Problem
Consider a set of optimization problems in form of:
Given
\[ g(x,y,z) = 0\]
Find
\[\max(f(x,y,z))\]
Or
\[\min(f(x,y,z))\]
Method
Define
\[L(x,y,z,\lambda) = f(x,y,z)-\lambda g(x,y,z)\]
or in short
\[ L = f - \lambda g\] then any local maximum or minimum of the function should satisfy:
\[ \frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} = \frac{\partial L}{\partial z}= \frac{\partial L}{\partial \lambda}=0\]
Deduction
Consider the gradient of \(f\) that represents the "direction and rate of fastest increase":
\[\Delta f = (\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}, \frac{\partial f}{\partial z})^T\]
The reason is because
\[\text{d}f = \frac{\partial f}{\partial x}\text{d}x+\frac{\partial f}{\partial y}\text{d}y+\frac{\partial f}{\partial z}\text{d}z\]
so
\[\text{d}f = (\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}, \frac{\partial f}{\partial z})^T\times(\text{d}x,\text{d}y,\text{d}z) \]
\[(df)^2 = ((\frac{\partial f}{\partial x})^2 + (\frac{\partial f}{\partial y})^2+ (\frac{\partial f}{\partial z})^2)((\text{d}x)^2+(\text{d}y)^2+(\text{d}z)^2)\cos^2 \theta\]
\[(df)^2 =(\Delta f)((\text{d}x)^2+(\text{d}y)^2+(\text{d}z)^2)\cos^2 \theta\]
so when \(\cos \theta\) is largest, which means that the gradient is perpendicular to the "direction of the curved surface", the rate of change of the \(f\) function is the largest.
In this type of problems, \(g\) determines the direction of the curve surface. So the direction of \(g\) should be perpendicular to the gradient of \(f\) at the local maximum or minimum.
In other words,
\[ \Delta f = \lambda \Delta g\]
\[ \Delta f - \lambda \Delta g = 0\]
\[ \Delta L = 0\]
This is not a strict prove but it shows how this theory make sense.
Extended
If there are multiple constraints, then
\[ \frac{\partial L}{\partial x} = \frac{\partial L}{\partial y} = \frac{\partial L}{\partial z}= \frac{\partial L}{\partial \lambda_1} = \frac{\partial L}{\partial \lambda_2} = \cdots = \frac{\partial L} {\partial \lambda_n}=0\]
Prove or deduction is ommited because I don't know how to prove or deduce this.
Exercise
\[g(x,y) = x^2+y^2-1\]
\[f(x,y) = (x+1)(y+2)\]
\[L(x,y,\lambda) = (xy+2x+y+2)-\lambda(x^2+y^2-1)\]
Use the method
\[\frac{\partial L}{\partial x} = (y+2)-\lambda(2x)=0\]
\[\frac{\partial L}{\partial y} = (x+1)-\lambda(2y)=0\]
\[\frac{\partial L}{\partial \lambda} = x^2+y^2-1=0\]
clean
\[y+2-2\lambda x= 0\]
\[x+1-2 \lambda y= 0\]
\[x^2+y^2 = 1\]
solve to obtain
\[4x^4 + 4x^3 +x^2 -2x - 3 = 0\]
the final steps are left for the readers to complete