Multivariable calculus and vector analysis

Taylor’s Theorem

Remember one-variable calculus Taylor’s theorem. Given a one variable function $f\left(x\right)$, you can fit it with a polynomial around $x=a$.

For example, the best linear approximation for $f\left(x\right)$ is

$$\begin{array}{lll}\hfill f\left(x\right)\approx f\left(a\right)+f{}^{\text{'}}\left(a\right)\left(x-a\right).& & \hfill \end{array}$$

This linear approximation fits $f\left(x\right)$ (shown in green below) with a line (shown in blue) through $x=a$ that matches the slope of $f$ at $a$.

We can add additional, higher-order terms, to approximate $f\left(x\right)$ better near $a$. The best quadratic approximation is

$$\begin{array}{lll}\hfill f\left(x\right)\approx f\left(a\right)+f{}^{\text{'}}\left(a\right)\left(x-a\right)+\frac{1}{2}f{}^{\text{'}\prime }\left(a\right){\left(x-a\right)}^2& & \hfill \end{array}$$

We could add third-order or even higher-order terms:

$$\begin{array}{lll}\hfill f\left(x\right)\approx f\left(a\right)+f{}^{\text{'}}\left(a\right)\left(x-a\right)+\frac{1}{2}f{}^{\text{'}\prime }\left(a\right){\left(x-a\right)}^2+\frac{1}{6}f{}^{\text{'}\prime \prime }\left(a\right){\left(x-a\right)}^3+\cdots .& & \hfill \end{array}$$

The important point is that this Taylor polynomial approximates $f\left(x\right)$ well for $x$ near $a$.

We want to generalize the Taylor polynomial to (scalar-valued) functions of multiple variables:

$$\begin{array}{lll}\hfill f\left(\mathbf{x}\right)=f\left(x_1,x_2,\dots ,x_n\right).& & \hfill \end{array}$$

We already know the best linear approximation to $f$. It involves the derivative,

$$\begin{array}{lll}\hfill f\left(\mathbf{x}\right)\approx f\left(\mathbf{a}\right)+Df\left(\mathbf{a}\right)\left(\mathbf{x}-\mathbf{a}\right).& & \hfill \text{ (1)}\end{array}$$

where $Df\left(\mathbf{a}\right)$ is the matrix of partial derivatives. The linear approximation is the first-order Taylor polynomial.

What about the second-order Taylor polynomial? To find a quadratic approximation, we need to add quadratic terms to our linear approximation. For a function of one-variable $f\left(x\right)$, the quadratic term was

$$\begin{array}{lll}\hfill \frac{1}{2}f{}^{\text{'}\prime }\left(a\right){\left(x-a\right)}^2.& & \hfill \text{ (2)}\end{array}$$

For a function of multiple variables $f\left(\mathbf{x}\right)$, what is analogous to the second derivative?

Since $f\left(\mathbf{x}\right)$ is scalar, the first derivative is $Df\left(\mathbf{x}\right)$, which we can view as a vector-valued function of $\mathbf{x}$. For the second derivative of $f\left(\mathbf{x}\right)$, we can take the matrix of partial derivatives of the function $Df\left(\mathbf{x}\right)$ (I suppose we could write this something like $DDf\left(\mathbf{x}\right)$ for the moment). This second derivative matrix is called the Hessian matrix of $f$, and is denoted $Hf\left(\mathbf{x}\right)$,

$$\begin{array}{lll}\hfill Hf\left(\mathbf{x}\right)=DDf\left(\mathbf{x}\right).& & \hfill \end{array}$$

When $f$ is a function of multiple variables, the second derivative term in the Taylor series will use the Hessian $Hf\left(\mathbf{a}\right)$. For the single-variable case, we could rewrite expression (2) as

$$\begin{array}{lll}\hfill \frac{1}{2}\left(x-a\right)f{}^{\text{'}\prime }\left(a\right)\left(x-a\right).& & \hfill \end{array}$$

The analog of this expression for the multivariable case is

$$\begin{array}{lll}\hfill \frac{1}{2}{\left(\mathbf{x}-\mathbf{a}\right)}^{T}Hf\left(\mathbf{a}\right)\left(\mathbf{x}-\mathbf{a}\right).& & \hfill \end{array}$$

The important point to remember is that we can add the above expression to our first-order Taylor polynomial (1) to obtain the second-order Taylor polynomial for functions of multiple variables:

$$\begin{array}{lll}\hfill f\left(\mathbf{x}\right)\approx f\left(\mathbf{a}\right)+Df\left(\mathbf{a}\right)\left(\mathbf{x}-\mathbf{a}\right)+\frac{1}{2}{\left(\mathbf{x}-\mathbf{a}\right)}^{T}Hf\left(\mathbf{a}\right)\left(\mathbf{x}-\mathbf{a}\right).& & \hfill \end{array}$$

The second-order Taylor polynomial is a better approximation of $f\left(\mathbf{x}\right)$ near $\mathbf{x}=\mathbf{a}$ than is the linear approximation (which is the same as the first-order Taylor polynomial). We’ll be able to use it for things such as finding a local minimum or local maximum of the function $f\left(\mathbf{x}\right)$.

You can read some examples here.

Contents by topic

1. Geometry in higher dimensions

2. Differential Calculus

2a. The partial derivative

2b. The derivative and differentiability

2c. The chain rule, directional derivative and gradient

2d. Applications of differential calculus

• Taylor's theorem
• Taylor polynomial example
• Local extrema
• Extrema examples

3. Integral calculus

4. Vector calculus

5. The fundamental theorems of vector calculus

6. Review material