Differentiability in higher dimensions

Review of differentiability in one-variable calculus

You learned in one-variable calculus that a function f : R → R may or may not be differentiable. In fact, a function may be differentiable at some places but not in others.

Consider the function r(i) that gives the output rate r of a neuron as a function of its input i. A neuron communicates to other neurons by sending output pulses to other neurons. In this simple model, we represent only the rate of these output pulses by the function r(i). Since a negative rate isn’t meaningful, we know that r(i) ≥ 0.

For this idealized neuron model, the neuron is completely silent when it has very little input, i.e., when the input i is small then the rate r(i) = 0. Only when the input exceeds some threshold I₀ does the neuron begins emitting output pulses. The output rates increases as i increases, as illustrated in this graph of r(i). (The curve levels off for high values of i because there is a limit on how fast the neuron can emit output pulses. For example, the limit might be 200 pulses per second.)

The function is differentiable everywhere except at the point where i = I₀. By differentiable, we mean that we can fit the graph of r(i) with a tangent line. Given any level of input, say i = a, we can find a line that closely approximates r(i) around a, as long as a≠I₀.

The function r(i) is not differentiable at i = I₀ because there is no tangent line at i = I₀. The graph of r has a kink there, so no matter what line we chose, it will fail to match the graph on either the left or the right side of I₀.

We can compute the equation of the tangent line at the point where i = a. It is simply

f(i) = r(a) + r'(a)(i - a),

(1)

where r'(a) is the derivative of r(i) at the point where i = a. We see that f(a) = r(a), so f(i) meets r(i) when i = a. Since r(i) is differentiable, we know that f(i) and r(i) are close to each other when i is near a. The fact that r(i) is differentiable means that it is nearly linear around i = a. In fact, we call the tangent line the linear approximation to r(i).

If we felt like it, we could use the tangent line to define the derivative of r(i). We could say that the derivative of r(i) at the point where i = a is the number m for which the line

f(i) = r(a) + m(i - a),

(2)

is tangent to the graph of r(i). The number m is the slope of the tangent line. (By comparing equations (1) and (2), you’ll see that, of course, m must be equal to r'(a).) This is, in effect, how we will define the derivative in higher dimensions.

Differentiability in two dimensions

Why did I go through that long-winded review of one-variable differentiability? To recast the derivative into the language we will use for multivariable differentiability.

To illustrate, let’s modify our neuron example. In turns out many neurons have “receptors” built right into them that respond to nicotine. For these neurons, the presence of nicotine alters their behavior. (Needless to say, many in the medical community are interested in the nicotine receptors as nicotine is a common drug of addiction.) We can model the effects of nicotine by defining a new function r : R² → R that gives the neural response as a function of both input i and nicotine level s. (I chose s for “smoke.”)

Let’s pretend that the effect of nicotine is to shift the threshold I₀ to smaller values and flatten out a neuron’s response to input. We write the response of a neuron to input i and nicotine s as r(i,s). If we look at the case when s = 0, we have the original curve r(i, 0) of neural output rate to input. If we add nicotine so some level, say s = 2 (in some arbitrary units), then the curve becomes r(i, 2), a shifted and flattened version of the original curve. Increasing nicotine further to s = 4 gives the further shifted and flattened curve r(i, 4).

To get a complete picture, we can plot the full function r(i,s). Here is a CVT of r(i,s), which you can rotate to view better.

One of the first things you may notice is that the surface r(i,s) is smooth except for a fold or crease along a line where r(i,s) becomes nonzero. This fold is analogous to the kink at I₀ that we saw in the original curve, above.

We want to define a notion of differentiability for our multivariable function r(i,s). As in the one-variable case, the function r(i,s) may be differentiable at some points and not at others. In fact, our definition of differentiability should distinguish the fold in the surface from the smooth parts of the surface. To be consistent with the one-variable case, the function r(i,s) should fail to be differentiable along the fold.

Given some point a = (a₁,a₂), the function r(i,s) is differentiable at the point where (i,s) = a if has a tangent plane at a. The definition of a tangent plane is analogous to the definition of a tangent line. At points (i,s) near a, the function r(i,s) is nearly identical to the tangent plane.

For example, the below graph shows that r(i,s) is differentiable at the point (i,s) = (3, 3) (shown by the green dot). By rotating the graph, you can see how the tangent plane touches the surface at the point (3, 3). On the other hand, if we tried to fit a plane at a point where the surface folds (e.g., the point shown by the red dot), we would never succeed. The plane will fail to match the graph on one side of the fold or the other. Hence the function r(i,s) is not differentiable at any point along the fold.

Let’s write down the form of the equation for the tangent plane. Since this reading is getting long, I’m just going to assert that we can write the equation for the tangent plane going through the point (i,s) = a where a = (a₁,a₂) as

f(i,s) = r(a₁,a₂) + m(i - a₁) + n(s - a₂).

(3)

The numbers m and n are the “slopes” of the plane in the i and s directions, respectively. Equation (3) is analogue of equation (2) for a function of two variables.

It turns out that this slopes m and n are the partial derivatives of r at a:

m = (a₁,a₂)
n = (a₁,a₂)

so we can write the equation of the tangent plane as

f(i,s) = r(a₁,a₂) + [ ]
∂r-
∂i(a1,a2)

(i - a₁) + [ ]
∂r-
∂s (a1,a2 )

(s - a₂).

(4)

Although it looks messier, I hope you can see that equation (4) is the analogue of equation (1) for a function of two variables.vector).

We’ll group the partial derivatives into a row matrix, called the matrix of partial derivatives of r at a = (a₁,a₂), and denoted by

Dr(a₁,a₂) =

IF the tangent plane of r(i,s) exists at (i,s) = (a₁,a₂), then Dr(a₁,a₂) is the derivative of r at (a₁,a₂). (The “IF” is important, and you can read more about some of the subtleties of differentiability in higher dimensions.)

In summary, if the function r(i,s) has a tangent plane at the point (i,s) = a, then it is differentiable at a. The slopes of the tangent plane are the partial derivatives of r. The matrix of partial derivatives Dr(a₁,a₂) is the derivative of the function r(i,s) at the point a. Equation (4) is the equation for the tangent plane. Just like in the one-variable case, the tangent plane f(i,s) is called the linear approximation to r(i,s). The fact that r(i,s) is differentiable means that it is close to its linear approximation around (i,s) = a.

If you made it this far, your neurons have probably been working overtime. You better let them relax for awhile. Given our current knowledge about nicotine, I can’t suggest using that to relax. Maybe going to some place warm would be in order.

Multivariable calculus and vector analysis

A set of on-line readings

Non-Math ML version