Local Extrema in Multiple Variables
Just like in one variable, we’re interested in local maxima and minima of a function , where
is an open region in
. Again, we say that
has a local minimum at a point
if there is some neighborhood
of
so that
for all
. A maximum is similarly defined, except that we require
in the neighborhood. As I alluded to recently, we can bring Fermat’s theorem to bear to determine a necessary condition.
Specifically, if we have coordinates on given by a basis
, we can regard
as a function of the
variables
. We can fix
of these variables
for
and let
vary in a neighborhood of
. If
has a local extremum at
, then in particular it has a local extremum along this coordinate line at
. And so we can use Fermat’s theorem to draw conclusions about the derivative of this restricted function at
, which of course is the partial derivative
.
So what can we say? For each variable , the partial derivative
either does not exist or is equal to zero at
. And because the differential subsumes the partial derivatives, if any of them fail to exist the differential must fail to exist as well. On the other hand, if they all exist they’re all zero, and so
as well. Incidentally, we can again make the connection to the usual coverage in a multivariable calculus course by remembering that the gradient
is the vector that corresponds to the linear functional of the differential
. So at a local extremum we must have
.
As was the case with Fermat’s theorem, this provides a necessary, but not a sufficient condition to have a local extremum. Anything that can go wrong in one dimension can be copied here. For instance, we could define . Then we find
, which is zero at
. But any neighborhood of this point will contain points
and
for small enough
, and we see that
, so the origin cannot be a local extremum.
But weirder things can happen. We might ask that have a local minimum at
along any line, like we tried with directional derivatives. But even this can go wrong. If we define
we can calculate
which again is zero at . Along any slanted line through the origin
we find
and so the second derivative is always positive at the origin, except along the -axis. For the vertical line, we find
so along all of these lines we have a local minimum at the origin by the second derivative test. And along the -axis, we have
, which has the origin as a local minimum.
Unfortunately, it’s still not a local minimum in the plane, since any neighborhood of the origin must contain points of the form for small enough
. For these points we find
and so cannot have a local minimum at the origin.
What we’ll do is content ourselves with this analogue and extension of Fermat’s theorem as a necessary condition, and then develop tools that can distinguish the common behaviors near such critical points, analogous to the second derivative test.
Sunday Samples 148
When I talked about David Bowie last week, I of course had to mention one of his recurring characters, Major Tom. Interestingly enough, Bowie was not the only one to use this character. In 1983, German singer Peter Schilling wrote “Major Tom (Coming Home)” for his first English-language album, and it shot to the top of the charts in many places around the world (though it only peaked at 14 in the United States).
Some months back, a band called Shiny Toy Guns made a cover of “Major Tom (Coming Home)”, very much in keeping with the spirit of the original, but swapping out all the guitar, bass, and drum parts for synthesized lines, and replacing Schiller’s vocals by those of a female lead with a much cleaner, more antiseptic tone to match the theme. Unfortunately, nothing much seems to have been done with it beyond using it in a car commercial.
Read more »
The Implicit Function Theorem II
Okay, today we’re going to prove the implicit function theorem. We’re going to think of our function as taking an
-dimensional vector
and a
-dimensional vector
and giving back an
-dimensional vector
. In essence, what we want to do is see how this output vector must change as we change
, and then undo that by making a corresponding change in
. And to do that, we need to know how changing the output changes
, at least in a neighborhood of
. That is, we’ve got to invert a function, and we’ll need to use the inverse function theorem.
But we’re not going to apply it directly as the above heuristic suggests. Instead, we’re going to “puff up” the function into a bigger function
that will give us some room to maneuver. For
we define
just copying over our original function. Then we continue by defining for
That is, the new component functions are just the coordinate functions
. We can easily calculate the Jacobian matrix
where is the
zero matrix and
is the
identity matrix. From here it’s straightforward to find the Jacobian determinant
which is exactly the determinant we assert to be nonzero at . We also easily see that
.
And so the inverse function theorem tells us that there are neighborhoods of
and
of
so that
is injective on
and
, and that there is a continuously differentiable inverse function
so that
for all
. We want to study this inverse function to recover our implicit function from it.
First off, we can write for two functions:
which takes
-dimensional vector values, and
which takes
-dimensional vector values. Our inverse relation tells us that
But since is injective from
onto
, we can write any point
as
, and in this case we must have
by the definition of
. That is, we have
And so we see that , where
is the
-dimensional vector so that
. We thus have
for every
.
Now define be the collection of vectors
so that
, and for each such
define
, so
. As a slice of the open set
in the product topology on
, the set
is open in
. Further,
is continuously differentiable on
since
is continuously differentiable on
, and the components of
are taken directly from those of
. Finally,
is in
since
, and
by assumption. This also shows that
.
The only thing left is to show that is uniquely defined. But there can only be one such function, by the injectivity of
. If there were another such function
then we’d have
, and thus
, or
for every
.
The Implicit Function Theorem I
Let’s consider the function . The collection of points
so that
defines a curve in the plane: the unit circle. Unfortunately, this relation is not a function. Neither is
defined as a function of
, nor is
defined as a function of
by this curve. However, if we consider a point
on the curve (that is, with
), then near this point we usually do have a graph of
as a function of
(except for a few isolated points). That is, as we move
near the value
then we have to adjust
to maintain the relation
. There is some function
defined “implicitly” in a neighborhood of
satisfying the relation
.
We want to generalize this situation. Given a system of functions of
variables
we consider the collection of points in
-dimensional space satisfying
.
If this were a linear system, the rank-nullity theorem would tell us that our solution space is (generically) dimensional. Indeed, we could use Gauss-Jordan elimination to put the system into reduced row echelon form, and (usually) find the resulting matrix starting with an
identity matrix, like
This makes finding solutions to the system easy. We put our variables into a column vector and write
and from this we find
Thus we can use the variables
as parameters on the space of solutions, and define each of the
as a function of the
.
But in general we don’t have a linear system. Still, we want to know some circumstances under which we can do something similar and write each of the as a function of the other variables
, at least near some known point
.
The key observation is that we can perform the Gauss-Jordan elimination above and get a matrix with rank if and only if the leading
matrix is invertible. And this is generalized to asking that some Jacobian determinant of our system of functions is nonzero.
Specifically, let’s assume that all of the are continuously differentiable on some region
in
-dimensional space, and that
is some point in
where
, and at which the determinant
where both indices and
run from
to
to make a square matrix. Then I assert that there is some
-dimensional neighborhood
of
and a uniquely defined, continuously differentiable, vector-valued function
so that
and
.
That is, near we can use the variables
as parameters on the space of solutions to our system of equations. Near this point, the solution set looks like the graph of the function
, which is implicitly defined by the need to stay on the solution set as we vary
. This is the implicit function theorem, and we will prove it next time.
The Inverse Function Theorem
At last we come to the theorem that I promised. Let be continuously differentiable on an open region
, and
. If the Jacobian determinant
at some point
, then there is a uniquely determined function
and two open sets
and
so that
, and
is injective on
is defined on
,
, and
for all
is continuously differentiable on
The Jacobian determinant is continuous as a function of
, so there is some neighborhood
of
so that the Jacobian is nonzero within
. Our second lemma tells us that there is a smaller neighborhood
on which
is injective. We pick some closed ball
centered at
, and use our first lemma to find that
must contain an open neighborhood
of
. Then we define
, which is open since both
and
are (the latter by the continuity of
). Since
is injective on the compact set
, it has a uniquely-defined continuous inverse
on
. This establishes the first four of the conditions of the theorem.
Now the hard part is showing that is continuously differentiable on
. To this end, like we did in our second lemma, we define the function
along with a neighborhood of
so that as long as all the
are within
this function is nonzero. Without loss of generality we can go back and choose our earlier neighborhood
so that
, and thus that
.
To show that the partial derivative exists at a point
, we consider the difference quotient
with also in
for sufficiently small
. Then writing
and
we find
. The mean value theorem then tells us that
for some (no summation on
). As usual,
is the Kronecker delta.
This is a linear system of equations, which has a unique solution since the determinant of its matrix is . We use Cramer’s rule to solve it, and get an expression for our difference quotient as a quotient of two determinants. This is why we want the form of the solution given by Cramer’s rule, and not by a more computationally-efficient method like Gaussian elimination.
As approaches zero, continuity of
tells us that
approaches
, and thus so do all of the
. Therefore the determinant in the denominator of Cramer’s rule is in the limit
, and thus limits of the solutions given by Cramer’s rule actually do exist.
This establishes that the partial derivative exists at each
. Further, since we found the limit of the difference quotient by Cramer’s rule, we have an expression given by the quotient of two determinants, each of which only involves the partial derivatives of
, which are themselves all continuous. Therefore the partial derivatives of
not only exist but are in fact continuous.
Cramer’s Rule
We’re trying to invert a function which is continuously differentiable on some region
. That is we know that if
is a point where
, then there is a ball
around
where
is one-to-one onto some neighborhood
around
. Then if
is a point in
, we’ve got a system of equations
that we want to solve for all the .
We know how to handle this if is defined by a linear transformation, represented by a matrix
:
In this case, the Jacobian transformation is just the function itself, and so the Jacobian determinant
is nonzero if and only if the matrix
is invertible. And so our solution depends on finding the inverse
and solving
This is the approach we’d like to generalize. But to do so, we need a more specific method of finding the inverse.
This is where Cramer’s rule comes in, and it starts by analyzing the way we calculate the determinant of a matrix . This formula
involves a sum over all the permutations , and we want to consider the order in which we add up these terms. If we fix an index
, we can factor out each matrix entry in the
th column:
where the hat indicates that we omit the th term in the product. For a given value of
, we can consider the restricted sum
which is times the determinant of the
-
“minor” of the matrix
. That is, if we strike out the row and column of
which contain
and take the determinant of the remaining
matrix, we multiply this by
to get
. These are the entries in the “adjugate” matrix
.
What we’ve shown is that
(no summation on ). It’s not hard to show, however, that if we use a different row from the adjugate matrix we find
That is, the adjugate times the original matrix is the determinant of times the identity matrix. And so if
we find
So what does this mean for our system of equations? We can write
But how does this sum differ from the one
we used before (without summing on
) to calculate the determinant of
? We’ve replaced the
th column of
by the column vector
, and so this is just another determinant, taken after performing this replacement!
Here’s an example. Let’s say we’ve got a system written in matrix form
The entry in the th row and
th column of the adjugate matrix is calculated by striking out the
th column and
th row of our original matrix, taking the determinant of the remaining matrix, and multiplying by
. We get
and thus we find
where we note that
In other words, our solution is given by ratios of determinants:
and similar formulae hold for larger systems of equations.
Another Lemma on Nonzero Jacobians
Sorry for the late post. I didn’t get a chance to get it up this morning before my flight.
Brace yourself. Just like last time we’ve got a messy technical lemma about what happens when the Jacobian determinant of a function is nonzero.
This time we’ll assume that is not only continuous, but continuously differentiable on a region
. We also assume that the Jacobian
at some point
. Then I say that there is some neighborhood
of
so that
is injective on
.
First, we take points
in
and make a function of them
That is, we take the th partial derivative of the
th component function and evaluate it at the
th sample point to make a matrix
, and then we take the determinant of this matrix. As a particular value, we have
Since each partial derivative is continuous, and the determinant is a polynomial in its entries, this function is continuous where it’s defined. And so there’s some ball of
so that if all the
are in
we have
. We want to show that
is injective on
.
So, let’s take two points and
in
so that
. Since the ball is convex, the line segment
is completely contained within
, and so we can bring the mean value theorem to bear. For each component function we can write
for some in
(no summation here on
). But like last time we now have a linear system of equations described by an invertible matrix. Here the matrix has determinant
which is nonzero because all the are inside the ball
. Thus the only possible solution to the system of equations is
. And so if
for points within the ball
, we must have
, and thus
is injective.
Sunday Samples 147
Okay, everybody knows David Bowie. The glitz and glam and all the characters that are no more escapable as an influence anymore than anything the Beatles or the Rolling Stones ever did. And we all know that Ziggy played guitar.
But before Ziggy Stardust, Bowie came up through the same blues and Merseybeat beginnings as a whole generation of British artists. It’s how he departs from that background where things get interesting. The late-’60s psychedelia influenced his still largely-acoustic sound in “Space Oddity”, released to coincide with the moon landing in 1969 and introducing the character of Major Tom, to whom he (and other artists) would return many times. On 1970’s The Man Who Sold the World he started working with the backing band who would eventually become the Spiders, and they pushed into a sound more like then-current British heavy metal artists Led Zeppelin and Black Sabbath.
While that style didn’t stick, the album also saw Bowie incorporating more avant garde artistic influences and references. These did resonate in the developing work, and 1971’s Hunky Dory saw Bowie plunging headlong towards the ignition of his glam rock career in the following year’s The Rise and Fall of Ziggy Stardust and the Spiders from Mars. Hunky Dory brought back the pop-singer sound from Space Oddity and blended it with everything from Brecht to Dylan to Warhol, and threw in, Buddhism, Lovecraftian themes, references to Nietzsche-infused esoteric and hermetic philosophies, and even some outright, self-proclaimed nonsense.
Smack in the middle of all of this comes “Quicksand”, which survives to this day as a concert staple.
Read more »
A Lemma on Nonzero Jacobians
Okay, let’s dive right in with a first step towards proving the inverse function theorem we talked about at the end of yesterday’s post. This is going to get messy.
We start with a function and first ask that it be continuous and injective on the closed ball
of radius
around the point
. Then we ask that all the partial derivatives of
exist within the open interior
— note that this is weaker than our existence condition for the differential of
— and that the Jacobian determinant
on
. Then I say that the image
actually contains a neighborhood of
. That is, the image doesn’t “flatten out” near
.
The boundary of the ball
is the sphere of radius
:
Now the Heine-Borel theorem says that this sphere, being both closed and bounded, is a compact subset of . We’ll define a function on this sphere by
which must be continuous and strictly positive, since if then
, but we assumed that
is injective on
. But we also know that the image of a continuous real-valued function on a compact, connected space must be a closed interval. That is,
, and there exists some point
on the sphere where this minimum is actually attained:
.
Now we’re going to let be the ball of radius
centered at
. We will show that
, and is thus a neighborhood of
contained within
. To this end, we’ll pick
and show that
.
So, given such a point , we define a new function on the closed ball
by
This function is continuous on the compact ball , so it again has an absolute minimum. I say that it happens somewhere in the interior
.
At the center of the ball, we have (since
), so the minimum must be even less. But on the boundary
, we find
so the minimum can’t happen on the boundary. So this minimum of happens at some point
in the open ball
, and so does the minimum of the square of
:
Now we can vary each component of
separately, and use Fermat’s theorem to tell us that the derivative in terms of
must be zero at the minimum value
. That is, each of the partial derivatives of
must be zero (we’ll come back to this more generally later):
This is the product of the vector by the matrix
. And the determinant of this matrix is
: the Jacobian determinant at
, which we assumed to be nonzero way back at the beginning! Thus the matrix must be invertible, and the only possible solution to this system of equations is for
, and so
.
The Jacobian of a Composition
Let’s start today by introducing some notation for the Jacobian determinant which we introduced yesterday. We’ll write the Jacobian determinant of a differentiable function at a point
as
. Or, in more of a Leibnizean style:
We’re interested in determining the Jacobian of the composite of two differentiable functions. To which end, suppose and
are differentiable functions on two open regions
and
in
, with
, and let
be their composite. Then the chain rule tells us that
where each differential is an matrix, and the right-hand side is a matrix multiplication.
But these matrices are exactly the Jacobian matrices of the functions! And since the by definition, the determinant of the product of two matrices is the product of their determinants. That is, we find the equation
Or, we could define and use the Leibniz notation to write
As a special case, let’s assume that the differentiable function is injective in some open neighborhood
of a point
. That is, every
is sent to a distinct point by
, making up the whole image
. Further, let’s suppose that the function
which sends each point
back to the point in
from which it came —
if and only if
— is also differentiable. Then we have the composition
, and thus we find
or
Thus, if a differentiable function has a differentiable inverse function defined in some neighborhood of a point
, then the Jacobian determinant of the function must be nonzero at that point. A fair bit of work will now be put to turning this statement around. That is, we seek to show that if the Jacobian determinant
, then
has a differentiable inverse in some neighborhood of
.