The Unapologetic Mathematician

Mathematics for the interested outsider

The Singular Value Decomposition

Now the real and complex spectral theorems give nice decompositions of self-adjoint and normal transformations, respectively. Each one is of a similar form

\displaystyle\begin{aligned}S&=O\Lambda O^*\\H&=U\Lambda U^*\end{aligned}

where O is orthogonal, U is unitary, and \Lambda (in either case) is diagonal. What we want is a similar decomposition for any transformation. And, in fact, we’ll get one that even works for transformations between different inner prouct spaces.

So let’s say we’ve got a transformation M:A\rightarrow B (we’re going to want to save U and V to denote transformations). We also have its adjoint M^*:B\rightarrow A. Then M^*M:A\rightarrow A is positive-semidefinite (and thus self-adjoint and normal), and so the spectral theorem applies. There must be a unitary transformation V:A\rightarrow A (orthogonal, if we’re working with real vector spaces) so that

\displaystyle V^*M^*MV=\begin{pmatrix}D&0\\{0}&0\end{pmatrix}

where D is a diagonal matrix with strictly positive entries.

That is, we can break A up as the direct sum A=A_1\oplus A_2. The diagonal transformation D:A_1\rightarrow A_1 is positive-definite, while the restriction of V^*M^*MV to A_2 is the zero transformation. We will restrict V to each of these subspaces, giving V_1:A_1\rightarrow A and V_2:A_2\rightarrow A, along with their adjoints V_1^*:A\rightarrow A_1 and V_2^*:A\rightarrow A_2. Then we can write

\displaystyle\begin{pmatrix}V_1^*\\V_2^*\end{pmatrix}M^*M\begin{pmatrix}V_1&V_2\end{pmatrix}=\begin{pmatrix}V_1^*M^*MV_1&V_1^*M^*MV_2\\V_2^*M^*MV_1&V_2^*M^*MV_1\end{pmatrix}=\begin{pmatrix}D&0\\{0}&0\end{pmatrix}

From this we conclude both that V_1^*M^*MV_1=D and that MV_2=0. We define U_1=MV_1D^{-\frac{1}{2}}:A_1\rightarrow B, where we get the last matrix by just taking the inverse of the square root of each of the diagonal entries of D (this is part of why diagonal transformations are so nice to work with). Then we can calculate

\displaystyle\begin{aligned}U_1D^\frac{1}{2}V_1^*&=MV_1D^{-\frac{1}{2}}D^\frac{1}{2}V_1^*\\&=MV_1V_1^*\\&=MV_1V_1^*+MV_2V_2^*\\&=M\left(V_1V_1^*+V_2V_2^*\right)\\&=MVV^*=M\end{aligned}

This is good, but we don’t yet have unitary matrices in our decomposition. We do know that V_1^*V_1=1_{A_1}, and we can check that

\displaystyle\begin{aligned}U_1^*U_1&=\left(MV_1D^{-\frac{1}{2}}\right)^*MV_1D^{-\frac{1}{2}}\\&=D^{-\frac{1}{2}}V_1^*M^*MV_1D^{-\frac{1}{2}}\\&=D^{-\frac{1}{2}}DD^{-\frac{1}{2}}=1_{A_1}\end{aligned}

Now we know that we can use V_2:A_2\rightarrow A to “fill out” V_1 to give the unitary transformation V. That is, V_1^*V_1=1_{A_1} (as we just noted), V_2^*V_2=1_{A_2} (similarly), V_1^*V_2 and V_2^*V_1 are both the appropriate zero transformation, and V_1V_1^*+V_2V_2^*=1_A. Notice that these are exactly stating that the adjoints V_1^* and V_2^* are the projection operators corresponding to the inclusions V_1 and V_2 in a direct sum representation of A as A_1\oplus A_2. It’s clear from general principles that there must be some projections, but it’s the unitarity of V that makes the projections be exactly the adjoints of the inclusions.

What we need to do now is to similarly fill out U_1 by supplying a corresponding U_2:B_2\rightarrow B that will similarly “fill out” a unitary transformation U. But we know that we can do this! Pick an orthonormal basis of A_1 and hit it with U_1 to get a bunch of orthonormal vectors in B (orthonormal because U_1^*U_1=1_{A_1}. Then fill these out to an orthonormal basis of all of B. Just set B_2 to be the span of all the new basis vectors, which is the orthogonal complement of the image of U_1, and let U_2 be the inclusion of B_2 into B. We can then combine to get a unitary transformation

\displaystyle U=\begin{pmatrix}U_1&U_2\end{pmatrix}

Finally, we define

\displaystyle\Sigma=\begin{pmatrix}D^\frac{1}{2}&0\\{0}&0\end{pmatrix}

where there are as many zero rows in \Sigma as we needed to add to fill out the basis of B (the dimension of B_2). I say that U\Sigma V^* is our desired decomposition. Indeed, we can calculate

\displaystyle\begin{aligned}U\Sigma V^*&=\begin{pmatrix}U_1&U_2\end{pmatrix}\begin{pmatrix}D^\frac{1}{2}&0\\{0}&0\end{pmatrix}\begin{pmatrix}V_1^*\\V_2^*\end{pmatrix}\\&=\begin{pmatrix}U_1D^\frac{1}{2}&0\end{pmatrix}\begin{pmatrix}V_1^*\\V_2^*\end{pmatrix}\\&=U_1D^\frac{1}{2}V_1^*=M\end{aligned}

where U and V are unitary on B and A, respectively, and \Sigma is a “diagonal” transformation (not strictly speaking in the case where A and B have different dimensions).

August 17, 2009 Posted by | Algebra, Linear Algebra | 3 Comments