# The Unapologetic Mathematician

## The Singular Value Decomposition

Now the real and complex spectral theorems give nice decompositions of self-adjoint and normal transformations, respectively. Each one is of a similar form

\displaystyle\begin{aligned}S&=O\Lambda O^*\\H&=U\Lambda U^*\end{aligned}

where $O$ is orthogonal, $U$ is unitary, and $\Lambda$ (in either case) is diagonal. What we want is a similar decomposition for any transformation. And, in fact, we’ll get one that even works for transformations between different inner prouct spaces.

So let’s say we’ve got a transformation $M:A\rightarrow B$ (we’re going to want to save $U$ and $V$ to denote transformations). We also have its adjoint $M^*:B\rightarrow A$. Then $M^*M:A\rightarrow A$ is positive-semidefinite (and thus self-adjoint and normal), and so the spectral theorem applies. There must be a unitary transformation $V:A\rightarrow A$ (orthogonal, if we’re working with real vector spaces) so that

$\displaystyle V^*M^*MV=\begin{pmatrix}D&0\\{0}&0\end{pmatrix}$

where $D$ is a diagonal matrix with strictly positive entries.

That is, we can break $A$ up as the direct sum $A=A_1\oplus A_2$. The diagonal transformation $D:A_1\rightarrow A_1$ is positive-definite, while the restriction of $V^*M^*MV$ to $A_2$ is the zero transformation. We will restrict $V$ to each of these subspaces, giving $V_1:A_1\rightarrow A$ and $V_2:A_2\rightarrow A$, along with their adjoints $V_1^*:A\rightarrow A_1$ and $V_2^*:A\rightarrow A_2$. Then we can write

$\displaystyle\begin{pmatrix}V_1^*\\V_2^*\end{pmatrix}M^*M\begin{pmatrix}V_1&V_2\end{pmatrix}=\begin{pmatrix}V_1^*M^*MV_1&V_1^*M^*MV_2\\V_2^*M^*MV_1&V_2^*M^*MV_1\end{pmatrix}=\begin{pmatrix}D&0\\{0}&0\end{pmatrix}$

From this we conclude both that $V_1^*M^*MV_1=D$ and that $MV_2=0$. We define $U_1=MV_1D^{-\frac{1}{2}}:A_1\rightarrow B$, where we get the last matrix by just taking the inverse of the square root of each of the diagonal entries of $D$ (this is part of why diagonal transformations are so nice to work with). Then we can calculate

\displaystyle\begin{aligned}U_1D^\frac{1}{2}V_1^*&=MV_1D^{-\frac{1}{2}}D^\frac{1}{2}V_1^*\\&=MV_1V_1^*\\&=MV_1V_1^*+MV_2V_2^*\\&=M\left(V_1V_1^*+V_2V_2^*\right)\\&=MVV^*=M\end{aligned}

This is good, but we don’t yet have unitary matrices in our decomposition. We do know that $V_1^*V_1=1_{A_1}$, and we can check that

\displaystyle\begin{aligned}U_1^*U_1&=\left(MV_1D^{-\frac{1}{2}}\right)^*MV_1D^{-\frac{1}{2}}\\&=D^{-\frac{1}{2}}V_1^*M^*MV_1D^{-\frac{1}{2}}\\&=D^{-\frac{1}{2}}DD^{-\frac{1}{2}}=1_{A_1}\end{aligned}

Now we know that we can use $V_2:A_2\rightarrow A$ to “fill out” $V_1$ to give the unitary transformation $V$. That is, $V_1^*V_1=1_{A_1}$ (as we just noted), $V_2^*V_2=1_{A_2}$ (similarly), $V_1^*V_2$ and $V_2^*V_1$ are both the appropriate zero transformation, and $V_1V_1^*+V_2V_2^*=1_A$. Notice that these are exactly stating that the adjoints $V_1^*$ and $V_2^*$ are the projection operators corresponding to the inclusions $V_1$ and $V_2$ in a direct sum representation of $A$ as $A_1\oplus A_2$. It’s clear from general principles that there must be some projections, but it’s the unitarity of $V$ that makes the projections be exactly the adjoints of the inclusions.

What we need to do now is to similarly fill out $U_1$ by supplying a corresponding $U_2:B_2\rightarrow B$ that will similarly “fill out” a unitary transformation $U$. But we know that we can do this! Pick an orthonormal basis of $A_1$ and hit it with $U_1$ to get a bunch of orthonormal vectors in $B$ (orthonormal because $U_1^*U_1=1_{A_1}$. Then fill these out to an orthonormal basis of all of $B$. Just set $B_2$ to be the span of all the new basis vectors, which is the orthogonal complement of the image of $U_1$, and let $U_2$ be the inclusion of $B_2$ into $B$. We can then combine to get a unitary transformation

$\displaystyle U=\begin{pmatrix}U_1&U_2\end{pmatrix}$

Finally, we define

$\displaystyle\Sigma=\begin{pmatrix}D^\frac{1}{2}&0\\{0}&0\end{pmatrix}$

where there are as many zero rows in $\Sigma$ as we needed to add to fill out the basis of $B$ (the dimension of $B_2$). I say that $U\Sigma V^*$ is our desired decomposition. Indeed, we can calculate

\displaystyle\begin{aligned}U\Sigma V^*&=\begin{pmatrix}U_1&U_2\end{pmatrix}\begin{pmatrix}D^\frac{1}{2}&0\\{0}&0\end{pmatrix}\begin{pmatrix}V_1^*\\V_2^*\end{pmatrix}\\&=\begin{pmatrix}U_1D^\frac{1}{2}&0\end{pmatrix}\begin{pmatrix}V_1^*\\V_2^*\end{pmatrix}\\&=U_1D^\frac{1}{2}V_1^*=M\end{aligned}

where $U$ and $V$ are unitary on $B$ and $A$, respectively, and $\Sigma$ is a “diagonal” transformation (not strictly speaking in the case where $A$ and $B$ have different dimensions).

August 17, 2009 - Posted by | Algebra, Linear Algebra