The Unapologetic Mathematician

Mathematics for the interested outsider

The Singular Value Decomposition

Now the real and complex spectral theorems give nice decompositions of self-adjoint and normal transformations, respectively. Each one is of a similar form

\displaystyle\begin{aligned}S&=O\Lambda O^*\\H&=U\Lambda U^*\end{aligned}

where O is orthogonal, U is unitary, and \Lambda (in either case) is diagonal. What we want is a similar decomposition for any transformation. And, in fact, we’ll get one that even works for transformations between different inner prouct spaces.

So let’s say we’ve got a transformation M:A\rightarrow B (we’re going to want to save U and V to denote transformations). We also have its adjoint M^*:B\rightarrow A. Then M^*M:A\rightarrow A is positive-semidefinite (and thus self-adjoint and normal), and so the spectral theorem applies. There must be a unitary transformation V:A\rightarrow A (orthogonal, if we’re working with real vector spaces) so that

\displaystyle V^*M^*MV=\begin{pmatrix}D&0\\{0}&0\end{pmatrix}

where D is a diagonal matrix with strictly positive entries.

That is, we can break A up as the direct sum A=A_1\oplus A_2. The diagonal transformation D:A_1\rightarrow A_1 is positive-definite, while the restriction of V^*M^*MV to A_2 is the zero transformation. We will restrict V to each of these subspaces, giving V_1:A_1\rightarrow A and V_2:A_2\rightarrow A, along with their adjoints V_1^*:A\rightarrow A_1 and V_2^*:A\rightarrow A_2. Then we can write


From this we conclude both that V_1^*M^*MV_1=D and that MV_2=0. We define U_1=MV_1D^{-\frac{1}{2}}:A_1\rightarrow B, where we get the last matrix by just taking the inverse of the square root of each of the diagonal entries of D (this is part of why diagonal transformations are so nice to work with). Then we can calculate


This is good, but we don’t yet have unitary matrices in our decomposition. We do know that V_1^*V_1=1_{A_1}, and we can check that


Now we know that we can use V_2:A_2\rightarrow A to “fill out” V_1 to give the unitary transformation V. That is, V_1^*V_1=1_{A_1} (as we just noted), V_2^*V_2=1_{A_2} (similarly), V_1^*V_2 and V_2^*V_1 are both the appropriate zero transformation, and V_1V_1^*+V_2V_2^*=1_A. Notice that these are exactly stating that the adjoints V_1^* and V_2^* are the projection operators corresponding to the inclusions V_1 and V_2 in a direct sum representation of A as A_1\oplus A_2. It’s clear from general principles that there must be some projections, but it’s the unitarity of V that makes the projections be exactly the adjoints of the inclusions.

What we need to do now is to similarly fill out U_1 by supplying a corresponding U_2:B_2\rightarrow B that will similarly “fill out” a unitary transformation U. But we know that we can do this! Pick an orthonormal basis of A_1 and hit it with U_1 to get a bunch of orthonormal vectors in B (orthonormal because U_1^*U_1=1_{A_1}. Then fill these out to an orthonormal basis of all of B. Just set B_2 to be the span of all the new basis vectors, which is the orthogonal complement of the image of U_1, and let U_2 be the inclusion of B_2 into B. We can then combine to get a unitary transformation

\displaystyle U=\begin{pmatrix}U_1&U_2\end{pmatrix}

Finally, we define


where there are as many zero rows in \Sigma as we needed to add to fill out the basis of B (the dimension of B_2). I say that U\Sigma V^* is our desired decomposition. Indeed, we can calculate

\displaystyle\begin{aligned}U\Sigma V^*&=\begin{pmatrix}U_1&U_2\end{pmatrix}\begin{pmatrix}D^\frac{1}{2}&0\\{0}&0\end{pmatrix}\begin{pmatrix}V_1^*\\V_2^*\end{pmatrix}\\&=\begin{pmatrix}U_1D^\frac{1}{2}&0\end{pmatrix}\begin{pmatrix}V_1^*\\V_2^*\end{pmatrix}\\&=U_1D^\frac{1}{2}V_1^*=M\end{aligned}

where U and V are unitary on B and A, respectively, and \Sigma is a “diagonal” transformation (not strictly speaking in the case where A and B have different dimensions).

August 17, 2009 - Posted by | Algebra, Linear Algebra


  1. […] Meaning of the SVD We spent a lot of time yesterday working out how to write down the singular value decomposition of a transformation , […]

    Pingback by The Meaning of the SVD « The Unapologetic Mathematician | August 18, 2009 | Reply

  2. […] Okay, let’s take the singular value decomposition and do something really neat with it. Specifically, we’ll start with an endomorphism and […]

    Pingback by Polar Decomposition « The Unapologetic Mathematician | August 19, 2009 | Reply

  3. […] any linear transformation between two inner product spaces we can find orthonormal bases giving the singular value decomposition. There are two unitary (or orthogonal) transformations and and a “diagonal” […]

    Pingback by Decompositions Past and Future « The Unapologetic Mathematician | August 24, 2009 | Reply

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

Join 424 other followers

%d bloggers like this: