A very useful structure to have on a complex vector space carrying a representation of a group is an “invariant form”. To start with, this is a complex inner product , which we recall means that it is
- linear in the second slot —
- conjugate symmetric —
- positive definite — for all
Again as usual these imply conjugate linearity in the first slot, so the form isn’t quite bilinear. Still, people are often sloppy and say “invariant bilinear form”.
Anyhow, now we add a new condition to the form. We demand that it be
- invariant under the action of —
Here I have started to write as shorthand for . We will only do this when the representation in question is clear from the context.
The inner product gives us a notion of length and angle. Invariance now tells us that these notions are unaffected by the action of . That is, the vectors and have the same length for all and . Similarly, the angle between vectors and is exactly the same as the angle between and . Another way to say this is that if the form is invariant for the representation , then the image of is actually contained in the
orthogonal group [commenter Eric Finster, below, reminds me that since we've got a complex inner product we're using the group of unitary transformations with respect to the inner product : ].
More important than any particular invariant form is this: if we have an invariant form on our space , then any reducible representation is decomposable. That is, if is a submodule, we can find another submodule so that as -modules.
If we just consider them as vector spaces, we already know this: the orthogonal complement is exactly the subspace we need, for . I say that if is a -invariant subspace of , then is as well, and so they are both submodules. Indeed, if , then we check that is as well:
where the first equality follows from the -invariance of our form; the second from the representation property; and the third from the fact that is an invariant subspace, so .
So in the presence of an invariant form, all finite-dimensional representations are “completely reducible”. That is, they can be decomposed as the direct sum of a number of irreducible submodules. If the representation is irreducible to begin with, we’re done. If not, it must have some submodule . Then the orthogonal complement is also a submodule, and we can write . Then we can treat both and the same way. The process must eventually bottom out, since each of and have dimension smaller than that of , which was finite to begin with. Each step brings the dimension down further and further, and it must stop by the time it reaches .
This tells us, for instance, that there can be no inner product on that is invariant under the representation of the group of integers we laid out at the end of last time. Indeed, that was an example of a reducible representation that is not decomposable, but if there were an invariant form it would have to decompose.