Next up is a family of interesting representations that are also applicable to any group . The main ingredient is a subgroup — a subset of the elements of so that the inverse of any element in is also in , and the product of any two elements of is also in .
Our next step is to use to break up into cosets. We consider and to be equivalent if . It’s easy to check that this is actually and equivalence relation (reflexive, symmetric, and transitive), and so it breaks up into equivalence classes. We write the coset of all that are equivalent to as , and we write the collection of all cosets of as .
We should note that we don’t need to worry about being a normal subgroup of , since we only care about the set of cosets. We aren’t trying to make this set into a group — the quotient group — here.
Let’s work out an example to see this a bit more explicitly. For our group, take the symmetric group , and for our subgroup let . Indeed, is closed under both inversion and multiplication. And we can break up into cosets:
where we have picked a “transversal” — one representative of each coset so that we can write them down more simply. It doesn’t matter whether we write or , since both are really the same set. Now we can write down the multiplication table for the group action. It takes and , and tells us which coset falls in:
This is our group action. Since there are three elements in the set , the permutation representation we get will be three-dimensional. We can write down all the matrices just by working them out from this multiplication table:
It turns out that these matrices are the same as we saw when writing down the defining representation of . There’s a reason for this, which we will examine later.
As special cases, if , then there is one coset for each element of , and the coset representation is the same as the left regular representation. At the other extreme, if , then there is only one coset and we get the trivial representation.