So what happens when fails to be invertible? Its image must miss some vectors in . That is, we have a nontrivial kernel. The index of is zero, so a trivial kernel would mean a trivial cokernel. We would then have a one-to-one and onto linear transformation, and would be invertible.
Let’s take a basis of . Since this is a linearly independent set spanning a subspace of , it can be completed to a basis for all of . Now we can use this basis of to write out the matrix of and use our formula from last time to calculate .
The th column of the matrix is the vector written out in terms of our basis. But since the first few basis vectors are in the kernel of we have at least . So the first column of the matrix must be all zeroes. Now as we pick a permutation and walk down the rows, some row is going to tell us to multiply by the entry in the first column, which we have just seen is zero. That is, for every permutation, the term in our determinant formula for that permutation is zero, and so the sum of them all — the determinant itself — must also be zero.
Notice that this still lets us think of the determinant as preserving multiplications in the algebra of endomorphisms of . Any noninvertible linear transformation is sent to zero. The product of a noninvertible transformation and any other transformation will be noninvertible, and the product of their determinants will be zero. This also gives us a test for invertibility! Take the linear transformation and run it through the determinant function. If the result is zero, then is noninvertible. If the result is nonzero, then is invertible.