When we say that a Lie algebra is nilpotent, another way of putting it is that for any sufficiently long sequence of elements of the nested adjoint
is zero for all . In particular, applying enough times will eventually kill any element of . That is, each is ad-nilpotent. It turns out that the converse is also true, which is the content of Engel’s theorem.
But first we prove this lemma: if is a linear Lie algebra on a finite-dimensional, nonzero vector space that consists of nilpotent endomorphisms, then there is some nonzero for which for all .
If then is spanned by a single nilpotent endomorphism, which has only the eigenvalue zero, and must have an eigenvector , proving the lemma in this case.
If is any nontrivial subalgebra of then is nilpotent for all . We also get an everywhere-nilpotent action on the quotient vector space . But since , the induction hypothesis gives us a nonzero vector that gets killed by every . But this means that for all , while . That is, is strictly contained in the normalizer .
Now instead of just taking any subalgebra, let be a maximal proper subalgebra in . Since is properly contained in , we must have , and thus is actually an ideal of . If then we could find an even larger subalgebra of containing , in contradiction to our assumption, so as vector spaces we can write for any .
Finally, let consist of those vectors killed by all , which the inductive hypothesis tells us is a nonempty collection. Since is an ideal, sends back into itself: . Picking a as above, its action on is nilpotent, so it must have an eigenvector with . Thus for all .
So, now, to Engel’s theorem. We take a Lie algebra consisting of ad-nilpotent elements. Thus the algebra consists of nilpotent endomorphisms on the vector space , and there is thus some nonzero for which . That is, has a nontrivial center — .
The quotient thus has a lower dimension than , and it also consists of ad-nilpotent elements. By induction on the dimension of we assume that is actually nilpotent, which proves that itself is nilpotent.