I’d like to go back and give a different proof that the Picard iteration converges — one which is closer to the spirit of Newton’s method. In that case, we proved that Newton’s method converged by showing that the derivative of the iterating function was less than one at the desired solution, making it an attracting fixed point.
In this case, however, we don’t have a derivative because our iteration runs over functions rather than numbers. We will replace it with a similar construction called the “functional derivative”, which is a fundamental part of the “calculus of variations”. I’m not really going to go too deep into this field right now, and I’m not going to prove the analogous result that a small functional derivative means an attracting fixed point, but it’s a nice exercise and introduction anyway.
So, we start with the Picard iteration again:
We consider what happens when we add an adjustment to :
We call the small change the “variation” of , and we write . Similarly, we call the difference between and the variation of and write . It turns out that controlling the size of the variation gives us some control over the size of the variation . To wit, if then we find
Now our proof that is locally Lipschitz involved showing that there’s a neighborhood of where we can bound by . Again we can pick a small enough so that implies that stays within this neighborhood, and also such that . And then we conclude that , which we can also write as
Now, admittedly this argument is a bit handwavy as it stands. Still, it does go to show the basic idea of the technique, and it’s a nice little introduction to the calculus of variations.