Unrolled Autodiff of iterative Algorithms

When you have iterative parts in a computational graph (like optimization problems, linear solves, root-finding etc.) you can either unroll differentiate or implicitly differentiate them. The former has a counter-intuitive Jacobian converge (=Curse of Unrolling). Code: github.com/Ceyron/machine-lea...
Here is the paper, this video is based on: arxiv.org/abs/2209.13271
------
👉 This educational series is supported by the world-leaders in integrating machine learning and artificial intelligence with simulation and scientific computing, Pasteur Labs and Institute for Simulation Intelligence. Check out simulation.science/ for more on their pursuit of 'Nobel-Turing' technologies (arxiv.org/abs/2112.03235 ), and for partnership or career opportunities.
-------
📝 : Check out the GitHub Repository of the channel, where I upload all the handwritten notes and source-code files (contributions are very welcome): github.com/Ceyron/machine-lea...
📢 : Follow me on LinkedIn or Twitter for updates on the channel and other cool Machine Learning & Simulation stuff: / felix-koehler and / felix_m_koehler
💸 : If you want to support my work on the channel, you can become a Patreon here: / mlsim
🪙: Or you can make a one-time donation via PayPal: www.paypal.com/paypalme/Felix...
-------
Timestamps:
00:00 Intro
00:50 Iterative processes and two modes of autodiff
02:00 Considered problem
04:32 Code Setup
04:51 Inner objective and analytical optimizer
07:19 Gradient Descent optimization
10:07 Primal/Forward convergence
14:58 Unrolled Differentiation
18:58 Assess Jacobian convergence
21:05 Explaining the result
23:48 Example plot for explanation
26:00 About the inflection point
27:00 Curse of Unrolling
28:00 Outro

[@YOak_ML]

Great video!
I think at 12:10 it should be inner_objective(gd_iterates,THETA) instead of solution_suboptimality. In this case it is still working since the analytical solution is 0.

[@MachineLearningSimulation]

Thanks for the kind comment 😊.
Good catch at 12:10. 👍 I updated the file on GitHub: github.com/Ceyron/machine-learning-and-simulation/blob/main/english/adjoints_sensitivities_automatic_differentiation/curse_of_unrolling.ipynb

[@diegoandrade3912]

This dude is always on point ... keep it coming!

[@MachineLearningSimulation]

Thanks ❤️
More good stuff to come.

[@Michael-vs1mw]

Really cool stuff, with a clear explanation in 1D, that seems rare. Great work, thanks!

[@MachineLearningSimulation]

Thanks 😊

[@fenglongsong4760]

Super interesting observation and super clear explanation! Thanks for sharing it with us!

[@MachineLearningSimulation]

Thanks for the kind words 😊. Much appreciated ❤️

[@aryanpandey7835]

please make a video series on Graph neural network

[@MachineLearningSimulation]

Hi, thanks for the suggestion. 👍
Unfortunately, unstructured data is not my field of expertise. I want to delve into it at some point, but for now I want to stick with structured data.