Neural ODEs - Pushforward/Jvp rule

Neural ODEs are inspired as the continuous-time analogy to ResNets. Although practically not as useful as the reverse-mode/adjoint-mode, this video will derive the pushforward/Jvp rule for forward-mode autodiff. Here are the notes github.com/Ceyron/machine-lea...
---
👉 This educational series is supported by the world-leaders in integrating machine learning and artificial intelligence with simulation and scientific computing, Pasteur Labs and Institute for Simulation Intelligence. Check out simulation.science/ for more on their pursuit of 'Nobel-Turing' technologies (arxiv.org/abs/2112.03235 ), and for partnership or career opportunities.
---
-------
📝 : Check out the GitHub Repository of the channel, where I upload all the handwritten notes and source-code files (contributions are very welcome): github.com/Ceyron/machine-lea...
📢 : Follow me on LinkedIn or Twitter for updates on the channel and other cool Machine Learning & Simulation stuff: / felix-koehler and / felix_m_koehler
💸 : If you want to support my work on the channel, you can become a Patreon here: / mlsim
🪙: Or you can make a one-time donation via PayPal: www.paypal.com/paypalme/Felix...
-------
⚙️ My Gear:
(Below are affiliate links to Amazon. If you decide to purchase the product or something else on Amazon through this link, I earn a small commission.)
- 🎙️ Microphone: Blue Yeti: amzn.to/3NU7OAs
- ⌨️ Logitech TKL Mechanical Keyboard: amzn.to/3JhEtwp
- 🎨 Gaomon Drawing Tablet (similar to a WACOM Tablet, but cheaper, works flawlessly under Linux): amzn.to/37katmf
- 🔌 Laptop Charger: amzn.to/3ja0imP
- 💻 My Laptop (generally I like the Dell XPS series): amzn.to/38xrABL
- 📱 My Phone: Fairphone 4 (I love the sustainability and repairability aspect of it): amzn.to/3Jr4ZmV
If I had to purchase these items again, I would probably change the following:
- 🎙️ Rode NT: amzn.to/3NUIGtw
- 💻 Framework Laptop (I do not get a commission here, but I love the vision of Framework. It will definitely be my next Ultrabook): frame.work
As an Amazon Associate I earn from qualifying purchases.
-------
Timestamps:
00:00 Neural ODEs as Final Time Integration
01:01 Typical Algorithms (e.g. RK 4/5)
01:23 Dimensionalities involved
02:13 We are only interested in the final state
03:23 Task: Forward-Propagation of tangent information
04:36 Find implicit rule without unrolling/piggybacking
06:14 General Pushforward/Jacobian-vector product rule
07:18 Assigning names to the tangent contributions
08:00 (1.1) General solution to an ODE
08:45 (1.2) Total derivative of general solution wrt parameter vector
10:27 (1.3) Multiplying with parameter tangent from the right
13:03 (1.4) Identifying the auxiliary tangent linear ODE for the first tangent contribution
15:44 (1.5) Discussing the tangent linear ODE for parameter propagation
18:52 (2.1) Again general solution to an ODE
20:01 (2.2) Total derivative wrt to inital condition
21:10 (2.3) Multiplying with the tangent initial condition from the right
23:02 (2.4) Identifying another auxiliary tangent linear ODE
24:04 (2.5) Discussing the found ODE
25:00 (3) Tangent contribution from the final time
28:14 Summary of the three tangent contributions
30:49 Full Pushforward rule
32:28 Joining the auxiliary tangent linear ODEs with the primal ODE
38:08 Auxiliary tangent linear problems will be at max as hard as the primal
39:55 Video summary
41:20 Outro

[@user-rt1qq5bm9e]

hey! it's me again! thank you for the video, i learnt a lot from you！RK415 is mentioned in many of your videos, I really want to know what is that

[@MachineLearningSimulation]

You're welcome, :) thanks for the kind comments. I'm very happy I can help. :)
RK4/5 refers to a Runge-Kutta 4-5 integrator. That is an explicit integration scheme for ordinary differential equations which adaptively chooses the size of its time step. In order to do this, it uses both a 4-th order and a 5-th order integrator to assess its error and iteratively reduces its time step until the error is below a prescribed tolerance. I would say it is the de-facto standard to solve most (well-behaved, not too stiff) ODE problems that arise in engineering. Note, that RK4/5 is just the general scheme, it requires specific values which are often arranged in a Butcher tableau. The standard ode integrator in MATLAB (ode45) and in Python's SciPy use the values due to Dormand & Prince (en.wikipedia.org/wiki/Dormand%E2%80%93Prince_method ). The anecdotal evidence from Julia's DifferentialEquations.jl package is that a recent improvement of these weights due to Tsitourias is most of the time more efficient discourse.julialang.org/t/tsit5-what-is-it/88237

[@user-rt1qq5bm9e]

@@MachineLearningSimulation Thank you for your detailed explanation! I will follow this to learn it!

[@user-ue5su2wj4g]

класс! cool

[@MachineLearningSimulation]

Thanks :)