L2 Loss (Least Squares) - Pullback/vJp rule

Deriving the L2 loss is typically the first step in backpropagation for Neural Networks when applied to regression problems (as they are common in Scientific Machine Learning). Let's derive the corresponding pullback (vJp) rule. Here are the notes: github.com/Ceyron/machine-lea...
---
👉 This educational series is supported by the world-leaders in integrating machine learning and artificial intelligence with simulation and scientific computing, Pasteur Labs and Institute for Simulation Intelligence. Check out simulation.science/ for more on their pursuit of 'Nobel-Turing' technologies (arxiv.org/abs/2112.03235 ), and for partnership or career opportunities.
---
If we additionally divided by the number of elements in the vector, the L2-loss would be identical to the Mean Squared Error (MSE). Hence, this is a really common loss metric, typically used as the final computation when stacking operations in an Artificial Neural Network. As part of backpropagation, we start with a cotangent information on the loss (which in most cases will just be 1.0). This video derives how to pullback the 1.0 back to the cotangent on the guessed solution to then reversely propagate through the layers.
Timestamps:
00:00 A typical loss metric in regression problems
00:48 Discussing the primal computation
01:17 Task: Backward propagate cotangent information
01:51 Relevant for backpropagation in Neural Networks (as part of Deep Learning)
02:25 Often not interested in reference solution cotangent
02:51 General vector-Jacobian product (pullback rule
03:35 Finding a closed-form expression for the Jacobian
04:31 Changing to index notation
09:25 Back to symbolic notation
10:26 The other Jacobian
11:38 Plugging Jacobians into vJp rule
13:38 Full Pullback rule
15:03 Some remarks
15:45 Outro
-------
📝 : Check out the GitHub Repository of the channel, where I upload all the handwritten notes and source-code files (contributions are very welcome): github.com/Ceyron/machine-lea...
📢 : Follow me on LinkedIn or Twitter for updates on the channel and other cool Machine Learning & Simulation stuff: / felix-koehler and / felix_m_koehler
💸 : If you want to support my work on the channel, you can become a Patreon here: / mlsim
🪙: Or you can make a one-time donation via PayPal: www.paypal.com/paypalme/Felix...
-------
⚙️ My Gear:
(Below are affiliate links to Amazon. If you decide to purchase the product or something else on Amazon through this link, I earn a small commission.)
- 🎙️ Microphone: Blue Yeti: amzn.to/3NU7OAs
- ⌨️ Logitech TKL Mechanical Keyboard: amzn.to/3JhEtwp
- 🎨 Gaomon Drawing Tablet (similar to a WACOM Tablet, but cheaper, works flawlessly under Linux): amzn.to/37katmf
- 🔌 Laptop Charger: amzn.to/3ja0imP
- 💻 My Laptop (generally I like the Dell XPS series): amzn.to/38xrABL
- 📱 My Phone: Fairphone 4 (I love the sustainability and repairability aspect of it): amzn.to/3Jr4ZmV
If I had to purchase these items again, I would probably change the following:
- 🎙️ Rode NT: amzn.to/3NUIGtw
- 💻 Framework Laptop (I do not get a commission here, but I love the vision of Framework. It will definitely be my next Ultrabook): frame.work
As an Amazon Associate I earn from qualifying purchases.
-------

[@roro4787]

you are awsome with your content

[@MachineLearningSimulation]

Thanks a lot for the support on this autodiff series 😊