L2 Loss (Least Squares) - Pushforward/Jvp rule

The L2-norm loss (which arises as the Maximum Likelihood Estimate - MLE - under Gaussian/Normal error assumption) is typical for regression problems. Let's derive how we forward propagate tangents. Here are the notes: github.com/Ceyron/machine-lea...
The L2-loss computes a scalar metric by taking an element-wise difference between some guessed prediction and a reference, then element-wise squaring it and summing all the entries. If we were to additionally divide by the number of elements, we would get the MSE, the mean-squared error. Now imagine, this computation is part of a computational graph, for which you want to do forward-mode automatic differentiation. The necessary pushforward or Jacobian-vector product rule is what we are going to derive in this video.
Timestams:
00:00 What is the L2-loss ([nonlinear] Least-Squares loss)
00:45 Dimensionalities involved
01:11 Typical for regression problems (like Neural Networks for Scientific Machine Learning)
01:41 Task: Forward propagate tangent information
02:15 General Jacobian-vector product (pushforward)
03:22 Finding closed-form expression of Jacobian
04:14 Moving to index notation
08:25 Result back in symbolic notation
10:40 Full Pushforward rule
11:36 Computational considerations: storing the elementwise delta
12:01 Often there is no tangent on the reference vector
12:31 Summary
12:48 Outro
-------
📝 : Check out the GitHub Repository of the channel, where I upload all the handwritten notes and source-code files (contributions are very welcome): github.com/Ceyron/machine-lea...
📢 : Follow me on LinkedIn or Twitter for updates on the channel and other cool Machine Learning & Simulation stuff: / felix-koehler and / felix_m_koehler
💸 : If you want to support my work on the channel, you can become a Patreon here: / mlsim
🪙: Or you can make a one-time donation via PayPal: www.paypal.com/paypalme/Felix...
-------
⚙️ My Gear:
(Below are affiliate links to Amazon. If you decide to purchase the product or something else on Amazon through this link, I earn a small commission.)
- 🎙️ Microphone: Blue Yeti: amzn.to/3NU7OAs
- ⌨️ Logitech TKL Mechanical Keyboard: amzn.to/3JhEtwp
- 🎨 Gaomon Drawing Tablet (similar to a WACOM Tablet, but cheaper, works flawlessly under Linux): amzn.to/37katmf
- 🔌 Laptop Charger: amzn.to/3ja0imP
- 💻 My Laptop (generally I like the Dell XPS series): amzn.to/38xrABL
- 📱 My Phone: Fairphone 4 (I love the sustainability and repairability aspect of it): amzn.to/3Jr4ZmV
If I had to purchase these items again, I would probably change the following:
- 🎙️ Rode NT: amzn.to/3NUIGtw
- 💻 Framework Laptop (I do not get a commission here, but I love the vision of Framework. It will definitely be my next Ultrabook): frame.work
As an Amazon Associate I earn from qualifying purchases.
-------