Simple reverse-mode Autodiff in Julia - Computational Chain

Reverse-Mode Automatic Differentiation (a generalization of backpropagation of Neural Networks) can be challenging to understand at first. Let's make clear by an easy hands-on implementation in the Julia programming language. Here is the code: github.com/Ceyron/machine-lea...
(Reverse-Mode) Automatic Differentiation is different from symbolic or numeric differentiation. It produces derivative estimates at machine precision by exploiting the computational graph (or in special cases just a computational chain) created for the evaluation of numerical computations on a computer.
00:00 Introduction
00:11 Define a simple function (+ interactive evaluation in REPL session)
00:40 Implement symbolic derivative
01:15 Check derivative by finite differences
01:38 We need reverse rules for backpropagation
01:55 Pullback/vJp rule for sine function
03:51 Pullback/vJp rule for exponential function
05:17 Implement reverse-mode engine
09:34 Setting up a computational chain for a first reverse-mode call
10:34 Obtaining derivative by evaluating vJp at 1.0
10:53 Syntactic sugar to get a derivative transformation
12:00 Outro
-------
📝 : Check out the GitHub Repository of the channel, where I upload all the handwritten notes and source-code files (contributions are very welcome): github.com/Ceyron/machine-lea...
📢 : Follow me on LinkedIn or Twitter for updates on the channel and other cool Machine Learning & Simulation stuff: / felix-koehler and / felix_m_koehler
💸 : If you want to support my work on the channel, you can become a Patreon here: / mlsim
🪙: Or you can make a one-time donation via PayPal: www.paypal.com/paypalme/Felix...
-------
⚙️ My Gear:
(Below are affiliate links to Amazon. If you decide to purchase the product or something else on Amazon through this link, I earn a small commission.)
- 🎙️ Microphone: Blue Yeti: amzn.to/3NU7OAs
- ⌨️ Logitech TKL Mechanical Keyboard: amzn.to/3JhEtwp
- 🎨 Gaomon Drawing Tablet (similar to a WACOM Tablet, but cheaper, works flawlessly under Linux): amzn.to/37katmf
- 🔌 Laptop Charger: amzn.to/3ja0imP
- 💻 My Laptop (generally I like the Dell XPS series): amzn.to/38xrABL
- 📱 My Phone: Fairphone 4 (I love the sustainability and repairability aspect of it): amzn.to/3Jr4ZmV
If I had to purchase these items again, I would probably change the following:
- 🎙️ Rode NT: amzn.to/3NUIGtw
- 💻 Framework Laptop (I do not get a commission here, but I love the vision of Framework. It will definitely be my next Ultrabook): frame.work
As an Amazon Associate I earn from qualifying purchases.
-------

[@liangli-wei9316]

Man this is great. Thanks!

[@MachineLearningSimulation]

Thanks a lot, you're welcome.

[@kefre10]

One time you evaluated the derivate at 1.0 instead of 2.0, why did you get the same result?

[@kefre10]

Ah it needs the cotangend at this point

[@MachineLearningSimulation]

Great question! 😊
Yes, you figured it out. The pullback at lines 57 and 61 give the derivative if evaluated at 1.0. Since they effectively evaluate the vector-Jacobian product we need to provide a vector (here just a scalar) to left-multiply the Jacobian with. Using just 1.0 gives the unscaled derivative.