Somebody finally explains that it's typically implemented by overloading basic math operations like + and sin(), where gradient values can be passed around along with the function value. I spent hours reading articles that never bothered with this essential bit of information.
Nice video, doing some research into the topic myself and must say this was summarized accurately.
Somebody finally explains that it's typically implemented by overloading basic math operations like + and sin(), where gradient values can be passed around along with the function value. I spent hours reading articles that never bothered with this essential bit of information.
So Automatic Differentiation (AD) produces exact gradients or approximations? I think he said at some point exact but the method doesn't seem exact?
Exact. Take a look at the code if you're more interested.
Exact up to machine precision
Check this for better background: ua-cam.com/video/vAp6nUMrKYg/v-deo.html
Is there a link to learn about the bayesian optimization for hyperparam opt?