Top Optimizers for Neural Networks

In this video, I cover 16 of the most popular optimizers used for training neural networks, starting from the basic Gradient Descent (GD), to the most recent ones, such as Adam, AdamW, and Lookahead.
#deeplearning #artificialintelligence
#neuralnetworks #computerscience
~~~~~~~~~~~~~~~
References
* Nestrov: proceedings.mlr.press/v28/sut...
* AdaGraD: www.jmlr.org/papers/volume12/...
* AdaDelta: arxiv.org/abs/1212.5701
* Adam & AdaMax: arxiv.org/abs/1412.6980
* AMSGrad: arxiv.org/abs/1904.03590
* AdaBound: arxiv.org/abs/1902.09843
* AdamW: arxiv.org/pdf/1711.05101v2.pdf
* Yogi: proceedings.neurips.cc/paper_...
* Nadam: openreview.net/pdf/OM0jvwB8jI...
* Lookahead: arxiv.org/abs/1907.08610

[@star-ro6oo]

Thank you for this video! Excellent review

[@benemdav]

This is fantastic! Thank you for taking the time to make this.

[@PyMLstudio]

Thanks for the feedback, glad you like the video!

[@fayezalhussein7115]

excellent

[@saqibsarwarkhan5549]

Great video. Thanks a lot.😊

[@PyMLstudio]

Glad you liked it!

[@MultiWarrr]

Super smart

[@SolathPrime]

Thanks for the explanation I wanna see a benchmark for them as an extension for his video

[@PyMLstudio]

Thanks for the suggestion, yes it’s a good idea and it makes sense. I’ll prepare a case study and benchmark different optimizers with it.

[@SolathPrime]

@@PyMLstudio
I'm working on one with graphs in mnist ds but for so reasons matplotlib doesn't work on my os
so I'm kinda re implementing the ploting algorithm my self in PILLOW

[@agenticmark]

you left out step and sine :D

[@shredder-31]

Great video❤
Can you send slides link pls

[@PyMLstudio]

Thanks for watching. This video was not made with PowerPoint. All the animations were made using Python and Manim package