Відео

Hi! Thanks for bringing this up! I've seen the equation written in both forms, but probably should've elected for the one suggested by you! This is what I was referring to for the equation: www.arxiv.org/abs/1609.04747

Awesome, thank you!

Wow, thank you!

I'm really glad to hear that!

Oh wow that's an interesting experiment to run! Glad you decided to try it out

Hi! I would recommend with setting up an IDE such as VSCode, along with a local Python environment with Anaconda. Then, you'll need to install and set up PyTorch, the instructions for which are in the repo. There are plenty of resources online about how to get started with programming so I would treat Google as your best friend!

i wonder if you can combine nag and fractional gradients to make a generally even better optimizer

Glad you liked it

Unfortunately, the code for the animations are not ready for the public haha. It's wayyy too messy. However, I didn't include the code for the optimizers because the equations are straight forward to implement, but how you use the gradients to update weights depends greatly on how the rest of the code is structured.

Hi! The total cost was about 2.8k although some parts I probably should’ve gone cheaper on like the motherboard. I have a full list of the parts in the description

@@sourishk07 Thank you! I did not notice the sheet in the description. Very Helpful!

I’m really glad you think so! Thanks

Love to hear it! Hopefully I’ll see you there

Haha love to see that. Thanks for watching!

I guess technically I didn’t talk about how the dataset was batched when performing GD, so no stochastic elements were touched upon. However, I just used SGD as a general term to talk about vanilla gradient descent, like how PyTorch and Tensorflow’s APIs are structured.

@@sourishk07 I see! It would be interesting to see if/how the stochastic element helps with the landscape l(x, y) = x^2 + a|y| or whatever that example was :)

Hi! Sorry, this is something I should've definitely clarified in the video! I've gotten a couple other comments about this as well. Everything in the formula is component-wise. You square each element in the gradient matrix individually & you perform component-wise division, along with the component-wise square root. Again, I really apologize for the confusion! I'll make sure to make these things clearer next time.

Appreciate the visit!

Thank you! And I’m glad you enjoyed it

Hi! Using reinforcement learning in the realm of optimizers is a fascinating concept and there's already research being done on it! Here are a couple cool papers that might be worth your time: 1. Learning to Learn by Gradient Descent by Gradient Descent (2016, Andrychowicz et al.) 2. Learning to Optimize (2017, Li and Malik) It would be fascinating to see GPT-4 help write more efficient optimizers though. LLMs helping accelerate the training process for other AI models seems like the gateway into AGI

@@sourishk07 Thanks for the answer!

Hello! Thanks for the comment. Optimizers like RMSProp and Adam do make step size dependent on gradient size, which I showcase in the video, so while there are other techniques to deal with slow convergence close to the optimum point due to small gradients, having these optimizers still help. Maybe I could've made this part clearer though. Also, from my understanding, learning rate decay is a pretty popular technique used so wouldn't that just slow down convergence even more as the learning rate decays & the loss approaches the area with smaller gradients? However, I definitely agree with your bigger point about these optimizers from preventing the loss from zig-zagging! In my RMSProp example, I do show how the loss is able to take a more direct route from the starting point to the minimum. Maybe I could've showcased a bigger example where SGD zig-zags more prominently to further illustrate the benefit that RMSProp & Adam bring to the table. I really appreciate you taking the time to give me feedback.

@@sourishk07 Yeah, I absolutely think the animations give good insight into the different strategies within "moment"-based optimizers. My point was more that even with "vanilla" gradient descent methods, the step sizes can be handled to not vanish as the gradient gets smaller, and that real benefit of the other methods is for altering the _direction_ of descent to deal with situations where eigenvalues of the (locally approximate) quatratic form differs in orders of magnitude. But I must also admit that (especially in the field of machine learning) the name SGD seem to be more or less _defined_ to include a fixed decay rate of step sizes, rather than just the method of finding a step direction (where finding step sizes would be a separate (sub-)problem), so your interpretation is probably more accurate than mine. Anyway, thanks for replying and I hope you continue making videos on the topic!

Thank you so much! I'm honored to hear that!

Hi! This is a fascinating point you bring up. I did say at the beginning that the scope of optimizers wasn't just limited to neural networks in high dimensions, but could also be applicable in lower dimensions. However, I probably should've added a section about saddle points to make this part of the video more thorough, so I really appreciate the feedback!

Haha stay tuned for a more upgraded one soon!

Hmm while this might be Adam's fault, I would encourage you to see if you can replicate the issue with SGD w/ Momentum or see if another optimizer without momentum solves it. I believe there are a wide array of reasons as to why this periodic behavior might emerge.

Hi! There seems to be many interesting papers about using metaheuristic approaches with machine learning, but I haven't seen too many applications of them in industry. However, this is a topic I haven't looked too deeply into! I simply wanted to discuss the strategies that are commonly used by modern day deep learning and maybe I'll make another video about metaheuristic approaches! Thanks for the idea!

Thanks so much! I've seen your videos before! I really liked your videos about Policy Gradients methods & Importance Sampling!!!

@@sourishk07 thanks! There was some hard work behind them, so I’m happy to hear they’re appreciated. But I don’t need to tell you that. This video is a master piece!

I really appreciate that coming from you!!

Thank you Gemini for watching, although I'm not sure you learned anything from this lol

Thank you so much. And there were so many topics I wanted to cram into this video but couldn't in the interest of time. That is a very interesting topic to cover and I'll add it to my list! Hopefully we can visit it soon :) I appreciate the idea

It really depends on the team! My work life balance is pretty good, but some nights I do have to work after I get back home!

Thank you so much!

Glad it was helpful! What concepts do you feel like you don’t understand yet?

I’m really glad to hear that! Thanks for those kind words

Yes, I think it’s a fun project for everyone to try out! I learned a lot of about hardware and the different softwares

Thank you so much for watching! Don't worry, I have many more videos like this planned! Stay tuned :)

If you're simply running main.py, then the checkpoints should be saved in the same directory as main.py under a folder titled 'output.' Let me know if that's what you were looking for!

@@sourishk07 what do I do if I can't find the output folder?

@@simsimhaningan Are there any errors while running main.py? My guess is you're not in the same folder as main.py when you run it. Make sure you're in the root directory of the repository when you run main.py!

LMAO idk man...

I'm really glad I was able to help!

Haha thank you!

Thank you for the feedback! These are great things to include in a part 2!

Thank you for those kind words! I'm glad you liked the video

Thanks!!!

Glad you enjoyed it

Hi! Thank you for the great video ideas. I'll definitely add those to my list!

I'm glad you enjoyed it! And thank you for the catch; The description has been updated!

Hello! That's a good question. Unfortunately, 70b models struggle to run. Llama 13b works pretty well. I think for my next server, I definitely want to prioritize more VRAM

The temperature has not been an issue with the same case size

LMAO

Haha thank you very much!

Thank you Raunak!

Don't expose me 😭😂

HAHAHA had to keep you on your toes!