I've Been Doing This Wrong The Whole Time ... The Right Way to Save Models In PyTorch

It turns out I wasn't saving PyTorch models correctly. You really need to save the optimizer state as well as the state of the current weights of your deep neural network. This is critical to getting deep Q learning and actor critic agents training on complex environments that may require multiple sessions spread over time.
Starter code for this video is here:
gist.github.co...
Learn how to turn deep reinforcement learning papers into code:
Get instant access to all my courses, including the new Prioritized Experience Replay course, with my subscription service. $29 a month gives you instant access to 42 hours of instructional content plus access to future updates, added weekly.
Discounts available for Udemy students (enrolled longer than 30 days). Just send an email to [e-mail]
www.neuralnet....
Or, pickup my Udemy courses here:
Deep Q Learning:
www.udemy.com/...
Actor Critic Methods:
www.udemy.com/...
Curiosity Driven Deep Reinforcement Learning
www.udemy.com/...
Natural Language Processing from First Principles:
www.udemy.com/...
Just getting started in deep reinforcement learning? Check out my intro level course through Manning Publications.
Reinforcement Learning Fundamentals
www.manning.co...
Here are some books / courses I recommend (affiliate links):
Grokking Deep Learning in Motion: bit.ly/3fXHy8W
Grokking Deep Learning: bit.ly/3yJ14gT
Grokking Deep Reinforcement Learning: bit.ly/2VNAXql
Come hang out on Discord here:
/ discord
Website: www.neuralnet.ai
Github: github.com/phi...
Twitter: / mlwithphil

[@forough2387]

Many thanks for updating us. I really appreciate you providing us with such wonderful courses.

[@MachineLearningwithPhil]

Thanks for the kind words

[@jose-alberto-salazar-jimenez]

Thanks for update. I wonder how would you go about loading the model, once trained, for "testing"? I've tried for example, loading the model state to q_eval, setting q_eval to ".eval()" mode, using "with torch.no_grad()", then getting the predictions with "model(observation)", but the model/agent doesn't perform as how it was doing during training (it does really bad in comparison).

[@adamjiang3044]

Hey Phil, how to save/reload the seed for numpy, env.seed, env.action_space.seed, T.cuda.manual_seed_all and other related seed setting? Just say if I need to run a long training, but have to switch over to different machines (very likely happen in HPC). Could you please provide some advice on this? Thanks in advance.

[@forough2387]

Dear Phil,
May I ask why q_next[done] = 0? Is not q_next a single value?

[@animesound]

this is gold, thank you very much

[@markadyash]

hey phil good to see you
what you working on RL or anything else?

[@MachineLearningwithPhil]

RL predominantly.

[@meowatthemoon6645]

Hi, thanks for the video. Can you explain what are the benefits of saving the state of the optimizer?

[@etterathe]

You have an answer in the description of the video

[@mikhailkhlyzov6205]

Saving optimizer allows for user to continue training from where he left. Most optimizers create some inner variables (e.g. momentum parameter in Adam, which depends on training history) that determine the way weights are being changed each step. Check out Adam's formula to understand what happens if we randomly zero-out the momentum mid training.
The old way Phil was saving models (only weights fo nets) was meant to allow "train on one machine and run on the other" and nothing else.
Of course for those who use SGD this is not as crucial LoL

[@meowatthemoon6645]

@@mikhailkhlyzov6205 Understood, thank you very much!

[@MrCmon113]

I did horrible on all of this organizational shit for by bachelor thesis, next time I wanna be organized from the start. Gonna watch what a bunch of different experienced people are doing.

[@DR-ot9hh]

You're an excellent teacher!

[@sayanchaudhuri2966]

thanks for the tips

[@killereks]

What are ur pc specs? Training is rly fast

[@MachineLearningwithPhil]

Intel i7 7820x with an rtx Titan

[@spsharan2000]

What keyboard do you use?

[@MachineLearningwithPhil]

Some hyper X model. I'm not a huge fan but it gets the job done.

[@fastaalemyapanadam]

Thanks for video.Please check your e mail