An introduction to Policy Gradient methods - Deep Reinforcement Learning
Вставка
- Опубліковано 21 тра 2024
- In this episode I introduce Policy Gradient methods for Deep Reinforcement Learning.
After a general overview, I dive into Proximal Policy Optimization: an algorithm designed at OpenAI that tries to find a balance between sample efficiency and code complexity. PPO is the algorithm used to train the OpenAI Five system and is also used in a wide range of other challenges like Atari and robotic control tasks.
If you want to support this channel, here is my patreon link:
/ arxivinsights --- You are amazing!! ;)
If you have questions you would like to discuss with me personally, you can book a 1-on-1 video call through Pensight: pensight.com/x/xander-steenbr...
Links mentioned in the video:
⦁ PPO paper: arxiv.org/abs/1707.06347
⦁ TRPO paper: arxiv.org/abs/1502.05477
⦁ OpenAI PPO blogpost: blog.openai.com/openai-baseli...
⦁ Aurelien Geron: KL divergence and entropy in ML: • A Short Introduction t...
⦁ Deep RL Bootcamp - Lecture 5: • Deep RL Bootcamp Lect...
⦁ RL-adventure PyTorch implementation: github.com/higgsfield/RL-Adve...
⦁ OpenAI Baselines TensorFlow implementation: github.com/openai/baselines - Наука та технологія
This is the best explanation of PPO on the net hands down
The value you provide in these videos is insane !
Thank you very much for guiding our learning process ;)
He is actually much better than Siraj Raval.
@@ahmadayazamin3313 what kind of scandal?
@@ahmadayazamin3313 I don't know it, just watch one or two videos of him demonstrate RL for trading
He seems to know what he's talking about.
He (Siraj) never explained any of what he was saying, and that was why I stopped watching him. He just rushed through the context and the results, explaining nothing.
@@oracletrading3000 'how to predict stock market in 5 minute'? More like, how to expose oneself as a fraud & end career in that time
I'm loving this RL series. Keep it up!
easily the best explanation of PPO I've ever seen. most papers and lectures get too tangled up in the probabilistic principals and advanced mathematic derivations and completely lose sight of what these models are doing in high level terms.
Explained so well and it was intuitive as well. I learnt more from this video than all the articles I found in the internet. Great job.
Amazing! This was the best explanation of PPO I have seen so far
Third video in a row. Really enjoy your work. Keep it up! And thank you!!!
This video was very well done, I definitely got a lot of value out of it. Thank you for your work!
Thank you so much for this video! This is way more insightful and intuitive than simply reading the papers!
I watched all your videos today, great works! Love them!
Thank you for including links for learning more on the description.
I actually understood your explanation cover to cover on first view and thought the 19 minutes felt more like 5.
Outstanding work.
One view, no pauses?! Not willing to be mean, but how can you be sure you've truly understood and weren't conquering Mount Stupid the whole time?
Dude, your channel is awesome! So glad I found it!
Great explanation with enough details! Thumbs up for all the free knowledge on the internet!
Fantastic review of policy gradients, and PPO as well! Best place for a refresh
As someone who is working in RL field .... you did very good job.
I watched this video more than 5 times and this was the best video about the PPO. Thank you for making great videos like this and keep up the good work. P.S: Your explanation was even simpler than the creator of this algorithm Schulman.)
Great breakdown of PPO. You've simplified a lot of complex concepts to make them understandable! Hahaha... and you can't beat an octopus slap!!!
Excellent video! Wonderful resource for anyone participating in AWS DeepRacer competitions.
Thank you for help me understand PPO faster, good explanation with useful resources included.
Thank you for the video, it is very helpful. The key concepts have been explained in just 20min, bravo. I would like to see more videos from your channel. Thank you.
Thank you for such a clear explanation. I was able to watch this at 2x speed and follow everything, which is a testament to your clarity. It really helped that you tied PPO back to previous work (e.g TRPO)
By far the best explanation on UA-cam.
Keep it up. Brevity is the soul of wit, it is indeed a skill to summarize the crux of a concept in such lucid way..!
This is really great! Keep up the good work!
This guy actually knows what he's talking about. Excellent video.
one of the best overview of PPO, clean.
best explanation of PPO I've found. Thanks
It was a great explanation!
Please do a video on Soft Actor-Critic and Maximum Entropy RL! That would be amazing!
This video is absolutely amazing!!
Wonderful, this is the first video i've seen on this channel. I suspect it won't be the last!
I love how you take the formula apart an look at it step by step. Great work!
keep up the good work , sir. thanks for this awesome explanation
Amazing explanation. Keep up the good work.
Excellent algorithm and explanation!
it is a long video, no doubt, but once you end watching it you think it was much better than actually reading the paper. thanks man!
I really love your video, professional and informative, thank.
great video for ppo! thanks a lot for you work!
Thank you! Learned a lot from this!
Love your videos!!
Great Video. Excellent intro to this topic .
12:19
Min operator also gets prefers old PO update
IF advantage is positive but probability of taking that action decreased, min operator selects unclipped objective here to undo the bad update
IF the advantage is negative but probability of taking that action increased, min operator also selects unclipped objective to undo bad update, just as mentioned in video.
more, just more videos. so well explained.
Thanks for this video! Really good teaching skills! :)
Pretty good explanation and very understandable thanks!
Coming back to this after thoroughly understanding Q-learning and looking into the advantage function in another network, this explanation is FAST, I wonder who would understand all that is happening without background knowledge
Well for AI/ML some background info is needed. If you taking multivariable calculus, its assumed you know calculus already. For those who already work in machine learning, this video is amazing. If i didn't get something i can research what he's talking about because he's using the proper technical terms, not dumbing it down. It's a wake up call for what I need to know to be knowledge. Great video.
*Great* video. *Great* explanation!
Very well explained, thank you
Thanks buddy, really appreciated!
Subscribed because the topics so cool!
Great explanation and references
Hat's off, mate. This is fantastic.
Thank you, it's really a good explanation
Amazing work :D
Congrats! You have a special skill to explain AI.
Great explanation!
Much cleaner than deep learning boot camp explanation
Thank you so much. Very helpful
awesome explanation !
Great explanation.
Thank you so much. I watch this videos the 10th time :-D
Great video! Thanx a lot
A big thanks for the video :)
very good explanations
I need to dive deeper into this.
Great video!
Great breakdown and links for additional resources
This is gold!!
Great video, but I have a couple of doubts:
1. In PPO, how does changing the objective help in restricting the updates to the policy? Wouldn’t it make more sense to restrict the gradient so that we don’t update the policy too much in one go?
2. In PPO, when A
YOU ARE AWESOME!
simply the best ever
great content keep up man
Sherlock xx
oh thank you so muchhhhhh
Amazing explanation.
Also. I just noticed that at 9:12 the seal slaps the guy with an octopus o.O
Hey! Your videos are great but wondering if you could go over some actual lines of code (implementation) for a new PPO video since this seems to be the algo that Openai uses as their default. Thanks for the link to the Pytorch implementation by RL-Adventure though! That is the clearest explanation of how to output mu and std from the actor network! I have been searching for 3 days for something like that because I just cannot implement from scratch based on my meager abilities.
Man you are good !
Thanks!
Hi, I'm really interested in your video and I found that these videos are really helpful for people who are learning RL in particular and A.I in general. Your way of representing the notion and key ideas behind these algorithms is amazing. It's too sad to found that you r not gonna update videos for 2 years. Do you have any another channel or anything I can learn from you. Please let me know :( It's my pleasure
I came here to say this
Great video as usual! Just a suggestion, maybe instead of diving directly into deep RL you can make videos (shorter if you don't have much time to devote) on simpler RL algorithms like DQN, Q-learning. That way someone who wants to know more about RL can build up the knowledge through more vanilla stuff. I admire your style and content like many others and would love to see it grow more :)
You're totally right that this is not easy to step into if you are new to RL, but I feel like there are tons and tons of good introduction resources out there for the simpler stuff.
I'm really trying to focus on the niche of people that already have this background and want to go a bit further :p
And I thing that makes this channel amazing. Practically following the state-of-art is a fantastic concept. As I am just curious about AI as a hobby and doing science in a different sector, I am glad, that I don't need to go through tons of articles by myself, but you show me the direction, where I should look to stay in a picture. Thank you.
I was about to say what @Arvix Insights is saying. There's an amazing book called: RL: An introduction by Richard S. Sutton and Andrew G. Barto, which introduces all the basic concepts that you're talking about. This second version of this book has been made publically available as a pdf and will be available on Amazin next month (don't quote me on the release date please :P).
I'm doing beginner level videos on my channel and would love your feedback... ua-cam.com/channels/rRTWfso9OS3D09-QSLA5jg.html
Sure, thanks for the videos :)
I watch Siraj Raval for the motivation, but I watch Arxiv Insights for the explanations
Just thought I would let you know that I just shared your video with my cohort in the Udacity Reinforcement Learning Nanodegree. We are going through PPO now and this video is relevant and timely - especially wrt to the clip region explanations. Any ideas on how to convert the outputs from discrete to continuous action space?
Sounds great, thx for sharing! Well as I mentioned, the PPO policy head outputs the parameters of a Gaussian distribution (so means and variances) for each action. At runtime, you can then sample from these distributions to get continuous output values and use the reparametrization trick to backpropagate gradients through this non-differentiable block --> check out my video on Variational Autoencoders for all the details on this!
I don't know if I'll get answers here but I have some questions:
1) Why are we taking the "min" in the loss function?
2) We are considering 1 in 1-e and 1+e because the reward we give for each positive action is 1, right? My question here is the scaling factor.
Can you do a video about DDPG?
Also, The PPO I know uses simply the discounted future returns in the loss. Is the variant using the Advantage instead the standard one?
Hi Andrew,
Can you please make a video explaining OpenAI's transformer model, Google's BERT & OpenAI's GPT&GPT-2 model?
I can't seem to wrap my head around them.
Hey @Arxiv Insights, I do have a basic question regarding Reinforcement Learning and would really appreciate your help. What is the basic difference between Reinforcement Learning, Deep Learning and Deep Reinforcement Learning? Does Basic Reinforcement Learning take advantage of Neural Networks to find the best solution and therefore uses Deep Learning? Thank you very much in advance, trying to get an overview and understand all the differences for my Master Thesis at the moment...
Very nice videos. FYI: Please watch at 0.75 speed for better understanding, LOL!
3:54 to 4:10 or so, why does that section remind me of the method used for ray marching in image rendering?
Video is very helpful, thank you! Personally I find the background music distracting and would prefer it if it didn't exist, but I know others may feel differently.
How do I tackle the moving target problem using methods from RL ? Where I have more than one reward ,3 possible actions to take and ofcourse state which include many factors / sources of information around the environment ,
Your help is highly appreciated
Hi can you talk about reinforcement learning in Adanet which automatically build the good structure of the neural network
Thanks for sharing this! I may be misunderstanding something, but it seems like there might be a mistake in the description. Specifically, the claim in 12:50 that "this is the only region where the unclipped part... has a lower value than the clipped version".
I think this claim might be wrong, because there could be another case where the unclipped version would be selected:
For example, if the ratio is e.g 0.5 (and we assume epsilon is 0.2), that would mean the ratio is smaller than the clipped version (which would be 0.8), and it would be selected.
Is that not the case?
Thanks
Great video! I had a question though, at 6:50, the objective function, which you called loss is actually the function that we'd want to maximize right? I mean calling it loss gave me the idea that we should minimize it. Correct me if I am wrong, please.
Yes, we are trying to maximise the advantage. It is called "loss" simply because it has the same function as (true) loss functions in other domains. It might get tricky when you implement a multi-head neural network with Actor-Critic-Methods where you combine different loss functions (GAE for the actor, lambda returns for the critic, entropy for exploration) as you have to make sure which "loss" you aim to maximise and which to minimise.
Very good thorough walkthrough ...but take a breath when your talking ...it takes a moment to catch up to what your saying sometimes but I guess thats the benefit of pause and play again. nice job!
Can I use this algorithm for "computation offloading in edge computing "
There is something I didn't understand : If you clip r then how will you do back propagation ?? the gradient will be just zero in the case r>1+epsilon or r
At 10:27 he says that first part in the min() expression is the "default policy gradient objective" but I do not really see how, since the objective function usually is J=E[R_t]. Does someone understand this?
16:46 How to apply gpu usage in PPO. For me, it was hard to implement collect experiences with cuda pytorch; Actually it seems even OpenAI didn't use gpu in the collecting process.
You need to vectorize the environments. PPO2 now does this (see twitter.com/openai/status/931226402048811008?lang=en). In my experience GPU usage is still very low during collection, however.
At 2:54 you talked about online vs offline learning, but the screen shows some comparisons between off-policy and on-policy learning. Otherwise cool video!