Proximal Policy Optimization Explained

Every "what is proximal policy optimization?", well this is the video for you. Proximal Policy Optimization (PPO) is a reinforcement learning training method. It falls into the category of policy gradient methods, which is where a predictor is trained on a gradient derived directly from a reward function. PPO is sample efficient and very stable which makes it great from RL control problems like robotics and also many other tasks.
RL theory series: • Reinforcement Learning...
^ Watch the series above if you were confused
PPO paper: arxiv.org/abs/1707.06347
TRPO paper: arxiv.org/abs/1502.05477

[@James-qv1lh]

Insanely good video! Simple and straight to the point - thanks so much! :)

[@aramvanbergen4489]

Thank you for the clear explanation! But next time please use screenshots of the actual formulas this way it is much more readable.

[@carloscampo9119]

That was very, very well done. Thank you for the clear explanation.

[@boldizsarszabo883]

This video was super helpful and informative! Thank you so much for your effort!

[@alph4b3th]

Sensational! Dude, you explain in such a simple way! I was wondering what the difference was between deep Q-Learning and PPO, and I was looking for exactly a video like this. Congratulations on your great didactic way of explaining the basic mathematical concepts and abstracting them to a more intuitive approach; you are really very good at this! Excellent video!

[@sordesderisor]

If you also read the TRPO and PPO paper this video provides the perfect concise summary of PPO !

[@alexkonopatski429]

I really love your vids and I also love how you explain things! And could you pls maybe make a video about TRPO, 'cause it is a really complex thing to understand in my opinion and the lack of available resources makes the situation not better. Therefor, I and I think a lot of others would be really glad about a good explanation!
Thanks in advance

[@datonefaridze1503]

Thank you for your effort, i really appreciate it, you are working for us to learn, thanks

[@sayyidj6406]

i wish i know this channel sooner. thanks for video

[@FlapcakeFortress]

Much appreciated. Cheers!

[@LatpateShubhamManikrao]

Nicely explained man

[@GnuSnu]

4:25 "let me write it real quick" 💀💀

[@anibus1106]

Thank you so much, you save my day

[@marcotroster8247]

Just evaluate the derivative of the policy gradients. Only then, you can really understand why PPO works.
PPO adds the policy ratio as a factor to the derivative of the vanilla policy gradients. The clipping erases samples from the dataset with bad policy ratios because the derivative of a constant is zero.
Also you need to understand from advantage actor-critic that the sign of the advantage determines whether the probabilities increase or decrease. Given the same training data, positive advantages will increase probs for good actions and decrease probs for bad actions.
And the min always picks the clipped objective for bad policy ratios, so the gradients become constants. Otherweise they're the same and make only steps of policy ratios withing the epsilon bound. And because the policy gradients are multiplied by the policy ratio, this actually works as expected and gives PPO its stability.

[@vadimavkhimenia5806]

Can you make a video on maddpg with code?

[@awaisahmad5908]

Thanks

[@ivanwong863]

DQN is not an offline method is it?

[@EdanMeyer]

My bad, I meant to say it’s an off-policy method, q-learning performs very poorly an in offline setting

[@hemanthvemuluri9997]

for DQN you mean Offpolicy method right? DQN is not an Offline method.

[@underlecht]

you should have re-filmed this