Reinforcement Learning Series: Overview of Methods
Вставка
- Опубліковано 2 січ 2022
- This video introduces the variety of methods for model-based and model-free reinforcement learning, including: dynamic programming, value and policy iteration, Q-learning, deep RL, TD-learning, SARSA, policy gradient optimization, among others.
Citable link for this video: doi.org/10.52843/cassyni.jcgdvc
This is the overview in a series on reinforcement learning, following the new Chapter 11 from the 2nd edition of our book "Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control" by Brunton and Kutz
Book Website: databookuw.com
Book PDF: databookuw.com/databook.pdf
RL Chapter: faculty.washington.edu/sbrunt...
Amazon: www.amazon.com/Data-Driven-Sc...
Brunton Website: eigensteve.com
This video was produced at the University of Washington - Наука та технологія
I deeply appreciate the quality of knowledge you are providing to the community. please continue to democratise knowledge.
This russian doll of dichotomies has always been a mind bender, often it seems the literature has nebulous definitions and the boundaries aren't so clear. Thank you for the great insights in this lecture, the graphic is superb.
Thanks!
Superb. One of the things I always I struggle with when learning something is having a well structured map in my head of the topic and subtopics and this does an extremely good job of doing that. Many thanks.
OK, this year is gonna be better than I thought. Thanks, professor!
Wow! Steve, you've managed to break this all down into bite-sized chunks. Thank you 🙏
I just started getting back into RL so this comes at a perfect time! Looking forward 👌
It's been a great series of videos on RL! I'm updating my research interests and now I want to combine MPC with RL in such a way that the resulting control structure can be safely implemented and has some stability guarantees. Thank you very much!
Fantastic opening video. You're a talented teacher and I appreciate this content. Looking forward to watching the entire series.
You helped me during my undergrad, now you're an inspiration to me during my masters.
Great video! Though imho the on/off-policy distinction explained at 14:24 might be a bit misleading. I believe both on/off policy can explore sub-optimal actions with something like epsilon-greedy.
Such a high quality course and a free book in description? You're awesome!!
Finally! Thank you for you posting. Can't wait to see the whole playlist.
Please make a separate playlist for reinforcement learning :-)
Good call -- will do
@@Eigensteve please put the videos in order, the current order is not correct. But great content.
An excellent arrangement of a very tough topic, logical and in the proper flow, keep up the very good job
Thank you.
Really really great overview for those new to learning about reinforcement learning! Thanks so much!
The heart of AI is reinforcement learning, it is the only most interesting in whole AI/ml. Basically original AI .
Thanks professor 🤝👍
Great synthetic and dense video ! Thank you very much for sharing !
Coming to this video after a while. Really great video, thank you!!
Great didactic, congratulations! I used to confuse myself frequently when dealing with these concepts.
I am really grateful for your eye-opening videos, especially this one
Thanks!
What an amazing start to the new year! 😍
The dichotomy break down are so awesome...
This is series is life saver ! Thanks Steve !
0:00 Intro
3:00 Background
7:54 Model & Model-Free Reinforcement Learning (RL)
8:29 Markov Decision Process (MDP)
10:25 Nonlinear Dynamics
13:02 Gradient & Gradient-Free RL
14:05 Off-Policy (Q Learning) & On-Policy (SARSA) RL
17:23 Policy Gradient Optimization
18:05 Deep RL
Thank you! Happy New Year!
Needed this..... Thank you Professor
Dear steve its amazing category to classify the reinforcement learning thanks alot
Thank you prof for providing your book in PDF format
Great overview, which is just what I need, thank you sir!
Great teacher and master of the art!
You really invest perfectly the time in your lessons, very very useful! great series!
Thanks! :)
Hi Mr. Brunton,
Your videos are impressive and thank you for making the content. A small suggestion though, it will be better for us to navigate if you make separate playlists with orders in a particular content.
this explanation is just beautiful! Thx so much
This is amazing. Thank you, steve!
I am a happy owner of you "Data Driven Science and Engineering" book. That fact that there will be much more content on RL in the 2nd edition is really good news! Will there also be a print version of the 2nd edition of your and Kutz' Book?
Thanks! Yes, the print edition should be out sometime later this year.
Thanks so much for such great video. Can you please tell where does Proximal Policy Optimization (PPO) fit in these categories.
For my case a small game, I know that I will be using model free RL, but not able to decide what else to use apart from Q-Learning.
Great presentation 👏
Dear Steve, great explanation. However, just wanna confirm: I thought Actor-Critic is a model-free model?
Nice lectures and lots of stuffs to learn. Thanks for sharing. Are the On policy and Off policy somehow related to exploitation and exploration concept?
Very nice overview video! There is a small typo in the non-linear dynamic equation, the superflous dt on the right. Regarding how MPC fit in the whole DP framework, I remember Prof. Bertsekas was presenting it as a way to approximate cost-to-go online.
Good catch on the typo! And interesting perspective on MPC -- thanks!
thanks for your great videos
Great as always.
Great job Prof. Steve. How about multi-agent based DRL especially graph learning based RL. That can be a remarkable addition to your playlist.
Good stuff! You need to remove the "dt" on the right side of your nonlinear dynamics equation.
Thanks, Sir, Please add some robotics-related examples in the upcoming series also.
Thanks for the suggestion!
Thnx for video professor.
Steve, you are amazing.
I would like to have RL as a career , and you would be the best lecturer for a kickstart.
I have Energy data and I need to implement RL on these data (Inverter) to achieve the best result (when charge/discharge battery, when is the best time to feed in grid,etc.) which algorithm should I use for that ?
Thanks for the wonderful videos. It would be great if you add real code to the end of the main videos. it would be very easier to understand with detail.
I enjoy your broad strokes topics. I was wondering can an AI write to its memory once it learns or discovers something new ? Or it doesn't work like that.
Great series! Can you make a contro bootcamp like series for non-linear control theory? Would love to see some simplified explanations for topics like PDE backstepping, reference governors, lyapanov stability criteria etc.
Would love to do a bootcamp on this -- maybe a goal for the new year! :)
At 20:07 Steve said about the "model of system" that if we have a "model of system" we use the model-base, and if we don't have a "model of system" just use model-free. So, can you explain me more about "model of system". What is it? Example? and Why? Thank you so much.
Thank you!
Actor critic should be in the policy gradient optimization no?
Thank you for sharing.
I was reading 1 journal article and found that the author claimed following a model-free RL problem but they have used Markov Decision process to model the problem? they have not mentioned probabilities for states. What does that mean? Also there is another paper which used probabilities for state transitions and solved the problem using Q-learning, so it's all confusing again.
At 14:20 to 14:41, you state On-Policy is always playing the best game possible. Is this approach the same as greedily picking the best action at each state? If so, would On-Policy algorithms not include exploration such as epsilon-greedy?
The way I understood On-Policy vs Off-Policy here is that On-Policy is purely exploitation whereas Off-Policy is both exploration & exploitation. Am I misunderstanding it?
Thanks!
I was also a bit confused at that point. But, I think you have probably misunderstood it since, for RL, we have to always ensure exploration & exploitation. I feel that what I understand is completely the opposite to Prof. Steve's description: on-policy uses the to be improved policy to select an action (meaning not always the best action), while the off-policy uses a different policy to decide which action to take (you may always choose the best action).
I hope Steve will elaborate more on it. :)
Why is "Actor Critic" assigned (only) to the left side? Isn't it (also) a combination of gradient free and gradient based algorithms, e.g. the Critic is a DQN and the Actor is a Deep Policy Network?
I just have one doubt, is A3C a model free one?
Sir, it model free algorithm uses Marakove decision proce (MDP)?
There is a typo at 10:08, the dynamic model in continuous time should be dx/dt=f(x,u,t) only
Good catch, thanks!
What is difference between deterministic policy and stochastic policy?
The distinction between On-Policy and Off-Policy explained in this video seems to be different from other sources on the internet. I'm trying to get my head around reinforcement learning and I have noticed that different people have different understandings of certain concepts. Model-free and Model-based are also given a different distinction by others, and this really throws me off. I'm not saying the explanation in this video is incorrect, but that there are different explanations elsewhere and I'm not sure which one is correct.
Excellent. What other topics will be included in the 2nd edition?
Updates throughout, all code in Python and Matlab (with R and Julia online), and new chapters on RL and physics informed machine learning
oil lamp except the clear liquid is a gradient and the bubbles are parameters that perform an action. then its p>p' goes to q, then flipflops p q>q'. p'=p or q'=q if the AI improves at a particular task. two randoms cancel each other out on an error function and acts like an implicit rolles theorem without explicitly stating d/dx=0.
Hello Professor, where will DDPG algorithm sit in this chart?
hi steve, I think actor critic are usually considered model-free
Dear steve we can use ls algorithm instead of gradient algorithm isn’t it
The link for the new chapter of the 2nd edition of the book is not working for me. Can someone post the correct link in the comments?
Yeah it doesnt work :(
Maybe use databookuw.com/databook.pdf
Thank you, this link is working, but it still shows the 1st edition of the book I think.
@@csalahuni Shoot, sorry, here is the chapter: faculty.washington.edu/sbrunton/databookRL.pdf added to description too
@@Eigensteve tnx professor 🙏
Can anyone explain RL by comparing it with ML mathematically? I know much about ML but getting trouble understanding RL.
Excellent sunmary
Off plolicy..on policy is slightly confusing here...isnt off policy, a setup where you have the prior data and cant continuously interact with the environment
Dear professor, please explain to us about how to use reinforcement learning to tune pid gains ❤️
Im looking forward to hearing from you
Sincerely mohammad
Dear professor please explain to us how to use reinforcement learning to tune pid controller gains
I’m looking forward to hearing from you
Sincerely mohammad
Will we see the programs teaching each other? (like chess)
Sound level is low.
Bated breath. No, really.
Hurry uuuuh-uuuup!