Professor I must sincerely thank you for the astonishing quality of this video. You were able to clearly explain an advanced concept without simplifying, going into the details and providing brilliant insights. Also I sincerely thank you for saving my GPA from my R.L. exam 😆
More casual example for TD-learning: Imagine a curious robot exploring a maze, searching for a hidden treasure. Unlike other methods that wait until it finds the treasure to learn, TD learning is all about learning on the fly. It uses what it already knows (like the estimated value of different paths) and immediate feedback (rewards) to improve its predictions about future moves. - The robot keeps track of a Q-value (Q(s_t, a_t)) for each path, which tells it how good it thinks that path is based on its past experiences. - When it takes a path and gets a Q-value (or reward) (like finding a clue), it compares that reward to what it expected (based on the Q-value - r_t + \gamma Q(s_{t + 1}, a_{t + 1})). This difference is called the prediction error. - If the reward is better than expected (positive error, or r_t + \gamma V^{old}(s_{t + 1}) - V^{old}(s_t) > 0), the robot increases the Q-value for that path, making it seem more attractive next time. - If the reward is worse than expected (negative error, or r_t + \gamma V^{old}(s_{t + 1}) - V^{old}(s_t) < 0), the robot decreases the Q-value, steering it away from less promising paths.
Prof. Burton, you are amazing. I never expected someone to take so much of time to explain a concept about TD. I'm one of the few people who hate reading text books to understand concepts. I rather see a video or learn about it from class. Thanks a lot
I do like the description of Q Learning. I had come up with another analogy for why it makes sense. If you took the action of going out to a party, and then happened to make some mistakes while there, we wouldn't want to say "you should never go out again." We'd want to reinforce the action of going out based on the best* possible outcome of that night, not the suboptimal action that was taken once there.
Thank you for the outstanding production quality and content of these lectures! I especially enjoy the structure diagram organizing the different RL methods.
Thank you dear Prof Brunton for this outstanding lecture. The detailed explanations and focus on subtleties are so important , Looking forward to your next videos.
Somehow I find that the explanations given by Prof. Brunton are easier to understand than those provided by video lectures from Stanford (which are also available on UA-cam).
Thank you so much for using very relevant analogies and very clear explanations. I think I have a much better grasp of the concepts behind Temporal Difference learning now.
These are fantastic lectures, I use these as an alternative explaination to David Silvers DeepmindxUCL 2015 lectures on the same topic, the different perspective really suits how my brain understands RL. Thank you!!
Great lecture! Concerning the difference between SARSA and Q-Learning, I didn't get the emphasis on Q-Learning being better for exploration. In principle, one can choose a epsilon-greedy for both methods. As a matter of fact, the SARSA method is defined in Sutton's book with an epsilon-greedy policy. I get the point that the TD target of Q-Learning does not depend on the policy itself and, therefore, is called an off-policy method. However, if one can choose a exploratory policy (e.g., epsilon-greedy) for both methods, why would SARSA be safer or less exploratory?
I got the impression he was asserting that the updates to the quality function can/will/often become an undesireable feedback loop when non-optimized states are used, and I would infer that means the training steps done on those states would have an undesireably high probability of entering such states. What you said does seem to be convincing evidence otherwise though.
That is a correct comment. The difference between the two lies in the Q-function updates. The way you choose your action is orthogonal (and can be more/less exploratory in either method). Also, from the video, Q-learning comes off as "better" method than SARSA, at least in problems where you don't need safe exploration, which is not accurate. It's more like a trade-off, where no method is clearly better. I love your videos in general. I think though, that this particular one needs a bit of a revision. Hope you don't see this as a critique, but rather as constructive feedback.
Thank you for the clear picture. It was really well explained and others already mentioned, now I can say that I understand these techniques quite fairly well. 🙏
Great videos. Definitely keep your face in the thumbnails. Even if the channel name doesn't ring a bell, sometimes I find your videos by just the thumbnails and always enjoy them
Hi Steve, Thanks for Amazing! lecture. I think the mail challenge in RL is designing our own custom environment (Multiple states and actions). It will be a great help if you can upload some lecture, suggest some link to do this job. Other comments are also welcome. Currently, I am doing some experiment on Retail pricing optimization using offline data. Looking forward.
Steve: thank you again. I appreciate your work. Trying to help, let me say that I believe there is a small typo, at minute 5:29 you wrote π(s,a) = argmax_a Q(s,a), should it be written π(s) = argmax_a Q(s,a)? Also, at time 22:35, the first equation has a sum over k, should it be over n? Anyway, this is a very good video.
One thing that seems to be either an error or just inconsistent notation is the use of TD(N) to mean an N-step TD. It seems like the value in the parentheses is supposed the value of lambda, not the number of steps of TD. TD(0) apparently should be read as, "N-step TD when lambda = 0, " while TD(1) means, "N-step TD when lambda = 1." I'm basing this off of the book "Reinforcement Learning: An Introduction - Second edition" by Sutton and Barto.
Great lecture, but I guess your definition of on/off policy is different from the definition of Sutton/Barto. On policy doesn't necessarily mean you always take the optimal action. "[on-policy] learns action values not for the optimal policy, but for a near-optimal policy that still explores" [excerpt from a different chapter but also valid for TD]. SARSA usually still follows a epsilon-greedy strategy.
Read the chapter, and I've been waiting for this video for a while. Happy to know I'm the first to comment :) Thanks, Steve. Can you explain more the bias (and especially the variance) in RL in a later video ?
@12:55 if pi were not made optimal the game would converge slowly the agent doesn't explore enough and acting outcomes become bounded in a local optimum with low next step rewards.
For MC you only get the reward at the end and then divide it up among all the states. But for TD, if you are only taking one step forward, where does the reward come from? A little confused here.
Hello :) thank you for the video. I have a small question. I don't fully understand the difference between TD(0) and SARSA. Indeed, if SARSA uses the optimal action 'a' at each time step 'k', doesn't the Q-function in SARSA equal the Value function in TD(0) ? Or was there an error in my understanding ? Can you please help me see more clearly ? :)
In the Monte Carlo method, you have the reward discounted by gamma. Why do you discount a reward function for the entire episode? Furthermore, 1/n(R) would not the average reward for any step k.
there is a question that pops up in my head; if SARSA is an on-policy method, then is it OK to use e-greedy algorithm in SARSA? as you mentioned it always take into account taking the safe and on-policy action rather than random and off-policy actions?
One question, in terms of updating the Q function using the observed(real) reward at state k + 1, how do we know the observed(real) reward at state k + 1 since it is one timestamp in future?
what is the reward for r_k if we don't receive a reward after every action? Is it just assumed zero and the value is based only off the quality function?
This is amazing content. I just have a question (still struggle with the concept of on and off policy)... at 30:18, the max Q (in Q-learning) .. which Q is it old or new ?
Instead of using so many words to explain, why couldn't you just use a couple of examples to explain the relationship between Q(s, a) new vs Q(s', a')? It would be so easy to understand through examples.
I personally love the big picture perspective that Prof. Brunton always shows. Please, continue to make these high quality videos!
👍👍👍
Professor I must sincerely thank you for the astonishing quality of this video. You were able to clearly explain an advanced concept without simplifying, going into the details and providing brilliant insights. Also I sincerely thank you for saving my GPA from my R.L. exam 😆
More casual example for TD-learning:
Imagine a curious robot exploring a maze, searching for a hidden treasure. Unlike other methods that wait until it finds the treasure to learn, TD learning is all about learning on the fly. It uses what it already knows (like the estimated value of different paths) and immediate feedback (rewards) to improve its predictions about future moves.
- The robot keeps track of a Q-value (Q(s_t, a_t)) for each path, which tells it how good it thinks that path is based on its past experiences.
- When it takes a path and gets a Q-value (or reward) (like finding a clue), it compares that reward to what it expected (based on the Q-value - r_t + \gamma Q(s_{t + 1}, a_{t + 1})). This difference is called the prediction error.
- If the reward is better than expected (positive error, or r_t + \gamma V^{old}(s_{t + 1}) - V^{old}(s_t) > 0), the robot increases the Q-value for that path, making it seem more attractive next time.
- If the reward is worse than expected (negative error, or r_t + \gamma V^{old}(s_{t + 1}) - V^{old}(s_t) < 0), the robot decreases the Q-value, steering it away from less promising paths.
Prof. Burton, you are amazing. I never expected someone to take so much of time to explain a concept about TD. I'm one of the few people who hate reading text books to understand concepts. I rather see a video or learn about it from class.
Thanks a lot
I was hoping that your next video would have been about Q-learning, and here it comes!
I do like the description of Q Learning. I had come up with another analogy for why it makes sense. If you took the action of going out to a party, and then happened to make some mistakes while there, we wouldn't want to say "you should never go out again." We'd want to reinforce the action of going out based on the best* possible outcome of that night, not the suboptimal action that was taken once there.
Thank you for the outstanding production quality and content of these lectures! I especially enjoy the structure diagram organizing the different RL methods.
Thank you dear Prof Brunton for this outstanding lecture. The detailed explanations and focus on subtleties are so important , Looking forward to your next videos.
I enjoy your talks. They are very clear and well structured and have the right level of detail. Thank you,
Somehow I find that the explanations given by Prof. Brunton are easier to understand than those provided by video lectures from Stanford (which are also available on UA-cam).
This is the best RL tutorial on the internet.
Thank you so much for using very relevant analogies and very clear explanations. I think I have a much better grasp of the concepts behind Temporal Difference learning now.
Thank you! It's a great video. My understanding in TD learning was deepened a lot.
CS PhD student here. This video provides such amazing content. Highly recommended.
These are fantastic lectures, I use these as an alternative explaination to David Silvers DeepmindxUCL 2015 lectures on the same topic, the different perspective really suits how my brain understands RL. Thank you!!
The video quality is incredible lol and all the concept is discussed extremely clear OMG!!
Brilliant masterpiece bro KEEP GOING !!
Great lecture! Concerning the difference between SARSA and Q-Learning, I didn't get the emphasis on Q-Learning being better for exploration. In principle, one can choose a epsilon-greedy for both methods. As a matter of fact, the SARSA method is defined in Sutton's book with an epsilon-greedy policy. I get the point that the TD target of Q-Learning does not depend on the policy itself and, therefore, is called an off-policy method. However, if one can choose a exploratory policy (e.g., epsilon-greedy) for both methods, why would SARSA be safer or less exploratory?
I got the impression he was asserting that the updates to the quality function can/will/often become an undesireable feedback loop when non-optimized states are used, and I would infer that means the training steps done on those states would have an undesireably high probability of entering such states.
What you said does seem to be convincing evidence otherwise though.
That is a correct comment. The difference between the two lies in the Q-function updates. The way you choose your action is orthogonal (and can be more/less exploratory in either method). Also, from the video, Q-learning comes off as "better" method than SARSA, at least in problems where you don't need safe exploration, which is not accurate. It's more like a trade-off, where no method is clearly better.
I love your videos in general. I think though, that this particular one needs a bit of a revision. Hope you don't see this as a critique, but rather as constructive feedback.
this was the best explanation ever!
thank you so much, professor!
Thank you for the clear picture. It was really well explained and others already mentioned, now I can say that I understand these techniques quite fairly well. 🙏
Excellent class! Extremely easy to understand!
Thanks a lot. Not just math but also the intuition that i was looking for
Great videos. Definitely keep your face in the thumbnails. Even if the channel name doesn't ring a bell, sometimes I find your videos by just the thumbnails and always enjoy them
This whole series was good, but this one pushed me past my confusion. My neural network finally learned to play tic-tac-toe!
Hi Prof. Brunton. Great vídeo as always! Please keep producing quality ML content
Thanks a bundle Steve, this was really well explained!
Hi Steve, Thanks for Amazing! lecture. I think the mail challenge in RL is designing our own custom environment (Multiple states and actions). It will be a great help if you can upload some lecture, suggest some link to do this job. Other comments are also welcome. Currently, I am doing some experiment on Retail pricing optimization using offline data. Looking forward.
You gave the best explanations I've ever seen!
I very like your descriptions about deep learning
I'm so much very grateful for these videos you make. Keep on the good work.
Thanksssss steveeee. I couldnt understand nothing before this video.Thanks again
being a phd student of you must be a gift :D
Steve: thank you again. I appreciate your work. Trying to help, let me say that I believe there is a small typo, at minute 5:29 you wrote π(s,a) = argmax_a Q(s,a), should it be written π(s) = argmax_a Q(s,a)? Also, at time 22:35, the first equation has a sum over k, should it be over n? Anyway, this is a very good video.
brilliant explanation
One thing that seems to be either an error or just inconsistent notation is the use of TD(N) to mean an N-step TD. It seems like the value in the parentheses is supposed the value of lambda, not the number of steps of TD. TD(0) apparently should be read as, "N-step TD when lambda = 0, " while TD(1) means, "N-step TD when lambda = 1." I'm basing this off of the book "Reinforcement Learning: An Introduction - Second edition" by Sutton and Barto.
The videos are wonderful! Thank you, professor.
Great lecture, but I guess your definition of on/off policy is different from the definition of Sutton/Barto. On policy doesn't necessarily mean you always take the optimal action. "[on-policy] learns action values not for the optimal policy, but for a near-optimal policy that still explores" [excerpt from a different chapter but also valid for TD].
SARSA usually still follows a epsilon-greedy strategy.
Thanks a lot, Prof. Brunton!
woww thank you , so well explained with lot of patience .. god bless
Thank you so much for these lectures sir!
Excellent Explaination
Read the chapter, and I've been waiting for this video for a while. Happy to know I'm the first to comment :) Thanks, Steve. Can you explain more the bias (and especially the variance) in RL in a later video ?
Such a great video, thank you!
Thanks for the fantastic explanation!
Loud and clear.
@12:55 if pi were not made optimal the game would converge slowly the agent doesn't explore enough and acting outcomes become bounded in a local optimum with low next step rewards.
Great explanation..very clear!
Very well explained.
ty 4 great explain greeze from switzerland :)
In value able Content.. Cant thank enough.
For MC you only get the reward at the end and then divide it up among all the states. But for TD, if you are only taking one step forward, where does the reward come from? A little confused here.
Could you please explain it with some examples, that will be really helpful to understand these formulas, thanks!
Help a lot with my AI course final!!!!
Great video! Would be nice a simple example
Hello :) thank you for the video. I have a small question. I don't fully understand the difference between TD(0) and SARSA. Indeed, if SARSA uses the optimal action 'a' at each time step 'k', doesn't the Q-function in SARSA equal the Value function in TD(0) ? Or was there an error in my understanding ? Can you please help me see more clearly ? :)
Brilliant! Thanks 🙏
In the Monte Carlo method, you have the reward discounted by gamma. Why do you discount a reward function for the entire episode?
Furthermore, 1/n(R) would not the average reward for any step k.
Nice talk al always. A question: at minute 5:00, does pi depend on a? It shoud not, right? The same holds for the previous video.
For TD0, Why not say we do alpha x the new value (s_k+1) + (1 - alpha) x the old value (s_k). It's a very basic update method...
I would like to know what is being used to project the equations. It is apparently not video editing because he points directly to them.
there is a question that pops up in my head; if SARSA is an on-policy method, then is it OK to use e-greedy algorithm in SARSA? as you mentioned it always take into account taking the safe and on-policy action rather than random and off-policy actions?
Great lecture!
Can you explain how off policy q learning can take a suboptimal route sometimes if you are always taking the max of the actions presented?
One question, in terms of updating the Q function using the observed(real) reward at state k + 1, how do we know the observed(real) reward at state k + 1 since it is one timestamp in future?
22:35 As someone else mentioned, the first equation has a sum over k, shouldn't it be over n?
very useful
what is the reward for r_k if we don't receive a reward after every action? Is it just assumed zero and the value is based only off the quality function?
Thank youuu !
Pi of s,a should only be a function of s since the RHS calculates a. What am I missing?
Great stuff
I cannot access the new chapter in the 2nd edition. Has anybody accessed the link?
You are the best:))))))
wow, nice
top prof!
Did not understand the TD part
👍
A+
I don't think the comparison of q learning and sarsa is accurate.
isnt q-learning, learning to play master yi
too much talk too few examples
This is amazing content.
I just have a question (still struggle with the concept of on and off policy)... at 30:18, the max Q (in Q-learning) .. which Q is it old or new ?
old, but at next state $s_{k+1}$
Instead of using so many words to explain, why couldn't you just use a couple of examples to explain the relationship between Q(s, a) new vs Q(s', a')? It would be so easy to understand through examples.
It’s a theoretical formula.
Instead of using so many words to complain, why couldn’t you just make a video illustrating these yourself?
@@stevewu1920worst take ever
Agreed