Reinforcement Learning Course - Full Machine Learning Tutorial
Вставка
- Опубліковано 20 тра 2024
- Reinforcement learning is an area of machine learning that involves taking right action to maximize reward in a particular situation. In this full tutorial course, you will get a solid foundation in reinforcement learning core topics.
The course covers Q learning, SARSA, double Q learning, deep Q learning, and policy gradient methods. These algorithms are employed in a number of environments from the open AI gym, including space invaders, breakout, and others. The deep learning portion uses Tensorflow and PyTorch.
The course begins with more modern algorithms, such as deep q learning and policy gradient methods, and demonstrates the power of reinforcement learning.
Then the course teaches some of the fundamental concepts that power all reinforcement learning algorithms. These are illustrated by coding up some algorithms that predate deep learning, but are still foundational to the cutting edge. These are studied in some of the more traditional environments from the OpenAI gym, like the cart pole problem.
💻Code: github.com/philtabor/UA-cam-...
⭐️ Course Contents ⭐️
⌨️ (00:00:00) Intro
⌨️ (00:01:30) Intro to Deep Q Learning
⌨️ (00:08:56) How to Code Deep Q Learning in Tensorflow
⌨️ (00:52:03) Deep Q Learning with Pytorch Part 1: The Q Network
⌨️ (01:06:21) Deep Q Learning with Pytorch part 2: Coding the Agent
⌨️ (01:28:54) Deep Q Learning with Pytorch part
⌨️ (01:46:39) Intro to Policy Gradients 3: Coding the main loop
⌨️ (01:55:01) How to Beat Lunar Lander with Policy Gradients
⌨️ (02:21:32) How to Beat Space Invaders with Policy Gradients
⌨️ (02:34:41) How to Create Your Own Reinforcement Learning Environment Part 1
⌨️ (02:55:39) How to Create Your Own Reinforcement Learning Environment Part 2
⌨️ (03:08:20) Fundamentals of Reinforcement Learning
⌨️ (03:17:09) Markov Decision Processes
⌨️ (03:23:02) The Explore Exploit Dilemma
⌨️ (03:29:19) Reinforcement Learning in the Open AI Gym: SARSA
⌨️ (03:39:56) Reinforcement Learning in the Open AI Gym: Double Q Learning
⌨️ (03:54:07) Conclusion
Course from Machine Learning with Phil. Check out his UA-cam channel: / @machinelearningwithphil
--
Learn to code for free and get a developer job: www.freecodecamp.org
Read hundreds of articles on programming: medium.freecodecamp.org
![Pointers in C / C++ [Full Course]](http://i.ytimg.com/vi/zuegQmMdy8M/mqdefault.jpg)
Here are some time stamps folks!
Intro 00:00:00
Intro to Deep Q Learning 00:01:30
How to Code Deep Q Learning in Tensorflow 00:08:56
Deep Q Learning with Pytorch Part 1: The Q Network 00:52:03
Deep Q Learning with Pytorch part 2: Coding the Agent 01:06:21
Deep Q Learning with Pytorch part 3: Coding the main loop 01:28:54
Intro to Policy Gradients 01:46:39
How to Beat Lunar Lander with Policy Gradients 01:55:01
How to Beat Space Invaders with Policy Gradients 02:21:32
How to Create Your Own Reinforcement Learning Environment Part 1 02:34:41
How to Create Your Own Reinforcement Learning Environment Part 2 02:55:39
Fundamentals of Reinforcement Learning 03:08:20
Markov Decision Processes 03:17:09
The Explore Exploit Dilemma 03:23:02
Reinforcement Learning in the Open AI Gym: SARSA 03:29:19
Reinforcement Learning in the Open AI Gym: Double Q Learning 03:39:56
Conclusion 03:54:07
I litetally love this guy i subscribed right after i watched this. Best guy to watch while in college.
Ooooo
Ppppp
Зю
😢😢
Anytime I fall asleep to anything I watch these videos haunt my youtube. I never intentionally watch this channel. What the flip guys?
broooooo, the last couple of nights when I fell asleep and my laptop was goin, Ive had these videos playing when i wake up in the morning.
Same!
Yes i have the same thing. Im 2 hours in at the moment 😂
Same
Same here! something is wrong with UA-cam algo
This is a great video if you already understand the topic, understand the code and just want a guy saying what he's typing out aloud, kinda explaining bits and pieces here and there.
yah and people in the comment section be like: thank you, what a great tutorial for free, great explanation, while they get nothing and just being smart in typing a comment
He said at the beginning no need to know about this and that. 14 minutes into the video he is typing the line 123. Honestly why didn't he copy and paste it? :))))
Worst video
yeah, i hae the same feeling, i didn t undesrtand a crap
@@ramtinnazeryan
Its a "course" - so it is what it is :D . Take it as an overview to the topic, watch him code to see whats up, but not code urself here.
One minor correction for those watching at 1:19:12 and trying to follow along (like myself): on line 77 after the "else", the "memStart = int(np.random.choice(range(self.memCntr - batch_size - 1)))" should actually be "memStart = int(np.random.choice(range(self.memSize - batch_size - 1)))".
The self.memSize is needed here instead of self.memCntr because at this point the self.memory list is now full (the "else" branch), but the self.memCntr value is continuing to grow and is now larger than the max self.memory size. That leads to line 78 giving miniBatch an empty list, [ ], leading to memory being an empty array, because memStart will be a larger value than the self.memory list length, while then being used as the index for grabbing the miniBatch from that same self.memory list -- no good. Ultimately that leads to an exception: "too many indices for array" on line 81 (since we are trying to forward an empty 1D numpy array and call 2D indices that don't exist). With self.memSize for line 77, that no longer happens, and memStart stays within the bounds of the self.memory length/size. With that, everything works, and you can watch the agent play :)
Bro I fell asleep watching a different video and woke up this morning to this video playing and I was an hour deep in it 😭
Extremely well explained. Kudos to the tutor. Simple explanation to workign code in less than an hour is amazing and yet very clearly laid out. Thanks for this upload.
This is the only issue that i often see on any "basic tutorial" videos. There's no explaination on the terminologies during the intros.
The terminology and concepts are explained in the two blocks starting 03:08:20
This is AWESOME! 👍👍👍 Thank you for this!
I'm a beginner and the Background loop seems more interesting than what he's talking. I hope I understand what he's saying someday
The same thing
The length of the flatten outputlayer can actually be calculated from first conv layer tracing the data through the network. Just use the function:
((dimension length - kernal size for the dimension + 2*padding)/stride)+1 = output length for the dimension
do this for each dimension for each conv layer and multiply by number of outputs in the end to find the length of the flat dimension as such:
1st conv layer: ((185 - 8 + 2*1)/4) + 1 = 44 (acutally 44.75 but you always round down, since there are no 0.75 pixels)
((95 - 8 + 2*1)/ 4) + 1 = 22 (rounded down from 22.25)
2nd conv: ((44 - 4 + 2*0)/2) + 1 = 21
((22 - 4 + 2*0)/2) + 1 = 10
3rd conv: ((21 - 3 + 2*0)/1) + 1 = 19
((10 - 3 + 2*0)/1) + 1 = 8
this means the 3rd layer outputs 128 frames with each having dimensions 19*8 and therefore if you wanted to flatten them into one you would get one dimension with 128*19*8 vectors.
Just neat little trick for those who want it
This video has helped me find clues that ultimately helped me to understand machine learning. Thanks blue Steve.
Anyone interested in learning the terminologies of what he is talking about should go check out the video lectures Stanford did on MDPs(Markov decisions processes and RL), they're about each an hour long and do go in depth behind the math for a lot of this stuff. Cheers!!!
Underrated. Thank you.
Thank you profoundly for sharing your knowledge!
Heads up: this one isn't for beginners.
@Black Hole lel , got him good
I'm immersed in this. I read a book with a similar theme, and I was completely immersed. "The Art of Saying No: Mastering Boundaries for a Fulfilling Life" by Samuel Dawn
This is one of the best free RL videos available. Please make some more.
Indeed
⭐️ Course Contents ⭐️ ⌨️ (00:00:00) Intro ⌨️ (00:01:30) Intro to Deep Q Learning ⌨️ (00:08:56) How to Code Deep Q Learning in Tensorflow ⌨️ (00:52:03) Deep Q Learning with Pytorch Part 1: The Q Network ⌨️ (01:06:21) Deep Q Learning with Pytorch part 2: Coding the Agent ⌨️ (01:28:54) Deep Q Learning with Pytorch part ⌨️ (01:46:39) Intro to Policy Gradients 3: Coding the main loop ⌨️ (01:55:01) How to Beat Lunar Lander with Policy Gradients ⌨️ (02:21:32) How to Beat Space Invaders with Policy Gradients ⌨️ (02:34:41) How to Create Your Own Reinforcement Learning Environment Part 1 ⌨️ (02:55:39) How to Create Your Own Reinforcement Learning Environment Part 2 ⌨️ (03:08:20) Fundamentals of Reinforcement Learning ⌨️ (03:17:09) Markov Decision Processes ⌨️ (03:23:02) The Explore Exploit Dilemma ⌨️ (03:29:19) Reinforcement Learning in the Open AI Gym: SARSA ⌨️ (03:39:56) Reinforcement Learning in the Open AI Gym: Double Q
Learning ⌨️ (03:54:07) Conclusion
man i just fell asleep on youtube and now i’ve been watching this for 2 hours 19 minutes
I sleep to this all night. UA-cam knows I’m using it as white noise at bedtime and offers sleepy stuff like this 😬
Reinforced learning, is like when you have to write 1000 times "I will not talk in class"?
Wow fcc really stepping up
thanks, so helpful video
grazie mille ho iniziato da poco a seguire il tuo corso ben fatto
for anyone watching, inheriting from object is implied, you haven't needed to type that since even the oldest versions of python 3, save yourself some time ;) `class foo(object):` is exactly the same as `class foo:`
the reason he types it here is probably for intercompatability between py2 and py3, but not even a year after this was uploaded py2 went end of life, so you shouldn't need to worry about that anymore :))
You had me at "Atari"!!!
Thank you. Great job in explaining the content.
When I wanted to implement a multi agent reinforcement learning envirenment for a soccer game (multiple agents, separately trained, with separate models) what algorithms would I use for a continious envirenment (not a grid world) where the players can walk/run/shot everywhere? - DQN Reinforcement Learning?
Clearly the video is only for people who have already researched about RL. Not for beginners at all!
Great class.
Keep up the good work.
Thank You,
Natasha Samuel
so you understood everything in the tutorial?
what can you recommend to watch of your other youtube videos before watching this one?
python guides and some linear algebra to understand what is happening
Check out his channel
his original one.
thanks for taking time out your busy day to teach us RL fellow martian!
I am currently creating an agent-based model that will generate x numver of agents. Each agent has a step function. I would LOVE to incorporate this reinforced learning method into the model. How would you adjust it from taking a visual frame like from a game to using only the global environment variables? Is it simply as easy as swapping out one for the other?
Woke up to this
Good awsome view and awsome channel, good work
when you do def(build network) wouldnt it be easier just to use keras
keep going with Sutton but please move to DLR too!
easy to understand, much pleasure
I saw this video
I'll see this video again
Thank you
I really appreciate the work you are doing . Could you mention which the development tool you are
using for the whole series?
Vscode
raise error.UnregisteredEnv("No registered env with id: {}".format(id))
gym.error.UnregisteredEnv: No registered env with id: Breakout-v0
I get errors when i run the pip install commands for box2d-py, tensorflow-gpu and torch. Is there something i'm missing?
The stacking frame reshaping code is wrong, it messes up the entire array. I am trying to debug it.
Amazing course, thanks alot Phil! One question, you were comparing policy gradient methods with reinforcement learning however after few searches it seems like policy gradient method is an algorithm within RL. Could you clarify?
ж0
How exactly do I run these python programs?
I'm using Atom IDE
You can run via the terminal(linux and macOS)/command prompt(windows) or you can download the Run Script addon in Atom(that's what i use)
Thank You a lot !!!!!!!!!!!!!!!!!!!!
Do you know why his kernel size is even number? Normally we use odd number for easy calculation.
why is that a difference?
@@underlecht Because a non-symmetric kernel (even number) yields a non-symmetric filter response. In the example above, this non-symmetry leads to a shift of the blurred image by half a pixel.
Can I really get a job after learning stuff like machine learning and etc online? I did BSc in EEE but as I was not interested I did no do well and did not learn much, only thing I somewhat enjoyed was the C and C++ courses and digital logic design and verilog/vhdl courses . Later I did python course from MITx and enjoyed solving all the exercises. Pls give me hope.
wild doggo appears. wild doggo regrets wasting his time.
✍️👍 Больше спасибо
2nd time waking up to this guy
Yep... He is theaching for who already knows.....
Hello Phil, I think there is another mistake in the code, in the learn function it should be reward_batch + gamma*np.max(Q_next, axis=1)*(1-terminal_batch) instead of just terminal_batch. Since we are passing int(done) as a stored observation. Therefore for done=False, int(done)=0 and vice versa. And if episode does not end that is done equals False then we need to add the next Q_value otherwise we only add reward. What do you think? Am I correct?
yeah i do think so. but i encourage you to try it both ways. sometime it trains the agent to finish the episode as fast as possible, that's not what we want.
Could you post the link to the lectures here?
Let me tell you
It is so good :)
How comes that I am not capable to UNDERSTAND this stuff? Is it OK to accept that I am dumb? Serious question... (I am a software engineer, did computer science, but had always problem with math, but did always the best mark when I had to deliver a project..)
Can you please make a full tutorial in flutter?
Thanks, I've been watching your tutorial for a long time.
Excellent work.
We're posting a full flutter tutorial next week. :)
Thanks
@@freecodecamp Thank you so much,
I'm looking forward for that tutorial.
💯💯💯
are you using tensor flow 2.0?
No, it's tensorflow 1.4.10. If you want to look at tensorflow 2.0 or keras, go check out Phil's UA-cam channel
This is awesome..
Beta tester od roku 2016 oceňujem,,💖💖💝
Is there anyone who knows which version of TensorFlow is used in this video?
Looks like TF1
Why are we onehot encoding the actions?
What program do you use to run your code?
It's the linux distribution native terminal, not sure about the exact distribution. Maybe ubuntu
Keyboard sound is disturbing.
Steve from blue's clues wasn't joking when he said he was going to college
my utils package do not have any plotLearning func...
អរគុណ
How much linear algebra and statistics should I know for this track?
Not much is needed. I explain it all as we go along.
not much alegbra and statistics i can say, but more about the basic reinforcement learning terms
Can you please provide an environment.yml file for this?
Please use some illustrations so that can understand more easily
so it seems I can't even install this or get started, I'm on windows 10 and I have python 3.7 working, I installed pip, gym, but when I got to tensorflow it's telling me I have "no matching distribution found for tensorflow-gpu", some people suggested that it's because I got the latest version of py, others suggest to use anaconda, what should I do?
so after a few hours I was able to figure out that anaconda and miniconda are the same thing and going through the github repo for box2d-py it just says to use anaconda, but I'd rather follow how you're doing this vs using a stupid VM with py on it, seems really ridiculous that installing a few things you show in your command line running so fluidly, like what am I missing? I reinstalled python 4 times and made sure my env variables were working correctly, each with a different python version each time and they all gave the same error with pip installing box2d-py and tensorflow-gpu, including 3.6 which is the version you're on. What shell are you using? I'm just using command prompt and I wonder if that's the problem.
lol after buckling and trying anaconda tensorflow fails, but everything else runs properly. Really fun tutorial.
@@jordanolson11 Sorry, just seeing this now. To the best of my knowledge, Tensorflow only works with python 3.6. You can just install 3.6 in parallel with other versions, without nuking the whole install, I believe.
Use Google colaboratory
Use Google Colaboratory
Is there any course (Reinforcement Learning code based) for beginners ?
Coursera has a really cool 4-course series "Reinforcement Learning Specialization"
self.q = tf.reduce_sum(tf.multiply(self.Q_values, self.actions))
Why are you doing this? I fail to understand the meaning of this line?? Thank you in advance :)
he made a mistake.... the github code is corrected.
@@mt345678 I see so many other tutorials which do the same thing, and I fail to understand why.
Information is so good but the background is grabbing the attention away from the information.
Is this something someone can learn without previous experience?
You sure can, I'll cover the basics as we go along.
Not really, i suggest you read about reinforcement learning basics before you watch this.
nice
YEET!
Phil, you are a fucking boss :>
새해 첫날 듣는 노래가 그 해 운이 달렸다는거 몰라서 2023년 첫곡을 니소식으로 들었다가 이틀만에 사귀던 남친이랑 헤어졌었는데 2024년은 내가제일잘나가 듣고 완전 끝내주게 살아보려고! 1년내내 연애도 끝났고 직장도 안좋게되서 속상했는데 겨우 [빠]칭코닷[컴]하면서 잊고지내고 우울한거도 잊으면서 지냈는데 2024년에는 더 잘될 수 있을것같아! 이제 돈도 잘 벌고 내가 더 행복한 사람되야지!
ok guys where is the code available
12 minutes in and scratching my head. Hi yes I'm lost
i feel uncomfortable with his hands
Thanks for your information
Can you help me to set an environment to trade forex and how to set it with metatrader 4
Please help me
i can hardly hear anything
When you mentioned a course on Full Machine Learning Tutorial - Reinforcement Learning and there is no proper order to it. I don't recommend watching this thing. There are tons of materials that are a lot simpler than this.
Not useful. Creator should at least run a code once before wrapping up a section. Small syntax/logical errors are fine but when there is error in code itself (eg: new_state_batch in first section) and there isn't much explanation about it, whole hour spent in coding goes for nothing.
Is this tf1 or tf2?
tf1
machine learning is really trending matter
25.10.22 22:30
Add something with udemy
Just need to get this over.
After I find her,
That's it.
I will share the technique.
She must code it.
Then its done.
I will go to desert.
If your thinking nuclear reactor?
In the middle of the congested city in the world with rascal scientist like me .
It will result to catastrophe.
please make complete course on WordPress 5.2
I have that T-Shirt
this is not for beginners !!!
okay this is not for beginners at all. wow.
I'm second
No
I'm First
Congratulations you both got prize money of 5k dollars
Lol @shirish
@@shirish3008 no thanks.
@@tejasjani2544 we will give you even you don't want...
🤝🌈👽🖖
Classic example of a great coder who can't teach :)
First
LMAO, im just watching him code at this point pretending i understand
lol yah
terrible tutorial, he's using outdated methods of tensorflow library for literally no reason
1.4 hours of coding without any execution. this is not a good way of learning