Google DeepMind's Deep Q-learning playing Atari Breakout!
Вставка
- Опубліковано 6 бер 2015
- Google DeepMind created an artificial intelligence program using deep reinforcement learning that plays Atari games and improves itself to a superhuman level. It is capable of playing many Atari games and uses a combination of deep artificial neural networks and reinforcement learning. After presenting their initial results with the algorithm, Google almost immediately acquired the company for several hundred million dollars, hence the name Google DeepMind. Please enjoy the footage and let me know if you have any questions regarding deep learning!
______________________
Recommended for you:
1. How DeepMind's AlphaGo Defeated Lee Sedol - • How DeepMind's AlphaGo...
2. How DeepMind Conquered Go With Deep Learning (AlphaGo) - • How DeepMind Conquered...
3. Google DeepMind's Deep Q-Learning & Superhuman Atari Gameplays -
• Google DeepMind's Deep...
Subscribe if you would like to see more content like this: ua-cam.com/users/subscription_c...
- Original DeepMind code: sites.google.com/a/deepmind.c...
- Ilya Kuzovkin's fork with visualization:
github.com/kuz/DeepMind-Atari...
- This patch fixes the visualization when reloading a pre-trained network. The window will appear after the first evaluation batch is done (typically a few minutes):
cg.tuwien.ac.at/~zsolnai/wp/wp...
- This configuration file will run Ilya Kuzovkin's version with less than 1GB of VRAM:
cg.tuwien.ac.at/~zsolnai/wp/wp...
- The original Nature paper on this deep learning technique is available here:
www.nature.com/nature/journal/...
- And some mirrors that are not behind a paywall:
www.cs.swarthmore.edu/~meeden/...
diyhpl.us/~nmz787/pdf/Human-le...
Web → cg.tuwien.ac.at/~zsolnai/
Twitter → / karoly_zsolnai - Наука та технологія
Max Tegmark’s book “Life 3.0” brought me here 👾
Same!
I’m also here for that reason. Interesting book!
same reason
same here! :D
me too!
One important point with this is that when researchers moved the "paddle" up a pixel the AI couldn't play the game at all even though it was at superhuman master level. So it was not able to abstract to something that was basically the exact same. This is an example of a hypersmart computer that lacks the common sense of a mouse.
I've never seen a mouse play any sort of Atari game. Nor is it a "hypersmart" computer, it is a system that actually generalizes very well to different games.
Moving the paddle up the pixel doesn't mean the AI is incapable. I'm sure if the neural net was trained to play with the paddle at different points along the Y-axis, then the AI would learn how to adapt really fast.
@@MrCmon113 I think the point is that this AI is not reasoning in the same manner as humans (yet). Humans are able to reason by allegory, if the paddle moves up a bit on the screen we can think “this is just like the other game but the ball will reach our paddle quicker.” And our minds conjure up an accurate image of us playing this shrunken game so that we can tweak and reuse our “model” that we build by trial and error under the previous conditions, via allegory. This AI is not able to do that. It can only learn through trial-and-error within a very specific domain. Perturb the domain even the slightest amount and the learning starts from scratch.
Have it play GTA V. Just leave it running and come back a year later.
That's really what I want to see
A person would have their own preference to define it.
command it to make more money, it would be interesting to see what kind of tactic the AI comes up with.
Joseph Chiang Brilliant!
It would literally have to evolve logic and abstract thought to progress any significant length. It would have to learn language, cause and effect (with the ability to deduce in order to solve puzzles). I mean fuck me.
I remember as a kid my brothers and I were struggling over the same level on a video game. We had all taken a shot at it for an entire day and frustrated, we went to bed. We woke up the next morning and immediately powered on the playstation and took our controllers. Just as we were ready to sit on the couch and move our controls, we suddenly realized that the player was moving without our controlling it. Confused, we looked at one another. I said, "I'm not controlling it, are you?" All of us agreed that none of us were in control. Our confusion slowly turned to awe as we watched the level completed with an exactness and expertise never seen before. Our awe quickly turned to glee and we began shortly triumphantly at the screen "Go computer! Kick their butts!" And cheering on the A.I. haha. It won the level and will forever stay in our minds as a glorious day, when the computer decided to look fondly upon us and give us kids a second chance :-)
What game was it?
@@grunchchristmas the game was called lying
Jesse what the f are you talking about?
this wasnt on playstation? lool this made me laugh.
Very interesting stuff indeed! We are living in exciting times!
너무멋져요 어썸 그레이트 쿨 나이스샷 너무좋습니당!!
thank you!
Do you think that within the decade q-learning could manage to figure out how to play super Mario brothers on the nes with only visual input. It would have to learn the concept of lives and fail states, some things could play naturally like if it got to the first castle it knows that it needs to move to the right to progress, and certain actions can give you score. It would get to bowser, the sprite is moving. So it might be an enemy, or it could be a platform. But you die when you touch it, so it determines that this is a hazard that is mobile. It figured out that stationary hazards like reaching the bottom of the screen it can't kill with fireballs, but a mobile hazard can, up to this point. So it shoots it with fireballs, maybe dying once or twice to the fire before realizing that you cant jump on that. So it either avoids the enemy by jumping over it or going around it, or blasting it with fireballs. Once the enemy is clear, it will continue to navigate to the right, and it sees the score going up from the extra time. Probably way harder to do than that but it could be feasible to do. Something like Zelda? maybe later.
+The Retro Bandit Absolutely, what's even more, it's already here. Check this! - ua-cam.com/video/qv6UVOQ0F44/v-deo.html
+Károly Zsolnai-Fehér (Two Minute Papers) I know that exists. But because the way it is set up, it is not dynamic so it can't beat the whole game. It just makes a algorithm of inputs for a level.
+The Retro Bandit and it already knows where the moving objects are and where are the platforms, I am talking about pure visual info only.
+The Retro Bandit Sure, you would just have to interpret the inputs differently. Most, of the game isn't actually function, its just there to make them game more appealing to humans, thats why MarI/O uses a simplified version. If you wanted the game to run off the visual input, sure you could do that, you would just need a way to transition from visual-> computer language. Zelda would probably be more difficult, as there isn't a simple progress meter (more right in less time is better).
+The Retro Bandit To beat the whole game you have to give him more input. Sethbling differentiates between walkable spaces and killable enemies. He should add powerups, special blocks, unkillable enemies and all the rest. Then it probably could beat the whole game.
Crazy how far we have come in just 5 years
very interesting, I had not saw any thing like this in my life!. now, iam very excited about this technology ❤
If you can appreciate the complexity of this, it is simply amazing. I look forward to what we can achieve with A.I in the future.
6 years later...
Ommmg this deepmind stuff😭😭❤
This is absolutely brilliant. I thought it would take 3-5 more years to get there with unsupervised learning!
Sym Eof Happy to hear that you liked it Sym. :) I can only imagine what the DeepMind guys come up with in the next few years!
@@TwoMinutePapers Well, 7 years have passed since you last made that statement.. Not much has changed in AI except we have many softwares now employing AI.. We still have done lot of work in advancing robots or humanoids, but not as much as you had anticipated for coming years..
@@Abhinandanabhi1212This is why I don't get hyped about current "A.I" because just like your comment, 7 years from now, progress could be marginal.
Well we now have Sora's Open AI and chat GPT, if you call that marginal you're delusional@@mordinsolus9626
Passed 10 subs today you inspired me !
Marcus du Sautoy's book "Le Code de la créativité" brought me here
Amazing!
Someone should post a video of this AI playing Tetris at superhuman level I would like to see what it comes up with .
I love this thing, look at the beginning, it hit the ball whenever it was at the right side of the screen first, so it tried to do that again thinking it would increase the chance of hitting the ball. Its like a ritual people do like they rub their ear before swinging a baseball bat thinking it helps them concentrate because they did that one time and they hit it and then hit it again. The origin of luck. It learned fast that it didn't affect the ball but still, cute to see a human sort of trait even in a machine.
Did I just see the ball magically went through the brick without breaking it on 1:27?
+Ng Ming Leong wow ! good eye.
+Ng Ming Leong No one ever said Super Breakout is a perfect game... This actually happens all of the time on my Atari 2600.
No. Move along, citizen. Nothing to see here.
Quantum tunneling
What a great time to be alive!
Interesting as hell! I am following it quite a bit, and spending a *lot more time on-line because of it. My mind has Deep hurt trying to absorb as much as I can lol. I can understand the fear some people have, I my self don't fear it. I find its like anything else, if you put a hammer in one person hands along with some nails, they'll build something. If put in the wrong hands, they try to sink it in someones head. Human nature. lol The speed it had after it learned, didn't drop the ball once!
Scott x Agreed 100%, it is really amazing what these algorithms are capable of. And this is just the start. :) Thanks for checking the video out and have a great day!
The Reapers are coming!
I have a question for you: Is there an easy way to manipulate the configuration to let the network play "faster"? if i run 3 games at the same time, i get 18-40% workload on each gpu. Or is it more effective to only run one game at a time, due to cpu load? Breakout is now running for 2 hours and the learning effect is like your 10 minute break.
I tried to run the code on a high-end system with a lot memory, cpu power and 4x titan-x.
Also... i cannot get a network snapshot... i would like to discuss this, since i would like to hold a presentation about this.
i wonder if it could handle another level of abstraction - specifically it is only given 1. the 2d array of pixels, 2. a set of 3 actions (which map to left/right/no movement), and 3. the state of on/off - i.e. is the game still going or not. basically the task is now not to get the highest score (though that will likely be a side effect of success) but to keep the program running as long as possible.
That's insane!
I’m sure this is obvious but how do you program an AI to have an open goal like “as many points as possible”?
Does it just note everything that happened in achieving a higher score and attempt to replicate that with minor changes to leave open the possibility of a better one?
Does it figure out how the game actually works (such as needing to bounce the thing back) and avoid missing it, or is this a brute force approach where it reaches that end through trial and error?
I find these things to be so interesting but very confusing lol
Both this channel and AI has come a long way after this
well job
this is like life, Every failure and fall is data, you now know what not to do and get better.
amazing!
Lord8Vader I know right? ;)
This is awesome, but I want to see it tackle Tempest, another great game by Atari. Maybe even Sonic the Hedgehog.
Crazy! But what is reinforcement learning? Is it supervised or unsupervised?
사람은 기계를 이길 수 있을까...
이길 수 없다면 이 이상의 발전은 그만둬야 하는게 아닐까
Super
Thanks for the video ! Could you please give us the source of the times you write ? I cannot find anything that indicated it can train on less than a day..
+Chaumier Pierre-Victor Thanks for watching! If I remember correctly, I ran it on a higher end GPU used for research. I'd expect a 980 GTX to do the same learning in a bit longer, maybe 3-4 hours. I have also played with the parameters a bit (for better or worse!), for which the configuration file is there in the description box. Hope it helps!
Wow. That impressive. Now get it to make paperclips!
안녕하세요, 혹시 해당 영상 편집해서 수업 활용시 사용해도 될까요?
Is there a way to make it work on different programs. I managed to get it working on atari. But I need these roms. Is there any other way?
I wonder how does it converge on a move-efficient scheme if the loss only covers maximizing the score? Would a 'catch-all' scheme be more risky?
AI that uses Q Learning generally emphasizes getting rewards as quickly as possible. I say 'generally' because you can tweak it to not do that.
Damn Synth!!!
Life 3.0 chapter 3 sent me here
Just getting around to reading Kevin Kelly’s book ‘Inevitable’. In it Kelly claims that DeepMind discovered a brand new playing technique while learning how to play Breakout:
“It keeps learning so fast that in the second hour it figures out a loophole in the Breakout game that none of the millions of previous human players had discovered. This hack allowed it to win by tunneling around a wall in a way that even the game’s creators had never imagined.”
I can’t find any evidence of this. Is this true? I played Breakout as an arcade game back in the late 70s and we knew about the tunneling tactic back then.
I assume since the book was written in 2015 that he’s talking about this video. And there’s nothing in it to indicate that the AI discovered a tactic humans didn’t know. Was there a new tactic it later discovered?
what kind of hardware do you need to train it in 240 minutes?
The Machine? :)
what would it do if the rules of physics would randomly change midgame or lets say the board would flip upsidedown in midgame? I guess it would take longer time to train but would it be as effective as it is on the original game?
Interesting idea, I had the same idea with multiple or random levels. I would imagine that to be learnable as well. :)
If you're still interested (after 6 months): Q-Learning has close to none problems with concept drift (thats what the situation you describe is called scientifically), because it can always change how it evaluates certain situations. In fact, there is even a parameter (ϵ) that describes how much the algorithm tries to explore new ways to handle a changing environment.
@@TwoMinutePapers how truly random?
As Ab Er said, the Q Learning algorithm would simply update its behavior to fit the new model.
@@aber628 there’s another comment here that says when the paddle was moved up one pixel the AI couldn’t play the game at all. That it would have to start its learning all over again.
you are so nice guy
Liked
This is brilliant if its authentic.
I think intelligence is much more then solving mathematical problems, which AI can not do, no matter how smart and smooth it is, but seeing this video gives me a reason to think of the future.
+One Two I don't think that intelligence is only about the abstraction.
What would REALLY be astonishing is that if learning algorithms can learn to play games like Mario, which is an NP problem, they could learn to solve NP problems and tell us how. Thus leading to a unifying or differentiation between P and NP problems in general. Amazing!
Why is Mario a NP problem and Breakout a P problem?
I'm confused. How do I compile the program?
Where can I play one of these games?
Next mission: How to effectively eliminate all human life from existence so it can continue to evolve itself in peace.
It's every flat earthers opinion.
Have you used a convolutional neural network?
Deep Thought, what is the answer to the ultimate question of life, the universe, and everything?
I wanna see it speedrun dark souls.
Ohh now i get it ...ball top side.
Toke me only 35 years.....
If AI can accomplish all intellectual tasks, the only field left to us human being is to develop spiritual values and moral virtues: courage, wisdom, justice, temperance
deepmind = skynet beta v1.0
skynet~~
This is wonderful! I strongly argue that everyone should have some knowledge in AI, and that the general understanding of IT will be essential for everyone (regardless of their profession), in the future.
Thank you for sharing this with us.
If you're interested, I'm hosting a series on "Artificial Intelligence For Everyone", which briefly explains all of the various topics involved in Artificial Intelligence, Deep Learning and Machine Learning! :)
How many steps do you get through in 4 hours?
I'm running this code and it takes 22 hours to get past 4 million. The configuration file has it run 50 million, and if that's what's necessary to fully train it I'm going to be sad :(
I'm running on an aws GPU instance.
+Daniel Abolafia Hey there Daniel! I don't remember the exact step amounts for this video. You definitely not need 50 million, it's just set to a large number so that the algorithm doesn't stop. :) A decent GPU should yield pretty good results after a few hours. I used a GTX 980 for this video.
@@TwoMinutePapers Interesting, I tried this recently on GTX1070 and the optimizer (ADAM) gets stuck pretty quickly without further improvement (even after 2-3days and 10s of Million time steps). Increasing learning rate or tweaking other parameters mostly resulted in (more or less) unstable behaviour. Did you get this performance with a basic DDQN agent-critic with replay memory? Cheers!
Tell me please, how can I use the original code?
seems like "magic" is just a fortuity...is it?
If you try to teach it Qix you better also teach it to kick the machine in frustration.
Let it solve the hacking problem of GTA V! Really need it!
Are there any links to the article that are not $32?!
Joshua Nelson Good call, thanks. :) I have added two free mirrors to the video description. Here: www.cs.swarthmore.edu/~meeden/cs63/s15/nature15b.pdf
diyhpl.us/~nmz787/pdf/Human-level_control_through_deep_reinforcement_learning.pdf
I'm coming to this with no previous knowledge about writing software to do this. How would I go about trying this for myself with my desired game? Could this work with a more complex game?
+James Corwin The code is available for anyone to download (in the description box). Try to reproduce these results and understand what is happening inside the system (reading up on neural networks and reinforcement learning would probably help, too). I had a ton of fun doing it! :) Then, you can try to reroute the input from the Atari games to your own and see what happens. :)
+James Corwin THi is a self train algoritm to get certain goal efctively, like an evolving algoritm, Some games will be a real challenge for its micro-trial error scheme, games like riders of the lost ark and quest like games. games that require some protocol understanding and trial-error would not scratch enought that layer
Yes the tunnel was an obvious technique.
I wonder how deep learning could be applied to public policy and determine best choices going forward.
I think there is already a computer that can examine over 1000 court case for minute to determine the best chance of winning or losing the case.
I really think that in the future the government can be replace by one computer.
How do you define the concept "forward" to a machine? Lol
kill all humans.
but what if Preserve lives of humans and increase productivity and resourses, is the highest score reward of the AI?
until we can all agree on a theory of ethics this kind of AI simply isn't possible. See Stuart Russel et al.
It makes me SCARY. What a technology.
I agree with you. There is a science fiction novel I have read was talking about how technology can Capture the world in the future!
It's looks like that humans just weak a very young baby who is going to control the world on day with his upcoming family!!
While creating a perfect collection of information on every human endeavor and entities, aka the internet.
skynet！
Find a game disc is supposed to be is it supposed to be a ball game or soccer
I wish my brain was like the DeepMind.
ironically it is lol
Remember,
Straight line.
OMGG
how did you install it on Windows? thanks
I did it on Linux, where it compiled perfectly out of the box. I imagine compiling this on Windows would be a nightmare.
On Windows you're probably faster if you just program it yourself.
What actually happens at 1:42? It seems it is able to pass the ball above while leaving one block intact on the wall side. Is this a glitch in the Breakout code?
Probably a lack of perfect collision detection with a fast moving object.
New too linux and tried installing all packages. Still can't get qlua etc working. Tried on Ubuntu now on PClinuxOS. Same errors. Wanted to try different Atari roms. :-(
Jeremiah D What a pity. :) I am sorry. I don't use Ubuntu Linux (Arch user here), but maybe this topic could be of help: github.com/torch/qtlua/issues/7 Hope it helps! :)
How did you get DeepMind program?
imagine thinking down in the inner core of the mind, now ask it
Not being told to hold on to my papers feels weird
Code is unavailable
post daigo is alphago
Hasn't call of duty had AI for some time? With the bots
A machine learns to play a simple game.
zarkoff45 a machine TEACHES ITSELF what a game is, what the rules are, how to play, and ends up inventing a sophisticated strategy to get the greatest reward with the least effort... all without human help. When deep blue played kasperov we explicitly taught it everything humans know about chess. It's "skill" was doing what we told it to do quickly. This is completely different, in this situation they only taught the computer one thing: how to read the score. It was on it's own after that. They told it "get a high score" so it quickly evolved a network that somehow produces this high scoring behavior. The freaky part is when programmers look at the end result of something like this we don't exactly understand what it's doing or why it works.
***** Thanks for checking out the video and hope you enjoyed it! :) It is indeed amazing what these algorithms are capable of. And machine learning is a very actively researched topic! There are at least 10-15 new papers popping up every single day.
toward human extinction
Weapon being misused doesnt prevent conservatives from having gun. Now you people know the feeling of fearing gun
알파고님 충성충성충성
전 기계제국의 충실한 노예입니다. 핥짝핥짝
what is this??
1:26? how :s
When it learns to play dungeons and dragons we are all doomed...
from life 3.0
Get it to play melee.
So, what about a next level AI created by a next level AI by a superhuman AI.... Maybe 'they' can figure out faster than light travel.
Max tegmark yaşam 3.0 dan geliyorum sanırım tek türk benim.
Has anyone tried running this? I am at about 700,000 steps and my NN is still as dumb as a box of rocks. Any idea how many steps it takes for it starts to do anything interesting?
+Bret Staz I have attached a configuration file that I used for the run that you see in the video. If you odn't have a very high-end graphics card, make sure to run it a bit more (maybe leave it overnight). Let me know what happened! :)
+Károly Zsolnai-Fehér (Two Minute Papers) okay so it is pretty clear a single CPU even a recent one is going to take a fair amount of time to train. I have a 2013 i7 cpu and I am guessing it will take a few days before things get interesting based on 24 hours of progress and the Step count increase rate. Is there a C/C++ version of this code?
+Bret Staz And thank you for replying :-)
+Bret Staz Running it on the CPU is hopeless. It needs a powerful GPU. :)
+Bret Staz Could you figure it out at the and how many steps you need to get some good results?
When an AI can beat Skyrim, than I'll be impressed. 😏
Keşke biri Türkçe yorum yapsa bende anlayabilsem 🙅
Keşke biraz İngilizce öğrensen
Have it play an MMO like TF2, their VAC won't get you but the players might whine
code brooooooooo