Training a Neural Network to operate drones using Genetic Algorithm
Вставка
- Опубліковано 20 сер 2024
- After my first try with flappy I wanted to see how would a genetic algorithm handle more complex situations.
Github github.com/joh...
Music used
freepd.com/mus...
freepd.com/mus...
freepd.com/mus...
Good idea, and I like your smoke!
Thanks! I think smoke is where I spent the most time :D
@@PezzzasWork Why do we get hung up on those small sidequests?
@@mendelovitch it’s an easy way to procrastinate the main problem
How or where you stimulate this in unity or special software
I love how they seem to move so organically even though it seems like a relatively simple model. I bet there's some really interesting optimisation problems and extra restrictions you could throw at this.
Also thanks for uploading the demo and source code, very fun to play around with!
I think that the need to center themselves perfectly with the sphere is what makes them not become speed machines. Because when they reach the target they always gotta somehow "dock". And that requires their inertia to be 0 when they reach that point so they have to slow down. If somehow this was changed by making the drones to just need to touch the point at any part and maybe making the orb bigger I would certainly expect that there would be more speedy manoeuvres to just arrive at the target and pass through it. Perhaps even in an elliptical patrolling. Would be certainly interesting to see.
im currently working on the same thing but with more inputs;
I will try ours too;
@@00swinter21 Don't forget to post the result on your UA-cam channel !
Exactly my thoughts. It looks like the target requires pixel perfect precision to count as a success. Careful approach is the only way when the targeting criteria are so unnecessarily strict.
@@Wock__ I believe you are right. On one of their videos, there is an actual clock face that counts down on top of the target, like a circular loading bar.
The goal is to dock, not to touch the target. Changing the goals to achieve a better outcome does not mean that your model improved. Making them just have to touch the target so they could go really fast does not mean that they are suddenly better. Your thinking is flawed.
I'd love to see a game where your enemies are all neural network trained AI, and the higher the difficulty, the more trained AI variant you will have to face
Give it 10 years
imagine if the AI is being trained while you play. The better you play the less hard the ai is, but if you slow down the difficulty increases
@@ChunkyWaterisReal it's already possible now lol
@@marfitrblx AI has been shit since the 64 hush yourself.
Or even the player being an AI - I can totally see a 2D game with your cursor being the target point, and the more you play/the more enemies you defeat/etc. the smarter your character gets
5:28, gen 900: Ok, you guys are too good and I'm tired now. Bye!!!
true
"I have to go now, my planet needs me"
Would be really interesting to add fuel consumption to the mix and watch them optimize their fuel economy
and give them more fuel for every target they reach as more reward for doing that
Very cool stuff, well done!
Now make them go through an obstacle course 😁
I am working on it ;)
a combination of the ants finding the optimal path and then the drones following that? :)
@@PezzzasWork where's the video 🗿
This channel is a gem
love this channel. what separates this guy from others is his consistent ability to make his sims look cool.
Great work. I think many people would appreciate seeing background of the work.
That end result with the live-tracking is so good! I wonder how viable it is to train simple neural networks like this for game enemy AI
Depends on the game, but on games with a clear goal, it is fairly trivial and will quickly surpass humans.
@@originalbillyspeed1 i guess for different difficulty levels game designer can use agents (enemies) from different generations, for example "easy" = generation 400, medium = generation 500, hard=generation 1000.
@@AB-bp9fi I don't think that would work for most applications. When you want to make enemy AI easier or harder, you always have to think of it in relation to the player - for instance, in a stealth game, harder AI could mean it detects you faster - which pushes the player to improve and be more careful. That won't happen if you just made the enemies drunk (which is basically what would happen if you pick bad neural networks) - it just adds randomness which can be annoying to deal with. Maybe it could work better in things like racing games though.
I'm now imagining a game cloud coordinating through the internet. The AI uses background CPU while the game is running to simulate and evolve against itself, spits its best results against the player to see how they fare, and takes those results as more data to go back to the cloud with to keep working. The bots will start laughably bad at first, but they'll learn how players act, and make players devise new tactics... You might even get good teammate and wingman AI out of it if you put those AIs on the player's side.
@@commenturthegreat2915 What about training AI to match the certain level of intelligence? Like if AI detects a player too fast, then it failed the test.
This is one of the coolest implementations i've seen. Nj!
Props to Gen 300 and 400 for beings underdogs and yet surviving for so long
This is one of the coolest projects I've ever seen. Would be awesome to extend to add walls and an environment! Great work.
I wrote my autopilot cargo drone for space engineers and still i am impressed by the work
Dude I love it when they get sooo roofless! So fun to watch!
very nice! I'd love to see the same tests, but with added random disturbances like wind gusts from the side, to see how well they can adapt to that!
The end of play lineup was a cute touch. Nice work!
Impressive Stuff! Had my hands on GAs too for my Bachelor Thesis but with a 6 DOF 3D acting robotic arm. Kinda addicting when you dive deep down in ML :)!
OMG This is so cool, your video actually change my attitude toward neural network from hate to love.
5:56 The drone in the left down corner synchronized with the beat in the music. Perfection.
gen 400 is like that one kid in your class that cant stand still when waiting in a queue
1:58 that faint Vader "noooooo" put me on the floor for some reason
It would be great to have a remake of this one
I am actually working on a follow up :)
@@PezzzasWork noice! I will certainly watch it
Very nice! Please make more such content, with neural network and drones! :)
That drone that got yeeted at 5:30 had me dieing 😂
xDDD the "ok..." almost kills me
The memes are fun on this vid
Getting some strong Factorio vibes at 4:57
Hello,
very interesting work !
Did you think about testing scenarios with obstacles ?
It would be also interesting to compare the last trajectories and controls with optimal control algorithms solutions.
Cheers.
I suggest to add more then just time to the fitness equation. Fe. Energy use, pressicion, stability of flight and adding external forces like wind. with these factors the movement would become smooth like silk. But nice project anyway
The current fitness evaluation takes speed, precision and stability into account. I tried to add wind after the training was done and it worked quite well :)
@@PezzzasWorkahh I see, but the angled engines while hovering still seem very inefficient to me :)
@@DeepRafterGaming Yes you're right and I don't really know why they do this. My assumption is that it is a way to reduce power, as if they couldn't go very close to 0 power so it is easier to add angle. This could be avoided by taking energy into account in the fitness function. If I increase gravity, they don't angle the thrusters to gain more power. Here is a windows demo with a config file if you want to try it out github.com/johnBuffer/AutoDrone/releases/tag/v1
@@PezzzasWork Yeah it's hard to tell why. The fitness function is the most complicated part of any neural network.
I would allway advocate for implementing energy use in any neural network because, if you think about it, if the network doesn't have to bother with the used energy it will always come up with unnecessary movement patterns that look jenky. It's more important than speed I'd say ^^
@@PezzzasWork if you watch the way generation 5500 flys sideways, it ends up with one thruster almost horizontal and the other almost vertical. They might like tilting the thrusters because its kind of an inbetween state between flying right and left. So when it gets a new target, it can start flying towards the target sooner. That might be part of the reason anyway.
You are amazing. Thank you for sharing your fascinating work.
Beginning of the video: LOL!! those squeaks as they fall are really funny
End of the video: let's run to buy some food cans before they come for me!!!
This video felt like it's 30 minutes because I somehow kept falling asleep every ten seconds or so.
And it's not boring and no I am not high, idk I guess I just got tired or something
Imagine spending hours and hours trying to get to something and then when you finally get there you just have to go to another one
7:24 Loved how the Gen-400's legs synced with the music...
Btw, How do we decide the size of the hidden layers? Is there some rule or formula for the best size approximation?
Thanks for the video! It's really inspiring.
i have no idea about how you did it ..but it seems like something fun to learn
Machine learning is extremely fun and addictive :)
@@PezzzasWork can confirm
You could turn the target tracking into a game, try to get the drone to lose control as quickly as possible, using your mouse as the target! Or, just play with it. It looks fun.
Give the target to another network that tries to learn how to get the drones to crash while the drones learn how not to crash
@@DogsRNice oh no the ai wars
Other than giving us almost 20 seconds to read 6 words at 4:39 this was very enjoyable to watch :p
I like how it learned to turn off its thrusters to arrest upward motion and to speed up descent.
im more impressed by the smoke, great project though!
somewhat smaller models and policy gradient following might have increased convergence speed. MLPs are differentiable, so you could just backpropagate through them, sampling distance to the target at every frame and accumulating rewards over the trajectory for an unbiased estimate of a policy’s optimality. you could even use a decay term to incentivize the robots to move faster by downweighting rewards acquired later in the trajectory: distance to the target is ideally the same in the end, but according to the gradient of this reward function, faster would be better.
the only thing left would be running the simulations in parallel or faster than real-time by simply not fully rendering the state of the environment at every training step
give a consolation prize to generation 300!
It deserves it all
Have you ever tried using a neural network on a hardware platform?
The target tracking would be cool for a background
400 was such a trooper
Pls make more vids like this I love them
i want to see how chaotic it will be if the drones had collision
I will try this, that’s a good idea ;)
I tryed the mouse controlled vesion what you uploaded on github. And i saw that it's easy to confuse the A.I. in that way to lose controll and fall off the map. I think if you crate a small Trainer A.I. for the target control what best interest to confuse the drone and make it fall off the map, it can train the drone to not fall off no matter how the target moves.
Yes I did a more robust version that I can upload as well
"Im a Hovercraft like my Father before me and his before him!"
Adding a fuel allowance would probably add a more varied result, possibly get those burn hard drones quicker. Also maybe increase your destination bubble a fraction ? This increase the prize rate and hopefully the drones would tighten up the homecoming naturally like the ants do for food routes
Which parameters give the drone positive or negative feedback?
Is flying time a positive or a negative parameter? An acceleration to the target?
"300! 400! you're embarrassing everyone!"
7:25 the music moves to your left and right ear as the drone in the top right moves it's power to it's left and right thruster.
oooh idea. Space Invaders: Drones Addition. Different levels use different generations of drones as enemies.
Micheal Reeves breaking out in a cold sweat in the corner
This would be a great premise for a game where the character tracks the mouse so instead of controlling the character you're directing it and it gets better as you play through AI learning
And you did it with two hidden layers, nice! Also, you have to give it a gun now I mean come on. Let's see the the level 5500 drones beat a human being.
Ok, now make these drones fight in groups of 5, they can kill other drones in 2 ways one is to ram into enemy drones (killing both of them instantaneously), or shooting them with miniguns (only killing the target if it is hit X amount of times). But every time when they die they respawn, smarter, faster, more accurate, etc.
I'm kind of upset that you didn't publish the thing at the end on itch. Its so satisfying to see the drone follow your mouse and I want to play around with it. Great video!
You can download the control demo here github.com/johnBuffer/AutoDrone/releases/tag/v1
@@PezzzasWork thank you! :)
so, it was you who programmed my dog to run after the laser pointer.... 🤔
1:05: this one looks like Los Angeles Battle drones
Great video! How long have you been training them? Greetings from Uruguay!
it's cool to see your using dropout, so it learns better
What about creating new variables? Like saving fuel or energy consumption, or giving priorities like speed over energy/fuel consumption
That was really cool.
if you had an body orientation/angle input they would have been able to recover from a spin out or even fly upsidedown
It’s like they’re scared to collect it, knowing that once they do they’re likely to die xD
Nice work! Can you propose me material so that I can understand in practice how to build a neural network? Something with examples.
That's a good tutorial idea, I will think about it :)
Gen 2600 was a big leap in speed and control.
Now you can make game with mouse controled drones
I would love to see what happens if you give them a finite amount of fuel to manage. Have the fuel decrease quickly/slowly depending on how hard they burn thrusters. Extra bonus for fuel remaining when the task is complete. Death if you run out of fuel.
i think just counting how much fuel is used based on thrusterpower and then rewarding low numbers is better then limiting fuel
why not add a fuel limitation (which would refill once they get to a point) forcing them to switch between points as quickly as possible from the beginning
wouldn't be necessary, they're already rewarded for speed
After a few tweaks, I have a feeling this could have real-world use.
Imagine one of these things chasing you irl.
I would love for you to make an eco system like the bibites using those drones
I like these projects !
You should make a game out of this, it looks very funny!!
I love this! I'm gonna implement it right now in Python. What genetic algorithm were you using? I'm planning on using Neat
how did you get this environment in Python? I want to test policy gradient RL algorithms
@@CE-ov7of not sure if you still need this question answering however i'll give it my shot. My guess is hes implementing the basic algorithm of the envirment in python using pygame and and numpy. Then for the AI my second guess is he'll be using NEAT Python library or custom AI/NN algorithm for the agent and training. That's my guess however if you want any question just reply and i'll do my best to help. Python isn't my strongest language however but i'll try my best.
Hey @@j_owatson , unfortunately this is not something I have time/interest for anymore.
But I really appreciate your willingness to help! This is what makes the software/tech community great!
Would be interesting to have a drone sumo where they can collide and try to shove each other out of a ring.
the target tracking drone would be a really cool and distracting extension, it follows your cursor around where ever you put it lol
Nobody:
Generation 200: SPEEN
Now create an additional network which positions the orange dot (target) to navigate around obstacles on its own.
Nice work!
What mutation/crossover did you use?
Totally amazing!!!
Gen 5500 appears to display knowing how to fall rather than turning the thrusters to push itself down.
Now make it 3d and hve the gen 5500 implemented in there, have them master the flight in 3d, then make the difficulties higher. Then after all that, put the best gen into a irl drone and have it fly around
But what should be the target
its so funny to me how people in comment sections always say "now do [INSERT UNREALISTIC EXPECTATION HERE]" like there's so much difference between simulating drones, and making a drone in real life
you should place the targets randomly and not in a specific order. And for more challenge, they only have a specific time to reach the target. After the time the target disappears. And finally, the targets are fuel. If they miss too often they run out of fuel.
Edit: maybe even add obstacles.
In the video the targets are in a specific order to be able to benchmark the different generations, for the training I used random sequences
@@PezzzasWork Ok, that makes sense
Programmers and scientists are going to have a lot to study on neural networks, maybe thats what new AI will provide. Just more information for humans to expand their minds
you should make a game where you control a small ship like asteroids and your goal is to juke out the drones and cause them to crash or see how long you can survive before they hit you or something
Very nice result!
Wow that's amazing and looks amazing, how did you cross the two neural networks?
1:41 Lmaoo. This looks like flappy bird in terms of difficulty.
man this is so cool. a bit off topic but how are you rendering the thruster particles and smoke?
The smoke is just made out of static sprites and the thruster particles are baked into the flame's texture
Gen 400 is an all time favourite haha
Really cool project!!!
I was wondering what fitness function you used?
Love that video
While it was nice for the visual of all the different generations together, I feel like it would have been better to randomize the dot locations so that they have to learn to adapt to a new path every time
You may have to select more aggressively for speed. They seem a bit slower than what the optimal handmade algorithm could do
Man it would be sick to model a 3d one
Dude the physics look so polished. This is amazing!
Acceleration (gravity, mass an inertia) is probably the simplest physics properties to program. Literally just adding or subtracting numbers. He does not require your compliments on the physics.
@@UnitSe7en ?
@@UnitSe7en shut the fuck up, he’s giving him a compliment