It would be really interesting to see something like a reinforcement learning algorithm or other one that could learn to drive faster use real N64s to train. It would take way longer than using an emulator of course, but if it has to compensate for the delay of the input chain it might learn a lot better. Maybe we could get 4 models all racing eachother on one N64
there is probably some better way by adding a delay to the emulator instead, that would give the best of both worlds: it would be emulated, so you can do training in parallel and faster, but you still have the delay of the controller for the better data for the console.
I'm currently building an NES controller adapter that can send/receive controller signals in any direction. One of the future projects I"d like to work on with it is something like what you've done here. Very cool to see it working!
Idea: Mario Kart: Super Circuit (GBA), the neural network plays on an emulator, you play on a real GBA, and you play multiplayer using the link cable (via some serial converter)
I'm very impressed that the model manages to drive on Kalimari Desert as well with only two laps of training data! But I think it could generalize even better if you trained it on tracks with a "road" texture for the drivable area but very diverse backgrounds. Also please teach it to drift!
Hi, i'm currently trying to implement tensorkart for MK8DX on yuzu and pretty much everything is done but I have one problem. The model has a hard time steering correctly. I trained it at half speed with cruise control just like you. The loss and validation loss stay low but when the model try to play the game, the steering value stays near 0 and is pretty much always the same (0.003...). Did you encounter the same problem and if so, how did you manage to solve it ? Thank you
This is really cool! Have you tried starting to optimize for lap times? Watching a computer drive an almost frame-perfect lap on a physical N64 would be pretty fun!
Amazing to see my favourite channel delving into one of my favourite subjects! The results are very impressive given the low amount of training data (atleast compared to what reinforcement learning would have taken to learn from visual observations alone...) That being said would love to see you try implementing RL next!
Ooh, this is so cool!! This also got me wondering about getting an ML model to play Pokémon Snap. Since there has been so much research into things like object identification, I wonder if it could be applied to a photography game. And then you’d have some pictures you could keep as a “souvenir” after 😁
Cool video! How about making an RC car drive like this? I actually used a very similar approach to make an RC car drive by itself and it's so fun seeing your work drive in the real world. I also made a short video about my project which might be worth checking out.
This is fascinating! Once it's comfortably trained to complete the track I'd be curious how it would handle when it's given item variables. Learning it can use mushrooms would be interesting to see if it will train itself on when to best use mushrooms or be stuck on having difficulty navigating the course
I wonder why SLAM (Simutanous Localization And Mapping) isnt used for this project, as its used a bunch for guiding a robot through a maze. Maybe a part of the model can learn the map, while the other is in charge of driving.
I Hope you understand the impact of the revolution you create in this world...with your awesome quality content , helping others understand an also inspiring to DIY...Thanks For your time and effort , God bless 🙂👍
This video is literally incredible. Is there a way you could showcase how you got the computer to control the n64 using the pico? I didn’t see much detail on that aspect in the GitHub other than the code.
I get the feeling that programming in assembly would be the best way to reduce the mock controller latency. A Pi Pico’s wanky python toolchain is about the opposite of that, though I can’t say for sure whether or not it would be significant compared to the other latency. Ideally I think you’d train the network on an emulator with maybe a bit of input lag added deliberately, but then deploy the trained network itself on a microcontroller (STM32G7 series would probably be my choice, if I could find one) that’s directly looking at the video output of the N64. Deploying trained neural networks onto microcontrollers will be very interesting in the next few years, I suspect.
It looks like you'd need the same thing self-driving cars need: find out what is road and what isn't road. Then send that to the steering module. The risk of just feeding raw images to the neural network is you're never sure if can even recognize what's a road, so trying to steer on it might be futile. Is the AI failing to see where the road goes, or is the AI failing to control the vehicle?
@@stacksmashing Yeah that's the hard part. Finding the keywords fit something super simple that isn't just the finished project and explains what is going on. If you have anything handy for that I'd love to check it out.
There's kind of no real practical use for this as you can just program the karts to drive a long the tracks with pathing which is what the cpu karts in the mario kart games do. They then create difficulty by using rubber banding so its impossible to get a big lead on the higher ccs.
I have done something like this but in css and cs1.6 where i used cuda sdk. Enemy detection was no issue even with mouseevent for aiming center of mass The main big problem i had was the movment for corridors
@@stacksmashing do you have a way i can send it privately as my project is not something I want to have public. Managed to do : enemy detection ( body form aka 15k 360 screenshots of enemy) Aimbot with mouseevent Path finding using bsp phrasing Corridor detection concept I really wanna go into details about my project as I have done some pretty wild examples. Searching: antiaimboss on yt will show videos of me using ai to raid servers using ai mesh Path finding I have some vids on my channel with some stuff I did but if you want I can show you all my work from 2011 of some advanced concepts Kind regards.
Someone made a machine that play the other machine.. We gone full circle.. One day someone will create something like that and the supreme beings will end the simulation saying after epoch 2021 they finally made it.
Hey man, are you interested to have a logo redesign for your channel that's simpler, clever, and functional which represent your contents to new audience and also memorable for everyone about your channel? I'm a freelance logo designer and I hope I have the honour to make use of my creativity to design one for you! Let me know if you interested for further details!
Wow, very impressive! One remark on the input lag: generic AV-to-HDMI adapters introduce a lot of input lag. You can significantly improve on that by using an upscaler that is specifically build for retro consoles, like the OSSC or RetroTink 5X.
What an awesome project! I'm curious if a Jetson nano could be used for the video processing and controller output to make somewhat of an all in one solution
Machine learning for self driving car : good. Machine learning for self driving Mario kart : better! I wonder if this could be use for the Mario kart live game!?, Imagine buying two and fight against a machine learning in real life?! XD.
This is bringing me back to Sethblings MarIO project from years ago.
exactly the reason i clicked this ahah
*mari0
@@newsuperbowserworldNo
@@AholicKnight who u?
Ikr!
It would be really interesting to see something like a reinforcement learning algorithm or other one that could learn to drive faster use real N64s to train.
It would take way longer than using an emulator of course, but if it has to compensate for the delay of the input chain it might learn a lot better.
Maybe we could get 4 models all racing eachother on one N64
@@kezif I did mention as much yes.
there is probably some better way by adding a delay to the emulator instead, that would give the best of both worlds: it would be emulated, so you can do training in parallel and faster, but you still have the delay of the controller for the better data for the console.
I'm currently building an NES controller adapter that can send/receive controller signals in any direction. One of the future projects I"d like to work on with it is something like what you've done here. Very cool to see it working!
Idea: Mario Kart: Super Circuit (GBA), the neural network plays on an emulator, you play on a real GBA, and you play multiplayer using the link cable (via some serial converter)
If you need help, I might be able to help you out on getting some things to work as I am working on a similar project myself
That would be really cool
CPU race with extra steps
I'm not smart enough to do this stuff myself, but this is super cool and I can't wait to see what else you do with this!
machine learning on it's own is already pretty impresive
but this took the cake 🎂 and the cherry 🍒
Thank you 😀
I enjoyed your training-train easter egg.
Great work, gg.
I'm very impressed that the model manages to drive on Kalimari Desert as well with only two laps of training data!
But I think it could generalize even better if you trained it on tracks with a "road" texture for the drivable area but very diverse backgrounds.
Also please teach it to drift!
what a fun way to showcase ML, love it! 😻 how did the best model compare to your best score (manually)?
It’s really interesting you choose the supervised learning approach here. Frankly wouldn’t have expected the model to perform so well.
Hi, i'm currently trying to implement tensorkart for MK8DX on yuzu and pretty much everything is done but I have one problem.
The model has a hard time steering correctly. I trained it at half speed with cruise control just like you. The loss and validation loss stay low but when the model try to play the game, the steering value stays near 0 and is pretty much always the same (0.003...). Did you encounter the same problem and if so, how did you manage to solve it ?
Thank you
This is really cool! Have you tried starting to optimize for lap times? Watching a computer drive an almost frame-perfect lap on a physical N64 would be pretty fun!
Love your stuff! Thanks for sharing!!
Amazing to see my favourite channel delving into one of my favourite subjects! The results are very impressive given the low amount of training data (atleast compared to what reinforcement learning would have taken to learn from visual observations alone...) That being said would love to see you try implementing RL next!
0:24 YOOOO shoutouts to the Silicon Valley reference!
Nice video again @stacksmashing !
Thank you!!
Man i just found about your stuff and i gotta say, you inspire me a lot. Thank you !
Thanks so much :)
Ooh, this is so cool!!
This also got me wondering about getting an ML model to play Pokémon Snap. Since there has been so much research into things like object identification, I wonder if it could be applied to a photography game. And then you’d have some pictures you could keep as a “souvenir” after 😁
The real hardware twist is insane. This is super impressive!
just yesterday i was wondering when you'd drop a new video. keep it up!
Cool video! How about making an RC car drive like this? I actually used a very similar approach to make an RC car drive by itself and it's so fun seeing your work drive in the real world. I also made a short video about my project which might be worth checking out.
Awesome video! Really nice work :)
These types of videos should be longer, way longer. I want to see the whole process!
This is fascinating! Once it's comfortably trained to complete the track I'd be curious how it would handle when it's given item variables. Learning it can use mushrooms would be interesting to see if it will train itself on when to best use mushrooms or be stuck on having difficulty navigating the course
I wonder why SLAM (Simutanous Localization And Mapping) isnt used for this project, as its used a bunch for guiding a robot through a maze. Maybe a part of the model can learn the map, while the other is in charge of driving.
I Hope you understand the impact of the revolution you create in this world...with your awesome quality content , helping others understand an also inspiring to DIY...Thanks For your time and effort , God bless 🙂👍
Just found your channel. Excellent Content! - Another sub for you sir.
You need to train the model for speedruns :)
Instructions unclear. Now my sister's Prius has taken off and we aren't sure where it went...
This video is literally incredible. Is there a way you could showcase how you got the computer to control the n64 using the pico? I didn’t see much detail on that aspect in the GitHub other than the code.
Brilliant work 👏🏼👏🏼
Can you teach it how to drift, use items or even speedrun the game? You could use TAS as training data perhaps.
I really love your videos. They are so well done.
So the pinnacle end goal is for it to do weathertenko right?
If I can learn to drive in Mario Kart, anything can.
nice vid man!
Nice project !
Very cool! You do the things others dream of, congratulations! What is your lap time on the circuit?
Finally.
I have someone to play with.
An IA that play temple run on cellphone could be really cool
Sethbling was experimenting with Mario Kart AI a few years back and made MariFlow and MarIQ.
His MarIO was so awesome, and he played Mario Kart on SNES :)
I get the feeling that programming in assembly would be the best way to reduce the mock controller latency. A Pi Pico’s wanky python toolchain is about the opposite of that, though I can’t say for sure whether or not it would be significant compared to the other latency.
Ideally I think you’d train the network on an emulator with maybe a bit of input lag added deliberately, but then deploy the trained network itself on a microcontroller (STM32G7 series would probably be my choice, if I could find one) that’s directly looking at the video output of the N64. Deploying trained neural networks onto microcontrollers will be very interesting in the next few years, I suspect.
I programmed the Pico in C, there’s no bottleneck on that side :)
Can you please please please explore the “blink” security camera. I have been working off and on for months with not much success
Such a fun project. Is it possible with this setup to race your bot? Did you try this?
Amazing !!! Give us more :)
Ok, now take it into battle mode.
very cool video!
It looks like you'd need the same thing self-driving cars need: find out what is road and what isn't road. Then send that to the steering module. The risk of just feeding raw images to the neural network is you're never sure if can even recognize what's a road, so trying to steer on it might be futile. Is the AI failing to see where the road goes, or is the AI failing to control the vehicle?
yes, and not to mention other karts and items
dall-e is open source but i havent been able to train my own network. are there any networks out there, or can you show us how to train it?
Very cool!
Super impressive work!! 👌
Thanks a lot!
Cool - we’ll done. I would love to see, how well an AI model could learn to play PAC-MAN in a comparable setup. Any thoughts on that?
Great work! Are you using an M1 or an Intel Mac?
I’m on an M1 Mac, but trained the model on a desktop computer (but M1 also only took 7 secs per epoch)
Train for every track possible besides Yoshi's Valley then see how it does there lol.
Awesome!
Trackmania loves AI
Is it possible to make Splatoon play automatically with deep reinforcement learning?
yes
Awesome project!
Nice video
I wonder how it would preform if you trained it by playing a tas on hardware.
Good video ✌️
Maybe a Minecraft learning algorithm that can beat the game. That would be so mega cool
This is so cool!
the silicon valley reference :,)
Finally someone notices it!!
ML Model for karts driving a kart on a cart!
When will it learn ssmts and how to race through custom tracks lol.
Good job by the way
I do want to do something similar to this, but I have no idea where to start.
Start with something simple and go from there :)
@@stacksmashing Yeah that's the hard part. Finding the keywords fit something super simple that isn't just the finished project and explains what is going on.
If you have anything handy for that I'd love to check it out.
I dont understand, is the AI using the RGB values of the road to associate them with where he should be?
There's kind of no real practical use for this as you can just program the karts to drive a long the tracks with pathing which is what the cpu karts in the mario kart games do. They then create difficulty by using rubber banding so its impossible to get a big lead on the higher ccs.
Can you let the AI handle second player input and race against it?
Probably :) Have to give it a try
Game&watch Zelda version is out....
I have done something like this but in css and cs1.6 where i used cuda sdk.
Enemy detection was no issue even with mouseevent for aiming center of mass
The main big problem i had was the movment for corridors
That sounds awesome! Do you have code or a write up somewhere?
@@stacksmashing do you have a way i can send it privately as my project is not something I want to have public.
Managed to do :
enemy detection ( body form aka 15k 360 screenshots of enemy)
Aimbot with mouseevent
Path finding using bsp phrasing
Corridor detection concept
I really wanna go into details about my project as I have done some pretty wild examples.
Searching: antiaimboss on yt will show videos of me using ai to raid servers using ai mesh Path finding
I have some vids on my channel with some stuff I did but if you want I can show you all my work from 2011 of some advanced concepts
Kind regards.
Easier option: make Mario one of the computer players
Isnt this basically tas
Someone made a machine that play the other machine.. We gone full circle.. One day someone will create something like that and the supreme beings will end the simulation saying after epoch 2021 they finally made it.
TAS the manual way
Hacking the Zelda Game & Watch video when...?
Mama mia.
Time to hack zelda game&watch
not even drifting SMH.
😀
yea now teach it asseto corsa
hi
you didn't learn anything from Ultron and Dune...
Deutsche da wer noch?
Hey man, are you interested to have a logo redesign for your channel that's simpler, clever, and functional which represent your contents to new audience and also memorable for everyone about your channel? I'm a freelance logo designer and I hope I have the honour to make use of my creativity to design one for you! Let me know if you interested for further details!
This is so cool!!!
Amazing!
Wow, very impressive!
One remark on the input lag: generic AV-to-HDMI adapters introduce a lot of input lag. You can significantly improve on that by using an upscaler that is specifically build for retro consoles, like the OSSC or RetroTink 5X.
Good point! Unfortunately I don't have an OSSC or so yet, but definitely something I need to look into :)
@@stacksmashing look into the GBS-C, it's a DIY upscaler (open-source firmware) that compete with those for very cheap. cheers !
Will check it out! Thanks!
@@stacksmashing Alternatively there are great HDMI mods for the N64 that extract raw RGB directly from the GPU's output and can manage
Or capture the AV signal directly instead of going to HDMI first.
Very cool! Especially that you are running it on real hardware!
4:57 So with an expensive Jetson kit it will be faster!
What an awesome project!
I'm curious if a Jetson nano could be used for the video processing and controller output to make somewhat of an all in one solution
Probably! I’ve got a couple of Jetson boards here but not sure I’ll find time to go down that rabbit hole😁
As someone mentioned, this reminds me a lot of MarI/O from Sethbling. I'd love to see you tackle machine learning SMB1 on real hardware!
I'd love to see a part 2 where you explain in detail what were the most difficult problems and how you managed to solve them :)
Machine learning for self driving car : good.
Machine learning for self driving Mario kart : better!
I wonder if this could be use for the Mario kart live game!?, Imagine buying two and fight against a machine learning in real life?! XD.
Good job! Hearing how it keyed off of the minimap is really neat.
I can't wait to watch this one!
Amazing work! any chance for the .h5 file?
I'll see whether I can upload it :)
@@stacksmashing you're the best, really
oh man, this is going to be awesome!
Some nintendo switch with asphalt 9 would be interesting! There are some arduino software based controllers available...
"Casual interest in machine learning"
Stop and go work at TESLA bro.
"Can an AI play Mario Kart 64 like a human?"
as stacks proceeds to put the apple font and mario from super mario kart in the thumbnail