Want early access to new videos and some behind the scenes content? Consider becoming a channel member ua-cam.com/channels/QvW_89l7f-hCMP1pzGm4xw.htmljoin Other videos you might enjoy: ua-cam.com/video/ILY7tYdIS2Y/v-deo.html ua-cam.com/video/_2GObcrrWq8/v-deo.html
No need, you can just get a softmodded PS2 with OPL, or maybe a legit copy of the game and get some shenanigans going with PS2LINK (though you would need to patch the ELF to work with it first).
@@SiliconExarch I haven't used the debugger in years, but it should at least be possible on a Fat PS2 using Kermit through FireWire. Not the most stable experience, though.
I knew the EULA would appear only in the multiplayer menu 😅, as it would be weird for a console game of that generation to show an eula in the main mode. Small detail: audio level is low
I've read somewhere about some other (pc) game doing that, but it was either changelog or MOTD (message of the day) being exploited. Can't remember what game it was. Apparently they got so good at it that they've eventually managed to write a legit updater onto people's machines at roughly the same time they've started shipping real patcher in the installer of that game.
The crazy part of all this is that these games actually do have a way to patch themselves but apparently this was just something Sony had kept to themselves. Any game that used medius and also included DNAS (I say this because the only games that seem to have these packet handlers all had DNAS while the ones that didn't (socom 1, twisted metal black) don't) had the ability to read and write memory from the server. Sucks that they had to go through these sorts of hoops when sony already had the tools available for situations like this. Dan uses this functionality for his patch and we use it for our patches for SOCOM 2 and Combined Assault.
Dude this stuff is amazing - thank you for showing others how people like us think through these systems. Seriously, the biggest thing I enjoy about your videos is how it logics through abstractness. This is an art form, my friend, and one that isn’t taught in school…we learned from guys like you, just in different ways (forums, IRC, more) - so thank you for carrying on the torch to keep others digging behind the code too!
Awesome video! I'm still really curious how the developers bridged the gap from the EULA buffer to the target function they were trying to overwrite without crashing the game. I would love a followup.
I was discussing this with the Horizon server devs, our best theory is that this was only used for rapid patching in early releases and the pointers may have been put closer to the buffer
@@nathanbaggsAnother possibility might have included clobbering data with the same values. If you are overwriting memory with known values, you can include the values you were clobbering in the payload itself to avoid crashing.
Have you seen the developers (Tony Garcia and Mike Stout) let's play with their commentary on that game? It's up on youtube and full of interesting info about how they've made all of this run on a PS2 hardware. Love it just as much as your video, it's fascinating just how many little tricks Insomniac had and their technical knowledge is truly underrated.
Hope your feeling better! Glad you completed this project it came out quite well. Never even thought about how a dev could use a vulnerability they really only have access to in order to send patches… crazy.
I wanna do a channel like yours some time, or a blog, to explain a weird zip-like format I found on a series of games... It took me ages to understand because I also stumbled upon lots of problems like you do... But also years have passed and I've forgotten some of the wrong paths I had taken.
There used to be a huge online forum called Xentax. It closed down a few years ago but it specialised in obscure and esoteric game file formats and had thousands of tools people had made to extract resources from just about any game, no matter how obscure the game or file type was. I'm taking, everything from AAA titles to proprietary 2010s commercial arcade game devices like Deal or No Deal. Real shame it shut down. That 100% would have had the tools you need.
So in rare instances, two wrongs DO make a right? Oops, game has a major flaw. Oops, we left a massive security vulnerability. Let's use one to fix the other.
This was one of the best videos I've seen in a long time. Nice work dude. Nothing to suggest you as sollution, you went way deeper than I could've been. Good luck :) Post an update once you have!
Have you tried using the emulator's own debugging tools rather than Ghidra? You're probably feeding the emulator's RAM back to the game, not the game's RAM. The payload needs to maintain a stable game state up until the vulnerable point that can run the exploit.
I once patched Tenchu: Fatal Shadows to swap the function pointer of the options menu callback to a debug model previewer left inside the retail game. Pretty nifty. Very cool dev trick from y2k. I always knew insomniac devs were based.
Ah, those were the days, I once patched an older version of the unreleased MU online client to handle a widescreen resolution it didn't support. The monkey wrenching in the executable was minimal.
I was going to get a membership the other day, but I forgot. This video made me laugh a bit though, and you definitely deserve it. Keep up the great work, I absolutely love coming home from work and watching your videos
From what I could understand of the Game Developer article, it seems to me that they utilized the EULA itself to patch the game. I visualize this as follows: 1. They replaced (parts of) the EULA with patch code (which would at this step be processed as mere text), and overfilled it past the brim. This overflow would eventually reach a variable that contained an address. 2. They replaced this address for an address within the EULA buffer, meaning the later function callback that used that variable would send the pointer back to the EULA. 3. The pointer would then process the patch code in the EULA as instructions, leaving them free to do as they liked (as long as their patch code didn't ruin the function callback).
Look at that! So if they never used original C "bad" strcpy, they probably would never be able to patch their game or would have more difficulty. A unsafe C function, was the key for their success! C "do whatever you want, even shoot yourself in the foot" philosophy, that many critic, apparently was what helped them in the end.
It's possible the game crashes only if the second packet doesn't arrive quickly enough. They might had even done something fancy and packed both command packets into the same IP packet so that there is minimal delay between the two events.
I packed all the data into one message, I'm not sure how the proxy actually handles splitting it up into packets but I couldn't see anyway to shovel the bytes down quicker
I played UYA for a little bit in 2007 while the servers were still active; I don't know how much memory would you need to overwrite for the buffer to overflow, but it couldn't have been that much, right? Otherwise it would have taken forever for the lobby to load on slow connections back then, and while my DSL wasn't the worst, I don't remember much waiting between the EULA and lobby screens.
@@nathanbaggs It would have to have been used every single time you started up multiplayer. The game didn't run off a hard drive so the patch would be lost.
could it be that the emulators heap allocator is just different from the original? Maybe it does a better job if defragging / hole filling, so some important stuff gets allocated in between the buffer and pointer, whereas on the original device that wasn't the case. The defragging may also be more volatile so whatever gets allocated in there may effectively be random or at least not exactly the same every run hence why it still crashes when you fill the packet with the same memory. It would be cool to analyse all heap allocations in that memory region.
As far as I can tell the buffer is in static memory. It’s always at the same address but I will admit to not knowing a lot about the inner workings of PS2 memory to be sure
I mean, we are literally asking developers to simulate the real world AND to run it optimized on all sorts of hardware. If that isn't what "making games" is all about then I don't know what is. It's the reason why technology has developed so quickly.
I would imagine it is something related to the emulator. They do a check somewhere or other for a valid memory which was not happening on the real hardware
Also a question here Why did they even bother ? I mean the game released , sold well and i don't think sony had them contracted to do any sort of post release updates (which they couldn't do easily ) So what purpose outside of some sort of pride did they do this for
Well, assuming there was no https or packet validation from server back then, and game code leaked IP's (assuming some p2p in chat/voice/server code) you could crash other people with this exploit in hand. Thinking similiar situation happening in COD lobbies of that time would be a huge impact. Glad we were limited to ratchet & clank
It only works if you can control the EULA, which means you’re pretty much limited to man in the middle-ing a server. It’s not something you can use to remotely target any player
@@JohnnyWednesdayI would refrain from that tone especially since it is far from certain that it was in use. Development culture did not shift to "https everywhere" until remarkably recently.
Game Development hasn't been hard for a decade. Just sayin. It may be hard to do it well, but overall, since the market is flooded with mountains of trash "product", it can't be that hard, can it? For every "difficult" game there's a thousand trash titles that were churned out over a few weekends. The illusion of difficulty in game development is nothing but a marketing ploy designed to embellish the value of any given product as it relates to the absolute Ocean of products they are meant to "rise above". You're not wrong, it is all smoke and mirrors. But you didn't notice all the mirrors on the way in.
"Game Development hasn't been hard for a decade." If you're talking about shovelware trash, yea it's piss easy to crank out that kind of crap, but in context of aaa games, it's gotten signifignantly harder. You ignorance of game development is staggering. There's a huge difference between an assetdump, and a game that's had actual effort put into it.
@@nathanbaggs it's never been easier than right now to make your own game and put it out there. Every tool you could ever need is available in half a dozen mouse clicks. You should just brainstorm a simple gameplay loop and make a demo. Put your own name on it and release it. Then you won't have that excuse in your pocket anymore.
Not an excuse on my end by any means, all software dev is hard and I don't want to belittle anyone's effort just because it would have been harder to do 20 years ago. But I will accept your point that the barrier to entry is much lower (which is a good thing) now (:
@@ZennExile Bro you legitimately have no fucking clue what you're talking about. Again, yea, you could make a game with "a few dozen clicks" but something that's actually worth playing and of good qualilty is signifignantly harder than that. There's a differerence between shovelware that you'd see on jim stearling's old vids and stuff actually made by devs. Just look at how much of a clusterfuck cyberpunk 2077, it's a daunting task to handle if you're doing anything of actual scale instead of just something shat out from unity/Unreal Engine.
Want early access to new videos and some behind the scenes content? Consider becoming a channel member ua-cam.com/channels/QvW_89l7f-hCMP1pzGm4xw.htmljoin
Other videos you might enjoy:
ua-cam.com/video/ILY7tYdIS2Y/v-deo.html
ua-cam.com/video/_2GObcrrWq8/v-deo.html
Someone send this man a PS2 devkit
That could be fun
Didn't LTT borrow one? Maybe ask them?
No need, you can just get a softmodded PS2 with OPL, or maybe a legit copy of the game and get some shenanigans going with PS2LINK (though you would need to patch the ELF to work with it first).
@@zummoneI'm pretty sure a TOOL is required to debug games which use the network because they reboot the IOP.
@@SiliconExarch I haven't used the debugger in years, but it should at least be possible on a Fat PS2 using Kermit through FireWire. Not the most stable experience, though.
Great video and explanation on such an iconic hack! Glad we were able to help!
Thanks for all the help!
I knew the EULA would appear only in the multiplayer menu 😅, as it would be weird for a console game of that generation to show an eula in the main mode.
Small detail: audio level is low
It’s easy to get wrapped up in things like this and not take the time to step back and see the obvious
I've read somewhere about some other (pc) game doing that, but it was either changelog or MOTD (message of the day) being exploited. Can't remember what game it was. Apparently they got so good at it that they've eventually managed to write a legit updater onto people's machines at roughly the same time they've started shipping real patcher in the installer of that game.
DNA Workshop. Poor Dan.
The crazy part of all this is that these games actually do have a way to patch themselves but apparently this was just something Sony had kept to themselves. Any game that used medius and also included DNAS (I say this because the only games that seem to have these packet handlers all had DNAS while the ones that didn't (socom 1, twisted metal black) don't) had the ability to read and write memory from the server. Sucks that they had to go through these sorts of hoops when sony already had the tools available for situations like this.
Dan uses this functionality for his patch and we use it for our patches for SOCOM 2 and Combined Assault.
I grow up playing r&c3. It's quite surreal to now see some of the underlying code that made my childhood and understand it. Amazing video, thank you!
the irony is that eula probably said you can't hack the game
There is a certain irony to all of this
Don't the EULA forbids others from hacking the game not the original developers?! That seems strange to me.
As Insomniac works for Sony and they broke the EULA, it's now canon
5:29 as a Cybersecurity guy, this one tickled me pink. Incredibly novel use for a very standard part of a hacker's toolkit
Dude this stuff is amazing - thank you for showing others how people like us think through these systems.
Seriously, the biggest thing I enjoy about your videos is how it logics through abstractness. This is an art form, my friend, and one that isn’t taught in school…we learned from guys like you, just in different ways (forums, IRC, more) - so thank you for carrying on the torch to keep others digging behind the code too!
Thanks for the kind words (:
Absolutely this. :)
Awesome video! I'm still really curious how the developers bridged the gap from the EULA buffer to the target function they were trying to overwrite without crashing the game. I would love a followup.
I was discussing this with the Horizon server devs, our best theory is that this was only used for rapid patching in early releases and the pointers may have been put closer to the buffer
@@nathanbaggsAnother possibility might have included clobbering data with the same values.
If you are overwriting memory with known values, you can include the values you were clobbering in the payload itself to avoid crashing.
well that brings back some memories! - good job Nathan! - and a great video!
Thanks! Hope it was accurate
Have you seen the developers (Tony Garcia and Mike Stout) let's play with their commentary on that game? It's up on youtube and full of interesting info about how they've made all of this run on a PS2 hardware. Love it just as much as your video, it's fascinating just how many little tricks Insomniac had and their technical knowledge is truly underrated.
No but sounds interesting!
This video is awesome. I can see you've put a lot of effort into it mate.
Thanks for the kind gesture 🩷
Hope your feeling better! Glad you completed this project it came out quite well. Never even thought about how a dev could use a vulnerability they really only have access to in order to send patches… crazy.
I wanna do a channel like yours some time, or a blog, to explain a weird zip-like format I found on a series of games... It took me ages to understand because I also stumbled upon lots of problems like you do... But also years have passed and I've forgotten some of the wrong paths I had taken.
Do it! It’s been more fun and rewarding than I ever thought it would be
have you worked with QuickBMS? it extracts all kinds of weird archive formats for thousands of games
There used to be a huge online forum called Xentax. It closed down a few years ago but it specialised in obscure and esoteric game file formats and had thousands of tools people had made to extract resources from just about any game, no matter how obscure the game or file type was. I'm taking, everything from AAA titles to proprietary 2010s commercial arcade game devices like Deal or No Deal.
Real shame it shut down. That 100% would have had the tools you need.
So in rare instances, two wrongs DO make a right? Oops, game has a major flaw. Oops, we left a massive security vulnerability. Let's use one to fix the other.
This was one of the best videos I've seen in a long time. Nice work dude. Nothing to suggest you as sollution, you went way deeper than I could've been. Good luck :) Post an update once you have!
Thanks!
so that's how they're able to play custom maps and game mods ! that's awesome!
Have you tried using the emulator's own debugging tools rather than Ghidra? You're probably feeding the emulator's RAM back to the game, not the game's RAM. The payload needs to maintain a stable game state up until the vulnerable point that can run the exploit.
As far as I can tell the snapshot is the game RAM (it had the callback pointers at the expected offsets)
I once patched Tenchu: Fatal Shadows to swap the function pointer of the options menu callback to a debug model previewer left inside the retail game. Pretty nifty.
Very cool dev trick from y2k. I always knew insomniac devs were based.
Ah, those were the days, I once patched an older version of the unreleased MU online client to handle a widescreen resolution it didn't support. The monkey wrenching in the executable was minimal.
Seeing the COP0 instruction in the disassemby output near the end of the video reminded me that the PS2 was MIPS-based...
Yes I had to give myself a crash course in MIPS
@@nathanbaggs I never got around to doing MIPS assembly... Although I did study the ISA extensively during my university days.
Glad this was suggested. I heard about this ratchet and clank update before, but there wasn't much info on it. Cool to see a video about it
Yeah I was surprised no one had delved into it before
I can vouch for Nath. I saw him copying his PS2 BIOS from my bathroom window.
Phew
i find it hilarious that the beta swears at you
Sure made me laugh
One of my favorite game series. So awesome to learn about such a unique scenario that occurred with this one!! Thank you!
You're quickly becoming my favorite channel on UA-cam man
Thanks!
We need to keep this old games alive. Modern gaming is absolute trash.
No it's not and if you think it is you are playing the wrong games.
My favorite developer among nauhgtydog with their technical skills they used and went with consoles of those era.
The patch they implemented was like 6 megs too and it was stored on your memory card. Your 8 meg memory card. Seriously though, UYA online was so fun!
I was going to get a membership the other day, but I forgot. This video made me laugh a bit though, and you definitely deserve it. Keep up the great work, I absolutely love coming home from work and watching your videos
Wow, thanks!
Out of interest, what was the cheat / hack that players used on the game prior to patching?
Haven't a clue what any of this means but still good to watch
From what I could understand of the Game Developer article, it seems to me that they utilized the EULA itself to patch the game. I visualize this as follows:
1. They replaced (parts of) the EULA with patch code (which would at this step be processed as mere text), and overfilled it past the brim. This overflow would eventually reach a variable that contained an address.
2. They replaced this address for an address within the EULA buffer, meaning the later function callback that used that variable would send the pointer back to the EULA.
3. The pointer would then process the patch code in the EULA as instructions, leaving them free to do as they liked (as long as their patch code didn't ruin the function callback).
Pretty sure that’s what I was aiming for
@@nathanbaggs Ah, so I did manage to follow. I don't have much experience with low level programming, so it all kinda made my head spin
I want more video of this kind!
That’s good as I want to make more videos like this (:
@@nathanbaggs Nice 😍
Look at that! So if they never used original C "bad" strcpy, they probably would never be able to patch their game or would have more difficulty. A unsafe C function, was the key for their success! C "do whatever you want, even shoot yourself in the foot" philosophy, that many critic, apparently was what helped them in the end.
glad he has a "Legit" bios
Video would have been a "no go" if I couldn't get one
Please do more videos like this, especially on how to reverse engineer programs written in different type of languages and architectures.
It's possible the game crashes only if the second packet doesn't arrive quickly enough.
They might had even done something fancy and packed both command packets into the same IP packet so that there is minimal delay between the two events.
I packed all the data into one message, I'm not sure how the proxy actually handles splitting it up into packets but I couldn't see anyway to shovel the bytes down quicker
LETS GO DAN
dan the man with the plan
and a pram above a dam
I played UYA for a little bit in 2007 while the servers were still active; I don't know how much memory would you need to overwrite for the buffer to overflow, but it couldn't have been that much, right? Otherwise it would have taken forever for the lobby to load on slow connections back then, and while my DSL wasn't the worst, I don't remember much waiting between the EULA and lobby screens.
There’s still some questions around how often this was actually used and on what versions. All part of the mystery
@@nathanbaggs It would have to have been used every single time you started up multiplayer. The game didn't run off a hard drive so the patch would be lost.
could it be that the emulators heap allocator is just different from the original? Maybe it does a better job if defragging / hole filling, so some important stuff gets allocated in between the buffer and pointer, whereas on the original device that wasn't the case. The defragging may also be more volatile so whatever gets allocated in there may effectively be random or at least not exactly the same every run hence why it still crashes when you fill the packet with the same memory.
It would be cool to analyse all heap allocations in that memory region.
As far as I can tell the buffer is in static memory. It’s always at the same address but I will admit to not knowing a lot about the inner workings of PS2 memory to be sure
Program at work, come home watch some nathanbaggs, program even more, sleep, repeat
Understood nothing of it but really enjoyed the video nonetheless! I hope you can figure it out later on
Did you consider asking the game developers for more info?
I did reach out to the original author of the article (who still works at insomniac), he left a nice comment on this video
Reminds me of how AIM exploited an RCE in their own DLLs to try and keep out third parties (particularly MSN)
2:59 I'm sure someone with your technical skills would never borrow a PS2 BIOS from someone else, right? :3
Who me? No of course not
I mean, we are literally asking developers to simulate the real world AND to run it optimized on all sorts of hardware. If that isn't what "making games" is all about then I don't know what is. It's the reason why technology has developed so quickly.
This is an amazing story!
I wonder if that EULA trick could be used to install freemcboot
Absolutely it could.
I would imagine it is something related to the emulator. They do a check somewhere or other for a valid memory which was not happening on the real hardware
Also a question here
Why did they even bother ?
I mean the game released , sold well and i don't think sony had them contracted to do any sort of post release updates (which they couldn't do easily )
So what purpose outside of some sort of pride did they do this for
As far as I can tell people were cheating in the multiplayer, so they wanted to patch it to improve them experience for everyone
I wonder if that Dan is the SnowDan.
9:00 LMAO it's DNA Workshop but your pronunciation was way better
We all make mistakes…
chaotic good
I believe his screen name is pronounced "DNA Workshop".
We all make mistakes
Amazing stuff
Finally, a EULA I want to read 😂
Ratchet deadlocked the best one, going commando 2nd best one don't @ me
@rch5395
words: "legit"
actions: ( ͡° ͜ʖ ͡°)
Commenting for algorithm, this is a cool vid
Really interesting
definitely legal copies
Only the most legalist
That was cool 👍
Well, assuming there was no https or packet validation from server back then, and game code leaked IP's (assuming some p2p in chat/voice/server code) you could crash other people with this exploit in hand. Thinking similiar situation happening in COD lobbies of that time would be a huge impact. Glad we were limited to ratchet & clank
The vulnerability is in the EULA code, not gameplay. The server owners/a man-in-the-middle are the only ones that can utilise it.
It only works if you can control the EULA, which means you’re pretty much limited to man in the middle-ing a server. It’s not something you can use to remotely target any player
You must be quite young - we had https in the 90s let alone 20 years ago
@@JohnnyWednesdayI would refrain from that tone especially since it is far from certain that it was in use. Development culture did not shift to "https everywhere" until remarkably recently.
@@JohnnyWednesdayyoure quite right we had https back then but was not mainstream like today and was not well utilised in gamecode
Hello 😊
Game Development hasn't been hard for a decade. Just sayin. It may be hard to do it well, but overall, since the market is flooded with mountains of trash "product", it can't be that hard, can it? For every "difficult" game there's a thousand trash titles that were churned out over a few weekends. The illusion of difficulty in game development is nothing but a marketing ploy designed to embellish the value of any given product as it relates to the absolute Ocean of products they are meant to "rise above". You're not wrong, it is all smoke and mirrors. But you didn't notice all the mirrors on the way in.
Maybe your experience of game dev differs to mine, I’ve never been able to see anything through to release so game dev is certainly hard for me
"Game Development hasn't been hard for a decade."
If you're talking about shovelware trash, yea it's piss easy to crank out that kind of crap, but in context of aaa games, it's gotten signifignantly harder.
You ignorance of game development is staggering.
There's a huge difference between an assetdump, and a game that's had actual effort put into it.
@@nathanbaggs it's never been easier than right now to make your own game and put it out there. Every tool you could ever need is available in half a dozen mouse clicks. You should just brainstorm a simple gameplay loop and make a demo. Put your own name on it and release it.
Then you won't have that excuse in your pocket anymore.
Not an excuse on my end by any means, all software dev is hard and I don't want to belittle anyone's effort just because it would have been harder to do 20 years ago. But I will accept your point that the barrier to entry is much lower (which is a good thing) now (:
@@ZennExile Bro you legitimately have no fucking clue what you're talking about.
Again, yea, you could make a game with "a few dozen clicks" but something that's actually worth playing and of good qualilty is signifignantly harder than that.
There's a differerence between shovelware that you'd see on jim stearling's old vids and stuff actually made by devs.
Just look at how much of a clusterfuck cyberpunk 2077, it's a daunting task to handle if you're doing anything of actual scale instead of just something shat out from unity/Unreal Engine.
Now the next step is to get a reverse shell on the target's machine 🫠