I love how he’s gotten to the point of resurrecting game consoles and now has surpassed that, and is now trying to bring back a save file. What comes after I don’t know and I’m interested to see.
Alright guys so my friend had a heart attack yesterday, and I've been getting into biocoding recently. Anyway turns out after a lot of necromancy research all he really needed was 12.5 V to pin 3.
I'm hoping he follows a similar pattern to the UA-camr jdh. He started by making Minecraft in 24 hours, then built his own game engine, then made his own operating system, his own graphics card, and finally his own computer from scratch on breadboards
You can easily do it with Soul Calibur 3, because it has a… well… save breaking bug. Not even a rare one, it just requires you to erase an save file, that‘s older than your SC3 save file. That definitely wasn‘t uncommon back then
It looks like those missing bytes were for the completion loading message. at 0:19 it has: "Only one profile can be loaded per game session. You will need to reset the console..." but when those bytes are missing at 16:19 the message stops at "You will" the end of the string could be those missing bytes.
@@aprofondir Not exactly. You need a lot more than 9 bytes to store the rest of the loading message in plaintext. The bytes probably tell the game which messages to display. You might get random pieces of other messages or crash the game when decoding fails. If this is the PAL version, there are probably multiple languages available which means there are multiple versions of the loading message (at least one for each language). For whatever reason, using the saved language setting wasn't good enough for the developers or they wanted additional padding for the checksum and encryption.
This reminds me of when I had to rescue my save file from the Spyro Reignited Trilogy on Steam. I too was going for an 100% save file and was nearly finished with the game, when while playing it suddenly froze while saving. I realized I had run out of disk space. Freeing up disk space didn't cause it to unfreeze, and I noticed it had already deleted the original file and started partially writing the updated save file. It was a zero byte file though, so there was no trace of it on the disk anymore. I ended up freeing up space and doing a memory dump of the process and manually searched through the game's memory. I figured that the save file was still in a buffer somewhere in memory, and to my astonishment, that guess was correct. After searching for the game's file header in memory, I came across a large block of save-file-looking data. I copied it out of the process memory and into the save file. Killed the process, restarted the game, and crossed my fingers...and it worked! As always, great video man!
At 1 point i was on a 100% run for Spyro 1 in the remastered trilogy but my heart sunk when i honestly completed everything but 1 room that has bonus treasure Gnasty's loot i think didn't open up so i had a save extremely close to 100% but it killed alot of feels to keep on playing the rest of the trilogy
I find it hilarious to imagine that those 9 bytes did actually mean something, and you have a random car that is missing a rear left wheel or something like that.
actually, if you look at the completion percentage screen when he loads the uncorrupted game for the first time, the text abruptly ends at "you will" so that means the only thing that was truly corrupted was the second half of the completion percentage text. Which is probable the luckiest thing possible to be corrupted, because as soon as the game autosaved, nothing was lost. but i still find it funnier if it was just some random aspect of some car
@@thekeyboardwarrior1018 If you are writing games in python... the code is plainly readable. Can't someone just look at the save file loading function and then reimplement that to get a readable file?
This was a really good one. Instead of centering on hardware like this channel usually does, this video is more about figuring out how to force software to do what you want. I really liked that.
Yeah, same with the lego island videos. Gotta love problem solving in a confined workspace (logically speaking, with the original game code setting the boundaries)
I remember completing The Great Escape on the PS2 and the save you get after finishing the game corrupted the entire memory card. Lost literally months of game work. >:(
it's also similar to developers taking 10+ hours to writing a script that automates a task that takes minutes to do and will have to be done like 10 times
Or putting a cutting edge capacitor in a console to power a massive, power hungry chip, just to keep time, instead of moving timekeeping to another chip and using a coin battery.
@@thesecretlifeofdukelee and you promise yourself you'll reuse the code when you do something similar but you end up writing it from scratch next time anyway
@@KosteonLink I might done that for several projects several times. I'm doing it right now. Losing time rewriting code instead of thinking about new features.
Another major problem with brute forcing a the reverse of a 32 bit checksum with more than 32 bits of variable data is that at least some (all, if good algorithm) checksums will have multiple solutions due to the pigeon hole principle.
@@_notch Same. I recently came across and watched his Mario 64 video and some of the cheap Japanese console videos among others, which made me get into his content. Good stuff!
Even if the bruteforcing could complete in our lifetime, the fact that the number of possible combinations is bigger than the size of the checksum itself (4 bytes = ~4 billion) means that there are most likely an overwhelming number of possible combinations for those 11 bytes that would produce the same checksum, without any way of determining which combination was the original one.
well, I mean that's just how checksums work. If you have 4*8 holes to stuff pigeons in that's 2^32 combinations, but you have several times pigeons than you have holes so some pigeons are going to need to split the rent. Checksums, hashes, etc. will *_always_* map several inputs to identical outputs, the important bit is that they data that *_does_* map to identical outputs is so far apart that it will fatally fail and won't be mistaken for real data. Even if every single other byte in the file is completely ruled out (which it can be) you still have 11 bytes of variance to map onto a 4 byte sum. The only way to have no repetition is to have a checksum/hash that's as large as the original file, at which point you can just send the file again.
Fun fact: I also lost all of my progress in Test Drive Unlimited once because I accidentally turned it off during one of the autosaves (it autosaves all the damn time). I had to start fresh and decided to turn off the setting just to be safe. Also, this game has full 16:9 widescreen support and it makes the game look much better. The only way to activate the setting is by actually setting the Playstation 2 itself to 16:9 in the main menu.
@@Timic83tc The autosaving? I'm gonna be honest I don't remember. I think the Xbox 360 version also always autosaves, but it's much quicker and therfore you're much less likely to accidentally power the console off during an autosave
“Dad? That old computer beeped!” “Oh, So it is finished.” “What’s this?” “It’s a checksum… that my ancestor wanted.” “What checksum, dad?” “Test drive unlimited” edit: ".. what's Test drive unlimited?" ".. i don't know"
"What's your favouirte episode, Billy?" "I like the one where MattKC explains checksums for a PS2 game from 2006 for a looong time!" "You're goddamn right you do."
You absolutely could've just replayed the game to get back to where you were before the save corrupted, but I believe there's much more value in this issue being motivating enough to drive you to make an entire video on it, giving your viewers extra entertainment and education that we would've missed out on had you given up or restarted. Thanks for that!
Counter offer: Play the game all the way back to where you were then fill in the original missing bytes with the bytes from the newer save file, restoring the original and then completing the game with *that* one.
0:58 For future reference, you should try opening and closing the disc tray. I haven't had to try during a save, but it's unfrozen my games in the past without repercussions
@@MuchWhittering That would be a little disappointing in my opinion. The fact that this could work on original hardware is what makes this so interesting to me
@@MuchWhittering He literally just made a video before this one showing how to ruin your PS2 with a bunch of random exploits, and said _in this video_ that he transferred the save over a USB drive...
Woah. It's like magic. Fun fact: when my PS2 broke, I had created a backup of save files just like you did, and when I tried to load the latest one in PCSX2, it said 'Data Corrupt' (Prince of Persia: Warrior Within), with a new logo (usually it would be the place and time where I'm at) Thankfully, I had backup saves.
@@gamechep oof, but you had backup saves. I do the same with my DS Flashcard and 3DS SD cards. This is mainly because one time I was putting in some files when the USB disconnected and it got corrupted. Nowadays, I always create a backup before doing anything to the SD, so if it disconnects while the PC is doing something, nothing bad will happen.
@@pacomatic9833 Try using FTPD if the files are not that big. I use it because otherwise I'd have to unscrew the back panel, one of whose screws is already broken.
@@gamechep I miss the SD card slot on the slide, I know they did it so people don't accidentally fuck something up but it made adding music and stuff SO much easier. Now I need to find my screwdrivers and microSD adapter.
This was fantastic! As a fan translator for many a game and of course many game file formats, it's always a treat to watch others doing similar reverse engineering work. Awesome~
Absolute madlad. I've tampered with savefiles, editing hex values and tried to gerenerate checksums myself, but to the point of putting the game inside a simulator to edit the assembly code, truly madlad
[For How Many Years It Would Take For All Approximate Combinations are only 134,430.635921 Times Longer Than The Age Of The Universe by using the diving factor of the universe's age being 13.8 Billion Years and dividing it by 1,855,142,775,714,214 or 1 Quadrillion, 855 Trillion, 142 Billion, 775 Million, 714 Thousand 214 Years To Get The Answer.] Number Location for 1,855,142,775,714,214 or 1 Quadrillion, 855 Trillion, 142 Billion, 775 Million, 714 Thousand 214: 10:42
This man is becoming too powerful for his own good, and I'm loving watching it. I doubt I'll ever be *this* good at software, but it's still damn interesting. I'd love to know a bit about your background, where you learned everything you know and what got you into all of this.
I greatly enjoy these stories of discovery and deduction with a gaming setting. I would honestly be thrilled to have an entire series on the bit level workings of various file formats... I've partially reverse engineered unity's gameobject-component format (with some very major and necessary bits missing). But the sheer complexity of some formats would likely make for very engaging content (at least for people like me). I'd definitely want it from someone like you, as well. The way you talk definitely makes these feel like proper stories and I really wish I could find more content like it. 8-Bit Guy sometimes puts out similar stuff, but this sort of thing is probably a little too far from his niche to be more than a blue moon sort of event. If any comment readers have suggestions for channels, I would love to have them.
Check out "Retro Game Mechanics Explained". It's not precisely file formats but should be in the vein of what you're looking for. "MissingNo.'s Glitchy Appearance Explained" and it's kind-of-sequel "Pokémon Sprite Decompression Explained" were in particular really interesting to me.
Not exactly what you're asking for, but I've really enjoyed Kaze Emanuar talk about optimizing the hell out of Mario 64 on real hardware, and suckerpinch talking about...lots of things, including "reverse emulating the NES."
Been trying to revive my Monster Hunter Freedom Unite save file, so this was pleasantly relevant to me (I have a backup at the 500 hour mark, so it's not a huge loss, but I figured I'd try to rescue my current save anyway).
Lol I had a MHFU save file corrupt after I dropped my psp while it was saving. The battery flew out and when I turned it back on the file was toast. I had literally just gotten to g rank too.
Was the FISH tag generated by the encryption function or some generic save routine? If it was the output of the encryption, then perhaps it's an indicator of the save using Blowfish for encryption?
Just a file format indicator. Every single (sensibly implemented) file format uses one. It's used mostly to check if it's reading the right file/it's in the right format.
Everything MattKC has made so far has been interesting for me. This video reminds me of the time I worked as an embedded systems developer, looking through memory dumps, trying to figure out what went wrong. I'd like to request for a video on how to debug using modern tools. I want to understand the process you used to find the checksum code in the disassembler.
idea for if your making an autosaving game: keep the original save file in the games memory till the game realises that the save is finished. Trying to load the save file and seeing it is corrupted, delete the new broken autosave and load the old one, although you might have to make more regular autosaves or you could lose a lot of time.
You don't necessarily need to load the save file to check if it's ok before continuing, since that is pretty unoptimized. Rather, include some sort of exit code for the save function. Some pseudocode: save file.temp exit ok if exit = ok, then move file.temp to file.sav if the game saves often enough, players likely won't lose more than a few minutes of progress. also you could add a bit in the load function to try to load file.temp if file.sav is corrupt in the event something goes wrong during the move (although moving files requires very little time and is handled by the filesystem so it's vanishingly unlikely anything will)
“Keep the original save file in memory until the game realizes the save is finished” This wouldn’t work for the example in this video, since the game crashed and froze mid save (it’d never get to the “realized the save is finished” part, and at that point it’d have already partially overwritten the save data on the memory card). The only way to account for the issue in the video would be to have two save files on the persistent storage device (memory card).
@@patientallison There's nothing inefficient about loading a save file to ensure it was written successfully. File IO isn't really a costly operation, and loading a save file doesn't necessarily mean also restoring the state that the save file describes. That would be the inefficient part.
The same corruption thing happened to my gran turismo 4 save after completing all of the license tests. Needless to say, I was quite mad at having to do those 2 hours of boring tests again.
This happened to me an hour ago, but with a 88.2% completion file, just after driving the WHOLE Nürburgring 24h endurance race. I lost over 100 cars and a whole year of playing this game, even the Formula GT car I had just earned by winning this event.
As someone who's been staring at a hex editor for the past 3 months smashing my head against the wall to reverse-engineer a file format, this was a very fun video to watch. Thank you.
seems more likely those nine missing bytes of "save game data" were really also related to the signing key. The game must have crashed trying to sign the file. Presumably a bug in the signing algorithm. This makes much more sense than it crashing trying to write the last 9 bytes out of nearly 100k of actual game play data. That's a very low probability...
@@skyleite2065 if you assume that there's an equal chance of it failing each time you write a byte then the vast majority of failures will be before the last 9 bytes. That's my model of what's going on. What's yours?
@@xeostube I don't follow. If there's an equal chance of it failing each time you write a byte, why would the majority of failures be before the last 9 bytes? Those two statements sound contradictory to me
@@skyleite2065I'm presuming it stops once it fails a single time. So it had to succeed each of those ~100k times, but then fail at the end. Here's a simulation in matlab of the basic idea, with a failure rate of 1% per write. a=[]; for (i=1:10000) x=find(rand(1,100)>.99, 1); if (x) a(end+1)=x(1); end end sum(a==100)/length(a)*100 the last line calculates the probability of a failure on the final 1% of the data written, which is 0.46% the time I ran it. Or in frequentist terms, out of 10,000 simulations, failure occurred writing the last 1% of data only 29 times; 6267 times it occurred at an earlier point.
Really interesting video. I remember when I was a kid in the PS2 era, coming across a few corrupted save files on my memory cards. They are still there to this day...
Just wanted to say thanks for documenting the process of how you went about debugging your corrupted save file. I recently wanted to play the game "Stuntman" on PCSX2, but due to a very well known arithmetic-inaccuracy emulation limitation the player is always stuck at a particular level and is forced to use in-game cheats to unlock all levels. Instead of cheating entirely, I tried modifying the save game, but was met with errors that hinted that the memory card file was 'corrupt'. After you pointed out how checksum are integrated into saved game data, I did note that the save game file had 4 hex values which would consistently change with every save, indicating that the values were used as a checksum. Using this information I utilised the debugging tools that PCSX2 has to find in memory where the checksum was written and read from, and found the specific branching condition executed which ultimately validated the checksum (with the valid checksum value also visible in the debugger). Bypassing the check by editing register values within the EE CPU let me load the save. Initial effort when editing save game data was very little and I wasn't as invested in getting something going, but after watching your video I had a bit more patience and motivation to try to at least understand what might be happening and adapt it accordingly to a slightly different problem in the same domain, as well as learn a thing or two and feel a bit more comfortable using the debugging tools... So yeah, thank you.
Test Drive Unlimited is a really interesting game on PS2 and PSP, given that it has a completely different (and I would say much better) handling model than the 360 and PC versions, plus the addition of clubs. Really fun game, would recommend to any person who likes cars or racing games.
and "TDU: Solar Crown" is due to come out in September, about 11 years after TDU2 was released, and 16 years after TDU1's initial release in 2006. So excited!
Man, I remember when I got my first PS2 save corruption. I was playing the PS2 version of Sonic Unleashed, and at one point (don't remember when or if I shut the console off during a save) the game kept saying that it couldn't load data because of a corruption on the memory card, and was asking me to format the memory card to solve the error. This was FAR before I became as tech savvy as I am today, so I had no idea what "formatting" meant. I just agreed to format the memory card thinking no harm would come of it. I had complete/near-complete save data for games like Kingdom Hearts, Ratchet and Clank, Jak and Daxter, Sly Cooper, the Spongebob PS2 games, and even more I can't easily name, on top of being in the process of completing Sonic Unleashed, so you can PROBABLY imagine how much I cried when I later found my memory card empty of months/years of game progression.
11:24 It shouldn't actually take that long. 1. The checksum only has 4 billion possible values not septillions. On average you will only need to calculate 2 billion. That changes the time to 5 days. 2. Even single threaded you can get WAY more than 5k checksums per second. I would expect hundreds of thousands of hashes per seconds. You can also optimize it by saving partial computations. So to incremented the last byte you only need to call checksum once since you already have the checksum of all of the previous bytes. It's also trivial to make this multithreaded by partitioning based off the first byte. With all of those optimizations you should see millions of checksums per second. It would then take only an average time of under a minute. Another potential optimization would be trying to solve for the last byte which would give you like a 100x speedup bringing you down to below a second to solve. In summary the time it takes to finish will be measured in seconds and not billions of years.
As a researcher, I know how that feel when you try to understand, and come up with a solution to a problem. Of course, it feel even greater if you can solve it in the end, but the whole problem solving process is already fun enough! I have subscribed to your channel, hope to see more of your thought process during problem solving!
Fun fact, Matt: That checksum algorithm used in the game was supposed to be the famous, _ubiquitous_ CRC-32. Just implemented kind of backwards, so the bits were reversed, and possibly using a different polynomial. And of course, messed up in such a way that ignores 75% of the information you give it (Like me...)
i loved this video! it's been great seeing you upload so much while still retaining quality. the solution at the end was so cool, with the reveal that the save essentially wasn't corrupt, and was able to run when brute forced through.
These videos are really nostalgic for me, I remember tinkering with my consoles back when I was a kid (homebrewing the Wii, hardmodding the OG Xbox, etc.) and I hope you never stop this content. Truly great stuff.
I've had to do the exact same thing on another game. Unfortunately the level of corruption was a lot more substantial for me and I wasn't able to rescue it. It was a fun attempt though. These sorts of diversions almost become part of the games for me.
Hell yeah dude, nice one. I had a similar issue with my Ocarina of Time save a number of years back and ended up deep down a rabbit hole with a GameShark, a parallel cable, an old Pentium II PC and some very janky software before I finally emerged with my childhood save intact. I feel like I went on a similar voyage of discovery regarding checksums etc. as you described here.
I remember as a kid messing with the original XWing game on PC to modify the save files. I used a hex editor to compare files and change stats to unlock missions and trophies or whatever they were called.. No checksums, no encryption. I then wrote a program that would read a file, ask you a bunch of questions about what you wanted to modify, and spit out a modified save file. (The initial version just modified every byte I found to give you full stats). Soon after I found a much more competently written Xwing Editor, but man that was fun.These days kids would have to learn about assembly, emulation, and cryptography to get anywhere with most games, so it's a much higher bar for entry.
I once (with help) fixed my corrupt Pokemon Diamond savegame! I tried tweaking, and accidentaly saved in the wrong spot (in the void). Couldn't load up my game at all anymore, blackscreen when I did. Since my New 2DS XL is jailbroken, I dumped the savegame and fixed it with a hex-editor and certain savegame-editors for Pokemon. When confirming it worked with an emulator, I wrote the savegame to the actual cartridge. Fun experience.
Disclaimer: I am not a programmer and do not know the technical details of how checksums are implemented. That said, something about needing to run through all possibilities for 11 bytes to match a 4-byte checksum seems off to me. Because you have more missing bytes of data than there are in the checksum, there's going to be lots of possibilities for those missing bytes that yield the same checksum. This seems like it would make it impossible to find the exact missing values but much easier to find *some* set of missing bytes that agree with the checksum. (Couldn't you get away with keeping 7 bytes 0 and only messing with the other 4? Statistically speaking, it seems likely that you'd be able match a 4-byte checksum by altering 4 bytes of data to all possible values.)
that WOULD be correct if your only objective was passing the checksum. however, the goal here is to recover the real save file using the checksum as a way to help, meaning we still need to search through all possible options to make sure we find the right one. also, brute forcing just the checksum, while not as long as the lifetime of the universe, still takes roughly 250 days at the rate Matt's program works, meaning its still faster to just play the game over again
What you're talking about is called a collision. That's when you can find two sets of input values that will create the same exact checksum value. The thing about collisions is that they can also take a long time to find. (This is especially true with hashes, which are similar to checksums.) Think of it this way -- in Matt's case, there is a "correct" input sequence of 11 bytes he is looking for that will create the 4 byte checksum he has. And yes, there are very likely other sequences of 11 bytes that will create the same 4 byte checksum he has (i.e. collisions). However, there's no way of Matt knowing how many collisions with his 4 byte checksum there will be. There could be trillions of collisions, or in theory, even zero collisions. Yes, intuition may say that if he only changed four of the 11 bytes, he could find a combination of just those four bytes that would create the same 4 byte checksum. However, that's not guaranteed at all; even in this case, there will be a lot of collisions where several input values will create the same 4 byte checksum (not necessarily the checksum he's looking for), meaning there will also be a lot of possible 4 byte checksum values that will never be seen, if that makes sense. And his could be one of them. A related thing you might find interesting is the Perfect Hash function. To be brief, it's a hash algorithm that creates a unique hash / checksum for every input value you give it. However, in order to make a perfect hash function, the final hash / checksum value MUST have more bits than the values you're hashing (44% more at the very least, but in practice 60%+ more, and even then it's hard to do).
@@devmas There can't be 0 collisions if the size of your data is larger than the size of the checksum. It's mathematically impossible. This is why hashing algorithms are so difficult, because they have to take an infinite number of inputs and produce a finite number of outputs while making it near impossible to find collisions, even though collisions must exist.
@@SmashhoofTheOriginal Sorry, I should've made it more clear that I meant zero collisions with that one specific checksum he's looking for. Of course it's mathematically impossible to have zero total collisions with a smaller checksum than data.
This reminds me of how I cracked some saves of DS games, those actually had about 4 CRC32 checksums to check you didn't tamper with it and the latest games of that series also added encryption and something else that seems like a homemade compression algorithm(which I haven't figured out yet unfortunately), had a lot of fun figuring all of this out and making a save editor with this info
this was the first ever mattkc video the youtube algorithm recommended to me like more than a year ago at this point. i like to revisit this one every once in a while to appreciate it.
This gave me hope to fix my Ape Escape 1 pal being stuck at 99.9% completion (yes I have absolutely everything). But 99.9% of this video went right over my head.
I'm so happy and surprised to see you talk about TDU1 on ps2, such an obscure and underrated game. It's a childhood classic for me as well. Props for using its fantastic OST in the video !
You've outdone yourself, this is one of your coolest videos. Also you made me want to play this version of the game. I played it some on the 360 and PSP but...eh, seems like it might be fun.
Nice to see TDU getting some love, grew up playing it on an AM2 Athlon X2 PC with an old Radeon RX600 ProGuru. Tremendous game that was so ahead of its time and so impressive in terms of variety, car selection, Terrain, Sheer map size and detail. It's a shame the sequel was such a departure leaning more towards the then popular Casino elements, but the 1st game was amazing and I can highly recommend it
This was so cool! I'm currently working on a project for school that involves similar crypto and byte examination, so seeing this come out was really neat and I actually understood a lot of it.
Awesome job! Seeing your process was very informative (even as a software developer) and it's a good reminder to keep stay aware of tunnelvision while working towards a goal. Patching the game was a good move and I am happy that you managed to play the game to completion!
Man, TDU was so good. I was just getting into sim racing; got my first force feedback wheel. Vividly remember going for an hour long drive for the sake of it in my teenage years, easily one of my first truly immersive experiences on my 23 inch Hitachi CRT monitor. While it wasn't a sim, that was vastly outweighed imo by granting one of the first true 'going for a drive' experiences; I almost enjoyed the more normal cars to the exotic ones for that.
9:40 Hold up; you're telling me that I've been making passwords that look like hashcode when I could've just used normal-ass words and I would've been *MORE* secure?!
The security of a password depends _both_ on how jumbly it is ("hashcode-like") and how long it is. The comic's author is advocating for taking advantage of the latter.
What a journey through the depths of this game's code! It was fascinating to watch you figure out and reverse-engineer all of these security measures. If you have the opportunity to do so again, I'd love to watch you do the same thing another time! :)
Nice job! I don’t understand any of the technical aspects of this, but your solution and tenacity at figuring this out is fascinating! So cool and so are all of your videos! Thanks! 😀
I can't get over how, from a technical standpoint, fortunate you were that it was just a signature creation corruption. If *any* of the rest of the save was corrupted/nullified, you probably wouldn't have had much of a chance of getting it back like you did. For instance, with a Bad Egg Pokémon, the corruption of the Pokémon's data varies. Sometimes the data can be so corrupt that there is no way whatsoever of restoring your Pokémon.
This is an incredible vid: extremely interesting and insightful, yet also wonderfully relaxing. Every time it’s thumbnail pops up at night I feel nostalgic
The main reason why I always do backup saves of all my games once or twice a month on a external drive. Especially when some of the games have more than 150h of progress lol
Please, do more of that Hex-code researching. I know it might be tiring, but it’s very interesting. Besides that, very well made video and I never get tired of your channel!
Video game save files are analogous to configuration files like autoexec.bat and config.sys, except they are normally binary. All save files have a specification, and you can figure things out by looking at the file, making educated guesses, changing bytes, and trying out your edits with the game. I haven't looked into it, but I bet that specs for save game files for common games are published online. I haven't written many games, but I've worked on many programs with configuration files.
Awesome video! Been super interested in a lot of your more recent videos, though I have never been too good with hardware so that has always been more of a thing I like watching but not doing. But software has been more of my strong suite and where I've found more of my interest lies and so its cool to see you doing something more like that, even with hex editing and stuff.
@@ComicusFreemanius sorry for the late reply, youtube didn't notify me of this and I only just noticed when going through older comments I left. First off thank you! I really appreciate you saying that. Secondly I have messed around with unity before, I had been working on a small side game project in it, that skatepark tycoon game, just haven't had the time to be able to put into it, been trying to find a job which has been stressful and still trying to get the last big update for my thpm mod done too so its been a lot, but I do hope to get back to it someday.
This is so interesting to me as I’m trying to research how to do something similar with a Nintendo switch save. There’s a lot less information to work with and a lot more security to get around, but the rude customer service agents and the lazy devs are what’s motivating me.
i love your videos! You have just the right amount of technical detail and you make me feel like I'm there with you doing the project. Keep it up! If you ever end up in Italy, beer's on me
I found this channel last night and I'm amazed at the quality of your content, as a game/console modding novice who's been at it since childhood watching these types of videos is as relaxing as it can get, keep up the good work! 💙
I love how he’s gotten to the point of resurrecting game consoles and now has surpassed that, and is now trying to bring back a save file. What comes after I don’t know and I’m interested to see.
Yes
Can I fix a CORRUPT Singularity in a black hole?
game necromancer
Alright guys so my friend had a heart attack yesterday, and I've been getting into biocoding recently. Anyway turns out after a lot of necromancy research all he really needed was 12.5 V to pin 3.
I'm hoping he follows a similar pattern to the UA-camr jdh. He started by making Minecraft in 24 hours, then built his own game engine, then made his own operating system, his own graphics card, and finally his own computer from scratch on breadboards
I selfishly hope there are many more corrupted save files in your future! Was very entertaining to watch you puzzling this out :)
Hi sebastion!
me too. but not because i want to watch you fix them but because im EVIL
Sebastian I love your videos!!
Of course.....Seb is back at it again
You can easily do it with Soul Calibur 3, because it has a… well… save breaking bug. Not even a rare one, it just requires you to erase an save file, that‘s older than your SC3 save file. That definitely wasn‘t uncommon back then
To be fair, reverse engineering the game save format sound like much more fun than playing the same game again to 75%
same for me, but it's not fun for everyone and you need some basic knowledge of file index,hex etc
It looks like those missing bytes were for the completion loading message.
at 0:19 it has: "Only one profile can be loaded per game session. You will need to reset the console..."
but when those bytes are missing at 16:19 the message stops at "You will"
the end of the string could be those missing bytes.
bro you just solved the mystery
Well, mystery solved
If he had just filled those bytes with random characters would it just have been nonsense text in the message?
@@aprofondir if what the comment is saying is true then prob yea
@@aprofondir Not exactly. You need a lot more than 9 bytes to store the rest of the loading message in plaintext. The bytes probably tell the game which messages to display. You might get random pieces of other messages or crash the game when decoding fails.
If this is the PAL version, there are probably multiple languages available which means there are multiple versions of the loading message (at least one for each language). For whatever reason, using the saved language setting wasn't good enough for the developers or they wanted additional padding for the checksum and encryption.
This reminds me of when I had to rescue my save file from the Spyro Reignited Trilogy on Steam. I too was going for an 100% save file and was nearly finished with the game, when while playing it suddenly froze while saving. I realized I had run out of disk space. Freeing up disk space didn't cause it to unfreeze, and I noticed it had already deleted the original file and started partially writing the updated save file. It was a zero byte file though, so there was no trace of it on the disk anymore.
I ended up freeing up space and doing a memory dump of the process and manually searched through the game's memory. I figured that the save file was still in a buffer somewhere in memory, and to my astonishment, that guess was correct. After searching for the game's file header in memory, I came across a large block of save-file-looking data. I copied it out of the process memory and into the save file. Killed the process, restarted the game, and crossed my fingers...and it worked!
As always, great video man!
this is an awesome story, good thinking!
At 1 point i was on a 100% run for Spyro 1 in the remastered trilogy but my heart sunk when i honestly completed everything but 1 room that has bonus treasure Gnasty's loot i think didn't open up so i had a save extremely close to 100% but it killed alot of feels to keep on playing the rest of the trilogy
@@cydragon2.099 rip
That's awesome. I wonder, though, whether Steam's cloud saves could have helped you recover your previous save file if it hadn't worked.
100% I think you mean 120%/100%/117%
I find it hilarious to imagine that those 9 bytes did actually mean something, and you have a random car that is missing a rear left wheel or something like that.
actually, if you look at the completion percentage screen when he loads the uncorrupted game for the first time, the text abruptly ends at "you will" so that means the only thing that was truly corrupted was the second half of the completion percentage text. Which is probable the luckiest thing possible to be corrupted, because as soon as the game autosaved, nothing was lost. but i still find it funnier if it was just some random aspect of some car
It would be even more funny if it was just a bit of dirt on the ground in the game
Why would game assets be stored on a save file?
@@thekeyboardwarrior1018 If you are writing games in python... the code is plainly readable. Can't someone just look at the save file loading function and then reimplement that to get a readable file?
@@tcoren1 what if its a pointer to a text?
This was a really good one. Instead of centering on hardware like this channel usually does, this video is more about figuring out how to force software to do what you want. I really liked that.
I initially subbed for software related content in Lego Island. So for my interests this was a return to form sort of.
Yeah, same with the lego island videos. Gotta love problem solving in a confined workspace (logically speaking, with the original game code setting the boundaries)
"Instead of centering on hardware like this channel usually does"
As someone who watched the lego island videos at least 5 times each, I disagree :D
I remember completing The Great Escape on the PS2 and the save you get after finishing the game corrupted the entire memory card. Lost literally months of game work. >:(
Oof
You should give it to Matt lol
Shit, you played it. Fuck it bowl
Ps2: imma ruin your day
331st 👍
"And, yeah, somehow, this still seemed preferable to replaying the game from scratch." Now where have I seen that energy before
it's also similar to developers taking 10+ hours to writing a script that automates a task that takes minutes to do and will have to be done like 10 times
@@thesecretlifeofdukelee yeeeeep
Or putting a cutting edge capacitor in a console to power a massive, power hungry chip, just to keep time, instead of moving timekeeping to another chip and using a coin battery.
@@thesecretlifeofdukelee and you promise yourself you'll reuse the code when you do something similar but you end up writing it from scratch next time anyway
@@KosteonLink I might done that for several projects several times. I'm doing it right now. Losing time rewriting code instead of thinking about new features.
Another major problem with brute forcing a the reverse of a 32 bit checksum with more than 32 bits of variable data is that at least some (all, if good algorithm) checksums will have multiple solutions due to the pigeon hole principle.
Wow, didn't know you watched Matt.
is this deadass notch
@@InitialAA He's great! I'm a fairly recent though fan, it took a while for the algorithm to figure out I'd like his stuff.
@@_notch Same. I recently came across and watched his Mario 64 video and some of the cheap Japanese console videos among others, which made me get into his content. Good stuff!
i did not expect you to be the actual notch lmao
Even if the bruteforcing could complete in our lifetime, the fact that the number of possible combinations is bigger than the size of the checksum itself (4 bytes = ~4 billion) means that there are most likely an overwhelming number of possible combinations for those 11 bytes that would produce the same checksum, without any way of determining which combination was the original one.
Funny thing though is that even though the data would be wrong, it would still *work.*
Well it would get past the 1st error message at least.
well, I mean that's just how checksums work. If you have 4*8 holes to stuff pigeons in that's 2^32 combinations, but you have several times pigeons than you have holes so some pigeons are going to need to split the rent. Checksums, hashes, etc. will *_always_* map several inputs to identical outputs, the important bit is that they data that *_does_* map to identical outputs is so far apart that it will fatally fail and won't be mistaken for real data. Even if every single other byte in the file is completely ruled out (which it can be) you still have 11 bytes of variance to map onto a 4 byte sum. The only way to have no repetition is to have a checksum/hash that's as large as the original file, at which point you can just send the file again.
@@robonator2945 a java String's hash code is the same for "String" and "StringAA" for any String. oops!
Fun fact: I also lost all of my progress in Test Drive Unlimited once because I accidentally turned it off during one of the autosaves (it autosaves all the damn time). I had to start fresh and decided to turn off the setting just to be safe. Also, this game has full 16:9 widescreen support and it makes the game look much better. The only way to activate the setting is by actually setting the Playstation 2 itself to 16:9 in the main menu.
Does this extend to the 360 version? Love this game
@@Timic83tc The autosaving? I'm gonna be honest I don't remember. I think the Xbox 360 version also always autosaves, but it's much quicker and therfore you're much less likely to accidentally power the console off during an autosave
@@mazda9624 the game is good but I think the handling/drovimg controlls is too strict on 360, this version seems easier (game was a tough SoB)
“Dad? That old computer beeped!”
“Oh, So it is finished.”
“What’s this?”
“It’s a checksum… that my ancestor wanted.”
“What checksum, dad?”
“Test drive unlimited”
edit:
".. what's Test drive unlimited?"
".. i don't know"
lol
😂😂😂
I see people are living past the end of the known universe.
“What’s that?”
“A game.”
“What’s a game dad?”
“…”
"What's your favouirte episode, Billy?"
"I like the one where MattKC explains checksums for a PS2 game from 2006 for a looong time!"
"You're goddamn right you do."
You absolutely could've just replayed the game to get back to where you were before the save corrupted, but I believe there's much more value in this issue being motivating enough to drive you to make an entire video on it, giving your viewers extra entertainment and education that we would've missed out on had you given up or restarted. Thanks for that!
not to mention you could of beat the game and lost everything
Counter offer: Play the game all the way back to where you were then fill in the original missing bytes with the bytes from the newer save file, restoring the original and then completing the game with *that* one.
@@Blik10but then you might as well just play the game from the new save
@@noaharkadedelgadoYeah but at least you can still educate the people on your youtube channel in the process.
0:58 For future reference, you should try opening and closing the disc tray. I haven't had to try during a save, but it's unfrozen my games in the past without repercussions
@@MuchWhittering That would be a little disappointing in my opinion. The fact that this could work on original hardware is what makes this so interesting to me
@@MuchWhittering 5:29 "I switched over to the emulator..."
@@MuchWhittering He literally just made a video before this one showing how to ruin your PS2 with a bunch of random exploits, and said _in this video_ that he transferred the save over a USB drive...
@@tomysshadow ruin? dont you mean make it better by letting you run homebrew and more?
gran turismo 4 be like
Woah. It's like magic. Fun fact: when my PS2 broke, I had created a backup of save files just like you did, and when I tried to load the latest one in PCSX2, it said 'Data Corrupt' (Prince of Persia: Warrior Within), with a new logo (usually it would be the place and time where I'm at) Thankfully, I had backup saves.
And I was at like 93% completion.
@@gamechep oof, but you had backup saves.
I do the same with my DS Flashcard and 3DS SD cards.
This is mainly because one time I was putting in some files when the USB disconnected and it got corrupted.
Nowadays, I always create a backup before doing anything to the SD, so if it disconnects while the PC is doing something, nothing bad will happen.
@@pacomatic9833 Try using FTPD if the files are not that big. I use it because otherwise I'd have to unscrew the back panel, one of whose screws is already broken.
@@gamechep I miss the SD card slot on the slide, I know they did it so people don't accidentally fuck something up but it made adding music and stuff SO much easier. Now I need to find my screwdrivers and microSD adapter.
This was fantastic! As a fan translator for many a game and of course many game file formats, it's always a treat to watch others doing similar reverse engineering work. Awesome~
Shout outs to you fan translators, you all do a fantastic service.
Absolute madlad. I've tampered with savefiles, editing hex values and tried to gerenerate checksums myself, but to the point of putting the game inside a simulator to edit the assembly code, truly madlad
Number Is [309,485,009,821,345,068,724,781,056 or 309 Septilion, 485 Sextillion, 9 Quintillion, 821 Quadrillion, 345 Trillion, 68 Billion, 724 Million, 781 Thousand 56] Number Location: 10:04
[For How Many Years It Would Take For All Approximate Combinations are only 134,430.635921 Times Longer Than The Age Of The Universe by using the diving factor of the universe's age being 13.8 Billion Years and dividing it by 1,855,142,775,714,214 or 1 Quadrillion, 855 Trillion, 142 Billion, 775 Million, 714 Thousand 214 Years To Get The Answer.] Number Location for 1,855,142,775,714,214 or 1 Quadrillion, 855 Trillion, 142 Billion, 775 Million, 714 Thousand 214: 10:42
This man is becoming too powerful for his own good, and I'm loving watching it. I doubt I'll ever be *this* good at software, but it's still damn interesting. I'd love to know a bit about your background, where you learned everything you know and what got you into all of this.
You’ll be your own version of good at software. That will help and inspire many.
I greatly enjoy these stories of discovery and deduction with a gaming setting. I would honestly be thrilled to have an entire series on the bit level workings of various file formats... I've partially reverse engineered unity's gameobject-component format (with some very major and necessary bits missing). But the sheer complexity of some formats would likely make for very engaging content (at least for people like me). I'd definitely want it from someone like you, as well. The way you talk definitely makes these feel like proper stories and I really wish I could find more content like it. 8-Bit Guy sometimes puts out similar stuff, but this sort of thing is probably a little too far from his niche to be more than a blue moon sort of event. If any comment readers have suggestions for channels, I would love to have them.
Check out "Retro Game Mechanics Explained". It's not precisely file formats but should be in the vein of what you're looking for. "MissingNo.'s Glitchy Appearance Explained" and it's kind-of-sequel "Pokémon Sprite Decompression Explained" were in particular really interesting to me.
@@nicolasmerz7765 I was about to suggest that UA-camr as well, he's great!
Not exactly what you're asking for, but I've really enjoyed Kaze Emanuar talk about optimizing the hell out of Mario 64 on real hardware, and suckerpinch talking about...lots of things, including "reverse emulating the NES."
Been trying to revive my Monster Hunter Freedom Unite save file, so this was pleasantly relevant to me (I have a backup at the 500 hour mark, so it's not a huge loss, but I figured I'd try to rescue my current save anyway).
Lol I had a MHFU save file corrupt after I dropped my psp while it was saving. The battery flew out and when I turned it back on the file was toast. I had literally just gotten to g rank too.
Was the FISH tag generated by the encryption function or some generic save routine? If it was the output of the encryption, then perhaps it's an indicator of the save using Blowfish for encryption?
Just a file format indicator. Every single (sensibly implemented) file format uses one. It's used mostly to check if it's reading the right file/it's in the right format.
I had the same thought exactly since i remember hearing that TDU2 uses blowfish for encryption on every packet.
Everything MattKC has made so far has been interesting for me. This video reminds me of the time I worked as an embedded systems developer, looking through memory dumps, trying to figure out what went wrong.
I'd like to request for a video on how to debug using modern tools. I want to understand the process you used to find the checksum code in the disassembler.
idea for if your making an autosaving game: keep the original save file in the games memory till the game realises that the save is finished. Trying to load the save file and seeing it is corrupted, delete the new broken autosave and load the old one, although you might have to make more regular autosaves or you could lose a lot of time.
You don't necessarily need to load the save file to check if it's ok before continuing, since that is pretty unoptimized. Rather, include some sort of exit code for the save function. Some pseudocode:
save file.temp
exit ok
if exit = ok, then move file.temp to file.sav
if the game saves often enough, players likely won't lose more than a few minutes of progress. also you could add a bit in the load function to try to load file.temp if file.sav is corrupt in the event something goes wrong during the move (although moving files requires very little time and is handled by the filesystem so it's vanishingly unlikely anything will)
“Keep the original save file in memory until the game realizes the save is finished”
This wouldn’t work for the example in this video, since the game crashed and froze mid save (it’d never get to the “realized the save is finished” part, and at that point it’d have already partially overwritten the save data on the memory card).
The only way to account for the issue in the video would be to have two save files on the persistent storage device (memory card).
this is called copy on write
@@patientallison There's nothing inefficient about loading a save file to ensure it was written successfully. File IO isn't really a costly operation, and loading a save file doesn't necessarily mean also restoring the state that the save file describes. That would be the inefficient part.
@@skyleite2065 actually, PS2 memory cards are extremely slow.
COMPLETION : 76% Cr. 5,322,348
Only one profile can be loaded per game session. You will
(X) YES | NO
So that's what it was. The missing bits were the end of that string that would be displayed upon loading.
Haha
@@AbjectPermanence that is honestly the luckiest thing to be corrupted, because after the autosave, literally nothing was lost.
yoU WiLl
@@AbjectPermanence Why on earth are they part of the save data and not just loaded from the game data?
The same corruption thing happened to my gran turismo 4 save after completing all of the license tests. Needless to say, I was quite mad at having to do those 2 hours of boring tests again.
Happened to me as well I was pissed cause I did it with a wheel which makes those tests feel longer.
This happened to my brother and me in Gran Turismo 3, we had to restart the game like 3 times
This happened to me an hour ago, but with a 88.2% completion file, just after driving the WHOLE Nürburgring 24h endurance race. I lost over 100 cars and a whole year of playing this game, even the Formula GT car I had just earned by winning this event.
@@ynz9214 same
As a victim of multiple corrupt save files myself on a few games, this was extremely satisfying to watch
As someone who's been staring at a hex editor for the past 3 months smashing my head against the wall to reverse-engineer a file format, this was a very fun video to watch. Thank you.
seems more likely those nine missing bytes of "save game data" were really also related to the signing key. The game must have crashed trying to sign the file. Presumably a bug in the signing algorithm. This makes much more sense than it crashing trying to write the last 9 bytes out of nearly 100k of actual game play data. That's a very low probability...
The probability of it crashing while writing the last 9 bytes out of a 100k file is the same as any other set of bytes
@@skyleite2065 if you assume that there's an equal chance of it failing each time you write a byte then the vast majority of failures will be before the last 9 bytes. That's my model of what's going on. What's yours?
@@xeostube I don't follow. If there's an equal chance of it failing each time you write a byte, why would the majority of failures be before the last 9 bytes? Those two statements sound contradictory to me
@@skyleite2065I'm presuming it stops once it fails a single time. So it had to succeed each of those ~100k times, but then fail at the end. Here's a simulation in matlab of the basic idea, with a failure rate of 1% per write.
a=[];
for (i=1:10000)
x=find(rand(1,100)>.99, 1);
if (x)
a(end+1)=x(1);
end
end
sum(a==100)/length(a)*100
the last line calculates the probability of a failure on the final 1% of the data written, which is 0.46% the time I ran it. Or in frequentist terms, out of 10,000 simulations, failure occurred writing the last 1% of data only 29 times; 6267 times it occurred at an earlier point.
Really interesting video. I remember when I was a kid in the PS2 era, coming across a few corrupted save files on my memory cards. They are still there to this day...
What years? I have one from back as far as 2006, and unfortunately I no longer remember what game it was xD
Just wanted to say thanks for documenting the process of how you went about debugging your corrupted save file. I recently wanted to play the game "Stuntman" on PCSX2, but due to a very well known arithmetic-inaccuracy emulation limitation the player is always stuck at a particular level and is forced to use in-game cheats to unlock all levels. Instead of cheating entirely, I tried modifying the save game, but was met with errors that hinted that the memory card file was 'corrupt'. After you pointed out how checksum are integrated into saved game data, I did note that the save game file had 4 hex values which would consistently change with every save, indicating that the values were used as a checksum. Using this information I utilised the debugging tools that PCSX2 has to find in memory where the checksum was written and read from, and found the specific branching condition executed which ultimately validated the checksum (with the valid checksum value also visible in the debugger). Bypassing the check by editing register values within the EE CPU let me load the save. Initial effort when editing save game data was very little and I wasn't as invested in getting something going, but after watching your video I had a bit more patience and motivation to try to at least understand what might be happening and adapt it accordingly to a slightly different problem in the same domain, as well as learn a thing or two and feel a bit more comfortable using the debugging tools... So yeah, thank you.
Test Drive Unlimited is a really interesting game on PS2 and PSP, given that it has a completely different (and I would say much better) handling model than the 360 and PC versions, plus the addition of clubs. Really fun game, would recommend to any person who likes cars or racing games.
They are both good games and very different. I just wish the PC port was better
and "TDU: Solar Crown" is due to come out in September, about 11 years after TDU2 was released, and 16 years after TDU1's initial release in 2006. So excited!
Man, I remember when I got my first PS2 save corruption. I was playing the PS2 version of Sonic Unleashed, and at one point (don't remember when or if I shut the console off during a save) the game kept saying that it couldn't load data because of a corruption on the memory card, and was asking me to format the memory card to solve the error. This was FAR before I became as tech savvy as I am today, so I had no idea what "formatting" meant. I just agreed to format the memory card thinking no harm would come of it. I had complete/near-complete save data for games like Kingdom Hearts, Ratchet and Clank, Jak and Daxter, Sly Cooper, the Spongebob PS2 games, and even more I can't easily name, on top of being in the process of completing Sonic Unleashed, so you can PROBABLY imagine how much I cried when I later found my memory card empty of months/years of game progression.
11:24 It shouldn't actually take that long.
1. The checksum only has 4 billion possible values not septillions. On average you will only need to calculate 2 billion. That changes the time to 5 days.
2. Even single threaded you can get WAY more than 5k checksums per second. I would expect hundreds of thousands of hashes per seconds. You can also optimize it by saving partial computations. So to incremented the last byte you only need to call checksum once since you already have the checksum of all of the previous bytes. It's also trivial to make this multithreaded by partitioning based off the first byte. With all of those optimizations you should see millions of checksums per second. It would then take only an average time of under a minute. Another potential optimization would be trying to solve for the last byte which would give you like a 100x speedup bringing you down to below a second to solve.
In summary the time it takes to finish will be measured in seconds and not billions of years.
As a researcher, I know how that feel when you try to understand, and come up with a solution to a problem. Of course, it feel even greater if you can solve it in the end, but the whole problem solving process is already fun enough! I have subscribed to your channel, hope to see more of your thought process during problem solving!
Fun fact, Matt: That checksum algorithm used in the game was supposed to be the famous, _ubiquitous_ CRC-32. Just implemented kind of backwards, so the bits were reversed, and possibly using a different polynomial. And of course, messed up in such a way that ignores 75% of the information you give it (Like me...)
i loved this video! it's been great seeing you upload so much while still retaining quality. the solution at the end was so cool, with the reveal that the save essentially wasn't corrupt, and was able to run when brute forced through.
Loving the constant upload stream!! Any Matt is good Matt.
These videos are really nostalgic for me, I remember tinkering with my consoles back when I was a kid (homebrewing the Wii, hardmodding the OG Xbox, etc.) and I hope you never stop this content. Truly great stuff.
I've had to do the exact same thing on another game. Unfortunately the level of corruption was a lot more substantial for me and I wasn't able to rescue it. It was a fun attempt though. These sorts of diversions almost become part of the games for me.
As someone who tries tinkering with things in hex editors from time to time, I have a real appreciation for this. Nicely done
This is why i make two saves. i was always scared this would happen xD love your videos!
but....it's a profile that auto-saves. It's not a JRPG. But yeah he probably could've made a second save manually periodically
Hell yeah dude, nice one. I had a similar issue with my Ocarina of Time save a number of years back and ended up deep down a rabbit hole with a GameShark, a parallel cable, an old Pentium II PC and some very janky software before I finally emerged with my childhood save intact. I feel like I went on a similar voyage of discovery regarding checksums etc. as you described here.
I remember as a kid messing with the original XWing game on PC to modify the save files. I used a hex editor to compare files and change stats to unlock missions and trophies or whatever they were called.. No checksums, no encryption. I then wrote a program that would read a file, ask you a bunch of questions about what you wanted to modify, and spit out a modified save file. (The initial version just modified every byte I found to give you full stats). Soon after I found a much more competently written Xwing Editor, but man that was fun.These days kids would have to learn about assembly, emulation, and cryptography to get anywhere with most games, so it's a much higher bar for entry.
10:52 OVER 1.8 QUADRILLION YEARS!!?!?!?!?
I once (with help) fixed my corrupt Pokemon Diamond savegame! I tried tweaking, and accidentaly saved in the wrong spot (in the void). Couldn't load up my game at all anymore, blackscreen when I did. Since my New 2DS XL is jailbroken, I dumped the savegame and fixed it with a hex-editor and certain savegame-editors for Pokemon. When confirming it worked with an emulator, I wrote the savegame to the actual cartridge. Fun experience.
This is relatable! It's always more fun to jerry-rig something that shouldn't work than to start over.
So basically there’s no hope for normal people to recover corrupted saves.
I bought a ps2 because of matt a few weeks back. I never know how cool they were until I saw your videos on them, Thanks!
Disclaimer: I am not a programmer and do not know the technical details of how checksums are implemented.
That said, something about needing to run through all possibilities for 11 bytes to match a 4-byte checksum seems off to me. Because you have more missing bytes of data than there are in the checksum, there's going to be lots of possibilities for those missing bytes that yield the same checksum. This seems like it would make it impossible to find the exact missing values but much easier to find *some* set of missing bytes that agree with the checksum. (Couldn't you get away with keeping 7 bytes 0 and only messing with the other 4? Statistically speaking, it seems likely that you'd be able match a 4-byte checksum by altering 4 bytes of data to all possible values.)
Yeah you're right about that
that WOULD be correct if your only objective was passing the checksum. however, the goal here is to recover the real save file using the checksum as a way to help, meaning we still need to search through all possible options to make sure we find the right one. also, brute forcing just the checksum, while not as long as the lifetime of the universe, still takes roughly 250 days at the rate Matt's program works, meaning its still faster to just play the game over again
What you're talking about is called a collision. That's when you can find two sets of input values that will create the same exact checksum value. The thing about collisions is that they can also take a long time to find. (This is especially true with hashes, which are similar to checksums.)
Think of it this way -- in Matt's case, there is a "correct" input sequence of 11 bytes he is looking for that will create the 4 byte checksum he has. And yes, there are very likely other sequences of 11 bytes that will create the same 4 byte checksum he has (i.e. collisions). However, there's no way of Matt knowing how many collisions with his 4 byte checksum there will be. There could be trillions of collisions, or in theory, even zero collisions.
Yes, intuition may say that if he only changed four of the 11 bytes, he could find a combination of just those four bytes that would create the same 4 byte checksum. However, that's not guaranteed at all; even in this case, there will be a lot of collisions where several input values will create the same 4 byte checksum (not necessarily the checksum he's looking for), meaning there will also be a lot of possible 4 byte checksum values that will never be seen, if that makes sense. And his could be one of them.
A related thing you might find interesting is the Perfect Hash function. To be brief, it's a hash algorithm that creates a unique hash / checksum for every input value you give it. However, in order to make a perfect hash function, the final hash / checksum value MUST have more bits than the values you're hashing (44% more at the very least, but in practice 60%+ more, and even then it's hard to do).
@@devmas There can't be 0 collisions if the size of your data is larger than the size of the checksum. It's mathematically impossible. This is why hashing algorithms are so difficult, because they have to take an infinite number of inputs and produce a finite number of outputs while making it near impossible to find collisions, even though collisions must exist.
@@SmashhoofTheOriginal Sorry, I should've made it more clear that I meant zero collisions with that one specific checksum he's looking for. Of course it's mathematically impossible to have zero total collisions with a smaller checksum than data.
This reminds me of how I cracked some saves of DS games, those actually had about 4 CRC32 checksums to check you didn't tamper with it and the latest games of that series also added encryption and something else that seems like a homemade compression algorithm(which I haven't figured out yet unfortunately), had a lot of fun figuring all of this out and making a save editor with this info
So it was clearly running the signing alrorithm, crashed, and failed to write anymore. Nice stuff! This was actually a fun walkthrough.
Matt's gonna bring Stephen Hawking back to life, hackintosh him, play halo on him then set him to work decompiling Lego island.
this was the first ever mattkc video the youtube algorithm recommended to me like more than a year ago at this point. i like to revisit this one every once in a while to appreciate it.
This gave me hope to fix my Ape Escape 1 pal being stuck at 99.9% completion (yes I have absolutely everything). But 99.9% of this video went right over my head.
I'm so happy and surprised to see you talk about TDU1 on ps2, such an obscure and underrated game. It's a childhood classic for me as well. Props for using its fantastic OST in the video !
You've outdone yourself, this is one of your coolest videos.
Also you made me want to play this version of the game. I played it some on the 360 and PSP but...eh, seems like it might be fun.
The ps2 version is very good, you should play it
Reminds me very much of recovering databases from a faulted state. Very cool! Nice work!
MAN this was great. You're a phenomenal technology detective.
Nice to see TDU getting some love, grew up playing it on an AM2 Athlon X2 PC with an old Radeon RX600 ProGuru. Tremendous game that was so ahead of its time and so impressive in terms of variety, car selection, Terrain, Sheer map size and detail. It's a shame the sequel was such a departure leaning more towards the then popular Casino elements, but the 1st game was amazing and I can highly recommend it
03:13 - in the top it says "FISH*". That asterisk could be interpreted as a star. A Fish-star. A star-fish!
waos!
something about watching you zero in on the problem and fix it was absolutely enthralling. great video!
you should make this mod public as a cheat tool - it makes all saves loadable and instantly makes them loadable in a normal copy
I always knew there were people capable of solving these sorts of problems with software but you're the first one who demonstrated it to me.
This was so cool! I'm currently working on a project for school that involves similar crypto and byte examination, so seeing this come out was really neat and I actually understood a lot of it.
Awesome job! Seeing your process was very informative (even as a software developer) and it's a good reminder to keep stay aware of tunnelvision while working towards a goal.
Patching the game was a good move and I am happy that you managed to play the game to completion!
What a coincidence! I got done with doing this game 100% about 2 weeks ago! Also a childhood favorite that I ripped straight from my DVD
This reminds me of when I was running a minecraft server, I tried to switch to an old map and instead somehow ended up combining them
I wanna see him take a normal save file and corrupt it.
Run it though the Vinesauce rom corruptor
Gotta admit, I was stunned when you got that save file working again. I'm very happy for you. You're really talented. :)
I was expecting a burst from an MG42 at 10:20.
Man, TDU was so good. I was just getting into sim racing; got my first force feedback wheel. Vividly remember going for an hour long drive for the sake of it in my teenage years, easily one of my first truly immersive experiences on my 23 inch Hitachi CRT monitor.
While it wasn't a sim, that was vastly outweighed imo by granting one of the first true 'going for a drive' experiences; I almost enjoyed the more normal cars to the exotic ones for that.
9:40 Hold up;
you're telling me that I've been making passwords that look like hashcode when I could've just used normal-ass words and I would've been *MORE* secure?!
The security of a password depends _both_ on how jumbly it is ("hashcode-like") and how long it is. The comic's author is advocating for taking advantage of the latter.
What a journey through the depths of this game's code! It was fascinating to watch you figure out and reverse-engineer all of these security measures. If you have the opportunity to do so again, I'd love to watch you do the same thing another time! :)
16:00 I count 10 "bytes", not 9.
Nice job! I don’t understand any of the technical aspects of this, but your solution and tenacity at figuring this out is fascinating! So cool and so are all of your videos! Thanks! 😀
I can't get over how, from a technical standpoint, fortunate you were that it was just a signature creation corruption.
If *any* of the rest of the save was corrupted/nullified, you probably wouldn't have had much of a chance of getting it back like you did.
For instance, with a Bad Egg Pokémon, the corruption of the Pokémon's data varies. Sometimes the data can be so corrupt that there is no way whatsoever of restoring your Pokémon.
This is an incredible vid: extremely interesting and insightful, yet also wonderfully relaxing. Every time it’s thumbnail pops up at night I feel nostalgic
You're telling me there's hope for my 15 year old 100% completed burnout revenge save?
17:05 just as well those cops never knew you'd been tampering with save security features.
You took a broken save to a fixed one, you took my time i shouldve spent studying into 18:09 minutes of youtube
Im not into programming at all but this theme of corrupted saves are definitely one of the most relatable and sad subjects to me
11:02 Worth it
I find data manipulation really neat and was blown away. Thank you for this!
4:37 “MD5 is insecure fo-“
The main reason why I always do backup saves of all my games once or twice a month on a external drive.
Especially when some of the games have more than 150h of progress lol
I'd love to see one of the original devs react to this
Please, do more of that Hex-code researching. I know it might be tiring, but it’s very interesting. Besides that, very well made video and I never get tired of your channel!
Video game save files are analogous to configuration files like autoexec.bat and config.sys, except they are normally binary. All save files have a specification, and you can figure things out by looking at the file, making educated guesses, changing bytes, and trying out your edits with the game. I haven't looked into it, but I bet that specs for save game files for common games are published online. I haven't written many games, but I've worked on many programs with configuration files.
Dang i have zero interest in programming because i don't like it, but you make it entertaining, the pacing, the way you tell the story... So cool m8
Awesome video! Been super interested in a lot of your more recent videos, though I have never been too good with hardware so that has always been more of a thing I like watching but not doing. But software has been more of my strong suite and where I've found more of my interest lies and so its cool to see you doing something more like that, even with hex editing and stuff.
I'm a big fan of your work but you should try Unity As satisfying as hacking is you might be able to make a game.
@@ComicusFreemanius sorry for the late reply, youtube didn't notify me of this and I only just noticed when going through older comments I left. First off thank you! I really appreciate you saying that. Secondly I have messed around with unity before, I had been working on a small side game project in it, that skatepark tycoon game, just haven't had the time to be able to put into it, been trying to find a job which has been stressful and still trying to get the last big update for my thpm mod done too so its been a lot, but I do hope to get back to it someday.
been a programmer for 30+ years and this episode resonated. very nice job and the absolute BEST motivation. 🤓
6:48 LOL
This is so interesting to me as I’m trying to research how to do something similar with a Nintendo switch save. There’s a lot less information to work with and a lot more security to get around, but the rude customer service agents and the lazy devs are what’s motivating me.
0:40 You forgot to add the screen with "Piracy is not a game!". 🤣
By the way, just meant as a joke, not offensive!
i love your videos! You have just the right amount of technical detail and you make me feel like I'm there with you doing the project. Keep it up! If you ever end up in Italy, beer's on me
10:35 would not have imagined seeing Slovenian in one of MattKC's videos.
What were you translating, need any help? :^)
I found this channel last night and I'm amazed at the quality of your content, as a game/console modding novice who's been at it since childhood watching these types of videos is as relaxing as it can get, keep up the good work! 💙
So, let me get this straight: the game decided to be a real meanie and steal your save. So you literally hacked it back.
Sweet!
Yeah, that's basically what he did!