BINARY vs TEXT File Serialization
Вставка
- Опубліковано 29 кві 2024
- To try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/TheCherno . You’ll also get 20% off an annual premium subscription.
Hazel ► get.hazelengine.com
Patreon ► / thecherno
Instagram ► / thecherno
Twitter ► / thecherno
Discord ► / discord
HxD hex editor ► mh-nexus.de/en/hxd/
Hazel ► hazelengine.com
🕹️ Play Dichotomy for FREE (made in Hazel!) ► studiocherno.itch.io/dichotomy
🌏 Need web hosting? ► hostinger.com/cherno
💰 Links to stuff I use:
⌨ Keyboard ► geni.us/T2J7
🐭 Mouse ► geni.us/BuY7
💻 Monitors ► geni.us/wZFSwSK
This video is sponsored by Brilliant.
Hope you enjoyed! ❤️
Don’t forget you can try everything Brilliant has to offer-free-for a full 30 days, visit brilliant.org/TheCherno . You’ll also get 20% off an annual premium subscription.
Sir , continue the C++ playlist if not then tell talented teacher for teaching in more advance and has been better as you.
Text format design is very human
'the design is very human'
😪
at the cost of speed
Ah, that brings back the nostalgia of opening an .ini file of an old game for the first time and changing things just to see that it actually changes what the game does. It broke things more often than not, but that was my first interaction with the concept of variables.
The best solution is USE BOTH.
On my current project(s), I use only binary because they are small projects.
If I had more time, the level files for example would be saved as xml or json. That way merging is easier in git, and if I break loading/saving code and save a broken file, I can edit the broken data manually. Then I'd write an exporter to export those text/json/xml into binary when I export the project data. Only the problem now becomes that you need an additional QA pass to check that binary data also works.
Although, player options are good to export as xml, so users can change things. Which is what I used on my previous project.
And if you're going to use binary, as ThePrimeagen said recently on his stream: ALWAYS start with a version at the start of that binary format! I'm using this in the current project and I can modify the saving/loading code for level data and still be able to load an older level to save it in the new format.
This is what I was thinking. (Haven't finished the video yet, but am close)
Serialisation is a topic I've wanted to see a video on for so long, I got so so excited when I saw this video was released!
You have a whole playlist on his channel about serialization
One thing to remember is that when using "plain" text that the encoding and file formats still matter quite a bit too. For instance, whether you use ASCII or UTF-8 can make a huge difference, and while YAML is an okay choice, TOML would be better because whitespace really shouldn't be significant in a user edited format.
I was just looking for this. U really read my mind.
Thank you so much
Can you do json vs yaml? Which is better for storing data, readability, commenting, which is better to use as config file? Are there better formats ?
in most cases json and yaml are pretty much exchangable. Technically yaml is a superset of json, as in it has more data types supported out of the box, and supports things like references within the document, but overall serializing/deserializing it will, in most cases, be more or less the same. In the end both are key-value pair, human-readable documents.
Readability is personal preference. I saw people preferring yaml, but personally I vastly prefer json format, but tbh.. But that's very personal opinion.
@@DemoBytom For some reason I feel that YAML must be easier to parse, given that it doesn't exactly use braces like JSON.
@Brahvim nah, for a serialer it would be easier to parse json than yaml. Json doesn't care much for indentation, nor white characters, while if you screw indentation in yaml the whole document is invalid.
Granted - both are pretty much non issue anyway, because both json and yaml can be validated by tools to be correct, and I in prolly 99% cases nobody writes their own serializers/deserializers for them, since better ones already exist in any language/framework already.
@@DemoBytom Just use TOML, whitespace isn't significant since it's basically just an extension of old INI files, but it adds some extra functionality.
Use TOML and stay away from YAML (Curse you flutter) and use JSON if you don't want TOML.
I've only ever done binary serialization in C/C++, in Go it is super easy to serialize to text (JSON) so I use that in most of my Go projects. Interesting video!
Great Video! Can't wait to see the follow up. Keep up the good work!
13:27 Absolutely! That's the kind of stuff I work with atm with modding, and optimisation is always something I'm seeking to learn
This and memory management is one of my favorite topics in programming. Thanks for the video!
Nice comparison, I would love to see the code as well in a future video!
Another reason for using text format is if your software runs on multiple hardware configurations/operating systems.
In web development it's common practice to use json serialization for almost everything.
My best programming and my favorite teacher in my whole Life.Thanks to you Sir very Thanks....
Can’t wait for the next video. Let’s get it!
For me one more good reason for text is future changes in the data format. If you add/remove some variables in your format text parsers will easily handle old versions by using defaults/ignoring some keys. But binary ones will need added logic for each change and there needs to be version of the format saved in the data. Of course you should always have format version in all data even the text ones.
In the game I'm currently working on we store quests as json in the editor, but use MemoryPack for the runtime
Im working on a game engine and am currently using binary serializing, similar to how you would if you were pushing/reading a stack (ie packet deserializing) and integrating a reverse dump so the runtime can be the editor
while it still comes down to whether the format is intended to be read by a human or not, another reason/side effect for using text formats in the shipped build is modding
on i can still remember is in GTA San Andreas the data for vehicle specifications was in a text file (csv iirc) and allowed for some fun things like insanely fast tank
while determined users will still find a way to parse and + attempt to edit anything the text serialization does make it easier which depending on whether the game devs want to encourage/discourage may be another factor one way or the other
Nice vid! Always has thoughts on this topic - Unity serializes in YAML text file, UE serializes in binary.
Personally I think the most ideal engine would store assets in text format in editor and during cook process would generate a binary version for that build for smaller size and some "encryption" of data, merging blueprints sucks merging scenes in Unity is somewhat doable from my experience
Haven't needed to do anything with binary for files in a while, largely setup any config type files as text exactly for the reasons described in the video.
However if I had to make a binary file format now I'd probably consider using something like Protocol Buffers, mainly for maintainability.
What notes/whiteboard software did you on this video?
If you need serialization there is nothing better then chunks with headers that you just memcpy + relative pointers
you bringing up that it's a hexadecimal representation of a binary reminds me of the bell curve memes.
(beginner) It's a binary file
(middle) It's hexadecimal
(expert) It's a binary file
Does Hezel support something like Unity's Addresables system? I'm meaning a system that allows content to be packaged separately from code into packages.
ASCII is binary data as well; it is just interpreted/read and/or written differently.
Personally, for hexeditors, I prefer imhex, because it provides a sort of scripting interface to automatically annotate data in the files.
I'd be interested in seeing how a GLTF/FBX/OBJ can be converted to an "engine-friendly" format for faster loading.
i will use OBJ as example because its the one im most familiar with thugh i expect the same logic applies for most other file formats
when you load an OBJ file into an engine the engine is going to have to loops through the data in that file and build up an array of vertices in memory with whatever data you have for each vertex (eg position/normal/UV coords)
one option that would be relatively simple and i think would result in the fastest loading would be to simply dump the resulting block of memory out to a binary file, then rather than needing to perform any processing at all you could just load the file into memory and point a variable at it
for a setup where the originals are in text based format and only the shipped version is binary that feels like it would be pretty close if not an exact explanation to what is done
1:35 no there is no link in the discription
Use binary if you wana store some data (which is not needed to be used by hummans) it can be effecint at reading time, storage etc
While use some Text if it's humman designed,it will be slower but readable
You can use JSON, or custom fornat or any other
Theres tons of them
Not packing the assets in the development phase is definitely understandable, however I believe keeping all data in binary on the disk is fine. Instead of keeping the files in text and parsing them on runtime, you can also parse the binary in the Game Engine editor and show the developer a text version or any kind of human-readable UI which gets automatically converted and saved back to binaries on disk. The big downside here is that extra work needs to be put in to create a nice editor, but it may be a nice middle ground for some scenarios.
Binary files contain data of a fixed length, and text files are also binary, with a lot of line breaks and a lot of human readable numerical values, which must be converted back from ASCII to numerical values that can simply be placed in memory.
could you serialize to text during dev, then automatically serialize to binary during production, so during dev you can easily edit the save file, but in production it is just stored as a .dat
I feel like the merging and version control point is the most important part of this. Git, SVN, Plastic... Whatever you use, it doesn't like binary files and will likely not be able to merge changes of two contributors into the same file in your repository.
If you're working in a team, trust me, this becomes a huge nightmare.
Amazing Contant , always Unique.
Finally, we are getting there
do the shipped files get enrcypted or otherwise protected against "malicious" changes/extracting game assets besides being in binary
fun fact your user needs to have decrypted files to use them so it literally doesnt matter in any way
a waste of time both in terms of development time trying to implement it and loading time having to decrypt at least for the stated goal
if you encrypt the assets, then you would also have to provide the user with the key to decrypt them so they could play it
not with games but i have seen people try to do this exact thing with software and its a complete waste of time in almost every scenario except for malware avoiding detection
@@custard131 Most games that don't have file encryption are hacked, attributes are modified and can generate a lot of strange behavior on the client or server itself, assuming that the client/server will trust the client's information "a little".
e.g.: A weapon has an ID and the client sends that it is using that ID, but the weapon's ID has been modified on disk and now the server "thinks" that the client is using another weapon, that's it, you've exploited a problem...
Many games encrypt and decrypt files in parallel, others prefer to hash the file and send it to the server to confirm that the file hasn't been modified. If you don't do either of these things, your game will be destroyed by cheaters.
If it's an offline game, no problem, but an online game can't do without this.
@@qqshutup7175 that is a problem that needs to be solved for online multiplayer games, but encrypting of assets doesnt solve that
in such a setup the server needs to keep its own copy of the attributes and use those rather than relying on ones submitted from the client
there is essentially nothing you can do to prevent someone modifying how thing appear in thier client, but in realising/accepting that there are ways to design it in such a way that it doesnt impact other users
signed data packets can be useful in some cases but i see that as a completely separate topic to encyption
the simple fact here is that in order for a user to be able to play a game where the assets are encrypted, they are going to need to be able to decrpyt them
@@qqshutup7175 if you're making a multiplayer game then you simply cant trust the data sent from a client. There is no encryption trick to stop hacking, you just need to receive the least amount of data from the user as possible, send them the least amount of data about other players as possible and analyze their data over time to observe impossibilities in the player state.
It would be really cool to see the implementation differences between text and binary serialization. Do you use a lot of bitwise operations with binary serialization? Is bitwise logic actually faster than standard logic in C++? I have wondered about these things for a while. Lol. Thank you for your videos!!!!
It depends on the problem you're trying to solve. When you're dealing with communication (protocols) where bitwise operations are very much in use and you need to splice your number into bits, then bitwise operations will be a faster and better choice by far. It's just a tool that could be useful, and there is no separation between bitwise logic and standard logic of CPP
quite useful
Informative video. I would love to see the code as well :)
A thing to note is text serialization is really not that bad on strings, it's only in numbers where the perf can get a bit worse.
I just realized that serialization is really not that different from code compilation.
Just like when you compile your human readable code into machine code, you compile your human readable text into a binary format.
(Serialize your values into a binary format.)
That’s all it is.
Is it for humans or machines? Your choice will make it easier for one of them.
Make it for humans and machines need to translate.
Make it for machines, and humans have to translate.
it really is diffrent, serialization is supposed to be lossless, while compilcation is lossy. compoilcation is more like deserialization, which is more lossy.
@@urisinger3412 ahhh so your reply was compiled then
@@urisinger3412 Combobulation
@@urisinger3412 what is being lost durring deserialization?
@@tedchirvasiu depndes how you serlize it, for exsample json deserlization losses some information like indentation
This is a good video..what about a video about cpu vs gpu in game programming.
Can gpu execute algorithm?
Are they only used for vector multiplication?
Modern GPUs can mostly run arbitrary code but there are some restrictions:
- no recursion (the maximum size of the call stack must be known at compile time and is very limited)
- no indirect function calls (for the same reason as no recursion)
- no dynamic memory allocation (so you can't use things like lists or hash maps)
- branches are potentially slow (because multiple vector cores get fed from the same instruction stream and can't execute different instructions at the same time)
- sequential code execution is much slower on GPU than CPU so in order to achieve a performance advantage, you must design your algorithms to run heavily parallelized.
There might be some more things I haven't mentioned here but these are the main limitations for GPU algorithms.
Look into compute shaders for executing code and receiving back data from the GPU.
I would love to see a vidio of the serialization code
The idea of content files not being human readable on a released product hits a snag if you factor in Modding, if you want games to be mod friendly a large part if not all of your config style content files (aka not 3d models or texures and so on), should be human readable, but, yes, that increases load time, so why not have the best of both worlds? provide a "development" build, might also include more verbose logging, command console, asserts and content / asset validators, which works with text files as well as the release build that pre-processes those files once into binary and from then on will load faster? remember, any modding support and / or tool provided will also likely benefit designers and content creators on the dev team, so everyone wins
I think always use binary for assets. But have an option to convert to text if human needs it
Yes please make a video on binary SerDes!
I'd love to see how you do good Serialization in C++
Have tried many times, but never got a satisfying and safe result
You didn't link the hex editor!
Thanks, added!
I'd put my cutoff at around 100MB. If you can save the file as a JSON string or other text format in less than 100MB, go for it.
I would like you to discus xml reading writing which are useful for small configuration data store.
Almost everyone in is moving away from xml and adopting JSON instead.
@@christianm4906 Agree, usually on the team, we will never approve the use of XML if we are not forced to.
for text data that you don't want to be text data in the published game, why not just represent it in some entire c++ constructor (or other language) in some file? that way its human readable, but the compiler does the job at serialising it for you
if"other humans come in and change it"... very alien like
wouldn't be great if we could go from text base programming to something even easier for humans like something visual or something we can't figure out yet... I always thought that when I was in college studying this stuff. 😆
You mean nodes? Like the ones they use in shader scripting or visual scripting.
TOML. Use TOML
I don't understand, why(at least in 2024) all the code editor and even notepad decompile or decode idk then compile or encode when you're done editing, so everything stays in binary, it's just test surely it the cost is negligible right?
Sir pls start a series on vulkan with c++
Why did you go with YAML
when I serialize gltf model to binary it loads extremely faster but it turns from 70 mb to 300 mb on desk !!!!
Modding text files is easier. Binaries will need exporters or converters
I do text most of the time. Binary is pain in the ass to extend
Nice Peruvian sweater!
Sometimes libraries force you to choose between these two formats
I use binary serialization in my game engine, similar to how unreal engine does. I prefer binary serialization because generally, it is faster to deserialize binary data.
That's old school and one of the reasons I kinda don't like UE anymore (Even though I regularly worked with it for past 5 years).
1-) You are forced to use editor at all times which is usually not open since you write cpp
2-) Since it's a development environment frequent changes and crashes occurs. This sometimes ends up in binary corruption without any way to recover. If it was a text file, at least I could see what went wrong to prevent it in future. I might even be able to recover
3-) Using text files doesn't meas you can't use binary files. A good approach is storing only metada as text (depends on asset type)
@@paradox8425 well I already made my custom binary asset serialization format LOL. I'd rather focus on the engine itself. Plus, the asset loading (deserialization) in my engine is super fast. And writing 2 different asset formats (text + binary) is too much for me.
Currently, I have a class called Object, and a class called Package (which is subclass of Object). An object can have children objects in a hierarchy format. And Package is the object that is serialized to disk with all the children objects serialized under it. So it's very much like Unreal engine lol.
@@neil_m899 You can do whatever you like. What I'm saying is, reliable and easy to use tooling is much important than most of the people realize. Sure, it might not increase performance of shipped game, but it can make you ship it 2x-5x faster or maybe even more depending on the exact situation. You say "binary assets is fast", but did you benchmark it with the text assets? Exactly. Just like cherno said most difference it can make is 2-3 seconds to load an entire scene which is usually differs much less. And by doing that you gain ability to review chanhes directly from source control and edit without editor open
An important thing to evaluate about performance tho, is how often will the reads happen? If it's just once at startup where you have to wait say 1 second instead of 0.1 seconds, who cares (of course if it's many files, that adds up)? But if it's inside some kind of loop that's executed regularly this difference does matter a lot (best is no file io in loops at all ofc).
High level: Ez
"Low" level languages for humans: too hard
I think this novel introduction of background music is kind of distracting, which in my opinion is not good for the content of this particular channel. Please take that as a constructive suggestion.
Said that, just to share my experience as a retired software developer: storing and loading application memory models in binary is virtually instantaneous but slightly more prone to corruption due to subtle bugs. Also versioning is more difficult to handle.
On the other hand, text based memory models are arguably slower but a lot more robust against bugs (particularly versioning bugs). But please do not use standard json libraries or other library based stuff because these tend to be slow as hell. Just do your own parsing and you will be fine and a lot faster than ANY existing serialising library.
In at least one occasion I chose to use both alternatives simultaneously in a single file bundle. Saves created both formats, but opens only used the binary form. However, the text version was always there as a fallback in case something went wrong as reported by the appropriate data integrity checks. But I mostly developed document based apps, not games, so these files were updated by the users by using the app, adding further risk to their integrity, and the need for robust versioning. Otherwise, I understand that loading heavy textures and 3D models for a game is just too much demanding for text.
During development I'd use text files .. but I'd never ship them .. just to prevent users from screwing around with data used by the game. So for shipping I'd use binary files.
As a gamer and programmer, I've always disliked binary configs for singleplayer, if I want to play the game with a modifued config I should be allowed to, if want to modify my game to give myself ten thousand items/coins, I should be allowed to do that, regardless of the outcome even if it means gaming breaking bugs, I messed with fire and I get burnt.
Multiplayer is a different story.
@@gmfCoding NO, you should NOT be allowed to unless the developer/publisher allows it. If you really need unlimited coins/health whatever - then write your own trainer/crack just like i did back in the good old 90s by analysing runtime code in assembler :P
@@herrdingenz6295 well that's my point then, more publishers should allow it, they certainly don't have to recommend it or at the very least say "if you do this you risk loss or corruption of save data".
This is in argument against your original opinion that: "just to prevent users from screwing around" IMO it's in the right of the player to screw around as long as it doesn't come to the detriment of the people's real health or others personal property. :)
If the developer has other reasons for using binary files such as for performance reasons or development complexity then it may be excused.
@@herrdingenz6295 Thing is, if you don't give some people the option to do that or make it harder that could mean the game is less enjoyable for them and maybe they're just not gonna play it.
E.g. The Sims is one of those games for me where I wouldn't have played it at all it if there weren't cheats because it allowed me to do what I enjoyed doing in that game (building houses).
Would I have played it if I instead had to change a config? Probably, that's easy enough.
Would I have played it if I had to install mods? Most likely not, having to download and install the mod loader and the mod itself (before ever doing anything I like in the game) would be annoying enough for me to just not play it.
Would I have played it if I had to write a mod myself? Definitely not.
Just the way players changing configs have to deal with any bugs or whatever that come with this, devs have to deal with players not enjoying the game or aspects of it, if they limit what players can do. Especially if it's just as simple as making a config file a text file instead of a binary, this freedom might be worth it.
So yeah, whatever both sides decide to do, so be it but it might come at a cost.
👍
My brain ( him not me): IF I HEAR THE WORD THUMBNAIL I'M GONNA BE LIKE 🙎🙆🙆
Pro-tip: Have the best of both worlds by making an editor that can directly visualize and modify the binary format.
2k lines will slow down a couple of nanoseconds, not worth to optimize it at all
i can open binary in notepad, i just can't edit it
me using base64 encoded textures so I can use git without lfs 😎
I code in binary
I code by manually manipulating the ions inside of my CPU using magnets
I code by making the world/earth itself my game and manipulating people
Imagine Serializing into c++ file...
On modern computers I think the difference between binary and text for a 2500 line file should be in the order of milliseconds. We're talking about kilobytes of data.
In the order of milliseconds is incredibly slow for asset loading
@@bits360wastaken Can you observe a 2ms difference?
@@tedchirvasiu Sometimes, depending on when and how it happens in relation to other things, and how often it happens.
@@jamesmcmanus I think the "how often it happens" is the most important thing. If it only happens at initialization it's fine, if it happens during the game regularly it's a problem.
@@jamesmcmanus we are talking about files which would make sense to be human readable / editable: scene files, prefab files, material files. All of these would probably sum up to a few megabytes for a decently large scene.
The big files such as mesh files, textures (images), audio files would never be edited by a human in notepad because nobody would adjust the position of a vertex in a 100k triangle mesh like that. And nobody would care to see how those vertices changed by looking at git diffs.
And when I said milliseconds I was extremely conservative, we're likely talking about microseconds.
I highly doubt parsing files measuring a few hunderds of Kb would amount to anything in comparision to other tasks.
and Unreal Engine still uses stupid binary serialization for all its assets. as for me, the biggest disadvantage of it.
tried to learn unreal using my c++ knowledge but the compile time ruins my whole motivation... Not forget to mention that it crashes when the components are not loaded properly, and has two different method of compiling 💀
But encryption, tho. 🤔
In Blueprints you can use Ctrl+A to select all the nodes, Ctrl+C to copy, which copies it to the clipboard so you can send it to someone else (with Ctrl+V), in text format. I think it works with other node-based assets, too, like materials. But sending the .asset files themselves is a... philosophical decision that Epic made a long time ago to prevent people from stealing in-game assets.
serialisation with a 'z' is a big crime
Oh no, guess I'm going to jail
Me too going to jail 😢
both are correct
programming war crimes, yay
Says who?
please make more videos on how to use a hex editor. please