i think it would be cool if underground there were rocks that aren't indestructible but hard to destroy so you have to dig around them (just as an obstacle)
Cool concept indeed but our current plan is not to discourage digging or building - which is why we're aiming towards making blocks that are placed by players have more HP.
Even tough optimizing isn't bad, just remember that the speedtest global index has the "worst average connection by country" at 30 megabits per second, which by the standards set by this video (1 megabit connection) is 2 seconds. Better be safe than sorry, but i wouldn't worry that much about it
Storing the RGB values for each block is a lot of data that I think you could trim down. R - 256 G -256 B - 256 Results in literally millions of color combinations that you probably don't need. Have you considered instead storing a block type? For example: Grass, Stone, Dirt, etc. Then mapping each of those statically to RGB values? If you want color variation you could even map to a set of possible RGB values. Or upon render add some random +- from the programmed color value. Using a block type instead you would only need 1 byte of data per block, for a max of 256 block types. This also opens the possibility for implementing more complex logic for specific block materials in the future.
I did consider a variation-based system. Minecraft does that to smooth between biomes but that's based on a global noise texture as opposed to data stored per-block. If you solely go with RGB textures, the map does become pretty contrasty and more noisy visually. A large palette does smooth things out quite nicely. We could give map makers control over the variation from the texture (store it per block), but I'm not sure how that'd play out when making a map; we're also not completely sold on using block textures just yet, though it is something we want to try.
You could just store all the RGB values in an array at the head of the map file. Presuming you don't need more than 256 block colours per map, you could save 3 bytes per stored block by just storing an index into that array for each block.
Separating the map into empty, surface, and solid spaces makes sense. Surface data is typically the most important (with the exception of transparency). This has been standard practice in computer graphics for years, but why not just encode the surface data with a Sparse Voxel Octree? Efficiently storing voxel data has been a solved problem for a long time. This custom "span" approach differs from industry standard while simultaneously being less efficient. It's interesting, maybe it's quicker to load into memory? I don't really understand why this method was chosen though.
Every format has its flaws - scenarios where it performs better or worse. The format explained in the video was chosen partly because it has been used before in this setting for voxel maps that wish to reduce data, and partly because it was familiar to me, so adapting it to add some extra data didn't seem too great a task. I believe part of its design is down to how voxlap renders, is efficient for it, but it's still pretty efficient on the compression side regardless. The vxl format does a pretty good job at reducing data and taking advantage of the usual composition of a map; how it does what it does makes a lot of sense. It's not so great when there's lots of air gaps for example, as that means more spans. It seemed a reasonable way of representing/compressing data while keeping all surface blocks stored fully. From my understanding of it, octrees work best with data that has larger groups of duplicate data? Like vxl it ignores air blocks and then cuts the terrain up into chunks that ideally are a single data point across an entire cell. Still need to look into them more. For raw voxel terrain, I can imagine that would be pretty great; for terrain that may have properties beyond just colour (e.g. material and others), I was not so sure about it initially. Would be interested in learning more about it, because I think I may be misunderstanding it. If releasing this video has taught us anything, it's that there's many more ways to compress voxel data, some of which seem promising ways to keep file sizes workable while potentially storing more data than vxl does. Octrees is one, a chunked format with a palette/atlas per chunk is another, and in general potentially having a different way of loading maps so that you only have to download the full map once, after which you only ever download changed chunks/areas (which then reduces the data transfer overall). Lots of interesting things. No doubt another video like this will come along again when we've settled into what we end up with!
@@GrandyB93 Interesting, I hadn't considered that map data is usually flat. As long as you aren't dealing with something like a dense forest it might be more efficient to store it that way. It mostly will depend on the roughness of your data and how close it is to being 2 dimensional. Rough/jagged height-map-based terrain is probably the best case scenario for your method.
@@adrianm7203 Exactly yeah; the vxl format is at its most efficient when it's solid terrain with just a single top layer block in each column - adding more spans with more air in the column decreases efficiency, which was covered in the video to some degree with the rough analysis of the Normandie Ace of Spades map. Forests generally add one extra span per column and a bunch of extra individual coloured blocks to store, still not too bad though. Worst case would be block air block air block air block air all the way up - very inefficient in this case. At the end of the day, it's a lossy compression and has its pitfalls - we're open to options and have had some good suggestions here in the comments!
It seems like all i ever hear about voxel engines from the devs is them being like "After 4 years in a hyperbolic time chamber, we were able to succesfully get the engine to render voxels one extra frame per second, resulting in a whopping 2 frames per second. Also, sometimes it doesn't work, and if at any point you have a power outage while the game is installed on your PC then you and your house will be erased from this timeline."
You are correct. Most game engines take years to develop as there are multiple paths to take in every decision you make. Same goes for both voxel based engines and regular polygonal engines. We're working a day a week so we things take even longer. We're hoping it'll be worth it in the long run.
I am in no way an expert, and frankly I didn't do a lot of compression stuff at all, and my suggestions might have downsides that are not worth the extra safed space but... If the map only uses a few colors for the whole map, instead of storing color for each block/span, you could make a place in the file for a "color pallet" where you store each RGB value used by the map, and then just reference It later as if accessing an array. If we assume a map will use no more then 255 colors total, we could refer to the color with just one byte index, saving 2 bytes for each color in the process (more or less) Also, if the map uses a lot of decoration, and that decoration repeats a lot and introduced a bit too many block-air-block spans, you could store a reference to the decoration as a pseudo object, with enough info on how to recreate it, reference that, and in the map itself you would just place a single point that refers to the object. again, assuming maximal savings, this would mean that every tree and building and whatever that repeats would only need to be saved once, and then when it is spawned it would only take one byte of space, assuming there would only be 255 decoration objects on each map max. This could potentially be a bit overkill thou, and would change the workflow of making maps and just be not worth the few extra bytes of saved space. I just discovered this project and men I used to grind ace of spades so much... I am glad I am not the only one that didn't forget that game and I really like what I am seeing so far, kinda feels like ace of spades but more like battlefield, definitely caught my interest. oh damn it was a year ago... yeah my suggestions even if applicable are a bit too late to the party, oh well.
We plan to launch the editor with a size limit of 512x512x64. We have plans to add additional sizes once it's all up and running. As for open-ended, prodedural generated maps like Minecraft; it's still too early to tell if we'll be able to support those but we'll see.
make a sparse terrain such that you cant compress it. then you see that your algorithm is only for certain types of terrain and voxel mesh. not generic, you have to render every voxel in some cases. yes RLE/height-map fails for sparse terrain, clouds, trees. ie when there are many independent voxel objects.
Feel free to ask questions, maybe with timestamps, and I can try explain it a different way? Very happy to do that if you wanted to give it another stab!
why storing colors, why not just store type then apply any property to a whole bunch of blocks of the same type? like a grass block would be green, etc..?
Only using textures makes for some rather contrasted edges between blocks. Some games do pretty well with their textures to try and overcome this, but it certainly introduces some visual noise that we'd prefer to keep to a minimum. Colour enables minor variations, which can be pretty helpful in smoothing out areas. Textures are something that are on our list, but probably as an addition to rather than a colour replacement. We'll see if that works, anyway.
I love this video, I appreciate the technical dive into the .vxl format! I wanted to say, though, you should really look into other lossless compression formats. Palette compression is the one I think makes the most sense. I am in active development of an MIT licensed open source voxel format conversion library called gvox. I implemented my own chunked palette compressor and find that my maps are almost always smaller than the more complex .vxl maps (ones with several spans) and with the added bonus of allowing map makers to have full control of the data underground!
Hi! Palette compression was something that Amir (our engine developer) initially had in mind - some kind of colour atlas. The concern was that limiting the colours might limit quality of maps, since when you work with mostly solid colour voxels, small variations in colour mean more and there's a limit on how many colours there should be in an atlas for it to be efficient, vs storing the colour against every block. Either way, when I began looking at the format we moved to looking at doing an extended .vxl style format and made this video. Since the chatter around it, palette compression has come back up, but it's something that'll need careful thought and balance on implementation, as you want as minimal impact as possible on the map makers themselves. The other aspect to this discussion is that we aren't just storing colour, but a couple other details, so some of the more traditional encoding methods aren't going to be quite as efficient. As for gvox, we have to weigh ease of implementation into our C# codebase, how well documented/supported it is and whether we want reliance on another library for this. Tricky to say if it (or other libraries) are worth it.
@@GrandyB93 fair enough. It's worth noting that the gvox palette compression does not rely on a pre-determined palette, it dynamically constructs the palettes, a unique one for every 8x8x8 region. It's completely lossless and allows for full 24 bit color, normals, etc. So has no impact on the artist. The gvox library already supports what we call "channels", which would be your data other than color - we have standard channels like per-voxel color, normal, roughness, hardness, emissivity, etc. As well as support for user-defined channels.
@@GabeRundlett an individual palette per 8x8x8? so, that's up to a max of 512 colours per chunk, so do you then store 10 bits per index? or is that more dynamic? Then there's no doubt a bunch of colours that are used map-wide, so are these duplicated in each 8x8x8? I struggle to picture the compression... so if you have more info I'd be grateful. What's the file size like for some 512x64x512 maps?
@@GrandyB93 great question. There are *up to* 9 bits per index, but it's dynamic based on how large the palette is. It runs over the voxels in the 8x8x8 region, figures out how many unique voxels there are, and then constructs a palette from that - so say there is a green color for grass, a gray color for stone, and a blue color for water that happen to appear in that chunk. Since there are only 3 unique voxel colors in that region, the per-voxel data (which was previously a full 32 bit integer) can now be packed into just 2 bits - each representing an index into the local palette. And there are only 3 entries in the palette - the green, gray and blue colors. So what we need is 2x512 bits per voxel and 3x32 bits for the palette in that region. The idea hinges greatly on the fact that the palette size is dynamic. [The palette and the bits for palette indices I refer to as the "payload" of a chunk] One great thing too is that when an entire chunk is uniform, there's no need to store a payload. Up front there's a uniform grid of payload headers - each representing their respective 8x8x8 chunk, and then a heap of bytes which contains all the payloads. A payload header just consists of 2 values, the size of it's palette, and a pointer to the payload. Here's that special case: If the size of the palette is 1, then the chunk was completely uniform. Therefore I can just put the voxel data directly into where the pointer would normally be, and completely omit a payload.
@@GrandyB93 Now for the question of what the compression rate is. The compression is better than the vxl format for the types of things that vxl is bad at, but it does have some overhead. The reason being is that vxl excels at maps which are primarily 2D, whereas this format I propose is designed for 3D data. (basically, the larger the vxl file, the better my format does in comparison). Let's take 2 example maps from the ace of spades party website, "Arab" and "Lost Valley Arena" both at 512x512x64. The Arab vxl file is 3,483 KB when you download it. When I convert it to my palette format with the gvox library, it becomes 4,028 KB. This is because, although detailed, this map is an example of where the spans do the map a lot of favors - lots of completely vertical regions. The second map, Lost Valley Arena vxl file is 6,772 KB when you download it, but converting it to gvox palette results in a 4,573 KB map. As you can see, the more complex Lost Valley Arena map was reduced in size because this map has much more overhangs and 3D details.
Couldn't you send some procedural data and only send parts of the map that are changed by the mapper and or players? It could be really simple like 2d noise, or complex like Minecraft's 3d noise. Then populated with the changes a mapper makes like adding buildings, a base, fancy trees, a specific cave, etc. Could even ship "prefabs" if you reuse things like foliage a lot. Question is, would the complexity be worth it and the performance outweigh the data.
@@StiekemeHenk It's an interesting idea, parts of which we've thought of before. My thinking is that we have a 'terrain' layer in the map editor which will store much of this information as you have described it; not so much as a dynamic template but more just as a "here's what you did to generate the terrain last time". Obviously terrain generation is also a feature for further down the line as well; getting regular maps playable is first priority. If we did what you describe, there's a few considerations: - We'd have to trust that the connecting clients are all generating the same thing without being tampered with - Storing stuff that isn't the terrain - possibly less efficient, depending how much of it there is. The idea of prefabs is an interesting one, I think some mappers use them a lot while others might not use them at all, so we'd have to see what the common usage is I guess - Storing deletions to the terrain - may make that data model a bit less straightforward, also efficiency if there's a bunch of tunnels made (either by the map maker or during gameplay) Sparse voxel octrees is an option. We're yet to do stuff like lighting, so may revisit it. For now I believe our current thinking is to do simple layers of compression on the entire map, downloaded once... and then only sending the 'changed' chunks over when actually connecting, which should reduce connection times a lot if you already have the base map file locally.
This same exact problem is always in the back of my head. I’ve always thought about grouping areas of the same type voxels into rectangular shapes and then storing those, but that sounds like a lot work for the cpu.
Would it be possible to store only what's been modified? The seed for generating the map should always be able to regenerate the same map, it would incure the cost of generating it again though.
Hi from Colombia: I have a question . Why u save the color ?, i think is better save the index of the color. for example 1 => FFFFFF 2 => FF6655 ETC. The palette is a table of 255 posible colors.
Using a palette with IDs is certainly a possibility, though we've been back and forth about how good/bad certain implementations of that might be. 256 colours sounds like a lot, but from some checks we did against Ace of Spades maps, they were numbers in the thousands - from something simple in Normandie (1835 colours) to some of the more complex maps like Harbor2 (58,779 colours). Terrain is often the main cause of this as the buildings etc are generally hand made/placed and models get re-used. Since a colour is 3x8 bits, that's a max of 16,777,215... 16 bits is 65,535... 12 bits is 4,095... it really depends on how much compression we want - whether we want maps to be lossless, or slightly lossy in terms of colour data. Since terrain is usually the main culprit for the extra colours, we could restrict the colour palette in-editor for map objects, and add those straight to the atlas, and then do some complicated (probably histogram-like) compression on terrain colours to fill the rest of the atlas. Might be possible without it being too easily noticed. As mentioned in another comment on this video by Gabe Rundlett, another option is a dynamic colour index per-chunk, which is a different but interesting approach to it. Keeps it lossless but in theory should have storage gains due to the index only being as big as it needs to be per chunk.
![Adding ray tracing (back) to my game engine [Voxel Devlog #17]](http://i.ytimg.com/vi/aY4Zet_C9Zs/mqdefault.jpg)
![Adding ray tracing (back) to my game engine [Voxel Devlog #17]](/img/tr.png)
great video! loved the technical deep dive.
🫡
Awesome!
i think it would be cool if underground there were rocks that aren't indestructible but hard to destroy so you have to dig around them (just as an obstacle)
Cool concept indeed but our current plan is not to discourage digging or building - which is why we're aiming towards making blocks that are placed by players have more HP.
Even tough optimizing isn't bad, just remember that the speedtest global index has the "worst average connection by country" at 30 megabits per second, which by the standards set by this video (1 megabit connection)
is 2 seconds.
Better be safe than sorry, but i wouldn't worry that much about it
Roger that.
Storing the RGB values for each block is a lot of data that I think you could trim down.
R - 256
G -256
B - 256
Results in literally millions of color combinations that you probably don't need.
Have you considered instead storing a block type? For example: Grass, Stone, Dirt, etc.
Then mapping each of those statically to RGB values?
If you want color variation you could even map to a set of possible RGB values. Or upon render add some random +- from the programmed color value.
Using a block type instead you would only need 1 byte of data per block, for a max of 256 block types.
This also opens the possibility for implementing more complex logic for specific block materials in the future.
I did consider a variation-based system. Minecraft does that to smooth between biomes but that's based on a global noise texture as opposed to data stored per-block. If you solely go with RGB textures, the map does become pretty contrasty and more noisy visually. A large palette does smooth things out quite nicely.
We could give map makers control over the variation from the texture (store it per block), but I'm not sure how that'd play out when making a map; we're also not completely sold on using block textures just yet, though it is something we want to try.
You could just store all the RGB values in an array at the head of the map file.
Presuming you don't need more than 256 block colours per map, you could save 3 bytes per stored block by just storing an index into that array for each block.
I am actually making a Minecraft-inspired voxel game in the Rust programming language and the Bevy game engine.
Just had a gander at your videos. Great stuff, soldier. Keep it up!
Oh, is this gonna be like Ace of Spades? I miss that game.
So do we, soldier. AoS was a hidden gem.
Separating the map into empty, surface, and solid spaces makes sense. Surface data is typically the most important (with the exception of transparency). This has been standard practice in computer graphics for years, but why not just encode the surface data with a Sparse Voxel Octree? Efficiently storing voxel data has been a solved problem for a long time. This custom "span" approach differs from industry standard while simultaneously being less efficient. It's interesting, maybe it's quicker to load into memory? I don't really understand why this method was chosen though.
Every format has its flaws - scenarios where it performs better or worse.
The format explained in the video was chosen partly because it has been used before in this setting for voxel maps that wish to reduce data, and partly because it was familiar to me, so adapting it to add some extra data didn't seem too great a task. I believe part of its design is down to how voxlap renders, is efficient for it, but it's still pretty efficient on the compression side regardless.
The vxl format does a pretty good job at reducing data and taking advantage of the usual composition of a map; how it does what it does makes a lot of sense. It's not so great when there's lots of air gaps for example, as that means more spans. It seemed a reasonable way of representing/compressing data while keeping all surface blocks stored fully.
From my understanding of it, octrees work best with data that has larger groups of duplicate data? Like vxl it ignores air blocks and then cuts the terrain up into chunks that ideally are a single data point across an entire cell. Still need to look into them more. For raw voxel terrain, I can imagine that would be pretty great; for terrain that may have properties beyond just colour (e.g. material and others), I was not so sure about it initially. Would be interested in learning more about it, because I think I may be misunderstanding it.
If releasing this video has taught us anything, it's that there's many more ways to compress voxel data, some of which seem promising ways to keep file sizes workable while potentially storing more data than vxl does. Octrees is one, a chunked format with a palette/atlas per chunk is another, and in general potentially having a different way of loading maps so that you only have to download the full map once, after which you only ever download changed chunks/areas (which then reduces the data transfer overall). Lots of interesting things.
No doubt another video like this will come along again when we've settled into what we end up with!
@@GrandyB93 Interesting, I hadn't considered that map data is usually flat. As long as you aren't dealing with something like a dense forest it might be more efficient to store it that way. It mostly will depend on the roughness of your data and how close it is to being 2 dimensional. Rough/jagged height-map-based terrain is probably the best case scenario for your method.
@@adrianm7203 Exactly yeah; the vxl format is at its most efficient when it's solid terrain with just a single top layer block in each column - adding more spans with more air in the column decreases efficiency, which was covered in the video to some degree with the rough analysis of the Normandie Ace of Spades map. Forests generally add one extra span per column and a bunch of extra individual coloured blocks to store, still not too bad though. Worst case would be block air block air block air block air all the way up - very inefficient in this case.
At the end of the day, it's a lossy compression and has its pitfalls - we're open to options and have had some good suggestions here in the comments!
It seems like all i ever hear about voxel engines from the devs is them being like "After 4 years in a hyperbolic time chamber, we were able to succesfully get the engine to render voxels one extra frame per second, resulting in a whopping 2 frames per second. Also, sometimes it doesn't work, and if at any point you have a power outage while the game is installed on your PC then you and your house will be erased from this timeline."
You are correct. Most game engines take years to develop as there are multiple paths to take in every decision you make. Same goes for both voxel based engines and regular polygonal engines. We're working a day a week so we things take even longer. We're hoping it'll be worth it in the long run.
I am in no way an expert, and frankly I didn't do a lot of compression stuff at all, and my suggestions might have downsides that are not worth the extra safed space but...
If the map only uses a few colors for the whole map, instead of storing color for each block/span, you could make a place in the file for a "color pallet" where you store each RGB value used by the map, and then just reference It later as if accessing an array. If we assume a map will use no more then 255 colors total, we could refer to the color with just one byte index, saving 2 bytes for each color in the process (more or less)
Also, if the map uses a lot of decoration, and that decoration repeats a lot and introduced a bit too many block-air-block spans, you could store a reference to the decoration as a pseudo object, with enough info on how to recreate it, reference that, and in the map itself you would just place a single point that refers to the object. again, assuming maximal savings, this would mean that every tree and building and whatever that repeats would only need to be saved once, and then when it is spawned it would only take one byte of space, assuming there would only be 255 decoration objects on each map max. This could potentially be a bit overkill thou, and would change the workflow of making maps and just be not worth the few extra bytes of saved space.
I just discovered this project and men I used to grind ace of spades so much... I am glad I am not the only one that didn't forget that game and I really like what I am seeing so far, kinda feels like ace of spades but more like battlefield, definitely caught my interest.
oh damn it was a year ago... yeah my suggestions even if applicable are a bit too late to the party, oh well.
How does the code work. how do i make an open sized map like Minecraft or AOS with full destruction block generation ?
We plan to launch the editor with a size limit of 512x512x64. We have plans to add additional sizes once it's all up and running. As for open-ended, prodedural generated maps like Minecraft; it's still too early to tell if we'll be able to support those but we'll see.
make a sparse terrain such that you cant compress it. then you see that your algorithm is only for certain types of terrain and voxel mesh. not generic, you have to render every voxel in some cases. yes RLE/height-map fails for sparse terrain, clouds, trees. ie when there are many independent voxel objects.
this is a uhhhhhhh video i don't understand :(
It's okay I only got it after editing the video and watching it several times. Not an easy thing to wrap your head around.
@@BorntoBuild hah. nice little fact. as always, great video!
🫡
Feel free to ask questions, maybe with timestamps, and I can try explain it a different way? Very happy to do that if you wanted to give it another stab!
Very cool to see development in the project.
Thank you for watching 🫡
why storing colors, why not just store type then apply any property to a whole bunch of blocks of the same type?
like a grass block would be green, etc..?
Only using textures makes for some rather contrasted edges between blocks. Some games do pretty well with their textures to try and overcome this, but it certainly introduces some visual noise that we'd prefer to keep to a minimum.
Colour enables minor variations, which can be pretty helpful in smoothing out areas. Textures are something that are on our list, but probably as an addition to rather than a colour replacement. We'll see if that works, anyway.
I love this video, I appreciate the technical dive into the .vxl format! I wanted to say, though, you should really look into other lossless compression formats. Palette compression is the one I think makes the most sense. I am in active development of an MIT licensed open source voxel format conversion library called gvox. I implemented my own chunked palette compressor and find that my maps are almost always smaller than the more complex .vxl maps (ones with several spans) and with the added bonus of allowing map makers to have full control of the data underground!
Hi!
Palette compression was something that Amir (our engine developer) initially had in mind - some kind of colour atlas. The concern was that limiting the colours might limit quality of maps, since when you work with mostly solid colour voxels, small variations in colour mean more and there's a limit on how many colours there should be in an atlas for it to be efficient, vs storing the colour against every block.
Either way, when I began looking at the format we moved to looking at doing an extended .vxl style format and made this video. Since the chatter around it, palette compression has come back up, but it's something that'll need careful thought and balance on implementation, as you want as minimal impact as possible on the map makers themselves.
The other aspect to this discussion is that we aren't just storing colour, but a couple other details, so some of the more traditional encoding methods aren't going to be quite as efficient.
As for gvox, we have to weigh ease of implementation into our C# codebase, how well documented/supported it is and whether we want reliance on another library for this. Tricky to say if it (or other libraries) are worth it.
@@GrandyB93 fair enough. It's worth noting that the gvox palette compression does not rely on a pre-determined palette, it dynamically constructs the palettes, a unique one for every 8x8x8 region. It's completely lossless and allows for full 24 bit color, normals, etc. So has no impact on the artist. The gvox library already supports what we call "channels", which would be your data other than color - we have standard channels like per-voxel color, normal, roughness, hardness, emissivity, etc. As well as support for user-defined channels.
@@GabeRundlett an individual palette per 8x8x8? so, that's up to a max of 512 colours per chunk, so do you then store 10 bits per index? or is that more dynamic?
Then there's no doubt a bunch of colours that are used map-wide, so are these duplicated in each 8x8x8? I struggle to picture the compression... so if you have more info I'd be grateful.
What's the file size like for some 512x64x512 maps?
@@GrandyB93 great question. There are *up to* 9 bits per index, but it's dynamic based on how large the palette is. It runs over the voxels in the 8x8x8 region, figures out how many unique voxels there are, and then constructs a palette from that - so say there is a green color for grass, a gray color for stone, and a blue color for water that happen to appear in that chunk. Since there are only 3 unique voxel colors in that region, the per-voxel data (which was previously a full 32 bit integer) can now be packed into just 2 bits - each representing an index into the local palette. And there are only 3 entries in the palette - the green, gray and blue colors. So what we need is 2x512 bits per voxel and 3x32 bits for the palette in that region. The idea hinges greatly on the fact that the palette size is dynamic. [The palette and the bits for palette indices I refer to as the "payload" of a chunk] One great thing too is that when an entire chunk is uniform, there's no need to store a payload. Up front there's a uniform grid of payload headers - each representing their respective 8x8x8 chunk, and then a heap of bytes which contains all the payloads. A payload header just consists of 2 values, the size of it's palette, and a pointer to the payload. Here's that special case: If the size of the palette is 1, then the chunk was completely uniform. Therefore I can just put the voxel data directly into where the pointer would normally be, and completely omit a payload.
@@GrandyB93 Now for the question of what the compression rate is. The compression is better than the vxl format for the types of things that vxl is bad at, but it does have some overhead. The reason being is that vxl excels at maps which are primarily 2D, whereas this format I propose is designed for 3D data. (basically, the larger the vxl file, the better my format does in comparison). Let's take 2 example maps from the ace of spades party website, "Arab" and "Lost Valley Arena" both at 512x512x64. The Arab vxl file is 3,483 KB when you download it. When I convert it to my palette format with the gvox library, it becomes 4,028 KB. This is because, although detailed, this map is an example of where the spans do the map a lot of favors - lots of completely vertical regions. The second map, Lost Valley Arena vxl file is 6,772 KB when you download it, but converting it to gvox palette results in a 4,573 KB map. As you can see, the more complex Lost Valley Arena map was reduced in size because this map has much more overhangs and 3D details.
Couldn't you send some procedural data and only send parts of the map that are changed by the mapper and or players?
It could be really simple like 2d noise, or complex like Minecraft's 3d noise. Then populated with the changes a mapper makes like adding buildings, a base, fancy trees, a specific cave, etc. Could even ship "prefabs" if you reuse things like foliage a lot.
Question is, would the complexity be worth it and the performance outweigh the data.
Of course this would only work for maps made in this way and not for AOS maps.
Another option I'm wondering about is anything like sparse voxel octrees?
@@StiekemeHenk It's an interesting idea, parts of which we've thought of before.
My thinking is that we have a 'terrain' layer in the map editor which will store much of this information as you have described it; not so much as a dynamic template but more just as a "here's what you did to generate the terrain last time". Obviously terrain generation is also a feature for further down the line as well; getting regular maps playable is first priority.
If we did what you describe, there's a few considerations:
- We'd have to trust that the connecting clients are all generating the same thing without being tampered with
- Storing stuff that isn't the terrain - possibly less efficient, depending how much of it there is. The idea of prefabs is an interesting one, I think some mappers use them a lot while others might not use them at all, so we'd have to see what the common usage is I guess
- Storing deletions to the terrain - may make that data model a bit less straightforward, also efficiency if there's a bunch of tunnels made (either by the map maker or during gameplay)
Sparse voxel octrees is an option. We're yet to do stuff like lighting, so may revisit it.
For now I believe our current thinking is to do simple layers of compression on the entire map, downloaded once... and then only sending the 'changed' chunks over when actually connecting, which should reduce connection times a lot if you already have the base map file locally.
Cool bro! I want watch gameplay!
Ty very much. So do we 😂
This same exact problem is always in the back of my head. I’ve always thought about grouping areas of the same type voxels into rectangular shapes and then storing those, but that sounds like a lot work for the cpu.
Would it be possible to store only what's been modified? The seed for generating the map should always be able to regenerate the same map, it would incure the cost of generating it again though.
Working on another devlog regarding the import process. Should hopefully answer your question, soldier.
Hi from Colombia: I have a question . Why u save the color ?, i think is better save the index of the color.
for example
1 => FFFFFF
2 => FF6655
ETC.
The palette is a table of 255 posible colors.
Using a palette with IDs is certainly a possibility, though we've been back and forth about how good/bad certain implementations of that might be.
256 colours sounds like a lot, but from some checks we did against Ace of Spades maps, they were numbers in the thousands - from something simple in Normandie (1835 colours) to some of the more complex maps like Harbor2 (58,779 colours). Terrain is often the main cause of this as the buildings etc are generally hand made/placed and models get re-used.
Since a colour is 3x8 bits, that's a max of 16,777,215... 16 bits is 65,535... 12 bits is 4,095... it really depends on how much compression we want - whether we want maps to be lossless, or slightly lossy in terms of colour data. Since terrain is usually the main culprit for the extra colours, we could restrict the colour palette in-editor for map objects, and add those straight to the atlas, and then do some complicated (probably histogram-like) compression on terrain colours to fill the rest of the atlas. Might be possible without it being too easily noticed.
As mentioned in another comment on this video by Gabe Rundlett, another option is a dynamic colour index per-chunk, which is a different but interesting approach to it. Keeps it lossless but in theory should have storage gains due to the index only being as big as it needs to be per chunk.