Your game synchronization is a form of Consistency. That's the C in CAP/PACELS theorem and the ACID rules. Basically one has to choose between latency (L) and loss of consistency (C). It's also related to merges in source control as you have to determine who "Wins" and what is irrelevant. Maybe the statement applies here "I'll do it in practice, because it will never work in theory". Best of luck, network multiplayer is a great feature to have, if you can pull it off.
Do you have your gamestate in a format where you can checksum it? If you have quicksave functionality, perhaps you can make a save (without writing to file) on each tick, checksum it, and compare the host and client checksum. This is essentially what a studio I used to work at did. The save format for all the games were a huge human-readable table of key-value pairs. When an out-of-sync was detected, the host and client saved the game to file, and we manually did a diff of the save files to see where the issue is located. Additionally, we logged each random number generator call so that we could diff these in case the host and client took a different path of execution. With this and other tracking it still took a lot of time to find and fix out-of-sync issues, though.
That's similar as to what I'm doing now, dump random number seeds and values when something goes wrong... Hopefully at some point all discrepancies are gone 😎
I use a custom GASP implementation. Interpolation is how you handle the lag. GASP is a great multiplayer specification. The downside of a multiplayer server model is cost. You might be better off with a non server model solution that is peer to peer where the server only holds the list of players in a given game. It would reduce cost, but it is not good when you have cheaters.
Interpolation won't work in this deyrrministic solution 🤔 but there are rollback and prediction options available. And it will be cliënt to cliënt, and its still coop playing together as a team, so cheating wouldn't make much sense 😁
@@orangepixelgames Interpolation does not eliminate the deterministic behavior of a server model. Yes I don't think cheaters would benefit much, but it is something to know about when using peer to peer.
Pure madness, but mad props for the attempt! 😎When RC had party AI, I thought it would be a great fit for 4-player co-op, but didn't suggest it for exactly this reason. I feel like multiplayer has got to be baked-in to the architecture, even if it's not implemented until later, otherwise you risk ripping the innards out trying to rearchitect an already-working game. That said, I think you're on the right track with creating a library out of this that you could overlay on your existing framework for future games. It could even be used as an enhancement (or DLC) down the line for your existing games that fit the bill. I'd recommend some kind of Friend Code rather than lobbies though, since players are way more likely to play indie games with known ppl vs random strangers. You might be able to use the code-sharing to exchange initial seed-sync and potential-lag info (region-checking) as well. Not my area (only did a little in LAN class), but it's fascinating to watch you unpack it all!
This was really interesting. I've thought about trying to add online play to the current project I am working on and have read up on the topic, but I decided it was too much extra work for a part-time/hobby project developed by one person. Maybe one day if I actually finish the project and if it gets a bit of an audience I'll do it for a sequel or future project.
Yeah it's a lot of work and if it's just for one game it's probably a waste of time.. but I figure I'll be using the code in different projects in the future! 🤞
This is one of those areas that I've been running all my life from, maybe one day... I will add this, but for now, I do not think there is much audience for my games yet, maybe someday!, keep it up and let us know how it worked at the end!
Great video Pascal. Really fun seeing you go through your work process. I only play single player games, but I know that online multiplayer really broadens your audience. I hope you manage to get the code working. Good luck.
client side prediction, lag compensation, scene management, sync vars, rpc calls, etc.. welcome to the rabbit hole. check mirror networking solutions, they explain these terms in detail.
@@mandisaw yes, I tried recently unity's netcode for game objects and kept studying their Boss Room official example for two weeks, surprisingly most of networking essential stuff weren't implemented things like lag compensation and prediction are absent, all good solutions out there are commercial unfortunately.
Isn't there a saying that if you think you're done with the game and just need to add multiplayer, you really haven't started yet? :) Good stuff as usual, Pascal-and I say this out of my own free nondeterministic will.
Great video 💯, I think you can always generate random numbers with host and client only gets them. I mean make it like host user CPU does all the things and the client user act like he has only second controller. This way things become a lot easier
Hey man, cool video as always, I like the realism! haha the problem with deterministic lockstep is the "determinism" as you noticed haha... you might have done that already and I dont want to be preaching to the choir here but in case any of this helps, make sure literally all numbers in the game are fixed point and that the code is executed in a fixed order. It is pure magic when you have the determinism nailed down, it is like making a single player game which is amazing, but unless its thought about from the very beginning it is likely very hard. For example, event driven architectures will make it nearly impossible to guarantee the order events are fired, so how do you do damage? how do you spawn bullets? can you batch those (as in, when you want to spawn a bullet, create a spawn bullet request, then process all together)? State based architectures are easier (like ECS) since you already need to process all "events" manually so you can control the execution order... and the problem is it is a pain to debug and find the source cause of where determinism was broken. Try and make sure that all spawn events (for bullets for example) happen in the same "network frame", all damage is processed in the same "network frame" as well, basically replace your "update" with network update so the game literally pauses if you stop receiving network packets.... and well, good luck! haha deterministic lockstep for action games is in my opinion the hardest to achieve (client prediction becomes mandatory and very complicated to handle because you need the server world state and the local world state and reconcile both all the time...) but if you can do it, you will have the most magical, efficient, cheap and expandable network architecture possible, which Im sure will be very valuable for all your games, which btw, congrats on your studio, love the way you work, the dev strategy and focus on making cool games, not easy to get there!
Thanks! but very aware of all of this 😁 AND! Getting closer to a perfect state as I iron out tiny issues left and right... Which will be a good lesson for future games I make with this tech. I do foresee the headaches of any game updates I do later on causing issues in the multiplayer stuff (as it needs a bunch of testing).. but that's a future problem!
I've done deterministic lock-step with just sending control inputs. It's my preferred method. I had it working of an old GBA game I was hired to make; and 99.99% of the time it worked (GBA's connected via cables), but every so often it would just 'drift off', absolutely no idea why to this day. Are you using some kind of library for the networking stuff?
What if you let it be a bit out of sync but then correct it from the host by sending important information and only letting non-game breaking things be a bit out of sync
Steam remote play is impressive, but not flawless! I should probably dive into tweaking the settings to make it smoother but not sure how many players can and will do that 🤔
Doing it using "player inputs" usually ends up going out of sync in my experience (I call it BMX Simulator syndrome because the replays on that did it), once that happens players will wander into walls etc missing things by a few ms each frame that add up (even on local replay from an array!). On unity thats what happens anyway, due to physics and not being fully deterministic I think. I am implementing something similar now but thinking of going with just coordinates and lerping between them, so u can pack in like x,y,, for each player. Its the only way I can see it working (2 player you can use a socket, for more players im thinking of using signal-R and a simple blazor server), had to give up on my previous 7 implementations :) Photon pun works but its a bit laggy.
That's the thing in fixing now, tiny stuff but feels like I'm still making progress on it. Sending just player coordinates will also need world state changes? So that's another whole mess to work out!
@@orangepixelgames true yeh, if they need to be able to open a door or kill an enemy etc. seems so easy at the start then seems impossible after a week of trying lol. i guess events like open door or kill someone could be a const int and an id, like 1,15 (open door, door id 15). and it all ends up a stream of the most simple commands separated by pipe or something.
I think this is never going to work if you only send the movemets of the players. For example, a player destroys a box and that package is lost, the other player will never see that box destroyed, the map is going to be completely different. Am I missing something?
That's why it's called 'deterministic' - it's like running a simulation step by step, and before each step you collect everyone's input, and only if you got all of that do you continue a single step in the 'simulation' (game loop). Players, objects, bullets, all do the same thing in the same situation. So the box will be destroyed in all simulations (all clients).. it's how many old games did multiplayer on old connections, and many MMORPG games still use this
I tried networking box2d once but gave up on it. Determinism seems impossible between different machines/architectures because they implement floating point math differently. Which may explain the discrepancy between your pseudo random generators.
My game is not deterministic. i made the host manage and sync the events that happen in the game. The host has all the data and just send it to the clients
I also don't have deterministic logic in my multiplayer game, so I use some lerping function on the server to smoothly adjust for the differences at a fixed speed so players still can't cheat movementspeed. Works pretty well but feels like a bandaid solution
Why don't you use a game engine instead? I think would be a lot easier to use game maker to do all things, game design, multi-player, platform porting etc, Also game maker it's not that expensive so I think worth the price
I think he's got at least 4 videos discussing that, from tech & business POVs. I use native-code for work and an engine for my games now, both have pros & cons.
I do enjoy these sort of realistic dev logs talking about the real technical aspect of development
It's fun to make these videos, but it's also been chaos! So glad (some) people enjoy it 😀
Your game synchronization is a form of Consistency. That's the C in CAP/PACELS theorem and the ACID rules. Basically one has to choose between latency (L) and loss of consistency (C). It's also related to merges in source control as you have to determine who "Wins" and what is irrelevant.
Maybe the statement applies here "I'll do it in practice, because it will never work in theory". Best of luck, network multiplayer is a great feature to have, if you can pull it off.
Do you have your gamestate in a format where you can checksum it? If you have quicksave functionality, perhaps you can make a save (without writing to file) on each tick, checksum it, and compare the host and client checksum.
This is essentially what a studio I used to work at did. The save format for all the games were a huge human-readable table of key-value pairs. When an out-of-sync was detected, the host and client saved the game to file, and we manually did a diff of the save files to see where the issue is located. Additionally, we logged each random number generator call so that we could diff these in case the host and client took a different path of execution. With this and other tracking it still took a lot of time to find and fix out-of-sync issues, though.
That's similar as to what I'm doing now, dump random number seeds and values when something goes wrong... Hopefully at some point all discrepancies are gone 😎
I use a custom GASP implementation. Interpolation is how you handle the lag. GASP is a great multiplayer specification. The downside of a multiplayer server model is cost. You might be better off with a non server model solution that is peer to peer where the server only holds the list of players in a given game. It would reduce cost, but it is not good when you have cheaters.
Interpolation won't work in this deyrrministic solution 🤔 but there are rollback and prediction options available. And it will be cliënt to cliënt, and its still coop playing together as a team, so cheating wouldn't make much sense 😁
@@orangepixelgames Interpolation does not eliminate the deterministic behavior of a server model. Yes I don't think cheaters would benefit much, but it is something to know about when using peer to peer.
Pure madness, but mad props for the attempt! 😎When RC had party AI, I thought it would be a great fit for 4-player co-op, but didn't suggest it for exactly this reason. I feel like multiplayer has got to be baked-in to the architecture, even if it's not implemented until later, otherwise you risk ripping the innards out trying to rearchitect an already-working game.
That said, I think you're on the right track with creating a library out of this that you could overlay on your existing framework for future games. It could even be used as an enhancement (or DLC) down the line for your existing games that fit the bill.
I'd recommend some kind of Friend Code rather than lobbies though, since players are way more likely to play indie games with known ppl vs random strangers. You might be able to use the code-sharing to exchange initial seed-sync and potential-lag info (region-checking) as well. Not my area (only did a little in LAN class), but it's fascinating to watch you unpack it all!
Right now it will be sharing your IP (or I might turn that into a code) and only play with friends.. the good old fashioned way!
Impressive how you learned yet another technique/system for Regulator City. I hope it works, but if anything there is cool local coop :D
I feel like I'm close to getting it working.. at the least for LAN parties 😁🤞
This was really interesting. I've thought about trying to add online play to the current project I am working on and have read up on the topic, but I decided it was too much extra work for a part-time/hobby project developed by one person. Maybe one day if I actually finish the project and if it gets a bit of an audience I'll do it for a sequel or future project.
Yeah it's a lot of work and if it's just for one game it's probably a waste of time.. but I figure I'll be using the code in different projects in the future! 🤞
This is one of those areas that I've been running all my life from, maybe one day... I will add this, but for now, I do not think there is much audience for my games yet, maybe someday!, keep it up and let us know how it worked at the end!
I always avoided multiplayer, but it's really cool if it works and would love to play Regulator City with friends online, so not giving up yet 😏
Great video Pascal. Really fun seeing you go through your work process. I only play single player games, but I know that online multiplayer really broadens your audience. I hope you manage to get the code working. Good luck.
Made some more progress after video was uploaded, so hopefully a follow up video soonish 🤞😁
Even if it doesn't 100% work for now, I am sure you can use the experience and use it on future games, interesting video as always :)
In my head I already work on game ideas where this will absolutely work 😁 but haven't given up on the Regulator City code either!
client side prediction, lag compensation, scene management, sync vars, rpc calls, etc..
welcome to the rabbit hole.
check mirror networking solutions, they explain these terms in detail.
Oh I've been diving into all of it this week... It's crazy but I do really like figuring these things out I guess!
Netcode is really an entire subdiscipline of development unto itself. That rabbit hole hides quite the iceberg 😅
@@mandisaw yes, I tried recently unity's netcode for game objects and kept studying their Boss Room official example for two weeks, surprisingly most of networking essential stuff weren't implemented things like lag compensation and prediction are absent, all good solutions out there are commercial unfortunately.
Awesome video and very informative for me as game developer. It puts me in the mood to try and do multiplayer myself.
It's certainly fun to tinker with! Even if it fails or isn't useful for all situations/games
Isn't there a saying that if you think you're done with the game and just need to add multiplayer, you really haven't started yet? :)
Good stuff as usual, Pascal-and I say this out of my own free nondeterministic will.
Haha yeah they also say: don't do multiplayer 😁 or 'never tag it onto an existing single player game' 😅
Great video 💯, I think you can always generate random numbers with host and client only gets them. I mean make it like host user CPU does all the things and the client user act like he has only second controller.
This way things become a lot easier
Hey man, cool video as always, I like the realism! haha the problem with deterministic lockstep is the "determinism" as you noticed haha... you might have done that already and I dont want to be preaching to the choir here but in case any of this helps, make sure literally all numbers in the game are fixed point and that the code is executed in a fixed order. It is pure magic when you have the determinism nailed down, it is like making a single player game which is amazing, but unless its thought about from the very beginning it is likely very hard. For example, event driven architectures will make it nearly impossible to guarantee the order events are fired, so how do you do damage? how do you spawn bullets? can you batch those (as in, when you want to spawn a bullet, create a spawn bullet request, then process all together)? State based architectures are easier (like ECS) since you already need to process all "events" manually so you can control the execution order... and the problem is it is a pain to debug and find the source cause of where determinism was broken. Try and make sure that all spawn events (for bullets for example) happen in the same "network frame", all damage is processed in the same "network frame" as well, basically replace your "update" with network update so the game literally pauses if you stop receiving network packets.... and well, good luck! haha deterministic lockstep for action games is in my opinion the hardest to achieve (client prediction becomes mandatory and very complicated to handle because you need the server world state and the local world state and reconcile both all the time...) but if you can do it, you will have the most magical, efficient, cheap and expandable network architecture possible, which Im sure will be very valuable for all your games, which btw, congrats on your studio, love the way you work, the dev strategy and focus on making cool games, not easy to get there!
Thanks! but very aware of all of this 😁 AND! Getting closer to a perfect state as I iron out tiny issues left and right... Which will be a good lesson for future games I make with this tech.
I do foresee the headaches of any game updates I do later on causing issues in the multiplayer stuff (as it needs a bunch of testing).. but that's a future problem!
I've done deterministic lock-step with just sending control inputs. It's my preferred method. I had it working of an old GBA game I was hired to make; and 99.99% of the time it worked (GBA's connected via cables), but every so often it would just 'drift off', absolutely no idea why to this day. Are you using some kind of library for the networking stuff?
I'm using Kryonet (Java) which works amazingly simple for all TCP/UDP stuff. Easy setup, easy usage.. saved me a lot of time!
@@orangepixelgames We could use a vid just on choosing & integrating a netcode library into your existing project / framework 😲
Good luck with it Pascal. I need to find a lightweight c++ networking library for RR2. Only need minimal functionality from it.
My choice was limited as Kryonet seems to be the default one and unbeaten one for many years for libgdx and java users
What if you let it be a bit out of sync but then correct it from the host by sending important information and only letting non-game breaking things be a bit out of sync
You can do some prediction and rollback with this method, so I might look into that if everything else actually works 😁
Love this!!! All I can say is, thank god for Steam Remote Play!!!
Steam remote play is impressive, but not flawless! I should probably dive into tweaking the settings to make it smoother but not sure how many players can and will do that 🤔
Doing it using "player inputs" usually ends up going out of sync in my experience (I call it BMX Simulator syndrome because the replays on that did it), once that happens players will wander into walls etc missing things by a few ms each frame that add up (even on local replay from an array!). On unity thats what happens anyway, due to physics and not being fully deterministic I think. I am implementing something similar now but thinking of going with just coordinates and lerping between them, so u can pack in like x,y,, for each player. Its the only way I can see it working (2 player you can use a socket, for more players im thinking of using signal-R and a simple blazor server), had to give up on my previous 7 implementations :) Photon pun works but its a bit laggy.
That's the thing in fixing now, tiny stuff but feels like I'm still making progress on it. Sending just player coordinates will also need world state changes? So that's another whole mess to work out!
@@orangepixelgames true yeh, if they need to be able to open a door or kill an enemy etc. seems so easy at the start then seems impossible after a week of trying lol. i guess events like open door or kill someone could be a const int and an id, like 1,15 (open door, door id 15). and it all ends up a stream of the most simple commands separated by pipe or something.
I think this is never going to work if you only send the movemets of the players.
For example, a player destroys a box and that package is lost, the other player will never see that box destroyed, the map is going to be completely different. Am I missing something?
That's why it's called 'deterministic' - it's like running a simulation step by step, and before each step you collect everyone's input, and only if you got all of that do you continue a single step in the 'simulation' (game loop). Players, objects, bullets, all do the same thing in the same situation. So the box will be destroyed in all simulations (all clients).. it's how many old games did multiplayer on old connections, and many MMORPG games still use this
@@orangepixelgames Yeah, but if you lose ONE packet of the other player, then, the output is not the same for both players
That's where the lockstep part comes in. You don't move until you have that package, and you re-request it if it's not there
Not sure how feasible this may be, but making this game moddable would give this game a community and give it longevity.
Level editor is most likely shipping with the game 😉
Can I ask how do you render crisp looking text in low resolution games ? I assume you still use libgdx ?
Yup, still using libgdx. I use fonts that can be used at 8pixels or 10pixels to make sure they are crisp
What Level editor are you using in 9:48 ?
Custom level editor build into Regulator City 😉
Very cool to see your journey with working on multiplayer, even if it hasn't worked out to your satisfaction yet. Nice dev vlog
Yet being the key word! 😁
I tried networking box2d once but gave up on it. Determinism seems impossible between different machines/architectures because they implement floating point math differently. Which may explain the discrepancy between your pseudo random generators.
Haha yeah I don't use physics engines, and no floating points for movements! So I'm safe on that end
How much do you pay Steam to use their serves for multi-player
no idea! for now I'm using "self-hosted" so the hosting player IS the server and the client connects to that one.. old school style.
My game is not deterministic. i made the host manage and sync the events that happen in the game. The host has all the data and just send it to the clients
That's another way to go, but I don't have the game setup for that method 😏
I also don't have deterministic logic in my multiplayer game, so I use some lerping function on the server to smoothly adjust for the differences at a fixed speed so players still can't cheat movementspeed. Works pretty well but feels like a bandaid solution
I hope this works
Me too!
Why don't you use a game engine instead? I think would be a lot easier to use game maker to do all things, game design, multi-player, platform porting etc,
Also game maker it's not that expensive so I think worth the price
Instead of paying to someone 50% of you revenue, you can do it yourself with yoyo games, would be a lot cheaper
I think he's got at least 4 videos discussing that, from tech & business POVs. I use native-code for work and an engine for my games now, both have pros & cons.