Really nice 😀! I wish that serie was available when I started my project! I had to write my own system for Godot. It's really good tho, it makes me validate my comprehension of networking and the concepts I implented! Thank you!
Holy cow. What an incredible addon and series! I'm pretty noob as a game developer, but this series made me feel like I could probably pull off rollback for a highly competitive platform fighter if I want to, although I'm sure it won't be easy! And actually, I'm also glad I watched this series because it gave me a rough idea of how much work will actually be required to make rollback work in my game. One thing I wanted to point out for other people considering rollback for their game, is that, at least as it is implemented in this (currently 11 part) tutorial series, this framework seems highly susceptible to cheating. I'm no expert on networking, but it seems to me like you could just write a script or find some way to purposefully lag your input uploads, meanwhile you download the opponent's perfectly streamlined inputs, and now your screen shows a game with 0 rollbacks with only 2 frames of input delay, and meanwhile your opponent is experiencing many artifacts and an effectively increased latency, giving you a huge reaction time advantage. Furthermore, it is reasonable for your opponent (who probably doesn't know anything about what's going on under the hood) to incorrectly assume that the cheater is also experiencing the same artifacts, and thus they may not be suspicious of any foul play. I'm sure there are many partial solutions to this problem, but it's at least something worth thinking about. And couldn't this situation arise even with no malicious intent? You mentioned "fairness" as a positive aspect of rollback, but I wasn't really convinced by your brief explanation that this is the case. I'd love to hear more about why you think this is the case, or what you have to say in response to what I mentioned above about attempts to cheat in more competitive games.
The cheating wouldn't really work because of the rollback re-simulation. The way you describe the cheat seems backwards; don't you want to lag your opponent's inputs instead? Why lag yours since that would completely screw you over?
I reckon rollback and prediction is pretty much every game dev's dream networking solution, ignoring complexity, resimulation cost and the requirement* for determinism. It works well on almost any network connection and still allows full client prediction. *You don't _really_ need determinism so long as you send state along with inputs. Personally state+input would be my preferred method for small-player-count arena/twitch shooters like Quake or Overwatch (where state data can be really small). Compare inputs as well as state and if either varies (within some margin to allow for floating point inaccuracy) rollback and resim. The more deterministic your game simulation, the better, but it doesn't need to be perfect (which is good because true determinism is actually kinda hard).
A very nice video, thank you. I have a couple of questions. 1. Since you only send input and no other game state, do you have to guarantee that every input packet will be delivered? Should a player expect a packet with the other player's input data at constant time intervals? And if it does not arrive what to do? Reuse the previous input? 2. How do you guarantee that everything is deterministc? Each player can play on a different CPU what could have implications on how floating-point numbers are rounded/truncated/combined/etc. This can lead to divergence between two players. 3. How to determine between the two players what frame they are in? A player that receives the other player's input state needs to know what frame index they are in, and what frame index the incoming input data was in when it was sent out. 4. When you have more than 2 players in p2p, do you sync each pair of players independently? 5. How are other objects simulated, like some physically-moving box knocked down? Is it responsibility of one player who sends state of that box to other players? 6. In the example you showed at the beginning of the video you demonstrated latency of 2-3 frames. But this latency can vary. How do you determine how much rollback to use? By constantly monitoring ping between the players?
Thanks! I'd recommend coming over to Discord (link in description) to discuss these questions, although, some of them are answered in later parts of the series. I'll attempt to answer a few here, within the limits of a UA-cam comment: 1. The addon is using a technique I call "the sliding window" to deal with packet loss - this is covered in one of the videos 2. The addon has tools to detect when there are state mismatches, but, yes, you do need to be very careful with floating point math. This is why I also created SG Physics 2D, which is a deterministic physics engine for Godot that uses fixed-point math 3. There is a sort of self-asserting loop at work here :-) If we start with the same state at roughly the same time, and I say the first tick is "tick 1" and the other client says their first tick is "tick 1", the other's input will arrive later and lead to the same subsequent state, making it actually true that they are the same tick. Might be easier to draw a picture showing how this works, but it kind of just works! I did an older vlog where I tested this out in detail 4. No state is transferred in "pure rollback". Everything must be deterministic, even random numbers (this specific case is covered later in the series), so everything will play out the same on all clients 5. As soon as input is received that doesn't match the input we predicted for that tick, we rollback to the tick before, inject the real input, and then simulate forward to return to the current tick. So, again, this is sort of self-defining. If the latency increases, we'll receive older input, and hence rollback further. If latency decreases, we'll receive newer input, and don't need to rollback quite as far Hope that helps!
Yes, there will be a Godot 4 version! I've been working for a few months on trying to bring SG Physics 2D (my deterministic physics engine) to Godot 4, and right after I finish that, I'm planning to move on to porting the rollback addon. Unfortunately, I'm stuck on some GDExtension bugs, but hopefully those will get worked out over the few couple months
Thanks 🙂 Yes, absolutely! In fact, you'll get much better predictions if you tailor them to your game, which is something we're going to cover in a later part in the series. But if you want to start messing with it now, take a look in the README on the GitLab project page: custom predictions are done via the _predict_remote_input() virtual method.
@@SnopekGames yeah! I'm curious how feasible is use external models. A hidden markov model for example. How hard is for you generate a dataset while creating this tutorial series? I would love messing with the data!
@@bioinfolucas5606 It'd be really interesting to see prediction done with a Hidden Markov model, but I don't think that's something I'm personally going to work on - I've got too much to do already! If this is something you want to explore, though, please come over to the Snopek Games discord (link on my website - UA-cam won't let me put it in a comment) - me and the folks there can help you out with any issues you may have getting started :-)
Yes. There's a NetworkAdaptor class that you can replace. The default uses Godot RPCs (which will work with the builtin ENet or WebSocket or WebRTC) but you can also make your own implementation to use any network transport (ex Steam, PlayFab, anything)
11:29 i'm curious about this part where it says that rollback and prediction would be unsuitable for 60+ player matches. 1. why would that be the case? 2. does anyone know how the large multiplayer tactical fps games like hell let loose and squad handle netcode? (100 player matches). i thought rollback and prediction would both be necessary for these; rollback for accurate-feeling shots and to minimise rubberbanding, prediction for a responsive feel.
There's a couple reasons: - In rollback, all clients have to simulate the entire match, which means simulating all 60 players and the full environment, rather than just the things near by - The more players you have, the more missed predictions there will be, meaning more rollbacks and artifacts. With even 6-8 players there can really start to be a lot of artifacts (which are also harder to hide with rollback) Most FPS synchronize state and use authoritative servers - rollback is rarely ever used with FPS's. Sometimes (because there is a complete lack of standardized terminology!) developers will refer to the way the server in an FPS attempts to work out the position of the player when they fired as "rollback", but that's actually something quite different. FPS netcode usually makes heavy use of prediction, though!
@@SnopekGames thanks. that's a bit clearer to me now. if i'm understanding correctly large FPS multiplayers typically use a scheme where all clients are seeing other players as they were slightly 'in the past' (from the servers point of view), which is different to what you're doing here which is using prediction to show the (best guess) of the other player's positions 'right now', and then fixing them with rollback as needed. in those FPS games the authoritative server 'rewinds' the game state when checking for bullet collisions, compensating for lag, so that players don't get the experience of a perfecly lined up shot missing the target. yeah i was thinking rollback referred to this but i appreciate it's a distinct thing. thanks.
Yes, absolutely! This is what the NetworkAdaptor class is for - you can use it to swap out the network transport for something else. I think someone may even have already done this with godot-steam, but I don't think they shared their code. It shouldn't be too hard to do in any case :-)
Hm, possibly! The challenge with a bullet hell is the large number of objects - you wouldn't want each bullet to be a node. Maybe if you had a single "BulletManager" node that handled the _network_process() and _load_state()/_save_state() on behalf of all the bullets? For rendering, you'd probably also want to interact directly with the VisualServer to draw the bullets (again, rather than having a node for each). If you attempt this, please let me know how it goes!
This is a great question! I haven't had a chance to try it, but since we've got rollback-aware deterministic random number generation, I think we could make NPCs that operate deterministically on all peers (after sharing a random seed) and not require any synchronization. In any case, rollback is more about making PvP fair and less about NPCs. Physics can be done so long as you have a deterministic physics engine. If you're doing 2D, check out SG Physics 2D :-)
I did, in fact, include that in an earlier version of this video! I have the slides and uncut video still, if someone wants to see them. And that second rollback does make for a much more interesting artifact. However, the original version of this video was ~26 minutes long, and that ended up being one of the things I cut in order to get it down to it's current ~16 minutes (which is still longer than I wanted it to be 🙂).
Rollback also could be used with determinism. Most games with deterministic logic are related with lockstep mechanism, but lockstep could be freezing if one of the clients have slow internet connection. Rollback with determinism is hard to make but if we use with ECS architecture it could be actually straightforward. All the game world could be divided with different world like different worlds for each clients. Each client world will have their own state and ticks and rollbacking will be applied for that client world only. There are ready solutions like Photon Engine Quantum (Proprietary) and ME.ECS (MIT) [next version: ME.BECS] but they have no proper documentations I don't know if Unity DOTS supports determinism
Heh, no, I was just naming examples of the sort of games that are a good fit for rollback that most people would know. :-) I could have said Brawlhalla, which does have rollback netcode, but far fewer people would have known what that is.
This man is absolitely the GOAT
Really nice 😀! I wish that serie was available when I started my project! I had to write my own system for Godot. It's really good tho, it makes me validate my comprehension of networking and the concepts I implented! Thank you!
Trying to retrofit Roblox with more FPS style rollback rather than fighting game rollback, but this series has been incredibly helpful!
Thanks :-)
Really good explanation of rollback netcode! Gonna go watch the second part!
Thanks!
Holy cow. What an incredible addon and series! I'm pretty noob as a game developer, but this series made me feel like I could probably pull off rollback for a highly competitive platform fighter if I want to, although I'm sure it won't be easy! And actually, I'm also glad I watched this series because it gave me a rough idea of how much work will actually be required to make rollback work in my game.
One thing I wanted to point out for other people considering rollback for their game, is that, at least as it is implemented in this (currently 11 part) tutorial series, this framework seems highly susceptible to cheating. I'm no expert on networking, but it seems to me like you could just write a script or find some way to purposefully lag your input uploads, meanwhile you download the opponent's perfectly streamlined inputs, and now your screen shows a game with 0 rollbacks with only 2 frames of input delay, and meanwhile your opponent is experiencing many artifacts and an effectively increased latency, giving you a huge reaction time advantage. Furthermore, it is reasonable for your opponent (who probably doesn't know anything about what's going on under the hood) to incorrectly assume that the cheater is also experiencing the same artifacts, and thus they may not be suspicious of any foul play. I'm sure there are many partial solutions to this problem, but it's at least something worth thinking about. And couldn't this situation arise even with no malicious intent?
You mentioned "fairness" as a positive aspect of rollback, but I wasn't really convinced by your brief explanation that this is the case. I'd love to hear more about why you think this is the case, or what you have to say in response to what I mentioned above about attempts to cheat in more competitive games.
The cheating wouldn't really work because of the rollback re-simulation. The way you describe the cheat seems backwards; don't you want to lag your opponent's inputs instead? Why lag yours since that would completely screw you over?
this is probably the best explanation on Rollback netcode i've seen 😁... Thank you so much, you're the best.. Suscribed!!
Great video! Been banging my head against this exact netcode wall and excited to watch more of these
I reckon rollback and prediction is pretty much every game dev's dream networking solution, ignoring complexity, resimulation cost and the requirement* for determinism. It works well on almost any network connection and still allows full client prediction.
*You don't _really_ need determinism so long as you send state along with inputs. Personally state+input would be my preferred method for small-player-count arena/twitch shooters like Quake or Overwatch (where state data can be really small). Compare inputs as well as state and if either varies (within some margin to allow for floating point inaccuracy) rollback and resim. The more deterministic your game simulation, the better, but it doesn't need to be perfect (which is good because true determinism is actually kinda hard).
Looking forward to implementing this. Thank you, oh wise one 🙏
A very nice video, thank you. I have a couple of questions.
1. Since you only send input and no other game state, do you have to guarantee that every input packet will be delivered? Should a player expect a packet with the other player's input data at constant time intervals? And if it does not arrive what to do? Reuse the previous input?
2. How do you guarantee that everything is deterministc? Each player can play on a different CPU what could have implications on how floating-point numbers are rounded/truncated/combined/etc. This can lead to divergence between two players.
3. How to determine between the two players what frame they are in? A player that receives the other player's input state needs to know what frame index they are in, and what frame index the incoming input data was in when it was sent out.
4. When you have more than 2 players in p2p, do you sync each pair of players independently?
5. How are other objects simulated, like some physically-moving box knocked down? Is it responsibility of one player who sends state of that box to other players?
6. In the example you showed at the beginning of the video you demonstrated latency of 2-3 frames. But this latency can vary. How do you determine how much rollback to use? By constantly monitoring ping between the players?
Thanks! I'd recommend coming over to Discord (link in description) to discuss these questions, although, some of them are answered in later parts of the series. I'll attempt to answer a few here, within the limits of a UA-cam comment:
1. The addon is using a technique I call "the sliding window" to deal with packet loss - this is covered in one of the videos
2. The addon has tools to detect when there are state mismatches, but, yes, you do need to be very careful with floating point math. This is why I also created SG Physics 2D, which is a deterministic physics engine for Godot that uses fixed-point math
3. There is a sort of self-asserting loop at work here :-) If we start with the same state at roughly the same time, and I say the first tick is "tick 1" and the other client says their first tick is "tick 1", the other's input will arrive later and lead to the same subsequent state, making it actually true that they are the same tick. Might be easier to draw a picture showing how this works, but it kind of just works! I did an older vlog where I tested this out in detail
4. No state is transferred in "pure rollback". Everything must be deterministic, even random numbers (this specific case is covered later in the series), so everything will play out the same on all clients
5. As soon as input is received that doesn't match the input we predicted for that tick, we rollback to the tick before, inject the real input, and then simulate forward to return to the current tick. So, again, this is sort of self-defining. If the latency increases, we'll receive older input, and hence rollback further. If latency decreases, we'll receive newer input, and don't need to rollback quite as far
Hope that helps!
Will there be a Godot 4 implementation and if yes when?
Or can i use this one for my project?
Yes, there will be a Godot 4 version! I've been working for a few months on trying to bring SG Physics 2D (my deterministic physics engine) to Godot 4, and right after I finish that, I'm planning to move on to porting the rollback addon. Unfortunately, I'm stuck on some GDExtension bugs, but hopefully those will get worked out over the few couple months
@@SnopekGames thank you so much for your response and work!
Nice introduction! Does Rollback netcode have support to custom predictions?
Thanks 🙂 Yes, absolutely! In fact, you'll get much better predictions if you tailor them to your game, which is something we're going to cover in a later part in the series. But if you want to start messing with it now, take a look in the README on the GitLab project page: custom predictions are done via the _predict_remote_input() virtual method.
@@SnopekGames yeah! I'm curious how feasible is use external models. A hidden markov model for example. How hard is for you generate a dataset while creating this tutorial series? I would love messing with the data!
@@bioinfolucas5606 It'd be really interesting to see prediction done with a Hidden Markov model, but I don't think that's something I'm personally going to work on - I've got too much to do already! If this is something you want to explore, though, please come over to the Snopek Games discord (link on my website - UA-cam won't let me put it in a comment) - me and the folks there can help you out with any issues you may have getting started :-)
Is this unrelated to the underlying network library and can be used with websockets for example in browser games?
Yes. There's a NetworkAdaptor class that you can replace. The default uses Godot RPCs (which will work with the builtin ENet or WebSocket or WebRTC) but you can also make your own implementation to use any network transport (ex Steam, PlayFab, anything)
@@SnopekGames Thank you so much, you're a legend!
11:29 i'm curious about this part where it says that rollback and prediction would be unsuitable for 60+ player matches.
1. why would that be the case?
2. does anyone know how the large multiplayer tactical fps games like hell let loose and squad handle netcode? (100 player matches). i thought rollback and prediction would both be necessary for these; rollback for accurate-feeling shots and to minimise rubberbanding, prediction for a responsive feel.
There's a couple reasons:
- In rollback, all clients have to simulate the entire match, which means simulating all 60 players and the full environment, rather than just the things near by
- The more players you have, the more missed predictions there will be, meaning more rollbacks and artifacts. With even 6-8 players there can really start to be a lot of artifacts (which are also harder to hide with rollback)
Most FPS synchronize state and use authoritative servers - rollback is rarely ever used with FPS's. Sometimes (because there is a complete lack of standardized terminology!) developers will refer to the way the server in an FPS attempts to work out the position of the player when they fired as "rollback", but that's actually something quite different. FPS netcode usually makes heavy use of prediction, though!
@@SnopekGames thanks. that's a bit clearer to me now.
if i'm understanding correctly large FPS multiplayers typically use a scheme where all clients are seeing other players as they were slightly 'in the past' (from the servers point of view), which is different to what you're doing here which is using prediction to show the (best guess) of the other player's positions 'right now', and then fixing them with rollback as needed.
in those FPS games the authoritative server 'rewinds' the game state when checking for bullet collisions, compensating for lag, so that players don't get the experience of a perfecly lined up shot missing the target. yeah i was thinking rollback referred to this but i appreciate it's a distinct thing. thanks.
that's awesome
Thanks! :-)
Hello, awesome tutorials. Would it be possible to handle the multiplayer with godot-steam instead of the godot multiplayer API?
Yes, absolutely! This is what the NetworkAdaptor class is for - you can use it to swap out the network transport for something else. I think someone may even have already done this with godot-steam, but I don't think they shared their code. It shouldn't be too hard to do in any case :-)
@@SnopekGames Awesome, cheers
Whoa! Awesome. Do you think it can be used in a multiplayer bullet hell in Godot? Thank you!
Hm, possibly! The challenge with a bullet hell is the large number of objects - you wouldn't want each bullet to be a node. Maybe if you had a single "BulletManager" node that handled the _network_process() and _load_state()/_save_state() on behalf of all the bullets? For rendering, you'd probably also want to interact directly with the VisualServer to draw the bullets (again, rather than having a node for each). If you attempt this, please let me know how it goes!
@@SnopekGames Thanks for your reply. For sure I'll let you know.
how would you simulate physics and npcs with rollback? It seems input based
This is a great question! I haven't had a chance to try it, but since we've got rollback-aware deterministic random number generation, I think we could make NPCs that operate deterministically on all peers (after sharing a random seed) and not require any synchronization. In any case, rollback is more about making PvP fair and less about NPCs. Physics can be done so long as you have a deterministic physics engine. If you're doing 2D, check out SG Physics 2D :-)
I think it will be better if you show illustration on tanks for the seconds rollback
I did, in fact, include that in an earlier version of this video! I have the slides and uncut video still, if someone wants to see them. And that second rollback does make for a much more interesting artifact. However, the original version of this video was ~26 minutes long, and that ended up being one of the things I cut in order to get it down to it's current ~16 minutes (which is still longer than I wanted it to be 🙂).
Is this addon getting updated for 4.0???
Yes! I've been super busy working on GDExtension, but porting this to Godot 4 is the next thing on my list
Rollback also could be used with determinism. Most games with deterministic logic are related with lockstep mechanism, but lockstep could be freezing if one of the clients have slow internet connection. Rollback with determinism is hard to make but if we use with ECS architecture it could be actually straightforward. All the game world could be divided with different world like different worlds for each clients. Each client world will have their own state and ticks and rollbacking will be applied for that client world only. There are ready solutions like Photon Engine Quantum (Proprietary) and ME.ECS (MIT) [next version: ME.BECS] but they have no proper documentations
I don't know if Unity DOTS supports determinism
Does this add-on support cross platform (mobile-desktop) connections?
Yes! It uses Godot's high level multiplayer API, which is cross platform
@@SnopekGames Awesome, thank you! I was never sure about that either
Was that a dig at smash ultimates netcode haha?
Heh, no, I was just naming examples of the sort of games that are a good fit for rollback that most people would know. :-) I could have said Brawlhalla, which does have rollback netcode, but far fewer people would have known what that is.
@@SnopekGames Smash Ultimate is lockstep, right? At least that's what it feels like when I've played online.