Grolo, your content is informed and well thought out. I enjoy looking behind the curtain of game systems with you, while you explain the ins and outs of server deployment and configurations . Thank you for the education and the time you spent making this videos. Can't wait for the next one.
Amazing explanation ! Just for information they just announced yesterday on the server meshing video that 1 stanton system is 700k entities at start and 3 million entities after 1 week.
A lot of people also forget that individual ec2 instances do not scale linearly in power and cost but exponentially. Once you go over certain common configurations, they get very expensive very fast. The trick is to find the sweet spot where you get the best RoI performance cost ratio. In my experience, most servers run out of memory way before they run out of CPU.
Loved this and past videos and love to see the passion you put in these. I am not an expert in tech (by my trade), just user and enthusiast. Understanding game dev just a little bit and now a bit of the systems behind it (incl. hardware) is great and gives me a way better idea of what the CIG dev team is going through in the long run. Thank you!
I am somewhat convinced that CIG themselves does not know yet the scaling properties of the server architecture. Something like this has never been done before and the system is not really functional yet. On top of that it might be necessary to scale peaks dynamically. It is gonna be very interesting what solutions they come up with and how this might impact online gaming in general. Because if this is going to be successful other companies will be encouraged to to something similar and the nay sayers will be quieter.
I think people underestimate the sheer scale of the backend tech they are building, if it succeeds it's going to be a bigger achievement than the game itself.
@@amineabdz indeed, it also doesn't' matter who achieves it first or who ends up finding better routes its in the long run a massive achievement and push forward for multiple industries even if the game ends up failing. Reality is most people have very limited knowledge about the basics of computers in general so even mentioning basic hardware or attempting to explain in laymen's terms what hardware does is already too much for people. It's a sad reality for those who work in these kinds of fields because people really are unable to comprehend the difficulty and technicality and effort that goes into these things. They just expect it to work and its the architects and engineers jobs to basically make that happen which creates a larger disconnect.
Nice video. I think their IT infrastructure costs are listed in their UK financials and are around 1.5 million last time I looked a few years ago. Might be worth a follow up pumping the latest of those back through this calculation. I think this demonstrates the value of on prem to them. A lot of engineers can be had for a few million pounds a year!
Appreciate the thoughts!! I was trying to outline how we might think about server costs and hardware come the release of 1.0, not today. But it may be interesting to compare to today's hardware & cost too!
Here's hoping it doesn't cost them about $72 per month per concurrent user! Or at least that the utilization per actively paying user is low. I guess if you assume a player is active on average around 10 hours a week (which would be pretty high) then an active player costs around $4 a month in hosting and it might be viable as long as the monetization makes sense. If $10 of the cost of the base game went towards hosting and players played on average 100 hours or less over their lifetime, then it might be feasible with just game sales. But if people want to be playing it more than that, it will need a subscription or constant sales of cosmetics/ships to stay afloat. These costs are always an interesting balancing act, thanks for doing a dive into it and making a good WAG.
Ah! That's better than Neptune IMO for the reasons I described in the video and the pitfalls of managed servers. That could reduce the operational cost (but raise administration cost, something I didn't even touch on). But most importantly, keep their options open.
@@grolo-af re the scaling, we heard from Chris R directly LAST September about this exactly. They reached their ingress cap and were going to go to vendor to find a solution. But clearly the issue is with their optimisation, hence RMQ this year to avoid all writes being direct to graph, and making some less urgent things queue
I wonder how, or if better server performance will improve the feeling. I can't tell it's the engine or the server performance or the synergy between those two. Is it even possible to make an FPS in that size responsive enough? I played some Arena (the FPS/Deathmatch mode) a few weeks ago, and ofc it's not Quake or even CoD. I don't expect that. But the hitreg was all over the place, Kills with 1-99 bullets. The player model movement not in sync - death animation playing seconds after the kill etc. So I wonder where the limitations are and what part the hardware will play. Wobbly ships that stutter around and explode. Or that way: How would the game feel if it was an offline game. What effects are caused by the server hardware, server-software/engine, or client-server connection. What effects/bugs/glitches will disappear in the future and what limitations will be there for the foreseeable future, no matter the hardware?
We'll know soon how it feels as an offline game. That's what Squadron 42 is, and we saw a pretty decent demo of it at the last Citizen Con. I do think most of the issues you're describing are related to server performance. To answer your question: yes, I think it's possible.
Wow! That’s awesome! A long time ago I got to share time on a mainframe. Very different and cool way of doing things. Would love to play with a modern one.
Agreed... on -prem doesn't make sense, at least right now. CIG has too much on its plate to worry about supporting multiple data centers all over the world.
Don't forget that there are parallel tasks that run on these servers, but also because this is based on cloud infrastructure... asynchronous! Things like physics are likely to run asynchronous to the other threads so you don't thread or spinlock waiting while you serve clients. In the world of Star Citizen the replication layer is the "database" where "game servers" each take control over a set of entities.
And I certainly hope "gamers" that sit out this entire video will understand why a subscription service is not all that strange... I know, but if you just try to explain the economics behind it... it would take an hour!
You're right of course. I simplify some things to convey things in a way that are easier to understand, and I try not to explain everything all at once. And therefore, it will always be easy for a technical person to come in and say "but what about" (I'm not saying YOU are doing that - I'm speaking generally here to explain to folks that may not get why I omit things). But one thing you learn as you go ever further into this career (or indeed likely most?) is that communication is often a barrier to progress & understanding. As the person with the technical understanding, it's _your_ responsibility to bridge that gap, and you can't do that for everything and everyone all at once, as the message needs contoured to the recipient(s) and the subject. Hopefully that resonates. But yes something like weather would likely be in another process altogether. Thinking of the replication layer as a database is a good way to explain it, though it's kind of inverted from most from the perspective of the DGS and client, in that data is being pushed in a constant stream, not really queried, unless you consider player "inputs" the query, at least on the player side. On the DGS side you just have dumb compute waiting for jobs to do basically : ).
@@grolo-af my understanding is that we, the clients, don't connect to the DGS either but the data is streamed from replication back and forth to the DGS that has authority. Star Citizen is also built heavily on top of ECS (Entity Component System), which adds a lot of overhead if you do runtime profiling. And besides that, we are on a 64bit coordinate system that also means at least a three fold of extra CPU cycles for every position... something that is usually optimized by packing 2x32, 4x16 or 8x8 into a single word by every game developer out there :)
@@sjoervanderploeg4340 Yeah, I'm not suggesting runtime profiling. I'm suggesting extensive profiling outside of the runtime env. You're correct about us not connecting to DGS directly. Not sure how clear I made that in this one, but in a previous video I talked extensively about that.
Thanks!! Ive neen wanting to apply at CIG. HA arch & low lvl systems. Is similiar to coding in '90s cause every nanosec & bit counts. Working in HA auto scaling complex system used for Big Data. Its easy for mgmt to set sacrifical deadlines and comput resources. If devs aren't listened too or given ability to communicate. Leading to sky rocketing costs and quadrupling the efforst to fix tech debt running in production env's.
If you apply, best of luck! They need those skills! I hear you on management and costs! Effificency is very easy to sacrifice, and it's also easy to hide, sadly. In some ways "clouds" have been a boon, in others a curse. When it's so easy to acquire more compute, its easier to be wasteful. In the "old days" server acquisition was harder, there was more incentive to clean up the code...
Interesting video with all the estimated costs.laid out. Not sure how CIG will be able to make enough profit from a subscription (which it seems they definitely will need). Especially when the subscription price will be limited to what customers will pay. Eve online subscription for comparison is £15.99 ish per month. But we will see.
Certainly curious. I'd like to see 'em get creative. Sell billboard space! Let our Org rent a huge display in Area 18. Let me as an individual join a concierge program that gets me access to a cool lounge in space ports... stuff like that. I think the market would gobble it up. Line out the door.
Thanks! I watched a lot of Yellowstone (starting from episode 1), and I also re-watched the entire LoTR movie trilogy. That was a solid 10 hours itself.
Nice job on collecting all the prices from AWS, their site gives me migranes. I would guess storage, I/O and network traffic could increase those cost by quite a lot. On the other hand they could negotiate a better price than the public one (especially if they endorse the Neptune DB). Still you might look at well above 10$/€ per player and month easily... Just for server cost... I dont see how this will work without a susbstantial subscription.
You could be right! There was another thread I just replied in asking about subs... I could easily see them doing an optional premium sub. Sell access to concierge lounges at space ports, or advertise your org on premium billboards.... that's the kind of stuff I'd be looking at. People would be lining up to pay. But you're right... the cost of running this is going to be high. But the cost will decrease over time.
@@timkreis8543 that is what i was thinking that $16 pm is pure server const. not office, not people, not development. that will be double or triple. That is a lot of skins one should buy each month, if that is their business model
One thing CIG has proven, is they're adept at financing this project : ). I'm not concerned about them figuring out how to financially support it. A subscription may be part of it. I'm not sure. I could easily see them doing a "premium subscription" where you get access to premium (but entirely superficial) in-game features..l like concierge lounges in space ports... that kind of thing :). Kind of like ESO has an optional,premium sub. People would line up at the door to pay it and help fund this thing for everyone I think.
Interesting. Thanks for sharing. Is this going to be a problem for CIG as the player base grows? We know that not all players do a session every day, most only in the weekends. So we know it only spikes in the weekend. And that is when the service is slow, with very low server tick(..) And thus a bad experience for the playerbase. Will it make sense to invest for more server time in those periods? It feels like they are just cramping the playerbase in to a tight set atm. If they don't/can't invest/expand it will be bleak for years to come it looks like.
In the modern, cloud, world infrastructure should be designed to be horizontally scalable. Ie automatically handles load and spins up VMs etc as required. At this point I’ve only watched 1/2 Riis vid so not sure if he goes into automation much.
@@beny9360 I understand(..) the mechanic. It's not about scalability but more about cost I'm guessing. There is only a curtain amount of power(..) per server, and it's clear at the weekends they don't add sufficient server power for the needs. Resulting in minimum service. If that is a cost thing, which it seems it is, then it will only get worse as more players join in the future and more features are made active. There are weekends when the server fps is 3-5 max on all shards, which is just about unplayable for most situations.
Well, right now they may not be auto scaling. First of all, they cannot scale the universe... it's bound to 1 server and 100 players. With 4.0 that will change. But beyond that, there's services sitting behind those game services... and I'm not sure if they're auto scaling yet either. It seems to me that concurrent player spikes negatively impact all servers and so there's some shared resources being overtaxes. All of this will need solved as outlined in my video. Is it a problem? I don't think so. No reason that can't solve it. I also don't expect it to be a cost problem. As I demonstrated, they can get the cost down to reasonable levels per player and maintain that. It's just about funding then which they've proven to be adept at : )
@@errgoth I can see a problem there, if you have tons of shards spinning up on weekend and then you need to scale back again, persistent stuff like base building will prevent you from going back too much.
I wonder if there is any precedent to having one DGS in the mesh be solely responsible for handling all of the AI in the shard. That way no other DGS has to worry about AI. This sounds like a nightmare of server communication and authority state handling so could be hard to implement, however could be good for performance, I don’t know.
The CTO recently gave an interview in french in which he addressed exactly this. I was going to discuss the translated text but it turns out their interviewing him on the next ISC on these topics so I'm going to wait and react to that and reference the translated text of the french interview as-needed. Basically what he said, is many of the sub services that service the game servers ARE game servers. Sounds like the game servers are heavily feature flagged to me. We'll talk about this soon :)
based on the general analysis here it would take (roughly) about 100 Million/year for a consistent player base of 200,000 people. The number of players that have signed up is 5 million. Is there a scaling efficiency that we are missing in this analysis, similar to how efficiencies of scale reduce the cost of commodity production like in the chemical industry for example? It seems to me CIG needs to find another 10 to 1 efficiency to make this viable. Is it possible they have found that efficiency as part of their development? CIG knows how to do this type of financial analysis as well, and I doubt they would going full steam ahead with a financial prognosis that would be indicated here; is it possible that cost had a hand in killing their previous false starts on server meshing? How might you envision that another 10 to 1 efficiency increase might be possible. Thanks for the great video.
Hey really appreciate the comment, question, and thoughts! I think assuming an average peak of 200k with just 5 million total players may be high? EVE Online had 9ish million subscribers with an average peak of 50k. If we make the math a little easier, maybe 200k peak avg would entail a player base of like... 40 million? 40 million players would have a much easier time of generating 100m per year that 5m would. But even 5 million players would probably average $20 per player per year to fund that $100m cost. And probably quite a bit more than $20 / yr to make the company profitable! Numbers scale quick :) Also keep in mind, the numbers I put forth are WAGs! I could be over/under estimating quite a bit which would skew these numbers... but even if my guess is close to the mark, the numbers here seem very feasible to me. Whatchya think?
@@grolo-af Interesting, that first part might be where I went wrong given the landscape I was painting in my head. I was unaware that it is potentially more realistic that these peak usage periods would only draw in between 0.5 to 1% of the player base at a given time. I was thinking CIG would have to support 10-20% of the player base at peak play periods, and asking from that vantage point. This would account for that 10-1 ratio I was searching for that makes their financial accounting make more sense to me.
@@grolo-af Thanks a lot for this video. Really instructive, what a wonderful project to follow. When we talk about costs, we also need to keep in mind that there is also the salary of the employees at CIG (I think they are around 1000?) - their offices etc. I am wondering what will be the business model at the release. Can they continue to sell ships (that will release alongside new features)? I dont think they can ask for a mandatory subscription anymore as it was never part of their plan (even if I would like it!).
Pick one programming language and go deep on it. Build a service. Deploy and manage it. That's a starting point. Your networking knowledge will be invaluable. Begin to stitch those two worlds together. There's no short cut. Takes time. Stick with it.
As much as I hope Server meshing succeeds I do think it is three to four years off to be best it can be. My biggest hope for CIG is to be able to 500 people into one Shard at 4.0 and think that is very achievable for CIG to do and will be best thing they can do at this stage. Then build from there.
Yeah since SQ42 is announced for 2026 I would simply say 1.0 is at least two years later and Server meshing is one of the most complex and critical parts that cannot be perfect until it goes live because of scaling.
@@caintindal1671 sadly they can't really take it slow, time isn't in their favor to the common person who has no real understanding of what's happening and even those who worked in the field who also undermine the development such as Thor whose generally a fairly intelligent person.
@@yulfine1688 if they rush it it will fail having 500 on the server with 4.0 will be a big win for first time Orgs that have over 100 people will able to work together, Having 6 Orgs could fight each other in PYRO. First time Human Pirates will have a targets and the first time Pirates will have to think on who they hit as they will have to think. If all goes well they get it up to 1000 by the end of 2025 know one at this stage knows if this will work even though the y have had 1500 on a shard so far. If they go to fast people will be more pissed than what they are right now.
@caintindal1671 agreed but reality is people are stupid and that's why so much effort has gone into 4.0 and why they put other features on hold to focus further on sever meshing. Granted they may delay 4.0 further to early next year which would send people into an absolute frenzy like the mindless fools they are.
This is a great question and one that I'll spend some more time talking about in my next video which will relate directly to this question (it's going to be a reaction to the upcoming interview of their CTO on ISC who will be talking about the backend). Short answer: it's possible still that they'll deliver something... but it won't be a copy of the PU. It'll be something much smaller, is my guess.
An IBM Power Server is what SC needs for the backend, x86 is limited ;) The IBM® Power® E1080 is the most powerful and scalable server of the IBM Power portfolio. The E1080 supports up to 240 Power10 processor cores and up to 64 TB memory, and is designed to run AIX®, IBM i and Linux®.
@@grolo-af You wouldn't believe I've got these offered for FREE. A decommissioned one. There are "small" 4-8U Power systems. You can jank out RAM or a CPU while its running, one of the coolest features :p
Ive just stumbled across this. and i dont have any better information or knowledge.. I just play lol My real question is Why do these game servers need to do SO much calculation and have so much power.. My local personal computer needs to do it sure. Needs to render all the objects and place them, calculate gravity ship/player movement.. Why does all that work need to happen on the server too? Why not just send location information updates around from player to player instead of calculating it in the server as well? It just seems very wasteful and expensive to have so much stuff calculated and/or handled by the server.. Now admittedly im ignorant about what IS needed and what ISNT needed.. But for example we can do simple math and time calculations to follow the path of planets in the real world universe.. Running an active simulation on the server is a massive waste of resources. No?
I have little to no knowledge about your question but From what I gather CIG is trying to get everything they can server side but haven’t reached that level yet and are kinda a mesh of communication between the server and client (more then would be necessary) depending on what is being calculated… I am curious as to the answer to your question
Great question! What you're asking about is called "client authority" where a server trusts a client to perform the calculations. What CIG wants is "Server authority". There's many reasons. The biggest is likely exploitation. Clients love to cheat :). Another is resolving differences in calculations between clients. They will all calculate slightly different results (due to latency and time drift and etc)... who wins? I could go on... but you get the picture. Surprisingly, it also has tremendous scaling challenges. It's much harder to scale the edge of a network (client connection to game server) than the network in your data center. If you want an example of client authority look at Elite Dangerous. It's doing what you're describing. But the max number of players in a shard is around 30 or 60 I believe?
@@grolo-af Ah I see. That's pretty enlightening. Though I would wonder if there can be some compromising to lighten the load on the servers because we as consumers are forced to install Easy Anti-Cheat on our local machines anyway. So perhaps some of it can be offloaded to the client and save quite a bit of money? All this to say I hope they make good decisions and not burn money where they don't have to.
28:20 $12,300 a day X 365 days = $4,489,500 PER YEAR for 8000 PLAYERS. = $516.75 per year per player (for 8000 players) = $43 per Month per 8000 Players (for 8000 Players) That's all I heard!!! Fact- Even with more players the server cost will increase to handle the load. There is NO Balance. There will be No Server Meshing. IT ALL COMES TO MONEY!!!! Thats all I heard. Ain't no way!!!!!
@grolo-af just a rough estimate of (if at all) the PLAYERS were to take on the costly role of PAYING FOR SERVER ACCESS (we obviously are paying for it indirectly at this present time). This is done as a loose example. YOU jumped directly into the financial aspect of running a server (FOR WHAT???, maybe to give an monetary excuse of why servers in Star Citizen are so horrible at this present time). So doing the math by YOUR EXAMPLE GIVEN, if the players are absorbing the cost of paying for servers to run SC (which we obviously are thru ship sales) then BY YOUR EXAMPLE, this is a breakdown of what (8000 players would pay) Directly or indirectly. Where else is the money to run the servers coming from Grolo???
I 'd also like to point out so that everyone understands.... You also stated that the server load will hit its peak and plateau. It is atr this point where present day technology will not be able handle player numbers per server load. Guess what doesn't plateau.... COST, MONEY THE PLAYERS PAY TO ACHIEVE A THEORETICALLY PROVEN UNACHIEVABLE GOAL. You already know this, by your example. Finance already knows this. But THE PLAYERS are BEING told we will achieve XXXXX amount of players per server. SO THE SHIP SALES CONTINUE. If you're presenting how servers work to us. You've been cleared to do so. So Chris Roberts knows this in advance. Believe me bro. I understand being in a job where boss wants you as an ant, to push a boulder up a hill and just make it work. Would appreciate it if my comments are not scrubbed. The Star Citizen community is a very intelligent community. But a lot of us are tired of the money hungry feasting off our dreams, desires and passion for gaming.
@@MurkMercy what are you even going on about? He doesn't work for CIG or any of those companies nor has any relations with them. Yes the main funding for the game is currently via ship sales and other smaller backers who invested, its why likely by 1.0 a lot of things will change, the ship store isn't going away probably ever as will other things like skins etc, they also have their two different subscription options many people pay into among other things.
Grolo, your content is informed and well thought out. I enjoy looking behind the curtain of game systems with you, while you explain the ins and outs of server deployment and configurations . Thank you for the education and the time you spent making this videos. Can't wait for the next one.
You rock. Really appreciate that!! : ] Still pretty new to this. Really awesome to hear when people dig it!
Amazing explanation ! Just for information they just announced yesterday on the server meshing video that 1 stanton system is 700k entities at start and 3 million entities after 1 week.
Aye! I posted a reaction to that. Was pretty pumped about the ISC topic this week : )
ua-cam.com/video/IRzlTcloEvo/v-deo.html
A lot of people also forget that individual ec2 instances do not scale linearly in power and cost but exponentially. Once you go over certain common configurations, they get very expensive very fast. The trick is to find the sweet spot where you get the best RoI performance cost ratio.
In my experience, most servers run out of memory way before they run out of CPU.
As a fellow developer just starting out in the industry, I really enjoy your content. Very informative.
Awesome to hear! And best of luck on your journey! Doesn't seem like as long ago as it actually is that I was at the start of it too... was fun!
Loved this and past videos and love to see the passion you put in these. I am not an expert in tech (by my trade), just user and enthusiast. Understanding game dev just a little bit and now a bit of the systems behind it (incl. hardware) is great and gives me a way better idea of what the CIG dev team is going through in the long run. Thank you!
You're exactly the type of person I'm trying to speak to in these. Glad you enjoyed it!! Thanks for taking the time to let me know. Means a LOT!
Great video... this Thursday there will a video of server meshing news on the offical Star Citizen channel, can't wait to learn about it!
Aye!! I will be reacting to it immediately after it's posted. Can't wait.
I am somewhat convinced that CIG themselves does not know yet the scaling properties of the server architecture. Something like this has never been done before and the system is not really functional yet. On top of that it might be necessary to scale peaks dynamically. It is gonna be very interesting what solutions they come up with and how this might impact online gaming in general.
Because if this is going to be successful other companies will be encouraged to to something similar and the nay sayers will be quieter.
Agreed; I'm watching closely - eager to see how it all plays out!
I think people underestimate the sheer scale of the backend tech they are building, if it succeeds it's going to be a bigger achievement than the game itself.
@@amineabdz indeed, it also doesn't' matter who achieves it first or who ends up finding better routes its in the long run a massive achievement and push forward for multiple industries even if the game ends up failing.
Reality is most people have very limited knowledge about the basics of computers in general so even mentioning basic hardware or attempting to explain in laymen's terms what hardware does is already too much for people.
It's a sad reality for those who work in these kinds of fields because people really are unable to comprehend the difficulty and technicality and effort that goes into these things.
They just expect it to work and its the architects and engineers jobs to basically make that happen which creates a larger disconnect.
Nice video. I think their IT infrastructure costs are listed in their UK financials and are around 1.5 million last time I looked a few years ago. Might be worth a follow up pumping the latest of those back through this calculation. I think this demonstrates the value of on prem to them. A lot of engineers can be had for a few million pounds a year!
Appreciate the thoughts!! I was trying to outline how we might think about server costs and hardware come the release of 1.0, not today. But it may be interesting to compare to today's hardware & cost too!
It's Grolo and I click!
o7
Here's hoping it doesn't cost them about $72 per month per concurrent user! Or at least that the utilization per actively paying user is low. I guess if you assume a player is active on average around 10 hours a week (which would be pretty high) then an active player costs around $4 a month in hosting and it might be viable as long as the monetization makes sense. If $10 of the cost of the base game went towards hosting and players played on average 100 hours or less over their lifetime, then it might be feasible with just game sales. But if people want to be playing it more than that, it will need a subscription or constant sales of cosmetics/ships to stay afloat.
These costs are always an interesting balancing act, thanks for doing a dive into it and making a good WAG.
Indeed they are. And of course. Hope we get to see how close it is to the mark someday : )
neo4j as far as we've determined from some specifics CIG have shared in the past, for the entity graph
Ah! That's better than Neptune IMO for the reasons I described in the video and the pitfalls of managed servers. That could reduce the operational cost (but raise administration cost, something I didn't even touch on). But most importantly, keep their options open.
@@grolo-af re the scaling, we heard from Chris R directly LAST September about this exactly. They reached their ingress cap and were going to go to vendor to find a solution. But clearly the issue is with their optimisation, hence RMQ this year to avoid all writes being direct to graph, and making some less urgent things queue
I wonder how, or if better server performance will improve the feeling. I can't tell it's the engine or the server performance or the synergy between those two. Is it even possible to make an FPS in that size responsive enough? I played some Arena (the FPS/Deathmatch mode) a few weeks ago, and ofc it's not Quake or even CoD. I don't expect that. But the hitreg was all over the place, Kills with 1-99 bullets. The player model movement not in sync - death animation playing seconds after the kill etc.
So I wonder where the limitations are and what part the hardware will play. Wobbly ships that stutter around and explode. Or that way: How would the game feel if it was an offline game. What effects are caused by the server hardware, server-software/engine, or client-server connection.
What effects/bugs/glitches will disappear in the future and what limitations will be there for the foreseeable future, no matter the hardware?
We'll know soon how it feels as an offline game. That's what Squadron 42 is, and we saw a pretty decent demo of it at the last Citizen Con. I do think most of the issues you're describing are related to server performance. To answer your question: yes, I think it's possible.
@@grolo-af Arena Commander is the same; it runs really well compared to the PU.
Love your videos, can't enough of this topic and you explain it very well!
Well I can't get enough of the encouragement; really appreciate it! : )
Tremendous. If you ever get tired of working in the industry you would make a great educator.
I have no idea how I got here. But this is certainly fascinating.
Well I'm glad you got here! Welcome! And thank you : D
Thanks for the deep dive! Super fascinating.
For sure! Thanks for watch'n!
first.
I've been playing SC a lot after watching your streams. Bored from the EVE spreadsheets. Hope to see more lives!
Aw man, glad to here it!!! o7
Again, this is super appreciated! I love these sorts of spherical cow estimations when it tips over into real world numbers.
Haha, I hadn't heard that phrase before. That's a really good way of saying it :)
Well done sir. I work on IBM mainframes so all this distributed platform info is very interesting.
Wow! That’s awesome! A long time ago I got to share time on a mainframe. Very different and cool way of doing things. Would love to play with a modern one.
Awesome, thank you for your hard work and presenting it in an interesting and understandable way
You bet! "My pleasure" feels cliche, but I really do enjoy it. And I enjoy the discussions that follow just as much!
Might have missed it but what is minimum upload per player for stable? Back in the day swbf original required 2mb per slot...?
Glad to hear you're recovering sir!
Agreed... on -prem doesn't make sense, at least right now. CIG has too much on its plate to worry about supporting multiple data centers all over the world.
Great analysis, your insight is much appreciated.
i am pretty shure if they get this to work we gonna be 200k plus online at all times
Could very well be! Their operations captain will be yell’n, “Deploy the servers!!”
I think that would be what we like to call "a good problem to have"
Don't forget that there are parallel tasks that run on these servers, but also because this is based on cloud infrastructure... asynchronous!
Things like physics are likely to run asynchronous to the other threads so you don't thread or spinlock waiting while you serve clients.
In the world of Star Citizen the replication layer is the "database" where "game servers" each take control over a set of entities.
And I certainly hope "gamers" that sit out this entire video will understand why a subscription service is not all that strange...
I know, but if you just try to explain the economics behind it... it would take an hour!
You're right of course. I simplify some things to convey things in a way that are easier to understand, and I try not to explain everything all at once. And therefore, it will always be easy for a technical person to come in and say "but what about" (I'm not saying YOU are doing that - I'm speaking generally here to explain to folks that may not get why I omit things). But one thing you learn as you go ever further into this career (or indeed likely most?) is that communication is often a barrier to progress & understanding. As the person with the technical understanding, it's _your_ responsibility to bridge that gap, and you can't do that for everything and everyone all at once, as the message needs contoured to the recipient(s) and the subject. Hopefully that resonates.
But yes something like weather would likely be in another process altogether. Thinking of the replication layer as a database is a good way to explain it, though it's kind of inverted from most from the perspective of the DGS and client, in that data is being pushed in a constant stream, not really queried, unless you consider player "inputs" the query, at least on the player side. On the DGS side you just have dumb compute waiting for jobs to do basically : ).
@@grolo-af my understanding is that we, the clients, don't connect to the DGS either but the data is streamed from replication back and forth to the DGS that has authority.
Star Citizen is also built heavily on top of ECS (Entity Component System), which adds a lot of overhead if you do runtime profiling.
And besides that, we are on a 64bit coordinate system that also means at least a three fold of extra CPU cycles for every position... something that is usually optimized by packing 2x32, 4x16 or 8x8 into a single word by every game developer out there :)
@@sjoervanderploeg4340 Yeah, I'm not suggesting runtime profiling. I'm suggesting extensive profiling outside of the runtime env.
You're correct about us not connecting to DGS directly. Not sure how clear I made that in this one, but in a previous video I talked extensively about that.
Glad you're back and feeling better!
Thanks!!
Ive neen wanting to apply at CIG. HA arch & low lvl systems. Is similiar to coding in '90s cause every nanosec & bit counts.
Working in HA auto scaling complex system used for Big Data. Its easy for mgmt to set sacrifical deadlines and comput resources. If devs aren't listened too or given ability to communicate. Leading to sky rocketing costs and quadrupling the efforst to fix tech debt running in production env's.
If you apply, best of luck! They need those skills!
I hear you on management and costs! Effificency is very easy to sacrifice, and it's also easy to hide, sadly. In some ways "clouds" have been a boon, in others a curse. When it's so easy to acquire more compute, its easier to be wasteful. In the "old days" server acquisition was harder, there was more incentive to clean up the code...
4milli entites will be the lower end...players are MESSY...
heeeyooooooooooo
Thanks for the deeper dive on my questions! Checking out the SC org now ;)
Awesome!! Ping me on Discord if you get in. I'm grolo there too : )
don't forget right now they are hosing different servers/location US, ASIA,EU... etc, + PTU + evocati
Yeah I didn't mention regions in this video... but they'll basically replicate the entire stack in each of those places, at different scales.
Interesting video with all the estimated costs.laid out. Not sure how CIG will be able to make enough profit from a subscription (which it seems they definitely will need). Especially when the subscription price will be limited to what customers will pay. Eve online subscription for comparison is £15.99 ish per month. But we will see.
Certainly curious. I'd like to see 'em get creative. Sell billboard space! Let our Org rent a huge display in Area 18. Let me as an individual join a concierge program that gets me access to a cool lounge in space ports... stuff like that. I think the market would gobble it up. Line out the door.
Glad the procedure went well! What shows did you binge?
Thanks! I watched a lot of Yellowstone (starting from episode 1), and I also re-watched the entire LoTR movie trilogy. That was a solid 10 hours itself.
Nice job on collecting all the prices from AWS, their site gives me migranes.
I would guess storage, I/O and network traffic could increase those cost by quite a lot. On the other hand they could negotiate a better price than the public one (especially if they endorse the Neptune DB).
Still you might look at well above 10$/€ per player and month easily... Just for server cost... I dont see how this will work without a susbstantial subscription.
You could be right! There was another thread I just replied in asking about subs... I could easily see them doing an optional premium sub. Sell access to concierge lounges at space ports, or advertise your org on premium billboards.... that's the kind of stuff I'd be looking at. People would be lining up to pay. But you're right... the cost of running this is going to be high. But the cost will decrease over time.
@@grolo-afI would love to see a subscription mode with no pay to win ships or items
@@timkreis8543 I'm all in on the sub model myself.
can GIC survive without a subscription model?
I would be extremly suprised if they could. Nobody wants to hear that, but the game is very expensive to run.
@@timkreis8543 that is what i was thinking that $16 pm is pure server const. not office, not people, not development. that will be double or triple.
That is a lot of skins one should buy each month, if that is their business model
One thing CIG has proven, is they're adept at financing this project : ). I'm not concerned about them figuring out how to financially support it. A subscription may be part of it. I'm not sure. I could easily see them doing a "premium subscription" where you get access to premium (but entirely superficial) in-game features..l like concierge lounges in space ports... that kind of thing :). Kind of like ESO has an optional,premium sub. People would line up at the door to pay it and help fund this thing for everyone I think.
TIL that WAG stands for Wild Ass Guess.
: ) One of my favorite TLA’s!! There’s a lot of them to choose from in this field…
Interesting. Thanks for sharing.
Is this going to be a problem for CIG as the player base grows?
We know that not all players do a session every day, most only in the weekends. So we know it only spikes in the weekend. And that is when the service is slow, with very low server tick(..) And thus a bad experience for the playerbase.
Will it make sense to invest for more server time in those periods? It feels like they are just cramping the playerbase in to a tight set atm. If they don't/can't invest/expand it will be bleak for years to come it looks like.
In the modern, cloud, world infrastructure should be designed to be horizontally scalable. Ie automatically handles load and spins up VMs etc as required. At this point I’ve only watched 1/2 Riis vid so not sure if he goes into automation much.
@@beny9360 I understand(..) the mechanic. It's not about scalability but more about cost I'm guessing. There is only a curtain amount of power(..) per server, and it's clear at the weekends they don't add sufficient server power for the needs. Resulting in minimum service. If that is a cost thing, which it seems it is, then it will only get worse as more players join in the future and more features are made active. There are weekends when the server fps is 3-5 max on all shards, which is just about unplayable for most situations.
Well, right now they may not be auto scaling. First of all, they cannot scale the universe... it's bound to 1 server and 100 players. With 4.0 that will change. But beyond that, there's services sitting behind those game services... and I'm not sure if they're auto scaling yet either. It seems to me that concurrent player spikes negatively impact all servers and so there's some shared resources being overtaxes.
All of this will need solved as outlined in my video. Is it a problem? I don't think so. No reason that can't solve it. I also don't expect it to be a cost problem. As I demonstrated, they can get the cost down to reasonable levels per player and maintain that. It's just about funding then which they've proven to be adept at : )
@@errgoth I can see a problem there, if you have tons of shards spinning up on weekend and then you need to scale back again, persistent stuff like base building will prevent you from going back too much.
Upon the 1.0 release you won't be spinning up new shards, you'll be spinning up additional capacity for existing shards.
I wonder if there is any precedent to having one DGS in the mesh be solely responsible for handling all of the AI in the shard. That way no other DGS has to worry about AI. This sounds like a nightmare of server communication and authority state handling so could be hard to implement, however could be good for performance, I don’t know.
The CTO recently gave an interview in french in which he addressed exactly this. I was going to discuss the translated text but it turns out their interviewing him on the next ISC on these topics so I'm going to wait and react to that and reference the translated text of the french interview as-needed. Basically what he said, is many of the sub services that service the game servers ARE game servers. Sounds like the game servers are heavily feature flagged to me. We'll talk about this soon :)
Pog
based on the general analysis here it would take (roughly) about 100 Million/year for a consistent player base of 200,000 people. The number of players that have signed up is 5 million. Is there a scaling efficiency that we are missing in this analysis, similar to how efficiencies of scale reduce the cost of commodity production like in the chemical industry for example? It seems to me CIG needs to find another 10 to 1 efficiency to make this viable. Is it possible they have found that efficiency as part of their development? CIG knows how to do this type of financial analysis as well, and I doubt they would going full steam ahead with a financial prognosis that would be indicated here; is it possible that cost had a hand in killing their previous false starts on server meshing?
How might you envision that another 10 to 1 efficiency increase might be possible. Thanks for the great video.
maybe, CIG is just betting on this phenomena you point out here 1:04:46 (for next minute)
Hey really appreciate the comment, question, and thoughts! I think assuming an average peak of 200k with just 5 million total players may be high? EVE Online had 9ish million subscribers with an average peak of 50k. If we make the math a little easier, maybe 200k peak avg would entail a player base of like... 40 million? 40 million players would have a much easier time of generating 100m per year that 5m would. But even 5 million players would probably average $20 per player per year to fund that $100m cost. And probably quite a bit more than $20 / yr to make the company profitable!
Numbers scale quick :) Also keep in mind, the numbers I put forth are WAGs! I could be over/under estimating quite a bit which would skew these numbers... but even if my guess is close to the mark, the numbers here seem very feasible to me. Whatchya think?
@@grolo-af Interesting, that first part might be where I went wrong given the landscape I was painting in my head. I was unaware that it is potentially more realistic that these peak usage periods would only draw in between 0.5 to 1% of the player base at a given time. I was thinking CIG would have to support 10-20% of the player base at peak play periods, and asking from that vantage point. This would account for that 10-1 ratio I was searching for that makes their financial accounting make more sense to me.
@@grolo-af Thanks a lot for this video. Really instructive, what a wonderful project to follow.
When we talk about costs, we also need to keep in mind that there is also the salary of the employees at CIG (I think they are around 1000?) - their offices etc.
I am wondering what will be the business model at the release. Can they continue to sell ships (that will release alongside new features)? I dont think they can ask for a mandatory subscription anymore as it was never part of their plan (even if I would like it!).
What would be your advice to a network engineer wanting to become a System Architect?
Pick one programming language and go deep on it. Build a service. Deploy and manage it. That's a starting point. Your networking knowledge will be invaluable. Begin to stitch those two worlds together. There's no short cut. Takes time. Stick with it.
What server are you on on Ashes?
We started on Lyneth, but just moved to Castus (fresh start)
As much as I hope Server meshing succeeds I do think it is three to four years off to be best it can be. My biggest hope for CIG is to be able to 500 people into one Shard at 4.0 and think that is very achievable for CIG to do and will be best thing they can do at this stage. Then build from there.
Yeah since SQ42 is announced for 2026 I would simply say 1.0 is at least two years later and Server meshing is one of the most complex and critical parts that cannot be perfect until it goes live because of scaling.
@@madrooky1398 Why I dont want them to rush and get maximum players in the game I rather see take it slow and learn.
@@caintindal1671 sadly they can't really take it slow, time isn't in their favor to the common person who has no real understanding of what's happening and even those who worked in the field who also undermine the development such as Thor whose generally a fairly intelligent person.
@@yulfine1688 if they rush it it will fail having 500 on the server with 4.0 will be a big win for first time Orgs that have over 100 people will able to work together, Having 6 Orgs could fight each other in PYRO. First time Human Pirates will have a targets and the first time Pirates will have to think on who they hit as they will have to think. If all goes well they get it up to 1000 by the end of 2025 know one at this stage knows if this will work even though the y have had 1500 on a shard so far. If they go to fast people will be more pissed than what they are right now.
@caintindal1671 agreed but reality is people are stupid and that's why so much effort has gone into 4.0 and why they put other features on hold to focus further on sever meshing.
Granted they may delay 4.0 further to early next year which would send people into an absolute frenzy like the mindless fools they are.
one of the kickstarter promises was private servers do you think thats still realistic?
Not self-hosted, no way that will happen (anytime soon).
An Arena server sure, but the entire PU nope..
I could imagine them renting out heavily scaled down shards on AWS... but your wallet isnt going to like that at all.
This is a great question and one that I'll spend some more time talking about in my next video which will relate directly to this question (it's going to be a reaction to the upcoming interview of their CTO on ISC who will be talking about the backend). Short answer: it's possible still that they'll deliver something... but it won't be a copy of the PU. It'll be something much smaller, is my guess.
He's back!
I'm back! Who did it better Terminator or Grolo
lol
this is very intersting :)
Great vid
Thank you!
Good Job!
An IBM Power Server is what SC needs for the backend, x86 is limited ;)
The IBM® Power® E1080 is the most powerful and scalable server of the IBM Power portfolio. The E1080 supports up to 240 Power10 processor cores and up to 64 TB memory, and is designed to run AIX®, IBM i and Linux®.
I can't comment as I know almost nothing about this class of hardware.... but what I can say, is I'd love to play with it!
@@grolo-af You wouldn't believe I've got these offered for FREE. A decommissioned one. There are "small" 4-8U Power systems. You can jank out RAM or a CPU while its running, one of the coolest features :p
Awesome 👌
PiP video is in front of the writing
Yeah I noticed that :(. I’ll do better next time. Sorry!!
Ive just stumbled across this. and i dont have any better information or knowledge.. I just play lol
My real question is Why do these game servers need to do SO much calculation and have so much power.. My local personal computer needs to do it sure. Needs to render all the objects and place them, calculate gravity ship/player movement.. Why does all that work need to happen on the server too? Why not just send location information updates around from player to player instead of calculating it in the server as well? It just seems very wasteful and expensive to have so much stuff calculated and/or handled by the server.. Now admittedly im ignorant about what IS needed and what ISNT needed.. But for example we can do simple math and time calculations to follow the path of planets in the real world universe.. Running an active simulation on the server is a massive waste of resources. No?
I have little to no knowledge about your question but From what I gather CIG is trying to get everything they can server side but haven’t reached that level yet and are kinda a mesh of communication between the server and client (more then would be necessary) depending on what is being calculated… I am curious as to the answer to your question
Great question! What you're asking about is called "client authority" where a server trusts a client to perform the calculations. What CIG wants is "Server authority". There's many reasons. The biggest is likely exploitation. Clients love to cheat :). Another is resolving differences in calculations between clients. They will all calculate slightly different results (due to latency and time drift and etc)... who wins? I could go on... but you get the picture. Surprisingly, it also has tremendous scaling challenges. It's much harder to scale the edge of a network (client connection to game server) than the network in your data center. If you want an example of client authority look at Elite Dangerous. It's doing what you're describing. But the max number of players in a shard is around 30 or 60 I believe?
@@grolo-af
Ah I see. That's pretty enlightening. Though I would wonder if there can be some compromising to lighten the load on the servers because we as consumers are forced to install Easy Anti-Cheat on our local machines anyway. So perhaps some of it can be offloaded to the client and save quite a bit of money?
All this to say I hope they make good decisions and not burn money where they don't have to.
Star Citizen Has Actual Servers??
I Swore They Only Had A Few Potatoes Hooked Up To A Single GTX 900.
That's what it feels like.
lol; good one : ). I mean look… I put “future” in the title. You could be right currently!
Who's the person to look for when it comes to the game engine?
The game engine sitting in the client? I have no idea.... good question. I'll look around and see if someone stands out.
What aspect are you interested in?
@@entityself3104 Usually, I look at netcode, but at this instance, it's resource management. Why is there such bad GC in the game engine.
@@kVidStream, are you referring to the items in the world not being removed, or what programmers mean by GC (obj/mem management)?
@@entityself3104 yes; removal of handle in memory (therefore cannot be referenced anymore) but the resource is still not returned to the OS.
28:20 $12,300 a day X 365 days = $4,489,500 PER YEAR for 8000 PLAYERS.
= $516.75 per year per player (for 8000 players)
= $43 per Month per 8000 Players (for 8000 Players)
That's all I heard!!!
Fact- Even with more players the server cost will increase to handle the load. There is NO Balance. There will be No Server Meshing.
IT ALL COMES TO MONEY!!!!
Thats all I heard. Ain't no way!!!!!
How do you get $43 per month? $43 for what?
@grolo-af just a rough estimate of (if at all) the PLAYERS were to take on the costly role of PAYING FOR SERVER ACCESS (we obviously are paying for it indirectly at this present time). This is done as a loose example. YOU jumped directly into the financial aspect of running a server (FOR WHAT???, maybe to give an monetary excuse of why servers in Star Citizen are so horrible at this present time). So doing the math by YOUR EXAMPLE GIVEN, if the players are absorbing the cost of paying for servers to run SC (which we obviously are thru ship sales) then BY YOUR EXAMPLE, this is a breakdown of what (8000 players would pay) Directly or indirectly. Where else is the money to run the servers coming from Grolo???
@grolo-af or shorter answer MATH.
I 'd also like to point out so that everyone understands.... You also stated that the server load will hit its peak and plateau. It is atr this point where present day technology will not be able handle player numbers per server load. Guess what doesn't plateau.... COST, MONEY THE PLAYERS PAY TO ACHIEVE A THEORETICALLY PROVEN UNACHIEVABLE GOAL. You already know this, by your example. Finance already knows this. But THE PLAYERS are BEING told we will achieve XXXXX amount of players per server. SO THE SHIP SALES CONTINUE. If you're presenting how servers work to us. You've been cleared to do so. So Chris Roberts knows this in advance. Believe me bro. I understand being in a job where boss wants you as an ant, to push a boulder up a hill and just make it work. Would appreciate it if my comments are not scrubbed. The Star Citizen community is a very intelligent community. But a lot of us are tired of the money hungry feasting off our dreams, desires and passion for gaming.
@@MurkMercy what are you even going on about? He doesn't work for CIG or any of those companies nor has any relations with them.
Yes the main funding for the game is currently via ship sales and other smaller backers who invested, its why likely by 1.0 a lot of things will change, the ship store isn't going away probably ever as will other things like skins etc, they also have their two different subscription options many people pay into among other things.