Development Update with Server Meshing Technology - 11AM PT Wednesday, July 3, 2024
Вставка
- Опубліковано 17 лип 2024
- This month, we shared in-depth details about our server meshing technology in Ashes of Creation, along with studio updates, and more! 👇 For Timestamps click "Show More"
Chapters
0:00 Intro
1:48 Reminders
5:56 Server Meshing Technology
1:28:52 Testing News & Answer Server Meshing Tech Questions
1:48:04 Studio Update
1:53:55 Art Update
1:57:17 Outro, Q&A - Didn't hear your question in our Q&A? Check to see if it was answered over on our forums: forums.ashesofcreation.com/di...
🔔Reminder that Ashes of Creation is an open and transparent development project. As such, you will see work-in-progress art, systems, and mechanics. Leave us your thoughts on our progress in the video comments!
💬 If you have any feedback on what was shown, please share it with us over on our Forums!
- For Server Meshing Technology Feedback forums.ashesofcreation.com/di...
Follow us on
👉 / ashesofcreation
👉 / ashesofcreation
👉 / ashesofcreation
👉 / ashesofcreation
🗣 Join the discussion by connecting to our official discord / discord
📺 Watch live the last Friday of every month at / ashesofcreation - Ігри
Glorious Ashes of Creation Community, a HUGE thank you for continually tuning into our development updates. We are getting SOOOOO CLOSE to A2 coming your way! Our June showcase was the most technically oriented we’ve ever done, and it’s all in an effort to share with our glorious audience what exactly goes into making this MMO, and what is being tested in A2. We know these kinds of updates aren’t as exciting for some folks as the gameplay-focused ones, but they are important. We would love your feedback, and we plan on announcing the A2 start date at the end of this month! Also, we plan to begin sending out test email invites to some A1 users this month as noted on the stream, so stay tuned for more information!
We all assume dear Steven is now better again. 😎 Thanks very much for (finally ^.^) another Art Update. I myself am really addicted to see anything new from your awesome Fantasy Universe your MMO Game plays in. 🙂
What a great job taking such a super techy topic and making it digestible to the masses.
As a IT professional I can certainly tell you non tech people cannot entertain it for very long lol.
Nothing can make your mother bored faster than taking about a motherboard.
I wish you guys were on the East Coast
As a software engineer, I have put a lot of thought into how I might design a dynamic gridding solution for multiplayer games (like MMOs). This stream was very interesting as well as reaffirming for me because the ways you are achieving scale are quite similar to my designs. I love the technical deep dive content like this every once in awhile. Keep it coming!
1
Stop saying it wasn't exciting because it was tech. We gamer nerds loooove this stuff! Stream was definitely a 10/10 💪😎
Awwwwww, thanks. Glad you thought it was exciting. We just know it's not everyone's cup of tea. ;)
@@AshesofCreation it is mine! :D
@@AshesofCreation I'd say this was top 3 best dev streams and I think even for non-nerds it might have been more interesting than you think. I would say that apart from the deep dive into network engineering which the team and the presentation did a great job explaining in layman terms it shows something we all MMO gamers know. And that is laggy servers because a world boss is being killed or an event is starting somewhere in the world.
What makes this dev stream so exciting apart from knowing how things work in the background is your problem solving, your goals and how you've taken on the challenges to solve them. You are not satisified creating a game with laggy server when world events happen like every other game has settled on and every MMO-gamer just has gotten used to without liking it. And you are pushing boundaries and tweaking the unreal engine to multi-threading just to get that effeciency to where you need it to be for your OWN standards and goals.
That is VERY exciting to see because that means soo much more in the long run. It means you deeply care about delivering and improving the experience to something not seen and experienced before. Even after the game has released we you choose to play the game will see those improvements over the years to come after release if you keep the same attitude forward. That is what this stream also told us if you read between the lines.
@@AshesofCreation Players who have played DAOC or Warhammer Online know how important this video is. Good gameplay without server/graphics stability is not good gameplay. Thank you for sharing.
All of you did a great job breaking down and explaining the most nerdy tech ever. Bravo for the awesome visual aids as well. Excited for good performance in large battles!
D'aww, thanks! 😊
As a game dev doing both front end and back end stuff, I find videos like these very interesting. Good job
This is great content! An MMO that isn't built as a distributed system and can't scale horizontally will always be gimped and immersion breaking. What you guys are doing is paving the way as the standard for all MMOs.
💖
I appreciate the troll, Steve N. Great stream. I'm a tech nerd so this was fun!
Haha! =P
All in good friend homie.
Did you guys get inspired by Star Citizen by any chance (I'm ok with that)? The reason I ask, it's because of things like server meshing and replication graph (especially the latter), which no other game has and Star Citizen was the first to talk about it publicly at least 5 years ago
It's interesting two games are working on the same tech at the same time.
Is going to be a new era for MMOs
I might get flak for this, but it’s the honest truth.
As a technically minded person, I haven’t bought into any of the ashes of creation “hype” until this video.
This sold me.
The sheer amount of innovation this is bringing to the table has made me a believer in this project.
More of these please!
WOW. That's crazy! I love the new techs you've created. We can finally get back “massive” in ”MMORPG”. I can't wait for this! Thank you for explaining it, I love it!
Really enjoyed this type of content on the stream, amazing job!
Thanks for tuning in!
For what it's worth, this is actually a much more interesting topic to me than a lot of the standard content updates.
My team is also building applications on Unreal Engine.
I'm really happy to see how much everyone wants to improve the game to be at its best. Great job hiring these invested professionals and awesome job done by them.
Amazing work! I can't wait to experience the final product!
Absolutely loved the in depth technical discussion! Didnt understand much but loved listening to the difficult challenges the team overcame and did what sounded to be impossible! Keep breaking boundaries team!!!
Also Im extremely excited to be able to purchase the A2 key cause im not a broke university student now! Hoping it'll be reasonably priced!
YAY! Glad you enjoyed it!
I remember some of the general ambitious ideas how to solve UE limitations mentioned on one of the very first videos and I thought you just abandoned this as it turned out to be too difficult, but man, here we are. I'm impressed.
This was bloody AMAZING!! One of the best streams honestly, loved it ⭐️
Aw, so glad you enjoyed it! Thanks for the kind words and following along with our development journey.
Great overview of the server meshing. The effort you are taking to ensure stability of the game is exciting. I have one thought though. At 29:35 your slide says that a server is responsible for one grid component. If memory serves this is how EVE did their system, and in massive battles load was still an issue. Assuming you do not dynamically subdivide as load increases, it may be wise to see if you can scale the containers your dedicated server system runs on and allow for multiple servers/processes for a grid. Keeping those in sync via gRPC or similar technologies wouldn't be that difficult in comparison to what you have already described. With a true no layer system in place you may have thousands, or tens of thousands, of players in an area battling(if history teaches us anything it is that players don't really think about server load when they plan massive PvP battles). I LOVE that y'all are talking about how you plan on doing your backend!
This is incredible! Server meshing is one of the primary issues Star Citizen is working to implement as well. I'm glad to see this fantastic technical deep dive; clearly great talent at AoC!
@MargaretKrohn of alllllll people, you were the one who helped me get a PS2 Beta key. I conversed with you all those years ago... Seeing you here makes me happy that you are continuing to prosper. I look forward to seeing AoC getting more work done!
Tech streams (and art >.> ) are my favorite!
Amazing video guys. I've been tuning in here and there for Ashes, but this showcase of how you're pushing the networking boundaries for MMOs is inspirational. Definitely makes me curious in providing feedback or even applying
Was watching this whit my friend we had fun and learned something new
🧠
Coldest line from this video "I did some research on multithreading on unreal and everyone said it's impossible..... but we did it anyways"
Just to give you some idea of public interest in server meshing discussion, I’m not an AoC backer or player and I probably won’t ever be but I’m here because of your work in server meshing. AoC and Star Citizen are the only games with static meshing AND working on dynamic meshing and there is a segment of gamers like me who have played games for a long time under the limitations of traditional MMO server architecture who are excited about any games being developed that are attempting to push past the limitations. That should give you some idea of the appeal of tech heavy discussions like this even outside your community. We know what you’re doing will seep out into other games in the years to come so we’re all rooting for you!
watched it twice. That's how good this showcase was. Thanks intrepide! GREAT WORK!
I really enjoyed the Tech discussion, even with little actual knowledge its really cool to take a look under the hood and admire the mass of moving parts and shed light on issues you guys are facing. It's been very informative. Thanks!
You're welcome. Thank YOU for joining and following along with our development journey.
This was great, as someone that has worked in similar technology spaces (albeit not in game development) this was very cool to see
To be honest, I'm always more interested in the underlying technology than the game itself. So I really appreciate the deep dive.
As someone that has recently started to learn about computer science, this was amazing stream. Thank you guys so much! Very inspiring!
Keep up the hard work us players cant wait to make it home. It will be super exciting to see the largest battles and fun game play ever created.
Honestly loved hearing the detailed explanation!
Great presentation! I think I understood everything, haha
Yay!!!
Absolutely phenomenal video, I don't understand the complex parts, but I get how insanely complex it is to do. If it were easy, everyone else would be doing it.
"But it's still not enough" hang onto that mantra
I loved this, I haven't been following these updates very much - but I might considering following a lot more if it we get techy updates like this
Really loved the deep dive into server meshing, thanks!
Great update. Very interesting to dive into the tech behind the game
I can't wait to play this game love you guys being so open about development.
Absolutely fantastic update! Major props to the team for executing such a well done presentation, as well as making some really heavy technical info palatable for the core audience. Round of applause, excited to hear more about the bard and A2 date at the end of the month 👏🏼
so excited to play this game in 10 years when its finally out
It's good to be an informed player. This video was great info.
💕
Thank you guys so much for your work. The work you guys are doing is what the gaming world NEEDS to move forward. I cant wait to help out in any way I can. Even if it is just on the forums. I also have a friend that will likely apply so I will send them a message today!
also personally I LOVED the more in depth tech side of this. This is literally what I spend my freetime reading up on and tinkering with and its how I made my career what it is. This is literally the best video I have seen for this kind of tech and explaining how it all works!
Thailand people are waiting to play ❤
Hey! Great choice with microservice architecture for your MMO RPG project. It’s an ideal fit! As a developer who loves optimizing service performance, I can really appreciate the noticeable impact of your work. Although my contributions are usually less visible, I can see the impressive results of your efforts.
I’ve always wanted to work with gRPC, and it seems like you’re using it for cross-server RPC calls. Your implementation of load balancing, server scaling and proxying is remarkable-you’ve practically built k8s for gaming. Kudos to you guys!
This is massive! Great job guys, hoping for a timely release now with minimal bugs!
For those unfamiliar with software development, here's some insides:
Story points - unit of effort measure for a certain development task
Backlog - it is the bag of all development related tasks that are yet to be done
Debug - it is the way of checking the code as it runs, or register (log) all what is happening
Server meshing - it is the mechanism of server interoperability which is the way they speak with each other to create a working server cluster
Milestone - it is a set of things that once done fulfill the milestone and normally creates a patch or a minor/major version to be tested out or deployed
Concurrence - entities competing each other for resources such as players asking for server responses
Expensive in development - how much computing and hardware resources a certain thing consumes
I'm a complete networking nerd, this was awesome. I run a 10gb home network that's turned into a chore with the new SSH vulnerabilities. LOL
I'm just biding my time now. Now it's Ramp Time for "Bardic Lore" and "Stand-Alone Character Creator"!!!
I may or may not get into spot testing, but I didn't get to test as much as I would have liked in A1, so it would be no surprise if I didn't make the cut before A2 comes anyhow.
💕
You guys are doing a stellar job! Making magic happen!
I'm super excited to see how this will work in game and the effects it will have for players.
In cities around main npcs and shops is going to be the main test though, you definitely want that big social side of things.
This is so exciting to know more about the in-depth processes, so don't say it's not interesting. It's awesome! Thank you guys! Waiting this game so much!
We need more tech stuff like this. This was gorgeous.
I’m super excited for this game, as a comp sci guy from the era of c++, make sure your deconstructor methods are clean so you avoid memory leaks when players jump from server to server
1️⃣ idea would be to have transmitters to turn proxy copies to auth copies, and vice versa and know which properties need added or sent to server cleanup, very cool tech
I really liked this tech showcase its intressteding to see what U r doing behinde the curtains and getting Infos on stuff a Player dosn't See in His normal gameplay
awesome, great job! super excited about all of this!
I'm a CS student, on my free time I have this little hobby game project I'm working on and it's a MMORPG, of course, it's a very simple one but still, having to build the dedicated server and learning all about things like rollbacks, sending deltas, serializing, load distribution and how everything connects is super insteresting. So I LOVE having in depth reviews of the networking, there is very very little content since this is a very complex topic but for those inside of it it's amazing to look at.
1️⃣ thing to know is I love this technical deep dive
I loved this stream! You guys are doing awesome work and I am in awe of the dedication and passion you guys are pouring into this game.
Wow, that slide deck is art/science on its own!
Love so much the team is sharing this work. Thank you all! 💖
Loved this update much more than usual. They are always great, but the tech is more interesting. I'd rather see this stuff than the usual "gameplay" stuff that what I think would spoil it for me when I do start playing. I'd like to experience that when the time comes rather than knowing what's coming. Amazing work peeps 👍🏻👌🏻🎉😁🔥
Ill be honest, i am a developer for internal systems for businesses (automation etc.) but listening to this type of problem to solve... i think im in the wrong field. This sounds fascinating to sink your teeth in. I dont think i live close enough to work but just the sheer passion is heard!
Awesome, Thanks for the in depth Update
as a gamer and a indie game dev i love these. these things are exciting :))
very cool breakdown. I can't wait for this game to come out.
Was a nice live stream! We watched it live from Germany we went like oooh and ooooh and wow already 2 hours in was it supposed to be this long? :D
How's the weather in Germany?
We love a good oohing and ahhing. ;)
as a developer I love this stuff.
We've evolved into Ashes of Citizen!
This was an awesome video. Best insight video I've watched on any game dev so far.
This was an amazing stream. Best one by far in my opinion. This really helps to appreciate the hard work going into this game and the future is off the charts bright. Thank you for this.
I don't understand network coding but because of my job I like to make things more efficient as an applied mathematician. I really liked the tech behind the scenes. I have seen similar attempts at this in other MMOs, most of which were abject failures. I have a feeling as a layman that things will be different here, you are going in the right direction, this will be huge step.
Glad, I can support this project as a backer.
This stream was FIYA!!!!!!!
It was a great stream, really missed these more technical updates and I'd love to see many more like it.
Can't wait to start testing and see you all ingame!
edit: Also, as a braver of worlds backer I'm perfectly fine with you selling more alpha keys. If it can help you stress test the server go for it, the more the merrier and more cash for you to work without pressure.
Can´t wait to play this
It was freaking amazing and inspiring! You should definitely bring this to GDC as a presentation, maybe it would attract some people to the team.
Also great to hear Alpha 2 purchase coming back, I missed my chance last year but I'll be sure to get it now.
The complexity of the tech used in mmos is probably one of the reasons why there's so few good mmos. Good job explaining this stuff.
I didn’t understand everything but it was really interesting to listen to. Hopefully there will be UA-camrs that will break down on what exactly was achieved that wasn’t before and how this translates to players.
My dream, a true massive multiplayer sandbox, great job!!!
I love this. Great video.
- Since you made multithreading work, have you tried to delegate work to multiple servers? (or is sending the data to process across servers too expensive?)
- You were talking about hotfixing faulty servers: isn't having a system to detect and replace them automatically possible? (backup servers)
- Is it possible to have a server-swap system ? (Having a bunch of monster-servers that takes authority over crowded areas (that are not usually crowded))
I'm not an MMO fan. I watched because I'm a tech fan. Question: are you working closely with Epic? Are they going to back port your multithreading changes to Unreal at some point?
I recall a game called Rift and it was all cloud based server side. One dev talked about seeing 1300 people in one zone, no issues. That was pretty awesome.
Very nice to see such an in-depth peek under the hood!! Thx... Being a techie, I do have (many) questions: does a client connect to a new server when crossing over a border? Or is there some other gateway in between? (would give a bit of extra delay)
Super excited for this game. Probably gonna be getting an alpha key when they go back on sale. Great work guys! Very promising insights on the progress of the game! I do wonder what you guys are shooting for for recommended and minimum requirements for this game though?
Wait, wait, wait ... ok, so in your display of a server going down @52:48 ... it doesn't only effect that server but every surrounding server that was in communication with it. Data is lost without a replication layer because no entity survived that has the current state within that server as well as all data that was to be communicated while the server went down. What happens at the authority borders between servers while it is down? Does it become an invisible zone wall until a new server takes over the other sides authority? The server that went down has all the entity data under its authority, so does that mean it can be the only server that can spin up to take that authority slot without major data loss?
nice, I thought of that as well. They must have a "backup service" server for each Data Server, to pull from on spin up.
If the crash and spin are reasonably fast then the other servers would show some minor glitching because there would be no transactions until spin up.
So, if you were a player attacking something across a boundary, it would appear to do nothing? That's my thought anyhow.
They cannot have you attacking the fake without response after all, that's too pure a cheese.
I LOVE this Technical stuff!
Deltas will occur I'm sure but it's not hard to recover db information running HA. I don't know for certain but a child authority will likely be used. Lots of ways to prevent data delta.
Made me more interested in the game, thanks for sharing the tech (i have no idea what you're talking about ngl)
The female host is killing it btw, for some reason it feels like shes the kind loot master /co raid leader (Edit, director of communication: i get it now)
This is truly magnificent !! Thank you !!!
🥰💕
Steven please buy Atlas from grapeshot and use that ocean and ships for ashes it would run a so good with your meshing it would be the best !
As a nerd I approve this video thanks for the transparency ashes team you guys are changing the way videogames are developed.
💖
The best development update I've seen ❤
This sounds cool. Also sounds like an easy way to shoot your self in the foot though. The trade off for compartmentalization or decentralization is always complexity, and complexity is also difficult to work with. I am praying for this to work cause it sounds great.
1:54:25 More of these molded,rotted out cloth textures pls. Looks like glove made out mycelium. Love it.
1:55:23 Sick. Too bad you don't have some sort of druid archetype that can rock that style
1:55:36 Good texture. Interesting decorations.
1:55:55 Absolutely fucking amazing. MORE of this torn used up cloth textures on everything.
1:56:11 You never disappoint with your mounts.
1:56:30 Same old swirl decorations on metal parts. Its fucking boring.
If you really want to go with that type of style, use art nouveau references. Worked perfectly for elves in LoTR
Childish looking blade shape on that sword. At least the part near the hilt.
Hilt, grip and pommel are perfect.
Gorget is amazing. Ttassels too. Thigh guards are connected via belts to armor. Just a heads up.
I dont know wtf are those rings under his armpits....
Helm is interesting and spaulders are not that boring for a change. Gambeson had a bit more folds than usual, but thats ok. You could cover it slightly with chainmail and rerebrace.
Love the color scheme. Stick to these less saturated colors.
Also, good balance between the busy detail and places where your eyes can rest.
Can't wait till the start of Alpha 2 Might start making content over this and take a deeeeeeeep dive into the mechanics and seeing what is actually good and could use a tweak It has me hyped watching this game for the past 5 years slowly being crafted and made by people who actually care about the creation they are making. 1
To whomever is handling the media on this channel, don't look down on the metrics- all this information is vital and needed for the community. Viewership will boom when the game is released.
Hey Steven, this update was great, as a computer scientist I really enjoyed the journey the team took us through with solutions and everything.
Still, I feel like one thing wasn't clear, how do you plan to fix the problem of having too many people in one place? To me it seems to have a problem because as you said, you cannot open a server for every person or a small area ofcourse. Has the team talked about slowing down mechanisms like eve online or queing for entering zones like albion online?
Another question is, when shards are small, suddenly you might interact with shards that are not directly connected to your server, is it still possible? Those servers are in vision of the player but seem to not be in connection
Great work!
Game changer congratz IS!
I hope combat Will be as well thought
i'll tell my grandkids to try this game when its finally comes out
Talk nerdy to me!
🧠🤓💕
Did they say how big each zone is (give or take)?
I hate that I missed getting a copy of the game for testing. Really looking forward to playing one day.
no one is playing at the moment so you didn't miss anything lol
Love this stuff networking is my passion, AOC devs if you need more servers hit me up also based in San Diego and I can get ya discounted servers pc's and parts also sounds like a good excuse to visit the studio. 😊
Does these alpha2 new keys give access to betas as well?
Any chances of a white paper or an overview of the protocols and rfc’s you used in making is work? Amazing job as a network engineer I am fascinated with what you have accomplished.
Probably will not be doing something like that, but we like your curiosity!
That would be soooooo cool. But doubt they can publish something like that :(