All of this is actually pretty straightforward, at least the individual pieces. Time travel debugging, incremental compilation, live patching, etc etc. But the whole seamless package is very impressive indeed.
This is what a modern day development environment should look like! Thank you for sharing this and showing what our PCs are capable of. It opened my eyes even more and am now very inspired to strive for such a development environment for myself.
This is beyond insane. A lot of work must have went into making the underlying system, but the developer experience looks absolutely stellar. Taking deterministic execution to a whole new level.
Geez you kind of blew my mind with the reverse stepping and it didn't stop the further you went. Being able to debug sometime else's session completely removes "works on my machine". That's brilliant. Major props getting this to work so well. -MOM4Evr
My son just sent me the link to this video telling me "you must watch", and I'm so glad I did. This is some absolutely insane tooling! I don't know why, but this code editor has a visual studio 6.0 vibe, which is pretty awesome. Hats off to you!
This is so mind-blowing that I'm pretty sure 50% of people who watch this will think your event timeline scrubber is just a video, despite you saying it's not a video, because of how incomprehensibly epic it is. Kudos, and thanks for sharing. Would love to hear more.
OMG, this is amazing! While the world industry's direction is something like "Let's multiply further the existing bloat with more AI", this IDE is on the opposite side: It values the developer's time, not wasting it.
Absolutely, while Copilot can sometimes make an interesting leap and automate a piece of glue code and save a whole bunch of time, it's underlying idea is ultimately raw dumb text completion, not a single special programming-related trick. But things like this demo and Jetbrains IDEs? Magic, pure and beautiful magic. Subtle machines crafted by those who know what they're doing.
@@mostm8589 Yeah exactly! Slapping AI on top of bad tools seems just so dumb. If the existing tools were better, no-one would be even considering something like Copilot to be a good idea
I don't think I've ever been so impressed with a tool. It has opened my mind to things I wouldn't have imagined being possible. Reverse stepping and having visuals available while hitting a breakpoint are two incredibly innovative things.
I've been coding for 25 years and doing it as a job for 15... This is hands down the most insane shit I've ever seen. I can't recall having ever been this impressed by anything. This is absolutely batshit.
This is the most mind boggling thing I've witnessed working with games..! I'm buying all the games just in hope of y'all keep creating tech like this 🤩
This is some of the coolest shit I've ever seen. Honest to god stuff of dreams for programmers, and makes you wish even half of our work could be this well designed. Truly awesome.
Looks great Allan! (I'm guessing the GOAL thing you read would have been in one of them ol' fashioned magazines given the era! I remember being really inspired at the time too.) I sometimes have visions of doing something like this, but it's hard to invest the required long-term effort, and also to decide where exactly to cut around in the design space (where to skimp/cheat, what pre-existing tools to use, etc.) I hope it works well for you and you can keep developing it. I'd love to hear more about how it all works if/when you have time.
Awesome tooling, I only have about 100 questions based on curiosity alone. That debugging feature would be a dream, it's incredible what one can do once you control the aspects of all the data. Beautifully put together!
This the sort of masterpiece that would make Brett Victor proud and vindicated, are you aware of him and his ideas ? I hope he sees this, and I hope the HN submission for this video blow up to the front page. From a guy absolutely in love with compilers, programming language engineering, and dev toolchains : thank you, you have successfully made me want to do something like this. (just like the demo that made you want to do this in the first place)
This is insane dude. I love that you made everything from scratch. Jonathan Blow is mad on twitter saying your video is fake because what you did is so obviously superior to what he is doing with Jai and he behaves like a petty spoiled child.
do you have a link? that sounds amusing... seeing how this team made their own language reminded me of JB, i was curious what he would think of this, it looks like something he would love. it's surprising to hear that he claimed it's fake.
Update: I showed it to Eskil and he said he was familiar and impressed. He had a similar thought to me that the only unfortunate thing is that it's a custom compiler and not a general purpose language so not particularly accessible to other devs. Still very inspiring tech, though!
Wow, looks like an environment which rewards good design! At a guess, you have a coherant state at least once per frame, which can be reproduced knowing the inputs, complete codebase and previous frame.
I have so many questions, like, do you have your own compiler? How big is the state? How do you store it? How big are these session files? Are you actually running the program backwards or are you jumping back to some previous state and running forward from there?
The size of the state will depend on the amount of mutable data that the game needs. A ton of runtime data will be read only assets, so you just need to capture the stuff that actually changes. The size of the session files is mainly determined by the size of your current work that hasn't been committed to source control yet.
@@retrogameinternals4707 Hi! For each tool that you show in the video, what is the % of time that you use that tool versus having to use a more standard tool? I just can't tell if they have completely replaced those standard tools for you, or not yet. (I'm not a programmer, so this might be a clunky question.)
That demonstrates that is better to build by yourself simple tools that performs a smooth and fast production and do not depend on big and closed engines like Unity and you can do always better like you did. I'm was impressed with your work!! At first time I suppossed that you used a C language in the code of the game but no, you built your own compiler!
That's true but you can't just setup reverse-debugger or live-reload. If you check Allan Blomquist, he knows lots of stuff about low-level programming. Reverse engineering, compilers etc.
Incredible. This is a bit like rr and my thesis project, hotload. Both are FOSS and for native code (C, C++, Rust, etc.) instead of a custom scripting language / VM
If this gets used for World of Goo 2, I would be highly interested in a video of the biggest development challenges. Like how Ori was ported to the Nintendo Switch at 60fps.
This is incredibly impressive, and some of those features make me quite jealous! The backwards-debugging in combination with the allocation tagging system feel like they could have saved me quite a lot of time elaborately searching for nasty, nasty bugs in the past. There's one thing I'm a bit curious about: You mentioned how you wrote your own compiler front end and back end. So what do you do when you want to port one of your games to a specific game console? I mean, they usually have their own architecture, compiler etc. Are you forced to make a version of your compiler that can compile to that specific target architecture? If so, doesn't that make ports to new systems very tricky? Or do you already have a solution for that in place as well?
In the past our compiler could target a made up bytecode that we could interpret at runtime and we shipped small features of our existing games using that. Now the target is native code on our development platform only like you said, so in order to ship in the future we will transpile the code to plain old C and then use the native toolchains on the target platforms to compile the final product.
@@retrogameinternals4707 I see, that's awesome! I actually thought it might be a transpiler, but wasn't sure how viable that was! Though I guess with your custom language being so close to C, anyways, there's probably a smaller chance of bugs sneaking in.
Just wow. This is looking incredible. I can't imagine the time you must have put into making all these tools. The end result just feels magical. I'm curious about the language used - it looks a lot like C++ (including STL things like push_back). Is it a subset of C++ or something else inspired by that?
@@retrogameinternals4707 Thank you, that sounds cool. I would love to know more details, especially what quality of life improvements you added. The only things that stood out to me were "in" for ranged-based for loops and a .. operator reminiscent of Rust.
That is awesome! Absolutely brilliant dev. environment and playback. Will you do any more about this? I'd love to know how much disk space is needed to save sessions and record all changes or how the environment was designed to keep disk usage to a minimum
This is astonishing. I know it’s real but I imagine this is just masterful use of sfx software, trolling developers. Every tool chain maker needs to see this.
Is it event-based "database" that underlines not only prod runtime, but also the whole development stage incl all the IDEs? that's awesome 😃👍 This is how interactive coding should look like 😃👍👍👍
I feel sad that at other studio folks may be spending half a day chasing down a bug when in reality they could be doing in a few minutes,... if their studio actually had the intelligence to invest in custom tools like your playback feature. I laugh out loud when I read sentences like "Unity or Unreal is good for making 2D games", your team has made tools for 2D games that are vastly superior to what a company with 1000s of employees can make. Phenomenal work!
Could you explain how the reverse debugging works? Is every line of code deterministic both forwards and backwards, or are you saving/restoring state snapshots? Incredible demo btw, I envy this workflow greatly.
Do you handle multithreaded code in the state capture system? If yes, how do you store and restore scheduling info, etc.? If no, any plans to do so in the future? I have been wondering how to solve that, but the only ways I can see is either having a high-level language "safe" language compile to fiber based code with userspace scheduling, or writing a OS driver to mess with the scheduling as needed. Would love to hear your views. Also, as an aside, during state capture, do you store entire memory image at beginning of frame and then run back and forth; or do you store each read and write instruction (like an undo buffer)? Thanks, and best of luck with this system. I am in process of developing something similar (also inspired by GOAL), and it's nice to see others trying their hands too.
No multithreaded support so far. If we were going to add some, it would probably have to be a constrained form of scatter/gather only where memory protection was used to make sure that no thread could read from any memory that any other thread had write access to at the same time. Or if not that, then some other harsh constraint that made it so that there just weren't any race conditions possible. Once you had that then you wouldn't need to worry about stepping the different threads in any particular order, you'd just sync back up after you've finished going wide and the main thread would consume the results. Whether or not the constraint that you would have to operate under would be so severe that you would still be able to get any benefit out of multiple threads I don't know, and I don't have any plans to investigate in the near future. :) To capture the state, we memcpy the simulation heap at the beginning of a frame. This is far less data than the entire process' memory image though since the bulk of the data in memory is effectively read only for the purposes of the simulation of the game (movies, sounds, level blueprints, code, dialog, etc.) and the stack isn't needed since we never capture the state within a frame.
@@retrogameinternals4707 when you say "the stack isn't needed" what do you mean? It seems clear that you're doing stack-walks (your profiler heat map has stacks, and you can click on these functions to go anywhere in your editor). Perhaps you meant that the stack data isn't copied, but then how does your reverse debugger know the (possibly stack-local) value of things? Like the NULL pointer or the floating point value in your example? Thanks for this really interesting presentation.
Just… how…? I would religiously watch a series of videos describing how each aspect of your environment, engine, runtime, and toolchain works.
seconding this comment, this would be an amazing video series to watch
All of this is actually pretty straightforward, at least the individual pieces. Time travel debugging, incremental compilation, live patching, etc etc. But the whole seamless package is very impressive indeed.
if this was a recruitment advertisement to try to attract new developers it totally worked on me 😂
I was gonna say, I moved away from C++, but with tooling this nice, I almost want to apply, holy jumping.
This is insane. I never knew I needed Braid mechanics in my game engine until now
Consider my mind officially blown. It's really impressive what is possible when you have full control over the entire tech stack.
This is what a modern day development environment should look like!
Thank you for sharing this and showing what our PCs are capable of.
It opened my eyes even more and am now very inspired to strive for such a development environment for myself.
This is incredibly beautiful. Add me to the chorus of people who would love a deep dive into how this works.
This is beyond insane. A lot of work must have went into making the underlying system, but the developer experience looks absolutely stellar. Taking deterministic execution to a whole new level.
Geez you kind of blew my mind with the reverse stepping and it didn't stop the further you went. Being able to debug sometime else's session completely removes "works on my machine". That's brilliant. Major props getting this to work so well.
-MOM4Evr
This is probably the nicest debugging experience I've ever seen.
I am honestly blown away. Great job, and thank you for showing the world that their standards for debugging are way below what's possible.
My son just sent me the link to this video telling me "you must watch", and I'm so glad I did. This is some absolutely insane tooling!
I don't know why, but this code editor has a visual studio 6.0 vibe, which is pretty awesome. Hats off to you!
I use VS2008 as my main dev environment so that makes sense!
@@retrogameinternals4707 the code editor from 2008. The programming environment/debugger from 3008. :D
@@retrogameinternals4707 I'm not typically into men, but I think I want to marry you.
This is so mind-blowing that I'm pretty sure 50% of people who watch this will think your event timeline scrubber is just a video, despite you saying it's not a video, because of how incomprehensibly epic it is. Kudos, and thanks for sharing. Would love to hear more.
The backward debugging is insane, very impressive work!
OMG, this is amazing!
While the world industry's direction is something like "Let's multiply further the existing bloat with more AI", this IDE is on the opposite side: It values the developer's time, not wasting it.
Absolutely, while Copilot can sometimes make an interesting leap and automate a piece of glue code and save a whole bunch of time, it's underlying idea is ultimately raw dumb text completion, not a single special programming-related trick.
But things like this demo and Jetbrains IDEs? Magic, pure and beautiful magic. Subtle machines crafted by those who know what they're doing.
@@mostm8589 Yeah exactly! Slapping AI on top of bad tools seems just so dumb. If the existing tools were better, no-one would be even considering something like Copilot to be a good idea
Not only the games are a masterpiece, but they are also an engineering masterpiece
I don't think I've ever been so impressed with a tool. It has opened my mind to things I wouldn't have imagined being possible. Reverse stepping and having visuals available while hitting a breakpoint are two incredibly innovative things.
I've been coding for 25 years and doing it as a job for 15... This is hands down the most insane shit I've ever seen. I can't recall having ever been this impressed by anything. This is absolutely batshit.
This is what every dev environment should be! Very impressive work, and I'm sure it's a huge multiplier to your team's productivity.
This is so good. I came back to this vid, because I have never seen something even close.
This is the most mind boggling thing I've witnessed working with games..! I'm buying all the games just in hope of y'all keep creating tech like this 🤩
Running code backwards to track down changes into variables is OP. Big props to you guys for implementing this.
Incredible work. Reminds me of Bret Victor's Inventing on Principle demos.
Watched this multiple times now, it’s having the same impact on me that the blog post about GOAL you referred to had on you.
This is something where you have to work at a company to really understand why this is the golden standard
This is some of the coolest shit I've ever seen. Honest to god stuff of dreams for programmers, and makes you wish even half of our work could be this well designed. Truly awesome.
i am beyond impressed by this sorcery! and as if the backwards stepping wasn't enough, you're using paint shop pro too! incredible stuff.
never restarting and reverse stepping sounds like heaven🤩
Looks great Allan! (I'm guessing the GOAL thing you read would have been in one of them ol' fashioned magazines given the era! I remember being really inspired at the time too.)
I sometimes have visions of doing something like this, but it's hard to invest the required long-term effort, and also to decide where exactly to cut around in the design space (where to skimp/cheat, what pre-existing tools to use, etc.) I hope it works well for you and you can keep developing it. I'd love to hear more about how it all works if/when you have time.
Awesome tooling, I only have about 100 questions based on curiosity alone. That debugging feature would be a dream, it's incredible what one can do once you control the aspects of all the data. Beautifully put together!
This the sort of masterpiece that would make Brett Victor proud and vindicated, are you aware of him and his ideas ? I hope he sees this, and I hope the HN submission for this video blow up to the front page.
From a guy absolutely in love with compilers, programming language engineering, and dev toolchains : thank you, you have successfully made me want to do something like this. (just like the demo that made you want to do this in the first place)
i feel like i just witnessed something magical
Thank you so much I'm a game dev beginner and this is a good spot to watch a video of my childhood game and how it operates. Btw I like the art
You guys would give many companies a run for their money if you licensed this stack. Amazing toolchain.
I think this should be used as reference/base goals that every major game engine can strive to achieve.
wow, so impressive tech demo. Incredible at each 20 seconds of the video.
This is insane dude. I love that you made everything from scratch. Jonathan Blow is mad on twitter saying your video is fake because what you did is so obviously superior to what he is doing with Jai and he behaves like a petty spoiled child.
do you have a link? that sounds amusing...
seeing how this team made their own language reminded me of JB, i was curious what he would think of this, it looks like something he would love. it's surprising to hear that he claimed it's fake.
that’s mindblowing. love the debugger
Mind blowingly impressive.
@Eskil Steenberg needs to see this..
Update: I showed it to Eskil and he said he was familiar and impressed. He had a similar thought to me that the only unfortunate thing is that it's a custom compiler and not a general purpose language so not particularly accessible to other devs. Still very inspiring tech, though!
Wow, looks like an environment which rewards good design! At a guess, you have a coherant state at least once per frame, which can be reproduced knowing the inputs, complete codebase and previous frame.
I agree with everybody: I'd like to see how your game engine architecture is designed to achieve this wonderful environement.
I have so many questions, like, do you have your own compiler? How big is the state? How do you store it? How big are these session files? Are you actually running the program backwards or are you jumping back to some previous state and running forward from there?
I asked myself all the same questions looking at this! (Hi by the way, long time no see)
The size of the state will depend on the amount of mutable data that the game needs. A ton of runtime data will be read only assets, so you just need to capture the stuff that actually changes. The size of the session files is mainly determined by the size of your current work that hasn't been committed to source control yet.
@@retrogameinternals4707 Hi! For each tool that you show in the video, what is the % of time that you use that tool versus having to use a more standard tool? I just can't tell if they have completely replaced those standard tools for you, or not yet. (I'm not a programmer, so this might be a clunky question.)
Bruh UA-cam is getting very very good a showing me exactly the videos that i wanna see. Excellent content thank you ! 👍
this is so cool. i wish other debuggers adopt these awesome features. especially backward stepping.
"...so i'll do a reverse step out of the function..."
words no other developers have ever said
Does it handle multithreading? How do you keep interactions between threads deterministic?
I'd be unstoppable if I had tools like these. Beautiful !
You see that? That's Genius at work, right here.
That demonstrates that is better to build by yourself simple tools that performs a smooth and fast production and do not depend on big and closed engines like Unity and you can do always better like you did. I'm was impressed with your work!! At first time I suppossed that you used a C language in the code of the game but no, you built your own compiler!
That's true but you can't just setup reverse-debugger or live-reload. If you check Allan Blomquist, he knows lots of stuff about low-level programming. Reverse engineering, compilers etc.
I don't doubt he has a special knowledge about compilers and other subjects about programming. I will check at his great posts on its blog.
This is absolutely NOT a simple tool
@@bamberghh1691 I completely agree! I wanted to said a complex tools that makes simple your life xD
wow this is literally magic
Incredible. This is a bit like rr and my thesis project, hotload. Both are FOSS and for native code (C, C++, Rust, etc.) instead of a custom scripting language / VM
My God that is great. Very well done. This is so freaking cool. I wish I had half of these.
This is completely inspiring!
Deep integration, impressive. Thanks a lot for sharing!
Looks great! I wonder how World of Goo source code would look with this environment 😅
If this gets used for World of Goo 2, I would be highly interested in a video of the biggest development challenges. Like how Ori was ported to the Nintendo Switch at 60fps.
This is incredibly impressive, and some of those features make me quite jealous! The backwards-debugging in combination with the allocation tagging system feel like they could have saved me quite a lot of time elaborately searching for nasty, nasty bugs in the past.
There's one thing I'm a bit curious about: You mentioned how you wrote your own compiler front end and back end. So what do you do when you want to port one of your games to a specific game console? I mean, they usually have their own architecture, compiler etc. Are you forced to make a version of your compiler that can compile to that specific target architecture? If so, doesn't that make ports to new systems very tricky? Or do you already have a solution for that in place as well?
In the past our compiler could target a made up bytecode that we could interpret at runtime and we shipped small features of our existing games using that. Now the target is native code on our development platform only like you said, so in order to ship in the future we will transpile the code to plain old C and then use the native toolchains on the target platforms to compile the final product.
@@retrogameinternals4707 I see, that's awesome! I actually thought it might be a transpiler, but wasn't sure how viable that was! Though I guess with your custom language being so close to C, anyways, there's probably a smaller chance of bugs sneaking in.
This is Gigachad among other debugers
Just wow. This is looking incredible. I can't imagine the time you must have put into making all these tools. The end result just feels magical. I'm curious about the language used - it looks a lot like C++ (including STL things like push_back). Is it a subset of C++ or something else inspired by that?
It looks a lot like C actually, not C++. I'm guessing C99 would be main point of reference.
We took the "C+" style of C++ that we usually write in anyway and codified it with some of the quality of life improvements we wished we had.
@@retrogameinternals4707 Thank you, that sounds cool. I would love to know more details, especially what quality of life improvements you added. The only things that stood out to me were "in" for ranged-based for loops and a .. operator reminiscent of Rust.
@@danielagentlienkvick729 The import statements are also far cleaner than normal C++'.
That is awesome! Absolutely brilliant dev. environment and playback. Will you do any more about this?
I'd love to know how much disk space is needed to save sessions and record all changes or how the environment was designed to keep disk usage to a minimum
Spectacular work!
You had me worried at including our own language, but it looks very familiar :)
I'm just sat here dueling to this!
Wow amazing! I've been working on my own language and this debugging system is a great inspiration.
Incredible work! Very inspiring. Thanks for sharing!
This is astonishing. I know it’s real but I imagine this is just masterful use of sfx software, trolling developers. Every tool chain maker needs to see this.
This looks incredible
backwards stepping??? seeing graphics get drawn as they get rendered while stepping??? that's amazing
wow, what an awesome demo and toolset that you build!
I can't call myself a developer anymore
The editor kinds looks like you took Visual Studio, and just removed all unnecessary stuff, and the bugs.
I like it.
Is it event-based "database" that underlines not only prod runtime, but also the whole development stage incl all the IDEs? that's awesome 😃👍 This is how interactive coding should look like 😃👍👍👍
Absolutely incredible
Impressive and inspiring! Thank you so much for sharing.
Now I have to look at my tooling and be disappointed 😅
Thats amazing, I knew I was right to subscribe to you forever ago! While you're back can you do more punch-out stuff?!
I would love to finish my punch out series! Just need to add an 8th day to the week and I should be all set.
Interesting... Curious to know what's next, I thought it was sensational
Absolutely amazing development environment!
Why would anything watch porn when this is available ? This look absolutely incredible you guys are absolute monsters
This is breathtaking
All I can say about this video is........that.... it's dreamy...
I feel sad that at other studio folks may be spending half a day chasing down a bug when in reality they could be doing in a few minutes,... if their studio actually had the intelligence to invest in custom tools like your playback feature. I laugh out loud when I read sentences like "Unity or Unreal is good for making 2D games", your team has made tools for 2D games that are vastly superior to what a company with 1000s of employees can make. Phenomenal work!
This is incredibly impressive and I think if you were to sell access to this toolchain you would have a lot of success!
This is great stuff, but I wonder how well this would scale.
I knew you guys were brilliant, but this just blew my mind!
Could you explain how the reverse debugging works? Is every line of code deterministic both forwards and backwards, or are you saving/restoring state snapshots?
Incredible demo btw, I envy this workflow greatly.
Yup, saving and restoring snapshots.
Well done! Looks incredibly useful
This looks like an impossible dream. How does this full determenism/rewind actually works? Does it store a full game state in a frame?
Do you handle multithreaded code in the state capture system? If yes, how do you store and restore scheduling info, etc.? If no, any plans to do so in the future? I have been wondering how to solve that, but the only ways I can see is either having a high-level language "safe" language compile to fiber based code with userspace scheduling, or writing a OS driver to mess with the scheduling as needed. Would love to hear your views.
Also, as an aside, during state capture, do you store entire memory image at beginning of frame and then run back and forth; or do you store each read and write instruction (like an undo buffer)?
Thanks, and best of luck with this system. I am in process of developing something similar (also inspired by GOAL), and it's nice to see others trying their hands too.
No multithreaded support so far. If we were going to add some, it would probably have to be a constrained form of scatter/gather only where memory protection was used to make sure that no thread could read from any memory that any other thread had write access to at the same time. Or if not that, then some other harsh constraint that made it so that there just weren't any race conditions possible. Once you had that then you wouldn't need to worry about stepping the different threads in any particular order, you'd just sync back up after you've finished going wide and the main thread would consume the results. Whether or not the constraint that you would have to operate under would be so severe that you would still be able to get any benefit out of multiple threads I don't know, and I don't have any plans to investigate in the near future. :)
To capture the state, we memcpy the simulation heap at the beginning of a frame. This is far less data than the entire process' memory image though since the bulk of the data in memory is effectively read only for the purposes of the simulation of the game (movies, sounds, level blueprints, code, dialog, etc.) and the stack isn't needed since we never capture the state within a frame.
@@retrogameinternals4707 when you say "the stack isn't needed" what do you mean? It seems clear that you're doing stack-walks (your profiler heat map has stacks, and you can click on these functions to go anywhere in your editor). Perhaps you meant that the stack data isn't copied, but then how does your reverse debugger know the (possibly stack-local) value of things? Like the NULL pointer or the floating point value in your example?
Thanks for this really interesting presentation.
where am I? super nice :) i most liked the video that isn't actually a video. :)
I would love to have something like this available. Are there any resources on building such tooling that you could share?
Great work!!! Amazing: leaves me puzzling with lot of questions. Are you using some kind of event sourcing pattern to keep the history of states?
Is the Information Superhighway game still being developed?
How much would a Kickstarter be, for you to open source these tools? Millions would be possible, I guess.
This is wild! Very cool stuff.
This is absolutely unreal!
Cool stuff! Are the build tools, compiler, editor, engine and runtime etc also written in your "C+" language?
Very cool! Thanks for sharing
I wish I could debug embedded systems with this debugger. It would save me days of debugging time.
this is incredible
Please, tell us more about how all this works together
Holy hell this is incredible