Incredible work, thanks a lot for this in-depth presentation. Using G-buffers to generate a semantics map seems so logical in retrospect! One can't help but wonder what could happen if we develop a game purposefully built to work synchronously with SVS by default, helping it parse the data.
Regarding the work at 40:00: It'd be interesting to see a graphic artist modify the look of the game to match the results produced by the network using more conventional methods: Tonemapping and color grading tweaks, subtle blur effects, texture swaps, material tweaks and lighting tweaks, etc. It'd be interesting to see if we can produce these results using techniques we already know, and if there is anything deeper that remains about what makes something look "real" that we can't explain yet. One thing that stood out to me was a structural change in the appearance of some vehicle lights. For example, at 42:06 you can see a small change to the rear lights on the right car (this change is more apparent in the other video about this work on this channel.)
I was also thinking about it and the conclusion I get is that AI can reproduce the patterns of light distribution throughout the environment as it happens in real life, it is like a real-time pathtracing but without the need to trace even a ray.
To have that level of detail you would need a supercomputer that's the reason game devs don't go for better graphics they go for more fps but maybe in like 5 10 year we might get stuff like this as a norm
No idea how they can incorporate this into games. I honestly don't ever see a game looking this good without this technology. It's not possible. The imagery in GTA is real life. It's that simple, the game now looks like real life.
Awesome .. Still waiting of this . This green color correction also used in MW 2005 X360 version . Very smart decision ... ( but not GTA IV lol .. - it simply uncolored + added orange tones and fully no hq postprocess ) Only with some enbs like Danvsw etc - it looks same . So Big thx from GTA IV-V modding community for THAT .
Cityscapes looks better in some weird way than Vistas. More cohesive color palette, it somehow sells it as "real" to me better. I mean it looks like dashcam with the green tint, but still it looks like real dashcam
@@shredd5705 the data set they trained it on was dashcam footage I think on foot games would be much harder to get a good data set, plus I think given how good RDR2 looks there is a bit of a diminishing returns to using this on it. Euro truck simulator on the other hand I think is a better fit since its all played within a dash cam perspective like their data, Its set in euro like their data, and given its letter base graphics we can see a much bigger improvement than GTA or RDR2
as someone interested in this stuff from a purely artistic perspective, the artifacts are quite lovely on the comparison models like TSIT and CUT , i can't lie. neural hallucination is so fascinating when it's in video/animation form. i also notice these photorealism translations are making the colour scheme look more like the gritty and green GTA IV than GTA V :D
so it could be new part of art direction and photography to film with a particular camera and particular light, to complete this new pass into games or other vfx, it's maybe not less human than we think because, as they said, you have to feed the IA with your vision, it's illustrates the need of references
In the future developers will create games with basic shapes like spheres, flats, cubes, ragdolls with corresponding tags like wall, ball, desk, "person with blue jeans, red shirt, blond hair, glasses" etc. and train the data with petabytes of real life images/videos. The ai will "pre-render" this information and apply the pretrained data on the fly on the customers computer, providing a realtime, high-fps, ultra-high realistic graphics game. We already have the technology to do so(see DALL-E), it just needs to be combined, and made applicable for graphics cards. GPU's might have special cores for this kind of AI in the future and be able to rise the performance for such applications like Nvidia did with RTX within 2 generations of GPU's for example.
I have to wonder if human visual perception is also based on some aggregation of prior experience rather than strictly the view itself. It would seem so when you consider optical illusions.
I am not into the topic very well but how does this AI able to read texture map information from the game by useing video footage? The video is not real time anymore, so isn't hooked into the game engine to read normal map data and so on.
@@stephanrichter7028 You wouldn't need to. All you need is to use this system to colorize assets/textures and choose attributes like reflectivity for them. Then build a regular game using those assets. The correct color grading and reflection levels baked into the assets. I think games mostly look wrong, because they pick just wrong colors for stuff (too saturated, mostly). RTX can create good reflections, if AI tells it how much each surface should reflect
What does it look like when you get out of the car & start walking around? It would be nice to see what it looks like in cutscenes with close ups to faces as well.
On 1 hand I'm amazed on the other hand I start to wonder if the appeal of playing video games will be lost by the realism. I play to escape the real world, I hope mimicking the real world to this extent doesn't remove the escapism. Also, I like adventuring through fantasy worlds, how can they get real video of a fantasy? If they can somehow record video from our brain during dreams?
@Kenneth Yes, and I love their battle to be able to reach realism 1st. It brings us some cool looking fantasy worlds of entertainment. Just wondering if when they do reach it, will it lose it's allure as something different.
To preface this, your work is impressive and I am psyched about it. My question is why you did it. What was your incentive to improve upon the previous papers and to accomplish this result?
I would rather see it tested in something that has top graphics (like Forza Horizon 4 or RDR2), and see how much it can be pushed further. BeamNG doesn't look that great
You can't and even if you could it won't work in real time aka it might be 1 frame every 10 mins (a random guess) there is no way in hell that is working at a playable framerate
Como ray tracing mal chegou? Ray tracing existe desde os anos 80. E vc pelo jeito sequer sabe o q é ray tracing, tampouco entendeu algo desse vídeo, e se entendesse veria q isso sequer é aplicável
Watch Intel build a GPU with this running...they will make millions off the bat this will end AMD and Nvidia just look at the clouds thats as close as we will get to reality in a game hands down
I'm not entirely convinced that road surfaces in the US are as smooth as those in Germany. On the photos in geoguessr or Google Streetview they sure look different. Frankly I think the GTA original footage looks more realistic than yours.
![The moment we stopped understanding AI [AlexNet]](http://i.ytimg.com/vi/UZDiGooFs54/mqdefault.jpg)
![The moment we stopped understanding AI [AlexNet]](/img/tr.png)
Words can't express how incredible this is
The future is now
Thank you for such great presentation. I wish more researchers would do such clear and deep explanation of their work publicly.
All games should have this option! Mindblowing! 🤯 Love the immersion! 🙌🏼 Peace from 🇩🇰
Incredible work, thanks a lot for this in-depth presentation. Using G-buffers to generate a semantics map seems so logical in retrospect!
One can't help but wonder what could happen if we develop a game purposefully built to work synchronously with SVS by default, helping it parse the data.
This blew away the papers of a certain fellow scholar no doubt.
I am sure he thinks this is absolute insanity! What a time to be alive!
deep learning ai is gonna get scary on the next few years xd
Was just starting to think about that same thing again…..
Nah, just more awesome.
@@onee imagine soo awesome that can destroy someones life like a DEEPFAKE
Regarding the work at 40:00:
It'd be interesting to see a graphic artist modify the look of the game to match the results produced by the network using more conventional methods: Tonemapping and color grading tweaks, subtle blur effects, texture swaps, material tweaks and lighting tweaks, etc. It'd be interesting to see if we can produce these results using techniques we already know, and if there is anything deeper that remains about what makes something look "real" that we can't explain yet.
One thing that stood out to me was a structural change in the appearance of some vehicle lights. For example, at 42:06 you can see a small change to the rear lights on the right car (this change is more apparent in the other video about this work on this channel.)
impossible to do, because the information about all light sources is not there
@@gnt3c In a game environment, you actually do have all that information. Can you clarify?
I was also thinking about it and the conclusion I get is that AI can reproduce the patterns of light distribution throughout the environment as it happens in real life, it is like a real-time pathtracing but without the need to trace even a ray.
To have that level of detail you would need a supercomputer that's the reason game devs don't go for better graphics they go for more fps but maybe in like 5 10 year we might get stuff like this as a norm
@@captainpringles5946 We don't need a supercomputer anymore. Look at Unreal Engine 5 and think about tonemapping. We are very close to the norm
No idea how they can incorporate this into games. I honestly don't ever see a game looking this good without this technology. It's not possible. The imagery in GTA is real life. It's that simple, the game now looks like real life.
Great video. Interesting how it changes the hills to green due to the way it depicts reality.
This is better than ray tracing which I thought is the holy grail of graphics.Actually this is the holy grail
Brilliant explanation with such great detail.What a beautiful thinking in all of those details.It is totally worth for the 2 years spent on it.
Truly amazing, only hard work can accomplish those results
How would you know? You are no computer scientist.
Awesome .. Still waiting of this .
This green color correction also used in MW 2005 X360 version . Very smart decision ... ( but not GTA IV lol .. - it simply uncolored + added orange tones and fully no hq postprocess ) Only with some enbs like Danvsw etc - it looks same .
So Big thx from GTA IV-V modding community for THAT .
Cityscapes looks better in some weird way than Vistas. More cohesive color palette, it somehow sells it as "real" to me better. I mean it looks like dashcam with the green tint, but still it looks like real dashcam
Great job..I like how he explains in depth what's going on..
looking forward to synthesizing entire video games in 10 or 20 years
This is insanely cool. You guys should try The Last of Us 2 next. Feed the AI Zombie apocalypse movies haha
I would love to see what this could look like if applied to Euro truck simulator
mannnnn
Isn't GTA5 better graphically. I'm wondering how RDR2 would look like with this, because it already looks stunning and is present day tech
@@shredd5705 the data set they trained it on was dashcam footage I think on foot games would be much harder to get a good data set, plus I think given how good RDR2 looks there is a bit of a diminishing returns to using this on it.
Euro truck simulator on the other hand I think is a better fit since its all played within a dash cam perspective like their data, Its set in euro like their data, and given its letter base graphics we can see a much bigger improvement than GTA or RDR2
as someone interested in this stuff from a purely artistic perspective, the artifacts are quite lovely on the comparison models like TSIT and CUT , i can't lie. neural hallucination is so fascinating when it's in video/animation form. i also notice these photorealism translations are making the colour scheme look more like the gritty and green GTA IV than GTA V :D
The future is now Old man
so it could be new part of art direction and photography to film with a particular camera and particular light, to complete this new pass into games or other vfx, it's maybe not less human than we think because, as they said, you have to feed the IA with your vision, it's illustrates the need of references
thats sick! a little bit to green in the tech demo (gta v) but with another pictures it will be awesome
In the future developers will create games with basic shapes like spheres, flats, cubes, ragdolls with corresponding tags like wall, ball, desk, "person with blue jeans, red shirt, blond hair, glasses" etc. and train the data with petabytes of real life images/videos. The ai will "pre-render" this information and apply the pretrained data on the fly on the customers computer, providing a realtime, high-fps, ultra-high realistic graphics game. We already have the technology to do so(see DALL-E), it just needs to be combined, and made applicable for graphics cards. GPU's might have special cores for this kind of AI in the future and be able to rise the performance for such applications like Nvidia did with RTX within 2 generations of GPU's for example.
John Connor is gonna stop that shit
They should use this in the Triangle club, I'm sure they have lots of appropriate data sets at their disposal.
Very interesting work! Also funny to me is that the synthetic rendered version looks like GTA4's New York City.
m$ buy them now!!! Now this will be a VR dream from heaven!!!!!
amazing job
I have to wonder if human visual perception is also based on some aggregation of prior experience rather than strictly the view itself. It would seem so when you consider optical illusions.
I am not into the topic very well but how does this AI able to read texture map information from the game by useing video footage? The video is not real time anymore, so isn't hooked into the game engine to read normal map data and so on.
Love It!, is it possible to make it in real time!
This version is not optimized for speed at all. I don't see any hard constraints in principle.
@@stephanrichter7028 You wouldn't need to. All you need is to use this system to colorize assets/textures and choose attributes like reflectivity for them. Then build a regular game using those assets. The correct color grading and reflection levels baked into the assets. I think games mostly look wrong, because they pick just wrong colors for stuff (too saturated, mostly). RTX can create good reflections, if AI tells it how much each surface should reflect
What does it look like when you get out of the car & start walking around? It would be nice to see what it looks like in cutscenes with close ups to faces as well.
On 1 hand I'm amazed on the other hand I start to wonder if the appeal of playing video games will be lost by the realism. I play to escape the real world, I hope mimicking the real world to this extent doesn't remove the escapism.
Also, I like adventuring through fantasy worlds, how can they get real video of a fantasy? If they can somehow record video from our brain during dreams?
@Kenneth Yes, and I love their battle to be able to reach realism 1st. It brings us some cool looking fantasy worlds of entertainment. Just wondering if when they do reach it, will it lose it's allure as something different.
To preface this, your work is impressive and I am psyched about it. My question is why you did it. What was your incentive to improve upon the previous papers and to accomplish this result?
Any hope to get it in realtime in near future ?
Yes!
How could we test this out ourselves?
Amazing!.
Such a fantastic results!
how you install this on GTA V ?
А как я например могу это применить? Это закрытый софт?
can you turn 30km of google 360 street view into a 3d environment?
License this to Nvidia?
clear thought! very helpful
how long have you been working on this, when did you start at this
How much lag does this add?
One video on character models plz...
So... How we can use it? Or it's just for showing, not for using?
The best MOD ever 🤯
Very interesting! I can't help but imagine this is what rockstar is doing for GTA 6
What happens if you leave the car?
What setup it's required to run this?
fantastic work!
Can i install this??Is this available to download?
@PrimaryHades can you share the code you download 5 month agp, maybe i can do something about it
We need this in Assetto Corsa
Please try this with the game BeamNG Drive
There is very realistic car damage
I think it will be interesting
Yes of course that would be nice
I would rather see it tested in something that has top graphics (like Forza Horizon 4 or RDR2), and see how much it can be pushed further. BeamNG doesn't look that great
@@shredd5705 But BeamNG has the world's best car damage and deformation physics.
@@fylhtq7779 don
if the GTA V post processing would be available as a mod for GTA, i would rather pay for that, than for NVE or redux or any of those.
How can i install in GTA 5? On the Code&data is WebSite files
You can't and even if you could it won't work in real time aka it might be 1 frame every 10 mins (a random guess) there is no way in hell that is working at a playable framerate
have you download the code a few month ago? can you share the files?
Looks great
The graphics timecyc is like gta iv
I’m tripping so hard right now
Please make this as a mod
👏👏👏👏 Very good 👏👏👏👏
oyunun orjinalini tercih ederim ama değişime uğramış olması ilginç
So, basically you're making a digital "unconscious collective"?
pretty much
when intel release this
nvidia : can i buy your project for 30 rtx 3090?
This need to be tested on an older driving game lie The Job or GTA 3
You need to get more colours in the GTA 5 its cool but a little Green for me
tnx for sharing
Ray tracing mal chegou já ta sendo passado 😂🤣 eu não to no ritmo do mundo mesmo hahahaha foda morar no Brazil.
A gente vai demorar uns 8 - 10 anos pra ver isso no Brasil, e até lá os americano vão tar mais evoluido ainda
Como ray tracing mal chegou? Ray tracing existe desde os anos 80.
E vc pelo jeito sequer sabe o q é ray tracing, tampouco entendeu algo desse vídeo, e se entendesse veria q isso sequer é aplicável
intel gaming edition
You should use footage from better cameras. Automotive dashcams are poor quality, their photorealism is...well, poor quality.
Watch Intel build a GPU with this running...they will make millions off the bat this will end AMD and Nvidia just look at the clouds thats as close as we will get to reality in a game hands down
it's uncanny
I'm not entirely convinced that road surfaces in the US are as smooth as those in Germany. On the photos in geoguessr or Google Streetview they sure look different. Frankly I think the GTA original footage looks more realistic than yours.
map is based on the real city of Los Angeles, California no europe
Si o no ps5 Gta 5
Aguante el htaa
fake tech bad skills
Hand-crafting all the individual frames would near impossible. What's your big idea?