If you're keen to try out the mods shown in the video, be sure to check out the video description where I've left links for you to check out. In the meantime, episode 62 on DLSS is coming your way very soon. I hope you enjoyed this episode, it's an interesting change of pace from the regular stuff we look at around here.
Just to let you know, great approach to this material. I usually get insight into these types of things on Digital Foundry, and I felt that you filled in some gaps for me! I'd guess this was achieved by presenting it from more of a gamer's perspective...
Just one small error in your Video. You suggest that DLSS needs to be trained for a specific game, afaik this was only true for early versions of DLSS.
One of the things I loved about Doom 3 is that the engine would sort the level's textures according to frequency of use before applying the compression necessary to fit them in a 64 meg texture budget. So the most-used textures (like a corridoor wall texture) would get the least amount of compression and a one-shot texture (like the scanner at the start of the game) would get the most. It really made the game look terrific.
@@Earo16 I think the book was called "The Making of Doom 3" by Steven L Kent. I haven't read it but the folks who quoted from it had some interesting trivia about the game. Id needed to fit the game's textures into a 64 meg texture budget because they wanted to release the game on XBox, as well as PC. (Even though most PC users had 128/256meg graphics cards at the time). As a result of that, there was some cut content. In the "Welcome to Mars City" level, there was a missing chapel and the Player character had to find his living quarters, before the game continued. By cutting that content, they freed up just enough resources to add the infirmary (where you see the injured marine being treated).
This was a super cool video thanks for sharing. It's clear you put a lot of time/research into it, and writing the script in a way players can understand!
I'been watching the channel Two Minute Papers here on UA-cam for quite some time now and it mind is blown regularly. The technology being created surpasses even recent science fiction. Up until maybe a couple hundred years ago society changing technology would come about every several generations but now it seems to happen at least once a decade. In some fashion this is also apparent in the evolution of video games over the console generations. Man what a time to be alive!
This was a bit of a left-field topic, but one that *absolutely* applies when you consider this channel is about A.I and games! Who's to say this should always refer to the A.I that governs how games operate? ;)
Whenever anyone talks about remasters, my mind immediately starts thinking about MGS. MGS3 subsistence would be my top choice (although a remake in the fox engine would be better) just because MGS3, but MGS1 probably needs it more.
Just found your channel as Im interested in learning how Ai remasters old video footage? Then I thought how Awesome it would be to have a higher quality version of Kingpin, I loved that game and Total war Shogun!!!!
Yes, a neural network could be trained to refine and increase the detail of a mesh beyond simply subdividing it. It'd be prudent to create different training sets for Humans, Organics, Machinery, etc, and could be further helped along by reading the texture / normal maps. That said, there's a limit to what can be realistically be done. I can't imagine that Mario as he appears in Super Mario 64 could ever be automatically updated to resemble the Mario of Super Mario Odyssey, but I do think neural networks could be used to bring a 2013 character up to 2021 standards. I think it's important to remember that the models of yesteryear aren't simply high-res models that have been scaled down to fit the limitations of a given console; but instead specifically created for those limitations. To go further back to retro Arcade games- pixel artists leveraged the bleeding of pixels on CRT screens to their advantage, and many arcade games appear to lose detail when displayed on modern monitors because the pixels don't bleed; so the colors don't mix. This is similarly true with early 3D models- They're designed to be low-poly, and more detail may not be an improvement.
@@anteshell True But hightmap for tesselation is created by humans :-) Now we can generate hightmaps by ML so tesselated objects would have more details.
@@igorthelight Not true. Tesselation is not made with a height map but with a shader in particular place in the rendering pipeline. The shader for creating the tesselation is written by humans and that is mostly due to the fact that it needs to be configures game by game basis depending on the model complexity and so on. However, my point was not to say anything about creating that shader but rather the process of tesselating itself. There is no human in that process. Also, there are generalized tesselation algorithms available that pretty much cuts the human interaction to zero, if not counting placing it in the right place. A height map can be used to feed data for the algorithm, but is not required. Besides, nowdays height maps are rarely made by hand.
Kind of surprised there wasn't much consideration of the pushback from artists. Artists on older games designed the art to work for low resolutions and different screens - they knew what they were making and adjusted for it, the same way cinematographers accounted for black-and-white film. You can't get out what the art might have looked like if it was made in a different medium, because the artist would have done things differently. The set for the Adam's Family was designed with bizarre colours so they'd show up well on film - if you recreated the colours from that, it would look awful. If you recreated the colours based on what you think the intended effect was, you're just guessing. That would be fine if mistakes were always obviously wrong, but sometimes they camouflage themselves well. Take the problems of StyleGAN - it creates fairly realistic human faces, but biases towards white features. This led to much hilarity when the PULSE Face Depixelizer, using StyleGAN, was shown to upscale a lowres Barak Obama into a poorly-lit blue-eyed, pink-lipped white dude. Pretty convincing face if you didn't know who it was supposed to be! It's not a problem when the AI interprets a tiny texture on a soda can as baked beans if it's just used for piles of trash, but becomes concerning when, after publishing it, it becomes clear it was also used on a desk in an office or bedroom, with the worrying implication that whoever sat there was drinking beans straight from the tin. But I'm more worried about negligent publishers throwing their old PS1/2 games at a dev team who never worked on them and telling them it's being released Q3, just get it done, the GAN knows what it's doing.
While I think part of the fun for developers is to make these textures faithful to the original, I can see what you mean. This could possibly be an issue for some, yes. I think that it can also be a welcome addition to devs who wish they had created larger textures in the first place.
@@florianhoste6969 Yeah, the number of custom HD texture packs for old games shows dedication can have amazing results, don't get me wrong. All remakes and remasters have challenges like these. I'm not against them by any means and AI upscaling can be amazing. It's just a little odd to not mention these angles in the video.
I can only speak for Metroid Prime here, but a big problem with the AI upscaled textures for that game is that they tend to be massive, poorly optimized, and contain a ton of artifacts. There's another project working on manually recreating the Prime series' textures at a higher resolution, which promises smaller file sizes and (imo) looks much better, even if it's currently incomplete.
This presentation was very well done. Instant Sub. I can't wait until one day we have super AI scaler dongles that give us exactly what we want in real-time for any console.
the whole video is about how this thing isnt really possible. The information is not there, you see. upscaling only really works when the game was prepared for upscaling at development.
Really cool stuff. Excited for the DLSS episode -- never even heard of it before but it looks incredibly promising as a way to downsize game files and texture load times
Interesting that it uses a comparative GAN. I thought it would use a variation on an Autoencoder trained to transform downscaled images to their pre-downscaling form.
Imagine if they merged this kind of thing with something like Nvidia Canvas and used it on older games like ps1 or ps2. They could take the baron landscapes and turn them into something amazing. Or train an AI with a modern remake of an old game, like the Crash N Sane remake and it's original PS1 games, so that you might be able to feed it something like the original crash and it spits out something looking more like the remake.
I'm a narrative and game designer. I love your channel mostly for A.I. in games, but I love to keep an eye on all new tech in a casual way. You said you have resources for the more scholarly, and thats me, just not in this field.
@@RobNomad True. In most cases you may force it in driver options. But there are modern implementations or old game engines where it works without additional headaches :-)
Funny enough I'm playing the diablo 2 remaster while playing this and I don't think publishers realize just how much money there is to be made by just making a clone of the old game (with major bugs/glitches/crashes fixed ofc) and putting on a new 21st century coat of paint. I'd buy so many older games again if they got HD remakes.
True. But vector graphics needs more computational power (every frame) because every detail is not just an array of pixels - it's a mathematical equation now ;-)
@@igorthelight ah..true. You'd need much less V-Ram, but one hell of a monster CPU. A hybrid system may work though, what if the textures were stored as vectors, but baked as rasters at the required resolution.
@@AIandGames Except some strange "mood changes" like different time of day or weather on a planet we visit in the games. Those changes are mostly negative in my opinion.
The upscaling on Doom 1 honestly still looks pretty bad. :/ Even with all the manual clean-up. It looks like it took a lot of work and I'm sorry. I feel like retro 3D is best treated like pixel art, but I might be alone in that. If the original game's visuals were designed with anti-aliasing in mind, that's when upscaling begins to look good to me.
Might just be me, bur the first image "upgrades looked rubbish (the two guys and the doom one). Seems like a nice idea but given the examples I guess it has mixed results?
📢Bro i No this is an older video but please Get ahold of Rock star And break down and explain this stuff to them👍💯 For their grand theft auto trilogy Definitive addition
The Mass Effect remaster is probably the worst example that you could've possibly came up with to show how AI upscaling can preserve the artists' original vision.
Sorry, but you definitely need to read more into how 3d art works. Saying "1080p texture" is completely incorrect. Screen resolution has nothing to do with texture resolution. High resolution screens may allow you to better see detail, but it does not "stretch" textures or manipulate them in any form. A texture that is considered "4k" is not "3840x2160" like your monitor, it is "4096x4096" which is a power-of-two texture. They need to be power-of-two for texture streaming.
If you're keen to try out the mods shown in the video, be sure to check out the video description where I've left links for you to check out.
In the meantime, episode 62 on DLSS is coming your way very soon. I hope you enjoyed this episode, it's an interesting change of pace from the regular stuff we look at around here.
Just to let you know, great approach to this material. I usually get insight into these types of things on Digital Foundry, and I felt that you filled in some gaps for me!
I'd guess this was achieved by presenting it from more of a gamer's perspective...
Just one small error in your Video. You suggest that DLSS needs to be trained for a specific game, afaik this was only true for early versions of DLSS.
@@MrNubix Indeed, as of version 2.0, the deep neural network is no longer trained per game. I explain that in detail in the DLSS video.
@@AIandGames
On it Captain.
One of the things I loved about Doom 3 is that the engine would sort the level's textures according to frequency of use before applying the compression necessary to fit them in a 64 meg texture budget. So the most-used textures (like a corridoor wall texture) would get the least amount of compression and a one-shot texture (like the scanner at the start of the game) would get the most. It really made the game look terrific.
@@Earo16 I think the book was called "The Making of Doom 3" by Steven L Kent. I haven't read it but the folks who quoted from it had some interesting trivia about the game. Id needed to fit the game's textures into a 64 meg texture budget because they wanted to release the game on XBox, as well as PC. (Even though most PC users had 128/256meg graphics cards at the time). As a result of that, there was some cut content. In the "Welcome to Mars City" level, there was a missing chapel and the Player character had to find his living quarters, before the game continued. By cutting that content, they freed up just enough resources to add the infirmary (where you see the injured marine being treated).
This was a super cool video thanks for sharing. It's clear you put a lot of time/research into it, and writing the script in a way players can understand!
Didn't Expect a SWTOR connoisseur in these comments
there is really like 3 piece of information crumb for the length of 20 minutes.
So you could say that using an AI requires alot of... Calibrations.
I'been watching the channel Two Minute Papers here on UA-cam for quite some time now and it mind is blown regularly. The technology being created surpasses even recent science fiction. Up until maybe a couple hundred years ago society changing technology would come about every several generations but now it seems to happen at least once a decade. In some fashion this is also apparent in the evolution of video games over the console generations. Man what a time to be alive!
It has been so long since UA-cam recommended one of your videos, I was checking my subscriptions and found this.
Great video.
This was a bit of a left-field topic, but one that *absolutely* applies when you consider this channel is about A.I and games! Who's to say this should always refer to the A.I that governs how games operate? ;)
8:15 - "Neural networks"
What I heard: "noodle network"
Me: giggles and scrolls back to check I heard that right.
This is great content, people should be more aware of this. I would love to play MGS1 with this enhancement!
Whenever anyone talks about remasters, my mind immediately starts thinking about MGS. MGS3 subsistence would be my top choice (although a remake in the fox engine would be better) just because MGS3, but MGS1 probably needs it more.
@@Lord_of_Dread;)
Just found your channel as Im interested in learning how Ai remasters old video footage?
Then I thought how Awesome it would be to have a higher quality version of Kingpin, I loved that game and Total war Shogun!!!!
Could this type of upscaling possibly be done on meshes? I don't know much about this topic.
Great video !
Yes, a neural network could be trained to refine and increase the detail of a mesh beyond simply subdividing it. It'd be prudent to create different training sets for Humans, Organics, Machinery, etc, and could be further helped along by reading the texture / normal maps. That said, there's a limit to what can be realistically be done. I can't imagine that Mario as he appears in Super Mario 64 could ever be automatically updated to resemble the Mario of Super Mario Odyssey, but I do think neural networks could be used to bring a 2013 character up to 2021 standards.
I think it's important to remember that the models of yesteryear aren't simply high-res models that have been scaled down to fit the limitations of a given console; but instead specifically created for those limitations. To go further back to retro Arcade games- pixel artists leveraged the bleeding of pixels on CRT screens to their advantage, and many arcade games appear to lose detail when displayed on modern monitors because the pixels don't bleed; so the colors don't mix. This is similarly true with early 3D models- They're designed to be low-poly, and more detail may not be an improvement.
That is what tesselation essentially is, and it is available in almost all modern games after around 2011.
@@anteshell True
But hightmap for tesselation is created by humans :-)
Now we can generate hightmaps by ML so tesselated objects would have more details.
@@igorthelight Not true. Tesselation is not made with a height map but with a shader in particular place in the rendering pipeline.
The shader for creating the tesselation is written by humans and that is mostly due to the fact that it needs to be configures game by game basis depending on the model complexity and so on.
However, my point was not to say anything about creating that shader but rather the process of tesselating itself. There is no human in that process. Also, there are generalized tesselation algorithms available that pretty much cuts the human interaction to zero, if not counting placing it in the right place.
A height map can be used to feed data for the algorithm, but is not required. Besides, nowdays height maps are rarely made by hand.
@@anteshell Thanks for the info!
Kind of surprised there wasn't much consideration of the pushback from artists.
Artists on older games designed the art to work for low resolutions and different screens - they knew what they were making and adjusted for it, the same way cinematographers accounted for black-and-white film. You can't get out what the art might have looked like if it was made in a different medium, because the artist would have done things differently. The set for the Adam's Family was designed with bizarre colours so they'd show up well on film - if you recreated the colours from that, it would look awful. If you recreated the colours based on what you think the intended effect was, you're just guessing.
That would be fine if mistakes were always obviously wrong, but sometimes they camouflage themselves well. Take the problems of StyleGAN - it creates fairly realistic human faces, but biases towards white features. This led to much hilarity when the PULSE Face Depixelizer, using StyleGAN, was shown to upscale a lowres Barak Obama into a poorly-lit blue-eyed, pink-lipped white dude. Pretty convincing face if you didn't know who it was supposed to be!
It's not a problem when the AI interprets a tiny texture on a soda can as baked beans if it's just used for piles of trash, but becomes concerning when, after publishing it, it becomes clear it was also used on a desk in an office or bedroom, with the worrying implication that whoever sat there was drinking beans straight from the tin.
But I'm more worried about negligent publishers throwing their old PS1/2 games at a dev team who never worked on them and telling them it's being released Q3, just get it done, the GAN knows what it's doing.
While I think part of the fun for developers is to make these textures faithful to the original, I can see what you mean. This could possibly be an issue for some, yes. I think that it can also be a welcome addition to devs who wish they had created larger textures in the first place.
this was very well said
@@florianhoste6969 Yeah, the number of custom HD texture packs for old games shows dedication can have amazing results, don't get me wrong. All remakes and remasters have challenges like these. I'm not against them by any means and AI upscaling can be amazing. It's just a little odd to not mention these angles in the video.
I can only speak for Metroid Prime here, but a big problem with the AI upscaled textures for that game is that they tend to be massive, poorly optimized, and contain a ton of artifacts. There's another project working on manually recreating the Prime series' textures at a higher resolution, which promises smaller file sizes and (imo) looks much better, even if it's currently incomplete.
you mannaged to explain dlss in a one min throw away segment far better than any of the 10 min lectures ive seen manage i actualy get it now thank you
What I appreciate is how well Bioware was able to create 'Rounded Rear Curves' 17 years ago. True Artistry! ^_~
This presentation was very well done. Instant Sub. I can't wait until one day we have super AI scaler dongles that give us exactly what we want in real-time for any console.
the whole video is about how this thing isnt really possible. The information is not there, you see. upscaling only really works when the game was prepared for upscaling at development.
@@Omertahun I strongly disagree. I think the future will surprise you.
we want more videos like this one
Really cool stuff. Excited for the DLSS episode -- never even heard of it before but it looks incredibly promising as a way to downsize game files and texture load times
I was interested in this fir a while now & u did a great job explaining thanks!
Interesting that it uses a comparative GAN. I thought it would use a variation on an Autoencoder trained to transform downscaled images to their pre-downscaling form.
The possibilities of deep learning texture upscaling algorithms are exciting.
As the second oldest A I channel, your channel being the oldest, I really enjoy your content.
I saw that Xbox swap out! Lmao, good stuff.
I wonder if it will be ever possible to apply tessellation or map displacements to specific modded textures.
Anybody know any particular sites to get texture packs from
Cool! I'm watching this while upscaling Red Dead Redemption(PC) textures! xD
Imagine if they merged this kind of thing with something like Nvidia Canvas and used it on older games like ps1 or ps2. They could take the baron landscapes and turn them into something amazing. Or train an AI with a modern remake of an old game, like the Crash N Sane remake and it's original PS1 games, so that you might be able to feed it something like the original crash and it spits out something looking more like the remake.
What about the Hitman series from Codename 47 to Blood Money? Those missions need the Remaster treatment!
have been watching 2 min papers for quite some time, but just where can I grab a GAN model to tweak myself?
Google for them - some of them are free on Github as far as I know ;-)
I'd really like to do this to some Splinter Cell games. Have zero idea on how to do it though lol. I'll check out some of the links.
I'm a narrative and game designer. I love your channel mostly for A.I. in games, but I love to keep an eye on all new tech in a casual way. You said you have resources for the more scholarly, and thats me, just not in this field.
Liking, commenting, and just subscribed!
Fascinating stuff, thanks for the video.
do you force 16x anisotropic filtering on all games in the nvidia control panel? i’m looking at morrowind in the first part of the video.
You may do that.
Or just use OpenMW - a more modern recreation of Morrowind's engine :-)
@@igorthelight yes, i meant that anisotropic is useful in many older games that didn’t have it in the options menu.
@@RobNomad True.
In most cases you may force it in driver options. But there are modern implementations or old game engines where it works without additional headaches :-)
@@igorthelight i always leave 16x activated in the drivers options, there are no drawbacks
@@RobNomad True.
Videocard must be VERY old to take penalties for using AF 16x
I'm talking about 3dfx/Voodoo era :-)
so what's keeping this same sort of thing from being applied to geometry?
Nothing.
It's just someone have to do that ;-)
I recently upscaled all the textures for The Thing 2002 using A.I. If anyone wants I can give a link to the upscaled texture pack :)
Great video!
So when will we have a real time overlay that reinterpretes each frame and doesnt care if its 2d or 3d?
Funny enough I'm playing the diablo 2 remaster while playing this and I don't think publishers realize just how much money there is to be made by just making a clone of the old game (with major bugs/glitches/crashes fixed ofc) and putting on a new 21st century coat of paint. I'd buy so many older games again if they got HD remakes.
The artists orginal intent vs CDs 700MB size?
I wish we could use vector graphics for textures, then we wouldn't need to worry about scaling changing resolution.
True.
But vector graphics needs more computational power (every frame) because every detail is not just an array of pixels - it's a mathematical equation now ;-)
@@igorthelight ah..true. You'd need much less V-Ram, but one hell of a monster CPU. A hybrid system may work though, what if the textures were stored as vectors, but baked as rasters at the required resolution.
@@MoonlightFox I heard about that somewhere...
Not sure is that worked or not - someone may try to make it :-)
DLSS sounds like an interesting piece of technology also the Mass Effect Legendary Edition looks amazing.
I've only played the first game so far (all three games are included) but it looks fantastic. They've done a great job.
@@AIandGames Except some strange "mood changes" like different time of day or weather on a planet we visit in the games.
Those changes are mostly negative in my opinion.
Great video
Great video Thanks
Thanks for the vid
Those Morrowind textures used as an example at around 3:20 don't actually look that bad, and it detracts from the point of example for me.
Very interesting... thx
I saw you try to sneak the xbox one in there for a second when you said "original xbox" you sneaky sneaker you
some of the ai upscaled texture mods look weird like on old games clothes look weird
6:17 Is that Tom Cruise lol, honestly its funny how I think it kinda looks like him
This is something I've wanted to do but haven't had the chance yet
The upscaling on Doom 1 honestly still looks pretty bad. :/ Even with all the manual clean-up. It looks like it took a lot of work and I'm sorry.
I feel like retro 3D is best treated like pixel art, but I might be alone in that. If the original game's visuals were designed with anti-aliasing in mind, that's when upscaling begins to look good to me.
I just wish someone could do it for the original dishonored, man that would rule
Videos like this are a stark reminder of how many people are much smarter than me 😖
Might just be me, bur the first image "upgrades looked rubbish (the two guys and the doom one). Seems like a nice idea but given the examples I guess it has mixed results?
📢Bro i No this is an older video but please Get ahold of Rock star And break down and explain this stuff to them👍💯 For their grand theft auto trilogy Definitive addition
that was funny @3:47 lol
can u upscale AION CLASSIC? :D
Well now i understand why it should have Been tested more for gta….
ua-cam.com/video/Fix6u4pksrg/v-deo.html Was that Xbox picture swapping a subtle jab at the confusing naming scheme of the consoles? 😁
Could u pls try it with Hitman?
I'm all for upscaling textures in older 3D games. But think classic/retro sprite art games should be left alone. just my two pennies worth.
AI is magic
"Wouldnt usually recommend reading reddit" sir how dare you.
@AI and Games PLS correct disciples 2 :c
A lot has changed, I wonder what is being done on 2024
I have a video on the to-do list about how God of War: Ragnarok does texture upscaling at runtime.
That xbox joke, GOLD.
Hello to Jason Isaacs
kermode gang worldwide
The Mass Effect remaster is probably the worst example that you could've possibly came up with to show how AI upscaling can preserve the artists' original vision.
I think textures are good.
But "mood changes" on many game levels are really... unfortunate :-)
Thumbs Down
You didnt "Upcale" Deus Ex , you just used low and high settings
You're adorable 🤣
Just interesting how much of a bottleneck hardware was then, 32 MG VRAM graphics cards couldn't pull much.
LOL the lack of the aspect ratio fix in Max Payne is atrocious. If you couldn't patch it, at least do the proper 4:3 and leave black bars.
I found this video through a ff14 upscale mod.
Loved your accent
Cool.
Jesus Christ it’s here boys
Now compare this to that shitty ai upscaling in GTA remastered 😂
You could say this video contains 'Deep Learning'! bdum tish...Sorry I'll see myself out
That was terrible, but I'll allow it.
Sorry, but you definitely need to read more into how 3d art works. Saying "1080p texture" is completely incorrect. Screen resolution has nothing to do with texture resolution. High resolution screens may allow you to better see detail, but it does not "stretch" textures or manipulate them in any form. A texture that is considered "4k" is not "3840x2160" like your monitor, it is "4096x4096" which is a power-of-two texture. They need to be power-of-two for texture streaming.
Up scaling textures only fixes half the problem,since you're usually still working with the same low-poly model as before
Soon, models could also be upscaled by ML :-)
Animations could be improved by key frames interpolations (simple math - no ML at all, but it may help)
@@StefanGhioci Interpolation could be non-linear if you want them to ;-)
@@StefanGhioci There will be :-)
4:33 40Gb RAM, dam
Yeah.... heh, that was my old machine too.
JUST GIMME GTA IV UPSCALE VANILLA TEXTURES AAAAAAH
Ah.... The Max payne remaster is a modd... Fair enough
Zoom and enhance.
Tha hell is this accent I've never heard anything like that before
Go watch Lord of the Rings. He's from the Shire obviously.
Is the accent south african?
first, and a great video.
My dream is to see SNK Samurai Shodown games upscaled, and depixalated. I love SNK games.
Great video
Cool.