I would so just try to utterly break the Network. I'd be like: "Computer, generate Elmo's Number adventure, but make it an open world third person RPG that crosses over into the Doom universe. Replace Big Bird with Count Chocula, and deep fry all the textures."
Hey nice video! After watching countless rtx 4000 DLSS 3 videos being pretty much the same information, this video in comparison differd and gave me an actual idea how DLSS 3.0 may work. Thanks a lot phillip
IMO nVidia is not so dumb that DLSS 3 would look anywhere close to videos in this one, but the responsiveness issue simply cannot be avoided unless the AI is smart enough to build a mini time travel machine inside each card.
DLSS 3 injects FAKE FRAMES to give a FAKE higher FPS. --> DON'T BELIEVE NGREEDIA'S MARKETING BULLSH#T...!!!! You eyes will be opened to the depth of Nvidia's anti-consumer practices once the reviews are out...practices we already know such as rebranding a 4070 as a much MUCH more expensive 3080 12GB.....!!!!
@@marceelino So it's not useless, considering the other uses you mentioned. Also for gamers who want to experience scenic visuals in exchange for a bit of latency. Not everyone plays games that demand very low input lag, so I don't know why people are saying that DLSS 3 is a failure.
Input lag was the first thing that popped into my mind when they said they were going to interplolate frames. I hope there's an option where you can easily disable that and only upscale the image, especially for in competitive games that are already well optimized.
The latency hit would make this unusable for VR, where that lag becomes something that causes nausea instead of just a mild annoyance depending on the game
Unless it was somehow in the single digits of milliseconds, completely agree. For VR, we just need more actual raw power, faking it can only get us so far.
Many games utilize asw which is frame rate interpolation. Some quest 2 games already run at 60 fps and get interpolated up to 120hz On PC the same thing happens, when your PC can't keep up with 90fps, it switches to 45 fps and frame rate interpolation is used
VR has had this tech on the oculus side of things since the original rift It's called ASW (Asynchronous Space Warp) and what it does is if your FPS can't match your headsets refresh rate then it will lock your framerate at half the refresh rate, so 45 for 90hz, 60 for 120hz, etc. and then it will interpret every other frame, I'm not 100% sure if ASW uses AI/Neural networks or not
Don't let this distract you from the fact the shader gap between the "4080" 12gb and 16 is larger than the gap between the 3080 and 3090ti that all used the same 102 die. Jensen trying to pad his wallet even more.
@UCEKJKJ3FO-9SFv5x5BzyxhQ I think they meant the 3080 and 3090ti used the same die, not that the 4080 and 4080 use the same die. According to the tech power up GPU database the 3080 and 3090tu both use the GA102 die Edit: he changed his comment while I was writing mine. Now mine doesn't make as much sense
@@2kliksphilip Looking around it's believed to be AD102, AD103 and AD104 for the RTX 4078 $899; interestingly VideoCardz thinks the 4080 uses 22.5GHz GDDR6x which is faster than the other models.
I feel this is similar to how supermarkets no longer dont stock 4 packs of beans and force you to buy them individually so overall you pay more. If DLSS 3.0 is just a marketing headline, these new GPU's are terrible value. Will be interesting to see how AMD responds as their cards were really competitive last gen.
They allready did by Lowering 6900xt to 780€ comapred to the 1100€ for the 12gb 4080 with more or less similar Raster Performance by looking at the Nvidia Benchmarks of RE:8, AC Valhalla and Division 2
Definitely in the same boat with you Philip with knowing about the wonders of frame interpolation but never thinking it could be applied in real time to games like this. But this is honestly the most comprehensive video I've found on DLSS so far, not even big journalists have broken it down as well as you have here. So hats off. Great vid. 🎉
@@2kliksphilip yeah we'll have to see. Hopefully you plan on covering the DLSS 3 situation more as it develops I'm really looking forward to that. And for the AI detail/colour overlay weirdness that the GTA example shows off. Really looking forward to stuff like that. But I am going to stand on the positive side of things here because I think if you're used to say 60fps and turn it on to get 120fps with the latency of 60fps you'll enjoy it. If you're used to 120fps and want to play a harder to run game with DLSS 3 to get 120fps you're probably going to find it laggy at best and sickening at worst. But time will tell. 🙂
@@2kliksphilip Reflex only help in situations where one is GPU limited (GPU Usage=100%), it doesn't decrease latency at all in CPU limited scenarios. Setting a framerate cap to limit GPU usage has the same effects on input latency as reflex does. I have seen ideas about this a while back on some forums but I never thought nvidia would come up with it out of the blue. There will be a delay added that cannot be circumvented in any way. My guess is it will be at least 2 frames.
@@eniff2925 just checked out the Digital Foundry DLSS 3 first showcase video. And they went over latency in scenarios where they are CPU limited and GPU limited (before turning on DLSS 3 frame generation) And whilst Nvidia are very careful about what they allowed Digital Foundry to show in that video. FPS numbers being a no no for example. The latency is better than native (due to native being pretty bad as it's a low frame rate) and about on par with DLSS 2 latency without Nvidia Reflex. Looked really good latency wise, only thing to keep in mind if that whenever they showed DLSS 3 frame generation, they also had DLSS upscaling in performance mode, so are always working with quite a high FPS (100fps turned to 200fps in Spiderman for example), using frame generation to go from 40fps to 80fps it might be much more of a problem. And of course the latency isnt really changed at all when you have DLSS 3 frame generation and Nviida Reflex both on. So whilst latency doesn't get worse, you're not getting half the benefit of the higher FPS, which is less latency. So will 200fps in Spiderman with DLSS 3 actually feel like 200fps. No, it will just look like it. Weird tech. But super cool, hopefully it feels great to usw for those 10 people who can afford a £950 RTX 4070 ☠️
@@HoneyTwee I also watched it. They had DLSS 2 active which more than doubled the framerate in some cases. That was the reason for the decrease of latency. Turning DLSS 3 on top of that increases latency, quite heavily, but it is offset by DLSS 2. Compared to native, DLSS 3 should increase latency by at least 2 frames according to my logic. Reflex is not connected to this at all. It cannot help directly in this use case. Reflex was created in response to battlenonsense's findings about input latency drastically increasing in GPU bound scenarios. This input lag can be completely eleminated by setting an FPS cap so reflex is not very beneficial if you know what you are doing.
@@eniff2925 True, it would have been cool to see more about the latency penalties of DLSS 3, but it seems they were quite limited on what they could show. But if i'm not mistaken, the latency they reported with DLSS 3 + DLSS 2 + reflex was still slightly better than DLSS 2 on its own, and a lot better than native. So I assume reflex is making quite a lot of difference even if the overall latency compared to native is probably mostly coming from DLSS 2. Honestly though DLSS 3 looked good from what they showed, I don't think its a make or break to have it or not yet, so i'll probably still get an AMD card like a RX 7800XT if it's £850 ish and is around a 4080 16G, but DLSS 3 looks like really promising technology that I will be glad to have by the time its in more games and hopefully by then AMD will have a competitor to it themselves.
With filming videos on smartphones we already have all the stuff filtered and processed while recording. So yes, I'm 100% with you on your last sentence.
One thing I really found unique about why this is the new DLSS 3.0 is because it's abusing the pipeline which contains motion vectors for the 3D scene already and is easy to update, maintain, and naturally slides into the DLSS framework without having to add more code into the project. It's all encapsulated into one single package rather than having two DLSS branches to integrate. Other frame interpolations have to predict and implement many other algorithms to create a "fake" 3D perspective in order to be able to keep the frame well interpolated in some 3D games. It creates those weird visual effects philip points out, but with DLSS's framework, they get the motion vector data from checking the velocity between the vertex data on each frame inside the vertex buffer. So that's why it's part of DLSS 3.0, because DLSS at it's core utilises motion vectors within it's algorithm, why bother creating an entirely new shader package for devs to integrate alongside DLSS 2.0 when they can just focus on installing the entire DLSS 3.0 umbrella. I think they're trying to extend DLSS more than just an upscale algorithm to more of a performance optimisation toolkit in the future. I feel like the only draw back to DLSS 3.0's interpolation isn't going to be weird visual artifacts, but more motion blur because it knows the motion of a scene with each frame and it isn't quite tuned to completely sharpen the image. The main issues we'll have is with interpolating transparent/fading objects, inconsistent performance, and weird race conditions when it comes to frame times in some titles. I can also imagine DLSS 3.0 being REALLY good for games which are frame-locked to 60 where when you try to break the frame limiter it messes with the physics/timesteps of the game, as it allows you to just get a smoother feel to the game rather than just the crappy locked framerate :) This probably won't impact the timestep calculations this way, and allows you to get a much smoother experience without breaking your entire gameplay experience. So emulators might benefit A LOT with this. Thanks for reading my tedtalk, I was interested into being a graphics programmer while I was studying in university but I gave up that path, but if you've got questions about things like this feel free to ask c:
In other words, allow 3000 series to use DLSS 3 and you’ll see less performance difference between 30 and 40 series. Not to mention nvidia is trying to sell x70 class gpu for x80 price.
Well it depends, because for Ray Tracing, there's like 4x as many RT cores as 30 series, so for RT performance, it'd still be well ahead, and they did show the difference between DLSS 3 and 2. It's also interesting that people keep saying it's a 70 class GPU at an 80 class price, because we will eventually have a 70 class GPU, and they've done stuff like this before, the 1060 3gb and 1060 6gb were really more a 1060 and 1060Ti because the 6gb had more of other things so it was faster in general. I also don't know why this is considered a bad thing because if they made this a 4070 at it's price people would have been more angry, this means the eventual 4070 will be cheaper
@@MrZodiac011 1060 3gb snd 6gb were much closer. in fact, no big performance difference as people claimed. These have much more cuda core difference. And just compare it to last gen. "4080" 12gb doesn't even beat 3090ti. 3070 beat 2080ti, 2070 beat 1080ti, 1070 beat 980ti, 970 beat 780ti.
@@Japixx no, we just dont have as much disposable income as you and are unable to spend at least $899 on a gpu with lackluster performance compared to previous generations. Furthurmore we wish to be able to have a relatively pricey gpu released only a few years ago be compatible with a software which seems to be very beneficial and amazing yet locked behind exclusivity deals
Couldn't stop laughing my ass off at 2:19, which made it impossible to hear what you were saying. Philip pls fix, Jack Torrence could get phased out of the next Shining if you're not careful xD
I'm not so sure about input latency being an acceptable trade-off for VR users. John Carmack talks at length about how the biggest hurdle getting VR to a usable quality was just reducing input latency, since delays are so noticeable and intolerable with a HMD. DLSS might not take VR back out of that acceptable range, but it may be the case that like competitive games, input latency for any VR games becomes a very sensitive benchmark. You can already see it when you look up benchmarks for VR setups, frametime is a oft included stat.
VR apps may even care about high frame rate less because it's smoother but more because higher frame rate lessens input lag, since you are waiting not as long for a new frame. But DLSS 3 does the opposite, it makes the input lag longer than it would have been with lower (non-interpolated) frame rate.
@@cube2fox I'm not sure, when it comes to VR, we tend to be a lot more sensitive to what we see, hence why higher frame rates are a must, DLSS 3 adding higher latency might be noticeable a lot more with VR then standard gaming.
If they want to target VR, they'll probably have to add predictive resolve mode to the algorithm, as opposed to interpolative resolve for quality-sensitive applications. But it's possible to do it then without introducing latency that you don't already have with framewarp, or just a little, in like 2ms range, just the tiniest little fraction of a frame worth. But also absolute latency maybe isn't really THAT much of an issue. You can then just hide it by reprojection, so your view is updated at minimum possible latency on head turn, no added latency, while the game events and movement receive a frame of additional latency. It might just work and not be noticeable. Worth trying!
I LOVE THE SELF VIDEO FOR PROOF. Proves the point perfectly, yet makes me just laugh and enjoy your content even more. I hope they add dlss 3.0 to csgo so u can go redo the input lag and hitbox videos again =)
0:22 "Why hasn't this been done for games already?" It kinda has for VR. Asynchronous Space Warp with Meta headsets, Motion Smoothing on SteamVR headsets. It's actually a pretty good experience on the Valve Index to do fixed framerate motion smoothing at 72hz to 144hz. I didn't like any lower than that. And when I play on Oculus I disable it entirely cause I'm already adding a lot of latency with Virtual Desktop streaming.
I think it would be best to have a way to integrate the tech into the rendering pipeline so that the frame generation tech uses an unfinished frame + the last frame instead of 2 of the last frames. The paint drawing -> realistic image tech you showed at the end is probably a step in that direction.
I'm not sure but this sounds a lot like oculus's asynchronous space warp (ASW) which halves your frames and interpolates the other half. It works well in slow games or race games. But less so in fast games as you run into the same artifact issues explained in your video.
EDIT: I was corrected and the claims made in this comment aren't true, the presentation contradicts them. There are a few reasons that I believe that DLSS 3 is, in fact, predicting frames as you proposed at the beginning of this section, and not placing them in-between already generated frames as you seem to believe is the case. Firstly, the article about DLSS 3 on the Nvidia website has two images which show the DLSS 2 and 3 pipelines, including the framerate and input lag after dlss has done its magic. These pictures both show an improvement in input lag when DLSS 3 is used compared to DLSS 2. That may be Reflex working its magic even if DLSS 3 is still adding lag by not predicting, but I find that unlikely. Of course, that's just my not massively informed opinion, so that alone is not particularly swaying My second reason is that there is precedent for versions of framerate interpolation that predict the next frame to reduce input lag - those seen in VR. Oculus Spacewarp and Valve's solution both attempt to do this exact thing with mixed quality. It has been a couple years, I think, since the release of these two methods; so it's not entirely surprising to me that a new set of graphics cards, with additional hardware dedicated to extrapolating useful information from previous frames (the Optical Flow Accelerator mentioned in the article), could do this with better results than the slightly older VR solutions). Of course, it's very early days, and the existence of methods that can predict doesn't necessarily mean that DLSS will. Perhaps we don't have enough information to make a strong statement either way, but those are the reasons that I believe it is predicting frames, and that I think it is incorrect to suggest that it's placing frames inbetween previously generated frames. If I lack critical information found elsewhere, or misinterpreted some part of the article, or maybe even misunderstood the video, please someone let me know.
I did an experiment once. Because UA-cam didn't support 60fps at the time, but i wanted to upload smooth looking synthetic footage. So i captured at 60fps, and wrote an avisynth script which upinterpolated it to 150 or 180fps i forget, then filtered it down to 30. So added synthetic motion blur basically. And it looked decent! Here's what i'm thinking. Let's say you don't have VRR, say you have a TV running at 60Hz. It looks good, but your GPU can only hit real 60 only most of the time; but the occasional drops down to 56, you'd like to hide them, and maybe you don't have enough CPU or compute throughput to make it happen without taking too much of a compromise elsewhere. In that case i think frame resynthesis from motion, similar to how VR does it (generally running at something like 90Hz there) might be the ticket. It absolutely makes sense to integrate it into DLSS since the resynthesis can leverage the same motion vector, lower-resolution and accumulation data just to do this framerate independently instead of frame synchronously. A hunch that the limitation of DLSS3 to upcoming series of GPUs is artificial though.
As someone who despises frame interpolation on televisions, I really, REALLY hope that developers implement the image upscaling and FPS aspects of DLSS separately, because I think this FPS interpolation thing undermines THE WHOLE POINT of having higher framerates: more responsiveness, less input lag.
@@egesanl1 I have used their Linux drivers on a laptop and while they are certainly more clunky and not open source (compared to AMD), I didn't get the vibe that they were all that different from Windows ones.
@@tablettablete186 Those are the server grade cards right? I don't know much about them but honestly I wouldn't be surprised if they did have subscription based drivers, not all that uncommon in the enterprise world.
Based on the available information, I believe DLSS 3.0 could be useful on strong PCs with very high refresh rate displays - 240Hz or 360Hz. If you have a PC that can already run games at say 120 or 144FPS without drops with a decent resolution, which is already smooth in regards to latency, and then run it with the new DLSS in an attempt to reach the higher frame rate, achieving a smooth gameplay and supremely smooth visuals. However, I think anything below 120 FPS is too low for this kind of use, and I don't think it's going to work well in regards to the latency - as was mentioned in the video, it will be worse the worse the base FPS is. This is also one of the reasons why I think the mid range GPUs were not shown yet - they can't run the games natively at a good enough resolution/framerate for the new DLSS to provide good enough results. I wouldn't be surprised if, in the end, they won't have this feature available, perhaps with the exception of 4070.
I tried the DLSS mode they just added to flight sim to see if i could improve my performance, it didn't do anything as i think i'm CPU limited but it also introduced really ugly ghosting on everything so i wouldn't use it anyway, it's the first time i've used DLSS and i'm disappointed, it usually looks good in Digital Foundry videos so i'm guessing there's something going wrong with its implementation here or my setup. Also if you were to use that interpolation feature for VR wouldn't the input latency increase nausea?
I think how useful this feature will be in VR is overstated. The fact is that image latency has a massive impact on the VR experience and is actually one of the greatest challenges in bringing cheap VR to mass market. Latency in VR causes VR sickness.
This is neat and all, but according to the couples last years trend it will be used as an excuse for bad game optimization and other stupid things rather than being a nice bonus on top of a well made game.
I think the best use case is to set a frame rate (60/120/…) and when the framerate dips below that threshold dlss kicks in and adds additional frames to the existing ones to keep it as smooth as possible
I feel like the biggest point of playing at a high fps is the lower input lag, this maybe makes sense for playing at very low frame rate with a controller.
What I personally miss is having VR technology make it back into regular gaming. Asynchronous reprojection and all that stuff for example, is brilliant.
If you have 30fps and dlls 3.0 brings it up to 90fps ... you'll still have the input lag of playing at 30fps. So yes, you said it right ... in order for the "motion flow" to work in a pleasant way, you'll need a reasonable input fps to begin with.
Well, no. DLSS reduces resolution and improves performance. So 30 fps becomes 45. Then DLSS interpolates, so 45 fps will feel like 45 fps and look like 90.
It's like how DLSS 2 works in the spatial dimension, but now in the temporal dimension. Better results the larger the data set you feed it to begin with. So it does a great job when fed 1440p and asked to output 4k, less so when fed 720p and asked to output 1080p etc. Likewise DLSS3 should do a great job when fed 60 and outputting 120, not so when fed 30 and outputting 60. Which means, if you can achieve a native 1440p 60fps without DLSS, expect DLSS 3 to give you 4k 120fps that's of comparable quality to native 4k 120fps except at a quarter the performance demands. That's huge.
In fact, you will have MORE input lag, since interpolation needs to know the NEXT frame to insert a generated one, not the current one! And of course, all of this is artificially locked not only to NVIDIA GPUs, but to the 4xxx series specifically...
I would have preferred if they simply focused on upscaling and threw all that performance at upcaling instead. I dont like the added latency which is what im trying to mimize with DLSS in the first place. What they could have done with DLSS if if they told game engine to render only the color (no shadows or GI). This would give them a geometrically accurate representation of game world. They could than feed this to AI to generate a realisticly illuminated version with very noisy ray tracing added in for consistency. Rendering only the color is much faster than full render pass, you can test this with UE5 pretty easily. It also wont suffer from input lag of having frame interpolation. Another trick is foveated ray tracing. There are only tech demos of this available atm but i think it is amazing idea
I'm not sure they have done it as suggested. Flight Simulator was CPU bound and input lag isn't a big deal. But Digital Foundry have clips from Cyber Punk, past DLSS caused input lag, so I'd be surprised if Nvidia made the same mistake twice. Anyway I know 3rd party reviewers will investigate the latency angle to check it out.
@@HeloisGevit Because they have really powerful AI compute power to cover it up. I mean right now they are rendering completely artificial frames, very noisy ray tracing can help massively with this
The real reason nvidia is focusing on dlss 3 and gatekeeping it is because the power difference between the two gens are actually not as big and they need to hook customers somehow. If Amd comes up with a similar solution they would walk back and enable it on old gen just like freesync and gsync.
Hmm you mention generating new frames when there's a stutter in vr, but that sounds almost the same as the motion smoothing oculus/meta is doing. Asynchronous spacewarp does help with small stutters but can be very noticeable
SteamVR also does this. Async warp just uses the last frame and warps it to match where the headset is a frame later. There are lots of artifacts because AFAIK no motion vector data is being used, just the headset motion. DLSS has access to that plus the next frame also so it has a lot more data to work with.
No, because those games don't support the suite of data that DLSS 2 needs to function (and by extension, DLSS 3). For instance: motion vectors. These need to be fed to the GPU and so they require the game developer to implement DLSS.
@@callumsmile I don't know how the emulator would be able to add motion vectors among other stuff, it's a pretty complex concept for such old engines unfortunately.
The thing you describe (generating a fake frame when the game can't keep up) is already a built-in feature of the Oculus runtime called Asynchronous Space Warp (ASW). It's extrapolative instead of interpolative though. Interpolative is a total no-go in VR, extra latency makes you very sick very quickly.
The point about it only being used to produce frames when needed is interesting. I have a 4k 120hz display that I use for most games and would only want extra frames generated in order to hit that 120fps mark when needed. If I can already get 120fps without any generated frames, then that is what I want to see. But I'm not sure it will be work that way. So by turning on DLSS 3, I'm essentially limiting my real framerate to far below what I usually would and I'm losing out on input lag whenever my machine could in theory be maintaining 120fps without DLSS 3.
DLSS 3 is just a way for them to advertise performance gains in a way that we've never done so before. More frames used to always mean lower input delay due to each frame coming to the screen sooner. Now they can basically fake performance gains while still having the overall input lag of what the game looks like before they add the frames.
2:32 this is much less of a concern with DLSS frame generation, because it gets extra information from the game, like motion vectors. So it knows how things are moving between frames as well. Also like you said, the framerates are already fairly high in comparison to videos that interpolation would usually be used on and because it only seems to be generating 1 frame in between, any interpolation error will be corrected in the next frame, making it much less noticeable in motion. If you want to see what the artifacts look like though, watch Digital Foundrys DLSS3 preview and go frame by frame before Spiderman jumps out of the window.
That, and for reducing input lag, they might also be using player input in the interpolation process, like the proper frame interpolation the Quest 2 uses, where input is resampled right before the image is rendered.
A lot of TVs have a motion interpolation feature, but I think Samsung deserves a special mention here. They are the only TV brand to offer a low-latency, gaming-optimized motion interpolation feature called "Game Motion Plus". This works only if you send the TV a non-VRR 60 Hz signal, and also only has ~30ms of input delay.
Because your eyes get to enjoy a 120fps image which is way smoother to look at. It won't feel any worse than what you could achieve without it, so what's the downside?
Also, something not discussed, Nvidia reflex usually decreases input lag by more then a 60 fps frame, so using it in 60+ fps will probably still result in a lag reduction
Nvidia refers to the two separate functions of DLSS 3 as Super Resolution and Frame Generation, and in fact it seems to basically be DLSS 2 but with that new Frame Generation part, still relying on many of the same inputs (motion vectors and depth info, combined with an "optical flow field"). Since the Super Resolution part of DLSS 3 is still backwards compatible with RTX Turing and Ampere, presumably the two features can be toggled individually. Or at least I hope it does in case someone likes one or the other instead of both. Either way, while the prices can suck a dick (especially for people on Europe who will get screwed over extra hard), I genuinely hope Nvidia pulls off real time frame interpolation successfully. We are continuously moving to a philosophy of "render smarter, not harder", and this would be a massive step forward. I won't be buying a 4000 series card, but I'm excited to see this technology. I wasn't expecting this sort of thing until a few GPU generations away.
Smart TVs already do real-time motion interpolation, and Nvidia has made fancy hardware for it, I'm sure it will be high quality. But for interactive content I don't know if interpolation will ever be good, I'm hoping this is a step towards extrapolation
4:28 The thing with VR though is that although 100 ish fps is ideal, lags are very noticeable when you move your head around and can induce motion sickness, if its present even a little bit, so dont think this is ideal for VR either.
My prediction is that because DLSS already introduces more latency to frame rendering than native resolution is that with the newer Tensor cores they reduced the upscaling latency to such a degree that inserting a artificial frame doesnt increase the latency to such a extend that is noticeable. The new Optical Flow Accelerators are MUCH faster than whats inside Turing. So yeah of course it will be a fake 120FPS without the responsiveness of a 120FPS image. It looks smoother and feels smoother regardless.
Input lag is not the measure of framerate, the user experience is. And that is subjective. Some prefer smoothness, some prefer input speed, and we tailor ourselves around that preference. Many of us like both. This technology seems to support both. For example -- something like gsync like will have more input lag than without, but - for many, it will be worth it for the immersion of frame smoothness and consistency. Same deal here. And this is coming from a guy that hates tv interpolation, yet understands this is a different beast.
VR headsets use it too avoid nausea. Oculus calls it space warp the industry calls it reprojection. Some implementations are better than others it depends it they use raw motion vector data. They are introduce some level of artifacting and latency.
"DLSS, features that would be most beneficial on lowered hardware, reserved for only the latest and greatest" Feels like giving fabulously wealthy celebrities free stuff.
Let's say we have 2 frames, frame 1 and frame 2, and dlss3 is gonna create frame 1.5 and place it between these two frames. Now, the scene being rendered is of a clock hand (second hand) moving across the small 60 dots around a clock. Now, on frame 1, we see the hand is on the first dot (right under the number 12 on the clock). On frame 2 we see the hand is on the 3rd dot, meaning the game engine didn't generate the frame on which the hand goes on the second dot (due to the gpu not being powerful enough). Now, DLSS3 sees these 2 frames, and decides that if on the first frame, the hand was on the first dot, and on the second frame the hand was on the 3rd dot, this means that on the frame between these two frames, the hand should be on the second dot, and so dlss3 will generate a fake frame, on which the hand is on the second dot. This will make the motion of the hand more smooth, and we can actually see the hand go on the second dot. No problem so far. BUT, what if this clock was made in a way that every other dot makes the hand change color. Meaning that when the hand in on the first dot, it's black, and when it's on the second dot, it turns red, and when it's on the 3rd dot, it's black again, and when it's on the 4th dot, it's red again and so on. Now, dlss3 has no way of knowing this about the clock hand, doesn't this mean that dlss3 is actually going to miss some important frames and actually create FALSE frames that can ruin the game? This example is about colors that change. but i have another example. think of a small particle that is moving from the left side of the screen to the right side. frame 1 shows this particle on the left side, and frame 2 shows it on the right side. Dlss 3 is gonna assume that the frame in-between should have the particle in the middle of the screen. BUT, what if this particle was actually moving in zigzag? Meaning that it's first on the left-middle side of the screen, and then it moves to the middle-up of the screen, and then it goes down to the right middle of the screen. Again, dlss3 doesn't know about the zigzag movement of this particle, and again it will create a FALSE frame that misrepresents what the game developer actually created. So, what do you think? Am I crazy for thinking this?
As with any signal processing, Nygvist theorem applies. Sample rate must be at least twice of the frequency of whatever we try to reproduce. Simply put, it is possible to fix those problems you describe, but easiest way is to render more frames.
I think the value here, given it's a feature of a flagship £1600 card, is to do ridiculous 4K 144fps gameplay rather than aiming it at 60fps where the input lag issue would presumably be much more noticable (along with any visual artifacts).
I honestly doubt the interpolation will have no artifacts. Don't all the interpolation programs that use a pre-recorded video also have access to the next frame? And don't they also suck for unpredictable motion? How would this one be different, specially when running in real time? I guess it could be better if it was somehow to first technique to utilize the next frame on the line but I very much doubt so.
AI video interpolators only have the frames and no motion data. A lot of the video interpolation process is guessing which parts of the frame are moving where, DLSS has access to motion vectors so it knows exactly where everything on the screen is moving because the game told it. That should lead to a lot less artifacting
You mentioned VR so I have to pop in and say that predicting future frames based on past frames (which is better than DLSS 3) is exactly what Asynchronous Spacewarp (Oculus' tech) does. Based on head movement, previous frame data, etc, it generates brand new frames to double FPS without introducing any latency. Of course, the downside is visual artifacts, as mentioned. But it does mean that DLSS 3 is way less revolutionary than, say, DLSS 2 was.
I really dont wanna hate, but the bit at 3m20s about how it would help weak machines doesnt apply now since DLSS3.0 is tied to only the RTX 4xxx series cards, maybe in 30 years. But the other part about helping insane high res and framerates on powerful machines is very cool. DLSS 3.0 sounds VERY cool, but I just hope AMD makes a version of these algorythms unthetered from new GPUs.
The latency in VR has been mentioned plenty of times, and so I don't think this helps. Besides, VR already has reprojection to deal with stutters in the source material, because the image *has* to move in response to your head moving, unless you want many people to get sick. On an unrelated note, you'll have less artifacting with NVIDIAs approach than traditional frame interpolation because it doesn't have to estimate the motion vectors, it gets them from the engine.
basically, use dlss 3/frame generation if your performance doesn't cap out your monitor refresh rate for the smoothest experience, but don't use it if you monitor can't make use of the new frames, as there is no benefit from it at this point and will only increase input lag.
Unfortunately it won't work with stutters, as the next frame is not available and the last frame already went out. The only solution I would see is delaying the last frame and slowing down time, as we see in video apps, when it drops frames.
I guess it should be aiming not to generating new frames to reach from 30 to like 60. BUT instead should be working towards converting existing ~60 fps to 144/165 fps since im with high refresh monitor tend to lower graphics to get at least 144. Maybe DLSS would make it not mandatory to lower graphics but still getting high fps with little to no input delay. My only hope is... that DLSS 3 wont make the image so smudged and smoothed as DLSS does now
@3:02 bad example, cos the its not focus and blurry, the picture in the game not matter if shown blurry the AI get it focus and clean great video tho thank you
@@2kliksphilip the blurry moving hand wont happened in games because the game always in focus and clean, ofc if the motion blur is off in the game. The camera had blurry shots and unfocused so the AI thought it part of video and added more frames between with that blur again so its not fit representing of DLSS 3 in games
Lets see if the dlss3 is optimized enough. I guess it will be like Ray Tracing on the 20x Series. A feature that will work well for the future generations. Time will tell.
Okay I have a question about this whole thing that maybe one of you can answer for me. isnt the point of having higher fps in game that you have less input lag then? if DLSS 3 gives you more fps but also higher input lag, doesnt that defeat the whole point?
I'm not sure about the added latency. As I understand it GPUs already queue up a bunch of frames in advance, and 3D engines have always sacrified latency in favour of raw framerate (with things like Async Compute and multicore rendering) since it's the only metric g4m3rz care about. Also, an interesting thing is that it will not be more taxing for the CPU, since it won't even be aware of those fake frames.
> As I understand it GPUs already queue up a bunch of frames in advance Only when the GPU is processing frames faster than the CPU is able to submit them, which is (fortunately) becoming less and less common.
It uses new application-specific hardware (optical flow accelerators) to interpolate without too much computation or artifacting. But with input lag etc I don't see why you'd even want it
They are output same as 'real' frames. So gsync will sync your panel to your framerate. Elsewise your screen wouldn't actually be displaying all the frames if it capped it's refresh lower.
Oh damn, I didn't pay enough attention to presentation, and thought that based on last few frames it predicts the new frame, with high enough input framerate (90 and above) I believe that could work. But I rewatched this part, and you are right. This implementation seems terrible for anything other than MS Flight Sim or similar games. Also I think actually it would make more sense if implemented in driver layer, to help old, single-threaded games, that are highly bottlenecked by cpus (we saw this in Crysis) My guess is they are working on it already.
It could be a combination of both and 2kliksphilip doesn't talk about it at all here. You could have DLSS 3-generated predictive frames based on the last couple sequential frames, that are only generated and sent to your monitor while the next DLSS 2 frame is being prepared, this way maximizing the benefits of both interpolation and extrapolation by being smart about it. Latency still goes up but not as much, artifacting can still happen but not nearly as badly, and you still get to see tons of "real" frames, as real as DLSS 2 frames are. This is likely what's actually going on with DLSS 3 but I can't say for sure until we get more information.
Yes, most VR platforms has some version of this. Only they also calculate it based on predicted motion of the headset. It does absolutely work, and is overall a benefit... but like pointed out in the video, when it fails it's extremely distracting.
I don’t know if the maths work out right to approximatley double the frame rate if the interpolated frame is generated after two frames are rendered. At best I’d think you’d get a 50% boost in frame rate. Example with a base of 60 fps: 1. 16.67 ms to render frame 1 2. 16.67 ms to render frame 3 3. 0 ms (for the sake of arguement) to render frame 2 (the interpolated frame) 4. Frame 1, 2, and 3 are displayed on screen for 11.1 ms each given the total of 33.333 ms take to render the three total frames. 11.1111 ms if frametime equals 90 fps so a boost from 60 fps to ‘90 fps’. Nvidia is claiming a bigger boost but I dunno how they are accomplishing that
I'm honestly skipping this generation mostly because I do AI workloads, and them selling a 2000 euro 24gb model running 450w at their top end that can't run Cyberpunk Native 4k RT 23fps is insane. Absolutely insane in the worst way.
@3:56...Hey looks the DLSS 3 propeller disappeared...don't worry about that...just pay $900 for a 70 series GPU, based on it's FPS it's definitely the next 80 series...put in you preorder before we actually send review samples out though...they won't last long...
I can't wait for DLSS 6.0 where you just play any game just by giving it a prompt and the game is generated on the fly by the neural network.
It could be like that yes... games/movies/VR/... will be generated on the fly, in realtime
This but unironically
6.0 is generous, at least it will be 60.0
I would so just try to utterly break the Network. I'd be like: "Computer, generate Elmo's Number adventure, but make it an open world third person RPG that crosses over into the Doom universe. Replace Big Bird with Count Chocula, and deep fry all the textures."
So playing a game with normally rendered frames?
Hey nice video! After watching countless rtx 4000 DLSS 3 videos being pretty much the same information, this video in comparison differd and gave me an actual idea how DLSS 3.0 may work. Thanks a lot phillip
IMO nVidia is not so dumb that DLSS 3 would look anywhere close to videos in this one, but the responsiveness issue simply cannot be avoided unless the AI is smart enough to build a mini time travel machine inside each card.
@@HyperOpticalSaint so it's useless. Usable in single player strategy games or sim games.
DLSS 3 injects FAKE FRAMES to give a FAKE higher FPS.
--> DON'T BELIEVE NGREEDIA'S MARKETING BULLSH#T...!!!!
You eyes will be opened to the depth of Nvidia's anti-consumer practices once the reviews are out...practices we already know such as rebranding a 4070 as a much MUCH more expensive 3080 12GB.....!!!!
@@marceelino So it's not useless, considering the other uses you mentioned. Also for gamers who want to experience scenic visuals in exchange for a bit of latency. Not everyone plays games that demand very low input lag, so I don't know why people are saying that DLSS 3 is a failure.
@@sayorimiko well the uses are minimal as you are getting the same value from FSR 1.0.
Input lag was the first thing that popped into my mind when they said they were going to interplolate frames. I hope there's an option where you can easily disable that and only upscale the image, especially for in competitive games that are already well optimized.
It seems that frame generation is a separate toggle for Dlss 3
The latency hit would make this unusable for VR, where that lag becomes something that causes nausea instead of just a mild annoyance depending on the game
Unless it was somehow in the single digits of milliseconds, completely agree. For VR, we just need more actual raw power, faking it can only get us so far.
Many games utilize asw which is frame rate interpolation.
Some quest 2 games already run at 60 fps and get interpolated up to 120hz
On PC the same thing happens, when your PC can't keep up with 90fps, it switches to 45 fps and frame rate interpolation is used
Oculus already has frame insertion when frames drop below a threshold and it's pretty decent.
VR has had this tech on the oculus side of things since the original rift
It's called ASW (Asynchronous Space Warp) and what it does is if your FPS can't match your headsets refresh rate then it will lock your framerate at half the refresh rate, so 45 for 90hz, 60 for 120hz, etc. and then it will interpret every other frame, I'm not 100% sure if ASW uses AI/Neural networks or not
No, it's not an issue, VR reprojects head movements so there's no latency, for that. For objects like hands there are artifacts.
God i cant express how greatfull i am for your tech videos. Thank you philip
You're welcome.
Don't let this distract you from the fact the shader gap between the "4080" 12gb and 16 is larger than the gap between the 3080 and 3090ti that all used the same 102 die.
Jensen trying to pad his wallet even more.
my wallet is augmented
@UCEKJKJ3FO-9SFv5x5BzyxhQ my arse
@UCEKJKJ3FO-9SFv5x5BzyxhQ I think they meant the 3080 and 3090ti used the same die, not that the 4080 and 4080 use the same die. According to the tech power up GPU database the 3080 and 3090tu both use the GA102 die
Edit: he changed his comment while I was writing mine. Now mine doesn't make as much sense
@@2kliksphilip Looking around it's believed to be AD102, AD103 and AD104 for the RTX 4078 $899; interestingly VideoCardz thinks the 4080 uses 22.5GHz GDDR6x which is faster than the other models.
@@2kliksphilip They all have a different die, 4090 : 102, 4080: 103, 4070(12gb): 104
First baked beans, now DLSS.
Thank you for blessing our day, Philip.
I feel this is similar to how supermarkets no longer dont stock 4 packs of beans and force you to buy them individually so overall you pay more. If DLSS 3.0 is just a marketing headline, these new GPU's are terrible value. Will be interesting to see how AMD responds as their cards were really competitive last gen.
This guy beans
Aldi 3.0
We still have 4 packs of beans in Australia but they are usually worse value than buying a single larger can.
Lidl still sells 4 packs ..
They allready did by Lowering 6900xt to 780€ comapred to the 1100€ for the 12gb 4080 with more or less similar Raster Performance by looking at the Nvidia Benchmarks of RE:8, AC Valhalla and Division 2
Why are you so weird but, yet, so cool. I can't stand liking this sense of humor of yours so much.
Definitely in the same boat with you Philip with knowing about the wonders of frame interpolation but never thinking it could be applied in real time to games like this.
But this is honestly the most comprehensive video I've found on DLSS so far, not even big journalists have broken it down as well as you have here. So hats off. Great vid. 🎉
@@2kliksphilip yeah we'll have to see. Hopefully you plan on covering the DLSS 3 situation more as it develops I'm really looking forward to that.
And for the AI detail/colour overlay weirdness that the GTA example shows off. Really looking forward to stuff like that.
But I am going to stand on the positive side of things here because I think if you're used to say 60fps and turn it on to get 120fps with the latency of 60fps you'll enjoy it.
If you're used to 120fps and want to play a harder to run game with DLSS 3 to get 120fps you're probably going to find it laggy at best and sickening at worst.
But time will tell. 🙂
@@2kliksphilip Reflex only help in situations where one is GPU limited (GPU Usage=100%), it doesn't decrease latency at all in CPU limited scenarios. Setting a framerate cap to limit GPU usage has the same effects on input latency as reflex does.
I have seen ideas about this a while back on some forums but I never thought nvidia would come up with it out of the blue.
There will be a delay added that cannot be circumvented in any way. My guess is it will be at least 2 frames.
@@eniff2925 just checked out the Digital Foundry DLSS 3 first showcase video. And they went over latency in scenarios where they are CPU limited and GPU limited (before turning on DLSS 3 frame generation)
And whilst Nvidia are very careful about what they allowed Digital Foundry to show in that video. FPS numbers being a no no for example. The latency is better than native (due to native being pretty bad as it's a low frame rate) and about on par with DLSS 2 latency without Nvidia Reflex.
Looked really good latency wise, only thing to keep in mind if that whenever they showed DLSS 3 frame generation, they also had DLSS upscaling in performance mode, so are always working with quite a high FPS (100fps turned to 200fps in Spiderman for example), using frame generation to go from 40fps to 80fps it might be much more of a problem.
And of course the latency isnt really changed at all when you have DLSS 3 frame generation and Nviida Reflex both on. So whilst latency doesn't get worse, you're not getting half the benefit of the higher FPS, which is less latency. So will 200fps in Spiderman with DLSS 3 actually feel like 200fps. No, it will just look like it. Weird tech. But super cool, hopefully it feels great to usw for those 10 people who can afford a £950 RTX 4070 ☠️
@@HoneyTwee I also watched it. They had DLSS 2 active which more than doubled the framerate in some cases. That was the reason for the decrease of latency. Turning DLSS 3 on top of that increases latency, quite heavily, but it is offset by DLSS 2. Compared to native, DLSS 3 should increase latency by at least 2 frames according to my logic. Reflex is not connected to this at all. It cannot help directly in this use case. Reflex was created in response to battlenonsense's findings about input latency drastically increasing in GPU bound scenarios. This input lag can be completely eleminated by setting an FPS cap so reflex is not very beneficial if you know what you are doing.
@@eniff2925 True, it would have been cool to see more about the latency penalties of DLSS 3, but it seems they were quite limited on what they could show.
But if i'm not mistaken, the latency they reported with DLSS 3 + DLSS 2 + reflex was still slightly better than DLSS 2 on its own, and a lot better than native. So I assume reflex is making quite a lot of difference even if the overall latency compared to native is probably mostly coming from DLSS 2.
Honestly though DLSS 3 looked good from what they showed, I don't think its a make or break to have it or not yet, so i'll probably still get an AMD card like a RX 7800XT if it's £850 ish and is around a 4080 16G, but DLSS 3 looks like really promising technology that I will be glad to have by the time its in more games and hopefully by then AMD will have a competitor to it themselves.
Laughed my as* off at motion extrapolation demo :D Anyway, nice informative video!
With filming videos on smartphones we already have all the stuff filtered and processed while recording.
So yes, I'm 100% with you on your last sentence.
One thing I really found unique about why this is the new DLSS 3.0 is because it's abusing the pipeline which contains motion vectors for the 3D scene already and is easy to update, maintain, and naturally slides into the DLSS framework without having to add more code into the project. It's all encapsulated into one single package rather than having two DLSS branches to integrate.
Other frame interpolations have to predict and implement many other algorithms to create a "fake" 3D perspective in order to be able to keep the frame well interpolated in some 3D games. It creates those weird visual effects philip points out, but with DLSS's framework, they get the motion vector data from checking the velocity between the vertex data on each frame inside the vertex buffer.
So that's why it's part of DLSS 3.0, because DLSS at it's core utilises motion vectors within it's algorithm, why bother creating an entirely new shader package for devs to integrate alongside DLSS 2.0 when they can just focus on installing the entire DLSS 3.0 umbrella. I think they're trying to extend DLSS more than just an upscale algorithm to more of a performance optimisation toolkit in the future.
I feel like the only draw back to DLSS 3.0's interpolation isn't going to be weird visual artifacts, but more motion blur because it knows the motion of a scene with each frame and it isn't quite tuned to completely sharpen the image. The main issues we'll have is with interpolating transparent/fading objects, inconsistent performance, and weird race conditions when it comes to frame times in some titles.
I can also imagine DLSS 3.0 being REALLY good for games which are frame-locked to 60 where when you try to break the frame limiter it messes with the physics/timesteps of the game, as it allows you to just get a smoother feel to the game rather than just the crappy locked framerate :) This probably won't impact the timestep calculations this way, and allows you to get a much smoother experience without breaking your entire gameplay experience. So emulators might benefit A LOT with this.
Thanks for reading my tedtalk, I was interested into being a graphics programmer while I was studying in university but I gave up that path, but if you've got questions about things like this feel free to ask c:
In other words, allow 3000 series to use DLSS 3 and you’ll see less performance difference between 30 and 40 series.
Not to mention nvidia is trying to sell x70 class gpu for x80 price.
Well it depends, because for Ray Tracing, there's like 4x as many RT cores as 30 series, so for RT performance, it'd still be well ahead, and they did show the difference between DLSS 3 and 2. It's also interesting that people keep saying it's a 70 class GPU at an 80 class price, because we will eventually have a 70 class GPU, and they've done stuff like this before, the 1060 3gb and 1060 6gb were really more a 1060 and 1060Ti because the 6gb had more of other things so it was faster in general. I also don't know why this is considered a bad thing because if they made this a 4070 at it's price people would have been more angry, this means the eventual 4070 will be cheaper
80 ti price really
@@MrZodiac011 1060 3gb snd 6gb were much closer.
in fact, no big performance difference as people claimed. These have much more cuda core difference.
And just compare it to last gen. "4080" 12gb doesn't even beat 3090ti.
3070 beat 2080ti, 2070 beat 1080ti, 1070 beat 980ti, 970 beat 780ti.
Salty 30 series owner?
@@Japixx no, we just dont have as much disposable income as you and are unable to spend at least $899 on a gpu with lackluster performance compared to previous generations. Furthurmore we wish to be able to have a relatively pricey gpu released only a few years ago be compatible with a software which seems to be very beneficial and amazing yet locked behind exclusivity deals
Couldn't stop laughing my ass off at 2:19, which made it impossible to hear what you were saying. Philip pls fix, Jack Torrence could get phased out of the next Shining if you're not careful xD
I'm not so sure about input latency being an acceptable trade-off for VR users. John Carmack talks at length about how the biggest hurdle getting VR to a usable quality was just reducing input latency, since delays are so noticeable and intolerable with a HMD. DLSS might not take VR back out of that acceptable range, but it may be the case that like competitive games, input latency for any VR games becomes a very sensitive benchmark. You can already see it when you look up benchmarks for VR setups, frametime is a oft included stat.
VR apps may even care about high frame rate less because it's smoother but more because higher frame rate lessens input lag, since you are waiting not as long for a new frame. But DLSS 3 does the opposite, it makes the input lag longer than it would have been with lower (non-interpolated) frame rate.
@@cube2fox I'm not sure, when it comes to VR, we tend to be a lot more sensitive to what we see, hence why higher frame rates are a must, DLSS 3 adding higher latency might be noticeable a lot more with VR then standard gaming.
If they want to target VR, they'll probably have to add predictive resolve mode to the algorithm, as opposed to interpolative resolve for quality-sensitive applications. But it's possible to do it then without introducing latency that you don't already have with framewarp, or just a little, in like 2ms range, just the tiniest little fraction of a frame worth.
But also absolute latency maybe isn't really THAT much of an issue. You can then just hide it by reprojection, so your view is updated at minimum possible latency on head turn, no added latency, while the game events and movement receive a frame of additional latency. It might just work and not be noticeable. Worth trying!
I LOVE THE SELF VIDEO FOR PROOF. Proves the point perfectly, yet makes me just laugh and enjoy your content even more.
I hope they add dlss 3.0 to csgo so u can go redo the input lag and hitbox videos again =)
As somebody put it so fittingly under a different video on the topic: DLSS is so amazing, it even upscales the prices!
0:22 "Why hasn't this been done for games already?"
It kinda has for VR. Asynchronous Space Warp with Meta headsets, Motion Smoothing on SteamVR headsets.
It's actually a pretty good experience on the Valve Index to do fixed framerate motion smoothing at 72hz to 144hz. I didn't like any lower than that.
And when I play on Oculus I disable it entirely cause I'm already adding a lot of latency with Virtual Desktop streaming.
That slow mo footage. Majestic.
babe wake up, new kliksphilip upscaling video
I think it would be best to have a way to integrate the tech into the rendering pipeline so that the frame generation tech uses an unfinished frame + the last frame instead of 2 of the last frames.
The paint drawing -> realistic image tech you showed at the end is probably a step in that direction.
I'm not sure but this sounds a lot like oculus's asynchronous space warp (ASW) which halves your frames and interpolates the other half. It works well in slow games or race games. But less so in fast games as you run into the same artifact issues explained in your video.
Honestly, I just hope the artificial frames thing is optional and DLSS at it's core keeps improving!
i think you can switch between 2.0 and 3.0
I doubt it unfortunately.. this new fake tech is the thing going forward vs improving 2.0
EDIT: I was corrected and the claims made in this comment aren't true, the presentation contradicts them.
There are a few reasons that I believe that DLSS 3 is, in fact, predicting frames as you proposed at the beginning of this section, and not placing them in-between already generated frames as you seem to believe is the case.
Firstly, the article about DLSS 3 on the Nvidia website has two images which show the DLSS 2 and 3 pipelines, including the framerate and input lag after dlss has done its magic. These pictures both show an improvement in input lag when DLSS 3 is used compared to DLSS 2. That may be Reflex working its magic even if DLSS 3 is still adding lag by not predicting, but I find that unlikely. Of course, that's just my not massively informed opinion, so that alone is not particularly swaying
My second reason is that there is precedent for versions of framerate interpolation that predict the next frame to reduce input lag - those seen in VR. Oculus Spacewarp and Valve's solution both attempt to do this exact thing with mixed quality. It has been a couple years, I think, since the release of these two methods; so it's not entirely surprising to me that a new set of graphics cards, with additional hardware dedicated to extrapolating useful information from previous frames (the Optical Flow Accelerator mentioned in the article), could do this with better results than the slightly older VR solutions).
Of course, it's very early days, and the existence of methods that can predict doesn't necessarily mean that DLSS will. Perhaps we don't have enough information to make a strong statement either way, but those are the reasons that I believe it is predicting frames, and that I think it is incorrect to suggest that it's placing frames inbetween previously generated frames.
If I lack critical information found elsewhere, or misinterpreted some part of the article, or maybe even misunderstood the video, please someone let me know.
@@2kliksphilip it's been confirmed in an interview that they are placing it between the two frames.
I did an experiment once. Because UA-cam didn't support 60fps at the time, but i wanted to upload smooth looking synthetic footage.
So i captured at 60fps, and wrote an avisynth script which upinterpolated it to 150 or 180fps i forget, then filtered it down to 30. So added synthetic motion blur basically. And it looked decent!
Here's what i'm thinking. Let's say you don't have VRR, say you have a TV running at 60Hz. It looks good, but your GPU can only hit real 60 only most of the time; but the occasional drops down to 56, you'd like to hide them, and maybe you don't have enough CPU or compute throughput to make it happen without taking too much of a compromise elsewhere. In that case i think frame resynthesis from motion, similar to how VR does it (generally running at something like 90Hz there) might be the ticket.
It absolutely makes sense to integrate it into DLSS since the resynthesis can leverage the same motion vector, lower-resolution and accumulation data just to do this framerate independently instead of frame synchronously.
A hunch that the limitation of DLSS3 to upcoming series of GPUs is artificial though.
As someone who despises frame interpolation on televisions, I really, REALLY hope that developers implement the image upscaling and FPS aspects of DLSS separately, because I think this FPS interpolation thing undermines THE WHOLE POINT of having higher framerates: more responsiveness, less input lag.
I despise frame interpolation on tvs as well. Yet this version genuinely excites me because of how fundamentally different it is.
What a time to be alive
I can't wait for the subsciption based DRM locked graphics cards of the future.
Future is now old man!
3 words
Nvidia on Linux
The way they make drivers for linux is absurd.
NVIDIA Grid cof cof...
@@egesanl1 I have used their Linux drivers on a laptop and while they are certainly more clunky and not open source (compared to AMD), I didn't get the vibe that they were all that different from Windows ones.
@@tablettablete186 Those are the server grade cards right? I don't know much about them but honestly I wouldn't be surprised if they did have subscription based drivers, not all that uncommon in the enterprise world.
@@bulutcagdas1071 Server and professional grade. This is needed if you want to use the virtualization capabilities of the cards as an example
Looks like your predictions were all correct. Well done!
Based on the available information, I believe DLSS 3.0 could be useful on strong PCs with very high refresh rate displays - 240Hz or 360Hz. If you have a PC that can already run games at say 120 or 144FPS without drops with a decent resolution, which is already smooth in regards to latency, and then run it with the new DLSS in an attempt to reach the higher frame rate, achieving a smooth gameplay and supremely smooth visuals.
However, I think anything below 120 FPS is too low for this kind of use, and I don't think it's going to work well in regards to the latency - as was mentioned in the video, it will be worse the worse the base FPS is. This is also one of the reasons why I think the mid range GPUs were not shown yet - they can't run the games natively at a good enough resolution/framerate for the new DLSS to provide good enough results. I wouldn't be surprised if, in the end, they won't have this feature available, perhaps with the exception of 4070.
I tried the DLSS mode they just added to flight sim to see if i could improve my performance, it didn't do anything as i think i'm CPU limited but it also introduced really ugly ghosting on everything so i wouldn't use it anyway, it's the first time i've used DLSS and i'm disappointed, it usually looks good in Digital Foundry videos so i'm guessing there's something going wrong with its implementation here or my setup.
Also if you were to use that interpolation feature for VR wouldn't the input latency increase nausea?
I think how useful this feature will be in VR is overstated. The fact is that image latency has a massive impact on the VR experience and is actually one of the greatest challenges in bringing cheap VR to mass market. Latency in VR causes VR sickness.
This is neat and all, but according to the couples last years trend it will be used as an excuse for bad game optimization and other stupid things rather than being a nice bonus on top of a well made game.
I think the best use case is to set a frame rate (60/120/…) and when the framerate dips below that threshold dlss kicks in and adds additional frames to the existing ones to keep it as smooth as possible
I feel like the biggest point of playing at a high fps is the lower input lag, this maybe makes sense for playing at very low frame rate with a controller.
What I personally miss is having VR technology make it back into regular gaming. Asynchronous reprojection and all that stuff for example, is brilliant.
we are on the cusp of hardware being good enough and cheap enough.
2:40 - ***scrolls down to the comments, to hide in fear***
I feel like we're going down the route of TVs advertising 5000Hz through interpolation.
Got excited about future videos of yours when DLSS 3 was announced.
If you have 30fps and dlls 3.0 brings it up to 90fps ... you'll still have the input lag of playing at 30fps.
So yes, you said it right ... in order for the "motion flow" to work in a pleasant way, you'll need a reasonable input fps to begin with.
So, totally not worth it ever. Got it.
Well, no.
DLSS reduces resolution and improves performance.
So 30 fps becomes 45.
Then DLSS interpolates, so 45 fps will feel like 45 fps and look like 90.
If it is actually waiting a full frame as talked about in the video you'll have more input lag.
It's like how DLSS 2 works in the spatial dimension, but now in the temporal dimension. Better results the larger the data set you feed it to begin with. So it does a great job when fed 1440p and asked to output 4k, less so when fed 720p and asked to output 1080p etc. Likewise DLSS3 should do a great job when fed 60 and outputting 120, not so when fed 30 and outputting 60.
Which means, if you can achieve a native 1440p 60fps without DLSS, expect DLSS 3 to give you 4k 120fps that's of comparable quality to native 4k 120fps except at a quarter the performance demands. That's huge.
In fact, you will have MORE input lag, since interpolation needs to know the NEXT frame to insert a generated one, not the current one!
And of course, all of this is artificially locked not only to NVIDIA GPUs, but to the 4xxx series specifically...
Also, DLSS3/motion interpolation is temporal upscaling.
I would have preferred if they simply focused on upscaling and threw all that performance at upcaling instead. I dont like the added latency which is what im trying to mimize with DLSS in the first place.
What they could have done with DLSS if if they told game engine to render only the color (no shadows or GI). This would give them a geometrically accurate representation of game world. They could than feed this to AI to generate a realisticly illuminated version with very noisy ray tracing added in for consistency.
Rendering only the color is much faster than full render pass, you can test this with UE5 pretty easily. It also wont suffer from input lag of having frame interpolation.
Another trick is foveated ray tracing. There are only tech demos of this available atm but i think it is amazing idea
Why would you want very noisy ray tracing, that's terrible.
I'm not sure they have done it as suggested. Flight Simulator was CPU bound and input lag isn't a big deal. But Digital Foundry have clips from Cyber Punk, past DLSS caused input lag, so I'd be surprised if Nvidia made the same mistake twice.
Anyway I know 3rd party reviewers will investigate the latency angle to check it out.
@@HeloisGevit Because they have really powerful AI compute power to cover it up. I mean right now they are rendering completely artificial frames, very noisy ray tracing can help massively with this
The real reason nvidia is focusing on dlss 3 and gatekeeping it is because the power difference between the two gens are actually not as big and they need to hook customers somehow. If Amd comes up with a similar solution they would walk back and enable it on old gen just like freesync and gsync.
@@huleyn135 it's 50% in games Nvidia picked, so it's probably max jump
Hmm you mention generating new frames when there's a stutter in vr, but that sounds almost the same as the motion smoothing oculus/meta is doing. Asynchronous spacewarp does help with small stutters but can be very noticeable
SteamVR also does this. Async warp just uses the last frame and warps it to match where the headset is a frame later. There are lots of artifacts because AFAIK no motion vector data is being used, just the headset motion. DLSS has access to that plus the next frame also so it has a lot more data to work with.
Could you use DLSS to interpolate frames for games with locked frame rates? Such as old emulated games? that would be a real game changer
No, because those games don't support the suite of data that DLSS 2 needs to function (and by extension, DLSS 3). For instance: motion vectors. These need to be fed to the GPU and so they require the game developer to implement DLSS.
@@TheDravic but could newer emulators like dolphin add these things or would Nvidia not allow that
@@callumsmile I don't know how the emulator would be able to add motion vectors among other stuff, it's a pretty complex concept for such old engines unfortunately.
The thing you describe (generating a fake frame when the game can't keep up) is already a built-in feature of the Oculus runtime called Asynchronous Space Warp (ASW). It's extrapolative instead of interpolative though. Interpolative is a total no-go in VR, extra latency makes you very sick very quickly.
Frame interpolation will just add a ton of input lag it doesn't belong in interactive media like games
The point about it only being used to produce frames when needed is interesting. I have a 4k 120hz display that I use for most games and would only want extra frames generated in order to hit that 120fps mark when needed. If I can already get 120fps without any generated frames, then that is what I want to see. But I'm not sure it will be work that way. So by turning on DLSS 3, I'm essentially limiting my real framerate to far below what I usually would and I'm losing out on input lag whenever my machine could in theory be maintaining 120fps without DLSS 3.
Exactly!!!
They should give an option to disable frame interpolation so that you'll still get the benefit of image upscaling.
You get it from switching from DLSS 3 to 2 pretty much
This will be insanely good for vr idc for games since I'm not dealing with delay in my comp games but vr idc much
Is DLSS 3 a software update or you actually need the newest hardware to have it? im confused.
DLSS 3 is just a way for them to advertise performance gains in a way that we've never done so before. More frames used to always mean lower input delay due to each frame coming to the screen sooner. Now they can basically fake performance gains while still having the overall input lag of what the game looks like before they add the frames.
2:32 this is much less of a concern with DLSS frame generation, because it gets extra information from the game, like motion vectors. So it knows how things are moving between frames as well.
Also like you said, the framerates are already fairly high in comparison to videos that interpolation would usually be used on and because it only seems to be generating 1 frame in between, any interpolation error will be corrected in the next frame, making it much less noticeable in motion.
If you want to see what the artifacts look like though, watch Digital Foundrys DLSS3 preview and go frame by frame before Spiderman jumps out of the window.
That, and for reducing input lag, they might also be using player input in the interpolation process, like the proper frame interpolation the Quest 2 uses, where input is resampled right before the image is rendered.
Now waitung for a ps vr2 predictions. ✌
A lot of TVs have a motion interpolation feature, but I think Samsung deserves a special mention here. They are the only TV brand to offer a low-latency, gaming-optimized motion interpolation feature called "Game Motion Plus". This works only if you send the TV a non-VRR 60 Hz signal, and also only has ~30ms of input delay.
0:18 And -Door Stuck- UA-cam's best video
I don't understand. Why bother getting 120 fps if it doesn't feel like 120 fps should.
Because your eyes get to enjoy a 120fps image which is way smoother to look at. It won't feel any worse than what you could achieve without it, so what's the downside?
Not all the benefit of 120 fps is low latency input, a lot of it is a smooth looking image. And, in that, it will look like 120 fps should
Also, something not discussed, Nvidia reflex usually decreases input lag by more then a 60 fps frame, so using it in 60+ fps will probably still result in a lag reduction
In Games with Low Interactions Like flight Sim it will be great
DLSS is getting wild
Nvidia refers to the two separate functions of DLSS 3 as Super Resolution and Frame Generation, and in fact it seems to basically be DLSS 2 but with that new Frame Generation part, still relying on many of the same inputs (motion vectors and depth info, combined with an "optical flow field"). Since the Super Resolution part of DLSS 3 is still backwards compatible with RTX Turing and Ampere, presumably the two features can be toggled individually. Or at least I hope it does in case someone likes one or the other instead of both.
Either way, while the prices can suck a dick (especially for people on Europe who will get screwed over extra hard), I genuinely hope Nvidia pulls off real time frame interpolation successfully. We are continuously moving to a philosophy of "render smarter, not harder", and this would be a massive step forward. I won't be buying a 4000 series card, but I'm excited to see this technology. I wasn't expecting this sort of thing until a few GPU generations away.
I'm definitely curious how well this works in practice since currently available frame interpolation is not good for real time purposes.
Smart TVs already do real-time motion interpolation, and Nvidia has made fancy hardware for it, I'm sure it will be high quality. But for interactive content I don't know if interpolation will ever be good, I'm hoping this is a step towards extrapolation
4:28 The thing with VR though is that although 100 ish fps is ideal, lags are very noticeable when you move your head around and can induce motion sickness, if its present even a little bit, so dont think this is ideal for VR either.
My prediction is that because DLSS already introduces more latency to frame rendering than native resolution is that with the newer Tensor cores they reduced the upscaling latency to such a degree that inserting a artificial frame doesnt increase the latency to such a extend that is noticeable. The new Optical Flow Accelerators are MUCH faster than whats inside Turing.
So yeah of course it will be a fake 120FPS without the responsiveness of a 120FPS image. It looks smoother and feels smoother regardless.
03:00 I can't freakin' believe it's voice of that guy xD
Input lag is not the measure of framerate, the user experience is. And that is subjective. Some prefer smoothness, some prefer input speed, and we tailor ourselves around that preference. Many of us like both. This technology seems to support both. For example -- something like gsync like will have more input lag than without, but - for many, it will be worth it for the immersion of frame smoothness and consistency. Same deal here. And this is coming from a guy that hates tv interpolation, yet understands this is a different beast.
Just what?
VR headsets use it too avoid nausea. Oculus calls it space warp the industry calls it reprojection. Some implementations are better than others it depends it they use raw motion vector data. They are introduce some level of artifacting and latency.
Therapist: "15 to 60 fps interpotlation 2kliksphilip doesnt exist"
This Video: 2:57
Am I wrong for only wanting to see native benchmarks?
"DLSS, features that would be most beneficial on lowered hardware, reserved for only the latest and greatest"
Feels like giving fabulously wealthy celebrities free stuff.
It's part of DLSS 3 due to requiring the same data that DLSS already uses, e.g. motion vectors.
when will they start reading my mind to know my moves before I make them?
Let's say we have 2 frames, frame 1 and frame 2, and dlss3 is gonna create frame 1.5 and place it between these two frames. Now, the scene being rendered is of a clock hand (second hand) moving across the small 60 dots around a clock. Now, on frame 1, we see the hand is on the first dot (right under the number 12 on the clock). On frame 2 we see the hand is on the 3rd dot, meaning the game engine didn't generate the frame on which the hand goes on the second dot (due to the gpu not being powerful enough). Now, DLSS3 sees these 2 frames, and decides that if on the first frame, the hand was on the first dot, and on the second frame the hand was on the 3rd dot, this means that on the frame between these two frames, the hand should be on the second dot, and so dlss3 will generate a fake frame, on which the hand is on the second dot. This will make the motion of the hand more smooth, and we can actually see the hand go on the second dot. No problem so far. BUT, what if this clock was made in a way that every other dot makes the hand change color. Meaning that when the hand in on the first dot, it's black, and when it's on the second dot, it turns red, and when it's on the 3rd dot, it's black again, and when it's on the 4th dot, it's red again and so on. Now, dlss3 has no way of knowing this about the clock hand, doesn't this mean that dlss3 is actually going to miss some important frames and actually create FALSE frames that can ruin the game? This example is about colors that change. but i have another example. think of a small particle that is moving from the left side of the screen to the right side. frame 1 shows this particle on the left side, and frame 2 shows it on the right side. Dlss 3 is gonna assume that the frame in-between should have the particle in the middle of the screen. BUT, what if this particle was actually moving in zigzag? Meaning that it's first on the left-middle side of the screen, and then it moves to the middle-up of the screen, and then it goes down to the right middle of the screen. Again, dlss3 doesn't know about the zigzag movement of this particle, and again it will create a FALSE frame that misrepresents what the game developer actually created. So, what do you think? Am I crazy for thinking this?
As with any signal processing, Nygvist theorem applies.
Sample rate must be at least twice of the frequency of whatever we try to reproduce.
Simply put, it is possible to fix those problems you describe, but easiest way is to render more frames.
I think the value here, given it's a feature of a flagship £1600 card, is to do ridiculous 4K 144fps gameplay rather than aiming it at 60fps where the input lag issue would presumably be much more noticable (along with any visual artifacts).
I honestly doubt the interpolation will have no artifacts. Don't all the interpolation programs that use a pre-recorded video also have access to the next frame? And don't they also suck for unpredictable motion? How would this one be different, specially when running in real time? I guess it could be better if it was somehow to first technique to utilize the next frame on the line but I very much doubt so.
AI video interpolators only have the frames and no motion data. A lot of the video interpolation process is guessing which parts of the frame are moving where, DLSS has access to motion vectors so it knows exactly where everything on the screen is moving because the game told it. That should lead to a lot less artifacting
Check out Digital Foundry's video at 1:32 if you want to see some artifacts. That's one of the worst case scenarios for DLSS 3 though
You mentioned VR so I have to pop in and say that predicting future frames based on past frames (which is better than DLSS 3) is exactly what Asynchronous Spacewarp (Oculus' tech) does. Based on head movement, previous frame data, etc, it generates brand new frames to double FPS without introducing any latency. Of course, the downside is visual artifacts, as mentioned. But it does mean that DLSS 3 is way less revolutionary than, say, DLSS 2 was.
I really dont wanna hate, but the bit at 3m20s about how it would help weak machines doesnt apply now since DLSS3.0 is tied to only the RTX 4xxx series cards, maybe in 30 years.
But the other part about helping insane high res and framerates on powerful machines is very cool.
DLSS 3.0 sounds VERY cool, but I just hope AMD makes a version of these algorythms unthetered from new GPUs.
The latency in VR has been mentioned plenty of times, and so I don't think this helps. Besides, VR already has reprojection to deal with stutters in the source material, because the image *has* to move in response to your head moving, unless you want many people to get sick.
On an unrelated note, you'll have less artifacting with NVIDIAs approach than traditional frame interpolation because it doesn't have to estimate the motion vectors, it gets them from the engine.
About VR, they won't love this at all. The added latency is way more noticeable on VR compared to monitors
So all rtx cards will benefit from Dlss 3 right?
basically, use dlss 3/frame generation if your performance doesn't cap out your monitor refresh rate for the smoothest experience, but don't use it if you monitor can't make use of the new frames, as there is no benefit from it at this point and will only increase input lag.
Unfortunately it won't work with stutters, as the next frame is not available and the last frame already went out.
The only solution I would see is delaying the last frame and slowing down time, as we see in video apps, when it drops frames.
I guess it should be aiming not to generating new frames to reach from 30 to like 60. BUT instead should be working towards converting existing ~60 fps to 144/165 fps since im with high refresh monitor tend to lower graphics to get at least 144. Maybe DLSS would make it not mandatory to lower graphics but still getting high fps with little to no input delay.
My only hope is... that DLSS 3 wont make the image so smudged and smoothed as DLSS does now
I understand a lot now.
@3:02 bad example, cos the its not focus and blurry, the picture in the game not matter if shown blurry
the AI get it focus and clean
great video tho thank you
@@2kliksphilip the blurry moving hand wont happened in games because the game always in focus and clean, ofc if the motion blur is off in the game.
The camera had blurry shots and unfocused so the AI thought it part of video and added more frames between with that blur again
so its not fit representing of DLSS 3 in games
@@2kliksphilip ua-cam.com/video/6pV93XhiC1Y/v-deo.html
Lets see if the dlss3 is optimized enough. I guess it will be like Ray Tracing on the 20x Series. A feature that will work well for the future generations.
Time will tell.
This is all very exciting, just wish they hadn't upscaled their prices too.
Okay I have a question about this whole thing that maybe one of you can answer for me.
isnt the point of having higher fps in game that you have less input lag then?
if DLSS 3 gives you more fps but also higher input lag, doesnt that defeat the whole point?
No
This is so confusing. I'm getting it though.
Kliksphilip videos always make them easier to understand for me at least
You're getting the $1600 4090?
The new Dlss the old VR trick it might work fine on a headset but it's going to feel awful on the monitor or a TV
2:30 The most cursed snapchat filter
I'm not sure about the added latency. As I understand it GPUs already queue up a bunch of frames in advance, and 3D engines have always sacrified latency in favour of raw framerate (with things like Async Compute and multicore rendering) since it's the only metric g4m3rz care about. Also, an interesting thing is that it will not be more taxing for the CPU, since it won't even be aware of those fake frames.
> As I understand it GPUs already queue up a bunch of frames in advance
Only when the GPU is processing frames faster than the CPU is able to submit them, which is (fortunately) becoming less and less common.
Thanks for the upload
My predications are that I won’t ever be able to use it because I think I’m done with nvidia after my 3080 and not getting support for dlss 3 with it.
How, as far as i know the 40 series it has dedicate part in the DIE for that
And asking 1200$ for the new 4080
It uses new application-specific hardware (optical flow accelerators) to interpolate without too much computation or artifacting. But with input lag etc I don't see why you'd even want it
How do the 'fake' frames work with gsync?
gsync doesn't see them till it renders so i doubt there is any change.
They are output same as 'real' frames. So gsync will sync your panel to your framerate. Elsewise your screen wouldn't actually be displaying all the frames if it capped it's refresh lower.
Oh damn, I didn't pay enough attention to presentation, and thought that based on last few frames it predicts the new frame, with high enough input framerate (90 and above) I believe that could work. But I rewatched this part, and you are right. This implementation seems terrible for anything other than MS Flight Sim or similar games. Also I think actually it would make more sense if implemented in driver layer, to help old, single-threaded games, that are highly bottlenecked by cpus (we saw this in Crysis) My guess is they are working on it already.
It could be a combination of both and 2kliksphilip doesn't talk about it at all here.
You could have DLSS 3-generated predictive frames based on the last couple sequential frames, that are only generated and sent to your monitor while the next DLSS 2 frame is being prepared, this way maximizing the benefits of both interpolation and extrapolation by being smart about it. Latency still goes up but not as much, artifacting can still happen but not nearly as badly, and you still get to see tons of "real" frames, as real as DLSS 2 frames are.
This is likely what's actually going on with DLSS 3 but I can't say for sure until we get more information.
Isn't VR headsets already have a Frame Generation feature?
Yes, most VR platforms has some version of this. Only they also calculate it based on predicted motion of the headset. It does absolutely work, and is overall a benefit... but like pointed out in the video, when it fails it's extremely distracting.
I don’t know if the maths work out right to approximatley double the frame rate if the interpolated frame is generated after two frames are rendered. At best I’d think you’d get a 50% boost in frame rate.
Example with a base of 60 fps:
1. 16.67 ms to render frame 1
2. 16.67 ms to render frame 3
3. 0 ms (for the sake of arguement) to render frame 2 (the interpolated frame)
4. Frame 1, 2, and 3 are displayed on screen for 11.1 ms each given the total of 33.333 ms take to render the three total frames.
11.1111 ms if frametime equals 90 fps so a boost from 60 fps to ‘90 fps’. Nvidia is claiming a bigger boost but I dunno how they are accomplishing that
I'm honestly skipping this generation mostly because I do AI workloads, and them selling a 2000 euro 24gb model running 450w at their top end that can't run Cyberpunk Native 4k RT 23fps is insane. Absolutely insane in the worst way.
@3:56...Hey looks the DLSS 3 propeller disappeared...don't worry about that...just pay $900 for a 70 series GPU, based on it's FPS it's definitely the next 80 series...put in you preorder before we actually send review samples out though...they won't last long...
4:31 wouldn't vr user notice lag even more?
I love it when AI touches things :)