'Upscaling' framerates in games - Using RIFE to prepare for DLSS 3
Вставка
- Опубліковано 15 лип 2024
- I don't have access to DLSS 3, but now's an excellent time to convert some videos using RIFE's AI and to see what sort of bugs come with framerate conversion- which DLSS 3 will have to somehow combat / mitigate.
0:00 - Framerate conversion
0:29 - Latency?!
2:19 - 5 to 60 FPS
4:38 - Benefits of higher starting real FPS
6:16 - 15 to 60 FPS
7:31 - The magical 24 FPS
8:00 - 30 to 60 FPS
9:43 - 60 to 120 FPS?
10:38 - Conclusion - Ігри
Rendering technology is so wild. It's hard to believe how just 30 years ago we had games at a resolution of 160x200 and now computers are just inventing frames whole cloth.
30 years is an eternity in modern technology terms.
Hell 30 years is a significant portion of a human lifetime.
Man insane how we went from the pinhole camera back in 1021 to what we have today in 2022 absolutely incredible
A Pentium from 30 years ago would have produced 10+ quadrillions of cycles by now.
The same amount of cycles today's 8 core cpu can do under 30 mins. That's how quick technology has improved and evolved.
30 years ago was 1992 and we had SNES with much higher resolution and only 4 years before N64 and SM64
As far as I'm aware only pre-8 bit consoles and handhelds ran at resolutions that low. Both the NES and SNES ran at 256x244, although for NTSC regions the image was actually rendered at 256x224. The N64 was capable of 480p image, although the vast majority of games used the 320x240 mode. The original GameBoy and Gameboy Colour both rendered at 160x144. Much older video formats also rendered at 144p.
I mean, yes, the fact that the Atari 2600 had pixels you could count on screen and the PS5 and current PCs are rendering at 3840x2160 (which was always a pet peeve of mine, 4K suggests a 4 times larger resolution but if 1080p is 1920x1080, then 4K is actually only double the resolution, the 4K coming from the horizontal axis and not the vertical axis but that's neither here nor there) but also time moves forward.
One thing to note about 24fps movies are that each frame is filled with motion blur which helps our head basically create detail from what is essentially a blurry mess.
A frame from a video game capped at 24fps would be essentially moments frozen in time with no motion blur, thus looking less smooth.
In fact, shutter angles are a consideration for DPs to vary the length of the motion blur depending on whether it's an action scene (making it look almost superhuman) or to mimic intoxication or being drugged by making the blur extra long.
usually shutter speed is double the fps you capture at so you get smooth more natural realistic motion blur
The rave scene in Blade being a good example of no blur, if I remember correctly
I dont know if you tried this already, but how does it look if you extract all the AI generated frames from the 30->60fps video and play them back without any real frames (and then ai generate again based on that). Might be a cool experiment to see how often you can generate until it looks bad or non-playable.
I really like this idea! It wouldn't be necessarily in context but it'd be interesting to watch at the very least
Cool idea
this has 3klik's written all over it, love it
Brilliant idea! Would make a perfect 3kliks video.
3KLIKS, 2KLIKS, KLIKS, ANYONE
Once again Philip is explaing this technology better than anybody else.
Somehow this CSGO UA-camr is my go to guy when telling people about AI technology.
Great video man. Really shows you're willing to go the extra mile with the effort you put into this stuff. 💙
I enjoy his videos, but Digital foundry is my go to if you want in depth analysis.
@@lolroflmaoization sure, I'd agree with some stuff. But with this very example of DLSS 3 digital foundry seems very restricted by Nvidia about what they can show. And didn't dedicate their video to exploring AI frame generation in its entirety, just showed some percentage increases when using DLSS 2 and DLSS 3 in like 2 Nvidia approved games and scenarios.
So at least for now, I think Philip has them beat in terms of a full comprehensive look at this technology. So hats off to him I say.
you must be thinking of 3kliksphillip. different guy. brother. father?
@@WhizXGames haha true, the CSGO mapper, the AI expert and the jelly baby eater. A true family.
@@eniff2925 ultimately DLSS 3 isn't out yet. So this video isn't trying to be a deep dive and review of DLSS 3 and if you should turn it on or not. It's meant as an overview of what frame interpolation is, how it works, the limitations of the technology and what artifacts to look for, what frame rates the technology works better at and the fact that you see much better results when interpolating smoother looking motion to begin with (such as 24fps+) and explained why with simple and easy to understand graphs about the time between each real frame and AI generated frame and that tells us about what kind of use cases DLSS 3 is going to be best at (60fps to begin with and up)
Like why so negative? This video is great for just expanding on a basic understanding of the technology and without early access from Nvidia is the best we can hope to do, it covers many things that weren't mentioned in digital founday's video.
And you're negative towards DLSS as well saying it's a huge sacrifice in either latency or resolution. Whilst ignoring the benefits which is much smoother motion to drive monitors like 240Hz and 360Hz and 480Hz at 4k. Or allow higher end demand rendering like ray tracing to become smooth looking.
The tech isn't out yet, we don't know how good it is. But even if you take digital founday's numbers on latency at their worst and say that latency is the same as native when you have reflex on and are rendering at 1/4th the resolution. Or a little higher when rendering at native res with frame generation and reflex on presumably. Is that really a massive deal? It's milliseconds more latency for a much much smoother image to drive these new 240Hz+ monitors. If you only focus on the downsides of a few ms higher latency it sounds terrible, but you're just ignoring the positives. Which doesn't really matter at the moment since neither of us can make a good conclusion about the tech yet. But it seems you are instinctively negative and bias against it for whatever reason, a sentiment I've seen a lot of due to the anger with Nvidia over priced and what not.
Just give the tech a chance, stop hating on a video for not being an in-depth look at DLSS 3's specific implementation of frame interpolation before the tech is even out, and stop being so negative. 👍
The biggest difference is that dlss has access to color, normal, depth and motion vectors straight from the engine.
I don't see why DLSS wouldn't have access to the vertex data too
Just wanted to say that I really appreciated the random Oblivion OST towards the end of the video, absolutely wonderful
Same taught. I had to check if anyone else noticed.
@@gsedej_MB same
Impressive stuff tho, I still remember when fading between frames to increase framerate was the state of the art
My old copy of Sony Vegas sure likes to remind me that resampling framerates was the future, when it makes even my hard cuts out of black blurry.
I've been pretty surprised that MPV seems to do that, while the Smooth Motion option in MPC-HC (video players) doesn't seem to do anything at all.
I can't believe that every single player in existence shouldn't be able to do better real time interpolation on PC and a 5600X and 3060Ti than my 10yo TV on some potato chip.
RIFE 4.0 via Flowframes is great and shockingly fast, compared to my previous experience 2 years ago with DAIN 0.36 and a GTX 960, but I'd still love to know about something that replicates that standard TV feature.
One thing to keep in mind is that rife can see all the frames while nvidia can only see two, the present frame and the next one.
Edit: IM WRONG rife only uses two frames despite having access to all of them. Thanks Quaterions
@@dan_loeb not only nvidia others upscale too. It's not special thing tô nvidia
Unfortunately RIFE only uses two frames regardless of the fact it has access to all of them...
@@baka2907 What does this have to do with anything? This is not upscaling.
@@baka2907 ???
RIFE does not use more than 2 frames.
I really appreciate the amount of effort that has gone into this video. Straightforward and easy to understand, even for someone who isn't very tech savvy. Thanks for the all the hard work, very enjoyable!
incredible work with the explanations and demos! The filmed video is literally perfect
I'm pretty sure the motion vectors in use will make a massive difference. The motion vectors are 3D, so it knows whether things are moving towards or away from you as well.
I just wish games would also use the "time warp" style fps boost that VR uses. With VR you have the drivers doing stuff with the depth buffers, distorting the previous frame, guided by the depth buffer, using your headset's input.
So it allows you to move your head in a scene, while technically the action in the scene isn't newly rendered yet, for a low head-movement input latency?
I never heard of this, cause I don't own a VR headset, but that's what I understand this as.
@@Rainquack yes! I encourage you to look it up on youtube. There are videos demonstrating it. The idea is to never let your headset's movement suffer from any framerate changes. When a frame isn't ready in time, it takes the previous frame and warps it according to the geometry of the scene (stored in the depth buffer), so you can always move around freely in the space in the full framerate, and if things are chugging along you will see items and characters and stuff get choppy, but your own movement in the space is always smooth. It's brilliant!
मेरे विचार से यह एक बहुत सुन्दर , उपयोगी और महत्वपूर्ण तकनीक है चित्रपट निर्माण के लिए । इस तकनीक द्वारा हम विशेष दृश्य प्रभाव जैसे मैट्रिक्स प्रभाव या टाइम-स्लाइस (जो की कई कैमरों का उपयोग करके बनाया जाता है ) ताकि काल धीमा हो जाये या स्थिर प्रतीत हो , बहुत सरलता और तीव्रता गति से से सम्पन किया जा सकता है।
वैसे आपने इस चित्रपट में गहन शोध , स्पष्ट अवलोकन , प्रज्ञावान विश्लेषण और बहुत सुन्दर प्रस्तुतीकरण किया है ।
Any time I notice music from Oblivion in the background of a video I have to go back and watch parts again because the music is so good it's distracting.
You released the video just in time. Was rewatching your older ones
2:33 this effect could be used purposefully to make some amazing visual stylisation
your videos are dope. been binging everything since i paused watching you from a few yrs ago, good stuff.
Thank you for your research! I've really enjoyed this type of content from you over the last few years!
0:00 You may not like it, but this is what the peak male form looks like.
Look at that powerful trunk. Phillip is built like a Roman column.
Seeing my own animation in your video caught me of guard. Very nice
excellent video as always, always a pleasure
love your work!
Wow, those morphing videos created by RIFE remind me of some rather spiritual experiences
?
@@HaxxorElite the RIFE merging looks really similar to an effect one can get while using psychedelics
@@logisticallogic8673 It has no connection
@@HaxxorElite ? I'm not quite sure what you mean, I'm only saying it looks similar to things I've personally seen before.
Great video. I've been waiting for real-time frame interpolation tech in games for years; it's very exciting to see it finally coming to life. Plus, the wise mystical music always gets me in a wise, mystical mood.
I enjoyed this video thank you for making it
Fricken great video!
I would love to see a dlss 3 example that shows the input video on the left, the "upscaled" result in the middle, and JUST the newly generated frames on the right. Would make for a really neat comparison.
I almost didn't click on this video, because the title sounded boring, but I am very glad that I changed my mind. This video is even better than a lot of your recent ones.
DLSS 3 will be HUGE for older games that have physics tied to fps. A great example is Ocarina of Time. Trying to play at 20fps gives me a headache sometimes, but I power through it since it's one of my favorite games ever. If DLSS can properly "upscale" it to 60fps on Project64, I'd buy a 40 series card alone for that.
Wait... is that Uncle Al? Oh god, he's broken into our physical dimension!
Aside from that, I did find this whole experiment/demonstration super fascinating. I've always been mesmerized with some plugins and software that can take 60fps and attempt to convert them into slow motion. The weird bending and smeary distortion can look really neat.
When using Rife with Flowframes, it does have an option to detect scene changes, so it will not interpolate between those 2 frames.
I think this is the most I have ever critically analyzed and learned from a clip of a half naked man
Other than maybe that unlisted 4kliksphillip video from a while back
I'm glad you sorted out the city packs for flight sim! Sorry, you were probably flooded with comments on the video on that...
Frame upscaling and the artifacts that come along with it look really similar to asynchronous space warp (frame interpolation for vr headsets)
This is not upscaling though, it's interpolation
@@theRPGmaster you could argue that its temporal upscaling 🤔
@@AngryApple You could argue it's temporal upscaling when plants grow
@@AngryApple no you cant because then it would be called temporal upscaling
that intro is gold.
I think it could be interesting to just show the 60 AI frames for the 120fps example, to judge what it is doing.
Very interesting.
Long time SVP user and RIFE user as of this year here. I would like to report that chain link fences are the bane of all things frame interpolation, and you should use it as a worst case scenario instead of grass!
doubling the replay speed to increase fps was the most smart thing i ever heard 😮
Don't undersell yourself - you are always quite mesmerizing to watch ;)
these vids are so cool 🕶
OOOOH the bugs you mention about framerate reminded me of an old bug. when you ran oldest cossacks game on 64 bit architecture it couldnt help but speed up the game so fast it was unplayable it was like running a speedhack on very fast setting. it was meant for 32bit architecture. there were also some differences of gameplay between different screen resolutions iirc.
Good video!
such an amazing video. holy shit..
Saving this to my "watch later" pile as I'm going to sleep. 😂
see you in the morning my dude
waited for this vid
Didn't expect to see David Bowie demonstrations today.
sneaky Oblivion music :)
it would be cool if there was like an option that would only activate the fps upscaling when your fps goes below a certain point.
for example you would normally play at 60 fps but when the game sometimes dropped to 40 it would generate new frames to hit the 60 fps again
Very interesting test and comparasion! But I wonder if DLSS3 creates its newly interpolated frames by interpolating between grainly DLSS-upscaled frames Or is this new frame interpolation an entirely separate tech which doesn't involve any upscaling at all? At the top of the video you mentioned that it doesn't allow for higher resolutions than before. So I assume its working like DLSS2 plus interpolated extra frames. I am looking forward to a possible future video where you are testing it out to see how it looks and behaves in detail! Big love from Europe!
I'm fairly sure the gpu will utilise more data to interpolate muuuuuuuuch better. A gpu has a z-buffer (depth image), movement vectors for each pixel and more data, basically for free. This additional data can make the output almost perfect, even for the grass situation for example.
The Oblivion score in the background was doing my head in trying to work out if I had another tab open with someone streaming gameplay lol.
Nice!
its going to be quite interesting for sure. With post processing you can't really test what would happen with a game. 5FPS decimated from a 60FPS video would be as you presented it, an even decimation of the frame rate and thusly, the frame times. but with a game, as i'm sure you know, you have a variable frame time. i'm the most interested in where you are getting, by all means, a measurable "60fps" or "120fps" but your frame TIMES are wonky and uneven. THAT is where i'm sure DLSS 3 will shine.
EDIT: and yeah, DLSS3 will also have the motion vectors and depth buffer among other things that can act as hints to it. this is going to be VERY VERY interesting to see.
Have you seen RTX remix phillip? Who am I kidding, of course you have, feel like it's a techology you'll love. I can't wait to go render so many old dx8 and 9 games with new ray tracing capabilities it'll be very neat to see them in a different light.
I really didn't understand DLSS as well as I thought till I discovered your channel, and since then I've learnt a lot. This technology is going to get pretty wild as time goes on.
Is there currently anything better than rife? I need 60fps to 120fps conversion for editing purposes.
M2M_VFI
@@D-K-C How is it better? Is it faster or is it more accurate? Or both?
@@Neekzu both
Woah, what is the name of the last song used in Conclusion chapter? It is so good, never heard it on another philip video before.
i also wanna know
Two things I would say-
The low frame rate interpolation shimmering could be used for some really good music videos
Hypothetically for the corner warping if the area it takes up is consistent enough would rendering the image at a slightly higher resolution and basically cropping to the desired resolution
the oblivion ost just made this video super
5:01 This came out really nice with the Vsauce-like music
Ray tracing and dlss are the technical computing miracle technologies of my time and I now understand what a lot of older gamers talk about about what I take for granted, it’s such an exciting time
Cool video! Thanks for making it. For the 30->60fps examples, could you show the real skipped frame and the interpolated frame side by side or even the difference between them? I feel like that might be really cool to look at. Cheers.
what looks REALLY good is already high fps footage upscaled to even higher (say 240 to 2x or 3x that speed) then RESAMPLED back down to 60fps with sony vegas smart resample. it looks insanely smooth good video btw
Would love to see this as well
8:25 - It’s impressive technology, but I feel like those camera “jumps” would drive me insane after a while. It’s similar to the later examples where you can’t exactly look for the issues, but you can FEEL their presence.
My Samsung smart TV upscales frames already it's called sharpness setting, really it's good as long as you don't go overboard with it (mine is set on 70/100). Especially during circular outer pans of camera, that's when you notice all the gaps in spacetime it can not possibly fill :D
2:40 that’s the effect I got when playing half life Alyx on my old PC.
yea the 5-60 csgo scene broke my brain, I have never in my life experienced motion sickness not even when I bought my vr set and instantly started with locomotive movement. But that scene broke me, jesus
this video reminds me of how much i hated using twixtor for editing haha, could never get the extreme slow motion looking right and now I find out why
yay dlss 3
Fallout 3, NV, and Skyrim, have mods/plugings that give high framerate without issues
(Some of these require some tweaking, but I have them all of them working)
I just hope that some form of it can be modded in or maybe even down the line officially supported on older cards, DLSS2 is great at what it does but fully raytraced cyberpunk still makes my gpu catch fire a bit even on Balanced
my curiosity is satisfied
That Oblivion music tho
In the 5 -> 60 flight sim, sure there was a lot of funny business going on over much of the screen, but there was also a lot of surprisingly smooth motion on the more distant objects. Definitely not how I'd want to game, but still super impressive that it can take 5 frames and turn them into 60 with any degree of success even if it's not exactly a good result overall.
I am curious how this program does its thing without itself reducing performance. If the computer is not fast enough to produce 60 real frames, how is it fast enough to invent fake frames to make up the difference? I get that the fake frames aren't as complex to resolve as real ones, but they aren't free either, the computer has to do some calculating to produce them. Is there a point at which the computer is spending so much time inventing fake frames that the real frame rate starts to suffer?
Why are your videos so good?
0:14 Now I really want to see 30 fps vs 60 fps comparison like that, just normal gameplay (not DLSS or anything like that), that is filmed from the same angle in both recordings so you can crop it and put 30 fps on one side and 60 fps on the other side. Searching for it on UA-cam, but I can only find comparisons where it's crop-split like that.
artifacts do not worry me. what worries me is the input latency
First 17 seconds of this video is the best news I've heard since the great war
👌👌👌👏👏👏👏
Well there is a game realm where something like this is already implemented - ASW in VR games, and it gives very convincing results.
OBLIVION MUSIC FTW!
Oblivion music = instant like
Something interesting is going to be if it's like minimum frame times, what if there's 10 frames between some of the "true" frames, after nothing but 1 frame between "true" frames. Something like a new form of stuttering...
Idea: The artifacting in the 60 to 120 FPS conversion becomes fast enough to trigger a kind of survival mechanism in the human brain, related to danger response and reflex. Like if you see something unexpectedly move in the corner of your vision, that may be a danger that you need to react to, so your brain distributes more resources to paying attention to it, and divides your attention between the flickering artifacts and whatever thing your trying to concentrate on.
I'm surprised that you, as an avid VR fan, made no comparison to the interpolation technology used in headsets! This kind of "Something is wrong, but you can't exactly tell what" happens ALL the time when you use interpolation to smooth out the framerates in VR. Things seem to "hang" on the edges of the screen sometimes and warp out of proportion. It is why I prefer to have a choppier framerate, as this kind of "wrong" feeling disappears, though I presume people who quickly get motion sick cannot enjoy the same luxury.
Specifically, I'm speaking of "Asynchronous Spacewarp", which adds frames when you move your head, to interpolate where you're about to look.
Wait I did not get the last part about raytracing. How would DLSS 3 help double the amount of rays fired for a raytracing scene while keeping the framerate same as before? Image interpolation and ray tracing are completely different scenarios of rendering. How is that related? Can someone enlighten me? 12:16 Or does he just mean essentially what Optix is doing? A form of AI denoising which helps get a better looking image with just a few rays fired.
I can really only see this being useful in the future when the RTX 4040 releases and can only play GTA 6 at 30 FPS. That and with RTX Remix on old games capped at under 165FPS (my monitor's refresh rate).
I wonder if this would be adopted in VR, instead of the Asynchronous Space Warp
The 5 real frames example gave me a headache. Really impressed with 24 though.
2:20 ah what I would give to get up to buttery smooth 5fps!
Hmmm Da Vinci Resolve has a very good slow motion feature, even in the free version it does an amazing job. Wonder what system it uses for the frame blending it has. Studio has an even better one.
I was interpolating some really fucked (GTA possibly 6) footage and it seemed to bug at tiles on rife 4.0. Also an AI denoiser also thought the tiles were noise and suppressed them. (They were the small tiles you'd find around pool areas and on walls)
That was a fucking great diagram
This may be great for VR where artifacts at the corner of your eyes may not be noticed
12:37 से 12:49 हाँ , यही है इस तकनीक का सटीक उपयोग है, पुरानी फिल्मों के फ्रेम दर को २४ से बढ़ाकर ५० या ६० तक कर देना , वैसे इस चलचित्र का क्या नाम है ?
wondering if this DLSS3.0 feature will be able to hide the shader stutters that we get in most open world games like...that would be "worthy gain" in my book.
in one of DF analysis, the frame interpolation process takes as little as 3ms so it s very flexible, they can also implement frame interpolation as a "substitute" frame., only delivering the interpolated one if the "genuine frame" does not show up on time, that would eliminate pretty much of the occasional stutters, the 1% low for sure, the 0.1% lows not sure
If a frame does not show up, dlss 3 doesn't have 2 frames to interpolate. So you'll get stutter anyways
@@iurigrang you are right, DLSS3.0 became meaning less to me now but thnx
You will bring us a video of you messing around on NVidia's Remix when it launches, wont you?
You made a great point at the end of the video. Past 60fps, traditionally I think the benefit of higher framerates has been the reduction in latency, not that it appears to move smoother. You can try looking over someone's shoulder as they play a video game and honestly anything above 50 looks reasonably smooth, but they can certainly feel that it's not perfect due to the latency. I'm really curious how DLSS 3 is going to feel on 30fps and 60fps games
It looks much nicer ngl.
Disagree, CRT's in motion resolution are equivalent to 1000fps, 60hz on a sample and hold LCD or OLED screen looks like a smeary mess, in fact resolution drops to 360 lines equivalent to 360p when panning the camera which looks ultra terrible on modern 4k displays. Latency is only 1/2 the picture and only really relevant for E-sport titles / twitchy games. Nvidia, Intel have both done research papers/tests that suggest past a certain point people don't care about latency. Also this testing methodology is useless as DLLS 3 has access to the games render pipeline and motion vectors so all those artefacts he experiences in his CS:GO example are a limitation of RIFE and can't be compared 1:1 with DLSS.
@@theemulationportal what bargain bin OLED do you have that smears 60fps content?
@@theemulationportal You're right about the motion vectors, I expect DLSS to outperform the methodology shown in the video in terms of image quality.
My argument was more that the increase in framerate is only really meaningful as long as you can feel improvements in latency. If you can't feel improvements in latency, you probably aren't benefitting anymore from high FPS from a smoothness standpoint either. I suppose we could actually test this empirically by having a 240hz screen and injecting input lag until it has the same latency as if you were playing at 60fps, might be an interesting test to do.
@@mechanicalmonk2020 For a 1000 pixels per second motion tracked by the eye, the object tracked is perceived to have a 16,6 pixel wide symmetrical smear on a sample and hold 60Hz display. Being OLED doesn't change that. A theoretical instant pixel transition technology would not change that either.
👍