How Super Resolution Works
Вставка
- Опубліковано 29 тра 2024
- Is it really possible to zoom and enhance images like in the CSI movies? Let's find out how image super-resolution works in the real world.
References
Image Super-Resolution Using Deep Convolutional Networks (SRCNN Paper)
arxiv.org/pdf/1501.00092.pdf
DIV2K dataset: DIVerse 2K resolution high quality images
data.vision.ee.ethz.ch/cvl/DI...
Mean Squared Error: Love It or Leave It?
ece.uwaterloo.ca/~z70wang/pub...
Al Bovik Gives Primetime Emmy Award Acceptance Speech
• Al Bovik Gives Primeti...
Image quality assessment: from error visibility to structural similarity (SSIM Paper)
ece.uwaterloo.ca/~z70wang/pub...
Loss Functions for Image Restoration with Neural Networks
arxiv.org/pdf/1511.08861.pdf
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network (SRGAN Paper)
arxiv.org/pdf/1609.04802.pdf
ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
arxiv.org/pdf/1809.00219.pdf
ESRGAN GitHub Repository
github.com/xinntao/ESRGAN
AI Neural Networks being used to generate HQ textures for older games
www.resetera.com/threads/ai-n...
Deep Network Interpolation for Continuous Imagery Effect Transition
openaccess.thecvf.com/content_...
Zoom to Learn, Learn to Zoom
vladlen.info/papers/zoom.pdf
See Better and Further with Super Res Zoom on the Pixel 3
ai.googleblog.com/2018/10/see...
Google Pixel Super Res Zoom
• Super Res Zoom
Handheld Multi-Frame Super-Resolution
arxiv.org/pdf/1905.03277.pdf
Pixel Recursive Super Resolution
arxiv.org/pdf/1702.00783.pdf
Deep Learning Crash Course References
Generative Adversarial Networks
• Generative Adversarial...
Optimization Tricks: momentum, batch-norm, and more
• Optimization Tricks: m... - Наука та технологія
This is one of the best videos I have seen so far related to super resolution. It explains a fantastic summary of many papers. Great work Leo!
Thanks, Utkarsh!
Wow, love how you are explaining what techniques and what papers use them, along with excellent details of super-resolution its self.
damn, one of the best SRGAN & ECRGAN explanation vids out there! It was straight to the point and the way of explaining was flawless. Thanks!
Your explanation is very concise and clear! It really helps me in my deep learning class. Thank you so much!
Thank you for this extensive review paper style video... It really helped me and I am sure it will also help every other undergrad like me who is looking to attempt Super Resolution. I have studied some papers online and was trying to figure out their place on the timeline and what are the current state-of-the-art architectures. I had also planned to investigate how all of this related to the famous Google Computational Photography but you have provided a fine overview...
Thanks for the links... I will be looking forward to more of your videos...
I think I hit a gold mine by finding this channel
One of the greatest, combined many courses with no realism to a daily operations using all of them theorems. Well done! :)
Your videos are always very interesting, thanks Leo
Awesome context, thank you for all of them. Great job!
Thanks for this great summary video. Best updated video on this topic for sure!
I saw this video a year back with limited deep learning knowledge, but still understood the concepts at a higher level. I understood that super resolution is possible with deep neural networks.
Now watching this video with more knowledge about different deep neural network architectures, gave me more understanding.
Just rushed into the topic and got scared by dense papers. Luckily I found this video and I feel much more familiar with the concept. Thank you for great summary and explanation!
nice very general view of the topic , we need more of this
This video is very effective for me. Thank You, Leo. You are the best👍
The amazing thing is that this is only going to keep getting better as more people use it.
Enjoyed a lot! Thanks for the well-made video.
Amazing summary, great video~!
The explanation about Data Processioning inequity was super, thank you (Y)
Love your videos Leo. Please keep it up.
Very nice overview ! Super-resolution is my ph.d researche field, and i am more interested in task-specific SR
Fantastic video ! Summarising some papers so good
Nice! Gonna read up more on your references for this
Great summary! Thanks!!
how interesting! thanks for your explanation :) I clicked the 'good' bb
Fantastic video, thank you for your great work. Love your channel! I've been wondering how AI upscaling is implemented so this came at a great time!
I find they have to down-scale the high-resolution image to produce low-resolution, and the low-resolution is not the natural image, when we train these low-resolution to produce high-resolution image, I doubt can we use these networks to train the natural lowresolution image?
@@renpulin8524 Renpu if we do a straight 2x downscale (half width, half height), or an even factor of that, then we take the average value of 4 pixels and put that color in 1 pixel, and the result is very close to what a camera would have captured in reality if the sensor was small. So downscaling is actually a natural image. :-)
And I don't think that there's any other way we can train anyway. How would we be able to capture 2 images of different resolutions with a single camera? Wouldn't work. So downscaling is the best thing we have.
And the results are good. Look at Topaz Gigapixel AI. A well-trained neural network which produces fantastic results with its "2x upscale" setting. It does pretty badly with 4x and 6x and higher, but that is because the training data was clearly focused on 2x (having halved the resolution in their training data as I described above).
If I need higher results in Topaz Gigapixel AI, I actually prefer to run a 2x resize and then take the result and put it in 2x resize again, because it will look more natural than a 4x or 6x setting. Which just proves that the 2x model is much better trained. And it is very good at 2x!
Excellent video!
Thanks for the input!
'Super' video of Super Resolution. 👌
Very clear, bro, thank u.
Thanks Leo. You are best
Nice!! Thanks for the sharing!!
You are the most underrated guy....
This is too much for me though 😅
Very informative
4:16 I think it does, if the hair is moving in the wrong direction, and the picture is sharp enough, it will look distinctly machine-generated, so perceptually it does matter, but not _necessarily_ for noise reduction
Give this man a medal.
Very good video ! I came to this from some Convolution Neural Network videos while I was googling AMDs new Super Resolution Feauture (competitor to nvidias DLSS).
Also did anyone ever tell you that you look like Quentin Tarantino‘s son? :D
Haha, yes I do get that sometimes :)
That madr my head spin a bit. However it's cool !👍
Does the procedure affect the respawn time?
outstanding !
thank you
If a game plays in native 4K is there any reason to even turn it on? Does it cause input lag if I do turn it on? I have it on low right now in game mode.
I think you last point is critical. The purpose of the completed image determines which SR methods should be used. If it is an artistic endeavor the simple fact that it is a good looking plausible image is all the is needed. If your trying to visualize something for study or measurement at a higher resolution then it must be an accurate representation of the object. This is not something guaranteed through AI. But multiple frame super resolution does give an accurate higher resolution image. It might not look as good but it is not just a plausible solution it is an accurate solution to the increased resolution of the image. So grandmothers picture is appropriate for AI based SR, medical imaging is not.
Exactly!
Sir, can I do a super resolution project as my academic project,
I am incredibly intrigued by Super Resolution. And while it is a great descriptive explanation of what, Super Resolution is. Other than the academic bookwork. How am I to get this for my historic, VHS, 4 x 3 productions.
However when cropping 260 line resolution, color under video to, 16 x 9. I'm reducing that 260 lines of resolution to something like, 175-200 lines of resolution. Along with the need to bump it to 1080 P at 16 x 9. And I have some interesting and pleasant results using Vegas video editor. But not to the extent of this Super Resolution, process.
So I'm preparing to archive some videos of a former Metropolitan Opera star from the late 1940s. With videos of her from the 1980s on VHS. And which already contains an excellent hi-Fidelity soundtrack in stereo. And would be much nicer to make the video look professional and acceptable.
So your explanation and description is great. How do I do it? How do I get it in video? One frame at a time? Like Photoshop? Or, because my mathematical aptitude sucks so badly. This might be out of the realm of my capabilities? And it will just have to look amateur VHS fuzzy. Until this process is refined and becomes a single click plug-in. In your choice of video editing software. But not quite there yet. Oh well. I'll just have to wait a long time to come before that happens. At least a month or two.
And what kind of resolution will we see. If Donald Trump gets reelected? As clear as shit. That's what. Clearing the way for 4 more years of insane shit.
I vote for Super Democracy Resolution. We need an enhanced democracy with increased resolution. And political transparency. If that were only possible?
I'm going to vote for increased democratic resolution
RemyRAD
Wow, I have no idea how to reply to this :) As for the VHS footages, EDVR (github.com/xinntao/BasicSR) may work well. I haven't tried it but it seems to do a good job on video super resolution. As for the increased democracy resolution, I'm not a US citizen yet, therefore cannot vote. So, whomever Americans want to see in the oval office, I respect their choice.
Interesting results
thanks...
Can you make a video summarizing some methods of applying computational photography techniques to smartphone cameras? More specifically, approaches that allow for more manual inputs compared to the automatic features found in the Google Pixel devices. I would love to see a video centered around the idea of capturing sensor data with a smartphone camera with the intention of using resource intensive techniques on a separate more powerful computer to process the image.
The basic computational photography techniques that smartphones use are similar to the ones that other digital cameras use. I made a video about that earlier: ua-cam.com/video/3E8DlKYKnO4/v-deo.html
Beyond those basic processing methods, Google Pixel and other advanced smartphone cameras use more sophisticated burst image processing methods to make up for the shortcomings of a small imaging sensor. I might cover those methods in a future video.
@@leoisikdogan I did see that video and as an explanation of the theoretical framework for image processing pipeline it was very good. However, I would like to see ways to apply computational photography techniques if one was to take a more manual approach as opposed to relying on the manufacturers implementation. This topic is relatively new to me so I may not be asking the correct questions. Basically, how much further can you push image quality of mobile photography beyond the manufacturer first party application?
@@morgan3913 You might find Marc Levoy's work interesting. He has been working on this topic for quite some time and now is leading Google Pixel's camera team.
Some of his papers:
Burst photography for high dynamic range and low-light imaging on mobile cameras
graphics.stanford.edu/papers/hdrp/hasinoff-hdrplus-sigasia16-preprint.pdf
Pixel Night Sight
ai.googleblog.com/2018/11/night-sight-seeing-in-dark-on-pixel.html
Handheld Mobile Photography in Very Low Light
arxiv.org/pdf/1910.11336.pdf
There are also these papers from Intel Labs (not my team though):
Learning to See in the Dark
openaccess.thecvf.com/content_cvpr_2018/papers/Chen_Learning_to_See_CVPR_2018_paper.pdf
Seeing Motion in the Dark
vladlen.info/papers/DRV.pdf
I guess even those papers alone have enough material for a new video :)
there should be a super like option in UA-cam for contents like this
Can we avoid using the term 'hallucinate' to describe detail generation? Hallucination is conjuring up images without any external stimuli causing it. In the case of super resolution, you definitely need the current pixel information to determine what gets drawn in the inbetween space, and in the case of GAN, the model draws upon what images it was trained on to create those detail. None of these come out of 'nowhere', so by dictionary definition hallucinate is a wrong word to use.
Some parties use the term 'dream' instead, and it's a bit more appropriate as it draws upon the knowledge/memory of the dreamer when making up its details.
That makes sense. I didn't come up with the usage of the word hallucination in the context of generative neural networks though. It's been widely used for methods that generate information that cannot be inferred from the input alone.
I believe that an illusion is the miss interpretation of sensory information, and hallucination is the addition of extra details which are not sensed by our sensory organs but are added by our brains... In that sense, this word seems quite appropriate for the action that it describes.
Does implementation of super resolution in a very rudimentary way implementation of AI?
In the gaming world, we don't use the esrgan model for real images, as assumed in the video, we have trained our own purpose-built models for different cases
As far as I know, the early examples used ESRGAN trained on natural images. I would be interested to see the results of models trained on video game graphics. Do you have any checkpoints for such models?
@@leoisikdogan yes, you can find our public models in the database at upscale.wiki/wiki/Model_Database . More information and showcases in the subreddit (www.reddit.com/r/GameUpscale/), the Discord channel (discord.gg/cpAUpDK) and some custom modifications and tests to the BasicSR codebase in my fork at github.com/victorca25/BasicSR .
Thanks for the links. It looks like the community has grown a lot. There are many model checkpoints fine tuned for different tasks.
@@leoisikdogan Indeed, we worked on many different cases and a parallel group grew besides the game upscale one, more focused on animation upscaling. We ended up creating models for many tasks, in my fork I included the code to automatically augment images with different types of degradations, but the other guys did the hard work of creating the datasets and training the models. I've been unable to participate for a while, but it's a great community, last time I was there, there was interest in DAIN as well.
Can't we just create synthetic burst of images to recreate google's one?
what if you put the upscaled image into the ESRGAN input and upscale again? INFINITE RESOLUTION!
Upscaling artifacts become more visible every time you upscale. It seems to produce decent results for up to 16x upscaling in both axes.
@@leoisikdogan nice
wow
what about video super resolution
nvidia is on that
This is fucking incredible
What's your view on the image processing experts in the Kyle Rittenhouse trial? I would love to see a review of the how accurate the expert advice was.
This video has only 5k views? wtf.
abi türk müsün
What happened to this channel???
The information in the training set does not count as information in the sense of data processing inequality. This is not actual super resolution; it's just educated guessing. All the details in *all* of these examples are completely made-up.
I like when somebody proudly speak about thinks he do not understand. Is it talking about gay resolution?
Someone speaking about things he does not understand? Ah, you are speaking about yourself, Markov.
This from your perpspective. Because you do not understand NOTHING from this matter, you listen and say "Waw this guy is genius" but he only is "genius" in the blind eye of peoples that understand nothing like you. For peoples that seriously work in this domain, this "presentations" is dummy funny.
@@RetroBulgaria Why are you so mad bro?
Because I'm tired of amateurs (claim to be professionals), dummies, morons, stupid peoples etc. that filled western world.
Your over-reaction to a nicely explained summary video is insane. And you called him gay for no reason and without proof. You seem to be both rude and stupid at the same time. ;-)