Amazing AR Effects Are Coming!
Вставка
- Опубліковано 30 чер 2020
- ❤️ Check out Weights & Biases and sign up for a free demo here: www.wandb.com/papers
Their mentioned post is available here:
app.wandb.ai/latentspace/publ...
📝 The paper "Consistent Video Depth Estimation" is available here:
roxanneluo.github.io/Consiste...
🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Aleksandr Mashrabov, Alex Haro, Alex Paden, Andrew Melnychuk, Angelos Evripiotis, Benji Rabhan, Bruno Mikuš, Bryan Learn, Christian Ahlin, Daniel Hasegan, Eric Haddad, Eric Martel, Gordon Child, Javier Bustamante, Lorin Atzberger, Lukas Biewald, Michael Albrecht, Nader S., Nikhil Velpanur, Owen Campbell-Moore, Owen Skarpness, Robin Graham, Steef, Sunil Kim, Taras Bobrovytsky, Thomas Krcmar, Torsten Reil, Tybie Fitzhugh.
More info if you would like to appear here: / twominutepapers
Károly Zsolnai-Fehér's links:
Instagram: / twominutepapers
Twitter: / twominutepapers
Web: cg.tuwien.ac.at/~zsolnai/ - Наука та технологія
Nothing like smell of a fresh 2 minute paper!
I love the smell of wood pulp in the morning!
Especially one that isn’t more fluid simulation lol
that runs for 5 minutes xD
Image the smell two papers down the line?
Nothing like the smell of a depth-mapped cat
Holy cow! Realtime depth information could be used for overlaying information for all sorts of displays: classroom learning, live events, concerts, task specific assistance. Might be a big leap forward for AR. Exciting tech.
Gonna be a crazy decade
Woah it's the guru himself!
That "dear fellow scholars" hits hard everytime
What a time to be alive!
Hold on to your papers!
*happy fellow scholar noises
it's their trademark
Lets add one more paper to this little one, becouse everybody needs a friend. Little neural network has to have a place to set there. There he goes...
the 2010s were the years of the smartphone
the 2020s will be the decade of AR and VR
@nasolem we can only hope and wait for a real SAO.
And the 2030s will be the decade of neural link and brain interfaces
@@cybercrazy1059 Hate to break it to you, but "mind upload" doesn't mean what you think it means. When you upload a file, it doesn't disappear from your computer. It simply creates a new file at the new location that is a copy of the original information. However, full-sensory simulation for virtual environments just need how the brain uses neural transmitters/inhibitors to be figured out for that to be achieved.
As we can see in the video above, we will always find ways to add cats to our new ARs and VRs.
@@nagualdesign Only an actual baby would be bothered by such an obvious troll.
*excited paper holding*
Hey, you are a Jonas tyroller viewer. RIGHT!!?
You can't expect us to keep a grip on these papers if you keep showing stuff like this!
My papers all emigrated to nigeria to start a new life
I had to order in a dump truck.
I get so excited whenever you upload!
every video you make blows me away! these advances are crazy and its so great to see these thing happening RIGHT NOW! Thanks for making these videos and bringing it to our attention
The next iPhone is expected to have a "LiDAR" sensor. Wonder if that will give that "perfect" AR effects.
We’re getting into some really exciting territory here!
It most certainly should, unless Apple messes up somehow. In my opinion, future is going to be about combining a very-cheap (sparse) but accurate lidar with depth estimator for filling in the gaps, it gives us precise values to work with. Lack of reference points is the biggest problem with monocular depth estimation.
doubt it. the ipad's lidar sensor, which i'm assuming is the same going into the iphone, is not nearly as fine on the details as faceid. the dots the lidar projector projects are spaced pretty far apart, which leaves the ipad to have to fill in the rest of the info.
@@cunty I'm confident higher resolution cameras and stronger parallel processors will enable NNs to do this from video only.
@SCUUBE Got reminded of the same. My tries with arcore didn't have flickering, tho it did sometimes take some time for it to discover the depth in certain areas. There's an arcore labs app in the store to try it out.
Amazing!!! This probably has tons of application for photogrammetry as well.
how do your videos so consistently blow my mind? You have a true talent for presenting dry academic information in a way that makes it exciting and understandable. Thank you for what you do.
Maybe google street view could use this to make more detailed 3D buildings?
This isn't something radically new, it's an improvement of existing methods. Making 3d models out of images is possible since 2009, find "building Rome in a day" paper and video. So I guess google street view isn't 3d not for technical reasons, but because their management don't want to bother making it in 3d.
@@luck3949 that's cool, I wonder what's possible with todays technology🤔.
@@rasp1628 I don't know, as I don't monitor this field. Look at LSD SLAM if you want to see something impressive, but it's 5 years old.
@@luck3949 Well, yes but improvements in the process could potentially make it more cost-effective and worth it for google, specially if you can teach an AI to just run through the preexisting library of images and produce decent results on its own.
@@luck3949 it's already 3d
Wow, this effects are so Amazing! Thanks for sharing this with us 😄
I love keeping myself updated with your videos. Thank you.
very very very cool! Can't wait to see the follow up papers
No way? That depth solve is so crisp and consistent, and that's all without any depth sensor or additional camera? Man I can only imagine what this means for VR and AR.
You read my mind! Just a few hours ago (I swear) I was thinking of having to search online to find an algorithm to get depth maps from a video stream. Although I can't use it yet for my project (as the depth maps are not detailed enough yet), imagine how detailed and fast it will be a few papers down the line! I can't wait!
I found your channel by pure luck and god damn. That's been the best subscription I ever made to a youtube channel.
I can't express how most of your video just blew my mind.
And the presentation is perfect. Bite sized yet explained so clearly.
Thank you for your content.
It'll be exciting to see this get sped up to support real-time depth mapping. From the paper, it took 40 minutes to process a video of 244 frames (approx a 9-second video on most devices) so there's quite a bit of work necessary to get to the AR-stage, but these results are already incredibly impressive. Now it's going to be all about speeeeeed!
thank you W&B. because of you, we can watch this channel. Thank you really.
2:34 At first I was kind of skeptical about the accuracy of the depth effect, until I saw that. The way the water refracts the fire hydrant is so cool.
I miss the times where you were also explaining the papers :(
From what I understand from their video: ua-cam.com/video/5Tia2oblJAg/v-deo.html , it's not real-time, it takes a video as input, and by taking from 2 random frames a pixel and trying to estimate it's depth, you'll get 2 approximations. The difference between these 2 is the error that is then backpropagated through a network that, after doing this a lot of times, will end up giving a way better and consistent approximation.
two papers down the line...
Yeah, that's what I starting thinking. I noticed that none of the examples shown was real-time, so this wouldn't work as well in driving ai.
What happens for moving objects?
@@MOOBBreezy I think you still would be able to train an ai on the results.
I still think that this will be the future of CGI tho
That's amazing. I'm glad I was holding onto my papers.
That's amazing. I work with images everyday and I can't imagine improving the result any further ... without including extra data sources, that is.
Your videos always give me the motivation to keep working on my AI degree with enthousiasm, so a huge thanks for that!
Absolutely amazing, kind thanks for sharing this!
Could you share some Al resources for people who aspire to do it too please
YES!!! you humans are doing so good! Keep going
The glowing particles effect... holy crap, this method comes with re-lighting FREE OF CHARGE!
Nice video! wish you success, keep it simple like you do and people will come back!
I've been following the Two Minute Papers channel for quite some time, I think it's been a few years since I found the channel and got in love with the content. Even if I know the intro says "Dear fellow scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér.", because Ive heard and read it so many times, even then I still hear the intro like if it says "Dear fellow scholars, this is too many papers with Carlos Jonas Ifahir". Please, tell me I'm not the only one xD
"What a time to be alive!" - I always loved that line. I can feel the sheer passion! :D
Imagine how this could be used with art sculpting software, where you can reach your hands inside an object and have them be properly occluded (or not occluded) based on this depth map. I had thought this would only be possible using something like lidar or a lytro camera, but it looks like itll work for any camera soon enough! AR is awesome
What a time to be alive!
Excellent! I cannot wait for somebody to code this into Adobe After Effects.
The AR effects look amazing!
Having worked with AR stuff, I can say that this looks amazingly promising.
that this works so well is beyond nutty.
"Hold on to your papers"
Damn! That was a nice one
It's amazing! Thanks!
one of the things I'm most excited for with these depth algorithms is the idea of 360 degree video being able to have depth information so it can be displayed properly in VR. Imagine being able to watch live events in proper 3D VR like you're actually there.
Thank you!
a new era for "here in my garage"-style videos.
The flickering ball on the table looks more like z-fighting to me. Can't wait for your take on GPT-3!
What an exciting time to be alive!
This is so fricking cool!
Well, this is impressive...
This should be very interesting in future...
This is really outstanding.
2:41 neural RGB - thats how i see images in my head when i have headache. That flickering.
This is exactly what I want/expect from a device like the Microsoft Hololens.
Mind, the dev kit I worked with back in 2016 was really pretty good for the form factor (it would scan space around it and produce a mesh from a voxelized space using 5cm voxels) but it couldn't handle real time changes to that environment. Things that didn't move (floor, ceiling, walls, furniture) would be generally stable, but if you were lucky enough to have a person walk by, you'd have a physics ghost of them for about five minutes before that region was rechecked.
But it could not see glass at all (and I suspect that will always be a problem) and I never got a chance to see how it dealt with mirrors.
This. Is. AMAZING!
I really love your Channel, merci beaucoup
This will be incredible for VFX artists like myself. God, the future looks beautiful.
Absolutely incredible
I'd love to see a video about your deal with W/B and learn about how partnerships and ownership circularly benefit the content creation, W/B, yourself, and your viewers.
BUt does it need a high end scanner like the one in the new iPad pro Lidar scanner ? or does it work with all kinds of general smartphone cams
Incredible !
I want this in ARCore, last week!
I'm curious how this handles light refracting through transparent/translucent objects.
Beautiful. Add some sensors (optional, so it can run without them) to the input of that and create a perfect 3D world for a car or machine. Imagine know mixing 2 cameras for a robot, cheap 3D world.
I'm really excited for this, not really for the AR effects, but an accurate depth maps from videos means smartphone cameras could finally record videos with bokeh, there is portrait mode for photos, but there is no good version for videos yet (i think)
IMHO, the quality of their results are already pretty usable for almost all consumer products!
Holy CRAP, I cannot WAIT for this to be implemented into XR, AND driverless cars!
this it the thing ive been waiting for years
shout out to the cat being the patient subject of this video!
that cat is so adorable by the way
I don't really understand half this stuff, but it's really cool to watch. It means automatic rotoscoping so then vfx artists don't have to do so much work by hand roto-ing stuff.
can anyone help me solve a problem of matching a head video of a person on the neck of other person when the portion of the other is not available. i.e cropped out video?
this is such a breakthrough
Dr! First time I hear the title, congrats (maybe it was a long time ago?)
This would be Amazing for VFX.
Blender so needs this.
This could speed up the process of adding visual effects in movies drastically
binocular camera and temporal frame by frame cleaning would help this a lot.
Hi Károly,
I've been following this channel and the general AI field for a few years now, but since I finished school and started working, I've really started to do something with it in my free time and apply the concepts I've been hearing about so much.
I would love your insight on where to go to continue my learning. Especially looking towards network architectures.
I love your videos! Could you do an update to your How To Get Started With Machine Learning video? It's been 4 years and so much has changed!
2^7th!!
This paper blew my mind so hard. I attended class about computer vision for a semester and the vision is so hard and complex (not to learn, to do at all)! Even with depth cameras!
I don't really care about video effects, it would be so amazingly useful without them alltogether!
Just awesome 😮😮
The issue I am having is that there is no hard edges. It's too smoothed. And it's coarse. The resolution is like 1/9 for G vs D.
And if you have multiple sensors, this should vastly improve due to stereo vision. Or even tof sensors build into phones now.
I would use this method in combination with a low resolution thermal camera to do a 3D reprojection for higher resolution and accuracy along usual photogrammetry solutions. I sketched it out a long time ago - but never moved on.
I have seen this paper a few weeks ago and looked into it a little. Can't wait to study this type of stuff at an academic level.
Wow, that's good depth quality; better than stereo depth sensors! Watch out Google Depth API :)
So, you need an AI to run first to create a sharper image to remove movement blur to get accurate object edges and another to apply the movement blurs to the areas that got sharper to keep camera movement consistency?
Interested that most of these clips seem to have a moving camera? Is this necessary? Obviously that would give more information on the depth of an object by comparing each frame to the last, does this method work on static images?
Oh Apples’ Glass team is going to enjoy this paper
I wonder what happens when you pair the colour and depth data with the past data of previous states and a classifier?
Now I want to see what that can do for selfdriving cars since it is much better than previous method.
This really reminds me of that 2012 game Kinect Party. Just makes you wonder how these experiences will look with current tech.
I thought this is how AR worked already! Any news to learn about current AR implementations?
I’ve seen a couple programs like this shown on your channel, where an input image or video can suddenly generate a better zooming effect or useful depth map, but do none of these have demos available?
Two Minute Papers, I almost hear "too many papers". of course, there's never too many papers !!
Just thinking about combining the algorithm to two cameras. Them seeing what we see, making a good 3d scanner
That cat is adorable!
Could this be used for 3D scanning objects with your phone better and more accurate?
This was amazing
There is another app out there that is relatively new called camtrackAR on the apple store that is free and auto tracks footage recorded from iPad or iPhone and creates a camera solve that can be used in blender. Free version allows one point to be used and paid version allows for more than 1 but one can still get some real nice results.
One of my fav parts of each video is the very beginning. I listen to the vid author pronounce his own name, and it sounds amazing. Then i try but its no good lol.
I wonder if this could be combined with an actual depth camera/stereoscopic cameras for higher performance
I feel like the water had a bit too high IOR. Great video.
looks great
2 more cats down the line, the picture will be even fuzzier
Snapchat: I own this now
Who are you?
@@somethingwithbungalows Joe
ًSomething with Bungalow joe mama
Joe okay. understandable. have a nice day, Joe Mama.
I think one important aspect that was left out is the time it takes to calculate these depth maps. If I remember correctly, it was around 20 minutes per frame.
This is going to completely replace bluescrren in the film industry in a few years. No more blue shining lights, no expensive LED background screens, just put up a gray neutral screen as background, or just use any old room. 1000 budget home made films could have the same visual effects as todays block busters :O
What a time to be alive is in two minute paper vedios