For anyone interested, here's my TLDR of the video: Biggest advantage of photogrammetry method (polygonal): model is metrically valid, can be used for measurements, model is fairly lightweight as it's a model, polygonal, and also less computationally taxing to post-process for the same reason, can use in many software aand share easily. Advantage of NeRFs (radiance field): includes more scene information (distance, sky), but this info is estimated, generated (via color info aka a radiance field) in realtime so perhaps it's not the most scientifically accurate. But because a full scene is being computed, creating new "footage" from novel viewpoints is possible. Advantage of Guassian Splat (static point cloud): includes scene information like NeRFs, but is not being computed in realtime aka it's a static point cloud (or splat cloud). Because the visualization is static, Guassian Splats can be used in many visualization software (game engines like Unity, Unreal, as well as 3D software like Blender, Cinema 4D). It's "the best of both worlds". Also, of course, it's the most fun to say :)
Hi, great video - I think it clarifies to a lot of people the difference between the three different technologies. Just one thing that would have been important to mention, is the biggest advantages of NERF and GS is the ability to capture reflections, and even transparency, which is just about impossible with Photogrammetrie. Granted that your example is, of course, not the right one to demonstrate these features but maybe you have a different set of images where you can demonstrate that difference in more detail.
You're absolutely right - in fact I just went out and took some video of my car and some reflective surfaces in the rain - which would be very hard for photogrammetry to reconstruct well - I'll do a side-by-side!
GS can also rasterize thin structures much better than mesh recon or nerf. Give it a few months my bet is GS can surpass meshing in rendering quality and file size.
I have never used either of these techniques, however ever since I've seen these technologies become popular, i had a really poor understanding and had no idea how they compared, but still remained curious. This video filled the gap in my understanding perfectly
It has an interesting backstory too! The legend goes that a utopian community wanted to hollow out the rock to use as a church - and even went so far as to begin chiseling a doorway (you can see the opening in the video/model).
In the future, the 2 others might become an option too, but my favorite aspect of photogrametry is that being a "standard" 3D model, it can be edited in something like Blender, fixed, simplified, reworked, given PBR textures, and then used as an optimized asset in a game engine, like Unity, and for exemple, uploaded to VRChat. So after having done all the drone videos, you could just, be there as a friend, with a properly scaled world, then have your friend walk towards it, showcasing just how big it actually is.
thank you for explaining. Recently these terms were thrown into the room, with a lot of "look at this", without going into the context of what is different about it to photogrammetry in terms of workflow. But now I got a clearer idea of what the future could look like. So when you train a model rather than calculating a static point cloud you are pretty flexible in terms of AI, mixing different models up. Possibly in the future we define a 3d objects by volumes and descriptions that are then filled by AI and either calculated into a performant geometry with another AI model, or the future of digital spaces is based on voxel clouds that are also animated by AI models. quite interesting
Thanks for making this video! I've been playing with photogrammetry for about 6 - 7 years, and have only been a curious bystander with respect to neRF and GS. It's interesting but I envisioned something like neRF and GS in conjunction with 3D meshes several years ago, before learning that they exist. Your video does a wonderful job of explaining the distinction between the three, especially between neRF and GS. I watched til the end! Thank you for creating it. I'll connect with you on your social accounts and hopefully we'll run into each other in the unfolding "metaverse".
There's also a problem with physics, interactivity and the rendering of anything besides real world data (procedural generation would be hell), which altogether makes it likely to only have niche use cases, though I do admit it's cool
Thank you very much for the detailed explanation of the three systems, I am teaching myself to use Agisfoft Metashape for developing 3d models of cave interiors. Your video has really helped, I have just rendered your zip file of Church rock with excellent results the output quality is confirming my techniques and results for image capture underground. Excellent video.
Could be cool to use an example with more variance in material properties, to showcase how different techniques deal with things like reflectivity and transparency.
thank you very much for this video :D superb for future developments in archaeology. I am already checking NERF with Kiri Engine, but I loved to see how to do it with the tools you mentioned. Coool!
I have absolutely no affiliation with anything 3D-related though somehow I find this Gaussian Splatting thing so intriguing though I had no clear understanding what it was haha. Just that you could make high quality looking 3D renders of things with just a video. Cool to actually see a more technical and comparison video about it. Thank you for making this video!
Thank you!! I’ve been looking for a video like this since I learnt about Gaussian splatting. Also the music you played in the showcases was chilling :))
Great comparision of different methods! Looking at the drone video vs the output, it seems clear that all these technologies will get better when we get more processing power. The current output is nowhere close the detail level of input video but there's no reason to think it couldn't be given enough computing resources.
really cool! if you map the panorama photograph to a hemisphere and a ground plane, you can get the model and the panorama to line up perfectly. the spherical mapping gizmo just needs to be in the location the panorama was captured in relation to the ground plane
Neat trick - I've always just mapped the pano to a sphere and set the gizmo to the origin/center of the sphere, but what you suggest could help with extending the ground plane imagery past the model geometry and giving the appearance of continuity. Thanks!
Very intriguing comparison. Had not heard of Gaussian Splatting previously. With Photogrammetry it would be possible to model different sun angle and lighting effects. Could be interesting to explore how the various could take advantage of video captures taken at different times of day. For example, could allow for animating time and motion. Just a thought that might be interesting to explore.
While it's true with photogrammetry you could relight the scene, the shadows are still "baked" into the texture. So in this case, Church Rock would still be casting that shadow even if you put virtual lights into the scene to reimagine it.
intersting fact is that nerf technically uses ml to search for the camera positions. the process stops when the render of the estimated volume matches the photo itself. ( within a threshold )
You're right - I captured at 1080 (my screen's max res) and upscaled it, but didn't realize it because I had my premiere clip previews set to 1/8 res, so didn't notice the blurriness! Sorry!
Would love to see an update on this tech and see what best option there is for making a model of a property for land development and architectural visualization purposes.
Photogrammetry is still the best option for rapid/inexpensive landscape digitization - then you'd want to bring it into CAD/Blender/Max/Unity/UE, etc and add your structures/trees/etc.
Seeing Matthew's analysis of these three technologies was quite insightful. It brought to mind a question an architect friend once posed: Could one feasibly craft an architectural blueprint of a home or apartment using video footage? Considering your experience with these three tech contenders, would you say we're on the brink of making this potential a reality?
You could definitely build a rough model from video footage (provided that the video entered every room). No digitization technology (yet) will output a plan useful to an architect without substantial work by hand- however the power of these techs is that you can achieve results based on very little information (I.e. a series of photos or video) that can then be interpreted by an architect or draftsman and turned into a polished representation, like a plan or section.
@@MatthewBrennan Thank you very much for your response; I have more clues so my friend can make things more efficient. It's a matter of time before new papers are published. 🔜
I think Gaussian splatting could be used more going forward, since it gives you a point cloud(which can be turned into a mesh) and also because it captures reflections. Btw it'd be a great tool for virtual production! Also thanks for this video
Played a bit with gaussian splatting, but found it really hard to get rid of the garbage points/splats that are clearly visible in this video too. You need at least to try to use some kind of select box/sphere to limit them to the object in focus but even then there are stray points that you need to manually select and zap from the data.
There is some functionality for this in the current Unity project, and the updated scripts have helped eliminate many of those "floaters". I've found that it really depends on high-quality, sharp input data - so it's best to use still images rather than video frames.
Right now I get the feeling it's very much a "solution" in search of a problem. But here's another video with some thoughts on the potential practicality: ua-cam.com/video/Ksi_RfY77SI/v-deo.html
What do you thing is currently the best technology to scan faces to create highly accurate 3d files? Which are also future proof in terms of working with them? The priority is accuracy and feasibility for a non-professional. If we are limiting to cost to maybe 5 to 10 grand? Photogrammetry, a 3D scanner like from Einscan, NeRF or anything else? Do you know of any software that could benefit from having more than 1 camera at a time for photogrammetry? When I look at professional studios, they have like a hundred cameras surrounding a person that gets scanned.
Photogrammetry would fit the bill, but you'd need a multi-camera rig. I've scanned a live subject (just the bust - shoulders + head) with a single camera, but it required quite a bit of cleanup in 3D sculpting software. A calibrated high-resolution, multi-camera solution would be the way to go.
Hi Matthew. Great video explanation. Which drone did you use for this test? Do you have by any chance any more drone footage? Have you use the DJI Mavick 3 pro Cine for photogrammetry? Thanks.
I used a Mavic 2 for this model. I have used a number of different drones for photogrammetry in the past, but haven't tried the Mavic 3 yet, although I don't think the Cine model adds anything particularly useful for traditional photogrammetry.
The photogrammetry and NeRF were processed on a desktop computer with a Ryzen 9 3900X CPU + 4070ti GPU. The Gaussian Splatting was processed with an nVidia A100 GPU.
3DGS does something similar, as everything is based on the interpolated “viewing angle”/position of the scene, which of course is tied to the conditions/time (shadow or sun) that each photo was captured under/during.
It's me or from 8:30 and a half the quality of video seems 480? I can appreciate the quality correctly because seems a 480p video rescalated. Aside of that the video is very interesting.
Right now, I think the primary application is in virtual production. This is a relatively new method, so I'm sure as it evolves, new applications will develop. At the moment it is not a straight replacement for any existing digitization or visualization technology.
like a comparison of the data/size on disk? 3DGS and photogrammetry are more lightweight than NeRF - particularly if you want to run it in real-time, where NeRF is very taxing on your GPU.
if you have an rtx series gpu may i suggest "nvidea broadcast" for your mic. it has amazing background noise suppression that works in real time and doesn't reduce the quality by much at all, especially compared to hearing air vents or water leaks.
That's why I'm here, I have driven past that rock hundreds if not thousands of times in my life. I have no knowledge about the actual subject matter of this video but I'll watch the whole thing just out of gratitude for the nostalgia.
Hi Matthew! Thanks for this video. I'm new and a total noob to this field. I'm a Marketer and my line of work I'm having to learn a lot of these things including reality capture, photogrammetry, NeRFs, 3D GS, Digital Twin. Do you have videos that are educational about these aspects which would help a beginner like me to understand the basics?
hey :) question regarding the 3Dmodel in the sphere. did u shoot the panorama exactly on top and center of the poi (the rock)? and is there a tutorial how to do the hole 3D process in blender? thx, kind regards sebastian
Yes, directly over it. I don’t know if there’s a tutorial…I had the idea years ago as a way of “faking” a more extensive scene (see some of ‘digital Hadrians villa’ project videos). Basically you just create a sphere/geo-sphere, invert the normals, and apply a spherically mapped panorama. Hope that helps!
Hello Matthew, thank you very much for your response. I assume I need to create an mp4 file if I want to scan a gaussian splatter. What are the settings? I am really a novice in this. Previously I was doing an fps 30. Is it convenient to set fps60? Thanks in advance.
In my experience, still images (photographs) work much better than video! Follow good photogrammetric practice for capture, and then process as a 3DGS.
Hello, Brennnan, May I ask you how long did the Gaussian Splatting method reconstruction takes you to get that church rock? I used a RTX 3080 machine and it will take me about 24 hours to reconstruct it(I start reconstruction last night, and it is still running now finished 45%). I dont know if it is OK......
It takes about 15 minutes using a 40GB a100… sounds like you could be running out of vram and instead of crashing it’s just hanging. Shouldn’t take that long…
Working with photogrametry for 15 years or more to. And i tested Nerf as soon as they got out, and all nice and cool untill you give them uniform reflective surfaces. Photogrametry and Nerfs do not like uniform glossy surfaces. GS have no problm, they even get transparency,. Is an insane technology, far from perfect, have very limited use case, but i can see already GS being perfect solution for then next generation of realistic 3D google maps. Are very small and a lot easyer to do then NerFs and can be used with night shots to.
So basically , photogrammetry is still king when it comes to realtime applications (games) as it holds the detail even up close and you can retopo it later, and things like gausiang splatting are currently good for video production/vfx, correct ?
@@MatthewBrennan In that case subscribing as I don't want to be left behind on the info XD (and difficult to keep track on everything) . I suspect that when a good and quality method for converting gaussian splatting to 3d mesh comes out, I will see the news here to keep up to date . Cheers mate !
This wasn't meant to be a technical video, but if you want to know more about the technicals behind 3DGS, this is a good one: ua-cam.com/video/HVv_IQKlafQ/v-deo.html
Great video, thank you. Could you advise me how to scan an object very sharply for gaussian splatting? As it happens, I can't get it sharp. The letters are not clear. I own an android (s22 samsung, nikon z50, insta 360 x3). Which one would be the most accurate? As I am using the texture for training purpose. Thanks in advance. Greetings, Sebas
The Nikon z50 would likely be the best (physical shutter + megapixels), although of course it depends on what lens you are using. In my experience, I get the best results using a high-resolution mirrorless camera (compared to an iphone or action camera). Of course - more data = longer processing times, so there is always a trade off or compromise.
Hi, you said we can export camera positions from 1st method and import it into NeRF to reduce amount of work for neural network and speed up a process. Don't you know how to do this with MeshRoom and colmap (or instant ngp)? Recently I was trying to make a NeRF with those tools but found that even after 2 days of work it was still processing the data (which are around 1200 frames from video).
Good video, but I have some tips. 1: Get a capture card so your system usage doesn't effect quality, it's hard to tell which method is better when the resource intensive ones are in 240p. And 2: either write a script or, if you want to maintain your natural manner of speaking, do a dry run first. Go though all your talking points once, then immediately start recording and do it again. Should help you move along a little faster and keep your pauses and "ums" to a minimum.
It would be nice if in a few years the Gaussian splatting has a way to erase things you don't want, or correct blurry parts of a scene. Gaussian to mesh would be the holy grail.
Very well done. You mentioned using data resulting from aligning photos in Metashape in place of ColeMap for NeRF studio. Can this substitution of processing also be done for Gaussian Splatting? I'd love to use the aligned photo data & camera model data from many Metashape (standard edition) projects for GS without using ColeMap as Im already confident in the alignment. Any suggestions on how to accomplish that?
@@MatthewBrennan Thanks! Would I be correct in assuming that this must be run in (or at least with) the professional edition (not the standard edition) of Metashape?
@@MatthewBrennanWondering if your still involved much with Gaussian Splatting as I haven't seen you post any recent content on UA-cam on the subject. What you did share was amazing! Not sure if you are aware that this summer Metashape added COLEMAP format export of the point cloud & undistorted images to the standard edition. Very happy with that as I can now process much more data with better results. I'd love to chat sometime to talk about GS and share insights on processing larger datasets and mixed images from multiple cameras (different fov & distortion). Any interest in that?
Could you define your usage of "photogrammetry"? Is it only one particular algorithm? I thought it's any method to convert photos to 3D, meaning nerf and gsplats are also photogrammetry technologies. It's like comparing "vehicles", "cars" and "bicycles".
I'm using the term "photogrammetry" to refer generally to the estimation of camera poses (photogrammetric alignment). Historically, this was done manually between a few photographs, with the intention of extracting measurements - but of course today we can use computing power to apply this to thousands of digital images. In my opinion, "photogrammetry" in the true sense is just the "first step" in a number of digitization/visualization workflows - pointcloud, 3D polygonal mesh, NeRF, 3DGS, etc...
Cool stuff! I was thinking regarding the fact that extracting images from a movie gives poor photos: is there a way to extract higher quality images that you're aware of? I'm thinking it would be fairly straight forward to create an ai that takes five images in a row and creates a de-blurred version of the middle image using data from all five images. Or are there additional issues?
The best type of image is a high resolution digital still photo, with low iso (low sensor noise) and high sharpness (I.e. high f-number… usually f/8 or higher)
@@MatthewBrennan I see - but much could be gained if we were able to produce similar images from moving images - it would just be way faster to capture that way, not only for drones. Probably someone will have a go at it. I'm feel better images than a random frame from a movie could be generated, but never as good as the type of image you're describing. BTW, someone mentioned that within a few years, a team shooting footage on location will map the site using photogrammetry as a matter of course. We're used to voice dubbing in post production, but it's hard to shoot a new scene on a site that's distant - or in different weather/season. I think there are interesting things ahead!
which would work best for a house with trees all around? and is there a way to see past the trees in any way? or would i need to fly my drone down into and past the tree line?
Thx for the vids - deep respect for your knowledge and sharing it. (needed to watch it in 1.5x tho - otherwise it would have been kind of as if told in slow motion xD)
I love how you can hear the fan going with the gaussian splatter
😂 my poor comp was breathing heavy
You can hear it from the jump when he switches to NeRF too.
@@MatthewBrennanThought it was raining there for a moment, then realized I’ve head that same sound in my office. Def taxing the GPU, but great video.
PC became a fighter jet with that one
NeRF and gaussian splatting are so realistic that they even captured the wind.
hahaha
For anyone interested, here's my TLDR of the video: Biggest advantage of photogrammetry method (polygonal): model is metrically valid, can be used for measurements, model is fairly lightweight as it's a model, polygonal, and also less computationally taxing to post-process for the same reason, can use in many software aand share easily. Advantage of NeRFs (radiance field): includes more scene information (distance, sky), but this info is estimated, generated (via color info aka a radiance field) in realtime so perhaps it's not the most scientifically accurate. But because a full scene is being computed, creating new "footage" from novel viewpoints is possible. Advantage of Guassian Splat (static point cloud): includes scene information like NeRFs, but is not being computed in realtime aka it's a static point cloud (or splat cloud). Because the visualization is static, Guassian Splats can be used in many visualization software (game engines like Unity, Unreal, as well as 3D software like Blender, Cinema 4D). It's "the best of both worlds". Also, of course, it's the most fun to say :)
well put!
Another advantage of Gaussian Splat over NeRF is that the splats are real objects in 3D; you can edit, move, delete parts you don't like.
Not exaggerating, when Gaussian Splatting became to sudden prominence, I was waiting for a video exactly like this.
Hi, great video - I think it clarifies to a lot of people the difference between the three different technologies.
Just one thing that would have been important to mention, is the biggest advantages of NERF and GS is the ability to capture reflections, and even transparency, which is just about impossible with Photogrammetrie. Granted that your example is, of course, not the right one to demonstrate these features but maybe you have a different set of images where you can demonstrate that difference in more detail.
You're absolutely right - in fact I just went out and took some video of my car and some reflective surfaces in the rain - which would be very hard for photogrammetry to reconstruct well - I'll do a side-by-side!
photogrammetry*
GS can also rasterize thin structures much better than mesh recon or nerf. Give it a few months my bet is GS can surpass meshing in rendering quality and file size.
Good point
I have never used either of these techniques, however ever since I've seen these technologies become popular, i had a really poor understanding and had no idea how they compared, but still remained curious. This video filled the gap in my understanding perfectly
The splatter is really cool looking. Looks like a hallucination or a dream
This was exactly what i was looking for! I tried gaussian in unity in vr. I have to try with your conclusion. Thanks!!! Subscribed! :)
That's an amazingly beautiful rock formation, not that it's the topic of the video but just felt like mentioning it
It has an interesting backstory too! The legend goes that a utopian community wanted to hollow out the rock to use as a church - and even went so far as to begin chiseling a doorway (you can see the opening in the video/model).
In the future, the 2 others might become an option too, but my favorite aspect of photogrametry is that being a "standard" 3D model, it can be edited in something like Blender, fixed, simplified, reworked, given PBR textures, and then used as an optimized asset in a game engine, like Unity, and for exemple, uploaded to VRChat.
So after having done all the drone videos, you could just, be there as a friend, with a properly scaled world, then have your friend walk towards it, showcasing just how big it actually is.
so can gausian splats
At last, a serious and in-depth video without hyperactive editing and gimmicks 👍 Amazing how NeRFs and splats give you that change in lighting.
thank you for explaining. Recently these terms were thrown into the room, with a lot of "look at this", without going into the context of what is different about it to photogrammetry in terms of workflow. But now I got a clearer idea of what the future could look like. So when you train a model rather than calculating a static point cloud you are pretty flexible in terms of AI, mixing different models up. Possibly in the future we define a 3d objects by volumes and descriptions that are then filled by AI and either calculated into a performant geometry with another AI model, or the future of digital spaces is based on voxel clouds that are also animated by AI models. quite interesting
Thanks for making this video! I've been playing with photogrammetry for about 6 - 7 years, and have only been a curious bystander with respect to neRF and GS. It's interesting but I envisioned something like neRF and GS in conjunction with 3D meshes several years ago, before learning that they exist. Your video does a wonderful job of explaining the distinction between the three, especially between neRF and GS. I watched til the end! Thank you for creating it. I'll connect with you on your social accounts and hopefully we'll run into each other in the unfolding "metaverse".
This is absolutely amazing! I think Gaussian Splatting might replace rasterised/polygonal rendering in the near future!
GS is great, but it will be a real challenge to invent an optimized way of applying real time lighting
@@dmitriytuchashvili8594 agree, interactive lightning is the main issue
Curious to see how GS handles reflections, normals, displacements, transluscency and so on.
There's also a problem with physics, interactivity and the rendering of anything besides real world data (procedural generation would be hell), which altogether makes it likely to only have niche use cases, though I do admit it's cool
@@constantinosschinas4503 see this video: ua-cam.com/video/gheD8vrOJNI/v-deo.html
Thank you very much for the detailed explanation of the three systems, I am teaching myself to use Agisfoft Metashape for developing 3d models of cave interiors. Your video has really helped, I have just rendered your zip file of Church rock with excellent results the output quality is confirming my techniques and results for image capture underground. Excellent video.
Could be cool to use an example with more variance in material properties, to showcase how different techniques deal with things like reflectivity and transparency.
See my video here :) Reflective Object: Gaussian Splatting radiance field vs. Photogrammetry mesh
ua-cam.com/video/gheD8vrOJNI/v-deo.html
Thanks for posting this. Adding Houdini to your tool set will give you the ability to leverage volumes and point clouds to their fullest potential.
Thanks for the good comparison video! You did a great job highlighting many of the topics of interest! 👏👏👏
thank you very much for this video :D superb for future developments in archaeology. I am already checking NERF with Kiri Engine, but I loved to see how to do it with the tools you mentioned. Coool!
Amazing comparison! And very cool flying scenes for NeRF and Gaussian Splats.
I have absolutely no affiliation with anything 3D-related though somehow I find this Gaussian Splatting thing so intriguing though I had no clear understanding what it was haha. Just that you could make high quality looking 3D renders of things with just a video. Cool to actually see a more technical and comparison video about it. Thank you for making this video!
Thank you, it was very educational and enjoyable video!
Thank you!! I’ve been looking for a video like this since I learnt about Gaussian splatting. Also the music you played in the showcases was chilling :))
Great comparision of different methods! Looking at the drone video vs the output, it seems clear that all these technologies will get better when we get more processing power. The current output is nowhere close the detail level of input video but there's no reason to think it couldn't be given enough computing resources.
honestly I love the gaussian splatting look. Especially when you move "too" close :D
Thanks for the video! The best way to learn the difference about NeRF and gaussian splatting 🙏🏻
Thanks a lot! That was exactly the quick introduction to the subject that I needed!
really cool! if you map the panorama photograph to a hemisphere and a ground plane, you can get the model and the panorama to line up perfectly. the spherical mapping gizmo just needs to be in the location the panorama was captured in relation to the ground plane
Neat trick - I've always just mapped the pano to a sphere and set the gizmo to the origin/center of the sphere, but what you suggest could help with extending the ground plane imagery past the model geometry and giving the appearance of continuity. Thanks!
Nobody gonna talk about how the rock looks like a space ship that turned into stone?
🚀
Enjoyable and educational! Thanks Matthew!
Very intriguing comparison. Had not heard of Gaussian Splatting previously. With Photogrammetry it would be possible to model different sun angle and lighting effects. Could be interesting to explore how the various could take advantage of video captures taken at different times of day.
For example, could allow for animating time and motion. Just a thought that might be interesting to explore.
While it's true with photogrammetry you could relight the scene, the shadows are still "baked" into the texture. So in this case, Church Rock would still be casting that shadow even if you put virtual lights into the scene to reimagine it.
I think photogrametry and some kind of Gaussian splatting will be fused one day
Very valuable information for us 3D creators. Thank you very much!
The fan kicking on when the Gaussian Splat started rendering got to me 🤣🤣
Good comparison and review. Good notes, links in the video info grey area.
intersting fact is that nerf technically uses ml to search for the camera positions. the process stops when the render of the estimated volume matches the photo itself. ( within a threshold )
I love the examples, and the discussion on the technical differences. Though could definitely use better quality screen capture.
You're right - I captured at 1080 (my screen's max res) and upscaled it, but didn't realize it because I had my premiere clip previews set to 1/8 res, so didn't notice the blurriness! Sorry!
@@MatthewBrennan Some great points in there. Thanks for the vid.
Looks more like 360p mate, maybe you set premiere to render using preview cache @@MatthewBrennan
@@lemovision could be- I think I fixed it for subsequent exports, at least 🙃
UA-cam compression likely not helping either.
Ooh thanks for this overview. Very helpful.
Great video, I think if you reuploaded this with better quality it could gather even more attention.
Unfortunately I don’t think you can replace previous uploads on UA-cam
Would love to see an update on this tech and see what best option there is for making a model of a property for land development and architectural visualization purposes.
Photogrammetry is still the best option for rapid/inexpensive landscape digitization - then you'd want to bring it into CAD/Blender/Max/Unity/UE, etc and add your structures/trees/etc.
Great comparison! Could you have cropped the splats that were interrupting the sky in your Unity example for a cleaner look - or is that limited?
In theory it should be possible, because the splats are directly related to the sparse cloud from COLMAP - I'm planning to investigate this.
11:26 bro uploaded the video in 4k but recorded the screen with an amazon doorbell camera 💀
😂 yep. Lesson learned. The renderings were 4K but my screen recording was 1080 🥲
Lol
Seeing Matthew's analysis of these three technologies was quite insightful. It brought to mind a question an architect friend once posed: Could one feasibly craft an architectural blueprint of a home or apartment using video footage? Considering your experience with these three tech contenders, would you say we're on the brink of making this potential a reality?
You could definitely build a rough model from video footage (provided that the video entered every room). No digitization technology (yet) will output a plan useful to an architect without substantial work by hand- however the power of these techs is that you can achieve results based on very little information (I.e. a series of photos or video) that can then be interpreted by an architect or draftsman and turned into a polished representation, like a plan or section.
@@MatthewBrennan Thank you very much for your response; I have more clues so my friend can make things more efficient.
It's a matter of time before new papers are published. 🔜
I think Gaussian splatting could be used more going forward, since it gives you a point cloud(which can be turned into a mesh) and also because it captures reflections. Btw it'd be a great tool for virtual production! Also thanks for this video
Thanks for the good video and for providing the nice dataset. What is the focal length of your drone's camera (full frame equivalent)?
35mm equivalent is ~28mm. It's a 1" CMOS 20mpx sensor (for still images). However this dataset uses 4k video.
@@MatthewBrennan thanks for your answer. I'm trying to run your dataset using micmac photogrammetry. I'll let you know if I get good results!
Played a bit with gaussian splatting, but found it really hard to get rid of the garbage points/splats that are clearly visible in this video too. You need at least to try to use some kind of select box/sphere to limit them to the object in focus but even then there are stray points that you need to manually select and zap from the data.
There is some functionality for this in the current Unity project, and the updated scripts have helped eliminate many of those "floaters". I've found that it really depends on high-quality, sharp input data - so it's best to use still images rather than video frames.
Thanks - I'd like to see a comparison of how you would actually use these in production - especially NERF and splatting.
Right now I get the feeling it's very much a "solution" in search of a problem. But here's another video with some thoughts on the potential practicality: ua-cam.com/video/Ksi_RfY77SI/v-deo.html
What do you thing is currently the best technology to scan faces to create highly accurate 3d files? Which are also future proof in terms of working with them? The priority is accuracy and feasibility for a non-professional. If we are limiting to cost to maybe 5 to 10 grand? Photogrammetry, a 3D scanner like from Einscan, NeRF or anything else? Do you know of any software that could benefit from having more than 1 camera at a time for photogrammetry? When I look at professional studios, they have like a hundred cameras surrounding a person that gets scanned.
Photogrammetry would fit the bill, but you'd need a multi-camera rig. I've scanned a live subject (just the bust - shoulders + head) with a single camera, but it required quite a bit of cleanup in 3D sculpting software. A calibrated high-resolution, multi-camera solution would be the way to go.
Wonderful work, really appreciate it
Hi Matthew. Great video explanation. Which drone did you use for this test? Do you have by any chance any more drone footage? Have you use the DJI Mavick 3 pro Cine for photogrammetry? Thanks.
I used a Mavic 2 for this model. I have used a number of different drones for photogrammetry in the past, but haven't tried the Mavic 3 yet, although I don't think the Cine model adds anything particularly useful for traditional photogrammetry.
Hi Matt, are you testing with NV Omniverse? Great video!
No. I tried it about a year ago but didn't find it very compelling.
Great comparison. Thanks!
The last shot is really good!
Thank you for explaining brother! 💪
Matthew, may i know what is your computer specification when creating this?
The photogrammetry and NeRF were processed on a desktop computer with a Ryzen 9 3900X CPU + 4070ti GPU. The Gaussian Splatting was processed with an nVidia A100 GPU.
Interesting how the NeRF has captured the changing cloud shadow over time as a positional aspect
3DGS does something similar, as everything is based on the interpolated “viewing angle”/position of the scene, which of course is tied to the conditions/time (shadow or sun) that each photo was captured under/during.
superb explanation...thank you for sharing your knowledge.
Pity about the low resolution. I assume it's low resolution as that's how it appears on my computer, even at 4K. Very interesting though!
Yea, I accidentally screen captured at 1080, but rendered everything at 4K!
Thanks for the video and links.
It seems that the next step is a neural/AI enhanced version of guassian splatting to improve quality/render performance/data size.
It's me or from 8:30 and a half the quality of video seems 480?
I can appreciate the quality correctly because seems a 480p video rescalated.
Aside of that the video is very interesting.
Is this purpose-driven for a particular field? I've always been fond of the intersection between geospatial tech and the broader CS/gaming world.
Right now, I think the primary application is in virtual production. This is a relatively new method, so I'm sure as it evolves, new applications will develop. At the moment it is not a straight replacement for any existing digitization or visualization technology.
Outstanding! How about size comparison? Thanks
like a comparison of the data/size on disk? 3DGS and photogrammetry are more lightweight than NeRF - particularly if you want to run it in real-time, where NeRF is very taxing on your GPU.
if you have an rtx series gpu may i suggest "nvidea broadcast" for your mic. it has amazing background noise suppression that works in real time and doesn't reduce the quality by much at all, especially compared to hearing air vents or water leaks.
Thanks for the suggestion - I'll check broadcast out. I learned a lot about recording videos at home (since) making this video.
Very nice!
interesting , good informative vid , thankyou, as a side note, lets consider Bob ross once painted 50 sheds and 1 tree.
Oh hey I've been there. Recognized it before you even said "Church Rock". Nice stretch of road just outside Canyonlands National jawdropping park.
That's why I'm here, I have driven past that rock hundreds if not thousands of times in my life. I have no knowledge about the actual subject matter of this video but I'll watch the whole thing just out of gratitude for the nostalgia.
you can also compare nvidia's neuralangelo. that would be great
Hi Matthew! Thanks for this video. I'm new and a total noob to this field. I'm a Marketer and my line of work I'm having to learn a lot of these things including reality capture, photogrammetry, NeRFs, 3D GS, Digital Twin. Do you have videos that are educational about these aspects which would help a beginner like me to understand the basics?
great video. except that the definition is not good even in 2160, it appeared blurry ?
4k60fps video. But you recorded your Desktop in 720p or what?
yep. Oops. I've fixed it in subsequent videos.
Great work! Keep doing the stuff! Like and subcribed!
It should be no problem to import the photogrammetry model into blender, import a world texture (photosphere) and render a similar camera path.
hey :) question regarding the 3Dmodel in the sphere. did u shoot the panorama exactly on top and center of the poi (the rock)? and is there a tutorial how to do the hole 3D process in blender? thx, kind regards sebastian
Yes, directly over it. I don’t know if there’s a tutorial…I had the idea years ago as a way of “faking” a more extensive scene (see some of ‘digital Hadrians villa’ project videos). Basically you just create a sphere/geo-sphere, invert the normals, and apply a spherically mapped panorama. Hope that helps!
thx for your response and infos! have tried it right now on one of my models and it was a success :) @@MatthewBrennan
@@klangr4usch Awesome! Send me a link if you post it on sketchfab!
Hello Matthew, thank you very much for your response. I assume I need to create an mp4 file if I want to scan a gaussian splatter. What are the settings? I am really a novice in this. Previously I was doing an fps 30. Is it convenient to set fps60? Thanks in advance.
In my experience, still images (photographs) work much better than video! Follow good photogrammetric practice for capture, and then process as a 3DGS.
That was good man - thanks
Where is the panorama sphere texture coming from that you used in the end?
It’s a panorama I took using a UAV directly above church rock.
very very interesting
I'd say, if your using reflective or translucent surfaces,i 100% recommend nerfs instead of photogrammatry, cuz it does it somuch better
I made another video explicitly comparing the two- the photogrammetry model actually turned out pretty well.
Why the video is blurred at 1440p?
Hello, Brennnan, May I ask you how long did the Gaussian Splatting method reconstruction takes you to get that church rock? I used a RTX 3080 machine and it will take me about 24 hours to reconstruct it(I start reconstruction last night, and it is still running now finished 45%). I dont know if it is OK......
It takes about 15 minutes using a 40GB a100… sounds like you could be running out of vram and instead of crashing it’s just hanging. Shouldn’t take that long…
@@MatthewBrennan Yes. I changed a 3090 machine and now it is less in one hour
what is the name of the music used at 14:30?
Working with photogrametry for 15 years or more to. And i tested Nerf as soon as they got out, and all nice and cool untill you give them uniform reflective surfaces. Photogrametry and Nerfs do not like uniform glossy surfaces. GS have no problm, they even get transparency,. Is an insane technology, far from perfect, have very limited use case, but i can see already GS being perfect solution for then next generation of realistic 3D google maps. Are very small and a lot easyer to do then NerFs and can be used with night shots to.
So basically , photogrammetry is still king when it comes to realtime applications (games) as it holds the detail even up close and you can retopo it later, and things like gausiang splatting are currently good for video production/vfx, correct ?
Yep, essentially :)
@@MatthewBrennan In that case subscribing as I don't want to be left behind on the info XD (and difficult to keep track on everything) . I suspect that when a good and quality method for converting gaussian splatting to 3d mesh comes out, I will see the news here to keep up to date .
Cheers mate !
thanks this was super helpful
I don't know what is the third method more than I did before starting your video
This wasn't meant to be a technical video, but if you want to know more about the technicals behind 3DGS, this is a good one: ua-cam.com/video/HVv_IQKlafQ/v-deo.html
Great video, thank you. Could you advise me how to scan an object very sharply for gaussian splatting? As it happens, I can't get it sharp. The letters are not clear. I own an android (s22 samsung, nikon z50, insta 360 x3). Which one would be the most accurate? As I am using the texture for training purpose. Thanks in advance. Greetings, Sebas
The Nikon z50 would likely be the best (physical shutter + megapixels), although of course it depends on what lens you are using. In my experience, I get the best results using a high-resolution mirrorless camera (compared to an iphone or action camera). Of course - more data = longer processing times, so there is always a trade off or compromise.
Hi, you said we can export camera positions from 1st method and import it into NeRF to reduce amount of work for neural network and speed up a process. Don't you know how to do this with MeshRoom and colmap (or instant ngp)? Recently I was trying to make a NeRF with those tools but found that even after 2 days of work it was still processing the data (which are around 1200 frames from video).
I use metashape instead of COLMAP, because COLMAP is very slow and gives subpar results (it’s open source though, which is nice).
@@MatthewBrennan I'll check this as well, thanks.
What dhrone and camera you used? :)
DJI Mavic 2
Good video, but I have some tips. 1: Get a capture card so your system usage doesn't effect quality, it's hard to tell which method is better when the resource
intensive ones are in 240p. And 2: either write a script or, if you want to maintain your natural manner of speaking, do a dry run first. Go though all your talking points once, then immediately start recording and do it again. Should help you move along a little faster and keep your pauses and "ums" to a minimum.
It would be nice if in a few years the Gaussian splatting has a way to erase things you don't want, or correct blurry parts of a scene. Gaussian to mesh would be the holy grail.
It's possible (albeit somewhat crudely) to edit the Gaussian cloud now, in Unity.
Very well done. You mentioned using data resulting from aligning photos in Metashape in place of ColeMap for NeRF studio. Can this substitution of processing also be done for Gaussian Splatting? I'd love to use the aligned photo data & camera model data from many Metashape (standard edition) projects for GS without using ColeMap as Im already confident in the alignment. Any suggestions on how to accomplish that?
Yes - I use this script: github.com/agisoft-llc/metashape-scripts/blob/master/src/export_for_gaussian_splatting.py
@@MatthewBrennan Thanks! Would I be correct in assuming that this must be run in (or at least with) the professional edition (not the standard edition) of Metashape?
hmm... I'm actually not sure if the standard version supports python scripts. I am using the professional version to process this data.
@@MatthewBrennanWondering if your still involved much with Gaussian Splatting as I haven't seen you post any recent content on UA-cam on the subject. What you did share was amazing! Not sure if you are aware that this summer Metashape added COLEMAP format export of the point cloud & undistorted images to the standard edition. Very happy with that as I can now process much more data with better results. I'd love to chat sometime to talk about GS and share insights on processing larger datasets and mixed images from multiple cameras (different fov & distortion). Any interest in that?
@@LaunchedPix Yes! Have been meaning to upload something - hopefully in the next week or so. Happy to chat sometime - just send me an email!
Very cool work thanks!
Could you define your usage of "photogrammetry"? Is it only one particular algorithm? I thought it's any method to convert photos to 3D, meaning nerf and gsplats are also photogrammetry technologies. It's like comparing "vehicles", "cars" and "bicycles".
I'm using the term "photogrammetry" to refer generally to the estimation of camera poses (photogrammetric alignment). Historically, this was done manually between a few photographs, with the intention of extracting measurements - but of course today we can use computing power to apply this to thousands of digital images. In my opinion, "photogrammetry" in the true sense is just the "first step" in a number of digitization/visualization workflows - pointcloud, 3D polygonal mesh, NeRF, 3DGS, etc...
@@MatthewBrennan Thanks!
Imaging google earth with gaussian splatting.
The capture of the SW is really blurry 😞
how did you get the camera positions?
photogrammetric alignment using Agisoft Metashape.
@@MatthewBrennan thank you!
Cool stuff! I was thinking regarding the fact that extracting images from a movie gives poor photos: is there a way to extract higher quality images that you're aware of? I'm thinking it would be fairly straight forward to create an ai that takes five images in a row and creates a de-blurred version of the middle image using data from all five images. Or are there additional issues?
The best type of image is a high resolution digital still photo, with low iso (low sensor noise) and high sharpness (I.e. high f-number… usually f/8 or higher)
@@MatthewBrennan I see - but much could be gained if we were able to produce similar images from moving images - it would just be way faster to capture that way, not only for drones. Probably someone will have a go at it. I'm feel better images than a random frame from a movie could be generated, but never as good as the type of image you're describing.
BTW, someone mentioned that within a few years, a team shooting footage on location will map the site using photogrammetry as a matter of course. We're used to voice dubbing in post production, but it's hard to shoot a new scene on a site that's distant - or in different weather/season. I think there are interesting things ahead!
which would work best for a house with trees all around? and is there a way to see past the trees in any way? or would i need to fly my drone down into and past the tree line?
I would use photogrammetry, then edit out/delete the trees
so go in front of the trees to get the house? drone whole area and then camera to move into more detailed spots? @@MatthewBrennan
Yes, you can also combine the drone data with photos taken by hand using a high quality mobile phone or still camera
@@MatthewBrennan awesome thank youf or the help.
Thx for the vids - deep respect for your knowledge and sharing it. (needed to watch it in 1.5x tho - otherwise it would have been kind of as if told in slow motion xD)
😂