I'm so glad you did this Proof of Concept, it looks absolutely amazing... and thanks for the shout out 😊 I hope this will be integrated in every 3D program at some point. There is so much room for optimization in this process. For example the point cloud generation could be done automatically inside the engine. Also, there are already early tests of dynamically lighten and animating Gaussian Splats. We've got an interesting tech right here that could revolutionize the way we render 3D scenes all together 😊 Keep up the fantastic work... 😊👍 Btw... I wonder how fictional or cartoony content would turn out 🤯
This 3DGS actually could be the "front end" to so many tools that have limited interfaces because the finished visual is a great approximation of reality. Every geographic information system, facility management system, BIM\Fabrication package & surveying application could use this technology. The limiting factor for so long has been differing file formats and interfaces. This could be the 3d printing press that makes digital data almost agnostic. 3D has come a long way since the 1980's but is still the same method and tools, but faster. If you use your room as an example you could have all the services, construction materials, fabrication plans, building approvals, energy calculations etc all use the same 3DGS format to display information on any pixel generating viewer. Exciting times.
Olli, this is a fascinating technique and the results in the real-time animation look amazingly good. Especially when you consider that it was the first attempt. Congratulations!
This is pretty insane. I think in theory it should be possible to render directly into 3d gaussian splats, without ever having to do any point clouds or training process.
Not sure about training, but the point cloud could definitely be generated directly from the geometry. Remember gaussian splats are an extension of a point cloud, there is no way to avoid a point cloud.
@@AMessfulI think you’d need to convert the mesh into a textured form first and derive the point cloud from that (using edge detection or similar). This cloud could probably be pruned a bit (more pruning means lower quality but faster). Then you could path trace from each point to get lighting/reflections etc. to set the spherical harmonics coefficients of the colour components. Just guessing here (I’m not a 3D graphics researcher) and I don’t know if that’s any more practical than just using carefully chosen probe cameras and doing the entire original pipeline. One thing that would probably be better though would be fewer floaters and more accurate edges (because we know exactly where in 3D each point is by transforming the texture coordinates by the geometry’s transform).
Great job, I think tihis is the future of 3d realtime graphics, and sure a new evolution of 3D gaussian will incorporate animation, with animated elements inside the scene
Olli Fantastic video and great show of the usecase! Even with NeRFs, a friend of ours managed to get a 3D scene created from a Spiderman Game and it was awesome! We were thinking of testing that out with Gaussians and see how it would turn out. Clearly it is going to look awesome!!
Ay, Bad Decisions! I just got finished watching you guys' video on this tech! It was really great, too! Scanning a 3D scene from a game is something I've wanted to try since I first heard about NeRFs, but I haven't known the best way to go about doing it. Has your friend made a video or documented how he did it?
Fantastic experiment! It'll be fantastic once we can go direct from scan or DCC into real time gaussian splats. I wonder how we could do dynamic lighting within Gaussian Splats though. Right now, it would probably be faster to bake lighting information into the textures of your 3D so you can get Cycles style lighting in real time in Eevee. Looking forward to seeing how this tech progresses!
OK, so this might be a game-changer for the interior design. My wife renders her Revit interiors using Corona. She positions the camera, sets lighting, and makes a rendered shot. Corona renders only on CPU so this takes up to one and a half hours for one shot. There are at LEAST 20 shots per project, which means 30-40 render iterations. If she could instead capture the whole project and just fly with the camera and make screenshots of such perfect quality, this would drastically lower the time and electricity that it takes to finish a project visualization. I would love to see how your project turns out. I can see big commercial (or open-source) potential in it! Would be glad to help if this goes open-source! Would gladly consider buying it for my wife if it is commercial!
Instantly though about this on the first video I saw, especially as here you've took the "classic" route, but what if point could be trained separately then put together with some kind of 3D tools (a bit like the simple color to image IA we had few time ago)? I really think that the capturing still image to cloud point step are not the big thing (as it's the same as classic photogrammetry obviously) and that 3DGS could lead to even more crazy result with another kind of source of data But first the obvious next step would be to generate automatically all the needed point of view and go straight from 3D rendering with maxed ray casting stuff to cloud points as the 3D engine would be able to identify points by itself. I wonder how much better points would help this tech
Fascinating idea and great video! Wonderful accent 🤗 The effort put into clear speech is much appreciated! A real problem for most "native speaking" channels 😉
This is really cool - I'm sure someone will move to generating these splats directly from blender and avoid the loss of detail from generating frames then training the splat. The renderer knows the geometry, camera pos, etc. and could provide much more useful detail when creating each surface instead of inferring. Interested to dive deeper in to this.
I would love to see someone use the 3d model to inform the machine learning model about generated point cloud in order to help remove noise/ghosts, and to make the models more accurate. or perhaps create the pointcloud directly from an engine
I'm loving these. The most obvious gaming usecase would be realtime cinematics. Cinematics are often one of two things: - an ingame cinematic with obviously game-level graphics/lighitng/models, or - a prerendered cinematic with much higher quality assets and methods that can't be done realtime. If you're using a mix of gaussian splatting for the environment and then spending all your traditional rendering power on your moving parts/characters, then you could have cinematics nearly as pretty as pre-rendered cinematics. And that's just until we get animated splats. This way you can have faultless splats too, because the camera would only ever use the path the splat input references took.
Yay, finally, someone tested it! 🙏 Now replace the Blender Render(!) step with a generative AI process, like Runway ML or ControlNet, and use just a 3D OpenGL output (as a "3D Skeleton") from Blender where colors correspond with different object prompts for the generative AI. Or any similar process you can make. Consistency is key though.
kinda imagine there must be a more direct way to convert from 3d model to the "point cloud" skipping the render , but it indeed is very promising technique
Basically future games could render a high quality interior simular to traditional texture/light baking such as in Unreal Engine, but then use it as Gaussian Splatting while at the same time having a defined playground that can't be exited because Gaussian Splatting will then blur the area. This would offer a overall higher quality photorealism experience then Unreal Engines light bakes for ex. . But the question is if that could be merged with interactions inside that world that it looks right. As the light on interactable props have to be rendered somehow. Also over the years the actual light baking that looks photorealistic haven't been used in games, mostly by archviz companys only. Interesting what could this offer then. I guess film production or product visualisations benefit more from that.
Brilliant stuff! Having well above real time rendering of a complex "photorealistic" 3D landscape is nothing less than sorcery! I also wonder what the uncapped FPS would have been.
Hi Olli, love your work - really appreciate your experiments. I have a somewhat similar question that you address in this video but with pre-existing point cloud data. Do you think it would be possible to create a gaussian splatting from point cloud data like .las, instead of images.
I’m not sure. Training needs also the source images to generate Gaussian Splatting model. The point cloud itself is not enought. This source code uses .ply point cloud files. But I think that any point cloud format can be convert to .ply format.
Interesting topic! But it would have been even better if you had added some computer specifications. For example what GPU and how much VRAM does it have?
Very interesting I indeed believe the point cloud could be directly produced from the 3d geometry tesselated to a high poly count and the texture baked to vertces. It should be faster than rendering all the images and their wouldn't be any forgotten or occluded portions of the model. Thank you for your good videos
The part that’s missing is still the path tracing. Though maybe there’s a way to bake that in as well? Failing that, I wonder if there’s a way to write an algorithm to calculate the optimal camera path through a scene to maximize coverage while minimizing redundancy.
@@mortenjorck Yeah I think you're right, but isn't like 99% of cycles render time effectively just calculating lighting? I can't see why you wouldn't get equally good results just by using Eevee with baked lighting.
Tätä tekniikkaa voisi käyttää myös 3D-pelien toteutukseen. Saisi ihan upeita pelejä kehitettyä, vaikka tässä onkin paljon rajoitteita verrattuna polygoniperustaiseen toteutukseen. Esimerkiksi joku lentosimulaattori voisi toimia ihan hyvin tai autopeli. Saisi tosi realistisen näköisiä pelimaailmoja tehtyä, kun voi ensinnä vaikka Unreal Enginellä renderöidä tosi yksityiskohtaisen ja upeasti valaistun maailman, jonka muuntaa tuollaiseen pistemalliin
With computer-generated geometry like in this example, I wonder if it would be better to ‘manually’ place gaussians according to real scene geometry, rather than using photogrammetry to place them.
thanks for doing this video.. I always assumed it would work in theory but was lazy and never created a scene to test it :). great stuff what about the refractions in the water? do those still hold up and change on view change?
I think right now within the scope of this project, 360 videos could be looked at for training data. Much more information and should fill in the gaps nicely. In the future it should be possible to extract texture information from the 3d file and use it along with the point cloud information to skip the pre-rendering entirely.
Very interseting worklow. And great idea to try this way. Btw. Unreal Engine can compute lightmaps for every surface, calculate how much light will be on every surface by using path tracing and then you can basically change location of camery and get instant raytraced result. It of course don't work when they are moving objects or light in the scene. But for static scene like this it is awesome. Would be great if Blender could to the same.
like ... it's cool ... but what is it for.... Are we going to see movies shot in some future tech 3DGS cameras maybe? For maybe stronger ai's to reinterpolate actual hybrid 3DGS and vector format or "baking"?
Fascinating or as we say: todella mielenkiintoista. En ole GausSplattingiä tutkinu aiemmin ja mietin että sellainen voisi olla kova jos esim tuosta huoneesta saisi exportattua ajettavan tiedoston ja asiakas voisi avata sen omalla koneellaan ja pyöritellä kameraa ympäriinsä. Kuinka isoja tiedostoja noi sitten on mitä python laskee?
Tää on erittäin uutta tekniikkaa. Lähdekoodi julkaistiin tuossa syyskuun alussa 2023. Katseluohjelmia on hyvin vähän vielä. 3DGS malleista voi tulla aika isojakin: 800Mb - 1.3gb mutta näihin on jo kehitelty pakkausmenetelmää jolla datasetin koko saadaan merkittävästi pienemmäksi. Mielenkiintoista tutkia mihin tämä homma menee.
Metal FX, DLSS & FSR have proved that image upscaling is the future of real-time graphics. Can this same method be used to build a real-time engine that can be comparable to cycles, arnold, octane, etc
Not really, it doesn't solve real time lighting, it's more like baking. So you will have to render your scene normally into a few shots and baked it into 3DGS. like he did.
I am absolutely fascinated by the potential of this stuff. I had a couple questions - if anyone has ideas or answers that'd be awesome. 1) Does Gaussian splatting work only on freeze frame moments, or would it be possible to record a real-world event (say a ballet dancer) from multiple fixed angles, then playing it back in 3D? 2) Would it be possible to integrate this with AR or VR and able to walk around pre-recorded events?
1) yes but it's costly, you will have to capture every frame with enough multiple views, each frame would cost the same of a single 3dgs, there is probably way to compress, but you would have to invent it, or hired a programmer. 2)yes it's been done, the problem is the raw file size, see if using less splat can help retain a good enough quality that fit the memory.
I recomment to follow Infinite-realities on Twitter (X). They have done experiments on animated sequences on Gaussian Splatting. And they have some special custom version on that SIBR viewer which I would be very interested to get my hands on. Check for example this: x.com/8infinite8/status/1699463085604397522?s=61
This try is great, I will follow your future tests, I quitted the computer graphics world, but still throwing an eye on new techniques. just one point : Nerf and 3D Gaussian Splatting are not exactly the same techniques (when saw D:\NeRF\gaussian-splatting)
Interesting for potential future prospects in generating real time 3d under specific parameters, but currently you would get better results converting the file to a real time 3d engine like Unreal.
Hey Olli! Thanks for the super cool videos you make, they are always a treat! Would you have an idea what the lowest amount of VRAM and CPU power are, necessary to view a Gaussian splatting scene? Or if there is still room for optimization on that front? I have a slight fascination for "beautiful graphics through optimization and new technology on old hardware", and I would think it would be super cool if something like this could be run on (very) low end hardware, like an older generation gaming console.
Yeah! I haven't tried it yet, but it seems that Gaussian splatting can be viewed even on older and less powerful devices. Creating Gaussian Splatting model it requires a RTX level card. But just for watching a pre-trained model it could work on less powerful PC also. SIBR Viewer for exsample does work on other cards as well. I tried it on my nephew's gaming PC with a basic GTX card. And it worked very well! It would be interesting to find out how low level of machines the viewer app would still work on.
UNless you use compute, it's equivalent to a massive particles only scene. Which mean tons of overdraws. If we can fit the code of a single splat inside the VU2 of the ps2, it's probably possible on ps2 with a cap at 4 millions particles. lol Overdraws, therefore fillrates, it's the main limiter for non compute (which would rasterize directly). Also current scene are probably the equivalent of using RAW 4k texture density everywhere. We can probably find way to optimize and compress scene, by first reducing "resolution" by dropping some particles, then find way to compress and sort the remaining into a nice format. If we can get rid of transparency too, I wonder how bad are splatting without transparent shapes, might be good enough for some use case. If I were to try, I would first sorts particle into a 3d grid, such that we can march the grid, per rays, and only get the relevant splats, then I would try to guess how to mipmap every sets in a grid, which could then load based on distance. Then I would find a way to trim the grid from empty spaces, then find a compression method for chunks of the grid.
@@NeoShameMan By coincidence, I saw an article on Hackernews this morning explaining that using clustering, it is possible to quite easily reduce the file size of a Gaussian Splatting scene by about a factor 10 without much loss of quality. You can reduce even further, but then you start noticing artifacts, although I still think the images look pretty good. The author notes that in the bike demo scene from the original paper, 4 million gaussians were used... I haven't read the original paper yet, nor do I know a lot about real time rendering, but if a gaussian splat equals a single particle, then 4 million particles + reduced file size might not be outright impossible on the PS2... although you would probably have to also optimize the renderer, like you described. I don't think you can post links on youtube, but the article name was "Making Gaussian Splats more smaller". The idea of the author seemed to be to reduce the file sizes, so that gaussian splatting scenes could be used in Unity. Pretty cool!
My assumption about gaussian on ps2 is based on gamehut's video: " Crash & LEGO Star Wars "Impossible" Effects - CODING SECRETS " ytcode: JK1aV_mzH3A and Aras unity's implementation which use billboards, gaussian rendering on a billboard is analogous to a stretch image, so we would simply stretch the billboard (using the matrix) and not do complex gaussian rendering, we would let the alpha saturate by itself@@impactguide But I think the bottleneck would be bandwidth to pass the unique position, in the video it's all procedural, so it can achieve peak more easily, and I have no idea how feasible is the spherical harmonics, but a close approximation should be possible anyway.
Well, this Gaussian splatting generation relies very heavily on CUDA, so I don't really think it would work on AMD cards. At least not yet. But the pre-calculated Gaussian model can at least be viewed on AMD cards as well. At least I would think so.
@@OlliHuttunen78 this was awesome video!!! is there repo of pre-calculated models that can be viewed? what format are these files? if its *.ply we can use any viewer?
Excelente, Es posible exportar la nube de puntos para convertirla en un mesh? Nuestro interés es tener el modelo 3d del terreno y de ahí poder hacer las modificaciones y lecturas necesarias para diseños constructivos. Actualmente lo hacemos con fotogrametria con fotos tomadas con dron, con esta tecnología podria hacerlo con un video.
How do you use the point cloud generated by Gaussian Splatting? Blender doesn't load the textures when importing the ply file. Couldn't find any help online about this issue I have
60 fps looked like it was locked to that, vsync maybe? Pretty great tech, i have seen many getting 500-600 fps on photoreal looking 3dgs models. Truly the beginning of the future of 3d worlds l capturing
Great video. I'm surprised by the lack of ports of these tools to Mac since the unified memory architecture means one could give the graphics 50-60GB of free memory. I still have hopes for projects built on Taichi as hopefully their backend could be ported from CUDA to Vulkan.
Wow FANTASTIC!!!. This is definitely what i am waiting for. I do architectural visualization and some interactive walkthru using UE5. My question is, is there anyway to view this in VR?
Yes. There is recently developed plugin for Unrel in UE markeplace. It is not free as the Unity plug in the github but with that you can probably make it work in Unreal with Unreals VR templates.
Yes! 6Gb can be enough up to a certain iteration point. And in the calculation, you only need to reach 7000 iterations to get a good enough Gaussian Splatting model. And even if the calculation crashes before 7000, there are tricks to succeed in the calculation, for example by reducing the resolution of the training images.
@@OlliHuttunen78 i wonder if there is relationship to the iterations and vram. for llm 32b fits around 28gb models and 4b models can be run with at least 4gb. not sure if its related or mere coincidences
@@OlliHuttunen78thank you! I definitely have to try it, tried nerf with nvidia ngp in the past and I was already amazed by the results, this looks even more incredible! Thank you for your kind answer :)
The better way wouldn't be going to 2D image space but compute the gaussians directly from the scene. It would be faster and more accurate. You convert the geometry directly to gaussians by sampling then do the training to reduce the number.
I should be possible to produce point cloud without intermediary step of Colmap, the blender already has all that information and it is redundant to restore it with noisy process of rendering images and then back to 3d.
Could the issue of flat backgrounds causing artefacts be remedied by marking parts of the images so that the software knows what it's looking at? For example mark the parts showing the sky so that it knows that it's supposed to be far away instead of confusing it with particles that are closer to the camera. Another thing I thought about was that couldn't you simply convert the 3D model into a point cloud? Of course it would lack the lighting and reflections present in the rendered images, and then new solutions would have to be deviced so that lighting and reflections could be introduced in the render afterwards. I know that these things would have to be coded first and no ready made software exists for them yet, but I'm just spitballing solutions ~
Yes having a ground truth point cloud should solve locations of splatts for the images to be fit in training that's what is missed in real world training sets. Perhaps even they could be improved by using lidar or stereo imagery
The lighting isn't a problem, these are baked data index using Spherical harmonics, the light isn't calculated and wouldn't cost anything. 3D gaussian IS a form of point cloud. The reason we can't convert a 3d mesh, is because it's not encoding surfaces per say, but ray direction BY a position and baking of color into the SH format, the gaussian part can be seen as a form of spatial compression by elongating an ellipse to any point of space sharing similar data, else you would need more points.
Do you think this would work if rendered in 360? Currently working on a project where I'm rendering out a bunch of very realistic 360 images for a restaurant so cooks and personnel can train how to use the space before it's built. The idea struck when watching this video that it would be cool to show the client the entire space in real time.
I'm trying to do this, but for some reason when I'm trying to create the splats, COLMAP is having trouble matching the images. I just did a pan around and object at 30fps, so the pictures should be good enough, but for some reason it's just not working Did you ever run into this problem at all?
I have the exact same 3070 GTX and i wonder why you don't just use Luma AI services, probably to benefits to the video and viewers but there is any other reasons we should know? Like maybe it gives better outputs than Luma AI? Thanks for your great videos !
This video was made before Luma AI released interactive scenes and offered the Gaussian Splatting training opportunity in their services. Now I use it a lot and it makes great results.
@@OlliHuttunen78 Its blurry, it´s not dynamic, it has a very big memory footprint, its not even clear to me that its faster than something similar done with realtime RT in Unreal.
i guess some problems can arise also with collision detection, unless the scene is not cleared someway, I suppose lot of random points can be floating around
@@BardCanning yes, you are right, yes maybe it can be manually specified, I was thinking more of some automatic way to derive it from the data, but I suppose it is possible with some automatic cleaning
I need to check out but somehow I remember that these PLY files can be quite heavy and large on their file size. This ArcViz room was something like 800Mb. But they can be compressed. Aras Pranckevičius who has created plug-in for Unity have also made a great options where you can compress these Gaussian models with very good results. They can be made much smaller trough that plug-in.
@@OlliHuttunen78 Thanks for the info, very much appreciated. I'm a dev and excited about optimizing ply's for games. (along with automating the creation of low-poly unrendered collision models). Aras_p's optimization work is excellent, and I believe it's what drives his buttons in the unitypackage for point density / size. Olli, may I ask please, would it be possible for me to do some experiments on your PLY for the rendered room? I promise to not share it or, whatever your wishes are. It would be a short-cut for me to start editing the PLY via scripts. Your ply is more applicable than any outdoor stuff available online. Thanks for considering this.
This is going to be a game changer! Clients always want to change camera movements like it doesn't take hours or even days to do so
Thats a bluddy briliant use case thfor GS that i never thaught possible.
The answer was on the surface all this time! This guy idea is genius, and thank you that you're point it out and show how to do!!
Very cool!!! Would love to try processing the room scene on my 3090 to se if it gets do 30k steps.
I'm so glad you did this Proof of Concept, it looks absolutely amazing... and thanks for the shout out 😊
I hope this will be integrated in every 3D program at some point. There is so much room for optimization in this process. For example the point cloud generation could be done automatically inside the engine. Also, there are already early tests of dynamically lighten and animating Gaussian Splats. We've got an interesting tech right here that could revolutionize the way we render 3D scenes all together 😊
Keep up the fantastic work... 😊👍
Btw... I wonder how fictional or cartoony content would turn out 🤯
This 3DGS actually could be the "front end" to so many tools that have limited interfaces because the finished visual is a great approximation of reality. Every geographic information system, facility management system, BIM\Fabrication package & surveying application could use this technology. The limiting factor for so long has been differing file formats and interfaces. This could be the 3d printing press that makes digital data almost agnostic. 3D has come a long way since the 1980's but is still the same method and tools, but faster. If you use your room as an example you could have all the services, construction materials, fabrication plans, building approvals, energy calculations etc all use the same 3DGS format to display information on any pixel generating viewer. Exciting times.
Natural next step. We'll done! I'm already attempting to try to fit the splats using genetic algorithms. Fingers crossed.
Olli, this is a fascinating technique and the results in the real-time animation look amazingly good. Especially when you consider that it was the first attempt. Congratulations!
This is a game changer for the future of VFX industry
This is pretty insane. I think in theory it should be possible to render directly into 3d gaussian splats, without ever having to do any point clouds or training process.
Not sure about training, but the point cloud could definitely be generated directly from the geometry. Remember gaussian splats are an extension of a point cloud, there is no way to avoid a point cloud.
Sierra is right. The points are at the basis of splats. Splats even keep their position and color while fitted.
But if you use the geometry as a point cloud, does it have the render quality on it? F. E. the reflections and refractions of the water bubbles?
@@AMessfulI think you’d need to convert the mesh into a textured form first and derive the point cloud from that (using edge detection or similar). This cloud could probably be pruned a bit (more pruning means lower quality but faster). Then you could path trace from each point to get lighting/reflections etc. to set the spherical harmonics coefficients of the colour components. Just guessing here (I’m not a 3D graphics researcher) and I don’t know if that’s any more practical than just using carefully chosen probe cameras and doing the entire original pipeline. One thing that would probably be better though would be fewer floaters and more accurate edges (because we know exactly where in 3D each point is by transforming the texture coordinates by the geometry’s transform).
I thought that this videos was about that
Great job, I think tihis is the future of 3d realtime graphics, and sure a new evolution of 3D gaussian will incorporate animation, with animated elements inside the scene
I was thinking about this as soon as you uploaded video about GS. Very interesting!! Great videos!
I'm amazed how casually you record video and you get theese cool ressults. THis test was also super super cool.
Fantastic. Makes me wonder if you could go to a film like the Matrix and use a bullet time scene as your image base and render that to splats
Well, this is something interesting and very useful
This is amazing…the unlocks so many creative possibilities. thanks for sharing!
this is the future of 3d
Olli Fantastic video and great show of the usecase! Even with NeRFs, a friend of ours managed to get a 3D scene created from a Spiderman Game and it was awesome! We were thinking of testing that out with Gaussians and see how it would turn out. Clearly it is going to look awesome!!
Yeah y’all definitely should glaze our life out with that. Yall the reason why I know about nerfs.
Ay, Bad Decisions! I just got finished watching you guys' video on this tech! It was really great, too!
Scanning a 3D scene from a game is something I've wanted to try since I first heard about NeRFs, but I haven't known the best way to go about doing it. Has your friend made a video or documented how he did it?
Hey mate
You can also check my video abot that topic here: ua-cam.com/video/GnJOFbEwXrw/v-deo.htmlsi=VBTzPpHA3FB8VSox
Do it with the first Matrix movie gun scene :)
Wow! This is amazing! One of the most impressive things I've seen all year. Thanks for sharing this experiment.
I was thinking about this ... thanks for posting!!! I know what I'm doing this weekend!? I'm in 3dsmax and corona.
Fantastic experiment! It'll be fantastic once we can go direct from scan or DCC into real time gaussian splats. I wonder how we could do dynamic lighting within Gaussian Splats though. Right now, it would probably be faster to bake lighting information into the textures of your 3D so you can get Cycles style lighting in real time in Eevee. Looking forward to seeing how this tech progresses!
OK, so this might be a game-changer for the interior design. My wife renders her Revit interiors using Corona. She positions the camera, sets lighting, and makes a rendered shot. Corona renders only on CPU so this takes up to one and a half hours for one shot. There are at LEAST 20 shots per project, which means 30-40 render iterations. If she could instead capture the whole project and just fly with the camera and make screenshots of such perfect quality, this would drastically lower the time and electricity that it takes to finish a project visualization. I would love to see how your project turns out. I can see big commercial (or open-source) potential in it! Would be glad to help if this goes open-source! Would gladly consider buying it for my wife if it is commercial!
Instantly though about this on the first video I saw, especially as here you've took the "classic" route, but what if point could be trained separately then put together with some kind of 3D tools (a bit like the simple color to image IA we had few time ago)?
I really think that the capturing still image to cloud point step are not the big thing (as it's the same as classic photogrammetry obviously) and that 3DGS could lead to even more crazy result with another kind of source of data
But first the obvious next step would be to generate automatically all the needed point of view and go straight from 3D rendering with maxed ray casting stuff to cloud points as the 3D engine would be able to identify points by itself. I wonder how much better points would help this tech
Looking forward to seeing how this progresses!
Amazing, thx for your experimentations !
I wonder if the old Iradiance map could be converted to Gaussian splats or same technique could be used to generate a perfect gs scene.
Maybe in the future 3D over-the-air broadcast will exist or 3D live streaming surveillance.
I now have infinite ideas thanks to this video.
Games were my first thought too. Very interesting!
Fascinating idea and great video! Wonderful accent 🤗 The effort put into clear speech is much appreciated! A real problem for most "native speaking" channels 😉
This is really cool - I'm sure someone will move to generating these splats directly from blender and avoid the loss of detail from generating frames then training the splat. The renderer knows the geometry, camera pos, etc. and could provide much more useful detail when creating each surface instead of inferring. Interested to dive deeper in to this.
I would love to see someone use the 3d model to inform the machine learning model about generated point cloud in order to help remove noise/ghosts, and to make the models more accurate. or perhaps create the pointcloud directly from an engine
Amazing! Thank you for sharing
awesome content
impressive
This is amazing! 🤯 Thanks for sharing....will follow with interest!
I'm loving these.
The most obvious gaming usecase would be realtime cinematics.
Cinematics are often one of two things:
- an ingame cinematic with obviously game-level graphics/lighitng/models, or
- a prerendered cinematic with much higher quality assets and methods that can't be done realtime.
If you're using a mix of gaussian splatting for the environment and then spending all your traditional rendering power on your moving parts/characters, then you could have cinematics nearly as pretty as pre-rendered cinematics. And that's just until we get animated splats.
This way you can have faultless splats too, because the camera would only ever use the path the splat input references took.
Yay, finally, someone tested it! 🙏 Now replace the Blender Render(!) step with a generative AI process, like Runway ML or ControlNet, and use just a 3D OpenGL output (as a "3D Skeleton") from Blender where colors correspond with different object prompts for the generative AI. Or any similar process you can make. Consistency is key though.
Amazing idea and implementation, thanks!
kinda imagine there must be a more direct way to convert from 3d model to the "point cloud" skipping the render , but it indeed is very promising technique
Basically future games could render a high quality interior simular to traditional texture/light baking such as in Unreal Engine, but then use it as Gaussian Splatting while at the same time having a defined playground that can't be exited because Gaussian Splatting will then blur the area. This would offer a overall higher quality photorealism experience then Unreal Engines light bakes for ex. . But the question is if that could be merged with interactions inside that world that it looks right. As the light on interactable props have to be rendered somehow. Also over the years the actual light baking that looks photorealistic haven't been used in games, mostly by archviz companys only. Interesting what could this offer then. I guess film production or product visualisations benefit more from that.
Very interesting stuff. Keep up the good work!
This idea for using #GaussianSplatting is so obvious. Brilliant! #b3d
Siiiiiiick. I love the future
I was thinking this same thing, where some intense renders could be viewed in real time.
THIS VIDEO WAS MADE FOR ME!!!!
Brilliant stuff! Having well above real time rendering of a complex "photorealistic" 3D landscape is nothing less than sorcery!
I also wonder what the uncapped FPS would have been.
Hi Olli, love your work - really appreciate your experiments.
I have a somewhat similar question that you address in this video but with pre-existing point cloud data.
Do you think it would be possible to create a gaussian splatting from point cloud data like .las, instead of images.
I’m not sure. Training needs also the source images to generate Gaussian Splatting model. The point cloud itself is not enought. This source code uses .ply point cloud files. But I think that any point cloud format can be convert to .ply format.
@@OlliHuttunen78 Thank you for the reply - appreciate it :)
Interesting topic! But it would have been even better if you had added some computer specifications. For example what GPU and how much VRAM does it have?
GPU: Nvidia RTX 3070 8Gb Vram
PC: Asus ROG Ryzen 7 64Gb ram
This is getting Nuts!
Very interesting
I indeed believe the point cloud could be directly produced from the 3d geometry tesselated to a high poly count and the texture baked to vertces. It should be faster than rendering all the images and their wouldn't be any forgotten or occluded portions of the model.
Thank you for your good videos
The part that’s missing is still the path tracing. Though maybe there’s a way to bake that in as well?
Failing that, I wonder if there’s a way to write an algorithm to calculate the optimal camera path through a scene to maximize coverage while minimizing redundancy.
@@mortenjorck Yeah I think you're right, but isn't like 99% of cycles render time effectively just calculating lighting? I can't see why you wouldn't get equally good results just by using Eevee with baked lighting.
Great test, this is the kind of content I'm excited about right now. I'm holding off trying it myself though until I can get a 4090 graphics card.
Tätä tekniikkaa voisi käyttää myös 3D-pelien toteutukseen. Saisi ihan upeita pelejä kehitettyä, vaikka tässä onkin paljon rajoitteita verrattuna polygoniperustaiseen toteutukseen. Esimerkiksi joku lentosimulaattori voisi toimia ihan hyvin tai autopeli. Saisi tosi realistisen näköisiä pelimaailmoja tehtyä, kun voi ensinnä vaikka Unreal Enginellä renderöidä tosi yksityiskohtaisen ja upeasti valaistun maailman, jonka muuntaa tuollaiseen pistemalliin
With computer-generated geometry like in this example, I wonder if it would be better to ‘manually’ place gaussians according to real scene geometry, rather than using photogrammetry to place them.
Awesome content!!
amazing !
thanks for doing this video.. I always assumed it would work in theory but was lazy and never created a scene to test it :). great stuff
what about the refractions in the water? do those still hold up and change on view change?
I think right now within the scope of this project, 360 videos could be looked at for training data. Much more information and should fill in the gaps nicely. In the future it should be possible to extract texture information from the 3d file and use it along with the point cloud information to skip the pre-rendering entirely.
Very interseting worklow. And great idea to try this way. Btw. Unreal Engine can compute lightmaps for every surface, calculate how much light will be on every surface by using path tracing and then you can basically change location of camery and get instant raytraced result. It of course don't work when they are moving objects or light in the scene. But for static scene like this it is awesome. Would be great if Blender could to the same.
wow what a great idea
Hey there, you have very effective way of doing it i would love to have the ready made scripts for this
very cool!
Very good, I must try it python stuff on my 4090
like ... it's cool ... but what is it for....
Are we going to see movies shot in some future tech 3DGS cameras maybe?
For maybe stronger ai's to reinterpolate actual hybrid 3DGS and vector format or "baking"?
The experimentation phase is always amazing. But at the moment its faster to bring the modeling to Unreal Engine if we need hi quality in realtime.
Please make a tutorial how to do GS model! And about installing software and what hardware you need.
It would be absolutely incredible if this could be incorporated with VR
awesome
Fascinating or as we say: todella mielenkiintoista. En ole GausSplattingiä tutkinu aiemmin ja mietin että sellainen voisi olla kova jos esim tuosta huoneesta saisi exportattua ajettavan tiedoston ja asiakas voisi avata sen omalla koneellaan ja pyöritellä kameraa ympäriinsä. Kuinka isoja tiedostoja noi sitten on mitä python laskee?
Tää on erittäin uutta tekniikkaa. Lähdekoodi julkaistiin tuossa syyskuun alussa 2023. Katseluohjelmia on hyvin vähän vielä. 3DGS malleista voi tulla aika isojakin: 800Mb - 1.3gb mutta näihin on jo kehitelty pakkausmenetelmää jolla datasetin koko saadaan merkittävästi pienemmäksi. Mielenkiintoista tutkia mihin tämä homma menee.
Metal FX, DLSS & FSR have proved that image upscaling is the future of real-time graphics. Can this same method be used to build a real-time engine that can be comparable to cycles, arnold, octane, etc
Not really, it doesn't solve real time lighting, it's more like baking. So you will have to render your scene normally into a few shots and baked it into 3DGS. like he did.
I am absolutely fascinated by the potential of this stuff. I had a couple questions - if anyone has ideas or answers that'd be awesome. 1) Does Gaussian splatting work only on freeze frame moments, or would it be possible to record a real-world event (say a ballet dancer) from multiple fixed angles, then playing it back in 3D? 2) Would it be possible to integrate this with AR or VR and able to walk around pre-recorded events?
1) yes but it's costly, you will have to capture every frame with enough multiple views, each frame would cost the same of a single 3dgs, there is probably way to compress, but you would have to invent it, or hired a programmer.
2)yes it's been done, the problem is the raw file size, see if using less splat can help retain a good enough quality that fit the memory.
I recomment to follow Infinite-realities on Twitter (X). They have done experiments on animated sequences on Gaussian Splatting. And they have some special custom version on that SIBR viewer which I would be very interested to get my hands on. Check for example this: x.com/8infinite8/status/1699463085604397522?s=61
@@OlliHuttunen78 Oh man, thanks for that recommendation. Love your work, super excited for all the insane stuff this new tech is opening up.
This try is great, I will follow your future tests,
I quitted the computer graphics world, but still throwing an eye on new techniques.
just one point : Nerf and 3D Gaussian Splatting are not exactly the same techniques (when saw D:\NeRF\gaussian-splatting)
Interesting for potential future prospects in generating real time 3d under specific parameters, but currently you would get better results converting the file to a real time 3d engine like Unreal.
Hey Olli! Thanks for the super cool videos you make, they are always a treat! Would you have an idea what the lowest amount of VRAM and CPU power are, necessary to view a Gaussian splatting scene? Or if there is still room for optimization on that front? I have a slight fascination for "beautiful graphics through optimization and new technology on old hardware", and I would think it would be super cool if something like this could be run on (very) low end hardware, like an older generation gaming console.
Yeah! I haven't tried it yet, but it seems that Gaussian splatting can be viewed even on older and less powerful devices. Creating Gaussian Splatting model it requires a RTX level card. But just for watching a pre-trained model it could work on less powerful PC also. SIBR Viewer for exsample does work on other cards as well. I tried it on my nephew's gaming PC with a basic GTX card. And it worked very well! It would be interesting to find out how low level of machines the viewer app would still work on.
UNless you use compute, it's equivalent to a massive particles only scene. Which mean tons of overdraws. If we can fit the code of a single splat inside the VU2 of the ps2, it's probably possible on ps2 with a cap at 4 millions particles. lol Overdraws, therefore fillrates, it's the main limiter for non compute (which would rasterize directly).
Also current scene are probably the equivalent of using RAW 4k texture density everywhere. We can probably find way to optimize and compress scene, by first reducing "resolution" by dropping some particles, then find way to compress and sort the remaining into a nice format. If we can get rid of transparency too, I wonder how bad are splatting without transparent shapes, might be good enough for some use case.
If I were to try, I would first sorts particle into a 3d grid, such that we can march the grid, per rays, and only get the relevant splats, then I would try to guess how to mipmap every sets in a grid, which could then load based on distance. Then I would find a way to trim the grid from empty spaces, then find a compression method for chunks of the grid.
@@NeoShameMan By coincidence, I saw an article on Hackernews this morning explaining that using clustering, it is possible to quite easily reduce the file size of a Gaussian Splatting scene by about a factor 10 without much loss of quality. You can reduce even further, but then you start noticing artifacts, although I still think the images look pretty good. The author notes that in the bike demo scene from the original paper, 4 million gaussians were used... I haven't read the original paper yet, nor do I know a lot about real time rendering, but if a gaussian splat equals a single particle, then 4 million particles + reduced file size might not be outright impossible on the PS2... although you would probably have to also optimize the renderer, like you described.
I don't think you can post links on youtube, but the article name was "Making Gaussian Splats more smaller". The idea of the author seemed to be to reduce the file sizes, so that gaussian splatting scenes could be used in Unity. Pretty cool!
My assumption about gaussian on ps2 is based on gamehut's video: " Crash & LEGO Star Wars "Impossible" Effects - CODING SECRETS " ytcode: JK1aV_mzH3A and Aras unity's implementation which use billboards, gaussian rendering on a billboard is analogous to a stretch image, so we would simply stretch the billboard (using the matrix) and not do complex gaussian rendering, we would let the alpha saturate by itself@@impactguide But I think the bottleneck would be bandwidth to pass the unique position, in the video it's all procedural, so it can achieve peak more easily, and I have no idea how feasible is the spherical harmonics, but a close approximation should be possible anyway.
over 9000!
This, this is what i first thought of when i found out about 3d gausian splatting. now i wonder how long until we get this but animated?
Really interesting, I need to test this at some point, do you know if this works with AMD cards as well?
Well, this Gaussian splatting generation relies very heavily on CUDA, so I don't really think it would work on AMD cards. At least not yet. But the pre-calculated Gaussian model can at least be viewed on AMD cards as well. At least I would think so.
@@OlliHuttunen78 Thanks, I know at least on linux using ROCm, you can run pytorch. But not on windows yet as far as I know.
@@OlliHuttunen78 this was awesome video!!! is there repo of pre-calculated models that can be viewed? what format are these files? if its *.ply we can use any viewer?
You are a genius
Excelente,
Es posible exportar la nube de puntos para convertirla en un mesh?
Nuestro interés es tener el modelo 3d del terreno y de ahí poder hacer las modificaciones y lecturas necesarias para diseños constructivos.
Actualmente lo hacemos con fotogrametria con fotos tomadas con dron, con esta tecnología podria hacerlo con un video.
How do you use the point cloud generated by Gaussian Splatting? Blender doesn't load the textures when importing the ply file.
Couldn't find any help online about this issue I have
60 fps looked like it was locked to that, vsync maybe? Pretty great tech, i have seen many getting 500-600 fps on photoreal looking 3dgs models. Truly the beginning of the future of 3d worlds l capturing
Yeah V-synch. Disabling it i get 120-150 fps
Great video. I'm surprised by the lack of ports of these tools to Mac since the unified memory architecture means one could give the graphics 50-60GB of free memory. I still have hopes for projects built on Taichi as hopefully their backend could be ported from CUDA to Vulkan.
Yeah it’s very weird
Wow FANTASTIC!!!. This is definitely what i am waiting for. I do architectural visualization and some interactive walkthru using UE5.
My question is, is there anyway to view this in VR?
At the moment you can import it into unity
@@Thats_Cool_Jack cool. Thanks for the info
Yes. There is recently developed plugin for Unrel in UE markeplace. It is not free as the Unity plug in the github but with that you can probably make it work in Unreal with Unreals VR templates.
@@OlliHuttunen78 ok cool. Thanks for the info
amazing, is it possible to train the data with a graphics card with just 6GB of VRAM?
Yes! 6Gb can be enough up to a certain iteration point. And in the calculation, you only need to reach 7000 iterations to get a good enough Gaussian Splatting model. And even if the calculation crashes before 7000, there are tricks to succeed in the calculation, for example by reducing the resolution of the training images.
@@OlliHuttunen78 i wonder if there is relationship to the iterations and vram. for llm 32b fits around 28gb models and 4b models can be run with at least 4gb. not sure if its related or mere coincidences
@@OlliHuttunen78thank you! I definitely have to try it, tried nerf with nvidia ngp in the past and I was already amazed by the results, this looks even more incredible! Thank you for your kind answer :)
The better way wouldn't be going to 2D image space but compute the gaussians directly from the scene. It would be faster and more accurate. You convert the geometry directly to gaussians by sampling then do the training to reduce the number.
Verry informative video! I wonder with what GPU you have achieve this result in such a short time?
I already had that idea, and thanks to you, now I know I can do it
I should be possible to produce point cloud without intermediary step of Colmap, the blender already has all that information and it is redundant to restore it with noisy process of rendering images and then back to 3d.
Could the issue of flat backgrounds causing artefacts be remedied by marking parts of the images so that the software knows what it's looking at? For example mark the parts showing the sky so that it knows that it's supposed to be far away instead of confusing it with particles that are closer to the camera.
Another thing I thought about was that couldn't you simply convert the 3D model into a point cloud? Of course it would lack the lighting and reflections present in the rendered images, and then new solutions would have to be deviced so that lighting and reflections could be introduced in the render afterwards.
I know that these things would have to be coded first and no ready made software exists for them yet, but I'm just spitballing solutions ~
Yes having a ground truth point cloud should solve locations of splatts for the images to be fit in training that's what is missed in real world training sets. Perhaps even they could be improved by using lidar or stereo imagery
The lighting isn't a problem, these are baked data index using Spherical harmonics, the light isn't calculated and wouldn't cost anything. 3D gaussian IS a form of point cloud. The reason we can't convert a 3d mesh, is because it's not encoding surfaces per say, but ray direction BY a position and baking of color into the SH format, the gaussian part can be seen as a form of spatial compression by elongating an ellipse to any point of space sharing similar data, else you would need more points.
But if I took so much time isn't better to bake textures in blender and than having a higher quality model render animations or use it in 3D in evvee?
enjoyed your video : )
Wonder if that viewer can run using a 3dconnexion space mouse!
Do you think this would work if rendered in 360? Currently working on a project where I'm rendering out a bunch of very realistic 360 images for a restaurant so cooks and personnel can train how to use the space before it's built. The idea struck when watching this video that it would be cool to show the client the entire space in real time.
I'm trying to do this, but for some reason when I'm trying to create the splats, COLMAP is having trouble matching the images. I just did a pan around and object at 30fps, so the pictures should be good enough, but for some reason it's just not working
Did you ever run into this problem at all?
What is your computer specs? on unity forums some are skeptical that's a replacement for nerf or any good.
Graphics card I have Nvidia RTX 3070 with 8gb Vram memory.
Thanks! @@OlliHuttunen78
When we can import this to blender ?
i wonder if AI could use the point cloud so you can skip the training time.
I have the exact same 3070 GTX and i wonder why you don't just use Luma AI services, probably to benefits to the video and viewers but there is any other reasons we should know? Like maybe it gives better outputs than Luma AI? Thanks for your great videos !
This video was made before Luma AI released interactive scenes and offered the Gaussian Splatting training opportunity in their services. Now I use it a lot and it makes great results.
@@OlliHuttunen78 Thanks a lot for your answer! Watching your videos from France and love it :)
Is there any reason why this wouldn't be used to make games run with prerendered raytraced scenes at a high frame rate?
Absolutely! I can't think of any reason why it couldn't.
@@OlliHuttunen78 Its blurry, it´s not dynamic, it has a very big memory footprint, its not even clear to me that its faster than something similar done with realtime RT in Unreal.
i guess some problems can arise also with collision detection, unless the scene is not cleared someway, I suppose lot of random points can be floating around
Isn't collision usually a separate invisible polygon layer?@@3DProgramming
@@BardCanning yes, you are right, yes maybe it can be manually specified, I was thinking more of some automatic way to derive it from the data, but I suppose it is possible with some automatic cleaning
question... how large were your PLY files? I'm interested in getting into optimizing ply's for games. Thanks for doing this experiment.
I need to check out but somehow I remember that these PLY files can be quite heavy and large on their file size. This ArcViz room was something like 800Mb. But they can be compressed. Aras Pranckevičius who has created plug-in for Unity have also made a great options where you can compress these Gaussian models with very good results. They can be made much smaller trough that plug-in.
@@OlliHuttunen78 Thanks for the info, very much appreciated. I'm a dev and excited about optimizing ply's for games. (along with automating the creation of low-poly unrendered collision models). Aras_p's optimization work is excellent, and I believe it's what drives his buttons in the unitypackage for point density / size. Olli, may I ask please, would it be possible for me to do some experiments on your PLY for the rendered room? I promise to not share it or, whatever your wishes are. It would be a short-cut for me to start editing the PLY via scripts. Your ply is more applicable than any outdoor stuff available online. Thanks for considering this.
@@alpaykasal2902 Sure! You can download this ArcVizRoom Gaussian Splatting PLY -file from this WeTransfer link: we.tl/t-F5fbgAMNY5
@@OlliHuttunen78 Amazing!!! i just grabbed it... i'll keep ni touch with any progress from my side. Thank you!