@@thenerfguru my personal models are pretty spotty due to still having an 8gb gpu but I'm planning to post a vid soon with it also in the premade models
That's great! did you run it in Quest? And can I ask which plug-in did you use? you used Oculus or OpenXR for this? because Oculus doesn't allow Graphics API Direct3d12 and the 3dgs project doesn't run in Direct3d11, so I tried OpenXR but got a bunch of errors.
@@smtabatabaie i used openxr in unity, their default vr toolkit setup and just made sure to follow a tutorial for getting a vr environment working. Had to sort through a few things but it was working pretty easily, just very laggy on my setup
Hey, Johnathan. Just wanted to send my thanks. I’m doing research on integration of Radiance fields in BIM/CIM softwares. Needless to say, your videos have helped me quite a bit. (Btw, screw unity)
Ha! Literally, the Unity announcement came at the WRONG time. Oh well. Would live to hear more about how you feel this fits in the BIM/CIM stack. I talked about NeRFs and Geospatial at GeoWeek last year.
Fantastic tutorial! But when I try to import the Gaussian splat in the tools menu, there is an error that I can't find anywhere on the internet: "the output folder must be within the project, it was "C:/User...". No matter what I do, this error keeps appearing. I'm using Unity version 2022.3.21f1.
Thank you! I felt like I botched this one when I messed up the post processing volume.
Рік тому+6
Thanks Jonathan for another great video, I am more into UE stuff. I am looking forward to see the upcoming tutorials about integrating gaussian splatting into UE5! 🚀
nice! btw. usually Post Processing layer is added into camera gameobject (but that scene didn't seem to have any camera yet, so it created camera on that gaussplat. gameobject itself) so the easy way would be adding Camera gameobject into hierarcy first and assign post process layer there. the post process volume can be added to any gameobject when that global checkbox is enabled. And one more note, this is using Standard Rendering Pipeline. (in case someone creates project with Universal Render Pipeline or High Def. Render Pipeline, this project might not work and post processing is built-in there)
Thank you! Aras Pranckevičius, the plugin creator, also reached out telling me essentially the same thing. This just shows that I am not all that familiar with Unity.
I have a question about these gaussian splatting scenes. Can you do modular environments with them? By that I mean can you isolate a part of the scene like a door or a car and reuse it or recombine it in a different way in another scene just like we do with photgrammetry scenes if you get what I mean? Or is the result totaly static with no creative freedom?
Hey, Jonathan. I have an important question. I was training images for a few hours, however because I was doing another render at the same it crashed. Is it possible to start training from where it last trained or does it have to train from the very beginning?
Hey Nerf Guru, thank you for the detailed tutorials that you've provided. Do you know if there is a method / tutorial demonstrating the creation of gaussian splatting from point cloud data like .las?
I do not. In fact, the main reason for a point cloud is for an initial set of points in space to start optimizing the gaussians. The whole process can be achieved without a point cloud too. What you really need is a sparse set of images with their known camera poses.
thanks so much for this vid! I'm looking to go down the 3DGS rabbit hole (have done a ton of photogrammetry and some nerfs) and have a test scan of an interior I'm looking to get into VR using 3DGS - I have more experience with UE but could go with Unity if it's the better path for this - in your experience is on better than the other with current implementation (UE vs Unity). I know the UE plugin is using niagra with particles but from what I've seen looks good. Also I scanned with Lidar plus bracketed photos so I have a great camera calibration and alignment (in Reality Capture) and would love to keep those intrinsics/extrinsics which have already solved instead of relying on colmap. I've run through some conversions for nerf from RC but have you played with similar workflows for GS? Thanks - awesome work!
I'm trying to use models from Luma, but the tool is saying "PLY vertex size mismatch, expected 248 but file has 27" and I'm not sure what's up. Do you know what might be the issue?
Thanks for the video. One quick question though, why is it that I can see gaussian splat in view port but not in the main camera. In other words, Unity does not render gaussian splat in any of the camera except view port. If you know, please share. Thanks
I want to express my gratitude to the UA-cam channel for its fantastic tutorials. Is there any way to reduce the size of the generated .ply files? Additionally, I'd like to better understand the difference between .ply and .splat files, as, following the tutorial, we mostly end up with .ply files. Thanks in advance for your help!
Sadly for me for some reason, when i try to import any ply (by obviously using input ), I got this error PLY Vertex size mismatch, expected 248 but the fas has 27 I tried with a few PLY files that are available on GitHub and all of them give the same error, mostly showing different numbers!
@@alejandrocapra8654 My problem was solved; I was trying to import the input.PLY, and in fact, you need to open the point_cloud folder and then choose the file you want to open inside the folders (iteration_7000 or iteration_3000, that will load the model correctly without that error
In my case, when I open the tools tab in Unity, instead of asking to import the folder, it asks to import the .ply file and I opened the first input.ply is an error, in fact the qrfile that should be opened, is in the point_cloud\iteration_30000 folder point_cloud.ply file - the manual Next up, create some GaussianSplat assets: open Tools -> Gaussian Splats -> Create GaussianSplatAsset menu within Unity. In the dialog, point Input PLY File to your Gaussian Splat file (note that it has to be a gaussian splat PLY file, not some other PLY file. E.g. in the official paper models, the correct files are under point_cloud/iteration_*/point_cloud.ply).
hi bro. How much data is needed for the model to perform well like bicycle ? and i have a 2 minute video from which i need to choose what fps per second. please give me your advice
good work. thanks. keep it up. I am following your progress. one question for you : do you know if we can merge multiple 3D gaussian scene to make a bigger one ? ( and same question for Nerf)
Maybe someone answered about your camera problems already... but I see your main camera was a child of the splat model... that means your main camera was inheriting all transforms from the splat's axis (which may have been wonky). If you keep the main camera at the top level, and set the position and rotation transforms to 0,0,0 things should behave a lot better.
How does it convert images to spatial data such that it creates an "navigable enclosed world"? Is it also possible with some python plugin by any chance?
The plugin on tools -> gaussian splatting -> create gaussian splatting assets, has not the option input folder, only input ply. And just by importing the ply is not working... Just renders a blank scene.
Im sure its possible but me being unexperienced with programming/coding and an amature in Unity, I wish I could upload one of my Gaussian Splats into VRChat but Vrchat's World SDK is only for Unity version 2019.4.31f1, I tried to get the gaussian splat unity asset to work in it, but the window doesnt show up in the tools tab, and trying to get the vrcsdk working in Unity version used in this video, I cant get the VRChat world splash screen opened to upload it...
hola mencionas que en el min 3:44 que hay que convertir en formato gaussiano, o si no ver tu tutorial para principiante podrias dejar el link, asimismo no se si lo que intentado esta bien intente de mi nube de puntos trabajada de una via llevarla con formato de extencion .ply para cargarla en unity pero me sale error supongo que asi sea en extencion .ply aun debe de transfromarse a gaussiano, espero puedas apoyar por favor , saludos desde peru
please add your pc configs somewhere in your channel or videos! It's really frustrating trying something only to realize it's not possible on my machine
The ply point cloud is just a starting point for where the Gaussian splats render. I foresee in the future, this visualization gets enveloped into the USD umbrella. It’s too early though.
I use Unity every day, and I'll just say you run into stupid rendering problems like you did with the camera flipping all the time. I recently had it so that the camera would only render 1 single column of pixels but smear them horizontally UNLESS the resolution was perfectly 6:9... only in the editor though, if I built the game it worked perfectly.
got this working in VR and scanned a few of the rooms in my house! really impressive how good some of the models can look that way!
I'd love to see some videos of your VR captures!
@@thenerfguru my personal models are pretty spotty due to still having an 8gb gpu but I'm planning to post a vid soon with it also in the premade models
Can you create a tutorial please? :) Definitely excited to try in VR.
That's great! did you run it in Quest? And can I ask which plug-in did you use? you used Oculus or OpenXR for this? because Oculus doesn't allow Graphics API Direct3d12 and the 3dgs project doesn't run in Direct3d11, so I tried OpenXR but got a bunch of errors.
@@smtabatabaie i used openxr in unity, their default vr toolkit setup and just made sure to follow a tutorial for getting a vr environment working. Had to sort through a few things but it was working pretty easily, just very laggy on my setup
literally needed this exact tutorial 5 hours ago. This was uploaded 3 hours ago. Amazing timing, great tutorial
Dang! I wished I had a Delorean so I could go back in time and post sooner!
DUDE I'm so excited to watch this!! Thank yoU!
Yes!!! I just hope people can make cooler things than me. You can now mix meshes with Splats in 1 scene.
This is so awesome!! Thanks for introducing this researches and making this video! It is a great help for me :D
Glad it was helpful! Follow Aras for more amazing work.
Hey, Johnathan. Just wanted to send my thanks.
I’m doing research on integration of Radiance fields in BIM/CIM softwares.
Needless to say, your videos have helped me quite a bit.
(Btw, screw unity)
Ha! Literally, the Unity announcement came at the WRONG time. Oh well. Would live to hear more about how you feel this fits in the BIM/CIM stack. I talked about NeRFs and Geospatial at GeoWeek last year.
Maybe you have come across information on how you can transfer footage from a 360 camera to a point cloud with a 360 camera
Hey im pretty new to this. Is it also possible to do it in a 3D VR project?
Fantastic tutorial! But when I try to import the Gaussian splat in the tools menu, there is an error that I can't find anywhere on the internet: "the output folder must be within the project, it was "C:/User...". No matter what I do, this error keeps appearing. I'm using Unity version 2022.3.21f1.
Great video again!
Thank you! I felt like I botched this one when I messed up the post processing volume.
Thanks Jonathan for another great video, I am more into UE stuff. I am looking forward to see the upcoming tutorials about integrating gaussian splatting into UE5! 🚀
I see there is a $99 plugin on the marketplace now. I am holding out, I am sure someone will hack a free solution.
Thanks for the heads up @@thenerfguru I will also hold out! :) Hopefully not for long...
Great content, keep up the great work!
Thanks!
Thanks a bunch for the workflow. Everyone will love you for a UE5 workflow though :D
There is now a $99 plugin for UE5. I haven't tried it yet.
nice! btw. usually Post Processing layer is added into camera gameobject (but that scene didn't seem to have any camera yet, so it created camera on that gaussplat. gameobject itself)
so the easy way would be adding Camera gameobject into hierarcy first and assign post process layer there. the post process volume can be added to any gameobject when that global checkbox is enabled. And one more note, this is using Standard Rendering Pipeline. (in case someone creates project with Universal Render Pipeline or High Def. Render Pipeline, this project might not work and post processing is built-in there)
Thank you! Aras Pranckevičius, the plugin creator, also reached out telling me essentially the same thing. This just shows that I am not all that familiar with Unity.
Thank you for this Splat Guru
Great Video and good lesson to use Unity.
I have a question about these gaussian splatting scenes. Can you do modular environments with them? By that I mean can you isolate a part of the scene like a door or a car and reuse it or recombine it in a different way in another scene just like we do with photgrammetry scenes if you get what I mean? Or is the result totaly static with no creative freedom?
You could mix and match as long as it's all in the same coordinate space. You could achieve it in this Unity environment.
Hey, Jonathan. I have an important question. I was training images for a few hours, however because I was doing another render at the same it crashed. Is it possible to start training from where it last trained or does it have to train from the very beginning?
Thanks for your tutorial!! Could 3D Gaussian splatting interact with light in the scene, and could it be captured by a reflection probe?
Hey Nerf Guru, thank you for the detailed tutorials that you've provided.
Do you know if there is a method / tutorial demonstrating the creation of gaussian splatting from point cloud data like .las?
I do not. In fact, the main reason for a point cloud is for an initial set of points in space to start optimizing the gaussians. The whole process can be achieved without a point cloud too. What you really need is a sparse set of images with their known camera poses.
@@thenerfguru thanks for clarifying - makes sense. It's a shame bc the company I work for has lots of survey / point cloud data
thanks so much for this vid! I'm looking to go down the 3DGS rabbit hole (have done a ton of photogrammetry and some nerfs) and have a test scan of an interior I'm looking to get into VR using 3DGS - I have more experience with UE but could go with Unity if it's the better path for this - in your experience is on better than the other with current implementation (UE vs Unity). I know the UE plugin is using niagra with particles but from what I've seen looks good. Also I scanned with Lidar plus bracketed photos so I have a great camera calibration and alignment (in Reality Capture) and would love to keep those intrinsics/extrinsics which have already solved instead of relying on colmap. I've run through some conversions for nerf from RC but have you played with similar workflows for GS? Thanks - awesome work!
I'm trying to use models from Luma, but the tool is saying "PLY vertex size mismatch, expected 248 but file has 27" and I'm not sure what's up. Do you know what might be the issue?
Not sure. I don’t think you can actually use the PLY file from Luma. Download the Gaussian Splat from the Unreal Engine section and that should work.
thank you very much
You’re welcome
Thanks for the video. One quick question though, why is it that I can see gaussian splat in view port but not in the main camera. In other words, Unity does not render gaussian splat in any of the camera except view port. If you know, please share. Thanks
I want to express my gratitude to the UA-cam channel for its fantastic tutorials.
Is there any way to reduce the size of the generated .ply files? Additionally, I'd like to better understand the difference between .ply and .splat files, as, following the tutorial, we mostly end up with .ply files. Thanks in advance for your help!
Sadly for me for some reason, when i try to import any ply (by obviously using input ), I got this error
PLY Vertex size mismatch, expected 248 but the fas has 27
I tried with a few PLY files that are available on GitHub and all of them give the same error, mostly showing different numbers!
same error
@@alejandrocapra8654 My problem was solved; I was trying to import the input.PLY, and in fact, you need to open the point_cloud folder and then choose the file you want to open inside the folders (iteration_7000 or iteration_3000, that will load the model correctly without that error
In my case, when I open the tools tab in Unity, instead of asking to import the folder, it asks to import the .ply file and I opened the first input.ply is an error, in fact the qrfile that should be opened, is in the point_cloud\iteration_30000 folder point_cloud.ply file
- the manual
Next up, create some GaussianSplat assets: open Tools -> Gaussian Splats -> Create GaussianSplatAsset menu within Unity. In the dialog, point Input PLY File to your Gaussian Splat file (note that it has to be a gaussian splat PLY file, not some other PLY file. E.g. in the official paper models, the correct files are under point_cloud/iteration_*/point_cloud.ply).
I found a fix, if its in a zip folder but you have the right file type, you have to extract it first.
Thank you for the video, what is your recommendation for the best drone for 3D Gaussian Splatting?
Maybe you have come across information on how you can transfer footage from a 360 camera to a point cloud with a 360 camera
hi bro. How much data is needed for the model to perform well like bicycle ? and i have a 2 minute video from which i need to choose what fps per second. please give me your advice
Great tutorial! I ant to know how can I import gaussian splatts into another unity project that I'm already working on...
we didn't see the assests folder in your git repo. can you help us?
good work. thanks. keep it up. I am following your progress. one question for you : do you know if we can merge multiple 3D gaussian scene to make a bigger one ? ( and same question for Nerf)
Possible, I’m not sure how to do it thought. Block NeRF stitched together multiple NeRFs.
Maybe someone answered about your camera problems already... but I see your main camera was a child of the splat model... that means your main camera was inheriting all transforms from the splat's axis (which may have been wonky). If you keep the main camera at the top level, and set the position and rotation transforms to 0,0,0 things should behave a lot better.
it doesn't seem to work what version of the release are you using?
Do you think using GS models of trees and plants in architectural rendering is possible of you have a whole garden design?
How does it convert images to spatial data such that it creates an "navigable enclosed world"? Is it also possible with some python plugin by any chance?
Do you mean this? ua-cam.com/video/UXtuigy_wYc/v-deo.html
The plugin on tools -> gaussian splatting -> create gaussian splatting assets, has not the option input folder, only input ply.
And just by importing the ply is not working... Just renders a blank scene.
How many pictures does it take to create the GaussianSplattingScene? Also, great videos, thank you!
Im sure its possible but me being unexperienced with programming/coding and an amature in Unity, I wish I could upload one of my Gaussian Splats into VRChat but Vrchat's World SDK is only for Unity version 2019.4.31f1, I tried to get the gaussian splat unity asset to work in it, but the window doesnt show up in the tools tab, and trying to get the vrcsdk working in Unity version used in this video, I cant get the VRChat world splash screen opened to upload it...
hola mencionas que en el min 3:44 que hay que convertir en formato gaussiano, o si no ver tu tutorial para principiante podrias dejar el link, asimismo no se si lo que intentado esta bien intente de mi nube de puntos trabajada de una via llevarla con formato de extencion .ply para cargarla en unity pero me sale error supongo que asi sea en extencion .ply aun debe de transfromarse a gaussiano, espero puedas apoyar por favor , saludos desde peru
Curious if this is possible with unity 2019 and vrchat...
Not sure. I guess you can give it a try?
please add your pc configs somewhere in your channel or videos! It's really frustrating trying something only to realize it's not possible on my machine
Sorry to hear that! I’ll follow your recommendation and do that.
Any thoughts on how to get this working with URP or HDRP?
No idea. I wish I knew more about how Unity works.
Hi, isn't that the .ply model?
Is it possible to convert it to usd/usdz?
The ply point cloud is just a starting point for where the Gaussian splats render. I foresee in the future, this visualization gets enveloped into the USD umbrella. It’s too early though.
Que componentes de PC tienes para estos renders ?
Can you try walking into a scene with virtual reality goggles on?
i heard about this plugin! i can't wait till theres an unreal one too
Hold onto your papers. It exists!
Also, I know how to use UE5 way easier than Unity
I imagine exporting into Unreal 5 would be epic too.
Can this be done in Unreal as well???
There is now a $99 plugin for UE5. I haven't tried it yet.
Is this possible in Blender?
Yes, it looks like Alex Carlier is working on it: x.com/alexcarliera/status/1698769174950985787?s=20
Good tutorial! I need to check the UE5 plugin tho
I’ll try to bust out a video tomorrow night.
I found one plugin on the marketplace, but it's $100, which sounds unreasonably expensive. @@thenerfguru
@@thenerfguru 🕺🕺
Can the scene be built and exported as stand alone software?
I assume so. I believe someone did this as a Gaussian Splatting VR Viewer.
Original author removed assets from GitHub, sadly I cannot reproduce steps mentioned
Oh I found he moved assets inside project folder
Good to know!
excelent video, UE5 please
There is now a $99 plugin for UE5. I haven't tried it yet.
On my way to create VRCHAT world with this level of graphics wish me luck
if you figured it out lmk
Bad timing, lol
But it's exciting to see that tech in engines. Maybe Unreal Engine though, or Godot
It’s now available on UE5 too.
@@thenerfguruThat was quick!!
Can you please link it?
This video just cost you $0.20 per click. Sincerely, the Unity Gestapo
Ha! Sad day... I really hope this all gets resolved in a way that is beneficial for the developers.
If unity, then fr unreal too i m hopeful ❤.. please try a solution for unreal engine.
I saw a plugin released today for UE5, I’ll check it out.
@@thenerfguru waiting 🙌
we should all switch to unreal
There is a $99 plugin for Unreal.
I use Unity every day, and I'll just say you run into stupid rendering problems like you did with the camera flipping all the time. I recently had it so that the camera would only render 1 single column of pixels but smear them horizontally UNLESS the resolution was perfectly 6:9... only in the editor though, if I built the game it worked perfectly.
Ugh! Maddening! I wished I knew Unity better. Seems fickle at times.
Unity is a 4 letter word these days
The timing couldn’t be any worse for this video. Haha
Yeah. I will be blissful if it is available for Unreal Engine
@@OriginRow it is through LumaAI. Not gaussian splattering, but NeRF at least
Shit crap cash grab
Bad timing...
I know 🙄
haha cool to see my uni in here