Відео

Why so?

@@infiniterealities4D its 100% going to be abused to forge incriminating footage or to claim that incriminating footage was made up

@@sascham3274 you mean like people do with photos and videos now.. these are baked captures just like video.

@@infiniterealities4D im dumbf af

4K @ 60fps

Thanks it's all real-time you can view it from any angle you need to.

Thanks. It's 3DGS as a flipbook essentially. We don't support Unreal at the moment as there is not a performant way to play them back quickly enough. We reached out to Epic but there was little interest.

@infiniterealities4D Thank you for your reply. So your implementation is to train each frame in 3DGS and then play it, right? Very cool! I never thought it could be implemented this way! You guys are really great. Yes, currently there are few DCC software or engines that support the preview of 3DGS or 4DGS. The way we are using is to convert Gaussian splashes into Unreal's Niagara particle system using the 3DGS plugin. Maybe we can convert the motion of each frame of 3DGS into particles? Anyway, seeing the exciting effect of your demo is so inspiring, looking forward to more of your technical demos and keeping in touch with you.

@@tonydu5658 Thank you!

@@infiniterealities4D This is amazing. I'm curious about this comment because I've heard Gaussian splats are more CPU friendly than pbr and photogrammetry. Also below you seem to say it can run on a PS5, so how can it not run in Unreal in a performant way? I am shocked Epic is not interested in this, this is the very cutting edge-how soon will these models be riggable for use? Thanks

No lidar is used.

176x and 360 all angles.

@@infiniterealities4D crap. So out of consumers reach. It loooks amazzzzing tho! ❤️

@@Instant_Nerf thanks!

176x

How cool is that! It's broken through to the mainstream. Great positive attitude :D

Lots of cameras and the 3DSG code base. Beeble's a multi-millionaire NFT baker he should have the resources to do this :)

@@infiniterealities4D please redo the same output with minimum number of phone, bringing it simplicity for indie creators.

@@justmovies682 not possible atm sadly, maybe in 5-10 years.

Currently each frame averages about 10mb. With better 4DGS compression code we should be able to reduce this or reuse the splats more over time.

The Playstation 4 could do with a similar type of process with "Dreams". The PS5 easily.

We've done that already. Full silver outfit + guitar playing. New video coming soon :)

The videos were captured using OBS from a real-time playback @60fps and composited side by side in After Effects.

Sorry, I meant for a single 4DGS what did you use to combine the different camera angles taken?

​@@mattm381 We're working with the original code base from Inria: repo-sam.inria.fr/fungraph/3d-gaussian-splatting/ to train the data. Plus we use our own code base for 4DGS playback.

Not sure where the confusion comes from. We're not stating that's possible with our data.. yet. We're just able to collect the data required for future processing. If you need confidence see Meta's Codec Avatar 3 relightable gaussians. We just don't have the resources that Meta does.

Hang on? I'm not missing the data or anything, it looks amazing, But I'm assuming that what your showing here is 1 gaussian splat captured in a well lit ambient studio, You've then taken that capture data and put it in different hdris environments stating it's a composite (i.e done in post using compositing software such as nuke or resolve fusion) So from a compositing standpoint, if you have a world normal pass of a render then you can relight the gaussian splatter to match another environment as you are shown there. But you also showed that the gaussian splat casts self shadows. Now if that was done in composite alone, I think that's cool and was wondering how?. But if you said this was done in the render engine and not post/composite then I understand ( still cool)

@@darmelli954 We're not showing the gs showing self cast shadows. There's no relighting or lights being used in the render engine, no shadowing. That's trained into the data from the baked lighting from the input images. "we captured a static pose of @henrypearce4D under varying "baked" illumination patterns." I hope that helps. But we can capture the normals, specular, roughness, albedo (+mesh surface) information now to train future ML databases for relighting shadow casting etc. We just cannot demo that atm.

@@infiniterealities4D Ah right, Sorry for the misunderstanding, And that's way cooler then I thought Thanks for the info

@@infiniterealities4D I wonder how you get the normal, do you capture it directly, or do you estimate them algorithmically using some SOTA method, many thanks

176 cameras.

@@infiniterealities4D wow. all synced with timecode?

@@spiderbite7849 yes with audio (multi mic) sync

​@@infiniterealities4D What kind of camera? and resolution.. Don't it need an RGB-D camera?

@@방현편 technically when you know the instrinsic values of the camera you can infer the depth from the disparity of the images- but there are newer neural network driven approaches too

Small enough and performant enough to play back on the Quest 3.

Thanks!

It varies. Our body system uses 176x cameras. This was captured on the head capture system with 64x cameras.

It's a Gaussian Splat... there is a digital model underneath it as well though. So tehcnically yes.

​@@infiniterealities4D do you think an average computer will become toast if there is an attempt to open this in blender?

@@aromncoronado2005 It depends what you mean by average. Modern gaming hardware is a given for any content that is next generation.

Thank you!

Thanks. Some custom processes and playback tools, most of the training code is original. Nearly all of the published 4DGS code online does not work (as advertised) especially with multi-camera data. So we've had to improvise for the time being.

Just need Apple to sell in the UK :D

This is bullet time 2.0

With our custom multi-camera and lighting rig.

Thanks Joe!

Thank yoU!

We have compression working for playback on VR devices. It's not our interest to pursue mobile or web camera capture? I'm sure academic (funded) researchers will be focused on that

You should drop us a DM on Twitter/X. We could potentially share some test data with you to experiment on.

Thanks. Imagine it as a stop-motion play back. Each frame is unique and played in sequence, very quickly.

@@infiniterealities4D Thanks. I see, does it take a long time to calculate? And how big is the size of the final result?

@@stickmanunivers5si9d We farm the training out, it can take up to 15m per frame depending on your settings. Sizes vary but anything from 25mb-100mb per frame (without compression) We now have compression methods so can reduce those to half

@@infiniterealities4D Thank you for your response, it was very insightful.

Thanks & yes!