Відео

That was really notable to me as well - absolutely; as they are very fine scale, and I was not sure how they would impact the reconstruction.

Many thanks - and I definitely agree; I always seem to end up with a very glacial motion, and adopting faster camera motion in the export is a priority going forward for sure.

This example was generated with the RTX4060 Ti 16GB, though for most of the examples, I was using an RTX3060 12GB (and some with RTX 3070 Ti 8GB Mobile; and one where I tested with RTX 2060 6GB and an i7-4790). In terms of CPU, almost all of them, excepting the laptop (12700h), were with 11700 desktop (non-K, all at default motherboard settings, Intel stock cooler etc.) During some steps of the processing, CPU is heavily utilized; the actual GS reconstruction phase is more dependent on the graphics card. I would say the latter is the main determinant of performance, generally speaking.

@ thank you! I’m hoping when they release the next Nvidia graphics cards 4090 series will become cheaper.

@@historyphotogrammetry2218 One other question about the processors. Does it matter if it is Intel vs AMD?

@@Dvm2003 I do not think it would matter. From my general impression, the modern Ryzen series would have all of the possible instruction sets that might be required. Given that it works on the 12700h, I don't think it requires AVX-512 for instance (though the most recent Zen5 do have that from memory); and prior testing had success on an i7-4790 (so a much earlier AVX version). In terms of performance, I suspect it would approximately follow the kind of performance benchmarked for things like Agisoft Metashape and Colmap, as the camera pose reconstruction phase of the process seems to be the most demanding in terms of CPU. It probably does tend to scale with single-thread more than multi-thread (impressionistically).

@ thank you so much for your help!

It is an interesting camera, and seems less frequently encountered than the other Finepix SuperCCD models. I am fairly sure that here, I was using the Chrome preset or similar color mode, which really does tend to dial up saturation (it is an early implementation of Fujifilm's film modes, which would become very popular in their later ILC range of cameras).

It is found under the F-Chrome setting in the F-Mode menu (under FinePix Color) - and the manual specifically notes saturation. It is a little overdone, but I do rather like it for some scenes.

Many thanks; I think here, it was probably a combination of features. The input images from the Fujifilm were very sharp, and the way the EXR CCD camera maps dynamic range into a JPEG seems to be very well suited to giving an even, balanced exposure (despite the comparative age and, in modern terms, very modest area of the sensor). The downsample setting was also set to allow a higher resolution input, and the step count was doubled from the default. Even still, I was quite surprised at how promising the output was - given it was a consumer compact camera, released well over a dozen years ago.

May I ask what software you are using? Postshot?

This example is Postshot. In other cases, I have used the original 2023 Gaussian Splatting as published with the Gaussian Splatting research paper.

I'm so pleased it was interesting - I've added the direct output from Postshot Beta here - the difference with the TVAI output is fairly subtle to my eye when viewed at 1080p. ua-cam.com/video/TQDf7JlfPuk/v-deo.html

@historyphotogrammetry2218 Thanks! I just asked as want to walk around in such stuff in VR...or is this just for one time render and not live walkable with rtx4090?

@historyphotogrammetry2218 And thanks a lot! :)

No trouble, and it is very much interactive, realtime once the scene has been generated. It seems like 60fps navigating realtime with mouse & keyboard on a 3060 12GB, 1080p. I'll check, but I imagine a 4090 would manage 90fps stereoscopic.

@historyphotogrammetry2218 Wow amazing! Is there a tutorial on how you do all that? I would use Insta360 Pro 2 or Qoocam 3 Ultra for capture :)

Thank you so much - I've been really fascinated with the kind of output, despite definite limits, can be generated with spontaneous, serendipitous capture in daily life.

Around two hours; I can check the log for greater precision. I've observed that a lot of the calculation time on some recent tests, with the 11700 and 3060, seems to be feature detection and camera tracking. That camera pose reconstruction is probably limited by the single-thread performance of the 11700 (non-K, all at defaults). Postshot Beta seems less demanding on VRAM than I had anticipated, even when experimenting with a less aggressive downsample.

Generally, especially at night, capture was at a more standard rate; though the navigation of the reconstructed scene and final export via Topaz VEAI I have experimented with higher frame rate. I think that in low-light, my expectation would be that 120fps would be highly undesirable as each frame is going to be at an exposure around 1/240s (assuming 180 shutter angle), which is very fast given the available light, and would likely mean an extremely high ISO being required for adequate exposure (and thus, substantial issues with noise). I could be mistaken however, but generally, the 120fps examples have been the navigation / export of flying through the reconstruction, not the capture stage (with a few exceptions in daytime, where capturing from slow-motion / high frame rate video). Hopefully that helps a little,

My apologies, I definitely have started drafting a tutorial - I had notes from some time ago, and it should not take that long to prepare a video. I'm delighted you are finding them engaging!

@@historyphotogrammetry2218 i lovely your work. thank!

That is great to hear - I will try to keep track of the training time; I noted it down for the most recent Postshot Beta example (82 mins). There does not seem to be a large ecosystem for benchmark results for this area just yet.

This is so wonderful and encouraging to read - thank you, I really appreciate such generous words!

Many thanks - it was a very interesting scene for sure. This is still the original version; I tend to find that is the one where I am most confident, though I definitely do want to explore the wider ecosystem (which seems to be developing in many promising directions).

As indicated in the description, around 125; approximately even distribution of views in a shallow hemispheric arc.

The settings were default in Postshot; I think it auto-downsampled the images to 1536 pixels wide.

I think so at this stage, though I have not checked recently.

Unfortunately, I think it does at the moment - at least an RTX 2060 (if working in Postshot), and maybe an RTX card with at least 8GB VRAM if using the software from the original publication.

Apologies, I did not keep a record - but it was reasonable, perhaps an hour or so, with defaults (it selected the best 400 images I think).

Thank you so much for the generous words! I am delighted it was of appeal. This was, I think, around 500 or 600 images. They were captured at full resolution, but had to be downsampled quite severely (less than 900p), to manage the VRAM limits.

Very much so - it is an interesting case where the deficits in fidelity are aligned to the conditions captured.

I'm so pleased - this example was acquired as still images (JPEG), in auto, mechanical shutter. I often do find it interesting to see which approach is more useful in which circumstance - usually decimating from video provides the better balance overall. I will post one of the same scene using images decimated out from a video capture.

@@historyphotogrammetry2218 Thanks for this! I am really enjoing your videos. Thanks for the frequent posts. Recently I also been trying to explore all posibilities using my DSLR camera and my 360 cmera.

I studied the input, and I used 341 frames decimated from Pixel 8 video I captured while transiting home (downsampled to I think 720p). I walked around the scene in a kind of hemispheric orbit, with the camera (main lens I think) pointing toward the areas I was seeking to capture.

@@historyphotogrammetry2218 thank you!

Looking at the video, I think I was walking alongside Erskine House, and moved the camera around the table in a kind of orbital motion. It was fairly brisk and incidental in terms of captures (the input imagery is taken from early 2017, and I did not have GS specifically in mind).

I'm so pleased to hear you are finding them interesting - I find the process intriguing, just seeing what works reasonably well, the interaction of camera motion and time acquired etc. With the Pixel 8, as with the Pixel 7 Pro, I just use the default auto everything JPEG from the Camera application (that is, not any specific GCam version or anything of that sort). It is an excellent general purpose camera I think, and I find it ideal for the types of situations where a smartphone is taking the pictures. I do, however, tend to find it has some elements that require care. The first, and perhaps it is just my experience, and in particular, use in often very warm conditions - but when taking a lot of still images in sequence, I have had repeated crashes (with both Pixel 8 and Pixel 7 Pro before it). This would not necessarily be a disaster, but it is frustrating because it tends to lose a lot of the images taken in that session. Often a salvaged version, of lower resolution, is saved as <PXL**numbers.RESTORED> but it does require care. Basically, the cadence between shots has to be lower I find, compared to say, the Xperia 1 (II). I appreciate that I am often asking it to do a lot - with short times between photos, but I do wish it would simply indicate that the queue of images to be processed was saturated and another shot was not possible. Nevertheless, it is fairly easy to avoid by lower cadence of imaging, and pausing while photos are processed. I've tended to pause, then close the camera, and then reopen it a few minutes later.

My other observation about Pixel 8 and Pixel 7 Pro is that the features which I tend to think make it amazing for still image capture in smartphone type situations - namely, an incredible HDR+ fusion of multiple exposures, can make it a bit more susceptible to a distinctive pattern of artefacts when processed. The local contrast enhancement and sharpening seems sometimes to perhaps be associated with floating, dark, linear-type features. Not always, but I have found the more naturalistic JPEGs of a small ILC, or premium compact, or a smartphone with less advanced processing.

Which leads me to a reflection on the still images versus video question. I tend to think it is very, very often the case that video acquisition is preferable - and especially so with the Pixel 8 and Pixel 7 Pro. You are acquiring vastly more samples per unit time, and the kind of HDR+ fusion process that seems to me liable to produce artefacts is not being undertaken. It is also very stable in my experience - I've never had the Pixel 8 or Pixel 7 Pro cameras crash when taking video. With that said, I think it can vary - obviously for low-light, a somewhat pathological case, still images with Night Sight seem much better generally than video capture. It's an interesting question with regard to locked AE. I definitely think that can help with photogrammetry, but I'm not sure with Gaussian Splatting. I haven't noticed an improvement (tested with an older ILC, the Fujifilm X-M1) - and it occurs to me that part of the effect of moving around the scene and having exposure adjust a little seems related to the view dependent aspects of recreating a surface, scene etc. I'm fascinated to hear of your use case - that sounds very impressive! I suspect if you have more time to acquire, you could consider experiments saving out the DNG with the JPEG. It would be quite a bit slower, but you would have the ability to adjust exposure in post, and to apply less heavily tone mapped curve in the RAW to JPEG processing.

I think the main thing I've found useful is being a bit adventurous in terms of what scenes to try to acquire - it sometimes gives a really engaging result in scenarios which intuitively do not seem like they would work at all.

This was using the application that accompanied the original paper (see detail and URL in description). I then ran through Topaz VEAI for upscaling the output.

I have not used Luna AI, but I imagine they would be comparable - it will be the same fundamental approach, and with identical input. I noticed modest differences between Postshot for example, and the classic 2023 software that accompanied the Gaussian Splatting research paper.

It was around 44 JPEGs, resized to 1192 x 898 (approximately) - and just taken as I was walking under this freeway. It is quite faithful to the scene - it did well given how sparse the input was, and how inattentive I was while capturing it. I think generally I've found Olli Huttunen's channel very useful for learning about GS: www.youtube.com/@OlliHuttunen78 Though I am not sure if there is a step by step illustration of the process - but it definitely has good guidance and introduction I've found.

Many thanks - here it is SDR UA-cam from what I think is HLG HEVC. I am investigating how to get a fully HDR process (most of the approaches I could think of are fairly inelegant, involving bracketed exposures run separately).

Another comment drew my attention to a new program, now in Beta, which appears to make the process quite straightforward - it looks to me that (at this stage) it just requires a free account for the Beta test. I've not tried it yet, but it looks very promising (Jawset Postshot). These involved smartphone imagery capture, then processing via the original software from the research publication listed in the description, then some upscaling and editing of the output video - so it is slightly intricate. My area is history, so it is not a very strong background in computers (I would say generally I mostly use Word, Powerpoint, and OCR software in my normal day) - and it is not too onerous to install and process things via command line, manage the dependency installs etc. But potentially the more integrated, GUI based program (Postshot) may now be the best avenue.

That's very kind - I must admit, I generally just try to have a camera with me across the course of the week, and to try to get kind of 'orbital' views of things, or for indoor spaces, a mixture of views across multiple planes of the scene. I generally am capturing things while on my to office or function, appointment etc., so it is mostly when there are few minutes spare or similar. I quite like seeing how different, particularly older, consumer cameras manage particular reconstruction techniques, at least in part because it suggests that older, lower fidelity, inputs are usable (I have had some success there with Video 8 Handycam frames from the early 1990s).

I am glad it was engaging as a demonstration, that's great to read. In terms of the cropping and upscaling, that is applied to the output of my moving the camera around the reconstructed scene - the input images are always the same focal length, processed from one camera (here the Ricoh GRDIII, from 2009), and uniformly downsampled (to diminish pressure on VRAM). So it is the output video where I have upscaled, and sometimes cropped the frame to zoom, not the input images. This scene was 102 JPEGs, downsampled to about 1198 x 898 pixels (from the original capture resolution of 10 megapixels). It took about five minutes, perhaps a little less.

I got it, thank you very much. so if I captured higher resolution images for example 8k video then crop it to small side by side images so each image resolution does not exceed 1.6K and depend on the camera movement direction I can collect much wider dataset with lower effort and less time, right? @@historyphotogrammetry2218

That is no trouble - and I will do my best to offer views on how to approach here. So I am not sure on the merits of cropping versus an overall downscale / supersample - there are instances where I have, equally across all input frames, cropped to a centre window of the image. But I have never, for instance, tried to crop varying windows of a frame changing across an exported video sequence. Colmap assumes a single camera (I think) in the aligning step - I don't know if varying the portion of a larger frame being cropped into would cause problems. I would suggest, just as a start, if you have 8k video, uniformly downsampling/resizing to 1600 x 900 and running that input as an initial test. The amount of decimation (every 2, every 3, or 5, or 6th frame) is, in my experience, somewhat dependent on the nature of the scene - and how quickly you are traversing it (e.g. for walking, every 4th frame perhaps, but if faster, perhaps every 2nd frame). I usually would use Lanczos2, but 8k is a very high resolution (I've never used it) - and the impact of particular resampling filters is quite complex; such a high supersample may produce oversharpening with Lanczos2 - though I tend to think it would actually work really well (as linear features for matching would be well reproduced etc.)

Overall though, I would say, for this technique - video capture is very much more efficient in terms of acquisition/capture time. With standard photogrammetry, it often seems worthwhile to adopt a still frame based capture strategy - whereas with gaussian splatting, I have found video often better (and almost always so if keeping capture time equal - that is, results/fidelity adjusted per minute of acquisition time).

145 JPEGs, downsampled to 1198 x 898 (from original 10 megapixel); it took around 5 minutes to take them.

An excellent question, the first 1m55s are just free movement with the keyboard - I then play back aligned to the reconstructed views (from about 1m56s onward). That section tends to have more abrupt shifts in perspective - but the actual fidelity does not seem dramatically better here (and arguably worse sometimes).

So in short, first third, off trajectory - last two thirds on trajectory.