[ICCV 2021] Pixel-Perfect Structure-from-Motion with Featuremetric Refinement

This is the 11-minute video for our ICCV 2021 paper:
"Pixel-Perfect Structure-from-Motion with Featuremetric Refinement"
Project Page: psarlin.com/pixsfm
Paper: arxiv.org/abs/2108.08291
Code: github.com/cvg/pixel-perfect-sfm
Authors: Philipp Lindenberger*, Paul-Edouard Sarlin*, Viktor Larsson, and Marc Pollefeys
(* equal contributions).
Abstract:
Finding local features that are repeatable across multiple views is a cornerstone of sparse 3D reconstruction. The classical image matching paradigm detects keypoints per-image once and for all, which can yield poorly-localized features and propagate large errors to the final geometry. In this paper, we refine two key steps of structure-from-motion by a direct alignment of low-level image information from multiple views: we first adjust the initial keypoint locations prior to any geometric estimation, and subsequently refine points and camera poses as a post-processing. This refinement is robust to large detection noise and appearance changes, as it optimizes a featuremetric error based on dense features predicted by a neural network. This significantly improves the accuracy of camera poses and scene geometry for a wide range of keypoint detectors, challenging viewing conditions, and off-the-shelf deep features. Our system easily scales to large image collections, enabling pixel-perfect crowd-sourced localization at scale. Our code will be publicly available at as an add-on to the popular SfM software COLMAP.

[@Patrick-vq4qz]

Awesome work!

[@johncasey434]

Excellent work!

[@Matojeje]

Can't wait to see a software implementation of this!

[@alpharealcat]

awesome work😃

[@v000000000000v]

very interesting work, the output is very clean
how does it deal with lens distortion? is it compensated by the FBA where the sample with the least distortion gets chosen as the reference? or does the FBA internally have some estimation of the distortion and applies that to all keypoints from the same sample?

[@zubairmalik613]

WOW!!! I will be shocked if they did not give you the Marr prize.

[@fbayes1811]

6啊老铁