TUM AI Lecture Series - The 3D Gaussian Splatting Adventure: Past, Present, Futur (George Drettakis)

Abstract:
Neural rendering has advanced at outstanding speed in recent years, with the advent of Neural Radiance Fields (NeRFs), typically based on volumetric ray-marching. Last year, our group developed an alternative approach, 3D Gaussian Splatting, that has better performance for training, display speed and visual quality and has seen widespread adoption both academically and industrially. In this talk, we describe the 20+ year process leading to the development of this method and discuss some future directions. We will start with a short historical perspective of our work on image-based and neural rendering over the years, outlining several developments that guided our thinking over the years. We then discuss a sequence of three point-based rasterization methods for novel view synthesis -- developed in the context of G. Kopanas' Ph.D. and the ERC Advanced Grant FUNGRAPH -- that culminated with 3D Gaussian Splatting. We will emphasize how we progressively overcame the challenges as the research progressed. We first discuss differentiable point splatting and how we extended it in our first approach that enhances points with neural features, optimizing geometry to correct reconstruction errors. We briefly review our second method that handles highly reflective objects, where we use multi-layer perceptrons (MLP), to learn the motion of reflections and to perform the final rendering of captured scenes. We then discuss 3D Gaussian Splatting, that provides the high-quality real-time rendering for novel view synthesis using a novel 3D scene representation based on 3D Gaussians and fast GPU rasterization, and present some recent improvements. We will conclude with a discussion of future directions for 3D Gaussian splatting with examples from recent work.

[@anatolicvs]

Excellent presentation and outstanding lecture by an eminent scientist. We should also thank @Matthias Niessner for sharing this fascinating video with us.

[@sumitdeshpande1]

Thank You Dr. George Drettakis.