[ICCV 2021] A Hierarchical Variational Neural Uncertainty Model for Stochastic Video Prediction

MERL Researchers Moitreya Chatterjee and Anoop Cherian presented their paper titled "A Hierarchical Variational Neural Uncertainty Model for Stochastic Video Prediction" for the IEEE/CVF International Conference on Computer Vision (ICCV), held virtually October 11-17 2021. The paper was co-authored with Prof. Narendra Ahuja at University of Illinois, Urbana-Champaign.
Paper: openaccess.thecvf.com/content... , merl.com/publications/TR2021-096
Abstract: Predicting the future frames of a video is a challenging task, in part due to the underlying stochastic real-world phenomena. Prior approaches to solve this task typically estimate a latent prior characterizing this stochasticity, however do not account for the predictive uncertainty of the (deep learning) model. Such approaches often derive the training signal from the mean-squared error (MSE) between the generated frame and the ground truth, which can lead to sub-optimal training, especially when the predictive uncertainty is high. Towards this end, we introduce Neural Uncertainty Quantifier (NUQ) - a stochastic quantification of the model's predictive uncertainty, and use it to weigh the MSE loss. We propose a hierarchical, variational framework to derive NUQ in a principled manner using a deep, Bayesian graphical model. Our experiments on four benchmark stochastic video prediction datasets show that our proposed framework trains more effectively compared to the state-of-the-art models (especially when the training sets are small), while demonstrating better video generation quality and diversity against several evaluation metrics.