[CVPR 2024] TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models

MERL former intern Haomiao Ni presents our paper, "TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models," at the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), held in Seattle, Washington June 17-21 2024. The paper was co-authored with MERL researchers Suhas Lohit, Anoop Cherian, Ye Wang, Toshiaki Koike-Akino, and Tim K. Marks, as well as MERL consultant Prof. Bernhard Egger from FAU and Prof. Sharon X. Huang from Penn State University.
Paper: www.merl.com/publications/TR2...
Abstract: Text-conditioned image-to-video generation (TI2V) aims to synthesize a realistic video starting from a given image (e.g., a woman's photo) and a text description (e.g., "a woman is drinking water."). Existing TI2V frameworks often require costly training on video-text datasets and specific model designs for text and image conditioning. In this paper, we propose TI2V-Zero, a zero-shot, tuning-free method that empowers a pretrained text-to-video (T2V) diffusion model to be conditioned on a provided image, enabling TI2V generation without any optimization, fine-tuning, or introducing external modules. Our approach leverages a pretrained T2V diffusion foundation model as the generative prior. To guide video generation with the additional image input, we propose a "Repeat-and-Slide" strategy that modulates the reverse denoising process, allowing the frozen diffusion model to synthesize a video frame-by-frame starting from the provided image. To ensure temporal continuity, we employ a DDPM inversion strategy to initialize Gaussian noise for each newly synthesized frame and a resampling technique to help preserve visual details. We conduct comprehensive experiments on both domain-specific and open-domain datasets, where TI2V-Zero consistently outperforms a very recent open-domain TI2V model. Furthermore, we show that TI2V-Zero can seamlessly extend to other tasks such as video infilling and prediction when provided with more images. Its autoregressive design also supports long video generation. Code will be made available upon publication.