Understanding and Mitigating Copying in Diffusion Models

A Google TechTalk, presented by Gowthami Somepalli , 2023-08-09
Abstract: Cutting-edge diffusion models produce high-quality and customizable images, enabling their use for commercial art and graphic design purposes. First, I will discuss our study of various frameworks to detect replication. Then, I will show how we used these frameworks to identify memorization in Stable Diffusion 1.4 model. In the second part, I will discuss various factors contributing to memorization in diffusion models. While it is widely believed that duplicated images in the training set are responsible for content replication at inference time, I will show results on how the text conditioning of the model also plays an important role. Lastly, I will discuss several techniques we proposed based on these findings for reducing data replication in both training and inference times.