Hossein Mobahi: Sharpness-Aware Minimization (SAM): Current Method and Future Directions

Slides: www.dropbox.com/s/66wet9ps2a6...
TITLE:
Sharpness-Aware Minimization (SAM): Current Method and Future Directions
ABSTRACT:
In today's heavily overparameterized models, the value of the training loss provides few guarantees on model generalization ability. Indeed, optimizing only the training loss value, as is commonly done, can easily lead to suboptimal model quality. Motivated by prior work connecting the geometry of the loss landscape and generalization, we introduce a new and effective procedure for instead simultaneously minimizing loss value and loss sharpness. In particular, our procedure, Sharpness-Aware Minimization (SAM), seeks parameters that lie in neighborhoods having uniformly low loss; this formulation results in a min-max optimization problem on which gradient descent can be performed efficiently. We present empirical results showing that SAM improves model generalization across a variety of benchmark datasets (e.g., CIFAR-10, CIFAR-100, ImageNet, finetuning tasks) and models, yielding novel state-of-the-art performance for several. Additionally, we find that SAM natively provides robustness to label noise on par with that provided by state-of-the-art procedures that specifically target learning with noisy labels. Finally, we will discuss possible directions for further research around SAM.
BIO:
Hossein Mobahi is a senior research scientist at Google Research. His current interests revolve around the interplay between optimization and generalization in deep neural networks. Prior to joining Google in 2016, he was a postdoctoral researcher in CSAIL of MIT. He obtained his PhD in Computer Science from the University of Illinois at Urbana-Champaign (UIUC).

[@toby3927]

Very helpful. A clear explanation of the idea behind the algorithm.

[@axe863]

Awesome idea.

[@gyeonghokim]

Thanks for a great talk

[@huanranchen]

Hi~ Is there a video about ASAM? I wanna learn more about ASAM