Ep 34. Deep Double Descent: Where Bigger Models and More Data Hurt

This episode delves into the concept of "double descent" in deep learning, challenging the traditional assumption that model complexity and performance are simply a U-shaped curve. The paper "Deep Double Descent: Where Bigger Models and More Data Hurt" highlights that performance can worsen initially as models grow larger and training time increases, before ultimately improving again. The episode also explores the concept of "effective model complexity" (EMC), suggesting that focusing on how effectively models utilize knowledge, rather than just raw knowledge, is crucial. The discussion emphasizes the prevalence of "model misspecification" and the importance of acknowledging the inherent noise in real-world data.
Original paper: arxiv.org/abs/...

[@collecct1on]

Great video.
The relation between labeled noise and
double descent was very interesting 👨🏻‍💻