Pr. Francesco Vaccarino was speaking about: Topological Constraints in Shallow ReLU Neural Networks: A Journey through Optimization Obstacles
Abstract:
This talk explores the topological properties of the parameter space in shallow neural networks, specifically those using ReLU activation functions. We present the discovery of a topological obstruction that limits gradient-based optimization within the network's loss landscape. Focusing on two-layer ReLU networks, we demonstrate how neurons’ gradient flow trajectories are confined to products of quadric hypersurfaces and examine how these constraints emerge from the network's initialization and symmetries. Calculating the invariant set's Betti numbers reveals conditions where the network's connected components limit optimal learning. The analysis draws connections between these obstructions, Segre varieties, and invariant theory, providing insights into navigating and potentially mitigating such obstacles. Joint work with Marco Nurisso and Pierrick Leroy.
To appear in Advances in Neural Information Processing Systems (NeurIPS 2024).
The recordings of the seminar BrAINs season are on the website and new youtube channel projects.learningplanetinstitute.org/projects/network-seminars-at-cri/description#tab
For descriptions of upcoming events, get the latest updates, or unsubscribe, check our Google Group!
groups.google.com/g/network-seminar-paris?pli=1
Some of the coming conference related to the topic of TDA
embedded-days.bunka.ai/