I really enjoyed giving this talk at the Weights & Biases salon, here is the outline: 00:00 - Intro 00:58 - The context of the project 02:52 - Numerical analysis & visualization complementing each other 03:27 - The importance of visualization 05:22 - Dimensionality reduction techniques 06:19 - The creative aspect of the project, improving understanding 07:09 - Connections between the geometry of the surfaces and properties of the networks 08:20 - Navigating high dimensional weight space 09:17 - PCA directions vs random directions 11:03 - The importance of normalizing the directions 12:31 - Using the eigenvalues of the hessian to verify the distribution of non-convexities & convexities in these representations 14:01 - Capturing the dynamics of the landscapes in movement 14:14 - Counter intuitive effects in the dynamics of these representations 14:30 - Our flatland reality 14:44 - Matt Parker's examples of counterintuitive effects between dimensions 15:55 - Filtering the high dimensional space through our flatland reality 16:34 - Noise in the morphology and the dynamics of the landscapes 17:25 - Tunneling, on the way towards the main convexity 18:17 - The blessing of dimensionality - Babak Hassibi 18:56 - Exploring cross-sections of the high dimensional spaces 19:34 - A multidisciplinary pipeline 20:00 - Learning rate stress tests 20:50 - Mode connectivity - Connecting minima while maintaining a low loss value 23:10 - Studies comparing specific parts of the landscapes 23:27 - Resnets, non-skip vs skip connections 23:40 - The loss landscape library project 24:13 - Mish, ReLU & Swish activation functions 25:10 - Lottery garden - The lottery ticket hypothesis 26:24 - Edge horizons, downfalls and minima, studies 28:23 - Dropout, static and in movement 29:07 - Bayesian deep learning, approximating the posterior 30:13 - Wasserstein GANs and the generator 31:10 - Geometric deep learning - Neural concept 31:44 - The Loss landscape explorer app
I really enjoyed giving this talk at the Weights & Biases salon, here is the outline:
00:00 - Intro
00:58 - The context of the project
02:52 - Numerical analysis & visualization complementing each other
03:27 - The importance of visualization
05:22 - Dimensionality reduction techniques
06:19 - The creative aspect of the project, improving understanding
07:09 - Connections between the geometry of the surfaces and properties of the networks
08:20 - Navigating high dimensional weight space
09:17 - PCA directions vs random directions
11:03 - The importance of normalizing the directions
12:31 - Using the eigenvalues of the hessian to verify the distribution of non-convexities & convexities in these representations
14:01 - Capturing the dynamics of the landscapes in movement
14:14 - Counter intuitive effects in the dynamics of these representations
14:30 - Our flatland reality
14:44 - Matt Parker's examples of counterintuitive effects between dimensions
15:55 - Filtering the high dimensional space through our flatland reality
16:34 - Noise in the morphology and the dynamics of the landscapes
17:25 - Tunneling, on the way towards the main convexity
18:17 - The blessing of dimensionality - Babak Hassibi
18:56 - Exploring cross-sections of the high dimensional spaces
19:34 - A multidisciplinary pipeline
20:00 - Learning rate stress tests
20:50 - Mode connectivity - Connecting minima while maintaining a low loss value
23:10 - Studies comparing specific parts of the landscapes
23:27 - Resnets, non-skip vs skip connections
23:40 - The loss landscape library project
24:13 - Mish, ReLU & Swish activation functions
25:10 - Lottery garden - The lottery ticket hypothesis
26:24 - Edge horizons, downfalls and minima, studies
28:23 - Dropout, static and in movement
29:07 - Bayesian deep learning, approximating the posterior
30:13 - Wasserstein GANs and the generator
31:10 - Geometric deep learning - Neural concept
31:44 - The Loss landscape explorer app
wtf only 104 views? this is amazing, thank you so much!
thank you for appreciating the vid Gustav :) well you know, there is so much competition in youtube, even having 100 views is not easy :)
Hermoso,muy buena presentación
thank you for appreciating the work and presentation