Highway Networks - Deep Neural Network Explained

Highway Networks are a type of network inspired by LSTM that make use of learnable information highway to let inputs flow unimpeded to subsequent layers.
They have a lot of similarities with residual neural networks and offer deep insight into how to make training deeper neural networks possible.
Table of Content
- Introduction: 0:00
- Degradation Problem: 0:38
- Idea Behind Highway Networks: 1:14
- Formulas: 2:11
- Training & Data: 2:57
- Plain VS Highway: 3:34
- MNIST Sanity Checks: 4:19
- FitNet vs Highway: 4:37
- SOTA vs Highway: 5:00
- Highway Activation Analysis: 5:26
- Highway Ablation Analysis: 7:34
- Conclusion: 8:39
Highway Networks Paper: arxiv.org/pdf/1505.00387v2
Training Very Deep Network Paper: arxiv.org/pdf/1507.06228
For an implementation of Highway Networks do check this repository:
github.com/protonx-tf-03-proj...
----
Join the Discord for general discussion: / discord
----
Follow Me Online Here:
Twitter: / yacineaxya
GitHub: github.com/yacineMahdid
LinkedIn: / yacinemahdid
___
Have a great week! 👋