Alex Graves invented CTC and RNNT, which is basically a modern e2e ASR model in 2013. It created tens of thousands of research jobs, and he left to seek his desire. His journey is inspiring. He doesn't seek fame or money or status. He seeks the answer to his internal curiosity. I wanna live like him.
A no-nonsense detailed attention based lectures. A very well prepared lecture for all (beginners and experienced deep learning practitioner). Greatly recommended for all who want a context on how attention is first thought through in the research world. Thank you Alex. Enjoyed the lecture.
Great Lecture ! Highly recommend anyone who is looking for indepth understanding of attention and different families of attention mechanism please watch this video . Its the best attention explanation available on the entire web.
hey! i really enjoyed the machine lecturing! BUT!!!! your name is graves but i dont see a scar and i played u quite a bit in aram and also no cigar and also no shotgun and also no collector in ur item list in the background! PROPS FOR THE BEARD!!! FRAUD!!!!!!!
For anyone that watched this lecture and his lecture from two years ago, is the difference large enough for me to watch the one from two years ago? Thanks
These things seem eeringly similar to an idea i had 5 years ago and wrote some innocuous linkedin article around tge same same transformers delved into attnetion mechanism. But, only those who actually are in Harvard, MIT, deep mind can actually implement it with the resources that are required for it.
No worries! Jürgen Schmidhuber already invented all those stuff and all relevant ideas at least 30-40 years ago. Maybe earlier, just around while he was a suckling infant.
Also, understand the difference between me and you, i never want credit for anything but i don't like stealing. Do you know who rudra is, that is what i become when i see too much injustice.. So better stay away from sucking any future generation's blood. If i see such injustice, believe me you will know God's wrath..
Alex Graves invented CTC and RNNT, which is basically a modern e2e ASR model in 2013. It created tens of thousands of research jobs, and he left to seek his desire. His journey is inspiring. He doesn't seek fame or money or status. He seeks the answer to his internal curiosity. I wanna live like him.
*DeepMind x UCL | Deep Learning Lectures | 8/12 | Attention and Memory in Deep Learning*
*My takeaways:*
*1. Introduction **1:24*
1.1 Attention, memory and cognition 1:28
1.2 Attention in neural networks 2:50
-Implicit
-Can be checked through Jacobian
1.3 Explicit attention: hard attention, non-differentiable 17:00
-It has several advantages over implicit attention
--Computational efficiency
--Scalability (e.g. fixed size glimpse for any size image)
--Sequential processing of static data (e.g. moving gaze)
--Easier to interpret
-Neural attention models 19:24
-Glimpse distribution 20:25
-Attention with reinforcement learning 21:12
-Complex glimpse 22:46
*2. Explicit attention: soft attention, differentiable **26:27*
2.1 Basic 28:15
2.2 Attention weights 29:22
2.3 An example: handwriting synthesis with RNNs 32:40
2.4 Associative attention 38:38
2.5 Differentiable visual attention 45:30
*3. Introspective attention **49:23*
3.1 Neural Turing Machine 51:02
3.2 Selective attention 52:53
3.3 Content-based and location-based attention 55:28
3.4 Differentiable Neural Computer 1:12:04
*4. Further topics **1:13:51*
4.1 Self-attention in Transformers 1:14:00
*5. Summary **1:34:14*
22:25 : I'm checking all my tabs for notifications
@jawad mansoor You’re welcome
A no-nonsense detailed attention based lectures. A very well prepared lecture for all (beginners and experienced deep learning practitioner). Greatly recommended for all who want a context on how attention is first thought through in the research world. Thank you Alex. Enjoyed the lecture.
From comment of Lei Xun (I added a 0.00 timestamp for the see chapters in the video)
0. Opening 0:00
1. Introduction 1:24
1.1 Attention, memory and cognition 1:28
1.2 Attention in neural networks 2:50
-Implicit
-Can be checked through Jacobian
1.3 Explicit attention: hard attention, non-differentiable 17:00
-It has several advantages over implicit attention
--Computational efficiency
--Scalability (e.g. fixed size glimpse for any size image)
--Sequential processing of static data (e.g. moving gaze)
--Easier to interpret
-Neural attention models 19:24
-Glimpse distribution 20:25
-Attention with reinforcement learning 21:12
-Complex glimpse 22:46
2. Explicit attention: soft attention, differentiable 26:27
2.1 Basic 28:15
2.2 Attention weights 29:22
2.3 An example: handwriting synthesis with RNNs 32:40
2.4 Associative attention 38:38
2.5 Differentiable visual attention 45:30
3. Introspective attention 49:23
3.1 Neural Turing Machine 51:02
3.2 Selective attention 52:53
3.3 Content-based and location-based attention 55:28
3.4 Differentiable Neural Computer 1:12:04
4. Further topics 1:13:51
4.1 Self-attention in Transformers 1:14:00
5. Summary 1:34:14
Great Lecture ! Highly recommend anyone who is looking for indepth understanding of attention and different families of attention mechanism please watch this video . Its the best attention explanation available on the entire web.
Great lecture! I really appreciate how he explains the thought process behind the new ideas.
Selam türk
This is one of my all-time favorite lectures, thanks for making this available. DNCs are very interesting.
This is sooooooo good. So well explained. It's like a Neuralink knowledge upload to my brain. Thanks, Alex!
I like the "Thank you very much for your attention" punch line at the end.
Another high quality course from deepmind, thanks !
You are really brilliant, sir. I am from your friend country Bangladesh 🇧🇩. Hope you will be more and more helpful
This is only the beginning.
Looking for a lecture on attention mechanism..and This was the best.
Revive UA-cam
Great lecture and big thanks to DeepMind for sharing this great content.
wonderful! Thanks for putting these lectures out!!
thank you for this lecture, learned a lot about attention
Excellent, Added To My Research Library, Sharing Through TheTRUTH Network...
Thanks Alex for the cool lecture and research!
Dear DeepMind, the link for the slides seems to be valid. Can anyone fix that?
Splendid!! 🎉
hey! i really enjoyed the machine lecturing! BUT!!!! your name is graves but i dont see a scar and i played u quite a bit in aram and also no cigar and also no shotgun and also no collector in ur item list in the background! PROPS FOR THE BEARD!!! FRAUD!!!!!!!
totally agree. needless to say he was camping base the entire time afk
For anyone that watched this lecture and his lecture from two years ago, is the difference large enough for me to watch the one from two years ago? Thanks
The section discussed after 1:14:00 (Further topic) is new.
@@mabbasiazad thanks!
Fantastic :)
Thanks Alex for a great lecture!
Memmory Augmented Neural Networks are the next big thing.
Amazing lecture!
really.....
so much interesting
These things seem eeringly similar to an idea i had 5 years ago and wrote some innocuous linkedin article around tge same same transformers delved into attnetion mechanism. But, only those who actually are in Harvard, MIT, deep mind can actually implement it with the resources that are required for it.
No worries! Jürgen Schmidhuber already invented all those stuff and all relevant ideas at least 30-40 years ago. Maybe earlier, just around while he was a suckling infant.
@@robensonlarokulu4963 yeah.. Also true that losers will be losers right from the time they are born.
Also, understand the difference between me and you, i never want credit for anything but i don't like stealing. Do you know who rudra is, that is what i become when i see too much injustice.. So better stay away from sucking any future generation's blood. If i see such injustice, believe me you will know God's wrath..
Thanks for sharing this
Thank you for such a good lecture! :=)
Amazing.
Police UA-cam
"Attention is all you need"? What a missed opportunity to call the paper "Give me some attention and I'll do everything you want". ;)