Everything that comes out of MIT is pure gold. You'd think that the concepts would be described at a high, inaccessible level, but that's not the case. The lectures are student friendly & homeworks are challenging and doable.
Thank you to Alexander Amini and Ava soleimany for making this course accessible to everyone, which otherwise is a distant dream for many people like myself to learn such high quality content.
Just amazing how well those two lectures are layed out, structured and explained, nothing comes close to them in my experience so far, thank you so much Alexander and Ava, heading for the first lab now.
this is genius. This lecture is pure gold. Such difficult concepts like transformers explained in a 15 minutes seems to be impossible but she did it. Thank you MIT!
"Attention Is All you Need" - The intuition of Query, Key and Value is one of the best from what I've read or watched (in other courses) until now....Excellent job Ava Soleimany, thank you
Excellent explanation! This is perhaps the best description about the roots of the attention mechanism, and the intuition behind it. People who follow the route of CNNs -> GANs -> ViTs in their deep learning journey have trouble in understanding the self-attention (without having much knowledge about RNNs). This is like an excellent "bridge" video that fills all the gaps! Great effort by Ava!
if you are watching, learning and practicing this video, you have be granted a visa to the future. Alexander Amini, Ava Solemany and the rest of the team thanks. you guy are amazing
This is definitely the best video for describing attention mechanisms and the logic behind them. Many videos only try to review as it is written in the paper. Thank you so much! It really helped me a lot to get the attention even more clearly!
This is by far the best explanation of attention that I've seen. It definitely deserves its own video. Maybe a video on transformers that covers attention and some more detail on the other components of the architecture?
Yes, I too would love to see another lecture on attention and transformers, with a more detailed treatment of the Q, K, V matrices and how they function. I understand what was said but can’t derive it in my own.
Takes a really bright mind and a lot of practice together with hands-on experience to pack so much information about a complex subject so briefly and elegantly. I bet it took some back propagation through time to arrive to that.
Figuring out attention mechanism in minutes is super cool and intuitive. Thanks ava for such a clarity of your train of thoughts throughout the lecture. God bless.
I struggled to wrap my head around neural networks for sequential data. The intuition provided for each concept is perfect, it was so engaging that the timeline of this lecture is now embedded in my head (pun intended ;))
Thanks a lot for uploading these videos. I have already done the amazing DLS by Andrew Ng, but this video is a perfect summary and a revision. By the way, at 32:30, it's the derivative that is assigned the value 1 whenever the argument is greater than 0. I guess that's a speaking error, but just wanted to put the forward.
43:00 "long memory" is a bit misleading as transformer has O(n^2) in its attention layer, which is not scalable either (there are variations with better O, but they are less poweful). so it might work better than rnn/lstm, it is not a silver bullet and only slighty increases the context size (to few thousand tokens). we still can't have "long memory" in the sense of hundreds of thousands or millions of tokens.
I love these series! Thank you for sharing the knowledge! I am listening to very word! Now I am getting Instagram ads for MIT Full AI course for the hefty price of $3300 USD , I wish I could afford it ;/
The attention mechanism is being used widely in computer vision (vit). In an example of image classification, what type of image data is treated as query, key and value ?
HI. i stll can not understand the encoding bottleneck about limitations of RNN, what exactly means about encoding bottleneck? can anyone explain it more clearly?
Since you get some intuition here, consider this lecture here ua-cam.com/video/S27pHKBEp30/v-deo.html I found his explanation about one of the shortcomings "encoding bottleneck" of the lstms that motivated the need for the attention mechanism useful. I hope you will find it helpful.
This is absolutely amazing! Does anyone know of any programs that can be used to generate such great looking neural network/code slides? Or the one that was used to create these?
I dont really understand the „many to many“ example in the sequence modeling applications. It looks to me that the output sequence has always the same length as the input sequence, which is rare for language translation. Also the first element of the output sequence is produced without any feedback from later timestamps and is not reevaluated later on.
Maybe as an example: the network translates english to german and the input sequence starts with „The“. The Network might then have a first output sequence element of „Das“, which is fine. Then the second element of the Input sequence is „Human“, the network gets „Mensch“ as second output element. Now it would have to change the first output from „Das“ to „Der“, which i dont think would ever happen with the presented Modeling.
I need to get some help regarding the process of getting admission in MIT EECS for under graduation that how can I apply and what are the requirements?? If someone has any idea kindly help me out with that I will really appreciate your effort...
I'm still trying to figure out how did you manage to perfectly describe the logic behind attention mechanisms in 10 minutes ...
It was absolutely amazing, especially how she connect the notion of search to attention. Thanks for the intuitive connection.
@@MahJohn .aaaaaaakaaaaaaa
Everything that comes out of MIT is pure gold. You'd think that the concepts would be described at a high, inaccessible level, but that's not the case. The lectures are student friendly & homeworks are challenging and doable.
Yeah unlike the Harvards of the world MIT always seemed to focus on having the richest education, not the richest students
just wanted to know is this just theoretical stuff or they also walk through any code for its implementation ??
Thank you to Alexander Amini and Ava soleimany for making this course accessible to everyone, which otherwise is a distant dream for many people like myself to learn such high quality content.
Just amazing how well those two lectures are layed out, structured and explained, nothing comes close to them in my experience so far, thank you so much Alexander and Ava, heading for the first lab now.
this is genius. This lecture is pure gold. Such difficult concepts like transformers explained in a 15 minutes seems to be impossible but she did it. Thank you MIT!
"Attention Is All you Need" - The intuition of Query, Key and Value is one of the best from what I've read or watched (in other courses) until now....Excellent job Ava Soleimany, thank you
Who needs GPT-3 when we have Ava? Amazingly clear, succinct, and enjoyable presentation. Thank you Ava!
This is by far the best explanation of Transformers that I have ever seen. It all makes more sense now. Thanks
Excellent explanation! This is perhaps the best description about the roots of the attention mechanism, and the intuition behind it. People who follow the route of CNNs -> GANs -> ViTs in their deep learning journey have trouble in understanding the self-attention (without having much knowledge about RNNs). This is like an excellent "bridge" video that fills all the gaps! Great effort by Ava!
if you are watching, learning and practicing this video, you have be granted a visa to the future. Alexander Amini, Ava Solemany and the rest of the team thanks. you guy are amazing
This is definitely the best video for describing attention mechanisms and the logic behind them. Many videos only try to review as it is written in the paper. Thank you so much! It really helped me a lot to get the attention even more clearly!
Thank you very much!
Feels like I'm waiting for a much awaited movie trailer! This is quality.
God bless MIT
Set reminder, patiently waiting. It's a great initiative, cant thank the organizers and instructors enough!
This is by far the best explanation of attention that I've seen. It definitely deserves its own video. Maybe a video on transformers that covers attention and some more detail on the other components of the architecture?
Yes, I too would love to see another lecture on attention and transformers, with a more detailed treatment of the Q, K, V matrices and how they function. I understand what was said but can’t derive it in my own.
Excellent presentation on the transition from RNN to Attention-based Transformer networks. Thank you
This is just the best, brilliant explanation of RNN and attention mechanism I've ever seen! Thank you guys for such a hard work!
Takes a really bright mind and a lot of practice together with hands-on experience to pack so much information about a complex subject so briefly and elegantly. I bet it took some back propagation through time to arrive to that.
The best explanation of attention mechanism I have ever seen. It is very intuitive and easy to understand.
Excellent lecture. Very well designed, clear, intuitive, well balanced. A lot was accomplished in one hour! I learned a lot.
Figuring out attention mechanism in minutes is super cool and intuitive. Thanks ava for such a clarity of your train of thoughts throughout the lecture. God bless.
unable to describe how amazing is this ... thank you Ava
Ava is a very talented lecturer - thanks for the cogent explanation of RNNs.
The "deep learning couple" is at it again! and congrats!
Finally, I understood the self attention mechanism completely.
Precise and very well explained. Thank you for making this course accessible.
Thanks for detailed explanations. Especially, attention!And finally attention all that we need and additionally understand thanks to you:-)
Amazing intuition behind Transformers, thank you!
this is one of the best lectures ever
43:26 - The Transformers: More than Meets the Eye. Missed it on the news, but saw it on Netflix.
Really intuitive way of teaching. The concepts are explained really well.
Good explanation on self-attention. It gives me better intuitions on the topic.
Yes I really appreciate the intuition of attention very much.
Passion for science and technology is just oozing out of Ava !! Persian Passion.
I struggled to wrap my head around neural networks for sequential data. The intuition provided for each concept is perfect, it was so engaging that the timeline of this lecture is now embedded in my head (pun intended ;))
Thanks a lot for uploading these videos. I have already done the amazing DLS by Andrew Ng, but this video is a perfect summary and a revision. By the way, at 32:30, it's the derivative that is assigned the value 1 whenever the argument is greater than 0. I guess that's a speaking error, but just wanted to put the forward.
God bless you, Alexander and co
the single most fascinating and entertaining video for me.... let me just express my appreciation YYDS (means best of ever)
nice and clear video explantion on RNNs and Attention
43:00 "long memory" is a bit misleading as transformer has O(n^2) in its attention layer, which is not scalable either (there are variations with better O, but they are less poweful). so it might work better than rnn/lstm, it is not a silver bullet and only slighty increases the context size (to few thousand tokens). we still can't have "long memory" in the sense of hundreds of thousands or millions of tokens.
Thank you for this lecture!
This is so well explained - thanks a lot
I am happy to be able to access this course. Your job is much appreciated. THANKS
Ava Khanum, you put together an excellent lecture. Thank you very much!
Such a great explanation! Thanks for sharing!!!
Nicee... Glad to see you guys are back.
This is very well made. Thanks a lot!
well done. proud of you when I see there are such persian speaking people.
Thanks for the content, all the details were amazing!
What a lecture! Thanks a lot and keep up the great work.
Why does the naive concatenated approach lose a notion of sequence? (I'm confused about the red X next to No Order at 42:40)
Loved the explanation of Transformers you made such a complicated topic so much elegant and easy to understand
you can directly use FIR and IIR filter methods to neural networks, and laplace transforms
you just have multiple output and inputs, maybe like image filtering
or video filtering, with the time data with frames included
I love these series! Thank you for sharing the knowledge! I am listening to very word! Now I am getting Instagram ads for MIT Full AI course for the hefty price of $3300 USD , I wish I could afford it ;/
Which course is this?
excellent lecture, I'm excited!
at time step 42:00 is she referring to encoder and decoder model
Impressive presentation, thanks a lot for sharing !
Very great explanation, completely understood. thanks to team MIT 😁
16:19 Shouldn't the last loss be Lt, rather than L3?
I suppose it's just some typo.
Yes, thanks for pointing that out
The attention mechanism is being used widely in computer vision (vit). In an example of image classification, what type of image data is treated as query, key and value ?
Oh Ava became your wife! congrats!
Awesome loved this knowledge shower from India
HI. i stll can not understand the encoding bottleneck about limitations of RNN, what exactly means about encoding bottleneck? can anyone explain it more clearly?
Since you get some intuition here, consider this lecture here ua-cam.com/video/S27pHKBEp30/v-deo.html
I found his explanation about one of the shortcomings "encoding bottleneck" of the lstms that motivated the need for the attention mechanism useful. I hope you will find it helpful.
Ava forces herself to be clear splitting in little piece complex concepts to be understandable by us.
What's really "self" in self-attention? If it's the "input" as mentioned in this video, how is it different from attention?
Reminder on,
Wonderful !!! Ava
This is absolutely amazing! Does anyone know of any programs that can be used to generate such great looking neural network/code slides? Or the one that was used to create these?
Thanks! We use Powerpoint for the slides.
Very cool lecture.
57:51
This is goldddd
Great video!
good work thanks a lot
but how we can get the linear layer or how we calculate it to get attention
I dont really understand the „many to many“ example in the sequence modeling applications. It looks to me that the output sequence has always the same length as the input sequence, which is rare for language translation. Also the first element of the output sequence is produced without any feedback from later timestamps and is not reevaluated later on.
Maybe as an example: the network translates english to german and the input sequence starts with „The“. The Network might then have a first output sequence element of „Das“, which is fine. Then the second element of the Input sequence is „Human“, the network gets „Mensch“ as second output element. Now it would have to change the first output from „Das“ to „Der“, which i dont think would ever happen with the presented Modeling.
Why we have same weights for different inputs??
Anyone tried Lab1 Part2? I wonder whether anyone get any songs in the last cell? Even the sample_song does not work.
Where i can find the link of attention mechanisme lab? i did check the link..
Great lecture
Awesome explanation about attention mechenism in a very shot time hahhahaha!
Bravo!
Is it possible to get a certificate from the related website? I'd be grateful if you guide me.
What is linear layer at 50:03?
A dense (i.e., fully connected) layer with a linear activation function (i.e., no activation function).
@@AAmini Got it. By the way, I was not expecting this urgent response. Thank you sir Mr. Amini.
Thanks for the question!
I need to get some help regarding the process of getting admission in MIT EECS for under graduation that how can I apply and what are the requirements?? If someone has any idea kindly help me out with that I will really appreciate your effort...
thank you
I'm waiting ....
How is this available for free
Very fast videos. Need to slow down and explain key concepts clearly. Otherwise it's like a sweet story.
Too good to be true! But why? Thanks a ton!
10:46
Can I give this video more than 1 like?
💯
Comment for algorithm
30:28 What exactly is gradient clipping ?
I'm waiting ....