this is absolutely FANTASTIC I watched Albert Gu's stanford lecture on state space models/ Mamba, and it was a great high level overview. But I really appreciate you taking it slower, and going farther into detail on the basic/ fundamental concepts. A lot of us aren't mathematicians or ML engineers, so it's much appreciated to be helped along with those concepts.
I rarely comment on videos, but this one was worth it. Thank you so much for such a clear explanation. You explained all the nuances that I previously did not understand in a very clear way. God bless you.
Your teaching approach is very good. You started from fundamental concepts and went deeper. This helped in gaining intuitions, understanding and avoid confusions in later part. Brilliant!
I just read about mamba and wanted to find a detailed explanation video. All you covered in this video is everything I need, thank you so much, keep on cooking
You are just too amazing! You can understand these stuff in great detail. Then you take the time and explain to us in educative videos. A true gem channel!
Thanks a ton! Excellent explanation and great analogies to introduce the more advanced material. This is an absolute masterclass on how to teach advanced material.
Thank you for this great and smooth explanation. I think the model you are showing at 36:14 is valid if matrix A ( and B also to send each input directly to the corresponding ssm) is diagonal. Now in this way each hidden state at different canonical direction ( or different element of the vector) is independent of each other. So if A is not diagonal then assuming an eigen decomposition exist, then we may say there exist an equivalent ssm which can be represented independent ( if we change the basis to eigen basis) .
As others have mentioned, you have a keen ability to explain difficult topics succinctly and completely. Keep up the awesome work! I could of used this when I took a class on time-series modeling! Hah!
Even I understood much of this. I have no education. Thank you! Mamba looks really cool. Especially like the long context and further refinement. It looks like a model that could be made to learn as it goes. Plasticity potential
Amazing explanation. I love this video because it covers sufficient depth and explains each concept with proper examples. I've subscribed instantly, and look forward to more such videos on recent papers.
Thanks for your clear explanation of MAMBA, coming from a control theory background, very much appreciate its usage in LLMs. One small error that I noted was that the A matrix must be N x N to translate the previous N-dimensional hidden states h(t-1) to h(t). I believe the A matrix is also time-varying to produce selective output tokens.
Dear Umar, referring to 53:50, recurrent SSM is indeed similar as prefix-sum (i.e., y=x_0+x_1+....x_N), but I the difference is that h_t=Ah_{t-1}+Bx_t, where h_{t_1} depends on h_{t-2}. I know how Blelloch parallel prefix scan works for calculating the sum of constants, but I do not know how parallel scan works for h_t=Ah_{t-1}+Bx_t. Could you please elaborate on it ? Thank you. @Umar
Hi, I was wondering if you could explain 36:40 a bit more where you talk about multi head attention. From what I understand each head in multi-head attention each head looks at the whole input vector. Our key value and query matrices are all of size Dx(head_size) where D being dimension of embedding, so when we find key say we do key = X @ key_matrix where X is an CxD dimensional matrix, C is context len. This means each head looks at the whole dimension of the embedding D and represents it a head_size vector meaning that arrows going into each head should point at every single input dim.
Very good lecture ! Thank you very much for putting this for free on youtube :) I have question though, if my understanding of the HiPPO framework is correct, the A matrix is built to uniformly approximate the input signal (name HiPPO LegS in the paper). "Our novel scaled Legendre measure (LegS) assigns uniform weight to all history [0, t]". But however at 41:49 you explain that it is decaying exponentially similarly to HiPPO LagT. Do they opt for HiPPO LagT when moving to s4 and Mamba or am I missing something ?
You're making very useful content, thank you!!! Maybe you could consider using larger text, so that one could read easily from a phone. Also a plus would be if the presentation were white on black (or bright color on black), it is less tiring to look at a dark screen for long periods of time.
Trying to follow the rationale/last-slides - one advantage of SSM/RNN was that they would scale to infinite context. But Mamba reintroduced L-lengthed parameters. Why is this not limiting to this architecture the same way it limits Transformers? Qualitatively, it seems the only remaining advantage over transformers is the inference is cheaper - could you help clarify? Thanks
Hey! Thanks for the details in this video. I'm confused about the HiPPO matrix, which seems to be fixed given N? However the paper stated that delta, A, B, C are all trainable. What did I miss?
Thanks for clarification. Could you please further explain how the parameter of A is (D, N) in S4? If I have D*SSMs, one for each embedding dimension, shouldn't A have DN^2 parameters?
this is so far the only video I found that described the math part in the mamba model. thanks a lot. One small issue. In 37:00, for the attention model, you mentioned each head takes only a portion of input dimensions, can you confirm this? I believe each head actually use all input dimensions.
Hello! First of all thanks for the kind words. Yes, in multi-head attention, the idea is that each head sees the entire sequence, but a different portion of the embedding of each token. This is to make each head relate tokens in different ways. This mechanism is described in my previous video on the Transformer model.
I have one question in terms of the example which you provided, 'the number of buddies'. I think the function should be like this : b(t)=5squ(3)^λt . please comment to me if I am wrong.
Hi Professor! Very good explanation as always. However, I have huge difficulties to understand the dimensions of objects. Why the hell A matrix would be of (D,N) dimensions since it is used to project a vector h_t-1 of N dimensions into N dimensions? By the way, why is it written "Represents structured N x N matrix" ?????!!!!
Coding one is not very interesting, because the most interesting part is the selective scan algorithm, which is a CUDA Kernel. The architecture is not so different from any other language model. Of course it would be super cool to code the CUDA kernel from scratch ;-)
Brilliant - you are easily one of the most lucid and accessible teachers of deep learning.
this is absolutely FANTASTIC
I watched Albert Gu's stanford lecture on state space models/ Mamba, and it was a great high level overview.
But I really appreciate you taking it slower, and going farther into detail on the basic/ fundamental concepts.
A lot of us aren't mathematicians or ML engineers, so it's much appreciated to be helped along with those concepts.
Thank you for your kind words. Please share the video in your network, it would help me a lot. Thanks!
I rarely comment on videos, but this one was worth it. Thank you so much for such a clear explanation. You explained all the nuances that I previously did not understand in a very clear way. God bless you.
Your teaching approach is very good. You started from fundamental concepts and went deeper. This helped in gaining intuitions, understanding and avoid confusions in later part. Brilliant!
🙌 Still working through Transformers from scratch. Hopefully a Mamba from scratch is in the future!
I just read about mamba and wanted to find a detailed explanation video. All you covered in this video is everything I need, thank you so much, keep on cooking
I'm so glad i found this channel, you are a gold mine for such content, please keep them coming.
Words will fall short to appreciate the work you put to create these videos. Simply BRILLIANT.
The whole lecture was very intuitive. Thanks for the efforts put into building this video!
You are just too amazing! You can understand these stuff in great detail. Then you take the time and explain to us in educative videos. A true gem channel!
You're amazing. God Bless you. You made this the best hour i've spent on trying to understand MAMBA. Keep up the great work.
Best ever presentation I saw so far, thanks so much
Very high quality, this is great. Hard to find good content like this. Thanks Umar!
After I saw this lecture, I subscribed your channel. It is the most easy to understand Mamba lecture I've seen.
Thanks for explaining it in a way that anyone with some high school math background can understand, keep this up!
Thanks for the amazing work as usual! Keep it up - this is probably one of the highest quality content on LLMs on youtube.
作为一个来自北京的大学生，谢谢你分享的这篇文章解析！best wishes！
Excited for the video. I was searching for a video on Mamba and today I saw this. Your Transformer video helped me alot previously. Keep it up!
Thanks!
This is gold! I really appreciate attention to the details. Thank you Umar!
Understanding mamba couldn't be better than this !
Thank you so much. Lots of useful details yet you curate through them at such a good tempo with easy to follow examples
This is the best deep learning video I've ever seen. I will surely use some of your slides to teach my students
Thank you so much for your detailed video and thoughtful thinking of you that we will need help with the equations! You are a savior!
Thanks a ton! Excellent explanation and great analogies to introduce the more advanced material. This is an absolute masterclass on how to teach advanced material.
This is one of the best ML explanations I've seen even though I didn't understand all of it but I definitely learnt something new.
Excellent video! Thank you. I have watched a few videos about mamba and this one was by far the best.
I'm very thankful for your explanation of this article, best wishes for you!
Thanks!
Great explanation!! This is the first video that mekes me comprenhad the whole mamba paper.
Danke!
Thank you very very very much for your generous support! Let's connect on LinkedIn!
Best MAMBA video at the moment!
Thank you so much for your efforts to make such an amazing video on Mamba architecture !!
Thank you. I appreciate the approach you took in explaining the major concepts.
Thank you for this great and smooth explanation. I think the model you are showing at 36:14 is valid if matrix A ( and B also to send each input directly to the corresponding ssm) is diagonal. Now in this way each hidden state at different canonical direction ( or different element of the vector) is independent of each other. So if A is not diagonal then assuming an eigen decomposition exist, then we may say there exist an equivalent ssm which can be represented independent ( if we change the basis to eigen basis) .
As others have mentioned, you have a keen ability to explain difficult topics succinctly and completely. Keep up the awesome work! I could of used this when I took a class on time-series modeling! Hah!
this is just a pure art; thanks so much
very good video!!! thanks a lot for your efforts!!!!
This is really helpful for another talk I am doing on Mamba. Thank you very much for putting this out.
Brilliant video! Really clear and with just the right amount of details!
Such a briliant work you have done. Really learned a lot, thanks!!!
Even I understood much of this. I have no education. Thank you! Mamba looks really cool. Especially like the long context and further refinement. It looks like a model that could be made to learn as it goes. Plasticity potential
Amazing explanation. I love this video because it covers sufficient depth and explains each concept with proper examples. I've subscribed instantly, and look forward to more such videos on recent papers.
OMG ! this is such as amazing description , you made my day
Ohhh Man, why did I discover this gem so late :( This guy is a rockstar!
Salute to consistency
Thanks Umar sir.
Thanks for your clear explanation of MAMBA, coming from a control theory background, very much appreciate its usage in LLMs. One small error that I noted was that the A matrix must be N x N to translate the previous N-dimensional hidden states h(t-1) to h(t). I believe the A matrix is also time-varying to produce selective output tokens.
some people are just born to teach.
Love it! Keep up the amazing work.
Great job on this video! I learned a lot
Really an amazing video! You save me a lot of time! Thank you!
I did learn a lot! Many thanks for making this video.
wow that's a great explanation , thanks for the efforts!
Dear Umar, referring to 53:50, recurrent SSM is indeed similar as prefix-sum (i.e., y=x_0+x_1+....x_N), but I the difference is that h_t=Ah_{t-1}+Bx_t, where h_{t_1} depends on h_{t-2}. I know how Blelloch parallel prefix scan works for calculating the sum of constants, but I do not know how parallel scan works for h_t=Ah_{t-1}+Bx_t. Could you please elaborate on it ? Thank you. @Umar
Hi, I was wondering if you could explain 36:40 a bit more where you talk about multi head attention. From what I understand each head in multi-head attention each head looks at the whole input vector. Our key value and query matrices are all of size Dx(head_size) where D being dimension of embedding, so when we find key say we do key = X @ key_matrix where X is an CxD dimensional matrix, C is context len. This means each head looks at the whole dimension of the embedding D and represents it a head_size vector meaning that arrows going into each head should point at every single input dim.
最清晰的讲解！
i always eagerly wait for your explainer. they are 🤯.
thank you :)
Thanks for the video! Very informative! Just to check: At @1:03:42, 3. be "... save back the result to HBM."?
Thanks man! This helped me a lot
This video is of great help!!Thank you very much.
In 57:00 isn't the time complexity reduced to 2*lg(n) in parallel scan? Thanks for the amazing explanation btw. 💚
Brilliant explanations. Thanks.
Very good lecture ! Thank you very much for putting this for free on youtube :) I have question though, if my understanding of the HiPPO framework is correct, the A matrix is built to uniformly approximate the input signal (name HiPPO LegS in the paper). "Our novel scaled Legendre measure (LegS) assigns uniform weight to all history [0, t]". But however at 41:49 you explain that it is decaying exponentially similarly to HiPPO LagT. Do they opt for HiPPO LagT when moving to s4 and Mamba or am I missing something ?
are going to code it as well.
I really liked the video it was easy and very comprehensive.
Amazing! So detailed. Well done sir
Absolutely amazing 🎉
Thanks Umar! 🥰Very amazing learning material for Mamba!
thank you for this video , really helped me
You're making very useful content, thank you!!! Maybe you could consider using larger text, so that one could read easily from a phone. Also a plus would be if the presentation were white on black (or bright color on black), it is less tiring to look at a dark screen for long periods of time.
One of the best! I have one question if we apply conv in S4 on sequence of length L, what will be size of conv layer?
Thanks for the awesome content! Hope the next one will be about DPO and coding it from scratch ❤
You're welcome: ua-cam.com/video/hvGa5Mba4c8/v-deo.html
@@umarjamilai Thank you!!! You're so talented at research and teaching!!!!
Great explanation!
Very nice talk, thank you.
Trying to follow the rationale/last-slides - one advantage of SSM/RNN was that they would scale to infinite context. But Mamba reintroduced L-lengthed parameters. Why is this not limiting to this architecture the same way it limits Transformers? Qualitatively, it seems the only remaining advantage over transformers is the inference is cheaper - could you help clarify? Thanks
absolutely fantastic
Awesome video as usual
Amazing video.
excellent work! Thank you
Great lecture! It is easier for me to understand the work with your lecture.
Can you give one for Reinforcement learning?
Hey! Thanks for the details in this video.
I'm confused about the HiPPO matrix, which seems to be fixed given N?
However the paper stated that delta, A, B, C are all trainable. What did I miss?
is HiPPO the initialization of A?
Yeah, just the initialization
Thanks for clarification.
Could you please further explain how the parameter of A is (D, N) in S4? If I have D*SSMs, one for each embedding dimension, shouldn't A have DN^2 parameters?
Thanks a lot that was very useful!
you are the best.
this is so far the only video I found that described the math part in the mamba model. thanks a lot.
One small issue. In 37:00, for the attention model, you mentioned each head takes only a portion of input dimensions, can you confirm this? I believe each head actually use all input dimensions.
It might be true for LLMs, but I believe this is not true for the original transformer model.
Hello! First of all thanks for the kind words.
Yes, in multi-head attention, the idea is that each head sees the entire sequence, but a different portion of the embedding of each token. This is to make each head relate tokens in different ways. This mechanism is described in my previous video on the Transformer model.
Amazing video
It was very informative
I have one question in terms of the example which you provided, 'the number of buddies'. I think the function should be like this : b(t)=5squ(3)^λt . please comment to me if I am wrong.
i've just started watching but guess this vid'll be much usefull
Bellissimo video, grazie!
Grazie a te!
Fantastic
Hi Professor! Very good explanation as always. However, I have huge difficulties to understand the dimensions of objects. Why the hell A matrix would be of (D,N) dimensions since it is used to project a vector h_t-1 of N dimensions into N dimensions? By the way, why is it written "Represents structured N x N matrix" ?????!!!!
Hi Umar, amazing video. You are the best teacher. You are Karpathy 2.0. :) Please make a video on DPO :)
Done: ua-cam.com/video/hvGa5Mba4c8/v-deo.html
@@umarjamilai thank you so much 😃
Awesome explanation. Really appreciate such content. Can you please make a similar explanation video on the Mamba-2 paper?
amazing explanation
waiting for new video
please upload soon
Great explanation. Very through. Loved it. I struggled with understanding the SSM paper. You explained all the bits beautifully
Great❤
very great
Thank you
Thanks, i hope you explain rwkv
Please can you make video on optimizers like adam,adagrad,...
Excellent video! I'm looking forward if you do a coding one. Thank you so much for your work to the AI community
Coding one is not very interesting, because the most interesting part is the selective scan algorithm, which is a CUDA Kernel. The architecture is not so different from any other language model. Of course it would be super cool to code the CUDA kernel from scratch ;-)