What are Transformer Neural Networks?
Вставка
- Опубліковано 27 чер 2024
- This short tutorial covers the basics of the Transformer, a neural network architecture designed for handling sequential data in machine learning.
Timestamps:
0:00 - Intro
1:18 - Motivation for developing the Transformer
2:44 - Input embeddings (start of encoder walk-through)
3:29 - Attention
6:29 - Multi-head attention
7:55 - Positional encodings
9:59 - Add & norm, feedforward, & stacking encoder layers
11:14 - Masked multi-head attention (start of decoder walk-through)
12:35 - Cross-attention
13:38 - Decoder output & prediction probabilities
14:46 - Complexity analysis
16:00 - Transformers as graph neural networks
Original Transformers paper:
Attention is All You Need - arxiv.org/abs/1706.03762
Other papers mentioned:
(GPT-3) Language Models are Few-Shot Learners - arxiv.org/abs/2005.14165
(DALL-E) Zero-Shot Text-to-Image Generation - arxiv.org/abs/2102.12092
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding - arxiv.org/abs/1810.04805
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity - arxiv.org/abs/2101.03961
Finetuning Pretrained Transformers into RNNs - arxiv.org/abs/2103.13076
Efficient Transformers: A Survey - arxiv.org/abs/2009.06732
Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth - arxiv.org/abs/2103.03404
Do Transformer Modifications Transfer Across Implementations and Applications? - arxiv.org/abs/2102.11972
Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies - ml.jku.at/publications/older/...
Transformers are Graph Neural Networks (blog post) - thegradient.pub/transformers-...
Video style inspired by 3Blue1Brown
Music: Trinkets by Vincent Rubinetti
Links:
UA-cam: / ariseffai
Twitter: / ari_seff
Homepage: www.ariseff.com
If you'd like to help support the channel (completely optional), you can donate a cup of coffee via the following:
Venmo: venmo.com/ariseff
PayPal: www.paypal.me/ariseff - Наука та технологія
Love the 3blue1brown esque style. Keep em coming
Thank you! Working on it :)
Appreciate both of you guys work in this field!
It's called manim, sir.
3blue1brown has become a classic in UA-cam videos. We would describe arts as Escheresque or poetry as shakespearean but when a streamer defines a style that is compared to as a subgenre is some remarkable of an achievement by 3blue1brown
Wish more videos were coming
I think I speak for everyone when I say.. please keep posting! This is excellent stuff!
Thank you for the clear and concise explanation! I understood this more than any other video I've seen yet (although I'm still learning). Looking forward to more videos like this!
i thought i was smart - but then i started getting my head around all this amazing machine learning and then i was humbled. thanks for sharing
I put 2 months into studying this. I’m a Cultural Science major and I can finally say that I understood every part of this video.
I’m really proud of myself and very thankful for your excellent didactic style.
Exceptionally explained! Please make more content! This stuff is worth paying for ;)
These are great! I can see this channel easily becoming as popular as Yannic Kilcher’s. Thank you for all your work, your explanations give a lot of clarity without sacrificing depth!
Hi Ari, Seriously great stuff! Really appreciate the videos.
Thanks for the nice content. I am not a beginner, but unfortunately it was hard for me to follow this video. There were many concepts/terms mentioned without a brief explanation, and the pace was rather fast. If you could publish the same video, with additional examples and clarifications would be much appreciated. I understand that one would need to look up some topics and references while watching the video, but in this case it felt like I have to look up things very often. Thanks again for your effort!
Absolutely. It started well then went off the rails.
you spoke for all of us, thank you for voicing our opinions. i'm a graduate data science alumni and i'm very familiar with deep learning. Watching this video, i wasn't learning anything rather i couldn't even follow-up with the video. I started litterally questioning my languistic and academic abilities. Then i realized that this guy is just summarizing the dreaded research paper "Attention is all you need" which is a bad idea if you're not going to cover explanation of terminologies and equations.
@@UpperM3 not really, to learn this concept it is necessary to at least have a basic idea of the concepts related to it. I think this is the best explaination of transformers in youTuble yet
@@UpperM3 this is more than data science, it's computer science. You need to have some understanding of comp sci concepts - parallelization, graphs, vectors, encodings, decodings. If you are "very familiar" with deep learning, you should know all this. But, yes this explanation is not geared towards beginners to deep learning/computer science. You need to understand neural networks, representations, communication paths, lengths etc. so the definitions are verbose at times. And you can always pause, and google the terms for more examples. This is technical and math jargon heavy, but yeah, it did not say it was explained for beginners. But, we can all spend some time to become more "expert". I suggest you to take notes, and fill in the gaps with Google.
For e.g. " They leverage what is called self-attention, to compute updated representations for each sequence in parallel. The attention mechanism is going to allow each representation to kind of differentially consider the representation in every other position. And the communication paths will have the same length for all pairs of elements"
@@UpperM3 there are also numerous videos that explain transformers without getting too technical -ua-cam.com/video/SZorAJ4I-sA/v-deo.html
The best help in understanding transformers while reading „Attention is all you need“ I have found.
I want to thank you for your effort on making these videos. I think your channel deserve more attention.
I watched a bunch of videos on this and I feel like after this, I actually understand it. Thank you!
This channel is an incredible gem, thanks so much for your work!
You did such a good job explaining a not very simple topic, thanks!
Thank YOU! this is the most precise and straight forward thing about Attention that I ever have. When you said the word compatibility, I finally understood why do I need to take dot product between Queries and Keys.
Our community is the best!!! 💪💪 Thank you very much for the amazing review!!!
Transformers are incredibly complicated but you tried to simplify it for us any way it does need another look
Thank you very much indeed
Maybe the best transformer explanation out there
I really like your videos. Please keep uploading !
Dude I was looking for a channel like yours for weeks !
incredible, insightful, delightful, enlightening
Thank you very much for sharing this. Wonderful work!
Outstanding high-quality video. Thanks!
best tutorial about transformer on youtube by far 🙌🏻
Really Attention is all we need.
best explanation I've found! great job
Much needed video, if you have time I would request you to make a video series on transformers. My personal request would be on "Do Transformer Modifications Transfer Across Implementations and Applications?" and "Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth - "
Again, I thank you for your work!
Very well illustrated! Thanks
Great explanation ! Thank you for making this video !
Thanks ! Beautiful work.
I really liked your video, I finally can say I'm starting to get how it works, thank you !
Thanks for the vid. This is exactly level of the details and explanation style I needed. Many other explanations are either too vague and miss important details or too hardcore and hard to follow. This is ideal: most of the technical details are here, but still easy to follow.
Excellent explanation! Thank you!!
Thanks Ari! Very useful.
This video is a mix of actual explanation about the nature of transformers and long tangents about implementation details that are mixed together so perfectly it ensures that it's impossible follow or even know if the author of this video understands how transformers work.
The attention part gave me the most headaches I almost cried, felt so stupid. I found a better video in AssemblyAI that explains it better if that's the same case for you about the attention part.
Very well explained. Nice video!
i am at 1/3rd of the video, stopped to come here express gratitude before getting back to it! 3B1B is a proven style, no harm in using it, later on adding to it.
Thanks for a wonderful explanation.
I appreciate the effort to put this together! 🤩 It is really beautiful! 😍
Unfortunately, the original formulation, equations, and diagram for the transformer are rather cryptic 😞
Thanks Alfredo! Means a lot. I hope people can gain some clarity here.
Outstanding explanation!
Fantastic voice and description. Keep posting
this is amazing, well done!!
Amazing video.
Please make more content!
Thanks for the video!
Love the style
Great explanation
Just awesome!
great content!
Good stuff!
Very good
Thanks!
I also feel the residual connection is definitely RNN inspired.
Good! Liked & Subscribed
great explanation but when you were explaining the self-attention mechanism before multi-head attention, you forgot to mention that they way you get Q, K, V is by multiplying the input X by the corresponding weight matrix WQ, WK, or WV. When you said that "q, k, v are all learned" it made it sound like we're somehow learning q, k, v directly where as in fact we are learning the weight matrices that help generate them
I can't thank you enough. :)
Great video! Do you know if layer normalization is performed exclusively along the features axis, or if it's done both along the feature axis and the tokens (words) axis? Different sources say different things :( thank you
Great explanation of the Transformer. Thank you!
Hi, thanks for the video! There are several things that are still unclear to me. First I do not understand well how the architecture is dynamic with respect to the size of the input. I mean what does change structurally when we change the size of the input, are there some inner parts that should be parallely repeated? or does this architecture fix a size of max window that we hope will be larger than any sequence input?
The other question is the most important one, it seems every explanation of transformer architecture I have found so far focuses one what we WANT a self attention or attention layer to do but never say a word of WHY after training those attention layers will do, by emergence, what we expect them to do. I guess it has something to do with the chosen structure of data in input and output of those layers, as well as the data flow which is forced but I do not have yet the revelation.
If you could help me with those, that would be great!
great video imagery... good job. Love seeing the flow of information, and little opinions about why things are happening. got a bit confusing when it went away from images to just pure equations, could do images and boxes with the equations?
Thanks for putting lot of efforts in making this video. I know how much hard work it takes to make such kind of videos. However, I would recommend as a viewer to explain with an example from the begining as it is really hard to understand with standard notations. I was really not able to follow after a while.
Badass
Definitely, UA-cam should review their suggestion algorithm. What do you think?
oh my brother, the rabbit hole you have sent me down.......... I'm very content . Send food from time to time please. Thank you for this video btw. I've had it on repeat for about a day. Well, ya-know, I ain't that smart. I did get CUDA installed and working in python though so that a win. Have a good day!
Around 7:20 you say Q, K, and V are all identical. What exactly do you mean? If we have separate matrices that generate these for each head (with the i indexing you mentioned), then doesn't that mean we will have h different WQ matrices, meaning we have h different Q vectors for each word (i.e. h different Q matrices for the sequence of words)?
This part could've been a little clearer. What I think can cause some confusion is that the letters Q, K and V are used in two different ways:
1) in the signature of the original Attention function (6:05)
2) in the signature of the MultiHead function (7:20)
You're exactly right that we will have h different projected Q vectors for each word (same with K and V). What I'm referring to is that, for the encoder, the three arguments of the function MultiHead(Q, K, V) are all identical. The Q argument of MultiHead in layer i+1 of the encoder will be the stacked word representations output by layer i. This will also be the case for K and V, thus Q=K=V. In cross-attention, this is not the case: Q comes from the previous decoder layer while K and V are produced by the encoder.
When we're doing multi-head attention, the three inputs to the original Attention function are *not* identical to each other, regardless of if we're doing self- or cross-attention. These inputs are produced by applying linear transformations to the arguments of MultiHead via the weight matrices.
Great video! But I stiil can't understand it thoroughly..... What prerequisite knowledge do i need? Thanks!🙏
hi. a small doubt: when you say the dot product of Query(Q) and Key vector(k), do you mean the "element-wise multiplication of each elements of Q & K followed by the sum of all these element-wise products such the resultant is only a single number" or simply "element-wise multiplication of each elements of Q & K, such that resultant is still a vector of the same dimension as Q&K"?
Why do we sum the positional and input embeddings? Wouldnt concatenating make more sense? How would that play with dimensions?
What is the meaning of fine-tuning and Pre-trained in Transformers?
Me Brain be melted in a good way .
2:45
Let's take a moment to note: Dark Reader rules! :D
During training you can feed your input encoder and decoder. What happens during prediction with new data? You don't know the output yet, How do you feed the decoder?
What animation software did you use?
This one used a combination of matplotlib, keynote, and FCP. I've also used manim in a couple videos.
great Video! But I start to lost from queries key values lol
Here when you use "sequence", you mean a sentence or the whole text ?
I have question about this fragment 06:58. Im not sure but I suppose that R dimensionality schould be d_k*D not D*d_k
So the transformers came back to their home planet actually :)
how decoder match the size of output of the encoder with his input in every step to could make the multiplication of the matrixes
The transformer architecture diagram resembles one of Jordan Peterson's book illustrations. Except the transformer is actually coherent and useful.
Do you have a Twitter? I always share excellent videos I find there, and would like to give you credit?
@ari_seff :)
8:35 I don't understand what is _pos_ and _i_ here. Here pos is the position in the sequence (0 for the first word, ...) but what is i ?
It looks like 256 and higher values of i in the positional encoding all go to 1 no matter the position value. So half the elements of the word embedding are wasted
9:51
This seems to be an error.
The posicional embedding is concatenated to the word embedding.
It is not a sum.
Also, the posicional embedding may have an arbitrary dimension.
EDIT: I was wrong
From the paper: "The positional encodings have the same dimension d_model as the embeddings, so that the two can be summed."
i.imgur.com/pJU5O9n.png
@@ariseffai Oh you are right! Thank you very much.
I am supprise I didn't realize this before. I have to change my code haha.
It seems weird to me because if you sum the positional to the word embedding, you are changing the information about the meaning of that particular dimension of the word embedding. I mean: Let's suppouse that the first number of the word embedding is 0.5. Let's suppose that that dimension has a meaning about the happiness of the word. That means that the word has a level of happiness of 0.5. If you add the positional embbeding you are changing that meaning of the word. You would mixing the information about happiness with some positional information. And yoy are doing this to EACH dimension of the word embedding. This seems messy to me. If you concatenate instaed of sum, this does not happen.
But okay, I really don't know anything.
13:06 is incorrect. The queries and keys come from encoder and the values come from previous attention block.
Authors write in the paper: "In "encoder-decoder attention" layers, the queries come from the previous decoder layer, and the memory keys and values come from the output of the encoder. This allows every position in the decoder to attend over all positions in the input sequence. This mimics the typical encoder-decoder attention mechanisms in sequence-to-sequence models". Of course, you're free to try out alternatives, but the video is not incorrect.
algorithm, more!
Did everybody get that? 😅
If this is for a university lecture, then this would be marvelous, however, most audiences are laymen, the explanation obfuscates the ease of understandig.
I’m going to guess that they are robots in disguise?
Someone needs to make an intuitive introduction to COMPLETE beginners, those who have no experience in AI, math, linear algebra, etc. etc. Almost all 'simple' tutorials I've come across never really give an intuitive explanation for the details, just a lot of jargon and math.
Completly agree with you. The most important is the intuition behind. Not how things are calculated.
I'm not that novice in AI but i confess i understood very little here. For instance, how come we're using sin and cos to encode a position?
We know the properties of those functions. So there's no point recalling them.
What is the intuition behind keys, values and queries?
We know how to project vectors and how to deal with linear transformations.
And last, we're trying to translate a sentence. How come the input of the decoder is the target sentence we're trying to obtain?
Those are the things a video should try to clarify. Otherwise, there's no value added.
It is not clear from the visualization what you are talking about. A lot of the words are not defined and the diagrams are missing a lot of stuff.
I like the animation style and the way you talk, but it was still really hard to follow. I did not understand much even though I study CS in a master
Almost none of the terms and jargon were explained. People who need this video won't understand it, people who don't won't click.
We turn off the music when we talk
This is probaby a good video...
...
...
...
Recorded with the
---
Pensive pauses trademarked by 3 blue 1 brown.
---
---
People say you're supposed to think during them
---
---
But I think while the person is talking
---
___
__
And so I have all of this time
...
...
..
Between the points the video makes
---
---
---
Alone and with no new info.
---
---
While everyone else thinks.
----
---
Teacher, is there a version of this that's already complete, where I could read it long-form without waiitng for others to think?
This is introduction to almost everything, I can't catch up or understand everything. Maybe this is for someone who understand them all already
what a disaster this transformers are. they did not know what they were doing ha ha ha
why does this video have back ground music?
I lost my attention half way to
Too much jargon without explanation mate
Whyyyyyy the annoying piano!!!!!
Its pretty obvious that you have 3Blue1Brown as source of aesthetics, down to the intonation. I do think you can at least make your own soundtrack choice, feels a bit too disingenuous and off putting to be this blatant. Somehow uncanny valley vibes. Having that said, a unique voice on youtube and really good content.