Positional embeddings in transformers EXPLAINED | Demystifying positional encodings.
Вставка
- Опубліковано 5 чер 2024
- What are positional embeddings / encodings?
📺 Follow-up video: Concatenate or add positional encodings? Learned positional embeddings. • Adding vs. concatenati...
➡️ AI Coffee Break Merch! 🛍️ aicoffeebreak.creator-spring....
► Outline:
00:00 What are positional embeddings?
03:39 Requirements for positional embeddings
04:23 Sines, cosines explained: The original solution from the “Attention is all you need” paper
📺 Transformer explained: • The Transformer neural...
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
NEW (channel update):
🔥 Optionally, pay us a coffee to boost our Coffee Bean production! ☕
Patreon: / aicoffeebreak
Ko-fi: ko-fi.com/aicoffeebreak
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
Paper 📄
Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." In Advances in neural information processing systems, pp. 5998-6008. 2017. proceedings.neurips.cc/paper/...
✍️ Arabic Subtitles by Ali Haidar Ahmad / ali-ahmad-0706a51bb .
Music 🎵 :
Discovery Hit by Kevin MacLeod is licensed under a Creative Commons Attribution 4.0 licence. creativecommons.org/licenses/...
Source: incompetech.com/music/royalty-...
Artist: incompetech.com/
---------------------------
🔗 Links:
AICoffeeBreakQuiz: / aicoffeebreak
Twitter: / aicoffeebreak
Reddit: / aicoffeebreak
UA-cam: / aicoffeebreak
#AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research
I found no explanation for this anywhere and when reading the paper missed the detail that each tokens positional encoding consists of multiple values (calculated by different sine functions). Your explanation and visual representation finally made me understand! Fourier transforms are genius and I'm amazed in how many different areas they show up.
love how the "Attention is all you need" paper appears with an epic-like bgm
It wasn't on purpose, but it is funny -- in hindsight 😅🤣
This video is a clear explaination of why you shouldn't add your positional encoding but concat.
Extra dimensions dedicated exclusively to encode position! Sure, but only if you have some extra to share. 😅
@@AICoffeeBreak this method relocates the embeddings in a specific direction in the embeddings space, so that new position in the relevant embedding cluster have "another" meaning to (say there is another instance of the same word later) other words of "same kind". But that place should be reserved other semantics, else the space is literally filled with "second position" coffee and "tenth position" me, "third position" good etc etc. This can go wrong in soooo many ways. Don't get me wrong, I am a clear cut "Chinese Room Experiment" guy, I don't think you can translate "he is a good doctor" before imagining an iconic low resolution male doctor and recall a memory of satisfaction and admiration of consumatory reward, but again, the "he" in "he did again" and "man, he did it again" should literally have the same representation in the network to start discussing things.
You are entirely right. I was short in my comment because I commented on the same issue in Cristian Garcia's comment. But there is no way you would have seen it, so I will copy paste it here: 😅
"Concatenating has the luxury of extra, exclusive dimensions dedicated to positional encoding with the upside of avoiding mixing up semantic and positional information. The downside is, you can afford those extra dimensions only if you have capacity to spare.
So adding the positional embeddings to initial vector representations saves some capacity by using it for both semantic and positional information, but with the danger of mixing these up if there is no careful tuning on this (for tuning, think about the division by 10000 in the sine formula in "attention is all you need")."
And you correctly read between the lines, because this was not explicitly mentioned in the video. In the video I explained what an act of balance it is between semantic and positional information, but you identified the solution: If adding them up causes such trouble, then... let's don't! 😂
@@AICoffeeBreak new to AI, but what do you mean by the word "capacity"? Do you mean RAM? Do you mean that if we concat positional encodings to original vector instead of adding, it will take up more RAM/memory and therefore make the training process slow?
+1 for more vids on positional encodings.
This is the most intuitive explanation of the positional encoding I have come across. Everyone out there explain how to do it, even with code, but not the why which is more important.
Keep this up. You have earned my subscription.
Wow, thanks a lot! :)
"A multi-dimensional, spurious correlation identifying beast..." ... wow. Douglas Adams would be proud of that.
Hope so. 😁
I had my morning coffee with this and will make an habit!
What a cool comment! Thanks.
Multi Dimensional Spurious Corelation Identifying Beast 🔥🔥
That's what I am calling transformers from now on
Great explanation !! And Yes Yes Yes.
Letitia's Explanation Is All You Need!
And comments like these!
+1
+ 1 for video on relative positional representations!
i've read numerous articles explaining the positional embedding so far.. however, it is surely the greatest & clearest ever
Thanks a lot! Glad we could help. :)
Dang it, i learned something and my morning coffee isn't even finished.
Its going to be one of *those* days.
Sound like a good day to me! 😅
Whish you a fruitful day!
Nice explanation
Thanks for appreciating!
You've solid understanding of Mathematics of Signal Processing
Very intuitive. I know there is sine cosine positional encoding but it is actually effective that I got it here..👍👍
Bro
Where have you been hiding all this time?
This is next level explaining
Make Ms. Coffee Bean spill the beans on positional embeddings for images
😂
great explanation :)
Thanks! 😁
Amazing! This is the best explanation for positional encodings period. Subscribed!!
Thank you!
Probably the clearest explanation for positional encoding:D
Thanks!
Thanks for the explanation
Interested to know about rotary position embedding
Just watched this again for a refresher; thee best video out there on the subject!
Wow, thanks!
BEST AI channel on UA-cam!!!!!
I was browsing through some channels after first stopping on Sean Cannells and I noticed your channel. You got a great little channel building up here. I decided to drop by and show some support. Keep up the great content and I hope you keep posting :)
Thanks for passing by and for the comment! I appreciate it!
Great explanation - It might be the best. I think I finally figured it out. I highly appreciate it.
Great stuff :) Would love to see more of that, especially for images or geometry!
Noted!
This explanation is incredible
Super intuitive explanation, nice!
Great explanation of the intuition of positional encodings used in the Transformer!
great explanation of positional embeddings. Just what I need.
the most brilliant positional encoding explanation EVER that was GOD Level explanation
Haha, thanks.
The best explanation of how exactly position embeddings work !
I finally understand the positinal encoding! Thanks
Happy to help. :)
i watched bunch of videos on the positional embedding , but this video makes me very clear intuition behind the positional embedding , thank you very much for this great video , 😊
Thanks, that's great to hear! ☺️
Explained really well, thank you 😊
Thanks, so nice of you!
This is probably the best explanation of this topic on UA-cam! Great work!
Thanks so much! 😊
What a great video, I'm studying my Ph.D. at NLU, so this came in very handy. Thank you very much and greetings from Ensenada Baja California Mexico :D!
Thanks, thanks for visiting from so far away! Greetings from Heidelberg Germany! 👋
Amazing content. After seeing this all the articles and research papers makes sense.
I loved your simple and explicit explanation. You've earned a sub and like!
Thanks, see you again soon!
Thanks, as always, for the great explanation!
It was Ms. Coffee Bean's pleasure! 😅
amazing explanation!!! I have been looking for a clear explanation on how the positional encodings actually work and this really helped! thank you :)
This was very intuitive, thank you very much! Needed this, please keep making videos
Great explanation - thanks very much for sharing this.
Glad you enjoyed it!
Brilliant video. This was the best explanation of positional encodings I have seen. It helped a TON!!!
I also make AI videos. I am more into the image side(convolutions and pooling) so it was great to see more AI educators.
Great! It took me forever, before I had found your videos, but finally I understand it. Thank you soooo much!
You're very welcome!
Sooo good!
This was so insightful!
excellent explanation!
This is a great video! Just found your channel and glad I did, instantly subscribed :)
Thank you so much, you honour us!
Excellent! Great work!
Super clear and amazing (as always) explanation of sines and cosines positional embeddings! 🙌
Thanks! Always happy when you visit!
I feel lucky to have found your channel. Simply amazing ❤️
Happy to have you here!
Thank you. One thing is to know the formula and applying it and other thing is to understand the intuition behind it. You made it very clear. All the best
Well said! -- Humbled to realize this was put in context with our video, thanks.
Thanks for watching!
Gemini: This video is about positional embeddings in transformers.
The video starts with an explanation of why positional embeddings are important. Transformers are a type of neural network that has become very popular for machine learning tasks, especially when there is a lot of data to train on. However, transformers do not process information in the order that it is given. This can be a problem for tasks where the order of the data is important, such as language translation. Positional embeddings are a way of adding information about the order of the data to the transformer.
The video then goes on to explain how positional embeddings work. Positional embeddings are vectors that are added to the input vectors of the transformer. These vectors encode the position of each element in the sequence. The way that positional embeddings are created is important. The embeddings need to be unique for each position, but they also need to be small enough that they do not overwhelm the signal from the original data.
The video concludes by discussing some of the different ways that positional embeddings can be created. The most common way is to use sine and cosine functions. These functions can be used to create embeddings that are both unique and small. The video also mentions that there are other ways to create positional embeddings, and that these methods may be more appropriate for some types of data.░
In another video I've seen, apparently it doesn't matter if positional embedding are learned or static. It seems as thiugh the rest of the model makes accurate deductions regardless.
This is why I was not surprised that Fourier transforms seem to work nearly as well as self attention.
COuld you please elaborate using an example maybe?
Thanks Letitia, great explanation
Hi Letitia. Thank you so much for your wonderful video! Your explanations are more intuitive than almost anything else I've seen on the internet. Could you also do a video on how positional encoding works for images, specifically? I assume they are different from text because images do not have the sequential pattern text data have. Thanks!
Thanks for the suggestion. I do not think I will come to do this in the next few months. But the idea of image position embeddings is that those representations are most often learned. The gist of it is to divide the image into patches, let's say 9. And then to number them from 1 to 9 (from the top-left to bottom right). Then let gradient descent learn better representations of these addresses.
amazing stuff! so clear and intuitive, exactly what I was looking for :)
Thanks for watching and appreciating! 😊
Lifesaver! Thank you for the explanation.
Very well explained! Thank you so much!
Great content! Can you do a video about Tabnet please? :)
Nice explanation! I’d love to hear more about multidimensional and learned position encodings
Noted!
These videos are priceless, thank you!
Your explanation was great! I got everything I wanted to know about positional embedding. thank you:)
Awesome, thanks for the visit! ☺️
Awesome. Thank you for making this great explanation. I highly appreciate it.
Hey, Great Explanation :). Love to see more videos.
Glad you found us! 😁
Thank you for the explanation! Can you please make a video on Positional information in Gated Positional Self Attention in ConViT paper?
Noted!
Great work! Clear and deep explanation!
Very good explanation!! Thanks for this 🙏🙏
Thank You for your wonderful message!
So helpful! Thank you a lot!
Really Thank you! Your video is just amazing!
Thanks Letitia! A vid on relative positional embeddings would be nice 😃
Implementations seems a bit involved so I've never used them in my toy examples.
Regarding this topic, I've seen positional embeddings sometimes being added and sometimes being concatenated with no real justification for either 😐
Concatenating has the luxury of extra, exclusive dimensions dedicated to positional encoding with the upside of avoiding mixing up semantic and positional information. The downside is, you can have those extra dimensions only if you have capacity to spare.
So adding the positional embeddings to initial vector representations saves some capacity by using it for both semantic and positional information with the danger of mixing these up if there is no careful tuning on this (for tuning, think about the division by 10000 in the sine formula in "attention is all you need").
Excellent explanation. Will certainly like to learn about other encodings in areas like image processing.
Great video, many thanks!
Amazing explanation! Thank you so much !
this helped me so much!! thank you!!!
best video about position emb i have ever heard
Thanks!
love your explanations !!!
🤗
Just an amazing explanation ...
Thanks, this was so clear! Finally understood position embeddings!
Fantastic!
Thanks, your videos helped the most
great explanation. really loved it. Thank you
Wow, thanks so much! This is so well explained!!
Glad it helped. :)
I really enjoy your videos 😄!
I would like a video about using transformers with time-series data.
Noted!
Loved your explanation, thank you for this video!
Glad you like it!
What a lovely explanation & video!.. Thank you!
Glad you enjoyed it! Thanks for the visit and leaving a comment.
@@AICoffeeBreak Thank you again!.. subscribed!!
@@bingochipspass08 Oh, great, then I'll see you on future videos as well.
This was awesome! I don't have a complete understanding but it definitely pushed me to the side of understanding. Did you make a video about relative positions?
Yes, I did! ua-cam.com/video/DwaBQbqh5aE/v-deo.html
so helpful, appreciate it!
Really great!
Best explanation I've seen - good work!
Thank you so much! 🫨
Cool explanation
The positional encodings are not that weird when you think about them like the hands on a clock. They're a way of expressing a continuous, unlimited sequence, at multiple levels of precision, within a confined space. The tips of clock hands also trace out a sin and cosine pattern at various frequencies.
Beautiful explanation
Glad you think so! :)
Good explanation.
Awesome, thanks!
This is an amazing explanation! Tku!!!🤓🥳🤩
Nice explanation!
Glad it was helpful!
Awesome video, thanks :)
amazing video - rockin!
🤘