The Narrated Transformer Language Model
Вставка
- Опубліковано 15 тра 2024
- AI/ML has been witnessing a rapid acceleration in model improvement in the last few years. The majority of the state-of-the-art models in the field are based on the Transformer architecture. Examples include models like BERT (which when applied to Google Search, resulted in what Google calls "one of the biggest leaps forward in the history of Search") and OpenAI's GPT2 and GPT3 (which are able to generate coherent text and essays).
This video by the author of the popular "Illustrated Transformer" guide will introduce the Transformer architecture and its various applications. This is a visual presentation accessible to people with various levels of ML experience.
Intro (0:00)
The Architecture of the Transformer (4:18)
Model Training (7:11)
Transformer LM Component 1: FFNN (10:01)
Transformer LM Component 2: Self-Attention(12:27)
Tokenization: Words to Token Ids (14:59)
Embedding: Breathe meaning into tokens (19:42)
Projecting the Output: Turning Computation into Language (24:11)
Final Note: Visualizing Probabilities (25:51)
The Illustrated Transformer:
jalammar.github.io/illustrate...
Simple transformer language model notebook:
github.com/jalammar/jalammar....
Philosophers On GPT-3 (updated with replies by GPT-3):
dailynous.com/2020/07/30/phil...
-----
Twitter: / jayalammar
Blog: jalammar.github.io/
Mailing List: jayalammar.substack.com/
More videos by Jay:
Jay's Visual Intro to AI
• Jay's Visual Intro to AI
How GPT-3 Works - Easily Explained with Animations
• How GPT3 Works - Easil...
Your blog on Illustrated Transformer was my intro to Deep Learning with NLP. Thanks for the amazing contributions for the community.
Yeah it is being referenced in my DL class too. Truly great content for new learners!
The Illustrated Transformer blog is a masterpiece!
Dear Teacher Alammar , thanks to this video I was able to accepted into BYU lab as an external researcher (even though I didn’t finish college) and have been invited by my professor to participate with the lab in CASP15 . You really changed the course of my life by demystifying such complex topics for non traditional learners like me . I’m eternally in your debt
Your ability to explain and breakdown complex topics into simpler and intuitive sections is legendary. Thank you for your contribution!
I’ve just read your “The illustrated transformer” article and I wanted to say that you made very smart and simple visual representations. It seems you put a lot of thought into that.
I remember Seeing your Transformer's Blog Jay.. It was legendary!! Was referred to by other youtubers as well... And thanks a lot for the wonderful explanation as well!
One of the most comprehensive video and blog overviews of Transformers I've seen. Thank you. 🙏
Thank you so much for all the tireless work you do for us visual learners out there! I’m looking forward to videos where you get into your excellent visualizations of the underlying matrix operations. Your visual abstractions both at the flow chart level and matrix/vector level have really shaped my mental model for what I think about when I’m engineering models. I’m so grateful and so excited to see what you come out with next (this library you hint at looks wonderful!)
Thanks Jack!
I haven't see such a clear explanation of Transformers and Decoder LM Models, Amazing Work Jay
Outstanding job demystifying the inner working details of the Transformer model architecture! All the illustrations and animations for the inference working are awesome. Thank you for taking all the time and sharing your understanding with all of us. Kudos! 👍
A huge thank you for this explanation!
Dude I freakin love your blog, keep up with the good work! Thanks for everything!
Thanks, your Blog is so clear!
You sir are an amazing teacher! I'm absolutely flabbergasted by how well you've explained, to think its all mathematics at the end of the day! Thank you for taking the time to put together such a concise yet complete guide to transformers!
2024, still a great reference to Transformers. Million thanks for the amazing work!
you have a gift for explaining complex materials... many other technical talks assumes the audience is very knowledgeable and are attending the session just for networking
27:56 - this explains a lot, thank you so much!
Amazing illustration. Keep going Jay.
Jay, many thanks for your work. These videos help me a lot to understand key concepts in NLP domain through visualization.
Thank you for sharing wonderful insight!
Thanks for creating this content. Your explanation is quite easy to follow, especially for someone like me who is just beginning to explore these areas of AI/ML.
Great explanation! Please keep doing this format.
Maybe the best video on this subject.
It would nice to have a step by step walkthrough of the training process. And why each of those steps makes sense intuitively.
Really enjoyed your blog post and video, super clear - thank you very much for this amazing resource :)
1 minute into the video and I already subscribed.
Just a personal comment on the format of the videos: I, personally, find that constant change of scene (like in "The architecture of the transformer" section) where the camera changes constantly showing you and then showing the computer screen and then back to you, is extremely annoying.
The content of the video itself was informative.
Thanks for the very clear and concise explanation, Jay!
Thank you so much for the clear and concise explanation. Keep it up the great work.
Thank you so much for you work on attention and transformers. Your posts and videos are the best i have encountered so far in terms of visualization and explanation. And you did it way better than my Professor. Again thank you :)
Thank you for this awesome introduction!
Really clear. amazing job!
Never been more excited by a UA-camr channel than when I saw this guy had a channel.
This is the best video I have seen by far in this domain. You strike a perfect balance in assuming the level of understanding of audience :)
Awesome! Glad you found it useful!
Thank you so much for your work ! The illustration help to clearly understand these models !!
Your blog was referred to me by my lecture Julia Kreutzer of Google Translate, it's just amazing piece of work. It has really helped me in my understanding of these concepts. Thanks.
I found this very helpful visual explainer, thanks so much for your time, and thanks for chopping it up into sections for easy revision 🤓!
Amazing explanation, my search to understand the transformers ended here, you done the wonderful job, thank you so much for the simplest explanation I ever seen.
Great video Jay, thank you so much!
A phenomenal extension of your blog post. Commenting for that bump in the recommendation algorithm!
Thank you! Much appreciated!
loved it. thanks. got some new neurons in my head created by this video.
Thank you for this great explanation. Visualize , visualize, visualize, the best way to undestand how it works.
I really appreciate your explanation about this topic. One more time, I check that DL is my new passion. Thanks a lot.
Amazing video. Thank you very much for making this topic accessible.
Your are really good (excellent) at explaining a complex topic in a simple way. Congratulations !!!!
Thank you for writing the blog. It has helped me .
Wow! One of THE best explanation of Transformers.. Thanks @Jay!!
Impressive. Thank you.
Jay, recentemente estive em um curso de I.A, Mas voce apresentou muito bem, de forma didática a PNL.... eu aprendi muito com voce.
Obrigado. Continue sendo este cara maravilhoso.
Definitely it is easier to understand in a vertical way. Thanks for everything!
Amazing!
Thank you for share!
The best explanation of the Transformer and GPT model !!
Amazinnnng illustration of language model transformers
I've ended up here to familiarize myself with NLP transformers. Your video was the optimal choice for me, as it' explains the concept in an understandable scientific manner. Thanks.
Wow! 🎉 Awesome into.
Great master piece explanation of NLP in real life scenario. Thank you
Jay, as a PhD student, I'm a fan of your ability to explain complex topics, in a very simple, illustrated and didactic way! I always recommend your ' illustrated' posts to my colleagues. Thanks again for this great video, keep up the good work!
Thanks Diogo!
Which university?
I see Miles Davis vinyl, kind of blue. Awesome album, and thanks for the video!
Awesome stuff. your blog really helped clarify my deep learning class.
Fantastic teacher. Thanks Jay!
I just found this now. it's super. thanks
This is really great! Highly recommend!
Great explanations!
One of the best videos of the subject
Great explanation
Great video! Best regards from Brazil!
great explanation! also love all the pop culture references in your room :p
Awesome content! thank you!
absolutely amazing video
I appreciate your detailed explanation, Mr. Jay. My first reaction was to read your article (The Illustrated Transformer) after watching the video. My question relates to the Transformer architecture, which consists of six encoders and six decoders layers, all of which seem to be very similar. What is the purpose of the six layers? since a sentence will be checked for relevant information in every word from the first layer using Self-Attention. In addition, Attention is used to boost training speed, so will these six layers slow it down?
I like the way you are teaching! !!
You are a great teacher!!! If you chek the EQ settings and lower the music at the beginning the video is perfect!!! Thanks a lot for sharing your knowledge in this very understandable way
Amazing work indeed thanks for simplifying things for everyone to understand this AI great work
Nice collection of albuns man! Miles Davis, Radiohead, John Coltrane, very classy! 👏👏👏
Spot on observation, kind of ironic to be listening to Ok Computer and teaching about artificial intelligence :D
Omg, thanks lot for these amazing videos. Your lectures and blogs are so easy to understand.
Small request, please pin the BGM you used in the video
loved the music behind ..
14:15 - so, the Self-Attention layer is actually the thing that’s trying to understand the meaning of the whole sequence? How does it work and how can it be trained? How long sequenced can it analyze?
Im doing a Twitter sentiment analysis and i couldn't wrap my head around BERT and i came across this video. Perfectly explained. Thanks alot
omg, man. I've found several posters of really good music in your room! I defenetely need your TOP-10 albums list!!))
Haha! I'm mainly displaying some of my favorites!
Watching it now, thanks so much! It's really helpful to go through these kinds of things with clear examples and explanations.
My only preference would've been to reduce the volume of the background music in the intro. So many podcasts do this and it's an annoying trend!
Thanks Neil! Noted on the audio!
Great video, thank you!
This video really aged well. It came out just after GPT3 and before ChatGPT. I love it how it gives massive insights to how current generative AI works behind the scenes (but obviously in a simplified way).
Thank you very much! this is awesome and easy to understand.
Thanks, I learn a lot!
Great video!
❤️ That library!!!!
It's been my entire focus the last few months. Stay tuned!
Great work 👍👍👍
Thanks, very intuitive…
Thank you !
i don't khnow how must say thank you, I just can say please continue uploading your amazing videos. I live in a constrained country and this video is my only hope for learning like other peoples. yours sincerely.
Ramin Bakhtiyari.
Great content! can't wait for more.
Thank you Yuchen!
Thanks for the explanation. Good music taste at the background by the way👍
Thank you!
I am trying to understand working of transformer, you explain it much accessible way. One small thing I wish the video had less of transitions between two cameras.
Thank you.
Thank you
Thanks for the great explanation. MLP (at 11:35) stands for multilayer perceptron :)
Amazing.
hey Jay! love the blog on illustrated transformer, do you also have a reference to your blog on vision transformers?
Thank you for your videos and blog posts. These were my inspiration to create a Java GPT-2 implementation for learning purposes. I can't use a link here, but as huplay I uploaded it to the biggest hosting site, and it is called gpt2-demo.
great dear