Attention is all you need; Attentional Neural Network Models | Łukasz Kaiser | Masterclass
Вставка
- Опубліковано 3 жов 2017
- Łukasz Kaiser - Research Scientist at Google Brain - talks about attentional neural network models and the quick developments that have been made in this recent field. In his talk, he explains how such models "look at the past and generate the next word of the output" and how to train them. In a new talk, Lukasz introduces a new efficient variant of the Transformer. Enjoy the video pischool.link/transformer-update
If you are a brilliant post-graduate Machine Learning engineer and want to practise in real-world projects, apply for the Session 12 of the Pi School of AI starting on November 21 2022.
Ten free grants are available only for the most brilliants minds pischool.link/sailk
I love how excited he is.
Starting @ 15:45, in well under 2 minutes, attention explained! Only a true master can do it. Love.
K is a matrix representing the T previously seen words and V is the matrix representing the full dictionary of words of the target language, right? But what are K and V exactly? What values do these matrices hold? Are they learned?
The passion that he transmits is priceless
Nice explain about position and multihead ...
Nice explanation!!!!
Great talk, Łukasz.
Great presentation, he's having fun explaining the bits....great camera work- it was fun watching a moving cam than a boring still view.
very nice lecture ...enjoyed it lot.
"there is this guy, he never got his bachelor but he wrote most of these papers" - appreciation
Where experience and 'thinking outside the box' can beat education in some cases. He should be getting an 'honorary' bachelor degree, if he hasn't already.
Excuse me? that's fake news! Even his linked in profile says Duke 1998 (yes it's the same noam shazeer from this exact same paper)... "Noam Shazeer is an Engineer at Google. He graduated from Duke in 1998 with a double major in Mathematics and Computer Science"
I think he meant that Noam doesn't have a PhD. Noam does have a bachelors degree, but he started working at Google pretty soon after graduating (literally decades ago) and has contributed to many important Google technologies in his time there. Noam was a Google old-timer back when I started working there in 2005.
This is the best explanation of attention I have seen so far! And I have been looking :)
Bcoz, he is one of the co-author of the revolutionary paper which introduced it
@@ksrajavel revolutionary indeed. As per Google Scholar, the paper has over 72000 citations in just 5 years. One of the most cited papers in the history of academia.
Great lecture. Best explanation of attention in just a few words.
Nice lecture. I enjoyed it
Great talk. Something very useful.
Best expaination!!
Thank you !
Great content in this video. Would love if you had the multi-headed devil covered! Though, great video to get the overall view..
Here I get it.
Thank u
Can I somehow get the machine produced texts which where shown at the beginning of the presentation?
great explanation !
@27:14 multitasking
What model is that at the beginning? Can I somehow get the machine produced texts which where shown at the beginning of the presentation?
"
I just wish he hadn't stood right in front of what he was trying to show people, but I love his passion for explaining what he's talking about.
Thanks for sharing! It'd be great if the video could pay more attention to the slides though.
Thank you for your comment, Franck! You can download the slides here: picampus-school.com/open-day-2017-presentations-download/
@@pischool6210 perfect, thanks!
Thanks buddy
nothing beats GPT 2 TL;DR summarization trick
Are the talks he is referring to (as the previous talks) available on UA-cam?
Hello! Sure. You can find all the Masterclasses from our Open Day here 👉ua-cam.com/play/PLU3hjga27ZUiuL8V0CVlidBK27CDxWf-F.html
They should invent a device that can always tell the time of day when the user wants.
where is the library he talks about to get the details of training the DL "right"?
31:55
@@kvsnoufal is there one for pytorch?
Is there a way to get the slides?
You can download them here: picampus-school.com/open-day-2017-presentations-download/
Pi School I have just checked and it seems the slides linked to the presentation is "tensorflow workshop", can you please double-check?
Dear Vladimir, have a look here: drive.google.com/file/d/0B8BcJC1Y8XqobGNBYVpteDdFOWc/view
"He didn't put a trophy into the suitcase because it was too small." is an ambiguous statement. "it" could refer to either the trophy or the suitcase. It seems like the answer is mainly decided on probability from past experience, rather than the intended (ambiguous) meaning, similar to a survey or experiment with too small a sample size. It is also possible that he didn't want to put a too small a trophy into the suitcase in case it ended up being jostled about too much, and became damaged; although that is a less likely, but still a possible explanation and would need a thought process to come to that conclusion, or some further context, to clarify the intended meaning. People on the Autistic spectrum (HFA / Asperger's) have that same problem when phrasing thoughts (ambiguous meaning), and are often misunderstood because of it. When a statement has two (or more) possible meanings, then it's probably unfair to judge the performance of a system in 'getting the answer right' as there isn't a definite correct answer to begin with, just a more likely one.
A word for word translation, with grammatical correction applied would probably achieve a better result in a case like this. Google translate seems to somewhat agree.
Original: He didn't put a trophy into the suitcase because it was too small
Google translate: Er hat keine Trophäe in den Koffer gesteckt, weil er zu klein war.
Back to english: He did not put a trophy in his suitcase because he was too small.
Word for word translation (incorrect, but probably still understandable if you speak German): er nicht stellen ein Trophäe in das koffer da es was auch klein.
Google translate of word to word to english (much better but still wrong - where did the 'also' come from?):he does not put a trophy in the suitcase as it is also small.
does embedding gets trained or key or query or value gets trained i am confused. please help
math majors/ graduate math students skip to 15:36
ChatGPT gave me link to this video)
Is there any way i could get the ppts for reference?
Hi, sure! You can download it here: picampus-school.com/open-day-2017-presentations-download/
@@pischool6210 thankyou so much!!!!
Transformers learned translation without language dependency O_o
Slides?
drive.google.com/file/d/0B8BcJC1Y8XqobGNBYVpteDdFOWc/view
Tarea 6
47:55 "We tried it on images it didn't work so well". 2020, Visual Transformer: am I a joke to you?
In VIT, it is clearly stated that a "small dataset" like imagenet doesnt show promising results but a larger dataset like the jft gives amazing results, so this maybe a start, but it is far from perfection. Btw, I am not contradicting your statement. 😁. and also JFT is not an open source dataset(yet)
@@souhamghosh8714 True Google folks ^^
“Hi, I am from google, you know what i got, TPUs..more than you can imagine”😂
Strange thing, he mention "attention" term before explaining what it is. What was EXACT meaning of this Query Key Value magic ??? I suspect speakers just copy thoughts of another people mechanically, not understaning real meaning of operations !
the video image is too poor, you need to fix it more
good video but his and the camera placements are subobtimal
...comes from Google - Check. ...TensorFlow T-shirt - Check. Most viewers therefore rate this lecture highly - Check.
This is very hand-wavy throughout with relatively no rigor shown. There are many lectures/presentations online which actually explain the nuts and bolts and wider use cases of Attention mechanisms. Maybe the title of this video should be something else, like "Our group's success with one use case (language translation) of Attention." Frankly, the drive-by treatment of the technical details of language translation case was almost terrible and should have probably been omitted.
which lectures do you like that explain attention mechanisms and their wider use?
Most I gather from this talk is that "attention" is a pretty terrible term. Something like "fuzzy lookup" or "matching" or "mapping" would have been much more descriptive, but oh well, which researcher needs to think about terminology before unleashing it on the world.
can't believe this guy was one of the authors of Transformer. He just can not explain what he was doing!
I thought it was a really insightful talk. I'm preparing a talk about Transformer myself, and this talk was super inspiring :)
I also couldn't understand.. The architecture diagram is not very intuitive, and I was expecting some elaborations.. However, the explanation was not clear...
If you don’t already know something about attention then it’s impossible to follow the presentation. Everything is explained very poorly...
His communication skill are like a runner who is tripping over his shoelaces.
Unfortunately, it seems to be quite a common ailment of even "brilliant" coders (or shall I say, scientists) that they can't explain their ideas to others clearly using natural language. It's like they have no model of someone else's knowledge and take so many things for granted that their attempts at "explanation" just sound like gobbledygook to those who expect to be taught something. That's why we have technical writers, teachers, popular science books etc.
@Yancy Stevens Yes, to communicate you have to model in your head whoever you are communicating to, what they know, don't know, and foremost what they want to know. Otherwise it's just a fail, no matter how much knowledge you have.