Це відео не доступне.
Перепрошуємо.
Speculative Decoding: When Two LLMs are Faster than One
- Додати в
- Мій плейлист
- Переглянути пізніше
- Поділитися
Поділитися
Вставка
Розмір відео:
- Опубліковано 19 сер 2024
КОМЕНТАРІ • 42
Наступне
Автоматичне відтворення
Rotary Positional Embeddings: Combining Absolute and RelativeEfficient NLP
Переглядів 29 тис.
Quantization vs Pruning vs Distillation: Optimizing NNs for InferenceEfficient NLP
Переглядів 17 тис.
Why Does Diffusion Work Better than Auto-Regression?Algorithmic Simplicity
Переглядів 282 тис.
«Приїхали Бандеру шукали. Який Бандера? Він помер взагалі-то. Ні, от сказали, у вас тут Бандера є»Суспільне Дніпро
Переглядів 247 тис.
Як на КУРЩИНІ ПАДАВ АКВАФРЕШ РФТСН
Переглядів 633 тис.
Державний Прапор України підняли на КУРЩИНІ в рф😍🇺🇦💪🏻 #курськ #прапорукраїни #рф #війна #наступТелеканал Конкурент TV - новини Луцька та Волині
Переглядів 433 тис.
«Синку, ти не переживай. Сюди вона не впаде, вони всі повз летять»: військовий про перші дні війниСуспільне Запоріжжя
Переглядів 766 тис.
Speculative Decoding ExplainedTrelis Research
Переглядів 3,6 тис.
The KV Cache: Memory Usage in TransformersEfficient NLP
Переглядів 35 тис.
Beyond the Hype: A Realistic Look at Large Language Models • Jodie Burchell • GOTO 2024GOTO Conferences
Переглядів 86 тис.
Inside the LLM: Visualizing the Embeddings Layer of Mistral-7B and Gemma-2BChris Hay
Переглядів 6 тис.
Accept-Reject Sampling : Data Science Conceptsritvikmath
Переглядів 65 тис.
Rejection Sampling - VISUALLY EXPLAINED with EXAMPLES!Kapil Sachdeva
Переглядів 25 тис.
I wish every AI Engineer could watch this.1littlecoder
Переглядів 79 тис.
Low-rank Adaption of Large Language Models: Explaining the Key Concepts Behind LoRAChris Alexiuk
Переглядів 104 тис.
I Challenged AI to Create a Better YouTube Channel Than Me... Here's What Happenedadumb
Переглядів 390 тис.
Sad To Announce I Did Not Qualify For Mens 2024 Olympic Gymnastics TeamThe Rock
Переглядів 28 млн
🔥 Уся правда про українську СУДЖУЗе Інтерв'юер
Переглядів 356 тис.
ПЕТРО ЧЕРНИК ПОВЕРНУВСЯ! Про Курськ, реакцію Путіна та стратегічний переламСпецпроєкти Сергія Мамаєва
Переглядів 173 тис.
«Ми так війну не закінчимо ніколи»: 22-річний морпіх про те, чому їм потрібні молоді #війна #зсуСлідство.Інфо | Розслідування, репортажі, викриття
Переглядів 707 тис.
Get 10 Mega Boxes OR 60 Starr Drops!!Brawl Stars
Переглядів 15 млн
KLAVDIA PETRIVNA, МАША КОНДРАТЕНКО - ЇДЕ ДАХ (OFFICIAL VIDEO)Klavdia Petrivna
Переглядів 930 тис.
РЕЙНДЖЕРС - ДИНАМО. Ліга чемпіонів. ПРЯМА ТРАНСЛЯЦІЯFootball Hub
Переглядів 458 тис.
Алексей Щербаков разнес ВДВшниковBody Mania
Переглядів 11 млн
Really well done, my brain bulb went light up when you show the table! Thank you, keep it up!
Very clear explanation, thank you!
Great video! Keep it up!
Thank you for explaining it!
Very easy to understand. Thanx so much.
I haven’t read the paper yet but my understanding is that we sample from q(x) - p(x) because we want the most surprising token that the draft model does not anticipate. It should maximize the entropy but then it should have log on the equation, anyway, I gotta read the paper to understand the math.
Great work! Thank you
really informative! One thing that I don't understand is how does the LLM knows the previous probability distributions in a single pass? I thought decoder llm's only outputs the new token's probability distribution
This is due to the way the transformer architecture is set up: during decoding it takes as input all previous tokens and computes the hidden states for all previous tokens at each layer. Since we have the final layer hidden states, it is possible to obtain the probability distributions for all previous tokens.
@@EfficientNLP Now I get it. Did you also explain this somewhere in the video? Maybe link to it in the describtion because this is also where I didnt understand it.
We can compute the probability for each token in a batch fashion. The trick is to use mask attention.
Thank you for the video, when i first heard this idea in February i was wondering how it made sense because i was picturing a large K, now seeing that the recommended K is about 3 I understand how most of the output will be the same.
Very good video!
Google and DeepMind doing the Spiderman meme 😅
great explanation
thank you
Love your video, thanks!
If I had to give one request/critique, it'd be that I wish there were some slides in here similar to Samuel Albanie's videos that are quite information-dense recaps that could be lifted out of the presentations and put into our notes (or into a powerpoint for a paper club, or something).
Interesting idea, though my videos often contain animations, drawings, screencasts, etc., and are not directly a recording of PowerPoint slides. Feel free to take screenshots of my videos for any educational purposes though!
Really helpful video!
Great video 👍
made very simple, but one more variable is choosing right draft model. Suppose if one chooses that is too too away from larger one's distribution then its also a problem.
If the draft model is far from the target model's distribution, then speculative decoding will be less effective because it will have a higher rejection rate, thus reducing the speedup. However, the algorithm guarantees that the output sequence will be identical; therefore, even if the draft model is of poor quality, the text generation quality will not be affected.
Thanks for sharing, I am wondering how target model check the generated tokens of draft model and produce probability distribution q of x for each token?
This is due to the parallel nature of transformers - when given a sequence of tokens, it can generate the logits for all of them in parallel, unlike generation which must be done autoregressively.
Interesting. Curious if we can use mutiple different fine-tuned small models to do the same task along with a bigger model.
That is an interesting idea-like using multiple smaller models to generate several completions and having the large model choose the best one. I'm not sure if anyone has tried this.
@@EfficientNLP Yeah, hopefully might increase (or match) accuracy than original model.
this is great! is there any chance you could demonstrate something like this in code?
Good question - it appears that neither of the deepmind or google papers have an official source code implementation, but there are several implementations of this idea on GitHub, but I have not looked at them.
Thanks for this! I've been enjoying your videos! Do you think you do a review / explanation on flash-decoding by tri dao? I have been reading the pytorch blog but I don't really understand it
Thanks for the suggestion, I will add it to my list of future topics!
thank you for the explanations and the visuals. Does speculative decoding work with beam search? I understand that for LLM we generally just do greedy decoding in one pass, but for translation models like whisper, the performance increase significantly if we use beam search. I see even from hugging face official post discussing how speculative decoding improve whisper large inference speed by 2x, but to be honest, for non english audio data, with greedy decoding whisper is barely usable...
Interesting idea, but I don't know of any attempts to combine them. Speculative decoding relies heavily on random sampling, whereas beam search is deterministic, so probably they are incompatible. For speeding up whisper inference, you might try using smaller or quantized models on faster engines like CTranslate2 or faster-whisper.
why does target model running with K new tokens spend almost the same computation than with just 1 new token? I know K new tokens can be computed in parallel at one single forward pass, but self-attension with K new tokens indeed need more works than 1 token (suppose KV-cache is used), isn't it?
It's true that the computation time in the two scenarios might not be exactly the same due to KV cache and other implementation details; however, for simplicity, we can assume one forward pass through a model as taking one unit of time. Both decoding one token and checking the probabilities of multiple tokens require one forward pass.
Thanks! But doesn't the google paper define Mq as the draft model i.e. flips the definitions?
You are right; the Google paper uses a different notation from the DeepMind paper, in this video I'm using the DeepMind notation.
For this to work, the two models need to have identical tokenizations right? Is there any way around it?
That's right - the two models need to use the same vocabulary so that we can compare their logits meaningfully.
@@EfficientNLP thank you for the quick response. that makes sense!
Not necessarily. We can retokenize the predicted text by draft model. That can be slow though.
Nothing really new about this, it seems that big tech companies really do have it easier when publishing research
That’s the way it tends to go! One small step at a time