🎯 Key Takeaways for quick navigation: 00:00 🤖 *Introduction to Low Rank Adaptation (LoRA) and QLoRA* - LoRA is a parameter-efficient fine-tuning method for large language models. - Explains the need for efficient fine-tuning in the training process of large language models. 02:29 🛡️ *Challenges of Full Parameter Fine-Tuning* - Full parameter fine-tuning updates all model weights, requiring massive memory. - Limits fine-tuning to very large GPUs or GPU clusters due to memory constraints. 04:19 💼 *How LoRA Solves the Memory Problem* - LoRA tracks changes to model weights instead of directly updating all parameters. - It uses rank-one matrices to efficiently calculate weight changes. 06:11 🎯 *Choosing the Right Rank for LoRA* - Rank determines the precision of the final output table in LoRA fine-tuning. - For most tasks, rank can be set lower without sacrificing performance. 08:12 🔍 *Introduction to Quantized LoRA (QLoRA)* - QLoRA is a quantized version of LoRA that reduces model size without losing precision. - It exploits the normal distribution of parameters to achieve compression and recovery. 10:46 📈 *Hyperparameters in LoRA and QLoRA* - Discusses hyperparameters like rank, alpha, and dropout in LoRA and QLoRA. - The importance of training all layers and the relationship between alpha and rank. 13:30 🧩 *Fine-Tuning with LoRA and QLoRA in Practice* - Emphasizes the need to experiment with hyperparameters based on your specific data. - Highlights the ease of using LoRA with integrations like Replicate and Gradient.
I have been using thiese techniques for a while now without having a good understanding of each of the prameters. Thanks for giving a good overview of both the techniques and the papers
This is the best detailed video and nicest explanation on youtube right now. I do think your channel will grow because you are doing an EXCELENT job. Thank you man.
I'm pulling for another vid on alpha. Oobabooga suggests twice your rank. The Chinese alpaca lora people use a rank 8 with alpha 32 and I guess it worked. I've tried high alphas that make the model kinda crazy. Need guidence.
When in doubt, set alpha = rank for the effective scale factor to be 1. There are better ways to have a larger impact on training than bluntly multiplying the change in weights, like improving your dataset or dialing in the learning rate.
In LoRA, Wupdated = Wo + BA, where B and A are decomposed matrices with low ranks, so i wanted to ask you that what does the parameters of B and A represent like are they both the parameters of pre trained model, or both are the parameters of target dataset, or else one (B) represents pre-trained model parameters and the other (A) represents target dataset parameters, please answer as soon as possible
Wo would be the original model parameters. A and B multiplied together represent the changes to the original parameters learned from your fine-tuning. So together they represent the difference between your final fine-tuned model parameters and the original model parameters. Individually A and B don't represent anything, they are just intermediate stores of data that save memory.
The matrices are multiplied together and the result is the changes to the LLM's weights. It should be explained clearly in the video, it may help to rewatch.
@EntryPointAI My understanding: Orignal weight = 10 * 10 to form a two decomposed matrices A and B let's take the rank as 1 so, The A is 10 * 1 and B is 1 * 10 total trainable parameters is A + B = 20 In Lora even without any dataset training if we simply add the A and B matrices with original matric we can improve the accuracy slighty And if we use custom dataset in Lora the custom dataset matrices will captured by A and B matrices Am I right @EntryPointAI?
@@ArunkumarMTamil Trainable parameters math looks right. But these decomposed matrices will be initialized as all zeroes so adding them without any custom training dataset will have no effect.
Not sure if that's a rhetorical question, but I'll give it a go. You can call it just "training," but that might imply that it's ready to do something useful when you're done. If you call it "pre-training" it implies that you'll train it more afterward, which is generally true. So it may be useful in being a little more specific.
Nice video, congrats! LoRA is about fine-tuning, but is it possible to use it to compress the original matrices to speed up inference? I mean decompose the original model's original weight matrices to products of low-rank matrices to reduce the number of weights.
Seems worth looking into, but I couldn't give you a definitive answer on what the pros/cons would be. Intuitively I would expect it could reduce the memory footprint but that it wouldn't be any faster.
@@TheBojdaCheers mate! Yes, in distillation there is student-teacher configuration and the student tries to be like teacher with less parameters (aka. weights). This can also be combined with quantisation to reduce memory footprint.
wow I'm noobie in this field n I been testing fine-tunen my own chatbot with differents techniques, n I found a lot of stuff, but It's not commonly find a some explanation to understand the main reason of the use of it, ty a lot < 3
@@EntryPointAI Thank you, that's kind. I did already go look it up. Sorry I was frustrated. It's very common for people to forget to putlikes to papers in show note :^)
LoRa (Long Range) is a physical proprietary radio communication technique that uses a spread spectrum modulation technique derived from chirp spread spectrum. It's a low powered wireless platform that has become the de facto wireless platform of Internet of Things (IoT). Get your own acronym! 😂
Thanks! This is among the clearest and most concise explanations of LoRA and QLoRA. Really great job.
🎯 Key Takeaways for quick navigation:
00:00 🤖 *Introduction to Low Rank Adaptation (LoRA) and QLoRA*
- LoRA is a parameter-efficient fine-tuning method for large language models.
- Explains the need for efficient fine-tuning in the training process of large language models.
02:29 🛡️ *Challenges of Full Parameter Fine-Tuning*
- Full parameter fine-tuning updates all model weights, requiring massive memory.
- Limits fine-tuning to very large GPUs or GPU clusters due to memory constraints.
04:19 💼 *How LoRA Solves the Memory Problem*
- LoRA tracks changes to model weights instead of directly updating all parameters.
- It uses rank-one matrices to efficiently calculate weight changes.
06:11 🎯 *Choosing the Right Rank for LoRA*
- Rank determines the precision of the final output table in LoRA fine-tuning.
- For most tasks, rank can be set lower without sacrificing performance.
08:12 🔍 *Introduction to Quantized LoRA (QLoRA)*
- QLoRA is a quantized version of LoRA that reduces model size without losing precision.
- It exploits the normal distribution of parameters to achieve compression and recovery.
10:46 📈 *Hyperparameters in LoRA and QLoRA*
- Discusses hyperparameters like rank, alpha, and dropout in LoRA and QLoRA.
- The importance of training all layers and the relationship between alpha and rank.
13:30 🧩 *Fine-Tuning with LoRA and QLoRA in Practice*
- Emphasizes the need to experiment with hyperparameters based on your specific data.
- Highlights the ease of using LoRA with integrations like Replicate and Gradient.
I have been using thiese techniques for a while now without having a good understanding of each of the prameters. Thanks for giving a good overview of both the techniques and the papers
This is the best detailed video and nicest explanation on youtube right now. I do think your channel will grow because you are doing an EXCELENT job. Thank you man.
Very useful! Marvelous clear explanation with the right amount of detail about a subject that’s worth understanding
Thanks for this clear explanation about the topic!
Your way of relating back to research papers is very interesting and helpful!
Good explanation of LoRA and QLoRA
I loved the explanation! Please make more such videos!
great video! referring to the paper and explaining each thing in detail really helps understand the concept to the fullest. Kudos!
I've watched a couple other of your videos - you're a very good teacher - thanks for doing this.
I love this video man. watched it at least 3 times and came back to it before a job interview also. Please do more tutorials /explanations !
This is really well presented
It was incredible and very helpful video. Thank you man!
great video, i think the best explanation i've seen on this, i'm also really confused about why they picked the rank and alpha that they did.
This video deserves more likes
This is really helpful, Thank you!!
Loved the contnt! Simply explained no BS.
Great explanation, best that I've seen
Dude u look like the lich king with those blue eyes
True. 😅 He should be in acting career ig.
You mean Lich King looks like me I think 🤪
can you share the presentation document
Thisa saved me. Thank you. Keep doing this :)
Very useful, thank you!
I'm pulling for another vid on alpha. Oobabooga suggests twice your rank. The Chinese alpaca lora people use a rank 8 with alpha 32 and I guess it worked. I've tried high alphas that make the model kinda crazy. Need guidence.
When in doubt, set alpha = rank for the effective scale factor to be 1. There are better ways to have a larger impact on training than bluntly multiplying the change in weights, like improving your dataset or dialing in the learning rate.
@@EntryPointAI this does make sense they way you put it. Thanks so much for your reply!
can we use QLoRA in a simple ML model like CNN for image classification ?
Great video!
Some nice details here. Keep on.
What's the name of the paper you referenced in the video?
Here's LoRA: arxiv.org/abs/2106.09685
and QLoRA: arxiv.org/abs/2305.14314
really grate explanation
In LoRA, Wupdated = Wo + BA, where B and A are decomposed matrices with low ranks, so i wanted to ask you that what does the parameters of B and A represent like are they both the parameters of pre trained model, or both are the parameters of target dataset, or else one (B) represents pre-trained model parameters and the other (A) represents target dataset parameters, please answer as soon as possible
Wo would be the original model parameters. A and B multiplied together represent the changes to the original parameters learned from your fine-tuning. So together they represent the difference between your final fine-tuned model parameters and the original model parameters. Individually A and B don't represent anything, they are just intermediate stores of data that save memory.
@@EntryPointAI got it!! Thank you
how is Lora fine-tuning track changes from creating two decomposition matrix?
The matrices are multiplied together and the result is the changes to the LLM's weights. It should be explained clearly in the video, it may help to rewatch.
@EntryPointAI
My understanding:
Orignal weight = 10 * 10
to form a two decomposed matrices A and B
let's take the rank as 1 so, The A is 10 * 1
and B is 1 * 10
total trainable parameters is A + B = 20
In Lora even without any dataset training if we simply add the A and B matrices with original matric we can improve the accuracy slighty
And if we use custom dataset in Lora the custom dataset matrices will captured by A and B matrices
Am I right @EntryPointAI?
@@ArunkumarMTamil Trainable parameters math looks right. But these decomposed matrices will be initialized as all zeroes so adding them without any custom training dataset will have no effect.
Thanks a ton!
Great explanation...
Why do we call training a model pre-training it ?
Not sure if that's a rhetorical question, but I'll give it a go. You can call it just "training," but that might imply that it's ready to do something useful when you're done. If you call it "pre-training" it implies that you'll train it more afterward, which is generally true. So it may be useful in being a little more specific.
This was wonderfully educational and very easy to follow. That either it makes you a great educator or me an idiot :P Regardless, thank you.
let's both say it's the former and call it good! 🤣
Nice video, congrats! LoRA is about fine-tuning, but is it possible to use it to compress the original matrices to speed up inference? I mean decompose the original model's original weight matrices to products of low-rank matrices to reduce the number of weights.
I think you mean distillation with quantisation?
Seems worth looking into, but I couldn't give you a definitive answer on what the pros/cons would be. Intuitively I would expect it could reduce the memory footprint but that it wouldn't be any faster.
@@rishiktiwari Ty. I learned something new. :) If I understand well, this is a form of distillation.
@@TheBojdaCheers mate! Yes, in distillation there is student-teacher configuration and the student tries to be like teacher with less parameters (aka. weights). This can also be combined with quantisation to reduce memory footprint.
Amazing for struggling students. Love from Korea😂
very well explained, thanks :)
Awesome explanation! Which camera you use?
Thanks, it’s a Canon 6d Mk II
so good!!
wow I'm noobie in this field n I been testing fine-tunen my own chatbot with differents techniques, n I found a lot of stuff, but It's not commonly find a some explanation to understand the main reason of the use of it, ty a lot < 3
Nice
Saving me somr embarrassment in future work meetings. :) thanks for sharing.
Bright eyes
QLORA let's me train on a 4070ti with only 12gb vram. Though I can't go over 7b model
I wish people would actually share links to papers they reference...
LoRA: arxiv.org/abs/2106.09685
QLoRA: arxiv.org/abs/2305.14314
Click "Download PDF" in top right to view the actual papers.
@@EntryPointAI Thank you, that's kind. I did already go look it up. Sorry I was frustrated. It's very common for people to forget to putlikes to papers in show note :^)
Oh my god your eyes 😍😍😍😍everybody deserves hot teacher😂❤
Are you trying to hypnotize us with those eyes 😜
Gojo ?
Eyes!
Ahh very interesting thank you!
*goes to fine tune pictures of anime girls*
LoRa (Long Range) is a physical proprietary radio communication technique that uses a spread spectrum modulation technique derived from chirp spread spectrum. It's a low powered wireless platform that has become the de facto wireless platform of Internet of Things (IoT). Get your own acronym! 😂
Fair - didn’t create it, just explaining it 😂
bradley cooper in limitless tf
omg I always distracted by his blue eyes😆and ignoring what his talking
your have beautiful eyes
Thank you!
blud why your eyes like that