This is second video Ive watched from this channel after "quantization". And frankly wanted to express my gratitude towards your work as it is very easy to follow and the level of abstractions is tenable to understand concepts holistically.
Great video, thanks for creating this. I have use DDP quite a lot but seeing the visualizations for communication overlap helped me build a very good mental model. Would love to see more content around distributed training - Deepspeed ZeRO, Megatron DP + TP + PP
That's an amazing resource! It's great to see you sharing such detailed information on a complex topic. Your effort to explain everything clearly will really help others understand and apply these concepts. Keep up the great work!
Starting to watch my 3rd video on this channel, after transformer from scratch and quantization. Thank you for the great content and also for the code and notes to look back again. Thank you.
The video was very interesting and useful. Please make a similar video on DeepSpeed functionality. And in general, how to train large models (for example LLaMa SFT) on distributed systems (Multi-Server) when GPUs are located on different PCs.
well explained each and every detail, Great work Great Explanation👍 can you make this type of detailed video on distributed training through tensor parallelism? it would be very helpful. Thank you!
If time permits for you, Please make an video for entire GPU and TPU and how to them effectively and most of us donno . please create a playlist for pytorch for beginners and intermediates. Thanks for reading.
In broadcast , if we are sending the copy of file from rank 0 and rank 4 node to other node. How is the total time still 10 second. Because still I am having same internet speed of 1MB/s. Could anyone explain? I am bit confused. Also what happens if I am having odd numbers of nodes
@@umarjamilai not sure if it’s in your plans, but if you are open to suggestions, I would love to watch a video on multimodal models. Again, awesome work!
Hi Umar, Great video and enjoyed thorughly but i have one question.why are we using the approach of sum(grad1+grad2+....+gradN), why cant we use Avg of Gradients.
Of course you can (but you don't have to) use the average of the gradients. Actually, people usually take the average of the gradients. The reason we use the average is because we want the loss to be (more of less) the same as the non-distributed model, so you can compare the plots of the two. I don't know if PyTorch internally automatically takes the average of the gradients, I'd have to check the documentation/source.
Wouldn't the accumulated gradient need to be divided by the total number of individual gradients summed (or the learning rate needs to be divided by this value) to make it equivalent?
Yes, if you want to treat the "cumulative gradient" as a big batch, then you'd usually divide it by the number of items to keep it equivalent to the single-item setup. But it's not mandatory: as a matter of fact, loss functions on PyTorch have a "reduction" parameter, which is usually set to "mean" (so dividing the loss by the number of items) but can also be set to "sum". One reason we usually calculate the "mean" loss is because we want to make comparisons between models with different hyperparameters (batch size), so the loss should not depend on the batch size. But remember that mathematically you don't have to
In my understanding, yes the loss is accumulated for one batch theoretically, and the gradients are computed based on this accumulated loss too. But in the parallel implementation, both the loss calculated in the feedforward process, and the gradients calculated in the back propagation process executed in a parallel way. Here @umarjamilai use a for loop to illustrate the de facto parallel mechanism.
This is second video Ive watched from this channel after "quantization". And frankly wanted to express my gratitude towards your work as it is very easy to follow and the level of abstractions is tenable to understand concepts holistically.
This is the best video about Torch distributed I have ever seen. Thanks for making this video!
I really love your vidoes. you have a natural talent on simplifying logic and code. in same capacity as Andrej
agree
Great video, thanks for creating this. I have use DDP quite a lot but seeing the visualizations for communication overlap helped me build a very good mental model.
Would love to see more content around distributed training - Deepspeed ZeRO, Megatron DP + TP + PP
Thank you for the tutorial. It is really helpful to learn beyond pytorch documentations.
That's an amazing resource! It's great to see you sharing such detailed information on a complex topic. Your effort to explain everything clearly will really help others understand and apply these concepts. Keep up the great work!
Great introduction. Love the pace of the class and the balance of breadth vs depth
Starting to watch my 3rd video on this channel, after transformer from scratch and quantization. Thank you for the great content and also for the code and notes to look back again. Thank you.
Umar hits the sweet spot (Goldilocks zone) by balancing theory and practical😄😄😄😄😄
Dang. Never thought learning DDP would be this easy. Another great content from Umar. Looking forward for FSDP
Amazing video. Ideal video of how a lecture on a video should be
Super high quality lecture. You have a gift of teaching, man. Thank you!
this channel is hidden gem
The video was very interesting and useful. Please make a similar video on DeepSpeed functionality. And in general, how to train large models (for example LLaMa SFT) on distributed systems (Multi-Server) when GPUs are located on different PCs.
absolutely amazing! You made these concepts so accessible!
Incredible content, Umar! Well done! 🎉
Another great video, Umar. Nice work
非常清楚的解释~
Awesome tutorial!
Thank you very much for your wonderful video. Can you teach a video on how to use the accelerate library with dpp?
Amazing content! Thanks for your sharing
well explained each and every detail, Great work Great Explanation👍
can you make this type of detailed video on distributed training through tensor parallelism? it would be very helpful. Thank you!
You deserve many more likes and subscribers!
Awesome video. Please make tutorial on FSDP as well
thanks for the video!! can you cover the Tensor, Sequence and Pipeline Parallel and D using Dtensors in Pytorch next?
Great work! thank you !
Thankyou so much for this amazing video. It is really informative.
Really impressive!
Federated learning basics please.❤
very nice and informative video. Thanks
If time permits for you, Please make an video for entire GPU and TPU and how to them effectively and most of us donno .
please create a playlist for pytorch for beginners and intermediates.
Thanks for reading.
In broadcast , if we are sending the copy of file from rank 0 and rank 4 node to other node. How is the total time still 10 second. Because still I am having same internet speed of 1MB/s.
Could anyone explain? I am bit confused.
Also what happens if I am having odd numbers of nodes
Amazing learning stuff ! Very Thanks !~ 🥰🥰🥰
Great intro video. Do you have any plans to also cover other parallelism: Model, Pipeline, Tensor, etc.
you teach soooooooo good
Great video
You rock always 😂
I wish i could like it twice
You can share it on social media. That's the best way to thank me 😇
@@umarjamilai not sure if it’s in your plans, but if you are open to suggestions, I would love to watch a video on multimodal models. Again, awesome work!
Check my latest video!
I'm always confused with DP and DDP. Can you please tell me the difference between them? While both of them belong to data parallelism method.
DP only works on a single machine, while DDP can work on multiple machines. However, PyTorch now recommends using DDP also for single-machine setup.
@@umarjamilai thank you for your reply
Thank you so much for this
please do some thing related to audio large models like conformers,quartznet ,etc
Please create a video on model parallelism and FSDP.
Hi Umar, Great video and enjoyed thorughly but i have one question.why are we using the approach of sum(grad1+grad2+....+gradN), why cant we use Avg of Gradients.
Of course you can (but you don't have to) use the average of the gradients. Actually, people usually take the average of the gradients. The reason we use the average is because we want the loss to be (more of less) the same as the non-distributed model, so you can compare the plots of the two. I don't know if PyTorch internally automatically takes the average of the gradients, I'd have to check the documentation/source.
@@umarjamilaithanks for the info.
fantastic
great video
Thanks!
谢谢你！我们在领英connect吧
thanks
another banger
could provide another videos with respect to model parallel and pipeline parallel ? thanks..
Working with fsdp and megatron now and I really want to figure this out from scratch haha, it sounds fun but a big headache
Great!
SUUUPERRRR
How to do in Kubernetes? Please explain it.
Wouldn't the accumulated gradient need to be divided by the total number of individual gradients summed (or the learning rate needs to be divided by this value) to make it equivalent?
Yes, if you want to treat the "cumulative gradient" as a big batch, then you'd usually divide it by the number of items to keep it equivalent to the single-item setup. But it's not mandatory: as a matter of fact, loss functions on PyTorch have a "reduction" parameter, which is usually set to "mean" (so dividing the loss by the number of items) but can also be set to "sum".
One reason we usually calculate the "mean" loss is because we want to make comparisons between models with different hyperparameters (batch size), so the loss should not depend on the batch size.
But remember that mathematically you don't have to
shouldnt loss be accumulated ? loss += (y_pred - y_actual)^0.5
In my understanding, yes the loss is accumulated for one batch theoretically, and the gradients are computed based on this accumulated loss too. But in the parallel implementation, both the loss calculated in the feedforward process, and the gradients calculated in the back propagation process executed in a parallel way. Here @umarjamilai use a for loop to illustrate the de facto parallel mechanism.
Valeu!
do you have a discord channel?
21:01
Always great to watch your video, excellent work
Thanks!
Valeu!