Bhai mera bhai 😍 Abto hazaro students aayenge bhai ke pass par Apne sabse pehle student ko mat bhulna bawa😜 Very proud of you bhai... And i can guarantee every1 here that he is the best teacher that there is❤️
My work laptop doesn't allow the gmail login so I usually do not comment or like. But for this, I searched for the same video on mobile just to like and comment. Brilliantly explained. Your channel is definitely underrated. Thank you for all the videos.
Bro, I am beginner but i was able to understand everything. Really great content and ur explanations was also amazing. Please continue doing such great videos. Thanks a lot for sharing .
no one teaches detailed way complex things like you no matter what please spread you're knowledge to world i am sure there must be people learn from you , remember you as master life long who settled in it job like me
Buddy! You got a new sub here. Loved your detailed explanation. I see no one explaining the query plain this detail and I believe this is the right way of learning. But I would love to see an entire Spark series.
This is really informative, such details are not even present in the O'Reilly Learning Spark Book. Please continue to make such content. Needless to say but I have already subscribed.
By watching your first 15mins of youtube video and I am awed beyond my words. What a great explanation @afaqueahmad. Kudos to you! Please make more videos of solving real time scenarios using PySpark & Cluster configuration. Again BIG THANKS!
Just 10 minutes into this notebook and I am awed beyond my words. What a great explanation Afaque. Kudos to you! Please make more videos of solving real time scenarios using Spark UI and one on Cluster configuration too. Again BIG THANKS!
One of the cleanest explanation I ever come across on the internals of Spark. Really appreciate all the effort you are putting into making these videos. If you don't mind, May I know which text editor are you are using when pasting the Physical plan?
I loved your explanation and understood it very well. Could you help me to understand at 23 mins, if we have join key as cid and group by region. how the hash partitioning works. will that consider both?
Thank you for taking the time to create such an in depth video for Spark Plans. This is very helpful ! Would you also be able to explain Spark Memory Tuning ? How do we decide how much resources to allocate (driver mem, executors mem , num executors , etc for a spark submit ? Also Data Structures Tuning, Garbage Collection Tuning ! Thanks again !
Thanks for the kind words @crystalllake3158 and the suggestion; currently the focus of the series is to cover all possible code level optimization. Resource level optimisations will come in much later, but no plans for the upcoming few months :)
Thanks for nice explanation. My question regarding COALESCE - what happens when the partition increases the default size. Let us say 128 MB is default partition size, and we are coalescing to 3 partitions. But the volume of data is 512 MB ? How would spark handle such scenarios?
Hey @ajaykiranchundi9979, the default partition size `spark.sql.files.maxPartitionBytes` applies at read time. In this case, 4 partitions will initially be read (512/128). After coalescing, each will have and approximate size of ~170MB
Can you please prepare a video showing storage anatomy of data during job execution cycle? I am sure there are many aspiring spark students who may be confused about the idea of RDD or dataframe and how it access data through apis (since spark is in memory computation) during job execution. It will help many upcoming spark developers.
Hey @udaymmmmmmmmmm, I added this video recently on Spark Memory Management. It talks about storage and responsibilities or each of memory components during job execution. You may want to have a look at it :) Link here: ua-cam.com/video/sXL1qgrPysg/v-deo.html
At the very end of the video 38:36, we see that the cast("int") filter is present in the parsed logical plan and Analyzed logical plan. I am a little confused as to when we refer those plans. Can you please explain?
I have doubt when the data will be distributed to executor is it before scheduling the task or after scheduling the task and who assign the data to executor
Hey @sangu2227, this requires an understanding of transformations/actions and lazy evaluation in Spark. Spark doesn't do anything (either scheduling a task or distributing data) until an action is called. The moment an action is invoked, Spark creates a logical -> physical plan and Spark's scheduler divides the work into tasks. Spark's driver and Cluster manager then distributes the data to the executors for processing :)
Hello @afaqueahmad7117, thanks for the great video. While explaining repartition, you mentioned you’ve a video on the AQE. Please can you link that as well?
Can't wait for that@@afaqueahmad7117 These UA-cam videos are so much more helpful. Hats down one of the best ones that explain the Spark performance tuning and internals in a very simplest of forms possible. Cheers!
You mentioned that for coalesce(2) shuffle will happen, but later you mentioned that shuffle will not happen in case of coalesce hence no partitioning scheme. Could you please explain it in detail?
So, coalesce will only incur a shuffle if its a very aggressive situation. If the objective can be achieved by merging (reducing) the partitions on the same executor, it will go ahead with it. In case of coalesce(2), its an aggressive reduction in the number of partitions, meaning that Spark has no other option but to move the partitions. As there were 3 executors (in the example I referenced in the video), even if it reduced the partitions on each executor to a single partition, it would end up with 3 partitions in total, therefore it incurs a shuffle to have 2 final partitions :)
if it is doing local aggregation before shuffling the data then why it will throw out of memory error while taking count of each key when the column has huge distinct values
200 is the default number of shuffle partitions. You can find the number here in this table by the property name "spark.sql.shuffle.partitions" spark.apache.org/docs/latest/sql-performance-tuning.html#other-configuration-options
I am doing coalesce(1) and getting error as : Unable to acquire 65536 bytes of memory, got 0. But when i am doing repartition(1), it worked. Can you please explain what happens internally in this case?
The error you’re seeing when using coalesce(1) versus repartition(1) in Spark is related to the way these two operations handle memory and shuffling. Here’s what’s happening internally with each: coalesce(1) > Purpose: coalesce is used to reduce the number of partitions in a DataFrame, ideally with minimal data movement across the cluster. > Memory Handling: When you call coalesce(1), Spark attempts to reduce the number of partitions to one. However, unlike repartition, it does so by consolidating data without shuffling across all nodes. This can cause an out-of-memory error if the data volume is too high to fit into one partition on a single node. > Error Explanation: The error message, "Unable to acquire 65536 bytes of memory, got 0," indicates that Spark couldn’t allocate the requested memory for the partition consolidation due to the memory limitations on a node. repartition(1) > Purpose: repartition increases or decreases the partitions by evenly redistributing the data through a full shuffle, which can spread the load more evenly. > Memory Handling: When you use repartition(1), Spark performs a full shuffle across all nodes, moving data as needed. This distribution helps avoid memory overload on a single node by balancing memory across the cluster, making it more reliable for large data volumes. > Why it Worked: Because repartition involves a shuffle, it doesn’t place as much of a memory burden on any one executor, unlike coalesce. This allowed your code to avoid the memory allocation issue and complete successfully. Key Differences > Shuffle: repartition triggers a shuffle, coalesce does not (or minimizes shuffling). > Memory Demand: coalesce is more efficient for reducing partitions only if the data size per partition is small enough to fit within the available memory. For large datasets or tight memory, repartition is often more reliable. In summary, repartition(1) worked because it redistributed the data across the cluster rather than trying to load it all into a single node's memory.
Hi sir i came across a doubt Consider the executor size 1gb/executor. We have 3 executors and intially 3 gb data gets distributed across 3 executors each executor is having 1gb partition after various transformations we came across a requirment to decrease the number of partitions to 1 partition for that we will use repartition(1) or coalesce(1). In this scenario all the 3 partitions will merges to 1 partition each partition is having size of 1 gb approximately. Collectively all the partitions size is 3 gb approximately. When repartition (1) or coalesce(1) all the 3 gb data should sit in 1 executor having capicity of 1gb only. So here the data is execeeding the executor size what happens in this scenario. Could you please make video on this requesting sir.
Hi @bhargaviakkineni, In the scenario you described above where the resulting partition size (3 GB) exceeds the memory available on a single executor (1 GB), Spark will attempt to spill data to disk. The spill to disk is going to help the application from crashing due to out-of-memory errors however, there is going to be a performance impact associated, because disk IO is slower. On a side note, as a best practice, It’s best to also think/re-evaluate the need to write to a single partition. Avoid writing to a single partition, because it generally creates a bottleneck if the sizes are large. Try to balance out the partitions with the resources of the cluster (executors/cores). Hope that clarifies :)
Local distinct on cust id doens't make sense and couldn't understand. How globally it does distinct count if the count is already computed. The reasoning behind why cast doens't push down predicate is not clearly explained and just as it's mentioned in the doc
After distinct on cust id and city locally and globally, the data size would be reduced massively. An easy way to understand, spark adds a df.dropDuplicate(A,B) before running groupBy(A).countDistinct(B)
Hi Afaque. A suggestion. You could start from the beginning to connect the DOTS! Like if in your scenario we have X Node machine with Y workers and Z exectors and if you do REPARTITION and fit the data like this then this could happen. Otherwise the Machine would sit idle and so on.
Hi Sir Please Make a Detailed course on apache spark which include every aspect of spark for Data Engineer role Please make sure there are a lot of begineer course here in market keep the course from intermediate level to advance level. Please tr to make video in Hindi it will be very helpful.
🔔🔔 Please remember to subscribe to the channel folks. It really motivates me to make more such videos :)
Your content is really awesome with out any further thought, and its very advance level pyspark understanding
Thanks a bunch. To my knowledge, no one has explained Spark explain function this detailed level. Very in-depth information.
Bhai mera bhai 😍 Abto hazaro students aayenge bhai ke pass par Apne sabse pehle student ko mat bhulna bawa😜
Very proud of you bhai... And i can guarantee every1 here that he is the best teacher that there is❤️
My work laptop doesn't allow the gmail login so I usually do not comment or like. But for this, I searched for the same video on mobile just to like and comment. Brilliantly explained. Your channel is definitely underrated.
Thank you for all the videos.
Hey @mohitshrivastava4967, thank you for this note and gesture, it means a lot to me. Really appreciate it, brother :)
Proud of you brother, looking forward to more of such videos. Great job!
After looking for some time for best material which truly explains this topic, and try to dig deep enough you clearly delivered, thanks Afaque.
Glad it was helpful, appreciate it :)
Simple and most effective explanation.
Appreciate it man :)
Amazing content.. I am a newbie into Spark but I am hooked.. Sir plz post the continued series.. awaiting for your video posts.. Amazing teacher
This takes me back to me YaarPadhade times. Great work Bhai much love!
rare content! please don't stop making these
Bro, I am beginner but i was able to understand everything. Really great content and ur explanations was also amazing. Please continue doing such great videos. Thanks a lot for sharing .
@thecodingmind9319 Thanks for the kind words, means a lot :)
Thanks for such an in-depth overview!! helps a lot to grow!!
Afaque, THANK YOU SO MUCH FOR THESE VIDEOS!!
They are so amazing for a fast paced learning experience.
Hope you soon upload much more!!
Thanks! From Brazil
Appreciate it, thanks man! :)
One of the best videos I have seen on Spark, waiting for your Spark Architecture Video
no one teaches detailed way complex things like you no matter what please spread you're knowledge to world i am sure there must be people learn from you , remember you as master life long who settled in it job like me
This is one of the best video about Spark I have seen recently!
It's great to see such useful contents in spark... an its helpful to understand clearer with your notes! you rock.... Thankless thanks !!
Explained the concept really well!
Buddy! You got a new sub here.
Loved your detailed explanation. I see no one explaining the query plain this detail and I believe this is the right way of learning. But I would love to see an entire Spark series.
Thank you @snehitvaddi for the kind appreciation. A full-fledged, in-depth course on Spark coming soon :)
@@afaqueahmad7117 Most awaited. Keep up the 🚀
My today's well spent 40 mins. Thanks for the knowledge sharing.
Bhai bhot bhadia content banaate ho. Love your vdos. Please keep it up. You have great teaching skills.
Bohot shukriya bhai sahab!
Great content with practical knowledge. Hats off to you !!!
This is the most detailed explanation I have ever seen.
Appreciate it man @ridewithsuraj-zz9cc :)
By far best content i have seen on explain query thing!!! Keep it brother. Good luck!
Glad, you liked it, thank you! :)
Beautifully explained. Many concepts got cleared. thanks a lot.Keep going.
You are a gem bro. The content that you bring here is terrific. ❤❤❤
Thanks man, @yashwantdhole7645. This means a lot!
This is really informative, such details are not even present in the O'Reilly Learning Spark Book. Please continue to make such content. Needless to say but I have already subscribed.
This is pure gold, congrats bro , keep the good work
Thank you @garydiaz8886, really appreciate it! :)
Great explanation of such a complex topic, thanks and keep up the good work.
Thanks man @Dhawal-ld2mc :)
one of the best videos i came across on spark query plan explanation. Thank you! :)
Appreciate it @myl1566, thank you!
Awesome video as always. Would really appreciate more videos explaining how DAG's can be read
Bro, This is really nice, I just love the way you teach and very very good content. Bahut bahut Sukriya and ton of love
Thank you @RaviSingh-dp6xc, appreciate it man, means a lot, thank you! :)
It's a great video with a great explanation. Awesome. Thank you for such a detailed explanation. Please keep doing such content.
Excellent Video
Thanks for the beautiful explanation
Appreciate it @Ilovefriendswebseries, glad to hear that :)
Absolute gem ❤❤ would like to have video on handling real time scenarios (handle slow running job, oom etc)..
I am sure that down the line, in a few years, you will cross 100k subscribers. Great content BTW.
Hey @CoolGuy , thanks man! Means a lot to me :)
Excellend content, please make more videos like this with deep understanding of "how stuff works"... Highly Appreciate it. Love from 🇵🇰
Thank you @MuhammadAhmad-do1sk for the appreciation, love from India :)
Bro, you dropped this👑
amazing explanation!
Appreciate it :)
By watching your first 15mins of youtube video and I am awed beyond my words.
What a great explanation @afaqueahmad. Kudos to you!
Please make more videos of solving real time scenarios using PySpark & Cluster configuration. Again BIG THANKS!
Hey @VenuuMaadhav, thank you for the kind words, means a lot. More coming soon :)
Very useful, video man, thanks for explaining things in so much details, keep doing the good work.
a lot of knowledge in just one video
Appreciate it @user-pq9tx6ui2t :)
Thank you so much for making this video. this is really very helpful.
Underrated pro max!
Very useful and explaining complex things in easy manner . Thanks and expect more videos from you
Love the video! great content and presentation. You definitely earned a subscriber in me and probably my fellow DE friends. :)
Glad to know you liked the video and for being a subscriber, thanks man :)
"God bless you! Great video! Learned a lot"
Explanation is so good
Thank you @satyajitmohanty5039 :)
Great Content. Nice and Detailed!!
Thank you @shaheelsahoo8535, appreciate it :)
Excellent job 🙌
Thanks @prasadrajupericharla5545, appreciate it :)
This is really good, thanks so much for this explanation!
Great Video!
Appreciate it @jjayeshpawar, thank you!
Very Informative.... Thanks for sharing 🙂
Great tutorials
Appreciate it! :)
Just 10 minutes into this notebook and I am awed beyond my words.
What a great explanation Afaque. Kudos to you!
Please make more videos of solving real time scenarios using Spark UI and one on Cluster configuration too. Again BIG THANKS!
Hi @mohitupadhayay1439, really appreciate the kind words, it means a lot. A lot coming soon :)
One of the cleanest explanation I ever come across on the internals of Spark. Really appreciate all the effort you are putting into making these videos.
If you don't mind, May I know which text editor are you are using when pasting the Physical plan?
Many thanks for the kind words @venkatyelava8043, means a lot. On the text editor - I'm using Notion :)
I loved your explanation and understood it very well. Could you help me to understand at 23 mins, if we have join key as cid and group by region. how the hash partitioning works. will that consider both?
Nice explanation
please do more vedios bro. love this one
Thank you @varunparuchuri9544, really appreciate it :)
Great explanation!!Keep uploading such quality content bro
Very Good explanation...Keep Going
Thank you!
Amazing content! Thank you for sharing!
Thank you @crazypri8, appreciate it :)
this is gold. Thank you very much!
@user-meowmeow1 Glad you found it helpful :)
Thank you for taking the time to create such an in depth video for Spark Plans. This is very helpful !
Would you also be able to explain Spark Memory Tuning ?
How do we decide how much resources to allocate (driver mem, executors mem , num executors , etc for a spark submit ?
Also Data Structures Tuning, Garbage Collection Tuning !
Thanks again !
Thanks for the kind words @crystalllake3158 and the suggestion; currently the focus of the series is to cover all possible code level optimization. Resource level optimisations will come in much later, but no plans for the upcoming few months :)
Thanks ! Please do keep uploading, love your videos !
Great explanation.
Great content brother. Please post more 😁
Thanks for nice explanation. My question regarding COALESCE - what happens when the partition increases the default size. Let us say 128 MB is default partition size, and we are coalescing to 3 partitions. But the volume of data is 512 MB ?
How would spark handle such scenarios?
Hey @ajaykiranchundi9979, the default partition size `spark.sql.files.maxPartitionBytes` applies at read time. In this case, 4 partitions will initially be read (512/128). After coalescing, each will have and approximate size of ~170MB
This was awesome!
Please do post videos very often
You are awesome man❤
Great explanation man! Thank you! What's the editor that you use in the video to read query plans?
Thanks @venkateshkannan7398, appreciate it. Using Notion :)
Good knowledge
Appreciate it, thank you @savitajade8425 :)
Great explanation! I love the simplicity of it! I wonder what is the app you use for having your Mac as a screenshot that you can edit with your iPad?
Thanks @kvin007! So, basically I join a zoom meeting with my own self and annotate, haha!
Great video thanks for sharing. I definitely subscribe
Can you please prepare a video showing storage anatomy of data during job execution cycle? I am sure there are many aspiring spark students who may be confused about the idea of RDD or dataframe and how it access data through apis (since spark is in memory computation) during job execution. It will help many upcoming spark developers.
Hey @udaymmmmmmmmmm, I added this video recently on Spark Memory Management. It talks about storage and responsibilities or each of memory components during job execution. You may want to have a look at it :)
Link here: ua-cam.com/video/sXL1qgrPysg/v-deo.html
At the very end of the video 38:36, we see that the cast("int") filter is present in the parsed logical plan and Analyzed logical plan. I am a little confused as to when we refer those plans. Can you please explain?
I have doubt when the data will be distributed to executor is it before scheduling the task or after scheduling the task and who assign the data to executor
Hey @sangu2227, this requires an understanding of transformations/actions and lazy evaluation in Spark. Spark doesn't do anything (either scheduling a task or distributing data) until an action is called.
The moment an action is invoked, Spark creates a logical -> physical plan and Spark's scheduler divides the work into tasks. Spark's driver and Cluster manager then distributes the data to the executors for processing :)
quality content
Thank you!
You were too good!
Thank you!
Thanks for the content and when can we expect new video?
Coming soon, in the next few days! :)
Hello @afaqueahmad7117, thanks for the great video. While explaining repartition, you mentioned you’ve a video on the AQE. Please can you link that as well?
Thanks @nijanthanvijayakumar, yes that video is upcoming in the next few days :)
Can't wait for that@@afaqueahmad7117
These UA-cam videos are so much more helpful. Hats down one of the best ones that explain the Spark performance tuning and internals in a very simplest of forms possible. Cheers!
You mentioned that for coalesce(2) shuffle will happen, but later you mentioned that shuffle will not happen in case of coalesce hence no partitioning scheme. Could you please explain it in detail?
So, coalesce will only incur a shuffle if its a very aggressive situation. If the objective can be achieved by merging (reducing) the partitions on the same executor, it will go ahead with it. In case of coalesce(2), its an aggressive reduction in the number of partitions, meaning that Spark has no other option but to move the partitions. As there were 3 executors (in the example I referenced in the video), even if it reduced the partitions on each executor to a single partition, it would end up with 3 partitions in total, therefore it incurs a shuffle to have 2 final partitions :)
@@afaqueahmad7117 Thanks for clarification.
What drawing board are you using for those notes?
Using "Notion" for text, "Nebo" on iPad for the diagrams
@@afaqueahmad7117cool thx!
if it is doing local aggregation before shuffling the data then why it will throw out of memory error while taking count of each key when the column has huge distinct values
Hi Sir, you mentioned that you referred AQE before. Can I get that link ? I want to know about AQE
Yes, I will be releasing the video in the next few days. :)
@@afaqueahmad7117 Thank you sir.
In exchange hashpartitioning what is the significance of number 200? what does that mean?
200 is the default number of shuffle partitions. You can find the number here in this table by the property name "spark.sql.shuffle.partitions" spark.apache.org/docs/latest/sql-performance-tuning.html#other-configuration-options
I am doing coalesce(1) and getting error as : Unable to acquire 65536 bytes of memory, got 0.
But when i am doing repartition(1), it worked. Can you please explain what happens internally in this case?
The error you’re seeing when using coalesce(1) versus repartition(1) in Spark is related to the way these two operations handle memory and shuffling.
Here’s what’s happening internally with each:
coalesce(1)
> Purpose: coalesce is used to reduce the number of partitions in a DataFrame, ideally with minimal data movement across the cluster.
> Memory Handling: When you call coalesce(1), Spark attempts to reduce the number of partitions to one. However, unlike repartition, it does so by consolidating data without shuffling across all nodes. This can cause an out-of-memory error if the data volume is too high to fit into one partition on a single node.
> Error Explanation: The error message, "Unable to acquire 65536 bytes of memory, got 0," indicates that Spark couldn’t allocate the requested memory for the partition consolidation due to the memory limitations on a node.
repartition(1)
> Purpose: repartition increases or decreases the partitions by evenly redistributing the data through a full shuffle, which can spread the load more evenly.
> Memory Handling: When you use repartition(1), Spark performs a full shuffle across all nodes, moving data as needed. This distribution helps avoid memory overload on a single node by balancing memory across the cluster, making it more reliable for large data volumes.
> Why it Worked: Because repartition involves a shuffle, it doesn’t place as much of a memory burden on any one executor, unlike coalesce. This allowed your code to avoid the memory allocation issue and complete successfully.
Key Differences
> Shuffle: repartition triggers a shuffle, coalesce does not (or minimizes shuffling).
> Memory Demand: coalesce is more efficient for reducing partitions only if the data size per partition is small enough to fit within the available memory. For large datasets or tight memory, repartition is often more reliable.
In summary, repartition(1) worked because it redistributed the data across the cluster rather than trying to load it all into a single node's memory.
Hi sir i came across a doubt
Consider the executor size 1gb/executor. We have 3 executors and intially 3 gb data gets distributed across 3 executors each executor is having 1gb partition after various transformations we came across a requirment to decrease the number of partitions to 1 partition for that we will use repartition(1) or coalesce(1). In this scenario all the 3 partitions will merges to 1 partition each partition is having size of 1 gb approximately. Collectively all the partitions size is 3 gb approximately. When repartition (1) or coalesce(1) all the 3 gb data should sit in 1 executor having capicity of 1gb only. So here the data is execeeding the executor size what happens in this scenario. Could you please make video on this requesting sir.
Hi @bhargaviakkineni, In the scenario you described above where the resulting partition size (3 GB) exceeds the memory available on a single executor (1 GB), Spark will attempt to spill data to disk. The spill to disk is going to help the application from crashing due to out-of-memory errors however, there is going to be a performance impact associated, because disk IO is slower.
On a side note, as a best practice, It’s best to also think/re-evaluate the need to write to a single partition. Avoid writing to a single partition, because it generally creates a bottleneck if the sizes are large. Try to balance out the partitions with the resources of the cluster (executors/cores).
Hope that clarifies :)
I love to work
Hi Afaque, how can I download the data files you are using? I want to try it hands on :)
Should be available here: github.com/afaqueahmad7117/spark-experiments :)
Bro please make more videos !!!
Local distinct on cust id doens't make sense and couldn't understand. How globally it does distinct count if the count is already computed. The reasoning behind why cast doens't push down predicate is not clearly explained and just as it's mentioned in the doc
After distinct on cust id and city locally and globally, the data size would be reduced massively.
An easy way to understand, spark adds a df.dropDuplicate(A,B) before running groupBy(A).countDistinct(B)
Hi Afaque.
A suggestion.
You could start from the beginning to connect the DOTS!
Like if in your scenario we have X Node machine with Y workers and Z exectors and if you do REPARTITION and fit the data like this then this could happen.
Otherwise the Machine would sit idle and so on.
Fab Cotenet
Great Work buddy keep it up .... love your content, very simple to understand @Afaque Ahmed
Thanks a ton!
Hi Sir Please Make a Detailed course on apache spark which include every aspect of spark for Data Engineer role Please make sure there are a lot of begineer course here in market keep the course from intermediate level to advance level. Please tr to make video in Hindi it will be very helpful.