Master Reading Spark Query Plans
Вставка
- Опубліковано 17 лип 2024
- Spark Performance Tuning
Dive deep into Apache Spark Query Plans to better understand how Apache Spark operates under the hood. We'll cover how Spark creates logical and physical plans, as well as the role of the Catalyst Optimizer in utilizing optimization techniques such as filter (predicate) pushdown and projection pushdown.
The video covers intermediate concepts of Apache Spark in-depth, detailed explanations on how to read the Spark UI, understand Apache Spark’s query plans through code snippets of various narrow and wide transformations like reading files, select, filter, join, group by, repartition, coalesce, hash partitioning, hashaggregate, round robin partitioning, range partitioning and sort-merge join. Understanding them is going to give you a grasp on reading Spark’s step-by-step thought process and help identify performance issues and possible optimizations.
📄 Complete Code on GitHub: github.com/afaqueahmad7117/sp...
🎥 Full Spark Performance Tuning Playlist: • Apache Spark Performan...
🔗 LinkedIn: / afaque-ahmad-5a5847129
Chapters:
00:00 Introduction
01:30 How Spark generates logical and physical plans?
04:46 Narrow transformations (filter, select, add or update columns) query plan explanation
09:02 Repartition query plan explanation
12:57 Coalesce query plan explanation
17:32 Joins query plan explanation
23:23 Group by count query plan explanation
27:04 Group by sum query plan explanation
28:05 Group by count distinct query plan explanation
33:59 Interesting observations on Spark’s query plans
36:56 When will predicate pushdown not work?
39:07 Thank you
#ApacheSpark #SparkPerformanceTuning #DataEngineering #SparkDAG #SparkOptimization
🔔🔔 Please remember to subscribe to the channel folks. It really motivates me to make more such videos :)
Proud of you brother, looking forward to more of such videos. Great job!
This is one of the best video about Spark I have seen recently!
One of the best videos I have seen on Spark, waiting for your Spark Architecture Video
Thanks for such an in-depth overview!! helps a lot to grow!!
Afaque, THANK YOU SO MUCH FOR THESE VIDEOS!!
They are so amazing for a fast paced learning experience.
Hope you soon upload much more!!
Very useful and explaining complex things in easy manner . Thanks and expect more videos from you
Explained the concept really well!
rare content! please don't stop making these
Thanks a bunch. To my knowledge, no one has explained Spark explain function this detailed level. Very in-depth information.
It's great to see such useful contents in spark... an its helpful to understand clearer with your notes! you rock.... Thankless thanks !!
This takes me back to me YaarPadhade times. Great work Bhai much love!
Beautifully explained. Many concepts got cleared. thanks a lot.Keep going.
Very useful, video man, thanks for explaining things in so much details, keep doing the good work.
Amazing content.. I am a newbie into Spark but I am hooked.. Sir plz post the continued series.. awaiting for your video posts.. Amazing teacher
Great content with practical knowledge. Hats off to you !!!
After looking for some time for best material which truly explains this topic, and try to dig deep enough you clearly delivered, thanks Afaque.
Glad it was helpful, appreciate it :)
You are a gem bro. The content that you bring here is terrific. ❤❤❤
Thanks man, @yashwantdhole7645. This means a lot!
one of the best videos i came across on spark query plan explanation. Thank you! :)
Appreciate it @myl1566, thank you!
Great explanation!!Keep uploading such quality content bro
It's a great video with a great explanation. Awesome. Thank you for such a detailed explanation. Please keep doing such content.
Thank you so much for making this video. this is really very helpful.
This is really good, thanks so much for this explanation!
This is pure gold, congrats bro , keep the good work
Thank you @garydiaz8886, really appreciate it! :)
My today's well spent 40 mins. Thanks for the knowledge sharing.
By far best content i have seen on explain query thing!!! Keep it brother. Good luck!
Glad, you liked it, thank you! :)
Bro, you dropped this👑
This was awesome!
Great content brother. Please post more 😁
"God bless you! Great video! Learned a lot"
no one teaches detailed way complex things like you no matter what please spread you're knowledge to world i am sure there must be people learn from you , remember you as master life long who settled in it job like me
Underrated pro max!
Absolute gem ❤❤ would like to have video on handling real time scenarios (handle slow running job, oom etc)..
Great explanation.
Amazing content! Thank you for sharing!
Thank you @crazypri8, appreciate it :)
Great Content. Nice and Detailed!!
Thank you @shaheelsahoo8535, appreciate it :)
This is really informative, such details are not even present in the O'Reilly Learning Spark Book. Please continue to make such content. Needless to say but I have already subscribed.
this is gold. Thank you very much!
@user-meowmeow1 Glad you found it helpful :)
Great video thanks for sharing. I definitely subscribe
You are awesome man❤
Excellent job 🙌
Thanks @prasadrajupericharla5545, appreciate it :)
Bro, I am beginner but i was able to understand everything. Really great content and ur explanations was also amazing. Please continue doing such great videos. Thanks a lot for sharing .
@thecodingmind9319 Thanks for the kind words, means a lot :)
Bhai bhot bhadia content banaate ho. Love your vdos. Please keep it up. You have great teaching skills.
Bohot shukriya bhai sahab!
Very Good explanation...Keep Going
Thank you!
Great Video!
Appreciate it @jjayeshpawar, thank you!
I am sure that down the line, in a few years, you will cross 100k subscribers. Great content BTW.
Hey @CoolGuy , thanks man! Means a lot to me :)
By watching your first 15mins of youtube video and I am awed beyond my words.
What a great explanation @afaqueahmad. Kudos to you!
Please make more videos of solving real time scenarios using PySpark & Cluster configuration. Again BIG THANKS!
Hey @VenuuMaadhav, thank you for the kind words, means a lot. More coming soon :)
Bhai mera bhai 😍 Abto hazaro students aayenge bhai ke pass par Apne sabse pehle student ko mat bhulna bawa😜
Very proud of you bhai... And i can guarantee every1 here that he is the best teacher that there is❤️
Excellend content, please make more videos like this with deep understanding of "how stuff works"... Highly Appreciate it. Love from 🇵🇰
Thank you @MuhammadAhmad-do1sk for the appreciation, love from India :)
Thank you for taking the time to create such an in depth video for Spark Plans. This is very helpful !
Would you also be able to explain Spark Memory Tuning ?
How do we decide how much resources to allocate (driver mem, executors mem , num executors , etc for a spark submit ?
Also Data Structures Tuning, Garbage Collection Tuning !
Thanks again !
Thanks for the kind words @crystalllake3158 and the suggestion; currently the focus of the series is to cover all possible code level optimization. Resource level optimisations will come in much later, but no plans for the upcoming few months :)
Thanks ! Please do keep uploading, love your videos !
please do more vedios bro. love this one
Thank you @varunparuchuri9544, really appreciate it :)
I loved your explanation and understood it very well. Could you help me to understand at 23 mins, if we have join key as cid and group by region. how the hash partitioning works. will that consider both?
quality content
Thank you!
You were too good!
Thank you!
Great explanation! I love the simplicity of it! I wonder what is the app you use for having your Mac as a screenshot that you can edit with your iPad?
Thanks @kvin007! So, basically I join a zoom meeting with my own self and annotate, haha!
Great explanation man! Thank you! What's the editor that you use in the video to read query plans?
Thanks @venkateshkannan7398, appreciate it. Using Notion :)
Hi Afaque.
Do we have any library or can we create a UDF for understanding why some records got corrupt while reading file?
I have a nested XML file with large number of columns and I want to understand why some columns are going into corrupt. Couldn't find anything helpful online.
This video would be greatly appreciated.
At the very end of the video 38:36, we see that the cast("int") filter is present in the parsed logical plan and Analyzed logical plan. I am a little confused as to when we refer those plans. Can you please explain?
if it is doing local aggregation before shuffling the data then why it will throw out of memory error while taking count of each key when the column has huge distinct values
Bro please make more videos !!!
I am doing coalesce(1) and getting error as : Unable to acquire 65536 bytes of memory, got 0.
But when i am doing repartition(1), it worked. Can you please explain what happens internally in this case?
Can you please prepare a video showing storage anatomy of data during job execution cycle? I am sure there are many aspiring spark students who may be confused about the idea of RDD or dataframe and how it access data through apis (since spark is in memory computation) during job execution. It will help many upcoming spark developers.
Hey @udaymmmmmmmmmm, I added this video recently on Spark Memory Management. It talks about storage and responsibilities or each of memory components during job execution. You may want to have a look at it :)
Link here: ua-cam.com/video/sXL1qgrPysg/v-deo.html
Hello @afaqueahmad7117, thanks for the great video. While explaining repartition, you mentioned you’ve a video on the AQE. Please can you link that as well?
Thanks @nijanthanvijayakumar, yes that video is upcoming in the next few days :)
Can't wait for that@@afaqueahmad7117
These UA-cam videos are so much more helpful. Hats down one of the best ones that explain the Spark performance tuning and internals in a very simplest of forms possible. Cheers!
In exchange hashpartitioning what is the significance of number 200? what does that mean?
200 is the default number of shuffle partitions. You can find the number here in this table by the property name "spark.sql.shuffle.partitions" spark.apache.org/docs/latest/sql-performance-tuning.html#other-configuration-options
Just 10 minutes into this notebook and I am awed beyond my words.
What a great explanation Afaque. Kudos to you!
Please make more videos of solving real time scenarios using Spark UI and one on Cluster configuration too. Again BIG THANKS!
Hi @mohitupadhayay1439, really appreciate the kind words, it means a lot. A lot coming soon :)
I have doubt when the data will be distributed to executor is it before scheduling the task or after scheduling the task and who assign the data to executor
Hey @sangu2227, this requires an understanding of transformations/actions and lazy evaluation in Spark. Spark doesn't do anything (either scheduling a task or distributing data) until an action is called.
The moment an action is invoked, Spark creates a logical -> physical plan and Spark's scheduler divides the work into tasks. Spark's driver and Cluster manager then distributes the data to the executors for processing :)
Hi Sir, you mentioned that you referred AQE before. Can I get that link ? I want to know about AQE
Yes, I will be releasing the video in the next few days. :)
@@afaqueahmad7117 Thank you sir.
Thanks for the content and when can we expect new video?
Coming soon, in the next few days! :)
You mentioned that for coalesce(2) shuffle will happen, but later you mentioned that shuffle will not happen in case of coalesce hence no partitioning scheme. Could you please explain it in detail?
So, coalesce will only incur a shuffle if its a very aggressive situation. If the objective can be achieved by merging (reducing) the partitions on the same executor, it will go ahead with it. In case of coalesce(2), its an aggressive reduction in the number of partitions, meaning that Spark has no other option but to move the partitions. As there were 3 executors (in the example I referenced in the video), even if it reduced the partitions on each executor to a single partition, it would end up with 3 partitions in total, therefore it incurs a shuffle to have 2 final partitions :)
@@afaqueahmad7117 Thanks for clarification.
What drawing board are you using for those notes?
Using "Notion" for text, "Nebo" on iPad for the diagrams
@@afaqueahmad7117cool thx!
Hi Afaque, how can I download the data files you are using? I want to try it hands on :)
Should be available here: github.com/afaqueahmad7117/spark-experiments :)
Hi sir i came across a doubt
Consider the executor size 1gb/executor. We have 3 executors and intially 3 gb data gets distributed across 3 executors each executor is having 1gb partition after various transformations we came across a requirment to decrease the number of partitions to 1 partition for that we will use repartition(1) or coalesce(1). In this scenario all the 3 partitions will merges to 1 partition each partition is having size of 1 gb approximately. Collectively all the partitions size is 3 gb approximately. When repartition (1) or coalesce(1) all the 3 gb data should sit in 1 executor having capicity of 1gb only. So here the data is execeeding the executor size what happens in this scenario. Could you please make video on this requesting sir.
Hi @bhargaviakkineni, In the scenario you described above where the resulting partition size (3 GB) exceeds the memory available on a single executor (1 GB), Spark will attempt to spill data to disk. The spill to disk is going to help the application from crashing due to out-of-memory errors however, there is going to be a performance impact associated, because disk IO is slower.
On a side note, as a best practice, It’s best to also think/re-evaluate the need to write to a single partition. Avoid writing to a single partition, because it generally creates a bottleneck if the sizes are large. Try to balance out the partitions with the resources of the cluster (executors/cores).
Hope that clarifies :)
Fab Cotenet
Local distinct on cust id doens't make sense and couldn't understand. How globally it does distinct count if the count is already computed. The reasoning behind why cast doens't push down predicate is not clearly explained and just as it's mentioned in the doc
Great Work buddy keep it up .... love your content, very simple to understand @Afaque Ahmed
Thanks a ton!