Awesome video! I've been searching for something like this and all the other videos I found don't get to the point and neither explain things as good as you do. I'm definitely subscribing and sharing this with other DE's from my team, please keep posting content like this!
Thank you Bro!! your videos are very informative and helpful. Can you please one video explaining setting up spark in local machine. That will be very helpful
At 16:49, as part of the AQE plan for the larger dataset, the way that I understood is 1 skewed partition was split in 12 and finally we had 24+12 = 36 partitions. We see the same on Job Id 9 at 13:40 that it had 36 tasks. But I heard you say that 36 partitions have been reduced to 24. Can you please help clear the confusion ? thank you.
I think in that AQE Step, AQEShuffleRead reads 200 partitions (as per previous node) from customers dataset, then coalesced to 24 then something happened and make them to 36 thats why that right side node is showing "number of of partitions 36". At left side for transactions dataset, this "number of of partitions 36" is appearing as last value where at right side for customers dataset its appearing as first value. But Im not sure what is that " something"???
Hi Afaque Ahmad At 13:37 you were saying that separate job for shuffle operation that one job for transactions dataset shuffle operation and one for customers dataset. Im bit confused why they need a separate job? As per my understanding, when spark encounters a shuffle operation, it just creates a new stage within that job right? When I execute the same code snippet, it create 5 jobs totally: two for metadata (expected), two for shuffle operation (not expected) and final one is for join operation. Many thanks
Hi Afaque Ahmad At 7:24, you were saying that a batch is a group of rows and its not same as a partition. Shall we assume something like a group of rows read from one or more partitions available in one or more executors (not from all executors) to match that df.show() count?
Hey @subaruhassufferredenough7892, Thanks you for the kind words, really appreciate it :) On Spark SQL, DAGs/Execution plans for both Spark SQL and non-SQL (python) are the same as they are compiled/optimized by the same underlying engine/catalyst optimizer.
Hello Bro, I have a doubt. at "23:30 min" playtime, it was mentioned that AQEShuffleRead: coalesced partitions into 1, then will the other worker nodes will sit ideal ? In the Video it is mentioned that even after shuffle, all A's will be in 1 partition and B's in another partition. can you please explain me, what do you actually mean by Number of Coalesced Partitions=1
Hey @rohitsharma-mg7hd, appreciate your attention to the video. Yes, it should be A1 B1 instead of A2 B1. If you've watched the complete video I corrected it at 29:57 :)
@@afaqueahmad7117 thanks i did only half video since i was also doing practical on my system along with you. Surely i will watch it today. very good luck to you . very good content
By default, shuffle partitions are 200, hence you see that in the 'Exchange' step. The reduction (optimization) to fewer partitions takes place in the 'AQEShuffleRead' step below.
Practically explained the AQE runtime optimization. Good Work!
Glad you found it helpful :)
🔔🔔 Please remember to subscribe to the channel folks. It really motivates me to make more such videos :)
Done - awesome videos will watch the rest of the series. Would be great to get some databricks oriented videos also when possible
this is probably the best explanation I've seen on spark DAG's. Please keep up the amazing content! thank you
again in depth content. Thanks a lot. Please discuss a scenario based question on todays topics.
Beautiful content. Very clear and crystal explanation. Thank you for doing this. ❤❤
Glad you enjoyed it!
Awesome video! I've been searching for something like this and all the other videos I found don't get to the point and neither explain things as good as you do. I'm definitely subscribing and sharing this with other DE's from my team, please keep posting content like this!
Appreciate the kind words @joseduarte5663 :)
Bro..Can't thank you enough... This is what exactly I was looking... Thanks a ton bro... 🎉
Nice serie about performance, waiting for more videos, tranks.
Doing fantastic work bro.... Keep this up 💪❤
very good explanation, thanks. you earned a new subscriber
@Afaque asually amazing vedio bro. It's been more than 1 month we are dying of waiting for vedios from you
A new playlist coming soon brother :)
@@afaqueahmad7117 waiting
amazing explanination ..Waiting for more videos from you
Great explanation bro,thanks for the great work you are doing
Thank you @Learner1234-hv4be, really appreciate it :)
It's really informative session, thank you!!
Amazing. This is just too good. Will share with my team also.
Really appreciate it @HarbeerKadian-m3u :)
Explained so well!! Crystal clear!
Done with the second video on this channel. See you tomorrow again.
Thank you Bro!! your videos are very informative and helpful. Can you please one video explaining setting up spark in local machine. That will be very helpful
Thanks @ankursinhaa2466, videos on deployment (local and cluster) coming soon :)
please make a full pyspark tutorial.. This is very interesting topic and explained very nicely 👍
Thanks @RaviSingh-dp6xc, more PySpark content coming soon! :)
Nice explanation.Great work.Thank you .Liked and Subscribed.
Thank you @balakrishna61, appreciate it :)
At 16:49, as part of the AQE plan for the larger dataset, the way that I understood is 1 skewed partition was split in 12 and finally we had 24+12 = 36 partitions. We see the same on Job Id 9 at 13:40 that it had 36 tasks. But I heard you say that 36 partitions have been reduced to 24. Can you please help clear the confusion ? thank you.
I think in that AQE Step, AQEShuffleRead reads 200 partitions (as per previous node) from customers dataset, then coalesced to 24 then something happened and make them to 36 thats why that right side node is showing "number of of partitions 36".
At left side for transactions dataset, this "number of of partitions 36" is appearing as last value where at right side for customers dataset its appearing as first value.
But Im not sure what is that " something"???
Hi Afaque Ahmad
At 13:37 you were saying that separate job for shuffle operation that one job for transactions dataset shuffle operation and one for customers dataset.
Im bit confused why they need a separate job? As per my understanding, when spark encounters a shuffle operation, it just creates a new stage within that job right?
When I execute the same code snippet, it create 5 jobs totally: two for metadata (expected), two for shuffle operation (not expected) and final one is for join operation.
Many thanks
very well explained!!
your expertise and explanations is like "filtered gold in one can " Can you make quick video on AQE in depth please. 1000 thanks
Thanks @jdisunil for the kind words. There's already an in-depth video on AQE.
You can refer here: ua-cam.com/video/bRjVa7MgsBM/v-deo.html
Hi Afaque Ahmad
At 7:24, you were saying that a batch is a group of rows and its not same as a partition.
Shall we assume something like
a group of rows read from one or more partitions available in one or more executors (not from all executors) to match that df.show() count?
Could you also do a video on Spark SQL and how to read DAGs/Execution Plans for that? Amazing video btw, subscribed!!
Hey @subaruhassufferredenough7892, Thanks you for the kind words, really appreciate it :)
On Spark SQL, DAGs/Execution plans for both Spark SQL and non-SQL (python) are the same as they are compiled/optimized by the same underlying engine/catalyst optimizer.
Please make a dedicated video on shuffle partition... how it behaves when it's increased or decrease from 200
Hey @SHUBHAM_707, have you watched this - ua-cam.com/video/q1LtBU_ca20/v-deo.html
Hello Bro,
I have a doubt. at "23:30 min" playtime, it was mentioned that AQEShuffleRead: coalesced partitions into 1, then will the other worker nodes will sit ideal ?
In the Video it is mentioned that even after shuffle, all A's will be in 1 partition and B's in another partition.
can you please explain me, what do you actually mean by Number of Coalesced Partitions=1
Amazing content. Keep it up
Appreciate it :)
Thank you very much for the explanation. But I want to know what is your source of knowledge. Where do you learn these things?
awesome explanation.👍
beautifully explained!
How do you access this spark UI?
Excellent bro
Thanks for this. When is the next video coming sir?
Coming soon in the next few days! :)
29:10 pe a1 b1 (in green) ayega instead of a2 b1 (green)
Hey @rohitsharma-mg7hd, appreciate your attention to the video. Yes, it should be A1 B1 instead of A2 B1. If you've watched the complete video I corrected it at 29:57 :)
@@afaqueahmad7117 thanks i did only half video since i was also doing practical on my system along with you. Surely i will watch it today. very good luck to you . very good content
Superb ❤
thank you
Appreciate it :)
why shuffle partitions made 200 hundread. when we have only 13 partitions max. at 14:55
By default, shuffle partitions are 200, hence you see that in the 'Exchange' step. The reduction (optimization) to fewer partitions takes place in the 'AQEShuffleRead' step below.
Bhai , can help to make a video on spark architecture as well for beginners
Haan bhai, ayega kuch time mein :)
While stages are created whenever a shuffle occurs, how are jobs created?
Hey @satheeshkumar2149, jobs are created whenever an actions is invoked. Examples of action in Apache Spark can be - collect(), count()
@@afaqueahmad7117 , but in some cases we have more than one job being created. This is where I find difficulty in understanding
No clarity is provided on when job is created. The stages are result of shuffle. The task is just a unit of execution