Nice explanation.A couple of questions 1) Repartitioning does ensure the data distribution is not skewed (unlike coalesce) 2) You said repartitioning uses the hash value to distribute the data (are you talking about bucketing ?)
If the partition key is non numeric then how to perform salting? like your tower ids were numeric, but if instead of being 1, 2, .. they are to be A, B, ...
video from 11:30, we are adding random key to exiting towerid key for Example. tower id: 101 and salt key : 67 then 101+67= 168 hash value of the 168 would be a final value right. what in case of partition column is string datatype. ??
@TechWithViresh I simply love your videos. I have watched your other tutorial videos too. They are awesome. I am interested in knowing how to do Iterative Broadcast Join with the SQL API. Any help is highly appreciated. Can you pls advise.
thank you so much , really good , so what is the difference b/w isolation salting and salting ? and what is difference b/w , isolation map join & map join ??
We could loose our key join by Salting key adding random numbers If we want to do join with the same key then problem May be join key could be the different on other than salted column
Thank you so much for the video. I seek some clarification though. In your example you did mapPartition. Means for each partition of different keys, you updated the key with salt. But still the records remained in the respective partitions only. How will those records be shuffled across partitions for equal distribution?
@@TechWithViresh I tried it so I believe a new DF will have to be created and REPARTITIONED again! in order for the records to be shuffle by updated salted keys. It wont just trigger shuffle on key update in mapPartition function! That only makes sense.
Very informative video 👍🙂
7:24, repartition does full shuffling and hence creates equal size partitions. i.e It always guarantess the equal sized partitions.
Great video, like the pace, like the presentation.
Glad you liked it!
Hello,
Your videos are very good,
Can you please do a video on incremental data load and full data load by taking an example?
Hi Viresh, Thanks for the video.... How can we achieve salting technique in Pyspark?
thank you so much for this video, but i am not able to find the 2nd part of this video.. Can you please comment the link for the part 2 video
Nice explanation.A couple of questions 1) Repartitioning does ensure the data distribution is not skewed (unlike coalesce) 2) You said repartitioning uses the hash value to distribute the data (are you talking about bucketing ?)
There are two provided partitioners in Spark 1. Hash partitioner and 2. Range partitioner.Default is Hash one.
If you repartition on column, there you can get skewed data. If you repartition by number of parts then distribution may be almost equal.
If the partition key is non numeric then how to perform salting? like your tower ids were numeric, but if instead of being 1, 2, .. they are to be A, B, ...
Thank you for making this video. Could you suggest on which column mean, medium and the mode are calculated?
The columns are those that are being shuffled by such as the join columns or group by columns. There is data skew when the distribution is not normal.
video from 11:30, we are adding random key to exiting towerid key
for Example. tower id: 101 and salt key : 67 then 101+67= 168 hash value of the 168 would be a final value right.
what in case of partition column is string datatype. ??
Incase of strings, we can add surrogate keys, based on string column values and then do the salting.
@TechWithViresh I simply love your videos. I have watched your other tutorial videos too. They are awesome. I am interested in knowing how to do Iterative Broadcast Join with the SQL API. Any help is highly appreciated. Can you pls advise.
Thanks for the video, no part 2 tho?
Hi Viresh, can you please share the link for part 2
thank you so much , really good , so what is the difference b/w isolation salting and salting ? and what is difference b/w , isolation map join & map join ??
We could loose our key join by Salting key adding random numbers
If we want to do join with the same key then problem
May be join key could be the different on other than salted column
Thank you so much for the video. I seek some clarification though.
In your example you did mapPartition. Means for each partition of different keys, you updated the key with salt. But still the records remained in the respective partitions only. How will those records be shuffled across partitions for equal distribution?
Partition will change with the change in the key, as it is essentially the hascode of key+salt now.
@@TechWithViresh I tried it so I believe a new DF will have to be created and REPARTITIONED again! in order for the records to be shuffle by updated salted keys. It wont just trigger shuffle on key update in mapPartition function! That only makes sense.
Hi sir, pls upload the spark interview question videos which were present earlier.. I'm not able to find them in your playlist
All the videos are uploaded, please check:)
Awesome video 🙏...can you pls share part2 video link
Coming Soon!! , Thanks :)
@@TechWithViresh when ?
where is Part 2 ?
please give some solid coding example with explaination
have u uploaded part 2 of this
Check out other videos in the playlist for performance optimization and executor tuning.
can u share part 2 video
Coming Soon!! , Thanks :)