Its ok, but seems like its getting covered like a common theory session skipping some parts here and there. I feel it could have been better in case it would have been demonstrated practically. I don't see anything talked about the performance improvement each of these optimizations caused, neither was skewness or spill shown in spark UI.
Hello Bhavana, If we are introducing salted skews, then will it not introduce more of shuffles when performing the joins? pls answer
After spark 3.0 AQE already enabled...so skewness can easily handled...
Its ok, but seems like its getting covered like a common theory session skipping some parts here and there. I feel it could have been better in case it would have been demonstrated practically. I don't see anything talked about the performance improvement each of these optimizations caused, neither was skewness or spill shown in spark UI.
Thanks Bhawna!
Great