Відео

Industry level content.

Is this applicable to cloud platform?

I never see such a detailed explanation. Thank you very much @afaque Ahmad. Is there any way we can get the document.

Awesome video! I've been searching for something like this and all the other videos I found don't get to the point and neither explain things as good as you do. I'm definitely subscribing and sharing this with other DE's from my team, please keep posting content like this!

Very informative video afaque. Please create more videos.

What if the nodes are of unequal size?

Excellent.. Keep going..

Am.not able to see spill details in spark 3.5.2 UI?

Excellent Content..!! Loved It. Such detailed explanation on Salting Technique with Graphical Representation.

Hi Sir Please Make a Detailed course on apache spark which include every aspect of spark for Data Engineer role Please make sure there are a lot of begineer course here in market keep the course from intermediate level to advance level. Please tr to make video in Hindi it will be very helpful.

Thank you for this video !!

Sir please do a video on executor out of memory in spark and driver out of memory in spark

This is helpful, but I still have a few doubts. 1. If Broadcast join is immune to skewness, why there is Salting technique? 2. In the Broadcast join example, the customer dataset appeared to be outside of any executor. Where is it actually stored? How can we specify its storage location?

Want to understand better, Read this: Spark CacheLLM Response: 1. **Why Use Caching?** - Caching can significantly improve performance by reusing persistent data instead of recomputing it - It helps avoid redundant computations on the same dataset across multiple actions 2. **Lazy Evaluation and Caching** - Apache Spark uses lazy evaluation, where transformations are not executed until an action is triggered - Caching can help by materializing the result of a long sequence of transformations, avoiding recomputation 3. **Spark's Lineage Graph** - Spark tracks the lineage of transformations using a Lineage Graph - Caching breaks the lineage, reducing the size of the graph and improving performance 4. **Caching vs. No Caching** - The demo shows a significant performance improvement when caching is used, as seen in the Spark UI 5. **Persist and Storage Levels** - The `persist()` method is used for caching, with different storage levels available - Storage levels like `MEMORY_ONLY`, `DISK_ONLY`, and combinations control memory/disk usage and replication - Choose the appropriate storage level based on your requirements and cluster resources 6. **When to Cache?** - Cache datasets that are reused multiple times, especially after a long sequence of transformations - Cache intermediate datasets that are expensive to recompute - Be mindful of cluster resources and cache judiciously 7. **Unpersist** - Use `unpersist()` to remove cached data and free up resources when no longer needed - Spark may automatically unpersist cached data if memory is needed If you liked it, Upvote it. NarutoLLM Response

Great tutorials 🙏, please create more videos on spark from beginners point of view

Your channel is so underrated, Please dont stop

i like very much of your videos, it's insightful. can you please make series/videos on Spark interview oriented questions. Thanks in advance

Hi Afaque. A suggestion. You could start from the beginning to connect the DOTS! Like if in your scenario we have X Node machine with Y workers and Z exectors and if you do REPARTITION and fit the data like this then this could happen. Otherwise the Machine would sit idle and so on.

Perfect explanation & perfect examples throughout the playlist, Bhai mere Change data capture aur Slowly changing dimension jo bhi apply hote hain project me uska bhi khel samza de.

a lot of knowledge in just one video

I spent INR 42000 on a Big Data course but could not understand this concept clearly because the trainer himself lacked clarity. Here I understood completely.

Great explanation of such a complex topic, thanks and keep up the good work.

Great explanation, Thank you, But how would we know how to configure exact (at least best) "spark.sql.shuffle.partitions" at run time? Because each run/day the volume of the data is going to be changed. So, how do we determine the data volume at run time to set the shuffle.partitions number?

Awesome explanations! pls share us more relevant videos

Dead gorgeous stuff.

Hey Afaque Great tutorials. You should consider doing a full end to end spark project with a Big volume of data so we can understand the challenges faced and how to tackle them. Would be really helpful!

Not easy to understand, but it great

Useful information

Thanks for the video! I also have a question: when I execute complex query, there will be multiple stage with different shuffle write sizes, which do I have to take in consideration for the computation of the optimal number of shuffle partitions?

All the concepts are clearly explained. Please do more videos.

Your teaching skill is very good, please make a full series on PySpark, that'll be helpful for so many aspiring data engineers.

Please make one on AWE aswell

Excellent 🎉 👍 appreciate your effort

Correct me if I'm wrong, but these calculations consider the execution of only one job at a time. How do the calculations change when there are multiple jobs running in a cluster, as often happens?

Buddy! You got a new sub here. Loved your detailed explanation. I see no one explaining the query plain this detail and I believe this is the right way of learning. But I would love to see an entire Spark series.

Why to give such a large fraction (0.4) to User memory as in the end when the transformations will be performed in a particular stage , whether we give it a user defined function or any other function execution memory will be only used . So Whats exactly the role of User Memory ??

could you please make video on stack overflow like what are scenario when it can occur and how to fix it

Thanks for the content, really appreciate it. My understanding is AQE take care of Shuffle Partition Optimization and we don't need to manually intervene (starting spark 3) to optimize shuffle partitions. Could you clarify this please?

Hey man. learnt a lot from the video. please help me out on this doubt for example 2, total executors = 44/4 = 11 you have said. But shouldn't we think machine by machine, here each machine can have, 15/4 === 3 executors if 4 core for each, giving total 3*3 nodes = 9. in your workout, it seems like there will be an executor which will use some cores from one node and some from other. Am I wrong in my thought process somewhere?

sorry if I am asking very basic question, can we set executors per spark job or per spark cluster? Also how to set this up using coding examples and all

Every-time I come here before attending an interview , I try to give this video a like , but end up realising that I already did it earlier. Best video on this topic on whole internet.

Really appreciate your efforts. This was very easy to understand and comprehensive as well.

One of the cleanest explanation I ever come across on the internals of Spark. Really appreciate all the effort you are putting into making these videos. If you don't mind, May I know which text editor are you are using when pasting the Physical plan?

Excellent and clear cut explanation, thanks much for taking time and preaparing the content. Please do more.

This is the most detailed explanation I have ever seen.

Explanation is so good

Hey @Afaque, great content as usual, but I thought this video could be a little concise, great work anyways!

Consider a scenario where my first data shuffle size is 100gb then giving more shuffle partitions make sense now in the last shuffle data size is drastically reduced to 10gb according to calculations how would be to give shuffle partitions giving 1500 would benefit for the first shuffle and not for the last shuffle. How do one approach this scenario

It's very useful ❤

perfect video