Thank you nilendra for the video, hope to see more videos reg this topic in your channel. One small suggestion please dont add back ground music in upcoming videos..😊
Hi Nilendra, Thank you for sharing .... I have one famous question, how many cores and executors are required for 100 gb of data ?? Could you please help us in understanding this ... Thanks in advance
The number of cores and executors needed to process 100 GB of data will depend on several factors, such as the complexity of the processing logic, the type of data, and the resources available on the Spark cluster. Here is a rough estimate based on some general assumptions: A single executor can process about 10 GB of data in a reasonable amount of time. (Again depends on core capacity ) A single core can handle about 2-3 executors, depending on the processing logic and the amount of memory required. Based on these assumptions, processing 100 GB of data would require: 10 executors * 2-3 executors per core = 20-30 cores However, this is just a rough estimate and your actual requirements may vary. It's always best to conduct performance testing with representative data to determine the actual resources required for your specific use case.
Great series . Please include a session on how to handle skew in a join , Bucket join vs shuffle hash join .
Thats next video. Will get published this week. Please share with your friends!
Thank You bro for the video
My pleasure
Thank you nilendra for the video, hope to see more videos reg this topic in your channel. One small suggestion please dont add back ground music in upcoming videos..😊
Suggestion noted. Next video uploaded too :)
Thanks 👍
Welcome 👍
Thanks sir 🙏
Hope you like the video and please share !
Hi Sir, i would request to please remove the background music 😊. However your content is awesome
Point noted. But to me it seems with my voice only video, viewers might get bored. Haha
@@dataengineeringforeveryone if content is good then nobody will get bored. Content is key.
This is my video on installing pyspark. I am also a data engineer
ua-cam.com/video/nOSXvFd4hoY/v-deo.html
@@dataengineeringforeveryone let the bg music be but reduce the volume level and not dominate your voice. Great content btw 😊
Can you provide link of this document
Hey unfortunately my onedrive is full and not sharable at the moment. I need to upgrade it.
Hi Nilendra, Thank you for sharing ....
I have one famous question, how many cores and executors are required for 100 gb of data ?? Could you please help us in understanding this ... Thanks in advance
The number of cores and executors needed to process 100 GB of data will depend on several factors, such as the complexity of the processing logic, the type of data, and the resources available on the Spark cluster.
Here is a rough estimate based on some general assumptions:
A single executor can process about 10 GB of data in a reasonable amount of time. (Again depends on core capacity )
A single core can handle about 2-3 executors, depending on the processing logic and the amount of memory required.
Based on these assumptions, processing 100 GB of data would require:
10 executors * 2-3 executors per core = 20-30 cores
However, this is just a rough estimate and your actual requirements may vary. It's always best to conduct performance testing with representative data to determine the actual resources required for your specific use case.
@@dataengineeringforeveryone Great Thanks for taking some time in replying... Your videos helped us a lot ... Thanks again....
Bhai background music mat use Kar.
Noted bhai!!
❣️ 'Promo sm'