Spark Memory Management

[@prafuljain3657]

This is very helpful video to start with spark memory management. Thanks for posting

[@svdfxd]

Great explanation on Spark Memory Management.

[@santoshkumarthammineni3897]

Thanks for your contributing your knowledge and experience to IT society.

[@BigDataThoughts]

Thanks Santosh

[@nnkumar13]

Hi Shreya, Very clear and precise info. Hoping for more on spark performance issues.

[@BigDataThoughts]

Thanks Naveen

[@ranjithpals]

Very insightful!! Great job Shreya!

[@BigDataThoughts]

Thanks Ranjith

[@rajnimehta5156]

Awesome and concepts are very nicely guided

[@BigDataThoughts]

Thanks Rajni

[@dn9416]

Hi Shreya, thanks for the video, all viewers be safe

[@BigDataThoughts]

Thanks DN

[@himanshgautam]

This is gold .

[@BigDataThoughts]

Thanks Himansh

[@venkatesanr9455]

1 second ago
Hi Shreya, Thanks for the video, wonderful presentation, and knowledge sharing. you are doing a great job and continue. Also, I like you to do practical experiments like spark in the future for practicing purpose.

[@BigDataThoughts]

Thanks Venkatesan

[@devanand9943]

Well explained. Please make videos on performance tuning in Spark

[@BigDataThoughts]

Thanks Anand. yes have plans to make on performance

[@ait8954]

Nice explanation 👌👍👏

[@gauravsehrawat2191]

This is what I am looking for
Thankyou 🙏

[@BigDataThoughts]

Thanks Gaurav

[@shivratanmishra1459]

Just awesome tutorial.

[@BigDataThoughts]

Thanks Shivratan

[@shankarsaripalli3260]

Clear and lucid presentation mam 🙂

[@BigDataThoughts]

Thanks Shankar

[@shivamahankali4252]

nice explanation mam

[@BigDataThoughts]

thanks shiva

[@GuruVideoAC]

Good job!

[@BigDataThoughts]

Thanks

[@SagarSingh-ie8tx]

Very nice

[@rajnimehta9189]

Its just awesome

[@BigDataThoughts]

Thanks Rajni

[@arnabdutta7876]

around approx 07:20 - 08:00 mark, isnt the spark.memory.fraction and not spark.memory.storagefraction value that defines the amount of memory available for Unified memory(Execution+Storage). Out of that default spark.memory.storagefraction defines how much of that memory can be blocked for Storage so that amount of data doesnt get evicted?

[@dhanunjaymantri1442]

yes the spark.memory.storageFracation is fraction of storage space out of total space allocated by spark.memory.fraction for execution and storage.

[@guptaashok121]

Why there is no need of GC in off heap, how does that do clean up.

[@anirbansom6682]

GC is only applicable for a JVM process. The off-heap memory is managed outside the executor JVM process. That's why GC cycle on executor JVM doesn't clean off-heap memory

[@udaynayak4788]

Can you please share information on calculating num executors, memory core, memory drivers

[@BigDataThoughts]

will make a video on that

[@vishalmishra1937]

plz cover encoders in spark

[@pranayr1718]

what is Kyro. you used this jargon so asking? how broadcasting works in detail is another question?

[@guptaashok121]

Thanks for this. Could you explain the meaning of propagating internal data between cluster, which is done in storgae memory, is it not same as shuffling, which is taken xare in execution memory

[@ravikirantuduru1061]

Can you make videos with practical example and can we use kryoserrialser in pyspark.

[@BigDataThoughts]

kyro is for java

[@guptaashok121]

If you could explain user memory in little more details.

[@ferrerolounge1910]

Good video but noise in background