What is Apache Spark? Learn Apache Spark in 15 Minutes

[@5t1300]

Great job breaking this down with clear examples

[@MohanKrishna-yi9cc]

OMG , I even tried many UDEMY courses to understand this. None of the tutor explained this clearly .. iam loving it ... Sir please start full databricks course to help us. Please.. 🙏

[@mr.ktalkstech]

Thank you so much :) Sure, will do :)

[@slicein]

Fantastic. Thank you for such an easy and efficient explanation. The restaurant example is apt for spark. Great work👌👍🙏❤❤

[@kiranzsai]

Excellent !! thank you for the explanation!! Thats Y I subscribed n became member to your channel!!

[@dprakash1793]

this is what i was looking for well explained. thank you

[@mr.ktalkstech]

Thank you so much :)

[@manikandan-fq5sh]

simply great explanation about SPARK architecture, how its connected step by step it connects all the dots in Spark.

[@mr.ktalkstech]

Thank you so much :)

[@manikandan-fq5sh]

@@mr.ktalkstech looking further concept in Spark, it would be great if you try full course

[@emily-bose]

Excellent and one of the best explanation of spark architecture...

[@dvg_ck]

Simple and brilliant analogy Mr K

[@shyammaths5705]

this is so simple and clear explanation that
it made to share to my friends.
keep making video
your efforts putting great impact in our life.

[@mr.ktalkstech]

Thank you so much :)

[@rakeshverma6867]

Simplest and excellent explanation Mr K.

[@mr.ktalkstech]

Thank you so much :)

[@062nanthagopalm6]

Wow! Just mind blowing brother💥💥!! Looking for more DE fundamentals videos ✨♥️👌

[@mr.ktalkstech]

Thank you so much :)

[@sharaniyaswaminathan8760]

Excellent! Thank you for explaining this.

[@mr.ktalkstech]

Thank you so much :)

[@benim1917]

Clear and well explained

[@mr.ktalkstech]

Thank you so much :)

[@smderoller]

Very well explained!!!

[@mr.ktalkstech]

Thank you so much :)

[@Bijuthtt]

Awesome explanation bro.

[@mr.ktalkstech]

Thank you so much :)

[@digitalabi]

I appreciate your explanation; it has clarified the topic for me. Thank you. 🙏🏼
However, I have one question: if the CSV files are split into two, how will one worker determine if there are any duplications with another worker work?

[@Abhinavkumar-kt8gj]

Excellent!

[@mr.ktalkstech]

Thank you so much :)

[@seethaba]

Great primer @Mr. K! Thanks. Quick question - How does the driver program create task partitions for the plan? For example, if there are duplicates across two worker nodes, wouldn't the count be misrepresented if it simply adds 4500 and 5500? Does this get auto-handled or do we have to control the partitioning logic?

[@Bhavik_P89]

According to number of partitions of the files. You can also control over task by setting up configuration of partitions limit after each transformation.by using below code
spark.conf.set("spark.sql.shuffle.partitions", num_partitions)
The task is always depends on the number of partitions.
Your question is that each and every worker nodes have duplicates and in count operation it will just sum the results right.
Ans- after getting the result from each worker nodes the driver program will again aggregate it and then give the final result

[@selvakumarr.k.8660]

Useful presentation

[@mr.ktalkstech]

Thank you so much :)

[@dogzrgood]

Great explanation. Do you have a full pyspark tutorial?

[@shabeerkhan379]

Really good

[@AlexFosterAI]

hey man, may be worth a shot checking out LakeSail's PySail built on rust. supposedly 4x faster with 90% less hardware a cost according to their latest benchmarks. and can migrate existing python code. might be cool to make a vid on!
love ur content!

[@neeraj_dama]

thanks for this

[@zakeerp]

Hi , what tools used to create this type of videos. Please help.

[@mr.ktalkstech]

Final cut pro, CapCut, PowerPoint and After effects.

[@zakeerp]

@@mr.ktalkstech thank you for the info

[@mgdesire9255]

waiting for your pyspark playlist:)

[@mr.ktalkstech]

Very soon :)

[@satish1012]

This is my understanding
- Apache Spark falls under the compute category.
-It's related to MapReduce but is faster due to in-memory processing.
-Spark can read large datasets from object stores like S3 or Azure Blob Storage.
-It dynamically scales compute resources, similar to autoscaling and Kubernetes orchestration.
-It processes the data to deliver analytics, ML models, or other results efficiently.

[@PatelTushya]

Respect++

[@kirankarthikeyan4940]

will this same topic be covered in the other channel (Mr.K Talks Tech Tamil)

[@mr.ktalkstech]

No brother :)

[@RijyosInfo-wn6wt]

RESPECT++++++++

[@neuera9556]

Did not said about rddd