Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code. www.learningjournal.guru/courses/
This is the best video....wt a explanation sir..mind blowing. I feel bad where bad teacher get too much attention where people like you don't get much... U r brilliant
I started learning Spark by enrolling to a self learning course in UDEMY but by far this is the best video i have ever watched which explained the core concepts of SPARK clear and precise. I appreciate your efforts
I like the animal logos for the 3 APIs, turtle for RDD (slowest), cat for SQL-Dataset (medium), rabbit for DataFrame (fastest), but see Brian M. Clapper recent video on Frameless API (fast, compile-safe, and more functional, i.e. can compose actions).
It should be a=> (a(1),1)) to get the second field if you want the result to be as shown in 9:30. First field of the array is empty string, but you can still reduce by key since it will be all empty string at the end but not if all files didnt contain /etc root path. I find it useful to run collect() at each step like kvRDD.collect() to see. Thank you very much for the best spark tutorial, I let the adds run to help out.
Thank you for the great content!! just one request though please add a highlighter to your pointer it is kind of hard to track its movement and often have to rewind to check what actually you clicked on
Thanks a lot, sir for your outstanding efforts in making us brilliant. Could you please add some more spark videos shared variables, detailed transformations, and actions. Really I am double satisfied with your explanations, going forward we want to see more from you on spark.
Nice video with very clear explanation. But We will have to wait very long for a new session . Please try to upload fast, otherwise it will take a year to learn Spark.
Hi Prashant Sir , First things first : I am planning to take-up the course . Your explanation viz etc. are awesome kudos for that. The only thing that concerns me is that I have lot of questions when I study anything , some silly as well. Is there any channel (Slack/Discord etc for enrolled students) , where the doubts are cleared. Some AMA kind of sessions etc , becuse going through stuffs and having uncleared doubts will leave a learner is almost the same state. Will share the same message with you over lnkedin as well , not sure how frequently you look into UA-cam comments.
hello... I had a question. The intermediate RDDs which are generated, the partitioned data under them would also be distributed to the executors? or would the redistribution only happen on an action? Please help! :)
Hi Thanks a lot for these videos. They are quite helpful. In this video you mentioned that RDD is immutable, but you have overridden same RDD right, by changing number of partitions. Also we can load different text file into the same named variable(RDD). Could you explain how it is immutable in this case. Thanks in advance for your help.
I am confused in one step. When we say, RDD distributes data into nodes. We create 5 partitions from RDD. It means RDD has loaded all the data and do partitioning, is it? Will it load data from different nodes to the driver node and keep it in memory and distribute across? If yes, it is not following data locality paradigm and data movement is very costly. (I am sure spark follows data locality) What am I missing?
Sir i have executed the textFile() command with 5 partitions in cluster mode (5 W) but every time I could see that job is being executed only 1 of the workers. I mean in every run only 1 worker was executing all the partitions. is there any extra configuration required here. i am using spark-shell mode
Hello Sir, loved this tutorial..thanks a lot. I have one doubt, consider following scenario: Input data size(raeding from hdfs): 20 GB No of executors: 2 executor memory : 8 GB RDD partition factor: 2 and we run a spark job in client mode. So in this case: 1. how total 20GB data will get processed through sparkjob? 2. How many stages and task will get created? 3. how total 20gb data will be partitioned?
1. Do you need 20GB memory to process 20 GB data? No. More memory can improve performance but you can still process it with less memory. 2. Stages depend on your logic and the number of task on executors. 3. You asked for two partition so it will shuffle and make it two in that stage. Next stage depends on other factors.
Thanks for the wonderful explanation in this video. Can you please tell if this is a general scenario for each job in spark, that map and reducebykey operations usually run in 2 different stages always or if there are cases that they can run in a single stage as well. Any examples or leads would be appreciated!
Also if possible please explain the code using Java APIs too. I am doing development using Java API, but some methods are not supporting even after it's mentioned in API document and throwing run-time error. Is that when we are doing development using Java API or Python API, will it get converted to Scala language internally?
Hello,Great tutorial. Can you please elaborate more on reduceByKey((x,y) => x+y) that you have used to count number of pairs with same key. I am a bit confused how x+y will give us the total count
Here reduceByKey is aggregating the result of array (for similar key) to one value. Suppose you have (key, value) list :- List((hello, 1), (world, 1), (hello, 1), (hello, 2)). reduceByKey will perform operation on similar key and x, y denotes the value only from key, value pair (you cannot perform operation on key here) For key: hello rdd.reduceByKey(x,y => x+y) -equivalent to (1,1 => 1+1) => List((hello, 2), (world, 1), (hello, 2)) There are still pair exists belongs to hello key here. so again operation will be perform rdd.reduceByKey(x,y => x+y) -equivalent to (2,2 => 2+2) => List((hello, 4), (world, 1)) Now list has only one hello key pair, so no further reduction can be possible here.
I am using spark 2.2.0 but the code shown by you in the video doesn't print any output in shell. val myrdd = sc.textFile("UserData.txt",4) myrdd.foreachPartition(x => println("No. of elements in partition: " + x.count(y=>true))) Please share the workable code.
hello , so first of all i want to thank you for this superb tutorial. please i have one question following your example of imagin we have 10 partitions and 2 executor and we lets suppose in this example we dont have transformation that gona cause shuffle how many task parrallel there is it 5 ? thank in advance for your answear
Why do you think it's going to be 5? Because 10 partitions /2 executors? Number of executor have nothing to do with how many tasks are created. Once tasks are created, they will run on only two executors because you have only two executors.
@@ScholarNest first of all thank for your quick reply. so i undersntand in my example number of task created per stage depend only by number of partition . number of execeutor have nothing to with it im a correct plaese ? so i my example i will still 10 task because i have 10 partition ?
Sir I am doing development using spring Kafka, could you please help me with consumer question? how do i poll in regular intervals e.g every 5 mins and how do I specify number of records to be read in every poll? I saw batch listener can be used to specify number of records to read but did not find polling interval option. Thanks.
Yes sir i have gone through ur videos. my requirement is as a consumer i want to put delay with every poll and also want to control number of records being read in every poll.
+Jainesh Modi to be honest, I haven't used spring Kafka, just saw the documentation. Looks interesting. I will plan for some time to evaluate it and send you details if I find an answer to your problem.
Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code.
www.learningjournal.guru/courses/
I really like your content, very easy to understand. THANK YOU. Have you covered any where how RDD helps recover from fault tolerance?
This is the best video....wt a explanation sir..mind blowing. I feel bad where bad teacher get too much attention where people like you don't get much...
U r brilliant
I love the curtains opening up special effect!
Your video is one of the best for Spark, not spend too much on theory and high level, but down to the earth, very practical.
Even beginners can understand the flow who has zero knowledge about spark.. Great explanation 😊
This is the easiest and most explanatory explanation of complete spark architecture one can ever get
Wow ! Very crisp and to-the-point explanation. Really helpful. Thank you Prashant!
Hands down the best explanation you'll find on youtube
I started learning Spark by enrolling to a self learning course in UDEMY but by far this is the best video i have ever watched which explained the core concepts of SPARK clear and precise. I appreciate your efforts
extremely lucid and to the point...congrats !
Images, Colors, examples and clear explanations! This video has everything! Keep up the good work! Thank you Sir.
Exceptional way of explaining and making concepts crystal clear. I am enjoying it the way I used to enjoy your earlier videos on Hadoop.
Awesome video with correct explanation.
Highly recommended for every Spark newbee. BTW thanks a lot..
Fantastic video,I have subscribed your video!
the best tutorial i have ever seen..simply awesome
Just brilliant 👌.. point to point..
Best video by far.. What a teacher you are.
I like the animal logos for the 3 APIs, turtle for RDD (slowest), cat for SQL-Dataset (medium), rabbit for DataFrame (fastest), but see Brian M. Clapper recent video on Frameless API (fast, compile-safe, and more functional, i.e. can compose actions).
It should be a=> (a(1),1)) to get the second field if you want the result to be as shown in 9:30. First field of the array is empty string, but you can still reduce by key since it will be all empty string at the end but not if all files didnt contain /etc root path. I find it useful to run collect() at each step like kvRDD.collect() to see. Thank you very much for the best spark tutorial, I let the adds run to help out.
Please post more videos,I following all Ur video,Ur videos are something different to others.....it easily understandable way
you mean Data Savvy :P , Yeah Prashant is really good in explanation
Thank you ver much!, nicely explained spark architecture. there is no other better way than this.. keep the good work.!
Great tutorial and Excellent teaching
It is really appreciated,i never ever seen an explanation like this,so thanks a lots sir for revealing such extraordinary skills.....
Excellent way of teaching ...Thank you
Simply Awesome explanation...
Thank you sir...super video
Your tutorials are simply awesome.. :-) Super Like
Outstanding..
Great video
excellent teaching skills,thanks a lot :)
This is an extraordinary explanation of spark architecture.
Sir, please pick a few examples to implement on cluster mode too.
Thank you for the great content!! just one request though please add a highlighter to your pointer it is kind of hard to track its movement and often have to rewind to check what actually you clicked on
Very very good explaination sir, am very much thankfull to you.
nicely explained.
Thanks a lot, sir for your outstanding efforts in making us brilliant. Could you please add some more spark videos shared variables, detailed transformations, and actions. Really I am double satisfied with your explanations, going forward we want to see more from you on spark.
excellent explanation !
Simply Best!
Great explanation sir. Waiting for new concepts to be covered in future videos.
just awesome
Nice way of explanation. Thank you so much for your effort on making so nice tutorials. I am becoming a fan of you man! keep it up (Y)
Very Good video. Thanks sir. But I didn't anything with Apache Spark -05 in Playlist. Do we have some more videos on architecture?
Great video 🤓🤓sir
nice explanation
Nice video with very clear explanation. But We will have to wait very long for a new session . Please try to upload fast, otherwise it will take a year to learn Spark.
It's really amazing it's really real time level
Very nice video sir. Thank you.
Great explanation !!
Really amazing explanation thank you
Thank you so much, Sir, for providing such a nice practical explanation.
very very amazing . thank you
thanks for sharing your knowledge.
How do you create animation shown from 4:50 to 4:55. Which software. I like how you bring clarity visually.
Power point :-) Office 365
Hi Prashant Sir ,
First things first :
I am planning to take-up the course . Your explanation viz etc. are awesome kudos for that. The only thing that concerns me is that I have lot of questions when I study anything , some silly as well.
Is there any channel (Slack/Discord etc for enrolled students) , where the doubts are cleared. Some AMA kind of sessions etc , becuse going through stuffs and having uncleared doubts will leave a learner is almost the same state. Will share the same message with you over lnkedin as well , not sure how frequently you look into UA-cam comments.
just too good, you need too much of knowledge, when you can explain the complex things such easily
thank you so much from the bottom of my heart. god make you happy.
xcellent sir....!
What is the default number of partitions if we are not defining it...
hello... I had a question. The intermediate RDDs which are generated, the partitioned data under them would also be distributed to the executors? or would the redistribution only happen on an action? Please help! :)
Everything is lazy so nothing happens until an action is executed.
Thanks so much, Sir
Hi
Thanks a lot for these videos. They are quite helpful. In this video you mentioned that RDD is immutable, but you have overridden same RDD right, by changing number of partitions. Also we can load different text file into the same named variable(RDD). Could you explain how it is immutable in this case.
Thanks in advance for your help.
I am confused in one step.
When we say, RDD distributes data into nodes.
We create 5 partitions from RDD. It means RDD has loaded all the data and do partitioning, is it?
Will it load data from different nodes to the driver node and keep it in memory and distribute across?
If yes, it is not following data locality paradigm and data movement is very costly. (I am sure spark follows data locality)
What am I missing?
One of the best video series for learning .. Do you also provide classroom trainings as well
Sir i have executed the textFile() command with 5 partitions in cluster mode (5 W) but every time I could see that job is being executed only 1 of the workers.
I mean in every run only 1 worker was executing all the partitions.
is there any extra configuration required here.
i am using spark-shell mode
Please ignore this i got my answer .
thanks anyways
Best tutorial thank you!
Hello Sir, loved this tutorial..thanks a lot.
I have one doubt, consider following scenario:
Input data size(raeding from hdfs): 20 GB
No of executors: 2
executor memory : 8 GB
RDD partition factor: 2
and we run a spark job in client mode.
So in this case:
1. how total 20GB data will get processed through sparkjob?
2. How many stages and task will get created?
3. how total 20gb data will be partitioned?
1. Do you need 20GB memory to process 20 GB data? No. More memory can improve performance but you can still process it with less memory.
2. Stages depend on your logic and the number of task on executors.
3. You asked for two partition so it will shuffle and make it two in that stage. Next stage depends on other factors.
Sincerely looking for spark streaming with Kafka tutorial sir...when r u pubishing sir..you are the best..
Thanks for the wonderful explanation in this video. Can you please tell if this is a general scenario for each job in spark, that map and reducebykey operations usually run in 2 different stages always or if there are cases that they can run in a single stage as well. Any examples or leads would be appreciated!
Think about it. Map and Reduce? You are already talking about two stages.
Why proc and opt came in 1 partition ??
Sir i have a doubt like how no of cores of executors and processing of partitions depend on ?
With spark 2.3.2, number of elements within each partition is not being displayed by the code that your have shown.
Thankyou
Very good one .any airflow demo?
That's still incubating...I do not use open source until they graduate to become production ready.
@@ScholarNest Thank you so much for this info.
is reducybykey a spark/scala specefic function?
Also if possible please explain the code using Java APIs too. I am doing development using Java API, but some methods are not supporting even after it's mentioned in API document and throwing run-time error. Is that when we are doing development using Java API or Python API, will it get converted to Scala language internally?
Hello,Great tutorial.
Can you please elaborate more on reduceByKey((x,y) => x+y) that you have used to count number of pairs with same key.
I am a bit confused how x+y will give us the total count
Here reduceByKey is aggregating the result of array (for similar key) to one value.
Suppose you have (key, value) list :- List((hello, 1), (world, 1), (hello, 1), (hello, 2)).
reduceByKey will perform operation on similar key and x, y denotes the value only from key, value pair (you cannot perform operation on key here)
For key: hello
rdd.reduceByKey(x,y => x+y) -equivalent to (1,1 => 1+1) => List((hello, 2), (world, 1), (hello, 2))
There are still pair exists belongs to hello key here. so again operation will be perform
rdd.reduceByKey(x,y => x+y) -equivalent to (2,2 => 2+2) => List((hello, 4), (world, 1))
Now list has only one hello key pair, so no further reduction can be possible here.
A supplementary video with Amazon EMR please.
thanks, I have a question, does the shuffle and sort executed in multiple nodes ?
Yes, every node that owns a partition must participate in shuffle & sort.
I am using spark 2.2.0 but the code shown by you in the video doesn't print any output in shell.
val myrdd = sc.textFile("UserData.txt",4)
myrdd.foreachPartition(x => println("No. of elements in partition: " + x.count(y=>true)))
Please share the workable code.
hello , so first of all i want to thank you for this superb tutorial. please i have one question following your example of imagin we have 10 partitions and 2 executor and we lets suppose in this example we dont have transformation that gona cause shuffle how many task parrallel there is it 5 ?
thank in advance for your answear
Why do you think it's going to be 5?
Because 10 partitions /2 executors?
Number of executor have nothing to do with how many tasks are created. Once tasks are created, they will run on only two executors because you have only two executors.
@@ScholarNest first of all thank for your quick reply. so i undersntand in my example number of task created per stage depend only by number of partition . number of execeutor have nothing to with it im a correct plaese ? so i my example i will still 10 task because i have 10 partition ?
Super happy
Thanks Sir, Can you please give me link for part-3. I couldn't find
Check the playlist
Thanks,will Check. you have explained all videos in very simple language. :)
Sir I have one doubt is no of partitions is equal to no of executors?
Not necessarily.
waiting for more videos :)
+Nidhi Dewan, coming soon.... I am slightly busy to code a website for learning journal. Just another week away from the release.
I thank you from the deep of my heart for your hard work....
Sir I am doing development using spring Kafka, could you please help me with consumer question? how do i poll in regular intervals e.g every 5 mins and how do I specify number of records to be read in every poll?
I saw batch listener can be used to specify number of records to read but did not find polling interval option.
Thanks.
+Jainesh Modi what is spring Kafka?
Learning Journal sir I meant to say Kafka with spring boot
+Jainesh Modi have you seen my Kafka videos? I have discussed consumer APIs in detail. I am not sure what do you mean my batch listener?
Yes sir i have gone through ur videos.
my requirement is as a consumer i want to put delay with every poll and also want to control number of records being read in every poll.
+Jainesh Modi to be honest, I haven't used spring Kafka, just saw the documentation. Looks interesting. I will plan for some time to evaluate it and send you details if I find an answer to your problem.
I have one doubt is no of partitons is equal to no of executive?
NO, Executor runs tasks in it.
Awesome explanation !!!!!