Apache Spark - 03 - Architecture - Part 1
Вставка
- Опубліковано 25 сер 2024
- Spark Programming and Azure Databricks ILT Master Class by Prashant Kumar Pandey - Fill out the google form for Course inquiry.
forms.gle/Nxk8...
-------------------------------------------------------------------
Data Engineering using is one of the highest-paid jobs of today.
It is going to remain in the top IT skills forever.
Are you in database development, data warehousing, ETL tools, data analysis, SQL, PL/QL development?
I have a well-crafted success path for you.
I will help you get prepared for the data engineer and solution architect role depending on your profile and experience.
We created a course that takes you deep into core data engineering technology and masters it.
If you are a working professional:
1. Aspiring to become a data engineer.
2. Change your career to data engineering.
3. Grow your data engineering career.
4. Get Databricks Spark Certification.
5. Crack the Spark Data Engineering interviews.
ScholarNest is offering a one-stop integrated Learning Path.
The course is open for registration.
The course delivers an example-driven approach and project-based learning.
You will be practicing the skills using MCQ, Coding Exercises, and Capstone Projects.
The course comes with the following integrated services.
1. Technical support and Doubt Clarification
2. Live Project Discussion
3. Resume Building
4. Interview Preparation
5. Mock Interviews
Course Duration: 6 Months
Course Prerequisite: Programming and SQL Knowledge
Target Audience: Working Professionals
Batch start: Registration Started
Fill out the below form for more details and course inquiries.
forms.gle/Nxk8...
--------------------------------------------------------------------------
Learn more at www.scholarnes...
Best place to learn Data engineering, Bigdata, Apache Spark, Databricks, Apache Kafka, Confluent Cloud, AWS Cloud Computing, Azure Cloud, Google Cloud - Self-paced, Instructor-led, Certification courses, and practice tests.
========================================================
SPARK COURSES
-----------------------------
www.scholarnes...
www.scholarnes...
www.scholarnes...
www.scholarnes...
www.scholarnes...
KAFKA COURSES
--------------------------------
www.scholarnes...
www.scholarnes...
www.scholarnes...
AWS CLOUD
------------------------
www.scholarnes...
www.scholarnes...
PYTHON
------------------
www.scholarnes...
========================================
We are also available on the Udemy Platform
Check out the below link for our Courses on Udemy
www.learningjo...
=======================================
You can also find us on Oreilly Learning
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
www.oreilly.co...
=========================================
Follow us on Social Media
/ scholarnest
/ scholarnesttechnologies
/ scholarnest
/ scholarnest
github.com/Sch...
github.com/lea...
========================================
Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code.
www.learningjournal.guru/courses/
atleast show courtesy to share link of 2nd part in description.
I love the concise language and diagrams you use to explain each concept. Makes it easy to understand for new learners like me. Many manuals and books are overcomplicating things by jargons and not goes straight to the point like you do.
Great Video .. I got this much information by reading thousand articles and getting confused. Worse , still in doubt. your one video in 2024 made it all clear.. Thanks
Best video to understand Spark Architecture
sir multiple people saying u became your self like a god...........then i did not belive but...after when i watched the video then i fixed s u bacame ur self like a god.....yes its true.....and one more thing i have seen multiple videos regarding spark architecture but i didnt get this much of clarity.....so tnx a lot.....its amazing video
Yu ll become trend setter the way ur showing slides the explanation evry thng peeks
You are pure gold. Content is amazing, structured and well deductively explained, thank you
Everything is very nicely explained. Thank you.
You are one of the best teachers on youtube. Your style of teaching with simple examples is really helpful. thank you so much for making these videos. you are awesome sir.
for all other videos i have to watch 5 to 6 times then only i can able to grasp few things but your videos are quite impressive in the single shot itself i can able to grasp more things ... thanks a lot.... i'm willing to donate you tooo
This video is pure gold.
I could never imagine I can learn Apache Spark without attending the physical sessions. You not only saved my time but give me confidence in all senses. Thanks for such a honest efforts !
the quality of your content is amazing.
Sir, i seriously suggest you to post your videos in UDEMY, to reach max and all. This is excellent and better than many paid videos. Excellent ,pls do share more such videos. we are thankful to you.
+ganesh sundarachary thanks for the suggestion. I will think about it. However, I know my viewers are good at sharing, and sooner or later, all those udemy viewers that are looking for Spark will reach to this place as well.
I am unable to stop myself to subscribe and like this video. its clean and fantastic explanation... making things clear.
Thanks 🙌🙌🙌
Best tutorial so far on youtube
I have been following your videos for quiet a long time.I found each and every videos of yours to be so informative and easy.Thanks Prashant for all of these videos
One of the best videos of Spark i have come across untill now.. Thank you for explaining it so beautifully.. :)
Wow, best lecture by far. Well done
This is mind blowing .. Fan of you
you are a awesome instructor. good job
Wow! This was fun to watch. Great demo. Thanks.
Pandey Sir... Teacher ho to aap sa.. warna na ho :). What a clear way of teaching.
One word - Best
AMAZING VIDEO AND EXPLANATIONS!
Wonderful. We are lucky to have your materials. Simple and powerful.
Dude I am falling in love with you..amazing teaching content and style.
sir.. great work.. very nice explanation. i have seen lot of links, gone thru many videos, but didn't get clarity. your way of explaining is unique.. thank you soo much sir..
one of the best spark arch explanation ever seen on youtube,,, wondeful!!! Thank u so much sir
No words to express..simply awesome video... :) thank a lot Prashant sir
wow Excellent !
Best in the internet...
Much appreciate the standard of the content.It's simple,precise and context.Thank you
Thank you sooo much for the great explanation!!
Superb explanation.Expecting more spark video from you.Thanks for sharing this video
Excellent explanation.
One more gem of a video from Mr. Pandey. Awesome work !
Great learning!!!!
Sir I'm very thankful for your video. If I stuck in any point I come here to clear my doubts. Just a small request what ever you explain on CLI please increase the font size to make it visible. Thanks.
Loved the content.
The pace at which you are explaining and the way is excellent. Please make videos on Hadoop on AWS
Great explanation.
This video is so amazing! It's very very clear. Thank you so much
Sir its avery informative video series..pls keep uploading..big fan after watching first few videos..thanks
WOW. This video was so helpful!! Thank you
This is very good explanation.. this is first video on UA-cam where I am commenting..
Hi Sir...I really like and admire your efforts that you put for us.
Thanks once again
Pariksheet
You are the most awesome teacher. hats off
Thank you very much.. This is one of the best tutorial I have come across.Appreciate your efforts.
Wow, vera level. Thanks for the course. im one of the follower in Udemy & Linkedin. Once again thanks for this low hanging fruitful tutorial :) :) :)
I got it what i was looking for :)
Thank you, for the very practical and compact introduction to Spark.
Prashant Sir, I don't know how to thank you for such a wonderful content. I had to revise the spark and hadoop concepts, and what a wonderful way to do this. Thank you Prashant.
Fantastic teaching style...please continue to do so
These videos are simply amazing. Thanks a lot. Hope to see more tutorials from you.
good explanation
Thanks sir for sharing your knowledge, expecting more videos like this
Sir ur explanation was very good. waiting for the next video
Million Thanks Sir, Course of the content amazing sir.
Great video.. Came to know many unknown things from this video... :)
Thank you billion times…..
OMG! This video is so amazing! Thank you so much for the amazing session.
Thank you sir, the quality of your content is amazing.
Very good demo, learned more today, thank you.
Great Video sir!!!.. Really clears my concepts Thank you
thank you so so so much Prashant sir
Very useful Sir.. In depth knowledge you are providing with proper examples. Thanks a lot..!!! Looking forward to watch your more videos. And Sir, My jupyter notebook is not working..I have specified my problem in that video. Look into that too.
suitable
Really nice... 👏
Very well explained the concepts in the video. Great work Sir!!!
You rock !!
Supperb explaination sir.... Thanks 😊
awesome video very useful thanks
Sir can you please upload videoes regarding the project in spark or Hadoop. If not please suggest me some project related to spark and Hadoop. Bcoz your way of explanation and clearness in voice is great
Excellent !
thank you, wonderful connect.
Awesome explanation sir
very nice explanation
Nice. Thank you
3:15 How spark execute a program on a cluster
5:04 Who executes where
8:09 Who controls the clusters
14:00 Demo
Sir what is the difference between spark session and spark context. What purpose does spark session solve? Does it helps in overcoming any shortcomings of spark context??
19:39 - is that correct? From what I can see, the driver is not on your local computer. I thought the driver would've started in the spark-shell process which you ran from the master node that you were SSH'd into. So is the driver actually running on the master in that case?
So much appreciated bow
+1
3. All the json or other files those you are processing , are residing in the same machine (node), in that case if we have 3(executor ) + 1 (master) architecture- does spark copy the file to each executor for processing , or master node shares the file partially with each node ?
If we need to connect to S3 to process a huge file ? How will be the executor processing the file?
Hi sir, your videos are great. I have a request. Will it be okay if you make a video on real time Spark applications getting slow or getting failed. The causes of such failures and slowness and what could be done to prevent it. Thanks in advance :)
When we start spark-shell through edge node of cluster,it is which mode(client/cluster),because driver in running on one node of cluster right.Could you please explain.
thank you so much!
sir, the way you explain very use and simple.could you please any for spark reference.Thanks in advance.
god bless you all your life sir
So do you mean driver process will be created in Node manager or Resource Manager. Also Application master is also Node manager ?
Good Explanation sir
Very informative sir thanks 🙏.
I have few questions-
1. Let’s suppose we have 3 nodes + 1 master - while staring the executors does that mean - each node will have 1 executor .
2. Do we need to keep the spark binaries in executor as well . Or that will be taken by spark master node
Excellent demo video... Thank you..I feel that i couldn't see spark shell content and spark summit command...It would be better if video quality is more on clarity
+sudhakar k, video is full HD 1920x1080. Check your internet speed. However, all the commands and the code is also available at www.learningjournal.guru for copy/paste.
Loved it. Thank you. Subscribed
Excellent explanation.
Do you have videos on spark streaming ?
which one is faster client or cluster
At -19:43 when you did a ssh to the master node and started spark-shell with 3 executors then why the driver will start from your machine ? It shd start from one of the nodes
I have a question. Why your available executors are 5 as a default? As you have 1 master node and 3 workers nodes with 2 cores for each node, I thought the available executors are 6 (1 executor per core). Or is it just a default behavior of YARN?
Also I am thinking that in YARN, we can increase the number of executors as much as we want, e.g. 100 executors on 3 worker nodes (though it is probably inefficient). Also based on my research online, I see that 5 executors per each core is ideal in YARN.
Number of available executers = (total cores/num-cores-per-executor) - (1 for application manager)
So in my case it is (6/1)-1=5
If you ask for more than what is available, it goes into a queue.
When I pasted my spark Web UI in my browser nothing popped up. What am I missing?
Do you provide training?