All about Spark DAGs

[@Themotivationstationpower]

Really appreciate your hard work. Thank you for the great explanation.

[@BigDataThoughts]

thanks

[@chinmaytornekar4683]

Thanks for such a clear Explanation!!Amazing Walk through the DAG visualization!!

[@BigDataThoughts]

Thanks

[@karthikbhat2945]

Clear and Precise Explanation. These videos needs more outreach

[@BigDataThoughts]

Thanks Karthik

[@anuraggupta-xw2jm]

Very good explanation..appreciate your efforts..keep it up👍🏻

[@souravkrburnwal3519]

Just Amazing!!! Detailed information through PPT walkthrough and great way to explain everything at once.

[@BigDataThoughts]

Thanks saurav

[@TotuBabyBird]

Thanks for such a detailed and crystal clear explanation. ❤

[@BigDataThoughts]

Thanks Sumiran

[@NehaParmar-t6b]

Awesome , Superb explanation. Good efforts.

[@BigDataThoughts]

Thanks

[@pradyumnaanand6892]

Clear and simple explanation. Thanks a ton.

[@BigDataThoughts]

Thanks

[@karthikeyanr1171]

your videos on spark are hidden gems

[@BigDataThoughts]

Thanks

[@mattasid85]

Thanks Shreya, this was a very helpful and informative video like any other video of yours. Looking forward to more such content on your channel!

[@BigDataThoughts]

Thanks Sid

[@shivarajmiraje6672]

Great explanation ❤

[@satviknaren9681]

Really like your content !! really helpful ! thank you so much shreya

[@BigDataThoughts]

Thanks

[@syedmudasser631]

perfect explanation
keep it up...👍

[@BigDataThoughts]

Thanks

[@saiprakash6916]

very Informative and understandable. Thanks shreya

[@BigDataThoughts]

Thanks sai prakash

[@sanketdhamane5941]

Really Thanks to Good And Indepth Explantion

[@BigDataThoughts]

Thanks

[@shivam-h2b5d]

Nicely explained and thanks. helping a lot

[@sonip472]

awesome explanation, thanks a lot.

[@BigDataThoughts]

Thanks soni

[@SanganaChaturvedi]

Great explanation. thanks

[@BigDataThoughts]

@@SanganaChaturvedi thanks

[@ankitachauhan6084]

Great work

[@srikanth4788]

Great Explanation

[@BigDataThoughts]

Tha ks srikanth

[@askdoubts6359]

Your all videos are grate and easy to understand.. please make more videos on spark with real time code examples

[@BigDataThoughts]

Thanks sure

[@Laikacc]

Very well explained

[@BigDataThoughts]

Thanks

[@reevarastogi6224]

very nice explanation!!

[@BigDataThoughts]

Thanks Reeva

[@souravdas9683ytm]

Logical plan is nothing but Lineage, which tells us how to create an RDD by applying what transformation. Whenever an action is called, this lineage will be converted into a DAG, basically a Physical plan.

[@syedhussain6358]

Excellent explanation

[@BigDataThoughts]

Thanks syed

[@maheshalam7893]

It is great video and helps alot.. Shreya .
Interesting to know below .
1 . Why complete stage is skipped
2 . Shuffle should create different stage . But In snap of DAG , it is in same stage .

[@BigDataThoughts]

Thanks Mahesh

[@alokranjan7323]

He had questions as well

[@prasadj6370]

Thanks for sharing the valuable information in most simplest way. It saved a lots of time in reading spark tutorial. sincere thanks to you.
I have one question - where can we see or generate this DAG visualization in spark ? We want to see the plan in case to analyse our spark programme ? Please let us know . 🙏 Thank you

[@BigDataThoughts]

Thanks Prasad yes DAG visualization can be seen on spark UI.

[@BigDataThoughts]

A detailed analysis stage wise can be done on the UI

[@ryas8360]

shouldn't linkage of RDD's represented by Lineage?

[@anirbansarkar6306]

Hi Ma'am, in that 16 stages snapshot, what is the difference between gray boxes and blue boxes?

[@artemvolkov5682]

I confused a bit... I just watched Spark Summit videos. It states that on Physical Planning stage Spark applies 'Strategy' (a function that converts a logical plan tree to a physical plan tree)... Where does DAG come into play?

[@BigDataThoughts]

Dag is created once a spark job is submitted. Think of the vertices as RDDs and the edges as the operations. The dag schedulers job is to convert the dag into stages and tasks that will perform actual execution .

[@mateen161]

Nice explanation. Thank you!

[@BigDataThoughts]

Thanks