Please make a short video on the relationship between stages, node, executor, dataframe/dataset/RDD, and core, partition, and task. Want to know what consists what ? And what contains what.
How cluster manager can create any of the nearest worker as application master because the configuration of the master can be different. So will it not go and create master to the machine that is configured for master role. With memory allocated to master depending on the type of task.
Hi Naresh, Thank you so much, This help me a lot. ❤ Naresh, I've execute my spark application in cluster mode (yarn) in emr cluster, My spark application is failing with an exception saying application master container failed 2 times, exists with 137 code. This exceptional is occuring for only one dataset which I'm processing with spark application. For other datasets, my spark application is working fine. The dataset for which spark application is failing having large input payload, ( one record with 25000+ characters ). I tried increasing the driver memory and executor memory, now this time , I'm getting an exception while deserialization of input payload. Any suggestions how to resolve this issue. It will be helpful, please
![PySpark Live Class Pre Intro Session [Part-1] | Spark SQL | Databricks](http://i.ytimg.com/vi/M6VTq-EKCDM/mqdefault.jpg)
![PySpark Live Class Pre Intro Session [Part-1] | Spark SQL | Databricks](/img/tr.png)
very good job done
Nice explained
Needed this one badly... Thanks Naresh
Hi Naresh
your way of explanation is excellent.
this is first time i understand spark architeecture is very easy way in Cluster Mode
Thank you Pavan.❤
Please make a short video on the relationship between stages, node, executor, dataframe/dataset/RDD, and core, partition, and task. Want to know what consists what ? And what contains what.
How cluster manager can create any of the nearest worker as application master because the configuration of the master can be different. So will it not go and create master to the machine that is configured for master role. With memory allocated to master depending on the type of task.
Hi Naresh,
Thank you so much,
This help me a lot. ❤
Naresh, I've execute my spark application in cluster mode (yarn) in emr cluster,
My spark application is failing with an exception saying application master container failed 2 times, exists with 137 code.
This exceptional is occuring for only one dataset which I'm processing with spark application.
For other datasets, my spark application is working fine.
The dataset for which spark application is failing having large input payload, ( one record with 25000+ characters ).
I tried increasing the driver memory and executor memory, now this time , I'm getting an exception while deserialization of input payload.
Any suggestions how to resolve this issue.
It will be helpful, please
What about the node manager do in this architecture