Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code. www.learningjournal.guru/courses/
Hello 'Sir' I am taking this course from your website and there was no section available for comment. So Just came here to 'Thank you'. The way that you taught all the lecture is very simple and crisp and easily understandable . Your Voice is very clear. Thank you so much
Cristel clear explanation,to the point covering of each points. A complete learning for Zero to Hero.Well done Prashant.We are looking some Big Video series which can make us near perfect in Spark and Hadoop.I am surprised,why the views are very less...may be people are sleepy.Great job keep us enlightening.
Following Learning Journal from Kafka, explanations are very simple, highly helpful and easy to understand. Looking forward to learn basics of all components of Spark. Thanks!
Question Sir: What if we use SQL statements instead of using the inbuilt functions? Spark session has a SQL function that helps us use SQL statements to transform dataframes after we create a tempview out of it. Are there any performance issues if we go by that method instead of memorizing spark functions?
Could you please explain about dataset as this tutorial talks about only dataframe? If u can explain difference between dataframe and dataset that would be great
Hi Learning Journal , --> Please help me with the below queries 1. We didn't used Spark Sql Context I understand that we does not require the define in CLI level but by default its name is Sc and Sqlsc for -Sqlcontext 2. Slight difference in terms of syntax for Sql -- why is it the new enhancement done in Spark version? for example we do val rdd2 = SQLSC.sql("Select * from table") 3. transformation syntax - I observe we have used something similar to Case but what is use of $? I think is related to value within that particular column? Thanks, Anuj
2 Questions : 1. What is the difference between typed & untyped transformations ? 2. At some points , you used $ with the column name and at others you dint, any particular reason ?
Sir, your videos are very helpful in understanding the concepts, I would really like to listen and understand Spark Streaming concepts, please let me know if you plan to add spark streaming videos.
Please upload more examples on dataFrames and dataSets with different file format too..it will be a great help..Thank you so much for your session loved your sessions...
Hello Sir. I have a question regarding the parseGender operation performed as a UDF. Rather than applying it after 2 select transformations, would it be better to do it at the first step itself ?
Awesome videos sir. This explains the spark in crystal clear way. I have one question, instead of using spark UDF, can we just write a function which standardizes the gender parts i.e i use java for writing spark applications, where i usually write functions on driver for handling such cases. How spark UDF is different from a normal function/method which we use in map operations.
Hi Prashanth, First your videos are awesome and is of great value addition for us. Can you please tell us what kind of applications needs a huge horizontally scaled spark cluster with hundreds of cores. Thanks & Regards Sathiyanarayana
Nice videos as usual , thank you so much sir , is there any video regarding how to design a project or else around it , I know it requires a lot of effort to make such video
hello prasanth, first i will appreciate for all of this teaching, and can you do one video about difference of dataframe and dataset in a deep please..
We will be using SBT to build Spark Jars. I will cover required dependencies in my videos. I have already explained Scala IDE and SBT in my Scala tutorials.
hi sir I have doudt . if i have one spark program I have 7 select statements within that I have 3 actions and another program I have 10 select statements but only one action so which one run faster?
Spark concepts explained in a very easy way. Marvel piece of work sir.. Many thanks! you are guru One question, we saw that spark will optimize the code. Does this mean that developers should just write the code & leave the code optimization on spark completely ? if No then can you throw some light in the area where developers can contribute for optimization?
Spark will take care of the obvious things as explained in the video. We as a developer should know about that, so we can ignore those concerns. But that doesn't mean that we should not worry about performance. I will cover some commonly used optimization and tuning in a separate video.
Sir, Thanks for your videos. Is it okay if I publish a blog by making use of the understanding gained through your videos? I will cite them as amongst the sources.
Hello Sir, first of all youe videos are awsome and very clear. Thank you for making them. Now I have a question. I have clearly understood what is pipeline omtimization and predicte pushdown optimization. But other terms that we so frequently hear regarding spark is "catalyst optimizer" and "tungsten optimizer". Can you please explain them too? Thanks in advance. :)
Spark dataframe use catalyst optimizer for optimizing query execution logical plan and tungsten optimizer use for optimize memory and java heap during table to jvm object conversion...
Sir, I am getting error in when val df2 = health_survey_df.select("Gender", (when("treatment" === "Yes", 1).otherwise(0)).alias("All-Yes"), I am using spark 2.3, Can you please help me what to use in place of when in spark 2.3
hi , That is not working for me as well , instead you guys can use :::: val df2 = df.groupBy($"Gender").agg(sum(when($"treatment" === "Yes",1).otherwise(0)), sum(when($"treatment" === "No",1).otherwise(0))) This is working :)
Hi prashant, i want to learn spark with python. could you please send me spark UDF examples in python. and also send me material on pyspark if you have any...
+veeru kbr source code will be available on my new website. My new website is ready, just final round of testing is going. I will publish it on this weekend and share the link.
Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code.
www.learningjournal.guru/courses/
best youtube teacher i found till date !!!
Best Apache Spark course ever.. You did great job explaining spark concepts in simple & clear manner
Hello 'Sir'
I am taking this course from your website and there was no section available for comment. So Just came here to 'Thank you'. The way that you taught all the lecture is very simple and crisp and easily understandable . Your Voice is very clear. Thank you so much
Thanks a lot for your feedback and support. It really matters.
@@ScholarNest :)
Awesome Way of explaining. Any other tut Could not be better than this. High Respect Sir . Carry on the good work. May GOD Bless you !
Cristel clear explanation,to the point covering of each points. A complete learning for Zero to Hero.Well done Prashant.We are looking some Big Video series which can make us near perfect in Spark and Hadoop.I am surprised,why the views are very less...may be people are sleepy.Great job keep us enlightening.
Following Learning Journal from Kafka, explanations are very simple, highly helpful and easy to understand. Looking forward to learn basics of all components of Spark. Thanks!
Your tutorials are very much informative for new learners. Great work.
Like your details and explanations in the demo, good job Mr. LJ.
Nice examples, beautifully explained
Question Sir: What if we use SQL statements instead of using the inbuilt functions? Spark session has a SQL function that helps us use SQL statements to transform dataframes after we create a tempview out of it. Are there any performance issues if we go by that method instead of memorizing spark functions?
Short, crisp and informative..!!
Nice video, thanks for helping in understanding concepts
Could you please explain about dataset as this tutorial talks about only dataframe?
If u can explain difference between dataframe and dataset that would be great
Video is very informative. I want to practice spark. Could you please provide more details about where you are executed code?
Hi Learning Journal
, --> Please help me with the below queries
1. We didn't used Spark Sql Context I understand that we does not require the define in CLI level but by default its name is Sc and Sqlsc for -Sqlcontext
2. Slight difference in terms of syntax for Sql -- why is it the new enhancement done in Spark version?
for example we do val rdd2 = SQLSC.sql("Select * from table")
3. transformation syntax - I observe we have used something similar to Case but what is use of $? I think is related to value within that particular column?
Thanks,
Anuj
2 Questions :
1. What is the difference between typed & untyped transformations ?
2. At some points , you used $ with the column name and at others you dint, any particular reason ?
+Kishor Kukreja will cover those items in coming videos.
where are the equivalent python code located?
Awesome video, kindly also explain the off heap memory management benefit of datataframe over RDD.
Could you give you the difference between CountByKey and reduceByKey
Sir, your videos are very helpful in understanding the concepts, I would really like to listen and understand Spark Streaming concepts, please let me know if you plan to add spark streaming videos.
Hello Sir, I want to understand why "Gender" column is unresolvedalias in the "Parsed Logical Plan" ?
Please upload more examples on dataFrames and dataSets with different file format too..it will be a great help..Thank you so much for your session loved your sessions...
Sure, more videos are on the way.
Hello Sir. I have a question regarding the parseGender operation performed as a UDF. Rather than applying it after 2 select transformations, would it be better to do it at the first step itself ?
Watch other videos. I explained predicate pushdown. Spark can take care of those things automatically.
Awesome videos sir. This explains the spark in crystal clear way. I have one question, instead of using spark UDF, can we just write a function which standardizes the gender parts i.e i use java for writing spark applications, where i usually write functions on driver for handling such cases. How spark UDF is different from a normal function/method which we use in map operations.
Hi Prashanth, First your videos are awesome and is of great value addition for us. Can you please tell us what kind of applications needs a huge horizontally scaled spark cluster with hundreds of cores.
Thanks & Regards
Sathiyanarayana
Just Awesome.
Nice videos as usual , thank you so much sir , is there any video regarding how to design a project or else around it , I know it requires a lot of effort to make such video
Very good tutorial
hello prasanth, first i will appreciate for all of this teaching, and can you do one video about difference of dataframe and dataset in a deep please..
I intend to do that in near future. Can't commit a timeline though.
Amazing tutorial....thank you so much ...sir why dont you put these samples on git ?
Code is hosted at www.learningjournal.guru
could you tell me how to create jar for spark-submit utility , how we can write code in eclipse
We will be using SBT to build Spark Jars. I will cover required dependencies in my videos. I have already explained Scala IDE and SBT in my Scala tutorials.
hi sir
I have doudt . if i have one spark program I have 7 select statements within that I have 3 actions and another program I have 10 select statements but only one action
so which one run faster?
I couldn't get your question.
Hi sir,
i can't understand can you explain this sir
Why does the Spark application code need to be an object rather than a class?
You need to learn Scala to understand the reason. We can't define a main method in a Scala class. So you need an object.
Spark concepts explained in a very easy way. Marvel piece of work sir.. Many thanks! you are guru
One question, we saw that spark will optimize the code. Does this mean that developers should just write the code & leave the code optimization on spark completely ? if No then can you throw some light in the area where developers can contribute for optimization?
Spark will take care of the obvious things as explained in the video. We as a developer should know about that, so we can ignore those concerns. But that doesn't mean that we should not worry about performance. I will cover some commonly used optimization and tuning in a separate video.
thanks sir for prompt response.. eagerly waiting....
Sir, Thanks for your videos. Is it okay if I publish a blog by making use of the understanding gained through your videos? I will cite them as amongst the sources.
Yes, you can as long as it is not an exact copy and you are adding some value.
Hello Sir, first of all youe videos are awsome and very clear. Thank you for making them.
Now I have a question. I have clearly understood what is pipeline omtimization and predicte pushdown optimization. But other terms that we so frequently hear regarding spark is "catalyst optimizer" and "tungsten optimizer". Can you please explain them too? Thanks in advance. :)
Ok. Will try to include then.
Spark dataframe use catalyst optimizer for optimizing query execution logical plan and tungsten optimizer use for optimize memory and java heap during table to jvm object conversion...
Sir, I am getting error in when
val df2 = health_survey_df.select("Gender",
(when("treatment" === "Yes", 1).otherwise(0)).alias("All-Yes"), I am using spark 2.3, Can you please help me what to use in place of when in spark 2.3
Same error for me as well, Can anyone correct pls
hi , That is not working for me as well , instead you guys can use ::::
val df2 = df.groupBy($"Gender").agg(sum(when($"treatment" === "Yes",1).otherwise(0)), sum(when($"treatment" === "No",1).otherwise(0)))
This is working :)
add an import to your line of code "import org.apache.spark.sql.functions.when"
@@deepuinutube add an import to your line of code "import org.apache.spark.sql.functions.when"
thank u.
Great
Hi prashant, i want to learn spark with python. could you please send me spark UDF examples in python. and also send me material on pyspark if you have any...
We do not recommend writing UDF in Python. My next video will explain how to create UDF in Scala and use it in Python.
Request you to share a code with dataset
+veeru kbr source code will be available on my new website. My new website is ready, just final round of testing is going. I will publish it on this weekend and share the link.
Thanks for the response