Applying Header Dynamically to a Dataframe | Spark With Scala | With example

[@reenageorge5817]

Hi Saravana , Good video , Can you tell me how do we handle if the headers are stored in a config table or in another data frame as rows

[@sravanalakshmipisupati6533]

Get the df columns and assign it to a val and use that val in the current dataframe

[@narayanareddy5837]

Great. explanations is good. could you please help on how to handle dynamically changing headers in csv . everyday we are getting different columns in particular file.

[@sravanalakshmipisupati6533]

Thank you very much for watching the video and providing the comments. You can change the columns in the file as mentioned in the video. Only a file will be changed and not the code. Also, you can pass this file as input to the Spark job with the latest columns.

[@narayanareddy5837]

@@sravanalakshmipisupati6533 thankyou

[@narayanareddy5837]

could please provide, how to store data in hdfs for real time kafka messaging data using spark streaming in scala language in intelljidea . its grateful for me if you can provide this.

[@sravanalakshmipisupati6533]

@@narayanareddy5837 Sure. I will work on it. Thank you.

[@narayanareddy5837]

@@sravanalakshmipisupati6533 Thankyou

[@sravankumar1767]

Nics explanation sravana, can u please explain how can we write in pyspark

[@Texas2Nellai]

How to apply schema dynamically ?

[@sravanalakshmipisupati6533]

Hi Joseph, thank you for watching the video. The option spark.read.option("inferschema",true) can be used for applying the schema dynamically from the file that we are reading as a DF.