96. Databricks | Pyspark | Real Time Scenario | Schema Comparison

Azure Databricks Learning: Schema Comparison
============================================
How to compare schemas of different dataframes and make it same through automated way?
Schema related operations are very common in Databricks development. Have explained different real time scenarios of schema related operations in this video
To get through understanding of this concept, please watch this video
#DatabricksSchemaComparison, #DatabricksStructType,#DatabricksMapType,#PysparkStructType,#PysparkMapType, #SparkStructType,#SparkMapType,#StructTypevsMapType,#PysparkStructTypevsMapType,#SchemaDefinition,#PysparkSchemaDefinition, #PysparkTips, #DatabricksRealtime, #SparkRealTime, #DatabricksInterviewQuestion, #DatabricksInterview, #SparkInterviewQuestion, #SparkInterview, #PysparkInterviewQuestion, #PysparkInterview, #BigdataInterviewQuestion, #BigdataInterviewQuestion, #BigDataInterview, #PysparkPerformanceTuning, #PysparkPerformanceOptimization, #PysparkPerformance, #PysparkOptimization, #PysparkTuning, #DatabricksTutorial, #AzureDatabricks, #Databricks, #Pyspark, #Spark, #AzureDatabricks, #AzureADF, #Databricks, #LearnPyspark, #LearnDataBRicks, #DataBricksTutorial, #azuredatabricks, #notebook, #Databricksforbeginners

[@vishalaaa1]

excellent video Raaj
the scenario on dynamically adding missing columns is awesome

[@rajasdataengineering7585]

Glad you liked it!

[@seenme2951]

That's great sir

[@rajasdataengineering7585]

Thank you

[@putuism1]

excellent video Raja , thank you

[@rajasdataengineering7585]

You are welcome! Glad it helps

[@mallikarjunad3549]

Hi Raja, could you please make a video on CICD pipeline for Azure databricks and ADF by using Azure Devops. Many thanks in advance

[@rajasdataengineering7585]

Sure I will

[@muvvalabhaskar3948]

Hi I have created one databricks job what ever output comes from it needs to save in s3 as log file how can I do that?

[@nagulmeerashaik5336]

Very nice👍👏😊.... Lot of content you have

[@rajasdataengineering7585]

Thank you 😊

[@sravankumar1767]

Hi Raja, how can we merge 3 dataframe using union or unionbgname in the user defined function

[@rajasdataengineering7585]

Hi Sravan, yes it possible
Df=df1.union(df2).union(df3)

[@sravankumar1767]

I have one doubt , how can we merge these 2 csv files as a single csv, can we use union by name

[@rajasdataengineering7585]

Yes we can

[@playsomeboards]

Thank you for your videos, It's important to point out that you keep calling things a UDF that are not UDFs. For example, addMissingColumns(df1, df2) is not a UDF

[@rajasdataengineering7585]

Addmissingcolumns is a user defined function. What do you call that?

[@alluchandrasekhar2992]

Useful content 🔥

[@rajasdataengineering7585]

Thank you

[@dataengineerazure2983]

@@rajasdataengineering7585 Thank you. Have you the data source(github)? thank you in advance

[@David-rm1wn]

HI thanks for very good videos...Is it possible to get free azure databricks service?

[@sabesanj5509]

Raja bro please make a video on frequently asked interview questions in hive and spark.

[@rajasdataengineering7585]

Sure bro, will make

[@sabesanj5509]

@@rajasdataengineering7585 Thanks bro