Schema Merge | Schema Evolution | Parquet| Spark with Scala | Scenario based questions
Вставка
- Опубліковано 23 чер 2021
- Hi Friends,
In today's video, i have discussed about Schema, Schema evoluation and mergeSchema option in Spark with a sample Scala code.
Please subscribe to my channel and provide your feedback in the comments section.
very clear explanation Mam, Thank You
Clear explanation mam.. thanks for this entire playlist.
btw I believe its schema evolution and evaluation ?
Thank you very much for watching the video. Yes, it's evolution 👍
Clear Explanation, Can you please share the Dataset and it will be good to start practice.
Thank you Naresh . Plz take *.parquet files from the GITHUB - github.com/sravanapisupati/SampleDataSet
@@sravanalakshmipisupati6533 thank you 😊
Hi Sravana,
Hope you are doing well!
I have been blocked for one of the scenarios in the project, I hope you provide guidance in regards to it.
Background of Project:
There are some 10 base tables, from each table there are bringing some 5 columns and creating other 5 derived tables as dataframe (Scala/spark) by joining all the columns from 10 base tables. After this they are making a parquet file and publishing the data in snowflake via DAG run and Databricks.
Business Requirement:
From one particular base table, I need to bring one particular column and need to add this column to all the 5 derived tables.
Current Situation:
I have brought that column from the base table and added that column in all the 5 derived table dataframe. After the DAG run, I am not able to see that column in snowflake tables.
Issue:
After running the DAG run it's showing an error that Exception found when writing the partition.
Observations:
Scala
Please provide some guidance, looking forward to your reply.
Regards,
Vikas
Observations:
In the Scala/spark code there is no mention of .options("schemamerge":true).
Hi Vikas, please check this video for schema merge - ua-cam.com/video/w2EJATgekUo/v-deo.html
@@sravanalakshmipisupati6533 Hi Lakshmi, even after adding .option("mergeSchema", "true"). Still the issue persist. Throwing the same error-- Exception found when writing the partition to the table.
@@vikastv9593 Please try to write the data to a new table. if the issue still persists, there might be issue with schema.