Thank You Very Much for Explaining the concept in simple way. I had no idea about Meta Data driven prior watching your video , Now I have a fair idea about it, Thanks again.
One quick Q, let's say my schema or Metadata has a column "ID" as Long and using that I already created my table in Lakehouse, after a month application team changed the ID from Long to String, how am I going to handle it without reprocessing the entire table?
yes for data type change you would need to reprocess(its a destructive change). Using a metadata framework doesn't mean that there would be no reprocessing at any point of time but it ensures we reduce code changes for many scenarios. Typically constructive changes like adding a new column.
Thank You Very Much for Explaining the concept in simple way. I had no idea about Meta Data driven prior watching your video , Now I have a fair idea about it, Thanks again.
thanks
Can you do one video or 2 on real-time issues in Apache spark
One quick Q, let's say my schema or Metadata has a column "ID" as Long and using that I already created my table in Lakehouse, after a month application team changed the ID from Long to String, how am I going to handle it without reprocessing the entire table?
yes for data type change you would need to reprocess(its a destructive change). Using a metadata framework doesn't mean that there would be no reprocessing at any point of time but it ensures we reduce code changes for many scenarios. Typically constructive changes like adding a new column.