Wonderful channel! I have watched all the videos on Databricks. Keep going! Just one question on this video. You are manually mentioning the versions. How this will be done in dynamically in production scenarios?
If you are asking about the versions that she is mentioning here Select * FROM table_changes('table_name', ) here we have the option to mention the timestamp as well instead of version like this getting all the changes between the given timeframe SELECT * FROM table_changes('tableName', '2021-04-21 05:45:46', '2021-05-21 12:00:00')
@@DevarshShah-zo3qm This did not answer the question. I had the same question. Finding the versions or timestamps insnt the question. The question is about her hardcoding the versions. in a production environment, you wont know how many versions there are so there has to be a way of checkpointing to know how to get just the changes dynamically
gr8 video, i would suggest to implement delete as well to gold table, currently deleted record from silver table is still showing up in gold table
Do you have Git repository of your Notebooks
How come you know that I need to add 2 and 5 as commit version?
How to make it dynamic like real world scenarios?
Can you share how to do this dynamically? Also what about rows which are deleted from source? How does this change flow to target?
Wonderful channel! I have watched all the videos on Databricks. Keep going!
Just one question on this video. You are manually mentioning the versions. How this will be done in dynamically in production scenarios?
If you are asking about the versions that she is mentioning here
Select * FROM table_changes('table_name', )
here we have the option to mention the timestamp as well instead of version like this
getting all the changes between the given timeframe
SELECT * FROM table_changes('tableName', '2021-04-21 05:45:46', '2021-05-21 12:00:00')
@@DevarshShah-zo3qm This did not answer the question. I had the same question. Finding the versions or timestamps insnt the question. The question is about her hardcoding the versions. in a production environment, you wont know how many versions there are so there has to be a way of checkpointing to know how to get just the changes dynamically
Thank you
nice mam