Great content, Thanks. How do you deal with deletion of records? I have an OLTP server (SqlServer) where application can delete records, so trying to understand 1. how to identify delete 2. Treatment for deleted records in the late (something like making them inactive or highenddated etc)
There are multiple way to handle this. Some of following way generally use. 1. Instead of Hard Delete use soft Delete with flag 2. if soft delete is not possible then you can set delete trigger and keep record in another table and use that table to update OLAP. 3. we can also enable audit or CDC to find out deleted records and using audit or CDC record we can update OLAP. let me know if this make sense
Hello, I have a problem with the incremental load I want to create an incremental pipeline from the Oracle on-premise server to Azure data lake(blob storage) I don't have Azure SQL. I just want to push in blob storage as a CSV file. in my case, I have confusion about where I should create the watermark table & store procedure. someone told me in your case you have to use parquet data. please help me with this I am stuck for many days.
watermarking is one of easiest and widely used solution where we do store last successful push time for given table and then use store proc to get and update this table. only catch here that table which you have to push must have a timestamp column to compare and find change after last run.
@@SoftWizCircle I am pushing table directly to blob storage account I don't have azure SQL and I am getting table from on premise server SQL so that's why I have a confusion where I need to create this. Please help me for this. Is it possible can give me 5mnt on call show I can show you my problem
Please try once. last modified date is a column in sql table in this demo and we can parametrize sql statements or stored proc etc and pass value from adf
Great content, Thanks. How do you deal with deletion of records?
I have an OLTP server (SqlServer) where application can delete records, so trying to understand 1. how to identify delete 2. Treatment for deleted records in the late (something like making them inactive or highenddated etc)
There are multiple way to handle this. Some of following way generally use.
1. Instead of Hard Delete use soft Delete with flag
2. if soft delete is not possible then you can set delete trigger and keep record in another table and use that table to update OLAP.
3. we can also enable audit or CDC to find out deleted records and using audit or CDC record we can update OLAP.
let me know if this make sense
Can u please mention the syllabus, which is sufficient to get job in ADF. Also kindly order ur playlist
sure. i will do in sometime. apologies for delay
Hello,
I have a problem with the incremental load I want to create an incremental pipeline from the Oracle on-premise server to Azure data lake(blob storage) I don't have Azure SQL. I just want to push in blob storage as a CSV file. in my case, I have confusion about where I should create the watermark table & store procedure. someone told me in your case you have to use parquet data. please help me with this I am stuck for many days.
watermarking is one of easiest and widely used solution where we do store last successful push time for given table and then use store proc to get and update this table. only catch here that table which you have to push must have a timestamp column to compare and find change after last run.
@@SoftWizCircle I am pushing table directly to blob storage account I don't have azure SQL and I am getting table from on premise server SQL so that's why I have a confusion where I need to create this. Please help me for this. Is it possible can give me 5mnt on call show I can show you my problem
Even I'm stuck on this issue
Good Article, Lack of Visibility of the Video.
Sorry for that I tried to record in HD this abut still seems somehow blur.
what the fuck....how can lower lastmodifytime can pass through it..meaning the copy condition gets false and data should not be copied
Please try once. last modified date is a column in sql table in this demo and we can parametrize sql statements or stored proc etc and pass value from adf