Azure Data Factory - Incremental Data Copy with Parameters

This Topic covers the following:
1. Incrementally Copy Data from one Database to Another
2. Parameters to Support - Can be run for multiple tables using the same pipeline
Other Videos on Azure:
Subscribe:
/ @azurebitsandbytes5804
Azure Portal:
• Azure Portal - Sign up...
Storage Account:
• Azure Storage Account ...
SQL:
• Azure SQL - Introducti...
Data Factory:
• Azure Data Factory - I...
• Azure Data Factory - ...

[@namangarg7023]

Tough to achieve in different real time scenarios but still a masterpiece activity explained for new users..tks

[@Rafian1924]

Lucid explanation. Good job brother.

[@shaileshsondawale8765]

Awesome Content.. Great work done Sir....

[@chandrasekharnallam2578]

nice explanation with easily understanding the concept. thank you

[@pushpakbansal7070]

Very good explanation, helped me in getting the job done.

[@kolliuma9509]

Great Explanation Sir with good content

[@Lihka080]

@
Azure Bits and Bytes
Hi Sir very useful and excellent, could you please add update logic, so if there is any change in source table records it should update the records in sink table then need to take latest from source to sink table. Thanks in advance

[@vishwanathr3507]

Hi
How to Filter multiple values in Parameter like
SELECT * FROM DimCustomer WHERE EnglishEducation IN (@'{Education('LookupEducation').Output}')
not sure it is correct way to pass multiple values in to parameter in adf paramter.
Thank you

[@ashwinds6293]

Am getting an error in pulling the data through ADF for Oracle as a Source . Could you please help me in this

[@juzerpatanwala5133]

where can we find this dataset of customer orders

[@nagarajupinnu3621]

why can't we update watermarkDate in a table directly with query instead of Stored procedure which is extra object

[@azurebitsandbytes5804]

The script activity was not present at the time of making this video. You can use this definitely.

[@krishnakumar-vi3tf]

It was nice,but one question how can i handle multiple tables passing dynamically to pipe line,right now you are entering table names at pipe line level manually but my requirement is i have to pass table list for both lookups dynamically

[@krishnakumar-vi3tf]

i got to know there is another video where we can pass tables dynamically with master pipe line..excellent..its an urgent requirement for me..thanks a lot

[@hungerspoint7358]

How it will handle duplicate data?? If pipeline runs twice at at time the. It will again copy the data??

[@azurebitsandbytes5804]

making use of tumbling window trigger will help avoid overlapping runs.

[@preetijaiswal9089]

Hi, This Really worked and helped me out . Now I need to do CDC based on other column like Id or other unique column so how to implement this as our watermarkcolumn is of datetime type.
Please Help me out.
Thanks in advance

[@azurebitsandbytes5804]

in your case, you can make use of Id value as the watermark column, and change the logic to extract a id range(from and to) and use that to fetch data from source system. Hope it helps.

[@preetijaiswal9089]

@@azurebitsandbytes5804 I need to do Capture Data Change and do the update only on the basis of UpdateTime so how to do that. I really appreciate for your assistance. Thanks.

[@RajeevRanjan-nf6xc]

Thanks a lot sir!!! Can I get ddl scripts for practice

[@sirigiriharikrishna1772]

@@preetijaiswal9089 @azurebitsandbytes5804
Please help how cdc will work in this case?

[@dataengineerazure2983]

PLEASE HOW GET FILE SOURCE FOR TRAINING? TKS

[@wheretastesdivedeep]

How to handle if the table don't have data?

[@azurebitsandbytes5804]

Which table?

[@rohitsethi5696]

no this is not useful.The table should be manually enter. This is 200% end developer can end wrong table or misspell or space in table name end or before. The best way to store all the table name in the Table then for each loop process