8.2 Incremental data load in Azure Data Factory

All Azure Data Factory Videos link : • 12 ADF Filter Aggregat...
Github Link: github.com/vin...
Github DataPipelines: github.com/vin...
Github DataFlow: github.com/vin...

[@shanthababu]

Excellent approach, Please explain adequately to learners.

[@saipraneeth1660]

thank you for the video sir.. yesterday i got same question in the interview

[@cloudguru3178]

Hope my explanation helped you Praneeth

[@SantoshKumar-yr2md]

you should explain from very begining like when you created table in your datasource, nice

[@teunjacobs5005]

Hi Cloud Guru!
First of all, thanks for the clear explanation. It really helped me in creating a incremental data load in ADF.
Never the less I was wondering if you could make a video of making the tables which should be synced incremental more dynamic. Now I've hard set in the lookup action that it should look for a specifc table. I've more tables which should be synced incremental, so I assume that should start working with an iterate activity like a for each(?). So that in a specific SQL table I've defined which table(s) should be synced incremental and that it does that automatically. Do you have any ideas of tips and trics for things like this? Thanks in advance!

[@davidjasonmusic]

Hello i need help - my data sources are APIs from BC...so what I am supposed to do with those datasets?....copy them into blob in order to filter by last modified date back in ADF? That makes no sense what about copying IN to the blob? I just have to copy the whole fact table over everytime? That cant be right. All these methods of incremental copy seem to start with a csv from blob storage. But how do I even get to that point?

[@suniguha]

Very useful content. Thank you.

[@cloudguru3178]

Thanks bro

[@trendy_techer2202]

Thank you so much sir for the valuable info

[@cloudguru3178]

Thank you

[@chandandacchufan3242]

should explain how you are updating the table tbl_control

[@souranwaris142]

Hello Sir.
I have a problem with the incremental load I want to create an incremental pipeline from the Oracle on-premise server to Azure data lake(blob storage) I don't have Azure SQL. I just want to push in blob storage as a CSV file. in my case, I have confusion about where I should create the watermark table and store procedure. someone told me in your case you have to use parquet data. please help me with this I am stuck for many days.

[@sainathreddynandyal8761]

I encountered with same use-case brother. Can you tell me if you found a solution please?

[@harrydadson7467]

Please can someone please assist me on how I'm not able to do a Copy Activity to Stored Procedure on MySQL table in sink setting. Please help. Thanks

[@alexcarter-jones3251]

Great video! Have you ever set up an incremental load between an Oracle source table and an SQL sink table before? I am currently trying to do this but can only get my delta load to work when copying from SQL to SQL as you did in this video, any guidance would be greatly appreciated :)

[@cloudguru3178]

Hi Alex, please try this
@Concat('select part of query ', 'WHERE si.schedule_date BETWEEN ' , formatDateTime({variables('LastDate')}, 'yyyyMMdd') , ' AND SYSDATE-1')

[@alexcarter-jones3251]

Hi @@cloudguru3178, thanks for replying and apologise for the delay in my reply. I managed to write my own working solution using a query script prior to a simple copy data pipeline then using an upsert on the ID column. Query script is as follows 'select * from OracleSourceTable where LAST_UPDATE_TSTMP > (CURRENT_TIMESTAMP - 0.04)' looks very similar to your solution. I think the main issue I was having was trying to use templates that are only set up for Azure SQL DB to Azure SQL DB transfer.

[@vasistasairam5483]

The explanation is very good, but while trying to implement the same, getting errors, especially with formulas. Kindly make videos little more detailed.

[@cloudguru3178]

Let me know what error you are facing and i shall help u

[@swaminoonsavath5363]

Expression of type: 'String' does not match the field: 'additionalColumns' I am getting this error .. MY source is Servicenow and Destination is AZURE SQL DB .. Please help me on this

[@superyan4100]

Remove the 'first row only' checkbox.

[@camvinh3522]

why do we need control table for the last update time - since we can get it info in destination table

[@cloudguru3178]

What if loading to destination tables fails ??

[@camvinh3522]

@@cloudguru3178 so it means that after we loading success to destination table then we update the control table ( if it fails we will not update the control table ) and base on the control table we can handle issue. Is it right ?

[@camvinh3522]

@@cloudguru3178 thanks you for your support

[@cloudguru3178]

@@camvinh3522 We will update the control table if it is failure or success. But to get the maximum date for table you need to consider only last successful load.

[@ssbeats677]

Hi sir, in my source path i have a files i have created a shedule trigger to run every 1 hour. My issue is from source to sink the files are getting triggered duplucates files
Eg : last hour i have 10 files trigger nxt in my source path i recived 5more files. When it trigger the files of last 10+5 files are getting to the sink path

[@ssbeats677]

I need only latest and modified files

[@cloudguru3178]

Kindly move the file to another location once it is loaded so that there will be no duplicates

[@cloudguru3178]

If you would like to connect then call me on 9113070560

[@muapatrick4174]

is it possible to add theses incrementals on parquet files on adls?

[@cloudguru3178]

Hi Patrick it is not possible with the files

[@muapatrick4174]

@@cloudguru3178 Thanks