Azure data Engineer project | Incremental data load in Azure Data Factory

[@safkaify7875]

Well prepared, brief but precise, well spoken, nicely explained. I learned more in 15 minutes than I could not learn spending two hours from ADF docs on a similar tutorial. Keep up the good work.

[@gargipatilakshmi1782]

Great explanation thank you this helped me in my project

[@johnnybegood948]

Thanks for the tutorial!
I got a questions. What if you had 2 new rows to inserted into the orders.
It only would insert the last on, right? Not the two new ones?

[@learnbydoingit]

Insert both the rows

[@SrinivaSLakma]

Excellent tutorial

[@manjuagrawal5777]

Thanks pls upload more videos 🎉

[@learnbydoingit]

Sure

[@tausiqzuman9196]

Great . Keep uploading new videos

[@learnbydoingit]

Thanks I will upload more

[@naveed3600]

what is the difference between this method and the upsert option available in the copy activity?

[@nagrajganiga9155]

Hi, it was an informative video, I have a question, if you have to do the same with source as parquet files and destination as azure sql server, how would you do it?

[@learnbydoingit]

Same process

[@abhishekasane2635]

Upsert function will work same, right ?

[@sarvesh714]

how will incremental upsert and incremental append work in this scenario?

[@prititiwari8156]

Thanks

[@gauravdevgan79]

how do we manage delete? if source table is huge (size =200gb) how do we identify if some of the rows are deleted from it before inserting into target table?

[@safkaify7875]

I have the same question.

[@sunithareddy850]

Can we do it on onprem instead of Azure SQL?

[@learnbydoingit]

Yes check other videos we have done that using onprem

[@sunithareddy850]

@@learnbydoingit I mean using same data

[@learnbydoingit]

@@sunithareddy850 yes u can do it

[@saranyasdiary9325]

How it can be implemented if source is sftp and sink is azure sql

[@esdras-rocha]

Thanks!

[@MultiVenu]

What if there's no column regarding time which should be taken into consideration. Then how can we achieve this?

[@learnbydoingit]

Ur csv or file time u can consider

[@pallavisanap4175]

can you create vdeo on like if few files like one palce 1000 files but i want 100 files csv need to copy from one place to another folder in storage account real time senario is this

[@Ramana6783]

Could please make a vesio of full load copy activity from sql on prime to azure sql or adls

[@learnbydoingit]

It's available in the channel

[@rajudasari8482]

What if we have date column without time stamp? how to do incremental load

[@learnbydoingit]

Better to maintain or u can use current timestamp and compare

[@rajudasari8482]

@@learnbydoingit yes, but source data contains only date without time stamp, in my requirement data will be updated for any every 2 hrs in a day without a timestamp column

[@HARIRAMA6]

Hi @learn by doing, This logic will not work for data that was inserted into source during the pipeline run since data and time will be same for new data inserted in to source as well the max time of sink. Its better that we copy data and time from source to sink.

[@learnbydoingit]

We have to schedule specific time not the same time

[@learnbydoingit]

Few overlap scenario also can occur

[@ayaansk99]

but how can we add insertion time in source table?

[@learnbydoingit]

Generally you will se date time filters in source like ...change time,insert time,current time like that
Or if you are preparing source use current date to insert it

[@t-bar-mranch1074]

what about deletes?

[@learnbydoingit]

Delete,?

[@prabhatgupta6415]

what about the update and scd?

[@learnbydoingit]

Will do

[@shannkarsap5576]

Please do lab on new & updated records
@@learnbydoingit

[@chitrarekhatiwari6629]

Thanks