Data engineering Project : Incremental loading on azure !!

Hello everyone,In this video i explained about the incremental loading an essential concept in data engineering,Please follow through this video fully to clarify your doubts
👥For Collab, Sponsors & Projects: [e-mail]
✅Keep supporting me to get 25K subscribers ❤
✅Important Links:
learn.microsof...
Pipeline template:github.com/akp...
Socials:
✅Ask your Doubts on Instagram: arun.codes
✅Telegram: t.me/s/akpython (1.5K subscribers )
✅Twitter: / ak_python
✅Codes & Donations:
🧑‍💻Github: github.com/akp...
✅If You get impressed with my work Then Buy me a cup of coffee:
💰www.buymeacoff...
🎶Music by:
chill. by sakura Hz / sakurahertz
Creative Commons - Attribution 3.0 Unported - CC BY 3.0
Free Download / Stream: bit.ly/chill-sa...
Music promoted by Audio Library • Chill - sakura Hz (No ...
✅Thanks ..!
🔊Disclaimer:
All the videos, songs, images, and graphics used in the video belong to their respective owners and I or this channel do not claim any right over them.
Copyright Disclaimer under section 107 of the Copyright Act of 1976, allowance is made for “fair use” for purposes such as criticism, comment, news reporting, teaching, scholarship, education, and research. Fair use is a use permitted by copyright statute that might otherwise be infringing.”

[@swa2u86]

Hi , how the steaming real time data will load with business logic can you pls share with example

[@prabhatgupta6415]

bring incremental loading using azure databricks

[@AKPython]

Yep sure

[@prabhatgupta6415]

Brother please bring..all real time scenarios on Databricks only real time which you face daily and developer should know. Bring something on autoloder..

[@prabhatgupta6415]

its not upsert man

[@AKPython]

Hi it's a different method using load time and load date, if you want upsert i recently uploaded a video using upsert using adf please check

[@prabhatgupta6415]

sure will check,@@AKPython

[@prabhatgupta6415]

basically what is followed by org?
Client sent data and we hv to load in adls till gold layer so for that we r really required to perform upsert right?? not only insert.
What is being followed in ur org?

[@AKPython]

It's depends on your projects, There two type of load are there one is full load and another one is delta load, Insert will suitable for full load and for delta upsert is perfect one