How to do Incremental Loading using CDC in Azure Data Factory to Data Lake

[@gunasekharraju]

The best video for CDC with ADF, thank you

[@pytalista]

Thanks 🙏🏻

[@shaikhjaved7839]

Nice video, but I have scenario where we have multiple tables with cdc enabled, do I need to create separate pipeline for each tables, because you just pick customer table, even MS document shows with one table, is there any limitation ?

[@junqueirapedro]

You can still have the same pipeline as a template but you have to have some sort of parametrization that changes the source and sink table and use the template for the datasets as well.
The way you do it you need to have a metadata table in a SQL db where you store the parameters to send to the pipeline as input, for example, where you can store the parameters. Then you use a lookup activity that reads the metadata then execute the same template cdc pipeline for each table in the metadata database.
This probably is a good idea for a video

[@pytalista]

Hi @shaikhjaved7839 this is the reply. Cheers

[@KhalidDalvi]

Hi Dear thanks for the video, Can you provide me query to How to do Incremental Loading using CDC in Azure Data Factory to Data Lake for Multiple table in one pipeline

[@pytalista]

For that you can loop this pipeline passing the parameter for each table