14. Enable partition discovery in copy activity in Azure data factory pipeline

We learnt enablement of partition discovery in copy activity in ADF pipeline
#adf #azuredatafactory #azuresynapseanalytics #datafactory #dataengineer #microsoft #cloud

[@Kyonyx.]

Thank you for your contribution

[@rianflamenco8212]

Very informative for a fundamental look. And I appreciate the walkthrough tests. Thank you!

[@abhayverma-xm7kg]

Nicely explained!

[@amahi-pu8bt]

Hi Annu, your videos are of great help. Can you also make an incremental data load(delta load, watermark table --- mentioning these technical terms in understandable way) from blob to sql or sql to sql on a priority basis please. Thank you in advance !

[@azurecontentannu6399]

Hello thankyou for watching the videos. Sure will try to create the mentioned scenario. Meanwhile, please check if the videos in synapse playlist helps you for incremental load: ua-cam.com/video/FXw1gPaa2-M/v-deo.html

[@amahi-pu8bt]

@@azurecontentannu6399 Sure, thank you for your response, will watch the content as you shared, consider my concern and please help in that incremental load scenario as well.

[@sureshraina321]

Mam , everything is clear.
Only one doubt from myside -
I have the following folder structure :
====================================
Input(container)/folderA(directory)/source_file.csv(file)
I enabled the partition discovery
But not provided the partition root path
Copy nature : File path available in dataset
In this case , how it will behave mam?
I ran with above config and I could see same as normal copy activity.
Please help me

[@venugopal-nc3nz]

Can you make video on what are common error occurs when a pipeline break and how do you recover it ?

[@nitinpandey4857]

The folder nomenclature needs to be in the format "column_name=row_data".?
Will it accept anything apart from = operator.?

[@azurecontentannu6399]

Hi Nitin
Yes the folder nomenclature needs to be in that way else it will fail

[@nitinpandey4857]

@@azurecontentannu6399 great, thanks for clarification.

[@sudarshanthota3369]

Can you please start videos on adls gen1 migration to gen2

[@surajbasha9062]

How can we do optimization in copy activity

[@PavanKumar-eu5qg]

Hi
Is there any way to capture list of folders and file names recursively for a selected container?

[@RobinPandita]

Get Metadata activity can help you to achieve the same.

[@swapnilnarvekar3139]

Hi , will this help with reading the latest incremental data from adls ?
If incremental data is loaded into year=2023/month=06/day=21/ folder then will this data be fetched automatically?