#31. Azure Data Factory - Mapping data flow with Derived column transformation added

Mapping data flow lets us load records from source to sink while performing some transformations. In this video we explain the properties of mapping data flow with a demo. Please watch and let me know your feedback or comments.

[@LearnerMsbi]

Very clear explanation between two types of data flow. Thank you.

[@suniguha]

Excellent. All your videos are full of clarity. Thank you.

[@sureshareti2843]

Easy to learn everyone 👍👍👍

[@AllAboutBI]

Glad you think so!

[@sivakumargannamani5145]

Thanks for your nice explanation, keep it up.

[@alavenkatarao685]

Hi mam if you have possible time please create the source and sink its much better understanding the class and more helpful to us.

[@sureshareti2843]

Nice explanation 👍

[@AllAboutBI]

Thanks 🙂

[@praveenau]

Hi mam
Your videos are very helpful and easy to understand.
Can you do a demo for this scenario
Delete rows for current year from existing parquet file in adls gen2 and connect to the onPrem SQL server and load the current year data and append to the same parquet file in adls gen2 lake storage.

[@chethangowda4816]

Hi Mam,
I have excel sheet with 3 columns and 10 records. 6 of the records have all the values in them and the remaining 4 records were detected as NULL's
how can we work on removing those 4 NULL records ??

[@arnabsengupta2562]

Thanks. This really helps

[@AllAboutBI]

Glad it helped!

[@suvratrai8873]

Hi... I am facing a problem in the data flow while working with column headers. Lets say I have 2 columns with headers in my source. If I preview it, instead of 2 it shows 4 columns (with header same as the source), the first two columns containing all NULL values. Could you suggest where I might be going wrong?
Note: I am facing the above problem only if my excel contains headers. Without header, it is working fine.

[@AllAboutBI]

hi Suvrat, in the excel, click on header, check each cell if it has some formatting applied apart from columns we want to use. for instance, if you hv data in column AA n BA, check the cells CA and DA. CA n DA should have no formatting applied. if there s no data and still hv formatting, they are considered to be valid cells.

[@suvratrai8873]

I checked the cells. None of them have any formatting. Actually the issue happened because of schema drifting. Out of those 4 columns, the ones with the NULL values were being treated as the source columns and the ones with the data were being treated as drifted columns. I had to click on 'Map drifted' and remove the NULL columns later for the proper output. Anyways, could you share the excel you used in this video, so that I can compare it with the excel I am using?

[@SantoshSingh-ki8bx]

I have created 5 columns in excel and one derived column. But in preview section of dataflow it is inserting 11 columns. i.e. first 5 column with null value and 6 actual expected column. I tried every possible way to resolve it. But issue is still there. Please help

[@AllAboutBI]

Pls share the file to [e-mail] I'll have a look and confirm