ADF Copy Activity - Source Schema not fixed

Kesavan R 1 Reputation point
2021-10-08T18:16:49.257+00:00

Hi Experts,

I am still in learning phase of ADF. I have a scenario wherein I use Copy activity with source dataset connecting to SFTP server and fetch csv files from there and storing it in Gen2 containers. Inside the Copy activity, schema mapping has been done between source and target fields and we run the copy activity on a daily basis as the source system guys uploads the file in STFP server once a day.

I have recently come to know that one of the fields in the csv file can go missing on certain days due to the fact that there is no data for that field on that particular day. Due to this, the pipeline run gets failed saying "Column 'XXX' specified in column mapping cannot be found in source data".

Is there any possible solution or workarounds to handle this kind of situation in ADF ? I have asked the source system guys to always upload the files for all the fields, with some blank values, even when there is no data for that field on any particular day. But I would like to know the alternatives or workarounds for this problem if they get back saying that's not possible :(

//Kesavan.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,172 questions
{count} votes