ADF Copy Task - Source Varying Column count

Ivan Nel 26 Reputation points
2021-08-16T08:40:13.157+00:00

ADF reading multiple txt files, there are two types of txt files, 1 type has 18 cols, the other has 24.
The first 18 columns are the same in both types of files, Type 2 only has an additional 6.

If i create my sink mapping with a 24 col file, the pipeline fails when it tries to read an 18 col file

Column 'Prop_19' specified in column mapping cannot be found in source data.

My sink table has 24 columns.

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,488 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,782 questions
{count} votes

Accepted answer
  1. Nandan Hegde 29,911 Reputation points MVP
    2021-08-16T09:52:07.193+00:00

    Hey,
    You can use getmetadata activity to identify the structure of header and the column count in the txt files.
    Based on the column count, you can create seperate copy activity via IF activity/Switch activity.

    One can also leverage a single copy activity and create dynamic mappings at run time based on column count:
    https://sqlitybi.com/dynamically-set-copy-activity-mappings-in-azure-data-factory-v2/

    Similar thread : https://stackoverflow.com/questions/68800592/adf-if-condition-count-source-columns/68800648#68800648

    2 people found this answer helpful.
    0 comments No comments

0 additional answers

Sort by: Most helpful