ADF Copy Task - Source Varying Column count

Question

ADF reading multiple txt files, there are two types of txt files, 1 type has 18 cols, the other has 24.
The first 18 columns are the same in both types of files, Type 2 only has an additional 6.

If i create my sink mapping with a 24 col file, the pipeline fails when it tries to read an 18 col file

Column 'Prop_19' specified in column mapping cannot be found in source data.

My sink table has 24 columns.

Answer 1

Hey,
You can use getmetadata activity to identify the structure of header and the column count in the txt files.
Based on the column count, you can create seperate copy activity via IF activity/Switch activity.

One can also leverage a single copy activity and create dynamic mappings at run time based on column count:
https://sqlitybi.com/dynamically-set-copy-activity-mappings-in-azure-data-factory-v2/

Similar thread : https://stackoverflow.com/questions/68800592/adf-if-condition-count-source-columns/68800648#68800648