Concatenate 2 files with different schema in Data Factory

Michael Lawless 0 Reputation points
2024-06-13T21:58:57.7033333+00:00

I have a CSV file that needs to be that need to be concatenated with the results of a pipeline. Essentially the first CSV file acts like the metadata for the pipeline file, so they do not have the same schema. I can't seem to merge them together using the standard tricks (I am not a SQL native).

For example:

File 1 (3 columns):

Location,Date,Version

Oakland,01/01/2024,1.0.0

CSV results of my process (5 columns):

Name,Description,Price,Attribute,Count

WidgetA,FancyWidget,$1.00,Round,3

WidgetB,LessFancyWidget,$4.50,Square,212

My output should look like this

Output File:

Location,Date,Version

Oakland,01/01/2024,1.0.0

Name,Description,Price,Attribute

WidgetA,FancyWidget,$1.00,Round

WidgetB,LessFancyWidget,$4.50,Square

I hope that makes sense.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,159 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Nandan Hegde 31,436 Reputation points MVP
    2024-06-14T05:27:56.94+00:00

    Below link can act as a reference to concatenate the data within a same csv file :

    https://stackoverflow.com/questions/66403101/need-to-add-header-and-trailer-record-in-a-csv-file-azure-data-factory

    0 comments No comments