Is it possible to merge files with headers and without headers?

Question

Hello.
I want to merge the file with headers and without headers in Azure Synapse pipeline, but I am facing a problem.
What I would like to achieve is as follows

When merging in a pipeline copy activity, the merge will not work because the files that can be selected for the source data set are either with or without headers.

・If the source data set is merged with a header, the first line of each file will be the header, so data from files without headers will be lost.
・If the source data set is made without headers, the headers of files that have headers will be included in the data.

I was using the method of removing headers from files with headers and manually entering the schema in the copy activity mapping, but it is nonsense to manually set the schema since I want the pipeline to run automatically every week.

Is there a way to merge only the data in each file using the headers of the file with headers?
Any answers would be appreciated.
Thank you.

Answer 1

In short, no you can't merge files with-headers and without-headers. All must be one or the other.

So, the next step is to alter them so they are the same!

A lookup activity can get just the first row. If you know what the headers look like, then use an if-condition activity to check whether the returned row looks like a header. Then a copy activity can remove the header by using 2 similar datasets, one with "first row as header" (source) and one without "first row as header" (sink). This re-writes the file in place.

Once all the files processed, we use a copy activity to merge them.

@Shunya Kakehi