How to merge or append two .csv files having same no. of columns without using copy activity merge files scenario. I need to either use a query or something else ??

Question

So, we have got a scenario, where we are copying a table data through ADF pipeline, then we are having a copy activity for getting log (which include the pipeline name, table name & records count) . While copying the log, we getting an individual log file as per the table. I need to get all the log table details in one .csv file . I have already tried copy activity -copy behavior -merge file scenario. It's increasing the performance time of the pipeline. any other way or recommendation. ??? would be helpful.

Answer 1

hello Prashant, Good day!

1. You can leverage dataflow activity as well to merge the files in to a single final aspect
2. Do you have Synapse serverless or synapse dedicated pool in your environment ?
3. If yes, then you can create an external table mapping to the folder location that contains all diff files with same schema and executing select * from external table would give you the overall data (union of all files) in your output which you can copy into a single file

Answer 2

Hello @Prashant Singh , Thanks for the question and using MS Q&A platform. I could request you to evaluate the wildcard option (assuming that the all the log files are in the same container\folder) and let us know if this works out for you.

Thanks Himanshu Please accept as "Yes" if the answer provided is useful , so that you can help others in the community looking for remediation for similar issues.