Problem merging files in copy activity

Quentin CHIFFOLEAU 1 Reputation point
2022-06-15T13:36:23.677+00:00

Hi everyone,
I'm trying to merge a list of XML files through a Copy Data activity in Azure Pipeline. The problem I encounter is that part of those files have a partially different column name than the others. Example : a column is named 'xxxxxxx' in most of the files but there's some files where the name of the column is 'xxxx[0]xxx'. Can I add dynamic content in the Copy Data mapping to have the source of the column to be : contains('xxxxxxx') instead of the fixed name of the column 'xxxxxxx'?
If It's not possible, is there any way to bypass that by using a Pyspark notebook for example?
Thanks in advance

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,346 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. AnnuKumari-MSFT 30,361 Reputation points Microsoft Employee
    2022-06-16T09:48:23.183+00:00

    Hi @Quentin CHIFFOLEAU ,

    Welcome to Microsoft Q&A platform and thanks for posting your query.

    As I understand your ask, you are trying to data from multiple files into a single file and issue here is that the schema for all the files is not matching. There are differences in the columnNames. Please let me know if my understanding about your query is incorrect.

    Unfortunately, it can not be achieved by copy activity. You need to use mapping data flow to achieve this requirement.

    You can use Select transformation inside Mapping data flow to change the column name from the source. Also , to merge the data from multiple files, you can make use of Union transformation in mapping data flow that will combine multiple data streams into one destination.

    For more info, please watch the following videos:
    select transformation in mapping data flow
    union transformation in mapping data flow

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators