Combine multiple rows into single row in dataflow in Azure Data Factory

Question

Hello All,

I have a source XML where the data is flatten based on lowest level. Post flatenning, my data is looking like this

Key Row1 Row2 Row3 Row4 Row5
101 A
101 NULL B
101 NULL NULL C
101 NULL NULL NULL D
101 NULL NULL NULL NULL E
102 A NULL NULL NULL NULL
102 NULL B NULL NULL NULL
102 NULL NULL NULL D NULL
102 NULL NULL NULL NULL E
103 A NULL NULL NULL NULL
103 NULL B NULL NULL NULL
103 NULL NULL C NULL NULL
103 NULL NULL NULL D NULL
104 NULL B NULL NULL NULL
104 NULL NULL C NULL NULL
104 NULL NULL NULL D NULL

Now, I want the output to be in a single row for each key field.

Key Row1 Row2 Row3 Row4 Row5
101 A B C D E
102 A B D E
103 A B C D
104 B C D

Is there any other method to achieve this with any transformation apart from a regular JOIN transformation?

Answer 1

Hi @Alekya Seemakurty, Sri ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query.

As I understand your query, you want to combine multiple columns into single column based on aggregation on Key column here. Please let me know if my understanding is incorrect.

You can try the following approach to achieve the above requirement:

1. Create a dataset pointing to the source file . I have stored the above provided source data in .csv file . Make sure to select Edit option in Column delimiter and provide Space ' ' in the textbox as your column is not separated by comma delimiter. Also, check first row as header.

Data looks like this :

2. Use Aggregate transformation and group by using 'Keys' column and use collect(Row1) in the expression of aggregate tab.

3. Use Derived column transformation to create Columns named - 'Row1','Row2','Row3','Row4','Row5'

4. Use Select Transformation to select only necessary columns

5. Use Sink transformation to load the data in csv. Check the first row as header option.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators