question

CorySeaman-6081 avatar image
0 Votes"
CorySeaman-6081 asked SathyamoorthyVijayakumar-MSFT commented

Azure Data Factory aggregating SQL rows into JSON with path data and values

I've seen how to use the Aggregate transformation in an ADF Data Flow along with a static hierarchy specified in a Derived Columns transformation and the collect() function to output custom JSON.

What I want to do is a little different. If my tabular data contains one column with a single attribute value, and another column with a string representing the JSON path where I want the first column's attribute value to be output in the JSON schema, how would this sort of transformation be achieved?

Table Input:
142233-image.png

JSON Output:

 {
  "user": {
  "id" : 1
  "firstName": "John",
  "lastName": "Doe",
  "address": {
  "city": "Pittsburgh",
  "state": "Pennsylvania"
  }
  },
  "user": {
  "id" : 2
  "firstName": "Jane",
  "lastName": "Doe",
  "address": {
  "city": "Pittsburgh",
  "state": "Pennsylvania"
  }
  },
 }


azure-data-factoryazure-sql-database
image.png (9.9 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

SathyamoorthyVijayakumar-MSFT avatar image
0 Votes"
SathyamoorthyVijayakumar-MSFT answered SathyamoorthyVijayakumar-MSFT commented

Hello @CorySeaman-6081,

Thanks for the question and using MS Q&A platform.

I was able to get this by combining Mappng data flow and Copy Activity. The mapping Data Flow to transform the columns.

142966-image.png

For Demonstration purpose,

I took only the below columns :

 user.name
 entity
 user.address.state
 user.address.city

Source :

142965-image.png

The complete DataFlow :

142986-image.png

The first thing I did was the transpose the AttributePath and Attribute Values using Pivot columns

143051-image.png

For 1 EntityID - there will be one row of the record.

The below is the Pivot Configuration I had

143061-pcgif.gif

There will be drifted columns generated - I mapped them with Window Schema Modifier. Or you can click on the "Map drifted Columns" in the data preview section of the Pivot Action.

143042-image.png

I outputted as a JSON file using a sink transformation

143062-image.png


Now with as a source file - output of the Above DataFlow,

I did copy activity to destination (sink) - as a JSON.

I imported schema from a sample file - this would be file with output that you had shared in the question,

143071-image.png

Now in the Copy activity, you could map the columns as required.

143043-image.png

Output :

142980-image.png

Hope this will help. Please let us know if any further queries.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification

  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators



image.png (10.9 KiB)
image.png (16.1 KiB)
image.png (15.3 KiB)
image.png (25.8 KiB)
pcgif.gif (103.0 KiB)
image.png (8.8 KiB)
image.png (18.6 KiB)
image.png (17.3 KiB)
image.png (30.7 KiB)
image.png (17.0 KiB)
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @CorySeaman-6081,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·

Hello @CorySeaman-6081,

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

0 Votes 0 ·

Hello @CorySeaman-6081,

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

0 Votes 0 ·