Dynamic nested json flattening

Question

We are using data flows to flatten multiple disparate nested json files in order to load the data in a tabular format in Azure SQL.

At this point we have created n number of data flows for n files and the mapping has been done manually per json file. With more incoming json files and their structure is changing constantly it is getting difficult to maintain the same.

Kindly suggest a better solution to handle the above situation.

Regards
Swapnil Sarkar

Answer 1

Hi @Swapnil Sarkar ,

Thankyou for using Microsoft Q&A platform and thanks for posting your question here.

As per my understanding, you have a requirement to flatten multiple JSON files dynamically . Please let me know if my understanding is incorrect.

In flatten transformation , for Unroll By, you can use Add Dynamic Content. Additionally for defining the columns, you can use Rule-based Mapping.

For dynamic names, use Rule-based mapping instead of fixed mapping. You can use regex and wildcards there as well as Deep column traversal to expand all nested elements.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

Hi Annu,

Thanks! for such a quick response.

I am comparatively new to using data flows and would really appreciate if you could share some link or tutorial video where this dynamic rule-based mapping has been applied.
Additionally, the Unroll By option when passed as an expression can it be used to flatten out the multiple array data in one go?

Adding a sample json for more clarity:
JSON File:
{
"type" : "abcd",
"request" : {
"header" : {
"authorization" : {
"classId" : "102481"
}
},
"body" : {
"subjects" : [
{
"sequence" : "1",
"person" : {
"givenName" : "TEST1",
"surname" : "value1"
}
},
{
"sequence" : "2",
"person" : {
"givenName" : "TEST2",
"surname" : "value2"
}
},
{
"sequence" : "3",
"person" : {
"givenName" : "TEST3",
"surname" : "value3"
}
}
]
}
}
}

SQL Table:
classId sequence givenName surname
102481 1 TEST1 value1
102481 2 TEST2 value2
102481 3 TEST3 value3

Regards
Swapnil