Dynamic nested json flattening

Swapnil Sarkar 6 Reputation points
2022-11-07T11:52:48.873+00:00

We are using data flows to flatten multiple disparate nested json files in order to load the data in a tabular format in Azure SQL.

At this point we have created n number of data flows for n files and the mapping has been done manually per json file. With more incoming json files and their structure is changing constantly it is getting difficult to maintain the same.

Kindly suggest a better solution to handle the above situation.

Regards
Swapnil Sarkar

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,897 questions
{count} vote

2 answers

Sort by: Most helpful
  1. AnnuKumari-MSFT 31,731 Reputation points Microsoft Employee
    2022-11-08T11:17:21.473+00:00

    Hi @Swapnil Sarkar ,

    Thankyou for using Microsoft Q&A platform and thanks for posting your question here.

    As per my understanding, you have a requirement to flatten multiple JSON files dynamically . Please let me know if my understanding is incorrect.

    In flatten transformation , for Unroll By, you can use Add Dynamic Content. Additionally for defining the columns, you can use Rule-based Mapping.

    For dynamic names, use Rule-based mapping instead of fixed mapping. You can use regex and wildcards there as well as Deep column traversal to expand all nested elements.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    1 person found this answer helpful.

  2. Swapnil Sarkar 6 Reputation points
    2022-11-08T12:56:53.467+00:00

    Hi Annu,

    Thanks! for such a quick response.

    I am comparatively new to using data flows and would really appreciate if you could share some link or tutorial video where this dynamic rule-based mapping has been applied.
    Additionally, the Unroll By option when passed as an expression can it be used to flatten out the multiple array data in one go?

    Adding a sample json for more clarity:
    JSON File:
    {
    "type" : "abcd",
    "request" : {
    "header" : {
    "authorization" : {
    "classId" : "102481"
    }
    },
    "body" : {
    "subjects" : [
    {
    "sequence" : "1",
    "person" : {
    "givenName" : "TEST1",
    "surname" : "value1"
    }
    },
    {
    "sequence" : "2",
    "person" : {
    "givenName" : "TEST2",
    "surname" : "value2"
    }
    },
    {
    "sequence" : "3",
    "person" : {
    "givenName" : "TEST3",
    "surname" : "value3"
    }
    }
    ]
    }
    }
    }

    SQL Table:
    classId sequence givenName surname
    102481 1 TEST1 value1
    102481 2 TEST2 value2
    102481 3 TEST3 value3

    Regards
    Swapnil