ADF Dataflow: Flatten json file from a dynamic source

alexis 101 Reputation points
2022-11-09T17:59:36.313+00:00

Hi,
I managed to flatten my json file with a static source but when I change the source to dynamic, the array that I try to flatten is not available.
Is it that data flow cannot accept dynamic source?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,928 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. MartinJaffer-MSFT 26,046 Reputation points
    2022-11-10T20:44:21.21+00:00

    Hello @alexis ,
    Thanks for the question and using MS Q&A platform.

    The drop-down menu used to select what to flatten ("Unroll by"), is populated by the schema (in the source) and anything added from previous transformations.

    So when the source schema is empty, it has nothing with which to populate the list of stuff you can flatten ("Unroll by").
    259177-image.png
    I suspect the "dynamic" source is clearing out the schema.
    259221-image.png

    I recall the flatten transformation gave me much trouble when I wanted to do something without schema. I think I ended up going with derived column and the unfold function. The unfold function is a lot like the flatten transformation, but it can be used as an expression.

    There may yet be another option in defining an expression, like using byPath.

    Before I get too far afield, what did you mean by static vs dynamic source? I thought you meant something like, a rest source whose relative URL changed, or a parameterized linked service, or a sql table chosen at run-time. Something which would break the guarantee of a stable schema.

    Please do let me if you have any queries.

    Thanks
    Martin


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
      • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators