question

ewinkiser avatar image
0 Votes"
ewinkiser asked MartinJaffer-MSFT commented

Need some ADF Guidence on Mapping Please

Hello! @nasreen-akter @MartinJaffer-MSFT
We don't have Data Flows in the Government Cloud yet, sadly. An issue came up today where my Manager would like to know the following. IF we change our json file and add a new field in it such as
85349-image.png

we just had to add hanfordID this morning to this json file. I use the Copy Activity and the json file as a Source and SQL DB as the sink. Ok, today I had to totally remap over 100 fields in in the json file just to get this field in the mapping; it was very time consuming. My Manager thinks surely ADF has a faster way to map when a json file (schema) changes instead of doing it over manually (to map to the sink). Is there a better way? I need to report this back to him. He is saying with Logic Apps it is transparent.

Thanks so much for your opinion...
Mike Kiser


azure-data-factory
image.png (123.6 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

In which gov't Azure region will you need data flows?

0 Votes 0 ·

@ewinkiser if any of the answers solved your issue, please mark as accepted answer. If you came up with your own solution, please share with the community.

0 Votes 0 ·
nasreen-akter avatar image
0 Votes"
nasreen-akter answered MartinJaffer-MSFT commented

Hi @ewinkiser,

In the ADF CopyActivity --> Mapping, if you hover over the source side, you will see "+" sign after each attributes. You will able to add array/object/node in any position. That way you can avoid remapping all the fields in the CopyActivity. Hope this helps! :)

85452-img30.jpg
85461-31.jpg



img30.jpg (56.6 KiB)
31.jpg (11.2 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@nasreen-akter ,
Very cool! I wasn't aware of that; it works great! Thanks once again so much!
Mike Kiser

0 Votes 0 ·

@ewinkiser if nasreen-akter's answer solved your issue, please mark as answer.

0 Votes 0 ·
MartinJaffer-MSFT avatar image
0 Votes"
MartinJaffer-MSFT answered

If you are technically savvy, you could automate this process (home-brewed, not a part of Data Factory).

  1. Write a script/function that can read the pipeline definition, locate the appropriate copy activity, insert new stuff into the mapping.

  2. Get a list of pipelines / copy activities in need of update.

  3. Write a script, that for each pipeline in the list, fetches the definition, uses the aforementioned function on the definition, then saves the altered definition back to the source (Data Factory / Git).

  4. Publish changes and test.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.