ADF Copy Activity dynamic mapping from json source to csv for multiple tables in single copy activity

Question

Hi Guys,

I am working on a task which is to query the salesforce objects by using the REST API and load into the blob storage from the ADF, this needs to be done for 40 objects in the for each loop, I am able dynamically pass the object/table names and querying the data in the copy activity, however I am getting data in the json format and I need to convert it into csv format dynamically in the copy activity,

Below is the sample json for one table, format will be same for other tables also but the columns will be changed.

[
{
"totalSize": 1,
"done": true,
"records": [
{
"attributes": {
"type": "Account",
"url": "/services/data/v52.0/sobjects/Account/0013O00000g7EmSQAU"
},
"Id": "0013O00000g7EmSQAU",
"IsDeleted": false,
"MasterRecordId": null,
"Name": "eon1623321508143 nach1623321508143",
"Type": "Prospect",
"RecordTypeId": "0120Y000000aOWBQA2",
}
]
}
]

if we can just provide the collection reference in the copy activity with out any mapping will it work?, i have tried already by giving $records, but it's not worked, if there is a simple solution for this please suggest.

Answer 1

Hi @Muralikrishna Pirla ,

Thank you for posting query in Microsoft Q&A Platform.

Only adding collection reference will not work, you need add mappings either manually or dynamically.

Below is the sample screenshot which shows mappings.

Manual mapping works only if all the 40 objects/Tables has same schema(column names). If its not same then you need consider dynamic mapping.

When I say dynamic mapping, you need to store your mappings json for each object in some file or tables and read that mapping dynamically and pass it in to Copy activity. To know more about this you can refer below video.
https://www.youtube.com/watch?v=b27gmOufge4

To know more about Schema and mappings in copy activity Click Here.

Hope this will help. Please let us know if any further queries. Thank you.

---------------------------

• Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
• Want a reminder to come back and check responses? Here is how to subscribe to a notification.