question

RiyazPatanwala-0761 avatar image
0 Votes"
RiyazPatanwala-0761 asked MartinJaffer-MSFT commented

ADF - How to post CSV format data in the Rest API Body

For Salesforce Bulk API Rest API, we need to post data in CSV format in the body. Our CSV file is stored in a Blob Storage, how can this file be referenced in the API body, so that data from the CSV file is posted to the API call.
In Web Activity we have a dataset setting, can this be leveraged to be used as source and its content posted in the body.

87164-adf.png


azure-data-factory
adf.png (47.0 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

MartinJaffer-MSFT avatar image
1 Vote"
MartinJaffer-MSFT answered MartinJaffer-MSFT commented

Hello @RiyazPatanwala-0761 and welcome to Microsoft Q&A.

Unfortunately, the web activity feature you are pointing out does not do what you think it does. To my understanding, it takes the definition of the Linked Service / Dataset and adds it to the body. It does not let you put the actual data into the body. A very niche case in my opinion.

Instead, the method I use to pass data in web activity, is to first fetch the data using another web activity.
That is, first a web activity points to the blob you want to read. This fetches the data and makes it available in the activity output.
Then re-use the output in the body of the "Insert Data" web activity.
Below are details of how to set up the reading web activity for blob.

87451-image.png

URL: https://myStorageAccount.blob.core.windows.net/myContainer/myFolder/myFile.txt
Headers: x-ms-version 2017-11-09
Advanced -> Authentication: Managed Identity
Advanced -> Authentication -> Resource: https://storage.azure.com/

By selecting Managed Identity authentication, you need to go to the storage account, and grant blob reader / blob contributor role to the Data Factory.

In the body of "Insert Data" use something like @activity('Get stuff').output.Response to pull in the results of the read.

Do keep in mind, the Web activity returns at most 4 MB of data. If your file is bigger, you may run into problems.


image.png (157.5 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks Martin
It worked like a charm. Appreciate your timely help.

Regards
Riyaz

0 Votes 0 ·

Thank you for the feedback. Have a great week!

0 Votes 0 ·