How to use Azure Data Factory Web Activity to post a file to a API

Question

Hi guys. In postman I can upload a file to an API. But how do you do it in ADF, using the web activity. I haven't found any detail explanation on the web activity dataset and link service properties. Is it possible in ADF to use this activity to post a file to an API? If so please may I ask if you can show me some documentation on how to achieve this, as it will be much appreciated. Thank you

Answer 1

Hi D Williams ,

Thank you for sharing the insights with the community on the above query, hope it helps others experiencing similar doubt. Since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others ", I'll repost your answer in case you'd like to "Accept " the answer.

As of now, there is no direct way to post a file to a API using web activity in azure data factory pipeline. The workaround you can try is to use logic app or azure function and write your own custom code to achieve the same.

Please accept the answer.

Answer 2

Hello @D Williams

Welcome to the Microsoft Q&A and thank you for posting your questions here.

To understand your question, you would like to upload file to API in Azure Data Factory using Web Activity, and you need documentations on how to do this or for references.

This is one of the common scenarios that many developers face when working with REST APIs.

1. One option is to use the Data Factory Web Activity to send a POST request with the file as a parameter. You can set the filename as a dataset parameter and then use a pipeline parameter to pass it to the web activity. You also need to set the appropriate headers and body for the request. You can find an example of this approach in this Stack Overflow answer by Leon Yue.
2. Another option is to use the Copy Activity to copy the file from a source dataset (such as Blob Storage) to a sink dataset (such as REST API). You can configure the sink dataset to use the POST, PUT or PATCH methods and specify the URL, headers and body for the request. You can find more details about this option in this Microsoft documentation.
3. A third option is to use a custom activity to execute a Python script that uploads the file using the requests library. This option gives you more flexibility and control over the request, but it also requires more resources and configuration. You can find an example of this option in this Stack Overflow question by Greg Low.

Finally, uploading a file to an API in Azure Data Factory using a Web Activity involves a few steps. You can use the Web activity to make an HTTP POST request to the API endpoint and send the file as part of the request.

For documentation purposes and references, use the following links:

• Azure Data Factory Documentation
• Web Activity in Azure Data Factory

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please remember to "Accept Answer" if answer helped, so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam