Azure Data Factory Web Activity save output

Courtney Haedke 151

Is there a way to save the output of an Azure Data Factory Web Activity into a dataset?

Here is my current use case:

I have to dynamically build a JSON post request
The API I'm trying to reach requires a SSL certificate so I have to use the Web Activity Authentication Client Certificate option.
The API also requires basic authentication so I input the Content -Type and authorization guid in the header section of the Web Activity.
Once I get the JSON response from my post request I need to save the response into a blob storage some where.
I tried using the Copy Data Set HTTPS or Rest API as a data set source but both only allow one type of authentication certificate or Basic authentication.

Is there a way I can configure the Rest API or HTTPS data set source handle both types of authentications (SSL and Basic Authorization) or capture all the Web Activity output into a blob storage?

Thank you all for your help! I'm desperate at the moment lol..

Here is what my Web Activity looks like (sorry I had hide part of the credentials for security purposes: Web Activity

MartinJaffer-MSFT 26,091 Reputation points

2020-10-16T20:36:03+00:00
Hello @Courtney Haedke , welcome to Microsoft Q&A, and think you for your question. I see you difficulty.

I would like to break this down into 2 parts:

How can I record the output of my web activity which is currently working?

Can I make this work with REST dataset or HTTPS dataset instead?

I have some experience with 1. and will try to address below.
I will need to do a little digging on 2.

Web Activity takes the data away from the realm of "Copy Activity". Therefore we need to find something else to write the data. I have several ways.
In the past I have sent the result of Web Activity to Azure Function App which wrote to blob.
I have also sent the output of Web Activity as input to body of another Web Activity which called the blob REST API, and wrote directly using that.
Courtney Haedke 151 Reputation points

2020-10-16T21:45:07.51+00:00

Hi Martin,

Thank you for getting back to me. Are there examples on how to sent the output of the web activity using another web activity?
Do you have an examples? I'm trying to figure out how to configure the second web activity.
Courtney Haedke 151 Reputation points

2020-10-17T11:09:32.43+00:00

I'm reading the article you sent me. Do I need to create an authorization header in the blob storage? Is there a simpler way to generate the header? A script available some where?
Courtney Haedke 151 Reputation points

2020-10-17T13:19:01.66+00:00

Do I need to generate a shared key every single time before I do my put request?
MartinJaffer-MSFT 26,091 Reputation points

2020-10-19T20:15:51.997+00:00

For placing the output of the first activity in the body of the second, the expression generally looks like:
@activity('Web1').output
or
@activity('Web1').output.data
or something similar depending upon what the output of the first activity looks like. Data Factory adds some properties to the output, such as headers, so your case will need a little customization. Do a debug run, and look at the output of the first web activity.

If you do not see the body section, check what http verb you are using. GET does not have a body, but PUT or POST do.
For example, I target a web activity at https://reqres.in/api/users/2

Since I want to pass the "data" and not the headers, I use @activity('Web1').output.data

Accepted answer

MartinJaffer-MSFT 26,091 Reputation points

2020-10-19T23:18:02.627+00:00
@Courtney Haedke I found an easier way to deal with the authorization. Use Managed Service Identity. This has 2 parts.

Go to your storage account. Go to the IAM / RBAC. Give your Data Factory the Storage Blob Data Contributor role.
Important: "Storage Blob Data Contributor" is not the same as "Contributor"

Go to the web activity. In the bottom under 'advanced' select "MSI". The resource should be https://storage.azure.com/
You may need to set the x-ms-version header to 2017-11-09 or higher.

This should effectively handle the authentication for you.
Please sign in to rate this answer.

2 people found this answer helpful.
Courtney Haedke 151 Reputation points

2020-10-20T12:03:09.19+00:00

This solution worked! Thank you for the great support! I'm excited to continue using Data Factory for my project.

MartinJaffer-MSFT 26,091 Reputation points

2020-10-20T17:15:46.177+00:00

Thank you for the feedback @Courtney Haedke ! I am converting some of the comments to answer to make them more visible to community. Could you please mark the appropriate one as accepted answer, or let me know if I need to convert other comments to answer please? Organizing and markign as accepted answer will improve visibility of this thread and help others benefit.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

1 additional answer

Courtney Haedke 151 Reputation points

2020-10-19T21:58:18.817+00:00

Thanks Martin! I understand that piece. What are the requirements to for the header to complete a put request to an azure blob storage.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure Data Factory Web Activity save output

1 additional answer

Your answer