How to extract a Function App activity output through a Rest Api Call from a different azure data factory.

Shivoy Thakral 20

Hi All, I am dealing with a complex problem statement. I am developing an ETL process which uses two data factories. I am triggering a particular pipeline from one data factory using a rest api call. Then through another rest api call - I am retrieving the run status of the triggered pipeline. All subsequent activities in this data factory can go on once the run status of the pipeline is "Success" or "Failed". Now - I have a function app activity in the pipeline that I have triggered via the rest api call. I want to retrieve the output of that activity so that I can use it in the subsequent activities in the ADF that originally triggered the pipeline. What can I set as the headers or the body , or what can I use (say azure monitor) to retrieve the output of the Function App activity in the triggered pipeline ?. It would be of great help if someone could help me out with this. Thanks a lot 🧿

Smaran Thoomu 15,845 Reputation points Microsoft Vendor

2023-09-27T09:17:33.11+00:00
@Shivoy Thakral
To obtain the output of a Function App activity in a triggered pipeline from another data factory, the most effective approach is to leverage the Azure Data Factory API. This API offers a mechanism for retrieving the output of any activity within a pipeline, including Function App activities.

To get the output of a Function App activity, you can use the following steps:

Get the ID of the Function App activity run. You can do this by calling the Get Activity Runs API.

Make a GET request to the Get Activity Run Output API. Be sure to specify the ID of the Function App activity run in the request.

Parse the JSON response to extract the output of the Function App activity.

Here is a sample code snippet for retrieving the output of a Function App activity:

Python import json from azure.identity import DefaultAzureCredential from azure.mgmt.datafactory import DataFactoryManagementClient # Set up the Azure Data Factory SDK client credential = DefaultAzureCredential() client = DataFactoryManagementClient(credential, "<your_subscription_id>") # Get the ID of the Function App activity run resource_group_name = "<your_resource_group_name>" factory_name = "<your_data_factory_name>" run_id = "<your_pipeline_run_id>" activity_name = "<your_function_app_activity_name>" activity_run = client.activity_runs.get(resource_group_name, factory_name, run_id, activity_name) # Make a GET request to the Get Activity Run Output API output_url = f"https://management.azure.com/{resource_group_name}/providers/Microsoft.DataFactory/factories/{factory_name}/pipelines/{run_id}/activities/{activity_name}/runs/{activity_run.id}/outputs" headers = { "Authorization": f"Bearer {credential.get_token()}" } response = requests.get(output_url, headers=headers) # Parse the JSON response to extract the output of the Function App activity function_app_output = json.loads(response.content) # Process the output of the Function App activity

The output of the Function App activity will be a JSON object. You can use the standard Python json library to parse the JSON object and extract the correct activity parameters.
Once you have extracted the correct activity parameters, you can pass them to the subsequent activities in the data factory that originally triggered the pipeline.

I hope this helps!
Shivoy Thakral 20 Reputation points

2023-09-27T10:06:41.5066667+00:00

Hi @Smaran Thoomu ,

Thank you for the response.

Just wanted to check one thing with you. So we have two ADF's - ADF1 and ADF2 (all permissions are granted to trigger runs).

ADF2 - sends an API request to trigger a pipeline run [Test] in ADF1 which has multiple child pipelines and subsequent activities.

ADF2 only has the run ID of the [Test] pipeline that was triggered in ADF1. Now using the same run id - is it possible to extract all the outputs of the subsequent child pipelines ?

One of those activities would be the function app output and we want to get the same output in ADF2 and pass on to subsequent activities.

Please let me know if this is feasible
Smaran Thoomu 15,845 Reputation points Microsoft Vendor

2023-09-28T11:01:34.4166667+00:00
Hi @Shivoy Thakral

Yes, it is possible to extract the outputs of the subsequent child pipelines in ADF1 using the same pipeline run ID that was triggered by ADF2. You can use the same DataFactoryManagementClient and RunFilterParameters objects to call the client.activity_runs.list_by_pipeline_run() method to retrieve a list of all activity runs for the specified pipeline run ID.

Here's an example code snippet in Python that demonstrates how to retrieve the outputs of all activities in a pipeline run:

from azure.identity import DefaultAzureCredential from azure.mgmt.datafactory import DataFactoryManagementClient from azure.mgmt.datafactory.models import RunFilterParameters # Set up the Azure Data Factory SDK client credential = DefaultAzureCredential() client = DataFactoryManagementClient(credential, "<your_subscription_id>") # Define the parameters for the Get Activity Run API resource_group_name = "<your_resource_group_name>" factory_name = "<your_data_factory_name>" run_id = "<your_pipeline_run_id>" # Call the List Activity Runs API to retrieve a list of all activity runs for the specified pipeline run ID filter_params = RunFilterParameters(start_time=None, end_time=None) activity_runs = client.activity_runs.list_by_pipeline_run( resource_group_name, factory_name, run_id, filter_params) # Extract the outputs of all activities in the pipeline run for activity_run in activity_runs: activity_name = activity_run.activity_name activity_output = activity_run.output # Process the activity output as needed

This code snippet assumes that you have already authenticated with Azure using the DefaultAzureCredential class, which provides a simple way to authenticate with Azure using your default credentials.

Once you have retrieved the outputs of all activities in the pipeline run, you can process the activity output of the Function App activity as needed and pass it on to subsequent activities in ADF2.

I hope this helps! Let me know if you have any further questions.
Smaran Thoomu 15,845 Reputation points Microsoft Vendor

2023-09-29T12:44:36.1466667+00:00

@Shivoy Thakral We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Sedat SALMAN 13,830 Reputation points

2023-09-25T02:07:38.4733333+00:00
To retrieve the output of a Function App activity from another Azure Data Factory using a REST API call, follow these steps:

Trigger the pipeline in the second ADF using its REST API. When you do this, you'll receive a runId in the response.

Monitor this pipeline by polling its status. Use the runId you got from the previous step to query the pipeline's status using the REST API. Repeat this until the status is either "Success" or "Failed".

Fetch the output of the Function App activity. With the same runId, query the list of activity runs. Search through the returned activities to find your Function App activity and retrieve its output.

Addiitonal Note: For authorization, ensure you set the Authorization header with a Bearer token from Azure Active Directory. If triggering from another Azure Data Factory Web activity, you can use Managed Identity for this.

By following this method, you can easily retrieve the desired output without the need for additional tools like Azure Monitor.
Please sign in to rate this answer.
Shivoy Thakral 20 Reputation points

2023-09-25T07:23:38.07+00:00

What are the parameters to send in the rest api activity using the run id to retrieve the output - please could you help with an example ?. And we have the right authorisations via managed identity

Shivoy Thakral 20 Reputation points

2023-09-25T09:29:06.87+00:00

Please could you help me out with a sample code - whether it will be a Get or a Post request with the run id to query the run activities.

Apart from that - the output will be a json with multiple activities. I want to understand how i can process the json output to extract only the correct activity parameters which can be further passed downstream.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to extract a Function App activity output through a Rest Api Call from a different azure data factory.

1 answer

Your answer