How to retrieve both debug and triggered pipeline run details for ADF using a single REST API call using ADF web activity

Question

How to retrieve both debug and triggered pipeline run details for ADF using a single REST API call using ADF web activity

Kumar 20

How can I obtain the latest run date of a pipeline (both Triggered and Debug run details) utilizing just the pipeline name?

The first URL retrieves the triggered pipeline details with the ADF web activity, and the second URL retrieves the debug pipeline details. However, I want to combine both the triggered and debug pipeline runs' output. Please let me know if this is possible and how to accomplish it if it is feasible.

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-06-11T08:13:01.19+00:00
@Cherukuri, Praveen Kumar

Thanks for the question and using MS Q&A platform.

You are absolutely right, relying solely on the pipeline name won't differentiate between triggered and debug runs. However, there's a way to combine the results and identify the latest run date considering both types. Here's how you can achieve this:

Make Separate API Calls:

Use the provided URLs to make separate API calls to both queryPipelineRuns and queryDebugPipelineRuns endpoints. Replace placeholders like {subscriptionId}, {resourceGroupName}, and {factoryName} with your actual values.

Parse the Responses:

Both API responses will be in JSON format. Parse them to extract relevant information about each pipeline run, including:

runStart (timestamp of the run start)

pipelineName (name of the pipeline)

Combine and Sort:

Combine the extracted data from both responses into a single list. Since each entry represents a single pipeline run (triggered or debug), sort the list based on the runStart timestamp in descending order (latest first).

Identify Latest Run Date:

The first element in the sorted list will now represent the latest run, regardless of whether it's a triggered or debug run. Access the runStart value of this element to obtain the latest run date.

You can leverage libraries in your chosen programming language (Python, PowerShell, etc.) to handle API calls, JSON parsing, and sorting.

If you only need the most recent successful run, you might need to filter the combined list based on the run status before sorting.

Hope this helps. Do let us know if you any further queries.
Kumar 20 Reputation points

2024-06-12T04:08:43.5166667+00:00

Can you please provide more details on how to combine both JSON responses from web activity in the ADF?
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-06-14T09:02:56.67+00:00
@Cherukuri, Praveen Kumar Absolutely! Here's how you can combine the JSON responses from both Web Activities in your ADF pipeline:

1. Define Variables:

Create two variables in your ADF pipeline to store the outputs of the Web Activities that call the triggered and debug run APIs. You can name them something like triggeredPipelineRuns and debugPipelineRuns.

2. Accessing Web Activity Outputs:

Within your next activity (likely a Data Flow or Script activity), use the @pipeline().variables['triggeredPipelineRuns'] expression to access the content of the triggeredPipelineRuns variable. This will contain the parsed JSON response from the triggered run Web Activity.

Similarly, use @pipeline().variables['debugPipelineRuns'] to access the content of the debugPipelineRuns variable, containing the parsed JSON response from the debug run Web Activity.

3. Combining the Data (using Python Script example):

Here's an example using a Python Script activity:

# Import libraries (assuming you have a Script activity with Python runtime) import json # Access Web Activity outputs triggered_runs = json.loads(pipeline().variables['triggeredPipelineRuns']) debug_runs = json.loads(pipeline().variables['debugPipelineRuns']) # Combine data into a single list all_runs = triggered_runs + debug_runs # Sort based on runStart (descending order) all_runs.sort(key=lambda run: run['runStart'], reverse=True) # Access the latest run details (assuming 'runStart' and 'pipelineName' are keys) latest_run = all_runs[0] latest_run_date = latest_run['runStart'] latest_pipeline_name = latest_run['pipelineName'] # You can now use these variables for further processing

Explanation

We import the json library to handle JSON data.

We access the Web Activity outputs using the expressions mentioned earlier.

We combine both lists (triggered_runs and debug_runs) into a single list named all_runs.

We sort all_runs based on the runStart key in descending order to get the latest run first.

Finally, we access the details of the latest run (timestamp and pipeline name) from the first element of the sorted list.

4. Script Activity Output (Optional):

You can define additional outputs for your Script activity to store the processed data (e.g., latestRunDate and latestPipelineName). This allows you to use this information in subsequent activities in your pipeline.

Remember:

Adapt the script code to match the specific structure of your JSON responses and the desired output format.

Ensure you have the necessary libraries installed in your Script activity's runtime environment (e.g., install the json library for Python).
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-06-18T07:26:52.6033333+00:00

@Kumar We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

1 answer

Your answer

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-06-11T08:13:01.19+00:00

@Cherukuri, Praveen Kumar

Thanks for the question and using MS Q&A platform.

You are absolutely right, relying solely on the pipeline name won't differentiate between triggered and debug runs. However, there's a way to combine the results and identify the latest run date considering both types. Here's how you can achieve this:

Make Separate API Calls:

Use the provided URLs to make separate API calls to both queryPipelineRuns and queryDebugPipelineRuns endpoints. Replace placeholders like {subscriptionId}, {resourceGroupName}, and {factoryName} with your actual values.

Parse the Responses:

Both API responses will be in JSON format. Parse them to extract relevant information about each pipeline run, including:

runStart (timestamp of the run start)

pipelineName (name of the pipeline)

Combine and Sort:

Combine the extracted data from both responses into a single list. Since each entry represents a single pipeline run (triggered or debug), sort the list based on the runStart timestamp in descending order (latest first).

Identify Latest Run Date:

The first element in the sorted list will now represent the latest run, regardless of whether it's a triggered or debug run. Access the runStart value of this element to obtain the latest run date.

You can leverage libraries in your chosen programming language (Python, PowerShell, etc.) to handle API calls, JSON parsing, and sorting.

If you only need the most recent successful run, you might need to filter the combined list based on the run status before sorting.

Hope this helps. Do let us know if you any further queries.
Kumar 20 Reputation points

2024-06-12T04:08:43.5166667+00:00

Can you please provide more details on how to combine both JSON responses from web activity in the ADF?
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2024-06-18T07:26:52.6033333+00:00

@Kumar We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

To retrieve both debug and triggered pipeline run details for an Azure Data Factory (ADF) pipeline using a single REST API call is not directly supported. However, you can achieve this by using an ADF web activity to make two separate API calls and then merge the results within your data factory workflow.

Use the following URL in your web activity to retrieve triggered pipeline run details.


https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/queryPipelineRuns?api-version=2018-06-01

Use the following URL in another web activity to retrieve debug pipeline run details.


https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/queryDebugPipelineRuns?api-version=2018-06-01

After each web activity, use a Set Variable activity to store the results of the triggered and debug pipeline runs separately.

You can use an Azure Function or a custom activity to merge the results from the two variables. The function will take the two sets of results and combine them into a single output.

Kumar 0 Reputation points

2024-06-12T04:32:03.5033333+00:00

Can you please provide more details on how I can achieve combining the result using Azure Functions?

Share via

How to retrieve both debug and triggered pipeline run details for ADF using a single REST API call using ADF web activity

1 answer

Your answer