Get pipeline run status using web activity from different ADF instance.

Question

We have two ADF instances named as 1 and 2 .And two pipeline A and B in each ADF instance respectively. When the execution of pipeline A starts in ADF 1 ,we are calling pipeline B, which is present in ADF 2, at very first using WEB activity inside pipeline A and we are succeeding into it.

But after completion of some activities of pipeline B we have to come back to pipeline A of ADF1 from pipeline B of ADF2 to check whether pipeline A completed or not. If it is completed, then proceed with further activities of pipeline B, else wait until pipeline A completes and then proceed ahead.

I am using GET API of Web activity inside until activity .

https://management.azure.com/subscriptions/<YOUR SUBSCRIPTION>/resourceGroups/<RESOURCE GROUP>/providers/Microsoft.DataFactory/factories/<FACTORYNAME>/debugpipelineruns/@{pipeline().RunId}?api-version=2018-06-01

When I am passing RunId manually to this above expression . It is giving me result.

But I want to pass that RunId dynamically. So it goes to pipeline A of ADF1 and take latest RunID of pipeline A.

I have created one parameter in pipeline A which is holding the pipeline RunId , but not sure how to pass that in this above expression , which I am writing in different ADF instance ADF2

Answer 1

Hi @Gulhasan.Siddiquee ,

Thanks for clarifying the requirement. I have responded to other thread you have mentioned and also sharing the same solution here as it might help others reading this thread.

To get the pipeline runID and pipeline run status, instead of using Pipeline Runs - Get (which requires a pipeline RunID) you can use Pipeline Runs - Query By Factory (Query pipeline runs in the factory based on input filter conditions.).

Below is the required configuration in your web activity. Please note that this method to be used is POST and you will have to pass the filter conditions in the body section of the web activity as below. lastUpdatedAfter and lastUpdatedBefore are required fields to filter the run history (you can configure according to your requirement) . In the below example I have passed the lastUpdatedAfter and lastUpdatedBefore values dynamically. For testing you can start with hardcoded values and once the testing is completed you can configure with dynamic expression as I did. (NOTE: The lastUpdatedAfter and lastUpdatedBefore values should be in UTC format since the backend logs being stored in UTC format)

In the below sample I used Authorization by passing the bearer token and authentication type as none. If you have different authentication types, you may configure accordingly.

Request URL:

POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/queryPipelineRuns?api-version=2018-06-01

sample request Body looks like below (the below will return all pipeline runs between provided lastUpdatetime):

{
  "lastUpdatedAfter": "2023-03-16T23:59:00.4384332Z",
  "lastUpdatedBefore": "2023-03-18T00:36:44.3345758Z",
  "filters": [
    {
      "operand": "PipelineName",
      "operator": "Equals",
      "values": [
        "pl_scriptActivity"
      ]
    }
  ]
}

The response from API would look like below:

{
    "value": [
        {
            "id": "/SUBSCRIPTIONS/Bxxxxxx-Cxxx-xxxxxx-xxxx-xxx/RESOURCEGROUPS/Sample-RG/PROVIDERS/MICROSOFT.DATAFACTORY/FACTORIES/KRANTHIADF/pipelineruns/212245c4-bb53-494d-85e0-15728868ed12",
            "runId": "212245c4-bb53-494d-85e0-15728868ed12",
            "debugRunId": null,
            "runGroupId": "212245c4-bb53-494d-85e0-15728868ed12",
            "pipelineName": "pl_scriptActivity",
            "parameters": {},
            "invokedBy": {
                "id": "cd0e5444864e45eb8fe0f221a8296831",
                "name": "Manual",
                "invokedByType": "Manual"
            },
            "runStart": "2023-03-16T23:59:12.4384332Z",
            "runEnd": "2023-03-16T23:59:18.7911191Z",
            "durationInMs": 6352,
            "status": "Succeeded",
            "message": "",
            "pipelineReturnValue": {},
            "lastUpdated": "2023-03-16T23:59:18.7915386Z",
            "annotations": [],
            "runDimension": {},
            "isLatest": true
        }
    ],
    "ADFWebActivityResponseHeaders": {
        "Pragma": "no-cache",
        "x-ms-correlation-request-id": "xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx",
        "x-ms-ratelimit-remaining-subscription-reads": "11999",
        "x-ms-request-id": "xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx",
        "x-ms-routing-request-id": "WESTUS2:20230317T014422Z:xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx",
        "Strict-Transport-Security": "max-age=31536000; includeSubDomains",
        "X-Content-Type-Options": "nosniff",
        "Cache-Control": "no-cache",
        "Date": "Fri, 17 Mar 2023 01:44:22 GMT",
        "Server": "Kestrel",
        "Content-Length": "921",
        "Content-Type": "application/json; charset=utf-8",
        "Expires": "-1"
    },
    "effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime (West US 2)",
    "executionDuration": 0,
    "durationInQueue": {
        "integrationRuntimeQueue": 0
    },
    "billingReference": {
        "activityType": "ExternalActivity",
        "billableDuration": [
            {
                "meterType": "AzureIR",
                "duration": 0.016666666666666666,
                "unit": "Hours"
            }
        ]
    }
}

In case if you multiple runID's returned, you can either try passing the closest recent run time of your querying pipeline or you can get add orderBy parameter in your request using RunStart and order = DESC

One you have the Web activity response, then you can extract the latest or most recent pipelinerunID and its status by using below dynamic expression.

For retrieving pipeline run ID:

@activity('web_GetPipelineRunDetails').output.value[0].runId

For retrieving pipeline run status:

@activity('web_GetPipelineRunDetails').output.value[0].status

Hope this info helps. You can also refer to below additional useful information.

1. Get Azure Data Factory logs
2. How to get meta data of the pipeline in DataFlow of Azure data factory? Want to create a debug pipeline
3. Related doc: https://learn.microsoft.com/en-us/rest/api/datafactory/pipeline-runs/query-by-factory?tabs=HTTP#runqueryorder

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.