How to pass run Id dynamically, while using web activity with GET Method

Question

I want to know the run status of a particular pipeline in ADF from different pipeline in same ADF instance.

I am using web activity with GET method, but issue with it is that everytime I have to put RunID of that pipeline manually in below URL. I am not able pass it dynamically .

GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns/{runId}?api-version=2018-06-01

Answer 1

Hi @Gulhasan.Siddiquee ,

Thanks for using Microsoft Q&A forum and posting your query.

To get the pipeline runID and pipeline run status, instead of using Pipeline Runs - Get (which requires a pipeline RunID) you can use Pipeline Runs - Query By Factory (Query pipeline runs in the factory based on input filter conditions.).

Below is the required configuration in your web activity. Please note that this method to be used is POST and you will have to pass the filter conditions in the body section of the web activity as below. lastUpdatedAfter and lastUpdatedBefore are required fields to filter the run history (you can configure according to your requirement) . In the below example I have passed the lastUpdatedAfter and lastUpdatedBefore values dynamically. For testing you can start with hardcoded values and once the testing is completed you can configure with dynamic expression as I did. (NOTE: The lastUpdatedAfter and lastUpdatedBefore values should be in UTC format since the backend logs being stored in UTC format)

In the below sample I used Authorization by passing the bearer token and authentication type as none. If you have different authentication types, you may configure accordingly.

Request URL:

POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/queryPipelineRuns?api-version=2018-06-01

sample request Body looks like below (the below will return all pipeline runs between provided lastUpdatetime):

{
  "lastUpdatedAfter": "2023-03-16T23:59:00.4384332Z",
  "lastUpdatedBefore": "2023-03-18T00:36:44.3345758Z",
  "filters": [
    {
      "operand": "PipelineName",
      "operator": "Equals",
      "values": [
        "pl_scriptActivity"
      ]
    }
  ]
}

The response from API would look like below:

{
    "value": [
        {
            "id": "/SUBSCRIPTIONS/Bxxxxxx-Cxxx-xxxxxx-xxxx-xxx/RESOURCEGROUPS/Sample-RG/PROVIDERS/MICROSOFT.DATAFACTORY/FACTORIES/KRANTHIADF/pipelineruns/212245c4-bb53-494d-85e0-15728868ed12",
            "runId": "212245c4-bb53-494d-85e0-15728868ed12",
            "debugRunId": null,
            "runGroupId": "212245c4-bb53-494d-85e0-15728868ed12",
            "pipelineName": "pl_scriptActivity",
            "parameters": {},
            "invokedBy": {
                "id": "cd0e5444864e45eb8fe0f221a8296831",
                "name": "Manual",
                "invokedByType": "Manual"
            },
            "runStart": "2023-03-16T23:59:12.4384332Z",
            "runEnd": "2023-03-16T23:59:18.7911191Z",
            "durationInMs": 6352,
            "status": "Succeeded",
            "message": "",
            "pipelineReturnValue": {},
            "lastUpdated": "2023-03-16T23:59:18.7915386Z",
            "annotations": [],
            "runDimension": {},
            "isLatest": true
        }
    ],
    "ADFWebActivityResponseHeaders": {
        "Pragma": "no-cache",
        "x-ms-correlation-request-id": "xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx",
        "x-ms-ratelimit-remaining-subscription-reads": "11999",
        "x-ms-request-id": "xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx",
        "x-ms-routing-request-id": "WESTUS2:20230317T014422Z:xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx",
        "Strict-Transport-Security": "max-age=31536000; includeSubDomains",
        "X-Content-Type-Options": "nosniff",
        "Cache-Control": "no-cache",
        "Date": "Fri, 17 Mar 2023 01:44:22 GMT",
        "Server": "Kestrel",
        "Content-Length": "921",
        "Content-Type": "application/json; charset=utf-8",
        "Expires": "-1"
    },
    "effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime (West US 2)",
    "executionDuration": 0,
    "durationInQueue": {
        "integrationRuntimeQueue": 0
    },
    "billingReference": {
        "activityType": "ExternalActivity",
        "billableDuration": [
            {
                "meterType": "AzureIR",
                "duration": 0.016666666666666666,
                "unit": "Hours"
            }
        ]
    }
}

In case if you multiple runID's returned, you can either try passing the closest recent run time of your querying pipeline or you can get add orderBy parameter in your request using RunStart and order = DESC

One you have the Web activity response, then you can extract the latest or most recent pipelinerunID and its status by using below dynamic expression.

For retrieving pipeline run ID:

@activity('web_GetPipelineRunDetails').output.value[0].runId

For retrieving pipeline run status:

@activity('web_GetPipelineRunDetails').output.value[0].status

Hope this info helps. You can also refer to below additional useful information.

1. Get Azure Data Factory logs
2. How to get meta data of the pipeline in DataFlow of Azure data factory? Want to create a debug pipeline
3. Related doc: https://learn.microsoft.com/en-us/rest/api/datafactory/pipeline-runs/query-by-factory?tabs=HTTP#runqueryorder

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

I noticed Bearer token is used here for authorization. I know manually we can generate it and it is valid only for an hour. Please let me know if there is a way to either generate Bearer token dynamically or if we can generate a bearer token with a longer validity period.