How to delete ADF Pipeline run history or configure to be deleted after X days ?

Question

Hello,

Is there a way to configure ADF Pipeline run history to delete after X days ( say 10 days) ? I'm not able to delete this from UI portal / ADF Studio. How can we delete pipeline run history in ADF ?

Thanks,

Balaji

Answer 1

In ADF there isn’t a built-in UI option to delete pipeline run history directly or configure it to automatically delete after a set number of days. However, you can manage this using a custom solution with Azure Logic Apps, Azure Functions, or an Azure Automation Runbook. Here’s how you can approach it:

Option 1: Use an Azure Logic App or Azure Function

1. Create an Azure Logic App or Function App that will periodically run a script to delete pipeline run history based on a retention period (for example 10 days).
2. Use the Azure Data Factory REST API to query pipeline runs and filter by LastUpdatedAfter and LastUpdatedBefore parameters to get runs older than the specified date range.
3. Delete each pipeline run by calling the CancelPipelineRun API if needed, or simply ignore it if it’s not affecting performance, as old runs don’t impact new executions.

To list runs older than 10 days:

GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns?api-version=2018-06-01&$filter=LastUpdatedBefore='{Date10DaysAgo}'

Option 2: Azure Automation Runbook

1. Create an Automation Runbook in Azure Automation and schedule it to run daily or weekly.
2. Use PowerShell in the runbook to interact with the Data Factory REST API.
3. Implement logic to filter pipeline runs based on date and remove those beyond the retention period.

$resourceGroupName = "yourResourceGroup"

$factoryName = "yourDataFactory"

$subscriptionId = "yourSubscriptionId"

$cutoffDate = (Get-Date).AddDays(-10).ToString("yyyy-MM-ddTHH:mm:ssZ")

# Authenticate using your Azure credentials

# Get pipeline runs older than cutoffDate

$pipelineRuns = Invoke-RestMethod -Uri "https://management.azure.com/subscriptions/$subscriptionId/resourceGroups/$resourceGroupName/providers/Microsoft.DataFactory/factories/$factoryName/pipelineruns?api-version=2018-06-01&$filter=LastUpdatedBefore='$cutoffDate'" -Method Get -Headers @{Authorization = "Bearer $token"}

# Loop through and delete or ignore old runs

foreach ($run in $pipelineRuns.value) {

    # Logic to delete or ignore runs

}