How to delete ADF Pipeline run history or configure to be deleted after X days ?

Question

Hello,

Is there a way to configure ADF Pipeline run history to delete after X days ( say 10 days) ? I'm not able to delete this from UI portal / ADF Studio. How can we delete pipeline run history in ADF ?

Thanks,

Balaji

Answer 1

In ADF there isn’t a built-in UI option to delete pipeline run history directly or configure it to automatically delete after a set number of days. However, you can manage this using a custom solution with Azure Logic Apps, Azure Functions, or an Azure Automation Runbook. Here’s how you can approach it:

Option 1: Use an Azure Logic App or Azure Function

1. Create an Azure Logic App or Function App that will periodically run a script to delete pipeline run history based on a retention period (for example 10 days).
2. Use the Azure Data Factory REST API to query pipeline runs and filter by LastUpdatedAfter and LastUpdatedBefore parameters to get runs older than the specified date range.
3. Delete each pipeline run by calling the CancelPipelineRun API if needed, or simply ignore it if it’s not affecting performance, as old runs don’t impact new executions.

To list runs older than 10 days:

GET https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/pipelineruns?api-version=2018-06-01&$filter=LastUpdatedBefore='{Date10DaysAgo}'

Option 2: Azure Automation Runbook

1. Create an Automation Runbook in Azure Automation and schedule it to run daily or weekly.
2. Use PowerShell in the runbook to interact with the Data Factory REST API.
3. Implement logic to filter pipeline runs based on date and remove those beyond the retention period.

$resourceGroupName = "yourResourceGroup"

$factoryName = "yourDataFactory"

$subscriptionId = "yourSubscriptionId"

$cutoffDate = (Get-Date).AddDays(-10).ToString("yyyy-MM-ddTHH:mm:ssZ")

# Authenticate using your Azure credentials

# Get pipeline runs older than cutoffDate

$pipelineRuns = Invoke-RestMethod -Uri "https://management.azure.com/subscriptions/$subscriptionId/resourceGroups/$resourceGroupName/providers/Microsoft.DataFactory/factories/$factoryName/pipelineruns?api-version=2018-06-01&$filter=LastUpdatedBefore='$cutoffDate'" -Method Get -Headers @{Authorization = "Bearer $token"}

# Loop through and delete or ignore old runs

foreach ($run in $pipelineRuns.value) {

    # Logic to delete or ignore runs

}

Answer 2

Hi @Balaji Dabbara

Welcome to Microsoft Q&A platform and thanks for posting your query here.

Currently, Azure Data Factory (ADF) does not provide a direct way to manually delete or configure the retention of pipeline run history through the UI or ADF Studio. However, there is a workaround you can consider managing pipeline run history:

Retention Settings for Diagnostics Logs in Azure Monitor:

• ADF pipeline run history logs are typically stored in Azure Monitor Logs (Log Analytics) if you have configured diagnostic settings. You can set a retention policy for your logs in Azure Monitor.
• To configure retention:
  1. Go to Azure Portal and navigate to your Log Analytics workspace.
  2. Select Usage and estimated costs under Settings.
  3. In the Data retention section, you can configure the number of days data should be retained.

For more information, please refer this: https://learn.microsoft.com/en-us/azure/devops/pipelines/policies/retention?view=azure-devops&tabs=yaml

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.