Avoid Data factory pipelines tmieout 7 Days Limits

Question

I have ADF pipeline which internally executes Azure Databricks notebook using Notebook activity. But this ADF pipeline is getting timed-out after 7 Days due to databricks notebook activity maximum run for 7 Days.

Are there any other ADF features/abilities to automatically restart sample ADF pipeline when its timed-out after 7 Days?

I tried to use retry Option from Azure Databricks notebook activity itself but that always retrying even though its the ADF failed/stopped/cancelled pipeline manually due to various reasons.

I am looking for way to restart to execute pipeline when its timed-out only or avoid time out limits.

Thanks
Nani

Answer 1

Hi @ohithareddy-2074,

Welcome to Microsoft Q&A and thanks for reaching out.

The max timeout for a pipeline activity runs is 7 days. It is a hard limit. But I am reaching out to internal sources to double check if there is any other way to overcome this limitation. Will keep you posted as soon as I have an update.

Ref: https://github.com/MicrosoftDocs/azure-docs/blob/master/includes/azure-data-factory-limits.md

Apologizes for delay in response.

Answer 2

Hi @OBULA REDDY SANA ,

You can achieve this by using two ADF pipelines

1. Your pipeline_A which runs beyond 7 days and you which to restart after 7 days.
2. New pipeline_B which triggers your pipeline_A. Pipeline_B would just contain simple one "Execute Pipeline" activity to execute pipeline_A.

Now in your pipeline_A you add one step after your activity which times out after 7 days on "Failure" like below. This will trigger pipeline_B after timeout and pipelune_B will again trigger pipeline_A, since it just has one activity to execute pipeline_A.