Avoid Data factory pipelines tmieout 7 Days Limits

OBULA REDDY SANA 6 Reputation points
2020-06-27T23:22:14.687+00:00

I have ADF pipeline which internally executes Azure Databricks notebook using Notebook activity. But this ADF pipeline is getting timed-out after 7 Days due to databricks notebook activity maximum run for 7 Days.

Are there any other ADF features/abilities to automatically restart sample ADF pipeline when its timed-out after 7 Days?

I tried to use retry Option from Azure Databricks notebook activity itself but that always retrying even though its the ADF failed/stopped/cancelled pipeline manually due to various reasons.

I am looking for way to restart to execute pipeline when its timed-out only or avoid time out limits.

Thanks
Nani

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,093 questions
0 comments No comments
{count} vote

2 answers

Sort by: Most helpful
  1. KranthiPakala-MSFT 46,437 Reputation points Microsoft Employee
    2020-06-30T02:51:24.973+00:00

    Hi @ohithareddy-2074,

    Welcome to Microsoft Q&A and thanks for reaching out.

    The max timeout for a pipeline activity runs is 7 days. It is a hard limit. But I am reaching out to internal sources to double check if there is any other way to overcome this limitation. Will keep you posted as soon as I have an update.

    Ref: https://github.com/MicrosoftDocs/azure-docs/blob/master/includes/azure-data-factory-limits.md

    Apologizes for delay in response.

    1 person found this answer helpful.

  2. Harsh Ghatge 1 Reputation point
    2022-03-15T05:23:38.503+00:00

    Hi @OBULA REDDY SANA ,

    You can achieve this by using two ADF pipelines

    1. Your pipeline_A which runs beyond 7 days and you which to restart after 7 days.
    2. New pipeline_B which triggers your pipeline_A. Pipeline_B would just contain simple one "Execute Pipeline" activity to execute pipeline_A.

    Now in your pipeline_A you add one step after your activity which times out after 7 days on "Failure" like below. This will trigger pipeline_B after timeout and pipelune_B will again trigger pipeline_A, since it just has one activity to execute pipeline_A.

    183171-image.png