question

ohithareddy-2074 avatar image
1 Vote"
ohithareddy-2074 asked jnw-1172 commented

Avoid Data factory pipelines tmieout 7 Days Limits

I have ADF pipeline which internally executes Azure Databricks notebook using Notebook activity. But this ADF pipeline is getting timed-out after 7 Days due to databricks notebook activity maximum run for 7 Days.

Are there any other ADF features/abilities to automatically restart sample ADF pipeline when its timed-out after 7 Days?

I tried to use retry Option from Azure Databricks notebook activity itself but that always retrying even though its the ADF failed/stopped/cancelled pipeline manually due to various reasons.

I am looking for way to restart to execute pipeline when its timed-out only or avoid time out limits.

Thanks
Nani

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

KranthiPakala-MSFT avatar image
1 Vote"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hi @ohithareddy-2074,


Welcome to Microsoft Q&A and thanks for reaching out.



The max timeout for a pipeline activity runs is 7 days. It is a hard limit. But I am reaching out to internal sources to double check if there is any other way to overcome this limitation. Will keep you posted as soon as I have an update.


Ref: https://github.com/MicrosoftDocs/azure-docs/blob/master/includes/azure-data-factory-limits.md


Apologizes for delay in response.



· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @ohithareddy-2074,

Update: This is a hard limit that max timeout for activity runs is 7 days. Unfortunately there is no other way to increase it. If you have any feedback/suggestion regarding this, I would recommend you to please share your idea in ADF user voice forum: https://feedback.azure.com/forums/270578-data-factory

Once the feedback is shared, please do post the feedback link here, so that other users with similar idea can up-vote and/or comment on it.



Thanks


0 Votes 0 ·

Hi @ohithareddy-2074,

Just checking to see if the above information was helpful? If you have any further query, please do let us know.


Thanks

0 Votes 0 ·

Hi @ohithareddy-2074,


Following up to see if the above information was helpful. If you have any further query do let us know.


0 Votes 0 ·
harshghatge avatar image
0 Votes"
harshghatge answered jnw-1172 commented

Hi @ohithareddy-2074 ,

You can achieve this by using two ADF pipelines
1. Your pipeline_A which runs beyond 7 days and you which to restart after 7 days.
2. New pipeline_B which triggers your pipeline_A. Pipeline_B would just contain simple one "Execute Pipeline" activity to execute pipeline_A.

Now in your pipeline_A you add one step after your activity which times out after 7 days on "Failure" like below. This will trigger pipeline_B after timeout and pipelune_B will again trigger pipeline_A, since it just has one activity to execute pipeline_A.

183171-image.png



image.png (10.4 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

But after setting this configuration, I met error "Error while publishing: Circular dependencies found, please review the following dependency path: Execute Pipeline => pipeline3 => Execute Pipeline" when publishing.
Had you met this error? How to solve?

0 Votes 0 ·