Error Rerunning Failed Data Factory Pipeline from Failed Activity with a Secure Output Activity

Question

Hi all,

I have a pipeline in Azure Data Factory that is run each day, which is made of several execute pipeline activities. Some of these pipelines have web activities in them which use secure output.

The error I am facing is that on occasion, one of the pipelines in the daily pipeline fails due to a server error, which can be fixed by rerunning the activity. My intention would be to rerun the daily pipeline from the failed activity. When this occurs, the pipeline runs from start to finish but skips over all activities that were previously run.

An error occurs when it tries to skip over the execute pipeline activity that contains a secure output web activity in it, which has already run. I get the following error:

Error encountered: Failed with message Activity 'ba63dcb2-1343-5642-a7b9-6753b1b2b5423' references the SecureOutput from previous activity 'Get app id'. Since ADF does not cache secure output, please rerun from activity 'Get app id' or rerun the whole pipeline run instead

Is there any way to fix this so that I can rerun from the failed activity and successfully skipping over the run pipeline with a secure output?

Thanks for any help

Answer 1

Hello Connor Welham,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you are having "Error Rerunning Failed Data Factory Pipeline from Failed Activity with a Secure Output Activity" and according to your question.

There are ways to fix it, and you can rerun from the failed activity and successfully skipping over the run pipeline with a secure output in Azure Data Factory.

1. Since Azure Data Factory does not cache secure output, you need to rerun the pipeline from the activity that generates the secure output. In your case, rerun from the 'Get app id' activity. This ensures that the secure output is regenerated and available for subsequent activities.
   1. Determine which activity in the pipeline failed.
   2. Modify the pipeline to allow reruns from the point of failure. This might involve using a control flow activity like If Condition or Switch to check the status of the previous run.
   3. Make sure that the rerun logic includes steps to regenerate any secure outputs that were not produced due to the failure.
2. Implement checkpoints in your pipeline to save the state of the pipeline at various stages. This way, you can restart the pipeline from a specific checkpoint rather than from the beginning. But you need to plan carefully.
3. You can also use the 'On Failure' path in your activities to trigger specific actions, such as sending notifications or retrying the failed activity by implement error handling.
4. Then, if you can configure retry logic for activities that are prone to transient errors. This can help mitigate issues caused by temporary server errors without requiring a full pipeline rerun.

Check out the links for more steps: https://learn.microsoft.com/en-us/azure/data-factory/tutorial-pipeline-return-value and https://azure.microsoft.com/en-us/blog/rerun-activities-inside-your-data-factory-pipelines

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.

Answer 2

Hello @Connor Welham

It seems that the error you are facing is due to the fact that Azure Data Factory does not cache secure output, which is causing the pipeline to skip over the execute pipeline activity that contains a secure output web activity in it, which has already run.

To fix this issue, you can try rerunning the pipeline from the activity that produced the secure output, which in your case is the Get app id activity.

Alternatively, you can rerun the entire pipeline run instead of just the failed activity.

If you want to rerun the pipeline from the failed activity, you can try modifying your pipeline to store the secure output in a persistent location, such as Azure Blob Storage or Azure Data Lake Storage, instead of using the secure output feature.

This will allow you to rerun the pipeline from the failed activity without encountering the error you are facing.

Another option is to use the activity-level retry feature in Azure Data Factory. You can set the retry count on the execute pipeline activity that contains the secure output web activity. During the pipeline execution, if this activity run fails, the next automatic retry will start from the last trial's failure point.

I hope this helps

Answer 3

@Connor Welham

Welcome to the Microsoft Q&A and thank you for posting your questions here.

It does seem like a limitation in Azure Data Factory’s handling of secure outputs. Here are a few potential workarounds you might consider:

Conditional Execution:

Use the If Condition activity to check if the secure output is needed. If not, you can skip the activity. This way, when you rerun the pipeline, it will conditionally skip over the secure output activity if it has already been executed.

Custom Status Tracking:

Implement a custom status tracking mechanism using Azure Table Storage or SQL Database. You can log the status of each activity and check this status before executing an activity. If an activity has already been completed, you can skip it based on the logged status.

Separate Pipelines:

Break down your pipeline into smaller, more granular pipelines. This way, you can rerun only the specific pipeline that failed without affecting the others. Each smaller pipeline can handle its own secure outputs independently.

Error Handling Logic:

Enhance your error handling logic to better manage retries and skips. For example, you can use the Until activity to retry a specific activity until it succeeds, or use the Switch activity to handle different error scenarios more gracefully.

Manual Intervention:

As a last resort, you might need to manually intervene and rerun specific activities through the Azure Data Factory UI. This can be done by navigating to the Monitor section, selecting the failed pipeline run, and rerunning the specific activity.

These workarounds can help you manage the secure output issue more effectively and avoid unnecessary reruns of the entire pipeline.

Hope this helps. Do let us know if you any further queries.