ADF activity error capture

Question

ADF activity error capture

arkiboys 9,706

hello,

In Adf,

1- what is the best way to capture errors?

2- I remember there was an over alert mechanism and it would send an automated email. IS there such option or similar?

thanks

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2023-10-02T10:46:21.9733333+00:00
@arkiboys

Hi Thanks for Approaching Microsoft Q&A

The best way to capture errors in Azure Data Factory (ADF) is to use error handling activities. An error handling activity is defined for the "Upon Failure" path, and will be invoked if the main activity fails. It should be incorporated as best practice for all mission-critical steps that need fallback alternatives or logging Here are some common patterns following a single activity:

Error handling: An error handling activity is defined for the "Upon Failure" path, and will be invoked if the main activity fails. It should be incorporated as best practice for all mission-critical steps that need fallback alternatives or logging.

Best effort steps: Connect both UponFailure and UponSkip paths from the last activity to the error handling activity. The last step, Generic Error Handling, will only run if any of the previous activities fail. It will not run if they all succeed. You can add multiple activities for error handling.

Yes, there is an option to send an automated email when an error occurs in ADF. One way to achieve this is by using Azure Monitor for logging and enabling Diagnostic Settings on your Data Factory resource Another way is to capture and persist ADF pipeline errors to an Azure SQL Database table You can also implement a process to track the log activity for ADF pipelines that run and persist the data Finally, you can use log analytics to view verbose logs, which should be enough for anything that you might need
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2023-10-03T10:07:38.96+00:00

@arkiboys We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Accepted answer

0 additional answers

Your answer

phemanth 15,755 Reputation points Microsoft External Staff Moderator

2023-10-02T10:46:21.9733333+00:00

@arkiboys

Hi Thanks for Approaching Microsoft Q&A

The best way to capture errors in Azure Data Factory (ADF) is to use error handling activities. An error handling activity is defined for the "Upon Failure" path, and will be invoked if the main activity fails. It should be incorporated as best practice for all mission-critical steps that need fallback alternatives or logging Here are some common patterns following a single activity:

Error handling: An error handling activity is defined for the "Upon Failure" path, and will be invoked if the main activity fails. It should be incorporated as best practice for all mission-critical steps that need fallback alternatives or logging.

Best effort steps: Connect both UponFailure and UponSkip paths from the last activity to the error handling activity. The last step, Generic Error Handling, will only run if any of the previous activities fail. It will not run if they all succeed. You can add multiple activities for error handling.

Yes, there is an option to send an automated email when an error occurs in ADF. One way to achieve this is by using Azure Monitor for logging and enabling Diagnostic Settings on your Data Factory resource Another way is to capture and persist ADF pipeline errors to an Azure SQL Database table You can also implement a process to track the log activity for ADF pipelines that run and persist the data Finally, you can use log analytics to view verbose logs, which should be enough for anything that you might need
phemanth 15,755 Reputation points Microsoft External Staff Moderator

2023-10-03T10:07:38.96+00:00

@arkiboys We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Hi arkiboys

Reg question #1 - If you have multiple activities in a pipeline, say PL1, and you want to capture errors in each of the activities when they fail, you can use a master pipeline, say PL_Main and call PL1 from PL_Main and then use a stored proc to log errors from PL1 or you can use copy activity to log errors.

SO thread that discusses this and other alternatives-

https://stackoverflow.com/questions/73537220/azure-data-factory-how-to-catch-any-error-on-any-activity-and-log-it-into-databa

Here is.a link to implement a stored proc for error logging.

Reg question #2 I believe you are asking about custom alerts., Please take a look at this video or go through this.

Hope it helps. Kindly let us know if you have further questions.

Share via

ADF activity error capture

0 additional answers

Your answer