ADF - Capture the error message when you manually cancel the ADF pipeline run.

Question

I am trying to build an error logging mechanism in Azure Data Factory to capture all the errors into a SQL table.

This works fine if a pipeline activity failed itself, I have a stored procedure connecting to the activity (when failed) to write the ADF error message <@activity('activity').error.message> for a specific runid <@pipeline().RunId> into a table.

Another scenario is that the pipeline could be manually canceled. I was wondering if there is a way to get and log the 'cancel' status into a database table? Because a job can be canceled at any stage, there is no way I can put an error catching in place.

Many thanks for your help, any hints are most welcome.

Answer 1

Hello @Finn and welcome to Microsoft Q&A.

If my understanding is correct, you want to capture that a pipeline was cancelled, (not some other error that happens in a run that gets cancelled).

You do raise a good point, that the act of cancelling a pipeline run precludes the ability to log that cancel via a pipeline activity.

So I have thought up another way to monitor the pipeline.

In the pipeline, make the first aactivity be a stored procedure which writes to your SQL the pipeline run ID ( @pipeline().RunId ). Then have your SQL server on a timed delay, query the Data Factory REST API to get the pipeline run's status by pipeline run ID. This should get you whether the pipeline is running, succeeded, cancelled, or failed.

This way, as long as the initial activity succeeds in sending the run id to your SQL, you should always be able to track it.