Capture Pipeline error messages

Question

Hello,
My requirement is to capture the error message of pipeline execution in Data factory and store it into a table. Currently I'm able to do it activity wise only ('@{activity('Copy data').output.errors[0].Message}'), which captures the message for the specific component. Since there are a number of other activities present, it would not be a best thing to replicate the same thing for every activity within a same pipeline and also it would give a shabby appearance.
So I would like to know if there is a better to do this so that whenever a pipeline fails, I can get a consolidated error message for the complete pipeline instead for every single component ?

Thanks,

Answer 1

Hey,
The below blog should help:
https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activities-in-azure-data-factory/

wherein we can have a single activity capture all details for all activities

Answer 2

Hello @Nandan Hegde ,
The solution that you gave does work well, but it still requires every component to have a connection. In the Monitor tab we do have the pipeline run history along with the error description. Is there any way to fetch the info from there and store it into a DB?

Thanks,