Share via

Azure Monitor detailed information on failure of azure data factory pipeline

Philip O'Rourke 111 Reputation points
2022-04-28T10:33:05.877+00:00

Hi,

I have a few questions regarding azure alerts and monitoring with regards data factory.

1) Have set up alert in azure data factory whenever activity on any pipeline fails, we get an alert but when hit the resource it just takes us to the azure data factory resource.

Is it possible for the link in the e-mail which takes you to the actual pipeline which caused the failure?

At moment we have 2 options we could go direct to monitor in azure data factory to see this or add in call to logicapps for info on pipeline which failed but then would need to make call to such in every pipeline if used this feature.

Looked at https://learn.microsoft.com/en-us/azure/data-factory/how-to-send-email

However, when put below as per the link above and call the pipeline containing the web activity (e.g. send_email_error) the e-mail you get shows the name of the pipleine as send_email_error rather than the pipeline which called this in first place as result of failure (e.g. calling_pipeline).

"pipelineName" : "@{pipeline().Pipeline}",

What is the best way to have the e-mail show the actual pipeline name of the calling pipleine rather than the pipleine which solely dedicated to the web activity sending the error e-mail itself. and do you have a link with such an example?

Could look to do with hard-coded parameter passing but reckon will be more optimal way.

2) What are the various levels of severity in azure data factory alerts e.g. believe 0 critical, 1 error, 4 verbose and do they only work with certain features. Found that setting pipleine fails to > 0 and sev 0 didn't send us an e-mail fro some reason so put 1 and activity runs failures > 0 and got alert e-mail as expected.

Is there a link for this e.g. success activities and sev 1 (error) would never fire I assume.

3) Noticed after got e-mail that when look at alert in azure data factory then get e-mail that alert deactivated- is this expected e.g. if don't go to azure portal assume alert stays activated?

Thanks

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
11,656 questions
0 comments
Accepted answer
  1. MartinJaffer-MSFT 26,236 Reputation points
    2022-04-29T21:59:15.997+00:00

    Hello @Philip O'Rourke and welcome to Microsoft Q&A.

    I am not that good with Azure Monitor, butI think I can help with part 1). The others will take me some time to look up.

    As for making the link go to a specific pipeline definition rather than the factory in general, I have something to share. You may not have noticed, but when you click to edit a pipeline, the URL bar changes to include the pipeline name. Reloading the browser then takes you back to the same pipeline view. 

    https://adf.azure.com/en/authoring/pipeline/{pipeline_name}?factory=...

    and when monitoring a pipeline run it is like 

    https://adf.azure.com/en/monitoring/pipelineruns/{pipeline_run_id}?factory=...

    For changing the link from the send_email pipeline to the caller, there are a couple options.
    You can pass the parent/calling_pipeline's pipeline name to the child/send_email pipeline as a parameter, and then you can do like 

    "pipelineName": "@{pipeline().parameters.CallerName}"

    or you could get fancy and ... actually, nevermind, passing a parameter is simplest.

    I imagine the calling pipeline information should be available in Azure Monitor too. It might be under invokedBy.pipelineName like in the result of using Get_pipeline_run_by_id. I'd need to set up and look myself.

    Let me know if this helps, or if you have any questions.

    2 people found this answer helpful.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.