Get log analytics data in a seperate table and as an alert

Adeleke Abimbola 231 Reputation points
2021-09-17T14:57:23.823+00:00

I want to be able to get alerts about failed pipeline runs as an alert and in a separate table outside. What I currently get with azure alert is the name of my data factory and a link to the pipeline but I would like to know the name and error message of the failed pipeline, I currently can view this information in log analytics after running my query but is there a way to get the same information as an email alert and be logged to a separate database.

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
2,925 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,904 questions
Azure Data Lake Analytics
0 comments No comments
{count} votes

Accepted answer
  1. Adeleke Abimbola 231 Reputation points
    2021-09-20T11:46:51.857+00:00

    Thank you very much @ShaikMaheer-MSFT , this was very helpful. I have been able to solve the issue with logic app since I posted this question. I realized that using this method isn't very efficient because I either have to do a stored procedure activity for every step in the pipeline or do an execute pipeline activity and insert my whole pipeline process in it.
    What I did with the logic app was to read the result of my log analytics query for every pipeline error and send the result as an email. I hope this is a better reply for anyone having this same issue.
    I'm currently trying to do a parallel action that can also log the data to a table as well. Any help with that?

    https://learn.microsoft.com/en-us/azure/azure-monitor/logs/logicapp-flow-connector


1 additional answer

Sort by: Most helpful
  1. ShaikMaheer-MSFT 38,306 Reputation points Microsoft Employee
    2021-09-20T11:31:27.957+00:00

    Hi @Adeleke Abimbola ,

    Welcome to Microsoft Q&A Platform. Thank you for posting your query here.

    You can create alerts on top of your query on log analytics. Click here to know more about it. To log your pipeline failures to separate database table you can have a stored procedure created in dB and call that stored procedure on pipeline activity failures to log information.

    Below link shows logging pipeline executions to database table using stored procedure activity. Sample example you tweak to log only failures.
    https://www.youtube.com/watch?v=yoX0wTZzIcg

    To access any activity failure error message use below expression,
    @activity('<activity name>').Error.message

    Below few links which explains logging pipeline failures to database table.

    Hope this will help. Please let us know if any further queries.


    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments