How to set DLT pipeline warning alert?

Question

How to set DLT pipeline warning alert?

John Rambo 20

Hi there,

How to implement a security threshold set for a field in the Streaming table in Delta Live Pipeline. When real-time data exceeds the threshold, an alarm email is sent to notify users. The user's mailbox is not the databricks user's mailbox.

Thanks&Regards,

zmsoft

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Chandra Boorla 11,750 Reputation points Microsoft External Staff

2025-04-04T16:37:04.85+00:00

@John Rambo

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Accepted answer

0 additional answers

Your answer

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Chandra Boorla 11,750 Reputation points Microsoft External Staff

2025-04-04T16:37:04.85+00:00

@John Rambo

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.

Answer 1

Chandra Boorla 11,750 Microsoft External Staff

@John Rambo

To set up a warning alert for a DLT pipeline when a specific field in a streaming table exceeds a security threshold, you can define custom actions using event hooks. While the context does not provide explicit instructions on implementing a threshold alert specifically, you can utilize the built-in monitoring, and observability features of DLT pipelines to configure alerts for pipeline events.

Define Custom Monitoring - Use event hooks to define actions that should be triggered when certain conditions are met, such as when data exceeds a threshold.

Email Notifications - You can configure email notifications to send alerts to specified email addresses when certain events occur, such as a pipeline update failing or when specific conditions in your data are met.

For detailed steps on how to implement these features, refer to the documentation on defining custom monitoring and adding email notifications.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.

Chandra Boorla 11,750 Reputation points Microsoft External Staff

2025-04-03T18:21:58.1733333+00:00

@John Rambo

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
John Rambo 20 Reputation points

2025-04-06T03:14:52.9966667+00:00

Hi @Chandra Boorla ,

Thanks for your answer.

Is there any samples of dlt user customer event for warning alert

Thanks&Regards

john

Chandra Boorla 11,750 Microsoft External Staff

@John Rambo

Sample DLT Pipeline with Custom Warning Alert

Here's an example of how to implement a custom warning alert in a Delta Live Tables (DLT) pipeline when a field value exceeds a threshold:

from pyspark.sql.functions import col, explode
import smtplib
from email.message import EmailMessage

def send_email_alert(max_value):
    msg = EmailMessage()
    msg.set_content(f"Security threshold exceeded! Current value: {max_value}")
    msg['Subject'] = "DLT Pipeline Alert"
    msg['From'] = "******@yourcompany.com"
    msg['To'] = "******@external.com"

    with smtplib.SMTP('your.smtp.server', 587) as server:
        server.starttls()
        server.login("username", "password")
        server.send_message(msg)

# Load event log
event_log = spark.read.format("delta").load("dbfs:/pipelines/<pipeline-id>/system/events/")

# Extract failed expectations
failed = (
    event_log
    .filter(col("event_type") == "flow_progress")
    .select(explode("details.expectations").alias("expectation"))
    .filter(col("expectation.name") == "threshold_check")
    .filter(col("expectation.failed_records") > 0)
)

# If any failures, send alert
if failed.count() > 0:
    max_val = failed.select(col("expectation.failed_records")).collect()[0][0]
    send_email_alert(max_val)

I hope this information helps.

Chandra Boorla 11,750 Reputation points Microsoft External Staff

2025-04-09T00:19:31.44+00:00

@John Rambo

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
John Rambo 20 Reputation points

2025-04-10T03:33:43.3066667+00:00

Hi @Chandra Boorla ,

Thanks for your answer, solved my issue.

I can't find the Accept answer button.

Thanks&Regards,

John
Chandra Boorla 11,750 Reputation points Microsoft External Staff

2025-04-10T06:24:54.7333333+00:00

@John Rambo

Glad that your issue got resolved. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Share via

How to set DLT pipeline warning alert?

0 additional answers

Your answer