Databricks automate

Question

Hello , I have scheduled data bricks automated mail for data quality checks for 15-20 datasets which send DQ report at certain scheduled time. Since all these mails goes to business in separate mailers, it wants all of them as a single mail. How to do this in data bricks or do I need to use logic apps or other service? thanks

Answer 1

@Samy A

To consolidate the Data Quality (DQ) reports for your 15–20 datasets into a single email, you have a couple of options depending on your setup and flexibility:

Approach 1 - Handle it directly in Databricks

If your current DQ checks are run in separate notebooks or jobs, you can instead:

• Create a master notebook or job that loops through all datasets,
• Runs the DQ checks for each one,
• Aggregates the results (as text, HTML, or dataframe),
• Sends a single email at the end using a Python library like smtplib or via a service like SendGrid.

This keeps everything inside Databricks and avoids sending separate emails from each job.

Approach 2 - Use an external orchestration tool (e.g., Azure Logic Apps)

If consolidating within Databricks is difficult, you can:

• Store individual DQ reports (e.g., as files in Blob Storage or a database),
• Use Azure Logic Apps or Power Automate to fetch those reports on a schedule,
• Merge them into one email (either inline or as attachments),
• Send it to the business users.

This is especially useful if your organization already uses these tools for workflow automation or centralized email handling.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.

Thank you.