Genie ZeroOps

Important

This feature is in Private Preview. To try it, reach out to your Azure Databricks contact.

Genie ZeroOps is a background agent that monitors your jobs, pipelines, and tables, detects failures and data quality issues, identifies the root cause, and proposes a fix. It is part of the Genie family of AI experiences in Azure Databricks.

To set up Genie ZeroOps and work with the insights it generates, see Get started with Genie ZeroOps.

What Genie ZeroOps does

Data teams spend significant time finding the source of a failure or a data quality issue. A job fails, a table goes stale, or an upstream schema change breaks a downstream table, and an engineer traces the problem across lineage, pipeline graphs, and run logs to understand what happened.

Genie ZeroOps does this work for you. It runs in the background across the assets you are responsible for, following four steps to help you find and fix issues:

1. Detection: Finds failing jobs and pipelines, jobs that succeed but produce stale or incomplete tables, and tables that are stale or missing data.
2. Root cause analysis: For each issue, identifies the underlying cause, such as a code change, a schema change, or an upstream failure.
3. Impact analysis: Shows the downstream assets that an issue affects, so you can prioritize the issues that matter most.
4. Resolution: Proposes a fix and links you to the source, such as the notebook, where you make the change.

How it works

You configure Genie ZeroOps on a workspace and select what it monitors, such as the jobs or schemas you own. It can monitor the jobs, pipelines, and tables that you have OWNER, CAN MANAGE, or CAN VIEW permission on. You can change this scope at any time.

Genie ZeroOps groups its findings into insights. An insight describes a single issue and its shared root cause. If one root cause affects several assets, Genie ZeroOps groups them into one insight instead of reporting each one separately. Insights appear in an inbox, where you review them, follow the proposed fix, and track them to resolution.

Each insight contains:

• A description of what was detected.
• The root cause of the issue.
• The downstream assets that the issue affects.
• A proposed fix and a link to the source where you make the change.

Data processing

When you select the assets for Genie ZeroOps to monitor, you are choosing to have Databricks process them using a large language model on a continuous, autonomous basis until you disable Genie ZeroOps.

Genie ZeroOps is powered by Anthropic on Databricks, which uses endpoints hosted by Databricks within the Databricks security perimeter.

Permissions and visibility

You must have at least CAN VIEW permission on an asset to see insights for it. These are the same permissions needed to set up Genie ZeroOps, and other users cannot see insights for assets they lack permission on.

When you can view an insight but not the asset that Genie ZeroOps identifies as the root cause, Genie ZeroOps masks the information you do not have access to. Instead of naming the upstream asset, it reports that an upstream job or pipeline failed.

Guardrails

Genie ZeroOps does not change production data or code without human review. It proposes fixes and links you to the source, but you generate the fix yourself, or by prompting Genie Code, then approve it through your regular workflow.

Limitations

• When you set up Genie ZeroOps, the resulting configuration and monitoring for insights is for your account only. The insights are not shared with other users in your workspace, and cannot be reassigned to others to view or for follow-up.
• Genie ZeroOps monitors jobs, pipelines, and tables. It does not monitor other asset types, such as models or apps.
• Genie ZeroOps detects job and pipeline failures and data quality issues. It does not detect performance or cost regressions.
• Genie ZeroOps does not validate fixes in a production sandbox.
• Genie ZeroOps operates at the workspace level, not the account level.
• Genie ZeroOps is not HIPAA compliant.
• While the Genie ZeroOps Private Preview is free, detecting data quality issues on your tables requires data quality monitoring, which Databricks bills separately at standard pricing.