Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
This feature is in Private Preview. To try it, reach out to your Azure Databricks contact.
Genie ZeroOps is a background agent that monitors your jobs, pipelines, and tables, detects failures and data quality issues, identifies the root cause, and proposes a fix. It is part of the Genie family of AI experiences in Azure Databricks. Unlike Genie Code, which responds when you prompt it, Genie ZeroOps runs continuously without a prompt and surfaces issues for you to act on.
To set up Genie ZeroOps and work with the insights it generates, see Get started with Genie ZeroOps.
What Genie ZeroOps does
Data teams spend significant time finding the source of a failure or a data quality issue. A job fails, a table goes stale, or an upstream schema change breaks a downstream table, and an engineer traces the problem across lineage, pipeline graphs, and run logs to understand what happened.
Genie ZeroOps does this work for you. It runs in the background across the assets you are responsible for, following four steps to help you find and fix issues:
- Detection: Finds failing jobs and pipelines, jobs that succeed but produce stale or incomplete tables, and tables that are stale or missing data.
- Root cause analysis: For each issue, identifies the underlying cause, such as a code change, a schema change, or an upstream failure.
- Impact analysis: Shows the downstream assets that an issue affects, so you can prioritize the issues that matter most.
- Resolution: Proposes a fix and links you to the source, such as the notebook, where you make the change. For a job that runs code from a Git repository, Genie ZeroOps can also push the fix to a new branch when you ask, so that you can open a pull request.
How Genie ZeroOps monitors assets and reports insights
You configure Genie ZeroOps on a workspace and select what it monitors, such as the jobs or schemas you own. It can monitor the jobs, pipelines, and tables that you have OWNER, CAN MANAGE, or CAN VIEW permission on. You can change this scope at any time.
Genie ZeroOps groups its findings into insights. An insight describes a single issue and its shared root cause. If one root cause affects several assets, Genie ZeroOps groups them into one insight instead of reporting each one separately. Insights appear in an inbox, where you review them, follow the proposed fix, and track them to resolution.
Each insight contains:
- A description of what was detected.
- The root cause of the issue.
- The downstream assets that the issue affects.
- A proposed fix and a link to the source where you make the change.
Git integration
A job can run code from a Git repository through a Git source or a bundle deployed from Git. When such a job fails, Genie ZeroOps reads the repository to find the root cause. When the fix is a code change, Genie ZeroOps prepares the change. When you ask, it pushes the change to a new branch so that you can open a pull request. See Create a pull request from a fix.
Genie ZeroOps reads the repository with your Git credential. To add a Git credential, see Add Git credentials.
Data processing
When you select the assets for Genie ZeroOps to monitor, you are choosing to have Databricks process them using a large language model on a continuous, autonomous basis until you disable Genie ZeroOps. For a job that runs code from a Git repository, the data that Genie ZeroOps processes includes the contents of the repository.
Genie ZeroOps is powered by Anthropic on Databricks, which uses endpoints hosted by Databricks within the Databricks security perimeter.
Permissions and visibility
You must have at least CAN VIEW permission on an asset to see insights for it. These are the same permissions needed to set up Genie ZeroOps, and other users cannot see insights for assets they lack permission on.
When you can view an insight but not the asset that Genie ZeroOps identifies as the root cause, Genie ZeroOps masks the information you do not have access to. Instead of naming the upstream asset, it reports that an upstream job or pipeline failed.
Guardrails
Genie ZeroOps doesn't change production data or code without human review. It proposes fixes and links you to the source, and you apply the fix yourself, prompt Genie Code, or, for jobs that run code from a Git repository, ask Genie ZeroOps to push it.
For a job that runs code from a Git repository, Genie ZeroOps doesn't write to your repository in the background. It pushes a fix only after you explicitly ask it to and confirm the push. Genie ZeroOps then commits the fix and pushes it to a new branch. It gives you a link to open a pull request in your Git provider, and the change goes through your regular review process.
Limitations
- When you set up Genie ZeroOps, the resulting configuration and monitoring for insights is for your account only. The insights are not shared with other users in your workspace, and cannot be reassigned to others to view or for follow-up.
- Genie ZeroOps monitors jobs, pipelines, and tables. It does not monitor other asset types, such as models or apps.
- Genie ZeroOps detects job and pipeline failures and data quality issues. It does not detect performance or cost regressions.
- Genie ZeroOps does not validate fixes in a production sandbox.
- Genie ZeroOps operates at the workspace level, not the account level.
- Genie ZeroOps is not HIPAA compliant.
- While the Genie ZeroOps Private Preview is free, detecting data quality issues on your tables requires data quality monitoring, which Databricks bills separately at standard pricing.