Find and resolve issues with Insights

Important

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Azure Databricks previews.

Insights is a background agent that periodically reviews a Lakebase project's telemetry and surfaces issues and early warnings without you having to ask. This page turns Insights on for a project, then walks through a real insight end to end: what it found, why it matters, and how to respond.

The Insights loop: telemetry is collected and written continuously to Unity Catalog; Insights reviews it on a recurring schedule; an issue is surfaced in the workspace-wide list; and you triage it by acting, handing it to Genie, or setting its status. The loop repeats on a schedule, so findings come to you.

For a conversational, on-demand alternative, or to continue investigating an insight further, see Diagnose and fix issues with Genie.

Note

There's no separate charge for observability. Its cost is bundled into products you already use: exporting telemetry counts toward your ingestion usage, storing it as Delta tables in the Lakehouse counts toward your Databricks-managed storage, and asking Genie counts toward your Genie usage.

Prerequisites

  • A Lakebase Autoscaling project on AWS or Azure, with a running compute.
  • A Unity Catalog catalog and schema to receive telemetry (not Default storage).

Turn on Insights for a project

Insights runs per project, controlled by an observability configuration. If you don't have one yet, create and assign one first.

In the configuration wizard, make sure Enable insights is on. Without it, a configuration can write telemetry to Unity Catalog without surfacing anything proactively. You can also set an Analysis frequency for how often Insights reasons over your telemetry.

Find an insight

Insights runs on a recurring schedule, not continuously, so expect a delay between a real issue occurring and an insight appearing for it. Give it some time after you turn the agent on, especially right after enabling it on a new project.

Insights appear in two places:

  • The workspace-wide list: in the left navigation, click Insights. It aggregates insights across every project the configuration applies to. Use Search or Filter by to narrow it, for example by project or severity.
  • A single project's view: open the project, go to Monitoring, and select the Insights tab. It shows only that project's insights.

Each row shows a Severity (for example Minor or Moderate), the Insight title, which Project it's from, the Time window the issue covers, and a Status you set (for example Open).

How to read an insight

Click an insight to open its detail page. Every insight follows the same structure, which reads top to bottom as a diagnosis, not just a notification:

  • Summary — what's happening, in plain language.
  • Evidence — the specific signals and values that support the diagnosis, in a table.
  • Recommendations — concrete next steps, with the reasoning behind each one.

What Insights finds, and how it words the diagnosis, depends entirely on your own telemetry, so the insight you see doesn't match the one below. The following is a real insight, included to show the level of detail you can expect.

Example: "Idle session locked out 4 updates"

This insight's subtitle, "Idle session held locks; UPDATE waits totalled 749 s," already tells you where to look:

Insight detail page for "Idle session locked out 4 updates," at Moderate severity, showing a Summary of an idle-in-transaction session blocking four UPDATE statements and an Evidence table with the terminating FATAL log, idle-in-transaction time, and total UPDATE wait time.

The Summary explains what happened: a session opened a transaction, went idle while still holding a row lock, and stayed that way for roughly 9.5 minutes until the database's own idle-in-transaction timeout forcibly closed it. The Evidence table backs that up with the specific numbers, including the terminating log entry and a total of 749 seconds of UPDATE wait time spread across 4 calls, averaging 187 seconds each for updates that would normally complete in milliseconds.

Recommendations:

  1. Tune idle_in_transaction_session_timeout to a safe value, such as 30-60 seconds, so an idle session releases its locks quickly instead of blocking other queries for minutes.
  2. Audit for unclosed transactions and use connection pooling with short checkout timeouts, so a client that opens a transaction and goes idle can't hold a lock indefinitely.

Notice what the evidence quantifies: not just "queries were slow," but exactly which session caused it, how long it held the lock, and how much wait time it cost other queries. Severity is inferred from the available data, so read the Evidence to confirm the root cause before you act on Severity alone.

Respond to an insight

After you've read an insight, you have three ways to close the loop:

  • Apply the recommendation yourself, using the evidence and recommendations as a guide.
  • Click Fix with Genie on the insight's detail page to hand it off to the interactive agent and keep investigating conversationally. This helps when the recommendation needs more context before you act, or when the fix involves a database-side change. See Diagnose and fix issues with Genie.
  • Update Status (for example to dismiss or mark resolved) after you've addressed it or determined no action is needed.

A finding can disappear on its own

Because Insights re-evaluates on its own schedule, a finding can drop off the list even if you never change its Status: a transient event like a one-time bulk load might not recur in a later window, or the compute behind it might scale to zero and resume fresh with the problematic session gone. A finding leaving the active list doesn't mean it was dismissed or that anything went wrong. It reflects that the condition is no longer present in the latest evaluation window.

Note

Viewing insights requires Workspace access in the workspace, separate from any permission needed to create the configuration. A user without Workspace access can't open insights even if they can see the project.

Next steps

  • Last updated on