Set up a detection warehouse

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

Scheduled detections run on a SQL warehouse that you provide. Lakewatch doesn't create or manage this warehouse. Instead, you create it in your workspace and select it when you author a scheduled detection. Creating and sizing a warehouse usually requires workspace admin permissions, so this is typically an admin setup task.

This page recommends a starting configuration for a detection warehouse. You can share a single warehouse across all of the scheduled detections in a workspace.

Prerequisites

A detection warehouse must run Databricks SQL 2026.35 and above. Lakewatch checks the version the warehouse is running, not its channel setting. If Databricks has pinned your warehouse to a specific Databricks SQL version, the pinned version must also meet this minimum.

To check the version a warehouse is running, run this query on that warehouse:

SELECT current_version().dbsql_version;

See current_version function.

Unsupported warehouse errors

When Lakewatch checks the version

Lakewatch validates the warehouse version at these points:

  • When you create a rule: Lakewatch rejects the rule if its warehouse is unsupported.
  • When you change a rule's SQL warehouse or run-as principal: Lakewatch rejects the change if the warehouse is unsupported.
  • When Lakewatch revalidates existing rules: If an existing rule's warehouse is unsupported, Lakewatch stops running the rule and records a system event for it.

If Lakewatch can't determine the version a warehouse is running, it treats the warehouse as unsupported.

Use a supported warehouse

If Lakewatch rejects a warehouse because of its version, the error message starts with one of the following:

  • SQL warehouse '<name>' (<id>) is running a version that Lakewatch does not support.
  • Lakewatch could not verify that SQL warehouse '<name>' (<id>) is supported.

To recover:

  1. Select a SQL warehouse that meets the minimum version, or contact Databricks Support for help updating the warehouse.
  2. Retry the rule creation or update.

If you switch an existing rule to a different supported warehouse, Lakewatch revalidates the rule right away. If the rule's current warehouse is updated to a supported version instead, Lakewatch resumes running the rule the next time it revalidates.

A different error means that Lakewatch couldn't run the version check at all:

  • Lakewatch could not validate the SQL warehouse runtime. Check warehouse availability and the rule's run-as permissions, then retry.

In this case, check that the warehouse is available and that the rule's run-as principal can use it, then retry.

For most workspaces, start with one serverless SQL warehouse shared by all of your scheduled detections:

Setting Recommended value
Type Serverless
Size 2X-Small
Clusters (min) 1
Clusters (max) ceil(peak concurrent rules / 10), up to the 40-cluster maximum
Auto stop 10 minutes

A 2X-Small warehouse can run 100 or more simple detections. Because a SQL warehouse is billed by cluster uptime rather than per query, the more rules that share a warehouse, the lower the cost per rule.

Note

A warehouse that runs only one or two rules can cost the same as, or more than, serverless compute. The cost advantage comes from sharing one warehouse across many detections.

When to adjust

Change the defaults based on your workload:

  • Many, high, or bursty rules: Raise the maximum cluster count so queries don't queue. Adding clusters (scaling out) is the main lever for concurrency. Each cluster provides roughly 10 query slots.
  • Latency-critical detections: Set a warm floor (minimum of 2 or more clusters) or a longer auto stop to avoid cold starts, at higher cost.
  • Few, infrequent, or cost-sensitive rules: Lower auto stop to 5 minutes (the minimum) to reduce idle time. A cold warehouse restarts in about 5 seconds.
  • Heavy or complex queries (large scans or joins): Increase the warehouse size (Small, Medium, or larger). Use a bigger size (scaling up) only when an individual query needs more memory or compute.

Cost attribution

To track detection cost on its own, use a dedicated warehouse for detection workloads rather than sharing one with other query activity. Because billing is per cluster uptime, a dedicated warehouse makes it straightforward to attribute cost to detection.

Next steps

  • Last updated on