Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.
Scheduled detections run on a SQL warehouse that you provide. Lakewatch doesn't create or manage this warehouse. Instead, you create it in your workspace and select it when you author a scheduled detection. Creating and sizing a warehouse usually requires workspace admin permissions, so this is typically an admin setup task.
This page recommends a starting configuration for a detection warehouse. You can share a single warehouse across all of the scheduled detections in a workspace.
Recommended configuration
For most workspaces, start with one serverless SQL warehouse shared by all of your scheduled detections:
| Setting | Recommended value |
|---|---|
| Type | Serverless |
| Size | 2X-Small |
| Clusters (min) | 1 |
| Clusters (max) | ceil(peak concurrent rules / 10), up to the 40-cluster maximum |
| Auto stop | 10 minutes |
A 2X-Small warehouse can run 100 or more simple detections. Because a SQL warehouse is billed by cluster uptime rather than per query, the more rules that share a warehouse, the lower the cost per rule.
Note
A warehouse that runs only one or two rules can cost the same as, or more than, serverless compute. The cost advantage comes from sharing one warehouse across many detections.
When to adjust
Change the defaults based on your workload:
- Many, high, or bursty rules: Raise the maximum cluster count so queries don't queue. Adding clusters (scaling out) is the main lever for concurrency. Each cluster provides roughly 10 query slots.
- Latency-critical detections: Set a warm floor (minimum of 2 or more clusters) or a longer auto stop to avoid cold starts, at higher cost.
- Few, infrequent, or cost-sensitive rules: Lower auto stop to 5 minutes (the minimum) to reduce idle time. A cold warehouse restarts in about 5 seconds.
- Heavy or complex queries (large scans or joins): Increase the warehouse size (Small, Medium, or larger). Use a bigger size (scaling up) only when an individual query needs more memory or compute.
Cost attribution
To track detection cost on its own, use a dedicated warehouse for detection workloads rather than sharing one with other query activity. Because billing is per cluster uptime, a dedicated warehouse makes it straightforward to attribute cost to detection.
Next steps
- Create and manage detection rules: Select this warehouse when you author a scheduled detection.