Create and manage detection rules

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

Detection rules are filters that run on your tables to identify anomalies and potential security threats. Rules typically consume data from the parsed tables written by your datasources, but can run against any table you can query. Rules identify signals based on your specified criteria.

This page covers how to create, view, update, and manage detection rules in the UI.

Create and manage detection rules in the UI

This page covers the configuration and deployment of detection rules in the UI. You'll configure rule execution parameters, scheduling, compute resources, and signal generation settings.

Before you begin

Prerequisites: You should have already designed your detection logic. This page configures how that logic executes in production.

Workflow: After completing this form, Lakewatch schedules and runs your detection rule according to your settings. Matches generate signals, which Lakewatch groups into alerts that appear in the Alerts side panel for analyst review. You can also query signals directly in the signals system table in Unity Catalog.

How do I create detection rules?

Go to Detection rules and click Create detection rule. The form includes SQL and query configuration, with settings organized in tabs:

• Execution: Schedule, active state, alert suppression, and run-as principal.
• Signals: Output summary, context fields, and deduplication.
• MITRE ATT&CK (optional): Map the detection to the MITRE ATT&CK framework.
• Details: Classification metadata (severity, category, fidelity, objective).

These header controls apply throughout:

• Create with Genie: Opens a docked side panel where Genie (an AI assistant) helps you write and refine detection logic. Genie's tagline: "Build, review, and refine detection rules with an AI partner."
• Edit as YAML: Switch between the form and editing the rule as YAML; the two stay in sync. See Edit a rule as YAML.
• Active (toggle): Whether the rule is enabled and runs on its schedule.
• Edit alert suppression (button): When suppression is on, an active rule still runs and produces signals, but those signals don't open alerts. Set an until date and time for when suppression ends (required). Times use your local timezone and are stored as UTC.

Rule configuration

Name the rule, define its detection logic, and select its compute warehouse:

• Rule name (required): Enter a name for the rule. It can't be changed after creation.
• Detection SQL (required): Write your detection logic as inline SQL. See Write detection rules for how the detection SQL works.
• SQL warehouse (required): A detection rule runs your detection SQL on a SQL warehouse you select, in its own isolated job. Rules don't use cluster grouping or custom compute settings. If you don't have a warehouse sized for detection workloads, see the detection warehouse guide.
• Description (optional): A short description of your rule's function, e.g. "Detects credential stuffing" or "Identifies large email attachments."
• Run-as principal (optional): The user or service principal the rule runs as. Defaults to the rule's creator.

Execution tab

Configure when the rule runs, whether it's active, and how alerts are suppressed:

• Active (toggle): Whether the rule is enabled and runs on its schedule. Default: on.
• Schedule (button "Edit schedule"): Set how often the rule runs, using one of two schedule types:
  • At least every: The rule runs at least this often (minimum 5 minutes), but may run more frequently under certain circumstances. Set an amount and a unit: minutes, hours, or days.
  • Exactly at: The rule runs on a cron schedule. Build it with the Every N control and a unit (minute, hour, day, week, or month) plus the at control, or select Show cron syntax to enter a Quartz cron expression directly.
  • Either way, the rule is a batch job that runs on the SQL warehouse selected in the Rule & input step, in its own isolated job. Cluster grouping and compute settings don't apply.
• Alert suppression (button "Edit alert suppression"): When on, the rule still runs and produces signals, but those signals don't open alerts. Set an until date and time for when suppression ends (required). Times use your local timezone and are stored as UTC.

Signals tab

Shape the signals the rule produces and control deduplication:

Summary

The signal summary appears in the detection results list and provides analysts with a quick, descriptive headline of what was detected. This field allows you to create dynamic alert messages by incorporating data from the original event using SQL expressions wrapped in curly braces {sql-expression}. Signals that share a summary, rule ID, and version are grouped into one alert; if their entire output rows also match within the deduplication window, the duplicate is dropped instead. To avoid creating an excessive number of alerts, keep the summary general and avoid field expansions that produce a unique value on every match, such as a timestamp or event ID.

Example:

Brute force attempt detected for `{user.name}`

Context

The context field attaches structured metadata to the signal event as key-value pairs. Each key is a label and each value can include SQL expressions wrapped in curly braces {sql-expression} to insert event details. For example, a key of source_ip with a value of {src_endpoint.ip}.

Alternatively, toggle Include all data to automatically record all columns from the input row that triggered the signal.

Deduplication

Control whether Lakewatch skips duplicate signals produced by this rule. See Signal deduplication for how duplicates are identified and how the time window works.

• Deduplication: Toggle on to drop signals whose entire output rows are identical within the time window; toggle off to record every signal.
• Deduplication time window: Automatic (recommended) derives the window from the rule's schedule. A custom window can't exceed 7 days.

MITRE ATT&CK tab

Map the detection to the MITRE ATT&CK framework for standardized threat categorization:

MITRE ATT&CK® is a globally-accessible knowledge base of adversary tactics and techniques based on real-world observations. The ATT&CK knowledge base is used as a foundation for the development of specific threat models and methodologies in the private sector, in government, and in the cybersecurity product and service community.

If the detection maps to the MITRE ATT&CK framework, complete the following fields to help analysts understand the threat context and enable better threat hunting:

Click Add to create a mapping, then fill in these dropdowns. Add as many mappings as apply:

• Taxonomy: The MITRE ATT&CK matrix, such as Enterprise.
• Tactic: The adversary's tactical goal.
• Technique: The method used to achieve the tactic.
• Sub-technique: A specific variation of the technique.

Details tab

Set the rule's classification metadata:

Required fields:

• Severity: An assessment of the severity of the threat, used to assign risk scores to signals. Values: Informational, Low, Medium, High, Critical.

• Category: A classification that describes the detection methodology. All Lakewatch workspaces include these default categories:

  • Static Signature - For rules matching exact patterns (e.g., known malware hashes)
  • Fuzzy Signature - For pattern matching with variation tolerance (e.g., similar phishing subjects)
  • Behavioral - For action-based detection (e.g., privilege escalation sequences)
  • Statistical - For threshold-based detection (e.g., unusual login volume)
  • Learning (AI/ML) - For ML-based detection (e.g., anomaly detection models)

  Custom categories can be added in workspace configuration. Categories help organize rules, filter detections by methodology, and understand detection coverage.

Conditionally required:

• Fidelity: The maturity level of the rule. Newly created, untested rules should be marked as Development or Investigative. Older, well-tested rules should be marked as High. This helps analysts determine how likely a signal is a false positive. Values: Development, Investigative, Medium, High. (Required only for rule creation; can be blank for drafts.)

Optional fields:

• Objective: A longer description for detailed rule documentation; you might include your reasoning for rule creation, testing methodology, and/or implementation notes.
• Annotations (Key/Value pairs): An additional categorization method using a flexible tagging system for custom organization.

Create or cancel the rule

Once you've filled in all required fields, click Create rule to save and deploy the rule. Click Cancel to discard changes.

Edit a rule as YAML

The rule form includes an Edit as YAML button that opens a YAML editor. The form and YAML editor stay in sync: changes you make in YAML populate the form, and the YAML always reflects the current form values. Use the YAML editor to review a complete rule at a glance, or to copy a rule and adjust a few fields.

A rule's YAML uses flat, snake_case field names, and each field maps to a control in the wizard. The following example defines a scheduled rule that reads parsed authentication events and generates a signal for each matching login:

catalog_name: security
schema_name: detections
rule_name: suspicious_login_activity

description: Detects login attempts from unusual locations.
severity: High
category: Behavioral
fidelity: FIDELITY_INVESTIGATIVE

schedule:
  enabled: true
  at_least_every: 1h

batch:
  sql:
    query: |
      SELECT
        time,
        actor.user.name AS user_name,
        src_endpoint.ip AS source_ip
      FROM security.parsed.authentication_events
      WHERE IN_TIMERANGE(time)
        AND activity_name = 'Logon'

output:
  event_time_column: time
  summary: 'Suspicious login from {source_ip} by {user_name}'
  context:
    user_name: '{user_name}'
    source_ip: '{source_ip}'
  default_context: false

Rule fields

Each field maps to a wizard control described above. For what a field does, see its wizard step.

YAML field	Wizard control	Required	Notes
catalog_name	Catalog	Yes	Can't be changed after the rule is created.
schema_name	Schema	Yes	Can't be changed after the rule is created.
rule_name	Name	Yes	Can't be changed after the rule is created.
description	Description	No
run_as_principal	Run-as identity	No	A user email or service principal application ID. Defaults to the rule's creator.
batch.sql.query	Detection SQL	Yes	The detection query. Use IN_TIMERANGE() to scope it to each run's window. See Write detection rules.
batch.warehouse_id	Compute	No	The ID of the SQL warehouse the rule runs on.
severity	Severity	Yes	One of Informational, Low, Medium, High, or Critical.
category	Category	No	A category label, such as Behavioral.
fidelity	Fidelity	Yes	One of FIDELITY_DEVELOPMENT, FIDELITY_INVESTIGATIVE, FIDELITY_MEDIUM, or FIDELITY_HIGH (shown in the wizard as Development, Investigative, Medium, and High).
objective	Objective	No
mitre	MITRE ATT&CK	No	A list of mappings, each with taxonomy, tactic, technique_id, technique, sub_technique_id, and sub_technique.
output.summary	Summary	No	Supports {sql-expression} substitutions.
output.context	Context	No	A key/value map. Values support {sql-expression} substitutions.
output.default_context	Include all data	No	When true, records every column of the triggering row.
output.event_time_column	—	No	The output column that holds the event timestamp.
deduplication	Deduplication	No	One of DEDUPLICATION_DEFAULT, DEDUPLICATION_ENABLED, or DEDUPLICATION_DISABLED.
deduplication_time_window	Deduplication time window	No	A duration such as 1h, 24h, or 7d, or automatic.
schedule.enabled	Active	No	Whether the rule runs on its schedule. Defaults to true.
schedule.at_least_every	At least every	One of	An interval such as 5m, 15m, or 1h (minimum 5 minutes), for an At least every schedule.
schedule.cron_expression	Exactly at	One of	A Quartz cron expression, for an Exactly at schedule.
suppress_until_time	Suppress until	No	A timestamp until which the rule's signals don't open alerts.
annotations	Annotations	No	A key/value map.