Parser reference

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

A parser is a YAML document that describes the output columns of a security log source. A parser is versioned and stored in a Unity Catalog catalog and schema; editing a parser appends a new immutable version, so a datasource that pins a version is not affected by later edits. This page describes the parser syntax that Lakewatch accepts.

Document structure

description: Product audit events # what this log source is
fields:
  - name: event_name
    type: STRING
    required: true
    validate:
      in: ['sale', 'buy']
  - name: source_ip
    type: STRING
    observables: [ip]
  - name: actor
    type: VARIANT
    fields:
      - name: id
        type: STRING
        observables: [actor_id]
time:
  coalesce:
    - path: published
      time_formats: ['#rfc3339']

You can provide description in the YAML as shown here, or set it in the UI when you create the parser (recommended).

Event time

Every parser produces a time column: the event timestamp Lakewatch clusters on. You must define it with a top-level time block rather than a field in fields; a custom parser without one is rejected. Because Lakewatch always produces the time column and it's always a TIMESTAMP, you don't set its name, type, or required. Don't add a time field to fields: the name is reserved for the column that the top-level time block produces, and a parser that declares it as a field is rejected.

The top-level time block takes a coalesce list: a prioritized list of sources that Lakewatch evaluates in order, using the first non-null value. If every source is null, Lakewatch falls back to the ingestion time and sets lw_meta.missing_timestamp to true. Each source is one of the following:

Path: read the value from a source location:

time:
  coalesce:
    - path: published
      time_formats: ['#rfc3339']

Path with a regular expression: extract the value from a source location with a regex:

time:
  coalesce:
    - path: raw_message
      regexp: '<pattern>'
      match_group_index: 1 # which capture group holds the value; defaults to 1
      time_formats: ['#unix']

time_formats uses the same format sets and patterns as a TIMESTAMP field. See Timestamps.

Because time isn't a field, it isn't used in classification. To require the source event-time field to be present for an event to match, add it as a required field in fields.

Field attributes

Attribute	Applies to	Description
`name`	All fields	Output column name. Also the default source key, unless `path` is set.
`type`	All fields	Output column type. See Supported types.
`required`	All fields	If `true`, the field must be present for an event to match this parser. Defaults to `false`.
`path`	All fields	Source location to read the value from, when it differs from `name`, such as `published` or `actor.id`.
`comment`	All fields	Description of the field, surfaced to help you write detection rules.
`time_formats`	`TIMESTAMP` types	List of formats to parse the timestamp with. See Timestamps.
`validate`	`STRING`	A single validator. See Validators.
`observables`	`STRING`	One or more observable types to extract. See Observables.
`scan_for_observables`	`VARIANT`	Key-matching rules for extracting observables from nested content. See Scan for observables.
`fields`	`VARIANT`	Nested field definitions.
`element`	`ARRAY`	The element definition of the array.

Supported types

STRING, BOOLEAN, TINYINT (BYTE), SMALLINT (SHORT), INT (INTEGER), BIGINT (LONG), FLOAT (REAL), DOUBLE, DECIMAL(p,s), TIMESTAMP, TIMESTAMP_LTZ, TIMESTAMP_NTZ, DATE, BINARY, VARIANT, STRUCT, and ARRAY.

Only the types listed above are supported. To model nested objects, use VARIANT with inner fields. A VARIANT tolerates additional, undeclared fields without breaking the parser.

Choosing between STRUCT and VARIANT

STRUCT and VARIANT both model nested data, but they make opposite trade-offs. A STRUCT declares the shape up front: you list its members and their types, and the parsed column holds exactly what you declare. A VARIANT keeps the raw, semi-structured value whole, and you read into it by path at query time, for example with try_variant_get. You can still declare fields under a VARIANT, but that only attaches validation and observables to those paths; it doesn't drop the rest of the value.

Use STRUCT when you know the shape and want a stable, typed contract:

The shape lives in the parser spec, so it's versioned and reviewed. Queries, detections, and dashboards read each member as a known type and don't break when the source data shifts.
Reads are cheaper. A STRUCT is a real typed column that you access with dot notation and that Spark can prune, whereas a VARIANT is decoded on every read.
The cost is rigidity: anything you don't declare is dropped, and you must update the parser when the shape changes.

Use VARIANT when the shape is unknown, evolving, or too large to model field by field:

It keeps up with schema changes on its own. Sources such as AWS CloudTrail and Okta add fields with no parser change.
It handles mixed or conflicting shapes, such as a path that's an object in one event and a string in another, which a single STRUCT type can't represent.
It preserves the original payload for later reprocessing, and lets you land data first and model it later.
It scales to payloads with hundreds of fields without declaring each one.
The cost is that you get no typing or validation at ingest, and you pay to extract fields on every query. A common pattern is to promote the few fields you query most into typed fields.

Timestamps

A TIMESTAMP field is parsed with the formats listed in time_formats, tried in order, using parse_timestamp. Each entry is either a predefined format keyword or a literal Spark datetime pattern:

Entry	Meaning
`#rfc3339`	The RFC 3339 family.
`#iso8601`, `#common`, `#extended`	Other predefined format keywords.
`#unix`	A numeric Unix timestamp, in seconds. Millisecond values aren't detected.
`MM/dd/yyyy hh:mm:ss a`	A literal pattern.

List the formats the log actually uses so that parsing is not overly permissive.

Validators

A validator asserts a condition on a STRING field. Use exactly one of the following:

Validator	True when
`in: [...]`	The value is one of the listed strings.
`not_in: [...]`	The value is not one of the listed strings.
`contains_any: [...]`	The value contains at least one listed substring.
`contains_none: [...]`	The value contains none of the listed substrings.

Observables

Marking a field with observables extracts its value into the shared lw_observables column for threat hunting. Each observable type captures a specific kind of value and populates one or more fields in lw_observables:

Observable	Captures	Populates in `lw_observables`
`ip`	IPv4 or IPv6 addresses	`ip_addresses`
`domain`	Domain names	`hostnames`
`hostname`	Hostnames, or IP addresses when the value is an IP	`hostnames`, `ip_addresses`
`url`	The host of a URL (a hostname or IP)	`hostnames`, `ip_addresses`
`hash`	MD5, SHA-1, and SHA-256 hashes	`hashes`
`aws_arn`	An AWS ARN, plus the account ID, instance ID, email, and username it contains	`account_ids`, `resource_ids`, `emails`, `usernames`, `device_ids`
`aws_account_id`	A 12-digit AWS account ID	`account_ids`
`aws_instance_id`	An EC2 instance ID	`device_ids`
`trace_id`	A request or trace ID	`request_trace_ids`
`username`	A username	`usernames`
`email`	An email address, and the username portion of it	`emails`, `usernames`
`net_addr`	The host of a `host:port` value (a hostname or IP)	`hostnames`, `ip_addresses`
`mac`	A MAC address	`mac_addresses`
`actor_id`	An actor ID	`actor_ids`
`serial_number`	A serial number	`serial_numbers`
`cve`	A CVE identifier, such as `CVE-2024-1234`	`cves`
`resource_tag`	A resource tag	`resource_tags`
`mitre_attack_technique`	A MITRE ATT&CK technique ID, such as `T1078`	`mitre_attack_technique_ids`

lw_observables is a Lakewatch-managed STRUCT column added to every parsed table, with one ARRAY<STRING> field per observable type. Lakewatch normalizes, deduplicates, and sorts each field's values and skips invalid ones; a field with no values is []. The column is present on every event, even when the parser declares no observables.

Scan for observables

Use the optional scan_for_observables attribute on a VARIANT field to extract observables from keys anywhere in the nested value, including content not declared in fields. The attribute is not valid on other field types.

Map each observable type that you want to extract to an explicit matcher object. A matcher can use equals, ends_with, and contains; each key accepts an array of key-name strings. Key matching is case-insensitive.

- name: request
  type: VARIANT
  scan_for_observables:
    ip:
      equals: [sourceIp, destinationIp]
      ends_with: [IpAddress]
      contains: [clientIp]
    email:
      equals: [principalEmail]
      ends_with: [Email]

Lakewatch recursively examines keys in the VARIANT value. When a key matches a configured rule, Lakewatch validates the candidate value for that observable type and adds valid values to the corresponding field in lw_observables. Matching uses only the rules in the parser; Lakewatch does not add built-in matching rules.

Nested and array fields

Use VARIANT with fields for nested objects, and ARRAY with element for lists. An element is defined like a field, but without name, required, or path:

- name: resources
  type: ARRAY
  path: resources
  element:
    type: VARIANT
    fields:
      - name: accountId
        type: STRING
        observables: [aws_account_id]
- name: source_ips
  type: ARRAY
  path: sourceIps
  element:
    type: STRING
    observables: [ip]

Classification

A datasource can have several parsers attached. Each incoming event is matched against the parsers in the order they are declared and lands in the first parser it matches. An event matches when its required fields are present, its typed fields convert to the declared types, and all validators pass. Unmatched events are routed to a dead-letter sentinel instead of failing the pipeline.

Built-in and custom parsers

Lakewatch provides built-in parsers for common security log sources, and lets you author custom parsers for sources that aren't covered or that you want to tailor. Both use the YAML format described above.

Discover and inspect built-in parsers

When you attach parsers to a datasource, you can browse the available built-in parsers and select the ones that match your source. Inspect a built-in parser to see the columns it produces (its fields, types, validators, and observables) before you attach it, so you can confirm it matches the events you're ingesting.

Built-in parsers are read-only. To customize one, use the Clone action on the parser to create an editable custom parser seeded from the built-in parser's definition, then adjust and save it. Cloning is a UI convenience that copies the definition into a new custom parser (version 1). It isn't a server-side clone operation, and the built-in parser itself is unchanged.

Author a custom parser

To create a custom parser, provide a YAML document in the format described above and save it to a Unity Catalog catalog and schema. Creating a parser stores the definition as version 1 and records you as the parser's owner.

Define your fields carefully, because an event that matches no parser is routed to a dead-letter sentinel rather than failing the pipeline. An event whose values don't convert to the declared types, or that fails a validator, won't match the parser.

Versioning and deletion

Editing a parser appends a new immutable version. Existing versions are never changed, and version numbers are never reused. The server assigns each new version the next number in sequence. Only the parser's owner can add a version.

A datasource pins a specific version, so it keeps using that version until you explicitly change it. See Change a parser's version.

You can delete a whole custom parser, which removes all of its versions. There is no operation to delete an individual version. Only the owner can delete a parser, and deletion is blocked while any datasource references any version of the parser. Before deleting, detach the parser from every datasource that uses it (change each datasource to a different parser and save the change) then delete the parser.

Limits

A parser document is at most 256 KB.

Feedback

Was this page helpful?

Last updated on 2026-10-16