Cloud storage

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

Use the Cloud storage ingestion method to ingest files from a supported cloud object store. Lakewatch reads the source with Auto Loader.

This page teaches you how to create a cloud storage datasource. For managing a datasource after it exists, see Create and manage datasources.

Lakewatch ingests from Azure Data Lake Storage through a Unity Catalog volume or external location.

Prerequisites

Before you create the datasource, make sure you meet the following requirements:

  • Source in Unity Catalog: Configure your storage as a Unity Catalog volume or external location. Because Unity Catalog governs access to the source, this is the recommended path. For setup, see Manage external locations.
  • Run-as access to the source: The datasource's run-as identity must be able to read the source: the READ FILES privilege on the external location, or READ VOLUME on the volume. Grant this on the external location or volume in Unity Catalog, not on the identity itself. For instructions, see Manage external locations or Privileges for Unity Catalog volumes. Lakewatch ingests as the run-as identity, not as your user account, so your individual account-level access to the data isn't sufficient. Access the identity inherits from a parent catalog or schema, or from a group, also satisfies this requirement.

Prepare your data for ingestion

Each Lakewatch datasource ingests from a single directory and reads all files using one file format. Before creating a datasource, organize your cloud storage so that each directory you plan to ingest contains:

  • One file format: All files must be the same format (for example, all JSON, all Parquet, or all CSV). Files in different formats cause parse errors.
  • One source type: Files should come from the same source system and share a compatible schema (for example, all AWS CloudTrail logs, or all CrowdStrike detection events). Minor schema variations within the same format are handled automatically, but mixing fundamentally different schemas in one directory isn't supported.

Example directory structure:

s3://my-security-bucket/
├── cloudtrail/           ← Datasource 1 (JSON)
│   ├── 2025/01/01/
│   │   ├── event1.json.gz
│   │   └── event2.json.gz
│   └── 2025/01/02/
│       └── event3.json.gz
├── vpc-flow-logs/        ← Datasource 2 (Parquet)
│   ├── region=us-east-1/
│   │   └── flowlogs.parquet
│   └── region=us-west-2/
│       └── flowlogs.parquet
└── waf-logs/             ← Datasource 3 (JSON)
    └── waf-events.json

Each top-level directory becomes a separate Lakewatch datasource. Subdirectories are traversed automatically, including Hive-style partitioned paths (like region=us-east-1/), which Lakewatch detects and maps as partition columns.

Create the datasource

From Datasources, click Add datasource and select Custom datasource. On the creation page, set the Source type to your storage location, then fill out the following fields:

  • Datasource name: Enter a unique name for the datasource (for example, Cloudflare_WAF).
  • Data location: Enter the path to the directory containing your data (for example, s3://my-bucket/cloudtrail/, dbfs:/mnt/logs/, or /Volumes/catalog/schema/volume/). The path must point to a directory, not an individual file. To ingest only files matching a pattern, you can include a glob pattern in the path. For example, s3://my-bucket/logs/{2025,2026}/*.json.gz. Supported characters are *, ?, {, }, [, and ]. For full syntax details, see the Databricks autoloader path filtering documentation.
  • File format: The format Lakewatch uses to parse the source data: JSON, JSONL, Parquet, CSV, XML, or text. Click Infer with Genie to have Genie suggest the format from a sample of your data.
  • Data ingestion: Choose how much of the source to ingest. This choice takes effect only at first start.
    • New data (default): ingest only files that arrive after the datasource first starts. Files already present at the location are skipped.
    • All data: ingest the files already present at the location, then continue with files that arrive later.
  • Advanced settings: Expand this optional section to configure:
    • Raw output table name: The name of the raw table that Lakewatch writes source events to before parsers run. Lakewatch creates it in your environment; the name is required.
    • Performance optimized: Use performance-optimized compute for faster ingestion (on) or standard compute (off). Defaults to the workspace ingestion-compute setting.

File discovery and credentials

Lakewatch detects new files automatically. If the source's Unity Catalog external location has file events configured, Lakewatch uses them; otherwise it lists the directory. There's no file-notification setting to configure in Lakewatch.

When the source is a Unity Catalog volume or external location, Lakewatch authenticates through it. There's no separate credential or secret-scope entry. Make sure the run-as identity can read the Unity Catalog object and its backing storage credential.

Preview the data

Once you set the data location and file format, review the Data preview at the bottom of the page. It shows the raw data as Lakewatch writes it: a data column holding each record and a _metadata column. Lakewatch samples a file from the configured Data location.

The sample affects only the preview; when the datasource runs, it ingests from its configured source.

Time column handling

The time column is critical for clustering, streaming watermarks, and query performance. Lakewatch auto-detects and extracts it during setup, including when the timestamp is nested inside a VARIANT structure. Before writing each row to the raw table, Lakewatch replaces null time values with the ingestion timestamp (lw_meta.ingestion_time), so the raw table's time column is never null; it sets lw_meta.missing_timestamp to true for those rows so you can tell original event times from ingestion-time fallbacks.

Attach parsers and create the datasource

On the same creation form, scroll to the Parsers section:

  1. Attach parsers. Attach one or more parsers to schematize the data: a built-in parser that matches your source or one you authored. If your source delivers enveloped records, also set the Unwrapping option, described in Unwrap enveloped records. For how parser matching works and the columns parsers produce, see Attach parsers.
  2. Create the datasource. Click Create datasource. The datasource starts running immediately on its processing schedule; to stop it, delete the datasource.

You can edit the datasource's other settings (processing schedule, compute, and annotations) any time on its Overview tab after creation. See Configure datasource settings.

Unwrap enveloped records

Some sources deliver several events inside a container rather than one record per line. When the sampled data is a JSON array of events, or events wrapped in an AWS CloudWatch Logs delivery envelope, Lakewatch can unwrap the container so that each event becomes its own row. In the parser step, use the Unwrapping option (the default is No unwrapping) to match how your records are wrapped. Only the record shapes surfaced during setup are supported. Unwrapping is available for cloud storage datasources but not for the Existing table method, which ingests a Unity Catalog table as it is.

  • JSON array: Enter the JSON array key: the root-level key containing the array of events (for example, Records). Nested paths aren't supported.
  • AWS CloudWatch Logs: Turn on Retain CloudWatch Logs headers to keep the original CloudWatch Logs envelope metadata on each unwrapped event. It's off by default.
  • Last updated on