Cloud storage

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

Use the Cloud storage ingestion method to ingest files from a supported cloud object store. Lakewatch reads the source with Auto Loader.

This page teaches you how to create a cloud storage datasource. For managing a datasource after it exists, see Create and manage datasources.

Lakewatch ingests from Azure Data Lake Storage through a Unity Catalog volume or external location.

Prerequisites

Before you create the datasource, make sure you meet the following requirements:

  • Source in Unity Catalog: If the source is a Unity Catalog volume or external location, set it up in Unity Catalog before you create the datasource. For setup, see Manage external locations.
  • Run-as access to the source: The datasource's run-as identity must be able to read the source: the READ FILES privilege on the external location, or READ VOLUME on the volume. Grant this on the external location or volume in Unity Catalog, not on the identity itself. For instructions, see Manage external locations or Privileges for Unity Catalog volumes. Lakewatch ingests as the run-as identity, not as your user account, so your individual account-level access to the data isn't sufficient. Access the identity inherits from a parent catalog or schema, or from a group, also satisfies this requirement.

Prepare your data for ingestion

Each Lakewatch datasource ingests from a single directory and reads all files using one file format. Before creating a datasource, organize your cloud storage so that each directory you plan to ingest contains:

  • One file format: All files must be the same format (for example, all JSON, all Parquet, or all CSV). Files in different formats cause parse errors.
  • One source type: Files should come from the same source system and share a compatible schema (for example, all AWS CloudTrail logs, or all CrowdStrike detection events). Minor schema variations within the same format are handled automatically, but mixing fundamentally different schemas in one directory isn't supported.

Example directory structure:

s3://my-security-bucket/
├── cloudtrail/           ← Datasource 1 (JSON)
│   ├── 2025/01/01/
│   │   ├── event1.json.gz
│   │   └── event2.json.gz
│   └── 2025/01/02/
│       └── event3.json.gz
├── vpc-flow-logs/        ← Datasource 2 (Parquet)
│   ├── region=us-east-1/
│   │   └── flowlogs.parquet
│   └── region=us-west-2/
│       └── flowlogs.parquet
└── waf-logs/             ← Datasource 3 (JSON)
    └── waf-events.json

Each top-level directory becomes a separate Lakewatch datasource. Subdirectories are traversed automatically, including Hive-style partitioned paths (like region=us-east-1/), which Lakewatch detects and maps as partition columns.

Create the datasource

From Datasources, click Add datasource. Under Custom & internal logs, select the storage type that holds your files:

  • Azure Blob Storage: A path in Azure Blob Storage or Azure Data Lake Storage, through a Unity Catalog external location.
  • Unity Catalog volume: A folder in a Unity Catalog volume.

The setup page for that storage type opens. To switch storage types later, use the Source type picker on the setup page. Fill out the following fields:

  • Datasource name: Enter a unique name for the datasource (for example, Cloudflare_WAF).

  • Source location: Point the datasource at the folder that contains your data. The location must be a folder, not an individual file. The fields depend on the storage type:

    • External location and Subfolder (optional): Click Browse to open Browse external location, choose a Unity Catalog external location, and then select its root or a folder within it. You can also enter the location and subfolder directly. Leave Subfolder blank to ingest the whole location.
    • Unity Catalog Volume path: Enter the folder that contains your log files (for example, /Volumes/catalog/schema/volume/logs/), or click Browse volumes.

  • File format: The format Lakewatch uses to parse the source data: JSON, JSONL, Parquet, CSV, XML, or text. Click Infer with Genie to have Genie suggest the format from a sample of your data. When you select CSV, you can also set CSV format options.

  • Ingest historical data: Turn this on to also ingest the files already present at the location. New files are ingested either way. This choice takes effect only at first start.

  • Advanced settings: Expand this optional section to configure:

    • Raw output table name: The name of the raw table that Lakewatch writes source events to before parsers run. Lakewatch creates it in your environment. The name is required.
    • Performance optimized: Use performance-optimized compute for faster ingestion (on) or standard compute (off). Defaults to the workspace ingestion-compute setting.

Set CSV format options

When you set File format to CSV, a CSV format options section appears so you can customize how Lakewatch reads your files. Click Add CSV option to add an option and set its value, or remove an option to use its default. Values are used exactly as entered, including spaces and empty values.

The most common options are Header, Delimiter, Quote, Escape, Multiline, Encoding, Null value, Date format, and Timestamp format. The other standard CSV read options appear under their option names. For the complete list and what each option does, see CSV options.

File discovery and credentials

Lakewatch detects new files automatically. If the source's Unity Catalog external location has file events configured, Lakewatch uses them. Otherwise, it lists the directory. There's no file-notification setting to configure in Lakewatch.

When the source is a Unity Catalog volume or external location, Lakewatch authenticates through it. There's no separate credential or secret-scope entry. Make sure the run-as identity can read the Unity Catalog object and its backing storage credential.

Preview the data

After you set the source location and file format, click Preview data in the Data preview panel to see a sample. The preview shows the raw data as Lakewatch writes it: a data column holding each record and a _metadata column. If you change settings after the preview runs, the preview is marked as outdated. Click Update preview to rerun it.

By default, Lakewatch samples from the configured source. To sample from a specific folder instead, click Edit preview source and select Use sample subfolder. The Preview path shows the inherited source location and subfolder as read-only. Enter a folder within it, or click Browse to choose one. The preview path must be a folder, not a file, and the run-as identity must be able to read it. Unity Catalog volume datasources preview from the volume path and don't offer a separate preview path.

The sample affects only the preview and isn't stored with the datasource. When the datasource runs, it ingests from its configured source.

Time column handling

The time column is critical for clustering, streaming watermarks, and query performance. Lakewatch auto-detects and extracts it during setup, including when the timestamp is nested inside a VARIANT structure. Before writing each row to the raw table, Lakewatch replaces null time values with the ingestion timestamp (lw_meta.ingestion_time), so the raw table's time column is never null; it sets lw_meta.missing_timestamp to true for those rows so you can tell original event times from ingestion-time fallbacks.

Attach parsers and create the datasource

On the same creation form, scroll to the Parsers section:

  1. Attach parsers. Attach one or more parsers to schematize the data: a built-in parser that matches your source or one you authored. If your source delivers enveloped records, also set the Unwrapping option, described in Unwrap enveloped records. For how parser matching works and the columns parsers produce, see Attach parsers.
  2. Create the datasource. Click Create datasource. The datasource starts running immediately on its processing schedule; to stop it, delete the datasource.

You can edit the datasource's other settings (processing schedule, compute, and annotations) any time on its Overview tab after creation. See Configure datasource settings.

Unwrap enveloped records

Some sources deliver several events inside a container rather than one record per line. When the sampled data is a JSON array of events, or events wrapped in an AWS CloudWatch Logs delivery envelope, Lakewatch can unwrap the container so that each event becomes its own row. In the parser step, use the Unwrapping option (the default is No unwrapping) to match how your records are wrapped. Only the record shapes surfaced during setup are supported. Unwrapping is available for cloud storage datasources but not for the Existing table method, which ingests a Unity Catalog table as it is.

  • JSON array: Enter the JSON array key: the root-level key containing the array of events (for example, Records). Nested paths aren't supported.
  • AWS CloudWatch Logs: Turn on Retain CloudWatch Logs headers to keep the original CloudWatch Logs envelope metadata on each unwrapped event. It's off by default.
  • Last updated on