Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Lakewatch is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage workspace-level previews.
Use the Cloud storage ingestion method to ingest files from a supported cloud object store. Lakewatch reads the source with Auto Loader.
This page describes how to create a cloud storage datasource. To manage a datasource after it exists, see Create and manage datasources.
Lakewatch ingests from Azure Data Lake Storage through a Unity Catalog volume or external location.
Prerequisites
Before you create the datasource, make sure you meet the following requirements:
- Source in Unity Catalog: If the source is a Unity Catalog volume or external location, set it up in Unity Catalog before you create the datasource. For setup, see Manage external locations.
- Run-as access to the source: The datasource's run-as identity must be able to read the source: the
READ FILESprivilege on the external location, orREAD VOLUMEon the volume. Grant this on the external location or volume in Unity Catalog, not on the identity itself. For instructions, see Manage external locations or Privileges for Unity Catalog volumes. Lakewatch ingests as the run-as identity, not as your user account, so your individual account-level access to the data isn't sufficient. Access the identity inherits from a parent catalog or schema, or from a group, also satisfies this requirement.
Prepare your data for ingestion
Each Lakewatch datasource ingests from a single directory and reads all files using one file format. Before creating a datasource, organize your cloud storage so that each directory you plan to ingest contains:
- One file format: All files must be the same format (for example, all JSON, all Parquet, or all CSV). Files in different formats cause parse errors.
- One source type: Files should come from the same source system and share a compatible schema (for example, all AWS CloudTrail logs, or all CrowdStrike detection events). Minor schema variations within the same format are handled automatically, but mixing fundamentally different schemas in one directory isn't supported.
The following example shows a directory structure with one datasource for each top-level directory:
s3://my-security-bucket/
├── cloudtrail/ ← Datasource 1 (JSON)
│ ├── 2025/01/01/
│ │ ├── event1.json.gz
│ │ └── event2.json.gz
│ └── 2025/01/02/
│ └── event3.json.gz
├── vpc-flow-logs/ ← Datasource 2 (Parquet)
│ ├── region=us-east-1/
│ │ └── flowlogs.parquet
│ └── region=us-west-2/
│ └── flowlogs.parquet
└── waf-logs/ ← Datasource 3 (JSON)
└── waf-events.json
Each top-level directory becomes a separate Lakewatch datasource. Subdirectories are traversed automatically, including Hive-style partitioned paths (like region=us-east-1/). Lakewatch reads only the contents of the files, so path segments such as region=us-east-1 aren't added as columns.
Create the datasource
From Datasources, select Add datasource. Under Custom & internal logs, select the storage type that holds your files:
- Azure Blob Storage: A path in Azure Blob Storage or Azure Data Lake Storage, through a Unity Catalog external location.
- Unity Catalog volume: A folder in a Unity Catalog volume.
The setup page for that storage type opens. Fill out the following fields:
Name: Enter a unique name for the datasource (for example,
Cloudflare_WAF).Source location: Point the datasource at the folder that contains your data. The location must be a folder, not an individual file. The fields depend on the storage type:
- External location and Subfolder (optional): Select a Unity Catalog external location, and then select Browse to choose its root or a folder within it, and select Choose folder. You can also enter the location and subfolder directly. Leave Subfolder blank to ingest the whole location.
- Unity Catalog Volume path: Enter the folder that contains your log files (for example,
/Volumes/catalog/schema/volume/logs/), or select Browse volumes.
File format: The format Lakewatch uses to parse the source data: JSON, JSONL, Parquet, CSV, or XML. The AWS CloudTrail and Azure Event Hubs presets extract each event from the JSON records array. Select Infer with Genie to have Genie suggest the format from a sample of your data. When you select CSV, you can also set CSV format options.
Ingest historical data: Turn this on to also ingest the files already present at the location. It's off by default, so the datasource ingests only files that arrive after you create it. When it's on, the first run lists and ingests every existing file in the location, which can take a long time for a location with many files. This setting takes effect only when the datasource first starts.
Advanced settings: Expand this optional section to configure:
- Schedule: How often the ingestion job runs. Defaults to at least every 10 minutes. You can also change it later on the Configuration tab.
- Raw output table name: The name of the raw table that Lakewatch writes source events to before parsers run. Lakewatch creates it in the datasource's folder. The name is required.
- Performance optimized: Use performance-optimized compute for faster ingestion (on) or standard compute (off). Defaults to the workspace ingestion-compute setting.
Set CSV format options
When you set File format to CSV, a CSV format options section appears so you can customize how Lakewatch reads your files. Select Add CSV option to add an option and set its value, or remove an option to use its default. Values are used exactly as entered, including spaces and empty values.
The most common options are Header, Delimiter, Quote, Escape, Multiline, Encoding, Null value, Date format, and Timestamp format. The other standard CSV read options appear under their option names. For the complete list and what each option does, see CSV options.
File discovery and credentials
Lakewatch detects new files automatically. If the source's Unity Catalog external location has file events configured, Lakewatch uses them. Otherwise, it lists the directory. There's no file-notification setting to configure in Lakewatch.
Lakewatch checks the external location for file events when you create the datasource and keeps using that discovery method. If you turn off file events on the external location afterward, the datasource doesn't switch to directory listing. Its runs fail with CF_MANAGED_FILE_EVENTS_ENDPOINT_NOT_FOUND, and Lakewatch records the failure as a system error. To recover, turn file events back on for the external location, or recreate the datasource.
When the source is a Unity Catalog volume or external location, Lakewatch authenticates through it. There's no separate credential or secret-scope entry. Make sure the run-as identity can read the Unity Catalog object and its backing storage credential.
Preview the data
After you set the source location and file format, select Test parsers in the Data preview panel to run a sample through the attached parsers. The preview shows the raw data as Lakewatch writes it, with a data column holding each record and a _metadata column, alongside the parsed output. If you change settings after the preview runs, the preview is marked Outdated. Select Test parsers again to rerun it. The preview runs only on a sample, so a clean preview doesn't guarantee that every event in the source matches a parser.
By default, Lakewatch samples from the configured source. To sample from a specific folder instead, select Edit preview source, and then select Use sample subfolder. The Preview path shows the inherited source location and subfolder as read-only. Enter a folder within it, or select Browse to find one. The preview path must be a folder, not a file, and the run-as identity must be able to read it. Unity Catalog volume datasources preview from the volume path and don't offer a separate preview path.
The sample affects only the preview and isn't stored with the datasource. When the datasource runs, it ingests from its configured source.
Time column handling
The time column is critical for clustering, streaming watermarks, and query performance. Lakewatch auto-detects and extracts it during setup, including when the timestamp is nested inside a VARIANT structure. The raw table stores each source record as read, with a _metadata column. When Lakewatch writes the parsed tables, it takes time from the event time that the parser defines. If every configured time source is null, it uses the ingestion timestamp (lw_meta.ingestion_time) instead, so each row in a parsed table has a time value.
Attach parsers and create the datasource
On the same creation form, scroll to the Parsers section:
- Attach parsers. Attach one or more parsers to schematize the data: a built-in parser that matches your source or one you authored. If your source delivers enveloped records, also set Unwrapping, described in Unwrap enveloped records. For how parser matching works and the columns parsers produce, see Parsers.
- Create the datasource. Select Create datasource. The datasource starts running immediately on its processing schedule. To stop it, delete the datasource.
After creation, edit the datasource's other settings (schedule, compute, and annotations) at any time on its Configuration tab. See Edit datasource settings.
Unwrap enveloped records
Some sources deliver several events inside a container rather than one record per line. Lakewatch can unwrap the container so that each event becomes its own row. Next to File format, set Unwrapping (the default is None) to match how your records are wrapped. Only the record shapes surfaced during setup are supported. Unwrapping is available for cloud storage datasources but not for the Existing table method, which ingests a Unity Catalog table as it is.
For a JSON array of events, select JSON array and enter the JSON array path: the root-level key containing the array of events (for example, Records). Nested paths aren't supported. After creation, the field is labeled JSON array key.