Create and manage datasources

Important

Lakewatch is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage workspace-level previews.

A Lakewatch datasource is a configured pipeline that ingests raw security data and writes it to schematized, governed Unity Catalog tables ready for security analysis. A datasource combines an ingestion method with one or more parsers. This page explains both components and how to create and manage datasources in the UI.

Ingestion methods

An ingestion method defines how a Lakewatch datasource pulls security data in, based on where that data lives. Each method has its own creation flow and output path. API connectors write schematized output directly to their configured destination tables.

Lakewatch offers three ingestion methods:

  • Cloud storage: Ingest files from cloud object storage through a Unity Catalog volume or external location. Lakewatch reads the source with Auto Loader, picking up new files as they arrive.
  • Existing table: Ingest a Unity Catalog Delta table or view that already exists in your workspace. Lakewatch reads the source directly and runs the attached parsers over it, rather than copying the data.
  • API connectors: Ingest security data from software as a service (SaaS) applications through managed connectors governed by Unity Catalog connections. All API connectors are paired with pre-built parsers.

Note

For cloud storage and existing-table sources, an ingestion method parses and normalizes the source into Lakewatch parsed tables. If a Unity Catalog table is already query-ready, you don't need to ingest it. Run queries and write detection rules against it directly.

Parsers

For cloud storage and existing-table datasources, parsers schematize the ingested data into parsed tables. Each parser describes the output columns of a log source and writes a parsed table with the Lakewatch-managed lw_id, lw_meta, and lw_observables columns.

For API connector datasources, Lakewatch selects the complete set of built-in parsers available for the connector. Every parser in this set is required. For cloud storage and existing-table datasources, select one or more parsers during creation. Select a built-in parser that matches your source, or create a custom parser from scratch or from a built-in parser.

When a cloud storage or existing-table datasource runs, each incoming event lands in the first attached parser it matches, in declaration order, so each event is written to at most one parsed table. The parser definitions do this routing, so you don't configure filters on the datasource. For the matching rules, see Classification.

For API connector datasources, the connector writes its schema directly to the configured built-in parser destinations and adds Lakewatch metadata and observables during ingestion. See Connector output tables.

When you create a datasource, Lakewatch checks the output table for each parser. If a table name is missing, repeats another parser's table, or the run-as identity can't use the table, Lakewatch shows Resolve parser issues before continuing. with a Resolve issues button. Select it to open Resolve destination issues, change the table names, and select Apply changes. An existing table must be readable and writable by the run-as identity. A new table must be creatable in the datasource's schema, which requires USE CATALOG, USE SCHEMA, and CREATE TABLE for the run-as identity. See Set the run-as identity.

Note

When you preview parsed output while creating a datasource, lw_meta.datasource_name appears as null. A datasource isn't named until you create it, so Lakewatch populates datasource_name only after the datasource is created and runs.

Create a new datasource

Go to Datasources and select Add datasource. The Add new datasource page offers two ways to start:

  • Managed connectors: Select the log source you want to ingest. Sources tagged Lakeflow Connect connect to a service's API. Sources tagged Files, tables & queues, such as Cloudflare audit logs, read the service's logs from storage and come with matching built-in parsers. The list starts with a few popular sources. Select Browse all to search every source. Sources marked Coming soon can't be selected.
  • Custom & internal logs: Select where your data lives, such as object storage, a message queue, a Unity Catalog table, or a volume, and then attach parsers.

The ingestion methods are:

When you create a datasource, choose its folder under Execution. The folder field appears only when more than one folder is available to you for datasources. The datasource's name can't contain spaces, periods, or forward slashes.

To create a datasource, you must have CREATE_DATASOURCE on the folder where you create it. The Datasource manager, Folder admin, and Lakewatch admin permission groups include this privilege. See Permission groups. If you lack it, the form shows You don't appear to have permission to create datasources in this schema. Choose another folder, or ask a folder admin for one of these permission groups. To create an API connector datasource, you must also have CREATE CONNECTION on the metastore to create a new connection, or USE CONNECTION on an existing one. These permissions are separate from the datasource's run-as identity, which must have its own access to read the source and write the datasource's tables.

Manage existing datasources

To manage a datasource, open it from Datasources. The details page has these tabs:

  • Overview: Ingestion activity (events ingested in the last 7 days, compared with the preceding 7 days), the status of the latest run, and a summary of the datasource's processing settings, source, raw landing table, owner, and run-as identity.
  • Parsers: The attached parsers, each with its pinned version, destination table, and row counts for the last 24 hours and last 7 days, plus each parser's share of rows over the last 7 days. Select a destination table to open a SQL draft of it in Lakewatch Query, or copy its path. Rows that matched no attached parser appear under Unmatched rows. When a newer version of a parser is available, its row menu offers Upgrade version. To change parsers, select Edit parsers, as described in Change parsers on a datasource.
  • Configuration: The datasource's editable settings, described in the next section.
  • Run history: The latest 25 runs, including runs in progress, with each run's start time, state, and details. Links to the underlying job runs are shown only to workspace admins and the datasource's run-as identity.
  • System errors: The latest 50 active system errors raised for this datasource. Select an error to open its details page, or select View all system errors to see every system error.

Access to a datasource comes from the permissions on its folder. To change who can view or manage datasources, edit the folder's permissions. See Environments and folders.

From the Datasource actions menu (⋮), select Edit as YAML or Delete datasource. Deleting is the only way to stop a running datasource. It permanently removes the datasource, but the tables it already wrote aren't deleted.

Edit datasource settings

Edit a datasource's settings on its Configuration tab. Description and Compute mode save as soon as you change them. To change the schedule, select the edit icon next to Schedule, and then select Save in the Processing schedule dialog. Annotations work the same way: select Add annotation or an annotation's edit icon, and then select Save. A datasource's source, name, schema, and owner can't be changed after creation.

To edit settings, you need the MANAGE privilege on the datasource, and the datasource must run as you or as a service principal you're eligible to use. If it runs as another user, the page shows a banner. To change the run-as identity first, select Change Run as in the banner, as described in Set the run-as identity.

Lakewatch validates and saves each change using the SQL warehouse selected in the top bar. If no warehouse is selected, the tab prompts you to select one before you can save.

If you edit the datasource as YAML, Lakewatch shows Identity, source, and health fields are read-only. Changes to these fields will not be saved. To change the source, create a new datasource.

If someone else changes the datasource while you have an unsaved YAML draft, Lakewatch shows This datasource changed elsewhere. Your draft is preserved; cancel and reload before saving. Select Cancel, confirm the discard, reopen the datasource, and make your edits again.

The Configuration tab has the following settings:

  • Description: A description of the datasource.
  • Schedule: How often the datasource's ingestion job runs. New datasources default to every 10 minutes.
    • At least every [unit of time]: Runs on a specified interval.
    • Exactly at [time]: Runs on an exact schedule. Build it with the visual picker, or select Show cron syntax and enter a Quartz cron expression. An invalid expression shows Enter a valid Quartz cron expression and prevents saving.
    • Continuous: Ingests new data as it arrives during a run of about 30 minutes, then immediately starts a new run. API connector datasources don't support Continuous. Use At least every or Exactly at instead.
  • Compute mode: Whether ingestion uses performance-optimized or standard compute. Select Standard or Performance optimized, or select Catalog default to use the workspace ingestion-compute setting.
  • Compute group: How the ingestion job is grouped onto compute. Select Edit compute, then choose Automatic (the default), Dedicated, or Shared job group and enter the group name. Select Save in Compute resources.
  • Annotations: Key-value pairs applied as tags to the datasource.

For cloud storage datasources that read JSON or JSONL, you can also change how enveloped records are unwrapped after creation. On the Overview tab, select Edit unwrapping in the source details. Changes apply only to newly ingested data. See Unwrap enveloped records.

If a run is in progress when you save a change, Lakewatch doesn't stop it. That run finishes with the previous settings, and the next run uses the new ones. For example, if you change Schedule from Continuous to At least every [unit of time], the current continuous run keeps going until it completes, and then the new schedule takes effect.

Set the run-as identity

Every datasource has a run-as identity, the user or service principal that Lakewatch uses to read the source, use any configured credential, and write the datasource's output tables. This identity is set during creation and can be edited after creation from the details page.

  • Default: If you don't select an identity when you create the datasource, the run-as identity defaults to you.
  • Choosing a service principal: To run as a service principal, you must have the Service Principal User role on that service principal. A role granted through group membership is active even though the service principal's details page shows only directly assigned roles. See Manage service principals.
  • Reassigning: You can change the run-as identity when you edit the datasource. Lakewatch re-validates source and credential access as the new identity when you save the change.

Because Lakewatch validates and runs the datasource as this identity, the run-as identity must hold every grant the datasource needs:

  • Read the source: The necessary access depends on the ingestion method. For example, the run-as identity needs READ FILES for an external location and READ VOLUME on a volume for cloud storage. For the specific access each method requires, see its page under Create a new datasource.
  • Use a configured credential: The run-as identity needs the ACCESS privilege on a Unity Catalog service credential, or read access to the secret scope that holds the credentials, when one is configured.
  • Write the output tables: The run-as identity needs USE CATALOG, USE SCHEMA, and CREATE TABLE on each schema where Lakewatch creates the datasource's output tables.

Lakewatch doesn't automatically grant the run-as identity access to your destination schemas or tables. Grant this access before attaching this identity to the datasource.

Change parsers on a datasource

To change parsers, open the datasource, select the Parsers tab, and select Edit parsers. Make your changes, then select Save. Which changes you can make depends on the ingestion method.

If you see Select a warehouse and give every parser a valid output table. Lakeflow Connect requires at least one parser., select a SQL warehouse in the top bar.

When you create an API connector datasource, Lakewatch attaches every parser for that connector, and you can't add or remove parsers, either during creation or afterward. You can still change a parser's version or its parsed destination. To use a different set of parsers, recreate the datasource.

For cloud storage and existing-table datasources:

  • Add a parser: Attach another built-in or authored parser. Incoming events are matched against the attached parsers in declaration order, so a newly added parser takes effect for events that don't match an earlier parser. The new parser starts from a recent point and reprocesses raw data from there. The parsers already attached keep their place and aren't reprocessed. See Parsers.
  • Remove a parser: Detach a parser you no longer need. New events stop being written to that parser's parsed table, and the previously written parsed table isn't deleted. Drop it separately in Unity Catalog if you no longer need the data. Because removing a parser changes how incoming events are routed, the remaining parsers reprocess recent raw data, which can produce duplicate rows in their parsed tables.
  • Reorder parsers: Parsers match in declaration order, so reordering them changes routing the same way adding or removing does. The datasource reprocesses recent raw data, which can produce duplicate rows in the affected parsed tables.

For all datasources, including API connector datasources:

  • Change a parser's version: A datasource pins a specific parser version, so it isn't affected by later edits to the parser. To pick up a newer version, select it when editing the datasource, or select Upgrade version in the parser's row menu on the Parsers tab. If the new version changes which events the parser matches, the datasource reprocesses recent raw data the same way adding or removing a parser does, which can produce duplicate rows. Otherwise, the new version applies to events ingested after you save the change, and existing parsed data isn't reprocessed.
  • Change the parsed destination: You can change the table a parser writes to. The table uses the datasource's catalog and schema, so only the table name can change. Use a destination name that doesn't collide with an existing table. Data already written to the old table isn't moved.

Troubleshoot datasource errors

The Datasources list shows each datasource's recent Run history, its Errors count, and its 7-day events trend. To see the underlying run details, open the datasource's Run history tab and select View job runs to go to the job, where you can inspect run history, per-run status, and logs. This link is shown only to workspace admins and the datasource's run-as identity.

When a datasource run fails, Lakewatch records the failure on a system error, which identifies the affected datasource and includes the error message. Use the system error together with View job runs to see which phase failed (for example, a pipeline error). For the full list of datasource-related triggers, see What triggers system errors?.

If the datasource's run-as identity loses read access to the source, runs fail and the datasource's Errors count rises.

To recover, complete the following steps:

  1. Restore the run-as identity's access to the source. If you keep the same run-as identity, no configuration change is needed. The next scheduled run picks up the restored access.
  2. If you instead reassign the datasource to a different run-as identity, or change another setting, make the change on the Configuration tab.
  3. Confirm recovery on the Run history tab. The system error stays open after runs succeed again, so change its state to Resolved when you've confirmed the fix.

If you can open a datasource but can't edit it, the page shows You need the MANAGE privilege to edit this datasource. Ask an owner to grant it.