Create and manage datasources

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

This page explains the components of a datasource and teaches you how to create and manage datasources in the UI. A datasource is a configured pipeline that ingests raw security data and writes it to schematized, governed Unity Catalog tables ready for security analysis. A datasource combines an ingestion method with one or more parsers.

Ingestion methods

An ingestion method defines how a Lakewatch datasource pulls raw security data in, based on where that data lives. Each method has its own creation flow, but they all land in a raw layer that parsers then normalize.

Lakewatch offers three ingestion methods:

• Cloud storage: Ingest files from a supported object store (Amazon S3, Azure Data Lake Storage, or Google Cloud Storage) through a Unity Catalog volume or external location. Lakewatch reads the source with Auto Loader, picking up new files as they arrive.
• Existing table: Ingest a Unity Catalog Delta table or view that already exists in your workspace. Lakewatch reads the source directly and runs the attached parsers over it, rather than copying the data.
• API connectors: Ingest security data from SaaS applications through managed connectors governed by Unity Catalog connections. All API connectors are paired with pre-built parsers.

Note

An ingestion method is for parsing and normalizing a source into Lakewatch parsed tables. If a Unity Catalog table is already query-ready, you don't need to ingest it: you can run queries and write detection rules against it directly.

Parsers

Parsers schematize the ingested data into parsed tables. Each parser describes the output columns of a log source and writes a parsed table with the Lakewatch-managed lw_id, lw_meta, and lw_observables columns.

Select one or more parsers during the datasource creation flow. You can select a built-in parser that matches your source, customize a built-in parser, or create a new parser from scratch.

When the datasource runs, each incoming event is matched against the attached parsers in the order they are declared and lands in the first parser it matches. An event matches when its required fields are present, its typed fields convert to the declared types, and all validators pass. For the matching rules and the full parser format, see Parser reference.

Note

When you preview parsed output while creating a datasource, lw_meta.datasource_name appears as null. A datasource isn't named until you create it, so Lakewatch populates datasource_name only after the datasource is created and runs.

Create a new datasource

Go to Datasources and click Add datasource. You'll see a picker that asks "What data do you want to onboard?". Select a built-in connector card for a SaaS source, or select Custom datasource and choose a Source type. The ingestion methods are:

• Cloud storage: files in Amazon S3, Azure Data Lake Storage, or Google Cloud Storage.
• Existing table: a Unity Catalog Delta table or view already in your workspace.
• API connectors: SaaS applications, via managed connectors.

Manage existing datasources

To manage a datasource, open it from Datasources. The details page has three tabs:

• Overview: The datasource's description, recent runs, connection, ingestion stats, attached parsers, and its editable settings (processing schedule, run-as identity, compute, and annotations). Edit a setting by selecting it, adjusting the values, and clicking Done; add or change parsers from the Parsers section.
• System errors: The system errors raised for this datasource, each listed with its title, state, severity, assignee, event count, and creation time. Search by title, description, or comments, and filter by state. Select an error's title to open its details page.
• Permissions: The Unity Catalog privileges granted on the datasource. Use Grant and Revoke to manage access.

From the overflow menu (⋮) you can edit the datasource as YAML and delete the datasource. Deleting is the only way to stop a running datasource; it permanently removes the datasource, but the tables it already wrote aren't deleted.

Edit datasource settings

On the Overview tab, you can edit the following settings:

• Processing schedule: How often the datasource's ingestion job runs. New datasources default to every 10 minutes.
  • At least every [unit of time]: Runs on a specified interval.
  • Exactly at [time]: Runs on an exact schedule. Build it with the visual picker, or select Show cron syntax and enter a Quartz cron expression. An invalid expression shows Enter a valid Quartz cron expression and prevents saving.
  • Continuous: Ingests new data on a loop as it arrives for a ~30-minute run, then immediately re-triggers a fresh run.
  • API connectors: Not applicable. API connector datasources run on a managed schedule that you can't change.
• Compute: How the datasource's ingestion is grouped onto compute, under Job grouping:
  • Automatic (default): Ingestion may be combined with other jobs processing the same data to optimize compute costs.
  • Dedicated: Ingestion runs with dedicated resources.
  • Job group: Ingestion runs together with other jobs in a named group you enter. Changing groups resets checkpoints, so processing resumes from the current time.
• Annotations: Key-value pairs applied as tags to the datasource.

Set the run-as identity

Every datasource has a run-as identity, the user or service principal that Lakewatch uses to read the source, use any configured credential, and write the raw and parsed tables. This identity is set during creation and can be edited after creation from the details page.

• Default: If you don't choose an identity when you create the datasource, the run-as identity defaults to you.
• Choosing a service principal: To run as a service principal, you must have the Service Principal User role on that service principal. A role granted through group membership is active even though the service principal's details page shows only directly assigned roles. See Manage service principals.
• Reassigning: You can change the run-as identity when you edit the datasource. Lakewatch re-validates source and credential access as the new identity when you save the change.

Because Lakewatch validates and runs the datasource as this identity, the run-as identity must hold every grant the datasource needs:

• Read the source: The necessary access depends on the ingestion method. For example, the run-as identity needs READ FILES for an external location and READ VOLUME on a volume for cloud storage. See your ingestion method's page for the specific access it requires.
• Use a configured credential: The run-as identity needs the ACCESS privilege on a Unity Catalog service credential, or read access to the AWS secret scope, when one is configured.
• Write the output tables: The run-as identity needs USE CATALOG, USE SCHEMA, and CREATE TABLE on each schema where Lakewatch creates the raw and parsed tables.

Lakewatch doesn't automatically grant the run-as identity access to your destination schemas or tables. Grant this access before attaching this identity to the datasource.

Change parsers on a datasource

How you change parsers depends on the ingestion method:

• API connector datasources don't support changing parsers after creation. To use different parsers, recreate the datasource. Changing a parser would rebuild the managed ingestion pipeline and recreate the datasource's tables, so parser editing is disabled on these datasources.
• Cloud storage and existing-table datasources let you change parsers inline. Open the datasource, make your changes, and click Save.

For cloud storage and existing-table datasources:

• Add a parser: Attach another built-in or authored parser. Incoming events are matched against the attached parsers in declaration order, so a newly added parser takes effect for events that don't match an earlier parser. The new parser starts from a recent point and reprocesses raw data from there; the parsers already attached keep their place and aren't reprocessed. See Attach parsers.
• Remove a parser: Detach a parser you no longer need. New events stop being written to that parser's parsed table, and the previously written parsed table isn't deleted. Drop it separately in Unity Catalog if you no longer need the data. Because removing a parser changes how incoming events are routed, the remaining parsers reprocess recent raw data, which can produce duplicate rows in their parsed tables.
• Reorder parsers: Parsers match in declaration order, so reordering them changes routing the same way adding or removing does. The datasource reprocesses recent raw data, which can produce duplicate rows in the affected parsed tables.
• Change a parser's version: A datasource pins a specific parser version, so it isn't affected by later edits to the parser. To pick up a newer version, select it when editing the datasource. A version change doesn't reprocess existing data; the new version applies to events ingested after you save the change.
• Change the parsed destination: You can change the table a parser writes to. Choose a destination name that doesn't collide with an existing table. Data already written to the old table isn't moved.

Troubleshoot datasource errors

The datasource overview page summarizes each datasource's recent Run history, its Errors count, Event Stats (records processed in the last 7 days and last 24 hours), and a 7-Day Trend. To see the underlying run details, open the datasource and click View job to reach the Databricks job run, where you can inspect run history, per-run status, and logs.

When a datasource's runs fail, Lakewatch records the failure as a system event attached to a system error, which identifies the affected datasource and includes the error message. Use the system error together with View job to see which phase failed (for example: a pipeline error). For the full list of datasource-related triggers, see What triggers system events?.

If the datasource's run-as identity loses read access to the source, runs fail and the datasource's Errors count rises.

To recover:

1. Restore the run-as identity's access to the source. If you keep the same run-as identity, no configuration change is needed. The next scheduled run picks up the restored access.
2. If you instead reassign the datasource to a different run-as identity, or change another setting, edit the datasource inline and click Save.
3. Confirm recovery in View job or the system errors. Once successful runs resume, the errors clear.