Existing table

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

Use the Existing table ingestion method to ingest a Unity Catalog Delta table or view that already exists in your workspace. Lakewatch reads the source table directly and runs your attached parsers over it to produce parsed tables, rather than copying the data.

Note

You don't have to ingest a table to use it in Lakewatch. If a Unity Catalog table is already query-ready, you can run queries and write detection rules against it directly, without creating a datasource. Use the Existing table method when you want Lakewatch to run parsers over the source and produce normalized parsed tables, with the Lakewatch-managed lw_id, lw_meta, and lw_observables columns.

This page is the complete guide to creating a datasource from an existing table. For managing a datasource after it exists (editing settings, changing parsers, and monitoring) see Create and manage datasources.

The source can be a managed or external Delta table, a streaming table, a Delta Sharing table, or a view that reads from a single upstream table. Views that combine multiple tables (for example, with UNION ALL) aren't supported.

Create the datasource

From Datasources, click Add datasource and select Custom datasource. On the creation page, set the Source type to Unity Catalog table.

Fill out the following fields on the creation form:

• Datasource name: Enter a unique name for the datasource.
• Source table or view: Enter the fully qualified name of the existing Unity Catalog table or view, in catalog.schema.table format, or click the browse icon to select one. After you select a table, a Data preview shows its data so you can confirm you picked the right one.

Set the run-as identity for the datasource. Expand Advanced settings to configure:

• Only process new rows: Controls whether Lakewatch backfills the rows already in the source table. It's selected by default, so Lakewatch ingests only the rows added after the datasource is created. Clear it to also ingest the rows already present (a backfill that takes longer on a large source) then continue with every new commit. This setting takes effect only at first start and can't be changed after creation. Delta Sharing tables don't support incremental reads, so this option doesn't apply to them. Lakewatch processes all rows.
• Performance optimized: Use performance-optimized compute for faster ingestion (on) or standard compute (off). Defaults to the workspace ingestion-compute setting.

You can't change the source table after the datasource is created. To change it, recreate the datasource.

Note

Queries and detection rules that read the datasource perform best when the source table is liquid-clustered on the column used for time. If it isn't, Lakewatch displays a warning during configuration. If that column is a top-level TIMESTAMP or TIMESTAMP_NTZ column, add clustering directly on the source table by running ALTER TABLE catalog.schema.my_source_table CLUSTER BY (my_time_column) in a notebook or SQL editor, where my_time_column is the source table's time column. See Enable liquid clustering on existing tables.

If the time column isn't a top-level column, isn't a TIMESTAMP or TIMESTAMP_NTZ type, or is derived by a SQL expression, clustering can't be added directly.

Attach parsers and create the datasource

1. Attach parsers. Attach one or more parsers to schematize the ingested data: a built-in parser that matches your source or one you authored. For how parser matching works and the columns parsers produce, see Attach parsers.
2. Create the datasource. Click Create datasource. The datasource starts running immediately on its processing schedule; to stop it, delete the datasource.

You can edit the datasource's other settings (processing schedule, compute, and annotations) any time on its Overview tab after creation. See Configure datasource settings.

Ingest AI Gateway trace logs

AI Gateway Unified Tracing records model and tool requests (including guardrail and policy decisions, token usage, and requester identity) to a Unity Catalog Delta table (unity_gateway_otel_spans) that you own. Because it's a standard Unity Catalog table, you can bring it into Lakewatch with the Existing table method and then query and run detection rules against it.

1. Enable tracing. Set up the trace table for your metastore in the AI Gateway UI. See Unified trace table.
2. Create the datasource. Follow the steps above, using the trace table's fully qualified name as the Source table or view.
3. Grant access. The datasource's run-as identity must be able to read the trace table and write to the destination schema. A detection rule you write against the data runs under its own run-as identity, which needs the same access.

You can also query the trace table directly on the query page or from a detection rule, without creating a datasource.