API connectors

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

API connectors are managed connectors that pull security data from popular enterprise applications directly into Lakewatch through Unity Catalog connections. When you create a datasource with an API connector, Lakewatch automatically creates and manages the underlying ingestion pipeline that replicates data from the external source.

For the connectors available and their source-specific setup guides, see Supported logs.

All API connectors in Lakewatch share the following characteristics:

• Managed ingestion: Lakewatch runs the connector on a managed schedule and handles retries and monitoring with no user-created jobs required. You can't change the schedule on a connector datasource, and continuous scheduling isn't supported.
• Serverless infrastructure: Connectors run on serverless Lakeflow Spark Declarative Pipelines (SDP).
• Append-only design: New records are appended to the datasource's raw table; in-place updates and row-level change tracking are not supported.
• Multi-cloud support: All connectors are compatible with Databricks on AWS, Azure, or GCP.

Prerequisites

Before setting up an API connector, ensure you have:

1. Serverless pipelines: Turn on serverless Lakeflow SDP in your workspace.
2. Permissions: Unity Catalog must be enabled, and you must have privileges to create or use a Unity Catalog connection. See Connection permissions.

Each connector may have additional source-specific prerequisites. See the individual source pages under Supported logs for details.

Connection permissions

API connectors authenticate to the source application through a Unity Catalog connection, a securable object governed by Unity Catalog privileges. Creating a connector-backed datasource requires both these connection privileges and the Unity Catalog datasource privileges.

When you create a connector-backed datasource, you set up this connection under Connection to the source in the first step of the flow: either select an existing connection from the list, or click + Create connection to open a source-specific dialog (for example, Create a connection to Workday) where you name the connection and enter the source's credentials. The privilege you need depends on which:

• To create a new connection: You need CREATE CONNECTION on the metastore.
• To use an existing connection: You need USE CONNECTION on that connection, or on the metastore.

A metastore admin can grant these privileges, and the owner of a connection can grant USE CONNECTION on it. For details, see Unity Catalog privileges and securable objects.

Create a datasource with a connector

Creating a connector-backed datasource involves two main steps:

1. Configure the source application: Set up authentication credentials (API tokens, OAuth clients, and so on) in the source system. Each connector has source-specific instructions; see its page under Supported logs.
2. Create a datasource in the Lakewatch UI. From Datasources, click Add datasource and select a connector from the picker. The datasource creation form opens with these sections:
   1. Configure a Unity Catalog connection: The form begins with the connection section. Create a new connection or select an existing one using the credentials from step 1. Each connector requires credentials specific to the source system (for example, an API token or OAuth client). For privileges, see Connection permissions.
   2. Name: Enter a name for the datasource. Lakewatch creates its tables in your current environment.
   3. Historical data: Optionally configure how far back Lakewatch ingests. See Choose how far back to ingest.
   4. Execution and parsers: Set the run-as identity and attach the parsers that schematize the ingested data.
   5. Advanced settings: Optionally set Performance optimized: use performance-optimized compute for faster ingestion (on) or standard compute (off). Defaults to the workspace ingestion-compute setting.

Lakewatch automatically handles pipeline creation, scheduling, and monitoring.

Choose how far back to ingest

On the datasource creation form, look for the Ingest historical data toggle in the Execution section. It's cleared by default, so Lakewatch starts at datasource creation and then continues from the last ingested event. Select it to backfill a window of source history. After you select it, helper text shows the maximum lookback for the connector. This setting takes effect only at first start and can't be changed after creation.

Note

Larger historical windows can take longer to ingest and may increase ingestion cost.

If the connector does not support historical lookback, Ingest historical data does not appear. For source retention limits, see the connector's page under Supported logs.

Attach and configure parsers

When you create a connector-backed datasource, Lakewatch automatically selects all built-in parsers that match the connector's source data. You can adjust this selection by deselecting or adding parsers, but at least one parser must remain selected. You can't change the parsers after the datasource is created; to use different parsers, recreate the datasource. See Attach parsers and Change parsers on a datasource.

Parsed destination. By default, each parser writes to a table in your environment, with the table name derived from the parser name. To send a parser's parsed output to a different table, set a parsed destination (target table) for that parser. Choose a destination name that doesn't collide with an existing table.

Manage the datasource after creation

The datasource starts running as soon as you create it. Its schedule is managed and can't be changed, but you can edit its other settings (compute and annotations) any time on the Overview tab, and monitor runs or delete it as described in Create and manage datasources.