API connectors

Important

Lakewatch is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage workspace-level previews.

API connectors are managed connectors that pull security data from popular enterprise applications directly into Lakewatch through Unity Catalog connections. When you create a datasource with an API connector, Lakewatch automatically creates and manages the underlying ingestion pipeline that replicates data from the external source.

For the connectors available and their source-specific setup guides, see Supported logs.

All API connectors in Lakewatch share the following characteristics:

  • Managed ingestion: Lakewatch runs the connector on the schedule you set and handles retries and monitoring with no user-created jobs required. Connector datasources support interval and cron schedules, but not continuous scheduling.
  • Serverless infrastructure: Connectors run on serverless Lakeflow pipelines.
  • Append-only design: New records are appended to the configured destination tables. In-place updates and row-level change tracking aren't supported.

Connector output tables

The connector writes each source table directly to the destination configured for its built-in parser. Lakewatch doesn't create a separate raw staging table or apply a second parsing step. The destination uses the connector's schema, including its typed source fields and Lakewatch columns.

During ingestion, the connector adds lw_meta with the datasource and connection identity, parser name and version, and ingestion time. It also populates lw_observables with security observables from the source records. The attached built-in parser sets each source table's destination.

For each connector's output schema and observables, see its page under Supported logs.

Prerequisites

Before you set up an API connector, verify that you meet the following requirements:

  1. Serverless pipelines: Turn on serverless Lakeflow pipelines in your workspace.
  2. Permissions: Your workspace must be enabled for Unity Catalog, and you must have privileges to create or use a Unity Catalog connection. See Connection permissions.
  3. Connector preview: For a connector listed in Turn on the connector preview, a workspace admin must turn on its preview from the Previews page. Until the preview is on, that connector doesn't appear in the datasource picker. Zendesk is generally available and doesn't require this opt-in.
  4. Network access: If your workspace's serverless network policy uses restricted access, an account admin must add the source's API domains to the policy's Allowed domains list. See Allow the source API domains in your network policy.

Some connectors have additional source-specific prerequisites. For details, see the individual source pages under Supported logs.

Turn on the connector preview

The connectors listed below are in Beta. To see one of these connectors in the Add datasource picker, a workspace admin must turn on its preview individually. If searching the picker for one of these connectors returns "No datasources match this search or category," check that its preview is on. Zendesk is generally available and doesn't require a connector-preview opt-in.

To turn on a connector preview:

  1. As a workspace admin, select your username in the top bar of the workspace, then select Previews.
  2. Turn on the preview for your connector. The following table lists each connector's preview name.
Connector Preview name
1Password Lakeflow Connect for 1Password Event Logs
Akamai WAF Lakeflow Connect for Akamai WAF
Atlassian Lakeflow Connect for Atlassian Audit Logs
Box Lakeflow Connect for Box Audit Logs
CrowdStrike Lakeflow Connect for CrowdStrike
GitHub Lakeflow Connect for GitHub
Google Workspace Lakeflow Connect for Google Workspace
Microsoft 365 Lakeflow Connect for Microsoft 365 Unified Audit Logs
Microsoft Graph Lakeflow Connect for Microsoft Graph Logs
Okta Lakeflow Connect for Okta System Logs
OpenAI Lakeflow Connect for OpenAI
Wiz Lakeflow Connect for Wiz Audit Logs
Workday Lakeflow Connect for Workday Activity Logging

See Manage Azure Databricks previews.

Connection permissions

API connectors authenticate to the source application through a Unity Catalog connection, a securable object governed by Unity Catalog privileges. Creating a connector-backed datasource requires both these connection privileges and the Unity Catalog datasource privileges.

When you create a connector-backed datasource, you set up this connection under Connection to the source in the first step of the flow: either select an existing connection from the list, or select Create connection to open a source-specific dialog (for example, Create a connection to Workday) where you name the connection and enter the source's credentials. The privilege you need depends on which option you use:

  • To create a new connection: You need CREATE CONNECTION on the metastore.
  • To use an existing connection: You need USE CONNECTION on that connection, or on the metastore. Connections you can't use are unavailable in the list and show You do not have permission to use this connection.

A metastore admin can grant these privileges, and the owner of a connection can grant USE CONNECTION on it. For details, see Unity Catalog privileges reference.

Allow the source API domains in your network policy

API connectors run on serverless compute, so the workspace's serverless network policy controls which hosts they can reach. If the policy uses Restricted access, the connector can reach only the domains on the policy's Allowed domains list. If a source's API domain isn't on the list, the connector can't resolve the source host and the pipeline update fails.

To let a connector reach its source, an account admin must add the connector's domains to the policy's Allowed domains list. See Set egress rules.

Connector Domains to allow
1Password The host of your Base URL (for example, events.1password.com).
Akamai WAF Your Host (for example, akab-xxxxxxxx.luna.akamaiapis.net).
Atlassian api.atlassian.com
Box api.box.com
CrowdStrike The host of your Base URL (for example, api.us-2.crowdstrike.com), and the event stream hosts that the CrowdStrike Event Streams API returns. To find the event stream hosts, check the denial logs.
GitHub api.github.com. For GitHub Enterprise Server, your Enterprise domain.
Google Workspace admin.googleapis.com
Microsoft 365 The Office 365 Management API host for your subscription plan: manage.office.com (Enterprise), manage-gcc.office.com (Government Community Cloud, or GCC), manage.office365.us (GCC High), or manage.protection.apps.mil (Department of Defense).
Microsoft Graph The Microsoft Graph host for your Microsoft cloud setting: graph.microsoft.com (commercial, including GCC), graph.microsoft.us (gcc_high), or dod-graph.microsoft.us (dod).
Okta Your Domain (for example, your-company.okta.com).
OpenAI api.openai.com
Wiz api.<tenant-data-center>.app.wiz.io, where <tenant-data-center> is your Tenant Data Center (for example, api.us17.app.wiz.io).
Workday The host of your Instance URL.
Zendesk The host of your Subdomain (for example, your-company.zendesk.com).

Most changes to the Allowed domains list reach serverless compute in ten minutes. See Apply network policy changes. To find domains that the policy blocked, see Check denial logs.

Create a datasource with a connector

Creating a connector-backed datasource involves two main steps:

  1. Configure the source application: Set up authentication credentials (API tokens, OAuth clients, and so on) in the source system. Each connector has source-specific instructions. See its page under Supported logs.

  2. Create a datasource in the Lakewatch UI: From Datasources, select Add datasource and select a connector from the picker. The datasource creation form opens with these sections:

    1. Name: Lakewatch prefills a name, such as okta_system_logs. You can keep it or enter a different one. The name can't contain spaces, periods, or forward slashes.
    2. Connection to the source: Create a new connection or select an existing one using the credentials from step 1. Each connector requires credentials specific to the source system (for example, an API token or OAuth client). For privileges, see Connection permissions.
    3. Execution: Set the run-as identity. If more than one folder is available to you, also choose the folder that holds the datasource's tables. If the connector supports historical lookback, Ingest historical data also appears here. See Choose how far back to ingest.
    4. Advanced settings: Optionally, set a Schedule: At least every for an interval, or Exactly at for a cron schedule. The default is at least every 10 minutes. See Configure the ingestion schedule. You can also turn on Performance optimized to use performance-optimized compute for faster ingestion, or leave it off for standard compute. It defaults to the workspace ingestion-compute setting.

    The Parsers card follows these sections. See Attach and configure parsers.

Lakewatch automatically handles pipeline creation, scheduling, and monitoring.

Configure the ingestion schedule

In Advanced settings, configure the Schedule:

  • At least every: Enter an interval and select its unit.
  • Exactly at: Configure an exact schedule with the visual picker, or select Show cron syntax to enter a Quartz cron expression.

API connectors don't support continuous scheduling. To change the schedule after creating the datasource, use the Configuration tab. See Manage the datasource after creation.

Choose how far back to ingest

On the datasource creation form, look for the Ingest historical data option in the Execution section. It's cleared by default, so Lakewatch starts at datasource creation and then continues from the last ingested event. Select it to backfill a window of source history. After you select it, helper text shows the maximum lookback for the connector. This setting takes effect only when the datasource first starts and can't be changed after creation.

Note

Larger historical windows can take longer to ingest and can increase ingestion cost.

If the connector doesn't support historical lookback, Ingest historical data doesn't appear. For source retention limits, see the connector's page under Supported logs.

Attach and configure parsers

When you create a connector-backed datasource, Lakewatch selects all of the connector's parsers. The connector requires every one of them, so they stay selected: you can't deselect a parser or add another one. To review a parser, select its name. For each parser, you can select its version and set its parsed destination.

After the datasource is created, you can't add or remove parsers. You can still change each parser's version and destination table. To use a different set of parsers, recreate the datasource. See Parsers and Change parsers on a datasource.

By default, each source table's destination is a table in your environment named <parser>_<datasource>, where <datasource> is the datasource name. For example, the Okta System Log parser writes to okta_system_log_okta_system_logs when the datasource name is okta_system_logs. To send the connector output to a different table, select the parser's name and change Destination table name. The table uses the datasource's catalog and schema, so only the table name can change. Use a destination name that doesn't collide with an existing table.

Manage the datasource after creation

The datasource starts running as soon as you create it. Open its Configuration tab to edit the processing schedule, compute mode, run-as identity, or annotations. The Overview tab shows the datasource's status and summary. To monitor runs or delete the datasource, see Create and manage datasources.

To edit its schedule:

  1. Open the datasource from Datasources and select the Configuration tab.
  2. Click the edit icon next to Schedule to open Processing schedule.
  3. Under Run, select At least every and set an interval, or select Exactly at and configure a cron schedule.
  4. Click Save.

The Schedule field shows the new interval or cron expression.