Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Lakewatch is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage workspace-level previews.
Use the Lakewatch API connector to ingest organization audit log events from the OpenAI API Platform. This page covers prerequisites, configuration instructions, the connector schema, and security observables.
Prerequisites
The connector requires:
- Serverless Lakeflow pipelines turned on in your workspace.
- Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
- The Lakeflow Connect for OpenAI preview turned on in your workspace by a workspace admin. See Turn on the connector preview.
- If your workspace's serverless network policy uses restricted access, the source's API domains must be on the policy's Allowed domains list. See Allow the source API domains in your network policy.
- Audit logging turned on for your OpenAI organization. After audit logging is turned on, only OpenAI support can turn it off. See OpenAI's Admin and audit logs API documentation.
- The Organization Owner role in your OpenAI organization to create the Admin API key that authenticates the connector. See OpenAI's Administration APIs guide.
Known limitations
- Slowly changing dimension (SCD) type 2 is not supported.
- The connector supports OpenAI Admin API key authentication. Standard project API keys and OAuth aren't supported.
- The first sync of the
audit_log_entriestable starts at the datasource start time, which must be within the last 365 days. If you don't set a start time, ingestion starts when you create the datasource. Later syncs resume from the last ingestedeffective_atvalue. See Choose how far back to ingest.
Configure OpenAI for ingestion
The connector authenticates to the OpenAI Administration API with an Admin API key, which it sends as a Bearer token. Admin API keys work only with the Administration API endpoints. For the full steps, see Configure authentication to OpenAI.
- Sign in to the OpenAI platform as an Organization Owner.
- Open the Admin keys page for your organization.
- Create an Admin API key, and set its permission to Read Only. Read-only access is sufficient for ingestion. See OpenAI's guide to assigning API key permissions.
- Copy the Admin API key. OpenAI shows the full key only one time.
Tip
When you create a connection through the Lakewatch UI, enter the Admin API Key that you copied in the previous step.
Next steps
After you configure OpenAI, create a datasource with the OpenAI connector in the Lakewatch UI. See API connectors for instructions.
Connector and Lakewatch schemas
Lakewatch ingests OpenAI organization audit log events into a single table named audit_log_entries.
| Source table | OpenAI data | Built-in Lakewatch parser |
|---|---|---|
audit_log_entries |
Organization audit log events, such as API key, user, project, and organization setting changes | openai_audit |
The connector writes each source table directly to the destination configured for its built-in parser. By default, the destination table is named <parser>_<datasource>, where <datasource> is the datasource name. For example, with the default datasource name openai, the table is openai_audit_openai. You can set a different destination during datasource creation. There is no separate raw staging table or later parsing step.
Lakewatch-managed columns
The connector uses the source id field as the non-null primary key and the source effective_at field as the cursor for incremental ingestion. Unlike connectors that synthesize a Lakewatch ID, this connector doesn't add an lw_id or time column. To scope a query or detection rule to a time window, pass the effective_at column to IN_TIMERANGE().
For Lakewatch datasources, the connector also adds:
lw_meta: The datasource and connection identity, parser name and version, and ingestion time.lw_observables: Security observables extracted by the connector.
Connector output table schema
The destination includes the source and event columns below, together with the lw_meta and lw_observables columns described above.
The audit_log_entries table writes one row per audit log event. Each row contains the event's ID, type, and time, together with the actor and the project that the event is scoped to. The data column stores the full original audit log record as VARIANT, including the details object named after the event type, such as project.updated.
Schema
| Column | Data type |
|---|---|
id |
STRING |
type |
STRING |
effective_at |
TIMESTAMP |
actor |
VARIANT |
project |
STRUCT<id: STRING, name: STRING> |
data |
VARIANT |
For field definitions, see OpenAI's audit log object documentation.
Security observables
The connector extracts these fields into lw_observables:
Observables
| Field | Observable |
|---|---|
actor.session.user.id |
actor_id |
actor.session.user.email |
email |
actor.session.ip_address |
ip |
actor.api_key.user.id |
actor_id |
actor.api_key.user.email |
email |
actor.api_key.service_account.id |
actor_id |
The fields in data depend on the event type. The connector scans data and extracts values into lw_observables by matching subfield names. Matching is case-insensitive.
Data observables
data subfield name |
Observable |
|---|---|
email |
email |
allowed_ips |
ip |
principal_id, service_account_id |
actor_id |
role_arn |
aws_arn |
account_id |
aws_account_id |