OpenAI audit logs

Important

Lakewatch is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage workspace-level previews.

Use the Lakewatch API connector to ingest organization audit log events from the OpenAI API Platform. This page covers prerequisites, configuration instructions, the connector schema, and security observables.

Prerequisites

The connector requires:

  1. Serverless Lakeflow pipelines turned on in your workspace.
  2. Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
  3. The Lakeflow Connect for OpenAI preview turned on in your workspace by a workspace admin. See Turn on the connector preview.
  4. If your workspace's serverless network policy uses restricted access, the source's API domains must be on the policy's Allowed domains list. See Allow the source API domains in your network policy.
  5. Audit logging turned on for your OpenAI organization. After audit logging is turned on, only OpenAI support can turn it off. See OpenAI's Admin and audit logs API documentation.
  6. The Organization Owner role in your OpenAI organization to create the Admin API key that authenticates the connector. See OpenAI's Administration APIs guide.

Known limitations

  • Slowly changing dimension (SCD) type 2 is not supported.
  • The connector supports OpenAI Admin API key authentication. Standard project API keys and OAuth aren't supported.
  • The first sync of the audit_log_entries table starts at the datasource start time, which must be within the last 365 days. If you don't set a start time, ingestion starts when you create the datasource. Later syncs resume from the last ingested effective_at value. See Choose how far back to ingest.

Configure OpenAI for ingestion

The connector authenticates to the OpenAI Administration API with an Admin API key, which it sends as a Bearer token. Admin API keys work only with the Administration API endpoints. For the full steps, see Configure authentication to OpenAI.

  1. Sign in to the OpenAI platform as an Organization Owner.
  2. Open the Admin keys page for your organization.
  3. Create an Admin API key, and set its permission to Read Only. Read-only access is sufficient for ingestion. See OpenAI's guide to assigning API key permissions.
  4. Copy the Admin API key. OpenAI shows the full key only one time.

Tip

When you create a connection through the Lakewatch UI, enter the Admin API Key that you copied in the previous step.

Next steps

After you configure OpenAI, create a datasource with the OpenAI connector in the Lakewatch UI. See API connectors for instructions.


Connector and Lakewatch schemas

Lakewatch ingests OpenAI organization audit log events into a single table named audit_log_entries.

Source table OpenAI data Built-in Lakewatch parser
audit_log_entries Organization audit log events, such as API key, user, project, and organization setting changes openai_audit

The connector writes each source table directly to the destination configured for its built-in parser. By default, the destination table is named <parser>_<datasource>, where <datasource> is the datasource name. For example, with the default datasource name openai, the table is openai_audit_openai. You can set a different destination during datasource creation. There is no separate raw staging table or later parsing step.

Lakewatch-managed columns

The connector uses the source id field as the non-null primary key and the source effective_at field as the cursor for incremental ingestion. Unlike connectors that synthesize a Lakewatch ID, this connector doesn't add an lw_id or time column. To scope a query or detection rule to a time window, pass the effective_at column to IN_TIMERANGE().

For Lakewatch datasources, the connector also adds:

  • lw_meta: The datasource and connection identity, parser name and version, and ingestion time.
  • lw_observables: Security observables extracted by the connector.

See Connector output tables.

Connector output table schema

The destination includes the source and event columns below, together with the lw_meta and lw_observables columns described above.

The audit_log_entries table writes one row per audit log event. Each row contains the event's ID, type, and time, together with the actor and the project that the event is scoped to. The data column stores the full original audit log record as VARIANT, including the details object named after the event type, such as project.updated.

Schema
Column Data type
id STRING
type STRING
effective_at TIMESTAMP
actor VARIANT
project STRUCT<id: STRING, name: STRING>
data VARIANT

For field definitions, see OpenAI's audit log object documentation.

Security observables

The connector extracts these fields into lw_observables:

Observables
Field Observable
actor.session.user.id actor_id
actor.session.user.email email
actor.session.ip_address ip
actor.api_key.user.id actor_id
actor.api_key.user.email email
actor.api_key.service_account.id actor_id

The fields in data depend on the event type. The connector scans data and extracts values into lw_observables by matching subfield names. Matching is case-insensitive.

Data observables
data subfield name Observable
email email
allowed_ips ip
principal_id, service_account_id actor_id
role_arn aws_arn
account_id aws_account_id