OpenAI audit logs

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

Use the Lakewatch API connector to ingest organization audit log events from the OpenAI API Platform. This page covers prerequisites, configuration instructions, the connector schema, and Lakewatch parser observables.

Prerequisites

The connector requires:

  1. Serverless Lakeflow pipelines turned on in your workspace.
  2. Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
  3. Audit logging turned on for your OpenAI organization. After audit logging is turned on, only OpenAI support can turn it off. See OpenAI's Admin and audit logs API documentation.
  4. The Organization Owner role in your OpenAI organization to create the Admin API key that authenticates the connector. See OpenAI's Administration APIs guide.

Known limitations

  • Slowly changing dimension (SCD) type 2 is not supported.
  • The connector supports OpenAI Admin API key authentication. Standard project API keys and OAuth aren't supported.
  • Historical lookback can't exceed 365 days. See Choose how far back to ingest.

Configure OpenAI for ingestion

The connector authenticates to the OpenAI Administration API with an Admin API key, which it sends as a Bearer token. Admin API keys work only with the Administration API endpoints. For the full steps, see Configure authentication to OpenAI.

  1. Sign in to the OpenAI platform as an Organization Owner.
  2. Open the Admin keys page for your organization.
  3. Create an Admin API key, and set its permission to Read Only. Read-only access is sufficient for ingestion. See OpenAI's guide to assigning API key permissions.
  4. Copy the Admin API key. OpenAI shows the full key only one time.

Tip

When you create a connection through the Lakewatch UI, enter the Admin API Key that you copied in the previous step.

Next steps

After you configure OpenAI, create a datasource with the OpenAI connector in the Lakewatch UI. See API connectors for instructions.

Connector and Lakewatch schemas

Lakewatch ingests OpenAI organization audit log events into a single table named audit_log_entries.

Source table OpenAI data Built-in Lakewatch parser
audit_log_entries Organization audit log events, such as API key, user, project, and organization setting changes openai_audit

The connector writes the source table to the datasource's raw destination. The built-in parser writes a separate parsed table named after the parser, unless you set a parsed destination when you configure the datasource.

Lakewatch-managed columns

The connector uses the source id field as the non-null primary key and the source effective_at field as the cursor for incremental ingestion. Unlike connectors that synthesize a Lakewatch ID, this connector doesn't add an lw_id column. The parser assigns the parsed time column from the source effective_at field.

For Lakewatch-created pipelines, the connector also adds these columns to the raw table, and the parsed table carries them forward:

  • lw_meta: Standard Lakewatch processing metadata.
  • lw_observables: Values extracted from fields marked as security observables in the parser.

See Parser reference.

Raw connector table schema

The audit_log_entries table writes one row per audit log event. Each row contains the event's ID, type, and time, together with the actor and the project that the event is scoped to. The data column stores the full original audit log record as VARIANT, including the details object named after the event type, such as project.updated.

Schema
Column Data type
id STRING
type STRING
effective_at TIMESTAMP
actor VARIANT
project STRUCT<id: STRING, name: STRING>
data VARIANT

For field definitions, see OpenAI's audit log object documentation.

Parser observables

The openai_audit parser writes nested objects as VARIANT and extracts these fields into lw_observables:

Observables
Field Observable
actor.session.user.id actor_id
actor.session.user.email email
actor.session.ip_address ip
actor.api_key.user.id actor_id
actor.api_key.user.email email
actor.api_key.service_account.id actor_id

The fields in data depend on the event type. The parser scans data and extracts values into lw_observables by matching subfield names. Matching is case-insensitive.

Data observables
data subfield name Observable
email email
allowed_ips ip
principal_id, service_account_id actor_id
role_arn aws_arn
account_id aws_account_id
  • Last updated on