Okta system logs

Important

Lakewatch is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage workspace-level previews.

Lakewatch provides an API connector to ingest system logs from Okta. This article covers prerequisites, configuration instructions, connector schemas, and security observables.

Prerequisites

The connector requires:

  1. Serverless Lakeflow pipelines turned on in your workspace.
  2. Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
  3. The Lakeflow Connect for Okta System Logs preview turned on in your workspace by a workspace admin. See Turn on the connector preview.
  4. If your workspace's serverless network policy uses restricted access, the source's API domains must be on the policy's Allowed domains list. See Allow the source API domains in your network policy.
  5. An active Okta super administrator, org administrator, or read-only administrator account. Databricks recommends using a read-only administrator account for least-privilege System Log access.
  6. Access to the Okta Admin Console.

Note

An API token has the same permissions as the administrator who creates it. If that administrator's permissions change or the account is deactivated, the token is affected. For details, see Manage Okta API tokens.

Known limitations

  • SCD type 2 is not supported, since Okta system logs are append-only.
  • The connector supports Authentication via API Token only.
  • Historical lookback can't exceed the 90 days Okta retains System Logs. See Choose how far back to ingest.

Configure Okta for ingestion

  1. Sign in to the Okta Admin Console with the administrator account that owns the token.
  2. Go to Security > API.
  3. Click the Tokens tab.
  4. Click Create token.
  5. Enter a name for the token.
  6. For API calls made with this token must originate from, select Any IP.
  7. Click Create token.
  8. Copy the displayed SSWS API token. Okta doesn't display the token again.
  9. Note your Okta organization domain, such as company-name.okta.com.
  10. On the Tokens tab, confirm that the token's Role is Super Admin, Organization Admin, or Read-only Admin.

For more information, see Manage Okta API tokens.

Tip

When creating a connection through the Lakewatch UI, you need the Domain and SSWS API Token gathered above.

Next steps

After completing the configuration above, create a datasource with the Okta connector in the Lakewatch UI. See API connectors for instructions.


Connector and Lakewatch schemas

The connector ingests the following source table from the default source schema:

Source table Okta data Built-in Lakewatch parser
system_logs Okta System Log security events okta_system_log

The connector writes each source table directly to the destination configured for its built-in parser. By default, the destination table is named <parser>_<datasource>, where <datasource> is the datasource name. For example, with the default datasource name okta_system_logs, the table is okta_system_log_okta_system_logs. You can set a different destination during datasource creation. There is no separate raw staging table or later parsing step.

Lakewatch-managed columns

The connector adds the following columns to each output row:

  • lw_id: A non-null string that Lakewatch uses as the primary key. The connector derives a stable, time-sortable ID from the Okta published and uuid fields. The ID supports deduplication across incremental syncs.
  • time: A non-null timestamp parsed from published as an RFC 3339 timestamp. The connector uses the source published value for incremental ordering.

For Lakewatch datasources, the connector also adds:

  • lw_meta: The datasource and connection identity, parser name and version, and ingestion time.
  • lw_observables: Security observables extracted by the connector.

For more information, see Connector output tables.

Connector output table schema

The destination includes the source and event columns below, together with the lw_meta and lw_observables columns described above.

The following schema combines the standard Lakeflow Connect fields with the Lakewatch-only lw_id and time columns. For nested field definitions, see the Okta System Log API reference.

Schema
Column Data type
lw_id STRING
time TIMESTAMP
uuid STRING
published TIMESTAMP
eventType STRING
version STRING
severity STRING
legacyEventType STRING
displayMessage STRING
actor STRUCT<id: STRING, displayName: STRING, type: STRING, alternateId: STRING, detailEntry: MAP<STRING, STRING>>
authenticationContext STRUCT<authenticationProvider: STRING, authenticationStep: INT, credentialProvider: STRING, credentialType: STRING, interface: STRING, externalSessionId: STRING, issuer: STRUCT<id: STRING, type: STRING>>
client STRUCT<id: STRING, ipAddress: STRING, userAgent: STRUCT<rawUserAgent: STRING, os: STRING, browser: STRING>, zone: STRING, device: STRING, geographicalContext: STRUCT<city: STRING, state: STRING, country: STRING, postalCode: STRING, geolocation: STRUCT<lat: DOUBLE, lon: DOUBLE>>>
debugContext STRUCT<debugData: MAP<STRING, STRING>>
outcome STRUCT<result: STRING, reason: STRING>
device STRUCT<id: STRING, name: STRING, os_platform: STRING, os_version: STRING, managed: BOOLEAN, registered: BOOLEAN, device_integrator: STRING, disk_encryption_type: STRING, screen_lock_type: STRING, jailbreak: BOOLEAN, secure_hardware_present: BOOLEAN>
transaction STRUCT<type: STRING, id: STRING, detail: MAP<STRING, STRING>>
request STRUCT<ipChain: ARRAY<STRUCT<ip: STRING, geographicalContext: STRUCT<city: STRING, state: STRING, country: STRING, postalCode: STRING, geolocation: STRUCT<lat: DOUBLE, lon: DOUBLE>>, version: STRING, source: STRING, ipDetails: STRUCT<asNumber: BIGINT, asOrg: STRING, isp: STRING, domain: STRING, ipServiceCategories: ARRAY<STRUCT<isAnonymous: BOOLEAN, operator: STRING, type: STRING>>>>>>
target ARRAY<STRUCT<id: STRING, type: STRING, alternateId: STRING, displayName: STRING, detailEntry: MAP<STRING, STRING>, changeDetails: STRUCT<from: STRUCT<id: STRING, type: STRING, alternateId: STRING, displayName: STRING, detailEntry: MAP<STRING, STRING>>, to: STRUCT<id: STRING, type: STRING, alternateId: STRING, displayName: STRING, detailEntry: MAP<STRING, STRING>>>>>
securityContext STRUCT<asNumber: BIGINT, asOrg: STRING, isp: STRING, domain: STRING, isProxy: BOOLEAN, ipDetails: STRUCT<asNumber: BIGINT, asOrg: STRING, isp: STRING, domain: STRING, ipServiceCategories: ARRAY<STRUCT<isAnonymous: BOOLEAN, operator: STRING, type: STRING>>>, risk: STRUCT<detectionName: STRING, issuer: STRING, level: STRING, previousLevel: STRING, reasons: ARRAY<STRING>>, UserBehaviours: ARRAY<STRUCT<id: STRING, name: STRING, result: STRING>>>

Security observables

The connector extracts the following fields into lw_observables.

Observables
Field Observable
actor.id actor_id
actor.alternateId email
actor.alternateId actor_id
actor.alternateId username
client.ipAddress ip
request.ipChain[].ip ip
request.ipChain[].ipDetails.domain domain
target[].id actor_id
target[].alternateId email
target[].alternateId actor_id
target[].alternateId username
securityContext.domain domain
securityContext.ipDetails.domain domain

Troubleshooting

I'm getting rate-limit errors while ingesting data.

  1. In the Okta Admin Panel, go to Security >> API.
  2. In the Tokens tab, find the API token that you used to Authenticate to Databricks.
  3. Click the name of the Token and in the Token rate limits section and then click Edit and increase the slider to more than 50% of Each API rate limit.