Okta system logs

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

Lakewatch provides an API connector to ingest system logs from Okta. This article covers prerequisites, configuration instructions, connector schemas, and Lakewatch parser observables.

Prerequisites

The connector requires:

  1. Serverless Lakeflow Spark Declarative Pipelines turned on in your workspace.
  2. Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
  3. An active Okta super administrator, org administrator, or read-only administrator account. Databricks recommends using a read-only administrator account for least-privilege System Log access.
  4. Access to the Okta Admin Console.

Note

An API token has the same permissions as the administrator who creates it. If that administrator's permissions change or the account is deactivated, the token is affected. For details, see Manage Okta API tokens.

Known limitations

  • SCD type 2 is not supported, since Okta system logs are append-only.
  • The connector supports Authentication via API Token only.
  • Historical lookback can't exceed the 90 days Okta retains System Logs. See Choose how far back to ingest.

Configure Okta for ingestion

  1. Sign in to the Okta Admin Console with the administrator account that owns the token.
  2. Go to Security > API.
  3. Click the Tokens tab.
  4. Click Create token.
  5. Enter a name for the token.
  6. For API calls made with this token must originate from, select Any IP.
  7. Click Create token.
  8. Copy the displayed SSWS API token. Okta doesn't display the token again.
  9. Note your Okta organization domain, such as company-name.okta.com.
  10. On the Tokens tab, confirm that the token's Role is Super Admin, Organization Admin, or Read-only Admin.

For more information, see Manage Okta API tokens.

Tip

When creating a connection through the Lakewatch UI, you need the Domain and SSWS API Token gathered above.

Next steps

After completing the configuration above, create a datasource with the Okta connector in the Lakewatch UI. See API connectors for instructions.

Connector and Lakewatch schemas

The connector ingests the following source table from the default source schema:

Source table Okta data Built-in Lakewatch parser
system_logs Okta System Log security events okta_system_log

The connector writes the source table to the datasource's raw destination. The built-in parser writes a separate parsed table named after the parser, unless you set a parsed destination when you configure the datasource.

Lakewatch-managed columns

The connector adds the following columns to each raw source row:

  • lw_id: A non-null string that Lakewatch uses as the primary key. The connector derives a stable, time-sortable ID from the Okta published and uuid fields. The ID supports deduplication across incremental syncs.
  • time: A non-null timestamp parsed from published as an RFC 3339 timestamp. The connector uses the source published value for incremental ordering.

The parsed table carries lw_id forward. The parser also assigns the parsed time column and adds:

  • lw_meta: Standard Lakewatch processing metadata.
  • lw_observables: Values extracted from fields marked as security observables in the parser.

For more information, see Parser reference.

Raw connector table schema

The following schema combines the standard Lakeflow Connect fields with the Lakewatch-only lw_id and time columns. For nested field definitions, see the Okta System Log API reference.

Schema
Column Data type
lw_id STRING
time TIMESTAMP
uuid STRING
published TIMESTAMP
eventType STRING
version STRING
severity STRING
legacyEventType STRING
displayMessage STRING
actor STRUCT<id: STRING, displayName: STRING, type: STRING, alternateId: STRING, detailEntry: MAP<STRING, STRING>>
authenticationContext STRUCT<authenticationProvider: STRING, authenticationStep: INT, credentialProvider: STRING, credentialType: STRING, interface: STRING, externalSessionId: STRING, issuer: STRUCT<id: STRING, type: STRING>>
client STRUCT<id: STRING, ipAddress: STRING, userAgent: STRUCT<rawUserAgent: STRING, os: STRING, browser: STRING>, zone: STRING, device: STRING, geographicalContext: STRUCT<city: STRING, state: STRING, country: STRING, postalCode: STRING, geolocation: STRUCT<lat: DOUBLE, lon: DOUBLE>>>
debugContext STRUCT<debugData: MAP<STRING, STRING>>
outcome STRUCT<result: STRING, reason: STRING>
device STRUCT<id: STRING, name: STRING, os_platform: STRING, os_version: STRING, managed: BOOLEAN, registered: BOOLEAN, device_integrator: STRING, disk_encryption_type: STRING, screen_lock_type: STRING, jailbreak: BOOLEAN, secure_hardware_present: BOOLEAN>
transaction STRUCT<type: STRING, id: STRING, detail: MAP<STRING, STRING>>
request STRUCT<ipChain: ARRAY<STRUCT<ip: STRING, geographicalContext: STRUCT<city: STRING, state: STRING, country: STRING, postalCode: STRING, geolocation: STRUCT<lat: DOUBLE, lon: DOUBLE>>, version: STRING, source: STRING, ipDetails: STRUCT<asNumber: BIGINT, asOrg: STRING, isp: STRING, domain: STRING, ipServiceCategories: ARRAY<STRUCT<isAnonymous: BOOLEAN, operator: STRING, type: STRING>>>>>>
target ARRAY<STRUCT<id: STRING, type: STRING, alternateId: STRING, displayName: STRING, detailEntry: MAP<STRING, STRING>, changeDetails: STRUCT<from: STRUCT<id: STRING, type: STRING, alternateId: STRING, displayName: STRING, detailEntry: MAP<STRING, STRING>>, to: STRUCT<id: STRING, type: STRING, alternateId: STRING, displayName: STRING, detailEntry: MAP<STRING, STRING>>>>>
securityContext STRUCT<asNumber: BIGINT, asOrg: STRING, isp: STRING, domain: STRING, isProxy: BOOLEAN, ipDetails: STRUCT<asNumber: BIGINT, asOrg: STRING, isp: STRING, domain: STRING, ipServiceCategories: ARRAY<STRUCT<isAnonymous: BOOLEAN, operator: STRING, type: STRING>>>, risk: STRUCT<detectionName: STRING, issuer: STRING, level: STRING, previousLevel: STRING, reasons: ARRAY<STRING>>, UserBehaviours: ARRAY<STRUCT<id: STRING, name: STRING, result: STRING>>>

Parser observables

The okta_system_log parser writes nested objects as VARIANT and extracts the following fields into lw_observables.

Observables
Field Observable
actor.id actor_id
actor.alternateId email
actor.alternateId actor_id
actor.alternateId username
client.ipAddress ip
request.ipChain[].ip ip
request.ipChain[].ipDetails.domain domain
target[].id actor_id
target[].alternateId email
target[].alternateId actor_id
target[].alternateId username
securityContext.domain domain
securityContext.ipDetails.domain domain

Troubleshooting

I'm getting rate-limit errors while ingesting data.

  1. In the Okta Admin Panel, go to Security >> API.
  2. In the Tokens tab, find the API token that you used to Authenticate to Databricks.
  3. Click the name of the Token and in the Token rate limits section and then click Edit and increase the slider to more than 50% of Each API rate limit.
  • Last updated on