Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Lakewatch is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage workspace-level previews.
Lakewatch provides an API connector to ingest system logs from Okta. This article covers prerequisites, configuration instructions, connector schemas, and security observables.
Prerequisites
The connector requires:
- Serverless Lakeflow pipelines turned on in your workspace.
- Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
- The Lakeflow Connect for Okta System Logs preview turned on in your workspace by a workspace admin. See Turn on the connector preview.
- If your workspace's serverless network policy uses restricted access, the source's API domains must be on the policy's Allowed domains list. See Allow the source API domains in your network policy.
- An active Okta super administrator, org administrator, or read-only administrator account. Databricks recommends using a read-only administrator account for least-privilege System Log access.
- Access to the Okta Admin Console.
Note
An API token has the same permissions as the administrator who creates it. If that administrator's permissions change or the account is deactivated, the token is affected. For details, see Manage Okta API tokens.
Known limitations
- SCD type 2 is not supported, since Okta system logs are append-only.
- The connector supports Authentication via API Token only.
- Historical lookback can't exceed the 90 days Okta retains System Logs. See Choose how far back to ingest.
Configure Okta for ingestion
- Sign in to the Okta Admin Console with the administrator account that owns the token.
- Go to Security > API.
- Click the Tokens tab.
- Click Create token.
- Enter a name for the token.
- For API calls made with this token must originate from, select Any IP.
- Click Create token.
- Copy the displayed SSWS API token. Okta doesn't display the token again.
- Note your Okta organization domain, such as
company-name.okta.com. - On the Tokens tab, confirm that the token's Role is Super Admin, Organization Admin, or Read-only Admin.
For more information, see Manage Okta API tokens.
Tip
When creating a connection through the Lakewatch UI, you need the Domain and SSWS API Token gathered above.
Next steps
After completing the configuration above, create a datasource with the Okta connector in the Lakewatch UI. See API connectors for instructions.
Connector and Lakewatch schemas
The connector ingests the following source table from the default source schema:
| Source table | Okta data | Built-in Lakewatch parser |
|---|---|---|
system_logs |
Okta System Log security events | okta_system_log |
The connector writes each source table directly to the destination configured for its built-in parser. By default, the destination table is named <parser>_<datasource>, where <datasource> is the datasource name. For example, with the default datasource name okta_system_logs, the table is okta_system_log_okta_system_logs. You can set a different destination during datasource creation. There is no separate raw staging table or later parsing step.
Lakewatch-managed columns
The connector adds the following columns to each output row:
lw_id: A non-null string that Lakewatch uses as the primary key. The connector derives a stable, time-sortable ID from the Oktapublishedanduuidfields. The ID supports deduplication across incremental syncs.time: A non-null timestamp parsed frompublishedas an RFC 3339 timestamp. The connector uses the sourcepublishedvalue for incremental ordering.
For Lakewatch datasources, the connector also adds:
lw_meta: The datasource and connection identity, parser name and version, and ingestion time.lw_observables: Security observables extracted by the connector.
For more information, see Connector output tables.
Connector output table schema
The destination includes the source and event columns below, together with the lw_meta and lw_observables columns described above.
The following schema combines the standard Lakeflow Connect fields with the Lakewatch-only lw_id and time columns. For nested field definitions, see the Okta System Log API reference.
Schema
| Column | Data type |
|---|---|
lw_id |
STRING |
time |
TIMESTAMP |
uuid |
STRING |
published |
TIMESTAMP |
eventType |
STRING |
version |
STRING |
severity |
STRING |
legacyEventType |
STRING |
displayMessage |
STRING |
actor |
STRUCT<id: STRING, displayName: STRING, type: STRING, alternateId: STRING, detailEntry: MAP<STRING, STRING>> |
authenticationContext |
STRUCT<authenticationProvider: STRING, authenticationStep: INT, credentialProvider: STRING, credentialType: STRING, interface: STRING, externalSessionId: STRING, issuer: STRUCT<id: STRING, type: STRING>> |
client |
STRUCT<id: STRING, ipAddress: STRING, userAgent: STRUCT<rawUserAgent: STRING, os: STRING, browser: STRING>, zone: STRING, device: STRING, geographicalContext: STRUCT<city: STRING, state: STRING, country: STRING, postalCode: STRING, geolocation: STRUCT<lat: DOUBLE, lon: DOUBLE>>> |
debugContext |
STRUCT<debugData: MAP<STRING, STRING>> |
outcome |
STRUCT<result: STRING, reason: STRING> |
device |
STRUCT<id: STRING, name: STRING, os_platform: STRING, os_version: STRING, managed: BOOLEAN, registered: BOOLEAN, device_integrator: STRING, disk_encryption_type: STRING, screen_lock_type: STRING, jailbreak: BOOLEAN, secure_hardware_present: BOOLEAN> |
transaction |
STRUCT<type: STRING, id: STRING, detail: MAP<STRING, STRING>> |
request |
STRUCT<ipChain: ARRAY<STRUCT<ip: STRING, geographicalContext: STRUCT<city: STRING, state: STRING, country: STRING, postalCode: STRING, geolocation: STRUCT<lat: DOUBLE, lon: DOUBLE>>, version: STRING, source: STRING, ipDetails: STRUCT<asNumber: BIGINT, asOrg: STRING, isp: STRING, domain: STRING, ipServiceCategories: ARRAY<STRUCT<isAnonymous: BOOLEAN, operator: STRING, type: STRING>>>>>> |
target |
ARRAY<STRUCT<id: STRING, type: STRING, alternateId: STRING, displayName: STRING, detailEntry: MAP<STRING, STRING>, changeDetails: STRUCT<from: STRUCT<id: STRING, type: STRING, alternateId: STRING, displayName: STRING, detailEntry: MAP<STRING, STRING>>, to: STRUCT<id: STRING, type: STRING, alternateId: STRING, displayName: STRING, detailEntry: MAP<STRING, STRING>>>>> |
securityContext |
STRUCT<asNumber: BIGINT, asOrg: STRING, isp: STRING, domain: STRING, isProxy: BOOLEAN, ipDetails: STRUCT<asNumber: BIGINT, asOrg: STRING, isp: STRING, domain: STRING, ipServiceCategories: ARRAY<STRUCT<isAnonymous: BOOLEAN, operator: STRING, type: STRING>>>, risk: STRUCT<detectionName: STRING, issuer: STRING, level: STRING, previousLevel: STRING, reasons: ARRAY<STRING>>, UserBehaviours: ARRAY<STRUCT<id: STRING, name: STRING, result: STRING>>> |
Security observables
The connector extracts the following fields into lw_observables.
Observables
| Field | Observable |
|---|---|
actor.id |
actor_id |
actor.alternateId |
email |
actor.alternateId |
actor_id |
actor.alternateId |
username |
client.ipAddress |
ip |
request.ipChain[].ip |
ip |
request.ipChain[].ipDetails.domain |
domain |
target[].id |
actor_id |
target[].alternateId |
email |
target[].alternateId |
actor_id |
target[].alternateId |
username |
securityContext.domain |
domain |
securityContext.ipDetails.domain |
domain |
Troubleshooting
I'm getting rate-limit errors while ingesting data.
- In the Okta Admin Panel, go to Security >> API.
- In the Tokens tab, find the API token that you used to Authenticate to Databricks.
- Click the name of the Token and in the Token rate limits section and then click Edit and increase the slider to more than 50% of Each API rate limit.