Atlassian audit logs

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

Lakewatch provides an API connector to ingest organization audit log events from Atlassian. This article covers prerequisites, configuration instructions, the connector schema, and Lakewatch parser observables.

Prerequisites

The connector requires:

  1. Serverless Lakeflow Spark Declarative Pipelines turned on in your workspace.
  2. Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
  3. An Atlassian Guard Standard, Atlassian Guard Premium, or Cloud Enterprise subscription. Without one of these, Atlassian doesn't populate the organization audit log. See Atlassian's audit log activities documentation.
  4. Organization admin permission to create the API key that authenticates the connector. See Atlassian's Organizations REST API documentation.

Known limitations

  • SCD type 2 is not supported.
  • The connector supports Atlassian organization API key authentication. OAuth and basic username and password authentication aren't supported.
  • Atlassian API keys expire after no more than one year. Create a new key and update the Unity Catalog connection before the current key expires.
  • Historical lookback can't exceed 180 days. See Choose how far back to ingest.

Configure Atlassian for ingestion

Atlassian authenticates admin API requests with an API key that the connector sends as a Bearer token. For more information, see Atlassian's guide to managing an organization with the admin APIs.

  1. Sign in to Atlassian Administration as a user with Organization admin permission. Select your organization if you manage more than one.
  2. Select Organization settings, then API keys.
  3. Select API keys with scopes, then select Create API key.
  4. Enter a Name for the key, and select an expiration date. Keys can't last longer than a year.
  5. Select the read:events:admin scope (shown as Read for Events). This is the only scope the connector requires. See Atlassian's Organizations REST API events documentation.
  6. Review the key, then select Create.
  7. Copy the Organization ID and API key. Atlassian doesn't show these values again. Use them to create a Unity Catalog connection in Databricks.

Note

Atlassian API keys expire after at most one year and don't renew automatically. Before the key expires, create a new organization API key and update the API Key on the existing Unity Catalog connection. Don't create a new connection.

Tip

When you create a connection through the Lakewatch UI, enter the Organization ID and API Key gathered above.

For more information, see Configure authentication to Atlassian.

Next steps

After completing the configuration above, create a datasource with the Atlassian connector in the Lakewatch UI. See API connectors for instructions.

Connector and Lakewatch schemas

The connector ingests the following source table from the default source schema:

Source table Atlassian data Built-in Lakewatch parser
audit_logs Organization user, security, and configuration change events atlassian_audit

The connector writes the source table to the datasource's raw destination. The built-in parser writes a separate parsed table named after the parser, unless you set a parsed destination when you configure the datasource.

Lakewatch-managed columns

The connector uses the source id field as the non-null primary key and the source time field as the cursor for incremental ingestion. Unlike connectors that synthesize a Lakewatch ID, this connector doesn't add an lw_id column. The parser assigns the parsed time column.

For Lakewatch-created pipelines, the connector also adds these columns to the raw table, and the parsed table carries them forward:

  • lw_meta: Standard Lakewatch processing metadata.
  • lw_observables: Values extracted from fields marked as security observables in the parser.

For more information, see Parser reference.

Raw connector table schema

The following schema includes the Atlassian organization audit event fields. For field definitions, see the Atlassian Audit Logs connector reference.

Schema
Column Data type
id STRING
type STRING
time TIMESTAMP
action STRING
actor STRUCT<id: STRING, name: STRING, email: STRING, picture: STRING, links: STRUCT<self: STRING, alt: STRING>>
context VARIANT
container VARIANT
location STRUCT<ip: STRING, geo: STRING, countryName: STRING, regionName: STRING, city: STRING>
message STRUCT<content: STRING, format: STRING>
links STRUCT<self: STRING, alt: STRING>
relations VARIANT

Parser observables

The atlassian_audit parser writes nested objects as VARIANT and extracts the following fields into lw_observables.

Observables
Field Observable
actor.id actor_id
actor.name username
actor.email email
actor.links.self url
actor.links.alt url
location.ip ip
links.self url
links.alt url
  • Last updated on