Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Lakewatch is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage workspace-level previews.
Lakewatch provides an API connector to ingest organization audit log events from Atlassian. This article covers prerequisites, configuration instructions, the connector schema, and security observables.
Prerequisites
The connector requires:
- Serverless Lakeflow pipelines turned on in your workspace.
- Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
- The Lakeflow Connect for Atlassian Audit Logs preview turned on in your workspace by a workspace admin. See Turn on the connector preview.
- If your workspace's serverless network policy uses restricted access, the source's API domains must be on the policy's Allowed domains list. See Allow the source API domains in your network policy.
- An Atlassian Guard Standard, Atlassian Guard Premium, or Cloud Enterprise subscription. Without one of these, Atlassian doesn't populate the organization audit log. See Atlassian's audit log activities documentation.
- Organization admin permission to create the API key that authenticates the connector. See Atlassian's Organizations REST API documentation.
Known limitations
- SCD type 2 is not supported.
- The connector supports Atlassian organization API key authentication. OAuth and basic username and password authentication aren't supported.
- Atlassian API keys expire after no more than one year. Create a new key and update the Unity Catalog connection before the current key expires.
- Historical lookback can't exceed 180 days. See Choose how far back to ingest.
Configure Atlassian for ingestion
Atlassian authenticates admin API requests with an API key that the connector sends as a Bearer token. For more information, see Atlassian's guide to managing an organization with the admin APIs.
- Sign in to Atlassian Administration as a user with Organization admin permission. Select your organization if you manage more than one.
- Select Organization settings, then API keys.
- Select API keys with scopes, then select Create API key.
- Enter a Name for the key, and select an expiration date. Keys can't last longer than a year.
- Select the
read:events:adminscope (shown as Read for Events). This is the only scope the connector requires. See Atlassian's Organizations REST API events documentation. - Review the key, then select Create.
- Copy the Organization ID and API key. Atlassian doesn't show these values again. Use them to create a Unity Catalog connection in Databricks.
Note
Atlassian API keys expire after at most one year and don't renew automatically. Before the key expires, create a new organization API key and update the API Key on the existing Unity Catalog connection. Don't create a new connection.
Tip
When you create a connection through the Lakewatch UI, enter the Organization ID and API Key gathered above.
For more information, see Configure authentication to Atlassian.
Next steps
After completing the configuration above, create a datasource with the Atlassian connector in the Lakewatch UI. See API connectors for instructions.
Connector and Lakewatch schemas
The connector ingests the following source table from the default source schema:
| Source table | Atlassian data | Built-in Lakewatch parser |
|---|---|---|
audit_logs |
Organization user, security, and configuration change events | atlassian_audit |
The connector writes each source table directly to the destination configured for its built-in parser. By default, the destination table is named <parser>_<datasource>, where <datasource> is the datasource name. For example, with the default datasource name atlassian_audit_logs, the table is atlassian_audit_atlassian_audit_logs. You can set a different destination during datasource creation. There is no separate raw staging table or later parsing step.
Lakewatch-managed columns
The connector uses the source id field as the non-null primary key and the source time field as the cursor for incremental ingestion. Unlike connectors that synthesize a Lakewatch ID, this connector doesn't add an lw_id column.
For Lakewatch datasources, the connector also adds:
lw_meta: The datasource and connection identity, parser name and version, and ingestion time.lw_observables: Security observables extracted by the connector.
For more information, see Connector output tables.
Connector output table schema
The destination includes the source and event columns below, together with the lw_meta and lw_observables columns described above.
The following schema includes the Atlassian organization audit event fields. For field definitions, see the Atlassian Audit Logs connector reference.
Schema
| Column | Data type |
|---|---|
id |
STRING |
type |
STRING |
time |
TIMESTAMP |
action |
STRING |
actor |
STRUCT<id: STRING, name: STRING, email: STRING, picture: STRING, links: STRUCT<self: STRING, alt: STRING>> |
context |
VARIANT |
container |
VARIANT |
location |
STRUCT<ip: STRING, geo: STRING, countryName: STRING, regionName: STRING, city: STRING> |
message |
STRUCT<content: STRING, format: STRING> |
links |
STRUCT<self: STRING, alt: STRING> |
relations |
VARIANT |
Security observables
The connector extracts the following fields into lw_observables.
Observables
| Field | Observable |
|---|---|
actor.id |
actor_id |
actor.name |
username |
actor.email |
email |
actor.links.self |
url |
actor.links.alt |
url |
location.ip |
ip |
links.self |
url |
links.alt |
url |