GitHub audit logs

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

Lakewatch provides an API connector to ingest organization audit logs from GitHub. This article covers prerequisites, configuration instructions, the connector schema, and Lakewatch parser observables.

Prerequisites

The connector requires:

  1. Serverless Lakeflow Spark Declarative Pipelines turned on in your workspace.
  2. Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
  3. A GitHub account with access to the organization that you want to ingest audit logs from.
  4. Permission to create OAuth Apps in your GitHub account's developer settings.

Known limitations

  • The audit_logs table is available only for organization accounts.
  • On the GitHub Free plan, audit log history on github.com is limited to 90 days.
  • Historical lookback can't exceed 180 days. See Choose how far back to ingest.
  • The connector supports only long-lived OAuth user-to-machine (U2M) access tokens. If the OAuth App issues expiring user access tokens, ingestion fails after the token expires.
  • GitHub includes actor IP addresses in audit events only when IP address display is enabled in the GitHub organization or enterprise settings.

Configure GitHub for ingestion

  1. Sign in to your GitHub account.

  2. In the upper-right corner of any page, click your profile photo to open the user navigation menu.

  3. Click Settings.

  4. Click Developer settings.

  5. In the OAuth Apps section, click New OAuth App.

  6. Enter a memorable application name in the Application name field.

  7. In the Homepage URL field, enter your Databricks instance's primary URL. For example, https://my-databricks-account.cloud.databricks.com. Optionally, enter an application description.

  8. Set the authorization callback URL to the following:

    https://<your_workspace_domain>/login/oauth/lakehousefederation.html

    For example:

    https://my-databricks-account.cloud.databricks.com/login/oauth/lakehousefederation.html

  9. Click Register Application. This takes you to the application's details page.

  10. Note the Client ID. You need it when you create your Unity Catalog connection.

  11. Click Generate a new client secret to generate a client secret for your application.

  12. Note the generated Client Secret. This value is displayed only one time.

Tip

When you create a connection through the Lakewatch UI, enter the Client ID and Client Secret gathered above. For GitHub Enterprise Server, also enter the optional Enterprise domain. You are redirected to GitHub to authorize the requested scopes.

For more information, see Configure OAuth U2M for GitHub ingestion.

Next steps

After completing the configuration above, create a datasource with the GitHub connector in the Lakewatch UI. See API connectors for instructions.

Connector and Lakewatch schemas

Lakewatch ingests the following source table from the GitHub connector's default source schema:

Source table GitHub data Built-in Lakewatch parser
audit_logs Organization actions performed by users, integrations, and systems github_audit_lfc

The connector writes the source table to the datasource's raw destination. The built-in parser writes a separate parsed table named after the parser, unless you set a parsed destination when you configure the datasource.

Lakewatch-managed columns

The connector adds the following columns to each raw source row:

  • lw_id: A non-null string that Lakewatch uses as the primary key. The connector derives a stable, time-sortable ID from the source @timestamp and _document_id fields.
  • time: A non-null UTC timestamp converted from the source @timestamp value, which GitHub reports as Unix epoch milliseconds. The connector uses time as the cursor field for incremental ingestion.

The parsed table carries lw_id forward. The parser also assigns the parsed time column and adds:

  • lw_meta: Standard Lakewatch processing metadata.
  • lw_observables: Values extracted from fields marked as security observables in the parser.

For more information, see Parser reference.

Raw connector table schema

The connector extracts common GitHub audit event fields into typed columns and preserves the complete source event in payload.

Schema
Column Data type
lw_id STRING
_document_id STRING
time TIMESTAMP
action STRING
actor STRING
org STRING
repo STRING
payload VARIANT

For field definitions, see the GitHub audit log API.

Parser observables

The github_audit_lfc parser keeps the complete source event as VARIANT in payload and extracts the following fields into lw_observables.

Observables
Field Observable
actor username
payload.actor_ip ip
payload.email email
payload.invitee_email email
payload.user username
payload.blocked_user username
payload.external_identity_nameid username
payload.external_identity_username username
payload.user_programmatic_access_name username
payload.invitee username
payload.inviter username
payload.authorized_actors[] username
payload.authorized_actor_names[] username
payload.actor_id actor_id
payload.user_id actor_id
payload.referrer url
payload.url url
payload.pull_request_url url
payload.config.url url
payload.config_was.url url
payload.head_sha hash
payload.after hash
payload.before hash
payload.calling_workflow_shas[] hash
  • Last updated on