GitHub audit logs

Important

Lakewatch is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage workspace-level previews.

Lakewatch provides an API connector to ingest organization audit logs from GitHub. This article covers prerequisites, configuration instructions, the connector schema, and security observables.

Prerequisites

The connector requires:

  1. Serverless Lakeflow pipelines turned on in your workspace.
  2. Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
  3. The Lakeflow Connect for GitHub preview turned on in your workspace by a workspace admin. See Turn on the connector preview.
  4. If your workspace's serverless network policy uses restricted access, the source's API domains must be on the policy's Allowed domains list. See Allow the source API domains in your network policy.
  5. An organization owner account for the GitHub organization whose audit logs you want to ingest. The organization must have GitHub Enterprise Cloud access to the organization audit-log REST API. GitHub Free organization access is insufficient.
  6. An OAuth authorization with the read:audit_log scope, granted by that organization owner.
  7. Permission to create OAuth Apps in your GitHub account's developer settings.

Known limitations

  • The connector supports GitHub.com only. GitHub Enterprise Server isn't supported. To ingest GitHub Enterprise Server audit logs, stream them to cloud storage and ingest them with a cloud storage datasource and the built-in github_audit parser.
  • The audit_logs table is available only for organization accounts.
  • Historical lookback can't exceed 180 days. See Choose how far back to ingest.
  • The connector supports only long-lived OAuth user-to-machine (U2M) access tokens. If the OAuth App issues expiring user access tokens, ingestion fails after the token expires.
  • GitHub includes actor IP addresses in audit events only when IP address display is enabled in the GitHub organization or enterprise settings.

Configure GitHub for ingestion

Record the organization login from its GitHub URL. For example, the login in https://github.com/example-org is example-org. Use this login, which can differ from the organization's display name, in the GitHub organization field when you create the datasource.

  1. Sign in to your GitHub account.

  2. In the upper-right corner of any page, click your profile photo to open the user navigation menu.

  3. Click Settings.

  4. Click Developer settings.

  5. In the OAuth Apps section, click New OAuth App.

  6. Enter a memorable application name in the Application name field.

  7. In the Homepage URL field, enter your Databricks instance's primary URL. For example, https://my-databricks-account.cloud.databricks.com. Optionally, enter an application description.

  8. Set the authorization callback URL to the following:

    https://<your_workspace_domain>/login/oauth/lakehousefederation.html

    For example:

    https://my-databricks-account.cloud.databricks.com/login/oauth/lakehousefederation.html

  9. Click Register Application. This takes you to the application's details page.

  10. Note the Client ID. You need it when you create your Unity Catalog connection.

  11. Click Generate a new client secret to generate a client secret for your application.

  12. Note the generated Client Secret. This value is displayed only one time.

Tip

When you create a connection through the Lakewatch UI, enter the Client ID and Client Secret gathered above. You are redirected to GitHub to authorize the requested scopes. Sign in as an organization owner and authorize read:audit_log.

For more information, see Configure OAuth U2M for GitHub ingestion.

Next steps

After completing the configuration above, create a datasource with the GitHub connector in the Lakewatch UI. Select or create the Unity Catalog connection, then enter the organization login in GitHub organization. See API connectors for instructions.


Connector and Lakewatch schemas

Lakewatch ingests the following source table from the GitHub connector. The source schema is the organization login entered in GitHub organization:

Source table GitHub data Built-in Lakewatch parser
audit_logs Organization actions performed by users, integrations, and systems github_audit_lfc

The connector writes each source table directly to the destination configured for its built-in parser. By default, the destination table is named <parser>_<datasource>, where <datasource> is the datasource name. For example, with the default datasource name github, the table is github_audit_lfc_github. You can set a different destination during datasource creation. There is no separate raw staging table or later parsing step.

Lakewatch-managed columns

The connector adds the following columns to each output row:

  • lw_id: A non-null string that Lakewatch uses as the primary key. The connector derives a stable, time-sortable ID from the source @timestamp and _document_id fields.
  • time: A non-null UTC timestamp converted from the source @timestamp value, which GitHub reports as Unix epoch milliseconds. The connector uses time as the cursor field for incremental ingestion.

For Lakewatch datasources, the connector also adds:

  • lw_meta: The datasource and connection identity, parser name and version, and ingestion time.
  • lw_observables: Security observables extracted by the connector.

For more information, see Connector output tables.

Connector output table schema

The destination includes the source and event columns below, together with the lw_meta and lw_observables columns described above.

The connector extracts common GitHub audit event fields into typed columns and preserves the complete source event in payload.

Schema
Column Data type
lw_id STRING
_document_id STRING
time TIMESTAMP
action STRING
actor STRING
org STRING
repo STRING
payload VARIANT

For field definitions, see the GitHub audit log API.

Security observables

The connector keeps the complete source event as VARIANT in payload and extracts the following fields into lw_observables.

Observables
Field Observable
actor username
payload.actor_ip ip
payload.email email
payload.invitee_email email
payload.user username
payload.blocked_user username
payload.external_identity_nameid username
payload.external_identity_username username
payload.user_programmatic_access_name username
payload.invitee username
payload.inviter username
payload.authorized_actors[] username
payload.authorized_actor_names[] username
payload.actor_id actor_id
payload.user_id actor_id
payload.referrer url
payload.url url
payload.pull_request_url url
payload.config.url url
payload.config_was.url url
payload.head_sha hash
payload.after hash
payload.before hash
payload.calling_workflow_shas[] hash