Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Lakewatch is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage workspace-level previews.
Lakewatch provides an API connector to ingest organization audit logs from GitHub. This article covers prerequisites, configuration instructions, the connector schema, and security observables.
Prerequisites
The connector requires:
- Serverless Lakeflow pipelines turned on in your workspace.
- Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
- The Lakeflow Connect for GitHub preview turned on in your workspace by a workspace admin. See Turn on the connector preview.
- If your workspace's serverless network policy uses restricted access, the source's API domains must be on the policy's Allowed domains list. See Allow the source API domains in your network policy.
- An organization owner account for the GitHub organization whose audit logs you want to ingest. The organization must have GitHub Enterprise Cloud access to the organization audit-log REST API. GitHub Free organization access is insufficient.
- An OAuth authorization with the
read:audit_logscope, granted by that organization owner. - Permission to create OAuth Apps in your GitHub account's developer settings.
Known limitations
- The connector supports GitHub.com only. GitHub Enterprise Server isn't supported. To ingest GitHub Enterprise Server audit logs, stream them to cloud storage and ingest them with a cloud storage datasource and the built-in
github_auditparser. - The
audit_logstable is available only for organization accounts. - Historical lookback can't exceed 180 days. See Choose how far back to ingest.
- The connector supports only long-lived OAuth user-to-machine (U2M) access tokens. If the OAuth App issues expiring user access tokens, ingestion fails after the token expires.
- GitHub includes actor IP addresses in audit events only when IP address display is enabled in the GitHub organization or enterprise settings.
Configure GitHub for ingestion
Record the organization login from its GitHub URL. For example, the login in https://github.com/example-org is example-org. Use this login, which can differ from the organization's display name, in the GitHub organization field when you create the datasource.
Sign in to your GitHub account.
In the upper-right corner of any page, click your profile photo to open the user navigation menu.
Click Settings.
Click Developer settings.
In the OAuth Apps section, click New OAuth App.
Enter a memorable application name in the Application name field.
In the Homepage URL field, enter your Databricks instance's primary URL. For example,
https://my-databricks-account.cloud.databricks.com. Optionally, enter an application description.Set the authorization callback URL to the following:
https://<your_workspace_domain>/login/oauth/lakehousefederation.htmlFor example:
https://my-databricks-account.cloud.databricks.com/login/oauth/lakehousefederation.htmlClick Register Application. This takes you to the application's details page.
Note the Client ID. You need it when you create your Unity Catalog connection.
Click Generate a new client secret to generate a client secret for your application.
Note the generated Client Secret. This value is displayed only one time.
Tip
When you create a connection through the Lakewatch UI, enter the Client ID and Client Secret gathered above. You are redirected to GitHub to authorize the requested scopes. Sign in as an organization owner and authorize read:audit_log.
For more information, see Configure OAuth U2M for GitHub ingestion.
Next steps
After completing the configuration above, create a datasource with the GitHub connector in the Lakewatch UI. Select or create the Unity Catalog connection, then enter the organization login in GitHub organization. See API connectors for instructions.
Connector and Lakewatch schemas
Lakewatch ingests the following source table from the GitHub connector. The source schema is the organization login entered in GitHub organization:
| Source table | GitHub data | Built-in Lakewatch parser |
|---|---|---|
audit_logs |
Organization actions performed by users, integrations, and systems | github_audit_lfc |
The connector writes each source table directly to the destination configured for its built-in parser. By default, the destination table is named <parser>_<datasource>, where <datasource> is the datasource name. For example, with the default datasource name github, the table is github_audit_lfc_github. You can set a different destination during datasource creation. There is no separate raw staging table or later parsing step.
Lakewatch-managed columns
The connector adds the following columns to each output row:
lw_id: A non-null string that Lakewatch uses as the primary key. The connector derives a stable, time-sortable ID from the source@timestampand_document_idfields.time: A non-null UTC timestamp converted from the source@timestampvalue, which GitHub reports as Unix epoch milliseconds. The connector usestimeas the cursor field for incremental ingestion.
For Lakewatch datasources, the connector also adds:
lw_meta: The datasource and connection identity, parser name and version, and ingestion time.lw_observables: Security observables extracted by the connector.
For more information, see Connector output tables.
Connector output table schema
The destination includes the source and event columns below, together with the lw_meta and lw_observables columns described above.
The connector extracts common GitHub audit event fields into typed columns and preserves the complete source event in payload.
Schema
| Column | Data type |
|---|---|
lw_id |
STRING |
_document_id |
STRING |
time |
TIMESTAMP |
action |
STRING |
actor |
STRING |
org |
STRING |
repo |
STRING |
payload |
VARIANT |
For field definitions, see the GitHub audit log API.
Security observables
The connector keeps the complete source event as VARIANT in payload and extracts the following fields into lw_observables.
Observables
| Field | Observable |
|---|---|
actor |
username |
payload.actor_ip |
ip |
payload.email |
email |
payload.invitee_email |
email |
payload.user |
username |
payload.blocked_user |
username |
payload.external_identity_nameid |
username |
payload.external_identity_username |
username |
payload.user_programmatic_access_name |
username |
payload.invitee |
username |
payload.inviter |
username |
payload.authorized_actors[] |
username |
payload.authorized_actor_names[] |
username |
payload.actor_id |
actor_id |
payload.user_id |
actor_id |
payload.referrer |
url |
payload.url |
url |
payload.pull_request_url |
url |
payload.config.url |
url |
payload.config_was.url |
url |
payload.head_sha |
hash |
payload.after |
hash |
payload.before |
hash |
payload.calling_workflow_shas[] |
hash |