Google Cloud audit logs

Important

Lakewatch is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage workspace-level previews.

Overview

Google Cloud Audit Logs record administrative activity, data access, and system events across your Google Cloud projects, folders, and organizations. Lakewatch ingests Google Cloud audit logs from cloud storage using Auto Loader and schematizes the data with the built-in gcp_audit_log parser. This is a cloud storage source, not an API connector.

Onboard Google Cloud audit logs

To ingest Google Cloud audit logs, create a cloud storage datasource and attach the built-in gcp_audit_log parser so events are written to a schematized parsed table. For prerequisites, Unity Catalog access, and the full datasource setup steps, see Cloud storage.

Parsed schema and observables

The built-in gcp_audit_log parser schematizes Google Cloud audit logs into a parsed table. Every parsed table also carries the Lakewatch-managed columns:

  • lw_id: A non-null string Lakewatch uses as the primary key, supporting deduplication across incremental reads.
  • time: A non-null event timestamp assigned by the parser.
  • lw_meta: Standard Lakewatch processing metadata.
  • lw_observables: Values extracted from fields marked as security observables in the parser.

See Parser reference.

Parsed table schema

The gcp_audit_log parser writes the following columns (generated from the parser definition):

Column Data type
time TIMESTAMP
logName STRING
severity STRING
insertId STRING
resource VARIANT
receiveTimestamp TIMESTAMP
labels VARIANT
operation VARIANT
trace STRING
httpRequest VARIANT
spanId STRING
traceSampled BOOLEAN
sourceLocation VARIANT
protoPayload VARIANT
Parser observables

The parser extracts the following fields into lw_observables (generated from the parser definition):

Field Observable
httpRequest.requestURL url
httpRequest.remoteIP ip
httpRequest.serverIP ip
httpRequest.referer url
protoPayload.authenticationInfo.principalEmail email
protoPayload.authenticationInfo.serviceAccountDelegationInfo[].firstPartyPrincipal.principalEmail email
protoPayload.requestMetadata.callerIP ip

The following VARIANT fields are scanned for observables by matching nested key names. The table lists the observable types extracted from each:

Field Observable types scanned
protoPayload.request domain, email, ip, username
protoPayload.response domain, email, ip, username
protoPayload.metadata domain, email, ip, username