Google Workspace audit activity

Important

Lakewatch is in Private Preview. To request access, reach out to your Azure Databricks account team.

Lakewatch provides an API connector to ingest audit activity from Google Workspace applications and services. This article covers prerequisites, configuration instructions, the connector schema, and Lakewatch parser observables.

Prerequisites

The connector requires:

  1. Serverless Lakeflow Spark Declarative Pipelines turned on in your workspace.
  2. Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
  3. A Google Workspace administrator account with the Reports privilege. Super administrators already have this privilege. Otherwise, create a custom administrator role that includes Reports, then assign the role to the account that authorizes the connection. Reports is listed under Reports in the privilege list.
    • To ingest Vault audit activity, include the Google Vault Access All Logs privilege on the same custom role, under Vault.
  4. Access to a Google Cloud project where you can enable the Admin SDK API and create OAuth credentials.
  5. A supported edition for each Google Workspace application that requires one. Lakewatch writes these applications into the combined activities table:

Known limitations

  • The connector ingests audit activity from the Admin SDK Reports API. It doesn't ingest mailbox contents, Google Drive files, or other Google Workspace resources.
  • Historical lookback can't exceed 179 days. Google retains audit activity reports for 180 days. See Choose how far back to ingest.
  • The connector supports OAuth user-to-machine (U2M) authentication. OAuth M2M, username and password authentication, and API key authentication aren't supported.
  • Google limits each Google Account and OAuth client combination to 100 refresh tokens. Creating another token invalidates the oldest token.

Configure Google Workspace for ingestion

Create a Google Cloud project, configure the OAuth consent screen, enable the Admin SDK API, and create OAuth client credentials. Use the resulting client ID and secret to create the Unity Catalog connection in Databricks. For more information, see Enable Google Workspace APIs and Create access credentials.

Create or select a Google Cloud project

  1. Log in to the Google Cloud console.
  2. At the top of the page, open the project picker. Select an existing project, or click Create project to create a new one.
  3. If you create a project, enter a descriptive Project name, choose a Location, then click Create. It takes a few seconds to create the project.
  4. In the left navigation menu, go to Cloud overview > Dashboard.
  5. Confirm that the project you selected or created is shown at the top of the page. If it isn't, open the project picker and select it.

Configure your Google Cloud app

  1. In the top search bar, search for OAuth consent screen and select it.
  2. On the OAuth consent screen page, click Get started.
  3. On the Branding page, enter the following, then leave the remaining fields blank:
    • App name: A name for the app.
    • User support email: Your email address.
    • Audience: Select Internal.
    • Developer contact information: Your email address.
  4. Click Save and continue.
  5. On the Data access page, click Add or remove scopes.
  6. In Manually add scopes, enter https://www.googleapis.com/auth/admin.reports.audit.readonly.
  7. Click Add to table, then click Update.
  8. Click Save.

Enable the Admin SDK API

  1. In the top search bar, search for Admin SDK API and select it.
  2. On the Admin SDK API page, click Enable.

Create OAuth credentials

  1. In the top search bar, search for API & Services and select it.
  2. In the left navigation menu, click Credentials.
  3. Click Create credentials.
  4. Select OAuth client ID.
  5. For Application type, select Web application, then enter a Name for the client.
  6. Under Authorized redirect URIs, click Add URI.
  7. Enter https://<workspace-url>/login/oauth/http.html. Replace <workspace-url> with the Databricks workspace host name.
  8. Click Create.
  9. In the dialog, record the Client ID and Client secret. Use these values to create the Unity Catalog connection.

Tip

When you create a connection through the Lakewatch UI, enter the Client ID and Client Secret gathered above, then click Sign in to Google and grant the requested permission.

For more information, see the Google Workspace source setup.

Next steps

After completing the configuration above, create a datasource with the Google Workspace connector in the Lakewatch UI. See API connectors for instructions.

Connector and Lakewatch schemas

Lakewatch combines audit activity from all supported Google Workspace applications into a single table named activities.

Source table Google Workspace data Built-in Lakewatch parser
activities Combined audit activity from all supported Google Workspace applications google_workspace_reports

The connector writes the source table to the datasource's raw destination. The built-in parser writes a separate parsed table named after the parser, unless you set a parsed destination when you configure the datasource.

Lakewatch-managed columns

The connector adds the following columns to each raw source row:

  • lw_id: A non-null string that Lakewatch uses as part of the primary key. The connector derives a stable, time-sortable ID from time and a stable hash of the source activity identity and event fields. The activities table uses lw_id and application as a composite primary key.
  • time: A non-null timestamp parsed from id.time. The connector uses time as the cursor field for incremental ingestion.

The parsed table carries lw_id forward. The parser also assigns the parsed time column and adds:

  • lw_meta: Standard Lakewatch processing metadata.
  • lw_observables: Values extracted from fields marked as security observables in the parser.

For more information, see Parser reference.

Raw connector table schema

The activities table writes one row per event. Each row contains the event's type, name, status, resource IDs, and application-specific parameters, together with shared activity details such as the actor, IP address, and event time. Application-specific parameters are stored as VARIANT.

Schema
Column Data type
lw_id STRING
application STRING
time TIMESTAMP
kind STRING
etag STRING
id STRUCT<time: STRING, uniqueQualifier: STRING, applicationName: STRING, customerId: STRING>
actor STRUCT<profileId: STRING, email: STRING, callerType: STRING, key: STRING, applicationInfo: STRUCT<oauthClientId: STRING, applicationName: STRING, impersonation: BOOLEAN>>
ownerDomain STRING
ipAddress STRING
isAgenticAction BOOLEAN
networkInfo STRUCT<ipAsn: ARRAY<INT>, regionCode: STRING, subdivisionCode: STRING>
type STRING
name STRING
status STRING
resourceIds ARRAY<STRING>
parameters VARIANT
userDeviceInfo VARIANT
resourceDetails VARIANT

For field definitions, see the Admin SDK Reports API activity resource.

Parser observables

The google_workspace_reports parser writes nested objects as VARIANT and extracts the following fields into lw_observables.

Observables
Field Observable
actor.email email
actor.profileId actor_id
ownerDomain domain
ipAddress ip

The parser also scans parameters by parameter name. Matching is case-insensitive.

Observables
Parameter names Observable
user_email, email_monitor_dest_email, group_email, affected_email_address, email_forwarding_destination_address, grantee_email, recipient_email, organizer_email, guardians, impacted_users, invited_emails, previous_email, previous_course_owner, target_user, owner, assignee_email, new_assignee_email, group_id email
domain_name, domain_alias, secondary_domain_name, link_domain domain
web_address, remote_ews_url, task_creation_point_url, task_origin_space, target_uri, source_uri url
email_log_search_smtp_sender_ip, email_log_search_smtp_recipient_ip, ip_address, actor_ip_address ip
apk_sha256_hash hash
  • Last updated on