Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
Lakewatch is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage workspace-level previews.
Lakewatch provides an API connector to ingest audit activity from Google Workspace applications and services. This article covers prerequisites, configuration instructions, the connector schema, and security observables.
Prerequisites
The connector requires:
- Serverless Lakeflow pipelines turned on in your workspace.
- Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
- The Lakeflow Connect for Google Workspace preview turned on in your workspace by a workspace admin. See Turn on the connector preview.
- If your workspace's serverless network policy uses restricted access, the source's API domains must be on the policy's Allowed domains list. See Allow the source API domains in your network policy.
- A Google Workspace administrator account with the Reports privilege. Super administrators already have this privilege. Otherwise, create a custom administrator role that includes Reports, then assign the role to the account that authorizes the connection. Reports is listed under Reports in the privilege list.
- To ingest Vault audit activity, include the Google Vault Access All Logs privilege on the same custom role, under Vault.
- Access to a Google Cloud project where you can enable the Admin SDK API and create OAuth credentials.
- A supported edition for each Google Workspace application that requires one. Lakewatch writes these applications into the combined
activitiestable:- Drive and Keep activity require a Business or Enterprise edition. See the Admin SDK Reports API reference.
- Access Transparency activity requires Frontline Plus, Enterprise Plus, Education Standard, Education Plus, or Enterprise Essentials Plus. See Access Transparency.
- Vault activity requires a Google Vault add-on license. See Google Vault licensing.
Known limitations
- The connector ingests audit activity from the Admin SDK Reports API. It doesn't ingest mailbox contents, Google Drive files, or other Google Workspace resources.
- Historical lookback can't exceed 179 days. Google retains audit activity reports for 180 days. See Choose how far back to ingest.
- The connector supports OAuth user-to-machine (U2M) authentication. OAuth M2M, username and password authentication, and API key authentication aren't supported.
- Google limits each Google Account and OAuth client combination to 100 refresh tokens. Creating another token invalidates the oldest token.
Configure Google Workspace for ingestion
Create a Google Cloud project, configure the OAuth consent screen, enable the Admin SDK API, and create OAuth client credentials. Use the resulting client ID and secret to create the Unity Catalog connection in Databricks. For more information, see Enable Google Workspace APIs and Create access credentials.
Create or select a Google Cloud project
- Log in to the Google Cloud console.
- At the top of the page, open the project picker. Select an existing project, or click Create project to create a new one.
- If you create a project, enter a descriptive Project name, choose a Location, then click Create. It takes a few seconds to create the project.
- In the left navigation menu, go to Cloud overview > Dashboard.
- Confirm that the project you selected or created is shown at the top of the page. If it isn't, open the project picker and select it.
Configure your Google Cloud app
- In the top search bar, search for OAuth consent screen and select it.
- On the OAuth consent screen page, click Get started.
- On the Branding page, enter the following, then leave the remaining fields blank:
- App name: A name for the app.
- User support email: Your email address.
- Audience: Select Internal.
- Developer contact information: Your email address.
- Click Save and continue.
- On the Data access page, click Add or remove scopes.
- In Manually add scopes, enter
https://www.googleapis.com/auth/admin.reports.audit.readonly. - Click Add to table, then click Update.
- Click Save.
Enable the Admin SDK API
- In the top search bar, search for Admin SDK API and select it.
- On the Admin SDK API page, click Enable.
Create OAuth credentials
- In the top search bar, search for API & Services and select it.
- In the left navigation menu, click Credentials.
- Click Create credentials.
- Select OAuth client ID.
- For Application type, select Web application, then enter a Name for the client.
- Under Authorized redirect URIs, click Add URI.
- Enter
https://<workspace-url>/login/oauth/http.html. Replace<workspace-url>with the Databricks workspace host name. - Click Create.
- In the dialog, record the Client ID and Client secret. Use these values to create the Unity Catalog connection.
Tip
When you create a connection through the Lakewatch UI, enter the Client ID and Client Secret gathered above, then click Sign in to Google and grant the requested permission.
For more information, see the Google Workspace source setup.
Next steps
After completing the configuration above, create a datasource with the Google Workspace connector in the Lakewatch UI. See API connectors for instructions.
Connector and Lakewatch schemas
Lakewatch combines audit activity from all supported Google Workspace applications into a single table named activities.
| Source table | Google Workspace data | Built-in Lakewatch parser |
|---|---|---|
activities |
Combined audit activity from all supported Google Workspace applications | google_workspace_reports |
The connector writes each source table directly to the destination configured for its built-in parser. By default, the destination table is named <parser>_<datasource>, where <datasource> is the datasource name. For example, with the default datasource name google_workspace, the table is google_workspace_reports_google_workspace. You can set a different destination during datasource creation. There is no separate raw staging table or later parsing step.
Lakewatch-managed columns
The connector adds the following columns to each output row:
lw_id: A non-null string that Lakewatch uses as part of the primary key. The connector derives a stable, time-sortable ID fromtimeand a stable hash of the source activity identity and event fields. Theactivitiestable useslw_idandapplicationas a composite primary key.time: A non-null timestamp parsed fromid.time. The connector usestimeas the cursor field for incremental ingestion.
For Lakewatch datasources, the connector also adds:
lw_meta: The datasource and connection identity, parser name and version, and ingestion time.lw_observables: Security observables extracted by the connector.
For more information, see Connector output tables.
Connector output table schema
The destination includes the source and event columns below, together with the lw_meta and lw_observables columns described above.
The activities table writes one row per event. Each row contains the event's type, name, status, resource IDs, and application-specific parameters, together with shared activity details such as the actor, IP address, and event time. Application-specific parameters are stored as VARIANT.
Schema
| Column | Data type |
|---|---|
lw_id |
STRING |
application |
STRING |
time |
TIMESTAMP |
kind |
STRING |
etag |
STRING |
id |
STRUCT<time: STRING, uniqueQualifier: STRING, applicationName: STRING, customerId: STRING> |
actor |
STRUCT<profileId: STRING, email: STRING, callerType: STRING, key: STRING, applicationInfo: STRUCT<oauthClientId: STRING, applicationName: STRING, impersonation: BOOLEAN>> |
ownerDomain |
STRING |
ipAddress |
STRING |
isAgenticAction |
BOOLEAN |
networkInfo |
STRUCT<ipAsn: ARRAY<INT>, regionCode: STRING, subdivisionCode: STRING> |
type |
STRING |
name |
STRING |
status |
STRING |
resourceIds |
ARRAY<STRING> |
parameters |
VARIANT |
userDeviceInfo |
VARIANT |
resourceDetails |
VARIANT |
For field definitions, see the Admin SDK Reports API activity resource.
Security observables
The connector extracts the following fields into lw_observables.
Observables
| Field | Observable |
|---|---|
actor.email |
email |
actor.profileId |
actor_id |
ownerDomain |
domain |
ipAddress |
ip |
The connector also scans parameters by parameter name. Matching is case-insensitive.
Observables
| Parameter names | Observable |
|---|---|
user_email, email_monitor_dest_email, group_email, affected_email_address, email_forwarding_destination_address, grantee_email, recipient_email, organizer_email, guardians, impacted_users, invited_emails, previous_email, previous_course_owner, target_user, owner, assignee_email, new_assignee_email, group_id |
email |
domain_name, domain_alias, secondary_domain_name, link_domain |
domain |
web_address, remote_ews_url, task_creation_point_url, task_origin_space, target_uri, source_uri |
url |
email_log_search_smtp_sender_ip, email_log_search_smtp_recipient_ip, ip_address, actor_ip_address |
ip |
apk_sha256_hash |
hash |