Gmail connector reference

This page contains reference material for the Gmail connector in Databricks Lakeflow Connect.

Important

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Azure Databricks previews.

General connector behavior

  • The connector is read-only. It talks only to https://gmail.googleapis.com and uses the https://www.googleapis.com/auth/gmail.readonly scope by default. It never modifies the source mailbox.
  • Each connection ingests a single mailbox. The connector stamps the mailbox value as a mailbox column on every row. To ingest more than one mailbox, create a separate connection and pipeline for each mailbox.
  • The source schema is default.
  • The messages and message_labels tables sync incrementally using the Gmail History API. The profile, labels, labels_details, drafts, and filters tables are full-refresh only.
  • Message attachments are contained within the messages table's payload column (payload.parts[].body.attachmentId). There is no separate attachments table.

Supported tables

The connector ingests the following tables from the default source schema.

Table Primary key Sync mode
profile emailAddress Full refresh
labels mailbox, id Full refresh
labels_details mailbox, id Full refresh
drafts id Full refresh
filters id Full refresh
messages id Incremental (Gmail History API, historyId)
message_labels message_id Incremental (Gmail History API, historyId)

Destination schema

The following sections describe the columns in each destination table.

profile

Column Type
emailAddress string (primary key)
messagesTotal long
threadsTotal long
historyId string
mailbox string

labels

Column Type
mailbox string (primary key)
id string (primary key)
name string
messageListVisibility string
labelListVisibility string
type string
messagesTotal long
messagesUnread long
threadsTotal long
threadsUnread long
color struct{textColor: string, backgroundColor: string}

labels_details

The labels_details table has the same columns as labels (mailbox, id, name, type, the visibility fields, the message and thread counts, and color). Each label is enriched with the response from the labels.get API.

drafts

Column Type
id string (primary key)
message struct{id: string, threadId: string}
mailbox string

filters

Column Type
id string (primary key)
criteria struct{from: string, to: string, subject: string, query: string, negatedQuery: string, hasAttachment: boolean, excludeChats: boolean, size: long, sizeComparison: string}
action struct{addLabelIds: array<string>, removeLabelIds: array<string>, forward: string}
mailbox string

messages

Column Type
id string (primary key)
threadId string
snippet string
historyId string
internalDate string
payload struct (see payload structure)
sizeEstimate long
mailbox string
_ingestion_timestamp timestamp
_row_deleted boolean
_row_truncated boolean

payload structure

The payload column materializes the message MIME tree up to 8 levels of nesting. Each level has the following structure:

struct{
  partId: string,
  mimeType: string,
  filename: string,
  headers: array<struct{name: string, value: string}>,
  body: struct{attachmentId: string, size: long, data: string},
  parts: array<payload>
}

Attachments are contained within payload.parts[].body.attachmentId. Parts nested more than 8 levels deep aren't expanded into struct columns.

message_labels

Column Type
message_id string (primary key)
threadId string
labelIds array<string>
mailbox string
_ingestion_timestamp timestamp
_row_deleted boolean
_row_truncated boolean

Incremental sync

The messages and message_labels tables sync incrementally:

  • The first run performs a full bootstrap crawl of the mailbox.
  • Subsequent runs call users.history.list, keyed on the historyId cursor taken from the profile resource, to fetch only the changes since the previous run.
  • Deletions are emitted as _row_deleted tombstones.
  • If Gmail expires the stored historyId (the History API returns a 404 because the cursor is older than Gmail's retention window), the connector automatically falls back to a full refresh of the affected table.

Important

Gmail retains history for a limited window, typically about seven days. Schedule the pipeline to run at least once every seven days so the stored historyId stays within that window. If the cursor expires, the next run performs a full refresh of messages and message_labels.

The messages and message_labels tables do not support SCD Type 2 history tracking; configuring SCD Type 2 for these tables causes pipeline validation to fail.

Rate limiting

When Gmail returns an HTTP 403 response, the connector reads the Retry-After header (with a minimum backoff of 1 second) and retries the request automatically.

  • Last updated on