Zerobus Ingest connector quotas

This page describes the default quotas for the Zerobus Ingest connector in Lakeflow Connect, along with how it works with your Delta tables, schema, and data types.

Availability

The Zerobus Ingest connector is only available in some regions. For a list of supported regions, see Ingestion availability.

All of the available regions provide multi-zonal service availability, except for westus and northcentralus that provide single-az availability.

Latency

Latency reflects both the durability acknowledgment and the time to materialize records into the target Delta table. Actual times vary based on region alignment and workload characteristics.

• Time to durability
  • P50 ≤ 150 ms
• Time to table
  • P50 ≤ 5 sec

Quotas

The following table lists the default quotas for the Zerobus Ingest connector.

Item	Default quota	Need more?
Throughput per stream (gRPC)	100 MB/second (benchmarked with 1 KB messages)	Adjustable
Throughput per target table (gRPC)	10 GB/second	Adjustable
Records per second per stream	100,000 (benchmarked with 1 KB messages)	Adjustable
REST requests per second	10,000	Adjustable

These are the default throughputs the connector is provisioned with, and they scale to meet higher workloads. To raise any of them, contact your Databricks account representative. For maximum throughput, keep your client app and the endpoint in the same geographic region.

Delivery guarantees

The Zerobus Ingest connector provides at-least-once guarantees.

Buffered data retention

If the Zerobus Ingest service cannot flush buffered data to your Delta table, it holds the data in service-managed durable storage and occasionally emits per-stream warning events.

If the data remains unflushed for 31 days, the service deletes the buffered data and emits a per-stream discard event recording the number of records and bytes dropped. You cannot recover the data after deletion.

Table behavior

How Zerobus Ingest works with your Delta tables.

Partitioned tables

When writing to partitioned tables, Zerobus Ingest does not support writing to more than 100 partitions in any 5-second interval. For best throughput, minimize the number of partitions in each 5-second interval.

Databricks recommends using liquid clustered tables with Zerobus Ingest instead of partitioning.

Workspace and target table

Ingestion works with the following workspace and target table setup.

• The connector supports writing only to managed Delta tables. Writing to default storage is not supported.
• The connector does not support writing to storage secured through a private endpoint.
• The connector does not support recreating a target table.
• The connector only supports table names with ASCII letters, digits, and underscores.
• Both the workspace and the target table need to be in one of the available regions, and both in the same region.

Liquid clustered tables

When using the Zerobus Ingest connector with liquid clustered tables, it's recommended to keep predictive optimization enabled for the target table. The connector writes data into the table, but optimal data clustering is applied asynchronously by the predictive optimization service. Disabling predictive optimization can result in suboptimal query performance on the ingested data.

Schema and data types

The following sections explain how to define your schema and the data types Zerobus Ingest supports.

Schema evolution

Zerobus Ingest will never auto-evolve your target table.

Zerobus Ingest supports continuous ingestion when nullable Delta columns are added to the target table. Missing columns are filled with NULL values, allowing you to send records with missing fields.

Protobuf schema

The protobuf schema definition must match 1:1 with the Delta table schema (excluding extra nullable delta columns, which are considered a non-breaking schema change). If the schema does not match, the API returns an error. This includes:

• Different number of columns

• Different column names

• Different column optionality (nullable and non-nullable)

• The connector does not support proto schemas with more than 2000 columns.

• The connector only supports table and column names with ASCII letters, digits, and underscores.

• The connector does not support using a different proto schema for "stream creation" and "ingest record" operations.

Record size

Each message has a maximum size of 10 MB. The record size maximum is 10,485,760 bytes. Headers required for communication take up 19 bytes.

Type support

The following table shows the supported Delta types and their corresponding Protobuf types for ingestion.

Delta types	Protobuf types
INTEGER	int32
STRING	string
FLOAT	float
LONG	int64
SHORT	int32
DOUBLE	double
BOOLEAN	bool
BINARY	bytes
DATE Should be converted to int32 (number of days since epoch).	int32
TIMESTAMP Should be converted to int64 (epoch time in microseconds).	int64
ARRAY<TYPE>	repeated TYPE
MAP<K,V>	map<K,V> The map Protobuf syntactic sugar is only available for Protobuf compilers version 3 and above.
STRUCT<FIELDS>	message Nested { FIELDS }
VARIANT Variant must be ingested as a JSON-encoded string with keys of type STRING. The connector writes the data unshredded into the column. Supported formats include: Objects: "{\"id\":0,\"example\":\"this is variant example\"}" Primitives: "5", "3.14", "\"string\"" Arrays: "[1,2,3]"	string