Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
This page describes the default quotas for Zerobus Ingest in Lakeflow Connect, along with how it works with your Delta tables, schema, and data types.
Availability
Zerobus Ingest is only available in some regions. For a list of supported regions, see Ingestion availability.
All of the available regions provide multi-zonal service availability, except for westus and northcentralus that provide single-az availability.
Latency
Latency reflects both the durability acknowledgment and the time to materialize records into the target Delta table. Actual times vary based on region alignment and workload characteristics.
- Time to durability
- Approximately 150 ms
- Time to table
- Approximately 5 seconds
Quotas
The following table lists the default quotas for Zerobus Ingest.
| Item | Default quota | Need more? |
|---|---|---|
| Throughput per stream (gRPC) | 100 MB/second (benchmarked with 1 KB messages) | Adjustable |
| Throughput per target table (gRPC) | 10 GB/second | Adjustable |
| Records per second per stream | 100,000 (benchmarked with 1 KB messages) | Adjustable |
| REST requests per second | 10,000 | Adjustable |
| Concurrent streams per workspace | Unlimited | N/A |
These are the default throughputs Zerobus Ingest is provisioned with, and they scale to meet higher workloads. To raise any of them, contact your Databricks account representative. For maximum throughput, keep your client app and the endpoint in the same geographic region.
You can open as many concurrent streams as your workload needs, and write to as many target tables. Opening more streams is the intended way to scale out, so the number of concurrent streams is unbounded.
The Kafka-compatible APIs are in Beta, with a default quota of 50,000 messages per second per workspace. To request a higher Beta quota, contact your Databricks account representative.
Delivery guarantees
Zerobus Ingest provides at-least-once guarantees.
Buffered data retention
If the Zerobus Ingest service cannot flush buffered data to your Delta table, it holds the data in service-managed durable storage and occasionally emits per-stream warning events.
If the data remains unflushed for 28 days, the service deletes the buffered data and emits a per-stream discard event recording the number of records and bytes dropped. You cannot recover the data after deletion.
Table behavior
How Zerobus Ingest works with your Delta tables.
Partitioned tables
When writing to partitioned tables, Zerobus Ingest does not support writing to more than 100 partitions in any 5-second interval. For best throughput, minimize the number of partitions in each 5-second interval.
Databricks recommends using liquid clustered tables with Zerobus Ingest instead of partitioning.
Workspace and target table
Ingestion works with the following workspace and target table setup.
- Zerobus Ingest supports writing to managed Delta tables.
- Zerobus Ingest does not support recreating a target table.
- Zerobus Ingest only supports table names with ASCII letters, digits, and underscores.
- Both the workspace and the target table need to be in one of the available regions.
Liquid clustered tables (Beta)
When using Zerobus Ingest with liquid clustered tables, it's recommended to keep predictive optimization enabled for the target table. Zerobus Ingest writes data into the table, but optimal data clustering is applied asynchronously by the predictive optimization service. Disabling predictive optimization can result in suboptimal query performance on the ingested data.
Schema and data types
Record size
Each message has a maximum size of 10 MB. The record size maximum is 10,485,760 bytes. Headers required for communication take up 19 bytes.
For how Zerobus Ingest validates records against your table schema, including schema evolution and the Protobuf schema rules, see Schema management. For the supported Delta and Protobuf data types, see Supported data types.