Zerobus Ingest connector limitations

Important

The Zerobus Ingest connector is in Public Preview. To try it, contact your Azure Databricks account representative.

This page lists limitations when using the Zerobus Ingest connector in Lakeflow Connect.

Durability

We only support single availability zone (single-az) durability. This means our service may experience downtime if our zone is unavailable.

Throughput restrictions

• Same AZ
• 100MB/second per stream (benchmarked with 1KB-sized messages)
• 15,000 rows per second per stream

Maximum throughput can be achieved when a client app and endpoint are in the same geo region. If you need higher throughput, contact your Databricks account representative.

Delivery guarantees

The connector only provides at-least-once guarantees.

Workspace

The workspace must be in one of the following regions:

• canadacentral
• westus
• eastus
• eastus2
• centralus
• northcentralus
• swedencentral
• westeurope
• northeurope
• australiaeast
• southeastasia

Target table

• We support writing to managed Delta tables only. We do not support writing to managed storage.

• We do not support recreating a target table.

• We only support table names with ASCII letters, digits, and underscores.

• The following regions are supported:

  • canadacentral
  • westus
  • eastus
  • eastus2
  • centralus
  • northcentralus
  • swedencentral
  • westeurope
  • northeurope
  • australiaeast
  • southeastasia

Schema evolution

We do not support schema evolutions on the target table.

Protobuf schema

• The protobuf schema definition must match 1:1 with the Delta table schema. If not, the API returns an error. This includes:

  • Number of columns
  • Column names
  • Column optionality must match between Delta and Protobuf schemas

• We do not support proto schemas with more than 2000 columns.

• We only support table and column names with ASCII letters/digits/underscores.

• We do not support using a different proto schema for “stream creation” and “ingest record”.

Record size

Each individual message is limited to 4MB in size. Record size max is 4,194,285 Bytes. Headers required for communication take up 19 Bytes.

Type support

Delta types	Protobuf types
Integer	Int32
String	String
Float	Float
Long	Int64
Short	Int32
Double	Double
Boolean	Bool
Binary	Bytes
Date*	Int32
Timestamp**	Int64
Array<TYPE>	repeated TYPE
Map<K,V>	map<K,V>
Struct<FIELDS>	message Nested { FIELDS }

*Should be converted to int32 (number of days since epoch)

** Should be converted to int64 (epoch time in microseconds)

***The map Protobuf syntactic sugar is available only for Protobuf compilers version 3 and above