API protocols

Zerobus Ingest exposes one endpoint with several API protocols: gRPC (through the SDKs), REST, OpenTelemetry (OTLP), MQTT, and Kafka-compatible APIs. All of them write directly into Unity Catalog Delta tables, so you choose the protocol that best fits each producer.

The following diagram shows how records from source streams and APIs flow through auto-scaling and load balancing to a horizontally scalable pool of stateless Zerobus nodes. Each node has a write-ahead log and a Lakehouse writer that batch-commits records into a Unity Catalog managed Delta table. The source labels in the diagram are representative. MQTT follows the same path through load balancing, durable storage, and table materialization.

Which protocol should you use?

Protocol	Best for	Why
gRPC (SDKs)	High-volume streaming producers: change data capture, clickstream, log and event forwarders.	Persistent connections deliver the highest sustained throughput and preserve per-stream ordering. Available in Python, Java, Rust, Go, TypeScript, and (in Beta) C++ and C# / .NET. See Write a client.
REST	Large fleets of lightweight or "chatty" clients, such as edge and IoT devices that report infrequently.	Stateless: each request stands alone, so clients don't hold an open connection. Simpler for devices that send data occasionally. See the REST example.
OpenTelemetry (OTLP)	Observability pipelines already emitting traces, logs, and metrics.	Point your existing OpenTelemetry SDKs or collectors at the endpoint with no custom integration. See Ingest OpenTelemetry data with Zerobus Ingest.
MQTT	IoT, edge, and telemetry applications that already publish MQTT v5 messages.	Reuse an MQTT v5 client to publish JSON records to a topic named after the target table. See Use MQTT with Zerobus Ingest.
Kafka-compatible APIs	Producers that already speak the Kafka protocol, or tooling that emits to Kafka, when you want that data in Delta with minimal code change.	Reuse an existing Kafka producer without a Azure Databricks SDK. Point it at the endpoint and produce to a topic named after your target table. See Use Kafka-compatible APIs with Zerobus Ingest.

gRPC with the SDKs

The Zerobus SDKs wrap a persistent, bidirectional gRPC connection called a stream. Because the connection stays open, gRPC achieves the highest sustained throughput and is the recommended path for continuous, high-volume ingestion. Each open stream is a long-lived connection, so a client's throughput scales with the number of streams it opens.

The SDKs handle connection management, offset tracking, and recovery for you, and are available in Python, Java, Rust, Go, TypeScript, and (in Beta) C++ and C# / .NET. They provide equivalent behavior across languages, so choose the one that fits your application. Over gRPC, the SDKs support JSON, protobuf, and Apache Arrow record formats. See Message types.

To write a client with the SDKs, including a per-language example for each, see Write a client.

REST

The REST interface is stateless: each request completes on its own without holding an open connection. This makes it a strong fit for large fleets of lightweight or intermittent producers, for example edge and IoT devices that report status infrequently, where maintaining a persistent connection per device would be impractical.

REST ingestion is governed by a request-rate quota (see Zerobus Ingest quotas). For a high-volume producer, the SDKs over gRPC will sustain higher throughput than issuing many individual REST requests. Reserve REST for producers that send infrequently or can't hold a persistent connection.

OpenTelemetry (OTLP)

Zerobus Ingest includes a native OpenTelemetry Protocol endpoint. If your systems already produce OpenTelemetry traces, logs, and metrics, you can point your existing OTLP exporters or collectors at Zerobus Ingest and land that telemetry directly in Delta tables you own, usually with just a configuration change. See Ingest OpenTelemetry data with Zerobus Ingest.

MQTT

Important

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Azure Databricks previews.

The MQTT interface accepts MQTT v5 connections over TLS on port 8883. One connection targets one table, and each message contains one UTF-8 JSON object. Use quality of service (QoS) level 1 when the producer requires a durable acknowledgment. For setup, authentication, limitations, and a Python example, see Use MQTT with Zerobus Ingest.

Kafka-compatible APIs

Important

The Kafka-compatible APIs are in Beta.

Zerobus Ingest offers Kafka-compatible producer APIs, so you can ingest with any Apache Kafka producer client without a Azure Databricks SDK. You point an existing Kafka producer at the endpoint and produce to a topic named after your target table, and the records land in a Unity Catalog Delta table. This is the best fit when you already run a Kafka producer, or a collector or agent that emits to Kafka, and want to route that data into Delta with minimal code change.

The Kafka-compatible APIs implement the producer-side subset of the Kafka protocol and are write-only: consumer, admin, and transactional APIs are not available. They accept JSON records only. Throughput is governed by a per-workspace request-rate quota rather than by individual connections. For the specific quota, see Zerobus Ingest quotas. For the highest throughput, per-record acknowledgments, and automatic recovery, use a Zerobus SDK over gRPC instead.

For setup, authentication, and a producer example, see Use Kafka-compatible APIs with Zerobus Ingest.

Related

• Message types: The record formats available over these protocols.
• Use Zerobus Ingest: Write a client using the SDKs or REST API.
• Ingest OpenTelemetry data with Zerobus Ingest: Ingest OpenTelemetry data.
• Use MQTT with Zerobus Ingest: Use MQTT v5 to publish JSON records.
• Zerobus Ingest release stages: Release stages for each interface and record format.
• Use Kafka-compatible APIs with Zerobus Ingest: Use Kafka-compatible producer APIs.