RabbitMQ connector reference

Important

This feature is in Beta.

This page contains reference documentation for the managed RabbitMQ connector in Lakeflow Connect.

Connector options

The following options configure the RabbitMQ source for each destination table in the ingestion pipeline. Specify these options under connector_options.rabbitmq_options in your pipeline definition. See Examples for full pipeline examples.

Option	Type	Default	Description
queue	String	—	(Required) Name of the RabbitMQ classic queue to consume from.

Connection properties

When you create the Unity Catalog RabbitMQ connection in Catalog Explorer, you must specify the following properties. See Create a RabbitMQ connection for connection creation steps.

Property	Description
Connection name	A unique name for the connection in Unity Catalog.
Connection type	Select RabbitMQ.
Host	The hostname of the RabbitMQ broker.
Username	The username used to authenticate with the RabbitMQ broker.
Password	The password used to authenticate with the RabbitMQ broker.
Virtual host	The virtual host of the RabbitMQ broker. The default virtual host is /.
Port	The port of the RabbitMQ broker. The default port is 5672.

Destination table schema

The RabbitMQ connector writes to streaming tables (append-only). By default, each destination table contains a single column:

Column	Type	Description
body	BINARY	The raw binary content of the RabbitMQ message body. The connector does not decode or deserialize the body during ingestion. Decode it downstream as needed.

Include source metadata

To also capture RabbitMQ message metadata, set the source_metadata_column option in the destination table's table_configuration. When set, the connector adds a struct column with that name alongside body, containing the following fields:

Field	Type	Description
exchange	STRING	The exchange the message was published to.
routing_key	STRING	The routing key the message was published with.
queue	STRING	The queue the message was consumed from.
redelivered	BOOLEAN	true if the message was redelivered after a failed acknowledgement.
message_id	STRING	The producer-set message ID, if present.
correlation_id	STRING	The producer-set correlation ID, if present.
content_type	STRING	The producer-set content type, if present.
content_encoding	STRING	The producer-set content encoding, if present.
timestamp	BIGINT	The producer-set message timestamp, if present.
headers	MAP<STRING, STRING>	The producer-set message headers, if present.

The name you choose must not be body (the reserved output column), must not be empty, and must not contain a space or any of the characters ,;{}(), a newline, a tab, or =. When source_metadata_column is not set, the connector writes only the body column.

For a pipeline example, see Pipeline with source metadata.