Integrate Apache Kafka Connect support on Azure Event Hubs

Apache Kafka Connect is a framework to connect and import/export data from/to any external system such as MySQL, HDFS, and file system through a Kafka cluster. This tutorial walks you through using Kafka Connect framework with Event Hubs.

This tutorial walks you through integrating Kafka Connect with an event hub and deploying basic FileStreamSource and FileStreamSink connectors. While these connectors aren't meant for production use, they demonstrate an end-to-end Kafka Connect scenario where Azure Event Hubs acts as a Kafka broker.


This sample is available on GitHub.

In this tutorial, you take the following steps:

  • Create an Event Hubs namespace
  • Clone the example project
  • Configure Kafka Connect for Event Hubs
  • Run Kafka Connect
  • Create connectors


To complete this walkthrough, make sure you have the following prerequisites:

Create an Event Hubs namespace

An Event Hubs namespace is required to send and receive from any Event Hubs service. See Creating an event hub for instructions to create a namespace and an event hub. Get the Event Hubs connection string and fully qualified domain name (FQDN) for later use. For instructions, see Get an Event Hubs connection string.

Clone the example project

Clone the Azure Event Hubs repository and navigate to the tutorials/connect subfolder:

git clone
cd azure-event-hubs-for-kafka/tutorials/connect

Configure Kafka Connect for Event Hubs

Minimal reconfiguration is necessary when redirecting Kafka Connect throughput from Kafka to Event Hubs. The following sample illustrates how to configure Connect to authenticate and communicate with the Kafka endpoint on Event Hubs:

# e.g.

# connect internal topic names, auto-created if not exists

# internal topic replication factors - auto 3x replication in Azure Storage



# required EH Kafka security settings
sasl.mechanism=PLAIN required username="$ConnectionString" password="{YOUR.EVENTHUBS.CONNECTION.STRING}";
producer.sasl.mechanism=PLAIN required username="$ConnectionString" password="{YOUR.EVENTHUBS.CONNECTION.STRING}";
consumer.sasl.mechanism=PLAIN required username="$ConnectionString" password="{YOUR.EVENTHUBS.CONNECTION.STRING}";

plugin.path={KAFKA.DIRECTORY}/libs # path to the libs directory within the Kafka release


Replace {YOUR.EVENTHUBS.CONNECTION.STRING} with the connection string for your Event Hubs namespace. For instructions on getting the connection string, see Get an Event Hubs connection string. Here's an example configuration: required username="$ConnectionString" password="Endpoint=sb://;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=XXXXXXXXXXXXXXXX";

Run Kafka Connect

In this step, a Kafka Connect worker is started locally in distributed mode, using Event Hubs to maintain cluster state.

  1. Save the above file locally. Be sure to replace all values in braces.
  2. Navigate to the location of the Kafka release on your machine.
  3. Run ./bin/ /PATH/TO/ The Connect worker REST API is ready for interaction when you see 'INFO Finished starting connectors and tasks'.


Kafka Connect uses the Kafka AdminClient API to automatically create topics with recommended configurations, including compaction. A quick check of the namespace in the Azure portal reveals that the Connect worker's internal topics have been created automatically.

Kafka Connect internal topics must use compaction. The Event Hubs team is not responsible for fixing improper configurations if internal Connect topics are incorrectly configured.

Create connectors

This section walks you through spinning up FileStreamSource and FileStreamSink connectors.

  1. Create a directory for input and output data files.

    mkdir ~/connect-quickstart
  2. Create two files: one file with seed data from which the FileStreamSource connector reads, and another to which our FileStreamSink connector writes.

    seq 1000 > ~/connect-quickstart/input.txt
    touch ~/connect-quickstart/output.txt
  3. Create a FileStreamSource connector. Be sure to replace the curly braces with your home directory path.

    curl -s -X POST -H "Content-Type: application/json" --data '{"name": "file-source","config": {"connector.class":"org.apache.kafka.connect.file.FileStreamSourceConnector","tasks.max":"1","topic":"connect-quickstart","file": "{YOUR/HOME/PATH}/connect-quickstart/input.txt"}}' http://localhost:8083/connectors

    You should see the event hub connect-quickstart on your Event Hubs instance after running the above command.

  4. Check status of source connector.

    curl -s http://localhost:8083/connectors/file-source/status

    Optionally, you can use Service Bus Explorer to verify that events have arrived in the connect-quickstart topic.

  5. Create a FileStreamSink Connector. Again, make sure you replace the curly braces with your home directory path.

    curl -X POST -H "Content-Type: application/json" --data '{"name": "file-sink", "config": {"connector.class":"org.apache.kafka.connect.file.FileStreamSinkConnector", "tasks.max":"1", "topics":"connect-quickstart", "file": "{YOUR/HOME/PATH}/connect-quickstart/output.txt"}}' http://localhost:8083/connectors
  6. Check the status of sink connector.

    curl -s http://localhost:8083/connectors/file-sink/status
  7. Verify that data has been replicated between files and that the data is identical across both files.

    # read the file
    cat ~/connect-quickstart/output.txt
    # diff the input and output files
    diff ~/connect-quickstart/input.txt ~/connect-quickstart/output.txt


Kafka Connect creates Event Hubs topics to store configurations, offsets, and status that persist even after the Connect cluster has been taken down. Unless this persistence is desired, it's recommended that these topics are deleted. You may also want to delete the connect-quickstart Event Hubs that were created during this walkthrough.

Next steps

To learn more about Event Hubs for Kafka, see the following articles: