How to consume kafka topic from azure data factory?

Question

I want to create a pipeline which will consume data from Kafka topic. But in ADF Kafka connector is not present. Is there a way to establish connection from ADF to Kafka.

Answer 1

There is no direct Kafka connector available.

You can use Kafka Connect REST https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-kafka#start-the-connector

Example :

Here is also a paid solution : https://www.getorchestra.io/guides/azure-data-factory-connector-series-apache-kafka-connector-simplified

Answer 2

Hello @Bahurupi, Neel

Yes, you can use Azure Data Factory (ADF) to consume data from Kafka topic.

However, as you mentioned, there is no built-in connector for Kafka in ADF.

But you can use the Kafka Connect API to create a custom connector for Kafka in ADF. Here are the high-level steps to create a custom connector for Kafka in ADF:

1. Create a Kafka Connect cluster in Azure HDInsight.
2. Install the Kafka Connect Azure Blob Storage Sink connector on the Kafka Connect cluster.
3. Configure the Kafka Connect Azure Blob Storage Sink connector to write data to Azure Blob Storage.
4. Create an Azure Blob Storage linked service in ADF.
5. Create a dataset in ADF that points to the Azure Blob Storage location where the Kafka Connect Azure Blob Storage Sink connector writes data.

6 Create a pipeline in ADF that uses the Kafka Connect REST API to start a Kafka Connect job that reads data from the Kafka topic and writes it to Azure Blob Storage. You can find more detailed information on how to create a custom connector for Kafka in ADF in this Microsoft documentation.

---

I hope that this response has addressed your query and helped you overcome your challenges. If so, please mark this response as Answered. This will not only acknowledge our efforts, but also assist other community members who may be looking for similar solutions.