How to deploy Kafka with AKS?

Question

Hi All,

I want to deploy Kafka in my AKS cluster. Is there any managed service in Azure for that? If not, is there any reference or exhaustive document or video to help me out implement kafka? I need a setup that is production ready and has all security features implemented. Thanks!

Answer 1

Hello @Kumar, Gaurav

Yes, you can use Azure Event Hubs for Kafka to stream data from Apache Kafka applications without setting up a Kafka cluster. Event Hubs provides a Kafka-compatible interface that you can use to stream data into Azure and process it with Azure services. Event Hubs provides a managed, scalable, and highly available service that you can use to stream data from Apache Kafka applications.

You can find the detailed information on how to use Azure Event Hubs for Kafka in this article: https://learn.microsoft.com/en-us/azure/event-hubs/azure-event-hubs-kafka-overview

You can also use Azure Kubernetes Service with Kafka on HDInsight. You can find the detailed information on how to use it in this article: https://learn.microsoft.com/en-us/azure/hdinsight/kafka/apache-kafka-azure-container-services

For security features, Event Hubs provides multiple options to authorize access to your secure resources, including OAuth 2.0 and Shared Access Signature (SAS). You can find the detailed information on security and authentication in this article:

https://learn.microsoft.com/en-us/azure/event-hubs/apache-kafka-developer-guide

I hope this helps! Let me know if you need anything else

Please accept answer and upvote if the above information is helpful for the benefit of the community.

Answer 2

The question was not answered to my satisfaction as the requirements is to deploy opensource Apache Kafka in Azure Kubernetes Services and not to use HDInsight.