question

DevArch avatar image
0 Votes"
DevArch asked DevArch commented

Utilize Kafka API through Azure Event Hubs VS Hosting Kafka in Azure Container Apps

This is an "are we thinking about this correctly" type of question.

Currently: 3 AWS EC2 instances each running Kafka (0.10.2.0) and Zookeeper to support syncing between two on-premise CRMs via on-prem Windows services.

Problem: Administering the ec2 instances (monitoring, log cycling etc.) has taken a back seat, and therefore has caused stability issues (Lag sometimes spikes until we can figure out the issue(s)).

Desired Future State: A managed "Kafka"* solution that can serve as the message broker for a couple of services running on-prem that keep the two CRM instances in sync.

Question: Is there any advantage to us spinning up 3 instances of Kafka in Azure Container Apps, rather than just utilizing the Kafka API to interact with Azure Event Hubs?

References:

Validates we can use the Kafka API with Event Hubs: https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview

A lot of articles reference spinning up your own app in Azure Container Apps, that interact with Kafka, but none that are specifically hosting Kafka on Azure Container apps, which made me start questioning that route. e.g. https://quarkus.io/quarkus-workshops/super-heroes/index-azure.html

Also seeing Kafka referenced as infrastructure that can inform the scaling of apps themselves, which makes me think I should not be thinking of hosting Kafka as an app itself: https://azure.microsoft.com/en-us/services/container-apps/#features


**"Kafka" because our services are currently interacting with the Kafka API, but if we can use something else (like Eventhubs), which mirrors the Kafka API, then that's fine as well.*

tag: azure-container-apps (Tag not available?)

azure-event-hubs
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Got some feedback from the table top discussion I attended by @Sealjay at Microsoft Build today, where he said that unless there was a specific need for a Kafka specific install, that event hubs might be the way to go here. It was a great discussion, and I would encourage anyone to checkout the rest of his talks: https://mybuild.microsoft.com/en-US/speakers/2b7d2da5-9915-4fe1-b48a-eba48201599f

Would be interested in any other thoughts others have as well though.

1 Vote 1 ·

Thank you for sharing your discussions and promoting Build , @DevArch !

1 Vote 1 ·

1 Answer

DevArch avatar image
2 Votes"
DevArch answered DevArch commented

Reached out on the ACA Discord as well, and this was one response I received:

"You can't install Kafka in ACA since we don't support all the features it needs. Even if you get it to work, I still recommend using a managed service for this like EventHub with Kafka API or other managed Kakfa providers." - Tomer

https://discord.com/channels/938116342538706975/979098661466288148/979139137250680862

So Azure Event Hubs is the answer I am going with for now. If I receive a better answer here, I can reevaluate.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I have a kafka connector to azure event hub in production . Works really well. What do you use to integrate with CRM? On my case I am also pushing the data into CRM (dynamics 365). I'm looking now on a solution to have azure functions to go straight to kafka using the azure function kafka trigger or this :https://docs.confluent.io/kafka-connect-azure-functions/current/overview.html

1 Vote 1 ·
DevArch avatar image DevArch BrunoLucas-9843 ·

Currently, the are a number of windows services acting as producers and consumers of the messages going between crm1 <-svc*-> Kafka <-svc*-> crm2 We will probably look at running these svcs as container apps, and scale the consumers based on messages in Kafka.

Good to hear that the Kafka connector --> event hubs is working for you. Thanks for validating that approach.

0 Votes 0 ·