Connect Synapse Spark Pool with Kafka on HDInsight

Question

I have created a Kafka on HDinsight cluster . I have also created Azure Synapse Analytics - Spark Pool on same region as HDinsight. I need guidance on how to consume topics from Kafka into Spark Structured Streaming. Any documentation or steps will be of great help.

To get started , just how to get the connection established between the two resources will suffice.

Answer 1

Hello @sql-seek ,

Thanks for the question and using MS Q&A platform.

(UPDATE: 15/06/2022): Product support group mentioned that synapse can't expose its vnet and therefore can't peer with Kafka on HDInsight. recommendation is to go with event hub.

-----------------------------------------

The Spark and HDI Kafka Cluster should be in the same VNet or there should be VNet pairing, so that Spark can communicate with the Kafka brokers.

This is a general document on how to use Spark with HDI Kafka: https://learn.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-kafka-spark-structured-streaming

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators