Azure HDInsight
An Azure managed cluster service for open-source analytics.
206 questions
Kafka REST proxy endpoint is corrupting and wrongly parsing data
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(169.60000000000002, 87%, 26%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
AwesomePaul
20
Reputation points
Hello Team,
I recently activated REST proxy endpoint for HdInsight kafka. The endpoint is used to produce data to a topic, and upon consuming from the topic I observed that the data is wrongly encoded and parsed. If I try to produce data from the client using kafka console producer, the data is parsed and consumed with no issues.
My Kafka cluster version is 5.1, and below are the screenshots from the producer and consumer.
Case 1 ( using Content-type as "application/json")
Producer
Consumer
Case 2 ( using Content-type as "application/vnd.kafka.json.v2+json")
Producer
Consumer
Case 3 (Using Kafka console producer from terminal)
Producer
Consumer
Please advise on possible solution.
Thank you.
{count} votes
-
PRADEEPCHEEKATLA-MSFT 85,026 Reputation points Microsoft Employee2024-07-12T08:36:11.2666667+00:00 @AwesomePaul - Thanks for the question and using MS Q&A platform.
It seems like you are facing an issue with the Kafka REST proxy endpoint for HDInsight Kafka. Based on the information you provided, it looks like the data is wrongly encoded and parsed when you consume it from the topic using the REST proxy endpoint. However, when you produce data using the Kafka console producer, the data is parsed and consumed with no issues.
To better understand the issue, could you please provide more information on how you are producing and consuming data using the REST proxy endpoint?
-
AwesomePaul 20 Reputation points
2024-07-15T12:26:47.5866667+00:00 Thanks for your response @PRADEEPCHEEKATLA-MSFT
I am producing using the REST proxy endpoint from my local PC, and I consume from a kafka console producer running on the header node of the cluster.
In summary, i'm working on building a pipeline that uses kafka as a message broker, and i'll be streaming data from a various sources (could be an sdk, iOT, website events etc) , which are outside of Azure environment. Hence, the reason i'm using the REST proxy to connect to kafka.
I also have another test consumer setup using spark streaming on databricks, and this consumer is also consuming data in a wrongly encoded format. Same issue with the console consumer i mentioned above.
Not sure why the data is wrongly encoded, but i'm basically pushing JSON, and unable to consume the JSON, even after trying to decode.
Kindly see attached screenshots below, thanks.
-
PRADEEPCHEEKATLA-MSFT 85,026 Reputation points Microsoft Employee
2024-07-16T04:30:59.88+00:00 @AwesomePaul - I agree that this issue looks strange and I wasn't able to reproduce this issue. If you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us?
Sign in to comment