This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(12.8, 15%, 11%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EAK%3C/text%3E%3C/svg%3E)
Hello,
I have json messages in event hub which are compressed using gzip. I am trying to read those messages by enabling compression field to gzip, however i am getting the following error.
Exception of type 'Microsoft.Streaming.Adapters.Exceptions.EventHubAdapterException' was thrown.
Is there a way i can read these compressed message in stream analytics.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(134.4, 9%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
Hi Admin (KK)
Just checking to see if you have got a chance to see below response from @Sander van de Velde | MVP . If so, could you please share your findings after the following the suggested inputs.
Thank you
Hello Admin (KK),
yes, Azure Stream Analytics supports automatic unzip/deflate of incoming messages.
See also this blog post.
Deflate works like gzip but ignore file system paths so it's even more compressed.
I'm not sure why you get this error, have you set the message ContentEncoding property?
Can you try it with deflate?
Yes, I tried the deflate as well and its not working as well.
Hello Admin (KK)
did you check for that ContentEncoding property?
I recommend trying out my coding example and seeing if that works. From there, spot the differences.
public static string CompressString(string text)
{
byte[] buffer = Encoding.ASCII.GetBytes(text);
MemoryStream ms = new MemoryStream();
using (GZipStream zip = new GZipStream(ms, CompressionMode.Compress, true))
{
zip.Write(buffer, 0, buffer.Length);
}
ms.Position = 0;
MemoryStream outStream = new MemoryStream();
byte[] compressed = new byte[ms.Length];
ms.Read(compressed, 0, compressed.Length);
return Convert.ToBase64String(compressed);
}
This is the method i am using for compressing the string to ingest the data to Event Hub and when i read the message from Event through C# i am able to uncompress it with out any issues
Decompress method (Azure Function with Event Hub Trigger)
string decompressed;
string eventBody = Encoding.UTF8.GetString(binaryData);
byte[] inputBytes = Convert.FromBase64String(eventBody);
using (var inputStream = new MemoryStream(inputBytes))
using (var gZipStream = new GZipStream(inputStream, CompressionMode.Decompress))
using (var streamReader = new StreamReader(gZipStream))
{
decompressed = streamReader.ReadToEnd();
}
return decompressed;
Hello Admin (KK),
yes, this seems OK code for compression.
This compressed payload must be added as body.
You need to add this body to the message.
And you need to set the ContentEncoding too:
var zippedMessageBytes = DeflateHelper.Zip(messageBytes);
using (var pipeMessage = new Message(zippedMessageBytes))
{
pipeMessage.ContentEncoding = "deflate";
pipeMessage.ContentType = "application/zip";
await moduleClient.SendEventAsync("output1", pipeMessage);
}
Probably you need to set the ContentType too.
Do you set the same properties too?
Hello, I was able to write a deserializer and able to read the event hub data via stream analytics. Now the issue i am not able to deploy it through the "Submit to Azure" from Visual Studio Code.
The Stream analytics jobs runs fine in debug mode and when i deploy the job gets created but doesn't run at all.
Hello Admin (KK),
I'm not sure why you need to write a custom deserializer.
If you use default GZIP or DEFLATE for compressing the body of your messages and you notify Azure Stream Analytics for this kind of content type, it should be working out of the box.
Regarding the VS Code interface, I'm not sure why this fails.
I was using this code to send the message
producerClient = new EventHubProducerClient(EhConnectionString);
using EventDataBatch eventBatch = await producerClient.CreateBatchAsync();
var eventData = eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes(CompressString(jsonString))));
try
{
await producerClient.SendAsync(eventBatch);
}
Hello @Admin (KK) ,
as shown above, Azure Stream Analytics identifies compressed messages based on the message properties.
Check the properties of the class, a 'ContentType' can be set.
As seen in my examples, please try:
message.ContentType = "application/zip"
Note: Turn 'new EventData(Encoding.UTF8.GetBytes(CompressString(jsonString)))' in an explicit variable first :-)
(my example is coming for the Azure IoT Device SDK and is based on the same principles)
If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.