Hi every one I've been making some test and realized that a given CSV payload is around a 35% size of the equivalent JSON payload. I want to propose my boss to send to Event Hub CSV and not JSON, and here are my questions: ¿is it posible? The Data Generator feature of Event Hub on the Azure Portal only supports JSON I've reading that by serializing any content can be pushed to Event Hub In this SO , they state that to send CSV files to Event Hub, I have to provide the header line for every single line, so I would end up with a payload similar to JSON. Regards Jona

Hello @Jona, welcome to this moderated Azure community forum. As you can see in this code example , the body of an Event Hub message is just a byte array, so you can send everything you want as long as you convert it to a byte array. You could send either each line separately (eg. as JSON) or you could send the complete file at once (if it fits). What is needed to send a file? You need to read the whole file and put it into the message body as a byte array. The filename could then be added in the application properties . This means the receiver of the message must know how to construct the file back from the message (turn the byte array into a string, construct a file out of it and give it the proper name). The maximum message size of an Event Hub message is 1 Mb . This could be a compressed message (eg. GZIP of Deflate) so the original file could be much larger at first. As you can see, you can test for the size using the batching method. If this size is not enough, you could store the file in a blob storage first and only forward de location and name. Then the file size is not a limitation anymore. If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

@Jona - Thanks for the question and using MS Q&A platform. Yes, it is possible to send CSV files to Event Hub. However, you need to provide the header line for every single line, so you would end up with a payload similar to JSON. This means that the size reduction you are expecting may not be achieved. The Data Generator feature of Event Hub on the Azure Portal only only supports sending data in JSON, XML, Text, and Binary formats. CSV is not supported by the Data Generator. However, you can use any programming language to serialize your CSV data and send it to Event Hub. You can use the Azure Event Hubs client library for .NET, Java, Python, or JavaScript to send data to Event Hub. Hope this helps. Do let us know if you any further queries. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Upload CSV to Event Hub ¿is it posible?

Jona 355

Hi every one

I've been making some test and realized that a given CSV payload is around a 35% size of the equivalent JSON payload. I want to propose my boss to send to Event Hub CSV and not JSON, and here are my questions:

¿is it posible?
The Data Generator feature of Event Hub on the Azure Portal only supports JSON
I've reading that by serializing any content can be pushed to Event Hub

In this SO, they state that to send CSV files to Event Hub, I have to provide the header line for every single line, so I would end up with a payload similar to JSON.

Regards

Jona

PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee

2024-04-10T03:11:36.34+00:00

@Jona - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Jona 355 Reputation points

2024-04-10T03:35:59.04+00:00

Thanks for worrying. However, the others answers I've been given seems to be more accurate. I've yet finished the POC if the payload reductions is achieved.

When done, I'll let both of you know.

regards
PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee

2024-04-12T06:32:21.7966667+00:00

@Jona - Please do update us once you are done with the POC.
Jona 355 Reputation points

2024-04-12T07:06:16.04+00:00

No problem. If you want I can share with the interested on this via email

2 answers

Sander van de Velde 28,706 Reputation points MVP

2024-04-04T06:02:52.7766667+00:00

Hello @Jona,

welcome to this moderated Azure community forum.

As you can see in this code example, the body of an Event Hub message is just a byte array, so you can send everything you want as long as you convert it to a byte array.

You could send either each line separately (eg. as JSON) or you could send the complete file at once (if it fits).

What is needed to send a file?

You need to read the whole file and put it into the message body as a byte array.

The filename could then be added in the application properties.

This means the receiver of the message must know how to construct the file back from the message (turn the byte array into a string, construct a file out of it and give it the proper name).

The maximum message size of an Event Hub message is 1 Mb.

This could be a compressed message (eg. GZIP of Deflate) so the original file could be much larger at first. As you can see, you can test for the size using the batching method.

If this size is not enough, you could store the file in a blob storage first and only forward de location and name. Then the file size is not a limitation anymore.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.
Please sign in to rate this answer.
Jona 355 Reputation points

2024-04-04T17:07:58.9366667+00:00

Sorry, but I'm consufed now because I'm reading that any content can be pushed as long as it's converted to byte array.

From @Sander van de Velde

What is needed to send a file? You need to read the whole file and put it into the message body as a byte array. The filename could then be added in the application properties.

From @PRADEEPCHEEKATLA-MSFT

Yes, it is possible to send CSV files to Event Hub. However, you need to provide the header line for every single line, so you would end up with a payload similar to JSON. This means that the size reduction you are expecting may not be achieved.

I'm about to make a POC in my suscription, however I want to what to expect. ¿why should it be needed to include the header in every single line?, ¿What about pushing the whole CSV file converted into a byte array? ...

¿would it be possible to achieve the size reduction in the payload?

Regards and thanks for answering

Sander van de Velde 28,706 Reputation points MVP

2024-04-04T21:01:24.9366667+00:00

Hello @Jona,

You have a CSV file and you want to send all data in it to another services via an Event Hub.

Then you have three way to do this.

approach 1:

Send each line in the CSV file separately.

Because then each line only contains some values (character separated), you probably want to provide the header too.

Thus you have to mix it with the original content or put the header in the application properties (together with some line counter?) or move over to JSON for each line to make sense.

As you can see, this approach makes it quite complicated to send a file.

approach 2:

Take the content of the complete file and send it as the body. Put the filename in the application properties.

This works great for file not larger than 1MB. Larger files could work if you compress them first as long as the byte array returned by the compression algorithm is smaller than 1MB. A possible approach to zip the file via code is this NuGet package.

Does this work out for you? Or is the file larger?

approach 3:

Put the (very) large file in some storage solution (like a database or Azure Blob Storage).

Then, share the reference to the location via an Event Hub message.

Both the sender and receiver of the message must be able to access the storage solution to either write or read the file.

Conclusion:

I hope this makes it more clear to you how you can pass on messages using the Event Hub.

Try out which approach best fits your requirements.

Do some experiments.

Please check out the code example seen above first so you understand how to send (and receive) messages.

Jona 355 Reputation points

2024-04-04T23:48:45.5233333+00:00

Hi @Sander van de Velde

Approach 2 is the one I want because what I'm looking for is payload reductions, thus using less Event Hub capacity (thanks about the idea of compression).

Have yout tried this or is just a not-proven idea?

regards and thanks

Jona 355 Reputation points

2024-04-05T03:07:39.4466667+00:00

According to the docs, Event Hub Standard Tier has a limit of 1 MB of publication

The maximum message size of an Event Hub message is 1 Mb.

However, the ingress limit is 1 MB/s or 1000 events. So I assume that the limit of 1MB of publication/payload is to honor the proccessing speed of 1 MB/sec.

As I said, the ingress limit is 1 MB/s or 1000 events, ¿Will events be rejected in case of one of these conditions being violated?

That brings me to the thought of taking care not only the size of the publication/payload, but also de quantity of events

In mi case, I should put attention to not have CSV files with more than 1000 rows so that the file can't be rejected (or some rows being rejected)

¿Am I right in my interpretation of the docs?

regards

Sander van de Velde 28,706 Reputation points MVP

2024-04-05T14:53:36.3+00:00

Hello Jona,

I used this compression technique in the past for messages ingested by an IoT Hub.

Decompression can be done using the Azure Stream Analytics job (or use some library like I did with this NuGet package).

You need to decouple the message size and throughput size.

You can try to send any amount of messages, both large ones and small ones.

The Event Hub will try to ingest the messages with a throughput limitation.

If the Event Hub is not able to cope with the ingestion limitations it will start throttling.

This means complete messages (not parts of messages) will be rejected with a certain error status code.

Then its up to the sender to try to resend the same message again after a cool down.

If this happens, the Event Hub metrics will show this as well:

The solution is simple, scale up the number of throughput units:

Please do some tests on the message size after compression and check if the publisher application is capable of throttling.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Jona 355 Reputation points

2024-04-05T22:51:59.0433333+00:00

Hi @Sander van de Velde

What about this (quoting myself)?

However, the ingress limit is 1 MB/s or 1000 events. So I assume that the limit of 1MB of publication/payload is to honor the proccessing speed of 1 MB/sec. As I said, the ingress limit is 1 MB/s or 1000 events, ¿Will events be rejected in case of one of these conditions being violated?

Best regards

Sander van de Velde 28,706 Reputation points MVP

2024-04-06T09:29:42.56+00:00

Hello Jona,

yes, if the EventHub get a higher load than it can process (eg. when auto-inflate is not activated) it will reject messages.

This is called throttling.

You then need to both scale up your Event Hub and resend the messages.

Check out this post with details about recognizing throttled messages and how to handle them.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Jona 355 Reputation points

2024-04-08T13:01:25.48+00:00

I'm building my experiment in my suscription. One last question @Sander van de Velde ...

If I send a whole file using approach #2, that sending ¿is it considered as one message or not? Because Event hub has no way know how many rows are being sent...

So, 1000 event per second would be applicable if Event Hub won't know que quantity of rows

I expect having expressed myself well

regards

Sander van de Velde 28,706 Reputation points MVP

2024-04-08T14:40:35.29+00:00

Hello @Jona,

yes, if the file content fits inside the message body, it is counted as one message.

The Event Hub is not that smart. It just keeps receives messages as long as the size of individual messages is honoured and the throughput limitations are not reached.

Make sure you send the filename (and extension) as an application property so you can reconstruct the file later on when it is received.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.
Sign in to comment
PRADEEPCHEEKATLA-MSFT 77,676 Reputation points Microsoft Employee

2024-04-04T06:22:37.98+00:00

@Jona - Thanks for the question and using MS Q&A platform.

Yes, it is possible to send CSV files to Event Hub. However, you need to provide the header line for every single line, so you would end up with a payload similar to JSON. This means that the size reduction you are expecting may not be achieved.

The Data Generator feature of Event Hub on the Azure Portal only only supports sending data in JSON, XML, Text, and Binary formats. CSV is not supported by the Data Generator.

However, you can use any programming language to serialize your CSV data and send it to Event Hub. You can use the Azure Event Hubs client library for .NET, Java, Python, or JavaScript to send data to Event Hub.

Hope this helps. Do let us know if you any further queries.

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment