"While Event Grid can successfully ingest data for blob sizes under 1 GB, I'm dealing with a situation where the blob size exceeds 1 GB. What approach should I take to ensure that Event Grid can still ingest this larger blob size?"

Question

"While Event Grid can successfully ingest data for blob sizes under 1 GB, I'm dealing with a situation where the blob size exceeds 1 GB. What approach should I take to ensure that Event Grid can still ingest this larger blob size?"

Bexy Morgan 260

1 answer

Your answer

Answer 1

Sander van de Velde | MVP 36,761 MVP Volunteer Moderator

Hello @Bexy Morgan ,

can you share with us the link to the page where this limitation is shown?

I checked the documentation and it only shows this 6 GB limitation:

The original uncompressed data size should be part of the blob metadata, or else Azure Data Explorer will estimate it. The ingestion uncompressed size limit per file is 6 GB.

This is because EventGrid only is used to notify the arrival of blobs on the storage instead of transferring them.

The full text is:

The recommended file size for ADX ingestion is between 100MB and 1GB uncompressed. In compact binary formats like Avro and Parquet especially with snappy encoding, these numbers should be even lower. There is a service-internal limit on disk size of 6GB per single file being ingested. The driving limitation to these restrictions is the hard requirement that any ingestion operation must complete within 1 hour (including update policies).

Documentation is revised to be more clear about this.

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Bexy Morgan 260 Reputation points

2023-08-08T08:29:31.5166667+00:00

Hi Sander, in this document https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-overviews, it shows 1 GB uncompressed.
Sander van de Velde | MVP 36,761 Reputation points MVP Volunteer Moderator

2023-08-08T10:33:46.8933333+00:00

Good catch Bexy, these two numbers are not helping us :-)

Please let me check with my friends on the Product group.

Update: team is informed, and waiting for final confirmation this is increased overtime to 6 GB.
Sander van de Velde | MVP 36,761 Reputation points MVP Volunteer Moderator

2023-08-09T08:03:01.8733333+00:00

Hello @Bexy Morgan ,

this is the answer I got from the team:

The recommended file size for ADX ingestion is between 100MB and 1GB uncompressed. In compact binary formats like Avro and Parquet especially with snappy encoding, these numbers should be even lower. There is a service-internal limit on disk size of 6GB per single file being ingested. The driving limitation to these restrictions is the hard requirement that any ingestion operation must complete within 1 hour (including update policies).

Documentation is revised to be more clear about this.

I updated the text of the answer above accordingly.
Bexy Morgan 260 Reputation points

2023-08-09T08:32:09.67+00:00

Can you please elaborate "There is a service-internal limit on disk size of 6GB per single file being ingested. The driving limitation to these restrictions is the hard requirement that any ingestion operation must complete within 1 hour (including update policies)". Thanks Sander
Sander van de Velde | MVP 36,761 Reputation points MVP Volunteer Moderator

2023-08-09T09:05:40.2166667+00:00

Hello Bexy,

This is the message I got from the Product team.

If these hard limits are not met (size, duration), the ingest process will be stopped and the already ingested data must be considered abandoned probably (I have no details on how this works internally).

I suggest testing ingestion in a load test environment with files having both the normal (expected) size and two or three times bigger. This gives you confidence for the future.
Bexy Morgan 260 Reputation points

2023-08-09T10:03:22.38+00:00

Thanks Sander.
AnnuKumari-MSFT 34,556 Reputation points Microsoft Employee Moderator

2023-08-18T16:05:14.08+00:00

Hi Bexy Morgan ,

Glad to know your query has been answered . Could you please take a moment to accept the above answer by clicking on Accept answer button as it helps in answer visibility for the users who would face similar issue in future . Thankyou

Share via

"While Event Grid can successfully ingest data for blob sizes under 1 GB, I'm dealing with a situation where the blob size exceeds 1 GB. What approach should I take to ensure that Event Grid can still ingest this larger blob size?"

1 answer

Your answer