High Ingestion Latency in Azure data explorer

siddharth bansal 326

I have a cluster with Standard_E8ads_v5 Tier and via data connections , IOT hub data is being pushed to adx tables but in the metrics , i can see an average ingestion latency of 5.20 min , on reading about the latency , i came across streaming ingestion and enabled it on cluster , but then the latency increased to 5.34 min , Kindly suggest how to reduce the latency and what are best practices that can be followed regarding the same .

adx 1

adx 2

KranthiPakala-MSFT 46,502 Reputation points Microsoft Employee

2023-10-17T00:58:07.6733333+00:00

@siddharth bansal Welcome to Microsoft Q&A forum and thanks for reaching out here.

This issue may need deeper investigation to understand the the root cause of the issue. But I'm reaching out to internal team to see if they have any inputs before we route you to support. I'll keep you posted as soon as we have an update from internal team.

Appreciate your patience.

Thank you
siddharth bansal 326 Reputation points

2023-10-17T04:00:59.12+00:00

Thanks for responding @KranthiPakala-MSFT , will wait for the update .
siddharth bansal 326 Reputation points

2023-10-20T12:03:29.68+00:00
, I tried updating the ingestion batching policy and it did reduced the latency to average of 30 sec and maximum latency now is coming as 1 min 30 sec
below is the command i ran to update ingestion batching policy

.alter database ['<db name>'] policy ingestionbatching

{ "MaximumBatchingTimeSpan" : "00:00:30", "MaximumNumberOfItems" : 1000, "MaximumRawDataSizeMB" : 1024 }

can the latency be reduced further ?if yes then how can it be done .

can I change anything in the above query to achieve the same .
will enabling streaming ingestion along with this alter policy ,help me in reducing the latency further .
KranthiPakala-MSFT 46,502 Reputation points Microsoft Employee

2023-10-20T17:55:06.5266667+00:00
@siddharth bansal Glad to know altering ingestion batching policy helped to reduce the latency.

Seems like you have already taken the right steps to reduce the ingestion latency by updating the ingestion batching policy.

Enabling streaming ingestion along with the updated ingestion batching policy may help reduce the latency further, but it is recommended to test the performance before implementing it in production.

As this is performance related, I cannot be on spot, but I suggest playing around with below properties:

Try use a smaller batch size. You can try reducing the MaximumNumberOfItems property in your ingestion batching policy to a smaller value, such as 100 or 200. This will reduce the number of items that are ingested in each batch and may help reduce the ingestion latency.

Or try using a smaller batch interval. You can try reducing the MaximumBatchingTimeSpan property in your ingestion batching policy to a smaller value, such as 10 or 15 seconds. This will reduce the amount of time that items are held in the batching queue before they are ingested and may help reduce the ingestion latency.

Hope this info helps. Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Accepted answer

KranthiPakala-MSFT 46,502 Reputation points Microsoft Employee

2023-10-19T23:08:54.66+00:00

@siddharth bansal As per my discussion with internal team and from the screenshots you shared, it looks like your cluster is working with the default 5 min batching step in the ingestion service.

You can control this batching step latency by altering the ingestion batching policy on table / database / cluster level. For more detailed info kindly refer to this article and see if that helps - IngestionBatching policy - Azure Data Explorer | Microsoft Learn

Hope this info helps.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

High Ingestion Latency in Azure data explorer

0 additional answers

Your answer