IoT Hub ingestion into Azure Data Explorer runs at 100% avg ingestion utilization

Kovács, Ferenc 61 Reputation points
2023-12-04T18:49:34.9866667+00:00

My setup consists of several databases in a single Kusto cluster, each having a landing table with ingestion mapping configured from IoT Hubs. The data in the landing table is then split with update policies into specialized tables.

I have no ingestion batching policy defined, so things run on defaults.

To me it looks like the cluster is not keeping up with ingesting data, ingestion utilization is high at 100%, but strangely upon restarting the cluster, it somehow manages to ingest everything quickly, after which things start accumulating again. Metrics also suggest nothing is ingested for days, then a whole lot is ingested at once (see spikes in metrics).
This leads me to believe I should change some configuration to help with ingestion - but what?

Another strange thing is that data queries run fine, but some management queries time out

  • .alter database <TABLE> policy ingestionbatching -> works
  • .show commands-and-queries | take 1 -> query expires / times out

Here are some of the metrics:

User's image

User's image

User's image

All recommendations green, cluster health green.

Thanks for all the support!

Azure Data Explorer
Azure Data Explorer
An Azure data analytics service for real-time analysis on large volumes of data streaming from sources including applications, websites, and internet of things devices.
507 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Sander van de Velde | MVP 31,126 Reputation points MVP
    2023-12-04T23:11:22.78+00:00

    Hello @Kovács, Ferenc,

    welcome to this moderated Azure community forum.

    Can you share the size and kind of your cluster?

    Have you tried scaling out already for a couple of days?

    If so, did this give another result?

    0 comments No comments