Share via

Azure EventHub Throttling Request - Standard Tier, 1TU

K S Devadoss, Vishnu Kumar 20 Reputation points
2023-08-24T11:55:22.5933333+00:00

Problem Statement
Single consumer is consuming the message from the single topic partition. The consuming topic partition has 8578 messages The observation from Azure Eventhub Metrics, Outgoing message is high and seeing the Throttling Request.
This test is performed in Azure Eventhub with Standard Tier with 1 Throughput Unit.

Expected Result
Should see only the 8578 message in Outgoing message metrics and no Throttling Request should see.

Confluent Kafka Nuget Package version 2.1.1 is used for this test

Consumer Configuration
AutoOffsetReset =AutoOffsetReset.Earliest
EnableAutoOffsetStore = true
EnableAutoCommit = false
AutoCommitIntervalMs = 10000 //Used only when "EnableAutoCommit " is set to true
EnablePartitionEof = true
ConnectionsMaxIdleMs = 180000
MaxPartitionFetchBytes = 12518400 // 11.9 MB
PartitionAssignmentStrategy = PartitionAssignmentStrategy.CooperativeSticky
IsolationLevel = IsolationLevel.ReadUncommitted
SocketNagleDisable = true
SocketKeepaliveEnable = true
MetadataMaxAgeMs = 180000
SessionTimeoutMs = 30000
MaxPollIntervalMs = 300000
CancellationDelayMaxMs = 200,
Debug = "consumer,fetch"

Consumer Topic Metrics

image

Note: The incoming messages is there is different time frame. Messages are accumulated once performing the test with different ConsumerGroup for each run for test

Producer Topic Metrics

image

Log details is uploaded in this (raw.githubusercontent.com/ksdvishnukumar/LogRepo/main/ForGithubLog.log)

The setting "MaxPartitionFetchBytes" is set as "12518400". Whether this number is set to 1 MB or 2 MB still see the same behavior.
But if i update AutoCommit is turned on, I see the below metrics
Consumer Metrics

User's image

Producer Metrics
User's image

Azure Monitor
Azure Monitor
An Azure service that is used to collect, analyze, and act on telemetry data from Azure and on-premises environments.
3,037 questions
Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
601 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Ryan Hill 27,111 Reputation points Microsoft Employee
    2023-09-05T13:14:23.0833333+00:00

    Apologies for the delayed response @K S Devadoss, Vishnu Kumar . I reached out to the product team and was informed that throttling isn't the main issue here. Throttling is expected when the incoming or outgoing message volume reaches the TU limits. In your case, it's 2MB/sec as the outgoing limit for 1 TU.

    They also agree that the outgoing message count does appear off and would like to look into the service logs. We can address this in the support case you opened.

    0 comments