Azure Event Hubs
An Azure real-time data ingestion service.
573 questions
Custom Scale Rule with type azure-eventhub in Azure Container Apps scales up replicas, but never scales down.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(185.6, 43%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EA%3C/text%3E%3C/svg%3E)
Alvin
25
Reputation points
Hello Team,
Our microservices architecture implemented in Azure Container Apps utilizes Dapr pub/sub for publishing/subscribing to position data events to Azure Event Hubs. A key component is the positions-processing microservice, which scales dynamically based on event throughput.
Currently, we have configured the service with two partitions per Event Hub. When significant position event volume spikes occur, we expect the positions-processor to scale out its replicas accordingly to meet demand. However, we have observed some inconsistent auto-scaling behaviours.
Specifically, during substantial load the service only scales out to 2 container replicas at peak, regardless of sustained high message volume flowing through the Event Hubs. Additionally, the replicas fail to scale back down to the defined minimum of 1 replica when the spike dissipates.
I have attached the current configuration details for the position-processing service’s auto-scaling rules in Azure Container Apps, which allow further analysis into the scaling constraints. Please advise if any modifications are required to enable more responsive demand-based scaling tailored to Event Hubs traffic patterns and processor workload. Let me know if further troubleshooting details could help determine next steps or a resolution.
Please I need any help I can to get this service to scale appropriately.
{count} vote
-
PRADEEPCHEEKATLA-MSFT 80,491 Reputation points Microsoft Employee2024-01-11T05:14:37.4066667+00:00 @Alvin - Thanks for the question and using MS Q&A platform.
It seems like you are experiencing some issues with the auto-scaling behavior of your positions-processing microservice in Azure Container Apps. I'd be happy to help you troubleshoot this issue.
Based on the information you provided, it appears that you have configured a custom scale rule with type azure-eventhub to scale the replicas of your microservice based on the throughput of the Event Hubs. However, you have observed that the service only scales out to 2 container replicas at peak, regardless of sustained high message volume flowing through the Event Hubs. Additionally, the replicas fail to scale back down to the defined minimum of 1 replica when the spike dissipates.
To troubleshoot this issue, we need to check the configuration details of your position-processing service's auto-scaling rules in Azure Container Apps. Could you please provide me with the configuration details of your custom scale rule with type azure-eventhub? This will help me to identify any potential issues with the configuration and suggest modifications to enable more responsive demand-based scaling tailored to Event Hubs traffic patterns and processor workload.
Also, it's worth noting that the auto-inflate feature of Azure Event Hubs can automatically scale up the throughput units (TUs) to meet usage needs. Have you considered enabling auto-inflate on your Event Hubs namespace to automatically scale up the TUs based on the increase in your traffic? This can help prevent throttling scenarios and ensure that your microservice can handle larger transfer rates.
Please let me know if you have any further questions or if there is any additional information that I can provide to help you troubleshoot this issue.
Hope this helps. Do let us know if you any further queries.
-
Alvin 25 Reputation points
2024-01-11T17:58:41.9633333+00:00 Hello @PRADEEPCHEEKATLA-MSFT
Thank you for your prompt response.
You must have missed the snapshot of my position-processing service's auto-scaling rules which I attached to the question.
I have attached the same snapshot to this response.
Auto-inflate is enabled in the EventHubs.
Please kindly let me know if you need any more information that can help with the troubleshooting. I look forward to your response.
Best Regards, Alvin.
-
Alvin 25 Reputation points
2024-01-11T18:02:56.71+00:00 Hello @PRADEEPCHEEKATLA-MSFT Thank you for the prompt response. You may have missed the snapshot of my position-processing service's auto-scaling rules in Azure Container Apps, which is attached the my Question.
I have attached same snapshot here.Auto-inflate is enabled in our EventHubs.
Please do let me know if you need further information that will help troubleshoot this issue.
Kind Regards, Alvin.
-
PRADEEPCHEEKATLA-MSFT 80,491 Reputation points Microsoft Employee
2024-01-12T01:48:19.0866667+00:00 @Alvin - Thanks for sharing the details. We are reaching out to the internal team to get more information related to your query and will get back to you as soon as we have an update.
-
PRADEEPCHEEKATLA-MSFT 80,491 Reputation points Microsoft Employee
2024-01-15T23:00:12.58+00:00 @Alvin - We are still awaiting for the internal team response. In case, if you need immediate assistance if you have a support plan could you please file a support ticket for deeper investigation and do share the SR# with us? In case if you don't have a support plan please let us know here.
Sign in to comment