Container Apps Dapr Errors During Scaling

Question

We are using Container Apps and leveraging Dapr extensively for service invocation. For the most part this has worked great. However, we have been having reliability issues with Dapr when running under a heavy load. There will be random error spikes in service invocation from all over the app at different intervals. From what I can tell, these spikes roughly correlate to scaleup/scale in operations. It seems like the sidecar will either not be ready yet during scale up, or already be killed during scale down, but the app is still trying to use it. This results in many errors in the logs. Sometimes retries resolve the issue, but sometimes they don't.

As an example, we have a durable function setup which is designed to process zip files. It leverages Dapr for some service invocation calls in some of the activities. There seems to be a direct correlation in errors during scaling events with lots of errors like this:

It would appear that the app is trying to use Dapr either before it is ready (scale out) or after it has already shut down (scale down). Even when using retries when calling the activity function, sometimes these errors are not recovered from.

What is the guidance on how to handle this? Any advice you can give on getting more root cause analysis on the issue?

Answer 1

Hello Phillip S. Stenger Jr •,

Thanks for reaching out!

The errors you're seeing are likely due to your app invoking Dapr before the sidecar is fully ready during scale-out, or after it's terminated during scale-in. This causes intermittent failures during scaling operations.

• Use Dapr sidecar health checks before making calls to ensure it's ready.
• Handle graceful shutdowns in your app to avoid calling Dapr after it's stopped.
• Implement retry logic with exponential backoff and circuit breakers.
• In Durable Functions, wrap Dapr calls in try-catch with retries and avoid placing them directly in orchestrations.
• Monitor sidecar and app lifecycle events via Dapr logs and Azure Monitor.
• Optimize your Container App scaling configuration to avoid frequent scale events.

These steps will help reduce invocation errors and improve reliability during scale transitions.
For your reference, please refer the following documentation:

Dapr Health Checks

lifecycle management in Azure Container Apps

container-apps/application-lifecycle-management

Hope this helps!