Queue-based scaling is killing container instances before completion (Azure Container Apps)

Question

Queue-based scaling is killing container instances before completion (Azure Container Apps)

Todorov, Rosen 11

In Azure Container Apps, we make use of automatic scaling rule (KEDA, event based), where we are pulling messages from a queue or event source and want to have it run to completion without scaling down. The rule is set to scale down to 0 (serverless), but we have no control over the killing of container instances, as we want it to run until completion, before killing it. Is there a possible way to do so, or a feasible work-around.

1 answer

Your answer

Answer 1

MughundhanRaveendran-MSFT 12,511

@Todorov, Rosen ,

Thanks for reaching out to Q&A.

Please set the properties.configuration.activeRevisionsMode property of the container app to single in the configuration Json file. If this doesnt work, please provide the configuration json file so that I can check if the app is properly configured. If this issue is urgent and needs immediate mitigation, you can set the minReplicas property to 1 and run the app.

Feel free to reach out to me if you have any queries or concerns.

Todorov, Rosen 11 Reputation points

2022-08-16T14:10:07.697+00:00

I cannot see what does the revision mode have to do with the scaling rule. However we tried it, it didn't help.
Also setting the value of minReplicas to 1 does not help, since the scaling rule is set to scale (up or down) on every 10 messages.
This means that on 20 messages, we have two running instances and on 10- we have one running instance and the other one was killed,
even though it has not completed its tasks..
James Greig 1 Reputation point

2022-11-11T15:11:15.607+00:00

Have you had any success with this? We are considering a solution with a similar architecture. We are trying to reduce the problem by not deleting messages in the queue until the job is finished but have seen no solution to the problem where a container with work in progress is shut down by the autoscaler.
Any thoughts or advise on this would be much appreciated.
Todorov, Rosen 11 Reputation points

2022-11-21T13:27:19.267+00:00

Our team decided to workaround this problem. Our functional requirements include persistence. This means that every time a running job is being killed,
the lock is not being renewed. The message becomes free and other instances can read it. One of the other working instances reads the message and continues the job,
from where the previous instance has ended, but this is only possible because of the implemented persistence and state-keeping (in azure tables).
Jasenka Todorovic 0 Reputation points

2023-04-20T15:28:30.3933333+00:00

I have similar issue as James Greig described.
I am running container which is processing long running task in Azure Container Apps environment.
I setup Azure Queue storage Scaler with queue length 1 because I want that with each new message on queue new replica spin up. New instance will receive first available (visible) message on queue, lock the message, do processing based on message information and when processing is finished delete the message from queue. Scale up of containers works fine, but after ~5 minutes containers which have not finished processing and intensively used memory (so container is not in any idle state in terms of resources) are just scale down (shut down) by scaler.
Is there any recommendation how to resolve this? I did not find way how to reconfigure scaler limits for scaling down.
Is there some other better solution for controlling scaling of long running containers in ACA environment?

Share via

Queue-based scaling is killing container instances before completion (Azure Container Apps)

1 answer

Your answer