Azure function not executing queue messages in parallel

Ana Cristina Stefan 1

Hello,

We've been trying to obtain parallelism in executing the queue messages of a kubernetes function app.
We tried adjusting the following params in config:
PSWorkerInProcConcurrencyUpperBound = 10
FUNCTIONS_WORKER_PROCESS_COUNT = 10

The result was that 2 messages were being picked up from the queue at a time, then after a couple of minutes other 2 messages were picked up..after a couple of minutes another 2 messages were picked up..all 6 running in parallel. We were expecting the parallelism to work like..picking up 10 messages at the same timestamp then another 10 etc.

We also tried adjusting the host.json file with the following (also tweaked the batches values to see a difference)

"extensions": {
"queues": {
"maxPollingInterval": "00:00:02",
"visibilityTimeout": "00:00:30",
"maxDequeueCount": 5,
"newBatchThreshold": 16,
"batchSize": 32
}
},

Unfortunately, this didn't give us the expected parallelism either.
Could you help us in understanding what we are doing wrong?

Many thanks,
Ana

Pramod Valavala 20,626 Reputation points Microsoft Employee

2021-08-02T05:48:36.443+00:00

@Ana Cristina Stefan Since you are running on Kubernetes, it would help to know the resource requests/limits that you've set for your function container and if you have HPA setup.

And depending on the set limits, you would want to set FUNCTIONS_WORKER_PROCESS_COUNT to a lower value since more worker processes come with CPU/Memory overhead.
Ana Cristina Stefan 1 Reputation point

2021-08-02T06:20:43.12+00:00

@Pramod Valavala
Thanks for your reply. We are not running this function app in kubernetes hence resource requests/limits are not relevant. This is a function app running as a PaaS in azure.
Pramod Valavala 20,626 Reputation points Microsoft Employee

2021-08-02T08:04:16.283+00:00

@Ana Cristina Stefan Got it. You mentioned "kubernetes function app" so I had to ask :)

On Azure, could you share the plan that you are deploying to?

Share via

Azure function not executing queue messages in parallel

Your answer