Queue trigger function app not processing messaging on queue as expected

Question

I have a few issues that that recently started appearing with my function app that should be triggered by messages arriving on a queue.

Firstly, I want to limit the number of concurrent messages that my function app can process to 2 at a time. However, I have observed more than 2 messages being attempted to be processed concurrently.

Secondly, I have noticed when my app is idle for some time, the function app is no longer started up by a message arriving on the queue. It is only when I go to the azure portal and click on the function app does it wake up and dequeues the messages sitting in the queue. As I expect long periods of inactivity I want to use the consumption feature that the app is not running when it use, but responds to messages arriving on the queue. All of these issues only seem to have started recently - I did not notice any of these being an issue a few months ago (I have not pushed any updates that I expect to have altered the behavior).

Thirdly, I have noticed that messages seemingly stop mid-way through being processed and are moved back to the queue (where they then get stuck and are not immediately dequeued again as expected).

In my host.json I have

  "extensions": {
    "queues": {
        "maxPollingInterval": "00:00:05",
        "visibilityTimeout" : "00:00:00",
        "batchSize": 1,
        "newBatchThreshold": 1,
        "maxDequeueCount": 25,
        "messageEncoding": "base64"
    }
  }

In my ARM template, in the function app resource I have defined:

"type": "Microsoft.Web/sites",
"apiVersion": "2023-01-01",
"name": "[parameters('function_app_name')]",
"location": "[parameters('location')]",
"kind": "functionapp,linux,container,azurecontainerapps",
"properties": {
    "siteConfig": {
       ...
       "functionAppScaleLimit": 1,
        "minimumElasticInstanceCount": 0,
        "appSettings": [
            {
                "name": "FUNCTIONS_WORKER_PROCESS_COUNT",
                "value": "1"
            },
            {
                "name": "PYTHON_THREADPOOL_THREAD_COUNT",
                "value": "2"
            },
             ...
        ],
     ...
    },
    "workloadProfileName": "Consumption",
    "resourceConfig": {
        "cpu": 1,
        "memory": "2Gi"
    },
}

This is my function app docker file:

FROM mcr.microsoft.com/azure-functions/python:4-python3.11@sha256:1ed5b5711c5e0480b87420d49468b342bcd5446ebccaa11b435794511f0688c4    

# Set the working directory in the container
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
    AzureFunctionsJobHost__Logging__Console__IsEnabled=true \
    AzureWebJobsFeatureFlags=EnableWorkerIndexing

WORKDIR /home/site/wwwroot

# Install any additional Python dependencies
COPY requirements.txt /
RUN pip install --no-cache-dir -r /requirements.txt

# Copy the function app code to the working directory
COPY /app .

Answer 1

Samuel hi,

thanks for posting your question here )) i'll give u both microsoft specific fixes and some general tricks that might help elsewhere too.

for your concurrent processing limit, u got the right idea with 'functionAppScaleLimit' but let's tweak it further. in your host.json, add this under 'queues' section

"concurrency": {
    "dynamicConcurrencyEnabled": false,
    "maximumConcurrency": 2
}

this should force exactly 2 messages at a time https://learn.microsoft.com/en-us/azure/azure-functions/functions-host-json#queues

about the idle app not waking up - classic cold start issue with consumption plan. try adding 'alwaysReady' instances in your arm template

"siteConfig": {
    "alwaysReady": [
        {
            "size": "small",
            "count": 1
        }
    ]
}

this keeps one instance warm https://learn.microsoft.com/en-us/azure/azure-functions/functions-infrastructure-as-code#alwaysready

for messages getting stuck mid-process, your visibilityTimeout '00:00:00' looks suspicious. set it to something like '00:10:00' (10 mins) to give your function time to finish. also check your function code for unhandled exceptions - they cause messages to reappear in queue.

as well check this - in your dockerfile, add these env vars

ENV WEBSITE_CONTENTAZUREFILECONNECTIONSTRING="" \
    WEBSITE_CONTENTSHARE="your-functionapp-name"

this helps with cold starts on containerized functions. worth looking into.

when using python with azure functions, keep an eye on that threadpool count. '2' is good for your case, but sometimes bumping it to '4' helps with throughput. test different values ))

and one more thing - verify your storage account connection. queue triggers need solid storage connection. if its flaky, u get exactly these symptoms. this might help in other tools too when dealing with azure queues.

try these changes and see if it behaves better. let us know how it goes....

Best regards,

Alex

and "yes" if you would follow me at Q&A - personaly thx.
P.S. If my answer help to you, please Accept my answer

https://ctrlaltdel.blog/