Azure Functions - Concurrent requests throttle function response for all requests

Question

Hey there!

Overview

I have a simple function that is quite resource intense. It generates an image using the Magick.NET package. The average response for one concurrent request is around 1 second, however, as more concurrent requests are added, the response time expands exponentially. (5 seconds for 3 concurrent requests, 25 seconds for 10 concurrent requests) This delay occurs to all requests regardless of invocation order - All requests fire off, then after a delay all requests respond at once.

Analysing the console during concurrent requests, I can see that each request is getting jammed on the Magick.NET image processing logic. I can only assume the issue is memory related, but my understand was that Azure Functions scaled the instance count to meet processing demands.

I'll attach a redacted version of the full function script at the bottom of this issue.

If you require any further information, just let me know. Thanks :)

Pseudo

General overview of the flow of the function

• Function Run (http request)
• Download random image file from blob storage
• Process file with Magick.NET
• Return file as byte array

Specs

Azure Function specification

• Dotnet 6.0
• Azure Function Runtime - v4.0
• Platform - 32bit

Attempted Solutions

To fix this, I have tried

• Increasing the Scale Out limit in Azure Portal to 200
• Modifying the maxConcurrencyRequests parameter in host.json
• Running the Magick.NET logic async
• Setting 'FUNCTIONS_WORKER_PROCESS_COUNT' application setting to 10

Host.json

{
    "version": "2.0",
    "logging": {
        "applicationInsights": {
            "samplingSettings": {
                "isEnabled": true,
                "excludedTypes": "Request"
            }
        }
    },
    "http": {
        "maxConcurrentRequests": 100
    }
}

Function.cs

public static class Generate
{
    [FunctionName("Generate")]
    public static async Task<IActionResult> Run([HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req, ILogger log, ExecutionContext context)
    {
        string responseMessage = "ok";
        log = logger;

        try
        {
            string Title = req.Query["title"];
            string Author = req.Query["author"];

            string FileName = "image.png";
            string blobConnection = config["AzureWebJobsStorage"];
            string blobContainer = config["AzureBlobStorageImageDirectory"];

            byte[] FileData = DownloadBlob(FileName, blobConnection, blobContainer);
            byte[] ImageData = GenerateImage(Title, Author, FileName, FileData);

            return new OkObjectResult(JsonConvert.SerializeObject(ImageData)) { StatusCode = 200 };
        }
        catch (Exception ex)
        {
            responseMessage = ex.Message;
            return new ObjectResult(responseMessage) { StatusCode = 500 };
        }
    }

    public static byte[] GenerateImage(string Title, string Author, string FileName, byte[] FileData)
    {
        try
        {
            MagickImage img = new MagickImage(FileData);
            img.Alpha(AlphaOption.Opaque);

            var authorlabel = new MagickImage("caption:" + Author, new MagickReadSettings { /* ... */ });
            var titlelabel = new MagickImage("caption:" + Title, new MagickReadSettings { /* ... */ });

            img.Composite(authorlabel, 10, 10, CompositeOperator.Over);
            img.Composite(titlelabel, 10, 100, CompositeOperator.Over);

            byte[] imageData = img.ToByteArray(MagickFormat.Png24);
            img.Dispose();

            return imageData;
        }
        catch (Exception ex)
        {
            log.LogError(ex, "ERR: An exception occurred in GenerateImage");
            throw;
        }
    }

    public static byte[] DownloadBlob(string FileName, string blobConnection, string blobContainer)
    {
        try
        {
            BlobContainerClient blobContainerClient = new BlobContainerClient(blobConnection, blobContainer);

            var blobClient = blobContainerClient.GetBlobClient(FileName);

            if (blobClient.ExistsAsync().Result)
            {
                using MemoryStream ms = new MemoryStream();
                Response blobResponse = blobClient.DownloadTo(ms);
                return ms.ToArray();
            }
            else
            {
                throw new Exception($"Blob client {blobClient.BlobContainerName}/{blobClient.Name} does not exist");
            }
        }
        catch (Exception ex)
        {
            log.LogError(ex, "ERR: An exception occurred in BlobDownload");
            throw;
        }
    }
}

Answer 1

Hello @Alasdair Hendry

Can you also try to up the limit on:

FUNCTIONS_WORKER_PROCESS_MEMORY_MB ?

Please let us know how it went !

In case this helped kindly mark it as Accepted!

BR