Functions does not seem to scale as I had hoped. A lot of missing function calls.

Le, Nghia 0

Plan: Flex plan.
Premise: We are testing the performance of the functions, uploading about 80k documents to a azure container. The BlobTrigger has a source of EventGrid.
Using: azcopy copy <source> <destination> -- recursive to copy 80k data.

Message Queues: Delivered 80k msgs, no errors nothing in the dead letter queue.
The function within the function app itself has one job. Print out data when it gets invoked, and we use application insights to query the invocation counts of the print outs to the messages in the log.

Variables: I've played with:
function memory (2048, 4096),
scaling,
batching. (The more I add to batch, the less invocations the functions get) So far Batch of 1 seems to yield the best results.
batch size: 64 - 512, 1024

All of my testing comes back with inconclusive or erratic results. There was only one time, where I managed to get 80k messages to the queue, and pull of 80k function invocations.

The odd thing is, there are no errors, nothing that would tell me why It's missing function calls. I would think messages get consumed. Things will get picked up in a queue. If it times out and can't finish in time, it'll log it. Here, it just doesn't call and doesn't tell us why.

Has anyone done this sort of testing on the event grid and functions before with better sucess?

Ryan Hill 29,481 Reputation points Microsoft Employee

2025-01-09T19:17:25.27+00:00

Hi @Le, Nghia, just to confirm, you're doing something similar to Tutorial: Trigger Azure Functions on blob containers using an event subscription | Microsoft Learn. If so, I can try to repro; see if I encounter the same. How far off are the invocations? What's your current stack and storage account SKU?
Le, Nghia 0 Reputation points

2025-01-09T19:46:28.8166667+00:00
Hi, @Ryan Hill yes essentially. I followed that trigger on azure functions on blob containers. Using the new event grid and flex plans. The function is simple. I just want to see if there's a 1:1 invocation between a blob that is created, we can act upon each invocation reliably.

I believe we have a pretty high tier support? I can provide that info if it helps. Please let me know if you have any questions, I've been evaluating this feature for the past 2+ days and was about to make a recommendation of ("not reliable") but I refuse to think that it can't handle this case. It has to be something I'm doing wrong.
I don't get consistent results. One run it will do all the invocations, one it will be missing 5% 10% some arbritary number of invactions, with no errors in app insights. Oddly enough, I find if I increase the messaging Batches to 10 or 50, it makes it worse.

Current Stack: Java
Destination Container (where trigger is)
Storage Account SKU Tier: Standard
Kind: StorageV2

Location:eastus2

Testing Premise:
Take a directory AZ storage container directory with Active blobs: 51,862 blobs, 10.42 GiB (11,188,458,368 bytes).
Then azcopy copy <src> <dest> --recursive
Where <dest> is where the trigger is listening.

Sample: And it fluctuates. Function Invocations

1/9/25 09:54AM 51488 calls. #missing 364 Expected 51862 Function Invocations

Test Env:

Function App Flex
East US2,
Instance Memory 4096
Max Instance Count: 40

@FunctionName("EventGridBlobTriggerJavaEvaluation") @StorageAccount("SANDBOX_BLOB_STORAGE") public void run( @BlobTrigger(name = "documentProcessingEventGridBlobTrigger", source = "eventGrid", path = "%"+DOC_MIGRATION_ROOT_CONTAINER+"%/{name}", dataType = "binary", connection = "SANDBOX_BLOB_STORAGE") byte[] content, @BindingName("BlobTrigger") String wholePath, final ExecutionContext context) { try { String fileLocation = System.getenv().get(DOC_MIGRATION_ROOT_CONTAINER); context.getLogger().info("### Start processing a document.... \n " + "wholePath: [" + wholePath + "] \n" "size: [" + content.length + "]"); } catch(Exception cause) { context.getLogger().log(Level.SEVERE, "### FAILED to process due to : " + cause.getMessage(), cause); throw cause; } }

Storage:
Events Grid Topic:
Type: Blob Created (only)
Subject Filtering : /blobServices/default/containers/poc-test/blobs
type: Webhook
Features: Max delivery attempts 20
Dead Letter Queue Enabled
Max Events Per batch : 1
Preferred Batch Size in KB: 512
Ryan Hill 29,481 Reputation points Microsoft Employee

2025-01-09T22:27:35.65+00:00

@Le, Nghia please respond the private message so I can assist you further.

Share via

Functions does not seem to scale as I had hoped. A lot of missing function calls.

Your answer