Functions does not seem to scale as I had hoped. A lot of missing function calls.

Question

Functions does not seem to scale as I had hoped. A lot of missing function calls.

Le, Nghia 40

Plan: Flex plan.
Premise: We are testing the performance of the functions, uploading about 80k documents to a azure container. The BlobTrigger has a source of EventGrid.
Using: azcopy copy <source> <destination> -- recursive to copy 80k data.

Message Queues: Delivered 80k msgs, no errors nothing in the dead letter queue.
The function within the function app itself has one job. Print out data when it gets invoked, and we use application insights to query the invocation counts of the print outs to the messages in the log.

Variables: I've played with:
function memory (2048, 4096),
scaling,
batching. (The more I add to batch, the less invocations the functions get) So far Batch of 1 seems to yield the best results.
batch size: 64 - 512, 1024

All of my testing comes back with inconclusive or erratic results. There was only one time, where I managed to get 80k messages to the queue, and pull of 80k function invocations.

The odd thing is, there are no errors, nothing that would tell me why It's missing function calls. I would think messages get consumed. Things will get picked up in a queue. If it times out and can't finish in time, it'll log it. Here, it just doesn't call and doesn't tell us why.

Has anyone done this sort of testing on the event grid and functions before with better sucess?

Ryan Hill 30,326 Reputation points Microsoft Employee Moderator

2025-01-09T19:17:25.27+00:00

Hi @Le, Nghia, just to confirm, you're doing something similar to Tutorial: Trigger Azure Functions on blob containers using an event subscription | Microsoft Learn. If so, I can try to repro; see if I encounter the same. How far off are the invocations? What's your current stack and storage account SKU?
Le, Nghia 40 Reputation points

2025-01-09T19:46:28.8166667+00:00
Hi, @Ryan Hill yes essentially. I followed that trigger on azure functions on blob containers. Using the new event grid and flex plans. The function is simple. I just want to see if there's a 1:1 invocation between a blob that is created, we can act upon each invocation reliably.

I believe we have a pretty high tier support? I can provide that info if it helps. Please let me know if you have any questions, I've been evaluating this feature for the past 2+ days and was about to make a recommendation of ("not reliable") but I refuse to think that it can't handle this case. It has to be something I'm doing wrong.
I don't get consistent results. One run it will do all the invocations, one it will be missing 5% 10% some arbritary number of invactions, with no errors in app insights. Oddly enough, I find if I increase the messaging Batches to 10 or 50, it makes it worse.

Current Stack: Java
Destination Container (where trigger is)
Storage Account SKU Tier: Standard
Kind: StorageV2

Location:eastus2

Testing Premise:
Take a directory AZ storage container directory with Active blobs: 51,862 blobs, 10.42 GiB (11,188,458,368 bytes).
Then azcopy copy <src> <dest> --recursive
Where <dest> is where the trigger is listening.

Sample: And it fluctuates. Function Invocations

1/9/25 09:54AM 51488 calls. #missing 364 Expected 51862 Function Invocations

Test Env:

Function App Flex
East US2,
Instance Memory 4096
Max Instance Count: 40

@FunctionName("EventGridBlobTriggerJavaEvaluation") @StorageAccount("SANDBOX_BLOB_STORAGE") public void run( @BlobTrigger(name = "documentProcessingEventGridBlobTrigger", source = "eventGrid", path = "%"+DOC_MIGRATION_ROOT_CONTAINER+"%/{name}", dataType = "binary", connection = "SANDBOX_BLOB_STORAGE") byte[] content, @BindingName("BlobTrigger") String wholePath, final ExecutionContext context) { try { String fileLocation = System.getenv().get(DOC_MIGRATION_ROOT_CONTAINER); context.getLogger().info("### Start processing a document.... \n " + "wholePath: [" + wholePath + "] \n" "size: [" + content.length + "]"); } catch(Exception cause) { context.getLogger().log(Level.SEVERE, "### FAILED to process due to : " + cause.getMessage(), cause); throw cause; } }

Storage:
Events Grid Topic:
Type: Blob Created (only)
Subject Filtering : /blobServices/default/containers/poc-test/blobs
type: Webhook
Features: Max delivery attempts 20
Dead Letter Queue Enabled
Max Events Per batch : 1
Preferred Batch Size in KB: 512
Ryan Hill 30,326 Reputation points Microsoft Employee Moderator

2025-01-09T22:27:35.65+00:00

@Le, Nghia please respond the private message so I can assist you further.
Le, Nghia 40 Reputation points

2025-03-14T15:47:19.6233333+00:00

Hi, @Ryan Hill apologies, I forgot to follow up with you on this. I did work with an amazing team of tech support people. On 2/24 we resolved the issue. So it turns out the new type of Plan "flex consumption" out of the box couldn't really handle the load we were giving it, and the Event Grid's “delivery and retry” feature kicked in and caused high delays.
The solution was to go with premium functions plan and EP2 which resolved the issues if functions not firing. Again, thank you so much for stepping in and helping out in the matter.

Answer accepted by question author

0 additional answers

Your answer

Ryan Hill 30,326 Reputation points Microsoft Employee Moderator

2025-01-09T19:17:25.27+00:00

Hi @Le, Nghia, just to confirm, you're doing something similar to Tutorial: Trigger Azure Functions on blob containers using an event subscription | Microsoft Learn. If so, I can try to repro; see if I encounter the same. How far off are the invocations? What's your current stack and storage account SKU?
Ryan Hill 30,326 Reputation points Microsoft Employee Moderator

2025-01-09T22:27:35.65+00:00

@Le, Nghia please respond the private message so I can assist you further.
Le, Nghia 40 Reputation points

2025-03-14T15:47:19.6233333+00:00

Hi, @Ryan Hill apologies, I forgot to follow up with you on this. I did work with an amazing team of tech support people. On 2/24 we resolved the issue. So it turns out the new type of Plan "flex consumption" out of the box couldn't really handle the load we were giving it, and the Event Grid's “delivery and retry” feature kicked in and caused high delays.
The solution was to go with premium functions plan and EP2 which resolved the issues if functions not firing. Again, thank you so much for stepping in and helping out in the matter.

Answer 1

Hello Le, Nghia,

Thank you for your patience and for sharing your feedback on the Q&A community platform. I’m glad to hear that you were able to resolve your issue, and I appreciate you sharing your solution! Your contribution is valuable and can help others in the community facing similar challenges.

As per the Microsoft Q&A community policy, "The question author cannot accept their own answer. They can only accept answers by others"

I’m reposting your solution here so you can mark it as accepted if it resolves your query:

The issue was related to the "flex consumption" plan, which couldn't handle the load. The Event Grid's “delivery and retry” feature caused significant delays. The solution was to switch to the premium functions plan and EP2, which resolved the problem of functions not firing.

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.

Share via

Functions does not seem to scale as I had hoped. A lot of missing function calls.

0 additional answers

Your answer