Got error in azure function durable when uploading to mounted file share

Question

Hi! I have Azure Function Durable which uses "copernicusmarine" API to upload three files (each around 2 GB) to mounted Azure file share. If it is done sequentially then everything goes fine. In parallel:

Is this the problem with max size?

Answer 1

Hello Yauheni Kisialiou,

Greetings! Welcome to Microsoft Q&A Platform.

I understand that you are encountering issues with parallel uploads in your Azure Durable Function. This could be related to several factors, including resource contention or limitations in handling large files concurrently.

Durable Functions is an extension of Azure Functions that lets you build serverless orchestrations using ordinary code. For more information on Durable Functions, see the Durable Functions overview. Durable Function orchestrations are implemented in code and can use the programming language's built-in error-handling features. There really aren't any new concepts you need to learn to add error handling and compensation into your orchestrations. However, there are a few behaviors that you should be aware of to resolve the errors,

• Ensure that your Durable Function is properly managing concurrency. You can use the Task.WhenAll method to handle parallel tasks but be mindful of the resource limits.
• Implement robust error handling in your orchestrator function. This can help you catch and manage exceptions more effectively.
• Check if your Azure File Share or the underlying storage account has any limits that might be causing the issue. Sometimes, increasing the storage tier or adjusting the performance settings can help.
• Use Azure Monitor and Application Insights to track the performance and errors of your Durable Function. This can provide more insights into what’s going wrong.
• Consider using the fan-out/fan-in pattern to manage parallel tasks more efficiently. This pattern allows you to distribute the workload across multiple functions and then aggregate the results.
• Since the error mentions HDFS dimscales, ensure that your HDFS configuration is optimized for handling large files and concurrent access.

reference docs- https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-troubleshooting-guide,

https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-error-handling?tabs=csharp-inproc,

https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-cloud-backup?tabs=csharp

Hope the answer helps! please let us know if you have any further queries. I’m happy to assist you further.

---

Please "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.