Optimizing Data Transfer: Challenges and Strategies with Azure Durable Functions and Azure Data Factory

Question

I’m exploring solutions for transferring larger data using Azure Durable Functions in conjunction with Azure Data Factory. Currently, I’ve implemented the Durable Function template (Starter, Orchestration, and Activity). While I anticipated that the durable function would address moving large data, I’m encountering a size limit error in Azure Data Factory (ADF). Specifically, during execution, I hit the ADF size limit error of around 4MB, necessitating manual adjustment of paging size to pass data, which is a cumbersome process.

I aim to stay within the Consumption plan of Azure Functions for economic reasons. Even when attempting a higher tier, like Premium, it only passes through the Azure Function area, increasing memory but still encountering a 4 MB limit error at the Web portion.

I’m curious about the best strategy to efficiently move larger data from Azure Function to ADF. Currently, I use Azure Blob files for storing data (both input and output in text files). Are there any settings I could configure in Blob storage or within ADF to make a difference? Given that ADF is designed to handle substantial data volumes in other scenarios, I’m puzzled about the source of this limitation—whether it’s in Blob storage, Azure Functions, or elsewhere.

Answer 1

Hi @Glasier ,

Thank you for reaching out to us with your query. 

As I understand that you are encountering a size limit error in Azure Data Factory (ADF) when transferring larger data using Azure Functions, please let me know if that is not.

Unfortunately, this is a hard limit on Web activity where the response size should be less than or equal to 4MB. Plese refer this documentation.

Azure Resource Manager has certain restrictions on the number of API calls you can make in a given period. Be sure to keep your call rate within these API limits. Additionally, the API schema used by a validation policy cannot exceed 4 MB in size. If the schema is larger than this, validation policies will encounter errors during runtime. For more details, please refer to this documentation: API Management policies to validate requests and responses.

And as a workaround you may try using ForEach Activity. Maybe you need to use paging query for your rest api/endpoint and return a limited number of the data each time. Then query your data in loop until the number of return data is lower than threshold value. Please refer to this source: Web activity throws overlimit error when calling rest api

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.