GatewayTimeout' while invoking 'POST

Question

We've developed an Azure Function in python that connect to a Blob Storage, read files and writes into Azure tables. During the process with pipeline, using Azure Functions & working fine for small files. The problem is that, when I try to execute bigger file (more 200 MB) it's giving below errors. So, can you please help how fix this issue.

Call to provided Azure function 'csv-validator' failed with status-'GatewayTimeout' while invoking 'POST' on '(https://-.azurewebsites.net)' and message - '504.0 GatewayTimeout'.

another one saw is Call to provided Azure function 'csv-validator' failed with status-'499' while invoking 'POST' on '[https://-.azurewebsites.net]' and message - 'Invoking Azure function failed with HttpStatusCode - 499.'.

Answer 1

Did you adjust the timeout settings for your Azure Function ? if you're using a Premium or Dedicated (App Service) plan, increase the timeout limit from the default.

If possible, try to streamline your function’s code for faster processing, such as:

• Using asynchronous methods where possible.
• Optimizing read/write operations with Blob Storage and Azure Tables.

For files over 200 MB, consider splitting the file into smaller chunks and processing each one in parallel or sequentially. You can use ADF to orchestrate chunking before calling the function.

If the function requires substantial processing time, consider implementing Durable Functions. They allow for long-running operations and better management of state between chunks.