Azure Function to read large file in chunks through http trigger request

AshikI 11 Reputation points
2021-07-12T07:05:41.18+00:00

Hi
I have an API in Azure function that uses Http trigger to receive data and send that across to the on-prem application. We have the UI front-end where user can upload large file size (no limit) and that would send data in chunks to the API.

I am aware that the limitation of the function App is 100MB and I also see the recommendation for handling large file size should be done using Blob Storage. However for synchronous process, we wanted to achieve this through the API (avoiding storing data in blob storage in the intermediate process).

Is there a way how I can receive data in chunks via the Http trigger request? eg: UI -> send data in chunks -> API (uses HttpTrigger) -> send data in chunks -> on-prem.

Appreciate your help in this regard.

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,264 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Samara Soucy - MSFT 5,051 Reputation points
    2021-07-14T21:46:09.653+00:00

    The main issue with chunking up the data this way is that Functions are stateless. So if you make multiple calls to your Function, each with 100MB of data, they won't be able to put the data back together without storing it somewhere outside of the function' s process. If you absolutely had to do it that way, the best way to do it would be to write each chunk to storage as it comes in. You'd still be using Storage as a staging ground, but now the processing of that file is more complicated than it would be if you write directly to storage.

    I'm trying to understand your comment about trying to make this a synchronous process- is the goal to be able to return a message to the user when the file has been successfully written to on-prem? This could still be done with functions, even with writing to storage in between. Durable Functions would be best suited to doing this since it offers tools for monitoring long running tasks like writing to storage. Keep in mind that HTTP triggered functions have a hard limit of 230 seconds to respond, so even a larger file would need to complete copying in that timeframe.

    UI -> Writes to storage and calls HTTP triggered function with info on where to find the file -> 1st activity function polls the storage account waiting for the file to complete uploading. -> 2nd activity retrieves the file and writes it on-premises -> (optional) if successful, go ahead and ask storage to delete it's copy -> return result to UI

    0 comments