Problem with downloading large numbers of files from Azure Storage

Question

I need to download a large number of files from Azure Storage. Then I need to create a zip file and send it to the client. If I do this in synchronous way, then the system will freeze for a significant amount of time to complete the process and download the zip file. Because the number of files could be more than 1000 and the size could be large.

So, I am taking a different approach. I will create a Background Service to handle the request. Here are the steps.

-         Download all the files.

-         Create a zip file and store that zip file to the azure container.

-         Create a link of that zip file and send that link to the client.

But I have a problem here. Should I download all the files into the Memory and create a zip file in the memory and then upload that zip file from memory to azure container? Remember my application is hosted into the Azure App Service. If I follow this step should it not consume lots of memory?

 

Or should I download all the files to a location in the Azure App Service and then convert that folder to a zip file and then upload that file to azure container? But would Azure allow me to download all the files into a directory within the application's execution context in azure host?

 

Can anyone suggest to me which way I should go?

Answer 1

Because you are dealing with large files, I would offload that.

Depending on the number of files, you could download it to an Azure file share (or fileserver using Hybrid workers), mount it - then zip it using an Azure Automation runbook, and upload it to Azure Blob storage with SFTP support, then the clients can grab it from there.

If it were only a small amount of files, I would have looked at Webjobs.

I'm not sure what files or what your app service is doing, but sounds like these files might be better stored in Azure blob storage in the long run, vs staying in the App Service.