Azure blob storage - compression on upload and push raw on to the server

Question

Hello, I am new here and I have a first problem.
I wish to use blob storage to serve files in case clients need those files the first time only (downloaded mostly once). The main idea is to upload files to blob storage but at the time of uploading (or after), files need to be compressed in bz2 and appear on the storage.

The second part of the flow is to push files automatically (at the same time) to the Linux server as uncompressed/raw data.

Do I need an Azure event grid for it?

Should I change the type of storage from Hot to Cold?

Answer 1

Hi,

I don't think you required an event grid.

What you need is:

• An Azure function that will compress your files
• A power automate flow that will trigger when a blob is added, then call an Azure function to compress the file and use an action to store it to your Linux server.

The type of storage to choose, i.e., cold or hot, depends on how often the user(s) will access the files. 

https://learn.microsoft.com/en-us/azure/storage/blobs/access-tiers-overview

Answer 2

There is no other way to do it than to use Azure Function, so why do we have events it is used to send messages only?

I'm guessing this is my answer:
https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-event-overview

Answer 3

Hello Konrad Janczarek,

Greetings! Welcome to Microsoft Q&A Platform.To achieve your goal of compressing files in bz2 format when uploading to Azure Blob Storage and simultaneously pushing uncompressed files to a Linux server, you can indeed use Azure Event Grid along with other Azure Function services as stated above,

Upload your files to a Blob Storage container.

• Use an Azure Function triggered by the Blob Storage event (e.g., BlobCreated) to compress the uploaded file into bz2 format. This function can then save the compressed file back to the Blob Storage.
• Configure Event Grid to trigger an event when a new file is uploaded to Blob Storage.
• Use another Azure Function or Logic App to handle the event. This function can transfer the uncompressed file to your Linux server using SCP, SFTP, or another suitable method.

You can set up an Azure Function with a Blob Trigger, so it gets triggered whenever a new zip file is uploaded to a specific Blob Container. Utilize streaming to handle the contents of the zip file. This involves reading the file in smaller chunks, which helps to manage memory consumption effectively. Implement the logic to unzip the files in a streaming manner. You can use classes like ZipArchive from the .NET Framework and ZipInputStream from the SharpZipLib library. These classes allow you to read and process the contents of the zip file without loading the entire file into memory.

Process each file individually as it's streamed from the zip file and store the processed files in another Blob Container or any other suitable storage depending on your requirements.

Or else, instead of processing directly in the Logic App, move the blob to an Azure Blob Storage container. Create an Azure Function with a Blob Trigger to process the contents of the Azure Blob Storage container.

Use streaming to handle the zip file contents and unzip files individually and if a file is valid, move it to Linux server for further processing.

Since your files are expected to be downloaded mostly once, you might consider using Cool or Archive storage tiers to save costs. However, if you need faster access times for the initial download, Hot storage might still be preferable.

Example Workflow

Upload File: User uploads a file to Blob Storage.

Trigger Event: Azure Event Grid detects the new file and triggers an Azure Function.

Compress File: The Azure Function compresses the file to bz2 format and saves it back to Blob Storage.

Push to Linux Server: Another Azure Function or Logic App, triggered by the same event, transfers the uncompressed file to the Linux server.

Configuration Steps

Create a Blob Storage Account: Ensure it’s a StorageV2 (general purpose v2) account to support Event Grid integration.

Set Up Event Grid- Subscribe to Blob Created events for your storage account.

Deploy Azure Functions: One for compressing files and another for transferring files to the Linux server.

This setup ensures that your files are compressed and stored efficiently while also being available in their original form on your Linux server.

refer - https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob-trigger?tabs=python-v2%2Cisolated-process%2Cnodejs-v4%2Cextensionv5&pivots=programming-language-csharp#concurrency-and-memory-usage,

https://learn.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage?tabs=cloud-event-schema

Hope this answer helps! please let us know if you have any further queries. I’m happy to assist you further.

Please "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members