How to fix Redacted x-ms-range issue in azure functions python code

Question

Hi,

I have python code in azure functions which loops through all files that are zipped and encrypted using gpg. the following piece of code is working for small files but gives below error for actual big files. we are trying to read small chunks of data from encrypted file, decrypt it and upload to blob

Code :

import io
import logging
import os
import azure.functions as func
from azure.storage.blob import BlobServiceClient
import gnupg
import tarfile
app = func.FunctionApp()
def decrypt_gpg(enc_file_stream: io.BytesIO, gnupghome: str) -> io.BytesIO:
    logging.info("Decrypting the file: ")
    try:
        gpg = gnupg.GPG()
        decrypt_data = gpg.decrypt_file(enc_file_stream, passphrase=gnupghome)
        return io.BytesIO(decrypt_data.data)
    except Exception as e:
        logging.error(f"Error during GPG decryption: {str(e)}")
        raise
def extract_files_from_tar(decrypted_data: io.BytesIO, dest_container_name: str, blob_name: str,
                           blob_service_client: BlobServiceClient):
    logging.info("Extracting the files from blob:")
    folder_name = os.path.splitext(os.path.split(blob_name)[0])[0]
    with tarfile.open(fileobj=decrypted_data, mode='r') as tar:
        for member in tar.getmembers():
            if member.isfile():
                file_stream = tar.extractfile(member)
                file_name = member.name
                logging.info(f"Extracting file: {file_name}")
                if file_stream is not None:
                    dest_blob_name = os.path.join(folder_name, file_name)
                    logging.info(f"Uploading to blob path: {dest_blob_name}")
                    dest_blob_client = blob_service_client.get_blob_client(container=dest_container_name,
                                                                           blob=dest_blob_name)
                    upload_blob_in_chunks(file_stream, dest_blob_client)
except tarfile.TarError as e:
        logging.error(f"Error extracting tar file: {str(e)}")
        raise
    except Exception as e:
        logging.error(f"Error during the file extraction: {str(e)}")
        raise
def upload_blob_in_chunks(source_stream: io.IOBase, dest_blob_client,
                          chunk_size:int = 32*1024*1024) -> None:
    logging.info("Uploading the chunk")

I am getting this error. any help is much appreciated.

024-10-29T15:00:53Z   [Information]   I'm here! with chunk size <built-in method count of bytes object at 0x7fd1c088a4d0> 2024-10-29T15:00:53Z   [Information]   Request URL: ' Request method: 'GET' Request headers:     'x-ms-range': 'REDACTED'     'If-Match': '"0x8DCE4B6AF2E2F30"'     'x-ms-version': 'REDACTED'     'Accept': 'application/xml'     'User-Agent': 'azsdk-python-storage-blob/12.23.1 Python/3.11.10 (Linux-5.10.102.2-microsoft-standard-x86_64-with-glibc2.31)'     'x-ms-date': 'REDACTED'     'x-ms-client-request-id': '97bf8388-9606-11ef-bc8f-00155ddfbdf2'     'Authorization': 'REDACTED' No body was attached to the request 2024-10-29T15:00:53Z   [Information]   Response status: 206 Response headers:     'Content-Length': '4194304'     'Content-Range': 'REDACTED'     'Last-Modified': 'Fri, 04 Oct 2024 20:54:05 GMT'     'Accept-Ranges': 'REDACTED'     'ETag': '"0x8DCE4B6AF2E2F30"'     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'     'x-ms-request-id': '57ba173f-901e-0039-0c13-2a1e66000000'     'x-ms-client-request-id': '97bf8388-9606-11ef-bc8f-00155ddfbdf2'     'x-ms-version': 'REDACTED'     'x-ms-resource-type': 'REDACTED'     'x-ms-creation-time': 'REDACTED'     'x-ms-lease-status': 'REDACTED'     'x-ms-lease-state': 'REDACTED'     'x-ms-blob-type': 'REDACTED'     'x-ms-server-encrypted': 'REDACTED'     'x-ms-owner': 'REDACTED'     'x-ms-group': 'REDACTED'     'x-ms-permissions': 'REDACTED'     'x-ms-acl': 'REDACTED'     'Date': 'Tue, 29 Oct 2024 15:00:52 GMT' 2024-10-29T15:00:55Z   [Information]   Response status: 206 Response headers:

can someone help on this please

Raj

Answer 1

Hi Raj,

This looks like an issue I’ve seen before! Large, encrypted files in Azure Functions can quickly run into memory and timeout limits, especially with GPG-encrypted data. Here’s how I’d handle it:

Chunked Processing: Rather than loading the entire file, I’d read and decrypt it in smaller chunks. This approach keeps memory usage manageable and avoids overloading the function.

Chunked Blob Uploads: For uploading to Azure Blob Storage, I’d break it down into smaller chunks (around 4 MB tends to work well). Each chunk can be uploaded independently, making retries easier if a network interruption happens without risking duplicates or partial uploads.

Enhanced Logging: Logging each stage—decryption, extraction, and upload—helps trace any issues, whether they stem from decryption or network stability.

Setting overwrite=True: Using overwrite=True during uploads ensures that retries don’t create duplicates or leave incomplete data behind.

With this setup, Azure Functions should handle large files more smoothly, keeping memory usage in check and making the whole process more resilient.

I hope this helps.