How To Resolve Azure Function `Stream Too Long` Error

Question

How To Resolve Azure Function `Stream Too Long` Error

Gerald Rupp 130

Hi everybody,

I am working at the moment on an azure function blob trigger. I want to read out a json file and process it. It works as expected but once the file is very large (~2.5 GiB), things begin to fail. I already increased function timeout to 30min in host.json but the error still appears.

"functionTimeout": "00:30:00"

It seems the file is to big to store in memory. Is this assumption correct? And how can I solve that problem?
With kind regards,
Gerald

Gerald Rupp 130

I've just observed another detail. I'll share my code for the Blobtrigger, followed by the error message below. From my analysis, it appears that the function never reaches the first line of the Blobtrigger function

logging.warning('START OF FUNCTION')

Consequently, I believe the error isn't tied to issues like running out of memory or the file being too large, as I haven't opened it yet. Do you share this assumption?

This is my blobtrigger code (init_.py)_

async def main(myblob: func.InputStream, starter: str):
    logging.warning('START OF FUNCTION')
    # Get the blob URI
    blob_uri = myblob.uri
    
    logging.info(f"Triggered by = '{blob_uri}'")
    
    if blob_uri.endswith('.json')==True:
        json_file='/'.join(myblob.name.split('/')[1:])
        print(json_file)
        # Create a DurableOrchestrationClient
        client = df.DurableOrchestrationClient(starter)

        # Start the orchestrator function and pass the blob URI as input
       
        instance_id = await client.start_new('OrchestratorRailster', None, {"blobjson": json_file})
        logging.warning(f"Started orchestration with ID = '{instance_id}'")
        

        
    else:
        logging.warning(f"no json-file was modified. No Action for: '{blob_uri}'")


    logging.warning(f"Ending orchestration with ID = '{instance_id}'")

And this is the error code:

[2023-11-21T12:33:19.024Z] Worker process started and initialized.
[2023-11-21T12:33:22.919Z] Host lock lease acquired by instance ID '000000000000000000000000AF1EBE1C'.
[2023-11-21T12:35:04.322Z] Executing 'Functions.BlobTrigger_railster' (Reason='New blob detected(LogsAndContainerScan): Stream was too long.', Id=8f57113b-8f3a-43cb-888b-3252cef42f9a)
[2023-11-21T12:35:04.324Z] Trigger Details: MessageId: 3d20c4c0-6d8f-4b3d-ba64-1d1ebe43d9e0, DequeueCount: 4, InsertedOn: 2023-11-21T12:28:29.000+00:00, BlobCreated: 2023-11-09T16:56:41.000+00:00, BlobLastModified: 2023-11-21T12:28:07.000+00:00
[2023-11-21T12:35:04.328Z] Executed 'Functions.BlobTrigger_railster' (Failed, Id=8f57113b-8f3a-43cb-888b-3252cef42f9a, Duration=105724ms)
[2023-11-21T12:35:04.329Z] System.Private.CoreLib: Exception while executing function: Functions.BlobTrigger_railster. Microsoft.Azure.WebJobs.Host: Exception binding parameter 'myblob'. System.Private.CoreLib: Stream was too long.

I was thinkting of creating a durable function and reading out the very big json file chunk by chunk. But will this really help due to the fact that the error appears before even opening the file?

Thank you very much for all your help :)

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
MayankBargali-MSFT 70,981 Reputation points Moderator

2023-11-22T11:13:59.9733333+00:00

@Gerald Rupp Thanks for reaching out.

Python specific case, in general no size limit on functions in general, however what often happens is how the trigger works. For example by default trigger passes in a "string" or a "byte[]" of data, which means all the data is loaded in memory and then passed to the function. Usually an "out of memory" exception when the content gets to big. To get around this in .NET at least, you can pass in the type of Stream so that the data isn't fully loaded in but streamed in (and ideally out) of the function and can process massive files. For non .NET languages there will definitely be a chance of memory contention here.

I don't see specifically call out for python function. But it looks like it may be load the data in stream. You can reach out to the expert here to confirm.

Can you confirm if you see multiple function getting executed on your function app as this would cause due to concurrent request utilizing all the memory of your function app. Have you review the diagnose and solve problem blade on your function app to review the memory issue.

The workaround could be use of event grid function rather than blob trigger function and use storage SDK to read the content as chunks.
Gerald Rupp 130 Reputation points

2023-11-23T10:15:26.3966667+00:00
Hi @MayankBargali-MSFT,
thank you very much for your response and assistance! :)

I am currently working with Durable Functions, executing multiple functions. However, as I mentioned in my comment, my script never reaches the point where these functions are executed. Therefore, I believe your explanation makes complete sense. My trigger attempts to load the file into memory but fails because the file is too large.

Have I understood that correctly so far?

What I don't understand is how I can instruct my function to stream the data.
To stream the data I would alter my code in a way like:

import azure.functions as func def main(blob: func.InputStream): # Access blob information directly from the function parameters blob_name = blob.name container_name = blob.container_name # Your existing script logic to process the blob in chunks chunk_size = 4 * 1024 * 1024 # 4 MB offset = 0 while offset < blob.length: chunk_data = blob.read(offset=offset, length=chunk_size) ### do something with chunk_data offset += chunk_size

Here, the Blobtrigger also passes the blob (in memory!?) to my script. I don't quite understand where something changes here, as my script crashes right from the start.

Thanks a lot for your help and kind regards,
Gerald
MayankBargali-MSFT 70,981 Reputation points Moderator

2023-11-27T10:15:35.62+00:00

@Gerald Rupp Your understanding is correct. But looks like the blob is loaded into the memory. Can you create the issue here so the experts can confirm and assist.
Gerald Rupp 130 Reputation points

2023-11-27T11:47:34.3366667+00:00

I created this issue here:

https://github.com/Azure/azure-sdk-for-python/issues/33296

Your answer

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
MayankBargali-MSFT 70,981 Reputation points Moderator

2023-11-22T11:13:59.9733333+00:00

@Gerald Rupp Thanks for reaching out.

Python specific case, in general no size limit on functions in general, however what often happens is how the trigger works. For example by default trigger passes in a "string" or a "byte[]" of data, which means all the data is loaded in memory and then passed to the function. Usually an "out of memory" exception when the content gets to big. To get around this in .NET at least, you can pass in the type of Stream so that the data isn't fully loaded in but streamed in (and ideally out) of the function and can process massive files. For non .NET languages there will definitely be a chance of memory contention here.

I don't see specifically call out for python function. But it looks like it may be load the data in stream. You can reach out to the expert here to confirm.

Can you confirm if you see multiple function getting executed on your function app as this would cause due to concurrent request utilizing all the memory of your function app. Have you review the diagnose and solve problem blade on your function app to review the memory issue.

The workaround could be use of event grid function rather than blob trigger function and use storage SDK to read the content as chunks.
Gerald Rupp 130 Reputation points

2023-11-23T10:15:26.3966667+00:00

Hi @MayankBargali-MSFT,
thank you very much for your response and assistance! :)

I am currently working with Durable Functions, executing multiple functions. However, as I mentioned in my comment, my script never reaches the point where these functions are executed. Therefore, I believe your explanation makes complete sense. My trigger attempts to load the file into memory but fails because the file is too large.

Have I understood that correctly so far?

What I don't understand is how I can instruct my function to stream the data.
To stream the data I would alter my code in a way like:

import azure.functions as func def main(blob: func.InputStream): # Access blob information directly from the function parameters blob_name = blob.name container_name = blob.container_name # Your existing script logic to process the blob in chunks chunk_size = 4 * 1024 * 1024 # 4 MB offset = 0 while offset < blob.length: chunk_data = blob.read(offset=offset, length=chunk_size) ### do something with chunk_data offset += chunk_size

Here, the Blobtrigger also passes the blob (in memory!?) to my script. I don't quite understand where something changes here, as my script crashes right from the start.

Thanks a lot for your help and kind regards,
Gerald
MayankBargali-MSFT 70,981 Reputation points Moderator

2023-11-27T10:15:35.62+00:00

@Gerald Rupp Your understanding is correct. But looks like the blob is loaded into the memory. Can you create the issue here so the experts can confirm and assist.
Gerald Rupp 130 Reputation points

2023-11-27T11:47:34.3366667+00:00

I created this issue here:

https://github.com/Azure/azure-sdk-for-python/issues/33296

Share via

How To Resolve Azure Function `Stream Too Long` Error

Your answer