Connection Timeout and Internal Server Error 500 with Azure Function Handling Large Files from ADLS Gen 2

Question

Connection Timeout and Internal Server Error 500 with Azure Function Handling Large Files from ADLS Gen 2

Mayur Patil 35

I have created a function in my local environment that performs the following tasks: it reads files from ADLS Gen 2, uploads them to a temporary directory, truncates specific fields, and resizes the data to a maximum size of 3900. After these modifications, the files are uploaded back to ADLS Gen 2. This function works seamlessly for smaller files (approximately 1 MB). However, for larger files, I implemented chunking to read data from storage, which takes around 5 minutes. The issue arises when attempting to upload the modified file—it results in a 'connection timeout' error. This issue persists even when chunking is utilized, both in the local environment and in Azure Function. Additionally, for larger files, the Azure Function using a consumption plan throws an 'Internal Server Error 500.' I am seeking advice on resolving these problems.

1 answer

Your answer

Answer 1

Alex Burlachenko 10,335

Hello Mayur Patil,

thanks for post your questio here, so I can see 2 ways short term and long term...

short term fix:

implement chunked uploads with a 4 MB chunk size and a 300-second timeout in the SDK. Increase the function timeout to 10 minutes in host.json. Add retry logic for uploads.

long term (probably solutions):

switch to the premium plan for longer timeouts and better resource allocation. Optimize memory usage with streaming and incremental uploads. Monitor ADLS Gen 2 metrics to adjust concurrency if needed.

By addressing both the timeout (via chunking and retries) and the 500 error (via timeout extension or plan upgrade), your function should handle large files more reliably.

Best regards,

Alex

P.S. If my answer help to you, please Accept my answer

Mayur Patil 35 Reputation points

2025-03-26T08:46:20.3566667+00:00

Hi @Alex Burlachenko , thank you for providing detailed information. In the local environment(Http Trigger), I tested various chunk sizes, including 4 MB, 50 MB, and 100 MB, and encountered no issues while downloading files from ADLS Gen 2. However, when modifying the file and attempting to upload it back to ADLS Gen 2, I am getting an error: 'An error occurred: ('Connection aborted.', TimeoutError('The write operation timed out')'. This issue persists despite specifying the function timeout duration in the host.json file
Alex Burlachenko 10,335 Reputation points

2025-03-26T09:01:03.88+00:00

Mayur,

tell me pls is u storage account is in the same Azure region as your function?

No VPN& ip restrictions?

rgds,

Alex
Mayur Patil 35 Reputation points

2025-03-26T09:46:27.48+00:00

@Alex Burlachenko - The storage and Azure Function are in the same location, with no VPN or IP restrictions in place. Additionally, the information I shared earlier is related to the local environment

Alex Burlachenko 10,335

lets exclude network issue...
Run this PowerShell command (tests raw upload speed to Azure)

Good result: Upload completes in <2 minutes (≈800 KB/s+)

Network issue: Fails or takes >5 minutes

# Replace with your storage account name
$storageAccount = "yourstorageaccount"
$filePath = "$env:TEMP\test_100mb.bin"

# Create a 100MB test file
fsutil file createnew $filePath 100000000

# Time the upload
Measure-Command {
    az storage blob upload --account-name $storageAccount `
        --container-name "$tmp" `
        --file "$filePath" `
        --name "network_test.blob"
}

Mayur Patil 35 Reputation points

2025-03-26T13:24:59.6133333+00:00

Hi @Alex Burlachenko I executed the PowerShell command, and it took 10 minutes to upload the file to Azure Storage
Alex Burlachenko 10,335 Reputation points

2025-03-26T13:26:28.0633333+00:00

Seems it is network issue
Mayur Patil 35 Reputation points

2025-03-26T13:30:53.9966667+00:00

@Alex Burlachenko - If it's a network issue, why does it only affect larger files?
Alex Burlachenko 10,335 Reputation points

2025-03-26T14:20:28.8666667+00:00
Hmmm, good question )))) small files upload quickly before any network problems can interrupt them. Large files take longer, so minor issues (like slow speeds or brief disconnections) add up until Azure gives up and throws a timeout. Your 10min upload for a 100MB file proves the network is the problem usually it should take under 1 minute on a good connection. How to confirm it’s the network? You can test from another location (e.g., home or mobile hotspot). If it’s faster, your office network is throttling you.

Run this command to check for packet loss (replace yourstorageaccount):

ping yourstorageaccount.blob.core.windows.net -n 100

If you see "Request timed out" or high latency (>100ms), the network is unstable.

How to fix it? Force smaller chunks (bypasses t1hrottling) c#

await blobClient.UploadAsync(data, new BlobUploadOptions { TransferOptions = new StorageTransferOptions { MaximumTransferSize = 4 * 1024 * 1024 // 4MB chunks } });

Use HTTP/2 (more stable for slow networks) c#

var client = new BlobClient(connectionString, options: new BlobClientOptions { Transport = new HttpClientTransport(new HttpClient { DefaultRequestVersion = HttpVersion.Version20 }) });

Upload to Blob Storage first, then move to ADLS Gen2 (Blob Storage handles slow networks better).

If nothing works

Contact your network team or ISP and ask:

"Are you throttling Azure Storage uploads? Can you whitelist *.blob.core.windows.net?"

As well try during off-hours (when network traffic is lower).

The issue is 100% fixable—start with chunking and HTTP/2.

rgds,

Alex

Share via

Connection Timeout and Internal Server Error 500 with Azure Function Handling Large Files from ADLS Gen 2

1 answer

Your answer