Post to Azure Function with body over 10K doesn't reach Azure Function

Don Gillett 0

We have an Azure Function with an HTTP binding. It is bound to POST. The call works fine when we call it with a body smaller than 10K. When the size goes above that the call fails. The log steaming on the Azure Functions console doesn't show it even making it to the server. The default max size for an Azure function is documented to be 100Mb. We have also looked at all the posts we can find about setting the size of the allowed post and none of the suggested methods (host file entry and env variables in console) have any effect. When we run the Azure function on a local dev machine it looks fine. I suspect it has to do with networking config on the Azure side as the error we get back is "failed to copy content", followed by a bunch of SSL handshake failures... although we think the later is unrelated fallout from the actual failure. Note that we deployed a completely clean and default version of our service and it still fails. Many thanks in advance for idea to look at.

3 answers

Konstantinos Passadis 19,066 Reputation points MVP

2024-08-05T19:15:31.1+00:00
Hello @Don Gillett

Welcome to Microsoft QnA!

Since you already tried some settings i suggest

Try the Premium SKU for Azure Functions or Scale Up the Service

Also

Try this Setting

Name: WEBSITE_MAX_REQUEST_LENGTH

Value: 104857600 // 100MB in bytes

Also

Use Azure Application Insights to capture detailed logs and telemetry from your Azure Function. This might provide more information about what's happening when the request fails.

In your Azure Function, enable Application Insights and configure it to capture HTTP request and dependency telemetry.

Check Network Settings and what is between the caller and the endpoint

Kindly let us know with some insights from Application Insights as well

--

I hope this helps!

Kindly mark the answer as Accepted and Upvote in case it helped!

Regards
Please sign in to rate this answer.
Don Gillett 0 Reputation points

2024-08-05T19:44:42.2266667+00:00

Thanks for your reply!

When I throttle the amount of data we send a successful trace output on the Azure Function side is as follows...

2024-08-05T19:30:16Z [Verbose] Route value '(null)' with key 'httpMethod' did not match the constraint 'Microsoft.AspNetCore.Routing.Constraints.HttpMethodRouteConstraint'

2024-08-05T19:30:16Z [Verbose] Route value '(null)' with key 'httpMethod' did not match the constraint 'Microsoft.AspNetCore.Routing.Constraints.HttpMethodRouteConstraint'

2024-08-05T19:30:16Z [Verbose] Request successfully matched the route with name 'PostTrackedObjects' and template 'api/accounts/{accountId}/sites/{siteId}/trackedobjects'

2024-08-05T19:30:16Z [Information] Executing 'Functions.PostTrackedObjects' (Reason='This function was programmatically called via the host APIs.', Id=f4097913-4155-4b2d-88f5-f903d5112bf1)

2024-08-05T19:30:16Z [Verbose] Sending invocation id: 'f4097913-4155-4b2d-88f5-f903d5112bf1

2024-08-05T19:30:16Z [Verbose] Posting invocation id:f4097913-4155-4b2d-88f5-f903d5112bf1 on workerId:dbbb7155-e48e-46a9-9e7d-365b26797647

2024-08-05T19:30:16Z [Information] IDX10242: Security token: '[PII of type 'Microsoft.IdentityModel.JsonWebTokens.JsonWebToken' is hidden. For more details, see https://aka.ms/IdentityModel/PII.]' has a valid signature.

2024-08-05T19:30:16Z [Information] IDX10239: Lifetime of the token is valid.

2024-08-05T19:30:16Z [Information] IDX10234: Audience Validated.Audience: '0136c655-9040-40b0-951a-72268d5396a2'

2024-08-05T19:30:16Z [Information] IDX10245: Creating claims identity from the validated token: '[PII of type 'Microsoft.IdentityModel.JsonWebTokens.JsonWebToken' is hidden. For more details, see https://aka.ms/IdentityModel/PII.]'.

2024-08-05T19:30:17Z [Information] Executing StatusCodeResult, setting HTTP status code 200

2024-08-05T19:30:17Z [Information] Executed 'Functions.PostTrackedObjects' (Succeeded, Id=f4097913-4155-4b2d-88f5-f903d5112bf1, Duration=62ms)

When I do not throttle the payload the call never gets to the Azure function... I get no log output at all which is why I suspect something in front of the Azure function is getting involved.

Here is the client side trace output for a call that is not throttled (but still well under 100 Mb limit) and fails without outputting any trace on the server.

Unhandled exception. System.Net.Http.HttpRequestException: Error while copying content to a stream.

---> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host..

---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.

--- End of inner exception stack trace ---

at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)

at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)

at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1TIOAdapter

at System.Net.Security.SslStream.WriteAsyncChunkedTIOAdapter

at System.Net.Security.SslStream.WriteAsyncInternalTIOAdapter

at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask)

--- End of inner exception stack trace ---

at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask)

at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken)

at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)

at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)

at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)

at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)

I have tried the setting named FUNCTIONS_REQUEST_BODY_SIZE_LIMIT and it had no effect. You are recommending WEBSITE_MAX_REQUEST_LENGTH. Is this a server side option set on the Azure function. Did this variable change its name? Why the difference between the documented setting and this one? I'll give it a shot... Just wanting to know the difference in the settings.

Don Gillett 0 Reputation points

2024-08-05T19:52:05.0133333+00:00

Thanks for the reply.

Setting WEBSITE_MAX_REQUEST_LENGTH did not change the behavior. The same client error (below) happens with or without this setting whenever I post over 10K. The server never outputs any trace when the limit is exceeded. When the limit is not exceeded the call shows up on log streaming just fine.

Unhandled exception. System.Net.Http.HttpRequestException: Error while copying content to a stream.

---> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host..

---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.

--- End of inner exception stack trace ---

at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)

at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)

at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|157_1TIOAdapter

at System.Net.Security.SslStream.WriteAsyncChunkedTIOAdapter

at System.Net.Security.SslStream.WriteAsyncInternalTIOAdapter

at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask)

--- End of inner exception stack trace ---

at System.Net.Http.HttpContent.<CopyToAsync>g__WaitAsync|56_0(ValueTask copyTask)

at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, Boolean async, CancellationToken cancellationToken)

at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)

at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)

at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)

at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)

at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)

Don Gillett 0 Reputation points

2024-08-05T20:04:50.84+00:00

Bumping to Premium also did not fix the issue.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Konstantinos Passadis 19,066 Reputation points MVP

2024-08-05T20:18:38.6766667+00:00
Hello @Don Gillett

Thanks for the input!

The error message "Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host" implies that the problem may arise before the request reaches the Azure Function. It usually points to a networking or transport layer issue, not an issue with the Azure Function.

The issue is likely related to the network or SSL/TLS configuration rather than the Azure Function itself. Investigate potential problems with proxies, SSL/TLS configurations, client-side network settings, and Azure service configurations.

Suggestions:

Try without anything between , directly with a Postman call for example

Check whatever is between your Function and the originator , i belive something refuses to pass on the payload !!!

--

I hope this helps!

Kindly mark the answer as Accepted and Upvote in case it helped!

Regards
Please sign in to rate this answer.
Don Gillett 0 Reputation points

2024-08-05T20:51:03.22+00:00

We have tried directly from postman and also using swagger UX. The behavior is the same in those situations. The small payload works and the large payload doesn't. Also the SSL/TLS configuration is working for small payload requests. So if its something SSL/TLS related it would be due to large payloads but not smaller ones and I don't know what to look at in SSL/TLS that would be related to this. While I appreciate the sentiment that it is not an Azure Function issue... I am utilizing the most basic Azure Functions scenario deployed with all the defaults. What does Azure put between the client and the Azure server that I can check on. I did not configure anything myself with regards to this. Does Azure configure routing or usage of a Front Door which could be preventing the documented 100 Mb limit? As I stated earlier I suspect something is getting in the way... but it's gotta be what Azure rolls out as default config as we are doing nothing special here

.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Konstantinos Passadis 19,066 Reputation points MVP

2024-08-05T21:45:33.2166667+00:00

Hello @Don Gillett

While i am investigating kindly validate:

You dont have nothing in front of AzureFunctions and also you dont have Network Injection

Please try this :

https://learn.microsoft.com/en-us/answers/questions/965564/how-to-solve-an-existing-connection-was-forcibly-c

Taken from the post to help you :

When connecting over HTTPS, you have to consider if target is using SSL, TLS 1.1 or TLS 1.2, and properly enable on your end. Also have in mind .NET will check for valid certificate of target address.

I know you working on .Net Core6, but for .Net Framework whe use: To enable SSL, TLS 1.1 and TLS 1.2 on your end. Important this set is Global (for .Net Framework). This should fix exception "An existing connection was ..." System.Net.ServicePointManager.SecurityProtocol = System.Net.ServicePointManager.SecurityProtocol Or System.Net.SecurityProtocolType.Tls11 Or System.Net.SecurityProtocolType.Tls12

system.net.servicepointmanager.securityprotocol

Look for system.net.servicepointmanager.servercertificatevalidationcallback if you want bypass certificate validation.

--

I hope this helps!

Kindly mark the answer as Accepted and Upvote in case it helped!

Regards
Please sign in to rate this answer.
Konstantinos Passadis 19,066 Reputation points MVP

2024-08-05T21:48:29.83+00:00

Also :

https://learn.microsoft.com/en-us/troubleshoot/sql/database-engine/connect/tls-exist-connection-closed

AND did you tried here :

I hope this helps!

Kindly mark the answer as Accepted and Upvote in case it helped!

Regards

Don Gillett 0 Reputation points

2024-08-06T00:53:20.99+00:00

Ok, I created what I think is the minimal repro. Checking Client certificate mode = Optional is enough to cause the break with large bodies.

I created a new Azure Function App with a function called LargePostTest. All it do in this function is read in the body as a string and return a message with the length. I deployed this to a new app and chose the Premium Azure Function offering to rule out an earlier suggestion to try that. I have a test client that I can call it with any length of random body and it was succeeding just fine with large bodies. I then added in the ability for my client to attach a client cert. The configuration change I made was to click the Client certificate mode = Optional (from the image in the prior post) and left TLS to 1.2. I then changed the server test method to return the client cert thumbprint if a client cert was attached.

What is interesting is that both cases where I call with a client cert and without a client cert, the call would start to fail for large post payloads. So its not the fact that I'm calling with a client cert but just the config change alone makes it so the body is limited. Now an interesting thing I want to look at is that I don't get the cert by calling req.HttpContext.Connection.ClientCertificate on the server side but instead have to get the cert from a header called X-ARR-ClientCert. (I figured this out from online samples.) So my guess is that by introducing the option to allow including a client side cert, it introduces some infrastructure that proxies the call to my Azure function. It's probably terminating the SSL and passing the certificate on in the X-ARR-ClientCert header to me. It probably also has to copy the body on to the proxied request and that proxy probably isn't looking at the same limits for Azure Functions. From ancient times I remember ARR used to stand for Application Request Router so I'm guessing that is where this header comes from. How is ARR getting introduced and can I can configure it (or avoid it) to support a larger body?

Below is my test code:
Client

using System.Security.Cryptography.X509Certificates;

using System.Security.Cryptography;

using System.Text;

namespace LargePostTestClient

{

internal class Program { static async Task Main(string[] args) { X509Certificate2 certificate = GenerateSelfSignedCertificate("CN=LargePostTestClient"); //X509Certificate2 certificate = null; string randomString = GenerateRandomString(2000000); // works before I enables client certs but fails afterwards //string randomString = GenerateRandomString(1000); // works before and after enabling client certs string functionUrl = "https://largeposttest.azurewebsites.net/api/largeposttest"; string msg = await PostStringToFunction(functionUrl, randomString, certificate); Console.WriteLine(msg); } static X509Certificate2 GenerateSelfSignedCertificate(string subjectName) { using (RSA rsa = RSA.Create(2048)) // You can choose the RSA key size { var request = new CertificateRequest(subjectName, rsa, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1); // Set the certificate validity period var notBefore = DateTimeOffset.Now; var notAfter = notBefore.AddYears(1); // Create the self-signed certificate var certificate = request.CreateSelfSigned(notBefore, notAfter); // Export the certificate with the private key return new X509Certificate2(certificate.Export(X509ContentType.Pfx, "password"), "password", X509KeyStorageFlags.Exportable); } } static string GenerateRandomString(int length) { const string chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"; var random = new Random(); var stringChars = new char[length]; for (int i = 0; i < stringChars.Length; i++) { stringChars[i] = chars[random.Next(chars.Length)]; } return new string(stringChars); } static async Task<string> PostStringToFunction(string url, string content, X509Certificate2? clientCert = null) { HttpClientHandler? handler = null; if (clientCert != null) { handler = new HttpClientHandler(); handler.ClientCertificates.Add(clientCert); } using (HttpClient client = handler != null ? new HttpClient(handler): new HttpClient()) { var stringContent = new StringContent(content, Encoding.UTF8, "application/json"); HttpResponseMessage response = await client.PostAsync(url, stringContent); response.EnsureSuccessStatusCode(); string responseBody = await response.Content.ReadAsStringAsync(); return responseBody; } } }

}

Server

using Microsoft.AspNetCore.Http;

using Microsoft.AspNetCore.Mvc;

using Microsoft.AspNetCore.Server.HttpSys;

using Microsoft.Azure.Functions.Worker;

using Microsoft.Extensions.Logging;

using Microsoft.Extensions.Primitives;

using System.Security.Cryptography.X509Certificates;

namespace LargePostTest

{

public class Function1 { private readonly ILogger<Function1> _logger; public Function1(ILogger<Function1> logger) { _logger = logger; } [Function("LargePostTest")] public async Task<IActionResult> LargePostTest([HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = $"largeposttest")] HttpRequest req) { string requestBody = await new StreamReader(req.Body).ReadToEndAsync(); var certificate = GetCertificate(req); int length = requestBody.Length; _logger.LogInformation($"{length} bytes in body, certificate thumbprint is {certificate?.Thumbprint}"); return new OkObjectResult($"{length} bytes in body, certificate thumbprint is {certificate?.Thumbprint}"); } private static X509Certificate2 GetCertificate(HttpRequest req) { StringValues cert; if (req.Headers.TryGetValue("X-ARR-ClientCert", out cert)) { byte[] clientCertBytes = Convert.FromBase64String(cert[0]); X509Certificate2 clientCert = new X509Certificate2(clientCertBytes); return clientCert; } return null; } }

}

Konstantinos Passadis 19,066 Reputation points MVP

2024-08-06T09:35:34.4666667+00:00

Hello @Don Gillett

thank you for your insightful input

Apart from adding an API Management instance which will intoduce a new service to manage and added costs, here is what i should do :

Bypass Client Certificate Handling in Azure: If the client certificate is not mandatory, you could disable client certificate handling in Azure and instead handle the SSL termination and certificate processing yourself. This might involve using a different service or gateway that passes the request directly to the Function App without introducing the ARR proxy.

--

I hope this helps!

Kindly mark the answer as Accepted and Upvote in case it helped!

Regards

Don Gillett 0 Reputation points

2024-08-06T14:16:49.7266667+00:00

Client Certs are required for my scenario. So is the statement here that it is not possible to use client certs with Azure Functions and to get the documented limits for body size?

Konstantinos Passadis 19,066 Reputation points MVP

2024-08-06T14:54:03.2733333+00:00

Hello @Don Gillett !

If i was trying that , i would try one or two things

instead of plain Azure Functions deploy the Function to Azure Container Apps . It may save the day !

APi Management is the next option but only last case scenario

Do you want to examine the Container Apps version ?

https://learn.microsoft.com/en-us/azure/azure-functions/functions-deploy-container-apps?tabs=acr%2Cbash&pivots=programming-language-csharp

--

I hope this helps!

Kindly mark the answer as Accepted and Upvote in case it helped!

Regards

Don Gillett 0 Reputation points

2024-08-06T17:17:24.8966667+00:00

Thanks for the recommendations. Yes, I'll kick the tired on Azure Container apps and report back if it helps in this scenario.

Konstantinos Passadis 19,066 Reputation points MVP

2024-08-08T22:15:32.8066667+00:00

Hello @Don Gillett !

I was wondering whether my answer helped you or if there is anything else we can assist you with!

Do you have any feedback or is your issue resolved?

In case any answer helped kindly mark it as Accepted and upvote so others can benefit as well!

Otherwise post your feedback to provide additional help!

Regards
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Post to Azure Function with body over 10K doesn't reach Azure Function

3 answers

Your answer