*System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing

Question

Hi,

Why my Polly Timeout handler not working?

I still getting: "System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing."

program.cs

builder.Services.AddHttpClient<Example>("Example", c1 =>
{
    c1.BaseAddress = new Uri("https://api.example.io");
    c1.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
    c1.DefaultRequestHeaders.TryAddWithoutValidation("Content-Type", "application/json; charset=utf-8");
    c1.DefaultRequestHeaders.CacheControl = new CacheControlHeaderValue { NoCache = true };
})
.AddPolicyHandler(PrimeIoT.SIM.PollyPolicies.GetTimeoutPolicy("Example"))
.AddPolicyHandler(PrimeIoT.SIM.PollyPolicies.GetRetryPolicy("Example"))
.AddPolicyHandler(PrimeIoT.SIM.PollyPolicies.GetCircuitBreakerPolicy("Example"))
.ConfigureHttpClient((c2) =>
    new HttpClientHandler
    {
        ServerCertificateCustomValidationCallback = (sender, cert, chain,
        sslPolicyErrors) =>
        {
            return sslPolicyErrors == SslPolicyErrors.None;
        }
    }
);

polly.cs

const int MAX_RETRIES = 3;

   internal static IAsyncPolicy<HttpResponseMessage> TimeoutPolicy(ResilienceSettings settings)
    => Policy
        .TimeoutAsync<HttpResponseMessage>( //Catches TaskCanceledException and throws instead TimeoutRejectedException
            timeout: TimeSpan.FromMilliseconds(settings.HttpRequestTimeoutInMilliseconds),
            timeoutStrategy: TimeoutStrategy.Pessimistic);

internal static AsyncRetryPolicy<HttpResponseMessage> GetTimeoutPolicy(string serviceName)
{
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .Or<TaskCanceledException>()
        .WaitAndRetryAsync(MAX_RETRIES, retryAttempt => TimeSpan.FromSeconds(30),
        onRetry: (exception, sleepDuration, attemptNumber, context) =>
        {
            if (attemptNumber == MAX_RETRIES)
            {
                string msg = serviceName + $":PollyPolicies:GetTimoutPolicy in {sleepDuration}. {attemptNumber}/{MAX_RETRIES}";

                Models.Audit.AddAsync((long)0, msg);
            }
        });
}

Answer 1

So as an update* The "resolution" we found to this (which to me is NOT a real resolution!) is to add the following to the Kestrel server build options and then allow it to run synchronously with snowflakedata's SnowflakeDbConnection. Running it Async seems to lead back to the timeout, but running the pod synchronously does work, but it is taking roughly 5 and 1/2 hours to return around rows of data. (That is a weeks worth of data, and snowflake is updated weekly so we were following suite). We are looking at throttling the queries even more but I wouldn't expect the network to take that long to return 300,000 rows of data.

Kestral Options (incase anyone else find this and needs it!)
.UseKestrel(opt =>

           {

               opt.Limits.MaxConcurrentConnections = 100;

               opt.Limits.MaxConcurrentUpgradedConnections = 100;

               opt.Limits.KeepAliveTimeout = TimeSpan.FromMinutes(120);

               opt.Limits.MaxRequestLineSize = 512000;

               opt.Limits.MaxRequestBufferSize = 90000000;

               opt.Limits.MaxResponseBufferSize = 90000000;

               opt.Limits.MaxRequestHeadersTotalSize = 512000;

               opt.Limits.MaxRequestBodySize = 90000000;

               opt.AllowSynchronousIO = true;

           })