API Management - Gateway getting 400 bad request, then 429 Throttling Errors fetching configuration

Toby Milne 1 Reputation point
2021-09-14T09:53:54.773+00:00

Hi,
We have some APIM Gateways hosted on prem, which are getting errors when they periodically fetch their config from the control plane.
It starts with a number of 400 error codes, but then as the gateway re-requests every second, soon Azure starts to reply with 429 (Throttled) errors.

It looks like the gateway is set up to try every second for config on an error, which seems to be too fast for Azure, could the retry implement a slower back off (it looks like you are using the Polly library so that supports progressively longer back offs)?

We are running the gateways via docker - image currently in use is mcr.microsoft.com/azure-api-management/gateway@sha256:7ca80c8314e192c440e6453fb7f18fb3e4dd893ab9e8741487ca5d5bb10d8b2a

400 errors look like this:
[Error]2021-09-2T08:59:28.750 [Error], exception: System.Net.Http.HttpRequestException: Response status code does not indicate success: 400 (Bad Request).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at Gateway.Host.AspNetCore.ExternalConfiguration.ServiceConfigurationSource.HandleConfigurationError(HttpResponseMessage response) in D:\a\1\s\Proxy\Gateway.Host.AspNetCore\ExternalConfiguration\ServiceConfigurationSource.cs:line 212
at Gateway.Host.AspNetCore.ExternalConfiguration.ServiceConfigurationSource.<>c__DisplayClass19_0.<<UpdateConfiguration>b__1>d.MoveNext() in D:\a\1\s\Proxy\Gateway.Host.AspNetCore\ExternalConfiguration\ServiceConfigurationSource.cs:line 179
--- End of stack trace from previous location where exception was thrown ---
at Polly.Retry.AsyncRetryEngine.ImplementationAsyncTResult, source: ServiceConfigurationSource

429 errors look like this:
[Error], exception: System.Net.Http.HttpRequestException: Response status code does not indicate success: 429 ().
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at Gateway.Host.AspNetCore.ExternalConfiguration.ServiceConfigurationSource.HandleConfigurationError(HttpResponseMessage response) in D:\a\1\s\Proxy\Gateway.Host.AspNetCore\ExternalConfiguration\ServiceConfigurationSource.cs:line 212
at Gateway.Host.AspNetCore.ExternalConfiguration.ServiceConfigurationSource.<>c__DisplayClass19_0.<<UpdateConfiguration>b__1>d.MoveNext() in D:\a\1\s\Proxy\Gateway.Host.AspNetCore\ExternalConfiguration\ServiceConfigurationSource.cs:line 179
--- End of stack trace from previous location where exception was thrown ---
at Polly.Retry.AsyncRetryEngine.ImplementationAsyncTResult, source: ServiceConfigurationSource [0m

Azure API Management
Azure API Management
An Azure service that provides a hybrid, multi-cloud management platform for APIs.
2,447 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.