Azure App service goes down with fail: Middleware[0] System.Net.Http.HttpRequestException exception

Question

Intermittently app service goes unresponsive.
In log stream we can only see following errors

2023-01-20T10:57:22.078727437Z: [INFO] fail: Middleware[0]

2023-01-20T10:57:22.078767038Z: [INFO] Failed to forward request to http://169.254.130.2:3000. Encountered a System.Net.Http.HttpRequestException exception after 0.595ms with message: Connection refused. Check application logs to verify the application is properly handling HTTP traffic.

2023-01-20T10:57:28.980765262Z: [INFO] fail: Middleware[0]

2023-01-20T10:57:28.980806963Z: [INFO] Failed to forward request to http://169.254.130.2:3000. Encountered a System.Net.Http.HttpRequestException exception after 0.557ms with message: Connection refused. Check application logs to verify the application is properly handling HTTP traffic.

2023-01-20T10:57:45.666191317Z: [INFO] fail: Middleware[0]

2023-01-20T10:57:45.666210217Z: [INFO] Failed to forward request to http://169.254.130.2:3000. Encountered a System.Net.Http.HttpRequestException exception after 0.760ms with message: Connection refused. Check application logs to verify the application is properly handling HTTP traffic.

2023-01-20T10:57:49.526518951Z: [INFO] fail: Middleware[0]

2023-01-20T10:57:49.526553351Z: [INFO] Failed to forward request to http://169.254.130.2:3000. Encountered a System.Net.Http.HttpRequestException exception after 0.576ms with message: Connection refused. Check application logs to verify the application is properly handling HTTP traffic.

2023-01-20T10:57:53.612619396Z: [INFO] fail: Middleware[0]

2023-01-20T10:57:53.612860398Z: [INFO] Failed to forward request to http://169.254.130.2:3000. Encountered a System.Net.Http.HttpRequestException exception after 0.537ms with message: Connection refused. Check application logs to verify the application is properly handling HTTP traffic.

And application logs stop being generated at the time of issue.

*Issue gets resolve every time we restart the service

Answer 1

@Muhammad Maroof Subhani ,

Apologies for any inconvenience with this issue and my delayed response from over the weekend.

 

Do you recall making any changes prior to this issue?

Have you observed this behavior since the Webapp first deployment or started seeing this more recently? What App Service Plan are you using?

I understand, a restart fixes the problem, and the issue is sporadic.  Please try the following steps to fetch more info:

1. You may leverage App Service diagnostics from Azure Portal> Navigate to your App Service app in the Azure Portal. (screenshot below)
   -- In the left navigation, click on Diagnose and solve problems - Review - “Configuration and Management and “Availability and Performance” options.
2. Navigate directly to https://<app-name>.scm.azurewebsites.net/api/logs/docker  for the Docker logs.
   Also, kindly check diagnostic logs troubleshoot-diagnostic-logs for more info
3. Review Service Health notification info. Kindly let us know how it goes, I'll follow-up further with you.

Answer 2

I am having the same issue with my Linux deployed web app on PremiumV2 (P1v2). The app works fine deployed to any other environment but as soon as it's deployed within Azure, I have this issue exactly as explained here. I have an open ticket with MS with damazo.perez

Answer 3

This is equally affecting me. Using a simple FastAPI backend that calls openAI endpoints which works locally, I can use a docker container locally, but with it hosted on Azure I will frequently see the API calls fail.

Middleware[0]2024-01-27T00:34:19.417375023Z Failed to forward request to http://169.254.129.2. Encountered a System.Net.Http.HttpRequestException exception after 23987.782ms with message: An error occurred while sending the request.. Check application logs to verify the application is properly handling HTTP traffic.

Answer 4

It appears I solved this issue on my end. Within my gunicorn.conf.py file, I needed to specify a timeout = value that was >30 (the default value). My API was expected to take >1 minute on certain requests, and I had to denote this. Additionally I needed to set the line in my Dockerfile to the following to enable this parameter to be enforced.

CMD ["gunicorn", "-c", "gunicorn.conf.py", "main:app"]