Azure App service goes down with fail: Middleware[0] System.Net.Http.HttpRequestException exception

Muhammad Maroof Subhani 40 Reputation points
2023-01-27T06:22:45.81+00:00

Intermittently app service goes unresponsive.
In log stream we can only see following errors

2023-01-20T10:57:22.078727437Z: [INFO] fail: Middleware[0]

2023-01-20T10:57:22.078767038Z: [INFO] Failed to forward request to http://169.254.130.2:3000. Encountered a System.Net.Http.HttpRequestException exception after 0.595ms with message: Connection refused. Check application logs to verify the application is properly handling HTTP traffic.

2023-01-20T10:57:28.980765262Z: [INFO] fail: Middleware[0]

2023-01-20T10:57:28.980806963Z: [INFO] Failed to forward request to http://169.254.130.2:3000. Encountered a System.Net.Http.HttpRequestException exception after 0.557ms with message: Connection refused. Check application logs to verify the application is properly handling HTTP traffic.

2023-01-20T10:57:45.666191317Z: [INFO] fail: Middleware[0]

2023-01-20T10:57:45.666210217Z: [INFO] Failed to forward request to http://169.254.130.2:3000. Encountered a System.Net.Http.HttpRequestException exception after 0.760ms with message: Connection refused. Check application logs to verify the application is properly handling HTTP traffic.

2023-01-20T10:57:49.526518951Z: [INFO] fail: Middleware[0]

2023-01-20T10:57:49.526553351Z: [INFO] Failed to forward request to http://169.254.130.2:3000. Encountered a System.Net.Http.HttpRequestException exception after 0.576ms with message: Connection refused. Check application logs to verify the application is properly handling HTTP traffic.

2023-01-20T10:57:53.612619396Z: [INFO] fail: Middleware[0]

2023-01-20T10:57:53.612860398Z: [INFO] Failed to forward request to http://169.254.130.2:3000. Encountered a System.Net.Http.HttpRequestException exception after 0.537ms with message: Connection refused. Check application logs to verify the application is properly handling HTTP traffic.

And application logs stop being generated at the time of issue.

*Issue gets resolve every time we restart the service

Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
6,902 questions

4 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. ajkuma 22,401 Reputation points Microsoft Employee
    2023-02-02T20:36:39.5266667+00:00

    @Muhammad Maroof Subhani ,

    Apologies for any inconvenience with this issue and my delayed response from over the weekend.

     

    Do you recall making any changes prior to this issue?

    Have you observed this behavior since the Webapp first deployment or started seeing this more recently? What App Service Plan are you using?

    I understand, a restart fixes the problem, and the issue is sporadic.  Please try the following steps to fetch more info:

    1. You may leverage App Service diagnostics from Azure Portal> Navigate to your App Service app in the Azure Portal. (screenshot below)
      -- In the left navigation, click on Diagnose and solve problems - Review - “Configuration and Management and “Availability and Performance” options.
    2. Navigate directly to https://<app-name>.scm.azurewebsites.net/api/logs/docker  for the Docker logs.
      Also, kindly check diagnostic logs troubleshoot-diagnostic-logs for more info
    3. Review Service Health notification info. Kindly let us know how it goes, I'll follow-up further with you.
    0 comments
  2. Bill Christo 0 Reputation points
    2023-02-17T03:52:58.29+00:00

    I am having the same issue with my Linux deployed web app on PremiumV2 (P1v2). The app works fine deployed to any other environment but as soon as it's deployed within Azure, I have this issue exactly as explained here. I have an open ticket with MS with damazo.perez

  3. Niral Patel 5 Reputation points
    2024-01-27T00:44:15.48+00:00

    This is equally affecting me. Using a simple FastAPI backend that calls openAI endpoints which works locally, I can use a docker container locally, but with it hosted on Azure I will frequently see the API calls fail.

    Middleware[0]2024-01-27T00:34:19.417375023Z Failed to forward request to http://169.254.129.2. Encountered a System.Net.Http.HttpRequestException exception after 23987.782ms with message: An error occurred while sending the request.. Check application logs to verify the application is properly handling HTTP traffic.

  4. Niral Patel 5 Reputation points
    2024-02-02T20:26:13.5433333+00:00

    It appears I solved this issue on my end. Within my gunicorn.conf.py file, I needed to specify a timeout = value that was >30 (the default value). My API was expected to take >1 minute on certain requests, and I had to denote this. Additionally I needed to set the line in my Dockerfile to the following to enable this parameter to be enforced.

    CMD ["gunicorn", "-c", "gunicorn.conf.py", "main:app"]
    0 comments