Intermittent ERR_HTTP2_PROTOCOL_ERROR / HTTP2_STREAM_ERROR / Server reset stream on App Service with Azure Front Door

Question

We have two applications, one is a static single-page web application (SPA), and one is an ASP.NET Web API application running in .NET Framework 4.7.2. We have two App Service plans, each one in a different region, and each one hosting the two applications as App Services. We have an Azure Front Door instance for each application, each of which load balances between the two App Service plans. Users fetch the front-end SPA on one domain and the SPA talks to the API on another domain.

In the last couple of weeks we have started getting intermittent errors in the form of ERR_HTTP2_PROTOCOL_ERROR. These are not specific to any particular API request, and even happen just requesting the static website. Therefore it is not limited to either App Service. However seeing the errors on the API requests is more common as the SPA generally only gets loaded once before making many API requests over a period of time.

The website will be fine for periods, but then intermittently one or more requests will fail with ERR_HTTP2_PROTOCOL_ERROR, often in a cluster of requests (all failing), and then a few moments later the same requests will succeed.

If I sit refreshing the SPA every couple of seconds, eventually the SPA will fail to load and Chrome will display "The site can't be reached" and "ERR_HTTP2_PROTOCOL_ERROR".

I've used chrome://net-export to record when this happens, both with the API and the SPA, and in both cases I get a similar output for a failed URL_REQUEST with "HTTP2_STREAM_ERROR" and "Server reset stream". I've included an example log when failing to load the SPA at the end of this post.

Because we are getting this issue with both the SPA and the API, it seems to indicate it is an issue with something sitting in front of those (either the App Service configuration, the App Service Plan, or Front Door).

One thing that made me think it might be a Front Door issue is that our App Services were configured to use HTTP 1.1 (I've since switched to HTTP 2, which didn't help), but the errors are related to HTTP 2. The Azure Front Door docs say:

HTTP/2 protocol support is available to clients connecting to Azure Front Door only. The communication to backends in the backend pool is over HTTP/1.1. HTTP/2 support is enabled by default.

Does anyone have any suggestions on where to go from here?

Example error output

t=652141 [st=   0] +REQUEST_ALIVE  [dt=9925]  
                    --> priority = "HIGHEST"  
                    --> traffic_annotation = 63171670  
                    --> url = "https://portal.mydomain.com/sign-in"  
t=652141 [st=   0]    NETWORK_DELEGATE_BEFORE_URL_REQUEST  [dt=0]  
t=652141 [st=   0]   +URL_REQUEST_START_JOB  [dt=9924]  
                      --> initiator = "https://portal.mydomain.com"  
                      --> load_flags = 65794 (BYPASS_CACHE | CAN_USE_RESTRICTED_PREFETCH | MAIN_FRAME_DEPRECATED)  
                      --> method = "GET"  
                      --> network_isolation_key = "https://mydomain.com https://mydomain.com"  
                      --> privacy_mode = "disabled"  
                      --> site_for_cookies = "SiteForCookies: {scheme=https; registrable_domain=mydomain.com; schemefully_same=true}"  
                      --> url = "https://portal.mydomain.com/sign-in"  
t=652141 [st=   0]      COOKIE_INCLUSION_STATUS  
                        --> operation = "send"  
                        --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"  
t=652141 [st=   0]      COOKIE_INCLUSION_STATUS  
                        --> operation = "send"  
                        --> status = "EXCLUDE_DOMAIN_MISMATCH, DO_NOT_WARN"  
t=652141 [st=   0]      NETWORK_DELEGATE_BEFORE_START_TRANSACTION  [dt=0]  
t=652142 [st=   1]      HTTP_CACHE_GET_BACKEND  [dt=0]  
t=652142 [st=   1]      HTTP_CACHE_DOOM_ENTRY  [dt=0]  
                        --> net_error = -2 (ERR_FAILED)  
t=652142 [st=   1]      HTTP_CACHE_CREATE_ENTRY  [dt=0]  
t=652142 [st=   1]      HTTP_CACHE_ADD_TO_ENTRY  [dt=0]  
t=652142 [st=   1]     +HTTP_STREAM_REQUEST  [dt=0]  
t=652142 [st=   1]        HTTP_STREAM_JOB_CONTROLLER_BOUND  
                          --> source_dependency = 37851 (HTTP_STREAM_JOB_CONTROLLER)  
t=652142 [st=   1]        HTTP_STREAM_REQUEST_BOUND_TO_JOB  
                          --> source_dependency = 37852 (HTTP_STREAM_JOB)  
t=652142 [st=   1]     -HTTP_STREAM_REQUEST  
t=652142 [st=   1]     +HTTP_TRANSACTION_SEND_REQUEST  [dt=0]  
t=652142 [st=   1]        HTTP_TRANSACTION_HTTP2_SEND_REQUEST_HEADERS  
                          --> :method: GET  
                              :authority: portal.mydomain.com  
                              :scheme: https  
                              :path: /sign-in  
                              pragma: no-cache  
                              cache-control: no-cache  
                              upgrade-insecure-requests: 1  
                              user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36  
                              accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9  
                              sec-fetch-site: same-origin  
                              sec-fetch-mode: navigate  
                              sec-fetch-user: ?1  
                              sec-fetch-dest: document  
                              accept-encoding: gzip, deflate, br  
                              accept-language: en-US,en;q=0.9  
t=652142 [st=   1]     -HTTP_TRANSACTION_SEND_REQUEST  
t=652142 [st=   1]     +HTTP_TRANSACTION_READ_HEADERS  [dt=9923]  
t=662065 [st=9924]        HTTP2_STREAM_ERROR  
                          --> description = "Server reset stream."  
                          --> net_error = "ERR_HTTP2_PROTOCOL_ERROR"  
                          --> stream_id = 5847  
t=662065 [st=9924]     -HTTP_TRANSACTION_READ_HEADERS  
                        --> net_error = -337 (ERR_HTTP2_PROTOCOL_ERROR)  
t=662065 [st=9924]   -URL_REQUEST_START_JOB  
                      --> net_error = -337 (ERR_HTTP2_PROTOCOL_ERROR)  
t=662065 [st=9924]    URL_REQUEST_DELEGATE_RESPONSE_STARTED  [dt=1]  
t=662066 [st=9925] -REQUEST_ALIVE  
                    --> net_error = -337 (ERR_HTTP2_PROTOCOL_ERROR)  

UPDATE:

Hi Mithun, thanks for the comment. That does sound like the same issue. I haven't resolved it yet on my side.

I've tried passing the x-azure-debuginfo: 1 header through with requests, from the SPA to the API, as mentioned in the docs here, but when the error occurs I don't receive any additional debug headers back.

I've tried enabling Front Door diagnostic logging, but the only ErrorInfo entries other than NoError are DNSNameNotResolved. This is odd in itself as it is resolving an azurewebsites.net address and the requests around it succeed, however looking at the timestamps vs when I get HTTP2 errors these aren't obviously related. I've included sanitized entries, with and without an error, which occurred within seconds of each other, below.

With Error:

{  
    "time": "2021-01-22T10:05:16.5111237Z",  
    "resourceId": "/SUBSCRIPTIONS/9C81A098-BAA3-4465-9502-E464004F2F6B/RESOURCEGROUPS/MY-RG/PROVIDERS/MICROSOFT.NETWORK/FRONTDOORS/MY-API",  
    "category": "FrontdoorAccessLog",  
    "operationName": "Microsoft.Network/FrontDoor/AccessLog/Write",  
    "properties": {  
        "trackingReference": "0XKMKYAAAAADXRIIsEmR+RoeguVrUl+LlTVVDMzBFREdFMDYxOQAxZTI5YTljNy01N2M1LTQyZGEtYjljMi1iNjYyMjY2MWQxYTE=",  
        "httpMethod": "OPTIONS",  
        "httpVersion": "2.0",  
        "requestUri": "https://api.mydomain.com:443/xyz/abc/def?",  
        "requestBytes": "646",  
        "responseBytes": "1470",  
        "userAgent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36",  
        "clientIp": "123.456.153.25",  
        "socketIp": "123.456.153.25",  
        "clientPort": "27796",  
        "timeToFirstByte": "0.082",  
        "timeTaken": "0.082",  
        "requestProtocol": "HTTPS",  
        "securityProtocol": "TLS 1.2",  
        "routingRuleName": "my-api",  
        "rulesEngineMatchNames": [],  
        "backendHostname": "my-api-msnsxfxpaaaaa.azurewebsites.net:443",  
        "isReceivedFromClient": true,  
        "httpStatusCode": "503",  
        "httpStatusDetails": "503",  
        "pop": "MUC",  
        "cacheStatus": "CONFIG_NOCACHE",  
        "ErrorInfo": "DNSNameNotResolved"  
    }  
}   

Without Error:

{  
    "time": "2021-01-22T10:05:22.3334673Z",  
    "resourceId": "/SUBSCRIPTIONS/9C81A098-BAA3-4465-9502-E464004F2F6B/RESOURCEGROUPS/MY-RG/PROVIDERS/MICROSOFT.NETWORK/FRONTDOORS/MY-API",  
    "category": "FrontdoorAccessLog",  
    "operationName": "Microsoft.Network/FrontDoor/AccessLog/Write",  
    "properties": {  
        "trackingReference": "0YaMKYAAAAADdAswx9kxdRbBqhPzDwxIlTVVDMzBFREdFMDYxOQAxZTI5YTljNy01N2M1LTQyZGEtYjljMi1iNjYyMjY2MWQxYTE=",  
        "httpMethod": "OPTIONS",  
        "httpVersion": "2.0",  
        "requestUri": "https://api.mydomain.com:443/xyz/abc/def?",  
        "requestBytes": "646",  
        "responseBytes": "978",  
        "userAgent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36",  
        "clientIp": "123.456.153.25",  
        "socketIp": "123.456.153.25",  
        "clientPort": "27796",  
        "timeToFirstByte": "0.791",  
        "timeTaken": "0.791",  
        "requestProtocol": "HTTPS",  
        "securityProtocol": "TLS 1.2",  
        "routingRuleName": "my-api",  
        "rulesEngineMatchNames": [],  
        "backendHostname": "my-api-msnsxfxpaaaaa.azurewebsites.net:443",  
        "isReceivedFromClient": true,  
        "httpStatusCode": "200",  
        "httpStatusDetails": "200",  
        "pop": "MUC",  
        "cacheStatus": "CONFIG_NOCACHE",  
        "ErrorInfo": "NoError"  
    }  
}  

I'll continue investigating.