ERR_HTTP2_PROTOCOL_ERROR on Application Gateway WAF_v2

Question

ERR_HTTP2_PROTOCOL_ERROR on Application Gateway WAF_v2

Jose Ramon Roca Garcia 10

We have an application gateway with an NGINX docker image behind. Connections over HTTP2 get random ERR_HTTP2_PROTOCOL_ERROR error.

How can we debug this?

ChaitanyaNaykodi-MSFT 27,476 Reputation points Microsoft Employee Moderator

2023-10-05T19:40:08.6466667+00:00
@Jose Ramon Roca Garcia

Thank you for reaching out I am currently following up on a similar issue internally and the product team is actively working on it. I will share an update as soon as the issue is resolved.

Meanwhile I think the work-around in this case will be to Disable HTTP2 for your Application Gateway. You can run the following PowerShell Command to disable HTTP2 or you can also disable HTTP2 support using the Azure portal by selecting Disabled under HTTP2 in Application gateway > Configuration.

$gw = Get-AzApplicationGateway -Name test -ResourceGroupName hm $gw.EnableHttp2 = $false Set-AzApplicationGateway -ApplicationGateway $gw

Thank you!
Jose Ramon Roca Garcia 10 Reputation points

2023-10-06T06:12:59.9633333+00:00

Where can i see when it is fixed?
Bastien NOEL 30 Reputation points

2023-10-06T08:56:43.6433333+00:00

@ChaitanyaNaykodi-MSFT Can you let us know when it's resolved?

We've also been affected and have disabled HTTP2 on our side for the time being following your recommendations.
Ventura Morcillo Valle 0 Reputation points

2023-10-06T09:43:34.2466667+00:00

Yes, we need a resolution of the issue since it is being affected our website internet scoring since http is less scored than http2.

Is there a Microsoft dashboard o health panel where we could be alerted of such issues instead of having to discover by clients?

Thanks,

Ventura.
Jose Ramon Roca Garcia 10 Reputation points

2023-10-06T10:24:01.5433333+00:00

Do you have any ETA for fixing? For us is very important to have https2 as Google wants it.

We have several Gateways, are all potencially affected?

We have deactivated WAF and switched to Standard just-in case, is ok?
GH 0 Reputation points

2023-10-06T15:14:39.3966667+00:00

Users of our web application are experiencing this and are unable to use the product. Urgent remediation is requested
ChaitanyaNaykodi-MSFT 27,476 Reputation points Microsoft Employee Moderator

2023-10-06T17:27:58.33+00:00

Hello ALL,

I am following up on this issue internal and will make a post here as soon as I have an update.
ChaitanyaNaykodi-MSFT 27,476 Reputation points Microsoft Employee Moderator

2023-10-06T19:54:20.37+00:00

@Jose Ramon Roca Garcia @Ventura Morcillo Valle @Bastien Noel @GH

In order to mitigate this issue can you please share the following details so I can ask the backend team to apply the mitigation until the fix is deployed. Thank you!

If you have a support ticket open for this issue you can ask the support engineer to share this information with the backend team.

If you do not have a support ticket open, please send an email to azcommunity@microsoft.com with the below details.

Subject : Attn Chaitanya

Thread URL: Link to this thread.

Subscription ID:

Resource ID of the affected Application Gateway:

Please let me know once you have done the same.

You can fetch the Resource ID from the Application Gateway Overview page on the portal in JSON View
GH 0 Reputation points

2023-10-06T20:14:03.0766667+00:00

@ChaitanyaNaykodi-MSFT email has been sent with the requested information
Ventura Morcillo Valle 0 Reputation points

2023-10-06T20:30:01.7566667+00:00

@ChaitanyaNaykodi-MSFT email has been sent as well. Please confirm details are the ones you need.
Ventura Morcillo Valle 0 Reputation points

2023-10-06T20:31:59.9066667+00:00

@ChaitanyaNaykodi-MSFT shall we reactivate https2 once you confirm those mitigations are in place? For us it is important due to Google scoring matters.
ChaitanyaNaykodi-MSFT 27,476 Reputation points Microsoft Employee Moderator

2023-10-06T21:21:55.5133333+00:00

@Ventura Morcillo Valle @GH

Thank you for getting back. I have replied to your email just now.

shall we reactivate https2 once you confirm those mitigations are in place? For us it is important due to Google scoring matters.

Yes, you can re-enable the HTTPS2 setting.
badtiming 0 Reputation points

2023-10-06T21:40:34.1133333+00:00

We had the same issue this morning and would like to know when the fix is in place to revert our changes. Thanks!
Bastien NOEL 30 Reputation points

2023-10-06T22:51:53.4733333+00:00

@ChaitanyaNaykodi-MSFT email has been sent as well.
ChaitanyaNaykodi-MSFT 27,476 Reputation points Microsoft Employee Moderator

2023-10-07T00:05:40.92+00:00

@Bastien Noel @badtiming

I have replied to your email just now.

@badtiming

Yes, I will make an update here as soon as the fix is rolled out.

@Jose Ramon Roca Garcia

Sorry I missed replying to your questions above.

We have several Gateways, are all potencially affected? We have deactivated WAF and switched to Standard just-in case, is ok?

The current work-around in this case is to disable the HTTP2 setting. In order to apply the mitigation please send me the details as requested above. Thank you!
Ventura Morcillo Valle 0 Reputation points

2023-10-07T06:04:58.8933333+00:00

@ChaitanyaNaykodi-MSFT a couple of app gws also sent for mitigation.

Thanks.
GH 0 Reputation points

2023-10-07T21:14:41.1733333+00:00

@ChaitanyaNaykodi-MSFT This issue is now happening again on our Gateway that you had previously performed the mitigation on.
FARRELL Lisalee T (Tunde) 0 Reputation points

2023-10-09T16:48:49.6066667+00:00

How can MSFT say 'disable HTTP2' as a work around? Thats not a work around. MSFT has been SILENT on this issue while others have had to debug, troubleshoot etc. for days. We are unable to run performance tests - so the workaround isn't helpful.
Catalao, Pablo 0 Reputation points

2023-10-10T12:59:19.7166667+00:00

I have the same issue since Friday. I have a ticket with Microsoft but have not heard back yet.
badtiming 0 Reputation points

2023-10-10T14:05:40.5466667+00:00
i submitted a ticket and the answer i got from the support engineer was:

known issue

we are working on it

workaround is disabling http2
FARRELL Lisalee T (Tunde) 0 Reputation points

2023-10-10T15:07:18.44+00:00

We reached out MSFT under our premier support contract - this is a known issue that for some reason MSFT has decided to let the technical community figure out themselves. If you send Support the AppGw URL they can do something to address the issue for you until they have a global response. I have requested responses from MSFT for several days regarding timeframe and so far i just get a vague timeline.

1 answer

Your answer

ChaitanyaNaykodi-MSFT 27,476 Reputation points Microsoft Employee Moderator

2023-10-05T19:40:08.6466667+00:00

@Jose Ramon Roca Garcia

Thank you for reaching out I am currently following up on a similar issue internally and the product team is actively working on it. I will share an update as soon as the issue is resolved.

Meanwhile I think the work-around in this case will be to Disable HTTP2 for your Application Gateway. You can run the following PowerShell Command to disable HTTP2 or you can also disable HTTP2 support using the Azure portal by selecting Disabled under HTTP2 in Application gateway > Configuration.

$gw = Get-AzApplicationGateway -Name test -ResourceGroupName hm $gw.EnableHttp2 = $false Set-AzApplicationGateway -ApplicationGateway $gw

Thank you!
Jose Ramon Roca Garcia 10 Reputation points

2023-10-06T06:12:59.9633333+00:00

Where can i see when it is fixed?
Bastien NOEL 30 Reputation points

2023-10-06T08:56:43.6433333+00:00

@ChaitanyaNaykodi-MSFT Can you let us know when it's resolved?

We've also been affected and have disabled HTTP2 on our side for the time being following your recommendations.
Ventura Morcillo Valle 0 Reputation points

2023-10-06T09:43:34.2466667+00:00

Yes, we need a resolution of the issue since it is being affected our website internet scoring since http is less scored than http2.

Is there a Microsoft dashboard o health panel where we could be alerted of such issues instead of having to discover by clients?

Thanks,

Ventura.
Jose Ramon Roca Garcia 10 Reputation points

2023-10-06T10:24:01.5433333+00:00

Do you have any ETA for fixing? For us is very important to have https2 as Google wants it.

We have several Gateways, are all potencially affected?

We have deactivated WAF and switched to Standard just-in case, is ok?
GH 0 Reputation points

2023-10-06T15:14:39.3966667+00:00

Users of our web application are experiencing this and are unable to use the product. Urgent remediation is requested
ChaitanyaNaykodi-MSFT 27,476 Reputation points Microsoft Employee Moderator

2023-10-06T17:27:58.33+00:00

Hello ALL,

I am following up on this issue internal and will make a post here as soon as I have an update.
ChaitanyaNaykodi-MSFT 27,476 Reputation points Microsoft Employee Moderator

2023-10-06T19:54:20.37+00:00

@Jose Ramon Roca Garcia @Ventura Morcillo Valle @Bastien Noel @GH

In order to mitigate this issue can you please share the following details so I can ask the backend team to apply the mitigation until the fix is deployed. Thank you!

If you have a support ticket open for this issue you can ask the support engineer to share this information with the backend team.

If you do not have a support ticket open, please send an email to azcommunity@microsoft.com with the below details.

Subject : Attn Chaitanya

Thread URL: Link to this thread.

Subscription ID:

Resource ID of the affected Application Gateway:

Please let me know once you have done the same.

You can fetch the Resource ID from the Application Gateway Overview page on the portal in JSON View
GH 0 Reputation points

2023-10-06T20:14:03.0766667+00:00

@ChaitanyaNaykodi-MSFT email has been sent with the requested information
Ventura Morcillo Valle 0 Reputation points

2023-10-06T20:30:01.7566667+00:00

@ChaitanyaNaykodi-MSFT email has been sent as well. Please confirm details are the ones you need.
Ventura Morcillo Valle 0 Reputation points

2023-10-06T20:31:59.9066667+00:00

@ChaitanyaNaykodi-MSFT shall we reactivate https2 once you confirm those mitigations are in place? For us it is important due to Google scoring matters.
ChaitanyaNaykodi-MSFT 27,476 Reputation points Microsoft Employee Moderator

2023-10-06T21:21:55.5133333+00:00

@Ventura Morcillo Valle @GH

Thank you for getting back. I have replied to your email just now.

shall we reactivate https2 once you confirm those mitigations are in place? For us it is important due to Google scoring matters.

Yes, you can re-enable the HTTPS2 setting.
badtiming 0 Reputation points

2023-10-06T21:40:34.1133333+00:00

We had the same issue this morning and would like to know when the fix is in place to revert our changes. Thanks!
Bastien NOEL 30 Reputation points

2023-10-06T22:51:53.4733333+00:00

@ChaitanyaNaykodi-MSFT email has been sent as well.
ChaitanyaNaykodi-MSFT 27,476 Reputation points Microsoft Employee Moderator

2023-10-07T00:05:40.92+00:00

@Bastien Noel @badtiming

I have replied to your email just now.

@badtiming

Yes, I will make an update here as soon as the fix is rolled out.

@Jose Ramon Roca Garcia

Sorry I missed replying to your questions above.

We have several Gateways, are all potencially affected? We have deactivated WAF and switched to Standard just-in case, is ok?

The current work-around in this case is to disable the HTTP2 setting. In order to apply the mitigation please send me the details as requested above. Thank you!
Ventura Morcillo Valle 0 Reputation points

2023-10-07T06:04:58.8933333+00:00

@ChaitanyaNaykodi-MSFT a couple of app gws also sent for mitigation.

Thanks.
GH 0 Reputation points

2023-10-07T21:14:41.1733333+00:00

@ChaitanyaNaykodi-MSFT This issue is now happening again on our Gateway that you had previously performed the mitigation on.
FARRELL Lisalee T (Tunde) 0 Reputation points

2023-10-09T16:48:49.6066667+00:00

How can MSFT say 'disable HTTP2' as a work around? Thats not a work around. MSFT has been SILENT on this issue while others have had to debug, troubleshoot etc. for days. We are unable to run performance tests - so the workaround isn't helpful.
Catalao, Pablo 0 Reputation points

2023-10-10T12:59:19.7166667+00:00

I have the same issue since Friday. I have a ticket with Microsoft but have not heard back yet.
badtiming 0 Reputation points

2023-10-10T14:05:40.5466667+00:00

i submitted a ticket and the answer i got from the support engineer was:

known issue

we are working on it

workaround is disabling http2
FARRELL Lisalee T (Tunde) 0 Reputation points

2023-10-10T15:07:18.44+00:00

We reached out MSFT under our premier support contract - this is a known issue that for some reason MSFT has decided to let the technical community figure out themselves. If you send Support the AppGw URL they can do something to address the issue for you until they have a global response. I have requested responses from MSFT for several days regarding timeframe and so far i just get a vague timeline.

Answer 1

@Jose Ramon Roca Garcia @Ventura Morcillo Valle @Bastien Noel @GH FARRELL Lisalee T (Tunde) FARRELL Lisalee T (Tunde) Catalao, Pablo badtiming

Thank you for reaching out. Posting this as an answer for visibility.

I have update from the product team that the underlying issue has been resolved and they will share a Root Cause Analysis soon. I will update this answer as soon as we receive the RCA.

Please tag me in your response if you are still facing this issue.

[Update 10/11]

We received the Root Cause Analysis for this issue from the Product Group:

In light of discovery of the new Rapid Reset Http2 attack , rate limiting was introduced to thwart such attacks. Platform team was in process of releasing the protection for http2 vulnerability ,We initially rolled out a rate limiting strategy which caused a genuine customer traffic to be throttled and resulted in too aggressive behavior. Rate limiting strategy was tweaked over a period of time, which is now stable
and thus reducing the false positive. This rate limiting was introduced in context of https://msrc.microsoft.com/blog/2023/10/microsoft-response-to-distributed-denial-of-service-ddos-attacks-against-http/2/

Thank you!

Regards,

Chaitanya

@Jose Ramon Roca Garcia It will help if you could mark this as answered as it will be easier for the community members to understand the resolution for this issue.

Share via

ERR_HTTP2_PROTOCOL_ERROR on Application Gateway WAF_v2

1 answer

Your answer