Teams webhook suddenly stopped working

Question

Teams webhook suddenly stopped working

Jon Baker 30

We have a CI/CD pipeline that sends messages to a Teams channel via the "Incoming Webhook" connector. Yesterday at 3:47PM Central was the last time that the webhook worked; today I'm getting a 200 response, but the body of the response shows that something is wrong... the gist of it is "XXX is unauthorized, the public key XXX is not found for ring WW, cannot validate signature."

I tried creating a new connector but am getting the same response. And, when I delete the new connector and submit the form to tell why, I get an HTTP error there too.

Is there an outage? Did something change that I need to address?

Leonardo Dourado 0 Reputation points

2023-10-16T20:59:09.9366667+00:00

I'm facing the same issue. Tried to create new connectors, but they keep giving error.
ChetanSharmamsft 1,036 Reputation points Microsoft External Staff Moderator

2023-10-17T04:53:55.7733333+00:00

Hello Everyone - Thanks for raising your query.
We will look into it and let you know the updates.
John Weng 0 Reputation points

2023-10-17T05:22:36.85+00:00

I have the same issue from yesterday and don't know how to get it supported ? any idea ?
Anonymous

2023-10-17T07:21:35.6533333+00:00

Could you please confirm if you are also facing similar error like the one mentioned here - https://techcommunity.microsoft.com/t5/teams-developer/webhook-messages-aren-t-delivered-from-azure-functions/m-p/3954072#M8597. Our incoming webhook connectors are working fine.
Matt Rydzik 200 Reputation points

2023-10-17T08:07:54.4266667+00:00

Our error is not the same as the linked. Incoming webhook connectors are not working fine for us and many others.

See https://answers.microsoft.com/en-us/msteams/forum/all/teams-webhook-error-despite-200-ok/920464d2-dd9c-4fae-847a-99855f6498f2 for others with the same issue.

The error given by the MS side is:
Microsoft.Exchange.Security.TokenIssuer.Common.SubstrateTokenRequestException: Request 427c2e45-22b1-43a3-94dd-e277dfd221e1;1697517710;1.0;48af08dc-f6d2-435f-b2a7-069abd99c086;https://outlook.office.com;;[],bd581b29e7bf4a48b16e7fc6bca2cb66,48af08dc-f6d2-435f-b2a7-069abd99c086,;GlobalActor;{TokenScopeType,CloudInstance},{AccessType,HpaAsApp},{PopKey,{"kty":"RSA","n":"v95EizVpL385wCMW8sD3n_qACwnPKv2IstraFBLo_RU5bq6Ky8BwSDw_U-SB42qdH-W8X7BKBE2IRdfrT4aJkGgfMqQhPFAmH-xr-hIbBr6b4bcfjaocrGLpGxqS5b60MszTKdooHIy6JQasfsnzDJ367mFvtPj9fwAgAynKr-PTfI3y8Pn2-oZcuaz9jX2aF0qEngWOasr4sLVrq654NxUD7MBf3He-Yd_1-bYAMtVNfsG0Iv8Ih6HJ4EJcJUbSfBvgc1SWqrg0IGGJJZYcsodJkQ36DnQeUk28NGT2KdNawyzKZDAaVpxunH6eFOCE2NguwjN_Y-TrNZSHIHEKaQ","e":"AQAB","alg":"RS256","kid":"749f5a89-100e-44fc-9b8c-5c4a014bb709"}};warqvR45thDtXFmf0RCvmtXXdLU=;lejdjFZ-mokS5YZuZ55T4fppdDY to https://localhost:444/sts/token/1 on server DM8PR01MB6856 failed with status code 401(Unauthorized) and reason {"Message":"STI server DM8PR01MB6856 failed to process request. Error: Request "427c2e45-22b1-43a3-94dd-e277dfd221e1;1697517710;1.0;48af08dc-f6d2-435f-b2a7-069abd99c086;https://outlook.office.com;;[],bd581b29e7bf4a48b16e7fc6bca2cb66,48af08dc-f6d2-435f-b2a7-069abd99c086,;GlobalActor;{TokenScopeType,CloudInstance},{AccessType,HpaAsApp},{PopKey,{"kty":"RSA","n":"v95EizVpL385wCMW8sD3n_qACwnPKv2IstraFBLo_RU5bq6Ky8BwSDw_U-SB42qdH-W8X7BKBE2IRdfrT4aJkGgfMqQhPFAmH-xr-hIbBr6b4bcfjaocrGLpGxqS5b60MszTKdooHIy6JQasfsnzDJ367mFvtPj9fwAgAynKr-PTfI3y8Pn2-oZcuaz9jX2aF0qEngWOasr4sLVrq654NxUD7MBf3He-Yd_1-bYAMtVNfsG0Iv8Ih6HJ4EJcJUbSfBvgc1SWqrg0IGGJJZYcsodJkQ36DnQeUk28NGT2KdNawyzKZDAaVpxunH6eFOCE2NguwjN_Y-TrNZSHIHEKaQ","e":"AQAB","alg":"RS256","kid":"749f5a89-100e-44fc-9b8c-5c4a014bb709"}};warqvR45thDtXFmf0RCvmtXXdLU=;lejdjFZ-mokS5YZuZ55T4fppdDY" is unauthorized, the public key warqvR45thDtXFmf0RCvmtXXdLU= is not found for ring WW, cannot validate signature"}. The response headers are: Date:Tue, 17 Oct 2023 04:41:49 GMT;Server:Microsoft-HTTPAPI/2.0.'
Thuan Le 0 Reputation points

2023-10-17T10:50:52.54+00:00

Same problem since yesterday.
Daniel Tebbutt 0 Reputation points

2023-10-17T11:06:43.95+00:00

We are having a similar issue with Webhooks called by ADF. Slowness noticed yesterday 16/10, but much worse today and causing timeout failures. There seems to be a backlog of requests, we get the response after approx 20 mins
Jon Baker 30 Reputation points

2023-10-17T13:11:10.2433333+00:00

@Anonymous No, it's not the same. The http response is actually a 200 response, but the body is showing the above error.
Jon Baker 30 Reputation points

2023-10-17T13:12:41.4266667+00:00

My Postman test was successful just now, so it's possible this issue has been resolved. I'll keep my eyes on it today.
Jon Baker 30 Reputation points

2023-10-17T13:16:11.62+00:00

(deleted, duplicate comment)
Thuan Le 0 Reputation points

2023-10-17T13:52:53.81+00:00

Mine still takes forever to send message. At least yesterday it had like 5% success rate.

Aleksandrovic Palma 5

I can report the same issue on our side. We set up the weebhooks in teams months ago with no issues. They started to present intermittent behavior on 11-10-2023.

We are using Python 3.11.5 and pymsteams 0.2.1.

Here our script to generate the message:

import pymsteams

ms_teams_webhook_url=<WebbhookUrl>
ms_teams_message=pymsteams.connectorcard(ms_teams_webhook_url)
ms_teams_message.text(message)
ms_teams_message.send()

The first error that we can report is this one:

Webhook message delivery failed with error: Microsoft Teams endpoint returned HTTP error 500 with ContextId tcid=0,server=msgapi-production-weu-08-0,cv=9By0aPSKMEC/06j+RzUy+Q.001=2..

Then, we have received the following error message on 13-10-2023 as reported in this issue:

Microsoft.Exchange.Security.TokenIssuer.Common.SubstrateTokenRequestException: ... is unauthorized, the public key <PUBLIC_KEY> is not found for ring WW, cannot validate signature"}. The response headers are: Date:Fri, 13 Oct 2023 12:30:44 GMT;Server:Microsoft-HTTPAPI/2.0.

Anonymous

2023-10-18T17:14:08.99+00:00

The issue has been mitigated, could you please confirm if it is working fine now.
Jon Baker 30 Reputation points

2023-10-18T17:30:29.86+00:00

It appears to be working for me. Thank you!

Answer accepted by question author

2 additional answers

Your answer

Leonardo Dourado 0 Reputation points

2023-10-16T20:59:09.9366667+00:00

I'm facing the same issue. Tried to create new connectors, but they keep giving error.
ChetanSharmamsft 1,036 Reputation points Microsoft External Staff Moderator

2023-10-17T04:53:55.7733333+00:00

Hello Everyone - Thanks for raising your query.
We will look into it and let you know the updates.
John Weng 0 Reputation points

2023-10-17T05:22:36.85+00:00

I have the same issue from yesterday and don't know how to get it supported ? any idea ?
Anonymous

2023-10-17T07:21:35.6533333+00:00

Could you please confirm if you are also facing similar error like the one mentioned here - https://techcommunity.microsoft.com/t5/teams-developer/webhook-messages-aren-t-delivered-from-azure-functions/m-p/3954072#M8597. Our incoming webhook connectors are working fine.
Thuan Le 0 Reputation points

2023-10-17T10:50:52.54+00:00

Same problem since yesterday.
Daniel Tebbutt 0 Reputation points

2023-10-17T11:06:43.95+00:00

We are having a similar issue with Webhooks called by ADF. Slowness noticed yesterday 16/10, but much worse today and causing timeout failures. There seems to be a backlog of requests, we get the response after approx 20 mins
Jon Baker 30 Reputation points

2023-10-17T13:11:10.2433333+00:00

@Anonymous No, it's not the same. The http response is actually a 200 response, but the body is showing the above error.
Jon Baker 30 Reputation points

2023-10-17T13:12:41.4266667+00:00

My Postman test was successful just now, so it's possible this issue has been resolved. I'll keep my eyes on it today.
Jon Baker 30 Reputation points

2023-10-17T13:16:11.62+00:00

(deleted, duplicate comment)
Thuan Le 0 Reputation points

2023-10-17T13:52:53.81+00:00

Mine still takes forever to send message. At least yesterday it had like 5% success rate.
Aleksandrovic Palma 5 Reputation points

2023-10-17T14:05:30.5766667+00:00

I can report the same issue on our side. We set up the weebhooks in teams months ago with no issues. They started to present intermittent behavior on 11-10-2023.

We are using Python 3.11.5 and pymsteams 0.2.1.

Here our script to generate the message:

import pymsteams ms_teams_webhook_url=<WebbhookUrl> ms_teams_message=pymsteams.connectorcard(ms_teams_webhook_url) ms_teams_message.text(message) ms_teams_message.send()

The first error that we can report is this one:

Webhook message delivery failed with error: Microsoft Teams endpoint returned HTTP error 500 with ContextId tcid=0,server=msgapi-production-weu-08-0,cv=9By0aPSKMEC/06j+RzUy+Q.001=2..

Then, we have received the following error message on 13-10-2023 as reported in this issue:

Microsoft.Exchange.Security.TokenIssuer.Common.SubstrateTokenRequestException: ... is unauthorized, the public key <PUBLIC_KEY> is not found for ring WW, cannot validate signature"}. The response headers are: Date:Fri, 13 Oct 2023 12:30:44 GMT;Server:Microsoft-HTTPAPI/2.0.
Anonymous

2023-10-18T17:14:08.99+00:00

The issue has been mitigated, could you please confirm if it is working fine now.
Jon Baker 30 Reputation points

2023-10-18T17:30:29.86+00:00

It appears to be working for me. Thank you!

Answer 1

Anonymous

There is an ongoing issue in Exchange Connectors. An incident has been raised related to this error - Web Hooks erroring randomly, unauthorized, the public key XXXXXXXXXXXXXXXXX = is not found for ring WW, cannot validate signature"}.

Anonymous

2023-10-18T17:13:55.1333333+00:00

The issue has been mitigated, could you please confirm if it is working fine now.

Answer 2

Thuan Le 0

It works normally today. Maybe the problem is fixed. Hope it won't come back to haunt us.

Answer 3

Matt Rydzik 200

Same thing here, retries sometimes work.

Share via

Teams webhook suddenly stopped working

2 additional answers

Your answer