Incoming webhook request never sends response and message delivered after hours

Rizqy 30

Hi, we have issue with incoming webhook on our teams.

First, the webhook request took forever and never sends a response back. It happened since like 2 weeks ago and getting worse. When I tried to send the request manually with Insomnia, I got the same result: no response. To anticipate it, we set a timeout to 10 seconds to prevent it from causing another issue on our system. On Insomnia, I need to cancel the request manually.

But surprisingly, the message was appearing on the Team's channel several hours later. All the webhook sent from the systems and Insomnia are present, but the orders are at random times. Later messages can appear first and vice versa. Sometimes they don't show at all.

Second, we tried to create a new webhook within the same channel, trying to isolate the issue. We assume the issue was related specifically to a webhook endpoint. Then we test the new endpoint with Insomnia. The result was the same as above. The request never ends and the message appeared hours later.

Third, we set up a new webhook in a new channel. Here we tried to isolate whether the issue was related to a specific channel or not. Again, the result was the same.

So, I guess this is our last effort bringing it up here. We suspect the issue was out of our hands. Please help.

Anonymous

2023-07-10T11:34:46.2133333+00:00

I'm experiencing exactly the same issue. Our incoming webhook url was working fine for about 2 years and it stopped working since yesterday. The symptom is 100% same as this post.

I regenerated new webhook url within the same channel, deleted the old one. But nothing solved. If there are 10 cards to send, only 5-6 messages are ended up shown up on our channel. Rest 5 to 4 giving me 'Read Timeout Error'. If I manually set the timeout to 5 minutes, same error occurs after 5 minutes.

Environment: Python -> pymsteams-> Teams incoming webhook
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Nivedipa-MSFT 3,021 Reputation points Microsoft Vendor

2023-07-11T06:42:03.42+00:00

@Rizqy - Could you please verify if the webhook endpoint or the system that receives the webhook requests is experiencing high load or performance issues? If there is a high load on the system, it may struggle to respond promptly, resulting in delays in receiving a response and messages appearing later than expected.
Rizqy 30 Reputation points

2023-07-11T07:45:36.39+00:00

@Nivedipa-MSFT Sorry, I think you got me wrong. My issue is regarding incoming webhook, not outgoing webhook. So we don't set an endpoint on our system, but we sent the request to webhook URL provided by Teams.
Nivedipa-MSFT 3,021 Reputation points Microsoft Vendor

2023-07-28T05:46:13.9466667+00:00

@Rizqy - We are checking this internally. We will update you soon.
Yen Li 0 Reputation points

2023-07-31T07:54:46.3533333+00:00

I 'm experiencing the same issue, sending POST request to to incoming webhook URL provided by teams, it never response.

If I use http instead of https, it response with "1" but no message received from teams.
Rizqy 30 Reputation points

2023-08-03T03:30:45.3833333+00:00

Thank you @Nivedipa-MSFT for your response. As subscriber, this is really disappointing and from the replies here, it might happened to other organizations as well. Do you have any ETA when will this issue could be solved?
Nivedipa-MSFT 3,021 Reputation points Microsoft Vendor

2023-08-09T09:48:48.9633333+00:00

Rizqy - Apologies for the delay. Could you please share the request details along with network trace so we can raise an ICM for this? Thanks.
Rizqy 30 Reputation points

2023-08-09T10:29:37.69+00:00
@Nivedipa-MSFT last time I tried, even the request with this simple payload was never finished:

{ "@type": "MessageCard", "@context": "https://schema.org/extensions", "themeColor": "48b6b5", "summary": "testing", "title": "testing" }

But this issue is intermittent. Sometimes it's running fine, in other times we got the timeout issues. The error rate today is decreasing with last error happened this morning at 6am.
Nivedipa-MSFT 3,021 Reputation points Microsoft Vendor

2023-08-11T09:52:40.0066667+00:00

@Rizqy - As mentioned before it would be helpful if you share the network trace along with request-id and timestamp when the request is getting delayed so that it would be easier for us to investigate further? Thanks.
Rizqy 30 Reputation points

2023-08-11T11:28:22.75+00:00

@Nivedipa-MSFT as I mentioned the first time, the request was never finished. How could we possibly identify the request ID? For the network trace, could you please share the example?
Rizqy 30 Reputation points

2023-08-11T11:31:05.5366667+00:00

@Nivedipa-MSFT as I mentioned the first time, the request was never finished. So how could we possibly identify the request ID? For the network trace, could you please share an example? Thank you.
Tim Schipper 0 Reputation points

2023-08-21T19:27:31.46+00:00

Also having this exact issue ... I do not get a response (at least not before I end the process after several minutes) and cards get created like at least an hour after the trigger ran.
Aiko Mastboom 0 Reputation points

2023-09-06T09:42:43.93+00:00

Same issue.. POSTs to webhook taking a very long time (seen up to 12 hours) to get handled.
Ian Chang 0 Reputation points

2023-09-28T03:46:57.18+00:00

Same issue here, totally the some.
Outgoing webhook request took very long time without response. But message comes hours later randomly.

this comment is trying to support the original author to indicate this is not a single case.
Schuchie 0 Reputation points

2023-10-17T09:45:15.68+00:00

We having the same issue since today 9am.

Results in a timeout via node js and postman:

FetchError: request to *** failed, reason: read ECONNRESET
52 at ClientRequest.<anonymous> (/opt/hostedtoolcache/node/20.8.0/x64/lib/node_modules/node_modules/node-fetch/lib/index.js:1501:11)
53 at ClientRequest.emit (node:events:514:28)
54 at TLSSocket.socketErrorListener (node:_http_client:495:9)
55 at TLSSocket.emit (node:events:514:28)
56 at emitErrorNT (node:internal/streams/destroy:151:8)
57 at emitErrorCloseNT (node:internal/streams/destroy:116:3)
58 at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
59 type: 'system',
60 errno: 'ECONNRESET',
61 code: 'ECONNRESET'
62}
Nivedipa-MSFT 3,021 Reputation points Microsoft Vendor

2023-10-18T12:23:13.7466667+00:00

@Schuchie - Could you please confirm if you are also facing similar error like the one mentioned here - https://techcommunity.microsoft.com/t5/teams-developer/webhook-messages-aren-t-delivered-from-azure-functions/m-p/3954072#M8597. Our incoming webhook connectors are working fine.

Schuchie 0

@Nivedipa-MSFT strangly the messages are working today again. It looks for me like a downtime somewhere between github and ms teams. We did not change anything at the messages at all.

Following our payload

{   "type": "message",   "attachments": [     {       "contentType": "application/vnd.microsoft.card.adaptive",       "contentUrl": null,       "content": {         "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",         "msteams": {           "width": "full"         },         "type": "AdaptiveCard",         "version": "1.4",         "body": [           {             "type": "Container",             "id": "a8a0a8d8-e873-ff2c-c6d2-d9a0b8bd7faf",             "padding": "Small",             "style": "attention",             "items": [               {                 "type": "TextBlock",                 "id": "0f04f954-50d9-ab29-f58e-c00d9c419f11",                 "text": "❌ WI-000537 validation failed ❌",                 "wrap": true,                 "weight": "Bolder",                 "size": "Large",                 "style": "heading",                 "color": "Attention",                 "horizontalAlignment": "Center",                 "spacing": "None"               }             ],             "spacing": "None"           },           {             "type": "Container",             "id": "f6ac87f5-c560-bb2f-cd2f-75e87b706bd9",             "padding": {               "top": "Small",               "bottom": "Default",               "left": "Small",               "right": "Small"             },             "items": [               {                 "type": "FactSet",                 "id": "3121e608-7133-3f9b-c8f2-a2d7f39437cc",                 "facts": [                   {                     "title": "Repository",                     "value": "USER/REPO"                   },                   {                     "title": "Branch",                     "value": "WI-000537"                   }                 ],                 "spacing": "None",                 "separator": true               }             ],             "minHeight": "45px"           },           {             "type": "Container",             "id": "c8472889-6d94-90e5-5311-3ede0967cd86",             "items": [               {                 "type": "Container",                 "separator": true,                 "items": [                   {                     "type": "ColumnSet",                     "columns": [                       {                         "type": "Column",                         "items": [                           {                             "type": "TextBlock",                             "weight": "Bolder",                             "text": "Error Details",                             "wrap": true,                             "spacing": "none"                           }                         ],                         "width": "stretch",                         "padding": "none"                       },                       {                         "type": "Column",                         "spacing": "Small",                         "verticalContentAlignment": "Center",                         "items": [                           {                             "type": "Image",                             "id": "ErrorDetailsDown",                             "url": "https://adaptivecards.io/content/down.png",                             "width": "20px",                             "altText": "Collapsed"                           },                           {                             "type": "Image",                             "id": "ErrorDetailsUp",                             "url": "https://adaptivecards.io/content/up.png",                             "width": "20px",                             "altText": "Expanded",                             "isVisible": false                           }                         ],                         "width": "auto"                       }                     ],                     "padding": "none"                   }                 ],                 "padding": "Default",                 "style": "emphasis",                 "selectAction": {                   "type": "Action.ToggleVisibility",                   "targetElements": [                     "ErrorDetailsWrapper",                     "ErrorDetailsDown",                     "ErrorDetailsUp"                   ]                 },                 "spacing": "none"               },               {                 "type": "Container",                 "id": "ErrorDetailsWrapper",                 "isVisible": false,                 "padding": "Small",                 "spacing": "none",                 "separator": true,                 "items": [                   {                     "type": "ColumnSet",                     "columns": [                       {                         "type": "Column",                         "items": [                           {                             "type": "TextBlock",                             "horizontalAlignment": "left",                             "text": "1.",                             "wrap": true,                             "size": "medium",                             "isSubtle": true                           }                         ],                         "width": "auto",                         "padding": "none",                         "spacing": "none"                       },                       {                         "type": "Column",                         "items": [                           {                             "type": "FactSet",                             "facts": [                               {                                 "title": "Profile Analytics",                                 "value": "Integration User                  Error          You cannot deploy to a required field: CategoryQuantityRule__c.Location__c"                               }                             ],                             "spacing": "none"                           }                         ],                         "width": "stretch",                         "spacing": "none",                         "padding": "None"                       }                     ],                     "padding": "None",                     "spacing": "none",                     "separator": true,                     "horizontalAlignment": "Left"                   },                   {                     "type": "ColumnSet",                     "columns": [                       {                         "type": "Column",                         "items": [                           {                             "type": "TextBlock",                             "horizontalAlignment": "left",                             "text": "2.",                             "wrap": true,                             "size": "medium",                             "isSubtle": true                           }                         ],                         "width": "auto",                         "padding": "none",                         "spacing": "none"                       },                       {                         "type": "Column",                         "items": [                           {                             "type": "FactSet",                             "facts": [                               {                                 "title": "Profile",                                 "value": "User Profile                        Error          You cannot deploy to a required field: CategoryQuantityRule__c.Location__c"                               }                             ],                             "spacing": "none"                           }                         ],                         "width": "stretch",                         "spacing": "none",                         "padding": "None"                       }                     ],                     "padding": "None",                     "spacing": "none",                     "separator": true,                     "horizontalAlignment": "Left"                   },                   {                     "type": "ColumnSet",                     "columns": [                       {                         "type": "Column",                         "items": [                           {                             "type": "TextBlock",                             "horizontalAlignment": "left",                             "text": "3.",                             "wrap": true,                             "size": "medium",                             "isSubtle": true                           }                         ],                         "width": "auto",                         "padding": "none",                         "spacing": "none"                       },                       {                         "type": "Column",                         "items": [                           {                             "type": "FactSet",                             "facts": [                               {                                 "title": "Profile BENEO",                                 "value": "Error          You cannot deploy to a required field: CategoryQuantityRule__c.Location__c"                               }                             ],                             "spacing": "none"                           }                         ],                         "width": "stretch",                         "spacing": "none",                         "padding": "None"                       }                     ],                     "padding": "None",                     "spacing": "none",                     "separator": true,                     "horizontalAlignment": "Left"                   },                   {                     "type": "ColumnSet",                     "columns": [                       {                         "type": "Column",                         "items": [                           {                             "type": "TextBlock",                             "horizontalAlignment": "left",                             "text": "4.",                             "wrap": true,                             "size": "medium",                             "isSubtle": true                           }                         ],                         "width": "auto",                         "padding": "none",                         "spacing": "none"                       },                       {                         "type": "Column",                         "items": [                           {                             "type": "FactSet",                             "facts": [                               {                                 "title": "Profile Cropenergies",                                 "value": "Error          You cannot deploy to a required field: CategoryQuantityRule__c.Location__c"                               }                             ],                             "spacing": "none"                           }                         ],                         "width": "stretch",                         "spacing": "none",                         "padding": "None"                       }                     ],                     "padding": "None",                     "spacing": "none",                     "separator": true,                     "horizontalAlignment": "Left"                   },                   {                     "type": "ColumnSet",                     "columns": [                       {                         "type": "Column",                         "items": [                           {                             "type": "TextBlock",                             "horizontalAlignment": "left",                             "text": "5.",                             "wrap": true,                             "size": "medium",                             "isSubtle": true                           }                         ],                         "width": "auto",                         "padding": "none",                         "spacing": "none"                       },                       {                         "type": "Column",                         "items": [                           {                             "type": "FactSet",                             "facts": [                               {                                 "title": "Profile Guest",                                 "value": "User                                Error          You cannot deploy to a required field: CategoryQuantityRule__c.Location__c"                               }                             ],                             "spacing": "none"                           }                         ],                         "width": "stretch",                         "spacing": "none",                         "padding": "None"                       }                     ],                     "padding": "None",                     "spacing": "none",                     "separator": true,                     "horizontalAlignment": "Left"                   },                   {                     "type": "ColumnSet",                     "columns": [                       {                         "type": "Column",                         "items": [                           {                             "type": "TextBlock",                             "horizontalAlignment": "left",                             "text": "6.",                             "wrap": true,                             "size": "medium",                             "isSubtle": true                           }                         ],                         "width": "auto",                         "padding": "none",                         "spacing": "none"                       },                       {                         "type": "Column",                         "items": [                           {                             "type": "FactSet",                             "facts": [                               {                                 "title": "Profile Hellma",                                 "value": "You cannot deploy to a required field: CategoryQuantityRule__c.Location__c"                               }                             ],                             "spacing": "none"                           }                         ],                         "width": "stretch",                         "spacing": "none",                         "padding": "None"                       }                     ],                     "padding": "None",                     "spacing": "none",                     "separator": true,                     "horizontalAlignment": "Left"                   },                   {                     "type": "ColumnSet",                     "columns": [                       {                         "type": "Column",                         "items": [                           {                             "type": "TextBlock",                             "horizontalAlignment": "left",                             "text": "7.",                             "wrap": true,                             "size": "medium",                             "isSubtle": true                           }                         ],                         "width": "auto",                         "padding": "none",                         "spacing": "none"                       },                       {                         "type": "Column",                         "items": [                           {                             "type": "FactSet",                             "facts": [                               {                                 "title": "Profile Integration",                                 "value": "Error          You cannot deploy to a required field: CategoryQuantityRule__c.Location__c"                               }                             ],                             "spacing": "none"                           }                         ],                         "width": "stretch",                         "spacing": "none",                         "padding": "None"                       }                     ],                     "padding": "None",                     "spacing": "none",                     "separator": true,                     "horizontalAlignment": "Left"                   },                   {                     "type": "ColumnSet",                     "columns": [                       {                         "type": "Column",                         "items": [                           {                             "type": "TextBlock",                             "horizontalAlignment": "left",                             "text": "8.",                             "wrap": true,                             "size": "medium",                             "isSubtle": true                           }                         ],                         "width": "auto",                         "padding": "none",                         "spacing": "none"                       },                       {                         "type": "Column",                         "items": [                           {                             "type": "FactSet",                             "facts": [                               {                                 "title": "Profile Partner",                                 "value": "User                            Error          You cannot deploy to a required field: CategoryQuantityRule__c.Location__c"                               }                             ],                             "spacing": "none"                           }                         ],                         "width": "stretch",                         "spacing": "none",                         "padding": "None"                       }                     ],                     "padding": "None",                     "spacing": "none",                     "separator": true,                     "horizontalAlignment": "Left"                   },                   {                     "type": "ColumnSet",                     "columns": [                       {                         "type": "Column",                         "items": [                           {                             "type": "TextBlock",                             "horizontalAlignment": "left",                             "text": "9.",                             "wrap": true,                             "size": "medium",                             "isSubtle": true                           }                         ],                         "width": "auto",                         "padding": "none",                         "spacing": "none"                       },                       {                         "type": "Column",                         "items": [                           {                             "type": "FactSet",                             "facts": [                               {                                 "title": "Profile Sugar",                                 "value": "Error          You cannot deploy to a required field: CategoryQuantityRule__c.Location__c"                               }                             ],                             "spacing": "none"                           }                         ],                         "width": "stretch",                         "spacing": "none",                         "padding": "None"                       }                     ],                     "padding": "None",                     "spacing": "none",                     "separator": true,                     "horizontalAlignment": "Left"                   }                 ]               }             ],             "padding": "none",             "separator": true,             "spacing": "none"           },           {             "type": "Container",             "id": "3dea3674-83a1-b445-f9ae-480924d4fca1",             "padding": {               "top": "Medium",               "bottom": "Small",               "left": "Small",               "right": "Small"             },             "items": [               {                 "type": "ActionSet",                 "id": "c8a5ba7e-7ec6-53ae-79da-0cfb952a527e",                 "actions": [                   {                     "type": "Action.OpenUrl",                     "title": "Open GitHub",                     "url": "https://github.com/USER/REPO"                   },                   {                     "type": "Action.OpenUrl",                     "title": "FAQ",                     "url": "https://url.com"                   }                 ]               }             ],             "spacing": "None",             "separator": true,             "minHeight": "40px",             "verticalContentAlignment": "Bottom"           }         ],         "padding": "None"       }     }   ] }

1 answer

Verakorn Suwansawhang 5 Reputation points

2023-10-17T07:48:32.6633333+00:00

We have the same issue. Do you have any solved answer.

for more information - it always got timeout when call webhook and i don't think it's the network problem because when we call without payload, it's return payload error and not timeout.

update - the next day, everything back to work! The messages we've sent and got timeout yesterday that appear delay after hours.
Please sign in to rate this answer.

1 person found this answer helpful.
Schuchie 0 Reputation points

2023-10-17T09:45:00.4+00:00

We having the same issue since today 9am.

Nikolay Buslaev 0 Reputation points

2023-10-17T11:30:02.0866667+00:00

We having same issue since yesterday

Roberto Viegas Dutra 0 Reputation points

2023-10-17T12:54:49.8633333+00:00

We're experiencing the same issue today

Vitor Peixoto 0 Reputation points

2023-10-17T14:25:20.54+00:00

Same here, nothing since this morning.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Incoming webhook request never sends response and message delivered after hours

1 answer

Your answer