SharePoint 2016 application pool crash

john chendorain 60

I have two Windows 2016 servers and SharePoint 2016 installed with IIS running, and recently after applying the Sept. 2023 OS security update the Central Administration app pool is crashing perodically with the following issue found after analyzing the .dmp file with DebugDiag.

"In w3wp.exe.16816.dmp the assembly instruction at ucrtbase!invoke_watson+18 in C:\Windows\System32\ucrtbase.dll from Microsoft Corporation has caused an unknown exception (0xc0000409)" This issue has occurred on two different 2016 web servers servers with SP 2016 installed

Not sure if the latest Windows server or SharePoint update has caused this issue or not, as the two updates were applied two days prior to first crash. Thank you

Accepted answer

Yanli Jiang - MSFT 26,016 Reputation points Microsoft Vendor

2023-10-06T05:29:22.66+00:00

Hi @john chendorain ,

Great to know that and thanks for sharing the update here.

By the way, since the Microsoft Q&A community has a policy that "The question author cannot accept their own answer. They can only accept answers by others.". and according to the scenario introduced here: Answering your own questions on Microsoft Q&A, I would make a brief summary of this thread:

[SharePoint 2016 application pool crash]

Issue Symptom:
I have two Windows 2016 servers and SharePoint 2016 installed with IIS running, and recently after applying the Sept. 2023 OS security update the Central Administration app pool is crashing perodically with the following issue found after analyzing the .dmp file with DebugDiag.

"In w3wp.exe.16816.dmp the assembly instruction at ucrtbase!invoke_watson+18 in C:\Windows\System32\ucrtbase.dll from Microsoft Corporation has caused an unknown exception (0xc0000409)" This issue has occurred on two different 2016 web servers servers with SP 2016 installed

Not sure if the latest Windows server or SharePoint update has caused this issue or not, as the two updates were applied two days prior to first crash.

Current status:
So the issue was that the Sept. 2023 SP 2016 security update contained an antimalware feature that was enabled for all web applications. The resolution was to disable the hidden feature.

You could click the "Accept Answer" button for this summary to close this thread, and this can make it easier for other community members to see the useful information when they meet the similar issue.

Thanks again for sharing. :-)
Please sign in to rate this answer.
Moll, Joe 0 Reputation points

2023-10-15T14:45:00.3066667+00:00

Can you share how you disabled hidden feature or what the feature is? I'm only seeing disabling SharePoint Antimalware service for each web app

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

1 additional answer

Yanli Jiang - MSFT 26,016 Reputation points Microsoft Vendor

2023-09-27T06:58:18.01+00:00

Hi @john chendorain ,

It may be caused by the update. The error message you provided indicates an unknown exception occurred in the ucrtbase.dll file, which is a system file related to the Universal C Runtime in Windows. This could be caused by a conflict between the update and SharePoint 2016.

Let's do some checks:

Check to see if the latest CU is installed

Check to see if the latest Windows update is installed

Check the Event logs to see if there is more information

If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.
Please sign in to rate this answer.
john chendorain 60 Reputation points

2023-09-27T18:23:11.9166667+00:00

So I have six VMs with SP 2016 installed and all updates match. SharePoint is at the latest patch level Sept 2023 and the 2016 Server is at the latest patch level Sept 2023. Two of the six VMs are having issues, with just two specific app pools, with one in error on each. The Dev VM web server has this error in the event logs e.g. "A process serving application pool 'DEV_SharePoint' suffered a fatal communication error with the Windows Process Activation Service" both VMs have a simeltaneous system log Service Control Manager error "The Diagnostic Service Host service failed to start due to the following error: A privilege that the service requires to function properly does not exist in the service account configuration. You may use the Services Microsoft Management Console (MMC) snap-in (services.msc) and the Local Security Settings MMC snap-in (secpol.msc) to view the service configuration and the account configuration." The second prod web server has addtional event log errors and are as follows: "A worker process '7712' serving application pool 'SharePoint Central Administration v4' failed to stop a listener channel for protocol 'http' in the allotted time. The data field contains the error number", and "A process serving application pool 'SharePoint Central Administration v4' failed to respond to a ping. The process id was '3248", and "A process serving application pool 'SharePoint Central Administration v4' exceeded time limits during shut down. The process id was '7712". All are Windows Activation Service errors in the system log. The second VM which is a production web server that host central admin site now has degraded and throws an exception when attempting to open cnetral adim web site.

Yanli Jiang - MSFT 26,016 Reputation points Microsoft Vendor

2023-09-28T09:41:04.3533333+00:00

Based on the provided context, the issue with the SharePoint 2016 application pool crash might be caused by the System Center 2016 Operations Manager Application Performance Monitoring (APM) Agent, which uses the .NET Framework 2.0 application pool. To work around this issue, you can uninstall the Operations Manager 2016 monitoring agent or reinstall the Operations Manager 2016 agent without APM. Additionally, you can capture a dump file of the w3wp.exe process just before it terminates using DebugDiag to determine exactly why it crashed.

For your reference:

System Center 2016 Operations Manager APM Agent causing heap corruption in SharePoint

APM feature causes a crash for the IIS application pool running under .NET Framework 2.0 runtime

Troubleshoot application pool crashes on a Cloud Services virtual machine

You can have a try and hope this helps.

Due to the limitations of the forum, the support that can be provided is limited. If you want more direct and faster support, we would suggest you to open a ticket to contact Microsoft.

john chendorain 60 Reputation points

2023-09-28T14:41:49.9433333+00:00

So thanks for that addtional info realted to the SC Operations Manager APM agent issue. And again I did analyze the .dmp both the dev server and prod server app pool crashes utilizing Debug Diag and found the exception realted to that core C:\Windows\System32\ucrtbase.dll file. So now the question is how do we fix that file and is that at the root of the issue? Anyway I will open a ticket with MS today.

Yanli Jiang - MSFT 26,016 Reputation points Microsoft Vendor

2023-10-02T09:49:41.4166667+00:00

Sorry for replying to you so late. I was on vacation before. According to your reply, this belongs to the category of Windows and is not what I am good at. I cannot give you a good reply. Opening a ticket with MS is indeed a good choice. I hope you can solve this problem as soon as possible. Any progress is glad to be shared here. :-)

john chendorain 60 Reputation points

2023-10-02T18:16:25.65+00:00

Thanks for jumping in, and I do need to correct the issue with the windows core file so I will bring that question into the windows server support group. Currently I do have app pools periodically now throwing the Event ID 5011 Windows Activation Service, A process serving application pool 'SharePoint Central Administration v4' suffered a fatal communication error with the Windows Process Activation Service. The process id was '4536'. The data field contains the error number. Only web servers and generally only Central Admin app pool. No app pool crashes in the last 7 days and I have an engineer looking at the reports I generated for him with no solution at this point.

john chendorain 60 Reputation points

2023-10-05T17:46:10.87+00:00

So the issue was that the Sept. 2023 SP 2016 security update contained an antimalware feature that was enabled for all web applications. The resolution was to disable the hidden feature. Thanks again for all your input!
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

SharePoint 2016 application pool crash

1 additional answer

Your answer