Jcoiffin's WebLog

ContentIndexState AutoSuspended for lagged copies

jcoiffin — Fri, 03 Mar 2017 15:30:27 GMT

Your Exchange 2016 lagged copies may stopped being indexed with ContentIndexState AutoSuspended after applying recent CU4 You may also see these events : ID: 1009 Level: Warning Source: MSExchangeFastSearch Message : The indexing of mailbox database encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---> Microsoft.Exchange.Search.Core.Abstraction.CatalogReseedException: The database 'guid ()' needs to be crawled but is not mounted on the preferred active server. Requesting a reseed. at Microsoft.Exchange.Search.Engine.SearchFeedingController.DetermineFeederStateAndStartFeeders() at Microsoft.Exchange.Search.Engine.SearchFeedingController.InternalExecutionStart() at Microsoft.Exchange.Search.Core.Common.Executable.InternalExecutionStart(Object state) --- End of inner exception stack trace --- at Microsoft.Exchange.Search.Core.Common.Executable.EndExecute(IAsyncResult asyncResult) at Microsoft.Exchange.Search.Engine.SearchRootController.ExecuteComplete(IAsyncResult asyncResult)" This is in fact expected behavior to Autosuspend lagged copies when possible. This change allows to save network bandwidth by suspending such index which would likely not be used. I would recommend to keep this normal behavior. But if for some reason you require such database to be indexed , you can use this command to block this feature and index all databases as in CU1 : New-SettingOverride -Name DisableSkipLagCopyForOnPrem2016 -Component Search -Section SkipLagCopy -Parameters @("Enabled=false") -Reason "Disable SkipLagCopy"

100% CPU on Exchange 2013/2016 - Check the number of CPUs is not too high

jcoiffin — Wed, 30 Nov 2016 13:37:15 GMT

100% CPU issues seen recently due to high number of CPU. Max recommended CPU is 24 CPU is usually consumed by mapihttp module. You will find below Resolution and Further details on the symptom and troubleshooting. Resolution : --------------- When there is too much CPU , more CPU time is spend due to minimum number of threads per CPU and synchronization named Contention Exchange/ .net is not designed/optimized for so much CPUs and this is not following Exchange architecture recommendations. There is a great article on Exchange dev team on this subject: https://blogs.technet.microsoft.com/exchange/2015/06/19/ask-the-perf-guy-how-big-is-too-big/ Such problem and far more can be avoided by following design recommended named Preferred Architecture, or requirement calculators which integrate this maximum CPU number and provide further details to get best performance. In most case the issue can be mitigated by disabling hyperthreading which is not recommended on Exchange and double artificially the number of CPU triggering this problem. When it is physical CPU without hyperthreading , there are sometimes hardware options to disable part of the CPUs in order to avoid exceeding max number of CPU recommendation. There is also a .net optimization for these contention generated by high number of CPU : .net fix KB3146718 (Web Installer) or KB3146717 both contains this optimization. Choose one package or the other depending if you need a web installer version or not For applying these hotfixes , it is also recommended to apply CU13 or upper : https://technet.microsoft.com/en-us/library/ff728623.aspx Such issues are fixed by applying this .net fix and disabling hyperthreading which reduce as well the number of CPU. Further details on symptom and troubleshooting: -------------------------------------------------------------------- Generic troubleshooting steps for 100% CPU is to check which process consume the CPU. This can be seen from task manager during the problem or using daily perfmon logs stored under \Microsoft\Exchange Server \V15\Logging\Diagnostics\DailyPerformanceLogs In most case I have seen, the process consuming most CPU in case of too much CPU is the w3wp.exe running MSExchangeMapiMailboxAppPool linked with mapihttp module. c:\windows\system32\inetsrv\w3wp.exe -ap "MSExchangeMapiMailboxAppPool" -v "v4.0" -c "C:\Program Files\Microsoft\Exchange Server\V15\bin\MSExchangeMapiMailboxAppPool_CLRConfig.config" -a \\.\pipe\iisipm93b74187-ee0a-4645-9295-38cfcbed72a4 -h "C:\inetpub\temp\apppools\MSExchangeMapiMailboxAppPool\MSExchangeMapiMailboxAppPool.config" -w "" -m 0 Tools to trace further causes of 100% CPU are usually Perfview or xperf trace as they help to narrow down the functions consuming most CPU. In these trace , you will see clr.dll!AwareLock::Contention the stack for main CPU consumption Another way to trace high CPU is using dumps . In these dumps , you will see lots of threads waiting on clr!AwareLock::Contention 00007ffee4b70c6a ntdll!ZwWaitForMultipleObjects+0xa 00007ffee1f913ed KERNELBASE!WaitForMultipleObjectsEx+0xe1 00007ffed8e38976 clr!WaitForMultipleObjectsEx_SO_TOLERANT+0x62 (Inline) clr!Thread::DoAppropriateAptStateWait+0x45 00007ffed8e3877a clr!Thread::DoAppropriateWaitWorker+0x1d0 00007ffed8e38591 clr!Thread::DoAppropriateWait+0x7d 00007ffed8e3883d clr!CLREventBase::WaitEx+0xc0 00007ffed8e53ed6 clr!AwareLock::EnterEpilogHelper+0xc6 00007ffed8e53dea clr!AwareLock::EnterEpilog+0x62 00007ffed8e54521 clr!AwareLock::Contention+0x1e3 00007ffed8e54293 clr!JITutil_MonContention+0xaf 00007ffed769bc35 mscorlib_ni!System.Threading.TimerQueueTimer.Fire()+0xc5

Troubleshoot Exchange 2013/2016 Managed Availability

jcoiffin — Wed, 21 Oct 2015 04:12:47 GMT

Exchange 2013/2016 has a Monitoring system called Managed Availability which detects and recovers automatically from problems as they occur and as they are discovered. The large majority of bluescreen and process recycle you can encounter on Exchange server are triggered by this module in order to restore the service as quick as possible by forcing a restart. I am sharing here a script that can help troubleshoot issues related to this module. Download the script from codeplex This tool helps troubleshoot these scenarios : ----------------------------------------------------------- - Bluescreen / Force reboot

Here is the information you will see if you capture a memory.dmp during a crash triggered by Managed Availability: The crash will be a stop F4 or EF on wininit. kd> !analyze –v … MODULE_NAME: wininit PROCESS_NAME: msexchangerepl BUGCHECK_STR: 0xEF_msexchangerepl DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT

- Service & application pool restart , components disabled , and all possible recovery actions triggered by Managed Availability - SCOM alerts reporting Managed Availability failures - Unhealthy monitor

ManagedAvailabilityTroubleshooter script allows to : ------------------------------------------------------------------ - Automatically identify related responder , monitor & probe for these different scenarios and dump their properties - Easily visualize the related events with detailed fields - Provide the PowerShell command to disable the recovery action as a workaround if needed while troubleshooting - Provide automatic resolution recommendation if you are looking at a probe which has known issue - provided PowerShell command used with verbose switch. Most properties can't be seen by opening related eventlog and require to run PowerShell commands to see related attributes. How to run the tool : --------------------------- Execute the script directly on the Exchange 2013 server you need to investigate using Exchange PowerShell. Copy the script to c:\temp Or Collect the logs & analyze on a different machine : Copy the script to c:\temp Run this command from Exchange PowerShell : .\ManagedAvailabilityTroubleshooter.ps1 –Collect This will generate a zip file with logs useful to investigate Managed Availability issues To Analyze the logs on a different machine : Extract the zip generated on the machine Copy the ManagedAvailabilityTroubleshooter script in the path of these collected logs. Run it using Windows PowerShell The script will use the files in local path. In case some files are missing or duplicate in your directory, you will get a warning. As an example here , I deleted the system log. Select the scenario you like to troubleshoot: -------------------------------------------------------- The ManagedAvailabilityTroubleshooter script will ask you to select the option that best describe the your issue Default is F to investigate force reboot. In this example , the problem is pop3 component marked as inactive. In that case , I type A and then enter to use the “All Recovery Actions” option. Most scenario will start by displaying the different failures /recovery actions you had on the machine and ask you to select the one you like investigate. Disabling the recovery action while troubleshooting in case this recovery action create more problem than solution : ------------------------------------------------------------------------------------------ Recovery actions as forcereboot via bluescreen are used to restore the service as quick as possible for end-users. In case this action does not help and appears too often , you may want to temporarily disable the responder while you investigate. The script will provide you the PowerShell command to run for that purpose. If you run the script directly on the server, you can set this override directly using the script and check if the change is applied. Troubleshooting the root cause : ------------------------------------------ The ManagedAvailabilityTroubleshooter script automatically identify related modules : Responder , Monitor and Probe. It will display their properties and relationship. The script automatically display last 200 probe events of the Probe which should be the root cause events. By default it is parsing only last 200 probe events for quicker response time. If your logs were not collected quickly after the issue , you have the option afterwards to parse all events but this is of course slower. You can use "Add criteria" button to filter results and for example look only at failures : The time of the events is displayed in GMT timezone. Automatic resolution recommendation: -------------------------------------------------- The script can provide automatic resolution recommendation if you are looking at a probe which has known issue/recommendation. This is the case here : it was due to non-default pop3 settings blocking 127.0.0.1 binding .As result pop3 availability test performed on this IP is failing. Checking a specific probe/Monitor/Responder: ----------------------------------------------------------- You may want to check events for a specific probe. In this example you may have fixed the pop3 setting and not encountering the issue anymore but like to check the current pop3 probe events to be sure they are fine. In that case when you run the ManagedAvailabilityTroubleshooter script , type S +Enter for checking a specific probe/monitor and then enter the name of the probe as in the printscreen below ( probe name is POPProxyTestProbe in this example as you can see in Troubleshooting section ) Verbose parameter allows to see PowerShell commands used : -------------------------------------------------------------------------------- You might want to run some PowerShell command manually to see similar information and eventually modify it. You can find the PowerShell command used by the tool using the -Verbose switch :

nombre Maximum de Logs De Transactions

jcoiffin — Thu, 27 Aug 2009 19:29:16 GMT

J'ai rencontré récemment différentes productions bloquées par cet événement :

Toutes les bases Exchange d'un groupe de stockage s'arrêtent et cet événement apparait dans le log d'application du serveur :

Event ID: 9518
Event Type: Error
Event Source: MSExchangeIS
Event Category: General
Description: Error 0xfffffddc starting Storage Group Path_of_Storage_Group on the Microsoft Exchange Information Store. Storage Group - Initialization of Jet failed.

Ce problème est bien connu et est référencé dans cet article de Microsoft https://support.microsoft.com/kb/830408.

Il est du au fait qu'on ait atteint le nombre maximum de log de transactions pour la base.

Si vous rencontrez actuellement ce problème :

Pour vous assurer que vous avez bien ce problème , vérifiez que vous avez bien dans l'événement 9518 le code 0xfffffddc qui signifie JET_errLogSequenceEnd.
S'il s'agit d'un autre code , vous rencontrez un autre problème que celui que je décris ici.

Si c'est bien le cas, vérifiez que chacune des bases du groupe de stockage s'est arrêtée correctement.
Pour faire cela :
    - lancez une commande cmd
    - allez dans le répertoire des binaires d'exchange (par exemple c:program filesexchsrvrbin )
    - lancez la commande eseutil /mh ( par exemple eseutil /mh "c:program filesexchsrvrmdbdatapriv1.edb"). La ligne State doit afficher Clean Shutdown.

Si toutes les bases du groupe de stockage sont bien en Clean Shutdown , vous déplacez tous les logs de transactions de ce groupe de stockage dans un répertoire de sauvegarde en utilisant un couper coller de sorte à ce qu'il ne reste plus de logs dans le répertoire de transaction du groupe de stockage.
Puis vous remontez les bases , une par une et l'incident est résolu.

Il est d'ailleurs important de faire une sauvegarde complète du groupe de stockage juste après cette manipulation car comme un nouveau jeux de logs a été généré , ces nouveaux logs de transactions ne peuvent pas être rejoués sur la base sauvegardée la veille.

Pour éviter de rencontrer ce problème :
Il y a un événement qui apparait quelques jours avant qu'on ait épuisé le nombre maximum de logs de transactions et qui vous permet donc de planifier l'opération et d'éviter cet arrêt brutal de production.
Voici l'événement :

Event ID : 514
Event Type: Warning
Source : ESE

Descriptif : log sequence numbers for this instance have almost been completely consumed. The current log generation is 1047950 wich is approching for the maximum log generation of 1048559.

Cet événement est normalement surveillé par SCOM ( System Center Operation Manager).
Si le serveur Exchange est surveillé par SCOM et que le management pack Exchange 2003 ( ou le mangement pack Exchange 2007 s’il s’agit d’un serveur Exchange 2007) et bien installé sur le serveur SCOM , une alerte doit vous avertir de l’événement et vous éviter l’arrêt de production.

Si vous utilisez un autre outil de surveillance , il faudrait s’assurer qu’il surveille bien cet événement 514.

Une bonne façon de s’en assurer est de le tester en générant un événement factice 514 et en regardant si votre outil de surveillance vous reporte une alerte.

Pour créer l’événement , sous Windows 2003 , lancez un cmd et tapez la commande :

eventcreate /L APPLICATION /t Warning /ID 514 /D “test de monitoring – event 514”

Une autre méthode est d’utiliser powershell ( cette méthode est plutôt pour Exchange 2007 où Powershell est déjà installé ):

$log = New-Object System.Diagnostics.EventLog
$log.set_log("Application")
$log.set_source("ESE")
$log.WriteEntry('Test event 514 monitoring',"Warning",514)