mahuynh

Setup of first Exchange 2013 or 2016 server fails in a child domain due to AD replication

Matthew Huynh [MSFT] — Fri, 16 Jun 2017 21:12:21 GMT

Let's say you have a parent-child Active Directory topology of tailspintoys.com and child.tailspintoys.com. You have Exchange 2010 installed in the parent domain, and you want to install 2013 or 2016 into the child domain. You run through all the AD prep work. You then run Exchange setup to install your first mailbox server and it bombs out. Being the handy Exchange admin that you are, you know to consult C:\ExchangeSetupLogs\ExchangeSetup.log for all the gory details about the error. You see various errors such as:

Couldn't find recipient "tailspintoys.com/Users/SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c}". Reason: tailspintoys.com/Users/SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c} isn't a mailbox user.
DSID-03152C45, problem 1005 (CONSTRAINT_ATT_TYPE), data 0, Att 200f4 (homeMDB)

There are various permutations of this issue, but the root cause is the same: slow AD replication between the root domain and the child domain. In one version of this issue, here is Exchange setup is doing:

A mailbox database is created in the child domain.
The arbitration mailbox (residing in the root domain) is mail-enabled which modifies its homeMDB value.
The root DC processing this modification rejects it because the homeMDB value does not point to a valid object.

The root DC thinks the new homeMDB value is not valid because it hasn't replicated the newly-created mailbox database object from the child domain yet. What should we do here? Unfortunately, there are no great workarounds. One option is to wait for AD replication to catch up (at least 15 minutes usually), and then re-run setup. The other option is a bit more involved:

Install the first Exchange 2013 or 2016 mailbox server into the root domain so that the arbitration mailboxes will be staged properly.
Install another Exchange 2013 or 2016 mailbox server into the child domain.
Move all the arbitration mailboxes to the child domain's Exchange server (Get-Mailbox -Arbitration | New-MoveRequest -TargetDatabase MyCoolDb)
Uninstall the root domain's Exchange server.

This solution is a bit tedious but it'll get the job done without potentially re-running setup multiple times.

Check Managed Availability Probe Results in application log when your Exchange 2013 server has sustained high CPU

Matthew Huynh [MSFT] — Sat, 17 Dec 2016 00:05:11 GMT

I stumbled upon this neat feature in Exchange 2013. When your Exchange 2013 (and probably later) server is experiencing sustained high CPU, an event 2 error occurs inside your Microsoft-Exchange-ActiveMonitoring\ProbeResult channel. Look for an event with ResultName of EDS/ExchangeProcessorTimeTrigger_Warning/_total. Inside the details view, there are a bunch of top N lists that will give you quick insights into what is happening:


 
 
 2
 0
 2
 2
 0
 0x800000000000000
 
 75627993
 
 
 Microsoft-Exchange-ActiveMonitoring/ProbeResult
 EXCH2013.contoso.com
 
 
 
 
 75697922
 EDS
 1
 EDS/ExchangeProcessorTimeTrigger_Warning/_total
 -74577656
 0
 EXCH2013
 The performance counter '\\EXCH2013\Processor(_Total)\% Processor Time' sustained a value of '89.80', for the '5' minute(s) interval starting at '12/14/2016 4:42:00 PM'.%nAdditional information: 
Top 10 process time breakdown:
CPU % Process Name
279.5 w3wp#MSExchangeSyncAppPool
184.2 Microsoft.Exchange.Store.Worker
141.8 Microsoft.Exchange.Store.Worker#2
111.9 NodeRunner#ContentEngineNode1
 86.9 w3wp#MSExchangeRpcProxyFrontEndAppPool
 66.4 Microsoft.Exchange.RpcClientAccess.Service
 61.1 powershell#4
 47.3 powershell#5
 43.8 MSExchangeDelivery
 40.4 NodeRunner#IndexNode1

The total CPU in Kernel : 17.55441

\\EXCH2013\MSExchange Active Manager(_Total)\Database Mounted : 2

\\EXCH2013\MSExchange Active Manager(_Total)\Database Copy Role Active : 3

\\EXCH2013\MSExchange Search Indexes(_Total)\Crawler: Mailboxes Remaining : 0

\\EXCH2013\MSExchange Search Indexes(_Total)\Notifications: Awaiting Processing : 10247.25

\\EXCH2013\MSExchange Search Indexes(_Total)\Retry: Retriable Items : 1356622

Top 15 W3WP AppPools with the highest request rate:
Requests/sec: W3WP AppPool
 25.4 Total
 9.9 MSExchangeServicesAppPool
 4.9 MSExchangeSyncAppPool
 4.8 MSExchangeRpcProxyFrontEndAppPool
 1.8 MSExchangeRpcProxyAppPool
 1.2 MSExchangeAutodiscoverAppPool
 1.0 MSExchangeOWAAppPool
 0.9 MSExchangePowerShellAppPool
 0.3 MSExchangeMapiFrontEndAppPool
 0.2 MSExchangeOABAppPool
 0.2 MSExchangeMapiMailboxAppPool
 0.1 MSExchangePowerShellFrontEndAppPool
 0.1 MSExchangeECPAppPool
 0.0 MSExchangeOWACalendarAppPool

Top 15 components with the highest ROPs rate:
RPC Operations/sec: Component
1,977.6 _total
982.4 momt
402.9 contentindexing
333.0 airsync
159.5 transport
 79.1 eventbasedassistants
 10.5 webservices
 4.6 timebasedassistants
 2.5 owa
 1.0 availabilityservice
 0.6 storeactivemonitoring
 0.6 monitoring
 0.4 pop
 0.2 simplemigration
 0.2 management

Top 15 components with the highest RPC average latency:
RPC Average Latency: Component
 11.7 transport
 10.3 airsync
 6.4 eventbasedassistants
 5.4 _total
 4.8 contentindexing
 4.2 momt
 3.4 webservices
 1.9 management
 1.8 pop
 1.6 owa
 1.3 availabilityservice
 1.2 monitoring
 1.2 timebasedassistants
 1.1 storeactivemonitoring
 0.6 simplemigration
. Trigger Name:ExchangeProcessorTimeTrigger. Instance:_total
 ...snip...

Script: Enable and collect ExTRA tracing across all Exchange servers

Matthew Huynh [MSFT] — Fri, 05 Aug 2016 22:19:21 GMT

Here is a PowerShell script that can be used to kick off an ExTRA trace on every Exchange server in the org and collect the traces into a single location. I've highlighted in red the variables you'll likely want to edit prior to running the script.

# Description:
#   Initiates an ExTRA trace on all the Exchange 2013 servers
#   in an environment and collects the results into a single location.
# Use:
#   In an Exchange Management Shell (EMS), run either of these commands:
#     .\extraTrace.ps1 -Start
#     .\extraTrace.ps1 -Stop
# Author:
#   mahuynh@microsoft.com
# Last modified:
#   2016-06-28

[CmdletBinding()]
Param(
 [switch]$Start,
 [switch]$Stop
)

# check that user ran with either Start or Stop switch params
if (($Start -and $Stop) -or (-not $Start -and -not $Stop)) {
 Write-Error "Please specify only 1 parameter: -Start or -Stop."
 exit
}

$traceConfigFilepath = "C:\EnabledTraces.Config"
$script:nl = "`r`n"
# network path to save the resulting traces (default is local c:\temp\extra)
$TRACES_FILEPATH = "\\" + (hostname) + "\c$\temp\extra"

function CreateExtraTraceConfig
{
 new-item -path $traceConfigFilepath -type file -force
 $string = "TraceLevels:Debug,Warning,Error,Fatal,Info,Performance,Function,Pfd" + $nl
 $string += "MSExchangeWebServices:AllRequests,AllResponses,SubscribeCall,Subscriptions,UnsubscribeCall" + $nl
 $string += "FilteredTracing:No" + $nl
 $string += "InMemoryTracing:No" + $nl
 $string | Out-File -filepath $traceConfigFilepath -Encoding ASCII -Append
}

# main script start

# get a list of all the Exchange servers
$servers = Get-ExchangeServer

if ($Stop) {
 Get-Date
 Write-Host "Stopping ExTRA traces..." -ForegroundColor Cyan $nl

# create target path if it does not exist yet
 if (-not (Test-Path $TRACES_FILEPATH)) {
 mkdir $TRACES_FILEPATH | Out-Null
 Get-Date
 Write-Host "Created $TRACES_FILEPATH as it did not exist yet" -ForegroundColor Green $nl
 }

# stop and delete the trace configs, collect the results
 foreach ($serv in $servers) {
 Get-Date
 Write-Host "Disabling ExTRA tracing on" ($serv.Name) -ForegroundColor green $nl
 logman stop ExchangeDebugTraces -s $serv.Name
 logman delete ExchangeDebugTraces -s $serv.Name
 $fileToMovePath = "\\" + $serv.Name + "\c$\tracing\" + $serv.Name + "_*.etl"
 Move-Item $fileToMovePath $TRACES_FILEPATH -Force
 }

Get-Date
 Write-Host "Finished copying data to" $TRACES_FILEPATH -ForegroundColor Green $nl

} elseif ($Start) {
 Get-Date
 Write-Host "Starting ExTRA traces..." -ForegroundColor Cyan
 
 # create a local extra config file
 Get-Date
 Write-host "Creating local ExTRA trace..." -foregroundcolor Green $nl
 CreateExtraTraceConfig

# for each server
 foreach ($serv in $servers) {
 # copy the extra config file as long as it's not the local computer
 Get-Date
 Write-Host "Enabling ExTRA tracing on" ($serv.Name) -ForegroundColor green $nl
 if ((hostname).ToString().CompareTo($serv.Name) -ne 0) {
 Write-Debug "copying config file..."
 Copy-Item $traceConfigFilepath ("\\" + $serv.Name + "\c$\EnabledTraces.config") -Force
 }

# create a trace
 $outputFilepath = "c:\tracing\" + $serv.Name + ".etl"
 logman create trace ExchangeDebugTraces -p "Microsoft Exchange Server 2010" -o $outputFilepath -s $serv.Name -ow -f bincirc -max 1024

# start the trace
 logman start ExchangeDebugTraces -s $serv.Name
 }
}

How to take an ExTRA trace of Exchange setup

Matthew Huynh [MSFT] — Fri, 05 Feb 2016 11:30:03 GMT

This blog post is mainly for us Exchange support engineers but I like to surface things externally because it's easier to search and link to. This blog post is heavily inspired by https://blogs.technet.com/b/mikelag/archive/2008/08/22/how-to-enable-store-pfd-tracing-upon-reboot.aspx.

With Exchange 2007 and onwards, debug code tracing is logged to the ETW provider GUID 79BB49E6-2A2C-46E4-9167-FA122525D540. When you install Exchange, this ETW provider is registered formally and the GUI ExTRA application works. However, we can still manually do capture an ExTRA trace even if Exchange is not installed; the most common use would be setup issues.

Launch an administrative command prompt.
Create the c:\tracing folder.
Create the trace:
1. logman create trace ExchangeDebugTraces -p {79bb49e6-2a2c-46e4-9167-fa122525d540} -nb 128 256 -bs 128 -o c:\tracing\ExchangeDebugTraces
Create the trace configuration file at C:\EnabledTraces.Config and put in what you need. Its content will look something like this:
1. TraceLevels:Debug,Warning,Error,Fatal,Info,Performance,Function,Pfd
  ADProvider:Connection,ConnectionDetails,GetConnection,ADFind,DirectoryException
  Setup:Trace
  TopologyService:SuitabilityVerifier
  FilteredTracing:No
  InMemoryTracing:No
Start the trace:
1. logman start ExchangeDebugTraces
Reproduce the issue.
Stop the trace:
1. logman stop ExchangeDebugTraces
Parse the trace ETLs with extrace.exe. Remember to use the proper version of Microsoft.Exchange.Diagnostics.dll for proper parsing.

Cheers,

Matt

EWS script to create an appointment in a set of mailboxes defined in a CSV file using an impersonation account

Matthew Huynh [MSFT] — Thu, 03 Dec 2015 11:42:51 GMT

A customer recently wanted a script that would allow him to create an appointment with a reminder in a set of mailboxes that he defined in a CSV file. I was happy to oblige and this is the result. My script is based off of https://mikepfeiffer.net/2011/01/creating-calendar-items-with-powershell-and-the-ews-managed-api/. I wanted to post my script as it does do a couple of things differently:

has reminder support
has free/busy status support
it's a script rather than a cmdlet
has logging so you know against which mailboxes the script worked
it's been validated against Office 365

To run the script, you must:

Allow execution of unsigned PowerShell scripts: Set-ExecutionPolicy unrestricted
Install the EWS Managed API 2.2 from https://www.microsoft.com/en-us/download/details.aspx?id=42951
Create an impersonation account using https://msdn.microsoft.com/en-us/library/office/dn722376%28v=exchg.150%29.aspx
Create a CSV file where one of the columns has a header named EmailAddress

Notes:

There should be no throttling concerns as impersonated actions are counted against the target user's throttling policy rather than the impersonation account.
This script should work against any modern versions of Exchange.

CreateAppointmentItem.ps1

Restoring mail-enabled public folders after accidental deletion

Matthew Huynh [MSFT] — Thu, 02 Jul 2015 14:02:00 GMT

Sometimes, your entire public folder tree gets accidentally deleted. Luckily for you, this has happened to enough of our customers that the steps to do an authoritative restore of Public Folders is outlined in https://blogs.technet.com/b/exchange/archive/2012/02/06/recovering-public-folders-after-accidental-deletion-part-1-recovery-process.aspx. Sometimes, after doing an auth restore of your PF, some or all of your mail-enabled PF aren't mail-enabled anymore. You're forced to mail disable and then mail enable each previously mail-enabled PF. Not only this, but you need to set up the email addresses, mail forwarding, and all the other settings for each MPF again. Why is that? Let's take a quick dive into how mail-enabled public folders work.

When you mail enable a PF, an Active Directory (AD) object of objectClass publicFolder is created inside the Microsoft Exchange System Objects (MESO) container.

[PS] C:\>Get-MailPublicFolder | fl legacy*,emailAddresses,externalEmailAddress,organizationalUnit,*forward*

LegacyExchangeDN           : /O=PULSARWINDS/OU=EXCHANGE ADMINISTRATIVE GROUP (FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=F8B6366B5B039A4590816BCB85ED7832-00000000000E
EmailAddresses             : {smtp:myLittlePF@pulsarwinds.com, smtp:mahuynh1@PulsarWinds.mail.onmicrosoft.com, SMTP:mahuynh1@pulsarwinds.com}
ExternalEmailAddress       : expf:F8B6366B5B039A4590816BCB85ED7832-00000000000E
OrganizationalUnit         : pulsarwinds.com/Microsoft Exchange System Objects
DeliverToMailboxAndForward : True
ForwardingAddress          : pulsarwinds.com/Users/Administrator

When you mail disable a PF, its corresponding AD object in the MESO container gets deleted and thus all settings are lost. When you mail re-enable a PF, various fields like LegacyExchangeDN and ExternalEmailAddress calculate to the same values but other fields like email addresses and mail forwarding (highlighted above) are not preserved. For example:

[PS] C:\>Get-PublicFolder \mahuynh | Disable-MailPublicFolder -Confirm:$false
[PS] C:\>Get-PublicFolder \mahuynh | Enable-MailPublicFolder
[PS] C:\>Get-MailPublicFolder | fl legacy*,emailAddresses,externalEmailAddress,organizationalUnit,*forward*

LegacyExchangeDN           : /O=PULSARWINDS/OU=EXCHANGE ADMINISTRATIVE GROUP (FYDIBOHF23SPDLT)/CN=RECIPIENTS/CN=F8B6366B5B039A4590816BCB85ED7832-00000000000E
EmailAddresses             : {smtp:mahuynh1@PulsarWinds.mail.onmicrosoft.com, SMTP:mahuynh1@pulsarwinds.com}
ExternalEmailAddress       : expf:F8B6366B5B039A4590816BCB85ED7832-00000000000E
OrganizationalUnit         : pulsarwinds.com/Microsoft Exchange System Objects
DeliverToMailboxAndForward : False
ForwardingAddress          :

The easiest resolution here is to restore the deleted publicFolder AD objects. If your forest functional level is below 2008 R2, you'll have to do an auth restore of the MESO container to a backup taken before the PF tree deletion. Otherwise, you can use the AD recycle bin, which I will show you now.

First, ensure Active Directory module for Windows PowerShell is installed on your domain-joined computer. Launch a PowerShell window, load the AD module, and find your deleted PF objects.

Import-Module ActiveDirectory
cd AD:
$deletedMPF = Get-ADObject –SearchBase "CN=Deleted Objects,DC=pulsarwinds,DC=com" -Filter {lastKnownParent -eq 'CN=Microsoft Exchange System Objects,DC=pulsarwinds,DC=com' -and objectClass -eq 'publicFolder'} -IncludeDeletedObjects -Properties *

You can drill into the objects to selectively pick out which ones you want to restore. Let's say I wanted to identify only objects deleted since midnight and restore them.

PS AD:\> $recentlyDeletedMPF = $deletedMPF | ? {$_.whenChanged -gt "7/2/2015"}
PS AD:\> $recentlyDeletedMPF | sort whenChanged -desc | ft mail,whenChanged -auto

mail whenChanged
---- -----------
mahuynh1@pulsarwinds.com 7/2/2015 4:26:11 PM

PS AD:\> $recentlyDeletedMPF | Restore-ADObject

Pretty easy and nifty, eh? Now go find out who deleted your PF tree and discipline them!

Mailbox and mail sending permissions

Matthew Huynh [MSFT] — Tue, 30 Dec 2014 11:59:00 GMT

I recently had a case where we encountered an "issue" where users granted full mailbox access and send-on-behalf-of to a shared mailbox were able to use OWA to send as the shared mailbox. First, let's take a step back and review permissions. I've summarized it in a table here:

Type	Permission	Description	How to Grant
Mailbox	Folder-level permissions	allows a user to access particular folders (e.g. Inbox, Calendar, Tasks, etc.) inside a mailbox	Add-MailboxFolderPermission -Identity manager:\Inbox -User delegate -AccessRights Author
Mailbox	Full mailbox access	allows a user to do a secondary logon and browse the contents of a mailbox (read messages, edit the calendar, add new items, etc.)	Add-MailboxPermission -Identity manager -User delegate -AccessRights FullAccess
Mail sending	Send as	allows a user to spoof a manager	Add-ADPermission -Identity manager -User delegate -AccessRights ExtendedRight -ExtendedRights "Send As"
Mail sending	Send on behalf of	allows a user to send an email on behalf of a manager	Set-Mailbox -Identity manager -GrantSendOnBehalfTo delegate

If a manager grants a user full mailbox access, they'll be able to log onto the mailbox (OWA -> Open mailbox, Outlook -> Open these additional mailboxes). If a manager grants a user delegate access (using Outlook), they'll have to use Outlook's File -> Open functionality and choose a particular folder. If it is a special/default folder like the Calendar, the delegate will be able to use OWA's calendar view to view/edit the manager's calendar. In both these cases, the delegates won't be able to send as or send on behalf of the manager because mailbox and mail sending permissions are of two different types. Also something else to note: there are many ways to grant these permissions and I've only put down the Exchange PowerShell version. You can use Outlook, Exchange Management Console, Exchange Admin Center, and a host of other GUI tools to achieve the right permissions as long as they modify the appropriate data structures. As it turns out, the "issue" I described earlier is a by-design behavior of Exchange and is documented here: https://blogs.technet.com/b/ehlro/archive/2012/04/06/full-mailbox-access-rights-send-on-behalf-send-as.aspx. Basically, full mailbox access + send-on-behalf-of rights = send as permissions. Even if you add an explicit deny for the send as right, delegates will still be able to send as the manager or shared mailbox. Usually, this issue surfaces with delegates because organizations grant delegates access to their manager's mailbox using 2 sets of permissions:

The manager uses Outlook to grant a delegate access to their mailbox so the delegate receives copies of meeting requests and other delegate-y features. What's happening under the covers is that Outlook is performing a variety of configuration changes, including granting folder-level permissions and send-on-behalf-of rights.
The Exchange administrator then grants the delegate full mailbox access to the manager's mailbox so the mailbox is automapped into the Outlook profile (what's really happening is that Autodiscover is returning an additional mailbox for Outlook to open).

I hope this adds some clarity to the different types of permissions used in Exchange and Outlook as I know it is a frequent source of confusion, even for me. Edit on 7/27/2015: As of Exchange 2010 SP3 RU8, the Exchange Product Group acknowledged that a delegate that has full mailbox access should not be able to send-as the manager but only on-behalf-of. The previous behavior is a bug (https://support.microsoft.com/en-us/kb/2987104) and is fixed in 2010 SP3 RU8 and future versions. Edit on 3/23/2017: TechNet has a good page summarizing these permissions as well: Manage permissions for recipients [Exchange 2016] Cheers, Matt

Exchange 2013 domainprep (prepareAD) fails with Microsoft.Exchange.Data.Directory.AdminLimitExceededException [ERROR_DS_ADMIN_LIMIT_EXCEEDED]

Matthew Huynh [MSFT] — Wed, 29 Oct 2014 15:03:00 GMT

We've seen 2 cases in support with this issue that affected hosting companies. I anticipate we'll see more instances of this issue as more customers migrate to Exchange 2013, especially hosting companies, so I've decided to blog it out.

As with any installation issue, you start off by inspecting C:\ExchangeSetupLogs\ExchangeSetup.log. In there, you'll find that we were in the midst of creating Microsoft Exchange Security Groups (setup is inside the function initialize-ExchangeUniversalGroups) and then this error happens:

[10/15/2014 19:49:01.0934] [2] [ERROR] The administrative limit for this request was exceeded.
[10/15/2014 19:49:01.0934] [2] [ERROR] The administration limit on the server was exceeded.
[10/15/2014 19:49:01.0965] [2] Ending processing initialize-ExchangeUniversalGroups
[10/15/2014 19:49:02.0059] [1] The following 1 error(s) occurred during task execution:
[10/15/2014 19:49:02.0106] [1] 0. ErrorRecord: The administrative limit for this request was exceeded.
[10/15/2014 19:49:02.0106] [1] 0. ErrorRecord: Microsoft.Exchange.Data.Directory.AdminLimitExceededException: The administrative limit for this request was exceeded. ---> System.DirectoryServices.Protocols.DirectoryOperationException: The administration limit on the server was exceeded.
   at System.DirectoryServices.Protocols.LdapConnection.ConstructResponse(Int32 messageId, LdapOperation operation, ResultAll resultType, TimeSpan requestTimeOut, Boolean exceptionOnTimeOut)
   at System.DirectoryServices.Protocols.LdapConnection.SendRequest(DirectoryRequest request, TimeSpan requestTimeout)
   at Microsoft.Exchange.Data.Directory.PooledLdapConnection.SendRequest(DirectoryRequest request, LdapOperation ldapOperation, Nullable`1 clientSideSearchTimeout, IActivityScope activityScope, String callerInfo)
   at Microsoft.Exchange.Data.Directory.ADDataSession.ExecuteModificationRequest(ADObject entry, DirectoryRequest request, ADObjectId originalId, Boolean emptyObjectSessionOnException, Boolean isSync)
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.Data.Directory.ADDataSession.AnalyzeDirectoryError(PooledLdapConnection connection, DirectoryRequest request, DirectoryException de, Int32 totalRetries, Int32 retriesOnServer)
   at Microsoft.Exchange.Data.Directory.ADDataSession.ExecuteModificationRequest(ADObject entry, DirectoryRequest request, ADObjectId originalId, Boolean emptyObjectSessionOnException, Boolean isSync)
   at Microsoft.Exchange.Data.Directory.ADDataSession.ExecuteModificationRequest(ADObject entry, DirectoryRequest request, ADObjectId originalId, Boolean emptyObjectSessionOnException)
   at Microsoft.Exchange.Data.Directory.ADDataSession.Save(ADObject instanceToSave, IEnumerable`1 properties, Boolean bypassValidation)
   at Microsoft.Exchange.Management.Tasks.InitializeExchangeUniversalGroups.CreateWKGuid(ADContainer container, ADObjectId dn, Guid wkGuid)
   at Microsoft.Exchange.Management.Tasks.InitializeExchangeUniversalGroups.CreateGroup(ADOrganizationalUnit usgContainer, String groupName, Int32 groupId, Guid wkGuid, String groupDescription, GroupTypeFlags groupType, Boolean createAsRoleGroup)
   at Microsoft.Exchange.Management.Tasks.InitializeExchangeUniversalGroups.CreateGroup(ADOrganizationalUnit usgContainer, String groupName, Int32 groupId, Guid wkGuid, String groupDescription)
   at Microsoft.Exchange.Management.Tasks.InitializeExchangeUniversalGroups.InternalProcessRecord()
   at Microsoft.Exchange.Configuration.Tasks.Task.b__b()
   at Microsoft.Exchange.Configuration.Tasks.Task.InvokeRetryableFunc(String funcName, Action func, Boolean terminatePipelineIfFailed)

What does this error mean? If you do some digging, you'll eventually find out that every attribute in Active Directory needs to fit into a 8KB ESE page. 8KB fits about 1200-1300 values in a nonlinked multivalued attribute. The question then becomes what attribute is causing our problem? We take an LDAP trace with the help of https://support.microsoft.com/kb/2221529 to determine the LDAP query causing the error and we see the following:

[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] ldap_modify called for connection 0x234e0898: DN is CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=test,DC=lab. Synchronous is 0x0.

[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] Unencrypted dump of Data sent on connection 0x234e0898
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 30 84 00 00 00 fb 02 02 10 3f 66 84 00 00 00 f1 0........?f.....
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 04 41 43 4e 3d 4d 69 63 72 6f 73 6f 66 74 20 45 .ACN=Microsoft.E
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 78 63 68 61 6e 67 65 2c 43 4e 3d 53 65 72 76 69 xchange,CN=Servi
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 63 65 73 2c 43 4e 3d 43 6f 6e 66 69 67 75 72 61 ces,CN=Configura
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 74 69 6f 6e 2c 44 43 3d 73 68 31 31 2c 44 43 3d tion,DC=test,DC=
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 6c 61 62 30 84 00 00 00 a8 30 84 00 00 00 a2 0a lab0.....0......
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 01 00 30 84 00 00 00 99 04 15 6f 74 68 65 72 57 ..0.......otherW
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 65 6c 6c 4b 6e 6f 77 6e 4f 62 6a 65 63 74 73 31 ellKnownObjects1
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 84 00 00 00 7c 04 7a 42 3a 33 32 3a 30 32 35 32 ....|.zB:32:0252
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 32 45 43 46 39 39 38 35 39 38 34 41 39 32 33 32 2ECF9985984A9232
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 30 35 36 46 43 37 30 34 43 43 38 42 3a 43 4e 3d 056FC704CC8B:CN=
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 4d 61 6e 61 67 65 64 20 41 76 61 69 6c 61 62 69 Managed.Availabi
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 6c 69 74 79 20 53 65 72 76 65 72 73 2c 4f 55 3d lity.Servers,OU=
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 4d 69 63 72 6f 73 6f 66 74 20 45 78 63 68 61 6e Microsoft.Exchan
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 67 65 20 53 65 63 75 72 69 74 79 20 47 72 6f 75 ge.Security.Grou
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 70 73 2c 44 43 3d 73 68 31 31 2c 44 43 3d 6c 61 ps,DC=test,DC=la
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] 62 b
[0]11D4.13D0::10/22/2014-12:33:06.489 [Microsoft-Windows-LDAP-Client/Debug] End of Unencrypted dump of send buffer.

[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] Unencrypted dump of Data received on connection 0x234e0898
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] 30 84 00 00 00 63 02 02 10 3f 67 84 00 00 00 59 0....c...?g....Y
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] 0a 01 0b 04 00 04 52 30 30 30 30 32 30 32 34 3a ......R00002024:
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] 20 53 76 63 45 72 72 3a 20 44 53 49 44 2d 30 32 .SvcErr:.DSID-02
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] 30 35 30 35 35 45 2c 20 70 72 6f 62 6c 65 6d 20 05055E,.problem.
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] 35 30 30 38 20 28 41 44 4d 49 4e 5f 4c 49 4d 49 5008.(ADMIN_LIMI
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] 54 5f 45 58 43 45 45 44 45 44 29 2c 20 64 61 74 T_EXCEEDED),.dat
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] 61 20 2d 31 30 32 36 0a 00 a.-1026..
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] End of Unencrypted dump of receive buffer.
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] CheckForWaiter setting MsgNo 0x103f, wait 0x234192a8
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] LdapGetResponseFromServer : searching for messages for msgid 0x103f
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] LdapGetResponseFromServer Message 0x103f marked received for connection 0x234e0bc8
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] ldap_result found results at 0x233f41b8 for request 0x233eaca8 msgid 0x103f
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] HrPeekTag ran out of data, length 0x69, offset 0x69.
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] ldap_result returned 0x67 for connection 0x234e0898.
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] HrPeekTag ran out of data, length 0x69, offset 0x69.
[0]11D4.13D0::10/22/2014-12:33:06.490 [Microsoft-Windows-LDAP-Client/Debug] HrPeekTag ran out of data, length 0x69, offset 0x69.
[2]11D4.13D0::10/22/2014-12:33:06.499 [Microsoft-Windows-LDAP-Client/Debug] ldap_delete called for connection 0x233ddb58: DN is CN=Managed Availability Servers,OU=Microsoft Exchange Security Groups,DC=test,DC=lab. Synchronous is 0x0.

Basically, setup was trying to modify otherWellKnownObjects on CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=test,DC=lab and got DSID-0205055E, problem 5008 (ADMIN_LIMIT_EXCEEDED) as a response. We consult an AD engineer and he tells us that when an attribute is "full," you won't be able to modify the object at all. We immediately check otherWellKnownObjects and see it only has 17 values; that can't be our issue. We leverage the script from https://www.open-a-socket.com/index.php/2014/09/19/multivalued-attribute-limits-in-active-directory/ and realize that several attributes, including globalAddressList and globalAddressList2 have over 1200 values.

Now for a brief history lesson. Exchange hosters create a lot of Global Address Lists - usually 1 for each tenant it hosts. globalAddressList is a nonlinked multivalued attribute that enumerates all the GALs in an Exchange organization. Exchange 2007 hit this 8KB limit and thus a tool named makeGalLinked.exe was made that converted the attribute from nonlinked to linked. For an explanation as to why linked attributes aren't subject to the 8KB limit, look under the Replication of Linked and Nonlinked Attributes in this TechNet article How the Active Directory Replication Model Works. Starting with Exchange 2010, our schema created a new value named globalAddressList2 which is a linked multivalued attribute and thus is not subject to the 8KB limit. In Exchange 2010 onwards, any new GAL will be stored into both attributes for legacy interop but once the limit is exceeded, a warning will appear informing administrators that any user who isn't on Exchange 2010 might have issues accessing their GAL (because the pointer to the GAL might not exist).

With that history in mind, think about why Exchange 2013 is failing. globalAddressList is hitting the limit because it is a full nonlinked attribute and thus causes AD to throw the error ADMIN_LIMIT_EXCEEDED. The resolution in both support cases was to remove some values from the globalAddressList and try /prepareAD again. There should be no data loss because globalAddressList2 is a superset of globalAddressList (it contains all the data plus more). All Exchange 2010 and future code should use globalAddressList2 and thus globalAddressList is a legacy attribute that eventually will be deprecated if it hasn't already been.

Cheers,

Matt

ROP Breakdown by User

Matthew Huynh [MSFT] — Thu, 25 Sep 2014 11:09:00 GMT

Hey all,

I'm back with some more info to empower your Exchange administration! Once you've enabled ROP logging on your servers, there's a lot of things you can do with the data because the operation-specific field is now populated. Let's say a particular user is pushing a lot of requests against the server and you want to know what s/he is doing. One way to determine their intent is by analyzing what ROPs they are sending against the server. For example, if someone is sending a lot of FastTransferSourceGetBuffer ROPs, they are attempting to pull data (download) from the server. If they are constantly sending a lot of FastTransferSourcePutBuffer, they are attempting to push data (upload) to the server. You can read more about Fast Transfer in the official specifications documentation or watch Joe Warren's presentation on it.

Since multiple ROPs can be packaged into a single request, we need to extract the ROPs out so we can do analysis on them. I've written a PowerShell script which I'll attach to this blog post that takes a directory path and parses every .log file inside it, producing a corresponding "expanded" .csv file that has 2 columns: client-name and ROP.

Running the script is easy. You invoke it in an administrative PowerShell window with a command like: .\RopBreakdownByUser.ps1 "d:\myRcaLogs"

Once the CSV files are produced, you can use Log Parser Studio or Excel to analyze the data. Let's say I choose to use LPS to get the top ROPs by user. I launch LPS, set my log directory to be the csv files the PS script produced earlier, set my Log Type to CSVLOG, input my query, and execute. Here's what I end up with:

Now again, ROPs are usually the building blocks to bigger operations and it is not immediately obvious what this user is doing. Experience and knowledge is the only way around this ambiguity. Here, we can at least infer the top user is doing a lot of downloading against the server. Perhaps his/her Outlook is simply doing an initial sync of their profile and thus this high activity is normal. Sometimes when a user is truly an outlier (ROP count is at least 3x the next user), their Outlook is probably misbehaving and trying to sync a corrupted item. Hands-on investigation is required and a trip to the user's desktop will usually reveal the issue like an item stuck in Outbox constantly attempting to sync or a single recurring meeting with hundreds of exceptions, etc.

Hope this helps,

Matt

RopBreakdownByUser.ps1

Enable ROP logging in Exchange 2010 and 2013

Matthew Huynh [MSFT] — Mon, 22 Sep 2014 08:31:00 GMT

This is just a short post because it seems like there's not yet one on how to do it. https://www.expta.com/2012/07/how-to-enable-logging-for-rpc-client.html shows us how to enable throttling logging - and enabling ROP logging follows the same procedure. It is an additional logging tag that must be enabled inside the Microsoft.Exchange.RpcClientAccess.Service.exe.config file. Usually when I enable ROP logging, I enable throttling logging as well thus my config file would have this LoggingTag key:

Now when I open a log file, I'd see lines like this [I split this request using a comma delimiter so it's more legible]:

date-time,session-id,seq-number,client-name,organization-info,client-software,client-software-version,client-mode,client-ip,client-connection-info,server-ip,protocol,application-id,request-ids,session-cookie,operation,rpc-status,processing-time,operation-specific,failures,performance-data,activity-context-data,user-email

2014-09-18T15:32:28.583Z
83723
690
/o=Pulsarwinds/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=MAHUYNH
OUTLOOK.EXE
14.0.7108.5000
Cached
ncacn_http
Client=MSExchangeRPC
0
00:00:00.0150000
">SetProperties; >SetProperties; >SaveChangesMessage; >GetPropertiesSpecific; Conn:7
MaxConn:40
MaxBurst:150000
Balance:149980.5
Cutoff:Unlimited
RechargeRate:900000
Policy:GlobalThrottlingPolicy_09d51525-29c1-44b3-babc-4daaa4add881
IsServiceAccount:False
LiveTime:04:44:32.0860602"
mahuynh@pulsarwinds.com

Remote operation (ROP) logging allows you to see the ROPs that are sent from clients to the server and back. When you see a ROP that is prepended with '>', then it is being sent from the client. When you see a ROP that is prepended with '<', then it is being sent from the server. We can see that 4 ROPs were packed into this single request here.

RCA logs have a wealth of information. The client is connecting via Outlook 2010 cached mode using RPC over HTTP (Outlook Anywhere). We can also tell that he has 7 connections out of a maximum 40 used. His budget/balance is 149980.5ms so he's used up 19.5ms of server time globally across his connections. If the balance goes negative, the user will start encountering microdelays. Since there's no cutoff (it's unlimited), the user will never be denied/rejected outright via the RPC protocol.

Using the operation-specific field, you can extract the ROPs out and identify what a user's top ROPs are. My next blog post will have a script that does that and some info on how to generate basic stats on the extracted data.

Edit: Here's the promised script: https://blogs.technet.com/b/mahuynh/archive/2014/09/25/rop-breakdown-by-user.aspx