John Bailey (EXCHANGE)

Netlogon: Cross-Forest Delayed Authentication Requests Cause Subsequent (and Continuous) Authentication Failures

John Bailey [MSFT] — Fri, 29 Dec 2017 11:22:23 GMT

One of the longest debugging experiences I've ever had to debug, so far, in Exchange was a code bug that exists in the Netlogon code. I hope to cover what this bug was, how it manifested, and the fix that was implemented by the Windows developer to resolve the issue. So, this is going to be a long one.... Picture (Worth 1,000 Words) Basics Netlogon sessions use RPC (remote-procedure call) sessions with domain controllers to communicate authentication requests to domains. In the cases of cross-forest authentication requests, regardless of the type of trust created (e.g.: one-way transitive, one-way intransitive, etc.), the cross-forest authentication requests are forwarded (via the trust) to the responsible domain. In this case, the responsible domains exist in the customer's on-premises environments. So, in the above example, you'll see that the client communicates with the Café server. During authentication, the Café passes the request to the managed domain controller. The managed domain controller will see the trust and communicate the authentication request across the trust and receive only an NT status response back for the request from the customer's domain controllers. When Repro Cometh The condition that causes repro to start occurring is when the local domain controller (in this case, the managed domain controller in the illustration above) is awaiting a response from the customer's domain controller for an authentication request. In the authentication pipeline, if this request times-out, it's considered a re-triable exception - which is important for later. Because this exception is retriable, a the Café server doesn't consider that the authentication request has failed. Also, during this same time, the Café server may build a new Netlogon session with a new domain controller, which is where our problem begins to surface. The Netlogon code has a single object reference for the domain controller's name for the current Netlogon session on the current RPC session it should be using. (If you're familiar with native/unmanaged code, the reference to the domain controller's name is a pointer to a wchar_t value.) But remember: We've not disposed of the previous session because it's considered re-triable. So, since Netlogon can only communicate on one session per one RPC channel, we now have two Netlogon sessions with two RPC channels. The non-disposed of session is in red and the new session is in green in the illustration above. The Bug The bug is that all subsequent authentication requests traverse the red authentication path but use the domain controller's name that was obtained from the creation of the green authentication path (the domain controller's name is supplied in the authentication request as is defined in the specifications). This causes all subsequent authentication requests to fail, no matter the destination forest, because the domain controller receives a request that it should not process. Verifying Repro The best way to verify the repro of this bug is to look at the Netlogon logs. If you see 0xc0000122 (STATUS_INVALID_COMPUTER_NAME), then you've hit repro of this specific condition. In Exchange, this will bubble-up via the app pool in IIS as a 401 Unauthorised (which makes chasing the bug a bit more complicated). The Fix Windows dev determined that the best way to fix this was to tear down both the Netlogon and RPC sessions, regardless of current status. This has been verified as working in RS3 builds of Windows 10/Server 2016 and is currently being tested in RS1 builds of Windows 10/Server 2016.

Windows 10: Removing Groove, Facebook, and Other Unwanted Apps

John Bailey [MSFT] — Wed, 27 Dec 2017 11:26:27 GMT

A few months ago, someone very dear to me asked why they couldn't remove unwanted apps from Windows 10 - mainly over privacy and right to decline concerns. I took it up as a sort of personal challenge to find a way to rid these apps for other people, as well, whilst I was attempting to figure out how to do this for them specifically. So, I give you the result - which is still a work in progress.


function Remove-UnwantedApps
{
# Attempt to remove Facebook (if it exists)
$facebookPkg = Get-AppxPackage -AllUsers | Where{$_.Name -like '*Facebook*'}
if($facebookPkg -ne $null)
{
Remove-AppxPackage $facebookPkg.PackageFullName -Confirm:$FALSE
}

# Attempt to remove Groove Music
$grooveMusicPkg = Get-AppxPackage -AllUsers | Where{$_.PackageFullName -ilike '*ZuneMusic*'}
if($grooveMusicPkg -ne $null)
{
Remove-AppxPackage $grooveMusicPkg.PackageFullName -Confirm:$FALSE
}

# Attempt to remove Groove Video
$grooveVideoPkg = Get-AppxPackage -AllUsers | Where{$_.Name -like '*ZuneVideo*'}
if($grooveVideoPkg -ne $null)
{
Remove-AppxPackage $grooveVideoPkg.PackageFullName -Confirm:$FALSE
}

# Attempt to remove all things XBox.
$xboxPkgs = Get-AppxPackage -AllUsers | Where{$_.Name -like '*Xbox*'}
foreach($xboxPkg in $xboxPkgs)
{
Remove-AppxPackage $xboxPkg.PackageFullName -Confirm:$FALSE
}

# Attempt to remove the Advertising apps
$advertPkgs = Get-AppxPackage -AllUsers | Where{$_.Name -like '*Microsoft.Advertising*'}
foreach($advertPkg in $advertPkgs)
{
Remove-AppxPackage $advertPkg.PackageFullName -Confirm:$FALSE
}

# Attempt to remove the Bing apps
$bingPkgs = Get-AppxPackage -AllUsers | Where{$_.Name -like '*Microsoft.Bing*'}
foreach($bingPkg in $bingPkgs)
{
Remove-AppxPackage $bingPkg.PackageFullName -Confirm:$FALSE
}

# Attempt to remove the Messaging package
$msgPkg = Get-AppxPackage -AllUsers | Where{$_.Name -like '*Microsoft.Messaging*'}
Remove-AppxPackage $msgPkg.PackageFullName -Confirm:$FALSE

# Attempt to remove Windows Store dependent packages
$storeDepPkgs = Get-AppxPackage -AllUsers | Where{$_.Dependencies -ilike '*Store*' -and $_.Name -inotlike '*WindowsStore*'}
foreach($sdPkg in $storeDepPkgs)
{
Remove-AppxPackage $sdPkg.PackageFullName -Confirm:$FALSE
}

# Attempt to remove Windows Store package
$storePkg = Get-AppxPackage -AllUsers | Where{$_.Name -ilike '*WindowsStore*'}
if($storePkg -ne $null)
{
Remove-AppxPackage $storePkg.PackageFullName -Confirm:$FALSE
}

# Attempt to disable Cortana via the Registry
$searchRegistyPath = Test-Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\Windows Search"
if($searchRegistyPath -eq $FALSE)
{
# Create the Windows Search Registry Key
New-Item -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows" -Name "Windows Search"

# Create the AllowCortana Registry Key
New-Item -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\Windows Search" -Name "AllowCortana"

# Disable Cortana via the path
Set-Item -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\Windows Search\AllowCortana" -Value 0
}
else
{
# Let's ensure the child item exists.
$allowCortanaRegistryPath = Test-Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\Windows Search\AllowCortana"
if($allowCortanaRegistryPath -eq $FALSE)
{
# Create the AllowCortana Registry Key
New-Item -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\Windows Search" -Name "AllowCortana"
}

# Disable Cortana via the path
Set-Item -Path "HKLM:\SOFTWARE\Policies\Microsoft\Windows\Windows Search\AllowCortana" -Value 0
}

# Stop the Windows Search Service
$searchService = Get-Service WSearch
Stop-Service -InputObject $searchService -Force

# Attempt to remove Cortana via the App Package
$cortanaPkg = Get-AppxPackage -AllUsers | Where{$_.PackageFullName -ilike '*Cort*'}
foreach($cPkg in $cortanaPkg)
{
Remove-AppxPackage $cPkg.PackageFullName -Confirm:$FALSE
}

# Restart the system, so the registy changes take effect in the Windows Search Service's OnStart() method body.
}

Please note that this has only been tested on a local machine and, so far, validates. This can produce undesired results, such as not being able to install packages from the Windows Store in the future, since the Windows Store is removed via this script. YOU HAVE BEEN WARNED. To use: Copy the red text into an Administrative session in PowerShell (right-click the PowerShell icon --> Run as Administrator). Run 'Remove-UnwantedApps' at the new line, after the red code lines from above have been pasted into the PowerShell session. Restart your machine. Profit.

O365 and Exchange 2016/Exchange 2013: Understanding the UserPhoto API

John Bailey [MSFT] — Fri, 16 Jun 2017 14:20:00 GMT

function Get-UserPhotoDataOnPremises { param([string]$User) Write-Warning -Message "The source of our powers comes from a Cracker Jack™ box, so this will take a minute. Please be patient..." if([System.String]::IsNullOrEmpty($User)) { throw [System.NullReferenceException]::new("The user value cannot be null.") } else { $mbx = Get-Mailbox $User } [string]$scriptPath = $env:ExchangeInstallPath + "Scripts\ManagedStoreDiagnosticFunctions.ps1" # Dot-load script into function . $scriptPath [GUID]$guid = $mbx.ExchangeGuid.Guid [int]$mbxNumber = Get-StoreQuery -Database $mbx.Database -Query "SELECT MailboxNumber FROM Mailbox WHERE MailboxGuid='$guid'" | Select -ExpandProperty MailboxNumber [string]$folderId = Get-StoreQuery -Database $mbx.Database -Query "SELECT FolderId FROM Folder WHERE MailboxNumber='$mbxNumber' AND DisplayName='$([System.String]::Empty)'" | Select -ExpandProperty FolderId $global:item = Get-StoreQuery -Database $mbx.Database -Query "SELECT * FROM Message WHERE MailboxNumber='$mbxNumber' AND FolderId='$folderId' AND MessageClass='IPM.UserPhoto'" -Unlimited [string]$previewPhotoCachedId = (Get-StoreQuery -Database $mbx.Database -Query "SELECT UserPhotoPreviewCacheId FROM Mailbox WHERE MailboxNumber='$mbxNumber'" -Unlimited).p7C1B0003 [string]$photoCacheId = (Get-StoreQuery -Database $mbx.Database -Query "SELECT UserPhotoCacheId FROM Mailbox WHERE MailboxNumber='$mbxNumber'" -Unlimited).p7C1A0003 # Obtain files on disk (if any) $smtp = $mbx.WindowsEmailAddress.Address $smtpAtIndex = $smtp.IndexOf("@") $smtpAtIndexPlusOne = $smtpAtIndex + 1 $smtpDotIndex = $smtp.LastIndexOf(".") $smtpNewLength = $smtpDotIndex - $smtpAtIndexPlusOne $subString = $smtp.Substring($smtpAtIndexPlusOne, $smtpNewLength) $queryString = "_$subString" $preSubString = $smtp.Substring(0, $smtpAtIndex) $srvr = Get-MailboxDatabaseCopyStatus $mbx.Database.Name | Where{$_.Status -contains 'Mounted'} | Select -ExpandProperty MailboxServer $folderPathUnc = "\\$($srvr)\" + $env:ExchangeInstallPath.Replace(":", "$") + "\ClientAccess\photos" $obj = Get-ChildItem -Path $folderPathUnc -Filter "*$($queryString)*" $folderPath = $obj.FullName $obj2 = Get-ChildItem -Path $folderPath $fullPaths = @() foreach($o in $obj2) { $picObj = Get-ChildItem -Path $o.FullName -Filter "*$($preSubString)*" $fullPaths += New-Object PSobject -Property @{ Name=$picObj.Name Size=$picObj.Length Location=$picObj.FullName } } if([System.String]::IsNullOrEmpty($item.MessageId) -eq $FALSE) { $string = "IPM.UserPhoto item found in user's mailbox. The object can be found in " + "$" + "item" Write-Host $string } if($previewPhotoCachedId) { Write-Host "UserPhotoPreviewCacheId found: $previewPhotoCachedId" } if($photoCacheId) { Write-Host "UserPhotoCacheId found: $photoCacheId" } Write-Host -ForegroundColor Green "Photo files found on disk on $($srvr) for $($smtp):" $fullPaths | FL } Here's an example as run from my lab:


[PS] E:\>Get-UserPhotoDataOnPremises -User Administrator
WARNING: The source of our powers comes from a Cracker Jack™ box, so this will take a minute. Please be patient...
IPM.UserPhoto item found in user's mailbox. The object can be found in $item
UserPhotoPreviewCacheId found: -88512737
UserPhotoCacheId found: -88512737
Photo files found on disk on [REDACTED] for Administrator@contoso.se:


Name     : _Administrator-8EEDD78A2D804372C17E9FABD151BCD5.jpg
Location : \\[REDACTED]\E$\exchsrvr\ClientAccess\photos\_contoso.se-BBED6250E228D4F38F5F19BF4F1A6823\HR648x648\_Administrator-8EEDD78A2D804372C17E9FABD151BCD5.jpg
Size     : 79838

Name     : _Administrator-8EEDD78A2D804372C17E9FABD151BCD5.jpg
Location : \\[REDACTED]\E$\exchsrvr\ClientAccess\photos\_contoso.se-BBED6250E228D4F38F5F19BF4F1A6823\HR96x96\_Administrator-8EEDD78A2D804372C17E9FABD151BCD5.jpg
Size     : 3224

I've also written a method in C# to test obtain the photo via the EWS GetUserPhoto REST method:


/// 
///     Creates an EWS request for a user's photo at each standard size.
///



2017 Project: Documented Code Samples
John Bailey [MSFT] — Sun, 26 Mar 2017 03:46:49 GMT
In an effort to empower more people across the planet to learn, but for it not to be an arduous journey in doing so, I've started heavily commenting code so that people can learn from it. This will be the first of many code ventures that I attempt to do and then share with comments, so that others can learn from it. The reason for the commenting with the links is because I, myself, have often wondered what 'x' does or why 'y' may have been used; so, in typical learning fashion, I set off to my favourite search engine and undertake to read more on the class, method, property, etc. To save someone the time and heartache of doing that, I include the .NET MSDN/TechNet documentation with the code - to assist self-driven learning.

In this first of series, DownloadBingImage, Wes (Mills) was making a PowerShell script to download the Bing Search Image. Kevin (Miller) asked him if it was in a service or task and Wes replied that it was a task, so I challenged myself to write it into a Windows Service. I rose to the occasion - mostly, out of self-interest in Windows Services, as it would help me to learn more about coding fundamentals in this area.

And, so, a few months of trial and error and I have a working version of the service installed on my local machine and it downloads the image[s]. This same exact code is what I'm sharing with you.

You can find a copy of the code with in-line comments and links to the classes/methods here. If you want to start at the main service entry point, look here.

It should go without saying that I am not a professional developer (albeit, my role does require understanding all aspects of the SDLC), so if you have comments or are confused as to why I did something a particular way (from a professional developer's perspective) and it doesn't quite make sense, that's why: I'm more of a hobbyist than what others would call a "professional", at this point.

I hope that this helps at least one person on their journey towards software development. If it's just one, then the effort was worth it. :)

Happy Coding!


EXODSnapIn: Alpha Release
John Bailey [MSFT] — Thu, 05 May 2016 00:19:27 GMT
I wanted to write a post about the project I've been working on since circa December 2015. It's approximately 1200 lines of code and 18 commands - all varying in intended design/function. The assembly is meant to be imported into your Exchange Management Shell (EMS) via importing the module: Import-Module .\EXODSnapIn.dll -Verbose The intended audience is any Exchange Administrator with access rights to 'Update-MailboxDatabaseCopy' at the most/highest. 

It should be important to note that some of these commands will work outside of EMS (e.g.: Get-WhoIsInformation) and do not, implicitly, require importation into EMS. 

To add the assembly as a default into PowerShell, browse to C:\Users\your user name\Documents\WindowsPowerShell and edit (or create it if it does not exist) the Microsoft.PowerShell_profile.ps1 file to include Import-Module .\EXODSnapIn.dll -Verbose.

Get-AdObjectProperties
This command uses the System.DirectoryServices.Protocols library to search the current domain context for an object by both the property and property value supplied. In order to prevent overhead, I only return the processed collection of attributes for the first object returned.

$obj = Get-AdObjectProperties -Property mail -PropertyValue johnbai@exchange.microsoft.com
$obj.whenCreated
12/20/2011 18:54:22


Get-InternetExchangeStatus
This command will present a new browser (or tab) showing the network graph of the Internet Exchange site specified (via switch parameter). Unfortunately, not all IX have their network status listed publicly; so, if you're looking for the status for San Antonio, for example, you will not be able to find it with this command.

Get-InternetExchangeStatus -Amsterdam -Dublin -Frankfurt -Hamburg -London -Madrid -Munich


Get-WhoisInformation
This command takes an IPAddress (or array of them) and queries ARIN for the owner. If the owner is found to be RIPE, the command makes a second query to RIPE for the data before displaying it. (APNIC and LACNIC will be coming in the future, assuming they make REST endpoints available.)

Get-WhoisInformation -IpAddresses "158.198.48.206", "4.4.2.2", "88.66.77.11", "1.1.1.1", "23.103.0.0", "80.0.0.0", "157.100.0.0", "8.8.8.8" | FT -AutoSize

Name                   OriginAs StartAddress EndAddress
----                   -------- ------------ ----------
APNIC-ERX-158-198-0-0           158.198.0.0  158.215.255.255
LVLT-STATIC-4-4-16              4.4.0.0      4.4.255.255
ARCOR-DSL-NET15                 88.64.208.0  88.66.79.255
APNIC-1                         1.0.0.0      1.255.255.255
MSFT                   AS8075   23.96.0.0    23.103.255.255
UK-NTLI-20010425                80.0.0.0     80.7.255.255
LACNIC-ERX-157-100-0-0          157.100.0.0  157.100.255.255
LVLT-GOGL-8-8-8                 8.8.8.0      8.8.8.255


Repair-MailboxSecurityDescriptor
This command queries the mailbox security descriptor for a specific user specified, removes the user's ACE, and re-applies all permissions on the object (without the removed ACE).

Repair-MailboxSecurityDescriptor -Mailbox someone@contoso.com -User "Contoso\Someone2"
WARNING: The ACE for Contoso\Someone2 on someone@contoso.com was not found. Please validate your input[s].


Get-ExchangeProcessDiagnostics
This command parses the XML from the original diagnostics command to only give you the process-related information you may be concerned with, when troubleshooting an issue related to the process (e.g.: working set, mem usage, lifetime, etc.).

Get-ExchangeProcessDiagnostics -ProcessName MSExchangeMailboxAssistants -Server $Server | FT -AutoSize

ProcessName                 LifeTime            ThreadCount HandleCount WorkingSet
-----------                 --------            ----------- ----------- ----------
MSExchangeMailboxAssistants 23.03:50:14.4678669         167        4315 570.8 MB (598,523,904 bytes)


Get-DeletionEventsForMailbox
This command queries for all of the Database Events related to deletions performed on the user's mailbox.

Get-DeletionEventsForMailbox -SmtpAddress someone@contoso.com | FT -AutoSize
WARNING: This command will take some time to complete. Please be patient.

  Counter CreateTime            ItemType     EventName     ObjectClass                                MailboxGuid                          ClientCategory       UserType ParentEntryId
  ------- ----------            --------     ---------     -----------                                -----------                          --------------       -------- -------------
 99999999 4/28/2016 12:50:27 PM MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100000001 4/28/2016 12:50:27 PM MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100000003 4/28/2016 12:50:27 PM MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100001179 4/28/2016 1:01:31 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 Transport                     00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001BD97D0000
100001180 4/28/2016 1:01:31 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 Transport                     00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100001184 4/28/2016 1:01:34 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100002827 4/28/2016 1:14:04 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100003580 4/28/2016 1:20:23 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 Transport                     00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001BD97D0000
100003581 4/28/2016 1:20:23 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 Transport                     00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100003585 4/28/2016 1:20:24 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100003590 4/28/2016 1:20:24 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100003780 4/28/2016 1:22:37 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100004093 4/28/2016 1:25:30 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 Transport                     00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001BD97D0000
100004094 4/28/2016 1:25:30 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 Transport                     00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100004126 4/28/2016 1:25:48 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100006291 4/28/2016 1:40:06 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100006589 4/28/2016 1:43:47 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100007329 4/28/2016 1:54:20 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 Transport                     00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001BD97D0000
100007330 4/28/2016 1:54:20 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 Transport                     00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100007338 4/28/2016 1:54:22 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100008983 4/28/2016 2:05:57 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 Transport                     00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001BD97D0000
100008984 4/28/2016 2:05:57 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 Transport                     00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100008997 4/28/2016 2:05:58 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100010278 4/28/2016 2:13:29 PM  MAPI_MESSAGE ObjectDeleted IPM.Schedule.Meeting.Request               5081e2de-4bff-422a-9601-eff4bd04fab3 EventBasedAssistants          00000000DA58A48EAD04D14E83249194E7B36105010022F02B7771143E4B8B48ABD43107FC9300000AF544800000
100010279 4/28/2016 2:13:29 PM  MAPI_MESSAGE ObjectDeleted IPM.Schedule.Meeting.Request               5081e2de-4bff-422a-9601-eff4bd04fab3 EventBasedAssistants          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100010424 4/28/2016 2:14:04 PM  MAPI_MESSAGE ObjectDeleted IPM.Appointment                            5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001CD03C0000
100010643 4/28/2016 2:16:15 PM  MAPI_MESSAGE ObjectDeleted IPM.Appointment                            5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010022F02B7771143E4B8B48ABD43107FC9300000AF544800000
100010644 4/28/2016 2:16:15 PM  MAPI_MESSAGE ObjectDeleted IPM.Appointment                            5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100010662 4/28/2016 2:16:16 PM  MAPI_MESSAGE ObjectDeleted IPM.Schedule.Meeting.Canceled              5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010022F02B7771143E4B8B48ABD43107FC9300000AF544800000
100010663 4/28/2016 2:16:16 PM  MAPI_MESSAGE ObjectDeleted IPM.Schedule.Meeting.Canceled              5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100010745 4/28/2016 2:16:37 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100010806 4/28/2016 2:16:56 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100015172 4/28/2016 2:43:27 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 Transport                     00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001BD97D0000
100015173 4/28/2016 2:43:27 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 Transport                     00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100015177 4/28/2016 2:43:29 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100015286 4/28/2016 2:44:14 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100022263 4/28/2016 3:13:05 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100023147 4/28/2016 3:17:52 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100023157 4/28/2016 3:17:53 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 Transport                     00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001BD97D0000
100023158 4/28/2016 3:17:53 PM  MAPI_MESSAGE ObjectDeleted IPM.Note                                   5081e2de-4bff-422a-9601-eff4bd04fab3 Transport                     00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001C60AF0000
100026697 4/28/2016 3:32:15 PM  MAPI_MESSAGE ObjectDeleted IPM.Appointment.MP                         5081e2de-4bff-422a-9601-eff4bd04fab3 MOMT                          00000000DA58A48EAD04D14E83249194E7B36105010025A7CA50FD477849BFCCB20CBA662EF20000001CD03C0000
100037710 4/28/2016 4:02:45 PM  MAPI_MESSAGE ObjectDeleted IPM.Contact                                5081e2de-4bff-422a-9601-eff4bd04fab3 TimeBasedAssistants           00000000DA58A48EAD04D14E83249194E7B36105010022F02B7771143E4B8B48ABD43107FC930000000060550000
100037711 4/28/2016 4:02:45 PM  MAPI_MESSAGE ObjectDeleted IPM.Contact                                5081e2de-4bff-422a-9601-eff4bd04fab3 TimeBasedAssistants           00000000DA58A48EAD04D14E83249194E7B36105010022F02B7771143E4B8B48ABD43107FC93000078A453560000
100037712 4/28/2016 4:02:45 PM  MAPI_MESSAGE ObjectDeleted IPM.Contact                                5081e2de-4bff-422a-9601-eff4bd04fab3 TimeBasedAssistants           00000000DA58A48EAD04D14E83249194E7B36105010022F02B7771143E4B8B48ABD43107FC93000078A457400000


Get-ImpactedUserCount
This command gets the total number of mailboxes on the database, as is listed in AD.

$dbs = Get-MailboxDatabase | Select -ExpandProperty Name
Get-ImpactedUserCount -Database $dbs
The total mailboxes on Database1 are 21
The total mailboxes on Database2 are 162
The total mailboxes on Database3 are 41
The total mailboxes on Database4 are 43
The total mailboxes are 267


Get-MailboxQuotaStatus
This command provides the current quota statistics for the mailbox in question.

Get-MailboxQuotaStatus -Identity someoneone@contoso.com | FT -AutoSize

ExchangeGuid                         Quarantined UseDbDefaults TotItemSize                     MsgTblTotalSize                TotDelItemSize
------------                         ----------- ------------- -----------                     ---------------                --------------
594274b1-9e6f-45aa-b912-9ddb3797b57c       False         False 19.04 GB (20,447,468,845 bytes) 7.134 GB (7,659,716,608 bytes) 2.269 GB (2,435,837,443 bytes)


Get-MailboxQuotaStatus -Identity someoneone@contoso.com -Archive | FT -AutoSize

ExchangeGuid                         Quarantined UseDbDefaults TotItemSize                     MsgTblTotalSize                 TotDelItemSize
------------                         ----------- ------------- -----------                     ---------------                 --------------
594274b1-9e6f-45aa-b912-9ddb3797b57c       False         False 60.81 GB (65,289,171,556 bytes) 15.13 GB (16,247,652,352 bytes) 532.7 KB (545,501 bytes)


Get-TotalItemCount
This command returns the total number of items contained within the MAPI message table.

Get-TotalItemCount -PrimarySmtp someone@contoso.com
250194
Get-TotalItemCount -PrimarySmtp someone@contoso.com -IncludeArchive
1249282


Get-TransportSendMomtEvents
This command looks for the MOMT events from a client (e.g.: Outlook) submission to StoreDriver.

Get-TransportSendMomtEvents -SenderSmtpAddress someone@contoso.com -MessageId "<4810ce213bfc4cd7af8b9421e7266e7f@server.contoso.com>" | FT -AutoSize
WARNING: This command will take some time to complete. Please be patient.

  Counter ItemType     EventName     ObjectClass MailboxGuid                          ClientCategory ParentEntryId
  ------- --------     ---------     ----------- -----------                          -------------- -------------
172362630 MAPI_MESSAGE ObjectCreated IPM.Note    594274b1-9e6f-45aa-b912-9ddb3797b57c MOMT           000000004435FCF3C07DBE4F974F9D7A807901D00100BDCAB4A4117C7547B38E0314FA2AF58900000000000D0000
172362631 MAPI_MESSAGE ObjectCreated IPM.Note    594274b1-9e6f-45aa-b912-9ddb3797b57c MOMT           000000004435FCF3C07DBE4F974F9D7A807901D00100BDCAB4A4117C7547B38E0314FA2AF589000000000BE80000
172362632 MAPI_MESSAGE ObjectCreated IPM.Note    594274b1-9e6f-45aa-b912-9ddb3797b57c MOMT           000000004435FCF3C07DBE4F974F9D7A807901D00100BDCAB4A4117C7547B38E0314FA2AF58900009B0A1F4A0000
172362636 MAPI_MESSAGE MailSubmitted IPM.Note    594274b1-9e6f-45aa-b912-9ddb3797b57c MOMT           000000004435FCF3C07DBE4F974F9D7A807901D00100BDCAB4A4117C7547B38E0314FA2AF58900000000000D0000
172362645 MAPI_MESSAGE ObjectDeleted IPM.Note    594274b1-9e6f-45aa-b912-9ddb3797b57c Transport      000000004435FCF3C07DBE4F974F9D7A807901D00100BDCAB4A4117C7547B38E0314FA2AF58900000000000D0000
172362646 MAPI_MESSAGE ObjectDeleted IPM.Note    594274b1-9e6f-45aa-b912-9ddb3797b57c Transport      000000004435FCF3C07DBE4F974F9D7A807901D00100BDCAB4A4117C7547B38E0314FA2AF589000000000BE80000
172362647 MAPI_MESSAGE ObjectDeleted IPM.Note    594274b1-9e6f-45aa-b912-9ddb3797b57c Transport      000000004435FCF3C07DBE4F974F9D7A807901D00100BDCAB4A4117C7547B38E0314FA2AF58900009B0A1F4A0000


Get-ActiveUserCount
This command checks the perf counters of a specified server for both RPC and OWA user counters.

Get-ActiveUserCount -Server SERVER

ComputerName   	ListSet                    Source               Value
------------   	-------                    ------               -----
SERVER 		MSExchange RpcClientAccess Active User Count        2
SERVER 		MSExchange OWA             Current Unique Users     1


Get-FailedHeathSetItems
This command obtains only the failed HealthSet entries from a query against the server.

Get-FailedHeathSetItems -Server $Server -HealthSet OWA | FT -AutoSize

Server       State         Name                          TargetResource HealthSetName AlertValue ServerComponent
------       -----         ----                          -------------- ------------- ---------- ---------------
SERVER 	     NotApplicable MaintenanceFailureMonitor.OWA                OWA           Unhealthy  None


Get-UserFromSid
This command converts any SID you provide it to a NT Account (e.g.: Forest\user), as long as the SID is discover-able in AD.

Get-UserFromSid -Sid S-1-5-21-2127521184-1604012920-1887927527-8406844
Domain\User
Get-UserFromSid -Sid S-1-5-21-2014436394-2366853861-4082801726-2365144
Domain\User2


Restart-ServiceOnAllNodes
This command will restart a service you specify across all nodes in a DAG (Database Availability Group).

Restart-ServiceOnAllNodes -Dag DAG1 -Service MSExchangeMailboxAssistants
WARNING: The MSExchangeMailboxAssistants service on SERVER1 is stopping.
WARNING: The MSExchangeMailboxAssistants service on SERVER1 is starting.
WARNING: The MSExchangeMailboxAssistants service on SERVER1 is starting.
WARNING: The MSExchangeMailboxAssistants service on SERVER1 is starting.
WARNING: The MSExchangeMailboxAssistants service on SERVER1 is starting.
WARNING: The MSExchangeMailboxAssistants service on SERVER1 is starting.
WARNING: The MSExchangeMailboxAssistants service on SERVER1 is starting.
WARNING: The MSExchangeMailboxAssistants service on SERVER2 is starting.
WARNING: The MSExchangeMailboxAssistants service on SERVER2 is starting.


Start-DatabaseReseed
This command attempts to reseed a database from the mounted copy; which is the preferred (recommended) way to reseed a database, so that a fail-over copy is available for HA (assuming more than two copies exist).

Start-DatabaseReseed -DatabaseCopy $Copy -DeleteExistingFiles:$TRUE


Test-ClientOwaEndPoints
This command obtains the IP address entries for the name specified and attempts to perform an HTTP request against OWA.

Test-ClientOwaEndPoints mail.contoso.com | FT -AutoSize

IpAddress   OwaStatusCode
---------   -------------
38.96.29.10        Unused
38.96.29.10        Unused


Test-TcpConnectivity
This command uses the basic .NET TcpClient to make a TCP connection to the specified host on the specified port.

Test-TcpConnectivity -Endpoint mail.contoso.com -Port 25
True
Test-TcpConnectivity -Endpoint mail.contoso.com -Port 443
True
Test-TcpConnectivity -Endpoint mail.contoso.com -Port 444
Exception Caught: No connection could be made because the target machine actively refused it 10.221.254.5:444


Show-NetlogonImpact
This command shows the Netlogon counters for a specified machine.


Show-NetlogonImpact -Server SERVER -Intervals 3

ComputerName   Source             Value
------------   ------             -----
SERVER 		Semaphore Waiters      0
SERVER 		Semaphore Holders      0
SERVER 		Semaphore TimeOuts     0
SERVER 		Semaphore Acquires 48614

SERVER 		Semaphore Holders      0
SERVER 		Semaphore TimeOuts     0
SERVER 		Semaphore Waiters      0
SERVER 		Semaphore Acquires 48616

SERVER 		Semaphore Waiters      0
SERVER 		Semaphore Acquires 48616
SERVER 		Semaphore TimeOuts     0
SERVER 		Semaphore Holders      0


Download
To download the assembly, please visit the TechNet Gallery here.


C# + AD: Dump COM and Use System.DirectoryServices.Protocols with LINQ to Query AD
John Bailey [MSFT] — Mon, 25 Apr 2016 15:23:24 GMT
A while back, I posted about using ADSI COM to get the data that you wanted. I want to show you how you can do this 10 times faster, using System.DirectoryServices.Protocols and LINQ.

Class for the actual AD query being performed:

    /// 
    /// Initializes a new instance of the  class.
    /// 
    internal class LdapSearcher
    {
        /// 
        /// Initializes a new instance of the  method.
        /// 
        /// The filter to use for the LDAP query.
        /// The limit of return objects for the query.
        /// 
        /// 
        public static async Task<SearchResponse> SearchTheForestTask(string filter, int? sizeLimit, bool configContainer)
        {
            // More information on the return statement can be found here: https://msdn.microsoft.com/en-us/library/1h3swy84.aspx
            // More information on the async/await keywords can be found here: https://msdn.microsoft.com/en-us/library/hh191443.aspx
            // More information on the Task.Run() method can be found here: https://msdn.microsoft.com/en-us/library/hh160382(v=vs.110).aspx
            return await Task.Run(async /* Run on asynchronous thread so we don't block. */
                () =>
                {
                    // More information on the return statement can be found here: https://msdn.microsoft.com/en-us/library/1h3swy84.aspx
                    // More information on the async/await keywords can be found here: https://msdn.microsoft.com/en-us/library/hh191443.aspx
                    // More information on the Task.Run() method can be found here: https://msdn.microsoft.com/en-us/library/hh160382(v=vs.110).aspx
                    return await Task.Run(() => SearchTheForest(filter, sizeLimit, configContainer));
                });
        }

        /// 
        /// Initializes a new instance of the  method.
        /// 
        /// The filter to use for the LDAP query.
        /// The limit of return objects for the query.
        /// 
        /// 
        private static SearchResponse SearchTheForest(string filter, int? sizeLimit, bool configContainer)
        {
            string targetServer = Domain.GetComputerDomain().FindDomainController().Name;

            // We can't await a Task to get RootDSE via async or it may execute AFTER we need it.
            DirectoryEntry rootDse = new DirectoryEntry("LDAP://RootDSE")
            {
                // More information on the DirectoryEntry.AuthenticationType property can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.directoryentry.authenticationtype(v=vs.110).aspx
                // More information on the AuthenticationTypes enumeration can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.authenticationtypes(v=vs.110).aspx
                AuthenticationType = AuthenticationTypes.Encryption
            };

            string searchRoot = AdMethods.GetSearchRootTask(rootDse, configContainer).Result;

            // We instantiate the object to clear data.
            Ad.NewSearchResponse = null;

            // If the size limit isn't defined, we set it to 1k.
            int pageSize = sizeLimit ?? 1000;

            // We specify the page size so that we don't break all the things.
            // More information on the PageResultRequestControl class can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.pageresultrequestcontrol(v=vs.100).aspx
            PageResultRequestControl newPageResultRequestControl = new PageResultRequestControl(pageSize);

            // We include deleted objects in the search.
            // More information on the ShowDeletedControl class can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.showdeletedcontrol(v=vs.110).aspx
            ShowDeletedControl newShowDeletedControl = new ShowDeletedControl();
            
            // Establish connection to the Directory.
            LdapConnection newLdapConnection = new LdapConnection(targetServer);

            // Create the Search Request.
            // More information on the SearchRequest class be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.searchrequest(v=vs.110).aspx
            SearchRequest newSearchRequest = new SearchRequest
            {
                // More information on the SearchRequest.Filter property can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.searchrequest.filter(v=vs.110).aspx
                Filter = filter,

                // We specify the search root for the query. Don't let the documentation lie to you,
                // this property does not specify the object to query FOR.
                // More information on the SearchRequest.DistinguishedName property can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.searchrequest.distinguishedname(v=vs.110).aspx
                DistinguishedName = searchRoot,

                // More information on the SearchRequest.Scope property can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.searchrequest.scope(v=vs.110).aspx
                // More information on the SearchScope enumeration can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.searchscope(v=vs.110).aspx
                Scope = System.DirectoryServices.Protocols.SearchScope.Subtree,

                // More information on the SearchRequest.TimeLimit property can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.searchrequest.timelimit(v=vs.110).aspx
                TimeLimit = TimeSpanConstants.OneMinuteTimeSpan,

                // More information on the SearchRequest.RequestId property can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.directoryrequest.requestid(v=vs.110).aspx
                RequestId = GuidConstants.RequestGuid.ToString(),

                // More information on the SearchRequest.Aliases property can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.searchrequest.aliases(v=vs.110).aspx
                // More information on the DereferenceAlias enumeration can be found there: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.dereferencealias(v=vs.100).aspx
                Aliases = System.DirectoryServices.Protocols.DereferenceAlias.Never,

                // More information on the SearchRequest.TypesOnly property can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.searchrequest.typesonly(v=vs.110).aspx
                TypesOnly = false,

                // More information on the SearchRequest.Controls property can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.directoryrequest.controls(v=vs.110).aspx
                Controls = { newPageResultRequestControl, newShowDeletedControl }
            };
            if (sizeLimit.HasValue)
            {
                // More information on the SearchRequest.SizeLimit property can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.searchrequest.sizelimit(v=vs.110).aspx
                newSearchRequest.SizeLimit = sizeLimit.Value;
            }

            // Since we're entering a critical section, let's lock so we block further calls.
            // More information on the lock keyword can be found here: https://msdn.microsoft.com/en-us/library/c5kehkcz.aspx
            lock (ObjectConstants.NewLock)
            {
                // Enter the critical section.
                // More information on the try keyword can be found here: https://msdn.microsoft.com/en-us/library/6dekhbbc.aspx
                try
                {
                    // More information on the SearchResponse class can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.searchresponse(v=vs.110).aspx
                    // More information on the LdapConnection.SendRequest(DirectoryRequest) method can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.ldapconnection.sendrequest(v=vs.110).aspx
                    Ad.NewSearchResponse = (SearchResponse) newLdapConnection.SendRequest(newSearchRequest, TimeSpanConstants.OneMinuteTimeSpan);
                }
                catch (LdapException exception)
                {
                    // We throw the base exception so it can be unwound by the calling thread.
                    // More information on the Exception.GetBaseException() method can be found here: https://msdn.microsoft.com/en-us/library/system.exception.getbaseexception(v=vs.110).aspx
                    throw exception.GetBaseException();
                }
            }

            // Since we need to await completion, we pass a value back.
            // More information on the return statement can be found here: https://msdn.microsoft.com/en-us/library/1h3swy84.aspx
            return Ad.NewSearchResponse;
        }
    }


The real magic - taking the return and using LINQ:

           // More information on the TaskFactory class can be found here: https://msdn.microsoft.com/en-us/library/system.threading.tasks.taskfactory(v=vs.110).aspx
            TaskFactory<SearchResponse> newTaskFactory = new TaskFactory<SearchResponse>();

            // More information on the using statement can be found here: https://msdn.microsoft.com/en-us/library/yh598w02.aspx
            // More information on the Task<TResult> class can be found here: https://msdn.microsoft.com/en-us/library/dd321424(v=vs.110).aspx
            // More information on the TaskFactory.StartNew(Action) method can be found here: https://msdn.microsoft.com/en-us/library/dd321439(v=vs.110).aspx
            using (Task<SearchResponse> newTask = newTaskFactory.StartNew(() => LdapSearcher.SearchTheForestTask(filter, null, false).Result))
            {
                // More information on the TaskAwaiter<TResult> class can be found here: https://msdn.microsoft.com/en-us/library/hh138386(v=vs.110).aspx
                // More information on the Task.GetAwaiter() method can be found here: https://msdn.microsoft.com/en-us/library/system.threading.tasks.task.getawaiter(v=vs.110).aspx
                TaskAwaiter<SearchResponse> newTaskAwaiter = newTask.GetAwaiter();

                // More information on the TaskAwaiter.IsCompleted property can be found here: https://msdn.microsoft.com/en-us/library/system.runtime.compilerservices.taskawaiter.iscompleted(v=vs.110).aspx
                while (!newTaskAwaiter.IsCompleted)
                {
                    // More information on the string.Format() method can be found here: https://msdn.microsoft.com/en-us/library/system.string.format(v=vs.110).aspx
                    this.WriteDebug(string.Format("Awaiting task with id'{0}' to finish.", newTask.Id));

                    // Let's wait for task completion.
                    System.Threading.Thread.Sleep(500);
                }

                // More information on the TaskAwaiter.IsCompleted property can be found here: https://msdn.microsoft.com/en-us/library/system.runtime.compilerservices.taskawaiter.iscompleted(v=vs.110).aspx
                if (newTaskAwaiter.IsCompleted)
                {
                    // More information on the string.Format() method can be found here: https://msdn.microsoft.com/en-us/library/system.string.format(v=vs.110).aspx
                    this.WriteDebug(string.Format("Task with Id '{0}' completed.", newTask.Id));

                    // More information on the Task.IsFaulted property can be found here: https://msdn.microsoft.com/en-us/library/system.threading.tasks.task.isfaulted(v=vs.110).aspx
                    if (!newTask.IsFaulted)
                    {
                        // More information on the TaskAwaiter.GetResult() method can be found here: https://msdn.microsoft.com/en-us/library/system.runtime.compilerservices.taskawaiter.getresult(v=vs.110).aspx
                        newTaskAwaiter.GetResult();

                        // More information on the SearchResultEntryCollection class can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.searchresultentrycollection(v=vs.110).aspx
                        SearchResultEntryCollection newSearchResultEntryCollection = Ad.NewSearchResponse.Entries;

                        // More information on the SearchResultEntry class can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.searchresultentry(v=vs.110).aspx
                        SearchResultEntry newSearchResultEntry = newSearchResultEntryCollection[0];

                        // More information on the SearchResultAttributeCollection class can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.searchresultattributecollection(v=vs.110).aspx
                        SearchResultAttributeCollection newSearchResultAttributeCollection = newSearchResultEntry.Attributes;

                        // Validate we don't hit the System.NullReferenceException
                        // More information on the SearchResultAttributeCollection.Values property can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.searchresultattributecollection.values(v=vs.110).aspx
                        if (newSearchResultAttributeCollection.Values != null)
                        {
                            // To save overhead from iterating through each attribute on the object, we need to convert it into a queryable.
                            // This takes some magic but the overall result is the same data, just slightly faster.

                            // More information on the SearchResultAttributeCollection.Values property can be found here: https://msdn.microsoft.com/en-us/library/system.directoryservices.protocols.searchresultattributecollection.values(v=vs.100).aspx
                            ICollection newCollection = newSearchResultAttributeCollection.Values;

                            // More information on the Enumerable.Cast<TResult>() method can be found here: https://msdn.microsoft.com/en-us/library/bb341406(v=vs.110).aspx
                            IEnumerable newDirectoryAttributesCollection = newCollection.Cast<DirectoryAttribute>();

                            // More information on the Queryable.AsQueryable() method can be found here: https://msdn.microsoft.com/en-us/library/bb353734(v=vs.110).aspx
                            IQueryable<DirectoryAttribute> newQueryable = (IQueryable<DirectoryAttribute>) newDirectoryAttributesCollection.AsQueryable();

                            // More information on the Where() method can be found here: https://msdn.microsoft.com/en-us/library/bb546161.aspx
                            // More information on the Select() method can be found here: https://msdn.microsoft.com/en-us/library/bb546168.aspx
                            IQueryable<object[]> newQueryableObjectUno = newQueryable.Where(x => x.Name.Equals("msExchMailboxSecurityDescriptor")).Select(x => x.GetValues(typeof (byte[])));

                            // More information on the FirstOrDefault() method can be found here: https://msdn.microsoft.com/en-us/library/bb546140.aspx
                            object[] newObjectsUno = newQueryableObjectUno.FirstOrDefault();

                            // If the data is null, we went wrong somewhere but it's best to prevent System.NullReferenceException.
                            if (newObjectsUno != null)
                            {
                                byte[] newByteArrayBytes = (byte[]) newObjectsUno[0];

                                // We decode the string.
                                CommonSecurityDescriptor newCommonSecurityDescriptor = new CommonSecurityDescriptor(true, true, newByteArrayBytes, 0);
                                string attrValue = newCommonSecurityDescriptor.GetSddlForm(AccessControlSections.All);

                                this.WriteObject(attrValue);
                            }
                        }
                    }
                }
            }


While this isn't quite straight-forward, what's happening is that we're obtaining all of the properties for the object in AD, in byte format, and then performing a select against the specific property that we want to see.

Happy coding! :) 


Debugging: When Recursive NDRs Bring the FSC to It's Knees And Causes Queuing
John Bailey [MSFT] — Tue, 05 Apr 2016 20:52:49 GMT
A while back (circa July 2015), we had alerting for queuing for a Dedicated customer. When we went and looked, the messages were just sitting in the Submission queue.

We dumped the EdgeTransport process and I cracked it open. When I looked at the dump, only 4 threads were at or above 30 frames and those threads had the same stack[s] (redacted for readability):

# Child-SP          RetAddr           Call Site
00 00000000`3587d298 000007fe`ff2e74f6 ntdll!ZwAlpcSendWaitReceivePort+0xa
01 00000000`3587d2a0 000007fe`ff2e1c32 rpcrt4!LRPC_CCALL::SendReceive+0x156
02 00000000`3587d360 000007fe`fdf228c0 rpcrt4!I_RpcSendReceive+0x42
03 00000000`3587d390 000007fe`fdf2282f ole32!ThreadSendReceive+0x40
04 00000000`3587d3e0 000007fe`fdf2265b ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xa3
05 00000000`3587d480 000007fe`fddddaaa ole32!CRpcChannelBuffer::SendReceive2+0x11b
06 00000000`3587d640 000007fe`fdddda0c ole32!CAptRpcChnl::SendReceive+0x52
07 00000000`3587d710 000007fe`fdf2205d ole32!CCtxComChnl::SendReceive+0x68
08 00000000`3587d7c0 000007fe`ff38fd61 ole32!NdrExtpProxySendReceive+0x45
09 00000000`3587d7f0 000007fe`fdf1f82f rpcrt4!NdrpClientCall2+0x9ea
0a 00000000`3587df60 000007fe`fdddd8a2 ole32!ObjectStublessClient+0x1ad
0b 00000000`3587e2f0 000007fe`f839a8b7 ole32!ObjectStubless+0x42
0c 00000000`3587e340 000007ff`0042901d mscorwks!DoCLRToCOMCall+0x177
0d 00000000`3587e3f0 000007ff`00424efc DomainBoundILStubClass.IL_STUB(System.Object, UInt32, UInt32 ByRef, UInt32 ByRef, System.Array ByRef)+0x15d
0e 00000000`3587e540 000007ff`001af3ec FSEAgent_8_0_685!Antigen.Scanner.Scan()+0x45c
0f 00000000`3587e640 000007fe`f74f2bbb FSEAgent_8_0_685!Antigen.Scanner.ScannerThread()+0x13c
10 00000000`3587e6e0 000007fe`f758aadd mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x9b
11 00000000`3587e730 000007fe`f839c6a2 mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()+0x4d
12 00000000`3587e780 000007fe`f825ff03 mscorwks!CallDescrWorker+0x82
13 00000000`3587e7d0 000007fe`f877a291 mscorwks!CallDescrWorkerWithHandler+0xd3
14 00000000`3587e870 000007fe`f81c37b9 mscorwks!MethodDesc::CallDescr+0x2b1
15 00000000`3587eab0 000007fe`f831d298 mscorwks!ThreadNative::KickOffThread_Worker+0x191
16 00000000`3587edd0 000007fe`f8230945 mscorwks!ManagedThreadBase_DispatchInner+0x2c
17 00000000`3587ee20 000007fe`f81fe52d mscorwks!ManagedThreadBase_DispatchMiddle+0x9d
18 00000000`3587eef0 000007fe`f835a2b5 mscorwks!ManagedThreadBase_DispatchOuter+0x31
19 00000000`3587ef30 000007fe`f832825b mscorwks!ManagedThreadBase_FullTransitionWithAD+0x35
1a 00000000`3587ef90 000007fe`f8320134 mscorwks!ThreadNative::KickOffThread+0xd3
1b 00000000`3587f070 00000000`779959dd mscorwks!Thread::intermediateThreadProc+0x78
1c 00000000`3587fc40 00000000`77aca651 kernel32!BaseThreadInitThunk+0xd
1d 00000000`3587fc70 00000000`00000000 ntdll!RtlUserThreadStart+0x1d


All of those aforementioned threads were passing to COM:

!findstack mscorwks!DoCLRToCOMCall
Thread 064, 1 frame(s) match
        * 12 000000003587e3e0 000007ff0042901d mscorwks!DoCLRToCOMCall+0x177

Thread 066, 1 frame(s) match
        * 12 0000000051cfe240 000007ff0042901d mscorwks!DoCLRToCOMCall+0x177

Thread 068, 1 frame(s) match
        * 12 0000000051dfddb0 000007ff0042901d mscorwks!DoCLRToCOMCall+0x177

Thread 070, 1 frame(s) match
        * 12 000000005205e250 000007ff0042901d mscorwks!DoCLRToCOMCall+0x177


And the messages that were being processed were NDRs that have been around the block or two (sorry for the wall of text):

0:064> !mdt 00000000e074c8b0
00000000e074c8b0 (System.String) Length=12515, String="Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undelivered Mail Returned to Sender"


0:070> !mdt 0000000100776d88
0000000100776d88 (System.String) Length=11719, String="Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Undeliverable: Delayed Mail (still being retried)"


The message details (in redacted form) were:

0:064> !mdt 0000000081097f30
0000000081097f30 (Antigen.AntigenMessage)
    m_folder:00000000810a01f0 (System.String) Length=36, String="C:\forefront\data\\Archive\Internal\"
    m_fileObject:0x250e0 (System.UInt32)
    m_bcc:00000000e0754c50 (System.String) Length=43, String=";"
    m_flags:0x0 (System.Int32)
    m_smtpSenderAddress:00000000907eb738 (System.String) Length=21, String=""Sender""
    m_smtpExternalIP:NULL (System.String)
    m_onlyTreatIfPurge:0x0 (System.UInt32)
    m_isSystemMessage:0x1 (System.UInt32)
    m_smtpExternalHostName:NULL (System.String)
    m_MessageDateTimeReceived:00000000e0754990 (System.String) Length=20, String="2015-07-21T08:41:46Z"
    m_MessageDateTimeSent:00000000e07549d8 (System.String) Length=20, String="2015-07-21T08:26:23Z"
    m_EHSBypassVirusScan:0x0 (System.UInt32)
    m_PremiumEngineSet:0x0 (System.UInt32)
    m_IsTransportDecrypted:0x0 (System.UInt32)
    m_ReplayHeaders:00000000810a0428 (System.String) Length=172, String="X-Receiver: <recipient> ORCPT=rfc822;<recipient>
X-Sender: <sender>
X-CreatedBy: MSExchange12"


When I checked message tracking, this was a partial indicator of what introduced the problem:

$msgTrk = @()
$hubz = Get-TransportServer
foreach($hub in $hubz)
{
     $msgTrk += Get-MessageTrackingLog -Server $hub.Name -Recipients  -ResultSize Unlimited
}

$msgTrk.Count
1491474


In the words of Strongbad, "That is not a small number! That's a very big number!"

This was a limitation in the Forefront hook in Transport that caused it to consume a lot of resources on the heap, which was exacerbated by a mail-storm from the customer's on-premises.


Debugging: Incremental Mailbox Sync Move Requests Cause 'StopAll' to be called on the Assistants Infrastructure Which Causes CalendarNotificationAssistant to Stop Processing Text Message (SMS) Notifications
John Bailey [MSFT] — Tue, 05 Apr 2016 16:56:38 GMT
Not too long ago, I debugged an issue with the Calendar Notification Assistant (CNA), wherein the Assistant would stop processing text message (SMS) notifications for events. The affected end-users noted that if they enabled text message (SMS) notifications in Outlook Web Access (OWA), the notifications would work for around 24 hours and then they would fall back into reproduction of the issue, in which the notifications stop altogether.

When I went looking in PowerShell, I could verify (via the Admin Audit Log) that the commands to set the configuration for SMS were ran and that it was enabled:
Search-AdminAuditLog -Cmdlets Set-TextMessagingAccount | Where{$_.ObjectModified -contains ''} | FL


RunspaceId         : c7683058-b25c-4557-818d-546cf5f178c6
ObjectModified     : 
CmdletName         : Set-TextMessagingAccount
CmdletParameters   : {CountryRegionId, MobileOperatorId, NotificationPhoneNumber, Identity}
ModifiedProperties : {}
Caller             : 
ExternalAccess     :
Succeeded          : True
Error              : None
RunDate            : 12/22/2015 1:15:17 PM
OriginatingServer  : 
Identity           : 
IsValid            : True
ObjectState        : New

We could also see the messages being sent via the Message Tracking Log (MTL):
$MsgTrk | Sort TimeStamp | FT EventId,Source,Sender,Recipients,MessageSubject,TimeStamp -AutoSize

EventId Source      Sender                       Recipients                                  MessageSubject 	Timestamp
------- ------      ------                       ----------                                  -------------- 	---------
RECEIVE STOREDRIVER  	{IMCEAMOBILE-+2B10008675309@domain.com}                		12/17/2015 13:18:34
SEND    AGENT        	{IMCEAMOBILE-+2B10008675309@domain.com}                		12/17/2015 13:18:34
RECEIVE STOREDRIVER  	{IMCEAMOBILE-+2B10008675309@domain.com}                		12/17/2015 13:19:46
SEND    AGENT        	{IMCEAMOBILE-+2B10008675309@domain.com}                		12/17/2015 13:19:46
RECEIVE STOREDRIVER  	{IMCEAMOBILE-+2B10008675309@domain.com}                		12/17/2015 16:15:02
SEND    AGENT        	{IMCEAMOBILE-+2B10008675309@domain.com}                		12/17/2015 16:15:02

Using this, I could tell the originator of the message: The Mailbox Assistants. This was done by the seeing that the client and the server were the same:
$MsgTrk[0] | FL


RunspaceId              : 13ad4581-25d9-4136-8ed1-a284b7e979db
Timestamp               : 12/21/2015 15:40:31
ClientIp                : 
ClientHostname          : 
ServerIp                : 
ServerHostname          : 
SourceContext           : 08D30340C2F43943
ConnectorId             :
Source                  : STOREDRIVER
EventId                 : RECEIVE
InternalMessageId       : 2429739
MessageId               : 
Recipients              : {IMCEAMOBILE-+2B10008675309@domain.com}
RecipientStatus         : {To}
TotalBytes              : 6386
RecipientCount          : 1
RelatedRecipientAddress :
Reference               :
MessageSubject          :
Sender                  : 
ReturnPath              : 
MessageInfo             : 04I:
MessageLatency          :
MessageLatencyType      : None
EventData               : {[MailboxDatabaseGuid, 347a971c-e8f5-45de-8e01-870ed4dc6901], [ItemEntryId, 00-00-00-00-1E-95-32-44-0A-EA-D6-47-A5-98-9B-F0-5F-89-B9-09-07-00-BE-9B-24-68-00-23-66-49-95-41-BB-56-C6-02-E3-D0-00-00-00-03-71-5C-00-00-75-FB-A6-9E-77-CF-BE-43-A8-2A-B9-7F-C1-34-9A-01-00-00-1C-C-91-D1-00-00]}

Once I knew this, I checked the Database Event Watermarks for any backlog:
\ CalendarNotificationAssistant             977234167 989106153    11871986

I now had enough data to assert that a dump of the process was necessary in order to see what's going on.

At the time of the dump, despite the fact that the user was no longer receiving text message (SMS) notifications, the process was processing (for lack of a better turn-of-phrase at the moment) a two week old message in thread 21 in the stack:
0:021> !mdt 00000000036e1088
00000000036e1088 (Microsoft.Exchange.MailboxAssistants.Assistants.CalendarNotification.CalendarEvent)
    <CalendarItemIdentity>k__BackingField:00000000036e1208 (Microsoft.Exchange.Data.Storage.StoreObjectId)
    <CalendarItemOccurrenceIdentity>k__BackingField:00000000036e1048 (Microsoft.Exchange.Data.Storage.OccurrenceStoreObjectId)
    <CalendarItemType>k__BackingField:0x1 (Microsoft.Exchange.Data.Storage.CalendarItemType)
    <NormalizedSubject>k__BackingField:00000000036e0880 (System.String) Length=5, String="Lunch"
    <Location>k__BackingField:00000000036e0860 (System.String) 
    <ReminderIsSet>k__BackingField:true (System.Boolean)
    <ReminderTime>k__BackingField:(Microsoft.Exchange.ExchangeSystem.ExDateTime) VALTYPE (MT=000007ff002c5cb0, ADDR=00000000036e10d0)
    <ReminderMinutesBeforeStart>k__BackingField:0xf (System.Int32)
    <StartTime>k__BackingField:(Microsoft.Exchange.ExchangeSystem.ExDateTime) VALTYPE (MT=000007ff002c5cb0, ADDR=00000000036e10f0)
    <EndTime>k__BackingField:(Microsoft.Exchange.ExchangeSystem.ExDateTime) VALTYPE (MT=000007ff002c5cb0, ADDR=00000000036e1110)
    <FreeBusyStatus>k__BackingField:0x2 (Microsoft.Exchange.Data.Storage.BusyType)
    <ChangeHighlight>k__BackingField:0x0 (Microsoft.Exchange.Data.Storage.ChangeHighlightProperties)
    <AppointmentState>k__BackingField:0x0 (Microsoft.Exchange.Data.Storage.AppointmentStateFlags)
    <CreationRequestTime>k__BackingField:(Microsoft.Exchange.ExchangeSystem.ExDateTime) VALTYPE (MT=000007ff002c5cb0, ADDR=00000000036e1130)
    <ResponseType>k__BackingField:0x0 (Microsoft.Exchange.Data.Storage.ResponseType)
    <OldStartTime>k__BackingField:(System.Nullable`1) VALTYPE (MT=000007fef6207cc8, ADDR=00000000036e1150)
    <OldEndTime>k__BackingField:(System.Nullable`1) VALTYPE (MT=000007fef6207cc8, ADDR=00000000036e1178)

We can see the date of the item from the backing field:
0:021> !mdt 000007ff002c5cb0 00000000036e10d0
(Microsoft.Exchange.ExchangeSystem.ExDateTime) VALTYPE (MT=000007ff002c5cb0, ADDR=00000000036e10d0)
    timeZone:00000000014bb040 (Microsoft.Exchange.ExchangeSystem.ExTimeZone)
    universalTime:(System.DateTime) 2015/12/23 16:15:00.000 VALTYPE (MT=000007fef62482d0, ADDR=00000000036e10d8)
    localTime:(System.Nullable`1) VALTYPE (MT=000007fef6207cc8, ADDR=00000000036e10e0)

And we can see that it's processing the item for the same user that lodged the complaint:
0:021> !mdt 0000000003761ad0
0000000003761ad0 (System.String) Length=129, String=""

Finally, after perusing the dump for an hour or more, I found what I believed to be introducing the issue:

<ContextInfo>k__BackingField:00000000032fe908 (System.String) Length=7, String="StopAll"

Given this, it was time to go to the devs and figure out what was going on. After some back-and-forth and a conference call with both the Store and Calendaring devs, we arrived at what was causing the issue.

First, it's important to understand that an incremental sync was created for the user but had not - as of the time of debugging - been completed:
Get-MoveRequestStatistics 957ca0a9-f84f-425e-a263-5b468e90d574 | FL *increm*


EffectiveIncrementalSyncInterval  : 1.00:00:00
ConfiguredIncrementalSyncInterval : 1.00:00:00

Since it syncs once a day, this aligns with the end-user's statement regarding reconfiguration but we needed proof.

When we went and looked at the Mailbox Database Events for the mailbox in question, we saw the 'MailboxMoveStarted' event fire:
$eventz | FT Counter,ItemType,EventName,ObjectClass,Mailbox,CreateTime -AutoSize

  Counter ItemType   EventName          ObjectClass Mailbox CreateTime
  ------- --------   ---------          ----------- ------- ----------
974015558 MAPI_STORE MailboxMoveStarted                     1/25/2016 21:32:01
974683553 MAPI_STORE MailboxMoveStarted                     1/26/2016 21:41:10
975414298 MAPI_STORE MailboxMoveStarted                     1/27/2016 21:44:00
976136834 MAPI_STORE MailboxMoveStarted                     1/28/2016 21:52:10
976842768 MAPI_STORE MailboxMoveStarted                     1/29/2016 22:02:30
977056359 MAPI_STORE MailboxMoveStarted                     1/30/2016 22:03:45
977195259 MAPI_STORE MailboxMoveStarted                     1/31/2016 22:15:01


What's important to note as a side-comment/take-away is that the 24-hour interval doesn't initiate from the same start time. This is because the counter for the 24-hour interval starts when the last sync stopped.

When we matched this against a trace of the Assistant, we saw the following:
"2016/01/13-03:06:05.279","9444","2480","Assistants","EventDispatcher","Dispatcher for  (957ca0a9-f84f-425e-a263-5b468e90d574) and Calendar Notification Assistant: Event handled successfully: Counter: 0x3B331E14; MailboxID: 957ca0a9-f84f-425e-a263-5b468e90d574; Mask: MailboxMoveStarted; Flags: Source; ExtendedFlags: None;Object Class: ; Created Time: 1/13/2016 11:06:04 AM;Item Type: MAPI_STOREItem EntryId: Parent entryId: Old Item entryId: Old parent entryId: SID: S-1-5-21-276227223-3107228690-1043224028-159164Client Type: MigrationDocument ID: 0" 
"85868820"        020B591B          Debug 2016/01/13-02:01:49.777 9444     2480            MailboxAssistants.Assistants.CalendarNotification Assistant           rmd----- caller: StopAll, subj: >, usr: c1, calItemId: RgAAAAAelTJECurWR6WYm/BfibkJBwC+myRoACNmSZVBu1bGAuPQAAAAA3FaAAB1+6aed8++Q6gquX/BNJoBAAAcyVmOAAAP, calItemOccId: , event_t: 01/13/2016 16:15:00, rmd_t: 01/13/2016 19:45:00, s_t: 01/13/2016 20:00:00, e_t: 01/13/2016 21:30:00
"8326743"         006B5AB0          Debug 2016/01/13-03:06:05.277 9444     2480            MailboxAssistants.Assistants.CalendarNotification UserSettings      Processing Disabling event for mailbox > on database 347a971c-e8f5-45de-8e01-870ed4dc6901

To elaborate on this bug and explain it further: Recall back in Exchange 2007 (E12) that mailbox moves caused the mailbox to be locked. When this occurred, it would've been necessary for any code that attempts to open the message store to stop processing the mailbox until the move is complete. Fast-forward a few years (nigh to a decade) and incremental sync move requests are a common, every day occurrence. When the incremental sync starts, there is no delineation between it and a regular move request, so a database event fires and this signals the assistant to stop processing the mailbox. When the text message (SMS) notification is reconfigured, it signals the Assistant to start processing on the mailbox. This explains the behavior seen by the end-users.

For those of you concerned about hitting this bug in the wild, we've only see it happen once and we're using that instance to drive a change in the Assistant logic in all versions of Exchange (assuming all have the same bug in the logic).


PowerShell + C#: Accessing the Current Runspace And Adding a Snap-In
John Bailey [MSFT] — Tue, 05 Apr 2016 14:18:42 GMT
I've been coding an assembly for import into the Exchange Management Shell (EMS) in C# for some time. One of the problems that I ran into was how to load the Support assembly, since it's not imported into the RunSpace by default. After a while of testing/trying/failing and repeat ad nauseam, I found the answer in the form of what I consider to be a cheat.

You see, when the module is loaded, I'm depending on the current runspace as it exists to execute commands. There's no proper way to access the runspace that you're currently executing to modify it's configuration by adding the snap-in. 

So, we arrive at the "cheat":
// We use the current runspace to cast our calls into.
// This grants us access to the current runspace, instead of instantiating a new one.
// More information on the PowerShell.Create method can be found here: https://msdn.microsoft.com/en-us/library/system.management.automation.powershell.create(v=vs.85).aspx
// More information on the RunspaceMode.CurrentRunspace enumeration value can be found here: https://msdn.microsoft.com/en-us/library/system.management.automation.runspacemode(v=vs.85).aspx
PowerShell initialPowerShell = PowerShell.Create(RunspaceMode.CurrentRunspace);

// More information on the PowerShell.AddScript method can be found here: https://msdn.microsoft.com/en-us/library/dd182436(v=vs.85).aspx
initialPowerShell.Commands.AddScript("Add-PSSnapin Microsoft.Exchange.Management.Powershell.Support");

// More information on the PowerShell.Invoke method can be found here: https://msdn.microsoft.com/en-us/library/dd182449(v=vs.85).aspx
initialPowerShell.Invoke();


If you're wondering about the code comments: One of the ways that people learn is by searching for examples or questions related to the problem they're trying to solve. This is something that I - admittedly - used to do. Once they find what they think might be the answer, they look at the methods/class in the documentation (at least, that's what I used to do). So, I figured why not shorten the effort down for those who want to learn? :)


Exchange 2010: AD Recipient Cache and Locking Threads in EdgeTransport - Or How We Figured Out Something Wasn't Right With Mail Stuck in Submission Queue
John Bailey [MSFT] — Mon, 15 Jun 2015 00:04:39 GMT
A while ago we had an issue, wherein SCOM fired for message queue length for messages in Submission. When we went to take a look, the messages were stuck in Submission and weren't really going anywhere. Took a dump of EdgeTransport, paused Transport, and the queues eventually flushed.
Opened Windbg against the dump and here's what was going on: Every time a flood of email would come, Transport would see that the local recipient cache for a group was out of date. Threads would spin-up (typically, per message) to update the recipient cache. Since this is an AD call, a lock is used. The lock is owned by the thread until it's released, which means we need to wait for the LDAP call to complete before the next thread can enter the critical section. Repeat ad infinim. The issue that complicated this matter is that the refresh was for the same group of sixteen-thousand-plus users, each time.
In the dump, this is how I went down the rabbit hole. First, I checked for any locks that may be blocking in the process:
 
!locks
 
CritSec Wldap32!SelectLock1+0 at 000007fefdd5ada0
WaiterWoken        No
LockCount          0
RecursionCount     1
OwningThread       2a18
EntryCount         0
ContentionCount    2
*** Locked
 
I then checked the another thread to see if it was aware that there was contention (it was):
 
00 00000000`2b8fd4f8 000007fe`fd991430 ntdll!ZwWaitForMultipleObjects+0xa
01 00000000`2b8fd500 00000000`77731723 KERNELBASE!WaitForMultipleObjectsEx+0xe8
02 00000000`2b8fd600 000007fe`f830d0a5 kernel32!WaitForMultipleObjectsExImplementation+0xb3
03 00000000`2b8fd690 000007fe`f8311aa9 mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0xc1
04 00000000`2b8fd730 000007fe`f8344e95 mscorwks!Thread::DoAppropriateAptStateWait+0x41
05 00000000`2b8fd790 000007fe`f8427d94 mscorwks!Thread::DoAppropriateWaitWorker+0x191
06 00000000`2b8fd890 000007fe`f83ee48a mscorwks!Thread::DoAppropriateWait+0x5c
07 00000000`2b8fd900 000007fe`f82f077d mscorwks!CLREvent::WaitEx+0xbe
08 00000000`2b8fd9b0 000007fe`f843e6ea mscorwks!AwareLock::EnterEpilog+0xc9
09 00000000`2b8fda80 000007fe`f83336db mscorwks!AwareLock::Enter+0x72
0a 00000000`2b8fdab0 000007fe`f88b308f mscorwks!AwareLock::Contention+0x1fb
0b 00000000`2b8fdb80 000007fe`f2ff2beb mscorwks!JITutil_MonContention+0xdf
0c 00000000`2b8fdd30 000007fe`f2ff318f Microsoft_Exchange_Data_Directory_ni!Microsoft.Exchange.Data.Directory.IsMemberOfProvider.AutoRefreshCache`3[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]].CreateAndCache(System.__Canon, System.__Canon, Boolean)+0x16cb
0d 00000000`2b8fddf0 000007fe`f324dd16 Microsoft_Exchange_Data_Directory_ni!Microsoft.Exchange.Data.Directory.IsMemberOfProvider.AutoRefreshCache`3[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]].GetValue(System.__Canon, System.__Canon)+0x192f
0e 00000000`2b8fdeb0 000007fe`f324e2a0 Microsoft_Exchange_Data_Directory_ni!Microsoft.Exchange.Data.Directory.IsMemberOfProvider.IsMemberOfResolver`1[[System.__Canon, mscorlib]].IsMemberOf(Microsoft.Exchange.Data.Directory.Recipient.ADRecipientSession, System.Guid, System.Guid)+0x25b6a6
0f 00000000`2b8fe090 000007fe`f0940c15 Microsoft_Exchange_Data_Directory_ni!Microsoft.Exchange.Data.Directory.IsMemberOfProvider.IsMemberOfResolver`1[[System.__Canon, mscorlib]].IsMemberOf(Microsoft.Exchange.Data.Directory.Recipient.ADRecipientSession, Microsoft.Exchange.Data.Directory.ADObjectId, System.__Canon)+0x25bfd0
10 00000000`2b8fe180 000007fe`e0fc7267 Microsoft_Exchange_Transport_ni!Microsoft.Exchange.Transport.RecipientAPI.AddressBookImpl.IsMemberOf(Microsoft.Exchange.Data.Transport.RoutingAddress, Microsoft.Exchange.Data.Transport.RoutingAddress)+0x135
11 00000000`2b8fe200 000007fe`f245c46a Microsoft_Exchange_MessagingPolicies_Rules_ni!Microsoft.Exchange.MessagingPolicies.Rules.MembershipChecker.Equals(System.String, System.String)+0x47
12 00000000`2b8fe250 000007fe`e0fd5584 Microsoft_Exchange_Net_ni!Microsoft.Exchange.MessagingPolicies.Rules.RuleUtils.CompareStringValues(System.Object, System.Object, Microsoft.Exchange.MessagingPolicies.Rules.IStringComparer)+0x7a
13 00000000`2b8fe2a0 000007fe`f2459efe Microsoft_Exchange_MessagingPolicies_Rules_ni!Microsoft.Exchange.MessagingPolicies.Rules.IsMemberOfPredicate.Evaluate(Microsoft.Exchange.MessagingPolicies.Rules.RulesEvaluationContext)+0x74
14 00000000`2b8fe2e0 000007fe`f2459efe Microsoft_Exchange_Net_ni!Microsoft.Exchange.MessagingPolicies.Rules.OrCondition.Evaluate(Microsoft.Exchange.MessagingPolicies.Rules.RulesEvaluationContext)+0x6e
15 00000000`2b8fe360 000007fe`f246a157 Microsoft_Exchange_Net_ni!Microsoft.Exchange.MessagingPolicies.Rules.OrCondition.Evaluate(Microsoft.Exchange.MessagingPolicies.Rules.RulesEvaluationContext)+0x6e
16 00000000`2b8fe3e0 000007fe`e0fd839f Microsoft_Exchange_Net_ni!Microsoft.Exchange.MessagingPolicies.Rules.RulesEvaluator.Run()+0x1b7
 
O.k., so if we’re update the cache, how many times are we doing it? Oh, I see…:
 
0:086> !findstack Microsoft_Exchange_Data_Directory_ni!Microsoft.Exchange.Data.Directory.IsMemberOfProvider.AutoRefreshCache`3 0
Thread 023, 2 frame(s) match
Thread 024, 2 frame(s) match
Thread 025, 2 frame(s) match
Thread 034, 2 frame(s) match
Thread 035, 2 frame(s) match
Thread 036, 2 frame(s) match
Thread 037, 2 frame(s) match
Thread 038, 2 frame(s) match
Thread 039, 2 frame(s) match
Thread 040, 2 frame(s) match
Thread 050, 2 frame(s) match
Thread 051, 2 frame(s) match
Thread 052, 2 frame(s) match
Thread 053, 2 frame(s) match
Thread 054, 2 frame(s) match
Thread 055, 2 frame(s) match
Thread 056, 2 frame(s) match
Thread 057, 2 frame(s) match
Thread 058, 2 frame(s) match
Thread 059, 2 frame(s) match
Thread 060, 2 frame(s) match
Thread 061, 2 frame(s) match
Thread 062, 2 frame(s) match
Thread 063, 2 frame(s) match
Thread 064, 2 frame(s) match
Thread 065, 2 frame(s) match
Thread 066, 2 frame(s) match
Thread 067, 2 frame(s) match
Thread 068, 2 frame(s) match
Thread 069, 2 frame(s) match
Thread 070, 2 frame(s) match
Thread 071, 2 frame(s) match
Thread 072, 2 frame(s) match
Thread 073, 2 frame(s) match
Thread 074, 2 frame(s) match
Thread 075, 2 frame(s) match
Thread 076, 2 frame(s) match
Thread 077, 2 frame(s) match
Thread 078, 2 frame(s) match
Thread 079, 2 frame(s) match
Thread 080, 2 frame(s) match
Thread 081, 2 frame(s) match
Thread 082, 2 frame(s) match
Thread 083, 2 frame(s) match
Thread 084, 2 frame(s) match
Thread 085, 2 frame(s) match
Thread 086, 2 frame(s) match
Thread 087, 2 frame(s) match
 
The thread with the lock has an LDAP call being made:
 
00 00000000`5443cfc8 000007fe`fce91e35 ntdll!ZwWaitForSingleObject+0xa
01 00000000`5443cfd0 000007fe`fce91f96 mswsock!SockWaitForSingleObject+0xe145
02 00000000`5443d050 000007fe`ff1f4efc mswsock!WSPSelect+0xac26
03 00000000`5443d1f0 000007fe`ff1f4e7d ws2_32!DPROVIDER::WSPSelect+0x3c
04 00000000`5443d230 000007fe`fdd13a19 ws2_32!select+0x75a
05 00000000`5443d330 000007fe`fdd199b5 Wldap32!DrainWinsock+0x417
06 00000000`5443d3c0 000007fe`fdd136ec Wldap32!LdapWaitForResponseFromServer+0x646
07 00000000`5443d480 000007fe`fdd1d967 Wldap32!ldap_result_with_error+0x279
08 00000000`5443d520 000007fe`f84abec7 Wldap32!ldap_result+0x74
*** WARNING: Unable to verify checksum for System.DirectoryServices.Protocols.ni.dll
09 00000000`5443d580 000007fe`f1c453ff mscorwks!DoNDirectCall__PatchGetThreadCall+0x7b
0a 00000000`5443d630 000007fe`f1c5c61b System_DirectoryServices_Protocols_ni!DomainBoundILStubClass.IL_STUB(IntPtr, Int32, Int32, System.DirectoryServices.Protocols.LDAP_TIMEVAL, IntPtr ByRef)+0x8f
0b 00000000`5443d720 000007fe`f1c59f79 System_DirectoryServices_Protocols_ni!System.DirectoryServices.Protocols.LdapConnection.ConstructResponse(Int32, System.DirectoryServices.Protocols.LdapOperation, System.DirectoryServices.Protocols.ResultAll, System.TimeSpan, Boolean)+0x2cb
 
It looks like we’re performing this query, at the time of the dump:
 
000000005443d538  00000000e0188bf8  System.String  "(&(memberOf=20ff7cf3-3b7e-4767-8669-d2295d6fb0f3>)(!(msExchCU=*)))"
 
0:086> !do 00000000e0188bf8
Name: System.String
MethodTable: 000007fef7747d90
EEClass: 000007fef734e560
Size: 170(0xaa) bytes
GC Generation: 2
(C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)
String: (&(memberOf=20ff7cf3-3b7e-4767-8669-d2295d6fb0f3>)(!(msExchCU=*)))
Fields:
              MT            Field           Offset                 Type VT             Attr            Value Name
000007fef774f000  4000096        8         System.Int32  1 instance               73 m_arrayLength
000007fef774f000  4000097        c         System.Int32  1 instance               72 m_stringLength
000007fef77497d8  4000098       10          System.Char  1 instance               28 m_firstChar
000007fef7747d90  4000099       20        System.String  0   shared           static Empty
                                 >> Domain:Value  00000000004e5e50:000000015fff0370 <<
000007fef7749688  400009a       28        System.Char[]  0   shared           static WhitespaceChars
                                 >> Domain:Value  00000000004e5e50:000000015fff0b90 <<
 
O.k., how many calls against this object are we making?:
 
0:086> !refs 00000000e0188bf8
This command will work faster with an SOSEX heap index.  To build an index, run !bhi.
Objects referenced by 00000000e0188bf8 (System.String):
NONE
 
Objects referencing 00000000e0188bf8 (System.String):
follow 00000000c0393040        80   System.Object[]
follow 00000000c0393108        96   System.Object[]
follow 00000000c04a1928        80   System.DirectoryServices.Protocols.SearchRequest
follow 00000000c04a2960        72   System.Object[]
follow 00000000c0689800        80   System.Object[]
follow 00000000c06898c8        96   System.Object[]
follow 00000000c07c3e68        80   System.DirectoryServices.Protocols.SearchRequest
follow 00000000c07c4de0        72   System.Object[]
follow 00000000c08ba200        80   System.Object[]
follow 00000000c08ba2c8        96   System.Object[]
follow 00000000c0976870        80   System.DirectoryServices.Protocols.SearchRequest
follow 00000000c09777e8        72   System.Object[]
follow 00000000c0fbaf58        80   System.Object[]
follow 00000000c0fbb020        96   System.Object[]
follow 00000000c103fa40        80   System.DirectoryServices.Protocols.SearchRequest
follow 00000000c1046fd0        72   System.Object[]
follow 00000000c113a868        80   System.Object[]
follow 00000000c113a930        96   System.Object[]
follow 00000000c11e75a0        80   System.DirectoryServices.Protocols.SearchRequest
follow 00000000c11e8518        72   System.Object[]
follow 00000000c13b1068        80   System.Object[]
follow 00000000c13b1130        96   System.Object[]
follow 00000000c1547260        80   System.DirectoryServices.Protocols.SearchRequest
follow 00000000c15481d8        72   System.Object[]
follow 00000000c1669338        80   System.Object[]
follow 00000000c1669400        96   System.Object[]
follow 00000000c175b778        80   System.DirectoryServices.Protocols.SearchRequest
follow 00000000c175c6f0        72   System.Object[]
follow 00000000c1904c80        80   System.Object[]
follow 00000000c1904d48        96   System.Object[]
follow 00000000c1acba70        80   System.DirectoryServices.Protocols.SearchRequest
follow 00000000c1acc9e8        72   System.Object[]
follow 00000000c1d03f08        80   System.Object[]
follow 00000000c1d03fd0        96   System.Object[]
follow 00000000c1f8f398        80   System.DirectoryServices.Protocols.SearchRequest
follow 00000000c1f90310        72   System.Object[]
follow 00000000c21bd2f0        80   System.Object[]
follow 00000000c21bd3b8        96   System.Object[]
follow 00000000c243d8a0        80   System.DirectoryServices.Protocols.SearchRequest
follow 00000000c243e818        72   System.Object[]
follow 00000000e0188510       176   Microsoft.Exchange.Data.Directory.ADPagedReader`1[[Microsoft.Exchange.Data.Directory.ADRawEntry, Microsoft.Exchange.Data.Directory]]
follow 00000000f096aea8        80   System.DirectoryServices.Protocols.SearchRequest
follow 00000001106ce9f8        80   System.DirectoryServices.Protocols.SearchRequest
 
CLR Thread 0x49
stack:000000005443d538
stack:000000005443d5d0
stack:000000005443d970
  
O.k., how many threads do we have waiting on this?:
 
0:086> !syncblk
Index         SyncBlock MonitorHeld Recursion Owning Thread Info          SyncBlock Owner
  148 000000004ef62350           95         1 000000004ef6a070  2a18  86   0000000080074a08 System.Object
            Waiting threads: 23 24 25 34 35 36 37 38 39 40 50 51 52 53 54 55 56 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 87
-----------------------------
Total           1548
CCW             0
RCW             0
ComClassFactory 0
Free            0
 
In the words of Strongbad: That is not a very small number. That is a very BIG number!
 
For those of you wondering what this means in common, everyday production use of Exchange: 
 
Mail would come in, destined for 'x' recipient. Transport would see a rule that has the condition, IsMemberOf. Then, Transport would see that the local recipient cache for the group it needs to verify that the recipient is a member of is out-of-date. It would call a thread to update it. Since threads aren't context-aware, this would occur for many of the recipients currently in the Submission Queue. The thread making the LDAP call would own the lock, as it entered the critical section. The other threads had to wait for the lock to be released to make their AD call. Since only one thread in the process' bounds can own a lock, at a time, this means the threads could only complete in a one-by-one scenario. This is also why the mail couldn't be categorized because the lock prevented the thread from validating IsMemberOf without a current cache.

 
For those of you concerned if you'll run into this issue: This condition only reproduces in a single instance and that instance has only recurred twice, so far, during the timespan of a year. Rest assured, that even though the perceivable affected footprint of this issue is small, it is being investigated by the Transport developers.