• MessageText=The specified object was not found in the store., Cannot decode the message -itemId: E4E_M_edc899287-179d-4ab4-a115-a9e2784d1dd2, -currentUser user42@contoso.com
• Cannot decode the message -itemId: E4E_M_fcf2a6f5-1396-434f-b376-67a0e2a01478, -currentUser user42@contoso.com","ResponseCode":"ErrorRightsManagementPermanentException","ResponseClass":"Error","Items":null"

• An item arrives in the mailbox on March 1^st 2012 at 15:37:16.714 UTC time.
• The user reads the item on March 10^th 2012 at 02:19:47:917 UTC time and never takes action on the item again.
• Today this item would still be sitting in the Inbox.

• An item arrives in the mailbox on March 1^st 2012 at 15:37:16.714 UTC time.
• The user reads the item on March 10^th 2012 at 02:19:47:917 UTC time.
• The user Shift deletes the item on April 3^rd 2012 at 20:05:52.574.  The mail item moves from Inbox to Recoverable Items\Deletions at the time of the Shift+delete.  Shift+Delete is considered a Hard Delete.  Without SingleItemRetention=$true the item would disappear from the mailbox instantly.  Since SingleItemRetention is the default behaviour for Exchange Online mailboxes the message moves to the Deletions folder.
• On April 17^th, 2012 at 20:05:52.575 the item is no longer protected by the RetainDeletedItemsFor property.  The database engine permanently removes the item within a few minutes.

• An item arrives in the mailbox on March 1^st 2012 at 15:37:16.714 UTC time.
• The user reads the item on March 10^th 2012 at 02:19:47:917 UTC time.
• The user Shift deletes the item on April 3^rd 2012 at 20:05:52.574.  The mail item moves from Inbox to Recoverable Items\Deletions at the time of the Shift+delete.  Shift+Delete is considered a Hard Delete.
• On April 17^th, 2012 at 20:05:52.575 the item is no longer protected by the RetainDeletedItemsFor property.  The database engine attempts to permanently remove the item within a few minutes.  The removal attempt cannot proceed because of the Litigation Hold.  The server does not want to leave the item where it can be recovered by Outlook or OWA.  Therefore it moves the item to the Purges folder.
• On March 2^nd, 2015 at 15:37:16.715 UTC time the protection of the Litigation Hold ends.  The database engine permanently removes the mail item from the Purges Folder.

• An item arrives in the mailbox on March 1^st 2012 at 15:37:16.714 UTC time.
• The user reads the item on March 10^th 2012 at 02:19:47:917 UTC time.
• The user Shift deletes the item on April 3^rd 2017 at 20:05:52.574.  The mail item moves from Inbox to Recoverable Items\Deletions at the time of the Shift+delete.  Shift+Delete is considered a Hard Delete.
• On April 17^th, 2017 at 20:05:52.575 the item is no longer protected by the RetainDeletedItemsFor property.  The database engine attempts to permanently remove the item within a few minutes.  The removal attempt succeeds because the Litigation Hold only protected this item for 1096 days from its arrival on March 1^st 2012 at 15:37:16.714 UTC time.  Therefore any delete of the item at or after March 2^nd, 2015 at 15:37:16.715 UTC time proceeds as if there was no Litigation Hold in place.

• An item arrives in the mailbox on March 1^st 2012 at 15:37:16.714 UTC time.
• MRM stamps the item with a retention date of March 1^st 2015 at 15:37:16.714 UTC time.
• The user reads the item on March 10^th 2012 at 02:19:47:917 UTC time.
• The user Shift deletes the item on April 3^rd 2012 at 20:05:52.574.  The mail item moves from Inbox to Recoverable Items\Deletions at the time of the Shift+delete.  Shift+Delete is considered a Hard Delete.
• On April 17^th, 2012 at 20:05:52.575 the item is no longer protected by the RetainDeletedItemsFor property.  The database engine attempts to permanently remove the item within a few minutes.  The removal attempt cannot proceed because of the Litigation Hold.  The server does not want to leave the item where it can be recovered by Outlook or OWA.  Therefore it moves the item to the Purges folder.
• On March 2^nd, 2015 at 15:37:16.715 UTC time the protection of the Litigation Hold ends.  The database engine permanently removes the mail item from the Purges Folder.  In this instance the 3 year delete policy has no effect on how the item is handled.

• An item arrives in the mailbox on March 1^st 2012 at 15:37:16.714 UTC time.
• MRM stamps the item with a retention date of March 1^st 2014 at 15:37:16.714 UTC time.
• The user reads the item on March 10^th 2012 at 02:19:47:917 UTC time.
• MRM's ManagedFolderAssistant completes at 15:30 on March 1^st, 2014.  The item is not deleted this day because MRM completed a few minutes before the expiry time of 15:37:16.714 UTC.
• MRM get throttled on each of the next 4 days and terminates for the day without deleting the item.
• The deletion that moves the item from Inbox to Recoverable Items\Purges takes place on March 6^th 2015 at 15:29:28.520 UTC time.
• On March 1st 2015 at 15:37:16.714 UTC time the item is no longer protected by the 3 year litigation hold.  The database engine deletes it within minutes.

• An item arrives in the mailbox on March 1st 2012 at 15:37:16.714 UTC time.
• MRM stamps the item with a retention date of March 1st 2015 at 15:37:16.714 UTC time.
• The user reads the item on March 10th 2012 at 02:19:47:917 UTC time.
• MRM's ManagedFolderAssistant completes at 15:30 on March 1st, 2015. The item is not deleted this day because MRM completed a few minutes before the expiry time of 15:37:16.714 UTC.
• MRM get throttled on each of the next 4 days and terminates for the day without deleting the item.
• The Managed Folder Assistant (MRM)  moves the item from Inbox to Recoverable Items\Purges on March 6th 2015 at 15:29:28.520 UTC time.
• On March 20th 2015 at 15:29:28.521 UTC time the item is no longer protected by the RetainDeletedItemsFor (this mailbox property defaults to 14 days). The database engine deletes it within minutes. The 3 year litigation hold does not prevent the removal from the Purges folder because the Litigation hold only protected the item from deletion between March 1st 2012 at 15:37:16.714 UTC and March 1st, 2015 at 15:37:16.714 UTC

# Enter your credentials when prompted

$session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://ps.outlook.com/powershell/" -Credential $O365Cred -Authentication Basic -AllowRedirection

Import-PSSession $session

Why do this?  There are two reasons.

• If you have a search that is failing or timing out a common troubleshooting step is to try the same search on a single mailbox.  If it works then the next question is does it work with 10 mailboxes?  a 100?  500?  Once you discover a comfortable threshold you might have thousands (in some Office 365 tenants hundreds of thousands) of mailboxes that need to have the same search applied.  Creating all of those searches by hand is at best tedious and at worst impossible.  The mailbox selection dialog in the web interface shows 500 mailboxes.
• You need to apply the search to more than 10,000 mailboxes.

If you need to know more about the basics of Discovery Searches before reading further you can get a great overview of how to handle Discovery Searches and in-place hold from part I and II of this blog:

https://blogs.technet.com/b/exchange/archive/2012/09/26/in-place-e-discovery-and-in-place-hold-in-the-new-exchange.aspx

The cloud does have some differences from the on-premise world:

• There is a maximum of 10,000 mailboxes per search
• Only 2 searches can run simultaneously (though 32 can be queued)
• In the on-prem world you can precisely measure the load on the server that is doing the Discovery Search.  In the cloud this detail is hidden.  If your search is failing due to timeouts you may have to break it into smaller pieces or try it at a time fewer administrators of other tenants are doing the same thing.  To some extent this is a little like deciding what time of day to go to the bank.  If everyone else goes to the back at the same time the Tellers (servers/bank employees) will take longer to serve you.  You might even give up and walk away (a timeout error).
• The destination Discovery Mailboxes are capped at 50GB.

NOTE:  Please remember that all the numbers given here are subject to change as Office 365 evolves.

So how do you work with a large number of users?  Most administrators find it awkward to deal with the Office 365 web interface for adding thousands of mailboxes to a query because the selection dialog only shows 500 mailboxes.  There are a couple ways to deal with this:

• Create Distribution Groups with up to 10,000 member in each.  You can then base the search off the distribution group's membership instead of adding the mailboxes
• Use a script like the one I provide in this blog to add the individual mailboxes to each search

The first way has the advantage that once the group is built you need only add one item to the Discovery Search you wish to run (remember the group membership must not exceed 10,000 mailboxes).  One catch is that if group membership changes, and the Discovery Search has already been started, the change of group membership is not automatically cascaded into search results.  If you want the latest membership changes accounted for you will have to restart the search.

Here are a couple of scenarios an administrator might encounter:

You have 33,000 users that need to have in-place hold instituted.  In this example you would need to create 4 distribution groups (three of 10,000 members and one of 3000 members).  You need not worry about the size of the discovery mailbox because it is not a required parameter for placing an in-place hold.

You need to conduct a Discovery Search against 67,000 mailboxes.  This would mean creating a minimum of 7 distribution groups.  However, trying to do searches against thousands of mailboxes might run into the limit on the size of a Discovery Search mailbox (presently 50GB).  Avoiding the size limit of the Discovery Search mailbox might cause you to go with 500 mailboxes per Discovery Search to avoid overflowing the Discovery Mailbox.  This means creating a lot more than 7 groups.  It also means creating additional Discovery Mailboxes so no one mailbox is overwhelmed with content.

To handle both of these hypothetical examples I have written a script that can create the necessary Discovery Searches and spread them over the list of Discovery Mailboxes that are supplied to the script via a TXT file:

#

#  Objective:

#  Create a new Discovery Search for every $mailboxesPerSearch users.  The search

# results are distributed across the Discovery Mailboxes specified in

# c:\o365\DiscoveryMbxs.txt.  The distribution is round robin.  The TXT file MUST

# contain the UPN of each Discovery Mailbox to be used and each UPN should be on its

# own line of the file.  No field name or column heading should be used.  THERE MUST

# BE AT LEAST ONE UPN IN c:\o365\DiscoveryMbxs.txt

# 

#  Each search name will be text followed by a sequential number.  For example: 

# Search1, Search2, etc.

#

#  The script does not start any of the Discovery Searches it creates.

#

#  For Demonstration purposes this script places 3 mailboxes in each search.  You can

# change this by altering the value of the $mailboxesPerSearch variable.

#

#   Load the list of discovery mailboxes into a variable.  Searches will be distributed

#  across the Discovery mailboxes in round robin fashion.

[array]$DiscoveryList = get-content "c:\o365\DiscoveryMbxs.txt"  

# Must use [array] to handle the case where the text file lists only one mailbox. 

#Without Array it gets treated as a string and fails later.

$DiscoveryCount = $DiscoveryList.count

$DiscoveryLoop = 0

#   Set the number of mailboxes that will be in each search.  Acceptable range is 1 to

# 10000.  3 is selected for illustration purposes in a lab.  Remember that discovery

# mailboxes go against the 10,000 max, but are not listed individually in the search.

# If all your mailboxes have archives then treat the maximum as 5,000.

$mailboxesPerSearch = 3                       

$base_search_name = "Legal-TESTsearch"

$loop_count = 0

# The line below obtains the list of mailboxes to be searched.

$mbxs = get-mailbox -resultsize unlimited -RecipientTypeDetails UserMailbox       

        # For additional filtering options see the help for get-mailbox

        # https://technet.microsoft.com/en-us/library/bb123685(v=exchg.150).aspx and for

        # where-object https://technet.microsoft.com/en-us/library/ee177028.aspx

$MbxRemaining = $mbxs.count

$low_bound = 0

while ($MbxRemaining -gt 0 ) {

    #   This loop creates one search per loop and decrements $MbxRemaining by the

    # number of mailboxes included in the search.

    $loop_count += 1

    $this_search = $base_search_name + $loop_count

    write-host "creating search: " $this_search -foregroundcolor cyan

    if($MbxRemaining -gt $mailboxesPerSearch) {

      #  There are at least $mailboxesPerSearch mailboxes left. 

      # This IF branch fills $search_list with $mailboxesPerSearch mailboxes.

      # Clear $search_list

      [array]$search_list = $NULL

      # Place mailboxes from the current value of low_bound to low_bound plus $mailboxesPerSearch

      #minus one in $search_list

      $mbxs[$low_bound..($mailboxesPerSearch * $loop_count -1)] | foreach {

         [array]$search_list=$search_list + [string]$_.UserPrincipalName

      }

     } else {

        # This branch fills $search_list with the mailboxes that are left when

        # there are less than $mailboxesPerSearch remaining.

        # Clear $search_list

        [array]$search_list = $NULL

        #Place mailboxes from the current value of low_bound to the end of the array in $search_list

        $mbxs[$low_bound..($mbxs.count -1)] | foreach {

         [array]$search_list=$search_list + [string]$_.UserPrincipalName

      }

    }

    #To create an In-Place Hold customize the line below to suit your needs

    #New-MailboxSearch -name $this_search -SourceMailboxes $search_list -InPlaceHoldEnabled $true -ItemHoldPeriod 730

    #To create an ordinary Discovery Search customize the line below to meet your needs

    New-MailboxSearch -name $this_search -SourceMailboxes $search_list -TargetMailbox $DiscoveryList[$DiscoveryLoop] -StartDate "12/31/2012" -SearchQuery "'10-K' OR '10K' OR '10k' OR '10k' OR 'annual report' OR 'fire' NEAR(15) ('insur*' OR 'pay*' OR 'claim*') OR 'fire' NEAR(20) 'insurance'"

    $DiscoveryLoop += 1

    if($DiscoveryLoop -ge $DiscoveryCount) { 

      #   If we have looped through all available Discovery Mailboxes reset the counter 

      # to the first element of the array.

      $DiscoveryLoop = 0

    }

    $low_bound = $mailboxesPerSearch * $loop_count

    $MbxRemaining -= $mailboxesPerSearch

}

If you are using the script above to place mailboxes on In-Place Hold I would suggest that you select a name the will cause the search to appear at the bottom of the list of searches.  The reason I suggest this is that the Search placing the mailbox on In-Place hold might be around a long time.  Prefixing the name with “z” or another character further down in the sort order will place all the In-Place Hold searches at the bottom of the list when you sort by name.

Some Discovery Search troubleshooting tips:

• Include fewer mailboxes in each search (modify the $mailboxesPerSearch variable in the script)
• Tighten the criteria applied to the get-mailbox cmdlet the script runs to make sure you are only including the mailboxes that truly need to be searched
• Cover a smaller date range in each search.  If you searched without a date range and encountered a timeout or other failure read the error message carefully for clues.  There have been instances in the past where specifying a start and end date allowed failing queries to complete
• Search during times when demands placed on the search servers are likely to be lower (this will vary by the region in which your mailboxes are hosted)
• Test skipping the "Include unsearchable items" and "enable de-duplication" options.  This will reduce the demands of your search and allow it to run faster.  If this test works try breaking the search into smaller units and run them individually with the required options.  If it still doesn't work you may need to discuss the search with Office 365 support.
• If you are exporting to PST make sure you have .Net 4.5 and its latest updates installed.

Above I mentioned scheduling your searches as a possible option.  If you want or need to run your search during a specific time window it is best to create a scheduled task to execute your PS1 file.  The PS1 file then logs on to the Office 365 environment and executes your commands.  Here is a sample script that will log on and start a pair of searches:

# Specify username and password.  The Password is stored in an encrypted text file in this example

$EXOAdmUser = “YourAccount@YourDomain.onmicrosoft.com” 

$pwd= Get-Content "c:\o365\password.txt" | ConvertTo-SecureString

$O365cred = New-Object System.Management.Automation.PSCredential $EXOAdmUser, $pwd

# Connect to Office 365

$session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://ps.outlook.com/powershell/" -Credential $O365Cred -Authentication Basic -AllowRedirection

Import-PSSession $session 

Start-MailboxSearch -identity "DiscoverySearch1" -force

Start-MailboxSearch -identity “DiscoverySearch2” -force

Exit

To create password.txt follow these steps:

• Log in to office 365 from a PowerShell session using the account that you will be using to create and execute Discovery Searches.  You can use these lines to do so:
  $O365Cred = (Get-Credential)
  $session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://ps.outlook.com/powershell/" -Credential $O365Cred -Authentication Basic -AllowRedirection
  Import-PSSession $session

• Run this command:  $o365cred.Password | ConvertFrom-SecureString | Set-Content c:\o365\password.txt

• Make sure password.txt is created.

To log in at the time you wish and start your script you would need to create a Scheduled Task.  In Scheduled Tasks you would use a command line like powershell.exe -Noninteractive -Noprofile -Command “&{<full path to your script>}”.

Start by following the link you were sent to the web page.  After this follow these screen shots:

Say Yes to the User Account Control Prompt.

 At some point in the future I might expand this to include the steps on the Exchange server.  However for the moment I will end here.

Event Type: Error
Event Source: MSExchangeIS
Event Category: Content Engine 
Event ID: 12800
Date: 1/27/2012
Time: 9:47:14 AM
User: N/A
Computer: Contoso1
Description:
Message processing failed because there is not enough available memory (8007000E-80000000). 

There are three basic courses that troubleshooting can follow here:

• Check for 9582 events.  If you have them start with these:  Collect performance data, make sure the updates from KB article https://support.microsoft.com/kb/960144 are applied, and follow the steps in https://support.microsoft.com/kb/325044
• If there are no 9582 events in your application log turn up Diagnostic logging on MSExchangeIS\Content Engine (this should generate 9684 events for bad calendar items)
• Use the CalCheck tool described in this blog:  https://blogs.technet.com/b/exchange/archive/2012/02/22/calcheck-the-outlook-calendar-checking-tool.aspx

The CalCheck tool has resolved a couple of these cases for me since it was released.  In these cases bad calendar items were blowing right through the limits that Exchange had imposed upon their conversion.  The conversion engine had only allocated a fixed amount of RAM and these items required more.  This caused the Out of Memory error to be reported.  When CalCheck examined these items it flagged them.  When the Administrator deleted them the 12800 events disappeared from the application logs of the Exchange Server.

Hopefully Randy's CalCheck tool will be equally effective for you if your server is encountering this event.  The tool should be good for locating bad calendar items whether you have the 9582 events or not.  In fact, once the bad calendar items are cleared out, it may lessen the 9582 events.  Just be careful not to assume that it will lessen or stop them.  The abbreviated steps I provided in the first bullet point are still your best bet if you are seeing 9582s.

I suggested increasing the size of the customer's cluster log, but to get the 72 hours Jeff recommends their cluster log was going to need to be huge.  We needed an alternative that would cover them if there was a problem on Friday night that nobody picked up until Monday morning.

The suggestion I gave them was simply to use the Windows 2008 Task Scheduler to generate their cluster logs.  Based on their scenario the last event in the sequence before the server began restarting services was Event ID 2137 from MSExchangeRepl.  Within Task Scheduler we followed these steps:

• Under Actions select Create Task
• Go to the Triggers tab and click New
• Change the Begin the Task selection to "On an event"
• We used Basic, selected the Application log, our source and the event ID
• We then chose to delay the task for 5 minutes (long enough for the services to restart) and clicked OK
• Go to the Actions tab and select New
• We chose to place our command to generate the cluster log (complete with all parameters) in a BAT file, so we specified that BAT file as the program to run and clicked OK.  For details on how to generate the cluster log please see this blog:  https://blogs.msdn.com/b/clustering/archive/2008/09/24/8962934.aspx
• On the General tab we gave our task a name and description
• We also selected "Run whether the user is logged on or not"

We left the task to run overnight and it successfully captured the cluster log for us.  The only real challenge for us was selecting an event that was consistently associated with their failure so that we would not miss our opportunity by generating a spurious log.  The MSExchangeRepl event worked well in this instance because it was unique.  I might have preferred using a 7031 from the System log, but the Task does not offer a chance to filter by the text in the description.  Therefore the 7031 event would not be unique and we might capture the wrong service failure.

Later we were able to agree on an increased size for their cluster log, but when I last spoke to them they were setting up the task just in case they overran their cluster log and an admin didn't respond to their alerts in time.

get-casmailbox -filter {OWAEnabled -eq $false}

In the output they saw numerous rows displayed where OWAEnabled was actually listed as $true.

They found that if they run the command below they get the correct result:

get-casmailbox | where-object {$_.owaenabled -eq $false}

So why the difference?  The OWAEnabled property is not actually assigned a default value.  When the get-casmailbox cmdlet is run it treats Not Set as $true when it displays the data.  The filter is returning everything that is False and that has not been set previously.

So how do you always get accurate results from the filter?  The only ways are to:

• explicitly define the value of OWAEnabled with the creation of each mailbox (easy to do if you use provisioning scripts, an extra manual step if you use the GUI)
• Run this command:
  get-casmailbox | where-object {$_.owaenabled -ne $false} | set-casmailbox -owaenabled $true
  This command will produce warnings for every user it touches, but does not need to change because the user has already been set to $true.

Hopefully this helps everyone get accurate results back from this cmdlet.  Have a good weekend!

In both cases the dumps were collected and they revealed that the store was busy working on an "empty" search.  What this means is that the store was working with the internal MAPI Search Folders that had been constructed for a folder that had too many items in it.

With Exchange 2010 the architecture has changed, but in Exchange 2007 and lower the store creates and maintains numerous MAPI Search Folders on each folder (also known as Restricted Search Views as outlined in this article https://technet.microsoft.com/en-us/library/cc535025(EXCHG.80).aspx).  There is a cap on the number, but until that cap is reached every unique Search or Sort that is done on the folder is tracked.  The resulting MAPI Search Folder is then updated with every change to the folder.  Typically the store maintains 3-5 MAPI Search Folders per folder, but if you are in Outlook (online mode) and change the Sort order to one Exchange did not already have built then Exchange will build a new MAPI Search Folder.  The database engine will then maintain that MAPI Search Folder until it expires (typically 40 days).

So you're probably thinking - Why is this bad?  Aren’t these just indexes?  That is what databases do.

Even if these were just indexes you would only be partially correct.  The issue comes in the number of MAPI Search Folders the database engine can end up maintaining and the resources it consumes to constantly update them.  Most databases do not dynamically create and destroy indexes.  If your IT staff or developers make you a SQL database they will optimize the indexes to give the greatest possible speed to the forms and queries that will be commonly used.  This allows them to account for performance considerations at design time.  Exchange and Outlook are creating most of these MAPI Search Folders on the fly.  This makes it impossible to foresee all performance scenarios.  In addition to this the Search Folder is more than an index.  It is actually a separate table that is linked back to the original.  Thinking of the MAPI Search Folders as an index is too much of a simplification.

When a new item is added to a folder and the Store begins updating the MAPI Search Folders those updates all go into a single transaction.  If there are 30 MAPI Search Folders to update then all of the pages that will be dirtied get loaded into the Exchange Version Store as part of a single transaction.  That causes the consumption of Version Buckets to rise (as seen in the counter "\Database(Information store)\Version buckets allocated" ).  The number of items influences how long it takes to traverse each MAPI Search Folder and carry out all the proper page splits, writes, etc. for the update.  If there are a lot of MAPI Search Folders that are very large they will each take longer to load, examine, and update.  Therefore pages in the Version Store will be held longer.  The longer those pages are held the greater the chance we will have a 623 event occur.

Now if this is just one mailbox that is over by a little bit this probably won't hurt and won't cause a problem.  However, if you have numerous mailboxes with high item counts and the users are active in the folders with the higher counts, then the cumulative effect can snowball into a 623 event.

By default Exchange only maintains a maximum of 11 Restricted Search Views per folder and updates each for 40 days (after which, if it hasn't been used in 40 days, it is removed).  However if an online mode client has multiple regional settings this can turn into one set of 11 per language.  Also Administrators can raise the number from 11 to whatever they like (a max of 50 is recommended in https://technet.microsoft.com/en-us/library/cc535025(EXCHG.80).aspx).  Raising the number from 11 can be desirable because you can prevent dynamically created Views from being bumped out of the list and replaced by more recently requested views (if users are constantly causing turnover between views it can cause other performance problems).

You can get short term relief from running Outlook /cleanviews.  However, because the views are created dynamically you can very quickly return to the problem.  In the relatively rare case where this problem is occurring it would be better to reduce item counts to avoid the problem.  You can check item counts with the steps outlined in this blog:   https://blogs.technet.com/b/exchange/archive/2009/12/07/3408973.aspx