Search and Replace Retention tag on Microsoft Exchange 2010 (MRM)
Messaging Records Management
development on Retention Policy Tag
Introduction
Microsoft introduces the messaging record management (MRM). In Exchange 2010, which helps the organizations to reduce legal risks associated with e-mail and other communications.
MRM makes it easier to keep messages needed to comply with company policy, government regulations, or legal needs, and to remove content that has no legal or business value.
To learn more about MRM and how to set it up, please visit https://technet.microsoft.com/en-us/library/dd297955.aspx
In this article, will go over what are retention policy tags? How it is being internally stored in MAPI objects? How programmatically change the retention policy tags in each item of the exchange.
What are retention policy tags?
MRM in Exchange 2010 is accomplished by using retention tags and retention policies. Before discussing the details about each of these retention features, it's important to learn how the features are used in the overall Exchange 2010 MRM strategy. This strategy is based on:
- Assigning retention policy tags (RPTs) to default folders, such as the Inbox
- Applying a default policy tag (DPT) to mailboxes to manage the retention of all untagged items
- Allowing the user to assign personal tags to custom folders and individual items
- Separating MRM functionality from users' Inbox management and filing habits.
- Users
aren't required to file messages in managed folders based on retention
requirements - Individual
messages can have a different retention tag than the one applied to the folder
in which they're located
- Users
Figure 1 illustrates
the tasks involved in implementing this strategy
Figure 1
Ref: https://technet.microsoft.com/en-us/library/dd297955.aspx
Retention Tags Types (Reference: https://technet.microsoft.com/en-us/library/dd297955.aspx)
As illustrated in Figure 1, retention tags are used to apply retention settings to folders and individual items such as messages, notes, and contacts. These settings specify what policy a message should have, how long a message remains in a mailbox and the action to be taken when the message reaches the specified retention age. When a message reaches its retention age, it's moved to the personal archive, deleted, or flagged for user attention.
Unlike managed folders (the MRM feature introduced in Exchange 2007), retention tags allow users to tag mailbox folders and individual items for retention. Users no longer have to file items in managed folders based on message retention requirements.
Retention tag configuration objects are stored in Active Directory (AD) in the Retention Policy Tag Container, which is located under the organization container. There are three types of retention tags:
- Retention Policy Tags
- Default Policy Tags
- Personal Tags
Retention Policy Tags (RPTs)
RPTs apply retention settings to default folders such as Inbox, Deleted Items, and Sent Items. Mailbox items in a default folder that have an RPT applied inherit the folder's tag. Although users can't apply a different tag to a default folder, they can apply a different tag to the items in a default folder.
You can create RPTs for the following default folders:
Calendar (requires Exchange 2010 SP1) |
Notes (requires Exchange 2010 SP1) |
Deleted Items |
Outbox |
Drafts |
Sent Items |
Inbox |
RSS Feeds |
Journal |
Sync Issues |
Junk E-mail |
Conversation History |
Important:
You can't include more than one RPT for the same default folder type in one retention policy. For example, if a retention policy has an Inbox tag, you can't add another RPT of type Inbox to that retention policy.
In Exchange 2010 RTM, RPTs are not supported for the Calendar and Notes default folders. In Exchange 2010 RTM and SP1, retention tags are not applied to Contacts.
Default Policy Tags (DPTs)
DPTs apply retention settings to untagged mailbox items. Untagged items are mailbox items that do not already have a retention tag applied, either by inheritance from the folder in which they are located or by the user. A retention policy cannot contain more than one DPT with the same action.
Personal Tags
Personal tags are available to Outlook 2010 and Outlook Web App users as part of their retention policy. Users can apply personal tags to folders they create or to individual items, even if those items already have a different tag applied.
Retention Policy Tag Attributes
MRM depends on the Active Directory (AD) directory to store configuration object settings used for driving Managed Folder assistant provisioning and retention actions. These configuration settings are retrieved and set using Exchange management tasks from EMS, EMC and ECP
Retention tag configuration objects are of objectClass msExchELCFolder and are located in the Retention Policy Tag Container directly under the Organization container. msExchgELCFlags, and msExchELCFolderTypes are the attributes used by MRM. For each retention tag there is a Content
Settings configuration object of objectClass msExchELCContentSettings that is a direct child object of the retention tag configuration object. Table 1 describes the attributes used by MRM.
Table 1: Content Setting Configuration Attributes
Attribute
msExchELCAutoCopyAddressLink
msExchELCExpiryAction
msExchELCExpiryAgeLimit
msExchELCFlags
msExchELCLabel
msExchELCMessageClass
Retention Policy Attributes
Retention policy configuration objects are of objectClass msExchRecipientTemplate and are located in the Retention Policies Container directly under the Organization container. Table 2 describes the attributes used by MRM.
Table 3: Retention Policy Configuration Attributes
Attribute
msExchELCFolderLink
msExchMailboxTemplateBL
msExchMinAdminVersion
msExchRecipientTemplateFlags
Mailbox Objects
User’s mailbox contains Information specific to MRM operation. This information is used by the Managed Folder Assistant for processing retention and archival actions, and by mailbox clients for displaying and managing relevant retention and archival elements in the user interface.
Configuration Message
The MRM configuration information for a mailbox to which a retention policy has been applied is stored in a hidden Folder Associated Item (FAI) message of type IPM.Configuration.MRM, located in the Associated Contents table of the Inbox folder. This message in the mailbox is not visible in Outlook or OWA to the end user. MRM uses the FAI item to communicate the Retention Policy to the mailbox client, and for processing retention tags that have been self-provisioned by the user.
The configuration information is stored as XML formatted data in property PR_ROAMING_XMLSTREAM (0x7C080102). This information can be examined using the MAPI Editor tool (MFCMAPI).
Important:
Once a retention tag is applied to a mailbox and the information for that tag is written to the FAI message, the information remains even if the tag is un-provisioned by the user or the tag is no longer applied by policy. When this occurs the IsVisible attribute is set to False so the user no longer sees the tag from the client.
However, the Managed Folder Assistant continues to process any messages that are tagged with the hidden retention tag as long as the configuration object for the tag is still available in AD. When the tag is removed from AD, the information for the tag is also removed from all FAI messages by the Managed Folder Assistant.
Updating the Configuration Message
The configuration information is updated by one of three
ways:
- Managed Folder Assistant – The Managed Folder Assistant creates the configuration message when a retention policy is first applied to the mailbox, and updates the message anytime changes are made to the retention tags or retention policy applied to the mailbox.
- Set-RetentionPolicyTag – This cmdlet makes it possible for the administrator to assign additional retention tags that are not already included in the retention policy applied to the mailbox. When the administrator adds or removes retention tags via this command, the configuration message is updated.
- Self-Provisioning of Retention Policies via ECP – When a user adds or removes retention tags via the ECP, the configuration message is updated. The administrator can retrieve a list of all retention policies applied to a mailbox by policy assignment, administrative assignment or self-provisioning by using the Get-RetentionPolicyTag cmdlet with the Mailbox parameter. The command reads directly from the configuration message to retrieve the current settings. Because of this the mailbox must be available for this operation to succeed.
Folder Properties
MRM stores retention settings as MAPI property values on mailbox folders. Table 4 describes these property values.
Table 4: Retention Setting Properties on Mailbox Folders
Property Name
propTagID
Type
Stamping Details
StartDateEtc
PR_START_DATE_ETC
0x30190102
GUID
The GUID of the retention tag applied to the folder. Stamped on all folders. Outlook changes this property when the user explicitly tags a folder. Outlook updates all the subfolders with this tag’s GUID as well.
RetentionPeriod
PR_RETENTION_PERIOD
0x301A0003
Number
Of Days
The retention period for keeping the item (if it’s a value like 0 or -1 it means never expire). Stamped on all folders. Outlook changes this property when the user explicitly tags a folder. Outlook updates all the subfolders with the retention period as well.
RetentionFlags
PR_RETENTION_FLAGS
0x301D0003
Number
The property representing if the recipient tag is inherited from a parent folder. Outlook stamps the property in offline/cached mode, Exchange stamps for online mode. If the user tags a folder, RetentionFlags is 0 for each sub-folder and non-zero for the folder that is explicitly tagged. If the least significant bit is 0 or if the property is not present, then the tag is implicit.
See RetentionFlags table for details.
Item Properties
MRM stores retention settings as MAPI property values on mailbox items. Table 5 describes these property values.
Table 5: Retention Setting Properties on Items
Property Name
propTagID
Type
Stamping Details
StartDateEtc
PR_START_DATE_ETC
0x301B0102
DateTime
This is a composite prop containing the Start Time and Default Retention Period. The first 4 bytes is the default retention period and the next 8 bytes is the date stamped on every item. Exchange Event-Based MRM Assistant sets this property. Outlook changes if necessary (Calendar and Task Item) Add the policy length to the start time to calculate expiry date.
RetentionPeriod
PR_RETENTION_PERIOD
0x301A0003
Number
(In Days)
The time period for keeping the item (if it’s a value like 0 or -1 it means never expire). Stamped only when item is explicitly tagged. Outlook changes the value when explicitly tagged.
DefaultRetentionPeriod
<same as Start Time>
Number
The length to retain an item if it is under the default policy (varies based on message class). Stamped on every item by Exchange Event-Based/Time-Based Assistant. Never stamped by Outlook.
RetentionDate
PR_RETENTION_DATE
0x301C0040
DateTime
DateTime for item expiration. This is a calculated prop when cached or offline and a property stamped by Exchange when online. Stamped on every item. Exchange stamps when online, Outlook calculates when cached or offline.
PolicyTag
PR_POLICY_TAG
0x30190102
GUID
The retention policy an item is under (either implicit or explicit). Stamped on every item. Outlook calculates when cached or offline. Exchange stamps when online.
RetentionFlags
PR_RETENTION_FLAGS
0x301D0003
Number
The property representing if the recipient tag is inherited from a parent folder. Outlook stamps the property in offline/cached mode, Exchange stamps for online mode. If the user tags an item, RetentionFlags is non-zero for the item that is explicitly tagged. If the least significant bit is 0 or if the property is not present, then the tag is implicit.
See RetentionFlags table for details.
ArchiveTag
PR_ARCHIVE_TAG
0x30180102
GUID
The archive policy an item is under (either implicit or explicit). Stamped on every item. Outlook calculates when cached or offline. Exchange stamps when online.
ArchivePeriod
PR_ARCHIVE_PERIOD
0x301E0003
Number
(In Days)
The time period for archiving the item (if it’s a value like 0 or -1 it means never expire). Stamped only when item is explicitly tagged. Outlook changes the value when explicitly tagged.
ArchiveDate
PR_ARCHIVE_DATE
0x301F0040
DateTime
DateTime for item archival. This is a calculated prop when cached or offline and a property stamped by Exchange when online. Stamped on every item. Exchange stamps when online, Outlook calculates when cached or offline.
Table 5 describes the values used for the RetentionFlags property on mailbox folders and items.
Table 5: RetentionFlags Bitmask Values
RetentionFlags
Bitmask
Hex
Dec
None
00000000 00000000
0x00
0
ExplicitTag
00000000 00000001
0x01
1
UserOverride
00000000 00000010
0x02
2
Autotag
00000000 00000100
0x04
4
PersonalTag
00000000 00001000
0x08
8
AllRetentionFlags
00000000 00001111
0x0F
15
ExplictArchiveTag
00000000 00010000
0x10
16
KeepInPlace
00000000 00100000
0x20
32
AllArchiveFlags
00000000 00110000
0x30
48
SystemData
00000000 01000000
0x40
64
NeedsRescan
00000000 10000000
0x80
128
PendingRescan
00000001 00000000
0x100
256
Exchange Web Service
Microsoft Exchange Server 2010 provides Exchange Web Services as an extensibility point for clients that connect to the Exchange server and consume information about user availability, and the manipulation of items that are located in the Exchange data store.
To learn more about Exchange web service please click here https://msdn.microsoft.com/en-us/library/bb204119.aspx
Programmatically change the retention policy tag on the Exchange
Business Scenario
For my current customer, we had a scenario, using PowerShell the admin needs to
- Search for all items which have a particular retention tag assigned AND belong to a specified policy
- Specify the new retention tag in that command and have it assigned to the found mail items
We don’t have out-of-box functionality in Exchange 2010 SP1 for the above requirement. Hence, I have to do custom coding.
Solution
The custom script created as a framework. Using this script admin can search and replace the retention policy tag. Also, just by modifying the script, admin can search on how many users having certain policy tags, get the policy tag for the specific user, get the policy tag for the specific folder, and get the expiration date.
Below is the conceptual architecture, and code.
Pre-requisite
- Only admin can run this utility as it needs impersonation (to search on other mailboxes)
- PowerShell 2.0
- Exchange 2010 SP2 Web Service SDK
- Replaceable tag must be part of the same retention policy.
Conceptual Architecture
Figure 2
As shown in the Figure 2, PowerShell will read the users from the UserAccounts.txt file. Admin will be providing the description of the retention policy tag, but internally MAPI stores only the GUID of the tag. Hence, there will be a look-up function to match the tag description with the GUID, for both search and replace tags. Then
the Exchange Web Service (EWS) will be invoked to impersonate each user’s mailbox and search all the folders. Upon search, if match found the GUID will be replaced with the replace tag GUID provided by the admin. Report of the operations will be written in the SharePoint.
Code
#This Script is to replace the retention policyTag
#Parmamters
param($args1, $args2)
$SearchIdTag = $args1
$ReplaceTag = $args2
#Lookup Function to get the GUID of the retention policy tag description
Function ConvertIdentityToGuid($IdTag)
{
#Remote Session to get the exact GUID and description of the PolicyTag
#$userCredential = get-Credential
$username = "DOMAIN\SuperUser"
$password = convertTo-SecureString "password" -AsPlainText -Force
$cred = New-Object System.Management.Automation.PSCredential($username, $password)
$session=New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://CasServer/PowerShell/ -Credential $cred
Import-PSSession $session -AllowClobber
$GuidTag = Get-RetentionPolicyTag -Identity $IdTag | Select GUID
$Tag=$GuidTag.Guid
$GuidPolicyTag = $Tag
get-Pssession |remove-Pssession
Write-output $GuidPolicyTag
}
#Replace the tag
Function ReplacePolicyTagOnItem($item)
{
$GuidReplacePolicyTagArray = ConvertIdentityToGuid($ReplaceTag);
$ReplaceGuidTag = $GuidReplacePolicyTagArray[1]
#PR_POLICY_TAG 0x3019
$PolicyTag = New-Object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(0x3019,[Microsoft.Exchange.WebServices.Data.MapiPropertyType]::Binary)
$PolicyTagGUID = New-Object GUID($ReplaceGuidTag)
$enumAlwaysOverWrite = [Microsoft.Exchange.WebServices.Data.ConflictResolutionMode]::AlwaysOverWrite
$item.SetExtendedProperty($PolicyTag,$PolicyTagGuid.ToByteArray())
$item.Update($enumAlwaysOverWrite)
#Write the report to SharePoint List
$WriteToSP = $global:AuditLog.WriteAuditLog("RetentionTagReplaced",$MailboxName,$item.Subject ,"User" ,$global:FolderName , $args1, $args2, "Item", "")
write-host "SharePoint: " $WriteToSP
}
#Search Function
Function SearchPolicyTagOnItem($SearchTag)
{
$fvItemView = New-Object Microsoft.Exchange.WebServices.Data.ItemView(10000)
$ItmpsPropertySet = new-object Microsoft.Exchange.WebServices.Data.PropertySet([Microsoft.Exchange.WebServices.Data.BasePropertySet]::FirstClassProperties)
#PR_POLICY_TAG 0x3019
$ItmPolicyTag = New-Object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(0x3019,[Microsoft.Exchange.WebServices.Data.MapiPropertyType]::Binary)
$ItmpsPropertySet.Add($ItmPolicyTag)
$fvItemView.PropertySet = $ItmpsPropertySet;
#Loop through the folders
while (($results = $ffFolder.FindItems($fvItemView)).Items.Count -gt 0)
{
#Loop through each item in the folder
foreach ($item in $results)
{
$ItemPolicyGuid=$null;
if ($item.TryGetProperty($ItmPolicyTag,[ref] $ItemPolicyGuid))
{
$Itmptag = [GUID]$ItemPolicyGuid
if ($SearchTag -eq ($Itmptag))
{
"Item Subject: " + $item.Subject
"Item Retention Policy Tag:" +$Itmptag
ReplacePolicyTagOnItem($item)
}
}
}
$offset += $results.Items.Count
$fvItemView = new-object Microsoft.Exchange.WebServices.Data.ItemView(100, $offset)
}
}
# Script starts here
#Global declaration
$global:AuditLog = New-Object RetentionTagAuditLog.AuditLog
$global:FolderName = $null
#Get the GUID of the provided retention policy tag description
$GuidPolicyTagArray = ConvertIdentityToGuid($SearchIdTag);
#SharePoint assembly referenced for Reports
[System.Reflection.Assembly]::LoadFrom("G:\PowerShell\RetentionTagAuditLog.dll")
#Read data from the UserAccounts.txt.
#Make sure this file exists in the same location as the script or provide the path
import-csv AliasAccounts.txt | foreach-object
{
$MailboxName = $_.EmailAddress.ToString()
#Load the Exchange Web Service
$dllpath = "C:\Program Files\Microsoft\Exchange\Web Services\1.1\Microsoft.Exchange.WebServices.dll"
[void][Reflection.Assembly]::LoadFile($dllpath)
$service = New-Object Microsoft.Exchange.WebServices.Data.ExchangeService([Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2010_SP1)
#imepersonation - need to have admin rights
$service.ImpersonatedUserId = new-object Microsoft.Exchange.WebServices.Data.ImpersonatedUserId([Microsoft.Exchange.WebServices.Data.ConnectingIdType]::SmtpAddress,$MailboxName);
#Get current user
$windowsIdentity = [System.Security.Principal.WindowsIdentity]::GetCurrent()
$sidbind = "LDAP://<SID=" + $windowsIdentity.user.Value.ToString() + ">"
$aceuser = [ADSI]$sidbind
$service.AutodiscoverUrl($aceuser.mail.ToString(),{$true})
$GuidPolicyTag = $GuidPolicyTagArray[1]
"Checking : " + $MailboxName
$folderidcnt = new-object Microsoft.Exchange.WebServices.Data.FolderId([Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::Root,$MailboxName)
$fvFolderView = New-Object Microsoft.Exchange.WebServices.Data.FolderView(10000)
$fvFolderView.Traversal = [Microsoft.Exchange.WebServices.Data.FolderTraversal]::Deep;
$psPropertySet = new-object Microsoft.Exchange.WebServices.Data.PropertySet([Microsoft.Exchange.WebServices.Data.BasePropertySet]::FirstClassProperties)
#PR_POLICY_TAG 0x3019
$PolicyTag = New-Object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(0x3019,[Microsoft.Exchange.WebServices.Data.MapiPropertyType]::Binary)
$psPropertySet.Add($PolicyTag)
$fvFolderView.PropertySet = $psPropertySet;
$fiResult = $Service.FindFolders($folderidcnt,$fvFolderView)
#Loop through each folder in the mailbox
foreach($ffFolder in $fiResult.Folders)
{
$PolicyGuid=$null;
if ($ffFolder.TryGetProperty($PolicyTag,[ref] $PolicyGuid))
{
$ptag = [GUID]$PolicyGuid
if ($ptag -eq $GuidPolicyTag)
{
$global:FolderName = $ffFolder.DisplayName
$WriteToSP = $global:AuditLog.WriteAuditLog("RetentionTagReplace",$MailboxName,"" ,"User" , $global:FolderName, $args1, $args2, "Folder", "")
}
}
"Traversing folder: " + $ffFolder.DisplayName
$global:FolderName = $ffFolder.DisplayName
SearchPolicyTagOnItem($GuidPolicyTag)
}
# catch excption
trap [System.Exception]
{
write-host ("Error: " +$_.Exception.Message)
#write in log file
continue
}
}
Screen Shots
1. Email with a retention policy tag
2. Mapi Properties of the message item. Please see the PR_Policy_Tag
3. Retention Policy Tag MAPI properties
4. Run the script
5. After running the script
6. Verify the MAPI Properties
7. The Retention Policy tag value has been changed
Reference
Comments
Anonymous
January 01, 2003
Looks like a false property name for in propTagID 0x30190102. Should be PR_POLICY_TAG.Anonymous
January 01, 2003
This is easily one of my top 2 blogs on EWS API, and Exchange Retention. If you have managers who are asking for you to start shifting retention policies around, you have come to the right place. Sundar will have you talking with managers and admins authoritatively on Exchange Retention in minutes with diagrams and the key values listed out for you. I was desperate for something more than 4 lines of code snippets that were in all of my books and Sundar delivered an enterprise solution. This is far and above pulling subject lines from inbox mails like most of the other blogs. Great job!Anonymous
November 15, 2011
The comment has been removedAnonymous
January 15, 2014
As the Exchange Retention and Archiving Policies is a topic many people run from, I think it is alwaysAnonymous
August 08, 2014
Great Script! Where do I pass the 'search' and 'replace' policy tags? Help?Anonymous
September 01, 2015
Superb Article...great detailed info