Eric Norberg @ Microsoft

E2K7: Creating Custom SMTP E-mail Address Policies Utilizing Various Replacement String Combinations

ericnor — Thu, 30 Apr 2009 17:56:00 GMT

I have recently received a number of questions surrounding the generation of e-mail addresses in Exchange 2007. Particularly, whether or not custom email addresses can be generated via “legacy” replacement strings to meet specific organizational naming conventions.

This post will present a brief overview of the process as well as offer numerous examples which should assist you when tasked with creating your own custom proxy address templates in Exchange 2007.

Legacy Behavior:

In previous versions of Exchange, administrators would commonly use variables and replacement strings within SMTP Recipient Policies to generate specific email addresses for their mail enabled users. Specifically they would use these strings and variables to generate the local part of a user’s SMTP address. Once the Recipient Policies were created and applied, the Recipient Update Service would “stamp” the proper email addresses on the objects based upon an LDAP Recipient Filter.

The most common variables and replacement strings are documented in MSKB 822447 (https://support.microsoft.com/kb/822447).

Exchange System Manager (ESM) itself did not provide a GUI to assist with the creation of the various combinations which could potentially be generated. As such, the Administrator(s) had to have detailed knowledge of the replacement strings as well as their possible uses in order to successfully generate the SMTP addresses for mail enabled objects. Once the replacement strings were added to the policy, the Recipient Update Service was then responsible for “stamping” the objects with the custom addresses based upon an LDAP filtering syntax (stored on the purportedSearch attribute of the Recipient Policy).

Exchange 2007 Behavior:

Today, Exchange 2007 offers a Graphical Interface within the Exchange Management Console (EMC) to help an administrator generate email addresses based upon the most common variable combinations. To create an E-Mail Address Policy via the EMC an Administrator performs the following:

· Opens the EMC.

· Navigates to the Organization Configuration, Hub Transport Node.

· Switches to the E-Mail Address Policy Tab.

· Provides a Name for the Policy and selects what type of Recipient Objects the policy will be responsible for as well as the conditions for how the policy will be applied.

· Once these basic pieces of information have been provided, the Administrator eventually arrives at the E-Mail Addresses option page.

On this page, an Administrator needs to define how the local part of the address will appear. As noted above, there are a number of pre-canned options to use for the alias. These options are provided in the following screen shot:

What should immediately stand out as a primary difference with that of legacy Exchange Recipient Policies is that there is no way to actually edit the alias generation portion of the address in the GUI. In short you can only select one of the pre-canned options and lack the ability to further modify the local part of the address.

In the example above, I have elected to generate based upon first initial of First Name (Given Name) and the full Last Name or Surname (e.g. for a user named “Test User” the user portion of the address would be “tuser”). Once I specify the FQDN for the address (or select a valid Accepted Domain) and select OK, the replacement variables become visible on the E-Mail Addresses portion of the policy (although I can’t further edit them, if you select “Edit” you will simply be returned to the pre-canned options page):

This presents a problem if as an Administrator I had been tasked with generating the user portion of the addresses in the form of:

· First 3 letters of the First Name (e.g. the Given Name)

· .

· Full Last Name (e.g. the Surname)@en164420.lab.

In this instance, I would want a replacement string in the form of:

%3g.%s.@en164420.lab

This string once successfully applied would render a mail enabled object named Test User with an SMTP address in the form of: Tes.User@en164420.lab (which would accomplish precisely what I had been tasked with).

However as evident from above, there is no pre-canned option for the generation of this type of address nor do I have any ability to directly modify it within the Exchange Management Console (e.g. via a Custom Address, etc.).

To accomplish the goal I essentially have 2 options:

· Keep an Exchange 2003 Server around so that custom Recipient Policies can be created, modified and applied by the Recipient Update Service (not a very attractive option).

· Build or modify my policies via the Exchange Management Shell.

Using SMTP Replacement Strings in Exchange Management Shell:

The Exchange Management Shell can be utilized to define more sophisticated address templates. Once created the syntax is fully visible in the Management Console (it’s just not editable).

Administrators new to Exchange 2007 also need to understand that E-mail Address Policies natively created in Exchange 2007 (or legacy polices that have had their filter syntax upgraded) utilize the new OPATH filtering syntax as opposed to LDAP. Although there is a bit of a learning curve the primary benefit is that the filters themselves are far easier to create and interpret than those created with LDAP. Legacy Recipient Filters that were created in previous versions of Exchange still function it is just not possible to edit those policies from within the EMC until they have been upgraded to OPATH.

To create a new E-mail Address Policy the New-EmailAddressPolicy cmdlet is utilized within the shell: https://technet.microsoft.com/en-us/library/aa996800.aspx

To modify an existing E-mail Address Policy the Set-EmailAddressPolicy is utilized within the shell: https://technet.microsoft.com/en-us/library/bb124517.aspx

In my previous example, I wanted to generate the user portion of the SMTP utilizing the following naming convention :

First 3 Letters of First name.Full Last Name@domain.com (the name of my lab domain is en164420.lab)

Example: Tes.User@en164420.lab

To create a new policy with this alias convention involves the following steps:

1. Deciding on a Name for the new E-mail Address Policy.

2. Deciding on the Recipient Types I want the policy to apply to (e.g. Users with Exchange Mailboxes, Resource Mailboxes, etc.).

3. Filtering Syntax (e.g. who do I want the policy to apply to)?

4. The format of the alias (e.g. what SMTP Replacement Strings to utilize).

5. Whether the address will apply to an Accepted Domain or an FQDN.

6. Actually creating the E-mail Addressing Policy via Exchange Management Shell.

For the purpose of this example, I’m going to create a new policy named “Test” that applies to all Recipient Types via an OPATH filter utilizing the %g and %s replacement variables.

· Open Exchange Management Shell.

· Issue following command:

New-EmailAddressPolicy -Name “Test” –IncludeRecipients AllRecipients –EnabledPrimarySMTPAddressTemplate “SMTP: %3g.%s.@en164420.lab”

· ENTER

Note: In this specific example, I used the “-IncludeRecipients” parameter as opposed to creating a conditional filter via the –RecipientFilter parameter. The –RecipientFilter parameter is very useful for creating Policies that filter on specific properties such as Company, Country Code, a Custom Attribute, a Database, Department, etc). I will provide some examples at the end of the post using conditional clauses with the –RecipientFilter parameter.

If your command completed successfully, you can now verify it by running Get-EmailAddressPolicy “Test” in Shell or going into the EMC and verifying that exists within the Organizational Configuration Node for Hub Transport.

Shell:

Exchange Management Console:

The actual replacement string is now available in both the EMC and EMS:

Exchange Management Console:

Exchange Management Shell:

Now to actually test that the policy applies an SMTP address in the proper format we are going to create a user called Test User and then Mailbox-Enable the object.

In Active Directory Users and Computers (ADUC), I created a user account with a First Name of “Test” and a last name of “User” (the user object itself has no mailbox):

I then Mailbox Enable “Test User”:

Enable-Mailbox –Identity “Test User” -Alias “tuser” -Database “MyServer\First Storage Group\Mailbox Database”

Next verify that the user portion of the address is accurate:

Exchange Management Console:

Exchange Management Shell:

Conclusion:

SMTP Replacement Strings can be utilized in Exchange 2007 just as in previous versions of Exchange. However, there is no editable GUI interface for them in Exchange 2007. Thus, to successfully generate aliases based upon replacement strings an administrator needs to utilize the Exchange Management Shell.

Additional Examples:

Example 1:

This example will create an email-address policy where the user portion of the SMTP address is in the form of FirstName.LastName(e.g. Eric.Norberg) where the domain is en164420.lab. It will also only apply the address to users who are on an Exchange server named “Executives” that have a title of “Executive”:

New-EmailAddressPolicy -Name "Executives" -RecipientFilter { ServerName -eq "Executives" -and Title -eq "Executive" } -EnabledPrimarySMTPAddressTemplate "SMTP:%g.%s.@en164420.lab"

Example 2:

This example will create an E-mail Address policy where the user portion of the SMTP Address is in the form of first 3 letters of First Name.Last Name (e.g. eri.norberg@en164420). It will also force that all characters in the user portion of the address be in lower case (Example: If you had a user named “John Doe”, the address would not be Joh.Doe but joh.doe@en16442.lab). The policy would also only apply to objects that had an associated mailbox and had the Company value set to “Microsoft”:

New-EmailAddressPolicy -Name "Global" -EnabledEmailAddressTemplate `SMTP: %rAa%rBb%rCc%rDd%rEe%rFf%rGg%rHh%rIi%rJj%rKk%rLl%rMm%rNn%rOo%rPp%rQq%rRr%rSs%rTt%rUu%rVv%rWw%rXx%rYy%rZz%3g.%s.@en164420.lab' -RecipientFilter {((RecipientType -eq 'UserMailbox') -and (Company -eq 'Microsoft'))}

Example 3

This example modifies the SMTP Template on a policy named "Test". It simply adjusts the SMTP user portion of the address template to display name by utilizing the %d replacment string:

Set-EmailAddressPolicy -Name "Test" -EnabledPrimarySMTPTemplate 'SMTP: %d.@en164420.lab'

Example 4:

This example creates a new policy titled "Redmond Users". The policy applies to user objects with mailboxes that exist in the state of Washington "WA" who additionally have the otherTelephone attribute set to an area code beginning with 425. The user portion of the address is simply being stamped as alias.

New-EmailAddressPolicy -Name "Redmond Users" -EnabledPrimarySMTPAddressTemplate 'SMTP: @en164420.lab' -RecipientFilter { RecipientType -eq 'UserMailbox' -and StateorProvince -eq "WA" -and OtherTelephone -like '425*' }

Additional Reading:

Understanding E-Mail Address Policies: https://technet.microsoft.com/en-us/library/bb232171.aspx
New-Email Address Policy Wizard / Custom Address: https://technet.microsoft.com/en-us/library/aa996030.aspx
Managing E-Mail Address Policies: https://technet.microsoft.com/en-us/library/aa998940.aspx
Filterable Properties for the -RecipientFilter Parameter: https://technet.microsoft.com/en-us/library/bb430771.aspx
New-EmailAddressPolicy: https://technet.microsoft.com/en-us/library/aa996800.aspx
Set-EmailAddressPolicy: https://technet.microsoft.com/en-us/library/bb124517.aspx
Update-EmailAddressPolicy: https://technet.microsoft.com/en-us/library/aa996869.aspx
Upgrading Custom LDAP Filters to OPATH: https://technet.microsoft.com/en-us/library/cc164375.aspx

How to Remove an Invisible or Corrupted "Out of Facility" message with MFCMAPI

ericnor — Sat, 07 Feb 2009 17:35:00 GMT

This article is going to address how to deal with 1-off issues surrounding "Out of Facility" responses in Outlook. Over the years I have seen many strange cases surrounding OOF responses. However, generally speaking they fell into one of the following categories:

A user compains that although he/she has composed and activated an OOF response within Outlook no OOF message is being sent.
The wrong or duplicate OOF messages are being sent.
An extremely old OOF response (invisible to the end user in Outlook) is being sent.

In my experience there have been lots of these sorts of issues and typically the "root cause" turns out to be minor corruption of the OOF message within the Outlook Inbox. "Usually", these issues boil down to how and when the original OOF message was created (e.g. the Inbox Assistant, Rules Wizard, OWA, etc.), versions of Exchange Server that have or "may have" been in existance at a company (and hence how many times the mailbox itself has been moved between different versions of Exchange).

It is not the intention of this article to provide a "slash and burn" approach to dealing with these sorts of issues. I would not recommend you begin your troubleshooting here. In short, it is my expectation that you have done your dillegence and have arrived here as a "last resort" or sorts. I would like to emphasize that if you do not feel comfortable with any of the steps I have listed here or if you have further questions/concerns that you open an issue with Microsoft Product Support. The engineer who assists you will be able to provide you with a detailed analysis, action items, and summary for your specific issue. Perhaps you might even reach me.

MFCMAPI

A thorough discussion of MFCMAPI is well outside the scope of this entry. There is a tremendous amount of public information about MFCMAPI for consumption. However the short story is that MFCMAPI is a graphical MAPI editior (much like Outlook) that allows a user (with sufficent permissions) the ability to access and modify items within a mailbox or public folder. It is commonly used within Microsoft Product Support to facilitate investigations of "strange" issues between Exchange and Outlook. It does have a wide range of applications but most commonly it is used to perform one of the following operations:

Delete Corrupted Mail or Public/System Folder Items that cannot be removed via conventional means (Outlook, OWA, etc.).
To clear (or "cleanse", as I like to say), the Exhcange Temp Tables on a given server.
Inspection of Hidden Folders present within a mailbox or public folder.
Troubleshooting Replication of Individaul Free Busy messages (this will be the subject of a future blog entry).
Removal of a corrupt rule.

Today we will focus on removing a corrupt Inbox OOF rule, although the steps presented here would be just as applicable to other rules with minor exceptions with respect to the Message Class of the rule, where it might be located, et cetera.

Hypothetical Issue:

A user contacts you (the administrator), because they are experiencing problems where an old OOF response if being sent out once the "Send Out of Office auto-replies" has been enabled within the Out of Office Assistant.

As an administrator you have done your dilligence and cannot locate this OOF response anywhere (in Rules and Alerts, within the OOF Assistant, et cetera). As such, you decide to further investigate with MFCMAPI.

Steps:

1. Prior to performing any editing you will want to take a valid export of all rules configured within the user's profile (Tools, Rules and Alerts, Options, Export Rules, give the export a name (file extension should be .rwz), and click OK. Verify the file has been created and that it is valid -- to really test, you could important the rules into a dummy mailbox and verify they are valid.

2. It would also be helpful to have a copy of the incorrect OOF message being sent (with the OOF Assistant enabled for the impacted user, send a test message and verify that you receive the invalid response).

3. Turn off the OOF Assistant if it currently enabled then close Outlook.

4. You will need to have a valid MAPI profile to use MFCMAPI. So you can download MFCMAPI to the user’s desktop (where Outlook is installed), or create a new MAPI profile for the user.

5. Start MFCMAPI and from the “Session Menu” select “Logon and Display Stores”, and choose the proper Outlook profile.

6. Open the Mailbox of the profile that you want to inspect or “delete” the OOF rule for (this can be done either via a double-click on the mailbox or a right-click with a selection of “Open Store”.

7. This will bring you to the root of the user’s mailbox. At this point you will want to navigate down to the Inbox Folder (path PRE-Exchange 2007: Root-Mailbox – IPM-Subtree – Inbox, Exchange 2007 Path: Root Container – Top of Information Store –Inbox).

8. Right-Click the Inbox folder and select “Open Associated Contents Table”.

9. Once you are in the associated Contents Table, you will want to sort on the “Message Class” column. The valid classes for OOF messages are:

· IPM.Note.Rules.OofTemplate.Microsoft

· IPM.Note.Rules.ReplyTemplate.Microsoft

10. If you are performing these operations on a workstation that has Outlook installed, you can actually open the item by right-clicking and selecting “Open Message” or simply double-clicking.

11. Open all items of type IPM.Note.Rules.OofTemplate until you locate the OOF template that matches the invalid one (verified in Step-2).

12. Initially, try just deleting the matching OofTemplate and moving on. You can always come back and delete all messages at a later point if need be.

13. To delete, simply select the OofTemplate , right click the instance, and select “Delete Message”.

14. Close MFCMAPI properly by selecting Exit to each screen that has been presented. On the Final dialog, where the actual tree is located, select the Session menu then select “Logoff” (this will clear the windows). Then from the file Menu, select Exit.

15. Start Outlook with the proper user profile.

16. Synchronize the mailbox with Exchange (if the user were in cached mode for example) by pressing F9 or the Send/Receive button.

17. Create a new OOF message and turn on the Automatic Replies for the OOF Assistant (note: at this point you could actually close Outlook, then double check that the new OOF message was created properly by using MFCMAPI to verify there is a new OOF template).

18. Test sending mail to the user and verify that the new OOF response is delivered successfully

Hope you find this useful.

Happy Trails.

Eric

Move-Mailbox Operations in Exchange 2007 Fail Due to An Attribute "Being Too Long"

ericnor — Sat, 24 Jan 2009 06:38:00 GMT

Move-Mailbox in Exchange 2007 Fails Due to An Attribute "Being Too Long"

The original scope or reason for this call was that Mailbox-Moves from Exchange 2003 to Exchange 2007
were failing. In this specific case (although it was never performed), mailbox moves would have also
failed if the source mailbox had resided on an Exchange 2007 server and the destination was either
Exchange 2007 or 2003. This is because the Exchange 2007 tools would have to be used to perform the
move.

So, if you are experiencing Move-Mailbox failures when using Exchange 2007 tools (such as the Exchange
Management console or the Exchange Powershell) and the reason being provided is that an "attribute is
too long", I'd like to personally welcome you.

This is definitely your article.

Primary Issue & Symptoms:

The individual I was working with was experiencing problems attempting to migrate Exchange 2003
mailboxes to Exchange 2007. He was receiving sporadic failures during the Move-Mailbox operations.
When the mailboxes failed to move he would receive the following error in Exchange Management Console
and Shell:

_______________________________________________________________________________________________________

(Note: In every example throughout the course of this post, I'm going to focus on Exchange
extensionAttribute2. However, the failure could occur on any attribute that exceeds the
predefined upper limit which will be explained later).
_______________________________________________________________________________________________________

Error:
Custom Attribute2 is too long: maximum length is 1024 and the actual length is "x" (where "x" is
something greater than 1024 characters)

Additionally, running Get-Mailbox in shell "" for the failed user would result in an error
about the object being corrupted:

Warning: Object domain.com/Users/TestUser has been corrupted and it is in an inconsistent state.
The following validation errors have occurred:

Warning: CustomAttribute2 is too long: maximum length is 1024 and the actual length is 1358 (the
character count on extension attribute2 for my example user).

Troubleshooting:

Prior to calling in the customer had tested whether or not one of these impacted users could be moved to
another Exchange 2003 database and server. The result of that test was that it could be moved successfully.
The issue only manifested itself when moving the mailbox to an Exchange 2007 mail store.

Obviously mailbox moves in Exchange 2007, use an entirely different tool set (EMC, and EMS) than that of
2003/2000.

The first thing I had this fellow do was take an LDP dump of an impacted user and send this to me for review.
A simple review of the LDP dump showed that extensionAttribute2 had a whole lot of characters present on the
user object. To get an exact count, I simply copied the character string in this attribute to Notepad
(disabled Word Wrap) then enabled the Status bar. A single Line in Notepad (with Word Wrap Disabled) can
only hold 1024 characters. So, I simply went to the end of line 2, to get the character count of line 2 and
added that value to 1024 (the character count of line 1).

I did this simply to verify that the Exchange Shell and Management Console were calculating this value
accurately (which they were):

1024 + 334 (character count of line 2) = 1358

Verifying the Limits of Extension Attribute 2 in the Windows Schema

In the Windows Schema there is an object of the attributeSchemaClass for Exchange Extension Attribute 2.
The actual adminDisplayName of this object is: ms-Exch-Extension-Attribute-2.

This object (as well as all other default Exchange related objects and classes) are created during Exchange
2007 /PrepareSchema (or in previousversions of Exchange /ForestPrep).

The minimum and maximum character values for objects of this type are defined by two attributes:

rangeLower (being the minimum)
rangeUpper (being the maximum).

I had this individual provide me with an LDP dump of the ms-Exch-Extension-Attribute-2 object from his
environment for me to review.

This looked like the following (keep in mind this is taken from my own personal lab environment):

Expanding base 'CN=ms-Exch-Extension-Attribute-2,CN=Schema,CN=Configuration,DC=,DC=com'...
Result <0>: (null)
Matched DNs:
Getting 1 entries:
>> Dn: CN=ms-Exch-Extension-Attribute-2,CN=Schema,CN=Configuration,DC= 2> objectClass: top; attributeSchema;
1> cn: ms-Exch-Extension-Attribute-2;
1> distinguishedName: CN=ms-Exch-Extension-Attribute-2,CN=Schema,CN=Configuration,DC=,DC=com;
1> instanceType: 0x4 = ( IT_WRITE );
1> whenCreated: 04/03/2002 15:15:43 Pacific Standard Time Pacific Daylight Time;
1> whenChanged: 09/03/2008 17:20:51 Pacific Standard Time Pacific Daylight Time;
1> uSNCreated: 6732;
1> attributeID: 1.2.840.113556.1.2.424;
1> attributeSyntax: 2.5.5.12;
1> isSingleValued: TRUE;
1> rangeLower: 1;
1> rangeUpper: 10240;
1> mAPIID: 32814;
1> uSNChanged: 46036019;
1> showInAdvancedViewOnly: TRUE;
1> adminDisplayName: ms-Exch-Extension-Attribute-2;
1> adminDescription: ms-Exch-Extension-Attribute-2;
1> oMSyntax: 64;
1> searchFlags: 17;
1> lDAPDisplayName: extensionAttribute2;
1> name: ms-Exch-Extension-Attribute-2;
1> objectGUID: bc303dd3-f496-4c60-a5fa-8647a42dd063;
1> schemaIDGUID: bf967969-0de6-11d0-a285-00aa003049e2;
1> attributeSecurityGUID: 1f298a89-de98-47b8-b5cd-572ad53d267e;
1> isMemberOfPartialAttributeSet: TRUE;
1> objectCategory: CN=Attribute-Schema,CN=Schema,CN=Configuration,DC=,DC=com;
-----------

So here we see that the rangerLower value is set to 1 (which would be expected as the attribute itself can
only be NOT SET or something > 1), and the rangeUpper value is set to 10240. This rangeUpper value explains
why I can have a character string stored in the AD larger than 1024 (in this case for extensionAttribute2
on a user object) but doesn't explain why the Mailbox Move operation would expect the value to be something
less than or equal to 1024 characters (e.g. 90% less than what is currently defined as the upper bounds on
the schemaClass for the attribute here).

At first glance this looks like a really big bug.

Why would we allow you to store something as large as 10240 characters in the AD and then fail the mailbox
move because we expect this value to be less than 1024 characters?

Time to look at the LDF file responsible for setting this value.

When Exchange setup /prepareschema is run, a series of LDF files are imported to define Classes and valid
attributes for mail enabled objects, etc. Amongst other things it will set valid ranges on some of the
attributes being added or modifed in the Schema. In Exchange 2007 there are 100 of these files within the
installation media named sequentially schema0.ldf - schema 99.ldf (in Exchange 2000/2003 there were 10,
although the file sizes were significantly larger).

That's a lot of files to look through to find the right object. Fortunately I got lucky and
ms-Exch-Extension-Attribute-2 is defined in the first file: schema0.ldf.

If you crack open that file in notepad, and search on the adminDisplayName for the object (in this case:
ms-Exch-Extension-Attribute-2) you find the default values that /PrepareSchema sets for this object:

dn: CN=ms-Exch-Extension-Attribute-2,
changetype: add
adminDescription: ms-Exch-Extension-Attribute-2
adminDisplayName: ms-Exch-Extension-Attribute-2
attributeID: 1.2.840.113556.1.2.424
attributeSyntax: 2.5.5.12
isMemberOfPartialAttributeSet: FALSE
isSingleValued: TRUE
lDAPDisplayName: extensionAttribute2
mapiId: 32814
name: ms-Exch-Extension-Attribute-2
oMSyntax: 64
objectCategory: CN=Attribute-Schema,
objectClass: attributeSchema
rangeLower: 1
rangeUpper: 1024
schemaIdGuid:: aXmWv+YN0BGihQCqADBJ4g==
searchFlags: 0

Conclusion 1:

Clearly an edit has occured to this object since the time /PrepareSchema was run. The rangeUpper value
/PrepareSchema set was 1024 not 10240.

Customer's rangeUpper Value from LDP Dump: 10240
Schema definition for rangeUpper: 1024

So, somebody or something performed a Schema Edit of this object by appending a 0 to its default value.

____________________________________________________________________________________________________________

Note:

For the record, the only person with permissions to make such a change would be an account with Schema
Admin permissions. So, if you are in a situaton where you need/"have to" figure out who made this change/edit,
you "should have" a relatively short list of people to discuss your concens with. ____________________________________________________________________________________________________________

At this point I was informed by the individual I was working with that there is a certain 3rd Party Password
History/Reset Tool, that uses extensionAttribute2 (although its apparently configurable) to write whatever
values it needs to provide for its additional functionality. Apparently an upper boundary of 1024 characters
was determined to be insufficent. Thus, he had been advised to perform the edit and increase its boundary
to 10240.

To be very, very clear here, Schema Edits/Modifications to the default Exchange and Windows objects have
never been supported. The Reason: It is entirely possible that functionality could be added/removed that
is dependent up the values we set/expect to be present (as in this case).

If you choose to go down this path, your risks include (but aren't necessairly limited to):

1. Your Exchange/Windows implementation not working as expected.
2. Being unsupported.

So at this point I think a summary is in order:

1. Mailbox Moves for specific users destined for Exchange 2007 fail when a value larger than 1024 on
extensionAttribute2 is encountered.

2. These failures only occur when Exchange 2007 Management tools are used.

3. An unsupported Schema Edit has occurred against the SchemaClass object (in this case:
ms-Exch-Extension-Attribute-2).

4. Similar problems are encountered when running Get-Mailbox for a user. The response is that the object
is actually corrupted.

Troubleshooting (Continued):

At this point, I recommended that we perform a test of removing some characters from extensionAttribute2 (to
get under 1024 charaters) for a single user, then re-trying the move. Prior to doing so, I ensured we had a
current and valid LDP dump saved for our user prior to removing any characters off the user. Once done, I
removed 335 characters off extensionAttribute2 for our guinea pig (e.g. "Test User"). This left 1023 characters
remaining on the user object.

It should go without saying that you don't just want to start removing random characters. I personally removed
the first 335 characters (for simplicity), but removing the last 335 would have been just as simple
(so use your creativity here).

I then forced Domain Controller synchronization within the domain where our "Test User" existed.

After verifying the change had replicated (used ADSI Edit and bound to several different DCs within the domain
to verify the change), we retried the move.

The Result: SUCCESS. Guinea Pig moved.

Conclusion 2:

We have our first work around. In this case extensionAttribute2 is expected to be < 1024. Reducing the characters
on extensionAttribute2 to something below the expected rangeUpper value (as defined by the Exchange Schema update)
allows the move to complete successfully.

It does add an additional layer of Administrative complexity (e.g. the users have to be defined and located, their
original values saved, characters removed from the production object to get under the limit, the move performed,
and once completed added back). But once done, the move will complete successfully.

Putting the Pieces Together: New Validation Checks in Exchange 2007 Move-Mailbox:

There are a host of articles and separate blog posts which specifically address this subject, so I'm not going to
go into considerable detail here. However what I will say is that Exchange 2007 Move-Mailbox has added a number
of new pre-move validation checks (these really didn't exist in Exchange 2003/2000/5.5). This was done in effort to
try and reduce situations where partially provisioned accounts/mailboxes would exist as the result of a partial move.
That is where to say from the "Exchange side" a mailbox move might have actually completed (e.g. the items were
copied out of the source mailbox, a new mailbox was created on the target, and said items placed into the target)
but some sort of other failure occurred during the update of the user's attributes (usually near the end of the move).
So, the net result of the failure might be the user having msExchHomeServer set to the original source server, but
homeMDB might be pointed to a database on the target server. The net result is that it is invalid, and now the user
can no longer logon. Anyone that has had to deal with these sorts of problems knows how tedious they can be to
resolve (in my line of work, I've worked way to many of these issues to ever want to have to do it again).

What's important with respect to the scope of this issue is to first know that these checks exist.

The "technical stuff" is that within Move-Mailbox code there is a hard-coded validation lookup table. A full
discussion of this is well outside the scope of this post, but let's just say that there is a global variable
defined for extensionAttribute2 and its rangeUpper value is defined as 1024 characters. Obvisously
extensionAttribute2 is one of many different checks that are performed against the user pre-move, so if you make
a modification to any other object to allow for a larger upper boundary, you run the distinct possibility that
the pre-validation check will fail and the mailbox move cancelled.

The important distinction and thing to remember here is that rangeUpper is hard coded within the validation lookup
table. Move-Mailbox will not make an LDAP call into the Active Directory Schema to determine what current rangeUpper
value has been defined for the attribute class. So even though the Edit in and of itself is technically sound,
the mailbox move will fail if it has more than 1024 characters.

Conclusion 3:

Unfortunately at this point the individual I was assisting was in an unsupported state. Which essentially left 3
hypothetical options (If the first work around was not acceptable):

1. Find a way to circumvent or disable the Move-Mailbox validation check.
2. Request a Development Change Request (DCR), to search the Schema as opposed to hard coding the rangeUpper
value in code.

3. Locate another attribute with sufficent size to store the values currently residing on extensionAttribute2.

Option 1: Not possible. There is no way to disable the validation lookup check.

Option 2: Has been requested before and has been denied by the Product Team because we do not support edits to the
Default Exchange Schema.

Which essentially leaves Option 3.

Exchange Extension Attributes have existed for a long time (I believe they were even present back in the Exchange
4.0 days, but don't quote me on that -- I know they were present in 5.5). They were added to provide for some
general flexibiliy for storing customer specific data (a good example of this might be a repository to store an
IP Phone Number, because no specific IP Phone attribute existed back at that time.). They have also been used
by Microsoft for very specific purposes such as migration tools (anybody remember NTDSNoMatch/NTDSAttrib? -- I
really miss the ADC it kept me busy for years).

Under closer inspection you will find that extension attributes 1-10 have an upper boundary set to 1024 characters.
However, 11-15 are double in size (2048 characters).

Here is a complete list with their corresponding values (taken from the SCHEMA*.LDF files):

ms-Exch-Extension-Attribute-1 -- rangeUpper: 1024
ms-Exch-Extension-Attribute-2 -- rangeUpper: 1024
ms-Exch-Extension-Attribute-3 -- rangeUpper: 1024
ms-Exch-Extension-Attribute-4 -- rangeUpper: 1024
ms-Exch-Extension-Attribute-5 -- rangeUpper: 1024
ms-Exch-Extension-Attribute-6 -- rangeUpper: 1024
ms-Exch-Extension-Attribute-7 -- rangeUpper: 1024
ms-Exch-Extension-Attribute-8 -- rangeUpper: 1024
ms-Exch-Extension-Attribute-9 -- rangeUpper: 1024
ms-Exch-Extension-Attribute-10 -- rangeUpper: 1024
ms-Exch-Extension-Attribute-11 -- rangeUpper: 2048
ms-Exch-Extension-Attribute-12 -- rangeUpper: 2048
ms-Exch-Extension-Attribute-13 -- rangeUpper: 2048
ms-Exch-Extension-Attribute-14 -- rangeUpper: 2048
ms-Exch-Extension-Attribute-15 -- rangeUpper: 2048

In the example I have been using througout this post, the current character size for "Test User' on
extensionAttribute2 has been 1358. That value is considerably less than 2048, so if its value were stored to say
extensionAttribute11, there would be no issue and the mailbox move would complete successfully. This is
potentially valid work-around 2.

However, if you cannot guarantee that values will always be less than 2048 (if using extension11-15) then the use
of an Exchange Extension Attribute is not a wise choice (to be clear you will be able to store them in the AD after
the edit (if done correctly) -- but anytime you need to perform a Mailbox-Move or Get-Mailbox procedure against the
user it will fail validation. This of course means you need to go with work-around 1.

In Summary:

As previously discussed Schema Edits of Default Exchange and Windows objects/attributes have never been supported
and this issue is a perfect example of why. These checks were put into place in an effort to help prevent partially
moved/provisioned mailboxes as well as a host of other scenarios to lengthy to discuss here. At the end of the day
the best overall solution to this problem would be for these 3rd party vendors to create their own unique schema
classes that "extend" (extend being the operative word here) the default User object class provided by Windows.
This is fully supported. Once done they could write whatever values and sizes they want to an "intentional" attribute
for whatever purpose they want. Because these attributes would be unique to their applications, they would never,
(will never say never) be checked in some sort of validation code (in this case move-mailbox).

A general outline for how this might be done:

1. Extend the Schema:
https://technet.microsoft.com/en-us/magazine/2008.05.schema.aspx?pr=blog

2. Extend the User Class in the AD Schema:
https://www.windowsitpro.com/Web/Article/ArticleID/9738/9738.html

3. Contact the IANA to acquire valid OIDs:

https://msdn.microsoft.com/en-us/library/ms677621.aspx
https://pen.iana.org/pen/PenApplication.page

I hope you find this interesting/useful.

Happy Trails.

Eric

Restore to Recovery Storage Group Fails with 0xa0008488 in Backup Executive

ericnor — Thu, 22 Jan 2009 22:05:00 GMT

This issue dealt specifically with a customer attempting to restore a 120GB mail store to a Recovery Storage Group.

The customer was using Exchange 2003 SP2 and Backup Executive 9.1.

Primary Issue & Symptoms:

Customer was attempting to restore an Exchange 2003 Mailbox Database to a Recovery Storage Group. Backup Executive would initalize the job successfully but the customer was noticing that the byte count never increased above 238 bytes. After 3 hours the restore would eventually timeout and the following error and status code would be documented in the restore log:

Job Competion Status
Job Ended:
Completed Status: Failed
Final Error: 0xa0008488 -- Access is Denied

Troubleshooting Steps:

Clearly the "Access Is Denied" error suggests a permissions related issue. Reviewing the configuration of the Recovery Storage Group showed that the Database Path for the linked store was being written to the f:\.

The first thing I did was review the NTFS permissions across the drives and did not notice any issues.

I then had the customer create a folder share on the root of the f:\, and verified that users could access the share remotely.

I then had the customer create a Test mail store (within the Production Storage Group where the database he was attempting to restore resided) and then had them add a single Test Mailbox to the store. We sent the Test User a single message and then backed up Test Store with NTBackup as well as Backup Executive. We added the Test Store as a "linked database" with the RSG, and ensured that the linked database and log file path was also confgured for the f:\. We then attempted a restore of Test via NTBackup.

This restore operation also failed, however NTBackup returned a different error stating that it was "Unable to Create the Log Path". A review of the application log showed the following additional error:

Event Type: Error
Event Source: ESE Backup
Event ID: 950
Description: Unexpected file system error 53 encountered while opening the restore environment file."

What is interesting about this event and error is:

1. It suggests that the natue of the error was during the creation of the Exchange restore environment file (e.g. restore.env).
2. The System Error Code of 53. This equates to ERROR_BAD_NETPATH

At this point, I wanted to verify that Test could be restored to a drive outside of the f:\. So, we removed the linked stores, and rebuilt the RSG linking Test then configuring the Log and Database paths to the e:\.

We then attempted a restore of Test with NTBackup which completed successfully.
Removed the stores and retested a restore with Backup Executive which also restored successfully.

Conclusion 1:

Again, clearly the f:\ is suspect here as evident by the fact that both NTBackup and Backup Executive can both restore to a location outside of f:\. This would also suggest no issue with Backup Executive or NTBackup as well as no issue with the configuration of the RSG itself (as restores can be completed regardless of Agent).

The obvious question here is what is unique about the f:\? At this point I was informed that the f:\ was in fact an External USB Drive that had "temporarily" been added for these specific recovery operations because there was not enough free space locally on any drive to support the size of the file being restored. In short, the customer had never tested this before.

Troubleshooting (Continued):

When any valid Backup/Restore agent (e.g. one that subscribes to the rules and methods of the APIs responsible for Exchange aware backup/restore) restores an Exchange Database (regardless of whether or not it resides in an RSG or production directory), ESE (the Extensible Storage Engine) needs to be able to physically map the path specified for the Temporary Location for the restore (e.g. the path specified for logs coming off tape as well as for the creation of the restore environment file). It does this via a UNC path that includes within the path the Admin Share for the target drive.

"\\ Server Name\\Temp Directory Specified for Log Files"

Example: \\ My Server\f$\MyRestore

So to verify that the shares were setup properly, I had the customer open CMD and enter: NET SHARE
NET SHARE will list all configured Shares on a System.
Reviewing the Output of NET SHARE, clearly showed that Default Admin share was not created for the f:\, but was present for all other drives. Which explained why a restore to the e:\ in the previous example worked successfully and also provided a logically sound reason for System Error 53 in the 950 Event.

Conclusion 2:

The fact that the Default Admin share for the f:\ was not present suggested the following:

1. An Administrator had manually removed the Admin Share (which seemed unlikely).
2. There was a problem with Disk Manager creating the share for External Storage (more likely).

Admin Shares (drive$) are created during startup when the SERVER service fully initalizes. My suspicion was that most likely the issue with the Admin Share not being created on the External Storage was occuring because the External Storage Device wasn't fully initalized by the Disk Manager System when the SERVER service has completed startup. As such, the SERVER service assigned no share to the drive because it wasn't technically valid at the time of assignment.

I tested this hypothesis internally with a personal USB Hard-Drive and witnessed the identical behavior (there was no "Default Share" for my USB Hard-Drive).

Troubleshooting (cont.):

On my own system (one without Exchange installed), I restarted the SERVER service and re-ran NET SHARE. The result was that my own personal External USB Drive now showed a proper ADMIN Share. This seemed to back up my conclusions about how the SERVER service creates shares for the External USB Drive.

The next logical step in this case was to restart the SERVER service on the Exchange Server where the RSG had been configured.

There are two primary Exchange Services that have a dependency of the SERVER service. Namely the Information Store (MSExchangeIS), and the System Attendant (MSExchangeSA).

Prior to restarting the SERVER service, I had the customer manually dismount all production stores (to force all pending writes into the store and bring the databases into a CONSISTENT or "Clean Shutdown" State), then proceeded to manually stop the Information Store and System Attendant (as well as the Computer Browser service which is dependent on the SERVER service) . With these sevices offline, I restarted the Server service from the command line:

net stop srv
net start srv

With the SERVER service back online, I proceeded to bring the Computer Browser, Exchange Information Store, and Exchange System Attendant Services back online.

At this point ran NET SHARE.

Result: The f:\ was now showing as have a valid Default Admin Share. From the cmd line you would see something like this:

NET SHARE

Share name Resource Remark
-------------------------------------------------------------------------------
F$ F:\ Default share

We then rebuilt our Recovery Storage Group, linked the original database back to the RSG, specificed our Database, Log and TEMP paths onto the f:\ and successfully restored the mail database via Backup Executive.

Conclusion:

It would appear that when External USB Storage is used, Windows does not automatically create the Admin Shares for the USB drive. As previously discussed, Admin Shares are created during server startup when the SERVER service fully initializes. My "professional opinion" (I am after all an Exchange Specialist not a Windows specialist), is that most likely this occurs because the External Storage isn't fully initialized by the Disk Manager System when the SERVER service has completed startup. This is why a "reboot" of the server wouldn't correct the issue (it would simply recreate the issue). By restarting the SERVER service with the USB Drive fully initalized, you effectively force the shares to be re-evaluated and recreated.

So what is important to remember here is that if you ever have to perform a similar operation (restoring an Exchange Database to an External Drive), you need to ensure that the Admin Share for the Drive is configured properly prior to attempting the restore (as Exchange needs to directly map to the the UNC path specified via the Admin Share). The Admin Share itself is not needed for a backup, but it should go without saying that no production Exchange Database should ever by physically located on a USB drive. Otherwise, you and I will probably be working on an Exchange Performance related issue in the near to distant future (hello RPC pop-up).

Hope you find this interesting.

Happy Trails.

Eric