Post MortemMigrating 240,000 Mailboxes And 29 Domains

Dennis Swenson

The Project:

I recently participated in a project with a large agency of the United States federal government as they migrated and consolidated their servers from Exchange Server 5.5 to Exchange 2000 Server and Exchange Server 2003. In order to perform the migration and consolidation, an easy-to-manage process needed to be developed to synchronize the Exchange Server 5.5 directory with the agency's existing Active Directory® forest.

Challenges:

• The Exchange Server 5.5 organization consisted of 272 Exchange Server 5.5 sites spread across the United States (including Alaska and Hawaii), Puerto Rico, and the Philippines—a very large project.
• A Huge number of Exchange Server 5.5 objects needed to migrate: approximately 240,000 mailboxes, 60,000 distribution lists, and 48,000 custom recipients.
• The agency's existing Active Directory forest consisted of 29 domains.
• The migration and consolidation project spanned multiple years, so the directory synchronization process needed to be easily supportable throughout the life of the agency's Exchange Server migration project.
• The agency's three administrative branches each had their own security concerns, management styles, and business functions, which had to be accounted for when developing the synchronization solution.

The Plan:

Following guidelines in the Microsoft Operations Framework (MOF) and Microsoft Solutions Framework (MSF) , a solution using Microsoft Identity Integration Server (MIIS) was to be evaluated in a test lab and then moved to limited pilot deployment for evaluation in the production environment. Barring any insurmountable technical difficulties, MIIS then would be deployed into production to begin the Exchange Server 5.5 to Active Directory forest directory synchronization process preceding the Exchange Server migration and consolidation project. One of the planning team's main goals was to limit the size of the server farm required to perform directory synchronization to a manageable number, preferably a single server if possible.

The Consultants:

Microsoft Consulting Services (MCS) partnered with AC Technologies Inc., an IT integrator and solutions and services provider for government and military agencies. This consultant team would perform the overall technical planning, testing, deployment, and management of the directory synchronization solution throughout the life of the Exchange Server migration and consolidation.

Background:

At the onset of the Exchange Server migration and consolidation project, approximately 80 percent of the users had been migrated to the Active Directory forest in a separate project. Therefore, the directory synchronization solution had to be able to integrate and readily handle the instances where users of Exchange Server had not yet migrated to the new Active Directory forest.

As the legacy Exchange Server 5.5 organization was a single entity, the synchronization process needed to be treated as such. That is to say, it would be nearly impossible for one agency branch to use the Active Directory Connector (ADC) to synchronize their portion of the Exchange Server 5.5 organization while another agency entity used a third-party Exchange Server migration tool that did its own style of directory synchronization. A one-size-fits-all solution was a definite requirement.

Procedure Analysis

Before beginning, let's take a quick high-level view of the MIIS component setup, shown in Figure 1. You can think of the MIIS metadirectory as a conglomeration of two subcomponents: the metaverse (the center circle) and the connector space (represented by boxes on either side of the metaverse). The conduit that connects the MIIS metadirectory to the directory of interest is called a management agent. I will refer to the diagram in Figure 1 throughout the discussion as the components listed in the diagram are defined and discussed in the following paragraphs.

Figure 1** MIIS Component Structure **

In order to read the entire Exchange Server 5.5 organization (remember, that's 272 sites) and still design an easily managed solution that could run on a single machine, the team had to develop an alternative to building a synchronization connector to each Exchange Server 5.5 site.

MIIS has many out-of-the-box connectors or management agents, including two management agents specialized for Exchange Server 5.5 directory connection. One of these Exchange Server 5.5 management agents is called the bridgehead management agent. This agent allows MIIS read/write access to a single Exchange Server 5.5 site while also allowing read-only access to the entire Exchange Server 5.5 organization, providing for organization-wide object discovery. This management agent is only able to write to the Exchange Server 5.5 site it is connected to, much like the Exchange Server 5.5 management UI.

When configuring a management agent required tasks include defining the objects and determining which object attributes will be copied in with the object. The agency decided that they wanted the Exchange Server 5.5 directory to own and propagate the mailbox attributes pertaining to personal and business information to the corresponding associated user object in the agency's Active Directory forest. This included attributes like departments, titles, street addresses, phone numbers, and postal addresses. All Exchange Server 5.5 mailbox objects pertinent to the migration were discovered and automatically placed in a special section of the MIIS database (a series of SQL tables) called the Exchange Server 5.5 bridgehead management agent connector space. Because the objects were from Exchange Server 5.5, MIIS was instructed to "project" the objects into another special area of the MIIS database called the metaverse. The metaverse is where MIIS is able to perform most of the identity management tasks when two or more disparate directories are involved.

Like the Exchange Server 5.5 management agent, MIIS also has a management agent specifically designed for connection to an Active Directory domain. This Active Directory management agent was configured to connect to different "partitions" in the Active Directory forest. For the purposes of this discussion, a partition is equivalent to an Active Directory domain in the forest. The user objects were discovered and placed into their own particular Active Directory management agent connector space of the MIIS database. Therefore, Exchange Server 5.5 management agent has its own connector space separate from the Active Directory management agent's connector space. This became the staging area for each management agent.

From here, the Active Directory management agent was configured to find a mailbox in the MIIS metaverse whose associated Windows NT® account security identifier (SID) matched the SID on the Active Directory user object (plus matching on first and last name as well). When a successful match was made, the Active Directory management agent was instructed to join the Active Directory user object to the corresponding mailbox in the MIIS metaverse. After this association is made, MIIS can freely flow mailbox attribute information out to the user object in Active Directory.

That briefly covers how user objects and mailboxes are matched and managed. In similar fashion, all distribution lists and custom recipients in the legacy Exchange Server 5.5 organization were discovered by the Exchange Server 5.5 management agent and projected up from the Exchange Server 5.5 connector space into the metaverse. Since there is no representation of these objects in the Active Directory forest, MIIS must create (provision) these objects in the forest so Exchange Server 2003 users can see these same objects in their version of the Global Address List (GAL), which, for Exchange Server 2003 users, is based on the Global Catalog. MIIS was instructed to provision these groups and custom recipients to the proper area in the Active Directory management agent connector space based on which Active Directory domain the two sets of objects were to be created in.

The domain owner was determined by looking at the Exchange Server 5.5 distinguished name on the object, then looking at a table to determine which domain correlated with that Exchange Server 5.5 site. Once the objects were created and exported to the appropriate Active Directory domain, the final step was to have the Active Directory management agent confirm the newly created distribution lists and custom recipients (now called distribution groups and contacts) via a delta import. Through provisioning, there is a natural join from the provisioned object to the metaverse object.

Again, as with the user objects and mailboxes, once the Active Directory management agent successfully provisioned and joined the distribution groups and contacts to their legacy Exchange Server 5.5 counterparts, MIIS was able to freely flow attributes from the Exchange Server 5.5 objects to the Active Directory objects. These attributes included, but were not limited to, distribution list memberships as well as personal information contained in the custom receipts.

For further illustration, consider Figure 2 which shows a provisioned distribution list, or DL, (now called distribution group, or DG) in the agency's Active Directory forest and a projected mailbox with a joined user object. MIIS can read attributes in from the Exchange Server 5.5 organization (such as the mailbox's primary SMTP address, mailbox's phone number, and distribution list's membership) and flow them from Exchange Server 5.5 to Active Directory.

Figure 2** Provisioned Distribution List **

Up to this point, the directories have a baseline synchronization process that keeps the objects in sync between the agency's Exchange Server 5.5 organization directory and the agency's Active Directory forest. MIIS can now simply monitor both directories for changes (adds, deletes, modifications) to objects in one directory and pass them on to the other directory. But, what about directory changes that occur as a direct result of Exchange Server 5.5 site migrations? Here's where it really gets interesting.

When an Exchange Server 5.5 site was about to be migrated, that site was put into what was called "migrate mode" by adding the Exchange Server 5.5 site name to an XML input sheet that is read in by the Exchange Server 5.5 management agent. For mailbox/user-joined metaverse objects, a series of well-orchestrated imports, synchronizations, and exports of Exchange Server 5.5 bridgehead management agent and the Active Directory management agent then took place, which essentially deleted the metaverse object and forced the exact opposite of the original mechanism to occur. Now, the Active Directory user object projected up to the metaverse and the Exchange Server 5.5 mailbox was joined to it. At this point, the ownership and flow of information also changed. Active Directory now owned the attributes and attribute change flow now went from Active Directory into Exchange Server 5.5. When this procedure was completed successfully for all mailboxes in the site, the site was free to commence with the mailbox migration.

At the end of a successful mailbox migration, the mailbox was simply deleted from the Exchange Server 5.5 site. The Exchange Server 5.5 management agent removed the mailbox from the Exchange Server 5.5 management agent connector space and from the joined metaverse Active Directory user object. MIIS then automatically created a representative custom recipient in the Exchange Server 5.5 site that the Exchange Server 5.5 management agent has write access to. This custom recipient then replicated throughout the Exchange Server 5.5 organization so that legacy Exchange Server 5.5 users can continue to see the migrated mailbox in the Exchange Server 5.5 GAL. MIIS flowed information to the newly provisioned custom recipient to allow this, namely by ensuring the custom recipient has an SMTP address that correlates to the target Exchange Server 2003 organization.

Responses to old e-mails are allowed through a similar mechanism, as MIIS flows information to the Active Directory user object regarding the mailbox's legacy X.500 addresses, such as where in the legacy organization the migrated mailbox came from. MIIS also adds this same legacy X.500 address to the newly provisioned representative custom recipient, thereby allowing replies to messages which originated prior to that mailbox's migration.

For distribution lists, the migration process is a little different. Once an Exchange Server 5.5 site is in migration mode, as with the mailboxes, MIIS changes the ownership and direction of attribute change flow to favor the agency's Active Directory forest. The distribution list "migration" was essentially a deletion of all distribution lists from the Exchange Server 5.5 site (they still exist in the agency's Active Directory forest via MIIS provisioning). Once MIIS detects the deletion of the distribution lists via the Exchange Server 5.5 management agent, it provisions a representative custom recipient to the Exchange Server 5.5 site to which MIIS has write access.

This new DL representation then replicates throughout the entire legacy Exchange Server 5.5 organization. However, in Exchange Server 5.5 sites with a larger number of DLs, it was found that sometimes their deletion would be detected by MIIS in more than one batch, which meant that if DLs were members of other DLs, there was a potential that the member DLs could drop out of membership. This was due to the latency of the Exchange Server 5.5 directory replication from the remote Exchange Server 5.5 site to the MIIS Exchange Server bridgehead site.

To counter this, the exact same procedure was performed on the mailboxes during the orchestrated "migrate mode" management agent runs. The metaverse objects pertaining to distribution lists were deleted and the Active Directory management agent projected the distribution groups from the Active Directory management agent connector space up into the metaverse. The DLs from the Exchange Server 5.5 connector space were then joined to the projected metaverse object. As a result, the Active Directory forest now owns all attributes (including membership) prior to the DLs being deleted, thereby safeguarding DL membership in another DL.

Custom recipient migration was a simple approach. MIIS provisioned all custom recipients in the agency's Active Directory forest (exactly like provisioning DLs) in their proper Active Directory domain based on the legacy Exchange Server 5.5 distinguished name. Once the entire Exchange Server 5.5 site had been migrated, the site was decommissioned, thereby leaving the remaining legacy Exchange Server 5.5 organization void of these custom recipients. This logic was based on the fact that those who created the custom recipient (CR) had been migrated over to the new Exchange Server 2003 organization, and this external recipient already existed over there as a contact (via MIIS provisioning).

The people who used the CR in Exchange Server 5.5 now have the same contact representation in the Exchange Server 2003 GAL. If the legacy users in other Exchange Server 5.5 sites needed to have this representation back, the CR was recreated manually in that Exchange Server 5.5 site. This was all in an effort to keep the MIIS provisioning code simpler. However, if it is required, these "migrated" custom recipients could be re-provisioned back into the legacy Exchange Server 5.5 MIIS bridgehead site in similar fashion to the migrated mailboxes and DLs.

Conclusion

Through testing and a well-planned pilot process, the directory synchronization design team was able to prove that MIIS could perform the synchronization tasks quite readily and predictably. Equally important is the fact that the design team met the goals of the planning team in that the directory synchronization process could be designed to run on a single server which simplified management of the solution considerably over what would be required for a multiserver solution.

MIIS Resources

• MIIS Technical Walk-Throughs for Basic Operations
• General Information on MIIS
• MIIS Design and Planning Documents and Worksheets Collection
• Exchange Server 2003 Information
• Exchange Server 2003 Downloads

For the directory synchronization efforts, only two MIIS management agents were used (one Active Directory management agent and one Exchange Server 5.5 bridgehead management agent). Compare this to the amount of ADC connection agreements that would have been required to accomplish the same synchronization task, plus the number of servers that would have been required in order to run that many ADC connection agreements.

Because this approach uses a single management agent and a single Exchange Server 5.5 site to discover a very large Exchange Server 5.5 organization of 272 sites, the project relied heavily on Exchange Server 5.5 directory replication. This reliance was identified during the planning and risk analysis phases to be the point of highest risk for the directory synchronization subproject. This can be somewhat mitigated by decreasing the directory replication interval in the Exchange Server 5.5 sites in the directory replication chain to the MIIS Exchange Server 5.5 bridgehead site.

Care should be taken to not make the directory replication cycle too aggressive, as it could affect overall Exchange Server 5.5 performance and/or network performance. The moral of this story is that for larger, geographically dispersed Exchange Server 5.5 deployments, do not put too many direct MIIS tasks on the reliance list of Exchange Server 5.5 directory replication. For example, don't wait to do something based on seeing a mailbox deletion at the MIIS Exchange Server 5.5 bridgehead site. That kind of latency can throw a wrench in the machine.

The key to pulling off a successful synchronization is planning in advance how each object will be handled throughout the life of the migration process. This includes where the object is to be provisioned, as well as any customized attribute mapping/modifications to be performed during the flow operations. All of this can readily be hashed out in a full lab or even a representative virtual lab. It is highly recommended that you test all directory synchronization for an Exchange Server migration in a lab setting prior to production piloting or deployment, as directed by the MSF and MOF principles and guidelines.

Dennis Swenson is a Senior Infrastructure Consultant for Microsoft Consulting Services Public Sector and has been involved with Active Directory and integrated Exchange Server since the beta version of Exchange 2000 Server. He loves to write customized code using application APIs, but don't tell his friends as he does not want to appear geeky. Reach him at dswenson@microsoft.com.

© 2008 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.