A)  What is Exchange Hybrid Deployment:

Hybrid allows on-premises organization and cloud organization work together like a single, seamless organization. In other word's hybrid deployment provides the seamless look and feel of a single Exchange organization between an on-premises Exchange Server 2013 organization and Exchange Online in Microsoft Office 365

B) Benefits of Exchange Hybrid:

• Exchange Online users and on-premises users can share free/busy calendar data vice a versa.
• Hybrid allows Secure mail routing between on-premises and Exchange Online organizations
• Administrators can use powerful and familiar Exchange management tools to move users to Exchange Online.
• OWA redirection allows for redirection from the on-premises environment to the Office 365 Outlook Web App environment.
• MailTips, out-of-office messages, and similar features understand that Office 365 and on-premises users are part of the same organization.
• Delivery reports and multi-mailbox search work with users who are on-premises and those working in Exchange Online.
• Authentication headers are preserved during cross-premises mail flow. So, all mail looks and feels like it is internal to the company (for example, recipient names resolve in the GAL).
• With the help of Directory Synchronization you get Unified GAL
• If necessary, administrators can easily move mailboxes back to the on-premises Exchange environment
• Cloud-based message archiving for on-premises Exchange mailboxes
• Administrators do not have to manually reconfigure Outlook profiles or resynchronize .OST files after they move users’ mailboxes

C) Exchange Hybrid Prerequisites:

• On-premises Exchange organization. You can choose to setup Hybrid with Exchange 2010 Sp3 or Exchange 2013 Server
• Office 365 for Enterprises (AdminDisplayVersion parameter value should be equal to or greater than 15.0.620.28)
• The Windows Azure Directory Synchronization tool or AD Sync
• A certificate from a trusted third-party CA
• AD FS is optional but strongly recommended

 More details : https://technet.microsoft.com/en-in/library/hh534377(v=exchg.150).aspx

D) Exchange Hybrid Considerations:

1. On-premises mailbox permissions such as Send As, Receive As, and Full Access that are explicitly applied on the mailbox are migrated to Exchange Online if the tenant in Exchange Online has been fully synchronized using Dirsync or AAD Sync.
2.  Inherited (non-explicit) mailbox permissions such as permissions applied to the mailbox database and any permissions on non-mailbox objects (such as distribution lists or a mail-enabled user) are not migrated. Therefore, you should recreate these permissions in Exchange Online using the Add-MailboxPermission or Add-RecipientPermission cmdlets.
3.   Cross-premises permissions Mailbox permissions such as Send As, Receive As, and Full Access are not supported if the user trying to access the mailbox is in Exchange Online but the target mailbox is on-premises, or vice versa. In order to overcome this limitation, mailboxes belonging to users who have access to the first mailbox should also be migrated at the same time to ensure the delegate scenarios continue to work.
4.  If your organization implements multiple on-premises Exchange organizations, you must deploy Exchange 2013 SP1 or greater servers in your on-premises organization to configure a hybrid deployment with Office 365.

C) What happens behind the scene when running Hybrid setup

Firstly ,Hybrid Configuration wizard creates the Hybrid Configuration object in your on-premises Active Directory. This Active Directory object stores the hybrid configuration information for the hybrid deployment and is updated by the Hybrid Configuration wizard.

1.The Update-HybridConfiguration cmdlet triggers the Hybrid Configuration Engine to start.

 2.The Hybrid Configuration Engine reads the “desired state” stored on the HybridConfiguration Active Directory object.

3. The Hybrid Configuration Engine connects via Remote PowerShell to both the on-premises and Exchange Online organizations

you can also view respective powershell connections made in Hybrid Update logs as follows

[12/20/2014 15:35:26] INFO:Opening runspace to https://onpremise/powershell?serializationLevel=Full
[12/20/2014 15:35:26] INFO:Successfully connected to On-Premises

[12/20/2014 15:35:26] INFO:Opening runspace to https://ps.outlook.com/powershell-liveid/powershell.htm?serializationLevel=Full;clientApplication=EMC;ExchClientVer=14.3.123.4
[12/20/2014 15:36:12] INFO:Successfully connected to Tenant

 4. The Hybrid Configuration Engine discovers topology data and current configuration from the on-premises Exchange organization and the Exchange Online organization.

 5. Based on the desired state, topology data, and current configuration, across both the on-premises Exchange and Exchange Online organizations, the Hybrid Configuration Engine establishes the “difference” and then executes configuration tasks to establish the “desired state.”

6. This is how Hybrid Configuration desired state look like.

 Now , let's look at steps involved in setting up Hybrid with my Exchange 2013 Sp1 server and Office 365 Tenant

1.Log on Exchange on premise server and kick Hybrid setup using EAC

 2.Select hybrid domain* as per your setup requirement.

 3.View/Copy domain proof token and post verification with ISP , hit next

4. Choose transport options.  This setting may vary/depend on your requirement

5. Choose receive 2013 CAS server(s)

6. Choose 2013 MBX server(s)

 7.Select transport certificate from the drop down

8. Enter External FQDN of CAS Server

9.Enter Org Management Administrator's AD account

10. Enter Global Admin Office 365 account

11. Choose Update to configure hybrid. It fire's the Hybrid engine and starts configuring in the back ground.

 12. Upon completion you can hit close.

Wow!! At this point my Hybrid is all set.

What's New with Exchange Hybrid

Exchange 2013 Sp1 Hybrid now supports multiple Exchange Organizations configured against a single O365 tenant.

Exchange Forest contoso.com  ======> Office 365 Tenant  <===== Exchange Forest fabrikam.com

 Multi-forest hybrid deployment prerequisites:

• Prerequisites for multi-forest hybrid deployment are nearly same as of single hybrid deployments  <refer Section C > with few exception mentioned below.
• Each Exchange organization should have Exchange 2013 with SP1
• Each Exchange organizatoin should have minimum one SMTP and Autodiscover namespace published in way that Office 365 can query Autodiscover for each forest successfully.
• Different public certificate should be configured on each Exchange forest.key thing to note here is that the certificate used for hybrid deployment features for each forest in a multi-forest organization must be issued by different third party CA. For example, VeriSign or Go Daddy. For example, one forest would have a certificate issued by VeriSign and one forest would have a certificate issued by Go Daddy. But then the certificate installed on the Mailbox and Client Access (and Edge Transport if deployed) servers in each Active Directory forest used for mail transport in the hybrid deployment must all be issued by the same CA and have the same common name.
• Also the common name (CN) of the digital certificate must match the host being authenticated and is typically the external hostname for the Client Access server in the Active Directory forest. For example, mail.contoso.com.
• Microsoft Forefront Identity Manager (FIM) 2010 R2 or greater and the Azure Active Directory (AAD) connector for Active Directory Synchronisation to synchronize mail recipients in each forest and the Office 365 tenant
• Single Sign on is optional but if administrator want's to use SSO in multi-org hybrid model then ADFS need's to setup in each Active Directory forest, or to configure a single SSO server if there is a two-way forest trust configured between the on-premises forests.
  
  

Configure a Hybrid deployment in a multi-forest organization (Flow)

1. Preparation
   
   Verify that you’ve met the hybrid deployment prerequisites as listed above
   Validate AutoDiscover is properly configured and published in each Exchange organization
   Validate public certificates for Exchange org are unique
   Create 2 way forest trust
   
   
2. Configure Mail Flow on-premise

Configure SMTP domain sharing as required
Configure mail flow between on-premise organizations

       3.  Configure Directory Synchronization

   Configure FIM + AAD Connector to synchronize mail recipients in each forest and the Office 365 tenant

       4.   Run Hybrid Configuration Wizard

   Prepare Office 365 Tenant
   Run the HCW in each Forest (More inputs : Create a hybrid deployment with the Hybrid Configuration wizard )
   Validate mail flow between all entities

        5. Configure ADFS (optional)

Please refer to part II blog for understanding & mitigating common Issues with deploying Hybrid setup.

This blog covers most of common errors and its fixes, which can be encountered during Cutover migration process. 

Error 1. Microsoft.Exchange.Data.Storage.Management.MigrationTransientException: We weren't able to connect to the remote server. Please verify that the migration endpoint settings are correct and your certificate is valid, and then try again. Consider using the Exchange Remote Connectivity Analyzer (https://testexchangeconnectivity.com

Fix:

1. This error occurs when Outlook Anywhere is broken for On premise Users.
2. Correct Authentication settings for Outlook Anywhere, See https://technet.microsoft.com/en-us/library/bb123542(v=exchg.80).aspx.
3. Open Registry Editor on the Client Access server or on the front-end server, and verify that the ValidPorts registry value exists under HKEY_LOCAL_MACHINE\Software\Microsoft\Rpc\Rpcproxy. Also confirm that the key includes the NetBIOS and FQDN for all mailbox servers and for each required port (that is, for ports 6001, 6002, and 6004).
4. Once the Outlook Anywhere is fixed for on premise User's the above error shall be resolved. To verify this,you can open a web browser, browse to https://testconnectivity.microsoft.com/?TabId=o365 and perform RPC over HTTP test against On premise Mailbox .If test passes you should be all set. To know other method's to validate if Outlook Anywhere is working fine ,you can refer to my Part 1 of this blog.

More info: https://support.microsoft.com/kb/2903050 

Error 2. "Unable to connect to the server"

Fix:

1. Assign the Migration administrator Receive-As permissions to the mailbox database or Full-Access permissions to every mailbox using Power shell command's listed below.

How to: This Power shell command assign's the migration administrator account the Full Access permission to all user mailboxes in the on-premises organization.

Get-Mailbox -ResultSize unlimited -Filter {(RecipientTypeDetails -eq 'UserMailbox')} | Add-MailboxPermission -User migadmin -AccessRights FullAccess -InheritanceType all

This command assigns the Receive As permission to the migration administrator account for the mailbox database named "Mailbox Database 1".

Add-ADPermission -Identity "Mailbox Database 1" -User migadmin -ExtendedRights receive-as

     2.Ensure fully qualified domain name (FQDN) or remote procedure call (RPC) proxy address is entered correctly in the E-mail Migration Wizard. Entering Just NetBIOS of server name doesn't work.
     3. In a web browser, browse to https://testconnectivity.microsoft.com/?TabId=o365 and perform RPC over HTTP test against On premise Mailbox to ensure Outlook Anywhere is working fine.

 More info: https://support.microsoft.com/kb/2389390 

Error 3.  "Couldn’t connect to the source mailbox"

Fix:

1. Assign the migration administrator Receive-As permissions to the mailbox database or Full-Access permissions to every mailbox using power shell command's list below " Applies to Exchange 2007,2010 & 2013"

How to: This Power shell command assign's the migration administrator account the Full Access permission to all user mailboxes in the on-premises organization.

Get-Mailbox -ResultSize unlimited -Filter {(RecipientTypeDetails -eq 'UserMailbox')} | Add-MailboxPermission -User MigrationAdmin -AccessRights FullAccess -InheritanceType all

This command assigns the Receive As permission to the migration administrator account for the mailbox database named "Mailbox Database 1".

Add-ADPermission -Identity "Mailbox Database 1" -User MigrationAdmin -ExtendedRights receive-as

 More info: https://technet.microsoft.com/en-us/library/jj898489(v=exchg.150).aspx

Error 4.  A source email address <user email address> couldn’t be found in the on-premises domain.

Fix:

1. Unhide the user from the Exchange server's Global Address List and perform the Migration.

How to:

For Exchange 2007 & Exchange 2013:

• Click Start > All Programs > Microsoft Exchange Server > Exchange Management Console Expand recipient Configuration > Mailbox > Locate the Mailbox for the user in question > Right Click the Mailbox > Properties > General Tab > Uncheck the "Hide From Exchange Address Lists" option and click Apply > Click Ok to close the window

For Exchange 2013:

•  Connect to the Exchange Admin center > Recipients > Mailboxes > Locate your user > Edit >  Uncheck "Hide From Exchange Address Lists" option > Apply > Click ok to close the window.

Error 5.  Default folders can't be moved or deleted.

Fix:

1. This error mainly occurs if default folders like Inbox, Sent Items are broken or corrupted due to third party Addins or memory corruption
2. Disable outlook Addins on user in question Outlook Client's machine.See https://office.microsoft.com/en-in/outlook-help/enable-or-disable-add-ins-in-office-programs-HA010034127.aspx#BM9
3. Click > Start > Run and type Outlook /Resetfolders Switch on outlook Client Machine of User in question. This command shall bring the default folders in to their default state.

More info: https://support.microsoft.com/kb/2939487

Error 6.  "/o=nt5/ou=1234e11b842d744d88734918addbfc84/cn=1234e11b842d744d88734918addbfc84 A required recipient field SmtpAddress is missing"

Fix:

1. Run following command let on Exchange onpremise Server to remove invalid SMTP address in Exchange Global address list

Update-GlobalAddressList -identity "Default Global Address List" -DomainController DomainControllerName

       2.  In some cases either Mailbox or Mail-Enabled contact which doesn’t have proper LegacyExchangeDN value along with matching SMTP address, to sync Cloud can also cause this error and best way to fix is by

a) Search for Mail recipient matching proxy address “1234e11b842d744d88734918addbfc84" and remove it

b) Convert Object Guid into Hex value and perform LDAP search with help of the Object Guid and remove the User object if not in use. See https://blogs.technet.com/b/ehlro/archive/2010/04/22/how-to-find-the-object-that-belongs-to-a-guid.aspx for more details. 

Error 7.  "Failed to update one of the recipient properties. Couldn't find object "ADCDisabledMail". Please make sure that it was spelled correctly or specify a different object."

Fix:

1. Navigate to User properties >Check if any disabled User object is granted Send on Behalf of Permission on his Mailbox and Accordingly remove disabled Object from there.
2. In Some Case if you have assigned delegates and that object is disabled or No more in Active Directory, You'd expect to this error. Therefore remove the entry from 'PublicDelegates' Attribute on User Object using Adsiedit.

Error 8.  " This mailbox exceeded the maximum number of large items that were specified for this request. --> MapiExceptionMaxSubmissionExceeded: IExchangeFastTransferEx.TransferBuffer failed (hr=0x80004005, ec=1242)"

Fix:

1. This error is commonly seen if the user's mailbox contains one or more email items that exceed the 35-megabyte (MB) message limit.
2. Using outlook change the View to List. Arrange all items in all folders of the user's mailbox by size. Sort the items from largest to smallest so that the largest items are at the top of the list.
3. Locate the largest Item larger than 35 MB. Depending on how important email item is, you can export into pst or shift delete from user mailbox and then proceed with migration.

Error 9. "A different type of Active Directory object already exists for user@domain.com in the target forest. Please verify that the SMTP address is correct"

Fix:

1. Log on to Office 365 Portal and Navigate to Users and Groups.
2. Search for User which Failed to Migrate
3. Delete the User Object permanently Using GUI or Exchange PowerShell using command let below (Imp : If you have configured onedrive or contain any sharepoint data for the user , ensure you take a back up prior removing the object)
   
   Remove-MsolUser -UserPrincipalName User@Domain.com-RemoveFromRecycleBin
4. Ensure User is Successfully Removed from User and Groups In the Portal
5. Perform the Cut over migration

Error 10. MigrationPoisonCountThresholdExceededException"

Fix:

1.  Run Powershell command to pull Migration statistics report and contact Microsoft Support for further investigation.

Get-MigrationBatch -Diagnostics -IncludeReport | Export-Clixml -Path C:\Migbatch.xml

PowerShell Command to Review Migration progress & Troubleshooting:  The Powershell output's shall help you reviewing the detailed insights about the failure / error messages and migration progress.

Get-Migrationconfig | fl
Get-Migrationendpoint | fl
Get-MigrationBatch | fl
Get-MigrationStatistics | fl
Get-Migrationuser –identity <userfailingmigration> | fl
Get-MigrationUserStatistics -id <userfailingmigration> | fl
Get-MigrationUserStatistics -id <userfailingmigration> -diagnostic | fl
Get-MigrationUserStatistics -id <userfailingmigration> -includereport | fl 

I hope you found this informative and helpful!!    Stay tuned for my next blog on Staged Migration!!

What is Cutover Migration :

Cutover Migration is one of the type of Exchange online migration where Mailbox are provisioned and corresponding mailbox content is migrated from Exchange on premise servers to Exchange online in single batch. It is the simplest way to get all your existing email into Office 365 and very suitable for the organizations who plan to move your entire email organization to Office 365 and manage user accounts in Office 365. Cutover migrations are supported for Exchange 2003, 2007,2010 and 2013 version of Exchange server.

  
Considerations:

• You can migrate up to 2000 mailboxes only.
• This migration method only moves mailboxes, mail users, mail contacts, and mail-enabled groups. Security Groups and system mailboxes are not Migrated.
• Cutover migration isn’t available for Microsoft Live@edu organizations.
• Partial Migration are not possible like folder exclusions , time range etc.
  
• Cache mode ost file of On premise mailboxes are not Migrated.
• Active Sync Devices need to reconfigure their profile post migration.
• Consider Time Factor to Migrate Mailbox Content

Preparation:

• Add your on-premises Exchange organization as an accepted domain of your Office 365 organization.
• Grant Full mailbox Access to Administrator Account (Admin who would be performing Migration) on User Mailboxes Which would be Migrated to Exchange online
• Ensure Windows Azure Active Directory Sync tool is Deactivated
• Disable Unified Messaging On on-premise Mailbox.
• Upto 2000 Mailboxes hosted on source onpremise Exchange server
  
• Configure and Validate if Outlook Anywhere is working well on your on-premises Exchange server.
  
  

How to check if Outlook Anywhere is functioning: Try any one of these methods to test your connection settings:

1. Use the Microsoft Exchange Remote Connectivity Analyzer to test your connection settings. Use the Outlook Anywhere (RPC over HTTP) or Outlook Autodiscover tests
2. Run Test-OutlookWebServices using on premise Exchange Powershell
3. Check if User hosted on On premise Exchange server can Successfully Configure Outlook Profile outside the domain using Autodiscover.
4. Run Test-MigrationServerAvailbility Powershell Command as mentioned below.

Launch Exchange Online PowerShell

$Credentials = Get-Credential
Test-MigrationServerAvailability -ExchangeOutlookAnywhere -Autodiscover -EmailAddress <email address for on-premises administrator> -Credentials $credentials

 Example:

 PS C:\Users\shahmul\Desktop> $Credentials = Get-Credential
 PS C:\Users\shahmul\Desktop> Test-MigrationServerAvailability -ExchangeOutlookAnywhere -Autodiscover -EmailAddress alex@shahmul.msftonlinerepro.com -Credentials $Credentials

RunspaceId         : 1ad9afcb-e806-47f2-aeca-e975b1efd980
Result             : Success

Note: Your Outlook Anywhere configuration must be configured with a certificate issued by a trusted certification authority (CA). It can't be configured with a self-signed certificate.

How Cutover Migration process Works:

Here's some description about what all stuff happens when you kick the Cutover Migration.

1. The administrator creates and starts a cutover migration batch using the migration dashboard in the EAC or Shell.
   
   
2.  Migration Service on Cloud makes a NSPI connection to the source server.
   
   
3.  Once NSPI Connection is Successful after Batch is triggered- what now?

a) It uses the address book on your on-premises Exchange server to identify the mailboxes, distribution groups, and contacts that will be migrated to your Office 365 organization.

b) It provisions new Exchange Online mailboxes, It creates distribution groups and contacts

c) Exchange Online migrates mailbox data, which includes email messages, contacts, and calendar items, from each on-premises mailbox to the corresponding Exchange Online mailbox. This part is also called as Initial Synchronization.

key thing to note here it uses NSPI interface to provision Users, Groups, Contacts , whereas during Mailbox data Movement to cloud it uses RPC over HTTP connection.

      4.Post provisioning process, and Mailbox Content is Moved to cloud. Exchange online sends status email to the Migration Admin which displays over all Migrations statistics like, Number of Mailbox Migrated successfully / Failed etc. along with 8 digit password per user

      5.Once you discover the Mailbox are synced to Cloud in the Migration Dashboard, the mailboxes in the on-premises Exchange org. and the corresponding Exchange Online mailboxes start synchronizing every 24 hours. This process is called as incremental synchronization.

This means that any new messages sent to the mailboxes on the on-premises Exchange server are copied to the corresponding Exchange Online mailboxes. This incremental Sync continues until the administrator stops or deletes the migration batch.

Imp: During incremental synchronization, deletions made in an on-premises mailbox are synchronized to the corresponding Exchange Online mailbox. This means that if a user deletes an item from an on-premises mailbox that was previously migrated to Exchange Online, that item will be deleted from the Exchange Online mailbox

      6.At this stage Mailbox are Migrated to Cloud and all users can successful log on to Exchange online mailbox. Admin updates Mx record to point to their Office 365 org. so that messages hit cloud mailbox directly instead of onpremise server.

      7.After Mx record is updated and users on cloud are able to receive emails, Admin is all set to delete the Cut Over Migration batch and now On premise Exchange Mailbox stop Synchronizing to Exchange online Mailboxes.

      8. At this point we seemed to be all good; However, there are few post migration tasks you need to take care of. you can refer my Post Migration Task section at the bottom part of this blog for more details.

What are Migration Steps:

This section will help you step by step how to migrate our on premise exchange Mailboxes to Cloud.

 Log on to Office 365 Portal and Open Exchange Admin Center

Click New  and then click Migrate to Exchange Online

On the Select a migration type page, click Cutover migration, and then click Next. as per screen shot below

 If outlook Anywhere is Configured Successfully on the Source Exchange Server,Connection settings are successfully discovered, the fully qualified domain name (FQDN) of your on-premises Exchange server and RPC proxy server are displayed on the Confirm the migration endpoint page. Verify the Settings and Click next as per Screen Shot below

  Enter Migration Batch Name and click Next.

On the Start the batch page, select the either of the option based on your migration Plan and click New

  At this time the migration batch is displayed on the Migration Dashboard 

You can also view the Migration batch and Migration Statistics details Using PowerShell

Syncing:

Synced:

• Assign License to Users Provisioned In Exchange online Tenant.
• Configure Desktop Services
• Configure Autodiscover Record to Point to Office 365
• Configure your MX records to point to Office 365.

Once you change the MX record to point at Office 365, there is some DNS replication time. During this time, email will be delivered to either Exchange 2007 or Office 365. It’s important to keep your migration batch job running for up to 72 hours after switching the MX record.After you change the MX record and verify that all email is being routed to Exchange Online mailboxes, you're ready to delete the cutover migration batch.

 Imp: When to Delete a Migration Batch:

• All users can successfully connect and user their Online Mailbox
• Emails is being sent directly to the Exchange Online mailboxes after you change your MX record to point to your Office 365 email organization.
• If you okay that after the batch is deleted, mail sent to mailboxes on the on-premises Exchange server will not be copied or Synced to the corresponding Exchange Online mailboxes.

Once DNS records are updated / Migration Batch is Deleted  and after you verify that all email is being routed to Exchange Online mailboxes - you are almost Done!!. Its the time to decommission the On premise Exchange Server. However, Before we decommissioning your on-premises Exchange organization, we recommend that you contact Microsoft Support.

 Common Error's and Troubleshooting:  You can refer to my Part II section of this blog. https://blogs.technet.com/b/shahnawaz/archive/2014/05/12/part-ii-cut-over-migration.aspx which has got more insights about common error's and how to diagnose Cutover Migration Issues.

I hope you found this informative!!