You have setup a hybrid between on-premises Microsoft Exchange and Office 365 (Exchange Online), but when you try and move a mailbox from on-premises to Office 365 it tells you that the mailbox already exists. What it is telling you is that the user has a mailbox on-premises and one in Office 365 (Exchange Online). Gulp!

There are two scenarios where I have seen this happen

Most commonly, when a mailbox has been created for a user in Exchange Online before Directory Synchronization between on-premises AD and Office 365 has been setup. You then add Directory Synchronization (DirSync) and it links up (joins) an AD account from on-premises AD with an Office 365 account. You now have a situation where the AD account has a mailbox on-premises and in Exchange Online. Even if you attempt to delete the mailbox in Exchange Online (by removing the Exchange Online licenses from the user), as soon as you assign the license back to the user, the mailbox is re-created, so you still cannot move the mailbox from on-premises to Exchange Online.

The second scenario is, you have deployed DirSync and it is working “well”. Subsequently you have created a user in AD with a local mailbox and then synced the user account across into Office 365. You very soon after sync (example; under 1 minute), you assign the synced user in Office 365 an Exchange Online Plan license, which in turn creates a mailbox for the user in Exchange Online.

Yes, this shouldn't happen, in theory, if a user has a mailbox on-premises, and you sync the account into Office 365, when you assign a license it shouldn't create a mailbox, it should know that the user has a mailbox on-premises. If you assign the license through the Office 365 Admin Portal, it should give you a dialog box saying that a mailbox wont be created for the user as the mailbox exists on-premises.

Why does this happen, and how to prevent this?

What you want, is for Dirsync to set a flag on the user object in Office 365, to indicate that the object has a mailbox on-premises, and therefore not to create one when you assign an Exchange Online license. Specially, the attribute in question is called  msExchangeReceipientType, and you want it in the cases described above for the value to be “–2147481594” before assigning an Exchange Online license. If you have DirSync in place, and the user does have a mailbox on-premises it will be this, except in the 2 scenarios described above. Sometimes, you are just so quick, that the msExchangeReceipientType hasn't been set correctly for the user object, and you assign the license, or when you don't have AD sync, then it obviously wont be that value.

What you could do to prevent the behavior seen in the scenarios above, is check the value before assigning a license e.g., if you assigning licenses to users on mass via PowerShell, check this value before assigning the license if it is not “-2147481594” don't assign the license.

To view the value of msExchangeReceipientType, use the Azure AD Module for PowerShell and connect into your tenant. Then run “Get-MSOLUser -UserPrincipalName ‘upnofuser’ | fl” to list all properties for the user object.

What happens if I have a user where the issue has already occurred, and now they have a mailbox in Office 365 and on-premises.

The problem is you cannot easily change the msExchangeReceipientType value manually e.g. via PowerShell. You need to get DirSync to set the value. So if you have an account where this is an issue, the best thing is to remove the account from being synchronized, remove the account from Office 365, and then re-sync the account.

Here are the steps in more detail:

1. Move the AD account into an OU which is not being synchronized with Office 365 (Azure AD).
2. Force a DirSync. This should force a deletion of the account in Office 365 (Azure AD)
3. Check that the account has been deleted from Azure AD, if it hasn’t, force DirSync again. Do this until AD account is removed from Azure AD
4. Use Azure AD remote PowerShell to remove the account from the user recycle bin. i.e. Remove-MsolUser -UserPrincipalName ‘upn’ -RemoveFromRecycleBin –force
5. Check through the O365 admin UI portal, that the account has gone from the recycle bin
6. Move the AD account back into an OU which is being synchronized with Office 365 (Azure AD)
7. Force DirSync. This will re-provision the account into Office 365 (Azure AD)
8. Check that msExchRecipientDisplayType is now what it should be e.g. “-2147481594”.
9. Try move mailbox again

Note: by removing the account in Office 365 (Azure AD), you will be deleting content associated to this account too e.g. mailbox and OneDrive for Business, so if you need data, make sure you backup before doing this.

On a similar note. I have seen a situation where an organization wants to move resource mailboxes from Exchange on-premises to Exchange Online, but when they attempt the move request they get an error saying they need to assign a license first. As you know you don't need to licenses resource mailboxes in Exchange Online, if they are created as a resource mailbox i.e. not a user mailbox behaving like a resource mailbox. What has often happened in this case, is that their on-premises resource mailboxes were actually created as user mailboxes and therefore need a license assigned. How does Exchange Online know this, because of the value of msExchangeReceipientType. In this case it is probably set to “-2147481594”, which denotes that it is a user mailbox. What you need to do, is check this value for your synced accounts to ensure that this is not what is set on resources like rooms etc.

As a reference to what the values of msExchangeReceipientType mean and for example should be for a resource mailbox, check out this blog post.

Hope this helps,

Michael

As you can imagine there are many reasons which could prevent Microsoft Outlook connecting to Office 365, most commonly because the Outlook software version is not patched to the minimum required level, or firewall / proxy issues. However in this article, I specifically wanted to cover scenarios that comes up which are related to the autodiscover service, and how this may prevent Outlook connecting to Office 365 (Exchange Online) mailbox, or even resolving correctly in updating or creating an Outlook profile.

The scenarios I describe below cover what should happen when the autodiscover DNS record(s) point to on-premises Exchange, but the users mailbox is in Office 365 (Exchange Online). Then, I cover three scenarios which I have seen which break this model.

What should happen

Microsoft Outlook connects to the local Exchange autodiscover service, which in turn redirects Outlook to Office 365, because the mailbox has been moved there. This is because the user object in on-premises Exchange is a remote-mailbox with a forwarding address to Office 365. Fiddler is a very handy tool (or even the test email connectivity dialog built into Outlook), to understand what is going on, and therefore determine where Outlook might be having an issue connecting. In most environments, you should see Outlook connect to CAS server on-premises, and then connect to the following addresses (because it is redirected), in the order shown below:

1. Internal requests to CAS Servers using SCP (AD method for determining autodiscover location), and requests to external autodiscover address e.g. autodiscover.contoso.com/autodiscover/autodiscover.xml (note, there will be a few similar requests to the autodiscover web address as it tries various formats). One of these should succeed, and it will then tell Outlook to redirect the request for autodiscover to the internal O365 email address for the user i.e. alias@tenantname.mail.onmicrosoft.com. This should result in the following web addresses successfully being access by Outlook.
2. http://autodiscover.tenantname.mail.onmicrosoft.com/autodiscover/autodiscover.xml (note HTTP, not HTTPS)
3. https://autodiscover-s.outlook.com/autodiscover/autodiscover.xml
4. At this point, Outlook should now know where the mailbox CAS services is and connect to it (hopefully now using MAPI over HTTP)

If you see different behavior to the above, then that can help you troubleshoot issue. Also, a simple test would be, can the user, via their web browser, access these autodiscover end points? Perhaps their proxy server is preventing access to one of them e.g. *.outlook.com. There are 3 common scenarios that you might come across worth covering.

Scenario 1

Companies block access to consumer email services, and in this case outlook.com. A URL in outlook.com is used to return autodiscover information (autodiscover-s.outlook.com), so a users PC needs access to this end point.

Scenario 2

Outlook correctly connects to the autodiscover end points in Office 365, and does connect to the final endpoint which is https://autodiscover-s.outlook.com/autodiscover/autodiscover.xml, but yet it still doesn’t setup the Outlook profile. This can happen when an Outlook client is getting many responses from internal CAS servers through SCP lookup, all correct responses i.e. redirecting Outlook to the forwarding email address, but because there are so many responses coming back from internal CAS, Outlook stops attempting to discover. This is a design feature of Outlook (and in theory all autodiscover clients), they should stop attempting to autodiscover after 10 responses from autodiscover services. This is designed to prevent infinite loops. This behavior can be seen when a PC is a member of an AD site, which doesn’t have a CAS server associated to it, and so many (in some cases all) internal CAS servers respond, and if you have more than 10 CAS servers, you can see this behavior. To test if this is something that is occurring in your environment, use the steps in the following KB article (http://support.microsoft.com/kb/2212902), to disable SCP i.e. just add ExcludeScpLookup as Dword and set to 1. This will force Outlook to only use the DNS record for autodiscover. Disabling SCP is not the ultimate solution, but perhaps configuring AD to have a CAS for the site is a better e.g. a set of CAS servers which should primarily respond to autodiscover request for PCs in that site. Here is the autodiscover design article which talks about the max 10 responses, and here is an article which talks about configuring a site to use specific CAS, using autodiscoversitescope feature.

Scenario 3

You haven't deployed a hybrid solution, and have created mailboxes in Office 365 with a different SMTP domain name to on-premises Exchange. So you don’t have autodiscover redirection setup in on-premises Exchange for mailbox in Office 365, but have created a DNS record to point autodisover.domainname.com to Exchange Online. In this case, on-premises AD SCP will mess you around even if the SMTP domain name between your existing on-premises Exchange mailboxes are different to Office 365 SMTP address. When the PC is on your network, it will still try and resolve the end point using SCP. As a test on the user’s PC, you want to disable SCP lookup and force the user’s PC to only use the DNS record for autodisover.domainname.com. As described in step above.

Please note: this information was accurate at the time of writing this article (30 April 2015). Service endpoints, such as autodiscover addresses in Office 365, may change over time, which would make some of the information in this article redundant. Keep in mind, Fiddler is a great friend.

Hope this information helps,

Michael

There are several reasons why your Office 365 Pro Plus might fail with the error, error code: 30088-27. However, in this article I point out one scenario which I believe could be common, and a second scenario, which is specific to early builds of the Office Deployment Tool.

First Common Scenario

This occurs when Office 365 Pro Plus is attempting to fetch updates. The process which fetches the updates, doesn't run as the user who is logged into the PC, it runs as “Local System”. If “Local System” cannot connect to the internet, you will receive this error. “Local System” might not be able to connect to the internet for a variety of reasons, but the most common appears to be when your company is using manual proxy settings in the web browser. When you configure manual proxy settings for your users, it doesn't effect the “Local System” account, rather Local System is always set to “Automatically detect settings”. Therefore if it cannot identify the proxy through “Automatically detect settings”, the update will fail as the updater process cannot connect to the internet. Interestingly, on earlier builds of Office 365 Pro Plus, you don't get an error, you just don't see Office ever update itself (even after turning update off, and back on again, as described here).

To prove if you are a victim of this issue, you can force Local System (and all winhttp applications) to use a proxy, by running the Netsh command. I don't believe the Netsh command is a long term solution, but rather helps you troubleshoot the issue. A long term solution is recommended later in this article.

Open the command prompt (run as administrator), and use the following command to import the manual proxy settings from IE:

netsh winhttp import proxy source=ie

Now rerun the update, and it should start work…well, it will work if you are a victim of the issue described above. To reset winhttp back, run the following command:

netsh winhttp reset proxy

2 other reasons why you might receive the same error code. 1 - It could be the Local System cannot authenticate to your web proxy. 2 - You might be updating from a local file share, and local system doesn't have rights to the local file.

Troubleshooting techniques that allowed us to find the route cause, that you might be interested in:

1. Use network monitor on the PC in question, and see what is happening under the hood. You should see Office reach out to a URL which looks something like this one; http://officecdn.microsoft.com/pr/39168D7E-077B-48E7-872C-B232C3E72675/Office/Data/v32.cab. The cab file contains the latest Office 365 Pro Plus version number, and the “Local System” account must be able to fetch this cab file. Using network monitor, you should be able to see if this is happening, if the request is hitting your proxy server, or trying to go via a direct route.

2. Enable version logging. This works for both installation and updates. These 2 articles are good references for verbose logging:

1. http://support.microsoft.com/kb/2826852
2. http://community.office365.com/en-us/blogs/office_365_community_blog/archive/2013/04/04/office-365-proplus-administrator-series-enabling-verbose-logging-for-troubleshooting-office-365-proplus-installations.aspx

Longer term solution:

The better long term solution for this, in my opinion, is that you rather want to use “automatically detect settings”, this is for both local system account, and also for users. This can be implemented with “not much effort”, by using WPAD with any existing web proxy solution. I am not an expert on WPAD, but here is a quick 101 on how it works.

Use a PAC / DAT file to hold the proxy configuration for the network. This is a text file, placed on a web server on your network, which holds configuration as to which proxy server(s) to use, for which end points. This allows you to specify which IPs / Host Names are internal, and gives greater flexibility than the manual settings you probably have today anyway. This file should be placed on a highly available web server, perhaps on your proxy server(s) is a good place.

Next, in your local DNS (or hosts file for testing), you register the name WPAD to point to the server(s) where the text file is located. That’s it. Now, any web browser which is configured to use “automaticly detect settings”, will fetch the proxy text file (http://wpad/wpad.dat) from the web server and work accordingly. This means, without changing any user’s PC settings, Office 365 Pro Plus will start working because local system account is set to automatic detect settings by default. You can then over time, change user’s manual settings to auto detect.

Second Scenario, specific to early build of the Office Deployment Tool

We (project team) found a specific issue, when you have deployed Office 365 Pro Plus, using an earlier build of the Office Deployment Tool. We found that specifically, build 15.0.4535.1503 of the Office Deployment Tool causes this issue. In this case, we found that a missing registry key was causing an issue that prevents Office 365 Pro Plus updating, and we would get the same error i.e. error code: 30088-27. The registry key in question, tells Office 365 Pro Plus where to go, to fetch updates. By manually adding the key, updating started to work (as long as you are not a victim of the “Local System” issue too).

What key?

You should have 1 of the following keys (just having one of them works, you don't need both).

• HKLM\Software\Microsoft\Office\15.0\ClickToRun\Configuration\CDNBaseUrl
• HKLM\Software\Microsoft\Office\15.0\ClickToRun\Configuration\updateurl

The value should be set to the path where updates are retrieved from. We copied ours from a value in the propertybag key. It will look something like this http://officecdn.microsoft.com/pr/39168D7E-077B-48E7-872C-B232C3E72675. Disclaimer: we haven't checked all deployments of Office Deployment Tool, so I am not sure if this value is the same for all users, countries, and versions of Office 365 Pro Plus, hence the suggestion to copy it from an existing key in your registry.

The image below shows this value:

Second Disclaimer: This has yet to be officially published as a bug by Microsoft, but is rather something that I found on a recent engagement with a customer. I accept no responsibility for your actions when editing the registry. Use this information to help identify the issue, and contact Microsoft support for guidance on how best to resolve it.

The HCW runs a series of PowerShell commands to setup hybrid, one of them is a command to get federation information from Exchange Online (for your O365 tenant email domain i.e. tenantname.mail.onmicrosoft.com). Note, as part of the wizard, it also performs the reverse query i.e. from Exchange Online, get federation information for on-premises domain. This article only covers the federation request to Exchange Online, perhaps in another article I can explain the reverse.

If you are having issues in the HCW with getting federation information from Exchange Online, the first thing to do is to run the PowerShell command yourself from the Exchange Management Console, with a verbose switch, so you can get more information about the issue. Example command:

Get-FederationInformation –domainname tenantname.mail.onmicrosoft.com –BypassAdditionalDomainValidation $true –verbose

The result of this might already point you in the direction of what the issue is. Now, let me explain what happens when you, or the HCW runs the Get-FederationInformation command to the onmicrosoft.com domain.

1. Uses AutoDiscover to locate the autodiscover.svc service in Exchange Online for the email domain in question
2. Calls the GetFederationInformation “method” on the autodiscover.svc service, using the service.svc end point location returned from step 1

The screen shot below shows the Fiddler trace for a successful call.

Note: The last request in the screenshot above is to the autodiscover.svc service.

As you can see, your Exchange Server where you are running this command from, needs access to the appropriate web end points in order for the command to succeed. In most cases where we see the command fail, it is because of an outbound access issue, and often a proxy server or firewall is blocking access to these resources, or the proxy configuration on the Exchange Server is not configured correctly.

“Obvious” things you would do on your Exchange Server to see if the server can access these resources:

1. Open web browser and try and access the URL’s in question, that is:
1. http://autodiscover.tenantname.mail.onmicrosoft.com/autodiscover/autodiscover.xml (note that this is HTTP, not HTTPS. This should redirect automatically to the URL below)
2. https://autodiscover-s.outlook.com/autodiscover/autodiscover.xml (this should now prompt for credentials. Ignore the response of the page, the point is you should get prompted which means you can access the resource.)
3. https://pod51050.outlook.com/autodiscover/autodiscover.svc (the server name will very likely be different for you. Like above you should be prompted for credentials and this proves connectivity)
4. The steps above confirm your browser with it's proxy settings can connect to the endpoints. Interestingly, it doesn't matter if your web browser can or cannot connect, it is important that the Get-FederationInformation PowerShell command can! Read on to understand more.
2. Telnet to:
1. autodiscover.tenantname.mail.onmicrosoft.com port 80
2. autodiscover-s.outlook.com port 443
3. pod51050.outlook.com port 443
4. As with the web browser tests, it is not that important if Telnet can or cannot connect, it is important that the Get-FederationInformation PowerShell command can!

It is still useful to run the above tests, because it allow you to understand if there are connection issues with the well known proxy configurations from the web browser or Telnet. However, those connection requests might all succeed, but the Get-FederationInformation PowerShell command doesn't. Why? This is because the Exchange Server or the Exchange Management Console PowerShell session might be using different proxy settings to your web browser or Telnet.

Here is a list of common proxy configuration “places” your Exchange Server would use, so check these to understand what your configuration looks like.

1. Web Browser – e.g. in Internet Explorer, this is under Tools, Internet Options, Connections
2. Windows HTTP (WinHTTP) – this allows for HTTP/S applications besides the web browser to use a proxy, and is more of a system wide setting on the server. This article describes how to view and set this.
3. Proxy client – you might have a proxy client application on your sever e.g. TMG Client, which has a proxy configuration option
4. Exchange Server – you can configure Exchange Server itself to use a specific proxy server. You will know this has been done because when you run the Get-FederationInformation command with the verbose switch, one of the first command outputs will tell you it is using a proxy server. You can also use the Get-ExchangeServer command to see if a proxy has been set for the Exchange Server.

Now, the last and probably most important thing to uncover, is how the Exchange Management Console PowerShell session connects to the web resources. It could be that it is not using any of the proxy configuration options mentioned above. In my scenario where we were getting a bad gateway error when running Get-FederationInformation, we could connect to the end points using the web browser and Telnet, but not through Exchange Management Console PowerShell. This is because Exchange Management Console PowerShell behind the scenes opens a remote PowerShell session to itself. Remote PowerShell runs in the context of IIS, and more specifically it runs in the context of an IIS application pool. Requests to the web resources are therefore made under the context of the IIS application pool, and this application therefore needs access to the end points. It is pretty rare for it not to have access, or have different proxy settings than the rest of the server, but it can happen.

The IIS application pools runs as the “Local System” account on the server. In our case this account had been sent a proxy setting via a GPO in the past and in theory it had since been removed from users, but it wasn't removed from the “Local System” account. Therefore, the Get-FederationInformation command was trying to connect via a proxy server that was no longer around. The screen shot below shows the 2 IIS PowerShell application pools. Note the “Local System” account.

To change the local system account settings, we needed to run through the steps discussed in this article.

In summary, your Exchange Server needs access to a bunch of end points in Office 365, and typically if you have issues it is because it cannot connect to these. Look at your proxy settings and understand how your Exchange Server is trying to connect to these end points. Ensure that the route allows for the connections to succeed. You could have firewall or proxy server rules preventing access, or you might have miss-configured proxy settings on the server. I prefer to bypass proxy servers and allow direct internet connections, but in some cases this might not be feasible in your environment.

Both Fiddler and Network Monitor are extremely useful tools when working with issues like these, but in some production environment having these installed might not be allowed by the organization.

Hopefully this article helps you understand what is happening under the covers, and allows for better troubleshooting if you come across Get-FederationInformation issues.

Good Luck,
Michael

During the configuration step, DirSync attempts to contact each domain in the AD Forest, and then attempts to grants rights to the MSOL service account in each domain (note: MSOL service account is created as part of DirSync configuration). It is during these tasks where it attempts to connect to one of the DC’s in each domain where it could throw 0x80005000 error, if it fails to connect to that DC.

I have seen a few reasons why this occurs:

1. You have domain / DC entries in your AD Forest which point to objects which no longer exist
2. You have a DC which is down
3. You have a DC which is inaccessible e.g. routing rules prevent access to the DC

Here I have some pseudo code, which shows what DirSync is attempting to do:

   1: foreach (ActiveDirectoryPartition directoryPartition in forest.Domains)

   2:   {

   3:       GiveMSOLUserRights(MSOLUser); //grant MSOL service account rights in the domain

   4:   }

There is nothing you can do to solve the error, other than ensure the DC it is trying to connect to is available / contactable. It is a hard requirement of the DirSync software. There are a few PowerShell scripts around, which you can use to find out which DCs are not accessible from your DirSync server e.g. http://www.itadmintools.com/2011/07/powershell-list-domain-controllers-in.html

If Outlook is going via a proxy server to Office 365, there is a specific issue which occurs if you have manually configured your proxy server address in IE (see screen shot below as an example). That is, Outlook thinks some IP ranges in Office 365 are internal, and therefore doesn't route the requests via the proxy server. Specifically, if you do a network monitor trace, you will see requests to a 132 and 154 range, are not routed to the proxy server.

How to solve this, some options:

1. Microsoft recommends that Outlook doesn't go via a proxy server when connecting to Office 365, instead it should have direct access to the internet (Office 365). There are many reasons why it is recommended to bypass proxy server e.g. performance.
2. Unselect “bypass proxy server for local addresses” in IE manual proxy configuration screen. This is a quick one to test, to see if you are a victim of the issue.
3. Have IE automatically detect proxy settings instead of manual settings (there is a larger discussion to be had on how this works than we have time for here)
4. I haven't tested, but perhaps a PAC file will work to force the connection to the proxy for those IP ranges.

Note: this looks like it might be a bug, but not confirmed yet. Therefore, this issue could be resolved in a future update to Outlook. For now, you will need to go with one of the recommended fixes above. Or, post an alternate suggestion to help the community out.

Update: this was confirmed to be a bug, and Microsoft has released a hotfix and process to resolve it. This is described here: http://support.microsoft.com/kb/2916915. Personal opinion is that I still prefer the auto detect option with proxy pac file though as a solution.

The Issue

During the installation of DirSync, on the final step in the wizard, where is does the installation of all components, the installation never completes. It seems to hang. The wizard suggests this could take up to 10 minutes. After 20 minutes of waiting we soon realized something was wrong.

The resolution

It turns out the issue lies in the installation of SQL Express 2012 SP1, which DirSync installs for you. The installation of SQL Express 2012 SP1 “hangs” and hence DirSync never completes the installation. The reason this happens, is because during SQL Express installation it tries to end the Windows Installer process, and isn't successful. It then goes into an infinite loop trying to end the process, and check if the process has ended. You will know this is happening to you, if you look in the SQL Express installation log file, and find the following entries repeatedly being added at the end of the file.

Log file path in our case: C:\Program Files\Microsoft SQL Server\110\Setup Bootstrap\Log\ID\Detail_ComponentUpdate.txt

   1: (01) 2013-11-27 16:13:09 Slp: Sco: Attempting to close service handle for service msiserver

   2: (01) 2013-11-27 16:13:09 Slp: Sco: Attempting to close SC Manager

   3: (01) 2013-11-27 16:13:14 Slp: Sco: Attempting to open SC Manager

   4: (01) 2013-11-27 16:13:14 Slp: Sco: Attempting to open service handle for service msiserver

   5: (01) 2013-11-27 16:13:14 Slp: Invoking QueryServiceStatus Win32 API

To solve the issue, once you have verified you are being subjected to this issue (because of the entries in the log file) open Task Manager and manually end the Windows Installer process. You will then notice SQL Express installs, and so does DirSync.

I am sure this will be resolved when its officially supported. Until then, hope this saves you some time.

On the 29th of March 2012, Microsoft South Africa is hosting a free event to show you how to achieve better ALM for SharePoint solutions with Team Foundation Server.

Have look here for more information and to register.

Michael

Obviously, from a South African perspective this is something that is welcomed and will be of great value to Office 365 users in our country. I have heard of some results from internal testing, and it really does sound phenomenal.

Here is the press release and a video for more detail.

Contact Riverbed South Africa if you would like more information.

Enjoy,

Michael