Chris Allen's (CHALLEN) Exchange Blog https://docs.microsoft.com/archive/blogs/exchangechallengeaccepted/feed.xml CHALLENge accepted Thu, 05 Jul 2018 14:36:50 GMT en-US hourly 1 What is KeepAliveTime used for in regards to Exchange? https://docs.microsoft.com/archive/blogs/exchangechallengeaccepted/what-is-keepalivetime-used-for-in-regards-to-exchange Sat, 15 Apr 2017 03:42:45 GMT https://blogs.technet.microsoft.com/exchangechallengeaccepted/?p=486 First of all, where is it? The KeepAliveTime is a setting that lives in the registry of your Windows Servers at the following location: \HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters It is a Dword value and its data in decimal is in milliseconds. If it is not present under that key, that means that it is set to the default, which is 7,200,000ms or 2 hours. What does it do? KeepAliveTime does nothing on its own. If a process uses it, however, a heartbeat will be sent at it's configured interval to the connected client over TCP to ensure the client is still there and responding. If it isn't, then it will close the connection. If the client responds, it will wait the pre-set amount of time and then do it again. What is the purpose of that? There are a finite number of connections available to Exchange. If every connected client held their connections open indefinitely, then Exchange would run out of connections. So it is preferred that when Exchange detects the client has closed down, or dropped from the network, that Exchange close that connection and free it up for another client. It also serves another purpose. When an Exchange client, like Outlook, or a mobile device, opens a connection to Exchange, it is basically saying, "Hey, I am open on this port and I will remain open on this port until something closes it".  Exchange will do the same saying "ok, I will use a port on my side and reserve it for your system to talk to mine". So both of those ports remain open and available to each other until Exchange no longer needs it, or the client disconnects. There is a misconception that there is an actual virtual tunnel there but in reality, it is just two open ports reserved for each others' use. The client will then use that connection to talk to a Client Access Server in Exchange. It fully expects that everything it does on Exchange will be done over that connection. Now there are things between the client and that connection that have the ability to close the connection as well. Things like Firewalls, MDM solutions, Load balancers, etc. Each of these will have a timeout setting on them with an ominous name like TCP Timeout, TCP TTL (time to live), or Idle Session Timeout. I will be calling all of these TCP Timeout for simplicity sake. What these do is they monitor connections established through them over TCP and if nothing happens on the connection in a certain amount of time, they terminate it. Exchange does not like it when that happens to one of its connections. Exchange expects that connection to remain open until it is done with it. If it sends a process to the client, or if the client sends one to it, depending on the process, it can take some time for a response. If the Network appliance times out the connection before the response comes in, it causes all kinds of inconvenient issues such as Outlook authentication prompts, IPhone full resyncs, and delayed responses. Basically, Exchange and the client have to reestablish their connection before it can continue, which means you have to authenticate again, or in the case of the IPhone, all of your emails disappear and then slowly start uploading to the device again. KeepAliveTime, when configured properly, prevents those timeouts from occurring so long as the application is set up to make use of them. Outlook, IPhone, and many other applications come configured that way out of the box. The idea is that the KeepAliveTime heartbeat will interrupt the idle TCP session, making it no longer idle for a few seconds,  and thereby cause any TCP timeouts to have to start their countdown over again. So if your firewall has a 30 minute TCP Timeout, but your KeepAliveTime heartbeat fires off every 15 minutes, the TCP Timeout will never trigger, because the session isn't idle for more than 15 minutes ever and Exchange will be able to hold that connection open until either the client goes offline, or the Exchange server finishes using it. However, if your KeepAliveTime is at the default of 2 hours, and the Firewall has a TCP timeout set at 30 minutes, your KeepAliveTime heartbeat will never have the opportunity to fire as the session will have closed an hour and a half earlier. Our recommendation is that you set the KeepAliveTime to 30 minutes, or 1,800,000ms. Any Network appliances such as Firewalls, MDM solutions and Load balancers should have their timeout values set to higher than 30 minutes, preferably progressively higher from the Exchange server out to the egress point. So for instance, if your Client connects through a firewall, then through an MDM solution, and then a Load Balancer, and then finally to the Exchange Client Access Server, you would want it to look something like this: Firewall (1Hour TCP Timeout)->MDM Solution (50minutes TCP Timeout)->Load Balancer (40minutes TCP Timeout)->CAS Server (KeepAliveTime 30 minutes) With this setup, the connections will remain open as long as they are being used and cannot be interrupted prematurely with maybe the exception of a network outage or hardware failure. If I change it, do I need to reboot? Absolutely. The KeepAliveTime setting is not an Exchange setting. It is a network stack setting in the OS, so the setting will not take effect until a reboot is completed. Is KeepAliveTime case sensitive? This is a big question and there are various opinions on it. Until I can get the F5 load balancer configured in my lab (which may be a while), I am going to suggest you do it the way it is documented in all of the Microsoft KB articles that mention it. None of them tell you that it is case sensitive, but all of them present it as upper case K, A and T. KeepAliveTime. If you do it that way, and it winds up that it is case sensitive for some applications and not for others, it will still work unhindered. It won't be rejected if the application is not case aware if you have it as KeepAliveTime and I have yet to run into an application that rejected it when it was set that way. However, I have worked cases where we seem to get a different behavior when it is in the correct case than when it isn't. Full disclosure though, there may have been other mitigating factors, which is why I ultimately want to prove or disprove it in my lab. I will update this blog should that ever happen. That's pretty much everything I know about the KeepAliveTime value, I hope you find this information useful.]]> How did that 1.5GB message get into my Exchange environment anyway? https://docs.microsoft.com/archive/blogs/exchangechallengeaccepted/how-did-that-1-5gb-message-get-into-my-exchange-environment-anyway Sat, 15 Apr 2017 01:07:03 GMT https://blogs.technet.microsoft.com/exchangechallengeaccepted/?p=465 ISSUE: Recently, I had a customer call in an issue where their transport services were failing on all Exchange 2013 servers usually within minutes of each other. Transport would restart on its own and then 15 minutes to an hour later, they would crash yet again. The following errors were showing up in the Event logs:
Log Name:      Application Source:        MSExchangeTransport Date:          4/14/2017 7:41:23 AM Event ID:      17017 Task Category: Storage Level:         Error Keywords:      Classic User:          N/A Computer:      Server1.contoso.com Description: Transport Mail Database: Quota was exceeded while performing a database operation. The Microsoft Exchange Transport service is shutting down. Exception details: Microsoft.Isam.Esent.Interop.EsentTransactionTooLongException: Too many outstanding generations between JetBeginTransaction and current generation.    at Microsoft.Isam.Esent.Interop.Api.JetSetColumn(JET_SESID sesid, JET_TABLEID tableid, JET_COLUMNID columnid, Byte[] data, Int32 dataSize, Int32 dataOffset, SetColumnGrbit grbit, JET_SETINFO setinfo)    at Microsoft.Exchange.Transport.Storage.DataStreamImmediateWriter.Write(Int64 position, Byte[] data)
Log Name:      Application Source:        MSExchange Common Date:          4/14/2017 7:41:24 AM Event ID:      4999 Task Category: General Level:         Error Keywords:      Classic User:          N/A Computer:      Server1.contoso.com Description: Watson report about to be sent for process id: 32864, with parameters: E12N, c-rtl-AMD64, 15.00.1104.005, edgetransport.exe, KERNELBASE.dll, 8b9c, c0020001, 3cb8, 6.3.9600.17031 (winblue_gdr.140221-1952). ErrorReportingEnabled: True %14
Log Name:      Application Source:        MSExchangeTransport Date:          4/14/2017 7:38:19 AM Event ID:      15004 Task Category: ResourceManager Level:         Warning Keywords:      Classic User:          N/A Computer:      Server1.contoso.com Description: The resource pressure increased from Medium to High.
The following resources are under pressure: Version buckets = 2585 [High] [Normal=1750 Medium=2000 High=2500]
The following components are disabled due to back pressure: Inbound mail submission from Hub Transport servers Inbound mail submission from the Internet Mail submission from Pickup directory Mail submission from Replay directory Mail submission from Mailbox server Mail delivery to remote domains Content aggregation Mail resubmission from the Message Resubmission component. Mail resubmission from the Shadow Redundancy Component
The following resources are in normal state: Queue database and disk space ("D:\TransportDB\data\Queue\mail.que") = 66% [Normal] [Normal=95% Medium=97% High=99%] Queue database logging disk space ("D:\TransportDB\data\Queue\") = 66% [Normal] [Normal=95% Medium=97% High=99%] Private bytes = 4% [Normal] [Normal=71% Medium=73% High=75%] Physical memory load = 67% [limit is 94% to start dehydrating messages.] Submission Queue = 0 [Normal] [Normal=2000 Medium=10000 High=15000] Temporary Storage disk space ("D:\TransportDB\data\Temp") = 66% [Normal] [Normal=95% Medium=97% High=99%]
In the Protocol logs for SmtpSend, we could see several 452 4.3.1 status codes where the system had insufficient system resources. All of these showed up on Shadow Copy traffic or XProxyFrom traffic.

CAUSE:

One of the first things we noticed was the Back Pressure event (15004) was happening about 3 minutes before the 17017 event every time. So we knew the MSExchange Transport service would crash in about 3 minutes after the 15004 event fired off. We decided we needed to see what was killing the transport service so we ran procdump. We got the pid of the edgetransport.exe process and then waited until the next 15004 event fired. At that point we ran the following command from an elevated command prompt: procdump -ma -s 60 -n 3 <edgetransport.exe pid> This captured 3 dumps, 60 seconds apart (since we knew the process would crash in 3 minutes). We also did a different dump to catch it upon crashing using the following command (You can change the path at the end of the command to something relevant for your system): procdump.exe -e 1 -ma -f *EsentTransactionTooLongException* edgetransport.exe -accepteula C:\dumps\crashdump.dmp We sent those dumps to our Escalation Engineer and ultimately he came back saying that a large message, over 1.5GB, was trying to come in, getting through the receive connector and then blowing up the Transport service when trying to proxy to the backend servers. He identified the message telling us the server name and IP that sent the message, the Exchange server name and IP it was trying to proxy to,   the Exchange server IP that had received the message from the sender, and the name of the connector that was proxying the message (not the receive connector the message came in under). The sending server the dump identified was an application server trying to deliver the message through the relay connectors in Exchange. The Escalation Engineer then provided us the messageID and sender of the problematic message: <2017041410354.AB1234@appserver1.contoso.com> App1@appserver1.contoso.com So, we have a really large message that has come in to the Exchange environment and was wreaking havoc on it.

How did that 1.5GB message get into my Exchange environment anyway?

This is the question that hounded the customer the most. How did this message get in? I mean, their relay connectors and their Default Connectors all had 20MB message limits. Why did that not stop the message from even making it into transport? Well, to answer that, we looked into all of the relay connectors to see which one was being used by the IP of the Application server. We found it on Relay3. We took a closer look at the relay and found that the relay receive connector under the Security tab, had been set up with Externally Secured for the authentication security mechanism, and Exchange servers as the Permission group. Externally secured requires you to set the Permission group to Exchange servers because that is what it treats those connections like. Anything coming in under this connector will be treated like an internal Exchange Server. It will assume the sender is 100% trusted and will not apply any checks to it, including the checks on message size. So that is how the 1.5GB message got in. Now, because the message had gotten in, and Transport was crashing before the message could deliver, the sending server, an Unix server using SendMail, never got a response from Exchange that the message had been delivered, nor did it get an error. All it got was a severed connection due to the transport service crashing. So it waited a few minutes and then resubmitted the message. Depending on the configuration of the sendMail server, it could potentially do that indefinitely, which was the case in our scenario. So the cycle of crashing transport could have gone on forever if left unchecked.

RESOLUTION:

We spent a lot of time trying to find the message and remove it from the queues before we realized it was being resubmitted over and over again after every crash. So we finally looked at the receive connectors and realized the issue was with how it was configured. For rules to apply to a receive connector, such as the Maximum receive message size, it simply couldn't be set up with externally secured. So how do we set it up so the application can still send messages through the relay to both the internal and external customers? It is easier than it sounds. First, uncheck all of the Authentication and Permission groups in the current relay connector, and then check the Anonymous users permission group. Save that, then from the Exchange Management Shell, run the following to give the relay receive connector the permissions it needs to send internal and externally: Get-ReceiveConnector “Server1\Relay 3” | Add-ADPermission -User “NT AUTHORITY\ANONYMOUS LOGON” -ExtendedRights “Ms-Exch-SMTP-Accept-Any-Recipient” For good measure, I would then restart the MSExchange Transport service. The next time the sendMail server tries to send that message into Exchange, Exchange will check it's message size and reject it immediately, sending a response over the SMTP connection back to the sending server that the message was rejected due to the message being too large. This would show up in the sendMail servers mail logs as a 5.3.4 status saying: "Message too big for system" or a 5.2.3 status stating: "Message too large". Since the sendMail server received the rejection response, it would discard the message and not try to send it in again.

NOTES:

Information on Anonymous versus Externally Secured permission groups can be found in the following article as well as the steps to set up either of the two types of relays: https://technet.microsoft.com/en-us/library/mt668454(v=exchg.160).aspx

WARNING:

If your distribution groups are set up to only accept mail from "Only senders inside my organization", then anonymous relays will not work for that. In order for an application to send as an internal sender so that it can successfully send to one of those distribution groups, it has to come in as Externally Secured. The other option is to set the distribution group for "Senders inside and outside my organization". Then anonymous relays would work fine and still grant you the protection of the size limitations and other rules. There is another scenario where Externally Secured is needed for a relay. If the application is trying to send from OnPrem to a distribution group that lives in O365 that is set to accept messages only from authenticated users, even though the O365 Tenant has the same name space as the OnPrem environment, the message would need to appear to be internal. When it comes through an anonymous relay, the header is stamped with X-MS-Exchange-Organization-AuthAs: Anonymous and will be rejected as it would not be authorized to send to that distribution group. If the relay is set as Externally Secured, then it comes in a X-MS-Exchange-Organization-AuthAs: Internal and O365 accepts the message. In a nutshell, only use the Externally Secured option when the application is 100% trusted to abide by the policies you have enforced on the Exchange environment. Make sure the application developers know that you have a size limit and make them code with that in mind. Otherwise, make them use the anonymous connectors and inform them they will not be able to send to any Distribution groups that are locked down to only receive from internal senders.

ACKNOWLEDGEMENTS:

Thanks to Kary Wall our friendly Escalation Engineer, Tom Kern and Stephen Gilbert our Exchange Connector SMEs, and Sainath Vijayaraghavan out local team lead, for their help in coming to these conclusions and being a sounding board for me as I worked this case.]]> Owaproxy component is in Inactive State and will not successfully activate https://docs.microsoft.com/archive/blogs/exchangechallengeaccepted/owaproxy-component-is-in-inactive-state-and-will-not-successfully-activate Fri, 14 Apr 2017 23:29:27 GMT https://blogs.technet.microsoft.com/exchangechallengeaccepted/?p=455 ISSUE: My customer put their Exchange 2016 environment into maintenance mode so they could apply bitlocker to their drives. After applying bitlocker following the guidances outlined in the following article, they were not able to set the ServerComponentState to Active for OWA Proxy: https://blogs.technet.microsoft.com/exchange/2015/10/20/enabling-bitlocker-on-exchange-servers/ The customer runs the following command to see whether the state is enabled or not: Get-ServerComponentState ServerName | ?{$_.Component -eq "OwaProxy"} It returns the following: Server                                  Component                               State ------                                     ---------                                      ----- ServerName                      OwaProxy                                  Inactive They then tried to set it manually, using the following command, specifying the Maintenance Requester because that is the one that is actually showing Inactive: Set-ServerComponentState ServerName -Component OwaProxy -Requester Maintenance -State Active We do not get an error but when we do the Get command again, it still shows inactive.

CAUSE:

We noticed in Active Directory, that the OwaProxy.Maintenance values in msExchComponentStates had two opposing settings that looked as such: 1:OwaProxy:Maintenance :0:636264117313332389 1:OwaProxy:Maintenance :1:636272763659703237 These values will also exist in the registry of the Exchange server. Because it has competing values in there, the server will side with the most restrictive which is to keep it at Inactive.

RESOLUTION:

In Active Directory Users and Computers, go to the computer object of the server that you are having the issue on, and right-click on it and select properties. Then click on the msExchComponentStates attribute. You should see the various component values. From there, delete the OwaProxy values from the attribute. Then go to the registry on the Exchange server and delete up the OWA Proxy reg key using the following process: Backup and delete the reg key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\ServerComponentStates\OwaProxy Allow enough time for Active Directory to replicate and then run the set command again: Set-ServerComponentState ServerName -Component OwaProxy -Requester Maintenance -State Active Finally, test again to see if the component is now active: Get-ServerComponentState DENXCHMBXC1N4P -Component OwaProxy | fl The component value should then be recreated in AD and in the registry reflecting the proper value for the active state.]]> Mailbox migration times out when going from Exchange OnPrem to O365 https://docs.microsoft.com/archive/blogs/exchangechallengeaccepted/mailbox-migration-times-out-when-going-from-exchange-onprem-to-o365 Fri, 14 Apr 2017 22:39:45 GMT https://blogs.technet.microsoft.com/exchangechallengeaccepted/?p=445 ISSUE: When attempting to migrate an onPrem User mailbox from Exchange 2013 to O365, the following error is generated and the migration fails: The request channel timed out while waiting for a reply after 00:00:00.0012345. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.

CAUSE:

This typically happens when the ExchangeGUID property of the mail-enabled user in O365 does not match the ExchangeGUID property of the corresponding on-premises mailbox that you are trying to migrate. The migration will then timeout because it is trying to look up the wrong ExchangeGUID and will never find it.

RESOLUTION:

Set the ExchangeGUID property of the mail-enabled user in Exchange Online (O365) to match the ExchangeGUID property of the corresponding on-premises mailbox, and then retry the move. To do this, follow these steps:
  1. Retrieve the value of the ExchangeGUID property for the on-premises mailbox that you want to move. To do this, open the Exchange Management Shell on the on-premises server, and then run the following command:

    Get-Mailbox <MailboxName> | Format-List ExchangeGUID

    Make a note of the value. You'll need it later in step 2B.

     
  2. Connect to Exchange Online by using remote PowerShell. Then, do the following:

    A. Determine the value of the ExchangeGUID property of the mail-enabled user in Exchange Online. To do this, run the following command:

    Get-MailUser -Identity <UserName> | fl displayname,ExchangeGuid

    B. Set the value of the ExchangeGUID property of the mail-enabled user in Exchange Online to match the ExchangeGUID property of the on-premises mailbox. To do this, run the following command:

    Set-MailUser -Identity <UserName> -ExchangeGUID <Value Retrieved From Step 1>

  3. Retry the move.

NOTES:

There are other similar issues but I see this one more than any other when it comes to failed migrations, so I am singling it out for the purpose of this article. If this does not address your issue, please refer to TechNet for other possibilities.]]> Mail not routing to on-premise Exchange 2013 environment from O365 with 450 4.7.320 Certificate validation failed error https://docs.microsoft.com/archive/blogs/exchangechallengeaccepted/mail-not-routing-to-on-premise-exchange-2013-environment-from-o365-with-450-4-7-320-certificate-validation-failed-error Fri, 14 Apr 2017 22:23:42 GMT https://blogs.technet.microsoft.com/exchangechallengeaccepted/?p=435 Issue: User is trying to send email from his O365 Tenant in to the OnPrem Exchange 2013 environment. The email fails with the following reason (as seen in the message tracking logs in O365: Reason: [{LED=450 4.7.320 Certificate validation failed};{MSG=SubjectMismatch};{FQDN=hybrid.domain.com};{IP=###.###.###.###};{LRT=4/14/2017 1:07:17 PM}]. OutboundProxyTargetIP: ###.###.###.###. OutboundProxyTargetHostName: hybrid.domain.com

Cause:

We found that the certificate it was trying to use for TLS in the Exchange OnPrem space was not the certificate that was being presented by the O365 connector. By looking in the SmtpReceive protocol logs in Exchange 2013 onPrem, we saw the following: 2017-04-14T17:00:41.125Z,SERVER1\Default Frontend SERVER1,08D479E14521204D,20,172.20.102.10:25,155.39.36.248:30648,*,,Sending certificate 2017-04-14T17:00:41.125Z,SERVER1\Default Frontend SERVER1,08D479E14521204D,21,172.20.102.10:25,155.39.36.248:30648,*,CN=SERVER1,Certificate subject 2017-04-14T17:00:41.125Z,SERVER1\Default Frontend SERVER1,08D479E14521204D,22,172.20.102.10:25,155.39.36.248:30648,*,CN=SERVER1,Certificate issuer name 2017-04-14T17:00:41.125Z,SERVER1\Default Frontend SERVER1,08D479E14521204D,23,172.20.102.10:25,155.39.36.248:30648,*,1234567890ABCDEF1234567890ABCDEF,Certificate serial number 2017-04-14T17:00:41.125Z,SERVER1\Default Frontend SERVER1,08D479E14521204D,24,172.20.102.10:25,155.39.36.248:30648,*,9876534210FEDCBA9876534210FEDCBA123456789,Certificate thumbprint 2017-04-14T17:00:41.125Z,SERVER1\Default Frontend SERVER1,08D479E14521204D,25,172.20.102.10:25,155.39.36.248:30648,*,SERVER1;SERVER1.domain.com,Certificate alternate names The certificate with the thumbprint of 9876534210FEDCBA9876534210FEDCBA123456789 was a self signed exchange certificate in the OnPrem environment and did not have the SAN name of Hybrid.domain.com in it. The correct certificate is a third party SAN certificate which was already installed in exchange but for whatever reason was not being used by the Default Frontend Receive Connector. For this articles sake, we will say that certificate had a thumbprint of 4C0FA622D0D66E777AA123B1AF123456F001AE23. We did a Get-ReceiveConnector "SERVER1\Default Frontend SERVER1" | fl TlsCertificateName and saw that it was set to Null.

Resolution:

We resolved this by adding the TlsCertificateName for the Default Frontend Receive connector using the following commands: $tlscert=Get-ExchangeCertificate 4C0FA622D0D66E777AA123B1AF123456F001AE23 $tlscertname="<I>$($TLScert.Issuer)<S>$($TLSCert.Subject)" Get-ReceiveConnector "SERVER1\Default Frontend SERVER1" | Set-ReceiveConnector -TlsCertificateName $tlscertname We then bounced the transport service and the issue remediated.

NOTE:

This issue can also occur in Exchange 2016. The fix is the same for both.]]> Email destined for recently moved Modern Public Folders are going to the Poison queue. https://docs.microsoft.com/archive/blogs/exchangechallengeaccepted/email-destined-for-recently-migrated-modern-public-folder-are-going-to-the-poison-queue Wed, 01 Feb 2017 22:05:21 GMT https://blogs.technet.microsoft.com/exchangechallengeaccepted/?p=425 Scenario I recently worked a case where the customer had moved all of the public folders from one Modern Public Folder mailbox to another and deleted the old Public Folder mailbox. This was on Exchange 2013 CU9. After he did that, email destined for what were previously mail-enabled Public Folders started to show up in the Poison queue. When we did a get-mailpublicfolder against one of the Public Folders (lets call it pf using SMTP address pf@instant-grits.com) we found the following: ContentMailbox      : instant-grits.com/Deleted Objects/PFMbx01 The correct email address is specified under the EmailAddresses property. So here's the problem; Exchange still sees the mailbox content as being on the Deleted Object, aka the old Public Folder Mailbox server. We also noted that the new public folder was not automatically being mail-enabled after being migrated from one Public Folder Mailbox to the other. So we tried to mail-enable it using the enable-mailPublicFolder. It mail-enabled, but the email addresses it assigned were enumerated, so not the original email address. This will cause mail stuck in the poison queue to never be able to successfully deliver to the new folder.

Why is this happening to me?

In a nutshell, the Microsoft Exchange System Object (MESO) that exists for that public folder is still attached to the deleted public folder. Since the MESO holds the mail properties for a mail-enabled object, mail-enabling the new folder will not be able to use the old address. Since pf@instant-grits.com is not available, the new object gets an enumerated version of it, so pf1@instant-grits.com. Since that does not match the address, and there is no X500 pointed to the deleted folder, Exchange does not know how to deliver it. Since the old address does exist, Exchange will try, but it will then go to the Poison queue as the Folder it is trying to go to is a deleted object.

So what do I do about it?

As always, you ask the best questions. Let's assume the public folder is in the following location in the public folder tree: \thingsthatgowithgrits\shrimp\pf. There are 5 steps to cleaning this up and getting your messages in the Poison queue and new messages to deliver successfully to the Public Folders in the new Public Folder Mailbox:
  1. Mail-disable all of the public folders in the new Public Folder Mailbox. You can do this from the Exchange Admin Center (EAC), or by using the disable-mailpublicfolder cmdlet using the full public folder path of the public folder, like such:disable-mailpublicfolder "\thingsthatgowithgrits\shrimp\pf"
  2. Once all of the public folders in the new Public Folder Mailbox have been mail-disabled, you need to find and delete all of the MESO objects that have the same name as the public folders you are cleaning up. Make sure you capture the SMTP address of those MESO objects. They should be in the EmailAddresses property of the get-mailpublicfolder cmdlet you ran earlier. A quick script that Bill Long wrote and I modified very slightly can dump all of the MESO objects to a Text file. From there you can sort it and go to Active Directory Users and Computer. Make sure you have it in Advanced view. Then browse to the Microsoft Exchange System Objects container, which is at the root of the domain, and delete all of the appropriate MESO objects. The script looks as follows:$mesoDN = "CN=Microsoft Exchange System Objects,DC=instant-grits,DC=com" $mesoContainer = [ADSI]("LDAP://" + $mesoDN) $sysMbxFinder = new-object System.DirectoryServices.DirectorySearcher $sysMbxFinder.SearchRoot = $mesoContainer $sysMbxFinder.PageSize = 1000 $sysMbxFinder.Filter = "(cn=*)" $sysMbxFinder.SearchScope = "OneLevel" $sysMbxResults = $sysMbxFinder.FindAll() $sysMbxResults.Path > c:\MESOObjects.txt
  3. Once all of the appropriate MESO objects have been deleted, wait until Active Directory has finished replicating the change. For single site environments, this is pretty instant. For multi-site environments, it will depend on your AD topology. It will take as long as your longest and slowest route. Ask your network guy how long that should be.
  4. Now that we are fully replicated, we can mail-enable the public folders. You can do this from the Exchange Admin Center (EAC), or by using the enable-mailpublicfolder cmdlet using the full public folder path of the public folder, like such:enable-mailpublicfolder "\thingsthatgowithgrits\shrimp\pf"Using the enable-mailpublicfolder cmdlet in my lab has produced mixed results, sometimes it works, sometimes it doesn't, likely due to AD replication or my Exchange environment being on too old of a Cumulative Update (CU). So be prepared to do it manually from the EAC.
  5. Once all of the public folders that are going to be mail-enabled have been, you can then go into the queue viewer and browse the poison queue. From there you should be able to select all messages and then click on Resume. If the resume fails for any reason, you either missed cleaning up one or more of the public folders, or there are messages that are in the poison queue for some other reason. At that time you are ready to bounce the transport service. You can do that from the Exchange Management Shell with the following command: stop-service MSExchangeTransport then start-service MSExchangeTransport After restarting the queue, you can wait for the Poison Queue to reattempt deliver on its messages. All that have properly cleaned up public folders should then deliver.

Note:

I do strongly urge you to get and keep your Exchange servers at a fully supported cumulative update level. That level is N-1. So the latest version minus 1. If we are currently at CU16, then you need to be at least on CU15.

Special Thanks:

Thanks to Bill Long for his impressive blogs on the topics of MESO objects. The one in particular that I referenced in this blog is as follows: https://bill-long.com/2014/01/11/cleaning-up-microsoft-exchange-system-objects/ Also thanks Matthew Huynh for offering up his lab to me so I can test the scenario.

Update:

I have tested this scenario in CU15 and I could not reproduce it. So before you move any public folders across public folder mailboxes, it is highly recommended that you get to the latest Cumulative Update first.]]> O365 Migration that completes with warnings due to insufficient rights. https://docs.microsoft.com/archive/blogs/exchangechallengeaccepted/o365-migration-that-completes-with-warnings-due-to-insufficient-rights Fri, 27 Jan 2017 05:02:28 GMT https://blogs.technet.microsoft.com/exchangechallengeaccepted/?p=405 The Problem: With the rise in Hybrid configurations around the world, we are starting to see some regular occurrences of issues around migrating from OnPremises exchange (called OnPrem henceforth) to O365. Perhaps the most common is one where most users migrate fine, but a select few only partially migrate. When you look at the migration logs, they typically look as follows: FailureType       : UpdateMovedMailboxPermanentException FailureHash       : 1304 FailureCode       : -2146233088 MapiLowLevelError : 0 FailureSide       : Source FailureSideInt    : 1 ExceptionTypes    : {Exchange, MRS, MRSPermanent, MRSUpdateMovedMailbox...} Message           : An error occurred while updating a user object after the move operation. --> Active Directory operation failed on DC1.instant-grits.com. This error is not retriable. Additional information: Insufficient access rights to perform the operation. Active directory response: 00002098: SecErr: DSID-03150BB9, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0 --> The user has insufficient access rights. So basically, the user now has a mailbox in the cloud and a mailbox onPrem, because the migration did not fully complete. They can only access their mailbox in the cloud using OWA.

Why?

In most every case, the user you are trying to migrate either belongs to a protected group of some sort or simply does not have permissions inheritance enabled. The protected groups, by design, do not inherit permissions in Active Directory (AD). Exchange OnPrem needs to have specific permissions set in order for a mailbox migration to O365 to complete. Specifically, it needs to be able to convert the OnPrem mailbox to an OnPrem remote mailbox and link it to the O365 mailbox. For Exchange to do that, two groups have to have rights on the mailbox: Exchange Trusted Subsystem Exchange Servers These rights are typically pushed down from the top of the domain using inheritance, so with inheritance disabled, you simply cannot migrate them without errors.

What can I do about that?

A most excellent question. There are a few options here. The ideal one would be to turn on inheritance for the onPrem mailbox. The problem with that in regards to the protected group members is there is this tiny little process called SDProp that will revert it back to disabled, usually within the hour. Not always long enough for the mailbox to finish migrating. It is a protection mechanism to make sure that the protected group member doesn't suddenly have a bunch of permissions thrust upon it by the OU that it is a member of thereby giving permissions to an OU administrator over the privileged account that it shouldn't have. So what do you do? Well, you can add the two Exchange groups to the user's permissions manually, which is a huge undertaking as you will have to look at a user who does have inheritance turned on to see exactly which rights the Exchange Trusted Subsystem and Exchange Servers groups has on the account and then mimic them on each mailbox you intend to migrate, or you can add them manually once to the AdminSDHolder container in AD. Or better yet, just enable inheritance on the AdminSDHolder container itself. Sometime in the next hour, it will update all of the protected group members at once, and then you can migrate them. Once migrated, you can turn inheritance back off and up to an hour later, all of those accounts would have it removed for them as well and they would revert back to their original permissions. enableinheritance Once the permissions have been inherited, Exchange will then have adequate permissions to convert the mailbox to a remote mailbox and link it to the new O365 mailbox. Once that has completed, the user should be able to access their O365 mailbox from Outlook and OWA.

Note

For more information on AdminSDHolder, SDProp and Protected Groups, check out the following TechNet article: https://technet.microsoft.com/en-us/library/2009.09.sdadminholder.aspx There is another option which involves manually completing the process for these accounts. That article can be found here: https://support.microsoft.com/en-us/kb/2745710 One of our partners has an addendum to the support article that goes into some additional steps which will help preserve your proxy addresses in that process: https://blogs.perficient.com/microsoft/2014/10/office-365-mailbox-fails-to-convert-during-migration/

THANKS

Special thanks to Raymond Fong for helping me iron out this issue and to Perficient for their excellent blog.]]> No properties available on the Exchange Org container in AD Sites and Services https://docs.microsoft.com/archive/blogs/exchangechallengeaccepted/no-properties-available-on-the-exchange-org-container-in-ad-sites-and-services Tue, 13 Sep 2016 20:59:26 GMT https://blogs.technet.microsoft.com/exchangechallengeaccepted/?p=325 Scenario: So I ran into an odd case this week where the customer could not see the properties of their Exchange Org container in AD Sites and Services. Not only could they not see it, it's icon, which should have been a folder icon, was actually the notorious Unknown icon. It looked as such: ADS&S-IncorrectView2 When they right-clicked on the org container, they were able to click on properties, but it returned a blank window, similar to the following: AS&S-BrokenProperties or ADS&S-DisplaySpecifierIssue2 (redacted customer's org name) As I am not an AD engineer, I reached out to our Directory Services team and worked with an Engineer named Gary. After some research, he said he felt like the issue was with the Display Specifiers in AD. With that in mind, he had the customer take a look at the Display Specifiers in ADSI Edit. Sure enough, under CN=409, which is the Display Specifier for the English language, there was an exchange object called msExchOrganizationContainer-Display. ADS&S-ADSIEditDisplaySpecifiers Exchange objects in the Display Specifier are from legacy versions of exchange and are not supported. By the object existing, it was breaking the view of the org container in AD Sites and Services.

So what do we do about it?

Simply put, you delete the object in ADSI Edit. Once we deleted the object, and AD replicated, we closed and re-opened AD Sites and Services and the Exchange Org container then appeared with the proper folder icon. ADS&S-CorrectView The properties were then showing properly as well. AS&S-FixedProperties

NOTES:

As always use extreme caution when deleting anything from ADSI Edit. One wrong move could render your entire AD and exchange environment useless. Information on Display Specifiers can be found in MSDN at the following URL: https://msdn.microsoft.com/en-us/library/ms675905(v=vs.85).aspx

Acknowledgements:

Special thanks to Gary Mudgett (garymu) on his help with this case. I had never heard of Display Specifiers prior to this and would likely have taken a really long time to figure this out. One Microsoft FTW.  ]]> Only seeing first receive event when doing message tracking against a public folder that was migrated to O365 from Legacy Exchange https://docs.microsoft.com/archive/blogs/exchangechallengeaccepted/only-seeing-first-receive-event-when-doing-message-tracking-against-a-public-folder-that-was-migrated-to-o365-from-legacy-exchange Thu, 25 Aug 2016 19:15:16 GMT https://blogs.technet.microsoft.com/exchangechallengeaccepted/?p=316 receive event when running message tracking against the legacy namespace.

Scenario

A Public Folder with a Primary SMTP email address of PF1@contoso.com gets migrated to O365. You are testing mail flow from an application that sends through your On-Premises environment. When you kick off message tracking and list PF1@contoso.com as the recipient, the logs will return the first receive event, but you do not see any sends or any other events.

Why is this happening?

First of all, when you send a message to a migrated public folder from on-premise Exchange, the first thing it does when received by the Exchange server is to kick off a process called the PFResolver. This process locates the O365 address, something like PF1@contoso.mail.onmicrosoft.com, and then reroutes the message to that address, stamping it as the recipient. It then adds an ORCPT attribute to the RCPT TO: entry and populates that with the legacy address. So in the protocol logs it would look as such: 2016-08-07T23:59:55.790Z,Outbound to Office 365,08D14E32D1DC272C,42,10.0.0.1:57888,207.46.163.138:25,>RCPT TO:<PF1@contoso.mail.onmicrosoft.com> ORCPT=rfc822;PF1@contoso.com Message tracking only uses the RCPT TO: address. It does not look for the ORCPT address. So when you search on the Legacy address, the only message that has that is the initially sent message when it is first received by Exchange before the PFResolver is run.

So what do I do to see the whole message trace?

In a nutshell, you have to run two separate message traces. First, one with the Legacy email address, and second, on with the O365 address. I.E.: Get-MessageTrackingLog -Recipients "PF1@contoso.com" Get-MessageTrackingLog -Recipients "PF1@contoso.mail.onmicrosoft.com" This will give you the initial receive and then all other events where the message had been resolved to the O365 address.]]> Event 1035, MSExchangeTransport - Inbound authentication failed with error UnexpectedExchangeAuthBlob for Receive connector https://docs.microsoft.com/archive/blogs/exchangechallengeaccepted/event-1035-msexchangetransport-inbound-authentication-failed-with-error-unexpectedexchangeauthblob-for-receive-connector Wed, 24 Aug 2016 23:42:47 GMT https://blogs.technet.microsoft.com/exchangechallengeaccepted/?p=305 Scenario Ran into an interesting case today where mail flow had quit working in one direction between sites. The environment was on Exchange 2010. Basically, Site A and Site B could not send mail to an exchange server in Site C. They could send to each other, and mail was flowing out from Site C, but inbound mail to Site C was queueing up. The error in the Application event logs looked as follows: Log Name:      Application Source:        MSExchangeTransport Date:          8/24/2016 1:21:43 PM Event ID:      1035 Task Category: SmtpReceive Level:         Warning Keywords:      Classic User:          N/A Computer:     SiteCserver.contoso.com Description: Inbound authentication failed with error UnexpectedExchangeAuthBlob for Receive connector Default SiteCServer. The authentication mechanism is ExchangeAuth. The source IP address of the client who tried to authenticate to Microsoft Exchange is [IP of Site A exchange server (or site B)]. Event Xml: <Event xmlns="https://schemas.microsoft.com/win/2004/08/events/event">   <System>     <Provider Name="MSExchangeTransport" />     <EventID Qualifiers="32772">1035</EventID>     <Level>3</Level>     <Task>1</Task>     <Keywords>0x80000000000000</Keywords>     <TimeCreated SystemTime="2016-08-24T17:21:43.000000000Z" />     <EventRecordID>10489441</EventRecordID>     <Channel>Application</Channel>     <Computer>SiteCserver.contoso.com</Computer>     <Security />   </System>   <EventData>     <Data>UnexpectedExchangeAuthBlob</Data>     <Data>Default SiteCServer</Data>     <Data>ExchangeAuth</Data>     <Data>[IP of Site A exchange server (or Site B)]</Data>   </EventData> </Event> There was no time skew in the environment and no backpressure events were showing up (Event IDs 15001 -15007) in the Application event logs.

Resolution

Normally, I would spend a great deal of time working through Protocol logs and network traces to figure out what is happening here, but luckily for me, I have an engineer named Miguel Ortiz on my team and he has seen this before. In this particular case, what was happening was, the Kerberos token from Site A and Site B destined for Site C had gotten corrupted. Site C had suffered a severe outage and we believe it was during this time that the Kerberos tokens became corrupted. We went through the following steps to remediate the issue:
    1. On the Site C domain controllers, we restarted the Kerberos Key Distribution Center service (KDC).
    2. On the Site C Exchange server we purged the Kerberos tickets using the KList Purge command from an administrative command prompt
      1. klist purge
    3. Next, we restarted the Netlogon service on the Site C Exchange server.
      1. Net Stop Netlogon && Net Start Netlogon
    4. Then we restarted the Transport Service on the Site C Exchange server.
      1. Net Stop MSExchangeTransport && Net Start MSExchangeTransport
    5. After transport came back up on Site C, we then repeated steps 2 through 4 on the Site A and Site B servers.
This process basically clears out the Kerberos tickets and forces Site A and Site B to re-send their Kerberos request to Site C. In other words, they cleared the corrupted token and requested a new one. Once this process completed, mail flow returned to normal and no further 1035 events were thrown.

Special Thanks

Thanks to Miguel Ortiz for helping me get through this case quickly and working out the steps to resolve it.  ]]>