The Adventures of an Exchange Engineer

Tales of the Weird: Unable to Send TLS-Encrypted email to Office 365

Eric Whitehill — Thu, 30 Aug 2018 12:42:42 GMT

Here is another case from the "Tales of the Weird and Unusual" bin at Microsoft Exchange Support. This article includes a little bit of information from both the Office 365 and the Exchange On-Premise worlds, which is a little bit different as most of this blog focuses in solely on Exchange On-Premise. Over the last week or so we have seen an uptick in cases where our customers are having difficulties sending emails to Office 365 from their Exchange On-Premise environment when using TLS. This does not seem to affect just one version of Exchange which also can complicate matters. Where we are seeing this happen is right after we send over the SSL certificate. Most customers will see this as their mail queue building up, and most likely it'll be triggered by either a call to your help desk or your SCOM alerts. Typically the first thing you will do is check your mail queue via the Get-Queue command and you'll see something similar to below:

RunspaceId : RunSpaceID_Would_Go_Here DeliveryType : DnsConnectorDelivery NextHopDomain : Contoso.mail.onmicrosoft.com TlsDomain : NextHopConnector : 00000000-0000-0000-0000-000000000000 Status : Active MessageCount : MessagesInQueueForOffice365 LastError : [{LED=451 4.4.397 Error communicating with target host. -> 421 4.4.2 Connection dropped due to TimedOut};{MSG=};Contoso-mail-onmicrosoft-com.mail.protection.outlook.com};{IP=Office365IPAddressHere};{LRT=8/20/2018 12:00:39 AM}]

When I duplicated this using a SmartHost in my lab we noted the following:

Note, I have changed/removed the RunSpaceID's and NextHopConnectors to protect the innocent. You would see your RunSpaceId's, etc in your own environments. Now, the next step in troubleshooting is to enable our verbose protocol logging via the following command:

Set-SendConnector "Outbound to Office 365" -ProtocolLoggingLevel verbose

This will allow us to see what is happening from the Send Connector. Most often for this command, your send connector will be "Outbound to Office 365", however for some organizations it will be different. Once you have the Send Connector Protocol Logging enabled you can grab your Protocol Logs which by default in Exchange 2013/2016 will be in %ExchangeInstallPath%TransportRoles\Logs\Hub\ProtocolLog\SmtpSend. Again, you can customize this to suit your needs. * Once you have the logs, you can easily import the logs into Excel to review as they are just .csv files. When you import them, select delimited, comma format, with your import starting at line 5 and you will be able to filter down your connections. Now, depending upon your error, you may see different behavior. However, for what we are focusing in on today is the "TLS negotiation failed with error TimedOut". Here's a snippit from the log in my lab:

date-time connector-id session-id sequence-number local-endpoint remote-endpoint event data context 2018-08-12-T12:4612.310Z Outbound to Office 365 1 2 192.168.1.1:45874 52.164.999.999:25 + 2018-08-12-T12:4612.342Z Outbound to Office 365 1 3 192.168.1.1:45874 52.164.999.999:25 < "220 mail.protection.outlook.com Microsoft ESMTP MAIL Service ready at Sun, 12 Aug 2018 12:46:19 +0000" 2018-08-12-T12:4612.342Z Outbound to Office 365 1 4 192.168.1.1:45874 52.164.999.999:25 > EHLO mail.contoso.com 2018-08-12-T12:4612.408Z Outbound to Office 365 1 5 192.168.1.1:45874 52.164.999.999:25 < 250 mail.protection.outlook.com Hello [192.168.1.1] 2018-08-12-T12:4612.409Z Outbound to Office 365 1 6 192.168.1.1:45874 52.164.999.999:25 > STARTTLS 2018-08-12-T12:4612.530Z Outbound to Office 365 1 7 192.168.1.1:45874 52.164.999.999:25 < 220 2.0.0 SMTP server ready 2018-08-12-T12:4612.530Z Outbound to Office 365 1 8 192.168.1.1:45874 52.164.999.999:25 * " CN=mail.contoso.com.com, OU=MyOU, OU=Domain Control Validated CN=MyCNand CA, O=Contoso, " Sending certificate Subject Issuer name Serial number Thumbprint Not before Not after Subject alternate names 2018-08-12-T12:4631.547Z Outbound to Office 365 1 9 192.168.1.1:45874 52.164.999.999:25 * TLS negotiation failed with error TimedOut 2018-08-12-T12:4631.648Z Outbound to Office 365 1 10 192.168.1.1:45874 52.164.999.999:25 - Remote

Now, it looks like our connection was just timing out - we were sending our SSL certificate, however we were never getting a reply from Office 365. Then, our next step was we tried a telnet from our Exchange server. We should see:

Trying 52.164.999.999... Connected 52.164.999.999. Escape character is '^]'. 220 mail.protection.outlook.com Microsoft ESMTP MAIL Service ready at Fri, 24 Aug 2018 14:12:07 +0000 EHLO contoso.com 250 mail.protection.outlook.com Hello [192.168.1.1] STARTTLS 220 2.0.0 SMTP server ready

But instead, we saw nothing.

We ended up taking a network capture, and in our network capture we were seeing us connect and send our certificate. However, we would never see a reply back from the service. Many times in cases like this, we do end up taking concurrent network captures - one from Exchange, and then two from our firewall device - one capture on the internal interface, and then another on the external interface. What we were seeing is the traffic was coming back from Office 365, but our firewall device instead was dropping the packets and never sending the reply back to Exchange. This would show up sometimes as a [RST, ACK, CWR] flag in our network traces. (The key being the congestion window reduced.) What we found was that in some of these cases the firewall was doing what is commonly referred to as SMTP inspection, along with the TLS component can sometimes trip up certain firewalls. In our case we were seeing Exchange attempt to negotiate TLS and then we immediately send a number of TCP retransmissions. Then, we see the connection reset. When we checked the logs in Office 365, we were never seeing the client certificate logged, which was then indicating the failure is below the application layer. Once we either bypassed the firewall device or disabled SMTP and TLS inspection, the issue was resolved. *NOTE: Make sure you disable your protocol logging via the Set-SendConnector "Outbound to Office 365" -ProtocolLoggingLevel none command when you are done troubleshooting, otherwise you could see your disk space usage jump dramatically quickly. Special thanks to the following for their assistance in writing this blog: Nick Ulmer, Exchange Support Escalation Engineer Shawn Bakken, Exchange Support Escalation Engineer Tom Kern, Exchange Online Senior Support Escalation Engineer

Tales of the Weird: One User Not Able to use any ActiveSync Device or OWA will not work.

Eric Whitehill — Fri, 10 Aug 2018 07:06:25 GMT

It's been a while, so here's another case from the "Tales of the Weird and Unusual" bin at Microsoft Exchange Support. I recently was working a case with a customer where they were having issues with one user syncing their ActiveSync device to Exchange. They were an environment which was primarily on Exchange 2010, however in our testing it would have affected Exchange 2013 or Exchange 2016 as well. What we discovered about this user is that no matter what device they had for ActiveSync, they would be unable to get everything to their ActiveSync device. We also tried to connect to the Mailbox in OWA and we would get log in, but then the web page would just sit there and spin. Normally when we have one user who cannot sync like this, two big items come to the forefront of what can cause it.

1. Issue with the load balancer

2. Corruption within the mailbox.

We then bypassed the load balancer, and we were getting the same results. Because no other users were reporting issues (for example, in a server performance case) we migrated the mailbox to a new database on a different server. When we move a mailbox we do a corruption check, and thus if we have any corrupted items those can be identified. However, after the mailbox move, we were still encountering the same issues - Active Sync would not complete, and we could not use the mailbox in OWA. However, Outlook in cached mode worked just fine. At this point we enabled Active Sync Mailbox Logging which proved useful. ActiveSync Mailbox Logging often times will scare people because of the verbosity and type of information we are able to get from it. However it can be very useful when troubleshooting sync issues such as these, as the Active Sync protocol is well documented and in some cases, we can see what happens from the mailbox side of things when gathering IIS logs is not prudent or useful. When we reviewed the Active Sync logs, we kept seeing the following repeatedly:

RequestHeader : POST /Microsoft-Server-ActiveSync/default.eas?User=User@Contoso.com&DeviceId=MYDEVICEIDISREMOVED&DeviceType=WindowsMail&Cmd=FolderSync HTTP/1.1 Connection: keep-alive Content-Length: 13 Content-Type: application/vnd.ms-sync.wbxml Accept: */* Accept-Encoding: br, gzip, deflate Accept-Language: en-us Authorization: ******** Host: 192.168.0.10 User-Agent: MyDeviceID MS-ASProtocolVersion: 14.1 X-MS-PolicyKey: 0 X-Forwarded-For: 192.168.0.1 RequestBody : 0

So, the key here I've highlighted above. What we kept doing was a folder sync repeatedly. When we look at our documentation for why we were seeing this, on https://msdn.microsoft.com/en-us/library/ee237648(v=exchg.80).aspx we state the following:

The FolderSync command synchronizes the collection hierarchy but does not synchronize the items in the collections themselves. This command works similarly to the Sync command (section 2.2.1.21). An initial FolderSync command with a SyncKey element (section 2.2.3.181.2) value of 0 (zero) is required in order to obtain the list of folders and the synchronization key associated with that list. The synchronization key is returned in the SyncKeyelement of the response if the FolderSync command succeeds. This synchronization key is used in subsequent FolderSync commands to obtain folder hierarchy changes. Unlike a Sync request, there is no GetChanges element (section 2.2.3.84) submitted in the FolderSync request (section 2.2.1.5) to get changes from the server. All folders MUST be returned to the client when initial folder synchronization is done with a synchronization key of 0 (zero). The FolderHierarchy namespace is the primary namespace for this section. Elements referenced in this section that are not defined in the FolderHierarchynamespace use the namespace prefixes defined in section 2.2.4.

Simply put, we were not able to complete the FolderSync within the allowed time period. We then went and revisited the user's mailbox to see if we can determine how many folders we were trying to sync. We did this via the Get-MailboxFolderStatistics -Identity user@contoso.com -FolderScope Inbox |Select Name,FolderSize,ItemsInFolder command, but there are a number of different ways you can do this. You will see something similar to below:

In my test lab, we don't have many folders configured, however in our users case we found over 40,000 folders. This in almost all instances is going to cause performance issues when attempting to connect via ActiveSync or OWA, no matter our Exchange version. We have some very well documented limits for Outlook, as available here. We know if we get above 500 folders, then we can have performance issues inside of the Outlook client. We also have limits for Exchange 2010/2013/2016 and Exchange Online. Note, our limits for Exchange 2010 are significantly different from Exchange 2013/2016/Exchange Online! However, in our case for Exchange 2010, which is where this mailbox was hosted, we were significantly over the limits for Exchange 2010. Since Exchange 2010 is also at End of Life we will be migrating to a newer version of Exchange we know migrating a user with over 40,000 folders will be almost impossible. Once we were able to get the user to our guidance and under 500 folders, the issue disappeared and the end user was able to sync again.

Tales of the Weird: High Memory Utilization within Exchange Servers running on Windows Server 2016

Eric Whitehill — Fri, 09 Mar 2018 13:09:08 GMT

Here is another case from the "Tales of the Weird and Unusual" bin at Microsoft Exchange Support. Over the last couple of weeks we have seen a few cases where customers are reporting very high memory utilization within their Exchange 2016 environments, however the catch is all of these customers were running on Windows Server 2016. While memory utilization on Exchange can sometimes take a very serious note in several cases we were seeing where customers were adding in additional RAM into their environments, but still seeing high memory utilization or we were seeing the servers complain about not having enough memory. As a result, we would also see excessive system paging. We may even see error messages in our application logs such as the following:

Time: 1/2/2018 1:15:00 PM ID: 906 Level: Warning Source: ESE Machine: MBX01.contoso.com Message: Information Store - MBXDB1 A significant portion of the database buffer cache has been written out to the system paging file. This may result in severe performance degradation. See help link for complete details of possible causes. Previous cache residency state: 100% (4657 out of 4657 buffers) (867 seconds ago) Current cache residency state: XX% (XXXX out of XXXX buffers) Current cache size vs. target: XX% (XXX / XXX MBs) Physical Memory / RAM size: 96 GB

You may even seen errors such as the following:

Time: 1/1/2018 1:57:29 PM ID: 1009 Level: Warning Source: MSExchangeFastSearch Machine: MBX01.contoso.com Message: The indexing of mailbox database MBXDB1 encountered an unexpected exception. Error details: Microsoft.Exchange.Search.Core.Abstraction.OperationFailedException: The component operation has failed. ---> Microsoft.Exchange.Search.Core.Abstraction.ComponentFailedPermanentException: Failed to read notifications, MDB: MBX-GUID-GOES-HERE. ---> Microsoft.Mapi.MapiExceptionMdbOffline: MapiExceptionMdbOffline: Unable to read events. (hr=0x80004005, ec=1142)

Now, if we are in a virtualized environment, one of the items your storage teams may report is your disk I/O rises dramatically. If you are using high speed storage, such as SSD's, you may not see anything even show up on your disk latencies, but your storage team would notice the increased disk I/O. What you also may see is SystemCacheWs using a great deal of working set memory. If you start seeing the above issues, your issue may be fixed by applying one or more of the items below. Our most common causes are as follows:

Not using static resources within your virtualized host.
Anti-Virus Exclusions not being abided by.
Update to the Windows System when using ReFS.

Static Resources and Virtualization:

We have discussed virtulization in great detail at https://technet.microsoft.com/en-us/library/jj619301(v=exchg.160).aspx, and there have been other articles put out by fellow engineers, but one of the biggest disagreements from customers is when we see is the requirement for static resources within our virtualized host, and why we have to do it. In our TechNet articles we state the following:

Some hypervisors have the ability to oversubscribe/overcommit or dynamically adjust the amount of memory available to a specific guest machine based on the perceived usage of memory in the guest machine as compared to the needs of other guest machines managed by the same hypervisor. This technology makes sense for workloads in which memory is needed for brief periods of time and then can be surrendered for other uses. However, it doesn't make sense for workloads that are designed to use memory on an ongoing basis. Exchange, like many server applications with optimizations for performance that involve caching of data in memory, is susceptible to poor system performance and an unacceptable client experience if it doesn't have full control over the memory allocated to the physical or virtual machine on which it's running. As a result, using dynamic memory or memory overcommit features for Exchange isn't supported.

In non-geek speak, what does this mean? This means we need to make sure we are using static resource allocations within our virtualization environment, otherwise we can see unexpected behavior, which would include (but is not limited to) resource constraints, poor system performance, and client connectivity issues. But, my virtualization team told me we have static resources already! Great! We can check this! Here's how we can check! We can use perfmon, and then look up a specific counter to see if we have static resources allocated to our Exchange servers! If you look for the VM Memory ==> Memory Reservation in MB that counter will show you if you have a static memory allocation for your Exchange server. If you don't, you will see something similar to below: If you see zeros all across, then you know you static resources are not configured, and we will need to configure this within our virtualization environment.

Anti-Virus Exclusions and You:

The other common item we see causing higher memory usage can be Anti-virus software. Modern AV software is very, very good at what it does, to the point we need to make sure we have our exclusions configured correctly, otherwise we can see memory go out of control, among a number of other performance issues. Even if our AV exclusions are configured correctly today, we still should check the exclusions and verify them with an EICAR test from time to time. As discussed in a previous blog post, if our exclusions are not configured correctly, we can see errors in our application logs like this:

Time: 1/1/2017 1:01:10 AM ID: 489 Level: Error Source: ESE Machine: mail.contoso.com Message: msexchangerepl (4456) An attempt to open the file "D:\Databases\DB01.EDB" for read only access failed with system error 32 (0x00000020): "The process cannot access the file because it is being used by another process. ". The open file operation will fail with error -1032 (0xfffffbf8).

Additionally, some of our customers have reported high disk I/O times, however we see no corresponding spike on our storage array.

Great! How can I check if my AV exclusions are not set right!

There are three easy ways we can detect if our AV exclusions are not configured correctly.

Check our application logs for the Event ID 489. If we see we are unable to open files because they are being locked, this is a sign our exclusions are not configured correctly.
Disk performance counters. If we open up performance monitor, and we look at the following counter:
1. MSExchange Database ==> Instances, and then you add in the I/O Database Reads (Attached) Average Latency and I/O Database Writes (Attached) Average Latency, and add in your specific databases that are mounted on your server, as seen below:
2. If you see spikes on your databases, specifically above 50ms, AND we do not see any corresponding spike in disk latency (LogicalDisk --> Average Disk Sec Read Logical Disk --> Average Disk Sec Write counters) then there is a high degree of likelihood we either have reconfigured backups or our Anti-Virus is scanning our databases, resulting in latencies. Because AV has to load our database files somewhere while it scans the metadata and does exactly what it is supposed to do, we can see very high latencies and see very high memory usage which may or may not be attributed to the AV software itself. Because modern AV software runs as part of the system, you will not necessarily see the spike go to the av process.
3. Perform an EICAR test. Place the EICAR test file in one of our database paths for example, and see if your AV software detects it. If it does, then we know we need to adjust our AV configuration.

Again, this isn't the fault of AV doing anything wrong or bad. Modern AV software is very good at what it does, and if the AV software gets into memory, and the filter drivers start to scan your Exchange databases and delay or prevent Exchange from performing certain tasks, this can cause higher memory usage which does not have an attributable process.

ReFS on Windows Server 2016

The last item we have seen is when you are using ReFS with Windows Server 2016. We discuss this here where we state:

To provide greater resiliency for its metadata, the Resilient File System (ReFS) in Windows Server 2016 uses allocate-on-write semantics for all metadata updates. This means that ReFS never makes in-place updates to metadata. Instead, it makes all writes to newly allocated regions.

However, allocating-on-write causes ReFS to issue more metadata I/O to new regions of the volume than write-in-place file systems do. Additionally, ReFS uses block caching logic to cache its metadata in RAM. This is not as resource-efficient as file caching logic.

Together, the ReFS block caching logic and allocate-on-write semantics cause ReFS metadata streams to be large. ReFS uses the cache manager to create the metadata streams, and the cache manager lazily unmaps inactive views. In some situations, this lazy unmapping causes the active working set on the server to grow. This creates memory pressure that can cause poor performance.

In order for us to change these registry settings, make sure you have applied KB 4013429 first. This is the KB which introduces the tune-able parameters. We have released a workaround for ReFS, and when we look at the options for resolution, it's been our experience most of our customers will implement all three options, or some combination thereof. CAUTION: With the workaround, you will be required to edit your registry. Make sure you apply the registry settings correctly, and always take backups.

One of the biggest questions we get is what do those registry changes do? We explain it further on:

To update its metadata, ReFS uses allocate-on-write instead of writing-in-place to improve its resiliency to corruption.

Writing-in-place is susceptible to torn writes. This occurs if a power failure or an unexpected dismount causes a write to be only partially completed.

Allocating-on-write enables ReFS to reliably maintain metadata consistency after a power failure or an unexpected dismount. This is because ReFS can still reference the previous, consistent metadata copy.

Keep in mind, a reboot is required after applying the above fix.

I hope this helps to address your high memory usage within your Exchange servers you may have been seeing recently. If you apply the above and the issue persists, or the information above does not apply to you, please feel free to open up a support case and one of our awesome Exchange Support Engineers would be happy to assist!

A special thank you to Josh Hagen and Joe "PP" Santwire for their assistance in this blog. -Eric Whitehill, Support Escalation Engineer

Tales of the weird: High CPU on Exchange when using CBA authentication

Eric Whitehill — Tue, 31 Oct 2017 08:31:49 GMT

Here's another one from the 'Tales of the Weird and Unusual' bin here at Microsoft Exchange Support. We recently had a case where a customer was experiencing high CPU on their Exchange 2016 servers. While experiencing high CPU on an Exchange environment can happen from time to time, in our case, the customer reported it started within 2 hours of them deploying iOS 11 to their organization. What we were able to determine very quickly is that this was NOT the same as identified in Apple's support article HT208136. What made this case more from the weird is that the customer was also using Certificate based authentication, like below: (screen shot from my CBA lab) If we removed the CBA, then the issue would not duplicate. When we've seen issues like this in the past, often times it can be caused due to poorly performing Filter Drivers, IPS, or even a WAN accelerators in one instance. We were able to eliminate these potential causes fairly quickly as the high CPU was becoming more of a concern. So, this made us go and get a few procdumps and we started our debugging. I know some people out there like to try debugging this on their own, and if you do venture into that realm, you may see it getting stuck in this manner:

Child-SP RetAddr Call Site 00 00000075` 00007ff9`ebb113e8 ntdll!ZwDeviceIoControlFile+0x14 01 00000075` 00007ff9`ebb1434d httpapi!HttpApiOverlappedDeviceControl+0x68 02 00000075` 00007ff9`9a8b288a httpapi!HttpReceiveClientCertificate+0x6d 03 00000075` 00007ff9`9a8eed27 w3dt!UL_NATIVE_REQUEST::ReceiveClientCertificate+0x152 04 00000075` 00007ff9`9a8de0c3 iiscore!W3_REQUEST::NegotiateClientCertificate+0x13f w3dt!UL_NATIVE_REQUEST::ReceiveClientCertificate

In the procdump, we were finding we were getting stuck in http.sys, and SSL decryption, where the device was repeatedly trying to perform an SSL handshake and failing. Now, we have seen this happen from time to time with customers who are on slow connections, for example, if they are doing CBA from somewhere with a very poor cellular connection (think rural countries, satellite Internet connections, etc.) however that was not true in this case. After discussing with our counterparts within Windows, there were some performance tweaks made with how SSL is handled in the October 2017 update for Windows Server 2016 ( KB 4041688). We were able to successfully apply KB 4041688, and the issue was resolved upon reboot. So, if you are working on doing an iOS 11 deployment, using CBA, and starting to experience high CPU in the MSExchangeSyncApp pool, then apply KB 4041688 and your issue may very well be resolved. Special thanks to Radomir Zaric, Nick Tilton, and Jason Slaughter for their help on this very weird case.

Tales of the weird: OAB distribution failing

Eric Whitehill — Mon, 09 Oct 2017 11:05:19 GMT

From 'Tales of the Weird and Unusual', we recently worked a long running incident where at customer was having issues with their OAB in Exchange 2010, and our customer was hoping we could do a blog article on it. What we were finding is that the generation was working as expected, however it was the distribution point from the generation server that was failing. (Basically the OAB's were not updating between CAS01 and CAS02 in the diagram below) In this instance, we are not using public folder distribution, but rather web distribution. We could see on CAS03 that our OAB's were getting generated, it's just the updated files were never making it to CAS01 or CAS02. After ruling out network as being an issue, this left us wondering what was going on and we were finding that we couldn't browse via SMB to the fileshares. We would set to C:\ for our fileshare, but we weren't even able to browse to it, but we could see where it was configured: We then had the customer patch the system, and make sure it was up to date 100% as the customer was missing some windows patches. This still did not fix it. This far along in our troubleshooting, we've eliminated the following:

-Firewall/Network.
-Proper OAB configuration.
-Windows Server patching.
No Antivirus/third party. (Uninstalled as part of the troubleshooting process, as we were going to have to take an iDNA and AntiVirus + iDNA do not behave well, as discussed here. )

It didn't leave us many other options, and we ended up taking an iDNA of explorer.exe to see if we could determine what was going on and where. In the iDNA, we saw there we were referencing a very odd registry key! The provider name under [HKLM\System\CurrentControlSet\Control\NetworkProvider\HwOrder] was referencing LanmanWorkstationt not LanmanWorkstation vs. What we should have saw: We then had the customer go back through their change control (and you are doing your change control like this, right?) and discovered some 3rd party software they were using had updated this registry key as part of it's patching process. Once we reverted the change and implemented the correct key, restarted the system, the OAB was being distributed to their CAS, and in turn was being sent to their clients.

Cookies, OWA, Exchange 2016, and Third Party Software

Eric Whitehill — Mon, 09 Oct 2017 08:13:24 GMT

Recently, we had a very interesting, long running case in Exchange. This was a case where the symptoms had many Exchange Engineers scratching their heads and trying to figure out what was going on, and where. Symptoms: Customer originally called in while having issues with OWA and Exchange 2016, with a 2010 Coexist while they transitioned. In this case, the customer had Exchange 2016 on prem, and was using ADFS for auth to OWA. Once they signed into OWA, if they close the browser and attempted to sign in again, it timed out immediately and they were unable to sign back in. However, if you cleared all the cookies and browser history, then you would be able to log in to OWA one time, then the next person would have to clear the cookies before logging in again. What made it even harder to troubleshoot is that the issue only occurred if we hit a different front end from where the mailbox was and had to proxy to that server. Technical Breakdown: What we discovered during troubleshooting was that we were seeing two TimeWindow cookies. These cookies, while troubleshooting, one was for the /owa directory, the other was for the / directory. It was resulting in one of the cookies not being cleared in a proxy scenario. We ended up taking a Fiddler trace, as normal browser debugging was not showing us what we needed to see. Based upon that, we were able to take an iDNA, which was showing the customer had a third party add in which was generating the second TimeWindow cookie, and not clearing it. Fix: Ultimately it was an issue with the third party software not handling cookies correctly. Once we uninstalled the customer's third party software, the issue was resolved.

Odd behavior with OWA when sending data through a load balancer

Eric Whitehill — Fri, 30 Jun 2017 06:54:07 GMT

One of my colleagues, Danny Pastuszynski and I, recently had the privilege of working a very odd case for a customer of ours, and after discussing with the customer, they suggested I write a blog post on it. So, here we go! Our customer was spinning up a new Exchange environment with multiple sites with a load balancer to help manage their traffic load. Everything from what they could tell was configured correctly, however what they were running into was when sending the traffic for a cross-site call for OWA, they were getting an error similar to below: Or:

Sorry, we can't get that information right now. Please try again later. If the problem continues, contact your helpdesk. X-ClientId: XXXX-XXXX-XXXX-XXXX X-OWA-Error: ClientError;exMsg='$SERVER' is undefined;file=ReferenceError: '$SERVER' is undefined at _h.$Fy.prototype.$Des (https://domain.contoso.com/owa/prem/15.0.1263.5/scripts/boot.0.mouse.js:82:1710720) at _h.$1az.prototype.$988 (https://domain.contoso.com/owa/prem/15.0.1263.5/scripts/boot.0.mouse.js:82:1622688) at _a.$Fj.prototype.$6O3 (https://domain.contoso.com/owa/prem/15.0.1263.5/scripts/boot.0.mouse.js:82:383225) at _a.$H8.prototype.$390 (https://domain.contoso.com/owa/prem/15.0.1263.5/scripts/boot.0.mouse.js:82:385523) at _a.$H8.prototype.$1U (https://domain.contoso.com/owa/prem/15.0.1263.5/scripts/boot.0.mouse.js:82:385302) at _brp.$AW.prototype.$2Pk (https://domain.contoso.com/owa/prem/15.0.1263.5/scripts/boot.1.mouse.js:2:13 X-OWA-Version: undefined X-FEServer: FrontEnd.Contoso.Com X-BEServer: undefined

Date: 6/30/2017 3:14:08 AM

Basically, the X-OWA-Error: ClientError;exMsg$MachineName is not defined;file=ReferenceError :$MachineName is not defined. Then, we would see a series of java scripting errors. However, when we would pull the data in a network capture between the front and and back end Exchange servers, everything looked good. We would see the cert exchange, and traffic would not look abnormal. (i.e. TTL lowered to somewhere in the 60's, TCP resets, etc.) We then reviewed our HTTP Proxy logs between the front and and back end servers - we would see http status codes of 200 between our front end and back end Exchange servers. This would indicate the communication was successfully happening, and that there was something with the rendering happening between the Exchange server front end and the client. However, since the message was indicating something was going wrong with the rendering, a fiddler capture would not be helpful to us here.

Resolution:

Our next troubleshooting step was to bypass the customers load balancer. This is a fairly easy step where you configure a hosts file or just type in the server FQDN of one of the Exchange servers to no longer point to your VIP name. Whenever we did this, we would never get the above error. This just helped to add in more pieces to the puzzle. We were able to engage the NLB vendor and the customer's network team, where after comparisons to their dev site and their production site, we were able to determine there was a slight configuration change between the two sites, in that the customer had enabled an analytics feature within their load balancer which was injecting java script into the OWA calls, which ended up breaking the OWA render.

Smart Forward on iOS devices, or "Why am I seeing so many calendar invites?"

Eric Whitehill — Wed, 10 May 2017 12:00:07 GMT

Over the last several days, our Outlook and Exchange support teams have noted an increase in the number of cases opened where customers are seeing a calendar invite sent over and over again, even with no interaction on their devices. Usually we will send our support article to our customers, but there have been questions over how would we see it inside our own environment. In some cases, this is what we will see in our Outlook Inbox: Then, for the actual meeting, we will see something similar to below: One of the questions we get is why is this happening? What can be done to stop it? We have been able to work with Apple and determine is that all of these users all happen to have an iOS device, which was recently updated. From what we can tell, the location field is what is triggering the issue. We would have the time to leave and location field will somehow trigger this and then send a SmartForward. When this happens, all recipients will then get a 'copy' of the updated meeting.

What if I suspect I have this issue?

There is a way to tell if you have this issue! The first step would be for you to create a ticket with Premier Support via the Premier Support Portal! If you do not have a Premier Service contract, you can also open up a case with our Professional Support Incident team. We will require the following data:

Calendar Diagnostic Logs from the "Get-CalendarDiagnosticLog" cmdlet. This data will need to be from the meeting originator (in our example above, from Tony G.) and the Send on behalf user. (In our example above, we would need it from Lee S.)
We may also need Active Sync Mailbox logging, however you need to check with your Support Professional first.

Within these logs, your support professional will look for specific calls from the affected devices and if we see the SmartForward triggered in a certain way. Please understand we have to be fairly vague about how this call is made, and your support professional may not be able to give you additional details.

Update as of 6/30/17:

We have been able to work with Apple and can confirm there is a fix in progress on the Apple side. However, we do not have an ETA on release, only that it will be in a future release. Per our discussion with Apple, I've added in some additional workarounds.

Update as of 7/20/17:

Apple has released iOS 10.3.3, which should contain the fix. Please update to iOS 10.3.3 and check to see if the issue is resolved.

Update as of 7/28/17

Apple has confirmed iOS 10.3.3 does include a fix for the SmartForward issue. Please note that ALL iOS and watchOS devices MUST be updated to prevent additional occurrences. Apple has published the release notes at https://support.apple.com/kb/DL1893?locale=en_US and https://support.apple.com/kb/DL1894?locale=en_US for the iOS and watchOS issues. While we are still seeing some trailing reports of meetings invites, this issue is different than the SmartForward attendees issue. As such, I'm going to remove the workarounds and instead replace it with the fixes:

Workarounds:

~~Use Outlook for iOS.~~
~~Remove the location of the meeting.~~
~~Disable the "Time to Leave" feature via the "Settings > Calendar > Default Alert times" commands~~
~~Update to iOS 10.3.3 beta or iOS 11 beta.~~
~~You can also contact Apple, as they may have additional workarounds.~~

Solution:

To correct the SmartFoward issue, please upgrade to iOS 10.3.3 and watchOS 3.2.3.

Reasons why we ask for removal of third party software when troubleshooting Exchange

Eric Whitehill — Sun, 26 Mar 2017 13:03:30 GMT

Over the last few months, we have seen an increase in the number of critical situations (referred to as CritSits) where third party software is causing issues within Exchange 2013 and 2016. As an Exchange Engineer, we have seen several trends with many of these cases, and this blog post is to help ease the fear of our customers when faced with a decision to disable or remove their third party software. When troubleshooting something as complex as Exchange, we often times ask the following questions:

What version of Exchange are you on?
What is your server/DAG configuration?
Has there been any recent change in the environment?
Can we remove any third party add ins?

Especially when we start troubleshooting client connectivity issues, otherwise known as XCSI. As Exchange engineers, we understand that removing third party add ins can be difficult to almost impossible for some organizations. Please understand we do not go and ask for the removal of this software lightly. Many of the support engineers you talk to, especially once you get to the tier 3 level, used to work with Exchange prior to joining Microsoft. We know it's not easy, but it's a valid and important part of the troubleshooting process. Some of the push back we get often times involves these questions:

Can't you just troubleshoot around it?
It's never been an issue in the past!
Our internal processes will not allow for removal of the product.

And while we understand the difficulty of our request, please understand that for many advanced forms of troubleshooting, removal of the product is going to be required. For example, in the last few months, a number of customers experienced the following:

A third party product was directly interfacing with store.exe, inhibiting the Exchange information store from functioning correctly. This was causing random failovers, as store would find itself crashing and then we would initiate a failover.
A third party product created an additional transport agent, which was not able to handle the amount of incoming mail for a customer's environment, causing significant Edge Transport delays.
A third party product was installed to help monitor the network traffic on the Exchange servers. However, this product was not capable of processing the amount of traffic Exchange handles, which resulted in high CPU and insufficient system resources to monitor the amount of incoming Active Sync and Outlook requests.
A third party product was installed, however it was not accepting the anti virus exclusions and instead was scanning the customer's databases, resulting in database lock and corruption.
An instance of where a third party product was interfering with HTTPS traffic, causing significant handle and memory usage, resulting in an Exchange outage, however the customer's internal security team would not allow removal of the product, resulting in a prolonged outage for the customer.

In many of these cases, the customer was very adamant that we could not uninstall or even disable their third party products, resulting in some very long critical situations for these customers and loss of revenue. Our goal as engineers is to work with you and your organization to get email resources restored as quickly as we can. In some cases, we can do troubleshooting around the third party products, such as when a third party transport agent is installed. We can take a pipeline trace, as outlined at https://technet.microsoft.com/en-us/library/bb125018(v=exchg.150).aspx. Then, we can see which transport agent is misbehaving. However, in many cases, we also end up having to disable the transport agent, especially if we see something like this:

Time: 1/01/2017 1:01:01 AM ID: 1050 Level: Warning Source: MSExchange Extensibility Machine: mail.contoso.com Message: The execution time of agent 'Third Party Agent' exceeded 90000 milliseconds while handling event 'OnRoutedMessage' for message with InternetMessageId: 'XXXXXXXXXX'. This is an unusual amount of time for an agent to process a single event. However, Transport will continue processing this message.

But, it is not always possible for us to do that in more complex environments. A question we often get is, can we just disable the software? In many cases, this is not possible due to File System Filter Drivers. These drivers, as defined at https://msdn.microsoft.com/en-us/windows/hardware/drivers/ifs/what-is-a-file-system-filter-driver- states:

A file system filter driver is an optional driver that adds value to or modifies the behavior of a file system. A file system filter driver is a kernel-mode component that runs as part of the Windows executive.

A file system filter driver can filter I/O operations for one or more file systems or file system volumes. Depending on the nature of the driver, filter can mean log, observe, modify, or even prevent. Typical applications for file system filter drivers include antivirus utilities, encryption programs, and hierarchical storage management systems.

Because of the way these file system filter drivers interject themselves into the system, in order to truly disable them, we need to uninstall the product. Again, this is not the fault of the software, but simply the way these drivers are designed to work. To borrow a line from the movie "Star Wars: Episode III - Revenge of the Sith" Obi-Wan Kenobi states "They are doing their job so we can do ours." However, we have seen where an older, out of date version, or a misconfiguration, such as no antivirus exceptions, or incorrect antivirus exceptions, are in place, it causes Exchange performance to suffer or fail. When misconfigured, we can see where third party software can cause files to lock, in a similar fashion to below:

Time: 1/1/2017 1:01:10 AM ID: 489 Level: Error Source: ESE Machine: mail.contoso.com Message: msexchangerepl (4456) An attempt to open the file "D:\Databases\DB01.EDB" for read only access failed with system error 32 (0x00000020): "The process cannot access the file because it is being used by another process. ". The open file operation will fail with error -1032 (0xfffffbf8).

If these files are locked during a crucial process, this can cause Exchange to fail over, information store to crash, or any number of other issues. There's several ways we can attempt to track this, including using process explorer. This is not always applicable in every case however, and in many cases, we need to exclude the products which we know can cause this issue. We also have instances where we need to take a process dump (many times called a ProcDump) or Time Travel Tracing (often referred to as an iDNA). With iDNA in particular, unless we have antivirus removed, the iDNA will in almost every occasion not complete successfully. In those instances, we will have to remove the third party software in order for us to continue to troubleshoot. What we have discovered in working with many third party vendors is we will find customers have either ignored the advice of their third party vendors or Microsoft. As Ross Smith IV once put it so eloquently in his blog titled "Concerning Trends Discovered During Several Critical Escalations":

"Another concerning trend I witnessed is that customers repeatedly ignored recommendations from their product vendors. There are many reasons I’ve heard to explain away why a vendor’s advice about configuring or managing their own product was ignored, but it’s rare to see a case where a customer honestly knows more about how a vendor’s product works than does the vendor. If the vendor tells you to configure X or update to version Y, chances are they are telling you for a reason, and you would be wise to follow that advice and not ignore it. Microsoft’s recommendations are grounded upon data– the data we collect during a support call, the data we collect during a Risk Assessment, and the data we get from you. All of this data is analyzed before recommendations are made. And because we have a lot of customers, the collective learnings we get from you plays a big part."

In one instance, a third party vendor told a customer to not install their product on Exchange, as it has difficulties handling the bursty nature of Exchange under high traffic loads, and recommended a different version which was in line with being supported on the customer's version of Exchange. However, this customer then turned to Microsoft and requested we assist them in configuring the original third party software, over the objections of their vendor. In cases like this, because Microsoft will not have any knowledge of the inner workings of the vendor's software, we were unable to help with this implementation. In almost every instance, we find the third party software doing exactly what it is supposed to be doing - have hooks into the operating system or processes to perform certain tasks, or prevent certain tasks from taking place. However, in extremely busy pieces of software, such as Exchange, we hit a code optimization path point, where the software can only handle so many instructions in a period of time. This could be due to a limitation within how the code is executed or any number of reasons. Our job is not to point blame, but to get your environment back up and operating as quickly as possible.

Help Us Help You

The biggest ways we can help you is to perform the actions we have. Help us to eliminate any road blocks which might be in the way to enacting the action plans. Even if removing the third party software doesn't eliminate the issue, it allows for us to perform additional troubleshooting steps which we may need to do. Remember, taking some more advanced troubleshooting such as procdumps and iDNA's are difficult, to if not impossible for us to take if we do not remove the hooks these third party pieces of software put into our products. We know uninstalling third party software is not what any of us want to do - from you, down to your account manager, down to the engineer you are working with. We understand there are risks associated with uninstalling AV and other products. We've been there - in your shoes, most likely trying to figure out the same question - 'How do I do what my vendor is saying?". However, in certain cases, it is unavoidable. In every scenario outlined above, the resolution and restoration of service step was the removal of the third party product. It wasn't necessarily due to the third party product doing anything wrong, however it was either misconfigured or simply could not handle the resource intensive requests which happens within Exchange environments. Troubleshooting is like taking a very large onion, peeling back the layers, and finding out what is going on. In large and complex environments, such as Exchange, we will find many more layers of onions or unnecessary complexity. As engineers, you may have been working with your environment for over ten years, however on the phone, we have to get up to speed in an hour or less. Help us to reduce the complexity, and help us to help you get your environments back up as quickly as we can. We understand talking to an Exchange Engineer was most likely not on your plan that day, but we are here to help you as much as we can. I hope this article helps to explain some of our steps and items we look for when troubleshooting large and complex Exchange environments.

Hello world!

Eric Whitehill — Tue, 07 Mar 2017 21:14:05 GMT

Welcome to Technet. This is your first post. Edit or delete it, then start blogging!