This issue manifests itself in a number of ways.  However the most common is likely that an iOS device is trying to send a large attachment (likely around 8MB or larger).  The result is that the device fails to send the attachment, and the following Event ID is logged on the Exchange Client Access server:

Event ID 1008
Message: An exception occurred and was handled by Exchange ActiveSync. This may have been caused by an outdated or corrupted Exchange ActiveSync device partnership. This can occur if a user tries to modify the same item from multiple computers. If this is the case, Exchange ActiveSync will re-create the partnership with the device. Items will be updated at the next synchronization.

If you look further down into the Event ID you will find Maximum Request Length Exceeded under the Exception Message.

This Event ID will likely repeat over and over as the iPhone continues to sync.  Most likely if you check the device (or devices) of any affected users you will see a message in the outbox with a large attachment (>8MB). 

This is due to the Max Request Length setting being 10MB by default.  So any message with an attachment that is around 8MB or so will fail to send as the request is too big.  The reason I say 8MB is that we have to take into consideration the 33% MIME conversion bloat added to the message after MIME conversion.  Once that 33% is taken into account, we will exceed the 10MB Max Request Length limitation.  The iPhone will continue to try and send the message until it is removed from the device. 

To resolve this issue with iOS devices, we typically recommend increasing the Maximum Request Length parameter on the web.config file for ActiveSync on the Client Access Server(s).   

• On the CAS servers, browse to the web.config file (under the ClientAccess\Sync folder)
• Open the web.config file
• Modify the <httpRuntime maxRequestLength=”10240″/> parameter to a higher value to something higher, such as 15360 (15MB).
• Save the updated web.config file
• Restart IIS (IISRESET)

That's it.  Naturally if a larger value is needed then just follow the same steps to increase the value that is required.  We don't recommend getting too carried away with this setting.  Common recommendations are 15-20MB... and usually no higher than 50MB.

~Craig

Everything was working fine except for the fact that several users would receive the following error when trying to configure Voice settings using ECP:

Even though this was not the most insightful error, it did point to a permissions problem.  So next we attempted to just login via ECP with the affected user, and of course that also failed. 

As it turned out OWA was disabled for all the affected users.  We simply enabled OWA access for these users and they could now successfully customize their voice settings via ECP.

~Craig

One of the users we were trying to modify was not receiving emails from an automated application in her inbox - they were going to Junk email.  This was happening in both Outlook and in OWA.  It turned out the e-mail address for the application server was on the user's BlockedSendersandDomains list.  However when we tried to modify the list in OWA we received the following error:

Junk e-mail validation error.  Value:  user@domain.com

Since OWA failed we tried to modify the Junk e-mail configuration from the Exchange Management Shell with a command similar to the following:

Get-MailboxJunkEmailConfiguration affect_user | Set-MailboxJunkEmailConfiguration -TrustedSendersAndDomains “name@domain.com”

However this command also failed with a similar validation error:

Junk e-mail validation error. Value: 'name@domain.com'.
    + CategoryInfo          : NotSpecified: (0:Int32) [Set-MailboxJunkEmailConfiguration], JunkEmailValidationException
    + FullyQualifiedErrorId : XXXXXXXX,Microsoft.Exchange.Management.StoreTasks.SetMailboxJunkEmailConfiguration

Luckily the resolution in this case was easier than I figured it would be.  Just before I was considering having the customer download MFCMAPI so we could delete the Junk e-mail message from her mailbox, we went back to OWA once more to have a look at the options.  That's when I noticed the option was checked/enabled to Trust Email from my Contacts.  Low and behold we disabled/unchecked this option in OWA and BINGO - we could now make changes to the Junk e-mail configuration settings from both OWA & the Exchange Shell without the validation error.  

It's amazing the effects a checkbox can have on certain functions.  

~Craig   

Typically this is a relatively straightforward process, and the following link details the Import & Export scripts you would run to perform this process: http://technet.microsoft.com/en-us/library/aa996008.aspx

We exported the configuration from the Source Edge server using a variation of the following command:

./ExportEdgeConfig -CloneConfigData:"C:\CloneConfigData.xml"

Then we copied the XML file to the TARGET Edge server and validated the configuration file & created an answer file with a variation of the following command:

./ImportEdgeConfig -CloneConfigData:"C:\CloneConfigData.xml" -IsImport $false -CloneConfigAnswer:"C:\CloneConfigAnswer.xml" 

Finally we attempted to import the configuration file to the TARGET Edge server with a variation of the following command:

./ImportEdgeConfig -CloneConfigData:"C:\CloneConfigData.xml" -IsImport $true -CloneConfigAnswer:"C:\CloneConfigAnswer.xml"

We received the following error:  Error: "Validation Failed for Fqdn element of type FQDN"

As it turned out we didn't save the CloneConfigAnswer.xml after modifying the last of the invalid settings.  Thus the FQDN in the CloneConfigAnswer.xml was reflecting the source Edge Transport server and not the target Edge Transport server. Thus it fails the FQDN Validation step of the cloning process. 

Validation errors are usually due to problems with the CloneConfigAnswer.xml file, I had just never actually received an error during the Validation step before.  At least the fix was straightforward, as we updated the FQDN in the CloneConfigAnswer.xml to reflect the FQDN of the target Edge Transport server that
the configuration was being copied to and then we were able to successfully clone the Edge Transport configuration.

~Craig 

• Exchange 2010 SP3 (KB2808208)

         http://www.microsoft.com/en-us/download/details.aspx?id=36768

• Exchange 2010 SP2 RU6 (KB2746164) 

         http://www.microsoft.com/en-us/download/details.aspx?id=36716

• Exchange 2007 SP3 RU10 (KB2788321)  

         http://www.microsoft.com/en-us/download/details.aspx?id=36708

Of course Exchange 2010 SP3 is a major update.  It not only addresses many known issues, but it is critical to Exchange 2010-2013 coexistence as well. 

~Craig

Microsoft has released a KB article that details what we know so far:  http://support.microsoft.com/kb/2814847

Apple has also recently released a support article for this issue: http://support.apple.com/kb/TS4532

*UPDATE: Apple has released 6.1.2 (http://support.apple.com/kb/TS4532) which should address the 6.1 issue.   

Essentially the symptoms we've seen so far are excessive log growth, resource consumption on Client Access & Mailbox servers, and performance-related problems due to high memory and/or CPU increases.

Here are a few of the workarounds we've been using thus far:

• Do not process Calendar items on the iOS device... especially meeting requests.
• Disable ActiveSync for the user(s) in question until the partnership can be removed and/or the device can be restarted
• Remove the device partnership and recreate it.  This option should fix the issue for the user(s) in question.
• Create a custom Throttling Policy for iOS 6.1 users (see the KB listed above for more information)
• Block iOS 6.1 users/phones.  This can be achieved by using the Allow/Block/Quarantine feature in Exchange 2010.  http://blogs.technet.com/b/exchange/archive/2010/11/15/3411539.aspx
• iRule customizations (See this link for F5 Forums: https://devcentral.f5.com/community/group/aft/2165837/asg/50 )
• TMG/ISA Http Filtering
• IIS URL Rewrite: http://social.technet.microsoft.com/wiki/contents/articles/15997.using-url-rewrite-to-block-certain-clients-from-exchange.aspx

Install 6.1.2 update from Apple to resolve the issue:

• http://support.apple.com/kb/TS4532

Identify iOS 6.1 Devices

To identify the issue in your environment, the process I've been using thus far is parsing through IIS logs.  The easiest way to do this is to use a tool called Log Parser Studio.  If you haven't been using this great tool, download it here: http://gallery.technet.microsoft.com/Log-Parser-Studio-cd458765

Log Parser Studio has several built in queries to collect ActiveSync information, such as the ActiveSync Report.  Any 6.1 devices should be noticeable by locating the User-Agent field, which will look something like the following for iOS 6.1:

• Apple-iPad3C3/1002.141
• Apple-iPhone4C1/1002.142

To create the query to check specifically for the iOS 6.1 issue (from the KB Article), open Log Parser Studio, click File, New, and Query. 

Then paste the following query (also in the KB) into the New Query window:

SELECT
   Cs-username AS User,
   MyDeviceId AS DeviceId,
   COUNT(*) AS Hits
USING
   EXTRACT_VALUE(cs-uri-query,'DeviceId') AS MyDeviceId
FROM '[LOGFILEPATH]'
WHERE  cs-uri-query LIKE '%Error:WrongObjectTypeException%'
GROUP BY DeviceId,User
ORDER BY Hits DESC

Now click File and Save Query (if you wish to save this new query).  Otherwise at this point you can load your IIS logs into Log Parser Studio and execute the query. 

Here are the list of iOS UserAgents to be on the lookout for:

6.1       1002.141

-or        1002.142

-or        1002.143

-or        1002.144

6.1.1                1002.145

6.1.2                1002.146

----or               1002.147

This Exchange Shell command should help you locate iOS 6.1 devices as well if you don't have Log Parser Studio:

Get-ActiveSyncDevice | where {$_.DeviceOs -match “iOS 6.1″}

*UPDATE from the Apple KB:

Apple has identified a fix and will make it available in an upcoming software update. In the meantime, you can avoid this bug by not responding to an exception to a recurring event on your iOS device. If you do experience the symptoms described above, disable then reenable the Exchange calendar on your iOS device using the steps below.
1. Go to Settings > Mail, Contacts, Calendars
2. Select the Exchange account from your Accounts list.
3. Turn the switch for Calendars to OFF.
4. Wait ten seconds.
5. Turn the switch for Calendars back to ON.

**UPDATE: Apple has released 6.1.2 (http://support.apple.com/kb/TS4532) which should address the 6.1 issue.  

~Craig

The purpose of this Blog post is to point out the differences in managing IIS on the Client Access (Front End) & Mailbox (Back End) servers.   

Whether you install your Exchange 2013 server as a Client Access Role or Mailbox Role you will have two Web Sites in IIS:

• Default Web Site
• Exchange Back End

The Client Access Role naturally uses the Default Web Site as the home for all the Exchange Virtual Directories (EWS, OAB, RPC, Autodiscover), just as it did in Exchange 2007 & 2010.  The Mailbox Role uses the Exchange Back End site as the home for the corresponding Exchange 2013 Web Applications.

               Client Access Role                                                     Mailbox Role

Although it would appear that these sites are relatively similar, the main difference between the two is not immediately visible until you go to PowerShell to manage the Virtual Directories.  Just as in Exchange 2007 & 2010, you will run the varios commands to manage your Virtual Directories:

• Get-WebServicesVirtualDirectory
• Get-OabVirtualDirectory
• Get-AutodiscoverVirtualDirectory
• Get-ActiveSyncVirtualDirectory

The first thing you will notice when running one of these commands is that you will get the Virtual Directories for your Front End Client Access servers, but not for the Mailbox servers:

You may be asking why that is... as from looking in IIS they look exactly the same (just in a different Web Site).  If you access the Exchange Admin Center, you will see the same thing:

Despite looking similar in IIS, the main difference between the roles is that all the Virtual Directories are actually located on the front end Client Access servers.  The web applications on the Mailbox Servers can be viewed through the following command:

 Get-WebApplication

Why is this important?  I wasn't worried about that answer until a recent customer case.  Essentially we were getting a 404 error when trying to browse EWS.  The Virtual Directories on the front end Client Access servers looked fine, and the logs indicated a problem from the Mailbox servers.  Further investigation on the Mailbox server led me to an interesting discovery... the EWS web application was missing on the Mailbox server!  That certainly explained the 404 error. 

I figured this would be no problem... we'll just create a new EWS Virtual Directory on the Mailbox Server.  Well there are no EWS Virtual Directories on the Mailbox server.  To create a new Virtual Directory requires the Client Access Role, as noted by the following error:

So what now...?     

As it turned out what I needed to do was recreate the EWS web application on the mailbox server using the following command:

New-WebApplication -Site "Exchange Back End" -Name EWS -PhysicalPath "C:\Program Files\Microsoft\Exchange Server\V15\ClientAccess\exchweb\EWS" -ApplicationPool MSExchangeServicesAppPool

Once this command completed my EWS web application was restored.  I simply had to go into IIS Manger and enable Windows Authentication and the customer's issue was resolved. 

I hope this post will help shed some additional light on how to manage IIS Virtual Directories & Web Applications on Exchange 2013 servers.

~Craig 

The first step is to launch the Exchange Management Shell on an Edge server (or Hub if there are no Edge servers). 

Next, use the following commands to view your Send Connectors on your Hub or Edge:

Get-SendConnector

In this Example you can see I only have one Send Connector, but if I had more than one then I would make sure to specify that I want to disable Opportunistic TLS on the appropriate Send Connector with the following command:

Set-SendConnector -Identity "Internet" -IgnoreSTARTTLS $true

After the command finishes you can verify the success by running this command:

Get-SendConnector | fl Name, IgnoreSTARTTLS

That's it!  I can now send outgoing email through the Internet Send Connector, and Exchange will no longer try and negotiate TLS for each connection.   

Note: Naturally in this example, ALL my outgoing messages are affected by this change.  If you had specific email domains that may need Opportunistic TLS, then creating a new Send Connector for those Address Spaces would be required.

~Craig

Essentially these are the main steps to configure Kerberos Authentication for Exchange 2010:

• Create an account to be used as the ASA credential
• Deploy the ASA credential to the CAS servers
• Convert the OAB Virtual Directory to a web application
• Assign the SPNs to the ASA credential account
• Testing

Create an account to be used as the ASA credential 

I prefer using a computer account so the password doesn’t expire, otherwise you’d have to make sure to update the credentials regularly.  The computer name you give it doesn’t matter, but I typically use ASA or CASARRAY.  In the description, make sure to put something along the lines of DO NOT DELETE: and followed by something like This account is used for Kerberos Authentication on the CASARRAY.     

Deploy the ASA credential to the CAS servers

There is a built-in script that will deploy the ASA account credentials to all CAS servers: RollAlternateServiceAccountPassword.ps1 

Here is the command to use once you have created the ASA account and the associated SPNs:   

.\RollAlternateserviceAccountPassword.ps1 -ToArrayMembers {CAS array name} -GenerateNewPasswordFor "{Domain}\{ASA}" –Verbose

Using our examples it would look something like this:

.\RollAlternateserviceAccountPassword.ps1 -ToArrayMembers outlook -GenerateNewPasswordFor “CONTOSO\ASA$" –Verbose

• Answer Yes when the script asks if you want to change the password for the ASA account at this time.
• The final output should tell you if the script has succeeded… or if it failed at any point.   

Convert the OAB virtual directory to a web application

Since the OAB virtual directory is not actually a web application, we must convert it to a web application to allow Kerberos requests to the OAB.   

To convert the OAB, you must download this script http://gallery.technet.microsoft.com/scriptcenter/525fb1dc-b612-4998-a2d1-55f32a6c35ac 

Then login to EACH Client Access server, copy over the script to the Scripts directory (Default location is C:\Program Files\Microsoft\Exchange Server\V14\Scripts) and run the ConvertOABDir.ps1 script 

NOTE: This step MUST be done from EACH Client Access Server.

Assign the SPNs to the ASA credential account

In this step we will assign the Exchange service principal names (SPNs) that will be associated with the ASA credential account we created in the previous step.  Since Exchange may also need the NETBIOS (short name), we will create SPNs for both the NETBIOS & FQDN for each.  Once this step is complete, clients will be able to start authenticating to the CAS Array via Kerberos Authentication.    

There are various SPNs that are required. 

• http:  This SPN will be used for AutoDiscover, Exchange Web Services (EWS), and the Offline Address Book (OAB).
• exchangeMDB:  This SPN is used for the RPC Client Access Service
• exchangeRFR:  This SPN is used for the Address Book Service
• exchangeAB:  This SPN is used for the Address Book Service

Example commands to create the SPNs: 

Note: the $ after the ASA credential account (ASA in this example) is required when a computer account is used.  Outlook refers to the FQDN of the Client Access Array in my lab.

setspn -s exchangeMDB/outlook ASA$

setspn -s exchangeRFR/outlook ASA$

setspn -s exchangeAB/outlook ASA$

setspn -s exchangeMDB/outlook.contosos.local ASA$

setspn -s exchangeRFR/outlook.contoso.local ASA$

setspn -s exchangeAB/outlook.contoso.local ASA$

setspn -s http/mail.contoso.local ASA$

setspn -s http/autodiscover.contoso.com ASA$ 

NOTE: To disable/remove Kerberos Authentication... simply remove these SPNs. 

Testing Kerberos Authentication

Now that you have completed all the steps we need to test if Kerberos authentication is working. 

Verify the Client Access Servers

Verify that all Client Access Servers in the CAS Array have accepted the ASA Credential by running the following command:

Get-ClientAccessServer -IncludeAlternateServiceAccountCredentialstatus |Fl Name, AlternateServiceAccountConfiguration

If one of the Client Access servers did not get the credentials, then you can run the following command to copy the credentials to any CAS server(s) that didn't get the credentials:

.\RollAlternateServiceAccountPassword.ps1 -CopyFrom ServerA -ToSpecificServers ServerB -Verbose

Verify the OAB virtual directory has been successfully converted to a web application by checking for a new application pool for the OAB virtual directory in IIS Manager.

Another good test is to open a current Address Book Log (Default location is C:\Program Files\Microsoft\Exchange server\v14\Logging\AddressBookService) and make sure that for new entries the Authentication method sent is Kerberos.  There may still be some NTLM as well, but you should now see Kerberos.

Verify from Outlook

To test from an Outlook client, you can open Outlook and verify that the Connection Status now shows Negotiate as the authentication type [Outlook 2013].

Configure your Outlook client to connect via Kerberos:

Close and then open Outlook... if it connects without any problems then Kerberos is working. 

As another test once Outlook is open, you can also run the KLIST command (WIN7 and above) from the command prompt to ensure you have Kerberos tickets generated for the SPNs we created. 

You will see something like this - look for the SPNs for your Exchange services:

#1            Client:  user @ CONTOSO.COM

                Server: exchangeMDB/Outlook.contoso.local @ contoso.local

                KerbTicket

                Ticket Flags

#2           Client:  user @ CONTOSO.COM

               Server:  exchangeAB/Outlook.contoso.local @ contoso.local

                KerbTicket…

                Ticket Flags…

#3           Client:  user @ CONTOSO.COM

                Server:  exchangeRFR/Outlook.contoso.local @ contoso.local

                KerbTicket...

                TicketFlags…

You have now successfully enabled Kerberos authentication for your Client Access Array!

~Craig

This bottleneck is due to there only being 2 MaxConcurrentAPI channels available (by default) in Windows Server 2003 or 2008 to service authentication requests.  The default value of 2 (that even Windows Server 2008 R2 uses by default) is a legacy setting that has been used since Windows 2000.  Naturally the MaxConcurrentAPI value of 2 was used back in the Windows 2000 days to avoid creating performance bottlenecks on the server, since the hardware capabilities back then were nowhere near what they are today.  Unfortunately today this antiquated setting is not aggressive enough for all the applications (i.e. Exchange, Lync, SharePoint etc.) that may be creating additional NTLM authentication traffic to your Domain Controllers.  

The most common symptom of an authentication bottleneck in Exchange 2010 is the Outlook client will receive an authentication prompt.  More often than not if you enter your credentials you will be prompted again and again, until the authentication bottleneck has subsided.  I typically explain it to customers as a busy 2-lane highway.  The bottlenecks occur during peak times as more cars try to get on the highway from ramps and by other means.  Naturally you may be waiting a while to get on the highway when it is full.  For Exchange, an authentication prompt will only wait so long to get "on the highway" before it times out.  Since that authentication attempt timed out, you must retry and thus you will be prompted for your authentication credentials again.

So what is the best way to fix this problem in Exchange 2010?  Naturally if you have more lanes available to you then more cars can get on the highway.  Imagine adding another 8 lanes to this two-lane highway.  You have just significantly increased the ability to get more and more cars onto the highway and avoiding bottlenecks (and the corresponding problems that come from those bottlenecks).  Windows Server 2012 now utilizes a default value of 10 for MaxConcurrentAPI.  However if you don't have Windows Server 2012, we can add "more lanes" to Windows Server 2003 & 2008 by adjusting the MaxConcurrentAPI value in the registry (see KB: http://support.microsoft.com/kb/975363). 

Windows Server 2003 supports a maximum MaxConcurrentAPI value of 10. 

Windows Server 2008 & 2008 R2 can be set up to 10 without a hotfix, and can go all the way to 150 with a hotfix.

Windows Server 2008 R2 SP1 does not require a hotfix to go all the way to 150.  

Thus we typically recommend changing the MaxConcurrentAPI value on Windows Server 2003 & 2008 to 10 on your Domain Controllers and Exchange 2010 Client Access Servers.  This will address most issues with NTLM authentication bottlenecks. 

If authentication issues still exist, as long as you have Windows 2008 DCs, you can load the hotfix and set the MaxConcurrentAPI value higher as needed.  Another recommendation we have for Exchange 2010 is enabling Kerberos authentication to help reduce the high volume of NTLM authentication in Exchange.  I will discuss enabling Kerberos Authentication in another blog post.  

~Craig

For more information on MaxConcurrentAPI, see the following links:

You are intermittently prompted for credentials or experience time-outs when you connect to Authenticated Services: http://support.microsoft.com/kb/975363  

How to do performance tuning for NTLM authentication by using the MaxConcurrentApi setting: http://support.microsoft.com/kb/2688798 

Optimizing NTLM authentication flow in multi-domainenvironments: http://blogs.technet.com/b/isrpfeplat/archive/2010/11/05/optimizing-ntlm-authentication-flow-in-multi-domain-environments.aspx

NTLM’s time has passed: http://blogs.technet.com/b/authentication/archive/2006/04/07/ntlm-s-time-has-passed.aspx

NTLM Bottlenecks and the RPC runtime: http://blogs.technet.com/b/askds/archive/2011/09/15/is-this-horse-dead-yet-ntlm-bottlenecks-and-the-rpc-runtime.aspx