The One With The FBA Redirect Loop

Article
01/15/2015

Applies to: Exchange 2013

Recently I came across a few cases where all users logging into OWA and ECP would be perpetually redirected back to the FBA logon page. The customer had just replaced their old certificate with a new one, from a supported 3rd-party vendor, and the certificate seemed to check out, having a trusted chain, accessible CRLs, etc. (TLDR)

Setup:

Exchange 2013 CAS
Forms-based authentication (FBA) enabled

Symptom:

Log on to OWA or ECP by browsing to the site which will redirect you (302) to the FBA page
Enter the credentials as you normally would and submit the form
Get redirected back to /owa without an error message or an event being thrown

Sample Log

Digging into the logs, we see a distinct pattern. First up, the IIS log:

 #Software: Microsoft Internet Information Services 8.5
#Version: 1.0
#Date: 2015-01-14 18:39:49
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2015-01-14 18:39:49 192.168.137.113 GET /owa &CorrelationID=;&cafeReqId=78641dc0-8e3c-47b5-a4bc-30e1d7d91546; 443 - 127.0.0.1 Mozilla/5.0+(Windows+NT+6.3;+WOW64) - 302 0 0 8514
2015-01-14 18:39:50 192.168.137.113 GET /owa/auth/logon.aspx url=https%3a%2f%2fmail.contoso.com%2fowa&reason=0&CorrelationID=;&cafeReqId=9a5cb876-58cf-4f35-9ccb-bf06b0980656; 443 - 127.0.0.1 Mozilla/5.0+(Windows+NT+6.3;+WOW64) - 200 0 0 1343
2015-01-14 18:39:51 192.168.137.113 GET /owa/auth/logon.aspx replaceCurrent=1&url=https%3a%2f%2fmail.contoso.com%2fowa&CorrelationID=;&cafeReqId=cdec7c7e-e80e-4218-b295-f2e623d8bf24; 443 - 127.0.0.1 Mozilla/5.0+(Windows+NT+6.3;+WOW64) https://mail.contoso.com/owa/auth/logon.aspx?url=https%3a%2f%2fmail.contoso.com%2fowa&reason=0 200 0 0 1156
2015-01-14 18:40:02 192.168.137.113 POST /owa/auth.owa &CorrelationID=;&cafeReqId=55c7a045-aad3-4fa3-a345-3fab3436da17; 443 contoso\jason 127.0.0.1 Mozilla/5.0+(Windows+NT+6.3;+WOW64) https://mail.contoso.com/owa/auth/logon.aspx?replaceCurrent=1&url=https%3a%2f%2fmail.contoso.com%2fowa 302 0 0 1688
2015-01-14 18:40:02 192.168.137.113 GET /owa &CorrelationID=;&cafeReqId=6afdd6df-f2ed-4c04-9bc1-ae870305f0b5; 443 - 127.0.0.1 Mozilla/5.0+(Windows+NT+6.3;+WOW64) https://mail.contoso.com/owa/auth/logon.aspx?replaceCurrent=1&url=https%3a%2f%2fmail.contoso.com%2fowa 302 0 0 62
2015-01-14 18:40:02 192.168.137.113 GET /owa/auth/logon.aspx url=https%3a%2f%2fmail.contoso.com%2fowa&reason=0&CorrelationID=;&cafeReqId=ebc7e50b-e71a-43bd-84a7-40e6f70fb2a3; 443 - 127.0.0.1 Mozilla/5.0+(Windows+NT+6.3;+WOW64) https://mail.contoso.com/owa/auth/logon.aspx?replaceCurrent=1&url=https%3a%2f%2fmail.contoso.com%2fowa 200 0 0 390
2015-01-14 18:40:02 192.168.137.113 GET /owa/auth/logon.aspx replaceCurrent=1&url=https%3a%2f%2fmail.contoso.com%2fowa&CorrelationID=;&cafeReqId=97fc51f2-b4d6-4dfb-bfc4-6692260543e4; 443 - 127.0.0.1 Mozilla/5.0+(Windows+NT+6.3;+WOW64) https://mail.contoso.com/owa/auth/logon.aspx?url=https%3a%2f%2fmail.contoso.com%2fowa&reason=0 200 0 0 140

There are a few things that are really important here: the URI, the HTTP VERB used, and the response code. When we remove the noise and focus on those, we get this pattern:

GET, /owa, 302
GET, /owa/auth/logon.aspx, 200
GET, /owa/auth/logon.aspx, 200
POST, /owa/auth.owa, 302
GET, /owa, 302 (starts repeating with step 2)

To get an idea of what's happening here, we need to understand how the FBA logon works. In a nutshell, this is what's supposed to happen:

A user browses to the OWA page, in general that's "/owa".
OWA needs to authenticate the user, so it redirects the browser to /owa/auth/logon.aspx.
Logon.aspx is the FBA page. The form POSTs the user name and password to "/owa/auth.owa".
As part of the FBA model, we store an encrypted cookie with a short TTL in the browser, and redirect the browser to /owa.
OWA allows the user to complete the login, and displays the user's mailbox.

In the customer's environment, the process breaks down on step 5. Instead of allowing the login, we're just redirected to the FBA again.

Unfortunately there's no clear indication of what's happening in the event or IIS logs. The only real clue is in the HttpProxy\Owa log:

 #Software: Microsoft Exchange Server
#Version: 15.00.0995.012
#Log-type: HttpProxy Logs
#Date: 2015-01-14T18:39:49.098Z
#Fields: DateTime,RequestId,MajorVersion,MinorVersion,BuildVersion,RevisionVersion,ClientRequestId,Protocol,UrlHost,UrlStem,ProtocolAction,AuthenticationType,IsAuthenticated,AuthenticatedUser,Organization,AnchorMailbox,UserAgent,ClientIpAddress,ServerHostName,HttpStatus,BackEndStatus,ErrorCode,Method,ProxyAction,TargetServer,TargetServerVersion,RoutingType,RoutingHint,BackEndCookie,ServerLocatorHost,ServerLocatorLatency,RequestBytes,ResponseBytes,TargetOutstandingRequests,AuthModulePerfContext,HttpPipelineLatency,CalculateTargetBackEndLatency,GlsLatencyBreakup,TotalGlsLatency,AccountForestLatencyBreakup,TotalAccountForestLatency,ResourceForestLatencyBreakup,TotalResourceForestLatency,ADLatency,SharedCacheLatencyBreakup,TotalSharedCacheLatency,ActivityContextLifeTime,ModuleToHandlerSwitchingLatency,ClientReqStreamLatency,BackendReqInitLatency,BackendReqStreamLatency,BackendProcessingLatency,BackendRespInitLatency,BackendRespStreamLatency,ClientRespStreamLatency,KerberosAuthHeaderLatency,HandlerCompletionLatency,RequestHandlerLatency,HandlerToModuleSwitchingLatency,ProxyTime,CoreLatency,RoutingLatency,HttpProxyOverhead,TotalRequestTime,UrlQuery,BackEndGenericInfo,GenericInfo,GenericErrors
2015-01-14T18:39:49.082Z,78641dc0-8e3c-47b5-a4bc-30e1d7d91546,15,0,995,12,,Owa,mail.contoso.com,/owa,,FBA,False,,,,Mozilla/5.0 (Windows NT 6.3; WOW64),127.0.0.1,CONTOSERVER,302,,,GET,,,,,,,,,0,,,,,,,,,,,,,,,531.2234,,,,,,,,,,,,,,616,,616,618,,,BeginRequest=2015-01-14T18:39:48.551Z;CorrelationID=;OnAuthenticate=259;NoCookies=302 - GET/E14AuthPost;EndRequest=2015-01-14T18:39:49.082Z;,
2015-01-14T18:39:50.473Z,9a5cb876-58cf-4f35-9ccb-bf06b0980656,15,0,995,12,,Owa,mail.contoso.com,/owa/auth/logon.aspx,,FBA,False,,,,Mozilla/5.0 (Windows NT 6.3; WOW64),127.0.0.1,CONTOSERVER,200,,,GET,,,,,,,,,0,,,,152,,,,,,,,,,,1234.2881,,,,,,,,,,,,,,1077,,1229,1229,?url=https%3a%2f%2fmail.contoso.com%2fowa&reason=0,,BeginRequest=2015-01-14T18:39:49.239Z;CorrelationID=;OnAuthenticate=108;SelectHandler=151;EndRequest=2015-01-14T18:39:50.473Z;,
2015-01-14T18:39:51.676Z,cdec7c7e-e80e-4218-b295-f2e623d8bf24,15,0,995,12,,Owa,mail.contoso.com,/owa/auth/logon.aspx,,FBA,False,,,,Mozilla/5.0 (Windows NT 6.3; WOW64),127.0.0.1,CONTOSERVER,200,,,GET,,,,,,,,,0,,,,0,,,,,,,,,,,1047.0835,,,,,,,,,,,,,,1055,,1055,1055,?replaceCurrent=1&url=https%3a%2f%2fmail.contoso.com%2fowa,,BeginRequest=2015-01-14T18:39:50.629Z;CorrelationID=;EndRequest=2015-01-14T18:39:51.676Z;I32:ATE.C[CONTOSERVER.contoso.com]=2;F:ATE.AL[CONTOSERVER.contoso.com]=16;I32:ADS.C[CONTOSERVER]=1;F:ADS.AL[CONTOSERVER]=3.2332;I32:ADR.C[CONTOSERVER]=1;F:ADR.AL[CONTOSERVER]=5.214,
2015-01-14T18:40:02.839Z,55c7a045-aad3-4fa3-a345-3fab3436da17,15,0,995,12,,Owa,mail.contoso.com,/owa/auth.owa,,FBA,True,contoso\jason,,Sid~S-1-5-12-3456789012-3456789012-3456789012-3456,Mozilla/5.0 (Windows NT 6.3; WOW64),127.0.0.1,CONTOSERVER,302,,,POST,,,,,WindowsIdentity,,CONTOSERVER,78,145,,,,892,629,,0,90;,90,7;11;,18,108,,0,1656.8489,36,,,,,,,,,101,727,0,,763,,1655,1655,,,BeginRequest=2015-01-14T18:40:01.182Z;CorrelationID=;OnAuthenticate=155;ProxyState-Run=None;ServerLocatorCall=47e9ec43-db6e-4278-b364-1d3049692e7a;FEAuth=BEVersion-1941996515;ProxyState-Complete=CalculateBackEnd;EndRequest=2015-01-14T18:40:02.839Z;I32:ATE.C[CONTOSERVER.contoso.com]=4;F:ATE.AL[CONTOSERVER.contoso.com]=11.75;I32:ADS.C[CONTOSERVER]=4;F:ADS.AL[CONTOSERVER]=6.2329;I32:ADR.C[CONTOSERVER]=1;F:ADR.AL[CONTOSERVER]=1.0308,
2015-01-14T18:40:02.902Z,6afdd6df-f2ed-4c04-9bc1-ae870305f0b5,15,0,995,12,,Owa,mail.contoso.com,/owa,,FBA,False,,,,Mozilla/5.0 (Windows NT 6.3; WOW64),127.0.0.1,CONTOSERVER,302,,,GET,,,,,,,,,0,,,,,,,,,,,,,,,31.26,,,,,,,,,,,,,,27,,27,27,,,BeginRequest=2015-01-14T18:40:02.870Z;CorrelationID=;NoCookies=302 - GET/E14AuthPost;EndRequest=2015-01-14T18:40:02.902Z;,
2015-01-14T18:40:02.933Z,ebc7e50b-e71a-43bd-84a7-40e6f70fb2a3,15,0,995,12,,Owa,mail.contoso.com,/owa/auth/logon.aspx,,FBA,False,,,,Mozilla/5.0 (Windows NT 6.3; WOW64),127.0.0.1,CONTOSERVER,200,,,GET,,,,,,,,,0,,,,0,,,,,,,,,,,0,,,,,,,,,,,,,,1,,1,1,?url=https%3a%2f%2fmail.contoso.com%2fowa&reason=0,,BeginRequest=2015-01-14T18:40:02.933Z;CorrelationID=;EndRequest=2015-01-14T18:40:02.933Z;,
2015-01-14T18:40:03.371Z,97fc51f2-b4d6-4dfb-bfc4-6692260543e4,15,0,995,12,,Owa,mail.contoso.com,/owa/auth/logon.aspx,,FBA,False,,,,Mozilla/5.0 (Windows NT 6.3; WOW64),127.0.0.1,CONTOSERVER,200,,,GET,,,,,,,,,0,,,,0,,,,,,,,,,,0,,,,,,,,,,,,,,2,,2,2,?replaceCurrent=1&url=https%3a%2f%2fmail.contoso.com%2fowa,,BeginRequest=2015-01-14T18:40:03.371Z;CorrelationID=;EndRequest=2015-01-14T18:40:03.371Z;,

The lines highlighted in blue correspond with lines 4 and 5 above -- the POST to auth.owa and subsequent GET of /owa. Notice that a username (contoso\jason) appears here, indicating successful authentication. A bad auth attempt would result in a 401 or 403. The next line doesn't show a user, and also shows NoCookies=302.

That pattern indicates we authenticated successfully, but then for some reason didn't have the expected cookie when coming back to /owa. So what happened?

It turns out the cookie we set earlier in step 4 fails to decrypt, and we send the user back to the FBA page to try again. There aren't a lot of scenarios where we should fail to decrypt a legitimate cookie, so let's look at the cert, which is what we use to encrypt/decrypt the cookie data. Remember, the customer just changed the cert, so we knew it was more than likely related, but standard cert checks succeeded.

We used certutil to verify most of this:

Certificate chain is valid, up to the CA root
The current date was within the certificate's validity period
CRL, AIA, and OCSP checks were successful
The names on the certificate (subject name, alternative names) matched the URL we were browsing

That's the standard list of things I check when running into your run-of-the-mill certificate issues, but since they checked out, I just dumped the cert for an eyeball test (grab the serial number from Get-ExchangeCertificate or the cert itself, no spaces):

certutil -store my certificateSerialNumber

The provider was listed as "Provider = Microsoft Software Key Storage Provider". The KSP is a provider that is used with Cryptography API: Next Generation (CNG). Exchange FBA does not support CNG certificates. Exchange only uses and supports the legacy CryptoAPI which uses Cryptographic Service Providers (CSP).

Solution

Migrate the certificate to a CSP or request a CSP certificate from your certificate provider. If you use a CSP or KSP from another software or hardware vendor, contact the relevant vendor for the equivalent instructions. For example this will be the case if you use a Microsoft RSA SChannel Cryptographic Provider and the certificate is not locked into a KSP:

Backup your existing certificate, including the private key. For more information on how to do this, please reference the Export-ExchangeCertificate cmdlet.
Note which services are bound to the certificate currently using Get-ExchangeCertificate.
Delete the certificate and private key. If you use Remove-ExchangeCertificate, you'll need to temporarily assign a different certificate to the bound services first. If you just use PowerShell or MMC you'll need to reboot the server afterwards as bound services cache the certificate.
Import the new certificate into a CSP using certutil (pfx/p12):

certutil -csp "Microsoft RSA SChannel Cryptographic Provider" -importpfx <CertificateFilename>
Ensure the certificate is still bound to the same services using Get-ExchangeCertificate, and if not, re-set them using Enable-ExchangeCertificate.
Restart the server (see step 3).

Verify the certificate now has its private key stored with a CSP:

certutil -store my <CertificateSerialNumber>
The output should now show: "Provider = Microsoft RSA SChannel Cryptographic Provider"

Jason Slaughter

Comments

Anonymous
January 01, 2003
Just experiented the same issue in Exchange 2013 at a customer. Thanks a lot for the detailed information.
I've earlier used openssl to convert the pfx-file from CNG to CryptoAPI-compatible provider using the following commands:

openssl.exe pkcs12 -in old_certificate.pfx -out certificate.pem -nodes
openssl.exe pkcs12 -export -in certificate.pem -out new_certificate.pfx

Cheers
Rolf
Anonymous
January 16, 2015
wow
Thanks for sharing
Anonymous
June 24, 2015
The comment has been removed
Anonymous
July 28, 2015
THX for sharing - I just spend hours with this problem and installed Server and Exchange again cause I thought I configured something wrong.
An observation: The same certificate works after first install (Win2012R2 with all Updates as of today, Exch 13 with SP1, changing Certificate - working even after reboot) - Login did no longer work after installing Cumulative Update 9 .... so what went wrong installing the update or whats wrong with the update?
Anonymous
August 20, 2015
Thanks to JasonSla for demystifying this and big thanks to rvaglid!!! openssl saved me! ;)
Anonymous
November 28, 2015
Thanks for this one!!
Anonymous
January 06, 2016
Got an error with the certutil option, but worked the openssl way. BTW it took days to figure out the problem and find this article. Lets add some more tags :) Exchange 2013 CU update OWA ECP LOGIN loop error
Anonymous
February 24, 2016
Thank you Jason!!
Anonymous
February 25, 2016
Well, if you ever come to Czech republic I owe you a beer :-D
I've spent the whole today on this issue and finally encountered your blog which has provided the perfect solution.
A month ago we have issued a new certificate for Exchange server using our new WS2012R2 CA, imported into Exchange and everything was fine.. until last week we rebooted the server after Windows Updates and the OWA and ECP have stopped working. I had no idea that Exchange services have cached the old certificate and the new certificate have become used after the server restart..

Your advice is really top. I've just had a problem with importing the pfx via certutil, I had to convert it using openSSL as rvaglid has stated. However I ended with provider type "Microsoft Enhanced Cryptographic Provider v1.0" but it seems to work well.
If you know how to change it to Microsoft RSA SChannel Cryptographic Provider, please let me know

Other than that it has all gone smooth and I thank you.

Michal
Anonymous
February 25, 2016
I'm not sure why, but certutil is having issues on later versions of Windows converting the certs. I've had a few cases now where we've had to use OpenSSL.

@Michal - That provider is listed as a CSP (https://msdn.microsoft.com/en-us/library/windows/desktop/aa386983(v=vs.85).aspx), and it appears to support RSA, but I haven't dug into it. I'm glad it's working though!
Anonymous
February 29, 2016
Hi Jason, thanks for update. Certutil acts really weird. I wasn't able to convert and import the certificate on WS2012R2, however I was able to do it on W10
I'll stick to the "Microsoft Enhanced Cryptographic Provider v1.0".

Michal
Anonymous
April 12, 2016
This has been one that has bugged me for a while, especially being more notable when you go to change the certificate on the server to something other than the self signed. Thanks for the education, and the chuckle on the Friends reference in the title ;)
- Anonymous
  May 26, 2016
  I loved Friends!
Anonymous
May 12, 2016
Have the same issue but only behind F5 loadbalancerChecked certutil and we have "Provider = Microsoft RSA SChannel Cryptographic Provider"Any other things to check?
- Anonymous
  May 26, 2016
  Apologies for the delay -- we switched blog platforms and I'm not getting comment notifications.It's very difficult to say without more data. Did you determine it's the same issue due to the pattern in IIS? What if you HOSTS file to a CAS, bypassing the F5?Either way, I'd suggest opening a case if it's still an issue. We'd need to dig into that.
Anonymous
May 16, 2016
While "the Internet says nothing" about this problem, you saved my day. Thanks for sharing!
Anonymous
August 02, 2016
We have this problem with our Exch server with the exact errors in the log files mentioned above. The problem is, when the certificate is looked at using Certutil it shows that its CSP is Microsoft RSA SChannel Cryptographic Provider. I'm not really sure where to go with this from here.
- Anonymous
  August 02, 2016
  There may be a different reason your CAS is unable to decrypt the cookie, and you'd get the same symptoms. You should open a case with support so we can help you directly.
  - Anonymous
    December 06, 2016
    @Don, hopefully you are monitoring this but we too are in same boat. Updated cert, starting seeing this issue logging into ecp and found this blog. Cert checks out fine with correct CSP listed yet we are still having this issue. Did you find any resolution to your issue that you can share?
    - Anonymous
      December 07, 2016
      Restarting IIS on all cas servers fixed our issue. It was never done after the cert renewal.
Anonymous
September 26, 2016
Hey Jason,thank you for the great article. It worked for me. I have one question, as I am not very familar with certificates. Why could that have happend? Was it because we made a mistake by creating the csr, or did we something wrong on the comodo request site ? Or war it maybe because of the import. That would help me understanding the issue. Thanks a lot
- Anonymous
  September 26, 2016
  The cert request should contain a hint to the OS on where it wants to have the key stored. This attribute is the "Enrollment CSP" and if you still have the request itself you can dump it to see what the CSR lists:[PS] C:\Users\Administrator\desktop>certutil.exe -dump .\star_contoso_com.txtPKCS10 Certificate Request:Version: 1Subject: C=US S=Washington L=Redmond O=Contoso Labs CN=*.contoso.com[snip] Attribute[1]: 1.3.6.1.4.1.311.13.2.2 (Enrollment CSP) Value[1][0], Length = 64 Unknown Attribute type CSP Provider Info KeySpec = 1 Provider = Microsoft RSA SChannel Cryptographic Provider Signature: UnusedBits=0[/snip]CertUtil: -dump command completed successfully.We can often override this using certutil, but the default is to use the CSP from the cert. When using the Exchange CSR tool (New-ExchangeCertificate) it should embed the attribute properly as "Microsoft RSA SChannel Cryptographic Provider". If you used New-ExchangeCertificate (or created the CSR via ECP) then it's possible your CA didn't honor this when generating the resulting cert, and replaced it with something else.If you used your CA's creation tools, they may not expose the option and use a default of their own. In general, most CAs will provide a template for IIS 7/8.x or even Exchange proper, and those normally use the Microsoft RSA SChannel provider we need.
Anonymous
December 22, 2016
THANK YOU! It took me hours to find the culprit. This is still valid for Exchange 2016CU3.Do not forget to change the default KSP when generating a new CSR (*.req) from The Windows Certificate SnapIn.

Share via