A HTTP Detective Story by Eric Lawrence

As a little kid, my dad read with me a lot; we usually read detective stories.  While of questionable literary merit, those books developed in me a burning desire to figure stuff out, to pull back the curtain, to understand the mysterious.  I still maintain that curiosity today-- I joined the IE team because I wanted to learn the browser inside out, and I developed Microsoft Fiddler to expose the secrets of HTTP in a user-friendly way.

I recently came across a bug in the bug database for IE7 which was intriguing— logins for Internet Explorer 7 and IE6 on XPSP2 had started failing for subscribers at a popular magazine’s website.  The site didn’t have any recommendations on how to fix the alleged “cookie flaw in SP2”, though the issue was discovered by their users during the SP2 beta.

Now, as most of you know, XP SP2 introduced major security improvements for users, while causing some issues for web developers.  Before we shipped SP2, we fixed a ton of compatibility issues to ensure that IE continued to work on the broadest range of sites without making security sacrifices.  It looks like we missed a compatibility break; this bug still hadn’t been resolved, so I dove in.

I fired up my trusty HTTP debugger, Microsoft Fiddler.  Some developers in the audience might be scratching their heads-- I’ve got the full source for IE on my computer, so why start by looking at the traffic?  Simple—it’s usually the clearest way to see what’s really going on.  HTTP is beautiful that way— if you can see both the traffic of a working scenario and the traffic from a broken scenario, you can resolve most problems of this nature. 

The site noted that subscribers using Mozilla FireFox or Opera could log in just fine, so I hooked FireFox up to Fiddler and watched the traffic flow by.  Nothing seemed amiss, and the login worked as expected.  Okay, now to try out IE on XPSP2…  I performed the same login procedure, using the same credentials, and… hrm. That’s odd.  The login silently failed and I was left sitting at the “Please sign in” button on the homepage.  Very mysterious-- my curiosity was piqued. 

I started by filtering the Fiddler session list to ignore all of the image and CSS requests, since they’re unlikely to be related to login problems.  This left two HTTPS pages and two HTTP pages. 

  1. https://example.com/loginform.asp submits to
  2. https://example.com/loginverify.asp which then 302 Redirects to
  3. https://example.com/auth.asp which then 302 Redirects to
  4. https://example.com/index.asp

Using the Microsoft WinDiff feature inside Fiddler (select any two sessions, right-click and choose WinDiff from the menu), I found the body of the POST was identical in both cases, so that wasn’t the culprit.  I also determined that there are no extra cookies being set or sent to FireFox, so the magazine’s hypothesis that this bug is caused by cookies is rejected. 

The only cookie being passed around is a standard ASP Session ID cookie; I eventually decided that this cookie was probably associated with a serverside session variable set inside of auth.asp, e.g. IsLoggedIn. 

I carefully looked for differences in the headers… Nothing jumped out at me.  On a hunch, I added the following rule to my Fiddler Rules file:

static function OnBeforeRequest(oSession:Fiddler.Session)


     oSession.oRequest["User-Agent"] += " Gecko";


This rule will append “Gecko” to the browser user-agent header.  I used Fiddler’s menu options to clear my cache and cookies and then retried the login using IE. And this time, login succeeded.  What?!? That can’t be right.  I disabled the rule, cleared the cache and cookies, and tried it again.  Login silently failed.  Hrm. Let’s try something else:

static function OnBeforeRequest(oSession:Fiddler.Session)


     oSession.oRequest["User-Agent"] = "Opera/8.0 (Windows NT 5.1; U; en)";


Login was also successful using “Opera” as the user-agent string.  Interesting. Someone doesn’t want IE to log in. But why?

As with many mysteries, even though I thought I had collared the culprit, it turns out that something far more interesting was afoot.  A single fact kept tickling at the back of my mind… The bug says this site used to work before XP SP2 was released. How could that be? I don’t have an unpatched copy of IE handy, but I might be able to fake it…

static function OnBeforeRequest(oSession:Fiddler.Session)


     oSession.oRequest["User-Agent"] = "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)";


Login still failed.  There must be something else that differs…   I then recalled that I’ve got a rusty old Windows 98 image on a Virtual PC.  I fired up the Virtual PC, pointed the proxy configuration at my XP SP2 box running Fiddler, and ran the login.  It worked.  Bingo.

With the click of a mouse, I performed a WinDiff of the Windows 98 login traffic against the XP SP2 login traffic…  And I had my man.

You see, the Windows 98 request for Auth.asp carried the header

Referer: https://example.com/loginverify.asp

This violates the RFC which states “Clients SHOULD NOT include a Referer header field in a (non-secure) HTTP request if the referring page was transferred with a secure protocol.”  We fixed a bug for XP SP2 where if a HTTPS page 302 redirected to a HTTP page, the Referer was incorrectly being sent.

Could it be so simple? I added the following rule

static function OnBeforeRequest(oSession:Fiddler.Session)


     oSession.oRequest["Referer"] = "https://example.com/loginverify.asp";


And login now worked on Windows XP SP2. 

Wow, that’s really silly. Rather than developing a real security mechanism, they’re using a crusty old bug to check to see if the user is coming out of the secure login page, under the assumption that if they’re getting to Auth.asp from loginverify.asp, the login info must be correct!

It’s been said before, but I’ll say it again… never fully trust header data coming from the client, including the Referer.  It’s horrendously insecure; as we’ve just shown, it can be bypassed in a single line of code. 

Without ever looking at the source for IE or the source for auth.asp, I was able to deduce:

  • The Auth.asp page logic expected a security flaw in Internet Explorer 6.
  • Opera and FireFox never sent the referer, so any request carrying their user-agent was automatically permitted.
  • Microsoft fixed the flaw for XPSP2, causing Auth.asp to fail to work as the author expected.

I sent off my analysis to the web developer for the magazine and as I type this, logins now work with XPSP2 and IE7.  Case closed.


PS: The second installment in the Fiddler on MSDN series is almost ready… The new article is all about improving your websites’ performance.

Updated: changed "site.com" examples to be "example.com", which is correctly reserved in RFC2606 for this purpose.