Shopping for a Certificate

In the past, I signed my code using a certificate from the Comodo Certificate Authority (CA), purchased through a popular reseller named Tucows. This time, I decided to shop around a bit. My first stop was GlobalSign, but after some back and forth, I learned that they no longer sell code-signing certificates for individuals. If I wanted to get a code-signing certificate from GlobalSign, I’d need to file formal legal paperwork with the state to register a business. Since I’m not really making money on my freeware, I decided to keep looking.

My next stop was DigiCert and it proved more fruitful. I’d heard a lot of good things about DigiCert because they offer discounted certificates for SysDevs and free certificates for Microsoft employees and MVPs.

I had initially hoped to get an Extended Validation Code Signing certificate to ensure that my users didn’t encounter any temporary “Unknown Reputation” warnings from SmartScreen Application Reputation when I first switch over to signing using the new certificate. Unfortunately, it turns out that EV Code-Signing Certificates are not available to individuals from any certificate authority, so I placed an order for a regular code-signing certificate instead.

While it’s unfortunate that some users might get some “Unknown Reputation” warnings, my software is popular enough that hopefully the reputation will build on my certificate in a few days or so. My new certificate is valid for three years, so a few days of user questions shouldn’t prove a huge burden.

Validating My Identity

In past years, validating my identity to obtain a certificate was a complicated, multi-day affair involving faxing personal documents like my passport, utility bills, bank statements to the CA. In one case, I even had to ping the CEO of a certificate authority (we had met during my work on EV for HTTPS) to get the process unblocked.

In contrast, my experience with DigiCert was much more straightforward. I simply uploaded a copy of my driver’s license to their secure portal, along with a scan of a notarized document containing my address, driver’s license information, and certificate information. I was initially worried about the expense and complexity of finding a notary to countersign my document, but this turned out to be very easy. The UPS Store has notaries on staff and they charge just $6 per page notarized; the entire process took less than 20 minutes. I've been told that some banks offer free notary services to their customers. After supplying my identity proof, I got a quick call from the CA validators and was granted permission to request a certificate.

Generating a Certificate

The DigiCert portal offers a simple push-button interface for requesting a certificate. At first, the button didn’t work and I was reminded that, for Internet Explorer’s Certificate Enrollment ActiveX Control to work properly, I should put the site in my Trusted Sites Zone (especially because I run in Enhanced Protected Mode). Unfortunately, even after trusting the site, I got the same error message:

Fortunately, I quickly realized the problem: I use ActiveX Filtering to reduce my attack surface (and block annoyances) and this was preventing my use of the necessary ActiveX interface. After unblocking ActiveX controls

…and refreshing the page, the control loaded properly. It requested permission to generate a private key and certificate request:

And after I chose Yes, the new certificate was generated and signed by the Certificate Authority. The new certificate is automatically placed in the Current User\Personal\Certificates store which can be found by launching CertMgr.msc or by following these steps.

Double-clicking the certificate shows the Certificate UI which confirms that Windows has the matching private key:

Exporting the Certificate

My current build process uses signcode.exe from the Windows SDK to sign code; I supply my certificate in a SPC file and the private key in a PVK file. In order to generate these files, I must first export my certificate to a CER and a PFX file. This also allows me to back up my key, since the only copy in existence is in my Windows Certificate store.

Doing so is simple: just right-click the certificate and choose Export. Choose Yes, export the private key and select the Personal Information Exchange - PKCS #12 (.PFX) format. Save as mycert.pfx.

You’ll be prompted for a password to protect the file: do NOT forget your choice! You should probably back up this file somewhere safe and offline (e.g. a USB drive or CD) in case your PC ever suffers from an unrecoverable problem.

After exporting the PFX file (which contains both the certificate and the private key) you can export a plain CER file, which contains only the certificate. In the Certificate Export Wizard, choose No, do not export the private key and choose DER encoded binary X.509 (.CER) as the format. Save as mycert.cer.

Converting Certificate and Key Files

Note: These steps are not always necessary, for instance if you use signtool or the DigiCert Certificate Utility to sign your code. 

With the CER and PFX files in hand, you now need to convert the files into SPC and PVK files used by the signing tool.

PFX->PVK

Unfortunately, generating the PVK file containing the private key is a little tricky. Most of the tutorials on the web suggest you download a zip file containing a converter and run it. If the prospect of downloading an unsigned executable over HTTP and supplying it with your PFX and its password doesn’t set off your internal ZOMG no!!! alarms, perhaps you’re working in the wrong field.

So, if that approach is out, what do we do instead?

Here, OpenSSL comes to the rescue. First, convert to a PEM file:

openssl.exe pkcs12 -in mycert.pfx -nocerts -nodes -out mycert.pem

Then convert the PEM file to a PVK file:

openssl.exe rsa –inform pem –in mycert.pem –outform pvk -out mycert.pvk

CER->SPC

Generating the SPC file from the CER is comparatively easy. Simply use the Cert2SPC.exe utility included in the Windows SDK:

cert2spc.exe mycert.cer mycert.spc

In the unlikely event that you don’t have the Windows SDK installed, you can generate the SPC using OpenSSL:

openssl.exe pkcs12 -in mycert.pfx -nokeys -out mycert.pem
openssl.exe crl2pkcs7 -nocrl -certfile mycert.pem -outform DER -out mycert.spc

A word about Intermediates

The SPC file is a serialized certificate store that contains one or more certificates. The Cert2SPC command line I show above includes only my code-signing certificate-- it does not include the Certificate Authority's Root Certificate or the Certificate Authority's Intermediate Certificate. In contrast, the OpenSSL command shown will include ALL certificates in the SPC file.

Why do we care at all? Because if it does not have the intermediate certificate, Windows must download it using the information provided in the AIA extension of the code-signing certificate. That download will obviously fail if the PC is offline, and the user will get a "Cannot verify" warning on the signature and the file will behave as if it were unsigned. If we include the intermediate in the SPC, then the verifier doesn't need to worry about downloading it.

However... Windows also has a feature whereby not all root certificates ship with Windows; many roots are downloaded on-demand as they are encountered. So, if the end-user's computer doesn't have the necessary root installed, the signature verification logic will try to fetch it from WindowsUpdate, and if that fails because the system is offline, the user gets the same "Cannot verify" warning. As a signer, you can't simply put the root in the SPC file and avoid a network hit, however, because Windows won't trust the root until it checks with WindowsUpdate.

So, what to do? My recommendation is that you include the end-entity certificate and the intermediate certificate, but not the root.  You can do this thusly:  

cert2spc.exe mycert.cer CAIntermediate.cer mycert.spc

You can get the CAIntermediate.cer file by opening mycert.cer in the Certificate Viewer, clicking the Certification Path tab, selecting the Intermediate node of the tree, clicking View Certificate, clicking Details, and pushing the Copy to File button.

You can view the contents of the SPC file by double-clicking it:

Cleanup and Backup

Signing Files

With all files in the proper formats, you can now sign your code:

signcode -spc \src\mycert.spc -v \src\mykey.pvk -n "My App Name" -i "https://www.example.com/myApp/" -a sha1 -t https://timestamp.digicert.com MySetup.exe

I explicitly specify -a sha1 for my hash algorithm, because the default algorithm (MD5) is NOT safe. I still support a few Windows XP users, and unfortunately XP doesn’t support SHA256 for Authenticode, even with Service Pack 3 installed. In another year or two, I will stop using sha1 and will use SHA256 instead.

I always provide a timestamp URL using the -t parameter to ensure that my program’s signature will remain valid even after the signing certificate expires.

Improving Security with Hardware

As users and security software are increasingly looking for digital signatures, bad guys are now looking for ways to get their malware signed. Perhaps the simplest way for them to do so is to hack into software developers’ PCs and steal their private keys, or submit their malware into automated processes configured to sign anything they receive.

While I’ve never configured automatic signing of anything, I’ve long been worried about the threat of a bad guy stealing my private key and signing his malware with my good name.

Fortunately, it’s now relatively easy to raise the bar against attackers.

While EV-Authenticode requires use of a hardware token for signing, even non-EV signers like me can benefit from hardware-based signing. Below is a eToken 72K security token; this one and similar products are available online at prices ranging from a few dollars to about $40.

When you buy an EV code-signing certificate, you’ll get a token with the certificate and private key installed; you don’t need to do much beyond updating the password.

In contrast, when you’re setting up a token yourself, there are a few steps; my token didn’t come with instructions, but it was pretty easy to figure out.

First, you need to install the appropriate software to use the token; in my case, it was the Aladdin Knowledge Systems eToken PKI client. Next, you need to know that the default password for a new eToken is 1234567890. Supply this password and then pick a strong new password to replace it. Your new password should be memorable (if you forget it, you’ll be in a world of hurt) and probably should be relatively easy to type, as you’ll be typing it each time you sign anything.

In the left pane, select your token:

At first, I was a bit worried about the word “Java” here—was this only for signing JAR files or something? No—it’s that the token itself is running Java for its own internal operating system.

After selecting your token at the top, click the yellow gear icon to go to Advanced View:

You can then right-click and choose Import Certificate.

Since we’ve left our certificate in the Windows Certificate Store, we’ll choose that option:

Next comes a very confusing prompt:

With fingers crossed, we click OK and get an error message:

Hrm. So we click Cancel and voila, our certificate in the Windows Certificate store appears:

My guess is that the first “Smart Card” prompt was there in case we wanted to copy a certificate and key from a different smartcard to our token. When we hit Cancel, it then just shows the Windows Certificate stores. We select the desired certificate and click OK. After supplying our token’s password, the certificate is imported.

After we’ve moved the token to the token, the certificate still appears within the Windows Certificate manager (CertMgr.msc):

…until we unplug the token and hit F5. At that point, the code-signing certificate disappears:

Great!

Signing Files From the Token

Because we’re no longer going to use the private key file from disk, we obliterate the mycert.pvk file and update our command line as follows:

signcode -p "eToken Base Cryptographic Provider" -spc \src\mycert.spc -k df2852d2a58a1cc5ce82d186e0fb6eda_0b960da4-1609-44a5-bfa9-aac9caea8170 -n "My App Name" -i "https://www.example.com/myApp/" -a sha1 -t https://timestamp.digicert.com MySetup.exe

We first specify that the signing should use the eToken Base Cryptographic Provider; you can find the list of available providers by looking in your registry under the HKLM\Software\Microsoft\Cryptography\Defaults\Provider node.

We next replace our reference to the PVK file with the name of the container containing the private key. Fortunately, the eToken software exposes the key’s Container name information directly in the UI:

Now, when we run the updated signcode command, the eToken software prompts for our password and returns the signature.

Here Be Dragons

At this point, I was pretty excited at how easy it was to use hardware to bolster security. For fun, I tried modifying my command line

signcode -spc \src\mycert.spc -k df2852d2a58a1cc5ce82d186e0fb6eda_0b960da4-1609-44a5-bfa9-aac9caea8170 -n "My App Name" -i "https://www.example.com/myApp/" -a sha1 -t https://timestamp.digicert.com MySetup.exe

…to omit the provider directive, and signing succeeded. And it didn’t prompt me for a password.

I had a bad feeling about this.

I unplugged my eToken from the PC and ran the command again.

And it succeeded, without either a password or the token.

What the what?!?

Fortunately, I have spent a few weeks banging my head against the wall with problems with Windows Certificate Key storage in the past, and I had a theory about what was going on—perhaps the process of importing the certificate to the token did remove the certificate from the Windows Storage, but did not properly blow away the private key? (That mistake recently got a bit of press because one high-profile piece of ransomware also left a copy of its RSA key locally.)

With this hunch, I searched my RSA keys folder for any files containing the key container name:

…and I got exactly one hit. After I deleted the key container file, signcode started behaving as expected: I could only sign the file using the eToken Base Cryptographic Provider and only when the token was inserted and the password supplied.

After having deleted my private key files (both the \Crypto\RSA\ file and the PVK file) from my hard disk, I ran cipher /w:C:\ to help ensure that the key files could not be recovered in the future.

While I feel pretty good about my new level of security, there’s no question that my private key would be more secure if I had been able to obtain an EV-Certificate token with a non-exportable key pre-installed.

Secure all the things!

-Eric Lawrence
MVP - Internet Explorer

Spartan PM Jacob Rossi wrote about the new Project Spartan rendering engine.

Spartan Developer Justin Rogers has a great new blog on development in general, including some tantalizing posts on evolving the Spartan codebase.

Windows 10 build 9926 has been released; Spartan is not yet in it, but you can enable the new EdgeHTML engine using the about:flags page; set the Experimental Flag to Enabled. Included in the build are support for HSTS, HTTP Live Streaming, Video Tracks, DOM L3 XPath, and improved developer tools make an appearance.

EdgeHTML includes support for Content Security Policy, a powerful technology to defang XSS and content-injection attacks. David Risney recently released a Content Security Policy rule generator extension for Fiddler. Surf your site with Fiddler running and this extension will compute the appropriate CSP rules for your site.

-Eric Lawrence

Securing your websites and services using HTTPS has never been more important, or more complicated. In this talk, a former browser Security Program Manager covers the best practices for using HTTPS today. Topics covered in this session include ciphers and hash algorithms, forward secrecy, handshakes and protocols, and new browser features like HSTS and PKP. Explore how attackers circumvent HTTPS, and what you can do to help protect your visitors and customers safe from snoops and bad guys. Secure all the things!

You can find the slides for my talk here (pptx; 4.5mb) and an audio recording of the presentation here (mp3; 30mb). During the talk, I mentioned a few topics repeatedly:

• SSLLabs' Server Test - Test your server's use of HTTPS
• Ivan Ristić’s excellent Bulletproof SSL and TLS book - Understand HTTPS, attacks, and how to deploy
• The Let's Encrypt Project

The last item is perhaps the most exciting: the EFF’s Let’s Encrypt project is slated to offer free HTTPS certificates starting in the summer of 2015. There’s a great summary of the project in this video presentation by Seth Schoen from late last year.

HSTS and Privacy

I also wanted to take this opportunity to provide some context related to recent press about the “super-cookie attack” against HTTP Strict Transport Security.

In this attack, which I alluded to at the end of my intro to HSTS post last year, a site generates a large set of HSTS rules and uses these to act as a type of “super-cookie” identifier that can be used to uniquely identify a given client. While much of the coverage is of the breathless “Wow, so clever! Browser designers are fools!” form, the reality is that this attack has been well known since the dawn of the HSTS proposals, and it’s even explicitly called out in the Chromium team’s (excellent) Technical Analysis of Client Identification Mechanisms document.

The attack vector forces browsers to make a direct tradeoff between privacy and security—ignoring HSTS decreases security, but respecting it can impact the user’s privacy. Safari’s approach (an undetectable, undeletable super-cookie) is clearly bad, while Chrome’s approach (non-Incognito-set HSTS rules carry over to Incognito mode, while the reverse is not true) more carefully straddles the line.

It’s not yet clear how Internet Explorer 12 will behave, leading to the following design considerations:

1. Will IE offer a “pre-load” list of HSTS sites, improving security with no privacy impact?
2. Will IE allow InPrivate-set HSTS rules to apply to non-InPrivate mode?
3. Will IE allow InPrivate-set HSTS rules to apply to future InPrivate sessions?
4. Will IE allow Non-InPrivate-set HSTS rules to apply to InPrivate sessions?
5. Will deleting browser history clear all or all non-preload HSTS rules?

Beyond expanding the pre-load lists, I’m hoping that future browsers can use some additional heuristics to balance the privacy/security tradeoff. For instance, browsers could partition header-set HSTS rules by BrowsingPrivacyMode and clear them when the user dumps history, but persist any rules for sites that heuristics suggest are “legitimate” (e.g. the user has created a favorite, directly navigated via the address bar, etc).

HTTPS Everything 

One fact is increasingly clear—moving to HTTPS is crucial to helping protect the user’s privacy. In just one example, it’s recently been observed that a popular Mobile ISP is injecting a tracking identifier into all outbound HTTP request’s headers (regardless of privacy mode) and this header is being used to reconstitute cookies that users have manually deleted. Moving to HTTPS helps defeat shenanigans like this.

As a user: Demand HTTPS.

As a site: Deploy HTTPS.

Secure all the things!

-Eric Lawrence
MVP, Internet Explorer

As you can see in the example provided in the aforementioned blog, when referenced as the source of a script element, the script runs, and when referenced as the source of an img element, the GIF image renders:

Neat. But the secret is quickly revealed if we peek at the raw bytes:

As you can see, this image starts with the standard GIF89a signature. Then, the cleverly-crafted file takes advantage of the fact that the subsequent bytes encode the size of the image. The creator specifies that the image shall be 8,253 pixels wide by 8231 tall, which is encoded using the hexadecimal values 0x203d 0x2027, which are encoded in GIFs as 3d20 and 2720 with the least significant byte first.

When interpreted by the JavaScript parser, these four bytes are interpreted as an equals sign, space, single quote, space. This, in turn, causes a JavaScript parser to read the file as a variable declaration, followed by more JavaScript statements, and ending with a long garbage comment.

So, why is this interesting? Is it a horrible security hole?

No, not really.

Back before IE8, IE would eagerly seek out HTML even inside files served as images, and as a consequence, websites that accepted user-supplied images and served those images were at risk because any HTML documents “sniffed” from the files would be running in the security context of the victim site. IE8 changed things such that HTML would never be sniffed from files delivered with an image/ MIME type.

MIME-Sniffing HTML from images was much more dangerous than sniffing script, because script runs in the context of the hosting page, not in the context of the server that delivers the image/script. So, in order for something bad to happen today, a victim site must not only allow polyglot files to be uploaded, but also it must allow an attacker to inject a <SCRIPT SRC> tag into the markup that the site serves. In most cases, this is not very practical/interesting (especially because any such injection probably would be an XSS in and of itself).

Nevertheless, there’s still a bit of interest here because polyglot files could conceivably be used as an attack against Content Security Policy. Consider a CSP-protected site that decides to allow visitors to specify any SCRIPT SRC they like, relying on CSP to ensure that the SRC attribute points to a “safe” server. Also, our victim site must allow image uploads to a path under the CSP-specified script-src. With a polyglot file, the server is conceivably at risk, despite CSP.

There are a few best practices that help protect against attacks like this:

1. Don’t ever serve user-submitted content from your application’s domain. If your application runs on example.com, serve user-submitted content from example-usercontent.com.
2. When specifying a CSP script-src directive, ensure that it specifies a location where only known and trusted scripts reside.
3. Consider resaving (and optimizing) user-submitted image files. Reencoding image files can improve performance, privacy (dropping GPS coordinates, etc), and security (by breaking polyglots).
4. When serving user-submitted image files, serve them with the X-Content-Type-Options: nosniff response header.

The last item bears some explanation. Back in IE9, the MIME-Sniffing logic was improved such that a SCRIPT element whose response bore the nosniff directive would be rejected if the MIME type does not match one of the following values ["text/javascript", "application/javascript", "text/ecmascript", "application/ecmascript", "text/x-javascript", "application/x-javascript", "text/jscript", "text/vbscript", "text/vbs"]. This protection was added to help protect against other types of attacks (specifically, the leaking of cross-origin information in JavaScript error objects) but it’s useful here as well.

We can add this response header using Fiddler's Filters tab to prototype the effect. As you can see below, Chrome 39 also supports the nosniff directive, but Firefox 36 does not:

As far-fetched as the attack scenario may be, it still may make sense for browser vendors to require a proper MIME type on any SCRIPT response when CSP is in use. They might even want to broaden the existing logic such that image/ responses can never be used as anything other than an image.

-Eric Lawrence
MVP, Internet Explorer

Virtually any network-based application can be made faster by optimizing the number of bytes transferred across the network. Taking advantage of caching is a great way to minimize transfer sizes, but just as important is to reduce the size of the resources you transfer.

Data compression is used throughout the protocols and formats used by browsers, and today’s post is a summary of where you can find compression in use and how you can optimize it to improve the performance of your sites and services.

Compression Categories

Conceptually, compression is simple: recognize patterns in data and reduce repetition to minimize the size of the data. There are two major categories of compression: lossy and lossless.

When information is compressed with lossy compression, reversing the process (decompression) doesn’t result in the original data, but instead a surrogate which resembles the original either closely or loosely depending on the quality of the compression. In browsers, the primary uses of lossy compression are in the JPEG image file format, the MP3 audio file format, and the MPEG video file format. These formats utilize understanding of human perception to drop details that are unlikely to be noticed by the viewer/listener, resulting in often-huge savings in data size. While important, this post won’t discuss lossy compression any further.

In contrast, decompressing data that was losslessly compressed results in an exact copy of the original data—every byte is identical. Lossless compression is used throughout the web platform, in both the HTTP layer and internally within many web file formats including PNG, PDF, SWF, WOFF, and many more. The remainder of this post explores the use and optimization of lossless compression.

HTTP Headers

HTTP requests advertise the decompression algorithms the client supports using the Accept-Encoding request header.

Servers indicate which compression algorithm was used for a given response using the Content-Encoding response header. Unfortunately, even the updated HTTP/1.1 specification implies that servers can use compression-related tokens in the Transfer-Encoding response header, but indicating compression using the Transfer-Encoding header does not work in any major browser (all support only the chunked token).

If the server utilizes compression, it should generally include a Vary: Accept-Encoding response header to help ensure that a cache does not mistakenly serve a compressed response to a client that cannot understand it.

Algorithms

The most popular compression algorithm in use on the web is the DEFLATE algorithm, specified in RFC 1951. DEFLATE combines the LZ77 algorithm with Huffman encoding; it is straightforward to implement and effectively compresses a wide variety of data types. You can read a straightforward explanation of the algorithm here, and watch a fun video showing how repeated byte sequences are referenced here.

In browsers, the most obvious use of the DEFLATE algorithm is for HTTP/1.1 Content-Encoding. DEFLATE is the algorithm underlying two of the three encodings defined in the HTTP specification (“Content-Encoding: gzip” and “Content-Encoding: deflate”).

DEFLATE vs. GZIP for Content-Encoding?

Given that virtually all clients allow both “Content-Encoding: gzip” and “Content-Encoding: deflate”, which should you use?

You’ll find conflicting opinions on the topic, many of which are written without the understanding that both encodings are based on exactly the same algorithm, and the encodings differ only in the header and trailer bytes that wrap the compressed data.

The GZIP encoding of a resource starts with two magic bytes 0x1F 0x8B, a byte representing the compression method (0x08, indicating DEFLATE), seven additional bytes of metadata, several optional fields (rarely used), and then the DEFLATE-compressed bytes. After the compressed data, a 32bit CRC-32 and 32bit original datasize field complete the format.

Despite its name, the DEFLATE encoding of a resource is specified to be the ZLIB data format. This format consists of two header bytes; the first contains the compression method and window-size information (the low bits of the byte are 0x08, indicating DEFLATE). The second header byte is a flag byte which indicates the compression strategy (minimize size vs. maximize speed) and serves as a checksum of the header bytes. The DEFLATE-compressed bytes follow. After the compressed data, a 32-bit ADLER32 checksum field completes the format.

So, all other things being equal, a C-E:GZIP-encoded resource will be exactly 11 bytes bigger than a C-E:DEFLATE-encoded resource, due to the seven additional header bytes and four additional trailer bytes. In practice, however, your compression program may generate wildly different results.

Now, note that I said the encoding is specified to be the ZLIB data format. Here’s what you’ll see in Internet Explorer if you try to load a page which has the ZLIB-wrapper around the DEFLATE data:

The problem is that Internet Explorer expects a bare DEFLATE stream without the ZLIB wrapper bytes. So, does that mean you can’t use Content-Encoding: deflate? Not really, no. In addition to supporting the proper ZLIB-wrapped data, Chrome, Firefox, Safari, and Opera all support bare DEFLATE.

Some folks recommend you avoid C-E:Deflate and just stick with C-E:GZIP. It’s hard to argue with that approach.

Optimizing DEFLATE

No matter which DEFLATE-based Content-Encoding you choose, the quality of the implementation underlying DEFLATE implementation is key to reducing the size of the data stream. Many developers mistakenly assume that “DEFLATE is DEFLATE”, and any compressor that implements the algorithm will get the same result. This is a mistake.

Instead, think of DEFLATE’ing as solving a maze. A really really complicated maze. You might run around as fast as you can and eventually stumble upon an exit, but that’s very different than finding the optimal path. If you instead spent much longer, frequently retracing your steps to find out whether there’s a shortcut you missed, you may find a much shorter path. DEFLATE works much the same way—you can expend more resources (CPU time and memory) in the compression process finding the optimal compression choices. The best part is that expending resources to optimize DEFLATE compression doesn’t typically increase decompression time—uncompressing DEFLATEd content is a comparatively straightforward process.

For static resources that will be reused often (think of jQuery.js and other frameworks) you should be delighted to trade a one-time compression cost for a millions-of-times transfer size savings. Fortunately, folks at Google have done the hard work of making a great DEFLATE implementation, called Zopfli (pronounced "zopflee"); you can read more about it and its real-world savings over here. Unfortunately, far too few teams have integrated Zopfli into their workflow; even Google hasn’t gotten around to using it for most of their resources… yet.

If Microsoft were to use Zopfli when building the browser-based versions of Office, they’d see significant savings; three of their largest files shrink by just over 4% each:

Of course, you won’t always be able to use Zopfli to compress your resources—the compression tradeoff isn’t appropriate for dynamically-generated responses which will only be served once. But you can use it for more than just your HTML, CSS, and JS. Read on for more details.

Exotic Encodings

One of the first criticisms of Zopfli was “it seems like an awful lot of effort for a small improvement. Perhaps it’s time to add a better compression method.” And this criticism is valid to a certain extent.

bzip2

For instance, the first versions of Google Chrome added support for bzip2 as a HTTP Content-Encoding, because (quoting a Google engineer) “well, we had the code laying around." Bzip2 yields much better compression than even Zopfli-optimized DEFLATE:

And bzip2 doesn’t even offer the highest compression ratios: lzma2 showed even better results when we looked at it back in the IE9 timeframe.

The challenge with more advanced compression schemes is that there are billions of clients that support DEFLATE-based schemes, and as we’ve seen previously, most implementations haven’t yet optimized their use of DEFLATE. Making matters worse are intermediaries: Google reportedly encountered network proxies and security applications that would corrupt bzip2-encoded traffic.

Xpress

Some Microsoft products (Exchange, Software Update Services, etc) use Content-Encoding: xpress, a scheme based on LZ77 (like DEFLATE), optimized for compression speed. Xpress encoding is not used by Internet Explorer, WinINET, WinHTTP, or System.NET, the dominant HTTP implementations on Windows. When run on Windows 8 or later, Fiddler can decompress Xpress-encoded content using the native libraries.

SCDH

Back in 2008, Google proposed the SCDH (Shared Dictionary Compression over HTTP) Content-Encoding, and support has subsequently been added to Chrome and Android. Despite offering large potential savings, this lossless compression scheme is not broadly used and is comparatively complex to implement. It also has privacy implications.

Compress

The only non-DEFLATE-based HTTP-specification defined encoding (“Content-Encoding: compress”) is based on the LZW compression algorithm. Compress is not broadly supported (it doesn’t work in any major browser) and a quick test suggests that it’s not even as effective as a basic DEFLATE encoder.

What Gets Compressed

In general, HTTP Content-Encoding based compression applies to only the response body. This means that HTTP headers are not compressed, a significant shortcoming in the compression scheme. For instance, I recently observed Facebook delivering a 49 byte GIF file with 1050 bytes of HTTP headers, an overhead of over 2000% for a single file. Similarly, Content-Encoding is rarely applied to request bodies.

TLS Compression

When negotiating a HTTPS connection, the client may indicate support for automatic compression of all data on the connection, which has the benefit of applying to both headers and bodies of both the request and the response. TLS-based compression was not broadly implemented (Microsoft products have never supported it) and in 2012 it was disabled entirely by major products to address an exploit called CRIME (Compression Ratio Info-leak Made Easy).

HTTP/2 Compression

The HTTP/2 draft 14 enables header compression using an algorithm called HPACK, designed to combat the CRIME exploit against the SPDY protocol (which used DEFLATE to compress header fields). Draft 12 of the specification removed per-frame GZIP compression for data; you’ll still use Content-Encoding.

Compressing WebSocket Data

Check out https://www.igvita.com/2013/11/27/configuring-and-optimizing-websocket-compression/ for discussion of a mechanism to use DEFLATE to compress data sent over WebSockets.

Compressing Request Bodies

In theory, HTTP allows clients to compress request bodies using the same Content-Encoding mechanism used for HTTP responses. In practice, however, this feature is not used by browsers and is only rarely used by other types of HTTP clients.

One problem is that a client does not know, a priori, whether a server accepts compressed requests. In contrast, a server knows whether a client accepts compressed responses by examining the Accept-Encoding request header. Many servers do not accept compressed requests, although some have no objection (e.g. onedrive.live.com allows them). One consequence of this is that HTML forms do not (yet?) expose any way for authors to specify that compression of the request body should be undertaken.

Modern web applications can workaround the shortcomings in browser APIs with script:

1. Use HTML5 File API to load a file for upload into JavaScript
2. Compress it using a script-based compression engine (of many, Pako and compressjs look nice)
3. Upload the compressed array to the server.

You can use Fiddler to see whether your server accepts request bodies with Content-Encoding; click Rules > Customize Rules. Scroll to OnBeforeRequest and add the following code inside the function:   

  if (oSession.HTTPMethodIs("POST"))
   {
     oSession["ui-backcolor"] = "yellow";
     oSession.utilGZIPRequest();  
   }

If the server accepts the upload, you should see a normal HTTP/200 response (although you may wish to ensure that the server properly recognized the compression and didn’t just treat it like binary garbage). If the server doesn’t accept compressed uploads, you will likely see a HTTP/400 or HTTP/500 error code.

Compression Bombs

One reason that servers might be reluctant to support compressed uploads is the fear of “Compression bombs”. The DEFLATE algorithm allows a peak compression ratio approaching 1032 to 1, so a one megabyte upload can explode to 1 gigabyte. Such attacks are generally possible against client applications like browsers, but tend to be much more potent against servers, where a single CPU is required to serve thousands of users simultaneously.

Protecting against maliciously crafted compression streams requires additional diligence on the part of implementers.

Best Practice: Minify, then Compress

Because DEFLATE yields such great reductions in size, you might be thinking “I don’t need to minify my assets. I can just serve them compressed.”

From a networking point of view, you’re mostly correct: the bytes-on-wire for Minify+DEFLATE will likely be nearly the same as DEFLATE alone. However, there are other concerns:

Memory: The browser must decompress your stylesheets, script, and HTML to strings, so a 50kb script file, compressed to 5kb, still uses 50kb of memory after it has been decompressed. If you can minify the script to 40kb before compressing, you can save at least 10kb of memory on the client (and likely more). This is particularly important on memory-constrained mobile devices.

Disk Cache: Some browsers store compressed responses in the cache in compressed form (and Firefox will even proactively compress uncompressed responses). However, Internet Explorer removes all Content-Encoding when writing to the cache file, meaning that your 50kb script file will occupy 50kb of the client’s cache. Smaller files will reload faster and are less enticing candidates for eviction when cache space is reclaimed.

File Formats

As mentioned previously, many file formats internally use DEFLATE compression; a great example is the PNG image format. Unfortunately, many generators of those formats opt for minimum compression time, failing to achieve maximum compression.

Fiddler now includes PNGDistill, a simple tool that removes unnecessary metadata chunks and recompresses image data streams using Zopfli. Zopfli-based recompression often shaves 10% or more from the size of PNG files. You can access this tool from the context menu of the ImageView inspector:

The “Minify-then-compress” Best Practice applies to image file types as well. While large fields of empty pixels compress really well, the browser must decompress those fields back into memory. So, if you're building a sprite image with all of your site's icons, don't leave a huge empty area in it.

Beyond PNG, the WOFF font format also uses DEFLATE. Google recently shrank their WOFF font library by an average of 6% by reencoding their internal DEFLATE streams using Zopfli. Not content to stop there, Google has also introduced the WOFF2 format, which utilizes an innovative new algorithm known as Brotli, yielding size reductions around 20% over WOFF. WOFF2 is currently usable in Chrome, Opera, and Firefox dev builds; see CanIUse.com and status.modern.ie.

Compression-Related Trivia

Here are a few points of compression-related arcana that have come up over the years…

1. The HTTP decompression implementation used by Internet Explorer 6 and earlier was inside URLMon; the WinINET HTTP stack did not itself support decompression. As a consequence, there was a “dance” whereby WinINET would download the compressed version and ask URLMon to decompress the response in place, replacing the original compressed body. This “dance” was buggy in many corner cases, and led to problems where content could not be decompressed, or Vary headers were ignored, or similar.

In Internet Explorer 7, decompression was moved from URLMon down into WinINET, eliminating the “dance” and slaying many bugs.

2. Unfortunately, the fact that decompression happens so low in the stack causes its own minor problems, like the F12 Developer Tools not being aware of the use of Content-Encoding.

3. Some Apache servers will mistakenly serve .tar.gz files with a Content-Encoding: gzip header, leading the browser to eagerly decompress the downloaded file into a raw tar file with now-misleading filename extension. Firefox works around this issue by ignoring the Content-Encoding header if the response Content-Type is application/x-gzip. Internet Explorer 11 recently and briefly had a regression in its handling of related scenarios.

4. Despite the fact that the GZIP footer includes a CRC32 and OriginalSize field to enable detection of corruption, no browser complains if these fields are incorrect or missing entirely (which makes some sense, insofar as the original requester has already read the entire response before these fields are available). But you should set these correctly, or various tools and middleware might reject your content.

5. Even worse, a small number of servers apparently fail to update the HTTP Content-Length header when compressing the body.

-Eric Lawrence
MVP, Internet Explorer

Yesterday, Microsoft released the newest version of Microsoft Message Analyzer (v1.1), which offers packet capture as well as more exotic capture types (ETW, etc). You can read the announcement and download from here: https://blogs.technet.com/b/messageanalyzer/archive/2014/09/15/announcing-the-message-analyzer-1-1-release.aspx.

Note: If someone else happens to send you a Message Analyzer capture in .cap format, you can extract any HTTP traffic from it into Fiddler by choosing File > Import > Packet Capture.

-Eric Lawrence
MVP, Internet Explorer

Digitally signing your code helps to ensure that it cannot be tampered with, either on your servers, or when it is being downloaded to a user’s computer, especially over an insecure protocol like HTTP or FTP. This provides the user the confidence of knowing that the program they’ve just downloaded is the same one that you built.

This summary is mostly correct, but it oversimplifies things a bit. In this post, we’ll take a deep dive into Authenticode to show that things aren’t quite as simple as they might appear.

Personalized Installers

Say you’re a service like Copilot, Dropbox, or GoToMeeting. Each of these services has a website and a client application that users download to interact with the service. However, unlike many traditional applications, each of these downloadable applications is meant to be personalized to an individual user; for instance, Copilot wants the application to connect to the remote assistance session without any prompts, GoToMeeting wants the user to immediately join an in-progress conference, and Dropbox wants their app to immediately begin syncing files to the user’s cloud account.

Each of these services is security-conscious, so they sign their installers with Authenticode to prevent tampering and to ensure that users have a painless install experience.

However, the use of Authenticode makes installer personalization tricky. There are three approaches:

1. Have the installer attempt to detect the user by groveling data from browsers
2. Dynamically generate and sign a new installer each user upon download
3. “Cheat” -- Find some way to use one Authenticode signature for multiple different installers

Browser Groveling

Approach #1 is tricky: there are at least three popular browsers on Windows (Chrome, Firefox, and Internet Explorer). Of those, only Internet Explorer offers a standard published API for native code applications to retrieve cookies or cached data. And unfortunately, Protected Mode and Enhanced Protected Mode made things much harder.

Prior to the introduction of Protected Mode, an application could simply call InternetGetCookieEx and grab Internet Explorer’s cookie for the service’s website. However, that API doesn’t work to grab Protected Mode cookies, and the replacement API (IEGetProtectedModeCookie) doesn’t work if the calling process is running as Administrator, which many installers do. And if the user is using the Windows 8 “Immersive IE” or manually enabled Enhanced Protected Mode, there’s no API available to get the cookie at all.

While the “Browser Groveling” approach was once popular (it’s why the IEGetProtectedModeCookie API was created to begin with), it probably isn’t used very often these days.

Dynamic Generation and Signing

Approach #2 is conceptually simple: every time you get a download request, a server-side process prepares a custom version of the installer executable (say, by replacing a reserved block of null bytes with a token) and then signs the resulting program.

On one hand, this is quite straightforward, but it might be computationally expensive to generate a new installer and signature on every single download. Furthermore, using this approach requires careful attention to security: it would be insanely risky to put the code-signing private key on the web server and have that server do the work; you’ll want to have the frontend web server invoke a backend service running on a different machine to do the generation and signing.

The Copilot.com installer appears to make use of this architecture.

Cheating Authenticode

Approach #3 seems either silly or crazy—if the whole point of using Authenticode is to ensure the integrity of the code, then there shouldn’t be any way around that, right?

The devil, as always, is in the details.

using Authenticode is to ensure the integrity of the code

Authenticode indeed verifies the integrity of the code. But the code doesn’t make up the entire downloaded executable.

Microsoft helpfully publishes the Windows Authenticode Portable Executable Signature Format (and the less relevant Microsoft PE and COFF Specification) document, which outlines in great detail how Authenticode works.

The first document includes this very helpful map:

Items marked with a grey background aren’t included in the signed hash; the latter two grey blocks come from the signature itself, and the injection of the signature changes the file’s embedded checksum which is recomputed.

So, this all looks good, right? The signer is hashing everything in the file except for the bits related to the signature itself.

And there’s the rub—the signature blocks themselves can contain data. This data isn’t validated by the hash verification process, and while it isn’t code per-se, an executable with such data could examine itself, find the data, and make use of it.

WIN_CERTIFICATE Structure Padding

The first place that sneaky developers found to shove additional data is after the PKCS#7-formatted data. The WIN_CERTIFICATE structure has a length field which specifies how many bytes will be used to hold certificate data; this value can be larger than the number of bytes used by the certificate data.

MVP Didier Stevens wrote a great blog post describing this technique and offering a downloadable tool (with source) that allows you to explore whether there is additional data stored after the embedded certificates.

For instance, you can see this trick used to embed tracking data in an old “offline installer” for Chrome:

In December 2013, Microsoft announced that some developers had foolishly used this trick to store URLs of code that would be downloaded and installed by a signed “stub” installer. The bad guys noticed that they could edit these “stub” installers, changing the embedded URLs to point to malware, and the signature of the stub wouldn’t change. The file would appear legitimate and would pass all Authenticode checks, and when run, would proceed to pwn the victim’s computer.

Microsoft announced that this trick would be blocked in a future update by changing Authenticode Verification to ignore the signature in any file with non-null bytes after the certificates. Originally, the plan was to enable stricter verification by default in June of 2014, but after evaluating the compatibility impact, Microsoft decided not to turn on the restrictions by default; a registry key must be set to enable enforcement (Change your mind? Undo).

Unvalidated Attributes

You can imagine my surprise when Dropbox announced in early August 2014 that they were sending signed personalized installers to users. Even with the EnableCertPaddingCheck registry key set, the Dropbox client’s signature was deemed valid. How was this possible?

Microsoft’s blog post provides a hint:

Although this change prevents one form of this unsafe practice, it is not capable of preventing all such forms; for example, an application developer can place unverified data within the PKCS #7 blob itself which will not be taken into account when verifying the Authenticode signature.

Looking back at the Authenticode specification, we see:

unauthenticatedAttributes

If present, this field contains an Attributes object that in turn contains a set of Attribute objects. In Authenticode, this set contains only one Attribute object, which contains an Authenticode timestamp.

The prose in red is a bit misleading—while Microsoft might have expected only a timestamp to be stored there, in practice, a signer can put anything they want. So they do.

In Windows Explorer, right-click DropboxInstaller.exe and choose Properties. Click the Digital Signatures tab and click the Details button. On the Advanced tab, scroll to the bottom. In addition to the attribute “Counter Sign” (which is a cryptographically-signed timestamp, as described in my original blog) there’s an attribute whose OID (1.3.6.1.4.1.42921.2.1) lacks a friendly name. That attribute contains a 1KB block of unsigned data, in the format described in Dropbox’s post.

The GoToWebinarLauncher.exe uses the same trick, with a different OID (1.3.6.1.4.1.3845.3.9876.1.1.1) and a 6840 byte data block:

Is Unverified Data Safe?

Embedding unverified data into signed executables is absolutely a way to shoot yourself in the foot, but it is possible to do it securely. If you plan to go this route  you must threat model and penetration test your approach to ensure that none of your code relying on such data trusts it. By sneaking around behind Authenticode, Windows is no longer validating the authenticity of that data—you must do so yourself.

Best Practices

The following are some best practices to consider:

• Don’t do this at all. Getting it right is very hard, and Windows may decide to break you in the future.
• Embed a signature within the data block yourself. If it doesn’t validate when your code examines it, reject the data.
• Do not embed private data in the block; the user may share the installer or fail to protect it because they don’t realize it contains their private data.
• If the data contains URLs to other code, validate that code’s signature when it is downloaded.

MD5

Last year, I sent an email to the Copilot team noting that they were still using the MD5 hash algorithm when signing their code. I pointed out that changing to use the more secure SHA algorithm would probably be a trivial matter of adding a command line argument to their existing signing command (e.g. -a sha1).

Unfortunately, they haven’t made this change yet [Update: They moved to SHA1 on 9/4/2014]; while they haven’t told me why, my guess is that it’s related to the architecture they invented to dynamically sign the installers that they generate.

Why does this matter? Because MD5 is scary broken and should not be used.

A digital signature is only as strong as the hash algorithm to which the signature was applied. If a hash algorithm is broken, signed hashes could be repurposed by bad guys.

When it comes to hashing, there are two types of attacks:

• Collision Attack
• Preimage Attack

In a collision attack, an attacker can carefully craft two different programs (one good, and one evil) that hash to the same value. In Authenticode, this would only be interesting if he could convince someone else to sign the “good” program; that signature he could then apply to the “evil” variant. This attack is possible against MD5 today.

In a second-preimage attack, an attacker takes anyone else’s insecurely-signed legitimate program as input and he generates a new evil program that hashes to the same value as the legitimate file. He can then copy the signature to his file and it will appear that the evil file was signed by the good guy. As far as we know, there aren’t cheap MD5 preimage attacks yet, but there will be.

Best Practice: Don’t use MD5.

While much stronger than MD5, SHA1, the default used by most certificates and tools, is starting to show weakness as well. Over the next few years, it's expected that most certificates and signatures will move to using SHA256 or SHA512 as the hash algorithm. To use SHA256 with signtool.exe, specify the /fd SHA256 command line argument. Note that SHA256 Authenticode signatures cannot be created using the older signcode.exe tool, and cannot be verified on Windows XP, even when Service Pack 3 (which supports the use of SHA256 certificates) is installed.

-Eric Lawrence
MVP

The first rule for building high performance web sites is to make fewer requests, and using CSS sprites is one key and commonly-deployed means of doing so. Unfortunately, however, some websites are trading one performance problem (too many requests) for another problem (increased memory load on the client).

The first problem is that some designers treat “empty space” inside sprite images as “free,” because empty areas compress so well that the file size doesn’t grow. As a consequence, they may generate images that are 50% empty or worse. The problem is that the client ultimately still needs to allocate memory for those regions when the image is decompressed for rendering. So, don’t leave large gaps in your sprite collections. You should leave a little padding between the sprites in the image, however, because if the sprites aren’t padded at all, and the user has zoomed the website, a GPU may sample adjacent pixels when rendering the scaled sprite.

Surprisingly, the shape of your sprite image matters as well. Consider Facebook’s primary sprite image. It’s 26 pixels wide and 8227 pixels tall, and uses 32bit color. To optimize performance and memory consumption on the client, sprite collection images should be square(ish) and their dimensions should be a power of 2; this image would consume less memory on the system and GPU if it were instead 512x512 pixels.

Note that further network optimization is available: If I use RIOT Optimizer to convert Facebook’s image to use an optimal 256 color palette, the image shrinks from 143kb to 60kb; I couldn’t see any visual differences.

You can easily find un-optimized sprite images with Fiddler. To add an Image Dimensions column to Fiddler’s UI, right-click the column headers, choose Customize Columns, and add the column like so:

-Eric Lawrence

Bonus Tip: When using JPEG (for images on your site), you should use Progressive format and 4:2:0 or 4:2:2 chroma subsampling to improve size and optimize memory usage. Fiddler’s ImageView inspector will tell you what chroma subsampling was used and whether or not a JPEG is Progressive.

1. All HTTP connections to the HSTS-enabled host must take place over HTTPS
2. Any errors in the HTTPS certificate or handshake are fatal without override

HSTS was standardized in RFC6797 back in 2012, but the original feature idea dates back to 2008.

While conceptually relatively simple, there are a lot of interesting intricacies around HSTS that we will explore in this post.

Threats Mitigated

HSTS was designed to combat the threat model whereby an active network attacker is able to control the connection between a victim browser and the Internet at large. For instance, imagine you pull out your laptop at the coffee shop and innocently type paypal.com in the address bar and hit Go. Your network request is likely to hit the network as an insecure request for http://paypal.com and the attacker can easily intercept this and reply with a fake version of PayPal to collect your credentials.

Similarly, even if you were to type https://paypal.com, the attacker could intercept the request, and, after supplying a fake certificate, pose as the secure site. Your browser will typically show a certificate error page at this point, but too many users ignore such errors and proceed anyway. Researchers have built a variety of hacking tools like sslstrip to automate such attacks.

The stricter handling imposed by HSTS helps combat both of these threats.

Opting-In

Typically, a site opts-in to HSTS by sending a Strict-Transport-Security response header with a max-age value and optionally an includeSubDomains directive. The max-age value defines the number of seconds for which the rule should remain in force; longer values are preferable. For instance, PayPal’s current rule specifies a max-age of 14400 seconds, which means that the protection expires every four hours. If you were to load PayPal at home before bed, your HSTS protection would have evaporated by the time you opened your laptop in Starbucks the following morning. In contrast, Twitter’s max-age=631138519 rule provides an incredible 20 years of protection, and addons.mozilla.org is protected for a reasonable one year (max-age=31536000).

The Bootstrap Problem

HSTS (and similar specifications) suffer from what’s called a “bootstrap problem”—until and unless a user visits the target site over a secure connection, the protection is not activated. To mitigate this problem, some browsers offer what’s called a preload list, in which some HSTS rules are distributed with the browser or its updates directly. This ensures that users are protected immediately and neatly resolves this problem as well as a privacy threat discussed later. The downside of this approach is that it requires the site operator collaborate with browser vendors instead of simply setting a HTTP header.

One exciting possibility is that some of the many new Top-Level-Domains may opt-in for their entire TLD tree, such that every single site under that domain (e.g. anexample1.secure, otherexample2.secure, etc) are all automatically protected.

The includeSubDomains Directive

The includeSubDomains directive enables a site to declare that the rule applies to subdomains of the current domain as well; a rule on a.example.com with includeSubDomains set will also apply to b.a.example.com and c.a.example.com, for instance. This directive is important because design flaws in HTTP cookies could lead to cookie injection and/or session fixation attacks (Bulletproof SSL & TLS explains at length) if includeSubDomains is not set.

The crux of the threat is that browsers can’t indicate where cookies originated when they are sent back to the server, and a child domain may set a cookie on its parent. So, even if you’re interacting with https://legit.example.com, an active network attacker could create a fake http://attacker.legit.example.com and use it to set a cookie that would be sent to the secure site. The secure site would not have a trivial way to detect the phony cookie.

By setting the includeSubDomains directive, a site can protect itself from an attacker-created insecure child, because any child domain would be required to present a valid certificate.

Aside: One proposal I liked for combatting the security limitations of cookies was nicknamed MagicNamedCookies—the idea was that we’d create some new rules for cookies based on their name-prefix. For instance, any cookie named starting with $$SEC- could only be set or received by a HTTPS site. Similarly, a $$SO- prefixed cookie would only ever be set in a same-origin/first-party context. Unfortunately, to my knowledge, no browser has plans to implement these modest and backwards-compatible improvements.

The Parent Problem

Unfortunately, even with the includeSubDomains directive, the unavailability of an includeParent directive means that an active man-in-the-middle attacker can perform a cookie-injection attack against an otherwise HSTS-protected victim domain. Consider the following scenario: The user visits https://sub.example.com and gets a HSTS policy with includeSubDomains set. All subsequent navigations to sub.example.com and its subdomains will be secure.

The attacker causes the victim's browser to navigate to http://example.com. Because the HSTS policy applies only to sub.example.com and its superdomain matches, this insecure navigation is not blocked by the user agent. The attacker intercepts this insecure request and returns a response that sets a cookie on the entire domain tree using a Set-Cookie header. All subsequent requests to https://sub.example.com carry the injected cookie, despite the use of HSTS.

Best Practice To mitigate this attack, HSTS-protected websites should perform a background fetch of a resource at the first-level domain. This resource should carry a HSTS header with an includeSubDomains directive that will apply to the entire domain and all subdomains.

Surveying for HSTS

In Fiddler, you can easily add a temporary column to show the server’s Strict-Transport-Security response header, if any. In the black QuickExec box below the Web Sessions list, type:

cols add HSTS @response.Strict-Transport-Security

… and hit Enter. The HSTS column will appear and begin showing the response header’s value.

To permanently add the HSTS column to Fiddler’s UI, right-click the column headers, choose Customize Columns, and add the column like so:

In Chrome, you can see more about the HSTS list by visiting chrome://net-internals/#hsts

Why click-throughs are so dangerous

When the IE team replaced the IE6-era “traffic light” HTTPS error prompt:

…with the new blocking full-page errors, we heard a lot of negative feedback, especially from technically-savvy users. For instance, many users complained that the new pages don’t offer an easy way to see the certificate offered by the site, not recognizing that the site’s certificate could be entirely full of lies, deceptions that even technical users are ill-qualified to recognize.

Some users argued that throwing a full-page block for a expired certificate was too much—surely a recently expired certificate wasn’t really unsafe, right? A few of those complainers were experts with advanced understanding of cryptography, but a poor understanding of how the Certificate Ecosystem actually works. Certificates have expiration dates, in part, because the Certificate Authorities that issue them do not want the burden of maintaining revocation information for compromised certificates indefinitely-- the same reason credit cards originally included an expiration date—keeping track of and distributing the list of stolen cards is expensive. When a compromised certificate expires, the CA is no longer under any obligation to warn clients that the certificate had been revoked. The experts in question were appalled to learn that their “expert understanding” of the technology had actually put them at greater risk of having their data compromised.

Other enthusiasts argued: “Well, okay, you’ve convinced me that a site with a bad certificate shouldn’t be trusted. But I enjoy watching Shark Week and I like looking at scary things. Why not just let me go through to the site but warn me if I try to enter any data in it?”

There are two serious problems with this. The first is cache poisoning. If an active man-in-the-middle is attacking your connection to PayPal.com and you click through the blocking page “just to look”, the attacker can send back malicious JavaScript and HTML files with expiration dates that are years in the future. The next time you visit PayPal, even on a trusted, uncompromised connection with the shiny green badge in the address bar, the bad guy’s code is still running in your browser:

The second problem is DOM hijacking. Most browsers do not store a “Bad certificate” flag on the Origin, which means that a browser window with a bogus version of the site (where you clicked through a certificate error) can freely use script to manipulate a browser window that loaded the site over a secure, trusted connection. Stated another way, if any of your connections to a site are compromised, in effect they all are.

Common Questions

Q: So, if click-through is so dangerous, why is it allowed for most errors?

A: Because legitimate sites often deploy HTTPS incorrectly, and if browsers blocked access to them, the user would find another browser. This isn’t a theory, it’s a demonstrated fact.

Q: Prior to the introduction of HSTS, does IE ever prevent override of certificate errors?

A: Yes. There are a few cases in which the user cannot override a certificate error:

1. If the certificate is revoked
2. If the certificate is deemed insecure (e.g. contains a 512-bit RSA key)
3. If the page is in a “pinned site” instance
4. If group policy is set to Prevent Ignoring Certificate Errors

HSTS Accidental Denial of Service

One risk with HSTS’ no override behavior is if there’s a configuration problem on the client that turns what would be an annoyance into a work-stopping issue. For instance, I recently sent my Surface2 into Microsoft for servicing, and when I got it back, unbeknownst to me, the system clock was set back a year. I was unable to apply any Windows Updates and it wasn’t until I tried to visit a HTTPS site that I recognized the problem with the clock.

Unfortunately, Internet Explorer 11’s and Chrome 37’s error pages only help you recognize the incorrect system date problem if you know what you’re looking for.

Internet Explorer 11

Chrome 37 Certificate Expired Page

Chrome plans to address this in a future update.

Firefox 32 does a little bit better: it at least shows the system’s reported “Current time” to allow a user to potentially say: “Wait, that’s not right!”

Firefox 32

I’ve filed a bug against IE to suggest that improving this error page should be a priority to help raise websites’ confidence when deploying HSTS.

Beyond the incorrect system date problem, another problem arises with captive portals, like those seen when you connect to WiFi at places like Starbucks—these locations intercept all traffic until you take some action like checking a “I agree with your terms of use” box on a page that the portal serves. Because the portal doesn’t have a valid certificate for the sites it is intercepting, HSTS will prevent loading of the “terms of use” page. Fortunately, most of these problems are easily resolved by simply visiting a HTTP page, and most modern operating systems offer a “captive portal detection” feature that will load an insecure page automatically when joining the network.

Implementation Concerns

For browser implementers, there are a number of interesting challenges when implementing HSTS. For instance, there’s a privacy concern that a local “attacker” could examine the browser’s data store for HSTS rules and thus determine whether the browser had ever visited a given site that uses HSTS but was not in the “pre-load” list. Similarly, a remote website could (by using a carefully constructed set of "probing" subdomains) create a token/identifier/"cookie" that could be probed to identify a user. Clearing the list of sites when the browser history is cleared, and partitioning when the browser is InPrivate mode mitigates this privacy risk but limits the effectiveness of the HSTS feature.

Unfortunately, the authors of the HSTS specification declined to require safe handling of Mixed Content (insecure resources pulled into a secure page) meaning that HSTS-implementers are free to allow Mixed Content into protected pages, or to allow users the ability to override mixed content blocking. This is a significant shortcoming in HSTS.

Implementers must also make a number of design decisions in their implementation; for instance, if a DOM includes an insecure reference to a site using HSTS, should that DOM reference itself be changed, or should the secure version of the URL silently be used? How should the “virtual redirection” be handled by Resource Timing and other specifications? Incorrect handling of these decisions could lead to probing attacks.

For Internet Explorer in particular, there's a question about the scope of the feature. Windows has several major HTTP stacks (WinINET, WinHTTP, and System.NET are the most common) and because IE extensions do not rely upon IE itself for networking, they could unknowingly circumvent HSTS protections unless HSTS state is shared across all of the relevant HTTP stacks.

Further Reading

If you’re interested in HSTS, you’ll also be interested in three other specifications:

• Public Key Pinning – PKP allows a site to limit which certificates are deemed acceptable for HTTPS connections.
• Content Security Policy – CSP allows a site to limit content retrieval to specific origins.
• Mixed Content Handling - Proposal for how User-Agents should handle Mixed Content.

-Eric Lawrence
MVP, Internet Explorer

And the answer, as befitting an IEInternals post, is surprisingly complicated. 

The simplistic answer is that WinINET.h defines INTERNET_MAX_URL_LENGTH as 2083 characters, and this limit remains in force in a number of places. However, the true limit can be both longer and shorter than this.

CURI Removes Limits

In Internet Explorer 7, the IE team began the process of replacing legacy string-based URL handling code with CURI, a new object that manages URLs and exposes functionality via the IUri interface. This object internally has no formal limit on URL length, allowing URIs up to the effective size limit of COM strings (at least several hundred megabytes in a 32bit process).

However, CURI was not plumbed everywhere throughout Internet Explorer, the network stack, and Windows at large, and the legacy 2083 character limit remained in practical effect until IE8. If a larger URL reached CURI but later needed to flow through a legacy codepath, the URL would either be truncated or the codepath would exit with an error.

In IE8, CURI was used more broadly, and HTTP and HTTPS URLs could exceed the old limit, but only if the traffic was not sent through a proxy server. The code that prepared requests for a proxy was only fixed later in IE9. However, even today there are code paths that will not handle URLs that exceed the legacy limit; for instance, the WinINET cache will not store and retrieve resources that were retrieved from a URL longer than the legacy limit.

Beyond the local limit, the network path itself may effectively enforce a limit; some servers will reject requests whose headers are longer than 4k or 16k, for instance, and because the URL is contained in the header, requests of huge URLs may fail.

Data URIs

Internet Explorer 8 introduced support for DataURIs, a method of encoding a resource directly within a URI using base64 encoding. For performance reasons, IE8 supported DataURIs up to 32kb in length and this longer limit required changes elsewhere. For example, some DOM attributes at the time were limited to 4096 characters, a factor that would have hampered use of DataURIs in the HREF attributes of IMG elements.

In IE9, the 32kb limit for DataURIs was removed, although for security reasons their use remains limited to certain contexts (specifically, scenarios that create a security context, like IFRAMES, are forbidden). You still should avoid using huge DataURIs in most cases, however, as performance of large DataURIs often lags other approaches.

Mailto URIs

Mailto is one of the most interesting protocols in the browser, as it’s implemented as both an Asynchronous Pluggable Protocol and an Application Protocol. This means that URLMon itself both implements the protocol (to make security decisions about it) and delegates its handling out to the system (to invoke a mail client).

Why does URLMon have an Asynchronous Pluggable Protocol implementation of mailto, instead of just treating it like any other Application Protocol? Because in the 1990s, it wasn’t uncommon to have <FORM ACTION="mailto:joe@example.com">. When the web form was submitted, the browser would serialize the data into the body of an email message and submit the form data to the specified email address using the visitor’s email client. This process goes down Internet Explorer’s navigation codepath and uses the URLMon APP workflow.

This use of the mailto syntax is now very uncommon, although it is still supported in Internet Explorer, Firefox, and Chrome. Because it necessarily reveals the sender’s email address, IE will show the following warning:

This arcana is relevant in this post only for one reason: Asynchronous Pluggable Protocols implement the IInternetProtocolInfo interface to allow the browser to query the protocol for information about URLs that utilize this protocol. The browser calls the ParseUrl to ask for a “security URL” that is equivalent to the target URL, and the simplistic mailto protocol implementation simply copies the entire mailto URL to the target buffer. This causes a failure because the GetSecurityId method does not accept Security IDs longer than MAX_SIZE_SECURITY_ID, 512 bytes, and as a consequence the attempt to launch longer mailto URLs will fail. Unfortunately, this failure doesn’t result in a meaningful error message; the browser isn’t robust against failures at this point and thus navigating to such URLs may result in an endlessly-spinning donut, a blank tab, or a “Page cannot be displayed” error.

There are some codepaths that manage to avoid the GetSecurityId problem, but not reliably (e.g. depending on the user’s IE version and Zone configuration) and thus it is recommended that you avoid mailto URLs longer than 512 bytes.

JavaScript URIs

The JavaScript protocol is used for bookmarklets (aka favlets), a lightweight form of extensibility that permits a user to click a button and run some stored JavaScript on the currently loaded page. In IE9, the team did some work to relax the length limit (from ~260 characters, if I recall correctly) to something significantly larger (~5kb, if I recall correctly).

However, within the browser itself, the legacy 2083 character limit remains in force when using JavaScript URIs as the source of frames or IFRAMEs (demo). While I’m not convinced that relaxing the limit is warranted (such URIs are of limited usefulness) I have filed a bug suggesting that the IE Developer tools should warn if overlong script URLs are encountered.

Application Protocols

Application Protocols are subject to even weirder limits. On Windows 8.1, for instance, if you directly pass a 4k Application Protocol URL to ShellExecute in native code, the URL is silently truncated at the 2083 character mark.

In Internet Explorer 11, attempting to launch an Application Protocol URL longer than 507 characters fails with an exception:

In Chrome, if you attempt to an Application Protocol URL over 2046 characters, you’ll see the regular Security Prompt, but clicking its “Launch Application” button does nothing.

UI Limits

Beyond the underlying limitations, the User Interface also enforces various limits around URL length. For instance, the browser address bar is capped at 2047 characters. Start > Run is limited to 259 characters (the MAX_PATH limit of 260, which leaves one character for the null-terminator). Internet Explorer versions up to version 11 do not allow you to bookmark URLs longer than 260 characters. The Windows RichEdit v2 control's Automatic Hyperlink behavior (EM_AUTOURLDETECT) truncates the link after around 512 characters; this limit is higher in RichEdit5W controls.

-Eric