The User-Agent String: Use and Abuse

Article
10/07/2009

When I first joined the IE team five years ago, I became responsible for the User-Agent string. While I’ve owned significantly more “important” features over the years, on a byte-for-byte basis, few have proved as complicated as the “simple” UA string.

I (and others) have written a lot about the UA string over the years. This post largely assumes that you’re familiar with what the user-agent string is and what it’s commonly (mis)used for.

In this post, I’ll try to summarize why the UA string causes so many problems (beyond browser version sniffing), and expose the complex tradeoff between compatibility and extensibility.

Background

First things first-- you can check the UA string currently sent by your browser using my User-Agent string test page.

Do you see anything in there that you weren’t expecting?

Changing the User-Agent String at Runtime

For IE8, we fixed significant bugs in the UrlMkSetSessionOption API, which allows setting of the User-Agent for the current process. Before IE8, calling this API inside IE would (depending on timing) set the User-Agent sent to the server by WinINET, or set the User-Agent property in the DOM, but never properly set both.

I developed a simple User-Agent Picker Add-on for IE8 that allows you to change your User-Agent string to whatever you like. You can then easily see how websites react to various UA strings. For instance, try sending the GoogleBot UA string to MSDN to see how that site is optimized for search.

Internally, the add-on simply exercises the URLMon API:

UrlMkSetSessionOption(URLMON_OPTION_USERAGENT, szNewUA, strlen(szNewUA), 0)

Alternatively, Web Browser Control hosts can change the User-Agent string sent by hyperlink navigations by overriding the OnAmbientProperty method for DISPID_AMBIENT_USERAGENT. However, the overridden property is not used when programmatically calling the Navigate method, and it will not impact the userAgent property of the DOM's navigator or clientInformation objects.

Extending the User-Agent String in the Registry

It’s trivial to add tokens to the User-Agent string using simple registry modifications. Tokens added to the registry keys are sent by all requests from Internet Explorer and other hosts of the Web Browser control. These registry keys have been supported since IE5, meaning that all currently supported IE versions will send these tokens.

Other browsers (Firefox, Chrome, etc) do not offer the same degree of ease in extending the UA string, so it’s uncommon for software to extend the UA string in non-IE browsers.

Update 3/23/2010: IEBlog announces that IE9 will no longer send registry tokens to the server.

The Fiasco

Unfortunately, the ease of extending IE’s UA string means that it’s a very common practice. That, in turn, leads to a number of major problems that impact normal folks who don’t even know what a UA string is.

A few of the problems include:

Many websites will return only error pages upon receiving a UA header over a fixed length (often 256 characters).
In IE7 and below, if the UA string grows to over 260 characters, the navigator.userAgent property is incorrectly computed.
Poorly designed UA-sniffing code may be confused and misinterpret tokens in the UA.
Poorly designed browser add-ons are known to misinterpret how the registry keys are used, and shove an entire UA string into one of the tokens, resulting in a “nested” UA string.
Because UA strings are sent for every HTTP request, they entail a significant performance cost. In degenerate cases, sending the UA string might consume 50% of the overall request bandwidth.

Two real-world examples:

My bank has problem #1. They have security software on their firewall looking for “suspicious” requests, and the developers assumed that they’d never see a UA over 256 bytes.

Some major sites are using super-liberal UA parsing code (problem #3) to detect mobile browsers. Unfortunately, for instance, Creative Labs adds the token “Creative AutoUpdate” to the UA string. Naive server code sees the characters pda inside that token and decides that the user must be on a mobile browser. The server might then return WML content that the desktop browser will not even render, or provide an otherwise degraded experience. Worse still, some sites don’t send a Vary: User-Agent response header when returning the mobile content, meaning that network proxies will sometimes start sending everyone content designed for mobile devices.

Ultimately, the problem is what economists call the Tragedy of the Commons, although personally I prefer the visual representation. You might remember that the extensibility of the Accept header leads to the same problem, although that header is sent so unreliably that no sane website would depend upon it.

Standards

It’s tempting to look to the standards for restrictions on the UA string. Unfortunately, the RFC for HTTP has little to say on the topic:

14.43 User-Agent

The User-Agent request-header field contains information about the user agent originating the request. This is for statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations. User agents SHOULD include this field with requests. The field can contain multiple product tokens (section 3.8) and comments identifying the agent and any subproducts which form a significant part of the user agent. By convention, the product tokens are listed in order of their significance for identifying the application.

User-Agent = "User-Agent" ":" 1*( product | comment )

Example:

User-Agent: CERN-LineMode/2.15 libwww/2.17b3

Notably, the RFC does not define a maximum length for the header value, and does not provide much guidance into what “subproducts which form a significant part of the user agent” means. It suggests a few broad uses of the UA string on the server-side, without discussion of what problems such usage might introduce.

Motivations for UA Modification

OEMs and ISVs have a number of motivations for adding to the UA string.

Metrics. Every server on the web can easily tell if your software is installed.
Client capability detection. JavaScript can easily detect if your (ActiveX control / Protocol Handler / Client application / etc) is available.
User Tracking. I don’t know of any current offenders, but at some point in the past some software would add a GUID token to the UA string. This token would effectively act as an invisible “super-cookie” that would be sent to every site the user ever visited.

Now, scenario #3 is clearly evil, and we have no desire to support it. Scenarios #1 and #2 aren’t inherently bad—but advertising to every site in the world that a given piece of software is available on the client is probably the wrong design.

Known UA Tokens

Here are some explanations of common tokens found in real-world IE UA strings.

Token	Meaning / Component
SV1	Security Version 1- Indicates that XP SP2 was installed. Removed from IE7.
SLCC1	Software Licensing Commerce Client- Indicates Vista+ AnyTime Upgrade component is available.
MS-RTC LM 8	Microsoft Real Time Conferencing Live Meeting version 8
InfoPath.2	InfoPath XML MIME Filter
GTB6	Google Toolbar
Creative AutoUpdate	Creative AutoUpdate software
Trident/4.0	IE8 version of HTML Renderer installed
Zune 3.0	Zune Software client
Media Center PC 6.0	It's a Media Center PC
Tablet PC 2.0	It's a TabletPC
.NET CLR 3.5.30729	The .NET Common Language Runtime
chromeframe	Google ChromeFrame addon
fdm	FreeDownloadManager.org add-on
Comcast Install 1.0	Comcast High-speed Internet installer
OfficeLiveConnector.1.3	Office Connector
OfficeLivePatch.0.0	??
WOW64	Running in 32bit IE on 64bit Windows
Win64; x64	Running in 64bit IE
msn OptimizedIE8	Installed with MSN branding and services
yie8	Installed with Yahoo! branding and services

Alternatives to UA Modification

In many cases, allowing client-side script to detect a capability without forcing the browser to send that information to the server would be sufficient. While new APIs might be proposed for this purpose, we need an alternative that already works in all versions of IE prior to Internet Explorer 10 Standards Mode.

You probably know that Conditional Comments can be used to detect the IE version, but they can also be used to detect custom information about any component listed in the registry’s version vector key. For instance, Windows 7 uses the new WindowsVersion entry to allow script to detect the OperatingSystemSKU.

To expose your capabilities via conditional comments, simply create a REG_SZ inside HKLM\SOFTWARE\Microsoft\Internet Explorer\Version Vector. The new entry should be named uniquely (e.g. EricLaw-SampleAddon) and contain a string in the format x.xxxx (e.g. 1.0002).

You can then detect the version (or absence) of your component using conditional comments:

These conditional comments are hidden from non-IE browsers, and will work properly in IE5 to IE9. IE10 Standards Mode removes support for Conditional Comments.

Conclusions?

Extensibility is an important aspect for any major software project, but can also be the source of severe compatibility problems that are extremely painful to fix in the future. As we increase the power of the web platform, we need to find ways to ensure that extension points and the tragedy of the commons don’t destroy the user’s experience.

Until next time,

-Eric