Understanding Web Proxy Configuration

Over the last decade, I’ve come to learn a lot about web proxies, having chosen to implement my web debugger as a proxy. In today’s post, I’ll provide an overview of proxy-related information, including information on changes in Internet Explorer 11 / Windows 8.1.

The “System Proxy”

Unlike on other systems like Mac OSX, Windows doesn’t really have the concept of a “system” proxy. Most applications respect the WinINET proxy setting, but a few do not.

• Firefox respects the WinINET setting only when configured to “Use System Proxy Settings” in its Network configuration.
• WinHTTP-based applications may or may not respect the WinINET setting (see below).
• Applications built on the .NET Framework typically will adopt the system proxy only when they start (and do not detect changes at runtime). Also, the proxy settings can be overridden by the app.exe.config or machine.config files.

Historically, most applications were built on WinINET and thus respected its proxy settings directly, but today more and more applications are built on WinHTTP (or a descendent architecture like BITS) and System.NET and thus may require additional configuration.

Settings are Per-User

WinINET proxy settings are typically per-user rather than per-machine. This means that individual (even non-admin) users can set their own proxy settings without impacting the proxy settings of other user-accounts. WinINET can be configured to apply proxy changes on a machine-wide basis by creating a registry DWORD named HKLM\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings\ProxySettingsPerUser with value 0. Subsequently, changes to the proxy settings can only be made by elevated applications running with Administrative permissions.

Configuring the Proxy

The WinINET proxy setting can be configured in Internet Explorer by clicking Tools > Internet Options > Connections. On the connections tab, select a Connection and click the Settings button. For most users, the proxy is configured by pressing the LAN Settings button at the bottom of the dialog:

Settings Precedence: Part 1: While the proxy settings for the Dialup/VPN/RAS connection and the LAN settings button can be configured independently, internally WinINET will use the settings from any active/connected dialup/VPN/RAS Connection. Otherwise, the proxy setting specified in LAN settings is used. Many users incorrectly expect WinINET to first determine which network interface is used to establish a given connection before selecting a proxy; it doesn’t work like this.

Proxy Configurations

When you configure your proxy settings, you’re presented with the following dialog box:

Settings Precedence: Part 2: The settings in this dialog box are presented in the order of their precedence. First, the Automatically detect settings option is consulted, next the automatic configuration script option is evaluated. If neither of those options is enabled or neither specifies a proxy to use, only then are the fixed proxy server settings consulted.

Automatically Detect Settings

When Automatically Detect Settings is enabled, Internet Explorer performs a process called Web Proxy Auto-Detection (WPAD). This process consists of two phases:

1. Issuing a DHCP INFORM query, asking DHCP to provide the URL of a proxy script
2. Searching for a server in the current domain prefixed with WPAD (e.g. wpad.corp.contoso.com)

These operations may or may not be performed in parallel, depending on the network stack. Some browsers (e.g. Firefox) only undertake DNS detection and skip DHCP detection. For performance reasons, some clients may cache the result of the detection on a per-network basis ("SmartWPAD"), skipping the WPAD process in future sessions when connected to networks known not to be using WPAD.

If the client is able to automatically detect a proxy script's URL, it will then attempt to download and use that proxy configuration script.

Automatic Configuration Script

The user may also directly specify the URL of a proxy configuration script using the second checkbox in the dialog. The URL field below points directly at the target script (e.g. https://proxy.contoso.com/proxy.pac).

Proxy configuration scripts, whether discovered via WPAD, or manually specified by the user, are JavaScript files that expose at least one function FindProxyForURL(url, host). This function is called by the browser each time that it needs to decide where to send a given request. It returns a string which is either:

1. "DIRECT" indicating that the request should bypass the proxy
2. "PROXY PrimaryProxy:8080; BackupProxy:81" indicating that the request should be forwarded to the proxy PrimaryProxy on port 8080, unless that server is unreachable in which case the request should be sent to BackupProxy on port 81.

WinINET also offers support for another function FindProxyForURLEx, an extension which supports IPv6-aware proxy scripts.

NOTE: Proxy Scripts Impact the Zone

Warning: One sometimes surprising aspect of proxy scripts is that they impact the Internet Explorer Security Zone determination. To summarize my long MSDN article and blog post, by default, if a proxy script is in use and returns DIRECT, the target site will be mapped to the Local Intranet Zone.

 

NOTE: File://-based Proxy Scripts Deprecated

In IE11, the WinINET team has made some changes to attempt to improve interoperability of proxy scripts.

Internet Explorer 10 and other WinINET-based clients would use the specified proxy script. However, WindowsUpdate and any other application built on a WinHTTP-based network stack would silently ignore the specified proxy script because WinHTTP has never supported the use of the file:// protocol as the origin of a proxy configuration script. Note: In November 2012, the .NET Framework was changed to use WinHTTP for PAC processing, which means that, by default, .NET Applications will not support file://-based proxy scripts any longer either.

In Internet Explorer 11, the WinINET team has disabled WinINET’s support for file:// based scripts to promote interoperability across network stacks. Corporations are advised to instead host their proxy configuration scripts on a HTTP or HTTPS server. As a temporary workaround, this block can be removed by setting the following registry key:

Key: HKLM\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings\
Value: EnableLegacyAutoProxyFeatures
Type: REG_DWORD
Data: 1

Keep in mind that this should only be a temporary measure, as this block was introduced for good reasons, and removing the block won’t magically fix your WinHTTP-based applications.

 

NOTE: File://-based Proxy Scripts Require “Unhealthy” Syntax

While you shouldn’t use a file://-based proxy script, if you do so, IE requires that you use the legacy “unhealthy syntax” for file URIs in the proxy configuration dialog.

As you can see, the “Healthy” syntax is incorrectly parsed as a relative URL.

The legacy “unhealthy” syntax works properly:

But, to reiterate, don’t use file:// to host your proxy script.

NOTE: IE11 Deprecates Non-Silent Auth when downloading Proxy Script

When downloading a proxy configuration script, it’s possible that the server will demand credentials from the client application. A problem arises if the server requests credentials using HTTP BASIC or HTTP DIGEST authentication, as these authentication methods require that the user respond to a credential prompt dialog box. (In contrast, NTLM and Negotiate authenticate silently).

Many applications cannot handle showing prompts as a part of this workflow and will fail silently. To promote interoperability, IE11 also blocks the use of Proxy Configuration scripts that require authentication. As a temporary workaround, this block can be removed by setting the following registry key:

Key: HKLM\SOFTWARE\Policies\Microsoft\Windows\CurrentVersion\Internet Settings\
Value: EnableLegacyAutoProxyFeatures
Type: REG_DWORD
Data: 1

Keep in mind that this should only be a temporary measure, as this block was introduced for good reasons, and removing the block won’t magically fix applications which cannot show UI prompts.

Generally speaking, you should attempt to migrate away from BASIC/DIGEST authentication both for access to the proxy script and for access to use the proxy itself. As I’ve noted previously, Manual Proxy-Authentication breaks many applications.

 

Fixed Proxy Configuration

Beyond automatic proxy detection, the user may manually configure “fixed” proxies. By default, the simple UI is shown:

…but clicking the Advanced button shows more options:

 

You can either specify a different proxy server/port for each protocol, or you may use the default option which applies the same proxy server to all protocols.

The Exceptions box allows you to specify what hostnames are configured to bypass the proxy. Beyond hostnames, it supports two special tokens:

• <local> – Matches “plain” hostnames (with no embedded dots)
• <-loopback> – Unmatches localhost, 127.0.0.1 and other “loopback” addresses.
  • WinINET defaults to automatically bypassing the proxy for loopback addresses; this token disables that behavior.
  • This token impacts WinINET and current versions of WinHTTP; the .NET Framework simply ignores it.

Note that Internet Explorer only supports SOCKSv4.

 

Setting the Proxy Programmatically 

Fiddler and some other applications attempt to programmatically adjust the WinINET proxy setting. Rather than attempting to “poke” the registry directly, the proper way to update the proxy setting is to use the InternetSetOption API.

Using the API is straightforward, whether you’re using C/C++ or .NET.

IE10+ Regression on System Shutdown

As a part of Internet Explorer 10, proxy configuration was centralized in an existing system service (still labeled “WinHTTP Web Proxy Auto-Discovery Service”) so that proxy configuration code no longer needs to run in every process using WinINET (e.g. each IE tab is in its own process).

Unfortunately, as a part of this update, a regression was inadvertently introduced. Code that attempts to change the WinINET proxy setting during a system shutdown (like Fiddler’s shutdown code, for instance) will find that the APIs return success, but upon restart you will see that the proxy settings change was discarded. No fix for this issue (also present in IE11) is yet available. A possible workaround for this problem would be to create an out-of-proc COM server and run a method (DllHostKeepAlive) that doesn’t return until the client calls a shutdown method (CancelDllHostKeepAlive). This would open a DLLHost that can be used to adjust the proxy settings during shutdown.

CONNECT Bug: https://connect.microsoft.com/IE/feedback/details/838086/internet-explorer-10-11-wininet-api-drops-proxy-change-events-during-system-shutdown

WinHTTP Considerations

Thanks to Eric Loewenthal, owner of WinHTTP's proxy code, for providing the expertise for this section

The WinHTTP stack was designed for use in services and other contexts where WinINET would be inappropriate (e.g. because WinINET may show UI or not scale well for unattended use). WinHTTP offers a limited subset of proxy support-- generally speaking, the calling code is expected to set the proxy configuration settings directly on the session or request objects. The user may attempt to configure the "default" proxy for WinHTTP using the proxycfg.exe (XP) or netsh.exe (Vista+) tools, but these tools only support fixed proxy settings (not autodetection or PAC script URLs) and WinHTTP-based applications may or may not use those settings.

The configuration tools were primarily intended or WinHTTP apps that are running as System (without a user to impersonate). 

Well-written WinHTTP apps will:

1. If not running in a user's process, impersonate a user
2. Call WinHttpGetIEProxyConfigForCurrentUser
3. Plug the returned settings into WinHttpGetProxyForUrl

Sample code for this technique can be downloaded from MSDN. 

Note: WinHTTP will not process a proxy script if the Content-Type response header is not "application/x-ns-proxy-autoconfig" or the URL's file extension is not one of .js, .pac, or .dat. Also, remember that WinHTTP will not use proxy script files from file:// URLs, on any version of Windows.

 

There’s still quite a bit more to say about proxies. I’ll update this post as time permits and questions arise.

thanks!

-Eric

Comments

• Anonymous
  October 14, 2013
  How do I inspect a proxy script obtained via WPAD?

• Anonymous
  October 14, 2013
  Proxy scripts are simply text files that can be examined in any text editor. If you're looking for an actual PAC-script debugger, check out chentiangemalc.wordpress.com/.../c-proxy-pacdbg-pac-debugging.

• Anonymous
  November 01, 2013
  Hey Eric, Great article - we came across this as we are facing a problem due to this change and we have a valid reason for wanting to have a file based proxy. Our product uses a bunch of URLs that go to our data center and integrates with a bunch of URLs that belong to the clients of our product. Our URLs have our proxy rules and client URLs have their own rules -- we make both work by downloading both scripts and merging them to a combined script file on the desktop. We then call InternetSetOption and point it to the local file... It seems this is no longer working

• Anonymous
  November 05, 2013
  I am behind a proxy server providing free internet via my town hall. Since upgrading to windows 8.1 and explorer 11 I receive error 8024401C when I try to connect to windows update. I believe this is a proxy server problem is there anything you can suggest to help? thanks EricLaw - What are your proxy settings? Did you download and run the WindowsUpdate Troubleshooter?

• Anonymous
  November 17, 2013
  It looks like IE10 and above doesn't look at dialup/VPN/RAS proxy setting any more, it seems LAN setting always override that. The testing shows this behaviour. Could you confirm if this is expected? [EricLaw] The WinINET team indicates that this issue was addressed by http://www.microsoft.com/en-us/download/details.aspx?id=40532. 

• Anonymous
  November 20, 2013
  The comment has been removed

• Anonymous
  December 08, 2013
  The comment has been removed

• Anonymous
  December 23, 2013
  Hi Eric, great article!  I'm having issues with IE11 on WIndows 8.1 with our TMG 2010 wpad.  It works fine on our Windows 7 SOE and with Chrome on 8.1, but IE11 doesn't seem to even look for it.  I've posted at social.technet.microsoft.com/.../windows-81-with-ie-1102-wpad-issue   - any ideas appreciated : ) Thanks, Simon

• Anonymous
  January 02, 2014
  Regarding this IE 10+ regression on system shutdown, do you know if this is applicable to any of the other events such as sleep, restart or logout? [EricLaw] It's unlikely that sleep is impacted. Restart and Logout might be; you could simply try it and then you'd know for sure.

• Anonymous
  January 17, 2014
  Thank you so much for your article. About the change in WinINET in IE11, I have one question. If I need access at our http external site for read the .pac (now are using file://) how work in my internal LAN? My http connection are via Proxy defined in the .pac I'm using the .pac file for set our internal corporate squid proxy or don't use proxy (by example in my house) [EricLaw]: The common configuration for this is to set the PAC file URL to a HTTP:// URL that is only available within your corporate network (e.g. http://proxy/pac.js). That way, the URL is inaccessible from outside your network, so that when you take your device home, the script is not used. Alternatively, set up your internal network to use WPAD and don't manually configure a PAC URL on the client directly at all.

• Anonymous
  January 22, 2014
  Hello Eric, I work on any enterprise which uses a third partner remote access solution, this third partner solution only support internet proxy hosted locally at %appdata% folder (using file://C:...) While the windows computers are connected remotely, the internet browsing through I.E. works fine, but every time the user try to run a Click Once application, the Click Once fails to read the script pac and try to connects directly. If we manually change the I.E. proxy settings for the same script.pac hosted at any webserver using Http://...) the Click Once application works fine. This happens with I.E 8 and later, do you know if this behavior is expected?

• Anonymous
  January 22, 2014
  Hi, @Diego! You say that IE "works fine" but keep in mind that, as this post describes, that will no longer be true when IE11 is installed. There was a security fix for .NET in November 2012 that changed System.NET to rely on WinHTTP for WPAD support (which in turn means that file://-sourced WPAD scripts will fail) unless the LegacyWPADSupport key is set. See http://technet.microsoft.com/en-us/security/bulletin/MS12-074: What does the update do? 
  The update addresses the vulnerability by updating Microsoft.Jscript within System.Net and restricting how the .NET Framework executes JavaScript contained within a WPAD configuration file. The update also changes the way WPAD is handled by using WinHTTP instead of Microsoft.Jscript. This change, while unlikely, may cause some compatibility issues for environments that have non-standard proxy auto-configuration environments. For more information about this change, see the More Information section in Microsoft Knowledge Base Article 2745030. That KB specifically notes: When you set the LegacyWPADSupport DWORD value to 1, the new proxy detection logic that is installed by this update is disabled. This value should be set only if you must support one or more of the following scenarios in which WPAD is used for proxy detection in the network: Lookup of proxies for FTP uniform resource identifiers (URIs) is needed (Windows XP).Lookup of proxies for URIs that have a scheme other than HTTP, HTTPS, or FTP is needed (Windows XP, Windows Vista, Windows 7, or Windows 8).The proxy auto-configuration (.pac) file is located on an FTP server or a file share instead of an HTTP server (Windows XP, Windows Vista, Windows 7, or Windows 8).     

• Anonymous
  January 24, 2014
  Hi Eric, Yes, IE 11 needs the workaround you posted, but we are running IE 10 yet. The .NET KB solved my problem for now.Do you know if MS will remove the I.E. and .NET workaround on later patches? What should I follow to keep updated about this? -Thanks, Diego [EricLaw] Unfortunately, there's no way to know whether the "escape hatch" registry key will be kept around indefinitely; sometimes they're removed in a later version and sometimes they linger on for a decade or more. It would probably be wise to push the vendor to explore updating their code to be compatible with future products.

• Anonymous
  February 05, 2014
  Eric, Great article and very helpful.  Wondering if you could help me out with something related to the PAC file. We are use a corporate proxy server and configured IE to point to a URL and Port (e.g. Use automatic configuration script.  Address = http://ur:port/).  Our Network Engineer is troubleshooting a problem and asked if I knew where the Proxy PAC file is stored in Windows. My initial assumption was that the PAC file is downloaded to the Temp I-Net files cache, but I have not been able to locate the file. I have done quite a bit of research and there are several references (Wikis, Proxy and PAC build articles) stating the PAC file is downloaded and process by the browser (e.g. support.microsoft.com/.../271361), but I cannot find any documentation stating where the PAC file is actually downloaded to. Do you know if IE (or any other browsers for that matter) downloads the PAC file to the local machine?  If so, do you know is it located? OS: Win7 64bit / IE9 or WinXP / IE8 [EricLaw] Your best bet is to watch file writes using SysInternals' Process Monitor, because the behavior will vary by OS and by the browser version installed. On Windows 8.1 with IE11, the "WinHTTP Web Proxy Auto-Discovery Service" running inside a SVCHost.exe instance will download and cache the PAC file to a rudimentary cache: the file is written to a randomly-named *.cache file inside the c:WindowsServiceProfilesLocalServicewinhttp folder. In IE8 on Windows XP, I see the PAC file getting cached to %USERPROFILE%Local SettingsTemporary Internet FilesContent.IE5randomproxy[2].pac. 

• Anonymous
  February 13, 2014
  The comment has been removed

• Anonymous
  February 18, 2014
  Eric, Thanks for your response. I tried workaround which is to set registry key "EnableLegacyAutoProxyFeatures" to 1. But this requires restart of IE which may not be acceptable for our customers. Is there a way to avoid IE restart ? I mean any other way to programmatically inform IE that value of "EnableLegacyAutoProxyFeatures" has changed ? Thanks, John [EricLaw] John, the registry key was doubtless intended for use only by organizations themselves, not ISVs building products that rely on the poorly-supported file scheme. I strongly urge you to update your product not to rely on file:// URIs for PAC files. To your specific question: if the existing INTERNET_OPTION_PROXY_SETTINGS_CHANGED flag passed when programmatically changing the proxy doesn't refresh from the EnableLegacyAutoProxyFeatures key, that strongly suggests that there's no mechanism to do so and the application must restart.

• Anonymous
  February 25, 2014
  hi Eric, thanks for this great blog article. Maybe you can help me out with one specific problem I can't figure out, because I guess I do not fully understand the whole WinINET / WinHTTP topic: We are using a automatic config script (PAC) in Internet Explorer to set proxy settings. The first checkbox (Automatically detect settings) is not used. Now I was asked to deactivate the "WinHTTP Web Proxy Auto-Discovery Service" on our Win 7 clients for security reasons. And, well... I do not have the slightest idea if this would impact any used application (reading your IE10 update part, even IE11 itself maybe) on our clients, and if it really is a good idea to do so. We currently do not set any WinHTTP proxy settings using netsh if this is important. We only use GPO to set the proxy settings in IE. So my questions are: Is it save to disable this service if WPAD is not used? If not, why, or can we do something about it (like configuring WinHTTP proxy using netsh, then disabling the service)? What would be our options if disabling the service is not a good idea at all? The main concern of our security department are WPAD spoofing attacks... regards, Dominik

• Anonymous
  February 27, 2014
  Hi Eric, I came across this great article when troubleshooting proxy behavior after upgrading my laptop from Windows7 to Windows8. I'm a consultant who works most of the time at customer premises. One of my customers provides pac URL for proxy configuration. Everything works fine except for Terminal Services Gateway and Outlook Anywhere. If I use proxy server URL and port (the same as given by pac file) both services work fine. This is strange that the pac file fails to allow TS Gateway and OA only on Windows 8 PC (with IE10), while the same pac file allows TS Gateway and OA on windows 7 boxes. So the question is, are there any differences between Windows 7 and 8 in regard to pac file processing for Terminal Services Gateway and Outlook 2013 connecting via Outlook Anywhere? Thanks, Marcin

• Anonymous
  February 27, 2014
  Forgot to add one more detail. It seems when pac is configured outlook is unable to complete autodiscover - it doesn't resolve autodiscover.domain.com/autodiscover/autodiscover.xml, while when proxy is configured it can do that without any issues - results taken from the Test Email Autoconfiguration outlook feature.

• Anonymous
  March 26, 2014
  There is another use case however: some people use these proxy files to block ads on their HOME computers. They have no web server to move the proxy file to. I hope that Microsoft comes up with a long term solution for them. [EricLaw] It's trivial to have FiddlerCore or any other software host a local webserver to serve this file, but even then there are better options than blocking traffic using a fake proxy script; for instance, IE's TPL and Zones features, and/or the Windows HOSTS file. Please see http://blogs.msdn.com/b/ie/archive/2010/11/30/selectively-filtering-content-in-web-browsers.aspx

• Anonymous
  March 27, 2014
  As I understand, Lync and Outlook uses both WinINet and WinHTTP for different purpose. When using Office 365 (Outlook Online and Lync Online) in a federated domain environment (ADFS), it's  (as I have been told) very important that all connections to the Online service comes from one single endpoint (read one proxy server, not proxy-server and FW, or two different proxy servers) for a single user session. Otherwise , the "SAML" ticket are not valid. This makes it very important to have WinHTTP configured in sync with WinINet. How can we make that happen automatically??? Also, As I read earlier in this article: Well-written WinHTTP apps will:

1. If not running in a user's process, impersonate a user
2. Call WinHttpGetIEProxyConfigForCurrentUser
3. Plug the returned settings into WinHttpGetProxyForUrl Is Outlook and Lync one of those well-written WinHTTP apps?

• Anonymous
  June 15, 2014
  We are observing a reversal of precendence with IE11 on Win7 64bit compared to using IE8 with Win XP Pro SP3.  The context is an enterprise network with Win 2008 R2 servers, ADS, and proxy settings controlled via a proxy.pac using http and also local proxy setting preset for the case should the proxy.pac webserver be unreachable. The default routing goes to the Internet gateway but where there are other partner organisations which are reached by their own gateways and proxies either directly via the proxy pac or which are listed in the exceptions to the local LAN proxy server.  With IE8 the precedence was proxy.pac, local exceptions, local proxy server.  However we are now seeing the precedence local proxy server, proxy pac and the local exceptions appear to be being ignored as soon as we enter an http proxy as a local proxy server, at least from within one application.  Normally the proxy settings are controlled by group policies, but having removed a test PC from the group policies this behaviour became evident on setting the http proxy and exceptions manually..  Any suggestions as to what might be the cause and therefore how we can reestablish the needed order of precedence?

• Anonymous
  July 31, 2014
  The comment has been removed

• Anonymous
  November 17, 2014
  The comment has been removed

• Anonymous
  December 17, 2014
  The comment has been removed

• Anonymous
  December 18, 2014
  Great information Eric, My problem is that I have a working proxy.pac file, but when I put the path http://servername/pac/proxy.pac into my Group Policy (just like I used to when file was allowed) I get a popup on my clients to open or save the file.  But if I go to the client and type that in the LAN settings under Use Auto Config Script it works fine.  So I am trying to figure out how to deploy it through my group policy rather than manually touching all my computers.  Thanks!

• Anonymous
  January 29, 2015
  Robert, you need to set your web server with the proper mime type (application/x-ns-proxy-autoconfig) associated with the .pac file.

• Anonymous
  February 12, 2015
  The comment has been removed