Matt’s IIS Support Tidbits https://docs.microsoft.com/archive/blogs/mist/feed.xml Tue, 16 Apr 2019 06:30:10 GMT en-US hourly 1 Windows Authentication HTTP Request Flow in IIS https://docs.microsoft.com/archive/blogs/mist/windows-authentication-http-request-flow-in-iis Wed, 14 Feb 2018 10:23:08 GMT http://blogs.technet.microsoft.com/mist/?p=45 Foundation IIS is a user mode application. It sits on top of HTTP.sys, which is the kernel mode driver in the Windows network stack that receives HTTP requests. IIS picks up requests from http.sys, processes them, and calls http.sys to send the response. IIS, with the release of version 7.0 (Vista/Server 2008), introduced Kernel Mode authentication for Windows Auth (Kerberos & NTLM), and it's enabled by default on all versions. This feature offloads the NTLM and Kerberos authentication work to http.sys. Http.sys, before the request gets sent to IIS, works with the Local Security Authority (LSA, lsass.exe) to authenticate the end user. IIS just receives the result of the auth attempt, and takes appropriate action based on that result. Before diving into both Kerberos and NTLM request/response flows, it's worth noting that the vast majority of HTTP clients (browsers, apps, etc.) don't send any credentials on their first request for a resource.  This means that first request is anonymous, even if credentials have been configured for that resource. This anonymous request, when Windows Auth is enabled and Anonymous Auth is disabled in IIS, results in an HTTP 401 status, which shows up as "401 2 5" in the normal IIS logs.  Both request flows below will demonstrate this with a browser, and show that it is normal. The NTLM and Kerberos exchanges occur via strings encoded into HTTP headers.

Kerberos

Request & Response #1

This is the initial anonymous request by the browser: GET / HTTP/1.1 Accept: text/html, application/xhtml+xml, image/jxr, */* Accept-Encoding: gzip, deflate, peerdist Accept-Language: en-US, en; q=0.5 Connection: Keep-Alive Host: server User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Edge/16.16299 I've configured Windows Authentication to only use the "Negotiate" provider, so these are the headers we get back in the HTTP 401 response to the anonymous request above: HTTP/1.1 401 Unauthorized Cache-Control: private Content-Length: 6055 Content-Type: text/html; charset=utf-8 Date: Tue, 13 Feb 2018 18:57:03 GMT Server: Microsoft-IIS/8.5 WWW-Authenticate: Negotiate X-Powered-By: ASP.NET We can see this response has been sent from IIS, per the "Server" header. During the course of processing the request and generating the response, the Windows Authentication module added the "WWW-Authenticate" header, with a value of "Negotiate" to match what was configured in IIS. This tells the client how the server expects a user to be authenticated. This response gets logged as a "401 2 5" in the IIS logs: sc-status = 401: Unauthorized sc-substatus = 2: Unauthorized due to server configuration (in this case because anonymous authentication is not allowed) sc-win32-status = 5: Access Denied Side note: the "Negotiate" provider itself includes both the Kerberos and NTLM packages. These can be discerned by looking at the encoded auth strings after the provider name. NTLM and its auth string is described later in this post. Side note 2: The default settings for Windows Authentication in IIS include both the "Negotiate" and "NTLM" providers. This means the standard HTTP 401 response to the anonymous request will actually include two "WWW-Authenticate" headers - one for "Negotiate" and the other for "NTLM." Clients generally choose the one listed first, which is "Negotiate" in a default setup.

Request & Response #2

The client browser has received the HTTP 401 with the additional "WWW-Authentication" header indicating the server accepts the "Negotiate" package. The client will prefer Kerberos over NTLM, and at this point will retrieve the user's Kerberos token. The browser then re-sends the initial request, now with the token (KRB_AP_REQ) added to the "Authorization" header: GET / HTTP/1.1 Accept: text/html, application/xhtml+xml, image/jxr, */* Accept-Encoding: gzip, deflate, peerdist Accept-Language: en-US, en; q=0.5 Authorization: Negotiate YIIg8gYGKwY[...]hdN7Z6yDNBuU= Connection: Keep-Alive Host: server User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Edge/16.16299 Notice the encoded auth string starts with "YII.." - this indicates it's a Kerberos token, and is how you can discern what package is being used, since "Negotiate" itself includes both NTLM and Kerberos. Side-note: The client device will reach out to Active Directory if it needs to get a token. This communication takes place after the server sends the initial 401 (response #1), and before the client sends request #2 above. Of course, if the client has a cached Kerberos token for the requested resource already, then this communication may not necessarily take place, and the browser will just send the token it has cached. Side-note 2: Troubleshooting Kerberos is out of the scope of this post. This post shows a healthy, successful, working authentication flow, and assumes there were no problems retrieving a Kerberos token on the client side, and no problems validating that token on the server side. Once the server has received the second request containing the encoded Kerberos token, http.sys works with LSA to validate that token. If everything is good, http.sys sets the user context on the request, and IIS picks it up. At this point, the response gets built and the requested resource delivered to the browser: HTTP/1.1 200 OK Content-Encoding: gzip Content-Length: 608 Content-Type: text/html Date: Tue, 13 Feb 2018 18:57:03 GMT ETag: "b03f2ab9db9d01:0" Last-Modified: Wed, 08 Jul 2015 16:42:14 GMT Persistent-Auth: true Server: Microsoft-IIS/8.5 WWW-Authenticate: Negotiate oYG3MIG0oAMKAQChC[...]k+zK X-Powered-By: ASP.NET We can see this request was ultimately serviced by IIS, per the "Server" header. This also means we'll see this particular request/response logged in the IIS logs with a "200 0 0" for the statuses. We can also see an additional "WWW-Authenticate" header - this one is the Kerberos Application Reply (KRB_AP_REP). This is so the client can authenticate if the server is genuine. It's certainly not obvious here that http.sys took care of user authentication for the 2nd request before IIS got involved - just know that it did, as long as Kernel Mode is enabled :)

NTLM

Request & Response #1

This is the initial anonymous request by the browser: GET / HTTP/1.1 Accept: text/html, application/xhtml+xml, image/jxr, */* Accept-Encoding: gzip, deflate, peerdist Accept-Language: en-US, en; q=0.5 Connection: Keep-Alive Host: server User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Edge/16.16299 I've configured Windows Authentication to only use the "NTLM" provider, so these are the headers we get back in the HTTP 401 response to the anonymous request above: HTTP/1.1 401 Unauthorized Cache-Control: private Content-Length: 6055 Content-Type: text/html; charset=utf-8 Date: Tue, 13 Feb 2018 17:57:26 GMT Server: Microsoft-IIS/8.5 WWW-Authenticate: NTLM X-Powered-By: ASP.NET We can see this response has been sent from IIS, per the "Server" header. During the course of processing the request and generating the response, the Windows Authentication module added the "WWW-Authenticate" header, with a value of "NTLM" to match what was configured in IIS. This tells the client how the server expects a user to be authenticated. This response gets logged as a "401 2 5" in the IIS logs: sc-status = 401: Unauthorized sc-substatus = 2: Unauthorized due to server configuration (in this case because anonymous authentication is not allowed) sc-win32-status = 5: Access Denied

Request & Response #2

The browser sees the server has requested NTLM authentication, so it re-sends the original request with an additional Authorization header, containing the NTLM Type-1 message: GET / HTTP/1.1 Accept: text/html, application/xhtml+xml, image/jxr, */* Accept-Encoding: gzip, deflate, peerdist Accept-Language: en-US, en; q=0.5 Authorization: NTLM TlRMTVN[...]ADw== Connection: Keep-Alive Host: server User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Edge/16.16299 Side note: we can tell this is NTLM because the base64-encoded auth string starts with "TlRM..." - this will also be the case when NTLM is used with the Negotiate provider. At this point, the server needs to generate the NTLM challenge (Type-2 message) based off the user and domain information that was sent by the client browser, and send that challenge back to the client. This is where the IIS/http.sys kernel mode setting is more apparent.  Instead of the HTTP request with the encoded auth string being sent all the way up to IIS, http.sys makes a call to the Local Security Authority (LSA -> lsass.exe) to retrieve the NTLM challenge. Once it has been received, http.sys generates the next HTTP response and sends the challenge back to the client. This is another 401: HTTP/1.1 401 Unauthorized Content-Length: 341 Content-Type: text/html; charset=us-ascii Date: Tue, 13 Feb 2018 17:57:26 GMT Server: Microsoft-HTTPAPI/2.0 WWW-Authenticate: NTLM TlRMTVN[...]AAA Note the "Server" header now - this indicates the response was generated and sent back to the client by http.sys, not IIS. We've also got another "WWW-Authenticate" header here, containing the "NTLM" provider indicator, followed by the base64-encoded NTLM Type-2 message string. Since this request never made it to IIS, so you will not see it logged in the IIS logs. It's not logged by http.sys, either. It, along with the other requests shown here, can be observed by using an HTTP message tracer, such as the Developer Tools built into all major browsers, Fiddler, etc.

Request & Response #3

At this point, the browser has received the NTLM Type-2 message containing the NTLM challenge. Now, it needs to send the original request one more time, and add the challenge response (NTLM Type-3 message): GET / HTTP/1.1 Accept: text/html, application/xhtml+xml, image/jxr, */* Accept-Encoding: gzip, deflate, peerdist Accept-Language: en-US, en; q=0.5 Authorization: NTLM TlRMTVN[... much longer ...]AC4A Connection: Keep-Alive Host: server User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Edge/16.16299 This completes the client-side portion, and now it's up to the server to finish the user authentication. Just like before, http.sys takes care of parsing the "Authorization" header and completing the authentication with LSA, before the request is handed over to IIS. Once authentication is complete, http.sys sets the user context to the authenticated user, and IIS picks up the request for processing. In other words, when IIS receives the request, the user has already been authenticated. If all went well, then the appropriate response is generated by IIS and the hosted page/app/etc., and the response is sent back to the user. In our case below, the response had a status of HTTP 200: HTTP/1.1 200 OK Content-Encoding: gzip Content-Length: 608 Content-Type: text/html Date: Tue, 13 Feb 2018 17:57:26 GMT ETag: "b03f2ab9db9d01:0" Last-Modified: Wed, 08 Jul 2015 16:42:14 GMT Persistent-Auth: true Server: Microsoft-IIS/8.5 X-Powered-By: ASP.NET We can see this request was serviced by IIS, per the "Server" header. This also means we'll see this particular request/response logged in the IIS logs with a "200 0 0" for the statuses.

Thoughts/Observations

This post shows what good, working HTTP requests and responses look like when Windows Authentication using Kerberos and NTLM is used successfully. The Kernel Mode aspects aren't as obvious at this level, with the exception of the NTLM Type-2 Message (the challenge) sent in the response from http.sys. You can also see that HTTP 401 statuses are completely normal in these scenarios, with Kerberos auth receiving just one 401 (for the initial anon request), and NTLM receiving two (one for the initial anon request, the second for the NTLM challenge). If you've stumbled across this post looking to understand why you're seeing 401s when nothing is actually wrong, hopefully this helps clear at least some of the smoke. All current browsers, at least that I know of, handle these authentication processes with no need for user intervention - the browser does all the heavy lifting to get this done. Generally, browsers will only prompt the user for credentials when something goes wrong with the flows shown above. The same goes for many applications using various kinds of frameworks, like .NET.

More Info

How the Kerberos Version 5 Authentication Protocol Works NTLM Messages]]> Things to check when a CDO mail-enabled Classic ASP site is not sending mail after a migration to IIS7+ https://docs.microsoft.com/archive/blogs/mist/things-to-check-when-a-cdo-mail-enabled-classic-asp-site-is-not-sending-mail-after-a-migration-to-iis7 Mon, 12 Oct 2015 16:26:00 GMT https://blogs.technet.microsoft.com/mist/2015/10/12/things-to-check-when-a-cdo-mail-enabled-classic-asp-site-is-not-sending-mail-after-a-migration-to-iis7/ Summary CDOSYS can send mail in a few different ways. If there is no code specifying which server to send mail from, then it attempts to send it from the server the code is hosted on with the SMTP server optional feature. Conversely, you can add code that specifies what server to use if you do not wish to use the built-in SMTP server. Each option has its own pros, cons and items to keep in mind. I outline these two scenarios below. As an aside, I was able to gain massive insight on how to resolve these customers' issues by analyzing Failed Request Traces (FREB) and seeing how IIS moved through the code and when/where any errors were presented. Having the customer's code handy also proved very helpful. One of the main benefits of FREB tracing is that just about every Classic ASP line that is processed has its own line on the trace. Coupled with the customer's code, for example, you can see when an if statement is hit, then see if the code inside the if block executed to determine how the condition resolved, etc. This can be very useful when troubleshooting. Note: This post assumes Classic ASP pages that are not mail-enabled are working just fine.

Scenarios

I will address two main scenarios on this page:
  1. Using the built-in SMTP server to send email from pages utilizing CDOSYS for email. The server will go through the motions to get the mail delivered. This is the prevalent scenario I've worked with customers on and does not require any code specifying how mail is sent.
    • This option is great if you have multiple pages that send mail the same way - the server changes handle all of them. If you've got 20 sites that send email the same exact way, then this option will very likely get them all working without having to touch those pages.
    • With some slight modifications this configuration can also be used to forward mail to a separate SMTP server for delivery. This can be useful if your organization requires all mail to pass through a specific server or cluster to be delivered. Using Office 365/Exchange Online as an SMTP service requires other actions to take place. Here is a TechNet guide on this.
    • This option does require certain security measures to ensure the server is not exploited as an open relay.

  2. Using code to connect to an external SMTP server for mail delivery.
    • This is beneficial if you have very few sites that send email this way and you are able to utilize a separate SMTP server. You don't have to worry about modifying the server or even installing the SMTP Server feature.
    • This option may not be the best if you have a high amount of sites that send email this way, as each one that sends email will need the code changes implemented. Along with code changes in Production come testing.
The most basic implementation of CDOSYS to send email looks something like this: Set myMail=CreateObject("CDO.Message") myMail.Subject="CDOSYS Email" myMail.From="robot@contoso.com" myMail.To="user@contoso.com" myMail.TextBody="This message is using CDOSYS." myMail.Send set myMail=nothing Below I dive into the two scenarios and detail what needs to be checked for each one.

Scenario 1: Server Configuration

In IIS Manager, navigate to the application pool that is hosting your mail-enabled site and note the identity it's running under. By default this is ApplicationPoolIdentity on IIS 7.5+, which is a good practice for single-server setups. If you're using a different account, like a domain account, then add it as a member to the local "IIS_IUSRS" group via the Local Users and Groups window (Start->Run->lusrmgr.msc). This will help us later and I am going to assume for the upcoming steps that this has been done. Ensure the "SMTP Server" feature is installed. This installs the "Simple Mail Transfer Protocol (SMTP)" Service, which, by default, will not be started and is set to a "Manual' startup type. Ensure you start the service and change it's startup type to "Automatic" or "Automatic (Delayed)." Note: Do not get this confused with the "SMTP Email" module you see in the standard IIS Manager. That module and its settings are for use with the System.Net.Mail API, not CDOSYS.

Folder Permissions

The SMTP Server feature adds a new folder inside C:\inetpub named "mailroot." On this folder, you'll need to add "Modify" permissions for the "IIS_IUSRS" group. Ensure this trickles down to all subfolders. This site has great explanations of the various subdirectories in the mailroot folder and how they are used.

SMTP Service Permissions

Now we need to take care of the permissions to actually use the SMTP service. When the SMTP Server feature was installed, it also installed the IIS 6 Management Console dependency. This means you now have "Internet Information Services (IIS) 6.0 Manager" installed alongside the standard IIS Manager for 7.0+. You can find it in Administrative Tools. When you need to administer the legacy SMTP service settings, IIS 6 Manager is where you'll do it. There is one exception to this, however; for the permissions we need to change, we need Metabase Explorer from the IIS 6 Resource Kit. You can download it from here. When going through the installation, choose a custom installation and select only Metabase Explorer since we don't need anything else for this step. Once it's installed, run as administrator and navigate to down to the LM/SmtpSvc node. Right-click on it and select "Permissions." Add the "IIS_IUSRS" local group with Full Control perms. Do the same thing for the "1" node under LM/SmtpSvc. Your sites should now be able to send email. But we aren't finished yet - right now we have an open relay that needs to be locked down.

SMTP Service Security

Note: Microsoft does not assume any liability if your SMTP server is used for malicious purposes, whether or not the hardening measures are applied here. Please ensure you thoroughly understand the security options available to you for this SMTP feature, including those not mentioned here. There will be some resources at the bottom of this post that you can utilize to gain a better understanding of the security principals needed here. By default, any machine can use the SMTP service to send and/or relay mail. Let's fix that. Open the IIS 6 Manager and navigate to SMTP Virtual Server #1, right-click->Properties. Navigate to the "Access" tab and click on "Connection" in the "Connection control" section. Make sure yours looks like the screenshot below (you'll need to add 127.0.0.1 to the list): Click OK and click on "Relay" just below the "Connection" button. This opens a similar window as before; go through the same steps and ensure it looks like this: Now the SMTP service will only accept mail sent from its host server. You can make modifications to these areas if you wish for other computers to be able to send email through this one. I do not cover those scenarios here. Note: Depending on your firewall software, it's possible an automatic rule for SMTP port 25 was created when the SMTP Server feature was installed to allow incoming SMTP traffic to your server. This is not needed if messages originate on this server. If that is the case, and no other servers or code anywhere will be relaying mail via this server, then you can close the incoming SMTP port(s) as an extra layer of security. All done! At this point your Classic ASP sites should be able to send email AND your SMTP server feature is more secure. There are other tweaks in this Properties window that you can make, such as limiting message sizes and where to send NDRs to as well. Take a look around to see if you want to modify any of those. If you wish for your server to forward mail to another SMTP server, then keep reading.

Forwarding Email to Another SMTP Server

If your organization wishes to internally forward email to another server to be processed and sent to the recipients, then a few more changes are needed. We are now moving to the "Delivery" tab in IIS6 Manager -> SMTP Virtual Server -> Properties. If your downstream server requires authentication, click the "Outbound Security..." button at the bottom and choose the appropriate settings. Click OK when finished. The default port the SMTP service uses to make outbound connections is 25. If you need to change this or want to limit the number of overall connections, click the "Outbound connections..." button. Click OK when finished. Now, in that Delivery tab, click on the "Advanced" button. In the "Smart host" field, enter the URL, FQDN or IP address (IP should be enclosed in brackets [ ]) where mail should be delivered for further processing. Click OK. Note: If you want to use Office 365/Exchange Online to send and/or relay email, there are other points that must be addressed. This page lists the options available and how to go about implementing them. All done!

Scenario 2: Code Changes

Note: If you did the server changes above, you do not need to make these code changes. Note: As with any Production code changes, ensure you have properly tested before deploying! Microsoft assumes no liability due to any issues that arise from making the changes mentioned here. The default way the code will send email is to use the built-in Windows SMTP Server on the same server the code is hosted. That is detailed in the above steps. If you want to modify your code to direct email elsewhere for delivery, here are the parameters you need to add. This does not require the built-in SMTP server to be installed. This site has a great tutorial on what settings can be changed to make this happen. Below is the same code as the top of this post with the most basic code changes applied: Set myMail=CreateObject("CDO.Message") myMail.Subject="CDOSYS Email" myMail.From="robot@contoso.com" myMail.To="user@contoso.com" myMail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2 myMail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserver") = “smtp.contoso.com” myMail.TextBody="This message is using CDOSYS." myMail.Send set myMail=nothing The first configuration item (sendusing) tells the server the code wants to send email using a remote server (option '2'), which is specified below it. By default, the server will attempt anonymous authentication over port 25. There are configurations for authentication (smtpauthenticate), using SSL/TLS (smtpusessl), port (smtpserverport), and so on if needed. The tutorial site mentioned above utilizes some of those other configuration items. This MSDN link contains all the configurable options.

Resources

]]> HTTP404 while using IIS URL Rewrite to present a different WebApp https://docs.microsoft.com/archive/blogs/mist/http404-while-using-iis-url-rewrite-to-present-a-different-webapp Tue, 06 Oct 2015 15:40:00 GMT https://blogs.technet.microsoft.com/mist/2015/10/06/http404-while-using-iis-url-rewrite-to-present-a-different-webapp/ I recently supported a customer who was implementing a reverse proxy in IIS using URL rewrite to present an SSRS report from a backend server via an external-facing IIS machine. They had developed the rewrite rule in their Dev environment and it worked perfectly. They moved it over to Prod and modified it accordingly; however, in Prod the page was returning an HTTP 404 File Not Found error.

Summary

A 404 while trying to implement a reverse proxy where all URLs are correct and the resources exist could indicate that ARR is not installed and/or enabled, causing IIS to attempt to satisfy the entire request all on its own.

Failed Request Tracing (FREB) proved invaluable to troubleshooting this issue.

Details

We first investigated the URL rewrite rule to confirm it was built correctly - it was. We then collected failed request (FREB) traces from the working and non-working scenarios. We saw from the FREB in both scenarios the URL rewrite module correctly matched the pattern and the correct backend URL to hit. Testing the generated SSRS report URL directly produced a working report. Conclusion: our problem was not the URL rewrite module. So I continued digging into the FREB traces to see how the request was treated. Below are what events took place after the URL rewrite module finished:

working trace:


nonworking trace:

You'll notice a different set of events immediately following the URL_REWRITE_END event. Most important is the HANDLER_CHANGED event. This indicates the request processing pipeline handed-off processing of the request from PageHandlerFactory-Integrated-4.0 to the ApplicationRequestRoutingHandler in the working scenario. In the nonworking scenario, the PageHandlerFactory[...] does not hand-off the request and continues to process it, eventually moving through the Windows Authentication module and on to the AspNet pipeline. Further down the line (not shown in the screenshot) I see all the normal request processing events as if IIS was trying to process the entire request itself. This explains why a 404 is being returned: the server is processing the request and attempts to present the SSRS report as if it is hosted on itself, instead of reaching out to the SSRS server. Even though IIS is changing the URL in the rewrite module, it's acting as if that URL is hitting just another site located on itself.

Implementing a reverse proxy in IIS that utilizes a separate backend server for requests requires Application Request Routing (ARR) on the proxy server. This module takes care of creating the request with the necessary headers, sending it to the downstream server (SSRS on top of IIS in this case), and capturing the response for further processing and flushing to the user. This is what we see in the working trace. To further cement this, we see the gap in time (54.208 to 54.536) between the URL_CHANGED event and the GENERAL_NOT_SEND_CUSTOM_ERROR event, whereas there is no gap of time whatsoever in the nonworking trace. The descriptions of the events indicate the request was successfully serviced by the downstream IIS machine and a response was received, which is then processed.

At this point I provided this information to the customer and they found that ARR was installed on the IIS proxy server, but not enabled in the IIS configuration. Once they enabled it, the issue resolved and the site started working as intended.

Below is a graphical representation of how this scenario works:

Resources

http://www.iis.net/learn/extensions/url-rewrite-module/reverse-proxy-with-url-rewrite-v2-and-application-request-routing
http://www.iis.net/learn/extensions/planning-for-arr/using-the-application-request-routing-module
http://www.iis.net/learn/extensions/url-rewrite-module/using-the-url-rewrite-module
http://www.iis.net/learn/troubleshoot/using-failed-request-tracing/troubleshoot-with-failed-request-tracing

]]>