Most SQL DBA's are aware of the stored procedure – sp_change_users_login which is generally used to report and fix orphaned SQL SERVER LOGIN’s. I am explicitly mentioning this since the stored procedure cannot be used for Windows Logins. This has been explicitly pointed out in books online –

sp_change_users_login cannot be used to map database users to Windows-level principals, certificates, or asymmetric keys.

If you still want to experiment -

EXEC sp_change_users_login 'Update_One', 'TestUser', 'Testdomain\TestUser';

GO

Then you will encounter the below error –

Msg 15291, Level 16, State 1, Procedure sp_change_users_login, Line 114

Terminating this procedure. The User name 'TestUser' is absent or invalid.

So what do you do when you want to fix orphaned Windows users.

In such case you can use alter user [user_name] with login = [login_name] to fix orphaned windows login. This command re-maps a user to another login by changing the user's Security Identifier (SID) to match the login's SID.

Additionally, while researching on this topic I also found that sp_change_users_login is deprecated. 

From books online again - This feature will be removed in a future version of Microsoft SQL Server. Avoid using this feature in new development work, and plan to modify applications that currently use this feature. Use ALTER USER instead.

Hence you can now use ALTER USER command for all type of users.

Incase you are interested in finding out orphaned Windows Users, you can use the below query –

SELECT NAME,SID FROM SYS.DATABASE_PRINCIPALS

WHERE

TYPE = 'U' --Users only

AND AUTHENTICATION_TYPE = 3 --Windows Logins only

AND NAME NOT LIKE 'DBO'

EXCEPT

(SELECT DP.NAME, DP.SID FROM SYS.DATABASE_PRINCIPALS DP

INNER JOIN SYS.SERVER_PRINCIPALS SP ON DP.SID = SP.SID 

WHERE DP.TYPE = 'U'

AND DP.AUTHENTICATION_TYPE = 3)

Hope this helps you in case you are stuck with these orphaned users! Do send in your comments/thoughts.

Disclaimer: The information in this weblog is provided "AS IS" with no warranties, and confers no rights. This weblog does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my opinion. Inappropriate comments will be deleted at the authors discretion. All code samples are provided "AS IS" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular.

Recently I stumbled upon a Login failure error while connecting to an SQL Instance remotely. The authentication scheme used in this case was Pass-through Authentication.

For those who are not aware of Pass-through authentication, it’s basically a technique in which a local windows user account is created on the source and destination server with the same password. If the user account has the same username and password, Windows authenticates this user and lets him access the resources. This scheme is generally used where there is prohibition of domain account.

So here’s the error which I got while connecting via SSMS –

Looking at the errorlog revealed the below errors – 

SSPI Handshake Failed errors happens usually when the user is not authenticated. Let’s see what the specific error code means. For this I generally use the command line utility Err.exe which is one of the must haves for an administrator.

Here is what I got – 

In this case the output was not very helpful. 

Next I looked at the security event logs and found this Audit failure– 

In this case, it could not be unknown Username since the account existed in SQL Server. But yeah, it could be a Bad password. And that it was, resetting the password on the machine from where I was connecting to SQL Server did the trick for me. 

Learnings

Password mismatch for the local windows account between the servers. For pass-through authentication to work it’s important that we have the same username and password.

Even though there is a mismatch in the password we do not get 18456 (Login failed due to password mismatch) because the authentication scheme is slightly different in this case. The username and passwords are itself used to match the login in pass-through authentication.

Disclaimer: The information in this weblog is provided "AS IS" with no warranties, and confers no rights. This weblog does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my opinion. Inappropriate comments will be deleted at the author’s discretion. All code samples are provided "AS IS" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular.

Next, you go to Server Certificate feature to check for the installed certificate before you can bind the certificate with the website.

But the Certificate you just installed is not showing in the console, its missing! What do you do?

You verify the certificate is valid by checking the following -

1. That the current date and time is within the "Valid from" and "Valid to" date range on the certificate.
2. That the certificate's "Common Name" (CN) matches the host header in the request. For example, if the client is making a request to http://www.contoso.com/, then the CN must also be http://www.contoso.com/.
3. That the issuer of the certificate is a known and trusted CA

All the above conditions are true but still the certificate is not showing up.

It happens that the Server Certificates module in IIS manager displays a list of certificates from the Local Machine SSL store. But it only lists the certificate if - 

1. The certificate has a private key

2. The certificate is meant for Server Authentication

In my case I found that the private key was missing in the certificate.

We can recover the private key by following the below steps –

To recover the private key on the certificate, we need to repair the certificate using the following command –

certutil –repairstore my <serial number>

where serial number can be obtained by looking up the certificate properties as indicated below.

Once the certificate was repaired, we were able to get back the private key.

If the above commands completes successfully but does not work to get back the private key then you may have to recreate the certificate.

Once my private key was retrieved, the Server Certificates console starting showing up the certificate as well.

References

Certutil - https://technet.microsoft.com/en-us/library/cc732443.aspx

http://blogs.msdn.com/b/jaskis/archive/2009/04/22/missing-private-keys-pair-after-deleting-pending-request-from-iis.aspx

Disclaimer: The information in this weblog is provided "AS IS" with no warranties, and confers no rights. This weblog does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my opinion. Inappropriate comments will be deleted at the author’s discretion. All code samples are provided "AS IS" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular.

In my day to day work, I do a lot of troubleshooting on the IIS Admin side but while delving into the scheme of things I realized that more often than not the issues lie in application code.

Most developers are able to troubleshoot code issues on their development boxes. But what if the issue is only happening on production servers and not on development machines. This will need live debugging, which is the idea behind this blog post.

For the purpose of this blog, I have created a Sample application which generates a Null pointer reference exception. But the debugging method can be used to troubleshoot any type of exception.

Here is the snip of the login form -

Once I click on the Submit button I get the below Server Error -

Note - This type of detailed errors will generally be seen when you are browsing the web page locally on the web server and not on remote clients (default settings). To change this you can follow - https://technet.microsoft.com/en-us/library/bb684665.aspx.

At the same time, you would generally see an event in the application system event log –

<snip> 

Event code: 3005

Event message: An unhandled exception has occurred.

Application information:

    Application domain: /LM/W3SVC/4/ROOT-1-130775233748599802

    Trust level: Full

    Application Virtual Path: /

    Application Path: E:\SampleWebApp1\

    Machine name: DESALG-LAPTOP

Process information:

    Process ID: 18744

    Process name: w3wp.exe

    Account name: IIS APPPOOL\SampleWebApp1

Exception information:

    Exception type: NullReferenceException

    Exception message: Object reference not set to an instance of an object.

   at SampleWebApp1.WebForm1.Button1_Click(Object sender, EventArgs e)

   at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument)

   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

Request information:

    Request URL: http://localhost:8080/WebForm1.aspx

    Request path: /WebForm1.aspx

    User host address: ::1

    User: FAREAST\desalg

    Is authenticated: True

    Authentication Type: Negotiate

    Thread account name: IIS APPPOOL\SampleWebApp1

<snip> 

From the stack trace, you can see that it tells us the function (SampleWebApp1.WebForm1.Button1_Click) in which the exception occurred but it does not tell us the line of code where the exception occurred.

Assuming this code is deployed on the web server, here is a way how you can debug this live using windbg.

You can download and install WinDBG from here - https://msdn.microsoft.com/en-us/windows/hardware/hh852365

Note – You can follow the link https://msdn.microsoft.com/en-us/library/dd465337(v=vs.110).aspx to deploy the attached project to this blog on the web server.

If you deployed this with the default settings then hitting the link - http://localhost:8080/WebForm1.aspx would open the web page and trigger the worker process on the server.

Step 1 – Setup the Source File path (Location of the application source files) – File > Source File Path …

Step 2 - Setup the Symbol File path (Location of the application pdb files) – File > Symbol File Path …

Step 3 – File > Attach to the process – Select the worker process

Once you click ‘Ok’, the worker process will get attached to WinDBG and the control will be with debugger.

Step 4 – Run the below command to load the appropriate version of SOS.dll in WinDBG which is required for the purpose of debugging.

0:035> .cordll -ve -u -l

Automatically loaded SOS Extension

CLRDLL: Loaded DLL C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscordacwks.dll

CLR DLL status: Loaded DLL C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscordacwks.dll

*** WARNING: Unable to verify checksum for SampleWebApp1.dll

0:035> .Chain –Gives the extensions loadedby windbg

<snipped>

Extension DLL chain:

    C:\Windows\Microsoft.NET\Framework64\v4.0.30319\sos: image 4.0.30319.34209, API 1.0.0, built Sat Apr 12 07:33:15 2014

        [path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\SOS.dll]

<snipped>

Step 5 – Type ‘g’ in the command line of WinDBG to pass the control back to the application.

Step 6 – Fill in the login form and Click on Submit to generate the exception. The control will be sent back to WinDBG when the exception is hit.

0:035> g

(4910.4a7c): CLR exception - code e0434352 (first chance)

CLR exception type: System.Web.HttpUnhandledException

    ""

(4910.1dd4): Access violation - code c0000005 (first chance)

First chance exceptions are reported before any exception handling.

This exception may be expected and handled.

SampleWebApp1!SampleWebApp1.WebForm1.Button1_Click(System.Object, System.EventArgs)+0x8:

00007ff7`c71318e8 8b00            mov     eax,dword ptr [rax] ds:00000000`00000008=????????

If your source code alignment is properly done, windbg does a pretty neat job of highlighting the line of code that has caused the exception. This should be good enough information for the developer of the code to take forward the debugging.

For advanced folks, little bit more debugging and you can pull more details about the exception like the call stack and the type of exception.

0:040> !clrstack  -- Gives the callstack of the thread which threw the exception

SampleWebApp1.WebForm1.Button1_Click(System.Object, System.EventArgs)

System.Web.UI.WebControls.Button.RaisePostBackEvent(System.String)

System.Web.UI.Page.ProcessRequestMain(Boolean, Boolean)

System.Web.UI.Page.ProcessRequest(Boolean, Boolean)

System.Web.UI.Page.ProcessRequest()

System.Web.UI.Page.ProcessRequest(System.Web.HttpContext)

System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)

System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)

System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)

System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)

System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)

System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)

DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)

[InlinedCallFrame: 0000003e724eeac8] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)

[InlinedCallFrame: 0000003e724eeac8] System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr, System.Web.RequestNotificationStatus ByRef)

DomainNeutralILStubClass.IL_STUB_PInvoke(IntPtr, System.Web.RequestNotificationStatus ByRef)

System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)

System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)

DomainNeutralILStubClass.IL_STUB_ReversePInvoke(Int64, Int64, Int64, Int32)

[ContextTransitionFrame: 0000003e724eefc8]

0:040> t –Step into the code

(4910.1dd4): Access violation - code c0000005 (first chance)

First chance exceptions are reported before any exception handling.

This exception may be expected and handled.

KERNELBASE!RaiseException+0x6c:

00007ff8`337f8b9c 488b8c24c0000000 mov     rcx,qword ptr [rsp+0C0h] ss:0000003e`724ed450=0000ed06eea53973

0:040> t

(4910.1dd4): CLR exception - code e0434352 (first chance)

clr!IEE+0x71a7:

00007ff8`268a9c03 483b4728        cmp     rax,qword ptr [rdi+28h] ds:00007ff8`096b77d8=00000000000cb27b

0:040> !pe –Prints exception on the callstack and will work if the SOS dll is loaded

Exception object: 0000003ca7048cb8

Exception type:   System.Web.HttpUnhandledException

Message:          <none>

InnerException:   System.NullReferenceException, Use !PrintException 0000003ca7048990 to see more.

StackTrace (generated):

<none>

StackTraceString: <none>

HResult: 80004005

There are nested exceptions on this thread. Run with -nested for details

References - http://blogs.msdn.com/b/kaevans/archive/2011/04/11/intro-to-windbg-for-net-developers.aspx

Disclaimer: The information in this weblog is provided "AS IS" with no warranties, and confers no rights. This weblog does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my opinion. Inappropriate comments will be deleted at the authors discretion. All code samples are provided "AS IS" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular.

SampleWebApp1.7z

Msg 3013, Level 16, State 1, Line 2

RESTORE LOG is terminating abnormally.

Msg 3441, Level 17, State 1, Line 2

During startup of warm standby database 'TestDB' (database ID 5), its standby file

 ('Q:\mssql\backup\TestDB\TestDB_20150419043800.trn') was inaccessible to the RESTORE statement.

The operating system error was '32(The process cannot access the file because it is being used by another process.)'.

Diagnose the operating system error, correct the problem, and retry startup.

If you look carefully at the error message – its saying that the standby file which it is looking for is “TestDB_20150419043800.trn”.

Strange!...This is the same backup file we are trying to restore using the below command -

restore log TestDB from disk = 'Q:\mssql\backup\TestDB\TestDB _20150419043800.trn' with recovery

How can this happen?

I suspected that this could be a problem with the restoration command used to put the database in standby. I asked the DBA, if he had the command which was used to put the database in standby mode. Unfortunately, the answer was a No. How could I confirm.

Ok, no problem. I asked him to send me the Default traces since I knew we track restore statements in the default trace file.

Hoping to find something useful, I opened the default trace and guess what I found –

Looks like the DBA had restored the previous transaction log file with standby but by mistake specified the same transaction log backup file in the standby clause. It did not give an error but effectively what happened was that the .trn file was having the standby information.

Thinking that there was some corruption with the transaction log backup file, he had gone ahead and replaced the log backup file (which now had the standby information) with the original transaction log backup file. By doing so, we lost the standby information and we could not restore further backups nor recover the database since it was looking for the standby file which was no longer present.

In this case, the only option to get out of this situation was to restore the database from Full backup and apply the log backups.

Moral of the story – Database restore sequences are very sensitive, be very careful.

Disclaimer: The information in this weblog is provided "AS IS" with no warranties, and confers no rights. This weblog does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my opinion. Inappropriate comments will be deleted at the authors discretion. All code samples are provided "AS IS" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular.

The HTTP request was forbidden with client authentication scheme 'Anonymous'.

He had done his homework before coming to me and said that he was able to upload files of smaller size but when he would try larger files (around 40-50 MB) he was getting the above error. This meant that at least the upload module was working fine.

I asked the developer to reproduce the issue and I looked at the IIS logs generated for the website.

Here’s what I found –

GET /Rejected-By-UrlScan ~/TestService/TestProtocolService.svc 88 - ::1 - - 404 0 2 100

GET /Rejected-By-UrlScan ~/TestService/TestProtocolService.svc 88 - ::1 - - 404 0 2 1

Oh, looks like there is a URLSCAN module installed on IIS which is rejecting the requests.

For those who are not aware of URLSCAN – It is basically security extension that restricts the types of HTTP requests that IIS will process. By blocking specific HTTP requests, the UrlScan helps to prevent potentially harmful requests from reaching applications on the server. This was used in older versions but newer versions (IIS 7 and above) has a new feature called as Request Filtering.

But why are these requests getting rejected, these seem to be legitimate requests.

I looked into the URLSCAN logs (Default Location - C:\Windows\System32\inetsrv\urlscan\logs) to understand the reason -

POST /TestService/TestProtocolService.svc Rejected Content+length+too+long Content-Length: 89437188 30000000

Here you go – the content length is too long. It also provides the length of the content attempted to be uploaded as well. Pretty neat logging!

Jumped directly into the URLSCAN configuration file (Default Location - C:\Windows\System32\inetsrv\urlscan\UrlScan.ini) and found the below comment in the configuration file –

;   - MaxAllowedContentLength specifies the maximum allowed

;     numeric value of the Content-Length request header.  For

;     example, setting this to 1000 would cause any request

;     with a content length that exceeds 1000 to be rejected.

;     The default is 30000000.

The default max. allowed content length was 30 MB. We altered this configuration and increased the parameter to a higher value to fix the problem. 

Disclaimer: The information in this weblog is provided "AS IS" with no warranties, and confers no rights. This weblog does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my opinion. Inappropriate comments will be deleted at the authors discretion. All code samples are provided "AS IS" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular.

This was actually the SQL Server resource which was not coming ONLINE. As usual the DBA said NOTHING changed, it just happened!!

One of the first things we normally do is to start the service from the SQL Server Configuration Manager/Services console so that we can isolate the issue. In this case we found that the service was failing to start with the below errors:-

---------------------
TITLE: Services
---------------------
Windows could not start the SQL Server (MSSQLSERVER) service on Local Computer.
Error 1053: The service did not respond to the start or control request in a timely Fashion.
Button: OK
--------------------

Generic error, you don’t expect much from the GUI anyways!! Went for the SQL Server Errorlogs and Application/System Event viewer logs. We did not find the Errorlog created in the SQL Server folder nor did any errors get logged in the Application/System logs.

Commonly, in such scenarios, the start-up parameters are goofed up but in this case it was not.

So the next logical step was to start the service in the command prompt in console mode:-

C:\Windows\system32>"C:\Program Files\Microsoft SQL Server\MSSQL10_50.R2\MSSQL\Binn\sqlservr.exe" -sR2 –c

_

It did show any message indicating that the service in not even initialising. It appeared that it was sort of in a hung state. Strange!!

Then I pulled out the next tool from my toolkit: - PROCESS MONITOR a.k.a PROCMON (I hope most of you get the reason why I put that in CAPS for those of you who did not ..read on)

So I collected a PROCMON while running the service in command prompt. I filter it on SQLServr.exe and here is what I find.

I see that SQL Server accesses the ERRORLOG file but continues to perform a delete operation on files like ERRORLOG.321234533, ERRORLOG.321234532, ERRORLOG.321234531..Why would SQL do this and why these weird files names??

Scratching my head for some time and looking at the PROCMON output for some more time I figured it out. I see that before SQL attempts this file deletion it accesses the below registry key

Event Class:        Registry

Operation:         RegQueryValue

Result:  SUCCESS

Path:     HKLM\SOFTWARE\MICROSOFT\Microsoft SQL Server\MSSQL10_50.R2\MSSQLServer\NumErrorLogs

Type:     REG_DWORD

Length: 4

Data:     321234534

But now what is NumErrorLogs and when would that be set??

Remember there is an option in SQL Server where you can customise the Number of Errorlogs to be retained, by default its 6.

To confirm my understanding, I set this to 99 and collected a PROCMON on my test machine.

I see the similar symptoms of the Errorlogs being attempted to be deleted.

In short, what is happening is that SQL Server is internally is attempting to delete/rename the Errorlog files and arrange it so that only 99 or less Errorlog files are retained!!! Now since the registry key is set to such a high value SQL Server is looking for the Errorlogs and decrementing the number every time and hence it SQL Server appears to be hung!!!

But if you notice, the maximum value which can be set here is 99. So how does the NumErrorLogs registry key has such a high value??

No points for cracking that. The possible options for that happening:-

1. Manual intervention of changing this registry key
2. Running something like below in SQL Server

USE [master]
GO
EXEC xp_instance_regwrite N'HKEY_LOCAL_MACHINE', N'Software\Microsoft\MSSQLServer\MSSQLServer', N'NumErrorLogs', REG_DWORD, 321234533
GO

So finally, we go ahead and change the registry key back to the default value and BINGO SQL Server starts up!! Issue resolved!!

Before signing off, credits to Mark Russinovich for the tool PROCESS MONITOR without which I feel this issue could NEVER have been resolved!!

Disclaimer: The information in this weblog is provided "AS IS" with no warranties, and confers no rights. This weblog does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my opinion. Inappropriate comments will be deleted at the authors discretion. All code samples are provided "AS IS" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular.