Hello,

There are some great blogs out there which gives a lot of in-depth information on debugging. My main aim with this blog is to give an introduction to those who are very much new to debugging. .

In this blog I will try to explain how well we can troubleshoot Worker process Crash issues due to Stack Over flow exceptions.

Setting up Windows Debugger

=========================

Go ahead and download windows Debugger from the site.

https://www.microsoft.com/whdc/devtools/debugging/default.mspx

By default it gets installed in C:\Program Files\Debugging Tools for Windows. Open Windbg à File àset the symbol file path as srv*C:\Symbols\Pub*https://msdl.microsoft.com/download/symbols.

Create a folder Symbols and a subfolder Pub under C:\, press ok and Save the work space. This folder is to store the publicly available symbols from Microsoft (https://msdl.microsoft.com/download/symbols)

Setting up Debug Diag to Capture Crash dumps

==========================================

Install the latest version of DebugDiag https://www.microsoft.com/downloads/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en

Configure DebugDiag to capture the memory dump when the ExitProcess or TerminateProcess  command is called. This will ensure that we capture the memory dump when the faulting thread is terminating the process.

1. Open DebugDiag

2. On the Rules tab, click Add Rule

3. Select Crash and click Next

4. Select "All IIS Processes" and click Next

5. Under Advanced Settings, click Breakpoints

6. Click Add Breakpoint...

7. Select KERNEL32!ExitProcess and Kernel32!TerminateProcess .Change Action Type to Full UserDump

8. Click OK

9. Click Save and Close

10. Click Next through the rest of the wizard

Now whenever the crash occurs DebugDiag will capture the dump. By default it will be under C:\Program Files\DebugDiag\Logs\Crash rule for all IIS_COM+ related processes .

Make sure that you disable all the health monitoring settings in IIS Application pool to prevent the dumps getting generated due to a normal application pool recycle.

Once you have the dump load the same in Windbg. File -->Open Crash dump.

Type in the following command in the debugger to get the symbols files for all the loaded modules from the symbol file path

.reload /f --> reloading the modules (F is for forcing reload)

Once the debugger finishes downloading the symbols, open up C:\Symbols\Pub folder and you can see the Public symbols that got downloaded. From next time onwards debugger will be getting symbols from this cached location.

In order to run the commands and look through the dumps we need the SOS.dll which comes along with .Net Framework. Copy SOS.dl from C:\WINNT\Microsoft.NET\Framework\<Respective Version>\ and put it under C:\Program Files\Debugging Tools for Windows folder.

For getting help on the commands available in SOS.dll type !help on the debugger window . You will be able to see the list of commands available. If you need further help on any command type in !help <Command Name> Eg :- !help eeheap

Now to look at the threads type in !threads.

We can see that thread 14 is the one which has caused the stack over flow exception.

We are thread 0 now. To move to thread 14 type ~14s . To see the native call stack type kpn

To see the Managed stack type !clrstack

We can see that a function UserLogin() is getting called recursively which us causing the stack to overflow. Now let us try getting more details on the function such as the module and assembly name from which this function gets called.

Select a Code adress (Second column) from the above stack and type in !ip2md 02575e3a

We can see the module name(02207a80), type in !dumpmodule to dump out the module information

You can see that the UserLogin() function is getting called from the assembly App_Web_itzw09g-.dll

You can dump out the module information using lmvm command

Save the assembly to local disc using the command !savemodule App_Web_itzw09g_ c:\Assembly.dll and use reflector to get the function details.

Download Reflector

https://www.aisto.com/Roeder/Frontier/Default.aspx?PermaLink=53

Pass your comments to know whether this blog was helpful.....

So till next time..Bye bye...

Ref:- https://support.microsoft.com/kb/892277

This post is an Add-on to my previous blog. Yesterday I got my 15th Session loss case. This is the second time the same issue is coming to me and it took hardly 15 minutes for me to solve the same. But for the first time when we were troubleshooting the issue it took more than 8 hours to figure out what’s happening. Ok let’s start.

Some Session variables are lost while moving between pages.

Session variables are suddenly lost when we move from one page to another. That too only some specific session variables. There is no application domain restart, issue happens to all users from all machines and is consistently reproducible.

This is a little tricky scenario where we end up using all our troubleshooting skills and still end up at the “No Where” Side.

Customer had created a simple ASP.Net application to reproduce the issue. There were only two pages, In Page 1 he was setting some Session variables .On click of  "Move to Page2" we are redirected to page 2 where we print the Session variables. Page 2 contains an <Img > tag whose source is pointed to an image which doesn’t exist at all. So when we access the page we will see a cross sign.

Page 1

=========================

Session["Value1"] = "Ani";

Session["Value2"] = "Menon";

Page 2

=======================

Label1.Text = "First Value : " + Session["Value1"].ToString();

Label2.Text = "Second Value : " + Session["Value2"].ToString();

Now when click on “Back to Page 1” button, we are redirected to page 1, where we are trying to print the Session Values and Surprisingly one Session variable is lost.

There is no other code in these pages and in Global.asax.cs (No session.clear() or Session.abandon etc).

Now the toughest part. How to troubleshoot this??  !!!! .Here comes the power of ASP.Net tracing. Go ahead and enable tracing in Web.config file.

<trace enabled="true"/>

We tried browsing the pages again and checked for the sequence of events happening and found something interesting.

See the figure above. We can see that there is a GET request for Error.aspx {from where this request came from !!}.

We checked the Virtual directory settings in IIS and found that under Custom Errors Tab customer has set a redirection to a custom Error Page for 404. File not found.

(For demo purposes I am just redirecting to an Error page within the same directory as of the application, but in a complex real time scenario, this won’t be the case.)

We checked the Error.aspx.cs page and found that the Session value was set to “null” based on some condition.

Session["Value1"] = null;

We removed the custom error page and that fixed the issue. Now the interesting thing is that if we have set a custom error page the 404 redirection won’t get logged in IIS logs or tools like fiddler (www.fiddlertool.com) as it is already been handled.

Fiddler trace with Custom Error Redirection

Fiddler trace without Custom Error Redirection

So ultimately what's happening was , when we are redirected to Page2.apsx a 404 is generated for <IMG>. This is handled by the custom Error page settings in IIS , which redirects the request to Error.aspx which which was clearing out the Session variable.

Hope this helps…Until Next time... Bye Bye

I joined MS in April2006 and now it’s high time that I realized the need for a blog. I am a member of the IIS/ASP.NET team here at Microsoft PSS. I troubleshoot, I deliver technical solutions to complex problems, and I work with software engineers/developers across the globe.

One of the common and most difficult problems that I have to troubleshoot here in MS is Session loss. In this blog I will try to explain the different ways to approach a Session loss issue and the various reasons that can cause the same.

Why to use ASP.Net Session?

HTTP is a stateless protocol, which means that it does not automatically indicate whether a sequence of requests is all from the same client or even whether a single browser instance is still actively viewing a page or site. As a result, building Web applications that need to maintain some cross-request state information (shopping carts, data scrolling, and so on) can be extremely challenging without additional infrastructure help.

Where is Session Data Stored?

Session Data is stored on one of the following places depending on your application’s requirement

• In the worker process memory (InProc – Default)
• ASP.NET State Service
• SQL Server 2000 or higher database.

Session data Characteristics:-

• Are stored per user.
• Are objects in memory and therefore must be checked before using.
• Are automatically discarded after a period of inactivity or on Session.Abandon(), Session.Clear().
• Session ID is issued on the first request by the server.
• Session ID is persisted in an in-memory cookie of the browser.

Identifying a session ID

• Using a 120-bit SessionID string containing only the ASCII characters that are allowed in URLs.

• Fgazx3fsdqkww045itmtld45

• The SessionID strings are communicated across client-server requests either by means of an HTTP cookie or a modified URL with the SessionID string embedded, depending on how you configure the application settings.

Ok. Enough with basics, Let us move on to the troubleshooting part. Do remember one thing try to collect as much data as possible and try to isolate the issue to the lowest possible level. Proper isolation is the Key in Troubleshooting. And that’s the reason we are there for… J

So what data we need and how to isolate? Let’s Begin. Try getting the answers for these questions.

•  What Kind of application is it, whether it is ASP, ASP.NET 1.1 or ASP.NET 2.0?
• What do you mean by session loss?

a. Are the users getting redirected to login page?

b. Is the Session variables in your application are giving Null values?

c. Are you seeing a custom error page which says session is Invalid?

• How often does it happen? On each request?
• When it happens? When users are browsing the page or when they are sitting idle?

If idle, issue could be due to Session time out settings in Web.config

(<sessionState mode="InProc" timeout="20" />), Application pool settings (Health monitoring), Forms Ticket getting expired etc.

• Does it happen simultaneously for all users or is it happening for only some users?

If all, then application domain or Worker process recycling would be the cause. If it’s happening for only some users there is a possibility that the Session state is set to (<sessionState mode="InProc/> and Web garden is enabled.

• Is this a Web farm scenario? If so what is the session state mode you are using? Inproc, State Server or SQl Server?

This can be checked in the web.config file in ASP.NET applications. (https://msdn.microsoft.com/library/default.asp?url=/library/en-us/iisref/htm/UsingASPNET.asp )

If you are running Web farm and using Inproc Session that could be the possible cause of session loss as all the Session information is stored within the Worker process.

Format of the sessionState section:

<sessionState

mode="InProc"

stateConnectionString="tcpip=127.0.0.1:42424"

sqlConnectionString="data source=127.0.0.1;user id=sa;password="

cookieless="false"

timeout="20"

/>

In order to use StateServer session state mode, you will have to start the ASP State Service on the machine specified in the stateConnectionString attribute.

Another common issue that we have seen with Web farm scenario is when the IIS Metabase is not same for all the Web servers in the Farm. The main issue comes with Web site identifier mismatch.

https://support.microsoft.com/?id=325056

• Are users getting redirected to Login page? Is he using forms Authentication?

Do remember that the answer to this question is very important as neither ASP nor ASP.Net has got any mechanism where in users would be redirected to any login page on Session loss.

There is a very good KB article by Jerry Orman on troubleshooting Forms Authentication issues.

https://support.microsoft.com/kb/910439

• Does the domain name/server name have special characters such as “_” in them? Check this https://support.microsoft.com/?id=316112
• Scenarios where all users get session loss from a specific machine and the same user can access from other machine?

In this case issue could be mainly due to the client browser dropping cookie. Best way to approach this will be to try browsing the site with an alternate browser or use tools such as fiddler or Network Monitor to capture the traffic and see what’s happening.

Troubleshooting Classic ASP Applications

Let us talk about how to troubleshoot Session Loss issues in a Classic ASP application scenario. We need to keep in mind that there is no inbuilt mechanism by which users get redirected to login page. The possible causes by which a Session loss can occur in an ASP application could be

Application domain recycle

To find whether the issue is due to application domain recycle we can add code to Application_OnEnd.

Sub Application_OnEnd

Set fso = CreateObject("Scripting.FileSystemObject")

WriteFile ="C:\Inetpub\wwwroot\aspsessionevent\files\app.txt"

Set fs =fso.OpenTextFile(WriteFile,8)

fs.WriteLine "Application resetting..."

fs.WriteLine now()

fs.WriteLine

fs.Close

Set fs =nothing

Set fso= nothing

End Sub

The possible causes by which Application domain can get recycled are Antivirus touching any of the configuration files, any modification happened to Global.asa file, any setting in application pool as discussed before etc.

Ref: - https://support.microsoft.com/?id=316148

https://support.microsoft.com/?id=324772

Another issue which can cause Session loss is the Session cookie getting dropped by client.

Session State requires that the client browsers accept cookies. Are cookies enabled? If not, they will have to use cookie-less session state. In ASP.NET this can be done in the web.config file in the sessionState attribute. However in classic ASP, there is no such option, browsers will have to accept cookies for session state to work correctly with classic ASP.

Checking Cookie Settings in IE (Enabling Cookies):

Go to the Menu Item for: Tools, ->Internet Options -> Privacy -> Advanced -> Check "Override automatic cookie handling", select "Prompt" for both FirstParty cookies and Third party cookies, click OK.

Also Session cookie can get dropped if the number of cookies exceeds the default limit. Cookies are FIFO (first in - first out). Needless to say, ASPSessionID cookie is the first in, and when the limit is reached, it's also the first one to get discarded. This will result in a loss of session data.

Ref: - https://support.microsoft.com/kb/306070 .

We can enable the extend logging in IIS and can parse IIS logs using tools such as Microsoft Excel to see whether the Session cookie is getting dropped from client.

To enabled extended logging in IIS follow the steps.

Open IIS Manager – Web site properties – Website Tab- Enable Logging Properties – Advanced – Select Cookie Logging.

Using Microsoft Excel to Parse the IIS log files.

Open Microsoft Excel – Click on File Menu-Open - Point to IIS log file – Select Delimited in the Text Import Wizard—Click Next – Check Space in Next screen—Click Next –Finish.

Delete the first 3 unwanted rows. Select the #Fields –Right click Delete and select the Shift cells Left option to align the cells properly. Now check for the cs(Cookie) column to find the ASP.Net Session Cookie. We need to see the request from the beginning and see whether the Session cookie is getting dropped or not.

If you are running on a Web farm scenario there are possibilities that you will lose sessions when you use Cisco Local Director and SSL.

Ref: - https://support.microsoft.com/?id=251027

Other possible causes could be Third party ISAPI filters or custom code in application which is causing a redirection. Make sure that you check all the include pages to check for any redirection code.

Also if your application contains empty Global.asa Session variables do not persist, as IIS fails when it tries to parse the file.

Ref: - https://support.microsoft.com/?id=281298

Are you using framesets? Check the below two articles.

Session variables are lost when ASP is located in Frame set

https://support.microsoft.com/?id=178037

There is another article which explains about Session loss due to use of frames . This applies to both ASP and ASP.Net applications.

https://support.microsoft.com/?id=323752

Troubleshooting ASP.NET 1.1 & 2.0 Applications

Before troubleshooting we need to make sure that you have installed all the latest windows updates and Service packs available for .Net Framework.

To find application domain recycles in ASP.NET applications we can use tools such as Performance Monitor or can add logging code under Application_End.

Using Performance Monitor to find Application domain restart

Open Performance monitor—Add counters – Select Correct ASP.Net version in Performance object – Select Application Restarts—Click on Add & Close.

If the application domain gets recycled due to some reason you can see an increase in the counter.

Adding logging code under Application_End

To figure out the exact cause of what is triggering ASP.NET application domain restarts in a production shared hosted environment (was it a web.config file change, a global.asax change, an app_code directory change, a directory delete change, max-num-compilations reached quota, \bin directory change, etc). Just add the code in the below blog to Application_End.

https://weblogs.asp.net/scottgu/archive/2005/12/14/433194.aspx

The code works with both ASP.NET V1.1 and ASP.NET V2.0.

More information about Application shutdown reason can be found at the following msdn article

https://msdn2.microsoft.com/en-us/library/system.web.applicationshutdownreason.aspx

Troubleshooting Forms Authentication cookie loss

The best way to troubleshoot Forms authentication cookie loss is to enable extended logging as described before and check for ASP.Net FormAuth cookie. We need to keep in mind that the Sliding expiration works different incase of Forms authentication.

https://support.microsoft.com/kb/910439

Excerpt:-

If you use sliding expiration, the scenario is a bit more complicated. The cookie and the resulting ticket are updated if the user visits the site after the expiration time is half-expired. For example, you set an expiration of 20 minutes by using sliding expiration. A user visits the site at 2:00 PM, and the user receives a cookie that is set to expire at 2:20 PM. The expiration is only updated if the user visits the site after 2:10 PM. If the user visits the site at 2:09 PM, the ticket is not updated because half of the expiration time has not passed. If the user then waits 12 minutes, visiting the site at 2:21 PM, the ticket will be expired. The user is redirected to the login page.

We can also use tools such as Netmon or fiddler to capture the network traffic and check for form authentication cookie.

Using Fiddler to check Forms Authentication cookie.

Down load Network Monitor 3.1:- https://www.microsoft.com/downloads/details.aspx?familyid=18b1d59d-f4d8-4213-8d17-2f6dde7d7aac&displaylang=en

Download Fiddler: - https://www.fiddlertool.com/fiddler/

Note: - Fiddler doesn’t capture network traffic over Local host. So use machine name or FQDN while trying to access the Web site.

Troubleshooting ASP.NET 2.0 Applications

We can follow the same troubleshooting steps as described in ASP.NET 1.1 section but ASP.NET 2.0 offers some advanced features which makes our life much better.

The Web Events feature in ASP.NET 2.0

One of the major new services provided in ASP.NET 2.0 is the Web Events feature provided by the health monitoring system. By modifying the Web.config file, you can use the health monitoring system to log unhandled exceptions, expired forms, authentication tickets, and any other data you want logged in your application.

https://support.microsoft.com/?id=893664

Some of the other known issues with ASP.NET 2.0 that can contribute to session loss include,

Deleting ASP.NET 2.0 Application Sub-Directories Shuts Down the AppDomain

https://blogs.msdn.com/toddca/archive/2005/12/01/499144.aspx

Unhandled exceptions cause ASP.NET-based applications to unexpectedly quit in the .NET Framework 2.0

https://support.microsoft.com/?id=911816

Overwhelming Change Notifications

https://support.microsoft.com/?id=920970

Another common issue that can cause Session loss is Application domain domain recycle due to file change notifications.

https://support.microsoft.com/kb/911272/en-us

Hmm... I have tried to consolidate most of the stuffs that will help us in troubleshooting Session loss. If you need more help we are there at Microsoft PSS to assist you.

Happy Troubleshooting… :-)