setup procmon as a background task to run at a specific time and capture a specific process

avinash upadhyay 96

I want to capture some details around a w3wp process which triggers at a particular time and fails intermittently.
How can I setup procmon with a process filter to run independently (may be Task Scheduler ) at a specific time, capture the log and save it for future review. and then exit out?

1 answer

Michael Taylor 54,311 Reputation points

2021-07-12T14:11:56.807+00:00

You can schedule procman to run via Task Scheduler. The challenge is when. If you have a specific time then you can set that. If you don't know when then it becomes harder as you are limited in startup options.

To stop procman from running you can either set a run length limit in Task Scheduler or you can tell procmon to stop collecting after a certain interval or event size limit.

Note however that I don't believe using procmon is the correct solution here. The biggest issue is that you're talking about IIS. The process by itself isn't sufficient to identify the site being run. If you are like most other companies then your IIS is hosting more than 1 site. There will be no easy way using procmon to tell it to monitor just 1 site out of all of them. w3wp is the process IIS uses to run all sites and therefore each app pool is going to have its own instance. Within the app pool any related sites are going to be running. Depending upon your IIS configuration you may be monitoring the wrong process or more than just the site you care about.

Personally I think the better option is to use IIS's custom logging directly, your own custom logging within your app, AppInsights for on premise or the older IIS monitoring tools. These will allow you more fine grained control over the site(s) to monitor and they will better integrate with how web apps actually work.
Please sign in to rate this answer.
avinash upadhyay 96 Reputation points

2021-07-12T18:01:37.61+00:00

thank you @Michael Taylor , in my case I am hosting a single website, now there are multiple w3wp worker processes though, for various app pools. I want to capture a w3wp process(lets say via PID) for a particular app pool running two w3wp worker processes. Lets says I have identified a time to be able to replicate the issue, how can I setup the filter correctly (possibly arguments in task scheduler).

These are asp.net calls which are getting resource (ISAPI module) locked during an API call from another software. From the FRT logs I captured for 500 Internal server errors, I suspect McAfee is messing up with the asp.net calls and intermittently causes the 500 errors.

The application logs is not of much help, I have gone through that.
What all additional IIS logging can be enabled, and what specific old IIS monitoring tool are you referring to? could you please provide a specific tool which may be able to tell me what resource is getting locked?

Michael Taylor 54,311 Reputation points

2021-07-12T18:23:26.597+00:00

You cannot use the PID as it changes. The only way to identify it in procmon is by process name and as you already noticed there are multiples. Since procmon doesn't understand IIS it won't be able to distinguish between them and there aren't any process data points that would help it tell the difference anyway.

One possible workaround would be to use Powershell as the started process. Within PS have it get the running app pool that you care about by name using something like this (not tested).

$id = dir 'IIS:\AppPools\<poolname>\WorkerProcesses\' | Select-Object -expand processId

Assuming it was running you now have the PID. Then call procmon although I don't see any cmdline arg that allows a PID so I assume you'll have to dynamically create a filter that you can pass to the program to filter by PID.

But if you're having file locking issues then personally I would just first try excluding the IIS webroot from Mcafee. If the problem goes away you know. Alternatively have procmon monitor everything (otherwise it might miss Mcafee calls). Just having procmon monitor the site might tell you there is a file lock issue but won't tell you who else has locked it. Monitoring the folder that is having the issue and all processes would narrow down all processes trying to use the files in the folder at the given time.

avinash upadhyay 96 Reputation points

2021-07-13T15:10:18.49+00:00

This is the error that gets registered in the FRT trace for failed 500 errors.
Also please note that the issue is intermittent where 98% calls are successful and the rest fails. only way to get it running is to perform a IISRESET or app pool recycle.

MODULE_SET_RESPONSE_ERROR_STATUS

Warning-

ModuleName="IsapiModule", Notification="EXECUTE_REQUEST_HANDLER", HttpStatus="500", HttpReason="Internal Server Error", HttpSubStatus="0", ErrorCode="The requested resource is in use.
(0x800700aa)", ConfigExceptionInfo=""

Any clues on this? I am unsure how and where to check as to why this error.

Michael Taylor 54,311 Reputation points

2021-07-13T15:17:20.263+00:00

That's just a standard 500 server error which means an exception occurred. You need to get the exception details. Hopefully your web app is using global exception handling to capture these kinds of exceptions and you can go look at the actual exception message. The callstack will help tell you what is going on.

Did you exclude IIS from Mcafee to see if the problem goes away? Note that this would be dangerous if you are allowing your users to upload files. Problems with AV slowing down network requests I thought was solved years ago. If there is a slight delay then you could work around it using a delay in your processing or perhaps a retry. For example if you're trying to clean up files on your server then implementing a retry loop is sufficient. If you are having issues with files uploaded by users then a retry would also probably work.

But given that you have to do an IIS reset to recover leads me to believe you have a thread running that is crashing and not releasing its lock. Given this is .NET I'd lean toward it being an unmanaged thread such that the lock won't be released until the process terminates. The callstack on the exception should tell you this.

avinash upadhyay 96 Reputation points

2021-07-13T16:07:23.963+00:00

yes I did apply a rule to exclude McAfee scan for w3wp calls and also the folder being accessed, the intermittent issue still persists.

Due to the intermittent nature of the issue and the fact that McAfee can not be the culprit here, I also believe there is a w3wp process thread crash going on here.
Also the work flow for these API calls to this resource is defined in a way to ensure the calls are staggered across time. Still the issue persists.

Strange is Event viewer does not register a single entry for these failures and I don't understand why. The application log itself does not reveal much apart from the generic 500 error.
would a w3wp crash dump during the issue help?
can some elaborate logging be enabled at the IIS or asp.net level?

Michael Taylor 54,311 Reputation points

2021-07-13T16:58:34.1+00:00

If your app crashes or throws an exception then an event log entry should be generated and it'll have the basic information. I'm not aware of a situation in which it wouldn't unless you're silently eating the exception which seems unlikely.

If it isn't AV then then next thing I'd lean toward is a recycled app pool. You didn't mention it but unless the resource you're accessing is per-application then do note that it is possible for multiple instances of your app to be running at the same time. If you are accessing a single shared resource from multiple applications at the same time you could run into issues. Is that possible here?

A global exception filter would be my next step to debug this if you don't already have one. Dumping the process would only be useful if you happen to dump it when the exception occurs.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

setup procmon as a background task to run at a specific time and capture a specific process

1 answer

Your answer