How to fix IIS Worker Process memory spike issue

Question

How to fix IIS Worker Process memory spike issue

Wang, Davis@CDCR 0

Hello. Users are currently experiencing huge delays with our application. It seems like the IIS Worker Process is consuming a lot of resources on the application server, 2000MB at times. Is there a way to mitigate this on the server side? Thanks in advance.

Lex Li (Microsoft) 6,022 Reputation points Microsoft Employee

2024-02-17T04:40:29.4566667+00:00

High memory usage and slow responses often come together, so if you want to learn the root cause then you need to involve the web app developers. They might use the proper tools to profile the memory usage and performance metrics, and tell what's wrong. If you want the server to remain usable before the root cause is found, the advice from @Michael Taylor below to configure application pool recycle might buy you some time.
Benjamin Fallar III 11 Reputation points

2024-02-17T04:57:18.71+00:00

High memory usage is usually caused by code that might produce memory leak. So try to perform code review and assess your application runtime by using tools that can detect and diagnose for memory leaks. Next, you can also consider hosting your application in a separate IIS application pool to isolate the security and runtime environment of your application if your are sharing with other applications. It could be possible that if you are using a shared IIS application pool, the other application can indirectly affect your application performance.

1 answer

Your answer

Lex Li (Microsoft) 6,022 Reputation points Microsoft Employee

2024-02-17T04:40:29.4566667+00:00

High memory usage and slow responses often come together, so if you want to learn the root cause then you need to involve the web app developers. They might use the proper tools to profile the memory usage and performance metrics, and tell what's wrong. If you want the server to remain usable before the root cause is found, the advice from @Michael Taylor below to configure application pool recycle might buy you some time.
Benjamin Fallar III 11 Reputation points

2024-02-17T04:57:18.71+00:00

High memory usage is usually caused by code that might produce memory leak. So try to perform code review and assess your application runtime by using tools that can detect and diagnose for memory leaks. Next, you can also consider hosting your application in a separate IIS application pool to isolate the security and runtime environment of your application if your are sharing with other applications. It could be possible that if you are using a shared IIS application pool, the other application can indirectly affect your application performance.

Answer 1

Michael Taylor 57,216

If one of the web apps you're running, which are the worker processes, is eating up memory then that is something the app needs to fix. You can use a tool like Process Explorer to figure out which app is actually using up the memory and where it is using the memory. From there you can use standard web debugging to see more details that are specific to the app.

If you need quick mitigation such that one bad app doesn't eat up all the memory on your server then you have a couple of options. The best option would be to change the app pool settings for that process.

Open IIS Manager and go to the app pools.
Right click the app pool hosting the bad worker process and select Advanced Settings.
Under Recycling set the limit for how much memory can be allocated (generally virtual) before the app pool recycles. You'll want to ensure you configure the logging to show these events so you can more easily track them.

If you set a recycling limit then when memory gets to that limit new requests will trigger a new worker process and once the old process has handled all its outstanding requests it'll get shut down. This is best for processes that are either leaking memory or failing to release memory fast enough.

If the process just keeps allocating memory while handling requests then setting a limit won't help most likely. In that case you may need to switch the app to x86 mode (General \ Enable 32-bit Applications) temporarily. This will put a fixed limit of 2GB on the process. Any attempts to exceed that cause an error that brings down the process. For this to work your web app will need to support running in x86 mode (many do provided they don't rely on native components) and this will likely result in more errors seen by the end user. But as a quick and dirty solution until the problem can get resolved this may be good enough.

I would start with limiting memory and see if that fixes the problem until the real issue can get resolved and escalate to x86 only if necessary.

Lex Li (Microsoft) 6,022 Reputation points Microsoft Employee

2024-02-17T04:35:26.2366667+00:00

This is a post with several red flags. 1) I don't think Process Explorer provides any help here. 2) "standard web debugging" is ambiguous so what were you referring to? 3) "you may need to switch the app to x86 mode" looks like a terrible suggestion. If this is already a web app suffering high memory consumption (and you suggested using application pool recycle as workaround), then limiting its memory usage to 2-GB (actually at about 1.5-GB) only makes the things worse.
Michael Taylor 57,216 Reputation points

2024-02-17T07:35:11.43+00:00
@Lex Li sorry you feel that way but every one of these suggestions are useful in the right situation. We use these techniques to help narrow down problem sites on the servers we manage and have for many years.

Process explorer can be used to quickly identify what app pool is associated with the runaway w3p process. Furthermore if you can catch the app increasing memory then it also gives you access to the threads and stack trace to help narrow down what might be using up memory if you don't otherwise have debugging tools on the machine.

Standard web debugging is the standard techniques used by web devs to debug problems like this and include logging, perf counters, etc. Anybody with web dev experience has a standard set of debugging tools so repeating them here seems redundant.

Also disagree that this is a terrible idea. We have had to use it in one case. It completely depends on the memory usage problem but if you have a limited amount of memory on the server and a web app is leaking memory or eating up all the memory then forcing a limit defined by x86 is a stopgap to prevent it from bringing down other apps until you can resolve it. We had to do this exact thing when a download webapp improperly allocated memory to download files resulting in other apps failing. Throttling is the better option, as I said, but flipping it to x86 is a stopgap. Just because you've never seen this useful doesn't mean it is a bad idea. It doesn't make things worse unless the web app just spins up another instance and starts again which is why I said it can cause more errors for the end users but more errors for an already broken app vs impacting other apps is a pretty obvious tradeoff.
Lex Li (Microsoft) 6,022 Reputation points Microsoft Employee

2024-02-18T02:55:30.8166667+00:00

Can you explain how threads and call stacks in Process Explorer can help analyze memory usage? That sounds like an interesting approach.

Similarly you mentioned logging, but did you write log entries for every objects a web app created through its lifecycle to track down memory issues?

You might have used the approach for years, but they are not well recognized for memory analysis. Articles like this one list some of the most efficient tools Microsoft recommends, which are also the tools Microsoft support team use to help resolve issues raised by customers. For example, appcmd list wp is commonly used to see relationship between application pools and worker processes, instead of Process Explorer.
Michael Taylor 57,216 Reputation points

2024-02-18T19:22:06.16+00:00

If you have additional solutions for the OP then feel free to share them with the OP. However posting a comment saying other suggestions are bad is disrespectful and unhelpful. Just because you don't find them useful in your experience doesn't mean anything. I have seen MS's recommendation for debugging and I've honestly rarely found it useful. Our experiences differ.

I'm not going to comment further to you about this as it in no way helps the OP which is the whole point of this forum. I will continue to help the OP in the manner I feel is most helpful and you can continue to do the same if you like.

Share via

How to fix IIS Worker Process memory spike issue

1 answer

Your answer