David Paulson – Exchange https://docs.microsoft.com/archive/blogs/dpaulexchange/feed.xml Exchange, Exchange 2010, Exchange 2013, Exchange 2016, Performance Tue, 16 Apr 2019 06:03:57 GMT en-US hourly 1 Tips and Tricks: How to determine which w3wp.exe process belongs to which Application Pool https://docs.microsoft.com/archive/blogs/dpaulexchange/tips-and-tricks-how-to-determine-which-w3wp-exe-process-belongs-to-which-application-pool Wed, 15 Mar 2017 11:16:08 GMT https://blogs.technet.microsoft.com/dpaulexchange/?p=205 How to determine which Application Pool is which w3wp.exe process from Performance Data A lot of the time when you are looking at performance data, you are seeing the process w3wp.exe is the one that is causing the problem, however, you don't know which application pool is causing the issue from the data that was collect. Well, as long as you collect at least "\Process(*)\ID Process" and "\W3SVC_W3WP(*)\*" counters, you will be able to determine it rather quickly by following these steps.

Know which w3wp.exe process you want to be looking at (w3wp, w3wp#1,w3wp#2...)
Load up the counter "\Process(the process you want to look at)\ID Process"
- Check to make sure the PID doesn't change. If it does, then we need to zoom into the area where we were having an issue

The load up the counters "\W3SVC_W3WP(*)\Active Requests"
- It doesn't need to be Active Requests, but any counter under here will work as it is the instances that we are looking at.
Then find the number in front of the App Pool that matches the "\Process(the process you want to look at)\ID Process"

Now you can see that we are having an issue with MSExchangeServicesAppPool and are able to continue to troubleshoot down the path of why that service may be having issues. (I do apologize about the image not being the best for this one, I don't think I have the original data to try and get an updated version.) You can also use this method in reverse in order to see what a particular App Pool is doing for CPU usage during an issue as well. Either way you want to use it, it makes it so much easier to review performance data that was collected without having to collect PIDs from the server to line them up, as you may run into issues like the PID changing on you because the service is having that much of an issue it crashed or was restarted.

How to determine which Application Pool is which w3wp.exe process from on the server

Typically, most admins determine that they are running into a CPU issue by just logging onto the server and noticing that their CPU usage is extremely higher than normal. The same problem can occur here with trying to which w3wp.exe process is consuming the most CPU usage, as by default the detailed information in task manager doesn't provide which w3wp.exe process lines up with which Application Pool. So, in order to have a better understanding quickly as to which Application Pool is taking up the most CPU, you can follow these steps to determine this.

Open Task Manager, if you don't have it open already
Go to View then Select Columns...
Select the option "Command Line"
Go to the w3wp.exe process that you are wanting to determine what Application Pool it belongs to, then look at the data provided in the column "Command Line"

From this information, you should be able to determine which w3wp.exe process is running which Application and can continue troubleshooting from there.]]> Tips and Tricks: How to determine how many connections are coming into each protocol on a server https://docs.microsoft.com/archive/blogs/dpaulexchange/tips-and-tricks-how-to-determine-how-many-connections-are-coming-into-each-protocol-on-a-server Wed, 15 Mar 2017 11:15:53 GMT https://blogs.technet.microsoft.com/dpaulexchange/?p=245

In the image above, we can see that we are looking at two different Instances, "_LM_W3SVC_1_ROOT_mapi" and "_LM_W3SVC_2_ROOT_mapi_emsmdb". Whenever you are looking at the Instances in this counter, the number value correlates to the Site that this Application is associated to. Typically, the protocol is going to be the same in Site 1 and in Site 2, but MAPI is a little different and we hit a different naming convention in the backend or mailbox role of Exchange. Here is a snippet of the instances that you should see in your performance data for Exchange.

Keep in mind that there can be some customization done to Exchange to have added a 3rd Site so you could see a number "3" in the mix as well. So keep an eye out for that as a possible cause of an issue with load balancing as well. From this set of counters of data, we can properly determine how many connections we are executing on each protocol. This allows us to determine if there was a large increase or decrease of requests and we can go from there to determine why that might have occurred. These set of counters are a better use to determine load balancing from a frontend connection as well as a backend connection. Normally admins will try to keep an equal number of users across on the databases, which is typically good till you mix in extreme users. For example, if you have users that are connecting to multiple mailboxes for a different reason, or better yet you have all your modern public folders on 1 server with normal users, we can see a dramatic difference in connection count on the backend connection of the server. The number of requests executing is going to give you a better representation of how busy this server is compared to another. Now, from this set of data, you can only determine if we all of a sudden had a spike or dip in Executing Requests of this server. So that doesn't provide us useful information of load balancing of the environment until we add preferably all the servers that are behind the NLB to the view to determine if there is a major difference in connections to each server. You can do this by adding the performance data from each server during the same time frame and pulling them up in the same view. When wanting to add the other server, you just need to select the drop down under "Select counters from computer:" and select the computer that you want. Then you can proceed to add the same set of counters and then compare them.

From this image, you can see that server number 5 is averaging about 7,000 for the frontend connection from the NLB and the server number 7 is only averaging about 5,000. This is a large difference that could be adding to the additional load on server 5 in comparison to server 7. Keep in mind that the CPU usage on a frontend connection should be extremely small in comparison to the mailbox role. However, even with lower CPU usage in a good environment, we shouldn't see about 2,000 difference in connections if the NLB is configured to evenly distribute the load. You can also take the same concept from above and apply it to the backend connections that are getting proxied to this server from different front ends. This will allow you to see if you are evenly distributed between all the server nodes. One key concept to keep in mind when looking at data is that you need to understand that these counters can be different from environment to environment, as to what we should be seeing and where a server can't handle any more of these types of connections. The limit values are going to be factored based on the mailbox profiles and the benchmarks of the server's performance capabilities as well as many other factors that can consume resources on the server. This is yet another reason why you need to be looking at performance data from multiple servers if possible even servers that aren't seeing the issue that is under the same load. By collecting this data, you can get a better analysis done by being able to do the comparison between servers.]]> How to troubleshoot Exchange Performance Issues at a basic level (Part 2) https://docs.microsoft.com/archive/blogs/dpaulexchange/how-to-troubleshoot-exchange-performance-issues-at-a-basic-level-part-2 Wed, 15 Mar 2017 11:15:37 GMT https://blogs.technet.microsoft.com/dpaulexchange/?p=46 part 1, I went over information that you needed to collect when you are suspecting an Exchange Performance issue within the environment. In this part, I will be going into depth of the Key Pointers when looking over the Data.

Knowing the Min, Max, and Average for a duration and an interval that matters

The reason why you need to know what your Min, Max, and Average are because this data can easily be skewed if you are not looking at a reasonable duration and timeframe for that capture. What I mean by this is that we typically try to capture performance data of about 8 hours during production hours, but we might only be seeing issues during part of the day. We don't care what the average is for the whole 8 hours of the capture, but smaller chunks of data when the issue was occurring. In order to get a good understanding of what is going on in those smaller timeframe windows, we need to have a lot of data points to paint a better picture. This is why we recommend collecting performance data with at most a 5-second interval. What I would suggest is a good average timeframe to get a better understanding of an issue is going to be from 5 - 20 minutes. Keep in mind that the timeframe window that you want to look at can dramatically change from issue to issue depending on the counter and how much higher than the recommended threshold value that we provide for that counter. Here is an example of what I am talking about. One of the biggest problems we had with Exchange 2013 was with Garage Collection (GC) within .NET. GC is basically .NET freeing up memory within a process. In order to identify this issue, we use the counter "\.NET CLR Memory(process)\% Time in GC" to see if the average is above our threshold of 10. Here is part of a data collection that we got from a server having an issue and we were checking for GC issues for a w3wp.exe process.

Here you can see that we have a Duration of 50 minutes for the file, and the average is below our threshold so this means that we are not hitting an issue, right? Well no, that is not the case this is a 50-minute capture and we can see the GC causing issues at random times, so we want to zoom into a more reasonable time frame to see if we go above this threshold. So, if we zoom into this area that I have highlighted below, we should get a better idea if we are seeing this issue.

Now we can see the below image what our average is for that timeframe now.

As you can see, we are hitting an average of 18.494 during a 14-minute duration. This could easily provide issues that you may be seeing with Exchange 2013 and you should install .NET Framework 4.5.2 if you don't have it already installed. For this issue, we installed .NET 4.5.2 and it fixed the issue that we were seeing.

Note: You should always have the latest version of .NET installed that is supported by the build that you are currently running for Exchange 2013/2016. Note: This type of issue was more common with the earlier releases of Exchange 2013 prior to .NET 4.5.2 was ever released.

Another example of why it is important to look at your Min and Max for the whole duration is to see if we are dramatically above or below the threshold for that counter. You can't always depend on the graphical view that you are seeing within Performance Monitor to see if we actually hit that threshold or not. For example, we don't want to see the counter "\Memory\Available Mbytes" ever hit 0. We would like to see it above 5% of the available physical RAM on the server, if possible. However, what you really don't want to see is it dipping below 1GB as you are causing the system to work extra hard to try and free up memory from processes. In this view of my collection, this is what you see.

As you can see we hit 0 for a Min, but in the graphical view, we don't ever see the line even come close to the bottom. So, is this accurate? Most of the time yes, I have seen some issues if the data has been stopped incorrectly or has gone through relog.exe where the min is 0 and isn't correct but typically those data points are at the begin or end of the file. But most of the time it is correct, you just need to zoom into the lowest point of the line to see the issue. So if you zoom into this section of the data.

You now see this.

So now you can clearly see the line hit 0. Because of the long duration of the capture and the multiple data points in this collection, we can't display them all to the correct location that is visible to the eye without zooming in. This also brings up a good reason why to keep the intervals low, like at 5 seconds. If we were to zoom in more yet to this area, this is what you would see.

As you can see from all the data points, we are collecting this data at a 5-second interval. Because we are doing this, we are able to see that we do in fact reach a value of 0 for this counter. If we were to have a 1-minute interval capture, like what is default in the Exchange 2013/2016 data collector sets, it would just show that we got really low and only dipped slightly below the 1GB threshold limit. This is why we want to collect data at a 5-second interval so we can see issues like this. This is also why it is critical to also collect Experfwiz data and not fully rely on the default data collector sets in these versions of Exchange for complex issues.

Knowing what your scale factor of your counter is set to

Knowing what your scale factor is rather important when you are dealing with counters that can dramatically vary in their values and when you are comparing them. So, the most common time I am aware of this is when I am looking at processes for the % Processor Time and Memory counters for those processes. Here is an example of what I am talking about.

From this data, we can see that we have 3 processes showing within this view. Based on the lines in the graph, you would think that they are consuming about the same amount of resources, right? Wrong, they aren't using the same amount of resources as they are all at different scale factors. In this view, we have 1 counter at a scale factor of .1, another at 1, and another at 100. The one at .1 is consuming the most during this view of the performance data that we are looking at. Providing that you used the "Scale Selected Counters" feature in Performance Monitor, that is how you can get different scale factors for the same counter. What that means is when you scale a counter to size (Scale Selected Counters), it will automatically calculate this based on the Max value that is in your current view of the data. So, if you have a crazy high data point within your view, your data is going to look like it just has a little spike vs a major spike if the correct scale value isn't used. Here is an example of what I am talking about.

In this example, we can see that we just have a little spike on the "Avg Disk sec/Read" counter. At first, you think this is an okay spike because it maybe hits 20ms based on this view. But if you look at the Max value, you can see that we are really hitting 717ms during this spike. Since our scale factor is at 100 it really skews with our view of this counter. So, if we set it to 1,000 we can see that this is what we are looking at now - we set this to 1,000 because there are 1,000 milliseconds (ms) in a Second.

As you can see this stands out more than the other scale to what we are actually seeing. This is why it is important to be looking at the numbers as well as the graph when looking at performance data. This spike is so high (717ms) that it should really be noted and addressed by the storage team/vendor to understand why this I/O latency issue occurring. This should just be noted and isn't cause for extreme alarm (again depending on the issue that you are troubleshooting), as this isn't likely the cause to your full issue but it is an indicator that we might be pushing our disks at the time that we see that spike occur and we should never see a spike this high on an Exchange Server.

Knowing what we recommend for a threshold of that counter

Keep in mind that you must know which counter that you are looking at in order to determine if it is normal or within our threshold values. Also, keep in mind that you can't have a spike that is extremely crazy for a critical counter. For example, "Avg Disk Read/Write\sec" has always been 20ms or lower for an average. However, if you are seeing spikes up to 500ms, that can be extremely bad depending on multiple factors. It is also extremely bad to have some spikes of averages that are maintained above our recommended average, for an extended period of time. What I mean by this is similar to by the previous post regarding the .NET GC issue, we don't want to see a period of time that is greater than our threshold. The higher the maintained spike, the less time that you want to be above that. For example with the "Avg Disk Read\sec" counter, I would be a concern if we maintained a 100ms average spike for about 60 seconds. I would consider that to be poor disk I/O latency, or getting to the point where we are having some issues keeping up with what Exchange needs. This is something that we need to keep in mind when we are seeing issues like this. Also, speaking strictly to the "Avg. Disk Read/Write\sec" counters, just because we see high spikes or averages on a particular drive, this doesn't mean that the disk(s) are bad. What this does mean, however, from the OS level prior to getting back to the application layer, we are seeing this as the Avg. latency for those I/O requests for that second. This could mean an issue with the drive, controllers, or just the path to the drive is where we are actually experiencing the latency. In a later post, I am hoping to get into this with a little more detail. Here is a list of links for our guidance of thresholds for some of the counters that we offer in Exchange, and you can use the Exchange 2013 counters for 2016. Exchange 2013 Performance Counters https://technet.microsoft.com/en-us/library/dn904093(v=exchg.150).aspx Performance and Scalability Counters Thresholds - 2010 https://technet.microsoft.com/en-us/library/dd335215(v=exchg.141).aspx Monitoring Without System Center Operations Manager - 2007 https://technet.microsoft.com/en-us/library/bb201720(v=exchg.80).aspx

How to determine which Web Application Pool is which w3wp.exe process

For information on how to determine this from both the performance data and live on the server, check out my Tips and Tricks blog post that I wrote "How to determine which w3wp.exe process belongs to which Application Pool"

How to determine if you are in a balanced load from performance counters

For information on how to properly determine if you are evenly balanced for your connections on both the frontend connections and backend connections, check out my Tips and Tracks blog post that I wrote "How to determine how many connections are coming into each protocol on a server"]]> How to troubleshoot Exchange Performance Issues at a basic level (Part 1) https://docs.microsoft.com/archive/blogs/dpaulexchange/how-to-troubleshoot-exchange-performance-issues-at-a-basic-level-part-1 Wed, 15 Mar 2017 11:15:23 GMT https://blogs.technet.microsoft.com/dpaulexchange/?p=35 What we need to know

What version of Exchange?
What are the symptoms?
- Don't just provide high CPU, provide details like the Avg. high with the process that is consuming the most with what it is at like below.
  - Example: Total CPU 95% - 100%, store.exe ~80%
What OS is Exchange running on?
Does this occur all the time?
Does this occur on all the servers?
What have you changed recently? (This includes if you are in the middle of a migration. More mailboxes, more load)
What 3rd party applications are you using? (backups, AV, Skype/Lync - even though it isn't 3rd party it still adds additional load to the servers, and users with multiple devices that use EAS, BES, or GOOD are not included in the sizing calculator etc...)
Are clients in online or cached mode (provide a guess if possible)

Exchange 2013 and newer we need to get additional information

What is the version of .NET installed?
Are the servers All in One (AIO)? - Only with Exchange 2013
What CU are we on, and are we on the same build for all the servers? (2013 and try to keep the servers in N - 1 for the builds)
How many users are on the servers? Per Database?
How many active and passive databases do you have? (Lot of cases that we have seen in the past is when we are undersized so this gets to be important later on)

Note: Most of the Exchange 2013 and newer additional and default information can be collected with our Health Checker Script, which you can download here.

If the Exchange Environment is virtualized, here are some additional questions that should be answered ASAP when dealing with performance issues.

Do we have dedicated resources for Exchange (Memory/CPU)?
What is the Processor of the host machine and are there multiple?
How much memory is installed on the host machine?
How many logical cores do we have presented on the host?
How many physical cores do we have presented on the host?
How many cores are assigned out to each guest machine, that is currently running?

Note: If you are seeing high CPU utilization on the Exchange guest machine, like 95-100%, the issue is likely not with the host or its configuration. Reason being is we are getting all the resources that we are allowed extremely quick that is allowing us to actually see an extremely high CPU usage on the guest machine. Note: If you have Exchange virtualized and is acting extremely slow within normal RDP usage and other funcationality but aren't seeing a High CPU usage, begin to question the host configuration as it may not be setup correctly.

What to do when you first think the issue might be related to Performance

We need to have performance data to troubleshoot a performance issue, and the quicker we get it the better. By collecting performance data, it provides evidence of the issue that is occurring and history of data that we can later go back to and compare against. I personally like to provide a comparison of before and after to show how much we have improved after making a change to the environment, other than relying on the end user's feedback to a problem. When collecting performance data, we should start to collect it from all the affected servers if possible or at least from about 4 servers to allow a proper analysis to occur. If possible, we should also collect from some servers that do not have the issue as well, that are under about the same amount of load. By collecting the performance data from the other servers that are not having the issue, we are able to use them as a baseline during the same time period to compare against the affected servers to allow a better analysis to occur. So, we should start off by running the following command with the newest Experfwiz, even if it doesn't say stable always run the newest unless you run into issues. Reason being is typically the newest one has all the latest counters added to it that we utilize for analysis. $location = "C:\location_of_performance_logs\" .\experfwiz.ps1 -filepath $location -interval 2 -ExMon -ExMonDuration 00:20:00; sleep 1260; .\experfwiz.ps1 -stop; .\experfwiz.ps1 -filepath $location -interval 5 -quiet

Note: The Exmon switches may not be needed in every scenario, but it is better to include them to rule out a single user or a group of users causing the issue vs needing to go back and collect that data again when the issue does occur to rule out that possibility. Note: With the experfwiz script we have the option to add -Threads. This is typically not needed to troubleshoot a performance issue right off the bat. It provides a lot of additional information that will add more I/O overhead to write this out and increase the amount of blg files. This information is also more useful with additional debugging information that is being collected synchronously.

By running this single cmdlet in Exchange Management Shell (EMS), it will automatically stop after 21 minutes and start up a new collection with the 5-second interval for the default time of 8 hours. Which may not be needed but it still allows us to collect data while we are looking into the issue. You should do this so you don't have to worry about stopping the collection after 21 minutes and starting up a different one. By collecting a smaller chunk of performance data when the issue is occurring, it will allow us to begin our analysis to get a better understanding of what is going on with the system. While we are waiting for the 21 minutes to be up, we should still be utilizing the time by either looking at performance data live, or application and system logs, and collection additional information about the environment. However, we still want to continue collecting the performance data as the 21 minutes is not enough to properly determine what is going on with the system. Keep in mind that this also might not provide the most accurate information, as we could be looking at sub-symptoms that are a side effect of what occurred prior. The best source of data to determine what is causing the performance issue is performance issue leading up to the start of the issue plus 20 minutes into it. With Exchange 2013, we can get a better idea of what is leading up to this initial issue if we also look at the daily performance logs.

Note: There are articles regarding how to disable the daily performance logs on a system, do not do this. The best source of information on getting a Root Cause Analysis of an issue that is no longer occurring are these logs along with some other default logging that is enabled.

How to look at the raw data

In this section, we are going to determine the best way to read the data that is collected with experfwiz, as it is the best way to understand everything that is going on with the server all at once. We could use the tool PAL, but you still need to understand how to read it properly in order to make sense of it and look over counters in comparison at the same time.

How to open a BLG file faster

Most people would typically just double click on the .blg file and wait for it to open up in Performance Monitor, and this process takes forever to load when you are trying to open files greater than 512MB. There is a faster way to look over the data and here are the steps:

Open Performance Monitor
Under Monitoring Tools, select Performance Monitor
Then select the Action tab at the top, then select the Properties option

Then in the new window, select the Source tab
Under Data Source select the Log files option

Browse to the location of your performance file and click Apply

By following these steps you will be able to open up a .blg file much faster than just trying to double click on it, and it also allows you to add multiple files to show you a much bigger time frame. This is great if you have multiple files because of the roll over on the max size that you have from the Experfwiz or are just looking at multiple servers.

Key pointers when looking over the data

When looking at the performance data, there are some factors that really matter when you are reviewing it.

Knowing the Min, Max, and Average for a duration and an interval that matters
Knowing what your scale factor of your counter is set to
Knowing what we recommend for a threshold of that counter
How to determine which Application Pool is which w3wp.exe process
How to determine if you are in a balanced load from performance counters

For the next part 2 of this blog post, please continue by clicking here. ]]> EHLO World https://docs.microsoft.com/archive/blogs/dpaulexchange/ehlo-world Thu, 05 Jan 2017 12:32:37 GMT https://blogs.technet.microsoft.com/dpaulexchange/?p=25 Hello, my name is David Paulson and I have been working with Exchange for Microsoft starting as a vendor back in Sept 2012, then I was later hired on as a FTE in June of 2013. I love working with Exchange and the challenges that it brings every day with the constant learning that I am able to do on a daily basis. I am starting up this blog because of some the complex cases that we do get where we learn something interesting, that should be shared with everyone to help have a better understanding of Exchange and everything that we rely on in order to have a good performing environment. A good amount the cases that we do see coming in revolve a little around performance, which is why I am a Performance Subject Matter Expert (SME) in this area. Most of these cases are challenging and can be very complex and time consuming because of the amount of data that you tend to need to look through, if you are doing it correctly. So most of my post will evolve around performance type work, but I will also post some content around Exchange in general or PowerShell scripting tips that I have learned along the way. Hopefully you find my blog post useful and will continue to follow them in the future. If there is content that you would also like, please feel free to comment below. ]]>