Team Foundation Server Team Project Limits
Bill Essary, Software Architect
Microsoft Corporation
November 2006
Applies to:
Microsoft Visual Studio 2005 Team Foundation Server
Summary: This article describes the new team project limit and monitoring recommendations for Visual Studio 2005 Team Foundation Server. (6 printed pages)
Download the sample Microsoft Excel spreadsheet used to calculate team project limits. (25 KB)
Contents
Introduction
Factors Influencing Team Project Limits
Team Project Limits for MSF Work Item Types
Team Project Limits for Custom Work Item Types
Managing a Team Foundation Server That Is Almost at the Team Project Limit
Conclusion
Introduction
Microsoft has updated its guidance for Visual Studio Team Foundation Server team project limits. This article describes the new team project limit recommendations, the factors that influence team project limits, the guidelines for estimating project limits, and recommendations for monitoring and managing team project limits and related data.
This article is intended for system administrators and system operators who deploy and maintain Team Foundation Server installations.
New Team Project Limits
The new recommended limits for team projects based on the MSF for CMMI Process Improvement template is 250 team projects. The new recommended limit for projects that are based on the MSF for Agile Software Development template is 500 team projects. The recommended limits for projects that use custom work item types are based on Microsoft test data.
For specific guidance, see Team Project Limits for MSF Work Item Types and Team Project Limits for Custom Work Item Types.
New Operational Guidance
This article explains the factors that influence team project limits in a Team Foundation Server deployment and provides guidelines that you can use to monitor a Team Foundation Server installation.
The team project limit for a Team Foundation Server can be expressed as a limit on the size of the work item tracking metadata cache. On a client computer, the size of the metadata cache is the size of the subdirectory that holds all of the cache files from the server. The work item tracking metadata cache size is the same for all client computers that have recently connected to the server. You should monitor the metadata cache size, and if a server is near the team project limit, make sure that client computers are brought online in small groups. You should also consider increasing the average amount of RAM in client computers to 1 GB. If most client computers are local to the server, this can help increase the size of the groups that can be brought online simultaneously. However, the increase in RAM does not change the team project limit itself.
Factors Influencing Team Project Limits
The team project limit is a soft limit based on resource use. However, overall system performance does not deteriorate gradually as the number of team projects increases. This section describes the key issues to consider and the symptoms that you might see when your Team Foundation Server approaches its team project limit.
As the number of team projects grows, the main resources that influence performance are available memory in the client computers and the network speed between the client and server. Resource consumption is a significant concern only when a server is communicating with newly connected client computers. A server that handles many team projects that were created over time does not show significant performance degradation when it updates client computers that connect to that server regularly. However, client computers that connect to a server for the first time or for the first time after many team projects have been created require the server to download more data and use a large amount of memory to process the data. For a client computer that has little available RAM or a slow connection to the server, the result can be a one-time delay that can last for minutes as Team Explorer tries to expand the Team Project node. The server can also become a bottleneck during the initial connection period if it must add a large number of new client computers at the same time.
The size of the work item tracking metadata cache affects the performance of a system that hosts many team projects. The number and complexity of work item type definitions used in team projects are the dominant factors that can increase metadata cache size. (There are other system factors — such as the total number of users and groups — that also contribute to metadata cache size.)
Work item type complexity is difficult to measure. It is a combination of the number of fields in a work item type, the number and complexity of the rules, the size of the allowed values lists, and all of the other data found in the definition. When work item types that are used to create team projects are constant (because all projects use the same process template), the metadata cache increases at the same rate as the number of team projects on the server. However, server performance decreases as a stair step function that follows the underlying linear trend. The metadata cache that a client computer downloads holds data for all work item types in all team projects hosted on the server, regardless of the number of team projects actually used by that client. After it downloads the cache for the first time, the client regularly exchanges metadata with the server to keep the cache up-to-date. If you have a large number of team projects and you add new client computers, server performance is affected primarily during the initial download of the metadata cache.
As soon as a client computer has a near-current cache, the download time and corresponding performance load imposed by a large number of team projects is relatively small at both the client and the server. For example, if you connect a 512 MB client computer with a near-current cache to a server that has one team project based on MSF for CMMI Process Improvement or to a server that has 200 team projects based on MSF for CMMI Process Improvement, there is only a one second difference (approximately) in startup time. However, the first time that you connect the same client computer to the same servers, it takes approximately 43 seconds longer to download the cache from the server with 200 team projects based on MSF for CMMI Process Improvement. A remote client that has a slow connection to the server incurs additional data transfer time. A client computer that has 1 GB of RAM or more has a significant decrease in the initial connection time because a large part of the 43 second difference is due to paging.
If many new client computers simultaneously request metadata from the server, the server can consume enough memory to trigger the default Internet Information Server (IIS) memory limits. If this occurs, the IIS will restart and then resume processing metadata cache requests. However, the system will experience significant response degradation if a large number of client computers continue to request data. This can occur if many team projects are created on a server in a short time and client computers cannot slowly collect metadata, or if a field in a work item type definition is deleted on a production server (this is a rare occurrence). Typically, a server experiences many new client computers trying to connect only after the server is restored from backup media.
Team Project Limits for MSF Work Item Types
If you use work item types using one of the MSF templates that are included with Team Foundation Server, use the following recommendations when you establish the team project limits:
- For team projects based on the MSF for CMMI Process Improvement template, the recommended metadata cache limit is 250 team projects.
- For team projects based on the MSF for Agile Software Development template, the recommended metadata cache limit is 500 team projects.
On the Team Foundation Server data tier, you can monitor the TFSWorkItemTracking.dbo.Rules data space size to determine the approximate metadata cache size and team project limits. The Rules table data space size is indirectly related to the metadata cache size, but is easier to monitor on most systems. The recommended cache limit is roughly equivalent to a 30 MB rules table data space size. Refer to the "Team Project Limit" section of the Team Foundation Server Administrator's Guide if the data space size reaches 24 MB (80 percent of the maximum).
To determine the Rules table size:
- Start SQL Server Management Studio, and connect to the Team Foundation Server data-tier computer.
- Expand the Databases node, and find the TfsWorkItem database.
- Expand the Tables folder of the TfsWorkItem node.
- Select the dbo.Rules table, and right-click to view its properties.
- Obtain the Data space number from the General properties page.
Team Project Limits for Custom Work Item Types
If you use custom work item types, you can use the following detailed experimental data to estimate the work item tracking client metadata cache size. You can then use the cache size to establish the team project limit for your system.
Microsoft used a 500 team project test to help predict the metadata cache size for systems with similar team project profiles. Figure 1 shows the limits for team projects in the system. The color coding indicates safe (green), precarious (yellow), and excessive (red) project levels. If you consider the key factors involved (client memory, remote connection bandwidth, and server memory profile under load), the safe threshold for the work item tracking metadata cache for 500 team projects is 65 MB.
Figure 1. Custom client metadata cache limit
The experimental data is available in a Microsoft Excel spreadsheet (see the download at the top of this article). You can use the spreadsheet to estimate the maximum number of team projects per server when you use custom work item types. To use the spreadsheet, update the cells highlighted in purple with data from a server that hosts team projects that you created by using a process template with the custom work item types.
To determine the Rules table size:
- Start SQL Server Management Studio, and connect to the Team Foundation Server data-tier computer.
- Expand the Databases node, and find the TfsWorkItem database.
- Expand the Tables folder of the TfsWorkItem node.
- Select the dbo.Rules table, and right-click to view its properties.
- Obtain the Data space number from the General properties page.
Perform this sizing process on a clean test system. You can obtain the best estimate by measuring the size of the Rules table after you create one team project of the target type, and then measure again after you create additional team projects. In the example shown in Figure 1, nine additional team projects were used for the second measurement. The model predicted the work item tracking metadata cache size well in an actual sizing experiment with hundreds of projects. If you do not have an available clean system, use a zero for the estimated size of the Rules table for one team project to generate a conservative team project limit.
The size calculations produced by this model are merely estimates, and you should use them as guidelines only. There are several factors that affect the quality of the predictions. The most significant is that the model assumes that all of the team projects on the system will use the same process template. If this is not true, you must complete additional modeling to generate a reasonable estimate. There are also system factors — such as the number of users or groups — that can influence the growth of the metadata cache. Finally, to account for system performance degradation, the model uses an analog measure of metadata cache size to predict the size of a number that increases in a stair-step manner.
**Note **When you determine the acceptable metadata cache size limit for your system, you should also consider the percentage of remote users in the system and the bandwidth of the connection to those users.
Managing a Team Foundation Server That Is Almost at the Team Project Limit
When you manage a Team Foundation Server, use the team project limit as a guideline, monitor the size of the work item tracking metadata cache, and consider the factors that affect how a system behaves when it approaches the team project limit. (A system is considered to be almost at its project limit when the metadata cache size is 80 percent of the recommended safe threshold.)
To measure the work item tracking metadata cache size:
- On any client computer in the system, start Microsoft Windows Explorer and locate the \Documents and Settings\<user>\Local Settings\Application Data\Microsoft\Team Foundation\1.0\Cache folder.
- Right-click a subdirectory that has a Team Foundation Server GUID as the name, select Properties, and read the size in the Properties dialog box.
If a client connects to more than one server, you might need to map the server GUID to a computer name. The easiest way to do this is to open the ServerMap file in the cache folder.
Take the following steps to manage a system that is running near the team project limit:
Bring new client computers online in groups.
When it connects for the first time, a client computer downloads the complete metadata cache. If your system has hundreds (or thousands) of client computers connecting for the first time, the initial connection could overwhelm the server. If the metadata cache size is 80 percent of the recommended safe threshold and the server must process more than approximately 20 simultaneous requests for work item tracking metadata, the server can experience significant memory pressure. This can trigger the default IIS memory threshold. To help avoid this problem, after any event that invalidates the work item tracking metadata cache, such as restoring a system from backups, bring the client computers back online in groups of 100 or less.
Increase the minimum amount of RAM in client computers to 1 GB.
A RAM increase can significantly improve the performance of client computers during the initial connection. This can also have a positive effect on the server because the server spends less time processing requests from individual client computers. As explained previously, when a client computer initially requests and processes metadata, it consumes 43 seconds less time if it has 1 GB of RAM than if it has only 512 MB of RAM.
Conclusion
This article explained the factors that you should consider when you establish and monitor the team project and metadata cache size limits for Team Foundation Server. In summary, consider the following guidelines:
- For team projects based on the MSF for CMMI Process Improvement template, the recommended metadata cache limit is 250 team projects.
- For team projects based on the MSF for Agile Software Development template, the recommended metadata cache limit is 500 team projects.
- To improve system performance, increase the RAM on client computers to 1 GB.
When you monitor the metadata cache size, follow these guidelines:
- On the Team Foundation Server data tier, monitor the TFSWorkItemTracking.dbo.Rules data space size. The Rules table data space size is indirectly related to the metadata cache size, but is easier to monitor in most cases.
- The recommended team project limit for TFS is roughly equivalent to a 30 MB Rules table data space size.
- Refer to the "Team Project Limit" section of the Team Foundation Server Administrator's Guide if the data space size reaches 24 MB (80 percent).
- Add contingency planning to the disaster recovery plan for Team Foundation Server.
- If the team project limit is less than the 80 percent threshold, no operational changes are necessary when you restore the system from backup media.
- If the team project limit exceeds the 80 percent threshold, bring client computers online in groups of 100 or fewer when you restore the system.