1. If your collection is not hosted on a sever which allows inbound internet traffic, you will have to restore the collection on an internet inbound/outbound enabled server virtual machine. To reduce overhead of network latency, It's  recommenced that you host the server/virtual machine in the same region as your planned VSTS account. Please refer to migration guide for the setup options.

2. Collection database Backup: Divide collection backup into multiple files (16 to 24 files) and use MAXTRANSFERSIZE hint. This approach reduced total backup duration by 75% for our collection database.

3. Use AZCopy to transfer backup to Azure blob/ VM: You can get optimal performance for database files copy to Azure blob/VM using AZCopy utility.

4. Initiate multiple instance of AZCopy simultaneously: Running multiple instances of AZCopy in parallel (for different files upload) would help speed-up the upload/download files to/from blob storage. You will have to use /z switch to run parallel instances of AZCopy

Note that you still need required permissions to run the tool.

o   In command prompt navigate to TFSMigrator tool directory

o   Execute below command (Update the collection URL and Database server name)

• Migration troubleshooting: Link
• Import process errors: Link
• Process config errors: Link

As a TFS/Collection administrator I come across scenarios to improve overall health of the TFS environment.

One common scenario is cleaning up data. Periodically I will clean up data to makes sure the TFS collection is in a manageable/healthy state. This also helps to improve overall TFS performance and reduced sync latency in processes like warehouse sync adaptors.

Below are SQL queries to find list of work items (List may vary based on the filter condition, change where clause based on business required):

For TFS 2013/2012:

SELECT ID  FROM [WorkItemsAre] WITH (NOLOCK)

WHERE [Created Date] < DATEADD(MONTH, -18, GETUTCDATE())

You may also consider deleting work items based on last updated/changed date. Below is the query to get the work item ID's:

SELECT ID  FROM [WorkItemsAre] WITH (NOLOCK)

WHERE [Changed Date] < DATEADD(MONTH, -18, GETUTCDATE())

For TFS 2015:

SELECT ID  FROM [tbl_WorkItemCoreLatest] WITH (NOLOCK)

WHERE [CreatedDate] < DATEADD(MONTH, -18, GETUTCDATE())

Note: Work item destroy command used in the PowerShell script below is non recoverable command. Take required database backups if work items may need to be accessed in future.

During the work item destroy operation Team Foundation Server warehouse database will not be updated. Based on the amount of data deleted warehouse may show data latency for hours until warehouse adaptor will catch with all the updates.

Copy and paste below script in a PowerShell file (with .ps1 extension), update the variable values mentioned in the list #4 below and run the command from a machine where witadmin tool is installed (Generally available after visual studio installation). Open PowerShell command window and execute the script.

Note: Account running below script should have team foundation administrator or collection administrator access.

To delete large number of records you may also run multiple instances of below script. I have not seen any issues while running 2 instances of the script to delete 300K records.

########TFS Work Items Bulk Destroy Automation Script##########

#Notes:

#1) This script requires to setup file/folder path, validate the file/folders path before running the script

#2) start the powershell window as Administrator and run the script

#3) This script requires share and admin access on the destination server, make sure your account or the account under which script is executing is member of admin group

#on the destination server

#4) Update following:

#   4.1: $CollectionURL

#   4.2: $WitAdmin tool location

        # For VS 2015, Default location is C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE

        # For VS 2013, Default location is C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE

#   4.3: $WI_List

#   4.4: $logfile

####################

$CollectionURL = "https://tfs:8080/tfs/CollectionName"

$WitAdminLocation = "C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE"

$WI_List =  Get-Content "C:\WI_List.txt"

$logfile="C:\log.txt"

$ExecutionStartTime = Get-Date

$WICount = 0

"## Starting WI Destroy @ $ExecutionStartTime ##"| Out-File $logfile -Append

"Collection URL: $CollectionURL" | Out-File $logfile -Append

foreach ($WIID in $WI_List)

    {

        CD $WitAdminLocation

        .\witadmin destroywi /collection:$CollectionURL /id:$WIID /noprompt

        "WI ID: $WIID Destroyed" | Out-File $logfile -Append

        $WICount = $WICount + 1

        Write-Host "$WICount Work Items Deleted"

    }

$ExecutionEndTime = Get-Date

"## WI Destroy Command Completed @ $ExecutionEndTime ##"| Out-File $logfile -Append

$TotalExecutionTime = $ExecutionEndTime - $ExecutionStartTime

"Total Work Items Deleted: $WICount"   | Out-File $logfile -Append

" Total Execution Time: $TotalExecutionTime"  | Out-File $logfile -Append

##End of script##

Other References:

witadmin destroywi command details here

TFS permission references here

Please note that some of the queries execution may impact TFS performance and it is not advised to make any changes via SQL backend.

TFS Version: TFS 2012 or higher

Top 10 Commands with highest average response time (in last 7 days)

USE [Tfs_DefaultCollection]

SELECT TOP 10 Application, Command, ROUND(SUM(CAST(ExecutionTime ASfloat)/1000000)/SUM(ExecutionCount),3)AS ResponseTimeInSeconds

FROM tbl_Command WITH (NOLOCK)

WHERE StartTime >DATEADD(DAY,-7,GETUTCDATE())

GROUP BYApplication,Command

ORDER BYSUM(CAST(ExecutionTime ASfloat)/1000000)/SUM(ExecutionCount)DESC

Number of files in a Team Project Collection (TPC)

 USE [Tfs_DefaultCollection]

SELECT OwnerId, COUNT(*)AS Cnt,SUM(FileLength)as filelengthSum,SUM(CompressedLength)AS CompressedLengthSum

FROM tbl_File WITH (NOLOCK)

WHERE PartitionId=1

GROUP BY ownerid

Get list of failed Jobs (TFS interal scheduled jobs)

USE [Tfs_DefaultCollection]

SELECT   d.JobName, q.QueueTime as NextRunTimeUTC, DATEDIFF(SECOND,GETUTCDATE(), q.QueueTime)AS NextRun,

                 JobState, ExecutionStartTime as ExecutionStartTimeUTC, StateChangeTime, q.QueuedReasons, q.QueueFlags, shp.MachineName as AgentName,

                shp.StartTime as AgentStartTimeUTC,jh.ResultMessage

FROM     tbl_jobqueue q

JOIN  tbl_JobDefinition d ON d.JobId=q.JobId

JOIN  tbl_JobHistory jh ON JH.JobId = q.JobId

LEFT JOIN tbl_ServiceHostProcess shp ON q.AgentId=shp.ProcessId

WHERE Result <> 0
ORDER BY jh.StartTime DESC

Get List of Workspaces not accessed in last 90 days

USE [Tfs_DefaultCollection]

SELECT COUNT(*)

FROM tbl_Workspace ws WITH (NOLOCK)

JOIN tbl_LocalVersion lv WITH (NOLOCK)

ON lv.WorkspaceId = ws.WorkspaceId

WHERE ws.Type = 0 and ws.LastAccessDate < DATEADD(DAY,-90,GETUTCDATE())

Number of TFS Builds Executed in Last 7 Days

USE [Tfs_DefaultCollection]

SELECT  P.ProjectName, BD.DefinitionName ,B.[BuildNumber] , B.[StartTime], B.[FinishTime] , BQ.Quality ,B.[BuildStatus] ,B.[CompilationStatus] ,B.[TestStatus] ,

B.[LastChangedOn] ,B.[KeepForever] ,B.[BuildUri] ,B.[Reason]   ,B.[Deleted] ,B.[DeletionStatus] ,B.[BuildId] ,B.[ContainerId]

FROM [tbl_Build] B WITH (NOLOCK)

LEFT OUTERJOIN tbl_BuildDefinition BD WITH (NOLOCK) ON B.DefinitionID = BD.DefinitionId

LEFT OUTERJOIN tbl_BuildQuality BQ  WITH (NOLOCK) ON BQ.QualityID = B.QualityId

LEFT OUTERJOIN [tbl_BuildGroup] BG WITH (NOLOCK) ON BG.GroupId = BD.GroupID

LEFT OUTERJOIN tbl_Project P WITH (NOLOCK)ON P.ProjectUri = BG.TeamProject

WHERE B.PartitionId = 1 AND BD.PartitionId = 1

AND B.[StartTime] > DATEADD(DAY,-7,GETUTCDATE())

ORDER BY B.StartTime

Get List of Application Tier Server(s):

SELECT * FROM [Tfs_Configuration].[dbo].[tbl_ServiceHostProcess] WITH (NOLOCK)

Sharing below resources which were extremely helpful in Clearing my certification exam. While these resources could be helpful in the exam there is no better way to read the information and practice on a test/non user impacting TFS instance.

Note: Information in this article is for reference purpose only, please go through entire content available online, in books to get extensive knowledge around each of the topic.

What’s in Exam?

Per Microsoft site https://www.microsoft.com/learning/en-us/exam-70-496.aspx , exam is divided into four quadrants:

1. Install and configure Team Foundation Server (TFS) (25‒30%)

2. Manage Team Foundation Server (25‒30%)

3. Customize Team Foundation Server for team use (20‒25%)

4. Administer version control (25‒30%)

My notes for each of the topic: Please download attachment

Useful Resources:

MSDN Link for Application Lifecycle Management overview using TFS :https://msdn.microsoft.com/en-us/library/fda2bad5(v=vs.110).aspx

Professional Team Foundation Server 2012: https://www.amazon.com/Professional-Team-Foundation-Server-2012/dp/1118314093

Awesome online training: https://www.microsoftvirtualacademy.com/training-courses/administering-visual-studio-tfs-2012-exam-496-jump-start

TFS Installation Guide: https://www.microsoft.com/en-us/download/details.aspx?id=29035

TfsAdministrationCertification.pdf

# This script will add multiple groups on multiple serevrs
# Make sure you have one server in each row in the servers text file
# you must have administrator access on the server

$ServersList = "D:\ServersList.txt"
$ServerNames = get-content $ServersList
$UserGroupFilePath = "D:\SecurityGroup.txt"
$UserGroupList = get-content $UserGroupFilePath
$DomainName ="Enter your domain name here"

foreach ($name in $ServerNames)
    {
        $localAdminGroup = [ADSI]("WinNT://$name/Administrators")
        # Add all the groups in text file to the current server
        foreach ($UserGroupName in $UserGroupList)
            {
                $AdminsG = [ADSI] "WinNT://$DomainName/$UserGroupName"
                $localAdminGroup.Add($AdminsG.PSBase.Path)
                Write-Host "Adding" $AdminsG.PSBase.Path "to" $name
            } # End of User Group Loop
    } # End of Server List Loop

Remove multiple security groups on multiple servers

# This script will delete multiple security groups on multiple servers
# Make sure you have one server in each row in the servers text file
# you must have administrator access on the server
 

$ServersList = "D:\ServersList.txt"
$ServerNames = get-content $ServersList
$UserGroupFilePath = "D:\SecurityGroup.txt"
$UserGroupList = get-content $UserGroupFilePath
$DomainName ="Enter your domain name here"
foreach ($name in $ServerNames)
    {
        $localAdminGroup = [ADSI]("WinNT://$name/Administrators")
        # Add all the groups in text file to the current server
        foreach ($UserGroupName in $UserGroupList)
            {
                $AdminsG = [ADSI] "WinNT://$DomainName/$UserGroupName"
                $localAdminGroup.remove($AdminsG.PSBase.Path)
                Write-Host "remove" $AdminsG.PSBase.Path "to" $name
            } # End of User Group Loop
    } # End of Server List Loop

Source files download operations could be pretty resource expensive operations and could impact overall TFS instance performance. A proper implementation of TFS proxy (for source code download operations) can help to increased TFS instance performance also reduce total time to complete the download operations. TFS proxy can also help to reduce total operational cost, increased TFS user’s productivity by decreasing total download operation time.

Our team provides support on software development tools to multiple groups within company. Some of the TFS implementations are the largest implementation of on-premises Team foundation servers.

Total databases sizes on all the instances is 110+ TB.

Some of the limitations which can severely impact TFS server performance:

1. Large number/size files downloads from TFS server
2. Location of TFS server, TFS proxy server
3. Frequent cleanup of TFS proxy server content
4. Users/automations not using correct proxy server name

Primary benefits of TFS proxy:

1. Load on the main server is reduced; this is really the main benefit and the reasoning behind the recommendation for everyone to use proxies whether remote or not
2. Performance is increased for users in remote sites, particularly for files that are retrieved multiple times or by more than one person

Introductory material on this feature can be found here.

Below proposed architectures provides details around possible solutions to make sure that TFS proxies are used and also is maintainable without impacting end users.

2. Proxy server hardware Recommendations

1. At-least one server in each region (like US, Europe, Singapore, India, China)
2. Standard web server/file server SKU
3. Data caching drive should be at-least 20-30% of total content, this will reduce possibility of frequent cleanup.
4. You will need about 2 IOPS per file, per second - so, say 1K downloads per sec requires 2KIOPS - 20 spindles or a couple SSDs.

3. Proxy Architecture

During the proxy setup evaluation phase we have considered following options;we have also done a proof of concept (POC) for all 4 options and based on the POC have documented advantages/disadvantages of each option to make an informed decision.

3.1. (Option#1) Proxy Setup Via GTM

Global Traffic Manager (GTM) is one of the standard ways to redirect traffic based on the client location.

This architecture gives us advantage of being able to redirect the traffic based on the client and in the case of one server failure, the client can perform the Get operation from the nearest available other proxy.

This architecture has other advantages, such as manageability and reliability. Recognizing that our team manages 750+ team project collections, when the proxy server name changes we do not have to make changes on all the team project collection, we just need to make changes on the virtual IP (VIP) level without affecting user’s configuration/settings.  

3.1.1. Architecture

3.1.2. Configuration on team project collection

This configuration is one time task.

To register a proxy for a specific team project collection execute the following command

1. You must have TFS administrator permission to setup proxies on project collection.
2. Open command prompt
3. Change default directory to TF.exe location directory
4. Execute below command

tf proxy /add <<ProxyGTMURL>> /default:global  /collection:<<Collection URL>>

Note: Replace <<>> text with appropriate values.

tf proxy command & Permission details are available here 

3.1.3. User configuration

1. Open command prompt
2. Change default directory to TF.exe location directory
3. Execute below command

tf proxy /configure /Collection:<<TeamProjectCollectionName>> 

3.1.4. Advantages 

1. End user & machines never have to update the proxy configure
2. If the proxy servers (Name or physical server) changes there is no impact on the end-user
3. If TFS admin decided to add new servers in the current architecture, we do not have to configure servers URL on 750+ TPC databases
4. Manageability of the proxies’ setup is easy. 

3.1.5. Limitations/Disadvantages

1. Virtual IP (VIP) & GTM options may not be available in all geographical locations.. 

3.2. (Option#2) Proxy Setup Via AD site name

Proxy configuration via AD-Site name is another supported way to configure proxy for users in various locations.

The advantage of this solution is that users don’t have to explicitly configure a proxy for each user which increases the possibility of proxy usage more than in the current state. Since the majority of the Get operation will be served by the proxy it will reduce the load on the TFS instance and help to improve overall system performance. 

3.2.1. Architecture

3.2.2. Configuration on team project collection

To register a proxy for a specific team project collection execute the following command

1. You must have TFS administrator permission to setup proxies on project collection.
2. Open command prompt
3. Change default directory to TF.exe location directory
4. Execute below command

tf proxy /add <<ProxyURL>> /default:site /site:<<AD-SITE-NAME>> /collection:<<Collection URL>>

Note: Replace <<>> text with appropriate values.

Note: If you have multiple team project collection or site names you can script out the implementation to run the above mentioned commands for all team project collection.

3.2.3. User configuration

1. Open command prompt
2. Change default directory to TF.exe location directory
3. Execute below command

tf proxy /configure /Collection:<<TeamProjectCollectionName>> 

3.2.4. Advantages 

1. End user & machines never have to update the proxy configure
2. If the proxy server changes (rename or new) there is no impact on the end-user
3. If TFS admin decided to add new servers in the current architecture, we do not have to configure servers URL on 750+ TPC databases
4. Manageability of the proxies’ setup is easy. 

3.2.5. Limitations/Disadvantages

1. In the case of a proxy server name change the change needs to be executed on 750+ TPC databases
2. For the best use of the proxy functionality TFS admin should configure all possible site names from all users’ (across the world).
3. Maintainability cost of this option is very high as the accuracy of the information is user dependent and Admins may not have visibility of new users/team additions on different locations.

3.3. (Option#3) Hybrid setup with GTM & AD site name

In the hybrid solution we are going to merge best of option #1 & Option #2.

For the locations where GTM is supported (Like Redmond, Dublin, Singapore) we will add the proxy name as GTM URL with AD-Site name.

For the locations where GTM/VIP is not available we will use the server friendly name (alias name for server, can be created via DNS) and will add the TFS proxy using friendly name. In future if the server need to be replaced we just have to change the pointer (with new server IP) to replace the server. There will not be any configuration change for users.

3.3.1. Architecture

3.3.2. Configuration on team project collection

To register a proxy for a specific team project collection execute the following command

1. You must have TFS administrator permission to setup proxies on project collection.
2. Open command prompt
3. Change default directory to TF.exe location directory
4. Execute below command

tf proxy /add <<ProxyURL>> /default:site /site:<<AD-SITE-NAME>> /collection:<<Collection URL>>

Note: Replace <<>> text with appropriate values.

3.3.3. User configuration 

1. Open command prompt
2. Change default directory to TF.exe location directory
3. Execute below command

tf proxy /configure  /Collection:<<TPCURL>> 

3.3.4. How the setup works on client side

1. TFS Client calls AD to get his site name.
2. Client gets response from AD "Your site name is SITE-NAME."
3. Client makes a call to TFS to check the configured proxy for SITE-NAME
4. Client gets response from TFS with proxy URL for SITE-NAME
5. Client usages the proxy URL for file download operations. 

3.3.5. Advantages 

1. End user & machines never have to update the proxy configure
2. If the proxy server changes (rename or new) there is no impact on the end-user
3. If TFS admin decided to add new servers in the current architecture, we do not have to configure servers URL on 750+ TPC databases
4. Manageability of the proxies’ setup is easy.

3.3.6. Limitations/Disadvantages

1. For the best use of the proxy functionality TFS admin have to configure all team project collections for all possible site names.

Special thanks to Rémi Lemarchand and Philip Kelley for guidance and contribution.

Disclaimer: Above information is provided based on proxy implementation done in my team, Users/TFS admins should test the scenarios for best possible options.