Upgrade Git

Check if these hotfixes are applicable if your TFS server is running Windows 2012 R2 or earlier

• A large file upload or a large repository clone fails on VSO in Windows Server 2012 R2
• Pushing large files to Team Foundation Server by Git client hangs

Check if you're using a proxy or load balancer

What if my proxy or load balancer is broken, but I don't have any control over it?

Is setting http.postBuffer harmful?

• Increasing it above the default may increase latency for larger pushes (since the client will buffer the HTTP request into larger chunks).
• If you set it larger than the HTTP chunk size limit for your HTTP server (e.g. maxAllowedContentLength and maxRequestLength in web.config for TFS servers), then all pushes larger than the chunk size limit will start failing.

How do I unset http.postBuffer if I've already set it?

Original Post: 

We removed some unwanted binaries from our repo on visualstudio.com by following the instructions at https://help.github.com/articles/remove-sensitive-data/. We force-pushed to master and deleted all our other branches. After running git gc locally, our local repo is now 5 MB, but git clone from visualstudio.com still returns 100MB. The old unreferenced blobs are still being sent down by the server. How do we git gc (or some equivalent) on the server as well?

1. There is no equivalent to git gc on VSTS yet. Our server preserves the history of every ref/branch update to Git repos, including deleted branches. This is analogous to the "reflog" in core Git. On VSTS, we expose the reflog via the REST API and the Branch Updates (i.e. pushes) tab in Web Access. Similarly to core Git, objects in the reflog are still considered to be referenced and will not be deleted by git gc. Core Git can eventually prune old reflog entries via git prune or git gc, but VSTS does not have that functionality yet.
2. Large fetches are expensive for the server to calculate, so we cheat a little. Large fetches (and clones) have historically been very expensive in both core Git and VSTS due to the "counting objects" phase. https://githubengineering.com/counting-objects/ has a nice explanation of the problem, as well as how core Git and GitHub have (cleverly) improved the perf w/ bitmap indexes. Unfortunately, VSTS does not have that perf fix yet. Instead, it cheats a bit and blindly streams back every object that exists on the server if the client has nothing and is asks for all branches and tags (e.g. for git clone). This is generally reasonable, until a user decides to dereference most of the objects in their repo to save space!

• Delete the repo from the server (EDIT: or rename it) and re-push it. This works, but is sub-optimal.  In the new repo, you won't be able to see old pull request details, branch update history, and any links from other areas like builds or work items.
• Trick the server by not cloning everything at once:

  mkdir newRepo
  git init
  git remote add origin 
  #fetch one branch first
  git fetch origin master
  #fetch everything else
  git fetch origin
  

• Git for Windows (https://git-scm.com/)
• Fiddler (https://www.telerik.com/fiddler)
• gitfiddler.cmd (attached at bottom)

Steps:

1. Run gitfiddler.cmd in a new command prompt.  This script configures git.exe to use Fiddler as a proxy, and then waits for a key press.  Once a key is pressed, the script clears the settings that it sets, and then exists.
• Leave the script running for now.
• READ the script before running it!
2. Start Fiddler
• If you are using HTTPS, enable HTTPS decryption (https://docs.telerik.com/fiddler/Configure-Fiddler/Tasks/DecryptHTTPS)
3. Run your git commands
4. Select the captured requests in Fiddler and save them in a ArchiveZip (*.saz) file.

gitfiddler.cmd

If you see one of the following errors when branching [1] or renaming a path in TFS:

TF14098: Access Denied: User DOMAIN\Bob needs Read permission(s) for $/p0/folder/*.

Or the following error when checking in:

TF14098: Access Denied: User DOMAIN\Bob needs Checkin permission(s) for $/p0/folder/*.

You can follow these steps to figure out which permission entries are blocking your operation.  The instructions are for when you do not have Read permissions, but apply to Checkin permissions as well (just replace Read with Checkin)

These are the scenarios that can result in DOMAIN\Bob not having Read permission on $/p0/folder/a.txt

1. The Read permission is explicitly denied for $/p0/folder/a.txt for DOMAIN\Bob, or a group that he belongs to.

• Note that group membership includes both direct and indirect membership (e.g. Bob belongs to subgroup which belongs to supergroup).

2. The Read permission is unset for a.txt and all parent folders that it's inheriting permissions from, for Bob or any groups he belongs to. Unset permissions default to implicitly denied.

• Note that you can hit this case when permission inheritance is turned off for a.txt or one of it's parents.

3. The Read permission is explicitly denied on a parent of $/p0/folder/a.txt for DOMAIN\bob or a group that he belongs to. If this permission is not overridden at a lower level, $/p0/folder/a.txt will have inherit the deny entry.

4. If there are multiple applicable entries, denies will override allows.  If Contributors are denied Read access, and Bob belongs to Contributors, giving Bob Read access will not actually give him Read access

• The way to fix this is to either scope the deny down to a smaller group that Bob does not belong to, or turn off inheritance and leave permissions unset instead of using deny entries at all.

To diagnose what entry is causing issues [2]:

1. The following steps are only guaranteed to work if you authenticate as a member of the Team Foundation Administrators group (for the entire instance) or the Team Project Collection Administrators group.  Being an ordinary user or a member of the Team Project Administrators group is insufficient.  That is because if you do not have Read permissions, TFS will not tell you don’t have Read permissions.  The exception is that the special Administrators groups are always allowed to see all paths and permissions.  This is to keep Administrators from accidentally locking themselves out. Administrators will still get Access Denied errors, but they can at least see and fix the offending permissions entries.
2. In a Visual Studio Command Prompt, run tfssecurity /imx DOMAIN\Bob /server:https://SERVER:8080/tfs to get the list of groups Bob belongs to, directly, or indirectly.
3. In the same prompt, run tf permission $/p0/folder /recursive > out.txt, and open out.txt in a text editor that supports regex search (like Visual Studio):
• Note the use of the /recursive flag
• We are redirecting output to a text file because there may be ton of entries
• Search for any paths that include the following entries:
• “Inherit: No” where there is no explicit Allow: Read entry for the user or one of the groups he belongs to directly/indirectly
• “Deny.*Read” (use a regex search) for Bob or any groups that tfssecurity showed.

[1] We block server side branching (which checks in immediately without pending changes in your workspace) if you don't have recursive Read permissions on the source. Operations that result in server-side branches include:

• tf branch /checkin
• the VersionControlServer.CreateBranch() client object model call
• Branching from a Branch Object in Source Control Explorer

We do this because:

• If the user doesn't have Read permission on an item in the source, he shouldn't be allowed to branch that item
• If TFS offered to skip those items, it could only tell the user that one or more items will be skipped, not which items will be skipped.  The user doesn't have Read permissions, so disclosing the affected paths could leak sensitive information.
• If we allow the user to skip any items, there is no way for the user to figure out if anything important is missing from the new branch without talking to an Admin.  In that case, the user might as well have the Admin perform the branch anyway.

If you pend a branch instead (a non-server side branch), then we silently skip files with no Read perms. I consider that a bug (and have filed it): it should show a warning that one or more items couldn't be branched. Otherwise there is no way to trust that the branch is complete. Even an admin may be affected by explicit denies or disabled inheritance.


[2]I've heard on the MSDN forums that the TFS Permissions Sidekick from Attrice has a better UI for figuring out where you don't have Read access, but I've never used it.

test post