Fatal problem when pushing huge repository to TFS GIT
Deepak Kumar Mishra, Support Engineer, EMEA TFS Team, brings this knowledge experience while working with one of his customers and this important piece of information might help a lot of users who are using Git repos in tfs and face issue while performing a push operation.
Issue Description:
A Customer is trying to push an 18GB Git repository in TFS 2015 Update 3. It worked locally to a local server - but we cannot get it to push to our global TFS server. While pushing the repository via SSH, we get the following errors in a Linux machine:
[xxxx@servername]$ grep tfs .git/config
url = ssh://tfs. servername:22/tfs/collection/teamproject/_git/repo
[xxxx@servername]$ git push
Counting objects: 95405, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (24474/24474), done.
Writing objects: 100% (95405/95405), 18.15 GiB | 8.34 MiB/s, done.
Total 95405 (delta 68425), reused 95401 (delta 68421)
remote: Analyzing objects... (95405/95405) (3030021 ms)
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Cause:
This is a bug in TFS related to SSh which will be fixed in upcoming release.
Our SSH framework times out the connection if there is no communication for over 5 minutes. If pushing a massive repo, the server could do over 5 minutes of work ‘silently’, leading to a connection drop.
The workaround is to make the initial push over https and then switch to SSH.
Troubleshooting and Resolution:
1.The size of the repository was big and so we tried to reduce the size first by using the following commands.
git gc --prune=now --aggressive
git repack -adf
then push
time git gc --prune=now --aggressive; time git repack -adf ; time git push
The command failed with the following errors:
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
2. Then we tried to do a push with Http first and decided to do the next clone and push via SSH. This gave the following error regarding certificate as HTTPS was configured in this case.
fatal: unable to access 'https://tfs.servername/tfs/collection/teamproject/_git/repo/ ': Peer certificate cannot be authenticated with known CA certificates
3. So we informed customer that they were using using a self-signed certificate to allow HTTPS in TFS on-premises They need to permanently add the HTTPS certificate as “trusted” on the machine and ignore the self-sign check for git by using the flag git -c http.sslVerify=false
4. Then We checked and found that the push operation worked as expected by using Https. [[xxxx@servername]$ git -c http.sslVerify=false push Counting objects: 95405, done. Delta compression using up to 4 threads. Compressing objects: 100% (24499/24499), done. Writing objects: 100% (95405/95405), 18.54 GiB | 4.79 MiB/s, done. Total 95405 (delta 68399), reused 95401 (delta 68395) remote: Analyzing objects... (95405/95405) (3894107 ms) remote: Storing packfile... done (2695635 ms) remote: Storing index... done (936 ms) To 'https://tfs.servername/tfs/collection/teamproject/_git/repo/ * [new branch] master -> master
5. Finally we were able to do the next clone and then push by using SSH
Written by: Deepak Kumar Mishra, Support Engineer
Reviewed by: Nitish Nagpal, Support Escalation Engineer