493 questions
Azure DevOps server failing to delete Git LFS objects after Repo has been Deleted
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(179.20000000000002, 1%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ENW%3C/text%3E%3C/svg%3E)
Noah Wolton
0
Reputation points
So we're running an Azure DevOps server on a Microsoft SQL Server, and recently forked a large (~30GB) git repo to test out git LFS. During this testing process the git repo was forked and deleted multiple times.
We realised recently that the deletion of this repo was being handled by a soft-delete process on ADO so the repo was only being marked as deleted but the data was being kept around. To delete the repo fully we used this REST API: https://learn.microsoft.com/en-us/rest/api/azure/devops/git/repositories/delete-repository-from-recycle-bin?view=azure-devops-rest-7.1
This seemed to work and I expected the deleted repos to be cleaned up by ADO the next time it ran a cleanup job (presumably some time in the next 24 hours). It's been over 48 hours and the storage usage by ADO has actually grown 10 GB. Digging into the SQL Database the deleted repos no longer appear in tbl_GitRepository (but do appear in tbl_GitDeletedRepository) but there are lots of objects still in GitOdb.tbl_LfsObject (and these deleted repos are the only ones with Git LFS objects on our server).
I suspect it is these LFS objects which ADO is struggling to delete and while the growth in storage space usage is rather worrying it's possible it is unrelated to this problem (if unlikely). Any ideas on where to go from here? I'm not an expert on how ADO is meant to handle the deletion process so it's possible we just need to wait longer for it to run a full cleanup of Git LFS, but any advice would be greatly appreciated!
Azure DevOps
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(163.2, 64%, 25%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EDR%3C/text%3E%3C/svg%3E)
Durga Reshma Malthi 4,430 Reputation points Microsoft External Staff Moderator2025-06-26T13:35:18.1+00:00 Hi Noah Wolton
The LFS objects are not always cleaned up automatically after repo deletion especially in on-premises Azure DevOps Server environments.
Could you please follow the below steps:
- On your collection database, run the following (with caution and backups in place):
These are safe to run and are designed to clean up orphaned file content. You may need to run the second one multiple times depending on the batch size.EXEC prc_CleanupDeletedFileContent 1; EXEC prc_DeleteUnusedFiles 1, 0, 100000; - Run a query like:
If the reference count is 0, those objects are orphaned and eligible for deletion.SELECT o.LfsObjectId, COUNT(r.RepositoryId) AS ReferenceCount FROM GitOdb.tbl_LfsObject AS o LEFT JOIN Git.tbl_LfsObjectToRepository AS r ON o.LfsObjectId = r.LfsObjectId GROUP BY o.LfsObjectId HAVING COUNT(r.RepositoryId) = 0 - Even if the SQL DB has cleaned up rows, physical disk usage might remain, So, Check
C:\Program Files\Azure DevOps Server\or the location of yourGitRepositoriesandLfsObjectsdirectories. - Azure DevOps has scheduled jobs that handle cleanup tasks, including the removal of orphaned LFS objects. However, these jobs may not run immediately or may have specific conditions for execution - https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage You can use the following command to remove LFS objects:
git lfs prune
Hope this helps!
Please Let me know if you have any queries.
- On your collection database, run the following (with caution and backups in place):
-
Noah Wolton 0 Reputation points
2025-06-26T14:41:30.1066667+00:00 Thanks so much for your quick response! We've backed up the server and I'm ready to run the steps but I'm getting an error with the
EXEC prc_CleanupDeletedFileContent 1;line it says it can't find that procedure.There seem to be lots of similar procedures but not that one, can I ignore that command or should I use a different procedure?
EDIT: Here are the procedures it can find:
EDIT 2: I also seem to be missing Git.tbl_LfsObjectToRepository completely and I can't see any other table in the list that looks equivalent. This might be the source of the issue!
EDIT 3: I'm also struggling to find where the actual git repos are stored sorry! Is there any easy way to find out where they should be?
Sorry for being so useless so far! As I said I'm quite out of my depth here I'm afraid.
-
Durga Reshma Malthi 4,430 Reputation points Microsoft External Staff Moderator
2025-06-26T15:24:50.0266667+00:00 Hi Noah Wolton
Based on the image, it appears you're working with a local Azure DevOps Server, possibly version 2022.1, and that the prc_CleanupDeletedFileContent procedure isn't present. Instead, your server includes several segmented cleanup procedures, such as:
- prc_CleanupDeletedFileContentSegmentedClearFiles
- ...ClearContainers, ...ClearArtifacts, and others.
These segmented procedures likely provide more control and safer batching. The non-segmented procedure (prc_CleanupDeletedFileContent) may have existed in earlier versions or certain database types but has been replaced or separated in newer versions.
Therefore, it's fine to disregard that command and use the segmented procedures directly, depending on what you need to clean (files, containers, artifacts, etc.).
If this Git.tbl_LfsObjectToRepository table is missing entirely, you may not have LFS configured on your instance.
Could you please try the next individual steps and let me know.
-
Durga Reshma Malthi 4,430 Reputation points Microsoft External Staff Moderator
2025-06-26T15:27:28.4733333+00:00 Hi Noah Wolton
Based on the image, it appears you're working with a local Azure DevOps Server, possibly version 2022.1, and that the prc_CleanupDeletedFileContent procedure isn't present. Instead, your server includes several segmented cleanup procedures, such as:
- prc_CleanupDeletedFileContentSegmentedClearFiles
- ...ClearContainers, ...ClearArtifacts, and others.
These segmented procedures likely provide more control and safer batching. The non-segmented procedure (prc_CleanupDeletedFileContent) may have existed in earlier versions or certain database types but has been replaced or separated in newer versions.
Therefore, it's fine to disregard that command and use the segmented procedures directly, depending on what you need to clean (files, containers, artifacts, etc.).
If this Git.tbl_LfsObjectToRepository table is missing entirely, you may not have LFS configured on your instance.
Could you please try the next individual steps and let me know.
-
Noah Wolton 0 Reputation points
2025-06-26T15:34:45.0333333+00:00 Thanks @Durga Reshma Malthi
So
EXEC prc_DeleteUnusedFiles 1, 0, 100000;produced:
Warning: The join order has been enforced because a local join hint is used.Warning: The join order has been enforced because a local join hint is used.Warning: The join order has been enforced because a local join hint is used.Warning: The join order has been enforced because a local join hint is used.Warning: The join order has been enforced because a local join hint is used.I ran it again and then got:
Command(s) completed successfully.I tried running step 2 but got
Invalid object name 'Git.tbl_LfsObjectToRepository'.Steps 3 and 4 require knowing where the server is storing the git repositories and unfortunately I do not know where that is.
Thanks again for your quick response and I apologise for the lack of result.
-
Durga Reshma Malthi 4,430 Reputation points Microsoft External Staff Moderator
2025-06-27T07:02:48.19+00:00 Hi Noah Wolton
You're doing great, and absolutely no need to apologize,
That table’s absence confirms your schema doesn’t use it—possibly due to version differences or how Git LFS was enabled. So instead of checking reference counts, we’ll pivot to age-based heuristics. Try this:
SELECT LfsObjectId, CreatedDate FROM GitOdb.tbl_LfsObject WHERE DATEDIFF(day, CreatedDate, GETDATE()) > 30;This will show you LFS objects older than 30 days, which are good candidates for being orphaned especially if you know the only LFS-using repos were deleted.
If you’ve confirmed these are indeed only tied to deleted repositories and you've taken backups, this will clean them up:
DELETE o FROM GitOdb.tbl_LfsObject o LEFT JOIN Git.tbl_GitDeletedRepository d ON o.RepositoryId = d.RepositoryId WHERE o.RepositoryId IS NOT NULL AND d.RepositoryId IS NOT NULL;You can first run it as a
SELECTto double-check what will be deleted:SELECT COUNT(*) AS LfsObjectsToDelete FROM GitOdb.tbl_LfsObject o LEFT JOIN Git.tbl_GitDeletedRepository d ON o.RepositoryId = d.RepositoryId WHERE o.RepositoryId IS NOT NULL AND d.RepositoryId IS NOT NULL;In Azure DevOps Server, Git repositories are stored inside the SQL database, not on disk like traditional Git servers. So:
- You won’t find
.gitfolders on the file system. - LFS blobs, however, can be cached on disk. Check:
-
C:\Program Files\Azure DevOps Server\ - Or wherever your Application Tier is installed—look for folders like
LfsObjects,FileCache, orArtifacts.
-
- Look in the web.config file: Go to: C:\Program Files\Azure DevOps Server <version>\Application Tier\Web Services\web.config and search for
GitRepositoryPathorLfsObjectPath.
Additional References:
Hope this helps!
Please Let me know if you have any queries.
- You won’t find
Sign in to comment
Sign in to answer