Azure DevOps server failing to delete Git LFS objects after Repo has been Deleted

Question

So we're running an Azure DevOps server on a Microsoft SQL Server, and recently forked a large (~30GB) git repo to test out git LFS. During this testing process the git repo was forked and deleted multiple times.

We realised recently that the deletion of this repo was being handled by a soft-delete process on ADO so the repo was only being marked as deleted but the data was being kept around. To delete the repo fully we used this REST API: https://learn.microsoft.com/en-us/rest/api/azure/devops/git/repositories/delete-repository-from-recycle-bin?view=azure-devops-rest-7.1

This seemed to work and I expected the deleted repos to be cleaned up by ADO the next time it ran a cleanup job (presumably some time in the next 24 hours). It's been over 48 hours and the storage usage by ADO has actually grown 10 GB. Digging into the SQL Database the deleted repos no longer appear in tbl_GitRepository (but do appear in tbl_GitDeletedRepository) but there are lots of objects still in GitOdb.tbl_LfsObject (and these deleted repos are the only ones with Git LFS objects on our server).

I suspect it is these LFS objects which ADO is struggling to delete and while the growth in storage space usage is rather worrying it's possible it is unrelated to this problem (if unlikely). Any ideas on where to go from here? I'm not an expert on how ADO is meant to handle the deletion process so it's possible we just need to wait longer for it to run a full cleanup of Git LFS, but any advice would be greatly appreciated!

Answer 1

Hi Noah Wolton

The LFS objects are not always cleaned up automatically after repo deletion especially in on-premises Azure DevOps Server environments.

Could you please follow the below steps:

1. On your collection database, run the following (with caution and backups in place):

      EXEC prc_CleanupDeletedFileContent 1;
      EXEC prc_DeleteUnusedFiles 1, 0, 100000;
   

   These are safe to run and are designed to clean up orphaned file content. You may need to run the second one multiple times depending on the batch size.
2. Run a query like:

      SELECT o.LfsObjectId, COUNT(r.RepositoryId) AS ReferenceCount
      FROM GitOdb.tbl_LfsObject AS o
      LEFT JOIN Git.tbl_LfsObjectToRepository AS r ON o.LfsObjectId = r.LfsObjectId
      GROUP BY o.LfsObjectId
      HAVING COUNT(r.RepositoryId) = 0
   

   If the reference count is 0, those objects are orphaned and eligible for deletion.
3. Even if the SQL DB has cleaned up rows, physical disk usage might remain, So, Check C:\Program Files\Azure DevOps Server\ or the location of your GitRepositories and LfsObjects directories.
4. Azure DevOps has scheduled jobs that handle cleanup tasks, including the removal of orphaned LFS objects. However, these jobs may not run immediately or may have specific conditions for execution - https://docs.github.com/en/repositories/working-with-files/managing-large-files/removing-files-from-git-large-file-storage You can use the following command to remove LFS objects:

      git lfs prune
   

Hope this helps!

Please Let me know if you have any queries.

Answer 2

Hi Noah Wolton

Based on the image, it appears you're working with a local Azure DevOps Server, possibly version 2022.1, and that the prc_CleanupDeletedFileContent procedure isn't present. Instead, your server includes several segmented cleanup procedures, such as:

• prc_CleanupDeletedFileContentSegmentedClearFiles
• ...ClearContainers, ...ClearArtifacts, and others.

These segmented procedures likely provide more control and safer batching. The non-segmented procedure (prc_CleanupDeletedFileContent) may have existed in earlier versions or certain database types but has been replaced or separated in newer versions.

Therefore, it's fine to disregard that command and use the segmented procedures directly, depending on what you need to clean (files, containers, artifacts, etc.).

If this Git.tbl_LfsObjectToRepository table is missing entirely, you may not have LFS configured on your instance.

Could you please try the next individual steps and let me know.