I'd approach the problem by starting fresh, and likely taking the opportunity to move compiled/large artefacts to other stores such as Azure Storage / Azure Artefacts or even just a different repo. I realise that this probably isn't palatable, as you're likely seeking to retain the commit history.
We recommend keeping your repository below 10 GB for optimal operation. If your repository exceeds this size, consider using Git-LFS, Scalar, or Azure Artifacts to refactor your development artifacts. REF: https://learn.microsoft.com/en-us/azure/devops/repos/git/limits?view=azure-devops#repository-size