@Danny Roy I will start with a couple of questions to get more clarity on your case:
1 - Can you post error message here?
2 - can you run 'remove-azvirtualnetworkGateway -Name IPMM-STS_VNG -ResourceGroupName "IPMM-STS_RG" -Verbose'
and put error message here
Possible solutions:
1 - make sure all connections on the gateway are removed first.
2 - I've already caught some cases that we managed to delete the public ip before the gateway and the gateway couldn't remove it because of this dependency.
In this case, only Microsoft support can remove it, you will have to open a support ticket
https://azure.microsoft.com/en-us/support/create-ticket
Get in touch if you need more help with this issue.
--please don't forget to "[Accept the answer]" if the reply is helpful--