Hi @Draggeta
From a similar question at https://learn.microsoft.com/en-us/answers/questions/460947/cannot-delete-firewall-policy.html (answered by suvasara-MSFT)
Apologies for the inconvenience caused. At this point, we think there may be orphaned link and we may need help from our internal team to delete it. I would suggest you drop a mail at AzCommunity@microsoft.com referencing the link to this post as well as engineer you are working with for immediate resolution.
You can also create a ticket with Azure Support at:
https://azure.microsoft.com/en-us/support/create-ticket/
----------------------------------------
If this is helpful please accept answer.