Dear Pieter,
I'm following up on the thread with you.
The command Remove-SPOSite -Identity https://contoso.sharepoint.com/sites/OldSiteName should remove the site redirect.
Based on the error message "The operation is not supported for the site," after you run the command in PowerShell in your environment, there may be some retention policies applied to the old site. I suggest you check whether there are some retention policies. If yes, you might need to remove the retention policy to delete the redirect.
For your reference: Manage site redirects
Sincerely,
Cliff | Microsoft Community Moderator