Changed primary email alias, it 'failed', and now UPN doesn't match

Question

I recently added an email alias to my account and then tried to set it as primary. I received a generic error indicating that the system was unable to set the address as primary. Despite this, it now shows the newly added address as primary and I was able to delete the old alias. The catch is that the old address still shows my UPN and it also appears in odd places in the portal.

Is there a way to convince azure that I don't want to use that old address anymore?