Do you mean that you would like to delete the addresses in the following screenshot?
I tested in my lab(Exchange 2019 CU2) and found that the address can be deleted by hanging your mouse on it and pressing delete on your keyboard.
Have you tried it?
If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.