You can just remove the add-in directly from Exchange admin center. You don't have to make it unavailable before removing it.
If the response is helpful, please click "Accept Answer" and upvote it.
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.