Thank you for your patience here. I got a response back from the team.
The error received in this case when you reach the limitation of 500 custom domains and while creating a new custom domain after this limit has been reached is
"message": "The number of Custom Domains created per profile exceeds quota."
Hope this helps! Please let me know if you have any additional questions. Thank you!
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.