@kage At this time, non-UTF8 characters are not supported while custom error pages are in public preview.
"Non-UTF8 character support is planned, but is not currently supported in preview."
Source:https://github.com/Azure/app-service-linux-docs/blob/master/HowTo/configure_custom_error_page.md
We are meeting with the product group later this month and we will bring up your use case to see if there is an updated timeline to add this functionality that we can share with you.
If you found this answer helpful, please accept it as the answer to help the community.