Thanks for sharing the details.
I would recommend connecting with your Microsoft (MS) Account team to find another license option. e.g. E5 is not needed to use AVD.
A Windows E3 or even a VDA per user might be a more economical option compared to E5 (assuming that you are not using any other E5 features).
You can take a look here: https://learn.microsoft.com/en-us/azure/virtual-desktop/overview#requirements
I checked with the Product team, currently they don't have any plans to change this behavior, but you can always submit this as a feature idea here:
Azure Virtual Desktop Feedback - Microsoft Tech Community
Or you can share it with your MS account team and they can submit this internally as a feature request.
Hope this helps.