Currently, Conditional access seems to be the only sure way since you can limit the use to an IP range with your network and VNet. A VPN connection would, of course, succeed since the VPN would provide the endpoint with a valid corp IP address.
We can't prevent clients from using TCP connection via the public endpoints for the AVD gateway/broker services.
It looks like you are looking for functionality like Private Link support, which is not currently available.
I will share this feedback with the Product team, which could help them plan the product roadmap.
There is an existing thread on achieving this using Conditional access: https://learn.microsoft.com/en-us/answers/questions/66463/wvd-inbound-ip-address-restrictions.html