The solution for this is pretty straightforward. Just assign the Entra group that can use Azure VPN enterprise application. (Toggle "Assignment required?" to Yes)
I have also created a Conditional Access policy that targets Azure VPN app that includes everyone but one Entra group and choose "Grant" option as "Block access".
When users outside the group tries to login, they will see the "You don't have access" screen.
One downside is that connected users won't be forced disconnect. It will take time until their session ends and Azure VPN Client tries to get new token and that's when the access will be blocked. (You can force disconnect them from vWAN blade but they will be able to reconnect as long as their token isn't expired yet) However, I believe if you are setting this up new and nobody has connected to the VPN yet, it will work right away.
This is what I have done for now, if anyone has any suggestion or alternate method, please share.
Thanks.