Hi @Alexis Pejouan ,
it's not possible control/mange which user get's access to which VM in the same AVD host pool. The user permission is set for all VMs in the same host pool.
The only way I can see is the option you mentioned with a second AVD host pool for users 4,5,6.
----------
(If the reply was helpful please don't forget to upvote and/or accept as answer, thank you)
Regards
Andreas Baumgarten