Hi Zach,
When you say you've already given IAM access to the Virtual Machine, do you explicitly mean either the Virtual Machine User Login role or the Virtual Machine Admin Login role? It must be one of these two, having contributor or owner at the VM or above is not a subsititute for one of these roles.
The other thing to check is the RDP Properties of the host pool, make sure, under the 'Connection information' tab that Microsoft Entra single sign-on = Connections will use Microsoft Entra authentication to provide single sign-on.