There are some steps you can follow to troubleshoot, there is an official link with several solution scenarios, just like conditional access.
Conditional access policies can sometimes interfere with SSO, requiring additional authentication steps that may not be correctly bypassed by SSO configuration. Review any conditional access policies applied to your users or the Azure Virtual Desktop application to ensure they don't inadvertently block or complicate the sign-in process.
In the link below there are several troubleshooting solutions for this case.
Reference: https://learn.microsoft.com/en-us/entra/identity/hybrid/connect/tshoot-connect-sso