Thank you for posting your question in Microsoft Q&A
Usually, for all guest user's authentication always happens in user's home tenant. Say for example, if user belongs to contoso.com domain which is verified in another tenant, and also let's say contoso.com domain is federated with ADFS.
If user@Company portal .com tries to access any application which is configured in Fabrikam.com tenant, then authentication for user@Company portal .com will happen in contoso.com tenant only. And since contoso.com domain is federated with ADFS, Azure AD will send the request to ADFS for authentication.
Post authentication token will be given back to tenant where application is configured.
In your scenario, this redirection is happening to ADFS because guest user is configured to authenticate with ADFS. There is no other workaround for this and this is by design.
Let me know if you have any further questions on this.
Please "Accept the answer" if the information helped you. This will help us and others in the community as well.