Google workspace external identity configuration issue

I have a configuration problem with using Google workspace as external identity provider in Azure B2B. The Microsoft official document only have ADFS as example. I looked at both the MS doc and Google doc and could not find some consistent instructions on doing do.
Anyway, I created a SAML\WS-Fed external provider configuration using the metadata from Google workspace. I also created a Web\Mobile app using the build-in Microsoft 365 template in the Google workspace admin console. When I tried the test SAML login from the app. configuration, I got the following error.
AADSTS50107: The requested federation realm object 'https://accounts.google.com/o/saml2?idpid=xxxxxxxxx' does not exist.
I am not sure where do I start my troubleshooting from here.
Anyone has done this before? Any help will be appreciated.
Thanks