Authenticator apps keeps registering an account automatically!

Question

Really strange issue here.

We created our first conditional access policy in Azure to block Teamsfrom everywhere except our home office. The conditional access rule is pretty simple... block all EXCEPT ...then we have a "named location" called home office with our subnet. Pretty simple right?

While testing this... we tried to access Teams via mobile on 5 G .. so it should be blocked... nope, it goes right through... but for a second we see authneticator app POP up and disappears.. .then it allows us in.

1. Why is the authenticator app even involved? We don't have an MFA policy at all.. just a single policy to block teams
2. We check default global settings in Azure AD and made sure its off.
3. We delete the account in authenticator and try signing in again... after you put in your credentials... we see auth app for a split second again! then we're allowed in and when we check... the account got recreated in authenticator APP... why..how?
4. When we look at the sign-in logs we see that the Application is called "Microsoft Authentication Broker" ... we assume thats the authenticator letting the user bypass the conditional access policy.
5. If there is no authenticator app then the conditional access policy works
6. We cannot tell people to remove authenticator app as they may have other profiles on there they use.

So now IF you have authenticator on your mobile, you'll be able to bypass the conditional access policy and sign in.

Please help!