How to evaluate the result of a Conditional Access Policy

Question

Hello,

1. Microsoft has created the following conditional access policy in Report-only mode for us:
   1. Block legacy authentication
2. I'd like to evaluate its result before turning it on so I go to Conditional Access Insights and reporting page. center and I pick that policy.
3. But all it shows under the result column is "reportOnlyNotApplied" which is not helpful.

Any idea what I can do the get a good evaluation?

Thank you

Answer 1

reza admin No, you don’t need to review every individual sign‑in. Use filtering instead. Go to Entra admin center → Monitoring → Sign‑in logs → Sign‑in (non‑interactive) and filter Client app = Other clients (this represents legacy authentication). Then add the Conditional Access column to the view. If you see Report‑only: Failure, it means legacy authentication sign‑ins would be blocked when the policy is enabled. If there are no such entries or everything shows reportOnlyNotApplied, then legacy authentication isn’t being used and enabling the policy won’t impact users.

Answer 2

"reportOnlyNotApplied" means that, for the sign-ins being analyzed, the policy conditions were never met, so the policy was not actually evaluated against any real traffic. To get a meaningful evaluation, the policy must be in scope for actual sign-ins.

Use these steps to understand and validate the policy:

1. Confirm policy scope and conditions
   In the Conditional Access policy:
   • Check Users or workload identities and Target resources (cloud apps) to ensure they include real users and apps that are actually using legacy authentication.
   • Verify Conditions (locations, device platforms, client apps, etc.) are not overly restrictive so that no sign-ins match.
2. Generate matching sign-ins
   Ensure there are sign-ins that use legacy authentication and match the policy scope. If the environment has already blocked or eliminated legacy auth, the policy will remain reportOnlyNotApplied because there is nothing to evaluate.
3. Use sign-in logs to see per-sign-in results
   • Go to Sign-in logs and open individual sign-in events.
   • Use the Conditional Access and Report-only tabs in the sign-in details to see, for each sign-in, whether the policy was:
     • Report-only: Success
     • Report-only: Failure
     • Report-only: User action required
     • Report-only: Not applied These results show exactly how the policy would behave if turned on.
4. Use the What If tool for targeted evaluation
   • In Microsoft Entra admin center → Entra ID → Conditional Access → Policies → What If, run an evaluation.
   • Supply a test identity, target resource, device platform, and client app that should represent a legacy auth scenario.
   • Run What If to see whether the policy would apply and what grant controls would be enforced.
5. Use Policy impact / Insights workbook for broader view
   • In Policy impact (preview) or the Conditional Access Insights and Reporting workbook, review the policy’s impact over the last 24 hours, 7 days, or 1 month.
   • If the policy still shows only reportOnlyNotApplied, it indicates no sign-ins matched the policy conditions in that period.

If, after verifying scope and generating test sign-ins, the policy still shows only reportOnlyNotApplied, it effectively means there is no active legacy authentication usage in scope, and enabling the block policy should have no impact on users.

---

References:

• Analyze Conditional Access Policy Impact
• Troubleshoot Conditional Access Policies with the What If Tool
• Learn about the sign-in log activity details
• Block authentication flows with Conditional Access policy
• Add Conditional Access to user flows in Azure Active Directory B2C (b2c-user-flow)
• Review and apply suggestions from the Conditional Access Optimization Agent