Sign-in logs show user logins failing but access is still granted

Sam Davidoff 1 Reputation point
2024-01-27T14:12:36.82+00:00

We have a conditional access policy set up to require users logging into to our Azure DevOps application to to reauthenticate every day. This is done via a policy that targets the Azure DevOps Resource and sets the session sign-in frequency to one day.

The policy is set to "On" and in the Entra ID sign-in logs we can see that it is applied to users who try to access Azure DevOps. But the access is not actually blocked/reauthentication required on a daily basis. In other words, notwithstanding that the policy is applied, user sessions to Azure Devops persist for (far) longer than 24 hours.

What's also strange is that, in the sign-in logs, these accesses are shown as "Failed" with a message

The refresh token has expired or is invalid due to sign-in frequency checks by conditional access. The token was issued on {issueDate} and the maximum allowed lifetime for this request is {time}.

Which makes sense. Excepts that the logins DONT actually fail.

We have similar refresh policies set up for other apps and they appear to be working.

Any suggestions on what could be going on here?

Microsoft Entra ID
Microsoft Entra ID
A Microsoft Entra identity service that provides identity management and access control capabilities. Replaces Azure Active Directory.
18,715 questions
0 comments No comments
{count} votes

3 answers

Sort by: Most helpful
  1. Andy David - MVP 138.6K Reputation points MVP
    2024-01-27T16:14:26.09+00:00

    You really can't target DevOps alone.

    You have to target the entre Azure mgmt API suite:

    https://learn.microsoft.com/en-us/entra/identity/conditional-access/concept-conditional-access-cloud-apps#windows-azure-service-management-api

    When you target the Windows Azure Service Management API application, policy is enforced for tokens issued to a set of services closely bound to the portal. This grouping includes the application IDs of:

    • Azure Resource Manager
    • Azure portal, which also covers the Microsoft Entra admin center
    • Azure Data Lake
    • Application Insights API
    • Log Analytics API

    Because the policy is applied to the Azure management portal and API, services, or clients with an Azure API service dependency, can indirectly be impacted. For example:

    • Classic deployment model APIs
    • Azure PowerShell
    • Azure CLI
    • Azure DevOps
    • Azure Data Factory portal
    • Azure Event Hubs
    • Azure Service Bus
    • Azure SQL Database
    • SQL Managed Instance
    • Azure Synapse
    • Visual Studio subscriptions administrator portal
    • Microsoft IoT Central

    Note The Windows Azure Service Management API application applies to Azure PowerShell, which calls the Azure Resource Manager API. It does not apply to Microsoft Graph PowerShell, which calls the Microsoft Graph API Also: Session timeouts work best when targeting users, not apps per se ,but if you target the Azure Service API , I would think it will work for you , so I would test that out:

    https://learn.microsoft.com/en-us/entra/identity/conditional-access/howto-conditional-access-session-lifetime#user-sign-in-frequency

    0 comments No comments

  2. Sam Davidoff 1 Reputation point
    2024-01-27T21:10:47.6433333+00:00

    Interesting. So I tried both things you recommended and neither block DevOps' persistence. I set a CAP for 1 day sign-in frequency targeting the Azure DevOps and Windows Azure Service Management API, and I set a separate policy that sets the same 1 day sign-in frequency but for a certain group of users regardless of app. The result is the same. In the sign-in logs, the sign-in is recorded as "Failed" with the same error message above, CAP shows that all both policies (app-specific and user-specific) were applied. And yet the DevOps session marches on undisturbed... Has anyway successfully used Conditional Access to increase the sign-in frequency for DevOps?


  3. Andy David - MVP 138.6K Reputation points MVP
    2024-01-28T22:39:42.98+00:00

    So I was curious about this and it appears there are scenarios where a CA policy is enforced with Dev Ops:https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/change-application-access-policies?view=azure-devops#cap-support-on-azure-devops As I mentioned before the block works from what I have seen using Azure Mgmt as the target app, but I wonder if there is something being missed here