Sign-in logs show user logins failing but access is still granted

Question

We have a conditional access policy set up to require users logging into to our Azure DevOps application to to reauthenticate every day. This is done via a policy that targets the Azure DevOps Resource and sets the session sign-in frequency to one day.

The policy is set to "On" and in the Entra ID sign-in logs we can see that it is applied to users who try to access Azure DevOps. But the access is not actually blocked/reauthentication required on a daily basis. In other words, notwithstanding that the policy is applied, user sessions to Azure Devops persist for (far) longer than 24 hours.

What's also strange is that, in the sign-in logs, these accesses are shown as "Failed" with a message

The refresh token has expired or is invalid due to sign-in frequency checks by conditional access. The token was issued on {issueDate} and the maximum allowed lifetime for this request is {time}.

Which makes sense. Excepts that the logins DONT actually fail.

We have similar refresh policies set up for other apps and they appear to be working.

Any suggestions on what could be going on here?

Answer 1

You really can't target DevOps alone.

You have to target the entre Azure mgmt API suite:

https://learn.microsoft.com/en-us/entra/identity/conditional-access/concept-conditional-access-cloud-apps#windows-azure-service-management-api

When you target the Windows Azure Service Management API application, policy is enforced for tokens issued to a set of services closely bound to the portal. This grouping includes the application IDs of:

• Azure Resource Manager
• Azure portal, which also covers the Microsoft Entra admin center
• Azure Data Lake
• Application Insights API
• Log Analytics API

Because the policy is applied to the Azure management portal and API, services, or clients with an Azure API service dependency, can indirectly be impacted. For example:

• Classic deployment model APIs
• Azure PowerShell
• Azure CLI
• Azure DevOps
• Azure Data Factory portal
• Azure Event Hubs
• Azure Service Bus
• Azure SQL Database
• SQL Managed Instance
• Azure Synapse
• Visual Studio subscriptions administrator portal
• Microsoft IoT Central

Note The Windows Azure Service Management API application applies to Azure PowerShell, which calls the Azure Resource Manager API. It does not apply to Microsoft Graph PowerShell, which calls the Microsoft Graph API Also: Session timeouts work best when targeting users, not apps per se ,but if you target the Azure Service API , I would think it will work for you , so I would test that out:

https://learn.microsoft.com/en-us/entra/identity/conditional-access/howto-conditional-access-session-lifetime#user-sign-in-frequency

Answer 2

Interesting. So I tried both things you recommended and neither block DevOps' persistence. I set a CAP for 1 day sign-in frequency targeting the Azure DevOps and Windows Azure Service Management API, and I set a separate policy that sets the same 1 day sign-in frequency but for a certain group of users regardless of app. The result is the same. In the sign-in logs, the sign-in is recorded as "Failed" with the same error message above, CAP shows that all both policies (app-specific and user-specific) were applied. And yet the DevOps session marches on undisturbed... Has anyway successfully used Conditional Access to increase the sign-in frequency for DevOps?

Answer 3

So I was curious about this and it appears there are scenarios where a CA policy is enforced with Dev Ops:https://learn.microsoft.com/en-us/azure/devops/organizations/accounts/change-application-access-policies?view=azure-devops#cap-support-on-azure-devops As I mentioned before the block works from what I have seen using Azure Mgmt as the target app, but I wonder if there is something being missed here

Answer 4

Hi @Sam Davidoff , if you're still having issues, can you please send me an email at "azcommunity@microsoft.com" with subject "ATTN: James Hamil" and your subscription ID? I can open a free support ticket for you.

Best,

James

Answer 5

It appears that ADO currently does not respect sign-in frequency, but that is expected to change soon:

https://github.com/MicrosoftDocs/azure-devops-docs/issues/13983