Conditional access in sign-in logs: why a "not included"?

Here is what we did:

• Bought a third-party web app.
• Installed the app in our internet-facing datacenter.
• Registered the app in our AAD tenant.
• Configured the client id and secret in the app configuration.
• Created a conditional access policy to require MFA + sign-in frequency (1 hour) for this app specifically (cloud app -> include -> select cloud app).

Here is what we observe:

• When users connect to the app, they are redirected to the Azure AD login screen.
• But they are only asked for a password.
• The app reacts correctly: users cannot log in directly, the sessions are established after successful password auth in Azure AD. Still, the MFA requirement is completely ignored.
• This was tested both with employee laptops (joined) and with two consultant laptops (neither registered nor joined) using both Chrome and Firefox both in incognito mode.
• When reviewing sign-in logs, we can see the user connection events, each successful access results in 2 rows: one with a status "Interrupted" and one with a status "Success".

The first event shows a status "Interrupted", with the following "authentication details":

• auth method: password
• auth method: password in the cloud
• succeeded: true
• result: correct password
• in the "conditional access" tab, it shows "not applicable"

The second event has a status "success":

• but "authentication details" is empty.
• the conditional access tab lists all conditional access policies in the tenant:
• No other CA policy was applied (they are either "not applied" or "disabled". For the moment, this is expected, each of them targets a specific cloud app.
• The policy we created for this app appears as "Not applied". When clicking "Show details", it shows that a message "Not Matched" over a "Not included". No other information is provided.

Here is what we verified:
Another CA policy is set, for another cloud app (Salesforce). 2FA + sign-in frequency are set, + joined device requirement. The CA policy activates correctly when an event is detected on the app that matches the identifier.

We also triple-checked the assignments in the CA policy for the app:

• the users accounts are included in the scope, there are no exclusions
• one cloud app is selected, and its identifier matches the one in the registered apps blade
• no additional filters are set.

We also checked whether the users had the correct license: they are MS365-E3 users with an Azure AD P1 license. AAD properties indicate a Premium P1 status.

We checked whether there were risky sign-ins or risky users, none are shown.

We tried switching the app to Twitter SSO instead of Azure. Setup was similar (create app, get clientid from Twitter). The users can log into the app, and even forced to Twitter's 2FA. We inferred that the app probably implements OAUTH2 flows correctly.

We are not the only customer of this app, and the editor confirms to us Azure AD auth works for its other customers.

We tested with the what-if tool. It shows the CA policy should apply when tested with data pulled from an actual sign-in log event with the following parameters set: user, ip, country, device platform (windows), client app (browser), device state (hybrid join), no filters.

The CA policy has no filters configured.

I am genuinely lost on what else to look into. The sign-in log show that the policy is not being applied, with a message "not matched" although it should match. Even the sign-in log events have the correct app name in the application name + the correct application id.

What am I missing? Why would this CA policy appear as "Not included / Not matched" in the sign-in logs? Is there some limitation in conditional access that I am not aware of? (e.g., incompatible with non-Microsoft OAUTH2 apps??) Any help greatly appreciated.