@Jake Seaborn Thanks for reaching out.
That should not be the case normally unless you might be getting targeted with another conditional access policy.
Make sure you are targeted with just the same set of CA policy like others.
You can also review the tenant wide configuration and confirm the settings as per : https://learn.microsoft.com/en-us/azure/active-directory/authentication/concepts-azure-multi-factor-authentication-prompts-session-lifetime#review-your-tenant-configuration
If you still have the issue, we will need to review your sign ins for which you will need to contact us at azcommunity@microsoft.com with subject "Atten-Vipul".
After further checks, we can create a support ticket for you, if required.
-----------------------------------------------------------------------------------------------------------------
If the suggested response helped you resolve your issue, please do not forget to accept the response as Answer and "Up-Vote" for the answer that helped you for benefit of the community.