This issue is more related to lifetime of tokens. You could change the LogonTokenCacheExpirationWindow to be less than the SAML TokenLifetime by the PowerShell command.
$sts = Get-SPSecurityTokenServiceConfig
$sts.LogonTokenCacheExpirationWindow = (New-TimeSpan –minutes 1)
$sts.Update()
iisreset
For more detailed information, you could refer to the article below.
The same client browser session has made '6' requests in the last '11' seconds.