The error message in the logs seem to be related to a stale Web Application Proxy server. If you have enabled Verbose audit, you should see an entry in the security eventlog too which will show the IP address of that WAP server (in case you don't know where it comes from).
You added the azure-ad-single-sign-on tag. Does this mean that Azure AD is involved in this too? Do you have the trust between Azure AD and Zoom or between ADFS and Zoom. If that's the first one, things works very differently and will also depends on your clients status (like is it a Windows 10 Hybrid-AD Join, etc...).
Regarding the SSO issue, if things get solved when you clear your browser cache, it is hard to see what could be wrong on the ADFS server without having a trace.
Ideally, take a Fiddler trace of a working and a non-working scenario and we will be able to see what's different and if that could be addressed at the ADFS level.