The Exchange Servers do not communicate with the ADFS servers.
The clients need to be able to connect on port 443 (and port 49443 for cert auth)
https://learn.microsoft.com/en-us/windows-server/identity/ad-fs/deployment/best-practices-securing-ad-fs#wap-and-users
Note only EAC and OWA support ADFS auth