Exchange calendar syc with Teams produce a lot of timeouts

Question

Exchange calendar syc with Teams produce a lot of timeouts

Helmut Sz 1

Setup:
Exchange Hybrid - 2 Server Cluster (DAG) running Exchange 2016 CU22
Mostly on-prem mailboxes with some mailboxes in ExO.

Description:
So for about 2 months now, I am facing massive sync issues with the Teams calendar.
Aproximately 90% of the sync requests time out after 15 seconds (that seems to be the threshhold Teams is willing to wait for a calendar request answer from Exchange).
The calendar will sync eventually but only after some to a lot of timeouts. This happens for every user.
I'm trying to figure out where the bottleneck might be. As it is basically working but just not nearly reliably enough, the configuration seems to be correct in general (otherwise it wouldn't work at all I guess).
Also when using the Microsoft Remote Connectivity Analyzer - Teams Calendar Tab check, all checks succeed except for:
Sending user properties settings request to the Teams middle-tier service.
"The Teams middle-tier service endpoint did not return a valid response. > The request to the user properties settings service timed out. Please retry. Timeout: 15 seconds"
Also the Exchange Servers themselves don't seem to be overly busy when checking the ressource monitor.
The Outlook calendar sync don't have any problem.

I already tried a lot of things and would realy appreciate any suggetions that can point me in the right direction.

Thank you for your suggestions

Yuki Sun-MSFT 27,096 Reputation points

2022-02-04T05:33:41.253+00:00

Hi @Helmut Sz ,

So for about 2 months now, I am facing massive sync issues with the Teams calendar.

Do you mean it was working fine 2 months ago? If this is true, any changes were made in your organization right before the issue started? Like installing updates on Exchange servers?
How about Teams web client, is it working there?
Regarding the request to the user properties settings, can it finally reach success after several timeouts?

Besides, if you have any firewall configured in your environment, you can also have a check to see if any settings there could impact the requests from Microsoft 365.
Helmut Sz 1 Reputation point

2022-02-04T05:47:12.523+00:00

Hi @Yuki Sun-MSFT ,

thank you for your answer.

Do you mean it was working fine 2 months ago?

Yes, it was working perfectly.

If this is true, any changes were made in your organization right before the issue started? Like installing updates on Exchange servers?

Yes, as it is usual with on-prem Exchange servers, it is unavoidbale to patch then regularly because of security issues as well as that was around the time when generally a lot of systems had to be patched because of log4j.
Still, I couldn't find any hint in the Serverpatch notes that would suggest such problems beeing caused by the updates.

Regarding the request to the user properties settings, can it finally reach success after several timeouts?

Yes, it will sync eventually and work, sometimes there is one timeout, sometimes there are 20 but given enough time, teh request will not timeout and succeed.

Besides, if you have any firewall configured in your environment, you can also have a check to see if any settings there could impact the requests from Microsoft 365.

We double and tripple checked the firewall settings and it does not look like the problem is with the firewall.

Also tracing the network packages I can see that the answer from Exchange to Teams is under way but is cut off by Teams when it doesn't complete after 15 seconds.
Yuki Sun-MSFT 27,096 Reputation points

2022-02-08T09:03:32.703+00:00

Hi @Helmut Sz ,

Have you had a chance to test in different networks to see if there would be any improvement?
And does it working fine when checking calendar in Teams web client(https://teams.microsoft.com)?
Helmut Sz 1 Reputation point

2022-02-08T09:10:44.373+00:00

Hi @Yuki Sun-MSFT ,

Yes I tried it from a different network and also with the web client from the same and another network but it's still the same problem. The only calendars that don't have any problems are the ExO hosted mailboxes but that was to be expected as their calendars are also on ExO and therefore circumvent the whole on-prem infrastructure.
Michel de Rooij 1,521 Reputation points MVP

2022-02-08T23:14:56.72+00:00

Any chance you are using SSL inspection devices or drivers/software?
Did you list the M365 IP ranges to bypass any inspection?
https://learn.microsoft.com/en-us/microsoft-365/enterprise/urls-and-ip-address-ranges?view=o365-worldwide

Yuki Sun-MSFT 27,096 Reputation points

2022-02-04T05:33:41.253+00:00

Hi @Helmut Sz ,

So for about 2 months now, I am facing massive sync issues with the Teams calendar.

Do you mean it was working fine 2 months ago? If this is true, any changes were made in your organization right before the issue started? Like installing updates on Exchange servers?
How about Teams web client, is it working there?
Regarding the request to the user properties settings, can it finally reach success after several timeouts?

Besides, if you have any firewall configured in your environment, you can also have a check to see if any settings there could impact the requests from Microsoft 365.
Helmut Sz 1 Reputation point

2022-02-04T05:47:12.523+00:00

Hi @Yuki Sun-MSFT ,

thank you for your answer.

Do you mean it was working fine 2 months ago?

Yes, it was working perfectly.

If this is true, any changes were made in your organization right before the issue started? Like installing updates on Exchange servers?

Yes, as it is usual with on-prem Exchange servers, it is unavoidbale to patch then regularly because of security issues as well as that was around the time when generally a lot of systems had to be patched because of log4j.
Still, I couldn't find any hint in the Serverpatch notes that would suggest such problems beeing caused by the updates.

Regarding the request to the user properties settings, can it finally reach success after several timeouts?

Yes, it will sync eventually and work, sometimes there is one timeout, sometimes there are 20 but given enough time, teh request will not timeout and succeed.

Besides, if you have any firewall configured in your environment, you can also have a check to see if any settings there could impact the requests from Microsoft 365.

We double and tripple checked the firewall settings and it does not look like the problem is with the firewall.

Also tracing the network packages I can see that the answer from Exchange to Teams is under way but is cut off by Teams when it doesn't complete after 15 seconds.
Yuki Sun-MSFT 27,096 Reputation points

2022-02-08T09:03:32.703+00:00

Hi @Helmut Sz ,

Have you had a chance to test in different networks to see if there would be any improvement?
And does it working fine when checking calendar in Teams web client(https://teams.microsoft.com)?
Helmut Sz 1 Reputation point

2022-02-08T09:10:44.373+00:00

Hi @Yuki Sun-MSFT ,

Yes I tried it from a different network and also with the web client from the same and another network but it's still the same problem. The only calendars that don't have any problems are the ExO hosted mailboxes but that was to be expected as their calendars are also on ExO and therefore circumvent the whole on-prem infrastructure.
Michel de Rooij 1,521 Reputation points MVP

2022-02-08T23:14:56.72+00:00

Any chance you are using SSL inspection devices or drivers/software?
Did you list the M365 IP ranges to bypass any inspection?
https://learn.microsoft.com/en-us/microsoft-365/enterprise/urls-and-ip-address-ranges?view=o365-worldwide

Exchange calendar syc with Teams produce a lot of timeouts

Activity