Internal Server Error connecting to O365 Exchange using OAuth

Question

Internal Server Error connecting to O365 Exchange using OAuth

Graham Jenkins 6

Hello

We have an app registered in Azure which allows our software to "import" emails from mailboxes into our software using OAuth; as well as send outbound emails via our software. Since yesterday, we have noticed that emails are getting duplicated in our system as well as connectivity errors for outbound emails.

We have tried various logging but the only error that is getting returned is "An internal server error occurred. The operation failed" (See Trace below) This is affecting multiple users across multiple sites connecting to their own O365 Exchange. Only affecting OAuth connections, Basic Authentication works fine. No updates to our software have been made.

Any ideas?

<Trace Tag="EwsResponse" Tid="24" Time="2022-08-24 14:19:16Z" Version="15.00.0913.015">
<?xml version="1.0" encoding="utf-8"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Header>
<Action s:mustUnderstand="1" xmlns="http://schemas.microsoft.com/ws/2005/05/addressing/none">*</Action>
</s:Header>
<s:Body>
<s:Fault>
<faultcode xmlns:a="http://schemas.microsoft.com/exchange/services/2006/types">a:ErrorInternalServerError</faultcode>
<faultstring xml:lang="en-US">An internal server error occurred. The operation failed.</faultstring>
<detail>
<e:ResponseCode xmlns:e="http://schemas.microsoft.com/exchange/services/2006/errors">ErrorInternalServerError</e:ResponseCode>
<e:Message xmlns:e="http://schemas.microsoft.com/exchange/services/2006/errors">An internal server error occurred. The operation failed.</e:Message>
</detail>
</s:Fault>
</s:Body>
</s:Envelope>
</Trace>

Joyce Shen - MSFT 16,541 Reputation points

2022-08-25T02:10:28.58+00:00

Hi @Graham Jenkins

According to your description above, your issue is more related to Azure side, so I would help you add the Azure related tags. Thanks for your understanding and hope your issue can be resolved soon!

https://learn.microsoft.com/en-us/answers/questions/861996/callback-from-oauth-server-results-in-500-internal.html

Chad Scott 31

We're also seeing this, also for many different customers. The problem appears to be pervasive throughout the service, yet Support has been completely unhelpful so far.

8/24/2022 18:53:30:261: curl debug: infotype=1: HTTP/1.1 500   
8/24/2022 18:53:30:261: curl debug: infotype=1: Server: Microsoft-IIS/10.0  
8/24/2022 18:53:30:261: curl debug: infotype=1: request-id: 355c8b2d-ee28-15fb-2347-01e1e9c48769  
8/24/2022 18:53:30:261: curl debug: infotype=1: Alt-Svc: h3=":443",h3-29=":443"  
8/24/2022 18:53:30:261: curl debug: infotype=1: X-CalculatedFETarget: CY5P221CU003.internal.outlook.com  
8/24/2022 18:53:30:261: curl debug: infotype=1: X-BackEndHttpStatus: 500  
8/24/2022 18:53:30:261: curl debug: infotype=1: X-CalculatedBETarget: CY4PR13MB1526.namprd13.PROD.OUTLOOK.COM  
8/24/2022 18:53:30:261: curl debug: infotype=1: X-BackEndHttpStatus: 500  
8/24/2022 18:53:30:276: curl debug: infotype=1: X-DiagInfo: CY4PR13MB1526  
8/24/2022 18:53:30:276: curl debug: infotype=1: X-BEServer: CY4PR13MB1526  
8/24/2022 18:53:30:276: curl debug: infotype=1: X-Proxy-RoutingCorrectness: 1  
8/24/2022 18:53:30:276: curl debug: infotype=1: X-Proxy-BackendServerStatus: 500  
8/24/2022 18:53:30:276: curl debug: infotype=1: X-FEProxyInfo: MW4PR04CA0259.NAMPRD04.PROD.OUTLOOK.COM  
8/24/2022 18:53:30:276: curl debug: infotype=1: X-FEServer: CY5P221CA0105  
8/24/2022 18:53:30:276: curl debug: infotype=1: X-FEServer: MW4PR04CA0259  
8/24/2022 18:53:30:276: TM Error: FindFolder, HTTP Response=500 (EWS: An internal server error occurred. The operation failed.)

Borsoi Emanuel 296 Reputation points

2022-08-26T05:39:00.913+00:00

Are you using EWS Api?
I have the same problem, without OAuth and Azure...
Chad Scott 31 Reputation points

2022-08-26T13:13:19.747+00:00

In our case, we are using (unmanaged) EWS with OAuth. I will be trying Basic Auth today and will report here if it works.
Chad Scott 31 Reputation points

2022-08-26T16:27:47.497+00:00

I can confirm what @Graham Jenkins discovered. The problem only happens for us with EWS + Modern Auth. Switching to Basic Auth with Application Impersonation allows our app to work normally again. Of course, that's a very temporary workaround since it requires degrading tenant security, and it won't work at all in several weeks, when Basic Auth is shut down for good.

It sure would be nice to hear confirmation that Microsoft is aware of this massive problem and is taking steps to remediate it.
Marco Ancona 6 Reputation points

2022-08-28T09:30:38.76+00:00

Our app logs show this error was (and is) affecting hundreds of users and it started last week.
Borsoi Emanuel 296 Reputation points

2022-08-29T06:24:22.007+00:00

I am working with EWS Api using username/passwrod + impersonation and It does throw error randomly.
Lukas Pokorny 6 Reputation points

2022-08-29T12:25:53.75+00:00

Several customers of our software libraries have reported many occurrences of what appears to be the same issue on August 24th and 25th . In all cases, the error occurs when accessing an O365 mailbox using EWS with app-only OAuth authentication. Eventually, after a while (usually several seconds, but sometimes minutes or even hours), the same EWS request that initially got rejected (with ErrorInternalServerError) succeeds.

We have been able to reproduce this issue by accessing a mailbox using app-only authentication that has not been accessed before (or that has not been accessed for several months). However, our customers reported encountering some mailboxes where the issue persisted for hours. One of the customers reported having used app-only authentication for years without ever encountering this error before August 24th

Based on our observations and other error reports here, we assume this is a server-side issue. However, some clarification from Microsoft would really be appreciated.
Borsoi Emanuel 296 Reputation points

2022-08-30T08:00:27.633+00:00

Last 48 hours with no errors, don't know if it it because it upgraded my project to .NET 4.8 or if Microsoft did something...
Dashore, Nitesh 1 Reputation point

2022-09-06T11:04:44.557+00:00

I am facing the same issue. I changes my authentication from Basic to Oauth but this error is coming only on the server.
Dashore, Nitesh 1 Reputation point

2022-09-06T11:09:14.553+00:00

Hey @Chad Scott , Have you got something regarding this issue? I am facing same issue with EWS + Oauth.
I am able to read mail box while locally debugging the code but error get recording on the server. I don't understand this.
Chad Scott 31 Reputation points

2022-09-06T13:05:22.297+00:00

The issue has been fixed for us. It was mostly resolved by early last week, but it took some time for things to propagate to all servers.
Dashore, Nitesh 1 Reputation point

2022-09-07T12:10:46.123+00:00

@Chad Scott I am still getting this error randomly, Should I create a support request on Microsoft for this ?

Cannot access mailbox for service account: Microsoft.Exchange.WebServices.Data.ServiceResponseException: An internal server error occurred. The operation failed.
Chad Scott 31 Reputation points

2022-09-07T12:13:05.683+00:00

After not seeing the issue for a week, I'm seeing it again, though far, far less frequently. Yeah, I'd recommend opening a support ticket and pointing back to this forum post and the others like it.
Mikhail 1 Reputation point

2022-09-07T18:14:53.417+00:00

I have the same issue.
And I discovered null prefix in token string in trace of EwsRequest:

...wsse:Security soap:mustUnderstand='1'> nulleyJ0eXAiOiJK...
Mikhail 1 Reputation point

2022-09-16T13:43:39.407+00:00
We have fixed this problem:

Get token with scope - https://outlook.office365.com/EWS.AccessAsUser.All

Add the token to headers: var service = new ExchangeService(ExchangeVersion.Exchange2010_SP2);
service.setUrl(URI.create("https://outlook.office365.com/ews/exchange.asmx"));
service.getHttpHeaders().put("Authorization", "Bearer " + token);

Joyce Shen - MSFT 16,541 Reputation points

2022-08-25T02:10:28.58+00:00

Hi @Graham Jenkins

According to your description above, your issue is more related to Azure side, so I would help you add the Azure related tags. Thanks for your understanding and hope your issue can be resolved soon!

https://learn.microsoft.com/en-us/answers/questions/861996/callback-from-oauth-server-results-in-500-internal.html
Chad Scott 31 Reputation points

2022-08-25T02:46:30.22+00:00

We're also seeing this, also for many different customers. The problem appears to be pervasive throughout the service, yet Support has been completely unhelpful so far.

8/24/2022 18:53:30:261: curl debug: infotype=1: HTTP/1.1 500 8/24/2022 18:53:30:261: curl debug: infotype=1: Server: Microsoft-IIS/10.0 8/24/2022 18:53:30:261: curl debug: infotype=1: request-id: 355c8b2d-ee28-15fb-2347-01e1e9c48769 8/24/2022 18:53:30:261: curl debug: infotype=1: Alt-Svc: h3=":443",h3-29=":443" 8/24/2022 18:53:30:261: curl debug: infotype=1: X-CalculatedFETarget: CY5P221CU003.internal.outlook.com 8/24/2022 18:53:30:261: curl debug: infotype=1: X-BackEndHttpStatus: 500 8/24/2022 18:53:30:261: curl debug: infotype=1: X-CalculatedBETarget: CY4PR13MB1526.namprd13.PROD.OUTLOOK.COM 8/24/2022 18:53:30:261: curl debug: infotype=1: X-BackEndHttpStatus: 500 8/24/2022 18:53:30:276: curl debug: infotype=1: X-DiagInfo: CY4PR13MB1526 8/24/2022 18:53:30:276: curl debug: infotype=1: X-BEServer: CY4PR13MB1526 8/24/2022 18:53:30:276: curl debug: infotype=1: X-Proxy-RoutingCorrectness: 1 8/24/2022 18:53:30:276: curl debug: infotype=1: X-Proxy-BackendServerStatus: 500 8/24/2022 18:53:30:276: curl debug: infotype=1: X-FEProxyInfo: MW4PR04CA0259.NAMPRD04.PROD.OUTLOOK.COM 8/24/2022 18:53:30:276: curl debug: infotype=1: X-FEServer: CY5P221CA0105 8/24/2022 18:53:30:276: curl debug: infotype=1: X-FEServer: MW4PR04CA0259 8/24/2022 18:53:30:276: TM Error: FindFolder, HTTP Response=500 (EWS: An internal server error occurred. The operation failed.)
Borsoi Emanuel 296 Reputation points

2022-08-26T05:39:00.913+00:00

Are you using EWS Api?
I have the same problem, without OAuth and Azure...
Chad Scott 31 Reputation points

2022-08-26T13:13:19.747+00:00

In our case, we are using (unmanaged) EWS with OAuth. I will be trying Basic Auth today and will report here if it works.
Chad Scott 31 Reputation points

2022-08-26T16:27:47.497+00:00

I can confirm what @Graham Jenkins discovered. The problem only happens for us with EWS + Modern Auth. Switching to Basic Auth with Application Impersonation allows our app to work normally again. Of course, that's a very temporary workaround since it requires degrading tenant security, and it won't work at all in several weeks, when Basic Auth is shut down for good.

It sure would be nice to hear confirmation that Microsoft is aware of this massive problem and is taking steps to remediate it.
Marco Ancona 6 Reputation points

2022-08-28T09:30:38.76+00:00

Our app logs show this error was (and is) affecting hundreds of users and it started last week.
Borsoi Emanuel 296 Reputation points

2022-08-29T06:24:22.007+00:00

I am working with EWS Api using username/passwrod + impersonation and It does throw error randomly.
Lukas Pokorny 6 Reputation points

2022-08-29T12:25:53.75+00:00

Several customers of our software libraries have reported many occurrences of what appears to be the same issue on August 24th and 25th . In all cases, the error occurs when accessing an O365 mailbox using EWS with app-only OAuth authentication. Eventually, after a while (usually several seconds, but sometimes minutes or even hours), the same EWS request that initially got rejected (with ErrorInternalServerError) succeeds.

We have been able to reproduce this issue by accessing a mailbox using app-only authentication that has not been accessed before (or that has not been accessed for several months). However, our customers reported encountering some mailboxes where the issue persisted for hours. One of the customers reported having used app-only authentication for years without ever encountering this error before August 24th

Based on our observations and other error reports here, we assume this is a server-side issue. However, some clarification from Microsoft would really be appreciated.
Borsoi Emanuel 296 Reputation points

2022-08-30T08:00:27.633+00:00

Last 48 hours with no errors, don't know if it it because it upgraded my project to .NET 4.8 or if Microsoft did something...
Dashore, Nitesh 1 Reputation point

2022-09-06T11:04:44.557+00:00

I am facing the same issue. I changes my authentication from Basic to Oauth but this error is coming only on the server.
Dashore, Nitesh 1 Reputation point

2022-09-06T11:09:14.553+00:00

Hey @Chad Scott , Have you got something regarding this issue? I am facing same issue with EWS + Oauth.
I am able to read mail box while locally debugging the code but error get recording on the server. I don't understand this.
Chad Scott 31 Reputation points

2022-09-06T13:05:22.297+00:00

The issue has been fixed for us. It was mostly resolved by early last week, but it took some time for things to propagate to all servers.
Dashore, Nitesh 1 Reputation point

2022-09-07T12:10:46.123+00:00

@Chad Scott I am still getting this error randomly, Should I create a support request on Microsoft for this ?

Cannot access mailbox for service account: Microsoft.Exchange.WebServices.Data.ServiceResponseException: An internal server error occurred. The operation failed.
Chad Scott 31 Reputation points

2022-09-07T12:13:05.683+00:00

After not seeing the issue for a week, I'm seeing it again, though far, far less frequently. Yeah, I'd recommend opening a support ticket and pointing back to this forum post and the others like it.
Mikhail 1 Reputation point

2022-09-07T18:14:53.417+00:00

I have the same issue.
And I discovered null prefix in token string in trace of EwsRequest:

...wsse:Security soap:mustUnderstand='1'> nulleyJ0eXAiOiJK...
Mikhail 1 Reputation point

2022-09-16T13:43:39.407+00:00

We have fixed this problem:

Get token with scope - https://outlook.office365.com/EWS.AccessAsUser.All

Add the token to headers: var service = new ExchangeService(ExchangeVersion.Exchange2010_SP2);
service.setUrl(URI.create("https://outlook.office365.com/ews/exchange.asmx"));
service.getHttpHeaders().put("Authorization", "Bearer " + token);

Internal Server Error connecting to O365 Exchange using OAuth

Activity