Exchange ImpersonatedUserId error on another MailBox

Question

A user B has a mailbox on which the administrator has given full read/write rights to user A
User A logs into outlook and adds mailbox B. Our outlook addin manages client side to consult emails from mailbox B, however we have a server side backend that uses EWS to get the full content of the message (HTML and parts attached).
This code works on the messages of box A but not on those of box B with the error below

We connect with the token obtained on the client side, we tested with the refreshed token when it is on box A but also while being on box B (with user A of course):


And we have a exception : 


ExchangeImpersonation SOAP header must not exist for this type of Oauth token