To me I suppose it is not possible.
The mobile number in Teams and Outlook is stored and extracted from Azure AD.
You may use some features like Address Book Policy or Information barrier to limit the read scope of the users in the XYZ OU.
But besides only hiding some of the contact information (like mobile phone number) to them, this would also prevent them from searching and communicating with other users.
If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.