Hi ShahbazShaikh3
Thanks for the response.
Regarding restricting users access to the address book, for teams you may refer to Use Microsoft Teams scoped directory search - Microsoft Teams | Microsoft Docs, the scoped directory helps an organization set boundaries that control how users can find and communicate with other users in an organization.
With reference to adding restrictions in Sharepoint and Delve, these are out of my support boundary.
Kind Regards,
Momo