@AdamCheng-8231
Thank you for your post!
As mentioned by AlanKinane, specific user accounts will not have permission to access the Key Vault if they aren't added to the Access Policies. In your specific scenario, your app's Service Principal will only be able to access the Key Vault.
Example: The Key Vault request operation flow with authentication
Additional Links:
Key Vault authentication options
Access model overview
If you have any other questions, please let me know.
Thank you for your time and patience throughout this issue.
----------
Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.