Thank you for your post!
As mentioned by AlanKinane, specific user accounts will not have permission to access the Key Vault if they aren't added to the Access Policies. In your specific scenario, your app's Service Principal will only be able to access the Key Vault.
If you have any other questions, please let me know.
Thank you for your time and patience throughout this issue.
Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.