The "assignment required" option won't help here, this is only useful when using an AAD app for authentication to your own applications.
You can restrict access to Key Vault further be blocking public access and accessing using a private endpoint. You would need to do the following:
- Create a Virtual Network
- Join the App Service to the virtual network using the regional vNet integration feature
- create a Private Endpoint attached to that vNet, and the Key Vault
- Disable public access to the Key Vault
By doing this, if someone got hold of your credentials, they would not be able to access the Key vault unless they were connected to the virtual network.