Hi Simon,
You can test your access to the secret by az cli before use in pipeline :
Using a personal account:
az keyvault secret show --name "<Your Secret Name>" --vault-name "<Your Key Vault Name>" --query value --output tsv
Using Service principal:
#Login with your Service principal
az login --service-principal -u <app-url> -p <password-or-cert> --tenant <tenant>
az keyvault secret show --name "<Your Secret Name>" --vault-name "<Your Key Vault Name>" --query value --output tsv
Let me know your output. if there is a problem with rbac permission you can add your user to Key Vault Administrator role , Please note that the Key Vault Administrator role only works for key vaults that use the ‘Azure role-based access control’ permission model.
https://learn.microsoft.com/en-us/azure/key-vault/general/rbac-guide?tabs=azure-cli
Cheers,
Luis