Intermittent authorisation on Key Vault

Question

I have a deployed Azure Function that accesses a key vault, I'm using the 3 environmental variables of AZURE_CLIENT_ID and AZURE_CLIENT_SECRET and AZURE_TENANT_ID to use the DefaultAzureCredential of azure.identity from the python SDK. I've also enabled the identity of the Azure Function and added it to the access policy on the key vault.

I should say that I have (what I believe to be) the exact same set-up working properly on separate key vault and functions/webapps - but for my current project it needs to be a new key vault.

Of 100 hits of the function, 10 succeed with 200's status responses and the correct values, 90 of them fail with 500 server errors of:

2020-09-09T08:20:14.933476669Z: [INFO] ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException: Result: Failure
2020-09-09T08:20:14.933482369Z: [INFO] Exception: KeyVaultErrorException: (Forbidden) The user, group or application 'appid=,#redacted#.;oid=#redacted#;iss=https://sts.windows.net/#redacted#/' does not have secrets get permission on key vault '#redacted;location=uksouth'. For help resolving this issue, please see https://go.microsoft.com/fwlink/?linkid=2125287

So in summary, 10 times it was authorised, hit the keyvault for a secret it needed and returned 200's, 90 times it was not authorised. This is where I'm struggling, it either is or it isn't!

As a final test, I ran the function another 100 times just now, same again 10 success, 90 fails - which seems fishy. Nothing within the function should be changing between executions.