VS is notorious for playing up with logged on credentials. I ended up creating a service principal that can access the resources I needed and specyfing its id and password in environmental variables on my dev machine. The variables are:
AZURE_TENANT_ID
AZURE_CLIENT_ID
AZURE_CLIENT_SECRET
DefaultAzureCredential works perfectly fine with them.