Looks like it has bugs on DefaultAzureCredential class, especially the VisualStudioCredential class being used internally. According to https://learn.microsoft.com/en-us/dotnet/api/overview/azure/identity-readme, I replaced the DefaultAzureCredential with ChainedTokenCredential, the key value secret was successfully retrieved.
Sample code:
var cred = new ChainedTokenCredential(new ManagedIdentityCredential(), new AzureCliCredential());
SecretClient client = new SecretClient(new Uri(keyvaultUri), cred);
Response<KeyVaultSecret> secret = await client.GetSecretAsync("kv-sec-test");
For using AzureCliCredential, login with azure cli is required. The details can be found in above link.