Azure Key Vault integration with Jboss

Question

Azure Key Vault integration with Jboss

Henry He 1

We are going to use Azure to host our application service, also want to use Azure Key Vault to keep the sensitive data. I'm using azure-sdk-for-java library with managed identity credential to connect to Azure Key Vault. The key vault created and assigned the access to VM already. It's test ok on VM by java command, but when it's running as a JBoss module, we meet the problem. Please help to analyze the log trace and let me know what's the root cause.

*

Caused by: java.lang.RuntimeException: Couldn't acquire access token from IMDS, verify your objectId, clientId or msiResourceId
    at com.azure.identity.implementation.IdentityClient.lambda$authenticateToIMDSEndpoint$49(IdentityClient.java:1177)
    at reactor.core.publisher.MonoCallable.call(MonoCallable.java:92)
    at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:174)
    ... 35 more
Caused by: com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Cannot construct instance of `com.azure.identity.implementation.MSIToken` (no Creators, like default constructor, exist): cannot deserialize from Object value (no delegate- or property-based Creator)
 at [Source: (String)"{"access_token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Imwzc1EtNTBjQ0g0eEJWWkxIVEd3blNSNzY4MCIsImtpZCI6Imwzc1EtNTBjQ0g0eEJWWkxIVEd3blNSNzY4MCJ9.eyJhdWQiOiJodHRwczovL3ZhdWx0LmF6dXJlLm5ldCIsImlzcyI6Imh0dHBzOi8vc3RzLndpbmRvd3MubmV0LzYwYzNlMmQwLTM3MGUtNGY2OC05MGUyLWJmNzg3MTA4MGUzMy8iLCJpYXQiOjE2MzQ3MTM2NjIsIm5iZiI6MTYzNDcxMzY2MiwiZXhwIjoxNjM0ODAwMzYyLCJhaW8iOiJFMlpnWUlncyt2SGUvMGE3dXRMZmpUUFdNcng5Q3dBPSIsImFwcGlkIjoiMzdjMDA1NTktMTYxNS00NDlmLTgxYWEtZWQ0YzkxOTk5M2RlIiwiYXBwaWRhY3IiOiIyIiwiaWRwIj"[truncated 1166 chars]; line: 1, column: 2]
    at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:67)
    at com.fasterxml.jackson.databind.DeserializationContext.reportBadDefinition(DeserializationContext.java:1764)
    at com.fasterxml.jackson.databind.DatabindContext.reportBadDefinition(DatabindContext.java:400)
    at com.fasterxml.jackson.databind.DeserializationContext.handleMissingInstantiator(DeserializationContext.java:1209)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1415)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:362)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:195)
    at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4593)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3548)
    at com.azure.core.util.serializer.JacksonAdapter.deserialize(JacksonAdapter.java:269)
    at com.azure.identity.implementation.IdentityClient.lambda$authenticateToIMDSEndpoint$49(IdentityClient.java:1137)
    ... 37 more

*

**It seems a bug for class MSIToken no default constructor, so I download the source code and add the constructor. Then the access token can be got but run into another issue: Status code 401 **

*2021-10-20 08:06:53,187 INFO  [com.azure.security.keyvault.secrets.SecretService.getSecret] (ServerService Thread Pool -- 41) --> GET https://smbc-keyvault.vault.azure.net/secrets/fscs/?api-version=REDACTEDTry count: 1(empty body)--> END GET
2021-10-20 08:06:53,312 INFO  [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version 5.0.8.Final-redhat-1
2021-10-20 08:06:53,347 INFO  [org.jboss.as.security] (MSC service thread 1-4) WFLYSEC0001: Current PicketBox version=5.0.3.Final-redhat-3
2021-10-20 08:06:53,410 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) WFLYJCA0018: Started Driver service with driver-name = h2
2021-10-20 08:06:53,503 INFO  [org.jboss.as.naming] (MSC service thread 1-4) WFLYNAM0003: Starting Naming Service
2021-10-20 08:06:53,536 INFO  [org.jboss.as.ejb3] (MSC service thread 1-2) WFLYEJB0482: Strict pool mdb-strict-max-pool is using a max instance size of 8 (per class), which is derived from the number of CPUs on this host.
2021-10-20 08:06:53,536 INFO  [org.jboss.as.ejb3] (MSC service thread 1-2) WFLYEJB0481: Strict pool slsb-strict-max-pool is using a max instance size of 32 (per class), which is derived from thread worker pool sizing.
2021-10-20 08:06:53,538 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-2) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]
2021-10-20 08:06:53,611 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0012: Started server default-server.
2021-10-20 08:06:53,628 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0018: Host default-host starting
2021-10-20 08:06:53,829 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0006: Undertow HTTP listener default listening on 127.0.0.1:8080
2021-10-20 08:06:53,987 INFO  [org.jboss.as.ejb3] (MSC service thread 1-2) WFLYEJB0493: EJB subsystem suspension complete
2021-10-20 08:06:55,488 INFO  [com.azure.identity.ManagedIdentityCredential] (OkHttp https://smbc-keyvault.vault.azure.net/...) Azure Identity => Managed Identity environment: AZURE VM IMDS ENDPOINT
2021-10-20 08:06:55,506 INFO  [com.azure.identity.ManagedIdentityCredential] (OkHttp https://smbc-keyvault.vault.azure.net/...) Azure Identity => getToken() result for scopes [https://vault.azure.net/.default]: SUCCESS
2021-10-20 08:06:55,516 INFO  [com.azure.security.keyvault.secrets.SecretService.getSecret] (OkHttp https://smbc-keyvault.vault.azure.net/...) --> GET https://smbc-keyvault.vault.azure.net/secrets/fscs/?api-version=REDACTEDTry count: 1(empty body)--> END GET
2021-10-20 08:06:55,524 INFO  [com.azure.security.keyvault.secrets.SecretService.getSecret] (OkHttp https://smbc-keyvault.vault.azure.net/...) <-- 401 https://smbc-keyvault.vault.azure.net/secrets/fscs/?api-version=REDACTED (1958 ms, 97-byte body)Response body:{"error":{"code":"Unauthorized","message":"AKV10000: Request is missing a Bearer or PoP token."}}<-- END HTTP
2021-10-20 08:06:55,906 WARN  [com.azure.security.keyvault.secrets.SecretAsyncClient] (OkHttp https://smbc-keyvault.vault.azure.net/...) Failed to get secret - fscs
Status code 401, "{"error":{"code":"Unauthorized","message":"[BearerReadAccessTokenFailed] Error validating token: 'S2S12005'."}}"
2021-10-20 08:06:55,956 INFO  [stdout] (ServerService Thread Pool -- 41) 08:06:55.926 [ServerService Thread Pool -- 41] ERROR com.pti.waf.ext.azure.AzureKeyVault - Status code 401, "{"error":{"code":"Unauthorized","message":"[BearerReadAccessTokenFailed] Error validating token: 'S2S12005'."}}"
2021-10-20 08:06:55,966 INFO  [com.azure.security.keyvault.secrets.SecretService.getSecret] (OkHttp https://smbc-keyvault.vault.azure.net/...) <-- 401 https://smbc-keyvault.vault.azure.net/secrets/fscs/?api-version=REDACTED (336 ms, 111-byte body)Response body:{"error":{"code":"Unauthorized","message":"[BearerReadAccessTokenFailed] Error validating token: 'S2S12005'."}}<-- END HTTP
2021-10-20 08:06:55,966 INFO  [stdout] (ServerService Thread Pool -- 41) com.azure.core.exception.HttpResponseException: Status code 401, "{"error":{"code":"Unauthorized","message":"[BearerReadAccessTokenFailed] Error validating token: 'S2S12005'."}}"*

JamesTran-MSFT 37,226 Reputation points Microsoft Employee Moderator

2021-10-21T18:55:24.9+00:00
@Henry He
Thank you for your post!

Based off the second log, it looks like you're getting a "code":"Unauthorized","message":"[BearerReadAccessTokenFailed] Error validating token... error message. In order to gain a better understanding of your issue:

From the first log Couldn't acquire access token from IMDS, verify your objectId, clientId or msiResourceId, can you make sure that you have the correct variables set?

Can you share any documentation you're following?

Any additional information is greatly appreciated!

I also tried decoding your access token and wasn't able to, so it looks like the error from the second log is correct - relating to the token you're getting.

If you have any other questions, please let me know.
Thank you for your time and patience throughout this issue.
Henry He 1 Reputation point

2021-10-22T03:20:07.237+00:00
Appreciate for you reply.
For your questions:

I'm sure with the correct settings for Key vault access, since I use the same code successfully access the key vault by running in standard JDK environment (by using whatever client secret credential or managed identity can work fine). And also it's successful running on JBoss by using client secret credential, but if I changed to System Managed Identity, it didn't work and running into above errors.

I really didn't find any documentation from the internet on how does Azure key vault integrate with JBoss. What my reference are
a). JBoss documentation : https://access.redhat.com/documentation/en-us/red_hat_jboss_enterprise_application_platform/7.4/html-single/how_to_configure_server_security/index#configure_red_hat_jboss_enterprise_application_platform_to_use_a\_custom_implementation_of_the_password_vault
I follow the section 3.2.9 to develop a customized SecurityVault by using azure-sdk-for-java library to retrieve value from Azure key vault b). Use a Windows VM system-assigned managed identity to access Azure Key Vault: https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/tutorial-windows-vm-access-nonaad
I'm using the official SDK in my code, and added the correct dependency 3rd party libraries.

Then I deploy the code into JBoss 7.2 EAP (openjdk 1.8) as an extra module and use it to retrieve password from Azure key vault in Jboss config file.
Henry He 1 Reputation point

2021-10-22T04:57:49.52+00:00

In addition, appreciate you can tell me if any variable can be use to tune my program by using ManagedIdentityCredential ?

1 answer

Your answer

JamesTran-MSFT 37,226 Reputation points Microsoft Employee Moderator

2021-10-21T18:55:24.9+00:00

@Henry He
Thank you for your post!

Based off the second log, it looks like you're getting a "code":"Unauthorized","message":"[BearerReadAccessTokenFailed] Error validating token... error message. In order to gain a better understanding of your issue:

From the first log Couldn't acquire access token from IMDS, verify your objectId, clientId or msiResourceId, can you make sure that you have the correct variables set?

Can you share any documentation you're following?

Any additional information is greatly appreciated!

I also tried decoding your access token and wasn't able to, so it looks like the error from the second log is correct - relating to the token you're getting.

If you have any other questions, please let me know.
Thank you for your time and patience throughout this issue.
Henry He 1 Reputation point

2021-10-22T03:20:07.237+00:00

Appreciate for you reply.
For your questions:

I'm sure with the correct settings for Key vault access, since I use the same code successfully access the key vault by running in standard JDK environment (by using whatever client secret credential or managed identity can work fine). And also it's successful running on JBoss by using client secret credential, but if I changed to System Managed Identity, it didn't work and running into above errors.

I really didn't find any documentation from the internet on how does Azure key vault integrate with JBoss. What my reference are
a). JBoss documentation : https://access.redhat.com/documentation/en-us/red_hat_jboss_enterprise_application_platform/7.4/html-single/how_to_configure_server_security/index#configure_red_hat_jboss_enterprise_application_platform_to_use_a\_custom_implementation_of_the_password_vault
I follow the section 3.2.9 to develop a customized SecurityVault by using azure-sdk-for-java library to retrieve value from Azure key vault b). Use a Windows VM system-assigned managed identity to access Azure Key Vault: https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/tutorial-windows-vm-access-nonaad
I'm using the official SDK in my code, and added the correct dependency 3rd party libraries.

Then I deploy the code into JBoss 7.2 EAP (openjdk 1.8) as an extra module and use it to retrieve password from Azure key vault in Jboss config file.
Henry He 1 Reputation point

2021-10-22T04:57:49.52+00:00

In addition, appreciate you can tell me if any variable can be use to tune my program by using ManagedIdentityCredential ?

Answer 1

@Henry He
Thank you for the quick follow up!

Since you're successfully able to retrieve the Key Vault secret on JBoss by using client secret credentials but are running into a "code":"Unauthorized","message":"[BearerReadAccessTokenFailed] Error validating token..., error message after changing to a System Managed Identity, can you make sure that you added the identity to your Key Vault access policies?

Additional Links:
Access Key Vault secret when deploying Azure Managed Applications
Azure Managed Application with Managed Identity

Lastly, since you're following the Red Hat JBOSS Enterprise Application Platform doc to implement this, I'd recommend reaching out to the Red Hat Community as well to see if they can provide any additional insights on this.

If you have any other questions, please let me know.
Thank you for your time and patience throughout this issue.

Share via

Azure Key Vault integration with Jboss

1 answer

Your answer