Edit

Migrate an application to use passwordless connections with Azure Event Hubs for Kafka

  • Article

This article explains how to migrate from traditional authentication methods to more secure, passwordless connections with Azure Event Hubs for Kafka.

Application requests to Azure Event Hubs for Kafka must be authenticated. Azure Event Hubs for Kafka provides different ways for apps to connect securely. One of the ways is to use a connection string. However, you should prioritize passwordless connections in your applications when possible.

Passwordless connections are supported since Spring Cloud Azure 4.3.0. This article is a migration guide for removing credentials from Spring Cloud Stream Kafka applications.

Compare authentication options

When the application authenticates with Azure Event Hubs for Kafka, it provides an authorized entity to connect the Event Hubs namespace. Apache Kafka protocols provide multiple Simple Authentication and Security Layer (SASL) mechanisms for authentication. According to the SASL mechanisms, there are two authentication options that you can use to authorize access to your secure resources: Microsoft Entra authentication and Shared Access Signature (SAS) authentication.

Microsoft Entra authentication

Microsoft Entra authentication is a mechanism for connecting to Azure Event Hubs for Kafka using identities defined in Microsoft Entra ID. With Microsoft Entra authentication, you can manage service principal identities and other Microsoft services in a central location, which simplifies permission management.

Using Microsoft Entra ID for authentication provides the following benefits:

  • Authentication of users across Azure services in a uniform way.
  • Management of password policies and password rotation in a single place.
  • Multiple forms of authentication supported by Microsoft Entra ID, which can eliminate the need to store passwords.
  • Customers can manage Event Hubs permissions using external (Microsoft Entra ID) groups.
  • Support for token-based authentication for applications connecting to Azure Event Hubs for Kafka.

SAS authentication

Event Hubs also provides Shared Access Signatures (SAS) for delegated access to Event Hubs for Kafka resources.

Although it's possible to connect to Azure Event Hubs for Kafka with SAS, it should be used with caution. You must be diligent to never expose the connection strings in an unsecure location. Anyone who gains access to the connection strings is able to authenticate. For example, there's a risk that a malicious user can access the application if a connection string is accidentally checked into source control, sent through an unsecure email, pasted into the wrong chat, or viewed by someone who shouldn't have permission. Instead, authorizing access using the OAuth 2.0 token-based mechanism provides superior security and ease of use over SAS. Consider updating your application to use passwordless connections.

Introducing passwordless connections

With a passwordless connection, you can connect to Azure services without storing any credentials in the application code, its configuration files, or in environment variables.

Many Azure services support passwordless connections, for example via Azure Managed Identity. These techniques provide robust security features that you can implement using DefaultAzureCredential from the Azure Identity client libraries. In this tutorial, you'll learn how to update an existing application to use DefaultAzureCredential instead of alternatives such as connection strings.

DefaultAzureCredential supports multiple authentication methods and automatically determines which should be used at runtime. This approach enables your app to use different authentication methods in different environments (local dev vs. production) without implementing environment-specific code.

The order and locations in which DefaultAzureCredential searches for credentials can be found in the Azure Identity library overview. For example, when working locally, DefaultAzureCredential will generally authenticate using the account the developer used to sign in to Visual Studio. When the app is deployed to Azure, DefaultAzureCredential will automatically switch to use a managed identity. No code changes are required for this transition.

To ensure that connections are passwordless, you must take into consideration both local development and the production environment. If a connection string is required in either place, then the application isn't passwordless.

In your local development environment, you can authenticate with Azure CLI, Azure PowerShell, Visual Studio, or Azure plugins for Visual Studio Code or IntelliJ. In this case, you can use that credential in your application instead of configuring properties.

When you deploy applications to an Azure hosting environment, such as a virtual machine, you can assign managed identity in that environment. Then, you won't need to provide credentials to connect to Azure services.

Note

A managed identity provides a security identity to represent an app or service. The identity is managed by the Azure platform and does not require you to provision or rotate any secrets. You can read more about managed identities in the overview documentation.

Migrate an existing application to use passwordless connections

The following steps explain how to migrate an existing application to use passwordless connections instead of a SAS solution.

0) Prepare the working environment for local development authentication

First, use the following command to set up some environment variables.

export AZ_RESOURCE_GROUP=<YOUR_RESOURCE_GROUP>
export AZ_EVENTHUBS_NAMESPACE_NAME=<YOUR_EVENTHUBS_NAMESPACE_NAME>
export AZ_EVENTHUB_NAME=<YOUR_EVENTHUB_NAME>

Replace the placeholders with the following values, which are used throughout this article:

  • <YOUR_RESOURCE_GROUP>: The name of the resource group you'll use.
  • <YOUR_EVENTHUBS_NAMESPACE_NAME>: The name of the Azure Event Hubs namespace you'll use.
  • <YOUR_EVENTHUB_NAME>: The name of the event hub you'll use.

1) Grant permission for Azure Event Hubs

If you want to run this sample locally with Microsoft Entra authentication, be sure your user account has authenticated via Azure Toolkit for IntelliJ, Visual Studio Code Azure Account plugin, or Azure CLI. Also, be sure the account has been granted sufficient permissions.

  1. In the Azure portal, locate your Event Hubs namespace using the main search bar or left navigation.

  2. On the Event Hubs overview page, select Access control (IAM) from the left-hand menu.

  3. On the Access control (IAM) page, select the Role assignments tab.

  4. Select Add from the top menu and then Add role assignment from the resulting drop-down menu.

    Screenshot of Azure portal Access Control (IAM) page of Event Hubs Namespace resource with Add role assignment highlighted.

  5. Use the search box to filter the results to the desired role. For this example, search for Azure Event Hubs Data Sender and Azure Event Hubs Data Receiver and select the matching result and then choose Next.

  6. Under Assign access to, select User, group, or service principal, and then choose Select members.

  7. In the dialog, search for your Microsoft Entra username (usually your user@domain email address) and then choose Select at the bottom of the dialog.

  8. Select Review + assign to go to the final page, and then Review + assign again to complete the process.

For more information about granting access roles, see Authorize access to Event Hubs resources using Microsoft Entra ID.

2) Sign in and migrate the app code to use passwordless connections

For local development, make sure you're authenticated with the same Microsoft Entra account you assigned the role to on your Event Hubs. You can authenticate via the Azure CLI, Visual Studio, Azure PowerShell, or other tools such as IntelliJ.

Sign in to Azure through the Azure CLI by using the following command:

az login

Next, use the following steps to update your Spring Kafka application to use passwordless connections. Although conceptually similar, each framework uses different implementation details.

  1. Inside your project, open the pom.xml file and add the following reference:

    <dependency>
   <groupId>com.azure</groupId>
   <artifactId>azure-identity</artifactId>
   <version>1.6.0</version>
</dependency>

  2. After migration, implement AuthenticateCallbackHandler and OAuthBearerToken in your project for OAuth2 authentication, as shown in the following example.

    public class KafkaOAuth2AuthenticateCallbackHandler implements AuthenticateCallbackHandler {

   private static final Duration ACCESS_TOKEN_REQUEST_BLOCK_TIME = Duration.ofSeconds(30);
   private static final String TOKEN_AUDIENCE_FORMAT = "%s://%s/.default";

   private Function<TokenCredential, Mono<OAuthBearerTokenImp>> resolveToken;
   private final TokenCredential credential = new DefaultAzureCredentialBuilder().build();

   @Override
   public void configure(Map<String, ?> configs, String mechanism, List<AppConfigurationEntry> jaasConfigEntries) {
      TokenRequestContext request = buildTokenRequestContext(configs);
      this.resolveToken = tokenCredential -> tokenCredential.getToken(request).map(OAuthBearerTokenImp::new);
   }

   private TokenRequestContext buildTokenRequestContext(Map<String, ?> configs) {
      URI uri = buildEventHubsServerUri(configs);
      String tokenAudience = buildTokenAudience(uri);

      TokenRequestContext request = new TokenRequestContext();
      request.addScopes(tokenAudience);
      return request;
   }

   @SuppressWarnings("unchecked")
   private URI buildEventHubsServerUri(Map<String, ?> configs) {
      String bootstrapServer = Arrays.asList(configs.get(BOOTSTRAP_SERVERS_CONFIG)).get(0).toString();
      bootstrapServer = bootstrapServer.replaceAll("\\[|\\]", "");
      URI uri = URI.create("https://" + bootstrapServer);
      return uri;
   }

   private String buildTokenAudience(URI uri) {
      return String.format(TOKEN_AUDIENCE_FORMAT, uri.getScheme(), uri.getHost());
   }

   @Override
   public void handle(Callback[] callbacks) throws UnsupportedCallbackException {
      for (Callback callback : callbacks) {
         if (callback instanceof OAuthBearerTokenCallback) {
            OAuthBearerTokenCallback oauthCallback = (OAuthBearerTokenCallback) callback;
            this.resolveToken
                    .apply(credential)
                    .doOnNext(oauthCallback::token)
                    .doOnError(throwable -> oauthCallback.error("invalid_grant", throwable.getMessage(), null))
                    .block(ACCESS_TOKEN_REQUEST_BLOCK_TIME);
         } else {
            throw new UnsupportedCallbackException(callback);
         }
      }
   }

   @Override
   public void close() {
      // NOOP
   }
}

    public class OAuthBearerTokenImp implements OAuthBearerToken {
    private final AccessToken accessToken;
    private final JWTClaimsSet claims;

    public OAuthBearerTokenImp(AccessToken accessToken) {
        this.accessToken = accessToken;
        try {
            claims = JWTParser.parse(accessToken.getToken()).getJWTClaimsSet();
        } catch (ParseException exception) {
            throw new SaslAuthenticationException("Unable to parse the access token", exception);
        }
    }

    @Override
    public String value() {
        return accessToken.getToken();
    }

    @Override
    public Long startTimeMs() {
        return claims.getIssueTime().getTime();
    }

    @Override
    public long lifetimeMs() {
        return claims.getExpirationTime().getTime();
    }

    @Override
    public Set<String> scope() {
        // Referring to https://docs.microsoft.com/azure/active-directory/develop/access-tokens#payload-claims, the scp
        // claim is a String, which is presented as a space separated list.
        return Optional.ofNullable(claims.getClaim("scp"))
                .map(s -> Arrays.stream(((String) s)
                .split(" "))
                .collect(Collectors.toSet()))
                .orElse(null);
    }

    @Override
    public String principalName() {
        return (String) claims.getClaim("upn");
    }

    public boolean isExpired() {
        return accessToken.isExpired();
    }
}

  3. When you create your Kafka producer or consumer, add the configuration needed to support the SASL/OAUTHBEARER mechanism. The following examples show what your code should look like before and after migration. In both examples, replace the <eventhubs-namespace> placeholder with the name of your Event Hubs namespace.

    Before migration, your code should look like the following example:

    Properties properties = new Properties();
properties.put(CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, "<eventhubs-namespace>.servicebus.windows.net:9093");
properties.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL");
properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.put(SaslConfigs.SASL_MECHANISM, "PLAIN");
properties.put(SaslConfigs.SASL_JAAS_CONFIG,
        String.format("org.apache.kafka.common.security.plain.PlainLoginModule required username=\"$ConnectionString\" password=\"%s\";", connectionString));
return new KafkaProducer<>(properties);

    After migration, your code should look like the following example. In this example, replace the <path-to-your-KafkaOAuth2AuthenticateCallbackHandler> placeholder with the full class name for your implemented KafkaOAuth2AuthenticateCallbackHandler.

    Properties properties = new Properties();
properties.put(CommonClientConfigs.BOOTSTRAP_SERVERS_CONFIG, "<eventhubs-namespace>.servicebus.windows.net:9093");
properties.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, "SASL_SSL");
properties.put(SaslConfigs.SASL_MECHANISM, "OAUTHBEARER");
properties.put(SaslConfigs.SASL_JAAS_CONFIG, "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required");
properties.put(SaslConfigs.SASL_LOGIN_CALLBACK_HANDLER_CLASS, "<path-to-your-KafkaOAuth2AuthenticateCallbackHandler>");
return new KafkaProducer<>(properties);

Run the app locally

After making these code changes, run your application locally. The new configuration should pick up your local credentials, assuming you're logged into a compatible IDE or command line tool, such as the Azure CLI, Visual Studio, or IntelliJ. The roles you assigned to your local dev user in Azure will allow your app to connect to the Azure service locally.

3) Configure the Azure hosting environment

After your application is configured to use passwordless connections and it runs locally, the same code can authenticate to Azure services after it's deployed to Azure. For example, an application deployed to an Azure Spring Apps instance that has a managed identity assigned can connect to Azure Event Hubs for Kafka.

In this section, you'll execute two steps to enable your application to run in an Azure hosting environment in a passwordless way:

  • Assign the managed identity for your Azure hosting environment.
  • Assign roles to the managed identity.

Note

Azure also provides Service Connector, which can help you connect your hosting service with Event Hubs. With Service Connector to configure your hosting environment, you can omit the step of assigning roles to your managed identity because Service Connector will do it for you. The following section describes how to configure your Azure hosting environment in two ways: one via Service Connector and the other by configuring each hosting environment directly.

Important

Service Connector's commands require Azure CLI 2.41.0 or higher.

Assign the managed identity for your Azure hosting environment

The following steps show you how to assign a system-assigned managed identity for various web hosting services. The managed identity can securely connect to other Azure Services using the app configurations you set up previously.

  1. On the main overview page of your Azure App Service instance, select Identity from the navigation pane.

  2. On the System assigned tab, make sure to set the Status field to on. A system assigned identity is managed by Azure internally and handles administrative tasks for you. The details and IDs of the identity are never exposed in your code.

    Screenshot of Azure portal Identity page of App Service resource with System assigned tab showing and Status field highlighted.

You can also assign managed identity on an Azure hosting environment by using the Azure CLI.

You can assign a managed identity to an Azure App Service instance with the az webapp identity assign command, as shown in the following example.

export AZURE_MANAGED_IDENTITY_ID=$(az webapp identity assign \
    --resource-group $AZ_RESOURCE_GROUP \
    --name <app-service-name> \
    --query principalId \
    --output tsv)

Assign roles to the managed identity

Next, grant permissions to the managed identity you created to access your Event Hubs namespace. You can grant permissions by assigning a role to the managed identity, just like you did with your local development user.

If you connected your services using the Service Connector, you don't need to complete this step. The following necessary configurations were handled for you:

  • If you selected a managed identity when you created the connection, a system-assigned managed identity was created for your app and assigned the Azure Event Hubs Data Sender and Azure Event Hubs Data Receiver roles on the Event Hubs.

  • If you chose to use a connection string, the connection string was added as an app environment variable.

Test the app

After making these code changes, browse to your hosted application in the browser. Your app should be able to connect to the Azure Event Hubs for Kafka successfully. Keep in mind that it may take several minutes for the role assignments to propagate through your Azure environment. Your application is now configured to run both locally and in a production environment without the developers having to manage secrets in the application itself.

Next steps

In this tutorial, you learned how to migrate an application to passwordless connections.

You can read the following resources to explore the concepts discussed in this article in more depth: