Edit

Tutorial: Create and use an Apache Kafka service for development

  • Article

Azure Container Apps allows you to connect to development and production-grade services to provide a wide variety of functionality to your applications.

In this tutorial, you learn to create and use a development Apache Kafka service.

Azure CLI commands and Bicep template fragments are featured in this tutorial. If you use Bicep, you can add all the fragments to a single Bicep file and deploy the template all at once.

  • Create a Container Apps environment to deploy your service and container app
  • Create an Apache Kafka service
  • Set up a command line app to use the dev Apache Kafka service
  • Deploy a kafka-ui app to view application data
  • Compile a final bicep template to deploy all resources using a consistent and predictable template deployment
  • Use an azd template for a one command deployment of all resources

Prerequisites

Note

For a one command deployment, skip to the last azd template step.

Setup

  1. Define variables for common values.

    RESOURCE_GROUP="kafka-dev"
LOCATION="northcentralus"
ENVIRONMENT="aca-env"
KAFKA_SVC="kafka01"
KAFKA_CLI_APP="kafka-cli-app"
KAFKA_UI_APP="kafka-ui-app"

  2. Log in to Azure.

    az login

  3. Upgrade the CLI to the latest version.

    az upgrade

  4. Upgrade Bicep to the latest version.

    az bicep upgrade

  5. Add th containerapp extension.

    az extension add --name containerapp --upgrade

  6. Register required namespaces.

    az provider register --namespace Microsoft.App

    az provider register --namespace Microsoft.OperationalInsights

Create a Container Apps environment

  1. Create a resource group.

    az group create \
    --name "$RESOURCE_GROUP" \
    --location "$LOCATION"

  2. Create a Container Apps environment.

    az containerapp env create \
  --name "$ENVIRONMENT" \
  --resource-group "$RESOURCE_GROUP" \
  --location "$LOCATION"

Create an Apache Kafka service

  1. Create an Apache Kafka service.

    ENVIRONMENT_ID=$(az containerapp env show \
  --name "$ENVIRONMENT" \
  --resource-group "$RESOURCE_GROUP" \
  --output tsv \
  --query id)

  2. Deploy the template.

        az containerapp add-on kafka create \
    --name "$KAFKA_SVC" \
    --resource-group "$RESOURCE_GROUP" \
    --environment "$ENVIRONMENT"

  3. View log output from the Kafka instance

    Use the logs command to view log messages.

    az containerapp logs show \
    --name $KAFKA_SVC \
    --resource-group $RESOURCE_GROUP \
    --follow --tail 30

    Screenshot of container app kafka service logs.

Create an app to test the service

When you create the app, you'll set it up to use ./kafka-topics.sh, ./kafka-console-producer.sh, and kafka-console-consumer.sh to connect to the Kafka instance.

  1. Create a kafka-cli-app app that binds to the Kafka service.

    az containerapp create \
    --name "$KAFKA_CLI_APP" \
    --image mcr.microsoft.com/k8se/services/kafka:3.4 \
    --bind "$KAFKA_SVC" \
    --environment "$ENVIRONMENT" \
    --resource-group "$RESOURCE_GROUP" \
    --min-replicas 1 \
    --max-replicas 1 \
    --command "/bin/sleep" "infinity"

  2. Run the CLI exec command to connect to the test app.

    az containerapp exec \
    --name $KAFKA_CLI_APP \
    --resource-group $RESOURCE_GROUP \
    --command /bin/bash

    When you use --bind or serviceBinds on the test app, the connection information is injected into the application environment. Once you connect to the test container, you can inspect the values using the env command.

    env | grep "^KAFKA_"

KAFKA_SECURITYPROTOCOL=SASL_PLAINTEXT
KAFKA_BOOTSTRAPSERVER=kafka01:9092
KAFKA_HOME=/opt/kafka
KAFKA_PROPERTIES_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka-user" password="7dw..." user_kafka-user="7dw..." ;
KAFKA_BOOTSTRAP_SERVERS=kafka01:9092
KAFKA_SASLUSERNAME=kafka-user
KAFKA_SASL_USER=kafka-user
KAFKA_VERSION=3.4.0
KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT
KAFKA_SASL_PASSWORD=7dw...
KAFKA_SASLPASSWORD=7dw...
KAFKA_SASL_MECHANISM=PLAIN
KAFKA_SASLMECHANISM=PLAIN

  3. Use kafka-topics.sh to create an event topic.

    Create a kafka.props file.

    echo "security.protocol=$KAFKA_SECURITY_PROTOCOL" >> kafka.props && \
echo "sasl.mechanism=$KAFKA_SASL_MECHANISM" >> kafka.props && \
echo "sasl.jaas.config=$KAFKA_PROPERTIES_SASL_JAAS_CONFIG" >> kafka.props

    Create a quickstart-events event topic.

    /opt/kafka/bin/kafka-topics.sh \
    --create --topic quickstart-events \
    --bootstrap-server $KAFKA_BOOTSTRAP_SERVERS \
    --command-config kafka.props
# Created topic quickstart-events.

/opt/kafka/bin/kafka-topics.sh \
    --describe --topic quickstart-events \
    --bootstrap-server $KAFKA_BOOTSTRAP_SERVERS \
    --command-config kafka.props
# Topic: quickstart-events	TopicId: lCkTKmvZSgSUCHozhhvz1Q	PartitionCount: 1	ReplicationFactor: 1	Configs: segment.bytes=1073741824
# Topic: quickstart-events	Partition: 0	Leader: 1	Replicas: 1	Isr: 1

  4. Use kafka-console-producer.sh to write events to the topic.

    /opt/kafka/bin/kafka-console-producer.sh \
    --topic quickstart-events \
    --bootstrap-server $KAFKA_BOOTSTRAP_SERVERS \
    --producer.config kafka.props

> this is my first event
> this is my second event
> this is my third event
> CTRL-C

    Note

    The ./kafka-console-producer.sh command prompts you to write events with >. Write some events as shown, then hit CTRL-C to exit.

  5. Use kafka-console-consumer.sh to read events from the topic.

    /opt/kafka/bin/kafka-console-consumer.sh \
     --topic quickstart-events \
    --bootstrap-server $KAFKA_BOOTSTRAP_SERVERS \
    --from-beginning \
    --consumer.config kafka.props

# this is my first event
# this is my second event
# this is my third event

Screenshot of container app kafka CLI output logs.

Using a dev service with an existing app

If you already have an app that uses Apache Kafka, you can change how connection information is loaded.

First, create the following environment variables.

KAFKA_HOME=/opt/kafka
KAFKA_PROPERTIES_SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="kafka-user" password="7dw..." user_kafka-user="7dw..." ;
KAFKA_BOOTSTRAP_SERVERS=kafka01:9092
KAFKA_SASL_USER=kafka-user
KAFKA_VERSION=3.4.0
KAFKA_SECURITY_PROTOCOL=SASL_PLAINTEXT
KAFKA_SASL_PASSWORD=7dw...
KAFKA_SASL_MECHANISM=PLAIN

Using the CLI (or Bicep) you can update the app to add --bind $KAFKA_SVC to use the dev service.

Binding to the dev service

Deploy kafka-ui to view and manage the Kafka instance.

See Bicep or azd example.

Screenshot of pgweb Container App connecting to Kafka service.

Deploy all resources

Use the following examples to if you want to deploy all resources at once.

Bicep

The following Bicep template contains all the resources in this tutorial.

You can create a kafka-dev.bicep file with this content.

targetScope = 'resourceGroup'
param location string = resourceGroup().location
param appEnvironmentName string = 'aca-env'
param kafkaSvcName string = 'kafka01'
param kafkaCliAppName string = 'kafka-cli-app'
param kafkaUiAppName string = 'kafka-ui'

resource logAnalytics 'Microsoft.OperationalInsights/workspaces@2022-10-01' = {
  name: '${appEnvironmentName}-log-analytics'
  location: location
  properties: {
    sku: {
      name: 'PerGB2018'
    }
  }
}

resource appEnvironment 'Microsoft.App/managedEnvironments@2023-04-01-preview' = {
  name: appEnvironmentName
  location: location
  properties: {
    appLogsConfiguration: {
      destination: 'log-analytics'
      logAnalyticsConfiguration: {
        customerId: logAnalytics.properties.customerId
        sharedKey: logAnalytics.listKeys().primarySharedKey
      }
    }
  }
}

resource kafka 'Microsoft.App/containerApps@2023-04-01-preview' = {
  name: kafkaSvcName
  location: location
  properties: {
    environmentId: appEnvironment.id
    configuration: {
      service: {
          type: 'kafka'
      }
    }
  }
}

resource kafkaCli 'Microsoft.App/containerApps@2023-04-01-preview' = {
  name: kafkaCliAppName
  location: location
  properties: {
    environmentId: appEnvironment.id
    template: {
      serviceBinds: [
        {
          serviceId: kafka.id
        }
      ]
      containers: [
        {
          name: 'kafka-cli'
          image: 'mcr.microsoft.com/k8se/services/kafka:3.4'
          command: [ '/bin/sleep', 'infinity' ]
        }
      ]
      scale: {
        minReplicas: 1
        maxReplicas: 1
      }
    }
  }
}

resource kafkaUi 'Microsoft.App/containerApps@2023-04-01-preview' = {
  name: kafkaUiAppName
  location: location
  properties: {
    environmentId: appEnvironment.id
    configuration: {
      ingress: {
        external: true
        targetPort: 8080
      }
    }
    template: {
      serviceBinds: [
        {
          serviceId: kafka.id
          name: 'kafka'
        }
      ]
      containers: [
        {
          name: 'kafka-ui'
          image: 'docker.io/provectuslabs/kafka-ui:latest'
          command: [
            '/bin/sh'
          ]
          args: [
            '-c'
            '''export KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS="$KAFKA_BOOTSTRAP_SERVERS" && \
            export KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG="$KAFKA_PROPERTIES_SASL_JAAS_CONFIG" && \
            export KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM="$KAFKA_SASL_MECHANISM" && \
            export KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL="$KAFKA_SECURITY_PROTOCOL" && \
            java $JAVA_OPTS -jar kafka-ui-api.jar'''
          ]
          resources: {
            cpu: json('1.0')
            memory: '2.0Gi'
          }
        }
      ]
    }
  }
}

output kafkaUiUrl string = 'https://${kafkaUi.properties.configuration.ingress.fqdn}'

output kafkaCliExec string = 'az containerapp exec -n ${kafkaCli.name} -g ${resourceGroup().name} --command /bin/bash'

output kafkaLogs string = 'az containerapp logs show -n ${kafka.name} -g ${resourceGroup().name} --follow --tail 30'

Use the Azure CLI to deploy it the template.

RESOURCE_GROUP="kafka-dev"
LOCATION="northcentralus"

az group create \
    --name "$RESOURCE_GROUP" \
    --location "$LOCATION"

az deployment group create -g $RESOURCE_GROUP \
    --query 'properties.outputs.*.value' \
    --template-file kafka-dev.bicep

Azure Developer CLI

A final template is available on GitHub.

Use azd up to deploy the template.

git clone https://github.com/Azure-Samples/aca-dev-service-kafka-azd
cd aca-dev-service-kafka-azd
azd up

Clean up resources

Once you're done, run the following command to delete the resource group that contains your Container Apps resources.

Caution

The following command deletes the specified resource group and all resources contained within it. If resources outside the scope of this tutorial exist in the specified resource group, they will also be deleted.

az group delete \
    --resource-group $RESOURCE_GROUP