Edit

Add Apache Kafka as source in Fabric Real-Time hub (preview)

This article describes how to add Apache Kafka as an event source in Fabric Real-Time hub.

Prerequisites

  • Access to the Fabric workspace with Contributor or above permissions.

  • An Apache Kafka cluster running.

  • Your Apache Kafka must be publicly accessible and not be behind a firewall or secured in a virtual network. If it resides in a protected network, connect to it by using Eventstream connector virtual network injection.

  • If you plan to use TLS/mTLS settings, make sure the required certificates are available in an Azure Key Vault:

    • Import the required certificates into Azure Key Vault in .pem format.
    • The user who configures the source and previews data must have permission to access the certificates in the Key Vault (for example, Key Vault Certificate User or Key Vault Administrator).
    • If the current user doesn’t have the required permissions, data can’t be previewed from this source in Eventstream.

Data sources page

  1. Sign in to Microsoft Fabric.

  2. If you see Power BI at the bottom-left of the page, switch to the Fabric workload by selecting Power BI and then by selecting Fabric.

    Screenshot that shows how to switch to the Fabric workload.

  3. Select Real-Time on the left navigation bar.

    Screenshot that shows how to launch Connect to data source experience.

  4. The Streaming data page opens by default. Click on the Add data button to get to the Add data page.

    Screenshot that shows the Add data page in the Real-Time hub.

    You can also get to the Add data page directly by selecting the Add data option in the left navigation bar.

    Screenshot that shows the Connect data source button.

Add Apache Kafka as a source

On the Add data page, type in the search bar and select Apache Kafka.

A screenshot of selecting Apache Kafka.

Configure Apache Kafka connector

  1. On the Connect page, select New connection.

    Screenshot that shows the selection of the New connection link on the Connect page of the Get events wizard.

  2. In the Connection settings section, for Bootstrap Server, enter one or more Kafka bootstrap server addresses. Separate multiple addresses with commas (,).

    Screenshot that shows the selection of the Apache Kafka Bootstrap server field on the Connect page of the Get events wizard.

  3. In the Connection credentials section, If you have an existing connection to the Apache Kafka cluster, select it from the dropdown list for Connection. Otherwise, follow these steps:

    1. For Connection name, enter a name for the connection.
    2. For Authentication kind, confirm that API Key is selected.
    3. For Key and Secret, enter API key and key Secret.

      Note

      If you only use mTLS to do the authentication, you can add any string in the Key section during connection creation.

  4. Select Connect. 

  5. Now, on the Connect page, follow these steps.

    1. For Topic, enter the Kafka topic.

    2. For Consumer group, enter the consumer group of your Apache Kafka cluster. This field provides you with a dedicated consumer group for getting events.

    3. Select Reset auto offset to specify where to start reading offsets if there's no commit.

    4. For Security protocol, select one of the following options:

      • SASL_SSL: Use this option when your Kafka cluster uses SASL-based authentication. By default, the Kafka broker’s server certificate must be signed by a Certificate Authority (CA) included in the trusted CA list. If your Kafka cluster uses a custom CA, you can configure it by using TLS/mTLS settings.
      • SSL (mTLS): Use this option when your Kafka cluster requires mTLS authentication, and you must configure both a custom server CA certificate and a client certificate in TLS/mTLS settings.

    5. The default SASL mechanism is typically PLAIN, unless configured otherwise. You can select the SCRAM-SHA-256 or SCRAM-SHA-512 mechanism that suits your security requirements.

    6. If your Kafka cluster uses a custom CA or requires mTLS, expand TLS/mTLS settings and configure the following options as needed:

      • Trust CA certificate: Enable this option to configure the server CA certificate. Select your subscription, resource group, and key vault, and then provide the certificate name.
      • Client certificate and key: Enable this option to configure the client certificate and key.
        • Use the same CA certificate key vault: Select this checkbox when both certificates are stored in the same key vault. Then provide the certificate name.
        • If you don't select this checkbox, select the subscription, resource group, and key vault, and then provide the certificate name.

      Note

      TLS/mTLS settings in this section are currently in preview.

      For sources in a private network, ensure that the Azure Key Vault containing your certificates is connected to the Azure virtual network used by the streaming virtual network data gateway for Eventstream connector virtual network injection (for example, via a private endpoint).

    Screenshot that shows the first page of the Apache Kafka connection settings.

TLS/mTLS certificate requirements

If you configured TLS/mTLS settings, refer to this section for certificate format specifications and common configuration mistakes when uploading to Azure Key Vault.

Certificate chain

Certificate Key size Signed by Purpose
CA certificate 4096-bit RSA Self-signed Trust anchor - the broker verifies client certificates against this CA.
Server certificate 2048-bit RSA CA Broker identity - the client verifies the broker is who it claims to be.
Client certificate 2048-bit RSA CA Client identity - the broker verifies that the connector is authorized.

Server certificate SAN requirements

The server certificate must include the broker's IP address and DNS name in the Subject Alternative Name (SAN) to pass hostname verification (ssl.endpoint.identification.algorithm=https):

subjectAltName:
  DNS.1 = {broker FQDN}
  DNS.2 = localhost
  IP.1  = {broker public IP}
  IP.2  = 127.0.0.1

Upload certificates to Azure Key Vault

Certificates are uploaded as Azure Key Vault certificate objects in PEM format. The PEM bundle file is certificate + private key concatenated in one file:

-----BEGIN CERTIFICATE-----
MIIExjCCA...
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIB...
-----END RSA PRIVATE KEY-----

Use an import policy that matches the key properties:

{
  "secretProperties": {
    "contentType": "application/x-pem-file"
  },
  "keyProperties": {
    "exportable": true,
    "keyType": "RSA",
    "keySize": 4096,
    "reuseKey": false
  },
  "issuerParameters": {
    "name": "Unknown"
  }
}

To import the certificate, run the following command:

az keyvault certificate import \
  --vault-name {kvName} \
  --name {certName} \
  --file {pemBundleFile} \
  --policy @{policyFile}

Common mistakes

Avoid Do this instead
Upload as PKCS#12/PFX Use PEM format with contentType: application/x-pem-file.
Upload certificate without private key The PEM bundle must contain both the certificate and the key.
Set keySize: 2048 for a 4096-bit key The keySize value must match the actual key size.
Set issuerParameters.name: "Self" Use "Unknown" for externally signed certificates.
Use Windows line endings (CRLF) The PEM file must use Unix line endings (LF only).

View data stream details

  1. On the Review + connect page, if you select Open eventstream, the wizard opens the eventstream that it created for you with the selected Apache Kafka source. To close the wizard, select Close at the bottom of the page.
  2. You should see the stream in the Recent streaming data section of the Real-Time hub home page. For detailed steps, see View details of data streams in Fabric Real-Time hub.

Related content

To learn about consuming data streams, see the following articles:

  • Last updated on