Edit

Share via

Add Confluent Kafka source to an eventstream (preview)

  • Article

This article shows you how to add Confluent Kafka source to an eventstream.

Confluent Cloud Kafka is a streaming platform offering powerful data streaming and processing functionalities using Apache Kafka. By integrating Confluent Cloud Kafka as a source within your eventstream, you can seamlessly process real-time data streams before routing them to multiple destinations within Fabric.

Important

Enhanced capabilities of Fabric event streams are currently in preview.

Note

This source is not supported in the following regions of your workspace capacity: West US3, Switzerland West.

Prerequisites

  • Access to the Fabric premium workspace with Contributor or above permissions.
  • A Confluent Cloud Kafka cluster and an API Key.
  • Your Confluent Cloud Kafka cluster must be publicly accessible and not be behind a firewall or secured in a virtual network.

Note

The maximum number of sources and destinations for one eventstream is 11.

Add Confluent Cloud Kafka as a source

  1. Create an eventstream with selecting the preview toggle.
  2. Select Add external source.

Configure and connect to Confluent Kafka

  1. On the Select a data source page, select Confluent.

    Screenshot that shows the selection of Confluent as the source type in the Get events wizard.

  2. To create a connection to the Confluent Cloud Kafka source, select New connection.

    Screenshot that shows the selection of the New connection link on the Connect page of the Get events wizard.

  3. In the Connection settings section, enter Confluent Bootstrap Server. Navigate to your Confluent Cloud home page, select Cluster Settings, and copy the address to your Bootstrap Server.

  4. In the Connection credentials section, If you have an existing connection to the Confluent cluster, select it from the drop-down list for Connection. Otherwise, follow these steps:

    1. For Connection name, enter a name for the connection.
    2. For Authentication kind, confirm that Confluent Cloud Key is selected.
    3. For API Key and API Key Secret:

      1. Navigate to your Confluent Cloud.

      2. Select API Keys on the side menu.

      3. Select the Add key button to create a new API key.

      4. Copy the API Key and Secret.

      5. Paste those values into the API Key and API Key Secret fields.

      6. Select Connect

        Screenshot that shows the first page of the Confluent connection settings.

  5. Scroll to see the Configure Confluent data source section on the page. Enter the information to complete the configuration of the Confluent data source.

    1. For Topic, enter a topic name from your Confluent Cloud. You can create or manage your topic in the Confluent Cloud Console.
    2. For Consumer group, Enter a consumer group of your Confluent Cloud. It provides you with the dedicated consumer group for getting the events from Confluent Cloud cluster.
    3. For Reset auto offset setting, select one of the following values:

      • Earliest – the earliest data available from your Confluent cluster

      • Latest – the latest available data

      • None – don't automatically set the offset.

        Screenshot that shows the second page - Configure Confluent data source page - of the Confluent connection settings.

  6. Select Next. On the Review and create screen, review the summary, and then select Add.

You see that the Confluent Cloud Kafka source is added to your eventstream on the canvas in Edit mode. To implement this newly added Confluent Cloud Kafka source, select Publish on the ribbon.

Screenshot that shows Confluent Kafka source in Edit view.

After you complete these steps, the Confluent Cloud Kafka source is available for visualization in Live view.

Screenshot that shows Confluent Kafka source in Live view.

Limitations

  • Confluent Kafka with JSON and Avro formats, using schema registry, is not currently supported.

Related content

Other connectors: