Add Confluent Kafka source to an eventstream

This article shows you how to add Confluent Kafka source to an eventstream.

Confluent Cloud Kafka is a streaming platform offering powerful data streaming and processing functionalities using Apache Kafka. By integrating Confluent Cloud Kafka as a source within your eventstream, you can seamlessly process real-time data streams before routing them to multiple destinations within Fabric.

Note

This source is not supported in the following regions of your workspace capacity: West US3, Switzerland West.

Prerequisites

• Access to a workspace in the Fabric capacity license mode (or) the Trial license mode with Contributor or higher permissions.
• A Confluent Cloud Kafka cluster and an API Key.
• Your Confluent Cloud Kafka cluster must be publicly accessible and not be behind a firewall or secured in a virtual network.
• If you don't have an eventstream, create an eventstream.

Launch the Select a data source wizard

If you haven't added any source to your eventstream yet, select Use external source tile.

If you're adding the source to an already published eventstream, switch to Edit mode, select Add source on the ribbon, and then select External sources.

 

Configure and connect to Confluent Kafka

1. On the Select a data source page, select Confluent.

   

2. To create a connection to the Confluent Cloud Kafka source, select New connection.

   

3. In the Connection settings section, enter Confluent Bootstrap Server. Navigate to your Confluent Cloud home page, select Cluster Settings, and copy the address to your Bootstrap Server.

4. In the Connection credentials section, If you have an existing connection to the Confluent cluster, select it from the drop-down list for Connection. Otherwise, follow these steps:

   1. For Connection name, enter a name for the connection.
   2. For Authentication kind, confirm that Confluent Cloud Key is selected.
   3. For API Key and API Key Secret:

      1. Navigate to your Confluent Cloud.

      2. Select API Keys on the side menu.

      3. Select the Add key button to create a new API key.

      4. Copy the API Key and Secret.

      5. Paste those values into the API Key and API Key Secret fields.

      6. Select Connect

         

5. Scroll to see the Configure Confluent data source section on the page. Enter the information to complete the configuration of the Confluent data source.

   1. For Topic, enter a topic name from your Confluent Cloud. You can create or manage your topic in the Confluent Cloud Console.
   2. For Consumer group, Enter a consumer group of your Confluent Cloud. It provides you with the dedicated consumer group for getting the events from Confluent Cloud cluster.
   3. For Reset auto offset setting, select one of the following values:

      • Earliest – the earliest data available from your Confluent cluster

      • Latest – the latest available data

      • None – don't automatically set the offset.

        

6. Select Next. On the Review and create screen, review the summary, and then select Add.

You see that the Confluent Cloud Kafka source is added to your eventstream on the canvas in Edit mode. To implement this newly added Confluent Cloud Kafka source, select Publish on the ribbon.

After you complete these steps, the Confluent Cloud Kafka source is available for visualization in Live view.

Limitations

• Confluent Kafka with JSON and Avro formats, using schema registry, is currently not supported.
• Decoding data from Confluent Kafka using the Confluent Schema Registry is currently not supported.

Note

The maximum number of sources and destinations for one eventstream is 11.

Related content

Other connectors:

• Amazon Kinesis Data Streams
• Azure Cosmos DB
• Azure Event Hubs
• Azure IoT Hub
• Azure SQL Database Change Data Capture (CDC)
• Custom endpoint
• Google Cloud Pub/Sub
• MySQL Database CDC
• PostgreSQL Database CDC
• Sample data
• Azure Blob Storage events
• Fabric workspace event