Edit

Share via

Add Confluent Cloud for Apache Kafka source to an eventstream

This article shows you how to add Confluent Cloud for Apache Kafka source to an eventstream.

Confluent Cloud for Apache Kafka is a streaming platform offering powerful data streaming and processing functionalities using Apache Kafka. By integrating Confluent Cloud for Apache Kafka as a source within your eventstream, you can seamlessly process real-time data streams before routing them to multiple destinations within Fabric.

Note

This source is not supported in the following regions of your workspace capacity: West US3, Switzerland West.

Prerequisites

  • Access to a workspace in the Fabric capacity license mode (or) the Trial license mode with Contributor or higher permissions.
  • A Confluent Cloud for Apache Kafka cluster and an API Key.
  • Your Confluent Cloud for Apache Kafka cluster must be publicly accessible and not be behind a firewall or secured in a virtual network.
  • If you don't have an eventstream, create an eventstream.

Launch the Select a data source wizard

If you haven't added any source to your eventstream yet, select Use external source tile.

Screenshot that shows the selection of Use external source tile.

If you're adding the source to an already published eventstream, switch to Edit mode, select Add source on the ribbon, and then select External sources.

Screenshot that shows the selection of Add source to External sources menu.  

Configure and connect to Confluent Cloud for Apache Kafka

  1. On the Select a data source page, select Confluent Cloud for Apache Kafka.

    Screenshot that shows the selection of Confluent as the source type in the Get events wizard.

  2. To create a connection to the Confluent Cloud for Apache Kafka source, select New connection.

    Screenshot that shows the selection of the New connection link on the Connect page of the Get events wizard.

  3. In the Connection settings section, enter Confluent Bootstrap Server. Navigate to your Confluent Cloud home page, select Cluster Settings, and copy the address to your Bootstrap Server.

  4. In the Connection credentials section, If you have an existing connection to the Confluent cluster, select it from the dropdown list for Connection. Otherwise, follow these steps:

    1. For Connection name, enter a name for the connection.
    2. For Authentication kind, confirm that Confluent Cloud Key is selected.
    3. For API Key and API Key Secret:

      1. Navigate to your Confluent Cloud.

      2. Select API Keys on the side menu.

      3. Select the Add key button to create a new API key.

      4. Copy the API Key and Secret.

      5. Paste those values into the API Key and API Key Secret fields.

      6. Select Connect

        Screenshot that shows the first page of the Confluent connection settings.

  5. Scroll to see the Configure Confluent Cloud for Apache Kafka data source section on the page. Enter the information to complete the configuration of the Confluent data source.

    1. For Topic name, enter a topic name from your Confluent Cloud. You can create or manage your topic in the Confluent Cloud Console.
    2. For Consumer group, enter a consumer group of your Confluent Cloud. It provides you with the dedicated consumer group for getting the events from Confluent Cloud cluster.
    3. For Reset auto offset setting, select one of the following values:

      • Earliest – the earliest data available from your Confluent cluster

      • Latest – the latest available data

      • None – don't automatically set the offset.

        Screenshot that shows the second page - Configure Confluent data source page - of the Confluent connection settings.

  6. Depending on whether your data is encoded using Confluent Schema Registry:

Connect to Confluent schema registry to decode data (preview)

Eventstream's Confluent Cloud for Apache Kafka streaming connector is capable of decoding data produced with Confluent serializer and its Schema Registry from Confluent Cloud. Data encoded with this serializer of Confluent schema registry require schema retrieval from the Confluent Schema Registry for decoding. Without access to the schema, Eventstream can't preview, process, or route the incoming data.

You may expand Advanced settings to configure Confluent Schema Registry connection:

  1. Define and serialize data: Select Yes allows you to serialize the data into a standardized format. Select No keeps the data in its original format and passes it through without modification.

  2. If your data is encoded using a schema registry, select Yes when choosing whether the data is encoded with a schema registry. Then, select New connection to configure access to your Confluent Schema Registry:

    • Schema Registry URL: The public endpoint of your schema registry.
    • API Key and API Key Secret: Navigate to Confluent Cloud Environment's Schema Registry to copy the API Key and API Secret. Ensure the account used to create this API key has DeveloperRead or higher permission on the schema.
    • Privacy Level: Choose from None, Private, Organizational, or Public.

  3. JSON output decimal format: Specifies the JSON serialization format for Decimal logical type values in the data from the source.

    • NUMERIC: Serialize as numbers.
    • BASE64: Serialize as base64 encoded data.

  4. Select Next. On the Review and create screen, review the summary, and then select Add to complete the setup.

    Screenshot that shows advanced settings of the Confluent schema registry settings.

You see that the Confluent Cloud for Apache Kafka source is added to your eventstream on the canvas in Edit mode. To implement this newly added Confluent Cloud for Apache Kafka source, select Publish on the ribbon.

Screenshot that shows Confluent Cloud for Apache Kafka source in Edit view.

After you complete these steps, the Confluent Cloud for Apache Kafka source is available for visualization in Live view.

Screenshot that shows Confluent Cloud for Apache Kafka source in Live view.

Limitations

  • Confluent Cloud for Apache Kafka with JSON and Avro formats, using schema registry, is currently not supported.
  • Decoding data from Confluent Cloud for Apache Kafka using the Confluent Schema Registry is currently not supported.

Related content

Other connectors: