Send events to and receive events from Azure Event Hubs - .NET (Azure.Messaging.EventHubs)

This quickstart shows how to send events to and receive events from an event hub using the Azure.Messaging.EventHubs .NET library.

Note

You can find all .NET samples for Event Hubs in our .NET SDK repository on GitHub.

Prerequisites

If you're new to Azure Event Hubs, see Event Hubs overview before you do this quickstart.

To complete this quickstart, you need the following prerequisites:

  • Microsoft Azure subscription. To use Azure services, including Azure Event Hubs, you need a subscription. If you don't have an existing Azure account, you can sign up for a free trial or use your MSDN subscriber benefits when you create an account.
  • Microsoft Visual Studio 2022. The Azure Event Hubs client library makes use of new features that were introduced in C# 8.0. You can still use the library with previous C# language versions, but the new syntax won't be available. To make use of the full syntax, we recommend that you compile with the .NET Core SDK 3.0 or higher and language version set to latest. If you're using Visual Studio, versions before Visual Studio 2022 aren't compatible with the tools needed to build C# 8.0 projects. Visual Studio 2022, including the free Community edition, can be downloaded here.
  • Create an Event Hubs namespace and an event hub. The first step is to use the Azure portal to create a namespace of type Event Hubs, and obtain the management credentials your application needs to communicate with the event hub. To create a namespace and an event hub, follow the procedure in this article.

Send events

This section shows you how to create a .NET Core console application to send events to an event hub.

Create a console application

  1. Start Visual Studio 2022.

  2. Select Create a new project.

  3. On the Create a new project dialog box, do the following steps: If you don't see this dialog box, select File on the menu, select New, and then select Project.

    1. Select C# for the programming language.

    2. Select Console for the type of the application.

    3. Select Console Application from the results list.

    4. Then, select Next.

      Image showing the New Project dialog box

  4. Enter EventHubsSender for the project name, EventHubsQuickStart for the solution name, and then select OK to create the project.

    Image showing the page where you enter solution and project names

Add the NuGet packages to the project

  1. Select Tools > NuGet Package Manager > Package Manager Console from the menu.

  2. Run the following command to install the Azure.Messaging.EventHubs and the Azure.Identity NuGet packages.

    Install-Package Azure.Messaging.EventHubs
    Install-Package Azure.Identity
    

Authenticate the app to Azure

This quick start shows you two ways of connecting to Azure Event Hubs: passwordless and connection string. The first option shows you how to use your security principal in Azure Active Directory and role-based access control (RBAC) to connect to an Event Hubs namespace. You don't need to worry about having hard-coded connection strings in your code or in a configuration file or in a secure storage like Azure Key Vault. The second option shows you how to use a connection string to connect to an Event Hubs namespace. If you're new to Azure, you may find the connection string option easier to follow. We recommend using the passwordless option in real-world applications and production environments. For more information, see Authentication and authorization. You can also read more about passwordless authentication on the overview page.

Assign roles to your Azure AD user

When developing locally, make sure that the user account that connects to Azure Event Hubs has the correct permissions. You'll need the Azure Event Hubs Data Owner role in order to send and receive messages. To assign yourself this role, you'll need the User Access Administrator role, or another role that includes the Microsoft.Authorization/roleAssignments/write action. You can assign Azure RBAC roles to a user using the Azure portal, Azure CLI, or Azure PowerShell. Learn more about the available scopes for role assignments on the scope overview page.

The following example assigns the Azure Event Hubs Data Owner role to your user account, which provides full access to Azure Event Hubs resources. In a real scenario, follow the Principle of Least Privilege to give users only the minimum permissions needed for a more secure production environment.

Azure built-in roles for Azure Event Hubs

For Azure Event Hubs, the management of namespaces and all related resources through the Azure portal and the Azure resource management API is already protected using the Azure RBAC model. Azure provides the below Azure built-in roles for authorizing access to an Event Hubs namespace:

If you want to create a custom role, see Rights required for Event Hubs operations.

Important

In most cases, it will take a minute or two for the role assignment to propagate in Azure. In rare cases, it may take up to eight minutes. If you receive authentication errors when you first run your code, wait a few moments and try again.

  1. In the Azure portal, locate your Event Hubs namespace using the main search bar or left navigation.

  2. On the overview page, select Access control (IAM) from the left-hand menu.

  3. On the Access control (IAM) page, select the Role assignments tab.

  4. Select + Add from the top menu and then Add role assignment from the resulting drop-down menu.

    A screenshot showing how to assign a role.

  5. Use the search box to filter the results to the desired role. For this example, search for Azure Event Hubs Data Owner and select the matching result. Then choose Next.

  6. Under Assign access to, select User, group, or service principal, and then choose + Select members.

  7. In the dialog, search for your Azure AD username (usually your user@domain email address) and then choose Select at the bottom of the dialog.

  8. Select Review + assign to go to the final page, and then Review + assign again to complete the process.

Launch Visual Studio and sign-in to Azure

You can authorize access to the service bus namespace using the following steps:

  1. Launch Visual Studio. If you see the Get started window, select the Continue without code link in the right pane.

  2. Select the Sign in button in the top right of Visual Studio.

    Screenshot showing a button to sign in to Azure using Visual Studio.

  3. Sign-in using the Azure AD account you assigned a role to previously.

    Screenshot showing the account selection.

Write code to send events to the event hub

Add the following code to the top of the Program.cs file.

Here are the important steps from the code:

  1. Creates an EventHubProducerClient object using the namespace and the event hub name.
  2. Invokes the CreateBatchAsync method on the EventHubProducerClient object to create an EventDataBatch object.
  3. Add events to the batch using the EventDataBatch.TryAdd method.
  4. Sends the batch of messages to the event hub using the EventHubProducerClient.SendAsync method.

In the code sample below, replace the <EVENT_HUB_NAMESPACE> and <HUB_NAME> placeholder values for the EventHubProducerClient parameters.

using Azure.Identity;
using Azure.Messaging.EventHubs;
using Azure.Messaging.EventHubs.Producer;
using System.Text;

// number of events to be sent to the event hub
int numOfEvents = 3;

// The Event Hubs client types are safe to cache and use as a singleton for the lifetime
// of the application, which is best practice when events are being published or read regularly.
// TODO: Replace the <EVENT_HUB_NAMESPACE> and <HUB_NAME> placeholder values
EventHubProducerClient producerClient = new EventHubProducerClient(
    "<EVENT_HUB_NAMESPACE>.servicebus.windows.net",
    "<HUB_NAME>",
    new DefaultAzureCredential());

// Create a batch of events 
using EventDataBatch eventBatch = await producerClient.CreateBatchAsync();

for (int i = 1; i <= numOfEvents; i++)
{
    if (!eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes($"Event {i}"))))
    {
        // if it is too large for the batch
        throw new Exception($"Event {i} is too large for the batch and cannot be sent.");
    }
}

try
{
    // Use the producer client to send the batch of events to the event hub
    await producerClient.SendAsync(eventBatch);
    Console.WriteLine($"A batch of {numOfEvents} events has been published.");
}
finally
{
    await producerClient.DisposeAsync();
}
  1. Build the project, and ensure that there are no errors.

  2. Run the program and wait for the confirmation message.

    A batch of 3 events has been published.
    
  3. In the Azure portal, you can verify that the event hub has received the events. Switch to Messages view in the Metrics section. Refresh the page to update the chart. It may take a few seconds for it to show that the messages have been received.

    Image of the Azure portal page to verify that the event hub received the events

    Note

    For the complete source code with more informational comments, see this file on the GitHub

Receive events

This section shows how to write a .NET Core console application that receives events from an event hub using an event processor. The event processor simplifies receiving events from event hubs by managing persistent checkpoints and parallel receptions from those event hubs. An event processor is associated with a specific event Hub and a consumer group. It receives events from multiple partitions in the event hub, passing them to a handler delegate for processing using code that you provide.

Warning

If you run this code on Azure Stack Hub, you will experience runtime errors unless you target a specific Storage API version. That's because the Event Hubs SDK uses the latest available Azure Storage API available in Azure that may not be available on your Azure Stack Hub platform. Azure Stack Hub may support a different version of Storage Blob SDK than those typically available on Azure. If you are using Azure Blob Storage as a checkpoint store, check the supported Azure Storage API version for your Azure Stack Hub build and target that version in your code.

For example, If you are running on Azure Stack Hub version 2005, the highest available version for the Storage service is version 2019-02-02. By default, the Event Hubs SDK client library uses the highest available version on Azure (2019-07-07 at the time of the release of the SDK). In this case, besides following steps in this section, you will also need to add code to target the Storage service API version 2019-02-02. For an example on how to target a specific Storage API version, see this sample on GitHub.

Create an Azure Storage Account and a blob container

In this quickstart, you use Azure Storage as the checkpoint store. Follow these steps to create an Azure Storage account.

  1. Create an Azure Storage account
  2. Create a blob container
  3. Authenticate to the blob container

When developing locally, make sure that the user account that is accessing blob data has the correct permissions. You'll need Storage Blob Data Contributor to read and write blob data. To assign yourself this role, you'll need to be assigned the User Access Administrator role, or another role that includes the Microsoft.Authorization/roleAssignments/write action. You can assign Azure RBAC roles to a user using the Azure portal, Azure CLI, or Azure PowerShell. You can learn more about the available scopes for role assignments on the scope overview page.

In this scenario, you'll assign permissions to your user account, scoped to the storage account, to follow the Principle of Least Privilege. This practice gives users only the minimum permissions needed and creates more secure production environments.

The following example will assign the Storage Blob Data Contributor role to your user account, which provides both read and write access to blob data in your storage account.

Important

In most cases it will take a minute or two for the role assignment to propagate in Azure, but in rare cases it may take up to eight minutes. If you receive authentication errors when you first run your code, wait a few moments and try again.

  1. In the Azure portal, locate your storage account using the main search bar or left navigation.

  2. On the storage account overview page, select Access control (IAM) from the left-hand menu.

  3. On the Access control (IAM) page, select the Role assignments tab.

  4. Select + Add from the top menu and then Add role assignment from the resulting drop-down menu.

    A screenshot showing how to assign a storage account role.

  5. Use the search box to filter the results to the desired role. For this example, search for Storage Blob Data Contributor and select the matching result and then choose Next.

  6. Under Assign access to, select User, group, or service principal, and then choose + Select members.

  7. In the dialog, search for your Azure AD username (usually your user@domain email address) and then choose Select at the bottom of the dialog.

  8. Select Review + assign to go to the final page, and then Review + assign again to complete the process.

Create a project for the receiver

  1. In the Solution Explorer window, right-click the EventHubQuickStart solution, point to Add, and select New Project.
  2. Select Console application, and select Next.
  3. Enter EventHubsReceiver for the Project name, and select Create.
  4. In the Solution Explorer window, right-click EventHubsReceiver, and select Set as a Startup Project.

Add the NuGet packages to the project

  1. Select Tools > NuGet Package Manager > Package Manager Console from the menu.

  2. In the Package Manager Console window, confirm that EventHubsReceiver is selected for the Default project. If not, use the drop-down list to select EventHubsReceiver.

  3. Run the following command to install the Azure.Messaging.EventHubs and the Azure.Identity NuGet packages.

    Install-Package Azure.Messaging.EventHubs
    Install-Package Azure.Messaging.EventHubs.Processor
    Install-Package Azure.Identity
    

Update the code

Replace the contents of Program.cs with the following code:

Here are the important steps from the code:

  1. Creates an EventProcessorClient object using the event hub namespace and the event hub name. You need to build BlobContainerClient object for the container in the Azure storage you created earlier.
  2. Specifies handlers for the ProcessEventAsync and ProcessErrorAsync events of the EventProcessorClient object.
  3. Starts processing events by invoking the StartProcessingAsync on the EventProcessorClient object.
  4. When user presses a key to end the processing, invokes the StopProcessingAsync on the EventProcessorClient object.

In the code sample below, replace the <STORAGE_ACCOUNT_NAME> and <BLOB_CONTAINER_NAME> placeholder values for the BlobContainerClient URI. Replace the <EVENT_HUB_NAMESPACE> and <HUB_NAME> placeholder values for the EventProcessorClient as well.

using Azure.Identity;
using Azure.Messaging.EventHubs;
using Azure.Messaging.EventHubs.Consumer;
using Azure.Messaging.EventHubs.Processor;
using Azure.Storage.Blobs;
using System.Text;

// Create a blob container client that the event processor will use
// TODO: Replace the <STORAGE_ACCOUNT_NAME> placeholder value
BlobContainerClient storageClient = new BlobContainerClient(
    new Uri("https://<STORAGE_ACCOUNT_NAME>.blob.core.windows.net/<BLOB_CONTAINER_NAME>"),,
    new DefaultAzureCredential());

// Create an event processor client to process events in the event hub
// TODO: Replace the <EVENT_HUBS_NAMESPACE> and <HUB_NAME> placeholder values
var processor = new EventProcessorClient(
    storageClient,
    EventHubConsumerClient.DefaultConsumerGroupName,
    "<EVENT_HUB_NAMESPACE>.servicebus.windows.net",
    "<HUB_NAME>",
    new DefaultAzureCredential());

// Register handlers for processing events and handling errors
processor.ProcessEventAsync += ProcessEventHandler;
processor.ProcessErrorAsync += ProcessErrorHandler;

// Start the processing
await processor.StartProcessingAsync();

// Wait for 30 seconds for the events to be processed
await Task.Delay(TimeSpan.FromSeconds(30));

// Stop the processing
await processor.StopProcessingAsync();

async Task ProcessEventHandler(ProcessEventArgs eventArgs)
{
    // Write the body of the event to the console window
    Console.WriteLine("\tReceived event: {0}", Encoding.UTF8.GetString(eventArgs.Data.Body.ToArray()));

    // Update checkpoint in the blob storage so that the app receives only new events the next time it's run
    await eventArgs.UpdateCheckpointAsync(eventArgs.CancellationToken);
}

Task ProcessErrorHandler(ProcessErrorEventArgs eventArgs)
{
    // Write details about the error to the console window
    Console.WriteLine($"\tPartition '{eventArgs.PartitionId}': an unhandled exception was encountered. This was not expected to happen.");
    Console.WriteLine(eventArgs.Exception.Message);
    return Task.CompletedTask;
}
  1. Build the project, and ensure that there are no errors.

    Note

    For the complete source code with more informational comments, see this file on the GitHub.

  2. Run the receiver application.

  3. You should see a message that the events have been received.

    Received event: Event 1
    Received event: Event 2
    Received event: Event 3    
    

    These events are the three events you sent to the event hub earlier by running the sender program.

  4. In the Azure portal, you can verify that there are 3 outgoing messages. These are the messages Event Hubs sent to the receiving application. Refresh the page to update the chart. It may take a few seconds for it to show that the messages have been received.

    Image of the Azure portal page to verify that the event hub sent events to the receiving app

Next steps

This quick start provides step-by-step instructions to implement a simple scenario of sending a batch of events to an event hub and then receiving them. For more samples on other and advanced scenarios, check out the following samples on GitHub.