Govern resources for client applications with application groups
Azure Event Hubs enables you to govern event streaming workloads for client applications that connect to Event Hubs by using application groups. For more information, see Resource governance with application groups.
This article shows you how to perform the following tasks:
- Create an application group.
- Enable or disable an application group
- Define threshold limits and apply throttling policies to an application group
- Validate throttling with Diagnostic Logs
Application groups are available only in premium and dedicated tiers.
Create an application group
This section shows you how to create an application group using Azure portal, CLI, PowerShell, and an Azure Resource Manager (ARM) template.
You can create an application group using the Azure portal by following these steps.
Navigate to your Event Hubs namespace.
On the left menu, select Application Groups under Settings.
On the Application Groups page, select + Application Group on the command bar.
On the Add application group page, follow these steps:
Specify a name for the application group.
Confirm that Enabled is selected. To have the application group in the disabled state first, clear the Enabled option. This flag determines whether the clients of an application group can access Event Hubs or not.
For Security context type, select Namespace Shared access policy, event hub Shared Access Policy or AAD application.Application group supports the selection of SAS key at either namespace or at entity (event hub) level. When you create the application group, you should associate with either a shared access signatures (SAS) or Azure Active Directory(Azure AD) application ID, which is used by client applications.
If you selected Namespace Shared access policy:
For SAS key name, select the SAS policy that can be used as a security context for this application group.You can select Add SAS Policy to add a new policy and then associate with the application group.
If you selected Event Hubs Shared access policy:
For SAS key name, copy the SAS policy name from Event Hubs "Shared Access Policies" Page and paste into textbox
If you selected AAD application:
- For AAD Application (client) ID, specify the Azure Active Directory (Azure AD) application or client ID.
Review the auto-generated Client group ID, which is the unique ID associated with the application group. The scope of application governance (namespace or entity level) would depend on the access level for the used Azure AD application ID. The following table shows auto generated Client Group ID for different security Context type:
|Security Context type||Auto-generated client group ID|
|Namespace shared access key||
|Azure AD Application||
|Event Hubs shared access key||
All existing application groups created with namespace shared access key would continue to work with client group ID starting with
SASKeyName. However all new application groups would have updated client group ID as shown above.
To add a policy, follow these steps:
Enter a name for the policy.
For Type, select Throttling policy.
For Metric ID, select one of the following options: Incoming messages, Outgoing messages, Incoming bytes, Outgoing bytes. In the following example, Incoming messages is selected.
For Rate limit threshold, enter the threshold value. In the following example, 10000 is specified as the threshold for the number of incoming messages.
Here's a screenshot of the page with another policy added.
Now, on the Add application group page, select Add.
Confirm that you see the application group in the list of application groups.
You can delete the application group in the list by selecting the trash icon button next to it in the list.
Enable or disable an application group
You can prevent client applications accessing your Event Hubs namespace by disabling the application group that contains those applications. When the application group is disabled, client applications won't be able to publish or consume data. Any established connections from client applications of that application group will also be terminated.
This section shows you how to enable or disable an application group using Azure portal, PowerShell, CLI, and ARM template.
On the Event Hubs Namespace page, select Application Groups on the left menu.
Select the application group that you want to enable or disable.
On the Edit application group page, clear checkbox next to Enabled to disable an application group, and then select Update at the bottom of the page. Similarly, select the checkbox to enable an application group.
Apply throttling policies
You can add zero or more policies when you create an application group or to an existing application group. For example, you can add throttling policies related to
OutgoingBytes to the
contosoAppGroup. These policies will get applied to event streaming workloads of client applications that use the SAS policy
To learn how to add policies while creating an application group, see the Create an application group section.
You can also add policies after an application group is created.
On the Event Hubs Namespace page, select Application Groups on the left menu.
Select the application group that you want to add, update, or delete a policy.
On the Edit application group page, you can do the following steps:
- Update settings (including threshold values) for existing policies
- Add a new policy
Decide threshold value for throttling policies
Azure Event Hubs supports Application Metric Logs functionality to observe usual throughput within your system and accordingly decide on the threshold value for application group. You can follow these steps to decide on a threshold value:
Turn on diagnostic settings in Event Hubs with Application Metric logs as selected category and choose Log Analytics as destination.
Create an empty application group without any throttling policy.
Continue sending messages/events to event hub at usual throughput.
Go to Log Analytics workspace and query for the right activity name (based on the (resource-governance-overview.md#throttling-policy---threshold-limits)) in AzureDiagnostics table. The following sample query is set to track threshold value for incoming messages:
AzureDiagnostics | where ActivityName_s =="IncomingMessages" | where Outcome_s =="Success"
Select the Chart section on Log Analytics workspace and plot a chart between time generated on Y axis and count of messages sent on x axis.
In this example, you can see that the usual throughput never crossed more than 550 messages (expected current throughput). This observation helps you define the actual threshold value.
Once you decide the threshold value, add a new throttling policy inside the application group.
Publish or consume events
Once you successfully add throttling policies to the application group, you can test the throttling behavior by either publishing or consuming events using client applications that are part of the
contosoAppGroup application group. To test, you can use either an AMQP client or a Kafka client application and same SAS policy name or Azure AD application ID that's used to create the application group.
When your client applications are throttled, you should experience a slowness in publishing or consuming data.
Validate Throttling with Application Groups
Similar to Deciding Threshold limits for Throttling Policies, you can use Application Metric logs to validate throttling and find more details.
You can use the below example query to find out all the throttled requests in certain timeframe. You must update the ActivityName to match the operation that you expect to be throttled.
AzureDiagnostics | where Category =="ApplicationMetricsLogs" | where ActivityName_s =="IncomingMessages" | where Outcome_s =="Throttled"
Due to restrictions at protocol level, throttled request logs are not generated for consumer operations within event hub (
OutgoingBytes). when requests are throttled at consumer side, you would observe sluggish egress throughput.