Controlling Telemetry Cost

Стаття
12/05/2023

Note

Azure Active Directory is now Microsoft Entra ID. Learn more

Azure Application Insights is billed based on the volume of telemetry data your application sends (data ingestion) and how long time you want data to be available (data retention).

Check the Azure Application Insights documentation for up-to-date information on pricing: https://azure.microsoft.com/pricing/details/monitor/.

Control data ingestion cost

To control data ingestion cost, you can:

sample to only ingest a percentage of the inbound data (learn more at Sampling in Application Insights.
set a daily limit of how much data can be ingested
set alerts on cost thresholds being exceeded to get notified when it happens
use data collection rules (DCR) to filter the type of data you want
use a custom endpoint (go to section)

The first three options for controlling data ingestion cost in Azure Application Insights are found under configure, usage, and estimated costs. Data collection rules are defined on your workspace under the menu item Tables.

Data ingestion cost strategies

To control data ingestion cost, consider the following practice when setting up telemetry:

Set a daily cap on how much data that can be ingested (and what you are willing to pay for)
Set up an alert when daily cap is reached (so that you know that you do not get all data)
Use sampling or Data Collection Rules to adjust data ingestion

For more information, see Cost optimization in Azure Monitor and Set daily cap on Log Analytics workspace.

Video guidance

The following video shows how to enable cost control strategies for telemetry.

Calculating the cost of data ingestion

Currently, the first 5GB of data ingested per month is free. To stay below this limit, you can set up a daily cap of 160 MB per day.

One telemetry event typically consumes 2-10 KB depending of the type of event. The max size for one event is 32 KB.

So with a 160 MB daily cap, you can receive between 5000 (worst case) and 80000 daily events (best case). With event size 10kb, this corresponds to 16000 daily events.

Current price for data ingestion is 2.76 USD/GB (for pay-as-you-go). If we assume 10KB event size, 1 GB can give you an additional 100.000 monthly events ~ 3300 daily events. Or 0.00276 cents/event.

Typically, non-interactive sessions such as web service calls or background sessions are the ones that can generate a lot of data. These are the ones you could consider filtering away to reduce cost.

Examples of cost of data ingestion

One partner that uses telemetry a lot reported that they on average spend 7.3 USD per customer per month on data ingestion. For their large customers they spend up to 178 USD per month (without setting up data collection rules, sampling, or daily cap).

Their top 10 customer cost per month on telemetry is:

Rank	Monthly Cost (in USD)
1	178.68
2	173.68
3	146.03
4	106.32
5	66.47
6	31.91
7	27.06
8	23.53
9	22.79
10	17.65

Note

These numbers can vary and should only be taken as guidance. You need to measure your own telemetry cost based on data ingestion volume.

Distribution of telemetry data

The easiest way to see the data distribution of different event IDs in your Azure Application Insights resource is to use the Data in Telemetry page on the Administration report in the Power BI app on telemetry data.

You can also run the KQL queries below.

// 30 day overview of event ingestion in your Application Insights resource
let lookback = 30d ;
let traces_stats = 
traces
| where timestamp > ago(lookback) 
| extend eventId = tostring( customDimensions.eventId )
| summarize count() by eventId
| extend table_name = 'traces'
| project eventId, table_name, count_
;
let pageview_stats =
pageViews
| where timestamp > ago(lookback) 
| extend eventId = tostring( customDimensions.eventID )
| summarize count() by eventId
| extend table_name = 'pageViews'
| project eventId, table_name, count_
;
traces_stats
| union pageview_stats
| order by count_ desc
| project eventId, table_name, event_count=count_
| order by event_count desc

Similarly, you can see the data distribution of different environments in your Azure Application Insights resource, by running the KQL queries below.

// 30 day overview of data ingestion by environment in your Application Insights resource
let lookback = 30d ;
union traces, pageViews
| where timestamp > ago(lookback) 
| extend aadTenantId = tostring( customDimensions.aadTenantId )
, environmentName = tostring( customDimensions.environmentName )
| summarize count() by aadTenantId, environmentName
| project aadTenantId, environmentName, count_
| order by aadTenantId, environmentName

Use data collection rules (DCR)

Azure Log Analytics (the backend of Azure Application Insights) has a feature called Data Collection Rules (DCR). DCRs allow you to define rules on what is ingested into your telemetry resource. Use them to filter away unneeded telemetry data, which speeds up query performance and saves cost.

To use DCRs, your Azure Application Insights resource must be converted to Workspace-based (which enables Log Analytics as your back-end). For recently created Azure Application Insights resources, Workspace-based is the default setting. For older Azure Application Insights resources, you need to convert them (takes a few minutes).

DCRs are defined on your workspace under the menu item Tables. Business Central currently logs to two tables: AppTraces and AppPageViews. These tables are the back-end tables for the tables traces and pageviews that you normally use in KQL queries. Right-click on a table and go to Edit transformation and select the Transformation Editor button to add a KQL query to filter away certain types of data.

To help you get started with setting up Data Collection Rules, we've added many sample KQL queries that illustrate common filter scenarios:

How to filter out one or more events (I don't need these events, no need to pay for them)
How to only filter part of an event (I only want to see events when something fails or I only want 10 percent of these events)
How to filter on Microsoft Entra tenant (I'm an ISV and I want to start slowly on telemetry, so only enabling it for a few customers)
How to filter on environment type (I'm an ISV and I only want data from production environments)
How to filter on app dimensions (I'm a VAR and this app/publisher is too noisy)

For sample queries, go to Data Collection Rule KQL samples.

Use a custom endpoint

Azure Application Insights support overriding the standard data ingestion endpoint provided in the connection string (available in the Azure Application Insights portal). This capability means you can send telemetry data to your own component to do post-processing. For example, you could filter or enrich before ingesting data into your data source of choice. The data source can be an Azure SQL database, a data lake, Azure Log Analytics, Azure Application Insights, or a third-party data store.

You can override the ingestion endpoint by using the IngestionEndpoint key in the Azure Application Insights connection string. Learn more in the documentation: Overriding the Azure Application Insights standard connection string.

Reducing data retention cost

To reduce data retention cost, you can

use the default retention for Azure Application Insights resources (currently 90 days). Different retention periods can be selected for each Application Insights resource. The full set of available retention periods is 30, 60, 90, 120, 180, 270, 365, 550, or 730 days.
purge data from your Application Insights resource using a set of user-defined filters. See Components - Purge for examples.

Поділитися через