Log Analytics workspace data export in Azure Monitor
Data export in a Log Analytics workspace lets you continuously export data per selected tables in your workspace. You can export to an Azure Storage account or Azure Event Hubs as the data arrives to an Azure Monitor pipeline. This article provides details on this feature and steps to configure data export in your workspaces.
Data in Log Analytics is available for the retention period defined in your workspace. It's used in various experiences provided in Azure Monitor and Azure services. There are cases where you need to use other tools:
- Tamper-protected store compliance: Data can't be altered in Log Analytics after it's ingested, but it can be purged. Export to a storage account set with immutability policies to keep data tamper protected.
- Integration with Azure services and other tools: Export to event hubs as data arrives and is processed in Azure Monitor.
- Long-term retention of audit and security data: Export to a storage account in the workspace's region. Or you can replicate data to other regions by using any of the Azure Storage redundancy options including GRS and GZRS.
After you've configured data export rules in a Log Analytics workspace, new data for tables in rules is exported from the Azure Monitor pipeline to your storage account or event hubs as it arrives.
Data is exported without a filter. For example, when you configure a data export rule for a SecurityEvent table, all data sent to the SecurityEvent table is exported starting from the configuration time. Alternatively, you can filter or modify exported data by configuring transformations in your workspace, which apply to incoming data, before it's sent to your Log Analytics workspaces and to export destinations.
Other export options
Log Analytics workspace data export continuously exports data that's sent to your Log Analytics workspace. There are other options to export data for particular scenarios:
- Configure diagnostic settings in Azure resources. Logs are sent to a destination directly. This approach has lower latency compared to data export in Log Analytics.
- Schedule export of data based on a log query you define with the Log Analytics query API. Use Azure Data Factory, Azure Functions, or Azure Logic Apps to orchestrate queries in your workspace and export data to a destination. This method is similar to the data export feature, but you can use it to export historical data from your workspace by using filters and aggregation. This method is subject to log query limits and isn't intended for scale. For more information, see Export data from a Log Analytics workspace to a storage account by using Logic Apps.
- One-time export to a local machine by using a PowerShell script. For more information, see Invoke-AzOperationalInsightsQueryExport.
- Custom logs created via HTTP Data Collector API, or 'dataSources' API won't be supported in export. This includes text logs consumed by MMA. Custom log created using data collection rule can be exported, including text based logs.
- We are support more tables in data export gradually, but currently limited to those specified in the supported tables section.
- You can define up to 10 enabled rules in your workspace, each can include multiple tables. You can create more rules in workspace in disabled state.
- Destinations must be in the same region as the Log Analytics workspace.
- The storage account must be unique across rules in the workspace.
- Table names can be 60 characters long when you're exporting to a storage account. They can be 47 characters when you're exporting to event hubs. Tables with longer names won't be exported.
- Currently, data export isn't supported in China.
Data export is optimized for moving large data volumes to your destinations. The export operation might fail for destinations capacity or availability, and a retry process continues for up to 12-hours. For more information, see Create or update a data export rule for destination limits and recommended alerts. If the destinations are still unavailable after the retry period, data is discarded. In certain retry conditions, retry can cause a fraction of duplicated records.
Data export charges are based on the volume of data exported measured in bytes. The size of data exported by Log Analytics Data Export is the number of bytes in the exported JSON-formatted data. Data volume is measured in GB (10^9 bytes).
For more information, including the data export billing timeline, see Azure Monitor pricing.
The data export destination must be available before you create export rules in your workspace. Destinations don't have to be in the same subscription as your workspace. When you use Azure Lighthouse, it's also possible to send data to destinations in another Azure Active Directory tenant.
You need to have write permissions to both workspace and destination to configure a data export rule on any table in a workspace. The shared access policy for the Event Hubs namespace defines the permissions that the streaming mechanism has. Streaming to event hubs requires manage, send, and listen permissions. To update the export rule, you must have the ListKey permission on that event hubs authorization rule.
Don't use an existing storage account that has other non-monitoring data to better control access to the data and prevent reaching storage ingress rate limit, failures, and latency.
To send data to an immutable storage account, set the immutable policy for the storage account as described in Set and manage immutability policies for Azure Blob Storage. You must follow all steps in this article, including enabling protected append blobs writes.
The storage account must be StorageV1 or later and in the same region as your workspace. If you need to replicate your data to other storage accounts in other regions, you can use any of the Azure Storage redundancy options, including GRS and GZRS.
Data is sent to storage accounts as it reaches Azure Monitor and exported to destinations located in a workspace region. A container is created for each table in the storage account with the name am- followed by the name of the table. For example, the table SecurityEvent would send to a container named am-SecurityEvent.
Blobs are stored in 5-minute folders in the following path structure: WorkspaceResourceId=/subscriptions/subscription-id/resourcegroups/<resource-group>/providers/microsoft.operationalinsights/workspaces/<workspace>/y=<four-digit numeric year>/m=<two-digit numeric month>/d=<two-digit numeric day>/h=<two-digit 24-hour clock hour>/m=<two-digit 60-minute clock minute>/PT05M.json. Appends to blobs are limited to 50-K writes. More blobs will be added in the folder as PT05M_#.json*, where # is the incremental blob count.
The format of blobs in a storage account is in JSON lines, where each record is delimited by a new line, with no outer records array and no commas between JSON records.
Don't use an existing event hub that has non-monitoring data to prevent reaching the Event Hubs namespace ingress rate limit, failures, and latency.
Data is sent to your event hub as it reaches Azure Monitor and is exported to destinations located in a workspace region. You can create multiple export rules to the same Event Hubs namespace by providing a different
event hub name in the rule. When an
event hub name isn't provided, a default event hub is created for tables that you export with the name am- followed by the name of the table. For example, the table SecurityEvent would be sent to an event hub named am-SecurityEvent.
The number of supported event hubs in Basic and Standard namespace tiers is 10. When you're exporting more than 10 tables to these tiers, either split the tables between several export rules to different Event Hubs namespaces or provide an event hub name to export all tables to it.
- The Basic Event Hubs namespace tier is limited. It supports lower event size and no Auto-inflate option to automatically scale up and increase the number of throughput units. Because data volume to your workspace increases over time and as a consequence event hub scaling is required, use Standard, Premium, or Dedicated Event Hubs tiers with the Auto-inflate feature enabled. For more information, see Automatically scale up Azure Event Hubs throughput units.
- Data export can't reach Event Hubs resources when virtual networks are enabled. You have to select the Allow Azure services on the trusted services list to access this storage account checkbox to bypass this firewall setting in an event hub to grant access to your event hubs.
Enable data export
The following steps must be performed to enable Log Analytics data export. For more information on each, see the following sections:
- Register the resource provider
- Allow trusted Microsoft services
- Create or update a data export rule
Register the resource provider
The Azure resource provider Microsoft.Insights needs to be registered in your subscription to enable Log Analytics data export.
This resource provider is probably already registered for most Azure Monitor users. To verify, go to Subscriptions in the Azure portal. Select your subscription and then select Resource providers under the Settings section of the menu. Locate Microsoft.Insights. If its status is Registered, then it's already registered. If not, select Register to register it.
You can also use any of the available methods to register a resource provider as described in Azure resource providers and types. The following sample command uses the Azure CLI:
az provider register --namespace 'Microsoft.insights'
The following sample command uses PowerShell:
Register-AzResourceProvider -ProviderNamespace Microsoft.insights
Allow trusted Microsoft services
If you've configured your storage account to allow access from selected networks, you need to add an exception to allow Azure Monitor to write to the account. From Firewalls and virtual networks for your storage account, select Allow Azure services on the trusted services list to access this storage account.
Monitor a storage account
Use a separate storage account for export.
Configure an alert on the metric:
Scope Metric namespace Metric Aggregation Threshold storage-name Account Ingress Sum 80% of maximum ingress per alert evaluation period. For example, the limit is 60 Gbps for general-purpose v2 in West US. The threshold is 14,400 Gb per 5-minute evaluation period.
Alert remediation actions:
- Use a separate storage account for export that isn't shared with non-monitoring data.
- Azure Storage Standard accounts support higher ingress limit by request. To request an increase, contact Azure Support.
- Split tables between more storage accounts.
Monitor event hubs
Configure alerts on the metrics:
Scope Metric namespace Metric Aggregation Threshold namespaces-name Event Hubs standard metrics Incoming bytes Sum 80% of maximum ingress per alert evaluation period. For example, the limit is 1 MB/s per unit (TU or PU) and five units used. The threshold is 1,200 MB per 5-minute evaluation period. namespaces-name Event Hubs standard metrics Incoming requests Count 80% of maximum events per alert evaluation period. For example, the limit is 1,000/s per unit (TU or PU) and five units used. The threshold is 1,200,000 per 5-minute evaluation period. namespaces-name Event Hubs standard metrics Quota exceeded errors Count Between 1% of request. For example, requests per 5 minutes is 600,000. The threshold is 6,000 per 5-minute evaluation period.
Alert remediation actions:
- Use a separate Event Hubs namespace for export that isn't shared with non-monitoring data.
- Configure the Auto-inflate feature to automatically scale up and increase the number of throughput units to meet usage needs.
- Verify the increase of throughput units to accommodate data volume.
- Split tables between more namespaces.
- Use Premium or Dedicated tiers for higher throughput.
Create or update a data export rule
A data export rule defines the destination and tables for which data is exported. You can create 10 rules in the Enabled state in your workspace. More rules are allowed in the Disabled state. The storage account must be unique across rules in the workspace. Multiple rules can use the same Event Hubs namespace when you're sending to separate event hubs.
- You can include tables that aren't yet supported in rules, but no data will be exported for them until the tables are supported.
- Export to a storage account: A separate container is created in the storage account for each table.
- Export to event hubs: If an event hub name isn't provided, a separate event hub is created for each table. The number of supported event hubs in Basic and Standard namespace tiers is 10. When you're exporting more than 10 tables to these tiers, either split the tables between several export rules to different Event Hubs namespaces or provide an event hub name in the rule to export all tables to it.
View data export rule configuration
Disable or update an export rule
You can disable export rules to stop the export for a certain period, such as when testing is being held. On the Log Analytics workspace menu in the Azure portal, select Data Export under the Settings section. Select the Status toggle to disable or enable the export rule.
Delete an export rule
On the Log Analytics workspace menu in the Azure portal, select Data Export under the Settings section. Select the ellipsis to the right of the rule and select Delete.
View all data export rules in a workspace
On the Log Analytics workspace menu in the Azure portal, select Data Export under the Settings section to view all export rules in the workspace.
If the data export rule includes an unsupported table, the configuration will succeed, but no data will be exported for that table. If the table is later supported, then its data will be exported at that time.
All data from the table will be exported unless limitations are specified. This list is updated as more tables are added.
|Alert||Partial support. Data ingestion for Zabbix alerts isn't supported.|
|ConfigurationData||Partial support. Some of the data is ingested through internal services that aren't supported in export. Currently, this portion is missing in export.|
|Event||Partial support. Data arriving from the Log Analytics agent (MMA) or Azure Monitor Agent (AMA) is fully supported in export. Data arriving via the Diagnostics extension agent is collected through storage. This path isn't supported in export.|
|InsightsMetrics||Partial support. Some of the data is ingested through internal services that aren't supported in export. Currently, this portion is missing in export.|
|Operation||Partial support. Some of the data is ingested through internal services that aren't supported in export. Currently, this portion is missing in export.|
|Perf||Partial support. Only Windows perf data is currently supported. Currently, the Linux perf data is missing in export.|
|SecurityEvent||Partial support. Data arriving from the Log Analytics agent or Azure Monitor Agent is fully supported in export. Data arriving via the Diagnostics extension agent is collected through storage. This path isn't supported in export.|
|Syslog||Partial support. Data arriving from the Log Analytics agent or Azure Monitor Agent is fully supported in export. Data arriving via the Diagnostics extension agent is collected through storage. This path isn't supported in export.|
|W3CIISLog||Partial support. Data arriving from Log Analytics agent (MMA) or Azure Monitor Agent (AMA) is fully supported in export. Data arriving via Diagnostics extension agent is collected through storage while this path isn’t supported in export.|