Log Analytics workspace data export in Azure Monitor

Data export in a Log Analytics workspace lets you continuously export data per selected tables in your workspace. You can export to an Azure Storage account or Azure Event Hubs as the data arrives to an Azure Monitor pipeline. This article provides details on this feature and steps to configure data export in your workspaces.

Overview

Data in Log Analytics is available for the retention period defined in your workspace. It's used in various experiences provided in Azure Monitor and Azure services. There are cases where you need to use other tools:

  • Tamper-protected store compliance: Data can't be altered in Log Analytics after it's ingested, but it can be purged. Export to a storage account set with immutability policies to keep data tamper protected.
  • Integration with Azure services and other tools: Export to event hubs as data arrives and is processed in Azure Monitor.
  • Long-term retention of audit and security data: Export to a storage account in the workspace's region. Or you can replicate data to other regions by using any of the Azure Storage redundancy options including GRS and GZRS.

After you've configured data export rules in a Log Analytics workspace, new data for tables in rules is exported from the Azure Monitor pipeline to your storage account or event hubs as it arrives.

Diagram that shows a data export flow.

Data is exported without a filter. For example, when you configure a data export rule for a SecurityEvent table, all data sent to the SecurityEvent table is exported starting from the configuration time. Alternatively, you can filter or modify exported data by configuring transformations in your workspace, which apply to incoming data, before it's sent to your Log Analytics workspaces and to export destinations.

Other export options

Log Analytics workspace data export continuously exports data that's sent to your Log Analytics workspace. There are other options to export data for particular scenarios:

  • Configure diagnostic settings in Azure resources. Logs are sent to a destination directly. This approach has lower latency compared to data export in Log Analytics.
  • Schedule export of data based on a log query you define with the Log Analytics query API. Use Azure Data Factory, Azure Functions, or Azure Logic Apps to orchestrate queries in your workspace and export data to a destination. This method is similar to the data export feature, but you can use it to export historical data from your workspace by using filters and aggregation. This method is subject to log query limits and isn't intended for scale. For more information, see Export data from a Log Analytics workspace to a storage account by using Logic Apps.
  • One-time export to a local machine by using a PowerShell script. For more information, see Invoke-AzOperationalInsightsQueryExport.

Limitations

  • Custom logs created via HTTP Data Collector API, or 'dataSources' API won't be supported in export. This includes text logs consumed by MMA. Custom log created using data collection rule can be exported, including text based logs.
  • We are support more tables in data export gradually, but currently limited to those specified in the supported tables section.
  • You can define up to 10 enabled rules in your workspace, each can include multiple tables. You can create more rules in workspace in disabled state.
  • Destinations must be in the same region as the Log Analytics workspace.
  • The storage account must be unique across rules in the workspace.
  • Table names can be 60 characters long when you're exporting to a storage account. They can be 47 characters when you're exporting to event hubs. Tables with longer names won't be exported.
  • Currently, data export isn't supported in China.

Data completeness

Data export is optimized for moving large data volumes to your destinations. The export operation might fail for destinations capacity or availability, and a retry process continues for up to 12-hours. For more information, see Create or update a data export rule for destination limits and recommended alerts. If the destinations are still unavailable after the retry period, data is discarded. In certain retry conditions, retry can cause a fraction of duplicated records.

Pricing model

Data export charges are based on the volume of data exported measured in bytes. The size of data exported by Log Analytics Data Export is the number of bytes in the exported JSON-formatted data. Data volume is measured in GB (10^9 bytes).

For more information, including the data export billing timeline, see Azure Monitor pricing.

Export destinations

The data export destination must be available before you create export rules in your workspace. Destinations don't have to be in the same subscription as your workspace. When you use Azure Lighthouse, it's also possible to send data to destinations in another Azure Active Directory tenant.

You need to have write permissions to both workspace and destination to configure a data export rule on any table in a workspace. The shared access policy for the Event Hubs namespace defines the permissions that the streaming mechanism has. Streaming to event hubs requires manage, send, and listen permissions. To update the export rule, you must have the ListKey permission on that event hubs authorization rule.

Storage account

Don't use an existing storage account that has other non-monitoring data to better control access to the data and prevent reaching storage ingress rate limit, failures, and latency.

To send data to an immutable storage account, set the immutable policy for the storage account as described in Set and manage immutability policies for Azure Blob Storage. You must follow all steps in this article, including enabling protected append blobs writes.

The storage account must be StorageV1 or later and in the same region as your workspace. If you need to replicate your data to other storage accounts in other regions, you can use any of the Azure Storage redundancy options, including GRS and GZRS.

Data is sent to storage accounts as it reaches Azure Monitor and exported to destinations located in a workspace region. A container is created for each table in the storage account with the name am- followed by the name of the table. For example, the table SecurityEvent would send to a container named am-SecurityEvent.

Blobs are stored in 5-minute folders in the following path structure: WorkspaceResourceId=/subscriptions/subscription-id/resourcegroups/<resource-group>/providers/microsoft.operationalinsights/workspaces/<workspace>/y=<four-digit numeric year>/m=<two-digit numeric month>/d=<two-digit numeric day>/h=<two-digit 24-hour clock hour>/m=<two-digit 60-minute clock minute>/PT05M.json. Appends to blobs are limited to 50-K writes. More blobs will be added in the folder as PT05M_#.json*, where # is the incremental blob count.

The format of blobs in a storage account is in JSON lines, where each record is delimited by a new line, with no outer records array and no commas between JSON records.

Screenshot that shows data format in a blob.

Event hubs

Don't use an existing event hub that has non-monitoring data to prevent reaching the Event Hubs namespace ingress rate limit, failures, and latency.

Data is sent to your event hub as it reaches Azure Monitor and is exported to destinations located in a workspace region. You can create multiple export rules to the same Event Hubs namespace by providing a different event hub name in the rule. When an event hub name isn't provided, a default event hub is created for tables that you export with the name am- followed by the name of the table. For example, the table SecurityEvent would be sent to an event hub named am-SecurityEvent.

The number of supported event hubs in Basic and Standard namespace tiers is 10. When you're exporting more than 10 tables to these tiers, either split the tables between several export rules to different Event Hubs namespaces or provide an event hub name to export all tables to it.

Note

  • The Basic Event Hubs namespace tier is limited. It supports lower event size and no Auto-inflate option to automatically scale up and increase the number of throughput units. Because data volume to your workspace increases over time and as a consequence event hub scaling is required, use Standard, Premium, or Dedicated Event Hubs tiers with the Auto-inflate feature enabled. For more information, see Automatically scale up Azure Event Hubs throughput units.
  • Data export can't reach Event Hubs resources when virtual networks are enabled. You have to select the Allow Azure services on the trusted services list to access this storage account checkbox to bypass this firewall setting in an event hub to grant access to your event hubs.

Enable data export

The following steps must be performed to enable Log Analytics data export. For more information on each, see the following sections:

  • Register the resource provider
  • Allow trusted Microsoft services
  • Create or update a data export rule

Register the resource provider

The Azure resource provider Microsoft.Insights needs to be registered in your subscription to enable Log Analytics data export.

This resource provider is probably already registered for most Azure Monitor users. To verify, go to Subscriptions in the Azure portal. Select your subscription and then select Resource providers under the Settings section of the menu. Locate Microsoft.Insights. If its status is Registered, then it's already registered. If not, select Register to register it.

You can also use any of the available methods to register a resource provider as described in Azure resource providers and types. The following sample command uses the Azure CLI:

az provider register --namespace 'Microsoft.insights'

The following sample command uses PowerShell:

Register-AzResourceProvider -ProviderNamespace Microsoft.insights

Allow trusted Microsoft services

If you've configured your storage account to allow access from selected networks, you need to add an exception to allow Azure Monitor to write to the account. From Firewalls and virtual networks for your storage account, select Allow Azure services on the trusted services list to access this storage account.

Screenshot that shows the option Allow Azure services on the trusted services list.

Monitor destinations

Important

Export destinations have limits and should be monitored to minimize throttling, failures, and latency. For more information, see Storage account scalability and Event Hubs namespace quotas.

Monitor a storage account

  1. Use a separate storage account for export.

  2. Configure an alert on the metric:

    Scope Metric namespace Metric Aggregation Threshold
    storage-name Account Ingress Sum 80% of maximum ingress per alert evaluation period. For example, the limit is 60 Gbps for general-purpose v2 in West US. The threshold is 14,400 Gb per 5-minute evaluation period.
  3. Alert remediation actions:

    • Use a separate storage account for export that isn't shared with non-monitoring data.
    • Azure Storage Standard accounts support higher ingress limit by request. To request an increase, contact Azure Support.
    • Split tables between more storage accounts.

Monitor event hubs

  1. Configure alerts on the metrics:

    Scope Metric namespace Metric Aggregation Threshold
    namespaces-name Event Hubs standard metrics Incoming bytes Sum 80% of maximum ingress per alert evaluation period. For example, the limit is 1 MB/s per unit (TU or PU) and five units used. The threshold is 1,200 MB per 5-minute evaluation period.
    namespaces-name Event Hubs standard metrics Incoming requests Count 80% of maximum events per alert evaluation period. For example, the limit is 1,000/s per unit (TU or PU) and five units used. The threshold is 1,200,000 per 5-minute evaluation period.
    namespaces-name Event Hubs standard metrics Quota exceeded errors Count Between 1% of request. For example, requests per 5 minutes is 600,000. The threshold is 6,000 per 5-minute evaluation period.
  2. Alert remediation actions:

    • Use a separate Event Hubs namespace for export that isn't shared with non-monitoring data.
    • Configure the Auto-inflate feature to automatically scale up and increase the number of throughput units to meet usage needs.
    • Verify the increase of throughput units to accommodate data volume.
    • Split tables between more namespaces.
    • Use Premium or Dedicated tiers for higher throughput.

Create or update a data export rule

A data export rule defines the destination and tables for which data is exported. You can create 10 rules in the Enabled state in your workspace. More rules are allowed in the Disabled state. The storage account must be unique across rules in the workspace. Multiple rules can use the same Event Hubs namespace when you're sending to separate event hubs.

Note

  • You can include tables that aren't yet supported in rules, but no data will be exported for them until the tables are supported.
  • Export to a storage account: A separate container is created in the storage account for each table.
  • Export to event hubs: If an event hub name isn't provided, a separate event hub is created for each table. The number of supported event hubs in Basic and Standard namespace tiers is 10. When you're exporting more than 10 tables to these tiers, either split the tables between several export rules to different Event Hubs namespaces or provide an event hub name in the rule to export all tables to it.
  1. On the Log Analytics workspace menu in the Azure portal, select Data Export under the Settings section. Select New export rule at the top of the pane.

    Screenshot that shows the data export entry point.

  2. Follow the steps, and then select Create.

    Screenshot of data export rule configuration.

View data export rule configuration

  1. On the Log Analytics workspace menu in the Azure portal, select Data Export under the Settings section.

    Screenshot that shows the Data Export screen.

  2. Select a rule for a configuration view.

    Screenshot of data export rule view.

Disable or update an export rule

You can disable export rules to stop the export for a certain period, such as when testing is being held. On the Log Analytics workspace menu in the Azure portal, select Data Export under the Settings section. Select the Status toggle to disable or enable the export rule.

Screenshot that shows disabling the data export rule.

Delete an export rule

On the Log Analytics workspace menu in the Azure portal, select Data Export under the Settings section. Select the ellipsis to the right of the rule and select Delete.

Screenshot that shows deleting the data export rule.

View all data export rules in a workspace

On the Log Analytics workspace menu in the Azure portal, select Data Export under the Settings section to view all export rules in the workspace.

Screenshot that shows the data export rules view.

Unsupported tables

If the data export rule includes an unsupported table, the configuration will succeed, but no data will be exported for that table. If the table is later supported, then its data will be exported at that time.

Supported tables

All data from the table will be exported unless limitations are specified. This list is updated as more tables are added.

 Table   Limitations 
AACAudit
AACHttpRequest
AADB2CRequestLogs
AADDomainServicesAccountLogon
AADDomainServicesAccountManagement
AADDomainServicesDirectoryServiceAccess
AADDomainServicesLogonLogoff
AADDomainServicesPolicyChange
AADDomainServicesPrivilegeUse
AADManagedIdentitySignInLogs
AADNonInteractiveUserSignInLogs
AADProvisioningLogs
AADRiskyServicePrincipals
AADRiskyUsers
AADServicePrincipalRiskEvents
AADServicePrincipalSignInLogs
AADUserRiskEvents
ABSBotRequests
ACICollaborationAudit
ACRConnectedClientList
ACSAuthIncomingOperations
ACSBillingUsage
ACSCallAutomationIncomingOperations
ACSCallDiagnostics
ACSCallRecordingSummary
ACSCallSummary
ACSChatIncomingOperations
ACSEmailSendMailOperational
ACSEmailStatusUpdateOperational
ACSEmailUserEngagementOperational
ACSNetworkTraversalDiagnostics
ACSNetworkTraversalIncomingOperations
ACSRoomsIncomingOperations
ACSSMSIncomingOperations
ADAssessmentRecommendation
ADFActivityRun
ADFAirflowTaskLogs
ADFAirflowWorkerLogs
ADFPipelineRun
ADFSSignInLogs
ADFTriggerRun
ADPAudit
ADPDiagnostics
ADPRequests
ADReplicationResult
ADSecurityAssessmentRecommendation
ADTDataHistoryOperation
ADTDigitalTwinsOperation
ADTEventRoutesOperation
ADTModelsOperation
ADTQueryOperation
ADXCommand
ADXQuery
AegDataPlaneRequests
AegDeliveryFailureLogs
AegPublishFailureLogs
AEWAuditLogs
AEWComputePipelinesLogs
AgriFoodApplicationAuditLogs
AgriFoodFarmManagementLogs
AgriFoodFarmOperationLogs
AgriFoodInsightLogs
AgriFoodJobProcessedLogs
AgriFoodModelInferenceLogs
AgriFoodProviderAuthLogs
AgriFoodSatelliteLogs
AgriFoodSensorManagementLogs
AgriFoodWeatherLogs
AGSGrafanaLoginEvents
AHDSMedTechDiagnosticLogs
Alert Partial support. Data ingestion for Zabbix alerts isn't supported.
AlertEvidence
AlertInfo
AmlOnlineEndpointConsoleLog
AmlOnlineEndpointEventLog
AmlOnlineEndpointTrafficLog
AMSKeyDeliveryRequests
AMSLiveEventOperations
AMSMediaAccountHealth
AMSStreamingEndpointRequests
ANFFileAccess
ApiManagementGatewayLogs
AppAvailabilityResults
AppBrowserTimings
AppCenterError
AppDependencies
AppEvents
AppExceptions
AppMetrics
AppPageViews
AppPerformanceCounters
AppPlatformSystemLogs
AppRequests
AppServiceAppLogs
AppServiceAuditLogs
AppServiceConsoleLogs
AppServiceFileAuditLogs
AppServiceHTTPLogs
AppServicePlatformLogs
AppServiceServerlessSecurityPluginData
AppSystemEvents
AppTraces
ASimDnsActivityLogs
ASimNetworkSessionLogs
ATCExpressRouteCircuitIpfix
AuditLogs
AutoscaleEvaluationsLog
AutoscaleScaleActionsLog
AVSSyslog
AWSCloudTrail
AWSGuardDuty
AWSVPCFlow
AZFWApplicationRule
AZFWApplicationRuleAggregation
AZFWDnsQuery
AZFWFatFlow
AZFWFlowTrace
AZFWIdpsSignature
AZFWInternalFqdnResolutionFailure
AZFWNatRule
AZFWNatRuleAggregation
AZFWNetworkRule
AZFWNetworkRuleAggregation
AZFWThreatIntel
AzureAssessmentRecommendation
AzureAttestationDiagnostics
AzureDevOpsAuditing
AzureLoadTestingOperation
BehaviorAnalytics
CassandraAudit
CassandraLogs
CCFApplicationLogs
CDBCassandraRequests
CDBControlPlaneRequests
CDBDataPlaneRequests
CDBGremlinRequests
CDBMongoRequests
CDBPartitionKeyRUConsumption
CDBPartitionKeyStatistics
CDBQueryRuntimeStatistics
CIEventsAudit
CIEventsOperational
CloudAppEvents
CommonSecurityLog
ComputerGroup
ConfidentialWatchlist
ConfigurationData Partial support. Some of the data is ingested through internal services that aren't supported in export. Currently, this portion is missing in export.
ContainerAppConsoleLogs
ContainerAppSystemLogs
ContainerImageInventory
ContainerInventory
ContainerLog
ContainerLogV2
ContainerNodeInventory
ContainerServiceLog
CoreAzureBackup
DatabricksAccounts
DatabricksClusters
DatabricksDBFS
DatabricksInstancePools
DatabricksJobs
DatabricksNotebook
DatabricksSecrets
DatabricksSQLPermissions
DatabricksSSH
DatabricksWorkspace
DevCenterDiagnosticLogs
DeviceTvmSecureConfigurationAssessment
DeviceTvmSoftwareInventory
DeviceTvmSoftwareVulnerabilities
DnsEvents
DnsInventory
DSMAzureBlobStorageLogs
DSMDataClassificationLogs
DSMDataLabelingLogs
Dynamics365Activity
DynamicSummary
EmailAttachmentInfo
EmailEvents
EmailPostDeliveryEvents
EmailUrlInfo
Event Partial support. Data arriving from the Log Analytics agent (MMA) or Azure Monitor Agent (AMA) is fully supported in export. Data arriving via the Diagnostics extension agent is collected through storage. This path isn't supported in export.
ExchangeAssessmentRecommendation
FailedIngestion
FunctionAppLogs
GCPAuditLogs
HDInsightAmbariClusterAlerts
HDInsightAmbariSystemMetrics
HDInsightGatewayAuditLogs
HDInsightHadoopAndYarnLogs
HDInsightHadoopAndYarnMetrics
HDInsightHBaseLogs
HDInsightHBaseMetrics
HDInsightHiveAndLLAPLogs
HDInsightHiveAndLLAPMetrics
HDInsightHiveQueryAppStats
HDInsightHiveTezAppStats
HDInsightJupyterNotebookEvents
HDInsightKafkaLogs
HDInsightKafkaMetrics
HDInsightOozieLogs
HDInsightRangerAuditLogs
HDInsightSecurityLogs
HDInsightSparkApplicationEvents
HDInsightSparkBlockManagerEvents
HDInsightSparkEnvironmentEvents
HDInsightSparkExecutorEvents
HDInsightSparkExtraEvents
HDInsightSparkJobEvents
HDInsightSparkLogs
HDInsightSparkSQLExecutionEvents
HDInsightSparkStageEvents
HDInsightSparkStageTaskAccumulables
HDInsightSparkTaskEvents
HDInsightStormLogs
HDInsightStormMetrics
HDInsightStormTopologyMetrics
Heartbeat
HuntingBookmark
IdentityDirectoryEvents
IdentityLogonEvents
IdentityQueryEvents
InsightsMetrics Partial support. Some of the data is ingested through internal services that aren't supported in export. Currently, this portion is missing in export.
IntuneAuditLogs
IntuneDevices
IntuneOperationalLogs
KubeEvents
KubeHealth
KubeMonAgentEvents
KubeNodeInventory
KubePodInventory
KubeServices
LAQueryLogs
McasShadowItReporting
MCCEventLogs
MCVPAuditLogs
MCVPOperationLogs
MicrosoftAzureBastionAuditLogs
MicrosoftDataShareReceivedSnapshotLog
MicrosoftDataShareSentSnapshotLog
MicrosoftHealthcareApisAuditLogs
MicrosoftPurviewInformationProtection
NetworkAccessTraffic
NSPAccessLogs
NWConnectionMonitorPathResult
NWConnectionMonitorTestResult
OEPAirFlowTask
OEPElasticOperator
OEPElasticsearch
OfficeActivity
OLPSupplyChainEntityOperations
OLPSupplyChainEvents
Operation Partial support. Some of the data is ingested through internal services that aren't supported in export. Currently, this portion is missing in export.
Perf Partial support. Only Windows perf data is currently supported. Currently, the Linux perf data is missing in export.
PFTitleAuditLogs
PowerBIActivity
PowerBIAuditTenant
PowerBIDatasetsTenant
PowerBIDatasetsWorkspace
PowerBIReportUsageWorkspace
ProjectActivity
PurviewDataSensitivityLogs
PurviewScanStatusLogs
PurviewSecurityLogs
ResourceManagementPublicAccessLogs
SCCMAssessmentRecommendation
SCOMAssessmentRecommendation
SecurityAlert
SecurityBaseline
SecurityBaselineSummary
SecurityDetection
SecurityEvent Partial support. Data arriving from the Log Analytics agent or Azure Monitor Agent is fully supported in export. Data arriving via the Diagnostics extension agent is collected through storage. This path isn't supported in export.
SecurityIncident
SecurityIoTRawEvent
SecurityNestedRecommendation
SecurityRecommendation
SentinelAudit
SentinelHealth
SfBAssessmentRecommendation
SfBOnlineAssessmentRecommendation
SharePointOnlineAssessmentRecommendation
SignalRServiceDiagnosticLogs
SigninLogs
SPAssessmentRecommendation
SQLAssessmentRecommendation
SQLSecurityAuditEvents
StorageAntimalwareScanResults
StorageCacheOperationEvents
StorageCacheUpgradeEvents
StorageCacheWarningEvents
SucceededIngestion
SynapseBigDataPoolApplicationsEnded
SynapseBuiltinSqlPoolRequestsEnded
SynapseGatewayApiRequests
SynapseIntegrationActivityRuns
SynapseIntegrationPipelineRuns
SynapseIntegrationTriggerRuns
SynapseLinkEvent
SynapseRbacOperations
SynapseScopePoolScopeJobsEnded
SynapseScopePoolScopeJobsStateChange
SynapseSqlPoolDmsWorkers
SynapseSqlPoolExecRequests
SynapseSqlPoolRequestSteps
SynapseSqlPoolSqlRequests
SynapseSqlPoolWaits
Syslog Partial support. Data arriving from the Log Analytics agent or Azure Monitor Agent is fully supported in export. Data arriving via the Diagnostics extension agent is collected through storage. This path isn't supported in export.
ThreatIntelligenceIndicator
UCClient
UCDOAggregatedStatus
UCDOStatus
Update Partial support. Some of the data is ingested through internal services that aren't supported in export. Currently, this portion is missing in export.
UpdateRunProgress
UpdateSummary
UrlClickEvents
Usage
UserAccessAnalytics
UserPeerAnalytics
VIAudit
VIIndexing
W3CIISLog Partial support. Data arriving from Log Analytics agent (MMA) or Azure Monitor Agent (AMA) is fully supported in export. Data arriving via Diagnostics extension agent is collected through storage while this path isn’t supported in export.
Watchlist
WindowsEvent
WindowsFirewall
WireData Partial support. Some of the data is ingested through internal services that aren't supported in export. Currently, this portion is missing in export.
WorkloadDiagnosticLogs
WVDAgentHealthStatus
WVDCheckpoints
WVDConnections
WVDErrors
WVDFeeds
WVDManagement

Next steps

Query the exported data from Azure Data Explorer