Migrate from Splunk to Azure Monitor Logs
Azure Monitor Logs is a cloud-based managed monitoring and observability service that provides many advantages in terms of cost management, scalability, flexibility, integration, and low maintenance overhead. The service is designed to handle large amounts of data and scale easily to meet the needs of organizations of all sizes.
Azure Monitor Logs collects data from a wide variety of sources, including Windows Event logs, Syslog, and custom logs, to provide a unified view of all Azure and non-Azure resources. Using a sophisticated query language and curated visualization you can quickly analyze millions of records to identify, understand, and respond to critical patterns in your monitoring data.
This article explains how to migrate your Splunk Observability deployment to Azure Monitor Logs for logging and log data analysis.
For information on migrating your Security Information and Event Management (SIEM) deployment from Splunk Enterprise Security to Azure Sentinel, see Plan your migration to Microsoft Sentinel.
Why migrate to Azure Monitor?
The benefits of migrating to Azure Monitor include:
- Fully managed, Software as a Service (SaaS) platform with:
- Cloud-native monitoring and observability, including:
- Native integration with a range of complementary Azure services, such as Microsoft Sentinel for security information and event management, Azure Logic Apps for automation, Azure Managed Grafana for dashboarding, and Azure Machine Learning for advanced analysis and response capabilities.
|Splunk offering||Azure offering|
|Splunk Observability||Azure Monitor is an end-to-end solution for collecting, analyzing, and acting on telemetry from your cloud, multicloud, and on-premises environments, built over a powerful data ingestion pipeline that's shared with Microsoft Sentinel. Azure Monitor offers enterprises a comprehensive solution for monitoring cloud, hybrid, and on-premises environments, with network isolation, resilience features and protection from data center failures, reporting, and alerts and response capabilities.|
|Splunk Security||Microsoft Sentinel is a cloud-native solution that runs over the Azure Monitor platform to provide intelligent security analytics and threat intelligence across the enterprise.|
Introduction to key concepts
|Azure Monitor Logs||Similar Splunk concept||Description|
|Log Analytics workspace||Namespace||A Log Analytics workspace is an environment in which you can collect log data from all Azure and non-Azure monitored resources. The data in the workspace is available for querying and analysis, Azure Monitor features, and other Azure services. Similar to a Splunk namespace, you can manage access to the data and artifacts, such as alerts and workbooks, in your Log Analytics workspace.|
|Table management||Indexing||Azure Monitor Logs ingests log data into tables in a managed Azure Data Explorer database. During ingestion, the service automatically indexes and timestamps the data, which means you can store various types of data and access the data quickly using Kusto Query Language (KQL) queries.
Use table properties to manage the table schema, data retention and archive, and whether to store the data for occasional auditing and troubleshooting or for ongoing analysis and use by features and services.
For a comparison of Splunk and Azure Data Explorer data handling and querying concepts, see Splunk to Kusto Query Language map.
|Basic and Analytics log data plans||Azure Monitor Logs offers two log data plans that let you reduce log ingestion and retention costs and take advantage of Azure Monitor's advanced features and analytics capabilities based on your needs.
The Analytics plan makes log data available for interactive queries and use by features and services.
The Basic log data plan provides a low-cost way to ingest and retain logs for troubleshooting, debugging, auditing, and compliance.
|Archiving and quick access to archived data||Data bucket states (hot, warm, cold, thawed), archiving, Dynamic Data Active Archive (DDAA)||The cost-effective archive option keeps your logs in your Log Analytics workspace and lets you access archived log data immediately, when you need it. Archive configuration changes are effective immediately because data isn't physically transferred to external storage. You can restore archived data or run a search job to make a specific time range of archived data available for real-time analysis.|
|Access control||Role-based user access, permissions||Role-based access control lets you define which people in your organization have access to read, write, and perform operations in a Log Analytics workspace. You can configure permissions at the workspace level, at the resource level, and at the table level, so you have granular control over specific resources and log types.|
|Data transformations||Transforms, field extractions||Transformations let you filter or modify incoming data before it's sent to a Log Analytics workspace. Use transformations to remove sensitive data, enrich data in your Log Analytics workspace, perform calculations, and filter out data you don't need to reduce data costs.|
|Data collection rules||Data inputs, data pipeline||Define which data to collect, how to transform that data, and where to send the data.|
|Kusto Query Language (KQL)||Splunk Search Processing Language (SPL)||Azure Monitor Logs uses a large subset of KQL that's suitable for simple log queries but also includes advanced functionality such as aggregations, joins, and smart analytics. Use the Splunk to Kusto Query Language map to translate your Splunk SPL knowledge to KQL. You can also learn KQL with tutorials and KQL training modules.|
|Log Analytics||Splunk Web, Search app, Pivot tool||A tool in the Azure portal for editing and running log queries in Azure Monitor Logs. Log Analytics also provides a rich set of tools for exploring and visualizing data without using KQL.|
|Cost optimization||Azure Monitor provides tools and best practices to help you understand, monitor, and optimize your costs based on your needs.|
1. Understand your current usage
Your current usage in Splunk will help you decide which pricing tier to select in Azure Monitor and estimate your future costs:
- Follow Splunk guidance to view your usage report.
- Estimate Azure Monitor usage and costs using the Pricing Calculator.
2. Set up a Log Analytics workspace
Your Log Analytics workspace is where you collect log data from all of your monitored resources. You can retain data in a Log Analytics workspace for up to seven years. Low-cost data archiving within the workspace lets you access archived data quickly and easily when you need it, without the overhead of managing an external data store.
We recommend collecting all of your log data in a single Log Analytics workspace for ease of management. If you're considering using multiple workspaces, see Design a Log Analytics workspace architecture.
To set up a Log Analytics workspace for data collection:
Azure Monitor Logs creates Azure tables in your workspace automatically based on Azure services you use and data collection settings you define for Azure resources.
Configure your Log Analytics workspace, including:
The default log data plan is Analytics, which lets you take advantage of Azure Monitor's rich monitoring and analytics capabilities.
Set a data retention and archiving policy for specific tables, if you need them to be different from the workspace-level data retention and archiving policy.
Modify the table schema based on your data model.
3. Migrate Splunk artifacts to Azure Monitor
To migrate most Splunk artifacts, you need to translate Splunk Processing Language (SPL) to Kusto Query Language (KQL). For more information, see the Splunk to Kusto Query Language map and Get started with log queries in Azure Monitor.
This table lists Splunk artifacts and links to guidance for setting up the equivalent artifacts in Azure Monitor:
|Splunk artifact||Azure Monitor artifact|
|Alert actions||Action groups|
|Apps||Azure Monitor Insights are a set of ready-to-use, curated monitoring experiences with pre-configured data inputs, searches, alerts, and visualizations to get you started analyzing data quickly and effectively.|
|Lookups||Azure Monitor provides various ways to enrich data, including:
- Data collection rules, which let you send data from multiple sources to a Log Analytics workspace, and perform calculations and transformations before ingesting the data.
- KQL operators, such as the join operator, which combines data from different tables, and the externaldata operator, which returns data from external storage.
- Integration with services, such as Azure Machine Learning or Azure Event Hubs, to apply advanced machine learning and stream in additional data.
|Namespaces||You can grant or limit permission to artifacts in Azure Monitor based on access control you define on your Log Analytics workspace or Azure resource groups.|
|Reports||Azure Monitor offers a range of options for analyzing, visualizing, and sharing data, including:
- Integration with Grafana
- Integration with Power BI
- Integration with Excel
|Source types||Define your data model in your Log Analytics workspace. Use ingestion-time transformations to filter, format, or modify incoming data.|
|Data collections methods||See Collect data for Azure Monitor tools designed for specific resources.|
For information on migrating Splunk SIEM artifacts, including detection rules and SOAR automation, see Plan your migration to Microsoft Sentinel.
4. Collect data
Azure Monitor provides tools for collecting data from log data sources on Azure and non-Azure resources in your environment.
To collect data from a resource:
- Set up the relevant data collection tool based on the table below.
- Decide which data you need to collect from the resource.
- Use transformations to remove sensitive data, enrich data or perform calculations, and filter out data you don't need, to reduce costs.
This table lists the tools Azure Monitor provides for collecting data from various resource types.
|Resource type||Data collection tool||Similar Splunk tool||Collected data|
|Azure||Diagnostic settings||Azure tenant - Azure Active Directory Audit Logs provide sign-in activity history and audit trail of changes made within a tenant.
Azure resources - Logs and performance counters.
Azure subscription - Service health records along with records on any configuration changes made to the resources in your Azure subscription.
|Application||Application insights||Splunk Application Performance Monitoring||Application performance monitoring data.|
|Container||Container insights||Container Monitoring||Container performance data.|
|Operating system||Azure Monitor Agent||Universal Forwarder, Heavy Forwarder||Monitoring data from the guest operating system of Azure and non-Azure virtual machines.|
|Non-Azure source||Logs Ingestion API||HTTP Event Collector (HEC)||File-based logs and any data you send to a data collection endpoint on a monitored resource.|
5. Transition to Azure Monitor Logs
A common approach is to transition to Azure Monitor Logs gradually, while maintaining historical data in Splunk. During this period, you can:
- Use the Log ingestion API to ingest data from Splunk.
- Use Log Analytics workspace data export to export data out of Azure Monitor.
To export your historical data from Splunk:
- Use one of the Splunk export methods to export data in CSV format.
- To collect the exported data:
Use Azure Monitor Agent to collect the data you export from Splunk, as described in Collect text logs with Azure Monitor Agent.
Collect the exported data directly with the Logs Ingestion API, as described in Send data to Azure Monitor Logs by using a REST API.
- Learn more about using Log Analytics and the Log Analytics Query API.
- Enable Microsoft Sentinel on your Log Analytics workspace.
- Take the Analyze logs in Azure Monitor with KQL training module.