Run search jobs in Azure Monitor
Search jobs are asynchronous queries that fetch records into a new search table within your workspace for further analytics. The search job uses parallel processing and can run for hours across large datasets. This article describes how to create a search job and how to query its resulting data.
The search job feature is currently not supported for the following cases:
- Workspaces with customer-managed keys.
- Workspaces in the China East 2 region.
When to use search jobs
Use a search job when the log query timeout of 10 minutes isn't sufficient to search through large volumes of data or if you're running a slow query.
Restoring data from Archived Logs for a specific time range.
Use restore when you have a temporary need to run many queries on a large volume of data.
Querying Basic Logs directly and paying for each query.
To determine which alternative is more cost-effective, compare the cost of querying Basic Logs with the cost of running a search job and storing the search job results.
What does a search job do?
A search job sends its results to a new table in the same workspace as the source data. The results table is available as soon as the search job begins, but it may take time for results to begin to appear.
The search job results table is an Analytics table that is available for log queries and other Azure Monitor features that use tables in a workspace. The table uses the retention value set for the workspace, but you can modify this value after the table is created.
The search results table schema is based on the source table schema and the specified query. The following other columns help you track the source records:
|_OriginalType||Type value from source table.|
|_OriginalItemId||_ItemID value from source table.|
|_OriginalTimeGenerated||TimeGenerated value from source table.|
|TimeGenerated||Time at which the search job ran.|
Queries on the results table appear in log query auditing but not the initial search job.
Run a search job
Run a search job to fetch records from large datasets into a new search results table in your workspace.
You incur charges for running a search job. Therefore, write and optimize your query in interactive query mode before running the search job.
To run a search job, in the Azure portal:
From the Log Analytics workspace menu, select Logs.
Select the ellipsis menu on the right-hand side of the screen and toggle Search job mode on.
Azure Monitor Logs intellisense supports KQL query limitations in search job mode to help you write your search job query.
Specify the search job date range using the time picker.
Type the search job query and select the Search Job button.
Azure Monitor Logs prompts you to provide a name for the result set table and informs you that the search job is subject to billing.
Enter a name for the search job result table and select Run a search job.
Azure Monitor Logs runs the search job and creates a new table in your workspace for your search job results.
When the new table is ready, select View tablename_SRCH to view the table in Log Analytics.
You can see the search job results as they begin flowing into the newly created search job results table.
Azure Monitor Logs shows a Search job is done message at the end of the search job. The results table is now ready with all the records that match the search query.
Get search job status and details
Delete a search job table
We recommend you delete the search job table when you're done querying the table. This reduces workspace clutter and extra charges for data retention.
Search jobs are subject to the following limitations:
- Optimized to query one table at a time.
- Search date range is up to one year.
- Supports long running searches up to a 24-hour time-out.
- Results are limited to one million records in the record set.
- Concurrent execution is limited to five search jobs per workspace.
- Limited to 100 search results tables per workspace.
- Limited to 100 search job executions per day per workspace.
When you reach the record limit, Azure aborts the job with a status of partial success, and the table will contain only records ingested up to that point.
KQL query limitations
Search jobs are intended to scan large volumes of data in a specific table. Therefore, search job queries must always start with a table name. To enable asynchronous execution using distribution and segmentation, the query supports a subset of KQL, including the operators:
You can use all functions and binary operators within these operators.
The charge for a search job is based on:
- Search job execution - the amount of data the search job needs to scan.
- Search job results - the amount of data ingested in the results table, based on the regular log data ingestion prices.
For example, if your table holds 500 GB per day, for a query on three days, you'll be charged for 1500 GB of scanned data. If the job returns 1000 records, you'll be charged for ingesting these 1000 records into the results table.
Search job execution is free until early 2023. In other words, until early 2023, you will only incur charges for ingesting the search results, not for executing the search job.
For more information, see Azure Monitor pricing.