Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Custom analyzers let you control how Azure DocumentDB processes text for full-text search. Define a tokenization and token-filtering pipeline to support case-insensitive matching, prefix search, hierarchical identifiers, and language-specific analysis.
You declare custom analyzers inside a search index definition and reference them by name from searchable string fields. A single index can cover multiple fields, and each field can use a different analyzer.
Note
Full-text search is in preview. Before using these examples, enable full-text search on your cluster.
How custom analyzers work
An analyzer transforms an input string into searchable tokens. It has two components:
- Tokenizer. Splits the input into tokens. Each analyzer has exactly one tokenizer.
- Token filters. Normalize or transform the tokens. An analyzer can have zero or more token filters, applied in the order you specify.
Analysis runs when documents are indexed and when query text is processed:
| Setting | Purpose |
|---|---|
analyzer |
Processes field values at index time. |
searchAnalyzer |
Processes query text at query time. If omitted, the index analyzer is also used for queries. |
multi |
Defines alternate analyzer mappings for the same string field. |
Set analyzer and searchAnalyzer at the index-definition level to provide defaults, or on a string field to override them. Query tokens match indexed tokens when the two pipelines produce the same normalized form.
The pipelines don't always need to be identical. For example, prefix search can generate prefixes at index time and preserve the complete query string at query time.
When to use custom analyzers
Use a custom analyzer when a built-in analyzer doesn't meet your indexing or query requirements. Choose one supported tokenizer and combine supported token filters in the order your workload needs.
| Requirement | Analyzer configuration |
|---|---|
| Case-insensitive matching | Add a lowerCase token filter. |
| Accent-insensitive matching | Add an asciiFolding token filter. |
| Prefix matching on part numbers or SKUs | Use an edgeGram tokenizer or token filter. |
| Hierarchical identifier search | Use a pathHierarchy tokenizer. |
| Matching word forms | Add a language-specific stemmer token filter. |
| Removing common words | Add a stopWords token filter. |
| Splitting identifiers on selected separators | Use a charGroup or pattern tokenizer. |
| Alternate analysis of the same field | Define named multi mappings. |
Supported tokenizers
Use one of the following tokenizer types in a custom analyzer.
| Tokenizer | Options | Purpose |
|---|---|---|
whitespace |
None. | Splits text on whitespace. |
keyword |
None. | Preserves the complete input as a single token. |
pattern |
Required pattern; optional group, default 0. |
Splits or extracts tokens using a regular expression. |
charGroup |
Required tokenizeOnChars array. |
Splits on whitespace or specified literal characters. |
pathHierarchy |
Optional delimiter, default /; replacement, default delimiter; skip, default 0; and reverse, default false. |
Generates tokens for successive levels of a hierarchical path. The delimiter and replacement are single characters. |
edgeGram |
Required minGram and maxGram. |
Generates prefixes anchored at the start of the input. |
nGram |
Required minGram and maxGram. |
Generates sliding character sequences throughout the input. |
simple |
None. | Lowercases text and splits on nonletter characters. |
chinese, japanese, korean, thai |
None. | Applies language-specific tokenization. |
For edgeGram and nGram, both gram lengths must be integers of at least 1, and minGram must not exceed maxGram.
For pattern, specify regular expression flags inline, such as (?i). A separate tokenizer.flags option isn't supported. The default capture group is 0, which identifies the whole match.
Supported token filters
Add token filters in the order you want to apply them.
| Token filter | Options | Purpose |
|---|---|---|
lowerCase |
None. | Converts tokens to lowercase. |
asciiFolding |
None. | Converts non-ASCII characters to their closest ASCII equivalents where available. |
removeLong |
None. | Removes tokens that exceed the analyzer's supported token length. |
length |
Optional min, default 0; optional max, default 255. |
Keeps tokens within a specified length range. |
stemmer |
Required language. |
Reduces inflected words to a language-specific root form. |
stopWords |
Either tokens or language. |
Removes an explicit list of tokens or language-specific stop words. |
edgeGram |
Required minGram and maxGram, both at least 1. |
Generates prefixes from each token produced by the tokenizer. |
patternCapture |
Required patterns array; optional preserveOriginal, default false. |
Extracts token content using regular expression patterns. |
patternReplace |
Required pattern and replacement; optional replaceAll, default true. |
Replaces matching content within tokens. |
For stopWords, specify either a tokens array or a language, not both. For the edgeGram filter, minGram must not exceed maxGram.
Create a search index with custom analyzers
The following examples use MongoDB Shell syntax and a connected Azure DocumentDB cluster with full-text search enabled. Use a new collection named products.
Add sample documents
Create the collection and add fictional product data for prefix, hierarchy, and text queries.
db.createCollection("products");
db.products.insertMany([
{
_id: 1,
partNumber: "SKU-SHOE-001",
alternatePartNumber: "ALT-SHOE-001",
basicNumber: "catalog/outdoor/running",
title: "Running shoes",
description: "Lightweight shoes for everyday running"
},
{
_id: 2,
partNumber: "SKU-BOOT-002",
alternatePartNumber: "SKU-SHOE-002",
basicNumber: "catalog/outdoor/hiking",
title: "Trail shoes",
description: "Durable shoes for outdoor trails"
},
{
_id: 3,
partNumber: "SKU-SANDAL-003",
alternatePartNumber: "ALT-SANDAL-003",
basicNumber: "catalog/indoor/casual",
title: "Walking shoes",
description: "Comfortable shoes for everyday walking"
}
]);
Define the analyzers and field mappings
Use createSearchIndexes to create a named search index. Set dynamic: false and declare each searchable field explicitly.
db.runCommand({
createSearchIndexes: "products",
indexes: [
{
name: "idx_products_analyzers",
definition: {
mappings: {
dynamic: false,
fields: {
partNumber: {
type: "string",
analyzer: "part_prefix",
searchAnalyzer: "identifier_query"
},
alternatePartNumber: {
type: "string",
analyzer: "part_prefix",
searchAnalyzer: "identifier_query"
},
basicNumber: {
type: "string",
analyzer: "basic_hierarchy",
searchAnalyzer: "identifier_query"
},
title: {
type: "string",
analyzer: "normalized_text",
multi: {
stems: {
type: "string",
analyzer: "english_stems"
}
}
},
description: {
type: "string",
analyzer: "normalized_text"
}
}
},
analyzers: [
{
name: "part_prefix",
tokenizer: {
type: "edgeGram",
minGram: 2,
maxGram: 20
},
tokenFilters: [
{ type: "lowerCase" }
]
},
{
name: "identifier_query",
tokenizer: { type: "keyword" },
tokenFilters: [
{ type: "lowerCase" }
]
},
{
name: "basic_hierarchy",
tokenizer: {
type: "pathHierarchy",
delimiter: "/"
},
tokenFilters: [
{ type: "lowerCase" }
]
},
{
name: "normalized_text",
tokenizer: { type: "whitespace" },
tokenFilters: [
{ type: "lowerCase" },
{ type: "asciiFolding" }
]
},
{
name: "english_stems",
tokenizer: { type: "whitespace" },
tokenFilters: [
{ type: "lowerCase" },
{ type: "stemmer", language: "english" }
]
}
]
}
}
]
});
Confirm the index is ready
Index creation can continue after the command returns. Inspect the named index and wait until it reports status: "READY" and queryable: true before running the queries.
db.products.aggregate([
{ $listSearchIndexes: { name: "idx_products_analyzers" } }
]);
Run a prefix query
Prefix search matches the beginning of a field value. It's useful for part numbers, SKUs, and other identifiers that users enter incrementally.
The part_prefix analyzer generates prefixes at index time. The identifier_query analyzer preserves the query as one lowercased token.
db.products.aggregate([
{
$search: {
index: "idx_products_analyzers",
text: {
query: "sku-shoe",
path: "partNumber"
}
}
},
{ $limit: 20 },
{
$project: {
_id: 0,
partNumber: 1,
score: { $meta: "searchScore" }
}
}
]);
This query matches SKU-SHOE-001. Choose minGram and maxGram for the prefix lengths your application needs. Prefixes outside that range aren't generated by this analyzer. Generating more prefixes increases index size.
An edgeGram tokenizer generates prefixes from the input. An edgeGram token filter generates prefixes from each token produced by the tokenizer. Use the filter when you need to tokenize text into words before generating prefixes.
Prefix matching with edgeGram doesn't require the autocomplete operator. That operator isn't currently supported.
Run a hierarchical query
The pathHierarchy tokenizer generates tokens for successive levels of an identifier. With / as the delimiter, catalog/outdoor/running produces:
catalog
catalog/outdoor
catalog/outdoor/running
Use the query-time keyword tokenizer to search for a particular hierarchy level without expanding the query into its parent levels.
db.products.aggregate([
{
$search: {
index: "idx_products_analyzers",
text: {
query: "catalog/outdoor",
path: "basicNumber"
}
}
},
{ $limit: 20 },
{
$project: {
_id: 0,
basicNumber: 1,
score: { $meta: "searchScore" }
}
}
]);
This query matches catalog/outdoor/running and catalog/outdoor/hiking. Use the optional replacement, skip, and reverse settings to customize how hierarchical tokens are generated.
Search multiple identifier fields
One search index can cover multiple identifier fields. Pass an explicit array of mapped paths to text to search them together.
db.products.aggregate([
{
$search: {
index: "idx_products_analyzers",
text: {
query: "sku-shoe",
path: ["partNumber", "alternatePartNumber"],
matchCriteria: "any"
}
}
},
{ $limit: 20 },
{
$project: {
_id: 0,
partNumber: 1,
alternatePartNumber: 1,
score: { $meta: "searchScore" }
}
}
]);
This query matches the first document through partNumber and the second document through alternatePartNumber. Every path must be mapped in the selected index. If no single index covers all requested paths, the query returns an error.
Use multiple analyzers on the same field
A multi mapping indexes the same string field with an alternate analyzer. This feature supports different matching behaviors without storing duplicate document fields.
The example index defines a stems mapping on title. Select that mapping with path: { value, multi }:
db.products.aggregate([
{
$search: {
index: "idx_products_analyzers",
text: {
query: "running",
path: {
value: "title",
multi: "stems"
}
}
}
},
{ $limit: 20 },
{
$project: {
_id: 0,
title: 1,
score: { $meta: "searchScore" }
}
}
]);
The alternate analyzer applies English stemming to indexed text and query text. A multi mapping differs from a multi-field index: multi provides several analyzer variants of one field, while a multi-field index covers several document fields.
Use language-specific analyzers
Azure DocumentDB provides built-in analyzers for Chinese, Japanese, Korean, and Thai. Reference these analyzers with the documentdb. prefix. Inside a custom pipeline, use the unprefixed tokenizer name.
| Language | Built-in analyzer | Custom tokenizer type |
|---|---|---|
| Chinese | documentdb.chinese |
chinese |
| Japanese | documentdb.japanese |
japanese |
| Korean | documentdb.korean |
korean |
| Thai | documentdb.thai |
thai |
Set a default analyzer
Set analyzer on the index definition to provide a default for mapped string fields. A field-level analyzer overrides that default. If you don't configure a searchAnalyzer, queries use the effective index-time analyzer for the field.
The following index sets documentdb.chinese as its default analyzer. The titleZh field inherits that default, while descriptionZh overrides it with a custom language-tokenizer pipeline:
db.runCommand({
createSearchIndexes: "products",
indexes: [
{
name: "idx_products_languages",
definition: {
analyzer: "documentdb.chinese",
mappings: {
dynamic: false,
fields: {
titleZh: {
type: "string"
},
descriptionZh: {
type: "string",
analyzer: "chinese_normalized"
}
}
},
analyzers: [
{
name: "chinese_normalized",
tokenizer: { type: "chinese" },
tokenFilters: [
{ type: "lowerCase" }
]
}
]
}
}
]
});
Current limitations
The following constraints apply to custom analyzers and their field mappings:
- Setting
dynamic: falseor omittingdynamicuses static mappings. You must explicitly declare every searchable field undermappings.fields. Settingdynamic: truecauses index creation to fail withmappings.dynamic is not supported in search indexes yet.Index creation also fails ifmappings.fieldsis missing or empty. - Custom analyzer names must be unique within the index definition and no longer than 32 characters.
- A field's analyzer settings must reference a supported built-in analyzer or a custom analyzer declared in the index definition.
- Character filters (
charFilters) and synonym definitions aren't supported. - The
autocompleteandqueryStringoperators aren't supported. - The
regexandwildcardoperators inside$searcharen't supported.