Custom analyzers

Custom analyzers let you control how Azure DocumentDB processes text for full-text search. Define a tokenization and token-filtering pipeline to support case-insensitive matching, prefix search, hierarchical identifiers, and language-specific analysis.

You declare custom analyzers inside a search index definition and reference them by name from searchable string fields. A single index can cover multiple fields, and each field can use a different analyzer.

Note

Full-text search is in preview. Before using these examples, enable full-text search on your cluster.

How custom analyzers work

An analyzer transforms an input string into searchable tokens. It has two components:

  • Tokenizer. Splits the input into tokens. Each analyzer has exactly one tokenizer.
  • Token filters. Normalize or transform the tokens. An analyzer can have zero or more token filters, applied in the order you specify.

Analysis runs when documents are indexed and when query text is processed:

Setting Purpose
analyzer Processes field values at index time.
searchAnalyzer Processes query text at query time. If omitted, the index analyzer is also used for queries.
multi Defines alternate analyzer mappings for the same string field.

Set analyzer and searchAnalyzer at the index-definition level to provide defaults, or on a string field to override them. Query tokens match indexed tokens when the two pipelines produce the same normalized form.

The pipelines don't always need to be identical. For example, prefix search can generate prefixes at index time and preserve the complete query string at query time.

When to use custom analyzers

Use a custom analyzer when a built-in analyzer doesn't meet your indexing or query requirements. Choose one supported tokenizer and combine supported token filters in the order your workload needs.

Requirement Analyzer configuration
Case-insensitive matching Add a lowerCase token filter.
Accent-insensitive matching Add an asciiFolding token filter.
Prefix matching on part numbers or SKUs Use an edgeGram tokenizer or token filter.
Hierarchical identifier search Use a pathHierarchy tokenizer.
Matching word forms Add a language-specific stemmer token filter.
Removing common words Add a stopWords token filter.
Splitting identifiers on selected separators Use a charGroup or pattern tokenizer.
Alternate analysis of the same field Define named multi mappings.

Supported tokenizers

Use one of the following tokenizer types in a custom analyzer.

Tokenizer Options Purpose
whitespace None. Splits text on whitespace.
keyword None. Preserves the complete input as a single token.
pattern Required pattern; optional group, default 0. Splits or extracts tokens using a regular expression.
charGroup Required tokenizeOnChars array. Splits on whitespace or specified literal characters.
pathHierarchy Optional delimiter, default /; replacement, default delimiter; skip, default 0; and reverse, default false. Generates tokens for successive levels of a hierarchical path. The delimiter and replacement are single characters.
edgeGram Required minGram and maxGram. Generates prefixes anchored at the start of the input.
nGram Required minGram and maxGram. Generates sliding character sequences throughout the input.
simple None. Lowercases text and splits on nonletter characters.
chinese, japanese, korean, thai None. Applies language-specific tokenization.

For edgeGram and nGram, both gram lengths must be integers of at least 1, and minGram must not exceed maxGram.

For pattern, specify regular expression flags inline, such as (?i). A separate tokenizer.flags option isn't supported. The default capture group is 0, which identifies the whole match.

Supported token filters

Add token filters in the order you want to apply them.

Token filter Options Purpose
lowerCase None. Converts tokens to lowercase.
asciiFolding None. Converts non-ASCII characters to their closest ASCII equivalents where available.
removeLong None. Removes tokens that exceed the analyzer's supported token length.
length Optional min, default 0; optional max, default 255. Keeps tokens within a specified length range.
stemmer Required language. Reduces inflected words to a language-specific root form.
stopWords Either tokens or language. Removes an explicit list of tokens or language-specific stop words.
edgeGram Required minGram and maxGram, both at least 1. Generates prefixes from each token produced by the tokenizer.
patternCapture Required patterns array; optional preserveOriginal, default false. Extracts token content using regular expression patterns.
patternReplace Required pattern and replacement; optional replaceAll, default true. Replaces matching content within tokens.

For stopWords, specify either a tokens array or a language, not both. For the edgeGram filter, minGram must not exceed maxGram.

Create a search index with custom analyzers

The following examples use MongoDB Shell syntax and a connected Azure DocumentDB cluster with full-text search enabled. Use a new collection named products.

Add sample documents

Create the collection and add fictional product data for prefix, hierarchy, and text queries.

db.createCollection("products");

db.products.insertMany([
  {
    _id: 1,
    partNumber: "SKU-SHOE-001",
    alternatePartNumber: "ALT-SHOE-001",
    basicNumber: "catalog/outdoor/running",
    title: "Running shoes",
    description: "Lightweight shoes for everyday running"
  },
  {
    _id: 2,
    partNumber: "SKU-BOOT-002",
    alternatePartNumber: "SKU-SHOE-002",
    basicNumber: "catalog/outdoor/hiking",
    title: "Trail shoes",
    description: "Durable shoes for outdoor trails"
  },
  {
    _id: 3,
    partNumber: "SKU-SANDAL-003",
    alternatePartNumber: "ALT-SANDAL-003",
    basicNumber: "catalog/indoor/casual",
    title: "Walking shoes",
    description: "Comfortable shoes for everyday walking"
  }
]);

Define the analyzers and field mappings

Use createSearchIndexes to create a named search index. Set dynamic: false and declare each searchable field explicitly.

db.runCommand({
  createSearchIndexes: "products",
  indexes: [
    {
      name: "idx_products_analyzers",
      definition: {
        mappings: {
          dynamic: false,
          fields: {
            partNumber: {
              type: "string",
              analyzer: "part_prefix",
              searchAnalyzer: "identifier_query"
            },
            alternatePartNumber: {
              type: "string",
              analyzer: "part_prefix",
              searchAnalyzer: "identifier_query"
            },
            basicNumber: {
              type: "string",
              analyzer: "basic_hierarchy",
              searchAnalyzer: "identifier_query"
            },
            title: {
              type: "string",
              analyzer: "normalized_text",
              multi: {
                stems: {
                  type: "string",
                  analyzer: "english_stems"
                }
              }
            },
            description: {
              type: "string",
              analyzer: "normalized_text"
            }
          }
        },
        analyzers: [
          {
            name: "part_prefix",
            tokenizer: {
              type: "edgeGram",
              minGram: 2,
              maxGram: 20
            },
            tokenFilters: [
              { type: "lowerCase" }
            ]
          },
          {
            name: "identifier_query",
            tokenizer: { type: "keyword" },
            tokenFilters: [
              { type: "lowerCase" }
            ]
          },
          {
            name: "basic_hierarchy",
            tokenizer: {
              type: "pathHierarchy",
              delimiter: "/"
            },
            tokenFilters: [
              { type: "lowerCase" }
            ]
          },
          {
            name: "normalized_text",
            tokenizer: { type: "whitespace" },
            tokenFilters: [
              { type: "lowerCase" },
              { type: "asciiFolding" }
            ]
          },
          {
            name: "english_stems",
            tokenizer: { type: "whitespace" },
            tokenFilters: [
              { type: "lowerCase" },
              { type: "stemmer", language: "english" }
            ]
          }
        ]
      }
    }
  ]
});

Confirm the index is ready

Index creation can continue after the command returns. Inspect the named index and wait until it reports status: "READY" and queryable: true before running the queries.

db.products.aggregate([
  { $listSearchIndexes: { name: "idx_products_analyzers" } }
]);

Run a prefix query

Prefix search matches the beginning of a field value. It's useful for part numbers, SKUs, and other identifiers that users enter incrementally.

The part_prefix analyzer generates prefixes at index time. The identifier_query analyzer preserves the query as one lowercased token.

db.products.aggregate([
  {
    $search: {
      index: "idx_products_analyzers",
      text: {
        query: "sku-shoe",
        path: "partNumber"
      }
    }
  },
  { $limit: 20 },
  {
    $project: {
      _id: 0,
      partNumber: 1,
      score: { $meta: "searchScore" }
    }
  }
]);

This query matches SKU-SHOE-001. Choose minGram and maxGram for the prefix lengths your application needs. Prefixes outside that range aren't generated by this analyzer. Generating more prefixes increases index size.

An edgeGram tokenizer generates prefixes from the input. An edgeGram token filter generates prefixes from each token produced by the tokenizer. Use the filter when you need to tokenize text into words before generating prefixes.

Prefix matching with edgeGram doesn't require the autocomplete operator. That operator isn't currently supported.

Run a hierarchical query

The pathHierarchy tokenizer generates tokens for successive levels of an identifier. With / as the delimiter, catalog/outdoor/running produces:

catalog
catalog/outdoor
catalog/outdoor/running

Use the query-time keyword tokenizer to search for a particular hierarchy level without expanding the query into its parent levels.

db.products.aggregate([
  {
    $search: {
      index: "idx_products_analyzers",
      text: {
        query: "catalog/outdoor",
        path: "basicNumber"
      }
    }
  },
  { $limit: 20 },
  {
    $project: {
      _id: 0,
      basicNumber: 1,
      score: { $meta: "searchScore" }
    }
  }
]);

This query matches catalog/outdoor/running and catalog/outdoor/hiking. Use the optional replacement, skip, and reverse settings to customize how hierarchical tokens are generated.

Search multiple identifier fields

One search index can cover multiple identifier fields. Pass an explicit array of mapped paths to text to search them together.

db.products.aggregate([
  {
    $search: {
      index: "idx_products_analyzers",
      text: {
        query: "sku-shoe",
        path: ["partNumber", "alternatePartNumber"],
        matchCriteria: "any"
      }
    }
  },
  { $limit: 20 },
  {
    $project: {
      _id: 0,
      partNumber: 1,
      alternatePartNumber: 1,
      score: { $meta: "searchScore" }
    }
  }
]);

This query matches the first document through partNumber and the second document through alternatePartNumber. Every path must be mapped in the selected index. If no single index covers all requested paths, the query returns an error.

Use multiple analyzers on the same field

A multi mapping indexes the same string field with an alternate analyzer. This feature supports different matching behaviors without storing duplicate document fields.

The example index defines a stems mapping on title. Select that mapping with path: { value, multi }:

db.products.aggregate([
  {
    $search: {
      index: "idx_products_analyzers",
      text: {
        query: "running",
        path: {
          value: "title",
          multi: "stems"
        }
      }
    }
  },
  { $limit: 20 },
  {
    $project: {
      _id: 0,
      title: 1,
      score: { $meta: "searchScore" }
    }
  }
]);

The alternate analyzer applies English stemming to indexed text and query text. A multi mapping differs from a multi-field index: multi provides several analyzer variants of one field, while a multi-field index covers several document fields.

Use language-specific analyzers

Azure DocumentDB provides built-in analyzers for Chinese, Japanese, Korean, and Thai. Reference these analyzers with the documentdb. prefix. Inside a custom pipeline, use the unprefixed tokenizer name.

Language Built-in analyzer Custom tokenizer type
Chinese documentdb.chinese chinese
Japanese documentdb.japanese japanese
Korean documentdb.korean korean
Thai documentdb.thai thai

Set a default analyzer

Set analyzer on the index definition to provide a default for mapped string fields. A field-level analyzer overrides that default. If you don't configure a searchAnalyzer, queries use the effective index-time analyzer for the field.

The following index sets documentdb.chinese as its default analyzer. The titleZh field inherits that default, while descriptionZh overrides it with a custom language-tokenizer pipeline:

db.runCommand({
  createSearchIndexes: "products",
  indexes: [
    {
      name: "idx_products_languages",
      definition: {
        analyzer: "documentdb.chinese",
        mappings: {
          dynamic: false,
          fields: {
            titleZh: {
              type: "string"
            },
            descriptionZh: {
              type: "string",
              analyzer: "chinese_normalized"
            }
          }
        },
        analyzers: [
          {
            name: "chinese_normalized",
            tokenizer: { type: "chinese" },
            tokenFilters: [
              { type: "lowerCase" }
            ]
          }
        ]
      }
    }
  ]
});

Current limitations

The following constraints apply to custom analyzers and their field mappings:

  • Setting dynamic: false or omitting dynamic uses static mappings. You must explicitly declare every searchable field under mappings.fields. Setting dynamic: true causes index creation to fail with mappings.dynamic is not supported in search indexes yet. Index creation also fails if mappings.fields is missing or empty.
  • Custom analyzer names must be unique within the index definition and no longer than 32 characters.
  • A field's analyzer settings must reference a supported built-in analyzer or a custom analyzer declared in the index definition.
  • Character filters (charFilters) and synonym definitions aren't supported.
  • The autocomplete and queryString operators aren't supported.
  • The regex and wildcard operators inside $search aren't supported.