Azure AI Search token filter equivalent to Multiplexer in ElasticSearch

Jay Lin 20 Reputation points
2024-05-30T06:39:04.2966667+00:00

Hi there,

Do we have the same token filter "Multiplexer" as ElasticSearch?

A token filter of type multiplexer will emit multiple tokens at the same position

https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-multiplexer-tokenfilter.html

Thanks.

Azure AI Search
Azure AI Search
An Azure search service with built-in artificial intelligence capabilities that enrich information to help identify and explore relevant content at scale.
809 questions
0 comments No comments
{count} votes

Accepted answer
  1. SnehaAgrawal-MSFT 19,511 Reputation points
    2024-06-06T17:02:31.1+00:00

    @Jay Lin Thanks for reply! Assuming the content is in a specific language, you could use the Microsoft analyzers (e.g. en.microsoft), which normalize numbers and dates.

    That would handle the thousand separator case, trailing zeroes after the decimal separator, and other variations that don't alter the number.

    From your list of examples, the one that wouldn't be addressed is 1234.56 -> 1234 because that's a lossy conversion.

    Check this document- Add language analyzers to string fields - Azure AI Search | Microsoft Learn. Using the /analyze endpoint in indexes is a great way to experiment and see how analyzers handle and tokenize different inputs.

    Hope this helps. Let me know


1 additional answer

Sort by: Most helpful
  1. SnehaAgrawal-MSFT 19,511 Reputation points
    2024-06-04T05:15:20.7766667+00:00

    @Jay Lin Thanks for asking question!

    Could you elaborate on if any specific scenario you are looking for, or are you simply looking for an equivalent feature?

    In Azure Search we don't have a like to like token filter- here are the token filters available in Azure Search. I didn't see any multiplexer among them: https://learn.microsoft.com/en-us/azure/search/index-add-custom-analyzers#token-filters.

    Please let us know, happy to assist further.