Getting additional records when filter applied in Azure search

Question

Getting additional records when filter applied in Azure search

Mukesh Kumar 0

Hi Team,

I am getting additional records when applying filter in Azure Cognitive Search. Please find the details below

Data source json file

[
    {
        "Emissions": "US EPA (Certified) Stationary Emergency|US EPA (Certified) Stationary Non-Emergency",
        "AzureSearch_DocumentKey": "aHR0cHM6Ly9waW1jb3Ntb3NmZWVkdGVzdC5ibG9iLmNvcmUud2luZG93cy5uZXQvYmxvb21yZWFjaC9pbmR1c3RyaWFsL0luZHVzdHJpYWwuanNvbjszNA2",
        "ProductProductNo": "250REZXB_60_DD"
    },
    {
        "Emissions": "US EPA (Certified) Stationary Emergency|US EPA (Certified) Stationary Non-Emergency",
        "AzureSearch_DocumentKey": "aHR0cHM6Ly9waW1jb3Ntb3NmZWVkdGVzdC5ibG9iLmNvcmUud2luZG93cy5uZXQvYmxvb21yZWFjaC9pbmR1c3RyaWFsL0luZHVzdHJpYWwuanNvbjszNA2",
        "ProductProductNo": "250REZXB_60_DF"
    },
    {
        "Emissions": "US EPA (Certified) Stationary and Mobile Emergency|US EPA (Certified) Stationary and Mobile Non-Emergency",
        "AzureSearch_DocumentKey": "aHR0cHM6Ly9waW1jb3Ntb3NmZWVkdGVzdC5ibG9iLmNvcmUud2luZG93cy5uZXQvYmxvb21yZWFjaC9pbmR1c3RyaWFsL0luZHVzdHJpYWwuanNvbjszNA2",
        "ProductProductNo": "250REZXB_60_EE"
    }
]

Request Body

{
    "count": true
    ,"top": 3000
    ,"skip": 0
    ,"search": "*"
    ,"orderby": ""
    ,"facets":["Emissions,count:100,sort:-count"]
    ,"filter": "search.ismatch('US EPA (Certified) Stationary Non-Emergency', 'Emissions', 'full','all')"
}

Response

{
    "@odata.context": "",
    "@odata.count": 86,
    "@search.facets": {
        "Emissions": [
            {
                "count": 3,
                "value": "US EPA (Certified) Stationary Non-Emergency, Prime, and Continuous"
            },
            {
                "count": 9,
                "value": "US EPA (Certified) Stationary and Mobile Emergency|US EPA (Certified) Stationary and Mobile Non-Emergency"
            },
            {
                "count": 10,
                "value": "US EPA (Certified) Stationary Emergency|US EPA (Certified) Stationary Non-Emergency"
            }
        ]
    },
"value":[]
}

From above response facet contains in first and third records but not sure why

"US EPA (Certified) Stationary and Mobile Emergency|US EPA (Certified) Stationary and Mobile Non-Emergency"

is returned in response. i have added analyzer as below still not able to get exact record

"analyzers": [
        {
            "@odata.type": "#Microsoft.Azure.Search.CustomAnalyzer",
            "name": "standardCmAnalyzer",
            "tokenizer": "standard_v2",
            "tokenFilters": [
                "lowercase",
                "asciifolding"
            ],
            "charFilters": []
        },
        {
            "@odata.type": "#Microsoft.Azure.Search.CustomAnalyzer",
            "name": "prefixCmAnalyzer",
            "tokenizer": "EdgeNGramTokenizer",
            "tokenFilters": [
                "lowercase",
                "asciifolding",
                "edgeNGramCmTokenFilter"
            ],
            "charFilters": []
        }
    ],
    "tokenizers": [
        {
            "@odata.type": "#Microsoft.Azure.Search.EdgeNGramTokenizer",
            "name": "EdgeNGramTokenizer",
            "minGram": 9,
            "maxGram": 40
        }
    ],
    "tokenFilters": [
        {
            "@odata.type": "#Microsoft.Azure.Search.EdgeNGramTokenFilterV2",
            "name": "edgeNGramCmTokenFilter",
            "minGram": 9,
            "maxGram": 40,
            "side": "front"
        }
    ],

Can you please help me om this

1 answer

Your answer

Answer 1

Deepanshukatara-6769 16,565 Moderator

Hi , hope doing good!

Based on your provided information, it seems to be related to how the data is indexed and how the filter is applied. Here are some suggestions to help you resolve the issue:

Analyzer Configuration: It's important to ensure that your analyzer configuration aligns with your search requirements. In your case, you're using standardCmAnalyzer and prefixCmAnalyzer, which involve tokenization, lowercase conversion, and ASCII folding. Make sure these analyzers are applied consistently during indexing and querying.
Index Field Configuration: Verify that the "Emissions" field in your index is using the correct analyzer. Ensure that it uses the same analyzer (e.g., standardCmAnalyzer or prefixCmAnalyzer) for indexing and searching to achieve the desired results. If you've recently updated your analyzer, reindexing the data might be necessary.
Search Query: Your filter query seems to be searching for an exact match using search.ismatch. If you want an exact match, you might want to use search.ismatch without the 'full' and 'all' options.

Example filter query:


"filter": "search.ismatch('US EPA (Certified) Stationary Non-Emergency', 'Emissions')"

Make sure the text you're searching for ('US EPA (Certified) Stationary Non-Emergency') exactly matches the indexed data.

4.Check for Special Characters: Sometimes, special characters or encoding issues can affect search results. Ensure that there are no hidden characters or encoding differences between your filter query and the indexed data.

5.Reindex Data: If you've made changes to your analyzer configuration or suspect data indexing issues, consider reindexing your data to ensure the new analyzer settings are applied correctly.

Please accept answer , if it helps , Thankyou!|

Mukesh Kumar 0

Hi Deepanshu,

Thanks for your reply.

I already tried those things in Azure Search.

Please check my index settings

{
            "name": "Emissions",
            "type": "Edm.String",
            "searchable": true,
            "filterable": true,
            "retrievable": true,
            "sortable": false,
            "facetable": true,
            "key": false,
            "indexAnalyzer": "prefixCmAnalyzer",
            "searchAnalyzer": "prefixCmAnalyzer",
            "analyzer": null,
            "synonymMaps": []
        },

Here i am considering prefixCmAnalyzer as analyzer for field.

And i removed "full" and "all" and now i am not getting any results for that.

I have added one mode field in filter

"filter": "search.ismatch('US EPA (Certified) Stationary Non-Emergency', 'Emissions') or search.ismatch('Non-Certified', 'Emissions')"

But for this i can able to get Non-Certified please check the response below

 "@search.facets": {
        "Emissions": [
            {
                "count": 64,
                "value": "Non-Certified"
            }
        ]
    },

i have recreated the complete index but still not getting proper result

Deepanshukatara-6769 16,565 Reputation points Moderator

2023-09-13T13:02:07.6033333+00:00
It seems some issue with filter

To review and test your filter logic in Azure Search, you can use the Azure Portal or the Azure Search REST API to perform test queries with your filter expressions. Here's a step-by-step guide on how to do this:

Azure Portal:

a. Log in to the Azure Portal (https://portal.azure.com).

b. Navigate to your Azure Search service.

c. In the left-hand menu, under "Settings," select "Search explorer."

d. In the "Search explorer" panel, you can construct and test your search queries with filter expressions.

e. In the "Search" box, you can enter your search query, including the filter expression. For example:

javaCopy code search=ismatch('US EPA (Certified) Stationary Non-Emergency', 'Emissions') or ismatch('Non-Certified', 'Emissions')

f. Click the "Search" button to execute the query.

g. Review the results to see if they match your expectations.

Share via

Getting additional records when filter applied in Azure search

1 answer

Your answer