Relevance score horribly low using @azure/search vs using search explorer in Portal

Question

Relevance score horribly low using @azure/search vs using search explorer in Portal

dude95 40

I'm getting totally different results with searching using the search explorer in the Portal and in my bot. Here is the portal search explorer query

{
    "search": "SuccessFactors 2025 date",
    "count": true,
    "queryType": "semantic",
    "semanticConfiguration": "default", // <<< USE YOUR ACTUAL CONFIG NAME
    "queryLanguage": "en-us", 
    "captions": "extractive", 
    "answers": "extractive|count-3", 
    "select": "docId,docTitle,description,filename,fileUrl,lastmodified,chunkindex,totalChuncks", 
    "top": 10 
    // Vector search is implicitly included and configured via the semanticConfiguration
}

Here is partial results

{
  "@odata.context": "https://zyptai-s11-search.search.windows.net/indexes('sharepointblobindx')/$metadata#docs(*)",
  "@odata.count": 63,
  "@search.answers": [
    {
      "key": "01ICKOLJXIBDIQTU2WJVEJ275LQA6B7OTA-44",
      "text": "It's got ... Yeah, the the goal would be to stand up everybody at S...",
      "score": 0.9900000095367432
    },
    {
      "key": "01ICKOLJR343RCVRR7WJGL7AGOZYBCZXHX-1",
      "text": "It's got ... Yeah, the the goal would be to stand up everybody at SuccessFactors ...",
      "score": 0.9850000143051147
    },

Code from my webbot

      const searchOptions: SearchOptions<SearchDocument> = {
        // Select fields based on what's needed and what portal test selected
        select: [
          "description",
          "chunkindex",
          "filename",
          "fileUrl",
          "docId",
          "docTitle",
          "lastmodified",
          "totalChuncks"
        ],
        top: MAX_RESULTS,
        // Enable Semantic Reranking features
        queryType: "semantic" as const,
        // Use the semantic search options object:
        semanticSearchOptions: {
          configurationName: "my-semantic-config",
          answers: { 
            answerType: "extractive", 
            count: 3
          },
          captions: { 
            captionType: "extractive", 
            highlight: true 
          }
        },
        // Vector search configuration
        vectorSearchOptions: {
          queries: [{
            kind: "vector" as const,
            fields: ["descriptionVector"], // Field containing the vector embeddings
            kNearestNeighborsCount: K_NEAREST_NEIGHBORS, // How many vector results to initially find
            vector: queryVector, // The query vector itself
          }]
        },
        searchMode: "any" as const, // Use 'any' for OR logic on keywords
        includeTotalCount: true // Request the total count of results
      };

The results are 10x worse - can't really recognize anything of value:
[2025-04-15T13:09:20.037Z] INFO: [1] File: Fake meeting template - just a little discussion.docx, RerankerScore: N/A, HybridScore: 0.0315, Chunk: 2

[2025-04-15T13:09:20.044Z] INFO: [2] File: Phase 2 Planning Summit Day 1 (1) (1) (1).pdf, RerankerScore: N/A, HybridScore: 0.0308, Chunk: 54

[2025-04-15T13:09:20.045Z] INFO: [3] File: Phase 2 Planning Summit Day 1 (1) (1) (1).pdf, RerankerScore: N/A, HybridScore: 0.0255, Chunk: 56

I feel like the frontend code should replicated the Portal call. Even ChatGPT thinks they are the same. Anyone have an idea of what went wrong?

dude95 40 Reputation points

2025-04-15T20:57:24.5+00:00
Great comments - default was just an original cut and paste from my ai. The code and index are actually my-semantic-config.

RerankerScore: N/A is a red flag - I'm coming to the same point. I removed the vector search and now am getting relevance scores almost at 1. Easiest way is to give you the back and forth with my ai.

Okay, thanks for sharing the logs from the isolation test (commenting out vectorSearchOptions).

Analysis of Test Results:

Search Executed: The logs confirm the search was executed with queryType: "semantic" but without vectorSearchOptions.

Reranker Score Still Missing: Critically, the result is the same as before: [4f466452][performHybridSearch] Search completed {"totalResultsFetched":50,"topResultScore":"11.4585","topResultRerankerScore":"N/A"}. The reranker score is still "N/A".

Hybrid Score High: For this specific query ("2025 SuccessFactors date"), the standard hybrid score (topResultScore) is very high (11.4585).

Relevance Check Passes (via Hybrid): Because the reranker score is N/A, ragOptimizer falls back to the hybrid score. Since 11.4585 is much greater than the threshold (currently 1.0 in the code, although the second log set shows 0.4), the relevance check passes ([1b9c241f] Evaluating relevance: Highest Score (Reranker Preferred)=11.4585, Threshold=1... -> isRelevant:true).

Correct RAG Answer: Consequently, the RAG context is used, and the user gets a relevant answer based on company data.

Why did it work this time but not initially?

The initial problem occurred because for queries like "Phase 2 Planning Summit data migration", the hybrid score was likely very low (e.g., the 0.01x originally observed), and the SDK wasn't returning the high reranker score. The low hybrid score failed the relevance threshold.

In this test with the simpler "2025 SuccessFactors date" query, the hybrid score just happened to be extremely high (11.45), so it passed the threshold even without the reranker score being available. The same happened for the "tell me about the phase 2 planning summit..." query in the second log set (hybrid score 20.82 > threshold 0.4).

Final Confirmed Diagnosis:

The isolation test definitively confirms the root cause:

The @azure/search-documents@12.1.0 SDK, in your specific setup, is not returning the @search.rerankerScore field when queryType: "semantic" is used, regardless of whether vector search is active.

Since the Azure Portal does return the score, the problem lies between the SDK call and the data parsing/return within the SDK library itself.Okay, thanks for sharing the logs from the isolation test (commenting out vectorSearchOptions).

Analysis of Test Results:

Search Executed: The logs confirm the search was executed with queryType: "semantic" but without vectorSearchOptions.

Reranker Score Still Missing: Critically, the result is the same as before: [4f466452][performHybridSearch] Search completed {"totalResultsFetched":50,"topResultScore":"11.4585","topResultRerankerScore":"N/A"}. The reranker score is still "N/A".

Hybrid Score High: For this specific query ("2025 SuccessFactors date"), the standard hybrid score (topResultScore) is very high (11.4585).

Relevance Check Passes (via Hybrid): Because the reranker score is N/A, ragOptimizer falls back to the hybrid score. Since 11.4585 is much greater than the threshold (currently 1.0 in the code, although the second log set shows 0.4), the relevance check passes ([1b9c241f] Evaluating relevance: Highest Score (Reranker Preferred)=11.4585, Threshold=1... -> isRelevant:true).

Correct RAG Answer: Consequently, the RAG context is used, and the user gets a relevant answer based on company data.

Why did it work this time but not initially?

The initial problem occurred because for queries like "Phase 2 Planning Summit data migration", the hybrid score was likely very low (e.g., the 0.01x originally observed), and the SDK wasn't returning the high reranker score. The low hybrid score failed the relevance threshold.

In this test with the simpler "2025 SuccessFactors date" query, the hybrid score just happened to be extremely high (11.45), so it passed the threshold even without the reranker score being available. The same happened for the "tell me about the phase 2 planning summit..." query in the second log set (hybrid score 20.82 > threshold 0.4).

Final Confirmed Diagnosis:

The isolation test definitively confirms the root cause:

The

Since the Azure Portal does return the score, the problem lies between the SDK call and the data parsing/return within the SDK library itself.
Laxman Reddy Revuri 5,475 Reputation points Microsoft External Staff Moderator

2025-04-17T23:19:10.5966667+00:00

Hi dude95,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
dude95 40 Reputation points

2025-04-18T01:30:15.88+00:00

Prabhavathi Manchala did answer the question - I'm looking all over to mark that as the answer but there is no button?

Answer accepted by question author

1 additional answer

Your answer

Laxman Reddy Revuri 5,475 Reputation points Microsoft External Staff Moderator

2025-04-17T23:19:10.5966667+00:00

Hi dude95,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
dude95 40 Reputation points

2025-04-18T01:30:15.88+00:00

Prabhavathi Manchala did answer the question - I'm looking all over to mark that as the answer but there is no button?

Answer 1

Hi dude95,

You are using the @azure/search-documents SDK with semantic search, but the reranker score (@search.rerankerScore) always shows as N/A in your bot or web client, even after disabling vector search. This causes the system to rely on the hybrid score, which works sometimes but misses relevant answers in complex queries.

The @azure/search-documents SDK (v12.1.0) has a known issue where it doesn't always return the @search.rerankerScore, even with correct semantic search setup. However, the Azure Portal Search Explorer works fine and returns the reranker score since it fully supports the semantic pipeline.

Use the REST API directly for testing reranker scores. This avoids SDK issues and shows raw results, including fields like @search.rerankerScore, @search.answers, and @search.captions.
https://learn.microsoft.com/en-us/azure/search/semantic-how-to-query-request?tabs=portal-query#set-up-the-query
If you prefer using the SDK, check if a newer version of @azure/search-documents (after v12.1.0) fixes the issue and correctly returns the @search.rerankerScore.
As a workaround, keep using the hybrid score fallback with a smart threshold, but note it's less accurate than semantic reranking.

Kindly refer to this document to understand how ranking is scored.

https://learn.microsoft.com/en-us/azure/search/semantic-search-overview#how-ranking-is-scored

Please accept as "Yes" if the answer provided is useful, so that you can help others in the community looking for remediation for similar issues.

Let me know if you have any further Queries.

Anonymous

2025-04-21T19:08:08.9866667+00:00

Hi dude95,

Please accept as "Yes" if the answer provided is useful, so that you can help others in the community looking for remediation for similar issues.

Answer 2

Hi there dude95

Thanks for using QandA platform

i see In the Portal you’re using "default", but in code it’s "my-semantic-config". Make sure both point to the same setup. the Portal likely runs hybrid automatically. In your code, adding vectorSearchOptions manually might bypass reranking. Try removing it and test again.

Your logs show RerankerScore: N/A. That’s a red flag—semantic reranking probably isn’t even running. Could be field mismatches or config issues.

TL;DR: Try removing vector search, match the semantic config name exactly, and check if reranker kicks in]

If this helps kindly accept the answer thanks much.

Share via

Relevance score horribly low using @azure/search vs using search explorer in Portal

1 additional answer

Your answer