Grounding score for RAG

Question

Grounding score for RAG

Afshin Oroojlooy 0

I was wondering if I can get the grounding score for the generated answer when using the RAG system?

I see that it is possible to get the groundedness score for a certain dataset through the online console:

https://learn.microsoft.com/en-us/azure/ai-studio/how-to/evaluate-generative-ai-app?pivots=ai-studio

But, that is not a viable approach for an online service where I want to check the groundedness score before passing the generated answer to the user.

AshokPeddakotla-MSFT 36,021 Reputation points Moderator

2024-03-21T06:42:01.4333333+00:00

Afshin Oroojlooy Greetings & Welcome to Microsoft Q&A forum!

Did you check this doc Metrics for multi-turn or single-turn chat with retrieval augmentation (RAG) already?

you can also try exploring other pre-built models in Azure Cognitive Services that may better suit your needs.

For example, the QnA Maker service in Azure Cognitive Services provides a way to create a knowledge base of question and answer pairs. The knowledge base can be queried using natural language, and the service returns the best answer from the knowledge base. The QnA Maker service also provides a confidence score for each answer, which can be used as a measure of how well the answer is supported by the evidence in the input text.

See the documentation custom question answering for more details.

Do let me know if that helps or have any further queries.
Afshin Oroojlooy 0 Reputation points

2024-03-21T18:23:28.64+00:00

Hi AshokPeddakotla-MSFT,

Yes, I have seen that page.

As I mentioned, I want to get the grounding score after each generation. I also can check the confidence level that you mentioned, but for implementing the ReAct method, I need to know the grounding score.

Is there any way to call the corresponding API from python?