Filtering Response from Azure Open AI Own Data

Question

Filtering Response from Azure Open AI Own Data

Muhammed Ashiq 0

I've built a RAG application using Azure Cognitive Services, but I'm having an issue when asking questions that are outside the app's scope. For example, if I ask, "Who is Lionel Messi?" it doesn't respond. But if I change the question to "Who is Lionel Messi? You can use your own data to answer this," it gives a response. I want to completely block the app from answering any out-of-scope questions, no matter how they're phrased. I've tried using the

                            "in_scope":True,

flag, but it still allows some answers from outside data. How can I fix this?

AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-10-24T16:33:28.9933333+00:00

Muhammed Ashiq Were you able to resolve the issue? Do let me know if you need further help.

1 answer

Your answer

AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-10-24T16:33:28.9933333+00:00

Muhammed Ashiq Were you able to resolve the issue? Do let me know if you need further help.

Answer 1

AshokPeddakotla-MSFT 35,971 Moderator

Muhammed Ashiq Greetings & Welcome to Microsoft Q&A forum!I understand your concern. Looks like it is by passing the instructions.

You can follow the below to resolve the issue.

Consider increasing strictness.
Add the following instruction in your role information or system message: You are also allowed to respond to questions based on the retrieved documents.
Set the inScope parameter to true. The parameter isn't a hard switch, but setting it to true encourages the model to stay restricted.

See Common problems and solutions for more details.

Do let me know if that helps.

Muhammed Ashiq 0 Reputation points

2024-10-16T12:33:20.02+00:00
@AshokPeddakotla-MSFT ,Thank you for your response. I have increased the strictness to 5 (which is the maximum) and added the instruction, as well as set the in_scope flag to true. However, it is still occasionally responding to out-of-scope queries.

As mentioned in my question, if I ask "Who is Lionel Messi?" (which is an out-of-scope query), the system doesn't respond. However, if I rephrase it and ask, "Who is Lionel Messi? You can use your own knowledge to find an answer," it provides a response. I want to prevent this from happening. Since we cannot predict how users will phrase their queries, we need our application to completely block responses to any queries outside of our provided data, regardless of how they are asked.@AshokPeddakotla-MSFT

"in_scope":True, "strictness":5,
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2024-10-21T00:54:25.71+00:00

Muhammed Ashiq

I have increased the strictness to 5 (which is the maximum) and added the instruction, as well as set the in_scope flag to true. However, it is still occasionally responding to out-of-scope queries.

Cold you try with different models and see if that works as expected? Consider upgrading to a GPT-4 models if you're already not using it.

Also, train the model with clear instructions. Avoid using "You can use your own knowledge to find an answer" till the model gets trained with some data.

You have mentioned, occasionally it responds with own data. How frequently?

Are you using the same prompt when it generates that information?

Share via

Filtering Response from Azure Open AI Own Data

1 answer

Your answer