Halcination measures and measures to improve response accuracy

Question

Learned the product manual pdf with Add your data.

With a description of the a function, how to operate it, and learning the location of the a function menu.

I asked the question, "How do I enter the a-function?"

I asked the question, "How do I enter the a function?

The answer was "a function can be entered and operated from the b category of the menu".

The answer was "No, it is not,

When I typed "No, it's not...", it replied

If you correct the answer to the question "I made a mistake in my previous answer, the a function can be entered and operated from the a category.

The correct answer will be returned.

Please try another query or topic.

GPT 3.5turbo is used as the model.

Keyword search is used for retrieval.

The training data is 40 pages of product manuals, divided into 1 page each, and imported into FR (because FR is free).

The systemMessage says "You are an AI assistant to help you find information. You must retrieve Japanese documents, read them carefully in Japanese, and answer in Japanese. If you don't understand, say you don't understand. and prompts.

Answer 1

@ryota.matsuda After reading through my previous answer, I felt there was an opportunity to provide you with a more comprehensive answer. I will keep the other answer up, but I feel the below information that I wrote up will be of more help.

My understanding of your ask is that you are looking to improve the accuracy of your responses. If so, below are some steps that I put together to help you.

1. Provide quality data: Make sure that the data you are using to train your model is accurate and relevant. If you're trying to build a classifier or get the model to follow a pattern, make sure that there are enough examples. Be sure to proofread your examples. The model is usually smart enough to see through basic spelling mistakes and give you a response, but it also might assume that the mistakes are intentional, and it can affect the response.
2. Check your settings: The temperature and top_p settings control how deterministic the model is in generating a response. If you're asking it for a response where there's only one right answer, then you'd want to set these settings to lower values. If you're looking for a response that's not obvious, then you might want to set them to higher values. The number one mistake people use with these settings is assuming that they're "cleverness" or "creativity" controls.
3. Use a diverse set of prompts: To improve the quality of the responses, you can make a longer more diverse list in your prompt. One way to do that is to start off with one example, let the API generate more and select the ones that you like best and add them to the list. A few more high-quality variations can dramatically improve the quality of the responses.
4. Test and Improve: Start testing your model with other utterances to get a sense of how the app behaves, and you can decide if any improvement is needed. You can improve your application by following best practices.
5. Make use of synonyms: These have the ability to help you with your answer accuracy.

Brief Tutorial on how to use synonyms to improve your accuracy:

1. Create a Synonym Map: First, you need to create a Synonym Map. A Synonym Map is a collection of synonyms that you want to use in your search queries. You can create a Synonym Map using the Azure portal, Azure CLI, or Azure PowerShell.
2. Add Synonyms to the Synonym Map: Once you have created a Synonym Map, you can add synonyms to it. You can add synonyms using the Azure portal, Azure CLI, or Azure PowerShell. You can add synonyms in two ways: by uploading a file that contains the synonyms or by adding them manually.
3. Associate the Synonym Map with an Index: After you have created a Synonym Map and added synonyms to it, you need to associate the Synonym Map with an Index. You can associate a Synonym Map with an Index using the Azure portal, Azure CLI, or Azure PowerShell.
4. Test the Synonyms: Once you have associated the Synonym Map with an Index, you can test the synonyms by running search queries that use the synonyms. You can test the synonyms using the Azure portal, Azure CLI, or Azure PowerShell.

More information: https://docs.microsoft.com/en-us/azure/search/search-synonyms-tutorial-sdk

Answer 2

@ryota.matsuda To improve the accuracy of your responses, you can consider the following measures:

1. Add synonyms: By adding synonyms to your keywords, you can improve the chances of matching user queries with the correct response. This can be done using the Authoring API, as explained in this tutorial.
2. Choose an appropriate confidence score threshold: The confidence score is a measure of how confident the system is in its response. By choosing an appropriate threshold, you can balance the accuracy and coverage of your responses.
3. Improve your training data: The quality of your training data can have a significant impact on the accuracy of your responses. Make sure your training data is relevant, up-to-date, and covers a wide range of possible user queries.
4. Use feedback to improve your responses: Collect feedback from users and use it to improve your responses. This can help you identify areas where your responses are inaccurate or incomplete.