Need help optimizing GPT-3.5 Turbo Chatbot Responses for Multilingual Document Retrieval

Question

Our company is planning to use GPT-3.5 Turbo to create our own custom chatbot that is supposed to help the users with their meditation journey. Currently our server is hosted by Microsoft, so we are using their OpenAI Studio to get things going faster. We have a document in Korean with basic information on meditation, like how to start, how often to practice meditation, etc. and we are using hybrid semantic search with text-embedding-ada-002 for vectorizing. When testing out the AI search, the information being retrieved is relevant enough. However the problem occurs when the data is being uploaded to the chat playground and the chat keeps responding with “This question is out of the scope of the retrieved documents. Therefore, I cannot provide an answer based on the documents.” message in English to a prompt written in Korean (same language as the uploaded data). I tried reflecting this in a system message, asking GPT to respond according to its own knowledge to questions for which no information was found in the uploaded document, however that did not work. At certain values of temperature and top P it gives out a relatively good answer, now considering results of the semantic search, but later with the same configurations for both temperature and top P and the exact same prompt, it again responds with “no relevant information was found”. I am not sure what to do with this issue, what exactly it is related to whether its language related or I have set some configurations wrong, so if you could please give any advice I would most certainly appreciate it. In the screenshots provided as you can see same parameters are being used, however one time it responded perfectly while in another case the chatbot apologizes for lack of information.

Answer 1

Hello Dana,

You may refer to this Microsoft Learn document to adjust the "Strictness" setting. Based on your situation, it is advisable to start by setting it to the lowest level and then gradually increase it to a level you find acceptable.

Strictness determines the system's aggressiveness in filtering search documents based on their similarity scores. Setting strictness to 5 indicates that the system will aggressively filter out documents, applying a very high similarity threshold. Semantic search can be helpful in this scenario because the ranking models do a better job of inferring the intent of the query. Lower levels of strictness produce more verbose answers, but might also include information that isn't in your index. This is set to 3 by default.

Best regards,
Charlie

---

If you find my response helpful, please consider accepting this answer and voting 'yes' to support the community. Thank you!

Answer 2

@Dana As discussed in the private chat, I have enabled a one-time, courtesy free support flag on your subscription ‘a1bXXX-XXX-XXX-XXXX-XXXXXcb84d’ for a quick and immediate assistance. Using this you can work with the Microsoft Support professional to activate your subscription. I have also emailed you the steps to create a support ticket.

Once you have a resolution for your issue, please post the fix in this thread, so that it benefits the community audience who encounter the same issue.

**

Please do not forget to "Accept the answer” and “up-vote” if this helps.