How to set up "Limit responses to your data content" on json file?

Question

How to set up "Limit responses to your data content" on json file?

N.SAITO 70

I'm developing chatbot using Azure openAI gpt-35-turbo-16k.

I could use Azure cognitive search with Azure openAI using json file.

But I could not find how to set up "Limit responses to your data content" and on json file.

LimitResponsesOnPlayground

Could you teach me how to setup it on json file?

AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2023-07-10T15:19:39.5033333+00:00

N.SAITO Greetings!

But I could not find how to set up "Limit responses to your data content" and on json file.

I understand that you are looking to setup the same in JSON File.

I'm looking into this issue and update you with my findings.

Appreciate your time and patience.
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2023-07-14T10:34:08.52+00:00

N.SAITO To increase the limit of the input text, you can change the max_tokens parameter in your code. By increasing this value, you can increase the maximum length of the input text.

For example, if you want to increase the maximum length of the input text to 2000 characters, you can set the max_tokens parameter to a value that corresponds to 2000 tokens. The number of tokens required to represent a given text varies depending on the model being used.

The gpt-4 model supports 8192 max input tokens and the gpt-4-32k model supports up to 32,768 tokens.

The gpt-35-turbo model supports 4096 max input tokens and the gpt-35-turbo-16k model supports up to 16,384 tokens.

gpt-35-turbo and gpt-35-turbo-16k share the same quota.

I hope this helps!
N.SAITO 70 Reputation points

2023-07-18T08:45:19.4733333+00:00

Thank you for your response.
But it is not that I want to know. I already know it.

I think "Limit responses to your data content" means "GPT use only the data in Azure Cognitive Search result."
Dairton Lopes Martins Filho 0 Reputation points

2023-08-02T01:13:09.66+00:00

Hi @N.SAITO

Were you able to make it work by writing this sentence in the message content?

thanks!
Yash Shukla 40 Reputation points

2023-08-23T11:16:42.93+00:00

Any updates on this concern? I am facing a similar issue.
Mohamad Bahri 180 Reputation points

2023-08-23T13:37:27.97+00:00

I am facing the same issue, but I don't just want the json format of it, I would like to have the python code that configures this option so that openai's model is only utilized for its language understanding capabilities and the result is just limited to the search results of a cognitive search service.
There should be a parameter in the ChatCompletion function that just limits the reply to the documents content.
Manu 0 Reputation points

2023-10-17T10:18:18.19+00:00

here you have the solution
https://learn.microsoft.com/en-us/answers/questions/1366572/im-using-rest-api-how-can-i-disable-the-limit-resp
parameter "inScope": False

Accepted answer

0 additional answers

Your answer

AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2023-07-10T15:19:39.5033333+00:00

N.SAITO Greetings!

But I could not find how to set up "Limit responses to your data content" and on json file.

I understand that you are looking to setup the same in JSON File.

I'm looking into this issue and update you with my findings.

Appreciate your time and patience.
AshokPeddakotla-MSFT 35,971 Reputation points Moderator

2023-07-14T10:34:08.52+00:00

N.SAITO To increase the limit of the input text, you can change the max_tokens parameter in your code. By increasing this value, you can increase the maximum length of the input text.

For example, if you want to increase the maximum length of the input text to 2000 characters, you can set the max_tokens parameter to a value that corresponds to 2000 tokens. The number of tokens required to represent a given text varies depending on the model being used.

The gpt-4 model supports 8192 max input tokens and the gpt-4-32k model supports up to 32,768 tokens.

The gpt-35-turbo model supports 4096 max input tokens and the gpt-35-turbo-16k model supports up to 16,384 tokens.

gpt-35-turbo and gpt-35-turbo-16k share the same quota.

I hope this helps!
N.SAITO 70 Reputation points

2023-07-18T08:45:19.4733333+00:00

Thank you for your response.
But it is not that I want to know. I already know it.

I think "Limit responses to your data content" means "GPT use only the data in Azure Cognitive Search result."
Dairton Lopes Martins Filho 0 Reputation points

2023-08-02T01:13:09.66+00:00

Hi @N.SAITO

Were you able to make it work by writing this sentence in the message content?

thanks!
Yash Shukla 40 Reputation points

2023-08-23T11:16:42.93+00:00

Any updates on this concern? I am facing a similar issue.
Mohamad Bahri 180 Reputation points

2023-08-23T13:37:27.97+00:00

I am facing the same issue, but I don't just want the json format of it, I would like to have the python code that configures this option so that openai's model is only utilized for its language understanding capabilities and the result is just limited to the search results of a cognitive search service.
There should be a parameter in the ChatCompletion function that just limits the reply to the documents content.
Manu 0 Reputation points

2023-10-17T10:18:18.19+00:00

here you have the solution
https://learn.microsoft.com/en-us/answers/questions/1366572/im-using-rest-api-how-can-i-disable-the-limit-resp
parameter "inScope": False

Answer 1

Mohamad Bahri 180

Alright so from what I have searched, there isn't any native configuration that can solve this issue, regardless of whether you are using azure openai or openai itself.
The best possible way that I found in order to limit the response is to either use a binary classifier (detect if a prompt is on-topic or off-topic) or to use a a similarity detector between whatever you are getting from the search results and the answer. I used a similarity detector that returns a score between 0 and 1 based on the similarity between the search results and the prompt, set up a threshold and if the search results fit the user prompt, the score will be high and it'll be on-topic.

I really hope that they add some sort of configuration setup but I think it'll be challenging as it is very difficult to limit the model's response, and even in the playground the sometimes hallucinates.

N.SAITO 70 Reputation points

2023-08-30T22:51:35.1433333+00:00

Thank you for your answer.
Mohamad Bahri 180 Reputation points

2023-09-04T06:13:00.4166667+00:00

A little update, in the sample-app-aoai-chatgpt github repo, they are using a different way to call the azure openai api, in app.py the function prepare_body_headers_with_data they prepare the body of the call and they use the dataSources property with the inScope property inside it, I found that this does limit the responses to a great extent but it still sometimes hallucinates and generates answers from the model's data. I think that combining that function with another similarity_detctor or on/off_topic classification model could lead to the best possible results.
this is the repo: https://github.com/microsoft/sample-app-aoai-chatGPT
its actually the same repo that you get when you add your data in the azure portal and then deploy to a web app resource.

Still, there should be a more configurable method that 100% guarantees that the generated data is only from the sources.

Share via

How to set up "Limit responses to your data content" on json file?

0 additional answers

Your answer