Azure Gpt35Turbo replies very differently from OpenAI Gpt3.5 turbo

Question

Azure Gpt35Turbo replies very differently from OpenAI Gpt3.5 turbo

Sora Teichman 0

Hi,

We are using AzureOpenAI with langchain to generate SQL queries and run them against our database to answer user queries. I noticed that when I use the AzureChatOpenAI llm, it often returns far inferior responses than the OpenAI llm. Here's an example: I ran the same query: How many sales orders were placed in November 2023?

OpenAI was able to formulate the correct sql and returned the correct response.

...

Thought: I should query the SalesOrders table to get the number of sales orders created in November 2023.

Action: sql_db_query

Action Input: SELECT COUNT(*) FROM SalesOrders WHERE CreatedOn BETWEEN '2023-11-01' AND '2023-11-30'

Observation: [(17,)]

Thought: I now know the final answer.

Final Answer: 17 sales orders were created in November 2023.

However when I use AzureChatOpenAI, it puts backticks before and after any query that it formats into more than one line, causing syntax errors when it tries to run it in sql. It does not listen to my custom instructions to leave out backticks.

I've tried it with different API versions (2023-05-01) but it does not help.

See the 2 versions of the code here:

toolkit = SQLDatabaseToolkit(db=db, llm=OpenAI(temperature=0, openai_api_key="sk-xxx"))


toolkit = SQLDatabaseToolkit(db=db, llm=AzureChatOpenAI(temperature=0, deployment_name="xx", openai_api_version="2023-09-01-preview", openai_api_type="azure", openai_api_base="https://xxx.openai.azure.com/"))

Saurabh Sharma 23,851 Reputation points Microsoft Employee Moderator

2023-12-07T00:44:38.3466667+00:00

Hi @Sora Teichman

Welcome to Microsoft Q&A! Thanks for posting the question.

Can you please try to modify your input prompt to specify that the output should not add any backticks, as well as provide examples of correct output your input. The more or better examples you give in your prompt the better. Please refer to Few shot prompting and Few shot prompting | Learn Prompting: Your Guide to Communicating with AI . .

Please let me know how it goes.

Thanks

Saurabh

Sora Teichman 0

Thanks, where is the best place to add this? I am currently using the prefix field in the create_sql_agent method but perhaps there is a better option.

agent_executor = create_sql_agent(
    llm=AzureChatOpenAI(temperature=0, model_name="gpt-35-turbo", openai_api_type="azure", openai_api_base="https://xxx.openai.azure.com/", openai_api_version="2023-05-15", deployment_name="xxx"),
    toolkit=toolkit,
    verbose=True,
    agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
    prefix=
    """Given an input question, first create a syntactically correct MSSQL query to run, then look at the results of the query and return the answer.
    Alias the tables in the query to avoid ambiguity. The Model Number is stored in the Number column of the Models table. The Purchase order line table
    productmodelid column references the modelid column of models. Any table that has a foreign key to the parties id has its name in the parties table.
    Search for names using like. If you get an error while executing a query, rewrite the query and try again.
    When asked about sales or cost, make sure to multiply rate by quantity.
    DO NOT make any DML statements (INSERT, UPDATE, DELETE, DROP etc.) to the database. If the question does not seem related to the database, just return "I don\'t know" as the answer.
    DO NOT use backticks before and after the query in the action input, even it it a multiline query. It causes syntax errors so the query can't run. This is important!"""
)

Saurabh Sharma 23,851 Reputation points Microsoft Employee Moderator

2024-01-08T18:08:51.7433333+00:00

Hi @Sora Teichman ,

We haven't heard back from you. Just wanted to check if you are you still facing the issue? In case If you already found a solution, would you please share it here with the community? Otherwise, let us know and we will continue to engage with you on the issue.

Thanks

Saurabh

1 answer

Your answer

Saurabh Sharma 23,851 Reputation points Microsoft Employee Moderator

2023-12-07T00:44:38.3466667+00:00

Hi @Sora Teichman

Welcome to Microsoft Q&A! Thanks for posting the question.

Can you please try to modify your input prompt to specify that the output should not add any backticks, as well as provide examples of correct output your input. The more or better examples you give in your prompt the better. Please refer to Few shot prompting and Few shot prompting | Learn Prompting: Your Guide to Communicating with AI . .

Please let me know how it goes.

Thanks

Saurabh
Sora Teichman 0 Reputation points

2023-12-07T07:37:46.77+00:00

Thanks, where is the best place to add this? I am currently using the prefix field in the create_sql_agent method but perhaps there is a better option.

agent_executor = create_sql_agent( llm=AzureChatOpenAI(temperature=0, model_name="gpt-35-turbo", openai_api_type="azure", openai_api_base="https://xxx.openai.azure.com/", openai_api_version="2023-05-15", deployment_name="xxx"), toolkit=toolkit, verbose=True, agent_type=AgentType.ZERO_SHOT_REACT_DESCRIPTION, prefix= """Given an input question, first create a syntactically correct MSSQL query to run, then look at the results of the query and return the answer. Alias the tables in the query to avoid ambiguity. The Model Number is stored in the Number column of the Models table. The Purchase order line table productmodelid column references the modelid column of models. Any table that has a foreign key to the parties id has its name in the parties table. Search for names using like. If you get an error while executing a query, rewrite the query and try again. When asked about sales or cost, make sure to multiply rate by quantity. DO NOT make any DML statements (INSERT, UPDATE, DELETE, DROP etc.) to the database. If the question does not seem related to the database, just return "I don\'t know" as the answer. DO NOT use backticks before and after the query in the action input, even it it a multiline query. It causes syntax errors so the query can't run. This is important!""" )
Saurabh Sharma 23,851 Reputation points Microsoft Employee Moderator

2024-01-08T18:08:51.7433333+00:00

Hi @Sora Teichman ,

We haven't heard back from you. Just wanted to check if you are you still facing the issue? In case If you already found a solution, would you please share it here with the community? Otherwise, let us know and we will continue to engage with you on the issue.

Thanks

Saurabh

Answer 1

Sora Teichman That is the good place to add any information that you want agent to know before generating a query. However, you should also be careful not to make the prefix too long or complex, as it might confuse the agent or reduce its performance. You can use the input_variables parameter to pass any variables that you want the agent to use in the prefix, such as the database name, the table names, the column names, etc. This way, you can reuse the same prefix for different databases or queries by changing the input variables. Can you please try using input_variables along with prefix ?

Ref: https://python.langchain.com/docs/use_cases/qa_structured/sql

Thanks

Saurabh

Share via

Azure Gpt35Turbo replies very differently from OpenAI Gpt3.5 turbo

1 answer

Your answer