My mistral-large-2407 serverless deployment api is suddenly failing

Question

I am using a serverless mistral-large-2407 model. I connect over langchain tothe OpenAI-Api-Endpoint. Since the 08 November 2024 my requests are failing. I connect to the OpenAI-Endpoint in the following way:

from langchain_openai import  ChatOpenAI

llm = ChatOpenAI(
model="mistral-large-2407",
api_key="xxxx",
base_url = "...models.ai.azure.com")                 
llm.invoke("hi")

This error is produced:
Error code: 400 - {'detail': "Extra parameters ['n'] are not allowed when extra-parameters is not set or set to be 'error'. Set extra-parameters to 'pass-through' to pass to the model."}

If i connect with the mistral client:

from langchain_mistralai import ChatMistralAI

llm = ChatMistralAI(model="mistral-large-2407",
           api_key="xxxxx",
           base_url="....models.ai.azure.com")
llm.invoke("hi")

This error is produced:

{"detail":"Extra parameters ['safe_prompt'] are not allowed when extra-parameters is not set or set to be 'error'. Set extra-parameters to 'pass-through' to pass to the model."

Thanks in advance!

Answer 1

Michael Jaumann I'm glad that your issue is resolved and thank you for posting your solution so that others experiencing the same thing can easily reference this!

Since the Microsoft Q&A community has a policy that the question author cannot accept their own answer, they can only accept answers by others, I'll repost your solution in case you'd like to Accept the answer.

Error Message:

Error code: 400 - {'detail': "Extra parameters ['n'] are not allowed when extra-parameters is not set or set to be 'error'. Set extra-parameters to 'pass-through' to pass to the model."}

Solution:

I have found a solution for this by using the OpenAI-compltible endpoint of the mistral model and setting the extra-parameter header.

from langchain_openai import  ChatOpenAI

mistral_model = ChatOpenAI(
        default_headers={"extra-parameters": "pass-through"},
        base_url=base_url_mistral,
        api_key=os.getenv("AZURE_MISTRAL_KEY"),
        model=deployment_name_mistral,
    )

# This produces an exception.
mistral_model.invoke("hi")

If you have any other questions, please let me know. Thank you again for your time and patience throughout this issue.

Answer 2

I have found a solution for this by using the OpenAI-compltible endpoint of the mistral model and setting the extra-parameter header.

from langchain_openai import  ChatOpenAI

mistral_model = ChatOpenAI(
        default_headers={"extra-parameters": "pass-through"},
        base_url=base_url_mistral,
        api_key=os.getenv("AZURE_MISTRAL_KEY"),
        model=deployment_name_mistral,
    )

# This produces an exception.
mistral_model.invoke("hi")

The extra-parameters header is documented here