Cohere rerank v4.0 fast returns 500 status for Azure Foundry

Question

Hi. Our Cohere Rerank V4.0 Fast cannot be used (tested via postman and azure js SDK). All request returned 500 Internal Server Error. This is not an authentication issue because a wrong key will return a 401 instead (correct behaviour). We encounter the same issue when using the v2 link.

We achieved the same issue with the javascript client:

const cohere = new CohereClient({
  token: `our token`,
  baseUrl: `https://<our base url>.services.ai.azure.com/providers/cohere/v2/rerank`
});

Failed to rank using cohere: Error: InternalServerError
Status code: 500
Body: {
  "statusCode": 500,
  "message": "Internal server error",
  "activityId": "66db4ff2-bcef-4606-acf5-0e147b05b545"
}

Answer 1

Hi Bagas

Agreed with above pointers.

Any slip in endpoint/model deployment name might trigger 500 internal server error even though correct key is passed.

Have skip v2/rerank in endpoint name

Attached reference code from my trials (Used new foundry UI)

Code that provided results with correct key

import cohere

co = cohere.ClientV2(
    base_url="https://<foundryproject>.services.ai.azure.com/providers/cohere/",
    api_key="2UYOigwIlCf7ddqPQS8gCuQlumdBW1V6IiNq5GMm3y7aT2FCFqCSJQQJ99CCACREanaXJ3w3AAAAACOGlNG",
)

import yaml

documents = [
    {
        "Title": "Incorrect Password",
        "Content": "Hello, I have been trying to access my account for the past hour and it keeps saying my password is incorrect. Can you please help me?",
    },
    {
        "Title": "Confirmation Email Missed",
        "Content": "Hi, I recently purchased a product from your website but I never received a confirmation email. Can you please look into this for me?",
    },
    {
        "Title": "Questions about Return Policy",
        "Content": "Hello, I have a question about the return policy for this product. I purchased it a few weeks ago and it is defective.",
    },
    {
        "Title": "Customer Support is Busy",
        "Content": "Good morning, I have been trying to reach your customer support team for the past week but I keep getting a busy signal. Can you please help me?",
    },
    {
        "Title": "Received Wrong Item",
        "Content": "Hi, I have a question about my recent order. I received the wrong item and I need to return it.",
    },
    {
        "Title": "Customer Service is Unavailable",
        "Content": "Hello, I have been trying to reach your customer support team for the past hour but I keep getting a busy signal. Can you please help me?",
    },
    {
        "Title": "Return Policy for Defective Product",
        "Content": "Hi, I have a question about the return policy for this product. I purchased it a few weeks ago and it is defective.",
    },
    {
        "Title": "Wrong Item Received",
        "Content": "Good morning, I have a question about my recent order. I received the wrong item and I need to return it.",
    },
    {
        "Title": "Return Defective Product",
        "Content": "Hello, I have a question about the return policy for this product. I purchased it a few weeks ago and it is defective.",
    },
]

response = co.rerank(
    model="Cohere-rerank-v4.0-fast",  # NOTE: For rerank v4, you must specify the model e.g. `Cohere-rerank-v4.0-fast` or `Cohere-rerank-v4.0-pro`
    documents=[yaml.dump(doc, sort_keys=False) for doc in documents],
    query="What emails have been about returning items?",
    top_n=5,
)
print(response)

Output

id='7b32b918-e582-470c-9a12-450c1c2996a7' results=[V2RerankResponseResultsItem(index=7, relevance_score=0.70869666), V2RerankResponseResultsItem(index=4, relevance_score=0.6822351), V2RerankResponseResultsItem(index=8, relevance_score=0.52692705), V2RerankResponseResultsItem(index=2, relevance_score=0.5035156), V2RerankResponseResultsItem(index=6, relevance_score=0.44707403)] ...

Code that throws 500 internal server error with correct key

response = co.rerank(
    model="Cohere-rerank-v4.0",  # Model name was corrupted knowingly her
    documents=[yaml.dump(doc, sort_keys=False) for doc in documents],
    query="What emails have been about returning items?",
    top_n=5,
)
print(response)

Reference used- https://github.com/Azure/azureml-examples/blob/main/sdk/python/foundation-models/cohere/rerank-cohere-client.ipynb

Please test above syntax and new foundry endpoint and share your observation if the issue still persits.

Thank you.

Answer 2

Hello Bagas,

Thanks for raising this question in Azure Q&A forum.

This is a known issue with the Cohere Rerank v4.0 Fast model deployment in Azure AI Foundry that has been reported by multiple users recently. The 500 error occurs because of two separate but related problems: an incorrect model name parameter and an incorrect endpoint URL shown in the Foundry portal.

Root Cause #1: Model Name Mismatch (causes 500 error) The model parameter in your API request must exactly match the deployment name created in Foundry. When you deploy Cohere Rerank v4.0 Fast, the portal creates a deployment named Cohere-rerank-v4.0-fast (note the exact casing and hyphens). Using any variation (like cohere-rerank-v4.0-fast or rerank-v4.0-fast) triggers a 500 internal server error.

Root Cause #2: Incorrect Endpoint URL (causes 404 error) The Azure AI Foundry portal currently displays an incorrect endpoint URL for Cohere model deployments this is a known UI bug. The URL shown in the deployment console will not work. Instead, you must manually construct the endpoint URL using this format:

text
https://<your-foundry-project-name>.services.ai.azure.com/providers/cohere/v2/rerank

Complete Fix

Verify your deployment name:

Go to Azure AI Foundry → your project → Models + Endpoints → Deployments

  Copy the exact name of your Cohere Rerank v4.0 Fast deployment (should be `Cohere-rerank-v4.0-fast`)
  
  **Construct the correct endpoint URL**:
  
  ```yaml
  text
  https://your-project-name.services.ai.azure.com/providers/cohere/v2/rerank
  ```
  
  Replace `your-project-name` with your actual Foundry **project name** (visible in the project breadcrumb)
  
  **Update your client code**:
  
  ```python
  python
  import cohere

co = cohere.ClientV2( api_key="<YOUR_FOUNDARY_API_KEY>", base_url="https://your-project-name.services.ai.azure.com/providers/cohere/v2/rerank" )

result = co.rerank( model="Cohere-rerank-v4.0-fast",
```

  **Get the correct API key**:
  
     In your Foundry deployment → **Get endpoint** → copy the API key from there (not a general Azure key)
     

Why This Happens The Cohere Rerank models deployed in Foundry are serverless endpoints that use a special proxy architecture, and the Foundry portal's endpoint display has not been updated to reflect the correct providers/cohere/v2/rerank path. Microsoft is aware of this UI issue but the manual URL construction works reliably.

This fix has resolved the 500/404 errors for all users who reported the same issue.

If it helps kindly accept the answer.

Best Regards,

Jerald Felix