Azure OpenAI GPT-4-Vision API error: "The deployed GPT model does not support Vision Enhancement and On Your Data (OYD) with images"

Question

Dear, experts.

I am having trouble implementing enhancements with GPT-4-vision models.

Here is the code I use in python in my app:

response = client.chat.completions.create(
                                model=engine,
                                messages=gpt_messages, 
                                temperature=temperature,
                                stream=stream,
                                max_tokens = 4096,
                                extra_body=gpt_vision_enhancements
                            )


gpt_vision_enhancements = {"dataSources": [{
                                        "type": "AzureComputerVision",
                                        "parameters": {
                                            "endpoint": computer_vision_endpoint,
                                            "key": computer_vision_api_key
                                            }
                                        }],
                                        "enhancements": {
                                            "ocr": {
                                                "enabled": True
                                            },
                                            "grounding": {
                                                "enabled": True
                                            }
                                        }
                                }

I have deployed two models: gpt-4 vision-preview (Australia East) & gpt-4 turbo-2024-04-09 (Sweden Central) but I get the same error for the both:

openai.BadRequestError: Error code: 400 - {'error': {'code': 'InvalidRequest', 'message': 'The deployed GPT model does not support Vision Enhancement and On Your Data (OYD) with images.'}}

However, when I use the same model deployment and Azure Vision resource through Chat Playground in Azure OpenAI studio it works fine:

What I also found is that when I click on "View code" in the chat playground, the code does not seem to contain any reference to a computer vision resource:

import os
import requests
import base64

# Configuration
GPT4V_KEY = "YOUR_API_KEY"
IMAGE_PATH = "YOUR_IMAGE_PATH"
encoded_image = base64.b64encode(open(IMAGE_PATH, 'rb').read()).decode('ascii')
headers = {
    "Content-Type": "application/json",
    "api-key": GPT4V_KEY,
}

# Payload for the request
payload = {
  "enhancements": {
    "ocr": {
      "enabled": True
    },
    "grounding": {
      "enabled": True
    }
  },
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "You are an AI assistant that helps people find information."
        }
      ]
    },
    {
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": f"data:image/jpeg;base64,{encoded_image}"
          }
        },
        {
          "type": "text",
          "text": "What's in the image?"
        }
      ]
    },
    {
      "role": "assistant",
      "content": [
        {
          "type": "text",
          "text": "The image shows an animated character that appears to be a blend of a cow and a tractor. The character has the face of a cow with bovine features, but the body resembles the engine part of a tractor, complete with exhaust pipes and what looks like a grill. The overall design is whimsical and creative, suggesting it might be from an animated movie or a cartoon."
        }
      ]
    }
  ],
  "temperature": 0.7,
  "top_p": 0.95,
  "max_tokens": 800
}

GPT4V_ENDPOINT = "https://oai-australiaeast-frdcsandbox.openai.azure.com/openai/deployments/gpt-4-vision/extensions/chat/completions?api-version=2024-02-15-preview"

# Send request
try:
    response = requests.post(GPT4V_ENDPOINT, headers=headers, json=payload)
    response.raise_for_status()  # Will raise an HTTPError if the HTTP request returned an unsuccessful status code
except requests.RequestException as e:
    raise SystemExit(f"Failed to make the request. Error: {e}")

# Handle the response as needed (e.g., print or process)
print(response.json())

Could you please advise what I am doing wrong. Thanks.

Regards,

Alex

Answer 1

Hello Alex, it seems that the error message you received indicates that the deployed GPT model does not support Vision Enhancement and On Your Data (OYD) with images. This means that the model you are using does not support the features you are trying to use with it. It is possible that the Chat Playground in Azure OpenAI studio is using a different model that supports these features. I would recommend checking the documentation or contacting the support team for the GPT-4-vision models you have deployed to confirm whether they support the Vision Enhancement and OYD features you are trying to use. If they do not support these features, you may need to use a different model that does support them. Additionally, I noticed that the code you provided for the Chat Playground in Azure OpenAI studio does not contain any reference to a computer vision resource. This may indicate that the computer vision resource is not required for the features you are trying to use. I hope this helps! Let me know if you have any further questions.