Issue with Vision Chat using Base64 Image String

Question

Hello everyone,

I have been following the tutorial titled "Call the Chat Completion APIs," and I am encountering an issue when attempting to upload an image in base64 string format. I consistently receive a 400 error, and the error message is as follows:

'{"error":{"code":"BadRequest","message":"Invalid image URL. The URL must be a valid HTTP or HTTPS URL or a data URL with base64 encoding.","param":null,"type":null}}'

I'm wondering if this issue might be related to the API version or the endpoint.

Here's a snippet of my request code:

import base64
import requests

# Encode the image to base64
sImageData = base64.b64encode(open(sLongImageFn, 'rb').read()).decode('ascii')

# dData is copied from the tutorial
dData = {
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe this picture:"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": sImageData
                    }
                }
            ]
        }
    ],
    "max_tokens": 100,
    "stream": False
}

# Make the API request
response = requests.post(
    f'{sEndpoint}openai/deployments/{sDeploymentName}/chat/completions?api-version=2023-12-01-preview',
    headers={'api-key': sKey, 'Content-Type': 'application/json'},
    json=dData
)

I also tried to use the OpenAI format, however it looks like the image is just skipped and the response text is short.

OpenAI format I tried

{
  "type": "image_url",
  "image_url": {
       "url": f"data:image/jpeg;base64,{sImageData}",
}

I appreciate any assistance you can provide. Thank you!

Answer 1

@Jhuze Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

.

I tried with the below sample code and it worked fine as expected. I was able to get the image description response with 200 (success) status code.

.

import base64
import requests

# Encode the image to base64
sLongImageFn='newimage.jpg'
sImageData = base64.b64encode(open(sLongImageFn, 'rb').read()).decode('ascii')
sEndpoint='https://navbaoaigptvision1.openai.azure.com/'
sKey='83f2b417c2a84a3a859d2e653c30270e'

# dData is copied from the tutorial
dData = {
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe this picture:"
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{sImageData}",
                    }
                }
            ]
        }
    ],
    "max_tokens": 100,
    "stream": False
}

# Make the API request
response = requests.post(
    f'{sEndpoint}openai/deployments/test/chat/completions?api-version=2023-12-01-preview',
    headers={'api-key': sKey, 'Content-Type': 'application/json'},
    json=dData
)

# Print the response
print(response.json())

Please try with the above sample code for now and let me know if that helps.
.
We currently support PNG (.png), JPEG (.jpeg and .jpg), WEBP (.webp), and non-animated GIF (.gif). Can you try with this image for now and check ?
.
Also please refer this limitations section for the image support with Gpt vision preview feature.
.

Hope this helps. If you have any follow-up questions, please let me know. I would be happy to help.