Azure OpenAI API with gpt-4o deployment fails to create message with type image_url

Question

I'm trying to create a message with Open AI Assistants API using gpt-4o model. One of the content has type "image_url". But i'm getting error that image_url is not supported and only text is supported. I checked this link https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models and it says the gpt 4 models accept image and text input.

Answer 1

Here is an example of node js code to use a reference of an image in restful call. You can pass in a url and we used it and can confirm working. The same can be done by passing a base64 encoded image data string.

const { OpenAIClient, AzureKeyCredential } = require("@azure/openai");
async function CompletionApi(prompt) {
    const options = {
        api_version: "2023-12-01-preview"
    };
    const client = new OpenAIClient(
        `https://xxxx-openai.openai.azure.com`,
        new AzureKeyCredential("<add key>"),
        options
    );
    const deploymentName = 'completions';
    const result = await client.getChatCompletions(deploymentName, prompt);
    console.log(JSON.stringify(result, null, 2));
    return ;
}
(async () => {
    const prompt = [
        { "role": "system", "content": "You are a helpful assistant." },
        {
            "role": "user", "content": [
                {
                    "type": "text",
                    "text": "Describe this picture:"
                },
                {
                    "type": "image_url",
                    "imageUrl": {
                        "url": `<image url>`
                    }
                }
            ]
        }
    ];
    const response = await CompletionApi(prompt);
    console.log(response);
})();

Answer 2

Hello rasyid prawira,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you are getting error that image_url is not supported in gpt-4o and only text is supported. Even, after you checked a link https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models that affirm the gpt 4 models accept image and text input.

So sorry for the inconveniences this might have caused you, the GPT-4o model only accept text input and not support image_url. You might need to use another model such as GPT-4-turbo which accept image_urls in user messages. Check this links for similar questions - https://community.openai.com/t/gpt-4o-error-image-urls-in-system-messages/745769 and https://learn.microsoft.com/en-us/answers/questions/1688434/gpt-4o-image-processing-issues

However, there is another solution in this link https://community.openai.com/t/gpt-4o-sometime-not-able-to-access-image-from-url/903528 that advise to encoding the image in base64 and sending them as part of the text content or use another GPT-4 that support image.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful