Прочитај на енглеском Уреди

Делите путем

Quickstart: Use images in your AI chats

  • Чланак
Choose your preferred usage method

Get started using GPT-4 Turbo with images with the Azure OpenAI Service.

Напомена

Model choice

The latest vision-capable models are gpt-4o and gpt-4o mini. These models are in public preview. The latest available GA model is gpt-4 version turbo-2024-04-09.

Важно

Extra usage fees might apply when using chat completion models with vision functionality.

Use this article to get started using Azure AI Foundry to deploy and test a chat completion model with image understanding.

Prerequisites

Prepare your media

You need an image to complete this quickstart. You can use this sample image or any other image you have available.

Photo of a car accident that can be used to complete the quickstart.

Go to Azure AI Foundry

  1. Browse to Azure AI Foundry and sign in with the credentials associated with your Azure OpenAI resource. During or after the sign-in workflow, select the appropriate directory, Azure subscription, and Azure OpenAI resource.
  2. Select the hub you'd like to work in.
  3. On the left nav menu, select Models + endpoints and select + Deploy model.
  4. Choose an image-capable deployment by selecting model name: gpt-4o or gpt-4o-mini. In the window that appears, select a name and deployment type. Make sure your Azure OpenAI resource is connected. For more information about model deployment, see the resource deployment guide.
  5. Select Deploy.
  6. Next, select your new model and select Open in playground. In the chat playground, the deployment you created should be selected in the Deployment dropdown.

Playground

In this chat session, you instruct the assistant to aid you in understanding images that you input.

For general help with assistant setup, chat sessions, settings, and panels, refer to the Chat quickstart.

Start a chat session to analyze images

In this chat session, you're instructing the assistant to aid in understanding images that you input.

  1. To start, make sure your image-capable deployment is selected in the Deployment dropdown.

  2. In the context text box on the Setup panel, provide this prompt to guide the assistant: "You're an AI assistant that helps people find information." Or, you can tailor the prompt to your image or scenario.

    Напомена

    We recommend you update the System Message to be specific to the task in order to avoid unhelpful responses from the model.

  3. Select Apply changes to save your changes.

  4. In the chat session pane, select the attachment button and then Upload image. Choose your image.

  5. Add the following prompt in the chat field: Describe this image, and then select the send icon to submit it.

  6. The send icon is replaced by a stop button. If you select it, the assistant stops processing your request. For this quickstart, let the assistant finish its reply.

  7. The assistant replies with a description of the image.

  8. Ask a follow-up question related to the analysis of your image. You could enter, "What should I highlight about this image to my insurance company?".

  9. You should receive a relevant response similar to what's shown here:

    When reporting the incident to your insurance company, you should highlight the following key points from the image:  

1. **Location of Damage**: Clearly state that the front end of the car, particularly the driver's side, is damaged. Mention the crumpled hood, broken front bumper, and the damaged left headlight.  

2. **Point of Impact**: Indicate that the car has collided with a guardrail, which may suggest that no other vehicles were involved in the accident.  

3. **Condition of the Car**: Note that the damage seems to be concentrated on the front end, and there is no visible damage to the windshield or rear of the car from this perspective.  

4. **License Plate Visibility**: Mention that the license plate is intact and can be used for identification purposes.  

5. **Environment**: Report that the accident occurred near a roadside with a guardrail, possibly in a rural or semi-rural area, which might help in establishing the accident location and context.  

6. **Other Observations**: If there were any other circumstances or details not visible in the image that may have contributed to the accident, such as weather conditions, road conditions, or any other relevant information, be sure to include those as well.  

Remember to be factual and descriptive, avoiding speculation about the cause of the accident, as the insurance company will conduct its own investigation.

View and export code

At any point in the chat session, you can enable the Show raw JSON switch at the top of the chat window to see the conversation formatted as JSON. Heres' what it looks like at the beginning of the quickstart chat session:

JSON 
[
	{
		"role": "system",
		"content": [
			"You are an AI assistant that helps people find information."
		]
	},
]

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Use this article to get started using the Azure OpenAI REST APIs to deploy and use the GPT-4 Turbo with Vision model.

Prerequisites

Напомена

It is currently not supported to turn off content filtering for the GPT-4 Turbo with Vision model.

Retrieve key and endpoint

To successfully call the Azure OpenAI APIs, you need the following information about your Azure OpenAI resource:

Variable Name Value
Endpoint api_base The endpoint value is located under Keys and Endpoint for your resource in the Azure portal. You can also find the endpoint via the Deployments page in Azure AI Foundry portal. An example endpoint is: https://docs-test-001.openai.azure.com/.
Key api_key The key value is also located under Keys and Endpoint for your resource in the Azure portal. Azure generates two keys for your resource. You can use either value.

Go to your resource in the Azure portal. On the navigation pane, select Keys and Endpoint under Resource Management. Copy the Endpoint value and an access key value. You can use either the KEY 1 or KEY 2 value. Having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Screenshot that shows the Keys and Endpoint page for an Azure OpenAI resource in the Azure portal.

Create a new Python application

Create a new Python file named quickstart.py. Open the new file in your preferred editor or IDE.

  1. Replace the contents of quickstart.py with the following code.

    Python 
    # Packages required:
import requests 
import json 

api_base = '<your_azure_openai_endpoint>' 
deployment_name = '<your_deployment_name>'
API_KEY = '<your_azure_openai_key>'

base_url = f"{api_base}openai/deployments/{deployment_name}" 
headers = {   
    "Content-Type": "application/json",   
    "api-key": API_KEY 
} 

# Prepare endpoint, headers, and request body 
endpoint = f"{base_url}/chat/completions?api-version=2023-12-01-preview" 
data = { 
    "messages": [ 
        { "role": "system", "content": "You are a helpful assistant." }, 
        { "role": "user", "content": [  
            { 
                "type": "text", 
                "text": "Describe this picture:" 
            },
            { 
                "type": "image_url",
                "image_url": {
                    "url": "<image URL>"
                }
            }
        ] } 
    ], 
    "max_tokens": 2000 
}   

# Make the API call   
response = requests.post(endpoint, headers=headers, data=json.dumps(data))   

print(f"Status Code: {response.status_code}")   
print(response.text)

  2. Make the following changes:

    1. Enter your endpoint URL and key in the appropriate fields.
    2. Enter your GPT-4 Turbo with Vision deployment name in the appropriate field.
    3. Change the value of the "image" field to the URL of your image.

      Савет

      You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

  3. Run the application with the python command:

    Console 
    python quickstart.py

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Use this article to get started using the Azure OpenAI Python SDK to deploy and use the GPT-4 Turbo with Vision model.

Library source code | Package (PyPi) |

Prerequisites

Set up

Install the OpenAI Python client library with:

Console 
pip install openai

Напомена

This library is maintained by OpenAI. Refer to the release history to track the latest updates to the library.

Retrieve key and endpoint

To successfully make a call against Azure OpenAI, you need an endpoint and a key.

Variable name Value
ENDPOINT The service endpoint can be found in the Keys & Endpoint section when examining your resource from the Azure portal. Alternatively, you can find the endpoint via the Deployments page in Azure AI Foundry portal. An example endpoint is: https://docs-test-001.openai.azure.com/.
API-KEY This value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.

Go to your resource in the Azure portal. The Keys & Endpoint section can be found in the Resource Management section. Copy your endpoint and access key as you'll need both for authenticating your API calls. You can use either KEY1 or KEY2. Always having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Screenshot of the overview UI for an Azure OpenAI resource in the Azure portal with the endpoint and access keys location circled in red.

Environment variables

Create and assign persistent environment variables for your key and endpoint.

Важно

We recommend Microsoft Entra ID authentication with managed identities for Azure resources to avoid storing credentials with your applications that run in the cloud.

Use API keys with caution. Don't include the API key directly in your code, and never post it publicly. If using API keys, store them securely in Azure Key Vault, rotate the keys regularly, and restrict access to Azure Key Vault using role based access control and network access restrictions. For more information about using API keys securely in your apps, see API keys with Azure Key Vault.

For more information about AI services security, see Authenticate requests to Azure AI services.

Windows Command Prompt 
setx AZURE_OPENAI_API_KEY "REPLACE_WITH_YOUR_KEY_VALUE_HERE" 
setx AZURE_OPENAI_ENDPOINT "REPLACE_WITH_YOUR_ENDPOINT_HERE"

Create a new Python application

Create a new Python file named quickstart.py. Open the new file in your preferred editor or IDE.

  1. Replace the contents of quickstart.py with the following code.

    Python 
    from openai import AzureOpenAI

api_base = os.getenv("AZURE_OPENAI_ENDPOINT")
api_key= os.getenv("AZURE_OPENAI_API_KEY")
deployment_name = '<your_deployment_name>'
api_version = '2023-12-01-preview' # this might change in the future

client = AzureOpenAI(
    api_key=api_key,  
    api_version=api_version,
    base_url=f"{api_base}/openai/deployments/{deployment_name}"
)

response = client.chat.completions.create(
    model=deployment_name,
    messages=[
        { "role": "system", "content": "You are a helpful assistant." },
        { "role": "user", "content": [  
            { 
                "type": "text", 
                "text": "Describe this picture:" 
            },
            { 
                "type": "image_url",
                "image_url": {
                    "url": "<image URL>"
                }
            }
        ] } 
    ],
    max_tokens=2000 
)

print(response)

  2. Make the following changes:

    1. Enter the name of your GPT-4 Turbo with Vision deployment in the appropriate field.
    2. Change the value of the "url" field to the URL of your image.

      Савет

      You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

  3. Run the application with the python command:

    Console 
    python quickstart.py

Важно

Use API keys with caution. Don't include the API key directly in your code, and never post it publicly. If you use an API key, store it securely in Azure Key Vault. For more information about using API keys securely in your apps, see API keys with Azure Key Vault.

For more information about AI services security, see Authenticate requests to Azure AI services.

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Use this article to get started using the OpenAI JavaScript SDK to deploy and use the GPT-4 Turbo with Vision model.

This SDK is provided by OpenAI with Azure specific types provided by Azure.

Reference documentation | Library source code | Package (npm) | Samples

Prerequisites

Напомена

This library is maintained by OpenAI. Refer to the release history to track the latest updates to the library.

Microsoft Entra ID prerequisites

For the recommended keyless authentication with Microsoft Entra ID, you need to:

  • Install the Azure CLI used for keyless authentication with Microsoft Entra ID.
  • Assign the Cognitive Services User role to your user account. You can assign roles in the Azure portal under Access control (IAM) > Add role assignment.

Set up

  1. Create a new folder vision-quickstart to contain the application and open Visual Studio Code in that folder with the following command:

    shell 
    mkdir vision-quickstart && cd vision-quickstart

  2. Create the package.json with the following command:

    shell 
    npm init -y

  3. Install the OpenAI client library for JavaScript with:

    Console 
    npm install openai

  4. For the recommended passwordless authentication:

    Console 
    npm install @azure/identity

Retrieve resource information

You need to retrieve the following information to authenticate your application with your Azure OpenAI resource:

Variable name Value
AZURE_OPENAI_ENDPOINT This value can be found in the Keys and Endpoint section when examining your resource from the Azure portal.
AZURE_OPENAI_DEPLOYMENT_NAME This value will correspond to the custom name you chose for your deployment when you deployed a model. This value can be found under Resource Management > Model Deployments in the Azure portal.
OPENAI_API_VERSION Learn more about API Versions.

You can change the version in code or use an environment variable.

Learn more about keyless authentication and setting environment variables.

Опрез

To use the recommended keyless authentication with the SDK, make sure that the AZURE_OPENAI_API_KEY environment variable isn't set.

Create a new JavaScript application for image prompts

Select an image from the azure-samples/cognitive-services-sample-data-files. Use the image URL in the code below or set the IMAGE_URL environment variable to the image URL.

Савет

You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

  1. Create the index.js file with the following code:

    JavaScript 
    const AzureOpenAI = require('openai').AzureOpenAI;
const { 
    DefaultAzureCredential, 
    getBearerTokenProvider 
} = require('@azure/identity');

// You will need to set these environment variables or edit the following values
const endpoint = process.env.AZURE_OPENAI_ENDPOINT || "Your endpoint";
const imageUrl = process.env.IMAGE_URL || "<image url>";

// Required Azure OpenAI deployment name and API version
const apiVersion = process.env.OPENAI_API_VERSION || "2024-07-01-preview";
const deploymentName = process.env.AZURE_OPENAI_DEPLOYMENT_NAME || "gpt-4-with-turbo";

// keyless authentication    
const credential = new DefaultAzureCredential();
const scope = "https://cognitiveservices.azure.com/.default";
const azureADTokenProvider = getBearerTokenProvider(credential, scope);

function getClient(): AzureOpenAI {
  return new AzureOpenAI({
    endpoint,
    azureADTokenProvider,
    apiVersion,
    deployment: deploymentName,
  });
}
function createMessages() {
  return {
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "Describe this picture:",
          },
          {
            type: "image_url",
            image_url: {
              url: imageUrl,
            },
          },
        ],
      },
    ],
    model: "",
    max_tokens: 2000,
  };
}
async function printChoices(completion) {
  for (const choice of completion.choices) {
    console.log(choice.message);
  }
}
export async function main() {
  console.log("== Get GPT-4 Turbo with vision Sample ==");

  const client = getClient();
  const messages = createMessages();
  const completion = await client.chat.completions.create(messages);
  await printChoices(completion);
}

main().catch((err) => {
  console.error("Error occurred:", err);
});

  2. Sign in to Azure with the following command:

    shell 
    az login

  3. Run the JavaScript file.

    shell 
    node index.js

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Use this article to get started using the OpenAI JavaScript SDK to deploy and use the GPT-4 Turbo with Vision model.

This SDK is provided by OpenAI with Azure specific types provided by Azure.

Reference documentation | Library source code | Package (npm) | Samples

Prerequisites

Напомена

This library is maintained by OpenAI. Refer to the release history to track the latest updates to the library.

Microsoft Entra ID prerequisites

For the recommended keyless authentication with Microsoft Entra ID, you need to:

  • Install the Azure CLI used for keyless authentication with Microsoft Entra ID.
  • Assign the Cognitive Services User role to your user account. You can assign roles in the Azure portal under Access control (IAM) > Add role assignment.

Set up

  1. Create a new folder vision-quickstart to contain the application and open Visual Studio Code in that folder with the following command:

    shell 
    mkdir vision-quickstart && cd vision-quickstart

  2. Create the package.json with the following command:

    shell 
    npm init -y

  3. Update the package.json to ECMAScript with the following command:

    shell 
    npm pkg set type=module

  4. Install the OpenAI client library for JavaScript with:

    Console 
    npm install openai

  5. For the recommended passwordless authentication:

    Console 
    npm install @azure/identity

Retrieve resource information

You need to retrieve the following information to authenticate your application with your Azure OpenAI resource:

Variable name Value
AZURE_OPENAI_ENDPOINT This value can be found in the Keys and Endpoint section when examining your resource from the Azure portal.
AZURE_OPENAI_DEPLOYMENT_NAME This value will correspond to the custom name you chose for your deployment when you deployed a model. This value can be found under Resource Management > Model Deployments in the Azure portal.
OPENAI_API_VERSION Learn more about API Versions.

You can change the version in code or use an environment variable.

Learn more about keyless authentication and setting environment variables.

Опрез

To use the recommended keyless authentication with the SDK, make sure that the AZURE_OPENAI_API_KEY environment variable isn't set.

Create a new JavaScript application for image prompts

Select an image from the azure-samples/cognitive-services-sample-data-files. Use the image URL in the code below or set the IMAGE_URL environment variable to the image URL.

Савет

You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

  1. Create the index.ts file with the following code:

    TypeScript 
    import { AzureOpenAI } from "openai";
import { 
    DefaultAzureCredential, 
    getBearerTokenProvider 
} from "@azure/identity";
import type {
  ChatCompletion,
  ChatCompletionCreateParamsNonStreaming,
} from "openai/resources/index";

// You will need to set these environment variables or edit the following values
const endpoint = process.env.AZURE_OPENAI_ENDPOINT || "Your endpoint";
const imageUrl = process.env["IMAGE_URL"] || "<image url>";

// Required Azure OpenAI deployment name and API version
const apiVersion = process.env.OPENAI_API_VERSION || "2024-07-01-preview";
const deploymentName = process.env.AZURE_OPENAI_DEPLOYMENT_NAME || "gpt-4-with-turbo";

// keyless authentication    
const credential = new DefaultAzureCredential();
const scope = "https://cognitiveservices.azure.com/.default";
const azureADTokenProvider = getBearerTokenProvider(credential, scope);

function getClient(): AzureOpenAI {
  return new AzureOpenAI({
    endpoint,
    azureADTokenProvider,
    apiVersion,
    deployment: deploymentName,
  });
}
function createMessages(): ChatCompletionCreateParamsNonStreaming {
  return {
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "Describe this picture:",
          },
          {
            type: "image_url",
            image_url: {
              url: imageUrl,
            },
          },
        ],
      },
    ],
    model: "",
    max_tokens: 2000,
  };
}
async function printChoices(completion: ChatCompletion): Promise<void> {
  for (const choice of completion.choices) {
    console.log(choice.message);
  }
}
export async function main() {
  console.log("== Get GPT-4 Turbo with vision Sample ==");

  const client = getClient();
  const messages = createMessages();
  const completion = await client.chat.completions.create(messages);
  await printChoices(completion);
}

main().catch((err) => {
  console.error("Error occurred:", err);
});

  2. Create the tsconfig.json file to transpile the TypeScript code and copy the following code for ECMAScript.

    JSON 
    {
    "compilerOptions": {
      "module": "NodeNext",
      "target": "ES2022", // Supports top-level await
      "moduleResolution": "NodeNext",
      "skipLibCheck": true, // Avoid type errors from node_modules
      "strict": true // Enable strict type-checking options
    },
    "include": ["*.ts"]
}

  3. Transpile from TypeScript to JavaScript.

    shell 
    tsc

  4. Sign in to Azure with the following command:

    shell 
    az login

  5. Run the code with the following command:

    shell 
    node index.js

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Use this article to get started using the Azure OpenAI .NET SDK to deploy and use the GPT-4 Turbo with Vision model.

Prerequisites

Set up

Retrieve key and endpoint

To successfully make a call against Azure OpenAI, you need an endpoint and a key.

Variable name Value
AZURE_OPENAI_ENDPOINT The service endpoint can be found in the Keys & Endpoint section when examining your resource from the Azure portal. Alternatively, you can find the endpoint via the Deployments page in Azure AI Foundry portal. An example endpoint is: https://docs-test-001.openai.azure.com/.
AZURE_OPENAI_API_KEY This value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.

Go to your resource in the Azure portal. The Endpoint and Keys can be found in the Resource Management section. Copy your endpoint and access key as you need both for authenticating your API calls. You can use either KEY1 or KEY2. Always having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Screenshot of the overview UI for an Azure OpenAI resource in the Azure portal with the endpoint & access keys location highlighted.

Create the .NET app

  1. Create a .NET app using the dotnet new command:

    .NET CLI 
    dotnet new console -n OpenAISpeech

  2. Change into the directory of the new app:

    .NET CLI 
    cd OpenAISpeech

Install the client library

Install the Azure.OpenAI client library:

.NET CLI 
dotnet add package Azure.AI.OpenAI

Passwordless authentication is recommended

Passwordless authentication is more secure than key-based alternatives and is the recommended approach for connecting to Azure services. If you choose to use Passwordless authentication, you'll need to complete the following:

  1. Add the Azure.Identity package.

    .NET CLI 
    dotnet add package Azure.Identity

  2. Assign the Cognitive Services User role to your user account. This can be done in the Azure portal on your OpenAI resource under Access control (IAM) > Add role assignment.

  3. Sign-in to Azure using Visual Studio or the Azure CLI via az login.

Update the app code

  1. Replace the contents of program.cs with the following code and update the placeholder values with your own.

    C# 
    using Azure;
using Azure.AI.OpenAI;
using Azure.Identity;
using OpenAI.Chat; // Required for Passwordless auth

var endpoint = new Uri("YOUR_AZURE_OPENAI_ENDPOINT");
var credentials = new AzureKeyCredential("YOUR_AZURE_OPENAI_KEY");
// var credentials = new DefaultAzureCredential(); // Use this line for Passwordless auth
var deploymentName = "gpt-4"; // Default name, update with your own if needed

var openAIClient = new AzureOpenAIClient(endpoint, credentials);
var chatClient = openAIClient.GetChatClient(deploymentName);

var imageUri = "YOUR_IMAGE_URL";

List<ChatMessage> messages = [
    new UserChatMessage(
        ChatMessageContentPart.CreateTextMessageContentPart("Please describe the following image:"),
        ChatMessageContentPart.CreateImageMessageContentPart(new Uri(imageUri), "image/png"))
];

ChatCompletion chatCompletion = await chatClient.CompleteChatAsync(messages);

Console.WriteLine($"[ASSISTANT]:");
Console.WriteLine($"{chatCompletion.Content[0].Text}");

Важно

We recommend Microsoft Entra ID authentication with managed identities for Azure resources to avoid storing credentials with your applications that run in the cloud.

Use API keys with caution. Don't include the API key directly in your code, and never post it publicly. If using API keys, store them securely in Azure Key Vault, rotate the keys regularly, and restrict access to Azure Key Vault using role based access control and network access restrictions. For more information about using API keys securely in your apps, see API keys with Azure Key Vault.

For more information about AI services security, see Authenticate requests to Azure AI services.

  1. Run the application using the dotnet run command or the run button at the top of Visual Studio:

    .NET CLI 
    dotnet run

The app generates an audio file at the location you specified for the speechFilePath variable. Play the file on your device to hear the generated audio.

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Next steps