Уреди

Делите путем

Quickstart: Use images in your AI chats

  • Чланак

Get started using GPT-4 Turbo with images with the Azure OpenAI Service.

Note

Model choice

The latest vision-capable models are gpt-4o and gpt-4o mini. These models are in public preview. The latest available GA model is gpt-4 version turbo-2024-04-09.

Important

Extra usage fees might apply when using chat completion models with vision functionality.

Use this article to get started using Azure AI Foundry to deploy and test a chat completion model with image understanding.

Prerequisites

Prepare your media

You need an image to complete this quickstart. You can use this sample image or any other image you have available.

Photo of a car accident that can be used to complete the quickstart.

Go to Azure AI Foundry

  1. Browse to Azure AI Foundry and sign in with the credentials associated with your Azure OpenAI resource. During or after the sign-in workflow, select the appropriate directory, Azure subscription, and Azure OpenAI resource.
  2. Select the hub you'd like to work in.
  3. On the left nav menu, select Models + endpoints and select + Deploy model.
  4. Choose an image-capable deployment by selecting model name: gpt-4o or gpt-4o-mini. In the window that appears, select a name and deployment type. Make sure your Azure OpenAI resource is connected. For more information about model deployment, see the resource deployment guide.
  5. Select Deploy.
  6. Next, select your new model and select Open in playground. In the chat playground, the deployment you created should be selected in the Deployment dropdown.

Playground

In this chat session, you instruct the assistant to aid you in understanding images that you input.

For general help with assistant setup, chat sessions, settings, and panels, refer to the Chat quickstart.

Start a chat session to analyze images

In this chat session, you're instructing the assistant to aid in understanding images that you input.

  1. To start, make sure your image-capable deployment is selected in the Deployment dropdown.

  2. In the context text box on the Setup panel, provide this prompt to guide the assistant: "You're an AI assistant that helps people find information." Or, you can tailor the prompt to your image or scenario.

    Note

    We recommend you update the System Message to be specific to the task in order to avoid unhelpful responses from the model.

  3. Select Apply changes to save your changes.

  4. In the chat session pane, select the attachment button and then Upload image. Choose your image.

  5. Add the following prompt in the chat field: Describe this image, and then select the send icon to submit it.

  6. The send icon is replaced by a stop button. If you select it, the assistant stops processing your request. For this quickstart, let the assistant finish its reply.

  7. The assistant replies with a description of the image.

  8. Ask a follow-up question related to the analysis of your image. You could enter, "What should I highlight about this image to my insurance company?".

  9. You should receive a relevant response similar to what's shown here:

    When reporting the incident to your insurance company, you should highlight the following key points from the image:  

1. **Location of Damage**: Clearly state that the front end of the car, particularly the driver's side, is damaged. Mention the crumpled hood, broken front bumper, and the damaged left headlight.  

2. **Point of Impact**: Indicate that the car has collided with a guardrail, which may suggest that no other vehicles were involved in the accident.  

3. **Condition of the Car**: Note that the damage seems to be concentrated on the front end, and there is no visible damage to the windshield or rear of the car from this perspective.  

4. **License Plate Visibility**: Mention that the license plate is intact and can be used for identification purposes.  

5. **Environment**: Report that the accident occurred near a roadside with a guardrail, possibly in a rural or semi-rural area, which might help in establishing the accident location and context.  

6. **Other Observations**: If there were any other circumstances or details not visible in the image that may have contributed to the accident, such as weather conditions, road conditions, or any other relevant information, be sure to include those as well.  

Remember to be factual and descriptive, avoiding speculation about the cause of the accident, as the insurance company will conduct its own investigation.

View and export code

At any point in the chat session, you can enable the Show raw JSON switch at the top of the chat window to see the conversation formatted as JSON. Heres' what it looks like at the beginning of the quickstart chat session:

[
	{
		"role": "system",
		"content": [
			"You are an AI assistant that helps people find information."
		]
	},
]

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Use this article to get started using the Azure OpenAI REST APIs to deploy and use the GPT-4 Turbo with Vision model.

Prerequisites

Note

It is currently not supported to turn off content filtering for the GPT-4 Turbo with Vision model.

Retrieve key and endpoint

To successfully call the Azure OpenAI APIs, you need the following information about your Azure OpenAI resource:

Variable Name Value
Endpoint api_base The endpoint value is located under Keys and Endpoint for your resource in the Azure portal. You can also find the endpoint via the Deployments page in Azure AI Foundry portal. An example endpoint is: https://docs-test-001.openai.azure.com/.
Key api_key The key value is also located under Keys and Endpoint for your resource in the Azure portal. Azure generates two keys for your resource. You can use either value.

Go to your resource in the Azure portal. On the navigation pane, select Keys and Endpoint under Resource Management. Copy the Endpoint value and an access key value. You can use either the KEY 1 or KEY 2 value. Having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Screenshot that shows the Keys and Endpoint page for an Azure OpenAI resource in the Azure portal.

Create a new Python application

Create a new Python file named quickstart.py. Open the new file in your preferred editor or IDE.

  1. Replace the contents of quickstart.py with the following code.

    # Packages required:
import requests 
import json 

api_base = '<your_azure_openai_endpoint>' 
deployment_name = '<your_deployment_name>'
API_KEY = '<your_azure_openai_key>'

base_url = f"{api_base}openai/deployments/{deployment_name}" 
headers = {   
    "Content-Type": "application/json",   
    "api-key": API_KEY 
} 

# Prepare endpoint, headers, and request body 
endpoint = f"{base_url}/chat/completions?api-version=2023-12-01-preview" 
data = { 
    "messages": [ 
        { "role": "system", "content": "You are a helpful assistant." }, 
        { "role": "user", "content": [  
            { 
                "type": "text", 
                "text": "Describe this picture:" 
            },
            { 
                "type": "image_url",
                "image_url": {
                    "url": "<image URL>"
                }
            }
        ] } 
    ], 
    "max_tokens": 2000 
}   

# Make the API call   
response = requests.post(endpoint, headers=headers, data=json.dumps(data))   

print(f"Status Code: {response.status_code}")   
print(response.text)

  2. Make the following changes:

    1. Enter your endpoint URL and key in the appropriate fields.
    2. Enter your GPT-4 Turbo with Vision deployment name in the appropriate field.
    3. Change the value of the "image" field to the URL of your image.

      Tip

      You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

  3. Run the application with the python command:

    python quickstart.py

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Use this article to get started using the Azure OpenAI Python SDK to deploy and use the GPT-4 Turbo with Vision model.

Library source code | Package (PyPi) |

Prerequisites

Set up

Install the OpenAI Python client library with:

pip install openai

Note

This library is maintained by OpenAI. Refer to the release history to track the latest updates to the library.

Retrieve key and endpoint

To successfully make a call against Azure OpenAI, you need an endpoint and a key.

Variable name Value
ENDPOINT The service endpoint can be found in the Keys & Endpoint section when examining your resource from the Azure portal. Alternatively, you can find the endpoint via the Deployments page in Azure AI Foundry portal. An example endpoint is: https://docs-test-001.openai.azure.com/.
API-KEY This value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.

Go to your resource in the Azure portal. The Keys & Endpoint section can be found in the Resource Management section. Copy your endpoint and access key as you'll need both for authenticating your API calls. You can use either KEY1 or KEY2. Always having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Screenshot of the overview UI for an Azure OpenAI resource in the Azure portal with the endpoint and access keys location circled in red.

Environment variables

Create and assign persistent environment variables for your key and endpoint.

Important

We recommend Microsoft Entra ID authentication with managed identities for Azure resources to avoid storing credentials with your applications that run in the cloud.

Use API keys with caution. Don't include the API key directly in your code, and never post it publicly. If using API keys, store them securely in Azure Key Vault, rotate the keys regularly, and restrict access to Azure Key Vault using role based access control and network access restrictions. For more information about using API keys securely in your apps, see API keys with Azure Key Vault.

For more information about AI services security, see Authenticate requests to Azure AI services.

setx AZURE_OPENAI_API_KEY "REPLACE_WITH_YOUR_KEY_VALUE_HERE" 
setx AZURE_OPENAI_ENDPOINT "REPLACE_WITH_YOUR_ENDPOINT_HERE"

Create a new Python application

Create a new Python file named quickstart.py. Open the new file in your preferred editor or IDE.

  1. Replace the contents of quickstart.py with the following code.

    from openai import AzureOpenAI

api_base = os.getenv("AZURE_OPENAI_ENDPOINT")
api_key= os.getenv("AZURE_OPENAI_API_KEY")
deployment_name = '<your_deployment_name>'
api_version = '2023-12-01-preview' # this might change in the future

client = AzureOpenAI(
    api_key=api_key,  
    api_version=api_version,
    base_url=f"{api_base}/openai/deployments/{deployment_name}"
)

response = client.chat.completions.create(
    model=deployment_name,
    messages=[
        { "role": "system", "content": "You are a helpful assistant." },
        { "role": "user", "content": [  
            { 
                "type": "text", 
                "text": "Describe this picture:" 
            },
            { 
                "type": "image_url",
                "image_url": {
                    "url": "<image URL>"
                }
            }
        ] } 
    ],
    max_tokens=2000 
)

print(response)

  2. Make the following changes:

    1. Enter the name of your GPT-4 Turbo with Vision deployment in the appropriate field.
    2. Change the value of the "url" field to the URL of your image.

      Tip

      You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

  3. Run the application with the python command:

    python quickstart.py

Important

Use API keys with caution. Don't include the API key directly in your code, and never post it publicly. If you use an API key, store it securely in Azure Key Vault. For more information about using API keys securely in your apps, see API keys with Azure Key Vault.

For more information about AI services security, see Authenticate requests to Azure AI services.

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Use this article to get started using the OpenAI JavaScript SDK to deploy and use the GPT-4 Turbo with Vision model.

This SDK is provided by OpenAI with Azure specific types provided by Azure.

Reference documentation | Library source code | Package (npm) | Samples

Prerequisites

Note

This library is maintained by OpenAI. Refer to the release history to track the latest updates to the library.

Microsoft Entra ID prerequisites

For the recommended keyless authentication with Microsoft Entra ID, you need to:

  • Install the Azure CLI used for keyless authentication with Microsoft Entra ID.
  • Assign the Cognitive Services User role to your user account. You can assign roles in the Azure portal under Access control (IAM) > Add role assignment.

Retrieve resource information

You need to retrieve the following information to authenticate your application with your Azure OpenAI resource:

Variable name Value
AZURE_OPENAI_ENDPOINT This value can be found in the Keys and Endpoint section when examining your resource from the Azure portal.
AZURE_OPENAI_DEPLOYMENT_NAME This value will correspond to the custom name you chose for your deployment when you deployed a model. This value can be found under Resource Management > Model Deployments in the Azure portal.
OPENAI_API_VERSION Learn more about API Versions.

Learn more about keyless authentication and setting environment variables.

Caution

To use the recommended keyless authentication with the SDK, make sure that the AZURE_OPENAI_API_KEY environment variable isn't set.

Create a Node application

In a console window (such as cmd, PowerShell, or Bash), create a new directory for your app, and navigate to it. Then run the npm init command to create a node application with a package.json file.

npm init

Install the client library

Install the client libraries with:

npm install openai @azure/identity

Your app's package.json file will be updated with the dependencies.

Create a new JavaScript application for image prompts

Select an image from the azure-samples/cognitive-services-sample-data-files and set the URL for an image in the environment variables.

  1. Replace the contents of quickstart.js with the following code.

    const AzureOpenAI = require('openai').AzureOpenAI;
const { 
    DefaultAzureCredential, 
    getBearerTokenProvider 
} = require('@azure/identity');

// You will need to set these environment variables or edit the following values
const endpoint = process.env["AZURE_OPENAI_ENDPOINT"] || "<endpoint>";
const imageUrl = process.env["IMAGE_URL"] || "<image url>";

// Required Azure OpenAI deployment name and API version
const apiVersion = "2024-07-01-preview";
const deploymentName = "gpt-4-with-turbo";

// keyless authentication    
const credential = new DefaultAzureCredential();
const scope = "https://cognitiveservices.azure.com/.default";
const azureADTokenProvider = getBearerTokenProvider(credential, scope);

function getClient(): AzureOpenAI {
  return new AzureOpenAI({
    endpoint,
    azureADTokenProvider,
    apiVersion,
    deployment: deploymentName,
  });
}
function createMessages() {
  return {
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "Describe this picture:",
          },
          {
            type: "image_url",
            image_url: {
              url: imageUrl,
            },
          },
        ],
      },
    ],
    model: "",
    max_tokens: 2000,
  };
}
async function printChoices(completion) {
  for (const choice of completion.choices) {
    console.log(choice.message);
  }
}
export async function main() {
  console.log("== Get GPT-4 Turbo with vision Sample ==");

  const client = getClient();
  const messages = createMessages();
  const completion = await client.chat.completions.create(messages);
  await printChoices(completion);
}

main().catch((err) => {
  console.error("Error occurred:", err);
});

  2. Make the following changes:

    1. Enter the name of your GPT-4 Turbo with Vision deployment in the appropriate field.
    2. Change the value of the "url" field to the URL of your image.

      Tip

      You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

  3. Run the application with the following command:

    node quickstart.js

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Use this article to get started using the OpenAI JavaScript SDK to deploy and use the GPT-4 Turbo with Vision model.

This SDK is provided by OpenAI with Azure specific types provided by Azure.

Reference documentation | Library source code | Package (npm) | Samples

Prerequisites

Note

This library is maintained by OpenAI. Refer to the release history to track the latest updates to the library.

Microsoft Entra ID prerequisites

For the recommended keyless authentication with Microsoft Entra ID, you need to:

  • Install the Azure CLI used for keyless authentication with Microsoft Entra ID.
  • Assign the Cognitive Services User role to your user account. You can assign roles in the Azure portal under Access control (IAM) > Add role assignment.

Retrieve resource information

You need to retrieve the following information to authenticate your application with your Azure OpenAI resource:

Variable name Value
AZURE_OPENAI_ENDPOINT This value can be found in the Keys and Endpoint section when examining your resource from the Azure portal.
AZURE_OPENAI_DEPLOYMENT_NAME This value will correspond to the custom name you chose for your deployment when you deployed a model. This value can be found under Resource Management > Model Deployments in the Azure portal.
OPENAI_API_VERSION Learn more about API Versions.

Learn more about keyless authentication and setting environment variables.

Caution

To use the recommended keyless authentication with the SDK, make sure that the AZURE_OPENAI_API_KEY environment variable isn't set.

Create a Node application

In a console window (such as cmd, PowerShell, or Bash), create a new directory for your app, and navigate to it. Then run the npm init command to create a node application with a package.json file.

npm init

Install the client library

Install the client libraries with:

npm install openai @azure/identity

Your app's package.json file will be updated with the dependencies.

Create a new JavaScript application for image prompts

Select an image from the azure-samples/cognitive-services-sample-data-files and set the URL for an image in the environment variables.

  1. Create a quickstart.ts and paste in the following code.

    import { AzureOpenAI } from "openai";
import { 
    DefaultAzureCredential, 
    getBearerTokenProvider 
} from "@azure/identity";
import type {
  ChatCompletion,
  ChatCompletionCreateParamsNonStreaming,
} from "openai/resources/index";

// You will need to set these environment variables or edit the following values
const endpoint = process.env["AZURE_OPENAI_ENDPOINT"] || "<endpoint>";
const imageUrl = process.env["IMAGE_URL"] || "<image url>";

// Required Azure OpenAI deployment name and API version
const apiVersion = "2024-07-01-preview";
const deploymentName = "gpt-4-with-turbo";

// keyless authentication    
const credential = new DefaultAzureCredential();
const scope = "https://cognitiveservices.azure.com/.default";
const azureADTokenProvider = getBearerTokenProvider(credential, scope);

function getClient(): AzureOpenAI {
  return new AzureOpenAI({
    endpoint,
    azureADTokenProvider,
    apiVersion,
    deployment: deploymentName,
  });
}
function createMessages(): ChatCompletionCreateParamsNonStreaming {
  return {
    messages: [
      { role: "system", content: "You are a helpful assistant." },
      {
        role: "user",
        content: [
          {
            type: "text",
            text: "Describe this picture:",
          },
          {
            type: "image_url",
            image_url: {
              url: imageUrl,
            },
          },
        ],
      },
    ],
    model: "",
    max_tokens: 2000,
  };
}
async function printChoices(completion: ChatCompletion): Promise<void> {
  for (const choice of completion.choices) {
    console.log(choice.message);
  }
}
export async function main() {
  console.log("== Get GPT-4 Turbo with vision Sample ==");

  const client = getClient();
  const messages = createMessages();
  const completion = await client.chat.completions.create(messages);
  await printChoices(completion);
}

main().catch((err) => {
  console.error("Error occurred:", err);
});

  2. Make the following changes:

    1. Enter the name of your GPT-4 Turbo with Vision deployment in the appropriate field.
    2. Change the value of the "url" field to the URL of your image.

      Tip

      You can also use a base 64 encoded image data instead of a URL. For more information, see the GPT-4 Turbo with Vision how-to guide.

  3. Build the application with the following command:

    tsc

  4. Run the application with the following command:

    node quickstart.js

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Use this article to get started using the Azure OpenAI .NET SDK to deploy and use the GPT-4 Turbo with Vision model.

Prerequisites

Set up

Retrieve key and endpoint

To successfully make a call against Azure OpenAI, you need an endpoint and a key.

Variable name Value
AZURE_OPENAI_ENDPOINT The service endpoint can be found in the Keys & Endpoint section when examining your resource from the Azure portal. Alternatively, you can find the endpoint via the Deployments page in Azure AI Foundry portal. An example endpoint is: https://docs-test-001.openai.azure.com/.
AZURE_OPENAI_API_KEY This value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.

Go to your resource in the Azure portal. The Endpoint and Keys can be found in the Resource Management section. Copy your endpoint and access key as you need both for authenticating your API calls. You can use either KEY1 or KEY2. Always having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Screenshot of the overview UI for an Azure OpenAI resource in the Azure portal with the endpoint & access keys location highlighted.

Create the .NET app

  1. Create a .NET app using the dotnet new command:

    dotnet new console -n OpenAISpeech

  2. Change into the directory of the new app:

    cd OpenAISpeech

Install the client library

Install the Azure.OpenAI client library:

dotnet add package Azure.AI.OpenAI

Passwordless authentication is more secure than key-based alternatives and is the recommended approach for connecting to Azure services. If you choose to use Passwordless authentication, you'll need to complete the following:

  1. Add the Azure.Identity package.

    dotnet add package Azure.Identity

  2. Assign the Cognitive Services User role to your user account. This can be done in the Azure portal on your OpenAI resource under Access control (IAM) > Add role assignment.

  3. Sign-in to Azure using Visual Studio or the Azure CLI via az login.

Update the app code

  1. Replace the contents of program.cs with the following code and update the placeholder values with your own.

    using Azure;
using Azure.AI.OpenAI;
using Azure.Identity;
using OpenAI.Chat; // Required for Passwordless auth

var endpoint = new Uri("YOUR_AZURE_OPENAI_ENDPOINT");
var credentials = new AzureKeyCredential("YOUR_AZURE_OPENAI_KEY");
// var credentials = new DefaultAzureCredential(); // Use this line for Passwordless auth
var deploymentName = "gpt-4"; // Default name, update with your own if needed

var openAIClient = new AzureOpenAIClient(endpoint, credentials);
var chatClient = openAIClient.GetChatClient(deploymentName);

var imageUri = "YOUR_IMAGE_URL";

List<ChatMessage> messages = [
    new UserChatMessage(
        ChatMessageContentPart.CreateTextMessageContentPart("Please describe the following image:"),
        ChatMessageContentPart.CreateImageMessageContentPart(new Uri(imageUri), "image/png"))
];

ChatCompletion chatCompletion = await chatClient.CompleteChatAsync(messages);

Console.WriteLine($"[ASSISTANT]:");
Console.WriteLine($"{chatCompletion.Content[0].Text}");

Important

We recommend Microsoft Entra ID authentication with managed identities for Azure resources to avoid storing credentials with your applications that run in the cloud.

Use API keys with caution. Don't include the API key directly in your code, and never post it publicly. If using API keys, store them securely in Azure Key Vault, rotate the keys regularly, and restrict access to Azure Key Vault using role based access control and network access restrictions. For more information about using API keys securely in your apps, see API keys with Azure Key Vault.

For more information about AI services security, see Authenticate requests to Azure AI services.

  1. Run the application using the dotnet run command or the run button at the top of Visual Studio:

    dotnet run

The app generates an audio file at the location you specified for the speechFilePath variable. Play the file on your device to hear the generated audio.

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

Next steps