Quickstart: Get started using GPT-4 Turbo with Vision on your images and videos in Azure AI Studio

Note

Azure AI Studio is currently in public preview. This preview is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

Use this article to get started using Azure AI Studio to deploy and test the GPT-4 Turbo with Vision model.

GPT-4 Turbo with Vision and Azure AI Vision offer advanced functionality including:

• Optical Character Recognition (OCR): Extracts text from images and combines it with the user's prompt and image to expand the context.
• Object grounding: Complements the GPT-4 Turbo with Vision text response with object grounding and outlines salient objects in the input images.
• Video prompts: GPT-4 Turbo with Vision can answer questions by retrieving the video frames most relevant to the user's prompt.

Extra usage fees might apply for using GPT-4 Turbo with Vision and Azure AI Vision functionality.

Prerequisites

Note

This feature isn't available if you created an Azure AI hub resource together with an existing Azure OpenAI Service resource. You must create an AI hub with an Azure AI services provider. We're gradually rolling out this feature to all customers. If you don't see it yet, check back later.

• An Azure subscription - Create one for free.

• Access granted to Azure OpenAI in the desired Azure subscription.

  Currently, access to this service is granted only by application. You can apply for access to Azure OpenAI by completing the form at https://aka.ms/oai/access. Open an issue on this repo to contact us if you have an issue.

• An Azure AI hub resource with a GPT-4 Turbo with Vision model deployed in one of the regions that support GPT-4 Turbo with Vision. When you deploy from your Azure AI project's Deployments page, select: gpt-4 as the model name and vision-preview as the model version.

• An Azure AI project in Azure AI Studio.

Start a chat session to analyze images or video

You need an image to complete the image quickstarts. You can use the following image or any other image you have available.

You need a video up to three minutes in length to complete the video quickstart.

Image prompts

In this chat session, you instruct the assistant to aid in understanding images that you input.

1. Sign in to Azure AI Studio.

2. Go to your project or create a new project in Azure AI Studio.

3. Select Build from the top menu and then select Playground from the collapsible left menu.

4. Make sure that Chat is selected from the Mode dropdown. Select your deployed GPT-4 Turbo with Vision model from the Deployment dropdown. Under the chat session text box, you should now see the option to select a file.

   

5. In the System message text box on the Assistant setup pane, provide this prompt to guide the assistant: "You're an AI assistant that helps people find information." You can tailor the prompt the image or scenario that you're uploading.

6. Select Apply changes to save your changes, and when prompted to see if you want to update the system message, select Continue.

7. In the chat session pane, select an image file and then select the right arrow icon to upload the image.

   

8. Enter enter the following question: "Describe this image", and then select the right arrow icon to send.

   

9. The square icon replaces the right arrow icon. If you select the square icon, the assistant stops processing your request. For this quickstart, let the assistant finish its reply. Don't select the square icon.

   

10. The assistant should reply with a description of the image.

    

11. Ask a follow-up question related to the analysis of your image. Enter "What should I highlight about this image to my insurance company" and then select the right arrow icon to send.

12. You should receive a relevant response similar to what's shown here:

    

Image prompt enhancements

In this chat session, you instruct the assistant to aid in understanding images that you input. Try out the capabilities of the augmented vision model.

1. Sign in to Azure AI Studio.

2. Select Build from the top menu and then select Playground from the collapsible left menu.

3. Select your deployed GPT-4 Turbo with Vision model from the Deployment dropdown.

4. In the Configuration pane on the right side of the chat experience, turn on the option for Vision under Enhancements.

   

5. Make sure that Chat is selected from the Mode dropdown. Under the chat session text box, you should see the option to select a file.

   

6. In the System message text box on the Assistant setup pane, provide this prompt to guide the assistant: "You're an AI assistant that helps people find information." You can tailor the prompt the image or scenario that you're uploading.

7. Select Apply changes to save your changes, and when prompted to see if you want to update the system message, select Continue.

8. In the chat session pane, select an image file and then select the right arrow icon to upload the image.

9. Enter enter the following question: "Describe this image", and then select the right arrow icon to send.

   

10. The square icon replaces the right arrow icon. If you select the square icon, the assistant stops processing your request. For this quickstart, let the assistant finish its reply. Don't select the square icon.

    

11. The assistant should reply with a description of the image with objects highlighted both in the text and in the image.

    

12. Ask a follow-up question related to the analysis of your image. Enter "What should I highlight about this image to my insurance company" and then select the right arrow icon to send.

13. You should receive a relevant response similar to what's shown here:

    

Video prompt enhancements

In this chat session, you'll be instructing the assistant to aid in understanding videos that you input. The assistant extracts several frames from the video and uses them to answer your questions.

1. Sign in to Azure AI Studio.

2. Select Build from the top menu and then select Playground from the collapsible left menu.

3. Select your deployed GPT-4 Turbo with Vision model from the Deployment dropdown.

4. In the Configuration pane on the right side of the chat experience, turn on the option for Vision under Enhancements.

   

5. Make sure that Chat is selected from the Mode dropdown. Under the chat session text box, you should now see the option to select a video file.

   

6. In the System message text box on the Assistant setup pane, provide this prompt to guide the assistant: "You're a car insurance and accident expert. Extract detailed information about the car's make, model, damage extent, license plate, airbag deployment status, mileage, and any other observations" You can tailor the prompt for the video or scenario.

7. Select Apply changes to save your changes, and when prompted to see if you want to update the system message, select Continue.

8. In the chat session pane, select a video file and then select the right arrow icon to upload the video.

9. Enter enter the following question: "Provide details from this car accident video", and then select the right arrow icon to send.

   

10. The square icon replaces the right arrow icon. If you select the square icon, the assistant stops processing your request. For this quickstart, let the assistant finish its reply. Don't select the square icon.

    

11. The assistant should reply with a description of the video.

    

12. Ask a follow-up question related to the analysis of your video. Enter "What should I highlight about this video to my insurance company" and then select the right arrow icon to send.

13. You should receive a relevant response similar to what's shown here:

    

Below are the known limitations of the video prompt enhancements.

• Low resolution: The frames are analyzed using GPT-4 Turbo with Vision's "low resolution" setting, which may affect the accuracy of small object and text recognition in the video.
• Video file limits: Both mp4 and mov file types are supported. In the Azure AI Playground, videos must be less than 3 minutes long. When using the API there is no such limitation.
• Prompt limits: Video prompts only contain one video and no images. In Playground, you can clear the session to try with another video or images.
• Limited frame selection: Currently the system selects 20 frames from the entire video, which might not capture all critical moments or details. Frame selection can either be approximately evenly spread through the video or focused by a specific Video Retrieval query, depending on the prompt.
• Language support: Currently, the system primarily supports English for grounding with transcripts. Transcripts don't provide accurate information on lyrics from songs.

At any point in the chat session, you can select the Show raw JSON option to see the conversation formatted as JSON. Heres' what it looks like at the beginning of the quickstart chat session:

[
	{
		"role": "system",
		"content": [
			"You are an AI assistant that helps people find information."
		]
	},
]

This has been a walkthrough of GPT-4 Turbo with Vision in the Azure AI Studio chat playground experience.

Clean up resources

To avoid incurring unnecessary Azure costs, you should delete the resources you created in this quickstart if they're no longer needed. To manage resources, you can use the Azure portal.

Next steps

• Create an Azure AI project
• Learn more about Azure AI Vision.
• Learn more about Azure OpenAI models.