Azure OpenAI enterprise chat with images using GPT-4 Turbo with Vision
Important
Some of the features described in this article might only be available in preview. This preview is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.
Use this article to learn how to provide your own image data for GPT-4 Turbo with Vision, Azure OpenAI's vision model. GPT-4 Turbo with Vision enterprise chat allows the model to generate more customized and targeted answers using retrieval augmented generation based on your own images and image metadata.
Tip
This article is for using your image data on the GPT-4 Turbo with Vision model. See Deploy an enterprise chat web app for a tutorial on how to deploy a chat web app using your text data.
Prerequisites
- An Azure subscription - Create one for free.
- An Azure OpenAI resource with the GPT-4 Turbo with Vision model deployed. For more information about model deployment, see the resource deployment guide.
- Be sure that you're assigned at least the Cognitive Services Contributor role for the Azure OpenAI resource.
- An Azure AI Search resource. See create an Azure AI Search service in the portal. If you don't have an Azure AI Search resource, you're prompted to create one when you add your data source later in this guide.
- An AI Studio hub with your Azure OpenAI resource and Azure AI Search resource added as connections.
Deploy a GPT-4 Turbo with Vision model
- Sign in to Azure AI Studio and select the hub you'd like to work in.
- On the left nav menu, select AI Services. Select the Try out GPT-4 Turbo panel.
- On the gpt-4 page, select Deploy. In the window that appears, select your Azure OpenAI resource. Select
vision-preview
as the model version. - Select Deploy.
- Next, go to your new model's page and select Open in playground. In the chat playground, the GPT-4 deployment you created should be selected in the Deployment dropdown.
Select your image data source
- On the left pane, select the Add your data tab and select Add a data source.
- In the window that appears, select a data source option. Each option uses an Azure AI Search index that's trained on your images and can be used for retrieval augmented generation in the chat playground.
- Azure AI Search: If you have an existing Azure AI Search index, you can use it as a data source.
- Azure Blob Storage: The Azure Blob storage option is especially useful if you have a large number of image files and don't want to manually upload each one. Azure AI Studio will generate an image search index for you.
- Upload image files and metadata: You can upload image files and metadata using the playground. This option is useful if you have a small number of image files. Azure AI Studio will generate an image search index for you.
Add your image data
If you have an existing Azure AI Search index, you can use it as a data source. If you don't already have a search index but you'd like to create one on your own, follow the AI Search vector search repository on GitHub, which provides scripts to create an index with your image files.
- In the playground's Select or add data source window, choose Azure AI Search and enter your index's details. Select the boxes to acknowledge that deployments and connections incur usage on your account.
- Optionally enable the Use custom field mapping option. This lets you control the mapping between the custom fields in your search index and the standard fields that Azure OpenAI chat models use during retrieval augmented generation.
- Select Next and review your settings on the next page. Then select Save and close.
- In the chat playground, you can see that your data has been added.
Use your data with your GPT-4 Turbo with Vision model
After you add your image data, you can try out a chat conversation that's grounded on your image data.
Use the attachment button in the chat window to upload a new image. Ask a question about its relationship to the other images in your data set.
The model will respond with an answer that's grounded on your image data.
Add and remove data sources
Azure OpenAI only allows one data source to be used per a chat session. If you want to add a new data source, you must remove the existing data source first. Do this by selecting Remove data source under your data source information.
When you remove a data source, you'll see a warning message. Removing a data source clears the chat session and resets all playground settings.
Next steps
Обратна връзка
https://aka.ms/ContentUserFeedback.
Очаквайте скоро: През цялата 2024 г. постепенно ще отстраняваме проблемите в GitHub като механизъм за обратна връзка за съдържание и ще го заменим с нова система за обратна връзка. За повече информация вижте:Подаване и преглед на обратна връзка за