Azure OpenAI on your data with images using GPT-4 Turbo with Vision (preview)

Note

Azure AI Studio is currently in public preview. This preview is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

Use this article to learn how to provide your own image data for GPT-4 Turbo with Vision, Azure OpenAI's vision model. GPT-4 Turbo with Vision on your data allows the model to generate more customized and targeted answers using retrieval augmented generation based on your own images and image metadata.

Tip

This article is for using your data on the GPT-4 Turbo with Vision model. See Deploy a web app for chat on your data for a tutorial on how to deploy a chat web app using your text data.

Prerequisites

• An Azure subscription - Create one for free.

• Access granted to Azure OpenAI in the desired Azure subscription.

  Currently, access to this service is granted only by application. You can apply for access to Azure OpenAI by completing the form at https://aka.ms/oai/access. Open an issue on this repo to contact us if you have an issue.

• An Azure OpenAI resource with the GPT-4 Turbo with Vision model deployed. For more information about model deployment, see the resource deployment guide.

• Be sure that you're assigned at least the Cognitive Services Contributor role for the Azure OpenAI resource.

• An Azure AI Search resource. See create an Azure AI Search service in the portal. If you don't have an Azure AI Search resource, you are prompted to create one when you add your data source later in this guide.

Start a playground session

This guide is scoped to the Azure AI Studio playground, but you can also add image data via your project's Data page. See Add data to your project for more information.

1. Sign in to Azure AI Studio.

2. Go to your project or create a new project in Azure AI Studio.

3. If you aren't already in the playground, select Build from the top menu and then select Playground from the collapsible left menu.

4. In the playground, make sure that Chat is selected from the Mode dropdown. Select your deployed GPT-4 Turbo with Vision model from the Deployment dropdown.

   

5. On the Assistant setup page, select Add your data > + Add a data source.

   

6. In the Select or add data source page, select a data source from the Select data source dropdown. See the next section in this guide for more information about each option.

Add your image data source

From the Azure AI Studio playground, you can choose how to add your image data for GPT-4 Turbo with Vision:

• Upload image files and metadata: You can upload image files and metadata in the playground. This option is useful if you have a small number of image files.
• Azure AI Search: If you have an existing Azure AI Search index, you can use it as a data source.
• Azure Blob Storage: The Azure Blob storage option is especially useful if you have a large number of image files and don't want to manually upload each one.

Each option uses an Azure AI Search index to do image-to-image search and retrieve the top search results for your input prompt image.

• When you upload files in the playground or when you use Azure Blob storage, Azure AI Studio will generate an image search index for you.
• For Azure AI Search, you need to have an image search index.

Upload image files and metadata

1. Start a playground session and select Add your data > + Add a data source, as described in the previous section.

2. In the Select or add data source page, select Upload files from the Select data source dropdown.

3. Enter your data source details:

   

   Note

   Azure OpenAI needs both a storage account resource and a search resource to access and index your data. Your data is stored securely in your Azure subscription.

   When adding data to the selected storage account for the first time in Azure AI Studio, you might be prompted to turn on cross-origin resource sharing (CORS). Azure AI Studio and Azure OpenAI need access your Azure Blob storage account.

   • Subscription: Select the Azure subscription that contains the Azure OpenAI resource you want to use.
   • Storage resource: Select the Azure Blob storage resource where you want to upload your files.
   • Azure AI Search resource: Select the Azure AI Search resource where your images will be indexed.
   • Index name: Enter the index name that will be used to reference this data source. A new image search index with the provided name (and -v suffix) is generated after data ingestion is complete.

4. Select the boxes to acknowledge that deployments and connections incur usage on your account. Then select Next.

5. On the Upload files page, select Browse for a file and select the files you want to upload. If you want to upload more than one file, do so now. You won't be able to add more files later in the same playground session.

   The following file types are supported for your image files up to 16 MB in size:

   • .jpg
   • .png
   • .gif
   • .bmp
   • .tiff

6. Select Upload to upload the files to your Azure Blob storage account. Then select Next.

   

7. On the Add metadata page, for each image file enter the metadata in the provided description fields. Then select Next.

   

8. Review the details you entered. You can see the names of the storage container and search index that will be created for you.

   

9. Select Save and close.

10. Now on the Assistant setup page, you can see that your data ingestion is in progress. Before proceeding, wait until you see the data source and index name in place of the status.

    

11. You can now chat with the model asking questions such as "What tent resembles this picture?".

    

Azure AI Search

If you have an existing Azure AI Search index, you can use it as a data source.

If you don't already have a search index created for your images:

• You can create one using the AI Search vector search repository on GitHub, which provides you with scripts to create an index with your image files.
• You can upload image files in the playground and then select the corresponding image search index that was created for you.

1. Start a playground session and select Add your data > + Add a data source, as described in the previous section.

2. In the Select or add data source page, select Azure AI Search from the Select data source dropdown.

3. Enter your data source details:

   

   • Subscription: Select the Azure subscription that contains the Azure OpenAI resource you want to use.
   • Azure AI Search service: Select your Azure AI Search service resource that has an image search index.
   • Azure AI Search index: Select the image search index to be used for data grounding.

4. Select the boxes to acknowledge that deployments and connections incur usage on your account. Then select Next.

5. Review the details you entered.

   

6. Select Save and close.

Azure Blob storage

If you have an existing Azure Blob Storage container, you can use it to create an image search index. If you want to create a new blob storage account, see the Azure Blob storage quickstart documentation.

Before proceeding with this guide, your Azure Blob storage account must contain image files and a JSON file with the image file paths and metadata.

Your metadata JSON file must:

• Have a file name that starts with the word metadata, all in lowercase without a space.
• Have a maximum of 10,000 image files. If you have more than this number of files in your container, you can have multiple JSON files each with up to this maximum.

Here's an example of a metadata JSON file:

[
    {
        "image_blob_path": "image1.jpg",
        "description": "description of image1"
    },
    {
        "image_blob_path": "image2.jpg",
        "description": "description of image2"
    },
    ...
    {
        "image_blob_path": "image50.jpg",
        "description": "description of image50"
    }
]

After you have a blob storage populated with image files and at least one metadata JSON file, you are ready to add the blob storage as a data source.

1. Start a playground session and select Add your data > + Add a data source, as described in the previous section.

2. In the Select or add data source page, select Azure Blob Storage from the Select data source dropdown.

3. Enter your data source details:

   Note

   Azure OpenAI needs both a storage account resource and a search resource to access and index your data. Your data is stored securely in your Azure subscription.

   When adding data to the selected storage account for the first time in Azure AI Studio, you might be prompted to turn on cross-origin resource sharing (CORS). Azure AI Studio and Azure OpenAI need access your Azure Blob storage account.

   

   • Subscription: Select the Azure subscription that contains the Azure OpenAI resource you want to use.
   • Storage resource and Storage container: Select the Azure Blob storage resource where the image files and metadata are already stored.
   • Azure AI Search resource: Select the Azure AI Search resource where your image and JSON files in the selected Blob storage container will be indexed.
   • Index name: A new image search index with the provided name (and -v suffix) is generated after data ingestion is complete.

4. Select the boxes to acknowledge that deployments and connections incur usage on your account. Then select Next.

5. Review the details you entered.

   

6. Select Save and close.

7. Now on the Assistant setup page, you can see that your data ingestion is in progress. Before proceeding, wait until you see the data source and index name in place of the status.

Using your ingested data with your GPT-4 Turbo with Vision model

After you add your image data as described in the previous section, you can chat with the model that's grounded on your image data.

1. Upload an image and ask a question such as "What tent resembles this picture?".

   

2. The model will respond with an answer that's grounded on your image data.

   

Additional tips

Adding and removing data sources

Azure OpenAI currently allows only one data source to be used per a chat session. If you would like to add a new data source, you must remove the existing data source first. This can be done by selecting Remove data source under your data source information.

When you remove a data source, you'll see a warning message. Removing a data source clears the chat session and resets all playground settings.

Next steps

• Learn how to create a project in Azure AI Studio.
• Deploy a web app for chat on your data