Azure OpenAI on your data with images using GPT-4 Turbo with Vision (preview) in Azure OpenAI studio

Use this article to learn how to provide your own image data for GPT-4 Turbo with Vision, Azure OpenAI’s vision model. GPT-4 Turbo with Vision on your data allows the model to generate more customized and targeted answers using Retrieval Augmented Generation based on your own images and image metadata.

Important

Once the GPT4-Turbo with vision preview model is deprecated, you will no longer be able to use Azure OpenAI On your image data. To implement a Retrieval Augmented Generation (RAG) solution with image data, see the following sample on github.

Prerequisites

• An Azure subscription - Create one for free.

• Access granted to Azure OpenAI in the desired Azure subscription.

  Currently, access to this service is granted only by application. You can apply for access to Azure OpenAI by completing the form at https://aka.ms/oai/access. Open an issue on this repo to contact us if you have an issue.

• An Azure OpenAI resource with the GPT-4 Turbo with Vision model deployed. For more information about model deployment, see the resource deployment guide.

• Be sure that you're assigned at least the Cognitive Services Contributor role for the Azure OpenAI resource.

Add your data source

Navigate to Azure OpenAI Studio and sign-in with credentials that have access to your Azure OpenAI resource. During or after the sign-in workflow, select the appropriate directory, Azure subscription, and Azure OpenAI resource.

On the Assistant setup tile, select Add your data (preview) > + Add a data source.

In the pane that appears after you select Add a data source, you'll see multiple options to select a data source.

You have three different options to add your data for GPT-4 Turbo with Vision’s data source:

• Using your own image files and image metadata
• Using Azure AI Search
• Using Azure Blob Storage

All three options use Azure AI Search index to do image-to-image search and retrieve the top search results for your input prompt image. For Azure Blob Storage and Upload files options, Azure OpenAI will generate an image search index for you. For Azure AI Search, you need to have an image search index. The following sections contain details on how to create the search index.

When using these options for the first time, you might see this red notice asking you to turn on Cross-origin resource sharing (CORS). This is a notice asking you to enable CORS, so that Azure OpenAI can access your blob storage account. To fix the warning, select Turn on CORS.

Add your data by uploading files

You can manually upload your image files and enter metadata of them manually, using Azure OpenAI. This is especially useful if you are experimenting with a small set of images and would like to build your data source.

1. Navigate to the Select a data source button in Azure OpenAI as described above. Select Upload files.

2. Select your subscription. Select an Azure Blob storage to which your uploaded image files will be stored to. Select an Azure AI Search resource in which your new image search index will be created. Enter the image search index name of your choice.

   Once you have filled out all the fields, check the two boxes at the bottom acknowledging the incurring usage, and select Next.

   

   The following file types are supported for your image files:

   • .jpg
   • .png
   • .gif
   • .bmp
   • .tiff

3. Select Browse for a file to select image files you would like to use from your local directory.

4. Once you select your image files, you'll see the image files selected in the right table. Select Upload files. Once you have uploaded the files, you'll see the status for each is Uploaded. Select Next.

   

5. For each image file, enter the metadata in the provided description fields. Once you have descriptions for each image, select Next.

   

6. Review that all the information is correct. Select Save and close.

Add your data using Azure AI Search

If you have an existing Azure AI search index, you can use it as a data source. If you don't already have a search index created for your images, you can create one using the AI Search vector search repository on GitHub, which provides you with scripts to create an index with your image files. This option is also great if you would like to create your data source using your own files like the option above, and then come back to the playground experience to select that data source you already have created but have not added yet.

1. Navigate to the Select a data source button in Azure OpenAI as described above. Select Azure AI Search.

   Tip

   You can select an image search index that you have created with the Azure Blob Storage or Upload files options.

2. Select your subscription, and the Azure AI Search service you used to create the image search index.

3. Select your Azure AI Search index you have created with your images.

4. After you have filled in all fields, select the two checkboxes at the bottom asking you to acknowledge the charges incurred from using GPT-4 Turbo with Vision vector embeddings and Azure AI Search. Select Next. If CORS isn't already turned on for the AI Search resource, you will see a warning. To fix the warning, select Turn on CORS.

   

5. Review the details, then select Save and close.

Add your data using Azure Blob Storage

If you have an existing Azure Blob Storage container, you can use it to create an image search index. If you want to create a new blob storage, see the Azure Blob storage quickstart documentation.

Your blob storage should contain image files and a JSON file with the image file paths and metadata. This option is especially useful if you have a large number of image files and don't want to manually upload each one.

If you don't already have a blob storage populated with these files, and would like to upload files one by one, you can upload your files using Azure OpenAI studio instead.

Before you start adding your Azure Blob Storage container as your data source, make sure your blob storage contains all the images that you would like to ingest, and a JSON file that contains the image file paths and metadata.

Important

Your metadata JSON file must:

• Have a file name that starts with the word “metadata”, all in lowercase without a space.
• Have a maximum of 10,000 image files. If you have more than this number of files in your container, you can have multiple JSON files each with up to this maximum.

[
    {
        "image_blob_path": "image1.jpg",
        "description": "description of image1"
    },
    {
        "image_blob_path": "image2.jpg",
        "description": "description of image2"
    },
    ...
    {
        "image_blob_path": "image50.jpg",
        "description": "description of image50"
    }
]

After you have a blob storage populated with image files and at least one metadata JSON file, you are ready to add the blob storage as a data source.

1. Navigate to the Select a data source button in Azure OpenAI as described above. Select Azure Blob Storage.

2. Select your subscription, Azure Blob storage, and storage container. You'll also need to select an Azure AI Search resource, as a new image search index will be created in this resource group. If you don't have an Azure AI Search resource, you can create a new one using the link below the dropdown. If CORS isn't already turned on for the Azure Blob storage resource, you will see a warning. To fix the warning, select Turn on CORS.

3. Once you've selected an Azure AI search resource, enter a name for the search index in the Index name field.

   Note

   The name of the index will be suffixed with –v, to indicate that this is an index with image vectors extracted from the images provided. The description filed in the metadata.json will be added as text metadata in the index.

4. After you've filled in all fields, select the two checkboxes at the bottom asking you to acknowledge the charges incurred from using GPT-4 Turbo with Vision vector embeddings and Azure AI Search. Select Next.

   

5. Review the details, then select Save and close.

Using your ingested data with your GPT-4 Turbo with Vision model

After you connect your data source using any of the three methods listed above, It will take some time for the data ingestion process to finish. You will see an icon and a Ingestion in progress message as the process progresses. Once the ingestion has been completed, you'll see that a data source has been created.

Once the data source has finished being ingested, you will see your data source details as well as the image search index name. Now this ingested data is ready to be used as the grounding data for your deployed GPT-4 Turbo with Vision model. Your model will use the top retrieval data from your image search index and generate a response specifically adhered to your ingested data.

Turn on CORS

If CORS isn't already turned on for your data source, you will see the following message appear.

If you see this message, select Turn on CORS when you connect your data source.

Additional Tips

Adding and Removing Data Sources

Azure OpenAI currently allows only one data source to be used per a chat session. If you would like to add a new data source, you must remove the existing data source first. This can be done by selecting Remove data source under your data source information.

When you remove a data source, you'll see a warning message. Removing a data source clears the chat session and resets all playground settings.

Important

If you switch to a model deployment which is not using the GPT-4 Turbo with Vision model, you will see a warning message for removing a data source. Please note that removing a data source will clear the chat session and reset all playground settings.

Next steps

• You can also chat on Azure OpenAI text models. See Use your text data for more information.
• Or, use GPT-4 Turbo with Vision in a chat scenario by following the quickstart.
• GPT-4 Turbo with Vision frequently asked questions
• GPT-4 Turbo with Vision API reference