AI Vision Vectorization in Promptflow

Question

AI Vision Vectorization in Promptflow

Seah Bryan 40

Hi, I recently created a search index consisting of images, vectorized with the Azure AI Vision during the index creation. I now want to access the index in promptflow via the index lookup tool, but I require to use the same AI vision model for the embedding of the query in the tool. Does anyone know how I can access the AI Vision vectorizatin model and use it in promptflow's index lookup tool? Thanks!

santoshkc 15,245 Reputation points Microsoft External Staff Moderator

2024-10-16T10:56:09.07+00:00

Hi @Seah Bryan,

Thank you for reaching out to Microsoft Q&A forum!

To effectively access your image search index in Promptflow, it’s essential to use the same AI Vision model for embedding your query images that you used during the index creation. This is because multimodal embedding converts images into vector representations that capture their features, and using different models can lead to compatibility issues.

When you vectorize an image with Azure AI Vision, it creates a specific vector embedding that allows for semantic similarity searches. Since vector search finds matches based on these embeddings, it’s crucial to ensure that the query image is embedded using the same model as the indexed images. This approach will enhance the accuracy and relevance of your search results in Promptflow, as it allows for precise comparisons between the vectors.

If you haven't already, make sure to integrate the Azure AI Vision API within Promptflow to generate these embeddings for your queries. This way, you'll maintain consistency across both the indexed images and your search queries, ensuring optimal performance in your image retrieval process.

For more info: Multimodal embeddings
Also refer to: Image retrieval using multimodal embeddings.

I hope this help. Thank you.
santoshkc 15,245 Reputation points Microsoft External Staff Moderator

2024-10-17T17:10:48.67+00:00

Hi @Seah Bryan,

Following up to see if the given response was helpful. Thank you.

Seah Bryan 40

Hi @santoshkc ,

Thank you for your response, and my apologies for the late reply. While I do understand the concept of using the same AI Vision embedding model to query against the search index, my question was pertaining to how exactly to access this model in Promptflow.

So far, I have found a way to access it by creating a Python node and using the SearchClient and VectorizableTextQuery classes to achieve, but it returns an error stating that AI Vision models are not supported in the current api.

This is my code:

from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizableTextQuery
from azure.core.credentials import AzureKeyCredential

search_service = "<search_service_url>"
index_name = "<vector name>"
credentials = AzureKeyCredential(<search_index_key>)

search_client = SearchClient(endpoint=search_service, index_name=index_name, credential=credentials)
text_vector_query = VectorizableTextQuery(text="Stealing", fields="image_vector")

results = search_client.search(vector_queries=[text_vector_query], top=3)
for result in results:
  print(result)

This is the error:

HttpResponseError: (FeatureNotSupportedInApiVersion) Vectorization of queries against fields using the 'aiServicesVision' vectorizer kind is not supported in the current api version.

santoshkc 15,245 Reputation points Microsoft External Staff Moderator

2024-10-19T15:24:26.0533333+00:00

Hi @Seah Bryan,

The error suggests that the AI Vision model is not supported in the current API version you're using. Ensure you're using the latest Azure Cognitive Search API version that supports AI Vision model vectorization. If not supported natively in Promptflow, a workaround is to manually use the AI Vision API to generate image embeddings outside of the search process. You can then pass these vectors directly to the SearchClient for querying.

I hope you understand! Thank you.

Your answer

santoshkc 15,245 Reputation points Microsoft External Staff Moderator

2024-10-16T10:56:09.07+00:00

Hi @Seah Bryan,

Thank you for reaching out to Microsoft Q&A forum!

To effectively access your image search index in Promptflow, it’s essential to use the same AI Vision model for embedding your query images that you used during the index creation. This is because multimodal embedding converts images into vector representations that capture their features, and using different models can lead to compatibility issues.

When you vectorize an image with Azure AI Vision, it creates a specific vector embedding that allows for semantic similarity searches. Since vector search finds matches based on these embeddings, it’s crucial to ensure that the query image is embedded using the same model as the indexed images. This approach will enhance the accuracy and relevance of your search results in Promptflow, as it allows for precise comparisons between the vectors.

If you haven't already, make sure to integrate the Azure AI Vision API within Promptflow to generate these embeddings for your queries. This way, you'll maintain consistency across both the indexed images and your search queries, ensuring optimal performance in your image retrieval process.

For more info: Multimodal embeddings
Also refer to: Image retrieval using multimodal embeddings.

I hope this help. Thank you.
santoshkc 15,245 Reputation points Microsoft External Staff Moderator

2024-10-17T17:10:48.67+00:00

Hi @Seah Bryan,

Following up to see if the given response was helpful. Thank you.
Seah Bryan 40 Reputation points

2024-10-18T10:01:15.66+00:00

Hi @santoshkc ,

Thank you for your response, and my apologies for the late reply. While I do understand the concept of using the same AI Vision embedding model to query against the search index, my question was pertaining to how exactly to access this model in Promptflow.

So far, I have found a way to access it by creating a Python node and using the SearchClient and VectorizableTextQuery classes to achieve, but it returns an error stating that AI Vision models are not supported in the current api.

This is my code:

from azure.search.documents import SearchClient from azure.search.documents.models import VectorizableTextQuery from azure.core.credentials import AzureKeyCredential search_service = "<search_service_url>" index_name = "<vector name>" credentials = AzureKeyCredential(<search_index_key>) search_client = SearchClient(endpoint=search_service, index_name=index_name, credential=credentials) text_vector_query = VectorizableTextQuery(text="Stealing", fields="image_vector") results = search_client.search(vector_queries=[text_vector_query], top=3) for result in results: print(result)

This is the error:

HttpResponseError: (FeatureNotSupportedInApiVersion) Vectorization of queries against fields using the 'aiServicesVision' vectorizer kind is not supported in the current api version.
santoshkc 15,245 Reputation points Microsoft External Staff Moderator

2024-10-19T15:24:26.0533333+00:00

Hi @Seah Bryan,

The error suggests that the AI Vision model is not supported in the current API version you're using. Ensure you're using the latest Azure Cognitive Search API version that supports AI Vision model vectorization. If not supported natively in Promptflow, a workaround is to manually use the AI Vision API to generate image embeddings outside of the search process. You can then pass these vectors directly to the SearchClient for querying.

I hope you understand! Thank you.

Share via

AI Vision Vectorization in Promptflow

Your answer