Document Indexing and Vector Embeddings for SharePoint files using Azure Cognitive Search

Question

Document Indexing and Vector Embeddings for SharePoint files using Azure Cognitive Search

Sarva, Pavan 45

Please let me know if SharePoint Indexer for Azure Cognitive Search includes the ability to index documents from SharePoint libraries and generate vector embeddings. If not, what is the recommended approach to produce embedding for documents in the SP library and persist them in Oder to execute vector search using Azure Cognitive Search?

1 answer

Your answer

Answer 1

brtrach-MSFT 17,741 Microsoft Employee Moderator

@Sarva, Pavan Yes, the SharePoint indexer in Azure Cognitive Search can index documents from SharePoint libraries. However, it does not generate vector embeddings by default.

To generate vector embeddings for documents in the SharePoint library and persist them for vector search using Azure Cognitive Search, you can use Azure OpenAI Service. You can use the OpenAI Service to generate embeddings for each document in the SharePoint library and store them in a separate field in the search index. Once the embeddings are stored in the search index, you can use Azure Cognitive Search to execute vector search queries against the embeddings.

Here is a high-level overview of the steps you can follow:

Set up a SharePoint indexer in Azure Cognitive Search to index documents from the SharePoint library.
Use Azure OpenAI Service to generate embeddings for each document in the SharePoint library.
Store the embeddings in a separate field in the search index.
Use Azure Cognitive Search to execute vector search queries against the embeddings.

Sarva, Pavan 45 Reputation points

2023-10-18T08:00:14.01+00:00

Thank you for your response. Please offer further information whether SharePoint indexer extension or customisation is possible. This is considering options such as executing an Azure Function upon the completion of a scheduled indexer run in order to update Fields for embeddings (steps 2, 3, and 4).
brtrach-MSFT 17,741 Reputation points Microsoft Employee Moderator

2023-10-20T21:47:52.1066667+00:00
@Sarva, Pavan Yes, it is possible to extend or customize the SharePoint indexer in Azure Cognitive Search to execute an Azure Function upon the completion of a scheduled indexer run.

You can use the Azure Function to generate embeddings for each document in the SharePoint library and store them in a separate field in the search index. To do this, you can use the Azure Cognitive Search REST API to update the search index with the embeddings generated by the Azure Function.

Here is a high-level overview of the steps you can follow:

Set up a SharePoint indexer in Azure Cognitive Search to index documents from the SharePoint library.

Create an Azure Function that generates embeddings for each document in the SharePoint library and stores them in a separate field in the search index.

Use the Azure Function to update the search index with the embeddings generated for each document.

Configure the SharePoint indexer to execute the Azure Function upon the completion of a scheduled indexer run.

To execute the Azure Function upon the completion of a scheduled indexer run, you can use the Azure Cognitive Search Webhooks feature. You can configure the indexer to send a webhook notification to the Azure Function URL upon the completion of a scheduled indexer run. The Azure Function can then use the notification to trigger the generation and storage of embeddings for each document in the SharePoint library.
Sarva, Pavan 45 Reputation points

2023-10-21T03:32:12.3666667+00:00

@brtrach-MSFT , Thank you som much for letting me know regarding the Azure Cognitive Search Webhooks feature. I coud not locate relavant content.

Please may I request you to share any links to the documentation on how to configure indexer to send a webhook notiifcation.
brtrach-MSFT 17,741 Reputation points Microsoft Employee Moderator

2023-10-25T02:31:51.57+00:00

I do not see any documentation on that. My suggestion would be to make a request to the doc author here. They can then review the feedback and add content.
Kshitiz Kalra 0 Reputation points Microsoft Employee

2023-11-24T09:57:18.2833333+00:00

@brtrach-MSFT Hi Sir, can you please let me know how it works for the SharePoint documents which have some sensitivity or security labels
Saurabh Bhadane 5 Reputation points

2025-04-10T14:54:52.9233333+00:00

is this functionality of generating embeddings using skillsets using OpenAiEmbeddings available in sharepoint indexer as of today ?
Didn't find any relevant guide or documents online ?

Share via

Document Indexing and Vector Embeddings for SharePoint files using Azure Cognitive Search

1 answer

Your answer