How can I create an Azure Cognitive Search indexer to retrieve data from SharePoint, split it into chunks, and generate embeddings using skills?

Question

How can I create an Azure Cognitive Search indexer to retrieve data from SharePoint, split it into chunks, and generate embeddings using skills?

Charlie 30

Hello,

I have a SharePoint data source, an Azure Cognitive Search resource, and an index setup. The index has fields that I want to search via ACS (name, content, name_vector, content_vector) and metadata fields (id, path, content_type, last_modifed_date, size).

I'm trying to create an indexer to pull data from the SharePoint data source, make necessary transformations, and push that data to the index. The indexer must split the SharePoint documents into chunks and generate embeddings. This data will then fill the index: chunk text -> content; chunk embeddings -> content_vector; name text -> name; name embeddings -> name_vector; etc. Here, 'name' is the name of the SharePoint document, which means there will be the same name (and other metadata fields) for multiple chunks/entries in the index. There is not a one-to-one relationship between SharePoint documents and entries of the index.

I saw this post which suggests that I would need an Azure Function to 'chunk and embed' the SharePoint data. That function then becomes the data source for the indexer. This might be a viable solution, but I'm curious - will the indexer still be able to detect changes on the SharePoint site if that is no longer the direct data source?

The other, and simpler, solution would be to add skills to the indexer. Microsoft has an AzureOpenAIEmbeddingSkill class and a SplitSkill class. How can I create the following indexer:

Input: SharePoint data source documents and metadata
Run Skill: Create chunks with SplitSkill
Run Skill: Generate embeddings from chunks with AzureOpenAIEmbeddingSkill
Output: Stores embeddings and metadata for each chunk in the index

What are the steps to set up this indexer? Do I need to create an Azure Function for this?

Thank you.

Grmacjon-MSFT 19,151 Reputation points Moderator

2023-10-26T23:34:08.7966667+00:00
Hi @Charlie thanks for the question.

here is one way to set up an Azure Cognitive Search indexer with skills to chunk SharePoint documents, generate embeddings, and index enriched data:

Set up a SharePoint Online data source and indexer into your target index. This ingests content and metadata:

Add a "Text Split" skill to break up content into smaller chunks. You can specify whether you want to break the text into sentences or into pages of a particular length. This skill is especially useful if there are maximum text length requirements in other skills downstream.

Configure the Split skill to output chunks to a custom field like "contentChunks".

Add the "AzureOpenAIEmbeddingSkill" to generate embeddings for each chunk:

Output chunk embeddings to a vector field like "contentVectors"

Map the final enriched fields into your index definition:

Enable change tracking and set indexer schedule:

This allows transforming SharePoint content directly in the indexer pipeline using chaining skills, no external function required.

Hope that helps.
Charlie 30 Reputation points

2023-11-01T16:33:17.6933333+00:00

Thanks for the response @Grmacjon-MSFT

So if I created a field called contentVectors, what would its type be? The contentChunks field would be an array of strings, but contentVectors would have to be an array of float arrays, which is not allowed for indexes. Splitting up each vector/chunk into its own field also wouldn't work because of max field limits and varying number of chunks.

Also, Azure Cognitive Search forces a 1-to-1 relationship between SharePoint documents and entries of the index. This means that I can't split SharePoint documents into separate chunks and have each chunk as its own entry/row of the index.

Source: https://learn.microsoft.com/en-us/azure/search/search-howto-index-sharepoint-online

Only metadata_spo_site_library_item_id may be used as the key field in an index populated by the SharePoint indexer. If a key field doesn't exist in the data source, metadata_spo_site_library_item_id is automatically mapped to the key field.

Any ideas on how I can get around this?

Currently, it doesn't seem possible to create a SharePoint indexer that can chunk documents, generate embeddings, and insert these chunks into an index.

Thanks
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Grmacjon-MSFT 19,151 Reputation points Moderator

2023-11-14T03:25:09.51+00:00

you're right, after reviewing the doc you shared I don't think there is currently a turn key solution to creating a SharePoint indexer that can chunk documents, generate embeddings, and insert chunks into an index directly. I will reach out to the engineering team internally to see if there is a possible work around for your scenario and get back to you. Thanks
P A 25 Reputation points

2023-12-05T11:05:23.42+00:00

Hi @Grmaca Anamaria ! Thanks for your anwser.
We have the same request as Charlie.

Do you have more information about it ?

It is unsuable for now, since we cannot retrieve individual chunks, and the total page does not fit in any LLM.

Regards,
BEKKOUCH Imad-Eddine 10 Reputation points

2023-12-05T11:08:51.97+00:00

Hi @Grmacjon-MSFT

we have a probleme also for tracking the page number in which each text is located. this allows us to have flexibility to integrate different internal tools for interaction with LLMs

Regards,

1 answer

Your answer

Grmacjon-MSFT 19,151 Reputation points Moderator

2023-10-26T23:34:08.7966667+00:00

Hi @Charlie thanks for the question.

here is one way to set up an Azure Cognitive Search indexer with skills to chunk SharePoint documents, generate embeddings, and index enriched data:

Set up a SharePoint Online data source and indexer into your target index. This ingests content and metadata:

Add a "Text Split" skill to break up content into smaller chunks. You can specify whether you want to break the text into sentences or into pages of a particular length. This skill is especially useful if there are maximum text length requirements in other skills downstream.

Configure the Split skill to output chunks to a custom field like "contentChunks".

Add the "AzureOpenAIEmbeddingSkill" to generate embeddings for each chunk:

Output chunk embeddings to a vector field like "contentVectors"

Map the final enriched fields into your index definition:

Enable change tracking and set indexer schedule:

This allows transforming SharePoint content directly in the indexer pipeline using chaining skills, no external function required.

Hope that helps.
Charlie 30 Reputation points

2023-11-01T16:33:17.6933333+00:00

Thanks for the response @Grmacjon-MSFT

So if I created a field called contentVectors, what would its type be? The contentChunks field would be an array of strings, but contentVectors would have to be an array of float arrays, which is not allowed for indexes. Splitting up each vector/chunk into its own field also wouldn't work because of max field limits and varying number of chunks.

Also, Azure Cognitive Search forces a 1-to-1 relationship between SharePoint documents and entries of the index. This means that I can't split SharePoint documents into separate chunks and have each chunk as its own entry/row of the index.

Source: https://learn.microsoft.com/en-us/azure/search/search-howto-index-sharepoint-online

Only metadata_spo_site_library_item_id may be used as the key field in an index populated by the SharePoint indexer. If a key field doesn't exist in the data source, metadata_spo_site_library_item_id is automatically mapped to the key field.

Any ideas on how I can get around this?

Currently, it doesn't seem possible to create a SharePoint indexer that can chunk documents, generate embeddings, and insert these chunks into an index.

Thanks
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Grmacjon-MSFT 19,151 Reputation points Moderator

2023-11-14T03:25:09.51+00:00

you're right, after reviewing the doc you shared I don't think there is currently a turn key solution to creating a SharePoint indexer that can chunk documents, generate embeddings, and insert chunks into an index directly. I will reach out to the engineering team internally to see if there is a possible work around for your scenario and get back to you. Thanks
P A 25 Reputation points

2023-12-05T11:05:23.42+00:00

Hi @Grmaca Anamaria ! Thanks for your anwser.
We have the same request as Charlie.

Do you have more information about it ?

It is unsuable for now, since we cannot retrieve individual chunks, and the total page does not fit in any LLM.

Regards,
BEKKOUCH Imad-Eddine 10 Reputation points

2023-12-05T11:08:51.97+00:00

Hi @Grmacjon-MSFT

we have a probleme also for tracking the page number in which each text is located. this allows us to have flexibility to integrate different internal tools for interaction with LLMs

Regards,

Answer 1

Andrej Melicher 90

I found this very useful article about using projection approach when creating Skillset with document chunking and generating embeddings for each chunk in single indexer.

The example is using Azure Blob Storage, but I was able to manage same results with Sharepoint Document Library as data source.

Simon Weh 0 Reputation points

2024-09-16T12:11:29.5366667+00:00

How did you manage to do this with SharePoint? Would it be possible for you to share the three JSON files with us: the Indexer JSON, Indexes JSON, and Skillset JSON?
Andrej Melicher 90 Reputation points

2024-09-16T12:47:24.26+00:00

Here is export of Postman collection
https://1drv.ms/u/s!AgBcXhqagjcQqr530l_mD4UkwPf75g?e=CzW1wn

I'm using various skills including custom skill for text summarization, you can remove it if you don't need it.

Share via

How can I create an Azure Cognitive Search indexer to retrieve data from SharePoint, split it into chunks, and generate embeddings using skills?

1 answer

Your answer