Hi Azure Support, I hope this message finds you well. I'm reaching out because I have encountered an issue while customizing an Azure Search Index using Python. Specifically, I am attempting to add the source page information to the index schema for reference purposes in my application. However, my attempts to achieve this by utilizing metadata have been unsuccessful. Here's a brief overview of my situation: I have created an Azure Search Index using Python code. I have customized the index schema to align with my specific use case. The documents utilized to create the index include PDFs, DOCX files, and PPTX files. My objective is to include the source page information within the index schema. Could you please provide guidance on how to successfully incorporate the source page information into the index schema for documents of these types? Your assistance in resolving this matter would be greatly appreciated. Thank you in advance for your support. Best regards, Deepak Dange

@Deepak Dange To include the source page information within the index schema, you can use the metadata_storage_content_type field. This field is automatically populated with the content type of the document being indexed. You can also use it to store additional metadata about the document, such as the source page information. Here's an example of how you can add the metadata_storage_content_type field to your index schema: from azure.search.documents.indexes import SearchFieldDataType, SearchableField fields = [ SearchableField(name="id", type=SearchFieldDataType.String, key=True), SearchableField(name="content", type=SearchFieldDataType.String), SearchableField(name="metadata_storage_content_type", type=SearchFieldDataType.String) ] In this example, we've added the metadata_storage_content_type field to the index schema. You can then populate this field with the source page information when you upload your documents to the index. Please note that the metadata_storage_content_type field is only available for certain file types, including PDFs, DOCX files, and PPTX files. If you're indexing other file types, you may need to use a different approach to store the source page information.

How to add page number (sourcepage) to the AI search Index

Deepak Dange 5

Hi Azure Support,

I hope this message finds you well. I'm reaching out because I have encountered an issue while customizing an Azure Search Index using Python. Specifically, I am attempting to add the source page information to the index schema for reference purposes in my application. However, my attempts to achieve this by utilizing metadata have been unsuccessful. Here's a brief overview of my situation:

I have created an Azure Search Index using Python code.
I have customized the index schema to align with my specific use case.
The documents utilized to create the index include PDFs, DOCX files, and PPTX files.
My objective is to include the source page information within the index schema.

Could you please provide guidance on how to successfully incorporate the source page information into the index schema for documents of these types? Your assistance in resolving this matter would be greatly appreciated. Thank you in advance for your support. Best regards, Deepak Dange

1 answer

brtrach-MSFT 16,506 Reputation points Microsoft Employee

2024-02-15T03:37:00.6133333+00:00
@Deepak Dange To include the source page information within the index schema, you can use the metadata_storage_content_type field. This field is automatically populated with the content type of the document being indexed. You can also use it to store additional metadata about the document, such as the source page information. Here's an example of how you can add the metadata_storage_content_type field to your index schema:

from azure.search.documents.indexes import SearchFieldDataType, SearchableField fields = [ SearchableField(name="id", type=SearchFieldDataType.String, key=True), SearchableField(name="content", type=SearchFieldDataType.String), SearchableField(name="metadata_storage_content_type", type=SearchFieldDataType.String) ]

In this example, we've added the metadata_storage_content_type field to the index schema. You can then populate this field with the source page information when you upload your documents to the index. Please note that the metadata_storage_content_type field is only available for certain file types, including PDFs, DOCX files, and PPTX files. If you're indexing other file types, you may need to use a different approach to store the source page information.
Please sign in to rate this answer.
Deepak Dange 5 Reputation points

2024-04-18T04:57:35.5233333+00:00

@brtrach-MSFT
Hi,

I have tried doing this approach but didn't work. Can you please elaborate it how can I add page number with the content.
How to make that integration can you please help with it.
I am having number of pdf documents and I want to store the content with actual page number from the document. I am using other skillsets like splitskill, ocr, merge skillset and openai embeddings skillset.

Please tell if the page numbers can be stored.

Mads Møller 0 Reputation points

2024-05-03T19:54:46.92+00:00

@brtrach-MSFT I could also use some code to store the source page of chunks when using the integrated vectorization in Azure AI Search...

brtrach-MSFT 16,506 Reputation points Microsoft Employee

2024-05-15T01:16:02.22+00:00

@Deepak Dange To store the page numbers along with the content in your Azure Search Index, you can utilize the skillsets like split skill to divide the document into individual pages and then use the OCR skill to extract text from each page. After that, you can use a custom skill to add the page number metadata to each indexed document.

Here’s a high-level overview of the steps you might follow:

Split Skill: Use the split skill to break down the document into pages. Each page becomes a separate document in the index with its own unique ID.

OCR Skill: Apply the OCR skill to extract text from each page.

Custom Skill: Create a custom skill to add the page number to the extracted content. This skill will take the output from the OCR skill and append the page number to it.

Merge Skill: If needed, use the merge skill to combine fields from different skills into a single document in the index.

For the custom skill, you would write a Python function that takes the output from the OCR skill and adds the page number. The function might look something like this:

def extract_page_number(documents): for document in documents: page_number = document["metadata_storage_path"].split("#")[-1] content = document["text"] document["text_with_page_number"] = f"{content} (Page {page_number})" return documents

You would then integrate this custom skill into your indexing pipeline by adding it to the skillset definition in your Python code.

Please note that the actual implementation will depend on the specifics of your documents and how the split skill outputs the page number information. You may need to adjust the logic in the extract_page_number function accordingly.

Remember to update your index schema to include the new field where the page number will be stored. For example:

from azure.search.documents.indexes import SearchFieldDataType, SearchableField fields = [ SearchableField(name="id", type=SearchFieldDataType.String, key=True), SearchableField(name="text_with_page_number", type=SearchFieldDataType.String), # Add any other fields you need here ] index = SearchIndex(name="my_index", fields=fields) index_client.create_index(index)

This approach should allow you to store the content with the actual page number from the document in your Azure Search Index.

Das Gupta, Abhijeet 100 Reputation points

2024-06-18T09:53:34.3533333+00:00

Hi @brtrach-MSFT , appreciate your answer on this topic! Just want to point out that I do not see the code for the custom skill (I just see some incomplete lines of code) and index schema update code in your comment. Could you please update your comment with the code completions? Thank you

Deepak Dange 5 Reputation points

2024-06-18T10:04:11.4733333+00:00

@brtrach-MSFT Thanks for your response. I was trying to do the exact thing you have mentioned. The code snippet you have provided is incomplete. Can you please share the code for custom skillset?

It would be great help for us to work further on it.

brtrach-MSFT 16,506 Reputation points Microsoft Employee

2024-06-19T19:55:45.2866667+00:00

@Das Gupta, Abhijeet @Deepak Dange My apologies for not seeing this sooner. I have updated my response from May 14th with the relevant code samples.

Let me know if you have any further questions.

Das Gupta, Abhijeet 100 Reputation points

2024-06-25T08:16:26.8366667+00:00

Hi @brtrach-MSFT Thank you updating your response!

I have a few follow up questions

Can the OCR skill extract page number from a text-based PDF containing no images?

If yes above, what should be the context parameter set to, inside the OCR skill definition? You have mentioned to apply the skill on each document page after the split skill, so should the context="/document/pages/*" . I ask this because from the examples of OCR skill I've gathered from the microsoft documentation, OCR is applied to /document/normalized_images/* context.

You mentioned to store the page number in the meta_storage_content_type field in your first answer, but in your last updated post you created a new field text_with_page_number to add to the index. Just want to confirm that both approaches are fine?

Could you also share how to map the output of the custom skillset to the text_with_page_number field?

If no, does that mean absolute page numbers have to be generated? What would be the approach here?
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

How to add page number (sourcepage) to the AI search Index

1 answer

Your answer