Index File content using Azure Cognitive Search

Question

Index File content using Azure Cognitive Search

teja_98666 36

Hi,

I'm exploring the capability of Azure congnitive search to index file content.
Looking at the documentation i could see that Azure Cognitive search can index files that are stored in Azure Files, Azure Blob storage but it can only index predefined meta data properties only.

search-file-storage-integration
search-howto-indexing-azure-blob-storage
search-blob-metadata-properties

What i'm looking for is an api that would index the file given its location (in Azure Blob or file storage) along with additional properties that i'm interested in. This way i can query the file content and get the custom meta data properties which i would want to use to get the related information from other applications.
Do we have any such apis?

Also, at the moment ACS supports indexing of specific set of file formats how do we index a custom file format that is not specified in the list?

ajkuma 28,036 Reputation points Microsoft Employee Moderator

2022-01-31T20:40:59.007+00:00

teja98666-4126, Apologies for the delay in responding from over the weekend. I'm discussing on this internally and will get back to you shortly. Thanks for your patience.

1 answer

Your answer

ajkuma 28,036 Reputation points Microsoft Employee Moderator

2022-01-31T20:40:59.007+00:00

teja98666-4126, Apologies for the delay in responding from over the weekend. I'm discussing on this internally and will get back to you shortly. Thanks for your patience.

Answer 1

ajkuma 28,036 Microsoft Employee Moderator

@teja_98666 ,

1) Pull indexers for pre-defined data sources are a tool you could leverage to index right away with the existent capabilities.

You may customize what data you want in the index when using a pull indexer using skillset with custom skills. Also, please check if any of the pre-defined skills would be useful your requirement.

cognitive-search-predefined-skills
cognitive-search-defining-skillset

--Additionally - However, you also have the option of creating your own indexer from scratch using push API or SDK to push data into the index as needed as defined here:
search-what-is-data-import

2) For file types that are not supported by the Search service for indexing. You may check the possibility of converting the format, into a supported format such as JSON or similar. As currently, only the listed file types are supported.

Thanks for your feedback and cooperation!

teja_98666 36 Reputation points

2022-02-01T12:27:57.293+00:00

Hi@Anonymous thanks for your reply.

I'm currently using the Push method to index my data from local oracle database to Azure Cognitive Search Index. This works great and i have no issues.

What i'm looking for is a way to index content of physical files along with some additional meta data of the file like Unique id, access level etc ( these attributes are pulled from database).

In SOLR we have a capability to pass in additional information along with the file path and Solr stores the file content along with the additional metadata. Do we have a similar functionality in Azure Cognitive Search?
I can have my files to be available in Azure Blob storage but i need a way to index the additional meta data.
ajkuma 28,036 Reputation points Microsoft Employee Moderator

2022-02-02T05:13:40.487+00:00

teja98666-4126, As I understand, you wish to understand on how you can pull extra metadata from Azure Blob Storage and then add it through push API to Cognitive Search index.

1.This is the API in order to retrieve any metadata from blob storage: Get Blob Properties (REST API) - Azure Storage | Microsoft Learn

2.Once you get the data (from 1), you can create a matching field in your Azure Cognitive Search index and push it through the Push API (as mentioned above post).

To benefit the community find the right answers, please do mark the post which was helpful by clicking on ‘Accept Answer’ & ‘Up-Vote’.
Dhananjay Wasekar 1 Reputation point

2023-01-20T13:43:43.93+00:00

Hello ajkuma, I have same similar doubt regarding push model. Is there any API to run indexer manually for some specific data? or for some specific blob id in blob storage? What I know is to index blob, document extraction is done by indexer and extracted text is stored in "content" key.

But that indexer will index the documents which are uploaded during scheduled timespan of previous and current running indexer. what I need here is to run indexer and get the extracted text for specific file which is uploaded.

something like we send an base64 string and get the extracted text in the API response. so that I can manually add that extracted text to content key and push into index.

references - Document Extraction Like here, we have sample input/output json defined.

Share via

Index File content using Azure Cognitive Search

1 answer

Your answer