Index custom meta data attributes of file in azure cognitive search

Question

Index custom meta data attributes of file in azure cognitive search

teja_98666 36

Hi,
I'm exploring Azure Cognitive search to index files. Files are stored in Azure blob storage.
I created an indexer, index and a data source and ran the indexer i could notice that it indexes only few meta data properties like file name, path etc.
Our application stores the custom meta data properties of files in database. How do i index these properties along with the pre-defined properties?
is there any api that helps me to push the file from blob storage to index and also stamp the custom meta data properties?
Thanks

1 answer

Your answer

Answer 1

SnehaAgrawal-MSFT 22,706 Moderator

Thanks for asking question!

As mentioned here in Azure Blob indexer - Azure Cognitive Search

custom metadata of blobs will be indexed as long as the user-specified (custom) metadata properties have the exact same value defined in the index so it is mapped.

"User-specified metadata properties are extracted verbatim. To receive the values, you must define field in the search index of type Edm.String, with same name as the metadata key of the blob. For example, if a blob has a metadata key of Sensitivity with value High, you should define a field named Sensitivity in your search index and it will be populated with the value High."

Let us know if further query.

teja_98666 36 Reputation points

2022-04-08T07:03:10.057+00:00

Hi,
Thanks for your quick response.
As i understand the custom meta data properties of the file must be defined on the file in the blob storage only then indexer will identify those to index the value in the matching field in the index.

Is there any other way to achieve this with out defining the meta data properties on the file? My application has about 10+ properties that can change over time, and i want these to be read from database and then pushed to the index instead of indexer doing it for us. Any way we can acheive this?
SnehaAgrawal-MSFT 22,706 Reputation points Moderator

2022-04-08T15:52:50.5+00:00

Thanks for reply! You can use Push API (https://learn.microsoft.com/en-us/azure/search/search-what-is-data-import) or SDK ( https://learn.microsoft.com/en-us/azure/search/tutorial-optimize-indexing-push-api) push properties by reading them off of a database to map them to specific existing index fields.

You can have some logic to keep adding new fields in the index (https://learn.microsoft.com/en-us/rest/api/searchservice/update-indexand) pushing custom metadata.
teja_98666 36 Reputation points

2022-04-19T10:01:12.647+00:00
Hi,
Apologies for the delay.

Here is what i need, index file content from the blob storage and update the indexed file with the custom meta data properties from the database.

I tried -

Created an indexer, index and datasource (azure blob) and ran the indexer it indexed the files in the blob and i could see the results in the search explorer.

Created a sample .NET applicaton to read the custom meta data properties of the file from the database and then push them to the index using SDK. I noticed that the MergeDocumentsAsync() api replaces the properties that are indexed from blob storage (metadata_storage_name,metadata_storage_path etc) to NULL after the update operation.

What is the right way to update a specific property of a document?
SnehaAgrawal-MSFT 22,706 Reputation points Moderator

2022-04-21T06:07:59.76+00:00

Thanks! MergeDocumentsAsync() should replace only the fields you are adding to be replaced.

Also, we would like to check how you are using this and check for current data in the index. I have reached out to you privately on this please respond there.

Share via

Index custom meta data attributes of file in azure cognitive search

1 answer

Your answer