Azure Blob indexer producing "null" values for user-defined blob metadata in Azure AI Search

Question

Azure Blob indexer producing "null" values for user-defined blob metadata in Azure AI Search

Berk Dogan 10

I created an index using the Import and vectorize data button through an Azure Blob Container successfully. However, I had already some custom metadata (url_sap) in the blobs that I wanted to transition to the Search Index as retrievable fields.

For that purpose, I created the url_sap String field in the Search Index and ran the indexer again (successfully ran through all documents), but I am getting url_sap as null for all documents. I also tried to include a pre-defined blob property metadata_storage_content_type in the Search Index, and ran the indexer again, but the new field is always null.

While initially creating the index through Import and vectorize data, I also enabled Extract text from images, disabled semantic ranking, enabled deletion tracking with Native blob soft delete.

ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-12-14T19:21:27.2033333+00:00

@Berk Dogan , Just checking in to see if you had got a chance to see the previous response. Please share the requested/more info to help you better. I'll follow-up further.

Note: Do not share any PII data on the public forum.

1 answer

Your answer

ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-12-14T19:21:27.2033333+00:00

@Berk Dogan , Just checking in to see if you had got a chance to see the previous response. Please share the requested/more info to help you better. I'll follow-up further.

Note: Do not share any PII data on the public forum.

Answer 1

ajkuma 28,036 Microsoft Employee Moderator

Thanks for posting this question.

As I understand from the issue description, you are getting null values for the custom metadata fields that you wanted to transition to the Search Index as retrievable fields. You also tried to include a pre-defined blob property metadata_storage_content_type in the Search Index, but the new field is always null.

While I'm checking on this, kindly check the logs to fetch more details about the error: Diagnostic settings.

Troubleshooting common indexer errors and warnings in Azure AI Search

ajkuma 28,036 Reputation points Microsoft Employee Moderator

2023-12-13T19:18:24.21+00:00

Following-up, kindly share your findings from the diagnostic logs.

For a deeper investigation, we may need additional info about your subscription and Azure resource (to check internal logs).

Also, if you haven't done, just to isolate, kindly review the skillset definitions, make sure that you are setting the right mappings.

Review the field mappings in the indexer definition. Make sure that the url_sap field is properly mapped to the corresponding metadata field in the blob.

Kindly let us know, I'll follow-up further.
Jeff Teeter (jefteete) 15 Reputation points

2024-01-22T16:59:54.09+00:00

I am having same issue. Understanding that the "Import and vectorize data Wizard" doesnt allow for the schema to be modified and that source fields include conent (chunked and vectorized), metadata_storage_name for title, and a metadata_storage_path for document key. As the original post, I also added manual custom metadata to pdf's in a blob container, added a field in the index (that matched the custom metadata name, and added mapping to the indexer for the custom metadata. I have also tried adding various configs to the skillset. Question, is customization allowed after the "Import and vectorize data Wizard" and if so, is there any documentation or help that i can reference.

i have seens some articales abouth this issue, and one possible solution using a custom web api - (https://python.plainenglish.io/unlocking-advanced-capabilities-enhancing-azure-ai-search-vector-setup-with-custom-metadata-and-5d84bf4670d2). But this doesn't specifically work for my situation. Any help you could give would be helpful, since this seeems to be a common issue, since people are now writing articales about it.
Marios Markakis 0 Reputation points

2024-08-01T09:40:15.44+00:00

Any updates on this topic? I am having the exact same issue as Berk

Share via

Azure Blob indexer producing "null" values for user-defined blob metadata in Azure AI Search

1 answer

Your answer