Null values not being picked up in azure search

Question

Null values not being picked up in azure search

Matt 0

Hello,

When the azure search indexer goes through my azure blob table storage it does not pick up null values but I thought at one point it did. For example: Metadata is present in blob storage Testfield: testString, this metadata is deleted making the value for the field Testfield: null. After running the indexer the metadata field does not get updated even though everything else does in that document. The Testfield:testString is still present.

The metadata field only gets updated if it is changed to something else but not deleted entirely.

Is there a solution in azure for this?

Matt 0 Reputation points

2024-11-07T13:50:49.5566667+00:00

Thank you for the reply,
It does allow null values

Use Soft Deletes: Instead of deleting, use placeholder values (e.g., "Deleted") to indicate missing data. This triggers updates. More details.

We do use a soft delete like this but this deletes the whole row or document. Do you mean when it is in the storage doing something like TestField: null , instead have it TestField:Deleted?

Reindex the Entire Dataset: Manually trigger a full reindex to process null values. Reindexing info.
When you say this, do you mean go to the indexer and do a reset? If so is there a way to have it do its custom indexer runs every 15 min which is how it is currently set and also have a separate setting to make it reset every night and run the whole thing again?
Shree Hima Bindu Maganti 4,925 Reputation points Microsoft External Staff Moderator

2024-11-08T07:32:46.0466667+00:00

Hi @Matt ,

Thankyou For Your Response
Soft Deletes: Yes, instead of removing the field value entirely, set the field in storage to TestField: "Deleted" rather than TestField: null. This way, the indexer will detect the change and update the document in Azure Cognitive Search without deleting the whole row or document.

Reindex the Entire Dataset:

Yes, to perform a full reindex, you can go to the indexer and manually select "reset." This will force the indexer to reprocess all documents, which is useful for picking up changes like null values.

Unfortunately, Azure Cognitive Search doesn’t support dual schedules within a single indexer. To accomplish both a 15-minute incremental update and a daily full reset, here are two options:

Azure Logic App or Azure Function: Set up a Logic App or Function with a daily trigger to send a reset request to the indexer via the Azure REST API, alongside your existing 15-minute schedule.

Separate Indexers: Configure two indexers: one with a 15-minute interval for incremental updates and another scheduled to run once nightly for a full reindex. Both can target the same data source and index.
you can refer to Azure Cognitive Search Indexer Documentation.
Matt 0 Reputation points

2024-11-08T20:17:58.8366667+00:00

Thank you,
Re triggering a full index does not work well with my setup since there are multitudes of documents it would have to run through every time.
I settled on using an empty string instead of null value. This way the search will pick it up

1 answer

Your answer

Matt 0 Reputation points

2024-11-07T13:50:49.5566667+00:00

Thank you for the reply,
It does allow null values

Use Soft Deletes: Instead of deleting, use placeholder values (e.g., "Deleted") to indicate missing data. This triggers updates. More details.

We do use a soft delete like this but this deletes the whole row or document. Do you mean when it is in the storage doing something like TestField: null , instead have it TestField:Deleted?

Reindex the Entire Dataset: Manually trigger a full reindex to process null values. Reindexing info.
When you say this, do you mean go to the indexer and do a reset? If so is there a way to have it do its custom indexer runs every 15 min which is how it is currently set and also have a separate setting to make it reset every night and run the whole thing again?
Shree Hima Bindu Maganti 4,925 Reputation points Microsoft External Staff Moderator

2024-11-08T07:32:46.0466667+00:00

Hi @Matt ,

Thankyou For Your Response
Soft Deletes: Yes, instead of removing the field value entirely, set the field in storage to TestField: "Deleted" rather than TestField: null. This way, the indexer will detect the change and update the document in Azure Cognitive Search without deleting the whole row or document.

Reindex the Entire Dataset:

Yes, to perform a full reindex, you can go to the indexer and manually select "reset." This will force the indexer to reprocess all documents, which is useful for picking up changes like null values.

Unfortunately, Azure Cognitive Search doesn’t support dual schedules within a single indexer. To accomplish both a 15-minute incremental update and a daily full reset, here are two options:

Azure Logic App or Azure Function: Set up a Logic App or Function with a daily trigger to send a reset request to the indexer via the Azure REST API, alongside your existing 15-minute schedule.

Separate Indexers: Configure two indexers: one with a 15-minute interval for incremental updates and another scheduled to run once nightly for a full reindex. Both can target the same data source and index.
you can refer to Azure Cognitive Search Indexer Documentation.
Matt 0 Reputation points

2024-11-08T20:17:58.8366667+00:00

Thank you,
Re triggering a full index does not work well with my setup since there are multitudes of documents it would have to run through every time.
I settled on using an empty string instead of null value. This way the search will pick it up

Answer 1

Hi @Matt ,
Welcome to the Microsoft Q&A Platform!
In Azure Cognitive Search, when an indexer processes data from a data source like Azure Blob Storage, it may not update fields that have null or empty values, depending on how the indexer is configured. This behavior is typically observed when a field is explicitly set to null or removed from the source but the indexer doesn't treat this as an update.
Ensure Field Allows Nulls: Check that your field is defined to allow nulls in the index. Learn more.

Use Soft Deletes: Instead of deleting, use placeholder values (e.g., "Deleted") to indicate missing data. This triggers updates. More details.

Reindex the Entire Dataset: Manually trigger a full reindex to process null values. Reindexing info.

Configure Data Source Correctly: Ensure the field is properly mapped between the data source and the index. Indexing configuration.

Trigger Reindex Manually: Use Azure REST API or the portal to force reindexing. API details.
By applying these methods, you can ensure that null and deleted fields are handled appropriately by your Azure Search indexer.
f the answer is helpful, please click "Accept Answer" and kindly upvote it.

Share via

Null values not being picked up in azure search

1 answer

Your answer