We are sorry to hear you're having difficulties using HighWaterMarkChangeDetectionPolicy field with the search service API
Based on the error message you shared, it looks like the _ts
field is not available in the search index schema, and therefore cannot be used as a sorting field in the search API call.
To use the HighWaterMarkChangeDetectionPolicy
in the search API, you can define a new field in your search index schema to store the timestamp value of the HighWaterMarkChangeDetectionPolicy
field from Cosmos DB. You can then use this field for sorting in the search API call.
Here's an example of how to modify your search index schema to include a new field for the HighWaterMarkChangeDetectionPolicy
value:
{
"name": "your-search-index",
"fields": [
{"name": "id", "type": "Edm.String", "key": true, "searchable": false},
{"name": "collectionName", "type": "Edm.String", "searchable": true},
{"name": "coverThumbnailURL", "type": "Edm.String", "searchable": true},
{"name": "type", "type": "Edm.String", "searchable": true},
{"name": "artists", "type": "Collection(Edm.String)", "searchable": true},
{"name": "highWaterMarkTimestamp", "type": "Edm.DateTimeOffset", "searchable": false}
]
}
Next, you can update your indexing code to extract the HighWaterMarkChangeDetectionPolicy
value and store it in the highWaterMarkTimestamp
field of the search index document. Here's an example:
using Microsoft.Azure.Search;
using Microsoft.Azure.Search.Models;
using Microsoft.Azure.Cosmos;
using Newtonsoft.Json.Linq;
// ...
var cosmosClient = new CosmosClient(endpointUrl, primaryKey);
var container = cosmosClient.GetContainer(databaseId, containerId);
var searchClient = new SearchServiceClient(searchServiceName, new SearchCredentials(apiKey));
var searchIndexClient = searchClient.Indexes.GetClient(searchIndexName);
var query = "SELECT * FROM c WHERE c._ts > @highWaterMarkTimestamp";
var parameters = new SqlParameterCollection();
parameters.Add(new SqlParameter("@highWaterMarkTimestamp", highWaterMarkTimestamp));
var iterator = container.GetItemQueryIterator<JObject>(
new QueryDefinition(query).WithParameterCollection(parameters));
while (iterator.HasMoreResults)
{
foreach (var item in await iterator.ReadNextAsync())
{
var searchDocument = new SearchDocument(item);
searchDocument["highWaterMarkTimestamp"] = (DateTimeOffset)item["_ts"];
var indexBatch = IndexBatch.MergeOrUpload(new[] { IndexAction.Upload(searchDocument) });
await searchIndexClient.Documents.IndexAsync(indexBatch);
}
}
// update highWaterMarkTimestamp value after indexing is complete
var newHighWaterMarkTimestamp = // get latest _ts value from Cosmos DB
Lastly, you can modify your search API query to sort by the new highWaterMarkTimestamp
field, like this:
{
"count": true,
"filter": "type eq 'collection'",
"queryType": "full",
"orderby": "highWaterMarkTimestamp",
"select": "id,collectionName,coverThumbnailURL,type,artists",
"skip": 0,
"top": 10
}
This should allow you to use the HighWaterMarkChangeDetectionPolicy
in conjunction with the search API in a way that enables you to perform incremental indexing with Cosmos DB.
Hope that helps.
-Grace
If the information helped address your question, please Accept the answer. This will help us and also improve searchability for others in the community who might be researching similar information.