A question regarding responseIterator from the queryChangeFeed method (java)

Question

Hello,

I've been playing around with the CosmosContainer class method queryChangeFeed to fetch the change feed for a specific cosmos db container. According to the Azure documentation, upon obtaining the query result as an iterator, iterator.hasNext() will always be true given the nature of change feed being effectively an infinite list of items encompassing all future writes and updates. However, when I try a while/do-white loop with the condition that checks whether iterator.hasNext() returns true, I can see that the loop comes to an end, indicating that iterator.hasNext() in fact returns false at some point(whether it has reached the last item in given container, etc).

Could someone please let me know if I am misinterpreting the documentation, and whether this is an expected behavior?

To provide additional details, the code block looks something like:

CosmosChangeFeedRequest options = CosmosChangeFeedRequestOptions.createForProcessingFromBeginning(FeedRange.forFullRange());
Iterator<FeedResponse<JsonNode>> feedResponseIterator = container.queryChangeFeed(options, JsonNode.class).iterableByPage().iterator();
while(feedResponseIterator.hasNext()){
FeedResponse<JsonNode> = feedResponseIterator.next();
//do something else
}

Change feed(pull model) documentation: https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/change-feed-pull-model?tabs=java#consuming-an-entire-containers-changes
CosmosContainer documentation: https://learn.microsoft.com/en-us/java/api/com.azure.cosmos.cosmoscontainer?view=azure-java-stable

Answer 1

Hi @Jae Kim

This is the expected behavior, because if there are no more changes in the feed container, feedResponse will exit at some point (if it has reached the last element).

In that case, you need to re-fetch the change feed data using the returned continuation token in the feedResponse as mentioned in the samples for change feed pull model here - https://github.com/Azure-Samples/azure-cosmos-java-sql-api-samples/blob/main/src/main/java/com/azure/cosmos/examples/changefeedpull/SampleChangeFeedPullModel.java#L81

If you still face more issues, please create a GitHub bug here https://github.com/Azure/azure-sdk-for-java/issues and our team can take a deeper look at it.

Regards,
Oury

Answer 2

Hi @Jae Kim

I checked internally with the content team and all of the information is true.
If you see the full context, there is the continuation token example. Because the change feed is effectively an infinite list of items encompassing all future writes and updates, the value of responseIterator.hasNext() is always true. Below is an example, which reads all changes starting from the beginning. Each iteration persists a continuation token after processing all events and will pick up from the last processed point in the change feed. What it is stating is that after reading from beginning, then you can use the continuation token to process more changes.
and when you do so, the feed response iterator will always have next.

int i = 0;
List<JsonNode> results;
while (responseIterator.hasNext()) {
FeedResponse<JsonNode> response = responseIterator.next();
results = response.getResults();
logger.info("Got " + results.size() + " items(s)");

// applying the continuation token  
// only after processing all events  
options = CosmosChangeFeedRequestOptions  
        .createForProcessingFromContinuation(response.getContinuationToken());  
i++;  
if (i >= 5) {  
    // artificially breaking out of loop - not required in a real app  
    System.out.println("breaking....");  
    break;  
}  

}

Please let us know if this not clear

Regards,
Oury