A question regarding responseIterator from the queryChangeFeed method (java)

Jae Kim 26 Reputation points
2022-11-03T15:46:40.973+00:00

Hello,

I've been playing around with the CosmosContainer class method queryChangeFeed to fetch the change feed for a specific cosmos db container. According to the Azure documentation, upon obtaining the query result as an iterator, iterator.hasNext() will always be true given the nature of change feed being effectively an infinite list of items encompassing all future writes and updates. However, when I try a while/do-white loop with the condition that checks whether iterator.hasNext() returns true, I can see that the loop comes to an end, indicating that iterator.hasNext() in fact returns false at some point(whether it has reached the last item in given container, etc).

Could someone please let me know if I am misinterpreting the documentation, and whether this is an expected behavior?

To provide additional details, the code block looks something like:

CosmosChangeFeedRequest options = CosmosChangeFeedRequestOptions.createForProcessingFromBeginning(FeedRange.forFullRange());
Iterator<FeedResponse<JsonNode>> feedResponseIterator = container.queryChangeFeed(options, JsonNode.class).iterableByPage().iterator();
while(feedResponseIterator.hasNext()){
FeedResponse<JsonNode> = feedResponseIterator.next();
//do something else
}

Change feed(pull model) documentation: https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/change-feed-pull-model?tabs=java#consuming-an-entire-containers-changes
CosmosContainer documentation: https://learn.microsoft.com/en-us/java/api/com.azure.cosmos.cosmoscontainer?view=azure-java-stable

Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,500 questions
{count} votes

Accepted answer
  1. Oury Ba-MSFT 17,141 Reputation points Microsoft Employee
    2022-11-08T00:03:39.843+00:00

    Hi @Jae Kim

    This is the expected behavior, because if there are no more changes in the feed container, feedResponse will exit at some point (if it has reached the last element).

    In that case, you need to re-fetch the change feed data using the returned continuation token in the feedResponse as mentioned in the samples for change feed pull model here - https://github.com/Azure-Samples/azure-cosmos-java-sql-api-samples/blob/main/src/main/java/com/azure/cosmos/examples/changefeedpull/SampleChangeFeedPullModel.java#L81

    If you still face more issues, please create a GitHub bug here https://github.com/Azure/azure-sdk-for-java/issues and our team can take a deeper look at it.

    Regards,
    Oury


1 additional answer

Sort by: Most helpful
  1. Oury Ba-MSFT 17,141 Reputation points Microsoft Employee
    2022-11-08T21:28:37.1+00:00

    Hi @Jae Kim

    I checked internally with the content team and all of the information is true.
    If you see the full context, there is the continuation token example. Because the change feed is effectively an infinite list of items encompassing all future writes and updates, the value of responseIterator.hasNext() is always true. Below is an example, which reads all changes starting from the beginning. Each iteration persists a continuation token after processing all events and will pick up from the last processed point in the change feed. What it is stating is that after reading from beginning, then you can use the continuation token to process more changes.
    and when you do so, the feed response iterator will always have next.

    int i = 0;
    List<JsonNode> results;
    while (responseIterator.hasNext()) {
    FeedResponse<JsonNode> response = responseIterator.next();
    results = response.getResults();
    logger.info("Got " + results.size() + " items(s)");

    // applying the continuation token  
    // only after processing all events  
    options = CosmosChangeFeedRequestOptions  
            .createForProcessingFromContinuation(response.getContinuationToken());  
    i++;  
    if (i >= 5) {  
        // artificially breaking out of loop - not required in a real app  
        System.out.println("breaking....");  
        break;  
    }  
    

    }

    Please let us know if this not clear

    Regards,
    Oury