Getting null continuation token ItemLinqQueryable

Question

I seem to be having a strange issue when trying to use GetItemLinqQueryable method from the .NET implementation of the Microsoft.Azure.Cosmos.

Specifically when I try to get around 40,000 records I got around ~3000 or so before I get a null continuation token.

Strangely, If I remove the Where it will then successfully returns all results

Example code

private async Task<IEnumerable<TEntity>> QueryWithContinuationTokens<TEntity>(Container container, Expression<Func<TEntity, bool>>? searchPredicate)
{
    string continuation = null;
    
    List<TEntity> results = new List<TEntity>();
    using (FeedIterator<TEntity> resultSetIterator = container.GetItemLinqQueryable<TEntity>(requestOptions: new QueryRequestOptions()
           {
               MaxItemCount = -1
           }).Where(searchPredicate).ToFeedIterator())
    {
        // Execute query and get 1 item in the results. Then, get a continuation token to resume later
        while (resultSetIterator.HasMoreResults)
        {
            FeedResponse<TEntity> response = await resultSetIterator.ReadNextAsync();

            results.AddRange(response);
            
            _logger.LogInformation("Adding {ResponseSize} items from first request", response.Count);
            

            // Get continuation token once we've gotten > 0 results. 
            if (response.Count > 0)
            {
                continuation = response.ContinuationToken;
                break;
            }
        }
    }

    // Check if query has already been fully drained
    if (continuation == null)
    {
        _logger.LogInformation("No Continuation token, returning results");
        return results;
    }
    

using (FeedIterator<TEntity> resultSetIterator = container.GetItemLinqQueryable<TEntity>(requestOptions: new QueryRequestOptions()
       {
           MaxItemCount = -1,
       }, continuationToken: continuation).Where(searchPredicate).ToFeedIterator())
{
    // Execute query and get 1 item in the results. Then, get a continuation token to resume later
    while (resultSetIterator.HasMoreResults)
    {
        FeedResponse<TEntity> response = await resultSetIterator.ReadNextAsync();

        results.AddRange(response);
        
        _logger.LogInformation("Adding {ResponseSize} items from first request", response.Count);
        

        // Get continuation token once we've gotten > 0 results. 
        if (response.Count > 0)
        {
            continuation = response.ContinuationToken;
            break;
        }
    }
		return results;
}

}

Answer 1

@Andrew Rogers

Issue: I seem to be having a strange issue when trying to use GetItemLinqQueryable method from the .NET implementation of the Microsoft.Azure.Cosmos.

Specifically when I try to get around 40,000 records I got around ~3000 or so before I get a null continuation token.

Solution

Turns out the real issue was related to enums not being serialized correctly when using linq as described here:

https://github.com/Azure/azure-cosmos-dotnet-v3/issues/3491

However, it was difficult to identify this as the data we had was a mixture of integer and string enum values in the database, so it appeared as if it was only grabbing partial results but actually it was only grabbing the records that had integer values.

Regards,

Oury

Answer 2

Hi @Oury Ba-MSFT thanks for reaching out,

Turns out the real issue was related to enums not being serialized correctly when using linq as described here:

https://github.com/Azure/azure-cosmos-dotnet-v3/issues/3491

However it was difficult to identify this as the data we had was a mixture of integer and string enum values in the database so it appeared as if it was only grabbing partial results but actually it was only grabbing the records that had integer values.