How to Fix: com.mongodb.MongoExecutionTimeoutException: Request timed out. Retries due to rate limiting: True.

Deepanshu Goyal 0 Reputation points
2023-01-20T07:54:32.59+00:00

Dear Support team, I am trying to fetch millions of records from DB In 10k batch.

But after some successfully fetching some batches, It throws below Exception:

com.mongodb.MongoExecutionTimeoutException: Request timed out. Retries due to rate limiting: True.

Please help me with a Java Solution, on how can I read all records from DB in any way.

Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,349 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Deepanshu Goyal 0 Reputation points
    2023-01-21T04:47:03.31+00:00

    this code block I am using to Fetch Products from DB. Which have StoreId = 3.

    
    int PER_BATCH_COUNT = 1000;
    List<Product> dataList = new ArrayList<Product>();
    int count = 77235; //no of records in DB
    
    int iterations = BigDecimal.valueOf(count).divide(BigDecimal.valueOf(PER_BATCH_COUNT), RoundingMode.UP).intValue();
    int iterationNo = 0;
     /*this works as a page no, to calcualte the    next start index*/
    int rowCount = 1;
    
    while(iterations != 0) {   
     iterationNo++;
    
    log.info("Getting result for Iteration No: " + iterationNo);
    int startIndex = (iterationNo-1)*PER_BATCH_COUNT+1;
    
    Query query = Query.query(Criteria.where(SearchParameters.STORE_ID.getAttributeName()).is("3"));
    query.skip(startIndex);
    query.limit(PER_BATCH_COUNT);
    
    dataList.addAll(mongoTemplate.find( query, Product.class));
    iterations--;
    }
    return dataList;
    

  2. ShaktiSingh-MSFT 12,231 Reputation points Microsoft Employee
    2023-01-23T13:39:48.5066667+00:00

    Hi @Deepanshu Goyal ,

    For Rate Limiting Timeout errors in Cosmos DB, it is recommended to increase Request Units.

    Azure Cosmos DB for MongoDB operations may fail with rate-limiting (16500/429) errors if they exceed a collection's throughput limit (RUs).

    You can enable the Server Side Retry (SSR) feature and let the server retry these operations automatically. The requests are retried after a short delay for all collections in your account. This feature is a convenient alternative to handling rate-limiting errors in the client application.

    Refer to the detailed documentation for the same here: Prevent rate-limiting errors for Azure Cosmos DB for MongoDB operations

    Hope this helps. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.