Optimizing Bulk Operations for Cosmos DB

Question

Hi, I have an Azure Function that is inserting and deleting more than 10 thousand items from Cosmos DB. Each document is around 1 kb is size. I am trying to reduce the number of calls made by the Azure Function to Cosmos DB for each function run.

So, I am planning to use the Bulk support in Cosmos DB to batch several requests to the Cosmos DB into a single request. However, based on Bulk Support Documentation, there can only be 100 sub requests within the single bulk request. So, it will still require hundreds of Bulk Requests to Cosmos DB in a single Azure Function run.

Is there any way to increase the limit of the bulk request? Or is there another other way to perform Cosmos DB operation for large amounts of data within an Azure Function?

Answer 1

Hi @Khushaal Kurswani

Thank you for reaching out regarding optimizing bulk operations for Cosmos DB.

Currently, the limit of 100 sub-requests per bulk request in Cosmos DB cannot be increased. However, there are a few strategies you can consider optimizing your operations:

Parallel Execution: You can run multiple bulk operations in parallel to speed up the process. This can be achieved by using asynchronous programming and parallel tasks in your Azure Function.

Partitioning: Ensure that your data is well-partitioned. Cosmos DB performs better with operations distributed across multiple partitions.

Throughput Management: Adjust the throughput (RU/s) of your Cosmos DB container to handle the increased load during bulk operations. This can help in processing requests faster.

Efficient SDK Usage: Make sure you are using the latest version of the Cosmos DB SDK, which includes performance improvements and optimizations for bulk operations.

Batching Logic: Implement custom logic to batch and process items in chunks that align with the bulk request limit, ensuring efficient use of resources.

Continue Using Bulk Mode with Parallelization:

• While each bulk request can handle up to ~100 items, you can parallelize multiple bulk requests within your Azure Function.

• Use asynchronous programming with Task. When All (if using C#) or equivalent parallel execution in your language to dispatch multiple batches concurrently.

• This maintains performance and avoids overwhelming the Cosmos DB throughput limits.

Use Azure Data Factory or Change Feed:

• For very large, infrequent migrations or processing, Azure Data Factory with Cosmos DB connector can help with bulk ingestion or deletion scenarios.

• For deletes, you might even use a Change Feed processor to detect items to delete and process them asynchronously at scale.

Hope this helps. Please do let us know if you have any further queries.