How to handle changes from different containers in Cosmos DB through Change Feed?

Ihor Dyrman 181 Reputation points
2021-01-27T16:46:34.947+00:00

I have some amount of containers in Cosmos DB that changes all the time. I need to provide some mechanism for reading all the changes from those containers.

I'm trying to implement builder/factory for Change Feed Processor (CFP). In my case, I have to create CFP instances dynamically for the different container. How I see the solution right now - I need a WebJob/Console Application that listens to the queue. When another application creates a new container in Cosmos DB it also sends a new message to the queue. Message in the queue contains all information for creating new CFP (connection string, collection name, lease container name, etc). The application creates new CFP and runs it in a new thread in the background forever.

Here is the code how I'm creating a new CFP

private void StartNewProcessor()
{
    new List<Task>().Add(Task.Run(async () =>
    {
        var container = Database.GetContainer(ContainerName);
        var lease = Database.GetContainer(LeaseName);

        var changeFeedProcessor = container.GetChangeFeedProcessorBuilder<Item>(ProcessorName, ProcessData)
            .WithLeaseContainer(lease)
            .WithInstanceName(InstanceName)
            .Build();

        await changeFeedProcessor.StartAsync();

        Console.WriteLine($"Change Feed Processor: {ProcessorName} have been started");

        Console.ReadKey(true);

        await changeFeedProcessor.StopAsync();
    }));
}

The problem is that it's a bad approach since there can be 100 and more collections in the future, so I'll need to create 100 extra threads in background. I'm looking for some ideas regarding the architecture application and how to do all that in the right way. It will be great if it is possible to handle changes for all containers in one application.

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,352 questions
Azure Cosmos DB
Azure Cosmos DB
An Azure NoSQL database service for app development.
1,466 questions
Azure App Service
Azure App Service
Azure App Service is a service used to create and deploy scalable, mission-critical web apps.
6,985 questions
0 comments No comments
{count} votes

Accepted answer
  1. Pramod Valavala 20,591 Reputation points Microsoft Employee
    2021-01-28T11:01:16.703+00:00

    Unfortunately, there is no way to work around this. Each container will require its own Change Feed Processor to listen for changes. Feel free to raise a feature request to include support for your scenario.

    But though you have many threads, most of them would be waiting on network for changes rather than performing any CPU intensive tasks. Based on that, one improvement would be to use a pull-based model for some containers that don't see lots of changes or don't require real time processing.

    0 comments No comments

0 additional answers

Sort by: Most helpful