Thoughts on coding ...

Moving to another blog engine...

Kaduk Frantisek — Fri, 26 Apr 2019 16:42:08 GMT

I'll continue my new posts here: https://fknet.wordpress.com/

How Skype modernized its backend infrastructure using Azure Cosmos DB

Kaduk Frantisek — Fri, 26 Apr 2019 16:40:13 GMT

Whole story starts here: https://azure.microsoft.com/en-us/blog/how-skype-modernized-its-backend-infrastructure-using-azure-cosmos-db-part-1/

CosmosDB change feed support for manual checkpoint

Kaduk Frantisek — Wed, 28 Nov 2018 22:55:16 GMT

It was quite long time I wrote last post about the change feed processor. Let's continue the series. This time, let's look into a manual checkpoint feature.

Basics

Let's refresh the knowledge. CosmosDB change feed processor SDK is a library for dispatching the document-related changes (inserts, updates) as a stream of documents. The documents can be split into multiple partitions (determined by partition key). The change feed processor is able to read the changes from all partitions and the reading is executed in the batches. In order to move forward with reading the documents, the change feed need to track the last processed batch. The tracking process is in the form of storing the continuation token from the last processed batch - that's done using "checkpoint" method. The checkpoints are kept in the lease documents.

Automatic checkpoint

By default, the change feed processor checkpoints the processed batches automatically. Updating the lease document takes some time and costs some request units (RUs). In order to fine-tune it, it's possible to set the frequency of this process. Let's look at the change feed processing logic for one partition in the pseudo steps:

read the change feed from the last known "continuation token"
dispatch the documents for processing by calling IObserver.ProcessChangesAsync
wait for the observer to process whole batch
if it's time to checkpoint (based on the configured checkpoint frequency), update the lease document with the continuation token from the last processed batch of documents. That's so called "checkpoint" step.
repeat from step 1

The whole process is shown on the following picture: In most cases, the above algorithm works without issues. But do you see a possible problem?

Stuck partition problem

It's the step 3. The automatic checkpoint algorithm waits until whole batch is processed and then checkpoints. While this waiting no other documents are read and hence processed. It could happen that, e.g. in the batch with 10 documents, there is one document whose processing takes several minutes. In such case, whole change feed partition processing is "paused" and you can't do anything about it. Such situation can be easily detected when change feed estimator is used (for more information see my previous post). See the following graph showing the remaining work on such stuck partition: I simulated that the processing of some document takes several hours. This stopped reading from the change feed (the number of polling reads per second dropped to zero). The remaining work started to grow. If there would be a defined SLA agreement stating that the events are scheduled for handling within 2 seconds from the time they were inserted into the collection, then the system would not able to meet it.

Solution - manual checkpoint

The solution is to use the manual checkpoint. It's possible to turn off that the automatic checkpoint process. See the following code: By doing this, the result lease document is never automatically updated. See the following: That's not good. After the process restart, the change feed would start back from the beginning. We need to checkpoint manually. That's possible using CheckpointAsync method exposed on the interface IChangeFeedObserverContext. But when? We can not wait for whole batch to be processed and then call CheckpointAsync. The simple solution could be like this:

read the change feed from the last known "continuation token"
dispatch the documents for processing by calling IObserver.ProcessChangesAsync
schedule processing all documents, this will create as many Tasks as we have documents,
register scheduled Tasks and IChangeFeedObserverContext instance into manual periodic checkpointer
repeat from step 1

See the sample code of the observer: The code above is for demonstrating purposes. It integrates the periodic partition checkpointer into the observer. The checkpointer is started when the observer is opened and stops when the observer is closed. In between, when the batches are dispatched for processing, the processing tasks are scheduled and the whole batch is enqueued into the checkpointer. The periodically checks for the last processed batch in the queue. For the running example check the sample here. After running the sample, the result is:

And the console would look like:

Prons and cons of the solutions

The above solution solves the issue with stuck partition, that's great (advantage). The change feed processor is reading the changes while the previously read documents are still being processed. But if the reading documents and scheduling the work is much faster than its actual processing then it's possible that the instance will be overloaded (disadvantage). It's necessary to limit the number of concurrent document processings, e.g. using Semaphore. That's all for now. In the future posts I'll describe the pitfalls of the change feed processing in general.

CosmosDB change feed processor 2.1 released (monitoring enhancements)

Kaduk Frantisek — Thu, 13 Sep 2018 21:55:58 GMT

There is new version of CosmosDB Change feed processor library released, version 2.1 and two main enhancements are:

Upgrade to Microsoft.Azure.DocumentDB 2.0 nuget package
Extensions to the monitoring

Microsoft.Azure.DocumentDB upgrade

The change feed library was built against latest SDK 2.0. Short! New SDK version brings several improvements and one of the most important change was the new multiplexing connection model. More about it in future blog post.

Extensions to the monitoring

When you take change feed processing seriously, I mean, production ready, you need to be sure that the processing of the feed is working as expected. You need to be sure that "the feed is flowing" which means the feed is in progress and as soon as there is a document change (insert, replace), the change feed processor receives it. I wrote about the monitoring in my previous post. The whole monitoring is built on so called "remaining work estimator" which estimates the number of the documents till the end of the feed. There was an improvement introduced into the library. It's possible to get the estimation per partition now! Why does it matter? Because it give you better visibility into the system. You are able to see what partition is left behind and how far. Let's see how to create the remaining work estimator instance: And let's see how to use it: And that's the result:

Real life scenario

I'm working on the event sourcing system powered by CosmosDB (I'll be writing about it in the future posts) and we are heavily dependent on the change feed. We need to be sure the system works 24/7. We need to be prepared for failures also when using change feed. In this case when a partition processing is stuck. So, we are monitoring change feed on several levels.

Who process what

We monitor what process consumes what partition. In other words, we are able to say what process is consuming what partitions. In practice, we record the following metric: data center, role, instance id, database account, partitions.

What's the estimated work left to process

We have a runner which runs periodically and reports what is the estimated work left per account/partition. That's the input for the graphs and alerting. It is built on top of the estimator shown in this post. If any estimated work hits a limit, it means we have a problem with "stuck" partition. See the graph of the simulation from our test environment: The red dotted line is an alert level. Once we are alerted, we are able to see what partitions are stuck. Because we record who process what, we are able to find out the instance which was processing the partition last time and diagnose the issue. That's all for now, happy monitoring! Previous posts:

CosmosDB change feed processor - leases and work balancing strategy

Kaduk Frantisek — Thu, 06 Sep 2018 19:51:16 GMT

The work distribution is the one of the core change feed processor features. The purpose of the change feed processor is to receive and dispatch all the documents from all partitions of the feed collection for further processing. In case of the high load, it's necessary to scale-out the number of processors, so that the documents are processed within desired time SLA. As you can see, the change feed processing system has to deal with it. The system has to distribute the work between instances somehow. That's the aim of the so called "partitions balancing strategy". The change feed processor library contains a default implementation of this strategy "EqualPartitionsBalancingStrategy".

EqualPartitionsBalancingStrategy

This strategy tries to distribute the work evenly between all active hosts. The algorithm is based on the lease documents. Just a recap. (Taken from my first post in this serie)There are two collections in the game:

Data collection - it's known also as a feed collection. It's a collection of the data over which change feed is processed. It could be created as a partitioned collection so that data are split into separated physical locations.
Lease collection - it's a collection where the change feed keeps the pointer of last processed document per partition.

Let's see an example of such lease document:

{
  "id": "account.documents.azure.com_EH9XAA==_EH9XAP7EOwA=..2",
  "PartitionId": "2",
  "Owner": "unique_owner_id",
  "ContinuationToken": "\"2468759\"",
  "properties": {},
  "timestamp": "2018-08-01T20:43:54.9476294Z"
}

There will be as many lease documents created as there are partitions in the feed collection.

Id

Each lease document has "id" which is in the following form: {prefix}{feed database account}_{feed database resource id}_{collection resource id}..{partition range ID} The prefix is used to differentiate the change feed processor groups if they use same lease collection. The rest of the parameters in the "id" is self-explanatory.

Owner

That's a string which is uniquely identifying the processor instance. Make sure this string is unique per each processor instance. Otherwise the same documents will be processed as many time as you have the processor instances. We used to name it the following way: {deployment id}_{processor name}_{instance id}

Continuation token

That's the pointer of the last successfully processed document change on the feed.

Timestamp

That's the epoch timestamp of the last change feed checkpoint time or heartbeat time. This timestamp is used by the work balancing strategy to determine whether the lease is expired or not. If the lease is expired, then it's available to be taken over by any change feed processor instance. If the lease is not expired, it means that the instance is active and the feed is processed. The aim of the strategy is to distribute the work evenly between all active instances.

Properties

That's the free form dictionary of string to string. The properties are the form how the strategy executions communicate between each other. It give you a possibility to create more advanced strategies, e.g. when shifting the load between "A/B" deployments. The properties can be manipulated by the strategy execution and then they will be persisted.

Configuration - Lease expiration interval

The strategy is configured by the ChangeFeedProcessorOptions.LeaseExpirationInterval. It's the interval for which the lease is taken by the particular owner. It's an interval determining how often the strategy is executed. So, if the lease is not renewed or checkpoint-ed within this interval, it will be identified as expired and ownership of the partition will move to another change feed processor instance. If the processor is stopped, the lease is freed-up. It means that the partition won't be processed until some processor instance takes it (which depends on lease expiration interval). There are special situations when you need to enhance the default work balancing strategy. Imagine you stop the instance. The partitions processed by the stopped instance are not processed until some other instance takes it over. This "gap" depends on the expiration interval. So, if you have strongly time-limited SLA, you will be required to write the custom strategy and leveraging the properties. Ping me if you need more information about it. Bear in mind, this is really rare condition and almost in all cases you will be more than happy with default strategy. We run quite heavy system using change feed processor with default strategy without issues.

Conclusion, consequences

In case the change feed processor instance crashed and then starts within lease expiration time, it will take back all previously owned leases immediately after start. In case the instance is gracefully stopped, the owner is reset which means the lease is free to be owned by other change feed processor instance. In case the number of change feed processor instances is scaled-out, the new ideal work balance is calculated based on the new active instances. The new instance(s) will try to take the ownership of partitions by 1 per each execution cycle configured by lease expiration interval. Keep in mind that the leases are stored in the lease collection. The collection has its own throughput (== money). In case you set low expiration interval, you can end up with the throtlling issues because leases are scanned once per lease expiration interval by each instance. This can cause that the leases are not taken for longer time and their processing will be delayed. Previous posts:

Multiple ways how to limit parallel tasks processing

Kaduk Frantisek — Sun, 02 Sep 2018 20:40:02 GMT

I decided to write this post because I saw several implementations how to solve the following interesting exercise across several reviews last days.

Exercise:

Let's have a list of items. We need to execute an asynchronous task for each of them in concurrently limited way (aka max degree of parallelism) and we need to collect all the results.

Solution #1 using TPL Dataflow library

There is a nuget System.Threading.Tasks.Dataflow used to solve so called data flow asynchronous processing tasks. Using its ActionBlock and the property ExecutionDataflowBlockOptions.MaxDegreeOfParallelism it is possible solve the goal. You can find more information here: https://docs.microsoft.com/en-us/dotnet/standard/parallel-programming/how-to-specify-the-degree-of-parallelism-in-a-dataflow-block but let's see how it could be solved.

Code

Result

Solution #2 using SemaphoreSlim

Another solution could be to limit the number of concurrent processing by a semaphore, it's kind of rate throttling.

Code

Result

Solution #3 using Enumerable.Range and Concurrent Queue

There is another solution based on the idea of running as many processing loops as we need degree of parallelism to be. The items are provided in thread safe way, using a ConcurrentQueue.

Code

The result

Solution #4 using Partitioner

The last solution is by using System.Collections.Concurrent.Partitioner which is aimed to split the input into the buckets (partitions) and process them in parallel.

Code

Result

That's all for now folks! All samples can be found here: https://github.com/kadukf/blog.msdn/blob/master/.NET/ParalellizedTasks

CosmosDB change feed monitoring

Kaduk Frantisek — Fri, 17 Aug 2018 13:15:58 GMT

Let's continue in the serie of the posts focused on the CosmosDB change feed. In this post we will focus on the diagnostics.

Goals

After running the change feed processor we need to be sure that:

the documents are processed from all partitions and
the age of the documents is within the limit and
the costs consumed by change feed processor is under control and
the communication between change feed processor and CosmosDB meets QoS requirements.

Let's deep dive each aspect from the bottom up.

The costs and QoS measurement

Change feed processor uses DocumentDB SDK to connect and communite with CosmosDB. It connects to two sources: document feed collection and lease collection. Just to recap:

feed collection is the collection providing the documents for processing
lease collection is a collection used by change feed processor in order to manage and distribute the partitions processing evenly between all change feed processors.

Change feed processor gives the developer a possibility to configure how to connect to the collections. By default it creates a DocumentClient under the hood. It also gives a possiblity to pass directly DocumentClient or IChangeFeedDocumentClient instances. We will use last option here (document client level). Using this approach it will be possible to catch all calls made from change feed processor to document client and meter the calls, frequency, costs and reliability. Let's define a metering reporter interface and its console implementation. Let's create a metering decorator for IChangeFeedDocumentClient: As you can see the change feed query has to be decorated too. Here is the decorator: And let's put it all together: That's all. It's possible to run it. From our experience we used this to see if we have throttling issues when accessing lease collections, when listing the leases, updating the leases, etc. It all adds to the feed processing time so it's worth measuring it. In addition to the monitoring, using the explicit document client give us a possiblity to fine-tune the connection policy (e.g. switching to direct TCP connection mode, open connection timeouts, etc).

Monitoring the age of the documents

This is the metric which defines how much the change feed processing lags behind. Each document has a field _ts (document level). This is a system property representing epoch time (it's the number of seconds! that have elapsed since 00:00:00 (UTC), 1 January 1970) when a document was last updated (e.g. create or replace). That's enough for this measurement. Let's see it in action: From our experience this is the one of the metrics we use to define SLA for our service where we measure e2e latency/processing time. It has one more requirement, time synchronization. So you need to ensure that the servers where the change feed processing is runnning are synchronized with closest NTP servers.

Processing the documents from all partitions are moving forward

This is the most import metric, I think. It's possible to do it on 2 levels:

observer level
document client level

As we saw in the first post from this series, the observer is the component which is getting the feed. Here is the interface: We will leverage OpenAsync/CloseAsync callback methods. The observer is called once the change feed processor instance opens or closes the processing of the partition. This could happen due to reasons like:

redistributing the partition processing load when scaling up/down the cluster or
redistributing the partition processing load when rollout/shutdown the cluster or
system error

Here is the complete list of the closing reasons: It's necessary to monitoring high and constant frequency of closing the partitions because it signals an issue, especially reasons Unknown and ObserverError. Unknow is due to internal change feed processor issues. In such case, inspect the logs (see previous post) and report the issue to https://github.com/Azure/azure-documentdb-changefeedprocessor-dotnet. In case of observer error the issue will be in your code ;) All other reasons signals transient issues and the system should recover from them. At the end of the day, you need to monitor that the number of open partitions is equal to the number of all partitions in the collection and there are no closings. This ensures that the processing is working smoothly. But that's not all, folks. It's also necessary to track that the partitions are being processed. It could happen that the partition processing is open but reading the feed per partition is stuck. Yes, we had such issues in past! So how to solve it? There is a cheap but "undocummented" way which we will try to get into the library.

Undocummented way

Reading the partition feed is done by change feed document client and its response has a session token and feed documents. Session token is in the format :. LSN is the last commited transaction number per partition. The document has also "_lsn" which is the LSN which commited the document change (create or replace). The whole point is to report that the reading the documents from the particular partitions occurs and that it is progressing till the end. One of the options is to report the remaining work (session LSN - document LSN).

Remaining work estimator

The previous approach was so call "undocummented". There is also other, documented approach built inside change feed processor SDK and exposed via IRemainingWorkEstimator. Let's see it: It has a drawback. The estimator calculates estimated work for whole consumer and collection, not just per partition. So the best option is to go with the undocumented way until it is exposed in SDK. We will fix it! This helped us to avoid incidents, especially in the early stages of integrating the change feed processor. Let's put it all together. As usual, the whole code is at my github repo. After running it, the output would be: Previous posts:

Logging in CosmosDB change feed processor

Kaduk Frantisek — Mon, 06 Aug 2018 14:41:16 GMT

The logging is important part of any library. It's the first thing the developers turn on in case they want to gather runtime information about the library and diagnose the issues. This post focuses on the logging aspect of the change feed processor library.

The logging integration

With new version of the library, there was introduced also new way of logging integration. While in the old 1.x version it was built on top of System.Diagnostics.TraceSource, the new version 2.x uses its own logging integration. The new logging integration automatically detects the logging framework (Serilog, NLog, log4Net, Enterprise library or Loupe log) used in runtime and uses it by default.

Integrating log4net

Let's see t in the action for using log4net: with the following configuration inside app.config The output will be: It's not necessary to do anything more.

System.Diagnostics integration

If your logging system depends on System.Diagnostics.TraceSource, don't worry. The new library version provides its integration. You will need to call the following: The logging integration is something that should be reviewed review because it should converge to the same practices as DocumentClient (especially when there is Microsoft.Extensions.Logging) . The main reasons the logging infrastructure diverges from DocumentClient is:

the lock contention issues when using System.Diagnostics.TraceSource
no support for the nested diagnostics context (NDC)

The TraceSource lock contention

In order to avoid the lock contention issue when using System.Diagnostics.TraceSource, you will need to configure your system the following way: This forwards all trace messages from DocumentClient as well as from change feed processor to destination but without causing the lock contention because all trace listeners are marked as thread-safe. Kudos to my colleague Roman for deep diving into this. Again, the whole example is published at: https://github.com/kadukf/blog.msdn/tree/master/CosmosDB/ChangeFeed/Sample2/DocumentDB.ChangeFeedProcessor.ConsoleApp

Benchmarks of the logging with LoggerMessage

Kaduk Frantisek — Sun, 05 Aug 2018 12:44:45 GMT

Recently, I came across the following site: High-performance logging with LoggerMessage in ASP.NET Core. It says that the logging using LoggerMessage.Define style is a high-performance logging pattern because of:

boxing of arguments when using extension methods
repetitive message template parsing

Fair enough, so I decided to do a benchmark comparing LoggerMessage vs Logger extension method vs. string.Format logging. I decided to do it because after checking the code:

boxing of the value types in LoggerMessage.Define style is happening anyway , just later, inside the class LogValues (see the method ToArray ).
The same is for the message template parsing (see Microsoft.Extensions.Logging.Internal.LogValuesFormatter.Format method).

The benchmark sample

In the sample for the benchmark I setup logging infrastructure without actually writing the messages anywhere. I wanted to measure just the overhead of the different logging style. The results are: As you can see, LoggerMessage style of logging is really faster then logging via extension methods but the difference is really small and the code change is quite significant. I saw a discussion that the team wanted to migrate the code base to LoggerMessage style because it's high performant logging. I wouldn't invest in it for already running project. At the end of the day, LoggerMessage looks promising, especially if the parsing the message template without boxing would be really implemented. I thought the LoggerMessage.Define parses the message template and generate a custom method on the fly for creating the result message. What do you think?

New Azure CosmosDB change feed processor released!

Kaduk Frantisek — Wed, 04 Jul 2018 10:59:04 GMT

Introduction

Microsoft Azure CosmosDB is an umbrella over different database services. One of them is the schema free database service called DocumentDB. You can read about it here, it has many great features and components. One of them is "change feed" which represents the feed of changed documents (upserts, replace) . The client component exposing this feed is called change feed processor. Change feed processor works by querying DocumentDB collection for changes. Then, it outputs the list of documents to the registered observer in the order in which they were modified. The observer can process the changes. After observer processed last document from the batch, the change feed processor could checkpoint (persists) the pointer of last document from the batch into the lease document (optional). The whole process then repeats. That's all folks! We use change feed quite heavily. It our core component. We built an event sourcing system with capture state pattern but I'll write about it in future posts. Now, let's focus on the change feed processor itself and what's possible with this component.

Package

Microsoft recently released new version (2.0.4) of change feed processor package. It's client side library built on top of DocumentDB SDK. The source code is located here. I'm one of the contributors to this project. The newly released package changed the new major package version, 2.x which means there were major changes, refactoring, etc. Upgrading the package from 1.x is seamless but there will be several warning messages generated about obsolete interfaces/classes usage. New package deprecates old construction logic and introduce new one. That's the main focus of this blog post.

High level

The following picture shows the idea behind the scene. As you can see, there are 3 parts which need to be provided from the developer's perspective:

Data collection - it's known also as a feed collection. It's a collection of the data over which change feed is processed. It could be created as a partitioned collection so that data are split into separated physical locations.
Lease collection - it's a collection where the change feed keeps the pointer of last processed document per partition.
Observer - a business logic responsible for processing the documents delivered over change feed.

Creating this whole machinery was changed and provides you different extensibility options. That is the focus of this series of blog posts about change feed.

Construction

It's possible to create change feed processor manually but the preferred way is to use the builder Microsoft.Azure.Documents.ChangeFeedProcessor.ChangeFeedProcessorBuilder. The most simple version is following:

Sample

I created whole sample showing this in the action. Source code is at https://github.com/kadukf/blog.msdn/tree/master/CosmosDB/ChangeFeed/Sample1/DocumentDB.ChangeFeedProcessor.ConsoleApp The sample requires local Azure CosmosDB emulator running. The sample console application ensures that the database with feed collection (Input) as well as lease collection (Input.Lease.ConsoleApp) is created. Then it starts the thread which inserts the documents and then it starts the change feed processing machinery with the console logging observer. The console observer logs when the partition processing is "opened", "closed" as well as when the documents are dispatched for processing. Because the sample requires at least 2 collections and Azure CosmosDB emulator supports wheter several single partition collections or 1 multipartition collection, just single partition collection is created in this case, so the processing partition ID is 0.