Here are some tips on the best way of using it in a service that you expect will be running 24/7.

- IFF you create a new HttpClient per request you should Dispose it (but don't create a new HttpClient per request).

The Dispose method is where the HttpClient cleans up the HttpMessageHandler. So if you don’t Dispose it and you keep creating new HttpClients per request you will eventually get out of resources. I recommend the "using" pattern to do this automatically for you.

BUT

- Do NOT recreate HTTPClient for each request. Reuse Httpclient as much as possible

When you create a new HttpClient to make a request there is a lot of overhead. A new connection will have to be established for each request so you'll have your Three-way handshake and if you are going through HTTPS the cost will be even higher. If you are interested in understanding services performance I recommend a lot that you take a look at "High Performance Browser Networking".

Httpclient was written so that you would keep it alive for as long as your application is alive. The methods on it are threadsafe and it will save you the latency cost of establishing a new connection for each request. And you don't even need to worry about adding locking primitives to it. It just works!

This is particularly important in services because you might be paying this cost for each customer request. So even though in your tests it doesn’t seem like the impact is a lot, once you deploy you will see this impact. I’ve actually seen profiles that indicate 5-7% CPU usage just spent on tearing down the HttpClients (this is a lot!).

Making it static is fine. This might not always be possible because you might need different properties but you should try to do it for related requests.

You can also use something like ConnectionGroupName  to group multiple requests.

- Use timeouts to control the behavior of your requests

Finding out if a connection was established is normally very quick. But if there is a network partition it might take longer. If you are making cross datacenter calls your latency might be even higher. Depending on what your service does, the default value of

HttpClient.TimeOut might not be what you want so make sure you add the right value to it.

- Reuse connections when you can (httpClient.DefaultRequestHeaders.ConnectionClose = false or HttpWebRequest.KeepAlive = true)

This setting controls if HttpClient will close the connection to the service after each request. "Obviously" this is not what you want to do so make sure you have set it to false.

However some services might actually ask you to close the connection after each request. In particular if your request goes through a load balancer by keeping the connection alive your requests will not go through the load balancer so you might end up getting throttled. I've seen this with some Azure Services. If that is the case see the following recommendation

- Don't hold those connections forever

If you set Keepalive to true then those connections will be held forever. If the backend machine is down it might take a long while for your service to realize that is the case (and during this time your service will not be available). Also as mentioned before this does not play very well with LoadBalancers.

But there is a setting that allows the client to close the active connection after some time has passed and refresh the connection to the backend. This is exactly what we want.

ServicePointManager.FindServicePoint(new Uri(url)).ConnectionLeaseTimeout = ConnectionLeaseTimeoutMs;

- Error management

HttpClient tries very had to not throw. This allows you to examine the Response Status Code and based on that log the result, retry, backoff, etc.

One exception of this is when there is a TimeOut. In this case The HttpClient will throw a TaskCanceledException so you can catch that and do what you need.

Some years ago my then boss was going on vacation. Before he left he asked me to investigate how to improve the performance of a WCF service that was being developed at Microsoft. So I did due diligence and talked with the owners of the system and they mentioned that at some point in the past they had taken profiles and that WCF was spending too much time deserializing headers. 

After that I talked with some WCF experts and they mentioned that indeed it could be the case that cracking open the headers to grab/modify a value could take some time. This being a messaging system I figured that was the problem and started working on fixing it.

My solution was (I thought) very clever. This service had a couple of parts. See the attached diagram

The first part was the Frontdoor which received the message from our customers, added some headers (the message would have headers from our customers so we would just append new ones), modified some others and then forwarded it to a different server

The business logic server would then take that message and use the information in the headers that we added to decide what to do with it (persist it, run some procedure, etc).

And the reason why adding and receiving headers can be expensive in WCF is that they use XML so we need to create a valid XML document and deserialize it to grab the property value. So my solution was to allow the front door to set a property instead of a header. Now typically these are not passed through the wire so my extension would take the message properties and append them to the message buffer. Then the business logic layer would have another piece that removes the property from the message buffer and creates a new property with that value. So there was cost of moving data around but it was measurably faster in my micro-benchmark (6-7% faster if I remember correctly).  

The project was very fun, I had to read the WCF Framing specification, learn about MultiByteInt31s (unsigned 31 bit integers in a variable length packet), how to create my custom payload so that it would not break WCF and how to insert it efficiently with minimum copying of bytes and with little pressure on the Garbage collector. It took me a couple of weeks of coding and testing it.

All of this was fine until I went back to the team and asked them for an end to end test to validate my improvements.

And that's when I had to face reality. Even though my micro-benchmark showed a measurable improvement in the end to end test the performance gains were not what I expected. I saw less than 2% improvement and profiles showed that the bulk of the CPU was being spent elsewhere.

This really brought home to me the point of measuring before working on a performance improvement. Of course I learned quite a bit about WCF so I would like to think it was not wasted time :)

So my recommendations when you are trying to fix the performance of a product are:  

• Measure before you start coding.
• Do not trust "word of mouth" data 
• Do not trust micro-benchmarks. it's very easy to game them. Even if you wrote them yourself.
• Look at the profile with your own eyes :)

So for those interested I have put the code in github. I've tried to comment the code so that it's obvious what each piece is doing. Here are the interesting files (the ones doing the bulk of the work are the first two):

• WrappedEncodingBindingElement - This bindingElement class is the one that actually modifies the MessageBuffer on the WritePath and extracts our property from the MessageBuffer on the ReadPath
• BinaryFormatHelper - This is the Helper class that modifies a Buffer to include the desired payload (AppendPayloadAsHeader). It also verifies if a given buffer includes our payload and if it does it extracts it from the buffer (GetAndRemoveHeaderPayload). It uses bufferManager for pooling our buffers (avoid generating garbage) and it also uses Buffer.BlockCopy to copy data efficiently.
• OutOfBandPayloadProperty - Our custom property. Includes the payload and the name so that we can distinguish it from other message properties
• WcfHelpers - Helpers for creating WCF messages, custom bindings and the test service
• Program.cs - includes 3 tests: EncodeDecodeNoWcf, EncodeDecodeWithWcf and EncodeDecodeWithSecureWcf

Again, learn from my mistake. I would only use something like this if you can measurably proof that you are spending too much time serializing/deserializing headers for your real scenarios.

Relational Databases: I had experience with SQL Server of course and when I worked in Azure Service Bus I learned how cool SQL Azure can be (or Microsoft Azure SQL Database as it's called now). It’s really a relational database in the cloud! I really liked the fact that you could have an application that worked on top of SQL Server and then use pretty much the same application in Azure. As some people know, Service Bus can be installed in your own servers and how you do that is by installing it on top of SQL Server. It was amazing to me that you could use the same profiler and use store procedures in both. It was almost perfect. Of course after working with it for a while I got to experience firsthand the disadvantages that it can have for a large project. At some point I felt as if we were trying to fit a square peg in a round hole. But I still remember how cool it felt that it was even possible. Overall that was a good learning experience.

Key/Value Storage: Of course when I worked in Azure I also got to work a lot with Azure Table Storage. It’s very interesting too: it’s really scalable and it forces you to understand how your data is going to be accessed so that you can partition it efficiently. And once you do that you don’t have to worry about having too much data. It helps you focus on the data, without having to worry about constraints/schemas/indices/relationships and stored procedures. All you need to worry is the data. Very simple and very powerful. However, to be honest I felt it was too much of a jump from the concept of database I had learned in school. Having to put all the business logic in the application layer felt to me closer to an Excel file than to a Database.

And then I got to work with MongoDB. 

 So what is Mongo?

• It’s a Document database. This means that we don’t have the concept of schemas. Your data is persisted in collections of documents.

  However unlike Azure Table Storage the documents are nested data. You could have documents inside your documents. Whatever makes sense to your application. So there is no need to flatten your data to match a “row”. You can think of documents almost as an object in a programming language.

• It’s a JSON database and you use Javascript language to talk to it. At first I was skeptical of this but I got used to it very quickly. It also makes it very easy to learn for new developers.
• It allows querying the nested data for your document. You can query values in arrays, etc. So the document is not just a blob.
• As I said there is no schema: some documents might have fields that other documents don’t have. This is very handy when adding new fields. You can have old and new data living in the same database without having to do data migration.
• You can add indexes to the nested data! This can make it really fast
• It’s super versatile. It allows changing the actual DB engine to one that fits your needs (more on this later)
• It’s really scalable

Scalability

• With a relational Database you typically need to scale up if you want to improve performance. That means that you need to add more memory or a better disk or even change to a more powerful machine. This gets expensive very quickly and there is a physical limit on how powerful you can get it.
• Instead of this Mongo supports Scaling out. That is: you can add extra servers to your existing database servers and use them to increase availability (replication) or performance (sharding). 
• Sharding means splitting your collection into pieces. This can help with throughput because different servers will be serving the different pieces.
• Replicating means copying data to other servers. This is done through Replicasets in MongoDB and they give us data consistency and partition tolerance. If one or more servers go down we can keep working.

Engines…

MongoDB was designed as a Pluggable Storage Engine. This means that if your application has a particular pattern (writeheavy or readheavy) you can select a different storage engine. This avoids “one size fits all” that make a DB not good for certain use cases

• MMAPv1 is a good default engine that has collection level concurrency control. However as of the latest versions of Mongo it has been slowly deprecated for Wiredtiger.
• WiredTiger provides more granular concurrency control. For some benchmarks it shows better performance (up to 10x) than MMAPv1.
• Facebook released the Rocks storage engine (write-optimized)
• TokuMX is an engine that uses data compression & Fractal Indexing to improve performance (Open Source)

Profile inside Mongo?

Unlike other databases the profiler is part of Mongo from the start. You can easily enable it for your application with just one line:

• db.setProfilingLevel(1)

   This will take a profile of every operation that takes longer than 100 m

Then you can sort to find the slowest ones like this:

• db.getCollection('system.profile').find({}).sort({millis:-1})

In addition of looking at the profiles you can also look at the logs where the slowest operations are logged by default.

MapReduce and Aggregation framework

Once you have your data it’s pretty typical that you want to extract information from it. Instead of having to pull all the data and doing your processing in your application layer MongoDB allows you to take advantage of the server to do some of this.

It has full support of the MapReduce paradigm. This allows you to do grouping operations on your collections such as counting, sums, projections, etc. The best part is that it can be done on your shards so you can do parallel processing of large amounts of data. And this works both for input and output so your results can go to sharded collections as well.

This is good but overkill for most reporting needs. Mongo also has support for in memory aggregation operations. These are really fast and almost as powerful as MapReduce.

Conclusion

• No joins (sometimes this means that you need multiple trips to the DB)
• No stored procedures (but you get a powerful Aggregation pipeline and MapReduce support instead)
• No transactions (you need to implement two-phase commit yourself)

But you get a

• Document DB
• Very powerful Indexes
• Familiar Javascript language to query it
• Aggregation pipeline and MapReduce support
• Replicasets and Sharding
• Multiple engines

It’s fast, it’s scalable and it’s flexible. 

 I spent the past year working with MongoDB. Since our service was used for Halo and Call of Duty we had very aggressive scale and performance targets. So as part of our stress runs we identified some scalability issues that had their root cause in our data layer. Luckily MongoDB is a database that has great tools for identifying performance problems. Here are some of the approaches I did for debugging/fixing performance problems in Mongo.

Normal operations take too long

Slow Queries: This is the most common and the easiest to fix. You need to find out if you have an index that covers the query that is slow. As I mentioned before having indexes will speed up significantly your queries and as a general rule you want to have them for most of your queries. Optimize your queries with indexes!

Slow updates or inserts: This one is trickier! :) If you follow the previous recommendation you might find that your DB writes are too slow. Here are some operations that can slow down your updates

• You are grabbing and or modifying large arrays inside your documents.
• You are modifying fields that are part of an array index.
• You increase the size of the document you are modifying. This could cause it to move which will cause some IO cost while Mongo is moving stuff around

The solution to these might sound like workarounds but they help:

• Limit the size of the arrays inside your documents
• Add padding to the documents when you create them if you know they will grow.
• Change your schema to simplify your queries.

 In our case we found that having indexes would help updates as well (because it would speed up the time it took to find the document to modify). But whenever the rate of updates/finds increased we found our finds would take longer (see below for the reason).

Normal operations are slow but they should not be slow

 We saw this in our system a lot. Some of the queries would show as optimized in the .explain (See the previous blog for an explanation on what that is) but they would take too long to execute. Or they would show up in our profile as taking longer than 100 ms.

In our case we found that they were blocking waiting on slow operations such as updates.

You can confirm this by using the top command. This tells you how much time was spent locking for reads or writes. You can also look at the profile for the queries and it will tell you how much time it spent executing vs yielding to other operations.

Basically in MongoDB the writer is greedy. What this means is that if there is a write and a read operation the reads will yield to the write. So even if you have a fast query (that takes milliseconds to run normally) it might take very long because it’s yielding to slow updates (It's a little more complicated than that. Concurrency behavior changes depending on the engine you use and also if you use sharding or replicasets. See the FAQ for the latest update that applies to your configuration).

You have a couple of options:

1. Follow the previous recommendations on speeding up updates
2. Consider sharding your data so that your reads don’t wait on that many writes.

Slow Aggregates

Aggregates in Mongo are super powerful! However unlike other queries they will be touching most data in the Database (because you probably are trying to generate aggregate data or some kind of report). So they can easily become the bottleneck of your service. Also as I mentioned in a previous blog post the .explain for it is not great because it will tell you information about the indexes but not about memory. 

• The first and very important tip is to take a backup of your production Database and use that to play around with the aggregates without fear of affecting your production system. Since it’s a backup with production data you will be able to measure your improvements. Super useful!
• Focus on Memory usage. MongoDB has a Hard limit for the data that is passed in the aggregation pipeline of 100MB. You can turn on Disk usage to go around this but it will really slow down everything. So make sure you are under this limit. (And yes you can save the output of the aggregates to a collection, this can can be great but there is a high perf cost to IO so you still need to reduce the memory usage as much as possible). I found that doing the aggregate in memory is the way to go, so making sure that we were passing the least amount of memory through each stage improved performance quite a bit. Use $Project as early as possible to remove fields you don’t use!
• Try to avoid tablescans or visiting every document in your collection! Again by filtering out documents that you don't care early in the pipeline you will improve performance. Use $match as soon as possible to make the aggregation pipeline happy!

• If you are using an operator like $sort, it has to go through all the documents. This can get very expensive if you haven’t done project + match before.
• Unwind creates new documents which duplicates data 

• Index will help both match and sorts. So use indexes as much as possible. Note that this are only useful before you transform stuff (project, group, unwind) so make sure you measure that it helps (otherwise why add an index that is not helping?)

Make sure you MEASURE your fixes through all the pipeline. That is the great thing of using a production backup. You can get an idea of the performance impact of your changes.

Combining Sort and limit will keep around only the amount of documents that you passed to limit. So even though it will need to go through the entire collection the aggregation pipeline is smart enough to discard the documents that it does not need. Good for memory!

Project is supposed to be optimized (as in, the pipeline will try to infer if you don't need a field and move the project to the earliest possible moment). However in my local tests I found that making the change myself was much better. 

So try to reduce work for the next stage in the pipeline in each step. And after every optimization make sure you measure!

Some other options (that we did not need to do)

1)      Upgrade your hardware! Make sure your working set fits in RAM. We use pretty beefy machines for our Database so we did not need to worry about this.

2)      Use fast disks (SSDs) to speed up writes and reads (try to avoid page faults if you can)

Good luck!

It’s been very fun! Here are some of my learnings related to indexes and how they are different to other DBs. Hoping this will save people some time when they need to work with MongoDb. 

How are Indexes special in MongoDB?

MongoDB allows you to do different operations to the data you store in it. A very big feature compared to other Document DBs is that it supports secondary indexes on its collections. You can create simple and complex indexes and they will be used for speeding up finds (queries), matches (in aggregates) and sorts. It has support for indexes on multiple fields, on arrays, on geospatial fields and even text indexes to speed up searching for string content.

There ain't no such thing as a free lunch though:

• Indexes speed up queries significantly but they slow down writes.
• Updates are a special case: they might speed up finding your document but if the value on any indexed fields is changed the index needs to be updated. So you might get a similar impact as in inserts.
• In my measurements I saw reads could improve up to 1000% but writes slowed down almost 30%. Overall it should be a big gain since we do a lot more reads than inserts.
• Something of note is that they not only affect write operations. They might slow down read operations as well (such as aggregations). This is because aggregations yield to writes so if your writes are slower your aggregation will take longer to complete. So make sure you measure your performance interleaving writes+reads (or run in stress).

  I think MongoDB’s QueryOptimizer does a good job. It can be overriden (.hints and $natural for forcing full collection scans) but I could not find a place where it's needed. It's nice having the option of course. 

How to know if an index is being used?

 In the Mongo shell you can use .explain() for normal queries and runCommand+explain for aggregates. Examples:

1)      db.myCollection.find({}).explain()

2)       db.runCommand({ aggregate: "myCollection", pipeline: [{"$match":{}}], explain: true }) 

Compound indexes

Compound index is when you have more than one field as part of your index. Like this:

Index: {ServiceId, LastRecordedState}

Query: {ServiceId, LastRecordedState} – Uses the index of course

Query: {ServiceId} - it WILL also use the index. This is pretty nice. Because of this once you have a compound index you can remove your simpler indexes if the compound index is a superset.

Query: {LastRecordedState} - index will not be used because there is no ServiceId in the query

SORTS:

db.collectionName.find().sort({ServiceId: 1,LastRecordedState:1}).explain() uses the index

db.collectionName.find().sort({ServiceId:-1,LastRecordedState:-1}).explain() uses the index

db.collectionName.find().sort({ServiceId: 1,LastRecordedState:-1}).explain() does not use the index

db.collectionName.find().sort({ServiceId:-1,LastRecordedState:1}).explain() does not use the index

Intersect indexes

 If you have these two indexes:

Index: {ServiceId}

Index: {LastRecordedState}

 Mongo will try to combine the results of the indexes. You can verify this by looking at the “explain” for your query and find a stage named: AND_SORTED

Query: {ServiceId, LastRecordedState} could try to combine both queries. However, this depends on the Query performance. In my tests the query with the simple Index would be selected always. (Because LastRecordedState would return too much data)  

Names for your indexes

You can use up 31 fields in a compound index, but the default name will be too long. So you should pass a different name:

Other tips:

• You can create indexes before the field exists in the collection. Or even before the collection exists
• You can query if indexes exist: db.system.indexes.find({ns:"databaseName.collectionName"})

If the query uses information that exists in the index the result could be much faster. Most of the time this won’t be the case (you might need to disable some fields using a projection), you can see in the explain if the INDEXONLY value is true or fales. 

 How to make sense of the output of Explain:

As mentioned before you can use the .Explain option to evaluate if a given query is using indexes or why it’s taking so long to execute. Here are some tips on how to make sense of the output of this:

"queryPlanner" : {

        "plannerVersion" : 1,

        "namespace" : "databaseName.collectionName",

        "indexFilterSet" : false,

        "parsedQuery" : {

            "$and" : []

        },

        "winningPlan" : {

            "stage" : "COLLSCAN",

            "filter" : {

                "$and" : []

            },

            "direction" : "forward"

        },

        "rejectedPlans" : []

    },

"executionStats" : {

        "executionSuccess" : true,

        "nReturned" : 12191,

        "executionTimeMillis" : 16,

        "totalKeysExamined" : 12191,

        "totalDocsExamined" : 12191,

WINNINGPLAN is the plan that Query Analyzer picked because it finished faster. It will cache it and reuse it for future queries (until you add indexes, reindex, make changes to the collection, etc)

Stages:

COLLSCAN for a full collection scan (did not use your index. You do NOT want this)

IXSCAN for scanning index keys (used the index to find)

FETCH for retrieving documents that it found (most of the time you will see this combined with IXSCAN)

KEEP_MUTATIONS stage: When documents are deleted or updated some Query stages can receive invalidation notices. Some stages flag documents that are deleted. The set of flagged documents is then read by the KEEP_MUTATIONS stage, which re-checks these documents against the query and returns them if they still match. Note that this is not used for the WiredTiger engine, only for the MMAPv1.

"AND_SORTED": You will see this when Mongo used a combined index

queryPlanner.winningPlan.inputStage.stage displays IXSCAN to indicate index use.

executionStats.nReturned displays 12191 to indicate that the query matches and returns 12191 documents.

executionStats.totalKeysExamined display 12191 to indicate that MongoDB scanned 12191 index entries (Note that this is a Full table scan).

executionStats.totalDocsExamined display 12191 to indicate that MongoDB scanned 12191 documents.

Does Event Hub support batch send?

YES!

Azure Event Hub was developed by Microsoft to support IoT scenarios. It was written by the same team that owns Azure Service Bus (that is why it’s under the Service Bus Namespace) so they share a big part of their codebase. As a matter of fact I worked on the Batch Send feature to improve the throughput of the Http code path for Eventhub. But as we were working on it we made sure it was working for Service Bus as well.

Since I’m serializing as JSON do I need to change something in the Receive?

Nothing changes in the receive path. Json is only used to create the batches of messages, but in the receive code path we will be receiving them separately. The batch is gone by then.

Can you show a few samples on how to send different payloads?

Like I said in my previous post this is the format for an Http Message:

  You don’t need to send the fields that you don’t use. So if you are only sending a text payload you can just send a message like this:

  This is equivalent to the Non-Batch Http Send. The receive code path does not change. You would just receive it like you would receive a regular text message (no need to deserialize, we default to text serialization).

However in the past you have been able to set some Broker Properties in your Http message (such as CorrelationId, SessionId, PartitionKey, TimeToLiveTimeSpan, etc) and we still support that. You can just set those as part of the BrokerProperties Dictionary and we will set them in the BrokeredMessage.

And again if you want to set your own Properties you can add them to the UserProperties Dictionary and we will populate the BrokeredMessage.Properties with them

And finally if you want to send an arbitrary object instead of plain text you can do it by using the IsBodyBase64 setting.

Here is an example of sending an object of a custom class called MyTestObject.

1)

2)

3)

As you can see we do the following:

• Define the class of our custom object
• We create a new object of this class
• We JsonifyTheObject (This creates the message that we will insert into a batch)

- Use BinaryFormatter (from System.Runtime.Serialization) to serialize our object.
- Convert that to a Base64 Encoded string (This is the body)
- Create an HttpBrokeredMessage that has this newly created body
- Serialize this to Json with NewtonSoft.JsonConvert

• Now we have a json string that contains one message. We can do the same with the rest of our messages and send them as a Json array “[message1, message2…]”

Now the interesting thing of sending binary messages is that when we are receiving them we can just do a Receive<MyTestObject> and the .Net client will do all the hard work of deserialization for you. So the code in the receive path is a lot simpler :)

Accepted types for UserProperties

We accept the same types used for regular Rest Send:

·         Boolean values

·         Null

·         Integers

·         Doubles

·         Strings

Sample receive code

 The receive code path has not changed at all. From the .Net client you can just do a Receive and get back your message. From Http you can use code similar to this:

For Text:

For Binary objects:

}

How to debug any problems with Batch Send?

1)      If when you receive the message you receive one IEnumerable object with n elements in it that means you are not going through the Batch Send pipeline. Double check if the Request content type is correct. It should be "application/vnd.microsoft.servicebus.json"

2)      If you get a 400 error when you send then that means something is wrong with your message. You can use this code to get more information from the web exception:

I have been a performance developer at Microsoft for a little more than 4 years. During these years I have done performance investigations for Visual Studio (Workflow Designer), .Net Framework (Windows Communication Foundation), Azure Service Bus, Azure Event Hub and Azure Api Management. And still some friends at Microsoft are not sure what I do in my day to day work.

Typically Microsoft used to have 3 clearly differentiated roles (This has been changing lately): Program Managers, Software Developer Engineers and Software Developer Engineer in Test. Some SDETs specialize in UI testing or Stress testing. The PM role has a couple specializations: Release PM or Feature PM. The Developer has a few specializations as well: one of these specializations is the Performance Developer.

We are developers that focus on finding, analyzing and fixing performance problems (throughput, latency or scalability). We also work on adding performance features and since performance and scalability are so closely related we are also pulled-in to work on issues that might affect how a system behaves at scale. So part of our day can be spent looking at a profile or we can be looking at crash dumps to figure out why a given product is crashing when it’s being hit by thousands of requests.

I personally work in a group called "Azure Application Fundamentals". This team owns the Stress team, the Performance team and the Servicing team for the Azure Application group. What this means is that any team that is part of the Azure Application can ask us to help them with performance investigations.

This is a list of some things we do as part of our daily work:

Measure scenarios: Work with the feature team to identify main scenarios that we want to measure, establish goals, test authoring, ensure the tests are getting executed regularly and that the results are reported somewhere.

Regression Prevention: Make sure that the performance of the main scenarios improves as time goes by.

Performance Analysis: Jump in and analyze the performance of any feature important to the team and figure out why it’s not meeting goals.

Optimization: Fix the performance problems or implement performance features to improve the performance of the product.

Education: Teach the feature team what things are not good for performance and pretty much make sure they can do any of the previous steps themselves. Also share what you learn about the different tools used to measure and analyze performance problems.

There is a lot of case by case investigation but something important is that we try to make sure that the teams that are under our purview are measuring and reporting all their performance scenarios to a centralized Reporting system. This makes it easier to understand how the team is related to performance. It makes it very clear to understand when things are getting better or worse.

I find a couple of things interesting about this role. It lets even junior developers learn about performance, investigate profiles and memory dumps and basically grow until they become more knowledgeable about performance. In my experience not a lot of developers know what to do when something is slow. Having a specialized role encourages sharing of this information.

Also it's not very common for a team to be interested in investing in performance during the early parts of the development cycle. They want to have something to show as soon as possible. However what I've seen is that if you leave the performance as the last thing you do you will not be able to fix much. You will find that your design has certain performance characteristics that are simply too risky to change at the end of the lifecycle of the product. You might also find that everything is slow: there is no clear "smoking bullet" to change. This can be because you have used pervasive slow techniques across your entire product. This causes in my opinion a lot of teams to just ship something even though they know it is slow. Having someone from outside focusing on performance helps to avoid this.

So for me it’s been a very cool experience and I highly recommend it!

If you are interested in performance here are a few blogs of other Performance Engineers at Microsoft:

Vance Morrison

Rico Mariani -

Sajay Antony- His blog has not been updated in a while but he has some neat projects in his Github

Dustin Metzgar

Bruce Dawson - Ex Msft – has fantastic posts about xperf

I was assigned as his mentor and since I was working for the Azure Service Bus team I chose his project to be “Amqp for Android”.

Jimmy did such a fast job that after about 7 weeks we had to find another project! :)

So here is the github link to the work he did:

https://github.com/AppPlatPerf/qpid-proton/tree/AndroidProton

He basically solved all the hurdles so that people that want to use Service Bus / Event Hub from Android devices can do it very easily. It builds on top of the Proton-C Amqp library.

Hope it's useful for people out there!

Proxy

Let’s start from the beginning. What is a proxy?

A server that acts as an intermediary for requests from clients seeking resources from other servers (Wikipedia)

There are many reasons why you would want to have a Proxy server. The intermediary can add value to the request or to the response. For example it can make sure your backend server is not getting too many requests. It can also add caching so that your backend server is not even called. Other examples of value add are: security, compression, encryption, load balancing, etc.

For the past few months I've been helping out a team in Microsoft that has one of these services.

 The default implementation of a proxy would be something like this:

1) The Proxy receives the request
2) The Proxy makes a request to the backend
3) The Proxy gets the response from the backend
4) The Proxy passes this response back to the user.

So the Proxy will have two streams for each request. One stream for the Backend Response and one stream for the Proxy Response that we are generating. And as we discussed above it should be copying data from the origin (Backend response) to the destination (Proxy response) as fast as possible.

The default way to copy data from one stream in .Net is something like this:

await backendContent.CopyToAsync(context.Response.OutputStream);

This implementation has a few flaws:

1)      It allocates too much. Like I’ve mentioned before Stream.CopyTo allocates an 80k buffer each time it is called. So the proxy would be allocating an 80k buffer regardless of the stream size for each request! This might not be a problem if it’s something that you only do once every month. But for a Proxy server this is not acceptable. Too much memory pressure.

There is a version of CopyToAsync that allows you to control the buffer size:

await backendContent.CopyToAsync(context.Response.OutputStream, _readChunkSize);

2)      The problem with this method overload is that you need the ability of passing your own buffer. Otherwise every request will allocate memory. You need to be able to pool memory so that you don’t allocate too much. See my previous blog on how to do that.

There is another problem with both implementations.

3) They are slow

Let’s measure the 3^rd point. I grabbed a couple of Azure VMs and did this:

1) Deploy Backend in Azure West (All it does is return a file with an arbitrary size)
2) Deploy Proxy in Azure East (All it does is call backend and return the same body)
3) Call Proxy from my dev box.

I used Fiddler to call my Proxy and measured how long it takes to download a 50MB file through it:

       Overall Elapsed:     0:00:24.021

It takes more than 20 seconds to download a 50 MB file. We’ll see how much we can improve this.

Reason of the problem

Let’s look at the implementation of Stream.CopyTo (which calls CopyToAsync:

    private async Task CopyToAsyncInternal(Stream destination, int bufferSize, CancellationToken cancellationToken)

    {

      byte[] buffer = new byte[bufferSize];

      while (true)

      {

        int num = await this.ReadAsync(buffer, 0, buffer.Length, cancellationToken).ConfigureAwait(false);

        int bytesRead;

        if ((bytesRead = num) != 0)

          await destination.WriteAsync(buffer, 0, bytesRead, cancellationToken).ConfigureAwait(false);

        else

          break;

      }

    }

As you can see it always dominated by the slowest part (reader or writer)

* If the writer takes too long to write, the reader cannot read more data.
* There is only one pending write on the wire.

So even though it is not blocking threads (it’s using await) it will not be as fast as possible. The underlying stream supports multiple writes on it. So there is no reason why we need to wait until it finishes writing it before we start writing another one.

Here is an example:

If your writes take 10 milliseconds to write to the wire you will be bound by the amount of data you can push every 10 milliseconds. So if each write is only pushing a 4K buffer you will be able to send only 400K data per second (4K * 10 * 10).

If you are able to read much faster than 400K/second this is not acceptable. What you want is to have the ability of scheduling more than one write at a time. This way you will be able to saturate your network link.

Proposal: Consumer – Producer pattern

My proposal to fix this is to decouple the reads from the writes. Of course we can only write what we have read but read and write operations happen at different rates. So I think it makes sense to make them independent.

A way to do this is to follow the Consumer/Producer pattern.

1)      We have a queue where the Producer (the reader) puts buffers as soon as he reads data.

2)      Then we have a Consumer (the writer) that grabs buffers as they become available and schedules a write to the wire. Note that it is critical that the consumer does not wait until the write is done before continuing. As mentioned before the HttpResponse OutputStream supports multiple writes.

These two will be running in independent threads and they will stop once the input stream is done and when we have written all the data to the output stream.

Read Loop

Action readLoop = async () =>

{

    while (true)

    {

        // Now wait until there is a free buffer to read into.

        ArraySegment<byte> freeBuffer = _freeReadBufferQueue.Take();

        byte[] readBuffer = freeBuffer.Array;

        int readBufferSize = readBuffer.Length;

        // Fill the read buffer before enqueueing it for writing

        int readTotal = 0;

        int readImmediate;

        do

        {

            readImmediate = await input.ReadAsync(readBuffer, readTotal, readBufferSize - readTotal);

            readTotal += readImmediate;

        }

        while (readImmediate > 0 && readTotal < readBufferSize);

        // Enqueue it in the dataQueue for this request

        dataQueue.Add(new ArraySegment<byte>(readBuffer, 0, readTotal));

        if (readTotal == 0)

        {

            // Done reading. Bail out of the read loop

            break;

        }

    }

};

1. The reader takes a buffer from the “Free buffers queue”. If this queue has a buffer it means there is space for him to read. If the queue is empty it means all of our buffers are being used.
2. Then it fills this buffer (note the do-while) until there is no more content or it’s out of space.
3. Then it adds this buffer to another queue. The dataqueue. This queue is the one that is read by the consumer.
4. When it’s done reading we break out of the loop. Otherwise it starts at 1 again.

Write Loop

Action writeLoop = async () =>

{

    while (true)

    {

        ArraySegment<byte> dataArraySegment = dataQueue.Take();

        int dataRead = dataArraySegment.Count;

        byte[] dataBuffer = dataArraySegment.Array;

        if (dataRead == 0)

        {

            // If the dataQueue has no data it means we have reached the end 

            // of the input stream. Time to bail out of the write loop

            _freeReadBufferQueue.Add(new ArraySegment<byte>(dataBuffer));

            break;

        }

        // Schedule async write. 

        // Make sure you don't wait until it's done before continuing 

        output.BeginWrite(dataBuffer, 0, dataRead, (iar) =>

        {

            try

            {

                output.EndWrite(iar);

            }

            finally

            {

                // Enqueue the buffer back to the pool so that we can read more data

                _freeReadBufferQueue.Add(new ArraySegment<byte>(dataBuffer));

            }

        }, null);

    }

    allWritesHaveBeenEnqueued.SetResult(null);

};

1. We grab a buffer from the dataQueue that was populated by the producer
2. If it’s of length zero it means we are done with the stream so we break out of the loop.
3. Otherwise we schedule an asynchronous write. It is the callback of this method the onw that will return the buffer back to the freeReadBufferQueue.

As you can see we have both loops working independent from each other and using the Queues to communicate when they have data and when they are done.

For the implementation of the queues I’m using BlockingCollection. This was released by Microsoft for .Net 4.5. The only downside of this is that it blocks when there is no data in the queue. This can be improved.

So what is the performance for this code?

Downloading the same file now takes:

                Overall Elapsed:                0:00:06.200

We got a 75% improvement in latency! Also if we go and measure the network utilization in our Proxy we can see that it nicely saturates the network when responding to our request.

The code for this article can be found here: https://github.com/krolth/Http-Proxy-dotnet

Notes

If you look at the attached code you will see that we are using preallocating the _freeReadBufferQueue in a static variable. We preallocate 1MB of memory and we use this for all of our requests. This way we ensure that regardless of how many requests we have we will not increase our memory footprint.

Another optimization we have is the use of a semaphore in CopyStreamAsync. This is to prevent one big request from hogging all the buffers from _freeReadBufferQueue. In my tests 40 buffers were good enough to return our payload. But if you are using this code I recommend you play around with these settings.

The reason I used 4k buffers is that I did not see a lot of improvement with 8k buffer.

Here is a table with the latency I saw with different buffer sizes.

One of the teams I work with in Azure uses it for their Portal Web experience.

The problem:

The load time of the website was very slow the first time each page was opened (cold start latency). It would sometimes take upwards of 10 seconds to open a page for the first time. Normally cold start is a very low priority for a web application. The reason being that a given page might be seen thousands of times so if you have a slow experience only one time of 1000 you will be ok.

Even when you have performance tests for your scenarios you focus on percentiles for your latencies measurements. So 99th percentile will tell you how long it’s taking for 99 percent of your users. As you can imagine it’s very easy to dismiss the cold start scenario.

In our project this was not the case. Due to many reasons all of our customers would see this at least once for each page they visited (not just the first customer). They would also see it again whenever the server was rebooted which would be at least once a week but probably more than that (since it's hosted in Azure).

Also any new customer would see this whenever they tried the service for the first time (So a very bad first experience).

All of these made it a deal breaker that had to be fixed.

Analysis

So I had to figure out what the problem was and how to fix it. I deployed the service and connected to it. Then I started Perfview to take a profile while the first page was getting loaded. Here is the result:

ASP.NET Stats View

Statistics Per Request URL

Processes view

Processes that did live for the entire trace.

Processes that did not live for the entire trace.

Let me translate that for you:

1)      Opening the root of our app took 24.5 seconds!

2)      The w3wp process (which is the ASP.NET worker process) only consumed 1.4 seconds of CPU cycles. The rest of the time it was waiting on something

3)      There is a bunch of csc processes running during this time

CSC is the C# compiler. But why am I seeing it here?

“By default, ASP.NET Web pages and code files are compiled dynamically when users first request a resource such as a page from the Web site. After pages and code files have been compiled the first time, the compiled resources are cached. Therefore, subsequent requests to the same page are very efficient.”

My first question was why would anyone want to do this? Why would you make your customer pay for this instead of doing it in your developer machine? The answer to that is that this improves the developer experience by a lot! Whenever you are iterating in a website you make many code changes and having to compile after each change is very time consuming. In this mode ASP.NET allows you to go to your website and modify the code (such as the content on your Razor views) and ASP.NET will realize that something changed and will recompile that page for you.

Now I’m not 100% sure this is very useful in actual production environments. But I’m not an expert on ASP.NET MVC. However the ASP.NET team released a tool that basically lets you precompile your site before you deploy it in case you don’t care about this dynamic changes.

Here is a guide on how to do it from MSDN.

Solution

So we have to use this aspnet_compiler tool to fix the problem. However after trying it on our Orchard site I hit some errors such as this:

Error      2              The type or namespace name 'Markdown' could not be found (are you missing a using directive or an assembly reference?)              c:\Users\shacorn\AppData\Local\Temp\Temporary ASP.NET Files\orchardlocal\2b51729c\84e739fa\App_Web_ztbavtrn.0.cs                29           0              Orchard.Web

..\Portal\0.0.2063.0\Modules\Orchard.Alias\web.config(35): error ASPCONFIG: Could not load file or assembly 'System.Web.Mvc, Version=5.1.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified.

Some Orchard forum posts mentioned that this is not supported. Such as this one.

---------------

I talked with one of the Orchard core developers (Sebastien Ros) and he helped me figure out how to make it work with Orchard.

Orchard will dynamically load modules that are in the Orchard\Modules directories. So to precompile it we need to do a little extra work:

1. Run the precompilation tool against a “deploy” version of your site. Not from the plain source directory. 
2. When you do “deploy” from Visual Studio there are some things specific for Orchard that make it necessary to do some other things. In particular due to the way that Orchard loads modules you need to build Orchard modules and pack them in assemblies. This can be done with this command from the Orchard root folder: 
   build /t:precompiled
3. Note that you need the entire source code for Orchard. Not just the redistributable. You can get it from here: https://www.orchardproject.net/download. Make sure you select “download the source”
4. Deploy your web application to a directory. For example: %outputDir%\Web
5. Make sure there are no csproj files under modules or under themes directories. Otherwise they will confuse the tool. Actually there should be no csproj files or cs files. We still will have ASP.NET views (cshtml).
   pushd .\Output\Release\Portals.Web\Modules   
   del *.csproj /s
6. Precompile with a command like this:
   aspnet_compiler -v / -p %outputDir%\Web %outputDir%\Web.Precompiled

 The parameters we are passing to the aspnet_compiler are the following:

• First we are telling the compiler the virtual path to be root or your site (-v /) and the physical path to be %outputDir%\Web. Note that this is the same path we gave as the output path for step (3) above.
• Then we pass the output directory for the precompiled version

The fact that we are passing root as the virtual path is important because this value will be inserted in the compiled files. So you want them to match the value in your deployed machines.

How to find the virtual path root for your app?

The way I found the correct value for this was a little roundabout. As I mentioned before whenever ASP.NET sees a view (cshtml file) it will compile it. The output of this is placed in the folder: “Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET Files”.

So I went to the machine where I took the profile and opened one of the generated .compiled files. That showed me something like this:

virtualPath="/Themes/Portals.Themes.Bootstrap/Views/Portals.Web.Authentication/Home/_ProviderInfo.cshtml"

Since Themes is a physical directory I know that the virtualPath should be “/”. So by passing root (“/”) I am able to match this.

Some other things to note about precompilation:

1)      It will definitely increase your build time. It added 10 minutes in my local machine. So this is something that you probably want to do as part of your deployment script, not something you do every time you build.

2)      It will find errors that you probably did not know you had. Some old views that you need to remove for example.

3)      Precompilation has a known issue with VirtualPathProvider. In our case passing (HostingEnvironment.VirtualPathProvider) to the VirtualPathProvider made the issue go away.

For us the result was dramatic. We were able to get up to 70% performance improvement in latency (99th percentile). The portal feels much faster and the overall experience is much better. Totally recommended :)