Tech Ed 2007 Orlando Day 4

  • Article

The Magic is Back! The performance of our 800 concurrent subscribers rocked!  

Less proved to be definitely better than More when it comes to thread pools on IIS.  Before launching the 800 subscriber test run, I throttled back the number of threads that are available in the 3 pools that do work on IIS.  The default setting in the registry is 20 which means that a total of 60 threads are pooled to perform operations against the .IN files, .OUT files, and the SQL Reconciler.  I speculated that my IIS servers with dual cores were spending too much time context switching between threads in the previous two test runs of 600 and 700 subscribers.  I also thought I might be overwhelming the SQL Reconciler but experimentation was needed.  I changed the registry setting to just 10 on both IIS servers so that each of them would only have 30 total threads in their respective pools.

The gamble paid off...

800 Subscribers:

  • IIS1 - CPU: 10% | Memory: 228 MB | Network Utilization: .95% | Disk: 3%
  • IIS2 - CPU: 9% | Memory: 227 MB | Network Utilization: .92% | Disk: 3%
  • SQL Distributor - CPU: 4% | Memory: 4.91 GB | Network Utilization: .58% | Disk: 40%
  • SQL Publisher - CPU: 32% | Memory: 4.29 GB | Network Utilization: 1% | Disk: 2%
  • Client Sync Times - High: 37.9 Minutes | Low: 1.7 Sec | Avg: 15.6 Minutes

As you can see by looking at the server numbers, CPU utilization, memory usage, network utilization, and disk utilization numbers are almost uniformly down accross the board compared to running 600 and 700 clients through a single IIS server with a higher thread pool.  Even more startling are the subscriber sync times.

The highest sync time of 37.9 minutes is faster than the 40.3 minute sync time we recorded from 600 subscribers and it's significantly faster than the 76.8 minute sync time logged by the 700 subscribers.  The average sync time of 15.6 minutes was almost 5 minutes faster than the average sync time recorded by the 700 subscribers.

So the key takeaway here is reduce the size of the thread pool and don't push each IIS server beyond 400 concurrent subscribers.  Reducing my thread pool to 10 on both IIS servers meant that context switching was reduced and I never had more than 20 connections to the SQL Reconciler.  This gave SQL Server room to breathe and perform its work faster than if it had to deal with the default of 40 connections (20 connections per IIS box) to the SQL Reconciler.  If I was going to add a third IIS server to accomodate 1,200 concurrent subscribers, I would give serious thought to reducing the thread pool even further to stay under 20 connections to the SQL Reconciler (6-7 threads per IIS box).  I would also add that you should thoroughly experiment with different thead pool sizes on IIS to determine which amount yields the best performance for the number of subscribers you need to replicate at the same time.  

I'm super excited that we beat our record of 720 simultaneous subscribers from MEDC 2007 both in scalability and performance!

Rob