Performance of ASP.NET Web Services, Enterprise Services, and .NET Remoting

 

Ingo Rammer
thinktecture.

Richard Turner
Program Manager
Microsoft Distributed Systems Group

August 2005

Summary: Compare and contrast performance characteristics of real-life ASP.NET Web services, .NET Enterprise Services components, and .NET Remoting components, and get recommendations on how to best use these technologies. (34 printed pages)

Download the associated code sample, ASMXvsEnterpriseServicesvsRemotingPerformanceTests.msi.

Contents

Introduction
Goals
   This Is Not a Benchmark
   The Tests
The Results
   Test 1: Creating and Storing Orders
   Test 2: Retrieving Northwind's Product Data
   Test 3: Retrieving Customer Information
.NET Framework 2.0 Performance
Conclusions
   Recommendations
Appendix A: Synthetic Baseline - Empty Method Test
Appendix B – Detailed Test Results
   Store Order as Object
   Store Order as DataSet
   Load Products as Objects
   Load Products as DataSet
   Load Customer as Object
   Load Customer as DataSet
   Empty Message, Cross-Process
   Empty Message. Cross-Machine
Appendix C: Test Application Notes
   Running the Test Application
   Interesting Factoid – The Test App Is Technology Agnostic!
Appendix D: Software and Hardware Setup
   Application Environment
   Infrastructure Environment

Introduction

While absolute performance is of paramount concern for several areas of technology (devices, hardware controllers, life and healthcare services, certain financial systems), these tend to be among the minority. Most business applications' primary goals are "correctness", "time to delivery", and the need to be as fast as is necessary, but not more so. The cost and effort of engineering applications that deliver absolutely maximum possible performance can be enormous; the considerable time and skill required for peak performance is often unnecessary for many business systems. However, while absolute maximum performance is often overkill, ensuring good overall system performance is still a goal for most businesses wanting to maximize their return on investment.

In this white paper, we will provide a comparative analysis of the relative performance levels of real-world components/services hosted within each of the three distributed component/service technologies available in .NET:

• .NET Enterprise Services (ES) hosted in COM+
• ASP.NET Web services (ASMX) hosted in IIS
• .NET Remoting hosted in IIS and custom hosts

(Note   The performance of System.Messaging vs. MSMQ COM APIs will be addressed in the System.Messaging Performance paper).

Goals

There is a never-ending debate about "which is the fastest Microsoft distributed application technology" or statements that "<insert technology name here> is too slow for us to use". The primary goal of this paper is to address and clarify many of the concerns, misconceptions, inaccuracies, and misinformation relating to the performance of Microsoft distributed technologies.

We aim to eradicate many currently held misconceptions about the relative performance characteristics of each of Microsoft's distributed component/service technologies, and to provide a clear set of illustrative tests and their results along with a set of concise guidance that will help you choose the most appropriate technology for your needs.

To summarize, our goals for this paper are:

1. To investigate what the relative performance differences are between the three technologies for the majority of business applications
2. To correct some assumptions regarding the perceived performance penalties of one technology over another
3. To help guide your decisions about where, when, and how to utilize each technology most appropriately
4. To provide a test application so that you can run these tests on your own machines and in your own environments. We strongly encourage you to build and run this test environment and to investigate and analyze the performance characteristics of these technologies. This way will you be able to fully comprehend the many factors that affect performance of distributed systems.

This Is Not a Benchmark

The tests presented in this whitepaper are explicitly designed to provide consistent comparative results between the specific technologies under test. These tests are not designed to measure the absolute maximum possible performance for each technology under load.

The test driver (client) application is single-threaded, making serial synchronous calls as quickly as the called service responds. With this design, the bottleneck is not always server CPU use.

Multiple clients, or a multithreaded client, may result in the server processing more calls per second. For the more CPU-intensive server application tests, multiple clients do not change significantly the absolute or relative performance of the technologies being tested.

For the lightest tests, significantly higher (2X) aggregate server throughput may be achieved by using multiple clients. Multiple clients may also change somewhat the relative performance of the various technologies.

Whether a single client or multiple clients is more realistic depends on where and how a Web service is deployed. The conclusions of the paper are unaffected by the decision to measure with a single client.

The Tests

In the following performance tests, we explore both synthetic baselines and real-world scenarios. The common questions we've set out to examine are:

1. Is .NET Remoting faster than ASMX?
2. Is ES slower than Remoting?
3. Is ASMX too slow for real-world scenarios?

• In order to examine these claims, we performed a number of tests. The primary tests examine the performance of making calls against actions that either accept a large request and perform significant work or are requested to perform significant work and return small or large result sets. The goal of these tests is to illustrate the performance of typical business scenarios that you are likely to experience as you build your systems.

All of these tests were run against each of the available technologies' protocols:

• Enterprise Services (using Just-In-Time Activation)
  • No authentication
  • Call-level authentication and enforced role-access checks
• ASP.NET Web services
  • No authentication
  • Username and password authentication
  • Integrated authentication
• .NET Remoting
  • TCP/Binary, unsecured
  • HTTP/Binary, unsecured
  • HTTP/SOAP, unsecured
  • HTTP/SOAP in IIS, unsecured
  • HTTP/Binary in IIS, unsecured
  • HTTP/SOAP in IIS, HTTP Basic Authentication
  • HTTP/Binary in IIS, HTTP Basic Authentication
  • HTTP/SOAP in IIS, Integrated Authentication
  • HTTP/Binary in IIS, Integrated Authentication

All of the following tests are based on a single client application that repeatedly invokes server-side methods. We calculated the average number of remote invocations/method calls per second for each test in the set and repeated the complete set 10 times to acquire average and standard deviation for a complete multi-stage test run.

The Results

Test 1: Creating and Storing Orders

This first test was designed to simulate the performance characteristics each technology exhibits in the most optimal case—where the action being called performs significant work. In order to perform significant work, the action being called stores orders in a database involving writing records to two tables within a transaction.

The caller creates an instance of the following Order class.

[Serializable]
public class Order
{
   public int OrderID;
   public String CustomerID;
   public int EmployeeID;
   public DateTime OrderDate;
   public Address ShippingAddress;
   public Address BillingAddress;
   public int ShipVia;
   public decimal Freight;
   public LineItem[] LineItems;
}

The Order class contains one child object of type Address, and an array of 20 LineItems:

[Serializable]
public class Address
{
   public String Name;
   public String Street;
   public string City;
   public string Region;
   public string PostalCode;
   public string Country;
}

[Serializable]
public class LineItem
{
   public int ProductID;
   public double UnitPrice;
   public short Quantity;
   public double Discount;
}

This data object is passed from the caller to the server during method invocation and is serialized onto the wire by the different technologies being tested.

The server-side implementation is similar to the following:

SqlConnection _conn;

private void InitializeComponent()
{
   // ... large parts of generated code removed 
    _conn = new SqlConnection();
   // ... large parts of generated code removed 
}

public void StoreOrder(Order o)
{
   using (_conn)
   {
      _conn.Open();
      using (SqlTransaction tx = _conn.BeginTransaction())
      {
         using (SqlCommand cmd = new SqlCommand(@"INSERT INTO Orders(EmployeeID, " + 
                      "CustomerID, OrderDate, RequiredDate, ShipVia, ShippedDate, " + 
                      "ShipName, Freight, ShipAddress, ShipCity, ShipRegion, " + 
                      "ShipCountry, ShipPostalCode) VALUES (@EmployeeID, @CustomerID, " +
                      "@OrderDate, @RequiredDate, @ShipVia, @ShippedDate, @ShipName, " +  
                      "@Freight, @ShipAddress, @ShipCity, @ShipRegion, @ShipCountry, " + 
                      "@ShipPostalCode); SELECT @@IDENTITY", _conn,tx))
         {
            cmd.Parameters.Add("@EmployeeID", o.EmployeeID);
            cmd.Parameters.Add("@CustomerID",o.CustomerID);
            cmd.Parameters.Add("@OrderDate",o.OrderDate);
            cmd.Parameters.Add("@RequiredDate",o.OrderDate.AddDays(10));
            cmd.Parameters.Add("@ShipVia",o.ShipVia);
            cmd.Parameters.Add("@ShippedDate",o.OrderDate.AddDays(5));
            cmd.Parameters.Add("@ShipName",o.ShippingAddress.Name);
            cmd.Parameters.Add("@Freight",o.Freight);
            cmd.Parameters.Add("@ShipAddress",o.ShippingAddress.Street);
            cmd.Parameters.Add("@ShipCity",o.ShippingAddress.City);
            cmd.Parameters.Add("@ShipRegion",o.ShippingAddress.Region);
            cmd.Parameters.Add("@ShipCountry",o.ShippingAddress.Country);
            cmd.Parameters.Add("@ShipPostalCode",o.ShippingAddress.PostalCode);
            
            decimal orderID = (decimal) cmd.ExecuteScalar();
            o.OrderID = (int) orderID;
         }

         using (SqlCommand cmd = new SqlCommand(@"INSERT INTO [Order Details] (OrderID, " +
                      "ProductID, UnitPrice, Quantity, Discount) VALUES (@OrderID, " + 
                      "@ProductID, @UnitPrice, @Quantity, @Discount)", _conn,tx))
         {
            cmd.Parameters.Add("@OrderID",SqlDbType.Int);
            cmd.Parameters.Add("@ProductID",SqlDbType.Int);
            cmd.Parameters.Add("@UnitPrice",SqlDbType.Money);
            cmd.Parameters.Add("@Quantity",SqlDbType.SmallInt);
            cmd.Parameters.Add("@Discount",SqlDbType.Real);

            foreach (LineItem itm in o.LineItems)
            {
               cmd.Parameters["@OrderID"].Value = o.OrderID;
               cmd.Parameters["@ProductID"].Value = itm.ProductID;
               cmd.Parameters["@UnitPrice"].Value = itm.UnitPrice;
               cmd.Parameters["@Quantity"].Value = itm.Quantity;
               cmd.Parameters["@Discount"].Value = itm.Discount;
               cmd.ExecuteNonQuery();
            }
         }
         
         tx.Commit();
      }
      _conn.Close();
   }
}

After the test run is completed, the server-side logic uses a second transaction to delete the inserted records, providing a "level playing field" no matter how many times this test runs.

Figure 1. Passing an "order" cross-machine with a real server-side implementation

When looking at the test results in Figure 1, note that there is just a 14% difference between ES @73 CPS and ASP.NET Web services @63 CPS, and that the maximum difference between the fastest and the slowest protocols is 45%. This clearly illustrates that the benefit of any given transport over another diminishes as the amount of work performed grows in proportion to the time taken to transport data.

We also created a matching ADO.NET DataSet test, building typed DataSets containing DataTables defined as in Figure 2.

Figure 2. This DataSet represents a purchase order

This DataSet is used with a generated SqlDataAdapter including SqlCommands similar to those shown in the code snippets above.

When the order is passed from the caller to the server as a DataSet and then stored, we get the results shown in Figure 3.

Figure 3. Storing a purchase order as DataSet

Notice the significant performance impact of using DataSets (around a 50% slowdown), along with the fact that the difference between the protocols decreases further still. It is also important to note that the slowest results from passing data objects exceed the best performance when passing ADO.NET DataSets.

This clearly illustrates that passing ADO.NET DataSets between tiers of a system incurs a significant performance penalty.

Test 2: Retrieving Northwind Product Data

The purpose of this test is to illustrate the performance characteristics of each technology when a relatively sizeable amount of data is returned from a service that doesn't perform significant work.

To model this scenario, we built methods that retrieve and return a list of "product" records from the well-known SQL Server "Northwind" sample database.

We expect that the binary serialization technologies (ES/COM+ and Remoting Binary/TCP) will perform best in this test; we also expect that the SOAP-based technologies will return lower levels of performance due to the larger amounts of data being transported, as well as their more complex serialization and deserialization mechanisms.

In the first test, we defined a [Serializable] class that has public members compatible with the XML Serializer:

[Serializable]
public class Product
{
   public int ProductID;
   public string ProductName;
   public int SupplierID;
   public int CategoryID;
   public string QuantityPerUnit;
   public decimal UnitPrice;
   public short UnitsInStock;
   public short UnitsOnOrder;
   public short ReorderLevel;
   public bool Discontinued;
}

In the server-side implementation, we open a database connection to SQL Server, executed a query against the database, and, using an ADO.NET SqlDataReader, populated an ArrayList containing 77 Product instances of this class. The ArrayList was then returned to the caller.

The results are shown in Figure 4.

Figure 4. Retrieving the Northwind product catalog as objects

These results show that Enterprise Services (over DCOM) and TCP-based binary Remoting provide equal levels of performance for unsecured calls. The results also show that ASP.NET Web services provides about 62% of the performance of Enterprise Services and that the slowest protocols only allow 17% of the number of calls per second compared to the fastest.

In this and all following tests it is clear why it is generally not recommended to use the .NET Remoting SoapFormatter for building SOAP services: the ASP.NET Web Service stack is about three times as fast.

In the second approach to this test, we implemented a method that returns the requested data in an ADO.NET DataSet object that is based directly on the underlying table layout, and populate the DataSet's DataTable using a SqlDataAdapter. The table definition is illustrated in Figure 5.

Figure 5. The typed DataSet to access the product catalog

Figure 6 illustrates the results:

Figure 6. Retrieving the Northwind product catalog as a DataSet

It's clear from the results of this test that returning DataSets from service to caller decreases performance significantly—by around 75% for the binary transports! Note also that the performance of the four fastest protocols is completely identical, meaning that all four technologies were throttled by the serialization, de-serialization, and transport of the DataSet. Also, note that the range of performance between the fastest (ASMX @ 39 cps) and the slowest (Remoting HTTP/SOAP (IIS) Integrated security @ 25 cps) are separated by just 14 cps—a mere 30% drop.

This pointedly illustrates that while it is convenient to use DataSets to pass information between tiers of your applications, there is a significant performance impact in doing so. We will return to this subject later.

It is also important to point out that it is quicker to iterate through an ADO.NET DataSet and construct a collection of data objects and serialize this collection than merely to return the ADO.NET DataSet to the caller! If this was a real system, implemented to pass DataSets between server and caller, we could almost quadruple the performance of this part of the system with just a small amount of work.

Test 3: Retrieving Customer Information

In the next test we wanted to examine the relative performance characteristics of each technology when transferring a very small amount of data from server to client by retrieving a single customer record from the "Northwind" SQL Server sample database.

We created the following [Serializable] Customer class to ferry the retrieved data back to the client from the server:

[Serializable]
public class Customer
{
   public String CustomerID;
   public String CompanyName;
   public String ContactName; 
   public String ContactTitle;
   public string Address; 
   public string City;
   public string Region;
   public string PostalCode; 
   public string Country;
   public string Phone;
   public string Fax;
}

The following shows a part of our server-side implementation that uses a SqlDataReader to populate a new instance of the Customer class:

public Customer GetCustomer(string id)
{
   using (SqlConnection conn = new SqlConnection(...))
   {
      conn.Open();

      String sql = "SELECT CustomerID, CompanyName, ContactName, "
                   "ContactTitle, Address, City, Region, " +
                   "PostalCode, Phone, Fax, Country FROM " + 
                   "Customers WHERE (CustomerID = @CustomerID)";

      using (SqlCommand cmd = new SqlCommand(sql,conn))
      {
         cmd.Parameters.Add("@CustomerID", id);
         SqlDataReader rdr = cmd.ExecuteReader();
         if (rdr.Read())
         {
            Customer c = new Customer();
            c.CustomerID = (string) rdr[0];
            c.CompanyName = (String) rdr[1];
            c.ContactName = (String) rdr[2];
            c.ContactTitle = (String) rdr[3];
            c.Address = (String) rdr[4];
            c.City = (String) rdr[5];
            c.Region = rdr.IsDBNull(6) ? "" : rdr[6] as string;
            c.PostalCode = (String) rdr[7];
            c.Phone = (String) rdr[8];
            c.Fax = (String) rdr[9];
            c.Country = (String) rdr[10];
            return c;
         }
         else
         {
            return null;
         }
      }
   }
}

The results of our performance comparison are shown Figure 7.

Figure 7. Retrieving a customer's data as an object

These results more prominently highlight the performance differentials between each of the underlying technologies. These differentials are so visible in this test because the work being done to retrieve and return a single record is a much lower percentage of the overall call cost, and thus the cost of transport plays a more significant role. Therefore, because SOAP/HTTP has a higher cost of transport than binary/DCOM, the SOAP transports show much lower levels of throughput than the binary mechanisms do.

Next, we tested the same operation returning a typed DataSet, shown in Figure 8.

Figure 8. The typed DataSet to access a customer's information

This DataSet is populated using a server-side implementation similar to the following:

SqlDataAdapter _customerAdapter;
SqlCommand _customerSelect;
SqlConnection _conn;

private void InitializeComponent()
{
   // ... large parts of generated code removed 

    _conn = new SqlConnection();
    _customerAdapter = SqlDataAdapter();
    _customerSelect = SqlCommand();

    _customerSelect.CommandText = "SELECT CustomerID, CompanyName, " +
               "ContactName, ContactTitle, Address, City, Region, " +
               "PostalCode, Phone, Fax, Country FROM Customers WHERE " + 
               "(CustomerID = @CustomerID)";

_customerSelect.Connection = _conn;

_customerSelect.Parameters.Add(new SqlParameter("@CustomerID", 
               SqlDbType.NVarChar, 5, "CustomerID"));

    _customerAdapter.SelectCommand = this.sqlSelectCommand3;

    _customerAdapter.TableMappings.AddRange(new DataTableMapping[] {
           new DataTableMapping("Table", "Customers", new DataColumnMapping[] {
                  new DataColumnMapping("CustomerID", "CustomerID"),
                  new DataColumnMapping("CompanyName", "CompanyName"),
                  new DataColumnMapping("ContactName", "ContactName"),
                  new DataColumnMapping("ContactTitle", "ContactTitle"),
                  new DataColumnMapping("Address", "Address"),
                  new DataColumnMapping("City", "City"),
                  new DataColumnMapping("Region", "Region"),
                  new DataColumnMapping("PostalCode", "PostalCode"),
                  new DataColumnMapping("Phone", "Phone"),
                  new DataColumnMapping("Fax", "Fax"),
                  new DataColumnMapping("Country", "Country")})});

   // ... large parts of generated code removed 
}


public PerfTestDataSet GetCustomerAsDataset(string id)
{
   using (_conn)
   {
      _conn.Open();
      customerAdapter.SelectCommand.Parameters["@CustomerID"].Value = id;
      
      PerfTestDataSet ds = new PerfTestDataSet();
      _customerAdapter.Fill(ds,"Customers");
      return ds;
   }
}

When running this code inside our test framework, we received the results shown in Figure 9.

Figure 9. Retrieving a customer's data as a DataSet

Similarly to the previous tests, it is clear that exchanging ADO.NET DataSets negatively impact performance, and the difference in throughput for all technologies becomes smaller than when returning serialized objects illustrating that ADO.NET DataSets are throttling the throughput.

.NET Framework 2.0 Performance

At the time of writing this paper, Microsoft is working to deliver the .NET Framework 2.0 (previously codenamed "Whidbey") which includes a huge number of improvements, enhancements and new capabilities including:

• Full support for x64 and IA64 based machines
• Brand new transactions support through System.Transactions
• Many enhancements to System.Net including connectivity awareness and support for proxies
• Significant enhancements for Windows Forms and Web Forms
• "ClickOnce" automatic application deployment infrastructure

In addition to the new and enhanced features included in the .NET Framework 2.0, a great deal of work has been done to significantly boost performance and improve memory utilization in many areas, including BCL, XML Serialization, networking throughput, ADO.NET, ASP.NET, etc.

Early examples of these performance and throughput improvements can be seen in the following whitepapers:

• Comparing XML Performance

  .NET Framework 2.0 Beta 2, .NET Framework 1.1, and Sun Java 1.5 Platforms

• Comparing Web Service Performance

  WS Test 1.1 Benchmark Results for the .NET Framework 2.0, .NET 1.1, Sun JWSDP 1.5, and IBM WebSphere 6.0

More performance analysis and comparison whitepapers will be published over the coming months as the .NET Framework 2.0 continues its trajectory towards RTM (release to manufacturing). In order to keep up to date with such whitepapers, please visit the following online resources:

• The .NET Framework Developer Center
• Competitive Reports and Evidence: Comparing .NET to J2EE

Conclusions

The results from the real-world tests above are illustrated in Table 1.

Table 1. Results summary

Calls per Second (rounded)	Enterprise Services	ASP.NET Web services	.NET Remoting TCP Binary
Test 1: Storing orders a) Serialized Data b) DataSet	73 35	63 34	72 34
Test 2: Retrieving products a) Serialized Data b) DataSet	147 39	93 39	149 39
Test 3: Retrieving customer a) Serialized Data b) DataSet	618 90	289 83	682 91

These results show the performance characteristics you can expect from real-world applications, and that the difference between Web services (which are usually perceived as being very slow) and DCOM (which is clearly one of the fastest distributed application technology in this test) is in fact very small for real applications.

Our observations of the technologies being tested are discussed in more detail in the following section, but can be summarized here:

• Whilst ASMX is not absolutely the fastest technology we currently ship, its performance is more than adequate for most business scenarios. Combined with the facts that ASMX services are easy to scale out and provide a wealth of other benefits such as interoperability and future compatibility, ASMX should be your primary choice for building services today.
• If absolute performance is of the utmost concern, Enterprise Services components should be used to build the most performance-critical portions of your system. COM+ performed best overall and is a safe, secure, reliable hosting environment for distributed components. ES/COM+ will also integrate well with Indigo services and it will be relatively simple to convert ES components to Indigo services.
• While .NET Remoting performs very well when using binary serialization over TCP, its performance diminishes when Remoting components are hosted in IIS and/or when sending and receiving SOAP messages. Combined with the fact that .NET Remoting components will not interoperate with anything other than .NET Remoting endpoints, we urge you to consider ASMX or ES before Remoting wherever possible.

A key observation to draw from these results is that if the amount of work performed at the server is not a significant percentage of the overall duration of each call, transport time will therefore account for a significant portion of the call time. If you move from a "fast" transport (such as DCOM under COM+) to a more "costly" transport (such as SOAP/HTTP under ASMX) you may experience a significant performance impact. Thus our guidance to perform as much work as possible at the server for each method call and to avoid making unnecessary network traversals is clearly illustrated.

The performance benefits of these technologies over others decrease as the amount of work being performed by the called service increases. This observation has been stated in many articles, including .NET Enterprise Services Performance.

It is also clear from these tests that the performance differences between these technologies are smaller than the differences between application-specific decisions such as the use of DataSets or serialized structures.

Whilst we have focused greatly on performance numbers in this paper, performance is not the only factor that you should consider when planning your development projects. Other factors such as security, scalability, management, developer productivity, interoperability, and the future direction of Microsoft distributed technologies all come into play. Developing Distributed Services Today presents the prescriptive guidance for how to best choose when and where to utilize each technology and what the future holds.

Recommendations

Based upon the results of these tests, and in accordance with our future technology direction and best practices, you will be able to build systems which are not as prone to performance issues by adhering to the following recommendations:

Use ASMX wherever possible

In order to provide the broadest reach for your services, consider publishing your services using ASMX wherever possible. This will enable any system that can talk SOAP over HTTP to call your service, regardless of what platform, device, or language that system may be running on or implemented in. As you can see from the results above, ASMX performs admirably in many scenarios. Even on our single-processor machine being called by a single-threaded application, ASMX is capable of processing some 63 heavy operations per second—only 10 calls per second less than the equivalent ES component! In a production environment, you can expect significantly more performance from a single box hosting ASMX services, so ASMX performance shouldn't block your adoption of this important technology.

Web services exhibit other useful traits and features, including (but not limited to):

• Scalability—It is very easy to scale out a Web service using load-balancing technology such as Windows Network Load Balancing, or hardware devices from vendors such as Cisco and F5. More on this subject to follow.
• Availability—ASMX Web services can be configured to be highly available using a combination of technologies such as load-balancing combined with the powerful capabilities of the IIS6 infrastructure built into Windows 2003 Server (such as automatic recycling and restarting of failed services).
• Interoperability—Web services are at the very core of the movement to standardize interoperable secure, reliable, transacted communications protocols enabling disparate, heterogeneous systems to communicate freely.
• Productivity—Building ASMX Web services is very easy for most developers through the judicious use of a few attributes and by following a few recommendations.

Use ASMX for Web services

There are many reasons why ASMX is a great platform for building and deploying services as described in the prescriptive guidance, and as can be seen from the results in this paper, ASMX usually outperforms Remoting when talking SOAP.

It is also important to note that while .NET Remoting is capable of supporting SOAP/HTTP (via its SoapFormatter and HttpChannel) we recommend against using Remoting to expose Web services. The .NET Remoting SoapFormatter uses RPC-Encoding, which is no longer the standard way of encoding SOAP (document-literal is the future standard encoding mechanism used by most vendors' technologies and platforms); therefore, interoperability with Remoting services will be very limited.

Choose Web services for scalability

One thing not illustrated above, but highly relevant to this discussion, is that Web services talk primarily over HTTP, a stateless communications technology that is easy to load-balance. This means that it is also possible to load-balance stateless Remoting components using TCP or HTTP (i.e., hosted within IIS), although this can get complex and problematic if session state is required.

To load-balance ES/COM+ components, you must use Component Load Balancing (CLB), a feature of Application Center 2000. Alas, AppCenter 2000 enters extended support in July 2006 and support ends in 2011. Therefore, in order to avoid having to re-architect your systems well before 2011 to no longer depend on CLB, it is strongly advisable to avoid this technology for new development and instead migrate to other strategies as soon as possible. A good migration strategy is to front your current COM+ components with ASMX Web services, which simply pass calls straight through to the COM+ component, and to load-balance your Web services as described above.

Use Enterprise Services only inside your services

.NET Enterprise Services is a layer of the .NET Framework which provides access to, and support for, the rich services offered by COM+. If you need any of the rich COM+ services such as distributed transactions that span several objects/services, fine-grained security, JITA, and object pooling, then you should choose Enterprise Services. Enterprise Services and COM+ are well-tested, highly optimized technologies that provide extremely fast, low-latency cross-process and cross-machine calls

However, we strongly recommend that you not broadly expose your Enterprise Services components and that you use this technology only inside your service boundary. If you wish to provide access to your service, we urge you to expose your service's capabilities via Web services using ASMX. If you have an established ES/COM+ code base, then we suggest fronting your COM+ services with ASMX Web services wherever possible.

Consider a hosted infrastructure over building your own

As can be seen above, ASP.NET delivers great-performing Web services that are highly interoperable and also benefit from the great hosting facilities provided by IIS—particularly IIS6 in Windows 2003 server.

COM+ provides the fastest component performance on the Microsoft platform along with being a tried, trusted, safe, secure, and reliable hosting environment for your Enterprise Services and COM+ components. It also provides distributed transaction support, security (authentication and encryption), reliability, availability, management services, and is a great choice for hosting complex component-based systems within your service.

While .NET Remoting using Binary serialization over TCP also provides great performance, it does not provide any hosting environment, security, reliability, availability, or management features "out of the box". In order to host Remoting components that use TCP, you must write your own (secure, scalable, reliable) hosting application, and this is not a trivial task. If you want to use Remoting over HTTP, you can optionally use IIS as a host, but as you can see, hosting your Remoting component in IIS and talking SOAP/HTTP impacts Remoting performance considerably—often to the point where ASMX performs better!

Therefore, we urge you to consider delivering your services in ASMX hosted in IIS or Enterprise Services components within COM+, rather than using Remoting, wherever possible.

Avoid passing DataSets between services

In our tests, choosing the way the services and callers exchange data—as DataSets or serialized structures—resulted in a much bigger performance impact than choosing any particular distributed systems technology over another.

From a performance perspective, we strongly urge you to limit your use of DataSets as a data transfer mechanism to the parts of your application where they are truly needed and that you opt for [Serializable] structures wherever possible.

ADO.NET DataSets provide a great way to retrieve, manipulate, sort, and shape data, store it locally for offline use, and synchronize changes back into a central database. If this is a requirement of your application, then of course, it's a valid choice to make. In the .NET Framework 1.x, DataSet values are always serialized to XML (even if you state that you want them serialized as binary or use them with ES or Remoting), and they also contain the XSD describing the data. This, of course, impacts performance.

What's more, remember that DataSets are a .NET-specific technology and will significantly limit your interoperability as other platforms won't necessarily be able to serialize and parse data containing serialized DataSets.

You can gain considerable performance improvements by instead exchanging serialized data structures. The built-in runtime formatting and XML serialization framework makes it easy, as shown in the tests above.

Use Authenticated Connection Sharing

All of the tests that use Internet Information Server as a host in conjunction with Integrated Security have been configured to run with the options useAuthenticatedConnectionSharing and useUnsafeAuthenticatedConnectionSharing which are available for ASP.NET Clients and .NET Remoting. This setting allows client and server to re-use an existing NTLM-authenticated connection.

You can see the performance results when we set this option in Figure 10 during the tests presented originally in Figure 1 (storing a purchase order as objects). Please note that this setting has an effect only if you use Internet Information Server (IIS) as a server and if you mandate "Windows In Security" in the configuration of your virtual directory.

The Web Q&A: Caching Transforms, Connection Sharing, and More column in the MSDN Magazine September 2004 issue discusses these tradeoffs.

Figure 10. The impact of UnsafeAuthenticatedConnectionSharing

You can use Windows Integrated Security together with this setting from within ASP.NET Web services like this:

MyWebService svc = new MyWebService();
svc.Credentials = System.Net.CredentialCache.DefaultCredentials;
svc.UnsafeAuthenticatedConnectionSharing = true;

And with .NET Remoting like this:

IMyRemote rem = (IMyRemote) Activator.GetObject(typeof(IMyRemote), "HTTP://...");
IDictionary snkprops = ChannelServices.GetChannelSinkProperties(rem);
snkprops["unsafeauthenticatedconnectionsharing"] = true;

Appendix A: Synthetic Baseline - Empty Method Test

This test "baselines" the performance differentials between the underlying technologies being examined. These tests make calls against actions which accept no parameters, perform no work, and return no results. The server-side implementation is as simple as:

public void TransferEmpty()
{
   // do nothing
}

Note   It is important to remember that these synthetic baseline figures are presented purely to illustrate the relative levels of performance of the communications infrastructure of each of the technologies being examined and does not reflect the performance characteristics expected in real-life systems.

Figure 11. Calling a method with no parameters cross-process (synthetic baseline)

The results of our tests in Figure 11 matched the common assumptions very closely: ES provides by far the best performance due to its reliance on COM+ and DCOM and the high performance LPC (Lightweight Procedure Call) on-box communications infrastructure. .NET Remoting wire protocols come in second. ASP.NET Web services comes third, but is still faster than .NET Remoting components hosted in IIS.

As a second step, we ran this test across-machines. We redirected the generated ASP.NET by setting its "Url" property, exported an application proxy for the COM+/Enterprise Services components, and used Activator.GetObject() for the .NET Remoting part of this test. The results turned out to be very much in-line with the previous tests, but with slightly smaller numbers of calls per second due to the network traversals involved:

Figure 12. Calling an empty method with no parameters cross-machine (synthetic baseline)

Note that the cross-machine network traversal has essentially throttled the fastest transports (ES and Remoting Binary/TCP), while barely affecting the slower transports. This clearly illustrates how effective the cross-process communications mechanisms of COM+ in particular are compared to the technologies built primarily for serving network traffic (i.e., IIS & ASMX).

It is important to remember, as mentioned above, that this test is very artificial and only illustrates the capabilities of the transports.

Appendix B – Detailed Test Results

Store Order as Object

Test	Avg. Calls/Second	Std. Deviation
Enterprise Services	73	0.39
Enterprise Services (auth)	73	0.34
Remoting TCP/Binary	72	2.71
Remoting HTTP/Binary	68	0.86
Remoting HTTP/Binary (IIS)	66	0.31
ASP.NET Web services	63	2.71
Remoting HTTP/Binary (IIS) Password	63	0.39
Remoting TCP/SOAP	56	1.97
Remoting HTTP/SOAP	53	0.57
Remoting HTTP/Binary (IIS) Integrated	51	0.28
Remoting HTTP/SOAP (IIS)	50	0.16
ASP.NET Web services - Integrated	50	0.30
Remoting HTTP/SOAP (IIS) Password	49	0.29
Remoting HTTP/SOAP (IIS) Integrated	40	0.84

Store Order as DataSet

Test	Avg. Calls/Second	Std. Deviation
Enterprise Services	35	0.54
Enterprise Services (auth)	35	0.51
ASP.NET Web services	34	0.43
Remoting TCP/Binary	34	0.85
Remoting HTTP/Binary	32	0.77
Remoting HTTP/Binary (IIS)	32	1.10
Remoting TCP/SOAP	32	0.77
Remoting HTTP/Binary (IIS) Password	31	1.47
Remoting HTTP/SOAP	30	0.68
Remoting HTTP/SOAP (IIS)	30	0.48
Remoting HTTP/SOAP (IIS) Password	30	0.46
ASP.NET Web services - Integrated	29	0.37
Remoting HTTP/Binary (IIS) Integrated	28	0.37
Remoting HTTP/SOAP (IIS) Integrated	26	0.31

Load Products as Objects

Test	Avg. Calls/Second	Std. Deviation
Remoting TCP/Binary	149	3.05
Enterprise Services	147	2.29
Enterprise Services (auth)	146	2.49
Remoting HTTP/Binary	118	2.13
Remoting HTTP/Binary (IIS)	114	0.63
Remoting HTTP/Binary (IIS) Password	106	1.19
ASP.NET Web services	93	1.04
Remoting HTTP/Binary (IIS) Integrated	76	0.81
ASP.NET Web services - Integrated	67	0.35
Remoting TCP/SOAP	33	0.34
Remoting HTTP/SOAP	30	0.32
Remoting HTTP/SOAP (IIS)	30	0.25
Remoting HTTP/SOAP (IIS) Password	29	0.16
Remoting HTTP/SOAP (IIS) Integrated	26	0.14

Load Products as DataSet

Test	Avg. Calls/Second	Std. Deviation
ASP.NET Web services	39	0.12
Enterprise Services	39	0.26
Enterprise Services (auth)	39	0.21
Remoting TCP/Binary	39	1.16
Remoting HTTP/Binary (IIS)	36	0.24
Remoting HTTP/Binary	35	1.10
Remoting HTTP/Binary (IIS) Password	35	0.17
ASP.NET Web services - Integrated	33	0.09
Remoting HTTP/Binary (IIS) Integrated	31	0.21
Remoting TCP/SOAP	30	1.27
Remoting HTTP/SOAP (IIS)	29	0.12
Remoting HTTP/SOAP	28	1.07
Remoting HTTP/SOAP (IIS) Password	28	0.06
Remoting HTTP/SOAP (IIS) Integrated	25	0.08

Load Customer as Object

Test	Avg. Calls/Second	Std. Deviation
Remoting TCP/Binary	682	12.32
Enterprise Services	618	13.78
Enterprise Services (auth)	616	7.76
Remoting HTTP/Binary	406	7.84
Remoting TCP/SOAP	359	11.62
Remoting HTTP/Binary (IIS)	324	4.26
ASP.NET Web services	289	2.68
Remoting HTTP/SOAP	267	6.18
Remoting HTTP/Binary (IIS) Password	261	2.39
Remoting HTTP/SOAP (IIS)	214	2.84
Remoting HTTP/SOAP (IIS) Password	186	0.81
Remoting HTTP/Binary (IIS) Integrated	134	0.95
ASP.NET Web services - Integrated	130	0.67
Remoting HTTP/SOAP (IIS) Integrated	112	0.55

Load Customer as DataSet

Test	Avg. Calls/Second	Std. Deviation
Remoting TCP/Binary	91	2.69
Enterprise Services	90	0.30
Enterprise Services (auth)	90	0.26
ASP.NET Web services	83	0.24
Remoting HTTP/Binary	80	0.67
Remoting TCP/SOAP	78	1.04
Remoting HTTP/Binary (IIS)	78	0.22
Remoting HTTP/Binary (IIS) Password	75	0.26
Remoting HTTP/SOAP	71	0.92
Remoting HTTP/SOAP (IIS)	67	0.34
Remoting HTTP/SOAP (IIS) Password	64	0.20
ASP.NET Web services - Integrated	62	0.14
Remoting HTTP/Binary (IIS) Integrated	59	0.15
Remoting HTTP/SOAP (IIS) Integrated	52	0.13

Empty Message, Cross-Process

Test	Avg. Calls/Second	Std. Deviation
Enterprise Services	14687	52.66
Enterprise Services (auth)	12293	31.71
Remoting TCP/Binary	3538	38.87
Remoting HTTP/Binary	1069	8.05
Remoting TCP/SOAP	1075	11.87
Remoting HTTP/SOAP	612	7.88
Remoting HTTP/Binary (IIS)	589	1.83
ASP.NET Web services	517	1.44
Remoting HTTP/Binary (IIS) Integrated	451	1.19
Remoting HTTP/Binary (IIS) Password	421	0.90
Remoting HTTP/SOAP (IIS)	406	1.76
ASP.NET Web services - Integrated	403	0.76
Remoting HTTP/SOAP (IIS) Integrated	336	0.57
Remoting HTTP/SOAP (IIS) Password	321	0.37

Empty Message. Cross-Machine

Test	Avg. Calls/Second	Std. Deviation
Enterprise Services (auth)	3068	25.35
Enterprise Services	3048	38.24
Remoting TCP/Binary	2609	77.28
Remoting TCP/SOAP	760	31.65
Remoting HTTP/Binary	704	8.00
Remoting HTTP/Binary (IIS)	479	2.60
Remoting HTTP/SOAP	413	10.63
ASP.NET Web services	399	2.00
Remoting HTTP/Binary (IIS) Password	351	1.98
Remoting HTTP/SOAP (IIS)	309	1.38
Remoting HTTP/SOAP (IIS) Password	252	1.25
Remoting HTTP/Binary (IIS) Integrated	156	0.52
ASP.NET Web services - Integrated	150	0.57
Remoting HTTP/SOAP (IIS) Integrated	133	0.28

Appendix C: Test Application Notes

Running the Test Application

You can download the test application at the location specified at the beginning of this article. To perform these tests on your own hardware, you'll need at least two machines: a client and a server. In addition, you'll need to have access to a SQL Server installation that includes the Northwind sample database. You will find a step-by-step guide on how to deploy, configure, and run this test suite in the file README.TXT that is part of the ZIP archive containing the source code.

Interesting Factoid – The Test App Is Technology Agnostic!

You will see in the test application that it is possible to create applications that are transport-agnostic. In the sample application, you'll find a method Test.CreateProxy() that creates a reference to a remote service and returns a client-side wrapper (a Web-service proxy or the corresponding TransparentProxy for ES and .NET Remoting) implementing a certain interface. When calling this method, you can specify which transport protocol should be used, which security settings, formatters, and so on. The remaining application does not care about the underlying protocol. This allows you to create a protocol-agnostic application, allowing you to choose the suitable protocols according to your application environment.

Appendix D: Software and Hardware Setup

The following tests have been performed with three machines: a client, an application server and a database server. All tests have been performed with a sufficient amount of available physical memory to prevent unnecessary paging.

Application Environment

• Client: Single-threaded console application, written in C#, built with Visual Studio.NET 2003 in Release Mode.
• ASP.NET Web service running in IIS 6.0 on Windows Server 2003
• Enterprise Services components in different configurations (Library Activation and Server Activation, no authentication and call-level authentication) running in COM+ 1.5 on Windows Server 2003
• A stand-alone .NET Remoting server as a C# console application, written in C#, built with Visual Studio .NET 2003 in Release Mode

Infrastructure Environment

Client:

• CPU: 2.4 GHz Intel P4
• Memory: 768 MB
• O/S: Windows Server 2003 Standard Edition (with all patches as of June 2004)
• .NET Framework 1.1 SP1

Application Server:

• CPU: 2.8 GHz Intel P4
• Memory: 1024 MB
• O/S: Windows Server 2003 Standard Edition
• .NET Framework 1.1 SP1

Database Server:

• CPU: 2.8 GHz Intel P4 Prescott, w/ HT, 800 MHz FSB
• Memory: 1024 MB
• O/S: Windows Server 2003 Standard Edition
• Database: SQL Server 2000 Enterprise Edition (with SP3A)
• .NET Framework 1.1 SP1

Network:

• 100 Mbps, switched Ethernet