jagan_peri https://docs.microsoft.com/archive/blogs/jagan_peri/feed.xml Wed, 17 Apr 2019 03:50:14 GMT en-US hourly 1 Using Enterprise Transient fault handling block with Azure Cache https://docs.microsoft.com/archive/blogs/jagan_peri/using-enterprise-transient-fault-handling-block-with-azure-cache Tue, 06 Nov 2012 08:31:00 GMT https://blogs.msdn.microsoft.com/jagan_peri/2012/11/06/using-enterprise-transient-fault-handling-block-with-azure-cache/  I mentioned in an earlier blog about the importance of having a retry mechanism in applications that use Azure Cache. An easy way to do this is by using the enterprise transient fault handling block.

I've noticed that an application using the Windows Azure SDK 1.8 (October 2012 release) and Transient Fault handling block (version 5.1.1209.0) can fail to start with the callstack below:

System.Reflection.TargetInvocationException occurred
 HResult=-2146232828
 Message=Exception has been thrown by the target of an invocation.
 Source=mscorlib
 StackTrace:
 at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)
 at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
 at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)
 at System.Activator.CreateInstance[T]()
 at Microsoft.Practices.TransientFaultHandling.RetryPolicy`1..ctor(RetryStrategy retryStrategy)
 at WebRole1._Default.Page_Load(Object sender, EventArgs e) in c:\Projects\WindowsAzure1\WebRole1\Default.aspx.cs:line 23
 InnerException: System.IO.FileLoadException
 HResult=-2146234304
 Message=Could not load file or assembly 'Microsoft.ApplicationServer.Caching.Core, Version=101.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
 Source=mscorlib
 FileName=Microsoft.ApplicationServer.Caching.Core, Version=101.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
 FusionLog==== Pre-bind state information ===

 

Here is a simple workaround to get past this issue - In your application config (app.config or web.config), add the following assembly redirection snippet:

 

<configuration>
 <runtime>
 <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
 <dependentAssembly>
 <assemblyIdentity name="Microsoft.ApplicationServer.Caching.Core"
 publicKeyToken="31bf3856ad364e35"
 culture="neutral" />
 <!-- Assembly versions can be redirected in application, 
 publisher policy, or machine configuration files. -->
 <bindingRedirect oldVersion="101.0.0.0" newVersion="1.0.0.0" />
 </dependentAssembly>
 </assemblyBinding>
 </runtime>
 </configuration>

 

 

]]> AppFabric 1.1 is now supported on Windows Server 2012 https://docs.microsoft.com/archive/blogs/jagan_peri/appfabric-1-1-is-now-supported-on-windows-server-2012 Tue, 06 Nov 2012 08:08:00 GMT https://blogs.msdn.microsoft.com/jagan_peri/2012/11/06/appfabric-1-1-is-now-supported-on-windows-server-2012/  

AppFabric 1.1 is now supported on Windows Server 2012. The download page has been updated to reflect this.

 

]]> Best Practices for using Windows Azure Cache/Windows Server Appfabric Cache https://docs.microsoft.com/archive/blogs/jagan_peri/best-practices-for-using-windows-azure-cachewindows-server-appfabric-cache Sun, 09 Sep 2012 00:16:00 GMT https://blogs.msdn.microsoft.com/jagan_peri/2012/09/09/best-practices-for-using-windows-azure-cachewindows-server-appfabric-cache/ We frequently get asked on best practices for using Windows Azure Cache/Appfabric cache.  The compilation below is an attempt at putting together an initial list of best practices. I’ll publish an update to this in the future if needed.

I’m breaking down the best practices to follow by various topics.

Using Cache APIs

1.    Have Retry wrappers around cache API calls

Calls into cache client can occasionally fail due to a number of reasons such as transient network errors, cache  servers being unavailable due to maintainance/upgrades or cache servers being low on memory. The cache client raises a DataCacheException with an errorcode that indicates the reason for the failure in these cases. There is a good overview of how an application should handle these exceptions in MSDN

It is a good practice for the application to implement a retry policy. You can implement a custom policy or consider using a framework like the TransientFault Handling Application Block

2.    Keep static instances of DataCache/DataCacheFactory

Instances of DataCacheFactory (and hence DataCache instances indirectly) maintain tcp connections to the cache servers. These objects are expensive to create and destroy. In addition, you want to have as few of these as needed to ensure cache servers are not overwhelmed with too many connections from clients.

You can find more details of connection management here. Please note that the ability to share connections across factories is currently available only in November 2011 release of the Windows Azure SDK (and higher versions). Windows Server appfabric 1.1 does not have this capability yet.

Overhead of creating new factory instances is lower if connection pooling is enabled. In general though, it is a good practice to pre-create an instance of DataCacheFactory/DataCache and use them for all subsequent calls to the APIs. Do avoid creating an instance of DataCacheFactory/DataCache on each of your request processing paths. 

3.    WCF Services using Cache Client

It is a common practice for WCF services to use cache to improve their performance. However, unlike asp.net web applications, wcf services are susceptible for IO-thread starvation issues when making blocking calls (such as cache API calls) that further require IO threads to receive responses (such as responses from cache servers).

This issue is described in detail in the following KB article – The typical symptom that surfaces if you run into this is that when you’ve a sudden burst of load, cache API calls timeout. You can confirm if you are
running into this situation by plotting the thread count values against incoming requests/second as shown in the KB article.

4.    If app is using Lock APIs - Handle ObjectLocked, ObjectNotLocked exceptions

If you are using lock related APIs, please ensure you are handling exceptions such as ObjectLocked ( and ObjectNotLocked (Object being referred to is not locked by any client) error.

GetAndLock can fail with “<ERRCA0011>:SubStatus<ES0001>:Object being referred to is currently locked, and cannot be accessed until it is unlocked by the locking client. Please retry later.” error if another caller has acquired a lock on the object.

The code should handle this error and implement an appropriate retry policy.

PutAndUnlock can fail with “<ERRCA0012>:SubStatus<ES0001>:Object being referred to is not locked by any client” error.

This typically means that the lock timeout specified when the lock was acquired was not long enough because the application request took longer to process. Hence the lock expired before the call to PutAndUnlock and the cache server returns this error code.

The typical fix here is to both review your request processing time as well as set a higher lock timeout when acquiring a lock.

You can also run into this error when using the session state provider for cache. If you are running into this error from session state provider, the typical solution is to set a higher executionTimeout for your web app.

Session State Provider Usage

You can find more info about asp.net session state providers for appfabric cache here and azure cache here.

The session state provider has an option to store the entire session as 1 blob (useBlobMode=”true” which is the default), or to store the session as individual key/value pairs.

useBlobMode="true" incurs fewer round trips to cache servers and works well for most applications.

If you've a mix of small and large objects in session, useBlobMode=”false” (a.ka. granular mode) might work better since it will avoid fetching the entire (large) session object for all requests. The cache should also be marked as nonEvictable cache if useBlobMode=”false” option is being used. Because Azure shared cache does not give you the ability to mark a cache as non evictable, please note that useBlobMode=”true” is the only supported option against Windows Azure Shared cache.

Performance Tuning and Monitoring

 

            Tune MaxConnectionsToServer

Connection management between cache clients and servers is described in more detail here. Consider tuning MaxConnectionToServer setting. This setting controls the number of connections from a client to cache
servers. (MaxConnectionsToServer * Number of DataCacheFactory Instances *Number of Application Processes) is a rough value for the number of connections that will be opened to each of the cache servers. So, if you have 2 instances of your web role with 1 cache factory and MaxConnectionsToServer set to 3, there will be 3*1*2 = 6 connections opened to each of the cache servers.

Setting this to number of cores (of the application machine) is a good place to start. If you set this too high, a large number of connections can get opened to each of the cache servers and can impact throughput.

If you are using Azure cache SDK 1.7, maxConnectionsToServer is set to the default of number of cores (of the application machine). The on-premise appfabric cache (v1.0/v1.1) had the default as one, so that value might need to be tuned if needed.

            Adjust Security Settings

The default security settings for on-premise appfabric cache is to run with security on at EncryptAndSign protection level. If you are running in a trusted environment and don’t need this capability, you can turn this off by explicitly setting security to off.

The security model for Azure cache is different and theabove adjustment is not needed for azure cache.

            Monitoring

There is also a good set of performance counters on the cache servers that you can monitor to get a better understanding of cache performance issues. Some of the counters that are typically useful to troubleshoot issues include:

1)     %cpu used up by cache service

2)     %time spent in GC by cache service

3)     Total cache misses/sec – A high value here can indicate your application performance might suffer because it is not able to fetch data from cache. Possible causes for this include eviction and/or expiry
of items from cache.

4)     Total object count – Gives an idea of how many items are in the cache. A big drop in object count could mean eviction or expiry is taking place.

5)     Total client reqs/sec – This counter is useful in giving an idea of how much load is being generated on the cache servers from the application. A low value here usually means some sort of a bottleneck
outside of the cache server (perhaps in the application or network) and hence very little load is being placed on cache servers.

6)     Total Evicted Objects – If cache servers are constantly evicting items to make room for newer objects in cache, it is usually a good indication that you will need more memory on the cache servers to hold
the dataset for your application.

7)     Total failure exceptions/sec and Total Retry exceptions/sec

 

 Lead host vs Offloading

This applies only for the on-premise appfabric cache deployments. There is a good discussion of the  tradeoffs/options in this blog - As noted in the blog, with v1.1, you can use sql to just store config info and use lead-host model for cluster runtime. This option is attractive if setting up a highly-available sql server for offloading purposes is hard.

Other Links

Here are a set of blogs/articles that provide more info on some of the topics covered above.

1)     Jason Roth and Jaime Alva have written an  article providing additional guidance to developers using Windows
Azure Caching.

2)     Jaime Alva’s blog post on logging/counters for On-premise appfabric cache.

3)     MSDN article about connection management between cache client & servers.

4)     Amit Yadav and Kalyan Chakravarthy’s blog on lead host vs offloading options for cache clusters.

5)     MSDN article on common cache exceptions and Transient Fault Handling Application Block

 

 

]]> Custom Serialization https://docs.microsoft.com/archive/blogs/jagan_peri/custom-serialization Wed, 22 Aug 2012 23:42:00 GMT https://blogs.msdn.microsoft.com/jagan_peri/2012/08/22/custom-serialization/ Azure Cache (Preview) release added support for using a custom serializer to serialize objects that are stored in the cache.

NetDataContractSerializer is the default serialization mechanism used by Azure cache. Many times, applications might need to use custom serialization for performance or compatibility reasons.

 An application could explicitly serialize objects and store them in cache as byte[]'s. However, this  results in sub-optimal performance when local cache is used since the cost of deserialization will be incurred even when retrieving items from local cache. Using the custom serializer support, access to objects in local cache does not incur any deserialization costs (on-par with default serialization approach).

 

Let's walk through a code example

 

Instead of the default NetDataContractSerializer, we will create a custom serializer that uses Json serialization to serialize instances of Person class.  Since we will use DataContractJsonSerializer for this example, we need to attribute this class as a DataContract.

 

     [DataContract]

    publicclassPerson

    {

        public Person(string firstName, string lastName)

        {

            this.FirstName = firstName;

            this.LastName = lastName;

        }

 

        [DataMember]

        publicstring FirstName { get; set; }

 

        [DataMember]

        publicstring LastName { get; set; }

 

    }

 

Implement the IDataCacheObjectSerializer interface

 

Custom serializers need to implement the IDataCacheObjectSerializer interface - It has 2 methods (one each for Serialize and Deserialize).  For this example, we will keep the implementation simple and use the System.Runtime.Serialization.Json.DataContractJsonSerializer class to do the actual work.

 namespace WorkerRole1

{

    /*

     * Simple implementation of IDataCacheObjectSerializer

     * Uses DataContractJsonSerializer

     */

    publicclassCustomSerializer: IDataCacheObjectSerializer

    {

        public CustomSerializer()

        { }

        publicobject Deserialize(System.IO.Stream stream)

        {

            Trace.WriteLine("CustomSerializer.Deserialize method called");

           

            DataContractJsonSerializer serializer = newDataContractJsonSerializer(typeof(Person));

 

            return serializer.ReadObject(stream);

 

        }

 

        publicvoid Serialize(System.IO.Stream stream, object value)

        {

 

            Trace.WriteLine("CustomSerializer.Serialize method called");

                       

            DataContractJsonSerializer serializer = newDataContractJsonSerializer(typeof(Person));

 

            serializer.WriteObject(stream, value);

 

 

        }

    }

} 

 

Configure Cache Client to use the Custom Serializer

 

This can be done either through xml configuration or through code.

 

Through XML Configuration: 

Add the <serializationProperties /> element specifying serializer="CustomSerializer" and setting customSerializerType to the type that implements the custom serialization. In this case, WorkerRole1.CustomSerializer is the name of the class located in WorkerRole1.dll

 

  <dataCacheClients>

    <tracing sinkType="DiagnosticSink" traceLevel="Verbose" />

    <!-- This is the default config which is used for all Named Caches

     This can be overriden by specifying other dataCacheClient sections with name being the NamedCache name -->

    <dataCacheClient name="default" useLegacyProtocol="false">

      <autoDiscover isEnabled="true" identifier="WorkerRole1" />

      <!--<localCache isEnabled="true" sync="TimeoutBased" objectCount="100000" ttlValue="300" />-->

      <serializationProperties serializer="CustomSerializer" customSerializerType="WorkerRole1.CustomSerializer,WorkerRole1" />

    </dataCacheClient>

  </dataCacheClients>

 

You can also configure the custom serialization properties through code as shown below:

 

            DataCacheSerializationProperties customSerializerProps = newDataCacheSerializationProperties(

                                        DataCacheObjectSerializerType.CustomSerializer,

                                        newCustomSerializer());

 

            DataCacheFactoryConfiguration config = newDataCacheFactoryConfiguration("default");

 

            // Specify CustomSerialization Properties.

            config.SerializationProperties = customSerializerProps;

 

            DataCacheFactory factory = newDataCacheFactory(config);

 

            DataCache cache = factory.GetDefaultCache();

 

Testing the custom serializer

 

 When you try out the code below that adds instances of type Person to cache, you can verify that the CustomSerializer class methods are being called.

Person p = new Person("Jagan", "Peri");
cache.Put("key1", p);
Person p2 = (Person)cache.Get("key1");

Limitations

 

1) As you can see from the above code snippets, there can be only one custom serializer specified at a DataCacheFactory level. If your application stores multiple types in the cache, the custom serializer needs to handle all these types.

2) Some APIs (increment, decrement, Append, Prepend) are not supported with custom serialization. These APIs require the cache server to understand the value stored in the cache
so it can perform these operations atomically on the value. Since custom serializer  is only available on the client, these APIs cannot be used with custom serialization.

 

Sample Project

 

 I've attached a sample worker role that uses co-located cache and demonstrates the use of the above interface.

Key files to look at in this project are:

Person.cs - Type that is stored in cache

CustomSerializer.cs - Custom Serializer implementation

WorkerRole.cs - Configures CustomSerialization and performs cache put/get API calls.

 

 

 

 

 

CustomSerializer.zip

]]> Azure Cache (Preview) https://docs.microsoft.com/archive/blogs/jagan_peri/azure-cache-preview Tue, 21 Aug 2012 10:51:00 GMT https://blogs.msdn.microsoft.com/jagan_peri/2012/08/21/azure-cache-preview/ As a member on the Azure cache team, I was very thrilled when we released the Azure Cache (Preview) as part of the June 2012 Azure SDK.

 

The preview offers a great caching solution for applications running on Azure. It provides a lot of benefits over Azure shared cache around aspects such as pricing, quotas/throttling and flexible deployment options. 

In addition to the above areas of improvement over Azure shared cache, we also added a lot of other enhancements as part of the preview release.  I've included an overview of these enhancements below.

 

New APIs

      The preview release contains the following new APIs:

 

  1. DataCache() - You don’t need to use DataCacheFactory any more to create an instance of DataCache.
  2. Increment Decrement APIs - Many applications use cache to maintain numeric counters. These APIs are pretty useful for operating on numeric types stored in cache.
  3. Append Prepend APIs - Append or Prepend to a string that is stored in cache.
  4. Cache level BulkGet - This API supports bulk get of items from a cache. The earlier version of the API only allowed bulk get of items in a named region.
  5. Clear()   - Clears a cache.

 

Memcache Protocol Support

      The preview release of Azure cache supports memcache protocol (both binary and text). Azure cache has many advantages over memcached. With this protocol support, existing applications that were using memcache can migrate easily to Azure and leverage the benefits of Azure Cache.

 

     MSDN has more info on how to enable memcache protocol support. I'll expand on the protocol differences between memcache and Azure Cache in a later post.

 

Appfabric Cache Feature Parity

     All the familiar features of Appfabric Cache (high-availability, notifications, regions, tags etc) can be used with the Azure Cache preview. One exception to this is Read-Through/Write-Behind capability. This is not yet supported in the preview.

 

Improved Performance

     There were a lot of under-the-hood changes to improve the performance (both latency and throughput) of Azure cache.

 

Access to Azure shared cache goes through a LoadBalanced input endpoint which has impact both on latency and throughput (more on this in a later blog). In the preview release, there is no need to go through a load balanced endpoint since the cache role is in the same deployment as the application, the application can use internal endpoints to connect to the cache role instances.

 

Also, as part of this preview, we implemented a more efficient wire protocol between the cache client and servers that has given us good results.

 

The above changes provide significant latency and throughput improvements compared to the Azure shared cache.

 

Improved Management Experience

You can manage the azure cache preview roles just like you manage any other azure application. You can use the service management APIs to increase/decrease the number of cache role instances. You can also monitor a rich set of performance counters to help understand how the cache is performing for your application.

 

You can remote desktop to the cache role instances to look at the cache perf counters, or configure the list of performance counters to collect through code

 

Useful Links

 

Here are a few links that I've found useful in getting started on the azure cache preview.

 

  1. MSDN's Azure Caching section has a lot of good info about the preview release.
  2. You can watch Shyam Seshadri's presentation at TechEd about these enhancements.
  3. Prashant Kumar's blog also gives a good overview of what is new in Azure caching.

 

Over the next few weeks, I plan to do a deep dive into many of the above aspects of Azure Cache Preview.

 

 

 

 

 

]]>