Windows Azure Roles recycling between Initialize.. busy… status, while deploying the application with Azure Cache (Preview) feature

In recent past, I have noticed that several Windows Azure Customers having issues, while deploying the applications, which leverages Windows Azure Cache (Preview) feature shipped with Azure SDK 1.7. The usual symptoms should be role recycling with various status messages like Initializing …. Busy … etc.   

After doing the detailed investigations on this kind of reported issues for the application, which leverages Windows Azure Cache (Preview) feature, it noticed that majority of this issue caused due to one of the below reasons.

 

1.Storage Account Details for maintaining the cache cluster’s runtime state : I have noticed that developers forget to update the proper Windows Azure Storage account credential for maintaining the cache cluster’s runtime state before deploying the Windows Azure Cache (Preview) feature enabled application to Azure Portal. So please verify, you have already modify the appropriate storage credential details for maintaining the cache cluster’s runtime state. Please refer below screen shot :

 

2. Inadequate Cache Size for Co-located Role : The deployed Web/Worker Role will be recycled, if you provide the Cache Size (%) below 15% , and also, if you provide the value less than 15%, VS 2010 does gives a warning message. So please make sure you have provided the appropriate Cache size (ideally 30%) for the Co-located Role . Please refer below screen shot for the same :

 

       Typically for #2, you will find below mentioned details from the memory crash dump for ‘CacheInstaller.exe’ from the VM:

 

0:000> |
. 0 id: ce0 examine name: E:\plugins\Caching\CacheInstaller.exe

0:000> !threads
PDB symbol for clr.dll not loaded
ThreadCount: 7
UnstartedThread: 0
BackgroundThread: 6
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
                                           PreEmptive Lock
       ID OSID ThreadOBJ State GC GC Alloc Context Domain Count APT Exception
   0 1 624 0000000000220850 a020 Enabled 00000000021abf48:00000000021abfd0 0000000000213a80 0 MTA Microsoft.ApplicationServer.Caching.DataCacheException (00000000020be488)
   2 2 d18 0000000000225a60 b220 Enabled 0000000000000000:0000000000000000 0000000000213a80 0 MTA (Finalizer)
   4 3 f68 000000000190f180 1009220 Enabled 0000000001d831b8:0000000001d83fd0 0000000000213a80 0 MTA (Threadpool Worker)
   5 4 c68 000000000190e9c0 1009220 Enabled 0000000001d12690:0000000001d13fd0 0000000000213a80 0 MTA (Threadpool Worker)
   7 5 f04 000000000192b120 1009220 Enabled 0000000001d7df30:0000000001d7dfd0 0000000000213a80 0 MTA (Threadpool Worker)
   8 6 5e4 000000001a8d68a0 8009220 Enabled 0000000000000000:0000000000000000 0000000000213a80 0 MTA (Threadpool Completion Port)
   9 7 bec 000000001a8e7770 100a220 Enabled 0000000000000000:0000000000000000 0000000000213a80 0 MTA (Threadpool Worker)

0:000> !pe 00000000020be488
Exception object: 00000000020be488
Exception type: Microsoft.ApplicationServer.Caching.DataCacheException
Message: Exception was thrown while setting cache host size
InnerException: Microsoft.ApplicationServer.Caching.DataCacheException, Use !PrintException 00000000020be1b8 to see more.
StackTrace (generated):
    SP IP Function
    00000000003BC6E0 000007FF0019DB8F Microsoft_ApplicationServer_Caching_DedicatedCacheProviders!Microsoft.ApplicationServer.Caching.Configuration.AzureClusterHostConfigurationReader.SetCacheHostSize()+0x32f
    00000000003BE810 000007FF001942B8 Microsoft_ApplicationServer_Caching_Core!Microsoft.ApplicationServer.Caching.ClusterConfigReader..ctor(Microsoft.ApplicationServer.Caching.ClusterConfigElement)+0x288
    00000000003BE8B0 000007FF00193E83 Microsoft_ApplicationServer_Caching_Core!Microsoft.ApplicationServer.Caching.ClusterConfigurationFactory.GetReader(Microsoft.ApplicationServer.Caching.ClusterConfigElement)+0x153
    00000000003BE8F0 000007FF00193904 Microsoft_ApplicationServer_Caching_AzureCommon!Microsoft.ApplicationServer.Caching.AzureCommon.Package.InitializeExternalStoreIfRequired()+0x54
    00000000003BE950 000007FF00193852 CacheInstaller!Microsoft.ApplicationServer.Caching.CacheInstaller.CloudPackage.InstallAndStart()+0x12
    00000000003BE990 000007FF001802E6 CacheInstaller!Microsoft.ApplicationServer.Caching.CacheInstaller.CacheInstaller.Main(System.String[])+0x146

0:000> !do 00000000020be0b8
Name: System.String
MethodTable: 000007fee7a76ac8
EEClass: 000007fee75fed58
Size: 254(0xfe) bytes
File: D:\windows\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String: Value of CacheSizePercentage too small. Please increase value to support at least 32 MB of Cache (user data space)

 

References :

Windows Azure Caching (Preview)
https://msdn.microsoft.com/en-us/library/windowsazure/hh914153.aspx

How to Use Windows Azure Caching (Preview)
https://www.windowsazure.com/en-us/develop/net/how-to-guides/cache/

Capacity Planning Considerations for Windows Azure Caching (Preview)
https://msdn.microsoft.com/en-us/library/windowsazure/hh914129.aspx