Windows Azure Storage Emulator 1.8

In our continuous endeavor to enrich the development experience, we are extremely pleased to announce the new Storage Emulator, which has much improved parity with the Windows Azure Storage cloud service.

What is Storage Emulator?

Storage Emulator emulates the Windows Azure Storage blob, table and queue cloud services on local machine which helps developers in getting started and basic testing of their storage applications locally without incurring the cost associated with cloud service. This version of Windows Azure Storage emulator supports Blob, Tables and Queues up until REST version 2012-02-12.

How it works?

Storage Emulator exposes different http end points (port numbers: 10000 for blob, 10001 for queue and 10002 for table services) on local host to receive and serve storage requests. Upon receiving a request, the emulator validates the request for its correctness, authenticates it, authorizes (if necessary) it, works with the data in SQL tables and file system and finally sends a response to the client.

Delving deeper into the internals, Storage Emulator efficiently stores the data associated with queues and tables in SQL tables. However, for blobs, it stores the metadata in SQL tables and actual data on local disk one file for each blob, for better performance. When deleting blobs, the Storage Emulator does not synchronously clean up unreferenced blob data on disk while performing blob operations. Instead it compacts and garbage collects such data in the background for better scalability and concurrency.

Storage Emulator Dependencies:

• SQL Express or LocalDB
• .NET 4.0 or later with SQL Express or .NET 4.0.2 or later with LocalDB

Installing Storage Emulator

Storage Emulator can work with LocalDB, SQL express or even a full blown SQL server as its SQL store.

The following steps would help in getting started with emulator using LocalDB.

1. Install .NET framework 4.5 from here.
2. Install X64 or X86 LocalDB from here.
3. Install the Windows Azure Emulator from here.

Alternatively, if you have storage emulator 1.7 installed, you can do an in place update to the existing emulator. Please note that storage emulator 1.8 uses a new SQL schema and hence a DB reset is required for doing an in place update, which would result in loss of your existing data.

The following steps would help in performing an in place update.

1. Shutdown the storage emulator, if running
2. Replace the binaries ‘Microsoft.WindowsAzure.DevelopmentStorage.Services.dll’, ‘Microsoft.WindowsAzure.DevelopmentStorage.Store.dll’ and ‘Microsoft.WindowsAzure.DevelopmentStorage.Storev4.0.2.dll’, located at storage emulator installation path (Default path is "%systemdrive%\Program Files\Microsoft SDKs\Windows Azure\Emulator\devstore") with those available here.
3. Open up the command prompt in admin mode and run ‘dsinit /forceCreate’ to recreate the DB. You can find the ‘dsinit’ tool at the storage emulator installation path.
4. Start the storage emulator

What’s new in 1.8?

Storage emulator 1.8 supports the REST version 2012-02-12, along with earlier versions. Below are the service specific enhancements.

Blob Service Enhancements:

In 2012-02-12 REST version, Windows Azure Storage cloud service introduced support for container leases, improved blob leases and asynchronous copy blob across different storage accounts. Also, there were enhancements for blob shared access signatures and blob leases in the 2012-02-12 version. All those new features are supported in Storage Emulator 1.8.

Since the emulator has just one built in account, one can initiate cross account copy blob by providing a valid cloud based URL. Emulator serves such cross account copy blob requests, asynchronously, by downloading the blob data, in chunks of 4MB, and updating the copy status.

To know more about the new features in general, the following links would be helpful:

• Updated Shared Access Signatures
• Container Leases and improved blob leases
• Asynchronous copy blob with ability to copy across accounts

Storage Emulator 1.8 also garbage collects the unreferenced page blob files which may be produced as a result of delete blob requests, failed copy blob requests etc.

Queue Service Enhancements:

In 2012-02-12 REST version, Windows Azure Storage cloud service introduced support for Queue shared access signatures (SAS). Storage Emulator 1.8 supports Queue SAS.

Table Service Enhancements:

In 2012-02-12 REST version, Windows Azure Storage cloud service introduced support for table shared access signatures (SAS). Storage Emulator 1.8 supports Table SAS.

In order to achieve full parity with Windows Azure Storage table service APIs, the table service in emulator is completely rewritten from scratch to support truly schema less tables and expose data for querying and updating using ODATA protocol. As a result, Storage Emulator 1.8 fully supports the below table operations which were not supported in Emulator 1.7.

• Query Projection: You can read more about it here.
• Upsert operations: You can read more about it here.

Known Issues/Limitations

• The storage emulator supports only a single fixed account and a well-known authentication key. They are: Account name: devstoreaccount1, Account key: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
• The URI scheme supported by the storage emulator differs from the URI scheme supported by the cloud storage services. The development URI scheme specifies the account name as part of the hierarchical path of the URI, rather than as part of the domain name. This difference is due to the fact that domain name resolution is available in the cloud but not on the local computer. For more information about URI differences in the development and production environments, see “Using storage service URIs” section in Overview of Running a Windows Azure Application with the Storage Emulator.
• The storage emulator does not support Set Blob Service Properties or SetServiceProperties for blob, queue and table services.
• Date properties in the Table service in the storage emulator support only the range supported by SQL Server 2005 (For example, they are required to be later than January 1, 1753). All dates before January 1, 1753 are changed to this value. The precision of dates is limited to the precision of SQL Server 2005, meaning that dates are precise to 1/300th of a second.
• The storage emulator supports partition key and row key property values of less than 900 bytes. The total size of the account name, table name, and key property names together cannot exceed 900 bytes.
• The storage emulator does not validate that the size of a batch in an entity group transaction is less than 4 MB. Batches are limited to 4 MB in Windows Azure, so you must ensure that a batch does not exceed this size before transitioning to the Windows Azure storage services.
• Avoid using ‘PartitionKey’ or ‘RowKey’ that contains ‘%’ character due to the double decoding bug
• Get messages from queue might not return messages in the strict increasing order of message’s ‘Insertion TimeStamp’ + ‘visibilitytimeout’

Summary

Storage Emulator 1.8 has a great extent of parity with the Windows Azure Storage cloud service in terms of API support and usability and we will continue to improve it. We hope you all like it and please share your feedback with us to make it better.

Nagarjun Guraja

Windows Azure Storage

Comments

• Anonymous
  October 30, 2012
  This is extremely useful. Thank you !!!

• Anonymous
  October 30, 2012
  The comment has been removed

• Anonymous
  November 03, 2012
  @Jabe, this is a known issue and we are looking at fixing this. Sorry for the inconvenience this has caused.

• Anonymous
  November 18, 2012
  Storage Emulator 1.8 does not support request URL with more than 300 characters, DataServiceRequestException will be thrown in this case: (Azure SDK 1.7) System.Data.Services.Client.DataServiceRequestException: An error occurred while processing this request. ---> System.Data.Services.Client.DataServiceClientException: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""www.w3.org/.../strict.dtd"> <HTML><HEAD> <META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD> <BODY><h2>Bad Request - Invalid URL</h2> <hr><p>HTTP Error 400. The request URL is invalid.</p> </BODY></HTML>   at System.Data.Services.Client.DataServiceContext.SaveResult.<HandleBatchResponse>d__1e.MoveNext()   --- End of inner exception stack trace ---   at System.Data.Services.Client.DataServiceContext.SaveResult.HandleBatchResponse()   at System.Data.Services.Client.DataServiceContext.SaveResult.EndRequest()   at System.Data.Services.Client.DataServiceContext.SaveChanges(SaveChangesOptions options) Test code:        public class TestEntity : TableServiceEntity        {            public TestEntity(string pk, string rk) : base(pk, rk) { }        }        static void TestUrlLimit(int urlLength)        {            var myAccount = CloudStorageAccount.DevelopmentStorageAccount;            var tableClient = myAccount.CreateCloudTableClient();            string table = "TestTable";            tableClient.CreateTableIfNotExist(table);            var context = tableClient.GetDataServiceContext();            string pk = "0";            string rk = new string('0', urlLength - 77); // Set the length of row key to build a url with "urlLength" characters            var entity = new TestEntityX(pk, rk);            context.AttachTo(table, entity);            context.UpdateObject(entity);            Uri uri;            context.TryGetUri(entity, out uri);            Console.WriteLine("URL length: {0}", uri.AbsoluteUri.Length);            try            {                context.SaveChanges();                Console.WriteLine("Success");            }            catch (Exception e)            {                Console.WriteLine("Error: {0}", e);            }        }        static void Test()        {            TestUrlLimit(300); // Success            TestUrlLimit(301); // Error        } I'm not sure whether it is a known issue/limitation of Azure Storage Emualtor. Currently, I just use "context.SaveChanges(SaveChangesOptions.Batch)" to avoid the exception in this case. Is there any way to solve this problem? Thanks, Dong

• Anonymous
  November 19, 2012
  The comment has been removed

• Anonymous
  November 20, 2012
  Hi Jean, I have checked the values under HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesHTTPParameters and found the root cause of URL length issue: The default value of UrlSegmentMaxLength is 260 (support.microsoft.com/.../820129), so the max length of Azure request url to emulator is 40 ("127.0.0.1/.../") + 260 (segment of pk & rk) = 300. I changed value of UrlSegmentMaxLength to 0 (length that is bounded by the maximum value of a ULONG) and there's no URL issue now. Many thanks for your help!