ESE/ESENT Database Stuff https://docs.microsoft.com/archive/blogs/laurionb/feed.xml Wed, 17 Apr 2019 03:24:11 GMT en-US hourly 1 ManagedEsent 1.6 released. LINQ support for PersistentDictionary. https://docs.microsoft.com/archive/blogs/laurionb/managedesent-1-6-released-linq-support-for-persistentdictionary Tue, 15 Feb 2011 15:09:20 GMT https://blogs.msdn.microsoft.com/laurionb/2011/02/15/managedesent-1-6-released-linq-support-for-persistentdictionary/ There is a new release of ManagedEsent on Codeplex. There are some new APIs, bugfixes and performance improvements.

The PersistentDictionary class now supports LINQ -- it can examine a query and retrieve only matching records from the database. Some examples of LINQ queries that work this way:

from x in persistentDictionary where x.Key > 3 select x.Value;

from x in persistentDictionary where x.Key.StartsWith("b") select x.Value

from x in persistentDictionary where (x.Key.CompareTo("a") > 0 && x.Key.CompareTo("c") <= 0) || x.Key.StartsWith("c") select x.Value;

I added a new sample application for the PersistentDictionary, which stores some performance data and allows a simple query. 

For IronPython users there are performance enhancements to the DBM-like esedb module and a new esedbshelve module that provides shelf functionality.

 

 

 

 

 

]]> ESENT Multivalue tutorial https://docs.microsoft.com/archive/blogs/laurionb/esent-multivalue-tutorial Thu, 03 Jun 2010 14:48:10 GMT https://blogs.msdn.microsoft.com/laurionb/2010/06/03/esent-multivalue-tutorial/ Multivalued columns can be very useful for simplifying data schema and improving performance, but they can be complex to use (they also make Data Normalization geeks feel uneasy). I put a multi-value tutorial up on the ManagedEsent wiki. It uses the managed APIs but would be easy to translate to unmanaged code.

https://managedesent.codeplex.com/wikipage?title=MultiValueTutorial

]]> Using ManagedEsent from F# https://docs.microsoft.com/archive/blogs/laurionb/using-managedesent-from-f Tue, 02 Mar 2010 11:32:00 GMT https://blogs.msdn.microsoft.com/laurionb/2010/03/02/using-managedesent-from-f/ Cameron Taggart has a blog entry on using ManagedEsent from F#: 

https://blog.ctaggart.com/2010/02/microsofts-extensible-storage-engine.html

The ESENT programming model centers around a database cursor (JET_TABLEID). The cursor is located on a record, can navigate to other records individual columns can be set/retrieved. This is quite different than a functional programming model, which would center around a stream (enumeration) of records instead of a stateful cursor. I find it interesting seeing what it takes to get the two models working together.

 

]]> Version 1.5 of ManagedEsent released. PersistentDictionary goodness https://docs.microsoft.com/archive/blogs/laurionb/version-1-5-of-managedesent-released-persistentdictionary-goodness Sat, 12 Sep 2009 22:02:00 GMT https://blogs.msdn.microsoft.com/laurionb/2009/09/12/version-1-5-of-managedesent-released-persistentdictionary-goodness/ There is a new release of ManagedEsent on Codeplex. There are some new APIs, constants for the new Windows 7 ESENT features, bugfixes and performance improvements.

 There is also a new Esent.Collections.dll which contains a PersistentDictionary generic class. This is a drop-in compatible replacement for the generic Dictionary or SortedDictionary classes for applications that need very simple data persistence. The database is stored in a location specified in the dictionary constructor.

var dict = new PersistentDictionary<string, int>("DataDirectory");
dict["foo"] = 17;
dict["bar"] = 18;

The dictionary entries are persisted automatically and updates are transacted providing data consistency and crash resilience.

]]> 6 New ESENT features in Windows 7 https://docs.microsoft.com/archive/blogs/laurionb/6-new-esent-features-in-windows-7 Tue, 18 Aug 2009 17:35:00 GMT https://blogs.msdn.microsoft.com/laurionb/2009/08/18/6-new-esent-features-in-windows-7/ A quick look at some of the new features that are available in the Windows 7 version of ESENT. You'll need esent.h from the Windows 7 SDK to see these definitions:

1. Column Compression

Columns of type JET_coltypLongBinary and JET_coltypLongText can now be compressed. There are two ways of enabling compression.

  • Create the column with the JET_bitColumnCompressed flag. This can only be used with long-value columns, otherwise ESENT will return JET_errColumnCannotBeCompressed.
  • Set the column with JET_bitSetCompressed. This flag can be used with any column type, we just ignore it for non-compressable columns. Alternatively you can use JET_bitSetUncompressed when updating a compressed column with data that you don't want compressed.

How does compression work?

For 'short' columns (less than 1kb in size) we look to see if all the data is 7-bit ASCII characters. If so we compress each character to its 7-bit value. This is the only compression done for short values. It was added to help with the fact that Exchange stores a lot of strings which are often just 7-bit ASCII values, e-mail headers for example. Turning a 16-bit Unicode character to a 7-bit ASCII value gives a large savings and is cheap to do.

For 'long' columns (greater than 1kb) we apply Microsoft's XPRESS compression, which is similar to ZIP or RAR compression. We only store the compressed data if it is smaller than the uncompressed data. Of course determining that data doesn't compress well costs CPU cycles so if you are storing already compressed data you shouldn't use compressed columns.

Unlike SQL Server's page compression this feature is designed to compress large column values, not remove duplication when multiple records have similar values. This is done because this feature was added for Microsoft Exchange, where most of the data is stored in relatively large BLOB columns (e.g. e-mail bodies).

What else do I need to know?

Nothing, really. A compressed column looks just like a normal column and can be indexed, set and retrieved as normal. There is no way to retrieve the compressed data, you will always get uncompressed data when calling JetRetrieveColumn or a similar API. If you want to know how well a record compressed use the new JetGetRecordSize2 API which gives both the logical (uncompressed) and physical (compressed) size of a record.

2. 32kb and 16kb page support

ESENT now supports 32kb, 16kb and 2kb pages. Changing the page size affects b-tree performance and you may find your application runs faster with bigger pages. Bigger pages also allow more columns to be set in one record before getting JET_errRecordTooLarge.

3. JetPrereadKeys

If you want to retrieve non-contiguous records quickly you can use JetMakeKey to calculate their keys and then call JetPrereadKeys. Prereading the records will issue the I/O as efficiently as possible, which makes retrieving the records a LOT faster if I/O is involved. Records must be preread in either ascending or descending key order. JetPrereadKeys may not be able to preread all the records so it tells you how many keys were preread so that you can issue a preread call for the rest after processing the initial batch.

4. Space Hints

In older versions of ESENT you could specify the initial size of a table/index, but didn't have much control over how the table size grew. Space hints allow you to give the initial size of a table/index and then indicate how much space the table should use as it grows. This feature is designed to allow databases with large numbers of tables where the tables start out small (perhaps one page) and then grow their allocation size so that they have contiguous space.

5. JET_paramWaypointLatency and JET_bitReplayIgnoreLostLogs

Write-ahead logging depends on the underlying storage being durable, which isn't really true for inexpensive desktop-class machines. To improve database recoverability we have added a 'Lost Log Resiliency' (LLR) feature to the Windows 7 version of ESENT. Setting the JET_paramWaypointLatency parameter to a non-zero value will enable the LLR feature, causing ESENT to delay writes to the database until N logfiles have been generated, where N is the waypoint depth.

For example, if the waypoint is set to 1 and the current log generation is 28 then ESENT will not write any pages to the database that have been modified by log records in generation 28. This means that the database can be recovered without the most recent log but there will be data loss! If you are comfortable with some potential data loss and want to emphasise database recovery then set JET_paramWaypointLatency to 1 (or a small value, there is a performance penalty for maintaining the waypoint). You have to use the JET_bitReplayIgnoreLostLogs option with JetInit2 to tell ESENT that it is OK to perform a lossy recovery.

This will allow you to recover your database even if a system crash loses the most recent log. In our power-down testing this improves the chances of a sucessful database recovery after a machine with non-battery-backed write caching is powered down.


6. JET_bitTermDirty

If you want to shutdown REALLY quickly you can use JET_bitTermDirty, which won't even flush the database cache. Recovery will take longer the next time but no data will be lost.


I know this was a very quick summary. If you have specific questions, leave them in the comments or use the 'contact me' area and I'll try to answer them or provide samples.

 

]]> Multimethods in C# 4.0 with 'dynamic' https://docs.microsoft.com/archive/blogs/laurionb/multimethods-in-c-4-0-with-dynamic Thu, 13 Aug 2009 15:01:00 GMT https://blogs.msdn.microsoft.com/laurionb/2009/08/13/multimethods-in-c-4-0-with-dynamic/ (Continuing from the post on implementing the Visitor pattern in C# 4.0)

A virtual method is polymorphic in one dimension (the runtime class of the object the method is attached to). Multimethods (multiple dispatch) are polymorphic in multiple dimensions, which can be very useful in some cases. The dynamic keyword in C# 4.0 allows for method selection that depends on the runtime class of the arguments, not just the attached object. This allows true multimethods. (Of course the lookup rules for dynamic are complex so extremely complex cases might not resolve the way you want).

The basic idea is to create method overloads which take all the (subclass1, subclass2) combinations that you are interested in and then have the (superclass, superclass) method cast its arguments to dynamic and then call the method again and let runtime resolution do its thing. In the spirit of the Wikipedia article on multimethods, here is a spaceship/asteroid collision example:

 (code edited to separate the names of the methods)

using System;

namespace MultiMethods
{
    class Program
    {
        class Thing { }
        class Asteroid : Thing { }
        class Spaceship : Thing { }

        static void CollideWithImpl(Asteroid x, Asteroid y)
        {
            Console.WriteLine("Asteroid hits an Asteroid");
        }

        static void CollideWithImpl(Asteroid x, Spaceship y)
        {
            Console.WriteLine("Asteroid hits a Spaceship");
        }

        static void CollideWithImpl(Spaceship x, Asteroid y)
        {
            Console.WriteLine("Spaceship hits an Asteroid");
        }

        static void CollideWithImpl(Spaceship x, Spaceship y)
        {
            Console.WriteLine("Spaceship hits a Spaceship");
        }

        static void CollideWith(Thing x, Thing y)
        {
            dynamic a = x;
            dynamic b = y;
            CollideWithImpl(a, b);
        }

        static void Main(string[] args)
        {
            var asteroid = new Asteroid();
            var spaceship = new Spaceship();
            CollideWith(asteroid, spaceship);
            CollideWith(spaceship, spaceship);
        }
    }
}

 

]]> Implementing the Visitor pattern with the 'dynamic' feature of C# 4.0 https://docs.microsoft.com/archive/blogs/laurionb/implementing-the-visitor-pattern-with-the-dynamic-feature-of-c-4-0 Thu, 28 May 2009 19:05:00 GMT https://blogs.msdn.microsoft.com/laurionb/2009/05/28/implementing-the-visitor-pattern-with-the-dynamic-feature-of-c-4-0/ A lot of the talk around the new dynamic keyword in C# 4.0 (available in the beta of Visual Studio 2010) is around interaction with external components (dynamic languages, HTML DOM, COM) but I'm most excited about being able to stragetically add dynamic lookup to an ordinary C# program. Normally, when an object is passed to a method the method lookup depends on the compile-time type of the object. This means that code like this:

class MyClass
{
 public void Foo(Stream s) { Console.WriteLine("Stream"); }
 public void Foo(MemoryStream s) { Console.WriteLine("MemoryStream"); }
}

class Program
{
 static void DoSomething(Stream s)
 {
  MyClass obj = new MyClass();
  obj.Foo(s);
 }

 static void Main(string[] args)
 {
  DoSomething(new MemoryStream());

 }

Will print out 'Stream', not 'MemoryStream'. Method resolution for dynamic objects has to use the runtime type of the object, so simply by changing DoSomething() to take a dynamic instead of a Stream we get a program that prints out 'MemoryStream'. Basically, dynamic lets us implement multi-methods in C#! (Which I really do find very exciting).

A good use of this feature is when implementing the Visitor pattern. C# code that visits all the nodes in a Linq expression tree tends to be quite ugly, with a huge switch statement and a lot of casts to traverse the tree. By casting the Expression class to dynamic we can call a Visit() method which depends on the exact subclass of Expression. The code below prints an expression tree without needing any explicit conditional or loop statements (using dynamic makes the runtime do all the boring lookup work for us). It is also quite a bit shorter than the C# 3.0 equivalent. Note that if there isn't an implementation for a specific expression type then the Visit method for the parent class will be called.

using System;
using System.CodeDom.Compiler;
using System.IO;
using System.Linq.Expressions;

namespace MultiMethods
{
    class ExpressionPrinter
    {
        private readonly IndentedTextWriter output;

        public ExpressionPrinter(TextWriter output)
        {
            this.output = new IndentedTextWriter(output);
        }

        public void Print(Expression exp)
        {
            dynamic d = exp;
            this.Visit(d);
        }

        private void Visit(Expression exp)
        {
            output.WriteLine("Expression");
        }

        private void Visit(BinaryExpression exp)
        {
            output.WriteLine("BinaryExpression: {0}", exp.NodeType);
            this.VisitSubExpression(exp.Left);
            this.VisitSubExpression(exp.Right);
        }

        private void Visit(ConditionalExpression exp)
        {
            output.WriteLine("ConditionalExpression");
            this.VisitSubExpression(exp.Test);
            this.VisitSubExpression(exp.IfTrue);
            this.VisitSubExpression(exp.IfFalse);
        }

        private void Visit(ConstantExpression exp)
        {
            output.WriteLine("ConstantExpression: {0}", exp.Value);
        }

        private void Visit(MethodCallExpression exp)
        {
            output.WriteLine("MethodCall: {0}", exp.Method.Name);
        }

        private void Visit(ParameterExpression exp)
        {
            output.WriteLine("ParameterExpression: {0}", exp.Name);
        }

        public void Visit(UnaryExpression exp)
        {
            output.WriteLine("UnaryExpression: {0}", exp.NodeType);
            this.VisitSubExpression(exp.Operand);
        }

        private void VisitSubExpression(Expression exp)
        {
            output.Indent++;
            dynamic d = exp;
            this.Visit(d);
            output.Indent--;
        }
    }

    class Program
    {
        static void Main(string[] args)
        {
            var p = new ExpressionPrinter(Console.Out);
            Expression<Func<int, int>> e = i => (0 == i%2) ? -i * i : Math.Abs(i);
            p.Print(e.Body);
        }
    }
}

The output is:

ConditionalExpression
    BinaryExpression: Equal
        ConstantExpression: 0
        BinaryExpression: Modulo
            ParameterExpression: i
            ConstantExpression: 2
    BinaryExpression: Multiply
        UnaryExpression: Negate
            ParameterExpression: i
        ParameterExpression: i
    MethodCall: Abs

]]> Kobe - the Web 2.0 application that wasn't https://docs.microsoft.com/archive/blogs/laurionb/kobe-the-web-2-0-application-that-wasnt Fri, 17 Apr 2009 15:22:00 GMT https://blogs.msdn.microsoft.com/laurionb/2009/04/17/kobe-the-web-2-0-application-that-wasnt/ Microsoft has released Kobe, "a getting started resource kit for planning, architecting, and implementing Web 2.0 applications and services using the Microsoft Platform" and a lot of the response has been quite critical (Ayende, Karl Seguin). While the code, architecture and lack of tests have all been thoroughly criticized I think there is an even higher-level point that has been missed -- Kobe, a Web 2.0 sample, was released and managed in a very Web 1.0 way.

Although the Kobe Project Overview video talks about a Web 2.0 application as having an online community and fostering "active and social user engagement" Kobe itself is available though an inactive, non-social MSDN page with no hint of community. The response on the other hand was far more Web 2.0, with people discussing the project through blogs and on Twitter. At this point the web page is updated to say "This sample application is being updated to incorporate community feedback. An updated version will be released shortly.", but that level of interaction is too little, too late.

I think the biggest mistake here is not releasing a bad Web 2.0 sample, but being unprepared to engage with the online community discussing your bad Web 2.0 sample. Authentic participation on twitter, informative blogging and frequent updates would go a long way here.

]]> iiobo C++ ESE library https://docs.microsoft.com/archive/blogs/laurionb/iiobo-c-ese-library Mon, 06 Apr 2009 23:54:00 GMT https://blogs.msdn.microsoft.com/laurionb/2009/04/06/iiobo-c-ese-library/ (I saw a link to this from the Wikipedia entry on esent)

A company called iiobo has released a C++ toolkit for using esent (I'm guessing they developed it for internal use). I haven't looked at the code, but their website says that they have:

"C++ classes that provide a higher level abstraction of Extensible Storage Engine e.g DbCursor, DbTable, DbSchema etc. These well designed abstractions allow you to seamlessly work with transactions and other database considerations inside object-oriented C++ code."

Their code is released under the BSD license.

https://www.iiobo.com/ExtensibleStorageEngine.aspx

]]> Version 1.2 of ManagedEsent available on CodePlex https://docs.microsoft.com/archive/blogs/laurionb/version-1-2-of-managedesent-available-on-codeplex Mon, 06 Apr 2009 23:07:00 GMT https://blogs.msdn.microsoft.com/laurionb/2009/04/06/version-1-2-of-managedesent-available-on-codeplex/ Version 1.2 of the ManagedEsent interop layer for the ESENT database engine is now available on CodePlex. The three most useful additions here are:

  • JetEscrowUpdate
  • JetIntersectIndexes
  • Helper functions to set/retrieve columns as DateTime

JetEscrowUpdate works like Interlocked.Add, allowing a column to be updated in an atomic fashion by mutliple sessions without conflicts. There are several restrictions -- the column must have a default value, has to be JET_coltypLong and cannot be indexed.

JetIntersectIndexes takes a set of  index ranges and returns the records which are found in all the index ranges. This can be used to select records that meet multiple criteria. The StockSample.cs sample application shows how to use JetIntersectIndexes.

The release is up on Codeplex.

The source code now contains the source code for Pixie which is a 'work in progress' managed object layer for ESENT. It is nowhere near complete and will be changing a lot over the coming months.

 

]]>