So the question came up about just what the heck is a Code Page. I think it comes out of a long history of Windows support for globalization. It goes back to probably the 1980s, since I remember pages in the appendix of the MS-DOS 5 manual on different code pages.

ASCII is technically 7-bits only. The values 0-127.

ANSI characters are 8-bits, but the meaning of characters 128-255 change depending on which 'code page' you choose. ESE mostly cares about UTF-16 (1200) and Latin (1252). See Wikipaedia for more details.

Code pages are defined on the column itself.

typedef struct
 {
 // ...
 unsigned short cp;
 unsigned short wCollate; /* Must be 0 */
 // ...
 } JET_COLUMNDEF;
 

But then you could say 'Code Page 1200', which meant Unicode (I think UCS-2 was the only form of Unicode initially, and UTF-7, UTF-8, UTF-16, and UTF-32 came later? But that's a different topic...).

And it was realized that you need to know the locale in order to sort Unicode text. First we used LCIDs (locale identifiers), and used them for many years. Then BCP-47 and Locale Names came along. Now, LCIDs are deprecated (starting in Windows Vista), and locale names are preferred. Example of an LCID: 1033 (0x409). The equivalent locale name is "en-US".

So we need to know the *locale* in order to know how to sort a Unicode column. Different locales sort characters in different ways. The LCID in in the JET_UNICODEINDEX structure, which is part of JET_INDEXCREATE/JET_INDEXCREATE2and the Locale Name is in JET_UNICODEINDEX2, which is referenced by JET_INDEXCREATE3. You need Windows 8 in order to use Locale Names. If you need to support Windows 7, you're stuck with LCIDs.

typedef struct tagJET_UNICODEINDEX
 {
 unsigned long lcid;
 unsigned long dwMapFlags;
 } JET_UNICODEINDEX;
 
 #if ( JET_VERSION >= 0x0602 )
 typedef struct tagJET_UNICODEINDEX2
 {
 WCHAR *szLocaleName;
 unsigned long dwMapFlags;
 } JET_UNICODEINDEX2;
 #endif //JET_VERSION >= 0x0602
 

The 'dwMapFlags' member is passed as 'dwMapFlags' to LCMapStringEx.

Note that the code below thinks that it's using Unicode data, because it stores and retrieves it successfully. But the index is completely confused.

 /// <summary>
 /// Repro of Fleischman index range
 /// </summary>
 [TestMethod]
 [Priority(2)]
 [Description("Repro of Fleischman index range.")]
 public void TestFleischmanIndexRange()
 {
 JET_COLUMNID columnidName;
 var columndefName = new JET_COLUMNDEF()
 {
 //// Uncomment this line to make it work properly.
 //// cp = JET_CP.Unicode,
 coltyp = JET_coltyp.LongText,
 grbit = ColumndefGrbit.ColumnNotNULL
 };
 Api.JetAddColumn(this.sesid, this.tableid, "Name", columndefName, null, 0, out columnidName);
 
 JET_COLUMNID columnidStatus;
 var columndefStatus = new JET_COLUMNDEF() { coltyp = JET_coltyp.UnsignedByte, };
 Api.JetAddColumn(this.sesid, this.tableid, "Status", columndefStatus, null, 0, out columnidStatus);
 
 JET_COLUMNID columnidDifferentiator;
 var columndefDifferentiator = new JET_COLUMNDEF()
 {
 cp = JET_CP.Unicode,
 coltyp = JET_coltyp.LongText,
 grbit = ColumndefGrbit.ColumnNotNULL
 };
 Api.JetAddColumn(this.sesid, this.tableid, "Differentiator", columndefDifferentiator, null, 0, out columnidDifferentiator);
 
 string indexDef = "+Name\0+Status\0\0";
 var indexcreate = new JET_INDEXCREATE[]
 {
 new JET_INDEXCREATE()
 {
 szIndexName = "fleischman",
 szKey = indexDef,
 cbKey = indexDef.Length,
 grbit = CreateIndexGrbit.None,
 ulDensity = 100,
 cbKeyMost = 1000,
 cbVarSegMac = 1000,
 pidxUnicode =
 new JET_UNICODEINDEX() { szLocaleName = "en-us", }
 },
 };
 
 Windows8Api.JetCreateIndex4(this.sesid, this.tableid, indexcreate, indexcreate.Length);
 
 var records = new[]
 {
 new { name = "a1", status = 0x2, differentiator = "a1-2" },
 new { name = "a2", status = 0x2, differentiator = "a2-2" },
 new { name = "n1", status = 0x2, differentiator = "n1-2" },
 new { name = "n1", status = 0x2, differentiator = "n1-2_dupe!" },
 new { name = "n2", status = 0x2, differentiator = "n2-2" },
 new { name = "q1", status = 0x2, differentiator = "q1-2" },
 new { name = "q2", status = 0x2, differentiator = "q2-2" },
 };
 
 using (Transaction trx = new Transaction(this.sesid))
 {
 foreach (var record in records)
 {
 using (var update = new Update(this.sesid, this.tableid, JET_prep.Insert))
 {
 Console.WriteLine("Inserting name = '{0}', status = '{1}'", record.name, record.status, record.differentiator);
 Api.SetColumn(this.sesid, this.tableid, columnidName, record.name, Encoding.Unicode);
 Api.SetColumn(this.sesid, this.tableid, columnidStatus, (byte)record.status);
 Api.SetColumn(this.sesid, this.tableid, columnidDifferentiator, record.differentiator, Encoding.Unicode);
 update.Save();
 }
 }
 trx.Commit(CommitTransactionGrbit.LazyFlush);
 }
 
 if (this.NamedKeyTrySeek(this.sesid, this.tableid, "fleischman", "n1", 2))
 {
 do
 {
 string retrievedName = Api.RetrieveColumnAsString(
 this.sesid,
 this.tableid,
 columnidName,
 Encoding.Unicode);
 byte retrievedStatus= Api.RetrieveColumnAsByte(
 this.sesid,
 this.tableid,
 columnidStatus).GetValueOrDefault();
 string retrievedDifferentiator= Api.RetrieveColumnAsString(
 this.sesid,
 this.tableid,
 columnidName,
 Encoding.Unicode);
 
 Console.WriteLine(
 "Retrieved name = '{0}', status = '{1}', differentiator = '{2}'",
 retrievedName,
 retrievedStatus,
 retrievedDifferentiator);
 }
 while (Api.TryMoveNext(this.sesid, this.tableid));
 }
 }
 
 /// <summary>
 /// Courtesy of Fleischman.
 /// </summary>
 /// <param name="sesid"></param>
 /// <param name="tableid"></param>
 /// <param name="indexName"></param>
 /// <param name="name">The first column to seek for.</param>
 /// <param name="status">The second column to seek for.</param>
 /// <returns></returns>
 public bool NamedKeyTrySeek(
 JET_SESID sesid,
 JET_TABLEID tableid,
 string indexName,
 string name,
 byte status)
 {
 Api.JetSetCurrentIndex(sesid, tableid, indexName);
 Api.MakeKey(sesid, tableid, name, Encoding.Unicode, MakeKeyGrbit.NewKey);
 Api.MakeKey(sesid, tableid, status, MakeKeyGrbit.None);
 bool ret = Api.TrySeek(sesid, tableid, SeekGrbit.SeekEQ);
 if (ret)
 {
 Api.MakeKey(sesid, tableid, name, Encoding.Unicode, MakeKeyGrbit.NewKey);
 Api.MakeKey(sesid, tableid, status, MakeKeyGrbit.None);
 ret = Api.TrySetIndexRange(sesid, tableid, SetIndexRangeGrbit.RangeInclusive | SetIndexRangeGrbit.RangeUpperLimit);
 }
 
 return ret;
 }
 

Normally, when we try to access ESE database files, we get ACCESS_DENIED, because the ESE engine opens files exclusively.

VSS (Volume Shadow Copy Services) allows us to grab a Snapshot of the volume, and access the file in a Read-Only mode.

But with a VSS writer, there is a callback that allows us to modify these files in a small window (OnPostSnapshot). We do this in 'esevss.dll', and it allows us to replay the log files on the in-use database, get to a clean state, and then have the Read-Only copy of the database in a clean state. Some examples are below.

This is rather useful for accessing database files without stopping the process. But beware that any transactions that haven't been committed at the time of the snapshot will be rolled back, and will be missing from the snapshotted database. Therefore it's more useful as a diagnostic tool, and not very useful for backups (well, some people may still find it useful to get most of the data...).

Performance implications: Yes, there are implications. VSS snapshots end up increasing the I/O load on the volume. The first write operation for a file turns in to THREE I/Os: (Read the old value; Write the old value in a special Diff area; Write the new value). I won't go in to the gory details here, because it's been around for years.

Normally, we can't access the database:

c:\Users\martinc\AppData\Local\Microsoft\Internet Explorer\Indexed DB $ esentutl -mh AppQuota.edb

Extensible Storage Engine Utilities for Microsoft(R) Windows(R)
Version 10.0
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating FILE DUMP mode...
Error: Access to source database 'AppQuota.edb' failed with Jet error -1032.

Operation terminated with error -1032 (JET_errFileAccessDenied, Cannot access file, the file is locked or in use) after 0.188 seconds.



We can dump an in-use database. Note that it's in a Dirty state. Check out the funky file name -- HarddiskVolumeShadowCopy1.
c:\Users\martinc\AppData\Local\Microsoft\Internet Explorer\Indexed DB $ esentutl -mh AppQuota.edb -vss

Extensible Storage Engine Utilities for Microsoft(R) Windows(R)
Version 10.0
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initializing VSS subsystem...

Initiating FILE DUMP mode...
 Database: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1\Users\martinc\AppData\Local\Microsoft\Internet Explorer\Indexed DB\AppQuota.edb


DATABASE HEADER:
Checksum Information:
Expected Checksum: 0x7ea2e77c
 Actual Checksum: 0x7ea2e77c

Fields:
 File Type: Database
 Checksum: 0x7ea2e77c
 Format ulMagic: 0x89abcdef
 Engine ulMagic: 0x89abcdef
 Format ulVersion: 0x620,20
 Engine ulVersion: 0x620,20
Created ulVersion: 0x620,20
 DB Signature: Create time:11/19/2014 18:38:18.049 Rand:408092127 Computer:
 cbDbPage: 32768
 dbtime: 65844 (0x10134)
 State: Dirty Shutdown
...

Operation completed successfully in 6.282 seconds.

Here's an example using vssrec to get it ta clean state first:
c:\Users\martinc\AppData\Local\Microsoft\Internet Explorer\Indexed DB $ esentutl -mh AppQuota.edb -vssrec edb .

Extensible Storage Engine Utilities for Microsoft(R) Windows(R)
Version 10.0
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initializing VSS subsystem...

Initiating FILE DUMP mode...
 Database: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\Users\martinc\AppData\Local\Microsoft\Internet Explorer\Indexed DB\AppQuota.edb


DATABASE HEADER:
Checksum Information:
Expected Checksum: 0x7fadd805
 Actual Checksum: 0x7fadd805

Fields:
 File Type: Database
 Checksum: 0x7fadd805
 Format ulMagic: 0x89abcdef
 Engine ulMagic: 0x89abcdef
...
 DB Signature: Create time:11/19/2014 18:38:18.049 Rand:408092127 Computer:
 cbDbPage: 32768
 dbtime: 65854 (0x1013e)
 State: Clean Shutdown
...


Operation completed successfully in 6.422 seconds.

Also very useful is the ability to copy the database, in case you want to do multiple operations (e.g. esentutl.exe -ms to dump the space used). This copies the file to c:\tmp\clean.edb.

c:\Users\martinc\AppData\Local\Microsoft\Internet Explorer\Indexed DB [14:24:49.24] +$ esentutl -y AppQuota.edb -d \tmp\clean.edb -vssrec edb .

Extensible Storage Engine Utilities for Microsoft(R) Windows(R)
Version 10.0
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initializing VSS subsystem...

Initiating COPY FILE mode...
 Source File: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy3\Users\martinc\Ap
Destination File: \tmp\clean.edb

 Copy Progress (% complete)

 0 10 20 30 40 50 60 70 80 90 100
 |----|----|----|----|----|----|----|----|----|----|
 ...................................................

 Total bytes read = 0x1c80000 (29884416) (28 MB)
 Total bytes written = 0x1c80000 (29884416) (28 MB)


Operation completed successfully in 8.359 seconds.

Theoretically, `esentutl.exe -y -vss` could work for any file that's locked exclusively.

We hope you find it useful!

-martin

What is a Sparse File?

A sparse file has ‘holes’.

The File System has a special FSCTL called FSCTL_SET_ZERO_DATA. Normally it is used to write a bunch of zeroes to part of a file, but if the file is marked as sparse, and this space gets released to the file system.

But what does the File System do with sparse regions?

Any reads from the file don’t go to disk; instead the FS knows to just return zeroes.

Any writes will need to first allocate a cluster from the file system, and then write to this new block. The new block could be in a completely different part of the disk though -- and the file is now fragmented.

Downsides of Sparse Files

Sparse Files are only supported on NTFS and ReFS; not FAT (and all of the FAT variants).

I do not know of any copy utility (cmd.exe copy, xcopy, robocopy) that preserves the ‘sparseness’ of a file. They all expand the copied file to its full length.

Windows 8.1 and earlier did not do a great job of allocating contiguous space when the sparse regions were re-written. For example, if there was a 10 MB hole in the middle of a file, and the application writes a single 1 MB chunk, the file system often split up that 1 MB in to many 64 kb segments (depending on the fragmented free space of the volume).

Fragmented files hurt sequential access performance, which is very important for fast data access on spinning media (disks). SSDs can also be affected by block size, since it is difficult to know their underlying ‘native’ block size. SSDs may need to do a ‘read-modify-write’ if a program writes to less than the native block size.

Prior to Windows 8.1, memory-mapped files could not be made sparse.

We may get out-of-disk errors in places where we never got them before.

How does ESE use Sparse Files?

ESE manages space hierarchically. We will allocate a large multi-page block of space to be used by a table. We call these blocks of space ‘extents’. The default size of an extent is 16 pages. A table has a main btree, but will also have a btree for each of its secondary indices, and also one for its Long Values (LVs).

An extent starts out being ‘Owned’ by the table, and initially all of the extent is ‘Available’. As the btrees grow and require more space, the pages are allocated from the ‘Available’ part of the extent. Only when there are no more pages in the Available Extent will we ask for another Extent.

When enough records on a page are deleted, the page can be freed, and returned back to the Available Extent.

When an entire extent is available, it can then be freed back to the file system.

Manually Examining Sparse Files

Many utilities don’t know whether a file is Sparse or not. Just doing a ‘dir’ won’t help. But compact.exe can be used. Normally it’s used for examining NTFS-compressed files, but it turns out that the Win32 API GetCompressedFileSize works for sparse files as well.

Let's use an example. Indexed DB is an HTML 5 feature, and Internet Explorer uses ESE for its storage. Here we see that the AppQuota.edb file is 8,454,144 bytes large, and takes up 8,454,144 bytes on disk. It has no space savings whatsoever.

c:\Users\martinc\AppData\Local\Microsoft\Internet Explorer\Indexed DB [19:42:30.44] ++$ compact.exe *edb

 Listing c:\Users\martinc\AppData\Local\Microsoft\Internet Explorer\Indexed DB\
 New files added to this directory will not be compressed.

  8454144 :  8454144 = 1.0 to 1   AppQuota.edb
151060480 : 139788288 = 1.1 to 1 d Internet.edb

Of 2 files within 1 directories
1 are compressed and 1 are not compressed.
159,514,624 total bytes of data are stored in 148,242,432 bytes.
The compression ratio is 1.1 to 1.

Now we can run esentutl.exe to manually walk the space trees and free up all of the space that it can. Use the -z option (Zero, originally meant to write zeroes to the deleted parts of a database) along with -t (Trim).

c:\Users\martinc\AppData\Local\Microsoft\Internet Explorer\Indexed DB [19:43:10.26] $ esentutl -z AppQuota.edb -t

Extensible Storage Engine Utilities for Microsoft(R) Windows(R) Version 6.3
Copyright (C) Microsoft Corporation. All Rights Reserved.

Initiating SECURE mode...
        Database: AppQuota.edb
                              :  AE[    3]:         47-   256 (210);    A    47-   256 (210) 
cpgSparseBefore:   0 cpgSparseAfter: 210 cpgSparseNew: 210

:       0 trimmed (before), 210 trimmed (after), 210 pages newly trimmed.

Warning:
  You MUST delete the logfiles for this database

Note:

  It is recommended that you immediately perform a full backup
  of this database. If you restore a backup made before the
  repair, the database will be rolled back to the state
  it was in at the time of that backup.

Operation completed successfully in 0.313 seconds.

(Ignore the bits about deleting the logfiles – that’s meant for securely zeroing out data, and it’s a reminder that the potentially-sensitive deleted data may still exist in the transaction logfiles.)

Hey look, it even mentions AE -- that’s Available Extents, mentioned above. We just trimmed (freed) 210 pages back to the file system. Compact.exe now says it only takes up 1,572,864 bytes:

c:\Users\martinc\AppData\Local\Microsoft\Internet Explorer\Indexed DB [19:43:30.18] $ compact.exe *edb

 Listing c:\Users\martinc\AppData\Local\Microsoft\Internet Explorer\Indexed DB\
 New files added to this directory will not be compressed.

  8454144 :   1572864 = 5.4 to 1 d AppQuota.edb
151060480 : 139788288 = 1.1 to 1 d Internet.edb
Of 2 files within 1 directories
2 are compressed and 0 are not compressed.
159,514,624 total bytes of data are stored in 141,361,152 bytes.
The compression ratio is 1.1 to 1.

Fsutil.exe is supposed to allow us to dump the allocated bits of a file, but for some reason it doesn’t dump everything. So we’ll try a tool that a friend of mine wrote to dump file fragmentation:

c:\Users\martinc\AppData\Local\Microsoft\Internet Explorer\Indexed DB [19:43:49.38] $ fsutil sparse queryrange AppQuota.edb
Allocated range[1]:Offset: 0x0         Length: 0x180000

c:\Users\martinc\AppData\Local\Microsoft\Internet Explorer\Indexed DB [19:47:35.09] $ diskbash -f AppQuota.edb
VCN:        0, LCN: 4270808, LEN:       16 (   65536 bytes)
VCN:       16, LCN: 19183938, LEN:      368 ( 1507328 bytes)
VCN:      384, LCN:       -1, LEN:     1680 ( 6881280 bytes)
============================================================
                  File: AppQuota.edb
             Fragments: 3, Bytes Allocated: 8454144
Fragment Sizes (bytes): Min= 65536, Max= 6881280, Avg= 2818048

The VCN is ‘Virtual Cluster Number’, and it is relative to the beginning of the file.

The LCN is ‘Logical Cluster Number’, and is relative to the beginning of the disk.

So the first block of the file is VCN #0, and there are 16 of them sequentially on the disk, starting at LCN #4270808. It’s a total of 65536 bytes, meaning that this disk has 65535/16=4096
clusters.

The next block of the file starts at VCN #16, for 368 clusters (about 1.5 MB), at LCN #19183938.

The last block of the file starts at VCN #384 (16+368), for 1680 clusters, and at LCN #-1. This means it isn’t actually on the disk!

Also note that this run of 1680 clusters matches the output above of freeing up 210 pages. Each 32k database page takes up eight 4k clusters, and 210*8=1680.

Programmatically Telling ESE to use Sparse Files

Using the regular C API, JET_paramEnableShrinkDatabase must be set to one of the following:

#define JET_bitShrinkDatabaseOff 0x0
 
#define JET_bitShrinkDatabaseOn 0x1 // Uses the file system's Sparse Files feature to release space in the middle of a file.
 
#define JET_bitShrinkDatabaseRealtime 0x2 // Attempts to reclaim space back to the file system after freeing significant amounts of data (when space is marked as Available to the Root space tree).
 

Normally JetResizeDatabase (http://msdn.microsoft.com/en-us/library/jj835047(v=exchg.10).aspx) will act the same as JetGrowDatabase (http://msdn.microsoft.com/en-us/library/gg294109(v=EXCHG.10).aspx) -- that is, the database will either stay the same size, or grow. In order to get JetResizeDatabase() to use sparse files, use JET_bitShrinkDatabaseOn. Then pass in 0 for the cpgDesired.

JET_bitShrinkDatabaseRealtime affects the runtime behavior of what happens when enough data is deleted in tables to free an extent. ESE will make that extent sparse, but unfortunately there are some other background tasks which may then cause some data to be written in this sparse area. It's not ideal, but it's reality.

With our C# layer (http://codeplex.com/managedesent), the logic is the same, but the names are slightly different. The setup:

InstanceParameters instanceParameters = new InstanceParameters(instance);
 instanceParameters.EnableShrinkDatabase = ShrinkDatabaseGrbit.On;
 

And the actual code:

 int actualPagesAfterTrim = 0;
 Windows8Api.JetResizeDatabase(sesid, dbid, 0, out actualPagesAfterTrim, ResizeDatabaseGrbit.None);
 

But then they were getting JET_errKeyTruncated when setting a value with length 0x204, which is obviously less than 0x20c. But why?

If the column was JET_coltypLongText and sorted as Unicode, the answer would be obvious. I'll back up a bit though:

When ESE creates indices, it normalizes the keys first so that 'memcmp' is able to work on the raw memory. That way, the B-Tree code doesn't have to know what type a particular key is; it just knows raw bytes and lengths:

• For unsigned integers, this is easy. (Although depending on the endian-ness of the platform, the bytes may need to be shuffled around first.)
• For signed integers, they need to be biased first, and then can be sorted as signed numbers. (e.g. for bytes, add 128 to them first, so that instead of [-128 .. 127] we end up storing [0 .. 255], which sort naturally.) When retrieving from the key, this can be trivially reversed. (Useful with RetrieveColumnGrbit.RetrieveFromIndex)
• Date/times are normalized to a floating point number.
• ASCII/ANSI text is all converted to upper case. This is the first of the 'lossy' normalizations, since we don't know what the initial case was.
• Unicode is more complicated. We use the OS's LCMapStringEx function to normalize the text. Because the world is a big place, it has a very hard problem to solve. 'U' and 'u' and 'u-with-umlaut' should all be sorted together (depending on the flags that are passed in). This ends up expanding a string to many times its original length. A 20-character string takes up 40 bytes, and the normalized binary form could be twice or even up to five times that size (200 bytes -- although that's pretty rare).

Now for LongBinary. Internally we break up the LongBinary values in to much smaller chunks, and add a bit of overhead. Every 8 bytes we store another header byte.

For that 0x204-length LongBinary column, we do:

(0x204 + (8-1) ) / 8 = 65. This is the number of 'chunks'. Each of which takes up 9 bytes. That's 585 bytes. And then add another header byte, and that's 586 bytes (0x24a). That's 70 extra bytes on a 516-byte column, or about 14% more.

Summary: You may need a fudge factor when calculating cbKeyMost. You can't just assume it's the same as the column's cbMax.

ManagedEsent is documented on MSDN:

http://msdn.microsoft.com/en-us/library/windows/desktop/dn375980.aspx

And it's part of Windows 8.1. No need to download from CodePlex! Here's what's on my 8.1 machine:

C:\Windows\assembly\NativeImages_v4.0.30319_32\Microsoft.Ic1a2041b#\3862e50725b3d2f5ef48c19f24b17ef3\Microsoft.Isam.Esent.Interop.ni.dll
C:\Windows\assembly\NativeImages_v4.0.30319_32\Microsoft.Ic1a2041b#\3862e50725b3d2f5ef48c19f24b17ef3\Microsoft.Isam.Esent.Interop.ni.dll.aux
C:\Windows\assembly\NativeImages_v4.0.30319_64\Microsoft.Ic1a2041b#\85216777bd911aef15b30bda8b0046b8\Microsoft.Isam.Esent.Interop.ni.dll
C:\Windows\assembly\NativeImages_v4.0.30319_64\Microsoft.Ic1a2041b#\85216777bd911aef15b30bda8b0046b8\Microsoft.Isam.Esent.Interop.ni.dll.aux
C:\Windows\Microsoft.NET\assembly\GAC_MSIL\microsoft.isam.esent.interop\v4.0_6.3.0.0__31bf3856ad364e35\Microsoft.Isam.Esent.Interop.dll
C:\Windows\WinSxS\msil_microsoft.isam.esent.interop_31bf3856ad364e35_6.3.9600.16384_none_b4cbc5488cc56490\Microsoft.Isam.Esent.Interop.dll

Enjoy!

-martin

Intersect is to save disk IO at the cost of CPU time.

The cost of intersect indices, is the cost of iterating over ALL the index ranges (choice #3 below).　 That may not sound like a savings, HOWEVER index entries are stored together, and we try to maintain sequentially … and so iterating over index entries generally speaking is very fast off disk.　 If you walk an index, and retrieve columns you have to get from the real record … you have to jump back and forth between the index and the table, you will end up seeking into the table, and seeking is slow if the cache is not full of that data.　 Likely your cache is full of data … but let me continue on the cold case points though …

Some examples help imagining cold IO cases:

• Base Index Range Scan – Scanning or counting a range of secondary index entries (where you do not have to retrieve the main data) you can walk 100s of thousands or even single digit millions of entries in a second, because the index entry data is often sequentially laid out and a good index range tells us exactly what you’ll want to read (so we preread aggressively often in a few IOs [1] … let’s say 3 IOs for argument… though a range of 241k entries would probably be more than 3 IOs), and the indices often have very dense / smaller entries … so we sometimes even end up being CPU bound here even if the index is cold.
• Base Index Range + Primary Data Scan – Generally on a _cold_ DB, you can walk like 100 entries / sec via a secondary index … b/c while you can retrieve a range of 100s or thousands of index entries in a second (per #1), each seek into the main table is cold, requires a disk operation and random disk operations are typically ~100 IO per sec (IOPS)[2] on HDs (not SSD).　 Such a case is thoroughly random-IO disk bound.
• a. Best Case 3-way Index Range – So if you have 3 index ranges that selects (as you have) 3k records, 240k records, and 71k records … and it turns out only 1 record satisfies all 3 index ranges, then it will only cost say 3 IOs * 3 index ranges + 1 seek IO for the final data of the primary record = 9 random IOs total.　 But how much did the 240k index range save you …
• b. Even the most costly 240k index range probably processes in ~1 second or less (maybe a bit more, not truly sure of the speed of our insert into the sort) … so if you dropped that, and it ended up giving you an extra 150 records to process (over the 1 that you’ll eventually select), you just barely lose out w/ the extra 150 random disk IOPS (1.5 seconds) from dropping that index range.
• c. If dropping the 240k index range gave you an extra 2999 records to process (b/c not sure if this is obvious, but implicitly we would know in this case the 71k index range didn’t add much additional selectivity over the 3k range), then that index range will have been EXTREMELY beneficial as that’d be 2999 extra seeks (an extra 30 seconds of disk IO savings) to determine the 1 true good record.　 Vs. ~1 second to process the super large index range, that’s extremely fast (30x improvement).
• If dropping the 240k index range gave you NO extra records (b/c there is already only 1 record in common between the 3k and the 71k index ranges … sort of the opposite of 3.b.), then the 241k is pure cost … this is of course difficult to predict.

Every time another index range is added it generally takes a few IOs[1] / 30 ms and let’s say 1 sec CPU / 250k entries of range.　 For a small index range of say 100 entries (where IO cost is dominant), if the selectivity of the eventual set of records you have to look up improves by only 3 records per 3 IOs, then you save those costly #2 type seeks that can only be performed at a measly 100 IOPS pace.　 For a large index range, if the selectivity of the eventual set of records improves by 100 records per 250k entries processed (CPU wise) then you save enough costly #2 type disk seeks that you win over the CPU cost.　 Keep in mind also you have 4 cores to process things, but generally speaking only 1 disk … so again there CPU over disk is nicer to the system as a whole.

However, if everything is warm, generally it becomes an "even numbers" game … where more entries is just more processing … it probably would’ve been better to just select the most selective (smallest) index range and process them straight off by applying the left over terms (residual predicate in query parlance) through the primary record … as that’s 2000 index entries + 2000 seeks (but no IO, just CPU)… every index range you add is a cost / penalty of CPU processing you add to your operational time at likely little improvement.　 Which is likely what you’re seeing here … but this is again another thing that is difficult to know a-priori.

[1] I am generally assuming we have generally good indices, like those over dates, auto-inc-like IDs, common terms or common names, etc.　 Keep in mind some really randomized indices, like GUID indices can have much poorer contiguity and thus can cost a lot more IO to traverse a range of them … and that can change the results dramatically.

[2] This is numbers for HDs … SSDs change things significantly, as they can often do 2000+ random IOPS, and don’t experience as much speed differential from sequential IOs.

 I am trying to get the space occupied by each table that we have by using JetGetObjectInfo. The api succeeds but it returns 0 for both number of cRecords and cPages.

JET_OBJECTINFO tableInfo = {0};
 
 hr = ::JetGetObjectInfoA(m_spSession->GetSessionId(), m_dbId, JET_objtypTable, nullptr, strTableName, &tableInfo, sizeof(tableInfo), JET_ObjInfo);
 
 if (FAILED(hr)) return hr;
 

JetGetTableInfo didnt help as well.

However if i open the cursor and do a JetIndexRecordCount I see lots of records present in this table/index.

If i use JetGetIndexInfo on the clustered index I get the space occupied (using JET_IdxInfoSpaceAlloc) i get 80 as result (I beleive this is in pages).

Please let me know if i am missing some options that ought to be passed (grbits in the tableInfo ?).

Thanks,

It's not obvious, but you need to call JetComputeStats() first.

Calculating these numbers can be expensive to calculate on a large table, which is why the default behaviour is to leave them at zero.

Documentation on JetComputeStats() can be found at http://msdn.microsoft.com/en-us/library/windows/desktop/gg269192(v=exchg.10).aspx

.

ESE allows you to insert data in a structured manner, with data types such as integer or strings. Then you need to be able to find the data, which is where indexing comes in. As one co-worker put it '80% of ESE's coolness is in its indexing'.

ESE stores records in a B+tree (http://en.wikipedia.org/wiki/B%2B_tree). This tree is ordered by the Primary Key. You can either use the default order (an internally-incrementing number, so any new records inserted just end up at the end), or you can explicitly define a primary index (sometimes called 'Clustered Index') by specifying JET_bitIndexPrimary to the index definition. This is very beneficial when you have a unique property that you know will be accessed sequentially. (For example sorting by Employee ID or by Social Security Number [US] or Social Insurance Number [Canada] could be useful. But a primary index on employee name wouldn't be good because two people may have the same name.)

It's usually more interesting to be able to sort on properties other than the Primary Index. For example, sorting by employee names can be useful. This can't be a primary index, since full names aren't unique. You may want to create an index over "+LastName\0+FirstName\0\0". And maybe an index over "+Salary\0\0". These would be secondary indices.

But how do secondary indices work? They are also B+trees. Take the following data:

(Obviously 'Salary' is not unique, but 'EmployeeId' is.)

The Primary Index over 'EmployeeId' will have the primary keys of {3, 5, 101, 202}. The contents in that tree will be the records themselves.

The LastName/FirstName index would be: { [ "AGYEMAN\0FREEMA\0", 5 ], [ "GILLAN\0KAREN\0", 101 ], [ "PERTWEE\0JON\0", 3 ], [ "SMITH\0MATT\0", 202 ] }.

The Salary index would be { [ 1000, 3], [2000, 202], [2200, 5], [2200, 101] }.

Some things to note are:
-The contents of the secondary indices are the primary keys.
-By default, ESE converts ANSI text to uppercase for sorting. (Unicode text is treated differently.)
-The Salary index has a couple of duplicate entries. it's not a Unique index, so this is OK. The tie-breaker isthe value of the Primary Key.

If one was to insert a new record (e.g. { 55, "Baker", "Tom", 3000 } ), then a few things happen:
-By inserting in the middle, if there is no space left on the database page, then it may be necessary to split the page in two. Doing this page split may result in some discontiguous data. So specifying an index density of less than 100 when creating the index can avoid splits. (Data contiguity in general should be another blog topic...)
-It also requires updating the two secondary indices (including uniqueness checks). The more secondary indices you have, the more expensive this will be.

The uniqueness check is done at JetUpdate() time, not JetSetColumn/JetSetColumns.

Conclusion:

That just scratches the surface of indexing. Secondary indices are useful, but can also expensive. Use them wisely!

After creating a database with ESENT, you need to close the returned JET_TABLEID.

The original way to create a usable table was to create the table with JetCreateTable, add some columns with repeated calls to JetAddColumn, then index the columns with JetCreateIndex. But in a multi-threaded concurrent program, you wouldn't want two threads to try and do the same thing, so the returned JET_TABLEID from JetCreateTable() is opened with exclusive access. No other JET_SESID can open that table.

Fast-forward to the introduction of JetCreateTableColumnIndex (and later JetCreateTableColumnIndex3, and in Windows 8 JetCreateTableColumnIndex4 [not yet on MSDN]). This API takes a large JET_TABLECREATE structure that holds the information needed for the table, column, and indices. Everything can be done with a single API call, without a need to do a bunch of loops.

But take a closer look at the JET_TABLECREATE structure:

typedef struct tagJET_TABLECREATE {
 unsigned long cbStruct;
 tchar* szTableName;
 tchar* szTemplateTableName;
 unsigned long ulPages;
 unsigned long ulDensity;
 JET_COLUMNCREATE* rgcolumncreate;
 unsigned long cColumns;
 JET_INDEXCREATE* rgindexcreate;
 unsigned long cIndexes;
 JET_GRBIT grbit;
 JET_TABLEID tableid;
 unsigned long cCreated;
 } JET_TABLECREATE;

There are two output variables there: tableid and cCreated. cCreated returns the number of objects returned (e.g. when create a table with 5 columns and 3 indices, cCreated will be 1+5+3). This helps narrow down which table/column/index had a problem if the API failed. tableid is the JET_TABLEID of the newly created table.

We don't like to have multiple implementations of the code lying around, so internally JetCreateTable() will call the same function as JetCreateTableColumnIndex. The tableid was opened exclusively, to be just like the JetCreateTable/JetAddColumn/JetCreateIndex combination, it was also opened exclusively. So now it should be closed with JetCloseTable().

What does this mean to you? If you write a Metro program in C#, you can grab ManagedEsent from CodePlex (http://managedesent.codeplex.com) and you can use your favourite ESENT APIs from from C#, and still pass certification.

The update is currently available only in source form (http://managedesent.codeplex.com/SourceControl/list/changesets). Binaries will be coming.

Don't hesitate to start new discussions or bug reports in the Discussions area (http://managedesent.codeplex.com/discussions).

-martin