1. Venger Index – Balancing Cost and Performance

1. Key Hashing. We use md5 to hash original keys to 128 bit integers, same size with GUID. The number of unique keys are so large that the chance of collision can be ignored. We can also extend the key size from 128b to 192b or 256b, without affecting memory cost or performance of the index.
2. Key Folding: In the index, we don't store the full 128b key, but a hashed partial key of 27 bit. This is cheating. We will explain later about how we get away with it, and why 27.
3. Hash Block Spilling. To further save memory, we spill most of the hash entries from memory to the SSD, and only load them to memory when needed. Unlike memory, the main media is not suited for accessing of small items such as one hash entry. Thus we group hash entries into Hash Blocks.

1.1 Index Architecture

1.2 Adding a new entry:

1. Write the new key-value pair into the data file
2. Locate the correct bucket by hashing the full 128b key.
3. If the first hash block of the bucket is full,
   1. create a new hash page, insert it into the head of the linked list.
   2. Kick off asynchronous spilling of the second page in the list.
4. Generate keys and populate the keys in the first hash block, according to Figure 3

1.3 Querying a Key

1.4 Relocation and Eviction

1.5 Performance vs. Memory Cost

2. Data Files

2.1 Circular Data File

2.2 Multi-Stream Data File

3. Conclusion

LSM Tree Compaction

Thread-per-Request

Events

// Dump data to console
void Output(void* buffer, int length)
{
  char* ptr = reinterpret_cast<char*>(buffer);
  for (int i=0; i< length; i++){
    cout << *(ptr++);
  }
} 

void DumpFiles(vector<string> allFiles)
{
  char buffer[BIG_ENOUGH]; 
  for (auto& filename : allFiles) {
    asyncReadFile(filename, buffer, &Output);
    // Output function will be triggered after file read complete
    // multiple file read may finish around the same time,
    // causing multiple instances of Output to run in parallel
  }
}

C++ Lambda

void DumpFiles(vector<string> allFiles)
{
   char buffer[BIG_ENOUGH]; 
   for (auto& filename : allFiles) {
      asyncReadFile(filename, buffer, 
          [filename&]
          (void* buffer, int length){
              // logic for dumping data here
          });
   }
}

Mala, strung beads

void dumpFiles(vector<string> allFiles, Scheduler& scheduler)
{
   // create a "string" to string all the beads
   auto pActivity = ActivityFactory(scheduler, "Test Activity");

   // context shared by all the beads
   auto pFilesRemain = new int(allFiles.size());
   for (auto& filename : allFiles){
      // open file
      auto f = AsyncFileFactory(filename, scheduler);
      char* readbuffer = new char[f.GetSize()];
      f->Read(0, f.GetSize(), readbuffer, // read entire file to buffer

         lambdaContinue(  // create a bead as call back
            *pActivity,   // all the beads are sequentialized by pActivity
            [pActivity, pFilesRemain, f, readBuffer]
         (WorkerThread&, intptr_t status, uint32_t length)
         {
            if (status != S_OK){
               cerr << "File read error " << f.GetName() << " \n";
            }
            else {
               cout << "==== File Start: " << f.GetName() << "====\n";
               for (int i=0; i < length; i++){
                  cout << readBuffer[i];
               }
               cout << endl;
            }
            delete readBuffer;
            f->Close();
            
            if (--(*pFilesRmain) == 0){
               delete pFilesRemain;
               pActivity->Shutdown();
            }
         })); // end of the bead

   } // end for each file
}