1. The C++ part has memory leak concerning the overwritten entries.
2. The rand()*rand() is a bad idea. Also, using different random generators(C routine vs. C# Random class) does not produce precisely aligned semantics.
3. There are overheads invoking the random generators.
4. For the C++ part, people demand better coding style :-)
5. Plot the memory consumption chart with an origin.
6. For better performance, call reserve() on the unordered_map.
7. For better performance, embed content buffer into the cell entries.
8. Why does the C# program only allocate buffer once outside the loop?

Let me first explain why some of these proposed changes to the test are ruled out. For reserve(), it does improve the performance on the C++ side. But please note that I did not make any optimizations on the C# side either. And for the purpose of a key-value store, there should not be pre-assumed space reservations hard coded into the program, because generally if you are going to put it online, instead of doing planned experiments, you cannot know the usage patterns in advance. For similar reasons, we cannot embed fixed-length std::array into cell entries, because the value blobs stored into a key-value store may vary in size. Plus, variable length entries do not fit into a map, or we will end up storing references to these content objects, which, logically does not make a difference than a void*. The program only allocate a single C# byte array, because the Graph Engine K/V store has its own memory management subsystem, which is not allocating from the CLR managed heap.

#include "stdafx.h"
#include <random>
#include <functional>
#include <unordered_map>
#include <algorithm>
#include <cstdint>
#include <cstdio>

#define ENTRY_COUNT        (64<<20)
#define ENTRY_MAX_ID       (ENTRY_COUNT * 4)
#define ENTRY_MAX_SIZE     (314 * 2)
#define BUCKET_COUNT       (64)
#define BUCKET_GRANULARITY (ENTRY_MAX_SIZE / BUCKET_COUNT)

struct param_entry
{
    int64_t cell_id;
    size_t cell_size;
};

int main(int argc, char* argv[])
{
    std::default_random_engine             rnd_gen;
    std::uniform_int_distribution<int64_t> cell_id_dist(0, ENTRY_MAX_ID);
    std::uniform_int_distribution<int32_t> cell_size_dist(1, ENTRY_MAX_SIZE);
    std::vector<param_entry>               param_entries(ENTRY_COUNT);
    std::unordered_map<int32_t, int32_t>   size_buckets;
    auto cell_id_gen   = std::bind(cell_id_dist, rnd_gen);
    auto cell_size_gen = std::bind(cell_size_dist, rnd_gen);

    for (auto& param : param_entries)
    {
        param.cell_id   = cell_id_gen();
        param.cell_size = cell_size_gen();
    }

    FILE* fp;
    fopen_s(&fp, "params.dat", "wb");
    fwrite(param_entries.data(), sizeof(param_entry), ENTRY_COUNT, fp);
    fclose(fp);

    /* statictical analysis */

    for (int i = 0; i < BUCKET_COUNT; ++i)
    {
        size_buckets[i] = 0;
    }

    for (auto& param : param_entries)
    {
        int bucket_idx = param.cell_size / BUCKET_GRANULARITY;
        ++size_buckets[bucket_idx];
    }

    std::sort(param_entries.begin(), param_entries.end(), [](param_entry &a, param_entry &b) {return a.cell_id < b.cell_id; });
    auto new_end               = std::unique(param_entries.begin(), param_entries.end() , [](param_entry &a, param_entry &b){ return a.cell_id == b.cell_id; });
    int32_t unique_entry_count = new_end - param_entries.begin();

    printf(">>>");
    printf("%d total entries\n", ENTRY_COUNT);
    printf(">>>");
    printf("%d unique entries\n", unique_entry_count);
    printf(">>>");
    printf("Size histogram:\n");
    for (int i = 0; i < BUCKET_COUNT; ++i)
    {
        printf("%-2d\t\t%d\n", i * BUCKET_GRANULARITY, size_buckets[i]);
    }

    return 0;
}
 

The resulting data file params.dat takes exactly 1GB of space, with the following statistics information:

>>>67108864 total entries
>>>59378342 unique entries
>>>Size histogram:
0               855411
9               962141
18              960462
27              961976
36              961641
45              961725
54              963224
63              961521
72              961336
81              961785
90              961053
99              964048
108             962261
117             961443
126             960895
135             962208
144             960792
153             960925
162             961486
171             961390
180             961762
189             961682
198             962320
207             962201
216             960415
225             962173
234             963833
243             960419
252             960049
261             963305
270             959831
279             959656
288             961434
297             961652
306             964231
315             963157
324             961786
333             960566
342             961645
351             961816
360             961421
369             961435
378             962095
387             960651
396             962341
405             961820
414             961355
423             961368
432             960645
441             961942
450             960869
459             962132
468             962618
477             960927
486             962067
495             960434
504             962018
513             962531
522             962315
531             961781
540             962139
549             961675
558             963721
567             961252
 

Uniform enough. I limited the range of cell ids to [0, 256M] to introduce some duplicated entries. And the test code, updated, here:

#include "stdafx.h"
#include <vector>
#include <iostream>
#include <unordered_map>
#include <random>
#include <functional>
#include <chrono>
#include <cstdint>
#include <ctime>
using namespace std;

#define ENTRY_COUNT (1<<20)*64

typedef int64_t cellid_t;

struct CellEntry
{
    void*    ptr;
    int32_t  len;
    uint16_t cellType;
};

struct param_entry
{
    int64_t cell_id;
    size_t cell_size;
};

param_entry                        param_entries[ENTRY_COUNT];
unordered_map<cellid_t, CellEntry> LocalStorage;

int main(int argc, char** argv)
{
    srand(time(NULL));
    std::chrono::steady_clock::time_point start_time, end_time;
    std::cout << "Preparing keys" << std::endl;

    FILE* fp;
    fopen_s(&fp, "params.dat", "rb");
    fread(param_entries, sizeof(param_entry), ENTRY_COUNT, fp);
    fclose(fp);

    std::cout << "Test started." << std::endl;
    start_time = std::chrono::steady_clock::now();

    for (auto entry : param_entries)
    {
        void* cell_buf      = new char[entry.cell_size];
        auto  upsert_result = LocalStorage.emplace(entry.cell_id, CellEntry{ cell_buf, entry.cell_size, 0 });

        if (!upsert_result.second)
        {
            std::swap(cell_buf, upsert_result.first->second.ptr);
            delete[] cell_buf;
        }

        if (rand() % 3 == 0)
        {
            delete[] upsert_result.first->second.ptr;
            LocalStorage.erase(entry.cell_id);
        }
    }

    end_time = std::chrono::steady_clock::now();

    std::cout << "done." << std::endl;
    std::cout << "Time = " << chrono::duration_cast<chrono::seconds>(end_time - start_time).count() << std::endl;

    getchar();

    return 0;
}
 

C#:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading.Tasks;
using Trinity;
using Trinity.Core.Lib;
namespace MemoryConsumptionTest_CSharp
{
    unsafe internal class Program
    {
        const int  ENTRY_COUNT    = (64 << 20);
        const int  ENTRY_MAX_SIZE = (314 * 2);
        [StructLayout(LayoutKind.Explicit)]
        struct param_entry
        {
            [FieldOffset(0)]
            public long cell_id;
            [FieldOffset(8)]
            public int cell_size;
            [FieldOffset(12)]
            private int __pad;
        };
        [DllImport("msvcrt.dll")]
        static extern void* fopen(string fname, string mode);
        [DllImport("msvcrt.dll")]
        static extern long fread(void* ptr, long size, long count, void* fp);
        [DllImport("msvcrt.dll")]
        static extern void fclose(void* fp);
        static void Main(string[] args)
        {
            // SaveCell will alloc memory inside GE,
            // so we just allocate a single buffer from C#
            byte* buf                        = (byte*)Memory.malloc(ENTRY_MAX_SIZE);
            param_entry* entries             = (param_entry*)Memory.malloc((ulong)(ENTRY_COUNT * sizeof(param_entry)));
            TrinityConfig.CurrentRunningMode = RunningMode.Embedded;
            Stopwatch stopwach               = new Stopwatch();
            Random dice                      = new Random();

            Console.WriteLine("Preparing keys");

            var fp = fopen("params.dat", "rb");
            fread(entries, sizeof(param_entry), ENTRY_COUNT, fp);
            fclose(fp);

            Console.WriteLine("Test started.");
            stopwach.Start();

            for(param_entry* p = entries, e = entries + ENTRY_COUNT; p != e; ++p)
            {
                Global.LocalStorage.SaveCell(p->cell_id, buf, p->cell_size, cellType: 0);

                if (dice.Next(3) == 0)
                {
                    Global.LocalStorage.RemoveCell(p->cell_id);
                }
            }

            stopwach.Stop();
            Console.WriteLine("Done.");
            Console.WriteLine("Time = {0}", stopwach.ElapsedMilliseconds / 1000);

            Console.ReadKey();
        }
    }
}
 

Updated performance result:

Looking forward to your comments!

To use these functions, create a “Graph Engine Application Project” in Visual Studio and add the namespace “Trinity.Core.Lib”. Compared with their P/Invoked counterparts, the Graph Engine wrappers can usually give us a little bit performance edge. Try out the following code snippet:

using System;

using System.Diagnostics;

using System.Runtime.InteropServices;

using System.Text;

using System.Threading.Tasks;

using Trinity;

using Trinity.Core.Lib;

namespace GraphEngineApp1

{

    unsafe internal class Program

    {

        static void Main(string[] args)

        {

            int count = 1000000;

            void* src = Memory.malloc(300);

            void* dst = Memory.malloc(300);

            Stopwatch sw = Stopwatch.StartNew();

            for (int i = 0; i < count; i++)

            {

                Memory.memcpy(dst, src, 300);

            }

            sw.Stop();

            Console.WriteLine(sw.ElapsedMilliseconds);

            Memory.free(src);

            Memory.free(dst);

        }

    }

}

Now replace the “Memory.memcpy(dst, src, 300) ” with the P/Invoked version, and run again. On my machine, the GE wrapper version takes 14 ms, while the P/Invoked version takes 25 ms.

Use P/Invoke if its overhead is negligible for your application. Consider giving the GE wrappers a shot if the overhead caused by P/Invoke starts to bother you.  

Well, virtually they do the same thing after all: to store key-value pairs. So why bother using Graph Engine against the other? I'm not going to talk about remote access, fault tolerance and other fancy stuff for now... Let's take a look at a much easier, but critical factor: memory consumption.

Data structures

First let's calibrate our data structures. For the Graph Engine side, we drop TSL (wait didn't you just write a long post about the importance of TSL?) and leave only the bare metal memory cache. It stores cell IDs (keys), cell types (unsigned short values), cell blob location and cell size. To simulate this, we use the following data structure in C++:

struct CellEntry
{
    void*    ptr;
    int32_t  len;
    uint16_t cellType;
};
 

Initially I picked map<K,V> instead of unordered_map<K,V>. After watching it crawling bytes at 20MB/s for minutes... ^C^C^C^C^C.

And here's the code.

C++:

#include <vector>
#include <iostream>
#include <unordered_map>
#include <cstdint>
#include <ctime>
using namespace std;

#define ENTRY_SIZE 314 
#define ENTRY_COUNT (1<<20)*64

struct CellEntry
{
    void*    ptr;
    int32_t  len;
    uint16_t cellType;
};
unordered_map<long, CellEntry> LocalStorage;

int _tmain(int argc, _TCHAR* argv[])
{
    srand(time(NULL));

    for (int i = 0; i < ENTRY_COUNT; ++i)
    {
        long id = rand() * rand();
        void* p = new char[ENTRY_SIZE];
        LocalStorage[id] = { p, ENTRY_SIZE, /*cellType:*/0 };
        if (rand() % 3 == 0)
        {
            LocalStorage.erase(id);
            delete[] p;
        }
    }
    return 0;
}
 

C# w/ Graph Engine:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Trinity;
namespace MemoryConsumptionTest_CSharp
{
    internal class Program
    {
        const int ENTRY_SIZE = 314;
        const int ENTRY_COUNT = (1 << 20) * 64;
        static void Main(string[] args)
        {
            Random r = new Random();
            byte[] buf = new byte[ENTRY_SIZE];
            TrinityConfig.CurrentRunningMode = RunningMode.Embedded;

            for (int i = 0; i < ENTRY_COUNT; ++i)
            {
                long id = r.Next() * r.Next();
                Global.LocalStorage.SaveCell(id, buf, 0, ENTRY_SIZE, cellType: 0);

                if(r.Next(3) == 0)
                {
                    Global.LocalStorage.RemoveCell(id);
                }
            }
        }
    }
}
 

We store cells with a fixed length 314, 64 millions of them. Both C++ and C# will assure that the allocated memory gets committed unlike the malloc() function, which would need a memset() to trigger that. Also, along cell insertions, we randomly remove entries with the probability 1/3... This is to test their behaviors with regard to the memory fragmentations introduced by removals.

If the environment is correctly set up, the two programs should have the same semantics (remember to compile 64-bit programs, and make sure your machine can handle that ~20GB of memory allocation). The following performance data is acquired on a server with Xeon X5650@2.67Ghz and 96GB of RAM:

The "Raw performance data" row shows the screen shots of the task manager memory window during the two runs. Notice that for GE, there's a slight memory usage drop (only about a hundred of megabytes). This is due to the garbage collector running in the background collecting the space taken by the removed entries. I will go into the details of our garbage collector in another post.

As we can see, Graph Engine outperforms the unordered_map in both memory consumption (nearly 2GB less space) and speed (nearly 20 seconds faster).
And we're talking about the comparison between a full-fledged key-value store and a data structure...

1. Is GE essentially a hosted version of Titan or neo4j?
2. Can I process non-graph data with GE?

For question one: "no" for "hosted". We do offer the ability to deploy your GE application to the cloud. But there is nothing stopping you to deploy and run it locally. By downloading the SDK from the Visual Studio gallery you have already acquired everything you needed to gear up your local environment, be it standalone or distributed. (Please check out https://www.graphengine.io/docs/manual/Utilities/geconfig.html, we have provided tools to configure a local GE cluster) Still not satisfied? Please move on to:

For question two: Yes, because you can use the underlying key-value store and the computation engine without introducing the graph model! No hacks or tweaks to the system! Indeed, the architecture of GE is illustrated as follows:

However, notice the inherent message: the Graph Model layer is defined with TSL(Trinity Specification Language), so that if you define something non-graph with TSL, your system will become like:

(Oh look, that makes a good millennium falcon.)

The key point here is about the TSL, of which the elements shape what GE really is. It defines both the storage schemata and the communication protocols, without which the GE would fall back to a basic memory cache. In other words, it defines how GE should understand your data. (Of course, if you do not want GE to understand the data, for example, when the data chunks are encrypted, the bare-metal distributed memory cache that GE offers is still a valid option). Let me go further and explain how to turn the ship^H^H^H^H GE into your very-own enhanced key-value store.

Schema matters.

There are quite a few options when we talk about the storage organization. For example, some stores allow you to save JSON document, others let you store primitive data structures, etc. These stores do not enforce a strong schema of the data. However, in many practical cases, the data is intrinsically structural. When you have a stream of many entries, it is likely that they will share some properties, which allows us to make an generalization of them. Also, to process the entries, we have to implement logics. The most obvious case is that, for statically-typed languages, people write data-access-objects(DAOs) to establish a connection between entries and runtime objects. Even if your language of choice is dynamically typed, despite not having to explicitly define the data types, logics do not run away -- your code is still data-driven, or you would end up ignoring the non-generalized part of the data. Hopefully the data does not get so complex that one has to leverage machine learning algorithms only for type inference.
Next, let's look at the physical storage layout when a data stream hits the storage, which is schema-free, and natively stores JSON objects. Suppose we have the following incoming json object(Taken from Twitter Developers https://dev.twitter.com/rest/reference/get/search/tweets):

{ 
    "statuses": [ 
    { 
        "coordinates": null,
        "favorited": false, 
        "truncated": false, 
        "created_at": "Mon Sep 24 03:35:21 +0000 2012",
        "id_str": "250075927172759552",
        /* Lots of other properties that fill up 5 blog pages... */ 
    },
    { 
        "coordinates": null,
        /* And it goes on... for another few pages... */ 
    },
    { 
        "coordinates": null,
        /* And on... */ 
    }, 
    { 
        /* Imaging that there are thousands of this... */  
    }], /* Other stuff... */ 
}
 

We can see that, even in a single JSON object, there are a lot of repetitive patterns. In the given example, the most significant member of the JSON object is "statuses", an array list of nested JSON objects, each complying a fixed schema. However, JSON does not enforce a fixed schema so that a JSON store should always store the field names together with their values, and their field types.

Again and again.

This is a typical example of, when one has a data schema, but suffers from it. Overhead (which becomes so significant that it takes up more than a half of the storage space) to store and overhead to retrieve(where one has to enumerate the fields and check by name one after another to get something). We want our users to benefit from the schemata, not to suffer from it. By taking advantage of the schemata defined in the TSL, GE organizes fields in a data entry in a fixed format so that it is not necessary to store the name and data type of each field. Also, during a field retrieval, GE does not have to enumerate all the fields, as it can calculate the offset of the target field according to the predefined data schemata.

Tables? Collections? Types.

In a relational database, we group data records of the same type into a table. When we want to find a relation between different types of objects, we join. What options do we have if we leave the relational world and dive into NOSQL? In the following example, I will introduce two types of objects: kung-fu master and apprentice. Let's assume that each character has a name, and additionally, each apprentice can identify his/her master. The sample data set is presented in the following table:

ID    Type         MasterID    Name
 1    Master              -    Shifu
 2    Apprentice          1    Panda
 3    Apprentice          1    Tiger
 4    Apprentice          1    Snake
 5    Apprentice          1    Crane
 6    Apprentice          1    Monkey
 7    Apprentice          1    Mantis
 8    Master              -    Oogway
 9    Apprentice          8    Fenghuang
10    Apprentice          8    Monkey
11    Apprentice          8    Junjie
12    Apprentice          8    Chao
13    Apprentice          8    Taotie
14    Apprentice          8    Wushen
15    Apprentice          8    Thundering Rhino
16    Apprentice          8    Storming Ox
17    Apprentice          8    Golden Takin
 

With an RDBMS, we can create two tables for master and apprentice respectively, which is a straightforward solution. Each table has its own schema and we can omit the type of the character. Nothing much to worry about.

With a document store, the most intuitive way, if you are coming from a SQL background, is to create two collections and regard each of them as a table in an RDBMS. However, most document stores do not provide cross-collection data access patterns and manual implementation (through map-reduce or client-side logic) of that would have high cost. Instead of thinking about tables, now we have to break the wall and put everything in a single bag. We can store the type of the character as a string to identify them. However, given the absence of "data types", it is unlikely that a document store would provide a facility to focus on a single type of objects, and we have to apply our own filtering logic and use string matching to achieve type discrimination. Not to mention joins, which would probably involve two layers of that.

With a raw key-value store, the situation is worsen, as the ability to handle hierarchical data would be lost, and we have to "flatten" our data (a type of serialization, in some sense) so that each object fits in a single value slot(or we lost atomicity). Be it either a mystical placement logic over a weakly-typed array/dictionary or just plain object-string (de)serialization, the form of the data will drift away from its logical shape. One would end up turning both data and logic into strings with a lot of manual parsing and string concatenations.

What about Graph Engine? As we already know, GE is more than a key-value store with fixed data types. It allows you to define your own data structures, so storing hierarchical data is natively supported, with data access objects(DAOs) automatically generated by the TSL compiler. On the other hand, GE is not a fully-fledged document store so you cannot throw in arbitrary JSON data, which does fit very well in some particular scenarios, and is especially useful when, for example, you have a recursively-defined document and you encode a whole tree into a document(like XML or HTML). So, we can see GE as a storage lying between a primitive key value store and a document store, which is why I call it an "enhanced key-value store" :-) Here is an example of how to model our sample data set:

Cell Master
{
    String Name;
}

Cell Apprentice
{ 
    CellId MyMaster;
    String Name;
}
 

Note that we still have to throw all the data into a single storage and there is no concept like the relational table. However, with data types defined in your schema, we provide type selectors on the storage, acting effectively as tables. Enumerating a type selector will return all the records of that type, plus that you can conduct LINQ queries over them.

From the schema we can see that, our masters are so forgetful that they do not remember their apprentices. Let's see how to help them to get a reunion with the help of a join operation:

static void Main(string[] args)
{
    TrinityConfig.CurrentRunningMode = RunningMode.Embedded;
    /* https://kungfupanda.wikia.com/wiki/Kung_Fu_Panda_Wiki */
    Global.LocalStorage.SaveMaster(1, "Shifu");
    Global.LocalStorage.SaveApprentice(2, 1, "Panda");
    Global.LocalStorage.SaveApprentice(3, 1, "Tiger");
    Global.LocalStorage.SaveApprentice(4, 1, "Snake");
    Global.LocalStorage.SaveApprentice(5, 1, "Crane");
    Global.LocalStorage.SaveApprentice(6, 1, "Monkey");
    Global.LocalStorage.SaveApprentice(7, 1, "Mantis");

    Global.LocalStorage.SaveMaster(8, "Oogway");
    Global.LocalStorage.SaveApprentice(9, 8, "Fenghuang");
    Global.LocalStorage.SaveApprentice(10, 8, "Monkey");
    Global.LocalStorage.SaveApprentice(11, 8, "Junjie");
    Global.LocalStorage.SaveApprentice(12, 8, "Chao");
    Global.LocalStorage.SaveApprentice(13, 8, "Taotie");
    Global.LocalStorage.SaveApprentice(14, 8, "Wushen");
    Global.LocalStorage.SaveApprentice(15, 8, "Thundering Rhino");
    Global.LocalStorage.SaveApprentice(16, 8, "Storming Ox");
    Global.LocalStorage.SaveApprentice(17, 8, "Golden Takin");

    var relations = from master in Global.LocalStorage.Master_Selector()
                    join apprentice in Global.LocalStorage.Apprentice_Selector() 
                    on master.CellID equals apprentice.MyMaster into apprenticeGroup
                    select new { MasterName = master.Name, Apprentices = apprenticeGroup };

    foreach(var relation in relations)
    {
        Console.WriteLine(relation.MasterName);
        foreach(var apprentice in relation.Apprentices)
        {
            Console.WriteLine(">> {0}", apprentice.Name);
        }
    }
}
 

In the example given above, I first store the sample dataset into the local storage. To get a list of all the apprentices for a kung-fu master, we have to join the two types, which is achieved with a LINQ query over Global.LocalStorage.Master_Selector() and Global.LocalStorage.Apprentice_Selector(): just think of these two selectors as two tables in a relational database.

Atomicity, more than the usual ones.

Graph Engine comes with atomicity guaranteed. Operations applied to a cell are serialized, and observed in the same order by all readers. What makes it more useful than the usual atomic semantics is that, a user is not restricted to pre-defined operators like GET and PUT. Instead, we allow arbitrary operations to be applied to a cell, so that tasks like atomic increment of a counter, or atomic insertion into a list, can be easily achieved with simple user code. Also, you are not restricted to performing one operation at a time during an atomicity-guaranteed period. For example, during a single atomic "session", you can perform an atomic increment to a counter, an atomic write to a string, and an atomic insert into a list, at the same time, which would likely require a transaction in other NOSQL stores.

RESTful APIs extend your call stack to the remote side.

 A stock Graph Engine instance does not provide any RESTful APIs. Instead of providing a fixed set of RESTful endpoints, we allow users to roll their own RESTful interfaces declaratively with the help of TSL. For example, you can define protocols to describe GET/PUT interfaces:

cell Entry
{
    string data;
}
struct GETRequest
{
    long entry_id;
}
protocol GET
{
    type:HTTP;
    request:GETRequest;
    response:Entry;
}
struct PUTResponse
{
    string message;
}
protocol PUT
{
    type:HTTP;
    request:Entry;
    response:PUTResponse;
}
server KeyValueStore
{
    protocol GET;
    protocol PUT;
}
 

And define their semantics with simple user code:

internal class KeyValueStore : KeyValueStoreBase
{
    public override void GETHandler(GETRequest request, out Entry response)
    {
        response = Global.LocalStorage.LoadEntry(request.entry_id);
    }

    public override void PUTHandler(Entry request, out PUTResponse response)
    {
        if (Global.LocalStorage.SaveEntry(request))
            response.message = "OK";
        else
            response.message = "FAIL";
    }
}
 

A more comprehensive example, which mimics many of Redis's interfaces can be found here: https://github.com/jiabaohan/SimpleRedis

At the end of this article I'd like to have some extra fun. Note that you can set the request or response type of a RESTful endpoint to "stream" so that you can directly operate on the HTTP stream. I'm going to stream some media through Graph Engine... Like this:

struct StreamingRequest
{
    string filename;    
}

protocol Streaming
{
    Type:HTTP;
    Request:StreamingRequest;
    Response:stream;    
}

server StreamingServer
{
    protocol Streaming; 
}
 

With a simple handler that pumps media data to the web endpoint:

internal class StreamingServer : Trinity.Extension.StreamingServerBase
{
    public override void StreamingHandler(StreamingRequest request, HttpListenerResponse response)
    {
        byte[] content = File.ReadAllBytes(request.filename);
        response.OutputStream.Write(content, 0, content.Length);
    }
}
 

Party on!

We are very pleased to announce that the Graph Engine 1.0 preview has finally been released to the public. Graph Engine, previously known as Trinity, is a distributed, in-memory, large graph processing engine.

Graphs play an indispensable role in a wide range of domains. Graph processing at scale, however, is facing challenges at all levels, ranging from system architectures to programming models. Underpinned by a strongly-typed RAM store and a general computation engine, Graph Engine helps users build both real-time online query processing applications and high-throughput offline analytics systems with ease.

Graph Engine’s RAM store can manage billions of run-time objects. The RAM store provides efficient memory management with high memory utilization ratios. It takes full advantage of the RAM to speed up data accesses and computations in a way that makes every single byte count.

As a versatile general-purpose computation engine, Graph Engine is extremely extensible. It provides a unified declarative language for both data modeling and message passing. Instead of using a fixed data schema and hard-wired computation paradigms, it allows users to freely define data schemata and extend the system capability by performing fine-grained message passing.

Graph Engine is ready to integrate with other system stacks as well. The user-defined APIs can be easily exposed via RESTful interfaces. An application powered by Graph Engine is accessible to any connected device.

Graph Engine has many built-in features for distributed programming, including:

• Declarative data modeling and network programming;
• Full IntelliSense support;
• Language-Integrated Query;
• Remote application deployment, control, monitoring, and debugging.

Graph Engine provides seamless integration with the powerful integrated development environment Visual Studio and Microsoft’s cloud computing platform Azure. It offers developers the best experience in both development and deployment on both the local and cloud sides. Developers can create Graph Engine applications in the all-purpose Visual Studio and deploy the applications to the cloud (Azure) with just a few clicks.

This is our first official public preview. We look forward to feedback and suggestions. We will continue to provide routine maintenance and regular new releases, turning our research ideas into real-life functionality.

To get more information, documents, and downloads, please visit: https://graphengine.io.