Yi Zhang's MSDN Blog

Top secret .NET handles - Part 1 - Dependent handles

Yi Zhang - CLR — Thu, 08 Feb 2018 09:00:50 GMT

.NET publicly has documented 4 kind of handles:

Weak (also called Short Weak) - Don’t keep target object alive and will return null when object is gone. The target will become null when the object enters for finalization.
WeakTrackResurrection (also called Long Weak) - Don’t keep target object alive and will return null when object is gone. It’ll return the object even when the object is being finalized or resurrected.
Normal (also called strong) - keeps target object alive. If you are not careful, you may leak the object.
Pinned - Keeps the target object alive and prevents GC from moving this object around. Useful when you are passing this object to native code, as native code won’t know if GC moved it. Note that using a lot of pinning handles may degrade GC performance. The most common offender is pinned strings/arrays in interop calls.

You can also find them described in GCHandle enumeration. However, besides these 4 types, there are actually more secret internal handle types that are not exposed. In this post I’ll be talking about dependent handle, and why it is totally awesome.

Caching without leaks

Let’s say you want to implement the best cache ever for your customer. Naturally you start with a Dictionary and start adding things into it. So far so good. Except that all of your customers are now leaking. It’s obvious - they are being held by the dictionary! If you are in total control of the lifetime of the Customer object (for example, you have a dispose), it’s straigh-forward to automatically remove it from cache when it disposes. However, this may not always be possible. Another obvious choice is to resort back to finalizers - adding a finalizer to Customer type which removes itself from the cache when it finalizes. It works, but it means you now have potentially lots of objects that are finalizable, and there is only one finalizer thread. There is something called finalizer starvation - meaning finalizer couldn’t catch up with the objects that are finalized. Oops. Now, what if the key is not held alive by the Dictionary, and instead is a WeakReference? But this doens’t really work - you are no longer leaking WeakReference, but you are still leaking CustomerData, and what if CustomerData has a reference back to Customer? You are back to square one. Before we give up, let’s try one more thing - let’s make both key and value to a WeakReference. Now we no longer leak Customer nor CustomerData. However, the cache doesn’t work quite right, some times you’d have null keys, or you have null values. You can keep trying, but the real problem is that we need two simple things:

Key and Value should both be alive, or not alive
If Key is not alive, we should allow key to be collected as if key is held by a weak reference

Now if this were custom data structure, this is easy to implement - just have the key reference the value thorugh a field, and have a WeakReference to the key. Done. But what if we want a general purpose data structure? You can’t exactly add a new field to arbitary T.

DependentHandle to the rescue

Dependent handle is designed to solve these kind of problems - creating a dependencybetween lifetime of two separate objects. Unlike regular handles, a dependent handle has two targets - primary and secondary. It has the same effect as a field reference - GC will scan primary and secondary, and will keep secondary alive if primary is alive. If you are familiar with GC in a high-level (there aren’t many true GC experts out there and I’m not one of them), you’ll know that .NET GC scans live objects by tracing through object references, starting from roots - that is static variables, thread locals, stack variables, and strong (normal) handles, and going object to object, field by field. Dependent handle are kinda special by itself - GC will scan a list of dependent handle, and will mark secondary (making it alive, surviving this GC) alive if primary is alive. Note that GC may have to scan the dependent handle list multiple times. If C -> A, A -> D, B -> C, the first scan may skip C -> A but would mark C eventually due to B is alive, then it has to come back and mark C -> A, A -> D, basically marking new objects. So GC will keep scanning the dependent handle list until there are no more new objects being marked (alive). Another reason that GC may need to do this is due to mark stack overflow (when GC ran out of stack space during mark). You might think this is not very efficient, and it is not super efficient indeed. In theory you could get rid of this multiple scan, because the problem is you don’t have all the edge information as you go. This is solvable by essentially traverse the list of dependent handles and builds up a graph - thus avoiding traversing the same node twice as you already have all the edges, similar to the regular object graph traversing case (you’d have all the fields that are your edges). However, because the list of dependent handles, and the objects they point to are dynamic, building up this graph can be expensive, and you would be doing this work pretty much every GC (since the graph could easily change). In practice, a complicated graph between dependent handles are not common, so the trade-off works out in our favor. But this implementation for sure may subject to future change. If you are curious you can refer to https://github.com/dotnet/coreclr/blob/release/2.0.0/src/gc/objecthandle.cpp#L1267:

// Scan the dependent handle table promoting any secondary object whose associated primary object is promoted.
//
// Multiple scans may be required since (a) secondary promotions made during one scan could cause the primary
// of another handle to be promoted and (b) the GC may not have marked all promoted objects at the time it
// initially calls us.
//
// Returns true if any promotions resulted from this scan.
bool Ref_ScanDependentHandlesForPromotion(DhContext *pDhContext)
{
    LOG((LF_GC, LL_INFO10000, "Checking liveness of referents of dependent handles in generation %u\n", pDhContext->m_iCondemned));
    uint32_t type = HNDTYPE_DEPENDENT;
    uint32_t flags = (pDhContext->m_pScanContext->concurrent) ? HNDGCF_ASYNC : HNDGCF_NORMAL;
    flags |= HNDGCF_EXTRAINFO;

    // Keep a note of whether we promoted anything over the entire scan (not just the last iteration). We need
    // to return this data since under server GC promotions from this table may cause further promotions in
    // tables handled by other threads.
    bool fAnyPromotions = false;

    // Keep rescanning the table while both the following conditions are true:
    //  1) There's at least primary object left that could have been promoted.
    //  2) We performed at least one secondary promotion (which could have caused a primary promotion) on the
    //     last scan.
    // Note that even once we terminate the GC may call us again (because it has caused more objects to be
    // marked as promoted). But we scan in a loop here anyway because it is cheaper for us to loop than the GC
    // (especially on server GC where each external cycle has to be synchronized between GC worker threads).
    do
    {
        // Assume the conditions for re-scanning are both false initially. The scan callback below
        // (PromoteDependentHandle) will set the relevant flag on the first unpromoted primary it sees or
        // secondary promotion it performs.
        pDhContext->m_fUnpromotedPrimaries = false;
        pDhContext->m_fPromoted = false;

        HandleTableMap *walk = &g_HandleTableMap;
        while (walk) 
        {
            for (uint32_t i = 0; i < INITIAL_HANDLE_TABLE_ARRAY_SIZE; i ++)
            {
                if (walk->pBuckets[i] != NULL)
                {
                    HHANDLETABLE hTable = walk->pBuckets[i]->pTable[getSlotNumber(pDhContext->m_pScanContext)];
                    if (hTable)
                    {
                        HndScanHandlesForGC(hTable,
                                            PromoteDependentHandle,
                                            uintptr_t(pDhContext->m_pScanContext),
                                            uintptr_t(pDhContext->m_pfnPromoteFunction),
                                            &type, 1,
                                            pDhContext->m_iCondemned,
                                            pDhContext->m_iMaxGen,
                                            flags );
                    }
                }
            }
            walk = walk->pNext;
        }

        if (pDhContext->m_fPromoted)
            fAnyPromotions = true;

    } while (pDhContext->m_fUnpromotedPrimaries && pDhContext->m_fPromoted);

    return fAnyPromotions;
}

ConditionalWeakTable

Now you might wonder: if dependent handle is totally awesome but why is it hidden and not used? That’s actually not quite correct. It is true that dependent handles are not directly exposed to developers, they are actually available through the ConditionalWeakTable class. In the future, I’m also planning to write a series of post talking about the implementations of .NET concurrent data structures. A natural first post would be talking about how the ConditionalWeakTable is implemented - how it is using DependentHandle, how it supports thread-safety, etc. You can find its implementation here if you are interested to take a look yourself first. (I must confess that I added the FindEquivalentKeyUnsafe method and I’m not proud of it). Besides ConditionalWeakTable, dependent handle is also used internally in the .NET runtime to report missing references in native code to GC so that GC can resolve cycles between native WinRT objects and managed objects. But that itself is a rather involved topic and deserve its own post.

What’s next

In the next Secret .NET handles post, I’ll talk about my favorite handle type (I’m certainly baised on this one) - a ref counted handle. It’s used in Com Callable Wrappers. For a complete list of handle types, see https://github.com/dotnet/coreclr/blob/release/2.0.0/src/gc/gcinterface.h#L241

Calling C functions from Python - part 3 - deep dive into ctypes implementation in CPython

Yi Zhang - CLR — Fri, 02 Feb 2018 17:00:03 GMT

Last time we’ve looked at using ctypes to call C API, and writing extension module using Python/C API. Now we can finally tie these two together - looking at how ctypes is actually implemented using mix of Python/C API and Python code.

You can find CPython source code here.
ctypes’ C implementation is here
ctypes’ python implementation is here.

Loading libraries

Recall that in ctypes we have cdll, windll, oledll object to help loading libraries. They are really LibraryLoader objects:

>>> print ctypes.cdll
<ctypes.LibraryLoader object at 0x000000000592F470>

And that type is just plain python code:

class LibraryLoader(object):
    def __init__(self, dlltype):
        self._dlltype = dlltype

    def __getattr__(self, name):
        if name[0] == '_':
            raise AttributeError(name)
        dll = self._dlltype(name)
        setattr(self, name, dll)
        return dll

    def __getitem__(self, name):
        return getattr(self, name)

    def LoadLibrary(self, name):
        return self._dlltype(name)

The __getattr__ is the magic that implements attribute-based library loading. Note that if the attribute is already there, CPython returns that attribute immediately without calling __getattr__. Otherwise you would end up with multiple copies of the same attribute, or keeping creating new library objects and discarding old ones - not very efficent. dlltype is the type for each kind of DLL, such as CDLL, PyDll, WinDll, OleDll. __getattr__ creates new instances of these types as needed. cdll, pydll, windll, oledll are simply instances of LibraryLoader class, which are created with corresponding dlltype.

cdll = LibraryLoader(CDLL)
pydll = LibraryLoader(PyDLL)

if _os.name == "nt":
    pythonapi = PyDLL("python dll", None, _sys.dllhandle)
elif _sys.platform == "cygwin":
    pythonapi = PyDLL("libpython%d.%d.dll" % _sys.version_info[:2])
else:
    pythonapi = PyDLL(None)

if _os.name == "nt":
    windll = LibraryLoader(WinDLL)
    oledll = LibraryLoader(OleDLL)

Let’s look at CDLL first - its init does a dlopen to load the library:

class CDLL(object):
    def __init__(self, name, mode=DEFAULT_MODE, handle=None,
        if handle is None:
            self._handle = _dlopen(self._name, mode)
        else:
            self._handle = handle

The attribute access are defined in __getattr__ as well - it gets translated to __getitem__call which creates a new _FuncPtr instance.

    def __getattr__(self, name):
        if name.startswith('__') and name.endswith('__'):
            raise AttributeError(name)
        func = self.__getitem__(name)
        setattr(self, name, func)
        return func
    def __getitem__(self, name_or_ordinal):
        func = self._FuncPtr((name_or_ordinal, self))
        if not isinstance(name_or_ordinal, int):
            func.__name__ = name_or_ordinal
        return func

We’ll look at _FuncPtr later - for now it’s good enough to know it represents function pointer. The difference between OleDll and WinDll is simply the default settings: For CDLL - the base class:

class CDLL(object):
    _func_flags_ = _FUNCFLAG_CDECL
    _func_restype_ = c_int

WinDll has StdCall as default calling convention, and deriving from CDLL:

    class WinDLL(CDLL):
        _func_flags_ = _FUNCFLAG_STDCALL

OleDll is like WinDll (in terms of calling convention), but the default return type is HRESULT.

    class OleDLL(CDLL):
        _func_flags_ = _FUNCFLAG_STDCALL
        _func_restype_ = HRESULT

Calling the function

In last section we discussed the how library are loaded and we didn’t talk about functions yet. Functions are presented as _FuncPtr which is basically a _CFuncPtr in _ctypes module:

        class _FuncPtr(_CFuncPtr):
            _flags_ = flags
            _restype_ = self._func_restype_

Now it’s type to put our Python/C API knowledge to good use - _CFuncPtr is implemented in C:

PyTypeObject PyCFuncPtr_Type = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "_ctypes.PyCFuncPtr",
    sizeof(PyCFuncPtrObject),                           /* tp_basicsize */
    0,                                          /* tp_itemsize */
    (destructor)PyCFuncPtr_dealloc,             /* tp_dealloc */
    0,                                          /* tp_print */
    0,                                          /* tp_getattr */
    0,                                          /* tp_setattr */
    0,                                          /* tp_reserved */
    (reprfunc)PyCFuncPtr_repr,                  /* tp_repr */
    &PyCFuncPtr_as_number,                      /* tp_as_number */
    0,                                          /* tp_as_sequence */
    0,                                          /* tp_as_mapping */
    0,                                          /* tp_hash */
    (ternaryfunc)PyCFuncPtr_call,               /* tp_call */
    0,                                          /* tp_str */
    0,                                          /* tp_getattro */
    0,                                          /* tp_setattro */
    &PyCData_as_buffer,                         /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /* tp_flags */
    "Function Pointer",                         /* tp_doc */
    (traverseproc)PyCFuncPtr_traverse,          /* tp_traverse */
    (inquiry)PyCFuncPtr_clear,                  /* tp_clear */
    0,                                          /* tp_richcompare */
    0,                                          /* tp_weaklistoffset */
    0,                                          /* tp_iter */
    0,                                          /* tp_iternext */
    0,                                          /* tp_methods */
    0,                                          /* tp_members */
    PyCFuncPtr_getsets,                         /* tp_getset */
    0,                                          /* tp_base */
    0,                                          /* tp_dict */
    0,                                          /* tp_descr_get */
    0,                                          /* tp_descr_set */
    0,                                          /* tp_dictoffset */
    0,                                          /* tp_init */
    0,                                          /* tp_alloc */
    PyCFuncPtr_new,                             /* tp_new */
    0,                                          /* tp_free */
}

Let’s look at the tp_new function PyCFuncPtr_new first:

PyCFuncPtr_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    // ...

    if (1 <= PyTuple_GET_SIZE(args) && PyTuple_Check(PyTuple_GET_ITEM(args, 0)))
        return PyCFuncPtr_FromDll(type, args, kwds);

PyCFuncPtr_FromDll has quite a bit of code, but in the end these two lines are the most important:

static PyObject *
PyCFuncPtr_FromDll(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    // ...

#ifdef MS_WIN32
    address = FindAddress(handle, name, (PyObject *)type);
    // ...
#else
    address = (PPROC)ctypes_dlsym(handle, name);
    // ...

In Windows it does a GetProcAddress and in linux/mac it does dlsym. As far as calling function goes, calling the _CFuncPtr effectively calls tp_call field which is PyCFuncPtr_call:

static PyObject *
PyCFuncPtr_call(PyCFuncPtrObject *self, PyObject *inargs, PyObject *kwds)
{
    // ...
    callargs = _build_callargs(self, argtypes,
                               inargs, kwds,
                               &outmask, &inoutmask, &numretvals); 
    // ...
    result = _ctypes_callproc(pProc,
                       callargs,
#ifdef MS_WIN32
                       piunk,
                       self->iid,
#endif
                       dict->flags,
                       converters,
                       restype,
                       checker);
    // ...
    return _build_result(result, callargs,
                         outmask, inoutmask, numretvals);
}

There are a lot of code in the function above, but it basically does 3 steps - preparing the arguments, making the call, and building the result and propagating the arguments back (for out/inout parameters). Eventually it uses ffi_call from FFI to make the call.

    if (FFI_OK != ffi_prep_cif(&cif,
                               cc,
                               argcount,
                               restype,
                               atypes)) {
        PyErr_SetString(PyExc_RuntimeError,
                        "ffi_prep_cif failed");
        return -1;
    }


    ffi_call(&cif, (void *)pProc, resmem, avalues);

FFI itself is quite complicated as it needs to understand all calling conventions and for different CPUs as well (for example, procedure calls in amd64 is drastically different in SPARC) - for now just think of it as a way of being able to say “I want to make a CDecl call to this function using these arguments”, without worrying about all the details in the ABI (Application Binary Interface) level.

Structs and metaclasses

Now that we’ve looked at library loading and function loading/calling, let’s take a look at how structure is implemented. Recall how you write a structure:

class VECTOR3(Structure):
    _fields_ = [("x", c_int), ("y", c_int), ("z", c_int)]

Somehow the VECTOR3 class gets the magic x, y, z attributes. How does this work? The magic is in the PyCStructType metaclass.

Metaclass is a type used to create other types - it is an alternative way of doing subclassing / inheritance in Python, and a very powerful one too. If you understand metaclass you understand Python’s type system. If you are curious, see Primer on metaclasses on a excellent explanation on metaclasses and Understanding Python Metaclasses for a deeper dive. There is also a presentation version as well.

ctypes.Structure is implemented in _ctypes module as Struct_Type, and type of Struct_Type is PyCStructType (PyCStructType_Type object).

    Py_TYPE(&Struct_Type) = &PyCStructType_Type;
    Struct_Type.tp_base = &PyCData_Type;

This makes PyCStructType` a metaclass. Whenever you are deriving from ctypes.Strucuture like following:

class VECTOR3(Structure):
    _fields_ = [("x", c_int), ("y", c_int), ("z", c_int)]

This effectively becomes:

VECTOR3 = PyCStructType('VECTOR3', (Structure), { 'fields' : [("x", c_int), ("y", c_int), ("z", c_int)]
})

Note that tp_new of PyCStructType_Type is PyCStructType_new:

PyTypeObject PyCStructType_Type = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "_ctypes.PyCStructType",                            /* tp_name */
    PyCStructType_setattro,                     /* tp_setattro */
    CDataType_methods,                          /* tp_methods */
    PyCStructType_new,                                  /* tp_new */
};

So this ends up calling PyCStructType_new with those arguments, which retrieves the _fields_ from supplied dictionary, and assign it to _fields_ attribute, triggering PyCStructType_setattro:

static PyObject *
PyCStructType_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    return StructUnionType_new(type, args, kwds, 1);
}

static PyObject *
StructUnionType_new(PyTypeObject *type, PyObject *args, PyObject *kwds, int isStruct)
{
    PyTypeObject *result;
    PyObject *fields;
    StgDictObject *dict;

    result = (PyTypeObject *)PyType_Type.tp_new(type, args, kwds);
    //...
    PyDict_Update((PyObject *)dict, result->tp_dict));
    //...
    fields = PyDict_GetItemString((PyObject *)dict, "_fields_");
    //...
    fields = PyDict_GetItemString((PyObject *)dict, "_fields_");
    //...
    PyObject_SetAttrString((PyObject *)result, "_fields_", fields));

tp_setattro catch the _field_ access (from PyObject_SetAttrString call) and update the internal dictionary on the newly created VECTOR3 type:

static int
PyCStructType_setattro(PyObject *self, PyObject *key, PyObject *value)
{
    /* XXX Should we disallow deleting _fields_? */
    if (-1 == PyType_Type.tp_setattro(self, key, value))
        return -1;

    if (value && PyUnicode_Check(key) &&
        _PyUnicode_EqualToASCIIString(key, "_fields_"))
        return PyCStructUnionType_update_stgdict(self, value, 1);
    return 0;
}

PyCStructUnionType_update_stgdict mostly traverse the list of fields and create necessary PyCField instances as corresponding attributes. Interestingly, the attribute assignment also triggers setattro, which simply let it through as it only cares about _fields_ access (otherwise this would be an infinite loop). When you are accessing myVector3.x, you are setting/getting PyCField instance, which are descriptor classes that binds to the owner class, which is the structure itself.

PyTypeObject PyCField_Type = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "_ctypes.CField",                                   /* tp_name */
    sizeof(CFieldObject),                       /* tp_basicsize */
    (reprfunc)PyCField_repr,                            /* tp_repr */
    "Structure/Union member",                   /* tp_doc */
    (descrgetfunc)PyCField_get,                 /* tp_descr_get */
    (descrsetfunc)PyCField_set,                 /* tp_descr_set */

PyCField_repr provides the nice output you see here:

>>> VECTOR3.x
<Field type=c_long, ofs=0, size=4>

While PyCField_get/PyCField_set provides access to the field on this structure (myVector3.x) through descriptor class and bindings to the structure instance:

static int
PyCField_set(CFieldObject *self, PyObject *inst, PyObject *value)
{
    CDataObject *dst;
    char *ptr;
    if (!CDataObject_Check(inst)) {
        PyErr_SetString(PyExc_TypeError,
                        "not a ctype instance");
        return -1;
    }
    dst = (CDataObject *)inst;
    ptr = dst->b_ptr + self->offset;
    if (value == NULL) {
        PyErr_SetString(PyExc_TypeError,
                        "can't delete attribute");
        return -1;
    }
    return PyCData_set(inst, self->proto, self->setfunc, value,
                     self->index, self->size, ptr);
}

In the above function, self is the PyCField instance, inst is VECTOR3 (or whatever structure you have), and value is the new value you are assigning with. Eventually it got set on the pointer to the structure + field offset, basically *(ptr + offset) = value. But where is that ptr come from? ctypes.Structure are essentially CDataObject*:

// Fields omitted for clarity 
static PyTypeObject Struct_Type = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "_ctypes.Structure",
    sizeof(CDataObject),                        /* tp_basicsize */
    GenericPyCData_new,                         /* tp_new */
};

A CDataObject looks like this:

struct tagCDataObject {
    PyObject_HEAD
    char *b_ptr;                /* pointer to memory block */
    int  b_needsfree;           /* need _we_ free the memory? */
    CDataObject *b_base;        /* pointer to base object or NULL */
    Py_ssize_t b_size;          /* size of memory block in bytes */
    Py_ssize_t b_length;        /* number of references we need */
    Py_ssize_t b_index;         /* index of this object into base's
                               b_object list */
    PyObject *b_objects;        /* dictionary of references we need to keep, or Py_None */
    union value b_value;
};

Just think of it as a generic holder of any value - like VARIANT (if COM is your thing). In particular, b_value field holds the well known simple data values (it’s a union) and b_ptrpoints to the underlying data if it is a more complex type, like structures. GenericPyCData_new is fairly straight-forward - it allocates enough memory as described by the internal stgdict dictionary, which you can treat it as physical layout information about its fields and total size, which is calculated when _fields_ get assigned.

static PyObject *
GenericPyCData_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    CDataObject *obj;
    StgDictObject *dict;

    dict = PyType_stgdict((PyObject *)type);
    if (!dict) {
        PyErr_SetString(PyExc_TypeError,
                        "abstract class");
        return NULL;
    }
    dict->flags |= DICTFLAG_FINAL;

    obj = (CDataObject *)type->tp_alloc(type, 0);
    if (!obj)
        return NULL;

    obj->b_base = NULL;
    obj->b_index = 0;
    obj->b_objects = NULL;
    obj->b_length = dict->length;

    if (-1 == PyCData_MallocBuffer(obj, dict)) {
        Py_DECREF(obj);
        return NULL;
    }
    return (PyObject *)obj;
}

PyCData_MallocBuffer handles two cases - if it is referring to a simple type (like c_int, etc), there is no need to allocate the int dynamically as it fits perfectly well in the b_value union field. Otherwise, it allocates the correct buffer size and assign to b_ptr.

static int PyCData_MallocBuffer(CDataObject *obj, StgDictObject *dict)
{
    if ((size_t)dict->size <= sizeof(obj->b_value)) {
        /* No need to call malloc, can use the default buffer */
        obj->b_ptr = (char *)&obj->b_value;
        /* The b_needsfree flag does not mean that we actually did
           call PyMem_Malloc to allocate the memory block; instead it
           means we are the *owner* of the memory and are responsible
           for freeing resources associated with the memory.  This is
           also the reason that b_needsfree is exposed to Python.
         */
        obj->b_needsfree = 1;
    } else {
        /* In python 2.4, and ctypes 0.9.6, the malloc call took about
           33% of the creation time for c_int().
        */
        obj->b_ptr = (char *)PyMem_Malloc(dict->size);
        if (obj->b_ptr == NULL) {
            PyErr_NoMemory();
            return -1;
        }
        obj->b_needsfree = 1;
        memset(obj->b_ptr, 0, dict->size);
    }
    obj->b_size = dict->size;
    return 0;
}

You might already noticed that the buffer is 0 initialized, and gets freed when it gets finalized. The finalization happens in PyCData_dealloc which does a free if needed.

Next in the series

Originally I was planning to write 3 part series. But then I got interested in PyPy and decided to research into PyPy a bit more. In particular I suspect CFFI might have much better perf (at least in theory) than ctypes with a proper JIT implementation since the arguments “marshaling” can be pretty much “inlined”, but that also requires JIT to be aware of various calling conventions, which is also a pretty daunting task as well (essentially implementing FFI in the JIT). I’ll update them with links once they become available:

Part 1 - CTypes
Part 2 - writing CPython extensions using Python/C API
Part 3 - Deep dive into ctypes implementation in CPython
Part 4 - PyPy and CFFI

You can also find this post in http://yizhang82.me/python-interop-inside-ctypes

Calling C functions from Python - part 2 - writing CPython extensions using Python/C API

Yi Zhang - CLR — Sat, 27 Jan 2018 17:00:58 GMT

In my previous post we’ve briefly looked at using ctypes module to call C APIs. In this post let’s dive into a completely different approach - writing a C extension using Python/C API. From the C extension you can then call whatever C/C++ code you want. This is a pretty popular approach to expose a 3rd party C library as python module. The downside is this module would only be loaded by CPython, which is usually not a big issue - unless you use other Python implementation like PyPy. Imagine if we want to build a better integer library supporting faster integer operations. In this example it won’t be really faster - just a hypothetical example :)

It’s all C code

CPython, as the name suggests, is written in C (mostly). It has a set of data types describes what are python objects, defines a set of operations on python objects, and use those definition + operations to implement core language functionality and internal modules. A subset of those APIs are opened for extension module writers to access the interpreter directly, pretty much like internal modules. Not surprisingly, this is extremely powerful - you have almost the same power as the interpreter itself. Unfortunately, this also means:

the extension module is bound to CPython. In theory another Python implementation could implement the entire CPython API surface area - but that effectively means that you are rewriting a significant part of CPython from scratch with the same interface, implementing it with your own Python implementation - a daunting (if not challenging) task for sure.
CPython always needs to implement those APIs which means that overhauling the interpreter is now more difficult and needs to happen incrementally while maintaining compatibility of the C API.

A module with only static methods

Let’s jump right in. Any Python/C extension module needs to start by including python.h to get all the definitions of Python/C API and types. Don’t worry about details of building yet - we’ll get to that later.

#include

Now we need to define what the static method looks like:

static PyMethodDef FastIntModule_StaticMethods[] = {
    { "add", FastInt_Add, METH_VARARGS, "Add two integers" },
    { NULL, NULL, 0, NULL }
};

PyMethodDef is the C struct type describing python methods. You need to supply:

the name - “add”
the actual C function implementation - FastInt_Add. We’ll fill it in later.
flags - METH_VARARGS meaning it accepts arbitary number of arguments via tuples
doc - “Add two integers”

The last one with all NULLs is a sentinel tell Python to stop looking at more entries, kinda like a \0 terminator. It’s a convention that you’ll need to remember. This way you don’t have to pass size around. Now we need to tell Python that we have a module with these static functions:

PYMODINIT_FUNC initfastint()
{
    Py_InitModule("fastint", FastIntModule_StaticMethods);
}

All python initialization module functions need to be named init - this is how python knows which function to call in your extension module. All it needs to do right now is to register the module with the list of static methods you supplied. With all the setup, now all we need to do is to add the real method:

static PyObject *FastInt_Add(PyObject *self, PyObject *args)
{
    int a, b;
    if (!PyArg_ParseTuple(args, "ii", &a, &b))
    {
        return NULL;
    }

    return Py_BuildValue("i", a + b);
}

There are quite a few things here and can look a bit scary first time. Let’s go through this one thing at a time. First, all Python objects are PyObject*. You can think this as the “base class pointer” as every python object are “derived” from PyObject. The code is written using C so the inheritance is really built by hand through inserting a PyObject field into all the subclasses. We’ll see more of that later. A PyObject is really a ref count + its type.

typedef struct _object {
    _PyObject_HEAD_EXTRA
    Py_ssize_t ob_refcnt;
    struct _typeobject *ob_type;
} PyObject;

Then, in FastInt_Add, it accepts self and rest of the arguments in args as a tuple. PyArg_ParseTuple is used to parse the arguments and break them down into dividual local variables, using a printf/scanf style format - ii means following two pointers are int*. The result is added together, and then returned in the same manner constructing a Python integer object. That’s it!

Buliding the module

In this case I’m using CMake. You can use whatever you want but CMake makes it pretty easy to build it in multiple platforms. First, you need to let CMakfind_package(PythonLibs 2.7 REQUIRED)e find the python package:

find_package(PythonLibs 2.7 REQUIRED)

This sets necessary PYTHON_INCLUDE_DIRS and PYTHON_LIBRARIES variables that you can use later. One interesting problem you might run into is that you need to have a matching bitness of python with your build. For example, if you are building x86 and yet your python is 64-bit, you may see a package lookup failure. Try adding -DCMAKE_GENERATOR_PLATFORM=x64 to building as 64-bit instead.

include_directories(${PYTHON_INCLUDE_DIRS})target_link_libraries(FASTINT_DLL ${PYTHON_LIBRARIES})

To build your code as a python extension module, it obviously needs to be a shared library (DLL/dylib/so), and should have the extension .pyd:

add_library(FASTINT_DLL SHARED main.c)
set_target_properties(FASTINT_DLL PROPERTIES SUFFIX ".pyd")

One interesting problem that I ran into in Windows 64-bit python 2.7 is that Python.h tries to redirect your lib to the debug lib python27_d.lib which doesn’t really exist in python distribution by default unless you build your own debug python. This is done in pyconfig.h:

/* For an MSVC DLL, we can nominate the .lib files used by extensions */
#ifdef MS_COREDLL
#	ifndef Py_BUILD_CORE /* not building the core - must be an ext */
#		if defined(_MSC_VER)
			/* So MSVC users need not specify the .lib file in
			their Makefile (other compilers are generally
			taken care of by distutils.) */
#			ifdef _DEBUG
#				pragma comment(lib,"python27_d.lib")
#			else
#				pragma comment(lib,"python27.lib")
#			endif /* _DEBUG */
#		endif /* _MSC_VER */
#	endif /* Py_BUILD_CORE */
#endif /* MS_COREDLL */

This is unnecessary in my opinion as it should be perfectly OK to build your extension as debug and use Python/C API as release. I tried to use some knobs such as MS_COREDLL but having _DEBUG flag ultimately force Python.h go down the path of linking against debug version of API:

main.obj : error LNK2019: unresolved external symbol Py_InitModule4TraceRefs_64 referenced in function initFastInt

Turns out Py_InitModule4TraceRefs_64 only exists in debug libs. The simpliest thing to do is to stick with release build - and use debug build of Python if you want to build your extension as debug. In my opinion this is not designed properly - _DEBUG should control debug-ness of the module and python itself should use a different one, but this is probably topic for another day.

Trying it out

Once we build the module using cmake, we can finally give this a try:

>>> import fastint
>>> fastint.add(10, 20)
30

If Python refuse to load the module, you might need to tweak syspath, such as:

sys.path.append('path/to/my/module')

Or, load it directly:

fastint = imp.load_dynamic('fastint', r'path/to/my/module/fastint.pyd')

Defining a fastint class

If we want to define a fastint type, it is a bit more involved. Let’s start by defining our own PyObject “subclass”:

typedef struct {
    PyObject_HEAD
    int value;
} FastIntObject;

Recall our earlier discussion on PyObject “manual” inheritance - PyObject_HEAD is the magic that inserts the PyObject fields:

#define PyObject_HEAD                   \
    _PyObject_HEAD_EXTRA                \
    Py_ssize_t ob_refcnt;               \
    struct _typeobject *ob_type;

And all what we need in addition to that is a int field. Once we have the object definition, let’s define the type:

static PyTypeObject FastIntType = {
    PyVarObject_HEAD_INIT(NULL, 0)
    "fastint.fastInt",         /* tp_name */
    sizeof(FastIntObject),     /* tp_basicsize */
    0,                         /* tp_itemsize */
    0,                         /* tp_dealloc */
    0,                         /* tp_print */
    0,                         /* tp_getattr */
    0,                         /* tp_setattr */
    0,                         /* tp_reserved */
    0,                         /* tp_repr */
    0,                         /* tp_as_number */
    0,                         /* tp_as_sequence */
    0,                         /* tp_as_mapping */
    0,                         /* tp_hash  */
    0,                         /* tp_call */
    0,                         /* tp_str */
    0,                         /* tp_getattro */
    0,                         /* tp_setattro */
    0,                         /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT,        /* tp_flags */
    "Fast int object",         /* tp_doc */    
    0,                         /* tp_traverse */
    0,                         /* tp_clear */
    0,                         /* tp_richcompare */
    0,                         /* tp_weaklistoffset */
    0,                         /* tp_iter */
    0,                         /* tp_iternext */
    FastInt_methods,           /* tp_methods */
    FastInt_members,           /* tp_members */
    0,                         /* tp_getset */
    0,                         /* tp_base */
    0,                         /* tp_dict */
    0,                         /* tp_descr_get */
    0,                         /* tp_descr_set */
    0,                         /* tp_dictoffset */
    (initproc)FastInt_init,    /* tp_init */
    0,                         /* tp_alloc */
    FastInt_new,               /* tp_new */
};

Although this looks rather intimidating at first, this is simply a struct with a bunch of fields that defines what this type is. This defines a type object - an object that represents the fastint type.

Every field is a well-defined operation

Every single field here is a well-defined operation understood by the interpreter. For example, tp_alloc is for memory allocation, and tp_new and tp_init is for object initialization (including initialization for its fields). If you are familiar with COM or C++, you’ll quickly notice that this is simply a v-table with function pointers. By providing field values, you are effectively overriding the base implementation. Oh, where is the actual inheritance, you say? It is handled by PyType_Ready helper - we need to call PyType_Ready to fill in the blanks - initializing fields from base type.

    if (PyType_Ready(&FastIntType) < 0)
        return;

FastIntType itself is an PyObject

Take a look at the PyVarObject_HEAD_INIT macro that initializes PyObject’s fields:

#define PyObject_HEAD_INIT(type)        \
    _PyObject_EXTRA_INIT                \
    1, type,

#define PyVarObject_HEAD_INIT(type, size)       \
    PyObject_HEAD_INIT(type) size,

As you can see above, it is giving a 1 ref count and NULL type (which we’ll fill in later). All Python objects are ref-counted and as a extension writer we need to maintain ref count correctly. A new object starts with 1, not surprisingly.

In theory CPython could be designed with a garbage collector and avoid ref counting. There are quite a few challenges implementing a real garbage collector, such as reducing stop-othe-world collections, fine-tune memory growth policy, knowing where stack and static objects are at all times, avoiding fragmentation, finding the right balance between keeping objects alive vs reclaiming objects, etc. At the end of the day, implementing ref counting is the simplest thing to do, and it stuck. In practice it works usually pretty well, except for cycles (and the perf cost of ref-counting). Python does have a cycle detection algorithm and tp_traverse to help with pointer traversal, but it’s optional.

`tp_new` field defines the new function

static PyObject *
FastInt_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    FastIntObject *self;

    self = (FastIntObject *)type->tp_alloc(type, 0);
    if (self != NULL) {
        self->value = 0;
    }

    return (PyObject *)self;
}

The new function does the allocation and minimum initialization. Conveniently, we’ll use the type object of fastint type to allocate a type object, using the tp_alloc helper. As mentioned previously, tp_alloc wasn’t provided by us but rather filled-in by PyType_Ready, kinda like calling the base constructor. Note that we also has a init function:

static int
FastInt_init(FastIntObject *self, PyObject *args, PyObject *kwds)
{
    if (! PyArg_ParseTuple(args, "i", &self->value))
        return -1;

    return 0;
}

It calls PyArg_ParseTuple to assign the internal value with the supplied int argument. Intuitively, you might think tp_init is the __init__ function and tp_new is the __new__function, and you are right. __new__ is the creation method while __init__ is the initialization method.

`tp_methods` define list of methods

We only have one:

static PyMethodDef FastInt_methods[] = {
    { "inc", (PyCFunction) FastInt_inc, METH_VARARGS, "inc method" },
    { NULL }
};

Again, NULLs are the sentinel. The inc method looks like this - and is pretty straight-forward:

static PyObject *
FastInt_inc(FastIntObject *self, PyObject *args)
{
    int operand;
    if (! PyArg_ParseTuple(args, "i", &operand))
        return NULL;

    self->value += operand;

    Py_INCREF(self);
    return (PyObject *)self;
}

Note that it returns self back to the caller. In Python, returning a object needs to come with an add ref (just like COM). So we need to call Py_INCREF. If you think about it, there is a good reason for it - the caller needs to be sure that the returned object is usable (otherwise the object could be gone after the return, or even before), and having a usable object meaning doing an explicit Py_INCREF. The caller can then use it knowing it is always safe, and release it by calling Py_DECREF when it’s done. There are exceptions to this rule - if by certain implied contract that the caller knows the returned object is always going to be alive (usually tied to a parent object), it can return the object without a addref (one such example being PyTuple_GetItem). Always consult the documentation.

tp_members define its members

We have one member - the value field:

static PyMemberDef FastInt_members[] = {
    { "value", T_INT, offsetof(FastIntObject, value), 0, "value of the integer" },
    { NULL }  /* Sentinel */
};

We basically tell python - here is a field with this offset and you can show it as a member called “value”. The struct field name can be whatever name you want since python doesn’t care what you call your struct fields - meaning that they don’t have to match, but it is nice that they do.

Initializing the type

Now that we have the type with all its members, we need to tell Python that it is there. Not surprisingly, the best place to do so is the module init function:

PYMODINIT_FUNC initfastint()
{
    PyObject *m;

    if (PyType_Ready(&FastIntType) < 0)
        return;

    m = Py_InitModule("fastint", FastIntModule_StaticMethods);
    if (m == NULL)
        return;

    Py_INCREF(&FastIntType);
    PyModule_AddObject(m, "fastInt", (PyObject *)&FastIntType);
}

PyType_Ready initialize the type object by assigning its fields with base and other necessary adjustments and error checking
PyModule_AddObject registers the type on the module so that Python knows this module now has this type.

Trying out fastInt type

Now it’s time to try out our new (not at all) fastint type.

>>> import fastint
>>> i = fastint.fastInt(10)
>>> i.value
10
>>> i.inc(20)
<fastint.fastInt object at 0x0000000004A610D8>
>>> i.value
30

Where is the code

You can find the code at github

What’s next

As previously mentioned, I’ll cover Python C API in the next post and dive into ctypes implementation in CPython (which are written using python C API). I’ll update them with links once they become available:

Part 1 - CTypes
Part 2 - writing CPython extensions using Python/C API
Part 3 - Deep dive into ctypes module in CPython

You can also find this post in http://yizhang82.me/python-interop-capi

Calling C functions from Python - part 1 - using ctypes

Yi Zhang - CLR — Sat, 20 Jan 2018 17:00:40 GMT

Recently I’ve been evaluating Python interop technologies for a project at work and I think it’ll made an interesting blog series. Let’s say your have following C code (add extern "C" if you are in C++ land) and compile it into a dynamic library (dll/.so/.dylib):

    int Print(const char *msg)
    {
        printf("%s", msg);
        return 0;
    }
    
    int Add(int a, int b)
    {
        return a + b;
    }
    
    struct Vector
    {
        int x;
        int y;
        int z;
    };

    struct Vector AddVector(struct Vector a, struct Vector b)
    {
        Vector v;
        v.x = a.x + b.x;
        v.y = a.y + b.y;
        v.z = a.z + b.z;
        return v;
    }

    typedef struct Vector (*pfnAddVectorCallback)(struct Vector a, struct Vector b);

    struct Vector AddVectorCallback(pfnAddVectorCallback callback, struct Vector a, struct Vector b)
    {
        return callback(a, b);
    }

One of the ways to call C API from Python is to use ctypes module. The tutorial in docs.python.org is fairly comprehensive and I certainly don’t intend to cover everything in the tutorial. Instead, I’ll cover it in a exploratory style to show you how what I did to understand these API, and add some fairly interesting details of the API not quite covered by the tutorial (some of the behavior of the API are a bit obscure). In a future post I’ll also deep dive into ctypes implementation in CPython, but for me to get to that, I need to cover the Python C API first in part 2 first, which makes the deep dive part 3. :) Anyway, let’s get started.

Getting started

First let’s import the ctypes module:

>>> from ctypes import *

To load a module, you can use cdll, windll, oledll library loader objects. For example, to load kernel32, you can do:

>>> cdll.kernel32
<CDLL 'kernel32', handle 56930000 at 508eb70>
>>> print vars(cdll)
{'kernel32': <CDLL 'kernel32', handle 56930000 at 508eb70>, '_dlltype': <class 'ctypes.CDLL'>}

Basically accessing its attribute would automatically load a DLL by name. This is implemented in Python by overriding __getattr__ and does a LoadLibrary. Obviously this either requires the DLL to be already loaded or searchable using various rules. Since every process effectively has kernel32.dll loaded in the process, you’ll always load kernel32 successfully. Let’s say we built our dll as MyDll, and try to load it:

>>> cdll.MyDll
Traceback (most recent call last):
  File "", line 1, in <module>
  File "C:\Python27\lib\ctypes\__init__.py", line 436, in __getattr__
    dll = self._dlltype(name)
  File "C:\Python27\lib\ctypes\__init__.py", line 366, in __init__
    self._handle = _dlopen(self._name, mode)
WindowsError: [Error 126] The specified module could not be found

Well, that didn’t work. This is because MyDll is not locatable in path, application directory, nor system32. OK. Let’s try again using cdll.LoadLibrary:

>>> cdll.LoadLibrary(r"D:\Projects\MyDll\Debug\mydll.dll")
Traceback (most recent call last):
  File "", line 1, in <module>
  File "C:\Python27\lib\ctypes\__init__.py", line 444, in LoadLibrary
    return self._dlltype(name)
  File "C:\Python27\lib\ctypes\__init__.py", line 366, in __init__
    self._handle = _dlopen(self._name, mode)
WindowsError: [Error 193] %1 is not a valid Win32 application

Hmm.. That didn’t work either. Unfortunately the error didn’t provide a good description of the actual problem. The problem is that I’ve compiled my dll as a 32-bit DLL while Python.exe is 64-bit, so it doesn’t think it’s a valid (64-bit) application (win32 application is just a general term for 32-bit/64-bit windows applications, as opposed to 16-bit windows). Recompiling the DLL as 64-bit fixed it:

>>> cdll.LoadLibrary(r"D:\Projects\MyDll\x64\Debug\mydll.dll")
<CDLL 'D:\Projects\MyDll\x64\Debug\mydll.dll', handle 4cae0000 at 5064ac8>

Interestingly, it doesn’t really show up in cdll, until you access cdll.mydll:

>>> print vars(cdll)
{'kernel32': <CDLL 'kernel32', handle 56930000 at 508eb70>, '_dlltype': <class 'ctypes.CDLL'>}
>>> cdll.mydll
mydll', handle 4cae0000 at 509d5f8>

This is because cdll.LoadLibrary only returns a new instance of CDLL object. Because garbage collector didn’t kick in yet, the DLL is still loaded in this process, and therefore accessing cdll.mydll would “just work”. However, do note that these two mydlls are separate Python objects (5064ac8 vs 509d5f8), but pointing to the same library (56930000). However, the best way is to keep the instance in a variable - there is no point loading this library twice (there is no harm though as DLL has a ref-count maintained by the OS and you wouldn’t load two copies - there is just one as long as it is the same one).

>>> mydll = cdll.LoadLibrary(r"D:\Projects\MyDll\x64\Debug\mydll.dll")

Calling the function

Let’s try calling Print - just call it as a magic attribute:

>>> print vars(mydll)
{'_FuncPtr': <class 'ctypes._FuncPtr'>, '_handle': 140734480777216L, '_name': 'D:\\Projects\\MyDll\\x64\\Debug\\mydll.dll'}

>>> ret = mydll.Print("abc\n")
abc

>>> print vars(mydll)
{'Print': <_FuncPtr object at 0x0000000005501528>, '_FuncPtr': ctypes._FuncPtr'>, '_handle': 140734480777216L, '_name': 'D:\\Projects\\MyDll\\x64\\Debug\\mydll.dll'}

Note that calling mydll.Print magically inserts a new attribute on the mydll object. Again, this is achieved through __getattr__ So how does ctypes call Print internally? A few things happens:

ctypes does a GetProcAddress (or dlsym) on Print to get the internal address
ctypes automatically recognize that you are passing a “abc”, and converts it to a char *
ctypes uses FFI to make the call, using cdecl calling convention. CDll by default uses cdecl.

Now let’s try doing an Add:

>>> mydll.Add(1, 2)
3

There is a bit ctypes magic at play: by default ctypes assumes every function returns a int, so this works out fairly well. If you want a different return type, you can change it by assigning a type to restype attribute. In this case, what we need is ctypes.c_char, which is the 1-byte char type in C.

>>> mydll.Add.restype = c_char
>>> mydll.Add(97, 1)  # this can be dangerous!
'b'

Now Add would interpret the returned int automatically as a char. Note that this can be dangerous as the size of int and char aren’t exactly the same. However, in most platforms / calling conventions, return value are returned via a register (EAX/RAX in intel platforms), so this simply involves a truncation and work out fine. But again, you don’t want to make such assumptions. So this is just for illustration purpose only. Besides CDLL, there is also windll and oledll. windll by default treat the function as stdcall, and oledll would treat it as a COM function, which means accessing the function by an vtable offset, with stdcall, and returning HRESULT.

Define your own struct

Let’s take a look at how to define your own struct. You can do that by deriving from ctypes.Structure type, and supply a set of fields through the magic _fields_ attribute:

>>> class VECTOR(Structure):
...     _fields_ = [("x", c_int), ("y", c_int), ("z", c_int)]
...

If you print out the individual fields in the VECTOR type, you’ll see magic attributes showing up:

>>> print VECTOR.x, VECTOR.y, VECTOR.z
<Field type=c_long, ofs=0, size=4> <Field type=c_long, ofs=4, size=4> <Field type=c_long, ofs=8, size=4>

Note that the individual fields are nicely laid out sequentially (ofs=0, 4, 8), just what you would expect from a good old C struct. Now we can create new instances of VECTOR and return back VECTOR:

>>> vector_a = VECTOR(1, 2, 3)
>>> vector_b = VECTOR(2, 3, 4)
>>> mydll.AddVector.restype = VECTOR
>>> vector_c = mydll.AddVector(vector_a, vector_b)
>>> print vector_c.x, vector_c.y, vector_c.z
3 5 7

Calling python code from C and some surpises

Let’s make this a bit more interesting. Let’s try to call AddVectorCallback while passinging a python function. To do this you need to make a callback function type first:

>>> ADDVECTORCALLBACK = CFUNCTYPE(VECTOR, VECTOR, VECTOR)

With this type we can then define a Python function that does the add:

>>> def AddVectorImpl(a, b):
...     return VECTOR(a.x + b.x, a.y + b.y, a.z + b.z)
...
>>> mydll.AddVectorCallback(ADDVECTORCALLBACK(AddVectorImpl), VECTOR(1, 2, 3), VECTOR(2, 3, 4))
Traceback (most recent call last):
  File "", line 1, in <module>
TypeError: invalid result type for callback function

Unfortunately, this doesn’t work. Only simple data types like c_int are supported. Complex data types like struct/union are not, because they didn’t provide a setfunc. We’ll cover more of these details in a future deepdive ctypes post.

        StgDictObject *dict = PyType_stgdict(restype);
        if (dict == NULL || dict->setfunc == NULL) {
          PyErr_SetString(PyExc_TypeError,
                          "invalid result type for callback function");
          goto error;
        }

The workaround is to pass in a pointer instead:

    typedef void (*pfnAddVectorCallback)(struct Vector a, struct Vector b, struct Vector *c);

    struct Vector AddVectorCallback(pfnAddVectorCallback callback, struct Vector a, struct Vector b)
    {
        Vector c;
        callback(a, b, &c);
        return c;
    }

>>> ADDVECTORCALLBACK = CFUNCTYPE(None, VECTOR, VECTOR, POINTER(VECTOR))
>>> def AddVectorImpl(a, b, c):
...     c.contents = VECTOR(a.x + b.x, a.y + b.y, a.z + b.z)

And let’s see if it works:

>>> vector = mydll.AddVectorCallback(ADDVECTORCALLBACK(AddVectorImpl), VECTOR(1,2,3), VECTOR(2,3,4))
>>> print vector.x, vector.y, vector.z
-858993460 -858993460 -858993460

OK. So nope. Appears that setting contents doesn’t do what we want. Reading the code - it actually simply swap the internal pointers of the pointer object and doesn’t do any assignment!

    *(void **)self->b_ptr = dst->b_ptr;

The correct way is to assign it on the VECTOR object returned from contents attribute directly:

>>> def AddVectorImpl(a, b, c):
...     c.contents.x = a.x + b.x
...     c.contexts.y = a.y + b.y
...     c.contents.z = a.z + b.z
>>> vector = mydll.AddVectorCallback(ADDVECTORCALLBACK(AddVectorImpl), VECTOR(1,2,3), VECTOR(2,3,4))
>>> print vector.x, vector.y, vector.z
3 5 7

The reason this works is that the VECTOR object internal b_str pointer points directly to the Vector struct pointed by Vector*, so changing this VECTOR object changes the output Vector struct.

What’s next

Part 1 - CTypes
Part 2 - writing CPython extensions using Python/C API
Part 3 - Deep dive into ctypes module in CPython

You can also find this post in http://yizhang82.me/python-interop-ctypes

Calling node.js sqlite callback function using promise and await

Yi Zhang - CLR — Wed, 17 Jan 2018 05:33:41 GMT

Callback pain

I was experimenting with sqlite package with node.js the other day and the async programming style is rather painful, so I did some experiment to wrap it with promise + async/await to make it easier to use. This post shows you how to do that. For example, let’s say I’m working with a database with a Voters table that has two fields - Name and Count.

var sqlite = require('sqlite3').verbose();
var db = new sqlite.Database('test_db');

console.log('sqlite3...');
var stmt = "CREATE TABLE IF NOT EXISTS Voters (Name TEXT, Count int)";
console.log(stmt);
db.run(stmt, function (err) {
    if (err) {
        console.log(JSON.stringify(err));                        
        return;
    }

    vote("john doe", function (err, val) {
        if (err) {
            console.log(JSON.stringify(err));
            return;
        }

        console.log(`New vote for John Doe is ${val}`);
    });
});

And I need to write a function that increase the vote count by 1 or add a new voter entry for a specific voter:

function vote(voter, callback) {
    var val;
    var getStmt = `SELECT Name, Count FROM Voters WHERE Name="${voter}"`;
    db.get(getStmt, function(err, row) {
        if (!row) {
            var insertSql = `INSERT INTO Voters (Name, Count) VALUES ("${voter}", 1)`;
            db.run(insertSql, function (err) {
                val = 1;
                callback(err, val);    
            });
        }
        else {
            val = row["Count"];
            val += 1;
            var updateSql = `UPDATE Voters SET Count = ${val} WHERE Name = "${voter}"`;
            db.run(updateSql, function (err) {
                callback(err, val);
            });
        }
    });
}

As you can see, the async callback style gets pretty painful here. You need to pass a callback to every function that returns an error or result, and within that callback you need to nest your remaining logic, and on and on. Writing a custom function vote also need to pass a callback and call when the final callback is done. The code is difficult to read and maintain.

The promise land

In ES6, you can wrap callback based functions with Promise, which are essentially a return value with state (pending, resolved, rejected). If a function returns a Promise, it’s the function’s job to either resolve or reject it eventually, so that the caller knows when it’s done - either successfully or unsuccessfully. It’s best to demonstrate this with an example:

db.getAsync = function (sql) {
    var that = this;
    return new Promise(function (resolve, reject) {
        that.get(sql, function (err, row) {
            if (err)
                reject(err);
            else
                resolve(row);
        });
    });
};

In this function, we are wrapping sqlite get method and make it returns a promise. When do we know the promise is going to be fullfilled? When the callback arrives. If it fails, call reject and pass the error. If it succeeded, pass the result. Once you have getAsync, you can now call the function like this:

db.getAsync(stmt).then((row) => {
    // print row
}).catch((err) => {
    // handle error
});    

With the returned promise, you need to chain the promise with the next work using then, and handle the unhandled error with catch. The way to think about the then function is to treat them as a list of work - the next work inside the then callback will only get executed when the previous promise gets fullfilled. If it succeeded, then will get called next, otherwise catch is called with the error. If your then callback wants to do more work, it needs to return another promise that you can chain further using then. With this, our vote code would look like this:

function voteAsync(voter) {
    var val;
    var getStmt = `SELECT Name, Count FROM Voters WHERE Name="${voter}"`;
    return db.getAsync(getStmt).then((row) => {
        if (!row) {
            var insertSql = `INSERT INTO Voters (Name, Count) VALUES ("${voter}", 1)`;
            val = 1;
            return db.runAsync(insertSql);  // more work to do
        }
        else {
            val = row["Count"];
            val += 1;
            var updateSql = `UPDATE Voters SET Count = ${val} WHERE Name = "${voter}"`;
            return db.runAsync(updateSql);  // more work to do
        }
    }).then(() => {
        return val;     // when runAsync is done
    });
}

Note how voteAsync chains multiple promise together by returning another promise (from runAsync) and then applying then on top of that promise. To call the code, it looks like this:

var stmt = "CREATE TABLE IF NOT EXISTS Voters (Name TEXT, Count int)";
db.runAsync(stmt).then(() => {
    return voteAsync("john doe");
}).then((val) => {
    console.log(`New vote for John Doe is ${val}`);
}).catch((err) => {
    console.log(JSON.stringify(err));
});    

This is an improvement, but still not quite as natural as you need to explicitly chain then using promise functions and follow those pattern.

Async/Aawait

Fortunately, there are better ways to do this. With async/await support in node.js, you can write these async code just like how you would write regular synchronous code. That is, all javascript statements are serialized. The async/await infrastructure makes sure your code is split into multiple “phases” just like how you would write it in an promise then pattern, except that compiler does all the job for you. And all you need to do is to write this:

async function voteAsync(voter) {
    var val;
    var getStmt = `SELECT Name, Count FROM Voters WHERE Name="${voter}"`;
    var row = await db.getAsync(getStmt);
    if (!row) {
        var insertSql = `INSERT INTO Voters (Name, Count) VALUES ("${voter}", 1)`;
        await db.runAsync(insertSql);
        val = 1;
        return;
    }
    else {
        val = row["Count"];
        val += 1;
        var updateSql = `UPDATE Voters SET Count = ${val} WHERE Name = "${voter}"`;
        await db.runAsync(updateSql);
    }
    return val;
}

The code isn’t significantly smaller, but the logic is much simplified. You don’t need to think in terms of promise chaining - you just write the code without worrying about the underlying “async-ness” of those calls. You do need to call await operator on functions that returns promise, and the compiler automatically

suspends the execution when the promise is in-progress but not done. This is very important as you don’t want to block node.js main event loop.
resumes execution when the promise resolves
throw exception when promise rejects.

The function also needs to have async keyword to indicate it is a async function. Without it compiler will refuse to understand await keyword. To call the async version, just call the function like a normal person (with await, of course):

async function main() {
    try {
        var stmt = "CREATE TABLE IF NOT EXISTS Voters (Name TEXT, Count int)";
        await db.runAsync(stmt);
        var val = await voteAsync("john doe");
        console.log(`New vote for John Doe is ${val}`);
    } 
    catch (e) {
        console.log(JSON.stringify(ex));
    }
}

Again, this is slightly more code due to the try/catch and function goo (javascript doesn’t let you call await without an async function), but the concepts are significantly simpler to understand and code much easier to write.

Conclusion

Javascript promise and async/await is reasonably straight-forward abstraction for async programming once you get used to the model. Wrapping an “classic” callback model API with promise then calling it with async/await greatly simplifies your client code. For your reference, I’m put 3 versions of javascript code that runs (provided that you created a npm and added sqlite package):

BTW, in the upcoming C++ coroutine series, I’ll be talking about how to wrap C async API with C++ coroutines, which are much more involved but would greatly help understand async/await model in a much deeper level. Hope this helps. You can also find this post in http://yizhang82.me/async-sqlite-as-promise.

C++ coroutine tutorial

Yi Zhang - CLR — Sat, 28 Oct 2017 07:32:25 GMT

I’ve been experimenting with coroutines recently and I found that information on C++ coroutines are very difficult to find. I’m planning to write a series of C++ coroutine blogs on how to use C++ coroutines, how they work, and how to write your own library that uses C++ coroutines. My last post Basic Concepts are probably a bit too high-level and is not really meant for people who are new to C++ coroutines. So I’m going to start over with a simple C++ coroutine tutorial instead.

Enough talking. Show me the code

OK. I’m not going to bore you with the technical details (any more), and let’s jump straight into some sample code!

future<int> async_fib(int n)
{
    if (n <= 2)
        co_return 1;

    int a = 1;
    int b = 1;

    // iterate computing fib(n)
    for (int i = 0; i < n - 2; ++i)
    {
        int c = co_await async_add(a, b);
        a = b;
        b = c;
    }

    co_return b;
}

As you can see, this code simply calculates fibonacci, and it’s not a very good one either (again, coroutine != performance). But it’s a good one to show some basic concepts:

A C++ coroutine (such as async_fib) returns a coroutine type.

In this case, we are returning std::future. This is a standard C++ library type. Unfortunately the default implementation in most compilers don’t support using future as a coroutine type. VC++ has its own extension that adds coroutine support to std::future. For the purpose of showing what coroutine is, we are going to assume `std::future` has that support. To run this code, you'll need VC++. So what is a coroutine type anyway? It is a type that is aware of coroutines and implements a bunch of contracts required by C++ compilers. Most these contracts are (synchronous) callback that compiler will call when it’s about to suspend, to resume, to retrieve return value, to record exception, etc. Again, we’ll talk about details in future posts.

C++ coroutines uses co_await / co_return operators

co_await operator means “fire this async operation, suspend my code (if necessary), and resume execution with the return value”. So in this case, when calling co_await async_add(a, b), it’ll let the “expensive” add operation happen in another thread, suspend the execution, and resume the c = asignment with return value, and proceed with next execution. The async operation itself needs to return a awaitable expression. But for now, let’s simplify this to say it has to return a coroutine type. Not quite correct, but for the purpose of this tutorial, this is good enough at the moment. co_return operator simply returns a return value to the coroutine, just like any other function. Note that for coroutines, you typically return the value type T in the coroutine type. In this case, the function signature returns future, so you need to return int. std::futurehere means : I promise I’ll give you a int value in the future when I’m done.

I get it now, but how do I implement an async operation that returns a coroutine type?

async_add is a good example:

future<int> async_add(int a, int b)
{
    auto fut = std::async([=]() {
        int c = a + b;
        return c;
    });

    return fut;
}

async_add returns a future type (note that this is different than earlier - it actually returns the correct type!)

This is effectively saying: I’m returning an object back to you, promising that I’ll give you an int when I’m done. That’s exactly what co_await operator needs.

The real operation is done inside std::async, which conveniently returns a future, that’ll resolve/complete when the async operation is finished. The async operation is running in another thread.

How do I run this thing?

I’m testing this on VC++ on VS 2017 for now. In order to compile the code, you need to pass /await as a compiler switch: For your convenience, I’ve shared the entire cpp file here If you want to use clang, you need to do a bit more work because most importantly futureisn’t a coroutine type there. I’ll have another post talk about how to run it in clang5.

What’s next

OK. I’m going to stop here for now. A few things I’d planning to cover in the future (no pun intended):

How to augment std::future to be a coroutine type, and running coroutines in clang 5
What’s an awaitable, and how to wrap your own async functions using your own awaitable
How to write your own version of future - let’s call it task
Gory details - what does compiler codegen looks like, how suspension/resume works, some additional subtle issues to consider

Hope this helps. You can also find this post in http://yizhang82.me/cpp-coroutines-async-fibonacci

Writing your own lock-free reader/writer lock

Yi Zhang - CLR — Sat, 07 Oct 2017 20:52:58 GMT

It is sort of puzzling for me why C++ standard doesn't have a implementation of reader/writer lock. So just like every other C++ developer, I've decided to roll my own.

A typical implementation might be using semaphore / mutex. However, this has a major disadvantage that it requires entering a lock mutex every time a reader comes in, and this can be expensive as this usually leads to a kernel transition that are costly and can become a bottle neck when multiple readers are trying to access. A lock-free implementation avoids these overhead/bottleneck and allows for better throughput.

I've moved my blog over to yizhang82.me. You can find the full post here: http://yizhang82.me/lock-free-rw-lock

Sharing .NET generic code under the hood

Yi Zhang - CLR — Sat, 01 Apr 2017 18:26:07 GMT

If you come from a C++ programming background, you are most likely already familiar with C++’s template code bloat problem. Each template instantiation gets its own copy of the code (of course, compiler/linker can optimize by throwing away unused methods). The reason being that C++ template are more like C macro on steriods. I know this is a great simplification, but at the end of the day, it is pretty much a code expansion feature with type safety. This grants C++ some powerful capabilities that C# developers don’t have - like template specialization, or calling arbitary methods on a template class, or a whole different programming paradigm that’s known as template meta-programming. On the other hand, .NET generics require you to define what operations can be perform on T using constraints (otherwise you are limited to a small set of operations such as casts, assignments, etc). However, this does give .NET a unique advantage - it can do a better job at code sharing. I've moved my blog over to yizhang82.me. You can find the full post here: http://yizhang82.me/dotnet-generics-sharing

C# value type boxing under the hood

Yi Zhang - CLR — Sun, 19 Mar 2017 07:42:25 GMT

I recently had some really interesting discussion with a .NET typesystem expert in the team, and during the conversation he had pointed out an interesting aspect of .NET value type boxing when using constraints. Intrigued by that discussion, I decided to take a further look. See the full article here in my new blog yizhang82.me

Moving my blog to GitHub pages - http://yizhang82.me

Yi Zhang - CLR — Sun, 12 Mar 2017 06:34:39 GMT

I haven't posted in a good while - it's because I've been busy working on .NET native! Now that things has settled down for a bit, I've decided to resume writing posts. After spending some time thinking, I've decided that I prefer writing my blog posts in plain text/markdown and working on Git repos, rather than wordpress (which MSDN blog has migrated to). Case in point, I have to write this post twice because I've lost the first draft when I'm working on the web editor after I hit publish! Anyway, the new blog is http://yizhang82.me where I'll keep writing about .NET / CLR, COM / WinRT, and many other things I've been looking into. See you there!

Yi Zhang's MSDN Blog

Top secret .NET handles - Part 1 - Dependent handles

Caching without leaks

DependentHandle to the rescue

ConditionalWeakTable

What’s next

Calling C functions from Python - part 3 - deep dive into ctypes implementation in CPython

Loading libraries

Calling the function

Structs and metaclasses

Next in the series

Calling C functions from Python - part 2 - writing CPython extensions using Python/C API

It’s all C code

A module with only static methods

Buliding the module

Trying it out

Defining a fastint class

Every field is a well-defined operation

FastIntType itself is an PyObject

tp_new field defines the new function

tp_methods define list of methods

tp_members define its members

Initializing the type

Trying out fastInt type

Where is the code

What’s next

Calling C functions from Python - part 1 - using ctypes

Getting started

Calling the function

Define your own struct

Calling python code from C and some surpises

What’s next

Calling node.js sqlite callback function using promise and await

Callback pain

The promise land

Async/Aawait

Conclusion

C++ coroutine tutorial

Enough talking. Show me the code

I get it now, but how do I implement an async operation that returns a coroutine type?

How do I run this thing?

What’s next

Writing your own lock-free reader/writer lock

Sharing .NET generic code under the hood

C# value type boxing under the hood

Moving my blog to GitHub pages - http://yizhang82.me

`tp_new` field defines the new function

`tp_methods` define list of methods