CNG signature provider registration problem

Question

Hello,

I've written a DLL which implements a signature algorithm using the example code from CPDK as a base. I cannot get Windows 10 certificate UI to honor calling my DLL when it gets the certificate. DLL works normally if you use BCryptOpenAlgorithmProvider() with my algorithm name, (which is unique), DLL gets loaded. Thus I presume the provider registration is fine. The separate part to DLL is OID registration where I get algorithm name in string in certificate UI (thus OID lookup in registry works), algorithm names match, but it isn't called.

There is no example for this class of provider in CPDK, nor for its OID registration. But there is for other kind(s) of providers.

There is CRYPT_OID_INFO structure in API for OID->algo registration but it seems to take two different 'names', pwszName labeled as 'provider name' and pwszCNGAlgid labeled as 'AlgId string passed to BCrypt API'. Setting these to same string which is algo name doesn't do anything. Setting the latter to algo name and former to something else just makes that something else appear in the certificate UI. Again I'm referring to HashProviderOIDRegistration example from CPDK which could be working differently.

Worth mentioning is that this chain of trust uses the custom algorithm through, thus CA/subCA will also be signed by it. The authority also uses same OID for algorithm and public key.