Azure APIM doesn't support custom assemblies. Please feel free to raise a feature request to add support for this by sharing your use case.
An alternative would be to have a function app using your custom assembly that is exposed over HTTP, which you could call from your policies using the send-request policy.
Since this will be over the network, latency is something you would have to consider. You could address this in a couple of ways