Service Fabric - Enable read on statefull secondary replicas

Question

Hi there,

We're tweaking the operations, especially read operation, on our SF cluster.
Some context: we have 1 stateful service which persists data that only changes once or twice a year. This service/data is partitioned in 8 partitions so at this level we already have spread requests accross the cluster. At this moment when fetching data we only use the primary replica per partition and we're trying to enable reads on secondary replicas as well but without success.

At listener side we changed our implementation from this

protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
{
        return this.CreateServiceRemotingReplicaListeners();
}

to this:

protected override IEnumerable<ServiceReplicaListener> CreateServiceReplicaListeners()
{
    if (!_args.ListenOnSecondary)
    {
        // use the default and preferred way of constructing the listeners
        return this.CreateServiceRemotingReplicaListeners();
    }

    var settings = new FabricTransportRemotingListenerSettings
    {
        UseWrappedMessage = true
    };

    return new[]
    {
        new ServiceReplicaListener(context => new FabricTransportServiceRemotingListener(context, this, settings),
                                   _args.ListenerName,
                                   _args.ListenOnSecondary)
    };
}

note: "_args" is just a class containing a couple of properties which is injected in the ctor of the base class including the setting "ListenOnSecondary" to enable this at the registered listener. Nothing special otherwise ;)

The consumer on it's turn calls the code

ServiceProxy.Create<T>(ServiceUri, new ServicePartitionKey(partitionId), replicaSelector)

ServiceUri => constructed based service name
partitionId => calculated based on the way our partitioning is set up for this service
replicaSelector => in this case set to "TargetReplicaSelector.RandomReplica"

When executing on our cluster we sometimes get this error (when a secondadary replica is used):

The primary or stateless instance for the partition '3509a0e4-ba3e-4a58-bc53-a25a73e7b7de' has invalid address, this means that right address from the replica/instance is not registered in the system.

Remark: when using the "default" (TargetReplicaSelector.Default) replica selector everything works fine!

Any thought or things we're missing?