stateless service with instance count 6 when node count 5 fails.

Question

hi,

we have simple stateless service with default instance count 6, deployed to local five node service fabric cluster, errors out as below at 6th instance creation. No port is given for service end point, ports are dynamic, no conflicts.

    System.CRM  ServiceReplicaUnplacedHealth_Secondary_9a78cf1e-252f- 
 4b29-9d0c-10947e92928b Tue, 03 Dec 2019 12:31:37 GMT   0.00:01:05.0     
 132198498979714109 true    false
    The Cluster Resource Manager was unable to find a placement for one or more of the Service's Replicas:
    Secondary replica could not be placed due to the following constraints and properties:  
    TargetReplicaSetSize: 6
    Placement Constraint: N/A
    Parent Service: N/A

    Constraint Elimination Sequence:
    Down nodes count 0, Deactivated nodes count 0, Deactivating nodes count 
 0
    Existing Secondary Replicas eliminated 5 possible node(s) for placement -- 
 0/5 node(s) remain.

    Nodes Eliminated By Constraints:

    Existing Secondary Replicas -- Nodes with Partition's Existing Secondary Replicas/Instances:
    --
    FaultDomain:fd:/4 NodeName:_Node_4 NodeType:NodeType4 NodeTypeName:NodeType4 UpgradeDomain:4 Deactivation Intent/Status: None/None
    FaultDomain:fd:/3 NodeName:_Node_3 NodeType:NodeType3 NodeTypeName:NodeType3 UpgradeDomain:3 Deactivation Intent/Status: None/None
    FaultDomain:fd:/2 NodeName:_Node_2 NodeType:NodeType2 NodeTypeName:NodeType2 UpgradeDomain:2 Deactivation Intent/Status: None/None
    FaultDomain:fd:/1 NodeName:_Node_1 NodeType:NodeType1 NodeTypeName:NodeType1 UpgradeDomain:1 Deactivation Intent/Status: None/None
    FaultDomain:fd:/0 NodeName:_Node_0 NodeType:NodeType0 NodeTypeName:NodeType0 UpgradeDomain:0 Deactivation Intent/Status: None/None

Not sure why 6 th instance creation is failing when there is no port, application is deployed on kestrel host in reliable service.

How can we prevent this error from coming and have the 6th instance created? any help is greatly appreciated.

Answer 1

You can scale to the max number of nodes only.

Reference from this document.

Actual statement from the document:

Maximum Instance Count defines the upper limit for scaling. If number of instances of the partition reaches this limit, then the service will not be scaled out, regardless of the load. It is possible to omit this limit by specifying value of -1, and in that case the service will be scaled out as much as possible (

the limit is the number of nodes that are available in the cluster

).