Is it possible to configure the Service Fabric Repair Manager Service?

Chris Sloma 1

When running a double fault domain failure test with losing 2 consecutive fault domains on a 13 node cluster, we had quorum loss on the Repair Manager Service. We are set up with 5 FD's and 5 UD's each with a primary. All the system services are spread across the 5 FD/UD's except for the Repair Manager Service.

vipullag-MSFT 24,441 Reputation points

2022-08-05T06:03:06.257+00:00

@Chris Sloma

Can you please elaborate more on how did you try "double fault domain failure test" ?

Chris Sloma 1

We created a tester using ChaosTester methods.
Snippet:
var faultDomains = await GetFaultDomains().ConfAwait();
var counter = 0;

    while (counter < faultDomains.Count - 1)  
    {  
        var nodes = faultDomains.ElementAt(counter).Value;  

        nodes.AddRange(faultDomains.ElementAt(counter + 1).Value);  

        await TransitionNodesAsync(nodes, Options.DurationInSeconds).ConfAwait(); //Stop  

        await Wait(TimeSpan.FromSeconds(options.DelayInSeconds)).ConfAwait();  

        await TransitionNodesAsync(nodes).ConfAwait(); //Start  

        await Wait(TimeSpan.FromSeconds(options.DelayInSeconds)).ConfAwait();  
    }

Chris Sloma 1 Reputation point

2022-08-05T18:56:57.073+00:00

Essentially the test will shutdown FD0 and FD1
Wait a given period of time
restarts FD0 and FD1
waits a period of time.
shutsdown FD1 and FD2
waits
restarts FD1 and FD2
waits
continues until all FD's have been shutdown/restarted.
vipullag-MSFT 24,441 Reputation points

2022-08-08T09:52:25.477+00:00

@Chris Sloma

Thanks for sharing the details.