Software-Defined Networking (SDN) State Database Synchronisation Failure

Question

Hi all,

Our SDN Network Controller cluster is currently reporting Service Unhealthy. Following a brief network partition, the state databases across the 3 controller VMs have fallen out of sync and failed to recover automatically.

Could anyone advise on the correct procedure to force database re-synchronisation across the SDN controller nodes? Specifically:

- Node State Verification: What is the recommended CLI command or log location to identify which node currently holds the authoritative database state versus which nodes are strictly out of sync?

- Forced Synchronisation / Re-clustering: How do we safely trigger a full database resync from the primary node to the secondary nodes without corrupting active fabric states?

- Quorum Recovery: If the cluster has lost quorum completely, what is the proper process to reset the cluster state and rejoin the secondary controller VMs?

Any help or steps to safely restore cluster health would be greatly appreciated.

Answer 1

the database is distributed across the cluster, there is no single master node. You must open an elevated PowerShell prompt and execute the Get-NetworkControllerNode cmdlet to query the overall Service Fabric health and identify the explicitly out-of-sync nodes.

To safely resynchronize the cluster, you must forcefully clean the corrupted state from the broken nodes. Execute the Repair-NetworkControllerNode cmdlet targeting the unhealthy controllers. This safely evicts the node, clears the local database corruption, and automatically triggers a fresh synchronization from the nodes that still maintain quorum. If your cluster has suffered a complete loss of quorum, the underlying Service Fabric ring is broken and cannot self-heal. You must deploy a fresh Network Controller cluster and restore your authoritative network state from a backup or have your management plane, such as System Center Virtual Machine Manager, push the configuration down again.

Hope this answer has brought you some useful information. If it did, please hit “accept answer”. Should you have any questions, feel free to leave a comment.

VPHAN