SQL Server 3-node cluster can't see remote DR node

Question

Hello All. I'm setting up a typical SQL 2022 3-node availability group cluster with two HA nodes in primary data center and DR node in remote data center. This is my first attempt at dedicating interfaces to replication traffic The two HA nodes have a secondary NIC to which I assign 192.168.100.101 and .102 respectively. I can ping the HA from each other on their private IPs. The dedicated NICs are interconnected directly; not plugged into a switch. I bind the endpoints to these IPs. I use the same SQL Server domain account to run all the services. I gave the account connect rights to the endpoint. When I created the AGs, I specified the private IPs. The primary and secondary replicas got created, but the DR node always has a red "X". The DR node connects to a switch and does not have dedicated NICs. So I think I have the permissions and the endpoints correct. When I do a PowerShell test using Test_NetConnection from the primary HA node to the DR node using either hostname or IP to connect tp the endpoints, it succeeds. Why is there still a red X on the DR node? Thanks.

Answer 1

The HA replicas’ endpoint URLs must use addresses reachable from the DR node; either move those endpoints from the directly connected 192.169.100.x interfaces to routable addresses/FQDNs, or provide routing from DR to that dedicated replication network.

Your successful test from an HA node to DR verifies only one direction. Every replica must be able to reach every other replica’s database-mirroring endpoint over TCP. Because the HA private NICs are connected only to each other, the DR node normally has no path back to those endpoint addresses. That leaves the DR replica DISCONNECTED, which SSMS displays as a red X.^1^

Confirm the mismatch

1. On every SQL Server instance, inspect the AG endpoint URL, actual endpoint port, state, and encryption:

SELECT replica_server_name, endpoint_url
FROM sys.availability_replicas;

SELECT name, state_desc, type_desc,
       protocol_desc, encryption_algorithm_desc
FROM sys.database_mirroring_endpoints;

SELECT type_desc, ip_address, port
FROM sys.tcp_endpoints
WHERE type_desc = 'DATABASE_MIRRORING';

The URL has the form TCP://<system-address>:<port>. Its address must identify the target computer through an address reachable by the other replicas; Microsoft recommends an FQDN, while a static IP is also supported.^2^

2. From the DR node, test each HA replica using the exact address and port shown in sys.availability_replicas:

Test-NetConnection -ComputerName [IP Address] -Port 5022
Test-NetConnection -ComputerName [IP Address] -Port 5022

Replace 5022 with the actual endpoint port. These tests are expected to fail if those NICs have only a direct HA-to-HA connection.

3. Also test the reverse direction—from each HA node to the DR endpoint—and verify TcpTestSucceeded : True. Microsoft’s disconnected-replica procedure explicitly requires testing both directions.^1^

Correct the topology

Choose one design:

• Preferred for this topology: configure each HA endpoint/replica URL with the server’s routable static address or FQDN that all three nodes can reach.
• Keep the dedicated replication network: connect it through switching/routing and give the DR node a valid route to both HA endpoint addresses. Confirm that return routing and firewalls allow the endpoint TCP port.
• Add a dedicated DR replication NIC/network: use this only if that network becomes routable end-to-end; merely adding another isolated interface does not solve reachability.

A database-mirroring endpoint is shared by all AG connections for that SQL Server instance. It is not a separate endpoint per remote replica, so binding the instance’s endpoint exclusively to the isolated HA link prevents that same endpoint from accepting the DR replica’s traffic.^3^

Do not reconfigure an active endpoint casually: changing or restarting it temporarily interrupts AG traffic and can affect automatic failover.^3^ Schedule the change, update the affected replica endpoint URLs, and then repeat all six directional tests.

If bidirectional tests succeed but the X remains

Compare all three endpoints for:

• state_desc = STARTED
• The URL port matching the actual DATABASE_MIRRORING port
• Identical compatible encryption configuration
• No other process or SQL instance owning the port

$port = 5022
Get-NetTCPConnection -LocalPort $port |
    Select-Object LocalAddress, LocalPort, State, OwningProcess

Get-Process -Id (Get-NetTCPConnection -LocalPort $port |
    Select-Object -First 1 -ExpandProperty OwningProcess)

Microsoft lists port conflicts, encryption mismatch, stopped/deleted endpoints, and connectivity as causes of a disconnected replica.^1^

Because all SQL Server services use the same domain account, the corresponding logins normally exist automatically in master; additional cross-account endpoint grants are generally required when replicas use different accounts.^4^ Therefore, endpoint permissions are not the leading issue here.

---

References

1. Availability replica is disconnected within an Always On availability group
2. Specify Endpoint URL - Adding or Modifying Availability Replica
3. The database mirroring endpoint (SQL Server)
4. Troubleshoot Always On Availability Groups Configuration (SQL Server)