SQL 2019 Basic Availability Group Cannot create listeners

Question

2x SQL 2019 Standard severs on Windows Server 2019 Standard.

One in PROD site and one in DR site. The two sites have large bandwidth (>400Mbps) and <2ms between them.

These have been built with the intention of hosting Basic Availability Groups (BAG) for all our compatible databases.
BAGs require one IP in each subnet for the listener.

We created the first Database BAG fine and it works well.

When we try to create further BAGs, we fail when we try to add the listeners (IPs in each subnet). The proposed IPs are not in use by any other system.

So current state:
1st BAG Listener 'AG-Database1' IPs 10.1.1.5/24 (prod) and 10.2.1.5/24 (DR) port 1433

Try to create 2nd listener:
'AG-Database2' IPs 10.1.1.6/24 and 10.2.1.6/24 port 1433
The BAG builds fine but listener add fails with:

The WSFC cluster could not bring the Network Name resource with DNS name 'AG-Database2' online. The DNS name may have been taken or have a conflict with >existing name services, or the WSFC cluster service may not be running or may be inaccessible. Use a different DNS name to resolve name conflicts, or check the WSFC >cluster log for more information.
The attempt to create the network name and IP address for the listener failed. If this is a WSFC availability group, the WSFC service may not be running or may be >inaccessible in its current state, or the values provided for the network name and IP address may be incorrect. Check the state of the WSFC cluster and validate the >network name and IP address with the network administrator. Otherwise, contact your primary support provider. (Microsoft SQL Server, Error: 19471)

The cluster and services are fine. The cluster validates OK. The proposed additional IPs are not in use (not in DNS and don't respond to ping).

We pre-stage the listener computer object and the cluster CNO "sqlcluser" has full control on the listener computer object in ADDS. This is the bit that most articles point to as the usual cause but it's not in our case.

The only relevant errors I can find in the cluster log talk about crypto errors and I haven't found much about them.

INFO [RES] Network Name: [NNLIB] FindSuitableDCNew - objectName AG-Database2, username - SQLCLUSTER$, firstChoiceDCName - \THEDC01.domain.com
INFO [RES] Network Name: [NNLIB] Found first choice DC that has the object AG-atabase2 DCName - \THEDC01.domain.com.
ERR [RES] Network Name: [NNLIB] Could not acquire crypto context, status 2148073487
ERR [RES] Network Name: [NN] Encrypting password using crypto access failed, error 2148073487

I'm stuck as to why I can't create further BAG listeners.

Answer 1

Could not acquire crypto context, status 2148073487

Converting that to a HRESULT = NTE_EXISTS (0x8009000F)

The key container already exists, but you are attempting to create it. If a previous attempt to open the key failed with NTE_BAD_KEYSET, it implies that access to the key container is denied.

Given the above, I'd run a procmon when attempting to create the network name to see if there are other access denied items in the crypto registry locations or filesystem.

Answer 2

Hi @SeanGallardy-1365

Thanks for that!

Procmon showed access denied on C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys

Some further searching indicates that changing SQL server service account through services.mmc instead of SQL Configuration Manager can mean that some ACLs are not populated correctly. This combined with a recent event where the SQL server ssl cert was changed and then the services refused to start may have contributed.

Resetting inheritance on that folder solved it. The server now creates the listeners.

(The process that accesses that folder is the RHS.exe (Resource Host for Windows Cluster running as Local System))

Answer 3

Random Question: What technique did you use to convert "2148073487" to "HRESULT = NTE_EXISTS (0x8009000F)" ?