After many many issues I finally figured out the issues.
There are so many with RDS. It's a 'cluster' :) of issues that pop up one after the other.
Things I did to get it working:
After adding high availability I started getting the error. Maybe restarting the system right away would have cured that but I didn't do that.
I had added a system right after and thats when I started getting issues.
Ultimately I had to be on the server I wanted to add to high availability. Like I'm on CB02 and adding.
Add-RDServer -ConnectionBroker "CB01" -Server "CB02" -Role RDS-CONNECTION-BROKER
I initially had another Connection Broker CB00. I had Remove-RDServer the role after adding the role to CB01 and moving the ActiveMangementServer. The error kept coming back.
- I had to go to CB00 and manually remove RDS services and restart. Through "Remove Roles and Features"
- I had to go into the High Availability Database that had been created and remove CB00 from the list of Brokers.
delete from [RDSConnectionBroker].[rds].[ConnectionBroker] where Name = 'CB00'