Remote Desktop hosts randomly stop communicating with Connection Broker and horde all connections

Question

I have a rather large setup with:

2x Connection Brokers (HA - MSSQL)
11x Collections
6x Gateways

4 of them have 25 rd hosts, the others 3 or less.

Load-balancing is done on a Fortigate appliance and cb

What happens is that after x amount of days, a random server in the large collections will stop redirection new sessions to other servers even though it is above relative weight.
Connections for that server also stop refreshing on the Connection broker and only way to fix it is to drain it and restart it.

If it not restarted, sessions will still get redirected to it, even though it is set to DRAIN both locally and on the Connection Broker (server manager) resulting in an error for users on screen saying remote logons are currently disabled

I am not finding any traces in event logs other than the absence of EventID 1307 under TerminalServices-SessionBroker-Client, endless 1301 and I know that server is having issues.