Server 2019 RDS Session Host randomly disconnects Administrative user at logon

We have two different farms with Citrix VDAs in two different domains. The VDAs are running Server 2019, and obviously have the Remote Desktop Services Session Host role installed.

Occasionally they experience a strange problem, which automatically fixes itself after 2-3 hours. Regular users will be able to log on to Citrix sessions just fine, but if an administrative user tries to start an unbrokered (direct) rdp session to the same server, the session will seemingly start, but the user will automatically be disconnected after 3-4 seconds. The screen only turns black, no notice about user profile loading nor session starting is shown.

Investigating the Event logs shows the following:

When the problem occurs for rdp login,
https://i.ibb.co/1sCPMn2/Failed-Login.png

EventID: 261, Source: TerminalServices-RemoteConnectionManager
Listener RDP-Tcp received a connection

EventID: 1149, Source: TerminalServices-RemoteConnectionManager
Remote Desktop Services: User authentication succeeded

EventID: 39, Source: TerminalServices-LocalSessionManager
Session 4 has been disconnected by session 4

EventID: 40, Source: TerminalServices-LocalSessionManager
Session 4 has been disconnected, reason code 11

*** - No profile is ever loaded/created for the user.

When the problem does not occur for rdp login,
https://i.ibb.co/j6zzhM9/Succeeded-Login.png

EventID: 261, Source: TerminalServices-RemoteConnectionManager
Listener RDP-Tcp received a connection

EventID: 1149, Source: TerminalServices-RemoteConnectionManager
Remote Desktop Services: User authentication succeeded

EventID: 36, Source: TerminalServices-PnPDevices
Redirection of additional supported devices is disabled by policy.

EventID: 36, Source: TerminalServices-ServerUSBDevices
Redirection of additional supported devices is disabled by policy.

EventID: 41, Source: TerminalServices-LocalSessionManager
Begin session arbitration

EvendID: 20521, TerminalServices-RemoteConnectionManager
User config info will be loaded from local machine for this RDP-Tcp connection

EventID: 42, Source: TerminalServices-LocalSessionManager
End session arbitration

EventID: 20482, Source: TerminalServices-RemoteConnectionManager
Remote Desktop Services Network Fair Share was enabled for the user account Domain\UserName with a weight of 1.

EventID: 21, Source: TerminalServices-LocalSessionManager
Remote Desktop Services: Session logon succeeded

EventID: 22, Source: TerminalServices-LocalSessionManager
Remote Desktop Services: Shell start notification received

The problem does not occur every day, but arbitrarily occurs every now and then. Without any real error message apart from "reason code 11" (which is seemingly user initiated disconnect/logoff), this is hard to troubleshoot. Has anyone experienced anything similar?