Network Stack Crash on RDS with FSLogix Profile Disks

Question

We are having an issue where on several RDS Hosts, in multiple different Collections experience an issue where they loose all ability to connect to anything network related. The only corrolation we find is that all of the effected hosts are in a collection that have multiple session hosts, and use FSlogix profile disks. I believe all of these used RDS Profile disks at one time and were converted to FSLogix as well. We have a mix of RDS servers that are cloned and each built individually.

When the Issue presents it self. Even a user without FSLogix, which is any of our support users and admins fail to login with the error cannot connect to the domain. When we do login with the local administrator account. We find that we can ping all of the domain controllers and the domain it's self. Can ping google.com or google DNS. But should we try to \domain in explorer it will fail, or if we open, Edge or Chrome and we get a connection time out. Even if we try and load a site by IP address in a browser we get a connection time out.

At this point the only corrective action we can do to get the servers back in a place to allow users to log in is to turn off the VM and turn it back on. We cannot gracefully reboot the server as the users that are still logged on will not log off since they cannot talk to their profile disks.

Our environment is a single deployment of RDS Gateway and Connection broker with multiple collections. Everything runs as VMs on Hyper-V both 2016 and 2019 servers. All RDS hosts are on a 2019 server in one of 2 HV clusters. All settings on each HV host are the same configured through VMM and some manual PowerShell scripts for things VMM won't set making sure all hosts are configured the same. The only difference at the host level is the hard ware running each cluster. but we've had the issues on the same VMs no matter which cluster they are on.

We are looking for any idea on a root cause so that we can stop this from occurring. I'm considering a new rebuild on each collection with a proper clone of the VM to the number of VMs we need to keep the user load per host at what we need.

Answer 1

Hello. Install all updates. Over the last half year there have been many performance-related fixes to terminal services.

https://support.microsoft.com/en-us/topic/november-27-2018-kb4467684-os-build-14393-2639-7eb61afe-e3de-b34d-0d30-a77670f355fe

Addresses an issue that slows server performance or causes the server to stop responding because of numerous Windows firewall rules. To enable the changes, add a new registry key “DeleteUserAppContainersOnLogoff” (DWORD) on “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy” using Regedit, and set it to 1.

More detail:

https://social.technet.microsoft.com/Forums/en-US/992e86c8-2bee-4951-9461-e3d7710288e9/windows-servr-2016-rdsh-firewall-rules-created-at-every-login?forum=winserverTS&ranMID=24542&ranEAID=TnL5HPStwNw&ranSiteID=TnL5HPStwNw-.elcBQh5cl6K6bHUZBREXQ&epi=TnL5HPStwNw-.elcBQh5cl6K6bHUZBREXQ&irgwc=1&OCID=AID2200057_aff_7593_1243925&tduid=%28ir__zlln3t6ibkkf6givh3h3re0cde2xrasi91hojrdm00%29%287593%29%281243925%29%28TnL5HPStwNw-.elcBQh5cl6K6bHUZBREXQ%29%28%29&irclickid=_zlln3t6ibkkf6givh3h3re0cde2xrasi91hojrdm00

• https://learn.microsoft.com/en-us/windows/application-management/per-user-services-in-windows

Beginning with Windows 10, version 1709 and Windows Server, version 1709, you can prevent the per-user service from being created by setting UserServiceFlags to 0 under the same service configuration in the registry

Disable unnecessary services for users.

• DisableTaskOffload - HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
  DisableTaskOffload
  REG_DWORD
  0x1 (1)

Answer 2

This is probably my issue. I had totally forgotten about this issue with UPD. I had thought that being fully patched and using FSLogix instead of UPD would have corrected that. Maybe I was wrong. I'll have to implement the Firewall Key and see.

Answer 3

Those settings made it better, but it still occurs just less freqently.