Windows Server 2019 RDS / Task Scheduler

Question

Hello everyone,

We use VMware ESXi to host multiple virtual Windows servers on 3 Hosts. 4 of those virtual servers are used as terminal servers with Remote Desktop Services installed and configured. Around 200 people work on these 4 rds servers. So far so good, they have Windows Server 2019 Version 1809 (Build 17763.1637) installed.

In the past we experienced problems with start menu not showing / start menu button not working, so the users had difficulties with logging out of their session of the rds farm. Without really getting to a solution for this problem we started to reboot these servers daily via task scheduler, and the problem was "solved" for the moment.

After a while with these automated reboots, which worked more or less, task scheduler stopped working like it should: Most of the tasks (system-tasks and user-defined tasks) in task scheduler are being queued. As i said, that happens for system tasks (something like GoogleUpdateTaskMachineCore for example) and user defined tasks, i.e. our reboot. You can manually start these tasks and they work just as they should, but automation stopped working.

We started googling and worked through the possible solutions on these threads:
https://social.technet.microsoft.com/Forums/en-US/990001d3-9229-4fae-aeda-1edb5391c750/task-scheduler-is-queuing-tasks-instead-of-running-them?forum=winserver8gen
https://social.technet.microsoft.com/Forums/en-US/2ba65e21-8f2e-4c0f-a961-3f3f3030fa87/task-scheduler-large-number-of-tasks-queue-status?forum=ws2019

So far we tried different options on the "Settings" tab of the task, i tried to run it just once instead of a daily schedule, i rebooted one of the servers manually, and windows updates are installed. The task even gets executed as domain admin with rights to EVERYTHING, password is saved on the task. I attached some screenshots of the task settings (which are in german but you'll get it).

Event Viewer -> TaskScheduler -> Operational only shows a warning that the tasks got queued (eventid 325), that's the only log that shows at least some kind of information.

If anyone's got a tip, it would be much appreciated. And if your first idea is to execute sfc /scannow please let it be. We did that, with no improvement on the problem.

Answer 1

We had the same issue with an Windows Server 2016 and 2019 RDS environment. We also experience problems that the start menu randomly didn’t open on multiple terminals servers. After some time we figured out it had something to do with the Windows firewall. On each terminal there where several incoming and outcoming rules created for each user that logged on to the terminal server. The rules where never removed at log off so the firewall started to get overloaded with thousands of rules.

Microsoft does have a fix so the firewall rules are removed after log off. https://support.microsoft.com/en-us/help/4490481/windows-10-update-kb4490481

Addresses an issue that slows server performance or causes the server to stop responding because of numerous Windows firewall rules. To enable this solution, use regedit to modify the following and set it to 1:
• Type: “DeleteUserAppContainersOnLogoff” (DWORD)
• Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SharedAccess\Parameters\FirewallPolicy

This fix does work but I first had to manually remove all the firewall rules from each terminal server. Because some servers couldn’t open the firewall settings anymore we’ve found a script that removed the firewall rules and notifications from the register.

But after the script removed "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Notifications" we started to see al lot of tasks being queued which will never run. This had nothing to do with the sysprep issues I have read about.

We took another fresh 2019 server and exported the "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Notifications" key (and sub keys) and imported it to one of the affected terminals. After a reboot the scheduled tasks where up an running again and the start-menu was still working fine.

Answer 2

Well, it's working again! The tip to import the "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Notifications" Key from another Server 2019 made our Task Scheduler work again. I tried installing the update but it seems it has already been replaced with another cumulative update.
BUT, as soon as i imported the registry key and rebooted the server, Task Scheduler started working again. I adjusted our Firewall-Rule-Cleaning-Script accordingly.

Thanks for the help, @DanielKelleners!

Answer 3

Forgot one of the screenshots.

Answer 4

Hi,
1.Per searching, Event ID 325 is probably caused by multiple instances being kicked off at the same time. Please open the scheduled task’s properties and navigate to History tab.

2.According to the symptoms you described, I found following post with similar issues and also shared the root cause might be related to a SysPrep issue.
Fix: Scheduled Task Get Queued and won't run - Server 2012 R2
https://www.reddit.com/r/sysadmin/comments/93uk01/fix_scheduled_task_get_queued_and_wont_run_server/

---

Hope this helps and please help to accept as Answer if the response is useful.

Thanks,
Jenny

Answer 5

Hello Jenny,

1. History tab of the scheduled task just states that it was put in queue, something like "The instance {random id} of task 'Daily Reboot' was put in queue by task scheduler"
2. I checked that, good idea, but when i check the registry keys that indicate this problem (HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Setup\State and the key HKLM\SYSTEM\Setup\Status\ChildCompletion) they don't match the values given in the reddit post: i got IMAGE_STATE_COMPLETE and 3 as values

Any other ideas? I could do sysprep just to test it, but sounds like a lot of work for 4 terminal servers, and i fear losing rdp configuration if i do that. Wouldn't surprise me.