Server 2022 and Teams Bloats HKEY_LOCAL_MACHINE\SYSTEM\Software\Microsoft\TIP\TestResults\

JJAComm 106

Hi, has anyone else deployed MS Teams on Server 2022 and noticed bloating of the registry key:
HKEY_LOCAL_MACHINE\SYSTEM\Software\Microsoft\TIP\TestResults\
We need to clear this daily to prevent slow downs. From what I can see, it seems to be only written to by MS Teams.
Cheers

Yuki Sun-MSFT 40,856 Reputation points

2022-02-28T02:49:20.59+00:00

Hi @JJAComm ,

I tried checking in my Windows Server 2022 with no Microsoft Teams installed, and noticed that the registry key you mentioned was already existent there.

Then I installed Teams and then logged in, the registry path is still there with no additional entries added:

So it seems to me that the bloating of the registry key is not due to the Microsoft Teams client.

Given this, personally I'd recommend clearing the registry key first, then trying to catch logs using tools like Process Monitor and see if it can help detect the "culprit".
JJAComm 106 Reputation points

2022-02-28T18:50:12.847+00:00

Hi @Yuki Sun-MSFT
Thanks for looking at this. I ran processmon and found that the keys are not touched apart from when you open Teams. when you open Teams you get the attached type of action (.pml file renamed to .log)
178625-logfile.log

I cannot get any action on this key apart from when opening Teams. I wonder if it is related to installed languages. I have English NZ, French, Maori, Chinese (Simplified, China) Installed.
Cheers
John
Yuki Sun-MSFT 40,856 Reputation points

2022-03-01T08:51:07.36+00:00

Hi @JJAComm ,

I wonder if it is related to installed languages. I have English NZ, French, Maori, Chinese (Simplified, China) Installed.

May I know what you mean by "installed languages"? Since as far as I know, the language setting for Teams can simply be changed via Settings -> General -> Language:

So are you referring to the languages installed on the Windows OS, see Install a language for Windows? If this is the case, in order to narrow down whether it's related to the languages, I'd suggest trying to remove some languages and see if there would be any improvement.
JJAComm 106 Reputation points

2022-03-03T18:39:24.267+00:00

OK- so I ran the teams machine wide installer on a new server with no other languages installed except English US

And it still wrote to those keys

So I can discount that theory but am still lost as to why Teams keeps writing to these keys and does not remove them afterwards. I have to schedule a script removal of the registry keys to run several times a day to keep the servers stable.
Yuki Sun-MSFT 40,856 Reputation points

2022-03-04T09:55:09.347+00:00

Hi @JJAComm ,

So these registry keys won't appear if the Teams client is not opened or installed on the server, right?
Do you perhaps have any devices like microphone which may be used by the Teams client? How about disconnect them if there are and see how it goes?

I tried searching further but still hardly found any information about this registry path "HKEY_LOCAL_MACHINE\SYSTEM\Software\Microsoft\TIP\TestResults\". Noticed you've also added other tags like "windows-server", may be we can wait for a few more days and see if their community can share some insights as well. But for now, agree with you on scheduling a script to remove these registry keys as a workaround.
IT-Guy-9874 86 Reputation points

2022-03-10T09:08:17.357+00:00

Hello,
We have a server 2022 using RDS that have a lot of freeze.
The explorer.exe is around 6% and we have a lot of registry activity as you : HKLM\SYSTEM\Software\Microsoft\TIP\TestResults
And the same in procmon.
We are not using teams.
Os is in english and french is installed on the server as a language pack.
IT-Guy-9874 86 Reputation points

2022-03-10T10:35:16.797+00:00

There is no link with the language pack.
We have a lot of ID under HKLM\SYSTEM\Software\Microsoft\TIP\TestResults\27641370 on a fresh install with Edge and C++ 2016-2019 and vmwaretools. Without language pack and no application.
Christian Feuerstein 26 Reputation points

2022-03-10T13:40:32.483+00:00

Same here,
I'm troubleshooting frequent Taskbarhangs and stumbled across these.
Would really be cool to have some Info about them...
IT-Guy-9874 86 Reputation points

2022-03-10T13:50:59.387+00:00

Same here, right click on taskbar do not respond.
Outlook is frozen and the right click on the desktop works...

We have deleted the keys on two servers.
Christian Feuerstein 26 Reputation points

2022-03-10T13:58:59.48+00:00

I think you're right to do it, because the really takes time even if I open it in regedit manually.
In Processmonitor this key gets queried a lot.
I suppose I'll stick with your approach and if it helps clean them ~every hour

Thanks for sharing!
IT-Guy-9874 86 Reputation points

2022-03-10T14:26:24.937+00:00

I made this script :)

Remove-Item -Path "HKLM:\SYSTEM\Software\Microsoft\TIP\TestResults\27641008" -Recurse
Remove-Item -Path "HKLM:\SYSTEM\Software\Microsoft\TIP\TestResults\27641370" -Recurse
Christian Feuerstein 26 Reputation points

2022-03-10T14:35:39.773+00:00

Thank you, I just changed Permission so that system is unable to write underneath TestResults ;)
But works like a charm - no more problems since the keys are gone.

Good thing that Teams is 3rd party :P
JJAComm 106 Reputation points

2022-03-10T18:47:50.48+00:00

Thanks for sharing guys, it is kind of good to know others are seeing the behaviour and it is not just me doing something wrong! Nice idea about changing the permissions on the key.
Stefan Runarsson 1 Reputation point

2022-03-10T20:31:25.43+00:00

Do you happen to run another app built with Electron like Teams is?
IT-Guy-9874 86 Reputation points

2022-03-11T13:40:57.417+00:00

Customer feedback: The deletion of the keys solved the issue.
Does one of you have already reported the bug to Microsoft?

Thanks for your help
Stefan Runarsson 1 Reputation point

2022-03-13T20:00:07.327+00:00

Yes, I have a case open with them at the moment #30006205

It has been escalated to the "senior internal consultations team", and I'm waiting on them to contact me.
IT-Guy-9874 86 Reputation points

2022-03-13T20:07:55.813+00:00

Ok, many thanks. Let me know if they need some information from us.
Martin Lowery 1 Reputation point

2022-03-17T10:54:45.947+00:00

Sorted our issue too. Server 2022 with Office 2019. No Teams installed

THANK YOU!
Jon Mckeown 6 Reputation points

2022-03-29T22:51:05.027+00:00

I found that I had to remove the 'Users' group from the permissions too, otherwise the keys still kept getting created by user sessions.

This issue was causing us major problems on a brand new 2022 RDS Session Host. Users were unable to drag tabs around in Edge and Chrome, and taskbar was freezing up. Clearing out HKLM\SYSTEM\Software\Microsoft\TIP\TestResults\ resolves the issue instantly.
Christian Feuerstein 26 Reputation points

2022-03-30T06:28:54.333+00:00

Thanks for sharing, in my case the keys were never recreated so i thought it was sufficient.
Christian Feuerstein 26 Reputation points

2022-03-30T06:36:42.75+00:00

Please keep in mind, that Office on Server 2022 is not supported - could be that you stumble across theese errors in near Future ;)
Valerius Brug 6 Reputation points

2022-04-08T14:18:05.933+00:00

It not only affects Office, our own application, which uses the on-screen keyboard, has also blown up the registry ;)

5 answers

Binh Tang 41 Reputation points

2022-10-19T01:46:20.537+00:00
THANK YOU EVERYONE for sharing!

I have a 2022 RDS Server and was beating my head with issue for about 3 months now! The issues were random lag and slowness and more so when switching between apps. Tried so many troubleshooting it was crazy. It wasn't until I stumbled across this post that after implementing the changes, users have now reported no issues with lag or slowness (implemented 6 days ago and no issues since)!

What lead me to this was finding out explorer.exe was randomly sitting at 12%-20% CPU for some users. If I restart explorer.exe for that user, it would fix temporarily but then another user would have the same issue. CPU would also spike overall to 70%-99%.

My changes:

The HKEY_LOCAL_MACHINE\SYSTEM\Software\Microsoft\TIP\TestResults reg key when I exported it was 42MB in size! That shows how massive it was. I deleted all subkeys under that

Disabled the TabletInputService service (Touch Keyboard and Handwriting Panel Service)

Took ownership of C:\Program Files (x86)\Common Files\Microsoft Shared\ink\TabTip32.exe and renamed it (to prevent it from starting up and adding to TestResults key)

Took ownership of C:\Program Files\Common Files\microsoft shared\ink\TabTip.exe and renamed it (to prevent it from starting up and adding to TestResults key)

Rebooted server

After the changes, Explorer.exe no longer has high CPU usage. CPU overall is good, and users are happy and not reporting any lag or slowness. Such a random and odd issue. Glad I've found this and thank you everyone for sharing.
Please sign in to rate this answer.

7 people found this answer helpful.
Leo 1 Reputation point

2022-11-03T19:00:16.217+00:00

I just wanted to thank you very much for this! I have one new Server 2022 RDS server with the exact same issue for months, and after following your steps its working great again!

Janusz Siudek 0 Reputation points

2023-01-17T21:58:58.99+00:00

I had the same problem. Thank you

03945648 1 Reputation point

2023-02-04T19:28:59.63+00:00

the export of TIP\TestResults in my case was 140 MB... that's a lot!
I don't have MS Teams on my server tho... Any idea what could possibly be the reason if it is not teams?

Kari Jurva 0 Reputation points

2023-02-24T06:34:54.35+00:00

For me, the symptom started suddenly in August 2022. Symptoms here disappeared after just deleting everything under the TestResults key. I didn't need to stop or restart anything. Not even Chrome with two tabs, running at the same time, immediately stopped to cause high CPU. Now, after 20 mins, a new key was created under TestResults.

Update 10 minutes later: I deleted the TestResults key and it was not created after restarting Teams. I hope not having the key doesn't have side effects.

JJAComm 106 Reputation points

2023-02-24T06:50:32.6666667+00:00

No side effects @Kari Jurva , I have been doing this on servers for a year now without issue.

REMZİ ÖZTEL 0 Reputation points

2023-04-11T14:33:52.5566667+00:00

Thank you so much. I had this issue for months. Whenever any chronium browser starting up (chrome, edge, opera) explorer.exe was begining to consume CPU usage above 10% and server was becoming freezed. Start button not working and browser was going unresponsive. I searched a lot, I did everything but no luck until I have come access your post. Thanks a lot. By the way, my TestResult folder more than 90 MB and I don't have Microsoft Teams installed.
Sign in to comment
JJAComm 106 Reputation points

2022-10-19T23:55:38.997+00:00

Here is what I've been running when I hit this issue now:

taskkill /f /IM tabtip.exe
taskkill /f /IM tabtip32.exe

takeown /F "C:\Program Files\Common Files\microsoft shared\ink\tabtip.exe"
cacls "C:\Program Files\Common Files\microsoft shared\ink\tabtip.exe" /T /E /P Administrators:F
cacls "C:\Program Files\Common Files\microsoft shared\ink\tabtip.exe" /T /E /P SYSTEM:F
ren "C:\Program Files\Common Files\microsoft shared\ink\tabtip.exe" tabtip.exe.old

takeown /F "C:\Program Files (x86)\Common Files\microsoft shared\Ink\tabtip32.exe"
cacls "C:\Program Files (x86)\Common Files\microsoft shared\Ink\tabtip32.exe" /T /E /P Administrators:F
cacls "C:\Program Files (x86)\Common Files\microsoft shared\Ink\tabtip32.exe" /T /E /P SYSTEM:F
ren "C:\Program Files (x86)\Common Files\microsoft shared\Ink\tabtip32.exe" tabtip32.exe.old

Reg Delete HKEY_LOCAL_MACHINE\SYSTEM\Software\Microsoft\TIP\TestResults\ /f
Please sign in to rate this answer.

4 people found this answer helpful.

0 comments No comments
Sign in to comment
JJAComm 106 Reputation points

2022-03-18T01:15:41.34+00:00

OK- Working on the theory that TIP stands for Text Input Panel I tried disabling the Touch Keyboard and Handwriting Panel service but could not.
Then I went into Teams Settings, App Permissions, One Note and disabled this.
The keys were no longer written to so for me at least, the issue here was around One Note.
Looking further, renamed out
C:\Program Files\Common Files\microsoft shared\ink\tabtip.exe.
Rebooted
Created new user profile and could no longer replicate the issue.
Please sign in to rate this answer.

1 person found this answer helpful.
Yuki Sun-MSFT 40,856 Reputation points

2022-03-18T09:51:15.197+00:00

Hi @JJAComm ,

Great to know that you've managed to sort it out and thanks for your valuable sharing!
Sign in to comment
Robin 6 Reputation points

2022-08-18T15:56:23+00:00

Thanks a lot for this helpfull post!
We have the same issue. We are still testing it but it looks like deleting the keys is giving already a huge difference.
Would it also be ok to disable the TabletInputService service (Touch Keyboard and Handwriting Panel Service) as replacement for renaming tabtip.exe? I tried it with no reboot and no new profiles with no luck. I can reboot and test again, all new profiles will be more difficult to do.
Please sign in to rate this answer.
JJAComm 106 Reputation points

2022-08-18T19:31:52.37+00:00

I was unable to disable the service, permissions issue. I've now had to do the exe rename tabtip.exe and tabtip32.exe on several servers, have not needed to recreate user profiles, just kill that functionality. Tabtip32.exe was causing pdf files to be slow to open and view using a certain document management system.

Robin 6 Reputation points

2022-09-02T09:25:10.7+00:00

Indeed, I can confirm.
This worked for me (on test and live environment):

1) Delete the keys in HKEY_LOCAL_MACHINE\SYSTEM\Software\Microsoft\TIP\TestResults\
2) Disabled the TabletInputService service (Touch Keyboard and Handwriting Panel Service), if you can not do that try the solution from JJACommar by renaming tabtip.exe and tabtip32.exe
3) Reboot the server

Thanks a lot, this was really helpfull!
Not you won't be able using the virtual keyboard on the server anymore (till Microsoft fixes this).

Hope this helps somebody else.
Sign in to comment
Master Control 1 Reputation point

2022-12-01T13:52:21.067+00:00

This occurred with us on a MS 2022 Server and when we spun up a MS2019 Server we had the same issue with RDP sessions having latency. It was very random and weird behavior because it wasn't consistent at all. Some RDP sessions would have no issues and at the same time other sessions would freeze until you disconnect/reconnect, or the latency was so bad the RDP wasn't usable at all. All connections were through VPN (both site to site and web VPN). It's believed the UDP setting/traffic was interfering even though all packets & traffic looked good through the VPN and there wasn't anything blocking the UDP traffic. The issue was resolved by turning off UDP on Client through group policy.

On PC: Open the RUN dialog > gpedit.msc > Administrative Templates > Windows Comonents > Remote Desktop Services > Remote Desktop Connection Client > Turn Off UDP on Client
and enable it and add any notes you wish.

On Server/Domain Controller go to GP Managment: Administrative Tools > Group Policy Management and create a group policy under the Domain
Please sign in to rate this answer.
Christian Feuerstein 26 Reputation points

2022-12-01T14:06:06.843+00:00

This seems for me more like some trouble with Network.
Either you're working with an IPS and your UDPFloodProtection is going nuts or your Network is unstable.
I'd recommend you to have a more detailed look in this issue since UDP is making things really smooth.
Cheers

Master Control 1 Reputation point

2022-12-05T15:01:24.493+00:00

@Christian Feuerstein
Network issue was worthy of looking into and was ruled out. IPS doesn't discriminate between MS Server 2012 and 2019 or 2022 especially when there isn't anything like an ACL to block it. We have no issues with the MS Server 2012 (R2) in the same environment. Throughput & bandwidth were also looked at. At some point even the application became suspect and then it was quickly ruled out without looking into it. Antivirus was ruled out. Our conclusion was MS Servers or PC's have a bug which needs to be fixed either on host PC OS, Server OS or both. Server VM's were also checked for CPU & RAM resources and still had issue after adding more RAM. RDP was running as expected by connecting exact same way from PC's to PC's, PC's to other MS 2012 (R2) Servers, and 2012 Servers to 2012 Servers in the network. The testing from 2019 & 2022 Servers to 2012 Servers in network were ruled as inconclusive tests even though the RDP sessions worked as expected because of weird & wonky and inconsistent behavior of the overall issue. (Sometimes sessions would just work but when they didn't the behavior too inconsistent to map out.) The Server to Server RDP sessions were only performed with 1 session at a time. It would have been better to do with numerous other sessions going on at the same time to the 2019 & 2022 Servers to show a true load during an average day of work, which for us is approx. 3-5 sessions but no more than 7 sessions to the Server.

Master Control 1 Reputation point

2022-12-05T15:22:28.59+00:00

@Christian Feuerstein
In the beginning, a Network issue was worthy of looking into and was ruled out. IPS doesn't discriminate between MS Server 2012 and 2019 or 2022 especially when there isn't anything like an ACL to block it. We have no issues with the MS Server 2012 (R2) in the same environment. Throughput & bandwidth were also looked at. At some point even the application became suspect and then it was quickly ruled out without looking into it. Antivirus was ruled out. Our conclusion was MS Servers or PC's have a bug which needs to be fixed either on host PC OS, Server OS or both. Server VM's were also checked for CPU & RAM resources and still had issue after adding more RAM. RDP was running as expected by connecting exact same way from PC's to PC's, PC's to other MS 2012 (R2) Servers, and 2012 Servers to 2012 Servers in the network. The testing from 2019 & 2022 Servers to 2012 Servers in network were ruled as inconclusive tests even though the RDP sessions worked as expected because of weird & wonky and inconsistent behavior of the overall issue. (Sometimes sessions would just work but when they didn't the behavior too inconsistent to map out.) The Server to Server RDP sessions were only performed with 1 session at a time. It would have been better to do with numerous other sessions going on at the same time to the 2019 & 2022 Servers to show a true load during an average day of work, which for us is approx. 3-5 sessions but no more than 7 sessions to the Server.
Sign in to comment