Some Multi-threaded terminal server applications crashes in Win 2008 R2

아티클
03/29/2010

In Win2008 R2 some Multi-threaded terminal server application may crashes with access violation in the test eax, eax cpu instruction with following symptoms. This issue is very intermittent.

2. You may find following 2 threads,

Executing a test instruction and causing AV.
Trying to change the protection of exactly same code memory page which the first thread is causing access violation ( Not waiting ).
If DEP for the process is turned off application is working fine.

Following are the analysis details,

The stored exception information can be accessed via .ecxr.

(428.b50): Access violation - code c0000005 (first/second chance not available)

eax=00000000 ebx=7740f85c ecx=74e92dd9 edx=00000000 esi=00000000 edi=00000001

eip=741c17cd esp=0018a4dc ebp=0018a508 iopl=0 nv up ei pl nz na pe nc

cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206

mswsock!SockWaitForSingleObject+0x3a:

741c17cd 85c0 test eax,eax

0:000> k100

ChildEBP RetAddr

0018a508 741c678c mswsock!SockWaitForSingleObject+0x3a

0018a5f4 75f04a20 mswsock!WSPSelect+0x3a6

*** ERROR: Symbol file could not be found. Defaulted to export symbols for DvZediRimac002.dll -

0018a674 4b6f447d ws2_32!select+0x494

WARNING: Stack unwind information not available. Following frames may be wrong.

0018e598 4b6e5a80 ThirdParty!ThirdParty +0xac

0018e6ac 774bfd6e ThirdParty!ThirdParty +0x2fc

0018e698 00000000 ntdll!RtlpValidateHeap+0x20

0:000> !analyze -v

*******************************************************************************

* *

* Exception Analysis *

* *

*******************************************************************************

Debugger CompCtrlDb Connection::Open failed 80004005

Debugger Dbgportaldb Connection::Open failed 80040e4d

Database Dbgportaldb not connected

FAULTING_IP:

mswsock!SockWaitForSingleObject+3a

741c17cd 85c0 test eax,eax

EXCEPTION_RECORD: ffffffff -- (.exr ffffffffffffffff)

ExceptionAddress: 741c17cd (mswsock!SockWaitForSingleObject+0x0000003a)

ExceptionCode: c0000005 (Access violation)

ExceptionFlags: 00000000

NumberParameters: 2

Parameter[0]: 00000008

Parameter[1]: 741c17cd

Attempt to execute non-executable address 741c17cd

PROCESS_NAME: KAREWE.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.

WRITE_ADDRESS: 741c17cd

BUGCHECK_STR: ACCESS_VIOLATION

LAST_CONTROL_TRANSFER: from 741c678c to 741c17cd

STACK_TEXT:

0018a508 741c678c 0000025c 00000264 00000001 mswsock!SockWaitForSingleObject+0x3a

0018a5f4 75f04a20 00000000 0018a6a0 00000000 mswsock!WSPSelect+0x3a6

0018a674 4b6f447d 00000000 0018a6a0 00000000 ws2_32!select+0x494

WARNING: Stack unwind information not available. Following frames may be wrong.

0018e598 4b6e5a80 0018e6fc 00000200 00000001 ThirdParty!ThirdParty +0xac

0018e6ac 774bfd6e 00000000 00000044 00310150 ThirdParty!ThirdParty +0x2fc

0018e698 00000000 02c30ac8 23040027 0000000d ntdll!RtlpValidateHeap+0x20

STACK_COMMAND: ~0s; .ecxr ; kb

PRIMARY_PROBLEM_CLASS: SOFTWARE_NX_FAULT_FALSE_POSITIVE

DEFAULT_BUCKET_ID: SOFTWARE_NX_FAULT_FALSE_POSITIVE

FAULTING_THREAD: 00000b50

FOLLOWUP_IP:

mswsock!SockWaitForSingleObject+3a

741c17cd 85c0 test eax,eax

SYMBOL_STACK_INDEX: 0

FOLLOWUP_NAME: wsstress

MODULE_NAME: mswsock

DEBUG_FLR_IMAGE_TIMESTAMP: 4a5bda77

SYMBOL_NAME: mswsock!SockWaitForSingleObject+3a

IMAGE_NAME: mswsock.dll

FAILURE_BUCKET_ID: ACCESS_VIOLATION_mswsock!SockWaitForSingleObject+3a

BUCKET_ID: ACCESS_VIOLATION_mswsock!SockWaitForSingleObject+3a

Followup: wsstress

---------

· The application is failing at

741c17cd 85c0 test eax,eax

due to access violation.

· I checked if any dll is tampered with

0:000> !for_each_module !chkimg @#ModuleName

found that the dll under question(mswsock) is not tampered.

Above instruction (test) don't access any memory location other than where EIP points.
So for the above instruction to generate an AV, the EIP should point to an address in a non executable page of memory.
Again from !address command we can see that this a code region and it is from a mapped dll which should ideally an executable page.

0:000> !address eip

TEB 7efdd000 in range 7efdb000 7efde000

TEB 7efda000 in range 7efd8000 7efdb000

ProcessParametrs 003213f0 in range 00320000 0038f000

Environment 00320810 in range 00320000 0038f000

741c0000 : 741c1000 - 00001000

Type 01000000 MEM_IMAGE

Protect 00000004 PAGE_READWRITE

State 00001000 MEM_COMMIT

Usage RegionUsageImage

FullPath C:\Windows\System32\mswsock.dll

On further debugging I found the following as well, which should be true in at least some cases of execution.

Only two threads in the application,

Ø Executing the test instruction and causing AV.

Ø Trying to change the protection of exactly same code memory page which the first thread is causing access violation.

0:000> ~

. 0 Id: 428.b50 Suspend: 1 Teb: 7efdd000 Unfrozen

1 Id: 428.900 Suspend: 1 Teb: 7efda000 Unfrozen

0:000> ~1s

eax=00000000 ebx=80000000 ecx=00000000 edx=00000000 esi=741c1098 edi=741c0000

eip=7740ffea esp=030afacc ebp=030afb08 iopl=0 nv up ei pl nz na po nc

cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202

ntdll!NtProtectVirtualMemory+0x12:

7740ffea 83c404 add esp,4

0:001> k100

ChildEBP RetAddr

030afacc 7416aa2c ntdll!NtProtectVirtualMemory+0x12

030afb08 7416ac22 TSAPPCMP!TsRedirectRegisteredImage+0xcf

030afb58 7416af58 TSAPPCMP!TsWalkProcessDlls+0xf5

030afb8c 4b740f0c TSAPPCMP!TLoadLibraryA+0x3a

WARNING: Stack unwind information not available. Following frames may be wrong.

030afbe8 4b73f737 ThirdParty!ThirdParty +0x38a9

030afc84 774236fa ThirdParty!ThirdParty +0x20d4

030afd78 4b6b83c3 ntdll!RtlpFreeHeap+0xb7a

030afde0 774236fa ThirdParty!ThirdParty +0x9b

030afddc 50000063 ntdll!RtlpFreeHeap+0xb7a

030afec4 774236fa 0x50000063

030affd4 77429d45 ntdll!RtlpFreeHeap+0xb7a

030affec 00000000 ntdll!_RtlUserThreadStart+0x1b

Looking at that stack based on following link and trying to sleuth the parameters, ( NtProtectVirtualMemory has FPO and is written in assembly so normal methods don't work )

https://undocumented.ntinternals.net/UserMode/Undocumented%20Functions/Memory%20Management/Virtual%20Memory/NtProtectVirtualMemory.html

NTSYSAPI

NTSTATUS

NTAPI

NtProtectVirtualMemory(

IN HANDLE ProcessHandle,

IN OUT PVOID *BaseAddress,

IN OUT PULONG NumberOfBytesToProtect,

IN ULONG NewAccessProtection,

OUT PULONG OldAccessProtection );

Picking up the function arguments from stack

0:001> dds esp

030afacc 7740ffea ntdll!NtProtectVirtualMemory+0x12

030afad0 7416aa2c TSAPPCMP!TsRedirectRegisteredImage+0xcf

030afad4 ffffffff

030afad8 030afafc

030afadc 030afb00

030afae0 00000004

030afae4 030afaf8

030afae8 774f020c ntdll!PebLdr+0xc

030afaec 0038af48

030afaf0 0038be08

030afaf4 00000004

030afaf8 00000020

030afafc 741c1098 mswsock!_imp__NtOpenKey

030afb00 00000004

030afb04 741c0000 mswsock!_imp__OutputDebugStringA <PERF> (mswsock+0x0)

030afb08 030afb58

030afb0c 7416ac22 TSAPPCMP!TsWalkProcessDlls+0xf5

030afb10 741e4818 mswsock!DNSAPI_NULL_THUNK_DATA_DLB+0x8

030afb14 00000000

030afb18 00000001

030afb1c 3c144767

030afb20 00000000

030afb24 4b7a203c DvZediRimac002!CWatchedProxyObj::`vftable'+0x30280

030afb28 00000000

030afb2c 001a0018

030afb30 74162684 TSAPPCMP!`string'

030afb34 030afb54

030afb38 741c00d8 mswsock!_imp__OutputDebugStringA <PERF> (mswsock+0xd8)

030afb3c 0038be08

030afb40 030afb1c

030afb44 00000000

030afb48 030afb7c

Unassembleing the call to NtProtectVirtualMemory.

0:001> ub 7416aa2c L10

TSAPPCMP!TsRedirectRegisteredImage+0xaa [d:\w7rtm\termsrv\tsappcmp\register.c @ 1368]:

7416aa07 59 pop ecx

7416aa08 59 pop ecx

7416aa09 e9c2000000 jmp TSAPPCMP!TsRedirectRegisteredImage+0x173 (7416aad0)

7416aa0e 6a04 push 4

7416aa10 58 pop eax

7416aa11 8d4df0 lea ecx,[ebp-10h]

7416aa14 51 push ecx

7416aa15 50 push eax

7416aa16 8945f8 mov dword ptr [ebp-8],eax

7416aa19 8d45f8 lea eax,[ebp-8]

7416aa1c 50 push eax

7416aa1d 8d45f4 lea eax,[ebp-0Ch]

7416aa20 50 push eax

7416aa21 6aff push 0FFFFFFFFh

7416aa23 8975f4 mov dword ptr [ebp-0Ch],esi

7416aa26 ff151c131674 call dword ptr [TSAPPCMP!_imp__NtProtectVirtualMemory (7416131c)]

What address is trying to change the protection.

0:001> dc 030afafc

030afafc 741c1098 00000004 741c0000 030afb58 ...t.......tX...

030afb0c 7416ac22 741e4818 00000000 00000001 "..t.H.t........

030afb1c 3c144767 00000000 4b7a203c 00000000 gG.<....< zK....

030afb2c 001a0018 74162684 030afb54 741c00d8 .....&.tT......t

030afb3c 0038be08 030afb1c 00000000 030afb7c ..8.........|...

030afb4c 7416e80a 4b084bdf 00000000 030afb8c ...t.K.K........

030afb5c 7416af58 3c1447b3 00000000 4b7a203c X..t.G.<....< zK

030afb6c 00000000 611a0000 030afb60 7499fa2e .......a`......t

Location being changed by NtProtectVirtualMemory.

0:001> !address 741c1098

TEB 7efdd000 in range 7efdb000 7efde000

TEB 7efda000 in range 7efd8000 7efdb000

ProcessParametrs 003213f0 in range 00320000 0038f000

Environment 00320810 in range 00320000 0038f000

741c0000 : 741c1000 - 00001000

Type 01000000 MEM_IMAGE

Protect 00000004 PAGE_READWRITE

State 00001000 MEM_COMMIT

Usage RegionUsageImage

FullPath C:\Windows\System32\mswsock.dll

Failing EIP in the other thread.

0:001> !address 741c17cd

TEB 7efdd000 in range 7efdb000 7efde000

TEB 7efda000 in range 7efd8000 7efdb000

ProcessParametrs 003213f0 in range 00320000 0038f000

Environment 00320810 in range 00320000 0038f000

741c0000 : 741c1000 - 00001000

Type 01000000 MEM_IMAGE

Protect 00000004 PAGE_READWRITE

State 00001000 MEM_COMMIT

Usage RegionUsageImage

FullPath C:\Windows\System32\mswsock.dll

Conclusion

This is an issue in TSAPPCMP.dll

Until the fix is release if you happen to run in to this issue following is the solution.

Set the following registry value:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server]

"IAT"=dword:00000001

다음을 통해 공유

Some Multi-threaded terminal server applications crashes in Win 2008 R2

추가 리소스