SQL Server service crash
Question
Monday, July 11, 2011 9:03 PM
Whenever I run a simple select query from a Posgres 8.1.3.12 linked server, SQL Server 2008 R2 crashes with the following error:
Faulting application name: sqlservr.exe, version: 2009.100.1600.1, time stamp: 0x4bb6b40b
Faulting module name: ntdll.dll, version: 6.1.7601.17514, time stamp: 0x4ce7c8f9
Exception code: 0xc0000374
Fault offset: 0x00000000000c40f2
Faulting process id: 0x1ce8
Faulting application start time: 0x01cc35c2bdeb4d00
Faulting application path: C:\Program Files\Microsoft SQL Server\MSSQL10_50\MSSQL\Binn\sqlservr.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: e3c2892a-a8b2-11e0-8cb7-001ec9fd5281
The crash dump looks like this:
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(12a8.2258): Unknown exception - code c0000374 (first/second chance not available)
ntdll!NtWaitForSingleObject+0xa:
00000000`76e1135a c3 ret
0:032> .ecxr
rax=000000008a657372 rbx=00000000c0000374 rcx=0000000016c9c590
rdx=0000000076efb450 rsi=0000000000000000 rdi=0000000076efb450
rip=0000000076e840f2 rsp=0000000016c9cba0 rbp=0000000000000000
r8=26517d1db35d05c4 r9=000000001c8a66cd r10=0000000000000000
r11=0000000000000286 r12=0000000000000008 r13=000000000062f520
r14=00000000005c7f43 r15=0000000000000000
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206
ntdll!RtlReportCriticalFailure+0x62:
00000000`76e840f2 eb00 jmp ntdll!RtlReportCriticalFailure+0x64 (00000000`76e840f4)
0:032> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
*** WARNING: Unable to verify checksum for psqlodbc35w.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for psqlodbc35w.dll -
*** ERROR: Symbol file could not be found. Defaulted to export symbols for msvcr80.dll -
GetPageUrlData failed, server returned HTTP status 404
FAULTING_IP:
ntdll!RtlReportCriticalFailure+62
00000000`76e840f2 eb00 jmp ntdll!RtlReportCriticalFailure+0x64 (00000000`76e840f4)
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000076e840f2 (ntdll!RtlReportCriticalFailure+0x0000000000000062)
ExceptionCode: c0000374
ExceptionFlags: 00000001
NumberParameters: 1
Parameter[0]: 0000000076efb450
DEFAULT_BUCKET_ID: STATUS_HEAP_CORRUPTION
PROCESS_NAME: sqlservr.exe
OVERLAPPED_MODULE: Address regions for 'XPStar' and 'sqlboot.dll' overlap
ERROR_CODE: (NTSTATUS) 0xc0000374 - A heap has been corrupted.
EXCEPTION_CODE: (NTSTATUS) 0xc0000374 - A heap has been corrupted.
EXCEPTION_PARAMETER1: 0000000076efb450
MOD_LIST: <ANALYSIS/>
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
FAULTING_THREAD: 0000000000002258
PRIMARY_PROBLEM_CLASS: STATUS_HEAP_CORRUPTION
BUGCHECK_STR: APPLICATION_FAULT_STATUS_HEAP_CORRUPTION
LAST_CONTROL_TRANSFER: from 0000000076e84736 to 0000000076e840f2
STACK_TEXT:
00000000`16c9cba0 00000000`76e84736 : 00000000`00000002 00000000`00000023 00000000`005ea200 00000000`00000003 : ntdll!RtlReportCriticalFailure+0x62
00000000`16c9cc70 00000000`76e85942 : 00000000`00530500 005c0002`00000003 00020000`00000000 00000000`04df360d : ntdll!RtlpReportHeapFailure+0x26
00000000`16c9cca0 00000000`76e875f4 : 00000000`003f0000 00000000`00000000 00000000`00000000 00000000`04df360d : ntdll!RtlpHeapHandleError+0x12
00000000`16c9ccd0 00000000`76e2dc8f : 00000000`005c7ed0 00000000`003f0000 00000000`005c7ee0 00000000`00000000 : ntdll!RtlpLogHeapFailure+0xa4
00000000`16c9cd00 000007fe`fdea10c4 : 00000000`005c7ee0 00000000`0000007b 00000000`00000000 00000000`00000000 : ntdll! ?? ::FNODOBFM::`string'+0x10c54
00000000`16c9cd80 00000000`04dfc137 : 00000000`000000dd 00000000`005c7ee0 00000000`00000000 00000000`00000001 : msvcrt!free+0x1c
00000000`16c9cdb0 00000000`04de0a59 : 00000000`04e2a0f8 00000000`0062f520 00000000`00000000 00000000`0062f520 : psqlodbc35w!forcelog+0x8b07
00000000`16c9d1f0 00000000`04e19d15 : 00000000`04e30000 00000000`00620001 00000000`0062f520 00000000`00000001 : psqlodbc35w!dconn_FDriverConnectProc+0x3589
00000000`16c9d270 000007fe`f5f41348 : 00000000`18e7f790 00000000`005c3fb0 00000000`005a8fd0 00000000`00000000 : psqlodbc35w!SQLExecute+0xb5
00000000`16c9d2a0 000007fe`e63ec8db : ffffffff`ffffffff 00000000`00000000 000007fe`e6371a98 00000000`16c9d490 : odbc32!SQLExecute+0x448
00000000`16c9d300 000007fe`e638eedd : 00000000`00000000 00000000`18e759c0 00000000`00000000 00000000`18e811b0 : msdasql!CUtlParam30::ExecuteWithParameters+0x52b
00000000`16c9d3b0 000007fe`e638dc47 : ffffffff`ffffffff 00000000`18e80f78 000007fe`e6371a98 00000000`00000000 : msdasql!CImpICommandText::ExecuteHelper+0x37d
00000000`16c9d430 000007fe`e64c460f : 00000000`18e5e500 00000000`00000000 00000000`16c9d628 00000000`00000000 : msdasql!CImpICommandText::Execute+0x13d7
00000000`16c9d5f0 000007fe`e646d65a : 00000104`00000000 00000000`c8b522be 00000000`00000000 00000000`00c04770 : oledb32!CCommandText::DoExecute+0x824
00000000`16c9d7f0 00000000`017315c3 : 00000000`00730240 00000000`00d4b0e4 00000000`00000000 00000000`74a168ee : oledb32!CCommandText::Execute+0xb93
00000000`16c9da20 00000000`016dee03 : 00000000`16c9db20 00000000`803861a0 00000000`007461a0 00000000`00000000 : sqlservr!CallProtectorImpl::CallWithSEH<IWrapInterface<ICommandText>::CallTraitsFpuPre,long,MethodCallBinder_5<long,ICommandText,long (__cdecl ICommand::*)(IUnknown * __ptr64,_GUID const & __ptr64,tagDBPARAMS * __ptr64,__int64 * __ptr64,IUnknown * __ptr64 * __ptr64) __ptr64,IUnknown * __ptr64,_GUID,tagDBPARAMS * __ptr64,__int64 * __ptr64,IUnknown * __ptr64 * __ptr64> const >+0x3f
00000000`16c9da70 00000000`016ca332 : 00000000`a20d42e0 00000000`996d9650 00000000`16c9dac0 00000000`16c9dac8 : sqlservr!CallProtectorImpl::CallExternalFull<IWrapInterface<ICommandText>::CallTraitsFpuPre,long,MethodCallBinder_5<long,ICommandText,long (__cdecl ICommand::*)(IUnknown * __ptr64,_GUID const & __ptr64,tagDBPARAMS * __ptr64,__int64 * __ptr64,IUnknown * __ptr64 * __ptr64) __ptr64,IUnknown * __ptr64,_GUID,tagDBPARAMS * __ptr64,__int64 * __ptr64,IUnknown * __ptr64 * __ptr64> const >+0xf7
00000000`16c9daf0 00000000`016454c1 : 00000000`00000000 00000000`00000000 00000000`803861a0 00000000`803861a0 : sqlservr!IWrapInterface<ICommandText>::HrCallExternal<MethodCallBinder_5<long,ICommandText,long (__cdecl ICommand::*)(IUnknown * __ptr64,_GUID const & __ptr64,tagDBPARAMS * __ptr64,__int64 * __ptr64,IUnknown * __ptr64 * __ptr64) __ptr64,IUnknown * __ptr64,_GUID,tagDBPARAMS * __ptr64,__int64 * __ptr64,IUnknown * __ptr64 * __ptr64> >+0x32
00000000`16c9db40 00000000`024dfa6e : 00000000`00000000 00000000`00000000 00000000`a187af80 00000000`996d9650 : sqlservr!CWrapICommandText::Execute+0x229
00000000`16c9dcf0 00000000`024de23d : 00000000`a187ad10 00000000`a187ad10 00000000`00000000 00000000`a187a780 : sqlservr!CQScanRmtQueryNew::CreateQueryRowset+0x62a
00000000`16c9e390 00000000`00ca00bc : 00000000`00000000 00000000`00000000 00000000`a187a780 00000000`996d9218 : sqlservr!CQScanRmtQueryNew::Open+0x65
00000000`16c9e480 00000000`00ca01ba : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`a187bc50 : sqlservr!CQueryScan::Startup+0x2d2
00000000`16c9e4d0 00000000`00c99a76 : 00000000`a187a140 00000000`00000000 00000000`996d93d0 00000000`00000000 : sqlservr!CXStmtQuery::SetupQueryScanAndExpression+0x2d4
00000000`16c9e530 00000000`00c9984b : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : sqlservr!CXStmtQuery::InitForExecute+0x3a
00000000`16c9e560 00000000`00c9a7e4 : 00000000`a187a140 00000000`996d93d0 00000000`a187a140 00000000`a1800140 : sqlservr!CXStmtQuery::ErsqExecuteQuery+0x602
00000000`16c9e6f0 00000000`00c59d59 : 00000000`00000000 00000000`a187a140 00000000`996d9418 00000000`996d9438 : sqlservr!CXStmtSelect::XretExecute+0x2dd
00000000`16c9e7c0 00000000`00c5a9b8 : 00000000`00000000 00000000`00d24912 00000000`803866c0 00000000`00000000 : sqlservr!CMsqlExecContext::ExecuteStmts<1,1>+0x377
00000000`16c9e8d0 00000000`00c5b30c : 00000000`a1800140 00000000`996d93d0 00000003`00000000 00000000`00000000 : sqlservr!CMsqlExecContext::FExecute+0x983
00000000`16c9ea50 00000000`00c5c1a6 : 00000000`00000000 00000000`80386100 00000000`00000000 00000000`00000000 : sqlservr!CSQLSource::Execute+0x7b2
00000000`16c9eb80 00000000`00ca5342 : 00000000`996d8fa0 00000000`83d426c0 00000000`00000000 00000000`00000000 : sqlservr!process_request+0x64b
00000000`16c9f1e0 00000000`00c0bbd8 : 00000000`00736080 00000000`00736080 00000000`803861a0 00000000`803861a0 : sqlservr!process_commands+0x4e5
00000000`16c9f3f0 00000000`00c0b8ba : ffffffff`00000000 00000000`00c0be73 00000000`00000000 00000000`803861a0 : sqlservr!SOS_Task::Param::Execute+0x12a
00000000`16c9f500 00000000`00c0b6ff : 00000000`00736080 00000000`00732988 00000000`00736080 00000000`803861a0 : sqlservr!SOS_Scheduler::RunTask+0x96
00000000`16c9f560 00000000`01128fb6 : 00000000`803861a0 00000000`803861a0 000007ff`fff774a8 00000000`00000000 : sqlservr!SOS_Scheduler::ProcessTasks+0x128
00000000`16c9f5d0 00000000`01129175 : 00000000`803861a0 000008b4`8a1d9285 000007ff`fff774a8 000007ff`fff774a8 : sqlservr!SchedulerManager::WorkerEntryPoint+0x2b6
00000000`16c9f6b0 00000000`01129839 : 00000000`00138250 00000000`00138250 00000000`00138268 00000000`00000200 : sqlservr!SystemThread::RunWorker+0xcc
00000000`16c9f6f0 00000000`01129502 : 00000000`00000000 000007ff`fff774a8 00000000`00138198 00000000`00000000 : sqlservr!SystemThreadDispatcher::ProcessWorker+0x2db
00000000`16c9f7a0 00000000`74a137d7 : 00000000`00699a60 000007ff`fff774a8 00000000`00699a60 00000000`00000000 : sqlservr!SchedulerManager::ThreadEntryPoint+0x173
00000000`16c9f840 00000000`74a13894 : 00000000`74ac95c0 00000000`00699a60 00000000`00000000 00000000`00000000 : msvcr80!endthreadex+0x47
00000000`16c9f870 00000000`76bb652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : msvcr80!endthreadex+0x104
00000000`16c9f8a0 00000000`76dec521 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`16c9f8d0 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d
FOLLOWUP_IP:
psqlodbc35w!forcelog+8b07
00000000`04dfc137 8a5c247f mov bl,byte ptr [rsp+7Fh]
SYMBOL_STACK_INDEX: 6
SYMBOL_NAME: psqlodbc35w!forcelog+8b07
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: psqlodbc35w
IMAGE_NAME: psqlodbc35w.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 4b4d62e1
STACK_COMMAND: ~32s; .ecxr ; kb
FAILURE_BUCKET_ID: STATUS_HEAP_CORRUPTION_c0000374_psqlodbc35w.dll!forcelog
BUCKET_ID: X64_APPLICATION_FAULT_STATUS_HEAP_CORRUPTION_psqlodbc35w!forcelog+8b07
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/sqlservr_exe/2009_100_1600_1/4bb6b40b/ntdll_dll/6_1_7601_17514/4ce7c8f9/c0000374/000c40f2.htm?Retriage=1
Followup: MachineOwner
Any help on this problem is greatly appreciated.
All replies (2)
Tuesday, July 12, 2011 2:52 AM ✅Answered
Seems to be the scenario of heap corruption. I have seen these kind of issues when we use linked server and normally we recommend to upgrade the drivers for the third party product. Also, Try running linked server out of process and check if that helps:
To run the Linked Server in “Out of Process“ mode:
- Create a new Linked Server (or delete and recreate).
- Click on the “Provider Options“.
- Uncheck the “Allow InProcess“ checkbox
if that doesn't help then upgrade the drivers for the third party product.
If above doesn't help, then you could enable gflag to detect heap corruption : - http://blogs.msdn.com/b/webdav_101/archive/2010/06/22/detecting-heap-corruption-using-gflags-and-dumps.aspx
It will generate a dump which will need analysis
Harsh Chawla Personal Blog:- http://blogs.msdn.com/batala Team Blog:- http://blogs.msdn.com/b/sqlserverfaq/
Wednesday, July 13, 2011 3:31 PM
Thanks!
I haven't had any more crashes since setting the linked server "out of process". I will continue to monitor.