Windows API - Win32
A core set of Windows application programming interfaces (APIs) for desktop and server applications. Previously known as Win32 API.
2,617 questions
CancelIoEx spuriously fails with ERROR_NOT_FOUND when there is actually pending IO
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(153.6, 10%, 24%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EI%3C/text%3E%3C/svg%3E)
Ilia -
1
Reputation point
Hi! This ^^ leads to "access violation executing location" exception on TpIo thread when the pending IO operation really completes:
00007ff7c000014b() Unknown No symbols loaded.
> kernel32.dll!BasepTpIoCallback() Unknown Symbols loaded.
ntdll.dll!TppIopExecuteCallback() Unknown Symbols loaded.
ntdll.dll!TppWorkerThread() Unknown Symbols loaded.
kernel32.dll!BaseThreadInitThunk() Unknown Symbols loaded.
ntdll.dll!RtlUserThreadStart() Unknown Symbols loaded.
The crash happens in the following code, when it calls _guard_dispatch_icall_nop and then jumps to the address taken from the freed OVERLAPPED*:
BasepTpIoCallback:
00007FFB30B1BAC0 48 89 5C 24 08 mov qword ptr [rsp+8],rbx
00007FFB30B1BAC5 48 89 6C 24 10 mov qword ptr [rsp+10h],rbp
00007FFB30B1BACA 48 89 74 24 18 mov qword ptr [rsp+18h],rsi
00007FFB30B1BACF 57 push rdi
00007FFB30B1BAD0 48 83 EC 40 sub rsp,40h
00007FFB30B1BAD4 33 C0 xor eax,eax
00007FFB30B1BAD6 49 8B D9 mov rbx,r9
00007FFB30B1BAD9 49 8B F0 mov rsi,r8
00007FFB30B1BADC 48 8B FA mov rdi,rdx
00007FFB30B1BADF 48 8B E9 mov rbp,rcx
00007FFB30B1BAE2 41 39 01 cmp dword ptr [r9],eax
00007FFB30B1BAE5 0F 8C CD 82 01 00 jl CreateToolhelp32Snapshot+0D9C8h (07FFB30B33DB8h)
00007FFB30B1BAEB 48 8B 4C 24 70 mov rcx,qword ptr [rsp+70h]
00007FFB30B1BAF0 44 8B C8 mov r9d,eax
00007FFB30B1BAF3 48 8B 57 18 mov rdx,qword ptr [rdi+18h]
00007FFB30B1BAF7 4C 8B C6 mov r8,rsi
00007FFB30B1BAFA 48 8B 07 mov rax,qword ptr [rdi]
00007FFB30B1BAFD 48 89 4C 24 28 mov qword ptr [rsp+28h],rcx
00007FFB30B1BB02 48 8B 4B 08 mov rcx,qword ptr [rbx+8]
00007FFB30B1BB06 48 89 4C 24 20 mov qword ptr [rsp+20h],rcx
00007FFB30B1BB0B 48 8B CD mov rcx,rbp
00007FFB30B1BB0E FF 15 04 77 06 00 call qword ptr [__guard_dispatch_icall_fptr (07FFB30B83218h)]
-> 00007FFB30B1BB14 48 8B 5C 24 50 mov rbx,qword ptr [rsp+50h]
00007FFB30B1BB19 48 8B 6C 24 58 mov rbp,qword ptr [rsp+58h]
00007FFB30B1BB1E 48 8B 74 24 60 mov rsi,qword ptr [rsp+60h]
00007FFB30B1BB23 48 83 C4 40 add rsp,40h
00007FFB30B1BB27 5F pop rdi
AFAIK there are no docs or examples that prohibit freeing it after ERROR_NOT_FOUND. https://learn.microsoft.com/en-us/windows/win32/fileio/cancelioex-func just says if the function succeeds I "*must not free or reuse the OVERLAPPED structure associated with the canceled I/O operations until they have completed*", but there is no such requirement in case of ERROR_NOT_FOUND.
The situation is similar in other places, but they also say nothing about ERROR_NOT_FOUND: this page https://learn.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o says "*do not deallocate or modify the OVERLAPPED structure or the data buffer until all asynchronous I/O operations to the file object have been completed*", and this page https://learn.microsoft.com/en-us/windows/win32/fileio/canceling-pending-i-o-operations says "*once the I/O operation completes (either successfully or with a canceled status) then the overlapped structure is no longer in use by the system and can be reused*".
In addition, Canceling Asynchronous I/O example (https://learn.microsoft.com/en-us/windows/win32/fileio/canceling-pending-i-o-operations) doesn't wait operation to complete when CancelIoEx() returns ERROR_NOT_FOUND:
result = CancelIoEx( hFile, lpOverlapped );
*pbCancelCalled = TRUE;
if (result == TRUE || GetLastError() != ERROR_NOT_FOUND)
{
// Wait for the I/O subsystem to acknowledge our cancellation.
// Depending on the timing of the calls, the I/O might complete with a
// cancellation status, or it might complete normally (if the ReadFile was
// in the process of completing at the time CancelIoEx was called, or if
// the device does not support cancellation).
// This call specifies TRUE for the bWait parameter, which will block
// until the I/O either completes or is canceled, thus resuming execution,
// provided the underlying device driver and associated hardware are functioning
// properly. If there is a problem with the driver it is better to stop
// responding here than to try to continue while masking the problem.
result = GetOverlappedResult( hFile, lpOverlapped, lpNumberOfBytesRead, TRUE );
// ToDo: check result and log errors.
}
Regarding TpIo, I use WaitForThreadpoolIoCallbacks() with fCancelPendingCallbacks=TRUE and there is a note (https://learn.microsoft.com/en-us/windows/win32/api/threadpoolapiset/nf-threadpoolapiset-waitforthreadpooliocallbacks) that "*... only queued callbacks are canceled. Pending I/O requests are not canceled. Therefore, the caller should call GetOverlappedResult for the OVERLAPPED structure to check whether the I/O operation has completed before freeing the structure. ... Be careful not to free the OVERLAPPED structure while I/O requests are still pending; use GetOverlappedResult to determine the status of the I/O operation and wait for the operation to complete*", but in my case, firstly I do CancelIoEx() which returns ERROR_NOT_FOUND, and only then call WaitForThreadpoolIoCallbacks() function, so there should be no pending IO requests at this moment.
Thus, all of this looks like CancelIoEx() erroneously returns ERROR_NOT_FOUND when there is actually a pending IO request.
Here is my program (x64/Release, VS2017, Windows 10 Pro, version 2004, build 19041.508):
#undef NDEBUG
#define CHECK assert
#include <assert.h>
#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <string>
#include <windows.h>
static void pipe(HANDLE& hRead, HANDLE& hWrite) {
GUID guid;
CHECK(CoCreateGuid(&guid) == S_OK);
std::string name = "\\\\.\\pipe\\" + std::to_string(guid.Data1) + std::to_string(guid.Data2) + std::to_string(guid.Data3) + std::to_string(*(uint64_t*)&guid.Data4[0]);
SECURITY_ATTRIBUTES sa = {};
sa.nLength = sizeof sa;
sa.bInheritHandle = TRUE;
hRead = CreateNamedPipeA(name.c_str(), PIPE_ACCESS_INBOUND | FILE_FLAG_FIRST_PIPE_INSTANCE | FILE_FLAG_OVERLAPPED, PIPE_TYPE_BYTE, 1, 4096, 4096, 0, &sa);
CHECK(hRead != INVALID_HANDLE_VALUE);
CHECK(SetHandleInformation(hRead, HANDLE_FLAG_INHERIT, 0));
hWrite = CreateFileA(name.c_str(), FILE_WRITE_DATA | SYNCHRONIZE, 0, &sa, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
CHECK(hWrite != INVALID_HANDLE_VALUE);
}
void WINAPI IoCompletionCallback(PTP_CALLBACK_INSTANCE Instance, void* Context, void* Overlapped, ULONG IoResult, ULONG_PTR NumberOfBytesTransferred, PTP_IO Io)
{
if (IoResult != ERROR_BROKEN_PIPE)
if (IoResult != ERROR_OPERATION_ABORTED)
printf("IoCompletionCallback err = %lu\n", IoResult);
}
int main(int argc, char* argv[])
{
int rc;
if (argc >= 2 && strcmp(argv[1], "--wait-and-exit") == 0)
{
Sleep(1);
return 0;
}
srand(time(NULL));
for (int i = 0; i < 100000; ++i)
{
HANDLE hPipeRead, hChildStdOut;
pipe(hPipeRead, hChildStdOut);
// start read aio
#define BUFFER_SIZE 1048576
char* buffer = new char[BUFFER_SIZE];
OVERLAPPED* overlapped = new OVERLAPPED{};
printf("%d: ovlp %p\n", i, overlapped);
PTP_IO tpio = CreateThreadpoolIo(hPipeRead, IoCompletionCallback, NULL, NULL);
CHECK(tpio != NULL);
StartThreadpoolIo(tpio);
rc = ReadFile(hPipeRead, buffer, BUFFER_SIZE, NULL, overlapped);
CHECK(!rc && GetLastError() == ERROR_IO_PENDING);
// start process and wait for it to exit
HANDLE hChildProcess;
{
std::string cmdl = std::string(argv[0]) + " --wait-and-exit";
STARTUPINFO si = {};
si.cb = sizeof si;
si.dwFlags |= STARTF_USESTDHANDLES;
si.hStdOutput = hChildStdOut;
PROCESS_INFORMATION pi;
CHECK(CreateProcessA(NULL, (char*)cmdl.data(), NULL, NULL, TRUE, 0, NULL, NULL, &si, &pi));
CloseHandle(hChildStdOut);
CloseHandle(pi.hThread);
hChildProcess = pi.hProcess;
}
CHECK(WaitForSingleObject(hChildProcess, INFINITE) == WAIT_OBJECT_0);
// cancel read aio
rc = CancelIoEx(hPipeRead, overlapped);
if (!rc)
{
if (GetLastError() != ERROR_NOT_FOUND)
printf("CancelIoEx = %d, err = %lu\n", rc, GetLastError()), exit(EXIT_FAILURE);
}
else
{
DWORD dwDummy = 0;
rc = GetOverlappedResult(hPipeRead, overlapped, &dwDummy, TRUE);
if (rc || GetLastError() != ERROR_OPERATION_ABORTED)
printf("GetOverlappedResult = %d, err = %lu\n", rc, GetLastError()), exit(EXIT_FAILURE);
}
// verify
DWORD dwDummy = 0;
rc = GetOverlappedResult(hPipeRead, overlapped, &dwDummy, FALSE);
if (rc || GetLastError() != ERROR_BROKEN_PIPE)
printf("GetOverlappedResult = %d, err = %lu\n", rc, GetLastError()), fflush(stdout); // here!
// cleanup
CloseHandle(hChildProcess);
WaitForThreadpoolIoCallbacks(tpio, TRUE);
CloseHandle(hPipeRead); // should be closed before CloseThreadpoolIo
CloseThreadpoolIo(tpio);
delete overlapped;
delete[] buffer;
}
getchar();
return 0;
}
Output:
...
880: ovlp 0000014CDA205CA0
881: ovlp 0000014CDA205A00
882: ovlp 0000014CDA205BE0
883: ovlp 0000014CDA205A30
884: ovlp 0000014CDA205F70
885: ovlp 0000014CDA2059D0
886: ovlp 0000014CDA205CD0
887: ovlp 0000014CDA2059A0
888: ovlp 0000014CDA205BB0
889: ovlp 0000014CDA205B50
890: ovlp 0000014CDA205F40
891: ovlp 0000014CDA205910
CHECK GetOverlappedResult = 0, err = 996
892: ovlp 0000014CDA2058B0
Debugger:
Watch 1:
Name Value
*(void**)0x07FFB30B83218 0x00007ffb30b263e0 {kernel32.dll!_guard_dispatch_icall_nop} void *
*(void**)0x0000014cda205910 0x00007ff7c000014b void *
Threads:
Not Flagged > 0x00009028 0x00 Worker Thread ntdll.dll!TppWorkerThread 00007ff7c000014b
00007ff7c000014b()
kernel32.dll!BasepTpIoCallback()
ntdll.dll!TppIopExecuteCallback()
ntdll.dll!TppWorkerThread()
kernel32.dll!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()
Not Flagged 0x0000475C 0x00 Main Thread Main Thread [Inline Frame] cancelioex_test.exe!globals::destroy
ntdll.dll!NtWaitForAlertByThreadId()
ntdll.dll!TppBarrierAdjust()
ntdll.dll!TpWaitForIoCompletion()
[Inline Frame] cancelioex_test.exe!globals::destroy() Line 36
cancelioex_test.exe!main(int argc, char * * argv) Line 162
[Inline Frame] cancelioex_test.exe!invoke_main() Line 78
cancelioex_test.exe!__scrt_common_main_seh() Line 283
kernel32.dll!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()
Not Flagged 0x00008CE8 0x00 Worker Thread ntdll.dll!TppWorkerThread ntdll.dll!NtWaitForWorkViaWorkerFactory
ntdll.dll!NtWaitForWorkViaWorkerFactory()
ntdll.dll!TppWorkerThread()
kernel32.dll!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()
Not Flagged 0x0000717C 0x00 Worker Thread ntdll.dll!TppWorkerThread ntdll.dll!NtWaitForWorkViaWorkerFactory
ntdll.dll!NtWaitForWorkViaWorkerFactory()
ntdll.dll!TppWorkerThread()
kernel32.dll!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()
Not Flagged 0x000052E8 0x00 Worker Thread ntdll.dll!TppWorkerThread ntdll.dll!NtWaitForWorkViaWorkerFactory
ntdll.dll!NtWaitForWorkViaWorkerFactory()
ntdll.dll!TppWorkerThread()
kernel32.dll!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()
Source code: 30023-source1cpp.txt
Feedback Hub: https://aka.ms/AA9v13c
{count} votes
-
Drake Wu - MSFT 991 Reputation points2020-10-05T08:41:33.23+00:00 Hi @Ilia - , I can reproduce this issue with your sample, it seems that the
ReadFilewas in the process of completing at the timeCancelIoExwas called. And theIoCompletionCallbackhasn't been called beforeGetOverlappedResult:
A workaround is to set the parameter
bWaitofGetOverlappedResultto TRUE to wait the operation complete. -
-
Drake Wu - MSFT 991 Reputation points
2020-10-07T02:18:17.783+00:00 I have submitted this case internally and will confirm this issue with the relevant engineers. Any progress will update to you.
-
Drake Wu - MSFT 991 Reputation points
2020-11-20T05:58:17.233+00:00 Hi, @Ilia - We’ve concluded our investigation into this issue and determined that the behavior is by design. This is how it’s supposed to work. The workaround of setting the
bWaitparameter ofGetOverlappedResulttoTRUEis the best way to handle it. -
Ilia - 1 Reputation point
2020-11-20T13:38:09.09+00:00 @Drake Wu - MSFT Thank you for investigation. But in this case, the example and the docs I referred above (and others, if any) must be updated, because at the moment they all imply a different behavior in the case of
ERROR_NOT_FOUND.Besides, it's still unclear what to do with those projects that copied the wrong pattern of using asynchronous I/O.
-
Drake Wu - MSFT 991 Reputation points
2020-11-23T06:47:13.457+00:00 Hi @Ilia - I have submit a request internally to add note and description of this situation. Thanks for your suggestion.
Sign in to comment
Sign in to answer