WSASend with overllaped sockets behavior

Question

Hello,

It would be nice if someone with knowledge of Windows internals could provide me with some more detailed info of how WSASend works in overlapped mode. Especially how the buffers are being handled in the kernel. The documentation is not very clear on that:

A pointer to an array of WSABUF structures. Each WSABUF structure contains a pointer to a buffer and the length, in bytes, of the buffer. For a Winsock application, once the WSASend function is called, the system owns these buffers and the application may not access them. This array must remain valid for the duration of the send operation.

Instinctually I would assume that the buffers should not be touched until I get completion notification - I am using IOCP.
However, I tried to prove that and it did not come out as expected.
I sent ~1.5GB buffer and as soon as WSASend returned I changed the last 4K of the buffer. The remote party received the original buffer values.
Then I sent 2662400 buffers of 1K each and soon as WSASend returned I changed the last buffer. Again the remote party received the original values in the all the buffers.

I doubt that WSASend copies all the data to the network provider before returning. I suspect, it is actually marking the pages for the buffers as copy on write.

Would be great to get an informed answer on this.

Thank you,
Bogdan

Answer 1

Hello @Bogdan Vasilescu ,

The array of WSABUF structures pointed to by the lpBuffers parameter is transient. If this operation completes in an overlapped manner, it is the service provider's responsibility to capture these WSABUF structures before returning from this call. This enables applications to build stack-based WSABUF arrays.

As for the content of the buf pointer points to, copy or not, it depends on the service provider. The general rule when dealing with buffers and asynchronous IO is that you shouldn’t do any modifying/freeing of the buffer until the asynchronous IO has completed.

What you has been doing so far seems correct.

Thank you!

---

If the answer is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

Answer 2

Hi @Rita Han - MSFT ,

Thank you very much for taking the time to respond.
We have a fairly complex application that uses WSASend in several places. During a review, I discovered that some parts of the application are deallocating the buffers after we receive the completion event [or if WSASend fails, etc]. However, some other places of the application use buffers 'allocated' on the stack or buffers for which the lifetime is controlled by containers on the stack - objects like std::vector 'allocated' on the stack.
The code is in the field for more than 10 years and was also stressed under heavy load. We never noticed any network related issues.

• I need to decide if the former approach is wasting resources by making an extra copy of these buffers or if the later approach is a bug waiting to happen that we need to fix.

Once we send the buffers to WSASend we do not need to access them anymore. So if I can just 'give' them to the OS it would be the easiest and better solution for us.
As you mentioned the documentation says: "may not" ... I am wonder if this was on purpose or the intention was to use "must not". Also the "system owns these buffers" formula is a little bit confusing.

Please let me know if you need additional info.

Thank you,
Bogdan