Nagle’s Algorithm is Not Friendly towards Small Requests
We had recommended in a forum post about how turning off Nagling can greatly improve throughput for Table Inserts and Updates. We have also been seeing a lot of improvement when application deals with small sized messages (http message size < 1460 bytes). So what is Nagling? Nagling is a TCP optimization on the sender and it is designed to reduce network congestion by coalescing small send requests into larger TCP segments. This is achieved by holding back small segments either until TCP has enough data to transmit a full sized segment or until all outstanding data has been acknowledged by the receiver
However Nagling interacts poorly with TCP Delayed ACKs, which is a TCP optimization on the receiver. It is designed to reduce the number of acknowledgement packets by delaying the ACK for a short time. RFC 1122 states that the delay should not be more than 500ms and there should be an ACK for every second segment. Since the receiver delays the ACK and the sender waits for the ACK before transmitting a small segment, the data transfer can get stalled until the delayed ACK arrives.
Since a picture speaks a thousand words, we have provided a graph of Queue PUT latencies with Nagling ON and OFF. The test was written by Shuitao, an Engineer in Storage team. The test is run as a worker role accessing our storage account in the same geo location. It inserts messages which are 480 bytes in length. The results show that the average latency improves by more than 600% with Nagling turned off.
Figure 1 – Nagling ON (Default). The x-axis shows the queue requests over time for the time period sampled. The y-axis shows the end to end time from the client in milliseconds to process the queue request.
Figure 2 - Nagling OFF. The x-axis shows the queue requests over time for the time period sampled. The y-axis shows the end to end time from the client in milliseconds to process the queue request.
Compare Nagling on/off using Wireshark
Let us now look at a Wireshark/Netmon trace to clearly see the delay in transmitting small packets. For a Put Message, which is transmitted as a POST request, the following packets are exchanged with Nagling on and off.
Nagling turned ON:
Frame 25 (349 bytes on wire, 349 bytes captured)
Hypertext Transfer Protocol
Time 2010-06-14 14:19:27.160472
Source 192.168.100.1
Destination 94.245.114.15
Protocol Info POST https://storagetest.queue.core.windows.net/container/messages?timeout=90
Description: Packet 1 from client to server.
The initial packet is 349 bytes and it contains just the request URI and
headers that needs to be sent for the POST request.
The message content (of length 737 bytes) is not part of this segment.
Frame 26 (60 bytes on wire, 60 bytes captured)
Time 2010-06-14 14:19:27.323342
Source 94.245.114.15
Destination 192.168.100.1
Protocol Info http > 53806 [ACK] Seq=296 Ack=653 Win=64883 Len=0
Description: Response from server to client.
Note the delayed ACK is received after 163ms.
Frame 27 (791 bytes on wire, 791 bytes captured)
Time 2010-06-14 14:19:27.323389
Source 192.168.100.1
Destination 94.245.114.15
Protocol Info Continuation or non-HTTP traffic
Description: Packet 2 from client to server.
This segment contains the payload for the Put Message HTTP request and
it was initially held back waiting for the server side ACK,
because Nagling was enabled and the payload size was less than
a full sized TCP segment (1460 bytes). However, once the delayed ACK is received above,
the segment is then transmitted.
Frame 28 (360 bytes on wire, 360 bytes captured)
Hypertext Transfer Protocol
Time 2010-06-14 14:19:27.365973
Source 94.245.114.15
Destination 192.168.100.1
Protocol Info HTTP/1.1 201 Created
Description: Response from server to client.
We receive success from the server for the POST request. HTTP/1.1
Now the same request is sent with Nagling turned off.
Frame 9 (349 bytes on wire, 349 bytes captured)
Hypertext Transfer Protocol
Time 2010-06-14 14:24:12.346146
Source 192.168.100.1
Destination 94.245.114.15
Protocol Info POST https://storagetest.queue.core.windows.net/container/messages?timeout=90
Description: Packet 1 from client to server.
The initial segment is 349 bytes and it contains just the request URI and
headers that need to be sent for the POST request.
The message content (of length 737 bytes) is not part of this segment.
Frame 10 (791 bytes on wire, 791 bytes captured)
Time 2010-06-14 14:24:12.348661
Source 192.168.100.1
Destination 94.245.114.15
Protocol Info Continuation or non-HTTP traffic
Description: Packet 2 from client to server.
This segment contains the payload for the PUT Message HTTP request and
it is sent immediately despite the size being less than a full segment (< 1460 bytes)
since Nagling is turned off. Since Nagling is off it does not have to wait for the delayed ACK.
Frame 11 (60 bytes on wire, 60 bytes captured)
Time 2010-06-14 14:24:12.349052
Source 94.245.114.15
Destination 192.168.100.1
Protocol Info http > 53811 [ACK] Seq=296 Ack=1390 Win=65535 Len=0
Description: Response from server to client.
ACK is sent by the server for frame 9 and 10 (packet 1 and 2).
Frame 12 (360 bytes on wire, 360 bytes captured)
Hypertext Transfer Protocol
Time 2010-06-14 14:24:12.395271
Source 94.245.114.15
Destination 192.168.100.1
Protocol Info HTTP/1.1 201 Created
Description: Response from server to client.
We receive success from the server for the POST request. HTTP/1.1
How to turn off Nagling?
Since Nagling is on by default, the way to turn this off is by resetting the flag in ServicePointManager. The ServicePointManager is a .NET class that allows you to manage ServicePoint where each ServicePoint provides HTTP connection management. ServicePointManager also allows you to control settings like maximum connections, Expect 100, and Nagle for all ServicePoint instances. Therefore, if you want to turn Nagle off for just tables or just queues or just blobs in your application, you need to turn it off for the specific ServicePoint object in the ServicePointManager. Here is a code example for turning Nagle off for just the Queue and Table ServicePoints, but not Blob:
// cxnString = "DefaultEndpointsProtocol=http;AccountName=myaccount;AccountKey=mykey"
CloudStorageAccount account = CloudStorageAccount.Parse(cxnString);
ServicePoint tableServicePoint = ServicePointManager.FindServicePoint(account.TableEndpoint);
tableServicePoint.UseNagleAlgorithm = false;
ServicePoint queueServicePoint = ServicePointManager.FindServicePoint(account.QueueEndpoint);
queueServicePoint.UseNagleAlgorithm = false;
If you instead want to set it for all of the service points on a given role (all blob, table and queue requests) you can just reset it at the very start of your application process by executing the following:
// This sets it globally for all new ServicePoints
ServicePointManager.UseNagleAlgorithm = false;
Note, this has to be set for the role (process) before any calls to blob, table and queue are done for the setting to be applied. In addition, this setting will only be applied to the process that executes it (it does not affect other processes) running inside the same VM instance.
Turning Nagle off should be considered for Table and Queue (and any protocol that deals with small sized messages). For large packet segments, Nagling does not have an impact since the segments will form a full packet and will not be withheld. But as always, we suggest that you test it for your data before turning Nagle off in production.
Jai Haridas
Comments
Anonymous
June 25, 2010
Great information. I assume turning Nagle off works from configuration settings as well, as you suggested in the original post. Is that still the case, since you don't mention it here?Anonymous
June 25, 2010
Yes, nagling can be turned off using the config setting as described in social.msdn.microsoft.com/.../d84ba34b-b0e0-4961-a167-bbe7618beb83. However, the above code allows you to turn it off per ServicePoint.Anonymous
June 27, 2010
Does turning off Nagling improve throughput for Selects?Anonymous
June 28, 2010
Hi Roman, A query if issued as a batch (which is sent as POST HTTP request), will have a payload and impacted by delayed ACK. However, a simple query which is sent as GET HTTP request, will not be impacted since it will not have a payload.Anonymous
June 28, 2010
If both Nagle and Delayed ACK work on TCP level, how they can differentiate GET and POST requests?Anonymous
June 28, 2010
Got it. With GET server sends result after recieving the first packet; with POST server waits for the second packet, but client doesn't send it until ACK is recieved.