The following is a summary of techniques to maximize TCP WAN
throughput.
TCP uses
what is called the “congestion window”, or CWND, to determine how many packets
can be sent at one time. The larger the congestion window size, the higher the throughput.
The TCP “slow start” and “congestion avoidance” algorithms determine
the size of the congestion window. The maximum congestion window is related to the amount
of buffer space that the kernel allocates for each socket. For each socket, there is a
default value for the buffer size, which can be changed by the program using a system
library call just before opening the socket. There is also a kernel enforced maximum
buffer size. The buffer size can be adjusted for both the send and receive ends of the
socket.
To get maximal throughput it is critical to use optimal TCP
send and receive socket buffer sizes for the link you are using. If the buffers are too
small, the TCP congestion window will never fully open up. If the receiver buffers are too
large, TCP flow control breaks and the sender can overrun the receiver, which will cause
the TCP window to shut down. This is likely to happen if the sending host is faster than
the receiving host. Overly large windows on the sending side is not a big problem as long
as you have excess memory.
The optimal buffer size is twice the bandwidth*delay
product of the link:
buffer size = 2 * bandwidth * delay
The ping program can be used to get the delay, and tools such as pathrate
to get the end-to-end capacity (the bandwidth
of the slowest hop in your path). Since ping gives the round trip time (RTT), this formula
can be used instead of the previous one:
buffer size = bandwidth * RTT.
For example, if your ping time is 50 ms, and the end-to-end
network consists of all 100 BT Ethernet and OC3 (155 Mbps), the TCP buffers should
be .05 sec * (100 Mbits / 8 bits) = 625 KBytes. (When in doubt, 10 MB/s is a good first
approximation for network bandwidth on ESnet/vBNS/Abilene-like networks).
There are 2 TCP settings you need to know about. The default
TCP send and receive buffer size, and the maximum TCP send and receive buffer size.
Note that most of today UNIX OS's by default have a maximum TCP buffer size of only 256
KB! For instructions on how to increase the maximum TCP buffer, see the
OS specific instructions for setting system defaults. You won't want to
increase your default buffer size greater than 128 KB, because this may adversely affect
LAN performance. Therefore you need to use the UNIX setsockopt
call in your sender and receiver to set the optimal buffer size for the link you
are using.