TSO

Another gotcha I've come across in the past is called TCP Segmentation Offload (known as TSO), which is used for outbound, egress traffic. TSO is well-intentioned and is designed to improve the throughput performance of a NIC while helping to cut down on the CPU hit for processing high levels of traffic.

The hardware helps to introduce a buffer that can then have its contents split up into segments. In short, TSO has control and may change how your TCP traffic flows. This feature works well until something breaks its functionality and you need to disable it quick smart. You can disable TSO with:

# ethtool -K eth0 tso off

Conversely, if you want to re-enable it, you can use the command:

# ethtool -K eth0 tso on

But, there's more. In the following command:

# ethtool -K eth0 tx off   sg off tso off

the tx relates to pause functionality for transmitted, egress data. The sg stands for scatter-gather. From what I can tell, scatter-gather is the act of pulling together data from different places and reformatting it so that you can output it as one data stream.

For a network card to support TSO, both checksumming and scatter-gather functionality must be present. Switching off the ability to pause traffic, as well as sg and TSO, is a good starting point for debugging. You have ufo (UDP Fragmentation) and gso (Generic Segmentation) options as well.

Incidentally, TSO's counterpart for ingress traffic is something called LRO (Large Receive Offload), which uses a very similar premise and the lro moniker as its syntax. Some reports say using LRO on Linux routers is a very bad idea that will cause significant performance headaches, so watch out with LRO if you're forwarding traffic.

Flow Control

Another common networking issue is flow control. In its first incarnation, flow control was defined in IEEE 802.3x, and later in IEEE 802.1Qbb, which describes "priority-based flow control."

Flow control works clevely by offering a helping hand to the part of the network link that has too much work to do. When the network is struggling, flow control allows a brief pause so that traffic can catch up. Unfortunately, in some situations, overall network speed can be severely affected if the clever flow control misbehaves.

One such nightmare scenario might be using Gigabit switches, Gigabit servers, and 100Mbps client machines. Badly behaving flow control might force all devices, Gigabit or otherwise, all the way down to 100Mbps.

You'll need to have a network card that supports pause functionality to adjust these settings. The most basic type of pause functionality lets you pause all traffic on a link; a newer version also lets you prioritize per traffic class. You can query your card with the -a option, which is the same as --show-pause:

# ethtool -a eth0
Pause parameters for eth0:
Autonegotiate:  on
RX:             on
TX:             on

For this NIC, all is well with hardware support.

You can disable this sometimes-pesky pause functionality with the following syntax:

# ethtool -a|--pause devname   [rx on|off] [tx on|off]   [autoneg on|off]

In other words, an example to switch off both ingress and egress "pause" functionality might look like the following:

# ethtool --pause eth0   rx off tx off autoneg off

Some users contend that making sure the auto-negotiation is switched off will ensure your settings are applied correctly. You can simply run:

# ethtool -a eth0

to check whether your new settings have taken effect.

Unconscious

At some point in the future, you'll be sitting impatiently, blaming a piece of software for your woes and the resulting bad mood your boss is in. Lo and behold the issue might be a network problem you can diagnose with ethtool.

Just remember to run through your "flow control" settings, test TSO both on and off, and force your speed and duplex settings. Fire up a tool like ngrep to sniff the data as it traverses your links, and bear in mind the fun and games you can experience with auto-negotiation too.

See the box titled "Mii-tool" for a look at an alternative for older systems that might not support ethtool.

# mii-tool -lw

# mii-tool --force 10baseT-HD

# mii-tool --reset