[Intel-wired-lan] e1000e echo request response latency and tcph->ack

[Intel-wired-lan] e1000e echo request response latency and tcph->ack

[Willem de Bruijn]

With RR tests on a pair of 82571EB I am seeing much lower latency with
TCP (40 usec) vs UDP (105 usec). But only for TCP with the ACK bit
set.

I observed the initial TCP vs UDP results with sockperf.  Digging into
this behavior by emulating the protocol headers with packet sockets
show that just toggling the ACK bit changes this behavior. I shared
the test program that does this on github [1].

Is this known behavior? Is there some fast path on the NIC triggered
only for TCP ACK packets? If so, then my real question is, is there a
way to make UDP packets take this fast path?

Output:

# emulate UDP
./pingpong_tcpudp -s 192.168.2.2 -d 192.168.2.1 -S ${smac} -D {$dmac} -i eth2 -u
mode: server
1. RTT: 140 usec
2. RTT: 108 usec
3. RTT: 106 usec
4. RTT: 105 usec
5. RTT: 106 usec
6. RTT: 105 usec
7. RTT: 106 usec
8. RTT: 105 usec
9. RTT: 106 usec

# emulate TCP
./pingpong_tcpudp -s 192.168.2.2 -d 192.168.2.1 -S ${smac} -D ${dmac} -i eth2
mode: server
1. RTT: 145 usec
2. RTT: 107 usec
3. RTT: 105 usec
4. RTT: 105 usec
5. RTT: 105 usec
6. RTT: 104 usec
7. RTT: 104 usec
8. RTT: 104 usec
9. RTT: 105 usec

# emulate TCP and set ACK bit
./pingpong_tcpudp -s 192.168.2.2 -d 192.168.2.1 -S ${smac} -D ${dmac} -i eth2 -a
mode: server
1. RTT: 79 usec
2. RTT: 41 usec
3. RTT: 41 usec
4. RTT: 41 usec
5. RTT: 39 usec
6. RTT: 39 usec
7. RTT: 39 usec
8. RTT: 39 usec
9. RTT: 40 usec


The hosts are connected through a switch, so there is some chance that
this delay is incurred there. It is unlikely, as other non-e1000e
devices do not show the same latency.

The machine is running Linux at current davem-net-next/master HEAD,
booted with idle=poll.

ethtool -i eth2
driver: e1000e
version: 3.2.6-k
firmware-version: 5.11-2

Interrupt moderation, RPS and RFS are disabled and the packets are
dropped to avoid ICMP responses:

modprobe e1000e
ip link set dev eth2 up
ip addr add 192.168.2.2/24 dev eth2
for dev in eth0 eth2; do ethtool -C $dev adaptive-rx off adaptive-tx
off rx-usecs 0 rx-frames 0 tx-usecs 0 tx-frames 0; done
echo 0 > /proc/sys/net/core/rps_sock_flow_entries
echo 0 > /sys/class/net/eth2/queues/rx-0/rps_cpus
echo 0 > /sys/class/net/eth2/queues/rx-0/rps_flow_cnt
iptables -A PREROUTING -t raw -p udp --dport 4369 -j DROP
iptables -A PREROUTING -t raw -p tcp --dport 4369 -j DROP
iptables -A PREROUTING -t raw -p udp --dport 8738 -j DROP
iptables -A PREROUTING -t raw -p tcp --dport 8738 -j DROP

[1] https://github.com/wdebruij/kerneltools/blob/master/tests/pingpong_tcpudp.c

[Intel-wired-lan] e1000e echo request response latency and tcph->ack

[Willem de Bruijn]

On Tue, May 2, 2017 at 1:15 PM, Willem de Bruijn
<willemdebruijn.kernel@gmail.com> wrote:
> With RR tests on a pair of 82571EB I am seeing much lower latency with
> TCP (40 usec) vs UDP (105 usec). But only for TCP with the ACK bit
> set.
>
> I observed the initial TCP vs UDP results with sockperf.  Digging into
> this behavior by emulating the protocol headers with packet sockets
> show that just toggling the ACK bit changes this behavior. I shared
> the test program that does this on github [1].
>
> Is this known behavior? Is there some fast path on the NIC triggered
> only for TCP ACK packets? If so, then my real question is, is there a
> way to make UDP packets take this fast path?

Found the cause. This happens because because on the 82571 the default
setting for RxIntDelay (0) is overridden by BURST_RDTR (0x20) in
e1000_configure_tx.

                /* override the delay timers for enabling bursting, only if
                 * the value was not set by the user via module options
                 */
                if (adapter->rx_int_delay == DEFAULT_RDTR)
                        adapter->rx_int_delay = BURST_RDTR;
                if (adapter->rx_abs_int_delay == DEFAULT_RADV)
                        adapter->rx_abs_int_delay = BURST_RADV;

Loading the module with modprobe e1000e RxIntDelay=1 avoids that
branch and removes almost all of the latency. Loading the module
explicitly with modprobe e1000e RxIntDelay=0 probably should avoid
that burst mode, too. I can send a patch.

As a matter of fact, both RDTR and RADV should perhaps be set to 0
whenever the administrator sets ITR is set to 0 (e.g., as programmed
through ethtool -C ethX rx-usecs 0 tx-usecs 0).

Or even initialized to zero unconditionally. The 82571/82572 errata
lists "69. Receive Packet Delayed When Using RDTR or RADV Register."
with suggested workaround "It is recommended that the RDTR and RADV
registers not be used for moderating Rx interrupts. The preferred
solution is to use the Interrupt Throttling Register; ITR."