KVM virtio network performance on RHEL5.4

KVM virtio network performance on RHEL5.4

[Tom Lendacky]

I've been trying to understand why the performance from guest to guest over a 
10GbE link using virtio, as measured by netperf, dramatically decreases when 
the socket buffer size is increased on the receiving guest.  This is an Intel 
X3210 4-core 2.13GHz system running RHEL5.4.  I don't see this drop in 
performance when going from guest to host or host to guest over the 10GbE 
link.  Here are the results from netperf:

Default socket buffer sizes:
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

 87380  16384  16384    60.01      2268.47   47.69    99.95    1.722   3.609

Receiver 256K socket buffer size (actually rmem_max * 2):
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

262142  16384  16384    60.00      1583.75   39.00    74.09    2.018   3.832

There is increased idle time in the receiver.  Using systemtap I found that 
the idle time is because we are waiting for data (tcp_recvmsg calling 
sk_wait_data).

I instrumented qemu on the receiver side to print out some statistics related 
to xmit/recv events.

"Rx-Could not receive" is incremented whenever "do_virtio_net_can_receive" 
returns 0
"Rx-Ring full" is incremented in "do_virtio_net_can_receive" whenever there 
are no available entries/space in the receive ring
"Rx-Count" is incremented whenever "virtio_net_receive2" is called (and can 
receive data)
"Rx-Bytes" is increased in "virtio_net_receive2" by the number of bytes to be 
read from the tap device
"Rx-Ring buffers" is increased by the number of buffers used for the data in 
"virtio_net_receive2"
"Tx-Notify" is incremented whenever "virtio_net_handle_tx" is invoked
"Tx-Sched BH" is incremented whenever "virtio_net_handle_tx" is invoked and 
the the qemu_bh hasn't been scheduled yet
"Tx-Packets" is incremented in "virtio_net_flush_tx" whenever a packet is 
removed from the transmit ring and sent to qemu
"Tx-Bytes" is increased in "virtio_net_flush_tx" by the number of bytes sent 
to qemu.

Here are the stats for the two cases:

                        Default                 256K
Rx-Could not receive    3,559                   0
Rx-Ring full            3,559                   0
Rx-Count                1,063,056               805,012
Rx-Bytes                18,131,704,980          12,593,270,826
Rx-Ring buffers         4,963,793               3,541,010
Tx-Notify               125,068                 125,702
Tx-Sched BH             125,068                 125,702
Tx-Packets              147,256                 232,219
Tx-Bytes                11,486,448              18,113,586

Dividing the Tx-Bytes by Tx-Packets in each case yields about 78 bytes/packet 
so these are most likely ACKs.  But why am I seeing almost 85,000 more of 
these in the 256K socket buffer case?  Also, dividing the Rx-Bytes by the Rx-
Count shows that the tap device is delivering about 1413 bytes less per call 
to qemu in the 256K socket buffer case.

Does anyone have some insight as to what is happening?

Thanks,
Tom Lendacky