Re: Improving OCTEON II 10G Ethernet performance

[David Daney <ddaney@caviumnetworks.com>]

On 08/24/2016 06:29 PM, Ed Swierk wrote:
> I'm trying to migrate from the Octeon SDK to a vanilla Linux 4.4
> kernel for a Cavium OCTEON II (CN6880) board running in 64-bit
> little-endian mode. So far I've gotten most of the hardware features I
> need working, including XAUI/RXAUI, USB, boot bus and I2C, with a
> fairly small set of patches.
> https://github.com/skyportsystems/linux/compare/master...octeon2
>

It is unclear what your motivations for doing this are, so I can think 
of several things you could do:

A) Get v4.4 based SDK from Cavium.

B) Major rewrite of octeon-ethernet driver.

C) Live with current staging driver.

> The biggest remaining hurdle is improving 10G Ethernet performance:
> iperf -P 10 on the SDK kernel gets close to 10 Gbit/sec throughput,
> while on my 4.4 kernel, it tops out around 1 Gbit/sec.
>
> Comparing the octeon-ethernet driver in the SDK
> (http://git.yoctoproject.org/cgit/cgit.cgi/linux-yocto-contrib/tree/drivers/net/ethernet/octeon?h=apaliwal/octeon)
> against the one in 4.4, the latter appears to utilize only a single
> CPU core for the rx path. It's not clear to me if there is a similar
> issue on the tx side, or other bottlenecks.

The main limiting factor to performance is single threaded RX 
processing.  The main manner this is handled in the out-of-tree vendor 
driver is to have multiple NAPI processing threads running against the 
same RX queue when there is a queue backlog.  The disadvantage of doing 
this is that packets may be received out of order due to 
non-synchronization across multiple CPUs.

On the TX side, the locks on the queuing discipline can become contended 
leading to cache line bouncing.  In the TX code of the driver itself, 
there should be no impediments to parallel TX operations.

Ideally we would configure the packet classifiers on the RX side to 
create multiple RX queues based on a hash of the TCP 5-tuple, and handle 
each queue with a single NAPI instance.  That should result in better 
performance while maintaining packet ordering.


>
> I started trying to port multi-CPU rx from the SDK octeon-ethernet
> driver, but had trouble teasing out just the necessary bits without
> following a maze of dependencies on unrelated functions. (Dragging
> major parts of the SDK wholesale into 4.4 defeats the purpose of
> switching to a vanilla kernel, and doesn't bring us closer to getting
> octeon-ethernet out of staging.)

Yes, you have identified the main problem with this code.

All the code managing the SerDes and other MAC functions needs a 
complete rewrite.  One main problem is that all the SerDes/MACs in the 
system are configured simultaneously instead of on a per device basis. 
There are also a plethora of different SerDes technologies in use: 
(RGMII, SGMII, QSGMII, XFI, XAUI, RXAUI, SPI-4.1, XLAUI, KR, ...)  The 
code that handles all of these is mixed together with huge case 
statements switching on interface mode all over the place.

There is also code to handle target-mode PCI/PCIe packet engines mixed 
in as well.  This stuff should probably be removed.


>
> Has there been any work on the octeon-ethernet driver since this patch
> set? https://www.linux-mips.org/archives/linux-mips/2015-08/msg00338.html
>
> Any hints on what to pick out of the SDK code to improve 10G
> performance would be appreciated.
>
> --Ed
>