Re: zero-copy between interfaces

From: Ryan Goodfellow <rgoodfel@isi.edu>
To: Maxim Mikityanskiy <maximmi@mellanox.com>
Cc: Magnus Karlsson <magnus.karlsson@gmail.com>,
	"xdp-newbies@vger.kernel.org" <xdp-newbies@vger.kernel.org>,
	Tariq Toukan <tariqt@mellanox.com>,
	Saeed Mahameed <saeedm@mellanox.com>,
	Moshe Shemesh <moshe@mellanox.com>
Subject: Re: zero-copy between interfaces
Date: Thu, 23 Jan 2020 05:43:55 +0800	[thread overview]
Message-ID: <20200122214352.GA13201@smtp.ads.isi.edu> (raw)
In-Reply-To: <4c03813d-5edf-7e9e-8905-31902b5acb71@mellanox.com>

On Tue, Jan 21, 2020 at 01:40:50PM +0000, Maxim Mikityanskiy wrote:
> >> I've posted output from the program in debugging mode here
> >>
> >> - https://gitlab.com/mergetb/tech/network-emulation/kernel/snippets/1930375
> >>
> >> Yes, you are correct in that forwarding works for a brief period and then stops.
> >> I've noticed that the number of packets that are forwarded is equal to the size
> >> of the producer/consumer descriptor rings. I've posted two ping traces from a
> >> client ping that shows this.
> >>
> >> - https://gitlab.com/mergetb/tech/network-emulation/kernel/snippets/1930376
> >> - https://gitlab.com/mergetb/tech/network-emulation/kernel/snippets/1930377
> 
> These snippets are not available.

Apologies, I had the wrong permissions set. They should be available now.

> 
> >>
> >> I've also noticed that when the forwarding stops, the CPU usage for the proc
> >> running the program is pegged, which is not the norm for this program as it uses
> >> a poll call with a timeout on the xsk fd.
> 
> This information led me to a guess what may be happening. On the RX 
> side, mlx5e allocates pages in bulks for performance reasons and to 
> leverage hardware features targeted to performance. In AF_XDP mode, 
> bulking of frames is also used (on x86, the bulk size is 64 with 
> striding RQ enabled, and 8 otherwise, however, it's implementation 
> details that might change later). If you don't put enough frames to XSK 
> Fill Ring, the driver will be demanding more frames and return from 
> poll() immediately. Basically, in the application, you should put as 
> many frames to the Fill Ring as you can. Please check if that could be 
> the root cause of your issue.

The code in this application makes an effort to relenish the fill ring as fast
as possible. The basic loop of the application is to first check if there are
any descriptors to be consumed from the completion queue or any descriptors that
can be added to the fill queue, and only then to move on to moving packets
through the rx and tx rings.

https://gitlab.com/mergetb/tech/network-emulation/kernel/blob/v5.5-moa/samples/bpf/xdpsock_multidev.c#L452-474

> 
> I tracked this issue in our internal bug tracker in case we need to 
> perform actual debugging of mlx5e. I'm looking forward to your feedback 
> on my assumption above.
> 
> >> The hardware I am using is a Mellanox ConnectX4 2x100G card (MCX416A-CCAT)
> >> running the mlx5 driver.
> 
> This one should run without striding RQ, please verify it with ethtool 
> --show-priv-flags (the flag name is rx_striding_rq).

I do not remember changing this option, so whatever the default is, is what it
was running with. I am traveling this week and do not have access to these
systems, but will ensure that this flag is set properly when I get back.

-- 
~ ry