Re: [PATCH bpf-next 11/14] xsk: add shared umem support between devices

From: Maxim Mikityanskiy <maximmi@mellanox.com>
To: Magnus Karlsson <magnus.karlsson@intel.com>
Cc: bjorn.topel@intel.com, ast@kernel.org, daniel@iogearbox.net,
	netdev@vger.kernel.org, jonathan.lemon@gmail.com,
	bpf@vger.kernel.org, jeffrey.t.kirsher@intel.com,
	maciej.fijalkowski@intel.com, maciejromanfijalkowski@gmail.com,
	cristian.dumitrescu@intel.com
Subject: Re: [PATCH bpf-next 11/14] xsk: add shared umem support between devices
Date: Thu, 9 Jul 2020 18:20:10 +0300	[thread overview]
Message-ID: <f6008a8d-92bd-85f5-1c84-8aa638bbacef@mellanox.com> (raw)
In-Reply-To: <1593692353-15102-12-git-send-email-magnus.karlsson@intel.com>

On 2020-07-02 15:19, Magnus Karlsson wrote:
> Add support to share a umem between different devices. This mode
> can be invoked with the XDP_SHARED_UMEM bind flag. Previously,
> sharing was only supported within the same device. Note that when
> sharing a umem between devices, just as in the case of sharing a
> umem between queue ids, you need to create a fill ring and a
> completion ring and tie them to the socket (with two setsockopts,
> one for each ring) before you do the bind with the
> XDP_SHARED_UMEM flag. This so that the single-producer
> single-consumer semantics of the rings can be upheld.

I also wonder what performance numbers you see when doing forwarding 
with xsk_fwd between two queues of the same netdev and between two 
netdevs. Could you share (compared to some baseline like xdpsock -l)?

> 
> Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com>
> ---
>   net/xdp/xsk.c | 11 ++++-------
>   1 file changed, 4 insertions(+), 7 deletions(-)
> 
> diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
> index 1abc222..b240221 100644
> --- a/net/xdp/xsk.c
> +++ b/net/xdp/xsk.c
> @@ -692,14 +692,11 @@ static int xsk_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
>   			sockfd_put(sock);
>   			goto out_unlock;
>   		}
> -		if (umem_xs->dev != dev) {
> -			err = -EINVAL;
> -			sockfd_put(sock);
> -			goto out_unlock;
> -		}
>   
> -		if (umem_xs->queue_id != qid) {
> -			/* Share the umem with another socket on another qid */
> +		if (umem_xs->queue_id != qid || umem_xs->dev != dev) {
> +			/* Share the umem with another socket on another qid
> +			 * and/or device.
> +			 */
>   			new_pool = xp_assign_umem(xs->pool, umem_xs->umem);
>   			if (!new_pool) {
>   				sockfd_put(sock);
> 