From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
To: Jason Wang <jasowang@redhat.com>
Cc: netdev@vger.kernel.org, "David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
"Michael S. Tsirkin" <mst@redhat.com>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Jesper Dangaard Brouer <hawk@kernel.org>,
John Fastabend <john.fastabend@gmail.com>,
virtualization@lists.linux-foundation.org, bpf@vger.kernel.org
Subject: Re: [PATCH net-next v1 00/19] virtio-net: support AF_XDP zero copy
Date: Tue, 17 Oct 2023 11:02:05 +0800 [thread overview]
Message-ID: <1697511725.2037013-1-xuanzhuo@linux.alibaba.com> (raw)
In-Reply-To: <CACGkMEs4u-4ch2UAK14hNfKeORjqMu4BX7=46OfaXpvxW+VT7w@mail.gmail.com>
On Tue, 17 Oct 2023 10:53:44 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Mon, Oct 16, 2023 at 8:00 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > ## AF_XDP
> >
> > XDP socket(AF_XDP) is an excellent bypass kernel network framework. The zero
> > copy feature of xsk (XDP socket) needs to be supported by the driver. The
> > performance of zero copy is very good. mlx5 and intel ixgbe already support
> > this feature, This patch set allows virtio-net to support xsk's zerocopy xmit
> > feature.
> >
> > At present, we have completed some preparation:
> >
> > 1. vq-reset (virtio spec and kernel code)
> > 2. virtio-core premapped dma
> > 3. virtio-net xdp refactor
> >
> > So it is time for Virtio-Net to complete the support for the XDP Socket
> > Zerocopy.
> >
> > Virtio-net can not increase the queue num at will, so xsk shares the queue with
> > kernel.
> >
> > On the other hand, Virtio-Net does not support generate interrupt from driver
> > manually, so when we wakeup tx xmit, we used some tips. If the CPU run by TX
> > NAPI last time is other CPUs, use IPI to wake up NAPI on the remote CPU. If it
> > is also the local CPU, then we wake up napi directly.
> >
> > This patch set includes some refactor to the virtio-net to let that to support
> > AF_XDP.
> >
> > ## performance
> >
> > ENV: Qemu with vhost-user(polling mode).
> >
> > Sockperf: https://github.com/Mellanox/sockperf
> > I use this tool to send udp packet by kernel syscall.
> >
> > xmit command: sockperf tp -i 10.0.3.1 -t 1000
> >
> > I write a tool that sends udp packets or recvs udp packets by AF_XDP.
> >
> > | Guest APP CPU |Guest Softirq CPU | UDP PPS
> > ------------------|---------------|------------------|------------
> > xmit by syscall | 100% | | 676,915
> > xmit by xsk | 59.1% | 100% | 5,447,168
> > recv by syscall | 60% | 100% | 932,288
> > recv by xsk | 35.7% | 100% | 3,343,168
>
> Any chance we can get a testpmd result (which I guess should be better
> than PPS above)?
Do you mean testpmd + DPDK + AF_XDP?
Yes. This is probably better because my tool does more work. That is not a
complete testing tool used by our business.
What I noticed is that the hotspot is the driver writing virtio desc. Because
the device is in busy mode. So there is race between driver and device.
So I modified the virtio core and lazily updated avail idx. Then pps can reach
10,000,000.
Thanks.
>
> Thanks
>
> >
> > ## maintain
> >
> > I am currently a reviewer for virtio-net. I commit to maintain AF_XDP support in
> > virtio-net.
> >
> > Please review.
> >
> > Thanks.
> >
> > v1:
> > 1. remove two virtio commits. Push this patchset to net-next
> > 2. squash "virtio_net: virtnet_poll_tx support rescheduled" to xsk: support tx
> > 3. fix some warnings
> >
> > Xuan Zhuo (19):
> > virtio_net: rename free_old_xmit_skbs to free_old_xmit
> > virtio_net: unify the code for recycling the xmit ptr
> > virtio_net: independent directory
> > virtio_net: move to virtio_net.h
> > virtio_net: add prefix virtnet to all struct/api inside virtio_net.h
> > virtio_net: separate virtnet_rx_resize()
> > virtio_net: separate virtnet_tx_resize()
> > virtio_net: sq support premapped mode
> > virtio_net: xsk: bind/unbind xsk
> > virtio_net: xsk: prevent disable tx napi
> > virtio_net: xsk: tx: support tx
> > virtio_net: xsk: tx: support wakeup
> > virtio_net: xsk: tx: virtnet_free_old_xmit() distinguishes xsk buffer
> > virtio_net: xsk: tx: virtnet_sq_free_unused_buf() check xsk buffer
> > virtio_net: xsk: rx: introduce add_recvbuf_xsk()
> > virtio_net: xsk: rx: introduce receive_xsk() to recv xsk buffer
> > virtio_net: xsk: rx: virtnet_rq_free_unused_buf() check xsk buffer
> > virtio_net: update tx timeout record
> > virtio_net: xdp_features add NETDEV_XDP_ACT_XSK_ZEROCOPY
> >
> > MAINTAINERS | 2 +-
> > drivers/net/Kconfig | 8 +-
> > drivers/net/Makefile | 2 +-
> > drivers/net/virtio/Kconfig | 13 +
> > drivers/net/virtio/Makefile | 8 +
> > drivers/net/{virtio_net.c => virtio/main.c} | 652 +++++++++-----------
> > drivers/net/virtio/virtio_net.h | 359 +++++++++++
> > drivers/net/virtio/xsk.c | 545 ++++++++++++++++
> > drivers/net/virtio/xsk.h | 32 +
> > 9 files changed, 1247 insertions(+), 374 deletions(-)
> > create mode 100644 drivers/net/virtio/Kconfig
> > create mode 100644 drivers/net/virtio/Makefile
> > rename drivers/net/{virtio_net.c => virtio/main.c} (91%)
> > create mode 100644 drivers/net/virtio/virtio_net.h
> > create mode 100644 drivers/net/virtio/xsk.c
> > create mode 100644 drivers/net/virtio/xsk.h
> >
> > --
> > 2.32.0.3.g01195cf9f
> >
>
next prev parent reply other threads:[~2023-10-17 3:11 UTC|newest]
Thread overview: 66+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-10-16 12:00 [PATCH net-next v1 00/19] virtio-net: support AF_XDP zero copy Xuan Zhuo
2023-10-16 12:00 ` [PATCH net-next v1 01/19] virtio_net: rename free_old_xmit_skbs to free_old_xmit Xuan Zhuo
2023-10-19 4:17 ` Jason Wang
2023-10-16 12:00 ` [PATCH net-next v1 02/19] virtio_net: unify the code for recycling the xmit ptr Xuan Zhuo
2023-10-19 4:23 ` Jason Wang
2023-10-16 12:00 ` [PATCH net-next v1 03/19] virtio_net: independent directory Xuan Zhuo
2023-10-19 6:10 ` Jason Wang
2023-10-16 12:00 ` [PATCH net-next v1 04/19] virtio_net: move to virtio_net.h Xuan Zhuo
2023-10-19 6:12 ` Jason Wang
2023-10-19 7:16 ` Xuan Zhuo
2023-10-20 6:59 ` Jason Wang
2023-10-16 12:00 ` [PATCH net-next v1 05/19] virtio_net: add prefix virtnet to all struct/api inside virtio_net.h Xuan Zhuo
2023-10-19 6:14 ` Jason Wang
2023-10-19 6:36 ` Michael S. Tsirkin
2023-10-16 12:00 ` [PATCH net-next v1 06/19] virtio_net: separate virtnet_rx_resize() Xuan Zhuo
2023-10-19 6:17 ` Jason Wang
2023-10-16 12:00 ` [PATCH net-next v1 07/19] virtio_net: separate virtnet_tx_resize() Xuan Zhuo
2023-10-19 6:18 ` Jason Wang
2023-10-16 12:00 ` [PATCH net-next v1 08/19] virtio_net: sq support premapped mode Xuan Zhuo
2023-10-20 6:50 ` Jason Wang
2023-10-20 7:16 ` Xuan Zhuo
2023-10-16 12:00 ` [PATCH net-next v1 09/19] virtio_net: xsk: bind/unbind xsk Xuan Zhuo
2023-10-20 6:51 ` Jason Wang
2023-10-20 7:28 ` Xuan Zhuo
2023-10-16 12:00 ` [PATCH net-next v1 10/19] virtio_net: xsk: prevent disable tx napi Xuan Zhuo
2023-10-20 6:51 ` Jason Wang
2023-10-16 12:00 ` [PATCH net-next v1 11/19] virtio_net: xsk: tx: support tx Xuan Zhuo
2023-10-20 6:52 ` Jason Wang
2023-10-20 8:06 ` Xuan Zhuo
2023-10-16 12:00 ` [PATCH net-next v1 12/19] virtio_net: xsk: tx: support wakeup Xuan Zhuo
2023-10-20 6:52 ` Jason Wang
2023-10-20 8:09 ` Xuan Zhuo
2023-10-16 12:00 ` [PATCH net-next v1 13/19] virtio_net: xsk: tx: virtnet_free_old_xmit() distinguishes xsk buffer Xuan Zhuo
2023-10-16 23:44 ` Jakub Kicinski
2023-10-17 2:02 ` Xuan Zhuo
2023-10-19 6:38 ` Michael S. Tsirkin
2023-10-19 7:13 ` Xuan Zhuo
2023-10-19 8:42 ` Michael S. Tsirkin
2023-10-16 12:00 ` [PATCH net-next v1 14/19] virtio_net: xsk: tx: virtnet_sq_free_unused_buf() check " Xuan Zhuo
2023-10-20 6:53 ` Jason Wang
2023-10-16 12:00 ` [PATCH net-next v1 15/19] virtio_net: xsk: rx: introduce add_recvbuf_xsk() Xuan Zhuo
2023-10-20 6:56 ` Jason Wang
2023-10-23 6:56 ` Xuan Zhuo
2023-10-16 12:00 ` [PATCH net-next v1 16/19] virtio_net: xsk: rx: introduce receive_xsk() to recv xsk buffer Xuan Zhuo
2023-10-20 6:57 ` Jason Wang
2023-10-23 2:39 ` Xuan Zhuo
2023-11-15 2:35 ` Xuan Zhuo
2023-10-16 12:00 ` [PATCH net-next v1 17/19] virtio_net: xsk: rx: virtnet_rq_free_unused_buf() check " Xuan Zhuo
2023-10-16 12:00 ` [PATCH net-next v1 18/19] virtio_net: update tx timeout record Xuan Zhuo
2023-10-20 6:57 ` Jason Wang
2023-10-16 12:00 ` [PATCH net-next v1 19/19] virtio_net: xdp_features add NETDEV_XDP_ACT_XSK_ZEROCOPY Xuan Zhuo
2023-10-17 2:53 ` [PATCH net-next v1 00/19] virtio-net: support AF_XDP zero copy Jason Wang
2023-10-17 3:02 ` Xuan Zhuo [this message]
2023-10-17 3:20 ` Jason Wang
2023-10-17 3:22 ` Xuan Zhuo
2023-10-17 3:28 ` Jason Wang
2023-10-17 5:27 ` Jason Wang
2023-10-17 6:06 ` Xuan Zhuo
2023-10-17 6:26 ` Jason Wang
2023-10-17 6:43 ` Xuan Zhuo
2023-10-17 11:19 ` Xuan Zhuo
2023-10-18 2:46 ` Jason Wang
2023-10-18 2:56 ` Xuan Zhuo
2023-10-18 3:32 ` Xuan Zhuo
2023-10-18 3:40 ` Jason Wang
2023-10-18 1:02 ` Jason Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1697511725.2037013-1-xuanzhuo@linux.alibaba.com \
--to=xuanzhuo@linux.alibaba.com \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=hawk@kernel.org \
--cc=jasowang@redhat.com \
--cc=john.fastabend@gmail.com \
--cc=kuba@kernel.org \
--cc=mst@redhat.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=virtualization@lists.linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).