From: "Michael S. Tsirkin" <mst@redhat.com>
To: Tiwei Bie <tiwei.bie@intel.com>
Cc: jasowang@redhat.com, virtualization@lists.linux-foundation.org,
linux-kernel@vger.kernel.org, netdev@vger.kernel.org,
virtio-dev@lists.oasis-open.org, wexu@redhat.com,
jfreimann@redhat.com
Subject: Re: [virtio-dev] Re: [PATCH net-next v2 0/5] virtio: support packed ring
Date: Wed, 12 Sep 2018 09:06:31 -0400 [thread overview]
Message-ID: <20180912085453-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <20180910030053.GA15645@debian>
On Mon, Sep 10, 2018 at 11:00:53AM +0800, Tiwei Bie wrote:
> On Fri, Sep 07, 2018 at 09:00:49AM -0400, Michael S. Tsirkin wrote:
> > On Fri, Sep 07, 2018 at 09:22:25AM +0800, Tiwei Bie wrote:
> > > On Mon, Aug 27, 2018 at 05:00:40PM +0300, Michael S. Tsirkin wrote:
> > > > Are there still plans to test the performance with vost pmd?
> > > > vhost doesn't seem to show a performance gain ...
> > > >
> > >
> > > I tried some performance tests with vhost PMD. In guest, the
> > > XDP program will return XDP_DROP directly. And in host, testpmd
> > > will do txonly fwd.
> > >
> > > When burst size is 1 and packet size is 64 in testpmd and
> > > testpmd needs to iterate 5 Tx queues (but only the first two
> > > queues are enabled) to prepare and inject packets, I got ~12%
> > > performance boost (5.7Mpps -> 6.4Mpps). And if the vhost PMD
> > > is faster (e.g. just need to iterate the first two queues to
> > > prepare and inject packets), then I got similar performance
> > > for both rings (~9.9Mpps) (packed ring's performance can be
> > > lower, because it's more complicated in driver.)
> > >
> > > I think packed ring makes vhost PMD faster, but it doesn't make
> > > the driver faster. In packed ring, the ring is simplified, and
> > > the handling of the ring in vhost (device) is also simplified,
> > > but things are not simplified in driver, e.g. although there is
> > > no desc table in the virtqueue anymore, driver still needs to
> > > maintain a private desc state table (which is still managed as
> > > a list in this patch set) to support the out-of-order desc
> > > processing in vhost (device).
> > >
> > > I think this patch set is mainly to make the driver have a full
> > > functional support for the packed ring, which makes it possible
> > > to leverage the packed ring feature in vhost (device). But I'm
> > > not sure whether there is any other better idea, I'd like to
> > > hear your thoughts. Thanks!
> >
> > Just this: Jens seems to report a nice gain with virtio and
> > vhost pmd across the board. Try to compare virtio and
> > virtio pmd to see what does pmd do better?
>
> The virtio PMD (drivers/net/virtio) in DPDK doesn't need to share
> the virtio ring operation code with other drivers and is highly
> optimized for network. E.g. in Rx, the Rx burst function won't
> chain descs.
> So the ID management for the Rx ring can be quite
> simple and straightforward, we just need to initialize these IDs
> when initializing the ring and don't need to change these IDs
> in data path anymore (the mergable Rx code in that patch set
> assumes the descs will be written back in order, which should be
> fixed. I.e., the ID in the desc should be used to index vq->descx[]).
> The Tx code in that patch set also assumes the descs will be
> written back by device in order, which should be fixed.
>
> But in kernel virtio driver, the virtio_ring.c is very generic.
> The enqueue (virtqueue_add()) and dequeue (virtqueue_get_buf_ctx())
> functions need to support all the virtio devices and should be
> able to handle all the possible cases that may happen. So although
> the packed ring can be very efficient in some cases, currently
> the room to optimize the performance in kernel's virtio_ring.c
> isn't that much. If we want to take the fully advantage of the
> packed ring's efficiency, we need some further e.g. API changes
> in virtio_ring.c, which shouldn't be part of this patch set. So
> I still think this patch set is mainly to make the kernel virtio
> driver to have a full functional support of the packed ring, and
> we can't expect impressive performance boost with it.
So what are the cases that make things complex?
Maybe we should drop support for them completely.
> >
> >
> > >
> > > >
> > > > On Wed, Jul 11, 2018 at 10:27:06AM +0800, Tiwei Bie wrote:
> > > > > Hello everyone,
> > > > >
> > > > > This patch set implements packed ring support in virtio driver.
> > > > >
> > > > > Some functional tests have been done with Jason's
> > > > > packed ring implementation in vhost:
> > > > >
> > > > > https://lkml.org/lkml/2018/7/3/33
> > > > >
> > > > > Both of ping and netperf worked as expected.
> > > > >
> > > > > v1 -> v2:
> > > > > - Use READ_ONCE() to read event off_wrap and flags together (Jason);
> > > > > - Add comments related to ccw (Jason);
> > > > >
> > > > > RFC (v6) -> v1:
> > > > > - Avoid extra virtio_wmb() in virtqueue_enable_cb_delayed_packed()
> > > > > when event idx is off (Jason);
> > > > > - Fix bufs calculation in virtqueue_enable_cb_delayed_packed() (Jason);
> > > > > - Test the state of the desc at used_idx instead of last_used_idx
> > > > > in virtqueue_enable_cb_delayed_packed() (Jason);
> > > > > - Save wrap counter (as part of queue state) in the return value
> > > > > of virtqueue_enable_cb_prepare_packed();
> > > > > - Refine the packed ring definitions in uapi;
> > > > > - Rebase on the net-next tree;
> > > > >
> > > > > RFC v5 -> RFC v6:
> > > > > - Avoid tracking addr/len/flags when DMA API isn't used (MST/Jason);
> > > > > - Define wrap counter as bool (Jason);
> > > > > - Use ALIGN() in vring_init_packed() (Jason);
> > > > > - Avoid using pointer to track `next` in detach_buf_packed() (Jason);
> > > > > - Add comments for barriers (Jason);
> > > > > - Don't enable RING_PACKED on ccw for now (noticed by Jason);
> > > > > - Refine the memory barrier in virtqueue_poll();
> > > > > - Add a missing memory barrier in virtqueue_enable_cb_delayed_packed();
> > > > > - Remove the hacks in virtqueue_enable_cb_prepare_packed();
> > > > >
> > > > > RFC v4 -> RFC v5:
> > > > > - Save DMA addr, etc in desc state (Jason);
> > > > > - Track used wrap counter;
> > > > >
> > > > > RFC v3 -> RFC v4:
> > > > > - Make ID allocation support out-of-order (Jason);
> > > > > - Various fixes for EVENT_IDX support;
> > > > >
> > > > > RFC v2 -> RFC v3:
> > > > > - Split into small patches (Jason);
> > > > > - Add helper virtqueue_use_indirect() (Jason);
> > > > > - Just set id for the last descriptor of a list (Jason);
> > > > > - Calculate the prev in virtqueue_add_packed() (Jason);
> > > > > - Fix/improve desc suppression code (Jason/MST);
> > > > > - Refine the code layout for XXX_split/packed and wrappers (MST);
> > > > > - Fix the comments and API in uapi (MST);
> > > > > - Remove the BUG_ON() for indirect (Jason);
> > > > > - Some other refinements and bug fixes;
> > > > >
> > > > > RFC v1 -> RFC v2:
> > > > > - Add indirect descriptor support - compile test only;
> > > > > - Add event suppression supprt - compile test only;
> > > > > - Move vring_packed_init() out of uapi (Jason, MST);
> > > > > - Merge two loops into one in virtqueue_add_packed() (Jason);
> > > > > - Split vring_unmap_one() for packed ring and split ring (Jason);
> > > > > - Avoid using '%' operator (Jason);
> > > > > - Rename free_head -> next_avail_idx (Jason);
> > > > > - Add comments for virtio_wmb() in virtqueue_add_packed() (Jason);
> > > > > - Some other refinements and bug fixes;
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Tiwei Bie (5):
> > > > > virtio: add packed ring definitions
> > > > > virtio_ring: support creating packed ring
> > > > > virtio_ring: add packed ring support
> > > > > virtio_ring: add event idx support in packed ring
> > > > > virtio_ring: enable packed ring
> > > > >
> > > > > drivers/s390/virtio/virtio_ccw.c | 14 +
> > > > > drivers/virtio/virtio_ring.c | 1365 ++++++++++++++++++++++------
> > > > > include/linux/virtio_ring.h | 8 +-
> > > > > include/uapi/linux/virtio_config.h | 3 +
> > > > > include/uapi/linux/virtio_ring.h | 43 +
> > > > > 5 files changed, 1157 insertions(+), 276 deletions(-)
> > > > >
> > > > > --
> > > > > 2.18.0
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > > > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> > > >
prev parent reply other threads:[~2018-09-12 13:06 UTC|newest]
Thread overview: 53+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-07-11 2:27 [PATCH net-next v2 0/5] virtio: support packed ring Tiwei Bie
2018-07-11 2:27 ` [PATCH net-next v2 1/5] virtio: add packed ring definitions Tiwei Bie
2018-09-07 13:51 ` Michael S. Tsirkin
2018-09-10 2:13 ` Tiwei Bie
2018-09-12 12:53 ` Michael S. Tsirkin
2018-07-11 2:27 ` [PATCH net-next v2 2/5] virtio_ring: support creating packed ring Tiwei Bie
2018-09-07 14:03 ` Michael S. Tsirkin
2018-09-10 2:28 ` Tiwei Bie
2018-09-12 7:51 ` Tiwei Bie
2018-09-12 16:12 ` Michael S. Tsirkin
2018-09-12 12:45 ` Michael S. Tsirkin
2018-07-11 2:27 ` [PATCH net-next v2 3/5] virtio_ring: add packed ring support Tiwei Bie
2018-09-07 13:49 ` Michael S. Tsirkin
2018-09-10 2:03 ` Tiwei Bie
2018-09-12 16:22 ` Michael S. Tsirkin
2018-11-07 17:48 ` Michael S. Tsirkin
2018-11-08 1:38 ` Tiwei Bie
2018-11-08 8:18 ` Jason Wang
2018-11-08 11:51 ` Tiwei Bie
2018-11-08 15:56 ` Michael S. Tsirkin
2018-11-09 1:50 ` Tiwei Bie
2018-11-09 2:30 ` Jason Wang
2018-11-09 4:00 ` Michael S. Tsirkin
2018-11-09 10:05 ` Jason Wang
2018-11-08 14:14 ` Michael S. Tsirkin
2018-11-09 2:25 ` Jason Wang
2018-11-09 3:58 ` Michael S. Tsirkin
2018-11-09 10:04 ` Jason Wang
2018-07-11 2:27 ` [PATCH net-next v2 4/5] virtio_ring: add event idx support in packed ring Tiwei Bie
2018-09-07 14:10 ` Michael S. Tsirkin
2018-09-10 2:35 ` [virtio-dev] " Tiwei Bie
2018-07-11 2:27 ` [PATCH net-next v2 5/5] virtio_ring: enable " Tiwei Bie
2018-07-11 2:52 ` [PATCH net-next v2 0/5] virtio: support " Jason Wang
2018-07-12 21:44 ` David Miller
2018-07-13 0:52 ` Jason Wang
2018-07-13 3:26 ` Michael S. Tsirkin
2018-08-27 14:00 ` Michael S. Tsirkin
2018-08-28 5:51 ` [virtio-dev] " Jens Freimann
2018-09-07 1:22 ` Tiwei Bie
2018-09-07 13:00 ` Michael S. Tsirkin
2018-09-10 3:00 ` Tiwei Bie
2018-09-10 3:33 ` Jason Wang
2018-09-11 5:37 ` Tiwei Bie
2018-09-12 16:16 ` Michael S. Tsirkin
2018-09-13 8:59 ` Tiwei Bie
2018-09-13 9:47 ` Jason Wang
2018-10-10 14:36 ` Michael S. Tsirkin
2018-10-11 12:12 ` Tiwei Bie
2018-10-11 13:48 ` Michael S. Tsirkin
2018-10-11 14:13 ` Tiwei Bie
2018-10-11 14:17 ` Michael S. Tsirkin
2018-10-11 14:34 ` Tiwei Bie
2018-09-12 13:06 ` Michael S. Tsirkin [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180912085453-mutt-send-email-mst@kernel.org \
--to=mst@redhat.com \
--cc=jasowang@redhat.com \
--cc=jfreimann@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=tiwei.bie@intel.com \
--cc=virtio-dev@lists.oasis-open.org \
--cc=virtualization@lists.linux-foundation.org \
--cc=wexu@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).