From: "Michael S. Tsirkin" <mst@redhat.com>
To: linux-kernel@vger.kernel.org
Cc: qemu-devel@nongnu.org, virtualization@lists.linux-foundation.org
Subject: [PATCH] virtio_ring: use smp_store_mb
Date: Thu, 17 Dec 2015 12:32:53 +0200 [thread overview]
Message-ID: <1450347932-16325-1-git-send-email-mst__46919.7600767531$1450348391$gmane$org@redhat.com> (raw)
We need a full barrier after writing out event index, using smp_store_mb
there seems better than open-coding.
As usual, we need a wrapper to account for strong barriers/non smp.
It's tempting to use this in vhost as well, for that, we'll
need a variant of smp_store_mb that works on __user pointers.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
---
Seems to give a speedup on my box but I'm less sure about this one. E.g. as
xchng faster than mfence on all/most intel CPUs? Anyone has an opinion?
include/linux/virtio_ring.h | 14 ++++++++++++++
drivers/virtio/virtio_ring.c | 15 +++++++++------
2 files changed, 23 insertions(+), 6 deletions(-)
diff --git a/include/linux/virtio_ring.h b/include/linux/virtio_ring.h
index 0135c16..8912189 100644
--- a/include/linux/virtio_ring.h
+++ b/include/linux/virtio_ring.h
@@ -47,6 +47,20 @@ static inline void virtio_wmb(bool weak_barriers)
wmb();
}
+static inline void virtio_store_mb(bool weak_barriers,
+ __virtio16 *p, __virtio16 v)
+{
+#ifdef CONFIG_SMP
+ if (weak_barriers)
+ smp_store_mb(*p, v);
+ else
+#endif
+ {
+ WRITE_ONCE(*p, v);
+ mb();
+ }
+}
+
static inline __virtio16 virtio_load_acquire(bool weak_barriers, __virtio16 *p)
{
if (!weak_barriers) {
diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index f822cab..b0aea67 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -517,10 +517,10 @@ void *virtqueue_get_buf(struct virtqueue *_vq, unsigned int *len)
/* If we expect an interrupt for the next entry, tell host
* by writing event index and flush out the write before
* the read in the next get_buf call. */
- if (!(vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) {
- vring_used_event(&vq->vring) = cpu_to_virtio16(_vq->vdev, vq->last_used_idx);
- virtio_mb(vq->weak_barriers);
- }
+ if (!(vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT))
+ virtio_store_mb(vq->weak_barriers,
+ &vring_used_event(&vq->vring),
+ cpu_to_virtio16(_vq->vdev, vq->last_used_idx));
#ifdef DEBUG
vq->last_add_time_valid = false;
@@ -653,8 +653,11 @@ bool virtqueue_enable_cb_delayed(struct virtqueue *_vq)
}
/* TODO: tune this threshold */
bufs = (u16)(vq->avail_idx_shadow - vq->last_used_idx) * 3 / 4;
- vring_used_event(&vq->vring) = cpu_to_virtio16(_vq->vdev, vq->last_used_idx + bufs);
- virtio_mb(vq->weak_barriers);
+
+ virtio_store_mb(vq->weak_barriers,
+ &vring_used_event(&vq->vring),
+ cpu_to_virtio16(_vq->vdev, vq->last_used_idx + bufs));
+
if (unlikely((u16)(virtio16_to_cpu(_vq->vdev, vq->vring.used->idx) - vq->last_used_idx) > bufs)) {
END_USE(vq);
return false;
--
MST
next reply other threads:[~2015-12-17 10:32 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-12-17 10:32 Michael S. Tsirkin [this message]
2015-12-17 10:32 [PATCH] virtio_ring: use smp_store_mb Michael S. Tsirkin
2015-12-17 10:52 ` Peter Zijlstra
2015-12-17 10:52 ` Peter Zijlstra
2015-12-17 13:16 ` Michael S. Tsirkin
2015-12-17 13:16 ` Michael S. Tsirkin
2015-12-17 13:57 ` Peter Zijlstra
2015-12-17 13:57 ` Peter Zijlstra
2015-12-17 14:33 ` Michael S. Tsirkin
2015-12-17 14:33 ` Michael S. Tsirkin
2015-12-17 14:39 ` Peter Zijlstra
2015-12-17 14:39 ` Peter Zijlstra
2015-12-17 14:43 ` Michael S. Tsirkin
2015-12-17 14:43 ` Michael S. Tsirkin
2015-12-17 11:22 ` Peter Zijlstra
2015-12-17 13:26 ` Michael S. Tsirkin
2015-12-17 13:26 ` Michael S. Tsirkin
2015-12-17 14:02 ` Peter Zijlstra
2015-12-17 14:02 ` Peter Zijlstra
2015-12-17 14:34 ` Michael S. Tsirkin
2015-12-17 14:34 ` Michael S. Tsirkin
2015-12-17 15:09 ` Peter Zijlstra
2015-12-17 15:09 ` Peter Zijlstra
2015-12-17 15:52 ` Will Deacon
2015-12-17 15:52 ` Will Deacon
2015-12-17 19:21 ` Michael S. Tsirkin
2015-12-17 19:21 ` Michael S. Tsirkin
2015-12-17 11:22 ` Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='1450347932-16325-1-git-send-email-mst__46919.7600767531$1450348391$gmane$org@redhat.com' \
--to=mst@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=qemu-devel@nongnu.org \
--cc=virtualization@lists.linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.