From: "Michael S. Tsirkin" <mst@redhat.com> To: linux-kernel@vger.kernel.org Cc: virtualization@lists.linux-foundation.org, Jason Wang <jasowang@redhat.com>, qemu-devel@nongnu.org Subject: [PATCH] virtio_ring: use smp_store_mb Date: Thu, 17 Dec 2015 12:32:53 +0200 [thread overview] Message-ID: <1450347932-16325-1-git-send-email-mst@redhat.com> (raw) We need a full barrier after writing out event index, using smp_store_mb there seems better than open-coding. As usual, we need a wrapper to account for strong barriers/non smp. It's tempting to use this in vhost as well, for that, we'll need a variant of smp_store_mb that works on __user pointers. Signed-off-by: Michael S. Tsirkin <mst@redhat.com> --- Seems to give a speedup on my box but I'm less sure about this one. E.g. as xchng faster than mfence on all/most intel CPUs? Anyone has an opinion? include/linux/virtio_ring.h | 14 ++++++++++++++ drivers/virtio/virtio_ring.c | 15 +++++++++------ 2 files changed, 23 insertions(+), 6 deletions(-) diff --git a/include/linux/virtio_ring.h b/include/linux/virtio_ring.h index 0135c16..8912189 100644 --- a/include/linux/virtio_ring.h +++ b/include/linux/virtio_ring.h @@ -47,6 +47,20 @@ static inline void virtio_wmb(bool weak_barriers) wmb(); } +static inline void virtio_store_mb(bool weak_barriers, + __virtio16 *p, __virtio16 v) +{ +#ifdef CONFIG_SMP + if (weak_barriers) + smp_store_mb(*p, v); + else +#endif + { + WRITE_ONCE(*p, v); + mb(); + } +} + static inline __virtio16 virtio_load_acquire(bool weak_barriers, __virtio16 *p) { if (!weak_barriers) { diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index f822cab..b0aea67 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -517,10 +517,10 @@ void *virtqueue_get_buf(struct virtqueue *_vq, unsigned int *len) /* If we expect an interrupt for the next entry, tell host * by writing event index and flush out the write before * the read in the next get_buf call. */ - if (!(vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) { - vring_used_event(&vq->vring) = cpu_to_virtio16(_vq->vdev, vq->last_used_idx); - virtio_mb(vq->weak_barriers); - } + if (!(vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) + virtio_store_mb(vq->weak_barriers, + &vring_used_event(&vq->vring), + cpu_to_virtio16(_vq->vdev, vq->last_used_idx)); #ifdef DEBUG vq->last_add_time_valid = false; @@ -653,8 +653,11 @@ bool virtqueue_enable_cb_delayed(struct virtqueue *_vq) } /* TODO: tune this threshold */ bufs = (u16)(vq->avail_idx_shadow - vq->last_used_idx) * 3 / 4; - vring_used_event(&vq->vring) = cpu_to_virtio16(_vq->vdev, vq->last_used_idx + bufs); - virtio_mb(vq->weak_barriers); + + virtio_store_mb(vq->weak_barriers, + &vring_used_event(&vq->vring), + cpu_to_virtio16(_vq->vdev, vq->last_used_idx + bufs)); + if (unlikely((u16)(virtio16_to_cpu(_vq->vdev, vq->vring.used->idx) - vq->last_used_idx) > bufs)) { END_USE(vq); return false; -- MST
WARNING: multiple messages have this Message-ID (diff)
From: "Michael S. Tsirkin" <mst@redhat.com> To: linux-kernel@vger.kernel.org Cc: Jason Wang <jasowang@redhat.com>, qemu-devel@nongnu.org, virtualization@lists.linux-foundation.org Subject: [Qemu-devel] [PATCH] virtio_ring: use smp_store_mb Date: Thu, 17 Dec 2015 12:32:53 +0200 [thread overview] Message-ID: <1450347932-16325-1-git-send-email-mst@redhat.com> (raw) We need a full barrier after writing out event index, using smp_store_mb there seems better than open-coding. As usual, we need a wrapper to account for strong barriers/non smp. It's tempting to use this in vhost as well, for that, we'll need a variant of smp_store_mb that works on __user pointers. Signed-off-by: Michael S. Tsirkin <mst@redhat.com> --- Seems to give a speedup on my box but I'm less sure about this one. E.g. as xchng faster than mfence on all/most intel CPUs? Anyone has an opinion? include/linux/virtio_ring.h | 14 ++++++++++++++ drivers/virtio/virtio_ring.c | 15 +++++++++------ 2 files changed, 23 insertions(+), 6 deletions(-) diff --git a/include/linux/virtio_ring.h b/include/linux/virtio_ring.h index 0135c16..8912189 100644 --- a/include/linux/virtio_ring.h +++ b/include/linux/virtio_ring.h @@ -47,6 +47,20 @@ static inline void virtio_wmb(bool weak_barriers) wmb(); } +static inline void virtio_store_mb(bool weak_barriers, + __virtio16 *p, __virtio16 v) +{ +#ifdef CONFIG_SMP + if (weak_barriers) + smp_store_mb(*p, v); + else +#endif + { + WRITE_ONCE(*p, v); + mb(); + } +} + static inline __virtio16 virtio_load_acquire(bool weak_barriers, __virtio16 *p) { if (!weak_barriers) { diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index f822cab..b0aea67 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -517,10 +517,10 @@ void *virtqueue_get_buf(struct virtqueue *_vq, unsigned int *len) /* If we expect an interrupt for the next entry, tell host * by writing event index and flush out the write before * the read in the next get_buf call. */ - if (!(vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) { - vring_used_event(&vq->vring) = cpu_to_virtio16(_vq->vdev, vq->last_used_idx); - virtio_mb(vq->weak_barriers); - } + if (!(vq->avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) + virtio_store_mb(vq->weak_barriers, + &vring_used_event(&vq->vring), + cpu_to_virtio16(_vq->vdev, vq->last_used_idx)); #ifdef DEBUG vq->last_add_time_valid = false; @@ -653,8 +653,11 @@ bool virtqueue_enable_cb_delayed(struct virtqueue *_vq) } /* TODO: tune this threshold */ bufs = (u16)(vq->avail_idx_shadow - vq->last_used_idx) * 3 / 4; - vring_used_event(&vq->vring) = cpu_to_virtio16(_vq->vdev, vq->last_used_idx + bufs); - virtio_mb(vq->weak_barriers); + + virtio_store_mb(vq->weak_barriers, + &vring_used_event(&vq->vring), + cpu_to_virtio16(_vq->vdev, vq->last_used_idx + bufs)); + if (unlikely((u16)(virtio16_to_cpu(_vq->vdev, vq->vring.used->idx) - vq->last_used_idx) > bufs)) { END_USE(vq); return false; -- MST
next reply other threads:[~2015-12-17 10:32 UTC|newest] Thread overview: 75+ messages / expand[flat|nested] mbox.gz Atom feed top 2015-12-17 10:32 Michael S. Tsirkin [this message] 2015-12-17 10:32 ` [Qemu-devel] [PATCH] virtio_ring: use smp_store_mb Michael S. Tsirkin 2015-12-17 10:52 ` Peter Zijlstra 2015-12-17 10:52 ` Peter Zijlstra 2015-12-17 10:52 ` [Qemu-devel] " Peter Zijlstra 2015-12-17 13:16 ` Michael S. Tsirkin 2015-12-17 13:16 ` Michael S. Tsirkin 2015-12-17 13:16 ` [Qemu-devel] " Michael S. Tsirkin 2015-12-17 13:57 ` Peter Zijlstra 2015-12-17 13:57 ` Peter Zijlstra 2015-12-17 13:57 ` [Qemu-devel] " Peter Zijlstra 2015-12-17 14:33 ` Michael S. Tsirkin 2015-12-17 14:33 ` Michael S. Tsirkin 2015-12-17 14:33 ` [Qemu-devel] " Michael S. Tsirkin 2015-12-17 14:39 ` Peter Zijlstra 2015-12-17 14:39 ` Peter Zijlstra 2015-12-17 14:39 ` [Qemu-devel] " Peter Zijlstra 2015-12-17 14:43 ` Michael S. Tsirkin 2015-12-17 14:43 ` Michael S. Tsirkin 2015-12-17 14:43 ` [Qemu-devel] " Michael S. Tsirkin 2015-12-20 9:25 ` new barrier type for paravirt (was Re: [PATCH] virtio_ring: use smp_store_mb) Michael S. Tsirkin 2015-12-20 9:25 ` Michael S. Tsirkin 2015-12-20 9:25 ` [Qemu-devel] " Michael S. Tsirkin 2015-12-20 17:07 ` Andrew Cooper 2015-12-20 17:07 ` [Xen-devel] " Andrew Cooper 2015-12-20 17:07 ` Andrew Cooper 2015-12-20 17:07 ` [Qemu-devel] " Andrew Cooper 2015-12-20 19:59 ` Peter Zijlstra 2015-12-20 19:59 ` Peter Zijlstra 2015-12-20 19:59 ` [Qemu-devel] " Peter Zijlstra 2015-12-21 7:10 ` Michael S. Tsirkin 2015-12-21 7:10 ` [Xen-devel] " Michael S. Tsirkin 2015-12-21 7:10 ` Michael S. Tsirkin 2015-12-21 7:10 ` [Qemu-devel] " Michael S. Tsirkin 2015-12-21 7:22 ` [PATCH RFC] smp_store_mb should use smp_mb Michael S. Tsirkin 2015-12-21 7:22 ` Michael S. Tsirkin 2015-12-21 7:22 ` Michael S. Tsirkin 2015-12-21 7:22 ` [Qemu-devel] " Michael S. Tsirkin 2015-12-20 19:59 ` new barrier type for paravirt (was Re: [PATCH] virtio_ring: use smp_store_mb) Peter Zijlstra 2015-12-21 10:47 ` David Vrabel 2015-12-21 10:47 ` [Xen-devel] " David Vrabel 2015-12-21 10:47 ` David Vrabel 2015-12-21 10:47 ` [Qemu-devel] " David Vrabel 2015-12-21 11:52 ` Michael S. Tsirkin 2015-12-21 11:52 ` [Xen-devel] " Michael S. Tsirkin 2015-12-21 11:52 ` Michael S. Tsirkin 2015-12-21 11:52 ` [Qemu-devel] " Michael S. Tsirkin 2015-12-21 14:50 ` Stefano Stabellini 2015-12-21 14:50 ` Stefano Stabellini 2015-12-21 14:50 ` Stefano Stabellini 2015-12-21 14:50 ` Stefano Stabellini 2015-12-21 14:50 ` [Qemu-devel] " Stefano Stabellini 2015-12-20 9:25 ` Michael S. Tsirkin 2015-12-17 11:22 ` [PATCH] virtio_ring: use smp_store_mb Peter Zijlstra 2015-12-17 11:22 ` [Qemu-devel] " Peter Zijlstra 2015-12-17 13:26 ` Michael S. Tsirkin 2015-12-17 13:26 ` Michael S. Tsirkin 2015-12-17 13:26 ` [Qemu-devel] " Michael S. Tsirkin 2015-12-17 14:02 ` Peter Zijlstra 2015-12-17 14:02 ` Peter Zijlstra 2015-12-17 14:02 ` [Qemu-devel] " Peter Zijlstra 2015-12-17 14:34 ` Michael S. Tsirkin 2015-12-17 14:34 ` Michael S. Tsirkin 2015-12-17 14:34 ` [Qemu-devel] " Michael S. Tsirkin 2015-12-17 15:09 ` Peter Zijlstra 2015-12-17 15:09 ` Peter Zijlstra 2015-12-17 15:09 ` [Qemu-devel] " Peter Zijlstra 2015-12-17 15:52 ` Will Deacon 2015-12-17 15:52 ` Will Deacon 2015-12-17 15:52 ` [Qemu-devel] " Will Deacon 2015-12-17 19:21 ` Michael S. Tsirkin 2015-12-17 19:21 ` Michael S. Tsirkin 2015-12-17 19:21 ` [Qemu-devel] " Michael S. Tsirkin 2015-12-17 11:22 ` Peter Zijlstra -- strict thread matches above, loose matches on Subject: below -- 2015-12-17 10:32 Michael S. Tsirkin
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1450347932-16325-1-git-send-email-mst@redhat.com \ --to=mst@redhat.com \ --cc=jasowang@redhat.com \ --cc=linux-kernel@vger.kernel.org \ --cc=qemu-devel@nongnu.org \ --cc=virtualization@lists.linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.