From: Martin Schwidefsky <schwidefsky@de.ibm.com>
To: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>,
linux-kernel@vger.kernel.org, Arnd Bergmann <arnd@arndb.de>,
linux-arch@vger.kernel.org,
Andrew Cooper <andrew.cooper3@citrix.com>,
virtualization@lists.linux-foundation.org,
Stefano Stabellini <stefano.stabellini@eu.citrix.com>,
Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@elte.hu>,
"H. Peter Anvin" <hpa@zytor.com>,
David Miller <davem@davemloft.net>,
linux-ia64@vger.kernel.org, linuxppc-dev@lists.ozlabs.org,
linux-s390@vger.kernel.org, sparclinux@vger.kernel.org,
linux-arm-kernel@lists.infradead.org,
linux-metag@vger.kernel.org, linux-mips@linux-mips.org,
x86@kernel.org, user-mode-linux-devel@lists.sourceforge.net,
adi-buildroot-devel@lists.sourceforge.net,
linux-sh@vger.kernel.org, linux-xtensa@linux-xtensa.org,
xen-devel@lists.xenproject.org,
Heiko Carstens <heiko.carstens@de.ibm.com>,
Ingo Molnar <mingo@kernel.org>,
Davidlohr Bueso <dave@stgolabs.net>,
Christian Borntraeger <borntraeger@de.ibm.com>,
Andrey Konovalov <andreyknvl@google.com>,
Carsten Otte <cotte@de.ibm.com>,
Christian Ehrhardt <ehrhardt@de.ibm.com>
Subject: Re: [PATCH v2 22/32] s390: define __smp_xxx
Date: Tue, 5 Jan 2016 13:08:52 +0100 [thread overview]
Message-ID: <20160105130852.11148a7f@mschwide> (raw)
In-Reply-To: <20160105105335-mutt-send-email-mst@redhat.com>
On Tue, 5 Jan 2016 11:30:19 +0200
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Tue, Jan 05, 2016 at 09:13:19AM +0100, Martin Schwidefsky wrote:
> > On Mon, 4 Jan 2016 22:18:58 +0200
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >
> > > On Mon, Jan 04, 2016 at 02:45:25PM +0100, Peter Zijlstra wrote:
> > > > On Thu, Dec 31, 2015 at 09:08:38PM +0200, Michael S. Tsirkin wrote:
> > > > > This defines __smp_xxx barriers for s390,
> > > > > for use by virtualization.
> > > > >
> > > > > Some smp_xxx barriers are removed as they are
> > > > > defined correctly by asm-generic/barriers.h
> > > > >
> > > > > Note: smp_mb, smp_rmb and smp_wmb are defined as full barriers
> > > > > unconditionally on this architecture.
> > > > >
> > > > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > > > Acked-by: Arnd Bergmann <arnd@arndb.de>
> > > > > ---
> > > > > arch/s390/include/asm/barrier.h | 15 +++++++++------
> > > > > 1 file changed, 9 insertions(+), 6 deletions(-)
> > > > >
> > > > > diff --git a/arch/s390/include/asm/barrier.h b/arch/s390/include/asm/barrier.h
> > > > > index c358c31..fbd25b2 100644
> > > > > --- a/arch/s390/include/asm/barrier.h
> > > > > +++ b/arch/s390/include/asm/barrier.h
> > > > > @@ -26,18 +26,21 @@
> > > > > #define wmb() barrier()
> > > > > #define dma_rmb() mb()
> > > > > #define dma_wmb() mb()
> > > > > -#define smp_mb() mb()
> > > > > -#define smp_rmb() rmb()
> > > > > -#define smp_wmb() wmb()
> > > > > -
> > > > > -#define smp_store_release(p, v) \
> > > > > +#define __smp_mb() mb()
> > > > > +#define __smp_rmb() rmb()
> > > > > +#define __smp_wmb() wmb()
> > > > > +#define smp_mb() __smp_mb()
> > > > > +#define smp_rmb() __smp_rmb()
> > > > > +#define smp_wmb() __smp_wmb()
> > > >
> > > > Why define the smp_*mb() primitives here? Would not the inclusion of
> > > > asm-generic/barrier.h do this?
> > >
> > > No because the generic one is a nop on !SMP, this one isn't.
> > >
> > > Pls note this patch is just reordering code without making
> > > functional changes.
> > > And at the moment, on s390 smp_xxx barriers are always non empty.
> >
> > The s390 kernel is SMP to 99.99%, we just didn't bother with a
> > non-smp variant for the memory-barriers. If the generic header
> > is used we'd get the non-smp version for free. It will save a
> > small amount of text space for CONFIG_SMP=n.
>
> OK, so I'll queue a patch to do this then?
Yes please.
> Just to make sure: the question would be, are smp_xxx barriers ever used
> in s390 arch specific code to flush in/out memory accesses for
> synchronization with the hypervisor?
>
> I went over s390 arch code and it seems to me the answer is no
> (except of course for virtio).
Correct. Guest to host communication either uses instructions which
imply a memory barrier or QDIO which uses atomics.
> But I also see a lot of weirdness on this architecture.
Mostly historical, s390 actually is one of the easiest architectures in
regard to memory barriers.
> I found these calls:
>
> arch/s390/include/asm/bitops.h: smp_mb__before_atomic();
> arch/s390/include/asm/bitops.h: smp_mb();
>
> Not used in arch specific code so this is likely OK.
This has been introduced with git commit 5402ea6af11dc5a9, the smp_mb
and smp_mb__before_atomic are used in clear_bit_unlock and
__clear_bit_unlock which are 1:1 copies from the code in
include/asm-generic/bitops/lock.h. Only test_and_set_bit_lock differs
from the generic implementation.
> arch/s390/kernel/vdso.c: smp_mb();
>
> Looking at
> Author: Christian Borntraeger <borntraeger@de.ibm.com>
> Date: Fri Sep 11 16:23:06 2015 +0200
>
> s390/vdso: use correct memory barrier
>
> By definition smp_wmb only orders writes against writes. (Finish all
> previous writes, and do not start any future write). To protect the
> vdso init code against early reads on other CPUs, let's use a full
> smp_mb at the end of vdso init. As right now smp_wmb is implemented
> as full serialization, this needs no stable backport, but this change
> will be necessary if we reimplement smp_wmb.
>
> ok from hypervisor point of view, but it's also strange:
> 1. why isn't this paired with another mb somewhere?
> this seems to violate barrier pairing rules.
> 2. how does smp_mb protect against early reads on other CPUs?
> It normally does not: it orders reads from this CPU versus writes
> from same CPU. But init code does not appear to read anything.
> Maybe this is some s390 specific trick?
>
> I could not figure out the above commit.
That smp_mb can be removed. The initial s390 vdso code is heavily influenced
by the powerpc version which does have a smp_wmb in vdso_init right before
the vdso_ready=1 assignment. s390 has no need for that.
>
> arch/s390/kvm/kvm-s390.c: smp_mb();
>
> Does not appear to be paired with anything.
This one does not make sense to me. Imho can be removed as well.
> arch/s390/lib/spinlock.c: smp_mb();
> arch/s390/lib/spinlock.c: smp_mb();
>
> Seems ok, and appears paired properly.
> Just to make sure - spinlock is not paravirtualized on s390, is it?
s390 just uses the compare-and-swap instruction for the basic lock/unlock
operation, this implies the memory barrier. We do call the hypervisor for
contended locks if the lock can not be acquired after a number of retries.
A while ago we did play with ticket spinlocks but they behaved badly in
out usual virtualized environments. If we find the time we might take a
closer look at the para-virtualized queued spinlocks.
> rch/s390/kernel/time.c: smp_wmb();
> arch/s390/kernel/time.c: smp_wmb();
> arch/s390/kernel/time.c: smp_wmb();
> arch/s390/kernel/time.c: smp_wmb();
>
> It's all around vdso, so I'm guessing userspace is using this,
> this is why there's no pairing.
Correct, this is the update count mechanics with the vdso user space code.
> > > Some of this could be sub-optimal, but
> > > since on s390 Linux always runs on a hypervisor,
> > > I am not sure it's safe to use the generic version -
> > > in other words, it just might be that for s390 smp_ and virt_
> > > barriers must be equivalent.
> >
> > The definition of the memory barriers is independent from the fact
> > if the system is running on an hypervisor or not.
> > Is there really
> > an architecture where you need special virt_xxx barriers?!?
>
> It is whenever host and guest or two guests access memory at
> the same time.
>
> The optimization where smp_xxx barriers are compiled out when
> CONFIG_SMP is cleared means that two UP guests running
> on an SMP host can not use smp_xxx barriers for communication.
>
> See explanation here:
> http://thread.gmane.org/gmane.linux.kernel.virtualization/26555
Got it, makes sense.
--
blue skies,
Martin.
"Reality continues to ruin my life." - Calvin.
next prev parent reply other threads:[~2016-01-05 12:09 UTC|newest]
Thread overview: 85+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-12-31 19:05 [PATCH v2 00/34] arch: barrier cleanup + barriers for virt Michael S. Tsirkin
2015-12-31 19:05 ` [PATCH v2 01/32] lcoking/barriers, arch: Use smp barriers in smp_store_release() Michael S. Tsirkin
2015-12-31 19:05 ` [PATCH v2 02/32] asm-generic: guard smp_store_release/load_acquire Michael S. Tsirkin
2015-12-31 19:06 ` [PATCH v2 03/32] ia64: rename nop->iosapic_nop Michael S. Tsirkin
2015-12-31 19:06 ` [PATCH v2 04/32] ia64: reuse asm-generic/barrier.h Michael S. Tsirkin
2015-12-31 19:06 ` [PATCH v2 05/32] powerpc: " Michael S. Tsirkin
2015-12-31 19:06 ` [PATCH v2 06/32] s390: " Michael S. Tsirkin
2016-01-04 13:20 ` Peter Zijlstra
2016-01-04 15:03 ` Martin Schwidefsky
2016-01-04 20:42 ` Michael S. Tsirkin
2016-01-05 8:03 ` Martin Schwidefsky
2016-01-04 20:34 ` Michael S. Tsirkin
2015-12-31 19:06 ` [PATCH v2 07/32] sparc: " Michael S. Tsirkin
2015-12-31 19:43 ` David Miller
2015-12-31 19:06 ` [PATCH v2 08/32] arm: " Michael S. Tsirkin
2016-01-02 11:20 ` Russell King - ARM Linux
2015-12-31 19:06 ` [PATCH v2 09/32] arm64: " Michael S. Tsirkin
2015-12-31 19:07 ` [PATCH v2 10/32] metag: " Michael S. Tsirkin
2016-01-04 23:24 ` James Hogan
2015-12-31 19:07 ` [PATCH v2 11/32] mips: " Michael S. Tsirkin
2016-01-04 13:26 ` Peter Zijlstra
2015-12-31 19:07 ` [PATCH v2 12/32] x86/um: " Michael S. Tsirkin
2016-01-05 23:12 ` Richard Weinberger
2015-12-31 19:07 ` [PATCH v2 13/32] x86: " Michael S. Tsirkin
2015-12-31 19:07 ` [PATCH v2 14/32] asm-generic: add __smp_xxx wrappers Michael S. Tsirkin
2015-12-31 19:07 ` [PATCH v2 15/32] powerpc: define __smp_xxx Michael S. Tsirkin
2016-01-05 1:36 ` Boqun Feng
2016-01-05 8:51 ` Michael S. Tsirkin
2016-01-05 9:53 ` Boqun Feng
2016-01-05 16:16 ` Michael S. Tsirkin
2016-01-06 1:51 ` Boqun Feng
2016-01-06 20:23 ` Michael S. Tsirkin
2016-01-07 0:43 ` Boqun Feng
2015-12-31 19:07 ` [PATCH v2 16/32] arm64: " Michael S. Tsirkin
2015-12-31 19:07 ` [PATCH v2 17/32] arm: " Michael S. Tsirkin
2016-01-02 11:24 ` Russell King - ARM Linux
2016-01-03 9:12 ` Michael S. Tsirkin
2016-01-04 13:36 ` Peter Zijlstra
2016-01-04 13:54 ` Peter Zijlstra
2016-01-04 13:59 ` Russell King - ARM Linux
2016-01-05 14:38 ` Michael S. Tsirkin
2016-01-04 20:39 ` Michael S. Tsirkin
2016-01-04 20:12 ` Michael S. Tsirkin
2015-12-31 19:08 ` [PATCH v2 18/32] blackfin: " Michael S. Tsirkin
2015-12-31 19:08 ` [PATCH v2 19/32] ia64: " Michael S. Tsirkin
2015-12-31 19:08 ` [PATCH v2 20/32] metag: " Michael S. Tsirkin
2016-01-04 13:41 ` Peter Zijlstra
2016-01-04 15:25 ` James Hogan
2016-01-04 15:30 ` Peter Zijlstra
2016-01-04 16:04 ` James Hogan
2016-01-05 0:09 ` James Hogan
2016-01-11 11:10 ` Michael S. Tsirkin
2015-12-31 19:08 ` [PATCH v2 21/32] mips: " Michael S. Tsirkin
2015-12-31 19:08 ` [PATCH v2 22/32] s390: " Michael S. Tsirkin
2016-01-04 13:45 ` Peter Zijlstra
2016-01-04 20:18 ` Michael S. Tsirkin
2016-01-05 8:13 ` Martin Schwidefsky
2016-01-05 9:30 ` Michael S. Tsirkin
2016-01-05 12:08 ` Martin Schwidefsky [this message]
2016-01-05 13:04 ` Michael S. Tsirkin
2016-01-05 14:21 ` Martin Schwidefsky
2016-01-05 15:39 ` Christian Borntraeger
2016-01-05 16:04 ` Michael S. Tsirkin
2015-12-31 19:08 ` [PATCH v2 23/32] sh: define __smp_xxx, fix smp_store_mb for !SMP Michael S. Tsirkin
2015-12-31 19:08 ` [PATCH v2 24/32] sparc: define __smp_xxx Michael S. Tsirkin
2015-12-31 19:44 ` David Miller
2015-12-31 19:09 ` [PATCH v2 25/32] tile: " Michael S. Tsirkin
2015-12-31 19:09 ` [PATCH v2 26/32] xtensa: " Michael S. Tsirkin
2015-12-31 19:09 ` [PATCH v2 27/32] x86: " Michael S. Tsirkin
2015-12-31 19:09 ` [PATCH v2 28/32] asm-generic: implement virt_xxx memory barriers Michael S. Tsirkin
2015-12-31 19:09 ` [PATCH v2 29/32] Revert "virtio_ring: Update weak barriers to use dma_wmb/rmb" Michael S. Tsirkin
2015-12-31 19:09 ` [PATCH v2 30/32] virtio_ring: update weak barriers to use __smp_XXX Michael S. Tsirkin
2016-01-01 10:21 ` [PATCH v2 30/32] virtio_ring: update weak barriers to use __smp_xxx Michael S. Tsirkin
2015-12-31 19:09 ` [PATCH v2 31/32] sh: support a 2-byte smp_store_mb Michael S. Tsirkin
2016-01-04 14:05 ` Peter Zijlstra
2015-12-31 19:09 ` [PATCH v2 32/32] virtio_ring: use virt_store_mb Michael S. Tsirkin
2016-01-01 17:23 ` Sergei Shtylyov
2016-01-03 9:01 ` Michael S. Tsirkin
2015-12-31 19:10 ` [PATCH v2 33/34] xenbus: use virt_xxx barriers Michael S. Tsirkin
2016-01-04 11:32 ` [Xen-devel] " David Vrabel
2016-01-04 12:03 ` Stefano Stabellini
2016-01-04 14:09 ` Peter Zijlstra
2015-12-31 19:10 ` [PATCH v2 34/34] xen/io: " Michael S. Tsirkin
2016-01-04 11:32 ` [Xen-devel] " David Vrabel
2016-01-04 12:05 ` Stefano Stabellini
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20160105130852.11148a7f@mschwide \
--to=schwidefsky@de.ibm.com \
--cc=adi-buildroot-devel@lists.sourceforge.net \
--cc=andrew.cooper3@citrix.com \
--cc=andreyknvl@google.com \
--cc=arnd@arndb.de \
--cc=borntraeger@de.ibm.com \
--cc=cotte@de.ibm.com \
--cc=dave@stgolabs.net \
--cc=davem@davemloft.net \
--cc=ehrhardt@de.ibm.com \
--cc=heiko.carstens@de.ibm.com \
--cc=hpa@zytor.com \
--cc=linux-arch@vger.kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-ia64@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-metag@vger.kernel.org \
--cc=linux-mips@linux-mips.org \
--cc=linux-s390@vger.kernel.org \
--cc=linux-sh@vger.kernel.org \
--cc=linux-xtensa@linux-xtensa.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=mingo@elte.hu \
--cc=mingo@kernel.org \
--cc=mst@redhat.com \
--cc=peterz@infradead.org \
--cc=sparclinux@vger.kernel.org \
--cc=stefano.stabellini@eu.citrix.com \
--cc=tglx@linutronix.de \
--cc=user-mode-linux-devel@lists.sourceforge.net \
--cc=virtualization@lists.linux-foundation.org \
--cc=x86@kernel.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).