From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1751625AbdG0O3z (ORCPT <rfc822;w@1wt.eu>);
        Thu, 27 Jul 2017 10:29:55 -0400
Received: from mail-pg0-f65.google.com ([74.125.83.65]:33925 "EHLO
        mail-pg0-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1751497AbdG0O3x (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Thu, 27 Jul 2017 10:29:53 -0400
Date: Thu, 27 Jul 2017 22:29:55 +0800
From: Boqun Feng <boqun.feng@gmail.com>
To: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>, linux-kernel@vger.kernel.org,
        mingo@kernel.org, jiangshanlai@gmail.com, dipankar@in.ibm.com,
        akpm@linux-foundation.org, mathieu.desnoyers@efficios.com,
        josh@joshtriplett.org, tglx@linutronix.de, rostedt@goodmis.org,
        dhowells@redhat.com, edumazet@google.com, fweisbec@gmail.com,
        oleg@redhat.com, will.deacon@arm.com
Subject: Re: [PATCH tip/core/rcu 4/5] sys_membarrier: Add expedited option
Message-ID: <20170727142955.bb2bdgbjcm745xwo@tardis>
References: <20170725202451.GC28975@worktop>
 <20170725211926.GA3730@linux.vnet.ibm.com>
 <20170725215510.GD28975@worktop>
 <20170725235936.GC3730@linux.vnet.ibm.com>
 <20170726074128.ybb3e4flnjkrpi74@hirez.programming.kicks-ass.net>
 <20170726154110.GN3730@linux.vnet.ibm.com>
 <20170727083003.ivb2fr47vepa2g6t@hirez.programming.kicks-ass.net>
 <20170727130816.GP3730@linux.vnet.ibm.com>
 <20170727135551.tg7cduw22krjzbvt@tardis>
 <20170727141633.GT3730@linux.vnet.ibm.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
        protocol="application/pgp-signature"; boundary="6gvmrpjsmh3cirdu"
Content-Disposition: inline
In-Reply-To: <20170727141633.GT3730@linux.vnet.ibm.com>
User-Agent: NeoMutt/20170609 (1.8.3)
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


--6gvmrpjsmh3cirdu
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Thu, Jul 27, 2017 at 07:16:33AM -0700, Paul E. McKenney wrote:
> On Thu, Jul 27, 2017 at 09:55:51PM +0800, Boqun Feng wrote:
> > Hi Paul,
> >=20
> > I have a side question out of curiosity:
> >=20
> > How does synchronize_sched() work properly for sys_membarrier()?
> >=20
> > sys_membarrier() requires every other CPU does a smp_mb() before it
> > returns, and I know synchronize_sched() will wait until all CPUs running
> > a kernel thread do a context-switch, which has a smp_mb(). However, I
> > believe sched flavor RCU treat CPU running a user thread as a quiesent
> > state, so synchronize_sched() could return without that CPU does a
> > context switch.=20
> >=20
> > So why could we use synchronize_sched() for sys_membarrier()?
> >=20
> > In particular, could the following happens?
> >=20
> > 	CPU 0:				CPU 1:
> > 	=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D	=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
> > 	<in user space>			<in user space>
> > 					{read Y}(reordered) <------------------------------+
> > 	store Y;			                                                   |
> > 					read X; --------------------------------------+    |
> > 	sys_membarrier():		<timer interrupt>                             |    |
> > 	  synchronize_sched();		  update_process_times(user): //user =3D=3D tr=
ue  |    |
> > 	  				    rcu_check_callbacks(usr):                 |    |
> > 					      if (user || ..) {                       |    |
> > 					        rcu_sched_qs()                        |    |
> > 						...                                   |    |
> > 						<report quesient state in softirq>    |    |
>=20
> The reporting of the quiescent state will acquire the leaf rcu_node
> structure's lock, with an smp_mb__after_unlock_lock(), which will
> one way or another be a full memory barrier.  So the reorderings
> cannot happen.
>=20
> Unless I am missing something subtle.  ;-)
>=20

Well, smp_mb__after_unlock_lock() in ARM64 is a no-op, and ARM64's lock
doesn't provide a smp_mb().

So my point is more like: synchronize_sched() happens to be a
sys_membarrier() because of some implementation detail, and if some day
we come up with a much cheaper way to implement sched flavor
RCU(hopefully!), synchronize_sched() may be not good for the job. So at
least, we'd better document this somewhere?

Regards,
Boqun

> 						Thanx, Paul
>=20
> > 					<return to user space>                        |    |
> > 					read Y; --------------------------------------+----+
> > 	store X;			                                              |
> > 					{read X}(reordered) <-------------------------+
> >=20
> > I assume the timer interrupt handler, which interrupts a user space and
> > reports a quiesent state for sched flavor RCU, may not have a smp_mb()
> > in some code path.
> >=20
> > I may miss something subtle, but it just not very obvious how
> > synchronize_sched() will guarantee a remote CPU running in userspace to
> > do a smp_mb() before it returns, this is at least not in RCU
> > requirements, right?
> >=20
> > Regards,
> > Boqun
>=20
>=20

--6gvmrpjsmh3cirdu
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQEzBAABCAAdFiEEj5IosQTPz8XU1wRHSXnow7UH+rgFAll5+N8ACgkQSXnow7UH
+rhSMAf/f+G/HbhWuQKVQAyJKGb5zPlA+RydGAWqRsVMvWbAarISc1ezxcV9SLdD
umLbyMbRB8HXhHc6u0FWr3eMbA3tNE0JOzoeNdeVrb9bQ7zofdBiVULlVuB+VO1c
ZmGfPeuoG3Apfa75ncENpOX4neipf0o85ihC0ETlYrIsfuvW1zYFwzio5HMPzhZE
nKsBLWA1Mhneed6s7nr85luOnycHMmmQXKfgHWSnAqPfSTlcZMsx2ErZhFHe1uRG
O/ajhDHhuYmykOhssj4my7VV5Cv9hGFQ/gP0QovXxCvYHMwIkCXFd/EU2PACcsj8
3xHBK5ph0hY32W9n1mRMERsmEYAEkA==
=qY3W
-----END PGP SIGNATURE-----

--6gvmrpjsmh3cirdu--