From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754435AbbJGOdf (ORCPT ); Wed, 7 Oct 2015 10:33:35 -0400 Received: from e34.co.us.ibm.com ([32.97.110.152]:33042 "EHLO e34.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754232AbbJGOdc (ORCPT ); Wed, 7 Oct 2015 10:33:32 -0400 X-IBM-Helo: d03dlp02.boulder.ibm.com X-IBM-MailFrom: paulmck@linux.vnet.ibm.com X-IBM-RcptTo: linux-kernel@vger.kernel.org Date: Wed, 7 Oct 2015 07:33:25 -0700 From: "Paul E. McKenney" To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, mingo@kernel.org, jiangshanlai@gmail.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@efficios.com, josh@joshtriplett.org, tglx@linutronix.de, rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com, dvhart@linux.intel.com, fweisbec@gmail.com, oleg@redhat.com, bobby.prani@gmail.com Subject: Re: [PATCH tip/core/rcu 02/18] rcu: Move rcu_report_exp_rnp() to allow consolidation Message-ID: <20151007143325.GF3910@linux.vnet.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com References: <20151006162907.GA12020@linux.vnet.ibm.com> <1444148977-14108-1-git-send-email-paulmck@linux.vnet.ibm.com> <1444148977-14108-2-git-send-email-paulmck@linux.vnet.ibm.com> <20151006202937.GX3604@twins.programming.kicks-ass.net> <20151006205850.GW3910@linux.vnet.ibm.com> <20151007075114.GW2881@worktop.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151007075114.GW2881@worktop.programming.kicks-ass.net> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 15100714-0017-0000-0000-00000E73D619 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Oct 07, 2015 at 09:51:14AM +0200, Peter Zijlstra wrote: > On Tue, Oct 06, 2015 at 01:58:50PM -0700, Paul E. McKenney wrote: > > On Tue, Oct 06, 2015 at 10:29:37PM +0200, Peter Zijlstra wrote: > > > On Tue, Oct 06, 2015 at 09:29:21AM -0700, Paul E. McKenney wrote: > > > > +static void __maybe_unused rcu_report_exp_rnp(struct rcu_state *rsp, > > > > + struct rcu_node *rnp, bool wake) > > > > +{ > > > > + unsigned long flags; > > > > + unsigned long mask; > > > > + > > > > + raw_spin_lock_irqsave(&rnp->lock, flags); > > > > > > Normally we require a comment with barriers, explaining the order and > > > the pairing etc.. :-) > > > > > > > + smp_mb__after_unlock_lock(); > > > > Hmmmm... That is not good. > > > > Worse yet, I am missing comments on most of the pre-existing barriers > > of this form. > > Yes I noticed.. :/ Will fix, though probably as a follow-up patch. Once I figure out what comment makes sense... > > The purpose is to enforce the heavy-weight grace-period memory-ordering > > guarantees documented in the synchronize_sched() header comment and > > elsewhere. > > > They pair with anything you might use to check for violation > > of these guarantees, or, simiarly, any ordering that you might use when > > relying on these guarantees. > > I'm sure you know what that means, but I've no clue ;-) That is, I > wouldn't know where to start looking in the RCU implementation to verify > the barrier is either needed or sufficient. Unless you mean _everywhere_ > :-) Pretty much everywhere. Let's take the usual RCU removal pattern as an example: void f1(struct foo *p) { list_del_rcu(p); synchronize_rcu_expedited(); kfree(p); } void f2(void) { struct foo *p; list_for_each_entry_rcu(p, &my_head, next) do_something_with(p); } So the synchronize_rcu_expedited() acts as an extremely heavyweight memory barrier that pairs with the rcu_dereference() inside of list_for_each_entry_rcu(). Easy enough, right? But what exactly within synchronize_rcu_expedited() provides the ordering? The answer is a web of lock-based critical sections and explicit memory barriers, with the one you called out as needing a comment being one of them. > > I could add something like "/* Enforce GP memory ordering. */" > > > > Or perhaps "/* See synchronize_sched() header. */" > > > > I do not propose reproducing the synchronize_sched() header on each > > of these. That would be verbose, even for me! ;-) > > > > Other thoughts? > > Well, this is an UNLOCK+LOCK on non-matching lock variables upgrade to > full barrier thing, right? Yep! > To me its not clear which UNLOCK we even match here. I've just read the > sync_sched() header, but that doesn't help me either, so referring to > that isn't really helpful either. Usually this pairs with an rcu_dereference() somewhere in the calling code. Some other task in the calling code, actually. > In any case, I don't want to make too big a fuzz here, but I just > stumbled over a lot of unannotated barriers and figured I ought to say > something about it. I do need to better document how this works, no two ways about it. Thanx, Paul