From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753638AbdERD6b (ORCPT ); Wed, 17 May 2017 23:58:31 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:56958 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753520AbdERD61 (ORCPT ); Wed, 17 May 2017 23:58:27 -0400 Date: Wed, 17 May 2017 20:58:19 -0700 From: "Paul E. McKenney" To: Peter Zijlstra Cc: Steven Rostedt , linux-kernel@vger.kernel.org, Thomas Gleixner , Ingo Molnar , Mathieu Desnoyers , Masami Hiramatsu Subject: Re: [RFC][PATCH 0/5] perf/tracing/cpuhotplug: Fix locking order Reply-To: paulmck@linux.vnet.ibm.com References: <20170512194956.GH4626@worktop.programming.kicks-ass.net> <20170512173448.5e2106b6@gandalf.local.home> <20170513134003.GA30927@linux.vnet.ibm.com> <20170515090318.amup4tbqujg27nrl@hirez.programming.kicks-ass.net> <20170515184043.GU3956@linux.vnet.ibm.com> <20170516081923.fxg67gawc44eg6i6@hirez.programming.kicks-ass.net> <20170516124606.GC3956@linux.vnet.ibm.com> <20170516142742.GA17599@linux.vnet.ibm.com> <20170517104010.5dfz7qqeigwbzb2u@hirez.programming.kicks-ass.net> <20170517145520.GI3956@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170517145520.GI3956@linux.vnet.ibm.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 x-cbid: 17051803-0056-0000-0000-0000036841D4 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00007080; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000212; SDB=6.00862043; UDB=6.00427622; IPR=6.00641702; BA=6.00005356; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015498; XFM=3.00000015; UTC=2017-05-18 03:58:22 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17051803-0057-0000-0000-0000079E6FB3 Message-Id: <20170518035819.GA2652@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-05-18_01:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1705180026 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, May 17, 2017 at 07:55:20AM -0700, Paul E. McKenney wrote: > On Wed, May 17, 2017 at 12:40:10PM +0200, Peter Zijlstra wrote: > > On Tue, May 16, 2017 at 07:27:42AM -0700, Paul E. McKenney wrote: > > > On Tue, May 16, 2017 at 05:46:06AM -0700, Paul E. McKenney wrote: [ . . . ] > > > > With the above change any chance of a race is gone and we don't need to > > worry about hotplug ordering too much. > > Nice!!! > > > Now I just need to do something about the swevent hash, because that's > > got a hole in now. > > On the styles, here is a more coherence list: > > 1. Use get_online_cpus(), but no CPU-hotplug notifiers. You can > use cpu_online() and cpu_is_offline() straightforwardly while > get_online_cpus() is in effect, but these two may only be used > for statistical/heuristic purposes otherwise. And I forgot about the disable-preemption trick... Of course you can also use cpu_online() and cpu_is_offline() more or less straightforwardly when preemption is disabled, but if your code is making use of subsystems that disable themselves early on outgoing CPUs or enable themselves late on incoming CPUs, life can be difficult. These subsystems can make things easier, for example, smp_call_function_single() will give a failure indication if it cannot help you because the CPU isn't all there. (RCU does OK, but could use a bit more improvement.) Also, you can use the disable-preemption trick to prevent a CPU from leaving (if you get there early enough in the offlining), but it won't necessarily prevent a CPU from showing up. Yes, there seem to be a fair number of notifiers that wait for grace periods (perhaps too many), but you cannot necessarily count on them being configured into the kernel. > 2a. Use CPU-hotplug notifiers, but not get_online_cpus(). > The notifiers maintain some sort of bitmask tracking which CPUs > are present from the viewpoint of this subsystem. This bitmask > provides exact CPU presence/absence indications, at least > assuming the appropriate lock is held. The cpu_online() and > cpu_is_offline() macros should be avoided, except possibly for > statistical/heuristic purposes. > > For your perf patch, the bitmask is a cpumask_var_t. For RCU, > it is the combination of the ->qsmaskinitnext fields of the leaf > rcu_node structures. > > 2b. Like 2a, except that instead of a bitmask, the CPU online/offline > information is implicit in other data structures. For example, > per-CPU structures might be allocated only when the corresponding > CPU is online, so that a given per-CPU pointer is non-NULL > iff the corresponding CPU is online. Here, disabling preemption will still prevent any CPU from responding to the stop-CPUs mechanism, but it cannot prevent your own notifier from executing. So care is required using cpu_online() and cpu_is_offline() even when preemption is disabled. Thanx, Paul > So I was (confusingly) initially using "style" to distinguish #1 on the > one hand from #2a and #2b on the other. Later on, I was using "style" > to distinguish #2a from #2b. > > At this point, I am not sure that it makes all that much sense to > distinguish 2a from 2b. Or you could argue that use of a cpumask_var_t > is its own substyle, with hand-crafted bitmasks (such as RCU's) are > separate substyles. Can't say that I care all that much about that > fine-grained gradiations. ;-) > > Thanx, Paul