From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754540Ab3GXWJO (ORCPT ); Wed, 24 Jul 2013 18:09:14 -0400 Received: from e35.co.us.ibm.com ([32.97.110.153]:38945 "EHLO e35.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753394Ab3GXWJJ (ORCPT ); Wed, 24 Jul 2013 18:09:09 -0400 Date: Wed, 24 Jul 2013 15:09:02 -0700 From: "Paul E. McKenney" To: Frederic Weisbecker Cc: linux-kernel@vger.kernel.org, mingo@elte.hu, laijs@cn.fujitsu.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@polymtl.ca, josh@joshtriplett.org, niv@us.ibm.com, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com, darren@dvhart.com, sbw@mit.edu Subject: Re: [PATCH RFC nohz_full 6/7] nohz_full: Add full-system-idle state machine Message-ID: <20130724220902.GA3889@linux.vnet.ibm.com> Reply-To: paulmck@linux.vnet.ibm.com References: <20130718004141.GI4161@linux.vnet.ibm.com> <20130718013259.GA7398@somewhere> <20130718033921.GL4161@linux.vnet.ibm.com> <20130718142450.GB7398@somewhere> <20130718164749.GV4161@linux.vnet.ibm.com> <20130718224620.GF7398@somewhere> <20130719002408.GB21367@linux.vnet.ibm.com> <20130719021207.GA19491@somewhere> <20130719050625.GC21367@linux.vnet.ibm.com> <20130724180903.GB23431@somewhere> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20130724180903.GB23431@somewhere> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-MML: No X-Content-Scanned: Fidelis XPS MAILER x-cbid: 13072422-4834-0000-0000-0000095ECD51 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Jul 24, 2013 at 08:09:04PM +0200, Frederic Weisbecker wrote: > On Thu, Jul 18, 2013 at 10:06:25PM -0700, Paul E. McKenney wrote: > > > Lets summarize the last sequence, the following happens ordered by time: > > > > > > CPU 0 CPU 1 > > > > > > cmpxchg(&full_sysidle_state, > > > RCU_SYSIDLE_SHORT, > > > RCU_SYSIDLE_LONG); > > > > > > smp_mb() //cmpxchg > > > > > > atomic_read(rdtp(1)->dynticks_idle) > > > > > > //CPU 0 goes to sleep > > > //CPU 1 wakes up > > > atomic_inc(rdtp(1)->dynticks_idle) > > > > > > smp_mb() > > > > > > ACCESS_ONCE(full_sysidle_state) > > > > > > > > > Are you suggesting that because the CPU 1 executes its atomic_inc() _after_ (in terms > > > of absolute time) the atomic_read of CPU 0, the ordering settled in both sides guarantees > > > that the value read from CPU 1 is the one from the cmpxchg that precedes the atomic_read, > > > or FULL or FULL_NOTED that happen later. > > > > > > If so that's a big lesson for me. > > > > It is not absolute time that matters. Instead, it is the fact that > > CPU 0, when reading from ->dynticks_idle, read the old value before the > > atomic_inc(). Therefore, anything CPU 0 did before that memory barrier > > preceding CPU 0's read must come before anything CPU 1 did after that > > memory barrier following the atomic_inc(). For this to work, there > > must be some access to the same variable on each CPU. > > Aren't we in the following situation? > > CPU 0 CPU 1 > > STORE A STORE B > LOAD B LOAD A > > > If so and referring to your perfbook, this is an "ears to mouth" situation. > And it seems to describe there is no strong guarantee in that situation. "Yes" to the first, but on modern hardware, "no" to the second. The key paragraph is Section 12.2.4.5: The following pairings from Table 12.1 can be used on modern hardware, but might fail on some systems that were produced in the 1990s. However, these can safely be used on all mainstream hardware introduced since the year 2000. That said, you are not the first to be confused by this, so I might need to rework this section to make it clear that each can in fact be used on modern hardware. If you happen to have an old Sequent NUMA-Q or Symmetry box lying around, things are a bit different. On the other hand, I don't believe that any of these old boxes are still running Linux. (Hey, I am as sentimental as the next guy, but there are limits!) I updated this section and pushed it, please let me know if this helps! Thanx, Paul > > Or, if you must think in terms of time, you need a separate independent > > timeline for each variable, with no direct mapping from one timeline to > > another, except resulting from memory-barrier interactions. > > > > Thanx, Paul > > >