Re: [PATCH 2/2] [PATCH] sched: Add smp_rmb() in task rq locking cycles

From: Chris Metcalf <cmetcalf@ezchip.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
	Manfred Spraul <manfred@colorfullife.com>,
	Oleg Nesterov <oleg@redhat.com>,
	Kirill Tkhai <ktkhai@parallels.com>,
	<linux-kernel@vger.kernel.org>, Ingo Molnar <mingo@redhat.com>,
	Josh Poimboeuf <jpoimboe@redhat.com>
Subject: Re: [PATCH 2/2] [PATCH] sched: Add smp_rmb() in task rq locking cycles
Date: Tue, 28 Apr 2015 12:58:55 -0400	[thread overview]
Message-ID: <553FBC4F.3070104@ezchip.com> (raw)
In-Reply-To: <20150428164033.GJ5029@twins.programming.kicks-ass.net>

On 04/28/2015 12:40 PM, Peter Zijlstra wrote:
> On Tue, Apr 28, 2015 at 11:53:21AM -0400, Chris Metcalf wrote:
>
>> The reason we use two 32-bit fields on tilepro is that the only available
>> atomic instruction is tns (test and set), which sets a 32-bit "1" value
>> into the target memory and returns the old 32-bit value.
> And you want a ticket lock as opposed to the test-and-set lock because
> with 64 tiles starvation under contention is a real worry?

We see substantial unfairness under load with a plain spinlock,
basically because nearer cores on the mesh network can exponentially
crowd out further cores.  The ticket lock avoids that, though we
have to be careful to do backoff when checking the lock to avoid
DDoS in the mesh network.

> Where sparc32 (which only has the load-store-unsigned-byte instruction,
> which is similar to your tns except it writes 0xff) has the benefit of
> not actually having many CPUs you get to do 64 cpus!

Yes, the 64-core thing is awesome, but the atomic instructions
we built into the first generation chip weren't really adequate
to the task.

-- 
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com