Re: [RFC PATCH v2 3/5] futex: Throughput-optimized (TO) futexes

From: Waiman Long <waiman.long@hpe.com>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>,
	Mike Galbraith <umgwanakikbuti@gmail.com>,
	Ingo Molnar <mingo@kernel.org>, Jonathan Corbet <corbet@lwn.net>,
	<linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
	Davidlohr Bueso <dave@stgolabs.net>,
	Jason Low <jason.low2@hpe.com>,
	Scott J Norton <scott.norton@hpe.com>,
	Douglas Hatch <doug.hatch@hpe.com>
Subject: Re: [RFC PATCH v2 3/5] futex: Throughput-optimized (TO) futexes
Date: Thu, 22 Sep 2016 17:13:07 -0400	[thread overview]
Message-ID: <57E44963.9010409@hpe.com> (raw)
In-Reply-To: <alpine.DEB.2.20.1609222222510.5640@nanos>

On 09/22/2016 04:26 PM, Thomas Gleixner wrote:
> On Thu, 22 Sep 2016, Waiman Long wrote:
>> On 09/22/2016 09:34 AM, Thomas Gleixner wrote:
>>> On Thu, 22 Sep 2016, Peter Zijlstra wrote:
>>>> I'd leave out the TO part entirely (or only mention it in changelogs).
>>>>
>>>> That is, I'd call the futex ops: FUTEX_LOCK and FUTEX_UNLOCK.
>>> That brings me to a different question:
>>>
>>> How is user space going to support this, i.e. is this some extra magic for
>>> code which implements its own locking primitives or is there going to be a
>>> wide use via e.g. glibc.
>> There are some applications that use futex(2) directly to implement their
>> synchronization primitives. For those applications, they will need to modify
>> their code to detect the presence of the new futexes. They can then use the
>> new futexes if available and use wait-wake futexes if not.
> That's what I suspected. Did you talk to the other folks who complain about
> futex performance (database, JVM, etc.) and play their own games with user
> space spinlocks and whatever?

I am also part of the team that help large application vendors to tune 
their application performance on our large SMP systems. Those 
application vendors tend to use futex directly instead of relying on 
glibc. We had seen spinlock contention in the futex could sometimes be a 
significant portion of the CPU cycles consumed depending on the 
workloads that were being run. We had been providing suggestions on the 
best practice of how to use futexes. But there is only so much you can 
do with tuning their locking code implementation. That is why I am also 
looking for way to improve the performance of the futex code in the kernel.

>
>> I am also planning to take a look at the pthread_mutex* APIs to see if they
>> can be modified to use the new futexes later on when the patch becomes more
>> mature.
> Please involve glibc people who are interested in the futex stuff early and
> discuss the concept before it's set in stone for your particular usecase.
>   

Sure, I will start to do some prototyping and performance testing with 
glibc and then engage those folks about that.

>>> Also what's the reason that we can't do probabilistic spinning for
>>> FUTEX_WAIT and have to add yet another specialized variant of futexes?
>>>
>> The main reason is that a FUTEX_WAIT waiter has no idea who the owner of the
>> futex is. We usually do spinning when the lock owner is running and abort when
>> it goes to sleep. We can't do that for FUTEX_WAIT.
> Fair enough. This wants to be spelled out in the changelog and explained a
> bit more detailed.

I will.

Cheers,
Longman