Re: [PATCH -v5 14/14] futex: futex_unlock_pi() determinism

[Peter Zijlstra <peterz@infradead.org>]

On Tue, Mar 07, 2017 at 03:31:50PM +0100, Thomas Gleixner wrote:
> On Sat, 4 Mar 2017, Peter Zijlstra wrote:
> 
> > The problem with returning -EAGAIN when the waiter state mismatches is
> > that it becomes very hard to proof a bounded execution time on the
> > operation. And seeing that this is a RT operation, this is somewhat
> > important.
> > 
> > While in practise it will be very unlikely to ever really take more
> > than one or two rounds, proving so becomes rather hard.
> 
> Oh no. Assume the following:
> 
> T1 and T2 are both pinned to CPU0. prio(T2) > prio(T1)
> 
> CPU0
> 
> T1 
>   lock_pi()
>   queue_me()  <- Waiter is visible
> 
> preemption
> 
> T2
>   unlock_pi()
>     loops with -EAGAIN forever

Ah! indeed.

> > Now that modifying wait_list is done while holding both hb->lock and
> > wait_lock, we can avoid the scenario entirely if we acquire wait_lock
> > while still holding hb-lock. Doing a hand-over, without leaving a
> > hole.
> 
> > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> > ---
> >  kernel/futex.c |   26 ++++++++++++--------------
> >  1 file changed, 12 insertions(+), 14 deletions(-)
> > 
> > --- a/kernel/futex.c
> > +++ b/kernel/futex.c
> > @@ -1391,16 +1391,11 @@ static int wake_futex_pi(u32 __user *uad
> >  	DEFINE_WAKE_Q(wake_q);
> >  	int ret = 0;
> >  
> >  	new_owner = rt_mutex_next_owner(&pi_state->pi_mutex);
> > +	if (WARN_ON_ONCE(!new_owner)) {
> >  		/*
> > +		 * Should be impossible now... but if weirdness happens,
> 
> 'now...' is not very useful 6 month from NOW :)

I'll put in a reference to the below comment in, that explains why this
should now be impossible.

> > +		 * returning -EAGAIN is safe and correct.
> >  		 */
> >  		ret = -EAGAIN;
> >  		goto out_unlock;
> > @@ -2770,15 +2765,18 @@ static int futex_unlock_pi(u32 __user *u
> >  		if (pi_state->owner != current)
> >  			goto out_unlock;
> >  
> > +		get_pi_state(pi_state);
> >  		/*
> > +		 * Since modifying the wait_list is done while holding both
> > +		 * hb->lock and wait_lock, holding either is sufficient to
> > +		 * observe it.
> >  		 *
> > +		 * By taking wait_lock while still holding hb->lock, we ensure
> > +		 * there is no point where we hold neither; and therefore
> > +		 * wake_futex_pi() must observe a state consistent with what we
> > +		 * observed.
> >  		 */


^^ that one.

> > +		raw_spin_lock_irq(&pi_state->pi_mutex.wait_lock);
> >  		spin_unlock(&hb->lock);