Re: 2.6.33.[56]-rt23: howto create repeatable explosion in wakeup_next_waiter()

[Darren Hart <dvhltc@us.ibm.com>]

On 07/09/2010 06:58 AM, Mike Galbraith wrote:
> On Fri, 2010-07-09 at 00:19 -0700, Darren Hart wrote:
>
>> Walking through it:
>>
>> First the dumps:
>> ------------[ cut here ]------------
>> WARNING: at kernel/rtmutex.c:583 wakeup_next_waiter+0x1ad/0x220()
>>
>>
>> WARN_ON(pendowner->pi_blocked_on != waiter);
>> The pi_blocked_on is not NULL, but it isn't the expected waiter either.
>> This means that the top waiter selected at the beginning of
>> wakeup_next_waiter() is now blocked on a lock with a different waiter
>> structure, possibly on a different lock.
>
> pendowner->pi_blocked_on changes while we're in wakeup_next_waiter().
> The below fi^Wmade it not do that any more.  We hold the wait_lock for
> this lock, but if the wakee blocks on another, what's protecting us?

If pendowner is blocked on "lock" to begin with (he should be as his 
waiter struct in in the rtmutex waiters list) then he can't block on 
someone else until he either acquires this one or removes himself as a 
waiter (due to a timeout for instance) - both of these operations 
require holding lock->wait_lock, which is held by the caller of 
wakeup_next_waiter().

Seems more likely that the below forces a missing memory barrier... not 
sure yet though. Good data point.

--
Darren

>
> bandaid-by: /me
>
> diff --git a/kernel/rtmutex.c b/kernel/rtmutex.c
> index 23dd443..dd91ede 100644
> --- a/kernel/rtmutex.c
> +++ b/kernel/rtmutex.c
> @@ -525,6 +525,8 @@ static void wakeup_next_waiter(struct rt_mutex *lock, int savestate)
>   	pendowner = waiter->task;
>   	waiter->task = NULL;
>
> +	raw_spin_lock(&pendowner->pi_lock);
> +
>   	/*
>   	 * Do the wakeup before the ownership change to give any spinning
>   	 * waiter grantees a headstart over the other threads that will
> @@ -577,8 +579,6 @@ static void wakeup_next_waiter(struct rt_mutex *lock, int savestate)
>   	else
>   		next = NULL;
>
> -	raw_spin_lock(&pendowner->pi_lock);
> -
>   	WARN_ON(!pendowner->pi_blocked_on);
>   	WARN_ON(pendowner->pi_blocked_on != waiter);
>   	WARN_ON(pendowner->pi_blocked_on->lock != lock);
>
>


-- 
Darren Hart
IBM Linux Technology Center
Real-Time Linux Team