Re: [BUG] "sched: Remove rq->lock from the first half of ttwu()" locks up on ARM

From: Peter Zijlstra <peterz@infradead.org>
To: monstr@monstr.eu
Cc: Russell King - ARM Linux <linux@arm.linux.org.uk>,
	Ingo Molnar <mingo@elte.hu>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Marc Zyngier <Marc.Zyngier@arm.com>,
	Frank Rowand <frank.rowand@am.sony.com>,
	Oleg Nesterov <oleg@redhat.com>,
	linux-kernel@vger.kernel.org, Yong Zhang <yong.zhang0@gmail.com>,
	linux-arm-kernel@lists.infradead.org
Subject: Re: [BUG] "sched: Remove rq->lock from the first half of ttwu()" locks up on ARM
Date: Tue, 31 May 2011 16:29:54 +0200	[thread overview]
Message-ID: <1306852194.2353.135.camel@twins> (raw)
In-Reply-To: <4DE4F66D.9040101@monstr.eu>

On Tue, 2011-05-31 at 16:08 +0200, Michal Simek wrote:
> Peter Zijlstra wrote:

> >> I would like to also check some things.
> >> 1. When schedule should be called from arch specific code?
> >> Currently we are calling schedule after syscall/exception/interrupt happen.
> >> Is there any place where schedule should/shouldn't be called?
> > 
> > It should be called on the return to userspace path when
> > TIF_NEED_RESCHED is set. 
> 
> Yes, we do that. (PTO + PT_MODE stores if return is to kernel or user space)
> 
> It should not be called from non-preemptible
> > contexts like non-zero preempt_count or IRQ-disabled.
> 
> Is this even when the return is to userspace?

Well, return to userspace should have preempt_count == 0 and IRQs
enabled, right?

> PREEMPT is not well tested feature but maybe it is right time to do so.
> There is only small part of code (ifdef CONFIG_PREEMPT) when irq happen and 
> there is return to the kernel. Is this correct?

I think so, never looked too closely, Ingo?

> > [ with the exception of CONFIG_PREEMPT which calls preempt_schedule()
> > which checks both those things ]
> 
> This is called only when IRQ happen right? We call preempt_schedule_irq because 
> irq are off and IRQ is ON by rtid below IRQ_return label.

Ah, there's also preempt_schedule_irq(), which can be called with
IRQs-disabled, not sure about the rules there though, Ingo?

> > 
> >> 2. For syscall and exception handling - interrupt is ON but it is only masked.
> > 
> > I'm having trouble understanding: on but masked.
> 
> Interrupt can't happen because some masking bits are setup. If you call 
> irgs_disabled() or others you will get that IRQ is ON but can't happen.

Ah, we generally ignore that state and only rely on state modified by
local_irq_enable/disable(), eg. your MSR_IE bit.

> >> When schedule is called from that any code has to enable IRQ if generic code 
> >> doesn't do that. Not sure if it does.
> > 
> > generic code isn't supposed to call schedule() with IRQs disabled (and
> > doesn't afaik)
> 
> OK. Which means I have to disable IRQ before schedule is called. Is that correct?

Hum, I might have mis-understood. No, schedule() assumes IRQs are
enabled and will disable IRQs itself quite early:

        raw_spin_lock_irq(&rq->lock);