Trouble booting PREEMPT_RT kernel on ARM platform, 2.6.33

Trouble booting PREEMPT_RT kernel on ARM platform, 2.6.33

[Jeremy Brown]

We're trying to get a PREEMPT_RT kernel running on a BeagleBoard C4
(OMAP3 Arm architecture).  We're using kernel version 2.6.33, with
patches from
patch-2.6.33.6-rt26.gz

On boot, we start seeing messages of this form:

----
[ 4528.227294] BUG: sleeping function called from invalid context at
kernel/rtmutex.c:684
[ 4528.227294] pcnt: 1 0 in_atomic(): 1, irqs_disabled(): 128, pid:
223, name: modprobe
[ 4528.227355] [<c0044490>] (unwind_backtrace+0x0/0xe8) from
[<c04cb4e4>] (rt_spin_lock+0x30/0x5c)
[ 4528.227386] [<c04cb4e4>] (rt_spin_lock+0x30/0x5c) from [<c032e3b0>]
(twl4030_sih_set_type+0x3c/0xac)
[ 4528.227416] [<c032e3b0>] (twl4030_sih_set_type+0x3c/0xac) from
[<c00aa944>] (__irq_set_trigger+0x34/0x88)
[ 4528.227447] [<c00aa944>] (__irq_set_trigger+0x34/0x88) from
[<c00ab2ec>] (__setup_irq+0x34c/0x3c0)
[ 4528.227478] [<c00ab2ec>] (__setup_irq+0x34c/0x3c0) from
[<c00ab430>] (request_threaded_irq+0xd0/0x148)
[ 4528.227508] [<c00ab430>] (request_threaded_irq+0xd0/0x148) from
[<bf021770>] (twl_rtc_probe+0x104/0x1fc [rtc_twl])
[ 4528.227539] [<bf021770>] (twl_rtc_probe+0x104/0x1fc [rtc_twl]) from
[<c0324e2c>] (platform_drv_probe+0x18/0x1c)
[ 4528.227569] [<c0324e2c>] (platform_drv_probe+0x18/0x1c) from
[<c0323eec>] (driver_probe_device+0x98/0x194)
[ 4528.227600] [<c0323eec>] (driver_probe_device+0x98/0x194) from
[<c0324074>] (__driver_attach+0x8c/0x90)
[ 4528.227600] [<c0324074>] (__driver_attach+0x8c/0x90) from
[<c0323744>] (bus_for_each_dev+0x60/0x8c)
[ 4528.227630] [<c0323744>] (bus_for_each_dev+0x60/0x8c) from
[<c0323014>] (bus_add_driver+0xa0/0x240)
[ 4528.227661] [<c0323014>] (bus_add_driver+0xa0/0x240) from
[<c0324324>] (driver_register+0x78/0x13c)
[ 4528.227661] [<c0324324>] (driver_register+0x78/0x13c) from
[<c003e3c0>] (do_one_initcall+0x30/0x1c0)
[ 4528.227691] [<c003e3c0>] (do_one_initcall+0x30/0x1c0) from
[<c0097e40>] (sys_init_module+0xc4/0x1f8)
[ 4528.227722] [<c0097e40>] (sys_init_module+0xc4/0x1f8) from
[<c003ea60>] (ret_fast_syscall+0x0/0x2c)
---

I'm hoping that someone will recognize this problem and be able to
suggest a simple fix.  Any help would be very much appreciated!

Thanks,
Jeremy

Re: Trouble booting PREEMPT_RT kernel on ARM platform, 2.6.33

[gowrishankar]

On Wednesday 21 July 2010 03:56 AM, Jeremy Brown wrote:
> We're trying to get a PREEMPT_RT kernel running on a BeagleBoard C4
> (OMAP3 Arm architecture).  We're using kernel version 2.6.33, with
>    
2.6.33 ! not 2.6.33.6 ???
> patches from
> patch-2.6.33.6-rt26.gz
>
> On boot, we start seeing messages of this form:
>
> ----
> [ 4528.227294] BUG: sleeping function called from invalid context at
> kernel/rtmutex.c:684
> [ 4528.227294] pcnt: 1 0 in_atomic(): 1, irqs_disabled(): 128, pid:
> 223, name: modprobe
> [ 4528.227355] [<c0044490>] (unwind_backtrace+0x0/0xe8) from
> [<c04cb4e4>] (rt_spin_lock+0x30/0x5c)
> [ 4528.227386] [<c04cb4e4>] (rt_spin_lock+0x30/0x5c) from [<c032e3b0>]
> (twl4030_sih_set_type+0x3c/0xac)
>    
sleeping spin lock is called while preempt/interrupt is disabled.

Can you try the below patch ?

Signed-off-by: Gowrishankar <gowrishankar.m@in.ibm.com>

Index: linux-2.6.33.6-rt26/drivers/mfd/twl4030-irq.c
===================================================================
--- linux-2.6.33.6-rt26.orig/drivers/mfd/twl4030-irq.c    2010-07-21 
12:40:42.000000000 +0530
+++ linux-2.6.33.6-rt26/drivers/mfd/twl4030-irq.c    2010-07-21 
12:41:16.000000000 +0530
@@ -633,14 +633,14 @@
      if (trigger & ~(IRQ_TYPE_EDGE_FALLING | IRQ_TYPE_EDGE_RISING))
          return -EINVAL;

-    spin_lock_irqsave(&sih_agent_lock, flags);
+    raw_spin_lock_irqsave(&sih_agent_lock, flags);
      if ((desc->status & IRQ_TYPE_SENSE_MASK) != trigger) {
          desc->status &= ~IRQ_TYPE_SENSE_MASK;
          desc->status |= trigger;
          sih->edge_change |= BIT(irq - sih->irq_base);
          queue_work(wq, &sih->edge_work);
      }
-    spin_unlock_irqrestore(&sih_agent_lock, flags);
+    raw_spin_unlock_irqrestore(&sih_agent_lock, flags);
      return 0;
  }

Thanks,
Gowri

> [ 4528.227416] [<c032e3b0>] (twl4030_sih_set_type+0x3c/0xac) from
> [<c00aa944>] (__irq_set_trigger+0x34/0x88)
> [ 4528.227447] [<c00aa944>] (__irq_set_trigger+0x34/0x88) from
> [<c00ab2ec>] (__setup_irq+0x34c/0x3c0)
> [ 4528.227478] [<c00ab2ec>] (__setup_irq+0x34c/0x3c0) from
> [<c00ab430>] (request_threaded_irq+0xd0/0x148)
> [ 4528.227508] [<c00ab430>] (request_threaded_irq+0xd0/0x148) from
> [<bf021770>] (twl_rtc_probe+0x104/0x1fc [rtc_twl])
> [ 4528.227539] [<bf021770>] (twl_rtc_probe+0x104/0x1fc [rtc_twl]) from
> [<c0324e2c>] (platform_drv_probe+0x18/0x1c)
> [ 4528.227569] [<c0324e2c>] (platform_drv_probe+0x18/0x1c) from
> [<c0323eec>] (driver_probe_device+0x98/0x194)
> [ 4528.227600] [<c0323eec>] (driver_probe_device+0x98/0x194) from
> [<c0324074>] (__driver_attach+0x8c/0x90)
> [ 4528.227600] [<c0324074>] (__driver_attach+0x8c/0x90) from
> [<c0323744>] (bus_for_each_dev+0x60/0x8c)
> [ 4528.227630] [<c0323744>] (bus_for_each_dev+0x60/0x8c) from
> [<c0323014>] (bus_add_driver+0xa0/0x240)
> [ 4528.227661] [<c0323014>] (bus_add_driver+0xa0/0x240) from
> [<c0324324>] (driver_register+0x78/0x13c)
> [ 4528.227661] [<c0324324>] (driver_register+0x78/0x13c) from
> [<c003e3c0>] (do_one_initcall+0x30/0x1c0)
> [ 4528.227691] [<c003e3c0>] (do_one_initcall+0x30/0x1c0) from
> [<c0097e40>] (sys_init_module+0xc4/0x1f8)
> [ 4528.227722] [<c0097e40>] (sys_init_module+0xc4/0x1f8) from
> [<c003ea60>] (ret_fast_syscall+0x0/0x2c)
> ---
>
> I'm hoping that someone will recognize this problem and be able to
> suggest a simple fix.  Any help would be very much appreciated!
>
> Thanks,
> Jeremy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>    

Re: Trouble booting PREEMPT_RT kernel on ARM platform, 2.6.33

[Thomas Gleixner]

On Wed, 21 Jul 2010, gowrishankar wrote:

> On Wednesday 21 July 2010 03:56 AM, Jeremy Brown wrote:
> > We're trying to get a PREEMPT_RT kernel running on a BeagleBoard C4
> > (OMAP3 Arm architecture).  We're using kernel version 2.6.33, with
> >    
> 2.6.33 ! not 2.6.33.6 ???
> > patches from
> > patch-2.6.33.6-rt26.gz
> > 
> > On boot, we start seeing messages of this form:
> > 
> > ----
> > [ 4528.227294] BUG: sleeping function called from invalid context at
> > kernel/rtmutex.c:684
> > [ 4528.227294] pcnt: 1 0 in_atomic(): 1, irqs_disabled(): 128, pid:
> > 223, name: modprobe
> > [ 4528.227355] [<c0044490>] (unwind_backtrace+0x0/0xe8) from
> > [<c04cb4e4>] (rt_spin_lock+0x30/0x5c)
> > [ 4528.227386] [<c04cb4e4>] (rt_spin_lock+0x30/0x5c) from [<c032e3b0>]
> > (twl4030_sih_set_type+0x3c/0xac)
> >    
> sleeping spin lock is called while preempt/interrupt is disabled.
> 
> Can you try the below patch ?
> 
> Signed-off-by: Gowrishankar <gowrishankar.m@in.ibm.com>
> 
> Index: linux-2.6.33.6-rt26/drivers/mfd/twl4030-irq.c
> ===================================================================
> --- linux-2.6.33.6-rt26.orig/drivers/mfd/twl4030-irq.c    2010-07-21
> 12:40:42.000000000 +0530
> +++ linux-2.6.33.6-rt26/drivers/mfd/twl4030-irq.c    2010-07-21
> 12:41:16.000000000 +0530
> @@ -633,14 +633,14 @@
>      if (trigger & ~(IRQ_TYPE_EDGE_FALLING | IRQ_TYPE_EDGE_RISING))
>          return -EINVAL;
> 
> -    spin_lock_irqsave(&sih_agent_lock, flags);
> +    raw_spin_lock_irqsave(&sih_agent_lock, flags);
>      if ((desc->status & IRQ_TYPE_SENSE_MASK) != trigger) {
>          desc->status &= ~IRQ_TYPE_SENSE_MASK;
>          desc->status |= trigger;
>          sih->edge_change |= BIT(irq - sih->irq_base);
>          queue_work(wq, &sih->edge_work);
>      }
> -    spin_unlock_irqrestore(&sih_agent_lock, flags);
> +    raw_spin_unlock_irqrestore(&sih_agent_lock, flags);

Nope, that's not going to fly. First it does not compile and second
you trigger the same problem in queue_work() which is called inside
that section. That needs a bit more work, will have a look. (/me
mentions, that todo list is already overly full, so do not expect
wonders).

Thanks,

	tglx

Re: Trouble booting PREEMPT_RT kernel on ARM platform, 2.6.33

[Jeremy Brown]

On Thu, Jul 22, 2010 at 12:44 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
> On Wed, 21 Jul 2010, gowrishankar wrote:
>
>> On Wednesday 21 July 2010 03:56 AM, Jeremy Brown wrote:
>> > We're trying to get a PREEMPT_RT kernel running on a BeagleBoard C4
>> > (OMAP3 Arm architecture).  We're using kernel version 2.6.33, with
>> >
>> 2.6.33 ! not 2.6.33.6 ???

I double faulted; my apologies.  We're using kernel 2.6.33.5
with corresponding 2.6.33.5-rt25 patches.

>> > patches from
>> > patch-2.6.33.6-rt26.gz
>> >
>> > On boot, we start seeing messages of this form:
>> >
>> > ----
>> > [ 4528.227294] BUG: sleeping function called from invalid context at
>> > kernel/rtmutex.c:684
>> > [ 4528.227294] pcnt: 1 0 in_atomic(): 1, irqs_disabled(): 128, pid:
>> > 223, name: modprobe
>> > [ 4528.227355] [<c0044490>] (unwind_backtrace+0x0/0xe8) from
>> > [<c04cb4e4>] (rt_spin_lock+0x30/0x5c)
>> > [ 4528.227386] [<c04cb4e4>] (rt_spin_lock+0x30/0x5c) from [<c032e3b0>]
>> > (twl4030_sih_set_type+0x3c/0xac)
>> >
>> sleeping spin lock is called while preempt/interrupt is disabled.
>>
>> Can you try the below patch ?
>>
>> Signed-off-by: Gowrishankar <gowrishankar.m@in.ibm.com>
>>
>> Index: linux-2.6.33.6-rt26/drivers/mfd/twl4030-irq.c
>> ===================================================================
>> --- linux-2.6.33.6-rt26.orig/drivers/mfd/twl4030-irq.c    2010-07-21
>> 12:40:42.000000000 +0530
>> +++ linux-2.6.33.6-rt26/drivers/mfd/twl4030-irq.c    2010-07-21
>> 12:41:16.000000000 +0530
>> @@ -633,14 +633,14 @@
>>      if (trigger & ~(IRQ_TYPE_EDGE_FALLING | IRQ_TYPE_EDGE_RISING))
>>          return -EINVAL;
>>
>> -    spin_lock_irqsave(&sih_agent_lock, flags);
>> +    raw_spin_lock_irqsave(&sih_agent_lock, flags);
>>      if ((desc->status & IRQ_TYPE_SENSE_MASK) != trigger) {
>>          desc->status &= ~IRQ_TYPE_SENSE_MASK;
>>          desc->status |= trigger;
>>          sih->edge_change |= BIT(irq - sih->irq_base);
>>          queue_work(wq, &sih->edge_work);
>>      }
>> -    spin_unlock_irqrestore(&sih_agent_lock, flags);
>> +    raw_spin_unlock_irqrestore(&sih_agent_lock, flags);
>
> Nope, that's not going to fly. First it does not compile and second
> you trigger the same problem in queue_work() which is called inside
> that section. That needs a bit more work, will have a look.


Although the patch does compile for us, we do see, as you predicted, a
problem in queue_work:
-----
[   42.879547] BUG: sleeping function called from invalid context at
kernel/rtmutex.c:684
[   42.879577] pcnt: 2 0 in_atomic(): 1, irqs_disabled(): 128, pid:
228, name: modprobe
[   42.879638] [<c0044490>] (unwind_backtrace+0x0/0xe8) from
[<c04cb50c>] (rt_spin_lock+0x30/0x5c)
[   42.879669] [<c04cb50c>] (rt_spin_lock+0x30/0x5c) from [<c007c0ec>]
(__queue_work+0x10/0x2c)
[   42.879669] [<c007c0ec>] (__queue_work+0x10/0x2c) from [<c007c164>]
(queue_work_on+0x44/0x5c)
[   42.879699] [<c007c164>] (queue_work_on+0x44/0x5c) from
[<c032e3f8>] (twl4030_sih_set_type+0x84/0xd0)
[   42.879730] [<c032e3f8>] (twl4030_sih_set_type+0x84/0xd0) from
[<c00aa944>] (__irq_set_trigger+0x34/0x88)
[   42.879760] [<c00aa944>] (__irq_set_trigger+0x34/0x88) from
[<c00ab2ec>] (__setup_irq+0x34c/0x3c0)
[   42.879791] [<c00ab2ec>] (__setup_irq+0x34c/0x3c0) from
[<c00ab430>] (request_threaded_irq+0xd0/0x148)
[   42.879821] [<c00ab430>] (request_threaded_irq+0xd0/0x148) from
[<bf027770>] (twl_rtc_probe+0x104/0x1fc [rtc_twl])
[   42.879852] [<bf027770>] (twl_rtc_probe+0x104/0x1fc [rtc_twl]) from
[<c0324e2c>] (platform_drv_probe+0x18/0x1c)
...
------


> (/me
> mentions, that todo list is already overly full, so do not expect
> wonders).

Understood.  We'd be happy to be guinea pigs!

Thanks,

Jeremy


> Thanks,
>
>        tglx
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Trouble booting PREEMPT_RT kernel on ARM platform, 2.6.33

[Jeremy Brown]

Whoops, trying this again without HTML, sorry for any double-sends:

On Thu, Jul 22, 2010 at 9:44 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>
> On Wed, 21 Jul 2010, gowrishankar wrote:
>
> > On Wednesday 21 July 2010 03:56 AM, Jeremy Brown wrote:
> > > We're trying to get a PREEMPT_RT kernel running on a BeagleBoard C4
> > > (OMAP3 Arm architecture).  We're using kernel version 2.6.33, with
> > >
> > 2.6.33 ! not 2.6.33.6 ???
> > > patches from
> > > patch-2.6.33.6-rt26.gz
> > >
> > > On boot, we start seeing messages of this form:
> > >
> > > ----
> > > [ 4528.227294] BUG: sleeping function called from invalid context at
> > > kernel/rtmutex.c:684
> > > [ 4528.227294] pcnt: 1 0 in_atomic(): 1, irqs_disabled(): 128, pid:
> > > 223, name: modprobe
> > > [ 4528.227355] [<c0044490>] (unwind_backtrace+0x0/0xe8) from
> > > [<c04cb4e4>] (rt_spin_lock+0x30/0x5c)
> > > [ 4528.227386] [<c04cb4e4>] (rt_spin_lock+0x30/0x5c) from [<c032e3b0>]
> > > (twl4030_sih_set_type+0x3c/0xac)
> > >
> > sleeping spin lock is called while preempt/interrupt is disabled.
> >
> > Can you try the below patch ?
> >
> > Signed-off-by: Gowrishankar <gowrishankar.m@in.ibm.com>
> >
> > Index: linux-2.6.33.6-rt26/drivers/mfd/twl4030-irq.c
> > ===================================================================
> > --- linux-2.6.33.6-rt26.orig/drivers/mfd/twl4030-irq.c    2010-07-21
> > 12:40:42.000000000 +0530
> > +++ linux-2.6.33.6-rt26/drivers/mfd/twl4030-irq.c    2010-07-21
> > 12:41:16.000000000 +0530
> > @@ -633,14 +633,14 @@
> >      if (trigger & ~(IRQ_TYPE_EDGE_FALLING | IRQ_TYPE_EDGE_RISING))
> >          return -EINVAL;
> >
> > -    spin_lock_irqsave(&sih_agent_lock, flags);
> > +    raw_spin_lock_irqsave(&sih_agent_lock, flags);
> >      if ((desc->status & IRQ_TYPE_SENSE_MASK) != trigger) {
> >          desc->status &= ~IRQ_TYPE_SENSE_MASK;
> >          desc->status |= trigger;
> >          sih->edge_change |= BIT(irq - sih->irq_base);
> >          queue_work(wq, &sih->edge_work);
> >      }
> > -    spin_unlock_irqrestore(&sih_agent_lock, flags);
> > +    raw_spin_unlock_irqrestore(&sih_agent_lock, flags);
>
> Nope, that's not going to fly. First it does not compile and second
> you trigger the same problem in queue_work() which is called inside
> that section. That needs a bit more work, will have a look. (/me
> mentions, that todo list is already overly full, so do not expect
> wonders).

Hi, Thomas.  Has this has risen to the top of your todo queue?   If
not, we're going to start poking at it here; any thoughts, advice,
and/or pointers to reference material would be greatly appreciated.

If we're fortunate enough to get things working, we'll certainly send
patches to this list.

Thanks,
Jeremy

> Thanks,
>
>        tglx
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html