3.14.3-rt5: at91: Crash early at boot

3.14.3-rt5: at91: Crash early at boot

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 4121 bytes --]

Hello everyone,

I just gave the latest preempt-rt release for 3.14 a try on an ARM
Atmel SAMA5D3 Xplained board, without any changes to the kernel but
the preempt-rt patches obviously.

The same non-rt version, with the same configuration (except for the
preempt-rt related options), boot fine.

And whenever I boot it, I get:

Uncompressing Linux... done, booting the kernel.
Booting Linux on physical CPU 0x0
Linux version 3.14.3-rt4+ (max@lukather) (gcc version 4.8.2 (Ubuntu/Linaro 4.8.2-16ubuntu4) ) #11 PREEMPT Fri Jun 13 16:03:53 CEST 2014
CPU: ARMv7 Processor [410fc051] revision 1 (ARMv7), cr=10c53c7d
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine model: SAMA5D3 Xplained
bootconsole [earlycon0] enabled
Memory policy: Data cache writeback
AT91: Detected soc type: sama5d3
AT91: Detected soc subtype: sama5d36
AT91: sram at 0x300000 of 0x20000 mapped at 0xfef58000
CPU: All CPU(s) started in SVC mode.
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 65024
Kernel command line: console=ttyS0,115200 earlyprintk mtdparts=atmel_nand:256k(at91bootstrap),512k(uboot),256k(env),256k(env-backup),128k(dtb),5m(kernel),2m(rootfs)ro,-(data) root=/dev/nfs ip=dhcp nfsroot=192.168.0.40:/srv/nfs/sysdev rw
PID hash table entries: 1024 (order: 0, 4096 bytes)
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 253432K/262144K available (4519K kernel code, 191K rwdata, 1292K rodata, 174K init, 120K bss, 8712K reserved)
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
    vmalloc : 0xd0800000 - 0xff000000   ( 744 MB)
    lowmem  : 0xc0000000 - 0xd0000000   ( 256 MB)
    modules : 0xbf000000 - 0xc0000000   (  16 MB)
      .text : 0xc0008000 - 0xc05b51bc   (5813 kB)
      .init : 0xc05b6000 - 0xc05e1984   ( 175 kB)
      .data : 0xc05e2000 - 0xc0611f60   ( 192 kB)
       .bss : 0xc0611f6c - 0xc06301bc   ( 121 kB)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
Preemptible hierarchical RCU implementation.
NR_IRQS:16 nr_irqs:16 16
Unable to handle kernel NULL pointer dereference at virtual address 00000000
pgd = c0004000
[00000000] *pgd=00000000
Internal error: Oops: 5 [#1] PREEMPT ARM
Modules linked in:
CPU: 0 PID: 0 Comm: swapper Not tainted 3.14.3-rt4+ #11
task: c05ecb28 ti: c05e2000 task.ti: c05e2000
PC is at wake_up_process+0x8/0x40
LR is at kthread_create_on_node+0xb0/0x178
pc : [<c003ece0>]    lr : [<c0036318>]    psr: 200001d3
sp : c05e3ef0  ip : 00000017  fp : 00000000
r10: 00000000  r9 : cf4024c0  r8 : ffffffff
r7 : c004cd24  r6 : cf402500  r5 : c05e3f0c  r4 : 00000000
r3 : c06147b0  r2 : 00000001  r1 : cf402554  r0 : 00000000
Flags: nzCv  IRQs off  FIQs off  Mode SVC_32  ISA ARM  Segment kernel
Control: 10c53c7d  Table: 20004059  DAC: 00000015
Process swapper (pid: 0, stack limit = 0xc05e2240)
Stack: (0xc05e3ef0 to 0xc05e4000)
3ee0:                                     c06147b0 cf402540 00000000 c0036318
3f00: 00000000 c004bd38 00000000 00000000 c05e3f10 c05e3f10 00000000 cf404c00
3f20: cf402500 c0318374 00000010 00000010 c004d474 c051dddc 00000010 c055c07c
3f40: 00000104 cf402500 cf404c00 c0318374 00000080 00000010 cf4024c0 c004d808
3f60: cfdf3ca4 c047cefc fefffc00 00000010 cf4024c0 cfffc240 00000000 c05d00ac
3f80: c055c07c cf4024c0 cfdf3ca4 c05e08a0 ffffffff c05ea000 c05daab8 c05cfecc
3fa0: c05ea000 c05e0a28 c05ea000 c0611f80 c05ea000 c05bd9d0 c05d9950 c05b6924
3fc0: ffffffff ffffffff c05b64ec 00000000 00000000 c05daab8 00000000 10c53c7d
3fe0: c05ea01c c05daab4 c05edbd8 20004059 410fc051 20008070 00000000 00000000
Code: e3a05001 eaffffcd e92d4818 e1a04000 (e5903000) 
---[ end trace 3406ff24bd97382e ]---
Kernel panic - not syncing: Attempted to kill the idle task!

Do you know what could be the cause of this, or is this any known issue?

Thanks,
Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: 3.14.3-rt5: at91: Crash early at boot

[jordan]

Maxim,

You may be one of the people affected by some upstream RT changes
[dating back to 3.12-rt] that causes linux-rt to fail to boot on some
H/W.

I myself, am affected, i know of lots of AMD users and the some ARM
h/w that doesn't boot, without reverting a couple of these changes.
[as noted here;
http://www.spinics.net/lists/linux-rt-users/msg11655.html]... You
could try this patch; http://pastebin.com/jb4tuNna - which is applied
after the rt patchset ... This is how i have been able to use RT on my
machines for many RT releases now.

your problem may or may not be the same. but it's probably worth a try.

Jordan



On Fri, Jun 13, 2014 at 10:25 AM, Maxime Ripard
<maxime.ripard@free-electrons.com> wrote:
> Hello everyone,
>
> I just gave the latest preempt-rt release for 3.14 a try on an ARM
> Atmel SAMA5D3 Xplained board, without any changes to the kernel but
> the preempt-rt patches obviously.
>
> The same non-rt version, with the same configuration (except for the
> preempt-rt related options), boot fine.
>
> And whenever I boot it, I get:
>
> Uncompressing Linux... done, booting the kernel.
> Booting Linux on physical CPU 0x0
> Linux version 3.14.3-rt4+ (max@lukather) (gcc version 4.8.2 (Ubuntu/Linaro 4.8.2-16ubuntu4) ) #11 PREEMPT Fri Jun 13 16:03:53 CEST 2014
> CPU: ARMv7 Processor [410fc051] revision 1 (ARMv7), cr=10c53c7d
> CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
> Machine model: SAMA5D3 Xplained
> bootconsole [earlycon0] enabled
> Memory policy: Data cache writeback
> AT91: Detected soc type: sama5d3
> AT91: Detected soc subtype: sama5d36
> AT91: sram at 0x300000 of 0x20000 mapped at 0xfef58000
> CPU: All CPU(s) started in SVC mode.
> Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 65024
> Kernel command line: console=ttyS0,115200 earlyprintk mtdparts=atmel_nand:256k(at91bootstrap),512k(uboot),256k(env),256k(env-backup),128k(dtb),5m(kernel),2m(rootfs)ro,-(data) root=/dev/nfs ip=dhcp nfsroot=192.168.0.40:/srv/nfs/sysdev rw
> PID hash table entries: 1024 (order: 0, 4096 bytes)
> Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
> Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
> Memory: 253432K/262144K available (4519K kernel code, 191K rwdata, 1292K rodata, 174K init, 120K bss, 8712K reserved)
> Virtual kernel memory layout:
>     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
>     fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
>     vmalloc : 0xd0800000 - 0xff000000   ( 744 MB)
>     lowmem  : 0xc0000000 - 0xd0000000   ( 256 MB)
>     modules : 0xbf000000 - 0xc0000000   (  16 MB)
>       .text : 0xc0008000 - 0xc05b51bc   (5813 kB)
>       .init : 0xc05b6000 - 0xc05e1984   ( 175 kB)
>       .data : 0xc05e2000 - 0xc0611f60   ( 192 kB)
>        .bss : 0xc0611f6c - 0xc06301bc   ( 121 kB)
> SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
> Preemptible hierarchical RCU implementation.
> NR_IRQS:16 nr_irqs:16 16
> Unable to handle kernel NULL pointer dereference at virtual address 00000000
> pgd = c0004000
> [00000000] *pgd=00000000
> Internal error: Oops: 5 [#1] PREEMPT ARM
> Modules linked in:
> CPU: 0 PID: 0 Comm: swapper Not tainted 3.14.3-rt4+ #11
> task: c05ecb28 ti: c05e2000 task.ti: c05e2000
> PC is at wake_up_process+0x8/0x40
> LR is at kthread_create_on_node+0xb0/0x178
> pc : [<c003ece0>]    lr : [<c0036318>]    psr: 200001d3
> sp : c05e3ef0  ip : 00000017  fp : 00000000
> r10: 00000000  r9 : cf4024c0  r8 : ffffffff
> r7 : c004cd24  r6 : cf402500  r5 : c05e3f0c  r4 : 00000000
> r3 : c06147b0  r2 : 00000001  r1 : cf402554  r0 : 00000000
> Flags: nzCv  IRQs off  FIQs off  Mode SVC_32  ISA ARM  Segment kernel
> Control: 10c53c7d  Table: 20004059  DAC: 00000015
> Process swapper (pid: 0, stack limit = 0xc05e2240)
> Stack: (0xc05e3ef0 to 0xc05e4000)
> 3ee0:                                     c06147b0 cf402540 00000000 c0036318
> 3f00: 00000000 c004bd38 00000000 00000000 c05e3f10 c05e3f10 00000000 cf404c00
> 3f20: cf402500 c0318374 00000010 00000010 c004d474 c051dddc 00000010 c055c07c
> 3f40: 00000104 cf402500 cf404c00 c0318374 00000080 00000010 cf4024c0 c004d808
> 3f60: cfdf3ca4 c047cefc fefffc00 00000010 cf4024c0 cfffc240 00000000 c05d00ac
> 3f80: c055c07c cf4024c0 cfdf3ca4 c05e08a0 ffffffff c05ea000 c05daab8 c05cfecc
> 3fa0: c05ea000 c05e0a28 c05ea000 c0611f80 c05ea000 c05bd9d0 c05d9950 c05b6924
> 3fc0: ffffffff ffffffff c05b64ec 00000000 00000000 c05daab8 00000000 10c53c7d
> 3fe0: c05ea01c c05daab4 c05edbd8 20004059 410fc051 20008070 00000000 00000000
> Code: e3a05001 eaffffcd e92d4818 e1a04000 (e5903000)
> ---[ end trace 3406ff24bd97382e ]---
> Kernel panic - not syncing: Attempted to kill the idle task!
>
> Do you know what could be the cause of this, or is this any known issue?
>
> Thanks,
> Maxime
>
> --
> Maxime Ripard, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com

Re: 3.14.3-rt5: at91: Crash early at boot

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 991 bytes --]

Hi Jordan,

On Fri, Jun 13, 2014 at 10:48:05AM -0400, jordan wrote:
> Maxim,
> 
> You may be one of the people affected by some upstream RT changes
> [dating back to 3.12-rt] that causes linux-rt to fail to boot on some
> H/W.
> 
> I myself, am affected, i know of lots of AMD users and the some ARM
> h/w that doesn't boot, without reverting a couple of these changes.
> [as noted here;
> http://www.spinics.net/lists/linux-rt-users/msg11655.html]... You
> could try this patch; http://pastebin.com/jb4tuNna - which is applied
> after the rt patchset ... This is how i have been able to use RT on my
> machines for many RT releases now.
> 
> your problem may or may not be the same. but it's probably worth a try.

I just gave your patch a try, and unfortunately, it doesn't change
anything, I still get the same panic at the same moment.

Thanks!
Maxime
-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: 3.14.3-rt5: at91: Crash early at boot

[Sebastian Andrzej Siewior]

* Maxime Ripard | 2014-06-13 16:25:06 [+0200]:

>Preemptible hierarchical RCU implementation.
>NR_IRQS:16 nr_irqs:16 16
>Unable to handle kernel NULL pointer dereference at virtual address 00000000
>pgd = c0004000
>[00000000] *pgd=00000000
>Internal error: Oops: 5 [#1] PREEMPT ARM
>Modules linked in:
>CPU: 0 PID: 0 Comm: swapper Not tainted 3.14.3-rt4+ #11
>task: c05ecb28 ti: c05e2000 task.ti: c05e2000
>PC is at wake_up_process+0x8/0x40
>LR is at kthread_create_on_node+0xb0/0x178
…
>Do you know what could be the cause of this, or is this any known issue?

No. This is very early in the boot process. I upgrade the at91's timer
patches but have no HW to actually check if this is still working.

>
>Thanks,
>Maxime
>

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3.14.3-rt5: at91: Crash early at boot

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 1042 bytes --]

Hi,

On Tue, Feb 17, 2015 at 09:38:59AM +0100, Sebastian Andrzej Siewior wrote:
> * Maxime Ripard | 2014-06-13 16:25:06 [+0200]:
> 
> >Preemptible hierarchical RCU implementation.
> >NR_IRQS:16 nr_irqs:16 16
> >Unable to handle kernel NULL pointer dereference at virtual address 00000000
> >pgd = c0004000
> >[00000000] *pgd=00000000
> >Internal error: Oops: 5 [#1] PREEMPT ARM
> >Modules linked in:
> >CPU: 0 PID: 0 Comm: swapper Not tainted 3.14.3-rt4+ #11
> >task: c05ecb28 ti: c05e2000 task.ti: c05e2000
> >PC is at wake_up_process+0x8/0x40
> >LR is at kthread_create_on_node+0xb0/0x178
> …
> >Do you know what could be the cause of this, or is this any known issue?
> 
> No. This is very early in the boot process. I upgrade the at91's timer
> patches but have no HW to actually check if this is still working.

Do you have a link to these patches?

Maybe we can test them out.

Thanks,
Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: 3.14.3-rt5: at91: Crash early at boot

[Alexandre Belloni]

Hi Maxime,

On 16/03/2015 at 19:19:00 +0100, Maxime Ripard wrote :
> Hi,
> 
> On Tue, Feb 17, 2015 at 09:38:59AM +0100, Sebastian Andrzej Siewior wrote:
> > * Maxime Ripard | 2014-06-13 16:25:06 [+0200]:
> > 
> > >Preemptible hierarchical RCU implementation.
> > >NR_IRQS:16 nr_irqs:16 16
> > >Unable to handle kernel NULL pointer dereference at virtual address 00000000
> > >pgd = c0004000
> > >[00000000] *pgd=00000000
> > >Internal error: Oops: 5 [#1] PREEMPT ARM
> > >Modules linked in:
> > >CPU: 0 PID: 0 Comm: swapper Not tainted 3.14.3-rt4+ #11
> > >task: c05ecb28 ti: c05e2000 task.ti: c05e2000
> > >PC is at wake_up_process+0x8/0x40
> > >LR is at kthread_create_on_node+0xb0/0x178
> > …
> > >Do you know what could be the cause of this, or is this any known issue?
> > 
> > No. This is very early in the boot process. I upgrade the at91's timer
> > patches but have no HW to actually check if this is still working.
> 
> Do you have a link to these patches?
> 
The issue has been solved at emtrion but I think the patches were not
sent yet. The issue is that threading the timer IRQs leads to a crash
because kthreadd is not started yet.


-- 
Alexandre Belloni, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3.14.3-rt5: at91: Crash early at boot

[Christian Jann]

Alexandre Belloni <alexandre.belloni <at> free-electrons.com> writes:

Hi all,

> 
> Hi Maxime,
> 
> On 16/03/2015 at 19:19:00 +0100, Maxime Ripard wrote :
> > Hi,
> > 
> > On Tue, Feb 17, 2015 at 09:38:59AM +0100, Sebastian Andrzej Siewior wrote:
> > > * Maxime Ripard | 2014-06-13 16:25:06 [+0200]:
> > > 
> > > >Preemptible hierarchical RCU implementation.
> > > >NR_IRQS:16 nr_irqs:16 16
> > > >Unable to handle kernel NULL pointer dereference at virtual address
00000000
> > > >pgd = c0004000
> > > >[00000000] *pgd=00000000
> > > >Internal error: Oops: 5 [#1] PREEMPT ARM
> > > >Modules linked in:
> > > >CPU: 0 PID: 0 Comm: swapper Not tainted 3.14.3-rt4+ #11
> > > >task: c05ecb28 ti: c05e2000 task.ti: c05e2000
> > > >PC is at wake_up_process+0x8/0x40
> > > >LR is at kthread_create_on_node+0xb0/0x178
> > > …
> > > >Do you know what could be the cause of this, or is this any known issue?
> > > 
> > > No. This is very early in the boot process. I upgrade the at91's timer
> > > patches but have no HW to actually check if this is still working.
> > 
> > Do you have a link to these patches?
> > 
> The issue has been solved at emtrion but I think the patches were not
> sent yet. The issue is that threading the timer IRQs leads to a crash
> because kthreadd is not started yet.
> 


Are there any updates concerning the patches or does anybody have some more
information on how the problem was resolved on the Atmel controller?

I'm encountering the same problem on a Tegra K1 processor with a
3.10.61-rt65 kernel and I'm unsure what's the best way to fix it.

Unable to handle kernel NULL pointer dereference at virtual address 00000000
[<c0072c80>] (wake_up_process+0x10/0x4c) from [<c005ff20>]
(kthread_create_on_node+0x84/0xec)
[<c005ff20>] (kthread_create_on_node+0x84/0xec) from [<c00bb67c>]
(__setup_irq+0x29c/0x4cc)
[<c00bb67c>] (__setup_irq+0x29c/0x4cc) from [<c00bb8f4>] (setup_irq+0x48/0x90)
[<c00bb8f4>] (setup_irq+0x48/0x90) from [<c06d9f38>]
(tegra3_register_wake_timer+0x28/0x64)
[<c06d9f38>] (tegra3_register_wake_timer+0x28/0x64) from [<c0cea0fc>]
(tegra_init_timer+0x23c/0x36c)
[<c0cea0fc>] (tegra_init_timer+0x23c/0x36c) from [<c0ce9a88>]
(clocksource_of_init+0x34/0x58)
[<c0ce9a88>] (clocksource_of_init+0x34/0x58) from [<c0cba2b4>]
(time_init+0x2c/0x38)
[<c0cba2b4>] (time_init+0x2c/0x38) from [<c0cb79ac>] (start_kernel+0x234/0x370)
Kernel panic - not syncing: Attempted to kill the idle task!

The problem is time_init() gets executed far before rest_init() where
kthreadd_task gets initialized and wake_up_process(kthreadd_task) failes in
kthread_create_on_node().

Best regards
Christian Jann

Re: 3.14.3-rt5: at91: Crash early at boot

[Sebastian Andrzej Siewior]

* Christian Jann | 2015-04-13 13:37:30 [+0000]:

>Are there any updates concerning the patches or does anybody have some more
>information on how the problem was resolved on the Atmel controller?
>
>I'm encountering the same problem on a Tegra K1 processor with a
>3.10.61-rt65 kernel and I'm unsure what's the best way to fix it.

in general what you want is to mark timer interrupt with IRQF_TIMER and
that should be enough. The at91 has a few problems here. One thing is
that it switches from periodic mode to oneshot and this seems not to
work perfectly. The other thing is that the timer interrupt is shared
with the uart.

>Best regards
>Christian Jann

Sebastian

Re: 3.14.3-rt5: at91: Crash early at boot

[Christian Jann]

Sebastian Andrzej Siewior <bigeasy <at> linutronix.de> writes:

> in general what you want is to mark timer interrupt with IRQF_TIMER and
> that should be enough. The at91 has a few problems here. One thing is
> that it switches from periodic mode to oneshot and this seems not to
> work perfectly. The other thing is that the timer interrupt is shared
> with the uart.

Thanks, I replaced the outdated IRQF_DISABLED flag with IRQF_TIMER and now
it works like a charm. :)