* rcu_preempt self-detected stall on CPU from 4.4-rc4, since 3.17
@ 2015-12-11 11:17 Ross Green
2015-12-11 11:30 ` Peter Zijlstra
2016-01-03 5:29 ` Ross Green
0 siblings, 2 replies; 12+ messages in thread
From: Ross Green @ 2015-12-11 11:17 UTC (permalink / raw)
To: linux-kernel
Cc: paulmck, mingo, jiangshanlai, dipankar, akpm, mathieu.desnoyers,
josh, tglx, peterz, rostedt, dhowells, edumazet, dvhart,
fweisbec, oleg, bobby.prani
I have been getting these stalls in kernels going back to 3.17.
This stall occurs usually under light load but often requires several
days to show itself. I have not found any simple way to trigger the
stall. Indeed heavy workloads seems not to show the fault.
Anyone have any thoughts here?
The recent patch by peterz with kernel/sched/wait.c I thought might
help the situation, but alas after a few days of running 4.4-rc4 the
following turned up.
[179922.003570] INFO: rcu_preempt self-detected stall on CPU
[179922.008178] INFO: rcu_preempt detected stalls on CPUs/tasks:
[179922.008178] 0-...: (1 ticks this GP) idle=a91/1/0
softirq=1296733/1296733 fqs=0
[179922.008178]
[179922.008209] (detected by 1, t=8775 jiffies, g=576439, c=576438, q=102)
[179922.008209] Task dump for CPU 0:
[179922.008209] swapper/0 R
[179922.008209] running
[179922.008209] 0 0 0 0x00000000
[179922.008209] Backtrace:
[179922.008239] Backtrace aborted due to bad frame pointer <c0907f54>
[179922.008239] rcu_preempt kthread starved for 8775 jiffies! g576439
c576438 f0x0 s3 ->state=0x1
[179922.060302] 0-...: (1 ticks this GP) idle=a91/1/0
softirq=1296733/1296733 fqs=0
[179922.068023] (t=8775 jiffies g=576439 c=576438 q=102)
[179922.073913] rcu_preempt kthread starved for 8775 jiffies! g576439
c576438 f0x2 s3 ->state=0x100
[179922.083587] Task dump for CPU 0:
[179922.087097] swapper/0 R running 0 0 0 0x00000000
[179922.093292] Backtrace:
[179922.096313] [<c0013ea8>] (dump_backtrace) from [<c00140a4>]
(show_stack+0x18/0x1c)
[179922.104675] r7:c0908514 r6:80080193 r5:00000000 r4:c090aca8
[179922.110809] [<c001408c>] (show_stack) from [<c005a858>]
(sched_show_task+0xbc/0x110)
[179922.119049] [<c005a79c>] (sched_show_task) from [<c005ccd4>]
(dump_cpu_task+0x40/0x44)
[179922.127624] r5:c0917680 r4:00000000
[179922.131042] [<c005cc94>] (dump_cpu_task) from [<c0082268>]
(rcu_dump_cpu_stacks+0x9c/0xdc)
[179922.140350] r5:c0917680 r4:00000001
[179922.143157] [<c00821cc>] (rcu_dump_cpu_stacks) from [<c008637c>]
(rcu_check_callbacks+0x504/0x8e4)
[179922.153808] r9:c0908514 r8:c0917680 r7:00000066 r6:2eeab000
r5:c0904300 r4:ef7af300
[179922.161499] [<c0085e78>] (rcu_check_callbacks) from [<c00895d0>]
(update_process_times+0x40/0x6c)
[179922.170898] r10:c009a584 r9:00000001 r8:ef7abc4c r7:0000a3a3
r6:4ec3391c r5:00000000
[179922.179901] r4:c090aca8
[179922.182708] [<c0089590>] (update_process_times) from [<c009a580>]
(tick_sched_handle+0x50/0x54)
[179922.192108] r5:c0907f10 r4:ef7abe40
[179922.195983] [<c009a530>] (tick_sched_handle) from [<c009a5d4>]
(tick_sched_timer+0x50/0x94)
[179922.204895] [<c009a584>] (tick_sched_timer) from [<c0089fe4>]
(__hrtimer_run_queues+0x110/0x1a0)
[179922.214324] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
[179922.220428] [<c0089ed4>] (__hrtimer_run_queues) from [<c008a674>]
(hrtimer_interrupt+0xac/0x1f8)
[179922.227111] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8
r6:ffffffff r5:00000003
[179922.238220] r4:ef7abc00
[179922.238220] [<c008a5c8>] (hrtimer_interrupt) from [<c00170ec>]
(twd_handler+0x38/0x48)
[179922.238220] r10:c09084e8 r9:fa241100 r8:00000011 r7:ef028780
r6:c092574c r5:ef005cc0
[179922.257110] r4:00000001
[179922.257110] [<c00170b4>] (twd_handler) from [<c007c8f8>]
(handle_percpu_devid_irq+0x74/0x8c)
[179922.269683] r5:ef005cc0 r4:ef7b1740
[179922.269683] [<c007c884>] (handle_percpu_devid_irq) from
[<c0078454>] (generic_handle_irq+0x2c/0x3c)
[179922.283233] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000
r5:00000000 r4:c09013e8
[179922.290985] [<c0078428>] (generic_handle_irq) from [<c007872c>]
(__handle_domain_irq+0x64/0xbc)
[179922.300842] [<c00786c8>] (__handle_domain_irq) from [<c00094c0>]
(gic_handle_irq+0x50/0x90)
[179922.303222] r9:fa241100 r8:fa240100 r7:c0907f10 r6:fa24010c
r5:c09087a8 r4:c0925748
[179922.315216] [<c0009470>] (gic_handle_irq) from [<c0014bd4>]
(__irq_svc+0x54/0x90)
[179922.319000] Exception stack(0xc0907f10 to 0xc0907f58)
[179922.331542] 7f00: 00000000
ef7ab390 fe600000 00000000
[179922.331542] 7f20: c0906000 c090849c c0900364 c06a8124 c0907f80
c0944563 c09084e8 c0907f6c
[179922.349029] 7f40: c0907f4c c0907f60 c00287ac c0010ba8 60080113 ffffffff
[179922.349029] r9:c0944563 r8:c0907f80 r7:c0907f44 r6:ffffffff
r5:60080113 r4:c0010ba8
[179922.357116] [<c0010b80>] (arch_cpu_idle) from [<c006f034>]
(default_idle_call+0x28/0x34)
[179922.368926] [<c006f00c>] (default_idle_call) from [<c006f154>]
(cpu_startup_entry+0x114/0x18c)
[179922.368926] [<c006f040>] (cpu_startup_entry) from [<c069fc6c>]
(rest_init+0x90/0x94)
[179922.385284] r7:ffffffff r4:00000002
[179922.393463] [<c069fbdc>] (rest_init) from [<c08bbcec>]
(start_kernel+0x370/0x37c)
[179922.400421] r5:c0947000 r4:00000000
[179922.400421] [<c08bb97c>] (start_kernel) from [<8000807c>] (0x8000807c)
$
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: rcu_preempt self-detected stall on CPU from 4.4-rc4, since 3.17
2015-12-11 11:17 rcu_preempt self-detected stall on CPU from 4.4-rc4, since 3.17 Ross Green
@ 2015-12-11 11:30 ` Peter Zijlstra
2016-01-03 5:29 ` Ross Green
1 sibling, 0 replies; 12+ messages in thread
From: Peter Zijlstra @ 2015-12-11 11:30 UTC (permalink / raw)
To: Ross Green
Cc: linux-kernel, paulmck, mingo, jiangshanlai, dipankar, akpm,
mathieu.desnoyers, josh, tglx, rostedt, dhowells, edumazet,
dvhart, fweisbec, oleg, bobby.prani
On Fri, Dec 11, 2015 at 10:17:33PM +1100, Ross Green wrote:
> I have been getting these stalls in kernels going back to 3.17.
>
> This stall occurs usually under light load but often requires several
> days to show itself. I have not found any simple way to trigger the
> stall. Indeed heavy workloads seems not to show the fault.
>
> Anyone have any thoughts here?
see lkml.kernel.org/r/564F3DCA.1080907@arm.com
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: rcu_preempt self-detected stall on CPU from 4.4-rc4, since 3.17
2015-12-11 11:17 rcu_preempt self-detected stall on CPU from 4.4-rc4, since 3.17 Ross Green
2015-12-11 11:30 ` Peter Zijlstra
@ 2016-01-03 5:29 ` Ross Green
2016-01-03 6:17 ` Paul E. McKenney
1 sibling, 1 reply; 12+ messages in thread
From: Ross Green @ 2016-01-03 5:29 UTC (permalink / raw)
To: linux-kernel
Cc: Paul McKenney, mingo, jiangshanlai, dipankar, akpm,
Mathieu Desnoyers, josh, tglx, peterz, rostedt, dhowells,
Eric Dumazet, dvhart, Frédéric Weisbecker, oleg,
pranith kumar
[-- Attachment #1: Type: text/plain, Size: 5574 bytes --]
Still seeing these rcu_preempt stalls on kernels through to 4.4-rc7
Still have not found a sure fire method to evoke this stall, but have
found that it will normally occur within a week of running a kernel -
usually when it is quiet with light load.
Have seen similar self detected stalls all the way back to 3.17.
Most recent kernels have included 4.4-rc5 4.4-rc6 and 4.4-rc7
Regards,
Ross
On Fri, Dec 11, 2015 at 10:17 PM, Ross Green <rgkernel@gmail.com> wrote:
> I have been getting these stalls in kernels going back to 3.17.
>
> This stall occurs usually under light load but often requires several
> days to show itself. I have not found any simple way to trigger the
> stall. Indeed heavy workloads seems not to show the fault.
>
> Anyone have any thoughts here?
>
> The recent patch by peterz with kernel/sched/wait.c I thought might
> help the situation, but alas after a few days of running 4.4-rc4 the
> following turned up.
>
> [179922.003570] INFO: rcu_preempt self-detected stall on CPU
> [179922.008178] INFO: rcu_preempt detected stalls on CPUs/tasks:
>
> [179922.008178] 0-...: (1 ticks this GP) idle=a91/1/0
> softirq=1296733/1296733 fqs=0
> [179922.008178]
> [179922.008209] (detected by 1, t=8775 jiffies, g=576439, c=576438, q=102)
> [179922.008209] Task dump for CPU 0:
> [179922.008209] swapper/0 R
> [179922.008209] running
> [179922.008209] 0 0 0 0x00000000
> [179922.008209] Backtrace:
>
> [179922.008239] Backtrace aborted due to bad frame pointer <c0907f54>
> [179922.008239] rcu_preempt kthread starved for 8775 jiffies! g576439
> c576438 f0x0 s3 ->state=0x1
>
> [179922.060302] 0-...: (1 ticks this GP) idle=a91/1/0
> softirq=1296733/1296733 fqs=0
> [179922.068023] (t=8775 jiffies g=576439 c=576438 q=102)
> [179922.073913] rcu_preempt kthread starved for 8775 jiffies! g576439
> c576438 f0x2 s3 ->state=0x100
> [179922.083587] Task dump for CPU 0:
> [179922.087097] swapper/0 R running 0 0 0 0x00000000
> [179922.093292] Backtrace:
> [179922.096313] [<c0013ea8>] (dump_backtrace) from [<c00140a4>]
> (show_stack+0x18/0x1c)
> [179922.104675] r7:c0908514 r6:80080193 r5:00000000 r4:c090aca8
> [179922.110809] [<c001408c>] (show_stack) from [<c005a858>]
> (sched_show_task+0xbc/0x110)
> [179922.119049] [<c005a79c>] (sched_show_task) from [<c005ccd4>]
> (dump_cpu_task+0x40/0x44)
> [179922.127624] r5:c0917680 r4:00000000
> [179922.131042] [<c005cc94>] (dump_cpu_task) from [<c0082268>]
> (rcu_dump_cpu_stacks+0x9c/0xdc)
> [179922.140350] r5:c0917680 r4:00000001
> [179922.143157] [<c00821cc>] (rcu_dump_cpu_stacks) from [<c008637c>]
> (rcu_check_callbacks+0x504/0x8e4)
> [179922.153808] r9:c0908514 r8:c0917680 r7:00000066 r6:2eeab000
> r5:c0904300 r4:ef7af300
> [179922.161499] [<c0085e78>] (rcu_check_callbacks) from [<c00895d0>]
> (update_process_times+0x40/0x6c)
> [179922.170898] r10:c009a584 r9:00000001 r8:ef7abc4c r7:0000a3a3
> r6:4ec3391c r5:00000000
> [179922.179901] r4:c090aca8
> [179922.182708] [<c0089590>] (update_process_times) from [<c009a580>]
> (tick_sched_handle+0x50/0x54)
> [179922.192108] r5:c0907f10 r4:ef7abe40
> [179922.195983] [<c009a530>] (tick_sched_handle) from [<c009a5d4>]
> (tick_sched_timer+0x50/0x94)
> [179922.204895] [<c009a584>] (tick_sched_timer) from [<c0089fe4>]
> (__hrtimer_run_queues+0x110/0x1a0)
> [179922.214324] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
> [179922.220428] [<c0089ed4>] (__hrtimer_run_queues) from [<c008a674>]
> (hrtimer_interrupt+0xac/0x1f8)
> [179922.227111] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8
> r6:ffffffff r5:00000003
> [179922.238220] r4:ef7abc00
> [179922.238220] [<c008a5c8>] (hrtimer_interrupt) from [<c00170ec>]
> (twd_handler+0x38/0x48)
> [179922.238220] r10:c09084e8 r9:fa241100 r8:00000011 r7:ef028780
> r6:c092574c r5:ef005cc0
> [179922.257110] r4:00000001
> [179922.257110] [<c00170b4>] (twd_handler) from [<c007c8f8>]
> (handle_percpu_devid_irq+0x74/0x8c)
> [179922.269683] r5:ef005cc0 r4:ef7b1740
> [179922.269683] [<c007c884>] (handle_percpu_devid_irq) from
> [<c0078454>] (generic_handle_irq+0x2c/0x3c)
> [179922.283233] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000
> r5:00000000 r4:c09013e8
> [179922.290985] [<c0078428>] (generic_handle_irq) from [<c007872c>]
> (__handle_domain_irq+0x64/0xbc)
> [179922.300842] [<c00786c8>] (__handle_domain_irq) from [<c00094c0>]
> (gic_handle_irq+0x50/0x90)
> [179922.303222] r9:fa241100 r8:fa240100 r7:c0907f10 r6:fa24010c
> r5:c09087a8 r4:c0925748
> [179922.315216] [<c0009470>] (gic_handle_irq) from [<c0014bd4>]
> (__irq_svc+0x54/0x90)
> [179922.319000] Exception stack(0xc0907f10 to 0xc0907f58)
> [179922.331542] 7f00: 00000000
> ef7ab390 fe600000 00000000
> [179922.331542] 7f20: c0906000 c090849c c0900364 c06a8124 c0907f80
> c0944563 c09084e8 c0907f6c
> [179922.349029] 7f40: c0907f4c c0907f60 c00287ac c0010ba8 60080113 ffffffff
> [179922.349029] r9:c0944563 r8:c0907f80 r7:c0907f44 r6:ffffffff
> r5:60080113 r4:c0010ba8
> [179922.357116] [<c0010b80>] (arch_cpu_idle) from [<c006f034>]
> (default_idle_call+0x28/0x34)
> [179922.368926] [<c006f00c>] (default_idle_call) from [<c006f154>]
> (cpu_startup_entry+0x114/0x18c)
> [179922.368926] [<c006f040>] (cpu_startup_entry) from [<c069fc6c>]
> (rest_init+0x90/0x94)
> [179922.385284] r7:ffffffff r4:00000002
> [179922.393463] [<c069fbdc>] (rest_init) from [<c08bbcec>]
> (start_kernel+0x370/0x37c)
> [179922.400421] r5:c0947000 r4:00000000
> [179922.400421] [<c08bb97c>] (start_kernel) from [<8000807c>] (0x8000807c)
> $
[-- Attachment #2: dmesg-4.4-rc7 --]
[-- Type: application/octet-stream, Size: 22889 bytes --]
rfg@bigred:~> ssh rfg@192.168.1.100
rfg@192.168.1.100's password:
$
$
$
$
$
$ dmesg
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 4.4.0-rc7-panda (rfg@red.dwarf) (gcc version 5.3.0 (Buildroot 2016.02-git-00183-g1682aee) ) #1 SMP PREEMPT Mon Dec 28 14:27:40 AEDT 2015
[ 0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] Machine model: TI OMAP4 PandaBoard-ES
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] OMAP4: Map 0xbfe00000 to fe600000 for dram barrier
[ 0.000000] On node 0 totalpages: 261632
[ 0.000000] free_area_init_node: node 0, pgdat c0945940, node_mem_map ef7f9000
[ 0.000000] Normal zone: 1536 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 196608 pages, LIFO batch:31
[ 0.000000] HighMem zone: 65024 pages, LIFO batch:15
[ 0.000000] OMAP4460 ES1.1
[ 0.000000] PERCPU: Embedded 12 pages/cpu @ef7ab000 s17920 r8192 d23040 u49152
[ 0.000000] pcpu-alloc: s17920 r8192 d23040 u49152 alloc=12*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260096
[ 0.000000] Kernel command line: root=/dev/mmcblk0p2 rootwait console=ttyO2,115200
[ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Memory: 1027368K/1046528K available (6783K kernel code, 250K rwdata, 2132K rodata, 296K init, 294K bss, 19160K reserved, 0K cma-reserved, 260096K highmem)
[ 0.000000] Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
vmalloc : 0xf0800000 - 0xff800000 ( 240 MB)
lowmem : 0xc0000000 - 0xf0000000 ( 768 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc08bd0d4 (8917 kB)
.init : 0xc08be000 - 0xc0908000 ( 296 kB)
.data : 0xc0908000 - 0xc0946920 ( 251 kB)
.bss : 0xc0949000 - 0xc0992bb8 ( 295 kB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 32.
[ 0.000000] NR_IRQS:16 nr_irqs:16 16
[ 0.000000] L2C: platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C: DT/platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C-310 enabling early BRESP for Cortex-A9
[ 0.000000] OMAP L2C310: ROM does not support power control setting
[ 0.000000] L2C-310 dynamic clock gating disabled, standby mode disabled
[ 0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
[ 0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e470000
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node bandgap_fclk
[ 0.000000] OMAP clockevent source: timer1 at 32768 Hz
[ 0.000000] clocksource: 32k_counter: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 58327039986419 ns
[ 0.000030] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65535999984741ns
[ 0.000030] OMAP clocksource: 32k_counter at 32768 Hz
[ 0.000885] Console: colour dummy device 80x30
[ 0.000885] Calibrating delay loop... 1397.55 BogoMIPS (lpj=6987776)
[ 0.089233] pid_max: default: 32768 minimum: 301
[ 0.089324] Security Framework initialized
[ 0.089355] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.089355] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.089965] Initializing cgroup subsys io
[ 0.089965] Initializing cgroup subsys devices
[ 0.089996] Initializing cgroup subsys freezer
[ 0.090026] CPU: Testing write buffer coherency: ok
[ 0.090240] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.090301] Setting up static identity map for 0x80008280 - 0x800082d8
[ 0.209228] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.209289] Brought up 2 CPUs
[ 0.209320] SMP: Total of 2 processors activated (2795.11 BogoMIPS).
[ 0.209320] CPU: All CPU(s) started in SVC mode.
[ 0.210113] devtmpfs: initialized
[ 0.227264] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[ 0.228942] omap_hwmod: l3_main_3 using broken dt data from ocp
[ 0.230499] omap_hwmod: l3_main_2 using broken dt data from ocp
[ 0.326385] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.326629] xor: measuring software checksum speed
[ 0.419219] arm4regs : 1006.000 MB/sec
[ 0.519195] 8regs : 804.800 MB/sec
[ 0.619201] 32regs : 760.800 MB/sec
[ 0.619232] xor: using function: arm4regs (1006.000 MB/sec)
[ 0.619476] pinctrl core: initialized pinctrl subsystem
[ 0.621337] NET: Registered protocol family 16
[ 0.622131] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.631469] OMAP GPIO hardware version 0.1
[ 0.636688] omap-gpmc 50000000.gpmc: GPMC revision 6.0
[ 0.636718] gpmc_mem_init: disabling cs 0 mapped at 0x0-0x1000000
[ 0.637237] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.637603] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.637969] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.649597] platform 4b501000.aes: Cannot lookup hwmod 'aes'
[ 0.649902] platform 480a5000.des: Cannot lookup hwmod 'des'
[ 0.655822] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[ 0.655822] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.656616] omap4_sram_init:Unable to get sram pool needed to handle errata I688
[ 0.656829] OMAP DMA hardware revision 0.0
[ 0.889404] raid6: int32x1 gen() 103 MB/s
[ 1.059417] raid6: int32x1 xor() 84 MB/s
[ 1.229278] raid6: int32x2 gen() 134 MB/s
[ 1.399444] raid6: int32x2 xor() 97 MB/s
[ 1.569580] raid6: int32x4 gen() 145 MB/s
[ 1.739349] raid6: int32x4 xor() 100 MB/s
[ 1.909271] raid6: int32x8 gen() 160 MB/s
[ 2.079315] raid6: int32x8 xor() 98 MB/s
[ 2.079315] raid6: using algorithm int32x8 gen() 160 MB/s
[ 2.079345] raid6: .... xor() 98 MB/s, rmw enabled
[ 2.079345] raid6: using intx1 recovery algorithm
[ 2.086578] omap-dma-engine 4a056000.dma-controller: OMAP DMA engine driver
[ 2.159332] reg-fixed-voltage wl12xx_vmmc: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_wl12xx_gpio, deferring probe
[ 2.163482] omap-iommu 4a066000.mmu: 4a066000.mmu registered
[ 2.163604] omap-iommu 55082000.mmu: 55082000.mmu registered
[ 2.164550] SCSI subsystem initialized
[ 2.164825] usbcore: registered new interface driver usbfs
[ 2.164947] usbcore: registered new interface driver hub
[ 2.165100] usbcore: registered new device driver usb
[ 2.166046] omap_i2c 48070000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c1_pins, deferring probe
[ 2.166107] omap_i2c 48072000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c2_pins, deferring probe
[ 2.166137] omap_i2c 48060000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c3_pins, deferring probe
[ 2.166168] omap_i2c 48350000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c4_pins, deferring probe
[ 2.166442] Linux video capture interface: v2.00
[ 2.167205] Advanced Linux Sound Architecture Driver Initialized.
[ 2.168579] clocksource: Switched to clocksource 32k_counter
[ 2.185943] NET: Registered protocol family 2
[ 2.186523] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[ 2.186645] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[ 2.186767] TCP: Hash tables configured (established 8192 bind 8192)
[ 2.186859] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 2.186920] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 2.187103] NET: Registered protocol family 1
[ 2.187438] RPC: Registered named UNIX socket transport module.
[ 2.187469] RPC: Registered udp transport module.
[ 2.187469] RPC: Registered tcp transport module.
[ 2.187469] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 2.189758] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[ 2.192687] futex hash table entries: 512 (order: 3, 32768 bytes)
[ 2.204956] NFS: Registering the id_resolver key type
[ 2.205017] Key type id_resolver registered
[ 2.205017] Key type id_legacy registered
[ 2.205047] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 2.205078] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 2.206542] ntfs: driver 2.1.32 [Flags: R/O].
[ 2.206939] fuse init (API version 7.23)
[ 2.211364] jitterentropy: Initialization failed with host not compliant with requirements: 2
[ 2.211456] bounce: pool size: 64 pages
[ 2.211456] io scheduler noop registered
[ 2.211486] io scheduler deadline registered
[ 2.211517] io scheduler cfq registered (default)
[ 2.212402] pinctrl-single 4a100040.pinmux: 203 pins at pa fc100040 size 406
[ 2.212615] pinctrl-single 4a31e040.pinmux: 28 pins at pa fc31e040 size 56
[ 2.215911] 58000000.dss supply vdda_video not found, using dummy regulator
[ 2.216033] OMAP DSS rev 4.0
[ 2.216186] omapdss_dss 58000000.dss: bound 58001000.dispc (ops dispc_component_ops)
[ 2.216400] omapdss_dss 58000000.dss: bound 58005000.encoder (ops dsi_component_ops)
[ 2.216949] omapdss_dss 58000000.dss: bound 58006000.encoder (ops hdmi4_component_ops)
[ 2.219360] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 2.221282] omap_uart 4806a000.serial: no wakeirq for uart0
[ 2.221405] 4806a000.serial: ttyO0 at MMIO 0x4806a000 (irq = 226, base_baud = 3000000) is a OMAP UART0
[ 2.221984] 4806c000.serial: ttyO1 at MMIO 0x4806c000 (irq = 227, base_baud = 3000000) is a OMAP UART1
[ 2.222534] 48020000.serial: ttyO2 at MMIO 0x48020000 (irq = 228, base_baud = 3000000) is a OMAP UART2
[ 3.180755] console [ttyO2] enabled
[ 3.184906] 4806e000.serial: ttyO3 at MMIO 0x4806e000 (irq = 229, base_baud = 3000000) is a OMAP UART3
[ 3.195892] [drm] Initialized drm 1.1.0 20060810
[ 3.224334] loop: module loaded
[ 3.231048] usbcore: registered new interface driver smsc95xx
[ 3.237091] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 3.244079] ehci-omap: OMAP-EHCI Host Controller driver
[ 3.269958] ehci-omap 4a064c00.ehci: EHCI Host Controller
[ 3.275634] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1
[ 3.283721] ehci-omap 4a064c00.ehci: irq 336, io mem 0x4a064c00
[ 3.308624] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00
[ 3.315002] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 3.322143] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 3.329711] usb usb1: Product: EHCI Host Controller
[ 3.334808] usb usb1: Manufacturer: Linux 4.4.0-rc7-panda ehci_hcd
[ 3.341278] usb usb1: SerialNumber: 4a064c00.ehci
[ 3.346862] hub 1-0:1.0: USB hub found
[ 3.350860] hub 1-0:1.0: 3 ports detected
[ 3.355743] usbcore: registered new interface driver cdc_wdm
[ 3.361816] usbcore: registered new interface driver usb-storage
[ 3.368530] i2c /dev entries driver
[ 3.372344] usbcore: registered new interface driver uvcvideo
[ 3.378356] USB Video Class driver (1.1.1)
[ 3.383880] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[ 3.391418] device-mapper: uevent: version 1.0.3
[ 3.396606] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com
[ 3.469665] hidraw: raw HID events driver (C) Jiri Kosina
[ 3.475524] usbcore: registered new interface driver usbhid
[ 3.481353] usbhid: USB HID core driver
[ 3.485870] emif 4c000000.emif: emif_probe: device configured with addr = f08a8000 and IRQ247
[ 3.495208] emif 4d000000.emif: emif_probe: device configured with addr = f08aa000 and IRQ248
[ 3.506134] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 3.514068] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 3.521545] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 3.527252] Initializing XFRM netlink socket
[ 3.531799] NET: Registered protocol family 17
[ 3.536743] Key type dns_resolver registered
[ 3.541412] twl: not initialized
[ 3.544830] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.553436] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.561950] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.570434] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.578918] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.587402] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.595886] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1410000 Vs max 1316660
[ 3.604370] omap2_set_init_voltage: unable to find boot up OPP for vdd_mpu
[ 3.611999] omap2_set_init_voltage: unable to set vdd_mpu
[ 3.617706] omap2_set_init_voltage: unable to find boot up OPP for vdd_core
[ 3.625091] omap2_set_init_voltage: unable to set vdd_core
[ 3.630920] omap2_set_init_voltage: unable to find boot up OPP for vdd_iva
[ 3.638214] omap2_set_init_voltage: unable to set vdd_iva
[ 3.644805] Power Management for TI OMAP4+ devices.
[ 3.649993] OMAP4 PM: u-boot >= v2012.07 is required for full PM support
[ 3.657196] ThumbEE CPU extension supported.
[ 3.661804] Registering SWP/SWPB emulation handler
[ 3.661834] SmartReflex Class3 initialized
[ 3.671295] usb 1-1: new high-speed USB device number 2 using ehci-omap
[ 3.696441] Btrfs loaded
[ 3.702056] dmm 4e000000.dmm: initialized all PAT entries
[ 3.722503] Skipping twl internal clock init and using bootloader value (unknown osc rate)
[ 3.753234] twl 0-0048: PIH (irq 340) nested IRQs
[ 3.759704] twl_rtc 48070000.i2c:twl@48:rtc: Enabling TWL-RTC
[ 3.768402] twl_rtc 48070000.i2c:twl@48:rtc: rtc core: registered 48070000.i2c:twl@48 as rtc0
[ 3.788146] twl6040 0-004b: clk32k is not handled
[ 3.795928] omap_i2c 48070000.i2c: bus 0 rev0.11 at 400 kHz
[ 3.802734] omap_i2c 48072000.i2c: bus 1 rev0.11 at 400 kHz
[ 3.809570] omap_i2c 48060000.i2c: bus 2 rev0.11 at 100 kHz
[ 3.816223] omap_i2c 48350000.i2c: bus 3 rev0.11 at 400 kHz
[ 3.869079] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
[ 3.876220] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 3.887817] hub 1-1:1.0: USB hub found
[ 3.892242] hub 1-1:1.0: 5 ports detected
[ 4.007568] omap-abe-twl6040 sound: twl6040-legacy <-> 40132000.mcpdm mapping ok
[ 4.019409] twl_rtc 48070000.i2c:twl@48:rtc: setting system clock to 2015-12-28 10:40:33 UTC (1451299233)
[ 4.029541] sr_init: No PMIC hook to init smartreflex
[ 4.034942] smartreflex smartreflex.0: omap_sr_probe: SmartReflex driver initialized
[ 4.043609] smartreflex smartreflex.1: omap_sr_probe: SmartReflex driver initialized
[ 4.052154] smartreflex smartreflex.2: omap_sr_probe: SmartReflex driver initialized
[ 4.071533] VDAC: disabling
[ 4.075012] VANA: disabling
[ 4.078216] mmc0: host does not support reading read-only switch, assuming write-enable
[ 4.087036] VUSB: disabling
[ 4.090423] ALSA device list:
[ 4.090423] #0: PandaBoardES
[ 4.097229] mmc0: new high speed SDHC card at address e624
[ 4.103149] Waiting for root device /dev/mmcblk0p2...
[ 4.113708] mmcblk0: mmc0:e624 SL32G 28.7 GiB
[ 4.146820] mmcblk0: p1 p2 p3 p4
[ 4.156127] omap_hsmmc 480d5000.mmc: card claims to support voltages below defined range
[ 4.174774] mmc1: queuing unknown CIS tuple 0x91 (3 bytes)
[ 4.181762] mmc1: new SDIO card at address 0001
[ 4.208618] usb 1-1.1: new high-speed USB device number 3 using ehci-omap
[ 4.232116] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
[ 4.241333] EXT4-fs (mmcblk0p2): mounting ext2 file system using the ext4 subsystem
[ 4.271850] EXT4-fs (mmcblk0p2): mounted filesystem without journal. Opts: (null)
[ 4.281280] VFS: Mounted root (ext2 filesystem) readonly on device 179:2.
[ 4.294158] devtmpfs: mounted
[ 4.294158] Freeing unused kernel memory: 296K (c08be000 - c0908000)
[ 4.329345] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[ 4.336639] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 4.347320] smsc95xx v1.0.4
[ 4.441009] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-4a064c00.ehci-1.1, smsc95xx USB 2.0 Ethernet, 6e:e4:2d:83:0b:9f
[ 4.707611] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 4.840332] random: dd urandom read with 32 bits of entropy available
[ 13.230682] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[ 22.965057] Bluetooth: Core ver 2.21
[ 22.969055] NET: Registered protocol family 31
[ 22.969177] Bluetooth: HCI device and connection manager initialized
[ 22.980987] Bluetooth: HCI socket layer initialized
[ 22.981048] Bluetooth: L2CAP socket layer initialized
[ 22.991943] Bluetooth: SCO socket layer initialized
[ 30.553955] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[ 30.565460] NFSD: starting 90-second grace period (net c093fe80)
[ 71.098693] random: nonblocking pool is initialized
[ 80.267089] wlcore: loaded
[ 103.294311] wlcore: firmware booted (Rev 6.3.10.0.133)
[ 106.967620] wlan0: authenticate with 90:f6:52:81:ee:70
[ 107.006164] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[ 107.014739] wlan0: authenticated
[ 107.035919] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[ 107.046752] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[ 107.065765] wlan0: associated
[ 107.124145] wlcore: Association completed.
[ 150.622131] NFSD: Unable to end grace period: -110
[ 168.167938] Adding 2097148k swap on /dev/mmcblk0p3. Priority:-1 extents:1 across:2097148k SS
[146355.192016] INFO: rcu_preempt detected stalls on CPUs/tasks:
[146355.198089] (detected by 0, t=8509 jiffies, g=470652, c=470651, q=111)
[146355.204528] All QSes seen, last rcu_preempt kthread activity 8508 (14605476-14596968), jiffies_till_next_fqs=1, root ->qsmask 0x0
[146355.217620] swapper/0 R running 0 0 0 0x00000000
[146355.224060] Backtrace:
[146355.227172] [<c0013e9c>] (dump_backtrace) from [<c0014098>] (show_stack+0x18/0x1c)
[146355.234313] r7:00000001 r6:2eea8000 r5:00000000 r4:c090cca8
[146355.241394] [<c0014080>] (show_stack) from [<c005a7f8>] (sched_show_task+0xbc/0x110)
[146355.249053] [<c005a73c>] (sched_show_task) from [<c00866f8>] (rcu_check_callbacks+0x8d0/0x8e4)
[146355.258911] r5:c0907300 r4:ef7af300
[146355.262481] [<c0085e28>] (rcu_check_callbacks) from [<c0089580>] (update_process_times+0x40/0x6c)
[146355.272277] r10:c009a534 r9:00000001 r8:ef7abc4c r7:0000851b r6:dfabbc40 r5:00000000
[146355.280487] r4:c090cca8
[146355.283416] [<c0089540>] (update_process_times) from [<c009a530>] (tick_sched_handle+0x50/0x54)
[146355.292785] r5:c0909f10 r4:ef7abe40
[146355.296234] [<c009a4e0>] (tick_sched_handle) from [<c009a584>] (tick_sched_timer+0x50/0x94)
[146355.305603] [<c009a534>] (tick_sched_timer) from [<c0089f94>] (__hrtimer_run_queues+0x110/0x1a0)
[146355.314178] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
[146355.320251] [<c0089e84>] (__hrtimer_run_queues) from [<c008a624>] (hrtimer_interrupt+0xac/0x1f8)
[146355.330078] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8 r6:ffffffff r5:00000003
[146355.338531] r4:ef7abc00
[146355.341125] [<c008a578>] (hrtimer_interrupt) from [<c00170ec>] (twd_handler+0x38/0x48)
[146355.349670] r10:c090a4e8 r9:fa241100 r8:00000011 r7:ef028780 r6:c092774c r5:ef005cc0
[146355.358001] r4:00000001
[146355.361297] [<c00170b4>] (twd_handler) from [<c007c8a8>] (handle_percpu_devid_irq+0x74/0x8c)
[146355.370361] r5:ef005cc0 r4:ef7b1740
[146355.374206] [<c007c834>] (handle_percpu_devid_irq) from [<c0078404>] (generic_handle_irq+0x2c/0x3c)
[146355.383514] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000 r5:00000000 r4:c09043e8
[146355.391876] [<c00783d8>] (generic_handle_irq) from [<c00786dc>] (__handle_domain_irq+0x64/0xbc)
[146355.401550] [<c0078678>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[146355.403930] r9:fa241100 r8:fa240100 r7:c0909f10 r6:fa24010c r5:c090a7a8 r4:c0927748
[146355.418792] [<c0009470>] (gic_handle_irq) from [<c0014bd4>] (__irq_svc+0x54/0x90)
[146355.418792] Exception stack(0xc0909f10 to 0xc0909f58)
[146355.428771] 9f00: 00000000 ef7ab390 fe600000 00000000
[146355.440979] 9f20: c0908000 c090a49c c0903364 c06aa124 c0909f80 c09466e3 c090a4e8 c0909f6c
[146355.448791] 9f40: c0909f4c c0909f60 c00287ac c0010ba8 60080113 ffffffff
[146355.456817] r9:c09466e3 r8:c0909f80 r7:c0909f44 r6:ffffffff r5:60080113 r4:c0010ba8
[146355.462127] [<c0010b80>] (arch_cpu_idle) from [<c006efe4>] (default_idle_call+0x28/0x34)
[146355.462127] [<c006efbc>] (default_idle_call) from [<c006f104>] (cpu_startup_entry+0x114/0x18c)
[146355.483032] [<c006eff0>] (cpu_startup_entry) from [<c06a1284>] (rest_init+0x90/0x94)
[146355.491302] r7:ffffffff r4:00000002
[146355.494537] [<c06a11f4>] (rest_init) from [<c08becec>] (start_kernel+0x370/0x37c)
[146355.501037] r5:c0949000 r4:00000000
[146355.503234] [<c08be97c>] (start_kernel) from [<8000807c>] (0x8000807c)
[146355.508789] rcu_preempt kthread starved for 8508 jiffies! g470652 c470651 f0x2 s3 ->state=0x100
$
$
$
$
$
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: rcu_preempt self-detected stall on CPU from 4.4-rc4, since 3.17
2016-01-03 5:29 ` Ross Green
@ 2016-01-03 6:17 ` Paul E. McKenney
2016-01-03 8:27 ` Ross Green
0 siblings, 1 reply; 12+ messages in thread
From: Paul E. McKenney @ 2016-01-03 6:17 UTC (permalink / raw)
To: Ross Green
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
Mathieu Desnoyers, josh, tglx, peterz, rostedt, dhowells,
Eric Dumazet, dvhart, Frédéric Weisbecker, oleg,
pranith kumar
On Sun, Jan 03, 2016 at 04:29:11PM +1100, Ross Green wrote:
> Still seeing these rcu_preempt stalls on kernels through to 4.4-rc7
>
> Still have not found a sure fire method to evoke this stall, but have
> found that it will normally occur within a week of running a kernel -
> usually when it is quiet with light load.
>
> Have seen similar self detected stalls all the way back to 3.17.
> Most recent kernels have included 4.4-rc5 4.4-rc6 and 4.4-rc7
>
> Regards,
>
> Ross
>
> On Fri, Dec 11, 2015 at 10:17 PM, Ross Green <rgkernel@gmail.com> wrote:
> > I have been getting these stalls in kernels going back to 3.17.
> >
> > This stall occurs usually under light load but often requires several
> > days to show itself. I have not found any simple way to trigger the
> > stall. Indeed heavy workloads seems not to show the fault.
> >
> > Anyone have any thoughts here?
> >
> > The recent patch by peterz with kernel/sched/wait.c I thought might
> > help the situation, but alas after a few days of running 4.4-rc4 the
> > following turned up.
> >
> > [179922.003570] INFO: rcu_preempt self-detected stall on CPU
> > [179922.008178] INFO: rcu_preempt detected stalls on CPUs/tasks:
> > [179922.008178] 0-...: (1 ticks this GP) idle=a91/1/0
CPU 0 is non-idle from an RCU perspective.
> > softirq=1296733/1296733 fqs=0
> > [179922.008178]
> > [179922.008209] (detected by 1, t=8775 jiffies, g=576439, c=576438, q=102)
> > [179922.008209] Task dump for CPU 0:
> > [179922.008209] swapper/0 R [179922.008209] running [179922.008209] 0 0 0 0x00000000
> > [179922.008209] Backtrace:
> >
> > [179922.008239] Backtrace aborted due to bad frame pointer <c0907f54>
Can't have everything, I guess...
> > [179922.008239] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x0 s3 ->state=0x1
Something is keeping the rcu_preempt grace-period kthread from
running. This far into the grace period, it should have a
timer event waking it every few jiffies. It is currently
in TASK_INTERRUPTIBLE state.
> > [179922.060302] 0-...: (1 ticks this GP) idle=a91/1/0 softirq=1296733/1296733 fqs=0
> > [179922.068023] (t=8775 jiffies g=576439 c=576438 q=102)
> > [179922.073913] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x2 s3 ->state=0x100
Same story, same grace period, pretty much same time. Now there is an FQS
request (f0x2) and the state is now TASK_WAKING (->state=0x100 == 256).
> > [179922.083587] Task dump for CPU 0:
> > [179922.087097] swapper/0 R running 0 0 0 0x00000000
> > [179922.093292] Backtrace:
> > [179922.096313] [<c0013ea8>] (dump_backtrace) from [<c00140a4>] (show_stack+0x18/0x1c)
> > [179922.104675] r7:c0908514 r6:80080193 r5:00000000 r4:c090aca8
> > [179922.110809] [<c001408c>] (show_stack) from [<c005a858>] (sched_show_task+0xbc/0x110)
> > [179922.119049] [<c005a79c>] (sched_show_task) from [<c005ccd4>] (dump_cpu_task+0x40/0x44)
> > [179922.127624] r5:c0917680 r4:00000000
> > [179922.131042] [<c005cc94>] (dump_cpu_task) from [<c0082268>] (rcu_dump_cpu_stacks+0x9c/0xdc)
> > [179922.140350] r5:c0917680 r4:00000001
> > [179922.143157] [<c00821cc>] (rcu_dump_cpu_stacks) from [<c008637c>] (rcu_check_callbacks+0x504/0x8e4)
> > [179922.153808] r9:c0908514 r8:c0917680 r7:00000066 r6:2eeab000
> > r5:c0904300 r4:ef7af300
> > [179922.161499] [<c0085e78>] (rcu_check_callbacks) from [<c00895d0>] (update_process_times+0x40/0x6c)
> > [179922.170898] r10:c009a584 r9:00000001 r8:ef7abc4c r7:0000a3a3
> > r6:4ec3391c r5:00000000
> > [179922.179901] r4:c090aca8
> > [179922.182708] [<c0089590>] (update_process_times) from [<c009a580>]
> > (tick_sched_handle+0x50/0x54)
> > [179922.192108] r5:c0907f10 r4:ef7abe40
> > [179922.195983] [<c009a530>] (tick_sched_handle) from [<c009a5d4>]
> > (tick_sched_timer+0x50/0x94)
> > [179922.204895] [<c009a584>] (tick_sched_timer) from [<c0089fe4>]
> > (__hrtimer_run_queues+0x110/0x1a0)
> > [179922.214324] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
> > [179922.220428] [<c0089ed4>] (__hrtimer_run_queues) from [<c008a674>]
> > (hrtimer_interrupt+0xac/0x1f8)
> > [179922.227111] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8
> > r6:ffffffff r5:00000003
> > [179922.238220] r4:ef7abc00
> > [179922.238220] [<c008a5c8>] (hrtimer_interrupt) from [<c00170ec>]
> > (twd_handler+0x38/0x48)
> > [179922.238220] r10:c09084e8 r9:fa241100 r8:00000011 r7:ef028780
> > r6:c092574c r5:ef005cc0
All interrupt stack up to this point.
It is quite possible that the stuff below here is at fault as well.
That said, the grace-period should actually get to execute at some
point. Do you have a heavy real-time load that might be starving
things?
Thanx, Paul
> > [179922.257110] r4:00000001
> > [179922.257110] [<c00170b4>] (twd_handler) from [<c007c8f8>] (handle_percpu_devid_irq+0x74/0x8c)
> > [179922.269683] r5:ef005cc0 r4:ef7b1740
> > [179922.269683] [<c007c884>] (handle_percpu_devid_irq) from [<c0078454>] (generic_handle_irq+0x2c/0x3c)
> > [179922.283233] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000
> > r5:00000000 r4:c09013e8
> > [179922.290985] [<c0078428>] (generic_handle_irq) from [<c007872c>] (__handle_domain_irq+0x64/0xbc)
> > [179922.300842] [<c00786c8>] (__handle_domain_irq) from [<c00094c0>]
> > (gic_handle_irq+0x50/0x90)
> > [179922.303222] r9:fa241100 r8:fa240100 r7:c0907f10 r6:fa24010c
> > r5:c09087a8 r4:c0925748
> > [179922.315216] [<c0009470>] (gic_handle_irq) from [<c0014bd4>]
> > (__irq_svc+0x54/0x90)
> > [179922.319000] Exception stack(0xc0907f10 to 0xc0907f58)
> > [179922.331542] 7f00: 00000000
> > ef7ab390 fe600000 00000000
> > [179922.331542] 7f20: c0906000 c090849c c0900364 c06a8124 c0907f80
> > c0944563 c09084e8 c0907f6c
> > [179922.349029] 7f40: c0907f4c c0907f60 c00287ac c0010ba8 60080113 ffffffff
> > [179922.349029] r9:c0944563 r8:c0907f80 r7:c0907f44 r6:ffffffff
> > r5:60080113 r4:c0010ba8
> > [179922.357116] [<c0010b80>] (arch_cpu_idle) from [<c006f034>]
> > (default_idle_call+0x28/0x34)
> > [179922.368926] [<c006f00c>] (default_idle_call) from [<c006f154>]
> > (cpu_startup_entry+0x114/0x18c)
> > [179922.368926] [<c006f040>] (cpu_startup_entry) from [<c069fc6c>]
> > (rest_init+0x90/0x94)
> > [179922.385284] r7:ffffffff r4:00000002
> > [179922.393463] [<c069fbdc>] (rest_init) from [<c08bbcec>]
> > (start_kernel+0x370/0x37c)
> > [179922.400421] r5:c0947000 r4:00000000
> > [179922.400421] [<c08bb97c>] (start_kernel) from [<8000807c>] (0x8000807c)
> > $
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: rcu_preempt self-detected stall on CPU from 4.4-rc4, since 3.17
2016-01-03 6:17 ` Paul E. McKenney
@ 2016-01-03 8:27 ` Ross Green
2016-01-03 18:15 ` Paul E. McKenney
0 siblings, 1 reply; 12+ messages in thread
From: Ross Green @ 2016-01-03 8:27 UTC (permalink / raw)
To: Paul McKenney
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
Mathieu Desnoyers, josh, tglx, peterz, rostedt, dhowells,
Eric Dumazet, dvhart, Frédéric Weisbecker, oleg,
pranith kumar
[-- Attachment #1: Type: text/plain, Size: 7459 bytes --]
I would not describe the load on this test machine as high or real time.
Apart from a number of standard daemons not much more is running at all!
I normally build a release kernel as soon as possible and set it running.
Typically I run a series of benchmarks to confirm most things appear
to be working and then just leave it running. During a normal day i
will check on the machine 4/5 times just to see how its going!
Typically I will logon remotely via wifi network connection.
just for your information i will include a few other stack traces from
previous kernels that may show some trend!
Please see the attachments.
Regards,
Ross
On Sun, Jan 3, 2016 at 5:17 PM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Sun, Jan 03, 2016 at 04:29:11PM +1100, Ross Green wrote:
>> Still seeing these rcu_preempt stalls on kernels through to 4.4-rc7
>>
>> Still have not found a sure fire method to evoke this stall, but have
>> found that it will normally occur within a week of running a kernel -
>> usually when it is quiet with light load.
>>
>> Have seen similar self detected stalls all the way back to 3.17.
>> Most recent kernels have included 4.4-rc5 4.4-rc6 and 4.4-rc7
>>
>> Regards,
>>
>> Ross
>>
>> On Fri, Dec 11, 2015 at 10:17 PM, Ross Green <rgkernel@gmail.com> wrote:
>> > I have been getting these stalls in kernels going back to 3.17.
>> >
>> > This stall occurs usually under light load but often requires several
>> > days to show itself. I have not found any simple way to trigger the
>> > stall. Indeed heavy workloads seems not to show the fault.
>> >
>> > Anyone have any thoughts here?
>> >
>> > The recent patch by peterz with kernel/sched/wait.c I thought might
>> > help the situation, but alas after a few days of running 4.4-rc4 the
>> > following turned up.
>> >
>> > [179922.003570] INFO: rcu_preempt self-detected stall on CPU
>> > [179922.008178] INFO: rcu_preempt detected stalls on CPUs/tasks:
>> > [179922.008178] 0-...: (1 ticks this GP) idle=a91/1/0
>
> CPU 0 is non-idle from an RCU perspective.
>
>> > softirq=1296733/1296733 fqs=0
>> > [179922.008178]
>> > [179922.008209] (detected by 1, t=8775 jiffies, g=576439, c=576438, q=102)
>> > [179922.008209] Task dump for CPU 0:
>> > [179922.008209] swapper/0 R [179922.008209] running [179922.008209] 0 0 0 0x00000000
>> > [179922.008209] Backtrace:
>> >
>> > [179922.008239] Backtrace aborted due to bad frame pointer <c0907f54>
>
> Can't have everything, I guess...
>
>> > [179922.008239] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x0 s3 ->state=0x1
>
> Something is keeping the rcu_preempt grace-period kthread from
> running. This far into the grace period, it should have a
> timer event waking it every few jiffies. It is currently
> in TASK_INTERRUPTIBLE state.
>
>> > [179922.060302] 0-...: (1 ticks this GP) idle=a91/1/0 softirq=1296733/1296733 fqs=0
>> > [179922.068023] (t=8775 jiffies g=576439 c=576438 q=102)
>> > [179922.073913] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x2 s3 ->state=0x100
>
> Same story, same grace period, pretty much same time. Now there is an FQS
> request (f0x2) and the state is now TASK_WAKING (->state=0x100 == 256).
>
>> > [179922.083587] Task dump for CPU 0:
>> > [179922.087097] swapper/0 R running 0 0 0 0x00000000
>> > [179922.093292] Backtrace:
>> > [179922.096313] [<c0013ea8>] (dump_backtrace) from [<c00140a4>] (show_stack+0x18/0x1c)
>> > [179922.104675] r7:c0908514 r6:80080193 r5:00000000 r4:c090aca8
>> > [179922.110809] [<c001408c>] (show_stack) from [<c005a858>] (sched_show_task+0xbc/0x110)
>> > [179922.119049] [<c005a79c>] (sched_show_task) from [<c005ccd4>] (dump_cpu_task+0x40/0x44)
>> > [179922.127624] r5:c0917680 r4:00000000
>> > [179922.131042] [<c005cc94>] (dump_cpu_task) from [<c0082268>] (rcu_dump_cpu_stacks+0x9c/0xdc)
>> > [179922.140350] r5:c0917680 r4:00000001
>> > [179922.143157] [<c00821cc>] (rcu_dump_cpu_stacks) from [<c008637c>] (rcu_check_callbacks+0x504/0x8e4)
>> > [179922.153808] r9:c0908514 r8:c0917680 r7:00000066 r6:2eeab000
>> > r5:c0904300 r4:ef7af300
>> > [179922.161499] [<c0085e78>] (rcu_check_callbacks) from [<c00895d0>] (update_process_times+0x40/0x6c)
>> > [179922.170898] r10:c009a584 r9:00000001 r8:ef7abc4c r7:0000a3a3
>> > r6:4ec3391c r5:00000000
>> > [179922.179901] r4:c090aca8
>> > [179922.182708] [<c0089590>] (update_process_times) from [<c009a580>]
>> > (tick_sched_handle+0x50/0x54)
>> > [179922.192108] r5:c0907f10 r4:ef7abe40
>> > [179922.195983] [<c009a530>] (tick_sched_handle) from [<c009a5d4>]
>> > (tick_sched_timer+0x50/0x94)
>> > [179922.204895] [<c009a584>] (tick_sched_timer) from [<c0089fe4>]
>> > (__hrtimer_run_queues+0x110/0x1a0)
>> > [179922.214324] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
>> > [179922.220428] [<c0089ed4>] (__hrtimer_run_queues) from [<c008a674>]
>> > (hrtimer_interrupt+0xac/0x1f8)
>> > [179922.227111] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8
>> > r6:ffffffff r5:00000003
>> > [179922.238220] r4:ef7abc00
>> > [179922.238220] [<c008a5c8>] (hrtimer_interrupt) from [<c00170ec>]
>> > (twd_handler+0x38/0x48)
>> > [179922.238220] r10:c09084e8 r9:fa241100 r8:00000011 r7:ef028780
>> > r6:c092574c r5:ef005cc0
>
> All interrupt stack up to this point.
>
> It is quite possible that the stuff below here is at fault as well.
> That said, the grace-period should actually get to execute at some
> point. Do you have a heavy real-time load that might be starving
> things?
>
> Thanx, Paul
>
>> > [179922.257110] r4:00000001
>> > [179922.257110] [<c00170b4>] (twd_handler) from [<c007c8f8>] (handle_percpu_devid_irq+0x74/0x8c)
>> > [179922.269683] r5:ef005cc0 r4:ef7b1740
>> > [179922.269683] [<c007c884>] (handle_percpu_devid_irq) from [<c0078454>] (generic_handle_irq+0x2c/0x3c)
>> > [179922.283233] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000
>> > r5:00000000 r4:c09013e8
>> > [179922.290985] [<c0078428>] (generic_handle_irq) from [<c007872c>] (__handle_domain_irq+0x64/0xbc)
>> > [179922.300842] [<c00786c8>] (__handle_domain_irq) from [<c00094c0>]
>> > (gic_handle_irq+0x50/0x90)
>> > [179922.303222] r9:fa241100 r8:fa240100 r7:c0907f10 r6:fa24010c
>> > r5:c09087a8 r4:c0925748
>> > [179922.315216] [<c0009470>] (gic_handle_irq) from [<c0014bd4>]
>> > (__irq_svc+0x54/0x90)
>> > [179922.319000] Exception stack(0xc0907f10 to 0xc0907f58)
>> > [179922.331542] 7f00: 00000000
>> > ef7ab390 fe600000 00000000
>> > [179922.331542] 7f20: c0906000 c090849c c0900364 c06a8124 c0907f80
>> > c0944563 c09084e8 c0907f6c
>> > [179922.349029] 7f40: c0907f4c c0907f60 c00287ac c0010ba8 60080113 ffffffff
>> > [179922.349029] r9:c0944563 r8:c0907f80 r7:c0907f44 r6:ffffffff
>> > r5:60080113 r4:c0010ba8
>> > [179922.357116] [<c0010b80>] (arch_cpu_idle) from [<c006f034>]
>> > (default_idle_call+0x28/0x34)
>> > [179922.368926] [<c006f00c>] (default_idle_call) from [<c006f154>]
>> > (cpu_startup_entry+0x114/0x18c)
>> > [179922.368926] [<c006f040>] (cpu_startup_entry) from [<c069fc6c>]
>> > (rest_init+0x90/0x94)
>> > [179922.385284] r7:ffffffff r4:00000002
>> > [179922.393463] [<c069fbdc>] (rest_init) from [<c08bbcec>]
>> > (start_kernel+0x370/0x37c)
>> > [179922.400421] r5:c0947000 r4:00000000
>> > [179922.400421] [<c08bb97c>] (start_kernel) from [<8000807c>] (0x8000807c)
>> > $
>
>
[-- Attachment #2: dmesg-4.4-rc5 --]
[-- Type: application/octet-stream, Size: 33888 bytes --]
rfg@red:~> ssh rfg@192.168.1.100
rfg@192.168.1.100's password:
$
$
$
$
$ uptime
19:19:28 up 6 days, 23:54, load average: 0.27, 0.13, 0.14
$
$
$
$ dmesg
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 4.4.0-rc5-panda (rfg@red.dwarf) (gcc version 5.3.0 (Buildroot 2016.02-git-00286-g6a69e28) ) #1 SMP PREEMPT Mon Dec 14 16:46:33 AEDT 2015
[ 0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] Machine model: TI OMAP4 PandaBoard-ES
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] OMAP4: Map 0xbfe00000 to fe600000 for dram barrier
[ 0.000000] On node 0 totalpages: 261632
[ 0.000000] free_area_init_node: node 0, pgdat c09437c0, node_mem_map ef7f9000
[ 0.000000] Normal zone: 1536 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 196608 pages, LIFO batch:31
[ 0.000000] HighMem zone: 65024 pages, LIFO batch:15
[ 0.000000] OMAP4460 ES1.1
[ 0.000000] PERCPU: Embedded 12 pages/cpu @ef7ab000 s17920 r8192 d23040 u49152
[ 0.000000] pcpu-alloc: s17920 r8192 d23040 u49152 alloc=12*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260096
[ 0.000000] Kernel command line: root=/dev/mmcblk0p2 rootwait console=ttyO2,115200
[ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Memory: 1027376K/1046528K available (6775K kernel code, 249K rwdata, 2128K rodata, 300K init, 294K bss, 19152K reserved, 0K cma-reserved, 260096K highmem)
[ 0.000000] Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
vmalloc : 0xf0800000 - 0xff800000 ( 240 MB)
lowmem : 0xc0000000 - 0xf0000000 ( 768 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc08ba0d4 (8905 kB)
.init : 0xc08bb000 - 0xc0906000 ( 300 kB)
.data : 0xc0906000 - 0xc09447a0 ( 250 kB)
.bss : 0xc0947000 - 0xc0990bb8 ( 295 kB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 32.
[ 0.000000] NR_IRQS:16 nr_irqs:16 16
[ 0.000000] L2C: platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C: DT/platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C-310 enabling early BRESP for Cortex-A9
[ 0.000000] OMAP L2C310: ROM does not support power control setting
[ 0.000000] L2C-310 dynamic clock gating disabled, standby mode disabled
[ 0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
[ 0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e470000
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node bandgap_fclk
[ 0.000000] OMAP clockevent source: timer1 at 32768 Hz
[ 0.000000] clocksource: 32k_counter: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 58327039986419 ns
[ 0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65535999984741ns
[ 0.000000] OMAP clocksource: 32k_counter at 32768 Hz
[ 0.000885] Console: colour dummy device 80x30
[ 0.000915] Calibrating delay loop... 1397.55 BogoMIPS (lpj=6987776)
[ 0.089111] pid_max: default: 32768 minimum: 301
[ 0.089172] Security Framework initialized
[ 0.089233] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.089233] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.089843] Initializing cgroup subsys io
[ 0.089843] Initializing cgroup subsys devices
[ 0.089874] Initializing cgroup subsys freezer
[ 0.089874] CPU: Testing write buffer coherency: ok
[ 0.090118] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.090179] Setting up static identity map for 0x80008280 - 0x800082d8
[ 0.209106] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.209167] Brought up 2 CPUs
[ 0.209197] SMP: Total of 2 processors activated (2795.11 BogoMIPS).
[ 0.209197] CPU: All CPU(s) started in SVC mode.
[ 0.209991] devtmpfs: initialized
[ 0.226806] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[ 0.228454] omap_hwmod: l3_main_3 using broken dt data from ocp
[ 0.229949] omap_hwmod: l3_main_2 using broken dt data from ocp
[ 0.324920] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.325164] xor: measuring software checksum speed
[ 0.419097] arm4regs : 1006.000 MB/sec
[ 0.519073] 8regs : 803.600 MB/sec
[ 0.619079] 32regs : 760.800 MB/sec
[ 0.619110] xor: using function: arm4regs (1006.000 MB/sec)
[ 0.619323] pinctrl core: initialized pinctrl subsystem
[ 0.621215] NET: Registered protocol family 16
[ 0.622070] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.631469] OMAP GPIO hardware version 0.1
[ 0.636718] omap-gpmc 50000000.gpmc: GPMC revision 6.0
[ 0.636718] gpmc_mem_init: disabling cs 0 mapped at 0x0-0x1000000
[ 0.637268] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.637634] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.638000] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.649780] platform 4b501000.aes: Cannot lookup hwmod 'aes'
[ 0.650054] platform 480a5000.des: Cannot lookup hwmod 'des'
[ 0.656066] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[ 0.656066] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.656860] omap4_sram_init:Unable to get sram pool needed to handle errata I688
[ 0.657073] OMAP DMA hardware revision 0.0
[ 0.869323] raid6: int32x1 gen() 103 MB/s
[ 1.039428] raid6: int32x1 xor() 84 MB/s
[ 1.209167] raid6: int32x2 gen() 134 MB/s
[ 1.379180] raid6: int32x2 xor() 98 MB/s
[ 1.549163] raid6: int32x4 gen() 145 MB/s
[ 1.719268] raid6: int32x4 xor() 98 MB/s
[ 1.889129] raid6: int32x8 gen() 160 MB/s
[ 2.059143] raid6: int32x8 xor() 94 MB/s
[ 2.059173] raid6: using algorithm int32x8 gen() 160 MB/s
[ 2.059173] raid6: .... xor() 94 MB/s, rmw enabled
[ 2.059173] raid6: using intx1 recovery algorithm
[ 2.066467] omap-dma-engine 4a056000.dma-controller: OMAP DMA engine driver
[ 2.139221] reg-fixed-voltage wl12xx_vmmc: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_wl12xx_gpio, deferring probe
[ 2.143402] omap-iommu 4a066000.mmu: 4a066000.mmu registered
[ 2.143524] omap-iommu 55082000.mmu: 55082000.mmu registered
[ 2.144470] SCSI subsystem initialized
[ 2.144744] usbcore: registered new interface driver usbfs
[ 2.144897] usbcore: registered new interface driver hub
[ 2.145019] usbcore: registered new device driver usb
[ 2.145965] omap_i2c 48070000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c1_pins, deferring probe
[ 2.146026] omap_i2c 48072000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c2_pins, deferring probe
[ 2.146087] omap_i2c 48060000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c3_pins, deferring probe
[ 2.146118] omap_i2c 48350000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c4_pins, deferring probe
[ 2.146362] Linux video capture interface: v2.00
[ 2.147155] Advanced Linux Sound Architecture Driver Initialized.
[ 2.148529] clocksource: Switched to clocksource 32k_counter
[ 2.165893] NET: Registered protocol family 2
[ 2.166473] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[ 2.166595] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[ 2.166717] TCP: Hash tables configured (established 8192 bind 8192)
[ 2.166809] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 2.166870] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 2.167053] NET: Registered protocol family 1
[ 2.167419] RPC: Registered named UNIX socket transport module.
[ 2.167419] RPC: Registered udp transport module.
[ 2.167419] RPC: Registered tcp transport module.
[ 2.167449] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 2.169769] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[ 2.172698] futex hash table entries: 512 (order: 3, 32768 bytes)
[ 2.184875] NFS: Registering the id_resolver key type
[ 2.184906] Key type id_resolver registered
[ 2.184906] Key type id_legacy registered
[ 2.184936] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 2.184967] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 2.186462] ntfs: driver 2.1.32 [Flags: R/O].
[ 2.186859] fuse init (API version 7.23)
[ 2.190551] jitterentropy: Initialization failed with host not compliant with requirements: 2
[ 2.190643] bounce: pool size: 64 pages
[ 2.190643] io scheduler noop registered
[ 2.190673] io scheduler deadline registered
[ 2.190734] io scheduler cfq registered (default)
[ 2.191589] pinctrl-single 4a100040.pinmux: 203 pins at pa fc100040 size 406
[ 2.191802] pinctrl-single 4a31e040.pinmux: 28 pins at pa fc31e040 size 56
[ 2.195129] 58000000.dss supply vdda_video not found, using dummy regulator
[ 2.195251] OMAP DSS rev 4.0
[ 2.195404] omapdss_dss 58000000.dss: bound 58001000.dispc (ops dispc_component_ops)
[ 2.195617] omapdss_dss 58000000.dss: bound 58005000.encoder (ops dsi_component_ops)
[ 2.196166] omapdss_dss 58000000.dss: bound 58006000.encoder (ops hdmi4_component_ops)
[ 2.197448] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 2.199401] omap_uart 4806a000.serial: no wakeirq for uart0
[ 2.199523] 4806a000.serial: ttyO0 at MMIO 0x4806a000 (irq = 226, base_baud = 3000000) is a OMAP UART0
[ 2.200103] 4806c000.serial: ttyO1 at MMIO 0x4806c000 (irq = 227, base_baud = 3000000) is a OMAP UART1
[ 2.200653] 48020000.serial: ttyO2 at MMIO 0x48020000 (irq = 228, base_baud = 3000000) is a OMAP UART2
[ 3.158843] console [ttyO2] enabled
[ 3.163085] 4806e000.serial: ttyO3 at MMIO 0x4806e000 (irq = 229, base_baud = 3000000) is a OMAP UART3
[ 3.174072] [drm] Initialized drm 1.1.0 20060810
[ 3.192504] loop: module loaded
[ 3.198944] usbcore: registered new interface driver smsc95xx
[ 3.205017] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 3.211853] ehci-omap: OMAP-EHCI Host Controller driver
[ 3.228576] ehci-omap 4a064c00.ehci: EHCI Host Controller
[ 3.234252] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1
[ 3.242340] ehci-omap 4a064c00.ehci: irq 336, io mem 0x4a064c00
[ 3.268585] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00
[ 3.274963] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 3.282073] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 3.289642] usb usb1: Product: EHCI Host Controller
[ 3.294769] usb usb1: Manufacturer: Linux 4.4.0-rc5-panda ehci_hcd
[ 3.301239] usb usb1: SerialNumber: 4a064c00.ehci
[ 3.306823] hub 1-0:1.0: USB hub found
[ 3.310821] hub 1-0:1.0: 3 ports detected
[ 3.315704] usbcore: registered new interface driver cdc_wdm
[ 3.321777] usbcore: registered new interface driver usb-storage
[ 3.328491] i2c /dev entries driver
[ 3.332275] usbcore: registered new interface driver uvcvideo
[ 3.338287] USB Video Class driver (1.1.1)
[ 3.343811] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[ 3.351348] device-mapper: uevent: version 1.0.3
[ 3.356506] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com
[ 3.429626] hidraw: raw HID events driver (C) Jiri Kosina
[ 3.435485] usbcore: registered new interface driver usbhid
[ 3.441345] usbhid: USB HID core driver
[ 3.445861] emif 4c000000.emif: emif_probe: device configured with addr = f08a8000 and IRQ247
[ 3.455169] emif 4d000000.emif: emif_probe: device configured with addr = f08aa000 and IRQ248
[ 3.466094] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 3.474060] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 3.481536] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 3.487243] Initializing XFRM netlink socket
[ 3.491790] NET: Registered protocol family 17
[ 3.496734] Key type dns_resolver registered
[ 3.501403] twl: not initialized
[ 3.504821] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.513427] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.521942] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.530426] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.538940] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.547393] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.555877] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1410000 Vs max 1316660
[ 3.564392] omap2_set_init_voltage: unable to find boot up OPP for vdd_mpu
[ 3.571990] omap2_set_init_voltage: unable to set vdd_mpu
[ 3.577728] omap2_set_init_voltage: unable to find boot up OPP for vdd_core
[ 3.585083] omap2_set_init_voltage: unable to set vdd_core
[ 3.590942] omap2_set_init_voltage: unable to find boot up OPP for vdd_iva
[ 3.598205] omap2_set_init_voltage: unable to set vdd_iva
[ 3.604827] Power Management for TI OMAP4+ devices.
[ 3.610015] OMAP4 PM: u-boot >= v2012.07 is required for full PM support
[ 3.617187] ThumbEE CPU extension supported.
[ 3.621826] Registering SWP/SWPB emulation handler
[ 3.621826] SmartReflex Class3 initialized
[ 3.631317] usb 1-1: new high-speed USB device number 2 using ehci-omap
[ 3.654510] Btrfs loaded
[ 3.660186] dmm 4e000000.dmm: initialized all PAT entries
[ 3.683074] Skipping twl internal clock init and using bootloader value (unknown osc rate)
[ 3.709075] twl 0-0048: PIH (irq 340) nested IRQs
[ 3.715484] twl_rtc 48070000.i2c:twl@48:rtc: Enabling TWL-RTC
[ 3.724151] twl_rtc 48070000.i2c:twl@48:rtc: rtc core: registered 48070000.i2c:twl@48 as rtc0
[ 3.743988] twl6040 0-004b: clk32k is not handled
[ 3.751770] omap_i2c 48070000.i2c: bus 0 rev0.11 at 400 kHz
[ 3.758483] omap_i2c 48072000.i2c: bus 1 rev0.11 at 400 kHz
[ 3.765319] omap_i2c 48060000.i2c: bus 2 rev0.11 at 100 kHz
[ 3.772155] omap_i2c 48350000.i2c: bus 3 rev0.11 at 400 kHz
[ 3.808563] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
[ 3.815704] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 3.825317] hub 1-1:1.0: USB hub found
[ 3.831329] hub 1-1:1.0: 5 ports detected
[ 3.958160] omap-abe-twl6040 sound: twl6040-legacy <-> 40132000.mcpdm mapping ok
[ 3.969451] twl_rtc 48070000.i2c:twl@48:rtc: setting system clock to 2015-12-14 08:25:18 UTC (1450081518)
[ 3.976165] sr_init: No PMIC hook to init smartreflex
[ 3.984924] smartreflex smartreflex.0: omap_sr_probe: SmartReflex driver initialized
[ 3.993499] smartreflex smartreflex.1: omap_sr_probe: SmartReflex driver initialized
[ 4.002105] smartreflex smartreflex.2: omap_sr_probe: SmartReflex driver initialized
[ 4.021301] VDAC: disabling
[ 4.024810] VANA: disabling
[ 4.028289] VUSB: disabling
[ 4.031707] ALSA device list:
[ 4.034881] #0: PandaBoardES
[ 4.038146] mmc0: host does not support reading read-only switch, assuming write-enable
[ 4.047180] Waiting for root device /dev/mmcblk0p2...
[ 4.050292] mmc0: new high speed SDHC card at address e624
[ 4.060913] mmcblk0: mmc0:e624 SL32G 28.7 GiB
[ 4.094848] mmcblk0: p1 p2 p3 p4
[ 4.103973] omap_hsmmc 480d5000.mmc: card claims to support voltages below defined range
[ 4.122924] mmc1: queuing unknown CIS tuple 0x91 (3 bytes)
[ 4.129913] mmc1: new SDIO card at address 0001
[ 4.158569] usb 1-1.1: new high-speed USB device number 3 using ehci-omap
[ 4.169555] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
[ 4.178771] EXT4-fs (mmcblk0p2): mounting ext2 file system using the ext4 subsystem
[ 4.223052] EXT4-fs (mmcblk0p2): mounted filesystem without journal. Opts: (null)
[ 4.231079] VFS: Mounted root (ext2 filesystem) readonly on device 179:2.
[ 4.243865] devtmpfs: mounted
[ 4.243896] Freeing unused kernel memory: 300K (c08bb000 - c0906000)
[ 4.302093] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[ 4.309936] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 4.320953] smsc95xx v1.0.4
[ 4.436096] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-4a064c00.ehci-1.1, smsc95xx USB 2.0 Ethernet, fe:4c:ae:bf:a1:eb
[ 4.665832] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 4.803436] random: dd urandom read with 30 bits of entropy available
[ 9.701538] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[ 19.545715] Bluetooth: Core ver 2.21
[ 19.550262] NET: Registered protocol family 31
[ 19.555084] Bluetooth: HCI device and connection manager initialized
[ 19.562103] Bluetooth: HCI socket layer initialized
[ 19.562164] Bluetooth: L2CAP socket layer initialized
[ 19.572875] Bluetooth: SCO socket layer initialized
[ 27.179107] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[ 27.190551] NFSD: starting 90-second grace period (net c093dd00)
[ 30.797882] random: nonblocking pool is initialized
[ 55.023803] wlcore: loaded
[ 72.401306] wlcore: firmware booted (Rev 6.3.10.0.133)
[ 75.983123] wlan0: authenticate with 90:f6:52:81:ee:70
[ 76.019744] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[ 76.027770] wlan0: authenticated
[ 76.042907] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[ 76.053680] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[ 76.072845] wlan0: associated
[ 76.131195] wlcore: Association completed.
[ 147.188598] NFSD: Unable to end grace period: -110
[ 155.990264] Adding 2097148k swap on /dev/mmcblk0p3. Priority:-1 extents:1 across:2097148k SS
[62662.555969] INFO: rcu_preempt self-detected stall on CPU
[62662.560150] INFO: rcu_preempt detected stalls on CPUs/tasks:
[62662.560180] 0-...: (1 GPs behind) idle=bcb/140000000000002/0 softirq=468761/468761 fqs=1
[62662.560180]
[62662.560180] (detected by 1, t=6599 jiffies, g=201679, c=201678, q=107)
[62662.560180] Task dump for CPU 0:
[62662.560180] kworker/u4:1 R
[62662.560180] running
[62662.560211] 0 28741 2 0x00000000
[62662.560211] Workqueue: wl12xx_wq wl1271_netstack_work [wlcore]
[62662.560211] Backtrace:
[62662.560211] Backtrace aborted due to bad frame pointer <eda5df0c>
[62662.610137] 0-...: (1 GPs behind) idle=bcb/140000000000002/0 softirq=468761/468761 fqs=2
[62662.610137] (t=6599 jiffies g=201679 c=201678 q=107)
[62662.620208] Task dump for CPU 0:
[62662.620208] kworker/u4:1 R running 0 28741 2 0x00000002
[62662.630157] Workqueue: wl12xx_wq wl1271_netstack_work [wlcore]
[62662.640716] Backtrace:
[62662.640716] [<c0013ea8>] (dump_backtrace) from [<c00140a4>] (show_stack+0x18/0x1c)
[62662.640716] r7:c0908514 r6:80080193 r5:00000002 r4:ef2f97c0
[62662.640716] [<c001408c>] (show_stack) from [<c005a7f8>] (sched_show_task+0xbc/0x110)
[62662.662384] [<c005a73c>] (sched_show_task) from [<c005cc80>] (dump_cpu_task+0x40/0x48)
[62662.670166] r5:c0917680 r4:00000000
[62662.670166] [<c005cc40>] (dump_cpu_task) from [<c0082218>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[62662.686492] r5:c0917680 r4:00000001
[62662.686828] [<c008217c>] (rcu_dump_cpu_stacks) from [<c008632c>] (rcu_check_callbacks+0x504/0x8e4)
[62662.699768] r9:c0908514 r8:c0917680 r7:0000006b r6:2eeab000 r5:c0904300 r4:ef7af300
[62662.708007] [<c0085e28>] (rcu_check_callbacks) from [<c0089580>] (update_process_times+0x40/0x6c)
[62662.708007] r10:c009a534 r9:00000001 r8:ef7abc4c r7:000038fd r6:cc516401 r5:00000000
[62662.725708] r4:ef2f97c0
[62662.727630] [<c0089540>] (update_process_times) from [<c009a530>] (tick_sched_handle+0x50/0x54)
[62662.730529] r5:eda5dd70 r4:ef7abe40
[62662.730529] [<c009a4e0>] (tick_sched_handle) from [<c009a584>] (tick_sched_timer+0x50/0x94)
[62662.750244] [<c009a534>] (tick_sched_timer) from [<c0089f94>] (__hrtimer_run_queues+0x110/0x1a0)
[62662.750244] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
[62662.750244] [<c0089e84>] (__hrtimer_run_queues) from [<c008a624>] (hrtimer_interrupt+0xac/0x1f8)
[62662.774841] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8 r6:ffffffff r5:00000003
[62662.777374] r4:ef7abc00
[62662.785858] [<c008a578>] (hrtimer_interrupt) from [<c00170ec>] (twd_handler+0x38/0x48)
[62662.794219] r10:ef00b000 r9:fa241100 r8:00000011 r7:ef028780 r6:c092574c r5:ef005cc0
[62662.794219] r4:00000001
[62662.805236] [<c00170b4>] (twd_handler) from [<c007c8a8>] (handle_percpu_devid_irq+0x74/0x8c)
[62662.810150] r5:ef005cc0 r4:ef7b1740
[62662.817962] [<c007c834>] (handle_percpu_devid_irq) from [<c0078404>] (generic_handle_irq+0x2c/0x3c)
[62662.822326] r9:fa241100 r8:ef008000 r7:00000001 r6:eda5de88 r5:00000000 r4:c09013e8
[62662.835357] [<c00783d8>] (generic_handle_irq) from [<c00786dc>] (__handle_domain_irq+0x64/0xbc)
[62662.844970] [<c0078678>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[62662.844970] r9:fa241100 r8:fa240100 r7:eda5dd70 r6:fa24010c r5:c09087a8 r4:c0925748
[62662.862030] [<c0009470>] (gic_handle_irq) from [<c0014bd4>] (__irq_svc+0x54/0x90)
[62662.869750] Exception stack(0xeda5dd70 to 0xeda5ddb8)
[62662.875305] dd60: 04208060 c09480c0 00000000 00000000
[62662.875305] dd80: 00000040 00000000 eda5c000 00000001 ef008000 fa241100 ef00b000 eda5de1c
[62662.892608] dda0: eda5de20 eda5ddc0 c0038908 c0038488 60080113 ffffffff
[62662.892730] r9:fa241100 r8:ef008000 r7:eda5dda4 r6:ffffffff r5:60080113 r4:c0038488
[62662.907836] [<c00383e0>] (__do_softirq) from [<c0038908>] (irq_exit+0xb8/0x120)
[62662.912475] r10:ef00b000 r9:fa241100 r8:ef008000 r7:00000001 r6:00000000 r5:00000000
[62662.920166] r4:c09013e8
[62662.920166] [<c0038850>] (irq_exit) from [<c00786e0>] (__handle_domain_irq+0x68/0xbc)
[62662.934844] r5:00000000 r4:c09013e8
[62662.934844] [<c0078678>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[62662.947479] r9:fa241100 r8:fa240100 r7:eda5de88 r6:fa24010c r5:c09087a8 r4:c0925748
[62662.952301] [<c0009470>] (gic_handle_irq) from [<c0014bd4>] (__irq_svc+0x54/0x90)
[62662.963623] Exception stack(0xeda5de88 to 0xeda5ded0)
[62662.963623] de80: ef00b000 00000004 ef00b014 000062a7 ee9d0f8c ed9ff900
[62662.977630] dea0: ef00b000 ef220f00 00000000 00000000 ef00b000 eda5dee4 eda5dee8 eda5ded8
[62662.980163] dec0: c004a360 c06a46c0 20080113 ffffffff
[62662.991607] r9:00000000 r8:00000000 r7:eda5debc r6:ffffffff r5:20080113 r4:c06a46c0
[62662.991607] [<c06a4698>] (_raw_spin_unlock_irq) from [<c004a360>] (process_one_work+0x114/0x3f0)
[62663.004577] [<c004a24c>] (process_one_work) from [<c004a688>] (worker_thread+0x4c/0x524)
[62663.017730] r10:ef00b000 r9:ed9ff900 r8:00000088 r7:eda5c000 r6:ef00b014 r5:ed9ff918
[62663.017730] r4:ef00b000
[62663.028137] [<c004a63c>] (worker_thread) from [<c0050130>] (kthread+0xe4/0xfc)
[62663.030151] r10:00000000 r9:00000000 r8:00000000 r7:c004a63c r6:ed9ff900 r5:eda89880
[62663.044677] r4:00000000
[62663.044677] [<c005004c>] (kthread) from [<c000fa98>] (ret_from_fork+0x14/0x3c)
[62663.055023] r7:00000000 r6:00000000 r5:c005004c r4:eda89880
[86437.030029] mmcblk0: error -110 sending status command, retrying
[86437.062286] mmcblk0: error -110 sending status command, retrying
[86437.094482] mmcblk0: error -110 sending status command, aborting
[86437.442474] mmc0: tried to reset card
[194501.545684] wlcore: Beacon loss detected. roles:0xff
[194503.184173] wlcore: Beacon loss detected. roles:0xff
[194504.822326] wlcore: Beacon loss detected. roles:0xff
[194506.375915] wlcore: Connection loss work (role_id: 0).
[194506.381500] wlan0: Connection to AP 90:f6:52:81:ee:70 lost
[194509.991455] wlan0: authenticate with 90:f6:52:81:ee:70
[194510.035125] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[194510.043334] wlan0: authenticated
[194510.048522] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[194510.063476] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[194510.083709] wlan0: associated
[194510.127319] wlcore: Association completed.
[414324.735809] wlcore: Beacon loss detected. roles:0xff
[414329.537109] wlcore: Connection loss work (role_id: 0).
[414329.542663] wlan0: Connection to AP 90:f6:52:81:ee:70 lost
[414333.142303] wlan0: authenticate with 90:f6:52:81:ee:70
[414333.184448] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[414333.192626] wlan0: authenticated
[414333.207672] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[414333.218475] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[414333.236541] wlan0: associated
[414333.298248] wlcore: Association completed.
[578668.876434] INFO: rcu_preempt self-detected stall on CPU
[578668.876464] INFO: rcu_preempt self-detected stall on CPU
[578668.876464] 1-...: (2 ticks this GP) idle=d17/1/0 softirq=5136014/5136014 fqs=0
[578668.876495] (t=10403 jiffies g=1859351 c=1859350 q=189)
[578668.876495] rcu_preempt kthread starved for 10403 jiffies! g1859351 c1859350 f0x0 s3 ->state=0x1
[578668.876495] Task dump for CPU 0:
[578668.876495] swapper/0 R running 0 0 0 0x00000000
[578668.876525] Backtrace:
[578668.876525] Backtrace aborted due to bad frame pointer <c0907f54>
[578668.876525] Task dump for CPU 1:
[578668.876556] swapper/1 R running 0 0 1 0x00000002
[578668.876556] Backtrace:
[578668.876556] [<c0013ea8>] (dump_backtrace) from [<c00140a4>] (show_stack+0x18/0x1c)
[578668.876556] r7:c0908514 r6:800c0193 r5:00000001 r4:ef076f80
[578668.876617] [<c001408c>] (show_stack) from [<c005a7f8>] (sched_show_task+0xbc/0x110)
[578668.876647] [<c005a73c>] (sched_show_task) from [<c005cc80>] (dump_cpu_task+0x40/0x48)
[578668.876647] r5:c0917680 r4:00000001
[578668.876678] [<c005cc40>] (dump_cpu_task) from [<c0082218>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[578668.876678] r5:c0917680 r4:00000002
[578668.876708] [<c008217c>] (rcu_dump_cpu_stacks) from [<c008632c>] (rcu_check_callbacks+0x504/0x8e4)
[578668.876739] r9:c0908514 r8:c0917680 r7:000000bd r6:2eeb7000 r5:c0904300 r4:ef7bb300
[578668.876739] [<c0085e28>] (rcu_check_callbacks) from [<c0089580>] (update_process_times+0x40/0x6c)
[578668.876770] r10:c009a534 r9:00000001 r8:ef7b7c4c r7:00020e4b r6:a4b17788 r5:00000000
[578668.876770] r4:ef076f80
[578668.876800] [<c0089540>] (update_process_times) from [<c009a530>] (tick_sched_handle+0x50/0x54)
[578668.876800] r5:ef09ff58 r4:ef7b7e40
[578668.876831] [<c009a4e0>] (tick_sched_handle) from [<c009a584>] (tick_sched_timer+0x50/0x94)
[578668.876831] [<c009a534>] (tick_sched_timer) from [<c0089f94>] (__hrtimer_run_queues+0x110/0x1a0)
[578668.876861] r7:00000000 r6:ef7b7c40 r5:ef7b7e40 r4:ef7b7c00
[578668.876861] [<c0089e84>] (__hrtimer_run_queues) from [<c008a624>] (hrtimer_interrupt+0xac/0x1f8)
[578668.876892] r10:ef7b7c78 r9:ef7b7c98 r8:ef7b7c14 r7:ef7b7cb8 r6:ffffffff r5:00000002
[578668.876892] r4:ef7b7c00
[578668.876892] [<c008a578>] (hrtimer_interrupt) from [<c00170ec>] (twd_handler+0x38/0x48)
[578668.876922] r10:c09084e8 r9:fa241100 r8:00000011 r7:ef028780 r6:c092574c r5:ef005cc0
[578668.876922] r4:00000001
[578668.876953] [<c00170b4>] (twd_handler) from [<c007c8a8>] (handle_percpu_devid_irq+0x74/0x8c)
[578668.876953] r5:ef005cc0 r4:ef7bd740
[578668.876983] [<c007c834>] (handle_percpu_devid_irq) from [<c0078404>] (generic_handle_irq+0x2c/0x3c)
[578668.876983] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000 r5:00000000 r4:c09013e8
[578668.877014] [<c00783d8>] (generic_handle_irq) from [<c00786dc>] (__handle_domain_irq+0x64/0xbc)
[578668.877014] [<c0078678>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[578668.877044] r9:fa241100 r8:fa240100 r7:ef09ff58 r6:fa24010c r5:c09087a8 r4:c0925748
[578668.877044] [<c0009470>] (gic_handle_irq) from [<c0014bd4>] (__irq_svc+0x54/0x90)
[578668.877044] Exception stack(0xef09ff58 to 0xef09ffa0)
[578668.877044] ff40: 00000000 ef7b7390
[578668.877075] ff60: fe600000 00000000 ef09e000 c090849c c0900364 c06a8124 ef09ffc8 c0944563
[578668.877075] ff80: c09084e8 ef09ffb4 ef09ff94 ef09ffa8 c00287ac c0010ba8 600c0113 ffffffff
[578668.877105] r9:c0944563 r8:ef09ffc8 r7:ef09ff8c r6:ffffffff r5:600c0113 r4:c0010ba8
[578668.877105] [<c0010b80>] (arch_cpu_idle) from [<c006efe4>] (default_idle_call+0x28/0x34)
[578668.877105] [<c006efbc>] (default_idle_call) from [<c006f104>] (cpu_startup_entry+0x114/0x18c)
[578668.877166] [<c006eff0>] (cpu_startup_entry) from [<c0016128>] (secondary_start_kernel+0x164/0x170)
[578668.877166] r7:c0947288 r4:c091a700
[578668.877166] [<c0015fc4>] (secondary_start_kernel) from [<8000958c>] (0x8000958c)
[578668.877197] r5:00000051 r4:af08406a
[578669.229797] 0-...: (1 GPs behind) idle=f2b/1/0 softirq=4285547/4285547 fqs=1
[578669.263763] (t=10403 jiffies g=1859351 c=1859350 q=189)
[578669.265899] Task dump for CPU 0:
[578669.265899] swapper/0 R running 0 0 0 0x00000000
[578669.265899] Backtrace:
[578669.283538] [<c0013ea8>] (dump_backtrace) from [<c00140a4>] (show_stack+0x18/0x1c)
[578669.290435] r7:c0908514 r6:80010193 r5:00000000 r4:c090aca8
[578669.295349] [<c001408c>] (show_stack) from [<c005a7f8>] (sched_show_task+0xbc/0x110)
[578669.304595] [<c005a73c>] (sched_show_task) from [<c005cc80>] (dump_cpu_task+0x40/0x48)
[578669.309692] r5:c0917680 r4:00000000
[578669.309692] [<c005cc40>] (dump_cpu_task) from [<c0082218>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[578669.325073] r5:c0917680 r4:00000001
[578669.325073] [<c008217c>] (rcu_dump_cpu_stacks) from [<c008632c>] (rcu_check_callbacks+0x504/0x8e4)
[578669.334594] r9:c0908514 r8:c0917680 r7:000000bd r6:2eeab000 r5:c0904300 r4:ef7af300
[578669.349029] [<c0085e28>] (rcu_check_callbacks) from [<c0089580>] (update_process_times+0x40/0x6c)
[578669.349029] r10:c009a534 r9:00000001 r8:ef7abc4c r7:00020e4b r6:a46fcf14 r5:00000000
[578669.349029] r4:c090aca8
[578669.369689] [<c0089540>] (update_process_times) from [<c009a530>] (tick_sched_handle+0x50/0x54)
[578669.371276] r5:c0907f10 r4:ef7abe40
[578669.382873] [<c009a4e0>] (tick_sched_handle) from [<c009a584>] (tick_sched_timer+0x50/0x94)
[578669.385711] [<c009a534>] (tick_sched_timer) from [<c0089f94>] (__hrtimer_run_queues+0x110/0x1a0)
[578669.401184] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
[578669.401184] [<c0089e84>] (__hrtimer_run_queues) from [<c008a624>] (hrtimer_interrupt+0xac/0x1f8)
[578669.411407] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8 r6:ffffffff r5:00000003
[578669.416687] r4:ef7abc00
[578669.416687] [<c008a578>] (hrtimer_interrupt) from [<c00170ec>] (twd_handler+0x38/0x48)
[578669.434600] r10:c09084e8 r9:fa241100 r8:00000011 r7:ef028780 r6:c092574c r5:ef005cc0
[578669.444732] r4:00000001
[578669.444732] [<c00170b4>] (twd_handler) from [<c007c8a8>] (handle_percpu_devid_irq+0x74/0x8c)
[578669.451293] r5:ef005cc0 r4:ef7b1740
[578669.456542] [<c007c834>] (handle_percpu_devid_irq) from [<c0078404>] (generic_handle_irq+0x2c/0x3c)
[578669.465942] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000 r5:00000000 r4:c09013e8
[578669.465942] [<c00783d8>] (generic_handle_irq) from [<c00786dc>] (__handle_domain_irq+0x64/0xbc)
[578669.483245] [<c0078678>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[578669.491943] r9:fa241100 r8:fa240100 r7:c0907f10 r6:fa24010c r5:c09087a8 r4:c0925748
[578669.504943] [<c0009470>] (gic_handle_irq) from [<c0014bd4>] (__irq_svc+0x54/0x90)
[578669.510375] Exception stack(0xc0907f10 to 0xc0907f58)
[578669.518402] 7f00: 00000000 ef7ab390 fe600000 00000000
[578669.518402] 7f20: c0906000 c090849c c0900364 c06a8124 c0907f80 c0944563 c09084e8 c0907f6c
[578669.535888] 7f40: c0907f4c c0907f60 c00287ac c0010ba8 60010113 ffffffff
[578669.538391] r9:c0944563 r8:c0907f80 r7:c0907f44 r6:ffffffff r5:60010113 r4:c0010ba8
[578669.549133] [<c0010b80>] (arch_cpu_idle) from [<c006efe4>] (default_idle_call+0x28/0x34)
[578669.559967] [<c006efbc>] (default_idle_call) from [<c006f104>] (cpu_startup_entry+0x114/0x18c)
[578669.564605] [<c006eff0>] (cpu_startup_entry) from [<c069fe4c>] (rest_init+0x90/0x94)
[578669.574615] r7:ffffffff r4:00000002
[578669.574615] [<c069fdbc>] (rest_init) from [<c08bbcec>] (start_kernel+0x370/0x37c)
[578669.584594] r5:c0947000 r4:00000000
[578669.584594] [<c08bb97c>] (start_kernel) from [<8000807c>] (0x8000807c)
$
[-- Attachment #3: dmesg-4.3 --]
[-- Type: application/octet-stream, Size: 36244 bytes --]
$
$ uptime
13:16:01 up 1 day, 17:43, load average: 0.13, 0.09, 0.12
$
$
$
$ cat /proc/version
Linux version 4.3.0-panda (rfg@red.dwarf) (gcc version 5.2.0 (Buildroot 2015.11-git-00627-g9731582) ) #1 SMP PREEMPT Mon Nov 2 19:21:52 AEDT 2015
$ dmesg
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 4.3.0-panda (rfg@red.dwarf) (gcc version 5.2.0 (Buildroot 2015.11-git-00627-g9731582) ) #1 SMP PREEMPT Mon Nov 2 19:21:52 AEDT 2015
[ 0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] Machine model: TI OMAP4 PandaBoard-ES
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] OMAP4: Map 0xbfe00000 to fe600000 for dram barrier
[ 0.000000] On node 0 totalpages: 261632
[ 0.000000] free_area_init_node: node 0, pgdat c09335c0, node_mem_map eeff9000
[ 0.000000] Normal zone: 1520 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 194560 pages, LIFO batch:31
[ 0.000000] HighMem zone: 67072 pages, LIFO batch:15
[ 0.000000] OMAP4460 ES1.1
[ 0.000000] PERCPU: Embedded 12 pages/cpu @eefaa000 s17920 r8192 d23040 u49152
[ 0.000000] pcpu-alloc: s17920 r8192 d23040 u49152 alloc=12*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260112
[ 0.000000] Kernel command line: root=/dev/mmcblk0p2 rootwait console=ttyO2,115200
[ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Memory: 1027436K/1046528K available (6731K kernel code, 249K rwdata, 2116K rodata, 292K init, 294K bss, 19092K reserved, 0K cma-reserved, 268288K highmem)
[ 0.000000] Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
vmalloc : 0xf0000000 - 0xff000000 ( 240 MB)
lowmem : 0xc0000000 - 0xef800000 ( 760 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc08ac0d4 (8849 kB)
.init : 0xc08ad000 - 0xc08f6000 ( 292 kB)
.data : 0xc08f6000 - 0xc0934580 ( 250 kB)
.bss : 0xc0937000 - 0xc0980a78 ( 295 kB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 32.
[ 0.000000] NR_IRQS:16 nr_irqs:16 16
[ 0.000000] L2C: platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C: DT/platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C-310 enabling early BRESP for Cortex-A9
[ 0.000000] OMAP L2C310: ROM does not support power control setting
[ 0.000000] L2C-310 dynamic clock gating disabled, standby mode disabled
[ 0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
[ 0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e470000
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node bandgap_fclk
[ 0.000000] OMAP clockevent source: timer1 at 32768 Hz
[ 0.000000] clocksource: 32k_counter: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 58327039986419 ns
[ 0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65535999984741ns
[ 0.000030] OMAP clocksource: 32k_counter at 32768 Hz
[ 0.001739] Console: colour dummy device 80x30
[ 0.001739] Calibrating delay loop... 1397.55 BogoMIPS (lpj=6987776)
[ 0.090270] pid_max: default: 32768 minimum: 301
[ 0.090362] Security Framework initialized
[ 0.090393] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.090393] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.090972] Initializing cgroup subsys io
[ 0.091003] Initializing cgroup subsys devices
[ 0.091033] Initializing cgroup subsys freezer
[ 0.091064] CPU: Testing write buffer coherency: ok
[ 0.091278] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.091339] Setting up static identity map for 0x80008280 - 0x800082d8
[ 0.210266] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.210327] Brought up 2 CPUs
[ 0.210357] SMP: Total of 2 processors activated (2795.11 BogoMIPS).
[ 0.210357] CPU: All CPU(s) started in SVC mode.
[ 0.211151] devtmpfs: initialized
[ 0.227813] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[ 0.229583] omap_hwmod: l3_main_3 using broken dt data from ocp
[ 0.231170] omap_hwmod: l3_main_2 using broken dt data from ocp
[ 0.329162] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.329406] xor: measuring software checksum speed
[ 0.420257] arm4regs : 1006.000 MB/sec
[ 0.520233] 8regs : 803.600 MB/sec
[ 0.620239] 32regs : 760.800 MB/sec
[ 0.620269] xor: using function: arm4regs (1006.000 MB/sec)
[ 0.620483] pinctrl core: initialized pinctrl subsystem
[ 0.622375] NET: Registered protocol family 16
[ 0.623199] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.631988] OMAP GPIO hardware version 0.1
[ 0.636535] omap-gpmc 50000000.gpmc: GPMC revision 6.0
[ 0.636566] gpmc_mem_init: disabling cs 0 mapped at 0x0-0x1000000
[ 0.637084] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.637451] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.637817] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.649475] platform 4b501000.aes: Cannot lookup hwmod 'aes'
[ 0.649780] platform 480a5000.des: Cannot lookup hwmod 'des'
[ 0.655822] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[ 0.655853] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.656982] omap4_sram_init:Unable to get sram pool needed to handle errata I688
[ 0.657196] OMAP DMA hardware revision 0.0
[ 0.890319] raid6: int32x1 gen() 103 MB/s
[ 1.060577] raid6: int32x1 xor() 82 MB/s
[ 1.230316] raid6: int32x2 gen() 134 MB/s
[ 1.400451] raid6: int32x2 xor() 98 MB/s
[ 1.570404] raid6: int32x4 gen() 145 MB/s
[ 1.740509] raid6: int32x4 xor() 97 MB/s
[ 1.910339] raid6: int32x8 gen() 160 MB/s
[ 2.080535] raid6: int32x8 xor() 94 MB/s
[ 2.080535] raid6: using algorithm int32x8 gen() 160 MB/s
[ 2.080535] raid6: .... xor() 94 MB/s, rmw enabled
[ 2.080535] raid6: using intx1 recovery algorithm
[ 2.087707] omap-dma-engine 4a056000.dma-controller: OMAP DMA engine driver
[ 2.160369] reg-fixed-voltage wl12xx_vmmc: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_wl12xx_gpio, deferring probe
[ 2.164459] omap-iommu 4a066000.mmu: 4a066000.mmu registered
[ 2.164581] omap-iommu 55082000.mmu: 55082000.mmu registered
[ 2.165496] SCSI subsystem initialized
[ 2.165771] usbcore: registered new interface driver usbfs
[ 2.165924] usbcore: registered new interface driver hub
[ 2.166046] usbcore: registered new device driver usb
[ 2.166992] omap_i2c 48070000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c1_pins, deferring probe
[ 2.167053] omap_i2c 48072000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c2_pins, deferring probe
[ 2.167083] omap_i2c 48060000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c3_pins, deferring probe
[ 2.167114] omap_i2c 48350000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c4_pins, deferring probe
[ 2.167388] Linux video capture interface: v2.00
[ 2.168151] Advanced Linux Sound Architecture Driver Initialized.
[ 2.169555] clocksource: Switched to clocksource 32k_counter
[ 2.183593] NET: Registered protocol family 2
[ 2.184173] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[ 2.184295] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[ 2.184417] TCP: Hash tables configured (established 8192 bind 8192)
[ 2.184509] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 2.184570] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 2.184753] NET: Registered protocol family 1
[ 2.185119] RPC: Registered named UNIX socket transport module.
[ 2.185119] RPC: Registered udp transport module.
[ 2.185119] RPC: Registered tcp transport module.
[ 2.185150] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 2.187072] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[ 2.189178] futex hash table entries: 512 (order: 3, 32768 bytes)
[ 2.208190] NFS: Registering the id_resolver key type
[ 2.208221] Key type id_resolver registered
[ 2.208221] Key type id_legacy registered
[ 2.208251] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 2.208282] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 2.219512] ntfs: driver 2.1.32 [Flags: R/O].
[ 2.219909] fuse init (API version 7.23)
[ 2.223205] jitterentropy: Initialization failed with host not compliant with requirements: 2
[ 2.223297] bounce: pool size: 64 pages
[ 2.223297] io scheduler noop registered
[ 2.223327] io scheduler deadline registered
[ 2.223358] io scheduler cfq registered (default)
[ 2.224243] pinctrl-single 4a100040.pinmux: 203 pins at pa fc100040 size 406
[ 2.224456] pinctrl-single 4a31e040.pinmux: 28 pins at pa fc31e040 size 56
[ 2.227142] 58000000.dss supply vdda_video not found, using dummy regulator
[ 2.227264] OMAP DSS rev 4.0
[ 2.227416] omapdss_dss 58000000.dss: bound 58001000.dispc (ops dispc_component_ops)
[ 2.227661] omapdss_dss 58000000.dss: bound 58005000.encoder (ops dsi_component_ops)
[ 2.228118] omapdss_dss 58000000.dss: bound 58006000.encoder (ops hdmi4_component_ops)
[ 2.229217] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 2.230865] omap_uart 4806a000.serial: no wakeirq for uart0
[ 2.230987] 4806a000.serial: ttyO0 at MMIO 0x4806a000 (irq = 226, base_baud = 3000000) is a OMAP UART0
[ 2.231475] 4806c000.serial: ttyO1 at MMIO 0x4806c000 (irq = 227, base_baud = 3000000) is a OMAP UART1
[ 2.231933] 48020000.serial: ttyO2 at MMIO 0x48020000 (irq = 228, base_baud = 3000000) is a OMAP UART2
[ 3.190063] console [ttyO2] enabled
[ 3.194213] 4806e000.serial: ttyO3 at MMIO 0x4806e000 (irq = 229, base_baud = 3000000) is a OMAP UART3
[ 3.204986] [drm] Initialized drm 1.1.0 20060810
[ 3.253234] loop: module loaded
[ 3.259002] usbcore: registered new interface driver smsc95xx
[ 3.265106] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 3.271972] ehci-omap: OMAP-EHCI Host Controller driver
[ 3.289611] ehci-omap 4a064c00.ehci: EHCI Host Controller
[ 3.295288] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1
[ 3.303375] ehci-omap 4a064c00.ehci: irq 336, io mem 0x4a064c00
[ 3.329589] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00
[ 3.335968] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 3.343109] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 3.350677] usb usb1: Product: EHCI Host Controller
[ 3.355773] usb usb1: Manufacturer: Linux 4.3.0-panda ehci_hcd
[ 3.361877] usb usb1: SerialNumber: 4a064c00.ehci
[ 3.367462] hub 1-0:1.0: USB hub found
[ 3.371459] hub 1-0:1.0: 3 ports detected
[ 3.376312] usbcore: registered new interface driver cdc_wdm
[ 3.382415] usbcore: registered new interface driver usb-storage
[ 3.389099] i2c /dev entries driver
[ 3.392913] usbcore: registered new interface driver uvcvideo
[ 3.398925] USB Video Class driver (1.1.1)
[ 3.404418] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[ 3.411956] device-mapper: uevent: version 1.0.3
[ 3.417144] device-mapper: ioctl: 4.33.0-ioctl (2015-8-18) initialised: dm-devel@redhat.com
[ 3.490661] hidraw: raw HID events driver (C) Jiri Kosina
[ 3.496520] usbcore: registered new interface driver usbhid
[ 3.502380] usbhid: USB HID core driver
[ 3.506835] emif 4c000000.emif: emif_probe: device configured with addr = f00a8000 and IRQ247
[ 3.516143] emif 4d000000.emif: emif_probe: device configured with addr = f00aa000 and IRQ248
[ 3.527313] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 3.535247] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 3.542755] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 3.548431] Initializing XFRM netlink socket
[ 3.553009] NET: Registered protocol family 17
[ 3.557952] Key type dns_resolver registered
[ 3.562652] twl: not initialized
[ 3.566070] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.574676] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.583190] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.591674] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.600158] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.608642] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.617126] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1410000 Vs max 1316660
[ 3.625640] omap2_set_init_voltage: unable to find boot up OPP for vdd_mpu
[ 3.633148] omap2_set_init_voltage: unable to set vdd_mpu
[ 3.638885] omap2_set_init_voltage: unable to find boot up OPP for vdd_core
[ 3.646270] omap2_set_init_voltage: unable to set vdd_core
[ 3.652099] omap2_set_init_voltage: unable to find boot up OPP for vdd_iva
[ 3.659362] omap2_set_init_voltage: unable to set vdd_iva
[ 3.665985] Power Management for TI OMAP4+ devices.
[ 3.671173] OMAP4 PM: u-boot >= v2012.07 is required for full PM support
[ 3.678375] ThumbEE CPU extension supported.
[ 3.682983] Registering SWP/SWPB emulation handler
[ 3.683013] SmartReflex Class3 initialized
[ 3.692504] usb 1-1: new high-speed USB device number 2 using ehci-omap
[ 3.703735] Btrfs loaded
[ 3.709381] dmm 4e000000.dmm: initialized all PAT entries
[ 3.717864] Skipping twl internal clock init and using bootloader value (unknown osc rate)
[ 3.740112] twl 0-0048: PIH (irq 340) nested IRQs
[ 3.746520] twl_rtc 48070000.i2c:twl@48:rtc: Enabling TWL-RTC
[ 3.755279] twl_rtc 48070000.i2c:twl@48:rtc: rtc core: registered 48070000.i2c:twl@48 as rtc0
[ 3.775146] twl6040 0-004b: clk32k is not handled
[ 3.782806] omap_i2c 48070000.i2c: bus 0 rev0.11 at 400 kHz
[ 3.789489] omap_i2c 48072000.i2c: bus 1 rev0.11 at 400 kHz
[ 3.796417] omap_i2c 48060000.i2c: bus 2 rev0.11 at 100 kHz
[ 3.803283] omap_i2c 48350000.i2c: bus 3 rev0.11 at 400 kHz
[ 3.870880] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
[ 3.879089] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 3.887451] hub 1-1:1.0: USB hub found
[ 3.891754] hub 1-1:1.0: 5 ports detected
[ 4.019531] omap-abe-twl6040 sound: twl6040-legacy <-> 40132000.mcpdm mapping ok
[ 4.030517] twl_rtc 48070000.i2c:twl@48:rtc: setting system clock to 2015-11-02 08:32:10 UTC (1446453130)
[ 4.040588] sr_init: No PMIC hook to init smartreflex
[ 4.046173] smartreflex smartreflex.0: omap_sr_probe: SmartReflex driver initialized
[ 4.054565] smartreflex smartreflex.1: omap_sr_probe: SmartReflex driver initialized
[ 4.062988] smartreflex smartreflex.2: omap_sr_probe: SmartReflex driver initialized
[ 4.081909] VDAC: disabling
[ 4.085357] VANA: disabling
[ 4.088775] VUSB: disabling
[ 4.092102] ALSA device list:
[ 4.095336] #0: PandaBoardES
[ 4.098602] mmc0: host does not support reading read-only switch, assuming write-enable
[ 4.107513] Waiting for root device /dev/mmcblk0p2...
[ 4.113098] mmc0: new high speed SDHC card at address e624
[ 4.119598] mmcblk0: mmc0:e624 SL32G 28.7 GiB
[ 4.154510] mmcblk0: p1 p2 p3 p4
[ 4.164215] omap_hsmmc 480d5000.mmc: card claims to support voltages below defined range
[ 4.182922] mmc1: queuing unknown CIS tuple 0x91 (3 bytes)
[ 4.190002] mmc1: new SDIO card at address 0001
[ 4.202850] usb 1-1.1: new high-speed USB device number 3 using ehci-omap
[ 4.238739] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
[ 4.248046] EXT4-fs (mmcblk0p2): mounting ext2 file system using the ext4 subsystem
[ 4.281005] EXT4-fs (mmcblk0p2): mounted filesystem without journal. Opts: (null)
[ 4.286132] VFS: Mounted root (ext2 filesystem) readonly on device 179:2.
[ 4.301849] devtmpfs: mounted
[ 4.301879] Freeing unused kernel memory: 292K (c08ad000 - c08f6000)
[ 4.355072] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[ 4.362457] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 4.373107] smsc95xx v1.0.4
[ 4.471405] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-4a064c00.ehci-1.1, smsc95xx USB 2.0 Ethernet, 2a:82:11:81:3c:12
[ 4.666351] EXT4-fs (mmcblk0p2): warning: mounting unchecked fs, running e2fsck is recommended
[ 4.720001] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 4.859710] random: dd urandom read with 29 bits of entropy available
[ 9.671081] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[ 20.064331] Bluetooth: Core ver 2.20
[ 20.068206] NET: Registered protocol family 31
[ 20.073333] Bluetooth: HCI device and connection manager initialized
[ 20.080322] Bluetooth: HCI socket layer initialized
[ 20.085754] Bluetooth: L2CAP socket layer initialized
[ 20.091705] Bluetooth: SCO socket layer initialized
[ 27.723480] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[ 27.734954] NFSD: starting 90-second grace period (net c092dbc0)
[ 40.496948] random: nonblocking pool is initialized
[ 77.554626] wlcore: loaded
[ 101.651580] wlcore: firmware booted (Rev 6.3.10.0.133)
[ 105.310150] wlan0: authenticate with 90:f6:52:81:ee:70
[ 105.340179] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[ 105.348236] wlan0: authenticated
[ 105.359619] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[ 105.370391] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[ 105.389373] wlan0: associated
[ 105.451141] wlcore: Association completed.
[ 147.773071] NFSD: Unable to end grace period: -110
[ 261.411010] Adding 2097148k swap on /dev/mmcblk0p3. Priority:-1 extents:1 across:2097148k SS
[16633.358551] INFO: rcu_preempt self-detected stall on CPU
[16633.358551] INFO: rcu_preempt self-detected stall on CPU
[16633.358551] 1: (1 ticks this GP) idle=09f/1/0 softirq=157843/157843 fqs=0
[16633.358551]
[16633.358551] (t=7094 jiffies g=51926 c=51925 q=189)
[16633.358612] rcu_preempt kthread starved for 7094 jiffies! g51926 c51925 f0x0 s3 ->state=0x1
[16633.358612] Task dump for CPU 0:
[16633.358612] swapper/0 R
[16633.358612] running
[16633.358612] 0 0 0 0x00000000
[16633.358642] Backtrace:
[16633.358642] Backtrace aborted due to bad frame pointer <c08f7f54>
[16633.358642] Task dump for CPU 1:
[16633.358642] swapper/1 R
[16633.358642] running
[16633.358642] 0 0 1 0x00000000
[16633.358673] Backtrace:
[16633.358673] [<c0013eac>] (dump_backtrace) from [<c00140a8>] (show_stack+0x18/0x1c)
[16633.358673] r7:c08f8518
[16633.358673] r6:800f0193
[16633.358673] r5:00000001
[16633.358673] r4:ee877200
[16633.358673] [<c0014090>] (show_stack) from [<c005c970>] (sched_show_task+0xbc/0x110)
[16633.358764] [<c005c8b4>] (sched_show_task) from [<c005edd8>] (dump_cpu_task+0x40/0x48)
[16633.358764] r5:c0907800
[16633.358764] r4:00000001
[16633.358795] [<c005ed98>] (dump_cpu_task) from [<c0083028>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[16633.358795] r5:c0907800
[16633.358795] r4:00000002
[16633.358825] [<c0082f8c>] (rcu_dump_cpu_stacks) from [<c00872ac>] (rcu_check_callbacks+0x524/0x908)
[16633.358825] r9:c08f8518
[16633.358825] r8:c0907800
[16633.358825] r7:000000bd
[16633.358825] r6:2e6c5000
[16633.358856] r5:c08f5100
[16633.358856] r4:eefba100
[16633.358856] [<c0086d88>] (rcu_check_callbacks) from [<c0089f38>] (update_process_times+0x40/0x6c)
[16633.358856] r10:c009ae5c
[16633.358856] r9:00000001
[16633.358886] r8:eefb6c4c
[16633.358886] r7:00000f1f
[16633.358886] r6:0c0484d3
[16633.358886] r5:00000000
[16633.358886] r4:ee877200
[16633.358917] [<c0089ef8>] (update_process_times) from [<c009ae58>] (tick_sched_handle+0x50/0x54)
[16633.358917] r5:ee89ff58
[16633.358917] r4:eefb6e40
[16633.358947] [<c009ae08>] (tick_sched_handle) from [<c009aeac>] (tick_sched_timer+0x50/0x94)
[16633.358947] [<c009ae5c>] (tick_sched_timer) from [<c008a94c>] (__hrtimer_run_queues+0x110/0x1a0)
[16633.358947] r7:00000000
[16633.358947] r6:eefb6c40
[16633.358947] r5:eefb6e40
[16633.358978] r4:eefb6c00
[16633.358978] [<c008a83c>] (__hrtimer_run_queues) from [<c008afdc>] (hrtimer_interrupt+0xac/0x1f8)
[16633.358978] r10:eefb6c78
[16633.358978] r9:eefb6c98
[16633.359008] r8:eefb6c14
[16633.359008] r7:eefb6cb8
[16633.359008] r6:ffffffff
[16633.359008] r5:00000003
[16633.359008] r4:eefb6c00
[16633.359039] [<c008af30>] (hrtimer_interrupt) from [<c0017148>] (twd_handler+0x38/0x48)
[16633.359039] r10:c08f84e8
[16633.359039] r9:fa241100
[16633.359039] r8:00000011
[16633.359039] r7:ee828780
[16633.359069] r6:c09158cc
[16633.359069] r5:ee805d80
[16633.359069] r4:00000001
[16633.359069] [<c0017110>] (twd_handler) from [<c007dda8>] (handle_percpu_devid_irq+0x74/0x8c)
[16633.359069] r5:ee805d80
[16633.359069] r4:eefbc740
[16633.359130] [<c007dd34>] (handle_percpu_devid_irq) from [<c0079c54>] (generic_handle_irq+0x2c/0x3c)
[16633.359130] r9:fa241100
[16633.359130] r8:ee808000
[16633.359130] r7:00000001
[16633.359130] r6:00000000
[16633.359130] r5:00000000
[16633.359130] r4:c08f23d0
[16633.359161] [<c0079c28>] (generic_handle_irq) from [<c0079f28>] (__handle_domain_irq+0x64/0xbc)
[16633.359161] [<c0079ec4>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[16633.359161] r9:fa241100
[16633.359161] r8:fa240100
[16633.359191] r7:ee89ff58
[16633.359191] r6:fa24010c
[16633.359191] r5:c08f879c
[16633.359191] r4:c09158c8
[16633.359191] [<c0009470>] (gic_handle_irq) from [<c0014c94>] (__irq_svc+0x54/0x90)
[16633.359222] Exception stack(0xee89ff58 to 0xee89ffa0)
[16633.359222] ff40: 00000000 eefb6390
[16633.359222] ff60: fe600000 00000000 ee89e000 c08f849c c08f1364 c069d320 ee89ffc8 c0934362
[16633.359252] ff80: c08f84e8 ee89ffb4 ee89ff94 ee89ffa8 c002a988 c0010ba8 600f0113 ffffffff
[16633.359252] r9:c0934362
[16633.359252] r8:ee89ffc8
[16633.359252] r7:ee89ff8c
[16633.359252] r6:ffffffff
[16633.359252] r5:600f0113
[16633.359283] r4:c0010ba8
[16633.359283] [<c0010b80>] (arch_cpu_idle) from [<c00707d4>] (default_idle_call+0x28/0x34)
[16633.359313] [<c00707ac>] (default_idle_call) from [<c00708f4>] (cpu_startup_entry+0x114/0x18c)
[16633.359313] [<c00707e0>] (cpu_startup_entry) from [<c001618c>] (secondary_start_kernel+0x160/0x16c)
[16633.359344] r7:c0937288
[16633.359344] r4:c090a880
[16633.359344] [<c001602c>] (secondary_start_kernel) from [<8000958c>] (0x8000958c)
[16633.359344] r5:00000051
[16633.359344] r4:ae88406a
[16633.731079] 0: (1 GPs behind) idle=ed9/1/0 softirq=109281/109282 fqs=35
[16633.737152] (t=7132 jiffies g=51926 c=51925 q=188)
[16633.737152] Task dump for CPU 0:
[16633.748687] swapper/0 R running 0 0 0 0x00000002
[16633.755432] Backtrace:
[16633.755432] [<c0013eac>] (dump_backtrace) from [<c00140a8>] (show_stack+0x18/0x1c)
[16633.755432] r7:c08f8518 r6:800c0193 r5:00000000 r4:c08fabe8
[16633.772064] [<c0014090>] (show_stack) from [<c005c970>] (sched_show_task+0xbc/0x110)
[16633.772064] [<c005c8b4>] (sched_show_task) from [<c005edd8>] (dump_cpu_task+0x40/0x48)
[16633.780273] r5:c0907800 r4:00000000
[16633.792449] [<c005ed98>] (dump_cpu_task) from [<c0083028>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[16633.792449] r5:c0907800 r4:00000001
[16633.804992] [<c0082f8c>] (rcu_dump_cpu_stacks) from [<c00872ac>] (rcu_check_callbacks+0x524/0x908)
[16633.804992] r9:c08f8518 r8:c0907800 r7:000000bc r6:2e6b9000 r5:c08f5100 r4:eefae100
[16633.822692] [<c0086d88>] (rcu_check_callbacks) from [<c0089f38>] (update_process_times+0x40/0x6c)
[16633.822692] r10:c009ae5c r9:00000001 r8:eefaac4c r7:00000f1f r6:0c0484d3 r5:00000000
[16633.840393] r4:c08fabe8
[16633.840393] [<c0089ef8>] (update_process_times) from [<c009ae58>] (tick_sched_handle+0x50/0x54)
[16633.840393] r5:c08f7f10 r4:eefaae40
[16633.856079] [<c009ae08>] (tick_sched_handle) from [<c009aeac>] (tick_sched_timer+0x50/0x94)
[16633.856079] [<c009ae5c>] (tick_sched_timer) from [<c008a94c>] (__hrtimer_run_queues+0x110/0x1a0)
[16633.864929] r7:00000000 r6:eefaac40 r5:eefaae40 r4:eefaac00
[16633.880218] [<c008a83c>] (__hrtimer_run_queues) from [<c008afdc>] (hrtimer_interrupt+0xac/0x1f8)
[16633.880218] r10:eefaac78 r9:eefaac98 r8:eefaac14 r7:eefaacb8 r6:ffffffff r5:00000003
[16633.889526] r4:eefaac00
[16633.889526] [<c008af30>] (hrtimer_interrupt) from [<c0017148>] (twd_handler+0x38/0x48)
[16633.889526] r10:c08f84e8 r9:fa241100 r8:00000011 r7:ee828780 r6:c09158cc r5:ee805d80
[16633.917205] r4:00000001
[16633.917205] [<c0017110>] (twd_handler) from [<c007dda8>] (handle_percpu_devid_irq+0x74/0x8c)
[16633.917205] r5:ee805d80 r4:eefb0740
[16633.932617] [<c007dd34>] (handle_percpu_devid_irq) from [<c0079c54>] (generic_handle_irq+0x2c/0x3c)
[16633.932617] r9:fa241100 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:c08f23d0
[16633.950408] [<c0079c28>] (generic_handle_irq) from [<c0079f28>] (__handle_domain_irq+0x64/0xbc)
[16633.959625] [<c0079ec4>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[16633.959625] r9:fa241100 r8:fa240100 r7:c08f7f10 r6:fa24010c r5:c08f879c r4:c09158c8
[16633.976684] [<c0009470>] (gic_handle_irq) from [<c0014c94>] (__irq_svc+0x54/0x90)
[16633.976684] Exception stack(0xc08f7f10 to 0xc08f7f58)
[16633.976684] 7f00: 00000000 eefaa390 fe600000 00000000
[16633.998596] 7f20: c08f6000 c08f849c c08f1364 c069d320 c08f7f80 c0934362 c08f84e8 c08f7f6c
[16634.007263] 7f40: c08f7f4c c08f7f60 c002a988 c0010ba8 600c0113 ffffffff
[16634.007263] r9:c0934362 r8:c08f7f80 r7:c08f7f44 r6:ffffffff r5:600c0113 r4:c0010ba8
[16634.007263] [<c0010b80>] (arch_cpu_idle) from [<c00707d4>] (default_idle_call+0x28/0x34)
[16634.007263] [<c00707ac>] (default_idle_call) from [<c00708f4>] (cpu_startup_entry+0x114/0x18c)
[16634.040191] [<c00707e0>] (cpu_startup_entry) from [<c069485c>] (rest_init+0x90/0x94)
[16634.040191] r7:ffffffff r4:00000002
[16634.040191] [<c06947cc>] (rest_init) from [<c08adcec>] (start_kernel+0x370/0x37c)
[16634.040191] r5:c0937000 r4:00000000
[16634.063903] [<c08ad97c>] (start_kernel) from [<8000807c>] (0x8000807c)
[31008.610656] INFO: rcu_preempt self-detected stall on CPU
[31008.610687] INFO: rcu_preempt self-detected stall on CPU
[31008.610687] 0: (1 GPs behind) idle=fdf/1/0 softirq=207429/207429 fqs=0
[31008.610687] (t=11770 jiffies g=96530 c=96529 q=189)
[31008.610717] rcu_preempt kthread starved for 11770 jiffies! g96530 c96529 f0x0 s3 ->state=0x1
[31008.610717] Task dump for CPU 0:
[31008.610717] swapper/0 R running 0 0 0 0x00000000
[31008.610717] Backtrace:
[31008.610778] [<c0013eac>] (dump_backtrace) from [<c00140a8>] (show_stack+0x18/0x1c)
[31008.610778] r7:c08f8518 r6:800c0193 r5:00000000 r4:c08fabe8
[31008.610809] [<c0014090>] (show_stack) from [<c005c970>] (sched_show_task+0xbc/0x110)
[31008.610809] [<c005c8b4>] (sched_show_task) from [<c005edd8>] (dump_cpu_task+0x40/0x48)
[31008.610839] r5:c0907800 r4:00000000
[31008.610839] [<c005ed98>] (dump_cpu_task) from [<c0083028>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[31008.610839] r5:c0907800 r4:00000001
[31008.610839] [<c0082f8c>] (rcu_dump_cpu_stacks) from [<c00872ac>] (rcu_check_callbacks+0x524/0x908)
[31008.610900] r9:c08f8518 r8:c0907800 r7:000000bd r6:2e6b9000 r5:c08f5100 r4:eefae100
[31008.610931] [<c0086d88>] (rcu_check_callbacks) from [<c0089f38>] (update_process_times+0x40/0x6c)
[31008.610931] r10:c009ae5c r9:00000001 r8:eefaac4c r7:00001c31 r6:15ca672c r5:00000000
[31008.610931] r4:c08fabe8
[31008.610961] [<c0089ef8>] (update_process_times) from [<c009ae58>] (tick_sched_handle+0x50/0x54)
[31008.610961] r5:c08f7f10 r4:eefaae40
[31008.610992] [<c009ae08>] (tick_sched_handle) from [<c009aeac>] (tick_sched_timer+0x50/0x94)
[31008.610992] [<c009ae5c>] (tick_sched_timer) from [<c008a94c>] (__hrtimer_run_queues+0x110/0x1a0)
[31008.611022] r7:00000000 r6:eefaac40 r5:eefaae40 r4:eefaac00
[31008.611022] [<c008a83c>] (__hrtimer_run_queues) from [<c008afdc>] (hrtimer_interrupt+0xac/0x1f8)
[31008.611053] r10:eefaac78 r9:eefaac98 r8:eefaac14 r7:eefaacb8 r6:ffffffff r5:00000003
[31008.611053] r4:eefaac00
[31008.611083] [<c008af30>] (hrtimer_interrupt) from [<c0017148>] (twd_handler+0x38/0x48)
[31008.611083] r10:c08f84e8 r9:fa241100 r8:00000011 r7:ee828780 r6:c09158cc r5:ee805d80
[31008.611083] r4:00000001
[31008.611083] [<c0017110>] (twd_handler) from [<c007dda8>] (handle_percpu_devid_irq+0x74/0x8c)
[31008.611083] r5:ee805d80 r4:eefb0740
[31008.611145] [<c007dd34>] (handle_percpu_devid_irq) from [<c0079c54>] (generic_handle_irq+0x2c/0x3c)
[31008.611175] r9:fa241100 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:c08f23d0
[31008.611175] [<c0079c28>] (generic_handle_irq) from [<c0079f28>] (__handle_domain_irq+0x64/0xbc)
[31008.611175] [<c0079ec4>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[31008.611206] r9:fa241100 r8:fa240100 r7:c08f7f10 r6:fa24010c r5:c08f879c r4:c09158c8
[31008.611206] [<c0009470>] (gic_handle_irq) from [<c0014c94>] (__irq_svc+0x54/0x90)
[31008.611236] Exception stack(0xc08f7f10 to 0xc08f7f58)
[31008.611236] 7f00: 00000000 eefaa390 fe600000 00000000
[31008.611236] 7f20: c08f6000 c08f849c c08f1364 c069d320 c08f7f80 c0934362 c08f84e8 c08f7f6c
[31008.611267] 7f40: c08f7f4c c08f7f60 c002a988 c0010ba8 600c0113 ffffffff
[31008.611267] r9:c0934362 r8:c08f7f80 r7:c08f7f44 r6:ffffffff r5:600c0113 r4:c0010ba8
[31008.611297] [<c0010b80>] (arch_cpu_idle) from [<c00707d4>] (default_idle_call+0x28/0x34)
[31008.611328] [<c00707ac>] (default_idle_call) from [<c00708f4>] (cpu_startup_entry+0x114/0x18c)
[31008.611328] [<c00707e0>] (cpu_startup_entry) from [<c069485c>] (rest_init+0x90/0x94)
[31008.611358] r7:ffffffff r4:00000002
[31008.611358] [<c06947cc>] (rest_init) from [<c08adcec>] (start_kernel+0x370/0x37c)
[31008.611358] r5:c0937000 r4:00000000
[31008.611419] [<c08ad97c>] (start_kernel) from [<8000807c>] (0x8000807c)
[31008.611419] Task dump for CPU 1:
[31008.611419] swapper/1 R running 0 0 1 0x00000002
[31008.611419] Backtrace:
[31008.611450] Backtrace aborted due to bad frame pointer <ee89ff9c>
[31008.988525] 1: (2 ticks this GP) idle=01d/1/0 softirq=288173/288173 fqs=0
[31008.990051] (t=11809 jiffies g=96530 c=96529 q=103)
[31008.999877] rcu_preempt kthread starved for 11810 jiffies! g96530 c96529 f0x0 s3 ->state=0x1
[31009.010162] Task dump for CPU 1:
[31009.013580] swapper/1 R running 0 0 1 0x00000002
[31009.016937] Backtrace:
[31009.022918] [<c0013eac>] (dump_backtrace) from [<c00140a8>] (show_stack+0x18/0x1c)
[31009.022918] r7:c08f8518 r6:800f0193 r5:00000001 r4:ee877200
[31009.036956] [<c0014090>] (show_stack) from [<c005c970>] (sched_show_task+0xbc/0x110)
[31009.042999] [<c005c8b4>] (sched_show_task) from [<c005edd8>] (dump_cpu_task+0x40/0x48)
[31009.053527] r5:c0907800 r4:00000001
[31009.054962] [<c005ed98>] (dump_cpu_task) from [<c0083028>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[31009.066070] r5:c0907800 r4:00000002
[31009.066070] [<c0082f8c>] (rcu_dump_cpu_stacks) from [<c00872ac>] (rcu_check_callbacks+0x524/0x908)
[31009.077575] r9:c08f8518 r8:c0907800 r7:00000067 r6:2e6c5000 r5:c08f5100 r4:eefba100
[31009.080444] [<c0086d88>] (rcu_check_callbacks) from [<c0089f38>] (update_process_times+0x40/0x6c)
[31009.096954] r10:c009ae5c r9:00000001 r8:eefb6c4c r7:00001c31 r6:1637d5e3 r5:00000000
[31009.098968] r4:ee877200
[31009.107971] [<c0089ef8>] (update_process_times) from [<c009ae58>] (tick_sched_handle+0x50/0x54)
[31009.111083] r5:ee89ff58 r4:eefb6e40
[31009.120971] [<c009ae08>] (tick_sched_handle) from [<c009aeac>] (tick_sched_timer+0x50/0x94)
[31009.128021] [<c009ae5c>] (tick_sched_timer) from [<c008a94c>] (__hrtimer_run_queues+0x110/0x1a0)
[31009.138977] r7:00000000 r6:eefb6c40 r5:eefb6e40 r4:eefb6c00
[31009.138977] [<c008a83c>] (__hrtimer_run_queues) from [<c008afdc>] (hrtimer_interrupt+0xac/0x1f8)
[31009.138977] r10:eefb6c78 r9:eefb6c98 r8:eefb6c14 r7:eefb6cb8 r6:ffffffff r5:00000002
[31009.162719] r4:eefb6c00
[31009.162719] [<c008af30>] (hrtimer_interrupt) from [<c0017148>] (twd_handler+0x38/0x48)
[31009.168975] r10:c08f84e8 r9:fa241100 r8:00000011 r7:ee828780 r6:c09158cc r5:ee805d80
[31009.178955] r4:00000001
[31009.184783] [<c0017110>] (twd_handler) from [<c007dda8>] (handle_percpu_devid_irq+0x74/0x8c)
[31009.193115] r5:ee805d80 r4:eefbc740
[31009.197509] [<c007dd34>] (handle_percpu_devid_irq) from [<c0079c54>] (generic_handle_irq+0x2c/0x3c)
[31009.204711] r9:fa241100 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:c08f23d0
[31009.215301] [<c0079c28>] (generic_handle_irq) from [<c0079f28>] (__handle_domain_irq+0x64/0xbc)
[31009.222198] [<c0079ec4>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[31009.232177] r9:fa241100 r8:fa240100 r7:ee89ff58 r6:fa24010c r5:c08f879c r4:c09158c8
[31009.236633] [<c0009470>] (gic_handle_irq) from [<c0014c94>] (__irq_svc+0x54/0x90)
[31009.249481] Exception stack(0xee89ff58 to 0xee89ffa0)
[31009.254821] ff40: 00000000 eefb6390
[31009.254821] ff60: fe600000 00000000 ee89e000 c08f849c c08f1364 c069d320 ee89ffc8 c0934362
[31009.268981] ff80: c08f84e8 ee89ffb4 ee89ff94 ee89ffa8 c002a988 c0010ba8 600f0113 ffffffff
[31009.278991] r9:c0934362 r8:ee89ffc8 r7:ee89ff8c r6:ffffffff r5:600f0113 r4:c0010ba8
[31009.280792] [<c0010b80>] (arch_cpu_idle) from [<c00707d4>] (default_idle_call+0x28/0x34)
[31009.297576] [<c00707ac>] (default_idle_call) from [<c00708f4>] (cpu_startup_entry+0x114/0x18c)
[31009.306701] [<c00707e0>] (cpu_startup_entry) from [<c001618c>] (secondary_start_kernel+0x160/0x16c)
[31009.311218] r7:c0937288 r4:c090a880
[31009.319000] [<c001602c>] (secondary_start_kernel) from [<8000958c>] (0x8000958c)
[31009.319000] r5:00000051 r4:ae88406a
[79853.606109] wlcore: Beacon loss detected. roles:0xff
[79858.406158] wlcore: Connection loss work (role_id: 0).
[79858.411621] wlan0: Connection to AP 90:f6:52:81:ee:70 lost
[79862.045989] wlan0: authenticate with 90:f6:52:81:ee:70
[79862.092102] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[79862.100311] wlan0: authenticated
[79862.104766] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[79862.118896] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[79862.137695] wlan0: associated
[79862.191680] wlcore: Association completed.
$
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: rcu_preempt self-detected stall on CPU from 4.4-rc4, since 3.17
2016-01-03 8:27 ` Ross Green
@ 2016-01-03 18:15 ` Paul E. McKenney
2016-01-03 22:00 ` Ross Green
0 siblings, 1 reply; 12+ messages in thread
From: Paul E. McKenney @ 2016-01-03 18:15 UTC (permalink / raw)
To: Ross Green
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
Mathieu Desnoyers, josh, tglx, peterz, rostedt, dhowells,
Eric Dumazet, dvhart, Frédéric Weisbecker, oleg,
pranith kumar
On Sun, Jan 03, 2016 at 07:27:17PM +1100, Ross Green wrote:
> I would not describe the load on this test machine as high or real time.
>
> Apart from a number of standard daemons not much more is running at all!
>
> I normally build a release kernel as soon as possible and set it running.
> Typically I run a series of benchmarks to confirm most things appear
> to be working and then just leave it running. During a normal day i
> will check on the machine 4/5 times just to see how its going!
> Typically I will logon remotely via wifi network connection.
>
> just for your information i will include a few other stack traces from
> previous kernels that may show some trend!
>
>
> Please see the attachments.
Thank you for the additional details. This does look similar to some
problems I am seeing, though only in heavy rcutorture workloads with
CPU hotplugging.
I have some crude workarounds in progress, see for example
2da26818e515 (rcu: Awaken grace-period kthread when stalled) at
git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git.
This workaround kicks the RCU grace-period kthread on every stall warning.
In my testing, this workaround results in slow but real forward progress.
I have a better workaround in progress, however, please note:
1. I have no intention of sending these workarounds upstream.
2. The workarounds will splat when they take effect. In other words,
the idea is not to paper over the problem, but instead to allow
me to separate testing concerns.
3. A fix is needed for the underlying bug, wherever it might be.
Thanx, Paul
> Regards,
>
> Ross
>
>
>
> On Sun, Jan 3, 2016 at 5:17 PM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Sun, Jan 03, 2016 at 04:29:11PM +1100, Ross Green wrote:
> >> Still seeing these rcu_preempt stalls on kernels through to 4.4-rc7
> >>
> >> Still have not found a sure fire method to evoke this stall, but have
> >> found that it will normally occur within a week of running a kernel -
> >> usually when it is quiet with light load.
> >>
> >> Have seen similar self detected stalls all the way back to 3.17.
> >> Most recent kernels have included 4.4-rc5 4.4-rc6 and 4.4-rc7
> >>
> >> Regards,
> >>
> >> Ross
> >>
> >> On Fri, Dec 11, 2015 at 10:17 PM, Ross Green <rgkernel@gmail.com> wrote:
> >> > I have been getting these stalls in kernels going back to 3.17.
> >> >
> >> > This stall occurs usually under light load but often requires several
> >> > days to show itself. I have not found any simple way to trigger the
> >> > stall. Indeed heavy workloads seems not to show the fault.
> >> >
> >> > Anyone have any thoughts here?
> >> >
> >> > The recent patch by peterz with kernel/sched/wait.c I thought might
> >> > help the situation, but alas after a few days of running 4.4-rc4 the
> >> > following turned up.
> >> >
> >> > [179922.003570] INFO: rcu_preempt self-detected stall on CPU
> >> > [179922.008178] INFO: rcu_preempt detected stalls on CPUs/tasks:
> >> > [179922.008178] 0-...: (1 ticks this GP) idle=a91/1/0
> >
> > CPU 0 is non-idle from an RCU perspective.
> >
> >> > softirq=1296733/1296733 fqs=0
> >> > [179922.008178]
> >> > [179922.008209] (detected by 1, t=8775 jiffies, g=576439, c=576438, q=102)
> >> > [179922.008209] Task dump for CPU 0:
> >> > [179922.008209] swapper/0 R [179922.008209] running [179922.008209] 0 0 0 0x00000000
> >> > [179922.008209] Backtrace:
> >> >
> >> > [179922.008239] Backtrace aborted due to bad frame pointer <c0907f54>
> >
> > Can't have everything, I guess...
> >
> >> > [179922.008239] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x0 s3 ->state=0x1
> >
> > Something is keeping the rcu_preempt grace-period kthread from
> > running. This far into the grace period, it should have a
> > timer event waking it every few jiffies. It is currently
> > in TASK_INTERRUPTIBLE state.
> >
> >> > [179922.060302] 0-...: (1 ticks this GP) idle=a91/1/0 softirq=1296733/1296733 fqs=0
> >> > [179922.068023] (t=8775 jiffies g=576439 c=576438 q=102)
> >> > [179922.073913] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x2 s3 ->state=0x100
> >
> > Same story, same grace period, pretty much same time. Now there is an FQS
> > request (f0x2) and the state is now TASK_WAKING (->state=0x100 == 256).
> >
> >> > [179922.083587] Task dump for CPU 0:
> >> > [179922.087097] swapper/0 R running 0 0 0 0x00000000
> >> > [179922.093292] Backtrace:
> >> > [179922.096313] [<c0013ea8>] (dump_backtrace) from [<c00140a4>] (show_stack+0x18/0x1c)
> >> > [179922.104675] r7:c0908514 r6:80080193 r5:00000000 r4:c090aca8
> >> > [179922.110809] [<c001408c>] (show_stack) from [<c005a858>] (sched_show_task+0xbc/0x110)
> >> > [179922.119049] [<c005a79c>] (sched_show_task) from [<c005ccd4>] (dump_cpu_task+0x40/0x44)
> >> > [179922.127624] r5:c0917680 r4:00000000
> >> > [179922.131042] [<c005cc94>] (dump_cpu_task) from [<c0082268>] (rcu_dump_cpu_stacks+0x9c/0xdc)
> >> > [179922.140350] r5:c0917680 r4:00000001
> >> > [179922.143157] [<c00821cc>] (rcu_dump_cpu_stacks) from [<c008637c>] (rcu_check_callbacks+0x504/0x8e4)
> >> > [179922.153808] r9:c0908514 r8:c0917680 r7:00000066 r6:2eeab000
> >> > r5:c0904300 r4:ef7af300
> >> > [179922.161499] [<c0085e78>] (rcu_check_callbacks) from [<c00895d0>] (update_process_times+0x40/0x6c)
> >> > [179922.170898] r10:c009a584 r9:00000001 r8:ef7abc4c r7:0000a3a3
> >> > r6:4ec3391c r5:00000000
> >> > [179922.179901] r4:c090aca8
> >> > [179922.182708] [<c0089590>] (update_process_times) from [<c009a580>]
> >> > (tick_sched_handle+0x50/0x54)
> >> > [179922.192108] r5:c0907f10 r4:ef7abe40
> >> > [179922.195983] [<c009a530>] (tick_sched_handle) from [<c009a5d4>]
> >> > (tick_sched_timer+0x50/0x94)
> >> > [179922.204895] [<c009a584>] (tick_sched_timer) from [<c0089fe4>]
> >> > (__hrtimer_run_queues+0x110/0x1a0)
> >> > [179922.214324] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
> >> > [179922.220428] [<c0089ed4>] (__hrtimer_run_queues) from [<c008a674>]
> >> > (hrtimer_interrupt+0xac/0x1f8)
> >> > [179922.227111] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8
> >> > r6:ffffffff r5:00000003
> >> > [179922.238220] r4:ef7abc00
> >> > [179922.238220] [<c008a5c8>] (hrtimer_interrupt) from [<c00170ec>]
> >> > (twd_handler+0x38/0x48)
> >> > [179922.238220] r10:c09084e8 r9:fa241100 r8:00000011 r7:ef028780
> >> > r6:c092574c r5:ef005cc0
> >
> > All interrupt stack up to this point.
> >
> > It is quite possible that the stuff below here is at fault as well.
> > That said, the grace-period should actually get to execute at some
> > point. Do you have a heavy real-time load that might be starving
> > things?
> >
> > Thanx, Paul
> >
> >> > [179922.257110] r4:00000001
> >> > [179922.257110] [<c00170b4>] (twd_handler) from [<c007c8f8>] (handle_percpu_devid_irq+0x74/0x8c)
> >> > [179922.269683] r5:ef005cc0 r4:ef7b1740
> >> > [179922.269683] [<c007c884>] (handle_percpu_devid_irq) from [<c0078454>] (generic_handle_irq+0x2c/0x3c)
> >> > [179922.283233] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000
> >> > r5:00000000 r4:c09013e8
> >> > [179922.290985] [<c0078428>] (generic_handle_irq) from [<c007872c>] (__handle_domain_irq+0x64/0xbc)
> >> > [179922.300842] [<c00786c8>] (__handle_domain_irq) from [<c00094c0>]
> >> > (gic_handle_irq+0x50/0x90)
> >> > [179922.303222] r9:fa241100 r8:fa240100 r7:c0907f10 r6:fa24010c
> >> > r5:c09087a8 r4:c0925748
> >> > [179922.315216] [<c0009470>] (gic_handle_irq) from [<c0014bd4>]
> >> > (__irq_svc+0x54/0x90)
> >> > [179922.319000] Exception stack(0xc0907f10 to 0xc0907f58)
> >> > [179922.331542] 7f00: 00000000
> >> > ef7ab390 fe600000 00000000
> >> > [179922.331542] 7f20: c0906000 c090849c c0900364 c06a8124 c0907f80
> >> > c0944563 c09084e8 c0907f6c
> >> > [179922.349029] 7f40: c0907f4c c0907f60 c00287ac c0010ba8 60080113 ffffffff
> >> > [179922.349029] r9:c0944563 r8:c0907f80 r7:c0907f44 r6:ffffffff
> >> > r5:60080113 r4:c0010ba8
> >> > [179922.357116] [<c0010b80>] (arch_cpu_idle) from [<c006f034>]
> >> > (default_idle_call+0x28/0x34)
> >> > [179922.368926] [<c006f00c>] (default_idle_call) from [<c006f154>]
> >> > (cpu_startup_entry+0x114/0x18c)
> >> > [179922.368926] [<c006f040>] (cpu_startup_entry) from [<c069fc6c>]
> >> > (rest_init+0x90/0x94)
> >> > [179922.385284] r7:ffffffff r4:00000002
> >> > [179922.393463] [<c069fbdc>] (rest_init) from [<c08bbcec>]
> >> > (start_kernel+0x370/0x37c)
> >> > [179922.400421] r5:c0947000 r4:00000000
> >> > [179922.400421] [<c08bb97c>] (start_kernel) from [<8000807c>] (0x8000807c)
> >> > $
> >
> >
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: rcu_preempt self-detected stall on CPU from 4.4-rc4, since 3.17
2016-01-03 18:15 ` Paul E. McKenney
@ 2016-01-03 22:00 ` Ross Green
2016-01-04 14:21 ` Ross Green
0 siblings, 1 reply; 12+ messages in thread
From: Ross Green @ 2016-01-03 22:00 UTC (permalink / raw)
To: Paul McKenney
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
Mathieu Desnoyers, josh, tglx, peterz, rostedt, dhowells,
Eric Dumazet, dvhart, Frédéric Weisbecker, oleg,
pranith kumar
[-- Attachment #1: Type: text/plain, Size: 10526 bytes --]
Thanks Paul for your analysis and investigation,
I understand your patches are designed not to "fix" the problem, but
more to move the problem ahead.
I will include a few more stack traces from various kernels. I can go
back to around 3.17, with similar trace results.
My observation is that the problem can occur at various times and with
out any "bad" effect other than more stalls could happen afterwards.
At first I wondered if they might actually be a false positive as the
kernel seemed to carry on and run quite happily. It is rare that I
find a kernel just locks up after observing such a stall, or a
complete kernel splat!
Unfortunately with my testing I have never been able to induce the
problem under any heavy load that would immediately trigger the
problem. Indeed most heavy cpu utilisation seemed to just sail on
quite nicely.
The time for a fault, seems so far, to be non-deterministic with
quiescent systems taking anywhere from a few hours through to some six
days before showing the problem.
More recent kernels seem to have richer stall and back trace
information so I was hoping that might shed some light on how they
might be occurring..
I usually only run a kernel till I get a new -rc release to test with,
so a run of 1 week is a typical cycle.
I just wish I could find a sure fire method to trigger the problem!!
I have included a few more traces of various kernels all showing the problem.
Regards,
Ross
On Mon, Jan 4, 2016 at 5:15 AM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Sun, Jan 03, 2016 at 07:27:17PM +1100, Ross Green wrote:
>> I would not describe the load on this test machine as high or real time.
>>
>> Apart from a number of standard daemons not much more is running at all!
>>
>> I normally build a release kernel as soon as possible and set it running.
>> Typically I run a series of benchmarks to confirm most things appear
>> to be working and then just leave it running. During a normal day i
>> will check on the machine 4/5 times just to see how its going!
>> Typically I will logon remotely via wifi network connection.
>>
>> just for your information i will include a few other stack traces from
>> previous kernels that may show some trend!
>>
>>
>> Please see the attachments.
>
> Thank you for the additional details. This does look similar to some
> problems I am seeing, though only in heavy rcutorture workloads with
> CPU hotplugging.
>
> I have some crude workarounds in progress, see for example
> 2da26818e515 (rcu: Awaken grace-period kthread when stalled) at
> git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git.
> This workaround kicks the RCU grace-period kthread on every stall warning.
> In my testing, this workaround results in slow but real forward progress.
>
> I have a better workaround in progress, however, please note:
>
> 1. I have no intention of sending these workarounds upstream.
>
> 2. The workarounds will splat when they take effect. In other words,
> the idea is not to paper over the problem, but instead to allow
> me to separate testing concerns.
>
> 3. A fix is needed for the underlying bug, wherever it might be.
>
> Thanx, Paul
>
>> Regards,
>>
>> Ross
>>
>>
>>
>> On Sun, Jan 3, 2016 at 5:17 PM, Paul E. McKenney
>> <paulmck@linux.vnet.ibm.com> wrote:
>> > On Sun, Jan 03, 2016 at 04:29:11PM +1100, Ross Green wrote:
>> >> Still seeing these rcu_preempt stalls on kernels through to 4.4-rc7
>> >>
>> >> Still have not found a sure fire method to evoke this stall, but have
>> >> found that it will normally occur within a week of running a kernel -
>> >> usually when it is quiet with light load.
>> >>
>> >> Have seen similar self detected stalls all the way back to 3.17.
>> >> Most recent kernels have included 4.4-rc5 4.4-rc6 and 4.4-rc7
>> >>
>> >> Regards,
>> >>
>> >> Ross
>> >>
>> >> On Fri, Dec 11, 2015 at 10:17 PM, Ross Green <rgkernel@gmail.com> wrote:
>> >> > I have been getting these stalls in kernels going back to 3.17.
>> >> >
>> >> > This stall occurs usually under light load but often requires several
>> >> > days to show itself. I have not found any simple way to trigger the
>> >> > stall. Indeed heavy workloads seems not to show the fault.
>> >> >
>> >> > Anyone have any thoughts here?
>> >> >
>> >> > The recent patch by peterz with kernel/sched/wait.c I thought might
>> >> > help the situation, but alas after a few days of running 4.4-rc4 the
>> >> > following turned up.
>> >> >
>> >> > [179922.003570] INFO: rcu_preempt self-detected stall on CPU
>> >> > [179922.008178] INFO: rcu_preempt detected stalls on CPUs/tasks:
>> >> > [179922.008178] 0-...: (1 ticks this GP) idle=a91/1/0
>> >
>> > CPU 0 is non-idle from an RCU perspective.
>> >
>> >> > softirq=1296733/1296733 fqs=0
>> >> > [179922.008178]
>> >> > [179922.008209] (detected by 1, t=8775 jiffies, g=576439, c=576438, q=102)
>> >> > [179922.008209] Task dump for CPU 0:
>> >> > [179922.008209] swapper/0 R [179922.008209] running [179922.008209] 0 0 0 0x00000000
>> >> > [179922.008209] Backtrace:
>> >> >
>> >> > [179922.008239] Backtrace aborted due to bad frame pointer <c0907f54>
>> >
>> > Can't have everything, I guess...
>> >
>> >> > [179922.008239] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x0 s3 ->state=0x1
>> >
>> > Something is keeping the rcu_preempt grace-period kthread from
>> > running. This far into the grace period, it should have a
>> > timer event waking it every few jiffies. It is currently
>> > in TASK_INTERRUPTIBLE state.
>> >
>> >> > [179922.060302] 0-...: (1 ticks this GP) idle=a91/1/0 softirq=1296733/1296733 fqs=0
>> >> > [179922.068023] (t=8775 jiffies g=576439 c=576438 q=102)
>> >> > [179922.073913] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x2 s3 ->state=0x100
>> >
>> > Same story, same grace period, pretty much same time. Now there is an FQS
>> > request (f0x2) and the state is now TASK_WAKING (->state=0x100 == 256).
>> >
>> >> > [179922.083587] Task dump for CPU 0:
>> >> > [179922.087097] swapper/0 R running 0 0 0 0x00000000
>> >> > [179922.093292] Backtrace:
>> >> > [179922.096313] [<c0013ea8>] (dump_backtrace) from [<c00140a4>] (show_stack+0x18/0x1c)
>> >> > [179922.104675] r7:c0908514 r6:80080193 r5:00000000 r4:c090aca8
>> >> > [179922.110809] [<c001408c>] (show_stack) from [<c005a858>] (sched_show_task+0xbc/0x110)
>> >> > [179922.119049] [<c005a79c>] (sched_show_task) from [<c005ccd4>] (dump_cpu_task+0x40/0x44)
>> >> > [179922.127624] r5:c0917680 r4:00000000
>> >> > [179922.131042] [<c005cc94>] (dump_cpu_task) from [<c0082268>] (rcu_dump_cpu_stacks+0x9c/0xdc)
>> >> > [179922.140350] r5:c0917680 r4:00000001
>> >> > [179922.143157] [<c00821cc>] (rcu_dump_cpu_stacks) from [<c008637c>] (rcu_check_callbacks+0x504/0x8e4)
>> >> > [179922.153808] r9:c0908514 r8:c0917680 r7:00000066 r6:2eeab000
>> >> > r5:c0904300 r4:ef7af300
>> >> > [179922.161499] [<c0085e78>] (rcu_check_callbacks) from [<c00895d0>] (update_process_times+0x40/0x6c)
>> >> > [179922.170898] r10:c009a584 r9:00000001 r8:ef7abc4c r7:0000a3a3
>> >> > r6:4ec3391c r5:00000000
>> >> > [179922.179901] r4:c090aca8
>> >> > [179922.182708] [<c0089590>] (update_process_times) from [<c009a580>]
>> >> > (tick_sched_handle+0x50/0x54)
>> >> > [179922.192108] r5:c0907f10 r4:ef7abe40
>> >> > [179922.195983] [<c009a530>] (tick_sched_handle) from [<c009a5d4>]
>> >> > (tick_sched_timer+0x50/0x94)
>> >> > [179922.204895] [<c009a584>] (tick_sched_timer) from [<c0089fe4>]
>> >> > (__hrtimer_run_queues+0x110/0x1a0)
>> >> > [179922.214324] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
>> >> > [179922.220428] [<c0089ed4>] (__hrtimer_run_queues) from [<c008a674>]
>> >> > (hrtimer_interrupt+0xac/0x1f8)
>> >> > [179922.227111] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8
>> >> > r6:ffffffff r5:00000003
>> >> > [179922.238220] r4:ef7abc00
>> >> > [179922.238220] [<c008a5c8>] (hrtimer_interrupt) from [<c00170ec>]
>> >> > (twd_handler+0x38/0x48)
>> >> > [179922.238220] r10:c09084e8 r9:fa241100 r8:00000011 r7:ef028780
>> >> > r6:c092574c r5:ef005cc0
>> >
>> > All interrupt stack up to this point.
>> >
>> > It is quite possible that the stuff below here is at fault as well.
>> > That said, the grace-period should actually get to execute at some
>> > point. Do you have a heavy real-time load that might be starving
>> > things?
>> >
>> > Thanx, Paul
>> >
>> >> > [179922.257110] r4:00000001
>> >> > [179922.257110] [<c00170b4>] (twd_handler) from [<c007c8f8>] (handle_percpu_devid_irq+0x74/0x8c)
>> >> > [179922.269683] r5:ef005cc0 r4:ef7b1740
>> >> > [179922.269683] [<c007c884>] (handle_percpu_devid_irq) from [<c0078454>] (generic_handle_irq+0x2c/0x3c)
>> >> > [179922.283233] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000
>> >> > r5:00000000 r4:c09013e8
>> >> > [179922.290985] [<c0078428>] (generic_handle_irq) from [<c007872c>] (__handle_domain_irq+0x64/0xbc)
>> >> > [179922.300842] [<c00786c8>] (__handle_domain_irq) from [<c00094c0>]
>> >> > (gic_handle_irq+0x50/0x90)
>> >> > [179922.303222] r9:fa241100 r8:fa240100 r7:c0907f10 r6:fa24010c
>> >> > r5:c09087a8 r4:c0925748
>> >> > [179922.315216] [<c0009470>] (gic_handle_irq) from [<c0014bd4>]
>> >> > (__irq_svc+0x54/0x90)
>> >> > [179922.319000] Exception stack(0xc0907f10 to 0xc0907f58)
>> >> > [179922.331542] 7f00: 00000000
>> >> > ef7ab390 fe600000 00000000
>> >> > [179922.331542] 7f20: c0906000 c090849c c0900364 c06a8124 c0907f80
>> >> > c0944563 c09084e8 c0907f6c
>> >> > [179922.349029] 7f40: c0907f4c c0907f60 c00287ac c0010ba8 60080113 ffffffff
>> >> > [179922.349029] r9:c0944563 r8:c0907f80 r7:c0907f44 r6:ffffffff
>> >> > r5:60080113 r4:c0010ba8
>> >> > [179922.357116] [<c0010b80>] (arch_cpu_idle) from [<c006f034>]
>> >> > (default_idle_call+0x28/0x34)
>> >> > [179922.368926] [<c006f00c>] (default_idle_call) from [<c006f154>]
>> >> > (cpu_startup_entry+0x114/0x18c)
>> >> > [179922.368926] [<c006f040>] (cpu_startup_entry) from [<c069fc6c>]
>> >> > (rest_init+0x90/0x94)
>> >> > [179922.385284] r7:ffffffff r4:00000002
>> >> > [179922.393463] [<c069fbdc>] (rest_init) from [<c08bbcec>]
>> >> > (start_kernel+0x370/0x37c)
>> >> > [179922.400421] r5:c0947000 r4:00000000
>> >> > [179922.400421] [<c08bb97c>] (start_kernel) from [<8000807c>] (0x8000807c)
>> >> > $
>> >
>> >
>
>
>
[-- Attachment #2: dmesg-4.2.3 --]
[-- Type: application/octet-stream, Size: 31771 bytes --]
$
$ uptime
12:57:13 up 6 days, 17:26, load average: 0.13, 0.10, 0.13
$ cat /proc/version
Linux version 4.2.3-panda (rfg@red.dwarf) (gcc version 5.2.0 (Buildroot 2015.08-rc1-00052-gef3d042) ) #1 SMP PREEMPT Sun Oct 4 17:33:09 AEDT 2015
$
$
$ dmesg
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 4.2.3-panda (rfg@red.dwarf) (gcc version 5.2.0 (Buildroot 2015.08-rc1-00052-gef3d042) ) #1 SMP PREEMPT Sun Oct 4 17:33:09 AEDT 2015
[ 0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] Machine model: TI OMAP4 PandaBoard-ES
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] On node 0 totalpages: 261888
[ 0.000000] free_area_init_node: node 0, pgdat c0912f00, node_mem_map eeff9000
[ 0.000000] Normal zone: 1520 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 194560 pages, LIFO batch:31
[ 0.000000] HighMem zone: 67328 pages, LIFO batch:15
[ 0.000000] OMAP4460 ES1.1
[ 0.000000] PERCPU: Embedded 11 pages/cpu @eefac000 s13632 r8192 d23232 u45056
[ 0.000000] pcpu-alloc: s13632 r8192 d23232 u45056 alloc=11*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260368
[ 0.000000] Kernel command line: root=/dev/mmcblk0p2 rootwait console=ttyO2,115200
[ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Memory: 1028600K/1047552K available (6643K kernel code, 247K rwdata, 2100K rodata, 268K init, 294K bss, 18952K reserved, 0K cma-reserved, 269312K highmem)
[ 0.000000] Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
vmalloc : 0xf0000000 - 0xff000000 ( 240 MB)
lowmem : 0xc0000000 - 0xef800000 ( 760 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc08920dc (8745 kB)
.init : 0xc0893000 - 0xc08d6000 ( 268 kB)
.data : 0xc08d6000 - 0xc0913ec0 ( 248 kB)
.bss : 0xc0916000 - 0xc095f974 ( 295 kB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] Additional per-CPU info printed with stalls.
[ 0.000000] Build-time adjustment of leaf fanout to 32.
[ 0.000000] NR_IRQS:16 nr_irqs:16 16
[ 0.000000] L2C: platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C: DT/platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C-310 enabling early BRESP for Cortex-A9
[ 0.000000] OMAP L2C310: ROM does not support power control setting
[ 0.000000] L2C-310 dynamic clock gating disabled, standby mode disabled
[ 0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
[ 0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e470000
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node bandgap_fclk
[ 0.000000] OMAP clockevent source: timer1 at 32768 Hz
[ 0.000000] clocksource: 32k_counter: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 58327039986419 ns
[ 0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65535999984741ns
[ 0.000000] OMAP clocksource: 32k_counter at 32768 Hz
[ 0.001647] Console: colour dummy device 80x30
[ 0.001678] Calibrating delay loop... 1397.55 BogoMIPS (lpj=6987776)
[ 0.090240] pid_max: default: 32768 minimum: 301
[ 0.090301] Security Framework initialized
[ 0.090362] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.090362] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.090972] Initializing cgroup subsys blkio
[ 0.090972] Initializing cgroup subsys devices
[ 0.091003] Initializing cgroup subsys freezer
[ 0.091003] CPU: Testing write buffer coherency: ok
[ 0.091217] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.091308] Setting up static identity map for 0x80008280 - 0x800082d8
[ 0.160217] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.160308] Brought up 2 CPUs
[ 0.160308] SMP: Total of 2 processors activated (2795.11 BogoMIPS).
[ 0.160339] CPU: All CPU(s) started in SVC mode.
[ 0.160797] devtmpfs: initialized
[ 0.178558] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[ 0.180297] omap_hwmod: l3_main_3 using broken dt data from ocp
[ 0.181854] omap_hwmod: l3_main_2 using broken dt data from ocp
[ 0.278686] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.278930] xor: measuring software checksum speed
[ 0.370208] arm4regs : 1090.800 MB/sec
[ 0.470214] 8regs : 857.200 MB/sec
[ 0.570220] 32regs : 798.000 MB/sec
[ 0.570220] xor: using function: arm4regs (1090.800 MB/sec)
[ 0.570465] pinctrl core: initialized pinctrl subsystem
[ 0.572326] NET: Registered protocol family 16
[ 0.573211] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.581970] OMAP GPIO hardware version 0.1
[ 0.586517] omap-gpmc 50000000.gpmc: GPMC revision 6.0
[ 0.586547] gpmc_mem_init: disabling cs 0 mapped at 0x0-0x1000000
[ 0.587066] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.587432] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.587799] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.599456] platform 4b501000.aes: Cannot lookup hwmod 'aes'
[ 0.599761] platform 480a5000.des: Cannot lookup hwmod 'des'
[ 0.605743] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[ 0.605743] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.606994] OMAP DMA hardware revision 0.0
[ 0.820312] raid6: int32x1 gen() 103 MB/s
[ 0.990447] raid6: int32x1 xor() 82 MB/s
[ 1.160675] raid6: int32x2 gen() 134 MB/s
[ 1.330444] raid6: int32x2 xor() 98 MB/s
[ 1.500335] raid6: int32x4 gen() 145 MB/s
[ 1.670257] raid6: int32x4 xor() 97 MB/s
[ 1.840301] raid6: int32x8 gen() 160 MB/s
[ 2.010437] raid6: int32x8 xor() 94 MB/s
[ 2.010467] raid6: using algorithm int32x8 gen() 160 MB/s
[ 2.010467] raid6: .... xor() 94 MB/s, rmw enabled
[ 2.010467] raid6: using intx1 recovery algorithm
[ 2.017669] omap-dma-engine 4a056000.dma-controller: OMAP DMA engine driver
[ 2.090362] reg-fixed-voltage wl12xx_vmmc: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_wl12xx_gpio, deferring probe
[ 2.094360] omap-iommu 4a066000.mmu: 4a066000.mmu registered
[ 2.094482] omap-iommu 55082000.mmu: 55082000.mmu registered
[ 2.095428] SCSI subsystem initialized
[ 2.095733] usbcore: registered new interface driver usbfs
[ 2.095855] usbcore: registered new interface driver hub
[ 2.096008] usbcore: registered new device driver usb
[ 2.096984] omap_i2c 48070000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c1_pins, deferring probe
[ 2.097015] omap_i2c 48072000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c2_pins, deferring probe
[ 2.097076] omap_i2c 48060000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c3_pins, deferring probe
[ 2.097106] omap_i2c 48350000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c4_pins, deferring probe
[ 2.097381] Linux video capture interface: v2.00
[ 2.098175] Advanced Linux Sound Architecture Driver Initialized.
[ 2.099517] clocksource: Switched to clocksource 32k_counter
[ 2.111785] NET: Registered protocol family 2
[ 2.112457] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[ 2.112579] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[ 2.112701] TCP: Hash tables configured (established 8192 bind 8192)
[ 2.112792] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 2.112854] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 2.113037] NET: Registered protocol family 1
[ 2.113372] RPC: Registered named UNIX socket transport module.
[ 2.113372] RPC: Registered udp transport module.
[ 2.113372] RPC: Registered tcp transport module.
[ 2.113372] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 2.115325] hw perfevents: Failed to parse /pmu/interrupt-affinity[0]
[ 2.115386] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[ 2.117523] futex hash table entries: 512 (order: 3, 32768 bytes)
[ 2.126373] NFS: Registering the id_resolver key type
[ 2.126403] Key type id_resolver registered
[ 2.126403] Key type id_legacy registered
[ 2.126434] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 2.126464] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 2.127502] ntfs: driver 2.1.32 [Flags: R/O].
[ 2.127807] fuse init (API version 7.23)
[ 2.131469] jitterentropy: Initialization failed with host not compliant with requirements: 2
[ 2.131530] bounce: pool size: 64 pages
[ 2.131561] io scheduler noop registered
[ 2.131561] io scheduler deadline registered
[ 2.131622] io scheduler cfq registered (default)
[ 2.132476] pinctrl-single 4a100040.pinmux: 203 pins at pa fc100040 size 406
[ 2.132690] pinctrl-single 4a31e040.pinmux: 28 pins at pa fc31e040 size 56
[ 2.135406] 58000000.dss supply vdda_video not found, using dummy regulator
[ 2.135528] OMAP DSS rev 4.0
[ 2.135742] omapdss_dss 58000000.dss: bound 58001000.dispc (ops dispc_component_ops)
[ 2.136016] omapdss_dss 58000000.dss: bound 58005000.encoder (ops dsi_component_ops)
[ 2.136444] omapdss_dss 58000000.dss: bound 58006000.encoder (ops hdmi4_component_ops)
[ 2.136718] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 2.138275] omap_uart 4806a000.serial: no wakeirq for uart0
[ 2.138397] 4806a000.serial: ttyO0 at MMIO 0x4806a000 (irq = 226, base_baud = 3000000) is a OMAP UART0
[ 2.138885] 4806c000.serial: ttyO1 at MMIO 0x4806c000 (irq = 227, base_baud = 3000000) is a OMAP UART1
[ 2.139373] 48020000.serial: ttyO2 at MMIO 0x48020000 (irq = 228, base_baud = 3000000) is a OMAP UART2
[ 3.094879] console [ttyO2] enabled
[ 3.098999] 4806e000.serial: ttyO3 at MMIO 0x4806e000 (irq = 229, base_baud = 3000000) is a OMAP UART3
[ 3.109802] [drm] Initialized drm 1.1.0 20060810
[ 3.117370] loop: module loaded
[ 3.123168] usbcore: registered new interface driver smsc95xx
[ 3.129211] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 3.136108] ehci-omap: OMAP-EHCI Host Controller driver
[ 3.159637] ehci-omap 4a064c00.ehci: EHCI Host Controller
[ 3.165283] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1
[ 3.173370] ehci-omap 4a064c00.ehci: irq 336, io mem 0x4a064c00
[ 3.199523] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00
[ 3.205902] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 3.213012] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 3.220581] usb usb1: Product: EHCI Host Controller
[ 3.225677] usb usb1: Manufacturer: Linux 4.2.3-panda ehci_hcd
[ 3.231781] usb usb1: SerialNumber: 4a064c00.ehci
[ 3.237335] hub 1-0:1.0: USB hub found
[ 3.241302] hub 1-0:1.0: 3 ports detected
[ 3.246185] usbcore: registered new interface driver cdc_wdm
[ 3.252258] usbcore: registered new interface driver usb-storage
[ 3.259002] i2c /dev entries driver
[ 3.262817] usbcore: registered new interface driver uvcvideo
[ 3.268798] USB Video Class driver (1.1.1)
[ 3.274291] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[ 3.281799] device-mapper: uevent: version 1.0.3
[ 3.286956] device-mapper: ioctl: 4.33.0-ioctl (2015-8-18) initialised: dm-devel@redhat.com
[ 3.296600] omap_hsmmc 4809c000.mmc: unable to get vmmc regulator -517
[ 3.330047] omap_hsmmc 480d5000.mmc: unable to get vmmc regulator -517
[ 3.360626] hidraw: raw HID events driver (C) Jiri Kosina
[ 3.366516] usbcore: registered new interface driver usbhid
[ 3.372375] usbhid: USB HID core driver
[ 3.376892] emif 4c000000.emif: emif_probe: device configured with addr = f00a8000 and IRQ247
[ 3.386108] emif 4d000000.emif: emif_probe: device configured with addr = f00aa000 and IRQ248
[ 3.397155] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 3.404998] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 3.412414] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 3.418029] Initializing XFRM netlink socket
[ 3.422546] NET: Registered protocol family 17
[ 3.427429] Key type dns_resolver registered
[ 3.432067] twl: not initialized
[ 3.435424] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.443939] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.452331] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.460723] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.469085] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.477478] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.485839] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1410000 Vs max 1316660
[ 3.494232] omap2_set_init_voltage: unable to find boot up OPP for vdd_mpu
[ 3.501770] omap2_set_init_voltage: unable to set vdd_mpu
[ 3.507415] omap2_set_init_voltage: unable to find boot up OPP for vdd_core
[ 3.514709] omap2_set_init_voltage: unable to set vdd_core
[ 3.520477] omap2_set_init_voltage: unable to find boot up OPP for vdd_iva
[ 3.527648] omap2_set_init_voltage: unable to set vdd_iva
[ 3.534210] Power Management for TI OMAP4+ devices.
[ 3.539306] OMAP4 PM: u-boot >= v2012.07 is required for full PM support
[ 3.546478] ThumbEE CPU extension supported.
[ 3.550994] Registering SWP/SWPB emulation handler
[ 3.550994] SmartReflex Class3 initialized
[ 3.560363] usb 1-1: new high-speed USB device number 2 using ehci-omap
[ 3.570404] Btrfs loaded
[ 3.576141] dmm 4e000000.dmm: initialized all PAT entries
[ 3.584442] Skipping twl internal clock init and using bootloader value (unknown osc rate)
[ 3.616180] twl 0-0048: PIH (irq 340) nested IRQs
[ 3.622680] twl_rtc 48070000.i2c:twl@48:rtc: Enabling TWL-RTC
[ 3.631378] twl_rtc 48070000.i2c:twl@48:rtc: rtc core: registered 48070000.i2c:twl@48 as rtc0
[ 3.651092] twl6040 0-004b: clk32k is not handled
[ 3.658630] omap_i2c 48070000.i2c: bus 0 rev0.11 at 400 kHz
[ 3.665344] omap_i2c 48072000.i2c: bus 1 rev0.11 at 400 kHz
[ 3.672210] omap_i2c 48060000.i2c: bus 2 rev0.11 at 100 kHz
[ 3.678833] omap_i2c 48350000.i2c: bus 3 rev0.11 at 400 kHz
[ 3.714263] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
[ 3.721313] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 3.729614] hub 1-1:1.0: USB hub found
[ 3.734130] hub 1-1:1.0: 5 ports detected
[ 3.891174] omap-abe-twl6040 sound: twl6040-legacy <-> 40132000.mcpdm mapping ok
[ 3.902160] twl_rtc 48070000.i2c:twl@48:rtc: setting system clock to 2000-01-01 00:11:54 UTC (946685514)
[ 3.912109] sr_init: No PMIC hook to init smartreflex
[ 3.916870] smartreflex smartreflex.0: omap_sr_probe: SmartReflex driver initialized
[ 3.926055] smartreflex smartreflex.1: omap_sr_probe: SmartReflex driver initialized
[ 3.934509] smartreflex smartreflex.2: omap_sr_probe: SmartReflex driver initialized
[ 3.951293] mmc0: host does not support reading read-only switch, assuming write-enable
[ 3.960571] VUSB: disabling
[ 3.963775] mmc0: new high speed SDHC card at address e624
[ 3.969818] VANA: disabling
[ 3.973327] VDAC: disabling
[ 3.976623] mmcblk0: mmc0:e624 SL32G 28.7 GiB
[ 3.982940] ALSA device list:
[ 3.983856] #0: PandaBoardES
[ 4.010803] mmcblk0: p1 p2 p3 p4
[ 4.020111] omap_hsmmc 480d5000.mmc: card claims to support voltages below defined range
[ 4.029174] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
[ 4.038391] EXT4-fs (mmcblk0p2): mounting ext2 file system using the ext4 subsystem
[ 4.038757] mmc1: queuing unknown CIS tuple 0x91 (3 bytes)
[ 4.039886] mmc1: new SDIO card at address 0001
[ 4.052246] usb 1-1.1: new high-speed USB device number 3 using ehci-omap
[ 4.088867] EXT4-fs (mmcblk0p2): mounted filesystem without journal. Opts: (null)
[ 4.096710] VFS: Mounted root (ext2 filesystem) readonly on device 179:2.
[ 4.108917] devtmpfs: mounted
[ 4.112060] Freeing unused kernel memory: 268K (c0893000 - c08d6000)
[ 4.171874] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[ 4.178802] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 4.189880] smsc95xx v1.0.4
[ 4.298248] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-4a064c00.ehci-1.1, smsc95xx USB 2.0 Ethernet, 56:8c:e4:0b:3c:fc
[ 4.556671] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 4.699798] random: dd urandom read with 29 bits of entropy available
[ 8.689666] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[ 18.428833] Bluetooth: Core ver 2.20
[ 18.432952] NET: Registered protocol family 31
[ 18.437622] Bluetooth: HCI device and connection manager initialized
[ 18.444335] Bluetooth: HCI socket layer initialized
[ 18.449768] Bluetooth: L2CAP socket layer initialized
[ 18.455230] Bluetooth: SCO socket layer initialized
[ 26.062011] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[ 26.073211] NFSD: starting 90-second grace period (net c090d6c0)
[ 42.884582] random: nonblocking pool is initialized
[ 43.695190] wlcore: loaded
[ 60.231079] wlcore: firmware booted (Rev 6.3.10.0.133)
[ 63.886505] wlan0: authenticate with 90:f6:52:81:ee:70
[ 63.923431] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[ 63.931488] wlan0: authenticated
[ 63.939697] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[ 63.952789] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[ 63.971496] wlan0: associated
[ 64.037994] wlcore: Association completed.
[ 146.409576] NFSD: Unable to end grace period: -110
[48067.061523] INFO: rcu_preempt self-detected stall on CPU
[48067.061523] INFO: rcu_preempt self-detected stall on CPU
[48067.061553] 0: (1 ticks this GP) idle=4a5/1/0 softirq=329263/329263 fqs=0
[48067.061553] (t=8616 jiffies g=148593 c=148592 q=154)
[48067.061553] rcu_preempt kthread starved for 8616 jiffies! g148593 c148592 f0x0
[48067.061584] Task dump for CPU 0:
[48067.061584] swapper/0 R running 0 0 0 0x00000000
[48067.061584] Backtrace:
[48067.061584] [<c0013540>] (dump_backtrace) from [<c001373c>] (show_stack+0x18/0x1c)
[48067.061645] r7:c08d8558 r6:80080193 r5:00000000 r4:c08daf68
[48067.061645] [<c0013724>] (show_stack) from [<c005cf1c>] (sched_show_task+0xbc/0x110)
[48067.061645] [<c005ce60>] (sched_show_task) from [<c005f374>] (dump_cpu_task+0x40/0x44)
[48067.061706] r5:c08e7900 r4:00000000
[48067.061706] [<c005f334>] (dump_cpu_task) from [<c0081164>] (rcu_dump_cpu_stacks+0x90/0xc8)
[48067.061737] r5:c08e7900 r4:00000001
[48067.061737] [<c00810d4>] (rcu_dump_cpu_stacks) from [<c0085148>] (rcu_check_callbacks+0x524/0x8e0)
[48067.061767] r9:c08d8558 r8:c08e7900 r7:c08e7900 r6:2e6da000 r5:c08d5100 r4:eefaf100
[48067.061767] [<c0084c24>] (rcu_check_callbacks) from [<c0087c88>] (update_process_times+0x40/0x6c)
[48067.061798] r10:c0098b18 r9:00000001 r8:eefacc4c r7:00002bb5 r6:6d57a89f r5:00000000
[48067.061798] r4:c08daf68
[48067.061798] [<c0087c48>] (update_process_times) from [<c0098b14>] (tick_sched_handle.isra.6+0x4c/0x50)
[48067.061798] r5:c08d7f18 r4:eeface40
[48067.061859] [<c0098ac8>] (tick_sched_handle.isra.6) from [<c0098b6c>] (tick_sched_timer+0x54/0x98)
[48067.061859] [<c0098b18>] (tick_sched_timer) from [<c0088ab0>] (__hrtimer_run_queues+0x110/0x1a0)
[48067.061889] r7:00000000 r6:eefacc40 r5:eeface40 r4:eefacc00
[48067.061889] [<c00889a0>] (__hrtimer_run_queues) from [<c0088d28>] (hrtimer_interrupt+0xac/0x1fc)
[48067.061920] r10:eefacc78 r9:eefacc98 r8:eefacc14 r7:eefaccb8 r6:ffffffff r5:00000003
[48067.061920] r4:eefacc00
[48067.061950] [<c0088c7c>] (hrtimer_interrupt) from [<c0016514>] (twd_handler+0x38/0x48)
[48067.061950] r10:c08d8528 r9:c0913ca5 r8:ee828700 r7:00000011 r6:c08f57c8 r5:ee805d80
[48067.061981] r4:00000001
[48067.061981] [<c00164dc>] (twd_handler) from [<c007c034>] (handle_percpu_devid_irq+0x74/0x8c)
[48067.061981] r5:ee805d80 r4:eefb1680
[48067.061981] [<c007bfc0>] (handle_percpu_devid_irq) from [<c0077e54>] (generic_handle_irq+0x34/0x44)
[48067.062042] r9:c0913ca5 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[48067.062042] [<c0077e20>] (generic_handle_irq) from [<c0078130>] (__handle_domain_irq+0x64/0xbc)
[48067.062042] r5:00000000 r4:c08d33d0
[48067.062072] [<c00780cc>] (__handle_domain_irq) from [<c000949c>] (gic_handle_irq+0x2c/0x64)
[48067.062072] r9:c0913ca5 r8:c08d7f80 r7:fa240100 r6:fa24010c r5:c08d7f18 r4:c08d87d4
[48067.062103] [<c0009470>] (gic_handle_irq) from [<c00142c0>] (__irq_svc+0x40/0x74)
[48067.062103] Exception stack(0xc08d7f18 to 0xc08d7f60)
[48067.062103] 7f00: 00000000 eefac390
[48067.062133] 7f20: 033fb4a4 c002ace4 c08d6000 c08d84dc c08d2364 c06874fc c08d7f80 c0913ca5
[48067.062133] 7f40: c08d8528 c08d7f6c c08d7f60 c08d7f60 c002aab0 c0010a30 a0080113 ffffffff
[48067.062133] r7:c08d7f4c r6:ffffffff r5:a0080113 r4:c0010a30
[48067.062164] [<c0010a08>] (arch_cpu_idle) from [<c006eadc>] (default_idle_call+0x28/0x34)
[48067.062194] [<c006eab4>] (default_idle_call) from [<c006ebfc>] (cpu_startup_entry+0x114/0x18c)
[48067.062194] [<c006eae8>] (cpu_startup_entry) from [<c067a8ac>] (rest_init+0x90/0x94)
[48067.062194] r7:ffffffff r4:00000002
[48067.062255] [<c067a81c>] (rest_init) from [<c0893cec>] (start_kernel+0x370/0x37c)
[48067.062255] r5:c0916000 r4:00000000
[48067.062255] [<c089397c>] (start_kernel) from [<8000807c>] (0x8000807c)
[48067.062286] Task dump for CPU 1:
[48067.062286] swapper/1 R running 0 0 1 0x00000002
[48067.062286] Backtrace:
[48067.062286] [<c0010a24>] (arch_cpu_idle) from [<c002aab0>] (omap_do_wfi+0x8/0x74)
[48067.062286] [<ee89ffb8>] (0xee89ffb8) from [<ffffffff>] (0xffffffff)
[48067.062347] Backtrace aborted due to bad frame pointer <c0010a30>
[48067.450897] 1: (2 ticks this GP) idle=a1f/1/0 softirq=319262/319262 fqs=39
[48067.450897] (t=8657 jiffies g=148593 c=148592 q=15)
[48067.460388] Task dump for CPU 1:
[48067.460388] swapper/1 R running 0 0 1 0x00000002
[48067.472137] Backtrace:
[48067.477691] [<c0013540>] (dump_backtrace) from [<c001373c>] (show_stack+0x18/0x1c)
[48067.477691] r7:c08d8558 r6:800c0193 r5:00000001 r4:ee877200
[48067.477691] [<c0013724>] (show_stack) from [<c005cf1c>] (sched_show_task+0xbc/0x110)
[48067.499633] [<c005ce60>] (sched_show_task) from [<c005f374>] (dump_cpu_task+0x40/0x44)
[48067.499633] r5:c08e7900 r4:00000001
[48067.499633] [<c005f334>] (dump_cpu_task) from [<c0081164>] (rcu_dump_cpu_stacks+0x90/0xc8)
[48067.520263] r5:c08e7900 r4:00000002
[48067.524017] [<c00810d4>] (rcu_dump_cpu_stacks) from [<c0085148>] (rcu_check_callbacks+0x524/0x8e0)
[48067.524017] r9:c08d8558 r8:c08e7900 r7:c08e7900 r6:2e6e5000 r5:c08d5100 r4:eefba100
[48067.524017] [<c0084c24>] (rcu_check_callbacks) from [<c0087c88>] (update_process_times+0x40/0x6c)
[48067.541503] r10:c0098b18 r9:00000001 r8:eefb7c4c r7:00002bb5 r6:6dcd7447 r5:00000000
[48067.541503] r4:ee877200
[48067.541503] [<c0087c48>] (update_process_times) from [<c0098b14>] (tick_sched_handle.isra.6+0x4c/0x50)
[48067.541503] r5:ee89ff60 r4:eefb7e40
[48067.575073] [<c0098ac8>] (tick_sched_handle.isra.6) from [<c0098b6c>] (tick_sched_timer+0x54/0x98)
[48067.575073] [<c0098b18>] (tick_sched_timer) from [<c0088ab0>] (__hrtimer_run_queues+0x110/0x1a0)
[48067.575073] r7:00000000 r6:eefb7c40 r5:eefb7e40 r4:eefb7c00
[48067.575073] [<c00889a0>] (__hrtimer_run_queues) from [<c0088d28>] (hrtimer_interrupt+0xac/0x1fc)
[48067.599548] r10:eefb7c78 r9:eefb7c98 r8:eefb7c14 r7:eefb7cb8 r6:ffffffff r5:00000002
[48067.616912] r4:eefb7c00
[48067.616912] [<c0088c7c>] (hrtimer_interrupt) from [<c0016514>] (twd_handler+0x38/0x48)
[48067.616912] r10:c08d8528 r9:c0913ca5 r8:ee828700 r7:00000011 r6:c08f57c8 r5:ee805d80
[48067.616912] r4:00000001
[48067.616912] [<c00164dc>] (twd_handler) from [<c007c034>] (handle_percpu_devid_irq+0x74/0x8c)
[48067.647491] r5:ee805d80 r4:eefbc680
[48067.651245] [<c007bfc0>] (handle_percpu_devid_irq) from [<c0077e54>] (generic_handle_irq+0x34/0x44)
[48067.651245] r9:c0913ca5 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[48067.651245] [<c0077e20>] (generic_handle_irq) from [<c0078130>] (__handle_domain_irq+0x64/0xbc)
[48067.651245] r5:00000000 r4:c08d33d0
[48067.677886] [<c00780cc>] (__handle_domain_irq) from [<c000949c>] (gic_handle_irq+0x2c/0x64)
[48067.690338] r9:c0913ca5 r8:ee89ffc8 r7:fa240100 r6:fa24010c r5:ee89ff60 r4:c08d87d4
[48067.690338] [<c0009470>] (gic_handle_irq) from [<c00142c0>] (__irq_svc+0x40/0x74)
[48067.690338] Exception stack(0xee89ff60 to 0xee89ffa8)
[48067.690338] ff60: 00000000 eefb7390 02d39a1e c002ace4 ee89e000 c08d84dc c08d2364 c06874fc
[48067.690338] ff80: ee89ffc8 c0913ca5 c08d8528 ee89ffb4 ee89ffa8 ee89ffa8 c002aab0 c0010a30
[48067.720092] ffa0: a00c0113 ffffffff
[48067.720092] r7:ee89ff94 r6:ffffffff r5:a00c0113 r4:c0010a30
[48067.732269] [<c0010a08>] (arch_cpu_idle) from [<c006eadc>] (default_idle_call+0x28/0x34)
[48067.732269] [<c006eab4>] (default_idle_call) from [<c006ebfc>] (cpu_startup_entry+0x114/0x18c)
[48067.732269] [<c006eae8>] (cpu_startup_entry) from [<c00155dc>] (secondary_start_kernel+0x160/0x16c)
[48067.732269] r7:c0916280 r4:c08ea8b0
[48067.765106] [<c001547c>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[48067.765106] r5:00000015 r4:ae88406a
[504201.421600] INFO: rcu_preempt self-detected stall on CPU
[504201.425231] INFO: rcu_preempt detected stalls on CPUs/tasks:
[504201.425231] 0: (1 ticks this GP) idle=841/1/0 softirq=3416105/3416105 fqs=0
[504201.425262]
[504201.425262] (detected by 1, t=6081 jiffies, g=1560600, c=1560599, q=116)
[504201.425262] Task dump for CPU 0:
[504201.425262] swapper/0 R
[504201.425262] running
[504201.425262] 0 0 0 0x00000000
[504201.425292] Backtrace:
[504201.425292] [<c0010a24>] (arch_cpu_idle) from [<c002aab0>] (omap_do_wfi+0x8/0x74)
[504201.425292] [<c08d7f70>] (0xc08d7f70) from [<ffffffff>] (0xffffffff)
[504201.425353] Backtrace aborted due to bad frame pointer <c0010a30>
[504201.425353] rcu_preempt kthread starved for 6081 jiffies! g1560600 c1560599 f0x0
[504201.489471] 0: (1 ticks this GP) idle=841/1/0 softirq=3416105/3416105 fqs=1
[504201.497375] (t=6081 jiffies g=1560600 c=1560599 q=116)
[504201.497375] Task dump for CPU 0:
[504201.497375] swapper/0 R running 0 0 0 0x00000000
[504201.513427] Backtrace:
[504201.513427] [<c0013540>] (dump_backtrace) from [<c001373c>] (show_stack+0x18/0x1c)
[504201.525329] r7:c08d8558 r6:800c0193 r5:00000000 r4:c08daf68
[504201.525329] [<c0013724>] (show_stack) from [<c005cf1c>] (sched_show_task+0xbc/0x110)
[504201.533416] [<c005ce60>] (sched_show_task) from [<c005f374>] (dump_cpu_task+0x40/0x44)
[504201.543426] r5:c08e7900 r4:00000000
[504201.543426] [<c005f334>] (dump_cpu_task) from [<c0081164>] (rcu_dump_cpu_stacks+0x90/0xc8)
[504201.560485] r5:c08e7900 r4:00000001
[504201.564331] [<c00810d4>] (rcu_dump_cpu_stacks) from [<c0085148>] (rcu_check_callbacks+0x524/0x8e0)
[504201.573425] r9:c08d8558 r8:c08e7900 r7:c08e7900 r6:2e6da000 r5:c08d5100 r4:eefaf100
[504201.580474] [<c0084c24>] (rcu_check_callbacks) from [<c0087c88>] (update_process_times+0x40/0x6c)
[504201.586364] r10:c0098b18 r9:00000001 r8:eefacc4c r7:0001ca84 r6:1012c513 r5:00000000
[504201.595520] r4:c08daf68
[504201.595520] [<c0087c48>] (update_process_times) from [<c0098b14>] (tick_sched_handle.isra.6+0x4c/0x50)
[504201.612182] r5:c08d7f18 r4:eeface40
[504201.614685] [<c0098ac8>] (tick_sched_handle.isra.6) from [<c0098b6c>] (tick_sched_timer+0x54/0x98)
[504201.616027] [<c0098b18>] (tick_sched_timer) from [<c0088ab0>] (__hrtimer_run_queues+0x110/0x1a0)
[504201.634735] r7:00000000 r6:eefacc40 r5:eeface40 r4:eefacc00
[504201.634735] [<c00889a0>] (__hrtimer_run_queues) from [<c0088d28>] (hrtimer_interrupt+0xac/0x1fc)
[504201.650024] r10:eefacc78 r9:eefacc98 r8:eefacc14 r7:eefaccb8 r6:ffffffff r5:00000003
[504201.656829] r4:eefacc00
[504201.656829] [<c0088c7c>] (hrtimer_interrupt) from [<c0016514>] (twd_handler+0x38/0x48)
[504201.669403] r10:c08d8528 r9:c0913ca5 r8:ee828700 r7:00000011 r6:c08f57c8 r5:ee805d80
[504201.673431] r4:00000001
[504201.673431] [<c00164dc>] (twd_handler) from [<c007c034>] (handle_percpu_devid_irq+0x74/0x8c)
[504201.673431] r5:ee805d80 r4:eefb1680
[504201.673431] [<c007bfc0>] (handle_percpu_devid_irq) from [<c0077e54>] (generic_handle_irq+0x34/0x44)
[504201.693450] r9:c0913ca5 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[504201.693450] [<c0077e20>] (generic_handle_irq) from [<c0078130>] (__handle_domain_irq+0x64/0xbc)
[504201.693450] r5:00000000 r4:c08d33d0
[504201.723937] [<c00780cc>] (__handle_domain_irq) from [<c000949c>] (gic_handle_irq+0x2c/0x64)
[504201.723937] r9:c0913ca5 r8:c08d7f80 r7:fa240100 r6:fa24010c r5:c08d7f18 r4:c08d87d4
[504201.723937] [<c0009470>] (gic_handle_irq) from [<c00142c0>] (__irq_svc+0x40/0x74)
[504201.723937] Exception stack(0xc08d7f18 to 0xc08d7f60)
[504201.752441] 7f00: 00000000 eefac390
[504201.762847] 7f20: 21f9b840 c002ace4 c08d6000 c08d84dc c08d2364 c06874fc c08d7f80 c0913ca5
[504201.764038] 7f40: c08d8528 c08d7f6c c08d7f60 c08d7f60 c002aab0 c0010a30 a00c0113 ffffffff
[504201.764038] r7:c08d7f4c r6:ffffffff r5:a00c0113 r4:c0010a30
[504201.783447] [<c0010a08>] (arch_cpu_idle) from [<c006eadc>] (default_idle_call+0x28/0x34)
[504201.783447] [<c006eab4>] (default_idle_call) from [<c006ebfc>] (cpu_startup_entry+0x114/0x18c)
[504201.803802] [<c006eae8>] (cpu_startup_entry) from [<c067a8ac>] (rest_init+0x90/0x94)
[504201.803802] r7:ffffffff r4:00000002
[504201.815826] [<c067a81c>] (rest_init) from [<c0893cec>] (start_kernel+0x370/0x37c)
[504201.815826] r5:c0916000 r4:00000000
[504201.823730] [<c089397c>] (start_kernel) from [<8000807c>] (0x8000807c)
$
[-- Attachment #3: dmesg-4.1 --]
[-- Type: application/octet-stream, Size: 140639 bytes --]
Welcome to minicom 2.6.1
OPTIONS: I18n
Port /dev/ttyUSB0
Press CTRL-A Z for help on special keys
#
#
#
# uptime
19:10:13 up 12 days, 19:45, load average: 0.10, 0.12, 0.13
#
#
#
#
#
#
# cat /proc/version
Linux version 4.1.0-panda (rfg@red.dwarf) (gcc version 5.1.0 (Buildroot 2015.08-git-00291-gf2aabda) ) #1 SMP PREEMPT Tue Jun 23 25
# dmesg
[102358.482238] r7:ee89ffa4 r6:ffffffff r5:a0080113 r4:c00109e0
[102358.494079] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[102358.505157] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[102358.510406] r7:c08f0280 r4:c08c6898
[102358.518524] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[102358.526336] r5:00000015 r4:ae88406a
[206047.594329] INFO: rcu_preempt self-detected stall on CPU { 0} (t=23615 jiffies g=645561 c=645560 q=107)
[206047.600372] INFO: rcu_preempt detected stalls on CPUs/tasks: { 0} (detected by 1, t=23615 jiffies, g=645561, c=645560, q=107)
[206047.600402] Task dump for CPU 0:
[206047.600402] swapper/0 R running 0 0 0 0x00000000
[206047.600402] Backtrace:
[206047.600433] [<c08b3f70>] (0xc08b3f70) from [<c08b44e8>] (0xc08b44e8)
[206047.600433] Backtrace aborted due to bad frame pointer <c08b3f80>
[206047.642211] Task dump for CPU 0:
[206047.642211] swapper/0 R running 0 0 0 0x00000002
[206047.652404] Backtrace:
[206047.652404] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[206047.663055] r7:c08b4518 r6:800c0193 r5:00000000 r4:c08b6ee8
[206047.669097] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[206047.671661] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[206047.685638] r5:c08c3b00 r4:00000000
[206047.685638] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[206047.685638] r5:c08c3b00 r4:00000001
[206047.700378] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[206047.710388] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e6fe000 r5:c08b1980 r4:eefaf980
[206047.710388] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[206047.729034] r10:00000000 r9:eefad438 r8:eefad518 r7:0000bb66 r6:89daba20 r5:00000000
[206047.737335] r4:c08b6ee8
[206047.737335] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[206047.740386] r5:c08b3f28 r4:eefad690
[206047.750396] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[206047.760375] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[206047.772247] r7:c00978d0 r6:eefad400 r5:eefad438 r4:eefad690
[206047.772247] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[206047.787628] r7:00000001 r6:eefad400 r5:0000bb66 r4:89daba20
[206047.787628] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[206047.796569] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[206047.810302] r4:00000001
[206047.810394] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[206047.810394] r5:ee803d80 r4:eefb1f00
[206047.810394] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[206047.830413] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[206047.840393] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[206047.850372] r5:00000000 r4:c08afc18
[206047.850372] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[206047.860382] r9:c08ed664 r8:c08b3f80 r7:fa240100 r6:c08b3f28 r5:c08b478c r4:fa24010c
[206047.870391] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[206047.880371] Exception stack(0xc08b3f28 to 0xc08b3f70)
[206047.880371] 3f20: 00000000 00000000 00000000 c002ad20 c08b2000 c08b449c
[206047.890380] 3f40: c08ae344 c066b7c8 c08b3f80 c08ed664 c08b44e8 c08b3f7c c08b3f70 c08b3f70
[206047.890380] 3f60: c002aaec c00109e0 a00c0113 ffffffff
[206047.909423] r7:c08b3f5c r6:ffffffff r5:a00c0113 r4:c00109e0
[206047.910369] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[206047.920379] [<c006e028>] (cpu_startup_entry) from [<c065e9bc>] (rest_init+0x90/0x94)
[206047.932373] r7:ffffffff r4:00000002
[206047.932373] [<c065e92c>] (rest_init) from [<c086ecd4>] (start_kernel+0x368/0x374)
[206047.940368] r5:c08f0000 r4:00000000
[206047.940368] [<c086e96c>] (start_kernel) from [<8000807c>] (0x8000807c)
[220770.783630] INFO: rcu_preempt self-detected stall on CPU
[220770.786773] INFO: rcu_preempt self-detected stall on CPU
[220770.786773] {
[220770.786773] 1
[220770.786773] }
[220770.786804] (t=13820 jiffies g=691256 c=691255 q=193)
[220770.786804] rcu_preempt kthread starved for 13820 jiffies!
[220770.786804] Task dump for CPU 0:
[220770.786804] swapper/0 R
[220770.786804] running
[220770.786804] 0 0 0 0x00000000
[220770.786804] Backtrace:
[220770.786834] [<c08b3f70>] (0xc08b3f70) from [<c08b44e8>] (0xc08b44e8)
[220770.786834] Backtrace aborted due to bad frame pointer <c08b3f80>
[220770.786834] Task dump for CPU 1:
[220770.786865] swapper/1 R
[220770.786865] running
[220770.786865] 0 0 1 0x00000002
[220770.786865] Backtrace:
[220770.786895] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[220770.786895] r7:c08b4518
[220770.786895] r6:800f0193
[220770.786895] r5:00000001
[220770.786926] r4:ee877200
[220770.786926] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[220770.786956] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[220770.786956] r5:c08c3b00
[220770.786956] r4:00000001
[220770.786987] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[220770.786987] r5:c08c3b00
[220770.786987] r4:00000002
[220770.787017] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[220770.787017] r9:c08b4518
[220770.787017] r8:c08c3b00
[220770.787017] r7:c08c3b00
[220770.787017] r6:2e709000
[220770.787017] r5:c08b1980
[220770.787017] r4:eefba980
[220770.787048] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[220770.787048] r10:00000000
[220770.787048] r9:eefb8438
[220770.787048] r8:eefb8518
[220770.787048] r7:0000c8ca
[220770.787078] r6:8cc90cbd
[220770.787078] r5:00000000
[220770.787078] r4:ee877200
[220770.787109] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[220770.787109] r5:ee89ff70
[220770.787109] r4:eefb8690
[220770.787109] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[220770.787139] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[220770.787139] r7:c00978d0
[220770.787139] r6:eefb8400
[220770.787139] r5:eefb8438
[220770.787139] r4:eefb8690
[220770.787170] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[220770.787170] r7:00000001
[220770.787170] r6:eefb8400
[220770.787170] r5:0000c8ca
[220770.787170] r4:8cc89587
[220770.787200] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[220770.787200] r10:c08b44e8
[220770.787200] r9:c08ed664
[220770.787200] r8:ee828940
[220770.787200] r7:00000011
[220770.787231] r6:c08d0208
[220770.787231] r5:ee803d80
[220770.787231] r4:00000001
[220770.787231] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[220770.787261] r5:ee803d80
[220770.787261] r4:eefbcf00
[220770.787261] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[220770.787261] r9:c08ed664
[220770.787292] r8:ee808000
[220770.787292] r7:00000001
[220770.787292] r6:00000000
[220770.787292] r5:00000000
[220770.787292] r4:00000011
[220770.787292] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[220770.787322] r5:00000000
[220770.787322] r4:c08afc18
[220770.787322] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[220770.787322] r9:c08ed664
[220770.787322] r8:ee89ffc8
[220770.787353] r7:fa240100
[220770.787353] r6:ee89ff70
[220770.787353] r5:c08b478c
[220770.787353] r4:fa24010c
[220770.787353] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[220770.787353] Exception stack(0xee89ff70 to 0xee89ffb8)
[220770.787384] ff60: 00000000 00000000 00000000 c002ad20
[220770.787384] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[220770.787384] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a00f0113 ffffffff
[220770.787414] r7:ee89ffa4
[220770.787414] r6:ffffffff
[220770.787414] r5:a00f0113
[220770.787414] r4:c00109e0
[220770.787445] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[220770.787445] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[220770.787445] r7:c08f0280
[220770.787475] r4:c08c6898
[220770.787475] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[220770.787475] r5:00000015
[220770.787475] r4:ae88406a
[220771.125854] {
[220771.125854] 0} (t=13820 jiffies g=691256 c=691255 q=193)
[220771.145141] rcu_preempt kthread starved for 13820 jiffies!
[220771.145843] Task dump for CPU 0:
[220771.145843] swapper/0 R running 0 0 0 0x00000000
[220771.161132] Backtrace:
[220771.161132] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[220771.165863] r7:c08b4518 r6:800c0193 r5:00000000 r4:c08b6ee8
[220771.175842] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[220771.185852] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[220771.194366] r5:c08c3b00 r4:00000000
[220771.197326] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[220771.197326] r5:c08c3b00 r4:00000001
[220771.210754] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[220771.210754] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e6fe000 r5:c08b1980 r4:eefaf980
[220771.225860] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[220771.237762] r10:00000000 r9:eefad438 r8:eefad518 r7:0000c8ca r6:8c9827c5 r5:00000000
[220771.237762] r4:c08b6ee8
[220771.237762] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[220771.257751] r5:c08b3f28 r4:eefad690
[220771.258605] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[220771.265869] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[220771.275848] r7:c00978d0 r6:eefad400 r5:eefad438 r4:eefad690
[220771.286987] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[220771.286987] r7:00000001 r6:eefad400 r5:0000c8ca r4:8c9827c5
[220771.302368] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[220771.302368] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[220771.315856] r4:00000001
[220771.321746] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[220771.325866] r5:ee803d80 r4:eefb1f00
[220771.325866] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[220771.344024] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[220771.352233] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[220771.357666] r5:00000000 r4:c08afc18
[220771.365234] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[220771.372497] r9:c08ed664 r8:c08b3f80 r7:fa240100 r6:c08b3f28 r5:c08b478c r4:fa24010c
[220771.379180] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[220771.385864] Exception stack(0xc08b3f28 to 0xc08b3f70)
[220771.395507] 3f20: 00000000 00000000 00000000 c002ad20 c08b2000 c08b449c
[220771.404144] 3f40: c08ae344 c066b7c8 c08b3f80 c08ed664 c08b44e8 c08b3f7c c08b3f70 c08b3f70
[220771.412780] 3f60: c002aaec c00109e0 a00c0113 ffffffff
[220771.417541] r7:c08b3f5c r6:ffffffff r5:a00c0113 r4:c00109e0
[220771.424163] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[220771.424163] [<c006e028>] (cpu_startup_entry) from [<c065e9bc>] (rest_init+0x90/0x94)
[220771.435852] r7:ffffffff r4:00000002
[220771.435852] [<c065e92c>] (rest_init) from [<c086ecd4>] (start_kernel+0x368/0x374)
[220771.435852] r5:c08f0000 r4:00000000
[220771.455841] [<c086e96c>] (start_kernel) from [<8000807c>] (0x8000807c)
[283214.963958] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 0, t=17730 jiffies, g=886402, c=886401, q=198)
[283214.975708] All QSes seen, last rcu_preempt kthread activity 17730 (28291637-28273907), jiffies_till_next_fqs=1, root ->qsmas0
[283214.987976] swapper/0 R running 0 0 0 0x00000000
[283214.994445] Backtrace:
[283214.997314] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[283215.005065] r7:c08c3b00 r6:2e6fe000 r5:00000000 r4:c08b6ee8
[283215.011047] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[283215.018859] [<c005b590>] (sched_show_task) from [<c00840c4>] (rcu_check_callbacks+0x8dc/0x8f4)
[283215.028778] r5:c08b1980 r4:eefaf980
[283215.032623] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[283215.041717] r10:00000000 r9:eefad438 r8:eefad518 r7:00010195 r6:72f09e00 r5:00000000
[283215.049530] r4:c08b6ee8
[283215.052764] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[283215.062805] r5:c08b3f28 r4:eefad690
[283215.066528] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[283215.075744] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[283215.084655] r7:c00978d0 r6:eefad400 r5:eefad438 r4:eefad690
[283215.090942] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[283215.099639] r7:00000001 r6:eefad400 r5:00010195 r4:72f09e00
[283215.106597] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[283215.114044] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[283215.122467] r4:00000001
[283215.125915] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[283215.134887] r5:ee803d80 r4:eefb1f00
[283215.138732] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[283215.147491] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[283215.156463] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[283215.165588] r5:00000000 r4:c08afc18
[283215.168487] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[283215.168487] r9:c08ed664 r8:c08b3f80 r7:fa240100 r6:c08b3f28 r5:c08b478c r4:fa24010c
[283215.186492] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[283215.194396] Exception stack(0xc08b3f28 to 0xc08b3f70)
[283215.197113] 3f20: 00000000 00000000 00000000 c002ad20 c08b2000 c08b449c
[283215.199829] 3f40: c08ae344 c066b7c8 c08b3f80 c08ed664 c08b44e8 c08b3f7c c08b3f70 c08b3f70
[283215.199829] 3f60: c002aaec c00109e0 a00c0113 ffffffff
[283215.217071] r7:c08b3f5c r6:ffffffff r5:a00c0113 r4:c00109e0
[283215.227111] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[283215.237152] [<c006e028>] (cpu_startup_entry) from [<c065e9bc>] (rest_init+0x90/0x94)
[283215.239868] r7:ffffffff r4:00000002
[283215.249176] [<c065e92c>] (rest_init) from [<c086ecd4>] (start_kernel+0x368/0x374)
[283215.254333] r5:c08f0000 r4:00000000
[283215.259857] [<c086e96c>] (start_kernel) from [<8000807c>] (0x8000807c)
[283215.259857] rcu_preempt kthread starved for 17759 jiffies!
[298409.007598] INFO: rcu_preempt self-detected stall on CPU { 1} (t=0 jiffies g=934032 c=934031 q=116)
[298409.017272] Task dump for CPU 0:
[298409.017272] swapper/0 R running 0 0 0 0x00000000
[298409.021270] Backtrace:
[298409.021270] [<c08b3f70>] (0xc08b3f70) from [<c08b44e8>] (0xc08b44e8)
[298409.036865] Backtrace aborted due to bad frame pointer <c08b3f80>
[298409.036865] Task dump for CPU 1:
[298409.036865] swapper/1 R running 0 0 1 0x00000000
[298409.053497] Backtrace:
[298409.053497] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[298409.053497] r7:c08b4518 r6:800c0193 r5:00000001 r4:ee877200
[298409.070190] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[298409.070190] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[298409.086730] r5:c08c3b00 r4:00000001
[298409.086730] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[298409.086730] r5:c08c3b00 r4:00000002
[298409.099304] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[298409.112579] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e709000 r5:c08b1980 r4:eefba980
[298409.112579] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[298409.112579] r10:00000000 r9:eefb8438 r8:eefb8518 r7:00010f67 r6:114c8603 r5:00000000
[298409.138427] r4:ee877200
[298409.138427] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[298409.138427] r5:ee89ff70 r4:eefb8690
[298409.154815] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[298409.164276] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[298409.164276] r7:c00978d0 r6:eefb8400 r5:eefb8438 r4:eefb8690
[298409.179382] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[298409.179382] r7:00000001 r6:eefb8400 r5:00010f67 r4:114c8603
[298409.179382] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[298409.194763] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[298409.211395] r4:00000001
[298409.211395] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[298409.223052] r5:ee803d80 r4:eefbcf00
[298409.223052] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[298409.236419] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[298409.236419] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[298409.236419] r5:00000000 r4:c08afc18
[298409.257629] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[298409.257629] r9:c08ed664 r8:ee89ffc8 r7:fa240100 r6:ee89ff70 r5:c08b478c r4:fa24010c
[298409.257629] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[298409.257629] Exception stack(0xee89ff70 to 0xee89ffb8)
[298409.287902] ff60: 00000000 00000000 00000000 c002ad20
[298409.296539] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[298409.296539] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a00c0113 ffffffff
[298409.296539] r7:ee89ffa4 r6:ffffffff r5:a00c0113 r4:c00109e0
[298409.296539] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[298409.296539] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[298409.296539] r7:c08f0280 r4:c08c6898
[298409.296539] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[298409.296539] r5:00000015 r4:ae88406a
[383033.393798] INFO: rcu_preempt self-detected stall on CPU
[383033.399108] INFO: rcu_preempt self-detected stall on CPU
[383033.399108] {
[383033.399108] 1
[383033.399108] }
[383033.399139] (t=23577 jiffies g=1199203 c=1199202 q=196)
[383033.399139] rcu_preempt kthread starved for 23577 jiffies!
[383033.399139] Task dump for CPU 0:
[383033.399139] swapper/0 R
[383033.399139] running
[383033.399139] 0 0 0 0x00000000
[383033.399169] Backtrace:
[383033.399169] [<c08b3f70>] (0xc08b3f70) from [<c08b44e8>] (0xc08b44e8)
[383033.399169] Backtrace aborted due to bad frame pointer <c08b3f80>
[383033.399200] Task dump for CPU 1:
[383033.399200] swapper/1 R
[383033.399200] running
[383033.399200] 0 0 1 0x00000002
[383033.399200] Backtrace:
[383033.399230] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[383033.399230] r7:c08b4518
[383033.399230] r6:800f0193
[383033.399230] r5:00000001
[383033.399230] r4:ee877200
[383033.399261] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[383033.399291] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[383033.399291] r5:c08c3b00
[383033.399291] r4:00000001
[383033.399291] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[383033.399291] r5:c08c3b00
[383033.399291] r4:00000002
[383033.399353] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[383033.399353] r9:c08b4518
[383033.399353] r8:c08c3b00
[383033.399353] r7:c08c3b00
[383033.399353] r6:2e709000
[383033.399383] r5:c08b1980
[383033.399383] r4:eefba980
[383033.399383] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[383033.399383] r10:00000000
[383033.399383] r9:eefb8438
[383033.399414] r8:eefb8518
[383033.399414] r7:00015c5e
[383033.399414] r6:0b1f9ee9
[383033.399414] r5:00000000
[383033.399414] r4:ee877200
[383033.399444] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[383033.399444] r5:ee89ff70
[383033.399444] r4:eefb8690
[383033.399444] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[383033.399475] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[383033.399475] r7:c00978d0
[383033.399475] r6:eefb8400
[383033.399475] r5:eefb8438
[383033.399475] r4:eefb8690
[383033.399505] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[383033.399505] r7:00000001
[383033.399505] r6:eefb8400
[383033.399505] r5:00015c5e
[383033.399505] r4:0b1f9ee9
[383033.399536] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[383033.399536] r10:c08b44e8
[383033.399536] r9:c08ed664
[383033.399536] r8:ee828940
[383033.399536] r7:00000011
[383033.399566] r6:c08d0208
[383033.399566] r5:ee803d80
[383033.399566] r4:00000001
[383033.399566] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[383033.399566] r5:ee803d80
[383033.399566] r4:eefbcf00
[383033.399566] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[383033.399627] r9:c08ed664
[383033.399627] r8:ee808000
[383033.399627] r7:00000001
[383033.399627] r6:00000000
[383033.399627] r5:00000000
[383033.399627] r4:00000011
[383033.399658] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[383033.399658] r5:00000000
[383033.399658] r4:c08afc18
[383033.399658] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[383033.399688] r9:c08ed664
[383033.399688] r8:ee89ffc8
[383033.399688] r7:fa240100
[383033.399688] r6:ee89ff70
[383033.399688] r5:c08b478c
[383033.399688] r4:fa24010c
[383033.399688] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[383033.399719] Exception stack(0xee89ff70 to 0xee89ffb8)
[383033.399719] ff60: 00000000 00000000 00000000 c002ad20
[383033.399719] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[383033.399749] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a00f0113 ffffffff
[383033.399749] r7:ee89ffa4
[383033.399749] r6:ffffffff
[383033.399749] r5:a00f0113
[383033.399749] r4:c00109e0
[383033.399780] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[383033.399780] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[383033.399810] r7:c08f0280
[383033.399810] r4:c08c6898
[383033.399810] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[383033.399810] r5:00000015
[383033.399810] r4:ae88406a
[383033.749328] {
[383033.749328] 0} (t=23577 jiffies g=1199203 c=1199202 q=196)
[383033.751159] rcu_preempt kthread starved for 23577 jiffies!
[383033.759124] Task dump for CPU 0:
[383033.764984] swapper/0 R running 0 0 0 0x00000000
[383033.764984] Backtrace:
[383033.773223] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[383033.779113] r7:c08b4518 r6:80010193 r5:00000000 r4:c08b6ee8
[383033.785247] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[383033.796600] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[383033.801605] r5:c08c3b00 r4:00000000
[383033.801605] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[383033.809112] r5:c08c3b00 r4:00000001
[383033.821350] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[383033.829101] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e6fe000 r5:c08b1980 r4:eefaf980
[383033.835327] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[383033.839111] r10:00000000 r9:eefad438 r8:eefad518 r7:00015c5e r6:0acdaa30 r5:00000000
[383033.849121] r4:c08b6ee8
[383033.859100] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[383033.869110] r5:c08b3f28 r4:eefad690
[383033.869110] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[383033.879119] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[383033.879119] r7:c00978d0 r6:eefad400 r5:eefad438 r4:eefad690
[383033.897583] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[383033.906921] r7:00000001 r6:eefad400 r5:00015c5e r4:0acdaa30
[383033.909118] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[383033.909118] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[383033.929595] r4:00000001
[383033.931274] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[383033.931274] r5:ee803d80 r4:eefb1f00
[383033.941253] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[383033.953491] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[383033.954620] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[383033.969879] r5:00000000 r4:c08afc18
[383033.969879] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[383033.969879] r9:c08ed664 r8:c08b3f80 r7:fa240100 r6:c08b3f28 r5:c08b478c r4:fa24010c
[383033.969879] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[383034.000732] Exception stack(0xc08b3f28 to 0xc08b3f70)
[383034.006103] 3f20: 00000000 00000000 00000000 c002ad20 c08b2000 c08b449c
[383034.014739] 3f40: c08ae344 c066b7c8 c08b3f80 c08ed664 c08b44e8 c08b3f7c c08b3f70 c08b3f70
[383034.014739] 3f60: c002aaec c00109e0 a0010113 ffffffff
[383034.023468] r7:c08b3f5c r6:ffffffff r5:a0010113 r4:c00109e0
[383034.034759] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[383034.043457] [<c006e028>] (cpu_startup_entry) from [<c065e9bc>] (rest_init+0x90/0x94)
[383034.043457] r7:ffffffff r4:00000002
[383034.051666] [<c065e92c>] (rest_init) from [<c086ecd4>] (start_kernel+0x368/0x374)
[383034.059600] r5:c08f0000 r4:00000000
[383034.067230] [<c086e96c>] (start_kernel) from [<8000807c>] (0x8000807c)
[419159.023437] INFO: rcu_preempt self-detected stall on CPU
[419159.023468] INFO: rcu_preempt self-detected stall on CPU
[419159.023468] {
[419159.023468] 0
[419159.023468] }
[419159.023468] (t=2477 jiffies g=1312846 c=1312845 q=163)
[419159.023498] rcu_preempt kthread starved for 2477 jiffies!
[419159.023498] Task dump for CPU 0:
[419159.023498] swapper/0 R
[419159.023498] running
[419159.023498] 0 0 0 0x00000000
[419159.023498] Backtrace:
[419159.023559] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[419159.023559] r7:c08b4518
[419159.023559] r6:800c0193
[419159.023559] r5:00000000
[419159.023559] r4:c08b6ee8
[419159.023590] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[419159.023590] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[419159.023620] r5:c08c3b00
[419159.023620] r4:00000000
[419159.023620] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[419159.023651] r5:c08c3b00
[419159.023651] r4:00000001
[419159.023651] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[419159.023651] r9:c08b4518
[419159.023681] r8:c08c3b00
[419159.023681] r7:c08c3b00
[419159.023681] r6:2e6fe000
[419159.023681] r5:c08b1980
[419159.023681] r4:eefaf980
[419159.023712] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[419159.023712] r10:00000000
[419159.023712] r9:eefad438
[419159.023712] r8:eefad518
[419159.023712] r7:00017d39
[419159.023712] r6:2cd7a7ce
[419159.023712] r5:00000000
[419159.023712] r4:c08b6ee8
[419159.023742] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[419159.023742] r5:c08b3f28
[419159.023742] r4:eefad690
[419159.023773] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[419159.023773] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[419159.023773] r7:c00978d0
[419159.023803] r6:eefad400
[419159.023803] r5:eefad438
[419159.023803] r4:eefad690
[419159.023803] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[419159.023803] r7:00000001
[419159.023803] r6:eefad400
[419159.023834] r5:00017d39
[419159.023834] r4:2cd7a7ce
[419159.023834] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[419159.023834] r10:c08b44e8
[419159.023864] r9:c08ed664
[419159.023864] r8:ee828940
[419159.023864] r7:00000011
[419159.023864] r6:c08d0208
[419159.023864] r5:ee803d80
[419159.023864] r4:00000001
[419159.023895] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[419159.023895] r5:ee803d80
[419159.023895] r4:eefb1f00
[419159.023925] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[419159.023925] r9:c08ed664
[419159.023925] r8:ee808000
[419159.023925] r7:00000001
[419159.023925] r6:00000000
[419159.023925] r5:00000000
[419159.023925] r4:00000011
[419159.023956] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[419159.023956] r5:00000000
[419159.023956] r4:c08afc18
[419159.023986] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[419159.023986] r9:c08ed664
[419159.023986] r8:c08b3f80
[419159.023986] r7:fa240100
[419159.023986] r6:c08b3f28
[419159.023986] r5:c08b478c
[419159.023986] r4:fa24010c
[419159.024017] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[419159.024017] Exception stack(0xc08b3f28 to 0xc08b3f70)
[419159.024017] 3f20: 00000000 00000000 00000000 c002ad20 c08b2000 c08b449c
[419159.024047] 3f40: c08ae344 c066b7c8 c08b3f80 c08ed664 c08b44e8 c08b3f7c c08b3f70 c08b3f70
[419159.024047] 3f60: c002aaec c00109e0 a00c0113 ffffffff
[419159.024047] r7:c08b3f5c
[419159.024047] r6:ffffffff
[419159.024047] r5:a00c0113
[419159.024047] r4:c00109e0
[419159.024047] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[419159.024108] [<c006e028>] (cpu_startup_entry) from [<c065e9bc>] (rest_init+0x90/0x94)
[419159.024108] r7:ffffffff
[419159.024108] r4:00000002
[419159.024139] [<c065e92c>] (rest_init) from [<c086ecd4>] (start_kernel+0x368/0x374)
[419159.024139] r5:c08f0000
[419159.024139] r4:00000000
[419159.024139] [<c086e96c>] (start_kernel) from [<8000807c>] (0x8000807c)
[419159.024139] Task dump for CPU 1:
[419159.024169] swapper/1 R
[419159.024169] running
[419159.024169] 0 0 1 0x00000000
[419159.024169] Backtrace:
[419159.024169] [<ee89ffb8>] (0xee89ffb8) from [<c08b44e8>] (0xc08b44e8)
[419159.024200] [<c08c688c>] (0xc08c688c) from [<c00109c4>] (arch_cpu_idle+0xc/0x44)
[419159.024200] Backtrace aborted due to bad frame pointer <ee89ffc8>
[419159.227844] {
[419159.392364] 1} (t=2498 jiffies g=1312846 c=1312845 q=106)
[419159.392364] rcu_preempt kthread starved for 2498 jiffies!
[419159.398590] Task dump for CPU 1:
[419159.398590] swapper/1 R running 0 0 1 0x00000002
[419159.407836] Backtrace:
[419159.407836] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[419159.415435] r7:c08b4518 r6:80080193 r5:00000001 r4:ee877200
[419159.427795] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[419159.427795] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[419159.437805] r5:c08c3b00 r4:00000001
[419159.447814] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[419159.458557] r5:c08c3b00 r4:00000002
[419159.458557] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[419159.471832] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e709000 r5:c08b1980 r4:eefba980
[419159.480041] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[419159.487823] r10:00000000 r9:eefb8438 r8:eefb8518 r7:00017d39 r6:2cd7a7ce r5:00000000
[419159.497680] r4:ee877200
[419159.500427] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[419159.500427] r5:ee89ff70 r4:eefb8690
[419159.514068] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[419159.517822] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[419159.517822] r7:c00978d0 r6:eefb8400 r5:eefb8438 r4:eefb8690
[419159.538635] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[419159.547973] r7:00000001 r6:eefb8400 r5:00017d39 r4:2cd7a7ce
[419159.554016] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[419159.558624] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[419159.568634] r4:00000001
[419159.568634] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[419159.582305] r5:ee803d80 r4:eefbcf00
[419159.582305] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[419159.595672] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[419159.602081] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[419159.602081] r5:00000000 r4:c08afc18
[419159.613555] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[419159.613555] r9:c08ed664 r8:ee89ffc8 r7:fa240100 r6:ee89ff70 r5:c08b478c r4:fa24010c
[419159.629302] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[419159.637817] Exception stack(0xee89ff70 to 0xee89ffb8)
[419159.637817] ff60: 00000000 00000000 00000000 c002ad20
[419159.647827] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[419159.647827] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a0080113 ffffffff
[419159.671417] r7:ee89ffa4 r6:ffffffff r5:a0080113 r4:c00109e0
[419159.671417] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[419159.671417] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[419159.692199] r7:c08f0280 r4:c08c6898
[419159.699554] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[419159.706573] r5:00000015 r4:ae88406a
[562461.509155] INFO: rcu_preempt self-detected stall on CPU { 1} (t=6120 jiffies g=1762892 c=1762891 q=197)
[562461.515380] INFO: rcu_preempt detected stalls on CPUs/tasks: { 1} (detected by 0, t=6120 jiffies, g=1762892, c=1762891, q=197)
[562461.515380] Task dump for CPU 1:
[562461.515380] swapper/1 R running 0 0 1 0x00000000
[562461.515380] Backtrace:
[562461.515411] [<ee89ffb8>] (0xee89ffb8) from [<c08b44e8>] (0xc08b44e8)
[562461.515441] [<c08c688c>] (0xc08c688c) from [<c00109c4>] (arch_cpu_idle+0xc/0x44)
[562461.515441] Backtrace aborted due to bad frame pointer <ee89ffc8>
[562461.564941] Task dump for CPU 1:
[562461.567901] swapper/1 R running 0 0 1 0x00000002
[562461.574493] Backtrace:
[562461.577850] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[562461.585906] r7:c08b4518 r6:800c0193 r5:00000001 r4:ee877200
[562461.591918] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[562461.600067] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[562461.608093] r5:c08c3b00 r4:00000001
[562461.612335] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[562461.620880] r5:c08c3b00 r4:00000002
[562461.624053] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[562461.634338] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e709000 r5:c08b1980 r4:eefba980
[562461.642547] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[562461.651763] r10:00000000 r9:eefb8438 r8:eefb8518 r7:0001ff8e r6:344d5632 r5:00000000
[562461.659576] r4:ee877200
[562461.662231] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[562461.672760] r5:ee89ff70 r4:eefb8690
[562461.675537] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[562461.685394] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[562461.695129] r7:c00978d0 r6:eefb8400 r5:eefb8438 r4:eefb8690
[562461.700317] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[562461.709960] r7:00000001 r6:eefb8400 r5:0001ff8e r4:344d5632
[562461.710784] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[562461.721954] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[562461.733184] r4:00000001
[562461.733184] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[562461.733184] r5:ee803d80 r4:eefbcf00
[562461.748657] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[562461.748657] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[562461.760345] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[562461.775573] r5:00000000 r4:c08afc18
[562461.775573] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[562461.788208] r9:c08ed664 r8:ee89ffc8 r7:fa240100 r6:ee89ff70 r5:c08b478c r4:fa24010c
[562461.796417] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[562461.796417] Exception stack(0xee89ff70 to 0xee89ffb8)
[562461.809692] ff60: 00000000 00000000 00000000 c002ad20
[562461.814514] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[562461.818328] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a00c0113 ffffffff
[562461.830688] r7:ee89ffa4 r6:ffffffff r5:a00c0113 r4:c00109e0
[562461.836853] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[562461.836853] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[562461.854522] r7:c08f0280 r4:c08c6898
[562461.854522] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[562461.864501] r5:00000015 r4:ae88406a
[574093.122222] INFO: rcu_preempt self-detected stall on CPU
[574093.122222] INFO: rcu_preempt self-detected stall on CPU
[574093.122222] {
[574093.122253] 1
[574093.122253] }
[574093.122253] (t=2988 jiffies g=1799308 c=1799307 q=196)
[574093.122253] rcu_preempt kthread starved for 2988 jiffies!
[574093.122253] Task dump for CPU 0:
[574093.122253] swapper/0 R
[574093.122253] running
[574093.122283] 0 0 0 0x00000000
[574093.122283] Backtrace:
[574093.122283] [<c08b3f70>] (0xc08b3f70) from [<c08b44e8>] (0xc08b44e8)
[574093.122314] Backtrace aborted due to bad frame pointer <c08b3f80>
[574093.122314] Task dump for CPU 1:
[574093.122314] swapper/1 R
[574093.122314] running
[574093.122314] 0 0 1 0x00000000
[574093.122314] Backtrace:
[574093.122314] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[574093.122314] r7:c08b4518
[574093.122314] r6:80080193
[574093.122375] r5:00000001
[574093.122375] r4:ee877200
[574093.122375] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[574093.122406] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[574093.122406] r5:c08c3b00
[574093.122406] r4:00000001
[574093.122436] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[574093.122436] r5:c08c3b00
[574093.122436] r4:00000002
[574093.122467] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[574093.122467] r9:c08b4518
[574093.122467] r8:c08c3b00
[574093.122467] r7:c08c3b00
[574093.122467] r6:2e709000
[574093.122467] r5:c08b1980
[574093.122497] r4:eefba980
[574093.122497] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[574093.122497] r10:00000000
[574093.122497] r9:eefb8438
[574093.122497] r8:eefb8518
[574093.122528] r7:00020a22
[574093.122528] r6:620ab67f
[574093.122528] r5:00000000
[574093.122528] r4:ee877200
[574093.122558] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[574093.122558] r5:ee89ff70
[574093.122558] r4:eefb8690
[574093.122558] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[574093.122589] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[574093.122589] r7:c00978d0
[574093.122589] r6:eefb8400
[574093.122589] r5:eefb8438
[574093.122589] r4:eefb8690
[574093.122619] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[574093.122619] r7:00000001
[574093.122619] r6:eefb8400
[574093.122619] r5:00020a22
[574093.122619] r4:620ab67f
[574093.122650] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[574093.122650] r10:c08b44e8
[574093.122650] r9:c08ed664
[574093.122650] r8:ee828940
[574093.122650] r7:00000011
[574093.122650] r6:c08d0208
[574093.122650] r5:ee803d80
[574093.122680] r4:00000001
[574093.122680] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[574093.122680] r5:ee803d80
[574093.122711] r4:eefbcf00
[574093.122711] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[574093.122711] r9:c08ed664
[574093.122711] r8:ee808000
[574093.122741] r7:00000001
[574093.122741] r6:00000000
[574093.122741] r5:00000000
[574093.122741] r4:00000011
[574093.122741] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[574093.122741] r5:00000000
[574093.122772] r4:c08afc18
[574093.122772] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[574093.122772] r9:c08ed664
[574093.122772] r8:ee89ffc8
[574093.122772] r7:fa240100
[574093.122772] r6:ee89ff70
[574093.122802] r5:c08b478c
[574093.122802] r4:fa24010c
[574093.122802] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[574093.122802] Exception stack(0xee89ff70 to 0xee89ffb8)
[574093.122833] ff60: 00000000 00000000 00000000 c002ad20
[574093.122833] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[574093.122833] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a0080113 ffffffff
[574093.122833] r7:ee89ffa4
[574093.122833] r6:ffffffff
[574093.122863] r5:a0080113
[574093.122863] r4:c00109e0
[574093.122863] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[574093.122863] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[574093.122863] r7:c08f0280
[574093.122863] r4:c08c6898
[574093.122924] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[574093.122924] r5:00000015
[574093.122924] r4:ae88406a
[574093.477539] {
[574093.477539] 0} (t=3023 jiffies g=1799308 c=1799307 q=153)
[574093.483825] rcu_preempt kthread starved for 3024 jiffies!
[574093.483825] Task dump for CPU 0:
[574093.489562] swapper/0 R running 0 0 0 0x00000000
[574093.494262] Backtrace:
[574093.499755] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[574093.503448] r7:c08b4518 r6:800c0193 r5:00000000 r4:c08b6ee8
[574093.503448] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[574093.524627] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[574093.532867] r5:c08c3b00 r4:00000000
[574093.532867] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[574093.544647] r5:c08c3b00 r4:00000001
[574093.544647] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[574093.558807] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e6fe000 r5:c08b1980 r4:eefaf980
[574093.567016] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[574093.568756] r10:00000000 r9:eefad438 r8:eefad518 r7:00020a22 r6:620ab67f r5:00000000
[574093.584655] r4:c08b6ee8
[574093.584655] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[574093.588745] r5:c08b3f28 r4:eefad690
[574093.598754] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[574093.610504] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[574093.618743] r7:c00978d0 r6:eefad400 r5:eefad438 r4:eefad690
[574093.625610] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[574093.628753] r7:00000001 r6:eefad400 r5:00020a22 r4:620ab67f
[574093.640991] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[574093.640991] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[574093.656524] r4:00000001
[574093.660369] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[574093.660980] r5:ee803d80 r4:eefb1f00
[574093.669281] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[574093.682647] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[574093.688751] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[574093.694274] r5:00000000 r4:c08afc18
[574093.702178] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[574093.702178] r9:c08ed664 r8:c08b3f80 r7:fa240100 r6:c08b3f28 r5:c08b478c r4:fa24010c
[574093.715332] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[574093.728759] Exception stack(0xc08b3f28 to 0xc08b3f70)
[574093.734130] 3f20: 00000000 00000000 00000000 c002ad20 c08b2000 c08b449c
[574093.740234] 3f40: c08ae344 c066b7c8 c08b3f80 c08ed664 c08b44e8 c08b3f7c c08b3f70 c08b3f70
[574093.740234] 3f60: c002aaec c00109e0 a00c0113 ffffffff
[574093.751403] r7:c08b3f5c r6:ffffffff r5:a00c0113 r4:c00109e0
[574093.762786] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[574093.771514] [<c006e028>] (cpu_startup_entry) from [<c065e9bc>] (rest_init+0x90/0x94)
[574093.779693] r7:ffffffff r4:00000002
[574093.779693] [<c065e92c>] (rest_init) from [<c086ecd4>] (start_kernel+0x368/0x374)
[574093.788757] r5:c08f0000 r4:00000000
[574093.791442] [<c086e96c>] (start_kernel) from [<8000807c>] (0x8000807c)
[581984.917449] INFO: rcu_preempt self-detected stall on CPU
[581984.918426] INFO: rcu_preempt self-detected stall on CPU
[581984.918426] {
[581984.918426] 0
[581984.918426] }
[581984.918457] (t=13954 jiffies g=1823613 c=1823612 q=11)
[581984.918457] rcu_preempt kthread starved for 13954 jiffies!
[581984.918457] Task dump for CPU 0:
[581984.918457] thttpd R
[581984.918457] running
[581984.918457] 0 30612 1150 0x00000000
[581984.918457] Backtrace:
[581984.918487] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[581984.918487] r7:c08b4518
[581984.918487] r6:800d0193
[581984.918487] r5:0000047e
[581984.918487] r4:eeabf200
[581984.918548] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[581984.918548] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[581984.918579] r5:c08c3b00
[581984.918579] r4:00000000
[581984.918579] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[581984.918609] r5:c08c3b00
[581984.918609] r4:00000001
[581984.918609] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[581984.918640] r9:c08b4518
[581984.918640] r8:c08c3b00
[581984.918640] r7:c08c3b00
[581984.918640] r6:2e6fe000
[581984.918640] r5:c08b1980
[581984.918640] r4:eefaf980
[581984.918670] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[581984.918670] r10:00000000
[581984.918670] r9:eefad438
[581984.918670] r8:eefad518
[581984.918670] r7:0002114f
[581984.918670] r6:d36b2f36
[581984.918670] r5:00000001
[581984.918701] r4:eeabf200
[581984.918701] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[581984.918701] r5:ee94dfb0
[581984.918701] r4:eefad690
[581984.918731] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[581984.918731] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[581984.918762] r7:c00978d0
[581984.918762] r6:eefad400
[581984.918762] r5:eefad438
[581984.918762] r4:eefad690
[581984.918762] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[581984.918762] r7:00000001
[581984.918792] r6:eefad400
[581984.918792] r5:0002114f
[581984.918792] r4:d3695260
[581984.918792] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[581984.918823] r10:0001ba38
[581984.918823] r9:0002e2c4
[581984.918823] r8:ee828940
[581984.918823] r7:00000011
[581984.918823] r6:c08d0208
[581984.918823] r5:ee803d80
[581984.918823] r4:00000001
[581984.918823] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[581984.918823] r5:ee803d80
[581984.918823] r4:eefb1f00
[581984.918884] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[581984.918884] r9:0002e2c4
[581984.918884] r8:ee808000
[581984.918884] r7:00000001
[581984.918884] r6:00000000
[581984.918884] r5:00000000
[581984.918914] r4:00000011
[581984.918914] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[581984.918914] r5:00000000
[581984.918914] r4:c08afc18
[581984.918945] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[581984.918945] r9:0002e2c4
[581984.918945] r8:10c5387d
[581984.918945] r7:fa240100
[581984.918945] r6:ee94dfb0
[581984.918945] r5:c08b478c
[581984.918975] r4:fa24010c
[581984.918975] [<c000947c>] (gic_handle_irq) from [<c0014528>] (__irq_usr+0x48/0x60)
[581984.918975] Exception stack(0xee94dfb0 to 0xee94dff8)
[581984.918975] dfa0: b6d67008 be9e7ed8 55923f64 00000ff6
[581984.919006] dfc0: b6d82ae8 0001bb40 0000049e 0002e41c be9e7ed8 0002e2c4 0001ba38 431bde83
[581984.919006] dfe0: 0002d55c be9e7d20 000194dd 000142a2 200d0130 ffffffff
[581984.919006] r7:10c5387d
[581984.919006] r6:ffffffff
[581984.919006] r5:200d0130
[581984.919036] r4:000142a2
[581984.919036] Task dump for CPU 1:
[581984.919036] swapper/1 R
[581984.919036] running
[581984.919036] 0 0 1 0x00000000
[581984.919036] Backtrace:
[581984.919067] [<ee89ffb8>] (0xee89ffb8) from [<c08b44e8>] (0xc08b44e8)
[581984.919067] [<c08c688c>] (0xc08c688c) from [<c00109c4>] (arch_cpu_idle+0xc/0x44)
[581984.919067] Backtrace aborted due to bad frame pointer <ee89ffc8>
[581985.227905] {
[581985.247924] 1} (t=13954 jiffies g=1823613 c=1823612 q=11)
[581985.247924] rcu_preempt kthread starved for 13954 jiffies!
[581985.253265] Task dump for CPU 1:
[581985.253265] swapper/1 R running 0 0 1 0x00000000
[581985.267913] Backtrace:
[581985.267913] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[581985.271942] r7:c08b4518 r6:800c0193 r5:00000001 r4:ee877200
[581985.283874] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[581985.287902] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[581985.297912] r5:c08c3b00 r4:00000001
[581985.306335] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[581985.309478] r5:c08c3b00 r4:00000002
[581985.309478] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[581985.328338] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e709000 r5:c08b1980 r4:eefba980
[581985.336547] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[581985.344909] r10:00000000 r9:eefb8438 r8:eefb8518 r7:0002114f r6:d35b5a18 r5:00000000
[581985.352874] r4:ee877200
[581985.356933] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[581985.358886] r5:ee89ff70 r4:eefb8690
[581985.370574] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[581985.380035] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[581985.389099] r7:c00978d0 r6:eefb8400 r5:eefb8438 r4:eefb8690
[581985.392120] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[581985.397918] r7:00000001 r6:eefb8400 r5:0002114f r4:d35b5a18
[581985.397918] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[581985.418853] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[581985.418853] r4:00000001
[581985.427917] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[581985.427917] r5:ee803d80 r4:eefbcf00
[581985.442626] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[581985.447906] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[581985.453186] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[581985.467926] r5:00000000 r4:c08afc18
[581985.467926] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[581985.482177] r9:c08ed664 r8:ee89ffc8 r7:fa240100 r6:ee89ff70 r5:c08b478c r4:fa24010c
[581985.487915] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[581985.498291] Exception stack(0xee89ff70 to 0xee89ffb8)
[581985.498291] ff60: 00000000 00000000 00000000 c002ad20
[581985.507904] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[581985.520904] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a00c0113 ffffffff
[581985.520904] r7:ee89ffa4 r6:ffffffff r5:a00c0113 r4:c00109e0
[581985.527923] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[581985.542663] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[581985.542663] r7:c08f0280 r4:c08c6898
[581985.553161] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[581985.562652] r5:00000015 r4:ae88406a
[588364.430389] INFO: rcu_preempt self-detected stall on CPU
[588364.433654] INFO: rcu_preempt self-detected stall on CPU
[588364.433654] {
[588364.433654] 0
[588364.433654] }
[588364.433685] (t=5200 jiffies g=1843491 c=1843490 q=197)
[588364.433685] rcu_preempt kthread starved for 5200 jiffies!
[588364.433685] Task dump for CPU 0:
[588364.433685] swapper/0 R
[588364.433685] running
[588364.433685] 0 0 0 0x00000002
[588364.433715] Backtrace:
[588364.433715] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[588364.433715] r7:c08b4518
[588364.433715] r6:800f0193
[588364.433715] r5:00000000
[588364.433715] r4:c08b6ee8
[588364.433715] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[588364.433807] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[588364.433807] r5:c08c3b00
[588364.433807] r4:00000000
[588364.433837] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[588364.433837] r5:c08c3b00
[588364.433837] r4:00000001
[588364.433868] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[588364.433868] r9:c08b4518
[588364.433868] r8:c08c3b00
[588364.433868] r7:c08c3b00
[588364.433868] r6:2e6fe000
[588364.433868] r5:c08b1980
[588364.433868] r4:eefaf980
[588364.433898] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[588364.433898] r10:00000000
[588364.433898] r9:eefad438
[588364.433898] r8:eefad518
[588364.433898] r7:0002171d
[588364.433898] r6:2b40379a
[588364.433929] r5:00000000
[588364.433929] r4:c08b6ee8
[588364.433929] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[588364.433929] r5:c08b3f28
[588364.433959] r4:eefad690
[588364.433959] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[588364.433990] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[588364.433990] r7:c00978d0
[588364.433990] r6:eefad400
[588364.433990] r5:eefad438
[588364.433990] r4:eefad690
[588364.433990] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[588364.434020] r7:00000001
[588364.434020] r6:eefad400
[588364.434020] r5:0002171d
[588364.434020] r4:2b3de38f
[588364.434051] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[588364.434051] r10:c08b44e8
[588364.434051] r9:c08ed664
[588364.434051] r8:ee828940
[588364.434051] r7:00000011
[588364.434051] r6:c08d0208
[588364.434051] r5:ee803d80
[588364.434051] r4:00000001
[588364.434082] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[588364.434082] r5:ee803d80
[588364.434082] r4:eefb1f00
[588364.434112] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[588364.434112] r9:c08ed664
[588364.434112] r8:ee808000
[588364.434112] r7:00000001
[588364.434112] r6:00000000
[588364.434143] r5:00000000
[588364.434143] r4:00000011
[588364.434143] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[588364.434143] r5:00000000
[588364.434143] r4:c08afc18
[588364.434173] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[588364.434173] r9:c08ed664
[588364.434173] r8:c08b3f80
[588364.434173] r7:fa240100
[588364.434173] r6:c08b3f28
[588364.434204] r5:c08b478c
[588364.434204] r4:fa24010c
[588364.434204] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[588364.434204] Exception stack(0xc08b3f28 to 0xc08b3f70)
[588364.434234] 3f20: 00000000 00000000 00000000 c002ad20 c08b2000 c08b449c
[588364.434234] 3f40: c08ae344 c066b7c8 c08b3f80 c08ed664 c08b44e8 c08b3f7c c08b3f70 c08b3f70
[588364.434234] 3f60: c002aaec c00109e0 a00f0113 ffffffff
[588364.434234] r7:c08b3f5c
[588364.434234] r6:ffffffff
[588364.434265] r5:a00f0113
[588364.434265] r4:c00109e0
[588364.434265] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[588364.434295] [<c006e028>] (cpu_startup_entry) from [<c065e9bc>] (rest_init+0x90/0x94)
[588364.434295] r7:ffffffff
[588364.434295] r4:00000002
[588364.434326] [<c065e92c>] (rest_init) from [<c086ecd4>] (start_kernel+0x368/0x374)
[588364.434326] r5:c08f0000
[588364.434326] r4:00000000
[588364.434356] [<c086e96c>] (start_kernel) from [<8000807c>] (0x8000807c)
[588364.434356] Task dump for CPU 1:
[588364.434356] swapper/1 R
[588364.434356] running
[588364.434356] 0 0 1 0x00000000
[588364.434356] Backtrace:
[588364.434387] [<ee89ffb8>] (0xee89ffb8) from [<c08b44e8>] (0xc08b44e8)
[588364.434387] [<c08c688c>] (0xc08c688c) from [<c00109c4>] (arch_cpu_idle+0xc/0x44)
[588364.434387] Backtrace aborted due to bad frame pointer <ee89ffc8>
[588364.797515] {
[588364.798950] 1} (t=5200 jiffies g=1843491 c=1843490 q=197)
[588364.803131] rcu_preempt kthread starved for 5200 jiffies!
[588364.809539] Task dump for CPU 1:
[588364.809539] swapper/1 R running 0 0 1 0x00000000
[588364.819732] Backtrace:
[588364.819732] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[588364.830383] r7:c08b4518 r6:80080193 r5:00000001 r4:ee877200
[588364.836425] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[588364.843139] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[588364.852966] r5:c08c3b00 r4:00000001
[588364.856811] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[588364.856811] r5:c08c3b00 r4:00000002
[588364.869354] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[588364.873138] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e709000 r5:c08b1980 r4:eefba980
[588364.873138] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[588364.896362] r10:00000000 r9:eefb8438 r8:eefb8518 r7:0002171d r6:2b0ded14 r5:00000000
[588364.896881] r4:ee877200
[588364.896881] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[588364.917205] r5:ee89ff70 r4:eefb8690
[588364.919372] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[588364.930480] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[588364.933135] r7:c00978d0 r6:eefb8400 r5:eefb8438 r4:eefb8690
[588364.945587] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[588364.946685] r7:00000001 r6:eefb8400 r5:0002171d r4:2b0ded14
[588364.960968] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[588364.967102] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[588364.969329] r4:00000001
[588364.969329] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[588364.987945] r5:ee803d80 r4:eefbcf00
[588364.987945] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[588365.002624] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[588365.003143] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[588365.003143] r5:00000000 r4:c08afc18
[588365.023590] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[588365.032653] r9:c08ed664 r8:ee89ffc8 r7:fa240100 r6:ee89ff70 r5:c08b478c r4:fa24010c
[588365.040863] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[588365.040863] Exception stack(0xee89ff70 to 0xee89ffb8)
[588365.048767] ff60: 00000000 00000000 00000000 c002ad20
[588365.061981] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[588365.071380] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a0080113 ffffffff
[588365.078369] r7:ee89ffa4 r6:ffffffff r5:a0080113 r4:c00109e0
[588365.084411] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[588365.084411] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[588365.093139] r7:c08f0280 r4:c08c6898
[588365.093139] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[588365.093139] r5:00000015 r4:ae88406a
[642940.609374] Clock: inserting leap second 23:59:60 UTC
[646597.228698] INFO: rcu_preempt self-detected stall on CPU
[646597.228729] INFO: rcu_preempt self-detected stall on CPU
[646597.228729] {
[646597.228729] 0
[646597.228729] }
[646597.228759] (t=3842 jiffies g=2026704 c=2026703 q=196)
[646597.228759] rcu_preempt kthread starved for 3842 jiffies!
[646597.228759] Task dump for CPU 0:
[646597.228759] swapper/0 R
[646597.228759] running
[646597.228759] 0 0 0 0x00000000
[646597.228790] Backtrace:
[646597.228790] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[646597.228790] r7:c08b4518
[646597.228790] r6:800c0193
[646597.228790] r5:00000000
[646597.228790] r4:c08b6ee8
[646597.228851] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[646597.228881] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[646597.228881] r5:c08c3b00
[646597.228881] r4:00000000
[646597.228881] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[646597.228881] r5:c08c3b00
[646597.228881] r4:00000001
[646597.228881] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[646597.228942] r9:c08b4518
[646597.228942] r8:c08c3b00
[646597.228942] r7:c08c3b00
[646597.228942] r6:2e6fe000
[646597.228942] r5:c08b1980
[646597.228942] r4:eefaf980
[646597.228973] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[646597.228973] r10:00000000
[646597.228973] r9:eefad438
[646597.228973] r8:eefad518
[646597.228973] r7:00024c13
[646597.228973] r6:7e6fb8cb
[646597.229003] r5:00000000
[646597.229003] r4:c08b6ee8
[646597.229003] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[646597.229003] r5:c08b3f28
[646597.229034] r4:eefad690
[646597.229034] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[646597.229064] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[646597.229064] r7:c00978d0
[646597.229064] r6:eefad400
[646597.229064] r5:eefad438
[646597.229064] r4:eefad690
[646597.229064] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[646597.229095] r7:00000001
[646597.229095] r6:eefad400
[646597.229095] r5:00024c13
[646597.229095] r4:7e6fb8cb
[646597.229125] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[646597.229125] r10:c08b44e8
[646597.229125] r9:c08ed664
[646597.229125] r8:ee828940
[646597.229125] r7:00000011
[646597.229125] r6:c08d0208
[646597.229125] r5:ee803d80
[646597.229125] r4:00000001
[646597.229156] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[646597.229156] r5:ee803d80
[646597.229156] r4:eefb1f00
[646597.229187] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[646597.229187] r9:c08ed664
[646597.229187] r8:ee808000
[646597.229187] r7:00000001
[646597.229187] r6:00000000
[646597.229217] r5:00000000
[646597.229217] r4:00000011
[646597.229217] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[646597.229217] r5:00000000
[646597.229217] r4:c08afc18
[646597.229248] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[646597.229248] r9:c08ed664
[646597.229248] r8:c08b3f80
[646597.229248] r7:fa240100
[646597.229248] r6:c08b3f28
[646597.229278] r5:c08b478c
[646597.229278] r4:fa24010c
[646597.229278] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[646597.229278] Exception stack(0xc08b3f28 to 0xc08b3f70)
[646597.229309] 3f20: 00000000 00000000 00000000 c002ad20 c08b2000 c08b449c
[646597.229309] 3f40: c08ae344 c066b7c8 c08b3f80 c08ed664 c08b44e8 c08b3f7c c08b3f70 c08b3f70
[646597.229309] 3f60: c002aaec c00109e0 a00c0113 ffffffff
[646597.229309] r7:c08b3f5c
[646597.229309] r6:ffffffff
[646597.229339] r5:a00c0113
[646597.229339] r4:c00109e0
[646597.229339] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[646597.229370] [<c006e028>] (cpu_startup_entry) from [<c065e9bc>] (rest_init+0x90/0x94)
[646597.229370] r7:ffffffff
[646597.229370] r4:00000002
[646597.229400] [<c065e92c>] (rest_init) from [<c086ecd4>] (start_kernel+0x368/0x374)
[646597.229400] r5:c08f0000
[646597.229400] r4:00000000
[646597.229431] [<c086e96c>] (start_kernel) from [<8000807c>] (0x8000807c)
[646597.229431] Task dump for CPU 1:
[646597.229431] swapper/1 R
[646597.229431] running
[646597.229431] 0 0 1 0x00000000
[646597.229431] Backtrace:
[646597.229461] [<ee89ffb8>] (0xee89ffb8) from [<c08b44e8>] (0xc08b44e8)
[646597.229461] [<c08c688c>] (0xc08c688c) from [<c00109c4>] (arch_cpu_idle+0xc/0x44)
[646597.229461] Backtrace aborted due to bad frame pointer <ee89ffc8>
[646597.595825] {
[646597.597656] 1} (t=3879 jiffies g=2026704 c=2026703 q=151)
[646597.597656] rcu_preempt kthread starved for 3879 jiffies!
[646597.605957] Task dump for CPU 1:
[646597.605957] swapper/1 R running 0 0 1 0x00000002
[646597.614257] Backtrace:
[646597.614257] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[646597.625976] r7:c08b4518 r6:80010193 r5:00000001 r4:ee877200
[646597.634735] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[646597.642913] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[646597.651275] r5:c08c3b00 r4:00000001
[646597.651275] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[646597.663116] r5:c08c3b00 r4:00000002
[646597.665954] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[646597.668792] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e709000 r5:c08b1980 r4:eefba980
[646597.668792] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[646597.694671] r10:00000000 r9:eefb8438 r8:eefb8518 r7:00024c13 r6:7e6fb8cb r5:00000000
[646597.697235] r4:ee877200
[646597.697235] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[646597.714691] r5:ee89ff70 r4:eefb8690
[646597.714691] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[646597.728790] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[646597.735961] r7:c00978d0 r6:eefb8400 r5:eefb8438 r4:eefb8690
[646597.737884] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[646597.737884] r7:00000001 r6:eefb8400 r5:00024c13 r4:7e6fb8cb
[646597.759277] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[646597.765960] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[646597.769836] r4:00000001
[646597.775970] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[646597.787567] r5:ee803d80 r4:eefbcf00
[646597.787567] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[646597.800933] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[646597.809143] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[646597.817810] r5:00000000 r4:c08afc18
[646597.822143] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[646597.829772] r9:c08ed664 r8:ee89ffc8 r7:fa240100 r6:ee89ff70 r5:c08b478c r4:fa24010c
[646597.837005] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[646597.845977] Exception stack(0xee89ff70 to 0xee89ffb8)
[646597.845977] ff60: 00000000 00000000 00000000 c002ad20
[646597.855987] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[646597.869689] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a0010113 ffffffff
[646597.876678] r7:ee89ffa4 r6:ffffffff r5:a0010113 r4:c00109e0
[646597.882232] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[646597.882232] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[646597.900970] r7:c08f0280 r4:c08c6898
[646597.904815] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[646597.911376] r5:00000015 r4:ae88406a
[670816.378265] INFO: rcu_preempt self-detected stall on CPU
[670816.380279] INFO: rcu_preempt detected stalls on CPUs/tasks:
[670816.380279] {
[670816.380279] 1
[670816.380310] }
[670816.380310] (detected by 0, t=18832 jiffies, g=2102385, c=2102384, q=153)
[670816.380310] Task dump for CPU 1:
[670816.380310] swapper/1 R
[670816.380310] running
[670816.380310] 0 0 1 0x00000000
[670816.380340] Backtrace:
[670816.380340] [<ee89ffb8>] (0xee89ffb8) from [<c08b44e8>] (0xc08b44e8)
[670816.380371] [<c08c688c>] (0xc08c688c) from [<c00109c4>] (arch_cpu_idle+0xc/0x44)
[670816.380371] Backtrace aborted due to bad frame pointer <ee89ffc8>
[670816.380371] rcu_preempt kthread starved for 18832 jiffies!
[670816.435424] {
[670816.436737] 1} (t=18832 jiffies g=2102385 c=2102384 q=153)
[670816.441436] rcu_preempt kthread starved for 18832 jiffies!
[670816.447418] Task dump for CPU 1:
[670816.450286] swapper/1 R running 0 0 1 0x00000000
[670816.457122] Backtrace:
[670816.459442] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[670816.468170] r7:c08b4518 r6:80080193 r5:00000001 r4:ee877200
[670816.474182] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[670816.481842] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[670816.491058] r5:c08c3b00 r4:00000001
[670816.494689] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[670816.503417] r5:c08c3b00 r4:00000002
[670816.506866] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[670816.516906] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e709000 r5:c08b1980 r4:eefba980
[670816.524932] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[670816.534393] r10:00000000 r9:eefb8438 r8:eefb8518 r7:0002621a r6:79f6e168 r5:00000000
[670816.542358] r4:ee877200
[670816.545501] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[670816.554534] r5:ee89ff70 r4:eefb8690
[670816.559143] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[670816.567840] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[670816.577606] r7:c00978d0 r6:eefb8400 r5:eefb8438 r4:eefb8690
[670816.583099] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[670816.589904] r7:00000001 r6:eefb8400 r5:0002621a r4:79f6e168
[670816.599090] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[670816.607452] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[670816.607452] r4:00000001
[670816.618011] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[670816.627380] r5:ee803d80 r4:eefbcf00
[670816.629913] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[670816.629913] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[670816.648620] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[670816.658142] r5:00000000 r4:c08afc18
[670816.659912] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[670816.670776] r9:c08ed664 r8:ee89ffc8 r7:fa240100 r6:ee89ff70 r5:c08b478c r4:fa24010c
[670816.670776] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[670816.670776] Exception stack(0xee89ff70 to 0xee89ffb8)
[670816.670776] ff60: 00000000 00000000 00000000 c002ad20
[670816.698150] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[670816.701660] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a0080113 ffffffff
[670816.710388] r7:ee89ffa4 r6:ffffffff r5:a0080113 r4:c00109e0
[670816.719909] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[670816.731262] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[670816.739898] r7:c08f0280 r4:c08c6898
[670816.739898] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[670816.749908] r5:00000015 r4:ae88406a
[735732.505676] INFO: rcu_preempt self-detected stall on CPU
[735732.505706] INFO: rcu_preempt detected stalls on CPUs/tasks:
[735732.505706] {
[735732.505706] 0
[735732.505737] }
[735732.505737] (detected by 1, t=2338 jiffies, g=2306148, c=2306147, q=24)
[735732.505737] Task dump for CPU 0:
[735732.505737] swapper/0 R
[735732.505737] running
[735732.505737] 0 0 0 0x00000000
[735732.505767] Backtrace:
[735732.505767] [<c08b3f70>] (0xc08b3f70) from [<c08b44e8>] (0xc08b44e8)
[735732.505767] Backtrace aborted due to bad frame pointer <c08b3f80>
[735732.505798] rcu_preempt kthread starved for 2337 jiffies!
[735732.531768] {
[735732.531768] 0} (t=0 jiffies g=2306149 c=2306148 q=25)
[735732.531768] Task dump for CPU 0:
[735732.562896] swapper/0 R running 0 0 0 0x00000002
[735732.564147] Backtrace:
[735732.573547] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[735732.580963] r7:c08b4518 r6:80080193 r5:00000000 r4:c08b6ee8
[735732.581542] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[735732.589569] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[735732.589569] r5:c08c3b00 r4:00000000
[735732.589569] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[735732.616668] r5:c08c3b00 r4:00000001
[735732.619567] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[735732.619567] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e6fe000 r5:c08b1980 r4:eefaf980
[735732.619567] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[735732.642425] r10:00000000 r9:eefad438 r8:eefad518 r7:00029d24 r6:f625939a r5:00000000
[735732.652404] r4:c08b6ee8
[735732.652404] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[735732.659576] r5:c08b3f28 r4:eefad690
[735732.669555] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[735732.669555] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[735732.689575] r7:c00978d0 r6:eefad400 r5:eefad438 r4:eefad690
[735732.689575] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[735732.699554] r7:00000001 r6:eefad400 r5:00029d24 r4:f625939a
[735732.712127] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[735732.720489] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[735732.726989] r4:00000001
[735732.726989] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[735732.740417] r5:ee803d80 r4:eefb1f00
[735732.744262] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[735732.753784] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[735732.754913] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[735732.769561] r5:00000000 r4:c08afc18
[735732.774902] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[735732.779571] r9:c08ed664 r8:c08b3f80 r7:fa240100 r6:c08b3f28 r5:c08b478c r4:fa24010c
[735732.779571] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[735732.779571] Exception stack(0xc08b3f28 to 0xc08b3f70)
[735732.779571] 3f20: 00000000 00000000 00000000 c002ad20 c08b2000 c08b449c
[735732.809600] 3f40: c08ae344 c066b7c8 c08b3f80 c08ed664 c08b44e8 c08b3f7c c08b3f70 c08b3f70
[735732.822540] 3f60: c002aaec c00109e0 a0080113 ffffffff
[735732.827911] r7:c08b3f5c r6:ffffffff r5:a0080113 r4:c00109e0
[735732.833953] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[735732.833953] [<c006e028>] (cpu_startup_entry) from [<c065e9bc>] (rest_init+0x90/0x94)
[735732.833953] r7:ffffffff r4:00000002
[735732.852386] [<c065e92c>] (rest_init) from [<c086ecd4>] (start_kernel+0x368/0x374)
[735732.859558] r5:c08f0000 r4:00000000
[735732.866424] [<c086e96c>] (start_kernel) from [<8000807c>] (0x8000807c)
[770462.031738] wlcore: Beacon loss detected. roles:0xff
[770463.669952] wlcore: Beacon loss detected. roles:0xff
[770465.310150] wlcore: Beacon loss detected. roles:0xff
[770466.837738] wlcore: Connection loss work (role_id: 0).
[770466.843231] wlan0: Connection to AP 90:f6:52:81:ee:70 lost
[770466.946655] wlcore: Beacon loss detected. roles:0xff
[770466.998962] cfg80211: Exceeded CRDA call max attempts. Not calling CRDA
[833088.720825] INFO: rcu_preempt self-detected stall on CPU { 1} (t=2656 jiffies g=2653594 c=2653593 q=198)
[833088.730957] rcu_preempt kthread starved for 2656 jiffies!
[833088.730957] Task dump for CPU 1:
[833088.740142] swapper/1 R running 0 0 1 0x00000002
[833088.746887] Backtrace:
[833088.746887] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[833088.757537] r7:c08b4518 r6:80010193 r5:00000001 r4:ee877200
[833088.763580] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[833088.763580] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[833088.780120] r5:c08c3b00 r4:00000001
[833088.780120] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[833088.780120] r5:c08c3b00 r4:00000002
[833088.780120] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[833088.805969] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e709000 r5:c08b1980 r4:eefba980
[833088.805969] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[833088.805969] r10:00000000 r9:eefb8438 r8:eefb8518 r7:0002f5b0 r6:3ffcc765 r5:00000000
[833088.831817] r4:ee877200
[833088.831817] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[833088.831817] r5:ee89ff70 r4:eefb8690
[833088.848205] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[833088.848205] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[833088.857666] r7:c00978d0 r6:eefb8400 r5:eefb8438 r4:eefb8690
[833088.857666] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[833088.882110] r7:00000001 r6:eefb8400 r5:0002f5b0 r4:3ffcc765
[833088.882110] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[833088.896484] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[833088.904785] r4:00000001
[833088.904785] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[833088.904785] r5:ee803d80 r4:eefbcf00
[833088.904785] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[833088.904785] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[833088.929809] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[833088.947174] r5:00000000 r4:c08afc18
[833088.947174] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[833088.947174] r9:c08ed664 r8:ee89ffc8 r7:fa240100 r6:ee89ff70 r5:c08b478c r4:fa24010c
[833088.947174] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[833088.947174] Exception stack(0xee89ff70 to 0xee89ffb8)
[833088.981292] ff60: 00000000 00000000 00000000 c002ad20
[833088.981292] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[833088.998565] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a0010113 ffffffff
[833088.998565] r7:ee89ffa4 r6:ffffffff r5:a0010113 r4:c00109e0
[833088.998565] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[833089.011596] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[833089.029846] r7:c08f0280 r4:c08c6898
[833089.029846] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[833089.033691] r5:00000015 r4:ae88406a
[833089.045318] INFO: rcu_preempt detected stalls on CPUs/tasks: { 0 1} (detected by 0, t=0 jiffies, g=2653595, c=2653594, q=239)
[833089.053070] Task dump for CPU 0:
[833089.060760] swapper/0 R running 0 0 0 0x00000002
[833089.060882] Backtrace:
[833089.070159] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[833089.070159] r7:c08b4518 r6:80000193 r5:00000000 r4:c08b6ee8
[833089.070159] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[833089.092376] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[833089.092376] r5:c08c3b00 r4:00000000
[833089.092376] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[833089.113281] r5:c08c3b00 r4:00000001
[833089.113281] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0084078>] (rcu_check_callbacks+0x890/0x8f4)
[833089.113281] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e6fe000 r5:c08b1980 r4:eefaf980
[833089.113281] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[833089.144134] r10:00000000 r9:eefad438 r8:eefad518 r7:0002f5b0 r6:4097d7a5 r5:00000000
[833089.144134] r4:c08b6ee8
[833089.144134] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[833089.164978] r5:c08b3f28 r4:eefad690
[833089.164978] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[833089.178253] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[833089.178253] r7:c00978d0 r6:eefad400 r5:eefad438 r4:eefad690
[833089.193359] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[833089.193359] r7:00000001 r6:eefad400 r5:0002f5b0 r4:4097d7a5
[833089.208740] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[833089.208740] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[833089.225372] r4:00000001
[833089.225372] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[833089.237030] r5:ee803d80 r4:eefb1f00
[833089.237030] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[833089.240875] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[833089.240875] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[833089.240875] r5:00000000 r4:c08afc18
[833089.271606] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[833089.271606] r9:c08ed664 r8:c08b3f80 r7:fa240100 r6:c08b3f28 r5:c08b478c r4:fa24010c
[833089.271606] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[833089.271606] Exception stack(0xc08b3f28 to 0xc08b3f70)
[833089.271606] 3f20: 00000000 00000000 00000000 c002ad20 c08b2000 c08b449c
[833089.310516] 3f40: c08ae344 c066b7c8 c08b3f80 c08ed664 c08b44e8 c08b3f7c c08b3f70 c08b3f70
[833089.310516] 3f60: c002aaec c00109e0 a0000113 ffffffff
[833089.310516] r7:c08b3f5c r6:ffffffff r5:a0000113 r4:c00109e0
[833089.330535] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[833089.339263] [<c006e028>] (cpu_startup_entry) from [<c065e9bc>] (rest_init+0x90/0x94)
[833089.347442] r7:ffffffff r4:00000002
[833089.347442] [<c065e92c>] (rest_init) from [<c086ecd4>] (start_kernel+0x368/0x374)
[833089.347442] r5:c08f0000 r4:00000000
[833089.359191] [<c086e96c>] (start_kernel) from [<8000807c>] (0x8000807c)
[893756.081939] INFO: rcu_preempt self-detected stall on CPU
[893756.087097] INFO: rcu_preempt detected stalls on CPUs/tasks:
[893756.087097] {
[893756.087097] 1
[893756.087097] }
[893756.087127] (detected by 0, t=2722 jiffies, g=2886909, c=2886908, q=198)
[893756.087127] Task dump for CPU 1:
[893756.087127] swapper/1 R
[893756.087127] running
[893756.087127] 0 0 1 0x00000002
[893756.087127] Backtrace:
[893756.087158] [<ee89ffb8>] (0xee89ffb8) from [<c08b44e8>] (0xc08b44e8)
[893756.087188] [<c08c688c>] (0xc08c688c) from [<c00109c4>] (arch_cpu_idle+0xc/0x44)
[893756.087188] Backtrace aborted due to bad frame pointer <ee89ffc8>
[893756.132812] {
[893756.134948] 1} (t=2722 jiffies g=2886909 c=2886908 q=198)
[893756.138610] Task dump for CPU 1:
[893756.142364] swapper/1 R running 0 0 1 0x00000002
[893756.149688] Backtrace:
[893756.152221] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[893756.160339] r7:c08b4518 r6:80080193 r5:00000001 r4:ee877200
[893756.166381] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[893756.174530] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[893756.182922] r5:c08c3b00 r4:00000001
[893756.186767] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[893756.195129] r5:c08c3b00 r4:00000002
[893756.198730] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[893756.208679] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e709000 r5:c08b1980 r4:eefba980
[893756.216278] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[893756.225646] r10:00000000 r9:eefb8438 r8:eefb8518 r7:00032cdd r6:5036860d r5:00000000
[893756.234374] r4:ee877200
[893756.236267] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[893756.247131] r5:ee89ff70 r4:eefb8690
[893756.250579] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[893756.260192] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[893756.269287] r7:c00978d0 r6:eefb8400 r5:eefb8438 r4:eefb8690
[893756.275482] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[893756.282653] r7:00000001 r6:eefb8400 r5:00032cdd r4:5036860d
[893756.290954] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[893756.296661] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[893756.296661] r4:00000001
[893756.296661] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[893756.319244] r5:ee803d80 r4:eefbcf00
[893756.319244] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[893756.326660] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[893756.336639] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[893756.346649] r5:00000000 r4:c08afc18
[893756.346649] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[893756.362640] r9:c08ed664 r8:ee89ffc8 r7:fa240100 r6:ee89ff70 r5:c08b478c r4:fa24010c
[893756.366638] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[893756.376647] Exception stack(0xee89ff70 to 0xee89ffb8)
[893756.383483] ff60: 00000000 00000000 00000000 c002ad20
[893756.390319] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[893756.396636] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a0080113 ffffffff
[893756.406646] r7:ee89ffa4 r6:ffffffff r5:a0080113 r4:c00109e0
[893756.414398] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[893756.423126] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[893756.423126] r7:c08f0280 r4:c08c6898
[893756.435913] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[893756.436645] r5:00000015 r4:ae88406a
[910026.874572] INFO: rcu_preempt self-detected stall on CPU
[910026.874572] INFO: rcu_preempt self-detected stall on CPU
[910026.874572] {
[910026.874603] 1
[910026.874603] }
[910026.874603] (t=2463 jiffies g=2949434 c=2949433 q=3)
[910026.874603] rcu_preempt kthread starved for 2463 jiffies!
[910026.874603] Task dump for CPU 0:
[910026.874603] swapper/0 R
[910026.874603] running
[910026.874633] 0 0 0 0x00000000
[910026.874633] Backtrace:
[910026.874633] [<c08b3f70>] (0xc08b3f70) from [<c08b44e8>] (0xc08b44e8)
[910026.874664] Backtrace aborted due to bad frame pointer <c08b3f80>
[910026.874664] Task dump for CPU 1:
[910026.874664] swapper/1 R
[910026.874664] running
[910026.874664] 0 0 1 0x00000000
[910026.874664] Backtrace:
[910026.874664] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[910026.874664] r7:c08b4518
[910026.874664] r6:80080193
[910026.874725] r5:00000001
[910026.874725] r4:ee877200
[910026.874725] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[910026.874755] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[910026.874755] r5:c08c3b00
[910026.874755] r4:00000001
[910026.874786] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[910026.874786] r5:c08c3b00
[910026.874786] r4:00000002
[910026.874816] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[910026.874816] r9:c08b4518
[910026.874816] r8:c08c3b00
[910026.874847] r7:c08c3b00
[910026.874847] r6:2e709000
[910026.874847] r5:c08b1980
[910026.874847] r4:eefba980
[910026.874847] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[910026.874847] r10:00000000
[910026.874877] r9:eefb8438
[910026.874877] r8:eefb8518
[910026.874877] r7:00033ba9
[910026.874877] r6:9c396abb
[910026.874877] r5:00000000
[910026.874877] r4:ee877200
[910026.874908] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[910026.874908] r5:ee89ff70
[910026.874908] r4:eefb8690
[910026.874938] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[910026.874938] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[910026.874938] r7:c00978d0
[910026.874938] r6:eefb8400
[910026.874938] r5:eefb8438
[910026.874969] r4:eefb8690
[910026.874969] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[910026.874969] r7:00000001
[910026.874969] r6:eefb8400
[910026.874969] r5:00033ba9
[910026.874999] r4:9c396abb
[910026.874999] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[910026.874999] r10:c08b44e8
[910026.874999] r9:c08ed664
[910026.874999] r8:ee828940
[910026.875030] r7:00000011
[910026.875030] r6:c08d0208
[910026.875030] r5:ee803d80
[910026.875030] r4:00000001
[910026.875061] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[910026.875061] r5:ee803d80
[910026.875061] r4:eefbcf00
[910026.875091] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[910026.875091] r9:c08ed664
[910026.875091] r8:ee808000
[910026.875091] r7:00000001
[910026.875091] r6:00000000
[910026.875091] r5:00000000
[910026.875091] r4:00000011
[910026.875122] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[910026.875122] r5:00000000
[910026.875122] r4:c08afc18
[910026.875152] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[910026.875152] r9:c08ed664
[910026.875152] r8:ee89ffc8
[910026.875152] r7:fa240100
[910026.875152] r6:ee89ff70
[910026.875152] r5:c08b478c
[910026.875152] r4:fa24010c
[910026.875183] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[910026.875183] Exception stack(0xee89ff70 to 0xee89ffb8)
[910026.875183] ff60: 00000000 00000000 00000000 c002ad20
[910026.875213] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[910026.875213] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a0080113 ffffffff
[910026.875213] r7:ee89ffa4
[910026.875213] r6:ffffffff
[910026.875213] r5:a0080113
[910026.875213] r4:c00109e0
[910026.875244] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[910026.875274] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[910026.875274] r7:c08f0280
[910026.875274] r4:c08c6898
[910026.875274] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[910026.875305] r5:00000015
[910026.875305] r4:ae88406a
[910027.229675] {
[910027.229675] 0} (t=2494 jiffies g=2949434 c=2949433 q=3)
[910027.229675] rcu_preempt kthread starved for 2494 jiffies!
[910027.238067] Task dump for CPU 0:
[910027.238067] swapper/0 R running 0 0 0 0x00000000
[910027.247985] Backtrace:
[910027.247985] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[910027.257904] r7:c08b4518 r6:800c0193 r5:00000000 r4:c08b6ee8
[910027.268402] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[910027.268402] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[910027.277954] r5:c08c3b00 r4:00000000
[910027.277954] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[910027.277954] r5:c08c3b00 r4:00000001
[910027.297912] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[910027.310791] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e6fe000 r5:c08b1980 r4:eefaf980
[910027.310791] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[910027.310791] r10:00000000 r9:eefad438 r8:eefad518 r7:00033ba9 r6:9c396abb r5:00000000
[910027.336639] r4:c08b6ee8
[910027.339385] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[910027.339385] r5:c08b3f28 r4:eefad690
[910027.339385] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[910027.339385] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[910027.371551] r7:c00978d0 r6:eefad400 r5:eefad438 r4:eefad690
[910027.371551] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[910027.386932] r7:00000001 r6:eefad400 r5:00033ba9 r4:9c396abb
[910027.388854] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[910027.397918] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[910027.397918] r4:00000001
[910027.409606] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[910027.416961] r5:ee803d80 r4:eefb1f00
[910027.425079] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[910027.430541] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[910027.430541] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[910027.451995] r5:00000000 r4:c08afc18
[910027.455841] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[910027.459960] r9:c08ed664 r8:c08b3f80 r7:fa240100 r6:c08b3f28 r5:c08b478c r4:fa24010c
[910027.472839] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[910027.480743] Exception stack(0xc08b3f28 to 0xc08b3f70)
[910027.480743] 3f20: 00000000 00000000 00000000 c002ad20 c08b2000 c08b449c
[910027.487915] 3f40: c08ae344 c066b7c8 c08b3f80 c08ed664 c08b44e8 c08b3f7c c08b3f70 c08b3f70
[910027.495086] 3f60: c002aaec c00109e0 a00c0113 ffffffff
[910027.508728] r7:c08b3f5c r6:ffffffff r5:a00c0113 r4:c00109e0
[910027.514770] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[910027.523498] [<c006e028>] (cpu_startup_entry) from [<c065e9bc>] (rest_init+0x90/0x94)
[910027.528594] r7:ffffffff r4:00000002
[910027.531677] [<c065e92c>] (rest_init) from [<c086ecd4>] (start_kernel+0x368/0x374)
[910027.537902] r5:c08f0000 r4:00000000
[910027.547241] [<c086e96c>] (start_kernel) from [<8000807c>] (0x8000807c)
[959885.223114] INFO: rcu_preempt self-detected stall on CPU
[959885.223602] INFO: rcu_preempt self-detected stall on CPU
[959885.223602] {
[959885.223602] 0
[959885.223602] }
[959885.223602] (t=7148 jiffies g=3140935 c=3140934 q=13)
[959885.223632] rcu_preempt kthread starved for 7148 jiffies!
[959885.223632] Task dump for CPU 0:
[959885.223632] smsd R
[959885.223632] running
[959885.223632] 0 1014 1 0x00000000
[959885.223632] Backtrace:
[959885.223663] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[959885.223693] r7:c08b4518
[959885.223693] r6:80000193
[959885.223693] r5:00000001
[959885.223693] r4:ee0ff200
[959885.223724] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[959885.223724] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[959885.223754] r5:c08c3b00
[959885.223754] r4:00000000
[959885.223754] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[959885.223785] r5:c08c3b00
[959885.223785] r4:00000001
[959885.223785] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[959885.223785] r9:c08b4518
[959885.223815] r8:c08c3b00
[959885.223815] r7:c08c3b00
[959885.223815] r6:2e6fe000
[959885.223815] r5:c08b1980
[959885.223815] r4:eefaf980
[959885.223846] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[959885.223846] r10:00000000
[959885.223846] r9:eefad438
[959885.223846] r8:eefad518
[959885.223846] r7:00036902
[959885.223846] r6:084d9b9f
[959885.223846] r5:00000001
[959885.223846] r4:ee0ff200
[959885.223876] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[959885.223876] r5:ee2d7fb0
[959885.223907] r4:eefad690
[959885.223907] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[959885.223907] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[959885.223937] r7:c00978d0
[959885.223937] r6:eefad400
[959885.223937] r5:eefad438
[959885.223937] r4:eefad690
[959885.223968] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[959885.223968] r7:00000001
[959885.223968] r6:eefad400
[959885.223968] r5:00036902
[959885.223968] r4:084b4795
[959885.223999] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[959885.223999] r10:0005fc3c
[959885.223999] r9:beb7dca4
[959885.223999] r8:ee828940
[959885.223999] r7:00000011
[959885.223999] r6:c08d0208
[959885.223999] r5:ee803d80
[959885.223999] r4:00000001
[959885.224029] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[959885.224029] r5:ee803d80
[959885.224060] r4:eefb1f00
[959885.224060] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[959885.224060] r9:beb7dca4
[959885.224060] r8:ee808000
[959885.224090] r7:00000001
[959885.224090] r6:00000000
[959885.224090] r5:00000000
[959885.224090] r4:00000011
[959885.224090] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[959885.224090] r5:00000000
[959885.224121] r4:c08afc18
[959885.224121] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[959885.224121] r9:beb7dca4
[959885.224121] r8:10c5387d
[959885.224121] r7:fa240100
[959885.224151] r6:ee2d7fb0
[959885.224151] r5:c08b478c
[959885.224151] r4:fa24010c
[959885.224151] [<c000947c>] (gic_handle_irq) from [<c0014528>] (__irq_usr+0x48/0x60)
[959885.224151] Exception stack(0xee2d7fb0 to 0xee2d7ff8)
[959885.224182] 7fa0: beb7dcbb 00000067 80000000 00006e69
[959885.224182] 7fc0: 0005fc53 00000017 00000000 00000017 beb7c600 beb7dca4 0005fc3c beb7c600
[959885.224182] 7fe0: beb7dcba beb7c0a0 b6eca429 b6ec1570 60000130 ffffffff
[959885.224212] r7:10c5387d
[959885.224212] r6:ffffffff
[959885.224212] r5:60000130
[959885.224212] r4:b6ec1570
[959885.224212] Task dump for CPU 1:
[959885.224212] swapper/1 R
[959885.224212] running
[959885.224243] 0 0 1 0x00000000
[959885.224243] Backtrace:
[959885.224243] [<ee89ffb8>] (0xee89ffb8) from [<c08b44e8>] (0xc08b44e8)
[959885.224273] [<c08c688c>] (0xc08c688c) from [<c00109c4>] (arch_cpu_idle+0xc/0x44)
[959885.224273] Backtrace aborted due to bad frame pointer <ee89ffc8>
[959885.363555] {
[959885.363555] 1} (t=7148 jiffies g=3140935 c=3140934 q=17)
[959885.558654] rcu_preempt kthread starved for 7148 jiffies!
[959885.558654] Task dump for CPU 1:
[959885.567840] swapper/1 R running 0 0 1 0x00000000
[959885.573699] Backtrace:
[959885.577239] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[959885.583343] r7:c08b4518 r6:800c0193 r5:00000001 r4:ee877200
[959885.583343] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[959885.599456] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[959885.599456] r5:c08c3b00 r4:00000001
[959885.611053] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[959885.613342] r5:c08c3b00 r4:00000002
[959885.613342] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[959885.633636] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e709000 r5:c08b1980 r4:eefba980
[959885.641845] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[959885.643341] r10:00000000 r9:eefb8438 r8:eefb8518 r7:00036902 r6:0845b116 r5:00000000
[959885.653320] r4:ee877200
[959885.659667] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[959885.663330] r5:ee89ff70 r4:eefb8690
[959885.673339] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[959885.683349] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[959885.683349] r7:c00978d0 r6:eefb8400 r5:eefb8438 r4:eefb8690
[959885.700439] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[959885.703338] r7:00000001 r6:eefb8400 r5:00036902 r4:0845b116
[959885.715820] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[959885.723327] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[959885.732452] r4:00000001
[959885.734191] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[959885.743347] r5:ee803d80 r4:eefbcf00
[959885.743347] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[959885.752471] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[959885.765686] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[959885.765686] r5:00000000 r4:c08afc18
[959885.778686] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[959885.778686] r9:c08ed664 r8:ee89ffc8 r7:fa240100 r6:ee89ff70 r5:c08b478c r4:fa24010c
[959885.793334] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[959885.793334] Exception stack(0xee89ff70 to 0xee89ffb8)
[959885.808959] ff60: 00000000 00000000 00000000 c002ad20
[959885.813323] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[959885.823333] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a00c0113 ffffffff
[959885.823333] r7:ee89ffa4 r6:ffffffff r5:a00c0113 r4:c00109e0
[959885.839263] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[959885.847991] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[959885.857513] r7:c08f0280 r4:c08c6898
[959885.857513] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[959885.857513] r5:00000015 r4:ae88406a
[998834.122436] INFO: rcu_preempt self-detected stall on CPU
[998834.127563] INFO: rcu_preempt detected stalls on CPUs/tasks:
[998834.127563] {
[998834.127563] 0
[998834.127563] }
[998834.127563] (detected by 1, t=3308 jiffies, g=3290917, c=3290916, q=162)
[998834.127593] Task dump for CPU 0:
[998834.127593] swapper/0 R
[998834.127593] running
[998834.127593] 0 0 0 0x00000002
[998834.127593] Backtrace:
[998834.127624] [<c08b3f70>] (0xc08b3f70) from [<c08b44e8>] (0xc08b44e8)
[998834.127624] Backtrace aborted due to bad frame pointer <c08b3f80>
[998834.165557] {
[998834.167816] 0} (t=3308 jiffies g=3290917 c=3290916 q=319)
[998834.171722] Task dump for CPU 0:
[998834.175750] swapper/0 R running 0 0 0 0x00000002
[998834.181915] Backtrace:
[998834.185058] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[998834.193145] r7:c08b4518 r6:80080193 r5:00000000 r4:c08b6ee8
[998834.199188] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[998834.207366] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[998834.215332] r5:c08c3b00 r4:00000000
[998834.219146] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[998834.227508] r5:c08c3b00 r4:00000001
[998834.231811] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[998834.241577] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e6fe000 r5:c08b1980 r4:eefaf980
[998834.249786] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[998834.258880] r10:00000000 r9:eefad438 r8:eefad518 r7:00038c6e r6:6db7fb31 r5:00000000
[998834.267364] r4:c08b6ee8
[998834.270019] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[998834.279296] r5:c08b3f28 r4:eefad690
[998834.283813] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[998834.293243] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[998834.302246] r7:c00978d0 r6:eefad400 r5:eefad438 r4:eefad690
[998834.308380] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[998834.317596] r7:00000001 r6:eefad400 r5:00038c6e r4:6db7fb31
[998834.323547] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[998834.332122] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[998834.332122] r4:00000001
[998834.343170] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[998834.349121] r5:ee803d80 r4:eefb1f00
[998834.349121] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[998834.357238] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[998834.372528] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[998834.382812] r5:00000000 r4:c08afc18
[998834.382812] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[998834.391632] r9:c08ed664 r8:c08b3f80 r7:fa240100 r6:c08b3f28 r5:c08b478c r4:fa24010c
[998834.403655] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[998834.407226] Exception stack(0xc08b3f28 to 0xc08b3f70)
[998834.407226] 3f20: 00000000 00000000 00000000 c002ad20 c08b2000 c08b449c
[998834.425567] 3f40: c08ae344 c066b7c8 c08b3f80 c08ed664 c08b44e8 c08b3f7c c08b3f70 c08b3f70
[998834.427276] 3f60: c002aaec c00109e0 a0080113 ffffffff
[998834.437255] r7:c08b3f5c r6:ffffffff r5:a0080113 r4:c00109e0
[998834.445587] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[998834.450500] [<c006e028>] (cpu_startup_entry) from [<c065e9bc>] (rest_init+0x90/0x94)
[998834.462493] r7:ffffffff r4:00000002
[998834.466339] [<c065e92c>] (rest_init) from [<c086ecd4>] (start_kernel+0x368/0x374)
[998834.468109] r5:c08f0000 r4:00000000
[998834.468109] [<c086e96c>] (start_kernel) from [<8000807c>] (0x8000807c)
[1101572.068176] INFO: rcu_preempt self-detected stall on CPU
[1101572.068176] INFO: rcu_preempt detected stalls on CPUs/tasks:
[1101572.068176] {
[1101572.068206] 1
[1101572.068206] }
[1101572.068206] (detected by 0, t=2638 jiffies, g=3687571, c=3687570, q=16)
[1101572.068206] Task dump for CPU 1:
[1101572.068206] swapper/1 R
[1101572.068206] running
[1101572.068237] 0 0 1 0x00000000
[1101572.068237] Backtrace:
[1101572.068237] [<ee89ffb8>] (0xee89ffb8) from [<c08b44e8>] (0xc08b44e8)
[1101572.068237] [<c08c688c>] (0xc08c688c) from [<c00109c4>] (arch_cpu_idle+0xc/0x44)
[1101572.068237] Backtrace aborted due to bad frame pointer <ee89ffc8>
[1101572.068237] rcu_preempt kthread starved for 2638 jiffies!
[1101572.125762] {
[1101572.127349] 1} (t=2645 jiffies g=3687571 c=3687570 q=18)
[1101572.132019] Task dump for CPU 1:
[1101572.135711] swapper/1 R running 0 0 1 0x00000002
[1101572.142395] Backtrace:
[1101572.145294] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[1101572.145294] r7:c08b4518 r6:80080193 r5:00000001 r4:ee877200
[1101572.145294] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[1101572.167785] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[1101572.167785] r5:c08c3b00 r4:00000001
[1101572.167785] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[1101572.188964] r5:c08c3b00 r4:00000002
[1101572.188964] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[1101572.188964] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e709000 r5:c08b1980 r4:eefba980
[1101572.210723] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[1101572.210723] r10:00000000 r9:eefb8438 r8:eefb8518 r7:0003e9de r6:b3ae6379 r5:00000000
[1101572.228546] r4:ee877200
[1101572.231384] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[1101572.231384] r5:ee89ff70 r4:eefb8690
[1101572.245208] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[1101572.245208] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[1101572.245208] r7:c00978d0 r6:eefb8400 r5:eefb8438 r4:eefb8690
[1101572.245208] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[1101572.279479] r7:00000001 r6:eefb8400 r5:0003e9de r4:b3ae6379
[1101572.279479] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[1101572.294036] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[1101572.302429] r4:00000001
[1101572.302429] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[1101572.302429] r5:ee803d80 r4:eefbcf00
[1101572.302429] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[1101572.302429] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[1101572.302429] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[1101572.345367] r5:00000000 r4:c08afc18
[1101572.345367] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[1101572.358184] r9:c08ed664 r8:ee89ffc8 r7:fa240100 r6:ee89ff70 r5:c08b478c r4:fa24010c
[1101572.358184] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[1101572.358184] Exception stack(0xee89ff70 to 0xee89ffb8)
[1101572.358184] ff60: 00000000 00000000 00000000 c002ad20
[1101572.358184] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[1101572.397369] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a0080113 ffffffff
[1101572.397369] r7:ee89ffa4 r6:ffffffff r5:a0080113 r4:c00109e0
[1101572.410583] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[1101572.419403] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[1101572.419403] r7:c08f0280 r4:c08c6898
[1101572.432952] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[1101572.432952] r5:00000015 r4:ae88406a
[1102135.210327] INFO: rcu_preempt self-detected stall on CPU { 0} (t=0 jiffies g=3689626 c=3689625 q=12)
[1102135.220184] Task dump for CPU 0:
[1102135.220184] swapper/0 R running 0 0 0 0x00000002
[1102135.224060] Backtrace:
[1102135.233337] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[1102135.233337] r7:c08b4518 r6:80030193 r5:00000000 r4:c08b6ee8
[1102135.233337] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[1102135.255828] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[1102135.255828] r5:c08c3b00 r4:00000000
[1102135.255828] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[1102135.277008] r5:c08c3b00 r4:00000001
[1102135.277008] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[1102135.277008] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e6fe000 r5:c08b1980 r4:eefaf980
[1102135.277008] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[1102135.277008] r10:00000000 r9:eefad438 r8:eefad518 r7:0003ea61 r6:d1c6800c r5:00000000
[1102135.308227] r4:c08b6ee8
[1102135.308227] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[1102135.308227] r5:c08b3f28 r4:eefad690
[1102135.308227] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[1102135.342803] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[1102135.342803] r7:c00978d0 r6:eefad400 r5:eefad438 r4:eefad690
[1102135.358093] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[1102135.358093] r7:00000001 r6:eefad400 r5:0003ea61 r4:d1c6800c
[1102135.358093] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[1102135.373657] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[1102135.390472] r4:00000001
[1102135.390472] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[1102135.390472] r5:ee803d80 r4:eefb1f00
[1102135.402313] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[1102135.415863] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[1102135.415863] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[1102135.415863] r5:00000000 r4:c08afc18
[1102135.437347] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[1102135.437347] r9:c08ed664 r8:c08b3f80 r7:fa240100 r6:c08b3f28 r5:c08b478c r4:fa24010c
[1102135.437347] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[1102135.437347] Exception stack(0xc08b3f28 to 0xc08b3f70)
[1102135.467987] 3f20: 00000000 00000000 00000000 c002ad20 c08b2000 c08b449c
[1102135.467987] 3f40: c08ae344 c066b7c8 c08b3f80 c08ed664 c08b44e8 c08b3f7c c08b3f70 c08b3f70
[1102135.467987] 3f60: c002aaec c00109e0 a0030113 ffffffff
[1102135.485443] r7:c08b3f5c r6:ffffffff r5:a0030113 r4:c00109e0
[1102135.497009] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[1102135.497009] [<c006e028>] (cpu_startup_entry) from [<c065e9bc>] (rest_init+0x90/0x94)
[1102135.514099] r7:ffffffff r4:00000002
[1102135.514099] [<c065e92c>] (rest_init) from [<c086ecd4>] (start_kernel+0x368/0x374)
[1102135.518035] r5:c08f0000 r4:00000000
[1102135.518035] [<c086e96c>] (start_kernel) from [<8000807c>] (0x8000807c)
[1102135.518035] Task dump for CPU 1:
[1102135.540496] swapper/1 R running 0 0 1 0x00000000
[1102135.540496] Backtrace:
[1102135.540496] [<ee89ffb8>] (0xee89ffb8) from [<c08b44e8>] (0xc08b44e8)
[1102135.556884] [<c08c688c>] (0xc08c688c) from [<c00109c4>] (arch_cpu_idle+0xc/0x44)
[1102135.556884] Backtrace aborted due to bad frame pointer <ee89ffc8>
[1106110.304504] INFO: rcu_preempt self-detected stall on CPU { 0} (t=2318 jiffies g=3704906 c=3704905 q=199)
[1106110.312896] INFO: rcu_preempt self-detected stall on CPU { 1} (t=2318 jiffies g=3704906 c=3704905 q=199)
[1106110.312896] rcu_preempt kthread starved for 2318 jiffies!
[1106110.312896] Task dump for CPU 0:
[1106110.312927] swapper/0 R running 0 0 0 0x00000000
[1106110.312927] Backtrace:
[1106110.312927] [<c08b3f70>] (0xc08b3f70) from [<c08b44e8>] (0xc08b44e8)
[1106110.312957] Backtrace aborted due to bad frame pointer <c08b3f80>
[1106110.312957] Task dump for CPU 1:
[1106110.312957] swapper/1 R running 0 0 1 0x00000002
[1106110.312957] Backtrace:
[1106110.312957] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[1106110.313018] r7:c08b4518 r6:800f0193 r5:00000001 r4:ee877200
[1106110.313018] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[1106110.313018] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[1106110.313018] r5:c08c3b00 r4:00000001
[1106110.313079] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[1106110.313079] r5:c08c3b00 r4:00000002
[1106110.313110] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[1106110.313140] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e709000 r5:c08b1980 r4:eefba980
[1106110.313140] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[1106110.313171] r10:00000000 r9:eefb8438 r8:eefb8518 r7:0003edff r6:55b6684c r5:00000000
[1106110.313171] r4:ee877200
[1106110.313171] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[1106110.313201] r5:ee89ff70 r4:eefb8690
[1106110.313201] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[1106110.313232] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[1106110.313232] r7:c00978d0 r6:eefb8400 r5:eefb8438 r4:eefb8690
[1106110.313262] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[1106110.313262] r7:00000001 r6:eefb8400 r5:0003edff r4:55b6684c
[1106110.313293] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[1106110.313293] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[1106110.313293] r4:00000001
[1106110.313323] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[1106110.313323] r5:ee803d80 r4:eefbcf00
[1106110.313354] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[1106110.313354] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[1106110.313384] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[1106110.313384] r5:00000000 r4:c08afc18
[1106110.313415] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[1106110.313415] r9:c08ed664 r8:ee89ffc8 r7:fa240100 r6:ee89ff70 r5:c08b478c r4:fa24010c
[1106110.313415] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[1106110.313446] Exception stack(0xee89ff70 to 0xee89ffb8)
[1106110.313446] ff60: 00000000 00000000 00000000 c002ad20
[1106110.313446] ff80: ee89e000 c08b449c c08ae344 c066b7c8 ee89ffc8 c08ed664 c08b44e8 ee89ffc4
[1106110.313476] ffa0: ee89ffb8 ee89ffb8 c002aaec c00109e0 a00f0113 ffffffff
[1106110.313476] r7:ee89ffa4 r6:ffffffff r5:a00f0113 r4:c00109e0
[1106110.313507] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[1106110.313507] [<c006e028>] (cpu_startup_entry) from [<c00155a0>] (secondary_start_kernel+0x160/0x16c)
[1106110.313537] r7:c08f0280 r4:c08c6898
[1106110.313537] [<c0015440>] (secondary_start_kernel) from [<8000956c>] (0x8000956c)
[1106110.313537] r5:00000015 r4:ae88406a
[1106110.668914] rcu_preempt kthread starved for 2318 jiffies!
[1106110.668914] Task dump for CPU 0:
[1106110.675598] swapper/0 R running 0 0 0 0x00000000
[1106110.675598] Backtrace:
[1106110.687866] [<c00134c0>] (dump_backtrace) from [<c00136e4>] (show_stack+0x18/0x1c)
[1106110.687866] r7:c08b4518 r6:80010193 r5:00000000 r4:c08b6ee8
[1106110.696716] [<c00136cc>] (show_stack) from [<c005b64c>] (sched_show_task+0xbc/0x110)
[1106110.706695] [<c005b590>] (sched_show_task) from [<c005ead0>] (dump_cpu_task+0x40/0x48)
[1106110.712280] r5:c08c3b00 r4:00000000
[1106110.722717] [<c005ea90>] (dump_cpu_task) from [<c007fa94>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[1106110.731536] r5:c08c3b00 r4:00000001
[1106110.733032] [<c007f9f8>] (rcu_dump_cpu_stacks) from [<c0083ce8>] (rcu_check_callbacks+0x500/0x8f4)
[1106110.744995] r9:c08b4518 r8:c08c3b00 r7:c08c3b00 r6:2e6fe000 r5:c08b1980 r4:eefaf980
[1106110.753295] [<c00837e8>] (rcu_check_callbacks) from [<c00866ec>] (update_process_times+0x40/0x6c)
[1106110.762725] r10:00000000 r9:eefad438 r8:eefad518 r7:0003edff r6:55365a1b r5:00000000
[1106110.768585] r4:c08b6ee8
[1106110.768585] [<c00866ac>] (update_process_times) from [<c00978cc>] (tick_sched_handle.isra.6+0x4c/0x50)
[1106110.783843] r5:c08b3f28 r4:eefad690
[1106110.787078] [<c0097880>] (tick_sched_handle.isra.6) from [<c0097924>] (tick_sched_timer+0x54/0x98)
[1106110.787780] [<c00978d0>] (tick_sched_timer) from [<c00874b8>] (__run_hrtimer.isra.5+0x4c/0xdc)
[1106110.806488] r7:c00978d0 r6:eefad400 r5:eefad438 r4:eefad690
[1106110.812286] [<c008746c>] (__run_hrtimer.isra.5) from [<c008780c>] (hrtimer_interrupt+0x100/0x2f4)
[1106110.817749] r7:00000001 r6:eefad400 r5:0003edff r4:55365a1b
[1106110.828155] [<c008770c>] (hrtimer_interrupt) from [<c00164b8>] (twd_handler+0x38/0x48)
[1106110.832305] r10:c08b44e8 r9:c08ed664 r8:ee828940 r7:00000011 r6:c08d0208 r5:ee803d80
[1106110.836608] r4:00000001
[1106110.846405] [<c0016480>] (twd_handler) from [<c007adf4>] (handle_percpu_devid_irq+0x74/0x8c)
[1106110.852294] r5:ee803d80 r4:eefb1f00
[1106110.852294] [<c007ad80>] (handle_percpu_devid_irq) from [<c0076cec>] (generic_handle_irq+0x34/0x44)
[1106110.862304] r9:c08ed664 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011
[1106110.862304] [<c0076cb8>] (generic_handle_irq) from [<c0076fc4>] (__handle_domain_irq+0x64/0xbc)
[1106110.862304] r5:00000000 r4:c08afc18
[1106110.891845] [<c0076f60>] (__handle_domain_irq) from [<c00094a8>] (gic_handle_irq+0x2c/0x64)
[1106110.893127] r9:c08ed664 r8:c08b3f80 r7:fa240100 r6:c08b3f28 r5:c08b478c r4:fa24010c
[1106110.902282] [<c000947c>] (gic_handle_irq) from [<c0014240>] (__irq_svc+0x40/0x74)
[1106110.917022] Exception stack(0xc08b3f28 to 0xc08b3f70)
[1106110.922485] 3f20: 00000000 00000000 00000000 c002ad20 c08b2000 c08b449c
[1106110.931213] 3f40: c08ae344 c066b7c8 c08b3f80 c08ed664 c08b44e8 c08b3f7c c08b3f70 c08b3f70
[1106110.931213] 3f60: c002aaec c00109e0 a0010113 ffffffff
[1106110.943786] r7:c08b3f5c r6:ffffffff r5:a0010113 r4:c00109e0
[1106110.951507] [<c00109b8>] (arch_cpu_idle) from [<c006e148>] (cpu_startup_entry+0x120/0x1a0)
[1106110.953063] [<c006e028>] (cpu_startup_entry) from [<c065e9bc>] (rest_init+0x90/0x94)
[1106110.953063] r7:ffffffff r4:00000002
[1106110.953063] [<c065e92c>] (rest_init) from [<c086ecd4>] (start_kernel+0x368/0x374)
[1106110.980499] r5:c08f0000 r4:00000000
[1106110.984069] [<c086e96c>] (start_kernel) from [<8000807c>] (0x8000807c)
#
#
#
#
[-- Attachment #4: dmesg-3.18 --]
[-- Type: application/octet-stream, Size: 143406 bytes --]
$
$ uptime
19:43:30 up 4:46, load average: 0.07, 0.09, 0.12
$
$
$ cat /proc/version
Linux version 3.18.0-panda (rfg@red.dwarf) (gcc version 4.9.2 (Buildroot 2014.11-git-01039-g9f8180d) ) #1 SMP PREEMPT Mon Dec 8 14:29:55 AEDT 2014
$
# dmesg
[ 3.717803] random: dd urandom read with 21 bits of entropy available
[ 7.814697] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[ 17.600524] Bluetooth: Core ver 2.19
[ 17.604431] NET: Registered protocol family 31
[ 17.608306] Bluetooth: HCI device and connection manager initialized
[ 17.615844] Bluetooth: HCI socket layer initialized
[ 17.620971] Bluetooth: L2CAP socket layer initialized
[ 17.625640] Bluetooth: SCO socket layer initialized
[ 23.349731] RPC: Registered named UNIX socket transport module.
[ 23.349822] RPC: Registered udp transport module.
[ 23.360870] RPC: Registered tcp transport module.
[ 23.360870] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 23.426574] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 23.440551] svc: failed to register nfsdv2 RPC service (errno 111).
[ 23.447357] svc: failed to register nfsaclv2 RPC service (errno 111).
[ 47.411773] cfg80211: Calling CRDA to update world regulatory domain
[ 47.486389] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 47.506591] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 47.793640] wlcore: loaded
[ 59.089721] random: nonblocking pool is initialized
[ 59.567352] wlcore: firmware booted (Rev 6.3.10.0.133)
[ 62.548858] wlan0: authenticate with 90:f6:52:81:ee:70
[ 62.590179] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[ 62.602386] wlan0: authenticated
[ 62.609344] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[ 62.623565] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[ 62.640441] wlan0: associated
[ 62.690673] wlcore: Association completed.
[ 4936.259094] INFO: rcu_preempt self-detected stall on CPU
[ 4936.261260] INFO: rcu_preempt self-detected stall on CPU
[ 4936.261260] {
[ 4936.261260] 0
[ 4936.261260] }
[ 4936.261260] (t=4097 jiffies g=15318 c=15317 q=198)
[ 4936.261260] Task dump for CPU 0:
[ 4936.261260] swapper/0 R
[ 4936.261260] running
[ 4936.261291] 0 0 0 0x00000002
[ 4936.261291] Backtrace:
[ 4936.261291] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[ 4936.261291] r6:800f0193
[ 4936.261291] r5:00000000
[ 4936.261291] r4:c07a0920
[ 4936.261291] r3:00000000
[ 4936.261352] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[ 4936.261383] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[ 4936.261383] r5:c07b1240
[ 4936.261383] r4:00000000
[ 4936.261413] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 4936.261413] r4:00000001
[ 4936.261413] r3:00000003
[ 4936.261413] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[ 4936.261444] r8:000000c6
[ 4936.261444] r7:c079e798
[ 4936.261444] r6:c07b1240
[ 4936.261444] r5:c07933d8
[ 4936.261444] r4:eefb13d8
[ 4936.261444] r3:00010001
[ 4936.261474] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[ 4936.261474] r10:00000001
[ 4936.261474] r9:eefb15d0
[ 4936.261474] r8:eefb15d0
[ 4936.261474] r7:00000000
[ 4936.261474] r6:00000000
[ 4936.261505] r5:c07a0920
[ 4936.261505] r4:c0796000
[ 4936.261505] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[ 4936.261505] r7:0000047d
[ 4936.261535] r6:44b319c0
[ 4936.261535] r5:c0797f20
[ 4936.261535] r4:eefb1858
[ 4936.261535] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[ 4936.261566] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[ 4936.261566] r7:c0093510
[ 4936.261566] r6:eefb15d0
[ 4936.261566] r5:eefb1608
[ 4936.261566] r4:eefb1858
[ 4936.261596] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[ 4936.261596] r7:0000047d
[ 4936.261596] r6:44b319c0
[ 4936.261596] r5:00000000
[ 4936.261596] r4:eefb15d0
[ 4936.261627] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[ 4936.261627] r10:c0789aac
[ 4936.261627] r9:c07d6bd9
[ 4936.261627] r8:0000001d
[ 4936.261627] r7:ee813b40
[ 4936.261627] r6:ee806600
[ 4936.261627] r5:c07b8e4c
[ 4936.261657] r4:00000001
[ 4936.261657] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[ 4936.261657] r4:eefb5100
[ 4936.261688] r3:c0015594
[ 4936.261688] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[ 4936.261688] r8:ee808400
[ 4936.261688] r7:00000001
[ 4936.261718] r6:00000000
[ 4936.261718] r5:00000000
[ 4936.261718] r4:0000001d
[ 4936.261718] r3:c00790dc
[ 4936.261718] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[ 4936.261718] r4:c0793cc0
[ 4936.261749] r3:000001ab
[ 4936.261749] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[ 4936.261749] r8:fa240100
[ 4936.261749] r7:c0797f20
[ 4936.261749] r6:c079e908
[ 4936.261779] r5:0000000d
[ 4936.261779] r4:fa24010c
[ 4936.261779] r3:c0797f20
[ 4936.261779] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[ 4936.261779] Exception stack(0xc0797f20 to 0xc0797f68)
[ 4936.261810] 7f20: eefb13c8 00000000 005d9656 c002cc04 c0796000 c079e488 c079e4d4 c059670c
[ 4936.261810] 7f40: c07d6bd9 c07d6bd9 c0789aac c0797f74 c0797f68 c0797f68 c002c814 c000fdcc
[ 4936.261810] 7f60: a00f0113 ffffffff
[ 4936.261840] r8:c07d6bd9
[ 4936.261840] r7:c0797f54
[ 4936.261840] r6:ffffffff
[ 4936.261840] r5:a00f0113
[ 4936.261840] r4:c000fdcc
[ 4936.261840] r3:c002c814
[ 4936.261871] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[ 4936.261871] [<c006ca7c>] (cpu_startup_entry) from [<c0589140>] (rest_init+0x8c/0x90)
[ 4936.261901] r7:ffffffff
[ 4936.261901] r3:c0596af8
[ 4936.261932] [<c05890b4>] (rest_init) from [<c0755cb4>] (start_kernel+0x368/0x374)
[ 4936.261932] r4:c079e598
[ 4936.261932] r3:00000001
[ 4936.261932] [<c075594c>] (start_kernel) from [<80008074>] (0x80008074)
[ 4936.261932] Task dump for CPU 1:
[ 4936.261962] swapper/1 R
[ 4936.261962] running
[ 4936.261962] 0 0 1 0x00000000
[ 4936.261962] Backtrace:
[ 4936.261962] [<ee89ffb0>] (0xee89ffb0) from [<00000000>] ( (null))
[ 4936.261962] Backtrace aborted due to bad frame pointer <c07d6bd9>
[ 4936.610534] {
[ 4936.612274] 1} (t=4097 jiffies g=15318 c=15317 q=198)
[ 4936.612274] Task dump for CPU 1:
[ 4936.619750] swapper/1 R running 0 0 1 0x00000000
[ 4936.624298] Backtrace:
[ 4936.628967] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[ 4936.633575] r6:800c0193 r5:00000001 r4:ee876f80 r3:00000000
[ 4936.636871] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[ 4936.643554] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[ 4936.651275] r5:c07b1240 r4:00000001
[ 4936.662902] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[ 4936.671539] r4:00000002 r3:00000001
[ 4936.671539] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[ 4936.684631] r8:000000c6 r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefba3d8 r3:00010001
[ 4936.692749] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[ 4936.700988] r10:00000001 r9:eefba5d0 r8:eefba5d0 r7:00000001 r6:00000000 r5:ee876f80
[ 4936.708129] r4:ee89e000
[ 4936.711303] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[ 4936.711303] r7:0000047d r6:44920ad0 r5:ee89ff68 r4:eefba858
[ 4936.722686] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[ 4936.722686] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[ 4936.747131] r7:c0093510 r6:eefba5d0 r5:eefba608 r4:eefba858
[ 4936.747131] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[ 4936.762420] r7:0000047d r6:44920ad0 r5:00000000 r4:eefba5d0
[ 4936.762420] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[ 4936.776611] r10:00000000 r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[ 4936.784820] r4:00000001
[ 4936.784820] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[ 4936.784820] r4:eefbe100 r3:c0015594
[ 4936.797760] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[ 4936.809478] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[ 4936.813110] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[ 4936.813110] r4:c0793cc0 r3:000001ab
[ 4936.830413] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[ 4936.831298] r8:fa240100 r7:ee89ff68 r6:c079e908 r5:0000000d r4:fa24010c r3:ee89ff68
[ 4936.843444] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[ 4936.847259] Exception stack(0xee89ff68 to 0xee89ffb0)
[ 4936.855072] ff60: eefba3c8 00000000 00472100 c002cc04 ee89e000 c079e488
[ 4936.865539] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[ 4936.871307] ffa0: c002c814 c000fdcc a00c0113 ffffffff
[ 4936.881561] r8:c07d6bd9 r7:ee89ff9c r6:ffffffff r5:a00c0113 r4:c000fdcc r3:c002c814
[ 4936.881561] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[ 4936.891418] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[ 4936.891418] r7:c07d7080 r3:00000001
[ 4936.911437] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[ 4936.911437] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
[78058.416564] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 0, t=2193 jiffies, g=242834, c=242833, q=202)
[78058.428131] INFO: Stall ended before state dump start
[81672.306396] INFO: rcu_preempt detected stalls on CPUs/tasks: {} (detected by 0, t=3467 jiffies, g=253949, c=253948, q=75)
[81672.314971] INFO: Stall ended before state dump start
[97165.336303] INFO: rcu_preempt self-detected stall on CPU
[97165.336303] INFO: rcu_preempt self-detected stall on CPU
[97165.336303] {
[97165.336334] 1
[97165.336334] }
[97165.336334] (t=3580 jiffies g=302147 c=302146 q=200)
[97165.336334] Task dump for CPU 0:
[97165.336334] swapper/0 R
[97165.336334] running
[97165.336334] 0 0 0 0x00000000
[97165.336364] Backtrace:
[97165.336364] [<c0797f68>] (0xc0797f68) from [<c0789aac>] (0xc0789aac)
[97165.336364] Backtrace aborted due to bad frame pointer <c07d6bd9>
[97165.336395] Task dump for CPU 1:
[97165.336395] swapper/1 R
[97165.336395] running
[97165.336395] 0 0 1 0x00000000
[97165.336395] Backtrace:
[97165.336425] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[97165.336425] r6:800c0193
[97165.336425] r5:00000001
[97165.336425] r4:ee876f80
[97165.336456] r3:00000000
[97165.336456] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[97165.336486] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[97165.336486] r5:c07b1240
[97165.336486] r4:00000001
[97165.336517] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[97165.336517] r4:00000002
[97165.336517] r3:00000001
[97165.336547] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[97165.336547] r8:000000c8
[97165.336547] r7:c079e798
[97165.336547] r6:c07b1240
[97165.336547] r5:c07933d8
[97165.336547] r4:eefba3d8
[97165.336547] r3:00010001
[97165.336578] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[97165.336578] r10:00000001
[97165.336578] r9:eefba5d0
[97165.336578] r8:eefba5d0
[97165.336578] r7:00000001
[97165.336608] r6:00000000
[97165.336608] r5:ee876f80
[97165.336608] r4:ee89e000
[97165.336608] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[97165.336639] r7:0000585e
[97165.336639] r6:f34faf9e
[97165.336639] r5:ee89ff68
[97165.336639] r4:eefba858
[97165.336639] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[97165.336669] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[97165.336669] r7:c0093510
[97165.336669] r6:eefba5d0
[97165.336669] r5:eefba608
[97165.336669] r4:eefba858
[97165.336700] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[97165.336700] r7:0000585e
[97165.336700] r6:f34faf9e
[97165.336700] r5:00000000
[97165.336700] r4:eefba5d0
[97165.336730] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[97165.336730] r10:00000000
[97165.336730] r9:c07d6bd9
[97165.336730] r8:0000001d
[97165.336730] r7:ee813b40
[97165.336730] r6:ee806600
[97165.336730] r5:c07b8e4c
[97165.336761] r4:00000001
[97165.336761] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[97165.336761] r4:eefbe100
[97165.336791] r3:c0015594
[97165.336791] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[97165.336791] r8:ee808400
[97165.336791] r7:00000001
[97165.336822] r6:00000000
[97165.336822] r5:00000000
[97165.336822] r4:0000001d
[97165.336822] r3:c00790dc
[97165.336822] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[97165.336822] r4:c0793cc0
[97165.336853] r3:000001ab
[97165.336853] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[97165.336853] r8:fa240100
[97165.336853] r7:ee89ff68
[97165.336853] r6:c079e908
[97165.336883] r5:0000000d
[97165.336883] r4:fa24010c
[97165.336883] r3:ee89ff68
[97165.336883] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[97165.336883] Exception stack(0xee89ff68 to 0xee89ffb0)
[97165.336914] ff60: eefba3c8 00000000 05c8c3c6 c002cc04 ee89e000 c079e488
[97165.336914] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[97165.336944] ffa0: c002c814 c000fdcc a00c0113 ffffffff
[97165.336944] r8:c07d6bd9
[97165.336944] r7:ee89ff9c
[97165.336944] r6:ffffffff
[97165.336944] r5:a00c0113
[97165.336944] r4:c000fdcc
[97165.336944] r3:c002c814
[97165.336975] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[97165.336975] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[97165.337005] r7:c07d7080
[97165.337005] r3:00000001
[97165.337005] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[97165.337005] r6:10c0387d
[97165.337005] r5:00000015
[97165.337036] r4:ae88406a
[97165.337036] r3:c00087ac
[97165.684082] {
[97165.684082] 0} (t=3615 jiffies g=302147 c=302146 q=151)
[97165.688079] Task dump for CPU 0:
[97165.688079] swapper/0 R running 0 0 0 0x00000002
[97165.702362] Backtrace:
[97165.702362] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[97165.702362] r6:80010193 r5:00000000 r4:c07a0920 r3:00000000
[97165.718780] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[97165.718780] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[97165.735137] r5:c07b1240 r4:00000000
[97165.735137] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[97165.735137] r4:00000001 r3:00000001
[97165.735137] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[97165.760620] r8:00000097 r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefb13d8 r3:00010001
[97165.760620] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[97165.760620] r10:00000001 r9:eefb15d0 r8:eefb15d0 r7:00000000 r6:00000000 r5:c07a0920
[97165.786193] r4:c0796000
[97165.786193] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[97165.786193] r7:0000585e r6:f34faf9e r5:c0797f20 r4:eefb1858
[97165.804595] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[97165.804595] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[97165.823120] r7:c0093510 r6:eefb15d0 r5:eefb1608 r4:eefb1858
[97165.823120] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[97165.823120] r7:0000585e r6:f34faf9e r5:00000000 r4:eefb15d0
[97165.838409] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[97165.852600] r10:c0789aac r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[97165.852600] r4:00000001
[97165.852600] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[97165.863464] r4:eefb5100 r3:c0015594
[97165.876007] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[97165.876007] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[97165.876007] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[97165.902648] r4:c0793cc0 r3:000001ab
[97165.906402] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[97165.906402] r8:fa240100 r7:c0797f20 r6:c079e908 r5:0000000d r4:fa24010c r3:c0797f20
[97165.906402] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[97165.906402] Exception stack(0xc0797f20 to 0xc0797f68)
[97165.906402] 7f20: eefb13c8 00000000 06a51e4a c002cc04 c0796000 c079e488 c079e4d4 c059670c
[97165.944854] 7f40: c07d6bd9 c07d6bd9 c0789aac c0797f74 c0797f68 c0797f68 c002c814 c000fdcc
[97165.953399] 7f60: a0010113 ffffffff
[97165.953399] r8:c07d6bd9 r7:c0797f54 r6:ffffffff r5:a0010113 r4:c000fdcc r3:c002c814
[97165.953399] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[97165.953399] [<c006ca7c>] (cpu_startup_entry) from [<c0589140>] (rest_init+0x8c/0x90)
[97165.981872] r7:ffffffff r3:c0596af8
[97165.981872] [<c05890b4>] (rest_init) from [<c0755cb4>] (start_kernel+0x368/0x374)
[97165.993438] r4:c079e598 r3:00000001
[97165.997192] [<c075594c>] (start_kernel) from [<80008074>] (0x80008074)
[98782.620971] INFO: rcu_preempt self-detected stall on CPU
[98782.620971] INFO: rcu_preempt self-detected stall on CPU
[98782.620971] {
[98782.620971] 0
[98782.620971] }
[98782.620971] (t=5467 jiffies g=307061 c=307060 q=200)
[98782.620971] Task dump for CPU 0:
[98782.620971] swapper/0 R
[98782.620971] running
[98782.621032] 0 0 0 0x00000000
[98782.621032] Backtrace:
[98782.621032] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[98782.621032] r6:800c0193
[98782.621032] r5:00000000
[98782.621032] r4:c07a0920
[98782.621032] r3:00000000
[98782.621093] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[98782.621124] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[98782.621124] r5:c07b1240
[98782.621124] r4:00000000
[98782.621154] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[98782.621154] r4:00000001
[98782.621154] r3:00000003
[98782.621154] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[98782.621185] r8:000000c8
[98782.621185] r7:c079e798
[98782.621185] r6:c07b1240
[98782.621185] r5:c07933d8
[98782.621185] r4:eefb13d8
[98782.621185] r3:00010001
[98782.621215] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[98782.621215] r10:00000001
[98782.621215] r9:eefb15d0
[98782.621215] r8:eefb15d0
[98782.621215] r7:00000000
[98782.621215] r6:00000000
[98782.621246] r5:c07a0920
[98782.621246] r4:c0796000
[98782.621246] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[98782.621246] r7:000059d7
[98782.621276] r6:8194b7f3
[98782.621276] r5:c0797f20
[98782.621276] r4:eefb1858
[98782.621276] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[98782.621307] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[98782.621307] r7:c0093510
[98782.621307] r6:eefb15d0
[98782.621307] r5:eefb1608
[98782.621307] r4:eefb1858
[98782.621337] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[98782.621337] r7:000059d7
[98782.621337] r6:8194b7f3
[98782.621337] r5:00000000
[98782.621337] r4:eefb15d0
[98782.621368] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[98782.621368] r10:c0789aac
[98782.621368] r9:c07d6bd9
[98782.621368] r8:0000001d
[98782.621368] r7:ee813b40
[98782.621368] r6:ee806600
[98782.621368] r5:c07b8e4c
[98782.621398] r4:00000001
[98782.621398] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[98782.621398] r4:eefb5100
[98782.621429] r3:c0015594
[98782.621429] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[98782.621429] r8:ee808400
[98782.621429] r7:00000001
[98782.621429] r6:00000000
[98782.621459] r5:00000000
[98782.621459] r4:0000001d
[98782.621459] r3:c00790dc
[98782.621459] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[98782.621459] r4:c0793cc0
[98782.621490] r3:000001ab
[98782.621490] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[98782.621490] r8:fa240100
[98782.621490] r7:c0797f20
[98782.621490] r6:c079e908
[98782.621490] r5:0000000d
[98782.621520] r4:fa24010c
[98782.621520] r3:c0797f20
[98782.621520] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[98782.621520] Exception stack(0xc0797f20 to 0xc0797f68)
[98782.621551] 7f20: eefb13c8 00000000 06c222a4 c002cc04 c0796000 c079e488 c079e4d4 c059670c
[98782.621551] 7f40: c07d6bd9 c07d6bd9 c0789aac c0797f74 c0797f68 c0797f68 c002c814 c000fdcc
[98782.621551] 7f60: a00c0113 ffffffff
[98782.621582] r8:c07d6bd9
[98782.621582] r7:c0797f54
[98782.621582] r6:ffffffff
[98782.621582] r5:a00c0113
[98782.621582] r4:c000fdcc
[98782.621582] r3:c002c814
[98782.621612] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[98782.621612] [<c006ca7c>] (cpu_startup_entry) from [<c0589140>] (rest_init+0x8c/0x90)
[98782.621643] r7:ffffffff
[98782.621643] r3:c0596af8
[98782.621643] [<c05890b4>] (rest_init) from [<c0755cb4>] (start_kernel+0x368/0x374)
[98782.621673] r4:c079e598
[98782.621673] r3:00000001
[98782.621673] [<c075594c>] (start_kernel) from [<80008074>] (0x80008074)
[98782.621673] Task dump for CPU 1:
[98782.621704] swapper/1 R
[98782.621704] running
[98782.621704] 0 0 1 0x00000000
[98782.621704] Backtrace:
[98782.621704] [<ee89ffb0>] (0xee89ffb0) from [<00000000>] ( (null))
[98782.621704] Backtrace aborted due to bad frame pointer <c07d6bd9>
[98782.972595] {
[98782.972595] 1} (t=5502 jiffies g=307061 c=307060 q=202)
[98782.975616] Task dump for CPU 1:
[98782.975616] swapper/1 R running 0 0 1 0x00000000
[98782.985382] Backtrace:
[98782.985382] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[98782.991210] r6:80070193 r5:00000001 r4:ee876f80 r3:00000000
[98783.004669] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[98783.013153] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[98783.015075] r5:c07b1240 r4:00000001
[98783.024139] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[98783.033782] r4:00000002 r3:00000001
[98783.035369] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[98783.035369] r8:000000ca r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefba3d8 r3:00010001
[98783.052856] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[98783.064117] r10:00000001 r9:eefba5d0 r8:eefba5d0 r7:00000001 r6:00000000 r5:ee876f80
[98783.064117] r4:ee89e000
[98783.075134] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[98783.075744] r7:000059d7 r6:8194b7f3 r5:ee89ff68 r4:eefba858
[98783.075744] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[98783.100311] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[98783.107849] r7:c0093510 r6:eefba5d0 r5:eefba608 r4:eefba858
[98783.115325] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[98783.115325] r7:000059d7 r6:8194b7f3 r5:00000000 r4:eefba5d0
[98783.130615] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[98783.134094] r10:00000000 r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[98783.144104] r4:00000001
[98783.149322] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[98783.158538] r4:eefbe100 r3:c0015594
[98783.158538] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[98783.158538] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[98783.175201] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[98783.188934] r4:c0793cc0 r3:000001ab
[98783.188934] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[98783.201385] r8:fa240100 r7:ee89ff68 r6:c079e908 r5:0000000d r4:fa24010c r3:ee89ff68
[98783.204101] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[98783.204101] Exception stack(0xee89ff68 to 0xee89ffb0)
[98783.222595] ff60: eefba3c8 00000000 05dfdc5e c002cc04 ee89e000 c079e488
[98783.231140] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[98783.239379] ffa0: c002c814 c000fdcc a0070113 ffffffff
[98783.239379] r8:c07d6bd9 r7:ee89ff9c r6:ffffffff r5:a0070113 r4:c000fdcc r3:c002c814
[98783.253082] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[98783.261718] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[98783.271148] r7:c07d7080 r3:00000001
[98783.274108] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[98783.282623] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
[126838.518463] INFO: rcu_preempt self-detected stall on CPU
[126838.519836] INFO: rcu_preempt detected stalls on CPUs/tasks:
[126838.519836] {
[126838.519836] 1
[126838.519836] }
[126838.519866] (detected by 0, t=2922 jiffies, g=394252, c=394251, q=199)
[126838.519866] Task dump for CPU 1:
[126838.519866] swapper/1 R
[126838.519866] running
[126838.519866] 0 0 1 0x00000000
[126838.519866] Backtrace:
[126838.519897] [<ee89ffb0>] (0xee89ffb0) from [<00000000>] ( (null))
[126838.519897] Backtrace aborted due to bad frame pointer <c07d6bd9>
[126838.561553] {
[126838.562957] 1} (t=2922 jiffies g=394252 c=394251 q=201)
[126838.567596] Task dump for CPU 1:
[126838.571136] swapper/1 R running 0 0 1 0x00000000
[126838.577148] Backtrace:
[126838.580535] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[126838.588531] r6:800c0193 r5:00000001 r4:ee876f80 r3:00000000
[126838.594573] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[126838.602264] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[126838.610992] r5:c07b1240 r4:00000001
[126838.614929] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[126838.623657] r4:00000002 r3:00000001
[126838.627410] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[126838.636962] r8:000000c9 r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefba3d8 r3:00010001
[126838.644317] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[126838.654541] r10:00000001 r9:eefba5d0 r8:eefba5d0 r7:00000001 r6:00000000 r5:ee876f80
[126838.662689] r4:ee89e000
[126838.664672] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[126838.675109] r7:0000735b r6:c09a155e r5:ee89ff68 r4:eefba858
[126838.681182] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[126838.691040] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[126838.700195] r7:c0093510 r6:eefba5d0 r5:eefba608 r4:eefba858
[126838.706237] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[126838.714904] r7:0000735b r6:c09a155e r5:00000000 r4:eefba5d0
[126838.721710] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[126838.730072] r10:00000000 r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[126838.730072] r4:00000001
[126838.739196] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[126838.749999] r4:eefbe100 r3:c0015594
[126838.753845] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[126838.753845] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[126838.769195] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[126838.780761] r4:c0793cc0 r3:000001ab
[126838.780761] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[126838.789215] r8:fa240100 r7:ee89ff68 r6:c079e908 r5:0000000d r4:fa24010c r3:ee89ff68
[126838.799224] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[126838.809204] Exception stack(0xee89ff68 to 0xee89ffb0)
[126838.814880] ff60: eefba3c8 00000000 07762036 c002cc04 ee89e000 c079e488
[126838.823516] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[126838.829223] ffa0: c002c814 c000fdcc a00c0113 ffffffff
[126838.829223] r8:c07d6bd9 r7:ee89ff9c r6:ffffffff r5:a00c0113 r4:c000fdcc r3:c002c814
[126838.845733] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[126838.849212] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[126838.863983] r7:c07d7080 r3:00000001
[126838.867828] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[126838.870574] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
[132963.666503] INFO: rcu_preempt self-detected stall on CPU
[132963.666687] INFO: rcu_preempt self-detected stall on CPU
[132963.666687] {
[132963.666687] 0
[132963.666687] }
[132963.666717] (t=2547 jiffies g=413073 c=413072 q=201)
[132963.666717] Task dump for CPU 0:
[132963.666717] smsd R
[132963.666717] running
[132963.666717] 0 967 1 0x00000000
[132963.666717] Backtrace:
[132963.666748] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[132963.666778] r6:80010193
[132963.666778] r5:00000001
[132963.666778] r4:ee154e40
[132963.666778] r3:00000000
[132963.666809] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[132963.666809] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[132963.666809] r5:c07b1240
[132963.666839] r4:00000000
[132963.666839] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[132963.666839] r4:00000001
[132963.666839] r3:00000003
[132963.666870] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[132963.666870] r8:000000c9
[132963.666870] r7:c079e798
[132963.666870] r6:c07b1240
[132963.666900] r5:c07933d8
[132963.666900] r4:eefb13d8
[132963.666900] r3:00010000
[132963.666900] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[132963.666931] r10:00000001
[132963.666931] r9:eefb15d0
[132963.666931] r8:eefb15d0
[132963.666931] r7:00000000
[132963.666931] r6:00000000
[132963.666931] r5:ee154e40
[132963.666931] r4:ee1e0000
[132963.666961] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[132963.666961] r7:000078ed
[132963.666961] r6:dec49a9e
[132963.666961] r5:ee1e1f08
[132963.666961] r4:eefb1858
[132963.666992] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[132963.666992] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[132963.666992] r7:c0093510
[132963.667022] r6:eefb15d0
[132963.667022] r5:eefb1608
[132963.667022] r4:eefb1858
[132963.667022] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[132963.667022] r7:000078ed
[132963.667022] r6:dec49a9e
[132963.667053] r5:00000000
[132963.667053] r4:eefb15d0
[132963.667053] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[132963.667053] r10:00000000
[132963.667083] r9:ee1e0000
[132963.667083] r8:0000001d
[132963.667083] r7:ee813b40
[132963.667083] r6:ee806600
[132963.667083] r5:c07b8e4c
[132963.667083] r4:00000001
[132963.667114] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[132963.667114] r4:eefb5100
[132963.667114] r3:c0015594
[132963.667144] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[132963.667144] r8:ee808400
[132963.667144] r7:00000001
[132963.667144] r6:00000000
[132963.667144] r5:00000000
[132963.667144] r4:0000001d
[132963.667175] r3:c00790dc
[132963.667175] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[132963.667175] r4:c0793cc0
[132963.667175] r3:000001ab
[132963.667205] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[132963.667205] r8:fa240100
[132963.667205] r7:ee1e1f08
[132963.667205] r6:c079e908
[132963.667205] r5:0000000d
[132963.667205] r4:fa24010c
[132963.667205] r3:ee1e1f08
[132963.667236] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[132963.667236] Exception stack(0xee1e1f08 to 0xee1e1f50)
[132963.667236] 1f00: 00000000 00041e48 00000100 00020000 00020000 ffffff9c
[132963.667266] 1f20: 00000008 00000005 c000ed48 ee1e0000 00000000 ee1e1f94 00000000 ee1e1f50
[132963.667266] 1f40: c00f3a4c c00f3910 60010113 ffffffff
[132963.667266] r8:c000ed48
[132963.667266] r7:ee1e1f3c
[132963.667266] r6:ffffffff
[132963.667297] r5:60010113
[132963.667297] r4:c00f3910
[132963.667297] r3:c00f3a4c
[132963.667297] [<c00f3858>] (do_sys_open) from [<c00f3a4c>] (SyS_open+0x24/0x28)
[132963.667297] r9:ee1e0000
[132963.667327] r8:c000ed48
[132963.667327] r7:00000005
[132963.667327] r6:00000008
[132963.667327] r5:00000000
[132963.667327] r4:000683d0
[132963.667327] [<c00f3a28>] (SyS_open) from [<c000eb80>] (ret_fast_syscall+0x0/0x30)
[132963.667358] Task dump for CPU 1:
[132963.667358] swapper/1 R
[132963.667358] running
[132963.667358] 0 0 1 0x00000000
[132963.667358] Backtrace:
[132963.667388] [<ee89ffb0>] (0xee89ffb0) from [<00000000>] ( (null))
[132963.667388] Backtrace aborted due to bad frame pointer <c07d6bd9>
[132964.008789] {
[132964.010620] 1} (t=2547 jiffies g=413073 c=413072 q=203)
[132964.010620] Task dump for CPU 1:
[132964.018371] swapper/1 R running 0 0 1 0x00000000
[132964.018371] Backtrace:
[132964.026702] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[132964.026702] r6:80070193 r5:00000001 r4:ee876f80 r3:00000000
[132964.026702] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[132964.049987] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[132964.056701] r5:c07b1240 r4:00000001
[132964.056701] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[132964.070007] r4:00000002 r3:00000001
[132964.074737] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[132964.082550] r8:000000cb r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefba3d8 r3:00010001
[132964.092376] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[132964.092773] r10:00000001 r9:eefba5d0 r8:eefba5d0 r7:00000001 r6:00000000 r5:ee876f80
[132964.110015] r4:ee89e000
[132964.112335] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[132964.121032] r7:000078ed r6:de809e2f r5:ee89ff68 r4:eefba858
[132964.128692] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[132964.132354] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[132964.146728] r7:c0093510 r6:eefba5d0 r5:eefba608 r4:eefba858
[132964.146728] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[132964.156707] r7:000078ed r6:de809e2f r5:00000000 r4:eefba5d0
[132964.156707] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[132964.177246] r10:00000000 r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[132964.177246] r4:00000001
[132964.186706] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[132964.197174] r4:eefbe100 r3:c0015594
[132964.201019] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[132964.201019] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[132964.212371] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[132964.227935] r4:c0793cc0 r3:000001ab
[132964.227935] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[132964.240570] r8:fa240100 r7:ee89ff68 r6:c079e908 r5:0000000d r4:fa24010c r3:ee89ff68
[132964.246704] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[132964.246704] Exception stack(0xee89ff68 to 0xee89ffb0)
[132964.246704] ff60: eefba3c8 00000000 07c61ea4 c002cc04 ee89e000 c079e488
[132964.270690] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[132964.270690] ffa0: c002c814 c000fdcc a0070113 ffffffff
[132964.284667] r8:c07d6bd9 r7:ee89ff9c r6:ffffffff r5:a0070113 r4:c000fdcc r3:c002c814
[132964.288726] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[132964.298706] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[132964.311157] r7:c07d7080 r3:00000001
[132964.311157] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[132964.311157] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
[144021.631317] INFO: rcu_preempt self-detected stall on CPU
[144021.631317] INFO: rcu_preempt self-detected stall on CPU
[144021.631317] {
[144021.631347] 1
[144021.631347] }
[144021.631347] (t=3658 jiffies g=447167 c=447166 q=1)
[144021.631347] Task dump for CPU 0:
[144021.631347] swapper/0 R
[144021.631347] running
[144021.631347] 0 0 0 0x00000000
[144021.631378] Backtrace:
[144021.631378] [<c0797f68>] (0xc0797f68) from [<c0789aac>] (0xc0789aac)
[144021.631408] Backtrace aborted due to bad frame pointer <c07d6bd9>
[144021.631408] Task dump for CPU 1:
[144021.631408] swapper/1 R
[144021.631408] running
[144021.631408] 0 0 1 0x00000000
[144021.631408] Backtrace:
[144021.631408] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[144021.631408] r6:80010193
[144021.631408] r5:00000001
[144021.631408] r4:ee876f80
[144021.631469] r3:00000000
[144021.631469] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[144021.631500] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[144021.631500] r5:c07b1240
[144021.631500] r4:00000001
[144021.631530] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[144021.631530] r4:00000002
[144021.631530] r3:00000001
[144021.631561] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[144021.631561] r8:00000001
[144021.631561] r7:c079e798
[144021.631561] r6:c07b1240
[144021.631561] r5:c07933d8
[144021.631561] r4:eefba3d8
[144021.631561] r3:00010001
[144021.631591] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[144021.631591] r10:00000001
[144021.631591] r9:eefba5d0
[144021.631591] r8:eefba5d0
[144021.631622] r7:00000001
[144021.631622] r6:00000000
[144021.631622] r5:ee876f80
[144021.631622] r4:ee89e000
[144021.631652] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[144021.631652] r7:000082fc
[144021.631652] r6:7d289c3b
[144021.631652] r5:ee89ff68
[144021.631652] r4:eefba858
[144021.631683] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[144021.631683] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[144021.631683] r7:c0093510
[144021.631683] r6:eefba5d0
[144021.631683] r5:eefba608
[144021.631683] r4:eefba858
[144021.631713] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[144021.631713] r7:000082fc
[144021.631713] r6:7d289c3b
[144021.631713] r5:00000000
[144021.631713] r4:eefba5d0
[144021.631744] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[144021.631744] r10:00000000
[144021.631744] r9:c07d6bd9
[144021.631744] r8:0000001d
[144021.631744] r7:ee813b40
[144021.631774] r6:ee806600
[144021.631774] r5:c07b8e4c
[144021.631774] r4:00000001
[144021.631774] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[144021.631805] r4:eefbe100
[144021.631805] r3:c0015594
[144021.631805] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[144021.631805] r8:ee808400
[144021.631835] r7:00000001
[144021.631835] r6:00000000
[144021.631835] r5:00000000
[144021.631835] r4:0000001d
[144021.631835] r3:c00790dc
[144021.631835] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[144021.631866] r4:c0793cc0
[144021.631866] r3:000001ab
[144021.631866] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[144021.631866] r8:fa240100
[144021.631866] r7:ee89ff68
[144021.631896] r6:c079e908
[144021.631896] r5:0000000d
[144021.631896] r4:fa24010c
[144021.631896] r3:ee89ff68
[144021.631896] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[144021.631927] Exception stack(0xee89ff68 to 0xee89ffb0)
[144021.631927] ff60: eefba3c8 00000000 0870d82e c002cc04 ee89e000 c079e488
[144021.631927] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[144021.631958] ffa0: c002c814 c000fdcc a0010113 ffffffff
[144021.631958] r8:c07d6bd9
[144021.631958] r7:ee89ff9c
[144021.631958] r6:ffffffff
[144021.631958] r5:a0010113
[144021.631958] r4:c000fdcc
[144021.631958] r3:c002c814
[144021.631958] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[144021.632019] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[144021.632019] r7:c07d7080
[144021.632019] r3:00000001
[144021.632019] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[144021.632019] r6:10c0387d
[144021.632049] r5:00000015
[144021.632049] r4:ae88406a
[144021.632049] r3:c00087ac
[144021.985839] {
[144021.985839] 0} (t=3693 jiffies g=447167 c=447166 q=1)
[144021.988128] Task dump for CPU 0:
[144021.988128] swapper/0 R running 0 0 0 0x00000000
[144022.001007] Backtrace:
[144022.001007] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[144022.008148] r6:80070193 r5:00000000 r4:c07a0920 r3:00000000
[144022.018646] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[144022.018646] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[144022.035186] r5:c07b1240 r4:00000000
[144022.038635] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[144022.038635] r4:00000001 r3:00000001
[144022.048645] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[144022.058135] r8:00000001 r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefb13d8 r3:00010001
[144022.069244] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[144022.069244] r10:00000001 r9:eefb15d0 r8:eefb15d0 r7:00000000 r6:00000000 r5:c07a0920
[144022.086883] r4:c0796000
[144022.089630] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[144022.098114] r7:000082fc r6:7d289c3b r5:c0797f20 r4:eefb1858
[144022.098114] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[144022.110687] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[144022.120666] r7:c0093510 r6:eefb15d0 r5:eefb1608 r4:eefb1858
[144022.130279] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[144022.135559] r7:000082fc r6:7d289c3b r5:00000000 r4:eefb15d0
[144022.143249] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[144022.143249] r10:c0789aac r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[144022.143249] r4:00000001
[144022.143249] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[144022.174041] r4:eefb5100 r3:c0015594
[144022.174041] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[144022.187408] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[144022.188140] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[144022.188140] r4:c0793cc0 r3:000001ab
[144022.208618] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[144022.217437] r8:fa240100 r7:c0797f20 r6:c079e908 r5:0000000d r4:fa24010c r3:c0797f20
[144022.218811] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[144022.218811] Exception stack(0xc0797f20 to 0xc0797f68)
[144022.233551] 7f20: eefb13c8 00000000 09f78412 c002cc04 c0796000 c079e488 c079e4d4 c059670c
[144022.242004] 7f40: c07d6bd9 c07d6bd9 c0789aac c0797f74 c0797f68 c0797f68 c002c814 c000fdcc
[144022.256164] 7f60: a0070113 ffffffff
[144022.258148] r8:c07d6bd9 r7:c0797f54 r6:ffffffff r5:a0070113 r4:c000fdcc r3:c002c814
[144022.258148] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[144022.268127] [<c006ca7c>] (cpu_startup_entry) from [<c0589140>] (rest_init+0x8c/0x90)
[144022.268127] r7:ffffffff r3:c0596af8
[144022.268127] [<c05890b4>] (rest_init) from [<c0755cb4>] (start_kernel+0x368/0x374)
[144022.268127] r4:c079e598 r3:00000001
[144022.298095] [<c075594c>] (start_kernel) from [<80008074>] (0x80008074)
[196894.452148] INFO: rcu_preempt self-detected stall on CPU { 0} (t=2510 jiffies g=611096 c=611095 q=199)
[196894.459533] INFO: rcu_preempt self-detected stall on CPU { 1} (t=2510 jiffies g=611096 c=611095 q=199)
[196894.459533] Task dump for CPU 0:
[196894.459564] swapper/0 R running 0 0 0 0x00000000
[196894.459564] Backtrace:
[196894.459594] [<c0797f68>] (0xc0797f68) from [<c0789aac>] (0xc0789aac)
[196894.459594] Backtrace aborted due to bad frame pointer <c07d6bd9>
[196894.459594] Task dump for CPU 1:
[196894.459594] swapper/1 R running 0 0 1 0x00000000
[196894.459625] Backtrace:
[196894.459625] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[196894.459655] r6:80000193 r5:00000001 r4:ee876f80 r3:00000000
[196894.459686] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[196894.459686] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[196894.459716] r5:c07b1240 r4:00000001
[196894.459716] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[196894.459716] r4:00000002 r3:00000001
[196894.459747] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[196894.459777] r8:000000c7 r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefba3d8 r3:00010001
[196894.459777] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[196894.459808] r10:00000001 r9:eefba5d0 r8:eefba5d0 r7:00000001 r6:00000000 r5:ee876f80
[196894.459808] r4:ee89e000
[196894.459838] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[196894.459838] r7:0000b312 r6:e0a9f816 r5:ee89ff68 r4:eefba858
[196894.459838] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[196894.459869] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[196894.459869] r7:c0093510 r6:eefba5d0 r5:eefba608 r4:eefba858
[196894.459899] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[196894.459899] r7:0000b312 r6:e0a9f816 r5:00000000 r4:eefba5d0
[196894.459930] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[196894.459930] r10:00000000 r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[196894.459930] r4:00000001
[196894.459960] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[196894.459960] r4:eefbe100 r3:c0015594
[196894.459991] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[196894.459991] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[196894.460021] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[196894.460021] r4:c0793cc0 r3:000001ab
[196894.460021] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[196894.460052] r8:fa240100 r7:ee89ff68 r6:c079e908 r5:0000000d r4:fa24010c r3:ee89ff68
[196894.460052] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[196894.460083] Exception stack(0xee89ff68 to 0xee89ffb0)
[196894.460083] ff60: eefba3c8 00000000 0b7e9e2c c002cc04 ee89e000 c079e488
[196894.460083] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[196894.460113] ffa0: c002c814 c000fdcc a0000113 ffffffff
[196894.460113] r8:c07d6bd9 r7:ee89ff9c r6:ffffffff r5:a0000113 r4:c000fdcc r3:c002c814
[196894.460144] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[196894.460144] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[196894.460174] r7:c07d7080 r3:00000001
[196894.460174] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[196894.460174] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
[196894.809600] Task dump for CPU 0:
[196894.809600] swapper/0 R running 0 0 0 0x00000000
[196894.814483] Backtrace:
[196894.814483] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[196894.824584] r6:80070193 r5:00000000 r4:c07a0920 r3:00000000
[196894.837890] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[196894.842803] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[196894.854431] r5:c07b1240 r4:00000000
[196894.856262] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[196894.856262] r4:00000001 r3:00000001
[196894.870819] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[196894.879730] r8:000000c7 r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefb13d8 r3:00010001
[196894.879730] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[196894.897827] r10:00000001 r9:eefb15d0 r8:eefb15d0 r7:00000000 r6:00000000 r5:c07a0920
[196894.906127] r4:c0796000
[196894.906127] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[196894.910186] r7:0000b312 r6:e038d0c8 r5:c0797f20 r4:eefb1858
[196894.910186] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[196894.934326] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[196894.939544] r7:c0093510 r6:eefb15d0 r5:eefb1608 r4:eefb1858
[196894.949523] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[196894.949554] r7:0000b312 r6:e038d0c8 r5:00000000 r4:eefb15d0
[196894.959533] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[196894.959533] r10:c0789aac r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[196894.979553] r4:00000001
[196894.979553] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[196894.979553] r4:eefb5100 r3:c0015594
[196894.979553] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[196894.999511] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[196895.009613] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[196895.019531] r4:c0793cc0 r3:000001ab
[196895.027862] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[196895.029541] r8:fa240100 r7:c0797f20 r6:c079e908 r5:0000000d r4:fa24010c r3:c0797f20
[196895.041198] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[196895.052764] Exception stack(0xc0797f20 to 0xc0797f68)
[196895.058135] 7f20: eefb13c8 00000000 0dab1b54 c002cc04 c0796000 c079e488 c079e4d4 c059670c
[196895.059539] 7f40: c07d6bd9 c07d6bd9 c0789aac c0797f74 c0797f68 c0797f68 c002c814 c000fdcc
[196895.059539] 7f60: a0070113 ffffffff
[196895.075408] r8:c07d6bd9 r7:c0797f54 r6:ffffffff r5:a0070113 r4:c000fdcc r3:c002c814
[196895.075408] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[196895.096069] [<c006ca7c>] (cpu_startup_entry) from [<c0589140>] (rest_init+0x8c/0x90)
[196895.104248] r7:ffffffff r3:c0596af8
[196895.106475] [<c05890b4>] (rest_init) from [<c0755cb4>] (start_kernel+0x368/0x374)
[196895.115631] r4:c079e598 r3:00000001
[196895.115997] [<c075594c>] (start_kernel) from [<80008074>] (0x80008074)
[231882.439544] INFO: rcu_preempt self-detected stall on CPU { 1} (t=2253 jiffies g=720205 c=720204 q=3)
[231882.449310] Task dump for CPU 0:
[231882.450836] INFO: rcu_preempt self-detected stall on CPU { 0} (t=2253 jiffies g=720205 c=720204 q=3)
[231882.450836] swapper/0 R running 0 0 0 0x00000000
[231882.469207] Backtrace:
[231882.469207] [<c0797f68>] (0xc0797f68) from [<c0789aac>] (0xc0789aac)
[231882.478607] Backtrace aborted due to bad frame pointer <c07d6bd9>
[231882.478607] Task dump for CPU 1:
[231882.478607] swapper/1 R running 0 0 1 0x00000002
[231882.495239] Backtrace:
[231882.495239] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[231882.495239] r6:800c0193 r5:00000001 r4:ee876f80 r3:00000000
[231882.511932] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[231882.511932] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[231882.528472] r5:c07b1240 r4:00000001
[231882.528472] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[231882.528472] r4:00000002 r3:00000001
[231882.541046] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[231882.554321] r8:00000003 r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefba3d8 r3:00010001
[231882.554321] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[231882.554321] r10:00000001 r9:eefba5d0 r8:eefba5d0 r7:00000001 r6:00000000 r5:ee876f80
[231882.580169] r4:ee89e000
[231882.580169] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[231882.580169] r7:0000d2e5 r6:1fd948c8 r5:ee89ff68 r4:eefba858
[231882.598846] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[231882.598846] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[231882.617553] r7:c0093510 r6:eefba5d0 r5:eefba608 r4:eefba858
[231882.617553] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[231882.617553] r7:0000d2e5 r6:1fd948c8 r5:00000000 r4:eefba5d0
[231882.633026] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[231882.647399] r10:00000000 r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[231882.647399] r4:00000001
[231882.658447] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[231882.658447] r4:eefbe100 r3:c0015594
[231882.671173] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[231882.671173] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[231882.671173] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[231882.698089] r4:c0793cc0 r3:000001ab
[231882.698089] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[231882.698089] r8:fa240100 r7:ee89ff68 r6:c079e908 r5:0000000d r4:fa24010c r3:ee89ff68
[231882.698089] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[231882.698089] Exception stack(0xee89ff68 to 0xee89ffb0)
[231882.698089] ff60: eefba3c8 00000000 0d80b2a4 c002cc04 ee89e000 c079e488
[231882.740844] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[231882.749481] ffa0: c002c814 c000fdcc a00c0113 ffffffff
[231882.749481] r8:c07d6bd9 r7:ee89ff9c r6:ffffffff r5:a00c0113 r4:c000fdcc r3:c002c814
[231882.763061] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[231882.763061] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[231882.763061] r7:c07d7080 r3:00000001
[231882.785156] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[231882.785156] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
[231882.785156] Task dump for CPU 0:
[231882.785156] swapper/0 R running 0 0 0 0x00000000
[231882.809173] Backtrace:
[231882.809173] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[231882.819824] r6:800d0193 r5:00000000 r4:c07a0920 r3:00000000
[231882.825866] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[231882.825866] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[231882.825866] r5:c07b1240 r4:00000000
[231882.825866] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[231882.854949] r4:00000001 r3:00000003
[231882.854949] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[231882.854949] r8:00000003 r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefb13d8 r3:00010001
[231882.854949] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[231882.854949] r10:00000001 r9:eefb15d0 r8:eefb15d0 r7:00000000 r6:00000000 r5:c07a0920
[231882.885803] r4:c0796000
[231882.885803] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[231882.885803] r7:0000d2e5 r6:20860af0 r5:c0797f20 r4:eefb1858
[231882.912750] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[231882.912750] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[231882.931457] r7:c0093510 r6:eefb15d0 r5:eefb1608 r4:eefb1858
[231882.931457] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[231882.946929] r7:0000d2e5 r6:20860af0 r5:00000000 r4:eefb15d0
[231882.946929] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[231882.961303] r10:c0789aac r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[231882.961303] r4:00000001
[231882.969604] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[231882.981231] r4:eefb5100 r3:c0015594
[231882.985076] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[231882.985076] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[231883.002807] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[231883.002807] r4:c0793cc0 r3:000001ab
[231883.011993] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[231883.024627] r8:fa240100 r7:c0797f20 r6:c079e908 r5:0000000d r4:fa24010c r3:c0797f20
[231883.024627] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[231883.024627] Exception stack(0xc0797f20 to 0xc0797f68)
[231883.024627] 7f20: eefb13c8 00000000 102205a4 c002cc04 c0796000 c079e488 c079e4d4 c059670c
[231883.046112] 7f40: c07d6bd9 c07d6bd9 c0789aac c0797f74 c0797f68 c0797f68 c002c814 c000fdcc
[231883.046112] 7f60: a00d0113 ffffffff
[231883.046112] r8:c07d6bd9 r7:c0797f54 r6:ffffffff r5:a00d0113 r4:c000fdcc r3:c002c814
[231883.046112] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[231883.084014] [<c006ca7c>] (cpu_startup_entry) from [<c0589140>] (rest_init+0x8c/0x90)
[231883.084045] r7:ffffffff r3:c0596af8
[231883.084045] [<c05890b4>] (rest_init) from [<c0755cb4>] (start_kernel+0x368/0x374)
[231883.084045] r4:c079e598 r3:00000001
[231883.107788] [<c075594c>] (start_kernel) from [<80008074>] (0x80008074)
[231883.107788] Task dump for CPU 1:
[231883.107788] swapper/1 R running 0 0 1 0x00000002
[231883.124877] Backtrace:
[231883.127532] [<ee89ffb0>] (0xee89ffb0) from [<00000000>] ( (null))
[231883.127532] Backtrace aborted due to bad frame pointer <c07d6bd9>
[249368.950958] wlcore: Beacon loss detected. roles:0xff
[249370.588195] wlcore: Beacon loss detected. roles:0xff
[249372.228973] wlcore: Beacon loss detected. roles:0xff
[249373.788665] wlcore: Connection loss work (role_id: 0).
[249373.794158] wlan0: Connection to AP 90:f6:52:81:ee:70 lost
[249373.907928] cfg80211: Calling CRDA to update world regulatory domain
[249376.796813] wlan0: authenticate with 90:f6:52:81:ee:70
[249376.828186] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[249376.836425] wlan0: authenticated
[249376.845977] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[249376.858032] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[249376.876220] wlan0: associated
[249376.919830] wlcore: Association completed.
[283161.271728] wlcore: Beacon loss detected. roles:0xff
[283163.345458] wlcore: Beacon loss detected. roles:0xff
[283165.989898] wlcore: Beacon loss detected. roles:0xff
[283166.101623] wlcore: Connection loss work (role_id: 0).
[283166.107116] wlan0: Connection to AP 90:f6:52:81:ee:70 lost
[283166.275543] cfg80211: Calling CRDA to update world regulatory domain
[283176.969390] wlan0: authenticate with 90:f6:52:81:ee:70
[283176.984771] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[283176.993011] wlan0: authenticated
[283176.997955] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[283177.011657] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[283177.029174] wlan0: associated
[283177.079742] wlcore: Association completed.
[292938.989196] INFO: rcu_preempt self-detected stall on CPU { 1} (t=4010 jiffies g=909942 c=909941 q=16)
[292938.999053] Task dump for CPU 1:
[292938.999053] swapper/1 R running 0 0 1 0x00000000
[292939.009246] Backtrace:
[292939.009246] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[292939.009246] r6:80070193 r5:00000001 r4:ee876f80 r3:00000000
[292939.009246] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[292939.009246] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[292939.009246] r5:c07b1240 r4:00000001
[292939.009246] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[292939.055053] r4:00000002 r3:00000001
[292939.055053] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[292939.055053] r8:00000010 r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefba3d8 r3:00010001
[292939.055053] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[292939.055053] r10:00000001 r9:eefba5d0 r8:eefba5d0 r7:00000001 r6:00000000 r5:ee876f80
[292939.085906] r4:ee89e000
[292939.085906] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[292939.085906] r7:00010a6c r6:e8afa689 r5:ee89ff68 r4:eefba858
[292939.106842] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[292939.106842] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[292939.106842] r7:c0093510 r6:eefba5d0 r5:eefba608 r4:eefba858
[292939.106842] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[292939.147033] r7:00010a6c r6:e8afa689 r5:00000000 r4:eefba5d0
[292939.147033] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[292939.161407] r10:00000000 r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[292939.169708] r4:00000001
[292939.169708] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[292939.169708] r4:eefbe100 r3:c0015594
[292939.169708] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[292939.194732] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[292939.194732] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[292939.194732] r4:c0793cc0 r3:000001ab
[292939.215942] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[292939.215942] r8:fa240100 r7:ee89ff68 r6:c079e908 r5:0000000d r4:fa24010c r3:ee89ff68
[292939.215942] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[292939.232971] Exception stack(0xee89ff68 to 0xee89ffb0)
[292939.240875] ff60: eefba3c8 00000000 10dfac9e c002cc04 ee89e000 c079e488
[292939.240875] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[292939.263488] ffa0: c002c814 c000fdcc a0070113 ffffffff
[292939.263488] r8:c07d6bd9 r7:ee89ff9c r6:ffffffff r5:a0070113 r4:c000fdcc r3:c002c814
[292939.263488] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[292939.263488] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[292939.263488] r7:c07d7080 r3:00000001
[292939.263488] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[292939.306976] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
[292939.306976] INFO: rcu_preempt detected stalls on CPUs/tasks: { 1} (detected by 0, t=4043 jiffies, g=909942, c=909941, q=14)
[292939.324127] Task dump for CPU 1:
[292939.328186] swapper/1 R running 0 0 1 0x00000000
[292939.334930] Backtrace:
[292939.337463] [<c007d9ac>] (rcu_eqs_enter_common.isra.49) from [<c007ecb4>] (rcu_idle_enter+0xa0/0xa4)
[292939.347229] r6:600f0193 r5:2e827000 r4:c07933c8
[292939.352081] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[292939.360900] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[292939.369781] r7:c07d7080 r3:00000001
[292939.374053] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[292939.376281] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
[314398.318481] wlcore: Beacon loss detected. roles:0xff
[314399.956848] wlcore: Beacon loss detected. roles:0xff
[314401.595275] wlcore: Beacon loss detected. roles:0xff
[314403.129943] wlcore: Connection loss work (role_id: 0).
[314403.135437] wlan0: Connection to AP 90:f6:52:81:ee:70 lost
[314403.247772] cfg80211: Calling CRDA to update world regulatory domain
[314403.652191] cfg80211: Calling CRDA to update world regulatory domain
[314406.188079] wlan0: authenticate with 90:f6:52:81:ee:70
[314406.203277] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[314406.211669] wlan0: authenticated
[314406.216369] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[314406.230743] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[314406.250244] wlan0: associated
[314406.287322] wlcore: Association completed.
[335868.421356] wlcore: Beacon loss detected. roles:0xff
[335870.059783] wlcore: Beacon loss detected. roles:0xff
[335873.257812] wlcore: Connection loss work (role_id: 0).
[335873.263366] wlan0: Connection to AP 90:f6:52:81:ee:70 lost
[335873.400665] cfg80211: Calling CRDA to update world regulatory domain
[335876.198364] wlan0: authenticate with 90:f6:52:81:ee:70
[335876.213378] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[335876.221832] wlan0: authenticated
[335876.230346] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[335876.243835] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[335876.263153] wlan0: associated
[335876.310363] wlcore: Association completed.
[479936.402893] INFO: rcu_preempt self-detected stall on CPU
[479936.402923] INFO: rcu_preempt self-detected stall on CPU
[479936.402923] {
[479936.402923] 0
[479936.402923] }
[479936.402923] (t=2118 jiffies g=1492226 c=1492225 q=199)
[479936.402954] Task dump for CPU 0:
[479936.402954] swapper/0 R
[479936.402954] running
[479936.402954] 0 0 0 0x00000000
[479936.402954] Backtrace:
[479936.402954] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[479936.402954] r6:800c0193
[479936.402954] r5:00000000
[479936.403015] r4:c07a0920
[479936.403015] r3:00000000
[479936.403015] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[479936.403015] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[479936.403015] r5:c07b1240
[479936.403015] r4:00000000
[479936.403076] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[479936.403076] r4:00000001
[479936.403076] r3:00000003
[479936.403106] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[479936.403106] r8:000000c7
[479936.403106] r7:c079e798
[479936.403106] r6:c07b1240
[479936.403106] r5:c07933d8
[479936.403106] r4:eefb13d8
[479936.403137] r3:00010001
[479936.403137] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[479936.403137] r10:00000001
[479936.403137] r9:eefb15d0
[479936.403167] r8:eefb15d0
[479936.403167] r7:00000000
[479936.403167] r6:00000000
[479936.403167] r5:c07a0920
[479936.403167] r4:c0796000
[479936.403198] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[479936.403198] r7:0001b47f
[479936.403198] r6:997a0207
[479936.403198] r5:c0797f20
[479936.403198] r4:eefb1858
[479936.403228] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[479936.403228] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[479936.403228] r7:c0093510
[479936.403228] r6:eefb15d0
[479936.403259] r5:eefb1608
[479936.403259] r4:eefb1858
[479936.403259] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[479936.403259] r7:0001b47f
[479936.403259] r6:997a0207
[479936.403259] r5:00000000
[479936.403289] r4:eefb15d0
[479936.403289] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[479936.403289] r10:c0789aac
[479936.403289] r9:c07d6bd9
[479936.403289] r8:0000001d
[479936.403320] r7:ee813b40
[479936.403320] r6:ee806600
[479936.403320] r5:c07b8e4c
[479936.403320] r4:00000001
[479936.403350] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[479936.403350] r4:eefb5100
[479936.403350] r3:c0015594
[479936.403381] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[479936.403381] r8:ee808400
[479936.403381] r7:00000001
[479936.403381] r6:00000000
[479936.403381] r5:00000000
[479936.403381] r4:0000001d
[479936.403381] r3:c00790dc
[479936.403411] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[479936.403411] r4:c0793cc0
[479936.403411] r3:000001ab
[479936.403411] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[479936.403411] r8:fa240100
[479936.403442] r7:c0797f20
[479936.403442] r6:c079e908
[479936.403442] r5:0000000d
[479936.403442] r4:fa24010c
[479936.403442] r3:c0797f20
[479936.403472] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[479936.403472] Exception stack(0xc0797f20 to 0xc0797f68)
[479936.403472] 7f20: eefb13c8 00000000 22262992 c002cc04 c0796000 c079e488 c079e4d4 c059670c
[479936.403503] 7f40: c07d6bd9 c07d6bd9 c0789aac c0797f74 c0797f68 c0797f68 c002c814 c000fdcc
[479936.403503] 7f60: a00c0113 ffffffff
[479936.403503] r8:c07d6bd9
[479936.403503] r7:c0797f54
[479936.403503] r6:ffffffff
[479936.403503] r5:a00c0113
[479936.403503] r4:c000fdcc
[479936.403533] r3:c002c814
[479936.403533] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[479936.403564] [<c006ca7c>] (cpu_startup_entry) from [<c0589140>] (rest_init+0x8c/0x90)
[479936.403564] r7:ffffffff
[479936.403564] r3:c0596af8
[479936.403594] [<c05890b4>] (rest_init) from [<c0755cb4>] (start_kernel+0x368/0x374)
[479936.403594] r4:c079e598
[479936.403594] r3:00000001
[479936.403625] [<c075594c>] (start_kernel) from [<80008074>] (0x80008074)
[479936.403625] Task dump for CPU 1:
[479936.403625] swapper/1 R
[479936.403625] running
[479936.403625] 0 0 1 0x00000000
[479936.403625] Backtrace:
[479936.403656] [<ee89ffb0>] (0xee89ffb0) from [<00000000>] ( (null))
[479936.403656] Backtrace aborted due to bad frame pointer <c07d6bd9>
[479936.759399] {
[479936.759399] 1} (t=2148 jiffies g=1492226 c=1492225 q=77)
[479936.759399] Task dump for CPU 1:
[479936.767730] swapper/1 R running 0 0 1 0x00000000
[479936.769073] Backtrace:
[479936.778472] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[479936.778472] r6:80070193 r5:00000001 r4:ee876f80 r3:00000000
[479936.787445] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[479936.797454] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[479936.809020] r5:c07b1240 r4:00000001
[479936.812866] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[479936.821441] r4:00000002 r3:00000001
[479936.821594] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[479936.834869] r8:0000004d r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefba3d8 r3:00010001
[479936.843078] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[479936.843078] r10:00000001 r9:eefba5d0 r8:eefba5d0 r7:00000001 r6:00000000 r5:ee876f80
[479936.860717] r4:ee89e000
[479936.860717] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[479936.860717] r7:0001b47f r6:997a0207 r5:ee89ff68 r4:eefba858
[479936.875457] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[479936.879394] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[479936.895507] r7:c0093510 r6:eefba5d0 r5:eefba608 r4:eefba858
[479936.904113] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[479936.911346] r7:0001b47f r6:997a0207 r5:00000000 r4:eefba5d0
[479936.919586] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[479936.927947] r10:00000000 r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[479936.935302] r4:00000001
[479936.935302] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[479936.947448] r4:eefbe100 r3:c0015594
[479936.951721] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[479936.961242] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[479936.965820] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[479936.978454] r4:c0793cc0 r3:000001ab
[479936.978454] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[479936.991271] r8:fa240100 r7:ee89ff68 r6:c079e908 r5:0000000d r4:fa24010c r3:ee89ff68
[479936.997467] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[479936.997467] Exception stack(0xee89ff68 to 0xee89ffb0)
[479936.997467] ff60: eefba3c8 00000000 1b792502 c002cc04 ee89e000 c079e488
[479936.997467] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[479937.027435] ffa0: c002c814 c000fdcc a0070113 ffffffff
[479937.035369] r8:c07d6bd9 r7:ee89ff9c r6:ffffffff r5:a0070113 r4:c000fdcc r3:c002c814
[479937.043579] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[479937.052307] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[479937.057434] r7:c07d7080 r3:00000001
[479937.057434] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[479937.067443] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
[509722.778808] INFO: rcu_preempt self-detected stall on CPU
[509722.778808] INFO: rcu_preempt self-detected stall on CPU
[509722.778808] {
[509722.778839] 0
[509722.778839] }
[509722.778839] (t=4065 jiffies g=1584636 c=1584635 q=13)
[509722.778839] Task dump for CPU 0:
[509722.778839] swapper/0 R
[509722.778839] running
[509722.778839] 0 0 0 0x00000000
[509722.778869] Backtrace:
[509722.778869] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[509722.778869] r6:800c0193
[509722.778869] r5:00000000
[509722.778869] r4:c07a0920
[509722.778869] r3:00000000
[509722.778930] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[509722.778961] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[509722.778961] r5:c07b1240
[509722.778961] r4:00000000
[509722.778961] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[509722.778991] r4:00000001
[509722.778991] r3:00000003
[509722.778991] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[509722.778991] r8:0000000d
[509722.779022] r7:c079e798
[509722.779022] r6:c07b1240
[509722.779022] r5:c07933d8
[509722.779022] r4:eefb13d8
[509722.779022] r3:00010001
[509722.779052] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[509722.779052] r10:00000001
[509722.779052] r9:eefb15d0
[509722.779052] r8:eefb15d0
[509722.779052] r7:00000000
[509722.779052] r6:00000000
[509722.779052] r5:c07a0920
[509722.779083] r4:c0796000
[509722.779083] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[509722.779083] r7:0001cf96
[509722.779083] r6:c2511cf6
[509722.779113] r5:c0797f20
[509722.779113] r4:eefb1858
[509722.779113] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[509722.779113] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[509722.779144] r7:c0093510
[509722.779144] r6:eefb15d0
[509722.779144] r5:eefb1608
[509722.779144] r4:eefb1858
[509722.779144] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[509722.779174] r7:0001cf96
[509722.779174] r6:c2511cf6
[509722.779174] r5:00000000
[509722.779174] r4:eefb15d0
[509722.779174] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[509722.779205] r10:c0789aac
[509722.779205] r9:c07d6bd9
[509722.779205] r8:0000001d
[509722.779205] r7:ee813b40
[509722.779205] r6:ee806600
[509722.779205] r5:c07b8e4c
[509722.779205] r4:00000001
[509722.779205] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[509722.779205] r4:eefb5100
[509722.779205] r3:c0015594
[509722.779266] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[509722.779266] r8:ee808400
[509722.779266] r7:00000001
[509722.779266] r6:00000000
[509722.779266] r5:00000000
[509722.779266] r4:0000001d
[509722.779296] r3:c00790dc
[509722.779296] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[509722.779296] r4:c0793cc0
[509722.779296] r3:000001ab
[509722.779327] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[509722.779327] r8:fa240100
[509722.779327] r7:c0797f20
[509722.779327] r6:c079e908
[509722.779327] r5:0000000d
[509722.779327] r4:fa24010c
[509722.779327] r3:c0797f20
[509722.779357] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[509722.779357] Exception stack(0xc0797f20 to 0xc0797f68)
[509722.779388] 7f20: eefb13c8 00000000 245116fc c002cc04 c0796000 c079e488 c079e4d4 c059670c
[509722.779388] 7f40: c07d6bd9 c07d6bd9 c0789aac c0797f74 c0797f68 c0797f68 c002c814 c000fdcc
[509722.779388] 7f60: a00c0113 ffffffff
[509722.779388] r8:c07d6bd9
[509722.779388] r7:c0797f54
[509722.779418] r6:ffffffff
[509722.779418] r5:a00c0113
[509722.779418] r4:c000fdcc
[509722.779418] r3:c002c814
[509722.779449] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[509722.779449] [<c006ca7c>] (cpu_startup_entry) from [<c0589140>] (rest_init+0x8c/0x90)
[509722.779449] r7:ffffffff
[509722.779449] r3:c0596af8
[509722.779479] [<c05890b4>] (rest_init) from [<c0755cb4>] (start_kernel+0x368/0x374)
[509722.779479] r4:c079e598
[509722.779479] r3:00000001
[509722.779510] [<c075594c>] (start_kernel) from [<80008074>] (0x80008074)
[509722.779510] Task dump for CPU 1:
[509722.779510] swapper/1 R
[509722.779510] running
[509722.779541] 0 0 1 0x00000000
[509722.779541] Backtrace:
[509722.779541] [<ee89ffb0>] (0xee89ffb0) from [<00000000>] ( (null))
[509722.779541] Backtrace aborted due to bad frame pointer <c07d6bd9>
[509723.105773] {
[509723.135803] 1} (t=4102 jiffies g=1584636 c=1584635 q=4)
[509723.141326] Task dump for CPU 1:
[509723.141326] swapper/1 R running 0 0 1 0x00000002
[509723.151519] Backtrace:
[509723.151519] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[509723.154632] r6:80070193 r5:00000001 r4:ee876f80 r3:00000000
[509723.154632] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[509723.154632] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[509723.154632] r5:c07b1240 r4:00000001
[509723.154632] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[509723.154632] r4:00000002 r3:00000001
[509723.201141] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[509723.201141] r8:00000004 r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefba3d8 r3:00010001
[509723.218811] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[509723.218811] r10:00000001 r9:eefba5d0 r8:eefba5d0 r7:00000001 r6:00000000 r5:ee876f80
[509723.236450] r4:ee89e000
[509723.236450] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[509723.236450] r7:0001cf96 r6:c2511cf6 r5:ee89ff68 r4:eefba858
[509723.236450] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[509723.264648] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[509723.264648] r7:c0093510 r6:eefba5d0 r5:eefba608 r4:eefba858
[509723.273834] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[509723.273834] r7:0001cf96 r6:c2511cf6 r5:00000000 r4:eefba5d0
[509723.295318] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[509723.295318] r10:00000000 r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[509723.311950] r4:00000001
[509723.311950] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[509723.323608] r4:eefbe100 r3:c0015594
[509723.323608] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[509723.323608] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[509723.345184] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[509723.345184] r4:c0793cc0 r3:000001ab
[509723.345184] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[509723.367004] r8:fa240100 r7:ee89ff68 r6:c079e908 r5:0000000d r4:fa24010c r3:ee89ff68
[509723.367004] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[509723.367004] Exception stack(0xee89ff68 to 0xee89ffb0)
[509723.388488] ff60: eefba3c8 00000000 1d259418 c002cc04 ee89e000 c079e488
[509723.388488] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[509723.388488] ffa0: c002c814 c000fdcc a0070113 ffffffff
[509723.411102] r8:c07d6bd9 r7:ee89ff9c r6:ffffffff r5:a0070113 r4:c000fdcc r3:c002c814
[509723.411102] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[509723.411102] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[509723.411102] r7:c07d7080 r3:00000001
[509723.411102] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[509723.411102] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
[548548.134063] wlcore: Beacon loss detected. roles:0xff
[548549.772338] wlcore: Beacon loss detected. roles:0xff
[548551.410797] wlcore: Beacon loss detected. roles:0xff
[548552.939575] wlcore: Connection loss work (role_id: 0).
[548552.954986] wlan0: Connection to AP 90:f6:52:81:ee:70 lost
[548553.049377] wlcore: Beacon loss detected. roles:0xff
[548553.062835] cfg80211: Calling CRDA to update world regulatory domain
[594970.248321] INFO: rcu_preempt self-detected stall on CPU { 0} (t=2788 jiffies g=1877742 c=1877741 q=203)
[594970.258453] Task dump for CPU 0:
[594970.258453] swapper/0 R running 0 0 0 0x00000000
[594970.267578] INFO: rcu_preempt self-detected stall on CPU { 1} (t=2788 jiffies g=1877742 c=1877741 q=203)
[594970.267578] Backtrace:
[594970.267578] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[594970.267578] r6:80070193 r5:00000000 r4:c07a0920 r3:00000000
[594970.267578] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[594970.267578] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[594970.267578] r5:c07b1240 r4:00000000
[594970.267578] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[594970.324523] r4:00000001 r3:00000003
[594970.324523] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[594970.324523] r8:000000cb r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefb13d8 r3:00010001
[594970.324523] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[594970.355377] r10:00000001 r9:eefb15d0 r8:eefb15d0 r7:00000000 r6:00000000 r5:c07a0920
[594970.355377] r4:c0796000
[594970.355377] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[594970.376312] r7:00021d1f r6:00edff42 r5:c0797f20 r4:eefb1858
[594970.376312] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[594970.391876] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[594970.391876] r7:c0093510 r6:eefb15d0 r5:eefb1608 r4:eefb1858
[594970.407073] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[594970.407073] r7:00021d1f r6:00edff42 r5:00000000 r4:eefb15d0
[594970.407073] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[594970.407073] r10:c0789aac r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[594970.439178] r4:00000001
[594970.439178] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[594970.450836] r4:eefb5100 r3:c0015594
[594970.450836] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[594970.464202] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[594970.464202] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[594970.464202] r4:c0793cc0 r3:000001ab
[594970.485412] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[594970.485412] r8:fa240100 r7:c0797f20 r6:c079e908 r5:0000000d r4:fa24010c r3:c0797f20
[594970.494232] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[594970.502441] Exception stack(0xc0797f20 to 0xc0797f68)
[594970.510345] 7f20: eefb13c8 00000000 2a3ae6f6 c002cc04 c0796000 c079e488 c079e4d4 c059670c
[594970.515716] 7f40: c07d6bd9 c07d6bd9 c0789aac c0797f74 c0797f68 c0797f68 c002c814 c000fdcc
[594970.532958] 7f60: a0070113 ffffffff
[594970.532958] r8:c07d6bd9 r7:c0797f54 r6:ffffffff r5:a0070113 r4:c000fdcc r3:c002c814
[594970.532958] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[594970.544921] [<c006ca7c>] (cpu_startup_entry) from [<c0589140>] (rest_init+0x8c/0x90)
[594970.544921] r7:ffffffff r3:c0596af8
[594970.561828] [<c05890b4>] (rest_init) from [<c0755cb4>] (start_kernel+0x368/0x374)
[594970.565673] r4:c079e598 r3:00000001
[594970.565673] [<c075594c>] (start_kernel) from [<80008074>] (0x80008074)
[594970.565673] Task dump for CPU 1:
[594970.584320] swapper/1 R running 0 0 1 0x00000000
[594970.584320] Backtrace:
[594970.584320] [<ee89ffb0>] (0xee89ffb0) from [<00000000>] ( (null))
[594970.584320] Backtrace aborted due to bad frame pointer <c07d6bd9>
[594970.603698] Task dump for CPU 0:
[594970.603698] swapper/0 R running 0 0 0 0x00000000
[594970.620330] Backtrace:
[594970.620330] [<c0797f68>] (0xc0797f68) from [<c0789aac>] (0xc0789aac)
[594970.620330] Backtrace aborted due to bad frame pointer <c07d6bd9>
[594970.636169] Task dump for CPU 1:
[594970.636169] swapper/1 R running 0 0 1 0x00000000
[594970.646362] Backtrace:
[594970.646362] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[594970.646362] r6:80010193 r5:00000001 r4:ee876f80 r3:00000000
[594970.646362] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[594970.646362] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[594970.646362] r5:c07b1240 r4:00000001
[594970.683410] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[594970.683410] r4:00000002 r3:00000001
[594970.692138] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[594970.705413] r8:000000cb r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefba3d8 r3:00010001
[594970.705413] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[594970.705413] r10:00000001 r9:eefba5d0 r8:eefba5d0 r7:00000001 r6:00000000 r5:ee876f80
[594970.731262] r4:ee89e000
[594970.731262] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[594970.731262] r7:00021d1f r6:0213d465 r5:ee89ff68 r4:eefba858
[594970.731262] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[594970.749938] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[594970.768646] r7:c0093510 r6:eefba5d0 r5:eefba608 r4:eefba858
[594970.768646] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[594970.768646] r7:00021d1f r6:0213d465 r5:00000000 r4:eefba5d0
[594970.790130] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[594970.790130] r10:00000000 r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[594970.798492] r4:00000001
[594970.798492] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[594970.818420] r4:eefbe100 r3:c0015594
[594970.818420] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[594970.818420] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[594970.818420] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[594970.818420] r4:c0793cc0 r3:000001ab
[594970.818420] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[594970.861816] r8:fa240100 r7:ee89ff68 r6:c079e908 r5:0000000d r4:fa24010c r3:ee89ff68
[594970.861816] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[594970.861816] Exception stack(0xee89ff68 to 0xee89ffb0)
[594970.861816] ff60: eefba3c8 00000000 22429af4 c002cc04 ee89e000 c079e488
[594970.883300] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[594970.883300] ffa0: c002c814 c000fdcc a0010113 ffffffff
[594970.905914] r8:c07d6bd9 r7:ee89ff9c r6:ffffffff r5:a0010113 r4:c000fdcc r3:c002c814
[594970.905914] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[594970.905914] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[594970.905914] r7:c07d7080 r3:00000001
[594970.936218] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[594970.936218] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
[599814.157684] INFO: rcu_preempt self-detected stall on CPU { 0} (t=2615 jiffies g=1896139 c=1896138 q=202)
[599814.163848] INFO: rcu_preempt self-detected stall on CPU { 1} (t=2615 jiffies g=1896139 c=1896138 q=202)
[599814.163879] Task dump for CPU 0:
[599814.163879] swapper/0 R running 0 0 0 0x00000002
[599814.163879] Backtrace:
[599814.163909] [<c0797f68>] (0xc0797f68) from [<c0789aac>] (0xc0789aac)
[599814.163909] Backtrace aborted due to bad frame pointer <c07d6bd9>
[599814.163909] Task dump for CPU 1:
[599814.163940] swapper/1 R running 0 0 1 0x00000002
[599814.163940] Backtrace:
[599814.163940] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[599814.163940] r6:800f0193 r5:00000001 r4:ee876f80 r3:00000000
[599814.164001] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[599814.164001] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[599814.164031] r5:c07b1240 r4:00000001
[599814.164031] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[599814.164062] r4:00000002 r3:00000001
[599814.164062] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[599814.164093] r8:000000ca r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefba3d8 r3:00010001
[599814.164093] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[599814.164123] r10:00000001 r9:eefba5d0 r8:eefba5d0 r7:00000001 r6:00000000 r5:ee876f80
[599814.164123] r4:ee89e000
[599814.164154] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[599814.164154] r7:00022186 r6:d115fc96 r5:ee89ff68 r4:eefba858
[599814.164184] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[599814.164184] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[599814.164184] r7:c0093510 r6:eefba5d0 r5:eefba608 r4:eefba858
[599814.164215] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[599814.164215] r7:00022186 r6:d113a88a r5:00000000 r4:eefba5d0
[599814.164245] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[599814.164245] r10:00000000 r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[599814.164245] r4:00000001
[599814.164276] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[599814.164276] r4:eefbe100 r3:c0015594
[599814.164306] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[599814.164337] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[599814.164337] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[599814.164337] r4:c0793cc0 r3:000001ab
[599814.164367] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[599814.164367] r8:fa240100 r7:ee89ff68 r6:c079e908 r5:0000000d r4:fa24010c r3:ee89ff68
[599814.164398] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[599814.164398] Exception stack(0xee89ff68 to 0xee89ffb0)
[599814.164398] ff60: eefba3c8 00000000 22901142 c002cc04 ee89e000 c079e488
[599814.164428] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[599814.164428] ffa0: c002c814 c000fdcc a00f0113 ffffffff
[599814.164428] r8:c07d6bd9 r7:ee89ff9c r6:ffffffff r5:a00f0113 r4:c000fdcc r3:c002c814
[599814.164428] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[599814.164489] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[599814.164489] r7:c07d7080 r3:00000001
[599814.164489] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[599814.164520] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
[599814.473236] Task dump for CPU 0:
[599814.520416] swapper/0 R running 0 0 0 0x00000002
[599814.523284] Backtrace:
[599814.523284] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[599814.529815] r6:800c0193 r5:00000000 r4:c07a0920 r3:00000000
[599814.540069] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[599814.549530] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[599814.553222] r5:c07b1240 r4:00000000
[599814.564208] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[599814.564208] r4:00000001 r3:00000001
[599814.573242] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[599814.583221] r8:000000ca r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefb13d8 r3:00010001
[599814.594421] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[599814.603759] r10:00000001 r9:eefb15d0 r8:eefb15d0 r7:00000000 r6:00000000 r5:c07a0920
[599814.609039] r4:c0796000
[599814.609039] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[599814.624694] r7:00022186 r6:d0b77514 r5:c0797f20 r4:eefb1858
[599814.626861] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[599814.635375] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[599814.635375] r7:c0093510 r6:eefb15d0 r5:eefb1608 r4:eefb1858
[599814.653228] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[599814.653228] r7:00022186 r6:d0b77514 r5:00000000 r4:eefb15d0
[599814.670928] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[599814.673217] r10:c0789aac r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[599814.683227] r4:00000001
[599814.690307] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[599814.696136] r4:eefb5100 r3:c0015594
[599814.699218] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[599814.712585] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[599814.720794] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[599814.728698] r4:c0793cc0 r3:000001ab
[599814.733795] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[599814.733795] r8:fa240100 r7:c0797f20 r6:c079e908 r5:0000000d r4:fa24010c r3:c0797f20
[599814.743835] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[599814.743835] Exception stack(0xc0797f20 to 0xc0797f68)
[599814.762603] 7f20: eefb13c8 00000000 2a8ddd74 c002cc04 c0796000 c079e488 c079e4d4 c059670c
[599814.766906] 7f40: c07d6bd9 c07d6bd9 c0789aac c0797f74 c0797f68 c0797f68 c002c814 c000fdcc
[599814.773223] 7f60: a00c0113 ffffffff
[599814.785064] r8:c07d6bd9 r7:c0797f54 r6:ffffffff r5:a00c0113 r4:c000fdcc r3:c002c814
[599814.793243] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[599814.793243] [<c006ca7c>] (cpu_startup_entry) from [<c0589140>] (rest_init+0x8c/0x90)
[599814.803222] r7:ffffffff r3:c0596af8
[599814.814025] [<c05890b4>] (rest_init) from [<c0755cb4>] (start_kernel+0x368/0x374)
[599814.814025] r4:c079e598 r3:00000001
[599814.825775] [<c075594c>] (start_kernel) from [<80008074>] (0x80008074)
[664246.427612] INFO: rcu_preempt self-detected stall on CPU
[664246.429473] INFO: rcu_preempt detected stalls on CPUs/tasks:
[664246.429473] {
[664246.429473] 1
[664246.429473] }
[664246.429473] (detected by 0, t=3407 jiffies, g=2142187, c=2142186, q=70)
[664246.429473] Task dump for CPU 1:
[664246.429473] swapper/1 R
[664246.429504] running
[664246.429504] 0 0 1 0x00000000
[664246.429504] Backtrace:
[664246.429504] [<ee89ffb0>] (0xee89ffb0) from [<00000000>] ( (null))
[664246.429534] Backtrace aborted due to bad frame pointer <c07d6bd9>
[664246.470031] {
[664246.472412] 1} (t=3407 jiffies g=2142187 c=2142186 q=70)
[664246.476959] Task dump for CPU 1:
[664246.479370] swapper/1 R running 0 0 1 0x00000000
[664246.487091] Backtrace:
[664246.488677] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[664246.497772] r6:80070193 r5:00000001 r4:ee876f80 r3:00000000
[664246.503112] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[664246.511596] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[664246.519836] r5:c07b1240 r4:00000001
[664246.523773] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[664246.532806] r4:00000002 r3:00000001
[664246.536437] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[664246.545928] r8:00000046 r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefba3d8 r3:00010001
[664246.553863] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[664246.563842] r10:00000001 r9:eefba5d0 r8:eefba5d0 r7:00000001 r6:00000000 r5:ee876f80
[664246.571014] r4:ee89e000
[664246.574890] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[664246.584045] r7:00025c20 r6:a322d707 r5:ee89ff68 r4:eefba858
[664246.590728] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[664246.600341] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[664246.609527] r7:c0093510 r6:eefba5d0 r5:eefba608 r4:eefba858
[664246.615539] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[664246.624542] r7:00025c20 r6:a322d707 r5:00000000 r4:eefba5d0
[664246.631011] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[664246.631011] r10:00000000 r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[664246.631011] r4:00000001
[664246.631011] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[664246.653442] r4:eefbe100 r3:c0015594
[664246.663146] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[664246.663543] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[664246.678497] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[664246.690063] r4:c0793cc0 r3:000001ab
[664246.690063] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[664246.693908] r8:fa240100 r7:ee89ff68 r6:c079e908 r5:0000000d r4:fa24010c r3:ee89ff68
[664246.708496] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[664246.708496] Exception stack(0xee89ff68 to 0xee89ffb0)
[664246.724182] ff60: eefba3c8 00000000 2685ad74 c002cc04 ee89e000 c079e488
[664246.731842] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[664246.731842] ffa0: c002c814 c000fdcc a0070113 ffffffff
[664246.731842] r8:c07d6bd9 r7:ee89ff9c r6:ffffffff r5:a0070113 r4:c000fdcc r3:c002c814
[664246.731842] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[664246.763763] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[664246.773223] r7:c07d7080 r3:00000001
[664246.777130] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[664246.779846] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
[671203.857025] INFO: rcu_preempt self-detected stall on CPU
[671203.857025] INFO: rcu_preempt self-detected stall on CPU { 0} (t=6703 jiffies g=2168410 c=2168409 q=200)
[671203.857055] Task dump for CPU 0:
[671203.857055] swapper/0 R running 0 0 0 0x00000000
[671203.857055] Backtrace:
[671203.857055] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[671203.857116] r6:80070193 r5:00000000 r4:c07a0920 r3:00000000
[671203.857116] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[671203.857147] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[671203.857147] r5:c07b1240 r4:00000000
[671203.857177] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[671203.857177] r4:00000001 r3:00000003
[671203.857208] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[671203.857208] r8:000000c8 r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefb13d8 r3:00010001
[671203.857238] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[671203.857238] r10:00000001 r9:eefb15d0 r8:eefb15d0 r7:00000000 r6:00000000 r5:c07a0920
[671203.857269] r4:c0796000
[671203.857269] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[671203.857299] r7:00026274 r6:8a5509f4 r5:c0797f20 r4:eefb1858
[671203.857299] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[671203.857330] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[671203.857330] r7:c0093510 r6:eefb15d0 r5:eefb1608 r4:eefb1858
[671203.857330] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[671203.857360] r7:00026274 r6:8a5509f4 r5:00000000 r4:eefb15d0
[671203.857360] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[671203.857391] r10:c0789aac r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[671203.857391] r4:00000001
[671203.857421] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[671203.857421] r4:eefb5100 r3:c0015594
[671203.857452] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[671203.857452] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[671203.857452] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[671203.857482] r4:c0793cc0 r3:000001ab
[671203.857482] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[671203.857513] r8:fa240100 r7:c0797f20 r6:c079e908 r5:0000000d r4:fa24010c r3:c0797f20
[671203.857513] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[671203.857513] Exception stack(0xc0797f20 to 0xc0797f68)
[671203.857543] 7f20: eefb13c8 00000000 2f704416 c002cc04 c0796000 c079e488 c079e4d4 c059670c
[671203.857543] 7f40: c07d6bd9 c07d6bd9 c0789aac c0797f74 c0797f68 c0797f68 c002c814 c000fdcc
[671203.857543] 7f60: a0070113 ffffffff
[671203.857574] r8:c07d6bd9 r7:c0797f54 r6:ffffffff r5:a0070113 r4:c000fdcc r3:c002c814
[671203.857604] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[671203.857604] [<c006ca7c>] (cpu_startup_entry) from [<c0589140>] (rest_init+0x8c/0x90)
[671203.857635] r7:ffffffff r3:c0596af8
[671203.857635] [<c05890b4>] (rest_init) from [<c0755cb4>] (start_kernel+0x368/0x374)
[671203.857666] r4:c079e598 r3:00000001
[671203.857666] [<c075594c>] (start_kernel) from [<80008074>] (0x80008074)
[671203.857666] Task dump for CPU 1:
[671203.857696] swapper/1 R running 0 0 1 0x00000002
[671203.857696] Backtrace:
[671203.857696] [<ee89ffb0>] (0xee89ffb0) from [<00000000>] ( (null))
[671203.857696] Backtrace aborted due to bad frame pointer <c07d6bd9>
[671204.211517] { 1} (t=6739 jiffies g=2168410 c=2168409 q=202)
[671204.211517] Task dump for CPU 1:
[671204.211517] swapper/1 R running 0 0 1 0x00000002
[671204.229736] Backtrace:
[671204.229736] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[671204.229736] r6:800c0193 r5:00000001 r4:ee876f80 r3:00000000
[671204.246429] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[671204.246429] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[671204.246429] r5:c07b1240 r4:00000001
[671204.266815] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[671204.266815] r4:00000002 r3:00000001
[671204.279357] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[671204.279357] r8:000000ca r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefba3d8 r3:00010001
[671204.279357] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[671204.306365] r10:00000001 r9:eefba5d0 r8:eefba5d0 r7:00000001 r6:00000000 r5:ee876f80
[671204.306365] r4:ee89e000
[671204.314666] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[671204.314666] r7:00026274 r6:8aa952dc r5:ee89ff68 r4:eefba858
[671204.314666] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[671204.342864] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[671204.342864] r7:c0093510 r6:eefba5d0 r5:eefba608 r4:eefba858
[671204.358062] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[671204.358062] r7:00026274 r6:8aa952dc r5:00000000 r4:eefba5d0
[671204.373535] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[671204.373535] r10:00000000 r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[671204.390167] r4:00000001
[671204.390167] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[671204.401824] r4:eefbe100 r3:c0015594
[671204.401824] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[671204.405670] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[671204.405670] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[671204.405670] r4:c0793cc0 r3:000001ab
[671204.436401] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[671204.445220] r8:fa240100 r7:ee89ff68 r6:c079e908 r5:0000000d r4:fa24010c r3:ee89ff68
[671204.445220] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[671204.445220] Exception stack(0xee89ff68 to 0xee89ffb0)
[671204.461334] ff60: eefba3c8 00000000 26ee7df6 c002cc04 ee89e000 c079e488
[671204.461334] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[671204.483947] ffa0: c002c814 c000fdcc a00c0113 ffffffff
[671204.489318] r8:c07d6bd9 r7:ee89ff9c r6:ffffffff r5:a00c0113 r4:c000fdcc r3:c002c814
[671204.489318] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[671204.489318] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[671204.489318] r7:c07d7080 r3:00000001
[671204.489318] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[671204.527435] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
[694287.886413] INFO: rcu_preempt self-detected stall on CPU
[694287.888641] INFO: rcu_preempt detected stalls on CPUs/tasks:
[694287.888641] {
[694287.888641] 1
[694287.888641] }
[694287.888641] (detected by 0, t=2869 jiffies, g=2256491, c=2256490, q=110)
[694287.888641] Task dump for CPU 1:
[694287.888641] swapper/1 R
[694287.888671] running
[694287.888671] 0 0 1 0x00000000
[694287.888671] Backtrace:
[694287.888702] [<ee89ffb0>] (0xee89ffb0) from [<00000000>] ( (null))
[694287.888702] Backtrace aborted due to bad frame pointer <c07d6bd9>
[694287.929687] {
[694287.931426] 1} (t=2869 jiffies g=2256491 c=2256490 q=110)
[694287.935974] Task dump for CPU 1:
[694287.938476] swapper/1 R running 0 0 1 0x00000000
[694287.945434] Backtrace:
[694287.948760] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[694287.956298] r6:80070193 r5:00000001 r4:ee876f80 r3:00000000
[694287.962738] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[694287.970825] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[694287.978912] r5:c07b1240 r4:00000001
[694287.982543] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[694287.991119] r4:00000002 r3:00000001
[694287.995758] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[694288.005157] r8:0000006e r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefba3d8 r3:00010001
[694288.013488] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[694288.021759] r10:00000001 r9:eefba5d0 r8:eefba5d0 r7:00000001 r6:00000000 r5:ee876f80
[694288.031127] r4:ee89e000
[694288.033874] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[694288.043487] r7:00027773 r6:376a590f r5:ee89ff68 r4:eefba858
[694288.049530] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[694288.058715] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[694288.067779] r7:c0093510 r6:eefba5d0 r5:eefba608 r4:eefba858
[694288.074035] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[694288.083038] r7:00027773 r6:376a590f r5:00000000 r4:eefba5d0
[694288.088348] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[694288.090026] r10:00000000 r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[694288.106658] r4:00000001
[694288.106658] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[694288.118316] r4:eefbe100 r3:c0015594
[694288.118316] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[694288.122161] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[694288.131744] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[694288.131744] r4:c0793cc0 r3:000001ab
[694288.152893] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[694288.161712] r8:fa240100 r7:ee89ff68 r6:c079e908 r5:0000000d r4:fa24010c r3:ee89ff68
[694288.163726] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[694288.163726] Exception stack(0xee89ff68 to 0xee89ffb0)
[694288.183197] ff60: eefba3c8 00000000 2843f09a c002cc04 ee89e000 c079e488
[694288.183197] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[694288.196838] ffa0: c002c814 c000fdcc a0070113 ffffffff
[694288.205657] r8:c07d6bd9 r7:ee89ff9c r6:ffffffff r5:a0070113 r4:c000fdcc r3:c002c814
[694288.213592] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[694288.218322] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[694288.228301] r7:c07d7080 r3:00000001
[694288.232299] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[694288.238311] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
[760503.480987] INFO: rcu_preempt self-detected stall on CPU
[760503.481811] INFO: rcu_preempt self-detected stall on CPU
[760503.481811] {
[760503.481811] 1
[760503.481811] }
[760503.481811] (t=2282 jiffies g=2509003 c=2509002 q=204)
[760503.481842] Task dump for CPU 0:
[760503.481842] modprobe R
[760503.481842] running
[760503.481842] 0 17889 17888 0x00000000
[760503.481842] Backtrace:
[760503.481842] [<eefb50f4>] (0xeefb50f4) from [<c0015594>] (twd_handler+0x0/0x48)
[760503.481842] Backtrace aborted due to bad frame pointer <c00155cc>
[760503.481842] Task dump for CPU 1:
[760503.481903] swapper/1 R
[760503.481903] running
[760503.481903] 0 0 1 0x00000000
[760503.481903] Backtrace:
[760503.481933] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[760503.481933] r6:80070193
[760503.481933] r5:00000001
[760503.481933] r4:ee876f80
[760503.481933] r3:00000000
[760503.481964] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[760503.481994] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[760503.481994] r5:c07b1240
[760503.481994] r4:00000001
[760503.481994] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[760503.482025] r4:00000002
[760503.482025] r3:00000001
[760503.482025] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[760503.482025] r8:000000cc
[760503.482055] r7:c079e798
[760503.482055] r6:c07b1240
[760503.482055] r5:c07933d8
[760503.482055] r4:eefba3d8
[760503.482055] r3:00010001
[760503.482086] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[760503.482086] r10:00000001
[760503.482086] r9:eefba5d0
[760503.482086] r8:eefba5d0
[760503.482086] r7:00000001
[760503.482086] r6:00000000
[760503.482086] r5:ee876f80
[760503.482116] r4:ee89e000
[760503.482116] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[760503.482116] r7:0002b3ac
[760503.482116] r6:406cb5e9
[760503.482147] r5:ee89ff68
[760503.482147] r4:eefba858
[760503.482147] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[760503.482177] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[760503.482177] r7:c0093510
[760503.482177] r6:eefba5d0
[760503.482177] r5:eefba608
[760503.482177] r4:eefba858
[760503.482208] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[760503.482208] r7:0002b3ac
[760503.482208] r6:406cb5e9
[760503.482208] r5:00000000
[760503.482208] r4:eefba5d0
[760503.482208] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[760503.482238] r10:00000000
[760503.482238] r9:c07d6bd9
[760503.482238] r8:0000001d
[760503.482238] r7:ee813b40
[760503.482238] r6:ee806600
[760503.482238] r5:c07b8e4c
[760503.482238] r4:00000001
[760503.482269] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[760503.482269] r4:eefbe100
[760503.482269] r3:c0015594
[760503.482299] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[760503.482299] r8:ee808400
[760503.482299] r7:00000001
[760503.482299] r6:00000000
[760503.482330] r5:00000000
[760503.482330] r4:0000001d
[760503.482330] r3:c00790dc
[760503.482330] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[760503.482330] r4:c0793cc0
[760503.482330] r3:000001ab
[760503.482360] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[760503.482360] r8:fa240100
[760503.482360] r7:ee89ff68
[760503.482360] r6:c079e908
[760503.482360] r5:0000000d
[760503.482391] r4:fa24010c
[760503.482391] r3:ee89ff68
[760503.482391] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[760503.482391] Exception stack(0xee89ff68 to 0xee89ffb0)
[760503.482421] ff60: eefba3c8 00000000 2c45ffae c002cc04 ee89e000 c079e488
[760503.482421] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[760503.482421] ffa0: c002c814 c000fdcc a0070113 ffffffff
[760503.482452] r8:c07d6bd9
[760503.482452] r7:ee89ff9c
[760503.482452] r6:ffffffff
[760503.482452] r5:a0070113
[760503.482452] r4:c000fdcc
[760503.482452] r3:c002c814
[760503.482482] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[760503.482482] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[760503.482513] r7:c07d7080
[760503.482513] r3:00000001
[760503.482513] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[760503.482513] r6:10c0387d
[760503.482513] r5:00000015
[760503.482543] r4:ae88406a
[760503.482543] r3:c00087ac
[760503.821777] {
[760503.821777] 0} (t=2282 jiffies g=2509003 c=2509002 q=202)
[760503.841796] Task dump for CPU 0:
[760503.841796] modprobe R running 0 17889 17888 0x00000000
[760503.851806] Backtrace:
[760503.851806] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[760503.851806] r6:80030193 r5:000045e0 r4:ee9e3900 r3:00000000
[760503.851806] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[760503.871795] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[760503.885467] r5:c07b1240 r4:00000000
[760503.889129] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[760503.891784] r4:00000001 r3:00000001
[760503.891784] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[760503.912292] r8:000000ca r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefb13d8 r3:00010000
[760503.912292] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[760503.922851] r10:00000001 r9:eefb15d0 r8:eefb15d0 r7:00000000 r6:00000000 r5:ee9e3900
[760503.938140] r4:ed6b6000
[760503.940704] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[760503.942871] r7:0002b3ac r6:4052a22e r5:ed6b7f30 r4:eefb1858
[760503.956817] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[760503.956817] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[760503.971862] r7:c0093510 r6:eefb15d0 r5:eefb1608 r4:eefb1858
[760503.971862] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[760503.981780] r7:0002b3ac r6:4052a22e r5:00000000 r4:eefb15d0
[760503.997009] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[760504.001800] r10:00000000 r9:ed6b6000 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[760504.001800] r4:00000001
[760504.001800] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[760504.021789] r4:eefb5100 r3:c0015594
[760504.029144] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[760504.034423] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[760504.041778] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[760504.056060] r4:c0793cc0 r3:000001ab
[760504.056060] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[760504.068695] r8:fa240100 r7:ed6b7f30 r6:c079e908 r5:0000000d r4:fa24010c r3:ed6b7f30
[760504.076904] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[760504.084808] Exception stack(0xed6b7f30 to 0xed6b7f78)
[760504.090179] 7f20: ee8c4d80 00000000 000dd588 00000005
[760504.096832] 7f40: 00000002 00000005 000dd588 00000000 ee8c4d80 ed6b6000 00000000 ed6b7fa4
[760504.104736] 7f60: ed6b7f48 ed6b7f78 c00e1478 c00cfe80 40030113 ffffffff
[760504.111785] r8:ee8c4d80 r7:ed6b7f64 r6:ffffffff r5:40030113 r4:c00cfe80 r3:c00e1478
[760504.114440] [<c00e13dc>] (SyS_mmap_pgoff) from [<c000eb80>] (ret_fast_syscall+0x0/0x30)
[760504.114440] r8:c000ed48 r7:000000c0 r6:00000000 r5:00000000 r4:00000000
[785859.968719] INFO: rcu_preempt self-detected stall on CPU { 1} (t=0 jiffies g=2605409 c=2605408 q=17)
[785859.978485] Task dump for CPU 1:
[785859.981872] swapper/1 R running 0 0 1 0x00000002
[785859.988372] Backtrace:
[785859.991363] [<c00126ec>] (dump_backtrace) from [<c0012900>] (show_stack+0x18/0x1c)
[785859.999206] r6:800c0193 r5:00000001 r4:ee876f80 r3:00000000
[785860.005279] [<c00128e8>] (show_stack) from [<c005c33c>] (sched_show_task+0xb4/0x108)
[785860.013427] [<c005c288>] (sched_show_task) from [<c005f240>] (dump_cpu_task+0x40/0x44)
[785860.021484] r5:c07b1240 r4:00000001
[785860.025268] [<c005f200>] (dump_cpu_task) from [<c007cee0>] (rcu_dump_cpu_stacks+0x94/0xd0)
[785860.034454] r4:00000002 r3:00000001
[785860.037902] [<c007ce4c>] (rcu_dump_cpu_stacks) from [<c0080764>] (rcu_check_callbacks+0x428/0x798)
[785860.047790] r8:00000011 r7:c079e798 r6:c07b1240 r5:c07933d8 r4:eefba3d8 r3:00010001
[785860.055053] [<c008033c>] (rcu_check_callbacks) from [<c0083520>] (update_process_times+0x48/0x68)
[785860.065338] r10:00000001 r9:eefba5d0 r8:eefba5d0 r7:00000001 r6:00000000 r5:ee876f80
[785860.073547] r4:ee89e000
[785860.075775] [<c00834d8>] (update_process_times) from [<c009350c>] (tick_sched_handle.isra.17+0x4c/0x50)
[785860.086303] r7:0002cabc r6:06560cb0 r5:ee89ff68 r4:eefba858
[785860.091583] [<c00934c0>] (tick_sched_handle.isra.17) from [<c0093564>] (tick_sched_timer+0x54/0x98)
[785860.101501] [<c0093510>] (tick_sched_timer) from [<c0083da4>] (__run_hrtimer.isra.34+0x4c/0xdc)
[785860.111053] r7:c0093510 r6:eefba5d0 r5:eefba608 r4:eefba858
[785860.117065] [<c0083d58>] (__run_hrtimer.isra.34) from [<c0084574>] (hrtimer_interrupt+0x124/0x2d4)
[785860.126495] r7:0002cabc r6:06560cb0 r5:00000000 r4:eefba5d0
[785860.131683] [<c0084450>] (hrtimer_interrupt) from [<c00155cc>] (twd_handler+0x38/0x48)
[785860.140899] r10:00000000 r9:c07d6bd9 r8:0000001d r7:ee813b40 r6:ee806600 r5:c07b8e4c
[785860.148315] r4:00000001
[785860.151947] [<c0015594>] (twd_handler) from [<c0079150>] (handle_percpu_devid_irq+0x74/0x8c)
[785860.160247] r4:eefbe100 r3:c0015594
[785860.163879] [<c00790dc>] (handle_percpu_devid_irq) from [<c00752fc>] (generic_handle_irq+0x34/0x44)
[785860.174224] r8:ee808400 r7:00000001 r6:00000000 r5:00000000 r4:0000001d r3:c00790dc
[785860.182434] [<c00752c8>] (generic_handle_irq) from [<c007559c>] (__handle_domain_irq+0x64/0xac)
[785860.182434] r4:c0793cc0 r3:000001ab
[785860.194061] [<c0075538>] (__handle_domain_irq) from [<c00086f0>] (gic_handle_irq+0x2c/0x64)
[785860.196319] r8:fa240100 r7:ee89ff68 r6:c079e908 r5:0000000d r4:fa24010c r3:ee89ff68
[785860.208557] [<c00086c4>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[785860.218536] Exception stack(0xee89ff68 to 0xee89ffb0)
[785860.218536] ff60: eefba3c8 00000000 2ddfb296 c002cc04 ee89e000 c079e488
[785860.226318] ff80: c079e4d4 c059670c c07d6bd9 c07d6bd9 00000000 ee89ffbc ee89ffb0 ee89ffb0
[785860.236328] ffa0: c002c814 c000fdcc a00c0113 ffffffff
[785860.236328] r8:c07d6bd9 r7:ee89ff9c r6:ffffffff r5:a00c0113 r4:c000fdcc r3:c002c814
[785860.256561] [<c000fda4>] (arch_cpu_idle) from [<c006cb7c>] (cpu_startup_entry+0x100/0x154)
[785860.264251] [<c006ca7c>] (cpu_startup_entry) from [<c00146b8>] (secondary_start_kernel+0x134/0x140)
[785860.270416] r7:c07d7080 r3:00000001
[785860.270416] [<c0014584>] (secondary_start_kernel) from [<800087c4>] (0x800087c4)
[785860.286468] r6:10c0387d r5:00000015 r4:ae88406a r3:c00087ac
$
[-- Attachment #5: dmesg-3.19 --]
[-- Type: application/octet-stream, Size: 34687 bytes --]
$
$
$
$ dmesg
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 3.19.0-panda (rfg@red.dwarf) (gcc version 4.9.2 (Buildroot 2015.02-git-00957-gf0ea0e7) ) #1 SMP PREEMPT Mon Feb 9 20:11:22 AEDT 2015
[ 0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] Machine model: TI OMAP4 PandaBoard-ES
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] On node 0 totalpages: 261888
[ 0.000000] free_area_init_node: node 0, pgdat c07eff40, node_mem_map eeff9000
[ 0.000000] Normal zone: 1520 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 194560 pages, LIFO batch:31
[ 0.000000] HighMem zone: 67328 pages, LIFO batch:15
[ 0.000000] OMAP4460 ES1.1
[ 0.000000] L2C: platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C: DT/platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C-310 enabling early BRESP for Cortex-A9
[ 0.000000] L2C-310 ID prefetch enabled, offset 1 lines
[ 0.000000] OMAP L2C310: ROM does not support power control setting
[ 0.000000] L2C-310 dynamic clock gating disabled, standby mode disabled
[ 0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
[ 0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x7e470000
[ 0.000000] PERCPU: Embedded 9 pages/cpu @eefb2000 s6976 r8192 d21696 u36864
[ 0.000000] pcpu-alloc: s6976 r8192 d21696 u36864 alloc=9*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260368
[ 0.000000] Kernel command line: root=/dev/mmcblk0p2 rootwait console=ttyO2,115200
[ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Memory: 1029824K/1047552K available (5763K kernel code, 251K rwdata, 1824K rodata, 256K init, 266K bss, 17728K reserved, 0K cma-reserved, 269312K highmem)
[ 0.000000] Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
vmalloc : 0xf0000000 - 0xff000000 ( 240 MB)
lowmem : 0xc0000000 - 0xef800000 ( 760 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc07710ac (7589 kB)
.init : 0xc0772000 - 0xc07b2000 ( 256 kB)
.data : 0xc07b2000 - 0xc07f0e40 ( 252 kB)
.bss : 0xc07f0e40 - 0xc08339f0 ( 267 kB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] NR_IRQS:16 nr_irqs:16 16
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node bandgap_fclk
[ 0.000000] OMAP clockevent source: timer1 at 32768 Hz
[ 0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65536000000000ns
[ 0.000030] OMAP clocksource: 32k_counter at 32768 Hz
[ 0.001556] Console: colour dummy device 80x30
[ 0.001586] Calibrating delay loop... 1397.55 BogoMIPS (lpj=6987776)
[ 0.090270] pid_max: default: 32768 minimum: 301
[ 0.090362] Security Framework initialized
[ 0.090393] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.090423] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.091064] Initializing cgroup subsys devices
[ 0.091094] Initializing cgroup subsys freezer
[ 0.091094] Initializing cgroup subsys blkio
[ 0.091125] CPU: Testing write buffer coherency: ok
[ 0.091369] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.091430] Setting up static identity map for 0x805a6760 - 0x805a67b8
[ 0.190246] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.190338] Brought up 2 CPUs
[ 0.190368] SMP: Total of 2 processors activated (2795.11 BogoMIPS).
[ 0.190368] CPU: All CPU(s) started in SVC mode.
[ 0.190887] devtmpfs: initialized
[ 0.191558] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[ 0.193237] omap_hwmod: l3_main_3 using broken dt data from ocp
[ 0.194732] omap_hwmod: l3_main_2 using broken dt data from ocp
[ 0.289825] xor: measuring software checksum speed
[ 0.380249] arm4regs : 1213.600 MB/sec
[ 0.480255] 8regs : 858.800 MB/sec
[ 0.580230] 32regs : 808.800 MB/sec
[ 0.580261] xor: using function: arm4regs (1213.600 MB/sec)
[ 0.580291] pinctrl core: initialized pinctrl subsystem
[ 0.597381] NET: Registered protocol family 16
[ 0.598144] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.605346] OMAP GPIO hardware version 0.1
[ 0.609802] omap-gpmc 50000000.gpmc: GPMC revision 6.0
[ 0.610351] irq: no irq domain found for /ocp/pinmux@4a100040 !
[ 0.610717] irq: no irq domain found for /ocp/pinmux@4a100040 !
[ 0.611053] irq: no irq domain found for /ocp/pinmux@4a100040 !
[ 0.622283] platform 4b501000.aes: Cannot lookup hwmod 'aes'
[ 0.622558] platform 480a5000.des: Cannot lookup hwmod 'des'
[ 0.628112] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[ 0.628112] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.629119] OMAP DMA hardware revision 0.0
[ 0.840698] raid6: int32x1 103 MB/s
[ 1.010406] raid6: int32x2 138 MB/s
[ 1.180419] raid6: int32x4 145 MB/s
[ 1.350341] raid6: int32x8 160 MB/s
[ 1.350341] raid6: using algorithm int32x8 (160 MB/s)
[ 1.350372] raid6: using intx1 recovery algorithm
[ 1.372222] omap-dma-engine 4a056000.dma-controller: OMAP DMA engine driver
[ 1.450408] reg-fixed-voltage wl12xx_vmmc: could not find pctldev for node /ocp/pinmux@4a100040/pinmux_wl12xx_gpio, deferring probe
[ 1.450439] platform wl12xx_vmmc: Driver reg-fixed-voltage requests probe deferral
[ 1.453643] omap-iommu 4a066000.mmu: 4a066000.mmu registered
[ 1.453765] omap-iommu 55082000.mmu: 55082000.mmu registered
[ 1.454711] SCSI subsystem initialized
[ 1.454986] usbcore: registered new interface driver usbfs
[ 1.455139] usbcore: registered new interface driver hub
[ 1.455261] usbcore: registered new device driver usb
[ 1.456207] omap_i2c 48070000.i2c: could not find pctldev for node /ocp/pinmux@4a100040/pinmux_i2c1_pins, deferring probe
[ 1.456237] platform 48070000.i2c: Driver omap_i2c requests probe deferral
[ 1.456268] omap_i2c 48072000.i2c: could not find pctldev for node /ocp/pinmux@4a100040/pinmux_i2c2_pins, deferring probe
[ 1.456298] platform 48072000.i2c: Driver omap_i2c requests probe deferral
[ 1.456329] omap_i2c 48060000.i2c: could not find pctldev for node /ocp/pinmux@4a100040/pinmux_i2c3_pins, deferring probe
[ 1.456359] platform 48060000.i2c: Driver omap_i2c requests probe deferral
[ 1.456390] omap_i2c 48350000.i2c: could not find pctldev for node /ocp/pinmux@4a100040/pinmux_i2c4_pins, deferring probe
[ 1.456390] platform 48350000.i2c: Driver omap_i2c requests probe deferral
[ 1.456665] Linux video capture interface: v2.00
[ 1.457519] Advanced Linux Sound Architecture Driver Initialized.
[ 1.458740] Switched to clocksource 32k_counter
[ 1.470886] NET: Registered protocol family 2
[ 1.471649] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[ 1.471740] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[ 1.471862] TCP: Hash tables configured (established 8192 bind 8192)
[ 1.471923] TCP: reno registered
[ 1.471954] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 1.471984] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 1.472198] NET: Registered protocol family 1
[ 1.474029] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[ 1.476623] futex hash table entries: 512 (order: 3, 32768 bytes)
[ 1.485382] ntfs: driver 2.1.31 [Flags: R/O].
[ 1.485656] fuse init (API version 7.23)
[ 1.487609] bounce: pool size: 64 pages
[ 1.487640] io scheduler noop registered
[ 1.487670] io scheduler deadline registered
[ 1.487701] io scheduler cfq registered (default)
[ 1.488555] pinctrl-single 4a100040.pinmux: 203 pins at pa fc100040 size 406
[ 1.488983] pinctrl-single 4a31e040.pinmux: 28 pins at pa fc31e040 size 56
[ 1.490509] OMAP DSS rev 4.0
[ 1.492828] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 1.494323] omap_uart 4806a000.serial: no wakeirq for uart0
[ 1.494445] 4806a000.serial: ttyO0 at MMIO 0x4806a000 (irq = 226, base_baud = 3000000) is a OMAP UART0
[ 1.494964] 4806c000.serial: ttyO1 at MMIO 0x4806c000 (irq = 227, base_baud = 3000000) is a OMAP UART1
[ 1.495452] 48020000.serial: ttyO2 at MMIO 0x48020000 (irq = 228, base_baud = 3000000) is a OMAP UART2
[ 2.321533] console [ttyO2] enabled
[ 2.325683] 4806e000.serial: ttyO3 at MMIO 0x4806e000 (irq = 229, base_baud = 3000000) is a OMAP UART3
[ 2.336242] [drm] Initialized drm 1.1.0 20060810
[ 2.343139] loop: module loaded
[ 2.348663] usbcore: registered new interface driver smsc95xx
[ 2.354766] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 2.361633] ehci-omap: OMAP-EHCI Host Controller driver
[ 2.378814] ehci-omap 4a064c00.ehci: EHCI Host Controller
[ 2.384460] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1
[ 2.392547] ehci-omap 4a064c00.ehci: irq 336, io mem 0x4a064c00
[ 2.418762] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00
[ 2.425140] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 2.432250] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 2.439819] usb usb1: Product: EHCI Host Controller
[ 2.444915] usb usb1: Manufacturer: Linux 3.19.0-panda ehci_hcd
[ 2.451110] usb usb1: SerialNumber: 4a064c00.ehci
[ 2.456634] hub 1-0:1.0: USB hub found
[ 2.460632] hub 1-0:1.0: 3 ports detected
[ 2.465484] usbcore: registered new interface driver cdc_wdm
[ 2.471527] usbcore: registered new interface driver usb-storage
[ 2.478210] i2c /dev entries driver
[ 2.481994] usbcore: registered new interface driver uvcvideo
[ 2.488006] USB Video Class driver (1.1.1)
[ 2.493469] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[ 2.500976] device-mapper: uevent: version 1.0.3
[ 2.506103] device-mapper: ioctl: 4.29.0-ioctl (2014-10-28) initialised: dm-devel@redhat.com
[ 2.515106] Driver 'mmcblk' needs updating - please use bus_type methods
[ 2.522735] omap_hsmmc 4809c000.mmc: unable to get vmmc regulator -517
[ 2.529632] platform 4809c000.mmc: Driver omap_hsmmc requests probe deferral
[ 2.537353] omap_hsmmc 480d5000.mmc: unable to get vmmc regulator -517
[ 2.544250] platform 480d5000.mmc: Driver omap_hsmmc requests probe deferral
[ 2.552612] hidraw: raw HID events driver (C) Jiri Kosina
[ 2.558502] usbcore: registered new interface driver usbhid
[ 2.564453] usbhid: USB HID core driver
[ 2.569000] emif 4c000000.emif: emif_probe: device configured with addr = f00a8000 and IRQ247
[ 2.578155] emif 4d000000.emif: emif_probe: device configured with addr = f00aa000 and IRQ248
[ 2.589080] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 2.596923] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 2.603973] platform sound: Driver omap-abe-twl6040 requests probe deferral
[ 2.611541] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 2.617156] TCP: cubic registered
[ 2.620666] Initializing XFRM netlink socket
[ 2.625152] NET: Registered protocol family 17
[ 2.630035] twl: not initialized
[ 2.633422] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 2.641784] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 2.650177] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 2.658538] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 2.667022] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 2.675415] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 2.683807] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1410000 Vs max 1316660
[ 2.692199] omap2_set_init_voltage: unable to find boot up OPP for vdd_mpu
[ 2.699371] omap2_set_init_voltage: unable to set vdd_mpu
[ 2.705017] omap2_set_init_voltage: unable to find boot up OPP for vdd_core
[ 2.712310] omap2_set_init_voltage: unable to set vdd_core
[ 2.718048] omap2_set_init_voltage: unable to find boot up OPP for vdd_iva
[ 2.725524] omap2_set_init_voltage: unable to set vdd_iva
[ 2.732025] Power Management for TI OMAP4+ devices.
[ 2.737121] OMAP4 PM: u-boot >= v2012.07 is required for full PM support
[ 2.744232] ThumbEE CPU extension supported.
[ 2.748748] Registering SWP/SWPB emulation handler
[ 2.753814] SmartReflex Class3 initialized
[ 2.760711] Btrfs loaded
[ 2.766296] dmm 4e000000.dmm: initialized all PAT entries
[ 2.772247] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 2.781585] Skipping twl internal clock init and using bootloader value (unknown osc rate)
[ 2.798828] usb 1-1: new high-speed USB device number 2 using ehci-omap
[ 2.812011] twl 0-0048: PIH (irq 340) nested IRQs
[ 2.818420] twl_rtc 48070000.i2c:twl@48:rtc: Enabling TWL-RTC
[ 2.827239] twl_rtc 48070000.i2c:twl@48:rtc: rtc core: registered 48070000.i2c:twl@48 as rtc0
[ 2.846832] twl6040 0-004b: clk32k is not handled
[ 2.852874] twl6040 0-004b: Failed to read revision register: -121
[ 2.859588] twl6040: probe of 0-004b failed with error -121
[ 2.865112] omap_i2c 48070000.i2c: bus 0 rev0.11 at 400 kHz
[ 2.872100] omap_i2c 48072000.i2c: bus 1 rev0.11 at 400 kHz
[ 2.879180] omap_i2c 48060000.i2c: bus 2 rev0.11 at 100 kHz
[ 2.885772] omap_i2c 48350000.i2c: bus 3 rev0.11 at 400 kHz
[ 2.964263] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
[ 2.971374] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 2.979614] hub 1-1:1.0: USB hub found
[ 2.983795] hub 1-1:1.0: 5 ports detected
[ 3.053039] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 3.061279] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 3.068328] platform sound: Driver omap-abe-twl6040 requests probe deferral
[ 3.075653] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 3.083801] twl_rtc 48070000.i2c:twl@48:rtc: setting system clock to 2015-02-09 09:19:30 UTC (1423473570)
[ 3.093841] sr_init: No PMIC hook to init smartreflex
[ 3.099212] smartreflex smartreflex.0: omap_sr_probe: SmartReflex driver initialized
[ 3.107513] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 3.115783] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 3.115783] smartreflex smartreflex.1: omap_sr_probe: SmartReflex driver initialized
[ 3.115997] smartreflex smartreflex.2: omap_sr_probe: SmartReflex driver initialized
[ 3.139251] platform sound: Driver omap-abe-twl6040 requests probe deferral
[ 3.146820] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 3.153778] VUSB: disabling
[ 3.154144] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 3.154144] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 3.154205] platform sound: Driver omap-abe-twl6040 requests probe deferral
[ 3.174560] mmc0: host does not support reading read-only switch, assuming write-enable
[ 3.178131] mmc0: new high speed SDHC card at address aaaa
[ 3.188934] mmcblk0: mmc0:aaaa SS16G 14.8 GiB
[ 3.198303] VANA: disabling
[ 3.201690] VDAC: disabling
[ 3.206604] ALSA device list:
[ 3.209869] mmcblk0: p1 p2 p3 p4
[ 3.213470] No soundcards found.
[ 3.218078] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 3.227264] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 3.229400] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
[ 3.229919] EXT4-fs (mmcblk0p2): mounting ext2 file system using the ext4 subsystem
[ 3.230072] omap_hsmmc 480d5000.mmc: card claims to support voltages below defined range
[ 3.240295] mmc1: queuing unknown CIS tuple 0x91 (3 bytes)
[ 3.241271] mmc1: new SDIO card at address 0001
[ 3.270996] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 3.272949] EXT4-fs (mmcblk0p2): mounted filesystem without journal. Opts: (null)
[ 3.272949] VFS: Mounted root (ext2 filesystem) readonly on device 179:2.
[ 3.287322] devtmpfs: mounted
[ 3.287628] Freeing unused kernel memory: 256K (c0772000 - c07b2000)
[ 3.302764] platform sound: Driver omap-abe-twl6040 requests probe deferral
[ 3.389678] usb 1-1.1: new high-speed USB device number 3 using ehci-omap
[ 3.519439] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[ 3.526397] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 3.537170] smsc95xx v1.0.4
[ 3.541809] EXT4-fs (mmcblk0p2): warning: mounting unchecked fs, running e2fsck is recommended
[ 3.554199] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 3.625549] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-4a064c00.ehci-1.1, smsc95xx USB 2.0 Ethernet, 4a:2a:af:53:c8:b9
[ 3.640411] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 3.647827] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 3.656585] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 3.664093] platform sound: Driver omap-abe-twl6040 requests probe deferral
[ 3.672210] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 3.676086] random: dd urandom read with 20 bits of entropy available
[ 3.686920] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 3.694946] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 3.702117] platform sound: Driver omap-abe-twl6040 requests probe deferral
[ 8.096282] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[ 18.093841] Bluetooth: Core ver 2.20
[ 18.098876] NET: Registered protocol family 31
[ 18.103759] Bluetooth: HCI device and connection manager initialized
[ 18.110839] Bluetooth: HCI socket layer initialized
[ 18.115997] Bluetooth: L2CAP socket layer initialized
[ 18.121551] Bluetooth: SCO socket layer initialized
[ 23.821105] RPC: Registered named UNIX socket transport module.
[ 23.827514] RPC: Registered udp transport module.
[ 23.832763] RPC: Registered tcp transport module.
[ 23.832763] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 23.901184] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 23.915374] svc: failed to register nfsdv2 RPC service (errno 111).
[ 23.922271] svc: failed to register nfsaclv2 RPC service (errno 111).
[ 96.279296] cfg80211: Calling CRDA to update world regulatory domain
[ 96.354003] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 96.361755] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 96.370208] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 96.377716] platform sound: Driver omap-abe-twl6040 requests probe deferral
[ 96.385223] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 96.392242] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 96.400451] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 96.402008] platform sound: Driver omap-abe-twl6040 requests probe deferral
[ 96.428741] random: nonblocking pool is initialized
[ 96.652435] wlcore: loaded
[ 117.599517] wlcore: firmware booted (Rev 6.3.10.0.133)
[ 120.692840] wlan0: authenticate with 90:f6:52:81:ee:70
[ 120.739807] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[ 120.748016] wlan0: authenticated
[ 120.758758] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[ 120.769531] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[ 120.787017] wlan0: associated
[ 120.838531] wlcore: Association completed.
[ 3538.725128] sched: RT throttling activated
[18129.342376] INFO: rcu_preempt self-detected stall on CPU
[18129.342376] INFO: rcu_preempt detected stalls on CPUs/tasks:
[18129.342407] {
[18129.342407] 1
[18129.342407] }
[18129.342407] (detected by 0, t=5521 jiffies, g=56712, c=56711, q=77)
[18129.342407] Task dump for CPU 1:
[18129.342407] swapper/1 R
[18129.342407] running
[18129.342437] 0 0 1 0x00000000
[18129.342437] Backtrace:
[18129.342437] [<ee899fac>] (0xee899fac) from [<ee899fb8>] (0xee899fb8)
[18129.377960] {
[18129.380096] 1} (t=5525 jiffies g=56712 c=56711 q=79)
[18129.384124] Task dump for CPU 1:
[18129.387481] swapper/1 R running 0 0 1 0x00000002
[18129.394134] Backtrace:
[18129.394165] [<c00127e0>] (dump_backtrace) from [<c0012a04>] (show_stack+0x18/0x1c)
[18129.394165] r6:80070193 r5:00000001 r4:ee872f80 r3:00000000
[18129.394165] [<c00129ec>] (show_stack) from [<c005a914>] (sched_show_task+0xc0/0x114)
[18129.394165] [<c005a854>] (sched_show_task) from [<c005db18>] (dump_cpu_task+0x40/0x44)
[18129.394165] r5:c07caf80 r4:00000001
[18129.394165] [<c005dad8>] (dump_cpu_task) from [<c007c98c>] (rcu_dump_cpu_stacks+0x94/0xd0)
[18129.439300] r4:00000002 r3:00000001
[18129.439300] [<c007c8f8>] (rcu_dump_cpu_stacks) from [<c008032c>] (rcu_check_callbacks+0x47c/0x7e8)
[18129.439300] r8:2e80b000 r7:c07ba504 r6:c07caf80 r5:c07b1740 r4:eefbc740 r3:ee898000
[18129.439300] [<c007feb0>] (rcu_check_callbacks) from [<c00831f8>] (update_process_times+0x40/0x6c)
[18129.469787] r10:00000001 r9:eefbb368 r8:eefbb368 r7:0000107d r6:127defe6 r5:00000000
[18129.469787] r4:ee872f80
[18129.480651] [<c00831b8>] (update_process_times) from [<c0093428>] (tick_sched_handle.isra.17+0x4c/0x50)
[18129.480651] r5:ee899f70 r4:eefbb5f8
[18129.480651] [<c00933dc>] (tick_sched_handle.isra.17) from [<c0093480>] (tick_sched_timer+0x54/0x98)
[18129.480651] [<c009342c>] (tick_sched_timer) from [<c0083a88>] (__run_hrtimer.isra.34+0x4c/0xdc)
[18129.512725] r7:c009342c r6:eefbb368 r5:eefbb3a0 r4:eefbb5f8
[18129.512725] [<c0083a3c>] (__run_hrtimer.isra.34) from [<c0084258>] (hrtimer_interrupt+0x124/0x2d4)
[18129.528015] r7:0000107d r6:127defe6 r5:00000000 r4:eefbb368
[18129.528015] [<c0084134>] (hrtimer_interrupt) from [<c0015708>] (twd_handler+0x38/0x48)
[18129.542205] r10:00000000 r9:c07f0ce1 r8:00000011 r7:ee821ac0 r6:ee803d80 r5:c07d2b8c
[18129.542205] r4:00000001
[18129.542205] [<c00156d0>] (twd_handler) from [<c0077e60>] (handle_percpu_devid_irq+0x74/0x8c)
[18129.542205] r4:eefbec80 r3:c00156d0
[18129.561889] [<c0077dec>] (handle_percpu_devid_irq) from [<c0073f2c>] (generic_handle_irq+0x34/0x44)
[18129.575073] r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011 r3:c0077dec
[18129.575073] [<c0073ef8>] (generic_handle_irq) from [<c0074204>] (__handle_domain_irq+0x64/0xbc)
[18129.575073] r4:c07b0a60 r3:00000159
[18129.575073] [<c00741a0>] (__handle_domain_irq) from [<c0008710>] (gic_handle_irq+0x2c/0x64)
[18129.604736] r8:fa240100 r7:ee899f70 r6:c07ba90c r5:0000000d r4:fa24010c r3:ee899f70
[18129.604736] [<c00086e4>] (gic_handle_irq) from [<c0013580>] (__irq_svc+0x40/0x74)
[18129.604736] Exception stack(0xee899f70 to 0xee899fb8)
[18129.604736] 9f60: 00000000 00000000 00de1460 c002ab04
[18129.634490] 9f80: ee898000 c07ba48c c07ba4d8 c05ab354 c07f0ce1 c07f0ce1 00000000 ee899fc4
[18129.634490] 9fa0: ee899fb8 ee899fb8 c002a714 c000fe1c a0070113 ffffffff
[18129.634490] r8:c07f0ce1 r7:ee899fa4 r6:ffffffff r5:a0070113 r4:c000fe1c r3:c002a714
[18129.649932] [<c000fdf4>] (arch_cpu_idle) from [<c006b5e4>] (cpu_startup_entry+0x100/0x154)
[18129.649932] [<c006b4e4>] (cpu_startup_entry) from [<c00147f4>] (secondary_start_kernel+0x130/0x13c)
[18129.649932] r7:c07f1180 r3:ee898000
[18129.649932] [<c00146c4>] (secondary_start_kernel) from [<800087e4>] (0x800087e4)
[18129.687591] r5:00000015 r4:ae87c06a
[18575.265441] INFO: rcu_preempt self-detected stall on CPU
[18575.265441] INFO: rcu_preempt self-detected stall on CPU
[18575.265441] {
[18575.265441] 0
[18575.265441] }
[18575.265441] (t=6131 jiffies g=57941 c=57940 q=200)
[18575.265441] Task dump for CPU 0:
[18575.265441] swapper/0 R
[18575.265502] running
[18575.265502] 0 0 0 0x00000000
[18575.265502] Backtrace:
[18575.265533] [<c00127e0>] (dump_backtrace) from [<c0012a04>] (show_stack+0x18/0x1c)
[18575.265533] r6:80070193
[18575.265533] r5:00000000
[18575.265533] r4:c07bcfa8
[18575.265533] r3:00000000
[18575.265563] [<c00129ec>] (show_stack) from [<c005a914>] (sched_show_task+0xc0/0x114)
[18575.265594] [<c005a854>] (sched_show_task) from [<c005db18>] (dump_cpu_task+0x40/0x44)
[18575.265594] r5:c07caf80
[18575.265594] r4:00000000
[18575.265624] [<c005dad8>] (dump_cpu_task) from [<c007c98c>] (rcu_dump_cpu_stacks+0x94/0xd0)
[18575.265624] r4:00000001
[18575.265624] r3:00000003
[18575.265655] [<c007c8f8>] (rcu_dump_cpu_stacks) from [<c008032c>] (rcu_check_callbacks+0x47c/0x7e8)
[18575.265655] r8:2e802000
[18575.265655] r7:c07ba504
[18575.265655] r6:c07caf80
[18575.265655] r5:c07b1740
[18575.265655] r4:eefb3740
[18575.265655] r3:c07b2000
[18575.265686] [<c007feb0>] (rcu_check_callbacks) from [<c00831f8>] (update_process_times+0x40/0x6c)
[18575.265686] r10:00000001
[18575.265686] r9:eefb2368
[18575.265686] r8:eefb2368
[18575.265686] r7:000010e4
[18575.265716] r6:e58d7ff1
[18575.265716] r5:00000000
[18575.265716] r4:c07bcfa8
[18575.265716] [<c00831b8>] (update_process_times) from [<c0093428>] (tick_sched_handle.isra.17+0x4c/0x50)
[18575.265747] r5:c07b3f28
[18575.265747] r4:eefb25f8
[18575.265747] [<c00933dc>] (tick_sched_handle.isra.17) from [<c0093480>] (tick_sched_timer+0x54/0x98)
[18575.265777] [<c009342c>] (tick_sched_timer) from [<c0083a88>] (__run_hrtimer.isra.34+0x4c/0xdc)
[18575.265777] r7:c009342c
[18575.265777] r6:eefb2368
[18575.265777] r5:eefb23a0
[18575.265777] r4:eefb25f8
[18575.265808] [<c0083a3c>] (__run_hrtimer.isra.34) from [<c0084258>] (hrtimer_interrupt+0x124/0x2d4)
[18575.265808] r7:000010e4
[18575.265808] r6:e58d7ff1
[18575.265808] r5:00000000
[18575.265808] r4:eefb2368
[18575.265838] [<c0084134>] (hrtimer_interrupt) from [<c0015708>] (twd_handler+0x38/0x48)
[18575.265838] r10:eefc6800
[18575.265838] r9:c07f0ce1
[18575.265838] r8:00000011
[18575.265838] r7:ee821ac0
[18575.265838] r6:ee803d80
[18575.265838] r5:c07d2b8c
[18575.265838] r4:00000001
[18575.265869] [<c00156d0>] (twd_handler) from [<c0077e60>] (handle_percpu_devid_irq+0x74/0x8c)
[18575.265869] r4:eefb5c80
[18575.265869] r3:c00156d0
[18575.265899] [<c0077dec>] (handle_percpu_devid_irq) from [<c0073f2c>] (generic_handle_irq+0x34/0x44)
[18575.265899] r8:ee808000
[18575.265899] r7:00000001
[18575.265899] r6:00000000
[18575.265930] r5:00000000
[18575.265930] r4:00000011
[18575.265930] r3:c0077dec
[18575.265930] [<c0073ef8>] (generic_handle_irq) from [<c0074204>] (__handle_domain_irq+0x64/0xbc)
[18575.265930] r4:c07b0a60
[18575.265930] r3:00000159
[18575.265960] [<c00741a0>] (__handle_domain_irq) from [<c0008710>] (gic_handle_irq+0x2c/0x64)
[18575.265960] r8:fa240100
[18575.265960] r7:c07b3f28
[18575.265960] r6:c07ba90c
[18575.265960] r5:0000000d
[18575.265991] r4:fa24010c
[18575.265991] r3:c07b3f28
[18575.265991] [<c00086e4>] (gic_handle_irq) from [<c0013580>] (__irq_svc+0x40/0x74)
[18575.265991] Exception stack(0xc07b3f28 to 0xc07b3f70)
[18575.266021] 3f20: 00000000 00000000 01815340 c002ab04 c07b2000 c07ba48c
[18575.266021] 3f40: c07ba4d8 c05ab354 c07f0ce1 c07f0ce1 eefc6800 c07b3f7c c07b3f70 c07b3f70
[18575.266021] 3f60: c002a714 c000fe1c a0070113 ffffffff
[18575.266052] r8:c07f0ce1
[18575.266052] r7:c07b3f5c
[18575.266052] r6:ffffffff
[18575.266052] r5:a0070113
[18575.266052] r4:c000fe1c
[18575.266052] r3:c002a714
[18575.266082] [<c000fdf4>] (arch_cpu_idle) from [<c006b5e4>] (cpu_startup_entry+0x100/0x154)
[18575.266082] [<c006b4e4>] (cpu_startup_entry) from [<c059e158>] (rest_init+0x8c/0x90)
[18575.266082] r7:ffffffff
[18575.266082] r3:c07b2000
[18575.266082] [<c059e0cc>] (rest_init) from [<c0772cb4>] (start_kernel+0x368/0x374)
[18575.266143] r4:c07ba5a0
[18575.266143] r3:c07b2000
[18575.266143] [<c077294c>] (start_kernel) from [<80008074>] (0x80008074)
[18575.266143] Task dump for CPU 1:
[18575.266174] swapper/1 R
[18575.266174] running
[18575.266174] 0 0 1 0x00000000
[18575.266174] Backtrace:
[18575.266174] [<ee899fac>] (0xee899fac) from [<ee899fb8>] (0xee899fb8)
[18575.610168] {
[18575.610168] 1} (t=6160 jiffies g=57941 c=57940 q=200)
[18575.615386] Task dump for CPU 1:
[18575.619384] swapper/1 R running 0 0 1 0x00000002
[18575.619384] Backtrace:
[18575.628601] [<c00127e0>] (dump_backtrace) from [<c0012a04>] (show_stack+0x18/0x1c)
[18575.628601] r6:800c0193 r5:00000001 r4:ee872f80 r3:00000000
[18575.628601] [<c00129ec>] (show_stack) from [<c005a914>] (sched_show_task+0xc0/0x114)
[18575.642456] [<c005a854>] (sched_show_task) from [<c005db18>] (dump_cpu_task+0x40/0x44)
[18575.642456] r5:c07caf80 r4:00000001
[18575.662567] [<c005dad8>] (dump_cpu_task) from [<c007c98c>] (rcu_dump_cpu_stacks+0x94/0xd0)
[18575.662567] r4:00000002 r3:00000001
[18575.662567] [<c007c8f8>] (rcu_dump_cpu_stacks) from [<c008032c>] (rcu_check_callbacks+0x47c/0x7e8)
[18575.662567] r8:2e80b000 r7:c07ba504 r6:c07caf80 r5:c07b1740 r4:eefbc740 r3:ee898000
[18575.684295] [<c007feb0>] (rcu_check_callbacks) from [<c00831f8>] (update_process_times+0x40/0x6c)
[18575.684295] r10:00000001 r9:eefbb368 r8:eefbb368 r7:000010e4 r6:e58d7ff1 r5:00000000
[18575.709869] r4:ee872f80
[18575.709869] [<c00831b8>] (update_process_times) from [<c0093428>] (tick_sched_handle.isra.17+0x4c/0x50)
[18575.709869] r5:ee899f70 r4:eefbb5f8
[18575.726074] [<c00933dc>] (tick_sched_handle.isra.17) from [<c0093480>] (tick_sched_timer+0x54/0x98)
[18575.735534] [<c009342c>] (tick_sched_timer) from [<c0083a88>] (__run_hrtimer.isra.34+0x4c/0xdc)
[18575.735534] r7:c009342c r6:eefbb368 r5:eefbb3a0 r4:eefbb5f8
[18575.750549] [<c0083a3c>] (__run_hrtimer.isra.34) from [<c0084258>] (hrtimer_interrupt+0x124/0x2d4)
[18575.750549] r7:000010e4 r6:e58d7ff1 r5:00000000 r4:eefbb368
[18575.750549] [<c0084134>] (hrtimer_interrupt) from [<c0015708>] (twd_handler+0x38/0x48)
[18575.765838] r10:00000000 r9:c07f0ce1 r8:00000011 r7:ee821ac0 r6:ee803d80 r5:c07d2b8c
[18575.782287] r4:00000001
[18575.782287] [<c00156d0>] (twd_handler) from [<c0077e60>] (handle_percpu_devid_irq+0x74/0x8c)
[18575.793762] r4:eefbec80 r3:c00156d0
[18575.793762] [<c0077dec>] (handle_percpu_devid_irq) from [<c0073f2c>] (generic_handle_irq+0x34/0x44)
[18575.793762] r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:00000011 r3:c0077dec
[18575.793762] [<c0073ef8>] (generic_handle_irq) from [<c0074204>] (__handle_domain_irq+0x64/0xbc)
[18575.824157] r4:c07b0a60 r3:00000159
[18575.824157] [<c00741a0>] (__handle_domain_irq) from [<c0008710>] (gic_handle_irq+0x2c/0x64)
[18575.836608] r8:fa240100 r7:ee899f70 r6:c07ba90c r5:0000000d r4:fa24010c r3:ee899f70
[18575.836608] [<c00086e4>] (gic_handle_irq) from [<c0013580>] (__irq_svc+0x40/0x74)
[18575.852539] Exception stack(0xee899f70 to 0xee899fb8)
[18575.857818] 9f60: 00000000 00000000 00e2d49a c002ab04
[18575.857818] 9f80: ee898000 c07ba48c c07ba4d8 c05ab354 c07f0ce1 c07f0ce1 00000000 ee899fc4
[18575.857818] 9fa0: ee899fb8 ee899fb8 c002a714 c000fe1c a00c0113 ffffffff
[18575.857818] r8:c07f0ce1 r7:ee899fa4 r6:ffffffff r5:a00c0113 r4:c000fe1c r3:c002a714
[18575.857818] [<c000fdf4>] (arch_cpu_idle) from [<c006b5e4>] (cpu_startup_entry+0x100/0x154)
[18575.898559] [<c006b4e4>] (cpu_startup_entry) from [<c00147f4>] (secondary_start_kernel+0x130/0x13c)
[18575.898559] r7:c07f1180 r3:ee898000
[18575.898559] [<c00146c4>] (secondary_start_kernel) from [<800087e4>] (0x800087e4)
[18575.898559] r5:00000015 r4:ae87c06a
$ uptime
06:51:45 up 10:32, load average: 0.01, 0.06, 0.04
$
[-- Attachment #6: dmesg-4.3 --]
[-- Type: application/octet-stream, Size: 36244 bytes --]
$
$ uptime
13:16:01 up 1 day, 17:43, load average: 0.13, 0.09, 0.12
$
$
$
$ cat /proc/version
Linux version 4.3.0-panda (rfg@red.dwarf) (gcc version 5.2.0 (Buildroot 2015.11-git-00627-g9731582) ) #1 SMP PREEMPT Mon Nov 2 19:21:52 AEDT 2015
$ dmesg
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 4.3.0-panda (rfg@red.dwarf) (gcc version 5.2.0 (Buildroot 2015.11-git-00627-g9731582) ) #1 SMP PREEMPT Mon Nov 2 19:21:52 AEDT 2015
[ 0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] Machine model: TI OMAP4 PandaBoard-ES
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] OMAP4: Map 0xbfe00000 to fe600000 for dram barrier
[ 0.000000] On node 0 totalpages: 261632
[ 0.000000] free_area_init_node: node 0, pgdat c09335c0, node_mem_map eeff9000
[ 0.000000] Normal zone: 1520 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 194560 pages, LIFO batch:31
[ 0.000000] HighMem zone: 67072 pages, LIFO batch:15
[ 0.000000] OMAP4460 ES1.1
[ 0.000000] PERCPU: Embedded 12 pages/cpu @eefaa000 s17920 r8192 d23040 u49152
[ 0.000000] pcpu-alloc: s17920 r8192 d23040 u49152 alloc=12*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260112
[ 0.000000] Kernel command line: root=/dev/mmcblk0p2 rootwait console=ttyO2,115200
[ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Memory: 1027436K/1046528K available (6731K kernel code, 249K rwdata, 2116K rodata, 292K init, 294K bss, 19092K reserved, 0K cma-reserved, 268288K highmem)
[ 0.000000] Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
vmalloc : 0xf0000000 - 0xff000000 ( 240 MB)
lowmem : 0xc0000000 - 0xef800000 ( 760 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc08ac0d4 (8849 kB)
.init : 0xc08ad000 - 0xc08f6000 ( 292 kB)
.data : 0xc08f6000 - 0xc0934580 ( 250 kB)
.bss : 0xc0937000 - 0xc0980a78 ( 295 kB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 32.
[ 0.000000] NR_IRQS:16 nr_irqs:16 16
[ 0.000000] L2C: platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C: DT/platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C-310 enabling early BRESP for Cortex-A9
[ 0.000000] OMAP L2C310: ROM does not support power control setting
[ 0.000000] L2C-310 dynamic clock gating disabled, standby mode disabled
[ 0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
[ 0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e470000
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node bandgap_fclk
[ 0.000000] OMAP clockevent source: timer1 at 32768 Hz
[ 0.000000] clocksource: 32k_counter: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 58327039986419 ns
[ 0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65535999984741ns
[ 0.000030] OMAP clocksource: 32k_counter at 32768 Hz
[ 0.001739] Console: colour dummy device 80x30
[ 0.001739] Calibrating delay loop... 1397.55 BogoMIPS (lpj=6987776)
[ 0.090270] pid_max: default: 32768 minimum: 301
[ 0.090362] Security Framework initialized
[ 0.090393] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.090393] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.090972] Initializing cgroup subsys io
[ 0.091003] Initializing cgroup subsys devices
[ 0.091033] Initializing cgroup subsys freezer
[ 0.091064] CPU: Testing write buffer coherency: ok
[ 0.091278] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.091339] Setting up static identity map for 0x80008280 - 0x800082d8
[ 0.210266] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.210327] Brought up 2 CPUs
[ 0.210357] SMP: Total of 2 processors activated (2795.11 BogoMIPS).
[ 0.210357] CPU: All CPU(s) started in SVC mode.
[ 0.211151] devtmpfs: initialized
[ 0.227813] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[ 0.229583] omap_hwmod: l3_main_3 using broken dt data from ocp
[ 0.231170] omap_hwmod: l3_main_2 using broken dt data from ocp
[ 0.329162] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.329406] xor: measuring software checksum speed
[ 0.420257] arm4regs : 1006.000 MB/sec
[ 0.520233] 8regs : 803.600 MB/sec
[ 0.620239] 32regs : 760.800 MB/sec
[ 0.620269] xor: using function: arm4regs (1006.000 MB/sec)
[ 0.620483] pinctrl core: initialized pinctrl subsystem
[ 0.622375] NET: Registered protocol family 16
[ 0.623199] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.631988] OMAP GPIO hardware version 0.1
[ 0.636535] omap-gpmc 50000000.gpmc: GPMC revision 6.0
[ 0.636566] gpmc_mem_init: disabling cs 0 mapped at 0x0-0x1000000
[ 0.637084] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.637451] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.637817] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.649475] platform 4b501000.aes: Cannot lookup hwmod 'aes'
[ 0.649780] platform 480a5000.des: Cannot lookup hwmod 'des'
[ 0.655822] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[ 0.655853] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.656982] omap4_sram_init:Unable to get sram pool needed to handle errata I688
[ 0.657196] OMAP DMA hardware revision 0.0
[ 0.890319] raid6: int32x1 gen() 103 MB/s
[ 1.060577] raid6: int32x1 xor() 82 MB/s
[ 1.230316] raid6: int32x2 gen() 134 MB/s
[ 1.400451] raid6: int32x2 xor() 98 MB/s
[ 1.570404] raid6: int32x4 gen() 145 MB/s
[ 1.740509] raid6: int32x4 xor() 97 MB/s
[ 1.910339] raid6: int32x8 gen() 160 MB/s
[ 2.080535] raid6: int32x8 xor() 94 MB/s
[ 2.080535] raid6: using algorithm int32x8 gen() 160 MB/s
[ 2.080535] raid6: .... xor() 94 MB/s, rmw enabled
[ 2.080535] raid6: using intx1 recovery algorithm
[ 2.087707] omap-dma-engine 4a056000.dma-controller: OMAP DMA engine driver
[ 2.160369] reg-fixed-voltage wl12xx_vmmc: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_wl12xx_gpio, deferring probe
[ 2.164459] omap-iommu 4a066000.mmu: 4a066000.mmu registered
[ 2.164581] omap-iommu 55082000.mmu: 55082000.mmu registered
[ 2.165496] SCSI subsystem initialized
[ 2.165771] usbcore: registered new interface driver usbfs
[ 2.165924] usbcore: registered new interface driver hub
[ 2.166046] usbcore: registered new device driver usb
[ 2.166992] omap_i2c 48070000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c1_pins, deferring probe
[ 2.167053] omap_i2c 48072000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c2_pins, deferring probe
[ 2.167083] omap_i2c 48060000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c3_pins, deferring probe
[ 2.167114] omap_i2c 48350000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c4_pins, deferring probe
[ 2.167388] Linux video capture interface: v2.00
[ 2.168151] Advanced Linux Sound Architecture Driver Initialized.
[ 2.169555] clocksource: Switched to clocksource 32k_counter
[ 2.183593] NET: Registered protocol family 2
[ 2.184173] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[ 2.184295] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[ 2.184417] TCP: Hash tables configured (established 8192 bind 8192)
[ 2.184509] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 2.184570] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 2.184753] NET: Registered protocol family 1
[ 2.185119] RPC: Registered named UNIX socket transport module.
[ 2.185119] RPC: Registered udp transport module.
[ 2.185119] RPC: Registered tcp transport module.
[ 2.185150] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 2.187072] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[ 2.189178] futex hash table entries: 512 (order: 3, 32768 bytes)
[ 2.208190] NFS: Registering the id_resolver key type
[ 2.208221] Key type id_resolver registered
[ 2.208221] Key type id_legacy registered
[ 2.208251] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 2.208282] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 2.219512] ntfs: driver 2.1.32 [Flags: R/O].
[ 2.219909] fuse init (API version 7.23)
[ 2.223205] jitterentropy: Initialization failed with host not compliant with requirements: 2
[ 2.223297] bounce: pool size: 64 pages
[ 2.223297] io scheduler noop registered
[ 2.223327] io scheduler deadline registered
[ 2.223358] io scheduler cfq registered (default)
[ 2.224243] pinctrl-single 4a100040.pinmux: 203 pins at pa fc100040 size 406
[ 2.224456] pinctrl-single 4a31e040.pinmux: 28 pins at pa fc31e040 size 56
[ 2.227142] 58000000.dss supply vdda_video not found, using dummy regulator
[ 2.227264] OMAP DSS rev 4.0
[ 2.227416] omapdss_dss 58000000.dss: bound 58001000.dispc (ops dispc_component_ops)
[ 2.227661] omapdss_dss 58000000.dss: bound 58005000.encoder (ops dsi_component_ops)
[ 2.228118] omapdss_dss 58000000.dss: bound 58006000.encoder (ops hdmi4_component_ops)
[ 2.229217] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 2.230865] omap_uart 4806a000.serial: no wakeirq for uart0
[ 2.230987] 4806a000.serial: ttyO0 at MMIO 0x4806a000 (irq = 226, base_baud = 3000000) is a OMAP UART0
[ 2.231475] 4806c000.serial: ttyO1 at MMIO 0x4806c000 (irq = 227, base_baud = 3000000) is a OMAP UART1
[ 2.231933] 48020000.serial: ttyO2 at MMIO 0x48020000 (irq = 228, base_baud = 3000000) is a OMAP UART2
[ 3.190063] console [ttyO2] enabled
[ 3.194213] 4806e000.serial: ttyO3 at MMIO 0x4806e000 (irq = 229, base_baud = 3000000) is a OMAP UART3
[ 3.204986] [drm] Initialized drm 1.1.0 20060810
[ 3.253234] loop: module loaded
[ 3.259002] usbcore: registered new interface driver smsc95xx
[ 3.265106] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 3.271972] ehci-omap: OMAP-EHCI Host Controller driver
[ 3.289611] ehci-omap 4a064c00.ehci: EHCI Host Controller
[ 3.295288] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1
[ 3.303375] ehci-omap 4a064c00.ehci: irq 336, io mem 0x4a064c00
[ 3.329589] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00
[ 3.335968] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 3.343109] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 3.350677] usb usb1: Product: EHCI Host Controller
[ 3.355773] usb usb1: Manufacturer: Linux 4.3.0-panda ehci_hcd
[ 3.361877] usb usb1: SerialNumber: 4a064c00.ehci
[ 3.367462] hub 1-0:1.0: USB hub found
[ 3.371459] hub 1-0:1.0: 3 ports detected
[ 3.376312] usbcore: registered new interface driver cdc_wdm
[ 3.382415] usbcore: registered new interface driver usb-storage
[ 3.389099] i2c /dev entries driver
[ 3.392913] usbcore: registered new interface driver uvcvideo
[ 3.398925] USB Video Class driver (1.1.1)
[ 3.404418] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[ 3.411956] device-mapper: uevent: version 1.0.3
[ 3.417144] device-mapper: ioctl: 4.33.0-ioctl (2015-8-18) initialised: dm-devel@redhat.com
[ 3.490661] hidraw: raw HID events driver (C) Jiri Kosina
[ 3.496520] usbcore: registered new interface driver usbhid
[ 3.502380] usbhid: USB HID core driver
[ 3.506835] emif 4c000000.emif: emif_probe: device configured with addr = f00a8000 and IRQ247
[ 3.516143] emif 4d000000.emif: emif_probe: device configured with addr = f00aa000 and IRQ248
[ 3.527313] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 3.535247] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 3.542755] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 3.548431] Initializing XFRM netlink socket
[ 3.553009] NET: Registered protocol family 17
[ 3.557952] Key type dns_resolver registered
[ 3.562652] twl: not initialized
[ 3.566070] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.574676] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.583190] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.591674] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.600158] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.608642] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.617126] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1410000 Vs max 1316660
[ 3.625640] omap2_set_init_voltage: unable to find boot up OPP for vdd_mpu
[ 3.633148] omap2_set_init_voltage: unable to set vdd_mpu
[ 3.638885] omap2_set_init_voltage: unable to find boot up OPP for vdd_core
[ 3.646270] omap2_set_init_voltage: unable to set vdd_core
[ 3.652099] omap2_set_init_voltage: unable to find boot up OPP for vdd_iva
[ 3.659362] omap2_set_init_voltage: unable to set vdd_iva
[ 3.665985] Power Management for TI OMAP4+ devices.
[ 3.671173] OMAP4 PM: u-boot >= v2012.07 is required for full PM support
[ 3.678375] ThumbEE CPU extension supported.
[ 3.682983] Registering SWP/SWPB emulation handler
[ 3.683013] SmartReflex Class3 initialized
[ 3.692504] usb 1-1: new high-speed USB device number 2 using ehci-omap
[ 3.703735] Btrfs loaded
[ 3.709381] dmm 4e000000.dmm: initialized all PAT entries
[ 3.717864] Skipping twl internal clock init and using bootloader value (unknown osc rate)
[ 3.740112] twl 0-0048: PIH (irq 340) nested IRQs
[ 3.746520] twl_rtc 48070000.i2c:twl@48:rtc: Enabling TWL-RTC
[ 3.755279] twl_rtc 48070000.i2c:twl@48:rtc: rtc core: registered 48070000.i2c:twl@48 as rtc0
[ 3.775146] twl6040 0-004b: clk32k is not handled
[ 3.782806] omap_i2c 48070000.i2c: bus 0 rev0.11 at 400 kHz
[ 3.789489] omap_i2c 48072000.i2c: bus 1 rev0.11 at 400 kHz
[ 3.796417] omap_i2c 48060000.i2c: bus 2 rev0.11 at 100 kHz
[ 3.803283] omap_i2c 48350000.i2c: bus 3 rev0.11 at 400 kHz
[ 3.870880] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
[ 3.879089] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 3.887451] hub 1-1:1.0: USB hub found
[ 3.891754] hub 1-1:1.0: 5 ports detected
[ 4.019531] omap-abe-twl6040 sound: twl6040-legacy <-> 40132000.mcpdm mapping ok
[ 4.030517] twl_rtc 48070000.i2c:twl@48:rtc: setting system clock to 2015-11-02 08:32:10 UTC (1446453130)
[ 4.040588] sr_init: No PMIC hook to init smartreflex
[ 4.046173] smartreflex smartreflex.0: omap_sr_probe: SmartReflex driver initialized
[ 4.054565] smartreflex smartreflex.1: omap_sr_probe: SmartReflex driver initialized
[ 4.062988] smartreflex smartreflex.2: omap_sr_probe: SmartReflex driver initialized
[ 4.081909] VDAC: disabling
[ 4.085357] VANA: disabling
[ 4.088775] VUSB: disabling
[ 4.092102] ALSA device list:
[ 4.095336] #0: PandaBoardES
[ 4.098602] mmc0: host does not support reading read-only switch, assuming write-enable
[ 4.107513] Waiting for root device /dev/mmcblk0p2...
[ 4.113098] mmc0: new high speed SDHC card at address e624
[ 4.119598] mmcblk0: mmc0:e624 SL32G 28.7 GiB
[ 4.154510] mmcblk0: p1 p2 p3 p4
[ 4.164215] omap_hsmmc 480d5000.mmc: card claims to support voltages below defined range
[ 4.182922] mmc1: queuing unknown CIS tuple 0x91 (3 bytes)
[ 4.190002] mmc1: new SDIO card at address 0001
[ 4.202850] usb 1-1.1: new high-speed USB device number 3 using ehci-omap
[ 4.238739] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
[ 4.248046] EXT4-fs (mmcblk0p2): mounting ext2 file system using the ext4 subsystem
[ 4.281005] EXT4-fs (mmcblk0p2): mounted filesystem without journal. Opts: (null)
[ 4.286132] VFS: Mounted root (ext2 filesystem) readonly on device 179:2.
[ 4.301849] devtmpfs: mounted
[ 4.301879] Freeing unused kernel memory: 292K (c08ad000 - c08f6000)
[ 4.355072] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[ 4.362457] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 4.373107] smsc95xx v1.0.4
[ 4.471405] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-4a064c00.ehci-1.1, smsc95xx USB 2.0 Ethernet, 2a:82:11:81:3c:12
[ 4.666351] EXT4-fs (mmcblk0p2): warning: mounting unchecked fs, running e2fsck is recommended
[ 4.720001] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 4.859710] random: dd urandom read with 29 bits of entropy available
[ 9.671081] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[ 20.064331] Bluetooth: Core ver 2.20
[ 20.068206] NET: Registered protocol family 31
[ 20.073333] Bluetooth: HCI device and connection manager initialized
[ 20.080322] Bluetooth: HCI socket layer initialized
[ 20.085754] Bluetooth: L2CAP socket layer initialized
[ 20.091705] Bluetooth: SCO socket layer initialized
[ 27.723480] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[ 27.734954] NFSD: starting 90-second grace period (net c092dbc0)
[ 40.496948] random: nonblocking pool is initialized
[ 77.554626] wlcore: loaded
[ 101.651580] wlcore: firmware booted (Rev 6.3.10.0.133)
[ 105.310150] wlan0: authenticate with 90:f6:52:81:ee:70
[ 105.340179] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[ 105.348236] wlan0: authenticated
[ 105.359619] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[ 105.370391] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[ 105.389373] wlan0: associated
[ 105.451141] wlcore: Association completed.
[ 147.773071] NFSD: Unable to end grace period: -110
[ 261.411010] Adding 2097148k swap on /dev/mmcblk0p3. Priority:-1 extents:1 across:2097148k SS
[16633.358551] INFO: rcu_preempt self-detected stall on CPU
[16633.358551] INFO: rcu_preempt self-detected stall on CPU
[16633.358551] 1: (1 ticks this GP) idle=09f/1/0 softirq=157843/157843 fqs=0
[16633.358551]
[16633.358551] (t=7094 jiffies g=51926 c=51925 q=189)
[16633.358612] rcu_preempt kthread starved for 7094 jiffies! g51926 c51925 f0x0 s3 ->state=0x1
[16633.358612] Task dump for CPU 0:
[16633.358612] swapper/0 R
[16633.358612] running
[16633.358612] 0 0 0 0x00000000
[16633.358642] Backtrace:
[16633.358642] Backtrace aborted due to bad frame pointer <c08f7f54>
[16633.358642] Task dump for CPU 1:
[16633.358642] swapper/1 R
[16633.358642] running
[16633.358642] 0 0 1 0x00000000
[16633.358673] Backtrace:
[16633.358673] [<c0013eac>] (dump_backtrace) from [<c00140a8>] (show_stack+0x18/0x1c)
[16633.358673] r7:c08f8518
[16633.358673] r6:800f0193
[16633.358673] r5:00000001
[16633.358673] r4:ee877200
[16633.358673] [<c0014090>] (show_stack) from [<c005c970>] (sched_show_task+0xbc/0x110)
[16633.358764] [<c005c8b4>] (sched_show_task) from [<c005edd8>] (dump_cpu_task+0x40/0x48)
[16633.358764] r5:c0907800
[16633.358764] r4:00000001
[16633.358795] [<c005ed98>] (dump_cpu_task) from [<c0083028>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[16633.358795] r5:c0907800
[16633.358795] r4:00000002
[16633.358825] [<c0082f8c>] (rcu_dump_cpu_stacks) from [<c00872ac>] (rcu_check_callbacks+0x524/0x908)
[16633.358825] r9:c08f8518
[16633.358825] r8:c0907800
[16633.358825] r7:000000bd
[16633.358825] r6:2e6c5000
[16633.358856] r5:c08f5100
[16633.358856] r4:eefba100
[16633.358856] [<c0086d88>] (rcu_check_callbacks) from [<c0089f38>] (update_process_times+0x40/0x6c)
[16633.358856] r10:c009ae5c
[16633.358856] r9:00000001
[16633.358886] r8:eefb6c4c
[16633.358886] r7:00000f1f
[16633.358886] r6:0c0484d3
[16633.358886] r5:00000000
[16633.358886] r4:ee877200
[16633.358917] [<c0089ef8>] (update_process_times) from [<c009ae58>] (tick_sched_handle+0x50/0x54)
[16633.358917] r5:ee89ff58
[16633.358917] r4:eefb6e40
[16633.358947] [<c009ae08>] (tick_sched_handle) from [<c009aeac>] (tick_sched_timer+0x50/0x94)
[16633.358947] [<c009ae5c>] (tick_sched_timer) from [<c008a94c>] (__hrtimer_run_queues+0x110/0x1a0)
[16633.358947] r7:00000000
[16633.358947] r6:eefb6c40
[16633.358947] r5:eefb6e40
[16633.358978] r4:eefb6c00
[16633.358978] [<c008a83c>] (__hrtimer_run_queues) from [<c008afdc>] (hrtimer_interrupt+0xac/0x1f8)
[16633.358978] r10:eefb6c78
[16633.358978] r9:eefb6c98
[16633.359008] r8:eefb6c14
[16633.359008] r7:eefb6cb8
[16633.359008] r6:ffffffff
[16633.359008] r5:00000003
[16633.359008] r4:eefb6c00
[16633.359039] [<c008af30>] (hrtimer_interrupt) from [<c0017148>] (twd_handler+0x38/0x48)
[16633.359039] r10:c08f84e8
[16633.359039] r9:fa241100
[16633.359039] r8:00000011
[16633.359039] r7:ee828780
[16633.359069] r6:c09158cc
[16633.359069] r5:ee805d80
[16633.359069] r4:00000001
[16633.359069] [<c0017110>] (twd_handler) from [<c007dda8>] (handle_percpu_devid_irq+0x74/0x8c)
[16633.359069] r5:ee805d80
[16633.359069] r4:eefbc740
[16633.359130] [<c007dd34>] (handle_percpu_devid_irq) from [<c0079c54>] (generic_handle_irq+0x2c/0x3c)
[16633.359130] r9:fa241100
[16633.359130] r8:ee808000
[16633.359130] r7:00000001
[16633.359130] r6:00000000
[16633.359130] r5:00000000
[16633.359130] r4:c08f23d0
[16633.359161] [<c0079c28>] (generic_handle_irq) from [<c0079f28>] (__handle_domain_irq+0x64/0xbc)
[16633.359161] [<c0079ec4>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[16633.359161] r9:fa241100
[16633.359161] r8:fa240100
[16633.359191] r7:ee89ff58
[16633.359191] r6:fa24010c
[16633.359191] r5:c08f879c
[16633.359191] r4:c09158c8
[16633.359191] [<c0009470>] (gic_handle_irq) from [<c0014c94>] (__irq_svc+0x54/0x90)
[16633.359222] Exception stack(0xee89ff58 to 0xee89ffa0)
[16633.359222] ff40: 00000000 eefb6390
[16633.359222] ff60: fe600000 00000000 ee89e000 c08f849c c08f1364 c069d320 ee89ffc8 c0934362
[16633.359252] ff80: c08f84e8 ee89ffb4 ee89ff94 ee89ffa8 c002a988 c0010ba8 600f0113 ffffffff
[16633.359252] r9:c0934362
[16633.359252] r8:ee89ffc8
[16633.359252] r7:ee89ff8c
[16633.359252] r6:ffffffff
[16633.359252] r5:600f0113
[16633.359283] r4:c0010ba8
[16633.359283] [<c0010b80>] (arch_cpu_idle) from [<c00707d4>] (default_idle_call+0x28/0x34)
[16633.359313] [<c00707ac>] (default_idle_call) from [<c00708f4>] (cpu_startup_entry+0x114/0x18c)
[16633.359313] [<c00707e0>] (cpu_startup_entry) from [<c001618c>] (secondary_start_kernel+0x160/0x16c)
[16633.359344] r7:c0937288
[16633.359344] r4:c090a880
[16633.359344] [<c001602c>] (secondary_start_kernel) from [<8000958c>] (0x8000958c)
[16633.359344] r5:00000051
[16633.359344] r4:ae88406a
[16633.731079] 0: (1 GPs behind) idle=ed9/1/0 softirq=109281/109282 fqs=35
[16633.737152] (t=7132 jiffies g=51926 c=51925 q=188)
[16633.737152] Task dump for CPU 0:
[16633.748687] swapper/0 R running 0 0 0 0x00000002
[16633.755432] Backtrace:
[16633.755432] [<c0013eac>] (dump_backtrace) from [<c00140a8>] (show_stack+0x18/0x1c)
[16633.755432] r7:c08f8518 r6:800c0193 r5:00000000 r4:c08fabe8
[16633.772064] [<c0014090>] (show_stack) from [<c005c970>] (sched_show_task+0xbc/0x110)
[16633.772064] [<c005c8b4>] (sched_show_task) from [<c005edd8>] (dump_cpu_task+0x40/0x48)
[16633.780273] r5:c0907800 r4:00000000
[16633.792449] [<c005ed98>] (dump_cpu_task) from [<c0083028>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[16633.792449] r5:c0907800 r4:00000001
[16633.804992] [<c0082f8c>] (rcu_dump_cpu_stacks) from [<c00872ac>] (rcu_check_callbacks+0x524/0x908)
[16633.804992] r9:c08f8518 r8:c0907800 r7:000000bc r6:2e6b9000 r5:c08f5100 r4:eefae100
[16633.822692] [<c0086d88>] (rcu_check_callbacks) from [<c0089f38>] (update_process_times+0x40/0x6c)
[16633.822692] r10:c009ae5c r9:00000001 r8:eefaac4c r7:00000f1f r6:0c0484d3 r5:00000000
[16633.840393] r4:c08fabe8
[16633.840393] [<c0089ef8>] (update_process_times) from [<c009ae58>] (tick_sched_handle+0x50/0x54)
[16633.840393] r5:c08f7f10 r4:eefaae40
[16633.856079] [<c009ae08>] (tick_sched_handle) from [<c009aeac>] (tick_sched_timer+0x50/0x94)
[16633.856079] [<c009ae5c>] (tick_sched_timer) from [<c008a94c>] (__hrtimer_run_queues+0x110/0x1a0)
[16633.864929] r7:00000000 r6:eefaac40 r5:eefaae40 r4:eefaac00
[16633.880218] [<c008a83c>] (__hrtimer_run_queues) from [<c008afdc>] (hrtimer_interrupt+0xac/0x1f8)
[16633.880218] r10:eefaac78 r9:eefaac98 r8:eefaac14 r7:eefaacb8 r6:ffffffff r5:00000003
[16633.889526] r4:eefaac00
[16633.889526] [<c008af30>] (hrtimer_interrupt) from [<c0017148>] (twd_handler+0x38/0x48)
[16633.889526] r10:c08f84e8 r9:fa241100 r8:00000011 r7:ee828780 r6:c09158cc r5:ee805d80
[16633.917205] r4:00000001
[16633.917205] [<c0017110>] (twd_handler) from [<c007dda8>] (handle_percpu_devid_irq+0x74/0x8c)
[16633.917205] r5:ee805d80 r4:eefb0740
[16633.932617] [<c007dd34>] (handle_percpu_devid_irq) from [<c0079c54>] (generic_handle_irq+0x2c/0x3c)
[16633.932617] r9:fa241100 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:c08f23d0
[16633.950408] [<c0079c28>] (generic_handle_irq) from [<c0079f28>] (__handle_domain_irq+0x64/0xbc)
[16633.959625] [<c0079ec4>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[16633.959625] r9:fa241100 r8:fa240100 r7:c08f7f10 r6:fa24010c r5:c08f879c r4:c09158c8
[16633.976684] [<c0009470>] (gic_handle_irq) from [<c0014c94>] (__irq_svc+0x54/0x90)
[16633.976684] Exception stack(0xc08f7f10 to 0xc08f7f58)
[16633.976684] 7f00: 00000000 eefaa390 fe600000 00000000
[16633.998596] 7f20: c08f6000 c08f849c c08f1364 c069d320 c08f7f80 c0934362 c08f84e8 c08f7f6c
[16634.007263] 7f40: c08f7f4c c08f7f60 c002a988 c0010ba8 600c0113 ffffffff
[16634.007263] r9:c0934362 r8:c08f7f80 r7:c08f7f44 r6:ffffffff r5:600c0113 r4:c0010ba8
[16634.007263] [<c0010b80>] (arch_cpu_idle) from [<c00707d4>] (default_idle_call+0x28/0x34)
[16634.007263] [<c00707ac>] (default_idle_call) from [<c00708f4>] (cpu_startup_entry+0x114/0x18c)
[16634.040191] [<c00707e0>] (cpu_startup_entry) from [<c069485c>] (rest_init+0x90/0x94)
[16634.040191] r7:ffffffff r4:00000002
[16634.040191] [<c06947cc>] (rest_init) from [<c08adcec>] (start_kernel+0x370/0x37c)
[16634.040191] r5:c0937000 r4:00000000
[16634.063903] [<c08ad97c>] (start_kernel) from [<8000807c>] (0x8000807c)
[31008.610656] INFO: rcu_preempt self-detected stall on CPU
[31008.610687] INFO: rcu_preempt self-detected stall on CPU
[31008.610687] 0: (1 GPs behind) idle=fdf/1/0 softirq=207429/207429 fqs=0
[31008.610687] (t=11770 jiffies g=96530 c=96529 q=189)
[31008.610717] rcu_preempt kthread starved for 11770 jiffies! g96530 c96529 f0x0 s3 ->state=0x1
[31008.610717] Task dump for CPU 0:
[31008.610717] swapper/0 R running 0 0 0 0x00000000
[31008.610717] Backtrace:
[31008.610778] [<c0013eac>] (dump_backtrace) from [<c00140a8>] (show_stack+0x18/0x1c)
[31008.610778] r7:c08f8518 r6:800c0193 r5:00000000 r4:c08fabe8
[31008.610809] [<c0014090>] (show_stack) from [<c005c970>] (sched_show_task+0xbc/0x110)
[31008.610809] [<c005c8b4>] (sched_show_task) from [<c005edd8>] (dump_cpu_task+0x40/0x48)
[31008.610839] r5:c0907800 r4:00000000
[31008.610839] [<c005ed98>] (dump_cpu_task) from [<c0083028>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[31008.610839] r5:c0907800 r4:00000001
[31008.610839] [<c0082f8c>] (rcu_dump_cpu_stacks) from [<c00872ac>] (rcu_check_callbacks+0x524/0x908)
[31008.610900] r9:c08f8518 r8:c0907800 r7:000000bd r6:2e6b9000 r5:c08f5100 r4:eefae100
[31008.610931] [<c0086d88>] (rcu_check_callbacks) from [<c0089f38>] (update_process_times+0x40/0x6c)
[31008.610931] r10:c009ae5c r9:00000001 r8:eefaac4c r7:00001c31 r6:15ca672c r5:00000000
[31008.610931] r4:c08fabe8
[31008.610961] [<c0089ef8>] (update_process_times) from [<c009ae58>] (tick_sched_handle+0x50/0x54)
[31008.610961] r5:c08f7f10 r4:eefaae40
[31008.610992] [<c009ae08>] (tick_sched_handle) from [<c009aeac>] (tick_sched_timer+0x50/0x94)
[31008.610992] [<c009ae5c>] (tick_sched_timer) from [<c008a94c>] (__hrtimer_run_queues+0x110/0x1a0)
[31008.611022] r7:00000000 r6:eefaac40 r5:eefaae40 r4:eefaac00
[31008.611022] [<c008a83c>] (__hrtimer_run_queues) from [<c008afdc>] (hrtimer_interrupt+0xac/0x1f8)
[31008.611053] r10:eefaac78 r9:eefaac98 r8:eefaac14 r7:eefaacb8 r6:ffffffff r5:00000003
[31008.611053] r4:eefaac00
[31008.611083] [<c008af30>] (hrtimer_interrupt) from [<c0017148>] (twd_handler+0x38/0x48)
[31008.611083] r10:c08f84e8 r9:fa241100 r8:00000011 r7:ee828780 r6:c09158cc r5:ee805d80
[31008.611083] r4:00000001
[31008.611083] [<c0017110>] (twd_handler) from [<c007dda8>] (handle_percpu_devid_irq+0x74/0x8c)
[31008.611083] r5:ee805d80 r4:eefb0740
[31008.611145] [<c007dd34>] (handle_percpu_devid_irq) from [<c0079c54>] (generic_handle_irq+0x2c/0x3c)
[31008.611175] r9:fa241100 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:c08f23d0
[31008.611175] [<c0079c28>] (generic_handle_irq) from [<c0079f28>] (__handle_domain_irq+0x64/0xbc)
[31008.611175] [<c0079ec4>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[31008.611206] r9:fa241100 r8:fa240100 r7:c08f7f10 r6:fa24010c r5:c08f879c r4:c09158c8
[31008.611206] [<c0009470>] (gic_handle_irq) from [<c0014c94>] (__irq_svc+0x54/0x90)
[31008.611236] Exception stack(0xc08f7f10 to 0xc08f7f58)
[31008.611236] 7f00: 00000000 eefaa390 fe600000 00000000
[31008.611236] 7f20: c08f6000 c08f849c c08f1364 c069d320 c08f7f80 c0934362 c08f84e8 c08f7f6c
[31008.611267] 7f40: c08f7f4c c08f7f60 c002a988 c0010ba8 600c0113 ffffffff
[31008.611267] r9:c0934362 r8:c08f7f80 r7:c08f7f44 r6:ffffffff r5:600c0113 r4:c0010ba8
[31008.611297] [<c0010b80>] (arch_cpu_idle) from [<c00707d4>] (default_idle_call+0x28/0x34)
[31008.611328] [<c00707ac>] (default_idle_call) from [<c00708f4>] (cpu_startup_entry+0x114/0x18c)
[31008.611328] [<c00707e0>] (cpu_startup_entry) from [<c069485c>] (rest_init+0x90/0x94)
[31008.611358] r7:ffffffff r4:00000002
[31008.611358] [<c06947cc>] (rest_init) from [<c08adcec>] (start_kernel+0x370/0x37c)
[31008.611358] r5:c0937000 r4:00000000
[31008.611419] [<c08ad97c>] (start_kernel) from [<8000807c>] (0x8000807c)
[31008.611419] Task dump for CPU 1:
[31008.611419] swapper/1 R running 0 0 1 0x00000002
[31008.611419] Backtrace:
[31008.611450] Backtrace aborted due to bad frame pointer <ee89ff9c>
[31008.988525] 1: (2 ticks this GP) idle=01d/1/0 softirq=288173/288173 fqs=0
[31008.990051] (t=11809 jiffies g=96530 c=96529 q=103)
[31008.999877] rcu_preempt kthread starved for 11810 jiffies! g96530 c96529 f0x0 s3 ->state=0x1
[31009.010162] Task dump for CPU 1:
[31009.013580] swapper/1 R running 0 0 1 0x00000002
[31009.016937] Backtrace:
[31009.022918] [<c0013eac>] (dump_backtrace) from [<c00140a8>] (show_stack+0x18/0x1c)
[31009.022918] r7:c08f8518 r6:800f0193 r5:00000001 r4:ee877200
[31009.036956] [<c0014090>] (show_stack) from [<c005c970>] (sched_show_task+0xbc/0x110)
[31009.042999] [<c005c8b4>] (sched_show_task) from [<c005edd8>] (dump_cpu_task+0x40/0x48)
[31009.053527] r5:c0907800 r4:00000001
[31009.054962] [<c005ed98>] (dump_cpu_task) from [<c0083028>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[31009.066070] r5:c0907800 r4:00000002
[31009.066070] [<c0082f8c>] (rcu_dump_cpu_stacks) from [<c00872ac>] (rcu_check_callbacks+0x524/0x908)
[31009.077575] r9:c08f8518 r8:c0907800 r7:00000067 r6:2e6c5000 r5:c08f5100 r4:eefba100
[31009.080444] [<c0086d88>] (rcu_check_callbacks) from [<c0089f38>] (update_process_times+0x40/0x6c)
[31009.096954] r10:c009ae5c r9:00000001 r8:eefb6c4c r7:00001c31 r6:1637d5e3 r5:00000000
[31009.098968] r4:ee877200
[31009.107971] [<c0089ef8>] (update_process_times) from [<c009ae58>] (tick_sched_handle+0x50/0x54)
[31009.111083] r5:ee89ff58 r4:eefb6e40
[31009.120971] [<c009ae08>] (tick_sched_handle) from [<c009aeac>] (tick_sched_timer+0x50/0x94)
[31009.128021] [<c009ae5c>] (tick_sched_timer) from [<c008a94c>] (__hrtimer_run_queues+0x110/0x1a0)
[31009.138977] r7:00000000 r6:eefb6c40 r5:eefb6e40 r4:eefb6c00
[31009.138977] [<c008a83c>] (__hrtimer_run_queues) from [<c008afdc>] (hrtimer_interrupt+0xac/0x1f8)
[31009.138977] r10:eefb6c78 r9:eefb6c98 r8:eefb6c14 r7:eefb6cb8 r6:ffffffff r5:00000002
[31009.162719] r4:eefb6c00
[31009.162719] [<c008af30>] (hrtimer_interrupt) from [<c0017148>] (twd_handler+0x38/0x48)
[31009.168975] r10:c08f84e8 r9:fa241100 r8:00000011 r7:ee828780 r6:c09158cc r5:ee805d80
[31009.178955] r4:00000001
[31009.184783] [<c0017110>] (twd_handler) from [<c007dda8>] (handle_percpu_devid_irq+0x74/0x8c)
[31009.193115] r5:ee805d80 r4:eefbc740
[31009.197509] [<c007dd34>] (handle_percpu_devid_irq) from [<c0079c54>] (generic_handle_irq+0x2c/0x3c)
[31009.204711] r9:fa241100 r8:ee808000 r7:00000001 r6:00000000 r5:00000000 r4:c08f23d0
[31009.215301] [<c0079c28>] (generic_handle_irq) from [<c0079f28>] (__handle_domain_irq+0x64/0xbc)
[31009.222198] [<c0079ec4>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[31009.232177] r9:fa241100 r8:fa240100 r7:ee89ff58 r6:fa24010c r5:c08f879c r4:c09158c8
[31009.236633] [<c0009470>] (gic_handle_irq) from [<c0014c94>] (__irq_svc+0x54/0x90)
[31009.249481] Exception stack(0xee89ff58 to 0xee89ffa0)
[31009.254821] ff40: 00000000 eefb6390
[31009.254821] ff60: fe600000 00000000 ee89e000 c08f849c c08f1364 c069d320 ee89ffc8 c0934362
[31009.268981] ff80: c08f84e8 ee89ffb4 ee89ff94 ee89ffa8 c002a988 c0010ba8 600f0113 ffffffff
[31009.278991] r9:c0934362 r8:ee89ffc8 r7:ee89ff8c r6:ffffffff r5:600f0113 r4:c0010ba8
[31009.280792] [<c0010b80>] (arch_cpu_idle) from [<c00707d4>] (default_idle_call+0x28/0x34)
[31009.297576] [<c00707ac>] (default_idle_call) from [<c00708f4>] (cpu_startup_entry+0x114/0x18c)
[31009.306701] [<c00707e0>] (cpu_startup_entry) from [<c001618c>] (secondary_start_kernel+0x160/0x16c)
[31009.311218] r7:c0937288 r4:c090a880
[31009.319000] [<c001602c>] (secondary_start_kernel) from [<8000958c>] (0x8000958c)
[31009.319000] r5:00000051 r4:ae88406a
[79853.606109] wlcore: Beacon loss detected. roles:0xff
[79858.406158] wlcore: Connection loss work (role_id: 0).
[79858.411621] wlan0: Connection to AP 90:f6:52:81:ee:70 lost
[79862.045989] wlan0: authenticate with 90:f6:52:81:ee:70
[79862.092102] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[79862.100311] wlan0: authenticated
[79862.104766] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[79862.118896] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[79862.137695] wlan0: associated
[79862.191680] wlcore: Association completed.
$
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: rcu_preempt self-detected stall on CPU from 4.4-rc4, since 3.17
2016-01-03 22:00 ` Ross Green
@ 2016-01-04 14:21 ` Ross Green
2016-01-06 23:10 ` Ross Green
0 siblings, 1 reply; 12+ messages in thread
From: Ross Green @ 2016-01-04 14:21 UTC (permalink / raw)
To: Paul McKenney
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
Mathieu Desnoyers, josh, tglx, peterz, rostedt, dhowells,
Eric Dumazet, dvhart, Frédéric Weisbecker, oleg,
pranith kumar
[-- Attachment #1: Type: text/plain, Size: 11924 bytes --]
Well with the release of 4.4-rc8 I have built and
started testing the kernel.
With some luck I managed to get a rcu_preempt stall within a few hours
of testing.
Upon booting, I ran a small series of bench marks to make sure
everything is running as expected. limited regression testing and then
just left the system to idle away, with periodic monitoring from the
network.
please find attached two stack traces from linux-4.4-rc8 and also linux-4.4-rc7.
The interesting thing with the rc7 trace is that there are multiple
stalls that have occurred over a 6 day period.
I realise Paul you have a number of changes pending for the the RCU
code. It would be good to try and establish what is happening with
these stalls before the impact of those changes given that timings and
dynamics might change whats happening in the current environment.
As reported earlier, i have never been able to induce these stalls
with heavy loading of the system. The only method I can be sure of is
to leave the system quiet and let time go by till a stall occurs.
Regards,
Ross
On Mon, Jan 4, 2016 at 9:00 AM, Ross Green <rgkernel@gmail.com> wrote:
> Thanks Paul for your analysis and investigation,
>
> I understand your patches are designed not to "fix" the problem, but
> more to move the problem ahead.
>
> I will include a few more stack traces from various kernels. I can go
> back to around 3.17, with similar trace results.
>
> My observation is that the problem can occur at various times and with
> out any "bad" effect other than more stalls could happen afterwards.
>
> At first I wondered if they might actually be a false positive as the
> kernel seemed to carry on and run quite happily. It is rare that I
> find a kernel just locks up after observing such a stall, or a
> complete kernel splat!
>
> Unfortunately with my testing I have never been able to induce the
> problem under any heavy load that would immediately trigger the
> problem. Indeed most heavy cpu utilisation seemed to just sail on
> quite nicely.
>
> The time for a fault, seems so far, to be non-deterministic with
> quiescent systems taking anywhere from a few hours through to some six
> days before showing the problem.
>
> More recent kernels seem to have richer stall and back trace
> information so I was hoping that might shed some light on how they
> might be occurring..
>
> I usually only run a kernel till I get a new -rc release to test with,
> so a run of 1 week is a typical cycle.
>
> I just wish I could find a sure fire method to trigger the problem!!
>
> I have included a few more traces of various kernels all showing the problem.
>
> Regards,
>
> Ross
>
> On Mon, Jan 4, 2016 at 5:15 AM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
>> On Sun, Jan 03, 2016 at 07:27:17PM +1100, Ross Green wrote:
>>> I would not describe the load on this test machine as high or real time.
>>>
>>> Apart from a number of standard daemons not much more is running at all!
>>>
>>> I normally build a release kernel as soon as possible and set it running.
>>> Typically I run a series of benchmarks to confirm most things appear
>>> to be working and then just leave it running. During a normal day i
>>> will check on the machine 4/5 times just to see how its going!
>>> Typically I will logon remotely via wifi network connection.
>>>
>>> just for your information i will include a few other stack traces from
>>> previous kernels that may show some trend!
>>>
>>>
>>> Please see the attachments.
>>
>> Thank you for the additional details. This does look similar to some
>> problems I am seeing, though only in heavy rcutorture workloads with
>> CPU hotplugging.
>>
>> I have some crude workarounds in progress, see for example
>> 2da26818e515 (rcu: Awaken grace-period kthread when stalled) at
>> git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git.
>> This workaround kicks the RCU grace-period kthread on every stall warning.
>> In my testing, this workaround results in slow but real forward progress.
>>
>> I have a better workaround in progress, however, please note:
>>
>> 1. I have no intention of sending these workarounds upstream.
>>
>> 2. The workarounds will splat when they take effect. In other words,
>> the idea is not to paper over the problem, but instead to allow
>> me to separate testing concerns.
>>
>> 3. A fix is needed for the underlying bug, wherever it might be.
>>
>> Thanx, Paul
>>
>>> Regards,
>>>
>>> Ross
>>>
>>>
>>>
>>> On Sun, Jan 3, 2016 at 5:17 PM, Paul E. McKenney
>>> <paulmck@linux.vnet.ibm.com> wrote:
>>> > On Sun, Jan 03, 2016 at 04:29:11PM +1100, Ross Green wrote:
>>> >> Still seeing these rcu_preempt stalls on kernels through to 4.4-rc7
>>> >>
>>> >> Still have not found a sure fire method to evoke this stall, but have
>>> >> found that it will normally occur within a week of running a kernel -
>>> >> usually when it is quiet with light load.
>>> >>
>>> >> Have seen similar self detected stalls all the way back to 3.17.
>>> >> Most recent kernels have included 4.4-rc5 4.4-rc6 and 4.4-rc7
>>> >>
>>> >> Regards,
>>> >>
>>> >> Ross
>>> >>
>>> >> On Fri, Dec 11, 2015 at 10:17 PM, Ross Green <rgkernel@gmail.com> wrote:
>>> >> > I have been getting these stalls in kernels going back to 3.17.
>>> >> >
>>> >> > This stall occurs usually under light load but often requires several
>>> >> > days to show itself. I have not found any simple way to trigger the
>>> >> > stall. Indeed heavy workloads seems not to show the fault.
>>> >> >
>>> >> > Anyone have any thoughts here?
>>> >> >
>>> >> > The recent patch by peterz with kernel/sched/wait.c I thought might
>>> >> > help the situation, but alas after a few days of running 4.4-rc4 the
>>> >> > following turned up.
>>> >> >
>>> >> > [179922.003570] INFO: rcu_preempt self-detected stall on CPU
>>> >> > [179922.008178] INFO: rcu_preempt detected stalls on CPUs/tasks:
>>> >> > [179922.008178] 0-...: (1 ticks this GP) idle=a91/1/0
>>> >
>>> > CPU 0 is non-idle from an RCU perspective.
>>> >
>>> >> > softirq=1296733/1296733 fqs=0
>>> >> > [179922.008178]
>>> >> > [179922.008209] (detected by 1, t=8775 jiffies, g=576439, c=576438, q=102)
>>> >> > [179922.008209] Task dump for CPU 0:
>>> >> > [179922.008209] swapper/0 R [179922.008209] running [179922.008209] 0 0 0 0x00000000
>>> >> > [179922.008209] Backtrace:
>>> >> >
>>> >> > [179922.008239] Backtrace aborted due to bad frame pointer <c0907f54>
>>> >
>>> > Can't have everything, I guess...
>>> >
>>> >> > [179922.008239] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x0 s3 ->state=0x1
>>> >
>>> > Something is keeping the rcu_preempt grace-period kthread from
>>> > running. This far into the grace period, it should have a
>>> > timer event waking it every few jiffies. It is currently
>>> > in TASK_INTERRUPTIBLE state.
>>> >
>>> >> > [179922.060302] 0-...: (1 ticks this GP) idle=a91/1/0 softirq=1296733/1296733 fqs=0
>>> >> > [179922.068023] (t=8775 jiffies g=576439 c=576438 q=102)
>>> >> > [179922.073913] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x2 s3 ->state=0x100
>>> >
>>> > Same story, same grace period, pretty much same time. Now there is an FQS
>>> > request (f0x2) and the state is now TASK_WAKING (->state=0x100 == 256).
>>> >
>>> >> > [179922.083587] Task dump for CPU 0:
>>> >> > [179922.087097] swapper/0 R running 0 0 0 0x00000000
>>> >> > [179922.093292] Backtrace:
>>> >> > [179922.096313] [<c0013ea8>] (dump_backtrace) from [<c00140a4>] (show_stack+0x18/0x1c)
>>> >> > [179922.104675] r7:c0908514 r6:80080193 r5:00000000 r4:c090aca8
>>> >> > [179922.110809] [<c001408c>] (show_stack) from [<c005a858>] (sched_show_task+0xbc/0x110)
>>> >> > [179922.119049] [<c005a79c>] (sched_show_task) from [<c005ccd4>] (dump_cpu_task+0x40/0x44)
>>> >> > [179922.127624] r5:c0917680 r4:00000000
>>> >> > [179922.131042] [<c005cc94>] (dump_cpu_task) from [<c0082268>] (rcu_dump_cpu_stacks+0x9c/0xdc)
>>> >> > [179922.140350] r5:c0917680 r4:00000001
>>> >> > [179922.143157] [<c00821cc>] (rcu_dump_cpu_stacks) from [<c008637c>] (rcu_check_callbacks+0x504/0x8e4)
>>> >> > [179922.153808] r9:c0908514 r8:c0917680 r7:00000066 r6:2eeab000
>>> >> > r5:c0904300 r4:ef7af300
>>> >> > [179922.161499] [<c0085e78>] (rcu_check_callbacks) from [<c00895d0>] (update_process_times+0x40/0x6c)
>>> >> > [179922.170898] r10:c009a584 r9:00000001 r8:ef7abc4c r7:0000a3a3
>>> >> > r6:4ec3391c r5:00000000
>>> >> > [179922.179901] r4:c090aca8
>>> >> > [179922.182708] [<c0089590>] (update_process_times) from [<c009a580>]
>>> >> > (tick_sched_handle+0x50/0x54)
>>> >> > [179922.192108] r5:c0907f10 r4:ef7abe40
>>> >> > [179922.195983] [<c009a530>] (tick_sched_handle) from [<c009a5d4>]
>>> >> > (tick_sched_timer+0x50/0x94)
>>> >> > [179922.204895] [<c009a584>] (tick_sched_timer) from [<c0089fe4>]
>>> >> > (__hrtimer_run_queues+0x110/0x1a0)
>>> >> > [179922.214324] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
>>> >> > [179922.220428] [<c0089ed4>] (__hrtimer_run_queues) from [<c008a674>]
>>> >> > (hrtimer_interrupt+0xac/0x1f8)
>>> >> > [179922.227111] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8
>>> >> > r6:ffffffff r5:00000003
>>> >> > [179922.238220] r4:ef7abc00
>>> >> > [179922.238220] [<c008a5c8>] (hrtimer_interrupt) from [<c00170ec>]
>>> >> > (twd_handler+0x38/0x48)
>>> >> > [179922.238220] r10:c09084e8 r9:fa241100 r8:00000011 r7:ef028780
>>> >> > r6:c092574c r5:ef005cc0
>>> >
>>> > All interrupt stack up to this point.
>>> >
>>> > It is quite possible that the stuff below here is at fault as well.
>>> > That said, the grace-period should actually get to execute at some
>>> > point. Do you have a heavy real-time load that might be starving
>>> > things?
>>> >
>>> > Thanx, Paul
>>> >
>>> >> > [179922.257110] r4:00000001
>>> >> > [179922.257110] [<c00170b4>] (twd_handler) from [<c007c8f8>] (handle_percpu_devid_irq+0x74/0x8c)
>>> >> > [179922.269683] r5:ef005cc0 r4:ef7b1740
>>> >> > [179922.269683] [<c007c884>] (handle_percpu_devid_irq) from [<c0078454>] (generic_handle_irq+0x2c/0x3c)
>>> >> > [179922.283233] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000
>>> >> > r5:00000000 r4:c09013e8
>>> >> > [179922.290985] [<c0078428>] (generic_handle_irq) from [<c007872c>] (__handle_domain_irq+0x64/0xbc)
>>> >> > [179922.300842] [<c00786c8>] (__handle_domain_irq) from [<c00094c0>]
>>> >> > (gic_handle_irq+0x50/0x90)
>>> >> > [179922.303222] r9:fa241100 r8:fa240100 r7:c0907f10 r6:fa24010c
>>> >> > r5:c09087a8 r4:c0925748
>>> >> > [179922.315216] [<c0009470>] (gic_handle_irq) from [<c0014bd4>]
>>> >> > (__irq_svc+0x54/0x90)
>>> >> > [179922.319000] Exception stack(0xc0907f10 to 0xc0907f58)
>>> >> > [179922.331542] 7f00: 00000000
>>> >> > ef7ab390 fe600000 00000000
>>> >> > [179922.331542] 7f20: c0906000 c090849c c0900364 c06a8124 c0907f80
>>> >> > c0944563 c09084e8 c0907f6c
>>> >> > [179922.349029] 7f40: c0907f4c c0907f60 c00287ac c0010ba8 60080113 ffffffff
>>> >> > [179922.349029] r9:c0944563 r8:c0907f80 r7:c0907f44 r6:ffffffff
>>> >> > r5:60080113 r4:c0010ba8
>>> >> > [179922.357116] [<c0010b80>] (arch_cpu_idle) from [<c006f034>]
>>> >> > (default_idle_call+0x28/0x34)
>>> >> > [179922.368926] [<c006f00c>] (default_idle_call) from [<c006f154>]
>>> >> > (cpu_startup_entry+0x114/0x18c)
>>> >> > [179922.368926] [<c006f040>] (cpu_startup_entry) from [<c069fc6c>]
>>> >> > (rest_init+0x90/0x94)
>>> >> > [179922.385284] r7:ffffffff r4:00000002
>>> >> > [179922.393463] [<c069fbdc>] (rest_init) from [<c08bbcec>]
>>> >> > (start_kernel+0x370/0x37c)
>>> >> > [179922.400421] r5:c0947000 r4:00000000
>>> >> > [179922.400421] [<c08bb97c>] (start_kernel) from [<8000807c>] (0x8000807c)
>>> >> > $
>>> >
>>> >
>>
>>
>>
[-- Attachment #2: dmesg-4.4-rc8 --]
[-- Type: application/octet-stream, Size: 28104 bytes --]
$ uptime
00:40:23 up 4:19, load average: 0.38, 0.20, 0.16
$
$
$
$ dmesg
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 4.4.0-rc8-panda (rfg@red.dwarf) (gcc version 5.3.0 (Buildroot 2016.02-git-00183-g1682aee) ) #1 SMP PREEMPT Mon Jan 4 20:00:17 AEDT 2016
[ 0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] Machine model: TI OMAP4 PandaBoard-ES
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] OMAP4: Map 0xbfe00000 to fe600000 for dram barrier
[ 0.000000] On node 0 totalpages: 261632
[ 0.000000] free_area_init_node: node 0, pgdat c0945940, node_mem_map ef7f9000
[ 0.000000] Normal zone: 1536 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 196608 pages, LIFO batch:31
[ 0.000000] HighMem zone: 65024 pages, LIFO batch:15
[ 0.000000] OMAP4460 ES1.1
[ 0.000000] PERCPU: Embedded 12 pages/cpu @ef7ab000 s17920 r8192 d23040 u49152
[ 0.000000] pcpu-alloc: s17920 r8192 d23040 u49152 alloc=12*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260096
[ 0.000000] Kernel command line: root=/dev/mmcblk0p2 rootwait console=ttyO2,115200
[ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Memory: 1027368K/1046528K available (6783K kernel code, 250K rwdata, 2132K rodata, 296K init, 294K bss, 19160K reserved, 0K cma-reserved, 260096K highmem)
[ 0.000000] Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
vmalloc : 0xf0800000 - 0xff800000 ( 240 MB)
lowmem : 0xc0000000 - 0xf0000000 ( 768 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc08bd0d4 (8917 kB)
.init : 0xc08be000 - 0xc0908000 ( 296 kB)
.data : 0xc0908000 - 0xc0946920 ( 251 kB)
.bss : 0xc0949000 - 0xc0992bb8 ( 295 kB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 32.
[ 0.000000] NR_IRQS:16 nr_irqs:16 16
[ 0.000000] L2C: platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C: DT/platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C-310 enabling early BRESP for Cortex-A9
[ 0.000000] OMAP L2C310: ROM does not support power control setting
[ 0.000000] L2C-310 dynamic clock gating disabled, standby mode disabled
[ 0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
[ 0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e470000
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node bandgap_fclk
[ 0.000000] OMAP clockevent source: timer1 at 32768 Hz
[ 0.000000] clocksource: 32k_counter: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 58327039986419 ns
[ 0.000030] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65535999984741ns
[ 0.000030] OMAP clocksource: 32k_counter at 32768 Hz
[ 0.000885] Console: colour dummy device 80x30
[ 0.000885] Calibrating delay loop... 1397.55 BogoMIPS (lpj=6987776)
[ 0.089233] pid_max: default: 32768 minimum: 301
[ 0.089294] Security Framework initialized
[ 0.089355] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.089355] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.089965] Initializing cgroup subsys io
[ 0.089965] Initializing cgroup subsys devices
[ 0.089996] Initializing cgroup subsys freezer
[ 0.090026] CPU: Testing write buffer coherency: ok
[ 0.090240] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.090301] Setting up static identity map for 0x80008280 - 0x800082d8
[ 0.209228] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.209289] Brought up 2 CPUs
[ 0.209320] SMP: Total of 2 processors activated (2795.11 BogoMIPS).
[ 0.209320] CPU: All CPU(s) started in SVC mode.
[ 0.210113] devtmpfs: initialized
[ 0.227355] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[ 0.229064] omap_hwmod: l3_main_3 using broken dt data from ocp
[ 0.230590] omap_hwmod: l3_main_2 using broken dt data from ocp
[ 0.327270] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.327514] xor: measuring software checksum speed
[ 0.419219] arm4regs : 1006.000 MB/sec
[ 0.519195] 8regs : 804.800 MB/sec
[ 0.619201] 32regs : 760.800 MB/sec
[ 0.619232] xor: using function: arm4regs (1006.000 MB/sec)
[ 0.619476] pinctrl core: initialized pinctrl subsystem
[ 0.621337] NET: Registered protocol family 16
[ 0.622131] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.631500] OMAP GPIO hardware version 0.1
[ 0.636749] omap-gpmc 50000000.gpmc: GPMC revision 6.0
[ 0.636749] gpmc_mem_init: disabling cs 0 mapped at 0x0-0x1000000
[ 0.637268] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.637634] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.638031] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.649719] platform 4b501000.aes: Cannot lookup hwmod 'aes'
[ 0.649993] platform 480a5000.des: Cannot lookup hwmod 'des'
[ 0.655944] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[ 0.655975] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.656768] omap4_sram_init:Unable to get sram pool needed to handle errata I688
[ 0.656982] OMAP DMA hardware revision 0.0
[ 0.889434] raid6: int32x1 gen() 103 MB/s
[ 1.059356] raid6: int32x1 xor() 83 MB/s
[ 1.229309] raid6: int32x2 gen() 134 MB/s
[ 1.399230] raid6: int32x2 xor() 98 MB/s
[ 1.569427] raid6: int32x4 gen() 145 MB/s
[ 1.739471] raid6: int32x4 xor() 100 MB/s
[ 1.909301] raid6: int32x8 gen() 160 MB/s
[ 2.079467] raid6: int32x8 xor() 98 MB/s
[ 2.079467] raid6: using algorithm int32x8 gen() 160 MB/s
[ 2.079498] raid6: .... xor() 98 MB/s, rmw enabled
[ 2.079498] raid6: using intx1 recovery algorithm
[ 2.086761] omap-dma-engine 4a056000.dma-controller: OMAP DMA engine driver
[ 2.159332] reg-fixed-voltage wl12xx_vmmc: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_wl12xx_gpio, deferring probe
[ 2.163482] omap-iommu 4a066000.mmu: 4a066000.mmu registered
[ 2.163604] omap-iommu 55082000.mmu: 55082000.mmu registered
[ 2.164550] SCSI subsystem initialized
[ 2.164825] usbcore: registered new interface driver usbfs
[ 2.164947] usbcore: registered new interface driver hub
[ 2.165100] usbcore: registered new device driver usb
[ 2.166046] omap_i2c 48070000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c1_pins, deferring probe
[ 2.166107] omap_i2c 48072000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c2_pins, deferring probe
[ 2.166137] omap_i2c 48060000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c3_pins, deferring probe
[ 2.166198] omap_i2c 48350000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c4_pins, deferring probe
[ 2.166442] Linux video capture interface: v2.00
[ 2.167236] Advanced Linux Sound Architecture Driver Initialized.
[ 2.168609] clocksource: Switched to clocksource 32k_counter
[ 2.185974] NET: Registered protocol family 2
[ 2.186553] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[ 2.186645] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[ 2.186798] TCP: Hash tables configured (established 8192 bind 8192)
[ 2.186859] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 2.186920] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 2.187103] NET: Registered protocol family 1
[ 2.187469] RPC: Registered named UNIX socket transport module.
[ 2.187469] RPC: Registered udp transport module.
[ 2.187469] RPC: Registered tcp transport module.
[ 2.187499] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 2.189788] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[ 2.192718] futex hash table entries: 512 (order: 3, 32768 bytes)
[ 2.205017] NFS: Registering the id_resolver key type
[ 2.205078] Key type id_resolver registered
[ 2.205078] Key type id_legacy registered
[ 2.205108] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 2.205139] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 2.206604] ntfs: driver 2.1.32 [Flags: R/O].
[ 2.207031] fuse init (API version 7.23)
[ 2.211425] jitterentropy: Initialization failed with host not compliant with requirements: 2
[ 2.211517] bounce: pool size: 64 pages
[ 2.211547] io scheduler noop registered
[ 2.211578] io scheduler deadline registered
[ 2.211608] io scheduler cfq registered (default)
[ 2.212493] pinctrl-single 4a100040.pinmux: 203 pins at pa fc100040 size 406
[ 2.212707] pinctrl-single 4a31e040.pinmux: 28 pins at pa fc31e040 size 56
[ 2.216033] 58000000.dss supply vdda_video not found, using dummy regulator
[ 2.216156] OMAP DSS rev 4.0
[ 2.216308] omapdss_dss 58000000.dss: bound 58001000.dispc (ops dispc_component_ops)
[ 2.216522] omapdss_dss 58000000.dss: bound 58005000.encoder (ops dsi_component_ops)
[ 2.217071] omapdss_dss 58000000.dss: bound 58006000.encoder (ops hdmi4_component_ops)
[ 2.219482] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 2.221405] omap_uart 4806a000.serial: no wakeirq for uart0
[ 2.221527] 4806a000.serial: ttyO0 at MMIO 0x4806a000 (irq = 226, base_baud = 3000000) is a OMAP UART0
[ 2.222106] 4806c000.serial: ttyO1 at MMIO 0x4806c000 (irq = 227, base_baud = 3000000) is a OMAP UART1
[ 2.222656] 48020000.serial: ttyO2 at MMIO 0x48020000 (irq = 228, base_baud = 3000000) is a OMAP UART2
[ 3.180816] console [ttyO2] enabled
[ 3.184997] 4806e000.serial: ttyO3 at MMIO 0x4806e000 (irq = 229, base_baud = 3000000) is a OMAP UART3
[ 3.195953] [drm] Initialized drm 1.1.0 20060810
[ 3.224395] loop: module loaded
[ 3.230957] usbcore: registered new interface driver smsc95xx
[ 3.237030] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 3.243927] ehci-omap: OMAP-EHCI Host Controller driver
[ 3.268615] ehci-omap 4a064c00.ehci: EHCI Host Controller
[ 3.274291] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1
[ 3.282379] ehci-omap 4a064c00.ehci: irq 336, io mem 0x4a064c00
[ 3.298645] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00
[ 3.305023] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 3.312133] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 3.319702] usb usb1: Product: EHCI Host Controller
[ 3.324798] usb usb1: Manufacturer: Linux 4.4.0-rc8-panda ehci_hcd
[ 3.331298] usb usb1: SerialNumber: 4a064c00.ehci
[ 3.336883] hub 1-0:1.0: USB hub found
[ 3.340850] hub 1-0:1.0: 3 ports detected
[ 3.345764] usbcore: registered new interface driver cdc_wdm
[ 3.351837] usbcore: registered new interface driver usb-storage
[ 3.358551] i2c /dev entries driver
[ 3.362365] usbcore: registered new interface driver uvcvideo
[ 3.368377] USB Video Class driver (1.1.1)
[ 3.373901] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[ 3.381439] device-mapper: uevent: version 1.0.3
[ 3.386627] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com
[ 3.459655] hidraw: raw HID events driver (C) Jiri Kosina
[ 3.465515] usbcore: registered new interface driver usbhid
[ 3.471343] usbhid: USB HID core driver
[ 3.475891] emif 4c000000.emif: emif_probe: device configured with addr = f08a8000 and IRQ247
[ 3.485198] emif 4d000000.emif: emif_probe: device configured with addr = f08aa000 and IRQ248
[ 3.496154] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 3.504119] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 3.511596] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 3.517272] Initializing XFRM netlink socket
[ 3.521850] NET: Registered protocol family 17
[ 3.526794] Key type dns_resolver registered
[ 3.531463] twl: not initialized
[ 3.534881] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.543487] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.551971] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.560455] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.568969] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.577423] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.585937] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1410000 Vs max 1316660
[ 3.594421] omap2_set_init_voltage: unable to find boot up OPP for vdd_mpu
[ 3.602020] omap2_set_init_voltage: unable to set vdd_mpu
[ 3.607727] omap2_set_init_voltage: unable to find boot up OPP for vdd_core
[ 3.615112] omap2_set_init_voltage: unable to set vdd_core
[ 3.620941] omap2_set_init_voltage: unable to find boot up OPP for vdd_iva
[ 3.628234] omap2_set_init_voltage: unable to set vdd_iva
[ 3.634826] Power Management for TI OMAP4+ devices.
[ 3.640014] OMAP4 PM: u-boot >= v2012.07 is required for full PM support
[ 3.647216] ThumbEE CPU extension supported.
[ 3.651855] Registering SWP/SWPB emulation handler
[ 3.651855] SmartReflex Class3 initialized
[ 3.661346] usb 1-1: new high-speed USB device number 2 using ehci-omap
[ 3.681457] Btrfs loaded
[ 3.687103] dmm 4e000000.dmm: initialized all PAT entries
[ 3.704772] Skipping twl internal clock init and using bootloader value (unknown osc rate)
[ 3.719787] twl 0-0048: PIH (irq 340) nested IRQs
[ 3.725524] twl_rtc 48070000.i2c:twl@48:rtc: Power up reset detected.
[ 3.733123] twl_rtc 48070000.i2c:twl@48:rtc: Enabling TWL-RTC
[ 3.741851] twl_rtc 48070000.i2c:twl@48:rtc: rtc core: registered 48070000.i2c:twl@48 as rtc0
[ 3.763427] twl6040 0-004b: clk32k is not handled
[ 3.771362] omap_i2c 48070000.i2c: bus 0 rev0.11 at 400 kHz
[ 3.778076] omap_i2c 48072000.i2c: bus 1 rev0.11 at 400 kHz
[ 3.785339] omap_i2c 48060000.i2c: bus 2 rev0.11 at 100 kHz
[ 3.792205] omap_i2c 48350000.i2c: bus 3 rev0.11 at 400 kHz
[ 3.844970] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
[ 3.852203] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 3.860351] hub 1-1:1.0: USB hub found
[ 3.865112] hub 1-1:1.0: 5 ports detected
[ 3.996490] omap-abe-twl6040 sound: twl6040-legacy <-> 40132000.mcpdm mapping ok
[ 4.008392] twl_rtc 48070000.i2c:twl@48:rtc: setting system clock to 2000-01-01 00:00:00 UTC (946684800)
[ 4.018402] sr_init: No PMIC hook to init smartreflex
[ 4.023803] smartreflex smartreflex.0: omap_sr_probe: SmartReflex driver initialized
[ 4.032348] smartreflex smartreflex.1: omap_sr_probe: SmartReflex driver initialized
[ 4.040557] smartreflex smartreflex.2: omap_sr_probe: SmartReflex driver initialized
[ 4.059844] VDAC: disabling
[ 4.062866] mmc0: host does not support reading read-only switch, assuming write-enable
[ 4.063293] VANA: disabling
[ 4.063751] VUSB: disabling
[ 4.064025] ALSA device list:
[ 4.064025] #0: PandaBoardES
[ 4.084106] Waiting for root device /dev/mmcblk0p2...
[ 4.089813] mmc0: new high speed SDHC card at address e624
[ 4.106292] mmcblk0: mmc0:e624 SL32G 28.7 GiB
[ 4.141418] mmcblk0: p1 p2 p3 p4
[ 4.151214] omap_hsmmc 480d5000.mmc: card claims to support voltages below defined range
[ 4.168914] usb 1-1.1: new high-speed USB device number 3 using ehci-omap
[ 4.177246] mmc1: queuing unknown CIS tuple 0x91 (3 bytes)
[ 4.184326] mmc1: new SDIO card at address 0001
[ 4.214752] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
[ 4.223846] EXT4-fs (mmcblk0p2): mounting ext2 file system using the ext4 subsystem
[ 4.252502] EXT4-fs (mmcblk0p2): mounted filesystem without journal. Opts: (null)
[ 4.264282] VFS: Mounted root (ext2 filesystem) readonly on device 179:2.
[ 4.276947] devtmpfs: mounted
[ 4.280456] Freeing unused kernel memory: 296K (c08be000 - c0908000)
[ 4.280731] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[ 4.280731] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 4.283599] smsc95xx v1.0.4
[ 4.361236] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-4a064c00.ehci-1.1, smsc95xx USB 2.0 Ethernet, c2:ff:4d:1d:f7:fc
[ 4.732299] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 4.870025] random: dd urandom read with 33 bits of entropy available
[ 10.194335] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[ 19.960144] Bluetooth: Core ver 2.21
[ 19.964447] NET: Registered protocol family 31
[ 19.969482] Bluetooth: HCI device and connection manager initialized
[ 19.977355] Bluetooth: HCI socket layer initialized
[ 19.982604] Bluetooth: L2CAP socket layer initialized
[ 19.988647] Bluetooth: SCO socket layer initialized
[ 27.644348] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[ 27.655883] NFSD: starting 90-second grace period (net c093fe80)
[ 39.803558] random: nonblocking pool is initialized
[ 53.613281] wlcore: loaded
[ 66.856719] wlcore: firmware booted (Rev 6.3.10.0.133)
[ 70.489837] wlan0: authenticate with 90:f6:52:81:ee:70
[ 70.505676] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[ 70.522064] wlan0: send auth to 90:f6:52:81:ee:70 (try 2/3)
[ 70.538452] wlan0: send auth to 90:f6:52:81:ee:70 (try 3/3)
[ 70.554504] wlan0: authentication with 90:f6:52:81:ee:70 timed out
[ 74.102203] wlan0: authenticate with 90:f6:52:81:ee:70
[ 74.117462] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[ 74.125823] wlan0: authenticated
[ 74.144989] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[ 74.155883] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=1)
[ 74.174682] wlan0: associated
[ 74.234130] wlcore: Association completed.
[ 122.623901] Adding 2097148k swap on /dev/mmcblk0p3. Priority:-1 extents:1 across:2097148k SS
[ 147.687713] NFSD: Unable to end grace period: -110
[ 8828.529388] INFO: rcu_preempt self-detected stall on CPU
[ 8828.529479] INFO: rcu_preempt self-detected stall on CPU
[ 8828.529510] 1-...: (1 GPs behind) idle=4b1/1/0 softirq=80317/80317 fqs=0
[ 8828.529510]
[ 8828.529510] (t=25041 jiffies g=27810 c=27809 q=3)
[ 8828.529510] rcu_preempt kthread starved for 25041 jiffies! g27810 c27809 f0x0 s3 ->state=0x1
[ 8828.529510] Task dump for CPU 0:
[ 8828.529510] swapper/0 R
[ 8828.529541] running
[ 8828.529541] 0 0 0 0x00000000
[ 8828.529541] Backtrace:
[ 8828.529541] Backtrace aborted due to bad frame pointer <c0909f54>
[ 8828.529541] Task dump for CPU 1:
[ 8828.529571] swapper/1 R
[ 8828.529571] running
[ 8828.529571] 0 0 1 0x00000000
[ 8828.529571] Backtrace:
[ 8828.529571] [<c0013e9c>] (dump_backtrace) from [<c0014098>] (show_stack+0x18/0x1c)
[ 8828.529571] r7:c090a514
[ 8828.529632] r6:80000193
[ 8828.529632] r5:00000001
[ 8828.529632] r4:ef076f80
[ 8828.529663] [<c0014080>] (show_stack) from [<c005a7d8>] (sched_show_task+0xbc/0x110)
[ 8828.529663] [<c005a71c>] (sched_show_task) from [<c005cc60>] (dump_cpu_task+0x40/0x48)
[ 8828.529693] r5:c0919680
[ 8828.529693] r4:00000001
[ 8828.529693] [<c005cc20>] (dump_cpu_task) from [<c00821f8>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[ 8828.529724] r5:c0919680
[ 8828.529724] r4:00000002
[ 8828.529724] [<c008215c>] (rcu_dump_cpu_stacks) from [<c008630c>] (rcu_check_callbacks+0x504/0x8e4)
[ 8828.529754] r9:c090a514
[ 8828.529754] r8:c0919680
[ 8828.529754] r7:00000003
[ 8828.529754] r6:2eeb4000
[ 8828.529754] r5:c0907300
[ 8828.529754] r4:ef7bb300
[ 8828.529754] [<c0085e08>] (rcu_check_callbacks) from [<c0089560>] (update_process_times+0x40/0x6c)
[ 8828.529754] r10:c009a514
[ 8828.529754] r9:00000001
[ 8828.529815] r8:ef7b7c4c
[ 8828.529815] r7:00000807
[ 8828.529815] r6:8d3a6293
[ 8828.529815] r5:00000000
[ 8828.529815] r4:ef076f80
[ 8828.529846] [<c0089520>] (update_process_times) from [<c009a510>] (tick_sched_handle+0x50/0x54)
[ 8828.529846] r5:ef09ff58
[ 8828.529846] r4:ef7b7e40
[ 8828.529876] [<c009a4c0>] (tick_sched_handle) from [<c009a564>] (tick_sched_timer+0x50/0x94)
[ 8828.529876] [<c009a514>] (tick_sched_timer) from [<c0089f74>] (__hrtimer_run_queues+0x110/0x1a0)
[ 8828.529876] r7:00000000
[ 8828.529907] r6:ef7b7c40
[ 8828.529907] r5:ef7b7e40
[ 8828.529907] r4:ef7b7c00
[ 8828.529907] [<c0089e64>] (__hrtimer_run_queues) from [<c008a604>] (hrtimer_interrupt+0xac/0x1f8)
[ 8828.529907] r10:ef7b7c78
[ 8828.529937] r9:ef7b7c98
[ 8828.529937] r8:ef7b7c14
[ 8828.529937] r7:ef7b7cb8
[ 8828.529937] r6:ffffffff
[ 8828.529937] r5:00000003
[ 8828.529937] r4:ef7b7c00
[ 8828.529968] [<c008a558>] (hrtimer_interrupt) from [<c00170ec>] (twd_handler+0x38/0x48)
[ 8828.529968] r10:c090a4e8
[ 8828.529968] r9:fa241100
[ 8828.529998] r8:00000011
[ 8828.529998] r7:ef028780
[ 8828.529998] r6:c092774c
[ 8828.529998] r5:ef005cc0
[ 8828.529998] r4:00000001
[ 8828.530029] [<c00170b4>] (twd_handler) from [<c007c888>] (handle_percpu_devid_irq+0x74/0x8c)
[ 8828.530029] r5:ef005cc0
[ 8828.530029] r4:ef7bd740
[ 8828.530059] [<c007c814>] (handle_percpu_devid_irq) from [<c00783e4>] (generic_handle_irq+0x2c/0x3c)
[ 8828.530059] r9:fa241100
[ 8828.530059] r8:ef008000
[ 8828.530059] r7:00000001
[ 8828.530059] r6:00000000
[ 8828.530090] r5:00000000
[ 8828.530090] r4:c09043e8
[ 8828.530090] [<c00783b8>] (generic_handle_irq) from [<c00786bc>] (__handle_domain_irq+0x64/0xbc)
[ 8828.530120] [<c0078658>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[ 8828.530120] r9:fa241100
[ 8828.530120] r8:fa240100
[ 8828.530120] r7:ef09ff58
[ 8828.530120] r6:fa24010c
[ 8828.530120] r5:c090a7a8
[ 8828.530120] r4:c0927748
[ 8828.530151] [<c0009470>] (gic_handle_irq) from [<c0014bd4>] (__irq_svc+0x54/0x90)
[ 8828.530151] Exception stack(0xef09ff58 to 0xef09ffa0)
[ 8828.530151] ff40: 00000000 ef7b7390
[ 8828.530181] ff60: fe600000 00000000 ef09e000 c090a49c c0903364 c06aa124 ef09ffc8 c09466e3
[ 8828.530181] ff80: c090a4e8 ef09ffb4 ef09ff94 ef09ffa8 c002878c c0010ba8 60000113 ffffffff
[ 8828.530181] r9:c09466e3
[ 8828.530181] r8:ef09ffc8
[ 8828.530181] r7:ef09ff8c
[ 8828.530212] r6:ffffffff
[ 8828.530212] r5:60000113
[ 8828.530212] r4:c0010ba8
[ 8828.530212] [<c0010b80>] (arch_cpu_idle) from [<c006efc4>] (default_idle_call+0x28/0x34)
[ 8828.530212] [<c006ef9c>] (default_idle_call) from [<c006f0e4>] (cpu_startup_entry+0x114/0x18c)
[ 8828.530212] [<c006efd0>] (cpu_startup_entry) from [<c0016128>] (secondary_start_kernel+0x164/0x170)
[ 8828.530273] r7:c0949288
[ 8828.530273] r4:c091c700
[ 8828.530273] [<c0015fc4>] (secondary_start_kernel) from [<8000958c>] (0x8000958c)
[ 8828.530273] r5:00000051
[ 8828.530303] r4:af08406a
[ 8828.902252] 0-...: (1 ticks this GP) idle=2cd/1/0 softirq=62833/62833 fqs=0
[ 8828.902252] (t=25079 jiffies g=27810 c=27809 q=2)
[ 8828.902252] rcu_preempt kthread starved for 25079 jiffies! g27810 c27809 f0x0 s3 ->state=0x1
[ 8828.922241] Task dump for CPU 0:
[ 8828.922241] swapper/0 R running 0 0 0 0x00000000
[ 8828.928619] Backtrace:
[ 8828.935760] [<c0013e9c>] (dump_backtrace) from [<c0014098>] (show_stack+0x18/0x1c)
[ 8828.937957] r7:c090a514 r6:800c0193 r5:00000000 r4:c090cca8
[ 8828.950653] [<c0014080>] (show_stack) from [<c005a7d8>] (sched_show_task+0xbc/0x110)
[ 8828.952209] [<c005a71c>] (sched_show_task) from [<c005cc60>] (dump_cpu_task+0x40/0x48)
[ 8828.962219] r5:c0919680 r4:00000000
[ 8828.972351] [<c005cc20>] (dump_cpu_task) from [<c00821f8>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[ 8828.972351] r5:c0919680 r4:00000001
[ 8828.984893] [<c008215c>] (rcu_dump_cpu_stacks) from [<c008630c>] (rcu_check_callbacks+0x504/0x8e4)
[ 8828.994384] r9:c090a514 r8:c0919680 r7:00000002 r6:2eea8000 r5:c0907300 r4:ef7af300
[ 8828.994384] [<c0085e08>] (rcu_check_callbacks) from [<c0089560>] (update_process_times+0x40/0x6c)
[ 8829.011993] r10:c009a514 r9:00000001 r8:ef7abc4c r7:00000807 r6:8d3a6293 r5:00000000
[ 8829.017883] r4:c090cca8
[ 8829.017883] [<c0089520>] (update_process_times) from [<c009a510>] (tick_sched_handle+0x50/0x54)
[ 8829.032196] r5:c0909f10 r4:ef7abe40
[ 8829.032196] [<c009a4c0>] (tick_sched_handle) from [<c009a564>] (tick_sched_timer+0x50/0x94)
[ 8829.042205] [<c009a514>] (tick_sched_timer) from [<c0089f74>] (__hrtimer_run_queues+0x110/0x1a0)
[ 8829.054138] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
[ 8829.054138] [<c0089e64>] (__hrtimer_run_queues) from [<c008a604>] (hrtimer_interrupt+0xac/0x1f8)
[ 8829.062194] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8 r6:ffffffff r5:00000003
[ 8829.072204] r4:ef7abc00
[ 8829.077758] [<c008a558>] (hrtimer_interrupt) from [<c00170ec>] (twd_handler+0x38/0x48)
[ 8829.077758] r10:c090a4e8 r9:fa241100 r8:00000011 r7:ef028780 r6:c092774c r5:ef005cc0
[ 8829.092956] r4:00000001
[ 8829.099822] [<c00170b4>] (twd_handler) from [<c007c888>] (handle_percpu_devid_irq+0x74/0x8c)
[ 8829.108734] r5:ef005cc0 r4:ef7b1740
[ 8829.112548] [<c007c814>] (handle_percpu_devid_irq) from [<c00783e4>] (generic_handle_irq+0x2c/0x3c)
[ 8829.112548] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000 r5:00000000 r4:c09043e8
[ 8829.130340] [<c00783b8>] (generic_handle_irq) from [<c00786bc>] (__handle_domain_irq+0x64/0xbc)
[ 8829.139556] [<c0078658>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[ 8829.148376] r9:fa241100 r8:fa240100 r7:c0909f10 r6:fa24010c r5:c090a7a8 r4:c0927748
[ 8829.156616] [<c0009470>] (gic_handle_irq) from [<c0014bd4>] (__irq_svc+0x54/0x90)
[ 8829.164520] Exception stack(0xc0909f10 to 0xc0909f58)
[ 8829.167755] 9f00: 00000000 ef7ab390 fe600000 00000000
[ 8829.178527] 9f20: c0908000 c090a49c c0903364 c06aa124 c0909f80 c09466e3 c090a4e8 c0909f6c
[ 8829.178527] 9f40: c0909f4c c0909f60 c002878c c0010ba8 600c0113 ffffffff
[ 8829.194183] r9:c09466e3 r8:c0909f80 r7:c0909f44 r6:ffffffff r5:600c0113 r4:c0010ba8
[ 8829.194183] [<c0010b80>] (arch_cpu_idle) from [<c006efc4>] (default_idle_call+0x28/0x34)
[ 8829.202423] [<c006ef9c>] (default_idle_call) from [<c006f0e4>] (cpu_startup_entry+0x114/0x18c)
[ 8829.219940] [<c006efd0>] (cpu_startup_entry) from [<c06a120c>] (rest_init+0x90/0x94)
[ 8829.228302] r7:ffffffff r4:00000002
[ 8829.228302] [<c06a117c>] (rest_init) from [<c08becec>] (start_kernel+0x370/0x37c)
[ 8829.234436] r5:c0949000 r4:00000000
[ 8829.243804] [<c08be97c>] (start_kernel) from [<8000807c>] (0x8000807c)
$ uptime
00:40:54 up 4:19, load average: 0.30, 0.20, 0.16
$
[-- Attachment #3: dmesg-4.4-rc7 --]
[-- Type: application/octet-stream, Size: 34681 bytes --]
rfg@bigred:~> ssh rfg@192.168.1.100
rfg@192.168.1.100's password:
$
$
$
$ uptime
14:28:27 up 6 days, 16:47, load average: 0.14, 0.14, 0.13
$
$
$ dmesg
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 4.4.0-rc7-panda (rfg@red.dwarf) (gcc version 5.3.0 (Buildroot 2016.02-git-00183-g1682aee) ) #1 SMP PREEMPT Mon Dec 28 14:27:40 AEDT 2015
[ 0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] Machine model: TI OMAP4 PandaBoard-ES
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] OMAP4: Map 0xbfe00000 to fe600000 for dram barrier
[ 0.000000] On node 0 totalpages: 261632
[ 0.000000] free_area_init_node: node 0, pgdat c0945940, node_mem_map ef7f9000
[ 0.000000] Normal zone: 1536 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 196608 pages, LIFO batch:31
[ 0.000000] HighMem zone: 65024 pages, LIFO batch:15
[ 0.000000] OMAP4460 ES1.1
[ 0.000000] PERCPU: Embedded 12 pages/cpu @ef7ab000 s17920 r8192 d23040 u49152
[ 0.000000] pcpu-alloc: s17920 r8192 d23040 u49152 alloc=12*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260096
[ 0.000000] Kernel command line: root=/dev/mmcblk0p2 rootwait console=ttyO2,115200
[ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Memory: 1027368K/1046528K available (6783K kernel code, 250K rwdata, 2132K rodata, 296K init, 294K bss, 19160K reserved, 0K cma-reserved, 260096K highmem)
[ 0.000000] Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
vmalloc : 0xf0800000 - 0xff800000 ( 240 MB)
lowmem : 0xc0000000 - 0xf0000000 ( 768 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc08bd0d4 (8917 kB)
.init : 0xc08be000 - 0xc0908000 ( 296 kB)
.data : 0xc0908000 - 0xc0946920 ( 251 kB)
.bss : 0xc0949000 - 0xc0992bb8 ( 295 kB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 32.
[ 0.000000] NR_IRQS:16 nr_irqs:16 16
[ 0.000000] L2C: platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C: DT/platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C-310 enabling early BRESP for Cortex-A9
[ 0.000000] OMAP L2C310: ROM does not support power control setting
[ 0.000000] L2C-310 dynamic clock gating disabled, standby mode disabled
[ 0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
[ 0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e470000
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node bandgap_fclk
[ 0.000000] OMAP clockevent source: timer1 at 32768 Hz
[ 0.000000] clocksource: 32k_counter: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 58327039986419 ns
[ 0.000030] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65535999984741ns
[ 0.000030] OMAP clocksource: 32k_counter at 32768 Hz
[ 0.000885] Console: colour dummy device 80x30
[ 0.000885] Calibrating delay loop... 1397.55 BogoMIPS (lpj=6987776)
[ 0.089233] pid_max: default: 32768 minimum: 301
[ 0.089324] Security Framework initialized
[ 0.089355] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.089355] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.089965] Initializing cgroup subsys io
[ 0.089965] Initializing cgroup subsys devices
[ 0.089996] Initializing cgroup subsys freezer
[ 0.090026] CPU: Testing write buffer coherency: ok
[ 0.090240] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.090301] Setting up static identity map for 0x80008280 - 0x800082d8
[ 0.209228] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.209289] Brought up 2 CPUs
[ 0.209320] SMP: Total of 2 processors activated (2795.11 BogoMIPS).
[ 0.209320] CPU: All CPU(s) started in SVC mode.
[ 0.210113] devtmpfs: initialized
[ 0.227264] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[ 0.228942] omap_hwmod: l3_main_3 using broken dt data from ocp
[ 0.230499] omap_hwmod: l3_main_2 using broken dt data from ocp
[ 0.326385] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.326629] xor: measuring software checksum speed
[ 0.419219] arm4regs : 1006.000 MB/sec
[ 0.519195] 8regs : 804.800 MB/sec
[ 0.619201] 32regs : 760.800 MB/sec
[ 0.619232] xor: using function: arm4regs (1006.000 MB/sec)
[ 0.619476] pinctrl core: initialized pinctrl subsystem
[ 0.621337] NET: Registered protocol family 16
[ 0.622131] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.631469] OMAP GPIO hardware version 0.1
[ 0.636688] omap-gpmc 50000000.gpmc: GPMC revision 6.0
[ 0.636718] gpmc_mem_init: disabling cs 0 mapped at 0x0-0x1000000
[ 0.637237] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.637603] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.637969] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.649597] platform 4b501000.aes: Cannot lookup hwmod 'aes'
[ 0.649902] platform 480a5000.des: Cannot lookup hwmod 'des'
[ 0.655822] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[ 0.655822] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.656616] omap4_sram_init:Unable to get sram pool needed to handle errata I688
[ 0.656829] OMAP DMA hardware revision 0.0
[ 0.889404] raid6: int32x1 gen() 103 MB/s
[ 1.059417] raid6: int32x1 xor() 84 MB/s
[ 1.229278] raid6: int32x2 gen() 134 MB/s
[ 1.399444] raid6: int32x2 xor() 97 MB/s
[ 1.569580] raid6: int32x4 gen() 145 MB/s
[ 1.739349] raid6: int32x4 xor() 100 MB/s
[ 1.909271] raid6: int32x8 gen() 160 MB/s
[ 2.079315] raid6: int32x8 xor() 98 MB/s
[ 2.079315] raid6: using algorithm int32x8 gen() 160 MB/s
[ 2.079345] raid6: .... xor() 98 MB/s, rmw enabled
[ 2.079345] raid6: using intx1 recovery algorithm
[ 2.086578] omap-dma-engine 4a056000.dma-controller: OMAP DMA engine driver
[ 2.159332] reg-fixed-voltage wl12xx_vmmc: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_wl12xx_gpio, deferring probe
[ 2.163482] omap-iommu 4a066000.mmu: 4a066000.mmu registered
[ 2.163604] omap-iommu 55082000.mmu: 55082000.mmu registered
[ 2.164550] SCSI subsystem initialized
[ 2.164825] usbcore: registered new interface driver usbfs
[ 2.164947] usbcore: registered new interface driver hub
[ 2.165100] usbcore: registered new device driver usb
[ 2.166046] omap_i2c 48070000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c1_pins, deferring probe
[ 2.166107] omap_i2c 48072000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c2_pins, deferring probe
[ 2.166137] omap_i2c 48060000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c3_pins, deferring probe
[ 2.166168] omap_i2c 48350000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c4_pins, deferring probe
[ 2.166442] Linux video capture interface: v2.00
[ 2.167205] Advanced Linux Sound Architecture Driver Initialized.
[ 2.168579] clocksource: Switched to clocksource 32k_counter
[ 2.185943] NET: Registered protocol family 2
[ 2.186523] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[ 2.186645] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[ 2.186767] TCP: Hash tables configured (established 8192 bind 8192)
[ 2.186859] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 2.186920] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 2.187103] NET: Registered protocol family 1
[ 2.187438] RPC: Registered named UNIX socket transport module.
[ 2.187469] RPC: Registered udp transport module.
[ 2.187469] RPC: Registered tcp transport module.
[ 2.187469] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 2.189758] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[ 2.192687] futex hash table entries: 512 (order: 3, 32768 bytes)
[ 2.204956] NFS: Registering the id_resolver key type
[ 2.205017] Key type id_resolver registered
[ 2.205017] Key type id_legacy registered
[ 2.205047] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 2.205078] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 2.206542] ntfs: driver 2.1.32 [Flags: R/O].
[ 2.206939] fuse init (API version 7.23)
[ 2.211364] jitterentropy: Initialization failed with host not compliant with requirements: 2
[ 2.211456] bounce: pool size: 64 pages
[ 2.211456] io scheduler noop registered
[ 2.211486] io scheduler deadline registered
[ 2.211517] io scheduler cfq registered (default)
[ 2.212402] pinctrl-single 4a100040.pinmux: 203 pins at pa fc100040 size 406
[ 2.212615] pinctrl-single 4a31e040.pinmux: 28 pins at pa fc31e040 size 56
[ 2.215911] 58000000.dss supply vdda_video not found, using dummy regulator
[ 2.216033] OMAP DSS rev 4.0
[ 2.216186] omapdss_dss 58000000.dss: bound 58001000.dispc (ops dispc_component_ops)
[ 2.216400] omapdss_dss 58000000.dss: bound 58005000.encoder (ops dsi_component_ops)
[ 2.216949] omapdss_dss 58000000.dss: bound 58006000.encoder (ops hdmi4_component_ops)
[ 2.219360] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 2.221282] omap_uart 4806a000.serial: no wakeirq for uart0
[ 2.221405] 4806a000.serial: ttyO0 at MMIO 0x4806a000 (irq = 226, base_baud = 3000000) is a OMAP UART0
[ 2.221984] 4806c000.serial: ttyO1 at MMIO 0x4806c000 (irq = 227, base_baud = 3000000) is a OMAP UART1
[ 2.222534] 48020000.serial: ttyO2 at MMIO 0x48020000 (irq = 228, base_baud = 3000000) is a OMAP UART2
[ 3.180755] console [ttyO2] enabled
[ 3.184906] 4806e000.serial: ttyO3 at MMIO 0x4806e000 (irq = 229, base_baud = 3000000) is a OMAP UART3
[ 3.195892] [drm] Initialized drm 1.1.0 20060810
[ 3.224334] loop: module loaded
[ 3.231048] usbcore: registered new interface driver smsc95xx
[ 3.237091] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 3.244079] ehci-omap: OMAP-EHCI Host Controller driver
[ 3.269958] ehci-omap 4a064c00.ehci: EHCI Host Controller
[ 3.275634] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1
[ 3.283721] ehci-omap 4a064c00.ehci: irq 336, io mem 0x4a064c00
[ 3.308624] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00
[ 3.315002] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 3.322143] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 3.329711] usb usb1: Product: EHCI Host Controller
[ 3.334808] usb usb1: Manufacturer: Linux 4.4.0-rc7-panda ehci_hcd
[ 3.341278] usb usb1: SerialNumber: 4a064c00.ehci
[ 3.346862] hub 1-0:1.0: USB hub found
[ 3.350860] hub 1-0:1.0: 3 ports detected
[ 3.355743] usbcore: registered new interface driver cdc_wdm
[ 3.361816] usbcore: registered new interface driver usb-storage
[ 3.368530] i2c /dev entries driver
[ 3.372344] usbcore: registered new interface driver uvcvideo
[ 3.378356] USB Video Class driver (1.1.1)
[ 3.383880] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[ 3.391418] device-mapper: uevent: version 1.0.3
[ 3.396606] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com
[ 3.469665] hidraw: raw HID events driver (C) Jiri Kosina
[ 3.475524] usbcore: registered new interface driver usbhid
[ 3.481353] usbhid: USB HID core driver
[ 3.485870] emif 4c000000.emif: emif_probe: device configured with addr = f08a8000 and IRQ247
[ 3.495208] emif 4d000000.emif: emif_probe: device configured with addr = f08aa000 and IRQ248
[ 3.506134] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 3.514068] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 3.521545] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 3.527252] Initializing XFRM netlink socket
[ 3.531799] NET: Registered protocol family 17
[ 3.536743] Key type dns_resolver registered
[ 3.541412] twl: not initialized
[ 3.544830] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.553436] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.561950] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.570434] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.578918] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.587402] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.595886] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1410000 Vs max 1316660
[ 3.604370] omap2_set_init_voltage: unable to find boot up OPP for vdd_mpu
[ 3.611999] omap2_set_init_voltage: unable to set vdd_mpu
[ 3.617706] omap2_set_init_voltage: unable to find boot up OPP for vdd_core
[ 3.625091] omap2_set_init_voltage: unable to set vdd_core
[ 3.630920] omap2_set_init_voltage: unable to find boot up OPP for vdd_iva
[ 3.638214] omap2_set_init_voltage: unable to set vdd_iva
[ 3.644805] Power Management for TI OMAP4+ devices.
[ 3.649993] OMAP4 PM: u-boot >= v2012.07 is required for full PM support
[ 3.657196] ThumbEE CPU extension supported.
[ 3.661804] Registering SWP/SWPB emulation handler
[ 3.661834] SmartReflex Class3 initialized
[ 3.671295] usb 1-1: new high-speed USB device number 2 using ehci-omap
[ 3.696441] Btrfs loaded
[ 3.702056] dmm 4e000000.dmm: initialized all PAT entries
[ 3.722503] Skipping twl internal clock init and using bootloader value (unknown osc rate)
[ 3.753234] twl 0-0048: PIH (irq 340) nested IRQs
[ 3.759704] twl_rtc 48070000.i2c:twl@48:rtc: Enabling TWL-RTC
[ 3.768402] twl_rtc 48070000.i2c:twl@48:rtc: rtc core: registered 48070000.i2c:twl@48 as rtc0
[ 3.788146] twl6040 0-004b: clk32k is not handled
[ 3.795928] omap_i2c 48070000.i2c: bus 0 rev0.11 at 400 kHz
[ 3.802734] omap_i2c 48072000.i2c: bus 1 rev0.11 at 400 kHz
[ 3.809570] omap_i2c 48060000.i2c: bus 2 rev0.11 at 100 kHz
[ 3.816223] omap_i2c 48350000.i2c: bus 3 rev0.11 at 400 kHz
[ 3.869079] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
[ 3.876220] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 3.887817] hub 1-1:1.0: USB hub found
[ 3.892242] hub 1-1:1.0: 5 ports detected
[ 4.007568] omap-abe-twl6040 sound: twl6040-legacy <-> 40132000.mcpdm mapping ok
[ 4.019409] twl_rtc 48070000.i2c:twl@48:rtc: setting system clock to 2015-12-28 10:40:33 UTC (1451299233)
[ 4.029541] sr_init: No PMIC hook to init smartreflex
[ 4.034942] smartreflex smartreflex.0: omap_sr_probe: SmartReflex driver initialized
[ 4.043609] smartreflex smartreflex.1: omap_sr_probe: SmartReflex driver initialized
[ 4.052154] smartreflex smartreflex.2: omap_sr_probe: SmartReflex driver initialized
[ 4.071533] VDAC: disabling
[ 4.075012] VANA: disabling
[ 4.078216] mmc0: host does not support reading read-only switch, assuming write-enable
[ 4.087036] VUSB: disabling
[ 4.090423] ALSA device list:
[ 4.090423] #0: PandaBoardES
[ 4.097229] mmc0: new high speed SDHC card at address e624
[ 4.103149] Waiting for root device /dev/mmcblk0p2...
[ 4.113708] mmcblk0: mmc0:e624 SL32G 28.7 GiB
[ 4.146820] mmcblk0: p1 p2 p3 p4
[ 4.156127] omap_hsmmc 480d5000.mmc: card claims to support voltages below defined range
[ 4.174774] mmc1: queuing unknown CIS tuple 0x91 (3 bytes)
[ 4.181762] mmc1: new SDIO card at address 0001
[ 4.208618] usb 1-1.1: new high-speed USB device number 3 using ehci-omap
[ 4.232116] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
[ 4.241333] EXT4-fs (mmcblk0p2): mounting ext2 file system using the ext4 subsystem
[ 4.271850] EXT4-fs (mmcblk0p2): mounted filesystem without journal. Opts: (null)
[ 4.281280] VFS: Mounted root (ext2 filesystem) readonly on device 179:2.
[ 4.294158] devtmpfs: mounted
[ 4.294158] Freeing unused kernel memory: 296K (c08be000 - c0908000)
[ 4.329345] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[ 4.336639] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 4.347320] smsc95xx v1.0.4
[ 4.441009] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-4a064c00.ehci-1.1, smsc95xx USB 2.0 Ethernet, 6e:e4:2d:83:0b:9f
[ 4.707611] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 4.840332] random: dd urandom read with 32 bits of entropy available
[ 13.230682] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[ 22.965057] Bluetooth: Core ver 2.21
[ 22.969055] NET: Registered protocol family 31
[ 22.969177] Bluetooth: HCI device and connection manager initialized
[ 22.980987] Bluetooth: HCI socket layer initialized
[ 22.981048] Bluetooth: L2CAP socket layer initialized
[ 22.991943] Bluetooth: SCO socket layer initialized
[ 30.553955] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[ 30.565460] NFSD: starting 90-second grace period (net c093fe80)
[ 71.098693] random: nonblocking pool is initialized
[ 80.267089] wlcore: loaded
[ 103.294311] wlcore: firmware booted (Rev 6.3.10.0.133)
[ 106.967620] wlan0: authenticate with 90:f6:52:81:ee:70
[ 107.006164] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[ 107.014739] wlan0: authenticated
[ 107.035919] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[ 107.046752] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=2)
[ 107.065765] wlan0: associated
[ 107.124145] wlcore: Association completed.
[ 150.622131] NFSD: Unable to end grace period: -110
[ 168.167938] Adding 2097148k swap on /dev/mmcblk0p3. Priority:-1 extents:1 across:2097148k SS
[146355.192016] INFO: rcu_preempt detected stalls on CPUs/tasks:
[146355.198089] (detected by 0, t=8509 jiffies, g=470652, c=470651, q=111)
[146355.204528] All QSes seen, last rcu_preempt kthread activity 8508 (14605476-14596968), jiffies_till_next_fqs=1, root ->qsmask 0x0
[146355.217620] swapper/0 R running 0 0 0 0x00000000
[146355.224060] Backtrace:
[146355.227172] [<c0013e9c>] (dump_backtrace) from [<c0014098>] (show_stack+0x18/0x1c)
[146355.234313] r7:00000001 r6:2eea8000 r5:00000000 r4:c090cca8
[146355.241394] [<c0014080>] (show_stack) from [<c005a7f8>] (sched_show_task+0xbc/0x110)
[146355.249053] [<c005a73c>] (sched_show_task) from [<c00866f8>] (rcu_check_callbacks+0x8d0/0x8e4)
[146355.258911] r5:c0907300 r4:ef7af300
[146355.262481] [<c0085e28>] (rcu_check_callbacks) from [<c0089580>] (update_process_times+0x40/0x6c)
[146355.272277] r10:c009a534 r9:00000001 r8:ef7abc4c r7:0000851b r6:dfabbc40 r5:00000000
[146355.280487] r4:c090cca8
[146355.283416] [<c0089540>] (update_process_times) from [<c009a530>] (tick_sched_handle+0x50/0x54)
[146355.292785] r5:c0909f10 r4:ef7abe40
[146355.296234] [<c009a4e0>] (tick_sched_handle) from [<c009a584>] (tick_sched_timer+0x50/0x94)
[146355.305603] [<c009a534>] (tick_sched_timer) from [<c0089f94>] (__hrtimer_run_queues+0x110/0x1a0)
[146355.314178] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
[146355.320251] [<c0089e84>] (__hrtimer_run_queues) from [<c008a624>] (hrtimer_interrupt+0xac/0x1f8)
[146355.330078] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8 r6:ffffffff r5:00000003
[146355.338531] r4:ef7abc00
[146355.341125] [<c008a578>] (hrtimer_interrupt) from [<c00170ec>] (twd_handler+0x38/0x48)
[146355.349670] r10:c090a4e8 r9:fa241100 r8:00000011 r7:ef028780 r6:c092774c r5:ef005cc0
[146355.358001] r4:00000001
[146355.361297] [<c00170b4>] (twd_handler) from [<c007c8a8>] (handle_percpu_devid_irq+0x74/0x8c)
[146355.370361] r5:ef005cc0 r4:ef7b1740
[146355.374206] [<c007c834>] (handle_percpu_devid_irq) from [<c0078404>] (generic_handle_irq+0x2c/0x3c)
[146355.383514] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000 r5:00000000 r4:c09043e8
[146355.391876] [<c00783d8>] (generic_handle_irq) from [<c00786dc>] (__handle_domain_irq+0x64/0xbc)
[146355.401550] [<c0078678>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[146355.403930] r9:fa241100 r8:fa240100 r7:c0909f10 r6:fa24010c r5:c090a7a8 r4:c0927748
[146355.418792] [<c0009470>] (gic_handle_irq) from [<c0014bd4>] (__irq_svc+0x54/0x90)
[146355.418792] Exception stack(0xc0909f10 to 0xc0909f58)
[146355.428771] 9f00: 00000000 ef7ab390 fe600000 00000000
[146355.440979] 9f20: c0908000 c090a49c c0903364 c06aa124 c0909f80 c09466e3 c090a4e8 c0909f6c
[146355.448791] 9f40: c0909f4c c0909f60 c00287ac c0010ba8 60080113 ffffffff
[146355.456817] r9:c09466e3 r8:c0909f80 r7:c0909f44 r6:ffffffff r5:60080113 r4:c0010ba8
[146355.462127] [<c0010b80>] (arch_cpu_idle) from [<c006efe4>] (default_idle_call+0x28/0x34)
[146355.462127] [<c006efbc>] (default_idle_call) from [<c006f104>] (cpu_startup_entry+0x114/0x18c)
[146355.483032] [<c006eff0>] (cpu_startup_entry) from [<c06a1284>] (rest_init+0x90/0x94)
[146355.491302] r7:ffffffff r4:00000002
[146355.494537] [<c06a11f4>] (rest_init) from [<c08becec>] (start_kernel+0x370/0x37c)
[146355.501037] r5:c0949000 r4:00000000
[146355.503234] [<c08be97c>] (start_kernel) from [<8000807c>] (0x8000807c)
[146355.508789] rcu_preempt kthread starved for 8508 jiffies! g470652 c470651 f0x2 s3 ->state=0x100
[303742.013122] INFO: rcu_preempt self-detected stall on CPU
[303742.013549] 1-...: (1 ticks this GP) idle=ef9/1/0 softirq=2937608/2937608 fqs=0
[303742.013549] (t=8232 jiffies g=975243 c=975242 q=8)
[303742.031890] rcu_preempt kthread starved for 8232 jiffies! g975243 c975242 f0x0 s3 ->state=0x1
[303742.031890] INFO: rcu_preempt self-detected stall on CPU
[303742.041076] 0-...: (1 ticks this GP) idle=83d/1/0 softirq=2182590/2182590 fqs=0
[303742.041076] (t=8232 jiffies g=975243 c=975242 q=8)
[303742.041076] rcu_preempt kthread starved for 8232 jiffies! g975243 c975242 f0x0 s3 ->state=0x1
[303742.041076] Task dump for CPU 0:
[303742.041107] swapper/0 R running 0 0 0 0x00000000
[303742.041107] Backtrace:
[303742.041107] [<c0013e9c>] (dump_backtrace) from [<c0014098>] (show_stack+0x18/0x1c)
[303742.041168] r7:c090a514 r6:800c0193 r5:00000000 r4:c090cca8
[303742.041168] [<c0014080>] (show_stack) from [<c005a7f8>] (sched_show_task+0xbc/0x110)
[303742.041168] [<c005a73c>] (sched_show_task) from [<c005cc80>] (dump_cpu_task+0x40/0x48)
[303742.041229] r5:c0919680 r4:00000000
[303742.041229] [<c005cc40>] (dump_cpu_task) from [<c0082218>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[303742.041259] r5:c0919680 r4:00000001
[303742.041259] [<c008217c>] (rcu_dump_cpu_stacks) from [<c008632c>] (rcu_check_callbacks+0x504/0x8e4)
[303742.041290] r9:c090a514 r8:c0919680 r7:00000008 r6:2eea8000 r5:c0907300 r4:ef7af300
[303742.041290] [<c0085e28>] (rcu_check_callbacks) from [<c0089580>] (update_process_times+0x40/0x6c)
[303742.041320] r10:c009a534 r9:00000001 r8:ef7abc4c r7:00011440 r6:496ee983 r5:00000000
[303742.041320] r4:c090cca8
[303742.041320] [<c0089540>] (update_process_times) from [<c009a530>] (tick_sched_handle+0x50/0x54)
[303742.041320] r5:c0909f10 r4:ef7abe40
[303742.041381] [<c009a4e0>] (tick_sched_handle) from [<c009a584>] (tick_sched_timer+0x50/0x94)
[303742.041381] [<c009a534>] (tick_sched_timer) from [<c0089f94>] (__hrtimer_run_queues+0x110/0x1a0)
[303742.041412] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
[303742.041412] [<c0089e84>] (__hrtimer_run_queues) from [<c008a624>] (hrtimer_interrupt+0xac/0x1f8)
[303742.041442] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8 r6:ffffffff r5:00000003
[303742.041442] r4:ef7abc00
[303742.041473] [<c008a578>] (hrtimer_interrupt) from [<c00170ec>] (twd_handler+0x38/0x48)
[303742.041473] r10:c090a4e8 r9:fa241100 r8:00000011 r7:ef028780 r6:c092774c r5:ef005cc0
[303742.041473] r4:00000001
[303742.041503] [<c00170b4>] (twd_handler) from [<c007c8a8>] (handle_percpu_devid_irq+0x74/0x8c)
[303742.041534] r5:ef005cc0 r4:ef7b1740
[303742.041534] [<c007c834>] (handle_percpu_devid_irq) from [<c0078404>] (generic_handle_irq+0x2c/0x3c)
[303742.041564] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000 r5:00000000 r4:c09043e8
[303742.041564] [<c00783d8>] (generic_handle_irq) from [<c00786dc>] (__handle_domain_irq+0x64/0xbc)
[303742.041564] [<c0078678>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[303742.041595] r9:fa241100 r8:fa240100 r7:c0909f10 r6:fa24010c r5:c090a7a8 r4:c0927748
[303742.041595] [<c0009470>] (gic_handle_irq) from [<c0014bd4>] (__irq_svc+0x54/0x90)
[303742.041625] Exception stack(0xc0909f10 to 0xc0909f58)
[303742.041625] 9f00: 00000000 ef7ab390 fe600000 00000000
[303742.041625] 9f20: c0908000 c090a49c c0903364 c06aa124 c0909f80 c09466e3 c090a4e8 c0909f6c
[303742.041656] 9f40: c0909f4c c0909f60 c00287ac c0010ba8 600c0113 ffffffff
[303742.041656] r9:c09466e3 r8:c0909f80 r7:c0909f44 r6:ffffffff r5:600c0113 r4:c0010ba8
[303742.041656] [<c0010b80>] (arch_cpu_idle) from [<c006efe4>] (default_idle_call+0x28/0x34)
[303742.041717] [<c006efbc>] (default_idle_call) from [<c006f104>] (cpu_startup_entry+0x114/0x18c)
[303742.041717] [<c006eff0>] (cpu_startup_entry) from [<c06a1284>] (rest_init+0x90/0x94)
[303742.041748] r7:ffffffff r4:00000002
[303742.041748] [<c06a11f4>] (rest_init) from [<c08becec>] (start_kernel+0x370/0x37c)
[303742.041748] r5:c0949000 r4:00000000
[303742.041809] [<c08be97c>] (start_kernel) from [<8000807c>] (0x8000807c)
[303742.041809] Task dump for CPU 1:
[303742.041809] swapper/1 R running 0 0 1 0x00000000
[303742.041809] Backtrace:
[303742.041839] Backtrace aborted due to bad frame pointer <ef09ff9c>
[303742.412719] Task dump for CPU 1:
[303742.412719] swapper/1 R running 0 0 1 0x00000000
[303742.422729] Backtrace:
[303742.422729] [<c0013e9c>] (dump_backtrace) from [<c0014098>] (show_stack+0x18/0x1c)
[303742.438903] r7:c090a514 r6:80080193 r5:00000001 r4:ef076f80
[303742.444458] [<c0014080>] (show_stack) from [<c005a7f8>] (sched_show_task+0xbc/0x110)
[303742.453308] [<c005a73c>] (sched_show_task) from [<c005cc80>] (dump_cpu_task+0x40/0x48)
[303742.453308] r5:c0919680 r4:00000001
[303742.465667] [<c005cc40>] (dump_cpu_task) from [<c0082218>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[303742.473846] r5:c0919680 r4:00000002
[303742.478393] [<c008217c>] (rcu_dump_cpu_stacks) from [<c008632c>] (rcu_check_callbacks+0x504/0x8e4)
[303742.478393] r9:c090a514 r8:c0919680 r7:00000008 r6:2eeb4000 r5:c0907300 r4:ef7bb300
[303742.495483] [<c0085e28>] (rcu_check_callbacks) from [<c0089580>] (update_process_times+0x40/0x6c)
[303742.505767] r10:c009a534 r9:00000001 r8:ef7b7c4c r7:00011440 r6:484f98ba r5:00000000
[303742.505767] r4:ef076f80
[303742.505767] [<c0089540>] (update_process_times) from [<c009a530>] (tick_sched_handle+0x50/0x54)
[303742.519683] r5:ef09ff58 r4:ef7b7e40
[303742.519683] [<c009a4e0>] (tick_sched_handle) from [<c009a584>] (tick_sched_timer+0x50/0x94)
[303742.532806] [<c009a534>] (tick_sched_timer) from [<c0089f94>] (__hrtimer_run_queues+0x110/0x1a0)
[303742.548461] r7:00000000 r6:ef7b7c40 r5:ef7b7e40 r4:ef7b7c00
[303742.554565] [<c0089e84>] (__hrtimer_run_queues) from [<c008a624>] (hrtimer_interrupt+0xac/0x1f8)
[303742.554565] r10:ef7b7c78 r9:ef7b7c98 r8:ef7b7c14 r7:ef7b7cb8 r6:ffffffff r5:00000003
[303742.572357] r4:ef7b7c00
[303742.575134] [<c008a578>] (hrtimer_interrupt) from [<c00170ec>] (twd_handler+0x38/0x48)
[303742.583068] r10:c090a4e8 r9:fa241100 r8:00000011 r7:ef028780 r6:c092774c r5:ef005cc0
[303742.590637] r4:00000001
[303742.594787] [<c00170b4>] (twd_handler) from [<c007c8a8>] (handle_percpu_devid_irq+0x74/0x8c)
[303742.594787] r5:ef005cc0 r4:ef7bd740
[303742.607696] [<c007c834>] (handle_percpu_devid_irq) from [<c0078404>] (generic_handle_irq+0x2c/0x3c)
[303742.614746] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000 r5:00000000 r4:c09043e8
[303742.625671] [<c00783d8>] (generic_handle_irq) from [<c00786dc>] (__handle_domain_irq+0x64/0xbc)
[303742.625671] [<c0078678>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[303742.639770] r9:fa241100 r8:fa240100 r7:ef09ff58 r6:fa24010c r5:c090a7a8 r4:c0927748
[303742.652221] [<c0009470>] (gic_handle_irq) from [<c0014bd4>] (__irq_svc+0x54/0x90)
[303742.655181] Exception stack(0xef09ff58 to 0xef09ffa0)
[303742.664672] ff40: 00000000 ef7b7390
[303742.674407] ff60: fe600000 00000000 ef09e000 c090a49c c0903364 c06aa124 ef09ffc8 c09466e3
[303742.680755] ff80: c090a4e8 ef09ffb4 ef09ff94 ef09ffa8 c00287ac c0010ba8 60080113 ffffffff
[303742.686614] r9:c09466e3 r8:ef09ffc8 r7:ef09ff8c r6:ffffffff r5:60080113 r4:c0010ba8
[303742.700225] [<c0010b80>] (arch_cpu_idle) from [<c006efe4>] (default_idle_call+0x28/0x34)
[303742.708892] [<c006efbc>] (default_idle_call) from [<c006f104>] (cpu_startup_entry+0x114/0x18c)
[303742.717071] [<c006eff0>] (cpu_startup_entry) from [<c0016128>] (secondary_start_kernel+0x164/0x170)
[303742.727752] r7:c0949288 r4:c091c700
[303742.727752] [<c0015fc4>] (secondary_start_kernel) from [<8000958c>] (0x8000958c)
[303742.739562] r5:00000051 r4:af08406a
[455008.512084] INFO: rcu_preempt detected stalls on CPUs/tasks:
[455008.516235] (detected by 1, t=12485 jiffies, g=1459674, c=1459673, q=5)
[455008.521606] All QSes seen, last rcu_preempt kthread activity 12484 (45470784-45458300), jiffies_till_next_fqs=1, root ->qsmask 0x0
[455008.536224] swapper/1 R running 0 0 1 0x00000000
[455008.536224] Backtrace:
[455008.544708] [<c0013e9c>] (dump_backtrace) from [<c0014098>] (show_stack+0x18/0x1c)
[455008.555511] r7:00000001 r6:2eeb4000 r5:00000001 r4:ef076f80
[455008.557250] [<c0014080>] (show_stack) from [<c005a7f8>] (sched_show_task+0xbc/0x110)
[455008.563323] [<c005a73c>] (sched_show_task) from [<c00866f8>] (rcu_check_callbacks+0x8d0/0x8e4)
[455008.579132] r5:c0907300 r4:ef7bb300
[455008.583038] [<c0085e28>] (rcu_check_callbacks) from [<c0089580>] (update_process_times+0x40/0x6c)
[455008.583221] r10:c009a534 r9:00000001 r8:ef7b7c4c r7:00019dd3 r6:c470b320 r5:00000000
[455008.600128] r4:ef076f80
[455008.601593] [<c0089540>] (update_process_times) from [<c009a530>] (tick_sched_handle+0x50/0x54)
[455008.611602] r5:ef09ff58 r4:ef7b7e40
[455008.615997] [<c009a4e0>] (tick_sched_handle) from [<c009a584>] (tick_sched_timer+0x50/0x94)
[455008.615997] [<c009a534>] (tick_sched_timer) from [<c0089f94>] (__hrtimer_run_queues+0x110/0x1a0)
[455008.635192] r7:00000000 r6:ef7b7c40 r5:ef7b7e40 r4:ef7b7c00
[455008.635223] [<c0089e84>] (__hrtimer_run_queues) from [<c008a624>] (hrtimer_interrupt+0xac/0x1f8)
[455008.635223] r10:ef7b7c78 r9:ef7b7c98 r8:ef7b7c14 r7:ef7b7cb8 r6:ffffffff r5:00000003
[455008.659118] r4:ef7b7c00
[455008.661895] [<c008a578>] (hrtimer_interrupt) from [<c00170ec>] (twd_handler+0x38/0x48)
[455008.661895] r10:c090a4e8 r9:fa241100 r8:00000011 r7:ef028780 r6:c092774c r5:ef005cc0
[455008.678771] r4:00000001
[455008.678771] [<c00170b4>] (twd_handler) from [<c007c8a8>] (handle_percpu_devid_irq+0x74/0x8c)
[455008.690582] r5:ef005cc0 r4:ef7bd740
[455008.691589] [<c007c834>] (handle_percpu_devid_irq) from [<c0078404>] (generic_handle_irq+0x2c/0x3c)
[455008.704132] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000 r5:00000000 r4:c09043e8
[455008.704132] [<c00783d8>] (generic_handle_irq) from [<c00786dc>] (__handle_domain_irq+0x64/0xbc)
[455008.721588] [<c0078678>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[455008.727569] r9:fa241100 r8:fa240100 r7:ef09ff58 r6:fa24010c r5:c090a7a8 r4:c0927748
[455008.731597] [<c0009470>] (gic_handle_irq) from [<c0014bd4>] (__irq_svc+0x54/0x90)
[455008.746704] Exception stack(0xef09ff58 to 0xef09ffa0)
[455008.752441] ff40: 00000000 ef7b7390
[455008.761199] ff60: fe600000 00000000 ef09e000 c090a49c c0903364 c06aa124 ef09ffc8 c09466e3
[455008.761962] ff80: c090a4e8 ef09ffb4 ef09ff94 ef09ffa8 c00287ac c0010ba8 600c0113 ffffffff
[455008.778686] r9:c09466e3 r8:ef09ffc8 r7:ef09ff8c r6:ffffffff r5:600c0113 r4:c0010ba8
[455008.787017] [<c0010b80>] (arch_cpu_idle) from [<c006efe4>] (default_idle_call+0x28/0x34)
[455008.791931] [<c006efbc>] (default_idle_call) from [<c006f104>] (cpu_startup_entry+0x114/0x18c)
[455008.791931] [<c006eff0>] (cpu_startup_entry) from [<c0016128>] (secondary_start_kernel+0x164/0x170)
[455008.814544] r7:c0949288 r4:c091c700
[455008.817047] [<c0015fc4>] (secondary_start_kernel) from [<8000958c>] (0x8000958c)
[455008.826354] r5:00000051 r4:af08406a
[455008.826354] rcu_preempt kthread starved for 12515 jiffies! g1459674 c1459673 f0x2 s3 ->state=0x100
[468942.424163] wlcore: Beacon loss detected. roles:0xff
[468947.227508] wlcore: Connection loss work (role_id: 0).
[468947.233062] wlan0: Connection to AP 90:f6:52:81:ee:70 lost
[468950.866912] wlan0: authenticate with 90:f6:52:81:ee:70
[468950.898040] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[468950.906616] wlan0: authenticated
[468950.934509] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[468950.945404] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=1)
[468950.964050] wlan0: associated
[468951.000183] wlcore: Association completed.
$
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: rcu_preempt self-detected stall on CPU from 4.4-rc4, since 3.17
2016-01-04 14:21 ` Ross Green
@ 2016-01-06 23:10 ` Ross Green
2016-01-08 18:35 ` Paul E. McKenney
0 siblings, 1 reply; 12+ messages in thread
From: Ross Green @ 2016-01-06 23:10 UTC (permalink / raw)
To: Paul McKenney
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
Mathieu Desnoyers, josh, tglx, peterz, rostedt, dhowells,
Eric Dumazet, dvhart, Frédéric Weisbecker, oleg,
pranith kumar
[-- Attachment #1: Type: text/plain, Size: 12747 bytes --]
Just for completeness I have attached another rcu_preempt stall
warning from a kinux-3.17.1 kernel.
So it looks like these stall warning go back that far.
It could be that the kernel instrumentation improved enough to detect
this situation around the 3.17 kernel. I can't find any earlier
records of these stall warnings before these kernels. I guess someone
must have suspected there was a potential for these stalls and hence
the detection facility.
Regards,
Ross
On Tue, Jan 5, 2016 at 1:21 AM, Ross Green <rgkernel@gmail.com> wrote:
> Well with the release of 4.4-rc8 I have built and
> started testing the kernel.
>
> With some luck I managed to get a rcu_preempt stall within a few hours
> of testing.
>
> Upon booting, I ran a small series of bench marks to make sure
> everything is running as expected. limited regression testing and then
> just left the system to idle away, with periodic monitoring from the
> network.
>
>
> please find attached two stack traces from linux-4.4-rc8 and also linux-4.4-rc7.
>
> The interesting thing with the rc7 trace is that there are multiple
> stalls that have occurred over a 6 day period.
>
> I realise Paul you have a number of changes pending for the the RCU
> code. It would be good to try and establish what is happening with
> these stalls before the impact of those changes given that timings and
> dynamics might change whats happening in the current environment.
>
> As reported earlier, i have never been able to induce these stalls
> with heavy loading of the system. The only method I can be sure of is
> to leave the system quiet and let time go by till a stall occurs.
>
> Regards,
>
> Ross
>
> On Mon, Jan 4, 2016 at 9:00 AM, Ross Green <rgkernel@gmail.com> wrote:
>> Thanks Paul for your analysis and investigation,
>>
>> I understand your patches are designed not to "fix" the problem, but
>> more to move the problem ahead.
>>
>> I will include a few more stack traces from various kernels. I can go
>> back to around 3.17, with similar trace results.
>>
>> My observation is that the problem can occur at various times and with
>> out any "bad" effect other than more stalls could happen afterwards.
>>
>> At first I wondered if they might actually be a false positive as the
>> kernel seemed to carry on and run quite happily. It is rare that I
>> find a kernel just locks up after observing such a stall, or a
>> complete kernel splat!
>>
>> Unfortunately with my testing I have never been able to induce the
>> problem under any heavy load that would immediately trigger the
>> problem. Indeed most heavy cpu utilisation seemed to just sail on
>> quite nicely.
>>
>> The time for a fault, seems so far, to be non-deterministic with
>> quiescent systems taking anywhere from a few hours through to some six
>> days before showing the problem.
>>
>> More recent kernels seem to have richer stall and back trace
>> information so I was hoping that might shed some light on how they
>> might be occurring..
>>
>> I usually only run a kernel till I get a new -rc release to test with,
>> so a run of 1 week is a typical cycle.
>>
>> I just wish I could find a sure fire method to trigger the problem!!
>>
>> I have included a few more traces of various kernels all showing the problem.
>>
>> Regards,
>>
>> Ross
>>
>> On Mon, Jan 4, 2016 at 5:15 AM, Paul E. McKenney
>> <paulmck@linux.vnet.ibm.com> wrote:
>>> On Sun, Jan 03, 2016 at 07:27:17PM +1100, Ross Green wrote:
>>>> I would not describe the load on this test machine as high or real time.
>>>>
>>>> Apart from a number of standard daemons not much more is running at all!
>>>>
>>>> I normally build a release kernel as soon as possible and set it running.
>>>> Typically I run a series of benchmarks to confirm most things appear
>>>> to be working and then just leave it running. During a normal day i
>>>> will check on the machine 4/5 times just to see how its going!
>>>> Typically I will logon remotely via wifi network connection.
>>>>
>>>> just for your information i will include a few other stack traces from
>>>> previous kernels that may show some trend!
>>>>
>>>>
>>>> Please see the attachments.
>>>
>>> Thank you for the additional details. This does look similar to some
>>> problems I am seeing, though only in heavy rcutorture workloads with
>>> CPU hotplugging.
>>>
>>> I have some crude workarounds in progress, see for example
>>> 2da26818e515 (rcu: Awaken grace-period kthread when stalled) at
>>> git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git.
>>> This workaround kicks the RCU grace-period kthread on every stall warning.
>>> In my testing, this workaround results in slow but real forward progress.
>>>
>>> I have a better workaround in progress, however, please note:
>>>
>>> 1. I have no intention of sending these workarounds upstream.
>>>
>>> 2. The workarounds will splat when they take effect. In other words,
>>> the idea is not to paper over the problem, but instead to allow
>>> me to separate testing concerns.
>>>
>>> 3. A fix is needed for the underlying bug, wherever it might be.
>>>
>>> Thanx, Paul
>>>
>>>> Regards,
>>>>
>>>> Ross
>>>>
>>>>
>>>>
>>>> On Sun, Jan 3, 2016 at 5:17 PM, Paul E. McKenney
>>>> <paulmck@linux.vnet.ibm.com> wrote:
>>>> > On Sun, Jan 03, 2016 at 04:29:11PM +1100, Ross Green wrote:
>>>> >> Still seeing these rcu_preempt stalls on kernels through to 4.4-rc7
>>>> >>
>>>> >> Still have not found a sure fire method to evoke this stall, but have
>>>> >> found that it will normally occur within a week of running a kernel -
>>>> >> usually when it is quiet with light load.
>>>> >>
>>>> >> Have seen similar self detected stalls all the way back to 3.17.
>>>> >> Most recent kernels have included 4.4-rc5 4.4-rc6 and 4.4-rc7
>>>> >>
>>>> >> Regards,
>>>> >>
>>>> >> Ross
>>>> >>
>>>> >> On Fri, Dec 11, 2015 at 10:17 PM, Ross Green <rgkernel@gmail.com> wrote:
>>>> >> > I have been getting these stalls in kernels going back to 3.17.
>>>> >> >
>>>> >> > This stall occurs usually under light load but often requires several
>>>> >> > days to show itself. I have not found any simple way to trigger the
>>>> >> > stall. Indeed heavy workloads seems not to show the fault.
>>>> >> >
>>>> >> > Anyone have any thoughts here?
>>>> >> >
>>>> >> > The recent patch by peterz with kernel/sched/wait.c I thought might
>>>> >> > help the situation, but alas after a few days of running 4.4-rc4 the
>>>> >> > following turned up.
>>>> >> >
>>>> >> > [179922.003570] INFO: rcu_preempt self-detected stall on CPU
>>>> >> > [179922.008178] INFO: rcu_preempt detected stalls on CPUs/tasks:
>>>> >> > [179922.008178] 0-...: (1 ticks this GP) idle=a91/1/0
>>>> >
>>>> > CPU 0 is non-idle from an RCU perspective.
>>>> >
>>>> >> > softirq=1296733/1296733 fqs=0
>>>> >> > [179922.008178]
>>>> >> > [179922.008209] (detected by 1, t=8775 jiffies, g=576439, c=576438, q=102)
>>>> >> > [179922.008209] Task dump for CPU 0:
>>>> >> > [179922.008209] swapper/0 R [179922.008209] running [179922.008209] 0 0 0 0x00000000
>>>> >> > [179922.008209] Backtrace:
>>>> >> >
>>>> >> > [179922.008239] Backtrace aborted due to bad frame pointer <c0907f54>
>>>> >
>>>> > Can't have everything, I guess...
>>>> >
>>>> >> > [179922.008239] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x0 s3 ->state=0x1
>>>> >
>>>> > Something is keeping the rcu_preempt grace-period kthread from
>>>> > running. This far into the grace period, it should have a
>>>> > timer event waking it every few jiffies. It is currently
>>>> > in TASK_INTERRUPTIBLE state.
>>>> >
>>>> >> > [179922.060302] 0-...: (1 ticks this GP) idle=a91/1/0 softirq=1296733/1296733 fqs=0
>>>> >> > [179922.068023] (t=8775 jiffies g=576439 c=576438 q=102)
>>>> >> > [179922.073913] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x2 s3 ->state=0x100
>>>> >
>>>> > Same story, same grace period, pretty much same time. Now there is an FQS
>>>> > request (f0x2) and the state is now TASK_WAKING (->state=0x100 == 256).
>>>> >
>>>> >> > [179922.083587] Task dump for CPU 0:
>>>> >> > [179922.087097] swapper/0 R running 0 0 0 0x00000000
>>>> >> > [179922.093292] Backtrace:
>>>> >> > [179922.096313] [<c0013ea8>] (dump_backtrace) from [<c00140a4>] (show_stack+0x18/0x1c)
>>>> >> > [179922.104675] r7:c0908514 r6:80080193 r5:00000000 r4:c090aca8
>>>> >> > [179922.110809] [<c001408c>] (show_stack) from [<c005a858>] (sched_show_task+0xbc/0x110)
>>>> >> > [179922.119049] [<c005a79c>] (sched_show_task) from [<c005ccd4>] (dump_cpu_task+0x40/0x44)
>>>> >> > [179922.127624] r5:c0917680 r4:00000000
>>>> >> > [179922.131042] [<c005cc94>] (dump_cpu_task) from [<c0082268>] (rcu_dump_cpu_stacks+0x9c/0xdc)
>>>> >> > [179922.140350] r5:c0917680 r4:00000001
>>>> >> > [179922.143157] [<c00821cc>] (rcu_dump_cpu_stacks) from [<c008637c>] (rcu_check_callbacks+0x504/0x8e4)
>>>> >> > [179922.153808] r9:c0908514 r8:c0917680 r7:00000066 r6:2eeab000
>>>> >> > r5:c0904300 r4:ef7af300
>>>> >> > [179922.161499] [<c0085e78>] (rcu_check_callbacks) from [<c00895d0>] (update_process_times+0x40/0x6c)
>>>> >> > [179922.170898] r10:c009a584 r9:00000001 r8:ef7abc4c r7:0000a3a3
>>>> >> > r6:4ec3391c r5:00000000
>>>> >> > [179922.179901] r4:c090aca8
>>>> >> > [179922.182708] [<c0089590>] (update_process_times) from [<c009a580>]
>>>> >> > (tick_sched_handle+0x50/0x54)
>>>> >> > [179922.192108] r5:c0907f10 r4:ef7abe40
>>>> >> > [179922.195983] [<c009a530>] (tick_sched_handle) from [<c009a5d4>]
>>>> >> > (tick_sched_timer+0x50/0x94)
>>>> >> > [179922.204895] [<c009a584>] (tick_sched_timer) from [<c0089fe4>]
>>>> >> > (__hrtimer_run_queues+0x110/0x1a0)
>>>> >> > [179922.214324] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
>>>> >> > [179922.220428] [<c0089ed4>] (__hrtimer_run_queues) from [<c008a674>]
>>>> >> > (hrtimer_interrupt+0xac/0x1f8)
>>>> >> > [179922.227111] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8
>>>> >> > r6:ffffffff r5:00000003
>>>> >> > [179922.238220] r4:ef7abc00
>>>> >> > [179922.238220] [<c008a5c8>] (hrtimer_interrupt) from [<c00170ec>]
>>>> >> > (twd_handler+0x38/0x48)
>>>> >> > [179922.238220] r10:c09084e8 r9:fa241100 r8:00000011 r7:ef028780
>>>> >> > r6:c092574c r5:ef005cc0
>>>> >
>>>> > All interrupt stack up to this point.
>>>> >
>>>> > It is quite possible that the stuff below here is at fault as well.
>>>> > That said, the grace-period should actually get to execute at some
>>>> > point. Do you have a heavy real-time load that might be starving
>>>> > things?
>>>> >
>>>> > Thanx, Paul
>>>> >
>>>> >> > [179922.257110] r4:00000001
>>>> >> > [179922.257110] [<c00170b4>] (twd_handler) from [<c007c8f8>] (handle_percpu_devid_irq+0x74/0x8c)
>>>> >> > [179922.269683] r5:ef005cc0 r4:ef7b1740
>>>> >> > [179922.269683] [<c007c884>] (handle_percpu_devid_irq) from [<c0078454>] (generic_handle_irq+0x2c/0x3c)
>>>> >> > [179922.283233] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000
>>>> >> > r5:00000000 r4:c09013e8
>>>> >> > [179922.290985] [<c0078428>] (generic_handle_irq) from [<c007872c>] (__handle_domain_irq+0x64/0xbc)
>>>> >> > [179922.300842] [<c00786c8>] (__handle_domain_irq) from [<c00094c0>]
>>>> >> > (gic_handle_irq+0x50/0x90)
>>>> >> > [179922.303222] r9:fa241100 r8:fa240100 r7:c0907f10 r6:fa24010c
>>>> >> > r5:c09087a8 r4:c0925748
>>>> >> > [179922.315216] [<c0009470>] (gic_handle_irq) from [<c0014bd4>]
>>>> >> > (__irq_svc+0x54/0x90)
>>>> >> > [179922.319000] Exception stack(0xc0907f10 to 0xc0907f58)
>>>> >> > [179922.331542] 7f00: 00000000
>>>> >> > ef7ab390 fe600000 00000000
>>>> >> > [179922.331542] 7f20: c0906000 c090849c c0900364 c06a8124 c0907f80
>>>> >> > c0944563 c09084e8 c0907f6c
>>>> >> > [179922.349029] 7f40: c0907f4c c0907f60 c00287ac c0010ba8 60080113 ffffffff
>>>> >> > [179922.349029] r9:c0944563 r8:c0907f80 r7:c0907f44 r6:ffffffff
>>>> >> > r5:60080113 r4:c0010ba8
>>>> >> > [179922.357116] [<c0010b80>] (arch_cpu_idle) from [<c006f034>]
>>>> >> > (default_idle_call+0x28/0x34)
>>>> >> > [179922.368926] [<c006f00c>] (default_idle_call) from [<c006f154>]
>>>> >> > (cpu_startup_entry+0x114/0x18c)
>>>> >> > [179922.368926] [<c006f040>] (cpu_startup_entry) from [<c069fc6c>]
>>>> >> > (rest_init+0x90/0x94)
>>>> >> > [179922.385284] r7:ffffffff r4:00000002
>>>> >> > [179922.393463] [<c069fbdc>] (rest_init) from [<c08bbcec>]
>>>> >> > (start_kernel+0x370/0x37c)
>>>> >> > [179922.400421] r5:c0947000 r4:00000000
>>>> >> > [179922.400421] [<c08bb97c>] (start_kernel) from [<8000807c>] (0x8000807c)
>>>> >> > $
>>>> >
>>>> >
>>>
>>>
>>>
[-- Attachment #2: dmesg-3.17.1 --]
[-- Type: application/octet-stream, Size: 26680 bytes --]
$ dmesg
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 3.17.1-panda (rfg@red.dwarf) (gcc version 4.9.1 (Buildroot 2014.11-git-00185-g643baa1) ) #1 SMP PREEMPT Thu Oct 16 16:13:03 AEDT 2014
[ 0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] Machine model: TI OMAP4 PandaBoard
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] On node 0 totalpages: 261888
[ 0.000000] free_area_init_node: node 0, pgdat c081d300, node_mem_map eeff9000
[ 0.000000] Normal zone: 1520 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 194560 pages, LIFO batch:31
[ 0.000000] HighMem zone: 526 pages used for memmap
[ 0.000000] HighMem zone: 67328 pages, LIFO batch:15
[ 0.000000] OMAP4460 ES1.1
[ 0.000000] L2C: platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C: DT/platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C-310 enabling early BRESP for Cortex-A9
[ 0.000000] L2C-310 ID prefetch enabled, offset 1 lines
[ 0.000000] OMAP L2C310: ROM does not support power control setting
[ 0.000000] L2C-310 dynamic clock gating disabled, standby mode disabled
[ 0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
[ 0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x7e470000
[ 0.000000] PERCPU: Embedded 7 pages/cpu @eefb4000 s7488 r8192 d12992 u32768
[ 0.000000] pcpu-alloc: s7488 r8192 d12992 u32768 alloc=8*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260368
[ 0.000000] Kernel command line: root=/dev/mmcblk0p2 rootwait console=ttyO2,115200
[ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Memory: 1029648K/1047552K available (5903K kernel code, 272K rwdata, 1840K rodata, 255K init, 278K bss, 17904K reserved, 269312K highmem)
[ 0.000000] Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xffc00000 - 0xffe00000 (2048 kB)
vmalloc : 0xf0000000 - 0xff000000 ( 240 MB)
lowmem : 0xc0000000 - 0xef800000 ( 760 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc0798064 (7745 kB)
.init : 0xc0799000 - 0xc07d8d40 ( 256 kB)
.data : 0xc07da000 - 0xc081e240 ( 273 kB)
.bss : 0xc081e240 - 0xc0863d38 ( 279 kB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] NR_IRQS:16 nr_irqs:16 16
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node bandgap_fclk
[ 0.000000] OMAP clockevent source: timer1 at 32768 Hz
[ 0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65536000000000ns
[ 0.000000] OMAP clocksource: 32k_counter at 32768 Hz
[ 0.001525] Console: colour dummy device 80x30
[ 0.001556] Calibrating delay loop... 1397.55 BogoMIPS (lpj=6987776)
[ 0.090240] pid_max: default: 32768 minimum: 301
[ 0.090332] Security Framework initialized
[ 0.090393] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.090393] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.091003] Initializing cgroup subsys devices
[ 0.091003] Initializing cgroup subsys freezer
[ 0.091033] Initializing cgroup subsys blkio
[ 0.091064] CPU: Testing write buffer coherency: ok
[ 0.091308] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.091369] Setting up static identity map for 0x805ca138 - 0x805ca190
[ 0.150726] CPU1: Booted secondary processor
[ 0.190216] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.190307] Brought up 2 CPUs
[ 0.190307] SMP: Total of 2 processors activated.
[ 0.190338] CPU: All CPU(s) started in SVC mode.
[ 0.190826] devtmpfs: initialized
[ 0.194458] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[ 0.196136] omap_hwmod: l3_main_3 using broken dt data from ocp
[ 0.197631] omap_hwmod: l3_main_2 using broken dt data from ocp
[ 0.292724] xor: measuring software checksum speed
[ 0.390197] arm4regs : 1212.800 MB/sec
[ 0.490203] 8regs : 858.800 MB/sec
[ 0.590209] 32regs : 808.800 MB/sec
[ 0.590240] xor: using function: arm4regs (1212.800 MB/sec)
[ 0.590240] pinctrl core: initialized pinctrl subsystem
[ 0.590789] regulator-dummy: no parameters
[ 0.606872] NET: Registered protocol family 16
[ 0.607666] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.618438] OMAP GPIO hardware version 0.1
[ 0.622863] omap-gpmc 50000000.gpmc: GPMC revision 6.0
[ 0.623321] irq: no irq domain found for /ocp/pinmux@4a100040 !
[ 0.623687] irq: no irq domain found for /ocp/pinmux@4a100040 !
[ 0.623992] irq: no irq domain found for /ocp/pinmux@4a100040 !
[ 0.635284] platform 4b501000.aes: Cannot lookup hwmod 'aes'
[ 0.635559] platform 480a5000.des: Cannot lookup hwmod 'des'
[ 0.641265] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[ 0.641296] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.642272] OMAP DMA hardware revision 0.0
[ 0.850799] raid6: int32x1 100 MB/s
[ 1.020599] raid6: int32x2 127 MB/s
[ 1.190460] raid6: int32x4 144 MB/s
[ 1.360534] raid6: int32x8 160 MB/s
[ 1.360565] raid6: using algorithm int32x8 (160 MB/s)
[ 1.360565] raid6: using intx1 recovery algorithm
[ 1.376068] omap-dma-engine 4a056000.dma-controller: OMAP DMA engine driver
[ 1.450256] hsusb1_vbus: 3300 mV
[ 1.450378] reg-fixed-voltage wl12xx_vmmc: could not find pctldev for node /ocp/pinmux@4a100040/pinmux_wl12xx_gpio, deferring probe
[ 1.450408] platform wl12xx_vmmc: Driver reg-fixed-voltage requests probe deferral
[ 1.453369] SCSI subsystem initialized
[ 1.453613] usbcore: registered new interface driver usbfs
[ 1.453704] usbcore: registered new interface driver hub
[ 1.453826] usbcore: registered new device driver usb
[ 1.454559] omap_i2c 48070000.i2c: could not find pctldev for node /ocp/pinmux@4a100040/pinmux_i2c1_pins, deferring probe
[ 1.454589] platform 48070000.i2c: Driver omap_i2c requests probe deferral
[ 1.454620] omap_i2c 48072000.i2c: could not find pctldev for node /ocp/pinmux@4a100040/pinmux_i2c2_pins, deferring probe
[ 1.454650] platform 48072000.i2c: Driver omap_i2c requests probe deferral
[ 1.454681] omap_i2c 48060000.i2c: could not find pctldev for node /ocp/pinmux@4a100040/pinmux_i2c3_pins, deferring probe
[ 1.454681] platform 48060000.i2c: Driver omap_i2c requests probe deferral
[ 1.454711] omap_i2c 48350000.i2c: could not find pctldev for node /ocp/pinmux@4a100040/pinmux_i2c4_pins, deferring probe
[ 1.454742] platform 48350000.i2c: Driver omap_i2c requests probe deferral
[ 1.454956] Linux video capture interface: v2.00
[ 1.455810] omap-iommu 4a066000.mmu: 4a066000.mmu registered
[ 1.455902] omap-iommu 55082000.mmu: 55082000.mmu registered
[ 1.456146] Advanced Linux Sound Architecture Driver Initialized.
[ 1.457214] Switched to clocksource 32k_counter
[ 1.470184] NET: Registered protocol family 2
[ 1.470916] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[ 1.471008] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[ 1.471160] TCP: Hash tables configured (established 8192 bind 8192)
[ 1.471221] TCP: reno registered
[ 1.471252] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 1.471282] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 1.471496] NET: Registered protocol family 1
[ 1.471801] RPC: Registered named UNIX socket transport module.
[ 1.471801] RPC: Registered udp transport module.
[ 1.471832] RPC: Registered tcp transport module.
[ 1.471832] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 1.473663] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[ 1.476379] futex hash table entries: 512 (order: 3, 32768 bytes)
[ 1.485931] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 1.486541] ntfs: driver 2.1.30 [Flags: R/O].
[ 1.486846] fuse init (API version 7.23)
[ 1.487396] msgmni has been set to 1485
[ 1.489349] bounce: pool size: 64 pages
[ 1.489379] io scheduler noop registered
[ 1.489379] io scheduler deadline registered
[ 1.489562] io scheduler cfq registered (default)
[ 1.490417] pinctrl-single 4a100040.pinmux: 203 pins at pa fc100040 size 406
[ 1.490600] pinctrl-single 4a31e040.pinmux: 28 pins at pa fc31e040 size 56
[ 1.492156] OMAP DSS rev 4.0
[ 1.494049] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 1.495544] omap_uart 4806a000.serial: no wakeirq for uart0
[ 1.495758] 4806a000.serial: ttyO0 at MMIO 0x4806a000 (irq = 104, base_baud = 3000000) is a OMAP UART0
[ 1.496276] 4806c000.serial: ttyO1 at MMIO 0x4806c000 (irq = 105, base_baud = 3000000) is a OMAP UART1
[ 1.496734] 48020000.serial: ttyO2 at MMIO 0x48020000 (irq = 106, base_baud = 3000000) is a OMAP UART2
[ 2.351318] console [ttyO2] enabled
[ 2.355468] 4806e000.serial: ttyO3 at MMIO 0x4806e000 (irq = 102, base_baud = 3000000) is a OMAP UART3
[ 2.366058] [drm] Initialized drm 1.1.0 20060810
[ 2.372985] loop: module loaded
[ 2.378387] usbcore: registered new interface driver smsc95xx
[ 2.384429] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 2.391357] ehci-omap: OMAP-EHCI Host Controller driver
[ 2.407287] ehci-omap 4a064c00.ehci: EHCI Host Controller
[ 2.412933] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1
[ 2.420989] ehci-omap 4a064c00.ehci: irq 109, io mem 0x4a064c00
[ 2.437225] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00
[ 2.443603] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 2.450714] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 2.458282] usb usb1: Product: EHCI Host Controller
[ 2.463378] usb usb1: Manufacturer: Linux 3.17.1-panda ehci_hcd
[ 2.469573] usb usb1: SerialNumber: 4a064c00.ehci
[ 2.475097] hub 1-0:1.0: USB hub found
[ 2.479064] hub 1-0:1.0: 3 ports detected
[ 2.483947] usbcore: registered new interface driver cdc_wdm
[ 2.490020] usbcore: registered new interface driver usb-storage
[ 2.496673] i2c /dev entries driver
[ 2.500457] usbcore: registered new interface driver uvcvideo
[ 2.506469] USB Video Class driver (1.1.1)
[ 2.511901] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[ 2.519409] device-mapper: uevent: version 1.0.3
[ 2.524566] device-mapper: ioctl: 4.27.0-ioctl (2013-10-30) initialised: dm-devel@redhat.com
[ 2.534118] omap_hsmmc 4809c000.mmc: unable to get vmmc regulator -517
[ 2.540985] platform 4809c000.mmc: Driver omap_hsmmc requests probe deferral
[ 2.548706] omap_hsmmc 480d5000.mmc: unable to get vmmc regulator -517
[ 2.555572] platform 480d5000.mmc: Driver omap_hsmmc requests probe deferral
[ 2.563934] hidraw: raw HID events driver (C) Jiri Kosina
[ 2.569854] usbcore: registered new interface driver usbhid
[ 2.575653] usbhid: USB HID core driver
[ 2.580230] emif 4c000000.emif: emif_probe: device configured with addr = f00b8000 and IRQ142
[ 2.589385] emif 4d000000.emif: emif_probe: device configured with addr = f00ba000 and IRQ143
[ 2.600402] omap-abe-twl6040 sound: ASoC: CODEC twl6040-codec not registered
[ 2.607818] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 2.614837] platform sound: Driver omap-abe-twl6040 requests probe deferral
[ 2.622497] TCP: cubic registered
[ 2.625976] Initializing XFRM netlink socket
[ 2.630462] NET: Registered protocol family 17
[ 2.635345] twl: not initialized
[ 2.638763] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 2.647094] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 2.655487] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 2.663879] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 2.672271] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 2.680664] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 2.689056] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1410000 Vs max 1316660
[ 2.697448] omap2_set_init_voltage: unable to find boot up OPP for vdd_mpu
[ 2.704620] omap2_set_init_voltage: unable to set vdd_mpu
[ 2.710296] omap2_set_init_voltage: unable to find boot up OPP for vdd_core
[ 2.717590] omap2_set_init_voltage: unable to set vdd_core
[ 2.723327] omap2_set_init_voltage: unable to find boot up OPP for vdd_iva
[ 2.730499] omap2_set_init_voltage: unable to set vdd_iva
[ 2.736999] Power Management for TI OMAP4+ devices.
[ 2.742156] Power Management for TI OMAP4.
[ 2.746429] OMAP4 PM: u-boot >= v2012.07 is required for full PM support
[ 2.753570] ThumbEE CPU extension supported.
[ 2.758087] Registering SWP/SWPB emulation handler
[ 2.758087] SmartReflex Class3 initialized
[ 2.770080] Btrfs loaded
[ 2.770202] dmm 4e000000.dmm: initialized all PAT entries
[ 2.781524] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 2.788879] vwl1271: 1800 mV
[ 2.793884] Skipping twl internal clock init and using bootloader value (unknown osc rate)
[ 2.802734] usb 1-1: new high-speed USB device number 2 using ehci-omap
[ 2.810058] twl 0-0048: PIH (irq 39) nested IRQs
[ 2.816314] twl_rtc 48070000.i2c:twl@48:rtc: Enabling TWL-RTC
[ 2.825164] twl_rtc 48070000.i2c:twl@48:rtc: rtc core: registered 48070000.i2c:twl@48 as rtc0
[ 2.834899] VAUX1_6030: 1000 <--> 3000 mV at 2800 mV
[ 2.841064] VAUX2_6030: 1200 <--> 2800 mV at 1800 mV
[ 2.847198] VAUX3_6030: 1000 <--> 3000 mV at 1200 mV
[ 2.853363] VMMC: 1200 <--> 3000 mV at 3000 mV
[ 2.858978] VPP: 1800 <--> 2500 mV at 1900 mV
[ 2.864532] VUSIM: 1200 <--> 2900 mV at 1800 mV
[ 2.870025] VDAC: 1800 mV
[ 2.873260] VANA: 2100 mV
[ 2.876983] VCXIO: 1800 mV
[ 2.880706] VUSB: 3300 mV
[ 2.884307] V1V8: 1800 mV
[ 2.888031] V2V1: 2100 mV
[ 2.892608] twl6040 0-004b: clk32k is not handled
[ 2.900115] omap_i2c 48070000.i2c: bus 0 rev0.11 at 400 kHz
[ 2.906677] omap_i2c 48072000.i2c: bus 1 rev0.11 at 400 kHz
[ 2.913970] omap_i2c 48060000.i2c: bus 2 rev0.11 at 100 kHz
[ 2.920715] omap_i2c 48350000.i2c: bus 3 rev0.11 at 400 kHz
[ 2.982635] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
[ 2.990142] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 2.998565] hub 1-1:1.0: USB hub found
[ 3.009216] hub 1-1:1.0: 5 ports detected
[ 3.129821] omap-abe-twl6040 sound: twl6040-legacy <-> 40132000.mcpdm mapping ok
[ 3.140350] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 3.148437] twl_rtc 48070000.i2c:twl@48:rtc: setting system clock to 2014-10-16 05:44:02 UTC (1413438242)
[ 3.154907] mmc0: host does not support reading read-only switch. assuming write-enable.
[ 3.157562] mmc0: new high speed SDHC card at address aaaa
[ 3.158111] mmcblk0: mmc0:aaaa SE32G 29.7 GiB
[ 3.172332] mmcblk0: p1 p2 p3 p4
[ 3.173522] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 3.178283] omap_hsmmc 480d5000.mmc: card claims to support voltages below defined range
[ 3.188201] mmc1: queuing unknown CIS tuple 0x91 (3 bytes)
[ 3.189178] mmc1: new SDIO card at address 0001
[ 3.206848] sr_init: No PMIC hook to init smartreflex
[ 3.212219] smartreflex smartreflex.0: omap_sr_probe: SmartReflex driver initialized
[ 3.220672] smartreflex smartreflex.1: omap_sr_probe: SmartReflex driver initialized
[ 3.220672] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 3.228820] smartreflex smartreflex.2: omap_sr_probe: SmartReflex driver initialized
[ 3.235839] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 3.251068] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 3.258972] VUSB: disabling
[ 3.262390] VANA: disabling
[ 3.265808] VDAC: disabling
[ 3.270874] ALSA device list:
[ 3.270935] #0: PandaBoardES
[ 3.278381] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
[ 3.287353] EXT4-fs (mmcblk0p2): mounting ext2 file system using the ext4 subsystem
[ 3.298126] EXT4-fs (mmcblk0p2): mounted filesystem without journal. Opts: (null)
[ 3.305999] VFS: Mounted root (ext2 filesystem) readonly on device 179:2.
[ 3.311004] usb 1-1.1: new high-speed USB device number 3 using ehci-omap
[ 3.322143] devtmpfs: mounted
[ 3.325317] Freeing unused kernel memory: 252K (c0799000 - c07d8000)
[ 3.428466] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[ 3.435729] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 3.446319] smsc95xx v1.0.4
[ 3.558715] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-4a064c00.ehci-1.1, smsc95xx USB 2.0 Ethernet, d6:b7:37:f3:59:2c
[ 3.571746] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 3.594085] EXT4-fs (mmcblk0p2): warning: mounting unchecked fs, running e2fsck is recommended
[ 3.606048] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 3.687591] random: dd urandom read with 37 bits of entropy available
[ 8.639373] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[ 18.613311] Bluetooth: Core ver 2.19
[ 18.617553] NET: Registered protocol family 31
[ 18.617889] Bluetooth: HCI device and connection manager initialized
[ 18.629180] Bluetooth: HCI socket layer initialized
[ 18.630920] Bluetooth: L2CAP socket layer initialized
[ 18.634368] Bluetooth: SCO socket layer initialized
[ 24.340332] random: nonblocking pool is initialized
[ 24.360046] svc: failed to register nfsdv2 RPC service (errno 111).
[ 54.992584] cfg80211: Calling CRDA to update world regulatory domain
[ 55.057495] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 55.068511] platform omapdrm.0: Driver omapdrm requests probe deferral
[ 55.342956] wlcore: loaded
[ 103.648193] wlcore: firmware booted (Rev 6.3.10.0.133)
[ 106.767578] wlan0: authenticate with 90:f6:52:81:ee:70
[ 106.782501] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[ 106.790893] wlan0: authenticated
[ 106.797241] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[ 106.810943] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=1)
[ 106.829010] wlan0: associated
[ 106.873229] wlcore: Association completed.
[94352.175659] INFO: rcu_preempt self-detected stall on CPU { 0} (t=4944 jiffies g=325932 c=325931 q=127)
[94352.183654] Task dump for CPU 0:
[94352.183654] swapper/0 R
[94352.188903] INFO: rcu_preempt self-detected stall on CPU
[94352.188903] {
[94352.189605] 1
[94352.189605] }
[94352.189605] (t=4944 jiffies g=325932 c=325931 q=127)
[94352.189605] running
[94352.201629] 0 0 0 0x00000000
[94352.206909] Backtrace:
[94352.206909] [<c0012590>] (dump_backtrace) from [<c001286c>] (show_stack+0x18/0x1c)
[94352.206909] r6:80070193 r5:00000000 r4:c07e4960 r3:00000000
[94352.223327] [<c0012854>] (show_stack) from [<c005ce50>] (sched_show_task+0xb4/0x108)
[94352.223327] [<c005cd9c>] (sched_show_task) from [<c005fc60>] (dump_cpu_task+0x40/0x44)
[94352.239685] r5:c07f6380 r4:00000000
[94352.239685] [<c005fc20>] (dump_cpu_task) from [<c007d56c>] (rcu_dump_cpu_stacks+0x98/0xc4)
[94352.239685] r4:00000000 r3:00000003
[94352.252075] [<c007d4d4>] (rcu_dump_cpu_stacks) from [<c0080ad0>] (rcu_check_callbacks+0x488/0x80c)
[94352.265167] r8:0000007f r7:c07e2794 r6:c07f6380 r5:c07d7524 r4:eefb4524 r3:00010001
[94352.265167] [<c0080648>] (rcu_check_callbacks) from [<c0083928>] (update_process_times+0x48/0x68)
[94352.265167] r10:00000001 r9:eefb45d0 r8:eefb45d0 r7:00000000 r6:00000000 r5:c07e4960
[94352.290740] r4:c07da000
[94352.290740] [<c00838e0>] (update_process_times) from [<c009394c>] (tick_sched_handle.isra.17+0x4c/0x50)
[94352.290740] r7:000055cf r6:fc731722 r5:c07dbf20 r4:eefb4858
[94352.309143] [<c0093900>] (tick_sched_handle.isra.17) from [<c00939a4>] (tick_sched_timer+0x54/0x80)
[94352.309143] [<c0093950>] (tick_sched_timer) from [<c00841ac>] (__run_hrtimer.isra.34+0x4c/0xdc)
[94352.309143] r7:c0093950 r6:eefb45d0 r5:eefb4608 r4:eefb4858
[94352.333618] [<c0084160>] (__run_hrtimer.isra.34) from [<c008497c>] (hrtimer_interrupt+0x124/0x2d4)
[94352.333618] r7:000055cf r6:fc731722 r5:00000000 r4:eefb45d0
[94352.333618] [<c0084858>] (hrtimer_interrupt) from [<c001547c>] (twd_handler+0x38/0x48)
[94352.333618] r10:c07cda3c r9:c081e099 r8:0000001d r7:ee828100 r6:ee806600 r5:c07ffc04
[94352.365356] r4:00000001
[94352.365356] [<c0015444>] (twd_handler) from [<c007961c>] (handle_percpu_devid_irq+0x74/0x8c)
[94352.376831] r4:eefb80c0 r3:c0015444
[94352.376831] [<c00795a8>] (handle_percpu_devid_irq) from [<c0075818>] (generic_handle_irq+0x34/0x44)
[94352.380584] r8:c081e099 r7:fa240100 r6:00000000 r5:0000001d r4:0000001d r3:c00795a8
[94352.380584] [<c00757e4>] (generic_handle_irq) from [<c000f964>] (handle_IRQ+0x48/0x98)
[94352.380584] r4:c07d7ca0 r3:000001ab
[94352.380584] [<c000f91c>] (handle_IRQ) from [<c00085a0>] (gic_handle_irq+0x34/0x64)
[94352.380584] r6:c07dbf20 r5:c07e290c r4:fa24010c r3:000000a0
[94352.423980] [<c000856c>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[94352.423980] Exception stack(0xc07dbf20 to 0xc07dbf68)
[94352.423980] bf20: eefb43c8 00000000 05d70646 c002cb98 c07da000 c07e2488 c07e24d4 c05cd274
[94352.423980] bf40: c081e099 c081e099 c07cda3c c07dbf74 c07dbf68 c07dbf68 c002c7a8 c000fcac
[94352.423980] bf60: a0070113 ffffffff
[94352.457794] r7:c07dbf54 r6:ffffffff r5:a0070113 r4:c000fcac
[94352.457794] [<c000fc84>] (arch_cpu_idle) from [<c006d3b4>] (cpu_startup_entry+0x100/0x154)
[94352.457794] [<c006d2b4>] (cpu_startup_entry) from [<c05c1a68>] (rest_init+0x8c/0x90)
[94352.472381] r7:ffffffff r3:c05cd65c
[94352.472381] [<c05c19dc>] (rest_init) from [<c0799c20>] (start_kernel+0x354/0x360)
[94352.472381] r4:c07e2598 r3:00000001
[94352.472381] [<c07998cc>] (start_kernel) from [<80008074>] (0x80008074)
[94352.472381] Task dump for CPU 1:
[94352.505950] swapper/1 R running 0 0 1 0x00000000
[94352.505950] Backtrace:
[94352.505950] [<ee89ffb0>] (0xee89ffb0) from [<00000000>] ( (null))
[94352.521606] Backtrace aborted due to bad frame pointer <c081e099>
[94352.521606] Task dump for CPU 0:
[94352.521606] swapper/0 R running 0 0 0 0x00000000
[94352.521606] Backtrace:
[94352.521606] [<c07dbf68>] (0xc07dbf68) from [<c07cda3c>] (0xc07cda3c)
[94352.547180] Backtrace aborted due to bad frame pointer <c081e099>
[94352.547180] Task dump for CPU 1:
[94352.547180] swapper/1 R running 0 0 1 0x00000000
[94352.563537] Backtrace:
[94352.563537] [<c0012590>] (dump_backtrace) from [<c001286c>] (show_stack+0x18/0x1c)
[94352.574005] r6:80000193 r5:00000001 r4:ee876f80 r3:00000000
[94352.574005] [<c0012854>] (show_stack) from [<c005ce50>] (sched_show_task+0xb4/0x108)
[94352.574005] [<c005cd9c>] (sched_show_task) from [<c005fc60>] (dump_cpu_task+0x40/0x44)
[94352.574005] r5:c07f6380 r4:00000001
[94352.574005] [<c005fc20>] (dump_cpu_task) from [<c007d56c>] (rcu_dump_cpu_stacks+0x98/0xc4)
[94352.608673] r4:00000001 r3:00000001
[94352.608673] [<c007d4d4>] (rcu_dump_cpu_stacks) from [<c0080ad0>] (rcu_check_callbacks+0x488/0x80c)
[94352.608673] r8:0000007f r7:c07e2794 r6:c07f6380 r5:c07d7524 r4:eefbc524 r3:00010001
[94352.608673] [<c0080648>] (rcu_check_callbacks) from [<c0083928>] (update_process_times+0x48/0x68)
[94352.639160] r10:00000001 r9:eefbc5d0 r8:eefbc5d0 r7:00000001 r6:00000000 r5:ee876f80
[94352.639160] r4:ee89e000
[94352.639160] [<c00838e0>] (update_process_times) from [<c009394c>] (tick_sched_handle.isra.17+0x4c/0x50)
[94352.659820] r7:000055cf r6:fd3da6c6 r5:ee89ff68 r4:eefbc858
[94352.659820] [<c0093900>] (tick_sched_handle.isra.17) from [<c00939a4>] (tick_sched_timer+0x54/0x80)
[94352.675201] [<c0093950>] (tick_sched_timer) from [<c00841ac>] (__run_hrtimer.isra.34+0x4c/0xdc)
[94352.675201] r7:c0093950 r6:eefbc5d0 r5:eefbc608 r4:eefbc858
[94352.690216] [<c0084160>] (__run_hrtimer.isra.34) from [<c008497c>] (hrtimer_interrupt+0x124/0x2d4)
[94352.690216] r7:000055cf r6:fd3da6c6 r5:00000000 r4:eefbc5d0
[94352.705505] [<c0084858>] (hrtimer_interrupt) from [<c001547c>] (twd_handler+0x38/0x48)
[94352.705505] r10:00000000 r9:c081e099 r8:0000001d r7:ee828100 r6:ee806600 r5:c07ffc04
[94352.721954] r4:00000001
[94352.724609] [<c0015444>] (twd_handler) from [<c007961c>] (handle_percpu_devid_irq+0x74/0x8c)
[94352.733428] r4:eefc00c0 r3:c0015444
[94352.733428] [<c00795a8>] (handle_percpu_devid_irq) from [<c0075818>] (generic_handle_irq+0x34/0x44)
[94352.746612] r8:c081e099 r7:fa240100 r6:00000000 r5:0000001d r4:0000001d r3:c00795a8
[94352.746612] [<c00757e4>] (generic_handle_irq) from [<c000f964>] (handle_IRQ+0x48/0x98)
[94352.746612] r4:c07d7ca0 r3:000001ab
[94352.763000] [<c000f91c>] (handle_IRQ) from [<c00085a0>] (gic_handle_irq+0x34/0x64)
[94352.763000] r6:ee89ff68 r5:c07e290c r4:fa24010c r3:000000a0
[94352.780578] [<c000856c>] (gic_handle_irq) from [<c0013440>] (__irq_svc+0x40/0x74)
[94352.780578] Exception stack(0xee89ff68 to 0xee89ffb0)
[94352.788391] ff60: eefbc3c8 00000000 0623679c c002cb98 ee89e000 c07e2488
[94352.788391] ff80: c07e24d4 c05cd274 c081e099 c081e099 00000000 ee89ffbc ee89ffb0 ee89ffb0
[94352.788391] ffa0: c002c7a8 c000fcac a0000113 ffffffff
[94352.816009] r7:ee89ff9c r6:ffffffff r5:a0000113 r4:c000fcac
[94352.816009] [<c000fc84>] (arch_cpu_idle) from [<c006d3b4>] (cpu_startup_entry+0x100/0x154)
[94352.816009] [<c006d2b4>] (cpu_startup_entry) from [<c0014568>] (secondary_start_kernel+0x134/0x140)
[94352.840026] r7:c081e580 r3:00000001
[94352.840026] [<c0014434>] (secondary_start_kernel) from [<80008664>] (0x80008664)
[94352.843780] r6:10c0387d r5:00000015 r4:ae88406a r3:c000864c
$
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: rcu_preempt self-detected stall on CPU from 4.4-rc4, since 3.17
2016-01-06 23:10 ` Ross Green
@ 2016-01-08 18:35 ` Paul E. McKenney
2016-01-14 5:39 ` Ross Green
0 siblings, 1 reply; 12+ messages in thread
From: Paul E. McKenney @ 2016-01-08 18:35 UTC (permalink / raw)
To: Ross Green
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
Mathieu Desnoyers, josh, tglx, peterz, rostedt, dhowells,
Eric Dumazet, dvhart, Frédéric Weisbecker, oleg,
pranith kumar
On Thu, Jan 07, 2016 at 10:10:54AM +1100, Ross Green wrote:
> Just for completeness I have attached another rcu_preempt stall
> warning from a kinux-3.17.1 kernel.
>
> So it looks like these stall warning go back that far.
>
> It could be that the kernel instrumentation improved enough to detect
> this situation around the 3.17 kernel. I can't find any earlier
> records of these stall warnings before these kernels. I guess someone
> must have suspected there was a potential for these stalls and hence
> the detection facility.
Thank you for testing this, Ross! For whatever it is worth, the
stall-detection code was added early on, back in the 2.6 days.
On the workaround end of things, I am having limited success by forcing
RCU's grace-period kthread off of outgoing CPUs at down-prepare time.
I really don't like this workaround because I would rather give the
sysadm full control of where these kthreads run, but workarounds are
workarounds...
See 3263d1f49276 (rcu: Migrate grace-period kthread off of outgoing CPU)
for the current version in -rcu.
Thanx, Paul
> Regards,
>
> Ross
>
> On Tue, Jan 5, 2016 at 1:21 AM, Ross Green <rgkernel@gmail.com> wrote:
> > Well with the release of 4.4-rc8 I have built and
> > started testing the kernel.
> >
> > With some luck I managed to get a rcu_preempt stall within a few hours
> > of testing.
> >
> > Upon booting, I ran a small series of bench marks to make sure
> > everything is running as expected. limited regression testing and then
> > just left the system to idle away, with periodic monitoring from the
> > network.
> >
> >
> > please find attached two stack traces from linux-4.4-rc8 and also linux-4.4-rc7.
> >
> > The interesting thing with the rc7 trace is that there are multiple
> > stalls that have occurred over a 6 day period.
> >
> > I realise Paul you have a number of changes pending for the the RCU
> > code. It would be good to try and establish what is happening with
> > these stalls before the impact of those changes given that timings and
> > dynamics might change whats happening in the current environment.
> >
> > As reported earlier, i have never been able to induce these stalls
> > with heavy loading of the system. The only method I can be sure of is
> > to leave the system quiet and let time go by till a stall occurs.
> >
> > Regards,
> >
> > Ross
> >
> > On Mon, Jan 4, 2016 at 9:00 AM, Ross Green <rgkernel@gmail.com> wrote:
> >> Thanks Paul for your analysis and investigation,
> >>
> >> I understand your patches are designed not to "fix" the problem, but
> >> more to move the problem ahead.
> >>
> >> I will include a few more stack traces from various kernels. I can go
> >> back to around 3.17, with similar trace results.
> >>
> >> My observation is that the problem can occur at various times and with
> >> out any "bad" effect other than more stalls could happen afterwards.
> >>
> >> At first I wondered if they might actually be a false positive as the
> >> kernel seemed to carry on and run quite happily. It is rare that I
> >> find a kernel just locks up after observing such a stall, or a
> >> complete kernel splat!
> >>
> >> Unfortunately with my testing I have never been able to induce the
> >> problem under any heavy load that would immediately trigger the
> >> problem. Indeed most heavy cpu utilisation seemed to just sail on
> >> quite nicely.
> >>
> >> The time for a fault, seems so far, to be non-deterministic with
> >> quiescent systems taking anywhere from a few hours through to some six
> >> days before showing the problem.
> >>
> >> More recent kernels seem to have richer stall and back trace
> >> information so I was hoping that might shed some light on how they
> >> might be occurring..
> >>
> >> I usually only run a kernel till I get a new -rc release to test with,
> >> so a run of 1 week is a typical cycle.
> >>
> >> I just wish I could find a sure fire method to trigger the problem!!
> >>
> >> I have included a few more traces of various kernels all showing the problem.
> >>
> >> Regards,
> >>
> >> Ross
> >>
> >> On Mon, Jan 4, 2016 at 5:15 AM, Paul E. McKenney
> >> <paulmck@linux.vnet.ibm.com> wrote:
> >>> On Sun, Jan 03, 2016 at 07:27:17PM +1100, Ross Green wrote:
> >>>> I would not describe the load on this test machine as high or real time.
> >>>>
> >>>> Apart from a number of standard daemons not much more is running at all!
> >>>>
> >>>> I normally build a release kernel as soon as possible and set it running.
> >>>> Typically I run a series of benchmarks to confirm most things appear
> >>>> to be working and then just leave it running. During a normal day i
> >>>> will check on the machine 4/5 times just to see how its going!
> >>>> Typically I will logon remotely via wifi network connection.
> >>>>
> >>>> just for your information i will include a few other stack traces from
> >>>> previous kernels that may show some trend!
> >>>>
> >>>>
> >>>> Please see the attachments.
> >>>
> >>> Thank you for the additional details. This does look similar to some
> >>> problems I am seeing, though only in heavy rcutorture workloads with
> >>> CPU hotplugging.
> >>>
> >>> I have some crude workarounds in progress, see for example
> >>> 2da26818e515 (rcu: Awaken grace-period kthread when stalled) at
> >>> git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git.
> >>> This workaround kicks the RCU grace-period kthread on every stall warning.
> >>> In my testing, this workaround results in slow but real forward progress.
> >>>
> >>> I have a better workaround in progress, however, please note:
> >>>
> >>> 1. I have no intention of sending these workarounds upstream.
> >>>
> >>> 2. The workarounds will splat when they take effect. In other words,
> >>> the idea is not to paper over the problem, but instead to allow
> >>> me to separate testing concerns.
> >>>
> >>> 3. A fix is needed for the underlying bug, wherever it might be.
> >>>
> >>> Thanx, Paul
> >>>
> >>>> Regards,
> >>>>
> >>>> Ross
> >>>>
> >>>>
> >>>>
> >>>> On Sun, Jan 3, 2016 at 5:17 PM, Paul E. McKenney
> >>>> <paulmck@linux.vnet.ibm.com> wrote:
> >>>> > On Sun, Jan 03, 2016 at 04:29:11PM +1100, Ross Green wrote:
> >>>> >> Still seeing these rcu_preempt stalls on kernels through to 4.4-rc7
> >>>> >>
> >>>> >> Still have not found a sure fire method to evoke this stall, but have
> >>>> >> found that it will normally occur within a week of running a kernel -
> >>>> >> usually when it is quiet with light load.
> >>>> >>
> >>>> >> Have seen similar self detected stalls all the way back to 3.17.
> >>>> >> Most recent kernels have included 4.4-rc5 4.4-rc6 and 4.4-rc7
> >>>> >>
> >>>> >> Regards,
> >>>> >>
> >>>> >> Ross
> >>>> >>
> >>>> >> On Fri, Dec 11, 2015 at 10:17 PM, Ross Green <rgkernel@gmail.com> wrote:
> >>>> >> > I have been getting these stalls in kernels going back to 3.17.
> >>>> >> >
> >>>> >> > This stall occurs usually under light load but often requires several
> >>>> >> > days to show itself. I have not found any simple way to trigger the
> >>>> >> > stall. Indeed heavy workloads seems not to show the fault.
> >>>> >> >
> >>>> >> > Anyone have any thoughts here?
> >>>> >> >
> >>>> >> > The recent patch by peterz with kernel/sched/wait.c I thought might
> >>>> >> > help the situation, but alas after a few days of running 4.4-rc4 the
> >>>> >> > following turned up.
> >>>> >> >
> >>>> >> > [179922.003570] INFO: rcu_preempt self-detected stall on CPU
> >>>> >> > [179922.008178] INFO: rcu_preempt detected stalls on CPUs/tasks:
> >>>> >> > [179922.008178] 0-...: (1 ticks this GP) idle=a91/1/0
> >>>> >
> >>>> > CPU 0 is non-idle from an RCU perspective.
> >>>> >
> >>>> >> > softirq=1296733/1296733 fqs=0
> >>>> >> > [179922.008178]
> >>>> >> > [179922.008209] (detected by 1, t=8775 jiffies, g=576439, c=576438, q=102)
> >>>> >> > [179922.008209] Task dump for CPU 0:
> >>>> >> > [179922.008209] swapper/0 R [179922.008209] running [179922.008209] 0 0 0 0x00000000
> >>>> >> > [179922.008209] Backtrace:
> >>>> >> >
> >>>> >> > [179922.008239] Backtrace aborted due to bad frame pointer <c0907f54>
> >>>> >
> >>>> > Can't have everything, I guess...
> >>>> >
> >>>> >> > [179922.008239] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x0 s3 ->state=0x1
> >>>> >
> >>>> > Something is keeping the rcu_preempt grace-period kthread from
> >>>> > running. This far into the grace period, it should have a
> >>>> > timer event waking it every few jiffies. It is currently
> >>>> > in TASK_INTERRUPTIBLE state.
> >>>> >
> >>>> >> > [179922.060302] 0-...: (1 ticks this GP) idle=a91/1/0 softirq=1296733/1296733 fqs=0
> >>>> >> > [179922.068023] (t=8775 jiffies g=576439 c=576438 q=102)
> >>>> >> > [179922.073913] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x2 s3 ->state=0x100
> >>>> >
> >>>> > Same story, same grace period, pretty much same time. Now there is an FQS
> >>>> > request (f0x2) and the state is now TASK_WAKING (->state=0x100 == 256).
> >>>> >
> >>>> >> > [179922.083587] Task dump for CPU 0:
> >>>> >> > [179922.087097] swapper/0 R running 0 0 0 0x00000000
> >>>> >> > [179922.093292] Backtrace:
> >>>> >> > [179922.096313] [<c0013ea8>] (dump_backtrace) from [<c00140a4>] (show_stack+0x18/0x1c)
> >>>> >> > [179922.104675] r7:c0908514 r6:80080193 r5:00000000 r4:c090aca8
> >>>> >> > [179922.110809] [<c001408c>] (show_stack) from [<c005a858>] (sched_show_task+0xbc/0x110)
> >>>> >> > [179922.119049] [<c005a79c>] (sched_show_task) from [<c005ccd4>] (dump_cpu_task+0x40/0x44)
> >>>> >> > [179922.127624] r5:c0917680 r4:00000000
> >>>> >> > [179922.131042] [<c005cc94>] (dump_cpu_task) from [<c0082268>] (rcu_dump_cpu_stacks+0x9c/0xdc)
> >>>> >> > [179922.140350] r5:c0917680 r4:00000001
> >>>> >> > [179922.143157] [<c00821cc>] (rcu_dump_cpu_stacks) from [<c008637c>] (rcu_check_callbacks+0x504/0x8e4)
> >>>> >> > [179922.153808] r9:c0908514 r8:c0917680 r7:00000066 r6:2eeab000
> >>>> >> > r5:c0904300 r4:ef7af300
> >>>> >> > [179922.161499] [<c0085e78>] (rcu_check_callbacks) from [<c00895d0>] (update_process_times+0x40/0x6c)
> >>>> >> > [179922.170898] r10:c009a584 r9:00000001 r8:ef7abc4c r7:0000a3a3
> >>>> >> > r6:4ec3391c r5:00000000
> >>>> >> > [179922.179901] r4:c090aca8
> >>>> >> > [179922.182708] [<c0089590>] (update_process_times) from [<c009a580>]
> >>>> >> > (tick_sched_handle+0x50/0x54)
> >>>> >> > [179922.192108] r5:c0907f10 r4:ef7abe40
> >>>> >> > [179922.195983] [<c009a530>] (tick_sched_handle) from [<c009a5d4>]
> >>>> >> > (tick_sched_timer+0x50/0x94)
> >>>> >> > [179922.204895] [<c009a584>] (tick_sched_timer) from [<c0089fe4>]
> >>>> >> > (__hrtimer_run_queues+0x110/0x1a0)
> >>>> >> > [179922.214324] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
> >>>> >> > [179922.220428] [<c0089ed4>] (__hrtimer_run_queues) from [<c008a674>]
> >>>> >> > (hrtimer_interrupt+0xac/0x1f8)
> >>>> >> > [179922.227111] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8
> >>>> >> > r6:ffffffff r5:00000003
> >>>> >> > [179922.238220] r4:ef7abc00
> >>>> >> > [179922.238220] [<c008a5c8>] (hrtimer_interrupt) from [<c00170ec>]
> >>>> >> > (twd_handler+0x38/0x48)
> >>>> >> > [179922.238220] r10:c09084e8 r9:fa241100 r8:00000011 r7:ef028780
> >>>> >> > r6:c092574c r5:ef005cc0
> >>>> >
> >>>> > All interrupt stack up to this point.
> >>>> >
> >>>> > It is quite possible that the stuff below here is at fault as well.
> >>>> > That said, the grace-period should actually get to execute at some
> >>>> > point. Do you have a heavy real-time load that might be starving
> >>>> > things?
> >>>> >
> >>>> > Thanx, Paul
> >>>> >
> >>>> >> > [179922.257110] r4:00000001
> >>>> >> > [179922.257110] [<c00170b4>] (twd_handler) from [<c007c8f8>] (handle_percpu_devid_irq+0x74/0x8c)
> >>>> >> > [179922.269683] r5:ef005cc0 r4:ef7b1740
> >>>> >> > [179922.269683] [<c007c884>] (handle_percpu_devid_irq) from [<c0078454>] (generic_handle_irq+0x2c/0x3c)
> >>>> >> > [179922.283233] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000
> >>>> >> > r5:00000000 r4:c09013e8
> >>>> >> > [179922.290985] [<c0078428>] (generic_handle_irq) from [<c007872c>] (__handle_domain_irq+0x64/0xbc)
> >>>> >> > [179922.300842] [<c00786c8>] (__handle_domain_irq) from [<c00094c0>]
> >>>> >> > (gic_handle_irq+0x50/0x90)
> >>>> >> > [179922.303222] r9:fa241100 r8:fa240100 r7:c0907f10 r6:fa24010c
> >>>> >> > r5:c09087a8 r4:c0925748
> >>>> >> > [179922.315216] [<c0009470>] (gic_handle_irq) from [<c0014bd4>]
> >>>> >> > (__irq_svc+0x54/0x90)
> >>>> >> > [179922.319000] Exception stack(0xc0907f10 to 0xc0907f58)
> >>>> >> > [179922.331542] 7f00: 00000000
> >>>> >> > ef7ab390 fe600000 00000000
> >>>> >> > [179922.331542] 7f20: c0906000 c090849c c0900364 c06a8124 c0907f80
> >>>> >> > c0944563 c09084e8 c0907f6c
> >>>> >> > [179922.349029] 7f40: c0907f4c c0907f60 c00287ac c0010ba8 60080113 ffffffff
> >>>> >> > [179922.349029] r9:c0944563 r8:c0907f80 r7:c0907f44 r6:ffffffff
> >>>> >> > r5:60080113 r4:c0010ba8
> >>>> >> > [179922.357116] [<c0010b80>] (arch_cpu_idle) from [<c006f034>]
> >>>> >> > (default_idle_call+0x28/0x34)
> >>>> >> > [179922.368926] [<c006f00c>] (default_idle_call) from [<c006f154>]
> >>>> >> > (cpu_startup_entry+0x114/0x18c)
> >>>> >> > [179922.368926] [<c006f040>] (cpu_startup_entry) from [<c069fc6c>]
> >>>> >> > (rest_init+0x90/0x94)
> >>>> >> > [179922.385284] r7:ffffffff r4:00000002
> >>>> >> > [179922.393463] [<c069fbdc>] (rest_init) from [<c08bbcec>]
> >>>> >> > (start_kernel+0x370/0x37c)
> >>>> >> > [179922.400421] r5:c0947000 r4:00000000
> >>>> >> > [179922.400421] [<c08bb97c>] (start_kernel) from [<8000807c>] (0x8000807c)
> >>>> >> > $
> >>>> >
> >>>> >
> >>>
> >>>
> >>>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: rcu_preempt self-detected stall on CPU from 4.4-rc4, since 3.17
2016-01-08 18:35 ` Paul E. McKenney
@ 2016-01-14 5:39 ` Ross Green
2016-01-16 0:39 ` Paul E. McKenney
0 siblings, 1 reply; 12+ messages in thread
From: Ross Green @ 2016-01-14 5:39 UTC (permalink / raw)
To: Paul McKenney
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
Mathieu Desnoyers, josh, tglx, peterz, rostedt, dhowells,
Eric Dumazet, dvhart, Frédéric Weisbecker, oleg,
pranith kumar
[-- Attachment #1: Type: text/plain, Size: 14558 bytes --]
Just for completeness I have tested the kernel 4.4 and have seen stall
warnings similar to the others reported.
I have included the dmesg dump with stack trace for peoples further
understanding.
Regards,
Ross
On Sat, Jan 9, 2016 at 5:35 AM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
> On Thu, Jan 07, 2016 at 10:10:54AM +1100, Ross Green wrote:
>> Just for completeness I have attached another rcu_preempt stall
>> warning from a kinux-3.17.1 kernel.
>>
>> So it looks like these stall warning go back that far.
>>
>> It could be that the kernel instrumentation improved enough to detect
>> this situation around the 3.17 kernel. I can't find any earlier
>> records of these stall warnings before these kernels. I guess someone
>> must have suspected there was a potential for these stalls and hence
>> the detection facility.
>
> Thank you for testing this, Ross! For whatever it is worth, the
> stall-detection code was added early on, back in the 2.6 days.
>
> On the workaround end of things, I am having limited success by forcing
> RCU's grace-period kthread off of outgoing CPUs at down-prepare time.
> I really don't like this workaround because I would rather give the
> sysadm full control of where these kthreads run, but workarounds are
> workarounds...
>
> See 3263d1f49276 (rcu: Migrate grace-period kthread off of outgoing CPU)
> for the current version in -rcu.
>
> Thanx, Paul
>
>> Regards,
>>
>> Ross
>>
>> On Tue, Jan 5, 2016 at 1:21 AM, Ross Green <rgkernel@gmail.com> wrote:
>> > Well with the release of 4.4-rc8 I have built and
>> > started testing the kernel.
>> >
>> > With some luck I managed to get a rcu_preempt stall within a few hours
>> > of testing.
>> >
>> > Upon booting, I ran a small series of bench marks to make sure
>> > everything is running as expected. limited regression testing and then
>> > just left the system to idle away, with periodic monitoring from the
>> > network.
>> >
>> >
>> > please find attached two stack traces from linux-4.4-rc8 and also linux-4.4-rc7.
>> >
>> > The interesting thing with the rc7 trace is that there are multiple
>> > stalls that have occurred over a 6 day period.
>> >
>> > I realise Paul you have a number of changes pending for the the RCU
>> > code. It would be good to try and establish what is happening with
>> > these stalls before the impact of those changes given that timings and
>> > dynamics might change whats happening in the current environment.
>> >
>> > As reported earlier, i have never been able to induce these stalls
>> > with heavy loading of the system. The only method I can be sure of is
>> > to leave the system quiet and let time go by till a stall occurs.
>> >
>> > Regards,
>> >
>> > Ross
>> >
>> > On Mon, Jan 4, 2016 at 9:00 AM, Ross Green <rgkernel@gmail.com> wrote:
>> >> Thanks Paul for your analysis and investigation,
>> >>
>> >> I understand your patches are designed not to "fix" the problem, but
>> >> more to move the problem ahead.
>> >>
>> >> I will include a few more stack traces from various kernels. I can go
>> >> back to around 3.17, with similar trace results.
>> >>
>> >> My observation is that the problem can occur at various times and with
>> >> out any "bad" effect other than more stalls could happen afterwards.
>> >>
>> >> At first I wondered if they might actually be a false positive as the
>> >> kernel seemed to carry on and run quite happily. It is rare that I
>> >> find a kernel just locks up after observing such a stall, or a
>> >> complete kernel splat!
>> >>
>> >> Unfortunately with my testing I have never been able to induce the
>> >> problem under any heavy load that would immediately trigger the
>> >> problem. Indeed most heavy cpu utilisation seemed to just sail on
>> >> quite nicely.
>> >>
>> >> The time for a fault, seems so far, to be non-deterministic with
>> >> quiescent systems taking anywhere from a few hours through to some six
>> >> days before showing the problem.
>> >>
>> >> More recent kernels seem to have richer stall and back trace
>> >> information so I was hoping that might shed some light on how they
>> >> might be occurring..
>> >>
>> >> I usually only run a kernel till I get a new -rc release to test with,
>> >> so a run of 1 week is a typical cycle.
>> >>
>> >> I just wish I could find a sure fire method to trigger the problem!!
>> >>
>> >> I have included a few more traces of various kernels all showing the problem.
>> >>
>> >> Regards,
>> >>
>> >> Ross
>> >>
>> >> On Mon, Jan 4, 2016 at 5:15 AM, Paul E. McKenney
>> >> <paulmck@linux.vnet.ibm.com> wrote:
>> >>> On Sun, Jan 03, 2016 at 07:27:17PM +1100, Ross Green wrote:
>> >>>> I would not describe the load on this test machine as high or real time.
>> >>>>
>> >>>> Apart from a number of standard daemons not much more is running at all!
>> >>>>
>> >>>> I normally build a release kernel as soon as possible and set it running.
>> >>>> Typically I run a series of benchmarks to confirm most things appear
>> >>>> to be working and then just leave it running. During a normal day i
>> >>>> will check on the machine 4/5 times just to see how its going!
>> >>>> Typically I will logon remotely via wifi network connection.
>> >>>>
>> >>>> just for your information i will include a few other stack traces from
>> >>>> previous kernels that may show some trend!
>> >>>>
>> >>>>
>> >>>> Please see the attachments.
>> >>>
>> >>> Thank you for the additional details. This does look similar to some
>> >>> problems I am seeing, though only in heavy rcutorture workloads with
>> >>> CPU hotplugging.
>> >>>
>> >>> I have some crude workarounds in progress, see for example
>> >>> 2da26818e515 (rcu: Awaken grace-period kthread when stalled) at
>> >>> git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git.
>> >>> This workaround kicks the RCU grace-period kthread on every stall warning.
>> >>> In my testing, this workaround results in slow but real forward progress.
>> >>>
>> >>> I have a better workaround in progress, however, please note:
>> >>>
>> >>> 1. I have no intention of sending these workarounds upstream.
>> >>>
>> >>> 2. The workarounds will splat when they take effect. In other words,
>> >>> the idea is not to paper over the problem, but instead to allow
>> >>> me to separate testing concerns.
>> >>>
>> >>> 3. A fix is needed for the underlying bug, wherever it might be.
>> >>>
>> >>> Thanx, Paul
>> >>>
>> >>>> Regards,
>> >>>>
>> >>>> Ross
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Sun, Jan 3, 2016 at 5:17 PM, Paul E. McKenney
>> >>>> <paulmck@linux.vnet.ibm.com> wrote:
>> >>>> > On Sun, Jan 03, 2016 at 04:29:11PM +1100, Ross Green wrote:
>> >>>> >> Still seeing these rcu_preempt stalls on kernels through to 4.4-rc7
>> >>>> >>
>> >>>> >> Still have not found a sure fire method to evoke this stall, but have
>> >>>> >> found that it will normally occur within a week of running a kernel -
>> >>>> >> usually when it is quiet with light load.
>> >>>> >>
>> >>>> >> Have seen similar self detected stalls all the way back to 3.17.
>> >>>> >> Most recent kernels have included 4.4-rc5 4.4-rc6 and 4.4-rc7
>> >>>> >>
>> >>>> >> Regards,
>> >>>> >>
>> >>>> >> Ross
>> >>>> >>
>> >>>> >> On Fri, Dec 11, 2015 at 10:17 PM, Ross Green <rgkernel@gmail.com> wrote:
>> >>>> >> > I have been getting these stalls in kernels going back to 3.17.
>> >>>> >> >
>> >>>> >> > This stall occurs usually under light load but often requires several
>> >>>> >> > days to show itself. I have not found any simple way to trigger the
>> >>>> >> > stall. Indeed heavy workloads seems not to show the fault.
>> >>>> >> >
>> >>>> >> > Anyone have any thoughts here?
>> >>>> >> >
>> >>>> >> > The recent patch by peterz with kernel/sched/wait.c I thought might
>> >>>> >> > help the situation, but alas after a few days of running 4.4-rc4 the
>> >>>> >> > following turned up.
>> >>>> >> >
>> >>>> >> > [179922.003570] INFO: rcu_preempt self-detected stall on CPU
>> >>>> >> > [179922.008178] INFO: rcu_preempt detected stalls on CPUs/tasks:
>> >>>> >> > [179922.008178] 0-...: (1 ticks this GP) idle=a91/1/0
>> >>>> >
>> >>>> > CPU 0 is non-idle from an RCU perspective.
>> >>>> >
>> >>>> >> > softirq=1296733/1296733 fqs=0
>> >>>> >> > [179922.008178]
>> >>>> >> > [179922.008209] (detected by 1, t=8775 jiffies, g=576439, c=576438, q=102)
>> >>>> >> > [179922.008209] Task dump for CPU 0:
>> >>>> >> > [179922.008209] swapper/0 R [179922.008209] running [179922.008209] 0 0 0 0x00000000
>> >>>> >> > [179922.008209] Backtrace:
>> >>>> >> >
>> >>>> >> > [179922.008239] Backtrace aborted due to bad frame pointer <c0907f54>
>> >>>> >
>> >>>> > Can't have everything, I guess...
>> >>>> >
>> >>>> >> > [179922.008239] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x0 s3 ->state=0x1
>> >>>> >
>> >>>> > Something is keeping the rcu_preempt grace-period kthread from
>> >>>> > running. This far into the grace period, it should have a
>> >>>> > timer event waking it every few jiffies. It is currently
>> >>>> > in TASK_INTERRUPTIBLE state.
>> >>>> >
>> >>>> >> > [179922.060302] 0-...: (1 ticks this GP) idle=a91/1/0 softirq=1296733/1296733 fqs=0
>> >>>> >> > [179922.068023] (t=8775 jiffies g=576439 c=576438 q=102)
>> >>>> >> > [179922.073913] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x2 s3 ->state=0x100
>> >>>> >
>> >>>> > Same story, same grace period, pretty much same time. Now there is an FQS
>> >>>> > request (f0x2) and the state is now TASK_WAKING (->state=0x100 == 256).
>> >>>> >
>> >>>> >> > [179922.083587] Task dump for CPU 0:
>> >>>> >> > [179922.087097] swapper/0 R running 0 0 0 0x00000000
>> >>>> >> > [179922.093292] Backtrace:
>> >>>> >> > [179922.096313] [<c0013ea8>] (dump_backtrace) from [<c00140a4>] (show_stack+0x18/0x1c)
>> >>>> >> > [179922.104675] r7:c0908514 r6:80080193 r5:00000000 r4:c090aca8
>> >>>> >> > [179922.110809] [<c001408c>] (show_stack) from [<c005a858>] (sched_show_task+0xbc/0x110)
>> >>>> >> > [179922.119049] [<c005a79c>] (sched_show_task) from [<c005ccd4>] (dump_cpu_task+0x40/0x44)
>> >>>> >> > [179922.127624] r5:c0917680 r4:00000000
>> >>>> >> > [179922.131042] [<c005cc94>] (dump_cpu_task) from [<c0082268>] (rcu_dump_cpu_stacks+0x9c/0xdc)
>> >>>> >> > [179922.140350] r5:c0917680 r4:00000001
>> >>>> >> > [179922.143157] [<c00821cc>] (rcu_dump_cpu_stacks) from [<c008637c>] (rcu_check_callbacks+0x504/0x8e4)
>> >>>> >> > [179922.153808] r9:c0908514 r8:c0917680 r7:00000066 r6:2eeab000
>> >>>> >> > r5:c0904300 r4:ef7af300
>> >>>> >> > [179922.161499] [<c0085e78>] (rcu_check_callbacks) from [<c00895d0>] (update_process_times+0x40/0x6c)
>> >>>> >> > [179922.170898] r10:c009a584 r9:00000001 r8:ef7abc4c r7:0000a3a3
>> >>>> >> > r6:4ec3391c r5:00000000
>> >>>> >> > [179922.179901] r4:c090aca8
>> >>>> >> > [179922.182708] [<c0089590>] (update_process_times) from [<c009a580>]
>> >>>> >> > (tick_sched_handle+0x50/0x54)
>> >>>> >> > [179922.192108] r5:c0907f10 r4:ef7abe40
>> >>>> >> > [179922.195983] [<c009a530>] (tick_sched_handle) from [<c009a5d4>]
>> >>>> >> > (tick_sched_timer+0x50/0x94)
>> >>>> >> > [179922.204895] [<c009a584>] (tick_sched_timer) from [<c0089fe4>]
>> >>>> >> > (__hrtimer_run_queues+0x110/0x1a0)
>> >>>> >> > [179922.214324] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
>> >>>> >> > [179922.220428] [<c0089ed4>] (__hrtimer_run_queues) from [<c008a674>]
>> >>>> >> > (hrtimer_interrupt+0xac/0x1f8)
>> >>>> >> > [179922.227111] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8
>> >>>> >> > r6:ffffffff r5:00000003
>> >>>> >> > [179922.238220] r4:ef7abc00
>> >>>> >> > [179922.238220] [<c008a5c8>] (hrtimer_interrupt) from [<c00170ec>]
>> >>>> >> > (twd_handler+0x38/0x48)
>> >>>> >> > [179922.238220] r10:c09084e8 r9:fa241100 r8:00000011 r7:ef028780
>> >>>> >> > r6:c092574c r5:ef005cc0
>> >>>> >
>> >>>> > All interrupt stack up to this point.
>> >>>> >
>> >>>> > It is quite possible that the stuff below here is at fault as well.
>> >>>> > That said, the grace-period should actually get to execute at some
>> >>>> > point. Do you have a heavy real-time load that might be starving
>> >>>> > things?
>> >>>> >
>> >>>> > Thanx, Paul
>> >>>> >
>> >>>> >> > [179922.257110] r4:00000001
>> >>>> >> > [179922.257110] [<c00170b4>] (twd_handler) from [<c007c8f8>] (handle_percpu_devid_irq+0x74/0x8c)
>> >>>> >> > [179922.269683] r5:ef005cc0 r4:ef7b1740
>> >>>> >> > [179922.269683] [<c007c884>] (handle_percpu_devid_irq) from [<c0078454>] (generic_handle_irq+0x2c/0x3c)
>> >>>> >> > [179922.283233] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000
>> >>>> >> > r5:00000000 r4:c09013e8
>> >>>> >> > [179922.290985] [<c0078428>] (generic_handle_irq) from [<c007872c>] (__handle_domain_irq+0x64/0xbc)
>> >>>> >> > [179922.300842] [<c00786c8>] (__handle_domain_irq) from [<c00094c0>]
>> >>>> >> > (gic_handle_irq+0x50/0x90)
>> >>>> >> > [179922.303222] r9:fa241100 r8:fa240100 r7:c0907f10 r6:fa24010c
>> >>>> >> > r5:c09087a8 r4:c0925748
>> >>>> >> > [179922.315216] [<c0009470>] (gic_handle_irq) from [<c0014bd4>]
>> >>>> >> > (__irq_svc+0x54/0x90)
>> >>>> >> > [179922.319000] Exception stack(0xc0907f10 to 0xc0907f58)
>> >>>> >> > [179922.331542] 7f00: 00000000
>> >>>> >> > ef7ab390 fe600000 00000000
>> >>>> >> > [179922.331542] 7f20: c0906000 c090849c c0900364 c06a8124 c0907f80
>> >>>> >> > c0944563 c09084e8 c0907f6c
>> >>>> >> > [179922.349029] 7f40: c0907f4c c0907f60 c00287ac c0010ba8 60080113 ffffffff
>> >>>> >> > [179922.349029] r9:c0944563 r8:c0907f80 r7:c0907f44 r6:ffffffff
>> >>>> >> > r5:60080113 r4:c0010ba8
>> >>>> >> > [179922.357116] [<c0010b80>] (arch_cpu_idle) from [<c006f034>]
>> >>>> >> > (default_idle_call+0x28/0x34)
>> >>>> >> > [179922.368926] [<c006f00c>] (default_idle_call) from [<c006f154>]
>> >>>> >> > (cpu_startup_entry+0x114/0x18c)
>> >>>> >> > [179922.368926] [<c006f040>] (cpu_startup_entry) from [<c069fc6c>]
>> >>>> >> > (rest_init+0x90/0x94)
>> >>>> >> > [179922.385284] r7:ffffffff r4:00000002
>> >>>> >> > [179922.393463] [<c069fbdc>] (rest_init) from [<c08bbcec>]
>> >>>> >> > (start_kernel+0x370/0x37c)
>> >>>> >> > [179922.400421] r5:c0947000 r4:00000000
>> >>>> >> > [179922.400421] [<c08bb97c>] (start_kernel) from [<8000807c>] (0x8000807c)
>> >>>> >> > $
>> >>>> >
>> >>>> >
>> >>>
>> >>>
>> >>>
>
>
[-- Attachment #2: dmesg-4.4 --]
[-- Type: application/octet-stream, Size: 25178 bytes --]
rfg@bigred:~> ssh rfg@192.168.1.100
rfg@192.168.1.100's password:
$
$
$
$
$ dmesg
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
[ 0.000000] Initializing cgroup subsys cpuacct
[ 0.000000] Linux version 4.4.0-panda (rfg@red.dwarf) (gcc version 5.3.0 (Buildroot 2016.02-git-00183-g1682aee) ) #1 SMP PREEMPT Mon Jan 11 14:02:24 AEDT 2016
[ 0.000000] CPU: ARMv7 Processor [412fc09a] revision 10 (ARMv7), cr=10c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[ 0.000000] Machine model: TI OMAP4 PandaBoard-ES
[ 0.000000] Memory policy: Data cache writealloc
[ 0.000000] OMAP4: Map 0xbfe00000 to fe600000 for dram barrier
[ 0.000000] On node 0 totalpages: 261632
[ 0.000000] free_area_init_node: node 0, pgdat c0945940, node_mem_map ef7f9000
[ 0.000000] Normal zone: 1536 pages used for memmap
[ 0.000000] Normal zone: 0 pages reserved
[ 0.000000] Normal zone: 196608 pages, LIFO batch:31
[ 0.000000] HighMem zone: 65024 pages, LIFO batch:15
[ 0.000000] OMAP4460 ES1.1
[ 0.000000] PERCPU: Embedded 12 pages/cpu @ef7ab000 s17920 r8192 d23040 u49152
[ 0.000000] pcpu-alloc: s17920 r8192 d23040 u49152 alloc=12*4096
[ 0.000000] pcpu-alloc: [0] 0 [0] 1
[ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 260096
[ 0.000000] Kernel command line: root=/dev/mmcblk0p2 rootwait console=ttyO2,115200
[ 0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[ 0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[ 0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[ 0.000000] Memory: 1027368K/1046528K available (6783K kernel code, 250K rwdata, 2132K rodata, 296K init, 294K bss, 19160K reserved, 0K cma-reserved, 260096K highmem)
[ 0.000000] Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xffc00000 - 0xfff00000 (3072 kB)
vmalloc : 0xf0800000 - 0xff800000 ( 240 MB)
lowmem : 0xc0000000 - 0xf0000000 ( 768 MB)
pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
.text : 0xc0008000 - 0xc08bd0d4 (8917 kB)
.init : 0xc08be000 - 0xc0908000 ( 296 kB)
.data : 0xc0908000 - 0xc0946920 ( 251 kB)
.bss : 0xc0949000 - 0xc0992bb8 ( 295 kB)
[ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[ 0.000000] Preemptible hierarchical RCU implementation.
[ 0.000000] Build-time adjustment of leaf fanout to 32.
[ 0.000000] NR_IRQS:16 nr_irqs:16 16
[ 0.000000] L2C: platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C: DT/platform modifies aux control register: 0x0e070000 -> 0x3e470000
[ 0.000000] L2C-310 enabling early BRESP for Cortex-A9
[ 0.000000] OMAP L2C310: ROM does not support power control setting
[ 0.000000] L2C-310 dynamic clock gating disabled, standby mode disabled
[ 0.000000] L2C-310 cache controller enabled, 16 ways, 1024 kB
[ 0.000000] L2C-310: CACHE_ID 0x410000c7, AUX_CTRL 0x4e470000
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node dss_fck
[ 0.000000] ti_dt_clocks_register: failed to lookup clock node bandgap_fclk
[ 0.000000] OMAP clockevent source: timer1 at 32768 Hz
[ 0.000000] clocksource: 32k_counter: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 58327039986419 ns
[ 0.000030] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 65535999984741ns
[ 0.000030] OMAP clocksource: 32k_counter at 32768 Hz
[ 0.000885] Console: colour dummy device 80x30
[ 0.000885] Calibrating delay loop... 1397.55 BogoMIPS (lpj=6987776)
[ 0.089233] pid_max: default: 32768 minimum: 301
[ 0.089294] Security Framework initialized
[ 0.089355] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.089355] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[ 0.089935] Initializing cgroup subsys io
[ 0.089965] Initializing cgroup subsys devices
[ 0.089996] Initializing cgroup subsys freezer
[ 0.089996] CPU: Testing write buffer coherency: ok
[ 0.090240] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[ 0.090301] Setting up static identity map for 0x80008280 - 0x800082d8
[ 0.209228] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[ 0.209289] Brought up 2 CPUs
[ 0.209320] SMP: Total of 2 processors activated (2795.11 BogoMIPS).
[ 0.209320] CPU: All CPU(s) started in SVC mode.
[ 0.210144] devtmpfs: initialized
[ 0.226409] VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
[ 0.228057] omap_hwmod: l3_main_3 using broken dt data from ocp
[ 0.229614] omap_hwmod: l3_main_2 using broken dt data from ocp
[ 0.324829] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.325073] xor: measuring software checksum speed
[ 0.419219] arm4regs : 1006.400 MB/sec
[ 0.519195] 8regs : 804.800 MB/sec
[ 0.619201] 32regs : 761.600 MB/sec
[ 0.619232] xor: using function: arm4regs (1006.400 MB/sec)
[ 0.619476] pinctrl core: initialized pinctrl subsystem
[ 0.621307] NET: Registered protocol family 16
[ 0.622100] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.631378] OMAP GPIO hardware version 0.1
[ 0.636566] omap-gpmc 50000000.gpmc: GPMC revision 6.0
[ 0.636566] gpmc_mem_init: disabling cs 0 mapped at 0x0-0x1000000
[ 0.637084] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.637451] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.637817] irq: no irq domain found for /ocp/l4@4a000000/scm@100000/pinmux@40 !
[ 0.649383] platform 4b501000.aes: Cannot lookup hwmod 'aes'
[ 0.649688] platform 480a5000.des: Cannot lookup hwmod 'des'
[ 0.655578] hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
[ 0.655609] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.656372] omap4_sram_init:Unable to get sram pool needed to handle errata I688
[ 0.656616] OMAP DMA hardware revision 0.0
[ 0.889221] raid6: int32x1 gen() 103 MB/s
[ 1.059326] raid6: int32x1 xor() 85 MB/s
[ 1.229278] raid6: int32x2 gen() 134 MB/s
[ 1.399414] raid6: int32x2 xor() 97 MB/s
[ 1.569458] raid6: int32x4 gen() 145 MB/s
[ 1.739440] raid6: int32x4 xor() 100 MB/s
[ 1.909271] raid6: int32x8 gen() 160 MB/s
[ 2.079467] raid6: int32x8 xor() 98 MB/s
[ 2.079467] raid6: using algorithm int32x8 gen() 160 MB/s
[ 2.079467] raid6: .... xor() 98 MB/s, rmw enabled
[ 2.079467] raid6: using intx1 recovery algorithm
[ 2.086669] omap-dma-engine 4a056000.dma-controller: OMAP DMA engine driver
[ 2.159332] reg-fixed-voltage wl12xx_vmmc: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_wl12xx_gpio, deferring probe
[ 2.163482] omap-iommu 4a066000.mmu: 4a066000.mmu registered
[ 2.163604] omap-iommu 55082000.mmu: 55082000.mmu registered
[ 2.164550] SCSI subsystem initialized
[ 2.164825] usbcore: registered new interface driver usbfs
[ 2.164947] usbcore: registered new interface driver hub
[ 2.165100] usbcore: registered new device driver usb
[ 2.166015] omap_i2c 48070000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c1_pins, deferring probe
[ 2.166076] omap_i2c 48072000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c2_pins, deferring probe
[ 2.166107] omap_i2c 48060000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c3_pins, deferring probe
[ 2.166168] omap_i2c 48350000.i2c: could not find pctldev for node /ocp/l4@4a000000/scm@100000/pinmux@40/pinmux_i2c4_pins, deferring probe
[ 2.166412] Linux video capture interface: v2.00
[ 2.167205] Advanced Linux Sound Architecture Driver Initialized.
[ 2.168548] clocksource: Switched to clocksource 32k_counter
[ 2.185699] NET: Registered protocol family 2
[ 2.186279] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[ 2.186370] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[ 2.186523] TCP: Hash tables configured (established 8192 bind 8192)
[ 2.186584] UDP hash table entries: 512 (order: 2, 16384 bytes)
[ 2.186645] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[ 2.186828] NET: Registered protocol family 1
[ 2.187194] RPC: Registered named UNIX socket transport module.
[ 2.187194] RPC: Registered udp transport module.
[ 2.187194] RPC: Registered tcp transport module.
[ 2.187225] RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 2.189483] hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
[ 2.192413] futex hash table entries: 512 (order: 3, 32768 bytes)
[ 2.204467] NFS: Registering the id_resolver key type
[ 2.204528] Key type id_resolver registered
[ 2.204528] Key type id_legacy registered
[ 2.204559] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[ 2.204589] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 2.206024] ntfs: driver 2.1.32 [Flags: R/O].
[ 2.206420] fuse init (API version 7.23)
[ 2.209442] jitterentropy: Initialization failed with host not compliant with requirements: 2
[ 2.209533] bounce: pool size: 64 pages
[ 2.209533] io scheduler noop registered
[ 2.209564] io scheduler deadline registered
[ 2.209625] io scheduler cfq registered (default)
[ 2.210479] pinctrl-single 4a100040.pinmux: 203 pins at pa fc100040 size 406
[ 2.210693] pinctrl-single 4a31e040.pinmux: 28 pins at pa fc31e040 size 56
[ 2.214019] 58000000.dss supply vdda_video not found, using dummy regulator
[ 2.214111] OMAP DSS rev 4.0
[ 2.214263] omapdss_dss 58000000.dss: bound 58001000.dispc (ops dispc_component_ops)
[ 2.214508] omapdss_dss 58000000.dss: bound 58005000.encoder (ops dsi_component_ops)
[ 2.215057] omapdss_dss 58000000.dss: bound 58006000.encoder (ops hdmi4_component_ops)
[ 2.217407] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 2.220275] omap_uart 4806a000.serial: no wakeirq for uart0
[ 2.220428] 4806a000.serial: ttyO0 at MMIO 0x4806a000 (irq = 226, base_baud = 3000000) is a OMAP UART0
[ 2.221008] 4806c000.serial: ttyO1 at MMIO 0x4806c000 (irq = 227, base_baud = 3000000) is a OMAP UART1
[ 2.221527] 48020000.serial: ttyO2 at MMIO 0x48020000 (irq = 228, base_baud = 3000000) is a OMAP UART2
[ 3.179473] console [ttyO2] enabled
[ 3.183654] 4806e000.serial: ttyO3 at MMIO 0x4806e000 (irq = 229, base_baud = 3000000) is a OMAP UART3
[ 3.194610] [drm] Initialized drm 1.1.0 20060810
[ 3.212921] loop: module loaded
[ 3.219329] usbcore: registered new interface driver smsc95xx
[ 3.225372] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[ 3.232269] ehci-omap: OMAP-EHCI Host Controller driver
[ 3.248596] ehci-omap 4a064c00.ehci: EHCI Host Controller
[ 3.254272] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1
[ 3.262359] ehci-omap 4a064c00.ehci: irq 336, io mem 0x4a064c00
[ 3.288604] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00
[ 3.294982] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 3.302093] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 3.309661] usb usb1: Product: EHCI Host Controller
[ 3.314788] usb usb1: Manufacturer: Linux 4.4.0-panda ehci_hcd
[ 3.320892] usb usb1: SerialNumber: 4a064c00.ehci
[ 3.326477] hub 1-0:1.0: USB hub found
[ 3.330444] hub 1-0:1.0: 3 ports detected
[ 3.335327] usbcore: registered new interface driver cdc_wdm
[ 3.341430] usbcore: registered new interface driver usb-storage
[ 3.348114] i2c /dev entries driver
[ 3.351928] usbcore: registered new interface driver uvcvideo
[ 3.357940] USB Video Class driver (1.1.1)
[ 3.363464] omap_wdt: OMAP Watchdog Timer Rev 0x01: initial timeout 60 sec
[ 3.371002] device-mapper: uevent: version 1.0.3
[ 3.376159] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: dm-devel@redhat.com
[ 3.449615] hidraw: raw HID events driver (C) Jiri Kosina
[ 3.455474] usbcore: registered new interface driver usbhid
[ 3.461334] usbhid: USB HID core driver
[ 3.465850] emif 4c000000.emif: emif_probe: device configured with addr = f08a8000 and IRQ247
[ 3.475189] emif 4d000000.emif: emif_probe: device configured with addr = f08aa000 and IRQ248
[ 3.486114] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 3.494049] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 3.501525] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 3.507232] Initializing XFRM netlink socket
[ 3.511779] NET: Registered protocol family 17
[ 3.516723] Key type dns_resolver registered
[ 3.521392] twl: not initialized
[ 3.524810] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.533416] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.541931] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.550415] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.558929] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.567382] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1375000 Vs max 1316660
[ 3.575866] twl6030_uv_to_vsel:OUT OF RANGE! non mapped vsel for 1410000 Vs max 1316660
[ 3.584381] omap2_set_init_voltage: unable to find boot up OPP for vdd_mpu
[ 3.591888] omap2_set_init_voltage: unable to set vdd_mpu
[ 3.597625] omap2_set_init_voltage: unable to find boot up OPP for vdd_core
[ 3.605010] omap2_set_init_voltage: unable to set vdd_core
[ 3.610839] omap2_set_init_voltage: unable to find boot up OPP for vdd_iva
[ 3.618103] omap2_set_init_voltage: unable to set vdd_iva
[ 3.624694] Power Management for TI OMAP4+ devices.
[ 3.629882] OMAP4 PM: u-boot >= v2012.07 is required for full PM support
[ 3.637084] ThumbEE CPU extension supported.
[ 3.641723] Registering SWP/SWPB emulation handler
[ 3.641723] SmartReflex Class3 initialized
[ 3.651245] usb 1-1: new high-speed USB device number 2 using ehci-omap
[ 3.671356] Btrfs loaded
[ 3.676727] dmm 4e000000.dmm: initialized all PAT entries
[ 3.694549] Skipping twl internal clock init and using bootloader value (unknown osc rate)
[ 3.728759] twl 0-0048: PIH (irq 340) nested IRQs
[ 3.735229] twl_rtc 48070000.i2c:twl@48:rtc: Enabling TWL-RTC
[ 3.744018] twl_rtc 48070000.i2c:twl@48:rtc: rtc core: registered 48070000.i2c:twl@48 as rtc0
[ 3.765716] twl6040 0-004b: clk32k is not handled
[ 3.771759] twl6040 0-004b: Failed to read revision register: -121
[ 3.778564] twl6040: probe of 0-004b failed with error -121
[ 3.784484] omap_i2c 48070000.i2c: bus 0 rev0.11 at 400 kHz
[ 3.791259] omap_i2c 48072000.i2c: bus 1 rev0.11 at 400 kHz
[ 3.798217] omap_i2c 48060000.i2c: bus 2 rev0.11 at 100 kHz
[ 3.805053] omap_i2c 48350000.i2c: bus 3 rev0.11 at 400 kHz
[ 3.812591] usb 1-1: New USB device found, idVendor=0424, idProduct=9514
[ 3.819763] usb 1-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 3.828460] hub 1-1:1.0: USB hub found
[ 3.832611] hub 1-1:1.0: 5 ports detected
[ 4.000213] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 4.000823] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 4.016784] twl_rtc 48070000.i2c:twl@48:rtc: setting system clock to 2016-01-11 05:22:55 UTC (1452489775)
[ 4.027038] sr_init: No PMIC hook to init smartreflex
[ 4.032440] smartreflex smartreflex.0: omap_sr_probe: SmartReflex driver initialized
[ 4.041046] smartreflex smartreflex.1: omap_sr_probe: SmartReflex driver initialized
[ 4.041046] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 4.041320] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 4.064605] smartreflex smartreflex.2: omap_sr_probe: SmartReflex driver initialized
[ 4.064880] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 4.064880] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 4.088348] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 4.096588] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 4.098876] VDAC: disabling
[ 4.099334] VANA: disabling
[ 4.099761] VUSB: disabling
[ 4.099945] ALSA device list:
[ 4.099945] No soundcards found.
[ 4.119689] Waiting for root device /dev/mmcblk0p2...
[ 4.120574] mmc0: host does not support reading read-only switch, assuming write-enable
[ 4.137084] mmc0: new high speed SDHC card at address e624
[ 4.153472] mmcblk0: mmc0:e624 SL32G 28.7 GiB
[ 4.168609] usb 1-1.1: new high-speed USB device number 3 using ehci-omap
[ 4.187561] mmcblk0: p1 p2 p3 p4
[ 4.192810] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 4.200958] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 4.212371] omap_hsmmc 480d5000.mmc: card claims to support voltages below defined range
[ 4.230957] mmc1: queuing unknown CIS tuple 0x91 (3 bytes)
[ 4.237945] mmc1: new SDIO card at address 0001
[ 4.253662] EXT4-fs (mmcblk0p2): couldn't mount as ext3 due to feature incompatibilities
[ 4.262908] EXT4-fs (mmcblk0p2): mounting ext2 file system using the ext4 subsystem
[ 4.304626] usb 1-1.1: New USB device found, idVendor=0424, idProduct=ec00
[ 4.311920] usb 1-1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[ 4.322601] smsc95xx v1.0.4
[ 4.322662] EXT4-fs (mmcblk0p2): mounted filesystem without journal. Opts: (null)
[ 4.322662] VFS: Mounted root (ext2 filesystem) readonly on device 179:2.
[ 4.344482] devtmpfs: mounted
[ 4.346893] Freeing unused kernel memory: 296K (c08be000 - c0908000)
[ 4.430908] smsc95xx 1-1.1:1.0 eth0: register 'smsc95xx' at usb-4a064c00.ehci-1.1, smsc95xx USB 2.0 Ethernet, a6:b3:57:8f:86:da
[ 4.443634] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 4.451721] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 4.767181] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[ 4.901000] random: dd urandom read with 31 bits of entropy available
[ 9.734039] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup
[ 19.645782] Bluetooth: Core ver 2.21
[ 19.650665] NET: Registered protocol family 31
[ 19.655975] Bluetooth: HCI device and connection manager initialized
[ 19.662902] Bluetooth: HCI socket layer initialized
[ 19.668151] Bluetooth: L2CAP socket layer initialized
[ 19.673583] Bluetooth: SCO socket layer initialized
[ 27.280883] NFSD: the nfsdcld client tracking upcall will be removed in 3.10. Please transition to using nfsdcltrack.
[ 27.292205] NFSD: starting 90-second grace period (net c093fe80)
[ 38.805725] random: nonblocking pool is initialized
[ 68.041259] Adding 2097148k swap on /dev/mmcblk0p3. Priority:-1 extents:1 across:2097148k SS
[ 88.292449] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 88.300750] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 88.453155] omap-abe-twl6040 sound: ASoC: CODEC DAI twl6040-legacy not registered
[ 88.462005] omap-abe-twl6040 sound: snd_soc_register_card() failed: -517
[ 88.843353] wlcore: loaded
[ 100.856964] wlcore: firmware booted (Rev 6.3.10.0.133)
[ 104.490417] wlan0: authenticate with 90:f6:52:81:ee:70
[ 104.506256] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[ 104.522888] wlan0: send auth to 90:f6:52:81:ee:70 (try 2/3)
[ 104.539062] wlan0: send auth to 90:f6:52:81:ee:70 (try 3/3)
[ 104.555969] wlan0: authentication with 90:f6:52:81:ee:70 timed out
[ 108.035308] wlan0: authenticate with 90:f6:52:81:ee:70
[ 108.069183] wlan0: send auth to 90:f6:52:81:ee:70 (try 1/3)
[ 108.077239] wlan0: authenticated
[ 108.093322] wlan0: associate with 90:f6:52:81:ee:70 (try 1/3)
[ 108.104034] wlan0: RX AssocResp from 90:f6:52:81:ee:70 (capab=0x431 status=0 aid=1)
[ 108.121459] wlan0: associated
[ 108.180480] wlcore: Association completed.
[ 147.295623] NFSD: Unable to end grace period: -110
[43727.971862] INFO: rcu_preempt self-detected stall on CPU
[43727.974334] INFO: rcu_preempt detected stalls on CPUs/tasks:
[43727.974365] 0-...: (1 GPs behind) idle=61d/1/0 softirq=319694/319695 fqs=1
[43727.974365]
[43727.974365] (detected by 1, t=9178 jiffies, g=140244, c=140243, q=189)
[43727.974365] Task dump for CPU 0:
[43727.974395] swapper/0 R
[43727.974395] running
[43727.974395] 0 0 0 0x00000000
[43727.974395] Backtrace:
[43727.974395] Backtrace aborted due to bad frame pointer <c0909f54>
[43728.016876] 0-...: (1 GPs behind) idle=61d/1/0 softirq=319694/319695 fqs=2
[43728.026092] (t=9178 jiffies g=140244 c=140243 q=189)
[43728.027770] Task dump for CPU 0:
[43728.034942] swapper/0 R running 0 0 0 0x00000002
[43728.034942] Backtrace:
[43728.041687] [<c0013e9c>] (dump_backtrace) from [<c0014098>] (show_stack+0x18/0x1c)
[43728.045379] r7:c090a514 r6:80080193 r5:00000000 r4:c090cca8
[43728.045379] [<c0014080>] (show_stack) from [<c005a7e0>] (sched_show_task+0xbc/0x110)
[43728.065887] [<c005a724>] (sched_show_task) from [<c005cc68>] (dump_cpu_task+0x40/0x48)
[43728.074340] r5:c0919680 r4:00000000
[43728.074920] [<c005cc28>] (dump_cpu_task) from [<c0082200>] (rcu_dump_cpu_stacks+0x9c/0xdc)
[43728.087463] r5:c0919680 r4:00000001
[43728.087463] [<c0082164>] (rcu_dump_cpu_stacks) from [<c0086314>] (rcu_check_callbacks+0x504/0x8e4)
[43728.100738] r9:c090a514 r8:c0919680 r7:000000bd r6:2eea8000 r5:c0907300 r4:ef7af300
[43728.108978] [<c0085e10>] (rcu_check_callbacks) from [<c0089568>] (update_process_times+0x40/0x6c)
[43728.108978] r10:c009a51c r9:00000001 r8:ef7abc4c r7:000027c5 r6:30022228 r5:00000000
[43728.126678] r4:c090cca8
[43728.126678] [<c0089528>] (update_process_times) from [<c009a518>] (tick_sched_handle+0x50/0x54)
[43728.138580] r5:c0909f10 r4:ef7abe40
[43728.138580] [<c009a4c8>] (tick_sched_handle) from [<c009a56c>] (tick_sched_timer+0x50/0x94)
[43728.151214] [<c009a51c>] (tick_sched_timer) from [<c0089f7c>] (__hrtimer_run_queues+0x110/0x1a0)
[43728.154327] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
[43728.166534] [<c0089e6c>] (__hrtimer_run_queues) from [<c008a60c>] (hrtimer_interrupt+0xac/0x1f8)
[43728.166534] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8 r6:ffffffff r5:00000003
[43728.166534] r4:ef7abc00
[43728.184143] [<c008a560>] (hrtimer_interrupt) from [<c00170ec>] (twd_handler+0x38/0x48)
[43728.194335] r10:c090a4e8 r9:fa241100 r8:00000011 r7:ef028780 r6:c092774c r5:ef005cc0
[43728.203521] r4:00000001
[43728.203521] [<c00170b4>] (twd_handler) from [<c007c890>] (handle_percpu_devid_irq+0x74/0x8c)
[43728.215148] r5:ef005cc0 r4:ef7b1740
[43728.215148] [<c007c81c>] (handle_percpu_devid_irq) from [<c00783cc>] (generic_handle_irq+0x2c/0x3c)
[43728.220336] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000 r5:00000000 r4:c09043e0
[43728.220336] [<c00783a0>] (generic_handle_irq) from [<c00786a4>] (__handle_domain_irq+0x64/0xbc)
[43728.245941] [<c0078640>] (__handle_domain_irq) from [<c00094c0>] (gic_handle_irq+0x50/0x90)
[43728.254333] r9:fa241100 r8:fa240100 r7:c0909f10 r6:fa24010c r5:c090a7a8 r4:c0927748
[43728.254791] [<c0009470>] (gic_handle_irq) from [<c0014bd4>] (__irq_svc+0x54/0x90)
[43728.270935] Exception stack(0xc0909f10 to 0xc0909f58)
[43728.274322] 9f00: 00000000 ef7ab390 fe600000 00000000
[43728.274322] 9f20: c0908000 c090a49c c0903364 c06aa120 c0909f80 c09466e3 c090a4e8 c0909f6c
[43728.284942] 9f40: c0909f4c c0909f60 c002878c c0010ba8 60080113 ffffffff
[43728.294372] r9:c09466e3 r8:c0909f80 r7:c0909f44 r6:ffffffff r5:60080113 r4:c0010ba8
[43728.304321] [<c0010b80>] (arch_cpu_idle) from [<c006efac>] (default_idle_call+0x28/0x34)
[43728.314331] [<c006ef84>] (default_idle_call) from [<c006f0cc>] (cpu_startup_entry+0x114/0x18c)
[43728.324340] [<c006efb8>] (cpu_startup_entry) from [<c06a1284>] (rest_init+0x90/0x94)
[43728.334716] r7:ffffffff r4:00000002
[43728.334716] [<c06a11f4>] (rest_init) from [<c08becec>] (start_kernel+0x370/0x37c)
[43728.344329] r5:c0949000 r4:00000000
[43728.346435] [<c08be97c>] (start_kernel) from [<8000807c>] (0x8000807c)
$
$
$
$
$
$
$
$ sync
$ sync
$ cat /proc/version
Linux version 4.4.0-panda (rfg@red.dwarf) (gcc version 5.3.0 (Buildroot 2016.02-git-00183-g1682aee) ) #1 SMP PREEMPT Mon Jan 11 14:02:24 AEDT 2016
$
$ uptime
13:12:58 up 2 days, 20:50, load average: 0.04, 0.09, 0.12
$
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: rcu_preempt self-detected stall on CPU from 4.4-rc4, since 3.17
2016-01-14 5:39 ` Ross Green
@ 2016-01-16 0:39 ` Paul E. McKenney
0 siblings, 0 replies; 12+ messages in thread
From: Paul E. McKenney @ 2016-01-16 0:39 UTC (permalink / raw)
To: Ross Green
Cc: linux-kernel, mingo, jiangshanlai, dipankar, akpm,
Mathieu Desnoyers, josh, tglx, peterz, rostedt, dhowells,
Eric Dumazet, dvhart, Frédéric Weisbecker, oleg,
pranith kumar
On Thu, Jan 14, 2016 at 04:39:31PM +1100, Ross Green wrote:
> Just for completeness I have tested the kernel 4.4 and have seen stall
> warnings similar to the others reported.
>
> I have included the dmesg dump with stack trace for peoples further
> understanding.
Thank you! Interestingly enough, although the dump looks similar, I
don't see the usual message about the RCU grace-period kthread being
starved. So this might be something different.
Could you please try current -rcu, booting with rcutree.rcu_kick_kthreads?
Thanx, Paul
> Regards,
>
> Ross
>
> On Sat, Jan 9, 2016 at 5:35 AM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> > On Thu, Jan 07, 2016 at 10:10:54AM +1100, Ross Green wrote:
> >> Just for completeness I have attached another rcu_preempt stall
> >> warning from a kinux-3.17.1 kernel.
> >>
> >> So it looks like these stall warning go back that far.
> >>
> >> It could be that the kernel instrumentation improved enough to detect
> >> this situation around the 3.17 kernel. I can't find any earlier
> >> records of these stall warnings before these kernels. I guess someone
> >> must have suspected there was a potential for these stalls and hence
> >> the detection facility.
> >
> > Thank you for testing this, Ross! For whatever it is worth, the
> > stall-detection code was added early on, back in the 2.6 days.
> >
> > On the workaround end of things, I am having limited success by forcing
> > RCU's grace-period kthread off of outgoing CPUs at down-prepare time.
> > I really don't like this workaround because I would rather give the
> > sysadm full control of where these kthreads run, but workarounds are
> > workarounds...
> >
> > See 3263d1f49276 (rcu: Migrate grace-period kthread off of outgoing CPU)
> > for the current version in -rcu.
> >
> > Thanx, Paul
> >
> >> Regards,
> >>
> >> Ross
> >>
> >> On Tue, Jan 5, 2016 at 1:21 AM, Ross Green <rgkernel@gmail.com> wrote:
> >> > Well with the release of 4.4-rc8 I have built and
> >> > started testing the kernel.
> >> >
> >> > With some luck I managed to get a rcu_preempt stall within a few hours
> >> > of testing.
> >> >
> >> > Upon booting, I ran a small series of bench marks to make sure
> >> > everything is running as expected. limited regression testing and then
> >> > just left the system to idle away, with periodic monitoring from the
> >> > network.
> >> >
> >> >
> >> > please find attached two stack traces from linux-4.4-rc8 and also linux-4.4-rc7.
> >> >
> >> > The interesting thing with the rc7 trace is that there are multiple
> >> > stalls that have occurred over a 6 day period.
> >> >
> >> > I realise Paul you have a number of changes pending for the the RCU
> >> > code. It would be good to try and establish what is happening with
> >> > these stalls before the impact of those changes given that timings and
> >> > dynamics might change whats happening in the current environment.
> >> >
> >> > As reported earlier, i have never been able to induce these stalls
> >> > with heavy loading of the system. The only method I can be sure of is
> >> > to leave the system quiet and let time go by till a stall occurs.
> >> >
> >> > Regards,
> >> >
> >> > Ross
> >> >
> >> > On Mon, Jan 4, 2016 at 9:00 AM, Ross Green <rgkernel@gmail.com> wrote:
> >> >> Thanks Paul for your analysis and investigation,
> >> >>
> >> >> I understand your patches are designed not to "fix" the problem, but
> >> >> more to move the problem ahead.
> >> >>
> >> >> I will include a few more stack traces from various kernels. I can go
> >> >> back to around 3.17, with similar trace results.
> >> >>
> >> >> My observation is that the problem can occur at various times and with
> >> >> out any "bad" effect other than more stalls could happen afterwards.
> >> >>
> >> >> At first I wondered if they might actually be a false positive as the
> >> >> kernel seemed to carry on and run quite happily. It is rare that I
> >> >> find a kernel just locks up after observing such a stall, or a
> >> >> complete kernel splat!
> >> >>
> >> >> Unfortunately with my testing I have never been able to induce the
> >> >> problem under any heavy load that would immediately trigger the
> >> >> problem. Indeed most heavy cpu utilisation seemed to just sail on
> >> >> quite nicely.
> >> >>
> >> >> The time for a fault, seems so far, to be non-deterministic with
> >> >> quiescent systems taking anywhere from a few hours through to some six
> >> >> days before showing the problem.
> >> >>
> >> >> More recent kernels seem to have richer stall and back trace
> >> >> information so I was hoping that might shed some light on how they
> >> >> might be occurring..
> >> >>
> >> >> I usually only run a kernel till I get a new -rc release to test with,
> >> >> so a run of 1 week is a typical cycle.
> >> >>
> >> >> I just wish I could find a sure fire method to trigger the problem!!
> >> >>
> >> >> I have included a few more traces of various kernels all showing the problem.
> >> >>
> >> >> Regards,
> >> >>
> >> >> Ross
> >> >>
> >> >> On Mon, Jan 4, 2016 at 5:15 AM, Paul E. McKenney
> >> >> <paulmck@linux.vnet.ibm.com> wrote:
> >> >>> On Sun, Jan 03, 2016 at 07:27:17PM +1100, Ross Green wrote:
> >> >>>> I would not describe the load on this test machine as high or real time.
> >> >>>>
> >> >>>> Apart from a number of standard daemons not much more is running at all!
> >> >>>>
> >> >>>> I normally build a release kernel as soon as possible and set it running.
> >> >>>> Typically I run a series of benchmarks to confirm most things appear
> >> >>>> to be working and then just leave it running. During a normal day i
> >> >>>> will check on the machine 4/5 times just to see how its going!
> >> >>>> Typically I will logon remotely via wifi network connection.
> >> >>>>
> >> >>>> just for your information i will include a few other stack traces from
> >> >>>> previous kernels that may show some trend!
> >> >>>>
> >> >>>>
> >> >>>> Please see the attachments.
> >> >>>
> >> >>> Thank you for the additional details. This does look similar to some
> >> >>> problems I am seeing, though only in heavy rcutorture workloads with
> >> >>> CPU hotplugging.
> >> >>>
> >> >>> I have some crude workarounds in progress, see for example
> >> >>> 2da26818e515 (rcu: Awaken grace-period kthread when stalled) at
> >> >>> git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git.
> >> >>> This workaround kicks the RCU grace-period kthread on every stall warning.
> >> >>> In my testing, this workaround results in slow but real forward progress.
> >> >>>
> >> >>> I have a better workaround in progress, however, please note:
> >> >>>
> >> >>> 1. I have no intention of sending these workarounds upstream.
> >> >>>
> >> >>> 2. The workarounds will splat when they take effect. In other words,
> >> >>> the idea is not to paper over the problem, but instead to allow
> >> >>> me to separate testing concerns.
> >> >>>
> >> >>> 3. A fix is needed for the underlying bug, wherever it might be.
> >> >>>
> >> >>> Thanx, Paul
> >> >>>
> >> >>>> Regards,
> >> >>>>
> >> >>>> Ross
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On Sun, Jan 3, 2016 at 5:17 PM, Paul E. McKenney
> >> >>>> <paulmck@linux.vnet.ibm.com> wrote:
> >> >>>> > On Sun, Jan 03, 2016 at 04:29:11PM +1100, Ross Green wrote:
> >> >>>> >> Still seeing these rcu_preempt stalls on kernels through to 4.4-rc7
> >> >>>> >>
> >> >>>> >> Still have not found a sure fire method to evoke this stall, but have
> >> >>>> >> found that it will normally occur within a week of running a kernel -
> >> >>>> >> usually when it is quiet with light load.
> >> >>>> >>
> >> >>>> >> Have seen similar self detected stalls all the way back to 3.17.
> >> >>>> >> Most recent kernels have included 4.4-rc5 4.4-rc6 and 4.4-rc7
> >> >>>> >>
> >> >>>> >> Regards,
> >> >>>> >>
> >> >>>> >> Ross
> >> >>>> >>
> >> >>>> >> On Fri, Dec 11, 2015 at 10:17 PM, Ross Green <rgkernel@gmail.com> wrote:
> >> >>>> >> > I have been getting these stalls in kernels going back to 3.17.
> >> >>>> >> >
> >> >>>> >> > This stall occurs usually under light load but often requires several
> >> >>>> >> > days to show itself. I have not found any simple way to trigger the
> >> >>>> >> > stall. Indeed heavy workloads seems not to show the fault.
> >> >>>> >> >
> >> >>>> >> > Anyone have any thoughts here?
> >> >>>> >> >
> >> >>>> >> > The recent patch by peterz with kernel/sched/wait.c I thought might
> >> >>>> >> > help the situation, but alas after a few days of running 4.4-rc4 the
> >> >>>> >> > following turned up.
> >> >>>> >> >
> >> >>>> >> > [179922.003570] INFO: rcu_preempt self-detected stall on CPU
> >> >>>> >> > [179922.008178] INFO: rcu_preempt detected stalls on CPUs/tasks:
> >> >>>> >> > [179922.008178] 0-...: (1 ticks this GP) idle=a91/1/0
> >> >>>> >
> >> >>>> > CPU 0 is non-idle from an RCU perspective.
> >> >>>> >
> >> >>>> >> > softirq=1296733/1296733 fqs=0
> >> >>>> >> > [179922.008178]
> >> >>>> >> > [179922.008209] (detected by 1, t=8775 jiffies, g=576439, c=576438, q=102)
> >> >>>> >> > [179922.008209] Task dump for CPU 0:
> >> >>>> >> > [179922.008209] swapper/0 R [179922.008209] running [179922.008209] 0 0 0 0x00000000
> >> >>>> >> > [179922.008209] Backtrace:
> >> >>>> >> >
> >> >>>> >> > [179922.008239] Backtrace aborted due to bad frame pointer <c0907f54>
> >> >>>> >
> >> >>>> > Can't have everything, I guess...
> >> >>>> >
> >> >>>> >> > [179922.008239] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x0 s3 ->state=0x1
> >> >>>> >
> >> >>>> > Something is keeping the rcu_preempt grace-period kthread from
> >> >>>> > running. This far into the grace period, it should have a
> >> >>>> > timer event waking it every few jiffies. It is currently
> >> >>>> > in TASK_INTERRUPTIBLE state.
> >> >>>> >
> >> >>>> >> > [179922.060302] 0-...: (1 ticks this GP) idle=a91/1/0 softirq=1296733/1296733 fqs=0
> >> >>>> >> > [179922.068023] (t=8775 jiffies g=576439 c=576438 q=102)
> >> >>>> >> > [179922.073913] rcu_preempt kthread starved for 8775 jiffies! g576439 c576438 f0x2 s3 ->state=0x100
> >> >>>> >
> >> >>>> > Same story, same grace period, pretty much same time. Now there is an FQS
> >> >>>> > request (f0x2) and the state is now TASK_WAKING (->state=0x100 == 256).
> >> >>>> >
> >> >>>> >> > [179922.083587] Task dump for CPU 0:
> >> >>>> >> > [179922.087097] swapper/0 R running 0 0 0 0x00000000
> >> >>>> >> > [179922.093292] Backtrace:
> >> >>>> >> > [179922.096313] [<c0013ea8>] (dump_backtrace) from [<c00140a4>] (show_stack+0x18/0x1c)
> >> >>>> >> > [179922.104675] r7:c0908514 r6:80080193 r5:00000000 r4:c090aca8
> >> >>>> >> > [179922.110809] [<c001408c>] (show_stack) from [<c005a858>] (sched_show_task+0xbc/0x110)
> >> >>>> >> > [179922.119049] [<c005a79c>] (sched_show_task) from [<c005ccd4>] (dump_cpu_task+0x40/0x44)
> >> >>>> >> > [179922.127624] r5:c0917680 r4:00000000
> >> >>>> >> > [179922.131042] [<c005cc94>] (dump_cpu_task) from [<c0082268>] (rcu_dump_cpu_stacks+0x9c/0xdc)
> >> >>>> >> > [179922.140350] r5:c0917680 r4:00000001
> >> >>>> >> > [179922.143157] [<c00821cc>] (rcu_dump_cpu_stacks) from [<c008637c>] (rcu_check_callbacks+0x504/0x8e4)
> >> >>>> >> > [179922.153808] r9:c0908514 r8:c0917680 r7:00000066 r6:2eeab000
> >> >>>> >> > r5:c0904300 r4:ef7af300
> >> >>>> >> > [179922.161499] [<c0085e78>] (rcu_check_callbacks) from [<c00895d0>] (update_process_times+0x40/0x6c)
> >> >>>> >> > [179922.170898] r10:c009a584 r9:00000001 r8:ef7abc4c r7:0000a3a3
> >> >>>> >> > r6:4ec3391c r5:00000000
> >> >>>> >> > [179922.179901] r4:c090aca8
> >> >>>> >> > [179922.182708] [<c0089590>] (update_process_times) from [<c009a580>]
> >> >>>> >> > (tick_sched_handle+0x50/0x54)
> >> >>>> >> > [179922.192108] r5:c0907f10 r4:ef7abe40
> >> >>>> >> > [179922.195983] [<c009a530>] (tick_sched_handle) from [<c009a5d4>]
> >> >>>> >> > (tick_sched_timer+0x50/0x94)
> >> >>>> >> > [179922.204895] [<c009a584>] (tick_sched_timer) from [<c0089fe4>]
> >> >>>> >> > (__hrtimer_run_queues+0x110/0x1a0)
> >> >>>> >> > [179922.214324] r7:00000000 r6:ef7abc40 r5:ef7abe40 r4:ef7abc00
> >> >>>> >> > [179922.220428] [<c0089ed4>] (__hrtimer_run_queues) from [<c008a674>]
> >> >>>> >> > (hrtimer_interrupt+0xac/0x1f8)
> >> >>>> >> > [179922.227111] r10:ef7abc78 r9:ef7abc98 r8:ef7abc14 r7:ef7abcb8
> >> >>>> >> > r6:ffffffff r5:00000003
> >> >>>> >> > [179922.238220] r4:ef7abc00
> >> >>>> >> > [179922.238220] [<c008a5c8>] (hrtimer_interrupt) from [<c00170ec>]
> >> >>>> >> > (twd_handler+0x38/0x48)
> >> >>>> >> > [179922.238220] r10:c09084e8 r9:fa241100 r8:00000011 r7:ef028780
> >> >>>> >> > r6:c092574c r5:ef005cc0
> >> >>>> >
> >> >>>> > All interrupt stack up to this point.
> >> >>>> >
> >> >>>> > It is quite possible that the stuff below here is at fault as well.
> >> >>>> > That said, the grace-period should actually get to execute at some
> >> >>>> > point. Do you have a heavy real-time load that might be starving
> >> >>>> > things?
> >> >>>> >
> >> >>>> > Thanx, Paul
> >> >>>> >
> >> >>>> >> > [179922.257110] r4:00000001
> >> >>>> >> > [179922.257110] [<c00170b4>] (twd_handler) from [<c007c8f8>] (handle_percpu_devid_irq+0x74/0x8c)
> >> >>>> >> > [179922.269683] r5:ef005cc0 r4:ef7b1740
> >> >>>> >> > [179922.269683] [<c007c884>] (handle_percpu_devid_irq) from [<c0078454>] (generic_handle_irq+0x2c/0x3c)
> >> >>>> >> > [179922.283233] r9:fa241100 r8:ef008000 r7:00000001 r6:00000000
> >> >>>> >> > r5:00000000 r4:c09013e8
> >> >>>> >> > [179922.290985] [<c0078428>] (generic_handle_irq) from [<c007872c>] (__handle_domain_irq+0x64/0xbc)
> >> >>>> >> > [179922.300842] [<c00786c8>] (__handle_domain_irq) from [<c00094c0>]
> >> >>>> >> > (gic_handle_irq+0x50/0x90)
> >> >>>> >> > [179922.303222] r9:fa241100 r8:fa240100 r7:c0907f10 r6:fa24010c
> >> >>>> >> > r5:c09087a8 r4:c0925748
> >> >>>> >> > [179922.315216] [<c0009470>] (gic_handle_irq) from [<c0014bd4>]
> >> >>>> >> > (__irq_svc+0x54/0x90)
> >> >>>> >> > [179922.319000] Exception stack(0xc0907f10 to 0xc0907f58)
> >> >>>> >> > [179922.331542] 7f00: 00000000
> >> >>>> >> > ef7ab390 fe600000 00000000
> >> >>>> >> > [179922.331542] 7f20: c0906000 c090849c c0900364 c06a8124 c0907f80
> >> >>>> >> > c0944563 c09084e8 c0907f6c
> >> >>>> >> > [179922.349029] 7f40: c0907f4c c0907f60 c00287ac c0010ba8 60080113 ffffffff
> >> >>>> >> > [179922.349029] r9:c0944563 r8:c0907f80 r7:c0907f44 r6:ffffffff
> >> >>>> >> > r5:60080113 r4:c0010ba8
> >> >>>> >> > [179922.357116] [<c0010b80>] (arch_cpu_idle) from [<c006f034>]
> >> >>>> >> > (default_idle_call+0x28/0x34)
> >> >>>> >> > [179922.368926] [<c006f00c>] (default_idle_call) from [<c006f154>]
> >> >>>> >> > (cpu_startup_entry+0x114/0x18c)
> >> >>>> >> > [179922.368926] [<c006f040>] (cpu_startup_entry) from [<c069fc6c>]
> >> >>>> >> > (rest_init+0x90/0x94)
> >> >>>> >> > [179922.385284] r7:ffffffff r4:00000002
> >> >>>> >> > [179922.393463] [<c069fbdc>] (rest_init) from [<c08bbcec>]
> >> >>>> >> > (start_kernel+0x370/0x37c)
> >> >>>> >> > [179922.400421] r5:c0947000 r4:00000000
> >> >>>> >> > [179922.400421] [<c08bb97c>] (start_kernel) from [<8000807c>] (0x8000807c)
> >> >>>> >> > $
> >> >>>> >
> >> >>>> >
> >> >>>
> >> >>>
> >> >>>
> >
> >
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2016-01-16 0:40 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-12-11 11:17 rcu_preempt self-detected stall on CPU from 4.4-rc4, since 3.17 Ross Green
2015-12-11 11:30 ` Peter Zijlstra
2016-01-03 5:29 ` Ross Green
2016-01-03 6:17 ` Paul E. McKenney
2016-01-03 8:27 ` Ross Green
2016-01-03 18:15 ` Paul E. McKenney
2016-01-03 22:00 ` Ross Green
2016-01-04 14:21 ` Ross Green
2016-01-06 23:10 ` Ross Green
2016-01-08 18:35 ` Paul E. McKenney
2016-01-14 5:39 ` Ross Green
2016-01-16 0:39 ` Paul E. McKenney
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.