linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Michael Nazzareno Trimarchi <michael@amarulasolutions.com>
To: LKML <linux-kernel@vger.kernel.org>, linux-rt-users@vger.kernel.org
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Subject: linux-rt stall on imx6dl on patch-5.4.22-rt13.patch
Date: Fri, 29 May 2020 10:08:01 +0200	[thread overview]
Message-ID: <CAOf5uw=DU4KdLXt=VEU+Uv3+W3jUMXE393FO3_v2smScR6Xf-Q@mail.gmail.com> (raw)

Hi all

I get this stack trace
echo 3 > /proc/sys/vm/drop_caches chrt -f 99 sysbench --test=cpu --num-threads=2

First of all, I need to know that if it is a valid use case because
was raised by the testing team and
how can I start and where to take a look at it

Best Regards
Michael

[  161.709260] 001: sh (480): drop_caches: 3
[  180.234358] 001: sched: RT throttling activated
[  180.283363] 001: fec 2188000.ethernet eth0: MDIO read timeout
[  180.283406] 001: ------------[ cut here ]------------
[  180.283416] 001: WARNING: CPU: 1 PID: 133 at
drivers/net/phy/phy.c:716 phy_er

                                               ror+0x24/0x64
[  180.283460] 001: Modules linked in:
[  180.283473] 001: CPU: 1 PID: 133 Comm: kworker/1:2 Tainted: G
 W

                          5.4.23-rt13 #1
[  180.283494] 001: Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[  180.283531] 001: Workqueue: events_power_efficient phy_state_machine
[  180.283550] 001: [<c0113880>] (unwind_backtrace) from [<c010d498>]
(show_stac

                         k+0x10/0x14)
[  180.283591] 001: [<c010d498>] (show_stack) from [<c0c95ee4>]
(dump_stack+0xd4

                               /0x108)
[  180.283624] 001: [<c0c95ee4>] (dump_stack) from [<c0129588>]
(__warn+0xc0/0x1

                               0c)
[  180.283657] 001: [<c0129588>] (__warn) from [<c0129998>]
(warn_slowpath_fmt+0

                                   x5c/0xc0)
[  180.283680] 001: [<c0129998>] (warn_slowpath_fmt) from [<c07b3fcc>]
(phy_erro

                        r+0x24/0x64)
[  180.283706] 001: [<c07b3fcc>] (phy_error) from [<c07b5198>]
(phy_state_machin

                                e+0x68/0x148)
[  180.283727] 001: [<c07b5198>] (phy_state_machine) from [<c014b25c>]
(process_

                        one_work+0x2c8/0x884)
[  180.283760] 001: [<c014b25c>] (process_one_work) from [<c014b844>]
(worker_th

                         read+0x2c/0x580)
[  180.283785] 001: [<c014b844>] (worker_thread) from [<c0153790>]
(kthread+0x13

                            4/0x148)
[  180.283815] 001: [<c0153790>] (kthread) from [<c01010bc>]
(ret_from_fork+0x14

                                  /0x38)
[  180.283836] 001: Exception stack(0xd8873fb0 to 0xd8873ff8)
[  180.283854] 001: 3fa0:                                     00000000
00000000

                        00000000 00000000
[  180.283874] 001: 3fc0: 00000000 00000000 00000000 00000000 00000000
00000000

                        00000000 00000000
[  180.283890] 001: 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  181.243070] 001: irq event stamp: 52613
[  181.243094] 001: hardirqs last  enabled at (52615): [<c0c9d23c>]
prb_unlock+0

                           x84/0x94
[  181.243134] 001: hardirqs last disabled at (52616): [<c0c9d138>]
prb_lock+0xb

                           4/0x134
[  181.243151] 001: softirqs last  enabled at (42468): [<c0ab1f80>]
addrconf_ver

                           ify_rtnl+0x20c/0x620
[  181.243180] 001: softirqs last disabled at (42461): [<c0ab1d88>]
addrconf_ver

                           ify_rtnl+0x14/0x620
[  181.243196] 001: ---[ end trace 0000000000000005 ]---
[  181.244845] 001: fec 2188000.ethernet eth0: Link is Down
[  182.243188] 001: fec 2188000.ethernet eth0: MDIO read timeout
[  205.253023] 001: rcu: INFO: rcu_preempt self-detected stall on CPU
[  205.253041] 001: rcu:        1-...!: (2599 ticks this GP)
idle=50a/0/0x1 softirq=0/0 fqs=0
[  205.253060] 001:     (t=2600 jiffies g=7097 q=301)
[  205.253061] 001: rcu: rcu_preempt kthread starved for 2600 jiffies!
g7097 f0x0 RCU_GP_WAIT_FQS(5) ->st

                        ate=0x0 ->cpu=0
[  205.253071] 001: rcu: RCU grace-period kthread stack dump:
[  205.253078] 001: rcu_preempt     R  running task        0    10
 2 0x00000000
[  205.253096] 001: [<c0cb23b4>] (__schedule) from [<c0cb2900>]
(schedule+0x50/0xf8)
[  205.253121] 001: [<c0cb2900>] (schedule) from [<c0cb5fe0>]
(schedule_timeout+0x1bc/0x39c)
[  205.253137] 001: [<c0cb5fe0>] (schedule_timeout) from [<c01ad90c>]
(rcu_gp_kthread+0x764/0x246c)
[  205.253160] 001: [<c01ad90c>] (rcu_gp_kthread) from [<c0153790>]
(kthread+0x134/0x148)
[  205.253176] 001: [<c0153790>] (kthread) from [<c01010bc>]
(ret_from_fork+0x14/0x38)
[  205.253190] 001: Exception stack(0xd80e7fb0 to 0xd80e7ff8)
[  205.253198] 001: 7fa0:                                     00000000
00000000 00000000 00000000
[  205.253208] 001: 7fc0: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[  205.253217] 001: 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  205.253231] 001: NMI backtrace for cpu 1
[  205.253238] 001: CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W
       5.4.23-rt13 #1
[  205.253246] 001: Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[  205.253253] 001: [<c0113880>] (unwind_backtrace) from [<c010d498>]
(show_stack+0x10/0x14)
[  205.253271] 001: [<c010d498>] (show_stack) from [<c0c95ee4>]
(dump_stack+0xd4/0x108)
[  205.253288] 001: [<c0c95ee4>] (dump_stack) from [<c0c9cc44>]
(nmi_cpu_backtrace+0x74/0xac)
[  205.253303] 001: [<c0c9cc44>] (nmi_cpu_backtrace) from [<c0c9cdc0>]
(nmi_trigger_cpumask_backtrace+0x1

                        44/0x1ac)
[  205.253315] 001: [<c0c9cdc0>] (nmi_trigger_cpumask_backtrace) from
[<c01b4678>] (rcu_dump_cpu_stacks+0

                         x98/0xd0)
[  205.253329] 001: [<c01b4678>] (rcu_dump_cpu_stacks) from
[<c01b35d4>] (rcu_sched_clock_irq+0x824/0xb48

                                   )
[  205.253344] 001: [<c01b35d4>] (rcu_sched_clock_irq) from
[<c01bc950>] (update_process_times+0x2c/0x60)
[  205.253361] 001: [<c01bc950>] (update_process_times) from
[<c01d31e0>] (tick_sched_timer+0x64/0xcc)
[  205.253376] 001: [<c01d31e0>] (tick_sched_timer) from [<c01bd9c8>]
(__hrtimer_run_queues+0x170/0x788)
[  205.253388] 001: [<c01bd9c8>] (__hrtimer_run_queues) from
[<c01bf378>] (hrtimer_interrupt+0x130/0x2d8)
[  205.253400] 001: [<c01bf378>] (hrtimer_interrupt) from [<c01128b4>]
(twd_handler+0x2c/0x38)
[  205.253413] 001: [<c01128b4>] (twd_handler) from [<c019ab28>]
(handle_percpu_devid_irq+0xf0/0x4dc)
[  205.253429] 001: [<c019ab28>] (handle_percpu_devid_irq) from
[<c01941d4>] (generic_handle_irq+0x20/0x3

                               4)
[  205.253446] 001: [<c01941d4>] (generic_handle_irq) from
[<c0194818>] (__handle_domain_irq+0x8c/0x128)
[  205.253457] 001: [<c0194818>] (__handle_domain_irq) from
[<c0550d6c>] (gic_handle_irq+0x4c/0xa8)
[  205.253473] 001: [<c0550d6c>] (gic_handle_irq) from [<c0101ab0>]
(__irq_svc+0x70/0xc4)
[  205.253483] 001: Exception stack(0xd80f9f10 to 0xd80f9f58)
[  205.253491] 001: 9f00:                                     00000001
00000006 00000000 d80ec600
[  205.253501] 001: 9f20: 00000000 c1318854 c1401768 ca083cee c9ae857c
0000002f dbbb2ab8 00000000
[  205.253510] 001: 9f40: 00000000 d80f9f60 c0187400 c08c60bc 200d0113 ffffffff
[  205.253516] 001: [<c0101ab0>] (__irq_svc) from [<c08c60bc>]
(cpuidle_enter_state+0x168/0x6c0)
[  205.253530] 001: [<c08c60bc>] (cpuidle_enter_state) from
[<c08c6650>] (cpuidle_enter+0x28/0x38)
[  205.253540] 001: [<c08c6650>] (cpuidle_enter) from [<c0164b00>]
(do_idle+0x258/0x2e4)
[  205.253553] 001: [<c0164b00>] (do_idle) from [<c0164ed8>]
(cpu_startup_entry+0x18/0x20)
[  205.253563] 001: [<c0164ed8>] (cpu_startup_entry) from [<1010296c>]
(0x1010296c)
[  227.234448] 000: mmc1: Timeout waiting for hardware interrupt.
[  227.234472] 000: mmc1: sdhci: ============ SDHCI REGISTER DUMP ===========
[  227.234486] 000: mmc1: sdhci: Sys addr:  0x18e7d800 | Version:  0x00000002
[  227.234502] 000: mmc1: sdhci: Blk size:  0x00000200 | Blk cnt:  0x00000000
[  227.234515] 000: mmc1: sdhci: Argument:  0x000570bc | Trn mode: 0x0000003b
[  227.234527] 000: mmc1: sdhci: Present:   0x01fd8008 | Host ctl: 0x00000013
[  227.234539] 000: mmc1: sdhci: Power:     0x00000002 | Blk gap:  0x00000080
[  227.234550] 000: mmc1: sdhci: Wake-up:   0x00000008 | Clock:    0x0000001f
[  227.234560] 000: mmc1: sdhci: Timeout:   0x0000008f | Int stat: 0x00000003
[  227.234570] 000: mmc1: sdhci: Int enab:  0x117f100b | Sig enab: 0x117f100b
[  227.234582] 000: mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000103
[  227.234593] 000: mmc1: sdhci: Caps:      0x07eb0000 | Caps_1:   0x0000a000
[  227.234606] 000: mmc1: sdhci: Cmd:       0x0000123a | Max curr: 0x00ffffff
[  227.234617] 000: mmc1: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0xffffffff
[  227.234629] 000: mmc1: sdhci: Resp[2]:   0x320f5903 | Resp[3]:  0x00000900
[  227.234640] 000: mmc1: sdhci: Host ctl2: 0x00000000
[  227.234650] 000: mmc1: sdhci: ADMA Err:  0x00000000 | ADMA Ptr: 0x2c047208
[  227.234661] 000: mmc1: sdhci: ============================================
[  283.293246] 001: rcu: INFO: rcu_preempt self-detected stall on CPU
[  283.293276] 001: rcu:        1-...!: (10403 ticks this GP)
idle=77a/1/0x40000002 softirq=0/0 fqs=0
[  283.293313] 001:     (t=10404 jiffies g=7097 q=827)
[  283.293316] 001: rcu: rcu_preempt kthread starved for 10404
jiffies! g7097 f0x0 RCU_GP_WAIT_FQS(5) ->s

                                tate=0x0 ->cpu=0
[  283.293336] 001: rcu: RCU grace-period kthread stack dump:
[  283.293350] 001: rcu_preempt     R  running task        0    10
 2 0x00000000
[  283.293381] 001: [<c0cb23b4>] (__schedule) from [<c0cb2900>]
(schedule+0x50/0xf8)
[  283.293430] 001: [<c0cb2900>] (schedule) from [<c0cb5fe0>]
(schedule_timeout+0x1bc/0x39c)
[  283.293460] 001: [<c0cb5fe0>] (schedule_timeout) from [<c01ad90c>]
(rcu_gp_kthread+0x764/0x246c)
[  283.293500] 001: [<c01ad90c>] (rcu_gp_kthread) from [<c0153790>]
(kthread+0x134/0x148)
[  283.293533] 001: [<c0153790>] (kthread) from [<c01010bc>]
(ret_from_fork+0x14/0x38)
[  283.293556] 001: Exception stack(0xd80e7fb0 to 0xd80e7ff8)
[  283.293574] 001: 7fa0:                                     00000000
00000000 00000000 00000000
[  283.293593] 001: 7fc0: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[  283.293610] 001: 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  283.293634] 001: NMI backtrace for cpu 1
[  283.293647] 001: CPU: 1 PID: 484 Comm: sysbench Tainted: G        W
        5.4.23-rt13 #1
[  283.293662] 001: Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[  283.293676] 001: [<c0113880>] (unwind_backtrace) from [<c010d498>]
(show_stack+0x10/0x14)
[  283.293713] 001: [<c010d498>] (show_stack) from [<c0c95ee4>]
(dump_stack+0xd4/0x108)
[  283.293743] 001: [<c0c95ee4>] (dump_stack) from [<c0c9cc44>]
(nmi_cpu_backtrace+0x74/0xac)
[  283.293772] 001: [<c0c9cc44>] (nmi_cpu_backtrace) from [<c0c9cdc0>]
(nmi_trigger_cpumask_backtrace+0x1

                        44/0x1ac)
[  283.293795] 001: [<c0c9cdc0>] (nmi_trigger_cpumask_backtrace) from
[<c01b4678>] (rcu_dump_cpu_stacks+0

                         x98/0xd0)
[  283.293829] 001: [<c01b4678>] (rcu_dump_cpu_stacks) from
[<c01b35d4>] (rcu_sched_clock_irq+0x824/0xb48

                                   )
[  283.293852] 001: [<c01b35d4>] (rcu_sched_clock_irq) from
[<c01bc950>] (update_process_times+0x2c/0x60)
[  283.293887] 001: [<c01bc950>] (update_process_times) from
[<c01d31e0>] (tick_sched_timer+0x64/0xcc)
[  283.293914] 001: [<c01d31e0>] (tick_sched_timer) from [<c01bd9c8>]
(__hrtimer_run_queues+0x170/0x788)
[  283.293939] 001: [<c01bd9c8>] (__hrtimer_run_queues) from
[<c01bf378>] (hrtimer_interrupt+0x130/0x2d8)
[  283.293962] 001: [<c01bf378>] (hrtimer_interrupt) from [<c01128b4>]
(twd_handler+0x2c/0x38)
[  283.293988] 001: [<c01128b4>] (twd_handler) from [<c019ab28>]
(handle_percpu_devid_irq+0xf0/0x4dc)
[  283.294016] 001: [<c019ab28>] (handle_percpu_devid_irq) from
[<c01941d4>] (generic_handle_irq+0x20/0x3

                               4)
[  283.294045] 001: [<c01941d4>] (generic_handle_irq) from
[<c0194818>] (__handle_domain_irq+0x8c/0x128)
[  283.294069] 001: [<c0194818>] (__handle_domain_irq) from
[<c0550d6c>] (gic_handle_irq+0x4c/0xa8)
[  283.294096] 001: [<c0550d6c>] (gic_handle_irq) from [<c0101ef4>]
(__irq_usr+0x54/0x80)
[  283.294116] 001: Exception stack(0xd8da7fb0 to 0xd8da7ff8)
[  283.294134] 001: 7fa0:                                     00000332
00000000 000010c0 00000004
[  283.294154] 001: 7fc0: 00000860 00000000 00000006 b6f42d68 00000199
00000000 00002710 00000666
[  283.294174] 001: 7fe0: 00000000 b6f42d44 004547bd 004548f0 800d0030 ffffffff
[  361.323392] 001: rcu: INFO: rcu_preempt self-detected stall on CPU
[  361.323420] 001: rcu:        1-...!: (18206 ticks this GP)
idle=a96/1/0x40000002 softirq=0/0 fqs=0
[  361.323456] 001:     (t=18207 jiffies g=7097 q=1165)
[  361.323459] 001: rcu: rcu_preempt kthread starved for 18207
jiffies! g7097 f0x0 RCU_GP_WAIT_FQS(5) ->s

                                tate=0x0 ->cpu=0
[  361.323479] 001: rcu: RCU grace-period kthread stack dump:
[  361.323491] 001: rcu_preempt     R  running task        0    10
 2 0x00000000
[  361.323523] 001: [<c0cb23b4>] (__schedule) from [<c0cb2900>]
(schedule+0x50/0xf8)
[  361.323571] 001: [<c0cb2900>] (schedule) from [<c0cb5fe0>]
(schedule_timeout+0x1bc/0x39c)
[  361.323600] 001: [<c0cb5fe0>] (schedule_timeout) from [<c01ad90c>]
(rcu_gp_kthread+0x764/0x246c)
[  361.323639] 001: [<c01ad90c>] (rcu_gp_kthread) from [<c0153790>]
(kthread+0x134/0x148)
[  361.323671] 001: [<c0153790>] (kthread) from [<c01010bc>]
(ret_from_fork+0x14/0x38)
[  361.323692] 001: Exception stack(0xd80e7fb0 to 0xd80e7ff8)
[  361.323711] 001: 7fa0:                                     00000000
00000000 00000000 00000000
[  361.323730] 001: 7fc0: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[  361.323747] 001: 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[  361.323770] 001: NMI backtrace for cpu 1
[  361.323783] 001: CPU: 1 PID: 484 Comm: sysbench Tainted: G        W
        5.4.23-rt13 #1
[  361.323798] 001: Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[  361.323812] 001: [<c0113880>] (unwind_backtrace) from [<c010d498>]
(show_stack+0x10/0x14)
[  361.323850] 001: [<c010d498>] (show_stack) from [<c0c95ee4>]
(dump_stack+0xd4/0x108)
[  361.323879] 001: [<c0c95ee4>] (dump_stack) from [<c0c9cc44>]
(nmi_cpu_backtrace+0x74/0xac)
[  361.323906] 001: [<c0c9cc44>] (nmi_cpu_backtrace) from [<c0c9cdc0>]
(nmi_trigger_cpumask_backtrace+0x1

                        44/0x1ac)
[  361.323929] 001: [<c0c9cdc0>] (nmi_trigger_cpumask_backtrace) from
[<c01b4678>] (rcu_dump_cpu_stacks+0

                         x98/0xd0)
[  361.323960] 001: [<c01b4678>] (rcu_dump_cpu_stacks) from
[<c01b35d4>] (rcu_sched_clock_irq+0x824/0xb48

                                   )
[  361.323985] 001: [<c01b35d4>] (rcu_sched_clock_irq) from
[<c01bc950>] (update_process_times+0x2c/0x60)
[  361.324018] 001: [<c01bc950>] (update_process_times) from
[<c01d31e0>] (tick_sched_timer+0x64/0xcc)
[  361.324047] 001: [<c01d31e0>] (tick_sched_timer) from [<c01bd9c8>]
(__hrtimer_run_queues+0x170/0x788)
[  361.324070] 001: [<c01bd9c8>] (__hrtimer_run_queues) from
[<c01bf378>] (hrtimer_interrupt+0x130/0x2d8)
[  361.324094] 001: [<c01bf378>] (hrtimer_interrupt) from [<c01128b4>]
(twd_handler+0x2c/0x38)
[  361.324119] 001: [<c01128b4>] (twd_handler) from [<c019ab28>]
(handle_percpu_devid_irq+0xf0/0x4dc)
[  361.324146] 001: [<c019ab28>] (handle_percpu_devid_irq) from
[<c01941d4>] (generic_handle_irq+0x20/0x3

                               4)
[  361.324175] 001: [<c01941d4>] (generic_handle_irq) from
[<c0194818>] (__handle_domain_irq+0x8c/0x128)
[  361.324200] 001: [<c0194818>] (__handle_domain_irq) from
[<c0550d6c>] (gic_handle_irq+0x4c/0xa8)
[  361.324227] 001: [<c0550d6c>] (gic_handle_irq) from [<c0101ef4>]
(__irq_usr+0x54/0x80)
[  361.324249] 001: Exception stack(0xd8da7fb0 to 0xd8da7ff8)
[  361.324264] 001: 7fa0:                                     00000e76
00000000 00000000 00000000
[  361.324283] 001: 7fc0: 00000800 00000000 0000000a b6f42d68 0000073b
00000000 00002710 00000003
[  361.324301] 001: 7fe0: 00000000 b6f42d44 004547bd 00454918 600d0030 ffffffff

             reply	other threads:[~2020-05-29  8:08 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-29  8:08 Michael Nazzareno Trimarchi [this message]
2020-05-29 13:31 ` linux-rt stall on imx6dl on patch-5.4.22-rt13.patch Sebastian Andrzej Siewior
2020-05-29 13:46   ` Michael Nazzareno Trimarchi
2020-05-29 14:48     ` Sebastian Andrzej Siewior

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CAOf5uw=DU4KdLXt=VEU+Uv3+W3jUMXE393FO3_v2smScR6Xf-Q@mail.gmail.com' \
    --to=michael@amarulasolutions.com \
    --cc=bigeasy@linutronix.de \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rt-users@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).