Reboot regression on Lager

Reboot regression on Lager

[Wolfram Sang]

[-- Attachment #1: Type: text/plain, Size: 6180 bytes --]

Hi,

I just discovered that my Lager can't reboot anymore. 5.10 is good,
current Linus' tree [1] is bad. I don't have time to debug this on my
own right now, so this is a reminder and notification for now.

[1] (614cb5894306 ("Merge tag 'acpi-5.11-rc1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm")

Happy hacking,

   Wolfram

===

[    8.280735] Unhandled fault: imprecise external abort (0x1406) at 0x00000000
[    8.287793] pgd = (ptrval)
[    8.290504] [00000000] *pgd=41d85835, *pte=00000000, *ppte=00000000
[    8.296794] Internal error: : 1406 [#1] ARM
[    8.300983] Modules linked in:
[    8.304044] CPU: 0 PID: 1173 Comm: init Tainted: G        W         5.10.0-12913-g614cb5894306 #1092
[    8.313187] Hardware name: Generic R-Car Gen2 (Flattened Device Tree)
[    8.319632] PC is at sh_mdio_ctrl+0x44/0x60
[    8.323831] LR is at sh_mmd_ctrl+0x20/0x24
[    8.327933] pc : [<c0424c04>]    lr : [<c0424c64>]    psr: 20000013
[    8.334203] sp : c1f93d18  ip : c1f93d38  fp : c1f93d34
[    8.339431] r10: 00000058  r9 : c1b55044  r8 : 00000001
[    8.344658] r7 : 00000001  r6 : 00000002  r5 : 00000000  r4 : c1b97a44
[    8.351189] r3 : f08a1320  r2 : 00000001  r1 : 00000002  r0 : c1b97a44
[    8.357720] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[    8.364863] Control: 10c53c7d  Table: 41b44059  DAC: 00000051
[    8.370611] Process init (pid: 1173, stack limit = 0x(ptrval))
[    8.376449] Stack: (0xc1f93d18 to 0xc1f94000)
[    8.380810] 3d00:                                                       c1b97a44 00000002
[    8.388997] 3d20: 00000020 0000001f c1f93d44 c1f93d38 c0424c64 c0424bcc c1f93d6c c1f93d48
[    8.397183] 3d40: c04227dc c0424c50 00000001 c1b97a44 00000001 c1f01000 c0b200ac c1b55044
[    8.405369] 3d60: c1f93d8c c1f93d70 c0422a30 c04227b0 c1f01000 c1f01000 00000001 0000001f
[    8.413555] 3d80: c1f93db4 c1f93d90 c041f224 c04229e4 c1f93db4 c1f93da0 c1f01458 c1f01000
[    8.421741] 3da0: 00000001 0000001f c1f93dd4 c1f93db8 c041f2b0 c041f1bc c1b55000 00000200
[    8.429926] 3dc0: c1f01470 ffffe000 c1f93de4 c1f93dd8 c0420118 c041f27c c1f93dfc c1f93de8
[    8.438112] 3de0: c0421224 c0420108 c1b55004 c1b55000 c1f93e0c c1f93e00 c041944c c04211ec
[    8.446298] 3e00: c1f93e1c c1f93e10 c041bee0 c0419424 c1f93e44 c1f93e20 c03d3578 c041bed4
[    8.454484] 3e20: 00000000 c0a0d0a8 00000000 4321fedc c0100264 c1f92000 c1f93e54 c1f93e48
[    8.462669] 3e40: c013dfb0 c03d3414 c1f93e6c c1f93e58 c013e04c c013df80 01234567 c0a0d0a8
[    8.470855] 3e60: c1f93f94 c1f93e70 c013e358 c013e03c 00000000 00000000 00000000 00000000
[    8.479041] 3e80: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    8.487226] 3ea0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    8.495412] 3ec0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    8.503597] 3ee0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    8.511783] 3f00: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    8.519968] 3f20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[    8.528154] 3f40: 00000000 00000000 00000000 c012851c c1f93f74 c1f93f60 c0128548 c01271f0
[    8.536340] 3f60: ffffe000 00000000 c1f93f8c c1f93f78 c0128810 c0a04248 00000000 00000000
[    8.544526] 3f80: 000bc7e0 00000058 c1f93fa4 c1f93f98 c013e468 c013e200 00000000 c1f93fa8
[    8.552711] 3fa0: c0100060 c013e45c 00000000 00000000 fee1dead 28121969 01234567 00000000
[    8.560897] 3fc0: 00000000 00000000 000bc7e0 00000058 000bc7e0 00000000 b6f7af6c 00000000
[    8.569082] 3fe0: 000bc31c be9f6d00 0007e82c b6f02928 60000010 fee1dead 00000000 00000000
[    8.577265] Backtrace: 
[    8.579715] [<c0424bc0>] (sh_mdio_ctrl) from [<c0424c64>] (sh_mmd_ctrl+0x20/0x24)
[    8.587218]  r7:0000001f r6:00000020 r5:00000002 r4:c1b97a44
[    8.592879] [<c0424c44>] (sh_mmd_ctrl) from [<c04227dc>] (mdiobb_cmd+0x38/0xa8)
[    8.600204] [<c04227a4>] (mdiobb_cmd) from [<c0422a30>] (mdiobb_read+0x58/0xdc)
[    8.607531]  r9:c1b55044 r8:c0b200ac r7:c1f01000 r6:00000001 r5:c1b97a44 r4:00000001
[    8.615278] [<c04229d8>] (mdiobb_read) from [<c041f224>] (__mdiobus_read+0x74/0xc0)
[    8.622952]  r7:0000001f r6:00000001 r5:c1f01000 r4:c1f01000
[    8.628613] [<c041f1b0>] (__mdiobus_read) from [<c041f2b0>] (mdiobus_read+0x40/0x54)
[    8.636370]  r7:0000001f r6:00000001 r5:c1f01000 r4:c1f01458
[    8.642031] [<c041f270>] (mdiobus_read) from [<c0420118>] (phy_read+0x1c/0x20)
[    8.649268]  r7:ffffe000 r6:c1f01470 r5:00000200 r4:c1b55000
[    8.654929] [<c04200fc>] (phy_read) from [<c0421224>] (kszphy_config_intr+0x44/0xb4)
[    8.662686] [<c04211e0>] (kszphy_config_intr) from [<c041944c>] (phy_disable_interrupts+0x34/0x40)
[    8.671665]  r5:c1b55000 r4:c1b55004
[    8.675241] [<c0419418>] (phy_disable_interrupts) from [<c041bee0>] (phy_shutdown+0x18/0x1c)
[    8.683698] [<c041bec8>] (phy_shutdown) from [<c03d3578>] (device_shutdown+0x170/0x200)
[    8.691721] [<c03d3408>] (device_shutdown) from [<c013dfb0>] (kernel_restart_prepare+0x3c/0x48)
[    8.700440]  r9:c1f92000 r8:c0100264 r7:4321fedc r6:00000000 r5:c0a0d0a8 r4:00000000
[    8.708187] [<c013df74>] (kernel_restart_prepare) from [<c013e04c>] (kernel_restart+0x1c/0x78)
[    8.716812] [<c013e030>] (kernel_restart) from [<c013e358>] (__do_sys_reboot+0x164/0x204)
[    8.725004]  r5:c0a0d0a8 r4:01234567
[    8.728579] [<c013e1f4>] (__do_sys_reboot) from [<c013e468>] (sys_reboot+0x18/0x1c)
[    8.736251]  r7:00000058 r6:000bc7e0 r5:00000000 r4:00000000
[    8.741912] [<c013e450>] (sys_reboot) from [<c0100060>] (ret_fast_syscall+0x0/0x58)
[    8.749582] Exception stack(0xc1f93fa8 to 0xc1f93ff0)
[    8.754639] 3fa0:                   00000000 00000000 fee1dead 28121969 01234567 00000000
[    8.762825] 3fc0: 00000000 00000000 000bc7e0 00000058 000bc7e0 00000000 b6f7af6c 00000000
[    8.771009] 3fe0: 000bc31c be9f6d00 0007e82c b6f02928
[    8.776066] Code: e5943004 e5935000 f57ff04f e3570000 (e5944004) 
[    8.782164] ---[ end trace db0ca13a19886fca ]---


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Reboot regression on Lager

[Geert Uytterhoeven]

Hi Wolfram,

On Wed, Dec 23, 2020 at 7:55 PM Wolfram Sang
<wsa+renesas@sang-engineering.com> wrote:
> I just discovered that my Lager can't reboot anymore. 5.10 is good,
> current Linus' tree [1] is bad. I don't have time to debug this on my
> own right now, so this is a reminder and notification for now.
>
> [1] (614cb5894306 ("Merge tag 'acpi-5.11-rc1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm")
>
> Happy hacking,
>
>    Wolfram
>
> ===
>
> [    8.280735] Unhandled fault: imprecise external abort (0x1406) at 0x00000000

That usually indicates that a module's registers are accessed,
while the module clock is disabled.

> [    8.287793] pgd = (ptrval)
> [    8.290504] [00000000] *pgd=41d85835, *pte=00000000, *ppte=00000000
> [    8.296794] Internal error: : 1406 [#1] ARM
> [    8.300983] Modules linked in:
> [    8.304044] CPU: 0 PID: 1173 Comm: init Tainted: G        W         5.10.0-12913-g614cb5894306 #1092
> [    8.313187] Hardware name: Generic R-Car Gen2 (Flattened Device Tree)
> [    8.319632] PC is at sh_mdio_ctrl+0x44/0x60
> [    8.323831] LR is at sh_mmd_ctrl+0x20/0x24

So sh_eth is already suspended at this point.

Can you reproduce it at will? Koelsch is still rebooting fine.
But I caught the following during s2ram once, which might be related:

 Disabling non-boot CPUs ...
 Enabling non-boot CPUs ...
+------------[ cut here ]------------
+WARNING: CPU: 0 PID: 21 at drivers/i2c/i2c-core.h:54 __i2c_transfer+0x464/0x4a0
+i2c i2c-6: Transfer while suspended
+CPU: 0 PID: 21 Comm: kworker/0:1 Not tainted
5.11.0-rc1-shmobile-00107-gcf9760aa181f #829
+Hardware name: Generic R-Car Gen2 (Flattened Device Tree)
+Workqueue: events_power_efficient sync_hw_clock
+[<c010dba4>] (unwind_backtrace) from [<c0109b28>] (show_stack+0x10/0x14)
+[<c0109b28>] (show_stack) from [<c07a120c>] (dump_stack+0x8c/0xa8)
+[<c07a120c>] (dump_stack) from [<c011c538>] (__warn+0xc0/0xec)
+[<c011c538>] (__warn) from [<c079a7bc>] (warn_slowpath_fmt+0x78/0xb0)
+[<c079a7bc>] (warn_slowpath_fmt) from [<c0566574>] (__i2c_transfer+0x464/0x4a0)
+[<c0566574>] (__i2c_transfer) from [<c0566608>] (i2c_transfer+0x58/0xf8)
+[<c0566608>] (i2c_transfer) from [<c0489f80>] (regmap_i2c_read+0x58/0x94)
+[<c0489f80>] (regmap_i2c_read) from [<c0485e00>] (_regmap_raw_read+0x108/0x1bc)
+[<c0485e00>] (_regmap_raw_read) from [<c0485ef8>] (_regmap_bus_read+0x44/0x68)
+[<c0485ef8>] (_regmap_bus_read) from [<c0484018>] (_regmap_read+0x84/0x100)
+[<c0484018>] (_regmap_read) from [<c0485444>] (_regmap_update_bits+0xa8/0xf4)
+[<c0485444>] (_regmap_update_bits) from [<c0485574>]
(_regmap_select_page+0xe4/0x100)
+[<c0485574>] (_regmap_select_page) from [<c0485664>]
(_regmap_raw_write_impl+0xd4/0x608)
+[<c0485664>] (_regmap_raw_write_impl) from [<c04863f4>]
(_regmap_raw_write+0xd8/0x114)
+[<c04863f4>] (_regmap_raw_write) from [<c0486488>] (regmap_raw_write+0x58/0x7c)
+[<c0486488>] (regmap_raw_write) from [<c04866cc>]
(regmap_bulk_write+0x118/0x13c)
+[<c04866cc>] (regmap_bulk_write) from [<c05605b4>]
(da9063_rtc_set_time+0x44/0x8c)
+[<c05605b4>] (da9063_rtc_set_time) from [<c055e428>] (rtc_set_time+0x8c/0x15c)
+[<c055e428>] (rtc_set_time) from [<c01872cc>] (sync_hw_clock+0x12c/0x210)
+[<c01872cc>] (sync_hw_clock) from [<c01337d0>] (process_one_work+0x1bc/0x2ac)
+[<c01337d0>] (process_one_work) from [<c0133b18>] (worker_thread+0x22c/0x2d0)
+[<c0133b18>] (worker_thread) from [<c01388a8>] (kthread+0x100/0x10c)
+[<c01388a8>] (kthread) from [<c0100150>] (ret_from_fork+0x14/0x24)
+Exception stack(0xc1195fb0 to 0xc1195ff8)
+5fa0:                                     00000000 00000000 00000000 00000000
+5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
+5fe0: 00000000 00000000 00000000 00000000 00000013 00000000
+---[ end trace 5d3a7a10ee0cec3d ]---
+da9063-rtc da9063-rtc: Failed to set RTC time data: -108
+da9063-rtc da9063-rtc: Failed to read RTC time data: -108
 CPU1 is up

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: Reboot regression on Lager

[Geert Uytterhoeven]

Hi Wolfram,

On Tue, Dec 29, 2020 at 9:40 AM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> On Wed, Dec 23, 2020 at 7:55 PM Wolfram Sang
> <wsa+renesas@sang-engineering.com> wrote:
> > I just discovered that my Lager can't reboot anymore. 5.10 is good,
> > current Linus' tree [1] is bad. I don't have time to debug this on my
> > own right now, so this is a reminder and notification for now.
> >
> > [1] (614cb5894306 ("Merge tag 'acpi-5.11-rc1-2' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm")

> But I caught the following during s2ram once, which might be related:
>
>  Disabling non-boot CPUs ...
>  Enabling non-boot CPUs ...
> +------------[ cut here ]------------
> +WARNING: CPU: 0 PID: 21 at drivers/i2c/i2c-core.h:54 __i2c_transfer+0x464/0x4a0
> +i2c i2c-6: Transfer while suspended

Unrelated.
https://lore.kernel.org/linux-renesas-soc/CAMuHMdVB9XMAaMDnKrRzkqvhFugrDGmj=00Vh5sDQT-idnA7DA@mail.gmail.com/

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds