6.7.11: Fails to hibernate - work queues still busy

6.7.11: Fails to hibernate - work queues still busy

[Martin Steigerwald]

Hi Thorsten, hi Greg,

6.7.9 + some bcachefs upgrade/downgrade fixes included in 6.7.11 works
okay. 6.7.11 fails. Two repeated attempts failed with a dmesg like this:

[  192.465064] r8169 0000:05:00.0 en1: Link is Down
[  193.246691] PM: hibernation: hibernation entry
[  193.622844] Filesystems sync: 0.097 seconds
[  193.623337] Freezing user space processes
[  193.627326] Freezing user space processes completed (elapsed 0.003 seconds)
[  193.627390] OOM killer disabled.
[  193.627889] PM: hibernation: Marking nosave pages: [mem 0x00000000-0x00000fff]
[  193.627933] PM: hibernation: Marking nosave pages: [mem 0x0009f000-0x000fffff]
[  193.627973] PM: hibernation: Marking nosave pages: [mem 0x09c00000-0x09d00fff]
[  193.628022] PM: hibernation: Marking nosave pages: [mem 0x09f00000-0x09f0ffff]
[  193.628063] PM: hibernation: Marking nosave pages: [mem 0xa22d1000-0xa22d1fff]
[  193.628099] PM: hibernation: Marking nosave pages: [mem 0xa22de000-0xa22dffff]
[  193.628135] PM: hibernation: Marking nosave pages: [mem 0xa22ed000-0xa22eefff]
[  193.628172] PM: hibernation: Marking nosave pages: [mem 0xa22ff000-0xa22fffff]
[  193.628210] PM: hibernation: Marking nosave pages: [mem 0xb9533000-0xb95c3fff]
[  193.628263] PM: hibernation: Marking nosave pages: [mem 0xbd9de000-0xcc3fdfff]
[  193.630011] PM: hibernation: Marking nosave pages: [mem 0xce000000-0xffffffff]
[  193.632545] PM: hibernation: Basic memory bitmaps created
[  193.639135] PM: hibernation: Preallocating image memory
[  195.755034] PM: hibernation: Allocated 2438707 pages for snapshot
[  195.755817] PM: hibernation: Allocated 9754828 kbytes in 2.11 seconds (4623.14 MB/s)
[  195.755842] Freezing remaining freezable tasks
[  215.764748] Freezing remaining freezable tasks failed after 20.009 seconds (0 tasks refusing to freeze, wq_busy=1):
[  215.764813] Showing freezable workqueues that are still busy:
[  215.764841] workqueue events_freezable: flags=0x4
[  215.764869]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=0 refcnt=2
[  215.764881]     inactive: pci_pme_list_scan
[  215.764895] workqueue usb_hub_wq: flags=0x4
[  215.764965]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2 refcnt=3
[  215.764974]     in-flight: 350:hub_event [usbcore] hub_event [usbcore]
[  215.765212] Restarting kernel threads ... done.
[  216.244833] PM: hibernation: Basic memory bitmaps freed
[  216.245961] OOM killer enabled.
[  216.246377] Restarting tasks ... done.
[  216.250708] thermal thermal_zone0: failed to read out thermal zone (-61)
[  216.252313] PM: hibernation: hibernation exit
[  216.276601] Generic FE-GE Realtek PHY r8169-0-200:00: attached PHY driver (mii_bus:phy_addr=r8169-0-200:00, irq=MAC)
[  216.871301] r8169 0000:02:00.0 en0: rtl_ep_ocp_read_cond == 0 (loop: 30, delay: 10000).
[  216.976901] r8169 0000:02:00.0 en0: Link is Down
[  217.003589] Generic FE-GE Realtek PHY r8169-0-500:00: attached PHY driver (mii_bus:phy_addr=r8169-0-500:00, irq=MAC)
[  217.169087] r8169 0000:05:00.0 en1: Link is Down
[  220.611547] r8169 0000:05:00.0 en1: Link is Up - 1Gbps/Full - flow control rx/tx

ThinkPad T14 Gen 1 with AMD Ryzen 4750U and 32 GiB of RAM.

Could that be related to the following issue?

* Hibernate stuck after recent kernel/workqueue.c changes in Stable 6.6.23
@ 2024-04-02  8:08 Linux regression tracking (Thorsten Leemhuis)

https://lore.kernel.org/regressions/ce4c2f67-c298-48a0-87a3-f933d646c73b@leemhuis.info/T/#u

However I did not find above work queue related error messages in the
dmesg in the bug tracker bug report mentioned there:

https://bugzilla.kernel.org/show_bug.cgi?id=218658

If really needed I could do a bisect, but it would take a while until I
can take time to do it.

Best,
-- 
Martin

Re: 6.7.11: Fails to hibernate - work queues still busy

[Martin Steigerwald]

Martin Steigerwald - 02.04.24, 21:29:50 CEST:
> 6.7.9 + some bcachefs upgrade/downgrade fixes included in 6.7.11 works
> okay. 6.7.11 fails. Two repeated attempts failed with a dmesg like this:

Also 6.7.11 appears not to be capable to reboot the machine. Runit says
it is rebooting and then it hangs there with no further output.

Which reminds me of:

* Re: [regression] 6.8.1: fails to hibernate with pm_runtime_force_suspend+0x0/0x120 returns -16
  2024-03-16 16:02 [regression] 6.8.1: fails to hibernate with pm_runtime_force_suspend+0x0/0x120 returns -16 Martin Steigerwald
[…]

https://lore.kernel.org/regressions/22240355.EfDdHjke4D@lichtvoll.de/T/#md9dd4c2abed8503c0613d39be3cdc833aadecd6d

I don't know whether those two relate as dmesg outputs differ.

The output from 6.8.2 on hibernation attempt

[  412.158400] port 0000:02:00.1:0.0: PM: dpm_run_callback(): pm_runtime_force_suspend+0x0/0x120 returns -16
[  412.158418] port 0000:02:00.1:0.0: PM: failed to freeze: error -16

versus the output 6.7.11 on hibernation attempt

[  215.764813] Showing freezable workqueues that are still busy:
[  215.764841] workqueue events_freezable: flags=0x4
[  215.764869]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=0 refcnt=2
[  215.764881]     inactive: pci_pme_list_scan
[  215.764895] workqueue usb_hub_wq: flags=0x4
[  215.764965]   pwq 4: cpus=2 node=0 flags=0x0 nice=0 active=2 refcnt=3
[  215.764974]     in-flight: 350:hub_event [usbcore] hub_event [usbcore]

does not really look related to me.

Yet both kernels do not seem to like to reboot the machine.

As written I am willing to bisect this 6.7.9 versus 6.7.11 issue, but not
6.8.1 versus some 6.7 issues cause I do not want to risk filesystem
corruption on a production machine by bisecting between stable and rc1.

Best,
-- 
Martin

Re: 6.7.11: Fails to hibernate - work queues still busy

[Linux regression tracking (Thorsten Leemhuis)]

On 02.04.24 22:03, Martin Steigerwald wrote:
> Martin Steigerwald - 02.04.24, 21:29:50 CEST:
> 
> As written I am willing to bisect this 6.7.9 versus 6.7.11 issue,

The last 6.7.y release is under review, so that likely is not worth it,
unless you are lucky and that way find the change that broke things in 6.8.

> but not
> 6.8.1 versus some 6.7 issues cause I do not want to risk filesystem
> corruption on a production machine by bisecting between stable and rc1.

As mentioned in the other mail just send: I don't think it's that risky
as you make it sound.

Ciao, Thorsten