* ndctl hangs with big memmap=! fakepmem
@ 2021-10-20 13:23 Adam Borowski
2021-10-20 14:03 ` Adam Borowski
0 siblings, 1 reply; 4+ messages in thread
From: Adam Borowski @ 2021-10-20 13:23 UTC (permalink / raw)
To: nvdimm, Dan Williams, Vishal Verma
Hi!
After bumping fakepmem sizes from 4G!20G 4G!36G to 32G!20G 32G!192G,
ndctl hangs. Eg, at boot:
[ 725.642546] INFO: task ndctl:2486 blocked for more than 604 seconds.
[ 725.649586] Not tainted 5.15.0-rc6-vanilla-00020-gd9abdee5fd5a #1
[ 725.656877] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 725.665378] task:ndctl state:D stack: 0 pid: 2486 ppid: 2433 flags:0x00004000
[ 725.674404] Call Trace:
[ 725.677539] ? __schedule+0x30b/0x14e0
[ 725.681975] ? kernfs_put.part.0+0xd4/0x1a0
[ 725.686841] ? kmem_cache_free+0x28b/0x2b0
[ 725.691622] ? schedule+0x44/0xb0
[ 725.695622] ? blk_mq_freeze_queue_wait+0x62/0x90
[ 725.701009] ? do_wait_intr_irq+0xc0/0xc0
[ 725.705703] ? del_gendisk+0xcf/0x220
[ 725.710050] ? release_nodes+0x38/0xa0
[ 725.714485] ? devres_release_all+0x9f/0xe0
[ 725.719352] ? __device_release_driver+0x18a/0x240
[ 725.724823] ? device_driver_detach+0x4a/0xc0
[ 725.729862] ? unbind_store+0x117/0x130
[ 725.734379] ? kernfs_fop_write_iter+0x15a/0x1e0
[ 725.739677] ? new_sync_write+0x11f/0x1b0
[ 725.744368] ? vfs_write+0x1f5/0x2a0
[ 725.748627] ? do_sys_openat2+0x95/0x170
[ 725.753233] ? ksys_write+0x6d/0xf0
[ 725.757405] ? do_syscall_64+0x3b/0xc0
[ 725.761838] ? entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 725.767746] INFO: task ndctl:2520 blocked for more than 604 seconds.
[ 725.774779] Not tainted 5.15.0-rc6-vanilla-00020-gd9abdee5fd5a #1
[ 725.782069] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 725.790570] task:ndctl state:D stack: 0 pid: 2520 ppid: 1 flags:0x00000000
[ 725.799594] Call Trace:
[ 725.802725] ? __schedule+0x30b/0x14e0
[ 725.807158] ? __cond_resched+0x16/0x40
[ 725.811676] ? __cond_resched+0x16/0x40
[ 725.816193] ? kmem_cache_alloc_trace+0x10/0x3d0
[ 725.821492] ? schedule+0x44/0xb0
[ 725.825492] ? schedule_preempt_disabled+0xa/0x10
[ 725.830876] ? __mutex_lock.constprop.0+0x288/0x410
[ 725.836437] ? flush_namespaces+0x15/0x30
[ 725.841131] ? nvdimm_bus_unlock+0x20/0x20
[ 725.845911] ? device_for_each_child+0x57/0x90
[ 725.851035] ? flush_regions_dimms+0x3d/0x50
[ 725.855987] ? wait_probe_show+0x60/0x60
[ 725.860590] ? device_for_each_child+0x57/0x90
[ 725.865718] ? wait_probe_show+0x46/0x60
[ 725.870324] ? dev_attr_show+0x23/0x50
[ 725.874757] ? sysfs_kf_seq_show+0x9b/0xf0
[ 725.879534] ? seq_read_iter+0x10e/0x4b0
[ 725.884142] ? new_sync_read+0x118/0x1a0
[ 725.888745] ? vfs_read+0x120/0x1c0
[ 725.892921] ? do_sys_openat2+0x95/0x170
[ 725.897527] ? ksys_read+0x6d/0xf0
[ 725.901613] ? do_syscall_64+0x3b/0xc0
[ 725.906053] ? entry_SYSCALL_64_after_hwframe+0x44/0xae
[~]# ps axl|grep ndctl
0 0 2486 2433 20 0 6484 1896 - D ? 0:00 ndctl create-namespace -e namespace0.0 -m devdax -f
1 0 2520 1 20 0 6484 440 - Ds ? 0:00 /usr/bin/ndctl monitor --daemon
0 1000 6179 3707 20 0 6520 1912 - D+ pts/11 0:00 ndctl list
Stuff unrelated to pmem appears to work fine (the box is my personal
desktop), yet after the night I found the box unresponsive even to SysRq
(but able to switch text consoles); alas, I dun goofed and lost logs from
serial -- thus I can't tell the cause. Everything was stable beforehand
thus I assume it's related.
Full dmesg at https://angband.pl/tmp/logs/dmesg-valinor-20211020.log
Meow!
--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ in the beginning was the boot and root floppies and they were good.
⢿⡄⠘⠷⠚⠋⠀ -- <willmore>
⠈⠳⣄⠀⠀⠀⠀
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: ndctl hangs with big memmap=! fakepmem
2021-10-20 13:23 ndctl hangs with big memmap=! fakepmem Adam Borowski
@ 2021-10-20 14:03 ` Adam Borowski
2021-10-20 15:01 ` Scargall, Steve
0 siblings, 1 reply; 4+ messages in thread
From: Adam Borowski @ 2021-10-20 14:03 UTC (permalink / raw)
To: nvdimm, Dan Williams, Vishal Verma
On Wed, Oct 20, 2021 at 03:23:08PM +0200, Adam Borowski wrote:
> Hi!
> After bumping fakepmem sizes from 4G!20G 4G!36G to 32G!20G 32G!192G,
> ndctl hangs. Eg, at boot:
>
> [ 725.642546] INFO: task ndctl:2486 blocked for more than 604 seconds.
> [ 725.649586] Not tainted 5.15.0-rc6-vanilla-00020-gd9abdee5fd5a #1
> [ 725.677539] ? __schedule+0x30b/0x14e0
> [ 725.681975] ? kernfs_put.part.0+0xd4/0x1a0
> [ 725.686841] ? kmem_cache_free+0x28b/0x2b0
> [ 725.691622] ? schedule+0x44/0xb0
> [ 725.695622] ? blk_mq_freeze_queue_wait+0x62/0x90
> [ 725.701009] ? do_wait_intr_irq+0xc0/0xc0
> [ 725.705703] ? del_gendisk+0xcf/0x220
> [ 725.710050] ? release_nodes+0x38/0xa0
On 5.14.14 all is fine. Should I bisect?
Meow!
--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Remember, the S in "IoT" stands for Security, while P stands
⢿⡄⠘⠷⠚⠋⠀ for Privacy.
⠈⠳⣄⠀⠀⠀⠀
^ permalink raw reply [flat|nested] 4+ messages in thread
* RE: ndctl hangs with big memmap=! fakepmem
2021-10-20 14:03 ` Adam Borowski
@ 2021-10-20 15:01 ` Scargall, Steve
2021-10-21 1:38 ` Yi Zhang
0 siblings, 1 reply; 4+ messages in thread
From: Scargall, Steve @ 2021-10-20 15:01 UTC (permalink / raw)
To: Adam Borowski, nvdimm, Williams, Dan J, Verma, Vishal L
Hi Adam,
This is likely related to an issue that was reported to the Linux NVDIMM email list (https://lore.kernel.org/linux-block/CAHj4cs87BapQJcV0a=M6=dc9PrsGH6qzqJEt9fbjLK1aShnMPg@mail.gmail.com/)
So the bisecting shows it was introduced with below commit:
commit 8e141f9eb803e209714a80aa6ec073893f94c526
Author: Christoph Hellwig <hch@lst.de>
Date: Wed Sep 29 09:12:40 2021 +0200
block: drain file system I/O on del_gendisk
/Steve
-----Original Message-----
From: Adam Borowski <kilobyte@angband.pl>
Sent: Wednesday, October 20, 2021 8:04 AM
To: nvdimm@lists.linux.dev; Williams, Dan J <dan.j.williams@intel.com>; Verma, Vishal L <vishal.l.verma@intel.com>
Subject: Re: ndctl hangs with big memmap=! fakepmem
On Wed, Oct 20, 2021 at 03:23:08PM +0200, Adam Borowski wrote:
> Hi!
> After bumping fakepmem sizes from 4G!20G 4G!36G to 32G!20G 32G!192G,
> ndctl hangs. Eg, at boot:
>
> [ 725.642546] INFO: task ndctl:2486 blocked for more than 604 seconds.
> [ 725.649586] Not tainted 5.15.0-rc6-vanilla-00020-gd9abdee5fd5a #1
> [ 725.677539] ? __schedule+0x30b/0x14e0 [ 725.681975] ?
> kernfs_put.part.0+0xd4/0x1a0 [ 725.686841] ?
> kmem_cache_free+0x28b/0x2b0 [ 725.691622] ? schedule+0x44/0xb0 [
> 725.695622] ? blk_mq_freeze_queue_wait+0x62/0x90
> [ 725.701009] ? do_wait_intr_irq+0xc0/0xc0 [ 725.705703] ?
> del_gendisk+0xcf/0x220 [ 725.710050] ? release_nodes+0x38/0xa0
On 5.14.14 all is fine. Should I bisect?
Meow!
--
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Remember, the S in "IoT" stands for Security, while P stands ⢿⡄⠘⠷⠚⠋⠀ for Privacy.
⠈⠳⣄⠀⠀⠀⠀
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: ndctl hangs with big memmap=! fakepmem
2021-10-20 15:01 ` Scargall, Steve
@ 2021-10-21 1:38 ` Yi Zhang
0 siblings, 0 replies; 4+ messages in thread
From: Yi Zhang @ 2021-10-21 1:38 UTC (permalink / raw)
To: Scargall, Steve; +Cc: Adam Borowski, nvdimm, Williams, Dan J, Verma, Vishal L
On Wed, Oct 20, 2021 at 11:11 PM Scargall, Steve
<steve.scargall@intel.com> wrote:
>
> Hi Adam,
>
> This is likely related to an issue that was reported to the Linux NVDIMM email list (https://lore.kernel.org/linux-block/CAHj4cs87BapQJcV0a=M6=dc9PrsGH6qzqJEt9fbjLK1aShnMPg@mail.gmail.com/)
>
This issue was fixed by this
patchset:https://lore.kernel.org/nvdimm/20211019073641.2323410-1-hch@lst.de/
And they are still in the review process.
> So the bisecting shows it was introduced with below commit:
>
> commit 8e141f9eb803e209714a80aa6ec073893f94c526
> Author: Christoph Hellwig <hch@lst.de>
> Date: Wed Sep 29 09:12:40 2021 +0200
>
> block: drain file system I/O on del_gendisk
>
> /Steve
>
> -----Original Message-----
> From: Adam Borowski <kilobyte@angband.pl>
> Sent: Wednesday, October 20, 2021 8:04 AM
> To: nvdimm@lists.linux.dev; Williams, Dan J <dan.j.williams@intel.com>; Verma, Vishal L <vishal.l.verma@intel.com>
> Subject: Re: ndctl hangs with big memmap=! fakepmem
>
> On Wed, Oct 20, 2021 at 03:23:08PM +0200, Adam Borowski wrote:
> > Hi!
> > After bumping fakepmem sizes from 4G!20G 4G!36G to 32G!20G 32G!192G,
> > ndctl hangs. Eg, at boot:
> >
> > [ 725.642546] INFO: task ndctl:2486 blocked for more than 604 seconds.
> > [ 725.649586] Not tainted 5.15.0-rc6-vanilla-00020-gd9abdee5fd5a #1
>
> > [ 725.677539] ? __schedule+0x30b/0x14e0 [ 725.681975] ?
> > kernfs_put.part.0+0xd4/0x1a0 [ 725.686841] ?
> > kmem_cache_free+0x28b/0x2b0 [ 725.691622] ? schedule+0x44/0xb0 [
> > 725.695622] ? blk_mq_freeze_queue_wait+0x62/0x90
> > [ 725.701009] ? do_wait_intr_irq+0xc0/0xc0 [ 725.705703] ?
> > del_gendisk+0xcf/0x220 [ 725.710050] ? release_nodes+0x38/0xa0
>
> On 5.14.14 all is fine. Should I bisect?
>
>
> Meow!
> --
> ⢀⣴⠾⠻⢶⣦⠀
> ⣾⠁⢠⠒⠀⣿⡁ Remember, the S in "IoT" stands for Security, while P stands ⢿⡄⠘⠷⠚⠋⠀ for Privacy.
> ⠈⠳⣄⠀⠀⠀⠀
>
--
Best Regards,
Yi Zhang
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2021-10-21 1:38 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-10-20 13:23 ndctl hangs with big memmap=! fakepmem Adam Borowski
2021-10-20 14:03 ` Adam Borowski
2021-10-20 15:01 ` Scargall, Steve
2021-10-21 1:38 ` Yi Zhang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).