* hang in dax_pmem_compat_release on changing namespace mode
@ 2019-09-19 11:55 Adam Borowski
2019-09-19 15:10 ` Dan Williams
0 siblings, 1 reply; 5+ messages in thread
From: Adam Borowski @ 2019-09-19 11:55 UTC (permalink / raw)
To: linux-nvdimm, Dan Williams
Hi!
If I try to change the mode of a devdax namespace that's in use (mapped by
some process), ndctl hangs:
[ 9546.754673] INFO: task ndctl:3907 blocked for more than 1208 seconds.
[ 9546.754677] Not tainted 5.3.0-00048-g7f09b8bce091 #1
[ 9546.754679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9546.754681] ndctl D 0 3907 3856 0x00004004
[ 9546.754684] Call Trace:
[ 9546.754689] ? __schedule+0x281/0x670
[ 9546.754692] ? __switch_to_asm+0x34/0x70
[ 9546.754694] ? __switch_to_asm+0x34/0x70
[ 9546.754696] schedule+0x39/0xa0
[ 9546.754699] schedule_timeout+0x22b/0x320
[ 9546.754701] ? __switch_to_asm+0x34/0x70
[ 9546.754703] ? __switch_to_asm+0x40/0x70
[ 9546.754705] ? __switch_to_asm+0x34/0x70
[ 9546.754707] ? __switch_to+0x162/0x440
[ 9546.754710] ? apic_timer_interrupt+0xa/0x20
[ 9546.754712] wait_for_completion+0x100/0x150
[ 9546.754714] ? wake_up_q+0x60/0x60
[ 9546.754718] dev_pagemap_cleanup+0x47/0x60
[ 9546.754720] devm_memremap_pages_release+0xc5/0x220
[ 9546.754724] release_nodes+0x221/0x270
[ 9546.754728] dax_pmem_compat_release+0x30/0x50 [dax_pmem_compat]
[ 9546.754730] ? dax_pmem_compat_remove+0x20/0x20 [dax_pmem_compat]
[ 9546.754733] device_for_each_child+0x57/0x90
[ 9546.754736] dax_pmem_compat_remove+0x13/0x20 [dax_pmem_compat]
[ 9546.754739] nvdimm_bus_remove+0x4e/0xc0
[ 9546.754741] device_release_driver_internal+0xd8/0x1b0
[ 9546.754743] unbind_store+0xff/0x130
[ 9546.754746] kernfs_fop_write+0x140/0x1b0
[ 9546.754749] vfs_write+0xe4/0x1d0
[ 9546.754751] ksys_write+0x70/0x100
[ 9546.754754] do_syscall_64+0x50/0x100
[ 9546.754756] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 9546.754758] RIP: 0033:0x7f2d375dfad4
[ 9546.754762] Code: Bad RIP value.
[ 9546.754763] RSP: 002b:00007ffd61eca4e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 9546.754766] RAX: ffffffffffffffda RBX: 000055fb48f0982f RCX: 00007f2d375dfad4
[ 9546.754767] RDX: 0000000000000007 RSI: 000055fb48f0982f RDI: 0000000000000003
[ 9546.754769] RBP: 0000000000000007 R08: 00000000ffffffff R09: 00007ffd61eca3c0
[ 9546.754770] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000003
[ 9546.754771] R13: 00007f2d37166e70 R14: 0000000000000000 R15: 000055fb48f0c900
Fake pmem (memmap=4G!16G), the command is:
ndctl create-namespace -e namespace0.0 -m fsdax -f
-f is needed as a label-less fake pmem namespace is always active.
According to the man page, reconfiguring in that case is not allowed (duh),
and the operation is supposed to gracefully fail.
Meow!
--
⢀⣴⠾⠻⢶⣦⠀ A MAP07 (Dead Simple) raspberry tincture recipe: 0.5l 95% alcohol,
⣾⠁⢠⠒⠀⣿⡁ 1kg raspberries, 0.4kg sugar; put into a big jar for 1 month.
⢿⡄⠘⠷⠚⠋⠀ Filter out and throw away the fruits (can dump them into a cake,
⠈⠳⣄⠀⠀⠀⠀ etc), let the drink age at least 3-6 months.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: hang in dax_pmem_compat_release on changing namespace mode
2019-09-19 11:55 hang in dax_pmem_compat_release on changing namespace mode Adam Borowski
@ 2019-09-19 15:10 ` Dan Williams
2019-09-19 15:47 ` Adam Borowski
0 siblings, 1 reply; 5+ messages in thread
From: Dan Williams @ 2019-09-19 15:10 UTC (permalink / raw)
To: Adam Borowski; +Cc: linux-nvdimm
On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski <kilobyte@angband.pl> wrote:
>
> Hi!
> If I try to change the mode of a devdax namespace that's in use (mapped by
> some process), ndctl hangs:
Is it merely mapped, or might the pages be actively pinned / in use by
another part of the kernel? The kernel has no choice but to wait for
active page pins to drain. Can you get a stack trace of the process
with the dev-dax instance mapped?
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: hang in dax_pmem_compat_release on changing namespace mode
2019-09-19 15:10 ` Dan Williams
@ 2019-09-19 15:47 ` Adam Borowski
2019-09-19 15:50 ` Dan Williams
2019-09-19 15:50 ` Adam Borowski
0 siblings, 2 replies; 5+ messages in thread
From: Adam Borowski @ 2019-09-19 15:47 UTC (permalink / raw)
To: Dan Williams; +Cc: linux-nvdimm
On Thu, Sep 19, 2019 at 08:10:47AM -0700, Dan Williams wrote:
> On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski <kilobyte@angband.pl> wrote:
> > Hi!
> > If I try to change the mode of a devdax namespace that's in use (mapped by
> > some process), ndctl hangs:
>
> Is it merely mapped, or might the pages be actively pinned / in use by
> another part of the kernel? The kernel has no choice but to wait for
> active page pins to drain. Can you get a stack trace of the process
> with the dev-dax instance mapped?
Looks like the behaviour is different depending on what the other process
is:
* with qemu, the hang is 100% reproducible, the guest continues to work and
cleanly exits -- qemu does not exit on its own (unlike normal case) but
SIGTERM terminates it correctly. Thus, qemu is not stuck, only ndctl is.
* with mere mmap() (I've used vmemcache) ndctl allows
reconfiguring the namespace. No hang.
My way to start qemu is:
.----
#!/bin/sh
NET="-net bridge -net nic"
DISK=eoan-devdax.disk
exec qemu-system-x86_64 -enable-kvm -m 4096,slots=2,maxmem=16G -smp 8 $NET \
-drive if=none,id=hd,file="$DISK",format=raw,cache=unsafe,discard=on \
-device virtio-scsi-pci,id=scsi -device scsi-hd,drive=hd \
-M pc,nvdimm,nvdimm-persistence=mem-ctrl \
-object memory-backend-file,id=mem1,share=on,mem-path=/dev/dax0.0,size=4225761280,align=2M,pmem=on \
-device nvdimm,id=nvdimm1,memdev=mem1,label-size=256K \
-vnc :5
`----
Meow!
--
⢀⣴⠾⠻⢶⣦⠀ A MAP07 (Dead Simple) raspberry tincture recipe: 0.5l 95% alcohol,
⣾⠁⢠⠒⠀⣿⡁ 1kg raspberries, 0.4kg sugar; put into a big jar for 1 month.
⢿⡄⠘⠷⠚⠋⠀ Filter out and throw away the fruits (can dump them into a cake,
⠈⠳⣄⠀⠀⠀⠀ etc), let the drink age at least 3-6 months.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: hang in dax_pmem_compat_release on changing namespace mode
2019-09-19 15:47 ` Adam Borowski
@ 2019-09-19 15:50 ` Dan Williams
2019-09-19 15:50 ` Adam Borowski
1 sibling, 0 replies; 5+ messages in thread
From: Dan Williams @ 2019-09-19 15:50 UTC (permalink / raw)
To: Adam Borowski; +Cc: linux-nvdimm
On Thu, Sep 19, 2019 at 8:47 AM Adam Borowski <kilobyte@angband.pl> wrote:
>
> On Thu, Sep 19, 2019 at 08:10:47AM -0700, Dan Williams wrote:
> > On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski <kilobyte@angband.pl> wrote:
> > > Hi!
> > > If I try to change the mode of a devdax namespace that's in use (mapped by
> > > some process), ndctl hangs:
> >
> > Is it merely mapped, or might the pages be actively pinned / in use by
> > another part of the kernel? The kernel has no choice but to wait for
> > active page pins to drain. Can you get a stack trace of the process
> > with the dev-dax instance mapped?
>
> Looks like the behaviour is different depending on what the other process
> is:
> * with qemu, the hang is 100% reproducible, the guest continues to work and
> cleanly exits -- qemu does not exit on its own (unlike normal case) but
> SIGTERM terminates it correctly. Thus, qemu is not stuck, only ndctl is.
> * with mere mmap() (I've used vmemcache) ndctl allows
> reconfiguring the namespace. No hang.
>
> My way to start qemu is:
> .----
> #!/bin/sh
> NET="-net bridge -net nic"
> DISK=eoan-devdax.disk
>
> exec qemu-system-x86_64 -enable-kvm -m 4096,slots=2,maxmem=16G -smp 8 $NET \
> -drive if=none,id=hd,file="$DISK",format=raw,cache=unsafe,discard=on \
> -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=hd \
> -M pc,nvdimm,nvdimm-persistence=mem-ctrl \
> -object memory-backend-file,id=mem1,share=on,mem-path=/dev/dax0.0,size=4225761280,align=2M,pmem=on \
> -device nvdimm,id=nvdimm1,memdev=mem1,label-size=256K \
> -vnc :5
Ok, I'll take a look. At first glance nothing in that config should be
holding an indefinite page pin, so it does smell like a kernel bug.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: hang in dax_pmem_compat_release on changing namespace mode
2019-09-19 15:47 ` Adam Borowski
2019-09-19 15:50 ` Dan Williams
@ 2019-09-19 15:50 ` Adam Borowski
1 sibling, 0 replies; 5+ messages in thread
From: Adam Borowski @ 2019-09-19 15:50 UTC (permalink / raw)
To: Dan Williams; +Cc: linux-nvdimm
On Thu, Sep 19, 2019 at 05:47:08PM +0200, Adam Borowski wrote:
> On Thu, Sep 19, 2019 at 08:10:47AM -0700, Dan Williams wrote:
> > On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski <kilobyte@angband.pl> wrote:
> > > If I try to change the mode of a devdax namespace that's in use (mapped by
> > > some process), ndctl hangs:
> >
> > Is it merely mapped, or might the pages be actively pinned / in use by
> > another part of the kernel? The kernel has no choice but to wait for
> > active page pins to drain. Can you get a stack trace of the process
> > with the dev-dax instance mapped?
>
> Looks like the behaviour is different depending on what the other process
> is:
> * with qemu, the hang is 100% reproducible, the guest continues to work and
> cleanly exits -- qemu does not exit on its own (unlike normal case) but
> SIGTERM terminates it correctly. Thus, qemu is not stuck, only ndctl is.
Correction: not 100%. I just had qemu die with SIGBUS instead.
Meow!
--
⢀⣴⠾⠻⢶⣦⠀ A MAP07 (Dead Simple) raspberry tincture recipe: 0.5l 95% alcohol,
⣾⠁⢠⠒⠀⣿⡁ 1kg raspberries, 0.4kg sugar; put into a big jar for 1 month.
⢿⡄⠘⠷⠚⠋⠀ Filter out and throw away the fruits (can dump them into a cake,
⠈⠳⣄⠀⠀⠀⠀ etc), let the drink age at least 3-6 months.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-09-19 15:50 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-19 11:55 hang in dax_pmem_compat_release on changing namespace mode Adam Borowski
2019-09-19 15:10 ` Dan Williams
2019-09-19 15:47 ` Adam Borowski
2019-09-19 15:50 ` Dan Williams
2019-09-19 15:50 ` Adam Borowski
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).