* hang in dax_pmem_compat_release on changing namespace mode
@ 2019-09-19 11:55 Adam Borowski
2019-09-19 15:10 ` Dan Williams
0 siblings, 1 reply; 5+ messages in thread
From: Adam Borowski @ 2019-09-19 11:55 UTC (permalink / raw)
To: linux-nvdimm, Dan Williams
Hi!
If I try to change the mode of a devdax namespace that's in use (mapped by
some process), ndctl hangs:
[ 9546.754673] INFO: task ndctl:3907 blocked for more than 1208 seconds.
[ 9546.754677] Not tainted 5.3.0-00048-g7f09b8bce091 #1
[ 9546.754679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9546.754681] ndctl D 0 3907 3856 0x00004004
[ 9546.754684] Call Trace:
[ 9546.754689] ? __schedule+0x281/0x670
[ 9546.754692] ? __switch_to_asm+0x34/0x70
[ 9546.754694] ? __switch_to_asm+0x34/0x70
[ 9546.754696] schedule+0x39/0xa0
[ 9546.754699] schedule_timeout+0x22b/0x320
[ 9546.754701] ? __switch_to_asm+0x34/0x70
[ 9546.754703] ? __switch_to_asm+0x40/0x70
[ 9546.754705] ? __switch_to_asm+0x34/0x70
[ 9546.754707] ? __switch_to+0x162/0x440
[ 9546.754710] ? apic_timer_interrupt+0xa/0x20
[ 9546.754712] wait_for_completion+0x100/0x150
[ 9546.754714] ? wake_up_q+0x60/0x60
[ 9546.754718] dev_pagemap_cleanup+0x47/0x60
[ 9546.754720] devm_memremap_pages_release+0xc5/0x220
[ 9546.754724] release_nodes+0x221/0x270
[ 9546.754728] dax_pmem_compat_release+0x30/0x50 [dax_pmem_compat]
[ 9546.754730] ? dax_pmem_compat_remove+0x20/0x20 [dax_pmem_compat]
[ 9546.754733] device_for_each_child+0x57/0x90
[ 9546.754736] dax_pmem_compat_remove+0x13/0x20 [dax_pmem_compat]
[ 9546.754739] nvdimm_bus_remove+0x4e/0xc0
[ 9546.754741] device_release_driver_internal+0xd8/0x1b0
[ 9546.754743] unbind_store+0xff/0x130
[ 9546.754746] kernfs_fop_write+0x140/0x1b0
[ 9546.754749] vfs_write+0xe4/0x1d0
[ 9546.754751] ksys_write+0x70/0x100
[ 9546.754754] do_syscall_64+0x50/0x100
[ 9546.754756] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 9546.754758] RIP: 0033:0x7f2d375dfad4
[ 9546.754762] Code: Bad RIP value.
[ 9546.754763] RSP: 002b:00007ffd61eca4e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 9546.754766] RAX: ffffffffffffffda RBX: 000055fb48f0982f RCX: 00007f2d375dfad4
[ 9546.754767] RDX: 0000000000000007 RSI: 000055fb48f0982f RDI: 0000000000000003
[ 9546.754769] RBP: 0000000000000007 R08: 00000000ffffffff R09: 00007ffd61eca3c0
[ 9546.754770] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000003
[ 9546.754771] R13: 00007f2d37166e70 R14: 0000000000000000 R15: 000055fb48f0c900
Fake pmem (memmap=4G!16G), the command is:
ndctl create-namespace -e namespace0.0 -m fsdax -f
-f is needed as a label-less fake pmem namespace is always active.
According to the man page, reconfiguring in that case is not allowed (duh),
and the operation is supposed to gracefully fail.
Meow!
--
⢀⣴⠾⠻⢶⣦⠀ A MAP07 (Dead Simple) raspberry tincture recipe: 0.5l 95% alcohol,
⣾⠁⢠⠒⠀⣿⡁ 1kg raspberries, 0.4kg sugar; put into a big jar for 1 month.
⢿⡄⠘⠷⠚⠋⠀ Filter out and throw away the fruits (can dump them into a cake,
⠈⠳⣄⠀⠀⠀⠀ etc), let the drink age at least 3-6 months.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: hang in dax_pmem_compat_release on changing namespace mode 2019-09-19 11:55 hang in dax_pmem_compat_release on changing namespace mode Adam Borowski @ 2019-09-19 15:10 ` Dan Williams 2019-09-19 15:47 ` Adam Borowski 0 siblings, 1 reply; 5+ messages in thread From: Dan Williams @ 2019-09-19 15:10 UTC (permalink / raw) To: Adam Borowski; +Cc: linux-nvdimm On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski <kilobyte@angband.pl> wrote: > > Hi! > If I try to change the mode of a devdax namespace that's in use (mapped by > some process), ndctl hangs: Is it merely mapped, or might the pages be actively pinned / in use by another part of the kernel? The kernel has no choice but to wait for active page pins to drain. Can you get a stack trace of the process with the dev-dax instance mapped? _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: hang in dax_pmem_compat_release on changing namespace mode 2019-09-19 15:10 ` Dan Williams @ 2019-09-19 15:47 ` Adam Borowski 2019-09-19 15:50 ` Dan Williams 2019-09-19 15:50 ` Adam Borowski 0 siblings, 2 replies; 5+ messages in thread From: Adam Borowski @ 2019-09-19 15:47 UTC (permalink / raw) To: Dan Williams; +Cc: linux-nvdimm On Thu, Sep 19, 2019 at 08:10:47AM -0700, Dan Williams wrote: > On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski <kilobyte@angband.pl> wrote: > > Hi! > > If I try to change the mode of a devdax namespace that's in use (mapped by > > some process), ndctl hangs: > > Is it merely mapped, or might the pages be actively pinned / in use by > another part of the kernel? The kernel has no choice but to wait for > active page pins to drain. Can you get a stack trace of the process > with the dev-dax instance mapped? Looks like the behaviour is different depending on what the other process is: * with qemu, the hang is 100% reproducible, the guest continues to work and cleanly exits -- qemu does not exit on its own (unlike normal case) but SIGTERM terminates it correctly. Thus, qemu is not stuck, only ndctl is. * with mere mmap() (I've used vmemcache) ndctl allows reconfiguring the namespace. No hang. My way to start qemu is: .---- #!/bin/sh NET="-net bridge -net nic" DISK=eoan-devdax.disk exec qemu-system-x86_64 -enable-kvm -m 4096,slots=2,maxmem=16G -smp 8 $NET \ -drive if=none,id=hd,file="$DISK",format=raw,cache=unsafe,discard=on \ -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=hd \ -M pc,nvdimm,nvdimm-persistence=mem-ctrl \ -object memory-backend-file,id=mem1,share=on,mem-path=/dev/dax0.0,size=4225761280,align=2M,pmem=on \ -device nvdimm,id=nvdimm1,memdev=mem1,label-size=256K \ -vnc :5 `---- Meow! -- ⢀⣴⠾⠻⢶⣦⠀ A MAP07 (Dead Simple) raspberry tincture recipe: 0.5l 95% alcohol, ⣾⠁⢠⠒⠀⣿⡁ 1kg raspberries, 0.4kg sugar; put into a big jar for 1 month. ⢿⡄⠘⠷⠚⠋⠀ Filter out and throw away the fruits (can dump them into a cake, ⠈⠳⣄⠀⠀⠀⠀ etc), let the drink age at least 3-6 months. _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: hang in dax_pmem_compat_release on changing namespace mode 2019-09-19 15:47 ` Adam Borowski @ 2019-09-19 15:50 ` Dan Williams 2019-09-19 15:50 ` Adam Borowski 1 sibling, 0 replies; 5+ messages in thread From: Dan Williams @ 2019-09-19 15:50 UTC (permalink / raw) To: Adam Borowski; +Cc: linux-nvdimm On Thu, Sep 19, 2019 at 8:47 AM Adam Borowski <kilobyte@angband.pl> wrote: > > On Thu, Sep 19, 2019 at 08:10:47AM -0700, Dan Williams wrote: > > On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski <kilobyte@angband.pl> wrote: > > > Hi! > > > If I try to change the mode of a devdax namespace that's in use (mapped by > > > some process), ndctl hangs: > > > > Is it merely mapped, or might the pages be actively pinned / in use by > > another part of the kernel? The kernel has no choice but to wait for > > active page pins to drain. Can you get a stack trace of the process > > with the dev-dax instance mapped? > > Looks like the behaviour is different depending on what the other process > is: > * with qemu, the hang is 100% reproducible, the guest continues to work and > cleanly exits -- qemu does not exit on its own (unlike normal case) but > SIGTERM terminates it correctly. Thus, qemu is not stuck, only ndctl is. > * with mere mmap() (I've used vmemcache) ndctl allows > reconfiguring the namespace. No hang. > > My way to start qemu is: > .---- > #!/bin/sh > NET="-net bridge -net nic" > DISK=eoan-devdax.disk > > exec qemu-system-x86_64 -enable-kvm -m 4096,slots=2,maxmem=16G -smp 8 $NET \ > -drive if=none,id=hd,file="$DISK",format=raw,cache=unsafe,discard=on \ > -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=hd \ > -M pc,nvdimm,nvdimm-persistence=mem-ctrl \ > -object memory-backend-file,id=mem1,share=on,mem-path=/dev/dax0.0,size=4225761280,align=2M,pmem=on \ > -device nvdimm,id=nvdimm1,memdev=mem1,label-size=256K \ > -vnc :5 Ok, I'll take a look. At first glance nothing in that config should be holding an indefinite page pin, so it does smell like a kernel bug. _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: hang in dax_pmem_compat_release on changing namespace mode 2019-09-19 15:47 ` Adam Borowski 2019-09-19 15:50 ` Dan Williams @ 2019-09-19 15:50 ` Adam Borowski 1 sibling, 0 replies; 5+ messages in thread From: Adam Borowski @ 2019-09-19 15:50 UTC (permalink / raw) To: Dan Williams; +Cc: linux-nvdimm On Thu, Sep 19, 2019 at 05:47:08PM +0200, Adam Borowski wrote: > On Thu, Sep 19, 2019 at 08:10:47AM -0700, Dan Williams wrote: > > On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski <kilobyte@angband.pl> wrote: > > > If I try to change the mode of a devdax namespace that's in use (mapped by > > > some process), ndctl hangs: > > > > Is it merely mapped, or might the pages be actively pinned / in use by > > another part of the kernel? The kernel has no choice but to wait for > > active page pins to drain. Can you get a stack trace of the process > > with the dev-dax instance mapped? > > Looks like the behaviour is different depending on what the other process > is: > * with qemu, the hang is 100% reproducible, the guest continues to work and > cleanly exits -- qemu does not exit on its own (unlike normal case) but > SIGTERM terminates it correctly. Thus, qemu is not stuck, only ndctl is. Correction: not 100%. I just had qemu die with SIGBUS instead. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ A MAP07 (Dead Simple) raspberry tincture recipe: 0.5l 95% alcohol, ⣾⠁⢠⠒⠀⣿⡁ 1kg raspberries, 0.4kg sugar; put into a big jar for 1 month. ⢿⡄⠘⠷⠚⠋⠀ Filter out and throw away the fruits (can dump them into a cake, ⠈⠳⣄⠀⠀⠀⠀ etc), let the drink age at least 3-6 months. _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-09-19 15:50 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-09-19 11:55 hang in dax_pmem_compat_release on changing namespace mode Adam Borowski 2019-09-19 15:10 ` Dan Williams 2019-09-19 15:47 ` Adam Borowski 2019-09-19 15:50 ` Dan Williams 2019-09-19 15:50 ` Adam Borowski
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).