hang in dax_pmem_compat_release on changing namespace mode

hang in dax_pmem_compat_release on changing namespace mode

[Adam Borowski]

Hi!
If I try to change the mode of a devdax namespace that's in use (mapped by
some process), ndctl hangs:

[ 9546.754673] INFO: task ndctl:3907 blocked for more than 1208 seconds.
[ 9546.754677]       Not tainted 5.3.0-00048-g7f09b8bce091 #1
[ 9546.754679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 9546.754681] ndctl           D    0  3907   3856 0x00004004
[ 9546.754684] Call Trace:
[ 9546.754689]  ? __schedule+0x281/0x670
[ 9546.754692]  ? __switch_to_asm+0x34/0x70
[ 9546.754694]  ? __switch_to_asm+0x34/0x70
[ 9546.754696]  schedule+0x39/0xa0
[ 9546.754699]  schedule_timeout+0x22b/0x320
[ 9546.754701]  ? __switch_to_asm+0x34/0x70
[ 9546.754703]  ? __switch_to_asm+0x40/0x70
[ 9546.754705]  ? __switch_to_asm+0x34/0x70
[ 9546.754707]  ? __switch_to+0x162/0x440
[ 9546.754710]  ? apic_timer_interrupt+0xa/0x20
[ 9546.754712]  wait_for_completion+0x100/0x150
[ 9546.754714]  ? wake_up_q+0x60/0x60
[ 9546.754718]  dev_pagemap_cleanup+0x47/0x60
[ 9546.754720]  devm_memremap_pages_release+0xc5/0x220
[ 9546.754724]  release_nodes+0x221/0x270
[ 9546.754728]  dax_pmem_compat_release+0x30/0x50 [dax_pmem_compat]
[ 9546.754730]  ? dax_pmem_compat_remove+0x20/0x20 [dax_pmem_compat]
[ 9546.754733]  device_for_each_child+0x57/0x90
[ 9546.754736]  dax_pmem_compat_remove+0x13/0x20 [dax_pmem_compat]
[ 9546.754739]  nvdimm_bus_remove+0x4e/0xc0
[ 9546.754741]  device_release_driver_internal+0xd8/0x1b0
[ 9546.754743]  unbind_store+0xff/0x130
[ 9546.754746]  kernfs_fop_write+0x140/0x1b0
[ 9546.754749]  vfs_write+0xe4/0x1d0
[ 9546.754751]  ksys_write+0x70/0x100
[ 9546.754754]  do_syscall_64+0x50/0x100
[ 9546.754756]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 9546.754758] RIP: 0033:0x7f2d375dfad4
[ 9546.754762] Code: Bad RIP value.
[ 9546.754763] RSP: 002b:00007ffd61eca4e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 9546.754766] RAX: ffffffffffffffda RBX: 000055fb48f0982f RCX: 00007f2d375dfad4
[ 9546.754767] RDX: 0000000000000007 RSI: 000055fb48f0982f RDI: 0000000000000003
[ 9546.754769] RBP: 0000000000000007 R08: 00000000ffffffff R09: 00007ffd61eca3c0
[ 9546.754770] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000003
[ 9546.754771] R13: 00007f2d37166e70 R14: 0000000000000000 R15: 000055fb48f0c900

Fake pmem (memmap=4G!16G), the command is:
    ndctl create-namespace -e namespace0.0 -m fsdax -f
-f is needed as a label-less fake pmem namespace is always active.

According to the man page, reconfiguring in that case is not allowed (duh),
and the operation is supposed to gracefully fail.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ A MAP07 (Dead Simple) raspberry tincture recipe: 0.5l 95% alcohol,
⣾⠁⢠⠒⠀⣿⡁ 1kg raspberries, 0.4kg sugar; put into a big jar for 1 month.
⢿⡄⠘⠷⠚⠋⠀ Filter out and throw away the fruits (can dump them into a cake,
⠈⠳⣄⠀⠀⠀⠀ etc), let the drink age at least 3-6 months.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: hang in dax_pmem_compat_release on changing namespace mode

[Dan Williams]

On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski <kilobyte@angband.pl> wrote:
>
> Hi!
> If I try to change the mode of a devdax namespace that's in use (mapped by
> some process), ndctl hangs:

Is it merely mapped, or might the pages be actively pinned / in use by
another part of the kernel? The kernel has no choice but to wait for
active page pins to drain. Can you get a stack trace of the process
with the dev-dax instance mapped?
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: hang in dax_pmem_compat_release on changing namespace mode

[Adam Borowski]

On Thu, Sep 19, 2019 at 08:10:47AM -0700, Dan Williams wrote:
> On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski <kilobyte@angband.pl> wrote:
> > Hi!
> > If I try to change the mode of a devdax namespace that's in use (mapped by
> > some process), ndctl hangs:
> 
> Is it merely mapped, or might the pages be actively pinned / in use by
> another part of the kernel? The kernel has no choice but to wait for
> active page pins to drain. Can you get a stack trace of the process
> with the dev-dax instance mapped?

Looks like the behaviour is different depending on what the other process
is:
* with qemu, the hang is 100% reproducible, the guest continues to work and
  cleanly exits -- qemu does not exit on its own (unlike normal case) but
  SIGTERM terminates it correctly.  Thus, qemu is not stuck, only ndctl is.
* with mere mmap() (I've used vmemcache) ndctl allows
  reconfiguring the namespace.  No hang.

My way to start qemu is:
.----
#!/bin/sh
NET="-net bridge -net nic"
DISK=eoan-devdax.disk

exec qemu-system-x86_64 -enable-kvm -m 4096,slots=2,maxmem=16G -smp 8 $NET \
 -drive if=none,id=hd,file="$DISK",format=raw,cache=unsafe,discard=on \
 -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=hd \
 -M pc,nvdimm,nvdimm-persistence=mem-ctrl \
 -object memory-backend-file,id=mem1,share=on,mem-path=/dev/dax0.0,size=4225761280,align=2M,pmem=on \
 -device nvdimm,id=nvdimm1,memdev=mem1,label-size=256K \
 -vnc :5
`----


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ A MAP07 (Dead Simple) raspberry tincture recipe: 0.5l 95% alcohol,
⣾⠁⢠⠒⠀⣿⡁ 1kg raspberries, 0.4kg sugar; put into a big jar for 1 month.
⢿⡄⠘⠷⠚⠋⠀ Filter out and throw away the fruits (can dump them into a cake,
⠈⠳⣄⠀⠀⠀⠀ etc), let the drink age at least 3-6 months.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: hang in dax_pmem_compat_release on changing namespace mode

[Dan Williams]

On Thu, Sep 19, 2019 at 8:47 AM Adam Borowski <kilobyte@angband.pl> wrote:
>
> On Thu, Sep 19, 2019 at 08:10:47AM -0700, Dan Williams wrote:
> > On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski <kilobyte@angband.pl> wrote:
> > > Hi!
> > > If I try to change the mode of a devdax namespace that's in use (mapped by
> > > some process), ndctl hangs:
> >
> > Is it merely mapped, or might the pages be actively pinned / in use by
> > another part of the kernel? The kernel has no choice but to wait for
> > active page pins to drain. Can you get a stack trace of the process
> > with the dev-dax instance mapped?
>
> Looks like the behaviour is different depending on what the other process
> is:
> * with qemu, the hang is 100% reproducible, the guest continues to work and
>   cleanly exits -- qemu does not exit on its own (unlike normal case) but
>   SIGTERM terminates it correctly.  Thus, qemu is not stuck, only ndctl is.
> * with mere mmap() (I've used vmemcache) ndctl allows
>   reconfiguring the namespace.  No hang.
>
> My way to start qemu is:
> .----
> #!/bin/sh
> NET="-net bridge -net nic"
> DISK=eoan-devdax.disk
>
> exec qemu-system-x86_64 -enable-kvm -m 4096,slots=2,maxmem=16G -smp 8 $NET \
>  -drive if=none,id=hd,file="$DISK",format=raw,cache=unsafe,discard=on \
>  -device virtio-scsi-pci,id=scsi -device scsi-hd,drive=hd \
>  -M pc,nvdimm,nvdimm-persistence=mem-ctrl \
>  -object memory-backend-file,id=mem1,share=on,mem-path=/dev/dax0.0,size=4225761280,align=2M,pmem=on \
>  -device nvdimm,id=nvdimm1,memdev=mem1,label-size=256K \
>  -vnc :5

Ok, I'll take a look. At first glance nothing in that config should be
holding an indefinite page pin, so it does smell like a kernel bug.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm

Re: hang in dax_pmem_compat_release on changing namespace mode

[Adam Borowski]

On Thu, Sep 19, 2019 at 05:47:08PM +0200, Adam Borowski wrote:
> On Thu, Sep 19, 2019 at 08:10:47AM -0700, Dan Williams wrote:
> > On Thu, Sep 19, 2019 at 4:56 AM Adam Borowski <kilobyte@angband.pl> wrote:
> > > If I try to change the mode of a devdax namespace that's in use (mapped by
> > > some process), ndctl hangs:
> > 
> > Is it merely mapped, or might the pages be actively pinned / in use by
> > another part of the kernel? The kernel has no choice but to wait for
> > active page pins to drain. Can you get a stack trace of the process
> > with the dev-dax instance mapped?
> 
> Looks like the behaviour is different depending on what the other process
> is:
> * with qemu, the hang is 100% reproducible, the guest continues to work and
>   cleanly exits -- qemu does not exit on its own (unlike normal case) but
>   SIGTERM terminates it correctly.  Thus, qemu is not stuck, only ndctl is.

Correction: not 100%.  I just had qemu die with SIGBUS instead.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ A MAP07 (Dead Simple) raspberry tincture recipe: 0.5l 95% alcohol,
⣾⠁⢠⠒⠀⣿⡁ 1kg raspberries, 0.4kg sugar; put into a big jar for 1 month.
⢿⡄⠘⠷⠚⠋⠀ Filter out and throw away the fruits (can dump them into a cake,
⠈⠳⣄⠀⠀⠀⠀ etc), let the drink age at least 3-6 months.
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm