* multi-threads libvmmalloc fork test hang @ 2016-10-27 11:22 ` Xiong Zhou 0 siblings, 0 replies; 14+ messages in thread From: Xiong Zhou @ 2016-10-27 11:22 UTC (permalink / raw) To: linux-nvdimm-y27Ovi1pjclAfugRpC6u6w, jack-AlSwsSmVLrQ Cc: linux-fsdevel-u79uwXL29TY76Z2rM5mHXA, linux-kernel-u79uwXL29TY76Z2rM5mHXA # description nvml test suite vmmalloc_fork test hang. $ ps -eo stat,comm | grep vmma S+ vmmalloc_fork Sl+ vmmalloc_fork Z+ vmmalloc_fork <defunct> Sl+ vmmalloc_fork Z+ vmmalloc_fork <defunct> Z+ vmmalloc_fork <defunct> Sl+ vmmalloc_fork Z+ vmmalloc_fork <defunct> Z+ vmmalloc_fork <defunct> Z+ vmmalloc_fork <defunct> dmesg: [ 250.499097] INFO: task vmmalloc_fork:9805 blocked for more than 120 seconds. [ 250.530667] Not tainted 4.9.09fe68ca+ #27 [ 250.550901] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 250.585752] vmmalloc_fork D[ 250.598362] ffffffff8171813c 0 9805 9765 0x00000080 [ 250.623445] ffff88075dc68f80[ 250.636052] 0000000000000000 ffff88076058db00 ffff88017c5b0000 ffff880763b19340[ 250.668510] ffffc9000fe1bbb0 ffffffff8171813c ffffc9000fe1bc20 ffffc9000fe1bbe0[ 250.704220] ffffffff82248898 ffff88076058db00 ffffffff82248898Call Trace: [ 250.738382] [<ffffffff8171813c>] ? __schedule+0x21c/0x6a0 [ 250.763404] [<ffffffff817185f6>] schedule+0x36/0x80 [ 250.786177] [<ffffffff81284471>] get_unlocked_mapping_entry+0xc1/0x120 [ 250.815869] [<ffffffff81283810>] ? iomap_dax_rw+0x110/0x110 [ 250.841350] [<ffffffff81284c0a>] grab_mapping_entry+0x4a/0x220 [ 250.868442] [<ffffffff812851e9>] iomap_dax_fault+0xa9/0x3b0 [ 250.894437] [<ffffffffa02b15fe>] xfs_filemap_fault+0xce/0xf0 [xfs] [ 250.922805] [<ffffffff811d3159>] __do_fault+0x79/0x100 [ 250.947035] [<ffffffff811d7a2b>] do_fault+0x49b/0x690 [ 250.970964] [<ffffffffa02b146c>] ? xfs_filemap_pmd_fault+0x9c/0x160 [xfs] [ 251.001812] [<ffffffff811d94ba>] handle_mm_fault+0x61a/0xa50 [ 251.027736] [<ffffffff8106c3da>] __do_page_fault+0x22a/0x4a0 [ 251.053700] [<ffffffff8106c680>] do_page_fault+0x30/0x80 [ 251.077962] [<ffffffff81003b55>] ? do_syscall_64+0x175/0x180 [ 251.103835] [<ffffffff8171e208>] page_fault+0x28/0x30 # kernel versions: v4.6 pass in seconds v4.7 hang v4.9-rc1 hang Linus tree to commit 9fe68ca hang bisect points to first bad commit: [ac401cc782429cc8560ce4840b1405d603740917] dax: New fault locking v4.7 with these 3 commits reverted pass: 4d9a2c8 - Jan Kara, 6 months ago : dax: Remove i_mmap_lock protection bc2466e - Jan Kara, 6 months ago : dax: Use radix tree entry lock to protect cow faults ac401cc - Jan Kara, 6 months ago : dax: New fault locking # nvml version: https://github.com/pmem/nvml.git to commit: feab4d6f65102139ce460890c898fcad09ce20ae # How reproducible: always # Test steps: <git clone and pmem0 setup> $cd nvml $make install -j64 $cat > src/test/testconfig.sh <<EOF PMEM_FS_DIR=/daxmnt NON_PMEM_FS_DIR=/tmp EOF $mkfs.xfs /dev/pmem0 $mkdir -p /daxmnt/ $mount -o dax /dev/pmem0 /daxmnt/ $make -C src/test/vmmalloc_fork/ TEST_TIME=60m clean $make -C src/test/vmmalloc_fork/ TEST_TIME=60m check $umount /daxmnt ^ permalink raw reply [flat|nested] 14+ messages in thread
* multi-threads libvmmalloc fork test hang @ 2016-10-27 11:22 ` Xiong Zhou 0 siblings, 0 replies; 14+ messages in thread From: Xiong Zhou @ 2016-10-27 11:22 UTC (permalink / raw) To: linux-nvdimm, jack; +Cc: linux-fsdevel, linux-kernel # description nvml test suite vmmalloc_fork test hang. $ ps -eo stat,comm | grep vmma S+ vmmalloc_fork Sl+ vmmalloc_fork Z+ vmmalloc_fork <defunct> Sl+ vmmalloc_fork Z+ vmmalloc_fork <defunct> Z+ vmmalloc_fork <defunct> Sl+ vmmalloc_fork Z+ vmmalloc_fork <defunct> Z+ vmmalloc_fork <defunct> Z+ vmmalloc_fork <defunct> dmesg: [ 250.499097] INFO: task vmmalloc_fork:9805 blocked for more than 120 seconds. [ 250.530667] Not tainted 4.9.09fe68ca+ #27 [ 250.550901] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 250.585752] vmmalloc_fork D[ 250.598362] ffffffff8171813c 0 9805 9765 0x00000080 [ 250.623445] ffff88075dc68f80[ 250.636052] 0000000000000000 ffff88076058db00 ffff88017c5b0000 ffff880763b19340[ 250.668510] ffffc9000fe1bbb0 ffffffff8171813c ffffc9000fe1bc20 ffffc9000fe1bbe0[ 250.704220] ffffffff82248898 ffff88076058db00 ffffffff82248898Call Trace: [ 250.738382] [<ffffffff8171813c>] ? __schedule+0x21c/0x6a0 [ 250.763404] [<ffffffff817185f6>] schedule+0x36/0x80 [ 250.786177] [<ffffffff81284471>] get_unlocked_mapping_entry+0xc1/0x120 [ 250.815869] [<ffffffff81283810>] ? iomap_dax_rw+0x110/0x110 [ 250.841350] [<ffffffff81284c0a>] grab_mapping_entry+0x4a/0x220 [ 250.868442] [<ffffffff812851e9>] iomap_dax_fault+0xa9/0x3b0 [ 250.894437] [<ffffffffa02b15fe>] xfs_filemap_fault+0xce/0xf0 [xfs] [ 250.922805] [<ffffffff811d3159>] __do_fault+0x79/0x100 [ 250.947035] [<ffffffff811d7a2b>] do_fault+0x49b/0x690 [ 250.970964] [<ffffffffa02b146c>] ? xfs_filemap_pmd_fault+0x9c/0x160 [xfs] [ 251.001812] [<ffffffff811d94ba>] handle_mm_fault+0x61a/0xa50 [ 251.027736] [<ffffffff8106c3da>] __do_page_fault+0x22a/0x4a0 [ 251.053700] [<ffffffff8106c680>] do_page_fault+0x30/0x80 [ 251.077962] [<ffffffff81003b55>] ? do_syscall_64+0x175/0x180 [ 251.103835] [<ffffffff8171e208>] page_fault+0x28/0x30 # kernel versions: v4.6 pass in seconds v4.7 hang v4.9-rc1 hang Linus tree to commit 9fe68ca hang bisect points to first bad commit: [ac401cc782429cc8560ce4840b1405d603740917] dax: New fault locking v4.7 with these 3 commits reverted pass: 4d9a2c8 - Jan Kara, 6 months ago : dax: Remove i_mmap_lock protection bc2466e - Jan Kara, 6 months ago : dax: Use radix tree entry lock to protect cow faults ac401cc - Jan Kara, 6 months ago : dax: New fault locking # nvml version: https://github.com/pmem/nvml.git to commit: feab4d6f65102139ce460890c898fcad09ce20ae # How reproducible: always # Test steps: <git clone and pmem0 setup> $cd nvml $make install -j64 $cat > src/test/testconfig.sh <<EOF PMEM_FS_DIR=/daxmnt NON_PMEM_FS_DIR=/tmp EOF $mkfs.xfs /dev/pmem0 $mkdir -p /daxmnt/ $mount -o dax /dev/pmem0 /daxmnt/ $make -C src/test/vmmalloc_fork/ TEST_TIME=60m clean $make -C src/test/vmmalloc_fork/ TEST_TIME=60m check $umount /daxmnt ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <20161027112230.wsumgs62fqdxt3sc-obHvtwIU90hQcClZ3XN9yxcY2uh10dtjAL8bYrjMMd8@public.gmane.org>]
* Re: multi-threads libvmmalloc fork test hang 2016-10-27 11:22 ` Xiong Zhou @ 2016-10-27 13:37 ` Jan Kara -1 siblings, 0 replies; 14+ messages in thread From: Jan Kara @ 2016-10-27 13:37 UTC (permalink / raw) To: Xiong Zhou Cc: linux-fsdevel-u79uwXL29TY76Z2rM5mHXA, linux-nvdimm-y27Ovi1pjclAfugRpC6u6w, jack-AlSwsSmVLrQ, linux-kernel-u79uwXL29TY76Z2rM5mHXA Thanks for the report. I'll try to reproduce it... Honza On Thu 27-10-16 19:22:30, Xiong Zhou wrote: > # description > > nvml test suite vmmalloc_fork test hang. > > $ ps -eo stat,comm | grep vmma > S+ vmmalloc_fork > Sl+ vmmalloc_fork > Z+ vmmalloc_fork <defunct> > Sl+ vmmalloc_fork > Z+ vmmalloc_fork <defunct> > Z+ vmmalloc_fork <defunct> > Sl+ vmmalloc_fork > Z+ vmmalloc_fork <defunct> > Z+ vmmalloc_fork <defunct> > Z+ vmmalloc_fork <defunct> > > dmesg: > > [ 250.499097] INFO: task vmmalloc_fork:9805 blocked for more than 120 seconds. > [ 250.530667] Not tainted 4.9.09fe68ca+ #27 > [ 250.550901] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 250.585752] vmmalloc_fork D[ 250.598362] ffffffff8171813c 0 9805 9765 0x00000080 > [ 250.623445] ffff88075dc68f80[ 250.636052] 0000000000000000 ffff88076058db00 ffff88017c5b0000 ffff880763b19340[ 250.668510] ffffc9000fe1bbb0 ffffffff8171813c ffffc9000fe1bc20 ffffc9000fe1bbe0[ 250.704220] ffffffff82248898 ffff88076058db00 ffffffff82248898Call Trace: > [ 250.738382] [<ffffffff8171813c>] ? __schedule+0x21c/0x6a0 > [ 250.763404] [<ffffffff817185f6>] schedule+0x36/0x80 > [ 250.786177] [<ffffffff81284471>] get_unlocked_mapping_entry+0xc1/0x120 > [ 250.815869] [<ffffffff81283810>] ? iomap_dax_rw+0x110/0x110 > [ 250.841350] [<ffffffff81284c0a>] grab_mapping_entry+0x4a/0x220 > [ 250.868442] [<ffffffff812851e9>] iomap_dax_fault+0xa9/0x3b0 > [ 250.894437] [<ffffffffa02b15fe>] xfs_filemap_fault+0xce/0xf0 [xfs] > [ 250.922805] [<ffffffff811d3159>] __do_fault+0x79/0x100 > [ 250.947035] [<ffffffff811d7a2b>] do_fault+0x49b/0x690 > [ 250.970964] [<ffffffffa02b146c>] ? xfs_filemap_pmd_fault+0x9c/0x160 [xfs] > [ 251.001812] [<ffffffff811d94ba>] handle_mm_fault+0x61a/0xa50 > [ 251.027736] [<ffffffff8106c3da>] __do_page_fault+0x22a/0x4a0 > [ 251.053700] [<ffffffff8106c680>] do_page_fault+0x30/0x80 > [ 251.077962] [<ffffffff81003b55>] ? do_syscall_64+0x175/0x180 > [ 251.103835] [<ffffffff8171e208>] page_fault+0x28/0x30 > > > # kernel versions: > > v4.6 pass in seconds > v4.7 hang > v4.9-rc1 hang > Linus tree to commit 9fe68ca hang > > bisect points to > first bad commit: [ac401cc782429cc8560ce4840b1405d603740917] dax: New fault locking > > v4.7 with these 3 commits reverted pass: > 4d9a2c8 - Jan Kara, 6 months ago : dax: Remove i_mmap_lock protection > bc2466e - Jan Kara, 6 months ago : dax: Use radix tree entry lock to protect cow faults > ac401cc - Jan Kara, 6 months ago : dax: New fault locking > > # nvml version: > https://github.com/pmem/nvml.git > to commit: > feab4d6f65102139ce460890c898fcad09ce20ae > > # How reproducible: > always > > # Test steps: > > <git clone and pmem0 setup> > > $cd nvml > $make install -j64 > > $cat > src/test/testconfig.sh <<EOF > PMEM_FS_DIR=/daxmnt > NON_PMEM_FS_DIR=/tmp > EOF > > $mkfs.xfs /dev/pmem0 > $mkdir -p /daxmnt/ > $mount -o dax /dev/pmem0 /daxmnt/ > > $make -C src/test/vmmalloc_fork/ TEST_TIME=60m clean > $make -C src/test/vmmalloc_fork/ TEST_TIME=60m check > $umount /daxmnt -- Jan Kara <jack-IBi9RG/b67k@public.gmane.org> SUSE Labs, CR ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: multi-threads libvmmalloc fork test hang @ 2016-10-27 13:37 ` Jan Kara 0 siblings, 0 replies; 14+ messages in thread From: Jan Kara @ 2016-10-27 13:37 UTC (permalink / raw) To: Xiong Zhou; +Cc: linux-nvdimm, jack, linux-fsdevel, linux-kernel Thanks for the report. I'll try to reproduce it... Honza On Thu 27-10-16 19:22:30, Xiong Zhou wrote: > # description > > nvml test suite vmmalloc_fork test hang. > > $ ps -eo stat,comm | grep vmma > S+ vmmalloc_fork > Sl+ vmmalloc_fork > Z+ vmmalloc_fork <defunct> > Sl+ vmmalloc_fork > Z+ vmmalloc_fork <defunct> > Z+ vmmalloc_fork <defunct> > Sl+ vmmalloc_fork > Z+ vmmalloc_fork <defunct> > Z+ vmmalloc_fork <defunct> > Z+ vmmalloc_fork <defunct> > > dmesg: > > [ 250.499097] INFO: task vmmalloc_fork:9805 blocked for more than 120 seconds. > [ 250.530667] Not tainted 4.9.09fe68ca+ #27 > [ 250.550901] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 250.585752] vmmalloc_fork D[ 250.598362] ffffffff8171813c 0 9805 9765 0x00000080 > [ 250.623445] ffff88075dc68f80[ 250.636052] 0000000000000000 ffff88076058db00 ffff88017c5b0000 ffff880763b19340[ 250.668510] ffffc9000fe1bbb0 ffffffff8171813c ffffc9000fe1bc20 ffffc9000fe1bbe0[ 250.704220] ffffffff82248898 ffff88076058db00 ffffffff82248898Call Trace: > [ 250.738382] [<ffffffff8171813c>] ? __schedule+0x21c/0x6a0 > [ 250.763404] [<ffffffff817185f6>] schedule+0x36/0x80 > [ 250.786177] [<ffffffff81284471>] get_unlocked_mapping_entry+0xc1/0x120 > [ 250.815869] [<ffffffff81283810>] ? iomap_dax_rw+0x110/0x110 > [ 250.841350] [<ffffffff81284c0a>] grab_mapping_entry+0x4a/0x220 > [ 250.868442] [<ffffffff812851e9>] iomap_dax_fault+0xa9/0x3b0 > [ 250.894437] [<ffffffffa02b15fe>] xfs_filemap_fault+0xce/0xf0 [xfs] > [ 250.922805] [<ffffffff811d3159>] __do_fault+0x79/0x100 > [ 250.947035] [<ffffffff811d7a2b>] do_fault+0x49b/0x690 > [ 250.970964] [<ffffffffa02b146c>] ? xfs_filemap_pmd_fault+0x9c/0x160 [xfs] > [ 251.001812] [<ffffffff811d94ba>] handle_mm_fault+0x61a/0xa50 > [ 251.027736] [<ffffffff8106c3da>] __do_page_fault+0x22a/0x4a0 > [ 251.053700] [<ffffffff8106c680>] do_page_fault+0x30/0x80 > [ 251.077962] [<ffffffff81003b55>] ? do_syscall_64+0x175/0x180 > [ 251.103835] [<ffffffff8171e208>] page_fault+0x28/0x30 > > > # kernel versions: > > v4.6 pass in seconds > v4.7 hang > v4.9-rc1 hang > Linus tree to commit 9fe68ca hang > > bisect points to > first bad commit: [ac401cc782429cc8560ce4840b1405d603740917] dax: New fault locking > > v4.7 with these 3 commits reverted pass: > 4d9a2c8 - Jan Kara, 6 months ago : dax: Remove i_mmap_lock protection > bc2466e - Jan Kara, 6 months ago : dax: Use radix tree entry lock to protect cow faults > ac401cc - Jan Kara, 6 months ago : dax: New fault locking > > # nvml version: > https://github.com/pmem/nvml.git > to commit: > feab4d6f65102139ce460890c898fcad09ce20ae > > # How reproducible: > always > > # Test steps: > > <git clone and pmem0 setup> > > $cd nvml > $make install -j64 > > $cat > src/test/testconfig.sh <<EOF > PMEM_FS_DIR=/daxmnt > NON_PMEM_FS_DIR=/tmp > EOF > > $mkfs.xfs /dev/pmem0 > $mkdir -p /daxmnt/ > $mount -o dax /dev/pmem0 /daxmnt/ > > $make -C src/test/vmmalloc_fork/ TEST_TIME=60m clean > $make -C src/test/vmmalloc_fork/ TEST_TIME=60m check > $umount /daxmnt -- Jan Kara <jack@suse.com> SUSE Labs, CR ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <20161027133720.GF19743-4I4JzKEfoa/jFM9bn6wA6Q@public.gmane.org>]
* Re: multi-threads libvmmalloc fork test hang 2016-10-27 13:37 ` Jan Kara @ 2017-01-03 16:58 ` Ross Zwisler -1 siblings, 0 replies; 14+ messages in thread From: Ross Zwisler @ 2017-01-03 16:58 UTC (permalink / raw) To: Jan Kara Cc: linux-fsdevel-u79uwXL29TY76Z2rM5mHXA, linux-nvdimm-y27Ovi1pjclAfugRpC6u6w, linux-kernel-u79uwXL29TY76Z2rM5mHXA On Thu, Oct 27, 2016 at 03:37:20PM +0200, Jan Kara wrote: > Thanks for the report. I'll try to reproduce it... > > Honza > > On Thu 27-10-16 19:22:30, Xiong Zhou wrote: > > # description > > > > nvml test suite vmmalloc_fork test hang. > > > > $ ps -eo stat,comm | grep vmma > > S+ vmmalloc_fork > > Sl+ vmmalloc_fork > > Z+ vmmalloc_fork <defunct> > > Sl+ vmmalloc_fork > > Z+ vmmalloc_fork <defunct> > > Z+ vmmalloc_fork <defunct> > > Sl+ vmmalloc_fork > > Z+ vmmalloc_fork <defunct> > > Z+ vmmalloc_fork <defunct> > > Z+ vmmalloc_fork <defunct> > > > > dmesg: > > > > [ 250.499097] INFO: task vmmalloc_fork:9805 blocked for more than 120 seconds. > > [ 250.530667] Not tainted 4.9.09fe68ca+ #27 > > [ 250.550901] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > [ 250.585752] vmmalloc_fork D[ 250.598362] ffffffff8171813c 0 9805 9765 0x00000080 > > [ 250.623445] ffff88075dc68f80[ 250.636052] 0000000000000000 ffff88076058db00 ffff88017c5b0000 ffff880763b19340[ 250.668510] ffffc9000fe1bbb0 ffffffff8171813c ffffc9000fe1bc20 ffffc9000fe1bbe0[ 250.704220] ffffffff82248898 ffff88076058db00 ffffffff82248898Call Trace: > > [ 250.738382] [<ffffffff8171813c>] ? __schedule+0x21c/0x6a0 > > [ 250.763404] [<ffffffff817185f6>] schedule+0x36/0x80 > > [ 250.786177] [<ffffffff81284471>] get_unlocked_mapping_entry+0xc1/0x120 > > [ 250.815869] [<ffffffff81283810>] ? iomap_dax_rw+0x110/0x110 > > [ 250.841350] [<ffffffff81284c0a>] grab_mapping_entry+0x4a/0x220 > > [ 250.868442] [<ffffffff812851e9>] iomap_dax_fault+0xa9/0x3b0 > > [ 250.894437] [<ffffffffa02b15fe>] xfs_filemap_fault+0xce/0xf0 [xfs] > > [ 250.922805] [<ffffffff811d3159>] __do_fault+0x79/0x100 > > [ 250.947035] [<ffffffff811d7a2b>] do_fault+0x49b/0x690 > > [ 250.970964] [<ffffffffa02b146c>] ? xfs_filemap_pmd_fault+0x9c/0x160 [xfs] > > [ 251.001812] [<ffffffff811d94ba>] handle_mm_fault+0x61a/0xa50 > > [ 251.027736] [<ffffffff8106c3da>] __do_page_fault+0x22a/0x4a0 > > [ 251.053700] [<ffffffff8106c680>] do_page_fault+0x30/0x80 > > [ 251.077962] [<ffffffff81003b55>] ? do_syscall_64+0x175/0x180 > > [ 251.103835] [<ffffffff8171e208>] page_fault+0x28/0x30 > > > > > > # kernel versions: > > > > v4.6 pass in seconds > > v4.7 hang > > v4.9-rc1 hang > > Linus tree to commit 9fe68ca hang > > > > bisect points to > > first bad commit: [ac401cc782429cc8560ce4840b1405d603740917] dax: New fault locking > > > > v4.7 with these 3 commits reverted pass: > > 4d9a2c8 - Jan Kara, 6 months ago : dax: Remove i_mmap_lock protection > > bc2466e - Jan Kara, 6 months ago : dax: Use radix tree entry lock to protect cow faults > > ac401cc - Jan Kara, 6 months ago : dax: New fault locking > > > > # nvml version: > > https://github.com/pmem/nvml.git > > to commit: > > feab4d6f65102139ce460890c898fcad09ce20ae > > > > # How reproducible: > > always > > > > # Test steps: > > > > <git clone and pmem0 setup> > > > > $cd nvml > > $make install -j64 > > > > $cat > src/test/testconfig.sh <<EOF > > PMEM_FS_DIR=/daxmnt > > NON_PMEM_FS_DIR=/tmp > > EOF > > > > $mkfs.xfs /dev/pmem0 > > $mkdir -p /daxmnt/ > > $mount -o dax /dev/pmem0 /daxmnt/ > > > > $make -C src/test/vmmalloc_fork/ TEST_TIME=60m clean > > $make -C src/test/vmmalloc_fork/ TEST_TIME=60m check > > $umount /daxmnt As I mentioned in the other thread I was able to reproduce this issue with v4.10-rc2, and am currently debugging. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: multi-threads libvmmalloc fork test hang @ 2017-01-03 16:58 ` Ross Zwisler 0 siblings, 0 replies; 14+ messages in thread From: Ross Zwisler @ 2017-01-03 16:58 UTC (permalink / raw) To: Jan Kara; +Cc: Xiong Zhou, linux-fsdevel, linux-nvdimm, linux-kernel On Thu, Oct 27, 2016 at 03:37:20PM +0200, Jan Kara wrote: > Thanks for the report. I'll try to reproduce it... > > Honza > > On Thu 27-10-16 19:22:30, Xiong Zhou wrote: > > # description > > > > nvml test suite vmmalloc_fork test hang. > > > > $ ps -eo stat,comm | grep vmma > > S+ vmmalloc_fork > > Sl+ vmmalloc_fork > > Z+ vmmalloc_fork <defunct> > > Sl+ vmmalloc_fork > > Z+ vmmalloc_fork <defunct> > > Z+ vmmalloc_fork <defunct> > > Sl+ vmmalloc_fork > > Z+ vmmalloc_fork <defunct> > > Z+ vmmalloc_fork <defunct> > > Z+ vmmalloc_fork <defunct> > > > > dmesg: > > > > [ 250.499097] INFO: task vmmalloc_fork:9805 blocked for more than 120 seconds. > > [ 250.530667] Not tainted 4.9.09fe68ca+ #27 > > [ 250.550901] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > > [ 250.585752] vmmalloc_fork D[ 250.598362] ffffffff8171813c 0 9805 9765 0x00000080 > > [ 250.623445] ffff88075dc68f80[ 250.636052] 0000000000000000 ffff88076058db00 ffff88017c5b0000 ffff880763b19340[ 250.668510] ffffc9000fe1bbb0 ffffffff8171813c ffffc9000fe1bc20 ffffc9000fe1bbe0[ 250.704220] ffffffff82248898 ffff88076058db00 ffffffff82248898Call Trace: > > [ 250.738382] [<ffffffff8171813c>] ? __schedule+0x21c/0x6a0 > > [ 250.763404] [<ffffffff817185f6>] schedule+0x36/0x80 > > [ 250.786177] [<ffffffff81284471>] get_unlocked_mapping_entry+0xc1/0x120 > > [ 250.815869] [<ffffffff81283810>] ? iomap_dax_rw+0x110/0x110 > > [ 250.841350] [<ffffffff81284c0a>] grab_mapping_entry+0x4a/0x220 > > [ 250.868442] [<ffffffff812851e9>] iomap_dax_fault+0xa9/0x3b0 > > [ 250.894437] [<ffffffffa02b15fe>] xfs_filemap_fault+0xce/0xf0 [xfs] > > [ 250.922805] [<ffffffff811d3159>] __do_fault+0x79/0x100 > > [ 250.947035] [<ffffffff811d7a2b>] do_fault+0x49b/0x690 > > [ 250.970964] [<ffffffffa02b146c>] ? xfs_filemap_pmd_fault+0x9c/0x160 [xfs] > > [ 251.001812] [<ffffffff811d94ba>] handle_mm_fault+0x61a/0xa50 > > [ 251.027736] [<ffffffff8106c3da>] __do_page_fault+0x22a/0x4a0 > > [ 251.053700] [<ffffffff8106c680>] do_page_fault+0x30/0x80 > > [ 251.077962] [<ffffffff81003b55>] ? do_syscall_64+0x175/0x180 > > [ 251.103835] [<ffffffff8171e208>] page_fault+0x28/0x30 > > > > > > # kernel versions: > > > > v4.6 pass in seconds > > v4.7 hang > > v4.9-rc1 hang > > Linus tree to commit 9fe68ca hang > > > > bisect points to > > first bad commit: [ac401cc782429cc8560ce4840b1405d603740917] dax: New fault locking > > > > v4.7 with these 3 commits reverted pass: > > 4d9a2c8 - Jan Kara, 6 months ago : dax: Remove i_mmap_lock protection > > bc2466e - Jan Kara, 6 months ago : dax: Use radix tree entry lock to protect cow faults > > ac401cc - Jan Kara, 6 months ago : dax: New fault locking > > > > # nvml version: > > https://github.com/pmem/nvml.git > > to commit: > > feab4d6f65102139ce460890c898fcad09ce20ae > > > > # How reproducible: > > always > > > > # Test steps: > > > > <git clone and pmem0 setup> > > > > $cd nvml > > $make install -j64 > > > > $cat > src/test/testconfig.sh <<EOF > > PMEM_FS_DIR=/daxmnt > > NON_PMEM_FS_DIR=/tmp > > EOF > > > > $mkfs.xfs /dev/pmem0 > > $mkdir -p /daxmnt/ > > $mount -o dax /dev/pmem0 /daxmnt/ > > > > $make -C src/test/vmmalloc_fork/ TEST_TIME=60m clean > > $make -C src/test/vmmalloc_fork/ TEST_TIME=60m check > > $umount /daxmnt As I mentioned in the other thread I was able to reproduce this issue with v4.10-rc2, and am currently debugging. ^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH] dax: fix deadlock with DAX 4k holes 2016-10-27 11:22 ` Xiong Zhou (?) @ 2017-01-03 21:36 ` Ross Zwisler -1 siblings, 0 replies; 14+ messages in thread From: Ross Zwisler @ 2017-01-03 21:36 UTC (permalink / raw) To: Xiong Zhou, stable, linux-kernel Cc: Jan Kara, Andrew Morton, linux-mm, linux-nvdimm, Dave Chinner, Christoph Hellwig, Dave Hansen Currently in DAX if we have three read faults on the same hole address we can end up with the following: Thread 0 Thread 1 Thread 2 -------- -------- -------- dax_iomap_fault grab_mapping_entry lock_slot <locks empty DAX entry> dax_iomap_fault grab_mapping_entry get_unlocked_mapping_entry <sleeps on empty DAX entry> dax_iomap_fault grab_mapping_entry get_unlocked_mapping_entry <sleeps on empty DAX entry> dax_load_hole find_or_create_page ... page_cache_tree_insert dax_wake_mapping_entry_waiter <wakes one sleeper> __radix_tree_replace <swaps empty DAX entry with 4k zero page> <wakes> get_page lock_page ... put_locked_mapping_entry unlock_page put_page <sleeps forever on the DAX wait queue> The crux of the problem is that once we insert a 4k zero page, all locking from then on is done in terms of that 4k zero page and any additional threads sleeping on the empty DAX entry will never be woken. Fix this by waking all sleepers when we replace the DAX radix tree entry with a 4k zero page. This will allow all sleeping threads to successfully transition from locking based on the DAX empty entry to locking on the 4k zero page. With the test case reported by Xiong this happens very regularly in my test setup, with some runs resulting in 9+ threads in this deadlocked state. With this fix I've been able to run that same test dozens of times in a loop without issue. Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com> Reported-by: Xiong Zhou <xzhou@redhat.com> Fixes: commit ac401cc78242 ("dax: New fault locking") Cc: Jan Kara <jack@suse.cz> Cc: stable@vger.kernel.org # 4.7+ --- This issue exists as far back as v4.7, and I was easly able to reproduce it with v4.7 using the same test. Unfortunately this patch won't apply cleanly to the stable trees, but the change is very simple and should be easy to replicate by hand. Please ping me if you'd like patches that apply cleanly to the v4.9 and v4.8.15 trees. --- mm/filemap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index d0e4d10..b772a33 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -138,7 +138,7 @@ static int page_cache_tree_insert(struct address_space *mapping, dax_radix_locked_entry(0, RADIX_DAX_EMPTY)); /* Wakeup waiters for exceptional entry lock */ dax_wake_mapping_entry_waiter(mapping, page->index, p, - false); + true); } } __radix_tree_replace(&mapping->page_tree, node, slot, page, -- 2.7.4 _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm ^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH] dax: fix deadlock with DAX 4k holes @ 2017-01-03 21:36 ` Ross Zwisler 0 siblings, 0 replies; 14+ messages in thread From: Ross Zwisler @ 2017-01-03 21:36 UTC (permalink / raw) To: Xiong Zhou, stable, linux-kernel Cc: Ross Zwisler, Andrew Morton, Christoph Hellwig, Dan Williams, Dave Chinner, Dave Hansen, Jan Kara, linux-mm, linux-nvdimm Currently in DAX if we have three read faults on the same hole address we can end up with the following: Thread 0 Thread 1 Thread 2 -------- -------- -------- dax_iomap_fault grab_mapping_entry lock_slot <locks empty DAX entry> dax_iomap_fault grab_mapping_entry get_unlocked_mapping_entry <sleeps on empty DAX entry> dax_iomap_fault grab_mapping_entry get_unlocked_mapping_entry <sleeps on empty DAX entry> dax_load_hole find_or_create_page ... page_cache_tree_insert dax_wake_mapping_entry_waiter <wakes one sleeper> __radix_tree_replace <swaps empty DAX entry with 4k zero page> <wakes> get_page lock_page ... put_locked_mapping_entry unlock_page put_page <sleeps forever on the DAX wait queue> The crux of the problem is that once we insert a 4k zero page, all locking from then on is done in terms of that 4k zero page and any additional threads sleeping on the empty DAX entry will never be woken. Fix this by waking all sleepers when we replace the DAX radix tree entry with a 4k zero page. This will allow all sleeping threads to successfully transition from locking based on the DAX empty entry to locking on the 4k zero page. With the test case reported by Xiong this happens very regularly in my test setup, with some runs resulting in 9+ threads in this deadlocked state. With this fix I've been able to run that same test dozens of times in a loop without issue. Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com> Reported-by: Xiong Zhou <xzhou@redhat.com> Fixes: commit ac401cc78242 ("dax: New fault locking") Cc: Jan Kara <jack@suse.cz> Cc: stable@vger.kernel.org # 4.7+ --- This issue exists as far back as v4.7, and I was easly able to reproduce it with v4.7 using the same test. Unfortunately this patch won't apply cleanly to the stable trees, but the change is very simple and should be easy to replicate by hand. Please ping me if you'd like patches that apply cleanly to the v4.9 and v4.8.15 trees. --- mm/filemap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index d0e4d10..b772a33 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -138,7 +138,7 @@ static int page_cache_tree_insert(struct address_space *mapping, dax_radix_locked_entry(0, RADIX_DAX_EMPTY)); /* Wakeup waiters for exceptional entry lock */ dax_wake_mapping_entry_waiter(mapping, page->index, p, - false); + true); } } __radix_tree_replace(&mapping->page_tree, node, slot, page, -- 2.7.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH] dax: fix deadlock with DAX 4k holes @ 2017-01-03 21:36 ` Ross Zwisler 0 siblings, 0 replies; 14+ messages in thread From: Ross Zwisler @ 2017-01-03 21:36 UTC (permalink / raw) To: Xiong Zhou, stable, linux-kernel Cc: Ross Zwisler, Andrew Morton, Christoph Hellwig, Dan Williams, Dave Chinner, Dave Hansen, Jan Kara, linux-mm, linux-nvdimm Currently in DAX if we have three read faults on the same hole address we can end up with the following: Thread 0 Thread 1 Thread 2 -------- -------- -------- dax_iomap_fault grab_mapping_entry lock_slot <locks empty DAX entry> dax_iomap_fault grab_mapping_entry get_unlocked_mapping_entry <sleeps on empty DAX entry> dax_iomap_fault grab_mapping_entry get_unlocked_mapping_entry <sleeps on empty DAX entry> dax_load_hole find_or_create_page ... page_cache_tree_insert dax_wake_mapping_entry_waiter <wakes one sleeper> __radix_tree_replace <swaps empty DAX entry with 4k zero page> <wakes> get_page lock_page ... put_locked_mapping_entry unlock_page put_page <sleeps forever on the DAX wait queue> The crux of the problem is that once we insert a 4k zero page, all locking from then on is done in terms of that 4k zero page and any additional threads sleeping on the empty DAX entry will never be woken. Fix this by waking all sleepers when we replace the DAX radix tree entry with a 4k zero page. This will allow all sleeping threads to successfully transition from locking based on the DAX empty entry to locking on the 4k zero page. With the test case reported by Xiong this happens very regularly in my test setup, with some runs resulting in 9+ threads in this deadlocked state. With this fix I've been able to run that same test dozens of times in a loop without issue. Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com> Reported-by: Xiong Zhou <xzhou@redhat.com> Fixes: commit ac401cc78242 ("dax: New fault locking") Cc: Jan Kara <jack@suse.cz> Cc: stable@vger.kernel.org # 4.7+ --- This issue exists as far back as v4.7, and I was easly able to reproduce it with v4.7 using the same test. Unfortunately this patch won't apply cleanly to the stable trees, but the change is very simple and should be easy to replicate by hand. Please ping me if you'd like patches that apply cleanly to the v4.9 and v4.8.15 trees. --- mm/filemap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/filemap.c b/mm/filemap.c index d0e4d10..b772a33 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -138,7 +138,7 @@ static int page_cache_tree_insert(struct address_space *mapping, dax_radix_locked_entry(0, RADIX_DAX_EMPTY)); /* Wakeup waiters for exceptional entry lock */ dax_wake_mapping_entry_waiter(mapping, page->index, p, - false); + true); } } __radix_tree_replace(&mapping->page_tree, node, slot, page, -- 2.7.4 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCH] dax: fix deadlock with DAX 4k holes 2017-01-03 21:36 ` Ross Zwisler (?) @ 2017-01-04 7:18 ` Jan Kara -1 siblings, 0 replies; 14+ messages in thread From: Jan Kara @ 2017-01-04 7:18 UTC (permalink / raw) To: Ross Zwisler Cc: Jan Kara, linux-nvdimm, Dave Chinner, linux-kernel, stable, Christoph Hellwig, linux-mm, Dave Hansen, Andrew Morton On Tue 03-01-17 14:36:05, Ross Zwisler wrote: > Currently in DAX if we have three read faults on the same hole address we > can end up with the following: > > Thread 0 Thread 1 Thread 2 > -------- -------- -------- > dax_iomap_fault > grab_mapping_entry > lock_slot > <locks empty DAX entry> > > dax_iomap_fault > grab_mapping_entry > get_unlocked_mapping_entry > <sleeps on empty DAX entry> > > dax_iomap_fault > grab_mapping_entry > get_unlocked_mapping_entry > <sleeps on empty DAX entry> > dax_load_hole > find_or_create_page > ... > page_cache_tree_insert > dax_wake_mapping_entry_waiter > <wakes one sleeper> > __radix_tree_replace > <swaps empty DAX entry with 4k zero page> > > <wakes> > get_page > lock_page > ... > put_locked_mapping_entry > unlock_page > put_page > > <sleeps forever on the DAX > wait queue> > > The crux of the problem is that once we insert a 4k zero page, all locking > from then on is done in terms of that 4k zero page and any additional > threads sleeping on the empty DAX entry will never be woken. Fix this by > waking all sleepers when we replace the DAX radix tree entry with a 4k zero > page. This will allow all sleeping threads to successfully transition from > locking based on the DAX empty entry to locking on the 4k zero page. > > With the test case reported by Xiong this happens very regularly in my test > setup, with some runs resulting in 9+ threads in this deadlocked state. > With this fix I've been able to run that same test dozens of times in a > loop without issue. > > Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com> > Reported-by: Xiong Zhou <xzhou@redhat.com> > Fixes: commit ac401cc78242 ("dax: New fault locking") > Cc: Jan Kara <jack@suse.cz> > Cc: stable@vger.kernel.org # 4.7+ Ah, very good catch. You can add: Reviewed-by: Jan Kara <jack@suse.cz> I wonder why I was not able to reproduce this... Probably the timing didn't work out right on my test machine. Honza > --- > > This issue exists as far back as v4.7, and I was easly able to reproduce it > with v4.7 using the same test. > > Unfortunately this patch won't apply cleanly to the stable trees, but the > change is very simple and should be easy to replicate by hand. Please ping > me if you'd like patches that apply cleanly to the v4.9 and v4.8.15 trees. > > --- > mm/filemap.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/mm/filemap.c b/mm/filemap.c > index d0e4d10..b772a33 100644 > --- a/mm/filemap.c > +++ b/mm/filemap.c > @@ -138,7 +138,7 @@ static int page_cache_tree_insert(struct address_space *mapping, > dax_radix_locked_entry(0, RADIX_DAX_EMPTY)); > /* Wakeup waiters for exceptional entry lock */ > dax_wake_mapping_entry_waiter(mapping, page->index, p, > - false); > + true); > } > } > __radix_tree_replace(&mapping->page_tree, node, slot, page, > -- > 2.7.4 > -- Jan Kara <jack@suse.com> SUSE Labs, CR _______________________________________________ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] dax: fix deadlock with DAX 4k holes @ 2017-01-04 7:18 ` Jan Kara 0 siblings, 0 replies; 14+ messages in thread From: Jan Kara @ 2017-01-04 7:18 UTC (permalink / raw) To: Ross Zwisler Cc: Xiong Zhou, stable, linux-kernel, Andrew Morton, Christoph Hellwig, Dan Williams, Dave Chinner, Dave Hansen, Jan Kara, linux-mm, linux-nvdimm On Tue 03-01-17 14:36:05, Ross Zwisler wrote: > Currently in DAX if we have three read faults on the same hole address we > can end up with the following: > > Thread 0 Thread 1 Thread 2 > -------- -------- -------- > dax_iomap_fault > grab_mapping_entry > lock_slot > <locks empty DAX entry> > > dax_iomap_fault > grab_mapping_entry > get_unlocked_mapping_entry > <sleeps on empty DAX entry> > > dax_iomap_fault > grab_mapping_entry > get_unlocked_mapping_entry > <sleeps on empty DAX entry> > dax_load_hole > find_or_create_page > ... > page_cache_tree_insert > dax_wake_mapping_entry_waiter > <wakes one sleeper> > __radix_tree_replace > <swaps empty DAX entry with 4k zero page> > > <wakes> > get_page > lock_page > ... > put_locked_mapping_entry > unlock_page > put_page > > <sleeps forever on the DAX > wait queue> > > The crux of the problem is that once we insert a 4k zero page, all locking > from then on is done in terms of that 4k zero page and any additional > threads sleeping on the empty DAX entry will never be woken. Fix this by > waking all sleepers when we replace the DAX radix tree entry with a 4k zero > page. This will allow all sleeping threads to successfully transition from > locking based on the DAX empty entry to locking on the 4k zero page. > > With the test case reported by Xiong this happens very regularly in my test > setup, with some runs resulting in 9+ threads in this deadlocked state. > With this fix I've been able to run that same test dozens of times in a > loop without issue. > > Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com> > Reported-by: Xiong Zhou <xzhou@redhat.com> > Fixes: commit ac401cc78242 ("dax: New fault locking") > Cc: Jan Kara <jack@suse.cz> > Cc: stable@vger.kernel.org # 4.7+ Ah, very good catch. You can add: Reviewed-by: Jan Kara <jack@suse.cz> I wonder why I was not able to reproduce this... Probably the timing didn't work out right on my test machine. Honza > --- > > This issue exists as far back as v4.7, and I was easly able to reproduce it > with v4.7 using the same test. > > Unfortunately this patch won't apply cleanly to the stable trees, but the > change is very simple and should be easy to replicate by hand. Please ping > me if you'd like patches that apply cleanly to the v4.9 and v4.8.15 trees. > > --- > mm/filemap.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/mm/filemap.c b/mm/filemap.c > index d0e4d10..b772a33 100644 > --- a/mm/filemap.c > +++ b/mm/filemap.c > @@ -138,7 +138,7 @@ static int page_cache_tree_insert(struct address_space *mapping, > dax_radix_locked_entry(0, RADIX_DAX_EMPTY)); > /* Wakeup waiters for exceptional entry lock */ > dax_wake_mapping_entry_waiter(mapping, page->index, p, > - false); > + true); > } > } > __radix_tree_replace(&mapping->page_tree, node, slot, page, > -- > 2.7.4 > -- Jan Kara <jack@suse.com> SUSE Labs, CR -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] dax: fix deadlock with DAX 4k holes @ 2017-01-04 7:18 ` Jan Kara 0 siblings, 0 replies; 14+ messages in thread From: Jan Kara @ 2017-01-04 7:18 UTC (permalink / raw) To: Ross Zwisler Cc: Xiong Zhou, stable, linux-kernel, Andrew Morton, Christoph Hellwig, Dan Williams, Dave Chinner, Dave Hansen, Jan Kara, linux-mm, linux-nvdimm On Tue 03-01-17 14:36:05, Ross Zwisler wrote: > Currently in DAX if we have three read faults on the same hole address we > can end up with the following: > > Thread 0 Thread 1 Thread 2 > -------- -------- -------- > dax_iomap_fault > grab_mapping_entry > lock_slot > <locks empty DAX entry> > > dax_iomap_fault > grab_mapping_entry > get_unlocked_mapping_entry > <sleeps on empty DAX entry> > > dax_iomap_fault > grab_mapping_entry > get_unlocked_mapping_entry > <sleeps on empty DAX entry> > dax_load_hole > find_or_create_page > ... > page_cache_tree_insert > dax_wake_mapping_entry_waiter > <wakes one sleeper> > __radix_tree_replace > <swaps empty DAX entry with 4k zero page> > > <wakes> > get_page > lock_page > ... > put_locked_mapping_entry > unlock_page > put_page > > <sleeps forever on the DAX > wait queue> > > The crux of the problem is that once we insert a 4k zero page, all locking > from then on is done in terms of that 4k zero page and any additional > threads sleeping on the empty DAX entry will never be woken. Fix this by > waking all sleepers when we replace the DAX radix tree entry with a 4k zero > page. This will allow all sleeping threads to successfully transition from > locking based on the DAX empty entry to locking on the 4k zero page. > > With the test case reported by Xiong this happens very regularly in my test > setup, with some runs resulting in 9+ threads in this deadlocked state. > With this fix I've been able to run that same test dozens of times in a > loop without issue. > > Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com> > Reported-by: Xiong Zhou <xzhou@redhat.com> > Fixes: commit ac401cc78242 ("dax: New fault locking") > Cc: Jan Kara <jack@suse.cz> > Cc: stable@vger.kernel.org # 4.7+ Ah, very good catch. You can add: Reviewed-by: Jan Kara <jack@suse.cz> I wonder why I was not able to reproduce this... Probably the timing didn't work out right on my test machine. Honza > --- > > This issue exists as far back as v4.7, and I was easly able to reproduce it > with v4.7 using the same test. > > Unfortunately this patch won't apply cleanly to the stable trees, but the > change is very simple and should be easy to replicate by hand. Please ping > me if you'd like patches that apply cleanly to the v4.9 and v4.8.15 trees. > > --- > mm/filemap.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/mm/filemap.c b/mm/filemap.c > index d0e4d10..b772a33 100644 > --- a/mm/filemap.c > +++ b/mm/filemap.c > @@ -138,7 +138,7 @@ static int page_cache_tree_insert(struct address_space *mapping, > dax_radix_locked_entry(0, RADIX_DAX_EMPTY)); > /* Wakeup waiters for exceptional entry lock */ > dax_wake_mapping_entry_waiter(mapping, page->index, p, > - false); > + true); > } > } > __radix_tree_replace(&mapping->page_tree, node, slot, page, > -- > 2.7.4 > -- Jan Kara <jack@suse.com> SUSE Labs, CR ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] dax: fix deadlock with DAX 4k holes 2017-01-03 21:36 ` Ross Zwisler @ 2017-01-04 14:26 ` Xiong Zhou -1 siblings, 0 replies; 14+ messages in thread From: Xiong Zhou @ 2017-01-04 14:26 UTC (permalink / raw) To: Ross Zwisler Cc: Xiong Zhou, stable, linux-kernel, Andrew Morton, Christoph Hellwig, Dan Williams, Dave Chinner, Dave Hansen, Jan Kara, linux-mm, linux-nvdimm On Tue, Jan 03, 2017 at 02:36:05PM -0700, Ross Zwisler wrote: > Currently in DAX if we have three read faults on the same hole address we > can end up with the following: > > Thread 0 Thread 1 Thread 2 > -------- -------- -------- > dax_iomap_fault > grab_mapping_entry > lock_slot > <locks empty DAX entry> > > dax_iomap_fault > grab_mapping_entry > get_unlocked_mapping_entry > <sleeps on empty DAX entry> > > dax_iomap_fault > grab_mapping_entry > get_unlocked_mapping_entry > <sleeps on empty DAX entry> > dax_load_hole > find_or_create_page > ... > page_cache_tree_insert > dax_wake_mapping_entry_waiter > <wakes one sleeper> > __radix_tree_replace > <swaps empty DAX entry with 4k zero page> > > <wakes> > get_page > lock_page > ... > put_locked_mapping_entry > unlock_page > put_page > > <sleeps forever on the DAX > wait queue> > > The crux of the problem is that once we insert a 4k zero page, all locking > from then on is done in terms of that 4k zero page and any additional > threads sleeping on the empty DAX entry will never be woken. Fix this by > waking all sleepers when we replace the DAX radix tree entry with a 4k zero > page. This will allow all sleeping threads to successfully transition from > locking based on the DAX empty entry to locking on the 4k zero page. > > With the test case reported by Xiong this happens very regularly in my test > setup, with some runs resulting in 9+ threads in this deadlocked state. > With this fix I've been able to run that same test dozens of times in a > loop without issue. > > Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com> > Reported-by: Xiong Zhou <xzhou@redhat.com> > Fixes: commit ac401cc78242 ("dax: New fault locking") > Cc: Jan Kara <jack@suse.cz> > Cc: stable@vger.kernel.org # 4.7+ > --- Positive test result of this patch for this issue and the regression tests. Great job! > > This issue exists as far back as v4.7, and I was easly able to reproduce it > with v4.7 using the same test. > > Unfortunately this patch won't apply cleanly to the stable trees, but the > change is very simple and should be easy to replicate by hand. Please ping > me if you'd like patches that apply cleanly to the v4.9 and v4.8.15 trees. > > --- > mm/filemap.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/mm/filemap.c b/mm/filemap.c > index d0e4d10..b772a33 100644 > --- a/mm/filemap.c > +++ b/mm/filemap.c > @@ -138,7 +138,7 @@ static int page_cache_tree_insert(struct address_space *mapping, > dax_radix_locked_entry(0, RADIX_DAX_EMPTY)); > /* Wakeup waiters for exceptional entry lock */ > dax_wake_mapping_entry_waiter(mapping, page->index, p, > - false); > + true); > } > } > __radix_tree_replace(&mapping->page_tree, node, slot, page, > -- > 2.7.4 > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH] dax: fix deadlock with DAX 4k holes @ 2017-01-04 14:26 ` Xiong Zhou 0 siblings, 0 replies; 14+ messages in thread From: Xiong Zhou @ 2017-01-04 14:26 UTC (permalink / raw) To: Ross Zwisler Cc: Xiong Zhou, stable, linux-kernel, Andrew Morton, Christoph Hellwig, Dan Williams, Dave Chinner, Dave Hansen, Jan Kara, linux-mm, linux-nvdimm On Tue, Jan 03, 2017 at 02:36:05PM -0700, Ross Zwisler wrote: > Currently in DAX if we have three read faults on the same hole address we > can end up with the following: > > Thread 0 Thread 1 Thread 2 > -------- -------- -------- > dax_iomap_fault > grab_mapping_entry > lock_slot > <locks empty DAX entry> > > dax_iomap_fault > grab_mapping_entry > get_unlocked_mapping_entry > <sleeps on empty DAX entry> > > dax_iomap_fault > grab_mapping_entry > get_unlocked_mapping_entry > <sleeps on empty DAX entry> > dax_load_hole > find_or_create_page > ... > page_cache_tree_insert > dax_wake_mapping_entry_waiter > <wakes one sleeper> > __radix_tree_replace > <swaps empty DAX entry with 4k zero page> > > <wakes> > get_page > lock_page > ... > put_locked_mapping_entry > unlock_page > put_page > > <sleeps forever on the DAX > wait queue> > > The crux of the problem is that once we insert a 4k zero page, all locking > from then on is done in terms of that 4k zero page and any additional > threads sleeping on the empty DAX entry will never be woken. Fix this by > waking all sleepers when we replace the DAX radix tree entry with a 4k zero > page. This will allow all sleeping threads to successfully transition from > locking based on the DAX empty entry to locking on the 4k zero page. > > With the test case reported by Xiong this happens very regularly in my test > setup, with some runs resulting in 9+ threads in this deadlocked state. > With this fix I've been able to run that same test dozens of times in a > loop without issue. > > Signed-off-by: Ross Zwisler <ross.zwisler@linux.intel.com> > Reported-by: Xiong Zhou <xzhou@redhat.com> > Fixes: commit ac401cc78242 ("dax: New fault locking") > Cc: Jan Kara <jack@suse.cz> > Cc: stable@vger.kernel.org # 4.7+ > --- Positive test result of this patch for this issue and the regression tests. Great job! > > This issue exists as far back as v4.7, and I was easly able to reproduce it > with v4.7 using the same test. > > Unfortunately this patch won't apply cleanly to the stable trees, but the > change is very simple and should be easy to replicate by hand. Please ping > me if you'd like patches that apply cleanly to the v4.9 and v4.8.15 trees. > > --- > mm/filemap.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/mm/filemap.c b/mm/filemap.c > index d0e4d10..b772a33 100644 > --- a/mm/filemap.c > +++ b/mm/filemap.c > @@ -138,7 +138,7 @@ static int page_cache_tree_insert(struct address_space *mapping, > dax_radix_locked_entry(0, RADIX_DAX_EMPTY)); > /* Wakeup waiters for exceptional entry lock */ > dax_wake_mapping_entry_waiter(mapping, page->index, p, > - false); > + true); > } > } > __radix_tree_replace(&mapping->page_tree, node, slot, page, > -- > 2.7.4 > ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2017-01-04 14:26 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-10-27 11:22 multi-threads libvmmalloc fork test hang Xiong Zhou
2016-10-27 11:22 ` Xiong Zhou
[not found] ` <20161027112230.wsumgs62fqdxt3sc-obHvtwIU90hQcClZ3XN9yxcY2uh10dtjAL8bYrjMMd8@public.gmane.org>
2016-10-27 13:37 ` Jan Kara
2016-10-27 13:37 ` Jan Kara
[not found] ` <20161027133720.GF19743-4I4JzKEfoa/jFM9bn6wA6Q@public.gmane.org>
2017-01-03 16:58 ` Ross Zwisler
2017-01-03 16:58 ` Ross Zwisler
2017-01-03 21:36 ` [PATCH] dax: fix deadlock with DAX 4k holes Ross Zwisler
2017-01-03 21:36 ` Ross Zwisler
2017-01-03 21:36 ` Ross Zwisler
2017-01-04 7:18 ` Jan Kara
2017-01-04 7:18 ` Jan Kara
2017-01-04 7:18 ` Jan Kara
2017-01-04 14:26 ` Xiong Zhou
2017-01-04 14:26 ` Xiong Zhou
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.