[BUG] dd doesn't return on ENOSPC and hang when fulfilling rmapbt XFS

[BUG] dd doesn't return on ENOSPC and hang when fulfilling rmapbt XFS

[Eryu Guan]

Hi all,

I hit a test hang in generic/224 when testing rmapbt enabled XFS on a
host that has non-zero sunit/swidth reported from underlying device. And
I simplified the reproducer to the following script, and the hang can be
reproduced on any host now.

-----
#!/bin/bash

dev=/dev/sda5
mnt=/mnt/xfs

mkfs -t xfs -m rmapbt=1 -d agcount=8,size=1g -f $dev
mount $dev $mnt
xfs_io -x -c "resblks 4" $mnt

dd if=/dev/zero of=/mnt/xfs/testfile

echo "dd should return and report ENOSPC"
-----

sysrq-w output:
sysrq: SysRq : Show Blocked State
  task                        PC stack   pid father
dd              D    0  2504   2491 0x00000080
 ffff88021077dd00 0000000000000000 ffff88021623be40 ffff88021fd99300
 ffff8802107467c0 ffffc900019dbc18 ffffffff816e2cf5 ffff88020fa8ce90
 ffffc900019dbc40 0000000000000286 ffff8802107467c0 ffff88020fa8ce90
Call Trace:
 [<ffffffff816e2cf5>] ? __schedule+0x195/0x630
 [<ffffffff816e31c6>] schedule+0x36/0x80
 [<ffffffff812534b4>] wb_wait_for_completion+0x64/0x90
 [<ffffffff810d2910>] ? prepare_to_wait_event+0xf0/0xf0
 [<ffffffff81255b3d>] sync_inodes_sb+0xad/0x290
 [<ffffffff81288bc0>] ? iomap_write_end+0x80/0x80
 [<ffffffff8128922c>] ? iomap_apply+0x6c/0x130
 [<ffffffffa014c2a8>] xfs_flush_inodes+0x28/0x40 [xfs]
 [<ffffffffa013370b>] xfs_file_buffered_aio_write+0x18b/0x280 [xfs]
 [<ffffffffa0133890>] xfs_file_write_iter+0x90/0x130 [xfs]
 [<ffffffff81226b52>] __vfs_write+0xe2/0x140
 [<ffffffff812279d2>] vfs_write+0xb2/0x1b0
 [<ffffffff81003510>] ? syscall_trace_enter+0x1d0/0x2b0
 [<ffffffff81228e25>] SyS_write+0x55/0xc0
 [<ffffffff81003a47>] do_syscall_64+0x67/0x180
 [<ffffffff816e796b>] entry_SYSCALL64_slow_path+0x25/0x25

I tested on 4.9-rc5 kernel, for-next branch from linux-xfs tree and
latest djwong-devel branch from Darrick's tree.

If you need more information please let me know.

Thanks,
Eryu

Re: [BUG] dd doesn't return on ENOSPC and hang when fulfilling rmapbt XFS

[Darrick J. Wong]

On Fri, Nov 18, 2016 at 12:35:15AM +0800, Eryu Guan wrote:
> Hi all,
> 
> I hit a test hang in generic/224 when testing rmapbt enabled XFS on a
> host that has non-zero sunit/swidth reported from underlying device. And
> I simplified the reproducer to the following script, and the hang can be
> reproduced on any host now.
> 
> -----
> #!/bin/bash
> 
> dev=/dev/sda5
> mnt=/mnt/xfs
> 
> mkfs -t xfs -m rmapbt=1 -d agcount=8,size=1g -f $dev

Hm.  I formatted with:
mkfs.xfs -m rmapbt=1 -d sunit=4096,swidth=40960 -f /dev/sdf

(made up sunit numbers just to see how whacky it could get)

and got a different hang instead.  It looks like we are unable to
allocate any blocks to the bmbt and various things blow up from
there.  Will go retry with tracepoints on to see if we're running
out of AG reservation or if we're really out of disk blocks or what.

Crash message attached at the end.

--D

> mount $dev $mnt
> xfs_io -x -c "resblks 4" $mnt
> 
> dd if=/dev/zero of=/mnt/xfs/testfile
> 
> echo "dd should return and report ENOSPC"
> -----
> 
> sysrq-w output:
> sysrq: SysRq : Show Blocked State
>   task                        PC stack   pid father
> dd              D    0  2504   2491 0x00000080
>  ffff88021077dd00 0000000000000000 ffff88021623be40 ffff88021fd99300
>  ffff8802107467c0 ffffc900019dbc18 ffffffff816e2cf5 ffff88020fa8ce90
>  ffffc900019dbc40 0000000000000286 ffff8802107467c0 ffff88020fa8ce90
> Call Trace:
>  [<ffffffff816e2cf5>] ? __schedule+0x195/0x630
>  [<ffffffff816e31c6>] schedule+0x36/0x80
>  [<ffffffff812534b4>] wb_wait_for_completion+0x64/0x90
>  [<ffffffff810d2910>] ? prepare_to_wait_event+0xf0/0xf0
>  [<ffffffff81255b3d>] sync_inodes_sb+0xad/0x290
>  [<ffffffff81288bc0>] ? iomap_write_end+0x80/0x80
>  [<ffffffff8128922c>] ? iomap_apply+0x6c/0x130
>  [<ffffffffa014c2a8>] xfs_flush_inodes+0x28/0x40 [xfs]
>  [<ffffffffa013370b>] xfs_file_buffered_aio_write+0x18b/0x280 [xfs]
>  [<ffffffffa0133890>] xfs_file_write_iter+0x90/0x130 [xfs]
>  [<ffffffff81226b52>] __vfs_write+0xe2/0x140
>  [<ffffffff812279d2>] vfs_write+0xb2/0x1b0
>  [<ffffffff81003510>] ? syscall_trace_enter+0x1d0/0x2b0
>  [<ffffffff81228e25>] SyS_write+0x55/0xc0
>  [<ffffffff81003a47>] do_syscall_64+0x67/0x180
>  [<ffffffff816e796b>] entry_SYSCALL64_slow_path+0x25/0x25
> 
> I tested on 4.9-rc5 kernel, for-next branch from linux-xfs tree and
> latest djwong-devel branch from Darrick's tree.
> 
> If you need more information please let me know.
> 
> Thanks,
> Eryu
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


XFS: Assertion failed: args.fsbno != NULLFSBLOCK, file: /storage/home/djwong/cdev/work/linux-xfs/fs/xfs/libxfs/xfs_bmap.c, line: 789
------------[ cut here ]------------
WARNING: CPU: 1 PID: 104 at /storage/home/djwong/cdev/work/linux-xfs/fs/xfs/xfs_message.c:113 assfail+0x31/0x40 [xfs]
Modules linked in: xfs libcrc32c dax_pmem dax nd_pmem sch_fq_codel af_packet
CPU: 1 PID: 104 Comm: kworker/u8:6 Not tainted 4.9.0-rc5-xfsx #41
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
Workqueue: writeback wb_workfn (flush-8:80)
 ffffc9000083b398 ffffffff81340643 0000000000000000 0000000000000000
 ffffc9000083b3d8 ffffffff8108690b 000000710083b3f8 ffff88006be0a080
 ffffc9000083b838 ffff88006d437000 0000000000000001 ffff88006e407350
Call Trace:
 [<ffffffff81340643>] dump_stack+0x85/0xc2
 [<ffffffff8108690b>] __warn+0xcb/0xf0
 [<ffffffff810869fd>] warn_slowpath_null+0x1d/0x20
 [<ffffffffa00e8971>] assfail+0x31/0x40 [xfs]
 [<ffffffffa0062df5>] xfs_bmap_extents_to_btree+0x235/0x780 [xfs]
 [<ffffffffa0066a91>] xfs_bmap_add_extent_delay_real+0x1af1/0x1ce0 [xfs]
 [<ffffffffa009eb0c>] ? xfs_iext_bno_to_ext+0x8c/0x170 [xfs]
 [<ffffffffa006d06f>] xfs_bmapi_write+0xb6f/0x1180 [xfs]
 [<ffffffff810e016d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffffa00dda24>] xfs_iomap_write_allocate+0x184/0x360 [xfs]
 [<ffffffffa00b9baf>] xfs_map_blocks+0x34f/0x4e0 [xfs]
 [<ffffffffa00ba920>] xfs_do_writepage+0x310/0x840 [xfs]
 [<ffffffff811abc91>] write_cache_pages+0x251/0x640
 [<ffffffffa00ba610>] ? xfs_aops_discard_page+0x140/0x140 [xfs]
 [<ffffffffa00ba043>] xfs_vm_writepages+0xa3/0xe0 [xfs]
 [<ffffffff811acb81>] do_writepages+0x21/0x30
 [<ffffffff81252031>] __writeback_single_inode+0x61/0x800
 [<ffffffff81252cf7>] writeback_sb_inodes+0x297/0x5e0
 [<ffffffff8125329c>] wb_writeback+0xfc/0x5c0
 [<ffffffff81255e8b>] wb_workfn+0x11b/0x680
 [<ffffffff810a62d8>] process_one_work+0x1f8/0x750
 [<ffffffff810a6259>] ? process_one_work+0x179/0x750
 [<ffffffff810a687b>] worker_thread+0x4b/0x4f0
 [<ffffffff810a6830>] ? process_one_work+0x750/0x750
 [<ffffffff810ad3c2>] kthread+0x102/0x120
 [<ffffffff810ad2c0>] ? kthread_park+0x60/0x60
 [<ffffffff8168243a>] ret_from_fork+0x2a/0x40
---[ end trace a2c2fccec3607587 ]---
XFS: Assertion failed: fsbno != NULLFSBLOCK, file: /storage/home/djwong/cdev/work/linux-xfs/fs/xfs/libxfs/xfs_btree.c, line: 662
------------[ cut here ]------------
WARNING: CPU: 1 PID: 104 at /storage/home/djwong/cdev/work/linux-xfs/fs/xfs/xfs_message.c:113 assfail+0x31/0x40 [xfs]
Modules linked in: xfs libcrc32c dax_pmem dax nd_pmem sch_fq_codel af_packet
CPU: 1 PID: 104 Comm: kworker/u8:6 Tainted: G        W       4.9.0-rc5-xfsx #41
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
Workqueue: writeback wb_workfn (flush-8:80)
 ffffc9000083b358 ffffffff81340643 0000000000000000 0000000000000000
 ffffc9000083b398 ffffffff8108690b 000000710083b3b8 ffffffffffffffff
 ffff88006e407350 0000000000000000 ffff88006d437000 ffff88006e407350
Call Trace:
 [<ffffffff81340643>] dump_stack+0x85/0xc2
 [<ffffffff8108690b>] __warn+0xcb/0xf0
 [<ffffffff810869fd>] warn_slowpath_null+0x1d/0x20
 [<ffffffffa00e8971>] assfail+0x31/0x40 [xfs]
 [<ffffffffa007602e>] xfs_btree_get_bufl+0x3e/0xb0 [xfs]
 [<ffffffffa0062e8e>] xfs_bmap_extents_to_btree+0x2ce/0x780 [xfs]
 [<ffffffffa0066a91>] xfs_bmap_add_extent_delay_real+0x1af1/0x1ce0 [xfs]
 [<ffffffffa009eb0c>] ? xfs_iext_bno_to_ext+0x8c/0x170 [xfs]
 [<ffffffffa006d06f>] xfs_bmapi_write+0xb6f/0x1180 [xfs]
 [<ffffffff810e016d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffffa00dda24>] xfs_iomap_write_allocate+0x184/0x360 [xfs]
 [<ffffffffa00b9baf>] xfs_map_blocks+0x34f/0x4e0 [xfs]
 [<ffffffffa00ba920>] xfs_do_writepage+0x310/0x840 [xfs]
 [<ffffffff811abc91>] write_cache_pages+0x251/0x640
 [<ffffffffa00ba610>] ? xfs_aops_discard_page+0x140/0x140 [xfs]
 [<ffffffffa00ba043>] xfs_vm_writepages+0xa3/0xe0 [xfs]
 [<ffffffff811acb81>] do_writepages+0x21/0x30
 [<ffffffff81252031>] __writeback_single_inode+0x61/0x800
 [<ffffffff81252cf7>] writeback_sb_inodes+0x297/0x5e0
 [<ffffffff8125329c>] wb_writeback+0xfc/0x5c0
 [<ffffffff81255e8b>] wb_workfn+0x11b/0x680
 [<ffffffff810a62d8>] process_one_work+0x1f8/0x750
 [<ffffffff810a6259>] ? process_one_work+0x179/0x750
 [<ffffffff810a687b>] worker_thread+0x4b/0x4f0
 [<ffffffff810a6830>] ? process_one_work+0x750/0x750
 [<ffffffff810ad3c2>] kthread+0x102/0x120
 [<ffffffff810ad2c0>] ? kthread_park+0x60/0x60
 [<ffffffff8168243a>] ret_from_fork+0x2a/0x40
---[ end trace a2c2fccec3607588 ]---
XFS (sdf): _xfs_buf_find: Block out of range: block 0x600000001fff8, EOFS 0x300000 
------------[ cut here ]------------
WARNING: CPU: 1 PID: 104 at /storage/home/djwong/cdev/work/linux-xfs/fs/xfs/xfs_buf.c:520 _xfs_buf_find+0x448/0x4c0 [xfs]
Modules linked in: xfs libcrc32c dax_pmem dax nd_pmem sch_fq_codel af_packetCPU: 1 PID: 104 Comm: kworker/u8:6 Tainted: G        W       4.9.0-rc5-xfsx #41
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
Workqueue: writeback wb_workfn (flush-8:80)
 ffffc9000083b288 ffffffff81340643 0000000000000000 0000000000000000
 ffffc9000083b2c8 ffffffff8108690b 000002080083b2e8 0000000000000008
 ffff88006e72a400 000600000001fff8 ffffc9000083b3c8 0000000000000001
Call Trace:
 [<ffffffff81340643>] dump_stack+0x85/0xc2
 [<ffffffff8108690b>] __warn+0xcb/0xf0
 [<ffffffff810869fd>] warn_slowpath_null+0x1d/0x20
 [<ffffffffa00c49e8>] _xfs_buf_find+0x448/0x4c0 [xfs]
 [<ffffffffa00c4e5a>] xfs_buf_get_map+0x2a/0x400 [xfs]
 [<ffffffffa0126692>] xfs_trans_get_buf_map+0x222/0x3f0 [xfs]
 [<ffffffffa0076089>] xfs_btree_get_bufl+0x99/0xb0 [xfs]
 [<ffffffffa0062e8e>] xfs_bmap_extents_to_btree+0x2ce/0x780 [xfs]
 [<ffffffffa0066a91>] xfs_bmap_add_extent_delay_real+0x1af1/0x1ce0 [xfs]
 [<ffffffffa009eb0c>] ? xfs_iext_bno_to_ext+0x8c/0x170 [xfs]
 [<ffffffffa006d06f>] xfs_bmapi_write+0xb6f/0x1180 [xfs]
 [<ffffffff810e016d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffffa00dda24>] xfs_iomap_write_allocate+0x184/0x360 [xfs]
 [<ffffffffa00b9baf>] xfs_map_blocks+0x34f/0x4e0 [xfs]
 [<ffffffffa00ba920>] xfs_do_writepage+0x310/0x840 [xfs]
 [<ffffffff811abc91>] write_cache_pages+0x251/0x640
 [<ffffffffa00ba610>] ? xfs_aops_discard_page+0x140/0x140 [xfs]
 [<ffffffffa00ba043>] xfs_vm_writepages+0xa3/0xe0 [xfs]
 [<ffffffff811acb81>] do_writepages+0x21/0x30
 [<ffffffff81252031>] __writeback_single_inode+0x61/0x800
 [<ffffffff81252cf7>] writeback_sb_inodes+0x297/0x5e0
 [<ffffffff8125329c>] wb_writeback+0xfc/0x5c0
 [<ffffffff81255e8b>] wb_workfn+0x11b/0x680
 [<ffffffff810a62d8>] process_one_work+0x1f8/0x750
 [<ffffffff810a6259>] ? process_one_work+0x179/0x750
 [<ffffffff810a687b>] worker_thread+0x4b/0x4f0
 [<ffffffff810a6830>] ? process_one_work+0x750/0x750
 [<ffffffff810ad3c2>] kthread+0x102/0x120
 [<ffffffff810ad2c0>] ? kthread_park+0x60/0x60
 [<ffffffff8168243a>] ret_from_fork+0x2a/0x40
---[ end trace a2c2fccec3607589 ]---
XFS (sdf): _xfs_buf_find: Block out of range: block 0x600000001fff8, EOFS 0x300000 
------------[ cut here ]------------
WARNING: CPU: 1 PID: 104 at /storage/home/djwong/cdev/work/linux-xfs/fs/xfs/xfs_buf.c:520 _xfs_buf_find+0x448/0x4c0 [xfs]
Modules linked in: xfs libcrc32c dax_pmem dax nd_pmem sch_fq_codel af_packet
CPU: 1 PID: 104 Comm: kworker/u8:6 Tainted: G        W       4.9.0-rc5-xfsx #41
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
Workqueue: writeback wb_workfn (flush-8:80)
 ffffc9000083b288 ffffffff81340643 0000000000000000 0000000000000000
 ffffc9000083b2c8 ffffffff8108690b 000002080083b2e8 0000000000000008
 ffff88006e72a400 000600000001fff8 ffffc9000083b3c8 0000000000000001
Call Trace:
 [<ffffffff81340643>] dump_stack+0x85/0xc2
 [<ffffffff8108690b>] __warn+0xcb/0xf0
 [<ffffffff810869fd>] warn_slowpath_null+0x1d/0x20
 [<ffffffffa00c49e8>] _xfs_buf_find+0x448/0x4c0 [xfs]
 [<ffffffffa00c5011>] xfs_buf_get_map+0x1e1/0x400 [xfs]
 [<ffffffffa0126692>] xfs_trans_get_buf_map+0x222/0x3f0 [xfs]
 [<ffffffffa0076089>] xfs_btree_get_bufl+0x99/0xb0 [xfs]
 [<ffffffffa0062e8e>] xfs_bmap_extents_to_btree+0x2ce/0x780 [xfs]
 [<ffffffffa0066a91>] xfs_bmap_add_extent_delay_real+0x1af1/0x1ce0 [xfs]
 [<ffffffffa009eb0c>] ? xfs_iext_bno_to_ext+0x8c/0x170 [xfs]
 [<ffffffffa006d06f>] xfs_bmapi_write+0xb6f/0x1180 [xfs]
 [<ffffffff810e016d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffffa00dda24>] xfs_iomap_write_allocate+0x184/0x360 [xfs]
 [<ffffffffa00b9baf>] xfs_map_blocks+0x34f/0x4e0 [xfs]
 [<ffffffffa00ba920>] xfs_do_writepage+0x310/0x840 [xfs]
 [<ffffffff811abc91>] write_cache_pages+0x251/0x640
 [<ffffffffa00ba610>] ? xfs_aops_discard_page+0x140/0x140 [xfs]
 [<ffffffffa00ba043>] xfs_vm_writepages+0xa3/0xe0 [xfs]
 [<ffffffff811acb81>] do_writepages+0x21/0x30
 [<ffffffff81252031>] __writeback_single_inode+0x61/0x800
 [<ffffffff81252cf7>] writeback_sb_inodes+0x297/0x5e0
 [<ffffffff8125329c>] wb_writeback+0xfc/0x5c0
 [<ffffffff81255e8b>] wb_workfn+0x11b/0x680
 [<ffffffff810a62d8>] process_one_work+0x1f8/0x750
 [<ffffffff810a6259>] ? process_one_work+0x179/0x750
 [<ffffffff810a687b>] worker_thread+0x4b/0x4f0
 [<ffffffff810a6830>] ? process_one_work+0x750/0x750
 [<ffffffff810ad3c2>] kthread+0x102/0x120
 [<ffffffff810ad2c0>] ? kthread_park+0x60/0x60
 [<ffffffff8168243a>] ret_from_fork+0x2a/0x40
---[ end trace a2c2fccec360758a ]---
BUG: unable to handle kernel NULL pointer dereference at 00000000000002a0
IP: [<ffffffffa0062e95>] xfs_bmap_extents_to_btree+0x2d5/0x780 [xfs]
PGD 76f16067 PUD 6f51c067 
PMD 0 
Oops: 0002 [#1] PREEMPT SMP
Dumping ftrace buffer:
   (ftrace buffer empty)
Modules linked in: xfs libcrc32c dax_pmem dax nd_pmem sch_fq_codel af_packet
CPU: 1 PID: 104 Comm: kworker/u8:6 Tainted: G        W       4.9.0-rc5-xfsx #41
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
Workqueue: writeback wb_workfn (flush-8:80)
task: ffff8800750f9680 task.stack: ffffc90000838000
RIP: 0010:[<ffffffffa0062e95>]  [<ffffffffa0062e95>] xfs_bmap_extents_to_btree+0x2d5/0x780 [xfs]
RSP: 0018:ffffc9000083b408  EFLAGS: 00010286
RAX: 0000000000000000 RBX: ffff88006be0a080 RCX: 0000000000000002
RDX: 0000000000000000 RSI: ffffffff819e97f8 RDI: 0000000000000246
RBP: ffffc9000083b520 R08: ffffea0000f8b090 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffc9000083b838
R13: ffff88006d437000 R14: 0000000000000001 R15: ffff88006e407350
FS:  0000000000000000(0000) GS:ffff88007f200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000002a0 CR3: 000000006e42f000 CR4: 00000000001406e0
Stack:
 0000000000001003 ffffc90000000009 ffffc9000083b568 0000000000000001
 0000000000000000 ffffc9000083b770 ffff880074d8c780 ffff88006e6f2db0
 ffffc9000083b848 ffff88006be0a0c8 ffff88006e407350 ffff88006d437000
Call Trace:
 [<ffffffffa0066a91>] xfs_bmap_add_extent_delay_real+0x1af1/0x1ce0 [xfs]
 [<ffffffffa009eb0c>] ? xfs_iext_bno_to_ext+0x8c/0x170 [xfs]
 [<ffffffffa006d06f>] xfs_bmapi_write+0xb6f/0x1180 [xfs]
 [<ffffffff810e016d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffffa00dda24>] xfs_iomap_write_allocate+0x184/0x360 [xfs]
 [<ffffffffa00b9baf>] xfs_map_blocks+0x34f/0x4e0 [xfs]
 [<ffffffffa00ba920>] xfs_do_writepage+0x310/0x840 [xfs]
 [<ffffffff811abc91>] write_cache_pages+0x251/0x640
 [<ffffffffa00ba610>] ? xfs_aops_discard_page+0x140/0x140 [xfs]
 [<ffffffffa00ba043>] xfs_vm_writepages+0xa3/0xe0 [xfs]
 [<ffffffff811acb81>] do_writepages+0x21/0x30
 [<ffffffff81252031>] __writeback_single_inode+0x61/0x800
 [<ffffffff81252cf7>] writeback_sb_inodes+0x297/0x5e0
 [<ffffffff8125329c>] wb_writeback+0xfc/0x5c0
 [<ffffffff81255e8b>] wb_workfn+0x11b/0x680
 [<ffffffff810a62d8>] process_one_work+0x1f8/0x750
 [<ffffffff810a6259>] ? process_one_work+0x179/0x750
 [<ffffffff810a687b>] worker_thread+0x4b/0x4f0
 [<ffffffff810a6830>] ? process_one_work+0x750/0x750
 [<ffffffff810ad3c2>] kthread+0x102/0x120
 [<ffffffff810ad2c0>] ? kthread_park+0x60/0x60
 [<ffffffff8168243a>] ret_from_fork+0x2a/0x40
Code: 00 00 00 48 83 83 08 03 00 00 01 e8 06 a3 0c 00 31 c9 4c 89 fe 4c 89 ef 48 8b 95 60 ff ff ff e8 62 31 01 00 48 89 85 08 ff ff ff <48> c7 80 a0 02 00 00 50 c8 13 a0 48 8b 80 68 01 00 00 48 89 85 
RIP  [<ffffffffa0062e95>] xfs_bmap_extents_to_btree+0x2d5/0x780 [xfs]
 RSP <ffffc9000083b408>
CR2: 00000000000002a0
---[ end trace a2c2fccec360758b ]---
------------[ cut here ]------------
WARNING: CPU: 1 PID: 104 at /storage/home/djwong/cdev/work/linux-xfs/kernel/exit.c:737 do_exit+0x4b/0xc40
Modules linked in: xfs libcrc32c dax_pmem dax nd_pmem sch_fq_codel af_packet
CPU: 1 PID: 104 Comm: kworker/u8:6 Tainted: G      D W       4.9.0-rc5-xfsx #41
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
 ffffc9000083be90 ffffffff81340643 0000000000000000 0000000000000000
 ffffc9000083bed0 ffffffff8108690b 000002e1819bf972 ffff8800750f9680
 0000000000000009 ffffc9000083b358 0000000000000002 000000000000000b
Call Trace:
 [<ffffffff81340643>] dump_stack+0x85/0xc2
 [<ffffffff8108690b>] __warn+0xcb/0xf0
 [<ffffffff810869fd>] warn_slowpath_null+0x1d/0x20
 [<ffffffff8108a99b>] do_exit+0x4b/0xc40
 [<ffffffff816836b7>] rewind_stack_do_exit+0x17/0x20
---[ end trace a2c2fccec360758c ]---
BUG: sleeping function called from invalid context at /storage/home/djwong/cdev/work/linux-xfs/include/linux/sched.h:3109
in_atomic(): 0, irqs_disabled(): 1, pid: 104, name: kworker/u8:6
INFO: lockdep is turned off.
irq event stamp: 403636
hardirqs last  enabled at (403635): [<ffffffff81205906>] kmem_cache_free+0x66/0x220
hardirqs last disabled at (403636): [<ffffffff81683489>] error_entry+0x69/0xc0
softirqs last  enabled at (403622): [<ffffffff81684c50>] __do_softirq+0x200/0x4bb
softirqs last disabled at (403615): [<ffffffff8108e979>] irq_exit+0xc9/0xd0
CPU: 1 PID: 104 Comm: kworker/u8:6 Tainted: G      D W       4.9.0-rc5-xfsx #41
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
Workqueue:  0xfffed4840f02c4f6 (�Ł��)
 ffffc9000083be58 ffffffff81340643 0000000000000000 ffff8800750f9680
 ffffc9000083be90 ffffffff810b4994 ffffffff819c0848 0000000000000c25
 0000000000000000 0000000000000002 000000000000000b ffffc9000083beb8
Call Trace:
 [<ffffffff81340643>] dump_stack+0x85/0xc2
 [<ffffffff810b4994>] ___might_sleep+0x174/0x260
 [<ffffffff810b4aca>] __might_sleep+0x4a/0x80
 [<ffffffff8109b1b4>] exit_signals+0x24/0x2c0
 [<ffffffff8108a9f4>] do_exit+0xa4/0xc40
 [<ffffffff816836b7>] rewind_stack_do_exit+0x17/0x20

Re: [BUG] dd doesn't return on ENOSPC and hang when fulfilling rmapbt XFS

[Darrick J. Wong]

On Thu, Nov 17, 2016 at 09:36:39AM -0800, Darrick J. Wong wrote:
> On Fri, Nov 18, 2016 at 12:35:15AM +0800, Eryu Guan wrote:
> > Hi all,
> > 
> > I hit a test hang in generic/224 when testing rmapbt enabled XFS on a
> > host that has non-zero sunit/swidth reported from underlying device. And
> > I simplified the reproducer to the following script, and the hang can be
> > reproduced on any host now.
> > 
> > -----
> > #!/bin/bash
> > 
> > dev=/dev/sda5
> > mnt=/mnt/xfs
> > 
> > mkfs -t xfs -m rmapbt=1 -d agcount=8,size=1g -f $dev
> 
> Hm.  I formatted with:
> mkfs.xfs -m rmapbt=1 -d sunit=4096,swidth=40960 -f /dev/sdf
> 
> (made up sunit numbers just to see how whacky it could get)
> 
> and got a different hang instead.  It looks like we are unable to
> allocate any blocks to the bmbt and various things blow up from
> there.  Will go retry with tracepoints on to see if we're running
> out of AG reservation or if we're really out of disk blocks or what.
> 
> Crash message attached at the end.

Hm.  Looking at the indlen calculations, I see that we don't include the
space that the rmapbt might need to store all the reverse mappings.  I
think this is a problem, since we decline delalloc reservations if (len
+ indlen) > fdblocks, but we potentially end up using more than indlen
blocks to map len blocks into the file, so the allocator goes nuts.

Eryu, does the following patch fix the problem you see?  I ran your
reproducer and mine and it fixed the problem in both cases.  I didn't
observe any issues running generic/224 either.

--D

---
From: Darrick J. Wong <darrick.wong@oracle.com>
Subject: [PATCH] xfs: factor rmap btree size into the indlen calculations

When we're estimating the amount of space it's going to take to satisfy
a delalloc reservation, we need to include the space that we might need
to grow the rmapbt.  This helps us to avoid running out of space later
when _iomap_write_allocate needs more space than we reserved.  Eryu Guan
observed this happening on generic/224 when sunit/swidth were set.

Reported-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 fs/xfs/libxfs/xfs_bmap.c |   17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_bmap.c b/fs/xfs/libxfs/xfs_bmap.c
index b80a294..afedf96 100644
--- a/fs/xfs/libxfs/xfs_bmap.c
+++ b/fs/xfs/libxfs/xfs_bmap.c
@@ -49,6 +49,7 @@
 #include "xfs_rmap.h"
 #include "xfs_ag_resv.h"
 #include "xfs_refcount.h"
+#include "xfs_rmap_btree.h"
 
 
 kmem_zone_t		*xfs_bmap_free_item_zone;
@@ -190,8 +191,12 @@ xfs_bmap_worst_indlen(
 	int		maxrecs;	/* maximum record count at this level */
 	xfs_mount_t	*mp;		/* mount structure */
 	xfs_filblks_t	rval;		/* return value */
+	xfs_filblks_t   orig_len;
 
 	mp = ip->i_mount;
+
+	/* Calculate the worst-case size of the bmbt. */
+	orig_len = len;
 	maxrecs = mp->m_bmap_dmxr[0];
 	for (level = 0, rval = 0;
 	     level < XFS_BM_MAXLEVELS(mp, XFS_DATA_FORK);
@@ -199,12 +204,20 @@ xfs_bmap_worst_indlen(
 		len += maxrecs - 1;
 		do_div(len, maxrecs);
 		rval += len;
-		if (len == 1)
-			return rval + XFS_BM_MAXLEVELS(mp, XFS_DATA_FORK) -
+		if (len == 1) {
+			rval += XFS_BM_MAXLEVELS(mp, XFS_DATA_FORK) -
 				level - 1;
+			break;
+		}
 		if (level == 0)
 			maxrecs = mp->m_bmap_dmxr[1];
 	}
+
+	/* Calculate the worst-case size of the rmapbt. */
+	if (xfs_sb_version_hasrmapbt(&mp->m_sb))
+		rval += 1 + xfs_rmapbt_calc_size(mp, orig_len) +
+				mp->m_rmap_maxlevels;
+
 	return rval;
 }

Re: [BUG] dd doesn't return on ENOSPC and hang when fulfilling rmapbt XFS

[Dave Chinner]

On Thu, Nov 17, 2016 at 12:11:02PM -0800, Darrick J. Wong wrote:
> On Thu, Nov 17, 2016 at 09:36:39AM -0800, Darrick J. Wong wrote:
> > On Fri, Nov 18, 2016 at 12:35:15AM +0800, Eryu Guan wrote:
> > > Hi all,
> > > 
> > > I hit a test hang in generic/224 when testing rmapbt enabled XFS on a
> > > host that has non-zero sunit/swidth reported from underlying device. And
> > > I simplified the reproducer to the following script, and the hang can be
> > > reproduced on any host now.
> > > 
> > > -----
> > > #!/bin/bash
> > > 
> > > dev=/dev/sda5
> > > mnt=/mnt/xfs
> > > 
> > > mkfs -t xfs -m rmapbt=1 -d agcount=8,size=1g -f $dev
> > 
> > Hm.  I formatted with:
> > mkfs.xfs -m rmapbt=1 -d sunit=4096,swidth=40960 -f /dev/sdf
> > 
> > (made up sunit numbers just to see how whacky it could get)
> > 
> > and got a different hang instead.  It looks like we are unable to
> > allocate any blocks to the bmbt and various things blow up from
> > there.  Will go retry with tracepoints on to see if we're running
> > out of AG reservation or if we're really out of disk blocks or what.
> > 
> > Crash message attached at the end.
> 
> Hm.  Looking at the indlen calculations, I see that we don't include the
> space that the rmapbt might need to store all the reverse mappings.  I
> think this is a problem, since we decline delalloc reservations if (len
> + indlen) > fdblocks, but we potentially end up using more than indlen
> blocks to map len blocks into the file, so the allocator goes nuts.
> 
> Eryu, does the following patch fix the problem you see?  I ran your
> reproducer and mine and it fixed the problem in both cases.  I didn't
> observe any issues running generic/224 either.
> 
> --D
> 
> ---
> From: Darrick J. Wong <darrick.wong@oracle.com>
> Subject: [PATCH] xfs: factor rmap btree size into the indlen calculations
> 
> When we're estimating the amount of space it's going to take to satisfy
> a delalloc reservation, we need to include the space that we might need
> to grow the rmapbt.  This helps us to avoid running out of space later
> when _iomap_write_allocate needs more space than we reserved.  Eryu Guan
> observed this happening on generic/224 when sunit/swidth were set.
> 
> Reported-by: Eryu Guan <eguan@redhat.com>
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  fs/xfs/libxfs/xfs_bmap.c |   17 +++++++++++++++--
>  1 file changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/xfs/libxfs/xfs_bmap.c b/fs/xfs/libxfs/xfs_bmap.c
> index b80a294..afedf96 100644
> --- a/fs/xfs/libxfs/xfs_bmap.c
> +++ b/fs/xfs/libxfs/xfs_bmap.c
> @@ -49,6 +49,7 @@
>  #include "xfs_rmap.h"
>  #include "xfs_ag_resv.h"
>  #include "xfs_refcount.h"
> +#include "xfs_rmap_btree.h"
>  
>  
>  kmem_zone_t		*xfs_bmap_free_item_zone;
> @@ -190,8 +191,12 @@ xfs_bmap_worst_indlen(
>  	int		maxrecs;	/* maximum record count at this level */
>  	xfs_mount_t	*mp;		/* mount structure */
>  	xfs_filblks_t	rval;		/* return value */
> +	xfs_filblks_t   orig_len;
>  
>  	mp = ip->i_mount;
> +
> +	/* Calculate the worst-case size of the bmbt. */
> +	orig_len = len;
>  	maxrecs = mp->m_bmap_dmxr[0];
>  	for (level = 0, rval = 0;
>  	     level < XFS_BM_MAXLEVELS(mp, XFS_DATA_FORK);
> @@ -199,12 +204,20 @@ xfs_bmap_worst_indlen(
>  		len += maxrecs - 1;
>  		do_div(len, maxrecs);
>  		rval += len;
> -		if (len == 1)
> -			return rval + XFS_BM_MAXLEVELS(mp, XFS_DATA_FORK) -
> +		if (len == 1) {
> +			rval += XFS_BM_MAXLEVELS(mp, XFS_DATA_FORK) -
>  				level - 1;
> +			break;
> +		}
>  		if (level == 0)
>  			maxrecs = mp->m_bmap_dmxr[1];
>  	}
> +
> +	/* Calculate the worst-case size of the rmapbt. */
> +	if (xfs_sb_version_hasrmapbt(&mp->m_sb))
> +		rval += 1 + xfs_rmapbt_calc_size(mp, orig_len) +
> +				mp->m_rmap_maxlevels;
> +
>  	return rval;
>  }

So, I wrote an identical patch back when rmap was first merged and
224 was failing, but the ENOSPC problem I was hitting with 224 went
away with the reflink merge a cycle later.

This change then uncovered a separate failure with rmap+delalloc
reservation: when we allocate the delalloc extent, we release the
entire remaining indlen reservation, and so it's not available for
the rmapbt allocation that it was reserved for. Hence we still
get lockups at ENOSPC. There were a series of patches to address
this that I never finished - let me go forward port it, smoke test
it and repost it....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

[PATCH 0/4] xfs: fix rmapbt ENOSPC hangs

[Dave Chinner]

Darrick,

This is the patchset I write to address the rmapbt enospc issue
that generic/224 was tripping across. I replaced the first patch
with yours as it's cleaner, and forward ported the other patches.

I stopped working on this because the reflink merge fixed the
problems I was seeing. maybe that was because I stopped testing
rmapbt=1 by itself, though, so the problem never really got fixed.

I'm running it through an auto test run right now, but I thought
I'd post it now so you can comment on the next three patches in the
series. Also to see if it passes the enospc tests on your setups...

Cheers,

Dave.

[PATCH 1/4] xfs: factor rmap btree size into the indlen calculations

[Dave Chinner]

From: "Darrick J. Wong" <darrick.wong@oracle.com>

When we're estimating the amount of space it's going to take to satisfy
a delalloc reservation, we need to include the space that we might need
to grow the rmapbt.  This helps us to avoid running out of space later
when _iomap_write_allocate needs more space than we reserved.  Eryu Guan
observed this happening on generic/224 when sunit/swidth were set.

Reported-by: Eryu Guan <eguan@redhat.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
---
 fs/xfs/libxfs/xfs_bmap.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_bmap.c b/fs/xfs/libxfs/xfs_bmap.c
index 5c3c4dd14735..00188f559c8d 100644
--- a/fs/xfs/libxfs/xfs_bmap.c
+++ b/fs/xfs/libxfs/xfs_bmap.c
@@ -49,6 +49,7 @@
 #include "xfs_rmap.h"
 #include "xfs_ag_resv.h"
 #include "xfs_refcount.h"
+#include "xfs_rmap_btree.h"
 
 
 kmem_zone_t		*xfs_bmap_free_item_zone;
@@ -190,8 +191,12 @@ xfs_bmap_worst_indlen(
 	int		maxrecs;	/* maximum record count at this level */
 	xfs_mount_t	*mp;		/* mount structure */
 	xfs_filblks_t	rval;		/* return value */
+	xfs_filblks_t   orig_len;
 
 	mp = ip->i_mount;
+
+	/* Calculate the worst-case size of the bmbt. */
+	orig_len = len;
 	maxrecs = mp->m_bmap_dmxr[0];
 	for (level = 0, rval = 0;
 	     level < XFS_BM_MAXLEVELS(mp, XFS_DATA_FORK);
@@ -199,12 +204,20 @@ xfs_bmap_worst_indlen(
 		len += maxrecs - 1;
 		do_div(len, maxrecs);
 		rval += len;
-		if (len == 1)
-			return rval + XFS_BM_MAXLEVELS(mp, XFS_DATA_FORK) -
+		if (len == 1) {
+			rval += XFS_BM_MAXLEVELS(mp, XFS_DATA_FORK) -
 				level - 1;
+			break;
+		}
 		if (level == 0)
 			maxrecs = mp->m_bmap_dmxr[1];
 	}
+
+	/* Calculate the worst-case size of the rmapbt. */
+	if (xfs_sb_version_hasrmapbt(&mp->m_sb))
+		rval += 1 + xfs_rmapbt_calc_size(mp, orig_len) +
+				mp->m_rmap_maxlevels;
+
 	return rval;
 }
 
-- 
2.8.0.rc3

[PATCH 2/4] xfs: add more AGF/AGFL manipulation tracepoints

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

So we can see what blocks are added to or removed from the AGFL during
allocation.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 fs/xfs/libxfs/xfs_alloc.c |  8 ++++++--
 fs/xfs/xfs_trace.h        | 11 ++++++++++-
 2 files changed, 16 insertions(+), 3 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_alloc.c b/fs/xfs/libxfs/xfs_alloc.c
index effb64cf714f..a5a9d8360e74 100644
--- a/fs/xfs/libxfs/xfs_alloc.c
+++ b/fs/xfs/libxfs/xfs_alloc.c
@@ -2264,10 +2264,12 @@ xfs_alloc_get_freelist(
 	xfs_mount_t	*mp = tp->t_mountp;
 	xfs_perag_t	*pag;	/* per allocation group data */
 
+	agf = XFS_BUF_TO_AGF(agbp);
+	trace_xfs_alloc_get_freelist(mp, agf, 0, _RET_IP_);
+
 	/*
 	 * Freelist is empty, give up.
 	 */
-	agf = XFS_BUF_TO_AGF(agbp);
 	if (!agf->agf_flcount) {
 		*bnop = NULLAGBLOCK;
 		return 0;
@@ -2392,8 +2394,9 @@ xfs_alloc_put_freelist(
 	__be32			*agfl_bno;
 	int			startoff;
 
-	agf = XFS_BUF_TO_AGF(agbp);
 	mp = tp->t_mountp;
+	agf = XFS_BUF_TO_AGF(agbp);
+	trace_xfs_alloc_put_freelist(mp, agf, 0, _RET_IP_);
 
 	if (!agflbp && (error = xfs_alloc_read_agfl(mp, tp,
 			be32_to_cpu(agf->agf_seqno), &agflbp)))
@@ -2555,6 +2558,7 @@ xfs_read_agf(
 	if (!*bpp)
 		return 0;
 
+	trace_xfs_read_agf_detail(mp, XFS_BUF_TO_AGF(*bpp), 0, _RET_IP_);
 	ASSERT(!(*bpp)->b_error);
 	xfs_buf_set_ref(*bpp, XFS_AGF_REF);
 	return 0;
diff --git a/fs/xfs/xfs_trace.h b/fs/xfs/xfs_trace.h
index 0907752be62d..73e001d795ce 100644
--- a/fs/xfs/xfs_trace.h
+++ b/fs/xfs/xfs_trace.h
@@ -1516,7 +1516,7 @@ TRACE_EVENT(xfs_trans_commit_lsn,
 		  __entry->lsn)
 );
 
-TRACE_EVENT(xfs_agf,
+DECLARE_EVENT_CLASS(xfs_agf_class,
 	TP_PROTO(struct xfs_mount *mp, struct xfs_agf *agf, int flags,
 		 unsigned long caller_ip),
 	TP_ARGS(mp, agf, flags, caller_ip),
@@ -1572,6 +1572,15 @@ TRACE_EVENT(xfs_agf,
 		  __entry->longest,
 		  (void *)__entry->caller_ip)
 );
+#define DEFINE_AGF_EVENT(name) \
+DEFINE_EVENT(xfs_agf_class, name, \
+	TP_PROTO(struct xfs_mount *mp, struct xfs_agf *agf, int flags, \
+		 unsigned long caller_ip), \
+	TP_ARGS(mp, agf, flags, caller_ip))
+DEFINE_AGF_EVENT(xfs_agf);
+DEFINE_AGF_EVENT(xfs_read_agf_detail);
+DEFINE_AGF_EVENT(xfs_alloc_get_freelist);
+DEFINE_AGF_EVENT(xfs_alloc_put_freelist);
 
 TRACE_EVENT(xfs_free_extent,
 	TP_PROTO(struct xfs_mount *mp, xfs_agnumber_t agno, xfs_agblock_t agbno,
-- 
2.8.0.rc3

[PATCH 3/4] xfs: hold AGF buffers over defer ops

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

We make space and AGFL adjustments in the initial transaction for
rmap defer ops to complete, but those reservations are only valid
while we hold the AGF locked. The issue here is that if we drop the
AGF lock, we can race withother operations that need AGFL
adjustments for their deferred rmap operations, and so we can get
multiple deferred ops running at the same that share a single AGFL
reservation instead of having one each. With enough defered rmap ops
running at the same time, we run the AGFL out of free blocks and
fail rmap insertions, resulting in  corruption shutdowns.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 fs/xfs/libxfs/xfs_bmap.c  | 14 ++++++++++
 fs/xfs/libxfs/xfs_defer.c | 69 ++++++++++++++++++++++++++++++++++++++++++++---
 fs/xfs/libxfs/xfs_defer.h |  6 +++++
 fs/xfs/xfs_trans_buf.c    | 12 ++++++++-
 4 files changed, 96 insertions(+), 5 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_bmap.c b/fs/xfs/libxfs/xfs_bmap.c
index 00188f559c8d..6211b4b5e826 100644
--- a/fs/xfs/libxfs/xfs_bmap.c
+++ b/fs/xfs/libxfs/xfs_bmap.c
@@ -3932,6 +3932,20 @@ xfs_bmap_btalloc(
 			ap->wasdel ? XFS_TRANS_DQ_DELBCOUNT :
 					XFS_TRANS_DQ_BCOUNT,
 			(long) args.len);
+
+		/*
+		 * If we are going to update the rmap btree, we need to join the
+		 * AGF used for this allocation to the defer ops processing so
+		 * that it stays locked until we've processed all the relevant
+		 * btree updates that are required.
+		 */
+		if (xfs_sb_version_hasrmapbt(&mp->m_sb)) {
+			ASSERT(args.agbp);
+			error = xfs_defer_join_buf(ap->tp, ap->dfops,
+						   args.agbp);
+			if (error)
+				return error;
+		}
 	} else {
 		ap->blkno = NULLFSBLOCK;
 		ap->length = 0;
diff --git a/fs/xfs/libxfs/xfs_defer.c b/fs/xfs/libxfs/xfs_defer.c
index 5c2929f94bd3..1e5a8b04c7f5 100644
--- a/fs/xfs/libxfs/xfs_defer.c
+++ b/fs/xfs/libxfs/xfs_defer.c
@@ -271,6 +271,13 @@ xfs_defer_trans_roll(
 		xfs_trans_ijoin(*tp, dop->dop_inodes[i], 0);
 	}
 
+	/* Rejoin the held buffers */
+	for (i = 0; i < XFS_DEFER_OPS_NR_BUFS && dop->dop_bufs[i]; i++) {
+		ASSERT(dop->dop_bufs[i]->b_transp == NULL);
+		xfs_trans_bjoin(*tp, dop->dop_bufs[i]);
+		xfs_trans_bhold(*tp, dop->dop_bufs[i]);
+	}
+
 	return error;
 }
 
@@ -297,7 +304,7 @@ xfs_defer_join(
 	for (i = 0; i < XFS_DEFER_OPS_NR_INODES; i++) {
 		if (dop->dop_inodes[i] == ip)
 			return 0;
-		else if (dop->dop_inodes[i] == NULL) {
+		if (!dop->dop_inodes[i]) {
 			dop->dop_inodes[i] = ip;
 			return 0;
 		}
@@ -307,6 +314,60 @@ xfs_defer_join(
 }
 
 /*
+ * Add this buffer to the deferred op.  If this is a new buffer we are joining
+ * to the deferred ops, we need to ensure it will be held locked across
+ * transaction commits. This means each joined buffer remains locked until
+ * the final defer op commits and we clean up the xfs_defer_ops structure. This
+ * ensures atomicity of access to buffers and their state while we perform
+ * multiple operations to them through deferred ops processing.
+ */
+int
+xfs_defer_join_buf(
+	struct xfs_trans	*tp,
+	struct xfs_defer_ops	*dop,
+	struct xfs_buf		*bp)
+{
+	int				i;
+
+	for (i = 0; i < XFS_DEFER_OPS_NR_BUFS; i++) {
+		if (dop->dop_bufs[i] == bp)
+			return 0;
+		if (!dop->dop_bufs[i]) {
+			xfs_trans_bhold(tp, bp);
+			dop->dop_bufs[i] = bp;
+			return 0;
+		}
+	}
+
+	return -EFSCORRUPTED;
+}
+
+/*
+ * When we are all done with the defer processing, the buffers we hold
+ * will still be locked. We need to unlock and release them now.
+ *
+ * We can get called in from a context that doesn't pass a transaction,
+ * but the buffers are still attached to a transaction. If this happens,
+ * pull the transaction from the buffer. If the buffer is not part of a
+ * transaction, then b_transp will be null and the buffer will be released
+ * normally.
+ */
+static void
+xfs_defer_brelse(
+	struct xfs_trans	*tp,
+	struct xfs_defer_ops	*dop)
+{
+	int				i;
+
+	for (i = 0; i < XFS_DEFER_OPS_NR_BUFS; i++) {
+		if (!tp)
+			tp = dop->dop_bufs[i]->b_transp;
+		if (dop->dop_bufs[i])
+			xfs_trans_bhold_release(tp, dop->dop_bufs[i]);
+	}
+}
+
+/*
  * Finish all the pending work.  This involves logging intent items for
  * any work items that wandered in since the last transaction roll (if
  * one has even happened), rolling the transaction, and finishing the
@@ -402,6 +463,7 @@ xfs_defer_finish(
 	}
 
 out:
+	xfs_defer_brelse(*tp, dop);
 	if (error)
 		trace_xfs_defer_finish_error((*tp)->t_mountp, dop, error);
 	else
@@ -449,6 +511,7 @@ xfs_defer_cancel(
 		ASSERT(dfp->dfp_count == 0);
 		kmem_free(dfp);
 	}
+	xfs_defer_brelse(NULL, dop);
 }
 
 /* Add an item for later deferred processing. */
@@ -502,9 +565,7 @@ xfs_defer_init(
 	struct xfs_defer_ops		*dop,
 	xfs_fsblock_t			*fbp)
 {
-	dop->dop_committed = false;
-	dop->dop_low = false;
-	memset(&dop->dop_inodes, 0, sizeof(dop->dop_inodes));
+	memset(dop, 0, sizeof(*dop));
 	*fbp = NULLFSBLOCK;
 	INIT_LIST_HEAD(&dop->dop_intake);
 	INIT_LIST_HEAD(&dop->dop_pending);
diff --git a/fs/xfs/libxfs/xfs_defer.h b/fs/xfs/libxfs/xfs_defer.h
index f6e93ef0bffe..036fde7d839b 100644
--- a/fs/xfs/libxfs/xfs_defer.h
+++ b/fs/xfs/libxfs/xfs_defer.h
@@ -59,6 +59,7 @@ enum xfs_defer_ops_type {
 };
 
 #define XFS_DEFER_OPS_NR_INODES	2	/* join up to two inodes */
+#define XFS_DEFER_OPS_NR_BUFS	2	/* join up to two buffers */
 
 struct xfs_defer_ops {
 	bool			dop_committed;	/* did any trans commit? */
@@ -68,6 +69,9 @@ struct xfs_defer_ops {
 
 	/* relog these inodes with each roll */
 	struct xfs_inode	*dop_inodes[XFS_DEFER_OPS_NR_INODES];
+
+	/* hold these buffers with each roll */
+	struct xfs_buf		*dop_bufs[XFS_DEFER_OPS_NR_BUFS];
 };
 
 void xfs_defer_add(struct xfs_defer_ops *dop, enum xfs_defer_ops_type type,
@@ -78,6 +82,8 @@ void xfs_defer_cancel(struct xfs_defer_ops *dop);
 void xfs_defer_init(struct xfs_defer_ops *dop, xfs_fsblock_t *fbp);
 bool xfs_defer_has_unfinished_work(struct xfs_defer_ops *dop);
 int xfs_defer_join(struct xfs_defer_ops *dop, struct xfs_inode *ip);
+int xfs_defer_join_buf(struct xfs_trans *tp, struct xfs_defer_ops *dop,
+		struct xfs_buf *bp);
 
 /* Description of a deferred type. */
 struct xfs_defer_op_type {
diff --git a/fs/xfs/xfs_trans_buf.c b/fs/xfs/xfs_trans_buf.c
index 8ee29ca132dc..f88552182256 100644
--- a/fs/xfs/xfs_trans_buf.c
+++ b/fs/xfs/xfs_trans_buf.c
@@ -66,6 +66,12 @@ xfs_trans_buf_item_match(
  * Add the locked buffer to the transaction.
  *
  * The buffer must be locked, and it cannot be associated with any
+ * transaction other than the one we pass in. This "join" recursion
+ * is a result of needing to hold buffers locked across multiple transactions in
+ * the defer ops processing. To hold the buffer when rolling the transaction we
+ * need to first join the buffer to transaction, and hence when this gets done
+ * in the normal course of the transaction we then get called a second time.
+ * Hence just do nothing if bp->b_transp already matches the incoming
  * transaction.
  *
  * If the buffer does not yet have a buf log item associated with it,
@@ -79,7 +85,11 @@ _xfs_trans_bjoin(
 {
 	struct xfs_buf_log_item	*bip;
 
-	ASSERT(bp->b_transp == NULL);
+	ASSERT(!bp->b_transp || bp->b_transp == tp);
+
+	/* already joined to this transaction from defer ops? */
+	if (bp->b_transp == tp)
+		return;
 
 	/*
 	 * The xfs_buf_log_item pointer is stored in b_fsprivate.  If
-- 
2.8.0.rc3

[PATCH 4/4] xfs: defer indirect delalloc rmap reservations

[Dave Chinner]

From: Dave Chinner <dchinner@redhat.com>

When we do rmap additions for delalloc extents, we need to ensure we
have space reserved for them. We do this by keeping the space
required in the indirect length associated with a delalloc extent.

However, when we allocate the extent, we immediately release the
unused portion of the indlen reservation, and hence when we come to
needing it when processing the deferred rmap btree insertion, it's
no longer available. Because it gets returned to the global free
space pool, other delalloc reservations can take it before we use
it, resulting in having no space available to fix up the free list
to the correct length before doing the rmap insertion. This results
in an insertion failure and shutdown.

To avoid this problem, rather than releasing the unused indlen
reservation, store it in the transaction to be released when the
transaction is finally committed. When we roll a transaction during
defer ops processing, we transfer the unused block reservation to
the new transaction before we commit the old one. This keeps the
unused reservation local to the deferred ops context. On final
commit, the unused reservation space can be returned to the global
pool.

The final piece of the puzzle is hooking this up to the free list
fixup that ensures we have enough blocks on the free list for the
rmap insert. In this case, ensure that xfs_rmapbt_alloc_block()
always decrements a block from the reservation on the transaction.
This will track the number of blocks we've actually consumed from
the free list for the rmapbt, hence ensuring that we accurately
account of those blocks when the final transaction commit occurs.

Because we now hold the delalloc rmapbt block reservation until
we've done all the rmapbt block allocation, we should not see ENOSPC
problems as a result of the AGFL being emptied during rmap btree
insertion operations.

Signed-off-by: Dave Chinner <dchinner@redhat.com>
---
 fs/xfs/libxfs/xfs_alloc.c | 14 ++++++++++++++
 fs/xfs/libxfs/xfs_bmap.c  | 16 ++++++++++++----
 fs/xfs/xfs_trans.c        | 28 ++++++++++++++++++++++++++++
 fs/xfs/xfs_trans.h        |  1 +
 4 files changed, 55 insertions(+), 4 deletions(-)

diff --git a/fs/xfs/libxfs/xfs_alloc.c b/fs/xfs/libxfs/xfs_alloc.c
index a5a9d8360e74..e835bf24a85b 100644
--- a/fs/xfs/libxfs/xfs_alloc.c
+++ b/fs/xfs/libxfs/xfs_alloc.c
@@ -2228,6 +2228,20 @@ xfs_alloc_fix_freelist(
 							agflbp, bno, 0);
 			if (error)
 				goto out_agflbp_relse;
+
+			/*
+			 * If we've just done a delayed allocation and now we
+			 * are processing a deferred metadata update (such as an
+			 * rmapbt update), we'll have a space reservation for
+			 * the rmapbt blocks that may be needed. These are
+			 * allocated from the freelist, so account for them here
+			 * when we refill the AGFL. We've held the AGF locked
+			 * across the defered transactions, so this should only
+			 * be refilling blocks we consumed from the AGFL in the
+			 * preceeding transaction.
+			 */
+			if (tp->t_blk_deferred)
+				tp->t_blk_deferred--;
 		}
 	}
 	xfs_trans_brelse(tp, agflbp);
diff --git a/fs/xfs/libxfs/xfs_bmap.c b/fs/xfs/libxfs/xfs_bmap.c
index 6211b4b5e826..f1db9a03e4c4 100644
--- a/fs/xfs/libxfs/xfs_bmap.c
+++ b/fs/xfs/libxfs/xfs_bmap.c
@@ -2203,10 +2203,19 @@ xfs_bmap_add_extent_delay_real(
 		temp2 = xfs_bmap_worst_indlen(bma->ip, temp2);
 		diff = (int)(temp + temp2 - startblockval(PREV.br_startblock) -
 			(bma->cur ? bma->cur->bc_private.b.allocated : 0));
+
 		if (diff > 0) {
+			/*
+			 * XXX (dgc): Ouch! Pulling more blocks from the free pool
+			 * during allocation during delalloc split. This will
+			 * fail at ENOSPC and it screws up rmapbt space
+			 * accounting. We need to know when this happens so
+			 * we can isolate the typical causes of reservation
+			 * underruns so that they never happen in production.
+			 */
+			ASSERT(0);
 			error = xfs_mod_fdblocks(bma->ip->i_mount,
 						 -((int64_t)diff), false);
-			ASSERT(!error);
 			if (error)
 				goto done;
 		}
@@ -2261,8 +2270,7 @@ xfs_bmap_add_extent_delay_real(
 			temp += bma->cur->bc_private.b.allocated;
 		ASSERT(temp <= da_old);
 		if (temp < da_old)
-			xfs_mod_fdblocks(bma->ip->i_mount,
-					(int64_t)(da_old - temp), false);
+			bma->tp->t_blk_deferred += (int64_t)(da_old - temp);
 	}
 
 	/* clear out the allocated field, done with it now in any case. */
@@ -5437,7 +5445,7 @@ xfs_bmap_del_extent(
 	 */
 	ASSERT(da_old >= da_new);
 	if (da_old > da_new)
-		xfs_mod_fdblocks(mp, (int64_t)(da_old - da_new), false);
+		tp->t_blk_deferred += (int64_t)(da_old - da_new);
 done:
 	*logflagsp = flags;
 	return error;
diff --git a/fs/xfs/xfs_trans.c b/fs/xfs/xfs_trans.c
index 70f42ea86dfb..0728ff7a04ab 100644
--- a/fs/xfs/xfs_trans.c
+++ b/fs/xfs/xfs_trans.c
@@ -103,6 +103,16 @@ xfs_trans_dup(
 	tp->t_rtx_res = tp->t_rtx_res_used;
 	ntp->t_pflags = tp->t_pflags;
 
+	/*
+	 * Transfer deferred block reservations to new transaction so the remain
+	 * available to the ongoing deferred ops processing. We clear the
+	 * existing transaction count so that the deferred block reservation is
+	 * not released when that transaction is committed (i.e. it's not a
+	 * regrantable reservation).
+	 */
+	ntp->t_blk_deferred = tp->t_blk_deferred;
+	tp->t_blk_deferred = 0;
+
 	xfs_trans_dup_dqinfo(tp, ntp);
 
 	atomic_inc(&tp->t_mountp->m_active_trans);
@@ -680,6 +690,21 @@ xfs_trans_unreserve_and_mod_sb(
 }
 
 /*
+ * Release unused defered block reservations back to the global free space pool.
+ * These blocks came from the in-core counter, so return them there.
+ */
+static void
+xfs_trans_release_deferred_blocks(
+	struct xfs_trans	*tp)
+{
+	ASSERT(tp->t_blk_deferred >= 0);
+	if (!tp->t_blk_deferred)
+		return;
+	xfs_mod_fdblocks(tp->t_mountp, tp->t_blk_deferred,
+			 !!(tp->t_flags & XFS_TRANS_RESERVE));
+}
+
+/*
  * Add the given log item to the transaction's list of log items.
  *
  * The log item will now point to its new descriptor with its li_desc field.
@@ -908,6 +933,7 @@ __xfs_trans_commit(
 	/*
 	 * If we need to update the superblock, then do it now.
 	 */
+	xfs_trans_release_deferred_blocks(tp);
 	if (tp->t_flags & XFS_TRANS_SB_DIRTY)
 		xfs_trans_apply_sb_deltas(tp);
 	xfs_trans_apply_dquot_deltas(tp);
@@ -931,6 +957,7 @@ __xfs_trans_commit(
 	return error;
 
 out_unreserve:
+	xfs_trans_release_deferred_blocks(tp);
 	xfs_trans_unreserve_and_mod_sb(tp);
 
 	/*
@@ -991,6 +1018,7 @@ xfs_trans_cancel(
 			ASSERT(!(lidp->lid_item->li_type == XFS_LI_EFD));
 	}
 #endif
+	xfs_trans_release_deferred_blocks(tp);
 	xfs_trans_unreserve_and_mod_sb(tp);
 	xfs_trans_unreserve_and_mod_dquots(tp);
 
diff --git a/fs/xfs/xfs_trans.h b/fs/xfs/xfs_trans.h
index 61b7fbdd3ebd..6126e6fb9f5c 100644
--- a/fs/xfs/xfs_trans.h
+++ b/fs/xfs/xfs_trans.h
@@ -132,6 +132,7 @@ typedef struct xfs_trans {
 	int64_t			t_rblocks_delta;/* superblock rblocks change */
 	int64_t			t_rextents_delta;/* superblocks rextents chg */
 	int64_t			t_rextslog_delta;/* superblocks rextslog chg */
+	int64_t			t_blk_deferred;	/* blocks for deferred ops */
 	struct list_head	t_items;	/* log item descriptors */
 	struct list_head	t_busy;		/* list of busy extents */
 	unsigned long		t_pflags;	/* saved process flags state */
-- 
2.8.0.rc3

Re: [PATCH 3/4] xfs: hold AGF buffers over defer ops

[Dave Chinner]

On Fri, Nov 18, 2016 at 10:55:34AM +1100, Dave Chinner wrote:
> +static void
> +xfs_defer_brelse(
> +	struct xfs_trans	*tp,
> +	struct xfs_defer_ops	*dop)
> +{
> +	int				i;
> +
> +	for (i = 0; i < XFS_DEFER_OPS_NR_BUFS; i++) {
> +		if (!tp)
> +			tp = dop->dop_bufs[i]->b_transp;
> +		if (dop->dop_bufs[i])
> +			xfs_trans_bhold_release(tp, dop->dop_bufs[i]);
> +	}

This let's the smoke out in generic/139. Will fix...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [BUG] dd doesn't return on ENOSPC and hang when fulfilling rmapbt XFS

[Eryu Guan]

On Thu, Nov 17, 2016 at 12:11:02PM -0800, Darrick J. Wong wrote:
> On Thu, Nov 17, 2016 at 09:36:39AM -0800, Darrick J. Wong wrote:
> > On Fri, Nov 18, 2016 at 12:35:15AM +0800, Eryu Guan wrote:
> > > Hi all,
> > > 
> > > I hit a test hang in generic/224 when testing rmapbt enabled XFS on a
> > > host that has non-zero sunit/swidth reported from underlying device. And
> > > I simplified the reproducer to the following script, and the hang can be
> > > reproduced on any host now.
> > > 
> > > -----
> > > #!/bin/bash
> > > 
> > > dev=/dev/sda5
> > > mnt=/mnt/xfs
> > > 
> > > mkfs -t xfs -m rmapbt=1 -d agcount=8,size=1g -f $dev
> > 
> > Hm.  I formatted with:
> > mkfs.xfs -m rmapbt=1 -d sunit=4096,swidth=40960 -f /dev/sdf
> > 
> > (made up sunit numbers just to see how whacky it could get)
> > 
> > and got a different hang instead.  It looks like we are unable to
> > allocate any blocks to the bmbt and various things blow up from
> > there.  Will go retry with tracepoints on to see if we're running
> > out of AG reservation or if we're really out of disk blocks or what.
> > 
> > Crash message attached at the end.
> 
> Hm.  Looking at the indlen calculations, I see that we don't include the
> space that the rmapbt might need to store all the reverse mappings.  I
> think this is a problem, since we decline delalloc reservations if (len
> + indlen) > fdblocks, but we potentially end up using more than indlen
> blocks to map len blocks into the file, so the allocator goes nuts.
> 
> Eryu, does the following patch fix the problem you see?  I ran your
> reproducer and mine and it fixed the problem in both cases.  I didn't
> observe any issues running generic/224 either.

I applied your patch (and only your patch, patches posted by Dave were
not included) on top of 4.9-rc5 kernel, and it passed my simplified
reproducer, but still failed generic/224 with

MKFS_OPTIONS="-b size=4k -m crc=1,rmapbt=1 -d agcount=8"

Not all the time, but easily to hit. And sysrq-w showed the same traces
as before.

SECTION       -- xfs_test
RECREATING    -- xfs on /dev/mapper/testvg-testlv1
FSTYP         -- xfs (non-debug)
PLATFORM      -- Linux/x86_64 ibm-x3550m3-05 4.9.0-rc5+
MKFS_OPTIONS  -- -f -f -b size=4k -m crc=1,rmapbt=1 -d agcount=8 /dev/mapper/testvg-testlv2
MOUNT_OPTIONS -- -o context=system_u:object_r:nfs_t:s0 /dev/mapper/testvg-testlv2 /mnt/testarea/scratch

generic/224 16s ...  <===== never return

My local.config file:
[default]
TEST_DEV=/dev/mapper/testvg-testlv1
TEST_DIR=/mnt/testarea/test
SCRATCH_MNT=/mnt/testarea/scratch
SCRATCH_DEV=/dev/mapper/testvg-testlv2

[xfs_test]
FSTYP=xfs
MKFS_OPTIONS="-f -b size=4k -m crc=1,rmapbt=1 -d agcount=8"
# other unrelated configs follow

But this patch does make some differences. Prior to this patch, I saw
thousands of dd processes hang, now there's only one or two.

Thanks,
Eryu

> 
> --D
> 
> ---
> From: Darrick J. Wong <darrick.wong@oracle.com>
> Subject: [PATCH] xfs: factor rmap btree size into the indlen calculations
> 
> When we're estimating the amount of space it's going to take to satisfy
> a delalloc reservation, we need to include the space that we might need
> to grow the rmapbt.  This helps us to avoid running out of space later
> when _iomap_write_allocate needs more space than we reserved.  Eryu Guan
> observed this happening on generic/224 when sunit/swidth were set.
> 
> Reported-by: Eryu Guan <eguan@redhat.com>
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  fs/xfs/libxfs/xfs_bmap.c |   17 +++++++++++++++--
>  1 file changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/xfs/libxfs/xfs_bmap.c b/fs/xfs/libxfs/xfs_bmap.c
> index b80a294..afedf96 100644
> --- a/fs/xfs/libxfs/xfs_bmap.c
> +++ b/fs/xfs/libxfs/xfs_bmap.c
> @@ -49,6 +49,7 @@
>  #include "xfs_rmap.h"
>  #include "xfs_ag_resv.h"
>  #include "xfs_refcount.h"
> +#include "xfs_rmap_btree.h"
>  
>  
>  kmem_zone_t		*xfs_bmap_free_item_zone;
> @@ -190,8 +191,12 @@ xfs_bmap_worst_indlen(
>  	int		maxrecs;	/* maximum record count at this level */
>  	xfs_mount_t	*mp;		/* mount structure */
>  	xfs_filblks_t	rval;		/* return value */
> +	xfs_filblks_t   orig_len;
>  
>  	mp = ip->i_mount;
> +
> +	/* Calculate the worst-case size of the bmbt. */
> +	orig_len = len;
>  	maxrecs = mp->m_bmap_dmxr[0];
>  	for (level = 0, rval = 0;
>  	     level < XFS_BM_MAXLEVELS(mp, XFS_DATA_FORK);
> @@ -199,12 +204,20 @@ xfs_bmap_worst_indlen(
>  		len += maxrecs - 1;
>  		do_div(len, maxrecs);
>  		rval += len;
> -		if (len == 1)
> -			return rval + XFS_BM_MAXLEVELS(mp, XFS_DATA_FORK) -
> +		if (len == 1) {
> +			rval += XFS_BM_MAXLEVELS(mp, XFS_DATA_FORK) -
>  				level - 1;
> +			break;
> +		}
>  		if (level == 0)
>  			maxrecs = mp->m_bmap_dmxr[1];
>  	}
> +
> +	/* Calculate the worst-case size of the rmapbt. */
> +	if (xfs_sb_version_hasrmapbt(&mp->m_sb))
> +		rval += 1 + xfs_rmapbt_calc_size(mp, orig_len) +
> +				mp->m_rmap_maxlevels;
> +
>  	return rval;
>  }

Re: [BUG] dd doesn't return on ENOSPC and hang when fulfilling rmapbt XFS

[Dave Chinner]

On Fri, Nov 18, 2016 at 01:26:33PM +0800, Eryu Guan wrote:
> On Thu, Nov 17, 2016 at 12:11:02PM -0800, Darrick J. Wong wrote:
> > On Thu, Nov 17, 2016 at 09:36:39AM -0800, Darrick J. Wong wrote:
> > > On Fri, Nov 18, 2016 at 12:35:15AM +0800, Eryu Guan wrote:
> > > > Hi all,
> > > > 
> > > > I hit a test hang in generic/224 when testing rmapbt enabled XFS on a
> > > > host that has non-zero sunit/swidth reported from underlying device. And
> > > > I simplified the reproducer to the following script, and the hang can be
> > > > reproduced on any host now.
> > > > 
> > > > -----
> > > > #!/bin/bash
> > > > 
> > > > dev=/dev/sda5
> > > > mnt=/mnt/xfs
> > > > 
> > > > mkfs -t xfs -m rmapbt=1 -d agcount=8,size=1g -f $dev
> > > 
> > > Hm.  I formatted with:
> > > mkfs.xfs -m rmapbt=1 -d sunit=4096,swidth=40960 -f /dev/sdf
> > > 
> > > (made up sunit numbers just to see how whacky it could get)
> > > 
> > > and got a different hang instead.  It looks like we are unable to
> > > allocate any blocks to the bmbt and various things blow up from
> > > there.  Will go retry with tracepoints on to see if we're running
> > > out of AG reservation or if we're really out of disk blocks or what.
> > > 
> > > Crash message attached at the end.
> > 
> > Hm.  Looking at the indlen calculations, I see that we don't include the
> > space that the rmapbt might need to store all the reverse mappings.  I
> > think this is a problem, since we decline delalloc reservations if (len
> > + indlen) > fdblocks, but we potentially end up using more than indlen
> > blocks to map len blocks into the file, so the allocator goes nuts.
> > 
> > Eryu, does the following patch fix the problem you see?  I ran your
> > reproducer and mine and it fixed the problem in both cases.  I didn't
> > observe any issues running generic/224 either.
> 
> I applied your patch (and only your patch, patches posted by Dave were
> not included) on top of 4.9-rc5 kernel, and it passed my simplified
> reproducer, but still failed generic/224 with
> 
> MKFS_OPTIONS="-b size=4k -m crc=1,rmapbt=1 -d agcount=8"
> 
> Not all the time, but easily to hit. And sysrq-w showed the same traces
> as before.
> 
> SECTION       -- xfs_test
> RECREATING    -- xfs on /dev/mapper/testvg-testlv1
> FSTYP         -- xfs (non-debug)
> PLATFORM      -- Linux/x86_64 ibm-x3550m3-05 4.9.0-rc5+
> MKFS_OPTIONS  -- -f -f -b size=4k -m crc=1,rmapbt=1 -d agcount=8 /dev/mapper/testvg-testlv2
> MOUNT_OPTIONS -- -o context=system_u:object_r:nfs_t:s0 /dev/mapper/testvg-testlv2 /mnt/testarea/scratch
> 
> generic/224 16s ...  <===== never return

My patchset does pass generic/224 here, but it fails lots of other tests
because of an accounting problem I've not yet found.

SECTION       -- xfs
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 test2 4.9.0-rc4-dgc+
MKFS_OPTIONS  -- -f -m rmapbt=1 -i sparse=1 /dev/sdg
MOUNT_OPTIONS -- -o context=system_u:object_r:nfs_t:s0 /dev/sdg /mnt/scratch

generic/224 25s ... 24s
Ran: generic/224
Passed all 1 tests

SECTION       -- xfs
=========================
Ran: generic/224
Passed all 1 tests

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [BUG] dd doesn't return on ENOSPC and hang when fulfilling rmapbt XFS

[Eryu Guan]

On Fri, Nov 18, 2016 at 04:46:20PM +1100, Dave Chinner wrote:
> On Fri, Nov 18, 2016 at 01:26:33PM +0800, Eryu Guan wrote:
> > On Thu, Nov 17, 2016 at 12:11:02PM -0800, Darrick J. Wong wrote:
> > > On Thu, Nov 17, 2016 at 09:36:39AM -0800, Darrick J. Wong wrote:
> > > > On Fri, Nov 18, 2016 at 12:35:15AM +0800, Eryu Guan wrote:
> > > > > Hi all,
> > > > > 
> > > > > I hit a test hang in generic/224 when testing rmapbt enabled XFS on a
> > > > > host that has non-zero sunit/swidth reported from underlying device. And
> > > > > I simplified the reproducer to the following script, and the hang can be
> > > > > reproduced on any host now.
> > > > > 
> > > > > -----
> > > > > #!/bin/bash
> > > > > 
> > > > > dev=/dev/sda5
> > > > > mnt=/mnt/xfs
> > > > > 
> > > > > mkfs -t xfs -m rmapbt=1 -d agcount=8,size=1g -f $dev
> > > > 
> > > > Hm.  I formatted with:
> > > > mkfs.xfs -m rmapbt=1 -d sunit=4096,swidth=40960 -f /dev/sdf
> > > > 
> > > > (made up sunit numbers just to see how whacky it could get)
> > > > 
> > > > and got a different hang instead.  It looks like we are unable to
> > > > allocate any blocks to the bmbt and various things blow up from
> > > > there.  Will go retry with tracepoints on to see if we're running
> > > > out of AG reservation or if we're really out of disk blocks or what.
> > > > 
> > > > Crash message attached at the end.
> > > 
> > > Hm.  Looking at the indlen calculations, I see that we don't include the
> > > space that the rmapbt might need to store all the reverse mappings.  I
> > > think this is a problem, since we decline delalloc reservations if (len
> > > + indlen) > fdblocks, but we potentially end up using more than indlen
> > > blocks to map len blocks into the file, so the allocator goes nuts.
> > > 
> > > Eryu, does the following patch fix the problem you see?  I ran your
> > > reproducer and mine and it fixed the problem in both cases.  I didn't
> > > observe any issues running generic/224 either.
> > 
> > I applied your patch (and only your patch, patches posted by Dave were
> > not included) on top of 4.9-rc5 kernel, and it passed my simplified
> > reproducer, but still failed generic/224 with
> > 
> > MKFS_OPTIONS="-b size=4k -m crc=1,rmapbt=1 -d agcount=8"
> > 
> > Not all the time, but easily to hit. And sysrq-w showed the same traces
> > as before.
> > 
> > SECTION       -- xfs_test
> > RECREATING    -- xfs on /dev/mapper/testvg-testlv1
> > FSTYP         -- xfs (non-debug)
> > PLATFORM      -- Linux/x86_64 ibm-x3550m3-05 4.9.0-rc5+
> > MKFS_OPTIONS  -- -f -f -b size=4k -m crc=1,rmapbt=1 -d agcount=8 /dev/mapper/testvg-testlv2
> > MOUNT_OPTIONS -- -o context=system_u:object_r:nfs_t:s0 /dev/mapper/testvg-testlv2 /mnt/testarea/scratch
> > 
> > generic/224 16s ...  <===== never return
> 
> My patchset does pass generic/224 here, but it fails lots of other tests
> because of an accounting problem I've not yet found.

I applied all four patches you posted on top of v.9-rc5 this time. And
generic/224 still failed my test (test hang).

> 
> SECTION       -- xfs
> FSTYP         -- xfs (debug)
> PLATFORM      -- Linux/x86_64 test2 4.9.0-rc4-dgc+
> MKFS_OPTIONS  -- -f -m rmapbt=1 -i sparse=1 /dev/sdg

Does appending "-d agcount=8" to MKFS_OPTIONS make any difference for
you? I cannot reproduce the hang either if I remove the agcount config.

Thanks,
Eryu