* [next-20220215] WARNING at fs/iomap/buffered-io.c:75 with xfstests
@ 2022-02-16 7:25 Sachin Sant
2022-02-16 7:39 ` Stephen Rothwell
2022-02-16 17:35 ` Darrick J. Wong
0 siblings, 2 replies; 6+ messages in thread
From: Sachin Sant @ 2022-02-16 7:25 UTC (permalink / raw)
To: linux-xfs; +Cc: riteshh, linuxppc-dev, linux-next
[-- Attachment #1: Type: text/plain, Size: 6049 bytes --]
While running xfstests on IBM Power10 logical partition (LPAR) booted
with 5.17.0-rc4-next-20220215 following warning was seen:
[ 2547.384210] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2547.389021] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x2bc2c0 len 32 error 5
[ 2547.389020] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x15cede0 len 32 error 5
[ 2547.389026] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0xc60 len 8 error 5
[ 2547.389032] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x3bffa30 len 8 error 5
[ 2547.389120] XFS (dm-0): log I/O error -5
[ 2547.389135] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2547.389195] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2547.662818] ------------[ cut here ]------------
[ 2547.662832] WARNING: CPU: 24 PID: 2463718 at fs/iomap/buffered-io.c:75 iomap_page_release+0x1b0/0x1e0
[ 2547.662840] Modules linked in: dm_thin_pool dm_persistent_data dm_bio_prison dm_snapshot dm_bufio loop dm_flakey xfs libcrc32c dm_mod rfkill bonding sunrpc pseries_rng xts vmx_crypto uio_pdrv_genirq uio sch_fq_codel ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: scsi_debug]
[ 2547.662866] CPU: 24 PID: 2463718 Comm: umount Not tainted 5.17.0-rc4-next-20220215 #1
[ 2547.662871] NIP: c000000000565b80 LR: c000000000565aa0 CTR: c000000000565e40
[ 2547.662874] REGS: c00000000f0035b0 TRAP: 0700 Not tainted (5.17.0-rc4-next-20220215)
[ 2547.662877] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44000228 XER: 20040000
[ 2547.662885] CFAR: c000000000565ac8 IRQMASK: 0
[ 2547.662885] GPR00: c00000000037dd3c c00000000f003850 c000000002a03300 0000000000000001
[ 2547.662885] GPR04: 0000000000000010 0000000000010000 c0000000df6e7bc0 0000000000000001
[ 2547.662885] GPR08: fffffffffffeffff 0000000000000000 0000000000000010 0000000000000228
[ 2547.662885] GPR12: c000000000565e40 c000000abfcfe680 0000000000000000 0000000000000000
[ 2547.662885] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.662885] GPR20: 0000000000000000 0000000000000000 ffffffffffffffff fffffffffffffffe
[ 2547.662885] GPR24: 0000000000000000 ffffffffffffffff c00000000f0038e0 c0000000243db278
[ 2547.662885] GPR28: 0000000000000012 c0000000df6e7ae0 0000000000000010 c00c00000240c100
[ 2547.662920] NIP [c000000000565b80] iomap_page_release+0x1b0/0x1e0
[ 2547.662924] LR [c000000000565aa0] iomap_page_release+0xd0/0x1e0
[ 2547.662927] Call Trace:
[ 2547.662928] [c00000000f003850] [c00000000f003890] 0xc00000000f003890 (unreliable)
[ 2547.662932] [c00000000f003890] [c00000000037dd3c] truncate_cleanup_folio+0x7c/0x140
[ 2547.662937] [c00000000f0038c0] [c00000000037f068] truncate_inode_pages_range+0x148/0x5e0
[ 2547.662942] [c00000000f003a40] [c0000000004c2058] evict+0x248/0x270
[ 2547.662946] [c00000000f003a80] [c0000000004c20fc] dispose_list+0x7c/0xb0
[ 2547.662951] [c00000000f003ac0] [c0000000004c2314] evict_inodes+0x1e4/0x300
[ 2547.662955] [c00000000f003b60] [c0000000004922d0] generic_shutdown_super+0x70/0x1e0
[ 2547.662959] [c00000000f003bd0] [c000000000492518] kill_block_super+0x38/0x90
[ 2547.662964] [c00000000f003c00] [c0000000004926e8] deactivate_locked_super+0x78/0xf0
[ 2547.662968] [c00000000f003c30] [c0000000004cde9c] cleanup_mnt+0xfc/0x1c0
[ 2547.662972] [c00000000f003c80] [c000000000189448] task_work_run+0xf8/0x170
[ 2547.662976] [c00000000f003cd0] [c000000000021b94] do_notify_resume+0x434/0x480
[ 2547.662981] [c00000000f003d80] [c0000000000338b0] interrupt_exit_user_prepare_main+0x1a0/0x260
[ 2547.662985] [c00000000f003de0] [c000000000033d74] syscall_exit_prepare+0x74/0x150
[ 2547.662989] [c00000000f003e10] [c00000000000c658] system_call_common+0xf8/0x270
[ 2547.662994] --- interrupt: c00 at 0x7fffa6bb7dfc
[ 2547.662996] NIP: 00007fffa6bb7dfc LR: 00007fffa6bb7dcc CTR: 0000000000000000
[ 2547.662999] REGS: c00000000f003e80 TRAP: 0c00 Not tainted (5.17.0-rc4-next-20220215)
[ 2547.663001] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28000202 XER: 00000000
[ 2547.663010] IRQMASK: 0
[ 2547.663010] GPR00: 0000000000000034 00007fffc7ca6a70 00007fffa6c87300 0000000000000000
[ 2547.663010] GPR04: 0000000000000000 00007fffc7ca6a88 0000000000000000 0000000166cf4e90
[ 2547.663010] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.663010] GPR12: 0000000000000000 00007fffa6f1d730 0000000000000000 0000000000000000
[ 2547.663010] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.663010] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.663010] GPR24: 000000012d3b5f70 000000012d3b65d0 000000012d3cf6e8 0000000166cf0b50
[ 2547.663010] GPR28: 00007fffa6e701a4 0000000166cf4e90 0000000000000000 0000000166cf0970
[ 2547.663042] NIP [00007fffa6bb7dfc] 0x7fffa6bb7dfc
[ 2547.663045] LR [00007fffa6bb7dcc] 0x7fffa6bb7dcc
[ 2547.663047] --- interrupt: c00
[ 2547.663048] Instruction dump:
[ 2547.663050] 38210040 e9230000 ebe1fff8 4e800020 0fe00000 60000000 60000000 60000000
[ 2547.663056] 0fe00000 60000000 60000000 60000000 <0fe00000> 60000000 60000000 60000000
[ 2547.663062] ---[ end trace 0000000000000000 ]---
[ 2547.663075] XFS (dm-0): Unmounting Filesystem
The warning is seen when test tries to unmount the file system. This problem is seen
while running generic/475 sub test. Have attached captured messages during the test
run of generic/475.
xfstest is a recent add to upstream regression bucket. I don’t have any previous data
to attempt a git bisect.
Let me know if any additional information is required.
Thanks
-Sachin
[-- Attachment #2: xfstest.txt --]
[-- Type: text/plain, Size: 76413 bytes --]
[ 2523.250933] run fstests generic/475 at 2022-02-15 15:42:10
[ 2523.366239] XFS (sdb1): Mounting V5 Filesystem
[ 2523.376679] XFS (sdb1): Ending clean mount
[ 2523.378107] xfs filesystem being mounted at /mnt/test supports timestamps until 2038 (0x7fffffff)
[ 2523.502357] XFS (dm-0): Mounting V5 Filesystem
[ 2523.508526] XFS (dm-0): Ending clean mount
[ 2523.509735] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2524.516736] dm-0: writeback error on inode 33750928, offset 655360, sector 21228312
[ 2524.517884] dm-0: writeback error on inode 100730311, offset 589824, sector 63235832
[ 2524.518348] dm-0: writeback error on inode 67165541, offset 258371584, sector 42405248
[ 2524.518625] dm-0: writeback error on inode 136194, offset 417792, sector 212888
[ 2524.518727] dm-0: writeback error on inode 114612, offset 1052672, sector 295824
[ 2524.518876] buffer_io_error: 6 callbacks suppressed
[ 2524.518878] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2524.518888] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2524.518892] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2524.518895] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2524.518898] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2524.518901] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2524.518905] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2524.518908] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2524.518911] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2524.518914] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2524.519703] dm-0: writeback error on inode 67200889, offset 6684672, sector 42237480
[ 2524.520249] dm-0: writeback error on inode 100772183, offset 30445568, sector 63105160
[ 2524.520258] dm-0: writeback error on inode 100835769, offset 270336, sector 63079496
[ 2524.520266] dm-0: writeback error on inode 114527, offset 40960, sector 84968
[ 2524.520272] dm-0: writeback error on inode 67244195, offset 25001984, sector 42327408
[ 2524.520710] XFS (dm-0): log I/O error -5
[ 2524.520719] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2524.520780] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2524.793985] XFS (dm-0): Unmounting Filesystem
[ 2524.870119] XFS (dm-0): Mounting V5 Filesystem
[ 2525.015369] XFS (dm-0): Starting recovery (logdev: internal)
[ 2525.418197] XFS (dm-0): Ending recovery (logdev: internal)
[ 2525.419156] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2525.425641] XFS (dm-0): log I/O error -5
[ 2525.425645] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x2c3c0 len 32 error 5
[ 2525.425647] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x3bffa80 len 32 error 5
[ 2525.425656] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2525.425711] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2525.592468] XFS (dm-0): Unmounting Filesystem
[ 2525.609374] XFS (dm-0): Mounting V5 Filesystem
[ 2525.681666] XFS (dm-0): Starting recovery (logdev: internal)
[ 2525.694463] XFS (dm-0): Ending recovery (logdev: internal)
[ 2525.696357] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2526.704897] XFS (dm-0): log I/O error -5
[ 2526.704923] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2526.705029] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2526.948328] XFS (dm-0): Unmounting Filesystem
[ 2527.040926] XFS (dm-0): Mounting V5 Filesystem
[ 2527.216170] XFS (dm-0): Starting recovery (logdev: internal)
[ 2527.749123] XFS (dm-0): Ending recovery (logdev: internal)
[ 2527.750067] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2527.755047] XFS (dm-0): log I/O error -5
[ 2527.755045] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x3bffa30 len 8 error 5
[ 2527.755061] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x13ffe18 len 8 error 5
[ 2527.755080] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2527.755161] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2527.923532] XFS (dm-0): Unmounting Filesystem
[ 2527.940178] XFS (dm-0): Mounting V5 Filesystem
[ 2527.999573] XFS (dm-0): Starting recovery (logdev: internal)
[ 2528.014009] XFS (dm-0): Ending recovery (logdev: internal)
[ 2528.016171] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2528.021163] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x15c0 len 32 error 5
[ 2528.021161] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x13ffe80 len 32 error 5
[ 2528.021247] XFS (dm-0): log I/O error -5
[ 2528.021260] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2528.021319] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2528.261691] XFS (dm-0): Unmounting Filesystem
[ 2528.278383] XFS (dm-0): Mounting V5 Filesystem
[ 2528.337759] XFS (dm-0): Starting recovery (logdev: internal)
[ 2528.352023] XFS (dm-0): Ending recovery (logdev: internal)
[ 2528.354722] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2528.359910] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x1415598 len 8 error 5
[ 2528.359909] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x3bffaa0 len 32 error 5
[ 2528.360016] XFS (dm-0): xfs_imap_lookup: xfs_ialloc_read_agi() returned error -5, agno 0
[ 2528.360045] XFS (dm-0): log I/O error -5
[ 2528.360016] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x1415598 len 8 error 5
[ 2528.360059] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2528.360128] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2528.603702] XFS (dm-0): Unmounting Filesystem
[ 2528.620367] XFS (dm-0): Mounting V5 Filesystem
[ 2528.629620] XFS (dm-0): Starting recovery (logdev: internal)
[ 2528.634048] XFS (dm-0): Ending recovery (logdev: internal)
[ 2528.635707] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2530.643300] buffer_io_error: 86 callbacks suppressed
[ 2530.643303] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2530.643319] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2530.643323] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2530.643326] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2530.643330] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2530.643333] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2530.643336] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2530.643339] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2530.643342] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2530.643345] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2530.644362] iomap_finish_ioend: 221 callbacks suppressed
[ 2530.644364] dm-0: writeback error on inode 34090876, offset 3014656, sector 22112200
[ 2530.644538] dm-0: writeback error on inode 34427948, offset 786432, sector 22282544
[ 2530.644760] dm-0: writeback error on inode 178971, offset 131072, sector 1225112
[ 2530.645682] dm-0: writeback error on inode 178976, offset 327680, sector 857912
[ 2530.645690] dm-0: writeback error on inode 101264072, offset 884736, sector 63870264
[ 2530.645694] dm-0: writeback error on inode 178976, offset 331776, sector 1089144
[ 2530.645699] dm-0: writeback error on inode 178976, offset 389120, sector 1089256
[ 2530.652881] dm-0: writeback error on inode 67151740, offset 7122944, sector 43064768
[ 2530.653420] dm-0: writeback error on inode 385792, offset 5693440, sector 1185032
[ 2530.653602] dm-0: writeback error on inode 34090849, offset 2056192, sector 21840064
[ 2530.653811] XFS (dm-0): log I/O error -5
[ 2530.653819] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2530.653898] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2530.894975] XFS (dm-0): Unmounting Filesystem
[ 2530.953709] XFS (dm-0): Mounting V5 Filesystem
[ 2531.116725] XFS (dm-0): Starting recovery (logdev: internal)
[ 2531.619363] XFS (dm-0): Ending recovery (logdev: internal)
[ 2531.620537] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2533.630182] XFS (dm-0): log I/O error -5
[ 2533.630204] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2533.630296] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2533.875912] XFS (dm-0): Unmounting Filesystem
[ 2533.944095] XFS (dm-0): Mounting V5 Filesystem
[ 2534.097933] XFS (dm-0): Starting recovery (logdev: internal)
[ 2534.545681] XFS (dm-0): Ending recovery (logdev: internal)
[ 2534.546748] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2534.552289] XFS (dm-0): log I/O error -5
[ 2534.552294] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x86540 len 32 error 5
[ 2534.552317] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2534.552363] XFS (dm-0): xfs_imap_lookup: xfs_ialloc_read_agi() returned error -5, agno 0
[ 2534.552378] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2534.752003] XFS (dm-0): Unmounting Filesystem
[ 2534.768805] XFS (dm-0): Mounting V5 Filesystem
[ 2534.827769] XFS (dm-0): Starting recovery (logdev: internal)
[ 2534.841175] XFS (dm-0): Ending recovery (logdev: internal)
[ 2534.843039] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2536.849519] iomap_finish_ioend: 102 callbacks suppressed
[ 2536.849523] dm-0: writeback error on inode 1413851, offset 89260032, sector 2095432
[ 2536.850440] buffer_io_error: 38 callbacks suppressed
[ 2536.850442] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2536.850453] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2536.850457] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2536.850460] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2536.850463] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2536.850467] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2536.850470] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2536.850473] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2536.850476] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2536.850479] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2536.851985] dm-0: writeback error on inode 1392803, offset 7856128, sector 2156768
[ 2536.851989] dm-0: writeback error on inode 1358735, offset 32309248, sector 1874480
[ 2536.852481] dm-0: writeback error on inode 102083501, offset 5750784, sector 65056496
[ 2536.857616] dm-0: writeback error on inode 1369530, offset 786432, sector 1936752
[ 2536.858012] dm-0: writeback error on inode 1729116, offset 1294336, sector 2157232
[ 2536.858019] dm-0: writeback error on inode 68591248, offset 29708288, sector 44249952
[ 2536.858025] dm-0: writeback error on inode 35290493, offset 67264512, sector 23190648
[ 2536.858030] dm-0: writeback error on inode 35290493, offset 77107200, sector 23302296
[ 2536.858035] dm-0: writeback error on inode 1369530, offset 868352, sector 1937336
[ 2536.858192] XFS (dm-0): log I/O error -5
[ 2536.858202] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2536.858282] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2537.084788] XFS (dm-0): Unmounting Filesystem
[ 2537.125735] XFS (dm-0): Mounting V5 Filesystem
[ 2537.252751] XFS (dm-0): Starting recovery (logdev: internal)
[ 2537.583254] XFS (dm-0): Ending recovery (logdev: internal)
[ 2537.584277] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2538.595093] XFS (dm-0): log I/O error -5
[ 2538.595110] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2538.595196] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2538.864000] XFS (dm-0): Unmounting Filesystem
[ 2538.905962] XFS (dm-0): Mounting V5 Filesystem
[ 2539.096169] XFS (dm-0): Starting recovery (logdev: internal)
[ 2539.633947] XFS (dm-0): Ending recovery (logdev: internal)
[ 2539.634958] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2541.640724] XFS (dm-0): log I/O error -5
[ 2541.640741] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2541.640802] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2541.935012] XFS (dm-0): Unmounting Filesystem
[ 2541.988684] XFS (dm-0): Mounting V5 Filesystem
[ 2542.116431] XFS (dm-0): Starting recovery (logdev: internal)
[ 2542.441152] XFS (dm-0): Ending recovery (logdev: internal)
[ 2542.442150] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2544.448191] iomap_finish_ioend: 46 callbacks suppressed
[ 2544.448194] dm-0: writeback error on inode 36077872, offset 73859072, sector 24110792
[ 2544.449711] buffer_io_error: 38 callbacks suppressed
[ 2544.449712] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2544.449723] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2544.449727] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2544.449728] dm-0: writeback error on inode 2692401, offset 458752, sector 2775128
[ 2544.449730] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2544.449735] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2544.449738] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2544.449742] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2544.449744] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2544.449748] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2544.449751] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2544.458404] dm-0: writeback error on inode 2391873, offset 466944, sector 2391824
[ 2544.458447] dm-0: writeback error on inode 103160128, offset 937984, sector 65927664
[ 2544.458543] dm-0: writeback error on inode 36054700, offset 33423360, sector 24188272
[ 2544.458549] dm-0: writeback error on inode 103092984, offset 28631040, sector 65845792
[ 2544.458553] dm-0: writeback error on inode 2392584, offset 74911744, sector 2755816
[ 2544.458558] dm-0: writeback error on inode 69726030, offset 21921792, sector 44540328
[ 2544.458563] dm-0: writeback error on inode 2669138, offset 1040384, sector 2801096
[ 2544.458566] dm-0: writeback error on inode 2391873, offset 458752, sector 2391960
[ 2544.458851] XFS (dm-0): log I/O error -5
[ 2544.458858] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2544.458917] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2544.683010] XFS (dm-0): Unmounting Filesystem
[ 2544.739531] XFS (dm-0): Mounting V5 Filesystem
[ 2544.914316] XFS (dm-0): Starting recovery (logdev: internal)
[ 2545.426712] XFS (dm-0): Ending recovery (logdev: internal)
[ 2545.427672] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2546.438765] XFS (dm-0): log I/O error -5
[ 2546.438781] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2546.438842] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2546.727004] XFS (dm-0): Unmounting Filesystem
[ 2546.769608] XFS (dm-0): Mounting V5 Filesystem
[ 2546.936362] XFS (dm-0): Starting recovery (logdev: internal)
[ 2547.383261] XFS (dm-0): Ending recovery (logdev: internal)
[ 2547.384210] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2547.389021] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x2bc2c0 len 32 error 5
[ 2547.389020] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x15cede0 len 32 error 5
[ 2547.389026] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0xc60 len 8 error 5
[ 2547.389032] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x3bffa30 len 8 error 5
[ 2547.389120] XFS (dm-0): log I/O error -5
[ 2547.389135] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2547.389195] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2547.662818] ------------[ cut here ]------------
[ 2547.662832] WARNING: CPU: 24 PID: 2463718 at fs/iomap/buffered-io.c:75 iomap_page_release+0x1b0/0x1e0
[ 2547.662840] Modules linked in: dm_thin_pool dm_persistent_data dm_bio_prison dm_snapshot dm_bufio loop dm_flakey xfs libcrc32c dm_mod rfkill bonding sunrpc pseries_rng xts vmx_crypto uio_pdrv_genirq uio sch_fq_codel ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: scsi_debug]
[ 2547.662866] CPU: 24 PID: 2463718 Comm: umount Not tainted 5.17.0-rc4-next-20220215 #1
[ 2547.662871] NIP: c000000000565b80 LR: c000000000565aa0 CTR: c000000000565e40
[ 2547.662874] REGS: c00000000f0035b0 TRAP: 0700 Not tainted (5.17.0-rc4-next-20220215)
[ 2547.662877] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44000228 XER: 20040000
[ 2547.662885] CFAR: c000000000565ac8 IRQMASK: 0
[ 2547.662885] GPR00: c00000000037dd3c c00000000f003850 c000000002a03300 0000000000000001
[ 2547.662885] GPR04: 0000000000000010 0000000000010000 c0000000df6e7bc0 0000000000000001
[ 2547.662885] GPR08: fffffffffffeffff 0000000000000000 0000000000000010 0000000000000228
[ 2547.662885] GPR12: c000000000565e40 c000000abfcfe680 0000000000000000 0000000000000000
[ 2547.662885] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.662885] GPR20: 0000000000000000 0000000000000000 ffffffffffffffff fffffffffffffffe
[ 2547.662885] GPR24: 0000000000000000 ffffffffffffffff c00000000f0038e0 c0000000243db278
[ 2547.662885] GPR28: 0000000000000012 c0000000df6e7ae0 0000000000000010 c00c00000240c100
[ 2547.662920] NIP [c000000000565b80] iomap_page_release+0x1b0/0x1e0
[ 2547.662924] LR [c000000000565aa0] iomap_page_release+0xd0/0x1e0
[ 2547.662927] Call Trace:
[ 2547.662928] [c00000000f003850] [c00000000f003890] 0xc00000000f003890 (unreliable)
[ 2547.662932] [c00000000f003890] [c00000000037dd3c] truncate_cleanup_folio+0x7c/0x140
[ 2547.662937] [c00000000f0038c0] [c00000000037f068] truncate_inode_pages_range+0x148/0x5e0
[ 2547.662942] [c00000000f003a40] [c0000000004c2058] evict+0x248/0x270
[ 2547.662946] [c00000000f003a80] [c0000000004c20fc] dispose_list+0x7c/0xb0
[ 2547.662951] [c00000000f003ac0] [c0000000004c2314] evict_inodes+0x1e4/0x300
[ 2547.662955] [c00000000f003b60] [c0000000004922d0] generic_shutdown_super+0x70/0x1e0
[ 2547.662959] [c00000000f003bd0] [c000000000492518] kill_block_super+0x38/0x90
[ 2547.662964] [c00000000f003c00] [c0000000004926e8] deactivate_locked_super+0x78/0xf0
[ 2547.662968] [c00000000f003c30] [c0000000004cde9c] cleanup_mnt+0xfc/0x1c0
[ 2547.662972] [c00000000f003c80] [c000000000189448] task_work_run+0xf8/0x170
[ 2547.662976] [c00000000f003cd0] [c000000000021b94] do_notify_resume+0x434/0x480
[ 2547.662981] [c00000000f003d80] [c0000000000338b0] interrupt_exit_user_prepare_main+0x1a0/0x260
[ 2547.662985] [c00000000f003de0] [c000000000033d74] syscall_exit_prepare+0x74/0x150
[ 2547.662989] [c00000000f003e10] [c00000000000c658] system_call_common+0xf8/0x270
[ 2547.662994] --- interrupt: c00 at 0x7fffa6bb7dfc
[ 2547.662996] NIP: 00007fffa6bb7dfc LR: 00007fffa6bb7dcc CTR: 0000000000000000
[ 2547.662999] REGS: c00000000f003e80 TRAP: 0c00 Not tainted (5.17.0-rc4-next-20220215)
[ 2547.663001] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28000202 XER: 00000000
[ 2547.663010] IRQMASK: 0
[ 2547.663010] GPR00: 0000000000000034 00007fffc7ca6a70 00007fffa6c87300 0000000000000000
[ 2547.663010] GPR04: 0000000000000000 00007fffc7ca6a88 0000000000000000 0000000166cf4e90
[ 2547.663010] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.663010] GPR12: 0000000000000000 00007fffa6f1d730 0000000000000000 0000000000000000
[ 2547.663010] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.663010] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[ 2547.663010] GPR24: 000000012d3b5f70 000000012d3b65d0 000000012d3cf6e8 0000000166cf0b50
[ 2547.663010] GPR28: 00007fffa6e701a4 0000000166cf4e90 0000000000000000 0000000166cf0970
[ 2547.663042] NIP [00007fffa6bb7dfc] 0x7fffa6bb7dfc
[ 2547.663045] LR [00007fffa6bb7dcc] 0x7fffa6bb7dcc
[ 2547.663047] --- interrupt: c00
[ 2547.663048] Instruction dump:
[ 2547.663050] 38210040 e9230000 ebe1fff8 4e800020 0fe00000 60000000 60000000 60000000
[ 2547.663056] 0fe00000 60000000 60000000 60000000 <0fe00000> 60000000 60000000 60000000
[ 2547.663062] ---[ end trace 0000000000000000 ]---
[ 2547.663075] XFS (dm-0): Unmounting Filesystem
[ 2547.679882] XFS (dm-0): Mounting V5 Filesystem
[ 2547.733927] XFS (dm-0): Starting recovery (logdev: internal)
[ 2547.745633] XFS (dm-0): Ending recovery (logdev: internal)
[ 2547.748178] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2549.753836] iomap_finish_ioend: 43 callbacks suppressed
[ 2549.753839] dm-0: writeback error on inode 103859326, offset 65249280, sector 66209096
[ 2549.753854] dm-0: writeback error on inode 36109178, offset 6733824, sector 24398488
[ 2549.753859] dm-0: writeback error on inode 70328836, offset 1802240, sector 45162048
[ 2549.753863] dm-0: writeback error on inode 36480701, offset 108093440, sector 24428192
[ 2549.753869] dm-0: writeback error on inode 69618107, offset 22708224, sector 45250888
[ 2549.753873] dm-0: writeback error on inode 36521924, offset 104202240, sector 24543472
[ 2549.753877] dm-0: writeback error on inode 36521924, offset 170885120, sector 24541200
[ 2549.753881] dm-0: writeback error on inode 70357590, offset 3289088, sector 45331584
[ 2549.753885] dm-0: writeback error on inode 103289852, offset 1003520, sector 66297216
[ 2549.754924] XFS (dm-0): log I/O error -5
[ 2549.754950] XFS (dm-0): log I/O error -5
[ 2549.754940] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2549.755035] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2549.756230] buffer_io_error: 38 callbacks suppressed
[ 2549.756233] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2549.756242] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2549.756246] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2549.756249] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2549.756252] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2549.756255] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2549.756259] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2549.756262] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2549.756265] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2549.756268] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2550.021468] XFS (dm-0): Unmounting Filesystem
[ 2550.080673] XFS (dm-0): Mounting V5 Filesystem
[ 2550.270646] XFS (dm-0): Starting recovery (logdev: internal)
[ 2550.760973] XFS (dm-0): Ending recovery (logdev: internal)
[ 2550.762099] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2551.767871] dm-0: writeback error on inode 70406127, offset 124911616, sector 45548040
[ 2551.767885] XFS (dm-0): log I/O error -5
[ 2551.768068] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2551.768140] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2552.001984] XFS (dm-0): Unmounting Filesystem
[ 2552.033560] XFS (dm-0): Mounting V5 Filesystem
[ 2552.202493] XFS (dm-0): Starting recovery (logdev: internal)
[ 2552.629573] XFS (dm-0): Ending recovery (logdev: internal)
[ 2552.630476] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2554.636725] XFS (dm-0): log I/O error -5
[ 2554.636740] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2554.636803] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2554.906255] XFS (dm-0): Unmounting Filesystem
[ 2554.964125] XFS (dm-0): Mounting V5 Filesystem
[ 2555.118190] XFS (dm-0): Starting recovery (logdev: internal)
[ 2555.480176] XFS (dm-0): Ending recovery (logdev: internal)
[ 2555.481202] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2556.487342] XFS (dm-0): log I/O error -5
[ 2556.487360] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2556.487423] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2556.487427] XFS (dm-0): log I/O error -5
[ 2556.487429] XFS (dm-0): log I/O error -5
[ 2556.487431] XFS (dm-0): log I/O error -5
[ 2556.489554] buffer_io_error: 38 callbacks suppressed
[ 2556.489556] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2556.489565] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2556.489569] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2556.489572] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2556.489575] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2556.489579] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2556.489582] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2556.489585] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2556.489588] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2556.489591] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2556.722895] XFS (dm-0): Unmounting Filesystem
[ 2556.781571] XFS (dm-0): Mounting V5 Filesystem
[ 2556.962599] XFS (dm-0): Starting recovery (logdev: internal)
[ 2557.386041] XFS (dm-0): Ending recovery (logdev: internal)
[ 2557.387016] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2558.400557] iomap_finish_ioend: 41 callbacks suppressed
[ 2558.400560] dm-0: writeback error on inode 104621403, offset 393216, sector 66985016
[ 2558.400788] dm-0: writeback error on inode 2973309, offset 3555328, sector 4043448
[ 2558.400797] dm-0: writeback error on inode 3532245, offset 27906048, sector 4008328
[ 2558.400802] dm-0: writeback error on inode 104656344, offset 114626560, sector 66996824
[ 2558.400808] dm-0: writeback error on inode 104621424, offset 77824, sector 66916624
[ 2558.400872] XFS (dm-0): log I/O error -5
[ 2558.400880] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2558.400967] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2558.625070] XFS (dm-0): Unmounting Filesystem
[ 2558.672009] XFS (dm-0): Mounting V5 Filesystem
[ 2558.811196] XFS (dm-0): Starting recovery (logdev: internal)
[ 2559.196329] XFS (dm-0): Ending recovery (logdev: internal)
[ 2559.197243] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2559.202340] XFS (dm-0): log I/O error -5
[ 2559.202345] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x3f19480 len 32 error 5
[ 2559.202361] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2559.202420] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2559.432336] XFS (dm-0): Unmounting Filesystem
[ 2559.449047] XFS (dm-0): Mounting V5 Filesystem
[ 2559.490582] XFS (dm-0): Starting recovery (logdev: internal)
[ 2559.499664] XFS (dm-0): Ending recovery (logdev: internal)
[ 2559.501334] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2561.508579] buffer_io_error: 38 callbacks suppressed
[ 2561.508582] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2561.508599] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2561.508603] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2561.508606] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2561.508609] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2561.508613] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2561.508615] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2561.508619] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2561.508622] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2561.508625] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2561.512702] dm-0: writeback error on inode 104703262, offset 147324928, sector 67200392
[ 2561.523022] dm-0: writeback error on inode 37886195, offset 131072, sector 25502904
[ 2561.550302] XFS (dm-0): log I/O error -5
[ 2561.550316] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2561.550376] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2561.550380] XFS (dm-0): log I/O error -5
[ 2561.550382] XFS (dm-0): log I/O error -5
[ 2561.550384] XFS (dm-0): log I/O error -5
[ 2561.789009] XFS (dm-0): Unmounting Filesystem
[ 2561.873689] XFS (dm-0): Mounting V5 Filesystem
[ 2562.009743] XFS (dm-0): Starting recovery (logdev: internal)
[ 2562.358919] XFS (dm-0): Ending recovery (logdev: internal)
[ 2562.359856] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2564.370069] dm-0: writeback error on inode 102464293, offset 2129920, sector 67036960
[ 2564.370076] dm-0: writeback error on inode 37987634, offset 62914560, sector 25414248
[ 2564.370102] dm-0: writeback error on inode 37987634, offset 128565248, sector 25743352
[ 2564.370113] dm-0: writeback error on inode 4143566, offset 1355776, sector 4470760
[ 2564.370119] dm-0: writeback error on inode 37386218, offset 1511424, sector 25184776
[ 2564.370125] dm-0: writeback error on inode 71426166, offset 44986368, sector 46349904
[ 2564.370130] dm-0: writeback error on inode 4233901, offset 22544384, sector 4480616
[ 2564.370135] dm-0: writeback error on inode 37935212, offset 44507136, sector 25651960
[ 2564.370141] dm-0: writeback error on inode 105027811, offset 581632, sector 67474168
[ 2564.370147] dm-0: writeback error on inode 37886177, offset 34197504, sector 25582624
[ 2564.370451] XFS (dm-0): log I/O error -5
[ 2564.370460] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2564.370521] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2564.603752] XFS (dm-0): Unmounting Filesystem
[ 2564.668356] XFS (dm-0): Mounting V5 Filesystem
[ 2564.815810] XFS (dm-0): Starting recovery (logdev: internal)
[ 2565.223316] XFS (dm-0): Ending recovery (logdev: internal)
[ 2565.224235] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2567.231741] buffer_io_error: 22 callbacks suppressed
[ 2567.231744] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2567.231762] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2567.231765] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2567.231768] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2567.231772] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2567.231775] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2567.231778] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2567.231781] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2567.231785] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2567.231788] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2567.442435] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x1c10 len 8 error 5
[ 2567.446460] XFS (dm-0): log I/O error -5
[ 2567.446469] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2567.446526] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2567.469623] XFS (dm-0): Unmounting Filesystem
[ 2567.469653] XFS (dm-0): Failing async write on buffer block 0x421c00. Retrying async write.
[ 2567.469660] XFS (dm-0): Failing async write on buffer block 0x421bd8. Retrying async write.
[ 2567.469663] XFS (dm-0): Failing async write on buffer block 0x2c47ac8. Retrying async write.
[ 2567.469666] XFS (dm-0): Failing async write on buffer block 0x3db7618. Retrying async write.
[ 2567.469669] XFS (dm-0): Failing async write on buffer block 0x3d39bc0. Retrying async write.
[ 2567.469672] XFS (dm-0): Failing async write on buffer block 0x3d39c00. Retrying async write.
[ 2567.469675] XFS (dm-0): Failing async write on buffer block 0x2c46c70. Retrying async write.
[ 2567.469679] XFS (dm-0): Failing async write on buffer block 0x1724220. Retrying async write.
[ 2567.469683] XFS (dm-0): Failing async write on buffer block 0x3d39c08. Retrying async write.
[ 2567.605306] XFS (dm-0): Mounting V5 Filesystem
[ 2567.778964] XFS (dm-0): Starting recovery (logdev: internal)
[ 2568.250498] XFS (dm-0): Ending recovery (logdev: internal)
[ 2568.251454] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2570.257461] iomap_finish_ioend: 93 callbacks suppressed
[ 2570.257465] dm-0: writeback error on inode 71736329, offset 557056, sector 46797816
[ 2570.277368] dm-0: writeback error on inode 101747424, offset 798720, sector 4883144
[ 2570.277386] dm-0: writeback error on inode 4112654, offset 65536, sector 4765656
[ 2570.277392] dm-0: writeback error on inode 71576811, offset 24985600, sector 46689208
[ 2570.277397] dm-0: writeback error on inode 71596503, offset 56389632, sector 46663520
[ 2570.277403] dm-0: writeback error on inode 71596485, offset 70684672, sector 46728408
[ 2570.277408] dm-0: writeback error on inode 105195353, offset 671744, sector 67776440
[ 2570.277414] dm-0: writeback error on inode 71399352, offset 10092544, sector 46274472
[ 2570.277419] dm-0: writeback error on inode 71399352, offset 34680832, sector 46800296
[ 2570.277424] dm-0: writeback error on inode 4500080, offset 18259968, sector 4845360
[ 2570.277599] XFS (dm-0): log I/O error -5
[ 2570.277608] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2570.277680] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2570.509378] XFS (dm-0): Unmounting Filesystem
[ 2570.565428] XFS (dm-0): Mounting V5 Filesystem
[ 2570.757446] XFS (dm-0): Starting recovery (logdev: internal)
[ 2571.291433] XFS (dm-0): Ending recovery (logdev: internal)
[ 2571.292377] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2572.300294] buffer_io_error: 22 callbacks suppressed
[ 2572.300297] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2572.300313] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2572.300317] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2572.300320] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2572.300323] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2572.300326] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2572.300329] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2572.300333] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2572.300336] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2572.300339] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2572.301248] XFS (dm-0): log I/O error -5
[ 2572.301257] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2572.301329] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2572.524483] XFS (dm-0): Unmounting Filesystem
[ 2572.561468] XFS (dm-0): Mounting V5 Filesystem
[ 2572.702822] XFS (dm-0): Starting recovery (logdev: internal)
[ 2573.072094] XFS (dm-0): Ending recovery (logdev: internal)
[ 2573.073038] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2574.091592] XFS (dm-0): log I/O error -5
[ 2574.091610] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2574.091672] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2574.332995] XFS (dm-0): Unmounting Filesystem
[ 2574.368980] XFS (dm-0): Mounting V5 Filesystem
[ 2574.511448] XFS (dm-0): Starting recovery (logdev: internal)
[ 2574.879902] XFS (dm-0): Ending recovery (logdev: internal)
[ 2574.880819] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2576.886825] iomap_finish_ioend: 23 callbacks suppressed
[ 2576.886828] dm-0: writeback error on inode 5086277, offset 23314432, sector 5450480
[ 2576.886844] dm-0: writeback error on inode 71905642, offset 63766528, sector 47007200
[ 2576.886849] dm-0: writeback error on inode 38936684, offset 585728, sector 26394984
[ 2576.886853] dm-0: writeback error on inode 71399332, offset 18481152, sector 47024616
[ 2576.886858] dm-0: writeback error on inode 105806417, offset 839680, sector 68139328
[ 2576.886863] dm-0: writeback error on inode 72124624, offset 13430784, sector 47057936
[ 2576.886867] dm-0: writeback error on inode 72081362, offset 110592, sector 46913208
[ 2576.886872] dm-0: writeback error on inode 3290291, offset 20537344, sector 5361800
[ 2576.886876] dm-0: writeback error on inode 5270796, offset 569344, sector 5267992
[ 2576.886880] dm-0: writeback error on inode 38888467, offset 1638400, sector 26450008
[ 2576.886995] XFS (dm-0): log I/O error -5
[ 2576.887009] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2576.887116] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2577.128229] XFS (dm-0): Unmounting Filesystem
[ 2577.165577] XFS (dm-0): Mounting V5 Filesystem
[ 2577.305996] XFS (dm-0): Starting recovery (logdev: internal)
[ 2577.711552] XFS (dm-0): Ending recovery (logdev: internal)
[ 2577.712499] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2577.717746] XFS (dm-0): log I/O error -5
[ 2577.717743] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x13ffe18 len 8 error 5
[ 2577.717749] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x3bffa18 len 8 error 5
[ 2577.717755] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x1c10 len 8 error 5
[ 2577.717797] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2577.717899] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2577.720017] buffer_io_error: 38 callbacks suppressed
[ 2577.720019] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2577.720031] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2577.720035] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2577.720038] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2577.720042] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2577.720045] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2577.720048] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2577.720051] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2577.720054] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2577.720057] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2577.963247] XFS (dm-0): Unmounting Filesystem
[ 2577.980236] XFS (dm-0): Mounting V5 Filesystem
[ 2578.029315] XFS (dm-0): Starting recovery (logdev: internal)
[ 2578.040983] XFS (dm-0): Ending recovery (logdev: internal)
[ 2578.042968] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2580.048771] XFS (dm-0): log I/O error -5
[ 2580.048789] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2580.048851] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2580.048855] XFS (dm-0): log I/O error -5
[ 2580.048857] XFS (dm-0): log I/O error -5
[ 2580.048860] XFS (dm-0): log I/O error -5
[ 2580.048862] XFS (dm-0): log I/O error -5
[ 2580.261156] XFS (dm-0): Unmounting Filesystem
[ 2580.317309] XFS (dm-0): Mounting V5 Filesystem
[ 2580.524344] XFS (dm-0): Starting recovery (logdev: internal)
[ 2581.064911] XFS (dm-0): Ending recovery (logdev: internal)
[ 2581.066023] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2582.071952] iomap_finish_ioend: 11 callbacks suppressed
[ 2582.071956] dm-0: writeback error on inode 39381706, offset 417792, sector 26804152
[ 2582.072263] dm-0: writeback error on inode 71802791, offset 50212864, sector 46680008
[ 2582.072837] dm-0: writeback error on inode 104258031, offset 27787264, sector 68423688
[ 2582.072956] dm-0: writeback error on inode 71802753, offset 3035136, sector 47233248
[ 2582.073108] dm-0: writeback error on inode 105687197, offset 262144, sector 67937776
[ 2582.073118] dm-0: writeback error on inode 105687197, offset 19116032, sector 68428184
[ 2582.073126] dm-0: writeback error on inode 5290517, offset 897024, sector 5292776
[ 2582.073134] dm-0: writeback error on inode 105828662, offset 162582528, sector 47243656
[ 2582.073141] dm-0: writeback error on inode 5275003, offset 4661248, sector 5660280
[ 2582.073149] dm-0: writeback error on inode 104258031, offset 54648832, sector 68428352
[ 2582.073301] XFS (dm-0): log I/O error -5
[ 2582.073312] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2582.073422] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2582.329214] XFS (dm-0): Unmounting Filesystem
[ 2582.417520] XFS (dm-0): Mounting V5 Filesystem
[ 2582.553350] XFS (dm-0): Starting recovery (logdev: internal)
[ 2582.842465] XFS (dm-0): Ending recovery (logdev: internal)
[ 2582.843363] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2584.848950] XFS (dm-0): log I/O error -5
[ 2584.848969] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2584.849030] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2584.849033] XFS (dm-0): log I/O error -5
[ 2584.849036] XFS (dm-0): log I/O error -5
[ 2584.849038] XFS (dm-0): log I/O error -5
[ 2584.849040] XFS (dm-0): log I/O error -5
[ 2584.851337] buffer_io_error: 38 callbacks suppressed
[ 2584.851340] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2584.851351] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2584.851355] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2584.851358] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2584.851362] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2584.851365] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2584.851368] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2584.851371] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2584.851374] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2584.851377] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2585.125926] XFS (dm-0): Unmounting Filesystem
[ 2585.165412] XFS (dm-0): Mounting V5 Filesystem
[ 2585.354102] XFS (dm-0): Starting recovery (logdev: internal)
[ 2585.860897] XFS (dm-0): Ending recovery (logdev: internal)
[ 2585.861828] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2586.870178] XFS (dm-0): log I/O error -5
[ 2586.870200] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2586.870286] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2587.095054] XFS (dm-0): Unmounting Filesystem
[ 2587.134976] XFS (dm-0): Mounting V5 Filesystem
[ 2587.275993] XFS (dm-0): Starting recovery (logdev: internal)
[ 2587.537406] XFS (dm-0): Ending recovery (logdev: internal)
[ 2587.538293] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2587.543301] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x1415598 len 8 error 5
[ 2587.543302] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x18 len 8 error 5
[ 2587.543303] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x3bffa18 len 8 error 5
[ 2587.543299] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x40f09c0 len 32 error 5
[ 2587.543358] XFS (dm-0): page discard on page 000000008449fa28, inode 0x64c7ea4, pos 589824.
[ 2587.543371] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x18 len 8 error 5
[ 2587.543429] XFS (dm-0): log I/O error -5
[ 2587.543450] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2587.543514] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2587.744847] XFS (dm-0): Unmounting Filesystem
[ 2587.814615] XFS (dm-0): Mounting V5 Filesystem
[ 2587.844735] XFS (dm-0): Starting recovery (logdev: internal)
[ 2587.853639] XFS (dm-0): Ending recovery (logdev: internal)
[ 2587.855380] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2588.861351] iomap_finish_ioend: 17 callbacks suppressed
[ 2588.861355] dm-0: writeback error on inode 5669593, offset 1048576, sector 27217304
[ 2588.861747] dm-0: writeback error on inode 72173797, offset 655360, sector 47498000
[ 2588.863767] dm-0: writeback error on inode 5686085, offset 81920, sector 5856320
[ 2588.864118] XFS (dm-0): log I/O error -5
[ 2588.864129] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2588.864192] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2589.087112] XFS (dm-0): Unmounting Filesystem
[ 2589.126196] XFS (dm-0): Mounting V5 Filesystem
[ 2589.256674] XFS (dm-0): Starting recovery (logdev: internal)
[ 2589.542959] XFS (dm-0): Ending recovery (logdev: internal)
[ 2589.544017] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2590.551430] buffer_io_error: 54 callbacks suppressed
[ 2590.551434] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2590.551450] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2590.551454] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2590.551457] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2590.551461] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2590.551464] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2590.551467] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2590.551470] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2590.551473] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2590.551476] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2590.776047] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x18 len 8 error 5
[ 2590.777568] dm-0: writeback error on inode 39082395, offset 2023424, sector 26498480
[ 2590.777572] dm-0: writeback error on inode 4054818, offset 58073088, sector 5992088
[ 2590.777578] dm-0: writeback error on inode 4054818, offset 173793280, sector 5998104
[ 2590.777583] dm-0: writeback error on inode 4054818, offset 178622464, sector 5999312
[ 2590.777587] dm-0: writeback error on inode 39629723, offset 909312, sector 27047544
[ 2590.777592] dm-0: writeback error on inode 5686139, offset 26587136, sector 6010928
[ 2590.777596] dm-0: writeback error on inode 72492761, offset 217088, sector 47563032
[ 2590.777692] XFS (dm-0): log I/O error -5
[ 2590.777699] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2590.777757] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2590.803098] XFS (dm-0): Unmounting Filesystem
[ 2590.803127] XFS (dm-0): Failing async write on buffer block 0x4b0. Retrying async write.
[ 2590.803136] XFS (dm-0): Failing async write on buffer block 0x171cf30. Retrying async write.
[ 2590.803140] XFS (dm-0): Failing async write on buffer block 0x171d098. Retrying async write.
[ 2590.803143] XFS (dm-0): Failing async write on buffer block 0x13ffe28. Retrying async write.
[ 2590.803147] XFS (dm-0): Failing async write on buffer block 0x13ffe20. Retrying async write.
[ 2590.803151] XFS (dm-0): Failing async write on buffer block 0x50. Retrying async write.
[ 2590.803155] XFS (dm-0): Failing async write on buffer block 0x18. Retrying async write.
[ 2590.803158] XFS (dm-0): Failing async write on buffer block 0x28. Retrying async write.
[ 2590.803161] XFS (dm-0): Failing async write on buffer block 0x20. Retrying async write.
[ 2590.898854] XFS (dm-0): Mounting V5 Filesystem
[ 2591.015030] XFS (dm-0): Starting recovery (logdev: internal)
[ 2591.269532] XFS (dm-0): Ending recovery (logdev: internal)
[ 2591.270553] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2592.278892] XFS (dm-0): log I/O error -5
[ 2592.278909] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2592.279024] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2592.534075] XFS (dm-0): Unmounting Filesystem
[ 2592.573250] XFS (dm-0): Mounting V5 Filesystem
[ 2592.700931] XFS (dm-0): Starting recovery (logdev: internal)
[ 2593.029802] XFS (dm-0): Ending recovery (logdev: internal)
[ 2593.030847] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2594.233003] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x18 len 8 error 5
[ 2594.233514] iomap_finish_ioend: 21 callbacks suppressed
[ 2594.233516] dm-0: writeback error on inode 38125892, offset 651264, sector 27377000
[ 2594.233526] dm-0: writeback error on inode 36841118, offset 83578880, sector 27377256
[ 2594.233533] dm-0: writeback error on inode 3267698, offset 7778304, sector 6173120
[ 2594.233540] dm-0: writeback error on inode 38125940, offset 524288, sector 27335560
[ 2594.233546] dm-0: writeback error on inode 38125940, offset 1290240, sector 27335264
[ 2594.233553] dm-0: writeback error on inode 3267673, offset 74072064, sector 6173256
[ 2594.233677] XFS (dm-0): log I/O error -5
[ 2594.233688] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2594.233789] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2594.250213] XFS (dm-0): Failing async write on buffer block 0x1869620. Retrying async write.
[ 2594.250220] XFS (dm-0): Failing async write on buffer block 0x1869600. Retrying async write.
[ 2594.259160] XFS (dm-0): Unmounting Filesystem
[ 2594.259192] XFS (dm-0): Failing async write on buffer block 0x4b0. Retrying async write.
[ 2594.259197] XFS (dm-0): Failing async write on buffer block 0x40a8590. Retrying async write.
[ 2594.259201] XFS (dm-0): Failing async write on buffer block 0x50. Retrying async write.
[ 2594.259205] XFS (dm-0): Failing async write on buffer block 0x19cb4a8. Retrying async write.
[ 2594.259212] XFS (dm-0): Failing async write on buffer block 0x598438. Retrying async write.
[ 2594.259215] XFS (dm-0): Failing async write on buffer block 0x2d560a0. Retrying async write.
[ 2594.259221] XFS (dm-0): Failing async write on buffer block 0x14020b0. Retrying async write.
[ 2594.358902] XFS (dm-0): Mounting V5 Filesystem
[ 2594.463445] XFS (dm-0): Starting recovery (logdev: internal)
[ 2594.706900] XFS (dm-0): Ending recovery (logdev: internal)
[ 2594.707815] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2596.713662] dm-0: writeback error on inode 6257128, offset 716800, sector 6320832
[ 2596.713668] dm-0: writeback error on inode 3509159, offset 10272768, sector 3534256
[ 2596.713682] dm-0: writeback error on inode 105936492, offset 1105920, sector 68385744
[ 2596.713687] dm-0: writeback error on inode 72828680, offset 1441792, sector 47710184
[ 2596.713805] XFS (dm-0): log I/O error -5
[ 2596.713816] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2596.713888] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2596.716177] buffer_io_error: 38 callbacks suppressed
[ 2596.716179] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2596.716188] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2596.716191] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2596.716195] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2596.716198] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2596.716201] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2596.716204] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2596.716207] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2596.716210] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2596.716213] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2596.944807] XFS (dm-0): Unmounting Filesystem
[ 2596.987515] XFS (dm-0): Mounting V5 Filesystem
[ 2597.146921] XFS (dm-0): Starting recovery (logdev: internal)
[ 2597.506121] XFS (dm-0): Ending recovery (logdev: internal)
[ 2597.507036] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2598.694283] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x20 len 8 error 5
[ 2598.695795] XFS (dm-0): log I/O error -5
[ 2598.695803] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2598.695860] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2598.695864] XFS (dm-0): log I/O error -5
[ 2598.695866] XFS (dm-0): log I/O error -5
[ 2598.695868] XFS (dm-0): log I/O error -5
[ 2598.695870] XFS (dm-0): log I/O error -5
[ 2598.721263] XFS (dm-0): Unmounting Filesystem
[ 2598.721294] XFS (dm-0): Failing async write on buffer block 0x4b0. Retrying async write.
[ 2598.721301] XFS (dm-0): Failing async write on buffer block 0x14082c0. Retrying async write.
[ 2598.721305] XFS (dm-0): Failing async write on buffer block 0x1402cf8. Retrying async write.
[ 2598.721308] XFS (dm-0): Failing async write on buffer block 0x3c01d80. Retrying async write.
[ 2598.721311] XFS (dm-0): Failing async write on buffer block 0x3c032d0. Retrying async write.
[ 2598.721314] XFS (dm-0): Failing async write on buffer block 0x1869810. Retrying async write.
[ 2598.721317] XFS (dm-0): Failing async write on buffer block 0x3c03808. Retrying async write.
[ 2598.721320] XFS (dm-0): Failing async write on buffer block 0x4092918. Retrying async write.
[ 2598.721324] XFS (dm-0): Failing async write on buffer block 0x680. Retrying async write.
[ 2598.819903] XFS (dm-0): Mounting V5 Filesystem
[ 2598.926017] XFS (dm-0): Starting recovery (logdev: internal)
[ 2599.185692] XFS (dm-0): Ending recovery (logdev: internal)
[ 2599.186561] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2599.232676] XFS (dm-0): log I/O error -5
[ 2599.232679] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2599.232679] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x358b80 len 32 error 5
[ 2599.232693] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2599.232755] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2599.463847] XFS (dm-0): Unmounting Filesystem
[ 2599.480839] XFS (dm-0): Mounting V5 Filesystem
[ 2599.512736] XFS (dm-0): Starting recovery (logdev: internal)
[ 2599.520518] XFS (dm-0): Ending recovery (logdev: internal)
[ 2599.522021] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2601.544049] iomap_finish_ioend: 13 callbacks suppressed
[ 2601.544052] dm-0: writeback error on inode 6344500, offset 77398016, sector 6522176
[ 2601.693208] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x20 len 8 error 5
[ 2601.695457] dm-0: writeback error on inode 73002350, offset 69632, sector 47941664
[ 2601.695637] dm-0: writeback error on inode 73002350, offset 180224, sector 47943056
[ 2601.695645] dm-0: writeback error on inode 6211174, offset 1847296, sector 6452984
[ 2601.695649] dm-0: writeback error on inode 72952535, offset 57147392, sector 47919256
[ 2601.695653] dm-0: writeback error on inode 72967749, offset 46862336, sector 47943344
[ 2601.695657] dm-0: writeback error on inode 106150432, offset 524288, sector 68803664
[ 2601.695661] dm-0: writeback error on inode 106150432, offset 17850368, sector 69367752
[ 2601.695665] dm-0: writeback error on inode 106502774, offset 84537344, sector 69368088
[ 2601.695670] dm-0: writeback error on inode 6447171, offset 106463232, sector 6626344
[ 2601.695792] XFS (dm-0): log I/O error -5
[ 2601.695799] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2601.695857] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2601.725357] XFS (dm-0): Unmounting Filesystem
[ 2601.725391] XFS (dm-0): Failing async write on buffer block 0x2d96240. Retrying async write.
[ 2601.725396] XFS (dm-0): Failing async write on buffer block 0x3fca268. Retrying async write.
[ 2601.725401] XFS (dm-0): Failing async write on buffer block 0x41f6820. Retrying async write.
[ 2601.725404] XFS (dm-0): Failing async write on buffer block 0x41f6800. Retrying async write.
[ 2601.725407] XFS (dm-0): Failing async write on buffer block 0x623aa0. Retrying async write.
[ 2601.725410] XFS (dm-0): Failing async write on buffer block 0x623a80. Retrying async write.
[ 2601.725413] XFS (dm-0): Failing async write on buffer block 0x280a380. Retrying async write.
[ 2601.725416] XFS (dm-0): Failing async write on buffer block 0x2d96280. Retrying async write.
[ 2601.725419] XFS (dm-0): Failing async write on buffer block 0x196c938. Retrying async write.
[ 2601.880399] XFS (dm-0): Mounting V5 Filesystem
[ 2602.048576] XFS (dm-0): Starting recovery (logdev: internal)
[ 2602.486670] XFS (dm-0): Ending recovery (logdev: internal)
[ 2602.487581] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2603.494852] buffer_io_error: 54 callbacks suppressed
[ 2603.494855] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2603.494873] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2603.494877] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2603.494880] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2603.494883] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2603.494887] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2603.494889] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2603.494893] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2603.494896] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2603.494899] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2603.721652] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x18 len 8 error 5
[ 2603.723184] XFS (dm-0): log I/O error -5
[ 2603.723194] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2603.723276] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2603.750782] XFS (dm-0): Unmounting Filesystem
[ 2603.750814] XFS (dm-0): Failing async write on buffer block 0x4b0. Retrying async write.
[ 2603.750820] XFS (dm-0): Failing async write on buffer block 0x1920760. Retrying async write.
[ 2603.750823] XFS (dm-0): Failing async write on buffer block 0x3c02ea0. Retrying async write.
[ 2603.750827] XFS (dm-0): Failing async write on buffer block 0x14020b0. Retrying async write.
[ 2603.750840] XFS (dm-0): Failing async write on buffer block 0x41bb2a0. Retrying async write.
[ 2603.750844] XFS (dm-0): Failing async write on buffer block 0x14155a0. Retrying async write.
[ 2603.750847] XFS (dm-0): Failing async write on buffer block 0x13ffe20. Retrying async write.
[ 2603.750850] XFS (dm-0): Failing async write on buffer block 0x3c01d80. Retrying async write.
[ 2603.750859] XFS (dm-0): Failing async write on buffer block 0x42058e0. Retrying async write.
[ 2603.911355] XFS (dm-0): Mounting V5 Filesystem
[ 2604.039283] XFS (dm-0): Starting recovery (logdev: internal)
[ 2604.316945] XFS (dm-0): Ending recovery (logdev: internal)
[ 2604.317844] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2604.322964] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c07f78 len 8 error 5
[ 2604.322964] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2604.323032] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2604.323042] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c07f78 len 8 error 5
[ 2604.323088] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2604.323104] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c07f78 len 8 error 5
[ 2604.323138] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2604.323169] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c07f78 len 8 error 5
[ 2604.323188] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2604.323247] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2604.324979] XFS (dm-0): log I/O error -5
[ 2604.324999] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2604.325082] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2604.554164] XFS (dm-0): Unmounting Filesystem
[ 2604.571381] XFS (dm-0): Mounting V5 Filesystem
[ 2604.613536] XFS (dm-0): Starting recovery (logdev: internal)
[ 2604.620050] XFS (dm-0): Ending recovery (logdev: internal)
[ 2604.621596] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2604.627298] XFS (dm-0): log I/O error -5
[ 2604.627307] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x1869600 len 32 error 5
[ 2604.627334] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2604.627438] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2604.813933] XFS (dm-0): Unmounting Filesystem
[ 2604.830589] XFS (dm-0): Mounting V5 Filesystem
[ 2604.875920] XFS (dm-0): Starting recovery (logdev: internal)
[ 2604.884974] XFS (dm-0): Ending recovery (logdev: internal)
[ 2604.886426] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2604.932799] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x174bee0 len 32 error 5
[ 2604.932802] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x49a920 len 32 error 5
[ 2604.932803] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x3bffa30 len 8 error 5
[ 2604.932900] XFS (dm-0): log I/O error -5
[ 2604.932915] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2604.932974] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2605.164824] XFS (dm-0): Unmounting Filesystem
[ 2605.181637] XFS (dm-0): Mounting V5 Filesystem
[ 2605.225897] XFS (dm-0): Starting recovery (logdev: internal)
[ 2605.232410] XFS (dm-0): Ending recovery (logdev: internal)
[ 2605.234452] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2607.239943] iomap_finish_ioend: 40 callbacks suppressed
[ 2607.239946] dm-0: writeback error on inode 40220526, offset 31604736, sector 27637824
[ 2607.239960] dm-0: writeback error on inode 40021920, offset 29814784, sector 27875352
[ 2607.239965] dm-0: writeback error on inode 104667118, offset 105447424, sector 69504936
[ 2607.239969] dm-0: writeback error on inode 104667118, offset 171417600, sector 69614576
[ 2607.239974] dm-0: writeback error on inode 6344484, offset 25358336, sector 6857952
[ 2607.240036] XFS (dm-0): log I/O error -5
[ 2607.240049] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2607.240137] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2607.515399] XFS (dm-0): Unmounting Filesystem
[ 2607.560484] XFS (dm-0): Mounting V5 Filesystem
[ 2607.709693] XFS (dm-0): Starting recovery (logdev: internal)
[ 2608.076848] XFS (dm-0): Ending recovery (logdev: internal)
[ 2608.077772] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2608.082680] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x1404e80 len 8 error 5
[ 2608.082679] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
[ 2608.082679] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x49a920 len 32 error 5
[ 2608.082680] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c07f78 len 8 error 5
[ 2608.082758] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x1404e80 len 8 error 5
[ 2608.082810] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x1404e80 len 8 error 5
[ 2608.082862] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x1404e80 len 8 error 5
[ 2608.082882] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x18 len 8 error 5
[ 2608.082962] XFS (dm-0): log I/O error -5
[ 2608.082977] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
[ 2608.083049] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2608.083059] XFS (dm-0): xfs_imap_lookup: xfs_ialloc_read_agi() returned error -5, agno 0
[ 2608.283930] XFS (dm-0): Unmounting Filesystem
[ 2608.301209] XFS (dm-0): Mounting V5 Filesystem
[ 2608.359920] XFS (dm-0): Starting recovery (logdev: internal)
[ 2608.370818] XFS (dm-0): Ending recovery (logdev: internal)
[ 2608.372848] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2608.377740] XFS (dm-0): log I/O error -5
[ 2608.377743] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x1404288 len 8 error 5
[ 2608.377745] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x3fc5f60 len 32 error 5
[ 2608.377761] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2608.377823] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2608.603843] XFS (dm-0): Unmounting Filesystem
[ 2608.620761] XFS (dm-0): Mounting V5 Filesystem
[ 2608.682744] XFS (dm-0): Starting recovery (logdev: internal)
[ 2608.695059] XFS (dm-0): Ending recovery (logdev: internal)
[ 2608.696923] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2609.704216] buffer_io_error: 102 callbacks suppressed
[ 2609.704219] Buffer I/O error on dev dm-0, logical block 10485488, async page read
[ 2609.704235] Buffer I/O error on dev dm-0, logical block 10485489, async page read
[ 2609.704239] Buffer I/O error on dev dm-0, logical block 10485490, async page read
[ 2609.704242] Buffer I/O error on dev dm-0, logical block 10485491, async page read
[ 2609.704246] Buffer I/O error on dev dm-0, logical block 10485492, async page read
[ 2609.704249] Buffer I/O error on dev dm-0, logical block 10485493, async page read
[ 2609.704252] Buffer I/O error on dev dm-0, logical block 10485494, async page read
[ 2609.704255] Buffer I/O error on dev dm-0, logical block 10485495, async page read
[ 2609.704258] Buffer I/O error on dev dm-0, logical block 10485496, async page read
[ 2609.704261] Buffer I/O error on dev dm-0, logical block 10485497, async page read
[ 2609.704971] dm-0: writeback error on inode 73088154, offset 87687168, sector 48133584
[ 2609.868626] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x48 len 8 error 5
[ 2609.869085] dm-0: writeback error on inode 40295105, offset 1048576, sector 27860880
[ 2609.869194] dm-0: writeback error on inode 106528698, offset 983040, sector 68387864
[ 2609.869443] dm-0: writeback error on inode 40295105, offset 479232, sector 27704168
[ 2609.869452] dm-0: writeback error on inode 39669888, offset 46268416, sector 28022120
[ 2609.869639] XFS (dm-0): log I/O error -5
[ 2609.869650] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
[ 2609.869750] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
[ 2609.897807] XFS (dm-0): Unmounting Filesystem
[ 2609.897839] XFS (dm-0): Failing async write on buffer block 0x4b0. Retrying async write.
[ 2609.897851] XFS (dm-0): Failing async write on buffer block 0x1404c08. Retrying async write.
[ 2609.897855] XFS (dm-0): Failing async write on buffer block 0x17ced58. Retrying async write.
[ 2609.897860] XFS (dm-0): Failing async write on buffer block 0x1402cf8. Retrying async write.
[ 2609.897863] XFS (dm-0): Failing async write on buffer block 0x5937e8. Retrying async write.
[ 2609.897866] XFS (dm-0): Failing async write on buffer block 0x680. Retrying async write.
[ 2609.897882] XFS (dm-0): Failing async write on buffer block 0x657400. Retrying async write.
[ 2609.897886] XFS (dm-0): Failing async write on buffer block 0x689620. Retrying async write.
[ 2609.897888] XFS (dm-0): Failing async write on buffer block 0x689600. Retrying async write.
[ 2610.051099] XFS (dm-0): Mounting V5 Filesystem
[ 2610.163869] XFS (dm-0): Starting recovery (logdev: internal)
[ 2610.426613] XFS (dm-0): Ending recovery (logdev: internal)
[ 2610.427541] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
[ 2610.435505] XFS (dm-0): Unmounting Filesystem
[ 2610.546863] XFS (sdb1): Unmounting Filesystem
[ 2611.152679] run fstests generic/476 at 2022-02-15 15:43:38
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [next-20220215] WARNING at fs/iomap/buffered-io.c:75 with xfstests
2022-02-16 7:25 [next-20220215] WARNING at fs/iomap/buffered-io.c:75 with xfstests Sachin Sant
@ 2022-02-16 7:39 ` Stephen Rothwell
2022-02-16 9:47 ` Sachin Sant
2022-02-16 17:35 ` Darrick J. Wong
1 sibling, 1 reply; 6+ messages in thread
From: Stephen Rothwell @ 2022-02-16 7:39 UTC (permalink / raw)
To: Sachin Sant; +Cc: linux-xfs, riteshh, linuxppc-dev, linux-next
[-- Attachment #1: Type: text/plain, Size: 821 bytes --]
Hi Sachin,
On Wed, 16 Feb 2022 12:55:02 +0530 Sachin Sant <sachinp@linux.ibm.com> wrote:
>
> While running xfstests on IBM Power10 logical partition (LPAR) booted
> with 5.17.0-rc4-next-20220215 following warning was seen:
>
> The warning is seen when test tries to unmount the file system. This problem is seen
> while running generic/475 sub test. Have attached captured messages during the test
> run of generic/475.
>
> xfstest is a recent add to upstream regression bucket. I don’t have any previous data
> to attempt a git bisect.
If you have time, could you test v5.17-rc4-2-gd567f5db412e (the commit
in Linus' tree that next-20220215 is based on) and if that OK, then a
bisect from that to 5.17.0-rc4-next-20220215 may be helpful.
Thanks for the report.
--
Cheers,
Stephen Rothwell
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [next-20220215] WARNING at fs/iomap/buffered-io.c:75 with xfstests
2022-02-16 7:39 ` Stephen Rothwell
@ 2022-02-16 9:47 ` Sachin Sant
2022-02-16 11:39 ` Stephen Rothwell
0 siblings, 1 reply; 6+ messages in thread
From: Sachin Sant @ 2022-02-16 9:47 UTC (permalink / raw)
To: Stephen Rothwell; +Cc: linux-xfs, riteshh, linuxppc-dev, linux-next
>> While running xfstests on IBM Power10 logical partition (LPAR) booted
>> with 5.17.0-rc4-next-20220215 following warning was seen:
>>
>> The warning is seen when test tries to unmount the file system. This problem is seen
>> while running generic/475 sub test. Have attached captured messages during the test
>> run of generic/475.
>>
>> xfstest is a recent add to upstream regression bucket. I don’t have any previous data
>> to attempt a git bisect.
>
> If you have time, could you test v5.17-rc4-2-gd567f5db412e (the commit
> in Linus' tree that next-20220215 is based on) and if that OK, then a
> bisect from that to 5.17.0-rc4-next-20220215 may be helpful.
Unfortunately I cannot recreate the problem consistently. I tried same test run with both
mainline as well as linux-next20220215. In both attempts I wasn’t able to recreate it.
Thanks
-Sachin
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [next-20220215] WARNING at fs/iomap/buffered-io.c:75 with xfstests
2022-02-16 9:47 ` Sachin Sant
@ 2022-02-16 11:39 ` Stephen Rothwell
0 siblings, 0 replies; 6+ messages in thread
From: Stephen Rothwell @ 2022-02-16 11:39 UTC (permalink / raw)
To: Sachin Sant; +Cc: linux-xfs, riteshh, linuxppc-dev, linux-next
[-- Attachment #1: Type: text/plain, Size: 1061 bytes --]
Hi Sachin,
On Wed, 16 Feb 2022 15:17:14 +0530 Sachin Sant <sachinp@linux.ibm.com> wrote:
>
> >> While running xfstests on IBM Power10 logical partition (LPAR) booted
> >> with 5.17.0-rc4-next-20220215 following warning was seen:
> >>
> >> The warning is seen when test tries to unmount the file system. This problem is seen
> >> while running generic/475 sub test. Have attached captured messages during the test
> >> run of generic/475.
> >>
> >> xfstest is a recent add to upstream regression bucket. I don’t have any previous data
> >> to attempt a git bisect.
> >
> > If you have time, could you test v5.17-rc4-2-gd567f5db412e (the commit
> > in Linus' tree that next-20220215 is based on) and if that OK, then a
> > bisect from that to 5.17.0-rc4-next-20220215 may be helpful.
>
> Unfortunately I cannot recreate the problem consistently. I tried same test run with both
> mainline as well as linux-next20220215. In both attempts I wasn’t able to recreate it.
No worries, thanks anyway.
--
Cheers,
Stephen Rothwell
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [next-20220215] WARNING at fs/iomap/buffered-io.c:75 with xfstests
2022-02-16 7:25 [next-20220215] WARNING at fs/iomap/buffered-io.c:75 with xfstests Sachin Sant
2022-02-16 7:39 ` Stephen Rothwell
@ 2022-02-16 17:35 ` Darrick J. Wong
2022-02-16 17:46 ` Matthew Wilcox
1 sibling, 1 reply; 6+ messages in thread
From: Darrick J. Wong @ 2022-02-16 17:35 UTC (permalink / raw)
To: Sachin Sant; +Cc: linux-xfs, linux-fsdevel, riteshh, linuxppc-dev, linux-next
[adding fsdevel to this party, since iomap is core code, not just XFS...]
On Wed, Feb 16, 2022 at 12:55:02PM +0530, Sachin Sant wrote:
> While running xfstests on IBM Power10 logical partition (LPAR) booted
> with 5.17.0-rc4-next-20220215 following warning was seen:
>
> [ 2547.384210] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2547.389021] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x2bc2c0 len 32 error 5
> [ 2547.389020] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x15cede0 len 32 error 5
> [ 2547.389026] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0xc60 len 8 error 5
> [ 2547.389032] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x3bffa30 len 8 error 5
> [ 2547.389120] XFS (dm-0): log I/O error -5
> [ 2547.389135] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
Hmm, the filesystem went down...
> [ 2547.389195] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2547.662818] ------------[ cut here ]------------
> [ 2547.662832] WARNING: CPU: 24 PID: 2463718 at fs/iomap/buffered-io.c:75 iomap_page_release+0x1b0/0x1e0
...and I think this is complaining about this debugging test in
iomap_page_release:
WARN_ON_ONCE(bitmap_full(iop->uptodate, nr_blocks) !=
folio_test_uptodate(folio));
which checks that we're not releasing or invalidating a page that's
partially uptodate on a blocksize < pagesize filesystem (or so I gather
from "POWER10 LPAR" (64k pages?) and "XFS" (4k blocks?))...
> [ 2547.662840] Modules linked in: dm_thin_pool dm_persistent_data dm_bio_prison dm_snapshot dm_bufio loop dm_flakey xfs libcrc32c dm_mod rfkill bonding sunrpc pseries_rng xts vmx_crypto uio_pdrv_genirq uio sch_fq_codel ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: scsi_debug]
> [ 2547.662866] CPU: 24 PID: 2463718 Comm: umount Not tainted 5.17.0-rc4-next-20220215 #1
> [ 2547.662871] NIP: c000000000565b80 LR: c000000000565aa0 CTR: c000000000565e40
> [ 2547.662874] REGS: c00000000f0035b0 TRAP: 0700 Not tainted (5.17.0-rc4-next-20220215)
> [ 2547.662877] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44000228 XER: 20040000
> [ 2547.662885] CFAR: c000000000565ac8 IRQMASK: 0
> [ 2547.662885] GPR00: c00000000037dd3c c00000000f003850 c000000002a03300 0000000000000001
> [ 2547.662885] GPR04: 0000000000000010 0000000000010000 c0000000df6e7bc0 0000000000000001
> [ 2547.662885] GPR08: fffffffffffeffff 0000000000000000 0000000000000010 0000000000000228
> [ 2547.662885] GPR12: c000000000565e40 c000000abfcfe680 0000000000000000 0000000000000000
> [ 2547.662885] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ 2547.662885] GPR20: 0000000000000000 0000000000000000 ffffffffffffffff fffffffffffffffe
> [ 2547.662885] GPR24: 0000000000000000 ffffffffffffffff c00000000f0038e0 c0000000243db278
> [ 2547.662885] GPR28: 0000000000000012 c0000000df6e7ae0 0000000000000010 c00c00000240c100
> [ 2547.662920] NIP [c000000000565b80] iomap_page_release+0x1b0/0x1e0
> [ 2547.662924] LR [c000000000565aa0] iomap_page_release+0xd0/0x1e0
> [ 2547.662927] Call Trace:
> [ 2547.662928] [c00000000f003850] [c00000000f003890] 0xc00000000f003890 (unreliable)
> [ 2547.662932] [c00000000f003890] [c00000000037dd3c] truncate_cleanup_folio+0x7c/0x140
> [ 2547.662937] [c00000000f0038c0] [c00000000037f068] truncate_inode_pages_range+0x148/0x5e0
> [ 2547.662942] [c00000000f003a40] [c0000000004c2058] evict+0x248/0x270
> [ 2547.662946] [c00000000f003a80] [c0000000004c20fc] dispose_list+0x7c/0xb0
> [ 2547.662951] [c00000000f003ac0] [c0000000004c2314] evict_inodes+0x1e4/0x300
> [ 2547.662955] [c00000000f003b60] [c0000000004922d0] generic_shutdown_super+0x70/0x1e0
...but in this particular case, we're dumping pages (and inodes) as part
of unmounting the filesystem, in which case we've already flushed dirty
data to disk, because sync_filesystem gets called before evict_inodes
here...
> [ 2547.662959] [c00000000f003bd0] [c000000000492518] kill_block_super+0x38/0x90
> [ 2547.662964] [c00000000f003c00] [c0000000004926e8] deactivate_locked_super+0x78/0xf0
> [ 2547.662968] [c00000000f003c30] [c0000000004cde9c] cleanup_mnt+0xfc/0x1c0
> [ 2547.662972] [c00000000f003c80] [c000000000189448] task_work_run+0xf8/0x170
> [ 2547.662976] [c00000000f003cd0] [c000000000021b94] do_notify_resume+0x434/0x480
> [ 2547.662981] [c00000000f003d80] [c0000000000338b0] interrupt_exit_user_prepare_main+0x1a0/0x260
> [ 2547.662985] [c00000000f003de0] [c000000000033d74] syscall_exit_prepare+0x74/0x150
> [ 2547.662989] [c00000000f003e10] [c00000000000c658] system_call_common+0xf8/0x270
> [ 2547.662994] --- interrupt: c00 at 0x7fffa6bb7dfc
> [ 2547.662996] NIP: 00007fffa6bb7dfc LR: 00007fffa6bb7dcc CTR: 0000000000000000
> [ 2547.662999] REGS: c00000000f003e80 TRAP: 0c00 Not tainted (5.17.0-rc4-next-20220215)
> [ 2547.663001] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28000202 XER: 00000000
> [ 2547.663010] IRQMASK: 0
> [ 2547.663010] GPR00: 0000000000000034 00007fffc7ca6a70 00007fffa6c87300 0000000000000000
> [ 2547.663010] GPR04: 0000000000000000 00007fffc7ca6a88 0000000000000000 0000000166cf4e90
> [ 2547.663010] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ 2547.663010] GPR12: 0000000000000000 00007fffa6f1d730 0000000000000000 0000000000000000
> [ 2547.663010] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ 2547.663010] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ 2547.663010] GPR24: 000000012d3b5f70 000000012d3b65d0 000000012d3cf6e8 0000000166cf0b50
> [ 2547.663010] GPR28: 00007fffa6e701a4 0000000166cf4e90 0000000000000000 0000000166cf0970
> [ 2547.663042] NIP [00007fffa6bb7dfc] 0x7fffa6bb7dfc
> [ 2547.663045] LR [00007fffa6bb7dcc] 0x7fffa6bb7dcc
> [ 2547.663047] --- interrupt: c00
> [ 2547.663048] Instruction dump:
> [ 2547.663050] 38210040 e9230000 ebe1fff8 4e800020 0fe00000 60000000 60000000 60000000
> [ 2547.663056] 0fe00000 60000000 60000000 60000000 <0fe00000> 60000000 60000000 60000000
> [ 2547.663062] ---[ end trace 0000000000000000 ]---
> [ 2547.663075] XFS (dm-0): Unmounting Filesystem
>
> The warning is seen when test tries to unmount the file system. This problem is seen
> while running generic/475 sub test. Have attached captured messages during the test
> run of generic/475.
...aha! The filesystem has gone down (hence the metadata IO errors at
the top of the report) during generic/475, which is a test that
exercises filesystem crash recovery. It simulates a crash by calling
EXT4_IOC_SHUTDOWN and unmounting.
Given that in this case we've already cleared SB_ACTIVE from the
superblock s_flags, I wonder if we could amend that code to read:
if (inode->i_sb->s_flags & SB_ACTIVE)
WARN_ON_ONCE(bitmap_full(iop->uptodate, nr_blocks) !=
folio_test_uptodate(folio));
Since we don't really care about memory pages that aren't fully up to
date if we're in the midst of freeing all the incore filesystem state.
Thoughts?
> xfstest is a recent add to upstream regression bucket.
Ooh, nice! :)
> I don’t have any previous data to attempt a git bisect.
It's been there since the start of fs/iomap, so bisecting wouldn't help
much anyway.
--D
> Let me know if any additional information is required.
>
> Thanks
> -Sachin
>
>
> [ 2523.250933] run fstests generic/475 at 2022-02-15 15:42:10
> [ 2523.366239] XFS (sdb1): Mounting V5 Filesystem
> [ 2523.376679] XFS (sdb1): Ending clean mount
> [ 2523.378107] xfs filesystem being mounted at /mnt/test supports timestamps until 2038 (0x7fffffff)
> [ 2523.502357] XFS (dm-0): Mounting V5 Filesystem
> [ 2523.508526] XFS (dm-0): Ending clean mount
> [ 2523.509735] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2524.516736] dm-0: writeback error on inode 33750928, offset 655360, sector 21228312
> [ 2524.517884] dm-0: writeback error on inode 100730311, offset 589824, sector 63235832
> [ 2524.518348] dm-0: writeback error on inode 67165541, offset 258371584, sector 42405248
> [ 2524.518625] dm-0: writeback error on inode 136194, offset 417792, sector 212888
> [ 2524.518727] dm-0: writeback error on inode 114612, offset 1052672, sector 295824
> [ 2524.518876] buffer_io_error: 6 callbacks suppressed
> [ 2524.518878] Buffer I/O error on dev dm-0, logical block 10485488, async page read
> [ 2524.518888] Buffer I/O error on dev dm-0, logical block 10485489, async page read
> [ 2524.518892] Buffer I/O error on dev dm-0, logical block 10485490, async page read
> [ 2524.518895] Buffer I/O error on dev dm-0, logical block 10485491, async page read
> [ 2524.518898] Buffer I/O error on dev dm-0, logical block 10485492, async page read
> [ 2524.518901] Buffer I/O error on dev dm-0, logical block 10485493, async page read
> [ 2524.518905] Buffer I/O error on dev dm-0, logical block 10485494, async page read
> [ 2524.518908] Buffer I/O error on dev dm-0, logical block 10485495, async page read
> [ 2524.518911] Buffer I/O error on dev dm-0, logical block 10485496, async page read
> [ 2524.518914] Buffer I/O error on dev dm-0, logical block 10485497, async page read
> [ 2524.519703] dm-0: writeback error on inode 67200889, offset 6684672, sector 42237480
> [ 2524.520249] dm-0: writeback error on inode 100772183, offset 30445568, sector 63105160
> [ 2524.520258] dm-0: writeback error on inode 100835769, offset 270336, sector 63079496
> [ 2524.520266] dm-0: writeback error on inode 114527, offset 40960, sector 84968
> [ 2524.520272] dm-0: writeback error on inode 67244195, offset 25001984, sector 42327408
> [ 2524.520710] XFS (dm-0): log I/O error -5
> [ 2524.520719] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2524.520780] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2524.793985] XFS (dm-0): Unmounting Filesystem
> [ 2524.870119] XFS (dm-0): Mounting V5 Filesystem
> [ 2525.015369] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2525.418197] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2525.419156] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2525.425641] XFS (dm-0): log I/O error -5
> [ 2525.425645] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x2c3c0 len 32 error 5
> [ 2525.425647] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x3bffa80 len 32 error 5
> [ 2525.425656] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2525.425711] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2525.592468] XFS (dm-0): Unmounting Filesystem
> [ 2525.609374] XFS (dm-0): Mounting V5 Filesystem
> [ 2525.681666] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2525.694463] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2525.696357] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2526.704897] XFS (dm-0): log I/O error -5
> [ 2526.704923] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2526.705029] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2526.948328] XFS (dm-0): Unmounting Filesystem
> [ 2527.040926] XFS (dm-0): Mounting V5 Filesystem
> [ 2527.216170] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2527.749123] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2527.750067] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2527.755047] XFS (dm-0): log I/O error -5
> [ 2527.755045] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x3bffa30 len 8 error 5
> [ 2527.755061] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x13ffe18 len 8 error 5
> [ 2527.755080] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2527.755161] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2527.923532] XFS (dm-0): Unmounting Filesystem
> [ 2527.940178] XFS (dm-0): Mounting V5 Filesystem
> [ 2527.999573] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2528.014009] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2528.016171] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2528.021163] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x15c0 len 32 error 5
> [ 2528.021161] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x13ffe80 len 32 error 5
> [ 2528.021247] XFS (dm-0): log I/O error -5
> [ 2528.021260] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
> [ 2528.021319] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2528.261691] XFS (dm-0): Unmounting Filesystem
> [ 2528.278383] XFS (dm-0): Mounting V5 Filesystem
> [ 2528.337759] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2528.352023] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2528.354722] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2528.359910] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x1415598 len 8 error 5
> [ 2528.359909] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x3bffaa0 len 32 error 5
> [ 2528.360016] XFS (dm-0): xfs_imap_lookup: xfs_ialloc_read_agi() returned error -5, agno 0
> [ 2528.360045] XFS (dm-0): log I/O error -5
> [ 2528.360016] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x1415598 len 8 error 5
> [ 2528.360059] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
> [ 2528.360128] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2528.603702] XFS (dm-0): Unmounting Filesystem
> [ 2528.620367] XFS (dm-0): Mounting V5 Filesystem
> [ 2528.629620] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2528.634048] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2528.635707] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2530.643300] buffer_io_error: 86 callbacks suppressed
> [ 2530.643303] Buffer I/O error on dev dm-0, logical block 10485488, async page read
> [ 2530.643319] Buffer I/O error on dev dm-0, logical block 10485489, async page read
> [ 2530.643323] Buffer I/O error on dev dm-0, logical block 10485490, async page read
> [ 2530.643326] Buffer I/O error on dev dm-0, logical block 10485491, async page read
> [ 2530.643330] Buffer I/O error on dev dm-0, logical block 10485492, async page read
> [ 2530.643333] Buffer I/O error on dev dm-0, logical block 10485493, async page read
> [ 2530.643336] Buffer I/O error on dev dm-0, logical block 10485494, async page read
> [ 2530.643339] Buffer I/O error on dev dm-0, logical block 10485495, async page read
> [ 2530.643342] Buffer I/O error on dev dm-0, logical block 10485496, async page read
> [ 2530.643345] Buffer I/O error on dev dm-0, logical block 10485497, async page read
> [ 2530.644362] iomap_finish_ioend: 221 callbacks suppressed
> [ 2530.644364] dm-0: writeback error on inode 34090876, offset 3014656, sector 22112200
> [ 2530.644538] dm-0: writeback error on inode 34427948, offset 786432, sector 22282544
> [ 2530.644760] dm-0: writeback error on inode 178971, offset 131072, sector 1225112
> [ 2530.645682] dm-0: writeback error on inode 178976, offset 327680, sector 857912
> [ 2530.645690] dm-0: writeback error on inode 101264072, offset 884736, sector 63870264
> [ 2530.645694] dm-0: writeback error on inode 178976, offset 331776, sector 1089144
> [ 2530.645699] dm-0: writeback error on inode 178976, offset 389120, sector 1089256
> [ 2530.652881] dm-0: writeback error on inode 67151740, offset 7122944, sector 43064768
> [ 2530.653420] dm-0: writeback error on inode 385792, offset 5693440, sector 1185032
> [ 2530.653602] dm-0: writeback error on inode 34090849, offset 2056192, sector 21840064
> [ 2530.653811] XFS (dm-0): log I/O error -5
> [ 2530.653819] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2530.653898] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2530.894975] XFS (dm-0): Unmounting Filesystem
> [ 2530.953709] XFS (dm-0): Mounting V5 Filesystem
> [ 2531.116725] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2531.619363] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2531.620537] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2533.630182] XFS (dm-0): log I/O error -5
> [ 2533.630204] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2533.630296] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2533.875912] XFS (dm-0): Unmounting Filesystem
> [ 2533.944095] XFS (dm-0): Mounting V5 Filesystem
> [ 2534.097933] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2534.545681] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2534.546748] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2534.552289] XFS (dm-0): log I/O error -5
> [ 2534.552294] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x86540 len 32 error 5
> [ 2534.552317] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2534.552363] XFS (dm-0): xfs_imap_lookup: xfs_ialloc_read_agi() returned error -5, agno 0
> [ 2534.552378] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2534.752003] XFS (dm-0): Unmounting Filesystem
> [ 2534.768805] XFS (dm-0): Mounting V5 Filesystem
> [ 2534.827769] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2534.841175] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2534.843039] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2536.849519] iomap_finish_ioend: 102 callbacks suppressed
> [ 2536.849523] dm-0: writeback error on inode 1413851, offset 89260032, sector 2095432
> [ 2536.850440] buffer_io_error: 38 callbacks suppressed
> [ 2536.850442] Buffer I/O error on dev dm-0, logical block 10485488, async page read
> [ 2536.850453] Buffer I/O error on dev dm-0, logical block 10485489, async page read
> [ 2536.850457] Buffer I/O error on dev dm-0, logical block 10485490, async page read
> [ 2536.850460] Buffer I/O error on dev dm-0, logical block 10485491, async page read
> [ 2536.850463] Buffer I/O error on dev dm-0, logical block 10485492, async page read
> [ 2536.850467] Buffer I/O error on dev dm-0, logical block 10485493, async page read
> [ 2536.850470] Buffer I/O error on dev dm-0, logical block 10485494, async page read
> [ 2536.850473] Buffer I/O error on dev dm-0, logical block 10485495, async page read
> [ 2536.850476] Buffer I/O error on dev dm-0, logical block 10485496, async page read
> [ 2536.850479] Buffer I/O error on dev dm-0, logical block 10485497, async page read
> [ 2536.851985] dm-0: writeback error on inode 1392803, offset 7856128, sector 2156768
> [ 2536.851989] dm-0: writeback error on inode 1358735, offset 32309248, sector 1874480
> [ 2536.852481] dm-0: writeback error on inode 102083501, offset 5750784, sector 65056496
> [ 2536.857616] dm-0: writeback error on inode 1369530, offset 786432, sector 1936752
> [ 2536.858012] dm-0: writeback error on inode 1729116, offset 1294336, sector 2157232
> [ 2536.858019] dm-0: writeback error on inode 68591248, offset 29708288, sector 44249952
> [ 2536.858025] dm-0: writeback error on inode 35290493, offset 67264512, sector 23190648
> [ 2536.858030] dm-0: writeback error on inode 35290493, offset 77107200, sector 23302296
> [ 2536.858035] dm-0: writeback error on inode 1369530, offset 868352, sector 1937336
> [ 2536.858192] XFS (dm-0): log I/O error -5
> [ 2536.858202] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2536.858282] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2537.084788] XFS (dm-0): Unmounting Filesystem
> [ 2537.125735] XFS (dm-0): Mounting V5 Filesystem
> [ 2537.252751] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2537.583254] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2537.584277] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2538.595093] XFS (dm-0): log I/O error -5
> [ 2538.595110] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2538.595196] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2538.864000] XFS (dm-0): Unmounting Filesystem
> [ 2538.905962] XFS (dm-0): Mounting V5 Filesystem
> [ 2539.096169] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2539.633947] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2539.634958] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2541.640724] XFS (dm-0): log I/O error -5
> [ 2541.640741] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2541.640802] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2541.935012] XFS (dm-0): Unmounting Filesystem
> [ 2541.988684] XFS (dm-0): Mounting V5 Filesystem
> [ 2542.116431] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2542.441152] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2542.442150] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2544.448191] iomap_finish_ioend: 46 callbacks suppressed
> [ 2544.448194] dm-0: writeback error on inode 36077872, offset 73859072, sector 24110792
> [ 2544.449711] buffer_io_error: 38 callbacks suppressed
> [ 2544.449712] Buffer I/O error on dev dm-0, logical block 10485488, async page read
> [ 2544.449723] Buffer I/O error on dev dm-0, logical block 10485489, async page read
> [ 2544.449727] Buffer I/O error on dev dm-0, logical block 10485490, async page read
> [ 2544.449728] dm-0: writeback error on inode 2692401, offset 458752, sector 2775128
> [ 2544.449730] Buffer I/O error on dev dm-0, logical block 10485491, async page read
> [ 2544.449735] Buffer I/O error on dev dm-0, logical block 10485492, async page read
> [ 2544.449738] Buffer I/O error on dev dm-0, logical block 10485493, async page read
> [ 2544.449742] Buffer I/O error on dev dm-0, logical block 10485494, async page read
> [ 2544.449744] Buffer I/O error on dev dm-0, logical block 10485495, async page read
> [ 2544.449748] Buffer I/O error on dev dm-0, logical block 10485496, async page read
> [ 2544.449751] Buffer I/O error on dev dm-0, logical block 10485497, async page read
> [ 2544.458404] dm-0: writeback error on inode 2391873, offset 466944, sector 2391824
> [ 2544.458447] dm-0: writeback error on inode 103160128, offset 937984, sector 65927664
> [ 2544.458543] dm-0: writeback error on inode 36054700, offset 33423360, sector 24188272
> [ 2544.458549] dm-0: writeback error on inode 103092984, offset 28631040, sector 65845792
> [ 2544.458553] dm-0: writeback error on inode 2392584, offset 74911744, sector 2755816
> [ 2544.458558] dm-0: writeback error on inode 69726030, offset 21921792, sector 44540328
> [ 2544.458563] dm-0: writeback error on inode 2669138, offset 1040384, sector 2801096
> [ 2544.458566] dm-0: writeback error on inode 2391873, offset 458752, sector 2391960
> [ 2544.458851] XFS (dm-0): log I/O error -5
> [ 2544.458858] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2544.458917] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2544.683010] XFS (dm-0): Unmounting Filesystem
> [ 2544.739531] XFS (dm-0): Mounting V5 Filesystem
> [ 2544.914316] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2545.426712] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2545.427672] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2546.438765] XFS (dm-0): log I/O error -5
> [ 2546.438781] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2546.438842] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2546.727004] XFS (dm-0): Unmounting Filesystem
> [ 2546.769608] XFS (dm-0): Mounting V5 Filesystem
> [ 2546.936362] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2547.383261] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2547.384210] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2547.389021] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x2bc2c0 len 32 error 5
> [ 2547.389020] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x15cede0 len 32 error 5
> [ 2547.389026] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0xc60 len 8 error 5
> [ 2547.389032] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x3bffa30 len 8 error 5
> [ 2547.389120] XFS (dm-0): log I/O error -5
> [ 2547.389135] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
> [ 2547.389195] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2547.662818] ------------[ cut here ]------------
> [ 2547.662832] WARNING: CPU: 24 PID: 2463718 at fs/iomap/buffered-io.c:75 iomap_page_release+0x1b0/0x1e0
> [ 2547.662840] Modules linked in: dm_thin_pool dm_persistent_data dm_bio_prison dm_snapshot dm_bufio loop dm_flakey xfs libcrc32c dm_mod rfkill bonding sunrpc pseries_rng xts vmx_crypto uio_pdrv_genirq uio sch_fq_codel ext4 mbcache jbd2 sd_mod t10_pi sg ibmvscsi ibmveth scsi_transport_srp fuse [last unloaded: scsi_debug]
> [ 2547.662866] CPU: 24 PID: 2463718 Comm: umount Not tainted 5.17.0-rc4-next-20220215 #1
> [ 2547.662871] NIP: c000000000565b80 LR: c000000000565aa0 CTR: c000000000565e40
> [ 2547.662874] REGS: c00000000f0035b0 TRAP: 0700 Not tainted (5.17.0-rc4-next-20220215)
> [ 2547.662877] MSR: 800000000282b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44000228 XER: 20040000
> [ 2547.662885] CFAR: c000000000565ac8 IRQMASK: 0
> [ 2547.662885] GPR00: c00000000037dd3c c00000000f003850 c000000002a03300 0000000000000001
> [ 2547.662885] GPR04: 0000000000000010 0000000000010000 c0000000df6e7bc0 0000000000000001
> [ 2547.662885] GPR08: fffffffffffeffff 0000000000000000 0000000000000010 0000000000000228
> [ 2547.662885] GPR12: c000000000565e40 c000000abfcfe680 0000000000000000 0000000000000000
> [ 2547.662885] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ 2547.662885] GPR20: 0000000000000000 0000000000000000 ffffffffffffffff fffffffffffffffe
> [ 2547.662885] GPR24: 0000000000000000 ffffffffffffffff c00000000f0038e0 c0000000243db278
> [ 2547.662885] GPR28: 0000000000000012 c0000000df6e7ae0 0000000000000010 c00c00000240c100
> [ 2547.662920] NIP [c000000000565b80] iomap_page_release+0x1b0/0x1e0
> [ 2547.662924] LR [c000000000565aa0] iomap_page_release+0xd0/0x1e0
> [ 2547.662927] Call Trace:
> [ 2547.662928] [c00000000f003850] [c00000000f003890] 0xc00000000f003890 (unreliable)
> [ 2547.662932] [c00000000f003890] [c00000000037dd3c] truncate_cleanup_folio+0x7c/0x140
> [ 2547.662937] [c00000000f0038c0] [c00000000037f068] truncate_inode_pages_range+0x148/0x5e0
> [ 2547.662942] [c00000000f003a40] [c0000000004c2058] evict+0x248/0x270
> [ 2547.662946] [c00000000f003a80] [c0000000004c20fc] dispose_list+0x7c/0xb0
> [ 2547.662951] [c00000000f003ac0] [c0000000004c2314] evict_inodes+0x1e4/0x300
> [ 2547.662955] [c00000000f003b60] [c0000000004922d0] generic_shutdown_super+0x70/0x1e0
> [ 2547.662959] [c00000000f003bd0] [c000000000492518] kill_block_super+0x38/0x90
> [ 2547.662964] [c00000000f003c00] [c0000000004926e8] deactivate_locked_super+0x78/0xf0
> [ 2547.662968] [c00000000f003c30] [c0000000004cde9c] cleanup_mnt+0xfc/0x1c0
> [ 2547.662972] [c00000000f003c80] [c000000000189448] task_work_run+0xf8/0x170
> [ 2547.662976] [c00000000f003cd0] [c000000000021b94] do_notify_resume+0x434/0x480
> [ 2547.662981] [c00000000f003d80] [c0000000000338b0] interrupt_exit_user_prepare_main+0x1a0/0x260
> [ 2547.662985] [c00000000f003de0] [c000000000033d74] syscall_exit_prepare+0x74/0x150
> [ 2547.662989] [c00000000f003e10] [c00000000000c658] system_call_common+0xf8/0x270
> [ 2547.662994] --- interrupt: c00 at 0x7fffa6bb7dfc
> [ 2547.662996] NIP: 00007fffa6bb7dfc LR: 00007fffa6bb7dcc CTR: 0000000000000000
> [ 2547.662999] REGS: c00000000f003e80 TRAP: 0c00 Not tainted (5.17.0-rc4-next-20220215)
> [ 2547.663001] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE> CR: 28000202 XER: 00000000
> [ 2547.663010] IRQMASK: 0
> [ 2547.663010] GPR00: 0000000000000034 00007fffc7ca6a70 00007fffa6c87300 0000000000000000
> [ 2547.663010] GPR04: 0000000000000000 00007fffc7ca6a88 0000000000000000 0000000166cf4e90
> [ 2547.663010] GPR08: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ 2547.663010] GPR12: 0000000000000000 00007fffa6f1d730 0000000000000000 0000000000000000
> [ 2547.663010] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ 2547.663010] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
> [ 2547.663010] GPR24: 000000012d3b5f70 000000012d3b65d0 000000012d3cf6e8 0000000166cf0b50
> [ 2547.663010] GPR28: 00007fffa6e701a4 0000000166cf4e90 0000000000000000 0000000166cf0970
> [ 2547.663042] NIP [00007fffa6bb7dfc] 0x7fffa6bb7dfc
> [ 2547.663045] LR [00007fffa6bb7dcc] 0x7fffa6bb7dcc
> [ 2547.663047] --- interrupt: c00
> [ 2547.663048] Instruction dump:
> [ 2547.663050] 38210040 e9230000 ebe1fff8 4e800020 0fe00000 60000000 60000000 60000000
> [ 2547.663056] 0fe00000 60000000 60000000 60000000 <0fe00000> 60000000 60000000 60000000
> [ 2547.663062] ---[ end trace 0000000000000000 ]---
> [ 2547.663075] XFS (dm-0): Unmounting Filesystem
> [ 2547.679882] XFS (dm-0): Mounting V5 Filesystem
> [ 2547.733927] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2547.745633] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2547.748178] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2549.753836] iomap_finish_ioend: 43 callbacks suppressed
> [ 2549.753839] dm-0: writeback error on inode 103859326, offset 65249280, sector 66209096
> [ 2549.753854] dm-0: writeback error on inode 36109178, offset 6733824, sector 24398488
> [ 2549.753859] dm-0: writeback error on inode 70328836, offset 1802240, sector 45162048
> [ 2549.753863] dm-0: writeback error on inode 36480701, offset 108093440, sector 24428192
> [ 2549.753869] dm-0: writeback error on inode 69618107, offset 22708224, sector 45250888
> [ 2549.753873] dm-0: writeback error on inode 36521924, offset 104202240, sector 24543472
> [ 2549.753877] dm-0: writeback error on inode 36521924, offset 170885120, sector 24541200
> [ 2549.753881] dm-0: writeback error on inode 70357590, offset 3289088, sector 45331584
> [ 2549.753885] dm-0: writeback error on inode 103289852, offset 1003520, sector 66297216
> [ 2549.754924] XFS (dm-0): log I/O error -5
> [ 2549.754950] XFS (dm-0): log I/O error -5
> [ 2549.754940] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2549.755035] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2549.756230] buffer_io_error: 38 callbacks suppressed
> [ 2549.756233] Buffer I/O error on dev dm-0, logical block 10485488, async page read
> [ 2549.756242] Buffer I/O error on dev dm-0, logical block 10485489, async page read
> [ 2549.756246] Buffer I/O error on dev dm-0, logical block 10485490, async page read
> [ 2549.756249] Buffer I/O error on dev dm-0, logical block 10485491, async page read
> [ 2549.756252] Buffer I/O error on dev dm-0, logical block 10485492, async page read
> [ 2549.756255] Buffer I/O error on dev dm-0, logical block 10485493, async page read
> [ 2549.756259] Buffer I/O error on dev dm-0, logical block 10485494, async page read
> [ 2549.756262] Buffer I/O error on dev dm-0, logical block 10485495, async page read
> [ 2549.756265] Buffer I/O error on dev dm-0, logical block 10485496, async page read
> [ 2549.756268] Buffer I/O error on dev dm-0, logical block 10485497, async page read
> [ 2550.021468] XFS (dm-0): Unmounting Filesystem
> [ 2550.080673] XFS (dm-0): Mounting V5 Filesystem
> [ 2550.270646] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2550.760973] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2550.762099] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2551.767871] dm-0: writeback error on inode 70406127, offset 124911616, sector 45548040
> [ 2551.767885] XFS (dm-0): log I/O error -5
> [ 2551.768068] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2551.768140] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2552.001984] XFS (dm-0): Unmounting Filesystem
> [ 2552.033560] XFS (dm-0): Mounting V5 Filesystem
> [ 2552.202493] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2552.629573] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2552.630476] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2554.636725] XFS (dm-0): log I/O error -5
> [ 2554.636740] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2554.636803] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2554.906255] XFS (dm-0): Unmounting Filesystem
> [ 2554.964125] XFS (dm-0): Mounting V5 Filesystem
> [ 2555.118190] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2555.480176] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2555.481202] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2556.487342] XFS (dm-0): log I/O error -5
> [ 2556.487360] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2556.487423] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2556.487427] XFS (dm-0): log I/O error -5
> [ 2556.487429] XFS (dm-0): log I/O error -5
> [ 2556.487431] XFS (dm-0): log I/O error -5
> [ 2556.489554] buffer_io_error: 38 callbacks suppressed
> [ 2556.489556] Buffer I/O error on dev dm-0, logical block 10485488, async page read
> [ 2556.489565] Buffer I/O error on dev dm-0, logical block 10485489, async page read
> [ 2556.489569] Buffer I/O error on dev dm-0, logical block 10485490, async page read
> [ 2556.489572] Buffer I/O error on dev dm-0, logical block 10485491, async page read
> [ 2556.489575] Buffer I/O error on dev dm-0, logical block 10485492, async page read
> [ 2556.489579] Buffer I/O error on dev dm-0, logical block 10485493, async page read
> [ 2556.489582] Buffer I/O error on dev dm-0, logical block 10485494, async page read
> [ 2556.489585] Buffer I/O error on dev dm-0, logical block 10485495, async page read
> [ 2556.489588] Buffer I/O error on dev dm-0, logical block 10485496, async page read
> [ 2556.489591] Buffer I/O error on dev dm-0, logical block 10485497, async page read
> [ 2556.722895] XFS (dm-0): Unmounting Filesystem
> [ 2556.781571] XFS (dm-0): Mounting V5 Filesystem
> [ 2556.962599] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2557.386041] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2557.387016] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2558.400557] iomap_finish_ioend: 41 callbacks suppressed
> [ 2558.400560] dm-0: writeback error on inode 104621403, offset 393216, sector 66985016
> [ 2558.400788] dm-0: writeback error on inode 2973309, offset 3555328, sector 4043448
> [ 2558.400797] dm-0: writeback error on inode 3532245, offset 27906048, sector 4008328
> [ 2558.400802] dm-0: writeback error on inode 104656344, offset 114626560, sector 66996824
> [ 2558.400808] dm-0: writeback error on inode 104621424, offset 77824, sector 66916624
> [ 2558.400872] XFS (dm-0): log I/O error -5
> [ 2558.400880] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2558.400967] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2558.625070] XFS (dm-0): Unmounting Filesystem
> [ 2558.672009] XFS (dm-0): Mounting V5 Filesystem
> [ 2558.811196] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2559.196329] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2559.197243] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2559.202340] XFS (dm-0): log I/O error -5
> [ 2559.202345] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x3f19480 len 32 error 5
> [ 2559.202361] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2559.202420] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2559.432336] XFS (dm-0): Unmounting Filesystem
> [ 2559.449047] XFS (dm-0): Mounting V5 Filesystem
> [ 2559.490582] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2559.499664] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2559.501334] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2561.508579] buffer_io_error: 38 callbacks suppressed
> [ 2561.508582] Buffer I/O error on dev dm-0, logical block 10485488, async page read
> [ 2561.508599] Buffer I/O error on dev dm-0, logical block 10485489, async page read
> [ 2561.508603] Buffer I/O error on dev dm-0, logical block 10485490, async page read
> [ 2561.508606] Buffer I/O error on dev dm-0, logical block 10485491, async page read
> [ 2561.508609] Buffer I/O error on dev dm-0, logical block 10485492, async page read
> [ 2561.508613] Buffer I/O error on dev dm-0, logical block 10485493, async page read
> [ 2561.508615] Buffer I/O error on dev dm-0, logical block 10485494, async page read
> [ 2561.508619] Buffer I/O error on dev dm-0, logical block 10485495, async page read
> [ 2561.508622] Buffer I/O error on dev dm-0, logical block 10485496, async page read
> [ 2561.508625] Buffer I/O error on dev dm-0, logical block 10485497, async page read
> [ 2561.512702] dm-0: writeback error on inode 104703262, offset 147324928, sector 67200392
> [ 2561.523022] dm-0: writeback error on inode 37886195, offset 131072, sector 25502904
> [ 2561.550302] XFS (dm-0): log I/O error -5
> [ 2561.550316] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2561.550376] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2561.550380] XFS (dm-0): log I/O error -5
> [ 2561.550382] XFS (dm-0): log I/O error -5
> [ 2561.550384] XFS (dm-0): log I/O error -5
> [ 2561.789009] XFS (dm-0): Unmounting Filesystem
> [ 2561.873689] XFS (dm-0): Mounting V5 Filesystem
> [ 2562.009743] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2562.358919] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2562.359856] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2564.370069] dm-0: writeback error on inode 102464293, offset 2129920, sector 67036960
> [ 2564.370076] dm-0: writeback error on inode 37987634, offset 62914560, sector 25414248
> [ 2564.370102] dm-0: writeback error on inode 37987634, offset 128565248, sector 25743352
> [ 2564.370113] dm-0: writeback error on inode 4143566, offset 1355776, sector 4470760
> [ 2564.370119] dm-0: writeback error on inode 37386218, offset 1511424, sector 25184776
> [ 2564.370125] dm-0: writeback error on inode 71426166, offset 44986368, sector 46349904
> [ 2564.370130] dm-0: writeback error on inode 4233901, offset 22544384, sector 4480616
> [ 2564.370135] dm-0: writeback error on inode 37935212, offset 44507136, sector 25651960
> [ 2564.370141] dm-0: writeback error on inode 105027811, offset 581632, sector 67474168
> [ 2564.370147] dm-0: writeback error on inode 37886177, offset 34197504, sector 25582624
> [ 2564.370451] XFS (dm-0): log I/O error -5
> [ 2564.370460] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2564.370521] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2564.603752] XFS (dm-0): Unmounting Filesystem
> [ 2564.668356] XFS (dm-0): Mounting V5 Filesystem
> [ 2564.815810] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2565.223316] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2565.224235] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2567.231741] buffer_io_error: 22 callbacks suppressed
> [ 2567.231744] Buffer I/O error on dev dm-0, logical block 10485488, async page read
> [ 2567.231762] Buffer I/O error on dev dm-0, logical block 10485489, async page read
> [ 2567.231765] Buffer I/O error on dev dm-0, logical block 10485490, async page read
> [ 2567.231768] Buffer I/O error on dev dm-0, logical block 10485491, async page read
> [ 2567.231772] Buffer I/O error on dev dm-0, logical block 10485492, async page read
> [ 2567.231775] Buffer I/O error on dev dm-0, logical block 10485493, async page read
> [ 2567.231778] Buffer I/O error on dev dm-0, logical block 10485494, async page read
> [ 2567.231781] Buffer I/O error on dev dm-0, logical block 10485495, async page read
> [ 2567.231785] Buffer I/O error on dev dm-0, logical block 10485496, async page read
> [ 2567.231788] Buffer I/O error on dev dm-0, logical block 10485497, async page read
> [ 2567.442435] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x1c10 len 8 error 5
> [ 2567.446460] XFS (dm-0): log I/O error -5
> [ 2567.446469] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2567.446526] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2567.469623] XFS (dm-0): Unmounting Filesystem
> [ 2567.469653] XFS (dm-0): Failing async write on buffer block 0x421c00. Retrying async write.
> [ 2567.469660] XFS (dm-0): Failing async write on buffer block 0x421bd8. Retrying async write.
> [ 2567.469663] XFS (dm-0): Failing async write on buffer block 0x2c47ac8. Retrying async write.
> [ 2567.469666] XFS (dm-0): Failing async write on buffer block 0x3db7618. Retrying async write.
> [ 2567.469669] XFS (dm-0): Failing async write on buffer block 0x3d39bc0. Retrying async write.
> [ 2567.469672] XFS (dm-0): Failing async write on buffer block 0x3d39c00. Retrying async write.
> [ 2567.469675] XFS (dm-0): Failing async write on buffer block 0x2c46c70. Retrying async write.
> [ 2567.469679] XFS (dm-0): Failing async write on buffer block 0x1724220. Retrying async write.
> [ 2567.469683] XFS (dm-0): Failing async write on buffer block 0x3d39c08. Retrying async write.
> [ 2567.605306] XFS (dm-0): Mounting V5 Filesystem
> [ 2567.778964] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2568.250498] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2568.251454] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2570.257461] iomap_finish_ioend: 93 callbacks suppressed
> [ 2570.257465] dm-0: writeback error on inode 71736329, offset 557056, sector 46797816
> [ 2570.277368] dm-0: writeback error on inode 101747424, offset 798720, sector 4883144
> [ 2570.277386] dm-0: writeback error on inode 4112654, offset 65536, sector 4765656
> [ 2570.277392] dm-0: writeback error on inode 71576811, offset 24985600, sector 46689208
> [ 2570.277397] dm-0: writeback error on inode 71596503, offset 56389632, sector 46663520
> [ 2570.277403] dm-0: writeback error on inode 71596485, offset 70684672, sector 46728408
> [ 2570.277408] dm-0: writeback error on inode 105195353, offset 671744, sector 67776440
> [ 2570.277414] dm-0: writeback error on inode 71399352, offset 10092544, sector 46274472
> [ 2570.277419] dm-0: writeback error on inode 71399352, offset 34680832, sector 46800296
> [ 2570.277424] dm-0: writeback error on inode 4500080, offset 18259968, sector 4845360
> [ 2570.277599] XFS (dm-0): log I/O error -5
> [ 2570.277608] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2570.277680] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2570.509378] XFS (dm-0): Unmounting Filesystem
> [ 2570.565428] XFS (dm-0): Mounting V5 Filesystem
> [ 2570.757446] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2571.291433] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2571.292377] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2572.300294] buffer_io_error: 22 callbacks suppressed
> [ 2572.300297] Buffer I/O error on dev dm-0, logical block 10485488, async page read
> [ 2572.300313] Buffer I/O error on dev dm-0, logical block 10485489, async page read
> [ 2572.300317] Buffer I/O error on dev dm-0, logical block 10485490, async page read
> [ 2572.300320] Buffer I/O error on dev dm-0, logical block 10485491, async page read
> [ 2572.300323] Buffer I/O error on dev dm-0, logical block 10485492, async page read
> [ 2572.300326] Buffer I/O error on dev dm-0, logical block 10485493, async page read
> [ 2572.300329] Buffer I/O error on dev dm-0, logical block 10485494, async page read
> [ 2572.300333] Buffer I/O error on dev dm-0, logical block 10485495, async page read
> [ 2572.300336] Buffer I/O error on dev dm-0, logical block 10485496, async page read
> [ 2572.300339] Buffer I/O error on dev dm-0, logical block 10485497, async page read
> [ 2572.301248] XFS (dm-0): log I/O error -5
> [ 2572.301257] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2572.301329] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2572.524483] XFS (dm-0): Unmounting Filesystem
> [ 2572.561468] XFS (dm-0): Mounting V5 Filesystem
> [ 2572.702822] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2573.072094] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2573.073038] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2574.091592] XFS (dm-0): log I/O error -5
> [ 2574.091610] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2574.091672] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2574.332995] XFS (dm-0): Unmounting Filesystem
> [ 2574.368980] XFS (dm-0): Mounting V5 Filesystem
> [ 2574.511448] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2574.879902] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2574.880819] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2576.886825] iomap_finish_ioend: 23 callbacks suppressed
> [ 2576.886828] dm-0: writeback error on inode 5086277, offset 23314432, sector 5450480
> [ 2576.886844] dm-0: writeback error on inode 71905642, offset 63766528, sector 47007200
> [ 2576.886849] dm-0: writeback error on inode 38936684, offset 585728, sector 26394984
> [ 2576.886853] dm-0: writeback error on inode 71399332, offset 18481152, sector 47024616
> [ 2576.886858] dm-0: writeback error on inode 105806417, offset 839680, sector 68139328
> [ 2576.886863] dm-0: writeback error on inode 72124624, offset 13430784, sector 47057936
> [ 2576.886867] dm-0: writeback error on inode 72081362, offset 110592, sector 46913208
> [ 2576.886872] dm-0: writeback error on inode 3290291, offset 20537344, sector 5361800
> [ 2576.886876] dm-0: writeback error on inode 5270796, offset 569344, sector 5267992
> [ 2576.886880] dm-0: writeback error on inode 38888467, offset 1638400, sector 26450008
> [ 2576.886995] XFS (dm-0): log I/O error -5
> [ 2576.887009] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2576.887116] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2577.128229] XFS (dm-0): Unmounting Filesystem
> [ 2577.165577] XFS (dm-0): Mounting V5 Filesystem
> [ 2577.305996] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2577.711552] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2577.712499] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2577.717746] XFS (dm-0): log I/O error -5
> [ 2577.717743] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x13ffe18 len 8 error 5
> [ 2577.717749] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x3bffa18 len 8 error 5
> [ 2577.717755] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x1c10 len 8 error 5
> [ 2577.717797] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2577.717899] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2577.720017] buffer_io_error: 38 callbacks suppressed
> [ 2577.720019] Buffer I/O error on dev dm-0, logical block 10485488, async page read
> [ 2577.720031] Buffer I/O error on dev dm-0, logical block 10485489, async page read
> [ 2577.720035] Buffer I/O error on dev dm-0, logical block 10485490, async page read
> [ 2577.720038] Buffer I/O error on dev dm-0, logical block 10485491, async page read
> [ 2577.720042] Buffer I/O error on dev dm-0, logical block 10485492, async page read
> [ 2577.720045] Buffer I/O error on dev dm-0, logical block 10485493, async page read
> [ 2577.720048] Buffer I/O error on dev dm-0, logical block 10485494, async page read
> [ 2577.720051] Buffer I/O error on dev dm-0, logical block 10485495, async page read
> [ 2577.720054] Buffer I/O error on dev dm-0, logical block 10485496, async page read
> [ 2577.720057] Buffer I/O error on dev dm-0, logical block 10485497, async page read
> [ 2577.963247] XFS (dm-0): Unmounting Filesystem
> [ 2577.980236] XFS (dm-0): Mounting V5 Filesystem
> [ 2578.029315] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2578.040983] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2578.042968] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2580.048771] XFS (dm-0): log I/O error -5
> [ 2580.048789] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2580.048851] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2580.048855] XFS (dm-0): log I/O error -5
> [ 2580.048857] XFS (dm-0): log I/O error -5
> [ 2580.048860] XFS (dm-0): log I/O error -5
> [ 2580.048862] XFS (dm-0): log I/O error -5
> [ 2580.261156] XFS (dm-0): Unmounting Filesystem
> [ 2580.317309] XFS (dm-0): Mounting V5 Filesystem
> [ 2580.524344] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2581.064911] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2581.066023] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2582.071952] iomap_finish_ioend: 11 callbacks suppressed
> [ 2582.071956] dm-0: writeback error on inode 39381706, offset 417792, sector 26804152
> [ 2582.072263] dm-0: writeback error on inode 71802791, offset 50212864, sector 46680008
> [ 2582.072837] dm-0: writeback error on inode 104258031, offset 27787264, sector 68423688
> [ 2582.072956] dm-0: writeback error on inode 71802753, offset 3035136, sector 47233248
> [ 2582.073108] dm-0: writeback error on inode 105687197, offset 262144, sector 67937776
> [ 2582.073118] dm-0: writeback error on inode 105687197, offset 19116032, sector 68428184
> [ 2582.073126] dm-0: writeback error on inode 5290517, offset 897024, sector 5292776
> [ 2582.073134] dm-0: writeback error on inode 105828662, offset 162582528, sector 47243656
> [ 2582.073141] dm-0: writeback error on inode 5275003, offset 4661248, sector 5660280
> [ 2582.073149] dm-0: writeback error on inode 104258031, offset 54648832, sector 68428352
> [ 2582.073301] XFS (dm-0): log I/O error -5
> [ 2582.073312] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2582.073422] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2582.329214] XFS (dm-0): Unmounting Filesystem
> [ 2582.417520] XFS (dm-0): Mounting V5 Filesystem
> [ 2582.553350] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2582.842465] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2582.843363] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2584.848950] XFS (dm-0): log I/O error -5
> [ 2584.848969] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2584.849030] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2584.849033] XFS (dm-0): log I/O error -5
> [ 2584.849036] XFS (dm-0): log I/O error -5
> [ 2584.849038] XFS (dm-0): log I/O error -5
> [ 2584.849040] XFS (dm-0): log I/O error -5
> [ 2584.851337] buffer_io_error: 38 callbacks suppressed
> [ 2584.851340] Buffer I/O error on dev dm-0, logical block 10485488, async page read
> [ 2584.851351] Buffer I/O error on dev dm-0, logical block 10485489, async page read
> [ 2584.851355] Buffer I/O error on dev dm-0, logical block 10485490, async page read
> [ 2584.851358] Buffer I/O error on dev dm-0, logical block 10485491, async page read
> [ 2584.851362] Buffer I/O error on dev dm-0, logical block 10485492, async page read
> [ 2584.851365] Buffer I/O error on dev dm-0, logical block 10485493, async page read
> [ 2584.851368] Buffer I/O error on dev dm-0, logical block 10485494, async page read
> [ 2584.851371] Buffer I/O error on dev dm-0, logical block 10485495, async page read
> [ 2584.851374] Buffer I/O error on dev dm-0, logical block 10485496, async page read
> [ 2584.851377] Buffer I/O error on dev dm-0, logical block 10485497, async page read
> [ 2585.125926] XFS (dm-0): Unmounting Filesystem
> [ 2585.165412] XFS (dm-0): Mounting V5 Filesystem
> [ 2585.354102] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2585.860897] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2585.861828] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2586.870178] XFS (dm-0): log I/O error -5
> [ 2586.870200] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2586.870286] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2587.095054] XFS (dm-0): Unmounting Filesystem
> [ 2587.134976] XFS (dm-0): Mounting V5 Filesystem
> [ 2587.275993] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2587.537406] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2587.538293] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2587.543301] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x1415598 len 8 error 5
> [ 2587.543302] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x18 len 8 error 5
> [ 2587.543303] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x3bffa18 len 8 error 5
> [ 2587.543299] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x40f09c0 len 32 error 5
> [ 2587.543358] XFS (dm-0): page discard on page 000000008449fa28, inode 0x64c7ea4, pos 589824.
> [ 2587.543371] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x18 len 8 error 5
> [ 2587.543429] XFS (dm-0): log I/O error -5
> [ 2587.543450] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
> [ 2587.543514] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2587.744847] XFS (dm-0): Unmounting Filesystem
> [ 2587.814615] XFS (dm-0): Mounting V5 Filesystem
> [ 2587.844735] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2587.853639] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2587.855380] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2588.861351] iomap_finish_ioend: 17 callbacks suppressed
> [ 2588.861355] dm-0: writeback error on inode 5669593, offset 1048576, sector 27217304
> [ 2588.861747] dm-0: writeback error on inode 72173797, offset 655360, sector 47498000
> [ 2588.863767] dm-0: writeback error on inode 5686085, offset 81920, sector 5856320
> [ 2588.864118] XFS (dm-0): log I/O error -5
> [ 2588.864129] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2588.864192] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2589.087112] XFS (dm-0): Unmounting Filesystem
> [ 2589.126196] XFS (dm-0): Mounting V5 Filesystem
> [ 2589.256674] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2589.542959] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2589.544017] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2590.551430] buffer_io_error: 54 callbacks suppressed
> [ 2590.551434] Buffer I/O error on dev dm-0, logical block 10485488, async page read
> [ 2590.551450] Buffer I/O error on dev dm-0, logical block 10485489, async page read
> [ 2590.551454] Buffer I/O error on dev dm-0, logical block 10485490, async page read
> [ 2590.551457] Buffer I/O error on dev dm-0, logical block 10485491, async page read
> [ 2590.551461] Buffer I/O error on dev dm-0, logical block 10485492, async page read
> [ 2590.551464] Buffer I/O error on dev dm-0, logical block 10485493, async page read
> [ 2590.551467] Buffer I/O error on dev dm-0, logical block 10485494, async page read
> [ 2590.551470] Buffer I/O error on dev dm-0, logical block 10485495, async page read
> [ 2590.551473] Buffer I/O error on dev dm-0, logical block 10485496, async page read
> [ 2590.551476] Buffer I/O error on dev dm-0, logical block 10485497, async page read
> [ 2590.776047] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x18 len 8 error 5
> [ 2590.777568] dm-0: writeback error on inode 39082395, offset 2023424, sector 26498480
> [ 2590.777572] dm-0: writeback error on inode 4054818, offset 58073088, sector 5992088
> [ 2590.777578] dm-0: writeback error on inode 4054818, offset 173793280, sector 5998104
> [ 2590.777583] dm-0: writeback error on inode 4054818, offset 178622464, sector 5999312
> [ 2590.777587] dm-0: writeback error on inode 39629723, offset 909312, sector 27047544
> [ 2590.777592] dm-0: writeback error on inode 5686139, offset 26587136, sector 6010928
> [ 2590.777596] dm-0: writeback error on inode 72492761, offset 217088, sector 47563032
> [ 2590.777692] XFS (dm-0): log I/O error -5
> [ 2590.777699] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2590.777757] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2590.803098] XFS (dm-0): Unmounting Filesystem
> [ 2590.803127] XFS (dm-0): Failing async write on buffer block 0x4b0. Retrying async write.
> [ 2590.803136] XFS (dm-0): Failing async write on buffer block 0x171cf30. Retrying async write.
> [ 2590.803140] XFS (dm-0): Failing async write on buffer block 0x171d098. Retrying async write.
> [ 2590.803143] XFS (dm-0): Failing async write on buffer block 0x13ffe28. Retrying async write.
> [ 2590.803147] XFS (dm-0): Failing async write on buffer block 0x13ffe20. Retrying async write.
> [ 2590.803151] XFS (dm-0): Failing async write on buffer block 0x50. Retrying async write.
> [ 2590.803155] XFS (dm-0): Failing async write on buffer block 0x18. Retrying async write.
> [ 2590.803158] XFS (dm-0): Failing async write on buffer block 0x28. Retrying async write.
> [ 2590.803161] XFS (dm-0): Failing async write on buffer block 0x20. Retrying async write.
> [ 2590.898854] XFS (dm-0): Mounting V5 Filesystem
> [ 2591.015030] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2591.269532] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2591.270553] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2592.278892] XFS (dm-0): log I/O error -5
> [ 2592.278909] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2592.279024] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2592.534075] XFS (dm-0): Unmounting Filesystem
> [ 2592.573250] XFS (dm-0): Mounting V5 Filesystem
> [ 2592.700931] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2593.029802] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2593.030847] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2594.233003] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x18 len 8 error 5
> [ 2594.233514] iomap_finish_ioend: 21 callbacks suppressed
> [ 2594.233516] dm-0: writeback error on inode 38125892, offset 651264, sector 27377000
> [ 2594.233526] dm-0: writeback error on inode 36841118, offset 83578880, sector 27377256
> [ 2594.233533] dm-0: writeback error on inode 3267698, offset 7778304, sector 6173120
> [ 2594.233540] dm-0: writeback error on inode 38125940, offset 524288, sector 27335560
> [ 2594.233546] dm-0: writeback error on inode 38125940, offset 1290240, sector 27335264
> [ 2594.233553] dm-0: writeback error on inode 3267673, offset 74072064, sector 6173256
> [ 2594.233677] XFS (dm-0): log I/O error -5
> [ 2594.233688] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2594.233789] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2594.250213] XFS (dm-0): Failing async write on buffer block 0x1869620. Retrying async write.
> [ 2594.250220] XFS (dm-0): Failing async write on buffer block 0x1869600. Retrying async write.
> [ 2594.259160] XFS (dm-0): Unmounting Filesystem
> [ 2594.259192] XFS (dm-0): Failing async write on buffer block 0x4b0. Retrying async write.
> [ 2594.259197] XFS (dm-0): Failing async write on buffer block 0x40a8590. Retrying async write.
> [ 2594.259201] XFS (dm-0): Failing async write on buffer block 0x50. Retrying async write.
> [ 2594.259205] XFS (dm-0): Failing async write on buffer block 0x19cb4a8. Retrying async write.
> [ 2594.259212] XFS (dm-0): Failing async write on buffer block 0x598438. Retrying async write.
> [ 2594.259215] XFS (dm-0): Failing async write on buffer block 0x2d560a0. Retrying async write.
> [ 2594.259221] XFS (dm-0): Failing async write on buffer block 0x14020b0. Retrying async write.
> [ 2594.358902] XFS (dm-0): Mounting V5 Filesystem
> [ 2594.463445] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2594.706900] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2594.707815] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2596.713662] dm-0: writeback error on inode 6257128, offset 716800, sector 6320832
> [ 2596.713668] dm-0: writeback error on inode 3509159, offset 10272768, sector 3534256
> [ 2596.713682] dm-0: writeback error on inode 105936492, offset 1105920, sector 68385744
> [ 2596.713687] dm-0: writeback error on inode 72828680, offset 1441792, sector 47710184
> [ 2596.713805] XFS (dm-0): log I/O error -5
> [ 2596.713816] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2596.713888] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2596.716177] buffer_io_error: 38 callbacks suppressed
> [ 2596.716179] Buffer I/O error on dev dm-0, logical block 10485488, async page read
> [ 2596.716188] Buffer I/O error on dev dm-0, logical block 10485489, async page read
> [ 2596.716191] Buffer I/O error on dev dm-0, logical block 10485490, async page read
> [ 2596.716195] Buffer I/O error on dev dm-0, logical block 10485491, async page read
> [ 2596.716198] Buffer I/O error on dev dm-0, logical block 10485492, async page read
> [ 2596.716201] Buffer I/O error on dev dm-0, logical block 10485493, async page read
> [ 2596.716204] Buffer I/O error on dev dm-0, logical block 10485494, async page read
> [ 2596.716207] Buffer I/O error on dev dm-0, logical block 10485495, async page read
> [ 2596.716210] Buffer I/O error on dev dm-0, logical block 10485496, async page read
> [ 2596.716213] Buffer I/O error on dev dm-0, logical block 10485497, async page read
> [ 2596.944807] XFS (dm-0): Unmounting Filesystem
> [ 2596.987515] XFS (dm-0): Mounting V5 Filesystem
> [ 2597.146921] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2597.506121] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2597.507036] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2598.694283] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x20 len 8 error 5
> [ 2598.695795] XFS (dm-0): log I/O error -5
> [ 2598.695803] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2598.695860] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2598.695864] XFS (dm-0): log I/O error -5
> [ 2598.695866] XFS (dm-0): log I/O error -5
> [ 2598.695868] XFS (dm-0): log I/O error -5
> [ 2598.695870] XFS (dm-0): log I/O error -5
> [ 2598.721263] XFS (dm-0): Unmounting Filesystem
> [ 2598.721294] XFS (dm-0): Failing async write on buffer block 0x4b0. Retrying async write.
> [ 2598.721301] XFS (dm-0): Failing async write on buffer block 0x14082c0. Retrying async write.
> [ 2598.721305] XFS (dm-0): Failing async write on buffer block 0x1402cf8. Retrying async write.
> [ 2598.721308] XFS (dm-0): Failing async write on buffer block 0x3c01d80. Retrying async write.
> [ 2598.721311] XFS (dm-0): Failing async write on buffer block 0x3c032d0. Retrying async write.
> [ 2598.721314] XFS (dm-0): Failing async write on buffer block 0x1869810. Retrying async write.
> [ 2598.721317] XFS (dm-0): Failing async write on buffer block 0x3c03808. Retrying async write.
> [ 2598.721320] XFS (dm-0): Failing async write on buffer block 0x4092918. Retrying async write.
> [ 2598.721324] XFS (dm-0): Failing async write on buffer block 0x680. Retrying async write.
> [ 2598.819903] XFS (dm-0): Mounting V5 Filesystem
> [ 2598.926017] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2599.185692] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2599.186561] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2599.232676] XFS (dm-0): log I/O error -5
> [ 2599.232679] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
> [ 2599.232679] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x358b80 len 32 error 5
> [ 2599.232693] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2599.232755] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2599.463847] XFS (dm-0): Unmounting Filesystem
> [ 2599.480839] XFS (dm-0): Mounting V5 Filesystem
> [ 2599.512736] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2599.520518] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2599.522021] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2601.544049] iomap_finish_ioend: 13 callbacks suppressed
> [ 2601.544052] dm-0: writeback error on inode 6344500, offset 77398016, sector 6522176
> [ 2601.693208] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x20 len 8 error 5
> [ 2601.695457] dm-0: writeback error on inode 73002350, offset 69632, sector 47941664
> [ 2601.695637] dm-0: writeback error on inode 73002350, offset 180224, sector 47943056
> [ 2601.695645] dm-0: writeback error on inode 6211174, offset 1847296, sector 6452984
> [ 2601.695649] dm-0: writeback error on inode 72952535, offset 57147392, sector 47919256
> [ 2601.695653] dm-0: writeback error on inode 72967749, offset 46862336, sector 47943344
> [ 2601.695657] dm-0: writeback error on inode 106150432, offset 524288, sector 68803664
> [ 2601.695661] dm-0: writeback error on inode 106150432, offset 17850368, sector 69367752
> [ 2601.695665] dm-0: writeback error on inode 106502774, offset 84537344, sector 69368088
> [ 2601.695670] dm-0: writeback error on inode 6447171, offset 106463232, sector 6626344
> [ 2601.695792] XFS (dm-0): log I/O error -5
> [ 2601.695799] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2601.695857] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2601.725357] XFS (dm-0): Unmounting Filesystem
> [ 2601.725391] XFS (dm-0): Failing async write on buffer block 0x2d96240. Retrying async write.
> [ 2601.725396] XFS (dm-0): Failing async write on buffer block 0x3fca268. Retrying async write.
> [ 2601.725401] XFS (dm-0): Failing async write on buffer block 0x41f6820. Retrying async write.
> [ 2601.725404] XFS (dm-0): Failing async write on buffer block 0x41f6800. Retrying async write.
> [ 2601.725407] XFS (dm-0): Failing async write on buffer block 0x623aa0. Retrying async write.
> [ 2601.725410] XFS (dm-0): Failing async write on buffer block 0x623a80. Retrying async write.
> [ 2601.725413] XFS (dm-0): Failing async write on buffer block 0x280a380. Retrying async write.
> [ 2601.725416] XFS (dm-0): Failing async write on buffer block 0x2d96280. Retrying async write.
> [ 2601.725419] XFS (dm-0): Failing async write on buffer block 0x196c938. Retrying async write.
> [ 2601.880399] XFS (dm-0): Mounting V5 Filesystem
> [ 2602.048576] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2602.486670] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2602.487581] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2603.494852] buffer_io_error: 54 callbacks suppressed
> [ 2603.494855] Buffer I/O error on dev dm-0, logical block 10485488, async page read
> [ 2603.494873] Buffer I/O error on dev dm-0, logical block 10485489, async page read
> [ 2603.494877] Buffer I/O error on dev dm-0, logical block 10485490, async page read
> [ 2603.494880] Buffer I/O error on dev dm-0, logical block 10485491, async page read
> [ 2603.494883] Buffer I/O error on dev dm-0, logical block 10485492, async page read
> [ 2603.494887] Buffer I/O error on dev dm-0, logical block 10485493, async page read
> [ 2603.494889] Buffer I/O error on dev dm-0, logical block 10485494, async page read
> [ 2603.494893] Buffer I/O error on dev dm-0, logical block 10485495, async page read
> [ 2603.494896] Buffer I/O error on dev dm-0, logical block 10485496, async page read
> [ 2603.494899] Buffer I/O error on dev dm-0, logical block 10485497, async page read
> [ 2603.721652] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x18 len 8 error 5
> [ 2603.723184] XFS (dm-0): log I/O error -5
> [ 2603.723194] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2603.723276] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2603.750782] XFS (dm-0): Unmounting Filesystem
> [ 2603.750814] XFS (dm-0): Failing async write on buffer block 0x4b0. Retrying async write.
> [ 2603.750820] XFS (dm-0): Failing async write on buffer block 0x1920760. Retrying async write.
> [ 2603.750823] XFS (dm-0): Failing async write on buffer block 0x3c02ea0. Retrying async write.
> [ 2603.750827] XFS (dm-0): Failing async write on buffer block 0x14020b0. Retrying async write.
> [ 2603.750840] XFS (dm-0): Failing async write on buffer block 0x41bb2a0. Retrying async write.
> [ 2603.750844] XFS (dm-0): Failing async write on buffer block 0x14155a0. Retrying async write.
> [ 2603.750847] XFS (dm-0): Failing async write on buffer block 0x13ffe20. Retrying async write.
> [ 2603.750850] XFS (dm-0): Failing async write on buffer block 0x3c01d80. Retrying async write.
> [ 2603.750859] XFS (dm-0): Failing async write on buffer block 0x42058e0. Retrying async write.
> [ 2603.911355] XFS (dm-0): Mounting V5 Filesystem
> [ 2604.039283] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2604.316945] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2604.317844] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2604.322964] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c07f78 len 8 error 5
> [ 2604.322964] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
> [ 2604.323032] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
> [ 2604.323042] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c07f78 len 8 error 5
> [ 2604.323088] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
> [ 2604.323104] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c07f78 len 8 error 5
> [ 2604.323138] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
> [ 2604.323169] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c07f78 len 8 error 5
> [ 2604.323188] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
> [ 2604.323247] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
> [ 2604.324979] XFS (dm-0): log I/O error -5
> [ 2604.324999] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2604.325082] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2604.554164] XFS (dm-0): Unmounting Filesystem
> [ 2604.571381] XFS (dm-0): Mounting V5 Filesystem
> [ 2604.613536] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2604.620050] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2604.621596] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2604.627298] XFS (dm-0): log I/O error -5
> [ 2604.627307] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x1869600 len 32 error 5
> [ 2604.627334] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
> [ 2604.627438] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2604.813933] XFS (dm-0): Unmounting Filesystem
> [ 2604.830589] XFS (dm-0): Mounting V5 Filesystem
> [ 2604.875920] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2604.884974] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2604.886426] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2604.932799] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x174bee0 len 32 error 5
> [ 2604.932802] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x49a920 len 32 error 5
> [ 2604.932803] XFS (dm-0): metadata I/O error in "xfs_btree_read_buf_block.constprop.29+0xd0/0x110 [xfs]" at daddr 0x3bffa30 len 8 error 5
> [ 2604.932900] XFS (dm-0): log I/O error -5
> [ 2604.932915] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
> [ 2604.932974] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2605.164824] XFS (dm-0): Unmounting Filesystem
> [ 2605.181637] XFS (dm-0): Mounting V5 Filesystem
> [ 2605.225897] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2605.232410] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2605.234452] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2607.239943] iomap_finish_ioend: 40 callbacks suppressed
> [ 2607.239946] dm-0: writeback error on inode 40220526, offset 31604736, sector 27637824
> [ 2607.239960] dm-0: writeback error on inode 40021920, offset 29814784, sector 27875352
> [ 2607.239965] dm-0: writeback error on inode 104667118, offset 105447424, sector 69504936
> [ 2607.239969] dm-0: writeback error on inode 104667118, offset 171417600, sector 69614576
> [ 2607.239974] dm-0: writeback error on inode 6344484, offset 25358336, sector 6857952
> [ 2607.240036] XFS (dm-0): log I/O error -5
> [ 2607.240049] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2607.240137] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2607.515399] XFS (dm-0): Unmounting Filesystem
> [ 2607.560484] XFS (dm-0): Mounting V5 Filesystem
> [ 2607.709693] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2608.076848] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2608.077772] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2608.082680] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x1404e80 len 8 error 5
> [ 2608.082679] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c060a8 len 8 error 5
> [ 2608.082679] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x49a920 len 32 error 5
> [ 2608.082680] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x3c07f78 len 8 error 5
> [ 2608.082758] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x1404e80 len 8 error 5
> [ 2608.082810] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x1404e80 len 8 error 5
> [ 2608.082862] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x1404e80 len 8 error 5
> [ 2608.082882] XFS (dm-0): metadata I/O error in "xfs_alloc_read_agfl+0x88/0xf0 [xfs]" at daddr 0x18 len 8 error 5
> [ 2608.082962] XFS (dm-0): log I/O error -5
> [ 2608.082977] XFS (dm-0): Metadata I/O Error (0x1) detected at xfs_trans_read_buf_map+0x31c/0x368 [xfs] (fs/xfs/xfs_trans_buf.c:296). Shutting down filesystem.
> [ 2608.083049] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2608.083059] XFS (dm-0): xfs_imap_lookup: xfs_ialloc_read_agi() returned error -5, agno 0
> [ 2608.283930] XFS (dm-0): Unmounting Filesystem
> [ 2608.301209] XFS (dm-0): Mounting V5 Filesystem
> [ 2608.359920] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2608.370818] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2608.372848] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2608.377740] XFS (dm-0): log I/O error -5
> [ 2608.377743] XFS (dm-0): metadata I/O error in "xfs_da_read_buf+0x12c/0x1a0 [xfs]" at daddr 0x1404288 len 8 error 5
> [ 2608.377745] XFS (dm-0): metadata I/O error in "xfs_imap_to_bp+0x64/0x98 [xfs]" at daddr 0x3fc5f60 len 32 error 5
> [ 2608.377761] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2608.377823] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2608.603843] XFS (dm-0): Unmounting Filesystem
> [ 2608.620761] XFS (dm-0): Mounting V5 Filesystem
> [ 2608.682744] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2608.695059] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2608.696923] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2609.704216] buffer_io_error: 102 callbacks suppressed
> [ 2609.704219] Buffer I/O error on dev dm-0, logical block 10485488, async page read
> [ 2609.704235] Buffer I/O error on dev dm-0, logical block 10485489, async page read
> [ 2609.704239] Buffer I/O error on dev dm-0, logical block 10485490, async page read
> [ 2609.704242] Buffer I/O error on dev dm-0, logical block 10485491, async page read
> [ 2609.704246] Buffer I/O error on dev dm-0, logical block 10485492, async page read
> [ 2609.704249] Buffer I/O error on dev dm-0, logical block 10485493, async page read
> [ 2609.704252] Buffer I/O error on dev dm-0, logical block 10485494, async page read
> [ 2609.704255] Buffer I/O error on dev dm-0, logical block 10485495, async page read
> [ 2609.704258] Buffer I/O error on dev dm-0, logical block 10485496, async page read
> [ 2609.704261] Buffer I/O error on dev dm-0, logical block 10485497, async page read
> [ 2609.704971] dm-0: writeback error on inode 73088154, offset 87687168, sector 48133584
> [ 2609.868626] XFS (dm-0): metadata I/O error in "xfs_buf_ioend+0x310/0x6c0 [xfs]" at daddr 0x48 len 8 error 5
> [ 2609.869085] dm-0: writeback error on inode 40295105, offset 1048576, sector 27860880
> [ 2609.869194] dm-0: writeback error on inode 106528698, offset 983040, sector 68387864
> [ 2609.869443] dm-0: writeback error on inode 40295105, offset 479232, sector 27704168
> [ 2609.869452] dm-0: writeback error on inode 39669888, offset 46268416, sector 28022120
> [ 2609.869639] XFS (dm-0): log I/O error -5
> [ 2609.869650] XFS (dm-0): Log I/O Error (0x2) detected at xlog_ioend_work+0xb4/0x100 [xfs] (fs/xfs/xfs_log.c:1377). Shutting down filesystem.
> [ 2609.869750] XFS (dm-0): Please unmount the filesystem and rectify the problem(s)
> [ 2609.897807] XFS (dm-0): Unmounting Filesystem
> [ 2609.897839] XFS (dm-0): Failing async write on buffer block 0x4b0. Retrying async write.
> [ 2609.897851] XFS (dm-0): Failing async write on buffer block 0x1404c08. Retrying async write.
> [ 2609.897855] XFS (dm-0): Failing async write on buffer block 0x17ced58. Retrying async write.
> [ 2609.897860] XFS (dm-0): Failing async write on buffer block 0x1402cf8. Retrying async write.
> [ 2609.897863] XFS (dm-0): Failing async write on buffer block 0x5937e8. Retrying async write.
> [ 2609.897866] XFS (dm-0): Failing async write on buffer block 0x680. Retrying async write.
> [ 2609.897882] XFS (dm-0): Failing async write on buffer block 0x657400. Retrying async write.
> [ 2609.897886] XFS (dm-0): Failing async write on buffer block 0x689620. Retrying async write.
> [ 2609.897888] XFS (dm-0): Failing async write on buffer block 0x689600. Retrying async write.
> [ 2610.051099] XFS (dm-0): Mounting V5 Filesystem
> [ 2610.163869] XFS (dm-0): Starting recovery (logdev: internal)
> [ 2610.426613] XFS (dm-0): Ending recovery (logdev: internal)
> [ 2610.427541] xfs filesystem being mounted at /mnt/scratch supports timestamps until 2038 (0x7fffffff)
> [ 2610.435505] XFS (dm-0): Unmounting Filesystem
> [ 2610.546863] XFS (sdb1): Unmounting Filesystem
> [ 2611.152679] run fstests generic/476 at 2022-02-15 15:43:38
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [next-20220215] WARNING at fs/iomap/buffered-io.c:75 with xfstests
2022-02-16 17:35 ` Darrick J. Wong
@ 2022-02-16 17:46 ` Matthew Wilcox
0 siblings, 0 replies; 6+ messages in thread
From: Matthew Wilcox @ 2022-02-16 17:46 UTC (permalink / raw)
To: Darrick J. Wong
Cc: linux-xfs, riteshh, linux-next, Sachin Sant, linux-fsdevel, linuxppc-dev
On Wed, Feb 16, 2022 at 09:35:04AM -0800, Darrick J. Wong wrote:
> On Wed, Feb 16, 2022 at 12:55:02PM +0530, Sachin Sant wrote:
> > [ 2547.662818] ------------[ cut here ]------------
> > [ 2547.662832] WARNING: CPU: 24 PID: 2463718 at fs/iomap/buffered-io.c:75 iomap_page_release+0x1b0/0x1e0
>
> ...and I think this is complaining about this debugging test in
> iomap_page_release:
>
> WARN_ON_ONCE(bitmap_full(iop->uptodate, nr_blocks) !=
> folio_test_uptodate(folio));
>
> which checks that we're not releasing or invalidating a page that's
> partially uptodate on a blocksize < pagesize filesystem (or so I gather
> from "POWER10 LPAR" (64k pages?) and "XFS" (4k blocks?))...
This happens _all_ _the_ _time_ in my testing. If your block size is
less than page size, you can expect it.
What it's supposed to be testing is that we remembered to set the
uptodate flag once all blocks in this page are uptodate. What's
tripping the check is iomap_writepage_map()'s stupid clearing of the
uptodate flag on the folio:
if (unlikely(error)) {
...
if (!count) {
folio_clear_uptodate(folio);
folio_unlock(folio);
goto done;
What particularly annoys me about this is that the folio _was_ uptodate,
and it was dirty, so it has newer data in it than is on disk, but we're
going to re-read the folio from disk and throw away that user data.
> Given that in this case we've already cleared SB_ACTIVE from the
> superblock s_flags, I wonder if we could amend that code to read:
>
> if (inode->i_sb->s_flags & SB_ACTIVE)
> WARN_ON_ONCE(bitmap_full(iop->uptodate, nr_blocks) !=
> folio_test_uptodate(folio));
>
> Since we don't really care about memory pages that aren't fully up to
> date if we're in the midst of freeing all the incore filesystem state.
>
> Thoughts?
Seems like papering over a bad design decision to me.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-02-16 21:13 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-02-16 7:25 [next-20220215] WARNING at fs/iomap/buffered-io.c:75 with xfstests Sachin Sant
2022-02-16 7:39 ` Stephen Rothwell
2022-02-16 9:47 ` Sachin Sant
2022-02-16 11:39 ` Stephen Rothwell
2022-02-16 17:35 ` Darrick J. Wong
2022-02-16 17:46 ` Matthew Wilcox
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).