* linux-next: xfs metadata corruption since 30 March
@ 2020-03-31 21:57 Qian Cai
2020-03-31 22:13 ` Dave Chinner
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Qian Cai @ 2020-03-31 21:57 UTC (permalink / raw)
To: Darrick J. Wong, Christoph Hellwig; +Cc: linux-xfs, LKML
Ever since two days ago, linux-next starts to trigger xfs metadata corruption
during compilation workloads on both powerpc and arm64,
I suspect it could be one of those commits,
https://lore.kernel.org/linux-xfs/20200328182533.GM29339@magnolia/
Especially, those commits that would mark corruption more aggressively?
[8d57c21600a5] xfs: add a function to deal with corrupt buffers post-verifiers
[e83cf875d67a] xfs: xfs_buf_corruption_error should take __this_address
[ce99494c9699] xfs: fix buffer corruption reporting when xfs_dir3_free_header_check fails
[1cb5deb5bc09] xfs: don't ever return a stale pointer from __xfs_dir3_free_read
[6fb5aac73310] xfs: check owner of dir3 free blocks
[a10c21ed5d52] xfs: check owner of dir3 data blocks
[1b2c1a63b678] xfs: check owner of dir3 blocks
[2e107cf869ee] xfs: mark dir corrupt when lookup-by-hash fails
[806d3909a57e] xfs: mark extended attr corrupt when lookup-by-hash fails
[29331.182313][ T665] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x2b8/0x350 [xfs], xfs_inode block 0xa9b97900 xfs_inode_buf_verify
xfs_inode_buf_verify at fs/xfs/libxfs/xfs_inode_buf.c:101
[29331.182373][ T665] XFS (dm-2): Unmount and run xfs_repair
[29331.182386][ T665] XFS (dm-2): First 128 bytes of corrupted metadata buffer:
[29331.182402][ T665] 00000000: 2f 2a 20 53 50 44 58 2d 4c 69 63 65 6e 73 65 2d /* SPDX-License-
[29331.182426][ T665] 00000010: 49 64 65 6e 74 69 66 69 65 72 3a 20 47 50 4c 2d Identifier: GPL-
[29331.182442][ T665] 00000020: 32 2e 30 2d 6f 72 2d 6c 61 74 65 72 20 2a 2f 0a 2.0-or-later */.
[29331.182467][ T665] 00000030: 2f 2a 0a 20 2a 20 44 65 66 69 6e 69 74 69 6f 6e /*. * Definition
[29331.182492][ T665] 00000040: 73 20 61 6e 64 20 70 6c 61 74 66 6f 72 6d 20 64 s and platform d
[29331.182517][ T665] 00000050: 61 74 61 20 66 6f 72 20 41 6e 61 6c 6f 67 20 44 ata for Analog D
[29331.182541][ T665] 00000060: 65 76 69 63 65 73 0a 20 2a 20 41 44 50 35 35 32 evices. * ADP552
[29331.182566][ T665] 00000070: 30 2f 41 44 50 35 35 30 31 20 4d 46 44 20 50 4d 0/ADP5501 MFD PM
[29331.182700][ T7490] XFS (dm-2): metadata I/O error in "xfs_imap_to_bp+0x88/0x130 [xfs]" at daddr 0xa9b97900 len 32 error 117
xfs_trans_read_buf at fs/xfs/xfs_trans.h:209
(inlined by) xfs_imap_to_bp at fs/xfs/libxfs/xfs_inode_buf.c:171
[29331.182812][ T7490] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.
[29331.345347][ T7490] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3754 of file fs/xfs/xfs_inode.c. Return address = 0000000058be213e
[29331.345378][ T7490] XFS (dm-2): Corruption of in-memory data detected. Shutting down filesystem
[29331.345402][ T7490] XFS (dm-2): Please unmount the filesystem and rectify the problem(s)
[29331.346474][ T498] dm-2: writeback error on inode 934606, offset 0, sector 961072
[29331.346502][ T498] dm-2: writeback error on inode 934607, offset 0, sector 961080
[29331.346624][ T498] dm-2: writeback error on inode 934608, offset 0, sector 961088
[29331.346683][ T498] dm-2: writeback error on inode 1074331758, offset 0, sector 948449704
[29331.347306][ T498] dm-2: writeback error on inode 1074331762, offset 0, sector 948558816
[29331.349165][ T498] dm-2: writeback error on inode 1074331759, offset 0, sector 948560984
[29331.349227][ T498] dm-2: writeback error on inode 1074331760, offset 0, sector 948562944
[29331.349303][ T498] dm-2: writeback error on inode 1074331761, offset 0, sector 948568000
[ 7762.204313][T124538] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x2b8/0x350 [xfs], xfs_inode block 0x712398e0 xfs_inode_buf_verify
[ 7762.204599][T124538] XFS (dm-2): Unmount and run xfs_repair
[ 7762.204625][T124538] XFS (dm-2): First 128 bytes of corrupted metadata buffer:
[ 7762.204654][T124538] 00000000: 77 65 72 70 63 2f 69 6e 63 6c 75 64 65 2f 67 65 werpc/include/ge
[ 7762.204672][T124538] 00000010: 6e 65 72 61 74 65 64 2f 75 61 70 69 2f 61 73 6d nerated/uapi/asm
[ 7762.204699][T124538] 00000020: 2f 72 65 73 6f 75 72 63 65 2e 68 20 5c 0a 20 69 /resource.h \. i
[ 7762.204727][T124538] 00000030: 6e 63 6c 75 64 65 2f 61 73 6d 2d 67 65 6e 65 72 nclude/asm-gener
[ 7762.204745][T124538] 00000040: 69 63 2f 72 65 73 6f 75 72 63 65 2e 68 20 69 6e ic/resource.h in
[ 7762.204783][T124538] 00000050: 63 6c 75 64 65 2f 75 61 70 69 2f 61 73 6d 2d 67 clude/uapi/asm-g
[ 7762.204820][T124538] 00000060: 65 6e 65 72 69 63 2f 72 65 73 6f 75 72 63 65 2e eneric/resource.
[ 7762.204858][T124538] 00000070: 68 20 5c 0a 20 69 6e 63 6c 75 64 65 2f 6c 69 6e h \. include/lin
[ 7762.205068][ T7510] XFS (dm-2): metadata I/O error in "xfs_imap_to_bp+0x88/0x130 [xfs]" at daddr 0x712398e0 len 32 error 117
[ 7762.205466][ T7510] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.
[ 7762.219267][ T7510] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3754 of file fs/xfs/xfs_inode.c. Return address = 000000006bce0de3
[ 7762.219291][ T7510] XFS (dm-2): Corruption of in-memory data detected. Shutting down filesystem
[ 7762.219306][ T7510] XFS (dm-2): Please unmount the filesystem and rectify the problem(s)
[ 1032.162278][ T1515] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x244/0x2bc [xfs], xfs_inode block 0xa2b75dc0 xfs_inode_buf_verify
[ 1032.176156][ T1515] XFS (dm-2): Unmount and run xfs_repair
[ 1032.181835][ T1515] XFS (dm-2): First 128 bytes of corrupted metadata buffer:
[ 1032.189140][ T1515] 00000000: 6e 29 20 22 22 20 76 61 6c 75 65 20 22 22 20 73 n) "" value "" s
[ 1032.197988][ T1515] 00000010: 75 62 73 74 72 28 6c 69 6e 65 2c 20 6c 65 6e 20 ubstr(line, len
[ 1032.206723][ T1515] 00000020: 2b 20 6b 65 79 6c 65 6e 20 2b 20 33 29 0a 20 20 + keylen + 3).
[ 1032.215675][ T1515] 00000030: 20 20 20 20 6c 65 6e 20 2b 3d 20 6c 65 6e 67 74 len += lengt
[ 1032.224537][ T1515] 00000040: 68 28 76 61 6c 75 65 29 20 2b 20 6c 65 6e 67 74 h(value) + lengt
[ 1032.233388][ T1515] 00000050: 68 28 66 69 65 6c 64 5b 2b 2b 69 5d 29 0a 20 20 h(field[++i]).
[ 1032.242234][ T1515] 00000060: 20 20 20 20 73 75 62 73 74 65 64 20 3d 20 31 0a substed = 1.
[ 1032.251077][ T1515] 00000070: 20 20 20 20 7d 20 65 6c 73 65 0a 20 20 20 20 20 } else.
[ 1032.260792][ T4119] XFS (dm-2): metadata I/O error in "xfs_imap_to_bp+0xd8/0x18c [xfs]" at daddr 0xa2b75dc0 len 32 error 117
[ 1032.273096][ T4119] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117.
[ 1032.283283][ T4119] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3754 of file fs/xfs/xfs_inode.c. Return address = 00000000d99a2721
[ 1032.296214][ T4119] XFS (dm-2): Corruption of in-memory data detected. Shutting down filesystem
[ 1032.305158][ T4119] XFS (dm-2): Please unmount the filesystem and rectify the problem(s)
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: linux-next: xfs metadata corruption since 30 March 2020-03-31 21:57 linux-next: xfs metadata corruption since 30 March Qian Cai @ 2020-03-31 22:13 ` Dave Chinner 2020-04-01 2:13 ` Qian Cai 2020-04-01 4:14 ` Chandan Rajendra 2020-04-01 12:34 ` Brian Foster 2 siblings, 1 reply; 11+ messages in thread From: Dave Chinner @ 2020-03-31 22:13 UTC (permalink / raw) To: Qian Cai; +Cc: Darrick J. Wong, Christoph Hellwig, linux-xfs, LKML On Tue, Mar 31, 2020 at 05:57:24PM -0400, Qian Cai wrote: > Ever since two days ago, linux-next starts to trigger xfs metadata corruption > during compilation workloads on both powerpc and arm64, Is this on an existing filesystem, or a new filesystem? > I suspect it could be one of those commits, > > https://lore.kernel.org/linux-xfs/20200328182533.GM29339@magnolia/ > > Especially, those commits that would mark corruption more aggressively? > > [8d57c21600a5] xfs: add a function to deal with corrupt buffers post-verifiers > [e83cf875d67a] xfs: xfs_buf_corruption_error should take __this_address > [ce99494c9699] xfs: fix buffer corruption reporting when xfs_dir3_free_header_check fails > [1cb5deb5bc09] xfs: don't ever return a stale pointer from __xfs_dir3_free_read > [6fb5aac73310] xfs: check owner of dir3 free blocks > [a10c21ed5d52] xfs: check owner of dir3 data blocks > [1b2c1a63b678] xfs: check owner of dir3 blocks > [2e107cf869ee] xfs: mark dir corrupt when lookup-by-hash fails > [806d3909a57e] xfs: mark extended attr corrupt when lookup-by-hash fails Doubt it - they only add extra detection code and these: > [29331.182313][ T665] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x2b8/0x350 [xfs], xfs_inode block 0xa9b97900 xfs_inode_buf_verify > xfs_inode_buf_verify at fs/xfs/libxfs/xfs_inode_buf.c:101 > [29331.182373][ T665] XFS (dm-2): Unmount and run xfs_repair > [29331.182386][ T665] XFS (dm-2): First 128 bytes of corrupted metadata buffer: > [29331.182402][ T665] 00000000: 2f 2a 20 53 50 44 58 2d 4c 69 63 65 6e 73 65 2d /* SPDX-License- > [29331.182426][ T665] 00000010: 49 64 65 6e 74 69 66 69 65 72 3a 20 47 50 4c 2d Identifier: GPL- Would get caught by the existing verifiers as they aren't valid metadata at all. Basically, you are getting file data where there should be inode metadata. First thing to do is fix the existing corruptions with xfs_repair - please post the entire output so we can see what was corruption and what it fixed. Then if the problem is still reproducable, I suspect you are going to have to bisect it. i.e. run test, get corruption, mark bisect bad, run xfs_repair or mkfs to fix mess, install new kernel, run test again.... Cheers, Dave. -- Dave Chinner david@fromorbit.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: linux-next: xfs metadata corruption since 30 March 2020-03-31 22:13 ` Dave Chinner @ 2020-04-01 2:13 ` Qian Cai 0 siblings, 0 replies; 11+ messages in thread From: Qian Cai @ 2020-04-01 2:13 UTC (permalink / raw) To: Dave Chinner; +Cc: Darrick J. Wong, Christoph Hellwig, linux-xfs, LKML > On Mar 31, 2020, at 6:13 PM, Dave Chinner <david@fromorbit.com> wrote: > > On Tue, Mar 31, 2020 at 05:57:24PM -0400, Qian Cai wrote: >> Ever since two days ago, linux-next starts to trigger xfs metadata corruption >> during compilation workloads on both powerpc and arm64, > > Is this on an existing filesystem, or a new filesystem? New. > >> I suspect it could be one of those commits, >> >> https://lore.kernel.org/linux-xfs/20200328182533.GM29339@magnolia/ >> >> Especially, those commits that would mark corruption more aggressively? >> >> [8d57c21600a5] xfs: add a function to deal with corrupt buffers post-verifiers >> [e83cf875d67a] xfs: xfs_buf_corruption_error should take __this_address >> [ce99494c9699] xfs: fix buffer corruption reporting when xfs_dir3_free_header_check fails >> [1cb5deb5bc09] xfs: don't ever return a stale pointer from __xfs_dir3_free_read >> [6fb5aac73310] xfs: check owner of dir3 free blocks >> [a10c21ed5d52] xfs: check owner of dir3 data blocks >> [1b2c1a63b678] xfs: check owner of dir3 blocks >> [2e107cf869ee] xfs: mark dir corrupt when lookup-by-hash fails >> [806d3909a57e] xfs: mark extended attr corrupt when lookup-by-hash fails > > Doubt it - they only add extra detection code and these: > >> [29331.182313][ T665] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x2b8/0x350 [xfs], xfs_inode block 0xa9b97900 xfs_inode_buf_verify >> xfs_inode_buf_verify at fs/xfs/libxfs/xfs_inode_buf.c:101 >> [29331.182373][ T665] XFS (dm-2): Unmount and run xfs_repair >> [29331.182386][ T665] XFS (dm-2): First 128 bytes of corrupted metadata buffer: >> [29331.182402][ T665] 00000000: 2f 2a 20 53 50 44 58 2d 4c 69 63 65 6e 73 65 2d /* SPDX-License- >> [29331.182426][ T665] 00000010: 49 64 65 6e 74 69 66 69 65 72 3a 20 47 50 4c 2d Identifier: GPL- > > Would get caught by the existing verifiers as they aren't valid > metadata at all. > > Basically, you are getting file data where there should be inode > metadata. First thing to do is fix the existing corruptions with > xfs_repair - please post the entire output so we can see what was > corruption and what it fixed. # xfs_repair -v /dev/mapper/rhel_hpe--apollo--cn99xx--11-home Phase 1 - find and verify superblock... - block cache size set to 4355512 entries Phase 2 - using internal log - zero log... zero_log: head block 793608 tail block 786824 ERROR: The filesystem has valuable metadata changes in a log which needs to be replayed. Mount the filesystem to replay the log, and unmount it before re-running xfs_repair. If you are unable to mount the filesystem, then use the -L option to destroy the log and attempt a repair. Note that destroying the log may cause corruption -- please attempt a mount of the filesystem before doing this. # mount /dev/mapper/rhel_hpe--apollo--cn99xx--11-home /home/ # umount /home/ # xfs_repair -v /dev/mapper/rhel_hpe--apollo--cn99xx--11-home Phase 1 - find and verify superblock... - block cache size set to 4355512 entries Phase 2 - using internal log - zero log... zero_log: head block 793624 tail block 793624 - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 1 - agno = 3 Phase 5 - rebuild AG headers and trees... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... XFS_REPAIR Summary Tue Mar 31 22:10:54 2020 Phase Start End Duration Phase 1: 03/31 22:10:45 03/31 22:10:45 Phase 2: 03/31 22:10:45 03/31 22:10:45 Phase 3: 03/31 22:10:45 03/31 22:10:46 1 second Phase 4: 03/31 22:10:46 03/31 22:10:53 7 seconds Phase 5: 03/31 22:10:53 03/31 22:10:53 Phase 6: 03/31 22:10:53 03/31 22:10:53 Phase 7: 03/31 22:10:53 03/31 22:10:53 Total run time: 8 seconds done > > Then if the problem is still reproducable, I suspect you are going > to have to bisect it. i.e. run test, get corruption, mark bisect > bad, run xfs_repair or mkfs to fix mess, install new kernel, run > test again.... > > Cheers, > > Dave. > -- > Dave Chinner > david@fromorbit.com ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: linux-next: xfs metadata corruption since 30 March 2020-03-31 21:57 linux-next: xfs metadata corruption since 30 March Qian Cai 2020-03-31 22:13 ` Dave Chinner @ 2020-04-01 4:14 ` Chandan Rajendra 2020-04-01 4:15 ` Qian Cai 2020-04-01 12:34 ` Brian Foster 2 siblings, 1 reply; 11+ messages in thread From: Chandan Rajendra @ 2020-04-01 4:14 UTC (permalink / raw) To: Qian Cai; +Cc: Darrick J. Wong, Christoph Hellwig, linux-xfs, LKML On Wednesday, April 1, 2020 3:27 AM Qian Cai wrote: > Ever since two days ago, linux-next starts to trigger xfs metadata corruption > during compilation workloads on both powerpc and arm64, Can you please provide the filesystem geometry information? You can get that by executing "xfs_info <mount-point>" command. -- chandan ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: linux-next: xfs metadata corruption since 30 March 2020-04-01 4:14 ` Chandan Rajendra @ 2020-04-01 4:15 ` Qian Cai 2020-04-01 4:45 ` Darrick J. Wong 0 siblings, 1 reply; 11+ messages in thread From: Qian Cai @ 2020-04-01 4:15 UTC (permalink / raw) To: Chandan Rajendra; +Cc: Darrick J. Wong, Christoph Hellwig, linux-xfs, LKML > On Apr 1, 2020, at 12:14 AM, Chandan Rajendra <chandan@linux.ibm.com> wrote: > > On Wednesday, April 1, 2020 3:27 AM Qian Cai wrote: >> Ever since two days ago, linux-next starts to trigger xfs metadata corruption >> during compilation workloads on both powerpc and arm64, > > Can you please provide the filesystem geometry information? > You can get that by executing "xfs_info <mount-point>" command. > == arm64 == # xfs_info /home/ meta-data=/dev/mapper/rhel_hpe--apollo--cn99xx--11-home isize=512 agcount=4, agsize=113568256 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 data = bsize=4096 blocks=454273024, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=221813, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 == powerpc == # xfs_info /home/ meta-data=/dev/mapper/rhel_ibm--p9wr--01-home isize=512 agcount=4, agsize=118489856 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 data = bsize=4096 blocks=473959424, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=231425, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 == x86 (not yet reproduced) == meta-data=/dev/mapper/rhel_hpe--dl380gen9--01-home isize=512 agcount=16, agsize=3283776 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 data = bsize=4096 blocks=52540416, imaxpct=25 = sunit=64 swidth=64 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=25664, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: linux-next: xfs metadata corruption since 30 March 2020-04-01 4:15 ` Qian Cai @ 2020-04-01 4:45 ` Darrick J. Wong 2020-04-01 6:10 ` Chandan Rajendra 2020-04-01 13:54 ` Qian Cai 0 siblings, 2 replies; 11+ messages in thread From: Darrick J. Wong @ 2020-04-01 4:45 UTC (permalink / raw) To: Qian Cai; +Cc: Chandan Rajendra, Christoph Hellwig, linux-xfs, LKML On Wed, Apr 01, 2020 at 12:15:32AM -0400, Qian Cai wrote: > > > > On Apr 1, 2020, at 12:14 AM, Chandan Rajendra <chandan@linux.ibm.com> wrote: > > > > On Wednesday, April 1, 2020 3:27 AM Qian Cai wrote: > >> Ever since two days ago, linux-next starts to trigger xfs metadata corruption > >> during compilation workloads on both powerpc and arm64, > > > > Can you please provide the filesystem geometry information? > > You can get that by executing "xfs_info <mount-point>" command. > > Hmm. Do the arm/ppc systems have 64k pages? kconfigs might be a good starting place. Also, does the xfs for-next branch exhibit this problem, or is it just the big -next branch that Stephen Rothwell puts out? --D > == arm64 == > # xfs_info /home/ > meta-data=/dev/mapper/rhel_hpe--apollo--cn99xx--11-home isize=512 agcount=4, agsize=113568256 blks > = sectsz=4096 attr=2, projid32bit=1 > = crc=1 finobt=1, sparse=1, rmapbt=0 > = reflink=1 > data = bsize=4096 blocks=454273024, imaxpct=5 > = sunit=0 swidth=0 blks > naming =version 2 bsize=4096 ascii-ci=0, ftype=1 > log =internal log bsize=4096 blocks=221813, version=2 > = sectsz=4096 sunit=1 blks, lazy-count=1 > realtime =none extsz=4096 blocks=0, rtextents=0 > > > == powerpc == > # xfs_info /home/ > meta-data=/dev/mapper/rhel_ibm--p9wr--01-home isize=512 agcount=4, agsize=118489856 blks > = sectsz=4096 attr=2, projid32bit=1 > = crc=1 finobt=1, sparse=1, rmapbt=0 > = reflink=1 > data = bsize=4096 blocks=473959424, imaxpct=5 > = sunit=0 swidth=0 blks > naming =version 2 bsize=4096 ascii-ci=0, ftype=1 > log =internal log bsize=4096 blocks=231425, version=2 > = sectsz=4096 sunit=1 blks, lazy-count=1 > realtime =none extsz=4096 blocks=0, rtextents=0 > > == x86 (not yet reproduced) == > meta-data=/dev/mapper/rhel_hpe--dl380gen9--01-home isize=512 agcount=16, agsize=3283776 blks > = sectsz=512 attr=2, projid32bit=1 > = crc=1 finobt=1, sparse=1, rmapbt=0 > = reflink=1 > data = bsize=4096 blocks=52540416, imaxpct=25 > = sunit=64 swidth=64 blks > naming =version 2 bsize=4096 ascii-ci=0, ftype=1 > log =internal log bsize=4096 blocks=25664, version=2 > = sectsz=512 sunit=0 blks, lazy-count=1 > realtime =none extsz=4096 blocks=0, rtextents=0 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: linux-next: xfs metadata corruption since 30 March 2020-04-01 4:45 ` Darrick J. Wong @ 2020-04-01 6:10 ` Chandan Rajendra 2020-04-01 13:54 ` Qian Cai 1 sibling, 0 replies; 11+ messages in thread From: Chandan Rajendra @ 2020-04-01 6:10 UTC (permalink / raw) To: Darrick J. Wong; +Cc: Qian Cai, Christoph Hellwig, linux-xfs, LKML On Wednesday, April 1, 2020 10:15 AM Darrick J. Wong wrote: > On Wed, Apr 01, 2020 at 12:15:32AM -0400, Qian Cai wrote: > > > > > > > On Apr 1, 2020, at 12:14 AM, Chandan Rajendra <chandan@linux.ibm.com> wrote: > > > > > > On Wednesday, April 1, 2020 3:27 AM Qian Cai wrote: > > >> Ever since two days ago, linux-next starts to trigger xfs metadata corruption > > >> during compilation workloads on both powerpc and arm64, > > > > > > Can you please provide the filesystem geometry information? > > > You can get that by executing "xfs_info <mount-point>" command. > > > > I wasn't able to recreate this issue on my P8 kvm guest. Can you provide, 1. The build command line you are using. 2. The number of online CPUs and amount of system memory. 2. As Darrick pointed out, Can you please provide the kconfig used (especially the one used for powerpc build). > Hmm. Do the arm/ppc systems have 64k pages? kconfigs might be a good > starting place. Also, does the xfs for-next branch exhibit this > problem, or is it just the big -next branch that Stephen Rothwell puts > out? > > > --D > > > == arm64 == > > # xfs_info /home/ > > meta-data=/dev/mapper/rhel_hpe--apollo--cn99xx--11-home isize=512 agcount=4, agsize=113568256 blks > > = sectsz=4096 attr=2, projid32bit=1 > > = crc=1 finobt=1, sparse=1, rmapbt=0 > > = reflink=1 > > data = bsize=4096 blocks=454273024, imaxpct=5 > > = sunit=0 swidth=0 blks > > naming =version 2 bsize=4096 ascii-ci=0, ftype=1 > > log =internal log bsize=4096 blocks=221813, version=2 > > = sectsz=4096 sunit=1 blks, lazy-count=1 > > realtime =none extsz=4096 blocks=0, rtextents=0 > > > > > > == powerpc == > > # xfs_info /home/ > > meta-data=/dev/mapper/rhel_ibm--p9wr--01-home isize=512 agcount=4, agsize=118489856 blks > > = sectsz=4096 attr=2, projid32bit=1 > > = crc=1 finobt=1, sparse=1, rmapbt=0 > > = reflink=1 > > data = bsize=4096 blocks=473959424, imaxpct=5 > > = sunit=0 swidth=0 blks > > naming =version 2 bsize=4096 ascii-ci=0, ftype=1 > > log =internal log bsize=4096 blocks=231425, version=2 > > = sectsz=4096 sunit=1 blks, lazy-count=1 > > realtime =none extsz=4096 blocks=0, rtextents=0 > > > > == x86 (not yet reproduced) == > > meta-data=/dev/mapper/rhel_hpe--dl380gen9--01-home isize=512 agcount=16, agsize=3283776 blks > > = sectsz=512 attr=2, projid32bit=1 > > = crc=1 finobt=1, sparse=1, rmapbt=0 > > = reflink=1 > > data = bsize=4096 blocks=52540416, imaxpct=25 > > = sunit=64 swidth=64 blks > > naming =version 2 bsize=4096 ascii-ci=0, ftype=1 > > log =internal log bsize=4096 blocks=25664, version=2 > > = sectsz=512 sunit=0 blks, lazy-count=1 > > realtime =none extsz=4096 blocks=0, rtextents=0 > -- chandan ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: linux-next: xfs metadata corruption since 30 March 2020-04-01 4:45 ` Darrick J. Wong 2020-04-01 6:10 ` Chandan Rajendra @ 2020-04-01 13:54 ` Qian Cai 1 sibling, 0 replies; 11+ messages in thread From: Qian Cai @ 2020-04-01 13:54 UTC (permalink / raw) To: Darrick J. Wong; +Cc: Chandan Rajendra, Christoph Hellwig, linux-xfs, LKML > On Apr 1, 2020, at 12:45 AM, Darrick J. Wong <darrick.wong@oracle.com> wrote: > > On Wed, Apr 01, 2020 at 12:15:32AM -0400, Qian Cai wrote: >> >> >>> On Apr 1, 2020, at 12:14 AM, Chandan Rajendra <chandan@linux.ibm.com> wrote: >>> >>> On Wednesday, April 1, 2020 3:27 AM Qian Cai wrote: >>>> Ever since two days ago, linux-next starts to trigger xfs metadata corruption >>>> during compilation workloads on both powerpc and arm64, >>> >>> Can you please provide the filesystem geometry information? >>> You can get that by executing "xfs_info <mount-point>" command. >>> > > Hmm. Do the arm/ppc systems have 64k pages? kconfigs might be a good > starting place. Also, does the xfs for-next branch exhibit this Yes, 64k pages. The configs are in, https://github.com/cailca/linux-mm > problem, or is it just the big -next branch that Stephen Rothwell puts > out? The later. > > --D > >> == arm64 == >> # xfs_info /home/ >> meta-data=/dev/mapper/rhel_hpe--apollo--cn99xx--11-home isize=512 agcount=4, agsize=113568256 blks >> = sectsz=4096 attr=2, projid32bit=1 >> = crc=1 finobt=1, sparse=1, rmapbt=0 >> = reflink=1 >> data = bsize=4096 blocks=454273024, imaxpct=5 >> = sunit=0 swidth=0 blks >> naming =version 2 bsize=4096 ascii-ci=0, ftype=1 >> log =internal log bsize=4096 blocks=221813, version=2 >> = sectsz=4096 sunit=1 blks, lazy-count=1 >> realtime =none extsz=4096 blocks=0, rtextents=0 >> >> >> == powerpc == >> # xfs_info /home/ >> meta-data=/dev/mapper/rhel_ibm--p9wr--01-home isize=512 agcount=4, agsize=118489856 blks >> = sectsz=4096 attr=2, projid32bit=1 >> = crc=1 finobt=1, sparse=1, rmapbt=0 >> = reflink=1 >> data = bsize=4096 blocks=473959424, imaxpct=5 >> = sunit=0 swidth=0 blks >> naming =version 2 bsize=4096 ascii-ci=0, ftype=1 >> log =internal log bsize=4096 blocks=231425, version=2 >> = sectsz=4096 sunit=1 blks, lazy-count=1 >> realtime =none extsz=4096 blocks=0, rtextents=0 >> >> == x86 (not yet reproduced) == >> meta-data=/dev/mapper/rhel_hpe--dl380gen9--01-home isize=512 agcount=16, agsize=3283776 blks >> = sectsz=512 attr=2, projid32bit=1 >> = crc=1 finobt=1, sparse=1, rmapbt=0 >> = reflink=1 >> data = bsize=4096 blocks=52540416, imaxpct=25 >> = sunit=64 swidth=64 blks >> naming =version 2 bsize=4096 ascii-ci=0, ftype=1 >> log =internal log bsize=4096 blocks=25664, version=2 >> = sectsz=512 sunit=0 blks, lazy-count=1 >> realtime =none extsz=4096 blocks=0, rtextents=0 ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: linux-next: xfs metadata corruption since 30 March 2020-03-31 21:57 linux-next: xfs metadata corruption since 30 March Qian Cai 2020-03-31 22:13 ` Dave Chinner 2020-04-01 4:14 ` Chandan Rajendra @ 2020-04-01 12:34 ` Brian Foster 2020-04-01 16:21 ` Brian Foster 2 siblings, 1 reply; 11+ messages in thread From: Brian Foster @ 2020-04-01 12:34 UTC (permalink / raw) To: Qian Cai; +Cc: Darrick J. Wong, Christoph Hellwig, linux-xfs, LKML On Tue, Mar 31, 2020 at 05:57:24PM -0400, Qian Cai wrote: > Ever since two days ago, linux-next starts to trigger xfs metadata corruption > during compilation workloads on both powerpc and arm64, > > I suspect it could be one of those commits, > > https://lore.kernel.org/linux-xfs/20200328182533.GM29339@magnolia/ > > Especially, those commits that would mark corruption more aggressively? > > [8d57c21600a5] xfs: add a function to deal with corrupt buffers post-verifiers > [e83cf875d67a] xfs: xfs_buf_corruption_error should take __this_address > [ce99494c9699] xfs: fix buffer corruption reporting when xfs_dir3_free_header_check fails > [1cb5deb5bc09] xfs: don't ever return a stale pointer from __xfs_dir3_free_read > [6fb5aac73310] xfs: check owner of dir3 free blocks > [a10c21ed5d52] xfs: check owner of dir3 data blocks > [1b2c1a63b678] xfs: check owner of dir3 blocks > [2e107cf869ee] xfs: mark dir corrupt when lookup-by-hash fails > [806d3909a57e] xfs: mark extended attr corrupt when lookup-by-hash fails > > > [29331.182313][ T665] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x2b8/0x350 [xfs], xfs_inode block 0xa9b97900 xfs_inode_buf_verify > xfs_inode_buf_verify at fs/xfs/libxfs/xfs_inode_buf.c:101 > [29331.182373][ T665] XFS (dm-2): Unmount and run xfs_repair > [29331.182386][ T665] XFS (dm-2): First 128 bytes of corrupted metadata buffer: > [29331.182402][ T665] 00000000: 2f 2a 20 53 50 44 58 2d 4c 69 63 65 6e 73 65 2d /* SPDX-License- > [29331.182426][ T665] 00000010: 49 64 65 6e 74 69 66 69 65 72 3a 20 47 50 4c 2d Identifier: GPL- > [29331.182442][ T665] 00000020: 32 2e 30 2d 6f 72 2d 6c 61 74 65 72 20 2a 2f 0a 2.0-or-later */. > [29331.182467][ T665] 00000030: 2f 2a 0a 20 2a 20 44 65 66 69 6e 69 74 69 6f 6e /*. * Definition > [29331.182492][ T665] 00000040: 73 20 61 6e 64 20 70 6c 61 74 66 6f 72 6d 20 64 s and platform d > [29331.182517][ T665] 00000050: 61 74 61 20 66 6f 72 20 41 6e 61 6c 6f 67 20 44 ata for Analog D > [29331.182541][ T665] 00000060: 65 76 69 63 65 73 0a 20 2a 20 41 44 50 35 35 32 evices. * ADP552 > [29331.182566][ T665] 00000070: 30 2f 41 44 50 35 35 30 31 20 4d 46 44 20 50 4d 0/ADP5501 MFD PM > [29331.182700][ T7490] XFS (dm-2): metadata I/O error in "xfs_imap_to_bp+0x88/0x130 [xfs]" at daddr 0xa9b97900 len 32 error 117 > xfs_trans_read_buf at fs/xfs/xfs_trans.h:209 > (inlined by) xfs_imap_to_bp at fs/xfs/libxfs/xfs_inode_buf.c:171 > [29331.182812][ T7490] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117. > [29331.345347][ T7490] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3754 of file fs/xfs/xfs_inode.c. Return address = 0000000058be213e > [29331.345378][ T7490] XFS (dm-2): Corruption of in-memory data detected. Shutting down filesystem > [29331.345402][ T7490] XFS (dm-2): Please unmount the filesystem and rectify the problem(s) I've actually been seeing similar corruption errors in the past day or two but I'd chalked it up to the work and testing I'm doing in my development branch. I'm also on a system/storage configuration that I haven't established trust in yet, fwiw. I have recently rebased to for-next, so I installed that baseline kernel and hit a similar crash after 20 minutes or so of running fsstress. I'm not sure how consistent this failure is yet but I'll see if I can back off from there and narrow it down to a patch... Brian > [29331.346474][ T498] dm-2: writeback error on inode 934606, offset 0, sector 961072 > [29331.346502][ T498] dm-2: writeback error on inode 934607, offset 0, sector 961080 > [29331.346624][ T498] dm-2: writeback error on inode 934608, offset 0, sector 961088 > [29331.346683][ T498] dm-2: writeback error on inode 1074331758, offset 0, sector 948449704 > [29331.347306][ T498] dm-2: writeback error on inode 1074331762, offset 0, sector 948558816 > [29331.349165][ T498] dm-2: writeback error on inode 1074331759, offset 0, sector 948560984 > [29331.349227][ T498] dm-2: writeback error on inode 1074331760, offset 0, sector 948562944 > [29331.349303][ T498] dm-2: writeback error on inode 1074331761, offset 0, sector 948568000 > > [ 7762.204313][T124538] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x2b8/0x350 [xfs], xfs_inode block 0x712398e0 xfs_inode_buf_verify > [ 7762.204599][T124538] XFS (dm-2): Unmount and run xfs_repair > [ 7762.204625][T124538] XFS (dm-2): First 128 bytes of corrupted metadata buffer: > [ 7762.204654][T124538] 00000000: 77 65 72 70 63 2f 69 6e 63 6c 75 64 65 2f 67 65 werpc/include/ge > [ 7762.204672][T124538] 00000010: 6e 65 72 61 74 65 64 2f 75 61 70 69 2f 61 73 6d nerated/uapi/asm > [ 7762.204699][T124538] 00000020: 2f 72 65 73 6f 75 72 63 65 2e 68 20 5c 0a 20 69 /resource.h \. i > [ 7762.204727][T124538] 00000030: 6e 63 6c 75 64 65 2f 61 73 6d 2d 67 65 6e 65 72 nclude/asm-gener > [ 7762.204745][T124538] 00000040: 69 63 2f 72 65 73 6f 75 72 63 65 2e 68 20 69 6e ic/resource.h in > [ 7762.204783][T124538] 00000050: 63 6c 75 64 65 2f 75 61 70 69 2f 61 73 6d 2d 67 clude/uapi/asm-g > [ 7762.204820][T124538] 00000060: 65 6e 65 72 69 63 2f 72 65 73 6f 75 72 63 65 2e eneric/resource. > [ 7762.204858][T124538] 00000070: 68 20 5c 0a 20 69 6e 63 6c 75 64 65 2f 6c 69 6e h \. include/lin > [ 7762.205068][ T7510] XFS (dm-2): metadata I/O error in "xfs_imap_to_bp+0x88/0x130 [xfs]" at daddr 0x712398e0 len 32 error 117 > [ 7762.205466][ T7510] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117. > [ 7762.219267][ T7510] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3754 of file fs/xfs/xfs_inode.c. Return address = 000000006bce0de3 > [ 7762.219291][ T7510] XFS (dm-2): Corruption of in-memory data detected. Shutting down filesystem > [ 7762.219306][ T7510] XFS (dm-2): Please unmount the filesystem and rectify the problem(s) > > > [ 1032.162278][ T1515] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x244/0x2bc [xfs], xfs_inode block 0xa2b75dc0 xfs_inode_buf_verify > [ 1032.176156][ T1515] XFS (dm-2): Unmount and run xfs_repair > [ 1032.181835][ T1515] XFS (dm-2): First 128 bytes of corrupted metadata buffer: > [ 1032.189140][ T1515] 00000000: 6e 29 20 22 22 20 76 61 6c 75 65 20 22 22 20 73 n) "" value "" s > [ 1032.197988][ T1515] 00000010: 75 62 73 74 72 28 6c 69 6e 65 2c 20 6c 65 6e 20 ubstr(line, len > [ 1032.206723][ T1515] 00000020: 2b 20 6b 65 79 6c 65 6e 20 2b 20 33 29 0a 20 20 + keylen + 3). > [ 1032.215675][ T1515] 00000030: 20 20 20 20 6c 65 6e 20 2b 3d 20 6c 65 6e 67 74 len += lengt > [ 1032.224537][ T1515] 00000040: 68 28 76 61 6c 75 65 29 20 2b 20 6c 65 6e 67 74 h(value) + lengt > [ 1032.233388][ T1515] 00000050: 68 28 66 69 65 6c 64 5b 2b 2b 69 5d 29 0a 20 20 h(field[++i]). > [ 1032.242234][ T1515] 00000060: 20 20 20 20 73 75 62 73 74 65 64 20 3d 20 31 0a substed = 1. > [ 1032.251077][ T1515] 00000070: 20 20 20 20 7d 20 65 6c 73 65 0a 20 20 20 20 20 } else. > [ 1032.260792][ T4119] XFS (dm-2): metadata I/O error in "xfs_imap_to_bp+0xd8/0x18c [xfs]" at daddr 0xa2b75dc0 len 32 error 117 > [ 1032.273096][ T4119] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117. > [ 1032.283283][ T4119] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3754 of file fs/xfs/xfs_inode.c. Return address = 00000000d99a2721 > [ 1032.296214][ T4119] XFS (dm-2): Corruption of in-memory data detected. Shutting down filesystem > [ 1032.305158][ T4119] XFS (dm-2): Please unmount the filesystem and rectify the problem(s) > ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: linux-next: xfs metadata corruption since 30 March 2020-04-01 12:34 ` Brian Foster @ 2020-04-01 16:21 ` Brian Foster 2020-04-01 18:24 ` Qian Cai 0 siblings, 1 reply; 11+ messages in thread From: Brian Foster @ 2020-04-01 16:21 UTC (permalink / raw) To: Qian Cai; +Cc: Darrick J. Wong, Christoph Hellwig, linux-xfs, LKML On Wed, Apr 01, 2020 at 08:34:47AM -0400, Brian Foster wrote: > On Tue, Mar 31, 2020 at 05:57:24PM -0400, Qian Cai wrote: > > Ever since two days ago, linux-next starts to trigger xfs metadata corruption > > during compilation workloads on both powerpc and arm64, > > > > I suspect it could be one of those commits, > > > > https://lore.kernel.org/linux-xfs/20200328182533.GM29339@magnolia/ > > > > Especially, those commits that would mark corruption more aggressively? > > > > [8d57c21600a5] xfs: add a function to deal with corrupt buffers post-verifiers > > [e83cf875d67a] xfs: xfs_buf_corruption_error should take __this_address > > [ce99494c9699] xfs: fix buffer corruption reporting when xfs_dir3_free_header_check fails > > [1cb5deb5bc09] xfs: don't ever return a stale pointer from __xfs_dir3_free_read > > [6fb5aac73310] xfs: check owner of dir3 free blocks > > [a10c21ed5d52] xfs: check owner of dir3 data blocks > > [1b2c1a63b678] xfs: check owner of dir3 blocks > > [2e107cf869ee] xfs: mark dir corrupt when lookup-by-hash fails > > [806d3909a57e] xfs: mark extended attr corrupt when lookup-by-hash fails > > > > > > [29331.182313][ T665] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x2b8/0x350 [xfs], xfs_inode block 0xa9b97900 xfs_inode_buf_verify > > xfs_inode_buf_verify at fs/xfs/libxfs/xfs_inode_buf.c:101 > > [29331.182373][ T665] XFS (dm-2): Unmount and run xfs_repair > > [29331.182386][ T665] XFS (dm-2): First 128 bytes of corrupted metadata buffer: > > [29331.182402][ T665] 00000000: 2f 2a 20 53 50 44 58 2d 4c 69 63 65 6e 73 65 2d /* SPDX-License- > > [29331.182426][ T665] 00000010: 49 64 65 6e 74 69 66 69 65 72 3a 20 47 50 4c 2d Identifier: GPL- > > [29331.182442][ T665] 00000020: 32 2e 30 2d 6f 72 2d 6c 61 74 65 72 20 2a 2f 0a 2.0-or-later */. > > [29331.182467][ T665] 00000030: 2f 2a 0a 20 2a 20 44 65 66 69 6e 69 74 69 6f 6e /*. * Definition > > [29331.182492][ T665] 00000040: 73 20 61 6e 64 20 70 6c 61 74 66 6f 72 6d 20 64 s and platform d > > [29331.182517][ T665] 00000050: 61 74 61 20 66 6f 72 20 41 6e 61 6c 6f 67 20 44 ata for Analog D > > [29331.182541][ T665] 00000060: 65 76 69 63 65 73 0a 20 2a 20 41 44 50 35 35 32 evices. * ADP552 > > [29331.182566][ T665] 00000070: 30 2f 41 44 50 35 35 30 31 20 4d 46 44 20 50 4d 0/ADP5501 MFD PM > > [29331.182700][ T7490] XFS (dm-2): metadata I/O error in "xfs_imap_to_bp+0x88/0x130 [xfs]" at daddr 0xa9b97900 len 32 error 117 > > xfs_trans_read_buf at fs/xfs/xfs_trans.h:209 > > (inlined by) xfs_imap_to_bp at fs/xfs/libxfs/xfs_inode_buf.c:171 > > [29331.182812][ T7490] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117. > > [29331.345347][ T7490] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3754 of file fs/xfs/xfs_inode.c. Return address = 0000000058be213e > > [29331.345378][ T7490] XFS (dm-2): Corruption of in-memory data detected. Shutting down filesystem > > [29331.345402][ T7490] XFS (dm-2): Please unmount the filesystem and rectify the problem(s) > > I've actually been seeing similar corruption errors in the past day or > two but I'd chalked it up to the work and testing I'm doing in my > development branch. I'm also on a system/storage configuration that I > haven't established trust in yet, fwiw. I have recently rebased to > for-next, so I installed that baseline kernel and hit a similar crash > after 20 minutes or so of running fsstress. I'm not sure how consistent > this failure is yet but I'll see if I can back off from there and narrow > it down to a patch... > A bisect lands at commit 5806165a6663 ("xfs: factor inode lookup from xfs_ifree_cluster"). The exact sequence of the error is not clear to me, but from inspection the following seems to fix it: diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index 5c930863ed5b..a455be84b2d4 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -2512,7 +2512,7 @@ static struct xfs_inode * xfs_ifree_get_one_inode( struct xfs_perag *pag, struct xfs_inode *free_ip, - int inum) + xfs_ino_t inum) { struct xfs_mount *mp = pag->pag_mount; struct xfs_inode *ip; Qian, care to confirm? Brian > Brian > > > [29331.346474][ T498] dm-2: writeback error on inode 934606, offset 0, sector 961072 > > [29331.346502][ T498] dm-2: writeback error on inode 934607, offset 0, sector 961080 > > [29331.346624][ T498] dm-2: writeback error on inode 934608, offset 0, sector 961088 > > [29331.346683][ T498] dm-2: writeback error on inode 1074331758, offset 0, sector 948449704 > > [29331.347306][ T498] dm-2: writeback error on inode 1074331762, offset 0, sector 948558816 > > [29331.349165][ T498] dm-2: writeback error on inode 1074331759, offset 0, sector 948560984 > > [29331.349227][ T498] dm-2: writeback error on inode 1074331760, offset 0, sector 948562944 > > [29331.349303][ T498] dm-2: writeback error on inode 1074331761, offset 0, sector 948568000 > > > > [ 7762.204313][T124538] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x2b8/0x350 [xfs], xfs_inode block 0x712398e0 xfs_inode_buf_verify > > [ 7762.204599][T124538] XFS (dm-2): Unmount and run xfs_repair > > [ 7762.204625][T124538] XFS (dm-2): First 128 bytes of corrupted metadata buffer: > > [ 7762.204654][T124538] 00000000: 77 65 72 70 63 2f 69 6e 63 6c 75 64 65 2f 67 65 werpc/include/ge > > [ 7762.204672][T124538] 00000010: 6e 65 72 61 74 65 64 2f 75 61 70 69 2f 61 73 6d nerated/uapi/asm > > [ 7762.204699][T124538] 00000020: 2f 72 65 73 6f 75 72 63 65 2e 68 20 5c 0a 20 69 /resource.h \. i > > [ 7762.204727][T124538] 00000030: 6e 63 6c 75 64 65 2f 61 73 6d 2d 67 65 6e 65 72 nclude/asm-gener > > [ 7762.204745][T124538] 00000040: 69 63 2f 72 65 73 6f 75 72 63 65 2e 68 20 69 6e ic/resource.h in > > [ 7762.204783][T124538] 00000050: 63 6c 75 64 65 2f 75 61 70 69 2f 61 73 6d 2d 67 clude/uapi/asm-g > > [ 7762.204820][T124538] 00000060: 65 6e 65 72 69 63 2f 72 65 73 6f 75 72 63 65 2e eneric/resource. > > [ 7762.204858][T124538] 00000070: 68 20 5c 0a 20 69 6e 63 6c 75 64 65 2f 6c 69 6e h \. include/lin > > [ 7762.205068][ T7510] XFS (dm-2): metadata I/O error in "xfs_imap_to_bp+0x88/0x130 [xfs]" at daddr 0x712398e0 len 32 error 117 > > [ 7762.205466][ T7510] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117. > > [ 7762.219267][ T7510] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3754 of file fs/xfs/xfs_inode.c. Return address = 000000006bce0de3 > > [ 7762.219291][ T7510] XFS (dm-2): Corruption of in-memory data detected. Shutting down filesystem > > [ 7762.219306][ T7510] XFS (dm-2): Please unmount the filesystem and rectify the problem(s) > > > > > > [ 1032.162278][ T1515] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x244/0x2bc [xfs], xfs_inode block 0xa2b75dc0 xfs_inode_buf_verify > > [ 1032.176156][ T1515] XFS (dm-2): Unmount and run xfs_repair > > [ 1032.181835][ T1515] XFS (dm-2): First 128 bytes of corrupted metadata buffer: > > [ 1032.189140][ T1515] 00000000: 6e 29 20 22 22 20 76 61 6c 75 65 20 22 22 20 73 n) "" value "" s > > [ 1032.197988][ T1515] 00000010: 75 62 73 74 72 28 6c 69 6e 65 2c 20 6c 65 6e 20 ubstr(line, len > > [ 1032.206723][ T1515] 00000020: 2b 20 6b 65 79 6c 65 6e 20 2b 20 33 29 0a 20 20 + keylen + 3). > > [ 1032.215675][ T1515] 00000030: 20 20 20 20 6c 65 6e 20 2b 3d 20 6c 65 6e 67 74 len += lengt > > [ 1032.224537][ T1515] 00000040: 68 28 76 61 6c 75 65 29 20 2b 20 6c 65 6e 67 74 h(value) + lengt > > [ 1032.233388][ T1515] 00000050: 68 28 66 69 65 6c 64 5b 2b 2b 69 5d 29 0a 20 20 h(field[++i]). > > [ 1032.242234][ T1515] 00000060: 20 20 20 20 73 75 62 73 74 65 64 20 3d 20 31 0a substed = 1. > > [ 1032.251077][ T1515] 00000070: 20 20 20 20 7d 20 65 6c 73 65 0a 20 20 20 20 20 } else. > > [ 1032.260792][ T4119] XFS (dm-2): metadata I/O error in "xfs_imap_to_bp+0xd8/0x18c [xfs]" at daddr 0xa2b75dc0 len 32 error 117 > > [ 1032.273096][ T4119] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117. > > [ 1032.283283][ T4119] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3754 of file fs/xfs/xfs_inode.c. Return address = 00000000d99a2721 > > [ 1032.296214][ T4119] XFS (dm-2): Corruption of in-memory data detected. Shutting down filesystem > > [ 1032.305158][ T4119] XFS (dm-2): Please unmount the filesystem and rectify the problem(s) > > > ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: linux-next: xfs metadata corruption since 30 March 2020-04-01 16:21 ` Brian Foster @ 2020-04-01 18:24 ` Qian Cai 0 siblings, 0 replies; 11+ messages in thread From: Qian Cai @ 2020-04-01 18:24 UTC (permalink / raw) To: Brian Foster; +Cc: Darrick J. Wong, Christoph Hellwig, linux-xfs, LKML > On Apr 1, 2020, at 12:21 PM, Brian Foster <bfoster@redhat.com> wrote: > > On Wed, Apr 01, 2020 at 08:34:47AM -0400, Brian Foster wrote: >> On Tue, Mar 31, 2020 at 05:57:24PM -0400, Qian Cai wrote: >>> Ever since two days ago, linux-next starts to trigger xfs metadata corruption >>> during compilation workloads on both powerpc and arm64, >>> >>> I suspect it could be one of those commits, >>> >>> https://lore.kernel.org/linux-xfs/20200328182533.GM29339@magnolia/ >>> >>> Especially, those commits that would mark corruption more aggressively? >>> >>> [8d57c21600a5] xfs: add a function to deal with corrupt buffers post-verifiers >>> [e83cf875d67a] xfs: xfs_buf_corruption_error should take __this_address >>> [ce99494c9699] xfs: fix buffer corruption reporting when xfs_dir3_free_header_check fails >>> [1cb5deb5bc09] xfs: don't ever return a stale pointer from __xfs_dir3_free_read >>> [6fb5aac73310] xfs: check owner of dir3 free blocks >>> [a10c21ed5d52] xfs: check owner of dir3 data blocks >>> [1b2c1a63b678] xfs: check owner of dir3 blocks >>> [2e107cf869ee] xfs: mark dir corrupt when lookup-by-hash fails >>> [806d3909a57e] xfs: mark extended attr corrupt when lookup-by-hash fails >>> >>> >>> [29331.182313][ T665] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x2b8/0x350 [xfs], xfs_inode block 0xa9b97900 xfs_inode_buf_verify >>> xfs_inode_buf_verify at fs/xfs/libxfs/xfs_inode_buf.c:101 >>> [29331.182373][ T665] XFS (dm-2): Unmount and run xfs_repair >>> [29331.182386][ T665] XFS (dm-2): First 128 bytes of corrupted metadata buffer: >>> [29331.182402][ T665] 00000000: 2f 2a 20 53 50 44 58 2d 4c 69 63 65 6e 73 65 2d /* SPDX-License- >>> [29331.182426][ T665] 00000010: 49 64 65 6e 74 69 66 69 65 72 3a 20 47 50 4c 2d Identifier: GPL- >>> [29331.182442][ T665] 00000020: 32 2e 30 2d 6f 72 2d 6c 61 74 65 72 20 2a 2f 0a 2.0-or-later */. >>> [29331.182467][ T665] 00000030: 2f 2a 0a 20 2a 20 44 65 66 69 6e 69 74 69 6f 6e /*. * Definition >>> [29331.182492][ T665] 00000040: 73 20 61 6e 64 20 70 6c 61 74 66 6f 72 6d 20 64 s and platform d >>> [29331.182517][ T665] 00000050: 61 74 61 20 66 6f 72 20 41 6e 61 6c 6f 67 20 44 ata for Analog D >>> [29331.182541][ T665] 00000060: 65 76 69 63 65 73 0a 20 2a 20 41 44 50 35 35 32 evices. * ADP552 >>> [29331.182566][ T665] 00000070: 30 2f 41 44 50 35 35 30 31 20 4d 46 44 20 50 4d 0/ADP5501 MFD PM >>> [29331.182700][ T7490] XFS (dm-2): metadata I/O error in "xfs_imap_to_bp+0x88/0x130 [xfs]" at daddr 0xa9b97900 len 32 error 117 >>> xfs_trans_read_buf at fs/xfs/xfs_trans.h:209 >>> (inlined by) xfs_imap_to_bp at fs/xfs/libxfs/xfs_inode_buf.c:171 >>> [29331.182812][ T7490] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117. >>> [29331.345347][ T7490] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3754 of file fs/xfs/xfs_inode.c. Return address = 0000000058be213e >>> [29331.345378][ T7490] XFS (dm-2): Corruption of in-memory data detected. Shutting down filesystem >>> [29331.345402][ T7490] XFS (dm-2): Please unmount the filesystem and rectify the problem(s) >> >> I've actually been seeing similar corruption errors in the past day or >> two but I'd chalked it up to the work and testing I'm doing in my >> development branch. I'm also on a system/storage configuration that I >> haven't established trust in yet, fwiw. I have recently rebased to >> for-next, so I installed that baseline kernel and hit a similar crash >> after 20 minutes or so of running fsstress. I'm not sure how consistent >> this failure is yet but I'll see if I can back off from there and narrow >> it down to a patch... >> > > A bisect lands at commit 5806165a6663 ("xfs: factor inode lookup from > xfs_ifree_cluster"). The exact sequence of the error is not clear to me, > but from inspection the following seems to fix it: > > diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c > index 5c930863ed5b..a455be84b2d4 100644 > --- a/fs/xfs/xfs_inode.c > +++ b/fs/xfs/xfs_inode.c > @@ -2512,7 +2512,7 @@ static struct xfs_inode * > xfs_ifree_get_one_inode( > struct xfs_perag *pag, > struct xfs_inode *free_ip, > - int inum) > + xfs_ino_t inum) > { > struct xfs_mount *mp = pag->pag_mount; > struct xfs_inode *ip; > > Qian, care to confirm? Yes, the fix works well. > > Brian > >> Brian >> >>> [29331.346474][ T498] dm-2: writeback error on inode 934606, offset 0, sector 961072 >>> [29331.346502][ T498] dm-2: writeback error on inode 934607, offset 0, sector 961080 >>> [29331.346624][ T498] dm-2: writeback error on inode 934608, offset 0, sector 961088 >>> [29331.346683][ T498] dm-2: writeback error on inode 1074331758, offset 0, sector 948449704 >>> [29331.347306][ T498] dm-2: writeback error on inode 1074331762, offset 0, sector 948558816 >>> [29331.349165][ T498] dm-2: writeback error on inode 1074331759, offset 0, sector 948560984 >>> [29331.349227][ T498] dm-2: writeback error on inode 1074331760, offset 0, sector 948562944 >>> [29331.349303][ T498] dm-2: writeback error on inode 1074331761, offset 0, sector 948568000 >>> >>> [ 7762.204313][T124538] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x2b8/0x350 [xfs], xfs_inode block 0x712398e0 xfs_inode_buf_verify >>> [ 7762.204599][T124538] XFS (dm-2): Unmount and run xfs_repair >>> [ 7762.204625][T124538] XFS (dm-2): First 128 bytes of corrupted metadata buffer: >>> [ 7762.204654][T124538] 00000000: 77 65 72 70 63 2f 69 6e 63 6c 75 64 65 2f 67 65 werpc/include/ge >>> [ 7762.204672][T124538] 00000010: 6e 65 72 61 74 65 64 2f 75 61 70 69 2f 61 73 6d nerated/uapi/asm >>> [ 7762.204699][T124538] 00000020: 2f 72 65 73 6f 75 72 63 65 2e 68 20 5c 0a 20 69 /resource.h \. i >>> [ 7762.204727][T124538] 00000030: 6e 63 6c 75 64 65 2f 61 73 6d 2d 67 65 6e 65 72 nclude/asm-gener >>> [ 7762.204745][T124538] 00000040: 69 63 2f 72 65 73 6f 75 72 63 65 2e 68 20 69 6e ic/resource.h in >>> [ 7762.204783][T124538] 00000050: 63 6c 75 64 65 2f 75 61 70 69 2f 61 73 6d 2d 67 clude/uapi/asm-g >>> [ 7762.204820][T124538] 00000060: 65 6e 65 72 69 63 2f 72 65 73 6f 75 72 63 65 2e eneric/resource. >>> [ 7762.204858][T124538] 00000070: 68 20 5c 0a 20 69 6e 63 6c 75 64 65 2f 6c 69 6e h \. include/lin >>> [ 7762.205068][ T7510] XFS (dm-2): metadata I/O error in "xfs_imap_to_bp+0x88/0x130 [xfs]" at daddr 0x712398e0 len 32 error 117 >>> [ 7762.205466][ T7510] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117. >>> [ 7762.219267][ T7510] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3754 of file fs/xfs/xfs_inode.c. Return address = 000000006bce0de3 >>> [ 7762.219291][ T7510] XFS (dm-2): Corruption of in-memory data detected. Shutting down filesystem >>> [ 7762.219306][ T7510] XFS (dm-2): Please unmount the filesystem and rectify the problem(s) >>> >>> >>> [ 1032.162278][ T1515] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x244/0x2bc [xfs], xfs_inode block 0xa2b75dc0 xfs_inode_buf_verify >>> [ 1032.176156][ T1515] XFS (dm-2): Unmount and run xfs_repair >>> [ 1032.181835][ T1515] XFS (dm-2): First 128 bytes of corrupted metadata buffer: >>> [ 1032.189140][ T1515] 00000000: 6e 29 20 22 22 20 76 61 6c 75 65 20 22 22 20 73 n) "" value "" s >>> [ 1032.197988][ T1515] 00000010: 75 62 73 74 72 28 6c 69 6e 65 2c 20 6c 65 6e 20 ubstr(line, len >>> [ 1032.206723][ T1515] 00000020: 2b 20 6b 65 79 6c 65 6e 20 2b 20 33 29 0a 20 20 + keylen + 3). >>> [ 1032.215675][ T1515] 00000030: 20 20 20 20 6c 65 6e 20 2b 3d 20 6c 65 6e 67 74 len += lengt >>> [ 1032.224537][ T1515] 00000040: 68 28 76 61 6c 75 65 29 20 2b 20 6c 65 6e 67 74 h(value) + lengt >>> [ 1032.233388][ T1515] 00000050: 68 28 66 69 65 6c 64 5b 2b 2b 69 5d 29 0a 20 20 h(field[++i]). >>> [ 1032.242234][ T1515] 00000060: 20 20 20 20 73 75 62 73 74 65 64 20 3d 20 31 0a substed = 1. >>> [ 1032.251077][ T1515] 00000070: 20 20 20 20 7d 20 65 6c 73 65 0a 20 20 20 20 20 } else. >>> [ 1032.260792][ T4119] XFS (dm-2): metadata I/O error in "xfs_imap_to_bp+0xd8/0x18c [xfs]" at daddr 0xa2b75dc0 len 32 error 117 >>> [ 1032.273096][ T4119] XFS (dm-2): xfs_imap_to_bp: xfs_trans_read_buf() returned error -117. >>> [ 1032.283283][ T4119] XFS (dm-2): xfs_do_force_shutdown(0x8) called from line 3754 of file fs/xfs/xfs_inode.c. Return address = 00000000d99a2721 >>> [ 1032.296214][ T4119] XFS (dm-2): Corruption of in-memory data detected. Shutting down filesystem >>> [ 1032.305158][ T4119] XFS (dm-2): Please unmount the filesystem and rectify the problem(s) ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2020-04-01 18:25 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-03-31 21:57 linux-next: xfs metadata corruption since 30 March Qian Cai 2020-03-31 22:13 ` Dave Chinner 2020-04-01 2:13 ` Qian Cai 2020-04-01 4:14 ` Chandan Rajendra 2020-04-01 4:15 ` Qian Cai 2020-04-01 4:45 ` Darrick J. Wong 2020-04-01 6:10 ` Chandan Rajendra 2020-04-01 13:54 ` Qian Cai 2020-04-01 12:34 ` Brian Foster 2020-04-01 16:21 ` Brian Foster 2020-04-01 18:24 ` Qian Cai
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).