linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Dave Chinner <david@fromorbit.com>
To: Qian Cai <cai@lca.pw>
Cc: "Darrick J. Wong" <darrick.wong@oracle.com>,
	Christoph Hellwig <hch@lst.de>,
	linux-xfs@vger.kernel.org, LKML <linux-kernel@vger.kernel.org>
Subject: Re: linux-next: xfs metadata corruption since 30 March
Date: Wed, 1 Apr 2020 09:13:24 +1100	[thread overview]
Message-ID: <20200331221324.GZ10776@dread.disaster.area> (raw)
In-Reply-To: <990EDC4E-1A4E-4AC3-84D9-078ACF5EB9CC@lca.pw>

On Tue, Mar 31, 2020 at 05:57:24PM -0400, Qian Cai wrote:
> Ever since two days ago, linux-next starts to trigger xfs metadata corruption
> during compilation workloads on both powerpc and arm64,

Is this on an existing filesystem, or a new filesystem?

> I suspect it could be one of those commits,
> 
> https://lore.kernel.org/linux-xfs/20200328182533.GM29339@magnolia/
> 
> Especially, those commits that would mark corruption more aggressively?
> 
>       [8d57c21600a5] xfs: add a function to deal with corrupt buffers post-verifiers
>       [e83cf875d67a] xfs: xfs_buf_corruption_error should take __this_address
>       [ce99494c9699] xfs: fix buffer corruption reporting when xfs_dir3_free_header_check fails
>       [1cb5deb5bc09] xfs: don't ever return a stale pointer from __xfs_dir3_free_read
>       [6fb5aac73310] xfs: check owner of dir3 free blocks
>       [a10c21ed5d52] xfs: check owner of dir3 data blocks
>       [1b2c1a63b678] xfs: check owner of dir3 blocks
>       [2e107cf869ee] xfs: mark dir corrupt when lookup-by-hash fails
>       [806d3909a57e] xfs: mark extended attr corrupt when lookup-by-hash fails

Doubt it - they only add extra detection code and these:

> [29331.182313][  T665] XFS (dm-2): Metadata corruption detected at xfs_inode_buf_verify+0x2b8/0x350 [xfs], xfs_inode block 0xa9b97900 xfs_inode_buf_verify
> xfs_inode_buf_verify at fs/xfs/libxfs/xfs_inode_buf.c:101
> [29331.182373][  T665] XFS (dm-2): Unmount and run xfs_repair
> [29331.182386][  T665] XFS (dm-2): First 128 bytes of corrupted metadata buffer:
> [29331.182402][  T665] 00000000: 2f 2a 20 53 50 44 58 2d 4c 69 63 65 6e 73 65 2d  /* SPDX-License-
> [29331.182426][  T665] 00000010: 49 64 65 6e 74 69 66 69 65 72 3a 20 47 50 4c 2d  Identifier: GPL-

Would get caught by the existing  verifiers as they aren't valid
metadata at all.

Basically, you are getting file data where there should be inode
metadata. First thing to do is fix the existing corruptions with
xfs_repair - please post the entire output so we can see what was
corruption and what it fixed.

Then if the problem is still reproducable, I suspect you are going
to have to bisect it. i.e. run test, get corruption, mark bisect
bad, run xfs_repair or mkfs to fix mess, install new kernel, run
test again....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

  reply	other threads:[~2020-03-31 22:13 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-31 21:57 Qian Cai
2020-03-31 22:13 ` Dave Chinner [this message]
2020-04-01  2:13   ` Qian Cai
2020-04-01  4:14 ` Chandan Rajendra
2020-04-01  4:15   ` Qian Cai
2020-04-01  4:45     ` Darrick J. Wong
2020-04-01  6:10       ` Chandan Rajendra
2020-04-01 13:54       ` Qian Cai
2020-04-01 12:34 ` Brian Foster
2020-04-01 16:21   ` Brian Foster
2020-04-01 18:24     ` Qian Cai

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200331221324.GZ10776@dread.disaster.area \
    --to=david@fromorbit.com \
    --cc=cai@lca.pw \
    --cc=darrick.wong@oracle.com \
    --cc=hch@lst.de \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-xfs@vger.kernel.org \
    --subject='Re: linux-next: xfs metadata corruption since 30 March' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).