From: Eric Sandeen <sandeen@sandeen.net>
To: Patrick Rynhart <patrick@rynhart.co.nz>, linux-xfs@vger.kernel.org
Subject: Re: Inflight Corruption of XFS filesystem on CentOS 7.7 VMs
Date: Fri, 15 Nov 2019 23:29:20 -0600 [thread overview]
Message-ID: <80429a04-4b19-ec4a-1255-67b15c7b01f5@sandeen.net> (raw)
In-Reply-To: <CAMbe+5D9cSEpR2YTWTmigi77caw93p6qR-iAYf-X_3_OJQMROw@mail.gmail.com>
On 11/15/19 9:33 PM, Patrick Rynhart wrote:
> Hi all,
>
> A small number of our CentOS VMs (about 4 out of a fleet of 200) are
> experiencing ongoing, regular XFS corruption - and I'm not sure how to
> troubleshoot the problem. They are all CentOS 7.7 VMs are are using
> VMWare Paravirtual SCSI. The version of xfsprogs being used is
> 4.5.0-20.el7.x86_64, and the kernel is 3.10.0-1062.1.2.el7.x86_64.
> The VMWare version is ESXi, 6.5.0, 14320405.
>
> When the fault happens - the VMs will go into single user mode with
> the following text displayed on the console:
>
> sd 0:0:0:0: [sda] Assuming drive cache: write through
> XFS (dm-0): Internal error XFS_WANT_CORRUPTED_GOTO at line 1664 of
> file fs/xfs/libxfs
> /xfs_alloc.c. Caller xfs_free_extent+0xaa/0x140 [xfs]
> XFS (dm-0): Internal error xfs_trans_cancel at line 984 of file
> fs/xfs/xfs_trans.c.
> Caller xfs_efi_recover+0x17d/0x1a0 [xfs]
> XFS (dm-0): Corruption of in-memory data detected. Shutting down filesystem
> XFS (dm-0): Please umount the filesystem and rectify the problem(s)
> XFS (dm-0): Failed to recover intents
Seems like this is not the whole relevant log; "Failed to recover intents"
indicates it was in log replay but we don't see that starting. Did you
cut out other interesting bits?
-Eric
next prev parent reply other threads:[~2019-11-16 5:29 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-11-16 3:33 Inflight Corruption of XFS filesystem on CentOS 7.7 VMs Patrick Rynhart
2019-11-16 5:29 ` Eric Sandeen [this message]
2019-11-16 5:51 ` Patrick Rynhart
2019-11-19 13:54 ` Brian Foster
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=80429a04-4b19-ec4a-1255-67b15c7b01f5@sandeen.net \
--to=sandeen@sandeen.net \
--cc=linux-xfs@vger.kernel.org \
--cc=patrick@rynhart.co.nz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).