XFS fs corruption - XFS_WANT_CORRUPTED_RETURN

XFS fs corruption - XFS_WANT_CORRUPTED_RETURN

[Christopher Pereira]

Hi,

I've been using XFS for many years on many qemu-kvm VMs without problems.
I do daily qcow2 snapshots and today I noticed that a snaphot I took on 
Jun  1 2022 has a corrupted XFS root partition and doesn't boot any more 
(on another VM instance).
The snapshot I took the day before is clean.
The VM is still running since May 11 2022, has not been rebooted and 
didn't crash which is the reason I'm reporting this issue.
This is a production VM with sensible data.

The kernel logged this error multiple times between 00:00:21 and 
00:03:31 on Jun 1:

Jun  1 00:00:21 *** kernel: XFS (dm-0): Internal error 
XFS_WANT_CORRUPTED_RETURN at line 337 of file 
fs/xfs/libxfs/xfs_alloc.c.  Caller xfs_alloc_ag_vextent_near+0x658/0xa60 
[xfs]
Jun  1 00:00:22 *** kernel: [<ffffffffa0230e5b>] 
xfs_error_report+0x3b/0x40 [xfs]
Jun  1 00:00:22 *** kernel: [<ffffffffa01f0588>] ? 
xfs_alloc_ag_vextent_near+0x658/0xa60 [xfs]
Jun  1 00:00:22 *** kernel: [<ffffffffa01ee684>] 
xfs_alloc_fixup_trees+0x2c4/0x370 [xfs]
Jun  1 00:00:22 *** kernel: [<ffffffffa01f0588>] 
xfs_alloc_ag_vextent_near+0x658/0xa60 [xfs]
Jun  1 00:00:22 *** kernel: [<ffffffffa01f120d>] 
xfs_alloc_ag_vextent+0xcd/0x110 [xfs]
Jun  1 00:00:22 *** kernel: [<ffffffffa01f1f89>] 
xfs_alloc_vextent+0x429/0x5e0 [xfs]
Jun  1 00:00:22 *** kernel: [<ffffffffa020237f>] 
xfs_bmap_btalloc+0x3af/0x710 [xfs]
Jun  1 00:00:22 *** kernel: [<ffffffffa02026ee>] xfs_bmap_alloc+0xe/0x10 
[xfs]
Jun  1 00:00:22 *** kernel: [<ffffffffa0203148>] 
xfs_bmapi_write+0x4d8/0xa90 [xfs]
Jun  1 00:00:22 *** kernel: [<ffffffffa023bd1b>] 
xfs_iomap_write_allocate+0x14b/0x350 [xfs]
Jun  1 00:00:22 *** kernel: [<ffffffffa0226dc6>] 
xfs_map_blocks+0x1c6/0x230 [xfs]
Jun  1 00:00:22 *** kernel: [<ffffffffa0227fe3>] 
xfs_vm_writepage+0x193/0x5d0 [xfs]
Jun  1 00:00:22 *** kernel: [<ffffffffa0227993>] 
xfs_vm_writepages+0x43/0x50 [xfs]
Jun  1 00:00:22 *** kernel: XFS (dm-0): page discard on page 
ffffea000cf60200, inode 0xc52bf7f, offset 0.

I'm running this (outdated) software:

- uname -a:
     Linux *** 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC 
2016 x86_64 x86_64 x86_64 GNU/Linux

- modinfo xfs
     filename: /lib/modules/3.10.0-327.22.2.el7.x86_64/kernel/fs/xfs/xfs.ko
     license:        GPL
     description:    SGI XFS with ACLs, security attributes, no debug 
enabled
     author:         Silicon Graphics, Inc.
     alias:          fs-xfs
     rhelversion:    7.2
     srcversion:     5F736B32E75482D75F98583
     depends:        libcrc32c
     intree:         Y
     vermagic:       3.10.0-327.22.2.el7.x86_64 SMP mod_unload modversions
     signer:         CentOS Linux kernel signing key
     sig_key: A9:80:1A:61:B3:68:60:1C:40:EB:DB:D5:DF:D1:F3:A7:70:07:BF:A4
     sig_hashalgo:   sha256

1) Is there any known issue with this xfs version?

2) How may I help you to trace this bug.
I could provide my WhatsApp number privately for direct communication.

Should I try a xfs_repair and post the logs here or via pastebin?

BTW: I'm a experienced developer and sysadmin, but have no experience 
regarding the XFS  driver.

Re: XFS fs corruption - XFS_WANT_CORRUPTED_RETURN

[Carlos Maiolino]

On Wed, Jun 30, 2022 at 12:23:36PM -0400, Christopher Pereira wrote:
> Hi,
> 
> I've been using XFS for many years on many qemu-kvm VMs without problems.
> I do daily qcow2 snapshots and today I noticed that a snaphot I took on
> Jun  1 2022 has a corrupted XFS root partition and doesn't boot any more
> (on another VM instance).
> The snapshot I took the day before is clean.
> The VM is still running since May 11 2022, has not been rebooted and
> didn't crash which is the reason I'm reporting this issue.
> This is a production VM with sensible data.
> 
> The kernel logged this error multiple times between 00:00:21 and
> 00:03:31 on Jun 1:
> 
> Jun  1 00:00:21 *** kernel: XFS (dm-0): Internal error
> XFS_WANT_CORRUPTED_RETURN at line 337 of file
> fs/xfs/libxfs/xfs_alloc.c.  Caller xfs_alloc_ag_vextent_near+0x658/0xa60
> [xfs]
> Jun  1 00:00:22 *** kernel: [<ffffffffa0230e5b>]
> xfs_error_report+0x3b/0x40 [xfs]
> Jun  1 00:00:22 *** kernel: [<ffffffffa01f0588>] ?
> xfs_alloc_ag_vextent_near+0x658/0xa60 [xfs]
> Jun  1 00:00:22 *** kernel: [<ffffffffa01ee684>]
> xfs_alloc_fixup_trees+0x2c4/0x370 [xfs]
> Jun  1 00:00:22 *** kernel: [<ffffffffa01f0588>]
> xfs_alloc_ag_vextent_near+0x658/0xa60 [xfs]
> Jun  1 00:00:22 *** kernel: [<ffffffffa01f120d>]
> xfs_alloc_ag_vextent+0xcd/0x110 [xfs]
> Jun  1 00:00:22 *** kernel: [<ffffffffa01f1f89>]
> xfs_alloc_vextent+0x429/0x5e0 [xfs]
> Jun  1 00:00:22 *** kernel: [<ffffffffa020237f>]
> xfs_bmap_btalloc+0x3af/0x710 [xfs]
> Jun  1 00:00:22 *** kernel: [<ffffffffa02026ee>] xfs_bmap_alloc+0xe/0x10
> [xfs]
> Jun  1 00:00:22 *** kernel: [<ffffffffa0203148>]
> xfs_bmapi_write+0x4d8/0xa90 [xfs]
> Jun  1 00:00:22 *** kernel: [<ffffffffa023bd1b>]
> xfs_iomap_write_allocate+0x14b/0x350 [xfs]
> Jun  1 00:00:22 *** kernel: [<ffffffffa0226dc6>]
> xfs_map_blocks+0x1c6/0x230 [xfs]
> Jun  1 00:00:22 *** kernel: [<ffffffffa0227fe3>]
> xfs_vm_writepage+0x193/0x5d0 [xfs]
> Jun  1 00:00:22 *** kernel: [<ffffffffa0227993>]
> xfs_vm_writepages+0x43/0x50 [xfs]
> Jun  1 00:00:22 *** kernel: XFS (dm-0): page discard on page
> ffffea000cf60200, inode 0xc52bf7f, offset 0.
> 
> I'm running this (outdated) software:
> 
> - uname -a:
>      Linux *** 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 23 17:05:11 UTC
> 2016 x86_64 x86_64 x86_64 GNU/Linux

> 1) Is there any known issue with this xfs version?
> 
> 2) How may I help you to trace this bug.
> I could provide my WhatsApp number privately for direct communication.
> 
> Should I try a xfs_repair and post the logs here or via pastebin?
> 
> BTW: I'm a experienced developer and sysadmin, but have no experience
> regarding the XFS  driver.

It seems like you are stepping into a corrupted btree, maybe try to xfs_repair
it and see if it fixes the problem. I can't tell you anything about known bugs,
you're using a (very) outdated kernel, so it's hard to say anything about bugs
here, you need to check this with the distribution directly. Maybe it will lit
a lamp on somebody's else mind, but it's unlikely you'll get much from here with
your current environment. If you repair the fs, and run into it again using
upstream code, it's another story, but by now, your best shot is run xfs_repair
on it and fix the corrupted tree.

You can use a more recent xfs_repair too without updating the kernel.

Cheers.

-- 
Carlos Maiolino