ext4_xattr_ibody_find:2180: inode #2669312: comm systemd-journal: corrupted in-inode xattr

ext4_xattr_ibody_find:2180: inode #2669312: comm systemd-journal: corrupted in-inode xattr

[Peter Geis]

Good Morning,

I have run into a fun issue where I have the follow error on boot,
which causes the device to drop to read only mode:
ext4_xattr_ibody_find:2180: inode #2669312: comm systemd-journal:
corrupted in-inode xattr
It marks the file system as unclean, which forces a fsck, but it seems
that even as of v1.46.2 e2fsprogs does not have the ability to resolve
this issue.
Any attempts to touch the corrupted files (I've found two so far)
leads to the same error, and I cannot find a way to remove or repair
the affected inodes without wiping the entire file system.

Any insight you can provide would be much appreciated.
Thanks,
Peter Geis

Re: ext4_xattr_ibody_find:2180: inode #2669312: comm systemd-journal: corrupted in-inode xattr

[Jan Kara]

Hello!

On Fri 28-05-21 08:36:17, Peter Geis wrote:
> I have run into a fun issue where I have the follow error on boot,
> which causes the device to drop to read only mode:
> ext4_xattr_ibody_find:2180: inode #2669312: comm systemd-journal:
> corrupted in-inode xattr
> It marks the file system as unclean, which forces a fsck, but it seems
> that even as of v1.46.2 e2fsprogs does not have the ability to resolve
> this issue.
> Any attempts to touch the corrupted files (I've found two so far)
> leads to the same error, and I cannot find a way to remove or repair
> the affected inodes without wiping the entire file system.

This seems to have fallen through the cracks. Have you managed to fix your
filesystem? If not, please obtain filesystem image using e2image (it will
contain only fs metadata, not any data), compress it and put it somewhere for
download so that we can better diagnose the issue. Thanks!

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: ext4_xattr_ibody_find:2180: inode #2669312: comm systemd-journal: corrupted in-inode xattr

[Peter Geis]

On Tue, Jul 13, 2021 at 11:43 AM Jan Kara <jack@suse.cz> wrote:
>
> Hello!

Good Evening,

>
> On Fri 28-05-21 08:36:17, Peter Geis wrote:
> > I have run into a fun issue where I have the follow error on boot,
> > which causes the device to drop to read only mode:
> > ext4_xattr_ibody_find:2180: inode #2669312: comm systemd-journal:
> > corrupted in-inode xattr
> > It marks the file system as unclean, which forces a fsck, but it seems
> > that even as of v1.46.2 e2fsprogs does not have the ability to resolve
> > this issue.
> > Any attempts to touch the corrupted files (I've found two so far)
> > leads to the same error, and I cannot find a way to remove or repair
> > the affected inodes without wiping the entire file system.
>
> This seems to have fallen through the cracks. Have you managed to fix your
> filesystem? If not, please obtain filesystem image using e2image (it will
> contain only fs metadata, not any data), compress it and put it somewhere for
> download so that we can better diagnose the issue. Thanks!

Unfortunately no I don't have the file system anymore.
I put about two weeks into trying to fix it, but in the end I had to
nuke it from orbit.

Thanks for reaching out,
Peter

>
>                                                                 Honza
> --
> Jan Kara <jack@suse.com>
> SUSE Labs, CR

Re: ext4_xattr_ibody_find:2180: inode #2669312: comm systemd-journal: corrupted in-inode xattr

[Theodore Y. Ts'o]

On Tue, Jul 13, 2021 at 04:52:11PM -0400, Peter Geis wrote:
> > This seems to have fallen through the cracks. Have you managed to fix your
> > filesystem? If not, please obtain filesystem image using e2image (it will
> > contain only fs metadata, not any data), compress it and put it somewhere for
> > download so that we can better diagnose the issue. Thanks!
> 
> Unfortunately no I don't have the file system anymore.
> I put about two weeks into trying to fix it, but in the end I had to
> nuke it from orbit.

Sorry we didn't get back to you right away.  As Jan said, if this
happens again, it would be really helpful to use e2image.  There is a
discussion about this in the e2fsck man page, in the "REPORTING BUGS"
section.

In this specific case where there was an inode with a corrupted
extended attribute stored in the inode, you can use debugfs to print
out the inode data structure:

debugfs:  inode_dump testfile
0000  a481 0000 0000 0000 cf23 ef60 d523 ef60  .........#.`.#.`
0020  cf23 ef60 0000 0000 0000 0100 0000 0000  .#.`............
0040  0000 0800 0100 0000 0af3 0000 0400 0000  ................
0060  0000 0000 0000 0000 0000 0000 0000 0000  ................
*
0140  0000 0000 6867 c27f 0000 0000 0000 0000  ....hg..........
0160  0000 0000 0000 0000 0000 0000 9fbc 0000  ................
0200  2000 e8f9 34cc 39ec fcaf ee61 fcaf ee61   ...4.9....a...a
0220  cf23 ef60 fcaf ee61 0000 0000 0000 0000  .#.`...a........
0240  0000 02ea 0401 5400 0000 0000 0600 0000  ......T.........
0260  0000 0000 7465 7374 0000 0000 0000 0000  ....test........
0300  0000 0000 0000 0000 0000 0000 0000 0000  ................
*
0360  0000 0000 0000 0000 7473 7476 616c 0000  ........tstval..

debugfs:  inode_dump -x testfile
magic = ea020000, length = 96, value_start =4 

offset = 4 (0004), name_len = 4, name_index = 1
value_offset = 84 (0130), value_inum = 0, value_size = 6
name = test
value = tstval

last entry found at offset 24 (0030)

(And if you don't have the pathname to the file, you can also specify
the inode using the inode number in angle brackers, e.g., "inode_dump <12>")

Hopefully this won't happen to you again, but if it does, please do
let us know.

Cheers,

						- Ted