catch22: xfs_repair fails, tells me to mount, which fails

catch22: xfs_repair fails, tells me to mount, which fails

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 3459 bytes --]

Hi,

this morning I woke up to a system without /home, the reason being:

  piper:~# mount /dev/mapper/vg0-home                                       [308]
  mount: Unknown error 990
  piper:~# dmesg | tail -20                                              [32,309]
  Starting XFS recovery on filesystem: dm-0 (logdev: internal)
  Filesystem "dm-0": xfs_inode_recover: Bad inode magic number, dino ptr = 0xffff810028a6bd00, dino bp = 0xffff81002851d140, ino = 33629
  Filesystem "dm-0": XFS internal error xlog_recover_do_inode_trans(1) at line 2352 of file fs/xfs/xfs_log_recover.c.  Caller 0xffffffff88307729

  Call Trace: <ffffffff8830753c>{:xfs:xlog_recover_commit_trans+4198}
        <ffffffff80231360>{submit_bio+184} <ffffffff88307729>{:xfs:xlog_recover_process_data+466}
        <ffffffff883081c6>{:xfs:xlog_do_recovery_pass+585} <ffffffff883096f4>{:xfs:xlog_recover+243}
        <ffffffff88305b2f>{:xfs:xfs_log_mount+1310} <ffffffff8830b671>{:xfs:xfs_mountfs+2116}
        <ffffffff880a0076>{:libata:ata_exec_command+0} <ffffffff8020bcdc>{_atomic_dec_and_lock+57}
        <ffffffff883119ac>{:xfs:xfs_mount+1900} <ffffffff88320bb0>{:xfs:xfs_fs_fill_super+126}
        <ffffffff802bc01d>{get_filesystem+18} <ffffffff802b530c>{sget+850}
        <ffffffff802b4c65>{set_bdev_super+0} <ffffffff802b4c74>{test_bdev_super+0}
        <ffffffff802b5dec>{bd_claim+24} <ffffffff802b5996>{get_sb_bdev+239}
        <ffffffff88320b32>{:xfs:xfs_fs_fill_super+0} <ffffffff802b5605>{do_kern_mount+157}
        <ffffffff802bd86e>{do_mount+1676} <ffffffff8022b2c2>{mntput_no_expire+25}
        <ffffffff802075bb>{find_get_page+33} <ffffffff80212173>{filemap_nopage+387}
        <ffffffff8020878d>{__handle_mm_fault+1276} <ffffffff8020a699>{do_page_fault+1151}
        <ffffffff80249b1f>{sys_mount+138} <ffffffff80259b0e>{system_call+126}
  XFS: log mount/recovery failed: error 990
  XFS: log mount failed

When I try to run xfs_repair, I get:

  piper:~# xfs_repair /dev/mapper/vg0-home                                  [310]
  Phase 1 - find and verify superblock...
  Phase 2 - using internal log
          - zero log...
  ERROR: The filesystem has valuable metadata changes in a log which needs to
  be replayed.  Mount the filesystem to replay the log, and unmount it before
  re-running xfs_repair.  If you are unable to mount the filesystem, then use
  the -L option to destroy the log and attempt a repair.
  Note that destroying the log may cause corruption -- please attempt a mount
  of the filesystem before doing this.

I have already tried the xfs_db method of the "dir2" faq entry,
trying to delete that offending inode, but setting core.mode = 0 for
that inode did not do anything.

I do not have enough space around to xfsdump those 250Gb, nor would
I know how, given that I cannot mount the filesystem and xfsdump
works on mounted filesystems only.

I've had really bad experiences with XFS lately [0], but I am not
giving up just yet. This time around, I guess I can wait a bit and
try to get things fixed.

0. http://blog.madduck.net/geek/2006.08.09-through-with-xfs

Would you have any idea what I should do?

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
spamtraps: madduck.bogus@madduck.net
 
"... (ethik und ästhetik sind eins.)"
                                                       -- wittgenstein

[-- Attachment #2: Digital signature (GPG/PGP) --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: catch22: xfs_repair fails, tells me to mount, which fails

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]

FYI, here is how I fixed it, using LVM snapshots to ensure that
xfs_repair -L would produce acceptable results:

  http://blog.madduck.net/geek/2006.08.30-lvm-for-filesystem-recovery

I'd still be interested to know how I could have gone about fixing
it in case I could not have afforded to purge the journal.

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
spamtraps: madduck.bogus@madduck.net
 
the reason that every major university
maintains a department of mathematics
is that it's cheaper than
institutionalizing all those people.

[-- Attachment #2: Digital signature (GPG/PGP) --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: catch22: xfs_repair fails, tells me to mount, which fails

[Shailendra Tripathi]

Hi Martin,
           In this particular case, the log written is corrupted. That's 
why, forcing to make the inode going corrupt would not help. The real 
issue is why log content is not valid.
        By any chance, did you saw any disk I/O error ? If there is no 
disk I/O error scenario, following information would be more useful.

1. /usr/sbin/xfs_logprint -s 0 -D <device>
2. /usr/sbin/xfs_logprint -d <device>


You can try mounting with -L option but the risk is:
"
   ERROR: The filesystem has valuable metadata changes in a log which 
needs to
   be replayed.  Mount the filesystem to replay the log, and unmount it 
before
   re-running xfs_repair.  If you are unable to mount the filesystem, 
then use
   the -L option to destroy the log and attempt a repair.
   Note that destroying the log may cause corruption -- please attempt a 
mount
   of the filesystem before doing this.
"
martin f krafft wrote:
> Hi,
> 
> this morning I woke up to a system without /home, the reason being:
> 
>   piper:~# mount /dev/mapper/vg0-home                                       [308]
>   mount: Unknown error 990
>   piper:~# dmesg | tail -20                                              [32,309]
>   Starting XFS recovery on filesystem: dm-0 (logdev: internal)
>   Filesystem "dm-0": xfs_inode_recover: Bad inode magic number, dino ptr = 0xffff810028a6bd00, dino bp = 0xffff81002851d140, ino = 33629
>   Filesystem "dm-0": XFS internal error xlog_recover_do_inode_trans(1) at line 2352 of file fs/xfs/xfs_log_recover.c.  Caller 0xffffffff88307729
> 
>   Call Trace: <ffffffff8830753c>{:xfs:xlog_recover_commit_trans+4198}
>         <ffffffff80231360>{submit_bio+184} <ffffffff88307729>{:xfs:xlog_recover_process_data+466}
>         <ffffffff883081c6>{:xfs:xlog_do_recovery_pass+585} <ffffffff883096f4>{:xfs:xlog_recover+243}
>         <ffffffff88305b2f>{:xfs:xfs_log_mount+1310} <ffffffff8830b671>{:xfs:xfs_mountfs+2116}
>         <ffffffff880a0076>{:libata:ata_exec_command+0} <ffffffff8020bcdc>{_atomic_dec_and_lock+57}
>         <ffffffff883119ac>{:xfs:xfs_mount+1900} <ffffffff88320bb0>{:xfs:xfs_fs_fill_super+126}
>         <ffffffff802bc01d>{get_filesystem+18} <ffffffff802b530c>{sget+850}
>         <ffffffff802b4c65>{set_bdev_super+0} <ffffffff802b4c74>{test_bdev_super+0}
>         <ffffffff802b5dec>{bd_claim+24} <ffffffff802b5996>{get_sb_bdev+239}
>         <ffffffff88320b32>{:xfs:xfs_fs_fill_super+0} <ffffffff802b5605>{do_kern_mount+157}
>         <ffffffff802bd86e>{do_mount+1676} <ffffffff8022b2c2>{mntput_no_expire+25}
>         <ffffffff802075bb>{find_get_page+33} <ffffffff80212173>{filemap_nopage+387}
>         <ffffffff8020878d>{__handle_mm_fault+1276} <ffffffff8020a699>{do_page_fault+1151}
>         <ffffffff80249b1f>{sys_mount+138} <ffffffff80259b0e>{system_call+126}
>   XFS: log mount/recovery failed: error 990
>   XFS: log mount failed
> 
> When I try to run xfs_repair, I get:
> 
>   piper:~# xfs_repair /dev/mapper/vg0-home                                  [310]
>   Phase 1 - find and verify superblock...
>   Phase 2 - using internal log
>           - zero log...
>   ERROR: The filesystem has valuable metadata changes in a log which needs to
>   be replayed.  Mount the filesystem to replay the log, and unmount it before
>   re-running xfs_repair.  If you are unable to mount the filesystem, then use
>   the -L option to destroy the log and attempt a repair.
>   Note that destroying the log may cause corruption -- please attempt a mount
>   of the filesystem before doing this.
> 
> I have already tried the xfs_db method of the "dir2" faq entry,
> trying to delete that offending inode, but setting core.mode = 0 for
> that inode did not do anything.
> 
> I do not have enough space around to xfsdump those 250Gb, nor would
> I know how, given that I cannot mount the filesystem and xfsdump
> works on mounted filesystems only.
> 
> I've had really bad experiences with XFS lately [0], but I am not
> giving up just yet. This time around, I guess I can wait a bit and
> try to get things fixed.
> 
> 0. http://blog.madduck.net/geek/2006.08.09-through-with-xfs
> 
> Would you have any idea what I should do?
> 

Re: catch22: xfs_repair fails, tells me to mount, which fails

[martin f krafft]

[-- Attachment #1: Type: text/plain, Size: 1036 bytes --]

also sprach Shailendra Tripathi <stripathi@agami.com> [2006.08.30.1551 +0200]:
>        By any chance, did you saw any disk I/O error ?

No, definitely not.

> If there is no disk I/O error scenario, following information
> would be more useful.
> 
> 1. /usr/sbin/xfs_logprint -s 0 -D <device>
> 2. /usr/sbin/xfs_logprint -d <device>

Thanks for the info. I managed to restore the filesystem with
xfs_repair -L in the mean time (see followup post), but this is what
I'll keep in mind for the next time.

Btw: there's the xfs and linux-xfs mailing list, and I am confused
about whether they're separate or merged, and which one I should be
using. If you have any input, I'd love to hear it.

-- 
martin;              (greetings from the heart of the sun.)
  \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck
 
spamtraps: madduck.bogus@madduck.net
 
no micro$oft components were used
in the creation or posting of this email.
therefore, it is 100% virus free
and does not use html by default (yuck!).

[-- Attachment #2: Digital signature (GPG/PGP) --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: catch22: xfs_repair fails, tells me to mount, which fails

[Eric Sandeen]

martin f krafft wrote:

> Btw: there's the xfs and linux-xfs mailing list, and I am confused
> about whether they're separate or merged, and which one I should be
> using. If you have any input, I'd love to hear it.

xfs@ is the current one, linux-xfs@ is the old one, and is aliased to 
xfs@ (there was probably a post about this when xfs@ was created).

-Eric