Possible to mount this XFS at least temporarily to retrieve files?

Possible to mount this XFS at least temporarily to retrieve files?

[Carsten Aulbert]

Hi

after some hiatus, back on this list with an incident which happened
yesterday:

On a Debian Jessie machine installed back in October 2016 there a re a
bunch of 3TB disks behind an Adaptec ASR-6405[1] in RAID6 configuration.
Yesterday, one of the disks failed and was subsequently replace. About
an hour into the rebuild the 28TB xfs on this block device gave up:

Oct 24 12:39:15 atlas8 kernel: [526440.956408] XFS (sdc1):
xfs_imap_to_bp: xfs_trans_read_buf() returned error 117.
Oct 24 12:39:15 atlas8 kernel: [526440.956452] XFS (sdc1):
xfs_do_force_shutdown(0x8) called from line 3242 of file
/build/linux-byISom/linux-3.16.43/fs/xfs/xfs_inode.c.  Return address =
0xffffffffa02c0b76
Oct 24 12:39:45 atlas8 kernel: [526471.029957] XFS (sdc1):
xfs_log_force: error 5 returned.
Oct 24 12:40:15 atlas8 kernel: [526501.154991] XFS (sdc1):
xfs_log_force: error 5 returned.

(mount options were probably (99% confidence)
rw,relatime,attr2,inode64,noquota
)

As we had several bind mounts as well as NFS clients on this one, I was
not able to clear all pending mounts - xfs_check/xfs_repair constantly
complaint about the file system still being mounted even though
/proc/self/mounts as well as fuser/lsof disagreed.

Anyway, we rebooted the system, tried to manually mount the file system
to replay any pending log but no luck as the primary superblock was not
found.

Running xfs_repair (from xfsprogs 3.2.1) on this first started looking
for a secondary superblock which apparently it found after about two
hours as it never again search for it afterwards.

However, after this running xfs_repait with and without the -L switch
stopped dead in phase 6 with the error that lost+found ran out of disk
space.

We then upgraded xfsprogs to 4.9.0+nmu1 and tried again and it failed
with the same error.

Another shot in the dark was rebooting the system with a more recent
kernel, this time 4.9.30-2+deb9u5~bpo8+1 instead of 3.16.43-2+deb8u5
which indeed changed the behaviour of xfs_repair:

# xfs_repair /dev/sdc1
Phase 1 - find and verify superblock...
sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with
calculated value 128
resetting superblock root inode pointer to 128
sb realtime bitmap inode 18446744073709551615 (NULLFSINO) inconsistent
with calculated value 129
resetting superblock realtime bitmap ino pointer to 129
sb realtime summary inode 18446744073709551615 (NULLFSINO) inconsistent
with calculated value 130
resetting superblock realtime summary ino pointer to 130
Phase 2 - using internal log
        - zero log...
Log inconsistent (didn't find previous header)
failed to find log head
zero_log: cannot find log head/tail (xlog_find_tail=5)
ERROR: The log head and/or tail cannot be discovered. Attempt to mount the
filesystem to replay the log or use the -L option to destroy the log and
attempt a repair.

xfs_repair -L /dev/sdc1
Phase 1 - find and verify superblock...
sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with
calculated value 128
resetting superblock root inode pointer to 128
sb realtime bitmap inode 18446744073709551615 (NULLFSINO) inconsistent
with calculated value 129
resetting superblock realtime bitmap ino pointer to 129
sb realtime summary inode 18446744073709551615 (NULLFSINO) inconsistent
with calculated value 130
resetting superblock realtime summary ino pointer to 130
Phase 2 - using internal log
        - zero log...
Log inconsistent (didn't find previous header)
failed to find log head
zero_log: cannot find log head/tail (xlog_find_tail=5)

an occasional trial mount fails with

# mount -vvv /dev/sdc1 /mnt
mount: /dev/sdc1: can't read superblock

and dmesg:
[46098.224814] XFS (sdc1): Mounting V4 Filesystem
[46098.340251] XFS (sdc1): Log inconsistent (didn't find previous header)
[46098.340290] XFS (sdc1): failed to find log head
[46098.340311] XFS (sdc1): log mount/recovery failed: error -5
[46098.340365] XFS (sdc1): log mount failed

I've run xfs_metadump just in case someone would be interested in this,
however this stops with

[...]
xfs_metadump: invalid magic in dir inode 95523308418 block 2669
xfs_metadump: invalid magic in dir inode 95523308418 block 2682
Copied 28488832 of 0 inodes (23 of 28 AGs)
xfs_metadump: suspicious count 2032 in bmap extent 84 in dir2 ino
98836491893
Copied 30292672 of 0 inodes (24 of 28 AGs)
xfs_metadump: suspicious count 1341 in bmap extent 249 in dir2 ino
103419978691
Copying log                                                Log
inconsistent (didn't find previous header)
failed to find log head
xlog_is_dirty: cannot find log head/tail (xlog_find_tail=5)

(but return value of 0)

So, I don't know if this is helpful at all or not - and it's quite large
1.7GB gzipped, about 12GB uncompressed.

Some more "random" output:

# xfs_db -r -c "sb 0" -c "p" -c "freesp" /dev/sdc1

magicnum = 0x58465342
blocksize = 4096
dblocks = 7313814267
rblocks = 0
rextents = 0
uuid = 9be23871-60a4-4deb-83bf-65e6e1efaf98
logstart = 3758096388
rootino = null
rbmino = null
rsumino = null
rextsize = 1
agblocks = 268435455
agcount = 28
rbmblocks = 0
logblocks = 521728
versionnum = 0xb4a4
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 28
rextslog = 0
inprogress = 0
imax_pct = 5
icount = 0
ifree = 0
fdblocks = 7313292427
frextents = 0
uquotino = null
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 1
features2 = 0x8a
bad_features2 = 0x8a
features_compat = 0
features_ro_compat = 0
features_incompat = 0
features_log_incompat = 0
crc = 0 (unchecked)
spino_align = 0
pquotino = 0
lsn = 0
meta_uuid = 00000000-0000-0000-0000-000000000000
   from      to extents  blocks    pct
      1       1    7326    7326   0.00
      2       3   10132   21624   0.00
      4       7   61789  251318   0.01
      8      15   54927  494942   0.01
     16      31   20357  399672   0.01
     32      63    6928  290701   0.01
     64     127    2956  254315   0.01
    128     255    1027  186825   0.00
    256     511    1054  402182   0.01
    512    1023    3980 3235959   0.07
   1024    2047     634  942129   0.02
   2048    4095     451 1340993   0.03
   4096    8191     267 1559353   0.04
   8192   16383     190 2332137   0.05
  16384   32767     114 2810123   0.07
  32768   65535      89 4339005   0.10
  65536  131071      46 4382290   0.10
 131072  262143      14 2596831   0.06
 262144  524287      12 4632120   0.11
 524288 1048575       8 6391289   0.15
1048576 2097151       9 14059493   0.33
2097152 4194303       8 20104228   0.47
4194304 8388607      16 103605889   2.40
8388608 16777215       1 10074277   0.23
16777216 33554431       1 20576975   0.48
33554432 67108863       2 109236674   2.53
67108864 134217727       2 212839297   4.92
134217728 268435455      21 3794702496  87.80

Now my "final" question: Is there a chance to get some/most files from
this hosed file system or am I just wasting my time[2]?

Any information I can share to help the issue?

Cheers

Carsten

[1] https://storage.microsemi.com/de-de/support/raid/sas_raid/sas-6405/
[2] The file system officially is used as "scratch" space, i.e. not
backed up. But eventually vital user data may end up there, thus the
quest of trying to restore whatever is possible.

-- 
Dr. Carsten Aulbert, Max Planck Institute for Gravitational Physics,
Callinstraße 38, 30167 Hannover, Germany
Phone: +49 511 762 17185

Re: Possible to mount this XFS at least temporarily to retrieve files?

[Stefan Ring]

On Wed, Oct 25, 2017 at 9:20 AM, Carsten Aulbert
<carsten.aulbert@aei.mpg.de> wrote:
> Hi
>
> after some hiatus, back on this list with an incident which happened
> yesterday:
>
> On a Debian Jessie machine installed back in October 2016 there a re a
> bunch of 3TB disks behind an Adaptec ASR-6405[1] in RAID6 configuration.
> Yesterday, one of the disks failed and was subsequently replace. About
> an hour into the rebuild the 28TB xfs on this block device gave up:

Did the RAID rebuild complete at least?

Re: Possible to mount this XFS at least temporarily to retrieve files?

[Carsten Aulbert]

Hi

On 10/25/17 13:32, Stefan Ring wrote:
> Did the RAID rebuild complete at least?
> 

sorry, forgot to mention. Yes, the rebuild finished and the controller
did not mention anything wrong during the rebuild - only after it now
tells us about a fan failure and a dead battery:

* Rebuild done at Tue Oct 24 18:08:57 UTC 2017

* Fan failure at Tue Oct 24 18:16:33 UTC 2017

* Battery failure at Tue Oct 24 18:16:44 UTC 2017

So yes, this controller should probably be exchanged as well.

But on the other hand, given that this was a single disk rebuild in a
RAID6 (10+2) in theory it should not impact anything working on the
block device level or above except for timeouts/slowness ;)

Cheers

Carsten

Re: Possible to mount this XFS at least temporarily to retrieve files?

[Stefan Ring]

On Wed, Oct 25, 2017 at 1:51 PM, Carsten Aulbert
<carsten.aulbert@aei.mpg.de> wrote:
> But on the other hand, given that this was a single disk rebuild in a
> RAID6 (10+2) in theory it should not impact anything working on the
> block device level or above except for timeouts/slowness ;)

True, in theory. Although it sounds suspiciously like this is
precisely what happened.

Re: Possible to mount this XFS at least temporarily to retrieve files?

[Stefan Ring]

Anyway, it would be good to know if the RAID got mixed up somehow or
not. In the former case, I would kiss the data good bye. Otherwise it
should be recoverable, although it might be time consuming.

Re: Possible to mount this XFS at least temporarily to retrieve files?

[Dave Chinner]

On Wed, Oct 25, 2017 at 09:20:03AM +0200, Carsten Aulbert wrote:
> Hi
> 
> after some hiatus, back on this list with an incident which happened
> yesterday:
> 
> On a Debian Jessie machine installed back in October 2016 there a re a
> bunch of 3TB disks behind an Adaptec ASR-6405[1] in RAID6 configuration.
> Yesterday, one of the disks failed and was subsequently replace. About
> an hour into the rebuild the 28TB xfs on this block device gave up:
> 
> Oct 24 12:39:15 atlas8 kernel: [526440.956408] XFS (sdc1):
> xfs_imap_to_bp: xfs_trans_read_buf() returned error 117.
> Oct 24 12:39:15 atlas8 kernel: [526440.956452] XFS (sdc1):
> xfs_do_force_shutdown(0x8) called from line 3242 of file
> /build/linux-byISom/linux-3.16.43/fs/xfs/xfs_inode.c.  Return address =
> 0xffffffffa02c0b76
> Oct 24 12:39:45 atlas8 kernel: [526471.029957] XFS (sdc1):
> xfs_log_force: error 5 returned.
> Oct 24 12:40:15 atlas8 kernel: [526501.154991] XFS (sdc1):
> xfs_log_force: error 5 returned.

That's a pretty good indication that th rebuild has gone
catastrophically wrong....

[....]

> Another shot in the dark was rebooting the system with a more recent
> kernel, this time 4.9.30-2+deb9u5~bpo8+1 instead of 3.16.43-2+deb8u5
> which indeed changed the behaviour of xfs_repair:
> 
> # xfs_repair /dev/sdc1
> Phase 1 - find and verify superblock...
> sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with
> calculated value 128

Which tends to indicate it found a secondary superblock in the place
of the primary superblock.....

> Phase 2 - using internal log
>         - zero log...
> Log inconsistent (didn't find previous header)
> failed to find log head
> zero_log: cannot find log head/tail (xlog_find_tail=5)

And the log isn't where it's supposed to be. 

> Some more "random" output:
> 
> # xfs_db -r -c "sb 0" -c "p" -c "freesp" /dev/sdc1

[...]

> rootino = null
> rbmino = null
> rsumino = null

These null inode pointers, and

[...]

> icount = 0
> ifree = 0
> fdblocks = 7313292427

this (inode counts zero and free blocks at 28TB) indicate we're
looking at a secondary superblock as written by mkfs.

This is a pretty good indication that the RAID rebuild has
completely jumbled up the disks and the data on the disks during
the rebuild.

> Now my "final" question: Is there a chance to get some/most files from
> this hosed file system or am I just wasting my time[2]?

It's a hardware raid controller that is having hardware problems
during a rebuild.  I'd say your filesystem is completely screwed
because the rebuild went wrong and you have no way of knowing what
blocks are good and what aren't, nor even whether the RAID has been
assembled correctly after the failure. Hence even if you could mount
it, the data in the files is likely to be corrupt/incorrect
anyway...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Possible to mount this XFS at least temporarily to retrieve files?

[Emmanuel Florac]

[-- Attachment #1: Type: text/plain, Size: 1253 bytes --]

Le Wed, 25 Oct 2017 09:20:03 +0200
Carsten Aulbert <carsten.aulbert@aei.mpg.de> écrivait:

> On a Debian Jessie machine installed back in October 2016 there a re a
> bunch of 3TB disks behind an Adaptec ASR-6405[1] in RAID6
> configuration. Yesterday, one of the disks failed and was
> subsequently replace. About an hour into the rebuild the 28TB xfs on
> this block device gave up:

This is a common problem with old adaptec RAID controllers. Don't do
any IO while it's rebuilding, else it may corrupt data. After rebuild
is complete, the volume will probably be easily repairable.

DON'T DO ANYTHING UNTIL REBUILD IS COMPLETE. Particularly don't try
running xfs_repair.

Afterwards, install the very latest firmware for the controller, go in
"Serial select" utility in BIOS and deactivate Disk write cache (this
is the individual disks write caching). It will (mostly) avoid the
problem in the future.

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

[-- Attachment #2: Signature digitale OpenPGP --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Possible to mount this XFS at least temporarily to retrieve files?

[Emmanuel Florac]

[-- Attachment #1: Type: text/plain, Size: 740 bytes --]

Le Wed, 25 Oct 2017 16:56:31 +0200
Stefan Ring <stefanrin@gmail.com> écrivait:

> Anyway, it would be good to know if the RAID got mixed up somehow or
> not. In the former case, I would kiss the data good bye. Otherwise it
> should be recoverable, although it might be time consuming.

The Adaptec RAID have a clear tendancy not to report overheating, but
nonetheless badly eating data when doing so...

-- 
------------------------------------------------------------------------
Emmanuel Florac     |   Direction technique
                    |   Intellique
                    |	<eflorac@intellique.com>
                    |   +33 1 78 94 84 02
------------------------------------------------------------------------

[-- Attachment #2: Signature digitale OpenPGP --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Possible to mount this XFS at least temporarily to retrieve files?

[Carsten Aulbert]

Hi Dave and all others who replied,

I just realized I have not answered on list to thank you all for the
analysis of his failure!

On 10/25/17 23:29, Dave Chinner wrote:
> 
> This is a pretty good indication that the RAID rebuild has
> completely jumbled up the disks and the data on the disks during
> the rebuild.

*sigh*

> It's a hardware raid controller that is having hardware problems
> during a rebuild.  I'd say your filesystem is completely screwed
> because the rebuild went wrong and you have no way of knowing what
> blocks are good and what aren't, nor even whether the RAID has been
> assembled correctly after the failure. Hence even if you could mount
> it, the data in the files is likely to be corrupt/incorrect
> anyway...

We will now replace these controllers with dumb SAS HBA and will try to
keep the "raid" features at a different level (md+xfs and/or zfs) which
hopefully will avoid this failure mode in the future.

Until then, I think we are lucky as we do not have hot spares defined on
these controllers as that would prohibit us to react properly and
eliminate host I/O at a time of our choosing...

Thanks a lot again!

Cheers

Carsten

-- 
Dr. Carsten Aulbert, Max Planck Institute for Gravitational Physics,
Callinstraße 38, 30167 Hannover, Germany
Phone: +49 511 762 17185