Missing superblock on two disks in raid5

Missing superblock on two disks in raid5

[Clemens Kiehas]

[-- Attachment #1: Type: text/plain, Size: 1902 bytes --]

Hello,

until recently I was using Unraid with 8 disks in an XFS array.
I installed Debian on a spare SSD on the same machine and started 
migrating disk by disk from the Unraid array to a raid5 array using a 
second server as temporay storage. So I switched between Debian and 
Unraid a lot to copy data and remove/add drives from/to arrays.

 From the beginning I always had to assemble the array without /dev/sdd, 
add it afterwards and let it rebuild - since the array was working fine 
afterwards I didn't really think much of it.
Appearently Unraid always overwrote the superblock of that 1 (and later 
2) disks (/dev/sdc and /dev/sdd) when I switched between the two OSs and 
now mdadm isn't recognizing those 2 disks and I can't assemble the array 
obviously.
At least that's what I think happened, since file tells me that the 
first 32k bytes are XFS:
# losetup -o 32768 -f /dev/sdd
# file -s /dev/loop0
/dev/loop0: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)

Right now mdadm only assembles 5 (instead of 7) disks as spares into an 
inactive array at boot:
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] 
[raid4] [raid10]
md0 : inactive sde[6](S) sdf1[5](S) sdg[1](S) sdh[3](S) sdi[8](S)
       21487980896 blocks super 1.2
unused devices: <none>

My system:
Linux titan 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64 
GNU/Linux
mdadm - v4.1 - 2018-10-01

Maybe I could try to assemble it with assume-clean and read-only?
I found some pages in the wiki but I'm not 100% sure that they will 
solve my problem and I don't want to make things worse.
https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID
https://raid.wiki.kernel.org/index.php/RAID_Recovery

I attached the remaining system information as textfiles in a zipfile, 
since the outputs are pretty long - i hope thats ok.

Thanks,
best regards
Clemens

[-- Attachment #2: missing_superblock_raid5.zip --]
[-- Type: application/x-zip-compressed, Size: 36472 bytes --]

Re: Missing superblock on two disks in raid5

[Wols Lists]

On 30/09/2021 18:21, Clemens Kiehas wrote:
> Hello,
> 
> until recently I was using Unraid with 8 disks in an XFS array.
> I installed Debian on a spare SSD on the same machine and started 
> migrating disk by disk from the Unraid array to a raid5 array using a 
> second server as temporay storage. So I switched between Debian and 
> Unraid a lot to copy data and remove/add drives from/to arrays.
> 
>  From the beginning I always had to assemble the array without /dev/sdd, 
> add it afterwards and let it rebuild - since the array was working fine 
> afterwards I didn't really think much of it.

Not wise - as you've found out it was trashing your redundancy ...

> Appearently Unraid always overwrote the superblock of that 1 (and later 
> 2) disks (/dev/sdc and /dev/sdd) when I switched between the two OSs and 
> now mdadm isn't recognizing those 2 disks and I can't assemble the array 
> obviously.
> At least that's what I think happened, since file tells me that the 
> first 32k bytes are XFS:
> # losetup -o 32768 -f /dev/sdd
> # file -s /dev/loop0
> /dev/loop0: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)

That feels good, but I'm not sure how to proceed from here.
> 
> Right now mdadm only assembles 5 (instead of 7) disks as spares into an 
> inactive array at boot:
> # cat /proc/mdstat
> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] 
> [raid4] [raid10]
> md0 : inactive sde[6](S) sdf1[5](S) sdg[1](S) sdh[3](S) sdi[8](S)
>        21487980896 blocks super 1.2
> unused devices: <none>
> 
> My system:
> Linux titan 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) x86_64 
> GNU/Linux
> mdadm - v4.1 - 2018-10-01
> 
> Maybe I could try to assemble it with assume-clean and read-only?
> I found some pages in the wiki but I'm not 100% sure that they will 
> solve my problem and I don't want to make things worse.
> https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID
> https://raid.wiki.kernel.org/index.php/RAID_Recovery
> 
The first page talks about overlays. THEY WILL PROTECT YOUR DATA.

Make sure every disk is overlayed, then try assembling the overlays with 
your assume-clean. You won't need read-only but do it if you like it 
then do.

If you can then mount and run a fsck or whatever it is over the file 
system and it says everything is clean, then you can redo it without the 
overlays and recover the array properly. If it's messed up, just tear 
down the overlays and you're back where you started.

If it doesn't work, come back and tell us what happened (if it does 
work, please let us know :-)

Cheers,
Wol

Re: Missing superblock on two disks in raid5

[Clemens Kiehas]

Hi Wol,

thank you.
I followed the overlay-procedure for all 7 disks and was able to start 
the array with:
mdadm --create --assume-clean --level=5 --raid-devices=7 /dev/md0 $OVERLAYS

# echo $OVERLAYS
/dev/mapper/sdd /dev/mapper/sdg /dev/mapper/sdh /dev/mapper/sdf1 
/dev/mapper/sde /dev/mapper/sdi /dev/mapper/sdc

# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] 
[raid4] [raid10]
md0 : active raid5 dm-5[6] dm-10[5] dm-8[4] dm-9[3] dm-11[2] dm-7[1] dm-6[0]
       23441316864 blocks super 1.2 level 5, 512k chunk, algorithm 2 
[7/7] [UUUUUUU]
       bitmap: 0/30 pages [0KB], 65536KB chunk

# mdadm --detail /dev/md0
/dev/md0:
            Version : 1.2
      Creation Time : Thu Sep 30 21:57:03 2021
         Raid Level : raid5
         Array Size : 23441316864 (22355.38 GiB 24003.91 GB)
      Used Dev Size : 3906886144 (3725.90 GiB 4000.65 GB)
       Raid Devices : 7
      Total Devices : 7
        Persistence : Superblock is persistent

      Intent Bitmap : Internal

        Update Time : Thu Sep 30 21:57:03 2021
              State : clean
     Active Devices : 7
    Working Devices : 7
     Failed Devices : 0
      Spare Devices : 0

             Layout : left-symmetric
         Chunk Size : 512K

Consistency Policy : bitmap

               Name : titan:0  (local to host titan)
               UUID : 45954f34:4559be76:61d18d85:f9e7ed80
             Events : 0

     Number   Major   Minor   RaidDevice State
        0     253        6        0      active sync   /dev/dm-6
        1     253        7        1      active sync   /dev/dm-7
        2     253       11        2      active sync   /dev/dm-11
        3     253        9        3      active sync   /dev/dm-9
        4     253        8        4      active sync   /dev/dm-8
        5     253       10        5      active sync   /dev/dm-10
        6     253        5        6      active sync   /dev/dm-5

I also examined all overlay-disks and they seem to have proper superblocks:
# mdadm -E /dev/mapper/sdc
/dev/mapper/sdc:
           Magic : a92b4efc
         Version : 1.2
     Feature Map : 0x1
      Array UUID : 45954f34:4559be76:61d18d85:f9e7ed80
            Name : titan:0  (local to host titan)
   Creation Time : Thu Sep 30 21:57:03 2021
      Raid Level : raid5
    Raid Devices : 7

  Avail Dev Size : 7813772976 (3725.90 GiB 4000.65 GB)
      Array Size : 23441316864 (22355.38 GiB 24003.91 GB)
   Used Dev Size : 7813772288 (3725.90 GiB 4000.65 GB)
     Data Offset : 264192 sectors
    Super Offset : 8 sectors
    Unused Space : before=264112 sectors, after=688 sectors
           State : clean
     Device UUID : 10d49408:d28fe8ee:58ce698e:a2d66ac0

Internal Bitmap : 8 sectors from superblock
     Update Time : Thu Sep 30 21:57:03 2021
   Bad Block Log : 512 entries available at offset 24 sectors
        Checksum : 68e54840 - correct
          Events : 0

          Layout : left-symmetric
      Chunk Size : 512K

    Device Role : Active device 6
    Array State : AAAAAAA ('A' == active, '.' == missing, 'R' == replacing)

But it looks like there is no data on it?
Initially I used LVM to create /dev/md0 as a physical volume, created a 
volume-group with it and a logical volume on it with all free space. 
Either the physical volume doesn't exist in this state, or I don't know 
to use it..
I tried pvs, pvscan, vgscan, dmsetup info, but it won't show up.
Do you think this is just a lvm problem now, or did I miss something?

Best regards,
Clemens

Am 30.09.2021 um 20:11 schrieb Wols Lists:
> On 30/09/2021 18:21, Clemens Kiehas wrote:
>> Hello,
>>
>> until recently I was using Unraid with 8 disks in an XFS array.
>> I installed Debian on a spare SSD on the same machine and started 
>> migrating disk by disk from the Unraid array to a raid5 array using a 
>> second server as temporay storage. So I switched between Debian and 
>> Unraid a lot to copy data and remove/add drives from/to arrays.
>>
>>  From the beginning I always had to assemble the array without 
>> /dev/sdd, add it afterwards and let it rebuild - since the array was 
>> working fine afterwards I didn't really think much of it.
>
> Not wise - as you've found out it was trashing your redundancy ...
>
>> Appearently Unraid always overwrote the superblock of that 1 (and 
>> later 2) disks (/dev/sdc and /dev/sdd) when I switched between the 
>> two OSs and now mdadm isn't recognizing those 2 disks and I can't 
>> assemble the array obviously.
>> At least that's what I think happened, since file tells me that the 
>> first 32k bytes are XFS:
>> # losetup -o 32768 -f /dev/sdd
>> # file -s /dev/loop0
>> /dev/loop0: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)
>
> That feels good, but I'm not sure how to proceed from here.
>>
>> Right now mdadm only assembles 5 (instead of 7) disks as spares into 
>> an inactive array at boot:
>> # cat /proc/mdstat
>> Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] 
>> [raid4] [raid10]
>> md0 : inactive sde[6](S) sdf1[5](S) sdg[1](S) sdh[3](S) sdi[8](S)
>>        21487980896 blocks super 1.2
>> unused devices: <none>
>>
>> My system:
>> Linux titan 5.10.0-8-amd64 #1 SMP Debian 5.10.46-4 (2021-08-03) 
>> x86_64 GNU/Linux
>> mdadm - v4.1 - 2018-10-01
>>
>> Maybe I could try to assemble it with assume-clean and read-only?
>> I found some pages in the wiki but I'm not 100% sure that they will 
>> solve my problem and I don't want to make things worse.
>> https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID
>> https://raid.wiki.kernel.org/index.php/RAID_Recovery
>>
> The first page talks about overlays. THEY WILL PROTECT YOUR DATA.
>
> Make sure every disk is overlayed, then try assembling the overlays 
> with your assume-clean. You won't need read-only but do it if you like 
> it then do.
>
> If you can then mount and run a fsck or whatever it is over the file 
> system and it says everything is clean, then you can redo it without 
> the overlays and recover the array properly. If it's messed up, just 
> tear down the overlays and you're back where you started.
>
> If it doesn't work, come back and tell us what happened (if it does 
> work, please let us know :-)
>
> Cheers,
> Wol

-- 
MatrNr: 11805432
Mobile: 06642573109
E-Mail: clemens@kiehas.at
Matrix: @clemens:matrix.spacebeard.tech