My superblocks have gone missing, can't reassemble raid5

My superblocks have gone missing, can't reassemble raid5

[Christopher Thomas]

Hi all,

I've updated my system & migrated my 3 raid5 component drives from the
old to the new, but now can't reassemble the array - mdadm just
doesn't recognize that these belong to an array at all.

The scenario:
For many years, I've run a raid5 array on a virtual Linux server
(Ubuntu 12.04) in VirtualBox on a Windows 10 host, with 3 2.7TB drives
attached to the virt in "Raw Disk" mode, and assembled into an array.
I recently upgraded to a completely different physical machine, but
still running Windows 10 and VirtualBox.  I'm reasonably sure that the
last time I shut it down, the array was clean.  Or at they very least,
the drives had superblocks.  I plugged the old drives into it,
migrated the virtual machine image to the new system, and attached
them as raw disks, just as in the old system.  And they show up as
/dev/sd[b-d], as before.  However, it's not recognized automatically
as an array at boot, and manual attempts to assemble & start the array
fail with 'no superblock'

The closest I've found online as a solution is to --create the array
again using the same parameters.  But it sounds like if I don't get
the drive order exactly the same, I'll lose the data.  Other solutions
hint at playing with the partition table, but I'm equally nervous
about that.  So I thought it was a good time to stop & ask for advice.

The details:

Here's my arrangement of disks now, where sd[bcd] are the components:

==========
chris@ursula:~$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0 20.1G  0 disk
├─sda1   8:1    0 19.2G  0 part /
├─sda2   8:2    0    1K  0 part
└─sda5   8:5    0  976M  0 part [SWAP]
sdb      8:16   0  2.7T  0 disk
└─sdb1   8:17   0  128M  0 part
sdc      8:32   0  2.7T  0 disk
└─sdc1   8:33   0  128M  0 part
sdd      8:48   0  2.7T  0 disk
└─sdd1   8:49   0  128M  0 part
sr0     11:0    1 1024M  0 rom

chris@ursula:~$ sudo /sbin/fdisk -l
Disk /dev/sda: 20.1 GiB, 21613379584 bytes, 42213632 sectors
Disk model: VBOX HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x4bbbafdf

Device     Boot    Start      End  Sectors  Size Id Type
/dev/sda1  *        2048 40212479 40210432 19.2G 83 Linux
/dev/sda2       40214526 42213375  1998850  976M  5 Extended
/dev/sda5       40214528 42213375  1998848  976M 82 Linux swap / Solaris


Disk /dev/sdb: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: VBOX HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 6489224B-FAF8-45E2-AB3D-C0D280F8E91E

Device     Start    End Sectors  Size Type
/dev/sdb1     34 262177  262144  128M Microsoft reserved


Disk /dev/sdc: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: VBOX HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 6497BDEB-A8D0-40D7-9CD2-D06018862F2B

Device     Start    End Sectors  Size Type
/dev/sdc1     34 262177  262144  128M Microsoft reserved


Disk /dev/sdd: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: VBOX HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 9DB2C3F2-F93D-4A6D-AE0E-CE28A8B8C4A2

Device     Start    End Sectors  Size Type
/dev/sdd1     34 262177  262144  128M Microsoft reserved
==========

Note: I always intended to use the whole disk, so I don't know why I
would have created a single large partition on each, and I don't
recall doing so.  But it's been a while, so I just might not be
remembering.

Here's what happens when I try to do anything with it:

===========
chris@ursula:~$ sudo /sbin/mdadm --verbose --assemble /dev/md0
/dev/sdb /dev/sdc /dev/sdd
mdadm: looking for devices for /dev/md0
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: /dev/sdb has no superblock - assembly aborted

chris@ursula:~$ sudo /sbin/mdadm --examine /dev/sd[bcd]*
/dev/sdb:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sdb1.
/dev/sdc:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sdc1.
/dev/sdd:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sdd1.
======

At some point on the old system, back when the array was still
working, I did dump the results of Examine, which looked like this:

==========
/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 36205acf:993973ba:05712a13:ff75c031
           Name : ursula:0  (local to host ursula)
  Creation Time : Fri Apr 26 23:15:04 2013
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : 8841770a:f653d990:d5db60a0:fe2e4276

    Update Time : Sun Jul  5 12:36:19 2020
       Checksum : 3a671053 - correct
         Events : 76713

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAA ('A' == active, '.' == missing)
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 36205acf:993973ba:05712a13:ff75c031
           Name : ursula:0  (local to host ursula)
  Creation Time : Fri Apr 26 23:15:04 2013
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 87fd8496:95c9cd5e:5caaa28a:25f6ab04

    Update Time : Sat May 30 02:02:45 2020
       Checksum : ce4cd20 - correct
         Events : 76711

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAA ('A' == active, '.' == missing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 36205acf:993973ba:05712a13:ff75c031
           Name : ursula:0  (local to host ursula)
  Creation Time : Fri Apr 26 23:15:04 2013
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
     Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
          State : active
    Device UUID : c796e484:b4ed6813:a97e0ce9:66a56758

    Update Time : Sun Jul  5 12:36:19 2020
       Checksum : 6235188e - correct
         Events : 76713

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAA ('A' == active, '.' == missing)
==========

Thank you for any ideas or guidance you can offer.
-Chris

Re: My superblocks have gone missing, can't reassemble raid5

[Christopher Thomas]

I should say that all of my current diagnostic output is actually from
a virt running Debian 10 (Kernel 4.19.0), not the original Ubuntu
12.04.  But the behavior is practically the same in both, and I'd
rather get it working on a modern system anyway.

-Chris

On Sun, May 16, 2021 at 9:16 PM Christopher Thomas <youkai@earthlink.net> wrote:
>
> Hi all,
>
> I've updated my system & migrated my 3 raid5 component drives from the
> old to the new, but now can't reassemble the array - mdadm just
> doesn't recognize that these belong to an array at all.
>
> The scenario:
> For many years, I've run a raid5 array on a virtual Linux server
> (Ubuntu 12.04) in VirtualBox on a Windows 10 host, with 3 2.7TB drives
> attached to the virt in "Raw Disk" mode, and assembled into an array.
> I recently upgraded to a completely different physical machine, but
> still running Windows 10 and VirtualBox.  I'm reasonably sure that the
> last time I shut it down, the array was clean.  Or at they very least,
> the drives had superblocks.  I plugged the old drives into it,
> migrated the virtual machine image to the new system, and attached
> them as raw disks, just as in the old system.  And they show up as
> /dev/sd[b-d], as before.  However, it's not recognized automatically
> as an array at boot, and manual attempts to assemble & start the array
> fail with 'no superblock'
>
> The closest I've found online as a solution is to --create the array
> again using the same parameters.  But it sounds like if I don't get
> the drive order exactly the same, I'll lose the data.  Other solutions
> hint at playing with the partition table, but I'm equally nervous
> about that.  So I thought it was a good time to stop & ask for advice.
>
> The details:
>
> Here's my arrangement of disks now, where sd[bcd] are the components:
>
> ==========
> chris@ursula:~$ lsblk
> NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
> sda      8:0    0 20.1G  0 disk
> ├─sda1   8:1    0 19.2G  0 part /
> ├─sda2   8:2    0    1K  0 part
> └─sda5   8:5    0  976M  0 part [SWAP]
> sdb      8:16   0  2.7T  0 disk
> └─sdb1   8:17   0  128M  0 part
> sdc      8:32   0  2.7T  0 disk
> └─sdc1   8:33   0  128M  0 part
> sdd      8:48   0  2.7T  0 disk
> └─sdd1   8:49   0  128M  0 part
> sr0     11:0    1 1024M  0 rom
>
> chris@ursula:~$ sudo /sbin/fdisk -l
> Disk /dev/sda: 20.1 GiB, 21613379584 bytes, 42213632 sectors
> Disk model: VBOX HARDDISK
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: dos
> Disk identifier: 0x4bbbafdf
>
> Device     Boot    Start      End  Sectors  Size Id Type
> /dev/sda1  *        2048 40212479 40210432 19.2G 83 Linux
> /dev/sda2       40214526 42213375  1998850  976M  5 Extended
> /dev/sda5       40214528 42213375  1998848  976M 82 Linux swap / Solaris
>
>
> Disk /dev/sdb: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
> Disk model: VBOX HARDDISK
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: gpt
> Disk identifier: 6489224B-FAF8-45E2-AB3D-C0D280F8E91E
>
> Device     Start    End Sectors  Size Type
> /dev/sdb1     34 262177  262144  128M Microsoft reserved
>
>
> Disk /dev/sdc: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
> Disk model: VBOX HARDDISK
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: gpt
> Disk identifier: 6497BDEB-A8D0-40D7-9CD2-D06018862F2B
>
> Device     Start    End Sectors  Size Type
> /dev/sdc1     34 262177  262144  128M Microsoft reserved
>
>
> Disk /dev/sdd: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
> Disk model: VBOX HARDDISK
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: gpt
> Disk identifier: 9DB2C3F2-F93D-4A6D-AE0E-CE28A8B8C4A2
>
> Device     Start    End Sectors  Size Type
> /dev/sdd1     34 262177  262144  128M Microsoft reserved
> ==========
>
> Note: I always intended to use the whole disk, so I don't know why I
> would have created a single large partition on each, and I don't
> recall doing so.  But it's been a while, so I just might not be
> remembering.
>
> Here's what happens when I try to do anything with it:
>
> ===========
> chris@ursula:~$ sudo /sbin/mdadm --verbose --assemble /dev/md0
> /dev/sdb /dev/sdc /dev/sdd
> mdadm: looking for devices for /dev/md0
> mdadm: Cannot assemble mbr metadata on /dev/sdb
> mdadm: /dev/sdb has no superblock - assembly aborted
>
> chris@ursula:~$ sudo /sbin/mdadm --examine /dev/sd[bcd]*
> /dev/sdb:
>    MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> mdadm: No md superblock detected on /dev/sdb1.
> /dev/sdc:
>    MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> mdadm: No md superblock detected on /dev/sdc1.
> /dev/sdd:
>    MBR Magic : aa55
> Partition[0] :   4294967295 sectors at            1 (type ee)
> mdadm: No md superblock detected on /dev/sdd1.
> ======
>
> At some point on the old system, back when the array was still
> working, I did dump the results of Examine, which looked like this:
>
> ==========
> /dev/sdb:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : 36205acf:993973ba:05712a13:ff75c031
>            Name : ursula:0  (local to host ursula)
>   Creation Time : Fri Apr 26 23:15:04 2013
>      Raid Level : raid5
>    Raid Devices : 3
>
>  Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
>      Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
>   Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
>     Data Offset : 262144 sectors
>    Super Offset : 8 sectors
>           State : active
>     Device UUID : 8841770a:f653d990:d5db60a0:fe2e4276
>
>     Update Time : Sun Jul  5 12:36:19 2020
>        Checksum : 3a671053 - correct
>          Events : 76713
>
>          Layout : left-symmetric
>      Chunk Size : 512K
>
>    Device Role : Active device 2
>    Array State : AAA ('A' == active, '.' == missing)
> /dev/sdc:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : 36205acf:993973ba:05712a13:ff75c031
>            Name : ursula:0  (local to host ursula)
>   Creation Time : Fri Apr 26 23:15:04 2013
>      Raid Level : raid5
>    Raid Devices : 3
>
>  Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
>      Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
>   Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
>     Data Offset : 262144 sectors
>    Super Offset : 8 sectors
>           State : clean
>     Device UUID : 87fd8496:95c9cd5e:5caaa28a:25f6ab04
>
>     Update Time : Sat May 30 02:02:45 2020
>        Checksum : ce4cd20 - correct
>          Events : 76711
>
>          Layout : left-symmetric
>      Chunk Size : 512K
>
>    Device Role : Active device 0
>    Array State : AAA ('A' == active, '.' == missing)
> /dev/sdd:
>           Magic : a92b4efc
>         Version : 1.2
>     Feature Map : 0x0
>      Array UUID : 36205acf:993973ba:05712a13:ff75c031
>            Name : ursula:0  (local to host ursula)
>   Creation Time : Fri Apr 26 23:15:04 2013
>      Raid Level : raid5
>    Raid Devices : 3
>
>  Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
>      Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
>   Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
>     Data Offset : 262144 sectors
>    Super Offset : 8 sectors
>           State : active
>     Device UUID : c796e484:b4ed6813:a97e0ce9:66a56758
>
>     Update Time : Sun Jul  5 12:36:19 2020
>        Checksum : 6235188e - correct
>          Events : 76713
>
>          Layout : left-symmetric
>      Chunk Size : 512K
>
>    Device Role : Active device 1
>    Array State : AAA ('A' == active, '.' == missing)
> ==========
>
> Thank you for any ideas or guidance you can offer.
> -Chris

Re: My superblocks have gone missing, can't reassemble raid5

[Roman Mamedov]

On Sun, 16 May 2021 21:16:22 -0700
Christopher Thomas <youkai@earthlink.net> wrote:

> Disk /dev/sdd: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
> Disk model: VBOX HARDDISK
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: gpt
> Disk identifier: 9DB2C3F2-F93D-4A6D-AE0E-CE28A8B8C4A2
> 
> Device     Start    End Sectors  Size Type
> /dev/sdd1     34 262177  262144  128M Microsoft reserved
> ==========
> 
> Note: I always intended to use the whole disk, so I don't know why I
> would have created a single large partition on each, and I don't
> recall doing so.  But it's been a while, so I just might not be
> remembering.

Unless you mean "why I *wouldn't* have created"... There isn't a single large
partition on each, the partitions are only 128 MB in size, and these are the
Microsoft Reserved partitions (MSR)[1]. And a GPT partition table, which
shouldn't be there either, if you used whole disks for md RAID. I guess either
of these have overwritten your superblocks, which with version 1.2 are stored
4k from the beginning of the device.
 
I'm not sure if Windows would just quietly create these on it's on, but it
certainly would if the user opened "Disk Manager", was prompted that there are
3 new uninitialized drives and clicked "OK" to initialize them as GPT.

For that exact reason it is not a good idea to use whole disks for RAID, since
other OSes see them as empty/uninitialized which they can use as they see fit.

As for recovery, you really might need to play with --create; to prevent the
chance of data loss, there's a way to experiment using "overlays", keeping the
original drives untouched; see [2] for more background on that, and [3]
provides steps for your actual situation. Don't forget to use "--assume-clean"
and "--readonly".

There also might be some filesystem damage on the array, depending on how much
and how important were the pieces overwritten at the beginning of all disks by
data stored in the 128 MB MSR.

[1]
https://superuser.com/questions/942065/is-msr-partition-required-on-secondary-gpt-drives

[2] https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID
[3] https://raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recovery

-- 
With respect,
Roman

Re: My superblocks have gone missing, can't reassemble raid5

[Wols Lists]

On 17/05/21 07:28, Roman Mamedov wrote:
> As for recovery, you really might need to play with --create; to prevent the
> chance of data loss, there's a way to experiment using "overlays", keeping the
> original drives untouched; see [2] for more background on that, and [3]
> provides steps for your actual situation. Don't forget to use "--assume-clean"
> and "--readonly".

Firstly, what I'd do is a hexdump of the fifth kb of each disk (ie
hopefully where the superblock is/was). Post that here and see if anyone
can decode it. It does look like something has created a gpt, so mdadm
is no longer looking at the raw disk for the array.

There MIGHT be enough information lying around for someone to tell you
what --create command to use. Another thing - are those the original
disks used to create the array? Have you modified the array in any way?
If you haven't modified the array, then a plain create using the same
version of mdadm that created the array should just put things back the
way they were. The reason you mustn't just use --create willy nilly is
that any modification of the array might move the data offset, and
different versions of mdadm may have different default settings.

The other option is BACK UP YOUR HARD DRIVES, delete the new gpt, and
then see if it will assemble.

I know I've seen this before, but I've a nasty feeling the previous time
the user had used partitions and the GPT had been trashed so it's not
quite the same. Still, you might well be lucky ...

Cheers,
Wol

Re: My superblocks have gone missing, can't reassemble raid5

[Roman Mamedov]

On Mon, 17 May 2021 05:36:42 -0500
Roger Heflin <rogerheflin@gmail.com> wrote:

> When I look at my 1.2 block, the mdraid header appears to start at 4k, and
> the gpt partition table starts at 0x0000 and ends before 4k.
> 
> He may be able to simply delete the partition and have it work.

Christopher wrote that he tried:

chris@ursula:~$ sudo /sbin/mdadm --verbose --assemble /dev/md0
/dev/sdb /dev/sdc /dev/sdd
mdadm: looking for devices for /dev/md0
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: /dev/sdb has no superblock - assembly aborted

I would have expected mdadm when passed entire devices (not partitions) to not
even look if there are any partitions, and directly proceed to checking if
there's a superblock at its supposed location. But maybe indeed, from the
messages it looks like it bails before that, on seeing "mbr metadata", i.e.
the enclosing MBR partition table of GPT.

-- 
With respect,
Roman

Re: My superblocks have gone missing, can't reassemble raid5

[Phil Turmel]

On 5/17/21 9:19 AM, Roman Mamedov wrote:
> On Mon, 17 May 2021 05:36:42 -0500
> Roger Heflin <rogerheflin@gmail.com> wrote:
> 
>> When I look at my 1.2 block, the mdraid header appears to start at 4k, and
>> the gpt partition table starts at 0x0000 and ends before 4k.
>>
>> He may be able to simply delete the partition and have it work.
> 
> Christopher wrote that he tried:
> 
> chris@ursula:~$ sudo /sbin/mdadm --verbose --assemble /dev/md0
> /dev/sdb /dev/sdc /dev/sdd
> mdadm: looking for devices for /dev/md0
> mdadm: Cannot assemble mbr metadata on /dev/sdb
> mdadm: /dev/sdb has no superblock - assembly aborted
> 
> I would have expected mdadm when passed entire devices (not partitions) to not
> even look if there are any partitions, and directly proceed to checking if
> there's a superblock at its supposed location. But maybe indeed, from the
> messages it looks like it bails before that, on seeing "mbr metadata", i.e.
> the enclosing MBR partition table of GPT.
> 

The Microsoft system partition starts on top of the location for v1.2 
metadata.

Just another reason to *never* use bare drives.

Sorry, Christopher.  You will have to experiment with --create.  Use 
overlays.

Phil

Re: My superblocks have gone missing, can't reassemble raid5

[Reindl Harald]

Am 18.05.21 um 19:47 schrieb Phil Turmel:
> On 5/17/21 9:19 AM, Roman Mamedov wrote:
>> On Mon, 17 May 2021 05:36:42 -0500
>> Roger Heflin <rogerheflin@gmail.com> wrote:
>>
>>> When I look at my 1.2 block, the mdraid header appears to start at 
>>> 4k, and
>>> the gpt partition table starts at 0x0000 and ends before 4k.
>>>
>>> He may be able to simply delete the partition and have it work.
>>
>> Christopher wrote that he tried:
>>
>> chris@ursula:~$ sudo /sbin/mdadm --verbose --assemble /dev/md0
>> /dev/sdb /dev/sdc /dev/sdd
>> mdadm: looking for devices for /dev/md0
>> mdadm: Cannot assemble mbr metadata on /dev/sdb
>> mdadm: /dev/sdb has no superblock - assembly aborted
>>
>> I would have expected mdadm when passed entire devices (not 
>> partitions) to not
>> even look if there are any partitions, and directly proceed to 
>> checking if
>> there's a superblock at its supposed location. But maybe indeed, from the
>> messages it looks like it bails before that, on seeing "mbr metadata", 
>> i.e.
>> the enclosing MBR partition table of GPT.
>>
> 
> The Microsoft system partition starts on top of the location for v1.2 
> metadata.
> 
> Just another reason to *never* use bare drives

the most important is that you have no guarantee that a replacement 
drive years later is 100% identical in size

leave some margin and padding around the used space solves that problem 
entirely and i still need to hear a single valid reason for using 
unpartitioned drives in a RAID

Re: My superblocks have gone missing, can't reassemble raid5

[Leslie Rhorer]

On 5/18/2021 1:31 PM, Reindl Harald wrote:
> 
> 
> Am 18.05.21 um 19:47 schrieb Phil Turmel:
>> On 5/17/21 9:19 AM, Roman Mamedov wrote:
>>> On Mon, 17 May 2021 05:36:42 -0500
>>> Roger Heflin <rogerheflin@gmail.com> wrote:
>>>
>>>> When I look at my 1.2 block, the mdraid header appears to start at 
>>>> 4k, and
>>>> the gpt partition table starts at 0x0000 and ends before 4k.
>>>>
>>>> He may be able to simply delete the partition and have it work.
>>>
>>> Christopher wrote that he tried:
>>>
>>> chris@ursula:~$ sudo /sbin/mdadm --verbose --assemble /dev/md0
>>> /dev/sdb /dev/sdc /dev/sdd
>>> mdadm: looking for devices for /dev/md0
>>> mdadm: Cannot assemble mbr metadata on /dev/sdb
>>> mdadm: /dev/sdb has no superblock - assembly aborted
>>>
>>> I would have expected mdadm when passed entire devices (not 
>>> partitions) to not
>>> even look if there are any partitions, and directly proceed to 
>>> checking if
>>> there's a superblock at its supposed location. But maybe indeed, from 
>>> the
>>> messages it looks like it bails before that, on seeing "mbr 
>>> metadata", i.e.
>>> the enclosing MBR partition table of GPT.
>>>
>>
>> The Microsoft system partition starts on top of the location for v1.2 
>> metadata.
>>
>> Just another reason to *never* use bare drives
> 
> the most important is that you have no guarantee that a replacement 
> drive years later is 100% identical in size
> 
> leave some margin and padding around the used space solves that problem 
> entirely and i still need to hear a single valid reason for using 
> unpartitioned drives in a RAID

	I can give you about a dozen.  We will start with this:

1. Partitioning is not necessary.  Doing something that is not necessary 
is not usually worthwhile.

2. Partitioning offers no advantages.  Doing something unnecessary is 
questionable.  Doing something that has no purpose at all is downright 
foolish.

3. Partitioning introduces an additional layer of activity.  This makes 
it both more complex and more wasteful of resources.  And yes, before 
you bring it up, the additional complexity and resource infringement are 
quite small.  They are not zero, however, and they are in essence 
continuous.  Every little bit counts.

4. There is no guarantee the partitioning that works today will work 
tomorrow.  It should, of course, and it probably will, but why take a 
risk when there is absolutely no gain whatsoever?

5. It is additional work that ultimately yields no positive result 
whatsoever.  Admittedly, partitioning one disk is not a lot of work. 
Partitioning 50 disks is another matter.  Partitioning 500 disks...

6. Partitioning has an intent.  That intent is of no relevance 
whatsoever on a device whose content is singular in scope.  Are you 
suggesting we should also partition tapes?  Ralph Waldo Emerson had 
something important to say about repeatedly doing things simply because 
they have been done before and elsewhere.

7. There is no downside to forfeiting the partition table.  Not needing 
to do something is an extremely good reason for not doing it.  This is 
of course a corollary to point #1.

Re: My superblocks have gone missing, can't reassemble raid5

[Phil Turmel]

On 5/19/21 9:20 AM, Leslie Rhorer wrote:
> 
> 
> On 5/18/2021 1:31 PM, Reindl Harald wrote:

[trim/]

>> leave some margin and padding around the used space solves that 
>> problem entirely and i still need to hear a single valid reason for 
>> using unpartitioned drives in a RAID
> 
>      I can give you about a dozen.  We will start with this:
> 
> 1. Partitioning is not necessary.  Doing something that is not necessary 
> is not usually worthwhile.

1a: sure.  1b:  I can think of many things that aren't *necessary* but 
are certainly worthwhile.  I can even throw a few out there, like 
personal hygiene, healthy diets, exercise.  In this context, I would 
list drive smart monitoring, weekly scrubs, and sysadmins with a clue.

> 2. Partitioning offers no advantages.  Doing something unnecessary is 
> questionable.  Doing something that has no purpose at all is downright 
> foolish.

Who says it has no purpose.  Its purpose is to segment a device into 
regions with associated metadata.

> 3. Partitioning introduces an additional layer of activity.  This makes 
> it both more complex and more wasteful of resources.  And yes, before 
> you bring it up, the additional complexity and resource infringement are 
> quite small.  They are not zero, however, and they are in essence 
> continuous.  Every little bit counts.

Hmm.  A sector offset and limit check, buried deep in the kernel's 
common code.  I dare you to measure the incremental impact.

> 4. There is no guarantee the partitioning that works today will work 
> tomorrow.  It should, of course, and it probably will, but why take a 
> risk when there is absolutely no gain whatsoever?

You assert "no gain", but you provide no support for your assertion.

> 5. It is additional work that ultimately yields no positive result 
> whatsoever.  Admittedly, partitioning one disk is not a lot of work. 
> Partitioning 50 disks is another matter.  Partitioning 500 disks...

You assert "no positive result whatsoever".  Sounds like #4.  With 
similar lack of support.  Fluffing up your list, much?

> 6. Partitioning has an intent.  That intent is of no relevance 
> whatsoever on a device whose content is singular in scope.  Are you 
> suggesting we should also partition tapes?  Ralph Waldo Emerson had 
> something important to say about repeatedly doing things simply because 
> they have been done before and elsewhere.

No relevance?  Metadata can be rather useful when operating systems 
encounter what looks like an empty disk.  Metadata that says "not 
empty!"  Especially valuable when metadata is *recognized* by all 
operating systems.  You know, like, a *standard*.  While MDraid places 
metadata on the devices it uses, only Linux *recognizes* it.

> 7. There is no downside to forfeiting the partition table.  Not needing 
> to do something is an extremely good reason for not doing it.  This is 
> of course a corollary to point #1.

Just more fluff.

Microsoft and a number of NAS products are known to corrupt drives with 
no partition table.  I vaguely recall hardware raid doing it, too. 
That's a damn good reason to add a tiny (measurable?) amount of overhead.

And dude, making a single partition on a disk can be /scripted/.  Might 
want to learn about that, if the pain of the driving fdisk/gdisk 
occasionally is too much for your delicate fingers.

Phil

Re: My superblocks have gone missing, can't reassemble raid5

[Leslie Rhorer]

On 5/19/2021 8:41 AM, Phil Turmel wrote:
> On 5/19/21 9:20 AM, Leslie Rhorer wrote:
>>
>>
>> On 5/18/2021 1:31 PM, Reindl Harald wrote:
> 
> [trim/]
> 
>>> leave some margin and padding around the used space solves that 
>>> problem entirely and i still need to hear a single valid reason for 
>>> using unpartitioned drives in a RAID
>>
>>      I can give you about a dozen.  We will start with this:
>>
>> 1. Partitioning is not necessary.  Doing something that is not 
>> necessary is not usually worthwhile.
> 
> 1a: sure.  1b:  I can think of many things that aren't *necessary* but 
> are certainly worthwhile.  I can even throw a few out there, like 
> personal hygiene, healthy diets, exercise.  In this context, I would 
> list drive smart monitoring, weekly scrubs, and sysadmins with a clue.

	Every one of those things are not only necessary but absolutely 
essential.  The consequences of failing in those areas can be absolutely 
devastating.  What's more, I did not say, "...Never".  I very 
specifically said , "...Not usually".  There was a reason I did that.

>> 2. Partitioning offers no advantages.  Doing something unnecessary is 
>> questionable.  Doing something that has no purpose at all is downright 
>> foolish.
> 
> Who says it has no purpose.  Its purpose is to segment a device into 
> regions with associated metadata.

	Context, please.  If there is only one region with data then there is 
no reason for any association with or existence of any metadata.  A 
system containing only one segment needs no internal identification of 
any sort.  A universe requires no names or labels of any sort.

	A system consisting of multiple segments is a very different thing.  In 
any such system, partitioning or something similar is essential.  That 
is why my main servers all have partitioned OS arrays and 
non-partitioned data arrays.


>> 3. Partitioning introduces an additional layer of activity.  This 
>> makes it both more complex and more wasteful of resources.  And yes, 
>> before you bring it up, the additional complexity and resource 
>> infringement are quite small.  They are not zero, however, and they 
>> are in essence continuous.  Every little bit counts.
> 
> Hmm.  A sector offset and limit check, buried deep in the kernel's 
> common code.  I dare you to measure the incremental impact.

	You are assuming, there.  The kernel can most certainly be compiled 
without that code.  In fact, I have done so on embedded systems.  I 
could measure the impact, if it were an issue for me.  It isn't.  The 
simple fact is every bit of code takes time to run.  I don't need a 
quantitative measure to confirm that.  I also know the run time of the 
code is only a few msec for the first run and a few nanosec for 
subsequent runs, assuming the blocks are kept in cache somewhere.  That 
is indeed minimal.

>> 4. There is no guarantee the partitioning that works today will work 
>> tomorrow.  It should, of course, and it probably will, but why take a 
>> risk when there is absolutely no gain whatsoever?
> 
> You assert "no gain", but you provide no support for your assertion.

	That is because it is not possible to provide support for nothing.  One 
cannot prove non-existence.  If you have support for the notion there is 
some sort of gain, then by all means provide it.  Short of that, there 
is no evidence there is some gain to be had, and my statement stands. 
Believe me, my feelings are not going to be hurt by being proved wrong.

>> 5. It is additional work that ultimately yields no positive result 
>> whatsoever.  Admittedly, partitioning one disk is not a lot of work. 
>> Partitioning 50 disks is another matter.  Partitioning 500 disks...
> 
> You assert "no positive result whatsoever".  Sounds like #4.  With 
> similar lack of support.  Fluffing up your list, much?

	No, it is the cost side of the argument, which is not the same as the 
consequential side.  They are the two sides of the cost / benefit 
analysis.  In #4, I assert the benefit is negligible, perhaps even zero. 
  You haven't bothered to refute this, by the way.  In this point, I 
highlight the fact there is some cost to the practice.  Again, you have 
not refuted this.  An ad hominem reference to some alleged procedural 
failure on my part does not support your position.  Note I would be 
happy for you to do so, but don't think criticizing my abilities, 
whether accurate or not, in no way supports your position.

>> 6. Partitioning has an intent.  That intent is of no relevance 
>> whatsoever on a device whose content is singular in scope.  Are you 
>> suggesting we should also partition tapes?  Ralph Waldo Emerson had 
>> something important to say about repeatedly doing things simply 
>> because they have been done before and elsewhere.
> 
> No relevance?  Metadata can be rather useful when operating systems 
> encounter what looks like an empty disk.

	I am afraid you are going to have to be more specific for me to concede 
this point.  I am having trouble coming up with a good example.

> Metadata that says "not 
> empty!"  Especially valuable when metadata is *recognized* by all 
> operating systems.  You know, like, a *standard*.

	Please name one such case.  I cannot think of even a single protocol or 
standard that is universally recognized.  Some are close, but then only 
those that are very, very old.

> While MDraid places 
> metadata on the devices it uses, only Linux *recognizes* it.

	The logical extension of this is no system, anywhere, should ever use 
RAID of any flavor.  Nor indeed, any file system.  Nor, for that matter, 
any file system.  Indeed, we should never use any hard drive, and 
definitely not any tape drives.

>> 7. There is no downside to forfeiting the partition table.  Not 
>> needing to do something is an extremely good reason for not doing it.  
>> This is of course a corollary to point #1.
> 
> Just more fluff.

	"Just" more ad hominem.  Going down a short side path for a moment, the 
term "just" is almost always an illegitimate attempt to belittle and 
avoid an idea with the intent of being dismissive to an opponent in a 
debate.  All it really does is highlight the speaker's inability to 
properly support the speaker's side of the debate.  I would request you 
please stop "justiing" and instead provide some real support for your 
argument.  It doesn't greatly matter to me who wins this argument, but 
having to reply to unsupported rhetoric wastes my time.

> Microsoft and a number of NAS products are known to corrupt drives with 
> no partition table.

	Which is a pretty good argument to avoid those products, if true. 
Actually, I can't speak to any particular NAS (most of which are Linux 
based, and so quite unlikely to behave in such a manner), but while MS 
was accused of such activity in the past.  It was not true up to and 
including Windows 7.  I don't know about Windows 10, but I doubt it. 
Notwithstanding, I still avoid Windows whenever possible.  I certainly 
avoid doing anything merely to accommodate Windows.

> I vaguely recall hardware raid doing it, too. 
> That's a damn good reason to add a tiny (measurable?) amount of overhead.

	No, it's a damn good reason to completely avoid any such products, for 
reasons far deeper than just mucking around with partition tables. 
Mucking around the system without the express intent and direction of 
the system supervisor is not acceptable under any circumstances.  It's 
also a damn good reason why the supervisor needs to know what they are 
doing.

> And dude, making a single partition on a disk can be /scripted/.  Might 
> want to learn about that, if the pain of the driving fdisk/gdisk 
> occasionally is too much for your delicate fingers.

	It's not too much.  It's just unnecessary.  So is typing <scriptname> 
when it is not necessary.  The fact it can be scripted is irrelevant. 
You are free to do whatever you like.  I am not going to try to stop 
you.  You asked for reasons why one may chose not to partition RAID 
members.  I gave you some.  Accept them or reject them.  I don't really 
care.  Show me where my statements were factually incorrect.  I am 
always happy to learn of my mistakes.

	By the way, I don't partition non-RAID drives, either, unless they 
require multiple segments.  Not all of the media have any file systems, 
either.

Re: My superblocks have gone missing, can't reassemble raid5

[Nix]

On 19 May 2021, Phil Turmel spake thusly:

> weekly scrubs

*Weekly*? Scrubbing my arrays takes three or four days. If I ran them
weekly the machine would never have time to do anything else!

(I run them every couple of months. Doing them more often than that
feels too much like the machine's only job is to scrub itself :)
obviously I have good backups too. The scrubs have never spotted
anything at fault in ten years or so of scrubbing at more or less this
frequency -- more often in the past, when disks were smaller and scrubs
took less time.)

If the storage machinery on this system is so badly off that it's
misreading bits more often than that I think I have bigger problems.

Re: My superblocks have gone missing, can't reassemble raid5

[Leslie Rhorer]

On 5/20/2021 2:37 PM, Nix wrote:
> On 19 May 2021, Phil Turmel spake thusly:
> 
>> weekly scrubs
> 
> *Weekly*? Scrubbing my arrays takes three or four days. If I ran them
> weekly the machine would never have time to do anything else!

	Three or four days?  How large are your arrays?  I have quite a few 
arrays up to 48T (64T of spindles), and none of them take more than 12 
hours to scrub.

> If the storage machinery on this system is so badly off that it's
> misreading bits more often than that I think I have bigger problems.

	That might be the case already.

Re: My superblocks have gone missing, can't reassemble raid5

[Andy Smith]

Hello,

On Wed, May 19, 2021 at 08:20:19AM -0500, Leslie Rhorer wrote:
> 1. Partitioning is not necessary.  Doing something that is not necessary is
> not usually worthwhile.

I'm with you in that I prefer not to use partitions when I don't
need to.

However, there are many reports of people suffering corrupted disks
that ended up being due to their motherboard helpfully deciding that
if it sees a disk with no MBR nor GPT then it should create its own
at every boot.

    http://forum.asrock.com/forum_posts.asp?TID=10174&title=asrock-motherboard-destroys-linux-software-raid

So, I'm not going to rail against anyone who prefer to always
partition.

Cheers,
Andy

Re: My superblocks have gone missing, can't reassemble raid5

[Leslie Rhorer]

On 5/19/2021 9:20 AM, Andy Smith wrote:
> Hello,
> 
> On Wed, May 19, 2021 at 08:20:19AM -0500, Leslie Rhorer wrote:
>> 1. Partitioning is not necessary.  Doing something that is not necessary is
>> not usually worthwhile.
> 
> I'm with you in that I prefer not to use partitions when I don't
> need to.
> 
> However, there are many reports of people suffering corrupted disks
> that ended up being due to their motherboard helpfully deciding that
> if it sees a disk with no MBR nor GPT then it should create its own
> at every boot.
> 
>      http://forum.asrock.com/forum_posts.asp?TID=10174&title=asrock-motherboard-destroys-linux-software-raid

	So to paraphrase the other fellow, "That is another reason not to use 
ASRock motherboards."  Indeed, it is IMO a far better reason not to 
support ASRock than it is to spuriously employ partitions.  Of copurse 
it is also just another reason to avoid Microsoft like the plague.

> So, I'm not going to rail against anyone who prefer to always
> partition.

	I am not going to rail against anyone.  They are their systems; they 
can do as they please with them.  Indeed, it is they are who are 
suggesting I should do something, not the other way around.  I suggest 
not doing it.  Honestly, it is not a huge deal, either way.

Re: My superblocks have gone missing, can't reassemble raid5

[Leslie Rhorer]

On 5/16/2021 11:16 PM, Christopher Thomas wrote:
> Hi all,
> 
> I've updated my system & migrated my 3 raid5 component drives from the
> old to the new, but now can't reassemble the array - mdadm just
> doesn't recognize that these belong to an array at all.
> 
> The scenario:
> For many years, I've run a raid5 array on a virtual Linux server
> (Ubuntu 12.04) in VirtualBox on a Windows 10 host, with 3 2.7TB drives
> attached to the virt in "Raw Disk" mode, and assembled into an array.
> I recently upgraded to a completely different physical machine, but
> still running Windows 10 and VirtualBox.  I'm reasonably sure that the
> last time I shut it down, the array was clean.  Or at they very least,
> the drives had superblocks.  I plugged the old drives into it,
> migrated the virtual machine image to the new system, and attached
> them as raw disks, just as in the old system.  And they show up as
> /dev/sd[b-d], as before.  However, it's not recognized automatically
> as an array at boot, and manual attempts to assemble & start the array
> fail with 'no superblock'
> 
> The closest I've found online as a solution is to --create the array
> again using the same parameters.  But it sounds like if I don't get
> the drive order exactly the same, I'll lose the data.  Other solutions
> hint at playing with the partition table, but I'm equally nervous
> about that.  So I thought it was a good time to stop & ask for advice.
> 
> The details:
> 
> Here's my arrangement of disks now, where sd[bcd] are the components:
> 
> ==========
> chris@ursula:~$ lsblk
> NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
> sda      8:0    0 20.1G  0 disk
> ├─sda1   8:1    0 19.2G  0 part /
> ├─sda2   8:2    0    1K  0 part
> └─sda5   8:5    0  976M  0 part [SWAP]
> sdb      8:16   0  2.7T  0 disk
> └─sdb1   8:17   0  128M  0 part
> sdc      8:32   0  2.7T  0 disk
> └─sdc1   8:33   0  128M  0 part
> sdd      8:48   0  2.7T  0 disk
> └─sdd1   8:49   0  128M  0 part
> sr0     11:0    1 1024M  0 rom

	The first thing you need to do is copy those drives into safety media. 
  I know this means a new drive, but an 8T drive is not that expensive. 
  I would format the drive and mount it to some directory:

mkfs /dev/sdX
mkdir /Safety
mount /dev/sdX /Safety
cd /Safety
ddrescue /dev/sdb disk1 /Safety/disk1-map
ddrescue /dev/sdc disk2 /Safety/disk2-map
ddrescue /dev/sdd disk3 /Safety/disk3-map

	As mentioned earlier in this thread, you can create overlays of the 
original disks.  That, or you can make loops of the backup files. 
Actually, if it were me, I would create two sets of backups and work on 
the second set, but then I am hyper-anal about such things.  I don't 
just employ a belt and suspenders.  I use a belt, suspenders, large 
numbers of staples, super glue, and a braided steel strap welded in 
place.  Use whatever level of redundancy with which you feel 
comfortable, but I *DEFINITELY* do not recommend working with the 
original media.  Indeed, I do not recommend attempting to recover the 
original media, at all.  Once you have a solution identified, I would 
employ new drives, keeping the old as backups.  (Which rather begs the 
question, "Why don't you have backups of the data, so you could simply 
create an entirely new empty array and copy the data to the new array 
from the backup?")

	I would then create a loopfile from each backup file:

losetup -fP disk1
losetup -fP disk2
losetup -fP disk3

	Then I would try recreating the RAID based upon the earlier Examine report:

mdadm -C -f -n 3 -l 5 -e 1.2 -c 512 -p ls /dev/md99 /dev/loop2 
/dev/loop0 /dev/loop1

	You may notice some of the command switches are defaults.  Remember 
what I said about a belt and suspenders?  Personally, in such a case I 
would not rely on defaults.

	Now try running a check on the assembled array:
fsck /dev/md99

	If that fails, shutdown the array with

mdadm -S /dev/md99

	and then try creating the array with a different drive order.  There 
are only two other possible permutations of three disks.  If none of 
those work, you have some more serious problems.

Re: My superblocks have gone missing, can't reassemble raid5

[antlists]

On 19/05/2021 15:48, Leslie Rhorer wrote:
>      Then I would try recreating the RAID based upon the earlier Examine 
> report:
> 
> mdadm -C -f -n 3 -l 5 -e 1.2 -c 512 -p ls /dev/md99 /dev/loop2 
> /dev/loop0 /dev/loop1
> 
>      You may notice some of the command switches are defaults.  Remember 
> what I said about a belt and suspenders?  Personally, in such a case I 
> would not rely on defaults.

Because defaults change?
> 
>      Now try running a check on the assembled array:
> fsck /dev/md99
> 
>      If that fails, shutdown the array with
> 
> mdadm -S /dev/md99
> 
>      and then try creating the array with a different drive order.  
> There are only two other possible permutations of three disks.  If none 
> of those work, you have some more serious problems.

And here you are oversimplifying the problem immensely. If those three 
drives aren't the originals, then the chances are HIGH that a simple 
re-assembly/creation is going to fail your simplistic scenario.

That said, I couldn't agree more about getting new drive(s) to take a 
backup before attempting recovery ...

Cheers,
Wol

Re: My superblocks have gone missing, can't reassemble raid5

[Leslie Rhorer]

On 5/19/2021 11:41 AM, antlists wrote:
> On 19/05/2021 15:48, Leslie Rhorer wrote:
>>      Then I would try recreating the RAID based upon the earlier 
>> Examine report:
>>
>> mdadm -C -f -n 3 -l 5 -e 1.2 -c 512 -p ls /dev/md99 /dev/loop2 
>> /dev/loop0 /dev/loop1
>>
>>      You may notice some of the command switches are defaults.  
>> Remember what I said about a belt and suspenders?  Personally, in such 
>> a case I would not rely on defaults.
> 
> Because defaults change?

	That is one reason, yes.  It also means no mater what else, I know what 
the system is being told to do.

>>      Now try running a check on the assembled array:
>> fsck /dev/md99
>>
>>      If that fails, shutdown the array with
>>
>> mdadm -S /dev/md99
>>
>>      and then try creating the array with a different drive order. 
>> There are only two other possible permutations of three disks.  If 
>> none of those work, you have some more serious problems.
> 
> And here you are oversimplifying the problem immensely.

	No, I am not.

> If those three 
> drives aren't the originals, then the chances are HIGH that a simple 
> re-assembly/creation is going to fail your simplistic scenario.

	Absolutely.  So what?  If it does, very little time and no data 
whatsoever is lost.  If it doesn't fail, then his problem is solved with 
very little trouble.  I have been troubleshooting technical issues for 
well over 50 years, and one of the very first lessons I learned is one 
should try the simplest solutions first.  Whenever they work, they can 
save a tremendous amount of time and effort.  After that, it's time to 
dig out the heavy tools.

Re: My superblocks have gone missing, can't reassemble raid5

[Leslie Rhorer]

On 5/19/2021 11:41 AM, antlists wrote:
> On 19/05/2021 15:48, Leslie Rhorer wrote:
> 
> Because defaults change?

	Oh, I also don't necessarily know which build of mdadm he is using. 
They might be default on my build but not his.

>>      Now try running a check on the assembled array:
>> fsck /dev/md99
>>
>>      If that fails, shutdown the array with
>>
>> mdadm -S /dev/md99
>>
>>      and then try creating the array with a different drive order. 
>> There are only two other possible permutations of three disks.  If 
>> none of those work, you have some more serious problems.
> 
> And here you are oversimplifying the problem immensely. If those three 
> drives aren't the originals

	Hang on.  Which drives do you mean?

Re: My superblocks have gone missing, can't reassemble raid5

[Wols Lists]

On 19/05/21 18:08, Leslie Rhorer wrote:
> 
> 
> On 5/19/2021 11:41 AM, antlists wrote:
>> On 19/05/2021 15:48, Leslie Rhorer wrote:
>>
>> Because defaults change?
> 
>     Oh, I also don't necessarily know which build of mdadm he is using.
> They might be default on my build but not his.
> 
>>>      Now try running a check on the assembled array:
>>> fsck /dev/md99
>>>
>>>      If that fails, shutdown the array with
>>>
>>> mdadm -S /dev/md99
>>>
>>>      and then try creating the array with a different drive order.
>>> There are only two other possible permutations of three disks.  If
>>> none of those work, you have some more serious problems.
>>
>> And here you are oversimplifying the problem immensely. If those three
>> drives aren't the originals
> 
>     Hang on.  Which drives do you mean?

The drives he originally ran --create on to create the array in the
first place.

The ONLY time you can be reasonably confident that running --create WILL
recover a damaged array is if it is still in its original state - no
drives swapped, no admin changes to the array, AND you're using the same
version of mdadm.

Cheers,
Wol

Re: My superblocks have gone missing, can't reassemble raid5

[Leslie Rhorer]

On 5/19/2021 1:00 PM, Wols Lists wrote:
> On 19/05/21 18:08, Leslie Rhorer wrote:
>>>> There are only two other possible permutations of three disks.  If
>>>> none of those work, you have some more serious problems.
>>>
>>> And here you are oversimplifying the problem immensely. If those three
>>> drives aren't the originals
>>
>>      Hang on.  Which drives do you mean?
> 
> The drives he originally ran --create on to create the array in the
> first place.

	OK, after a double-take, I wasn't sure.

> The ONLY time you can be reasonably confident that running --create WILL
> recover a damaged array is if it is still in its original state - no
> drives swapped, no admin changes to the array, AND you're using the same
> version of mdadm.

	That's a little bit of an overstatement, depending on what you mean by 
"reasonably confident".  Swapped drives should not ordinarily cause an 
issue, especially with RAID 4 or 5.  The parity is, after all, 
numerically unique.  Admin changes to the array should be similarly 
fully established provided the rebuild completed properly.  I don't 
think the parity algorythms have changed over time in mdadm, either. 
Had they done so, mdadm would not be able to assemble arrays from 
previous versions regardless of whether the superblock was intact.

	The main point with respect to my previous post, however, is one 
needn't be confident at all.  Hopeful, perhaps, but one needn't have any 
certainty at all the attempt will work, since one is no worse off if the 
attempt fails than prior to the attempt.  I certainly would not bet the 
farm on it working.  After all, as I mentioned before, there may well be 
something else really, really wrong going on here.

Re: My superblocks have gone missing, can't reassemble raid5

[antlists]

On 19/05/2021 20:01, Leslie Rhorer wrote:
>> The ONLY time you can be reasonably confident that running --create WILL
>> recover a damaged array is if it is still in its original state - no
>> drives swapped, no admin changes to the array, AND you're using the same
>> version of mdadm.
> 
>      That's a little bit of an overstatement, depending on what you mean 
> by "reasonably confident".  Swapped drives should not ordinarily cause 
> an issue, especially with RAID 4 or 5.  The parity is, after all, 
> numerically unique.  Admin changes to the array should be similarly 
> fully established provided the rebuild completed properly.  I don't 
> think the parity algorythms have changed over time in mdadm, either. Had 
> they done so, mdadm would not be able to assemble arrays from previous 
> versions regardless of whether the superblock was intact.


You said there's only three possible combinations of three drives. Every 
change I've mentioned adds another variable - more options ...

The data offset is not fixed, for one. Swapping a drive could mean 
drives have different offsets which means NO combination of drives, with 
default options, will work. Rebuilding an array moves everything around.

Yes you can explicitly specify everything, and get mdadm to recover the 
array if the superblocks have been lost, but it's nowhere as simple as 
"there are only three possible combinations".

Cheers,
Wol

Re: My superblocks have gone missing, can't reassemble raid5

[Leslie Rhorer]

On 5/19/2021 3:01 PM, antlists wrote:
> On 19/05/2021 20:01, Leslie Rhorer wrote:
>>> The ONLY time you can be reasonably confident that running --create WILL
>>> recover a damaged array is if it is still in its original state - no
>>> drives swapped, no admin changes to the array, AND you're using the same
>>> version of mdadm.
>>
>>      That's a little bit of an overstatement, depending on what you 
>> mean by "reasonably confident".  Swapped drives should not ordinarily 
>> cause an issue, especially with RAID 4 or 5.  The parity is, after 
>> all, numerically unique.  Admin changes to the array should be 
>> similarly fully established provided the rebuild completed properly.  
>> I don't think the parity algorythms have changed over time in mdadm, 
>> either. Had they done so, mdadm would not be able to assemble arrays 
>> from previous versions regardless of whether the superblock was intact.
> 
> 
> You said there's only three possible combinations of three drives. Every 
> change I've mentioned adds another variable - more options ...

	I meant 6, not 3.

> The data offset is not fixed, for one.

	All of the data offsets were 262144 sectors in the listed Examine were 
262144 sectors.

> Swapping a drive could mean 
> drives have different offsets which means NO combination of drives

	I think not.  That is to say, it is certainly not impossible for it to 
change, but it is quite unlikely.  For one thing, if the offset is more, 
then all the data will no longer fit on the device, which would be a bit 
of a disaster.

	My main arrays consist of 8 drives each.  Every drive in both arrays 
has been replaced numerous times, starting with 1T drives more than 10 
years ago.  Now they are all 8T drives.  All 16 have data offsets of 
262144 sectors, just like the report from his drives.  Note I have seen 
arrays with different offsets, and of course any array on a partition 
will have a different offset.

	So while it is possible the offsets on his drives have changed, it is 
not at all likely.  Again. likely or not, it will not hurt for him to try.

> Yes you can explicitly specify everything, and get mdadm to recover the 
> array if the superblocks have been lost, but it's nowhere as simple as 
> "there are only three possible combinations".

	The number of permutations of the order of three drives is precisely 
six.  The permutations are:

1, 2, 3

1, 3, 2

2, 1, 3

2, 3, 1

3, 1, 2

3, 2, 1

	The Examine stated it was 2, 3, 1, but the device order is not at all 
unlikely to have changed.  Once again, I was very explicit in saying the 
simple create may not work and if not he is facing some much more 
serious issues.  That was and is in no way incorrect.  The odds one of 
them will work are not terrible, and if so, he is in good shape.  Are 
you seriously implying there is no way any of them could possibly work? 
  Are you seriously suggesting he should not try the simple approach 
before digging deep into the data to try to figure out all the 
parameters when the system was shut down?

Re: My superblocks have gone missing, can't reassemble raid5

[Nix]

On 20 May 2021, Leslie Rhorer outgrape:

> On 5/19/2021 3:01 PM, antlists wrote:
>> Yes you can explicitly specify everything, and get mdadm to recover the array if the superblocks have been lost, but it's nowhere
>> as simple as "there are only three possible combinations".
>
> 	The number of permutations of the order of three drives is precisely six.  The permutations are:
>
> 1, 2, 3
>
> 1, 3, 2
>
> 2, 1, 3
>
> 2, 3, 1
>
> 3, 1, 2
>
> 3, 2, 1
>
> 	The Examine stated it was 2, 3, 1, but the device order is not
> at all unlikely to have changed. Once again, I was very explicit in
> saying the simple create may not work and if not he is facing some
> much more serious issues. That was and is in no way incorrect. The
> odds one of them will work are not terrible, and if so, he is in good
> shape. Are you seriously implying there is no way any of them could
> possibly work?

Of course not, but it's *likely* none will work. mdadm's default chunk
size has probably changed since the array was created if it's more than
a few years old, for starters.

Re: My superblocks have gone missing, can't reassemble raid5

[Leslie Rhorer]

On 5/20/2021 3:49 PM, Nix wrote:

>> odds one of them will work are not terrible, and if so, he is in good
>> shape. Are you seriously implying there is no way any of them could
>> possibly work?
> 
> Of course not, but it's *likely* none will work.

	Well, it is not unlikely none will work.  I would estimate the OP has a 
pretty fair chance it will work.

> mdadm's default chunk
> size has probably changed since the array was created if it's more than
> a few years old, for starters.

	Which is precisely why I specified the chunk size (-c 512) reported by 
his Examine on the command line.  This eliminates any ambiguity.  Since 
the build in Debian Buster is also using a default chunk size of 512K, 
it is unlikely the default has changed since he created the array. 
Older versions used 64K.  Specifying the chunk size on the command line 
makes the issue moot.

Re: My superblocks have gone missing, can't reassemble raid5

[Leslie Rhorer]

Christopher,

	What was the outcome?  Were you able to recover your array?

Re: My superblocks have gone missing, can't reassemble raid5

[Nix]

On 19 May 2021, Leslie Rhorer spake thusly:

> 	That's a little bit of an overstatement, depending on what you
> mean by "reasonably confident". Swapped drives should not ordinarily
> cause an issue, especially with RAID 4 or 5. The parity is, after all,
> numerically unique. Admin changes to the array should be similarly
> fully established provided the rebuild completed properly. I don't
> think the parity algorythms have changed over time in mdadm, either.

All sorts of creation-time things have changed over time: default chunk
sizes, drive ordering (particularly but not only for RAID10), data
offset, etc etc etc. The list is really rather long and the number of
possible combinations astronomical. (Sure, it's less astronomical if you
know what version of mdadm you made the array with in the first place,
but hardly anyone who hasn't already been bitten tracks that, and if
they do they probably recorded all the other relevant state too by
preserving --examine output, so no guesswork is needed.)

> Had they done so, mdadm would not be able to assemble arrays from
> previous versions regardless of whether the superblock was intact.

Naah. Most of this stuff is recorded *in* the superblock, so mdadm can
assemble without difficulty or assistance: it doesn't do it by picking
defaults from inside mdadm's source code! *Those* defaults only apply at
array creation time. But when recreating an array over the top of a
vanished one with the same parameters, you have to *remember* those
parameters...

-- 
NULL && (void)

Re: My superblocks have gone missing, can't reassemble raid5

[Leslie Rhorer]

On 5/20/2021 3:48 PM, Nix wrote:
> On 19 May 2021, Leslie Rhorer spake thusly:
> 
>> 	That's a little bit of an overstatement, depending on what you
>> mean by "reasonably confident". Swapped drives should not ordinarily
>> cause an issue, especially with RAID 4 or 5. The parity is, after all,
>> numerically unique. Admin changes to the array should be similarly
>> fully established provided the rebuild completed properly. I don't
>> think the parity algorythms have changed over time in mdadm, either.
> 
> All sorts of creation-time things have changed over time: default chunk
> sizes, drive ordering (particularly but not only for RAID10), data
> offset, etc etc etc.

	Which is why I explicitly specified those parameters (other than the 
data offset) in the examp0le commands I listed.  There is a build of 
mdadm out there that allows one to specify the offset, as well.

> The list is really rather long and the number of
> possible combinations astronomical. (Sure, it's less astronomical if you

	It's not astronomical, but in terms of just guessing it is indeed 
rather large.

> know what version of mdadm you made the array with in the first place,
> but hardly anyone who hasn't already been bitten tracks that, and if
> they do they probably recorded all the other relevant state too by
> preserving --examine output, so no guesswork is needed.)

	Which he did.  This limits the field substantially.

>> Had they done so, mdadm would not be able to assemble arrays from
>> previous versions regardless of whether the superblock was intact.
> 
> Naah. Most of this stuff is recorded *in* the superblock, so mdadm can
> assemble without difficulty or assistance:

	Point taken.  The OP's downfall will be if the superblocks were fiddled 
substantially after he saved them.  If not, he should be golden.

> it doesn't do it by picking
> defaults from inside mdadm's source code! *Those* defaults only apply at
> array creation time.

	Yes, of course.

> But when recreating an array over the top of a
> vanished one with the same parameters, you have to *remember* those
> parameters...

	Unless, of course, the defaults in the build match up with those used 
in the last organization of the array.  Fingers crossed, as it were.