* My superblocks have gone missing, can't reassemble raid5
@ 2021-05-17 4:16 Christopher Thomas
2021-05-17 4:23 ` Christopher Thomas
` (2 more replies)
0 siblings, 3 replies; 27+ messages in thread
From: Christopher Thomas @ 2021-05-17 4:16 UTC (permalink / raw)
To: linux-raid
Hi all,
I've updated my system & migrated my 3 raid5 component drives from the
old to the new, but now can't reassemble the array - mdadm just
doesn't recognize that these belong to an array at all.
The scenario:
For many years, I've run a raid5 array on a virtual Linux server
(Ubuntu 12.04) in VirtualBox on a Windows 10 host, with 3 2.7TB drives
attached to the virt in "Raw Disk" mode, and assembled into an array.
I recently upgraded to a completely different physical machine, but
still running Windows 10 and VirtualBox. I'm reasonably sure that the
last time I shut it down, the array was clean. Or at they very least,
the drives had superblocks. I plugged the old drives into it,
migrated the virtual machine image to the new system, and attached
them as raw disks, just as in the old system. And they show up as
/dev/sd[b-d], as before. However, it's not recognized automatically
as an array at boot, and manual attempts to assemble & start the array
fail with 'no superblock'
The closest I've found online as a solution is to --create the array
again using the same parameters. But it sounds like if I don't get
the drive order exactly the same, I'll lose the data. Other solutions
hint at playing with the partition table, but I'm equally nervous
about that. So I thought it was a good time to stop & ask for advice.
The details:
Here's my arrangement of disks now, where sd[bcd] are the components:
==========
chris@ursula:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 20.1G 0 disk
├─sda1 8:1 0 19.2G 0 part /
├─sda2 8:2 0 1K 0 part
└─sda5 8:5 0 976M 0 part [SWAP]
sdb 8:16 0 2.7T 0 disk
└─sdb1 8:17 0 128M 0 part
sdc 8:32 0 2.7T 0 disk
└─sdc1 8:33 0 128M 0 part
sdd 8:48 0 2.7T 0 disk
└─sdd1 8:49 0 128M 0 part
sr0 11:0 1 1024M 0 rom
chris@ursula:~$ sudo /sbin/fdisk -l
Disk /dev/sda: 20.1 GiB, 21613379584 bytes, 42213632 sectors
Disk model: VBOX HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x4bbbafdf
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 40212479 40210432 19.2G 83 Linux
/dev/sda2 40214526 42213375 1998850 976M 5 Extended
/dev/sda5 40214528 42213375 1998848 976M 82 Linux swap / Solaris
Disk /dev/sdb: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: VBOX HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 6489224B-FAF8-45E2-AB3D-C0D280F8E91E
Device Start End Sectors Size Type
/dev/sdb1 34 262177 262144 128M Microsoft reserved
Disk /dev/sdc: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: VBOX HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 6497BDEB-A8D0-40D7-9CD2-D06018862F2B
Device Start End Sectors Size Type
/dev/sdc1 34 262177 262144 128M Microsoft reserved
Disk /dev/sdd: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: VBOX HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 9DB2C3F2-F93D-4A6D-AE0E-CE28A8B8C4A2
Device Start End Sectors Size Type
/dev/sdd1 34 262177 262144 128M Microsoft reserved
==========
Note: I always intended to use the whole disk, so I don't know why I
would have created a single large partition on each, and I don't
recall doing so. But it's been a while, so I just might not be
remembering.
Here's what happens when I try to do anything with it:
===========
chris@ursula:~$ sudo /sbin/mdadm --verbose --assemble /dev/md0
/dev/sdb /dev/sdc /dev/sdd
mdadm: looking for devices for /dev/md0
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: /dev/sdb has no superblock - assembly aborted
chris@ursula:~$ sudo /sbin/mdadm --examine /dev/sd[bcd]*
/dev/sdb:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
mdadm: No md superblock detected on /dev/sdb1.
/dev/sdc:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
mdadm: No md superblock detected on /dev/sdc1.
/dev/sdd:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
mdadm: No md superblock detected on /dev/sdd1.
======
At some point on the old system, back when the array was still
working, I did dump the results of Examine, which looked like this:
==========
/dev/sdb:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 36205acf:993973ba:05712a13:ff75c031
Name : ursula:0 (local to host ursula)
Creation Time : Fri Apr 26 23:15:04 2013
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : active
Device UUID : 8841770a:f653d990:d5db60a0:fe2e4276
Update Time : Sun Jul 5 12:36:19 2020
Checksum : 3a671053 - correct
Events : 76713
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : AAA ('A' == active, '.' == missing)
/dev/sdc:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 36205acf:993973ba:05712a13:ff75c031
Name : ursula:0 (local to host ursula)
Creation Time : Fri Apr 26 23:15:04 2013
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : clean
Device UUID : 87fd8496:95c9cd5e:5caaa28a:25f6ab04
Update Time : Sat May 30 02:02:45 2020
Checksum : ce4cd20 - correct
Events : 76711
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : AAA ('A' == active, '.' == missing)
/dev/sdd:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : 36205acf:993973ba:05712a13:ff75c031
Name : ursula:0 (local to host ursula)
Creation Time : Fri Apr 26 23:15:04 2013
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
State : active
Device UUID : c796e484:b4ed6813:a97e0ce9:66a56758
Update Time : Sun Jul 5 12:36:19 2020
Checksum : 6235188e - correct
Events : 76713
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAA ('A' == active, '.' == missing)
==========
Thank you for any ideas or guidance you can offer.
-Chris
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-17 4:16 My superblocks have gone missing, can't reassemble raid5 Christopher Thomas
@ 2021-05-17 4:23 ` Christopher Thomas
2021-05-17 6:28 ` Roman Mamedov
2021-05-19 14:48 ` Leslie Rhorer
2 siblings, 0 replies; 27+ messages in thread
From: Christopher Thomas @ 2021-05-17 4:23 UTC (permalink / raw)
To: Christopher Thomas; +Cc: linux-raid
I should say that all of my current diagnostic output is actually from
a virt running Debian 10 (Kernel 4.19.0), not the original Ubuntu
12.04. But the behavior is practically the same in both, and I'd
rather get it working on a modern system anyway.
-Chris
On Sun, May 16, 2021 at 9:16 PM Christopher Thomas <youkai@earthlink.net> wrote:
>
> Hi all,
>
> I've updated my system & migrated my 3 raid5 component drives from the
> old to the new, but now can't reassemble the array - mdadm just
> doesn't recognize that these belong to an array at all.
>
> The scenario:
> For many years, I've run a raid5 array on a virtual Linux server
> (Ubuntu 12.04) in VirtualBox on a Windows 10 host, with 3 2.7TB drives
> attached to the virt in "Raw Disk" mode, and assembled into an array.
> I recently upgraded to a completely different physical machine, but
> still running Windows 10 and VirtualBox. I'm reasonably sure that the
> last time I shut it down, the array was clean. Or at they very least,
> the drives had superblocks. I plugged the old drives into it,
> migrated the virtual machine image to the new system, and attached
> them as raw disks, just as in the old system. And they show up as
> /dev/sd[b-d], as before. However, it's not recognized automatically
> as an array at boot, and manual attempts to assemble & start the array
> fail with 'no superblock'
>
> The closest I've found online as a solution is to --create the array
> again using the same parameters. But it sounds like if I don't get
> the drive order exactly the same, I'll lose the data. Other solutions
> hint at playing with the partition table, but I'm equally nervous
> about that. So I thought it was a good time to stop & ask for advice.
>
> The details:
>
> Here's my arrangement of disks now, where sd[bcd] are the components:
>
> ==========
> chris@ursula:~$ lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sda 8:0 0 20.1G 0 disk
> ├─sda1 8:1 0 19.2G 0 part /
> ├─sda2 8:2 0 1K 0 part
> └─sda5 8:5 0 976M 0 part [SWAP]
> sdb 8:16 0 2.7T 0 disk
> └─sdb1 8:17 0 128M 0 part
> sdc 8:32 0 2.7T 0 disk
> └─sdc1 8:33 0 128M 0 part
> sdd 8:48 0 2.7T 0 disk
> └─sdd1 8:49 0 128M 0 part
> sr0 11:0 1 1024M 0 rom
>
> chris@ursula:~$ sudo /sbin/fdisk -l
> Disk /dev/sda: 20.1 GiB, 21613379584 bytes, 42213632 sectors
> Disk model: VBOX HARDDISK
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: dos
> Disk identifier: 0x4bbbafdf
>
> Device Boot Start End Sectors Size Id Type
> /dev/sda1 * 2048 40212479 40210432 19.2G 83 Linux
> /dev/sda2 40214526 42213375 1998850 976M 5 Extended
> /dev/sda5 40214528 42213375 1998848 976M 82 Linux swap / Solaris
>
>
> Disk /dev/sdb: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
> Disk model: VBOX HARDDISK
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: gpt
> Disk identifier: 6489224B-FAF8-45E2-AB3D-C0D280F8E91E
>
> Device Start End Sectors Size Type
> /dev/sdb1 34 262177 262144 128M Microsoft reserved
>
>
> Disk /dev/sdc: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
> Disk model: VBOX HARDDISK
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: gpt
> Disk identifier: 6497BDEB-A8D0-40D7-9CD2-D06018862F2B
>
> Device Start End Sectors Size Type
> /dev/sdc1 34 262177 262144 128M Microsoft reserved
>
>
> Disk /dev/sdd: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
> Disk model: VBOX HARDDISK
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: gpt
> Disk identifier: 9DB2C3F2-F93D-4A6D-AE0E-CE28A8B8C4A2
>
> Device Start End Sectors Size Type
> /dev/sdd1 34 262177 262144 128M Microsoft reserved
> ==========
>
> Note: I always intended to use the whole disk, so I don't know why I
> would have created a single large partition on each, and I don't
> recall doing so. But it's been a while, so I just might not be
> remembering.
>
> Here's what happens when I try to do anything with it:
>
> ===========
> chris@ursula:~$ sudo /sbin/mdadm --verbose --assemble /dev/md0
> /dev/sdb /dev/sdc /dev/sdd
> mdadm: looking for devices for /dev/md0
> mdadm: Cannot assemble mbr metadata on /dev/sdb
> mdadm: /dev/sdb has no superblock - assembly aborted
>
> chris@ursula:~$ sudo /sbin/mdadm --examine /dev/sd[bcd]*
> /dev/sdb:
> MBR Magic : aa55
> Partition[0] : 4294967295 sectors at 1 (type ee)
> mdadm: No md superblock detected on /dev/sdb1.
> /dev/sdc:
> MBR Magic : aa55
> Partition[0] : 4294967295 sectors at 1 (type ee)
> mdadm: No md superblock detected on /dev/sdc1.
> /dev/sdd:
> MBR Magic : aa55
> Partition[0] : 4294967295 sectors at 1 (type ee)
> mdadm: No md superblock detected on /dev/sdd1.
> ======
>
> At some point on the old system, back when the array was still
> working, I did dump the results of Examine, which looked like this:
>
> ==========
> /dev/sdb:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 36205acf:993973ba:05712a13:ff75c031
> Name : ursula:0 (local to host ursula)
> Creation Time : Fri Apr 26 23:15:04 2013
> Raid Level : raid5
> Raid Devices : 3
>
> Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
> Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
> Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> State : active
> Device UUID : 8841770a:f653d990:d5db60a0:fe2e4276
>
> Update Time : Sun Jul 5 12:36:19 2020
> Checksum : 3a671053 - correct
> Events : 76713
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 2
> Array State : AAA ('A' == active, '.' == missing)
> /dev/sdc:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 36205acf:993973ba:05712a13:ff75c031
> Name : ursula:0 (local to host ursula)
> Creation Time : Fri Apr 26 23:15:04 2013
> Raid Level : raid5
> Raid Devices : 3
>
> Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
> Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
> Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> State : clean
> Device UUID : 87fd8496:95c9cd5e:5caaa28a:25f6ab04
>
> Update Time : Sat May 30 02:02:45 2020
> Checksum : ce4cd20 - correct
> Events : 76711
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 0
> Array State : AAA ('A' == active, '.' == missing)
> /dev/sdd:
> Magic : a92b4efc
> Version : 1.2
> Feature Map : 0x0
> Array UUID : 36205acf:993973ba:05712a13:ff75c031
> Name : ursula:0 (local to host ursula)
> Creation Time : Fri Apr 26 23:15:04 2013
> Raid Level : raid5
> Raid Devices : 3
>
> Avail Dev Size : 5860271024 (2794.40 GiB 3000.46 GB)
> Array Size : 5860270080 (5588.79 GiB 6000.92 GB)
> Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
> Data Offset : 262144 sectors
> Super Offset : 8 sectors
> State : active
> Device UUID : c796e484:b4ed6813:a97e0ce9:66a56758
>
> Update Time : Sun Jul 5 12:36:19 2020
> Checksum : 6235188e - correct
> Events : 76713
>
> Layout : left-symmetric
> Chunk Size : 512K
>
> Device Role : Active device 1
> Array State : AAA ('A' == active, '.' == missing)
> ==========
>
> Thank you for any ideas or guidance you can offer.
> -Chris
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-17 4:16 My superblocks have gone missing, can't reassemble raid5 Christopher Thomas
2021-05-17 4:23 ` Christopher Thomas
@ 2021-05-17 6:28 ` Roman Mamedov
2021-05-17 9:30 ` Wols Lists
[not found] ` <CAAMCDec=H=6ceP9bKjSnsQyvmZ0LqTAYzJTDmDQoBOHSJV+hDw@mail.gmail.com>
2021-05-19 14:48 ` Leslie Rhorer
2 siblings, 2 replies; 27+ messages in thread
From: Roman Mamedov @ 2021-05-17 6:28 UTC (permalink / raw)
To: Christopher Thomas; +Cc: linux-raid
On Sun, 16 May 2021 21:16:22 -0700
Christopher Thomas <youkai@earthlink.net> wrote:
> Disk /dev/sdd: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
> Disk model: VBOX HARDDISK
> Units: sectors of 1 * 512 = 512 bytes
> Sector size (logical/physical): 512 bytes / 512 bytes
> I/O size (minimum/optimal): 512 bytes / 512 bytes
> Disklabel type: gpt
> Disk identifier: 9DB2C3F2-F93D-4A6D-AE0E-CE28A8B8C4A2
>
> Device Start End Sectors Size Type
> /dev/sdd1 34 262177 262144 128M Microsoft reserved
> ==========
>
> Note: I always intended to use the whole disk, so I don't know why I
> would have created a single large partition on each, and I don't
> recall doing so. But it's been a while, so I just might not be
> remembering.
Unless you mean "why I *wouldn't* have created"... There isn't a single large
partition on each, the partitions are only 128 MB in size, and these are the
Microsoft Reserved partitions (MSR)[1]. And a GPT partition table, which
shouldn't be there either, if you used whole disks for md RAID. I guess either
of these have overwritten your superblocks, which with version 1.2 are stored
4k from the beginning of the device.
I'm not sure if Windows would just quietly create these on it's on, but it
certainly would if the user opened "Disk Manager", was prompted that there are
3 new uninitialized drives and clicked "OK" to initialize them as GPT.
For that exact reason it is not a good idea to use whole disks for RAID, since
other OSes see them as empty/uninitialized which they can use as they see fit.
As for recovery, you really might need to play with --create; to prevent the
chance of data loss, there's a way to experiment using "overlays", keeping the
original drives untouched; see [2] for more background on that, and [3]
provides steps for your actual situation. Don't forget to use "--assume-clean"
and "--readonly".
There also might be some filesystem damage on the array, depending on how much
and how important were the pieces overwritten at the beginning of all disks by
data stored in the 128 MB MSR.
[1]
https://superuser.com/questions/942065/is-msr-partition-required-on-secondary-gpt-drives
[2] https://raid.wiki.kernel.org/index.php/Recovering_a_damaged_RAID
[3] https://raid.wiki.kernel.org/index.php/Irreversible_mdadm_failure_recovery
--
With respect,
Roman
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-17 6:28 ` Roman Mamedov
@ 2021-05-17 9:30 ` Wols Lists
[not found] ` <CAAMCDec=H=6ceP9bKjSnsQyvmZ0LqTAYzJTDmDQoBOHSJV+hDw@mail.gmail.com>
1 sibling, 0 replies; 27+ messages in thread
From: Wols Lists @ 2021-05-17 9:30 UTC (permalink / raw)
To: Roman Mamedov, Christopher Thomas; +Cc: linux-raid
On 17/05/21 07:28, Roman Mamedov wrote:
> As for recovery, you really might need to play with --create; to prevent the
> chance of data loss, there's a way to experiment using "overlays", keeping the
> original drives untouched; see [2] for more background on that, and [3]
> provides steps for your actual situation. Don't forget to use "--assume-clean"
> and "--readonly".
Firstly, what I'd do is a hexdump of the fifth kb of each disk (ie
hopefully where the superblock is/was). Post that here and see if anyone
can decode it. It does look like something has created a gpt, so mdadm
is no longer looking at the raw disk for the array.
There MIGHT be enough information lying around for someone to tell you
what --create command to use. Another thing - are those the original
disks used to create the array? Have you modified the array in any way?
If you haven't modified the array, then a plain create using the same
version of mdadm that created the array should just put things back the
way they were. The reason you mustn't just use --create willy nilly is
that any modification of the array might move the data offset, and
different versions of mdadm may have different default settings.
The other option is BACK UP YOUR HARD DRIVES, delete the new gpt, and
then see if it will assemble.
I know I've seen this before, but I've a nasty feeling the previous time
the user had used partitions and the GPT had been trashed so it's not
quite the same. Still, you might well be lucky ...
Cheers,
Wol
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
[not found] ` <CAAMCDec=H=6ceP9bKjSnsQyvmZ0LqTAYzJTDmDQoBOHSJV+hDw@mail.gmail.com>
@ 2021-05-17 13:19 ` Roman Mamedov
2021-05-18 17:47 ` Phil Turmel
0 siblings, 1 reply; 27+ messages in thread
From: Roman Mamedov @ 2021-05-17 13:19 UTC (permalink / raw)
To: Roger Heflin; +Cc: Christopher Thomas, Linux RAID
On Mon, 17 May 2021 05:36:42 -0500
Roger Heflin <rogerheflin@gmail.com> wrote:
> When I look at my 1.2 block, the mdraid header appears to start at 4k, and
> the gpt partition table starts at 0x0000 and ends before 4k.
>
> He may be able to simply delete the partition and have it work.
Christopher wrote that he tried:
chris@ursula:~$ sudo /sbin/mdadm --verbose --assemble /dev/md0
/dev/sdb /dev/sdc /dev/sdd
mdadm: looking for devices for /dev/md0
mdadm: Cannot assemble mbr metadata on /dev/sdb
mdadm: /dev/sdb has no superblock - assembly aborted
I would have expected mdadm when passed entire devices (not partitions) to not
even look if there are any partitions, and directly proceed to checking if
there's a superblock at its supposed location. But maybe indeed, from the
messages it looks like it bails before that, on seeing "mbr metadata", i.e.
the enclosing MBR partition table of GPT.
--
With respect,
Roman
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-17 13:19 ` Roman Mamedov
@ 2021-05-18 17:47 ` Phil Turmel
2021-05-18 18:31 ` Reindl Harald
0 siblings, 1 reply; 27+ messages in thread
From: Phil Turmel @ 2021-05-18 17:47 UTC (permalink / raw)
To: Roman Mamedov, Roger Heflin; +Cc: Christopher Thomas, Linux RAID
On 5/17/21 9:19 AM, Roman Mamedov wrote:
> On Mon, 17 May 2021 05:36:42 -0500
> Roger Heflin <rogerheflin@gmail.com> wrote:
>
>> When I look at my 1.2 block, the mdraid header appears to start at 4k, and
>> the gpt partition table starts at 0x0000 and ends before 4k.
>>
>> He may be able to simply delete the partition and have it work.
>
> Christopher wrote that he tried:
>
> chris@ursula:~$ sudo /sbin/mdadm --verbose --assemble /dev/md0
> /dev/sdb /dev/sdc /dev/sdd
> mdadm: looking for devices for /dev/md0
> mdadm: Cannot assemble mbr metadata on /dev/sdb
> mdadm: /dev/sdb has no superblock - assembly aborted
>
> I would have expected mdadm when passed entire devices (not partitions) to not
> even look if there are any partitions, and directly proceed to checking if
> there's a superblock at its supposed location. But maybe indeed, from the
> messages it looks like it bails before that, on seeing "mbr metadata", i.e.
> the enclosing MBR partition table of GPT.
>
The Microsoft system partition starts on top of the location for v1.2
metadata.
Just another reason to *never* use bare drives.
Sorry, Christopher. You will have to experiment with --create. Use
overlays.
Phil
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-18 17:47 ` Phil Turmel
@ 2021-05-18 18:31 ` Reindl Harald
2021-05-19 13:20 ` Leslie Rhorer
0 siblings, 1 reply; 27+ messages in thread
From: Reindl Harald @ 2021-05-18 18:31 UTC (permalink / raw)
To: Phil Turmel, Roman Mamedov, Roger Heflin; +Cc: Christopher Thomas, Linux RAID
Am 18.05.21 um 19:47 schrieb Phil Turmel:
> On 5/17/21 9:19 AM, Roman Mamedov wrote:
>> On Mon, 17 May 2021 05:36:42 -0500
>> Roger Heflin <rogerheflin@gmail.com> wrote:
>>
>>> When I look at my 1.2 block, the mdraid header appears to start at
>>> 4k, and
>>> the gpt partition table starts at 0x0000 and ends before 4k.
>>>
>>> He may be able to simply delete the partition and have it work.
>>
>> Christopher wrote that he tried:
>>
>> chris@ursula:~$ sudo /sbin/mdadm --verbose --assemble /dev/md0
>> /dev/sdb /dev/sdc /dev/sdd
>> mdadm: looking for devices for /dev/md0
>> mdadm: Cannot assemble mbr metadata on /dev/sdb
>> mdadm: /dev/sdb has no superblock - assembly aborted
>>
>> I would have expected mdadm when passed entire devices (not
>> partitions) to not
>> even look if there are any partitions, and directly proceed to
>> checking if
>> there's a superblock at its supposed location. But maybe indeed, from the
>> messages it looks like it bails before that, on seeing "mbr metadata",
>> i.e.
>> the enclosing MBR partition table of GPT.
>>
>
> The Microsoft system partition starts on top of the location for v1.2
> metadata.
>
> Just another reason to *never* use bare drives
the most important is that you have no guarantee that a replacement
drive years later is 100% identical in size
leave some margin and padding around the used space solves that problem
entirely and i still need to hear a single valid reason for using
unpartitioned drives in a RAID
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-18 18:31 ` Reindl Harald
@ 2021-05-19 13:20 ` Leslie Rhorer
2021-05-19 13:41 ` Phil Turmel
2021-05-19 14:20 ` Andy Smith
0 siblings, 2 replies; 27+ messages in thread
From: Leslie Rhorer @ 2021-05-19 13:20 UTC (permalink / raw)
To: Reindl Harald, Phil Turmel, Roman Mamedov, Roger Heflin
Cc: Christopher Thomas, Linux RAID
On 5/18/2021 1:31 PM, Reindl Harald wrote:
>
>
> Am 18.05.21 um 19:47 schrieb Phil Turmel:
>> On 5/17/21 9:19 AM, Roman Mamedov wrote:
>>> On Mon, 17 May 2021 05:36:42 -0500
>>> Roger Heflin <rogerheflin@gmail.com> wrote:
>>>
>>>> When I look at my 1.2 block, the mdraid header appears to start at
>>>> 4k, and
>>>> the gpt partition table starts at 0x0000 and ends before 4k.
>>>>
>>>> He may be able to simply delete the partition and have it work.
>>>
>>> Christopher wrote that he tried:
>>>
>>> chris@ursula:~$ sudo /sbin/mdadm --verbose --assemble /dev/md0
>>> /dev/sdb /dev/sdc /dev/sdd
>>> mdadm: looking for devices for /dev/md0
>>> mdadm: Cannot assemble mbr metadata on /dev/sdb
>>> mdadm: /dev/sdb has no superblock - assembly aborted
>>>
>>> I would have expected mdadm when passed entire devices (not
>>> partitions) to not
>>> even look if there are any partitions, and directly proceed to
>>> checking if
>>> there's a superblock at its supposed location. But maybe indeed, from
>>> the
>>> messages it looks like it bails before that, on seeing "mbr
>>> metadata", i.e.
>>> the enclosing MBR partition table of GPT.
>>>
>>
>> The Microsoft system partition starts on top of the location for v1.2
>> metadata.
>>
>> Just another reason to *never* use bare drives
>
> the most important is that you have no guarantee that a replacement
> drive years later is 100% identical in size
>
> leave some margin and padding around the used space solves that problem
> entirely and i still need to hear a single valid reason for using
> unpartitioned drives in a RAID
I can give you about a dozen. We will start with this:
1. Partitioning is not necessary. Doing something that is not necessary
is not usually worthwhile.
2. Partitioning offers no advantages. Doing something unnecessary is
questionable. Doing something that has no purpose at all is downright
foolish.
3. Partitioning introduces an additional layer of activity. This makes
it both more complex and more wasteful of resources. And yes, before
you bring it up, the additional complexity and resource infringement are
quite small. They are not zero, however, and they are in essence
continuous. Every little bit counts.
4. There is no guarantee the partitioning that works today will work
tomorrow. It should, of course, and it probably will, but why take a
risk when there is absolutely no gain whatsoever?
5. It is additional work that ultimately yields no positive result
whatsoever. Admittedly, partitioning one disk is not a lot of work.
Partitioning 50 disks is another matter. Partitioning 500 disks...
6. Partitioning has an intent. That intent is of no relevance
whatsoever on a device whose content is singular in scope. Are you
suggesting we should also partition tapes? Ralph Waldo Emerson had
something important to say about repeatedly doing things simply because
they have been done before and elsewhere.
7. There is no downside to forfeiting the partition table. Not needing
to do something is an extremely good reason for not doing it. This is
of course a corollary to point #1.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-19 13:20 ` Leslie Rhorer
@ 2021-05-19 13:41 ` Phil Turmel
2021-05-19 16:54 ` Leslie Rhorer
2021-05-20 19:37 ` Nix
2021-05-19 14:20 ` Andy Smith
1 sibling, 2 replies; 27+ messages in thread
From: Phil Turmel @ 2021-05-19 13:41 UTC (permalink / raw)
To: Leslie Rhorer, Reindl Harald, Roman Mamedov, Roger Heflin
Cc: Christopher Thomas, Linux RAID
On 5/19/21 9:20 AM, Leslie Rhorer wrote:
>
>
> On 5/18/2021 1:31 PM, Reindl Harald wrote:
[trim/]
>> leave some margin and padding around the used space solves that
>> problem entirely and i still need to hear a single valid reason for
>> using unpartitioned drives in a RAID
>
> I can give you about a dozen. We will start with this:
>
> 1. Partitioning is not necessary. Doing something that is not necessary
> is not usually worthwhile.
1a: sure. 1b: I can think of many things that aren't *necessary* but
are certainly worthwhile. I can even throw a few out there, like
personal hygiene, healthy diets, exercise. In this context, I would
list drive smart monitoring, weekly scrubs, and sysadmins with a clue.
> 2. Partitioning offers no advantages. Doing something unnecessary is
> questionable. Doing something that has no purpose at all is downright
> foolish.
Who says it has no purpose. Its purpose is to segment a device into
regions with associated metadata.
> 3. Partitioning introduces an additional layer of activity. This makes
> it both more complex and more wasteful of resources. And yes, before
> you bring it up, the additional complexity and resource infringement are
> quite small. They are not zero, however, and they are in essence
> continuous. Every little bit counts.
Hmm. A sector offset and limit check, buried deep in the kernel's
common code. I dare you to measure the incremental impact.
> 4. There is no guarantee the partitioning that works today will work
> tomorrow. It should, of course, and it probably will, but why take a
> risk when there is absolutely no gain whatsoever?
You assert "no gain", but you provide no support for your assertion.
> 5. It is additional work that ultimately yields no positive result
> whatsoever. Admittedly, partitioning one disk is not a lot of work.
> Partitioning 50 disks is another matter. Partitioning 500 disks...
You assert "no positive result whatsoever". Sounds like #4. With
similar lack of support. Fluffing up your list, much?
> 6. Partitioning has an intent. That intent is of no relevance
> whatsoever on a device whose content is singular in scope. Are you
> suggesting we should also partition tapes? Ralph Waldo Emerson had
> something important to say about repeatedly doing things simply because
> they have been done before and elsewhere.
No relevance? Metadata can be rather useful when operating systems
encounter what looks like an empty disk. Metadata that says "not
empty!" Especially valuable when metadata is *recognized* by all
operating systems. You know, like, a *standard*. While MDraid places
metadata on the devices it uses, only Linux *recognizes* it.
> 7. There is no downside to forfeiting the partition table. Not needing
> to do something is an extremely good reason for not doing it. This is
> of course a corollary to point #1.
Just more fluff.
Microsoft and a number of NAS products are known to corrupt drives with
no partition table. I vaguely recall hardware raid doing it, too.
That's a damn good reason to add a tiny (measurable?) amount of overhead.
And dude, making a single partition on a disk can be /scripted/. Might
want to learn about that, if the pain of the driving fdisk/gdisk
occasionally is too much for your delicate fingers.
Phil
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-19 13:20 ` Leslie Rhorer
2021-05-19 13:41 ` Phil Turmel
@ 2021-05-19 14:20 ` Andy Smith
2021-05-19 14:59 ` Leslie Rhorer
1 sibling, 1 reply; 27+ messages in thread
From: Andy Smith @ 2021-05-19 14:20 UTC (permalink / raw)
To: Linux RAID
Hello,
On Wed, May 19, 2021 at 08:20:19AM -0500, Leslie Rhorer wrote:
> 1. Partitioning is not necessary. Doing something that is not necessary is
> not usually worthwhile.
I'm with you in that I prefer not to use partitions when I don't
need to.
However, there are many reports of people suffering corrupted disks
that ended up being due to their motherboard helpfully deciding that
if it sees a disk with no MBR nor GPT then it should create its own
at every boot.
http://forum.asrock.com/forum_posts.asp?TID=10174&title=asrock-motherboard-destroys-linux-software-raid
So, I'm not going to rail against anyone who prefer to always
partition.
Cheers,
Andy
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-17 4:16 My superblocks have gone missing, can't reassemble raid5 Christopher Thomas
2021-05-17 4:23 ` Christopher Thomas
2021-05-17 6:28 ` Roman Mamedov
@ 2021-05-19 14:48 ` Leslie Rhorer
2021-05-19 16:41 ` antlists
2 siblings, 1 reply; 27+ messages in thread
From: Leslie Rhorer @ 2021-05-19 14:48 UTC (permalink / raw)
To: Linux RAID
On 5/16/2021 11:16 PM, Christopher Thomas wrote:
> Hi all,
>
> I've updated my system & migrated my 3 raid5 component drives from the
> old to the new, but now can't reassemble the array - mdadm just
> doesn't recognize that these belong to an array at all.
>
> The scenario:
> For many years, I've run a raid5 array on a virtual Linux server
> (Ubuntu 12.04) in VirtualBox on a Windows 10 host, with 3 2.7TB drives
> attached to the virt in "Raw Disk" mode, and assembled into an array.
> I recently upgraded to a completely different physical machine, but
> still running Windows 10 and VirtualBox. I'm reasonably sure that the
> last time I shut it down, the array was clean. Or at they very least,
> the drives had superblocks. I plugged the old drives into it,
> migrated the virtual machine image to the new system, and attached
> them as raw disks, just as in the old system. And they show up as
> /dev/sd[b-d], as before. However, it's not recognized automatically
> as an array at boot, and manual attempts to assemble & start the array
> fail with 'no superblock'
>
> The closest I've found online as a solution is to --create the array
> again using the same parameters. But it sounds like if I don't get
> the drive order exactly the same, I'll lose the data. Other solutions
> hint at playing with the partition table, but I'm equally nervous
> about that. So I thought it was a good time to stop & ask for advice.
>
> The details:
>
> Here's my arrangement of disks now, where sd[bcd] are the components:
>
> ==========
> chris@ursula:~$ lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
> sda 8:0 0 20.1G 0 disk
> ├─sda1 8:1 0 19.2G 0 part /
> ├─sda2 8:2 0 1K 0 part
> └─sda5 8:5 0 976M 0 part [SWAP]
> sdb 8:16 0 2.7T 0 disk
> └─sdb1 8:17 0 128M 0 part
> sdc 8:32 0 2.7T 0 disk
> └─sdc1 8:33 0 128M 0 part
> sdd 8:48 0 2.7T 0 disk
> └─sdd1 8:49 0 128M 0 part
> sr0 11:0 1 1024M 0 rom
The first thing you need to do is copy those drives into safety media.
I know this means a new drive, but an 8T drive is not that expensive.
I would format the drive and mount it to some directory:
mkfs /dev/sdX
mkdir /Safety
mount /dev/sdX /Safety
cd /Safety
ddrescue /dev/sdb disk1 /Safety/disk1-map
ddrescue /dev/sdc disk2 /Safety/disk2-map
ddrescue /dev/sdd disk3 /Safety/disk3-map
As mentioned earlier in this thread, you can create overlays of the
original disks. That, or you can make loops of the backup files.
Actually, if it were me, I would create two sets of backups and work on
the second set, but then I am hyper-anal about such things. I don't
just employ a belt and suspenders. I use a belt, suspenders, large
numbers of staples, super glue, and a braided steel strap welded in
place. Use whatever level of redundancy with which you feel
comfortable, but I *DEFINITELY* do not recommend working with the
original media. Indeed, I do not recommend attempting to recover the
original media, at all. Once you have a solution identified, I would
employ new drives, keeping the old as backups. (Which rather begs the
question, "Why don't you have backups of the data, so you could simply
create an entirely new empty array and copy the data to the new array
from the backup?")
I would then create a loopfile from each backup file:
losetup -fP disk1
losetup -fP disk2
losetup -fP disk3
Then I would try recreating the RAID based upon the earlier Examine report:
mdadm -C -f -n 3 -l 5 -e 1.2 -c 512 -p ls /dev/md99 /dev/loop2
/dev/loop0 /dev/loop1
You may notice some of the command switches are defaults. Remember
what I said about a belt and suspenders? Personally, in such a case I
would not rely on defaults.
Now try running a check on the assembled array:
fsck /dev/md99
If that fails, shutdown the array with
mdadm -S /dev/md99
and then try creating the array with a different drive order. There
are only two other possible permutations of three disks. If none of
those work, you have some more serious problems.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-19 14:20 ` Andy Smith
@ 2021-05-19 14:59 ` Leslie Rhorer
0 siblings, 0 replies; 27+ messages in thread
From: Leslie Rhorer @ 2021-05-19 14:59 UTC (permalink / raw)
To: Linux RAID
On 5/19/2021 9:20 AM, Andy Smith wrote:
> Hello,
>
> On Wed, May 19, 2021 at 08:20:19AM -0500, Leslie Rhorer wrote:
>> 1. Partitioning is not necessary. Doing something that is not necessary is
>> not usually worthwhile.
>
> I'm with you in that I prefer not to use partitions when I don't
> need to.
>
> However, there are many reports of people suffering corrupted disks
> that ended up being due to their motherboard helpfully deciding that
> if it sees a disk with no MBR nor GPT then it should create its own
> at every boot.
>
> http://forum.asrock.com/forum_posts.asp?TID=10174&title=asrock-motherboard-destroys-linux-software-raid
So to paraphrase the other fellow, "That is another reason not to use
ASRock motherboards." Indeed, it is IMO a far better reason not to
support ASRock than it is to spuriously employ partitions. Of copurse
it is also just another reason to avoid Microsoft like the plague.
> So, I'm not going to rail against anyone who prefer to always
> partition.
I am not going to rail against anyone. They are their systems; they
can do as they please with them. Indeed, it is they are who are
suggesting I should do something, not the other way around. I suggest
not doing it. Honestly, it is not a huge deal, either way.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-19 14:48 ` Leslie Rhorer
@ 2021-05-19 16:41 ` antlists
2021-05-19 17:03 ` Leslie Rhorer
2021-05-19 17:08 ` Leslie Rhorer
0 siblings, 2 replies; 27+ messages in thread
From: antlists @ 2021-05-19 16:41 UTC (permalink / raw)
To: Leslie Rhorer, Linux RAID
On 19/05/2021 15:48, Leslie Rhorer wrote:
> Then I would try recreating the RAID based upon the earlier Examine
> report:
>
> mdadm -C -f -n 3 -l 5 -e 1.2 -c 512 -p ls /dev/md99 /dev/loop2
> /dev/loop0 /dev/loop1
>
> You may notice some of the command switches are defaults. Remember
> what I said about a belt and suspenders? Personally, in such a case I
> would not rely on defaults.
Because defaults change?
>
> Now try running a check on the assembled array:
> fsck /dev/md99
>
> If that fails, shutdown the array with
>
> mdadm -S /dev/md99
>
> and then try creating the array with a different drive order.
> There are only two other possible permutations of three disks. If none
> of those work, you have some more serious problems.
And here you are oversimplifying the problem immensely. If those three
drives aren't the originals, then the chances are HIGH that a simple
re-assembly/creation is going to fail your simplistic scenario.
That said, I couldn't agree more about getting new drive(s) to take a
backup before attempting recovery ...
Cheers,
Wol
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-19 13:41 ` Phil Turmel
@ 2021-05-19 16:54 ` Leslie Rhorer
2021-05-20 19:37 ` Nix
1 sibling, 0 replies; 27+ messages in thread
From: Leslie Rhorer @ 2021-05-19 16:54 UTC (permalink / raw)
To: Linux RAID
On 5/19/2021 8:41 AM, Phil Turmel wrote:
> On 5/19/21 9:20 AM, Leslie Rhorer wrote:
>>
>>
>> On 5/18/2021 1:31 PM, Reindl Harald wrote:
>
> [trim/]
>
>>> leave some margin and padding around the used space solves that
>>> problem entirely and i still need to hear a single valid reason for
>>> using unpartitioned drives in a RAID
>>
>> I can give you about a dozen. We will start with this:
>>
>> 1. Partitioning is not necessary. Doing something that is not
>> necessary is not usually worthwhile.
>
> 1a: sure. 1b: I can think of many things that aren't *necessary* but
> are certainly worthwhile. I can even throw a few out there, like
> personal hygiene, healthy diets, exercise. In this context, I would
> list drive smart monitoring, weekly scrubs, and sysadmins with a clue.
Every one of those things are not only necessary but absolutely
essential. The consequences of failing in those areas can be absolutely
devastating. What's more, I did not say, "...Never". I very
specifically said , "...Not usually". There was a reason I did that.
>> 2. Partitioning offers no advantages. Doing something unnecessary is
>> questionable. Doing something that has no purpose at all is downright
>> foolish.
>
> Who says it has no purpose. Its purpose is to segment a device into
> regions with associated metadata.
Context, please. If there is only one region with data then there is
no reason for any association with or existence of any metadata. A
system containing only one segment needs no internal identification of
any sort. A universe requires no names or labels of any sort.
A system consisting of multiple segments is a very different thing. In
any such system, partitioning or something similar is essential. That
is why my main servers all have partitioned OS arrays and
non-partitioned data arrays.
>> 3. Partitioning introduces an additional layer of activity. This
>> makes it both more complex and more wasteful of resources. And yes,
>> before you bring it up, the additional complexity and resource
>> infringement are quite small. They are not zero, however, and they
>> are in essence continuous. Every little bit counts.
>
> Hmm. A sector offset and limit check, buried deep in the kernel's
> common code. I dare you to measure the incremental impact.
You are assuming, there. The kernel can most certainly be compiled
without that code. In fact, I have done so on embedded systems. I
could measure the impact, if it were an issue for me. It isn't. The
simple fact is every bit of code takes time to run. I don't need a
quantitative measure to confirm that. I also know the run time of the
code is only a few msec for the first run and a few nanosec for
subsequent runs, assuming the blocks are kept in cache somewhere. That
is indeed minimal.
>> 4. There is no guarantee the partitioning that works today will work
>> tomorrow. It should, of course, and it probably will, but why take a
>> risk when there is absolutely no gain whatsoever?
>
> You assert "no gain", but you provide no support for your assertion.
That is because it is not possible to provide support for nothing. One
cannot prove non-existence. If you have support for the notion there is
some sort of gain, then by all means provide it. Short of that, there
is no evidence there is some gain to be had, and my statement stands.
Believe me, my feelings are not going to be hurt by being proved wrong.
>> 5. It is additional work that ultimately yields no positive result
>> whatsoever. Admittedly, partitioning one disk is not a lot of work.
>> Partitioning 50 disks is another matter. Partitioning 500 disks...
>
> You assert "no positive result whatsoever". Sounds like #4. With
> similar lack of support. Fluffing up your list, much?
No, it is the cost side of the argument, which is not the same as the
consequential side. They are the two sides of the cost / benefit
analysis. In #4, I assert the benefit is negligible, perhaps even zero.
You haven't bothered to refute this, by the way. In this point, I
highlight the fact there is some cost to the practice. Again, you have
not refuted this. An ad hominem reference to some alleged procedural
failure on my part does not support your position. Note I would be
happy for you to do so, but don't think criticizing my abilities,
whether accurate or not, in no way supports your position.
>> 6. Partitioning has an intent. That intent is of no relevance
>> whatsoever on a device whose content is singular in scope. Are you
>> suggesting we should also partition tapes? Ralph Waldo Emerson had
>> something important to say about repeatedly doing things simply
>> because they have been done before and elsewhere.
>
> No relevance? Metadata can be rather useful when operating systems
> encounter what looks like an empty disk.
I am afraid you are going to have to be more specific for me to concede
this point. I am having trouble coming up with a good example.
> Metadata that says "not
> empty!" Especially valuable when metadata is *recognized* by all
> operating systems. You know, like, a *standard*.
Please name one such case. I cannot think of even a single protocol or
standard that is universally recognized. Some are close, but then only
those that are very, very old.
> While MDraid places
> metadata on the devices it uses, only Linux *recognizes* it.
The logical extension of this is no system, anywhere, should ever use
RAID of any flavor. Nor indeed, any file system. Nor, for that matter,
any file system. Indeed, we should never use any hard drive, and
definitely not any tape drives.
>> 7. There is no downside to forfeiting the partition table. Not
>> needing to do something is an extremely good reason for not doing it.
>> This is of course a corollary to point #1.
>
> Just more fluff.
"Just" more ad hominem. Going down a short side path for a moment, the
term "just" is almost always an illegitimate attempt to belittle and
avoid an idea with the intent of being dismissive to an opponent in a
debate. All it really does is highlight the speaker's inability to
properly support the speaker's side of the debate. I would request you
please stop "justiing" and instead provide some real support for your
argument. It doesn't greatly matter to me who wins this argument, but
having to reply to unsupported rhetoric wastes my time.
> Microsoft and a number of NAS products are known to corrupt drives with
> no partition table.
Which is a pretty good argument to avoid those products, if true.
Actually, I can't speak to any particular NAS (most of which are Linux
based, and so quite unlikely to behave in such a manner), but while MS
was accused of such activity in the past. It was not true up to and
including Windows 7. I don't know about Windows 10, but I doubt it.
Notwithstanding, I still avoid Windows whenever possible. I certainly
avoid doing anything merely to accommodate Windows.
> I vaguely recall hardware raid doing it, too.
> That's a damn good reason to add a tiny (measurable?) amount of overhead.
No, it's a damn good reason to completely avoid any such products, for
reasons far deeper than just mucking around with partition tables.
Mucking around the system without the express intent and direction of
the system supervisor is not acceptable under any circumstances. It's
also a damn good reason why the supervisor needs to know what they are
doing.
> And dude, making a single partition on a disk can be /scripted/. Might
> want to learn about that, if the pain of the driving fdisk/gdisk
> occasionally is too much for your delicate fingers.
It's not too much. It's just unnecessary. So is typing <scriptname>
when it is not necessary. The fact it can be scripted is irrelevant.
You are free to do whatever you like. I am not going to try to stop
you. You asked for reasons why one may chose not to partition RAID
members. I gave you some. Accept them or reject them. I don't really
care. Show me where my statements were factually incorrect. I am
always happy to learn of my mistakes.
By the way, I don't partition non-RAID drives, either, unless they
require multiple segments. Not all of the media have any file systems,
either.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-19 16:41 ` antlists
@ 2021-05-19 17:03 ` Leslie Rhorer
2021-05-19 17:08 ` Leslie Rhorer
1 sibling, 0 replies; 27+ messages in thread
From: Leslie Rhorer @ 2021-05-19 17:03 UTC (permalink / raw)
To: antlists, Linux RAID
On 5/19/2021 11:41 AM, antlists wrote:
> On 19/05/2021 15:48, Leslie Rhorer wrote:
>> Then I would try recreating the RAID based upon the earlier
>> Examine report:
>>
>> mdadm -C -f -n 3 -l 5 -e 1.2 -c 512 -p ls /dev/md99 /dev/loop2
>> /dev/loop0 /dev/loop1
>>
>> You may notice some of the command switches are defaults.
>> Remember what I said about a belt and suspenders? Personally, in such
>> a case I would not rely on defaults.
>
> Because defaults change?
That is one reason, yes. It also means no mater what else, I know what
the system is being told to do.
>> Now try running a check on the assembled array:
>> fsck /dev/md99
>>
>> If that fails, shutdown the array with
>>
>> mdadm -S /dev/md99
>>
>> and then try creating the array with a different drive order.
>> There are only two other possible permutations of three disks. If
>> none of those work, you have some more serious problems.
>
> And here you are oversimplifying the problem immensely.
No, I am not.
> If those three
> drives aren't the originals, then the chances are HIGH that a simple
> re-assembly/creation is going to fail your simplistic scenario.
Absolutely. So what? If it does, very little time and no data
whatsoever is lost. If it doesn't fail, then his problem is solved with
very little trouble. I have been troubleshooting technical issues for
well over 50 years, and one of the very first lessons I learned is one
should try the simplest solutions first. Whenever they work, they can
save a tremendous amount of time and effort. After that, it's time to
dig out the heavy tools.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-19 16:41 ` antlists
2021-05-19 17:03 ` Leslie Rhorer
@ 2021-05-19 17:08 ` Leslie Rhorer
2021-05-19 18:00 ` Wols Lists
1 sibling, 1 reply; 27+ messages in thread
From: Leslie Rhorer @ 2021-05-19 17:08 UTC (permalink / raw)
To: antlists, Linux RAID
On 5/19/2021 11:41 AM, antlists wrote:
> On 19/05/2021 15:48, Leslie Rhorer wrote:
>
> Because defaults change?
Oh, I also don't necessarily know which build of mdadm he is using.
They might be default on my build but not his.
>> Now try running a check on the assembled array:
>> fsck /dev/md99
>>
>> If that fails, shutdown the array with
>>
>> mdadm -S /dev/md99
>>
>> and then try creating the array with a different drive order.
>> There are only two other possible permutations of three disks. If
>> none of those work, you have some more serious problems.
>
> And here you are oversimplifying the problem immensely. If those three
> drives aren't the originals
Hang on. Which drives do you mean?
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-19 17:08 ` Leslie Rhorer
@ 2021-05-19 18:00 ` Wols Lists
2021-05-19 19:01 ` Leslie Rhorer
0 siblings, 1 reply; 27+ messages in thread
From: Wols Lists @ 2021-05-19 18:00 UTC (permalink / raw)
To: Leslie Rhorer, Linux RAID
On 19/05/21 18:08, Leslie Rhorer wrote:
>
>
> On 5/19/2021 11:41 AM, antlists wrote:
>> On 19/05/2021 15:48, Leslie Rhorer wrote:
>>
>> Because defaults change?
>
> Oh, I also don't necessarily know which build of mdadm he is using.
> They might be default on my build but not his.
>
>>> Now try running a check on the assembled array:
>>> fsck /dev/md99
>>>
>>> If that fails, shutdown the array with
>>>
>>> mdadm -S /dev/md99
>>>
>>> and then try creating the array with a different drive order.
>>> There are only two other possible permutations of three disks. If
>>> none of those work, you have some more serious problems.
>>
>> And here you are oversimplifying the problem immensely. If those three
>> drives aren't the originals
>
> Hang on. Which drives do you mean?
The drives he originally ran --create on to create the array in the
first place.
The ONLY time you can be reasonably confident that running --create WILL
recover a damaged array is if it is still in its original state - no
drives swapped, no admin changes to the array, AND you're using the same
version of mdadm.
Cheers,
Wol
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-19 18:00 ` Wols Lists
@ 2021-05-19 19:01 ` Leslie Rhorer
2021-05-19 20:01 ` antlists
2021-05-20 20:48 ` Nix
0 siblings, 2 replies; 27+ messages in thread
From: Leslie Rhorer @ 2021-05-19 19:01 UTC (permalink / raw)
To: Linux RAID
On 5/19/2021 1:00 PM, Wols Lists wrote:
> On 19/05/21 18:08, Leslie Rhorer wrote:
>>>> There are only two other possible permutations of three disks. If
>>>> none of those work, you have some more serious problems.
>>>
>>> And here you are oversimplifying the problem immensely. If those three
>>> drives aren't the originals
>>
>> Hang on. Which drives do you mean?
>
> The drives he originally ran --create on to create the array in the
> first place.
OK, after a double-take, I wasn't sure.
> The ONLY time you can be reasonably confident that running --create WILL
> recover a damaged array is if it is still in its original state - no
> drives swapped, no admin changes to the array, AND you're using the same
> version of mdadm.
That's a little bit of an overstatement, depending on what you mean by
"reasonably confident". Swapped drives should not ordinarily cause an
issue, especially with RAID 4 or 5. The parity is, after all,
numerically unique. Admin changes to the array should be similarly
fully established provided the rebuild completed properly. I don't
think the parity algorythms have changed over time in mdadm, either.
Had they done so, mdadm would not be able to assemble arrays from
previous versions regardless of whether the superblock was intact.
The main point with respect to my previous post, however, is one
needn't be confident at all. Hopeful, perhaps, but one needn't have any
certainty at all the attempt will work, since one is no worse off if the
attempt fails than prior to the attempt. I certainly would not bet the
farm on it working. After all, as I mentioned before, there may well be
something else really, really wrong going on here.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-19 19:01 ` Leslie Rhorer
@ 2021-05-19 20:01 ` antlists
2021-05-19 23:45 ` Leslie Rhorer
2021-05-20 20:48 ` Nix
1 sibling, 1 reply; 27+ messages in thread
From: antlists @ 2021-05-19 20:01 UTC (permalink / raw)
To: Leslie Rhorer, Linux RAID
On 19/05/2021 20:01, Leslie Rhorer wrote:
>> The ONLY time you can be reasonably confident that running --create WILL
>> recover a damaged array is if it is still in its original state - no
>> drives swapped, no admin changes to the array, AND you're using the same
>> version of mdadm.
>
> That's a little bit of an overstatement, depending on what you mean
> by "reasonably confident". Swapped drives should not ordinarily cause
> an issue, especially with RAID 4 or 5. The parity is, after all,
> numerically unique. Admin changes to the array should be similarly
> fully established provided the rebuild completed properly. I don't
> think the parity algorythms have changed over time in mdadm, either. Had
> they done so, mdadm would not be able to assemble arrays from previous
> versions regardless of whether the superblock was intact.
You said there's only three possible combinations of three drives. Every
change I've mentioned adds another variable - more options ...
The data offset is not fixed, for one. Swapping a drive could mean
drives have different offsets which means NO combination of drives, with
default options, will work. Rebuilding an array moves everything around.
Yes you can explicitly specify everything, and get mdadm to recover the
array if the superblocks have been lost, but it's nowhere as simple as
"there are only three possible combinations".
Cheers,
Wol
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-19 20:01 ` antlists
@ 2021-05-19 23:45 ` Leslie Rhorer
2021-05-20 20:49 ` Nix
0 siblings, 1 reply; 27+ messages in thread
From: Leslie Rhorer @ 2021-05-19 23:45 UTC (permalink / raw)
To: antlists, Linux RAID
On 5/19/2021 3:01 PM, antlists wrote:
> On 19/05/2021 20:01, Leslie Rhorer wrote:
>>> The ONLY time you can be reasonably confident that running --create WILL
>>> recover a damaged array is if it is still in its original state - no
>>> drives swapped, no admin changes to the array, AND you're using the same
>>> version of mdadm.
>>
>> That's a little bit of an overstatement, depending on what you
>> mean by "reasonably confident". Swapped drives should not ordinarily
>> cause an issue, especially with RAID 4 or 5. The parity is, after
>> all, numerically unique. Admin changes to the array should be
>> similarly fully established provided the rebuild completed properly.
>> I don't think the parity algorythms have changed over time in mdadm,
>> either. Had they done so, mdadm would not be able to assemble arrays
>> from previous versions regardless of whether the superblock was intact.
>
>
> You said there's only three possible combinations of three drives. Every
> change I've mentioned adds another variable - more options ...
I meant 6, not 3.
> The data offset is not fixed, for one.
All of the data offsets were 262144 sectors in the listed Examine were
262144 sectors.
> Swapping a drive could mean
> drives have different offsets which means NO combination of drives
I think not. That is to say, it is certainly not impossible for it to
change, but it is quite unlikely. For one thing, if the offset is more,
then all the data will no longer fit on the device, which would be a bit
of a disaster.
My main arrays consist of 8 drives each. Every drive in both arrays
has been replaced numerous times, starting with 1T drives more than 10
years ago. Now they are all 8T drives. All 16 have data offsets of
262144 sectors, just like the report from his drives. Note I have seen
arrays with different offsets, and of course any array on a partition
will have a different offset.
So while it is possible the offsets on his drives have changed, it is
not at all likely. Again. likely or not, it will not hurt for him to try.
> Yes you can explicitly specify everything, and get mdadm to recover the
> array if the superblocks have been lost, but it's nowhere as simple as
> "there are only three possible combinations".
The number of permutations of the order of three drives is precisely
six. The permutations are:
1, 2, 3
1, 3, 2
2, 1, 3
2, 3, 1
3, 1, 2
3, 2, 1
The Examine stated it was 2, 3, 1, but the device order is not at all
unlikely to have changed. Once again, I was very explicit in saying the
simple create may not work and if not he is facing some much more
serious issues. That was and is in no way incorrect. The odds one of
them will work are not terrible, and if so, he is in good shape. Are
you seriously implying there is no way any of them could possibly work?
Are you seriously suggesting he should not try the simple approach
before digging deep into the data to try to figure out all the
parameters when the system was shut down?
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-19 13:41 ` Phil Turmel
2021-05-19 16:54 ` Leslie Rhorer
@ 2021-05-20 19:37 ` Nix
2021-06-07 9:52 ` Leslie Rhorer
1 sibling, 1 reply; 27+ messages in thread
From: Nix @ 2021-05-20 19:37 UTC (permalink / raw)
To: Phil Turmel
Cc: Leslie Rhorer, Roman Mamedov, Roger Heflin, Christopher Thomas,
Linux RAID
On 19 May 2021, Phil Turmel spake thusly:
> weekly scrubs
*Weekly*? Scrubbing my arrays takes three or four days. If I ran them
weekly the machine would never have time to do anything else!
(I run them every couple of months. Doing them more often than that
feels too much like the machine's only job is to scrub itself :)
obviously I have good backups too. The scrubs have never spotted
anything at fault in ten years or so of scrubbing at more or less this
frequency -- more often in the past, when disks were smaller and scrubs
took less time.)
If the storage machinery on this system is so badly off that it's
misreading bits more often than that I think I have bigger problems.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-19 19:01 ` Leslie Rhorer
2021-05-19 20:01 ` antlists
@ 2021-05-20 20:48 ` Nix
2021-05-21 3:56 ` Leslie Rhorer
1 sibling, 1 reply; 27+ messages in thread
From: Nix @ 2021-05-20 20:48 UTC (permalink / raw)
To: Leslie Rhorer; +Cc: Linux RAID
On 19 May 2021, Leslie Rhorer spake thusly:
> That's a little bit of an overstatement, depending on what you
> mean by "reasonably confident". Swapped drives should not ordinarily
> cause an issue, especially with RAID 4 or 5. The parity is, after all,
> numerically unique. Admin changes to the array should be similarly
> fully established provided the rebuild completed properly. I don't
> think the parity algorythms have changed over time in mdadm, either.
All sorts of creation-time things have changed over time: default chunk
sizes, drive ordering (particularly but not only for RAID10), data
offset, etc etc etc. The list is really rather long and the number of
possible combinations astronomical. (Sure, it's less astronomical if you
know what version of mdadm you made the array with in the first place,
but hardly anyone who hasn't already been bitten tracks that, and if
they do they probably recorded all the other relevant state too by
preserving --examine output, so no guesswork is needed.)
> Had they done so, mdadm would not be able to assemble arrays from
> previous versions regardless of whether the superblock was intact.
Naah. Most of this stuff is recorded *in* the superblock, so mdadm can
assemble without difficulty or assistance: it doesn't do it by picking
defaults from inside mdadm's source code! *Those* defaults only apply at
array creation time. But when recreating an array over the top of a
vanished one with the same parameters, you have to *remember* those
parameters...
--
NULL && (void)
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-19 23:45 ` Leslie Rhorer
@ 2021-05-20 20:49 ` Nix
2021-05-21 4:07 ` Leslie Rhorer
0 siblings, 1 reply; 27+ messages in thread
From: Nix @ 2021-05-20 20:49 UTC (permalink / raw)
To: Leslie Rhorer; +Cc: antlists, Linux RAID
On 20 May 2021, Leslie Rhorer outgrape:
> On 5/19/2021 3:01 PM, antlists wrote:
>> Yes you can explicitly specify everything, and get mdadm to recover the array if the superblocks have been lost, but it's nowhere
>> as simple as "there are only three possible combinations".
>
> The number of permutations of the order of three drives is precisely six. The permutations are:
>
> 1, 2, 3
>
> 1, 3, 2
>
> 2, 1, 3
>
> 2, 3, 1
>
> 3, 1, 2
>
> 3, 2, 1
>
> The Examine stated it was 2, 3, 1, but the device order is not
> at all unlikely to have changed. Once again, I was very explicit in
> saying the simple create may not work and if not he is facing some
> much more serious issues. That was and is in no way incorrect. The
> odds one of them will work are not terrible, and if so, he is in good
> shape. Are you seriously implying there is no way any of them could
> possibly work?
Of course not, but it's *likely* none will work. mdadm's default chunk
size has probably changed since the array was created if it's more than
a few years old, for starters.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-20 20:48 ` Nix
@ 2021-05-21 3:56 ` Leslie Rhorer
0 siblings, 0 replies; 27+ messages in thread
From: Leslie Rhorer @ 2021-05-21 3:56 UTC (permalink / raw)
To: Linux RAID
On 5/20/2021 3:48 PM, Nix wrote:
> On 19 May 2021, Leslie Rhorer spake thusly:
>
>> That's a little bit of an overstatement, depending on what you
>> mean by "reasonably confident". Swapped drives should not ordinarily
>> cause an issue, especially with RAID 4 or 5. The parity is, after all,
>> numerically unique. Admin changes to the array should be similarly
>> fully established provided the rebuild completed properly. I don't
>> think the parity algorythms have changed over time in mdadm, either.
>
> All sorts of creation-time things have changed over time: default chunk
> sizes, drive ordering (particularly but not only for RAID10), data
> offset, etc etc etc.
Which is why I explicitly specified those parameters (other than the
data offset) in the examp0le commands I listed. There is a build of
mdadm out there that allows one to specify the offset, as well.
> The list is really rather long and the number of
> possible combinations astronomical. (Sure, it's less astronomical if you
It's not astronomical, but in terms of just guessing it is indeed
rather large.
> know what version of mdadm you made the array with in the first place,
> but hardly anyone who hasn't already been bitten tracks that, and if
> they do they probably recorded all the other relevant state too by
> preserving --examine output, so no guesswork is needed.)
Which he did. This limits the field substantially.
>> Had they done so, mdadm would not be able to assemble arrays from
>> previous versions regardless of whether the superblock was intact.
>
> Naah. Most of this stuff is recorded *in* the superblock, so mdadm can
> assemble without difficulty or assistance:
Point taken. The OP's downfall will be if the superblocks were fiddled
substantially after he saved them. If not, he should be golden.
> it doesn't do it by picking
> defaults from inside mdadm's source code! *Those* defaults only apply at
> array creation time.
Yes, of course.
> But when recreating an array over the top of a
> vanished one with the same parameters, you have to *remember* those
> parameters...
Unless, of course, the defaults in the build match up with those used
in the last organization of the array. Fingers crossed, as it were.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-20 20:49 ` Nix
@ 2021-05-21 4:07 ` Leslie Rhorer
2021-06-07 9:55 ` Leslie Rhorer
0 siblings, 1 reply; 27+ messages in thread
From: Leslie Rhorer @ 2021-05-21 4:07 UTC (permalink / raw)
To: Linux RAID
On 5/20/2021 3:49 PM, Nix wrote:
>> odds one of them will work are not terrible, and if so, he is in good
>> shape. Are you seriously implying there is no way any of them could
>> possibly work?
>
> Of course not, but it's *likely* none will work.
Well, it is not unlikely none will work. I would estimate the OP has a
pretty fair chance it will work.
> mdadm's default chunk
> size has probably changed since the array was created if it's more than
> a few years old, for starters.
Which is precisely why I specified the chunk size (-c 512) reported by
his Examine on the command line. This eliminates any ambiguity. Since
the build in Debian Buster is also using a default chunk size of 512K,
it is unlikely the default has changed since he created the array.
Older versions used 64K. Specifying the chunk size on the command line
makes the issue moot.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-20 19:37 ` Nix
@ 2021-06-07 9:52 ` Leslie Rhorer
0 siblings, 0 replies; 27+ messages in thread
From: Leslie Rhorer @ 2021-06-07 9:52 UTC (permalink / raw)
To: Linux RAID
On 5/20/2021 2:37 PM, Nix wrote:
> On 19 May 2021, Phil Turmel spake thusly:
>
>> weekly scrubs
>
> *Weekly*? Scrubbing my arrays takes three or four days. If I ran them
> weekly the machine would never have time to do anything else!
Three or four days? How large are your arrays? I have quite a few
arrays up to 48T (64T of spindles), and none of them take more than 12
hours to scrub.
> If the storage machinery on this system is so badly off that it's
> misreading bits more often than that I think I have bigger problems.
That might be the case already.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: My superblocks have gone missing, can't reassemble raid5
2021-05-21 4:07 ` Leslie Rhorer
@ 2021-06-07 9:55 ` Leslie Rhorer
0 siblings, 0 replies; 27+ messages in thread
From: Leslie Rhorer @ 2021-06-07 9:55 UTC (permalink / raw)
To: Linux RAID
Christopher,
What was the outcome? Were you able to recover your array?
^ permalink raw reply [flat|nested] 27+ messages in thread
end of thread, other threads:[~2021-06-07 9:55 UTC | newest]
Thread overview: 27+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-17 4:16 My superblocks have gone missing, can't reassemble raid5 Christopher Thomas
2021-05-17 4:23 ` Christopher Thomas
2021-05-17 6:28 ` Roman Mamedov
2021-05-17 9:30 ` Wols Lists
[not found] ` <CAAMCDec=H=6ceP9bKjSnsQyvmZ0LqTAYzJTDmDQoBOHSJV+hDw@mail.gmail.com>
2021-05-17 13:19 ` Roman Mamedov
2021-05-18 17:47 ` Phil Turmel
2021-05-18 18:31 ` Reindl Harald
2021-05-19 13:20 ` Leslie Rhorer
2021-05-19 13:41 ` Phil Turmel
2021-05-19 16:54 ` Leslie Rhorer
2021-05-20 19:37 ` Nix
2021-06-07 9:52 ` Leslie Rhorer
2021-05-19 14:20 ` Andy Smith
2021-05-19 14:59 ` Leslie Rhorer
2021-05-19 14:48 ` Leslie Rhorer
2021-05-19 16:41 ` antlists
2021-05-19 17:03 ` Leslie Rhorer
2021-05-19 17:08 ` Leslie Rhorer
2021-05-19 18:00 ` Wols Lists
2021-05-19 19:01 ` Leslie Rhorer
2021-05-19 20:01 ` antlists
2021-05-19 23:45 ` Leslie Rhorer
2021-05-20 20:49 ` Nix
2021-05-21 4:07 ` Leslie Rhorer
2021-06-07 9:55 ` Leslie Rhorer
2021-05-20 20:48 ` Nix
2021-05-21 3:56 ` Leslie Rhorer
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.