Mixing partitioned and non-partitioned discs in a RAID?

Mixing partitioned and non-partitioned discs in a RAID?

[Bearcat Şándor]

I have a btrfs non-raid file system that i'd like to convert to
raid10. This single device has my efi boot partion on it, so it's
partitioned into sda1 and sda2. I have 3 other discs (sdc-sde) that
i'd like to make partition-less systems and then add them to the first
disc (sda) using the steps in the conversion section of the wiki
(https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices#Conversion).

My concern in that i'll be mixing a partitioned and partition-less
system.  Will that work or will i hose myself?

Thank you

Bearcat

Re: Mixing partitioned and non-partitioned discs in a RAID?

[Andrei Borzenkov]

20.08.2016 07:00, Bearcat Şándor пишет:
> I have a btrfs non-raid file system that i'd like to convert to
> raid10. This single device has my efi boot partion on it, so it's
> partitioned into sda1 and sda2. I have 3 other discs (sdc-sde) that
> i'd like to make partition-less systems and then add them to the first
> disc (sda) using the steps in the conversion section of the wiki
> (https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices#Conversion).
> 
> My concern in that i'll be mixing a partitioned and partition-less
> system.  Will that work or will i hose myself?
> 
Assuming all disks are of equal size, you will lose space on
non-partitioned disks which is at least equal to your EFI boot
partition. So it does not offer any advantages; in general the more
uniform configuration, the better.

Re: Mixing partitioned and non-partitioned discs in a RAID?

[Duncan]

Bearcat Şándor posted on Fri, 19 Aug 2016 22:00:17 -0600 as excerpted:

> I have a btrfs non-raid file system that i'd like to convert to raid10.
> This single device has my efi boot partion on it, so it's partitioned
> into sda1 and sda2. I have 3 other discs (sdc-sde) that i'd like to make
> partition-less systems and then add them to the first disc (sda) using
> the steps in the conversion section of the wiki
> (https://btrfs.wiki.kernel.org/index.php/
Using_Btrfs_with_Multiple_Devices#Conversion).
> 
> My concern in that i'll be mixing a partitioned and partition-less
> system.  Will that work or will i hose myself?

It should work fine as far as btrfs is concerned.

Since you mentioned raid10, however, it's worth noting a difference 
between btrfs raid10 and conventional raid10 that takes some people by 
surprise, tho I don't know whether it'll be a factor for you or not.  
Btrfs raid is per-chunk, not per-device, and different chunks can be 
assigned differently.  What that tends to mean in practice is that while 
with conventional raid10 you can lose a second device without loss of 
data as long as it's from the other raid1 underneath the raid0, with 
btrfs raid10, losing a second device is likely to lose data, because with 
the per-chunk assignment, it's relatively likely that the two bad devices 
will contain both copies of the data for at least some chunks.  So with 
btrfs raid10, don't expect to be able to lose more than one device 
without data-loss, no matter /which/ devices they are.

One way around that, that still retains the checksum error correction 
features of btrfs raid1/10, is to put btrfs raid1 on top of a pair of md/
dmraid0s.  That's normally considered a raid01 instead of a raid10 and is 
discouraged as it makes rebuild more difficult, but btrfs' checksummed 
data error detection and with raid1/10, correction from the remaining 
good copy, makes it worth doing anyway.   And that way you can for 
instance put one of the underlying raid0s in one data cabinet and the 
other in another, and lose the connection to everything in the one, 
without loss of data as raid0 on the other leg of the btrfs raid1 will 
still be operating.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Mixing partitioned and non-partitioned discs in a RAID?

[Chris Murphy]

On Fri, Aug 19, 2016 at 10:00 PM, Bearcat Şándor
<bearcatsandor@gmail.com> wrote:
> I have a btrfs non-raid file system that i'd like to convert to
> raid10. This single device has my efi boot partion on it, so it's
> partitioned into sda1 and sda2. I have 3 other discs (sdc-sde) that
> i'd like to make partition-less systems and then add them to the first
> disc (sda) using the steps in the conversion section of the wiki
> (https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices#Conversion).
>
> My concern in that i'll be mixing a partitioned and partition-less
> system.  Will that work or will i hose myself?

That will work, but you'll likely be in a hosed situation later should
you ever lose a drive so I don't recommend this layout.

a. You have the ESP on only one drive, so it's still a single point of
failure. If that drive dies you can't boot. There are various ways
around that but they all pretty much suck right now from an end user
perspective. Basically redundant boot on UEFI with Linux right now is
pretty much shit. You have to be a bootloader expert to set it up,
maintain it with software updates, and fix it should things go bad.

b. Current udev and btrfs device ready code means degraded boot isn't
possible, because any missing device means the volume isn't ready, so
systemd won't even try to mount it (even with -o degraded). If you
deleted this udev rule, you run the risk of some boots sometimes
having a slightly delayed drive becoming available and the volume
wrongly being mounted degraded when it's not necessary. And since
Btrfs does not automatically fix up delayed devices later, you run the
risk of data loss later on; or worse, corruption of the entire file
system is possible if more than one device becomes stale at the same
time.

Between a. and b. it requires so much attention that it's just not
worth it at all. If you're really doing this to avoid downtime and the
ability to reboot degraded, the only two fairly easy and reliable ways
to do this on UEFI is firmware or hardware RAID. Firmware RAID permits
assembly of the RAID in firmware, so even the ESP can be mirrored, and
then there's a handoff during boot to the md driver during normal
operation and is managed by mdadm.


-- 
Chris Murphy

Re: Mixing partitioned and non-partitioned discs in a RAID?

[Chris Murphy]

On Sat, Aug 20, 2016 at 9:21 AM, Chris Murphy <lists@colorremedies.com> wrote:

>If you
> deleted this udev rule, you run the risk of some boots sometimes
> having a slightly delayed drive becoming available and the volume
> wrongly being mounted degraded when it's not necessary.

Yuck. Revision time!

If you were to delete this udev rule, it's possible to end up with an
unnecessary degraded mount, simply due to a drive spinning up slower
or otherwise having delayed readiness.

Add on: Silently always mounting degraded in order to anticipate
needing unattended degraded mounts will almost certainly lead to data
loss. Btrfs just isn't ready for this yet. It has no concept of device
faulty state still although patches for this are available for testing
(not merged). So yeah, buyer beware. If you don't care about uptime
and unattended operation, just self-healing during normal operation,
then it's OK but you have to remain vigilant, and aware of Btrfs
limitations, including it's lazy repair of formerly missing devices
that reappear. It only fixes passively on reads. To completely fix it,
you have to do a scrub, and you must do it before another drive goes
missing or you can lose the whole file system. Btrfs has no partial
resync like what's offered with mdadm arrays with a write-intent
bitmap.



-- 
Chris Murphy

Re: Mixing partitioned and non-partitioned discs in a RAID?

[Chris Murphy]

On Sat, Aug 20, 2016 at 1:38 PM, Alberto Bursi <alberto.bursi@outlook.it> wrote:
>
>
> On 08/20/2016 05:21 PM, Chris Murphy wrote:
>>
>> On Fri, Aug 19, 2016 at 10:00 PM, Bearcat Şándor
>> <bearcatsandor@gmail.com> wrote:
>>>
>>> I have a btrfs non-raid file system that i'd like to convert to
>>> raid10. This single device has my efi boot partion on it, so it's
>>> partitioned into sda1 and sda2. I have 3 other discs (sdc-sde) that
>>> i'd like to make partition-less systems and then add them to the first
>>> disc (sda) using the steps in the conversion section of the wiki
>>>
>>> (https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices#Conversion).
>>>
>>> My concern in that i'll be mixing a partitioned and partition-less
>>> system.  Will that work or will i hose myself?
>>
>> That will work, but you'll likely be in a hosed situation later should
>> you ever lose a drive so I don't recommend this layout.
>>
>> a. You have the ESP on only one drive, so it's still a single point of
>> failure. If that drive dies you can't boot. There are various ways
>> around that but they all pretty much suck right now from an end user
>> perspective. Basically redundant boot on UEFI with Linux right now is
>> pretty much shit. You have to be a bootloader expert to set it up,
>> maintain it with software updates, and fix it should things go bad.
>
> The usual solution to this is keeping the ESP on a separate (small) flash
> drive.

It's a good idea. It'd be nice if installers offered to put the ESP on
non-boot USB (so it's not overwriting the installer media).



> I still vehemently hate the fact that they couldn't provide like 2 MB of
> space on an onboard NOR flash chip to store a bootloader. Seriously, rEFInd
> with filesystem drivers to boot off btrfs and ext and whatever needs like
> 1.5 MB.


Needs to be bigger. The secure boot stuff takes up more room than
this, about 6-8MiB depending on the distro. And then more room needed
for dual booting let alone triple, which of course no manufacturer
really cares about.


> But no, they need to make a stupid system that is hardcoded to a disk, and
> to a crappy filesystem.

Ahh well the onboard flash could just appear as a small SATA or sdcard
or USB mass storage devices, whatever's easiest to implement.

FAT leaves a lot to be desired but it's pretty universally supported
and almost trivial to repair *if* the volume is repairable in the
first place. The much bigger issue with ESP on Linux is this neurotic
tendency of distros to persistently mount shit that does not need to
be mounted. Like the ESP, and even the dedicated boot volume. They
only need to be mounted when being updated and then should be
umounted. And worse the convention is to do nested mount with /boot
and then /boot/efi for the ESP so it's twice as bad a practice. By
virtue of mounting the ESP the dirty bit is set, so any crash means it
must be fsck'd and if that doesn't work, it's game over for that
volume. Fragile setup.

-- 
Chris Murphy

Re: Mixing partitioned and non-partitioned discs in a RAID?

[Duncan]

Chris Murphy posted on Sat, 20 Aug 2016 18:36:21 -0600 as excerpted:

> FAT leaves a lot to be desired but it's pretty universally supported and
> almost trivial to repair *if* the volume is repairable in the first
> place. The much bigger issue with ESP on Linux is this neurotic tendency
> of distros to persistently mount shit that does not need to be mounted.
> Like the ESP, and even the dedicated boot volume. They only need to be
> mounted when being updated and then should be umounted. And worse the
> convention is to do nested mount with /boot and then /boot/efi for the
> ESP so it's twice as bad a practice. By virtue of mounting the ESP the
> dirty bit is set, so any crash means it must be fsck'd and if that
> doesn't work, it's game over for that volume. Fragile setup.

Depends on the distro.  On gentoo, you set it up the way you want of 
course, but the recommendation has always been /boot, and now the ESP, 
not mounted by default.

But that would be /expected/ on gentoo, since being able to configure it 
the way you want is the whole /point/ of running gentoo in the first 
place.  Sort of like arch, only much more so.

Meanwhile, I'm kinda partial to booting in legacy mode in grub2, with 
grub2 installed to the legacy-BIOS partion on multiple gpt-partitioned 
devices, which each one having its own /boot as well, setup so the 
firmware boot device selector can choose which device and thus which 
grub2 installation on that device, I boot from.  That gives me as many 
backup grub2 installations on the legacy BIOS partitions, pointing at as 
many backup /boots on the same device by default, as I care to setup.  Of 
course what's extra nice about that is that you can switch to the /boot 
on another device direct from the grub2 prompt, emergency mode if it 
can't load the /boot off that device, if you like.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Mixing partitioned and non-partitioned discs in a RAID?

[Bearcat Şándor]

Thanks for the responses all. I'm on Gentoo as well for what that
matters. I think what i'm going to do (if i don't abandon EFI
entirely) is just to get a small (16gb), $25 m.2 since my board
supports that, and use that as my boot drive. Then mkfs other all
discs to be partitionless and away i go.

Not sure what i'll use the extra 15g on that m.2 for.

I learned a lot from your long response Chris. Thank you.

On Sat, Aug 20, 2016 at 8:19 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Chris Murphy posted on Sat, 20 Aug 2016 18:36:21 -0600 as excerpted:
>
>> FAT leaves a lot to be desired but it's pretty universally supported and
>> almost trivial to repair *if* the volume is repairable in the first
>> place. The much bigger issue with ESP on Linux is this neurotic tendency
>> of distros to persistently mount shit that does not need to be mounted.
>> Like the ESP, and even the dedicated boot volume. They only need to be
>> mounted when being updated and then should be umounted. And worse the
>> convention is to do nested mount with /boot and then /boot/efi for the
>> ESP so it's twice as bad a practice. By virtue of mounting the ESP the
>> dirty bit is set, so any crash means it must be fsck'd and if that
>> doesn't work, it's game over for that volume. Fragile setup.
>
> Depends on the distro.  On gentoo, you set it up the way you want of
> course, but the recommendation has always been /boot, and now the ESP,
> not mounted by default.
>
> But that would be /expected/ on gentoo, since being able to configure it
> the way you want is the whole /point/ of running gentoo in the first
> place.  Sort of like arch, only much more so.
>
> Meanwhile, I'm kinda partial to booting in legacy mode in grub2, with
> grub2 installed to the legacy-BIOS partion on multiple gpt-partitioned
> devices, which each one having its own /boot as well, setup so the
> firmware boot device selector can choose which device and thus which
> grub2 installation on that device, I boot from.  That gives me as many
> backup grub2 installations on the legacy BIOS partitions, pointing at as
> many backup /boots on the same device by default, as I care to setup.  Of
> course what's extra nice about that is that you can switch to the /boot
> on another device direct from the grub2 prompt, emergency mode if it
> can't load the /boot off that device, if you like.
>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Bearcat M. Şándor
Feline Soul Systems LLC
Voice: 872.CAT.SOUL (872.228.7685)
Fax: 406.235.7070

Re: Mixing partitioned and non-partitioned discs in a RAID?

[Kai Krakow]

Am Sun, 21 Aug 2016 02:19:33 +0000 (UTC)
schrieb Duncan <1i5t5.duncan@cox.net>:

> Chris Murphy posted on Sat, 20 Aug 2016 18:36:21 -0600 as excerpted:
> 
> > FAT leaves a lot to be desired but it's pretty universally
> > supported and almost trivial to repair *if* the volume is
> > repairable in the first place. The much bigger issue with ESP on
> > Linux is this neurotic tendency of distros to persistently mount
> > shit that does not need to be mounted. Like the ESP, and even the
> > dedicated boot volume. They only need to be mounted when being
> > updated and then should be umounted. And worse the convention is to
> > do nested mount with /boot and then /boot/efi for the ESP so it's
> > twice as bad a practice. By virtue of mounting the ESP the dirty
> > bit is set, so any crash means it must be fsck'd and if that
> > doesn't work, it's game over for that volume. Fragile setup.  
> 
> Depends on the distro.  On gentoo, you set it up the way you want of 
> course, but the recommendation has always been /boot, and now the
> ESP, not mounted by default.
> 
> But that would be /expected/ on gentoo, since being able to configure
> it the way you want is the whole /point/ of running gentoo in the
> first place.  Sort of like arch, only much more so.

systemd systems (I'm booting Gentoo with systemd) should auto-mount ESP
to /boot on access, and auto-unmount after a short timeout. So the
solution to this problem is already wired into systemd if you use (a)
proper GPT setup (with correct GUIDs) and (b) do not mention /boot in
fstab.

-- 
Regards,
Kai

Replies to list-only preferred.

Re: Mixing partitioned and non-partitioned discs in a RAID?

[Duncan]

Kai Krakow posted on Tue, 13 Sep 2016 00:21:10 +0200 as excerpted:

> Am Sun, 21 Aug 2016 02:19:33 +0000 (UTC)
> schrieb Duncan <1i5t5.duncan@cox.net>:
> 
>> Chris Murphy posted on Sat, 20 Aug 2016 18:36:21 -0600 as excerpted:
>> 
>> > FAT leaves a lot to be desired but it's pretty universally supported
>> > and almost trivial to repair *if* the volume is repairable in the
>> > first place. The much bigger issue with ESP on Linux is this neurotic
>> > tendency of distros to persistently mount shit that does not need to
>> > be mounted. Like the ESP, and even the dedicated boot volume. They
>> > only need to be mounted when being updated and then should be
>> > umounted. And worse the convention is to do nested mount with /boot
>> > and then /boot/efi for the ESP so it's twice as bad a practice. By
>> > virtue of mounting the ESP the dirty bit is set, so any crash means
>> > it must be fsck'd and if that doesn't work, it's game over for that
>> > volume. Fragile setup.
>> 
>> Depends on the distro.  On gentoo, you set it up the way you want of
>> course, but the recommendation has always been /boot, and now the ESP,
>> not mounted by default.
>> 
>> But that would be /expected/ on gentoo, since being able to configure
>> it the way you want is the whole /point/ of running gentoo in the first
>> place.  Sort of like arch, only much more so.
> 
> systemd systems (I'm booting Gentoo with systemd) should auto-mount ESP
> to /boot on access, and auto-unmount after a short timeout. So the
> solution to this problem is already wired into systemd if you use (a)
> proper GPT setup (with correct GUIDs) and (b) do not mention /boot in
> fstab.

No "automagic" on-access mounting here.  The kernel options aren't turned 
on for it, neither do I want them on (the systemd ebuild checks for and 
recommends but does not force them, and systemd functions fine without 
them except it doesn't automount on access, which is pretty much the 
point).  What gets mounted on boot is what I have in fstab (plus a few 
things like cgroups that systemd handles entirely on its own).  After 
that, nothing else is mounted unless I mount it, or unmounted, unless I 
umount it or systemd does it as part of shutdown, etc.

However, good point about systemd, since it should more or less 
standardize handling across mainline distros over the longer term, except 
of course where admins/distros specifically configure it otherwise, as 
I've done here.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Mixing partitioned and non-partitioned discs in a RAID?

[Chris Murphy]

On Mon, Sep 12, 2016 at 4:21 PM, Kai Krakow <hurikhan77@gmail.com> wrote:
> Am Sun, 21 Aug 2016 02:19:33 +0000 (UTC)
> schrieb Duncan <1i5t5.duncan@cox.net>:
>
>> Chris Murphy posted on Sat, 20 Aug 2016 18:36:21 -0600 as excerpted:
>>
>> > FAT leaves a lot to be desired but it's pretty universally
>> > supported and almost trivial to repair *if* the volume is
>> > repairable in the first place. The much bigger issue with ESP on
>> > Linux is this neurotic tendency of distros to persistently mount
>> > shit that does not need to be mounted. Like the ESP, and even the
>> > dedicated boot volume. They only need to be mounted when being
>> > updated and then should be umounted. And worse the convention is to
>> > do nested mount with /boot and then /boot/efi for the ESP so it's
>> > twice as bad a practice. By virtue of mounting the ESP the dirty
>> > bit is set, so any crash means it must be fsck'd and if that
>> > doesn't work, it's game over for that volume. Fragile setup.
>>
>> Depends on the distro.  On gentoo, you set it up the way you want of
>> course, but the recommendation has always been /boot, and now the
>> ESP, not mounted by default.
>>
>> But that would be /expected/ on gentoo, since being able to configure
>> it the way you want is the whole /point/ of running gentoo in the
>> first place.  Sort of like arch, only much more so.
>
> systemd systems (I'm booting Gentoo with systemd) should auto-mount ESP
> to /boot on access, and auto-unmount after a short timeout. So the
> solution to this problem is already wired into systemd if you use (a)
> proper GPT setup (with correct GUIDs) and (b) do not mention /boot in
> fstab.

What distro uses this layout by default? It's certainly incompatible
with multiboot where either Windows or macOS exist first. It also
necessitates an excessively large ESP to store all possible kernels
and initrds for every OS, making it all the harder to keep sync'd in
any case where even a simple raid1 setup is desired. The ESP is a file
system that doesn't support permissions, acls, or security labels
either. So I'm not really a fan of this, or even nested /boot/efi
either for that matter.  They should be mounted in /run only on demand
and only by the process that has permission to modify each.



-- 
Chris Murphy

Re: Mixing partitioned and non-partitioned discs in a RAID?

[Kai Krakow]

Am Tue, 13 Sep 2016 04:07:37 +0000 (UTC)
schrieb Duncan <1i5t5.duncan@cox.net>:

> Kai Krakow posted on Tue, 13 Sep 2016 00:21:10 +0200 as excerpted:
> 
> > Am Sun, 21 Aug 2016 02:19:33 +0000 (UTC)
> > schrieb Duncan <1i5t5.duncan@cox.net>:
> >   
> >> Chris Murphy posted on Sat, 20 Aug 2016 18:36:21 -0600 as
> >> excerpted: 
>  [...]  
> >> 
> >> Depends on the distro.  On gentoo, you set it up the way you want
> >> of course, but the recommendation has always been /boot, and now
> >> the ESP, not mounted by default.
> >> 
> >> But that would be /expected/ on gentoo, since being able to
> >> configure it the way you want is the whole /point/ of running
> >> gentoo in the first place.  Sort of like arch, only much more so.  
> > 
> > systemd systems (I'm booting Gentoo with systemd) should auto-mount
> > ESP to /boot on access, and auto-unmount after a short timeout. So
> > the solution to this problem is already wired into systemd if you
> > use (a) proper GPT setup (with correct GUIDs) and (b) do not
> > mention /boot in fstab.  
> 
> No "automagic" on-access mounting here.  The kernel options aren't
> turned on for it, neither do I want them on (the systemd ebuild
> checks for and recommends but does not force them, and systemd
> functions fine without them except it doesn't automount on access,
> which is pretty much the point).  What gets mounted on boot is what I
> have in fstab (plus a few things like cgroups that systemd handles
> entirely on its own).  After that, nothing else is mounted unless I
> mount it, or unmounted, unless I umount it or systemd does it as part
> of shutdown, etc.

https://www.freedesktop.org/software/systemd/man/systemd-gpt-auto-generator.html

FD> Mount and automount units for the EFI System Partition (ESP),
FD> mounting it to /boot, are generated on EFI systems where the boot
FD> loader communicates the used ESP to the operating system. Since
FD> this generator creates an automount unit, the mount will only be
FD> activated on-demand, when accessed. On systems where /boot is an
FD> explicitly configured mount (for example, listed in fstab(5)) or
FD> where the /boot mount point is non-empty, no mount units are
FD> generated.

So there are a few points when this does not happen. It does not
blindly auto-mount ESP.

> However, good point about systemd, since it should more or less
> standardize handling across mainline distros over the longer term,
> except of course where admins/distros specifically configure it
> otherwise, as I've done here.
 
I also explicitly put it in fstab - so the generator doesn't work for
me.


-- 
Regards,
Kai

Replies to list-only preferred.