Install to or Recover RAID Array Subvolume Root?

Install to or Recover RAID Array Subvolume Root?

[David Alcorn]

Debian's default installer (1) can not create a BTRFS raid array
during installation, and (2) installs to the default subvol of the
BTRFS target.  The default subvol is 5 (BTRFS root) unless (i) prior
to installation a BTRFS file-system was created, (ii) the default
subvol is set to something other than 5, and (iii) you do not format
the installation target during install. This perspective benefits from
https://aykevl.nl/2015/11/debian-btrfs-subvolume.

Can Debian install BTRFS to a pre-existing BTRFS RAID 6 array?  If the
array's default is set to ,say subvolid 257, can I install to array
subvolume 257?  The goal is to install to a subvolume on the array
without disturbing date on other array subvolumes.

I erred and shutdown my NAS during a balance.  Grub lost track of my
root.  Root was on RAID 6 array subvolid 257.  I can boot a different
root from a USB flash drive but neither update-grub not install-grub
sees my old root on array subvolid 257.  I am happy to either recover
or lose array subvolid 257 but do not want to lose data on other array
subvol's.  I prefer to have my root on the array rather than a flash
drive.  The balance completed successfully after I booted from the
flash drive.

I am running a debian back-ported 4.4.0 kernel with btrfs-progs v4.4
on both my flash drive and array subvolid 257.  Both installs are
UEFI.  df -h reports:

Filesystem      Size  Used Avail Use% Mounted on
udev             10M     0   10M   0% /dev
tmpfs           790M  8.6M  781M   2% /run
/dev/sdb3       3.8G  1.1G  2.4G  31% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/sdb1        93M  311K   93M   1% /boot/efi
/dev/sde         22T  4.9T   13T  29% /lit

Re: Install to or Recover RAID Array Subvolume Root?

[Austin S. Hemmelgarn]

On 2016-04-18 01:22, David Alcorn wrote:
> Debian's default installer (1) can not create a BTRFS raid array
> during installation, and (2) installs to the default subvol of the
> BTRFS target.  The default subvol is 5 (BTRFS root) unless (i) prior
> to installation a BTRFS file-system was created, (ii) the default
> subvol is set to something other than 5, and (iii) you do not format
> the installation target during install. This perspective benefits from
> https://aykevl.nl/2015/11/debian-btrfs-subvolume.
>
> Can Debian install BTRFS to a pre-existing BTRFS RAID 6 array?  If the
> array's default is set to ,say subvolid 257, can I install to array
> subvolume 257?  The goal is to install to a subvolume on the array
> without disturbing date on other array subvolumes.
This is really something you should be asking on the Debian mailing 
list, or on one of their IRC channels.   That said,  based on my limited 
knowledge of the current state of the Debian installer, I think it 
should work, but I"m not certain.  I don't really use Debian much 
anymore except on a couple of Raspberry Pi's at work, so I really can't 
give a much better answer (Ironically, limitations like this in the 
installer are part of why I switched to Gentoo).
>
> I erred and shutdown my NAS during a balance.  Grub lost track of my
> root.  Root was on RAID 6 array subvolid 257.  I can boot a different
> root from a USB flash drive but neither update-grub not install-grub
> sees my old root on array subvolid 257.  I am happy to either recover
> or lose array subvolid 257 but do not want to lose data on other array
> subvol's.  I prefer to have my root on the array rather than a flash
> drive.  The balance completed successfully after I booted from the
> flash drive.
>
> I am running a debian back-ported 4.4.0 kernel with btrfs-progs v4.4
> on both my flash drive and array subvolid 257.  Both installs are
> UEFI.
It's rather refreshing to see somebody using an up to date kernel and 
userspace for once.  The issue with GRUB is however likely an issue with 
GRUB itself though (or possibly with something it uses for detecting 
filesystems), which brings up the question: How recent is the version of 
GRUB you're trying to install?  I would not be surprised if their 
current support for BTRFS raid5 and raid6 is not particularly good, 
especially considering that their support for BTRFS raid1 was kind of 
shoddy at first (and it still has occasional issues, like not working 
right half the time if one of the devices is missing).

Re: Install to or Recover RAID Array Subvolume Root?

[Chris Murphy]

On Mon, Apr 18, 2016 at 6:31 AM, Austin S. Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2016-04-18 01:22, David Alcorn wrote:
>>
>> Debian's default installer (1) can not create a BTRFS raid array
>> during installation, and (2) installs to the default subvol of the
>> BTRFS target.  The default subvol is 5 (BTRFS root) unless (i) prior
>> to installation a BTRFS file-system was created, (ii) the default
>> subvol is set to something other than 5, and (iii) you do not format
>> the installation target during install. This perspective benefits from
>> https://aykevl.nl/2015/11/debian-btrfs-subvolume.
>>
>> Can Debian install BTRFS to a pre-existing BTRFS RAID 6 array?  If the
>> array's default is set to ,say subvolid 257, can I install to array
>> subvolume 257?  The goal is to install to a subvolume on the array
>> without disturbing date on other array subvolumes.
>
> This is really something you should be asking on the Debian mailing list, or
> on one of their IRC channels.   That said,  based on my limited knowledge of
> the current state of the Debian installer, I think it should work, but I"m
> not certain.  I don't really use Debian much anymore except on a couple of
> Raspberry Pi's at work, so I really can't give a much better answer
> (Ironically, limitations like this in the installer are part of why I
> switched to Gentoo).
>>
>>
>> I erred and shutdown my NAS during a balance.  Grub lost track of my
>> root.  Root was on RAID 6 array subvolid 257.  I can boot a different
>> root from a USB flash drive but neither update-grub not install-grub
>> sees my old root on array subvolid 257.  I am happy to either recover
>> or lose array subvolid 257 but do not want to lose data on other array
>> subvol's.  I prefer to have my root on the array rather than a flash
>> drive.  The balance completed successfully after I booted from the
>> flash drive.
>>
>> I am running a debian back-ported 4.4.0 kernel with btrfs-progs v4.4
>> on both my flash drive and array subvolid 257.  Both installs are
>> UEFI.
>
> It's rather refreshing to see somebody using an up to date kernel and
> userspace for once.  The issue with GRUB is however likely an issue with
> GRUB itself though (or possibly with something it uses for detecting
> filesystems), which brings up the question: How recent is the version of
> GRUB you're trying to install?  I would not be surprised if their current
> support for BTRFS raid5 and raid6 is not particularly good, especially
> considering that their support for BTRFS raid1 was kind of shoddy at first
> (and it still has occasional issues, like not working right half the time if
> one of the devices is missing).

I can't tell if this setup is Btrfs on md/mdadm raid6, or if it's
Btrfs raid6? GRUB has supported mdadm raid6 for a long time, but I
don't think it understands Btrfs raid56 at all.

The only GUI installer I know that explicitly permits installation of
an OS to an existing Btrfs is Anaconda (mainly Red Hat and Fedora).
There's a bug in the storage backend that prevents this from working
in Fedora 23, exposed by a Btrfs change disallowing simultaneous use
of subvolid and subvol, resulting in mount failure, which in turn
causes the installer to crash. But it works in Fedora 22, and it will
work in Fedora 24. On the other hand, no Fedora supports Btrfs for
/boot, it has to be separate and it has to be ext4 or XFS (and maybe
FAT).



-- 
Chris Murphy

Re: Install to or Recover RAID Array Subvolume Root?

[Austin S. Hemmelgarn]

On 2016-04-18 11:12, Chris Murphy wrote:
> On Mon, Apr 18, 2016 at 6:31 AM, Austin S. Hemmelgarn
> <ahferroin7@gmail.com> wrote:
>> On 2016-04-18 01:22, David Alcorn wrote:
>>>
>>>
>>> I erred and shutdown my NAS during a balance.  Grub lost track of my
>>> root.  Root was on RAID 6 array subvolid 257.  I can boot a different
>>> root from a USB flash drive but neither update-grub not install-grub
>>> sees my old root on array subvolid 257.  I am happy to either recover
>>> or lose array subvolid 257 but do not want to lose data on other array
>>> subvol's.  I prefer to have my root on the array rather than a flash
>>> drive.  The balance completed successfully after I booted from the
>>> flash drive.
>>>
>>> I am running a debian back-ported 4.4.0 kernel with btrfs-progs v4.4
>>> on both my flash drive and array subvolid 257.  Both installs are
>>> UEFI.
>>
>> It's rather refreshing to see somebody using an up to date kernel and
>> userspace for once.  The issue with GRUB is however likely an issue with
>> GRUB itself though (or possibly with something it uses for detecting
>> filesystems), which brings up the question: How recent is the version of
>> GRUB you're trying to install?  I would not be surprised if their current
>> support for BTRFS raid5 and raid6 is not particularly good, especially
>> considering that their support for BTRFS raid1 was kind of shoddy at first
>> (and it still has occasional issues, like not working right half the time if
>> one of the devices is missing).
>
> I can't tell if this setup is Btrfs on md/mdadm raid6, or if it's
> Btrfs raid6? GRUB has supported mdadm raid6 for a long time, but I
> don't think it understands Btrfs raid56 at all.
>
> The only GUI installer I know that explicitly permits installation of
> an OS to an existing Btrfs is Anaconda (mainly Red Hat and Fedora).
> There's a bug in the storage backend that prevents this from working
> in Fedora 23, exposed by a Btrfs change disallowing simultaneous use
> of subvolid and subvol, resulting in mount failure, which in turn
> causes the installer to crash. But it works in Fedora 22, and it will
> work in Fedora 24. On the other hand, no Fedora supports Btrfs for
> /boot, it has to be separate and it has to be ext4 or XFS (and maybe
> FAT).
>
I don't know about the current state of the Debian installer, but I know 
back when I used Debian regularly and used the standard text based 
installer, as long as I didn't format things from the UI, I could 
provision the filesystems however the hell I wanted manually and point 
the installer at the appropriate ones for each mount point, and it worked.

Re: Install to or Recover RAID Array Subvolume Root?

[Chris Murphy]

On Mon, Apr 18, 2016 at 9:15 AM, Austin S. Hemmelgarn
<ahferroin7@gmail.com> wrote:
>
>>
> I don't know about the current state of the Debian installer, but I know
> back when I used Debian regularly and used the standard text based
> installer, as long as I didn't format things from the UI, I could provision
> the filesystems however the hell I wanted manually and point the installer
> at the appropriate ones for each mount point, and it worked.

No doubt this is a catch 22. I think it's reasonable for an installer
to have an aging logic that allows it to be more conservative than the
kernel's support for compat_ro_flags and incompat_flags, and refuse to
install if it's not recently formatted.

Now that I go look for this, I'm not sure what either flags offset
0x38 or compat_flags offset 0xac are about.

Also, I didn't mention that Anaconda does require root fs to be in a
new subvolume. It won't install to either subvolid 5, or any other
existing subvolume.

-- 
Chris Murphy

Re: Install to or Recover RAID Array Subvolume Root?

[Austin S. Hemmelgarn]

On 2016-04-18 11:39, Chris Murphy wrote:
> On Mon, Apr 18, 2016 at 9:15 AM, Austin S. Hemmelgarn
> <ahferroin7@gmail.com> wrote:
>>
>>>
>> I don't know about the current state of the Debian installer, but I know
>> back when I used Debian regularly and used the standard text based
>> installer, as long as I didn't format things from the UI, I could provision
>> the filesystems however the hell I wanted manually and point the installer
>> at the appropriate ones for each mount point, and it worked.
>
> No doubt this is a catch 22. I think it's reasonable for an installer
> to have an aging logic that allows it to be more conservative than the
> kernel's support for compat_ro_flags and incompat_flags, and refuse to
> install if it's not recently formatted.
>
> Now that I go look for this, I'm not sure what either flags offset
> 0x38 or compat_flags offset 0xac are about.
>
> Also, I didn't mention that Anaconda does require root fs to be in a
> new subvolume. It won't install to either subvolid 5, or any other
> existing subvolume.
>
I'm not certain about how Debian would handle it, but I think they'll 
assume that the user knows what they're doing and just do what they say 
(which can admittedly be a very dangerous assumption sometimes).

Like I said in one of my earlier e-mails though, these kind of 
limitations are part of why I switched to Gentoo, there's no GUI 
installer, but you can put the system together however the hell you want 
(which is especially nice with BTRFS, because none of the installers out 
there will let you use BTRFS on top of LVM, which is useful for things 
like BTRFS raid1 on top of DM-RAID0).

Re: Install to or Recover RAID Array Subvolume Root?

[Chris Murphy]

On Mon, Apr 18, 2016 at 9:52 AM, Austin S. Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2016-04-18 11:39, Chris Murphy wrote:
>>
>> On Mon, Apr 18, 2016 at 9:15 AM, Austin S. Hemmelgarn
>> <ahferroin7@gmail.com> wrote:
>>>
>>>
>>>>
>>> I don't know about the current state of the Debian installer, but I know
>>> back when I used Debian regularly and used the standard text based
>>> installer, as long as I didn't format things from the UI, I could
>>> provision
>>> the filesystems however the hell I wanted manually and point the
>>> installer
>>> at the appropriate ones for each mount point, and it worked.
>>
>>
>> No doubt this is a catch 22. I think it's reasonable for an installer
>> to have an aging logic that allows it to be more conservative than the
>> kernel's support for compat_ro_flags and incompat_flags, and refuse to
>> install if it's not recently formatted.
>>
>> Now that I go look for this, I'm not sure what either flags offset
>> 0x38 or compat_flags offset 0xac are about.
>>
>> Also, I didn't mention that Anaconda does require root fs to be in a
>> new subvolume. It won't install to either subvolid 5, or any other
>> existing subvolume.
>>
> I'm not certain about how Debian would handle it, but I think they'll assume
> that the user knows what they're doing and just do what they say (which can
> admittedly be a very dangerous assumption sometimes).
>
> Like I said in one of my earlier e-mails though, these kind of limitations
> are part of why I switched to Gentoo, there's no GUI installer, but you can
> put the system together however the hell you want (which is especially nice
> with BTRFS, because none of the installers out there will let you use BTRFS
> on top of LVM, which is useful for things like BTRFS raid1 on top of
> DM-RAID0).

In a GUI context, the OS installer should be point and shoot to a
volume. It shouldn't care how it was built. Partitioning is a separate
thing. But in Linux land, these things get conflated, and makes them
more complicated and buggy. The result is excessive resources
maintaining and fixing them all the time.


-- 
Chris Murphy

Re: Install to or Recover RAID Array Subvolume Root?

[Nicholas D Steeves]

On 18 April 2016 at 11:52, Austin S. Hemmelgarn <ahferroin7@gmail.com> wrote:
> On 2016-04-18 11:39, Chris Murphy wrote:
>>
>> On Mon, Apr 18, 2016 at 9:15 AM, Austin S. Hemmelgarn
>> <ahferroin7@gmail.com> wrote:
> Like I said in one of my earlier e-mails though, these kind of limitations
> are part of why I switched to Gentoo, there's no GUI installer, but you can
> put the system together however the hell you want (which is especially nice
> with BTRFS, because none of the installers out there will let you use BTRFS
> on top of LVM, which is useful for things like BTRFS raid1 on top of
> DM-RAID0).

Limitations?  ;-)  Debian has a variety of bootstrapping methods,
though I forget if they're more analogous to starting a Gentoo
installation from stage2 or stage3...I haven't used Gentoo since 2002.
Please consult the following doc for one of the methods:
https://www.debian.org/releases/stable/amd64/apds03.html.en

Best regards,
Nicholas

Re: Install to or Recover RAID Array Subvolume Root?

[Nicholas D Steeves]

On 18 April 2016 at 01:22, David Alcorn <nroclaed@gmail.com> wrote:
> The goal is to install to a subvolume on the array
> without disturbing date on other array subvolumes.
>
> I erred and shutdown my NAS during a balance.  Grub lost track of my
> root.  Root was on RAID 6 array subvolid 257.  I can boot a different
> root from a USB flash drive but neither update-grub not install-grub
> sees my old root on array subvolid 257.  I am happy to either recover
> or lose array subvolid 257 but do not want to lose data on other array
> subvol's.  I prefer to have my root on the array rather than a flash
> drive.  The balance completed successfully after I booted from the
> flash drive.

Is your flash drive formatted btrfs?  If it is, you could always
snapshot it, send the snapshot to your array, set property of that
subvolume to RW, chroot, update fstab to mount / with the appropriate
subvol=option, update /etc/default/grub, reinstall grub and
update-grub, and reboot with your / as a subvolume on your array.  I'm
in the process of documenting how to do this on the Debian wiki.
Please let me know if I should put a rush on it.  It uses the subvol=
option rather than changing the volume's default subvol.

Cheers,
Nicholas

Re: Install to or Recover RAID Array Subvolume Root?

[David Alcorn]

Nicolas:

My flash drive uses BTRFS and I am comfortable with your instructions
with one exception.  What does "update /etc/default/grub" mean?

Currently, I am waiting for a scrub to verify that all is in good
order before fixing the problem.


On 4/19/16, Nicholas D Steeves <nsteeves@gmail.com> wrote:
> On 18 April 2016 at 01:22, David Alcorn <nroclaed@gmail.com> wrote:
>> The goal is to install to a subvolume on the array
>> without disturbing date on other array subvolumes.
>>
>> I erred and shutdown my NAS during a balance.  Grub lost track of my
>> root.  Root was on RAID 6 array subvolid 257.  I can boot a different
>> root from a USB flash drive but neither update-grub not install-grub
>> sees my old root on array subvolid 257.  I am happy to either recover
>> or lose array subvolid 257 but do not want to lose data on other array
>> subvol's.  I prefer to have my root on the array rather than a flash
>> drive.  The balance completed successfully after I booted from the
>> flash drive.
>
> Is your flash drive formatted btrfs?  If it is, you could always
> snapshot it, send the snapshot to your array, set property of that
> subvolume to RW, chroot, update fstab to mount / with the appropriate
> subvol=option, update /etc/default/grub, reinstall grub and
> update-grub, and reboot with your / as a subvolume on your array.  I'm
> in the process of documenting how to do this on the Debian wiki.
> Please let me know if I should put a rush on it.  It uses the subvol=
> option rather than changing the volume's default subvol.
>
> Cheers,
> Nicholas
>

Re: Install to or Recover RAID Array Subvolume Root?

[Austin S. Hemmelgarn]

On 2016-04-18 16:34, Nicholas D Steeves wrote:
> On 18 April 2016 at 11:52, Austin S. Hemmelgarn <ahferroin7@gmail.com> wrote:
>> On 2016-04-18 11:39, Chris Murphy wrote:
>>>
>>> On Mon, Apr 18, 2016 at 9:15 AM, Austin S. Hemmelgarn
>>> <ahferroin7@gmail.com> wrote:
>> Like I said in one of my earlier e-mails though, these kind of limitations
>> are part of why I switched to Gentoo, there's no GUI installer, but you can
>> put the system together however the hell you want (which is especially nice
>> with BTRFS, because none of the installers out there will let you use BTRFS
>> on top of LVM, which is useful for things like BTRFS raid1 on top of
>> DM-RAID0).
>
> Limitations?  ;-)  Debian has a variety of bootstrapping methods,
> though I forget if they're more analogous to starting a Gentoo
> installation from stage2 or stage3...I haven't used Gentoo since 2002.
> Please consult the following doc for one of the methods:
> https://www.debian.org/releases/stable/amd64/apds03.html.en
The closest I've ever seen for Debian to a Gentoo stage3 installation 
(the developers discourage usage of stage2 installs these days unless 
you're bootstrapping _everything_ yourself) is debbootstrap, and I could 
never get that to work reliably.

FWIW, the installer wasn't the only reason I switched to Gentoo, the two 
bigger ones for me were wanting newer software versions and needing 
different sets of features enabled on packages than the default builds, 
I ended up switching at the point that I was building more locally than 
I was installing from the repositories.

Re: Install to or Recover RAID Array Subvolume Root?

[Nicholas D Steeves]

On 18 April 2016 at 23:06, David Alcorn <nroclaed@gmail.com> wrote:
> Nicolas:
>
> My flash drive uses BTRFS and I am comfortable with your instructions
> with one exception.  What does "update /etc/default/grub" mean?
>
> Currently, I am waiting for a scrub to verify that all is in good
> order before fixing the problem.

I meant that more as a general precaution and good habit.  The most
common check/change would be to make sure the "resume=foo" option
matches the UUID or /dev/sdX of the swap partition; it's mostly
relevant to laptop users.

More to the point, as Austin and Chris mentioned the tricky bit is
going to get GRUB to boot from raid6 profile btrfs if your /boot is
part of your btrfs volume.  I honestly don't know if it will work...
Do you have a separate /boot partition?  What is your /dev/sda being
used for?  UEFI firmware loads GRUB's EFI payload, which loads the
different stages of grub that allow file system access, which is
necessary for grub to be able to find the kernel.  The EFI payload is
installed to your FAT-formatted ESP partition, which is usually
mounted to /boot/grub/efi/EFI.  I also suspect that without a separate
/boot partition GRUB won't be able to find the kernel
(/boot/vmlinuz-4.4.0-1-amd64).  If I remember correctly GRUB's stage1
talks to your motherboard's firmware, stage2 enables filesystem access
(/boot/grub/x86_64-efi/btrfs.mod), and stage3 loads the kernel.  En
bref, if GRUB has insufficient support for btrfs' raid6 profile then
grub will either be unable to access btrfs.mod, or btrfs.mod will be
unable to enable access /boot/vmlinuz-4.4.0-1-amd64.

I suspect the following worst-case scenario if you don't have a
partition you can use for /boot, and didn't leave any unallocated
space on any of your drives, and if you can't shrink something like a
swap partition to make room for /boot:  No need to backup/restore if
you have a usb port to dedicate to /boot.  A more exotic solution
would be using a small SATADOM to hold it, but then you lose a SATA
port ;-)  After sending the rootfs of your USB flash installation to a
subvolume of your raid6, you can manually use the GRUB command line on
your existing USB stick to attempt to boot the rootfs subvolume of
your raid6.

Cheers,
Nicholas

Re: Install to or Recover RAID Array Subvolume Root?

[Nicholas D Steeves]

On 19 April 2016 at 07:14, Austin S. Hemmelgarn <ahferroin7@gmail.com> wrote:
> The closest I've ever seen for Debian to a Gentoo stage3 installation (the
> developers discourage usage of stage2 installs these days unless you're
> bootstrapping _everything_ yourself) is debbootstrap, and I could never get
> that to work reliably.

Fair, I remember Gentoo stage3+documentation was more straightforward
than debbootstrap+documentation.  There are a couple of wrappers
around debbootstrap and better documentation now, but I think there
are still rare times when bootstrapping from unstable or testing will
fail because a major transition is in progress (like the libc5 ->
libc6 transition, the ulibc -> glibc one, possibly the multiarch or
multilib transition, sometimes a GCC one, etc.)

> FWIW, the installer wasn't the only reason I switched to Gentoo, the two
> bigger ones for me were wanting newer software versions and needing
> different sets of features enabled on packages than the default builds, I
> ended up switching at the point that I was building more locally than I was
> installing from the repositories.

Ah yes, a convenient stream of fresh updates and the power of USE
flags :-)  For my needs, security-only updates + a backport whenever I
need a more up-to-date package of saves me time.  That's what the
choice of tool comes down to, right?  Does what you need it to, and
saves you time.

Best regards,
Nicholas

Re: Install to or Recover RAID Array Subvolume Root?

[David Alcorn]

On 4/19/16, Nicholas D Steeves <nsteeves@gmail.com> wrote:
....
> Is your flash drive formatted btrfs?  If it is, you could always
> snapshot it, send the snapshot to your array, set property of that
> subvolume to RW, chroot, update fstab to mount / with the appropriate
> subvol=option, update /etc/default/grub, reinstall grub and
> update-grub, and reboot with your / as a subvolume on y>
....
>I'm
> in the process of documenting how to do this on the Debian wiki.
> Please let me know if I should put a rush on it.  It uses the subvol=
> option rather than changing the volume's default subvol.
>
> Cheers,
> Nicholas
>


First, I verified that while the Debian Installer will install to a
pre set default BTRFS RAID6 subvolume, the Grub install step fails.
The alternative to restore installation to a RAID6 subvolume requires
installation to a non RAID6 subvolume and then send|receive the
snapshotted installation to the array.  To prepare for this attempt, I
reinstalled BTRFS (Debian stable) to a flash drive using separate
partitions for efi, /boot/ and / (in a subvolume).  The default
subvolume was set to 5 for both the flash / partition and also the
RAID6 array.  I used a separate /boot partition to reduce complexity.
Both the kernel and btrfs tools were upgraded to 4.4.  I soon
thereafter got lost.

The steps "snapshot it, send the snapshot to your array, set property of that
subvolume to RW, chroot, update fstab to mount / with the appropriate
subvol=option," posed no problem, then I got confused.  Should the
next step "update /etc/default/grub" be done within the chroot
environment and  what does "update /etc/default/grub" mean?  I assume
update=grub is done outside the grub environment because it fails to
execute within the chroot/.  What am I missing?  Outside the chroot
environment I did both an update-grub and a grub-install.  My nas
still boots using root on the flash drive. How do I fix this?  I guess
I am requesting that you "put a rush" on you documentation effort.

I observe that "/boot/grub/grub.cfg" still references the flash drive
root rather than the array root.  The nas functions flawlessly but I
would sleep better with root on the RAID6 array rather than on a flash
drive.  Both the efi and /boot partitions are almost read only so
using cheap flash here is not a substantive risk.

Thanks in advance:
David Alcorn

Re: Install to or Recover RAID Array Subvolume Root?

[Nicholas D Steeves]

On 22 April 2016 at 06:44, David Alcorn <nroclaed@gmail.com> wrote:
>
> First, I verified that while the Debian Installer will install to a
> pre set default BTRFS RAID6 subvolume, the Grub install step fails.
> The alternative to restore installation to a RAID6 subvolume requires
> installation to a non RAID6 subvolume and then send|receive the
> snapshotted installation to the array.  To prepare for this attempt, I
> reinstalled BTRFS (Debian stable) to a flash drive using separate
> partitions for efi, /boot/ and / (in a subvolume).  The default
> subvolume was set to 5 for both the flash / partition and also the
> RAID6 array.  I used a separate /boot partition to reduce complexity.
> Both the kernel and btrfs tools were upgraded to 4.4.  I soon
> thereafter got lost.

1. Have you partially filled your RAID6 array?  If so, do you have
current backups for everything you care about?
2. Please indicate whether you prefer to mount by LABEL, UUID, or /dev
3. If it's by /dev, please send the output of: parted -l
4. If it's by LABEL or UUID, please also send the output of: blkid

Sincerely,
Nicholas

Re: Install to or Recover RAID Array Subvolume Root?

[David Alcorn]

On 4/26/16, Nicholas D Steeves <nsteeves@gmail.com> wrote:
> On 22 April 2016 at 06:44, David Alcorn <nroclaed@gmail.com> wrote:
>>
>> First, I verified that while the Debian Installer will install to a
>> pre set default BTRFS RAID6 subvolume, the Grub install step fails.
>> The alternative to restore installation to a RAID6 subvolume requires
>> installation to a non RAID6 subvolume and then send|receive the
>> snapshotted installation to the array.  To prepare for this attempt, I
>> reinstalled BTRFS (Debian stable) to a flash drive using separate
>> partitions for efi, /boot/ and / (in a subvolume).  The default
>> subvolume was set to 5 for both the flash / partition and also the
>> RAID6 array.  I used a separate /boot partition to reduce complexity.
>> Both the kernel and btrfs tools were upgraded to 4.4.  I soon
>> thereafter got lost.
>
> 1. Have you partially filled your RAID6 array?  If so, do you have
> current backups for everything you care about?
> 2. Please indicate whether you prefer to mount by LABEL, UUID, or /dev
> 3. If it's by /dev, please send the output of: parted -l
> 4. If it's by LABEL or UUID, please also send the output of: blkid
>
> Sincerely,
> Nicholas
>


Nicholas:

1. My RAID array is partially filled and backed up.
2. I prefer to mount by UUID.
3. n/a
4. the relevant content of my blkid is:
"/dev/sda1: UUID="AEF6-E013" TYPE="vfat"
PARTUUID="b4b9c894-aa4f-4c83-ba27-8919eaaeac49"
/dev/sda2: UUID="a428a1ea-5174-47b6-a894-521166a7a354" TYPE="ext2"
PARTUUID="e25bcabb-2fd3-4515-bf74-3a2f0c548fec"
/dev/sda3: UUID="1b413c7c-d39d-4c10-9f1d-4f3a21791c50"
UUID_SUB="e2d12168-c216-425f-9a82-6d46dad8ccc8" TYPE="btrfs"
PARTLABEL="primary" PARTUUID="1651d9f1-35ff-458b-ae64-bb6003c72159"
/dev/sdb: UUID="c2cf44d3-28e0-492a-9d51-00a41b71428d"
UUID_SUB="b4c6bbc5-ea80-4a79-89ca-333d351b09e5" TYPE="btrfs"
/dev/sdc: UUID="c2cf44d3-28e0-492a-9d51-00a41b71428d"
UUID_SUB="c4ac65e7-45bb-4450-9aa4-80a7fbebed3a" TYPE="btrfs"
/dev/sdd: UUID="c2cf44d3-28e0-492a-9d51-00a41b71428d"
UUID_SUB="5f55bf3b-8e82-4b50-8b62-5dc45c892281" TYPE="btrfs"
/dev/sde: UUID="c2cf44d3-28e0-492a-9d51-00a41b71428d"
UUID_SUB="5a12cbd5-bdc2-4258-9bc8-23d95cbb1db6" TYPE="btrfs"


I omitted /dev/sdf from my blkid content as it is an unused,
transitory drive that will be redeployed after "/" is moved to the
array.  /dev/sda is my flash drive with separate partitions for efi
(vfat), /boot/ (ext2) and "/" (btrfs subvolume).

David Alcorn

Re: Install to or Recover RAID Array Subvolume Root?

[Nicholas D Steeves]

Hi David,

Thanks for the update, sorry for the delay, this email was sitting in
my drafts folder.  Instructions follow.

On 26 April 2016 at 05:23, David Alcorn <nroclaed@gmail.com> wrote:
>
> Nicholas:
>
> 1. My RAID array is partially filled and backed up.
> 2. I prefer to mount by UUID.
> 3. n/a
> 4. the relevant content of my blkid is:
> "/dev/sda1: UUID="AEF6-E013" TYPE="vfat"
> PARTUUID="b4b9c894-aa4f-4c83-ba27-8919eaaeac49"
> /dev/sda2: UUID="a428a1ea-5174-47b6-a894-521166a7a354" TYPE="ext2"
> PARTUUID="e25bcabb-2fd3-4515-bf74-3a2f0c548fec"
> /dev/sda3: UUID="1b413c7c-d39d-4c10-9f1d-4f3a21791c50"
> UUID_SUB="e2d12168-c216-425f-9a82-6d46dad8ccc8" TYPE="btrfs"
> PARTLABEL="primary" PARTUUID="1651d9f1-35ff-458b-ae64-bb6003c72159"
> /dev/sdb: UUID="c2cf44d3-28e0-492a-9d51-00a41b71428d"
> UUID_SUB="b4c6bbc5-ea80-4a79-89ca-333d351b09e5" TYPE="btrfs"
> /dev/sdc: UUID="c2cf44d3-28e0-492a-9d51-00a41b71428d"
> UUID_SUB="c4ac65e7-45bb-4450-9aa4-80a7fbebed3a" TYPE="btrfs"
> /dev/sdd: UUID="c2cf44d3-28e0-492a-9d51-00a41b71428d"
> UUID_SUB="5f55bf3b-8e82-4b50-8b62-5dc45c892281" TYPE="btrfs"
> /dev/sde: UUID="c2cf44d3-28e0-492a-9d51-00a41b71428d"
> UUID_SUB="5a12cbd5-bdc2-4258-9bc8-23d95cbb1db6" TYPE="btrfs"
>
>
> I omitted /dev/sdf from my blkid content as it is an unused,
> transitory drive that will be redeployed after "/" is moved to the
> array.  /dev/sda is my flash drive with separate partitions for efi
> (vfat), /boot/ (ext2) and "/" (btrfs subvolume).

Note: I'm guessing you read the Debian wiki entry on btrfs before
2016-03-14?  If you read it after this date, I might need to make the
fact that UEFI systems cannot boot from raw drives more explicit.

Ok, here is how you move everything on your raid6 array to a
subvolume--the goal is to have two subvolumes on your RAID6, one for
rootfs, and one for whatever you're using the bulk storage for.  I'm
assuming you haven't yet created one, but I think these commands will
still work, even if you have; you'll just have nested subvolumes in
that case.  Personally, I prefer alphanumeric subvolume naming, but
I'll use the @ convention for clarity.

These instructions assume that you've booted from you flash disk.
Please note that I'm not sure if this will work if your flash rootfs
is installed on subvolid=5...

btrfs sub list /
# If there is no output, make a note of it.

mkdir /tmp/tank
mount -o noatime UUID="c2cf44d3-28e0-492a-9d51-00a41b71428d" /tmp/tank
btrfs sub list /tmp/tank

cd /tmp/tank
btrfs sub create @tank
ls -1 | egrep -v '@tank' while read d; do mv "$d" @tank/; done
# There!  Now all your data is in a subvolume

btrfs sub snap -r / /rootfs-snapshot
btrfs send /rootfs-snapshot | btrfs receive /tmp/tank
# if this fails for any reason, send the output to this mailing list,
and mention if "btrfs sub list /"
# had any output.
# then umount /tmp/tank && sync

btrfs property set /tmp/tank/rootfs-snapshot ro false && mv rootfs-snapshot @
# There! now everything except for /boot and /boot/efi is on your raid6

# You have a choice between following the following instructions or
editing your grub command line.  Personally I would choose to edit the
grub command line.
# if you choose do it this way, run these commands:
#
# sync
# umount /tmp/tank
# sync
# reboot
#
# edit your grub command line by hitting the "e" key when the menu
comes up.  Find the bit that
# says root= and make sure it looks something like
# root=/dev/sdb ro rootflags=subvol=@

Boot it, and scroll down to the "IF YOU FOLLOWED" section.

##### alternatively #####

cd /tmp/tank/@
mount none -t proc proc
mount -o bind /dev dev
mount none -t sysfs sysfs
mount -o bind /boot boot
mount -o bind /boot/efi
chroot ./


#### IF YOU FOLLOWED the "edit grub command line" method, start here ####

editor /etc/fstab
# change the first line, for /

UUID=c2cf44d3-28e0-492a-9d51-00a41b71428d    /    btrfs
subvol=/@,noatime    0    0

# if you have a line for your raid6 subvolume, find it, and change it to

UUID=c2cf44d3-28e0-492a-9d51-00a41b71428d    /some/location    btrfs
 subvol=/@tank,noatime    0    0

# You just need to run the following command to get grub to use the
rootfs on your raid6

cp -arx /boot /boot.bak
update-grub
grep '/@/' /boot/grub/grub.cfg
# this should output something!  If it doesn't, seek help on IRC.

# What I would personally do is manually edit /boot/grub/grub.cfg...
# At any rate if that grep command outputted nothing, your system won't boot.
# restore booting from you /dev/sda with the following command
# cp -arx /boot.bak/* /boot
# /\ this /\ should allow your usb stick to continue booting.
# Continuing could trash your grub installation on /dev/sda if grep
'/@/' /boot/grub/grub.cfg
# didn't provide any output.

# If that grep verified that you'll be booting to @ as your rootfs,
then it's safe to do the following:
grub-install /dev/sda
sync

#### IF YOU FOLLOWED the "edit grub command line" method, you're done! ###

### alternatively (continued) ####
exit
umount proc dev sys boot/efi
umount boot
cd ../ && umount tank
sync
reboot

And it should work ;-)
Nicholas