Conversion to btrfs raid1 profile on added ext device renders some systems unable to boot into converted rootfs

Conversion to btrfs raid1 profile on added ext device renders some systems unable to boot into converted rootfs

[Tony Prokott]

Good day. My technical trouble seems to be beyond the scope of active helpers on debian's irc support channel. Reasonable supposition that it's quite particular to the development stage of btrfs infrastructure on 4.17.xxx backport kernels and userland tools available on debian 9.5 stretch as well as buster, the testing suite to be released in the next several months as 10.0 stable. 

 > / # uname -a; lsb_release -a
 > Linux localhost 4.17.0-0.bpo.3-amd64 #1 SMP Debian 4.17.17-1~bpo9+1 (2018-08-27) x86_64 GNU/Linux
 > Distributor ID: LinuxMint
 > Description: LMDE 3 Cindy
 > Release: 3
 > Codename: cindy
 > 
 > / # btrfs --version
 > btrfs-progs v4.7.3
 > 
 > / # btrfs fi sh
 > Label: 'sys'  uuid: [snip]
 > Total devices 2 FS bytes used 24.07GiB
 > devid    1 size 401.59GiB used 26.03GiB path /dev/sda2
 > devid    2 size 401.76GiB used 26.03GiB path /dev/sdc1
 > 
 > / # btrfs fi df /
 > Data, RAID1: total=24.00GiB, used=23.27GiB
 > System, RAID1: total=32.00MiB, used=16.00KiB
 > Metadata, RAID1: total=2.00GiB, used=820.00MiB
 > GlobalReserve, single: total=69.17MiB, used=0.00B
 > 
 > / # btrfs su li -ta /
 > ID	gen	top level	path
 > --	---	---------	----
 > 260	115103	5		<FS_TREE>/d9
 > 261	115103	5		<FS_TREE>/d10
 > 262	123876	5		<FS_TREE>/home
 > 263	115148	261		<FS_TREE>/d10/@
 > 264	115136	261		<FS_TREE>/d10/@home
 > 443	123874	447		<FS_TREE>/md3/@
 > 444	123876	447		<FS_TREE>/md3/@home
 > 447	115103	5		<FS_TREE>/md3
 > 451	115144	260		<FS_TREE>/d9/@
 > 452	115136	260		<FS_TREE>/d9/@home

Providing no dmesg content so far, as it doesn't bear on the kind of difficulty in question. My system requires expert help now to restore bootability to 2 of its OS installations; it has a btrfs root file system in subvolumes for stretch, buster, and LMDE3(cindy) which derives directly from stretch and so has most core elements if not cfg defaults in common; even kernel versions are alike, besides buster. subvolid=262 is a  /home fs shared among  linux distros; 451, 263, and 443 are rootfs for stretch, buster and cindy respectively.

All 3 installations had been booting and running fine when data block group profile was "single" on an internal sata HDD /dev/sda2; then an external usb3 drive enclosure's sata HDD partition /dev/sdc1, also of size ~0.4TiB, was added and balanced as btrfs "raid1"; raid conversion did not damage subvolume content or filesystem integrity afaict, but rather rendered stretch and buster unbootable (more to follow), whereas cindy carried on without hiccup.

At first it seemed as though the initrd's might be missing a module or so, to allow access to external drives -- i.e. grub starts the unbootable kernel/initrd but drops to busybox prompt right away without starting external drives, referring to allegedly "missing" btrfs device's UUID_SUB.

But after chrooting to update-initramfs and cataloging resulting image content, usb_storage and uas were present under /lib/modules/xxx already, and failing systems still just busybox without a real rootfs rather than launch systemd; even tried kernel option "rootwait" which had no effect on access to ext storage; udev still seems not to have noticed the ext drives once busybox had control.

I could list all initrd modules present in cindy & absent for others, but need better knowledge than my reasonable guesses of what's required to make btrfs volume companion devices cooperate at boot time, as initrd transitions to steady state rootfs.

What would be a more practical diagnostic? Could stretch & buster initrd's somehow be failing to do a btrfs device scan at the proper moment? Not so interested in giving up on btrfs software raid so early in the game.

thanks in advance-
TP [not a list subscriber]

Re: Conversion to btrfs raid1 profile on added ext device renders some systems unable to boot into converted rootfs

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 4874 bytes --]



On 2018/10/18 上午12:38, Tony Prokott wrote:
> Good day. My technical trouble seems to be beyond the scope of active helpers on debian's irc support channel. Reasonable supposition that it's quite particular to the development stage of btrfs infrastructure on 4.17.xxx backport kernels and userland tools available on debian 9.5 stretch as well as buster, the testing suite to be released in the next several months as 10.0 stable. 
> 
>  > / # uname -a; lsb_release -a
>  > Linux localhost 4.17.0-0.bpo.3-amd64 #1 SMP Debian 4.17.17-1~bpo9+1 (2018-08-27) x86_64 GNU/Linux
>  > Distributor ID: LinuxMint
>  > Description: LMDE 3 Cindy
>  > Release: 3
>  > Codename: cindy
>  > 
>  > / # btrfs --version
>  > btrfs-progs v4.7.3
>  > 
>  > / # btrfs fi sh
>  > Label: 'sys'  uuid: [snip]
>  > Total devices 2 FS bytes used 24.07GiB
>  > devid    1 size 401.59GiB used 26.03GiB path /dev/sda2
>  > devid    2 size 401.76GiB used 26.03GiB path /dev/sdc1
>  > 
>  > / # btrfs fi df /
>  > Data, RAID1: total=24.00GiB, used=23.27GiB
>  > System, RAID1: total=32.00MiB, used=16.00KiB
>  > Metadata, RAID1: total=2.00GiB, used=820.00MiB
>  > GlobalReserve, single: total=69.17MiB, used=0.00B
>  > 
>  > / # btrfs su li -ta /
>  > ID	gen	top level	path
>  > --	---	---------	----
>  > 260	115103	5		<FS_TREE>/d9
>  > 261	115103	5		<FS_TREE>/d10
>  > 262	123876	5		<FS_TREE>/home
>  > 263	115148	261		<FS_TREE>/d10/@
>  > 264	115136	261		<FS_TREE>/d10/@home
>  > 443	123874	447		<FS_TREE>/md3/@
>  > 444	123876	447		<FS_TREE>/md3/@home
>  > 447	115103	5		<FS_TREE>/md3
>  > 451	115144	260		<FS_TREE>/d9/@
>  > 452	115136	260		<FS_TREE>/d9/@home
> 
> Providing no dmesg content so far, as it doesn't bear on the kind of difficulty in question. My system requires expert help now to restore bootability to 2 of its OS installations; it has a btrfs root file system in subvolumes for stretch, buster, and LMDE3(cindy) which derives directly from stretch and so has most core elements if not cfg defaults in common; even kernel versions are alike, besides buster. subvolid=262 is a  /home fs shared among  linux distros; 451, 263, and 443 are rootfs for stretch, buster and cindy respectively.
> 
> All 3 installations had been booting and running fine when data block group profile was "single" on an internal sata HDD /dev/sda2; then an external usb3 drive enclosure's sata HDD partition /dev/sdc1, also of size ~0.4TiB, was added and balanced as btrfs "raid1"; raid conversion did not damage subvolume content or filesystem integrity afaict, but rather rendered stretch and buster unbootable (more to follow), whereas cindy carried on without hiccup.
> 
> At first it seemed as though the initrd's might be missing a module or so, to allow access to external drives -- i.e. grub starts the unbootable kernel/initrd but drops to busybox prompt right away without starting external drives, referring to allegedly "missing" btrfs device's UUID_SUB.
> 
> But after chrooting to update-initramfs and cataloging resulting image content, usb_storage and uas were present under /lib/modules/xxx already, and failing systems still just busybox without a real rootfs rather than launch systemd; even tried kernel option "rootwait" which had no effect on access to ext storage; udev still seems not to have noticed the ext drives once busybox had control.

Still looks like a initramfs problem other than btrfs problem.

In the busybox environment, have you tried listing /dev to see if that
external device is found?

> 
> I could list all initrd modules present in cindy & absent for others, but need better knowledge than my reasonable guesses of what's required to make btrfs volume companion devices cooperate at boot time, as initrd transitions to steady state rootfs.

Since you have a busybox environment, have you checked if "btrfs"
command lives in the initramfs?

IIRC at least you need the following things/abilities to boot:

1) usb and sata drivers
   Means you could see both devices in the busybox environment under
   /dev

2) "Btrfs" command
   Mostly for scan

Then you could try the following commands under busybox environment:

# btrfs device scan
# mount <device> <some temporary mount point>

If it works, it may mean you're missing "btrfs device scan" during boot
so kernel can't see all RAID1 disks for btrfs and failed to boot.

Please refer to your distribution initramfs creation tool to see how to
add that scan. (Some distro has special hook for btrfs to handle such case).

Thanks,
Qu

> 
> What would be a more practical diagnostic? Could stretch & buster initrd's somehow be failing to do a btrfs device scan at the proper moment? Not so interested in giving up on btrfs software raid so early in the game.
> 
> thanks in advance-
> TP [not a list subscriber]
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Conversion to btrfs raid1 profile on added ext device renders some systems unable to boot into converted rootfs

[Tony Prokott]

 ---- On Wed, 17 Oct 2018 17:57:25 -0700 Qu Wenruo <quwenruo.btrfs@gmx.com> wrote ---- 
...
 > > But after chrooting to update-initramfs and cataloging resulting image content, usb_storage and uas were present under /lib/modules/xxx already, and failing systems still just busybox without a real rootfs rather than launch systemd; even tried kernel option "rootwait" which had no effect on access to ext storage; udev still seems not to have noticed the ext drives once busybox had control. 
 >  
 > Still looks like a initramfs problem other than btrfs problem. 
 >  
 > In the busybox environment, have you tried listing /dev to see if that 
 > external device is found? 

agreed that initramfs smells bad, but it hadn't been a problem until btrfs mounts (external-raid) had to rely on the usb channel;
in busybox, ext drives/partitions are all missing from /dev; can't tell why so, ahci and usb modules are loaded afaict

 > Since you have a busybox environment, have you checked if "btrfs" command lives in the initramfs? 

yes btrfs command works from busybox

 > IIRC at least you need the following things/abilities to boot: 
 >  
 > 1) usb and sata drivers 
 >    Means you could see both devices in the busybox environment under /dev 
 >  
 > 2) "Btrfs" command 
 >    Mostly for scan 
 > Then you could try the following commands under busybox environment: 
 > # btrfs device scan 
 > # mount <device> <some temporary mount point> 

"btrfs dev scan" runs but doesn't indicate recognizing any; since raid1 conversion, ext drives are required for any btrfs mounts to be seen whole.
When manually trying to mount in busybox, it gives a similar error about missing external device by UUID_SUB

 > If it works, it may mean you're missing "btrfs device scan" during boot 
 > so kernel can't see all RAID1 disks for btrfs and failed to boot. 
 >  
 > Please refer to your distribution initramfs creation tool to see how to 
 > add that scan. (Some distro has special hook for btrfs to handle such case). 

may have to tweak the /etc/initramfs-tools/initramfs.conf or modules list; MODULES=dep setting might act better than MODULES=most
will look into this further to see about contrasting block device modules between cindy and the others

appreciate the timely response-
TP

Re: Conversion to btrfs raid1 profile on added ext device renders some systems unable to boot into converted rootfs

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 3338 bytes --]



On 2018/10/18 下午2:16, Tony Prokott wrote:
>  ---- On Wed, 17 Oct 2018 17:57:25 -0700 Qu Wenruo <quwenruo.btrfs@gmx.com> wrote ---- 
> ...
>  > > But after chrooting to update-initramfs and cataloging resulting image content, usb_storage and uas were present under /lib/modules/xxx already, and failing systems still just busybox without a real rootfs rather than launch systemd; even tried kernel option "rootwait" which had no effect on access to ext storage; udev still seems not to have noticed the ext drives once busybox had control. 
>  >  
>  > Still looks like a initramfs problem other than btrfs problem. 
>  >  
>  > In the busybox environment, have you tried listing /dev to see if that 
>  > external device is found? 
> 
> agreed that initramfs smells bad, but it hadn't been a problem until btrfs mounts (external-raid) had to rely on the usb channel;

By default, btrfs must see *all* devices to mount RAID1/10/5/6/0.
Unless you're using "degraded" mount option.

You could argue it's a bad decision, but still you have the choice.

> in busybox, ext drives/partitions are all missing from /dev; can't tell why so, ahci and usb modules are loaded afaict
> 
>  > Since you have a busybox environment, have you checked if "btrfs" command lives in the initramfs? 
> 
> yes btrfs command works from busybox
> 
>  > IIRC at least you need the following things/abilities to boot: 
>  >  
>  > 1) usb and sata drivers 
>  >    Means you could see both devices in the busybox environment under /dev 
>  >  
>  > 2) "Btrfs" command 
>  >    Mostly for scan 
>  > Then you could try the following commands under busybox environment: 
>  > # btrfs device scan 
>  > # mount <device> <some temporary mount point> 
> 
> "btrfs dev scan" runs but doesn't indicate recognizing any; since raid1 conversion, ext drives are required for any btrfs mounts to be seen whole.
> When manually trying to mount in busybox, it gives a similar error about missing external device by UUID_SUB

Then it's still something wrong about the initramfs.

From your description, it looks pretty like the lack of external disk
driver is the root cause.

Could you try "btrfs fi show" to see which devices is missing?
I strongly suspect it's the external device, if so, then it is the
mkinitramfs driver causing the problem.

> 
>  > If it works, it may mean you're missing "btrfs device scan" during boot 
>  > so kernel can't see all RAID1 disks for btrfs and failed to boot. 
>  >  
>  > Please refer to your distribution initramfs creation tool to see how to 
>  > add that scan. (Some distro has special hook for btrfs to handle such case). 
> 
> may have to tweak the /etc/initramfs-tools/initramfs.conf or modules list; MODULES=dep setting might act better than MODULES=most
> will look into this further to see about contrasting block device modules between cindy and the others

Sorry I can't help much in this case, as I'm not debian user.

But in Archlinux, it's pretty easy by just adding a 'block' hook.
And in that case, Archlinux will install all kernel modules under
'drivers/usb/storage' directory.

For details, you could refer to this file:
https://git.archlinux.org/mkinitcpio.git/tree/install/block

Thanks,
Qu

> 
> appreciate the timely response-
> TP
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Conversion to btrfs raid1 profile on added ext device renders some systems unable to boot into converted rootfs

[Tony Prokott]

The trouble is yet unresolved, symptoms are as they were, but I've diagnosed a step further. Maybe you can help me advance the diagnosis or better pose my question among debian experts, related to adjusting the building of initrd.

 ---- On Thu, 18 Oct 2018 00:08:08 -0700 Qu Wenruo <quwenruo.btrfs@gmx.com> wrote ---- 
 > >  > Still looks like a initramfs problem [rather] than btrfs problem.  
Yes, but linux-btrfs list still knows better than I how best to proceed, mainly how to distill the trouble description using proper terms, also lending broader understanding of what named modules serve what device activate&cfg or storage-access purpose.

 > >  > In the busybox environment, have you tried listing /dev to see if that external device is found?  
External usb attached drives are definitely not found by a newly launched kernel, and particulars of why are still not self evident. Boot loader grub2 all along still has no trouble accessing -- presumably it's not able to leverage raid1 redundancy in btrfs but does have access to the ext mirror device and takes notice in passing of matching UUID's.

 > By default, btrfs must see *all* devices to mount RAID1/10/5/6/0. Unless you're using "degraded" mount option. 
 > You could argue it's a bad decision, but still you have the choice. 
Yes did manage to mount it degraded, just to see that content demirrored is also unclobbered, however can't finish the job by such means; hopefully in the process no metadata or other junk was written unintentionally (forgot to mount readonly - degraded)

The task now seems to be finishing resolving which modules can bring in the rest of the critical infrastructure to allow access to the drives that had been no customized bother to bring online, prior to rootfs raid1 conversion. A recently found item of great interest is module "autofs4" which has userland friends such as systemd; it's present in cindy(LMDE3) which boots fine in spite of deriving from stretch, and was absent in stretch & buster which no longer boot.

 > > When manually trying to mount in busybox, it gives a similar error about missing the same external device, by its UUID_SUB 
 > Then it's still something wrong about the initramfs. From your description, it looks pretty like the lack of external disk driver is the root cause. 
Agreed *something* missing in initrd and-or module deployment is the cause of failing access to ext usb3 drive enclosure devices, but am still tracking down which other missing blobs may be of concern; since busybox won't allow "lsblk" "lshw" or "lsusb" -- only "blkid" or "ls /dev" work to detect devices -- my confidence and expediency in tracking is reduced; haven't yet happened upon the debian feature that lets more nonkernel programs and libraries--not just modules-- be built into initrd.

As an aside probably not germane, the grub.cfg "linux" line to load the kernel in some cases has "ro" readonly option and others not; what's the difference signify? How to make informed choice whether to use ro? Why mount a real disk rootfs without write access, before corruption's detected? Would that not potentially cripple some vital features such as logging?

So far it's clear that uas, usb_storage, & autofs4 may be built into initrd and then load ok, but they're not enough to restore normal systemd launch. Setting "MODULES=dep" in /etc/initramfs-tools/conf.d/driver-policy seems not smart enough for building in all necessary objects. Ideas welcome.

regards-
TP

Re: Conversion to btrfs raid1 profile on added ext device renders some systems unable to boot into converted rootfs

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 4577 bytes --]

Add Cc to debian user list.

On 2018/10/24 上午3:21, Tony Prokott wrote:
> The trouble is yet unresolved, symptoms are as they were, but I've diagnosed a step further. Maybe you can help me advance the diagnosis or better pose my question among debian experts, related to adjusting the building of initrd.
> 
>  ---- On Thu, 18 Oct 2018 00:08:08 -0700 Qu Wenruo <quwenruo.btrfs@gmx.com> wrote ---- 
>  > >  > Still looks like a initramfs problem [rather] than btrfs problem.  
> Yes, but linux-btrfs list still knows better than I how best to proceed, mainly how to distill the trouble description using proper terms, also lending broader understanding of what named modules serve what device activate&cfg or storage-access purpose.

Personally speaking, distro guys are much better at such problem.

In fact I'm not even a debian guy, and for my distro, it's super easy to
setup initramfs to boot from USB device.

> 
>  > >  > In the busybox environment, have you tried listing /dev to see if that external device is found?  
> External usb attached drives are definitely not found by a newly launched kernel, and particulars of why are still not self evident. Boot loader grub2 all along still has no trouble accessing -- presumably it's not able to leverage raid1 redundancy in btrfs but does have access to the ext mirror device and takes notice in passing of matching UUID's.

So missing drivers.
Without the driver, btrfs scan doesn't make sense at all.

It needs not only usb drivers but also some other drivers.
Normally distro should provide such tools to do runtime probe and
generate all needed drivers and their dependency, or manually setup
needed drivers.

BTW, have you tried something like fallback initramfs?
Generally speaking such fallback initramfs should have more modules thus
higher chance to detect external usb drivers.

> 
>  > By default, btrfs must see *all* devices to mount RAID1/10/5/6/0. Unless you're using "degraded" mount option. 
>  > You could argue it's a bad decision, but still you have the choice. 
> Yes did manage to mount it degraded, just to see that content demirrored is also unclobbered, however can't finish the job by such means; hopefully in the process no metadata or other junk was written unintentionally (forgot to mount readonly - degraded)
> 
> The task now seems to be finishing resolving which modules can bring in the rest of the critical infrastructure to allow access to the drives that had been no customized bother to bring online, prior to rootfs raid1 conversion. A recently found item of great interest is module "autofs4" which has userland friends such as systemd; it's present in cindy(LMDE3) which boots fine in spite of deriving from stretch, and was absent in stretch & buster which no longer boot.
> 
>  > > When manually trying to mount in busybox, it gives a similar error about missing the same external device, by its UUID_SUB 
>  > Then it's still something wrong about the initramfs. From your description, it looks pretty like the lack of external disk driver is the root cause. 
> Agreed *something* missing in initrd and-or module deployment is the cause of failing access to ext usb3 drive enclosure devices, but am still tracking down which other missing blobs may be of concern; since busybox won't allow "lsblk" "lshw" or "lsusb" -- only "blkid" or "ls /dev" work to detect devices -- my confidence and expediency in tracking is reduced; haven't yet happened upon the debian feature that lets more nonkernel programs and libraries--not just modules-- be built into initrd.

At least for my distro, it's pretty easy to inject external commands
into initramfs.
(Archlinux provides "add_binary" function in mkinitcpio, which will not
only add the binary but also its dependency)

So it's really dependent on the distro.

Thanks,
Qu

> 
> As an aside probably not germane, the grub.cfg "linux" line to load the kernel in some cases has "ro" readonly option and others not; what's the difference signify? How to make informed choice whether to use ro? Why mount a real disk rootfs without write access, before corruption's detected? Would that not potentially cripple some vital features such as logging?
> 
> So far it's clear that uas, usb_storage, & autofs4 may be built into initrd and then load ok, but they're not enough to restore normal systemd launch. Setting "MODULES=dep" in /etc/initramfs-tools/conf.d/driver-policy seems not smart enough for building in all necessary objects. Ideas welcome.
> 
> regards-
> TP
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]