Re: btrfs as / filesystem in RAID1

From: "Austin S. Hemmelgarn" <ahferroin7@gmail.com>
To: Andrei Borzenkov <arvidjaar@gmail.com>, linux-btrfs@vger.kernel.org
Cc: waxhead@dirtcellar.net, Stefan K <shadow_7@gmx.net>
Subject: Re: btrfs as / filesystem in RAID1
Date: Fri, 8 Feb 2019 07:54:49 -0500	[thread overview]
Message-ID: <1569cf0a-05d3-a45e-ea60-274c7e64df7a@gmail.com> (raw)
In-Reply-To: <df1bde98-8849-581a-8f96-3a2045fe6274@gmail.com>

On 2019-02-07 23:51, Andrei Borzenkov wrote:
> 07.02.2019 22:39, Austin S. Hemmelgarn пишет:
>> The issue with systemd is that if you pass 'degraded' on most systemd
>> systems,  and devices are missing when the system tries to mount the
>> volume, systemd won't mount it because it doesn't see all the devices.
>> It doesn't even _try_ to mount it because it doesn't see all the
>> devices.  Changing to degraded by default won't fix this, because it's a
>> systemd problem.
>>
> 
> Oh no, not again. It was discussed millions of times already - systemd
> is using information that btrfs provides.
And we've already told the systemd developers to quit using the ioctl 
they're using because it causes this issue and also introduces a TOCTOU 
race condition that can be avoided by just trying to mount the volume 
with the provided options.
> 
>> The same issue also makes it a serious pain in the arse to recover
>> degraded BTRFS volumes on systemd systems, because if the volume is
>> supposed to mount normally on that system, systemd will unmount it if it
>> doesn't see all the devices, regardless of how it got mounted in the
>> first place.
>>
> 
> *That* would be systemd issue indeed. If someone can reliably reproduce
> it, systemd bug report would certainly be in order.
> 
It's been a few months since I dealt with it last (I don't use systemd 
on my everyday systems, because of this and a bunch of other issues I 
have with it (mostly design complaints, not bugs FWIW)), but the general 
process is as follows:

1. Configure a multi-device BTRFS volume such that removal of one device 
will cause the DEVICE_READY ioctl to return false.
2. Set it up in `/etc/fstab` or as a mount unit such that it will 
normally get mounted at boot, but won't prevent the system from booting 
if it fails.
3. Reboot with one of the devices missing.
4. Attempt to manually mount the volume using the regular `mount` 
command with the `degraded` option.
5. Check the mount table, there should be no entry for the volume you 
just mounted in it.

After dealing with this the first time (multiple years ago now), I took 
the time to trace system calls, and found that systemd was unmounting 
the volume immediately after I mounted it.