Seed device is broken, again.

Seed device is broken, again.

[Qu Wenruo]

Hi,

The very basic seed device usage is broken again:

	mkfs.btrfs -f $dev1 > /dev/null
	btrfstune -S 1 $dev1
	mount $dev1 $mnt
	btrfs dev add $dev2 $mnt
	umount $mnt


I'm not sure how many guys are really using seed device.

But I see a lot of weird operations, like calling a definite write 
operation (device add) on a RO mounted fs.

Can we make at least the seed sprouting part into btrfs-progs instead?

And can seed device even support the upcoming extent-tree-v2?

Personally speaking I prefer to mark seed device deprecated completely.

The call trace:

  assertion failed: sb_write_started(fs_info->sb), in 
fs/btrfs/volumes.c:3244
  ------------[ cut here ]------------
  kernel BUG at fs/btrfs/ctree.h:3556!
  invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
  CPU: 11 PID: 626 Comm: btrfs Not tainted 5.17.0-rc5-custom+ #2
  Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
  RIP: 0010:assertfail.constprop.0+0x18/0x1a [btrfs]
  Code: 87 ff ff 4c 89 e1 4c 89 ea 48 c7 c6 68 5f b2 c0 eb e4 89 f1 48 
c7 c2 8f cb b1 c0 48 89 fe 48 c7 c7 90 5f b2 c0 e8 c8 cd e0 c8 <0f> 0b 
49 8b 85 28 11 00 00 48 c7 c6 00 60 b2 c0 4c 89 ef 8b 90 fc
  RSP: 0018:ffffbaa04148bc78 EFLAGS: 00010246
  RAX: 000000000000004b RBX: ffff97cb45671000 RCX: 0000000000000000
  RDX: 0000000000000000 RSI: ffff97cbbd0e1aa0 RDI: ffff97cbbd0e1aa0
  RBP: ffff97cb4478c000 R08: 0000000000000000 R09: ffffbaa04148bab0
  R10: ffffbaa04148baa8 R11: ffffffff8a4e6968 R12: 0000000000000002
  R13: 0000000021d00000 R14: ffff97cb4478dfe0 R15: ffff97cb44dfc770
  FS:  00007fc02f8fb2c0(0000) GS:ffff97cbbd0c0000(0000) 
knlGS:0000000000000000
  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
  CR2: 000056327627ac88 CR3: 0000000008938000 CR4: 0000000000750ee0
  PKRU: 55555554
  Call Trace:
   <TASK>
   btrfs_relocate_chunk.cold+0x42/0x67 [btrfs]
   btrfs_init_new_device+0x11e5/0x1780 [btrfs]
   ? btrfs_ioctl+0x1f20/0x32c0 [btrfs]
   btrfs_ioctl+0x1f20/0x32c0 [btrfs]
   ? find_held_lock+0x2b/0x80
   ? mntput_no_expire+0x7c/0x480
   ? lock_release+0xca/0x2d0
   ? __x64_sys_ioctl+0x82/0xb0
   __x64_sys_ioctl+0x82/0xb0
   do_syscall_64+0x3b/0x90
   entry_SYSCALL_64_after_hwframe+0x44/0xae
  RIP: 0033:0x7fc02f9fc59b
  Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c 
c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 
01 f0 ff ff 73 01 c3 48 8b 0d a5 a8 0c 00 f7 d8 64 89 01 48
  RSP: 002b:00007fffc2620878 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
  RAX: ffffffffffffffda RBX: 0000556ccf6be870 RCX: 00007fc02f9fc59b
  RDX: 00007fffc26208d0 RSI: 000000005000940a RDI: 0000000000000003
  RBP: 00007fffc26208d0 R08: 0000000000000010 R09: 00007fffc261e6c0
  R10: 0000000000000031 R11: 0000000000000202 R12: 00007fffc2621a90
  R13: 00007fffc2621a98 R14: 0000000000000000 R15: 0000000000000000
   </TASK>
  Modules linked in: target_core_user uio target_core_mod btrfs 
blake2b_generic xor intel_rapl_msr iTCO_wdt raid6_pq iTCO_vendor_support 
snd_hda_codec_generic snd_hda_intel intel_rapl_common snd_intel_dspcfg 
crct10dif_pclmul snd_hda_codec crc32_pclmul ghash_clmulni_intel 
snd_hwdep aesni_intel nls_iso8859_1 snd_hda_core crypto_simd cryptd 
joydev vfat snd_pcm fat psmouse mousedev snd_timer pcspkr i2c_i801 snd 
i2c_smbus soundcore lpc_ich intel_agp intel_gtt qemu_fw_cfg agpgart drm 
fuse ip_tables x_tables xfs libcrc32c crc32c_generic dm_mod virtio_rng 
virtio_scsi virtio_blk rng_core virtio_console virtio_balloon virtio_net 
net_failover failover crc32c_intel serio_raw virtio_pci 
virtio_pci_legacy_dev virtio_pci_modern_dev usbhid
  Dumping ftrace buffer:
     (ftrace buffer empty)
  ---[ end trace 0000000000000000 ]---

Thanks,
Qu

Re: Seed device is broken, again.

[Filipe Manana]

On Fri, Feb 25, 2022 at 06:08:20PM +0800, Qu Wenruo wrote:
> Hi,
> 
> The very basic seed device usage is broken again:
> 
> 	mkfs.btrfs -f $dev1 > /dev/null
> 	btrfstune -S 1 $dev1
> 	mount $dev1 $mnt
> 	btrfs dev add $dev2 $mnt
> 	umount $mnt
> 
> 
> I'm not sure how many guys are really using seed device.
> 
> But I see a lot of weird operations, like calling a definite write operation
> (device add) on a RO mounted fs.
> 
> Can we make at least the seed sprouting part into btrfs-progs instead?
> 
> And can seed device even support the upcoming extent-tree-v2?
> 
> Personally speaking I prefer to mark seed device deprecated completely.
> 
> The call trace:
> 
>  assertion failed: sb_write_started(fs_info->sb), in fs/btrfs/volumes.c:3244

I think you are overreacting a bit about it being broken.

This is a new assertion recently added by Aota, and it's also failing when
balance is resumed on mount, see:

https://lore.kernel.org/linux-btrfs/cover.1645157220.git.naohiro.aota@wdc.com/

Adding him to cc, so that he's aware of this other case.
This only affects misc-next and not any release.

>  ------------[ cut here ]------------
>  kernel BUG at fs/btrfs/ctree.h:3556!
>  invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
>  CPU: 11 PID: 626 Comm: btrfs Not tainted 5.17.0-rc5-custom+ #2
>  Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
>  RIP: 0010:assertfail.constprop.0+0x18/0x1a [btrfs]
>  Code: 87 ff ff 4c 89 e1 4c 89 ea 48 c7 c6 68 5f b2 c0 eb e4 89 f1 48 c7 c2
> 8f cb b1 c0 48 89 fe 48 c7 c7 90 5f b2 c0 e8 c8 cd e0 c8 <0f> 0b 49 8b 85 28
> 11 00 00 48 c7 c6 00 60 b2 c0 4c 89 ef 8b 90 fc
>  RSP: 0018:ffffbaa04148bc78 EFLAGS: 00010246
>  RAX: 000000000000004b RBX: ffff97cb45671000 RCX: 0000000000000000
>  RDX: 0000000000000000 RSI: ffff97cbbd0e1aa0 RDI: ffff97cbbd0e1aa0
>  RBP: ffff97cb4478c000 R08: 0000000000000000 R09: ffffbaa04148bab0
>  R10: ffffbaa04148baa8 R11: ffffffff8a4e6968 R12: 0000000000000002
>  R13: 0000000021d00000 R14: ffff97cb4478dfe0 R15: ffff97cb44dfc770
>  FS:  00007fc02f8fb2c0(0000) GS:ffff97cbbd0c0000(0000)
> knlGS:0000000000000000
>  CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>  CR2: 000056327627ac88 CR3: 0000000008938000 CR4: 0000000000750ee0
>  PKRU: 55555554
>  Call Trace:
>   <TASK>
>   btrfs_relocate_chunk.cold+0x42/0x67 [btrfs]
>   btrfs_init_new_device+0x11e5/0x1780 [btrfs]
>   ? btrfs_ioctl+0x1f20/0x32c0 [btrfs]
>   btrfs_ioctl+0x1f20/0x32c0 [btrfs]
>   ? find_held_lock+0x2b/0x80
>   ? mntput_no_expire+0x7c/0x480
>   ? lock_release+0xca/0x2d0
>   ? __x64_sys_ioctl+0x82/0xb0
>   __x64_sys_ioctl+0x82/0xb0
>   do_syscall_64+0x3b/0x90
>   entry_SYSCALL_64_after_hwframe+0x44/0xae
>  RIP: 0033:0x7fc02f9fc59b
>  Code: ff ff ff 85 c0 79 9b 49 c7 c4 ff ff ff ff 5b 5d 4c 89 e0 41 5c c3 66
> 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff
> 73 01 c3 48 8b 0d a5 a8 0c 00 f7 d8 64 89 01 48
>  RSP: 002b:00007fffc2620878 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
>  RAX: ffffffffffffffda RBX: 0000556ccf6be870 RCX: 00007fc02f9fc59b
>  RDX: 00007fffc26208d0 RSI: 000000005000940a RDI: 0000000000000003
>  RBP: 00007fffc26208d0 R08: 0000000000000010 R09: 00007fffc261e6c0
>  R10: 0000000000000031 R11: 0000000000000202 R12: 00007fffc2621a90
>  R13: 00007fffc2621a98 R14: 0000000000000000 R15: 0000000000000000
>   </TASK>
>  Modules linked in: target_core_user uio target_core_mod btrfs
> blake2b_generic xor intel_rapl_msr iTCO_wdt raid6_pq iTCO_vendor_support
> snd_hda_codec_generic snd_hda_intel intel_rapl_common snd_intel_dspcfg
> crct10dif_pclmul snd_hda_codec crc32_pclmul ghash_clmulni_intel snd_hwdep
> aesni_intel nls_iso8859_1 snd_hda_core crypto_simd cryptd joydev vfat
> snd_pcm fat psmouse mousedev snd_timer pcspkr i2c_i801 snd i2c_smbus
> soundcore lpc_ich intel_agp intel_gtt qemu_fw_cfg agpgart drm fuse ip_tables
> x_tables xfs libcrc32c crc32c_generic dm_mod virtio_rng virtio_scsi
> virtio_blk rng_core virtio_console virtio_balloon virtio_net net_failover
> failover crc32c_intel serio_raw virtio_pci virtio_pci_legacy_dev
> virtio_pci_modern_dev usbhid
>  Dumping ftrace buffer:
>     (ftrace buffer empty)
>  ---[ end trace 0000000000000000 ]---
> 
> Thanks,
> Qu
> 

Re: Seed device is broken, again.

[David Sterba]

On Fri, Feb 25, 2022 at 06:08:20PM +0800, Qu Wenruo wrote:
> Hi,
> 
> The very basic seed device usage is broken again:
> 
> 	mkfs.btrfs -f $dev1 > /dev/null
> 	btrfstune -S 1 $dev1
> 	mount $dev1 $mnt
> 	btrfs dev add $dev2 $mnt
> 	umount $mnt
> 
> 
> I'm not sure how many guys are really using seed device.
> 
> But I see a lot of weird operations, like calling a definite write 
> operation (device add) on a RO mounted fs.

That's how the seeding device works, in what way would you want to do
the ro->rw change?

> Can we make at least the seed sprouting part into btrfs-progs instead?

How? What do you mean? This is an in kernel operation that is done on a
mounted filesystem, progs can't do that.

> And can seed device even support the upcoming extent-tree-v2?

I should, it operates on the device level.

> Personally speaking I prefer to mark seed device deprecated completely.

If we did that with every feature some developer does not like we'd have
nothing left you know. Seeding is a documented usecase, as long as it
works it's fine to have it available.

> The call trace:
> 
>   assertion failed: sb_write_started(fs_info->sb), in 
> fs/btrfs/volumes.c:3244

Yeah the asserts now appear and we need to fix the write annotations
first. I've moved the patches out of misc-next again, it's now only in
for-next so it does not block testing.

Re: Seed device is broken, again.

[Nikolay Borisov]

On 25.02.22 г. 12:08 ч., Qu Wenruo wrote:
> Hi,
> 
> The very basic seed device usage is broken again:
> 
>      mkfs.btrfs -f $dev1 > /dev/null
>      btrfstune -S 1 $dev1
>      mount $dev1 $mnt
>      btrfs dev add $dev2 $mnt
>      umount $mnt
> 
> 

This usage is tested by btrfs/161 no, so it should have been caught 
during testing of the feature that implemented the assert.

Re: Seed device is broken, again.

[Qu Wenruo]

On 2022/2/25 19:47, David Sterba wrote:
> On Fri, Feb 25, 2022 at 06:08:20PM +0800, Qu Wenruo wrote:
>> Hi,
>>
>> The very basic seed device usage is broken again:
>>
>> 	mkfs.btrfs -f $dev1 > /dev/null
>> 	btrfstune -S 1 $dev1
>> 	mount $dev1 $mnt
>> 	btrfs dev add $dev2 $mnt
>> 	umount $mnt
>>
>>
>> I'm not sure how many guys are really using seed device.
>>
>> But I see a lot of weird operations, like calling a definite write
>> operation (device add) on a RO mounted fs.
>
> That's how the seeding device works, in what way would you want to do
> the ro->rw change?

In progs-only, so that in kernel we can make dev add ioctl as a real RW
operation, not adding an exception for dev add.

>
>> Can we make at least the seed sprouting part into btrfs-progs instead?
>
> How? What do you mean? This is an in kernel operation that is done on a
> mounted filesystem, progs can't do that.

Why not?

Progs has the same ability to read-write the fs, I see no reason why we
shouldn't do it in user space.

We're just adding a device, update related trees (which will only be
written to the new device). I see no special reason why it can't be done
in user space.

Furthermore, the ability to add device in user space can be a good
safenet for some ENOSPC space.

>
>> And can seed device even support the upcoming extent-tree-v2?
>
> I should, it operates on the device level.
>
>> Personally speaking I prefer to mark seed device deprecated completely.
>
> If we did that with every feature some developer does not like we'd have
> nothing left you know. Seeding is a documented usecase, as long as it
> works it's fine to have it available.

A documented usecase doesn't mean it can't be deprecated.

Furthermore, a documented use case doesn't validate the use case itself.

So, please tell me when did you use seed device last time?
Or anyone in the mail list, please tell me some real world usecase for
seed devices.

I did remember some planned use case like a way to use seed device as a
VM/container template, but no, it doesn't get much attention.


I'm not asking for deprecate the feature just because I don't like it.

This feature is asking for too many exceptions, from the extra
fs_devices hack (we have in fact two fs_devices, one for rw devices, one
for seed only), to the dev add ioctl.

But the little benefit is not really worthy for the cost.

Thanks,
Qu

>
>> The call trace:
>>
>>    assertion failed: sb_write_started(fs_info->sb), in
>> fs/btrfs/volumes.c:3244
>
> Yeah the asserts now appear and we need to fix the write annotations
> first. I've moved the patches out of misc-next again, it's now only in
> for-next so it does not block testing.
>

Re: Seed device is broken, again.

[Omar Sandoval]

On Fri, Feb 25, 2022 at 09:36:32PM +0800, Qu Wenruo wrote:
> 
> 
> On 2022/2/25 19:47, David Sterba wrote:
> > On Fri, Feb 25, 2022 at 06:08:20PM +0800, Qu Wenruo wrote:
> > > Hi,
> > > 
> > > The very basic seed device usage is broken again:
> > > 
> > > 	mkfs.btrfs -f $dev1 > /dev/null
> > > 	btrfstune -S 1 $dev1
> > > 	mount $dev1 $mnt
> > > 	btrfs dev add $dev2 $mnt
> > > 	umount $mnt
> > > 
> > > 
> > > I'm not sure how many guys are really using seed device.
> > > 
> > > But I see a lot of weird operations, like calling a definite write
> > > operation (device add) on a RO mounted fs.
> > 
> > That's how the seeding device works, in what way would you want to do
> > the ro->rw change?
> 
> In progs-only, so that in kernel we can make dev add ioctl as a real RW
> operation, not adding an exception for dev add.
> 
> > 
> > > Can we make at least the seed sprouting part into btrfs-progs instead?
> > 
> > How? What do you mean? This is an in kernel operation that is done on a
> > mounted filesystem, progs can't do that.
> 
> Why not?
> 
> Progs has the same ability to read-write the fs, I see no reason why we
> shouldn't do it in user space.
> 
> We're just adding a device, update related trees (which will only be
> written to the new device). I see no special reason why it can't be done
> in user space.
> 
> Furthermore, the ability to add device in user space can be a good
> safenet for some ENOSPC space.
> 
> > 
> > > And can seed device even support the upcoming extent-tree-v2?
> > 
> > I should, it operates on the device level.
> > 
> > > Personally speaking I prefer to mark seed device deprecated completely.
> > 
> > If we did that with every feature some developer does not like we'd have
> > nothing left you know. Seeding is a documented usecase, as long as it
> > works it's fine to have it available.
> 
> A documented usecase doesn't mean it can't be deprecated.
> 
> Furthermore, a documented use case doesn't validate the use case itself.
> 
> So, please tell me when did you use seed device last time?
> Or anyone in the mail list, please tell me some real world usecase for
> seed devices.
> 
> I did remember some planned use case like a way to use seed device as a
> VM/container template, but no, it doesn't get much attention.
> 
> 
> I'm not asking for deprecate the feature just because I don't like it.
> 
> This feature is asking for too many exceptions, from the extra
> fs_devices hack (we have in fact two fs_devices, one for rw devices, one
> for seed only), to the dev add ioctl.
> 
> But the little benefit is not really worthy for the cost.

We use seed devices in production. The use case is for servers where we
don't want to persist any sensitive data. The seed device contains a
basic boot environment, then we sprout it with a dm-crypt device and
throw away the encryption key. This ensures that nothing sensitive can
ever be recovered. We previously did this with overlayfs, but seed
devices ended up working better for reasons I don't remember.

Davide Cavalca also told me that "Fedora is also interested in
leveraging seed devices down the road. e.g. doing seed/sprout for cloud
provisioning, and using seed devices for OEM factory recovery on
laptops."

There are definitely hacks we need to fix for seed devices, but there
are valid use cases and we can't just deprecate it.

Re: Seed device is broken, again.

[Qu Wenruo]

On 2022/2/26 03:18, Omar Sandoval wrote:
> On Fri, Feb 25, 2022 at 09:36:32PM +0800, Qu Wenruo wrote:
>>
>>
>> On 2022/2/25 19:47, David Sterba wrote:
>>> On Fri, Feb 25, 2022 at 06:08:20PM +0800, Qu Wenruo wrote:
>>>> Hi,
>>>>
>>>> The very basic seed device usage is broken again:
>>>>
>>>> 	mkfs.btrfs -f $dev1 > /dev/null
>>>> 	btrfstune -S 1 $dev1
>>>> 	mount $dev1 $mnt
>>>> 	btrfs dev add $dev2 $mnt
>>>> 	umount $mnt
>>>>
>>>>
>>>> I'm not sure how many guys are really using seed device.
>>>>
>>>> But I see a lot of weird operations, like calling a definite write
>>>> operation (device add) on a RO mounted fs.
>>>
>>> That's how the seeding device works, in what way would you want to do
>>> the ro->rw change?
>>
>> In progs-only, so that in kernel we can make dev add ioctl as a real RW
>> operation, not adding an exception for dev add.
>>
>>>
>>>> Can we make at least the seed sprouting part into btrfs-progs instead?
>>>
>>> How? What do you mean? This is an in kernel operation that is done on a
>>> mounted filesystem, progs can't do that.
>>
>> Why not?
>>
>> Progs has the same ability to read-write the fs, I see no reason why we
>> shouldn't do it in user space.
>>
>> We're just adding a device, update related trees (which will only be
>> written to the new device). I see no special reason why it can't be done
>> in user space.
>>
>> Furthermore, the ability to add device in user space can be a good
>> safenet for some ENOSPC space.
>>
>>>
>>>> And can seed device even support the upcoming extent-tree-v2?
>>>
>>> I should, it operates on the device level.
>>>
>>>> Personally speaking I prefer to mark seed device deprecated completely.
>>>
>>> If we did that with every feature some developer does not like we'd have
>>> nothing left you know. Seeding is a documented usecase, as long as it
>>> works it's fine to have it available.
>>
>> A documented usecase doesn't mean it can't be deprecated.
>>
>> Furthermore, a documented use case doesn't validate the use case itself.
>>
>> So, please tell me when did you use seed device last time?
>> Or anyone in the mail list, please tell me some real world usecase for
>> seed devices.
>>
>> I did remember some planned use case like a way to use seed device as a
>> VM/container template, but no, it doesn't get much attention.
>>
>>
>> I'm not asking for deprecate the feature just because I don't like it.
>>
>> This feature is asking for too many exceptions, from the extra
>> fs_devices hack (we have in fact two fs_devices, one for rw devices, one
>> for seed only), to the dev add ioctl.
>>
>> But the little benefit is not really worthy for the cost.
>
> We use seed devices in production. The use case is for servers where we
> don't want to persist any sensitive data. The seed device contains a
> basic boot environment, then we sprout it with a dm-crypt device and
> throw away the encryption key. This ensures that nothing sensitive can
> ever be recovered. We previously did this with overlayfs, but seed
> devices ended up working better for reasons I don't remember.
>
> Davide Cavalca also told me that "Fedora is also interested in
> leveraging seed devices down the road. e.g. doing seed/sprout for cloud
> provisioning, and using seed devices for OEM factory recovery on
> laptops."
>
> There are definitely hacks we need to fix for seed devices, but there
> are valid use cases and we can't just deprecate it.

OK, then it looks we need to keep the feature.

Then would you mind to share your preference between things like:

a) The current way
    mkfs + btrfstune + mount + dev add

b) All user space way
    mkfs /dev/seed
    btrfstune -S 1 /dev/seed
    mkfs -S /dev/seed /dev/new
    (using seed device as seed, and sprout to the new device)

With method b, at least we can make dev add ioctl to be completely RW in
the long run.

Thanks,
Qu

Re: Seed device is broken, again.

[Anand Jain]

On 28/02/2022 07:56, Qu Wenruo wrote:
> 
> 
> On 2022/2/26 03:18, Omar Sandoval wrote:
>> On Fri, Feb 25, 2022 at 09:36:32PM +0800, Qu Wenruo wrote:
>>>
>>>
>>> On 2022/2/25 19:47, David Sterba wrote:
>>>> On Fri, Feb 25, 2022 at 06:08:20PM +0800, Qu Wenruo wrote:
>>>>> Hi,
>>>>>
>>>>> The very basic seed device usage is broken again:
>>>>>
>>>>>     mkfs.btrfs -f $dev1 > /dev/null
>>>>>     btrfstune -S 1 $dev1
>>>>>     mount $dev1 $mnt
>>>>>     btrfs dev add $dev2 $mnt
>>>>>     umount $mnt
>>>>>
>>>>>
>>>>> I'm not sure how many guys are really using seed device.
>>>>>
>>>>> But I see a lot of weird operations, like calling a definite write
>>>>> operation (device add) on a RO mounted fs.
>>>>
>>>> That's how the seeding device works, in what way would you want to do
>>>> the ro->rw change?
>>>
>>> In progs-only, so that in kernel we can make dev add ioctl as a real RW
>>> operation, not adding an exception for dev add.
>>>
>>>>
>>>>> Can we make at least the seed sprouting part into btrfs-progs instead?
>>>>
>>>> How? What do you mean? This is an in kernel operation that is done on a
>>>> mounted filesystem, progs can't do that.
>>>
>>> Why not?
>>>
>>> Progs has the same ability to read-write the fs, I see no reason why we
>>> shouldn't do it in user space.
>>>
>>> We're just adding a device, update related trees (which will only be
>>> written to the new device). I see no special reason why it can't be done
>>> in user space.
>>>
>>> Furthermore, the ability to add device in user space can be a good
>>> safenet for some ENOSPC space.
>>>
>>>>
>>>>> And can seed device even support the upcoming extent-tree-v2?
>>>>
>>>> I should, it operates on the device level.
>>>>
>>>>> Personally speaking I prefer to mark seed device deprecated 
>>>>> completely.
>>>>
>>>> If we did that with every feature some developer does not like we'd 
>>>> have
>>>> nothing left you know. Seeding is a documented usecase, as long as it
>>>> works it's fine to have it available.
>>>
>>> A documented usecase doesn't mean it can't be deprecated.
>>>
>>> Furthermore, a documented use case doesn't validate the use case itself.
>>>
>>> So, please tell me when did you use seed device last time?
>>> Or anyone in the mail list, please tell me some real world usecase for
>>> seed devices.
>>>
>>> I did remember some planned use case like a way to use seed device as a
>>> VM/container template, but no, it doesn't get much attention.
>>>
>>>
>>> I'm not asking for deprecate the feature just because I don't like it.
>>>
>>> This feature is asking for too many exceptions, from the extra
>>> fs_devices hack (we have in fact two fs_devices, one for rw devices, one
>>> for seed only), to the dev add ioctl.
>>>
>>> But the little benefit is not really worthy for the cost.
>>
>> We use seed devices in production. The use case is for servers where we
>> don't want to persist any sensitive data. The seed device contains a
>> basic boot environment, then we sprout it with a dm-crypt device and
>> throw away the encryption key. This ensures that nothing sensitive can
>> ever be recovered. We previously did this with overlayfs, but seed
>> devices ended up working better for reasons I don't remember.
>>
>> Davide Cavalca also told me that "Fedora is also interested in
>> leveraging seed devices down the road. e.g. doing seed/sprout for cloud
>> provisioning, and using seed devices for OEM factory recovery on
>> laptops."
>>
>> There are definitely hacks we need to fix for seed devices, but there
>> are valid use cases and we can't just deprecate it.
> 
> OK, then it looks we need to keep the feature.
> 
> Then would you mind to share your preference between things like:
> 
> a) The current way
>     mkfs + btrfstune + mount + dev add

   And further, dev-remove seed if needed.

> b) All user space way
>     mkfs /dev/seed
>     btrfstune -S 1 /dev/seed
>     mkfs -S /dev/seed /dev/new
>     (using seed device as seed, and sprout to the new device)

  Does the -S option copy all blocks here?
  How does it work if /dev/new needed to be an independent filesystem
  only at some later point? Add another btrfstune option?

> With method b, at least we can make dev add ioctl to be completely RW in
> the long run.

  Could you please add more clarity here? Very confusing.

Thanks, Anand

> Thanks,
> Qu

Re: Seed device is broken, again.

[Qu Wenruo]

On 2022/2/28 10:01, Anand Jain wrote:
> On 28/02/2022 07:56, Qu Wenruo wrote:
>>
>>
>> On 2022/2/26 03:18, Omar Sandoval wrote:
>>> On Fri, Feb 25, 2022 at 09:36:32PM +0800, Qu Wenruo wrote:
>>>>
>>>>
>>>> On 2022/2/25 19:47, David Sterba wrote:
>>>>> On Fri, Feb 25, 2022 at 06:08:20PM +0800, Qu Wenruo wrote:
>>>>>> Hi,
>>>>>>
>>>>>> The very basic seed device usage is broken again:
>>>>>>
>>>>>>     mkfs.btrfs -f $dev1 > /dev/null
>>>>>>     btrfstune -S 1 $dev1
>>>>>>     mount $dev1 $mnt
>>>>>>     btrfs dev add $dev2 $mnt
>>>>>>     umount $mnt
>>>>>>
>>>>>>
>>>>>> I'm not sure how many guys are really using seed device.
>>>>>>
>>>>>> But I see a lot of weird operations, like calling a definite write
>>>>>> operation (device add) on a RO mounted fs.
>>>>>
>>>>> That's how the seeding device works, in what way would you want to do
>>>>> the ro->rw change?
>>>>
>>>> In progs-only, so that in kernel we can make dev add ioctl as a real RW
>>>> operation, not adding an exception for dev add.
>>>>
>>>>>
>>>>>> Can we make at least the seed sprouting part into btrfs-progs
>>>>>> instead?
>>>>>
>>>>> How? What do you mean? This is an in kernel operation that is done
>>>>> on a
>>>>> mounted filesystem, progs can't do that.
>>>>
>>>> Why not?
>>>>
>>>> Progs has the same ability to read-write the fs, I see no reason why we
>>>> shouldn't do it in user space.
>>>>
>>>> We're just adding a device, update related trees (which will only be
>>>> written to the new device). I see no special reason why it can't be
>>>> done
>>>> in user space.
>>>>
>>>> Furthermore, the ability to add device in user space can be a good
>>>> safenet for some ENOSPC space.
>>>>
>>>>>
>>>>>> And can seed device even support the upcoming extent-tree-v2?
>>>>>
>>>>> I should, it operates on the device level.
>>>>>
>>>>>> Personally speaking I prefer to mark seed device deprecated
>>>>>> completely.
>>>>>
>>>>> If we did that with every feature some developer does not like we'd
>>>>> have
>>>>> nothing left you know. Seeding is a documented usecase, as long as it
>>>>> works it's fine to have it available.
>>>>
>>>> A documented usecase doesn't mean it can't be deprecated.
>>>>
>>>> Furthermore, a documented use case doesn't validate the use case
>>>> itself.
>>>>
>>>> So, please tell me when did you use seed device last time?
>>>> Or anyone in the mail list, please tell me some real world usecase for
>>>> seed devices.
>>>>
>>>> I did remember some planned use case like a way to use seed device as a
>>>> VM/container template, but no, it doesn't get much attention.
>>>>
>>>>
>>>> I'm not asking for deprecate the feature just because I don't like it.
>>>>
>>>> This feature is asking for too many exceptions, from the extra
>>>> fs_devices hack (we have in fact two fs_devices, one for rw devices,
>>>> one
>>>> for seed only), to the dev add ioctl.
>>>>
>>>> But the little benefit is not really worthy for the cost.
>>>
>>> We use seed devices in production. The use case is for servers where we
>>> don't want to persist any sensitive data. The seed device contains a
>>> basic boot environment, then we sprout it with a dm-crypt device and
>>> throw away the encryption key. This ensures that nothing sensitive can
>>> ever be recovered. We previously did this with overlayfs, but seed
>>> devices ended up working better for reasons I don't remember.
>>>
>>> Davide Cavalca also told me that "Fedora is also interested in
>>> leveraging seed devices down the road. e.g. doing seed/sprout for cloud
>>> provisioning, and using seed devices for OEM factory recovery on
>>> laptops."
>>>
>>> There are definitely hacks we need to fix for seed devices, but there
>>> are valid use cases and we can't just deprecate it.
>>
>> OK, then it looks we need to keep the feature.
>>
>> Then would you mind to share your preference between things like:
>>
>> a) The current way
>>     mkfs + btrfstune + mount + dev add
>
>    And further, dev-remove seed if needed.
>
>> b) All user space way
>>     mkfs /dev/seed
>>     btrfstune -S 1 /dev/seed
>>     mkfs -S /dev/seed /dev/new
>>     (using seed device as seed, and sprout to the new device)
>
>   Does the -S option copy all blocks here?

Nope, just the same as dev add.

>   How does it work if /dev/new needed to be an independent filesystem
>   only at some later point? Add another btrfstune option?

The same dev remove.

>
>> With method b, at least we can make dev add ioctl to be completely RW in
>> the long run.
>
>   Could you please add more clarity here? Very confusing.

Isn't it an awful thing that device add ioctl can even be executed on a
RO mounted fs?

Thanks,
Qu

>
> Thanks, Anand
>
>> Thanks,
>> Qu
>

Re: Seed device is broken, again.

[Anand Jain]

On 28/02/2022 10:35, Qu Wenruo wrote:
> 
> 
> On 2022/2/28 10:01, Anand Jain wrote:
>> On 28/02/2022 07:56, Qu Wenruo wrote:
>>>
>>>
>>> On 2022/2/26 03:18, Omar Sandoval wrote:
>>>> On Fri, Feb 25, 2022 at 09:36:32PM +0800, Qu Wenruo wrote:
>>>>>
>>>>>
>>>>> On 2022/2/25 19:47, David Sterba wrote:
>>>>>> On Fri, Feb 25, 2022 at 06:08:20PM +0800, Qu Wenruo wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> The very basic seed device usage is broken again:
>>>>>>>
>>>>>>>     mkfs.btrfs -f $dev1 > /dev/null
>>>>>>>     btrfstune -S 1 $dev1
>>>>>>>     mount $dev1 $mnt
>>>>>>>     btrfs dev add $dev2 $mnt
>>>>>>>     umount $mnt
>>>>>>>
>>>>>>>
>>>>>>> I'm not sure how many guys are really using seed device.
>>>>>>>
>>>>>>> But I see a lot of weird operations, like calling a definite write
>>>>>>> operation (device add) on a RO mounted fs.
>>>>>>
>>>>>> That's how the seeding device works, in what way would you want to do
>>>>>> the ro->rw change?
>>>>>
>>>>> In progs-only, so that in kernel we can make dev add ioctl as a 
>>>>> real RW
>>>>> operation, not adding an exception for dev add.
>>>>>
>>>>>>
>>>>>>> Can we make at least the seed sprouting part into btrfs-progs
>>>>>>> instead?
>>>>>>
>>>>>> How? What do you mean? This is an in kernel operation that is done
>>>>>> on a
>>>>>> mounted filesystem, progs can't do that.
>>>>>
>>>>> Why not?
>>>>>
>>>>> Progs has the same ability to read-write the fs, I see no reason 
>>>>> why we
>>>>> shouldn't do it in user space.
>>>>>
>>>>> We're just adding a device, update related trees (which will only be
>>>>> written to the new device). I see no special reason why it can't be
>>>>> done
>>>>> in user space.
>>>>>
>>>>> Furthermore, the ability to add device in user space can be a good
>>>>> safenet for some ENOSPC space.
>>>>>
>>>>>>
>>>>>>> And can seed device even support the upcoming extent-tree-v2?
>>>>>>
>>>>>> I should, it operates on the device level.
>>>>>>
>>>>>>> Personally speaking I prefer to mark seed device deprecated
>>>>>>> completely.
>>>>>>
>>>>>> If we did that with every feature some developer does not like we'd
>>>>>> have
>>>>>> nothing left you know. Seeding is a documented usecase, as long as it
>>>>>> works it's fine to have it available.
>>>>>
>>>>> A documented usecase doesn't mean it can't be deprecated.
>>>>>
>>>>> Furthermore, a documented use case doesn't validate the use case
>>>>> itself.
>>>>>
>>>>> So, please tell me when did you use seed device last time?
>>>>> Or anyone in the mail list, please tell me some real world usecase for
>>>>> seed devices.
>>>>>
>>>>> I did remember some planned use case like a way to use seed device 
>>>>> as a
>>>>> VM/container template, but no, it doesn't get much attention.
>>>>>
>>>>>
>>>>> I'm not asking for deprecate the feature just because I don't like it.
>>>>>
>>>>> This feature is asking for too many exceptions, from the extra
>>>>> fs_devices hack (we have in fact two fs_devices, one for rw devices,
>>>>> one
>>>>> for seed only), to the dev add ioctl.
>>>>>
>>>>> But the little benefit is not really worthy for the cost.
>>>>
>>>> We use seed devices in production. The use case is for servers where we
>>>> don't want to persist any sensitive data. The seed device contains a
>>>> basic boot environment, then we sprout it with a dm-crypt device and
>>>> throw away the encryption key. This ensures that nothing sensitive can
>>>> ever be recovered. We previously did this with overlayfs, but seed
>>>> devices ended up working better for reasons I don't remember.
>>>>
>>>> Davide Cavalca also told me that "Fedora is also interested in
>>>> leveraging seed devices down the road. e.g. doing seed/sprout for cloud
>>>> provisioning, and using seed devices for OEM factory recovery on
>>>> laptops."
>>>>
>>>> There are definitely hacks we need to fix for seed devices, but there
>>>> are valid use cases and we can't just deprecate it.
>>>
>>> OK, then it looks we need to keep the feature.
>>>
>>> Then would you mind to share your preference between things like:
>>>
>>> a) The current way
>>>     mkfs + btrfstune + mount + dev add
>>
>>    And further, dev-remove seed if needed.
>>
>>> b) All user space way
>>>     mkfs /dev/seed
>>>     btrfstune -S 1 /dev/seed
>>>     mkfs -S /dev/seed /dev/new
>>>     (using seed device as seed, and sprout to the new device)
>>
>>   Does the -S option copy all blocks here?
> 
> Nope, just the same as dev add.
> 
>>   How does it work if /dev/new needed to be an independent filesystem
>>   only at some later point? Add another btrfstune option?
> 
> The same dev remove.
> 
>>
>>> With method b, at least we can make dev add ioctl to be completely RW in
>>> the long run.
>>
>>   Could you please add more clarity here? Very confusing.
> 
> Isn't it an awful thing that device add ioctl can even be executed on a
> RO mounted fs?
> 

Ah. That's fine, IMO. It is a matter of understanding the nature of the
seed device. No?
The RO mount used to turn into an RW mount after the device-add a long
ago. It got changed without a trace.

Thanks, Anand


> Thanks,
> Qu
> 
>>
>> Thanks, Anand
>>
>>> Thanks,
>>> Qu
>>

Re: Seed device is broken, again.

[Qu Wenruo]

On 2022/2/28 11:24, Anand Jain wrote:
>
>
> On 28/02/2022 10:35, Qu Wenruo wrote:
>>
>>
>> On 2022/2/28 10:01, Anand Jain wrote:
>>> On 28/02/2022 07:56, Qu Wenruo wrote:
>>>>
>>>>
>>>> On 2022/2/26 03:18, Omar Sandoval wrote:
>>>>> On Fri, Feb 25, 2022 at 09:36:32PM +0800, Qu Wenruo wrote:
>>>>>>
>>>>>>
>>>>>> On 2022/2/25 19:47, David Sterba wrote:
>>>>>>> On Fri, Feb 25, 2022 at 06:08:20PM +0800, Qu Wenruo wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> The very basic seed device usage is broken again:
>>>>>>>>
>>>>>>>>     mkfs.btrfs -f $dev1 > /dev/null
>>>>>>>>     btrfstune -S 1 $dev1
>>>>>>>>     mount $dev1 $mnt
>>>>>>>>     btrfs dev add $dev2 $mnt
>>>>>>>>     umount $mnt
>>>>>>>>
>>>>>>>>
>>>>>>>> I'm not sure how many guys are really using seed device.
>>>>>>>>
>>>>>>>> But I see a lot of weird operations, like calling a definite write
>>>>>>>> operation (device add) on a RO mounted fs.
>>>>>>>
>>>>>>> That's how the seeding device works, in what way would you want
>>>>>>> to do
>>>>>>> the ro->rw change?
>>>>>>
>>>>>> In progs-only, so that in kernel we can make dev add ioctl as a
>>>>>> real RW
>>>>>> operation, not adding an exception for dev add.
>>>>>>
>>>>>>>
>>>>>>>> Can we make at least the seed sprouting part into btrfs-progs
>>>>>>>> instead?
>>>>>>>
>>>>>>> How? What do you mean? This is an in kernel operation that is done
>>>>>>> on a
>>>>>>> mounted filesystem, progs can't do that.
>>>>>>
>>>>>> Why not?
>>>>>>
>>>>>> Progs has the same ability to read-write the fs, I see no reason
>>>>>> why we
>>>>>> shouldn't do it in user space.
>>>>>>
>>>>>> We're just adding a device, update related trees (which will only be
>>>>>> written to the new device). I see no special reason why it can't be
>>>>>> done
>>>>>> in user space.
>>>>>>
>>>>>> Furthermore, the ability to add device in user space can be a good
>>>>>> safenet for some ENOSPC space.
>>>>>>
>>>>>>>
>>>>>>>> And can seed device even support the upcoming extent-tree-v2?
>>>>>>>
>>>>>>> I should, it operates on the device level.
>>>>>>>
>>>>>>>> Personally speaking I prefer to mark seed device deprecated
>>>>>>>> completely.
>>>>>>>
>>>>>>> If we did that with every feature some developer does not like we'd
>>>>>>> have
>>>>>>> nothing left you know. Seeding is a documented usecase, as long
>>>>>>> as it
>>>>>>> works it's fine to have it available.
>>>>>>
>>>>>> A documented usecase doesn't mean it can't be deprecated.
>>>>>>
>>>>>> Furthermore, a documented use case doesn't validate the use case
>>>>>> itself.
>>>>>>
>>>>>> So, please tell me when did you use seed device last time?
>>>>>> Or anyone in the mail list, please tell me some real world usecase
>>>>>> for
>>>>>> seed devices.
>>>>>>
>>>>>> I did remember some planned use case like a way to use seed device
>>>>>> as a
>>>>>> VM/container template, but no, it doesn't get much attention.
>>>>>>
>>>>>>
>>>>>> I'm not asking for deprecate the feature just because I don't like
>>>>>> it.
>>>>>>
>>>>>> This feature is asking for too many exceptions, from the extra
>>>>>> fs_devices hack (we have in fact two fs_devices, one for rw devices,
>>>>>> one
>>>>>> for seed only), to the dev add ioctl.
>>>>>>
>>>>>> But the little benefit is not really worthy for the cost.
>>>>>
>>>>> We use seed devices in production. The use case is for servers
>>>>> where we
>>>>> don't want to persist any sensitive data. The seed device contains a
>>>>> basic boot environment, then we sprout it with a dm-crypt device and
>>>>> throw away the encryption key. This ensures that nothing sensitive can
>>>>> ever be recovered. We previously did this with overlayfs, but seed
>>>>> devices ended up working better for reasons I don't remember.
>>>>>
>>>>> Davide Cavalca also told me that "Fedora is also interested in
>>>>> leveraging seed devices down the road. e.g. doing seed/sprout for
>>>>> cloud
>>>>> provisioning, and using seed devices for OEM factory recovery on
>>>>> laptops."
>>>>>
>>>>> There are definitely hacks we need to fix for seed devices, but there
>>>>> are valid use cases and we can't just deprecate it.
>>>>
>>>> OK, then it looks we need to keep the feature.
>>>>
>>>> Then would you mind to share your preference between things like:
>>>>
>>>> a) The current way
>>>>     mkfs + btrfstune + mount + dev add
>>>
>>>    And further, dev-remove seed if needed.
>>>
>>>> b) All user space way
>>>>     mkfs /dev/seed
>>>>     btrfstune -S 1 /dev/seed
>>>>     mkfs -S /dev/seed /dev/new
>>>>     (using seed device as seed, and sprout to the new device)
>>>
>>>   Does the -S option copy all blocks here?
>>
>> Nope, just the same as dev add.
>>
>>>   How does it work if /dev/new needed to be an independent filesystem
>>>   only at some later point? Add another btrfstune option?
>>
>> The same dev remove.
>>
>>>
>>>> With method b, at least we can make dev add ioctl to be completely
>>>> RW in
>>>> the long run.
>>>
>>>   Could you please add more clarity here? Very confusing.
>>
>> Isn't it an awful thing that device add ioctl can even be executed on a
>> RO mounted fs?
>>
>
> Ah. That's fine, IMO. It is a matter of understanding the nature of the
> seed device. No?
> The RO mount used to turn into an RW mount after the device-add a long
> ago. It got changed without a trace.

Think twice about this, have you every seen another fs allowing a RO
mount to be converted to RW without even remounting?

Still think this doesn't provide any surprise to end users?

Thanks,
Qu

>
> Thanks, Anand
>
>
>> Thanks,
>> Qu
>>
>>>
>>> Thanks, Anand
>>>
>>>> Thanks,
>>>> Qu
>>>

Re: Seed device is broken, again.

[David Sterba]

On Mon, Feb 28, 2022 at 11:27:16AM +0800, Qu Wenruo wrote:
> > Ah. That's fine, IMO. It is a matter of understanding the nature of the
> > seed device. No?
> > The RO mount used to turn into an RW mount after the device-add a long
> > ago. It got changed without a trace.
> 
> Think twice about this, have you every seen another fs allowing a RO
> mount to be converted to RW without even remounting?

There's no other filesystem with a remotely close feature so we can't
follow an established pattern.

The ro->rw transition can be done from inside the filesystem too and
btrfs sort of does that in btrfs_mount, calling kern_mount.

> Still think this doesn't provide any surprise to end users?

The RO status means the filesystem does not support any write operations
from the user perspective that go through VFS. Adding the device in the
seed mode modifies the filesystem structures, ie. changes the block
device, but does not change the VFS status regarding writability.  The
read-write remount is mandatory to actually make use of the writable
device. This is documented and I don't find it confusing from the end
user perspective.

If you're concerned that there's a write to the block device then yes,
it's a bug that the mnt_set_write should be done just before we start
any change operation.

There was a discussion some time ago if the log replay should happen on
a read-only mount, or any potential repair action that could happen
regardless of the ro/rw mount status. The conclusion was that it could
happen, and guaranteeing no writes to the block device should be done
externally eg. by blockdev --setro. But I think we opted not to do any
writes anyway for btrfs.

Re: Seed device is broken, again.

[Qu Wenruo]

On 2022/3/1 02:40, David Sterba wrote:
> On Mon, Feb 28, 2022 at 11:27:16AM +0800, Qu Wenruo wrote:
>>> Ah. That's fine, IMO. It is a matter of understanding the nature of the
>>> seed device. No?
>>> The RO mount used to turn into an RW mount after the device-add a long
>>> ago. It got changed without a trace.
>>
>> Think twice about this, have you every seen another fs allowing a RO
>> mount to be converted to RW without even remounting?
>
> There's no other filesystem with a remotely close feature so we can't
> follow an established pattern.
>
> The ro->rw transition can be done from inside the filesystem too and
> btrfs sort of does that in btrfs_mount, calling kern_mount.
>
>> Still think this doesn't provide any surprise to end users?
>
> The RO status means the filesystem does not support any write operations
> from the user perspective that go through VFS. Adding the device in the
> seed mode modifies the filesystem structures, ie. changes the block
> device, but does not change the VFS status regarding writability.  The
> read-write remount is mandatory to actually make use of the writable
> device. This is documented and I don't find it confusing from the end
> user perspective.
>
> If you're concerned that there's a write to the block device then yes,
> it's a bug that the mnt_set_write should be done just before we start
> any change operation.

I'm not concerned with that at all.

>
> There was a discussion some time ago if the log replay should happen on
> a read-only mount, or any potential repair action that could happen
> regardless of the ro/rw mount status. The conclusion was that it could
> happen, and guaranteeing no writes to the block device should be done
> externally eg. by blockdev --setro. But I think we opted not to do any
> writes anyway for btrfs.

My main concern is still there, we change RO to RW without any remount.

It's weird from the beginning, but we just accept that.

If we have chance to rethink this, would we still take the same path?
Other than making seed device into user space tool like mkfs?

THanks,
Qu

Re: Seed device is broken, again.

[Chris Murphy]

On Sun, Feb 27, 2022 at 7:35 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
> >> With method b, at least we can make dev add ioctl to be completely RW in
> >> the long run.
> >
> >   Could you please add more clarity here? Very confusing.
>
> Isn't it an awful thing that device add ioctl can even be executed on a
> RO mounted fs?

I think it's more idiosyncratic than awful. It is definitely
confusing. But I think the explanation is that the ro mount is a VFS
option, not a Btrfs option. It only indicates we cannot write from
user space to this file system while it's ro. Btrfs still can,
including playing the tree log (and I assume it updates trees on disk,
even if ro but without notreelog.

There's a wide assortment of confusion in the `mount` command's
results. You just have to depend on rote memorization to know what
things are VFS or file system specific. Maybe it'd be nice if the
output were clearer but ...

/dev/nvme0n1p5 on /home type btrfs
(rw,noatime,seclabel,compress=zstd:1,ssd,space_cache=v2,subvolid=271,subvol=/home)

Is seclabel VFS? Or yet a third domain? Anyway, for example:

/dev/nvme0n1p5 on /home type btrfs
(VFS:rw,noatime,seclabel)(btrfs:compress=zstd:1,ssd,space_cache=v2,subvolid=271,subvol=/home)


-- 
Chris Murphy

Re: Seed device is broken, again.

[Chris Murphy]

On Mon, Feb 28, 2022 at 5:13 PM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
>
>
> On 2022/3/1 02:40, David Sterba wrote:
> > On Mon, Feb 28, 2022 at 11:27:16AM +0800, Qu Wenruo wrote:
> >>> Ah. That's fine, IMO. It is a matter of understanding the nature of the
> >>> seed device. No?
> >>> The RO mount used to turn into an RW mount after the device-add a long
> >>> ago. It got changed without a trace.
> >>
> >> Think twice about this, have you every seen another fs allowing a RO
> >> mount to be converted to RW without even remounting?
> >
> > There's no other filesystem with a remotely close feature so we can't
> > follow an established pattern.
> >
> > The ro->rw transition can be done from inside the filesystem too and
> > btrfs sort of does that in btrfs_mount, calling kern_mount.
> >
> >> Still think this doesn't provide any surprise to end users?
> >
> > The RO status means the filesystem does not support any write operations
> > from the user perspective that go through VFS. Adding the device in the
> > seed mode modifies the filesystem structures, ie. changes the block
> > device, but does not change the VFS status regarding writability.  The
> > read-write remount is mandatory to actually make use of the writable
> > device. This is documented and I don't find it confusing from the end
> > user perspective.
> >
> > If you're concerned that there's a write to the block device then yes,
> > it's a bug that the mnt_set_write should be done just before we start
> > any change operation.
>
> I'm not concerned with that at all.
>
> >
> > There was a discussion some time ago if the log replay should happen on
> > a read-only mount, or any potential repair action that could happen
> > regardless of the ro/rw mount status. The conclusion was that it could
> > happen, and guaranteeing no writes to the block device should be done
> > externally eg. by blockdev --setro. But I think we opted not to do any
> > writes anyway for btrfs.
>
> My main concern is still there, we change RO to RW without any remount.

Well, it hasn't been changed to rw yet - `btrfs dev add` is only btrfs
kernel code writing to it, user space still can't write until the user
`mount -o remount,rw` and now you can write to the sprout or remove
the seed to replicate it.

> It's weird from the beginning, but we just accept that.

It's unexpected but I think the story is still consistent.

> If we have chance to rethink this, would we still take the same path?
> Other than making seed device into user space tool like mkfs?

I do not mind revising this so that device add sprout is only
supported in user space.


-- 
Chris Murphy

Re: Seed device is broken, again.

[David Sterba]

On Tue, Mar 01, 2022 at 08:13:28AM +0800, Qu Wenruo wrote:
> 
> >
> > There was a discussion some time ago if the log replay should happen on
> > a read-only mount, or any potential repair action that could happen
> > regardless of the ro/rw mount status. The conclusion was that it could
> > happen, and guaranteeing no writes to the block device should be done
> > externally eg. by blockdev --setro. But I think we opted not to do any
> > writes anyway for btrfs.
> 
> My main concern is still there, we change RO to RW without any remount.

We also do emergency remount from RO to RW, so I take it as that
changing the status is partiall in the hands of filesystem, not
violating APIs and such.

> It's weird from the beginning, but we just accept that.
> 
> If we have chance to rethink this, would we still take the same path?
> Other than making seed device into user space tool like mkfs?

I'm not sure it would work the same way as now, the seeding device can
be mounted several times in parallel as the UUID is generated at each
mount, then added the writeable device, then thrown away. Any detour
through userspace make it more complicated, but I haven't thought that
through. We could add it no top of the on-line seeding as another
usecase but if current users are fine with how it works now I don't
think we need to spend time on implementing it.

Re: Seed device is broken, again.

[Qu Wenruo]

On 2022/3/2 01:09, David Sterba wrote:
> On Tue, Mar 01, 2022 at 08:13:28AM +0800, Qu Wenruo wrote:
>>
>>>
>>> There was a discussion some time ago if the log replay should happen on
>>> a read-only mount, or any potential repair action that could happen
>>> regardless of the ro/rw mount status. The conclusion was that it could
>>> happen, and guaranteeing no writes to the block device should be done
>>> externally eg. by blockdev --setro. But I think we opted not to do any
>>> writes anyway for btrfs.
>>
>> My main concern is still there, we change RO to RW without any remount.
>
> We also do emergency remount from RO to RW, so I take it as that
> changing the status is partiall in the hands of filesystem, not
> violating APIs and such.
>
>> It's weird from the beginning, but we just accept that.
>>
>> If we have chance to rethink this, would we still take the same path?
>> Other than making seed device into user space tool like mkfs?
>
> I'm not sure it would work the same way as now, the seeding device can
> be mounted several times in parallel as the UUID is generated at each
> mount, then added the writeable device, then thrown away. Any detour
> through userspace make it more complicated, but I haven't thought that
> through. We could add it no top of the on-line seeding as another
> usecase but if current users are fine with how it works now I don't
> think we need to spend time on implementing it.

Then let me try that.

At least to me, using seed device to create a new RW fs really looks
like a perfect match to mkfs.btrfs.

We already have things like mkfs.btrfs -R to populate the fs.
Using seed device would be more suitable to me.

Thanks,
Qu

Re: Seed device is broken, again.

[Neal Gompa]

On Fri, Feb 25, 2022 at 9:44 PM David Sterba <dsterba@suse.cz> wrote:
>
> On Fri, Feb 25, 2022 at 06:08:20PM +0800, Qu Wenruo wrote:
> > Hi,
> >
> > The very basic seed device usage is broken again:
> >
> >       mkfs.btrfs -f $dev1 > /dev/null
> >       btrfstune -S 1 $dev1
> >       mount $dev1 $mnt
> >       btrfs dev add $dev2 $mnt
> >       umount $mnt
> >
> >
> > I'm not sure how many guys are really using seed device.
> >
> > But I see a lot of weird operations, like calling a definite write
> > operation (device add) on a RO mounted fs.
>
> That's how the seeding device works, in what way would you want to do
> the ro->rw change?
>
> > Can we make at least the seed sprouting part into btrfs-progs instead?
>
> How? What do you mean? This is an in kernel operation that is done on a
> mounted filesystem, progs can't do that.
>
> > And can seed device even support the upcoming extent-tree-v2?
>
> I should, it operates on the device level.
>
> > Personally speaking I prefer to mark seed device deprecated completely.
>
> If we did that with every feature some developer does not like we'd have
> nothing left you know. Seeding is a documented usecase, as long as it
> works it's fine to have it available.
>

For what it's worth, I've been interested in using seed-sprout for
live media for a while now. I just haven't had time to experiment with
it yet...

-- 
真実はいつも一つ！/ Always, there's only one truth!