Adding Device Fails - Why?

Adding Device Fails - Why?

[Christian Völker]

Hi all,

I am using BTRS on a Debian10 system. I am trying to extend my existing 
filesystem with another device but adding it fails for no reason.

This is my setup of existing btrfs:

  2x DRBD Devices (Network RAID1)
  top of each a luks encrypted device (crypt_drbd1 and crypt_drbd3):

vdb                         254:16   0  1,1T  0 disk
└─drbd1                     147:1    0  1,1T  0 disk
   └─crypt_drbd1             253:3    0  1,1T  0 crypt
vdc                         254:32   0  900G  0 disk
└─drbd2                     147:2    0  900G  0 disk
   └─crypt2                  253:4    0  900G  0 crypt
vdd                         254:48   0  800G  0 disk
└─drbd3                     147:3    0  800G  0 disk
   └─crypt_drbd3             253:5    0  800G  0 crypt /var/lib/backuppc



I have now a third drbd device (drbd2) which I encrypted, too (crypt2). 
And tried to add to existing fi.
Here further system information:

Linux backuppc41 5.10.0-3-amd64 #1 SMP Debian 5.10.13-1 (2021-02-06) 
x86_64 GNU/Linux
btrfs-progs v5.10.1

root@backuppc41:~# btrfs fi sh
Label: 'backcuppc'  uuid: 73b98c7b-832a-437a-a15b-6cb00734e5db
         Total devices 2 FS bytes used 1.83TiB
         devid    3 size 799.96GiB used 789.96GiB path dm-5
         devid    4 size 1.07TiB used 1.06TiB path dm-3


I can create an additional btrfs filesystem with mkfs.btrfs on the new 
device without any issues:

root@backuppc41:~# btrfs fi sh
Label: 'backcuppc'  uuid: 73b98c7b-832a-437a-a15b-6cb00734e5db
         Total devices 2 FS bytes used 1.83TiB
         devid    3 size 799.96GiB used 789.96GiB path dm-5
         devid    4 size 1.07TiB used 1.06TiB path dm-3

Label: none  uuid: b111a08e-2969-457a-b9f1-551ff65451d1
         Total devices 1 FS bytes used 128.00KiB
         devid    1 size 899.96GiB used 2.02GiB path /dev/mapper/crypt2


But I can not add this device to the existing btrfs fi:
root@backuppc41:~# wipefs /dev/mapper/crypt2 -a
/dev/mapper/crypt2: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 
42 48 52 66 53 5f 4d

root@backuppc41:~# btrfs device add /dev/mapper/crypt2 /var/lib/backuppc/
ERROR: error adding device 'dm-4': No such file or directory

This is what I see in dmesg:
[43827.535383] BTRFS info (device dm-5): disk added /dev/drbd2
[43868.910994] BTRFS info (device dm-5): device deleted: /dev/drbd2
[48125.323995] BTRFS: device fsid 2b4b631c-b500-4f8d-909c-e88b012eba1e 
devid 1 transid 5 /dev/mapper/crypt2 scanned by mkfs.btrfs (4937)
[57799.499249] BTRFS: device fsid b111a08e-2969-457a-b9f1-551ff65451d1 
devid 1 transid 5 /dev/mapper/crypt2 scanned by mkfs.btrfs (5178)



And these are the mapping in dm:

root@backuppc41:~# ll /dev/mapper/
insgesamt 0
lrwxrwxrwx 1 root root       7 28. Feb 21:08 backuppc41--vg-root -> ../dm-1
lrwxrwxrwx 1 root root       7 28. Feb 21:08 backuppc41--vg-swap_1 -> 
../dm-2
crw------- 1 root root 10, 236 28. Feb 21:08 control
lrwxrwxrwx 1 root root       7  1. Mär 12:12 crypt2 -> ../dm-4
lrwxrwxrwx 1 root root       7 28. Feb 20:21 crypt_drbd1 -> ../dm-3
lrwxrwxrwx 1 root root       7 28. Feb 20:21 crypt_drbd3 -> ../dm-5
lrwxrwxrwx 1 root root       7 28. Feb 21:08 vda5_crypt -> ../dm-0


Anyone having an idea why I can not add the device to the existing 
filesystem? The error message is not really helpful...

Thanks a lot!

/KNEBB

Re: Adding Device Fails - Why?

[Hugo Mills]

On Mon, Mar 01, 2021 at 12:19:12PM +0100, Christian Völker wrote:
> I am using BTRS on a Debian10 system. I am trying to extend my existing
> filesystem with another device but adding it fails for no reason.
> 
> This is my setup of existing btrfs:
> 
>  2x DRBD Devices (Network RAID1)
>  top of each a luks encrypted device (crypt_drbd1 and crypt_drbd3):
> 
> vdb                         254:16   0  1,1T  0 disk
> └─drbd1                     147:1    0  1,1T  0 disk
>   └─crypt_drbd1             253:3    0  1,1T  0 crypt
> vdc                         254:32   0  900G  0 disk
> └─drbd2                     147:2    0  900G  0 disk
>   └─crypt2                  253:4    0  900G  0 crypt
> vdd                         254:48   0  800G  0 disk
> └─drbd3                     147:3    0  800G  0 disk
>   └─crypt_drbd3             253:5    0  800G  0 crypt /var/lib/backuppc
> 
> 
> 
> I have now a third drbd device (drbd2) which I encrypted, too (crypt2). And
> tried to add to existing fi.
> Here further system information:
> 
> Linux backuppc41 5.10.0-3-amd64 #1 SMP Debian 5.10.13-1 (2021-02-06) x86_64
> GNU/Linux
> btrfs-progs v5.10.1
> 
> root@backuppc41:~# btrfs fi sh
> Label: 'backcuppc'  uuid: 73b98c7b-832a-437a-a15b-6cb00734e5db
>         Total devices 2 FS bytes used 1.83TiB
>         devid    3 size 799.96GiB used 789.96GiB path dm-5
>         devid    4 size 1.07TiB used 1.06TiB path dm-3
> 
> 
> I can create an additional btrfs filesystem with mkfs.btrfs on the new
> device without any issues:
> 
> root@backuppc41:~# btrfs fi sh
> Label: 'backcuppc'  uuid: 73b98c7b-832a-437a-a15b-6cb00734e5db
>         Total devices 2 FS bytes used 1.83TiB
>         devid    3 size 799.96GiB used 789.96GiB path dm-5
>         devid    4 size 1.07TiB used 1.06TiB path dm-3
> 
> Label: none  uuid: b111a08e-2969-457a-b9f1-551ff65451d1
>         Total devices 1 FS bytes used 128.00KiB
>         devid    1 size 899.96GiB used 2.02GiB path /dev/mapper/crypt2
> 
> But I can not add this device to the existing btrfs fi:
> root@backuppc41:~# wipefs /dev/mapper/crypt2 -a
> /dev/mapper/crypt2: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42
> 48 52 66 53 5f 4d
> 
> root@backuppc41:~# btrfs device add /dev/mapper/crypt2 /var/lib/backuppc/
> ERROR: error adding device 'dm-4': No such file or directory
> 
> This is what I see in dmesg:
> [43827.535383] BTRFS info (device dm-5): disk added /dev/drbd2
> [43868.910994] BTRFS info (device dm-5): device deleted: /dev/drbd2
> [48125.323995] BTRFS: device fsid 2b4b631c-b500-4f8d-909c-e88b012eba1e devid
> 1 transid 5 /dev/mapper/crypt2 scanned by mkfs.btrfs (4937)
> [57799.499249] BTRFS: device fsid b111a08e-2969-457a-b9f1-551ff65451d1 devid
> 1 transid 5 /dev/mapper/crypt2 scanned by mkfs.btrfs (5178)

   We had someone on IRC a couple of days ago with exactly the same
kind of problem. I don't think I have a record of the solution in my
IRC logs, though, and I don't think we got to the bottom of it. From
memory, a reboot helped.

   Hugo.

> And these are the mapping in dm:
> 
> root@backuppc41:~# ll /dev/mapper/
> insgesamt 0
> lrwxrwxrwx 1 root root       7 28. Feb 21:08 backuppc41--vg-root -> ../dm-1
> lrwxrwxrwx 1 root root       7 28. Feb 21:08 backuppc41--vg-swap_1 ->
> ../dm-2
> crw------- 1 root root 10, 236 28. Feb 21:08 control
> lrwxrwxrwx 1 root root       7  1. Mär 12:12 crypt2 -> ../dm-4
> lrwxrwxrwx 1 root root       7 28. Feb 20:21 crypt_drbd1 -> ../dm-3
> lrwxrwxrwx 1 root root       7 28. Feb 20:21 crypt_drbd3 -> ../dm-5
> lrwxrwxrwx 1 root root       7 28. Feb 21:08 vda5_crypt -> ../dm-0
> 
> 
> Anyone having an idea why I can not add the device to the existing
> filesystem? The error message is not really helpful...
> 
> Thanks a lot!
> 
> /KNEBB

-- 
Hugo Mills             | Releasing out of hours. A Haiku:
hugo@... carfax.org.uk | Simply merge PR
http://carfax.org.uk/  | It is wrong. Buildkite, cancel!
PGP: E2AB1DE4          | gitops now corrupt

Re: Adding Device Fails - Why?

[Qu Wenruo]

[-- Attachment #1: Type: text/plain, Size: 4259 bytes --]



On 2021/3/1 下午7:19, Christian Völker wrote:
> Hi all,
>
> I am using BTRS on a Debian10 system. I am trying to extend my existing
> filesystem with another device but adding it fails for no reason.
>
> This is my setup of existing btrfs:
>
>   2x DRBD Devices (Network RAID1)
>   top of each a luks encrypted device (crypt_drbd1 and crypt_drbd3):
>
> vdb                         254:16   0  1,1T  0 disk
> └─drbd1                     147:1    0  1,1T  0 disk
>    └─crypt_drbd1             253:3    0  1,1T  0 crypt
> vdc                         254:32   0  900G  0 disk
> └─drbd2                     147:2    0  900G  0 disk
>    └─crypt2                  253:4    0  900G  0 crypt
> vdd                         254:48   0  800G  0 disk
> └─drbd3                     147:3    0  800G  0 disk
>    └─crypt_drbd3             253:5    0  800G  0 crypt /var/lib/backuppc
>
>
>
> I have now a third drbd device (drbd2) which I encrypted, too (crypt2).
> And tried to add to existing fi.
> Here further system information:
>
> Linux backuppc41 5.10.0-3-amd64 #1 SMP Debian 5.10.13-1 (2021-02-06)
> x86_64 GNU/Linux
> btrfs-progs v5.10.1
>
> root@backuppc41:~# btrfs fi sh
> Label: 'backcuppc'  uuid: 73b98c7b-832a-437a-a15b-6cb00734e5db
>          Total devices 2 FS bytes used 1.83TiB
>          devid    3 size 799.96GiB used 789.96GiB path dm-5
>          devid    4 size 1.07TiB used 1.06TiB path dm-3
>
>
> I can create an additional btrfs filesystem with mkfs.btrfs on the new
> device without any issues:
>
> root@backuppc41:~# btrfs fi sh
> Label: 'backcuppc'  uuid: 73b98c7b-832a-437a-a15b-6cb00734e5db
>          Total devices 2 FS bytes used 1.83TiB
>          devid    3 size 799.96GiB used 789.96GiB path dm-5
>          devid    4 size 1.07TiB used 1.06TiB path dm-3
>
> Label: none  uuid: b111a08e-2969-457a-b9f1-551ff65451d1
>          Total devices 1 FS bytes used 128.00KiB
>          devid    1 size 899.96GiB used 2.02GiB path /dev/mapper/crypt2
>
>
> But I can not add this device to the existing btrfs fi:
> root@backuppc41:~# wipefs /dev/mapper/crypt2 -a
> /dev/mapper/crypt2: 8 bytes were erased at offset 0x00010040 (btrfs): 5f
> 42 48 52 66 53 5f 4d
>
> root@backuppc41:~# btrfs device add /dev/mapper/crypt2 /var/lib/backuppc/
> ERROR: error adding device 'dm-4': No such file or directory
>
> This is what I see in dmesg:
> [43827.535383] BTRFS info (device dm-5): disk added /dev/drbd2
> [43868.910994] BTRFS info (device dm-5): device deleted: /dev/drbd2
> [48125.323995] BTRFS: device fsid 2b4b631c-b500-4f8d-909c-e88b012eba1e
> devid 1 transid 5 /dev/mapper/crypt2 scanned by mkfs.btrfs (4937)
> [57799.499249] BTRFS: device fsid b111a08e-2969-457a-b9f1-551ff65451d1
> devid 1 transid 5 /dev/mapper/crypt2 scanned by mkfs.btrfs (5178)

If you can recompile btrfs module, would you mind to test the following
diff to help us debugging the problem?

Of course, after applying the diff and recompiling the module, you still
need to use the new btrfs module to try add the device again to trigger
the error.

Thanks,
Qu
>
>
>
> And these are the mapping in dm:
>
> root@backuppc41:~# ll /dev/mapper/
> insgesamt 0
> lrwxrwxrwx 1 root root       7 28. Feb 21:08 backuppc41--vg-root -> ../dm-1
> lrwxrwxrwx 1 root root       7 28. Feb 21:08 backuppc41--vg-swap_1 ->
> ../dm-2
> crw------- 1 root root 10, 236 28. Feb 21:08 control
> lrwxrwxrwx 1 root root       7  1. Mär 12:12 crypt2 -> ../dm-4
> lrwxrwxrwx 1 root root       7 28. Feb 20:21 crypt_drbd1 -> ../dm-3
> lrwxrwxrwx 1 root root       7 28. Feb 20:21 crypt_drbd3 -> ../dm-5
> lrwxrwxrwx 1 root root       7 28. Feb 21:08 vda5_crypt -> ../dm-0
>
>
> Anyone having an idea why I can not add the device to the existing
> filesystem? The error message is not really helpful...
>
> Thanks a lot!
>
> /KNEBB
>
>
>

[-- Attachment #2: diff --]
[-- Type: text/plain, Size: 2078 bytes --]

diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index b8fab44394f5..9e759a0f9c10 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -2586,8 +2586,11 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 
 	bdev = blkdev_get_by_path(device_path, FMODE_WRITE | FMODE_EXCL,
 				  fs_info->bdev_holder);
-	if (IS_ERR(bdev))
+	if (IS_ERR(bdev)) {
+		pr_info("%s: failed to get blkdev: %ld\n", __func__,
+			PTR_ERR(bdev));
 		return PTR_ERR(bdev);
+	}
 
 	if (!btrfs_check_device_zone_type(fs_info, bdev)) {
 		ret = -EINVAL;
@@ -2617,6 +2620,7 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 	if (IS_ERR(device)) {
 		/* we can safely leave the fs_devices entry around */
 		ret = PTR_ERR(device);
+		pr_info("%s: failed to alloc dev: %d\n", __func__, ret);
 		goto error;
 	}
 
@@ -2637,6 +2641,7 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 	trans = btrfs_start_transaction(root, 0);
 	if (IS_ERR(trans)) {
 		ret = PTR_ERR(trans);
+		pr_info("%s: failed to start transaction: %d\n", __func__, ret);
 		goto error_free_zone;
 	}
 
@@ -2741,8 +2746,10 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 		up_write(&sb->s_umount);
 		locked = false;
 
-		if (ret) /* transaction commit */
+		if (ret) /* transaction commit */ {
+			pr_info("%s: failed to commit trans: %d\n", __func__, ret);
 			return ret;
+		}
 
 		ret = btrfs_relocate_sys_chunks(fs_info);
 		if (ret < 0)
@@ -2753,6 +2760,7 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 			if (PTR_ERR(trans) == -ENOENT)
 				return 0;
 			ret = PTR_ERR(trans);
+			pr_info("%s: failed to attach trans: %d\n", __func__, ret);
 			trans = NULL;
 			goto error_sysfs;
 		}
@@ -2771,6 +2779,7 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 	/* Update ctime/mtime for blkid or udev */
 	update_dev_time(device_path);
 
+	pr_info("%s: final ret: %d\n", __func__, ret);
 	return ret;
 
 error_sysfs:

Re: Adding Device Fails - Why?

[Christian Völker]

Hi,

unfortunately I do not have any sources or compilers installed on this 
box. So I am unable to install this here. I can not even take a snapshot 
to install and compile because it is a two-node setup with drbd so 
resetting to a snapshot will fail for the drbd sync.

A reboot did not help.

I am happy in helping to debug this issue. But compiling on the 
"production" machine is not really fine...

I will setup a different machine to see if I can reproduce it there (I 
doubt it, though)....

/KNEBB

Am 01.03.2021 um 13:06 schrieb Qu Wenruo:
>
>
> On 2021/3/1 下午7:19, Christian Völker wrote:
>> Hi all,
>>
>> I am using BTRS on a Debian10 system. I am trying to extend my existing
>> filesystem with another device but adding it fails for no reason.
>>
>> This is my setup of existing btrfs:
>>
>>   2x DRBD Devices (Network RAID1)
>>   top of each a luks encrypted device (crypt_drbd1 and crypt_drbd3):
>>
>> vdb                         254:16   0  1,1T  0 disk
>> └─drbd1                     147:1    0  1,1T  0 disk
>>    └─crypt_drbd1             253:3    0  1,1T  0 crypt
>> vdc                         254:32   0  900G  0 disk
>> └─drbd2                     147:2    0  900G  0 disk
>>    └─crypt2                  253:4    0  900G  0 crypt
>> vdd                         254:48   0  800G  0 disk
>> └─drbd3                     147:3    0  800G  0 disk
>>    └─crypt_drbd3             253:5    0  800G  0 crypt /var/lib/backuppc
>>
>>
>>
>> I have now a third drbd device (drbd2) which I encrypted, too (crypt2).
>> And tried to add to existing fi.
>> Here further system information:
>>
>> Linux backuppc41 5.10.0-3-amd64 #1 SMP Debian 5.10.13-1 (2021-02-06)
>> x86_64 GNU/Linux
>> btrfs-progs v5.10.1
>>
>> root@backuppc41:~# btrfs fi sh
>> Label: 'backcuppc'  uuid: 73b98c7b-832a-437a-a15b-6cb00734e5db
>>          Total devices 2 FS bytes used 1.83TiB
>>          devid    3 size 799.96GiB used 789.96GiB path dm-5
>>          devid    4 size 1.07TiB used 1.06TiB path dm-3
>>
>>
>> I can create an additional btrfs filesystem with mkfs.btrfs on the new
>> device without any issues:
>>
>> root@backuppc41:~# btrfs fi sh
>> Label: 'backcuppc'  uuid: 73b98c7b-832a-437a-a15b-6cb00734e5db
>>          Total devices 2 FS bytes used 1.83TiB
>>          devid    3 size 799.96GiB used 789.96GiB path dm-5
>>          devid    4 size 1.07TiB used 1.06TiB path dm-3
>>
>> Label: none  uuid: b111a08e-2969-457a-b9f1-551ff65451d1
>>          Total devices 1 FS bytes used 128.00KiB
>>          devid    1 size 899.96GiB used 2.02GiB path /dev/mapper/crypt2
>>
>>
>> But I can not add this device to the existing btrfs fi:
>> root@backuppc41:~# wipefs /dev/mapper/crypt2 -a
>> /dev/mapper/crypt2: 8 bytes were erased at offset 0x00010040 (btrfs): 5f
>> 42 48 52 66 53 5f 4d
>>
>> root@backuppc41:~# btrfs device add /dev/mapper/crypt2 
>> /var/lib/backuppc/
>> ERROR: error adding device 'dm-4': No such file or directory
>>
>> This is what I see in dmesg:
>> [43827.535383] BTRFS info (device dm-5): disk added /dev/drbd2
>> [43868.910994] BTRFS info (device dm-5): device deleted: /dev/drbd2
>> [48125.323995] BTRFS: device fsid 2b4b631c-b500-4f8d-909c-e88b012eba1e
>> devid 1 transid 5 /dev/mapper/crypt2 scanned by mkfs.btrfs (4937)
>> [57799.499249] BTRFS: device fsid b111a08e-2969-457a-b9f1-551ff65451d1
>> devid 1 transid 5 /dev/mapper/crypt2 scanned by mkfs.btrfs (5178)
>
> If you can recompile btrfs module, would you mind to test the following
> diff to help us debugging the problem?
>
> Of course, after applying the diff and recompiling the module, you still
> need to use the new btrfs module to try add the device again to trigger
> the error.
>
> Thanks,
> Qu
>>
>>
>>
>> And these are the mapping in dm:
>>
>> root@backuppc41:~# ll /dev/mapper/
>> insgesamt 0
>> lrwxrwxrwx 1 root root       7 28. Feb 21:08 backuppc41--vg-root -> 
>> ../dm-1
>> lrwxrwxrwx 1 root root       7 28. Feb 21:08 backuppc41--vg-swap_1 ->
>> ../dm-2
>> crw------- 1 root root 10, 236 28. Feb 21:08 control
>> lrwxrwxrwx 1 root root       7  1. Mär 12:12 crypt2 -> ../dm-4
>> lrwxrwxrwx 1 root root       7 28. Feb 20:21 crypt_drbd1 -> ../dm-3
>> lrwxrwxrwx 1 root root       7 28. Feb 20:21 crypt_drbd3 -> ../dm-5
>> lrwxrwxrwx 1 root root       7 28. Feb 21:08 vda5_crypt -> ../dm-0
>>
>>
>> Anyone having an idea why I can not add the device to the existing
>> filesystem? The error message is not really helpful...
>>
>> Thanks a lot!
>>
>> /KNEBB
>>
>>
>>

Re: Adding Device Fails - Why?

[Christian Völker]

Hi,

just a little update on the issue.

As soon as I omit the encryption part I can easily add the device to the 
btrfs filesystem. It does not matter if the crypted device is on top of 
DRBD or directly on the /dev/sdc. In both cases btrs refuses to add the 
device when a luks-encrypted device is on top.

In case I am swapping my setup (drbd on top of encryption) and add the 
drbd device to btrfs it works without any issues.

However, I prefer the other way round- and as the other two btrfs 
devices are both encryption on top of drbd it should work...

It appears it does not like to add a third device-mapper device...

Let me know how I can help in debugging. If i have some time I will 
setup a machine trying to reproduce this.

any ideas otherwise? Let me know!

Thanks!

/KNEBB

Re: Adding Device Fails - Why?

[Qu Wenruo]

On 2021/3/2 上午1:24, Christian Völker wrote:
> Hi,
>
> just a little update on the issue.
>
> As soon as I omit the encryption part I can easily add the device to the
> btrfs filesystem. It does not matter if the crypted device is on top of
> DRBD or directly on the /dev/sdc. In both cases btrs refuses to add the
> device when a luks-encrypted device is on top.
>
> In case I am swapping my setup (drbd on top of encryption) and add the
> drbd device to btrfs it works without any issues.
>
> However, I prefer the other way round- and as the other two btrfs
> devices are both encryption on top of drbd it should work...
>
> It appears it does not like to add a third device-mapper device...
>
> Let me know how I can help in debugging. If i have some time I will
> setup a machine trying to reproduce this.

Got the problem reproduced here.

And surprisingly, it's something related to btrfs-progs, not the kernel.

I just added one debug info in btrfs-progs, it shows:

$ sudo ./btrfs dev add /dev/test/scratch2  /mnt/btrfs
cmd_device_add: path=dm-5
ERROR: error adding device 'dm-5': No such file or directory

See the problem?

The path which should be passed to kernel lacks the "/dev/test/" prefix,
thus it's not pointing to correct path and cause the ENOENT error, since
there is no "dm-5" in current path.

Thankfully it's already fixed in devel branch with commit 2347b34af4d8
("btrfs-progs: fix device mapper path canonicalization").

The offending patch is 922eaa7b5472 ("btrfs-progs: build: fix linking
with static libmount"), which is in v5.10.1.

You can revert back to v5.10 to workaroud it.


TO David,

Would you consider to add a new v5.10.2 to fix the problem? As it seems
to affect the end user quite badly.

Thanks,
Qu
>
> any ideas otherwise? Let me know!
>
> Thanks!
>
> /KNEBB
>

Re: Adding Device Fails - Why?

[Christian Völker]

Hi Qu Wenro,

thanks for debugging this. What I am wondering- why did it work when 
creating the initial device?
For Debian it looks like there is no other version of btrfs-progs 
available, so I used this version (5.10.1-1) to create the btrfs device.

And may I workaround this issue by manually creating a softlink 
/dev/manualdevice to /dev/mappper/crypt_drbd3? The link will be gone 
after reboot but btrfs should then find the device by it's UUID, 
shouldn't it?

Greetings

/KNEBB


Am 02.03.2021 um 02:18 schrieb Qu Wenruo:
>
>
> On 2021/3/2 上午1:24, Christian Völker wrote:
>> Hi,
>>
>> just a little update on the issue.
>>
>> As soon as I omit the encryption part I can easily add the device to the
>> btrfs filesystem. It does not matter if the crypted device is on top of
>> DRBD or directly on the /dev/sdc. In both cases btrs refuses to add the
>> device when a luks-encrypted device is on top.
>>
>> In case I am swapping my setup (drbd on top of encryption) and add the
>> drbd device to btrfs it works without any issues.
>>
>> However, I prefer the other way round- and as the other two btrfs
>> devices are both encryption on top of drbd it should work...
>>
>> It appears it does not like to add a third device-mapper device...
>>
>> Let me know how I can help in debugging. If i have some time I will
>> setup a machine trying to reproduce this.
>
> Got the problem reproduced here.
>
> And surprisingly, it's something related to btrfs-progs, not the kernel.
>
> I just added one debug info in btrfs-progs, it shows:
>
> $ sudo ./btrfs dev add /dev/test/scratch2  /mnt/btrfs
> cmd_device_add: path=dm-5
> ERROR: error adding device 'dm-5': No such file or directory
>
> See the problem?
>
> The path which should be passed to kernel lacks the "/dev/test/" prefix,
> thus it's not pointing to correct path and cause the ENOENT error, since
> there is no "dm-5" in current path.
>
> Thankfully it's already fixed in devel branch with commit 2347b34af4d8
> ("btrfs-progs: fix device mapper path canonicalization").
>
> The offending patch is 922eaa7b5472 ("btrfs-progs: build: fix linking
> with static libmount"), which is in v5.10.1.
>
> You can revert back to v5.10 to workaroud it.
>
>
> TO David,
>
> Would you consider to add a new v5.10.2 to fix the problem? As it seems
> to affect the end user quite badly.
>
> Thanks,
> Qu
>>
>> any ideas otherwise? Let me know!
>>
>> Thanks!
>>
>> /KNEBB
>>

Re: Adding Device Fails - Why?

[Christian Völker]

Ok,

tried to do with a symlink. No working, getting segfault:

 >root@backuppc41:/dev# ln -s  mapper/crypt_test  crypt_temp
 >root@backuppc41:/dev# btrfs de ad /dev/crypt_temp /var/lib/backuppc/
 >Speicherzugriffsfehler

Greetings

/KNEBB


Am 02.03.2021 um 02:18 schrieb Qu Wenruo:
>
>
> On 2021/3/2 上午1:24, Christian Völker wrote:
>> Hi,
>>
>> just a little update on the issue.
>>
>> As soon as I omit the encryption part I can easily add the device to the
>> btrfs filesystem. It does not matter if the crypted device is on top of
>> DRBD or directly on the /dev/sdc. In both cases btrs refuses to add the
>> device when a luks-encrypted device is on top.
>>
>> In case I am swapping my setup (drbd on top of encryption) and add the
>> drbd device to btrfs it works without any issues.
>>
>> However, I prefer the other way round- and as the other two btrfs
>> devices are both encryption on top of drbd it should work...
>>
>> It appears it does not like to add a third device-mapper device...
>>
>> Let me know how I can help in debugging. If i have some time I will
>> setup a machine trying to reproduce this.
>
> Got the problem reproduced here.
>
> And surprisingly, it's something related to btrfs-progs, not the kernel.
>
> I just added one debug info in btrfs-progs, it shows:
>
> $ sudo ./btrfs dev add /dev/test/scratch2  /mnt/btrfs
> cmd_device_add: path=dm-5
> ERROR: error adding device 'dm-5': No such file or directory
>
> See the problem?
>
> The path which should be passed to kernel lacks the "/dev/test/" prefix,
> thus it's not pointing to correct path and cause the ENOENT error, since
> there is no "dm-5" in current path.
>
> Thankfully it's already fixed in devel branch with commit 2347b34af4d8
> ("btrfs-progs: fix device mapper path canonicalization").
>
> The offending patch is 922eaa7b5472 ("btrfs-progs: build: fix linking
> with static libmount"), which is in v5.10.1.
>
> You can revert back to v5.10 to workaroud it.
>
>
> TO David,
>
> Would you consider to add a new v5.10.2 to fix the problem? As it seems
> to affect the end user quite badly.
>
> Thanks,
> Qu
>>
>> any ideas otherwise? Let me know!
>>
>> Thanks!
>>
>> /KNEBB
>>

Re: Adding Device Fails - Why?

[Qu Wenruo]

On 2021/3/2 下午3:09, Christian Völker wrote:
> Hi Qu Wenro,
>
> thanks for debugging this. What I am wondering- why did it work when
> creating the initial device?

Fs creation (mkfs) doesn't canonicalize the path, and it all happens in
user space, so no problem.

Only device add is affected, and if the device to be added is not a
device mapped dev (aka /dev/dm-*) then even with the offending v5.10.1
you can still add the device, just like what you tried by using drbd
over luks.

> For Debian it looks like there is no other version of btrfs-progs
> available, so I used this version (5.10.1-1) to create the btrfs device.
>
> And may I workaround this issue by manually creating a softlink
> /dev/manualdevice to /dev/mappper/crypt_drbd3? The link will be gone
> after reboot but btrfs should then find the device by it's UUID,
> shouldn't it?

I just tried, that, it doesn't work.

The path canonicalization happens by following the softlink until it
reaches the final /dev/dm-* block file, and then cut the "/dev/" prefix
wrongly, leaving just "dm-*".

So even you added a softlink, it won't help.
I just tried that, and it failed:
  $ sudo ln -sf /dev/test/scratch2  /dev/whatever
  $ sudo btrfs dev add /dev/whatever /mnt/btrfs/ -f
  ERROR: error adding device 'dm-5': No such file or directory


But, if you manually create a block device file, with the same device
major/minor number, then you can add it using the newly created block
device file, just like this:

  $ ls -alh /dev/dm-5
  brw-rw---- 1 root disk 253, 5 Mar  2 15:24 /dev/dm-5
  $ sudo mknod ~/whatever b 253 5
  $ sudo btrfs dev add ~/whatever /mnt/btrfs/ -f
  $ sudo btrfs fi show /mnt/btrfs/
  Label: none  uuid: 82631967-0b16-40f2-b545-0039fe9b9653
         Total devices 2 FS bytes used 192.00KiB
         devid    1 size 10.00GiB used 536.00MiB path dm-4
         devid    2 size 10.00GiB used 0.00B path /home/adam/whatever

Thanks,
Qu

>
> Greetings
>
> /KNEBB
>
>
>
>
> Am 02.03.2021 um 02:18 schrieb Qu Wenruo:
>>
>>
>> On 2021/3/2 上午1:24, Christian Völker wrote:
>>> Hi,
>>>
>>> just a little update on the issue.
>>>
>>> As soon as I omit the encryption part I can easily add the device to the
>>> btrfs filesystem. It does not matter if the crypted device is on top of
>>> DRBD or directly on the /dev/sdc. In both cases btrs refuses to add the
>>> device when a luks-encrypted device is on top.
>>>
>>> In case I am swapping my setup (drbd on top of encryption) and add the
>>> drbd device to btrfs it works without any issues.
>>>
>>> However, I prefer the other way round- and as the other two btrfs
>>> devices are both encryption on top of drbd it should work...
>>>
>>> It appears it does not like to add a third device-mapper device...
>>>
>>> Let me know how I can help in debugging. If i have some time I will
>>> setup a machine trying to reproduce this.
>>
>> Got the problem reproduced here.
>>
>> And surprisingly, it's something related to btrfs-progs, not the kernel.
>>
>> I just added one debug info in btrfs-progs, it shows:
>>
>> $ sudo ./btrfs dev add /dev/test/scratch2  /mnt/btrfs
>> cmd_device_add: path=dm-5
>> ERROR: error adding device 'dm-5': No such file or directory
>>
>> See the problem?
>>
>> The path which should be passed to kernel lacks the "/dev/test/" prefix,
>> thus it's not pointing to correct path and cause the ENOENT error, since
>> there is no "dm-5" in current path.
>>
>> Thankfully it's already fixed in devel branch with commit 2347b34af4d8
>> ("btrfs-progs: fix device mapper path canonicalization").
>>
>> The offending patch is 922eaa7b5472 ("btrfs-progs: build: fix linking
>> with static libmount"), which is in v5.10.1.
>>
>> You can revert back to v5.10 to workaroud it.
>>
>>
>> TO David,
>>
>> Would you consider to add a new v5.10.2 to fix the problem? As it seems
>> to affect the end user quite badly.
>>
>> Thanks,
>> Qu
>>>
>>> any ideas otherwise? Let me know!
>>>
>>> Thanks!
>>>
>>> /KNEBB
>>>
>

Re: Adding Device Fails - Why?

[Christian Völker]

Hi,

thanks to Qu Wenruo it worked for me as he suggested.
So I created my additional drbd device and put the encryption on top of 
it. To workaround the issue with adding a device with /dev/mapper/ as 
prefix I created in my /root a node with the same major and minor 
numbers as the dm-* device. Once the device got added I removed the 
created node and everything went fine. Trying to use a created symlink 
did not work, neither adding the dm-* device directly.

Steps in detail:
 >root@backuppc41:/dev/mapper# ll
 >insgesamt 0
 >crw------- 1 root root 10, 236  2. Mär 13:27 control
 >lrwxrwxrwx 1 root root       7  2. Mär 13:48 crypt_drbd1 -> ../dm-3
 >lrwxrwxrwx 1 root root       7  2. Mär 13:48 crypt_drbd2 -> ../dm-4
 >lrwxrwxrwx 1 root root       7  2. Mär 13:48 crypt_drbd3 -> ../dm-5

The device to be added is crypt_drbd3 so I need to find the major/minor 
number of the dm-5 device:

 >root@backuppc41:/dev# ll /dev| grep dm-5
 >brw-rw---- 1 root disk    253,   5  2. Mär 13:48 dm-5

Ok, got all I need. Now going to create the (temp) device node:
 >root@backuppc41:~# mknod ~/temp_node b 253 2

Now I was finally able to add the device by the created node:
 >root@backuppc41:~# btrfs de add /root/temp_node /mnt

Now remove the temp_node and I rebooted the box. After reboot the device 
was still added to my btrfs-filesystem. Perfect!

Thanks again to all!

/KNEBB