Linux-BTRFS Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH] btrfs_progs: mkfs: match devid order to the stripe index
@ 2019-06-28  2:26 Anand Jain
  2019-06-28  2:44 ` Qu Wenruo
                   ` (3 more replies)
  0 siblings, 4 replies; 19+ messages in thread
From: Anand Jain @ 2019-06-28  2:26 UTC (permalink / raw)
  To: linux-btrfs

At the time mkfs.btrfs the device id and stripe index gets reversed as
shown in [1]. This patch helps to keep them in order at the time of
mkfs.btrfs. And makes it easier to debug.

Before:
Stripe 0 is on devid 2; Stipe 1 is on devid 1;

./mkfs.btrfs -fq -draid1 -mraid1 /dev/sdb /dev/sdc && btrfs in dump-tree -d /dev/sdb | grep -A 10000 "chunk tree" | grep -B 10000 "device tree" | grep -A 13  "FIRST_CHUNK_TREE CHUNK_ITEM"
	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975 itemsize 112
		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 1048576
			dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
			stripe 1 devid 1 offset 22020096
			dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863 itemsize 112
		length 268435456 owner 2 stripe_len 65536 type METADATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 9437184
			dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
			stripe 1 devid 1 offset 30408704
			dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 298844160) itemoff 15751 itemsize 112
		length 314572800 owner 2 stripe_len 65536 type DATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 2 offset 277872640
			dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
			stripe 1 devid 1 offset 298844160
			dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab

After:
Stripe 0 is on devid 1; Stripe 1 is on devid 2

./mkfs.btrfs -fq -draid1 -mraid1 /dev/sdb /dev/sdc && btrfs in dump-tree -d /dev/sdb | grep -A 10000 "chunk tree" | grep -B 10000 "device tree" | grep -A 13  "FIRST_CHUNK_TREE CHUNK_ITEM"
/dev/sdb: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
/dev/sdc: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975 itemsize 112
		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 1 offset 22020096
			dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
			stripe 1 devid 2 offset 1048576
			dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863 itemsize 112
		length 268435456 owner 2 stripe_len 65536 type METADATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 1 offset 30408704
			dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
			stripe 1 devid 2 offset 9437184
			dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 298844160) itemoff 15751 itemsize 112
		length 314572800 owner 2 stripe_len 65536 type DATA|RAID1
		io_align 65536 io_width 65536 sector_size 4096
		num_stripes 2 sub_stripes 0
			stripe 0 devid 1 offset 298844160
			dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
			stripe 1 devid 2 offset 277872640
			dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d

Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
 volumes.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/volumes.c b/volumes.c
index 79d1d6a07fb7..8c8b17e814b8 100644
--- a/volumes.c
+++ b/volumes.c
@@ -1109,7 +1109,7 @@ again:
 			return ret;
 		cur = cur->next;
 		if (avail >= min_free) {
-			list_move_tail(&device->dev_list, &private_devs);
+			list_move(&device->dev_list, &private_devs);
 			index++;
 			if (type & BTRFS_BLOCK_GROUP_DUP)
 				index++;
@@ -1166,7 +1166,7 @@ again:
 		/* loop over this device again if we're doing a dup group */
 		if (!(type & BTRFS_BLOCK_GROUP_DUP) ||
 		    (index == num_stripes - 1))
-			list_move_tail(&device->dev_list, dev_list);
+			list_move(&device->dev_list, dev_list);
 
 		ret = btrfs_alloc_dev_extent(trans, device, key.offset,
 			     calc_size, &dev_offset);
-- 
2.20.1 (Apple Git-117)


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH] btrfs_progs: mkfs: match devid order to the stripe index
  2019-06-28  2:26 [PATCH] btrfs_progs: mkfs: match devid order to the stripe index Anand Jain
@ 2019-06-28  2:44 ` Qu Wenruo
  2019-06-28  3:28   ` Anand Jain
  2019-07-03 13:21 ` David Sterba
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 19+ messages in thread
From: Qu Wenruo @ 2019-06-28  2:44 UTC (permalink / raw)
  To: Anand Jain, linux-btrfs

[-- Attachment #1.1: Type: text/plain, Size: 4964 bytes --]



On 2019/6/28 上午10:26, Anand Jain wrote:
> At the time mkfs.btrfs the device id and stripe index gets reversed as
> shown in [1]. This patch helps to keep them in order at the time of
> mkfs.btrfs. And makes it easier to debug.
> 
> Before:
> Stripe 0 is on devid 2; Stipe 1 is on devid 1;
> 
> ./mkfs.btrfs -fq -draid1 -mraid1 /dev/sdb /dev/sdc && btrfs in dump-tree -d /dev/sdb | grep -A 10000 "chunk tree" | grep -B 10000 "device tree" | grep -A 13  "FIRST_CHUNK_TREE CHUNK_ITEM"
> 	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975 itemsize 112
> 		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
> 		io_align 65536 io_width 65536 sector_size 4096
> 		num_stripes 2 sub_stripes 0
> 			stripe 0 devid 2 offset 1048576
> 			dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
> 			stripe 1 devid 1 offset 22020096
> 			dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
> 	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863 itemsize 112
> 		length 268435456 owner 2 stripe_len 65536 type METADATA|RAID1
> 		io_align 65536 io_width 65536 sector_size 4096
> 		num_stripes 2 sub_stripes 0
> 			stripe 0 devid 2 offset 9437184
> 			dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
> 			stripe 1 devid 1 offset 30408704
> 			dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
> 	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 298844160) itemoff 15751 itemsize 112
> 		length 314572800 owner 2 stripe_len 65536 type DATA|RAID1
> 		io_align 65536 io_width 65536 sector_size 4096
> 		num_stripes 2 sub_stripes 0
> 			stripe 0 devid 2 offset 277872640
> 			dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
> 			stripe 1 devid 1 offset 298844160
> 			dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
> 
> After:
> Stripe 0 is on devid 1; Stripe 1 is on devid 2
> 
> ./mkfs.btrfs -fq -draid1 -mraid1 /dev/sdb /dev/sdc && btrfs in dump-tree -d /dev/sdb | grep -A 10000 "chunk tree" | grep -B 10000 "device tree" | grep -A 13  "FIRST_CHUNK_TREE CHUNK_ITEM"
> /dev/sdb: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
> /dev/sdc: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
> 	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975 itemsize 112
> 		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
> 		io_align 65536 io_width 65536 sector_size 4096
> 		num_stripes 2 sub_stripes 0
> 			stripe 0 devid 1 offset 22020096
> 			dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
> 			stripe 1 devid 2 offset 1048576
> 			dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
> 	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863 itemsize 112
> 		length 268435456 owner 2 stripe_len 65536 type METADATA|RAID1
> 		io_align 65536 io_width 65536 sector_size 4096
> 		num_stripes 2 sub_stripes 0
> 			stripe 0 devid 1 offset 30408704
> 			dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
> 			stripe 1 devid 2 offset 9437184
> 			dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
> 	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 298844160) itemoff 15751 itemsize 112
> 		length 314572800 owner 2 stripe_len 65536 type DATA|RAID1
> 		io_align 65536 io_width 65536 sector_size 4096
> 		num_stripes 2 sub_stripes 0
> 			stripe 0 devid 1 offset 298844160
> 			dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
> 			stripe 1 devid 2 offset 277872640
> 			dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
> 
> Signed-off-by: Anand Jain <anand.jain@oracle.com>

Reviewed-by: Qu Wenruo <wqu@suse.com>

But please also check the comment inlined below.
> ---
>  volumes.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/volumes.c b/volumes.c
> index 79d1d6a07fb7..8c8b17e814b8 100644
> --- a/volumes.c
> +++ b/volumes.c
> @@ -1109,7 +1109,7 @@ again:
>  			return ret;
>  		cur = cur->next;
>  		if (avail >= min_free) {
> -			list_move_tail(&device->dev_list, &private_devs);
> +			list_move(&device->dev_list, &private_devs);

This is OK since current btrfs-progs chunk allocator doesn't follow the
kernel behavior by sorting devices with its unallocated space.
So it's completely devid based.

But please keep in mind that, if we're going to unify the chunk
allocator behavior of kernel and btrfs-progs, the behavior will change.

As the initial temporary chunk is always allocated on devid 1, reducing
its unallocated space thus reducing its priority in chunk allocator, and
making the devid sequence more unreliable.

Thanks,
Qu

>  			index++;
>  			if (type & BTRFS_BLOCK_GROUP_DUP)
>  				index++;
> @@ -1166,7 +1166,7 @@ again:
>  		/* loop over this device again if we're doing a dup group */
>  		if (!(type & BTRFS_BLOCK_GROUP_DUP) ||
>  		    (index == num_stripes - 1))
> -			list_move_tail(&device->dev_list, dev_list);
> +			list_move(&device->dev_list, dev_list);
>  
>  		ret = btrfs_alloc_dev_extent(trans, device, key.offset,
>  			     calc_size, &dev_offset);
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH] btrfs_progs: mkfs: match devid order to the stripe index
  2019-06-28  2:44 ` Qu Wenruo
@ 2019-06-28  3:28   ` Anand Jain
  2019-06-28  6:01     ` Qu Wenruo
  0 siblings, 1 reply; 19+ messages in thread
From: Anand Jain @ 2019-06-28  3:28 UTC (permalink / raw)
  To: Qu Wenruo, linux-btrfs

On 28/6/19 10:44 AM, Qu Wenruo wrote:
> 
> 
> On 2019/6/28 上午10:26, Anand Jain wrote:
>> At the time mkfs.btrfs the device id and stripe index gets reversed as
>> shown in [1]. This patch helps to keep them in order at the time of
>> mkfs.btrfs. And makes it easier to debug.
>>
>> Before:
>> Stripe 0 is on devid 2; Stipe 1 is on devid 1;
>>
>> ./mkfs.btrfs -fq -draid1 -mraid1 /dev/sdb /dev/sdc && btrfs in dump-tree -d /dev/sdb | grep -A 10000 "chunk tree" | grep -B 10000 "device tree" | grep -A 13  "FIRST_CHUNK_TREE CHUNK_ITEM"
>> 	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975 itemsize 112
>> 		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
>> 		io_align 65536 io_width 65536 sector_size 4096
>> 		num_stripes 2 sub_stripes 0
>> 			stripe 0 devid 2 offset 1048576
>> 			dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
>> 			stripe 1 devid 1 offset 22020096
>> 			dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
>> 	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863 itemsize 112
>> 		length 268435456 owner 2 stripe_len 65536 type METADATA|RAID1
>> 		io_align 65536 io_width 65536 sector_size 4096
>> 		num_stripes 2 sub_stripes 0
>> 			stripe 0 devid 2 offset 9437184
>> 			dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
>> 			stripe 1 devid 1 offset 30408704
>> 			dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
>> 	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 298844160) itemoff 15751 itemsize 112
>> 		length 314572800 owner 2 stripe_len 65536 type DATA|RAID1
>> 		io_align 65536 io_width 65536 sector_size 4096
>> 		num_stripes 2 sub_stripes 0
>> 			stripe 0 devid 2 offset 277872640
>> 			dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
>> 			stripe 1 devid 1 offset 298844160
>> 			dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
>>
>> After:
>> Stripe 0 is on devid 1; Stripe 1 is on devid 2
>>
>> ./mkfs.btrfs -fq -draid1 -mraid1 /dev/sdb /dev/sdc && btrfs in dump-tree -d /dev/sdb | grep -A 10000 "chunk tree" | grep -B 10000 "device tree" | grep -A 13  "FIRST_CHUNK_TREE CHUNK_ITEM"
>> /dev/sdb: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
>> /dev/sdc: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
>> 	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975 itemsize 112
>> 		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
>> 		io_align 65536 io_width 65536 sector_size 4096
>> 		num_stripes 2 sub_stripes 0
>> 			stripe 0 devid 1 offset 22020096
>> 			dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
>> 			stripe 1 devid 2 offset 1048576
>> 			dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
>> 	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863 itemsize 112
>> 		length 268435456 owner 2 stripe_len 65536 type METADATA|RAID1
>> 		io_align 65536 io_width 65536 sector_size 4096
>> 		num_stripes 2 sub_stripes 0
>> 			stripe 0 devid 1 offset 30408704
>> 			dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
>> 			stripe 1 devid 2 offset 9437184
>> 			dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
>> 	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 298844160) itemoff 15751 itemsize 112
>> 		length 314572800 owner 2 stripe_len 65536 type DATA|RAID1
>> 		io_align 65536 io_width 65536 sector_size 4096
>> 		num_stripes 2 sub_stripes 0
>> 			stripe 0 devid 1 offset 298844160
>> 			dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
>> 			stripe 1 devid 2 offset 277872640
>> 			dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
>>
>> Signed-off-by: Anand Jain <anand.jain@oracle.com>
> 
> Reviewed-by: Qu Wenruo <wqu@suse.com>
> 
> But please also check the comment inlined below.
>> ---
>>   volumes.c | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/volumes.c b/volumes.c
>> index 79d1d6a07fb7..8c8b17e814b8 100644
>> --- a/volumes.c
>> +++ b/volumes.c
>> @@ -1109,7 +1109,7 @@ again:
>>   			return ret;
>>   		cur = cur->next;
>>   		if (avail >= min_free) {
>> -			list_move_tail(&device->dev_list, &private_devs);
>> +			list_move(&device->dev_list, &private_devs);
> 
> This is OK since current btrfs-progs chunk allocator doesn't follow the
> kernel behavior by sorting devices with its unallocated space.
> So it's completely devid based.
> 
> But please keep in mind that, if we're going to unify the chunk
> allocator behavior of kernel and btrfs-progs, the behavior will change.
> 
> As the initial temporary chunk is always allocated on devid 1, reducing
> its unallocated space thus reducing its priority in chunk allocator, and
> making the devid sequence more unreliable.

  Right. For the debug here, I have an experimental code which disables
  the unallocated space sort in the kernel. I don't have a strong reason
  to disable the sort in the kernel so didn't send the patch.

Thanks, Anand

> Thanks,
> Qu
> 
>>   			index++;
>>   			if (type & BTRFS_BLOCK_GROUP_DUP)
>>   				index++;
>> @@ -1166,7 +1166,7 @@ again:
>>   		/* loop over this device again if we're doing a dup group */
>>   		if (!(type & BTRFS_BLOCK_GROUP_DUP) ||
>>   		    (index == num_stripes - 1))
>> -			list_move_tail(&device->dev_list, dev_list);
>> +			list_move(&device->dev_list, dev_list);
>>   
>>   		ret = btrfs_alloc_dev_extent(trans, device, key.offset,
>>   			     calc_size, &dev_offset);
>>
> 


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH] btrfs_progs: mkfs: match devid order to the stripe index
  2019-06-28  3:28   ` Anand Jain
@ 2019-06-28  6:01     ` Qu Wenruo
  0 siblings, 0 replies; 19+ messages in thread
From: Qu Wenruo @ 2019-06-28  6:01 UTC (permalink / raw)
  To: Anand Jain, linux-btrfs

[-- Attachment #1.1: Type: text/plain, Size: 7079 bytes --]



On 2019/6/28 上午11:28, Anand Jain wrote:
> On 28/6/19 10:44 AM, Qu Wenruo wrote:
>>
>>
>> On 2019/6/28 上午10:26, Anand Jain wrote:
>>> At the time mkfs.btrfs the device id and stripe index gets reversed as
>>> shown in [1]. This patch helps to keep them in order at the time of
>>> mkfs.btrfs. And makes it easier to debug.
>>>
>>> Before:
>>> Stripe 0 is on devid 2; Stipe 1 is on devid 1;
>>>
>>> ./mkfs.btrfs -fq -draid1 -mraid1 /dev/sdb /dev/sdc && btrfs in
>>> dump-tree -d /dev/sdb | grep -A 10000 "chunk tree" | grep -B 10000
>>> "device tree" | grep -A 13  "FIRST_CHUNK_TREE CHUNK_ITEM"
>>>     item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975
>>> itemsize 112
>>>         length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
>>>         io_align 65536 io_width 65536 sector_size 4096
>>>         num_stripes 2 sub_stripes 0
>>>             stripe 0 devid 2 offset 1048576
>>>             dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
>>>             stripe 1 devid 1 offset 22020096
>>>             dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
>>>     item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863
>>> itemsize 112
>>>         length 268435456 owner 2 stripe_len 65536 type METADATA|RAID1
>>>         io_align 65536 io_width 65536 sector_size 4096
>>>         num_stripes 2 sub_stripes 0
>>>             stripe 0 devid 2 offset 9437184
>>>             dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
>>>             stripe 1 devid 1 offset 30408704
>>>             dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
>>>     item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 298844160) itemoff 15751
>>> itemsize 112
>>>         length 314572800 owner 2 stripe_len 65536 type DATA|RAID1
>>>         io_align 65536 io_width 65536 sector_size 4096
>>>         num_stripes 2 sub_stripes 0
>>>             stripe 0 devid 2 offset 277872640
>>>             dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
>>>             stripe 1 devid 1 offset 298844160
>>>             dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
>>>
>>> After:
>>> Stripe 0 is on devid 1; Stripe 1 is on devid 2
>>>
>>> ./mkfs.btrfs -fq -draid1 -mraid1 /dev/sdb /dev/sdc && btrfs in
>>> dump-tree -d /dev/sdb | grep -A 10000 "chunk tree" | grep -B 10000
>>> "device tree" | grep -A 13  "FIRST_CHUNK_TREE CHUNK_ITEM"
>>> /dev/sdb: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48
>>> 52 66 53 5f 4d
>>> /dev/sdc: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48
>>> 52 66 53 5f 4d
>>>     item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975
>>> itemsize 112
>>>         length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
>>>         io_align 65536 io_width 65536 sector_size 4096
>>>         num_stripes 2 sub_stripes 0
>>>             stripe 0 devid 1 offset 22020096
>>>             dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
>>>             stripe 1 devid 2 offset 1048576
>>>             dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
>>>     item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863
>>> itemsize 112
>>>         length 268435456 owner 2 stripe_len 65536 type METADATA|RAID1
>>>         io_align 65536 io_width 65536 sector_size 4096
>>>         num_stripes 2 sub_stripes 0
>>>             stripe 0 devid 1 offset 30408704
>>>             dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
>>>             stripe 1 devid 2 offset 9437184
>>>             dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
>>>     item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 298844160) itemoff 15751
>>> itemsize 112
>>>         length 314572800 owner 2 stripe_len 65536 type DATA|RAID1
>>>         io_align 65536 io_width 65536 sector_size 4096
>>>         num_stripes 2 sub_stripes 0
>>>             stripe 0 devid 1 offset 298844160
>>>             dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
>>>             stripe 1 devid 2 offset 277872640
>>>             dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
>>>
>>> Signed-off-by: Anand Jain <anand.jain@oracle.com>
>>
>> Reviewed-by: Qu Wenruo <wqu@suse.com>
>>
>> But please also check the comment inlined below.
>>> ---
>>>   volumes.c | 4 ++--
>>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/volumes.c b/volumes.c
>>> index 79d1d6a07fb7..8c8b17e814b8 100644
>>> --- a/volumes.c
>>> +++ b/volumes.c
>>> @@ -1109,7 +1109,7 @@ again:
>>>               return ret;
>>>           cur = cur->next;
>>>           if (avail >= min_free) {
>>> -            list_move_tail(&device->dev_list, &private_devs);
>>> +            list_move(&device->dev_list, &private_devs);
>>
>> This is OK since current btrfs-progs chunk allocator doesn't follow the
>> kernel behavior by sorting devices with its unallocated space.
>> So it's completely devid based.
>>
>> But please keep in mind that, if we're going to unify the chunk
>> allocator behavior of kernel and btrfs-progs, the behavior will change.
>>
>> As the initial temporary chunk is always allocated on devid 1, reducing
>> its unallocated space thus reducing its priority in chunk allocator, and
>> making the devid sequence more unreliable.
> 
>  Right. For the debug here, I have an experimental code which disables
>  the unallocated space sort in the kernel. I don't have a strong reason
>  to disable the sort in the kernel so didn't send the patch.

I'd say that unallocated sort is a hidden way to prevent starvation.

The mostly common case is 3 disk RAID1. (1024M X 3)
With the unallocated space sort, we can take full use of 1.5T.

While without that, we can only use 1T, as all allocation will happen on
the first (or last) 2 devices, not utilize the remaining disk at all.

So that kernel part is very helpful to prevent starvation.

Thanks,
Qu

> 
> Thanks, Anand
> 
>> Thanks,
>> Qu
>>
>>>               index++;
>>>               if (type & BTRFS_BLOCK_GROUP_DUP)
>>>                   index++;
>>> @@ -1166,7 +1166,7 @@ again:
>>>           /* loop over this device again if we're doing a dup group */
>>>           if (!(type & BTRFS_BLOCK_GROUP_DUP) ||
>>>               (index == num_stripes - 1))
>>> -            list_move_tail(&device->dev_list, dev_list);
>>> +            list_move(&device->dev_list, dev_list);
>>>             ret = btrfs_alloc_dev_extent(trans, device, key.offset,
>>>                    calc_size, &dev_offset);
>>>
>>
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH] btrfs_progs: mkfs: match devid order to the stripe index
  2019-06-28  2:26 [PATCH] btrfs_progs: mkfs: match devid order to the stripe index Anand Jain
  2019-06-28  2:44 ` Qu Wenruo
@ 2019-07-03 13:21 ` David Sterba
  2019-08-27  2:02   ` Anand Jain
  2019-09-03 10:46 ` Johannes Thumshirn
  2019-09-10  7:37 ` [PATCH] btrfs_progs: mkfs: match devid order to the stripe index Anand Jain
  3 siblings, 1 reply; 19+ messages in thread
From: David Sterba @ 2019-07-03 13:21 UTC (permalink / raw)
  To: Anand Jain; +Cc: linux-btrfs

On Fri, Jun 28, 2019 at 10:26:11AM +0800, Anand Jain wrote:
> At the time mkfs.btrfs the device id and stripe index gets reversed as
> shown in [1]. This patch helps to keep them in order at the time of
> mkfs.btrfs. And makes it easier to debug.
> 
> Before:
> Stripe 0 is on devid 2; Stipe 1 is on devid 1;
> 
> ./mkfs.btrfs -fq -draid1 -mraid1 /dev/sdb /dev/sdc && btrfs in dump-tree -d /dev/sdb | grep -A 10000 "chunk tree" | grep -B 10000 "device tree" | grep -A 13  "FIRST_CHUNK_TREE CHUNK_ITEM"

I've reformatted that so it's not overly long line. For dumps it's ok
but a command can be split by && or | .

> 	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975 itemsize 112
> 		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
> 		io_align 65536 io_width 65536 sector_size 4096
> 		num_stripes 2 sub_stripes 0
> 			stripe 0 devid 2 offset 1048576
> 			dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
> 			stripe 1 devid 1 offset 22020096
> 			dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
> 	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863 itemsize 112
> 		length 268435456 owner 2 stripe_len 65536 type METADATA|RAID1
> 		io_align 65536 io_width 65536 sector_size 4096
> 		num_stripes 2 sub_stripes 0
> 			stripe 0 devid 2 offset 9437184
> 			dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
> 			stripe 1 devid 1 offset 30408704
> 			dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
> 	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 298844160) itemoff 15751 itemsize 112
> 		length 314572800 owner 2 stripe_len 65536 type DATA|RAID1
> 		io_align 65536 io_width 65536 sector_size 4096
> 		num_stripes 2 sub_stripes 0
> 			stripe 0 devid 2 offset 277872640
> 			dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
> 			stripe 1 devid 1 offset 298844160
> 			dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
> 
> After:
> Stripe 0 is on devid 1; Stripe 1 is on devid 2
> 
> ./mkfs.btrfs -fq -draid1 -mraid1 /dev/sdb /dev/sdc && btrfs in dump-tree -d /dev/sdb | grep -A 10000 "chunk tree" | grep -B 10000 "device tree" | grep -A 13  "FIRST_CHUNK_TREE CHUNK_ITEM"
> /dev/sdb: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
> /dev/sdc: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
> 	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975 itemsize 112
> 		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
> 		io_align 65536 io_width 65536 sector_size 4096
> 		num_stripes 2 sub_stripes 0
> 			stripe 0 devid 1 offset 22020096
> 			dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
> 			stripe 1 devid 2 offset 1048576
> 			dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
> 	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863 itemsize 112
> 		length 268435456 owner 2 stripe_len 65536 type METADATA|RAID1
> 		io_align 65536 io_width 65536 sector_size 4096
> 		num_stripes 2 sub_stripes 0
> 			stripe 0 devid 1 offset 30408704
> 			dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
> 			stripe 1 devid 2 offset 9437184
> 			dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
> 	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 298844160) itemoff 15751 itemsize 112
> 		length 314572800 owner 2 stripe_len 65536 type DATA|RAID1
> 		io_align 65536 io_width 65536 sector_size 4096
> 		num_stripes 2 sub_stripes 0
> 			stripe 0 devid 1 offset 298844160
> 			dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
> 			stripe 1 devid 2 offset 277872640
> 			dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
> 
> Signed-off-by: Anand Jain <anand.jain@oracle.com>

Added to devel, thanks.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH] btrfs_progs: mkfs: match devid order to the stripe index
  2019-07-03 13:21 ` David Sterba
@ 2019-08-27  2:02   ` Anand Jain
  2019-09-02  8:01     ` Anand Jain
  0 siblings, 1 reply; 19+ messages in thread
From: Anand Jain @ 2019-08-27  2:02 UTC (permalink / raw)
  To: dsterba, linux-btrfs

On 3/7/19 9:21 PM, David Sterba wrote:
> On Fri, Jun 28, 2019 at 10:26:11AM +0800, Anand Jain wrote:
>> At the time mkfs.btrfs the device id and stripe index gets reversed as
>> shown in [1]. This patch helps to keep them in order at the time of
>> mkfs.btrfs. And makes it easier to debug.
>>
>> Before:
>> Stripe 0 is on devid 2; Stipe 1 is on devid 1;
>>
>> ./mkfs.btrfs -fq -draid1 -mraid1 /dev/sdb /dev/sdc && btrfs in dump-tree -d /dev/sdb | grep -A 10000 "chunk tree" | grep -B 10000 "device tree" | grep -A 13  "FIRST_CHUNK_TREE CHUNK_ITEM"
> 
> I've reformatted that so it's not overly long line. For dumps it's ok
> but a command can be split by && or | .
> 
>> 	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975 itemsize 112
>> 		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
>> 		io_align 65536 io_width 65536 sector_size 4096
>> 		num_stripes 2 sub_stripes 0
>> 			stripe 0 devid 2 offset 1048576
>> 			dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
>> 			stripe 1 devid 1 offset 22020096
>> 			dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
>> 	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863 itemsize 112
>> 		length 268435456 owner 2 stripe_len 65536 type METADATA|RAID1
>> 		io_align 65536 io_width 65536 sector_size 4096
>> 		num_stripes 2 sub_stripes 0
>> 			stripe 0 devid 2 offset 9437184
>> 			dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
>> 			stripe 1 devid 1 offset 30408704
>> 			dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
>> 	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 298844160) itemoff 15751 itemsize 112
>> 		length 314572800 owner 2 stripe_len 65536 type DATA|RAID1
>> 		io_align 65536 io_width 65536 sector_size 4096
>> 		num_stripes 2 sub_stripes 0
>> 			stripe 0 devid 2 offset 277872640
>> 			dev_uuid d9fe51c4-6e79-446d-87ee-5be3184798cd
>> 			stripe 1 devid 1 offset 298844160
>> 			dev_uuid 16f626ca-1a54-469b-ac7e-25623af884ab
>>
>> After:
>> Stripe 0 is on devid 1; Stripe 1 is on devid 2
>>
>> ./mkfs.btrfs -fq -draid1 -mraid1 /dev/sdb /dev/sdc && btrfs in dump-tree -d /dev/sdb | grep -A 10000 "chunk tree" | grep -B 10000 "device tree" | grep -A 13  "FIRST_CHUNK_TREE CHUNK_ITEM"
>> /dev/sdb: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
>> /dev/sdc: 8 bytes were erased at offset 0x00010040 (btrfs): 5f 42 48 52 66 53 5f 4d
>> 	item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096) itemoff 15975 itemsize 112
>> 		length 8388608 owner 2 stripe_len 65536 type SYSTEM|RAID1
>> 		io_align 65536 io_width 65536 sector_size 4096
>> 		num_stripes 2 sub_stripes 0
>> 			stripe 0 devid 1 offset 22020096
>> 			dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
>> 			stripe 1 devid 2 offset 1048576
>> 			dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
>> 	item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 30408704) itemoff 15863 itemsize 112
>> 		length 268435456 owner 2 stripe_len 65536 type METADATA|RAID1
>> 		io_align 65536 io_width 65536 sector_size 4096
>> 		num_stripes 2 sub_stripes 0
>> 			stripe 0 devid 1 offset 30408704
>> 			dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
>> 			stripe 1 devid 2 offset 9437184
>> 			dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
>> 	item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 298844160) itemoff 15751 itemsize 112
>> 		length 314572800 owner 2 stripe_len 65536 type DATA|RAID1
>> 		io_align 65536 io_width 65536 sector_size 4096
>> 		num_stripes 2 sub_stripes 0
>> 			stripe 0 devid 1 offset 298844160
>> 			dev_uuid 6abc88fa-f42e-4f0c-9bc3-2225735e51d1
>> 			stripe 1 devid 2 offset 277872640
>> 			dev_uuid 73746d27-13a6-4d58-ac6b-48c90c31d94d
>>
>> Signed-off-by: Anand Jain <anand.jain@oracle.com>
> 
> Added to devel, thanks.
> 

  I don't see this patch is integrated. Any idea?

Thanks, Anand

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH] btrfs_progs: mkfs: match devid order to the stripe index
  2019-08-27  2:02   ` Anand Jain
@ 2019-09-02  8:01     ` Anand Jain
  2019-09-02 16:22       ` David Sterba
  0 siblings, 1 reply; 19+ messages in thread
From: Anand Jain @ 2019-09-02  8:01 UTC (permalink / raw)
  To: dsterba; +Cc: linux-btrfs


David,

  I don't see this patch is integrated. Can you please integrated this 
patch thanks.

Thanks, Anand


>>> Signed-off-by: Anand Jain <anand.jain@oracle.com>
>>
>> Added to devel, thanks.
>>
> 
>   I don't see this patch is integrated. Any idea?
> 
> Thanks, Anand


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH] btrfs_progs: mkfs: match devid order to the stripe index
  2019-09-02  8:01     ` Anand Jain
@ 2019-09-02 16:22       ` David Sterba
  2019-09-03 12:06         ` David Sterba
  0 siblings, 1 reply; 19+ messages in thread
From: David Sterba @ 2019-09-02 16:22 UTC (permalink / raw)
  To: Anand Jain; +Cc: dsterba, linux-btrfs

On Mon, Sep 02, 2019 at 04:01:56PM +0800, Anand Jain wrote:
> 
> David,
> 
>   I don't see this patch is integrated. Can you please integrated this 
> patch thanks.

I don't know why but the patch got lost somewhere, adding to devel
again.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH] btrfs_progs: mkfs: match devid order to the stripe index
  2019-06-28  2:26 [PATCH] btrfs_progs: mkfs: match devid order to the stripe index Anand Jain
  2019-06-28  2:44 ` Qu Wenruo
  2019-07-03 13:21 ` David Sterba
@ 2019-09-03 10:46 ` Johannes Thumshirn
  2019-09-04 12:54   ` [PATCH] btrfs: misc-tests-021 fix restore overlapped on disk's stale data Anand Jain
  2019-09-04 13:29   ` [PATCH Fix-title-prefix] btrfs-progs: " Anand Jain
  2019-09-10  7:37 ` [PATCH] btrfs_progs: mkfs: match devid order to the stripe index Anand Jain
  3 siblings, 2 replies; 19+ messages in thread
From: Johannes Thumshirn @ 2019-09-03 10:46 UTC (permalink / raw)
  To: Anand Jain, linux-btrfs, David Sterba

Hi Anand,

This patch breaks misc-tests/021-image-multi-devices for me:

jthumshirn@adalid:btrfs-progs ((b41eba548d89...)|BISECTING)$ git bisect log
git bisect start
# good: [9a85732d8beaae4b80cab98bb3355660389c1d36] Btrfs progs v5.2.1
git bisect good 9a85732d8beaae4b80cab98bb3355660389c1d36
# bad: [7757aa659556e3654bdd2066ac977cf3c1cf9a4e] btrfs-progs:
fsck-tests: Add test image for valid half-dropped orphan inode
git bisect bad 7757aa659556e3654bdd2066ac977cf3c1cf9a4e
# good: [70351ae92834ff6405b464053871770245165c9b] btrfs-progs: copy
btrfsck.h to check/common.h
git bisect good 70351ae92834ff6405b464053871770245165c9b
# good: [d0848c91d916a203ca45409ed7538120e3b27f69] btrfs-progs: tests:
mkfs and extra large devices
git bisect good d0848c91d916a203ca45409ed7538120e3b27f69
# good: [aa0d7b57658d0aebb8be84456782a55212b49292] btrfs-progs: tests:
cli/003: add resize checks with 'max'
git bisect good aa0d7b57658d0aebb8be84456782a55212b49292
# good: [de8112e7dc53a753a9fbd8c0fb9c7d46359efc56] btrfs-progs:
print-tree add missing DEV_STATS
git bisect good de8112e7dc53a753a9fbd8c0fb9c7d46359efc56
# bad: [1613a542dfaadc9e1d105f6fd077ca7c4fed5900] btrfs-progs:
check/lowmem: Skip nbytes check for orphan inodes
git bisect bad 1613a542dfaadc9e1d105f6fd077ca7c4fed5900
# bad: [17bb2860247876c34bf67f5526cb142f45f2e630] btrfs_progs: mkfs:
match devid order to the stripe index
git bisect bad 17bb2860247876c34bf67f5526cb142f45f2e630
# first bad commit: [17bb2860247876c34bf67f5526cb142f45f2e630]
btrfs_progs: mkfs: match devid order to the stripe index

Can you please have a look?

Thanks,
	Johannes
-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
(HRB 247165, AG München)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH] btrfs_progs: mkfs: match devid order to the stripe index
  2019-09-02 16:22       ` David Sterba
@ 2019-09-03 12:06         ` David Sterba
  2019-09-04 11:10           ` Anand Jain
  2019-09-12 17:54           ` David Sterba
  0 siblings, 2 replies; 19+ messages in thread
From: David Sterba @ 2019-09-03 12:06 UTC (permalink / raw)
  To: dsterba, Anand Jain, linux-btrfs

On Mon, Sep 02, 2019 at 06:22:30PM +0200, David Sterba wrote:
> On Mon, Sep 02, 2019 at 04:01:56PM +0800, Anand Jain wrote:
> > 
> > David,
> > 
> >   I don't see this patch is integrated. Can you please integrated this 
> > patch thanks.
> 
> I don't know why but the patch got lost somewhere, adding to devel
> again.

Not lost, but dropped, misc-tests/021 fails. So dropped again, please
fix it and test before posting again. Thanks.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH] btrfs_progs: mkfs: match devid order to the stripe index
  2019-09-03 12:06         ` David Sterba
@ 2019-09-04 11:10           ` Anand Jain
  2019-09-12 17:54           ` David Sterba
  1 sibling, 0 replies; 19+ messages in thread
From: Anand Jain @ 2019-09-04 11:10 UTC (permalink / raw)
  To: dsterba, linux-btrfs



On 9/3/19 8:06 PM, David Sterba wrote:
> On Mon, Sep 02, 2019 at 06:22:30PM +0200, David Sterba wrote:
>> On Mon, Sep 02, 2019 at 04:01:56PM +0800, Anand Jain wrote:
>>>
>>> David,
>>>
>>>    I don't see this patch is integrated. Can you please integrated this
>>> patch thanks.
>>
>> I don't know why but the patch got lost somewhere, adding to devel
>> again.
> 
> Not lost, but dropped, misc-tests/021 fails. So dropped again, please
> fix it and test before posting again. Thanks.
> 

I have unit tested this patch successfully, misc-tests/021 is a false 
positive failure. Because it assumes btrfs-image collects the data and 
attempts to read the file on the restored image, and rightly ends up 
with IO error.

-------
md5sum: /Volumes/ws/btrfs-progs/tests//mnt/foobar: Input/output error
failed: md5sum /Volumes/ws/btrfs-progs/tests//mnt/foobar
------

The test case is successful so far because the images is restored on the 
same loop device, and unfortunately stale data on the loop device 
matched and helped read from the restored image to succeed.

Tweak the test case a little as below and it fails even without this patch.

----------------------
diff --git a/tests/misc-tests/021-image-multi-devices/test.sh 
b/tests/misc-tests/021-image-multi-devices/test.sh
index b1013b5d2596..05f4146b6007 100755
--- a/tests/misc-tests/021-image-multi-devices/test.sh
+++ b/tests/misc-tests/021-image-multi-devices/test.sh
@@ -17,7 +17,7 @@ loop2=${loopdevs[2]}

  # Create the test file system.

-run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$loop1" "$loop2"
+run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$loop2" "$loop1"
  run_check $SUDO_HELPER mount "$loop1" "$TEST_MNT"
  run_check $SUDO_HELPER dd bs=1M count=1 if=/dev/zero of="$TEST_MNT/foobar"
  orig_md5=$(run_check_stdout md5sum "$TEST_MNT/foobar" | cut -d ' ' -f 1)
----------------------


This patch is good to integrate.

Thanks, Anand

^ permalink raw reply	[flat|nested] 19+ messages in thread

* [PATCH] btrfs: misc-tests-021 fix restore overlapped on disk's stale data
  2019-09-03 10:46 ` Johannes Thumshirn
@ 2019-09-04 12:54   ` Anand Jain
  2019-09-04 13:29   ` [PATCH Fix-title-prefix] btrfs-progs: " Anand Jain
  1 sibling, 0 replies; 19+ messages in thread
From: Anand Jain @ 2019-09-04 12:54 UTC (permalink / raw)
  To: linux-btrfs; +Cc: jthumshirn, dsterba

As misc-tests/021 image dump is restored on the same original loop
device, this overlaps with the stale data and makes the test pass
falsely.

Fix this by using a new device for restore.

And also, the btrfs-image dump and restore doesn't not collect the data,
so any read on the files should be avoided. So instead of file data use
file stat data for the md5sum.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
Reported-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 tests/misc-tests/021-image-multi-devices/test.sh | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/tests/misc-tests/021-image-multi-devices/test.sh b/tests/misc-tests/021-image-multi-devices/test.sh
index b1013b5d2596..5ed8f4b01457 100755
--- a/tests/misc-tests/021-image-multi-devices/test.sh
+++ b/tests/misc-tests/021-image-multi-devices/test.sh
@@ -10,17 +10,18 @@ check_prereq btrfs
 
 setup_root_helper
 
-setup_loopdevs 2
+setup_loopdevs 3
 prepare_loopdevs
 loop1=${loopdevs[1]}
 loop2=${loopdevs[2]}
+loop3=${loopdevs[3]}
 
 # Create the test file system.
 
 run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$loop1" "$loop2"
 run_check $SUDO_HELPER mount "$loop1" "$TEST_MNT"
 run_check $SUDO_HELPER dd bs=1M count=1 if=/dev/zero of="$TEST_MNT/foobar"
-orig_md5=$(run_check_stdout md5sum "$TEST_MNT/foobar" | cut -d ' ' -f 1)
+orig_md5=$(run_check_stdout stat "$TEST_MNT/foobar" | md5sum | cut -d ' ' -f 1)
 run_check $SUDO_HELPER umount "$TEST_MNT"
 
 # Create the image to restore later.
@@ -32,13 +33,13 @@ run_check $SUDO_HELPER "$TOP/btrfs-image" "$loop1" "$IMAGE"
 run_check $SUDO_HELPER wipefs -a "$loop1"
 run_check $SUDO_HELPER wipefs -a "$loop2"
 
-run_check $SUDO_HELPER "$TOP/btrfs-image" -r "$IMAGE" "$loop1"
+run_check $SUDO_HELPER "$TOP/btrfs-image" -r "$IMAGE" "$loop3"
 
 # Run check to make sure there is nothing wrong for the recovered image
-run_check $SUDO_HELPER "$TOP/btrfs" check "$loop1"
+run_check $SUDO_HELPER "$TOP/btrfs" check "$loop3"
 
-run_check $SUDO_HELPER mount "$loop1" "$TEST_MNT"
-new_md5=$(run_check_stdout md5sum "$TEST_MNT/foobar" | cut -d ' ' -f 1)
+run_check $SUDO_HELPER mount "$loop3" "$TEST_MNT"
+new_md5=$(run_check_stdout stat "$TEST_MNT/foobar" | md5sum | cut -d ' ' -f 1)
 run_check $SUDO_HELPER umount "$TEST_MNT"
 
 cleanup_loopdevs
-- 
1.8.3.1


^ permalink raw reply	[flat|nested] 19+ messages in thread

* [PATCH Fix-title-prefix] btrfs-progs: misc-tests-021 fix restore overlapped on disk's stale data
  2019-09-03 10:46 ` Johannes Thumshirn
  2019-09-04 12:54   ` [PATCH] btrfs: misc-tests-021 fix restore overlapped on disk's stale data Anand Jain
@ 2019-09-04 13:29   ` " Anand Jain
  2019-09-04 14:05     ` Johannes Thumshirn
                       ` (3 more replies)
  1 sibling, 4 replies; 19+ messages in thread
From: Anand Jain @ 2019-09-04 13:29 UTC (permalink / raw)
  To: linux-btrfs; +Cc: jthumshirn, dsterba

As misc-tests/021 image dump is restored on the same original loop
device, this overlaps with the stale data and makes the test pass
falsely.

Fix this by using a new device for restore.

And also, the btrfs-image dump and restore doesn't not collect the data,
so any read on the files should be avoided. So instead of file data use
file stat data for the md5sum.

Signed-off-by: Anand Jain <anand.jain@oracle.com>
Reported-by: Johannes Thumshirn <jthumshirn@suse.de>
---
 tests/misc-tests/021-image-multi-devices/test.sh | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/tests/misc-tests/021-image-multi-devices/test.sh b/tests/misc-tests/021-image-multi-devices/test.sh
index b1013b5d2596..5ed8f4b01457 100755
--- a/tests/misc-tests/021-image-multi-devices/test.sh
+++ b/tests/misc-tests/021-image-multi-devices/test.sh
@@ -10,17 +10,18 @@ check_prereq btrfs
 
 setup_root_helper
 
-setup_loopdevs 2
+setup_loopdevs 3
 prepare_loopdevs
 loop1=${loopdevs[1]}
 loop2=${loopdevs[2]}
+loop3=${loopdevs[3]}
 
 # Create the test file system.
 
 run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$loop1" "$loop2"
 run_check $SUDO_HELPER mount "$loop1" "$TEST_MNT"
 run_check $SUDO_HELPER dd bs=1M count=1 if=/dev/zero of="$TEST_MNT/foobar"
-orig_md5=$(run_check_stdout md5sum "$TEST_MNT/foobar" | cut -d ' ' -f 1)
+orig_md5=$(run_check_stdout stat "$TEST_MNT/foobar" | md5sum | cut -d ' ' -f 1)
 run_check $SUDO_HELPER umount "$TEST_MNT"
 
 # Create the image to restore later.
@@ -32,13 +33,13 @@ run_check $SUDO_HELPER "$TOP/btrfs-image" "$loop1" "$IMAGE"
 run_check $SUDO_HELPER wipefs -a "$loop1"
 run_check $SUDO_HELPER wipefs -a "$loop2"
 
-run_check $SUDO_HELPER "$TOP/btrfs-image" -r "$IMAGE" "$loop1"
+run_check $SUDO_HELPER "$TOP/btrfs-image" -r "$IMAGE" "$loop3"
 
 # Run check to make sure there is nothing wrong for the recovered image
-run_check $SUDO_HELPER "$TOP/btrfs" check "$loop1"
+run_check $SUDO_HELPER "$TOP/btrfs" check "$loop3"
 
-run_check $SUDO_HELPER mount "$loop1" "$TEST_MNT"
-new_md5=$(run_check_stdout md5sum "$TEST_MNT/foobar" | cut -d ' ' -f 1)
+run_check $SUDO_HELPER mount "$loop3" "$TEST_MNT"
+new_md5=$(run_check_stdout stat "$TEST_MNT/foobar" | md5sum | cut -d ' ' -f 1)
 run_check $SUDO_HELPER umount "$TEST_MNT"
 
 cleanup_loopdevs
-- 
1.8.3.1


^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH Fix-title-prefix] btrfs-progs: misc-tests-021 fix restore overlapped on disk's stale data
  2019-09-04 13:29   ` [PATCH Fix-title-prefix] btrfs-progs: " Anand Jain
@ 2019-09-04 14:05     ` Johannes Thumshirn
  2019-09-04 14:06     ` Nikolay Borisov
                       ` (2 subsequent siblings)
  3 siblings, 0 replies; 19+ messages in thread
From: Johannes Thumshirn @ 2019-09-04 14:05 UTC (permalink / raw)
  To: Anand Jain, linux-btrfs; +Cc: dsterba

Reviewed-by: Johannes Thumshirn <jthumshirn@suse.de>

-- 
Johannes Thumshirn                            SUSE Labs Filesystems
jthumshirn@suse.de                                +49 911 74053 689
SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
(HRB 247165, AG München)
Key fingerprint = EC38 9CAB C2C4 F25D 8600 D0D0 0393 969D 2D76 0850

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH Fix-title-prefix] btrfs-progs: misc-tests-021 fix restore overlapped on disk's stale data
  2019-09-04 13:29   ` [PATCH Fix-title-prefix] btrfs-progs: " Anand Jain
  2019-09-04 14:05     ` Johannes Thumshirn
@ 2019-09-04 14:06     ` Nikolay Borisov
  2019-09-10  7:43     ` Anand Jain
  2019-09-12 17:49     ` David Sterba
  3 siblings, 0 replies; 19+ messages in thread
From: Nikolay Borisov @ 2019-09-04 14:06 UTC (permalink / raw)
  To: Anand Jain, linux-btrfs; +Cc: dsterba, jthumshirn



On 4.09.19 г. 16:29 ч., Anand Jain wrote:
> As misc-tests/021 image dump is restored on the same original loop
> device, this overlaps with the stale data and makes the test pass
> falsely.
> 
> Fix this by using a new device for restore.
> 
> And also, the btrfs-image dump and restore doesn't not collect the data,
> so any read on the files should be avoided. So instead of file data use
> file stat data for the md5sum.
> 
> Signed-off-by: Anand Jain <anand.jain@oracle.com>
> Reported-by: Johannes Thumshirn <jthumshirn@suse.de>

Reviewed-by: Nikolay Borisov <nborisov@suse.com>

> ---
>  tests/misc-tests/021-image-multi-devices/test.sh | 13 +++++++------
>  1 file changed, 7 insertions(+), 6 deletions(-)
> 
> diff --git a/tests/misc-tests/021-image-multi-devices/test.sh b/tests/misc-tests/021-image-multi-devices/test.sh
> index b1013b5d2596..5ed8f4b01457 100755
> --- a/tests/misc-tests/021-image-multi-devices/test.sh
> +++ b/tests/misc-tests/021-image-multi-devices/test.sh
> @@ -10,17 +10,18 @@ check_prereq btrfs
>  
>  setup_root_helper
>  
> -setup_loopdevs 2
> +setup_loopdevs 3
>  prepare_loopdevs
>  loop1=${loopdevs[1]}
>  loop2=${loopdevs[2]}
> +loop3=${loopdevs[3]}
>  
>  # Create the test file system.
>  
>  run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$loop1" "$loop2"
>  run_check $SUDO_HELPER mount "$loop1" "$TEST_MNT"
>  run_check $SUDO_HELPER dd bs=1M count=1 if=/dev/zero of="$TEST_MNT/foobar"
> -orig_md5=$(run_check_stdout md5sum "$TEST_MNT/foobar" | cut -d ' ' -f 1)
> +orig_md5=$(run_check_stdout stat "$TEST_MNT/foobar" | md5sum | cut -d ' ' -f 1)
>  run_check $SUDO_HELPER umount "$TEST_MNT"
>  
>  # Create the image to restore later.
> @@ -32,13 +33,13 @@ run_check $SUDO_HELPER "$TOP/btrfs-image" "$loop1" "$IMAGE"
>  run_check $SUDO_HELPER wipefs -a "$loop1"
>  run_check $SUDO_HELPER wipefs -a "$loop2"
>  
> -run_check $SUDO_HELPER "$TOP/btrfs-image" -r "$IMAGE" "$loop1"
> +run_check $SUDO_HELPER "$TOP/btrfs-image" -r "$IMAGE" "$loop3"
>  
>  # Run check to make sure there is nothing wrong for the recovered image
> -run_check $SUDO_HELPER "$TOP/btrfs" check "$loop1"
> +run_check $SUDO_HELPER "$TOP/btrfs" check "$loop3"
>  
> -run_check $SUDO_HELPER mount "$loop1" "$TEST_MNT"
> -new_md5=$(run_check_stdout md5sum "$TEST_MNT/foobar" | cut -d ' ' -f 1)
> +run_check $SUDO_HELPER mount "$loop3" "$TEST_MNT"
> +new_md5=$(run_check_stdout stat "$TEST_MNT/foobar" | md5sum | cut -d ' ' -f 1)
>  run_check $SUDO_HELPER umount "$TEST_MNT"
>  
>  cleanup_loopdevs
> 

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH] btrfs_progs: mkfs: match devid order to the stripe index
  2019-06-28  2:26 [PATCH] btrfs_progs: mkfs: match devid order to the stripe index Anand Jain
                   ` (2 preceding siblings ...)
  2019-09-03 10:46 ` Johannes Thumshirn
@ 2019-09-10  7:37 ` Anand Jain
  3 siblings, 0 replies; 19+ messages in thread
From: Anand Jain @ 2019-09-10  7:37 UTC (permalink / raw)
  To: linux-btrfs


Ping.

Thanks, Anand

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH Fix-title-prefix] btrfs-progs: misc-tests-021 fix restore overlapped on disk's stale data
  2019-09-04 13:29   ` [PATCH Fix-title-prefix] btrfs-progs: " Anand Jain
  2019-09-04 14:05     ` Johannes Thumshirn
  2019-09-04 14:06     ` Nikolay Borisov
@ 2019-09-10  7:43     ` Anand Jain
  2019-09-12 17:49     ` David Sterba
  3 siblings, 0 replies; 19+ messages in thread
From: Anand Jain @ 2019-09-10  7:43 UTC (permalink / raw)
  To: linux-btrfs; +Cc: jthumshirn, dsterba


This one too.
Thanks, Anand

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH Fix-title-prefix] btrfs-progs: misc-tests-021 fix restore overlapped on disk's stale data
  2019-09-04 13:29   ` [PATCH Fix-title-prefix] btrfs-progs: " Anand Jain
                       ` (2 preceding siblings ...)
  2019-09-10  7:43     ` Anand Jain
@ 2019-09-12 17:49     ` David Sterba
  3 siblings, 0 replies; 19+ messages in thread
From: David Sterba @ 2019-09-12 17:49 UTC (permalink / raw)
  To: Anand Jain; +Cc: linux-btrfs, jthumshirn, dsterba

On Wed, Sep 04, 2019 at 09:29:47PM +0800, Anand Jain wrote:
> As misc-tests/021 image dump is restored on the same original loop
> device, this overlaps with the stale data and makes the test pass
> falsely.
> 
> Fix this by using a new device for restore.
> 
> And also, the btrfs-image dump and restore doesn't not collect the data,
> so any read on the files should be avoided. So instead of file data use
> file stat data for the md5sum.
> 
> Signed-off-by: Anand Jain <anand.jain@oracle.com>
> Reported-by: Johannes Thumshirn <jthumshirn@suse.de>
> ---

Applied, thanks.

^ permalink raw reply	[flat|nested] 19+ messages in thread

* Re: [PATCH] btrfs_progs: mkfs: match devid order to the stripe index
  2019-09-03 12:06         ` David Sterba
  2019-09-04 11:10           ` Anand Jain
@ 2019-09-12 17:54           ` David Sterba
  1 sibling, 0 replies; 19+ messages in thread
From: David Sterba @ 2019-09-12 17:54 UTC (permalink / raw)
  To: dsterba, Anand Jain, linux-btrfs

On Tue, Sep 03, 2019 at 02:06:03PM +0200, David Sterba wrote:
> On Mon, Sep 02, 2019 at 06:22:30PM +0200, David Sterba wrote:
> > On Mon, Sep 02, 2019 at 04:01:56PM +0800, Anand Jain wrote:
> > > 
> > > David,
> > > 
> > >   I don't see this patch is integrated. Can you please integrated this 
> > > patch thanks.
> > 
> > I don't know why but the patch got lost somewhere, adding to devel
> > again.
> 
> Not lost, but dropped, misc-tests/021 fails. So dropped again, please
> fix it and test before posting again. Thanks.

With the test misc/021 updated, this patch has been added to devel.
Thanks.

^ permalink raw reply	[flat|nested] 19+ messages in thread

end of thread, back to index

Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-06-28  2:26 [PATCH] btrfs_progs: mkfs: match devid order to the stripe index Anand Jain
2019-06-28  2:44 ` Qu Wenruo
2019-06-28  3:28   ` Anand Jain
2019-06-28  6:01     ` Qu Wenruo
2019-07-03 13:21 ` David Sterba
2019-08-27  2:02   ` Anand Jain
2019-09-02  8:01     ` Anand Jain
2019-09-02 16:22       ` David Sterba
2019-09-03 12:06         ` David Sterba
2019-09-04 11:10           ` Anand Jain
2019-09-12 17:54           ` David Sterba
2019-09-03 10:46 ` Johannes Thumshirn
2019-09-04 12:54   ` [PATCH] btrfs: misc-tests-021 fix restore overlapped on disk's stale data Anand Jain
2019-09-04 13:29   ` [PATCH Fix-title-prefix] btrfs-progs: " Anand Jain
2019-09-04 14:05     ` Johannes Thumshirn
2019-09-04 14:06     ` Nikolay Borisov
2019-09-10  7:43     ` Anand Jain
2019-09-12 17:49     ` David Sterba
2019-09-10  7:37 ` [PATCH] btrfs_progs: mkfs: match devid order to the stripe index Anand Jain

Linux-BTRFS Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-btrfs/0 linux-btrfs/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-btrfs linux-btrfs/ https://lore.kernel.org/linux-btrfs \
		linux-btrfs@vger.kernel.org linux-btrfs@archiver.kernel.org
	public-inbox-index linux-btrfs


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-btrfs


AGPL code for this site: git clone https://public-inbox.org/ public-inbox