Question: how understand the raid profile of a btrfs filesystem

Question: how understand the raid profile of a btrfs filesystem

[Goffredo Baroncelli]

Hi all,

for a btrfs filesystem, how an user can understand which is the {data,mmetadata,system} [raid] profile in use ? E.g. the next chunk which profile will have ?
For simple filesystem it is easy looking at the output of (e.g)  "btrfs fi df" or "btrfs fi us". But what if the filesystem is not simple ?

btrfs fi us t/.
Overall:
     Device size:		  40.00GiB
     Device allocated:		  19.52GiB
     Device unallocated:		  20.48GiB
     Device missing:		     0.00B
     Used:			  16.75GiB
     Free (estimated):		  12.22GiB	(min: 8.27GiB)
     Data ratio:			      1.90
     Metadata ratio:		      2.00
     Global reserve:		   9.06MiB	(used: 0.00B)

Data,single: Size:1.00GiB, Used:512.00MiB (50.00%)
    /dev/loop0	   1.00GiB

Data,RAID5: Size:3.00GiB, Used:2.48GiB (82.56%)
    /dev/loop1	   1.00GiB
    /dev/loop2	   1.00GiB
    /dev/loop3	   1.00GiB
    /dev/loop0	   1.00GiB

Data,RAID6: Size:4.00GiB, Used:3.71GiB (92.75%)
    /dev/loop1	   2.00GiB
    /dev/loop2	   2.00GiB
    /dev/loop3	   2.00GiB
    /dev/loop0	   2.00GiB

Data,RAID1C3: Size:2.00GiB, Used:1.88GiB (93.76%)
    /dev/loop1	   2.00GiB
    /dev/loop2	   2.00GiB
    /dev/loop3	   2.00GiB

Metadata,RAID1: Size:256.00MiB, Used:9.14MiB (3.57%)
    /dev/loop2	 256.00MiB
    /dev/loop3	 256.00MiB

System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%)
    /dev/loop2	   8.00MiB
    /dev/loop3	   8.00MiB

Unallocated:
    /dev/loop1	   5.00GiB
    /dev/loop2	   4.74GiB
    /dev/loop3	   4.74GiB
    /dev/loop0	   6.00GiB

This is an example of a strange but valid filesystem. So the question is: the next chunk which profile will have ?
Is there any way to understand what will happens ?

I expected that the next chunk will be allocated as the last "convert". However I discovered that this is not true.

Looking at the code it seems to me that the logic is the following (from btrfs_reduce_alloc_profile())

         if (allowed & BTRFS_BLOCK_GROUP_RAID6)
                 allowed = BTRFS_BLOCK_GROUP_RAID6;
         else if (allowed & BTRFS_BLOCK_GROUP_RAID5)
                 allowed = BTRFS_BLOCK_GROUP_RAID5;
         else if (allowed & BTRFS_BLOCK_GROUP_RAID10)
                 allowed = BTRFS_BLOCK_GROUP_RAID10;
         else if (allowed & BTRFS_BLOCK_GROUP_RAID1)
                 allowed = BTRFS_BLOCK_GROUP_RAID1;
         else if (allowed & BTRFS_BLOCK_GROUP_RAID0)
                 allowed = BTRFS_BLOCK_GROUP_RAID0;

         flags &= ~BTRFS_BLOCK_GROUP_PROFILE_MASK;

So in the case above the profile will be RAID6. And in the general if a RAID6 chunk is a filesystem, it wins !

But I am not sure.. Moreover I expected to see also reference to DUP and/or RAID1C[34] ...

Does someone have any suggestion ?

BR
G.Baroncelli

Re: Question: how understand the raid profile of a btrfs filesystem

[Zygo Blaxell]

On Fri, Mar 20, 2020 at 06:56:38PM +0100, Goffredo Baroncelli wrote:
> Hi all,
> 
> for a btrfs filesystem, how an user can understand which is the {data,mmetadata,system} [raid] profile in use ? E.g. the next chunk which profile will have ?

It's the profile used by the highest-numbered block group for the
allocation type (one for data, one for metadata/system).  There
are two profiles to consider, one for data and one for metadata.
'btrfs fi df', 'btrfs fi us', or 'btrfs dev usage' will all indicate
which profiles these are.

It is valid for the two profiles to be different, and different profile
combinations have different use cases.  In most cases the only one that
matters is the data profile, as that's the one POSIX 'df' reports and
data block writes consume, and the one that typically occupies more than
99% of the total space.  Adminstrators and system designers have to be
more aware of metadata usage when filesystems become extremely full or
extremely small (less than 16 GB).  Users (without root or CAP_SYS_ADMIN)
generally can't do anything about metadata usage, except as a tiny
side-effect of their data usage.

> For simple filesystem it is easy looking at the output of (e.g)  "btrfs fi df" or "btrfs fi us". But what if the filesystem is not simple ?

"Not simple" is not a normal operating mode for btrfs.  The filesystem
allows multiple profiles to be active, so the filesystem can be
converted to a new profile while old data is still accessible; however,
the conversion is expected to end at some point, and all block groups
will use the same profile when that happens.

The allocator will only use one RAID profile, and will ignore free
space in block groups of other profiles, while 'df' reports the total
space on the filesystem in each profile, and metadata allocation does
something else.  'btrfs fi us' reports a mess and can't give any accurate
free space estimate.  Disk space will apparently be free while writes
fail with ENOSPC.

This is not a problem if a conversion is running to eliminate all the
"competing" profiles, but if the conversion stops, you can expect some
problems with space until it resumes again.

> btrfs fi us t/.
> Overall:
>     Device size:		  40.00GiB
>     Device allocated:		  19.52GiB
>     Device unallocated:		  20.48GiB
>     Device missing:		     0.00B
>     Used:			  16.75GiB
>     Free (estimated):		  12.22GiB	(min: 8.27GiB)
>     Data ratio:			      1.90
>     Metadata ratio:		      2.00
>     Global reserve:		   9.06MiB	(used: 0.00B)
> 
> Data,single: Size:1.00GiB, Used:512.00MiB (50.00%)
>    /dev/loop0	   1.00GiB
> 
> Data,RAID5: Size:3.00GiB, Used:2.48GiB (82.56%)
>    /dev/loop1	   1.00GiB
>    /dev/loop2	   1.00GiB
>    /dev/loop3	   1.00GiB
>    /dev/loop0	   1.00GiB
> 
> Data,RAID6: Size:4.00GiB, Used:3.71GiB (92.75%)
>    /dev/loop1	   2.00GiB
>    /dev/loop2	   2.00GiB
>    /dev/loop3	   2.00GiB
>    /dev/loop0	   2.00GiB
> 
> Data,RAID1C3: Size:2.00GiB, Used:1.88GiB (93.76%)
>    /dev/loop1	   2.00GiB
>    /dev/loop2	   2.00GiB
>    /dev/loop3	   2.00GiB
> 
> Metadata,RAID1: Size:256.00MiB, Used:9.14MiB (3.57%)
>    /dev/loop2	 256.00MiB
>    /dev/loop3	 256.00MiB
> 
> System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%)
>    /dev/loop2	   8.00MiB
>    /dev/loop3	   8.00MiB
> 
> Unallocated:
>    /dev/loop1	   5.00GiB
>    /dev/loop2	   4.74GiB
>    /dev/loop3	   4.74GiB
>    /dev/loop0	   6.00GiB
> 
> This is an example of a strange but valid filesystem. 

Valid, but the filesystem is in a state designed for temporary use during
conversions, and you will want to exit that state as soon as possible.

> So the question is: the next chunk which profile will have ?
> Is there any way to understand what will happens ?
> 
> I expected that the next chunk will be allocated as the last "convert". However I discovered that this is not true.

That's correct in most cases--a convert will create a new block group,
which will have the highest bytenr in the filesystem, and its profile
will be used to allocate new data, thus converting the filesystem to
the new profile; however, if you pause the convert and delete all the
files in the new block group, it's possible that the new block group gets
deleted too, and then the filesystem reverts to the previous RAID profile.
Again, not a problem if you run the convert until it completely removes
all old block groups!

> Looking at the code it seems to me that the logic is the following (from btrfs_reduce_alloc_profile())
> 
>         if (allowed & BTRFS_BLOCK_GROUP_RAID6)
>                 allowed = BTRFS_BLOCK_GROUP_RAID6;
>         else if (allowed & BTRFS_BLOCK_GROUP_RAID5)
>                 allowed = BTRFS_BLOCK_GROUP_RAID5;
>         else if (allowed & BTRFS_BLOCK_GROUP_RAID10)
>                 allowed = BTRFS_BLOCK_GROUP_RAID10;
>         else if (allowed & BTRFS_BLOCK_GROUP_RAID1)
>                 allowed = BTRFS_BLOCK_GROUP_RAID1;
>         else if (allowed & BTRFS_BLOCK_GROUP_RAID0)
>                 allowed = BTRFS_BLOCK_GROUP_RAID0;
> 
>         flags &= ~BTRFS_BLOCK_GROUP_PROFILE_MASK;
> 
> So in the case above the profile will be RAID6. And in the general if a RAID6 chunk is a filesystem, it wins !

This code is used to determine whether a conversion reduces the level of
redundancy, e.g. you are going from raid6 (2 redundant disks) to raid5
(1 redundant disk) or raid0 (0 redundant disks).  There are warnings and
a force flag required when that happens.  It doesn't determine the raid
profile of the next block group--that's just a straight copy of the raid
profile of the last block group.

> But I am not sure.. Moreover I expected to see also reference to DUP and/or RAID1C[34] ...

If you get through that 'if' statement without hitting any of the
branches, then you're equal to raid0 (0 redundant disks) but raid0
is a special case because it requires 2 disks for allocation.  'dup'
(0 redundant disks) and 'single' (which is the absence of any profile
bits) also have 0 redundant disks and require only 1 disk for allocation,
there is no need to treat them differently.

raid1c[34] probably should be there.  Patches welcome.

> Does someone have any suggestion ?
> 
> BR
> G.Baroncelli
> 

Re: Question: how understand the raid profile of a btrfs filesystem

[Andrei Borzenkov]

21.03.2020 06:29, Zygo Blaxell пишет:
> On Fri, Mar 20, 2020 at 06:56:38PM +0100, Goffredo Baroncelli wrote:
>> Hi all,
>>
>> for a btrfs filesystem, how an user can understand which is the {data,mmetadata,system} [raid] profile in use ? E.g. the next chunk which profile will have ?
> 
> It's the profile used by the highest-numbered block group for the
> allocation type (one for data, one for metadata/system).

Is "highest-numbered" block group always the last one created? Can block
group numbers wrap around?

Recently someone reported that after conversion block groups with old
profile remained and this probably explains it - conversion races with
new allocation.

>> So the question is: the next chunk which profile will have ?
>> Is there any way to understand what will happens ?

Well, from that explanation it is not possible using standard tools -
one needs to crawl btrfs internals to find out the "last" block group.

Re: Question: how understand the raid profile of a btrfs filesystem

[Zygo Blaxell]

On Sat, Mar 21, 2020 at 08:40:50AM +0300, Andrei Borzenkov wrote:
> 21.03.2020 06:29, Zygo Blaxell пишет:
> > On Fri, Mar 20, 2020 at 06:56:38PM +0100, Goffredo Baroncelli wrote:
> >> Hi all,
> >>
> >> for a btrfs filesystem, how an user can understand which is the {data,mmetadata,system} [raid] profile in use ? E.g. the next chunk which profile will have ?
> > 
> > It's the profile used by the highest-numbered block group for the
> > allocation type (one for data, one for metadata/system).
> 
> Is "highest-numbered" block group always the last one created? 

It's not required by the filesystem format but it is the current behavior
of the implementation.

> Can block group numbers wrap around?

In theory, yes, but they are 64 bits long and correspond to bytes in the
filesystem's address space.  If you loop balancing a filesystem with a
single 4K data block and you can do it at 1000 block groups per second,
you'll wrap around in a little over six months.  Typical use cases
(and even extreme ones) will take centuries to wrap around if you
are converting all the time.

> Recently someone reported that after conversion block groups with old
> profile remained and this probably explains it - conversion races with
> new allocation.

Conversion *is* new allocation, no race is possible because they are
the same thing.

While a conversion is running, the conversion itself forces the
raid profile of newly created block groups, so there is no race.
After conversion is completed, there is special case code to prevent the
last empty block group in the filesystem from being deleted; otherwise,
btrfs would lose information about the selected raid profile.

When a conversion is paused or cancelled, new allocations normally
continue using the conversion target profile; however, if all block
groups of the new profile are deleted (i.e. all the data contained in
the new block groups are removed) then it is possible to revert back to
allocating using an older profile.  e.g. if you want to combine a balance
convert with a device remove, you have to let the convert run long enough
to ensure several block groups of the new raid profile exist on other
drives than the drive being removed.  The device remove will delete all
block groups on the removed device, in reverse device physical offset
order which is often (but not necessarily) reverse block group order.
This leads to device remove switching back to the old RAID profile.
This example is not any kind of race--the result can be produced
deterministically, and the conversion must be paused first.

A conversion can be forcibly stopped by various events:  crashes,
unmounting the filesystem, having an unrecoverable read or write error,
or running out of space.  These events will leave block groups with old
profiles on the disk.  Generally if an external event forces conversion
to stop, then it will need to be manually restarted.

If there are uncorrectable read errors on the filesystem then affected
data blocks must be removed from the filesystem before conversion can
be completed.  Same with free space, you must have enough to complete.

Old versions of mkfs.btrfs had bugs which would leave empty block groups
with different profiles on the filesystem.  When in doubt, or if you have
an older vintage btrfs filesystem, run a converting balance with the
desired raid profile and the 'soft' filter to be sure only one profile
is present--it will be a no-op if conversion is complete; otherwise,
it will finish the conversion.

> >> So the question is: the next chunk which profile will have ?
> >> Is there any way to understand what will happens ?
> 
> Well, from that explanation it is not possible using standard tools -
> one needs to crawl btrfs internals to find out the "last" block group.

This is required only during the conversion process.  In normal cases
users can assume the only profile present is the one that will be used.

The python-btrfs package contains an example of listing block groups.
The last entry in the list will have the current allocation profile.

An unprivileged user can monitor 'btrfs fi df' output over time.
Used space will increase or decrease in the current profile, and
only decrease in the other profiles.

Re: Question: how understand the raid profile of a btrfs filesystem

[Goffredo Baroncelli]

On 3/21/20 4:29 AM, Zygo Blaxell wrote:
> On Fri, Mar 20, 2020 at 06:56:38PM +0100, Goffredo Baroncelli wrote:
>> Hi all,
>>
>> for a btrfs filesystem, how an user can understand which is the {data,mmetadata,system} [raid] profile in use ? E.g. the next chunk which profile will have ?
> 
> It's the profile used by the highest-numbered block group for the
> allocation type (one for data, one for metadata/system).  There
> are two profiles to consider, one for data and one for metadata.
> 'btrfs fi df', 'btrfs fi us', or 'btrfs dev usage' will all indicate
> which profiles these are.
> 

What do you think as "highest-numbered block group", the value in the "offset" filed.
If so it doesn't make sense because it could be relocated easily.

Anyway what are you describing is not what I saw. In the test above I create a raid5 filesystem, filled 1 chunk at 100% and a second chunk for few MB. Then I convert the most empty chunk as single. Then I fill the last chunk (the single one) and force to create a new chunk. What I saw is that the new chunk is raid5 mode.

$ sudo mkfs.btrfs -draid5  /dev/loop[012]
$ dd if=/dev/zero of=t/file-2.128gb_5 bs=1M count=$((2024+128)) # fill two chunk raid 5
$ sudo btrfs fi du t/. # see what is the situation
[...]
Data,RAID5: Size:4.00GiB, Used:2.10GiB (52.57%)
    /dev/loop0	   2.00GiB
    /dev/loop1	   2.00GiB
    /dev/loop2	   2.00GiB
[...]
$ sudo btrfs balance start -dconvert=single,usage=50 t/. # convert the latest chunk to single
$ sudo btrfs fi us t/.	# see what is the situation
[...]
Data,single: Size:1.00GiB, Used:259.00MiB (25.29%)
    /dev/loop0	   1.00GiB

Data,RAID5: Size:2.00GiB, Used:1.85GiB (92.47%)
    /dev/loop0	   1.00GiB
    /dev/loop1	   1.00GiB
    /dev/loop2	   1.00GiB
[...]

# fill the latest chunk and created a new one
$ dd if=/dev/zero of=t/file-1.128gb_6 bs=1M count=$((1024+128))

$ sudo btrfs fi us t/. # see what is the situation
[...]
Data,single: Size:1.00GiB, Used:259.00MiB (25.29%)
    /dev/loop0	   1.00GiB

Data,RAID5: Size:4.00GiB, Used:1.85GiB (46.24%)
    /dev/loop0	   2.00GiB
    /dev/loop1	   2.00GiB
    /dev/loop2	   2.00GiB
[...]

Expected results: the "single" chunk should pass from 1GB to 2GB. What it is observed is that the raid5 (the oldest chunk) passed from 2GB to 4GB.

[...]
>> Looking at the code it seems to me that the logic is the following (from btrfs_reduce_alloc_profile())
>>
>>          if (allowed & BTRFS_BLOCK_GROUP_RAID6)
>>                  allowed = BTRFS_BLOCK_GROUP_RAID6;
>>          else if (allowed & BTRFS_BLOCK_GROUP_RAID5)
>>                  allowed = BTRFS_BLOCK_GROUP_RAID5;
>>          else if (allowed & BTRFS_BLOCK_GROUP_RAID10)
>>                  allowed = BTRFS_BLOCK_GROUP_RAID10;
>>          else if (allowed & BTRFS_BLOCK_GROUP_RAID1)
>>                  allowed = BTRFS_BLOCK_GROUP_RAID1;
>>          else if (allowed & BTRFS_BLOCK_GROUP_RAID0)
>>                  allowed = BTRFS_BLOCK_GROUP_RAID0;
>>
>>          flags &= ~BTRFS_BLOCK_GROUP_PROFILE_MASK;
>>
>> So in the case above the profile will be RAID6. And in the general if a RAID6 chunk is a filesystem, it wins !
> 
> This code is used to determine whether a conversion reduces the level of
> redundancy, e.g. you are going from raid6 (2 redundant disks) to raid5
> (1 redundant disk) or raid0 (0 redundant disks).  There are warnings and
> a force flag required when that happens.  It doesn't determine the raid
> profile of the next block group--that's just a straight copy of the raid
> profile of the last block group.

To me it seems that this function decides the allocation of the next chunk. The chain of call is the following:

btrfs_force_chunk_alloc
	btrfs_get_alloc_profile
		get_alloc_profile
			btrfs_reduce_alloc_profile
	btrfs_chunk_alloc
		btrfs_alloc_chunk
			__btrfs_alloc_chunk

or another one is

btrfs_alloc_data_chunk_ondemand
	btrfs_data_alloc_profile
		btrfs_get_alloc_profile
			get_alloc_profile
				btrfs_reduce_alloc_profile
	btrfs_chunk_alloc
		btrfs_alloc_chunk
			__btrfs_alloc_chunk


The btrfs_get_alloc_profile/get_alloc_profile/btrfs_reduce_alloc_profile chain decides which profile has to be allocated.
The current actives profiles are took and then filtered by the possible allowed on the basis of the number of disk. Which means that if a raid6 profile chunk exists (and there is a enough number of devices), the next chunk will be allocated as raid6.

So is how I read the code, and what suggest my tests...

My conclusion is: if you have multiple raid profile per disk, the next chunk allocation doesn't depend by the latest "balance", but but by the above logic.
The recipe is: when you made a balance, pay attention to not leave any chunk in old format

> 
>> But I am not sure.. Moreover I expected to see also reference to DUP and/or RAID1C[34] ...
> 
> If you get through that 'if' statement without hitting any of the
> branches, then you're equal to raid0 (0 redundant disks) but raid0
> is a special case because it requires 2 disks for allocation.  'dup'
> (0 redundant disks) and 'single' (which is the absence of any profile
> bits) also have 0 redundant disks and require only 1 disk for allocation,
> there is no need to treat them differently.
> 
> raid1c[34] probably should be there.  Patches welcome.
> 
>> Does someone have any suggestion ?
>>
>> BR
>> G.Baroncelli
>>


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: Question: how understand the raid profile of a btrfs filesystem

[Zygo Blaxell]

On Sat, Mar 21, 2020 at 10:55:32AM +0100, Goffredo Baroncelli wrote:
> On 3/21/20 4:29 AM, Zygo Blaxell wrote:
> > On Fri, Mar 20, 2020 at 06:56:38PM +0100, Goffredo Baroncelli wrote:
> > > Hi all,
> > > 
> > > for a btrfs filesystem, how an user can understand which is the {data,mmetadata,system} [raid] profile in use ? E.g. the next chunk which profile will have ?
> > 
> > It's the profile used by the highest-numbered block group for the
> > allocation type (one for data, one for metadata/system).  There
> > are two profiles to consider, one for data and one for metadata.
> > 'btrfs fi df', 'btrfs fi us', or 'btrfs dev usage' will all indicate
> > which profiles these are.
> > 
> 
> What do you think as "highest-numbered block group", the value in the "offset" filed.

Objectid field (the offset field is the size for block group items).

> If so it doesn't make sense because it could be relocated easily.

Relocation will create a new block group with the filesystem's current
profile, which is the conversion target profile if present (all conversion
is relocation), but some other profile in use on the filesystem otherwise.

> Anyway what are you describing is not what I saw. In the test above
> I create a raid5 filesystem, filled 1 chunk at 100% and a second chunk
> for few MB. Then I convert the most empty chunk as single. 

OK, I was missing some details:  At mount time all the block group items
are read in order, and each one adjusts the allocator profile bits for
the entire filesystem.  The last block group is the one that has the
*most influence* over the profile when no conversion is running, but
doesn't set the profile alone.

If there is a partial conversion, then the behavior changes as you note.

When a conversion is active, the conversion target profile overrides
everything else.  That is how you can get a single block group on a
filesystem that is entirely raid5.

So...TL;DR if you're not running a conversion, the next block group will
use some RAID profile already present on the filesystem, and it may not
be the one you want it to be.

> Then I fill
> the last chunk (the single one) and force to create a new chunk. What
> I saw is that the new chunk is raid5 mode.
> 
> $ sudo mkfs.btrfs -draid5  /dev/loop[012]
> $ dd if=/dev/zero of=t/file-2.128gb_5 bs=1M count=$((2024+128)) # fill two chunk raid 5
> $ sudo btrfs fi du t/. # see what is the situation
> [...]
> Data,RAID5: Size:4.00GiB, Used:2.10GiB (52.57%)
>    /dev/loop0	   2.00GiB
>    /dev/loop1	   2.00GiB
>    /dev/loop2	   2.00GiB
> [...]
> $ sudo btrfs balance start -dconvert=single,usage=50 t/. # convert the latest chunk to single
> $ sudo btrfs fi us t/.	# see what is the situation
> [...]
> Data,single: Size:1.00GiB, Used:259.00MiB (25.29%)
>    /dev/loop0	   1.00GiB
> 
> Data,RAID5: Size:2.00GiB, Used:1.85GiB (92.47%)
>    /dev/loop0	   1.00GiB
>    /dev/loop1	   1.00GiB
>    /dev/loop2	   1.00GiB
> [...]
> 
> # fill the latest chunk and created a new one
> $ dd if=/dev/zero of=t/file-1.128gb_6 bs=1M count=$((1024+128))
> 
> $ sudo btrfs fi us t/. # see what is the situation
> [...]
> Data,single: Size:1.00GiB, Used:259.00MiB (25.29%)
>    /dev/loop0	   1.00GiB
> 
> Data,RAID5: Size:4.00GiB, Used:1.85GiB (46.24%)
>    /dev/loop0	   2.00GiB
>    /dev/loop1	   2.00GiB
>    /dev/loop2	   2.00GiB
> [...]
> 
> Expected results: the "single" chunk should pass from 1GB to 2GB. What it is observed is that the raid5 (the oldest chunk) passed from 2GB to 4GB.

...but now you are not running conversion any more, and have multiple
profiles.  It's not really specified what will happen under those
conditions, nor is it obvious what the correct behavior should be.

The on-disk format does not have a field for "target profile".
Adding one would be a disk format change.

> [...]
> > > Looking at the code it seems to me that the logic is the following (from btrfs_reduce_alloc_profile())
> > > 
> > >          if (allowed & BTRFS_BLOCK_GROUP_RAID6)
> > >                  allowed = BTRFS_BLOCK_GROUP_RAID6;
> > >          else if (allowed & BTRFS_BLOCK_GROUP_RAID5)
> > >                  allowed = BTRFS_BLOCK_GROUP_RAID5;
> > >          else if (allowed & BTRFS_BLOCK_GROUP_RAID10)
> > >                  allowed = BTRFS_BLOCK_GROUP_RAID10;
> > >          else if (allowed & BTRFS_BLOCK_GROUP_RAID1)
> > >                  allowed = BTRFS_BLOCK_GROUP_RAID1;
> > >          else if (allowed & BTRFS_BLOCK_GROUP_RAID0)
> > >                  allowed = BTRFS_BLOCK_GROUP_RAID0;
> > > 
> > >          flags &= ~BTRFS_BLOCK_GROUP_PROFILE_MASK;
> > > 
> > > So in the case above the profile will be RAID6. And in the general if a RAID6 chunk is a filesystem, it wins !
> > 
> > This code is used to determine whether a conversion reduces the level of
> > redundancy, e.g. you are going from raid6 (2 redundant disks) to raid5
> > (1 redundant disk) or raid0 (0 redundant disks).  There are warnings and
> > a force flag required when that happens.  It doesn't determine the raid
> > profile of the next block group--that's just a straight copy of the raid
> > profile of the last block group.
> 
> To me it seems that this function decides the allocation of the next chunk. The chain of call is the following:

Sorry, in my earlier mail I thought we were talking about a different
piece of code that tries to enforce a similar rule.

> btrfs_force_chunk_alloc
> 	btrfs_get_alloc_profile
> 		get_alloc_profile
> 			btrfs_reduce_alloc_profile
> 	btrfs_chunk_alloc
> 		btrfs_alloc_chunk
> 			__btrfs_alloc_chunk
> 
> or another one is
> 
> btrfs_alloc_data_chunk_ondemand
> 	btrfs_data_alloc_profile
> 		btrfs_get_alloc_profile
> 			get_alloc_profile
> 				btrfs_reduce_alloc_profile
> 	btrfs_chunk_alloc
> 		btrfs_alloc_chunk
> 			__btrfs_alloc_chunk
> 
> 
> The btrfs_get_alloc_profile/get_alloc_profile/btrfs_reduce_alloc_profile chain decides which profile has to be allocated.
> The current actives profiles are took and then filtered by the possible allowed on the basis of the number of disk. Which means that if a raid6 profile chunk exists (and there is a enough number of devices), the next chunk will be allocated as raid6.
> 
> So is how I read the code, and what suggest my tests...
> 
> My conclusion is: if you have multiple raid profile per disk, the next chunk allocation doesn't depend by the latest "balance", but but by the above logic.
> The recipe is: when you made a balance, pay attention to not leave any chunk in old format

Well, yes, that is what I've been saying:  don't expect btrfs to do sane
things with a mixture of profiles.  Stick to just one profile, except
in the special case of a conversion.  You wouldn't leave an array in
degraded mode for long, and you need to balance after adding a single
drive to a raid1 or striped-profile raid array.  Partially converted
filesystems fall into this category too.

> > > But I am not sure.. Moreover I expected to see also reference to DUP and/or RAID1C[34] ...
> > 
> > If you get through that 'if' statement without hitting any of the
> > branches, then you're equal to raid0 (0 redundant disks) but raid0
> > is a special case because it requires 2 disks for allocation.  'dup'
> > (0 redundant disks) and 'single' (which is the absence of any profile
> > bits) also have 0 redundant disks and require only 1 disk for allocation,
> > there is no need to treat them differently.
> > 
> > raid1c[34] probably should be there.  Patches welcome.
> > 
> > > Does someone have any suggestion ?
> > > 
> > > BR
> > > G.Baroncelli
> > > 
> 
> 
> -- 
> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
> 

Re: Question: how understand the raid profile of a btrfs filesystem

[Goffredo Baroncelli]

Hi Zygo,

On 3/22/20 12:26 AM, Zygo Blaxell wrote:
> On Sat, Mar 21, 2020 at 10:55:32AM +0100, Goffredo Baroncelli wrote:
>> On 3/21/20 4:29 AM, Zygo Blaxell wrote:
>>> On Fri, Mar 20, 2020 at 06:56:38PM +0100, Goffredo Baroncelli wrote:
>>>> Hi all,
>>>>

[...]


> ...but now you are not running conversion any more, and have multiple
> profiles.  It's not really specified what will happen under those
> conditions, nor is it obvious what the correct behavior should be.
> 
> The on-disk format does not have a field for "target profile".

Ok, I looked for a confirmation of that.

> Adding one would be a disk format change.

Yes but I think that it would be done in a backward compatible way. I
think to add a field "target profile" in the super-block. The old
kernels will ignore this field, and behave as today. The new ones will
allocates the new chunk according to this field.

To me it seems complicated to

Any thoughts ?

BR
G.Baroncelli


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: Question: how understand the raid profile of a btrfs filesystem

[Goffredo Baroncelli]

On 3/22/20 9:34 AM, Goffredo Baroncelli wrote:

> 
> To me it seems complicated to
[sorry I push the send button too early]

To me it seems too complicated (and error prone) to derive the target profile from an analysis of the filesystem.

Any thoughts ?
  
BR
G.Baroncelli

-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: Question: how understand the raid profile of a btrfs filesystem

[Zygo Blaxell]

On Sun, Mar 22, 2020 at 09:38:30AM +0100, Goffredo Baroncelli wrote:
> On 3/22/20 9:34 AM, Goffredo Baroncelli wrote:
> 
> > 
> > To me it seems complicated to
> [sorry I push the send button too early]
> 
> To me it seems too complicated (and error prone) to derive the target profile from an analysis of the filesystem.
> 
> Any thoughts ?

I still don't understand the use case you are trying to support.

There are 3 states for a btrfs filesystem:

	1.  All block groups use the same profile.  Pick any one,
	use its profile for future block groups.  Avoid deleting the
	last one.  Simple and easy to implement.

	2.  A conversion is in progress.  Look in fs_info->balance_ctl
	for a 'convert' filter.  If there is one, that's the profile for
	new block groups.  Old block groups will be emptied and destroyed
	by conversion, and then we automatically go back to state #1.

	3.  A conversion is interrupted prior to completion.  Sysadmin is
	expected to proceed immediately back to state #2, possibly after
	taking any necessary recovery actions that triggered entry into
	state #3.  It doesn't really matter what the current allocation
	profile is, since it is likely to change before we allocate
	any more block groups.

You seem to be trying to sustain or support a filesystem in state #3 for
a prolonged period of time.  Why would we do that?  If your use case is
providing information or guidance to a user, tell them how to get back
to state #2 ASAP, so that they can then return to state #1 where they
should be.

Suppose your use case does involve staying in state #3 for a prolonged
period of time--let's say e.g. you want to be able to use file attributes
to put some file data on single profile while putting other files on raid5
profile.  That use case would need to come with a bunch of infrastructure
to support it, i.e. you'd need to define what the attributes are, and
how btrfs could map those to device subsets and raid profiles.  None of
this exists, and even if it did, it would conflict with the "store the
[singular] target profile on disk" idea.

There could be a warning message in dmesg if we enter state #3.
This message would appear after a converting balance is cancelled or
aborted, and on mount when we scan block groups (which we would still need
to do even after we added a "target profile" field to the superblock).
Userspace like 'btrfs fi df' could also put out a warning like "multiple
allocation profiles detected, but conversion is not in progress.  Please
finish conversion at your earliest convenience to avoid disappointment."
I don't see the need to do anything more about it.

We only get to state #3 if the automation has already failed, or has
been explicitly cancelled at sysadmin request.  It is better to wait
for the sysadmin to decide what to do next, especially if the sysadmin's
prior choice led to us entering this state (e.g. not enough space to
complete a conversion to the target profile, so we can no longer use
the target profile for new allocations).  Picking a target profile
at random (from the set of profiles already used in the filesystem)
is no better or worse than any deterministic algorithm--it will always
be wrong in some situations, and a good choice in other situations.

I'd even consider removing the heuristics that are already there for
prioritizing profiles.  They are just surprising and undocumented
behavior, and it would be better to document it as "random, BTW you
should finish your conversion now."  It doesn't help if e.g. you
want to convert from raid6 to raid1, since the heuristic assumes you
only want to go the other way.


> BR
> G.Baroncelli
> 
> -- 
> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5
> 

Re: Question: how understand the raid profile of a btrfs filesystem

[Goffredo Baroncelli]

On 3/23/20 12:49 AM, Zygo Blaxell wrote:
> On Sun, Mar 22, 2020 at 09:38:30AM +0100, Goffredo Baroncelli wrote:
>> On 3/22/20 9:34 AM, Goffredo Baroncelli wrote:
>>
>>>
>>> To me it seems complicated to
>> [sorry I push the send button too early]
>>
>> To me it seems too complicated (and error prone) to derive the target profile from an analysis of the filesystem.
>>
>> Any thoughts ?
> 
> I still don't understand the use case you are trying to support.
> 
> There are 3 states for a btrfs filesystem:
> 
[...]
> 
> 	3.  A conversion is interrupted prior to completion.  Sysadmin is
> 	expected to proceed immediately back to state #2, possibly after
> 	taking any necessary recovery actions that triggered entry into
> 	state #3.  It doesn't really matter what the current allocation
> 	profile is, since it is likely to change before we allocate
> 	any more block groups.
> 
> You seem to be trying to sustain or support a filesystem in state #3 for
> a prolonged period of time.  Why would we do that?  If your use case is
> providing information or guidance to a user, tell them how to get back
> to state #2 ASAP, so that they can then return to state #1 where they
> should be.

Believe me: I *don't want* to sustain #3 at all; btrfs is already too
complex. Supporting multiple profile is the worst thing that we can do.
However #3 exists and it could cause unexpected results. I think that on
this we agree.
> 
[...]

> There could be a warning message in dmesg if we enter state #3.
> This message would appear after a converting balance is cancelled or
> aborted, and on mount when we scan block groups (which we would still need
> to do even after we added a "target profile" field to the superblock).
> Userspace like 'btrfs fi df' could also put out a warning like "multiple
> allocation profiles detected, but conversion is not in progress.  Please
> finish conversion at your earliest convenience to avoid disappointment."
> I don't see the need to do anything more about it.

It would help that every btrfs command should warn the users about an
"un-wanted" state like this.

> 
> We only get to state #3 if the automation has already failed, or has
> been explicitly cancelled at sysadmin request.  
>
Not only, you can enter in state #3 if you do something like:

$ sudo btrfs balance start -dconvert=single,usage=50 t/.

where you convert some chunk but not other.

This is the point: we can consider the "failed automation" an unexpected
event, however doing "btrfs bal stop" or the command above cannot be
considered as unexpected event.

[...]

> I'd even consider removing the heuristics that are already there for
> prioritizing profiles.  They are just surprising and undocumented
> behavior, and it would be better to document it as "random, BTW you
> should finish your conversion now."

I agree that we should remove this kind of heuristic.
Doing so I think that, with moderate effort, btrfs can track what is the
wanted profile (i.e. the one at the mkfs time or the one specified in last balance
w/convert [*]) and uses it. To me it seems the natural thing to do. Noting more
nothing less.
We can't prevent a mixed profile filesystem (it has to be allowed the
possibility to stop a long activity like the balance), but we should
prevent the unexpected behavior: if we change the profile and something
goes wrong, the next chunk allocation should be clear. The user don't have
to read the code to understand what will happen.
  [*] we can argue which would be the expected profile after an interrupted balance:
the former one or the latter one ?


BR
G.Baroncelli

-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: Question: how understand the raid profile of a btrfs filesystem

[Graham Cobb]

On 23/03/2020 20:50, Goffredo Baroncelli wrote:
> On 3/23/20 12:49 AM, Zygo Blaxell wrote:
>> On Sun, Mar 22, 2020 at 09:38:30AM +0100, Goffredo Baroncelli wrote:
>>> On 3/22/20 9:34 AM, Goffredo Baroncelli wrote:
>>>
>>>>
>>>> To me it seems complicated to
>>> [sorry I push the send button too early]
>>>
>>> To me it seems too complicated (and error prone) to derive the target
>>> profile from an analysis of the filesystem.
>>>
>>> Any thoughts ?
>>
>> I still don't understand the use case you are trying to support.
>>
>> There are 3 states for a btrfs filesystem:
>>
> [...]
>>
>>     3.  A conversion is interrupted prior to completion.  Sysadmin is
>>     expected to proceed immediately back to state #2, possibly after
>>     taking any necessary recovery actions that triggered entry into
>>     state #3.  It doesn't really matter what the current allocation
>>     profile is, since it is likely to change before we allocate
>>     any more block groups.
>>
>> You seem to be trying to sustain or support a filesystem in state #3 for
>> a prolonged period of time.  Why would we do that?  

In real life situations (particularly outside a commercial datacentre)
this situation can persist for quite a while.  I recently found myself
in a real-life situation where this situation was not only in existence
for weeks but was, at some times, getting worse (I was getting further
away from my target configuration, not closer).

In this case, the original trigger was a disk in a well over 10TB
filesystem beginning to go bad. My strategy for handling that was to
replace the failing disk asap, and then rearrange the disk usage on the
system later. In order to handle the immediate emergency, I made use of
existing free space in LVM volume groups to replace the failing disk,
but that meant I had some user data and backups on the same physical
disk for a while (although I have plenty of other backups available I
like to keep my first-tier backups on separate local disks).

So, once the immediate crisis was over, I needed to move disks around
between the filesystems. It was weeks before I had managed to do
sufficient disk adds, removes and replaces to have all the filesystems
back to having data and backups on separate disks and all the data and
metadata in the profiles I wanted. Just doing a replace for one disk
took many days for the system to physically copy the data from one disk
to the other.

As this system was still in heavy use, this was made worse by btrfs
deciding to store data in profiles I did not want (at that point in the
manipulation) and forcing me to rebalance the data that had been written
during the last disk change before I could start on the next one.

Bottom line: although not the top priority in btrfs development, a
simple way to control the profile to be used for new data and metadata
allocations would have real benefit to overstretched sysadmins.

Re: Question: how understand the raid profile of a btrfs filesystem

[Zygo Blaxell]

On Mon, Mar 23, 2020 at 09:50:03PM +0100, Goffredo Baroncelli wrote:
> On 3/23/20 12:49 AM, Zygo Blaxell wrote:
> > On Sun, Mar 22, 2020 at 09:38:30AM +0100, Goffredo Baroncelli wrote:
> > > On 3/22/20 9:34 AM, Goffredo Baroncelli wrote:
> > > 
> > > > 
> > > > To me it seems complicated to
> > > [sorry I push the send button too early]
> > > 
> > > To me it seems too complicated (and error prone) to derive the target profile from an analysis of the filesystem.
> > > 
> > > Any thoughts ?
> > 
> > I still don't understand the use case you are trying to support.
> > 
> > There are 3 states for a btrfs filesystem:
> > 
> [...]
> > 
> > 	3.  A conversion is interrupted prior to completion.  Sysadmin is
> > 	expected to proceed immediately back to state #2, possibly after
> > 	taking any necessary recovery actions that triggered entry into
> > 	state #3.  It doesn't really matter what the current allocation
> > 	profile is, since it is likely to change before we allocate
> > 	any more block groups.
> > 
> > You seem to be trying to sustain or support a filesystem in state #3 for
> > a prolonged period of time.  Why would we do that?  If your use case is
> > providing information or guidance to a user, tell them how to get back
> > to state #2 ASAP, so that they can then return to state #1 where they
> > should be.
> 
> Believe me: I *don't want* to sustain #3 at all; btrfs is already too
> complex. Supporting multiple profile is the worst thing that we can do.
> However #3 exists and it could cause unexpected results. I think that on
> this we agree.
> > 
> [...]
> 
> > There could be a warning message in dmesg if we enter state #3.
> > This message would appear after a converting balance is cancelled or
> > aborted, and on mount when we scan block groups (which we would still need
> > to do even after we added a "target profile" field to the superblock).
> > Userspace like 'btrfs fi df' could also put out a warning like "multiple
> > allocation profiles detected, but conversion is not in progress.  Please
> > finish conversion at your earliest convenience to avoid disappointment."
> > I don't see the need to do anything more about it.
> 
> It would help that every btrfs command should warn the users about an
> "un-wanted" state like this.

Patches welcome...

> > been explicitly cancelled at sysadmin request.
> > 
> Not only, you can enter in state #3 if you do something like:
> 
> $ sudo btrfs balance start -dconvert=single,usage=50 t/.
> 
> where you convert some chunk but not other.

Sure, but now you're intentionally doing weird (or sufficiently advanced)
stuff.  Given a combination of balance flags like that (convert + other
restrictions), we should assume the user knows what they're doing, and
stay out of the way.

The existing code that inserts 'usage=90' when resuming a balance,
though highly questionable, still presumes the user knows what they're
doing when a balance has a convert in it, and doesn't modify the usage
filter setting in that case.

It's fairly normal to want to run something like this when changing
RAID profiles on a big array:

	# Make lots of free space quickly
	for x in $(seq 0 100); do
		btrfs balance start -dconvert=single,soft,usage=$x t/.
	done
	# OK now do the full BGs, will be slow
	btrfs balance start -dconvert=single,soft t/.

Should that print 101 warnings as it runs?  What if the user is using
python-btrfs (e.g. to order the block groups by usage) and not the
btrfs-progs tools, or some other UI?  Do we write warnings from inside
the kernel?  Will there be a "--quiet" option that suppresses the warning?
(I suppose if the answer to the last two questions is "yes" then we just
need patches to get it done).

> This is the point: we can consider the "failed automation" an unexpected
> event, however doing "btrfs bal stop" or the command above cannot be
> considered as unexpected event.

Balance cancel is always unexpected.

"balance cancel" is a sysadmin forcing balance to exit using the error
recovery code.  If early termination of a conversion was _expected_,
the sysadmin would have used 'limit' or 'vrange' or 'usage' or 'devid'
or some other filter parameter so that balance does what it was told
to do _without being cancelled_.

> [...]
> 
> > I'd even consider removing the heuristics that are already there for
> > prioritizing profiles.  They are just surprising and undocumented
> > behavior, and it would be better to document it as "random, BTW you
> > should finish your conversion now."
> 
> I agree that we should remove this kind of heuristic.
> Doing so I think that, with moderate effort, btrfs can track what is the
> wanted profile (i.e. the one at the mkfs time or the one specified in last balance
> w/convert [*]) and uses it. To me it seems the natural thing to do. Noting more
> nothing less.

It already kind of does--the balance convert parameters are stored on
disk so it can be resumed after a umount or pause.  "Pause" implies
resuming later, and saving all the state required to do so.  "Cancel"
says something different, "forget what you were doing and wait for new
instructions," so cancel wipes out the conversion target profile.

> We can't prevent a mixed profile filesystem (it has to be allowed the
> possibility to stop a long activity like the balance), but we should
> prevent the unexpected behavior: if we change the profile and something
> goes wrong, the next chunk allocation should be clear. The user don't have
> to read the code to understand what will happen.
>  [*] we can argue which would be the expected profile after an interrupted balance:
> the former one or the latter one ?

If we can argue about it, then there's no right answer, and the status
quo is fine (or we need a more complete solution).

> BR
> G.Baroncelli
> 
> -- 
> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: Question: how understand the raid profile of a btrfs filesystem

[Anand Jain]

On 3/21/20 1:56 AM, Goffredo Baroncelli wrote:
> Hi all,
> 
> for a btrfs filesystem, how an user can understand which is the 
> {data,mmetadata,system} [raid] profile in use ? E.g. the next chunk 
> which profile will have ?
> For simple filesystem it is easy looking at the output of (e.g)  "btrfs 
> fi df" or "btrfs fi us". But what if the filesystem is not simple ?
> 
> btrfs fi us t/.
> Overall:
>      Device size:          40.00GiB
>      Device allocated:          19.52GiB
>      Device unallocated:          20.48GiB
>      Device missing:             0.00B
>      Used:              16.75GiB
>      Free (estimated):          12.22GiB    (min: 8.27GiB)
>      Data ratio:                  1.90
>      Metadata ratio:              2.00
>      Global reserve:           9.06MiB    (used: 0.00B)
> 
> Data,single: Size:1.00GiB, Used:512.00MiB (50.00%)
>     /dev/loop0       1.00GiB
> 
> Data,RAID5: Size:3.00GiB, Used:2.48GiB (82.56%)
>     /dev/loop1       1.00GiB
>     /dev/loop2       1.00GiB
>     /dev/loop3       1.00GiB
>     /dev/loop0       1.00GiB
> 
> Data,RAID6: Size:4.00GiB, Used:3.71GiB (92.75%)
>     /dev/loop1       2.00GiB
>     /dev/loop2       2.00GiB
>     /dev/loop3       2.00GiB
>     /dev/loop0       2.00GiB
> 
> Data,RAID1C3: Size:2.00GiB, Used:1.88GiB (93.76%)
>     /dev/loop1       2.00GiB
>     /dev/loop2       2.00GiB
>     /dev/loop3       2.00GiB
> 
> Metadata,RAID1: Size:256.00MiB, Used:9.14MiB (3.57%)
>     /dev/loop2     256.00MiB
>     /dev/loop3     256.00MiB
> 
> System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%)
>     /dev/loop2       8.00MiB
>     /dev/loop3       8.00MiB
> 
> Unallocated:
>     /dev/loop1       5.00GiB
>     /dev/loop2       4.74GiB
>     /dev/loop3       4.74GiB
>     /dev/loop0       6.00GiB
> 
> This is an example of a strange but valid filesystem. So the question 
> is: the next chunk which profile will have ?
> Is there any way to understand what will happens ?
> 
> I expected that the next chunk will be allocated as the last "convert". 
> However I discovered that this is not true.
> 
> Looking at the code it seems to me that the logic is the following (from 
> btrfs_reduce_alloc_profile())
> 
>          if (allowed & BTRFS_BLOCK_GROUP_RAID6)
>                  allowed = BTRFS_BLOCK_GROUP_RAID6;
>          else if (allowed & BTRFS_BLOCK_GROUP_RAID5)
>                  allowed = BTRFS_BLOCK_GROUP_RAID5;
>          else if (allowed & BTRFS_BLOCK_GROUP_RAID10)
>                  allowed = BTRFS_BLOCK_GROUP_RAID10;
>          else if (allowed & BTRFS_BLOCK_GROUP_RAID1)
>                  allowed = BTRFS_BLOCK_GROUP_RAID1;
>          else if (allowed & BTRFS_BLOCK_GROUP_RAID0)
>                  allowed = BTRFS_BLOCK_GROUP_RAID0;
> 
>          flags &= ~BTRFS_BLOCK_GROUP_PROFILE_MASK;
> 
> So in the case above the profile will be RAID6. And in the general if a 
> RAID6 chunk is a filesystem, it wins !

  That's arbitrary and doesn't make sense to me, IMO mkfs should save
  default profile in the super-block (which can be changed using ioctl)
  and kernel can create chunks based on the default profile. This
  approach also fixes chunk size inconsistency between progs and kernel
  as reported/fixed here
    https://patchwork.kernel.org/patch/11431405/

Thanks, Anand

> But I am not sure.. Moreover I expected to see also reference to DUP 
> and/or RAID1C[34] ...
> 
> Does someone have any suggestion ?
> 
> BR
> G.Baroncelli
> 

Re: Question: how understand the raid profile of a btrfs filesystem

[Goffredo Baroncelli]

On 3/24/20 5:55 AM, Anand Jain wrote:
> On 3/21/20 1:56 AM, Goffredo Baroncelli wrote:
>> Hi all,
[..]
>> Looking at the code it seems to me that the logic is the following (from btrfs_reduce_alloc_profile())
>>
>>          if (allowed & BTRFS_BLOCK_GROUP_RAID6)
>>                  allowed = BTRFS_BLOCK_GROUP_RAID6;
>>          else if (allowed & BTRFS_BLOCK_GROUP_RAID5)
>>                  allowed = BTRFS_BLOCK_GROUP_RAID5;
>>          else if (allowed & BTRFS_BLOCK_GROUP_RAID10)
>>                  allowed = BTRFS_BLOCK_GROUP_RAID10;
>>          else if (allowed & BTRFS_BLOCK_GROUP_RAID1)
>>                  allowed = BTRFS_BLOCK_GROUP_RAID1;
>>          else if (allowed & BTRFS_BLOCK_GROUP_RAID0)
>>                  allowed = BTRFS_BLOCK_GROUP_RAID0;
>>
>>          flags &= ~BTRFS_BLOCK_GROUP_PROFILE_MASK;
>>
>> So in the case above the profile will be RAID6. And in the general if a RAID6 chunk is a filesystem, it wins !
> 
>   That's arbitrary and doesn't make sense to me, IMO mkfs should save
>   default profile in the super-block (which can be changed using ioctl)
>   and kernel can create chunks based on the default profile. 

I'm working on this idea (storing the target profile in super-block). Of course this increase the consistency, but
doesn't prevent the possibility that a mixed profiles filesystem could happen. And in this case is the user that
has to solve the issue.

Zygo, suggested also to add a mixed profile warning to btrfs (prog). And I agree with him. I think that we can use
the space info ioctl (which doesn't require root privileges).

BR
G.Baroncelli

> This
>   approach also fixes chunk size inconsistency between progs and kernel
>   as reported/fixed here
>     https://patchwork.kernel.org/patch/11431405/
> 
> Thanks, Anand
> 
>> But I am not sure.. Moreover I expected to see also reference to DUP and/or RAID1C[34] ...
>>
>> Does someone have any suggestion ?
>>
>> BR
>> G.Baroncelli
>>
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: Question: how understand the raid profile of a btrfs filesystem

[Andrei Borzenkov]

24.03.2020 20:59, Goffredo Baroncelli пишет:
> On 3/24/20 5:55 AM, Anand Jain wrote:
>> On 3/21/20 1:56 AM, Goffredo Baroncelli wrote:
>>> Hi all,
> [..]
>>> Looking at the code it seems to me that the logic is the following
>>> (from btrfs_reduce_alloc_profile())
>>>
>>>          if (allowed & BTRFS_BLOCK_GROUP_RAID6)
>>>                  allowed = BTRFS_BLOCK_GROUP_RAID6;
>>>          else if (allowed & BTRFS_BLOCK_GROUP_RAID5)
>>>                  allowed = BTRFS_BLOCK_GROUP_RAID5;
>>>          else if (allowed & BTRFS_BLOCK_GROUP_RAID10)
>>>                  allowed = BTRFS_BLOCK_GROUP_RAID10;
>>>          else if (allowed & BTRFS_BLOCK_GROUP_RAID1)
>>>                  allowed = BTRFS_BLOCK_GROUP_RAID1;
>>>          else if (allowed & BTRFS_BLOCK_GROUP_RAID0)
>>>                  allowed = BTRFS_BLOCK_GROUP_RAID0;
>>>
>>>          flags &= ~BTRFS_BLOCK_GROUP_PROFILE_MASK;
>>>
>>> So in the case above the profile will be RAID6. And in the general if
>>> a RAID6 chunk is a filesystem, it wins !
>>
>>   That's arbitrary and doesn't make sense to me, IMO mkfs should save
>>   default profile in the super-block (which can be changed using ioctl)
>>   and kernel can create chunks based on the default profile. 
> 
> I'm working on this idea (storing the target profile in super-block).

What about per-subvolume profile? This comes up every now and then, like

https://lore.kernel.org/linux-btrfs/cd82d247-5c95-18cd-a290-a911ff69613c@dirtcellar.net/

May be it could be subvolume property?

> Of
> course this increase the consistency, but
> doesn't prevent the possibility that a mixed profiles filesystem could
> happen. And in this case is the user that
> has to solve the issue.
> 
> Zygo, suggested also to add a mixed profile warning to btrfs (prog). And
> I agree with him. I think that we can use
> the space info ioctl (which doesn't require root privileges).
> 
> BR
> G.Baroncelli
> 
>> This
>>   approach also fixes chunk size inconsistency between progs and kernel
>>   as reported/fixed here
>>     https://patchwork.kernel.org/patch/11431405/
>>
>> Thanks, Anand
>>
>>> But I am not sure.. Moreover I expected to see also reference to DUP
>>> and/or RAID1C[34] ...
>>>
>>> Does someone have any suggestion ?
>>>
>>> BR
>>> G.Baroncelli
>>>
>>
> 
> 

Re: Question: how understand the raid profile of a btrfs filesystem

[Zygo Blaxell]

On Mon, Mar 23, 2020 at 10:48:44PM +0000, Graham Cobb wrote:
> On 23/03/2020 20:50, Goffredo Baroncelli wrote:
> > On 3/23/20 12:49 AM, Zygo Blaxell wrote:
> >> On Sun, Mar 22, 2020 at 09:38:30AM +0100, Goffredo Baroncelli wrote:
> >>> On 3/22/20 9:34 AM, Goffredo Baroncelli wrote:
> >>>
> >>>>
> >>>> To me it seems complicated to
> >>> [sorry I push the send button too early]
> >>>
> >>> To me it seems too complicated (and error prone) to derive the target
> >>> profile from an analysis of the filesystem.
> >>>
> >>> Any thoughts ?
> >>
> >> I still don't understand the use case you are trying to support.
> >>
> >> There are 3 states for a btrfs filesystem:
> >>
> > [...]
> >>
> >>     3.  A conversion is interrupted prior to completion.  Sysadmin is
> >>     expected to proceed immediately back to state #2, possibly after
> >>     taking any necessary recovery actions that triggered entry into
> >>     state #3.  It doesn't really matter what the current allocation
> >>     profile is, since it is likely to change before we allocate
> >>     any more block groups.
> >>
> >> You seem to be trying to sustain or support a filesystem in state #3 for
> >> a prolonged period of time.  Why would we do that?  
> 
> In real life situations (particularly outside a commercial datacentre)
> this situation can persist for quite a while.  I recently found myself
> in a real-life situation where this situation was not only in existence
> for weeks but was, at some times, getting worse (I was getting further
> away from my target configuration, not closer).
> 
> In this case, the original trigger was a disk in a well over 10TB
> filesystem beginning to go bad. My strategy for handling that was to
> replace the failing disk asap, and then rearrange the disk usage on the
> system later. In order to handle the immediate emergency, I made use of
> existing free space in LVM volume groups to replace the failing disk,
> but that meant I had some user data and backups on the same physical
> disk for a while (although I have plenty of other backups available I
> like to keep my first-tier backups on separate local disks).

I've done those.  And the annoying thing about them was...

> So, once the immediate crisis was over, I needed to move disks around
> between the filesystems. It was weeks before I had managed to do
> sufficient disk adds, removes 

Disk removes are where the current system breaks down.  'btrfs device
remove' is terrible:

	- can't cancel a remove except by rebooting or forcing ENOSPC

	- can't resume automatically after a reboot (probably a good
	thing for now, given there's no cancel)

	- can't coexist with a balance, even when paused--device remove
	requires the balance to be _cancelled_ first

	- doesn't have any equivalent to the 'convert' filter raid
	profile target in balance info

so if you need to remove a device while you're changing profiles, you
have to abort the profile change and then relocate a whole lot of data
without being able to specify the correct target profile.

The proper fix would be to reimplement 'btrfs dev remove' using pieces of
the balance infrastructure (it kind of is now, except where it's not),
and so 'device remove' can keep the 'convert=' target.  Then you don't
have to lose the target profile while doing removes (and fix the other
problems too).

Or just move it from the balance info to the superblock, as suggested
elsewhere in the thread (none of these changes can be done without
changing something in the on-disk format).  But definitely don't have
the target profile in both places!

> and replaces to have all the filesystems
> back to having data and backups on separate disks and all the data and
> metadata in the profiles I wanted. Just doing a replace for one disk
> took many days for the system to physically copy the data from one disk
> to the other.
> 
> As this system was still in heavy use, this was made worse by btrfs
> deciding to store data in profiles I did not want (at that point in the
> manipulation) and forcing me to rebalance the data that had been written
> during the last disk change before I could start on the next one.
> 
> Bottom line: although not the top priority in btrfs development, a
> simple way to control the profile to be used for new data and metadata
> allocations would have real benefit to overstretched sysadmins.
> 

RE: Question: how understand the raid profile of a btrfs filesystem

[Paul Jones]

> -----Original Message-----
> From: linux-btrfs-owner@vger.kernel.org <linux-btrfs-
> owner@vger.kernel.org> On Behalf Of Zygo Blaxell
> Sent: Wednesday, 25 March 2020 3:10 PM
> To: Graham Cobb <g.btrfs@cobb.uk.net>
> Cc: linux-btrfs <linux-btrfs@vger.kernel.org>
> Subject: Re: Question: how understand the raid profile of a btrfs filesystem

> Disk removes are where the current system breaks down.  'btrfs device
> remove' is terrible:
> 
> 	- can't cancel a remove except by rebooting or forcing ENOSPC
> 
> 	- can't resume automatically after a reboot (probably a good
> 	thing for now, given there's no cancel)
> 
> 	- can't coexist with a balance, even when paused--device remove
> 	requires the balance to be _cancelled_ first
> 
> 	- doesn't have any equivalent to the 'convert' filter raid
> 	profile target in balance info
> 
> so if you need to remove a device while you're changing profiles, you have to
> abort the profile change and then relocate a whole lot of data without being
> able to specify the correct target profile.
> 
> The proper fix would be to reimplement 'btrfs dev remove' using pieces of
> the balance infrastructure (it kind of is now, except where it's not), and so
> 'device remove' can keep the 'convert=' target.  Then you don't have to lose
> the target profile while doing removes (and fix the other problems too).

I've often thought it would be handy to be able to forcefully set the disk size or free space to zero, like how it is reported by 'btrfs fi sh' during a remove operation. That way a balance operation can be used for various things like profile changes or multiple disk removals (like replacing 4x1T drives with 1x4T drive) without unintentionally writing a bunch of data to a disk you don't want to write to anymore.
It would also allow for a more gradual removal for disks that need replacing but not as an emergency, as data will gradually migrate itself to other discs as it is COWed.

Paul.

Re: Question: how understand the raid profile of a btrfs filesystem

[Goffredo Baroncelli]

On 3/25/20 5:09 AM, Andrei Borzenkov wrote:
> 24.03.2020 20:59, Goffredo Baroncelli пишет:
>> On 3/24/20 5:55 AM, Anand Jain wrote:
>>> On 3/21/20 1:56 AM, Goffredo Baroncelli wrote:
>>>> Hi all,
>> [..]
>>>> Looking at the code it seems to me that the logic is the following
>>>> (from btrfs_reduce_alloc_profile())
>>>>
>>>>           if (allowed & BTRFS_BLOCK_GROUP_RAID6)
>>>>                   allowed = BTRFS_BLOCK_GROUP_RAID6;
>>>>           else if (allowed & BTRFS_BLOCK_GROUP_RAID5)
>>>>                   allowed = BTRFS_BLOCK_GROUP_RAID5;
>>>>           else if (allowed & BTRFS_BLOCK_GROUP_RAID10)
>>>>                   allowed = BTRFS_BLOCK_GROUP_RAID10;
>>>>           else if (allowed & BTRFS_BLOCK_GROUP_RAID1)
>>>>                   allowed = BTRFS_BLOCK_GROUP_RAID1;
>>>>           else if (allowed & BTRFS_BLOCK_GROUP_RAID0)
>>>>                   allowed = BTRFS_BLOCK_GROUP_RAID0;
>>>>
>>>>           flags &= ~BTRFS_BLOCK_GROUP_PROFILE_MASK;
>>>>
>>>> So in the case above the profile will be RAID6. And in the general if
>>>> a RAID6 chunk is a filesystem, it wins !
>>>
>>>    That's arbitrary and doesn't make sense to me, IMO mkfs should save
>>>    default profile in the super-block (which can be changed using ioctl)
>>>    and kernel can create chunks based on the default profile.
>>
>> I'm working on this idea (storing the target profile in super-block).
> 
> What about per-subvolume profile? This comes up every now and then, like
> 
> https://lore.kernel.org/linux-btrfs/cd82d247-5c95-18cd-a290-a911ff69613c@dirtcellar.net/
> 
> May be it could be subvolume property?


The idea is nice. However I fear the mess that it could cause. Even now, with a
more simpler system where there is a "per filesystem" profile, there are a lot of corner
cases when something goes wrong (an interrupted balance, or a disk failed).
In case of multiple profiles on sub-volume basis there is no simple answer in situation like:
- when I make a snapshot of a sub-volumes, and then I change the profile of the original one,
which is the profile of the files contained in the snapshot and in the original subvolumes ?

Frankly speaking, if you want different profiles you need different filesystem...

BR
G.Baroncelli

> 
>> Of
>> course this increase the consistency, but
>> doesn't prevent the possibility that a mixed profiles filesystem could
>> happen. And in this case is the user that
>> has to solve the issue.
>>
>> Zygo, suggested also to add a mixed profile warning to btrfs (prog). And
>> I agree with him. I think that we can use
>> the space info ioctl (which doesn't require root privileges).
>>
>> BR
>> G.Baroncelli
>>
>>> This
>>>    approach also fixes chunk size inconsistency between progs and kernel
>>>    as reported/fixed here
>>>      https://patchwork.kernel.org/patch/11431405/
>>>
>>> Thanks, Anand
>>>
>>>> But I am not sure.. Moreover I expected to see also reference to DUP
>>>> and/or RAID1C[34] ...
>>>>
>>>> Does someone have any suggestion ?
>>>>
>>>> BR
>>>> G.Baroncelli
>>>>
>>>
>>
>>
> 


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Re: Question: how understand the raid profile of a btrfs filesystem

[Zygo Blaxell]

On Wed, Mar 25, 2020 at 04:30:16AM +0000, Paul Jones wrote:
> > -----Original Message-----
> > From: linux-btrfs-owner@vger.kernel.org <linux-btrfs-
> > owner@vger.kernel.org> On Behalf Of Zygo Blaxell
> > Sent: Wednesday, 25 March 2020 3:10 PM
> > To: Graham Cobb <g.btrfs@cobb.uk.net>
> > Cc: linux-btrfs <linux-btrfs@vger.kernel.org>
> > Subject: Re: Question: how understand the raid profile of a btrfs filesystem
> 
> > Disk removes are where the current system breaks down.  'btrfs device
> > remove' is terrible:
> > 
> > 	- can't cancel a remove except by rebooting or forcing ENOSPC
> > 
> > 	- can't resume automatically after a reboot (probably a good
> > 	thing for now, given there's no cancel)
> > 
> > 	- can't coexist with a balance, even when paused--device remove
> > 	requires the balance to be _cancelled_ first
> > 
> > 	- doesn't have any equivalent to the 'convert' filter raid
> > 	profile target in balance info
> > 
> > so if you need to remove a device while you're changing profiles, you have to
> > abort the profile change and then relocate a whole lot of data without being
> > able to specify the correct target profile.
> > 
> > The proper fix would be to reimplement 'btrfs dev remove' using pieces of
> > the balance infrastructure (it kind of is now, except where it's not), and so
> > 'device remove' can keep the 'convert=' target.  Then you don't have to lose
> > the target profile while doing removes (and fix the other problems too).
> 
> I've often thought it would be handy to be able to forcefully set the
> disk size or free space to zero, like how it is reported by 'btrfs
> fi sh' during a remove operation. That way a balance operation can be
> used for various things like profile changes or multiple disk removals
> (like replacing 4x1T drives with 1x4T drive) without unintentionally
> writing a bunch of data to a disk you don't want to write to anymore.

I forgot "can only remove one disk at a time" in the list above.  We can
add multiple disks at once (well, add one at a time, then use balance to
do all the relocation at once), but the opposite operation isn't possible.

That is an elegant way to set up balances to do a device delete/shrink,
too.

> It would also allow for a more gradual removal for disks that need
> replacing but not as an emergency, as data will gradually migrate
> itself to other discs as it is COWed.
> 
> Paul.

Re: Question: how understand the raid profile of a btrfs filesystem

[Zygo Blaxell]

On Wed, Mar 25, 2020 at 06:14:05PM +0100, Goffredo Baroncelli wrote:
> On 3/25/20 5:09 AM, Andrei Borzenkov wrote:
> > 24.03.2020 20:59, Goffredo Baroncelli пишет:
> > > On 3/24/20 5:55 AM, Anand Jain wrote:
> > > > On 3/21/20 1:56 AM, Goffredo Baroncelli wrote:
> > > > > Hi all,
> > > [..]
> > > > > Looking at the code it seems to me that the logic is the following
> > > > > (from btrfs_reduce_alloc_profile())
> > > > > 
> > > > >           if (allowed & BTRFS_BLOCK_GROUP_RAID6)
> > > > >                   allowed = BTRFS_BLOCK_GROUP_RAID6;
> > > > >           else if (allowed & BTRFS_BLOCK_GROUP_RAID5)
> > > > >                   allowed = BTRFS_BLOCK_GROUP_RAID5;
> > > > >           else if (allowed & BTRFS_BLOCK_GROUP_RAID10)
> > > > >                   allowed = BTRFS_BLOCK_GROUP_RAID10;
> > > > >           else if (allowed & BTRFS_BLOCK_GROUP_RAID1)
> > > > >                   allowed = BTRFS_BLOCK_GROUP_RAID1;
> > > > >           else if (allowed & BTRFS_BLOCK_GROUP_RAID0)
> > > > >                   allowed = BTRFS_BLOCK_GROUP_RAID0;
> > > > > 
> > > > >           flags &= ~BTRFS_BLOCK_GROUP_PROFILE_MASK;
> > > > > 
> > > > > So in the case above the profile will be RAID6. And in the general if
> > > > > a RAID6 chunk is a filesystem, it wins !
> > > > 
> > > >    That's arbitrary and doesn't make sense to me, IMO mkfs should save
> > > >    default profile in the super-block (which can be changed using ioctl)
> > > >    and kernel can create chunks based on the default profile.
> > > 
> > > I'm working on this idea (storing the target profile in super-block).
> > 
> > What about per-subvolume profile? This comes up every now and then, like
> > 
> > https://lore.kernel.org/linux-btrfs/cd82d247-5c95-18cd-a290-a911ff69613c@dirtcellar.net/
> > 
> > May be it could be subvolume property?

...or inode.

> The idea is nice. However I fear the mess that it could cause. Even now, with a
> more simpler system where there is a "per filesystem" profile, there are a lot of corner
> cases when something goes wrong (an interrupted balance, or a disk failed).

It can't be worse than qgroups.  (only half kidding)

Thinking aloud, you could even set up coarse-but-fast quotas that
way--limit the number of data block groups allocated to a subvol.
No sharing of block groups between subvols though, unless one subvol is a
snapshot of the other.  Also, limiting usage by block group includes free
space within the block group, so it's inaccurate (i.e. coarse, effectively
allocating space with multi-GB granularity and large error bars).
If you have 20 users, and you want to give them each about 400GB but
don't really care if they get 390GB or 410GB, then maybe it's not so bad.

> In case of multiple profiles on sub-volume basis there is no simple answer in situation like:
> - when I make a snapshot of a sub-volumes, and then I change the profile of the original one,
> which is the profile of the files contained in the snapshot and in the original subvolumes ?

It shouldn't be different from compress:  you look up either the inode
or the root, and it tells you what kind of extent you can allocate next.
Any existing data stays where it is until it is deleted (or overwritten
by CoW).  If you start cloning between subvols then things get a
little interesting (especially if you balance those afterwards) but not
unsolvable if "when two or more answers are possible, it's undefined
which one btrfs picks" is allowed in the solution.

You'd have the same problem with no-longer-allocatable block groups that
don't match the currently selected profile as you do now with mixed
block group profiles.  As the unallocatable block groups empty out,
the storage density of the used space within them goes up, space appears
to disappear, etc.  This is state #3, after all, and it would take some
work to make btrfs as happy in this state as it is in state #1.

> Frankly speaking, if you want different profiles you need different filesystem...

Well, there is that.  Keeping the status quo (or small modifications
thereof) is far easier to document, and it's not like we don't have a
huge list of RAID-related things to fix already.

> BR
> G.Baroncelli
> 
> > 
> > > Of
> > > course this increase the consistency, but
> > > doesn't prevent the possibility that a mixed profiles filesystem could
> > > happen. And in this case is the user that
> > > has to solve the issue.
> > > 
> > > Zygo, suggested also to add a mixed profile warning to btrfs (prog). And
> > > I agree with him. I think that we can use
> > > the space info ioctl (which doesn't require root privileges).
> > > 
> > > BR
> > > G.Baroncelli
> > > 
> > > > This
> > > >    approach also fixes chunk size inconsistency between progs and kernel
> > > >    as reported/fixed here
> > > >      https://patchwork.kernel.org/patch/11431405/
> > > > 
> > > > Thanks, Anand
> > > > 
> > > > > But I am not sure.. Moreover I expected to see also reference to DUP
> > > > > and/or RAID1C[34] ...
> > > > > 
> > > > > Does someone have any suggestion ?
> > > > > 
> > > > > BR
> > > > > G.Baroncelli
> > > > > 
> > > > 
> > > 
> > > 
> > 
> 
> 
> -- 
> gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
> Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5

Question: how understand the raid profile of a btrfs filesystem

[Goffredo Baroncelli]

Hi all,

for a btrfs filesystem, how an user can understand which is the {data,mmetadata,system} [raid] profile in use ? E.g. the next chunk which profile will have ?
For simple filesystem it is easy looking at the output of (e.g)  "btrfs fi df" or "btrfs fi us". But what if the filesystem is not simple ?

btrfs fi us t/.
Overall:
     Device size:		  40.00GiB
     Device allocated:		  19.52GiB
     Device unallocated:		  20.48GiB
     Device missing:		     0.00B
     Used:			  16.75GiB
     Free (estimated):		  12.22GiB	(min: 8.27GiB)
     Data ratio:			      1.90
     Metadata ratio:		      2.00
     Global reserve:		   9.06MiB	(used: 0.00B)

Data,single: Size:1.00GiB, Used:512.00MiB (50.00%)
    /dev/loop0	   1.00GiB

Data,RAID5: Size:3.00GiB, Used:2.48GiB (82.56%)
    /dev/loop1	   1.00GiB
    /dev/loop2	   1.00GiB
    /dev/loop3	   1.00GiB
    /dev/loop0	   1.00GiB

Data,RAID6: Size:4.00GiB, Used:3.71GiB (92.75%)
    /dev/loop1	   2.00GiB
    /dev/loop2	   2.00GiB
    /dev/loop3	   2.00GiB
    /dev/loop0	   2.00GiB

Data,RAID1C3: Size:2.00GiB, Used:1.88GiB (93.76%)
    /dev/loop1	   2.00GiB
    /dev/loop2	   2.00GiB
    /dev/loop3	   2.00GiB

Metadata,RAID1: Size:256.00MiB, Used:9.14MiB (3.57%)
    /dev/loop2	 256.00MiB
    /dev/loop3	 256.00MiB

System,RAID1: Size:8.00MiB, Used:16.00KiB (0.20%)
    /dev/loop2	   8.00MiB
    /dev/loop3	   8.00MiB

Unallocated:
    /dev/loop1	   5.00GiB
    /dev/loop2	   4.74GiB
    /dev/loop3	   4.74GiB
    /dev/loop0	   6.00GiB

This is an example of a strange but valid filesystem. So the question is: the next chunk which profile will have ?
Is there any way to understand what will happens ?

I expected that the next chunk will be allocated as the last "convert". However I discovered that this is not true.

Looking at the code it seems to me that the logic is the following (from btrfs_reduce_alloc_profile())

         if (allowed & BTRFS_BLOCK_GROUP_RAID6)
                 allowed = BTRFS_BLOCK_GROUP_RAID6;
         else if (allowed & BTRFS_BLOCK_GROUP_RAID5)
                 allowed = BTRFS_BLOCK_GROUP_RAID5;
         else if (allowed & BTRFS_BLOCK_GROUP_RAID10)
                 allowed = BTRFS_BLOCK_GROUP_RAID10;
         else if (allowed & BTRFS_BLOCK_GROUP_RAID1)
                 allowed = BTRFS_BLOCK_GROUP_RAID1;
         else if (allowed & BTRFS_BLOCK_GROUP_RAID0)
                 allowed = BTRFS_BLOCK_GROUP_RAID0;

         flags &= ~BTRFS_BLOCK_GROUP_PROFILE_MASK;

So in the case above the profile will be RAID6. And in the general if a RAID6 chunk is a filesystem, it wins !

But I am not sure.. Moreover I expected to see also reference to DUP and/or RAID1C[34] ...

Does someone have any suggestion ?

BR
G.Baroncelli


-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5