Switch raid mode without rebalance?

Switch raid mode without rebalance?

[Gert Menke]

Hi,

I recently created a new btrfs on two disks - one 6TB, one 2TB - for 
temporary backup purposes.
It apparently defaulted to raid0 for data, and I didn't realize at the 
time that this would become a problem.
Now the 2TB is almost full, and df tells me I only have about 200GB of 
free space. Which makes sense, because raid0 spreads the data evenly on 
all disks.
However, I know that btrfs can have different raid modes on the same 
filesystem at the same time. So I was hoping I could just tell it to 
"switch to single mode for all new data", but I don't have a clue how to 
do that. I *could* rebalance, of course, but is that really necessary? 
How does btrfs find out which raid mode to use when writing new data?

Gert

Re: Switch raid mode without rebalance?

[Justin Kilpatrick]

I don't think its possible to change raid levels without using a balance.

I'm not sure why you want to avoid a balance, its a background process that
will auto resume if interrupted. You don't have to leave your machine on for
weeks at a time and it shouldn't slow down an active application using the disks
very much if at all.

btrfs balance start -dconvert=single -mconvert=raid1 /mnt

Will keep your metadata on two disks so if one disk fails you at least know
what was lost. That's only an extra couple of gigs of space.

Once you start this command all the new data should follow the new rules.


On Thu, Aug 25, 2016 at 2:16 PM, Gert Menke <gert@menke.ac> wrote:
> Hi,
>
> I recently created a new btrfs on two disks - one 6TB, one 2TB - for
> temporary backup purposes.
> It apparently defaulted to raid0 for data, and I didn't realize at the time
> that this would become a problem.
> Now the 2TB is almost full, and df tells me I only have about 200GB of free
> space. Which makes sense, because raid0 spreads the data evenly on all
> disks.
> However, I know that btrfs can have different raid modes on the same
> filesystem at the same time. So I was hoping I could just tell it to "switch
> to single mode for all new data", but I don't have a clue how to do that. I
> *could* rebalance, of course, but is that really necessary? How does btrfs
> find out which raid mode to use when writing new data?
>
> Gert
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Switch raid mode without rebalance?

[Gert Menke]

Hi,

On 2016-08-25 20:26, Justin Kilpatrick wrote:

> I'm not sure why you want to avoid a balance,
I didn't check, but I imagined it would slow down my rsync 
significantly.

> Once you start this command all the new data should follow the new 
> rules.
Ah, now that's interesting.
When the balance is running, df shows 4TB free space; when I cancel the 
balance, the free space goes back to a few 100GB.

But if the balancing only happens when the disk would otherwise be idle, 
and the mere fact that there is a balance process running will cause the 
new data to be written in single mode, I'm all set. I would not even 
have to wait for the balance to finish after the rsync is done; I could 
just cancel it and unmount the disks. A bit hacky, but in this case 
totally acceptable.

Thanks!
Gert

Re: Switch raid mode without rebalance?

[Austin S. Hemmelgarn]

On 2016-08-25 18:32, Gert Menke wrote:
> Hi,
>
> On 2016-08-25 20:26, Justin Kilpatrick wrote:
>
>> I'm not sure why you want to avoid a balance,
> I didn't check, but I imagined it would slow down my rsync significantly.
It will slow it down, but I can't tell you exactly how much (there are 
too many variables to give a good answer in this case).
>
>> Once you start this command all the new data should follow the new rules.
> Ah, now that's interesting.
> When the balance is running, df shows 4TB free space; when I cancel the
> balance, the free space goes back to a few 100GB.
Regular 'df' isn't to be trusted when dealing with BTRFS, the only 
reason we report anything there is because many things break horribly if 
we don't.
>
> But if the balancing only happens when the disk would otherwise be idle,
> and the mere fact that there is a balance process running will cause the
> new data to be written in single mode, I'm all set. I would not even
> have to wait for the balance to finish after the rsync is done; I could
> just cancel it and unmount the disks. A bit hacky, but in this case
> totally acceptable.
The balancing operation runs in the foreground (unless you daemonize it 
in a new enough btrfs-progs), it isn't restricted to just using idle 
time, and as explained by Chris Murphy in the other branch of this 
thread, it's not flipping some binary switch telling the FS to write all 
new data in the new profile.

Additionally, while running with multiple profiles while not balancing 
should work, it's pretty much untested, and any number of things may 
break.  Assuming your two disks have similar latency and transfer speed, 
you're almost certainly better off just converting completely to single 
mode (which works like raid0, just at the chunk level instead of the 
block level).  Doing so will likely have near zero impact on 
performance, will also be easier to recover data from if one disk dies, 
and will avoid the mostly untested situation of running with multiple 
chunk profiles.

On a slightly separate note, if your doing backups more frequently than 
once a week, your probably better off just leaving the disks connected 
and running.  Regular load/unload cycles are generally harder on the 
mechanical components in modern disks than just leaving them running 24/7.

Re: Switch raid mode without rebalance?

[Gert Menke]

Hi,

On 2016-08-26 13:52, Austin S. Hemmelgarn wrote:

> Regular 'df' isn't to be trusted when dealing with BTRFS, the only
> reason we report anything there is because many things break horribly
> if we don't.
Yeah, I noticed. Seems to produce a reasonable guess, though.

> Additionally, while running with multiple profiles while not balancing
> should work, it's pretty much untested, and any number of things may
> break.
Oh. Good to know.

> Assuming your two disks have similar latency and transfer
> speed, you're almost certainly better off just converting completely
> to single mode (which works like raid0, just at the chunk level
> instead of the block level).
Okay, I see.

> On a slightly separate note, if your doing backups more frequently
> than once a week, your probably better off just leaving the disks
> connected and running.  Regular load/unload cycles are generally
> harder on the mechanical components in modern disks than just leaving
> them running 24/7.
True. A bit of context: I first want to make a full backup locally, then 
use the disks off-site for nightly incremental backups via internet.

Cheers
Gert

Re: Switch raid mode without rebalance?

[Chris Murphy]

On Thu, Aug 25, 2016 at 12:16 PM, Gert Menke <gert@menke.ac> wrote:

> However, I know that btrfs can have different raid modes on the same
> filesystem at the same time.

It's incidental right now. It's not something controllable or intended
to have enduring mixed profile block groups.

>So I was hoping I could just tell it to "switch
> to single mode for all new data", but I don't have a clue how to do that.

Such a switch doesn't exist, there's no way to define what files,
directories, or subvolumes, have what profiles.


>I
> *could* rebalance, of course, but is that really necessary?

Yes.

> How does btrfs
> find out which raid mode to use when writing new data?

That's kindof an interesting question. If you were to do 'btrfs
balance start -dconvert=single -mconvert=raid1' and very soon after
that do 'btrfs balance cancel' you'll end up with one or a few new
chunks with those profiles. When data is allocated to those chunks,
they will have those profile characteristics. When data is allocated
to old chunks that are still raid0, it will be raid0. The thing is,
you can't really tell or control what data will be placed in what
chunk. So it's plausible that some new data goes in old raid0 chunk,
and some old data goes in new single/raid1 chunks.


-- 
Chris Murphy

Re: Switch raid mode without rebalance?

[Gert Menke]

Hi,

On 2016-08-25 21:50, Chris Murphy wrote:
> It's incidental right now. It's not something controllable or intended
> to have enduring mixed profile block groups.
I see. (Kindof)

> Such a switch doesn't exist, there's no way to define what files,
> directories, or subvolumes, have what profiles.
Well it kind of does - a running balance process seems to have just that 
effect, it's just not persistent (and has the side effect of, well, 
balancing the existing data).

>> How does btrfs
>> find out which raid mode to use when writing new data?
> 
> That's kindof an interesting question. If you were to do 'btrfs
> balance start -dconvert=single -mconvert=raid1' and very soon after
> that do 'btrfs balance cancel' you'll end up with one or a few new
> chunks with those profiles. When data is allocated to those chunks,
> they will have those profile characteristics. When data is allocated
> to old chunks that are still raid0, it will be raid0. The thing is,
> you can't really tell or control what data will be placed in what
> chunk. So it's plausible that some new data goes in old raid0 chunk,
> and some old data goes in new single/raid1 chunks.

I'm not quite familiar with the concept of a chunk here.
Are chunks allocated for new data, or is the unallocated space divided 
into chunks, too?
In the former case, when creating a new chunk, does btrfs just look into 
a random already existing chunk and copy the raid mode from there?
In the latter case, could you (in theory) change the raid mode of all 
empty chunks only?

I know this is not an intended usage scenario; just being curious here.

Thanks!
Gert

Re: Switch raid mode without rebalance?

[Chris Murphy]

On Thu, Aug 25, 2016 at 4:33 PM, Gert Menke <gert@menke.ac> wrote:

>> Such a switch doesn't exist, there's no way to define what files,
>> directories, or subvolumes, have what profiles.
>
> Well it kind of does - a running balance process seems to have just that
> effect, it's just not persistent (and has the side effect of, well,
> balancing the existing data).

No, it's not a file, directory or subvolume specific command. It
applies to a whole volume.


>>> How does btrfs
>>> find out which raid mode to use when writing new data?
>>
>>
>> That's kindof an interesting question. If you were to do 'btrfs
>> balance start -dconvert=single -mconvert=raid1' and very soon after
>> that do 'btrfs balance cancel' you'll end up with one or a few new
>> chunks with those profiles. When data is allocated to those chunks,
>> they will have those profile characteristics. When data is allocated
>> to old chunks that are still raid0, it will be raid0. The thing is,
>> you can't really tell or control what data will be placed in what
>> chunk. So it's plausible that some new data goes in old raid0 chunk,
>> and some old data goes in new single/raid1 chunks.
>
>
> I'm not quite familiar with the concept of a chunk here.

Chunk is the same thing as a block group. And it's a collection of
extents. Basically all addresses used by Btrfs internally are logical
addresses, they don't map to sectors or devices. It's the chunk tree
which lists the chunks, that does the mapping from logical address to
device and sector. A file will have one logical address in Btrfs, but
that logical address can be looked up and return multiple devices and
sectors for where the file is physically located. So the raid stuff
happens at the chunk stripe level.



> Are chunks allocated for new data, or is the unallocated space divided into
> chunks, too?

Allocation happens on demand. Empty chunks are supposed to
automatically be deallocated, depending on the kernel version.


> In the former case, when creating a new chunk, does btrfs just look into a
> random already existing chunk and copy the raid mode from there?

The profile is a function of a chunk. On a two device raid0 Btrfs
volume, you'd see something like this for the chunk tree:

chunk tree
leaf 20987904 items 5 free space 15626 generation 5 owner 3
fs uuid 44d6f5e9-6be0-4472-9357-02bf1b1c3d99
chunk uuid 183d3ff9-377d-4afe-8621-d8bafdf06cf9
item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
dev item devid 1 total_bytes 10737418240 bytes used 2155872256
dev uuid d8ecbb0b-5b8d-49f7-a33e-2e3f1dc8d240
item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
dev item devid 2 total_bytes 10737418240 bytes used 2155872256
dev uuid d170bfba-8f7c-4154-8997-4d1117aeee32
item 2 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520) itemoff 15975 itemsize 112
chunk length 8388608 owner 2 stripe_len 65536
type SYSTEM|RAID1 num_stripes 2
stripe 0 devid 2 offset 1048576
dev uuid: d170bfba-8f7c-4154-8997-4d1117aeee32
stripe 1 devid 1 offset 20971520
dev uuid: d8ecbb0b-5b8d-49f7-a33e-2e3f1dc8d240
item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 29360128) itemoff 15863 itemsize 112
chunk length 1073741824 owner 2 stripe_len 65536
type METADATA|RAID1 num_stripes 2
stripe 0 devid 2 offset 9437184
dev uuid: d170bfba-8f7c-4154-8997-4d1117aeee32
stripe 1 devid 1 offset 29360128
dev uuid: d8ecbb0b-5b8d-49f7-a33e-2e3f1dc8d240
item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 1103101952) itemoff 15751 itemsize 112
chunk length 2147483648 owner 2 stripe_len 65536
type DATA|RAID0 num_stripes 2
stripe 0 devid 2 offset 1083179008
dev uuid: d170bfba-8f7c-4154-8997-4d1117aeee32
stripe 1 devid 1 offset 1103101952
dev uuid: d8ecbb0b-5b8d-49f7-a33e-2e3f1dc8d240


Here you can see there are three chunks, system, metadata, data.
System and metadata chunks have profile raid1, you can see they have
two stripes each (stripe 0 and stripe 1) which are mirrors on two
different devices. And data chunk is profile raid0, also two stripes
which are alternating stripe elements of 64KiB each across the two
devices.

If I add another file, I'll get another data chunk allocated, and
it'll be added to the chunk tree as item 5, and it'll have its own
physical  offset on each device.

item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 3250585600) itemoff 15639 itemsize 112
chunk length 2147483648 owner 2 stripe_len 65536
type DATA|RAID0 num_stripes 2
stripe 0 devid 2 offset 2156920832
dev uuid: d170bfba-8f7c-4154-8997-4d1117aeee32
stripe 1 devid 1 offset 2176843776
dev uuid: d8ecbb0b-5b8d-49f7-a33e-2e3f1dc8d240

That chunk is just a collection of data extents.

So the point now is, in order to change the profile of a chunk, it has
to be completely rewritten.

> In the latter case, could you (in theory) change the raid mode of all empty
> chunks only?

Nope. In theory, there are no empty chunks.

To do what you want is planned, with no work picked up yet as far as I
know. It'd probably involve some work to associate something like an
xattr to let the allocator know which profile the user wants for the
data, and then to allocate it to the proper existing chunk or create a
new chunk with that profile as needed.


-- 
Chris Murphy

Re: Switch raid mode without rebalance?

[Gert Menke]

Hi Chris,

first off, thank you for the detailled explanations!

On 2016-08-26 01:04, Chris Murphy wrote:
> No, it's not a file, directory or subvolume specific command. It
> applies to a whole volume.
You are right, but all I was after in the first place was a way to 
change the mode for new data on the whole volume.
But I understand now that there is no simple switch for that. (And to be 
honest it's not overly important, I was just being curious.)

> If I add another file, I'll get another data chunk allocated, and
> it'll be added to the chunk tree as item 5, and it'll have its own
> physical  offset on each device.
And this chunk just uses the same profile as the last one (or the parent 
in the tree), I suppose.

> So the point now is, in order to change the profile of a chunk, it has
> to be completely rewritten.
That makes sense.

> To do what you want is planned, with no work picked up yet as far as I
> know. It'd probably involve some work to associate something like an
> xattr to let the allocator know which profile the user wants for the
> data, and then to allocate it to the proper existing chunk or create a
> new chunk with that profile as needed.
I see.

I think it would be very nice to have something like that for the 
different compression modes. For example, use LZ4 for daily use but LZMA 
for the subvolume that stores backups, and no compression at all for 
/boot, so the bootloader doesn't have to know about all the different 
compression algorithms.

Speaking of which, I read here: 
https://btrfs.wiki.kernel.org/index.php/Compression#Why_does_not_du_report_the_compressed_size.3F
that du will not tell me the compressed size of a file. This is very 
counter-intuitive, isn't it?
The reason stated is that some tools apparently determine the 
sparse-ness of a file by comparing the size with the stat.st_blocks 
value.
I do not know if there is a better way to do that, so maybe my argument 
falls apart right here, BUT: this looks to me like working around one 
bug by introducing another. Wouldn't it be better to have a mount option 
"make_du_lie_for_buggy_tools" for those that really need it? BTW, which 
tools would an honest du break, and how? (What harm is there in thinking 
that a compressed file is sparse?)

Thanks!
Gert

Re: Switch raid mode without rebalance?

[Chris Murphy]

On Fri, Aug 26, 2016 at 1:18 PM, Gert Menke <gert@menke.ac> wrote:
> Hi Chris,
>
> first off, thank you for the detailled explanations!
>
> On 2016-08-26 01:04, Chris Murphy wrote:
>>
>> No, it's not a file, directory or subvolume specific command. It
>> applies to a whole volume.
>
> You are right, but all I was after in the first place was a way to change
> the mode for new data on the whole volume.

New chunks, yes in a rudimentary sense (btrfs balance start then
cancel with a dconvert/mconvert), but there's no user control to
provide a hint to the allocator which chunks to put new data into. It
can go into new chunks or old chunks.

Also, I *think* if you cancel the balance, a new balance is not going
to complete the conversion - it will do the balance maintaining the
mixed profile state of existing chunks. So you'd want to do something
like -dconvert=profile,soft -mconvert=profile,soft



>> If I add another file, I'll get another data chunk allocated, and
>> it'll be added to the chunk tree as item 5, and it'll have its own
>> physical  offset on each device.
>
> And this chunk just uses the same profile as the last one (or the parent in
> the tree), I suppose.

I honestly don't know how it determines what the profile will be; it
might be the profile of the chunk with the most recent generation.



https://btrfs.wiki.kernel.org/index.php/Compression#Why_does_not_du_report_the_compressed_size.3F
> that du will not tell me the compressed size of a file. This is very
> counter-intuitive, isn't it?
> The reason stated is that some tools apparently determine the sparse-ness of
> a file by comparing the size with the stat.st_blocks value.
> I do not know if there is a better way to do that, so maybe my argument
> falls apart right here, BUT: this looks to me like working around one bug by
> introducing another. Wouldn't it be better to have a mount option
> "make_du_lie_for_buggy_tools" for those that really need it? BTW, which
> tools would an honest du break, and how? (What harm is there in thinking
> that a compressed file is sparse?)

I don't know anything about this so someone else will have to chime in.

-- 
Chris Murphy