some principal understanding problems (balance and free space)

some principal understanding problems (balance and free space)

[Lentes, Bernd]

[-- Attachment #1: Type: text/plain, Size: 3622 bytes --]

Hi,

OMG. Why is BTRFS in some cases so complicated? I just expect that a FS does its job, nothing else.
I read the wiki and Merlin's Blog, but the more i read the more i get confused.
Please help me, i'd like to use BTRFS, especially because of the snapshot feature which i'm missing in the most others FS.
And i like to understand what i do.

Let's take this:

root@pc65472:~# btrfs fi df /
Data, single: total=361.00GiB, used=343.29GiB
System, single: total=32.00MiB, used=80.00KiB
Metadata, single: total=9.00GiB, used=7.10GiB
root@pc65472:~#
root@pc65472:~# btrfs fi show /
Label: none  uuid: 3a623645-a5e1-438e-b0f3-f02520f1a2eb
        Total devices 1 FS bytes used 350.39GiB
        devid    1 size 420.00GiB used 372.03GiB path /dev/mapper/vg1-lv_root

What does "Data, single: total=361.00GiB, used=343.29GiB" mean ?
Having 343GB pure data occupying 361GB ?
From what i've read before my understanding was that 361GB are reversed for data allocation ("data will be stored there").
And 343GB are really used for data, so 18Gb pure data can still be saved. Is that correct ?
What if i save now 18GB, so the total and used value are equal ? Does BTRFS claim new space so that the total value is growing ?

Or are the 18GB lost and unusable ?

I read that if Metadata is occupying more than 75% of the total value you need to react.
I did some "btrfs balance start / -dusage=5", increasing the value of dusage in steps of 5.
I expected that i get more total space or less used space for Metadata. But it didn't.
What happened is that the total value for Data shrinked a bit, from 363GB to 361GB.

I did then some "btrfs balance start / -musage=5" increasing in steps of 5. I expected that "used" for Metadata decreased, but it didn't.
Finally the value "total" for Metadata decreased (from 9GB to 8GB), which isn't completely contrary to what i've expected:

root@pc65472:~# btrfs fi df /
Data, single: total=361.00GiB, used=343.46GiB
System, single: total=32.00MiB, used=80.00KiB
Metadata, single: total=8.00GiB, used=7.10GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

That would mean that after my "btrfs balance ... musage= " there is now less space for Metadata than before. Why to balance then ?

OS is Ubuntu 16.04, kernel is 4.4.0-66-generic.

What is with the most recent kernels ? Is there an automatic "btrfs balance" or do i still have to check my BTRFS regulary ?

Thanks for helping me to sort that out.

Bernd


-- 

Bernd Lentes 
System Administrator 
Institute for Metabolism and Cell Death (MCD) 
Building 25 - office 122 
HelmholtzZentrum München 
bernd.lentes@helmholtz-muenchen.de 
phone: +49 89 3187 1241 
phone: +49 89 3187 3827 
fax: +49 89 3187 2294 
http://www.helmholtz-muenchen.de/mcd 


Public key: 

30 82 01 0a 02 82 01 01 00 b3 72 3e ce 2c 0a 6f 58 49 2c 92 23 c7 b9 c1 ff 6c 3a 53 be f7 9e e9 24 b7 49 fa 3c e8 de 28 85 2c d3 ed f7 70 03 3f 4d 82 fc cc 96 4f 18 27 1f df 25 b3 13 00 db 4b 1d ec 7f 1b cf f9 cd e8 5b 1f 11 b3 a7 48 f8 c8 37 ed 41 ff 18 9f d7 83 51 a9 bd 86 c2 32 b3 d6 2d 77 ff 32 83 92 67 9e ae ae 9c 99 ce 42 27 6f bf d8 c2 a1 54 fd 2b 6b 12 65 0e 8a 79 56 be 53 89 70 51 02 6a eb 76 b8 92 25 2d 88 aa 57 08 42 ef 57 fb fe 00 71 8e 90 ef b2 e3 22 f3 34 4f 7b f1 c4 b1 7c 2f 1d 6f bd c8 a6 a1 1f 25 f3 e4 4b 6a 23 d3 d2 fa 27 ae 97 80 a3 f0 5a c4 50 4a 45 e3 45 4d 82 9f 8b 87 90 d0 f9 92 2d a7 d2 67 53 e6 ae 1e 72 3e e9 e0 c9 d3 1c 23 e0 75 78 4a 45 60 94 f8 e3 03 0b 09 85 08 d0 6c f3 ff ce fa 50 25 d9 da 81 7b 2a dc 9e 28 8b 83 04 b4 0a 9f 37 b8 ac 58 f1 38 43 0e 72 af 02 03 01 00 01

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2217 bytes --]

Re: some principal understanding problems (balance and free space)

[Nikolay Borisov]

On 14.10.21 г. 15:57, Lentes, Bernd wrote:
> Hi,
> 
> OMG. Why is BTRFS in some cases so complicated? I just expect that a FS does its job, nothing else.
> I read the wiki and Merlin's Blog, but the more i read the more i get confused.
> Please help me, i'd like to use BTRFS, especially because of the snapshot feature which i'm missing in the most others FS.
> And i like to understand what i do.
> 
> Let's take this:
> 
> root@pc65472:~# btrfs fi df /
> Data, single: total=361.00GiB, used=343.29GiB
> System, single: total=32.00MiB, used=80.00KiB
> Metadata, single: total=9.00GiB, used=7.10GiB
> root@pc65472:~#
> root@pc65472:~# btrfs fi show /
> Label: none  uuid: 3a623645-a5e1-438e-b0f3-f02520f1a2eb
>         Total devices 1 FS bytes used 350.39GiB
>         devid    1 size 420.00GiB used 372.03GiB path /dev/mapper/vg1-lv_root

In order to understand those numbers and interpret them correctly you
need to understand how btrfs manages its memory. Essentially we take a
linear device (or multiple devices, but for this example it will be
linear) and we split them in Metadata/Data chunks (this is what the
Metadata/Data mean above). I will ignore system as it's not important.


Respectively metadata/data writes would go to the respective space. For
example say you want to write 4megabytes if there are no free data
groups i.e total == used in the "DATA" line we'd allocate a data block
group with a size of 1gb, so now total would be 1g larger than Used, and
we'll be able to write our 4 megabytes. Now what happens in this case is
that we've allocated 1 gigabyte for data and only used 4 megabytes of
it, in this case you might argue we are wasting the rest of the 996
megabytes. However, if you want to write further data you'd be able to
do so within those 996megabytes, after they are exhausted btrfs will go
and fetch another chunk of 1 gigabytes. This works the same way for
metadata.


So based on the numbers above you have a device whose actual physical
size is 420 gigabytes, of it 372 are allocated i.e split among
data/metadata groups, but only 350 giigabytes of those 372 which are
allocated are used. Looking further we can see that you have 9gb of
metadata allocated and only 7.1g of this is used. BY the same logic you
have 361g of data allocated, but only 343 used.

And you have close to 40g of "unallocated" space i.e one which is not
dedicated to either data or metadata. So that's a healthy filesystem state.

The reason why you'd want to run balance is if you have some
pathological workload which creates a lot of blockgroups of either type
(Data/Metadata) but which are sparsely populated. For example you write
a bunch of files which cause the FS to allocate 100g (i.e 100
blockgroups) for data, then you free 95% of every block group. The end
result would be you will have 100 blockgroup but only 5 of each is going
to be populated, the rest of the space is going to be "wasted" because
it is dedicated for data but you are not using it AT THE MOMENT, this
doesn't mean you won't use it in the future. This situation can become
problematic if you run out of unallocated space and your workload
requires you to allocated a metadata chunk, in this case the filesystem
might give you an ENOSPC error because it will see that there is no
unallocated space but in reality you could squash those 100, very
lightly used blockgroup into 1 or 2 almost full block group which would
render the rest of the space "unallocated", meaning it can be used for
either data/metadata.

Does that answer your question?


> 
> What does "Data, single: total=361.00GiB, used=343.29GiB" mean ?
> Having 343GB pure data occupying 361GB ?
> From what i've read before my understanding was that 361GB are reversed for data allocation ("data will be stored there").
> And 343GB are really used for data, so 18Gb pure data can still be saved. Is that correct ?
> What if i save now 18GB, so the total and used value are equal ? Does BTRFS claim new space so that the total value is growing ?
> 
> Or are the 18GB lost and unusable ?
> 
> I read that if Metadata is occupying more than 75% of the total value you need to react.
> I did some "btrfs balance start / -dusage=5", increasing the value of dusage in steps of 5.
> I expected that i get more total space or less used space for Metadata. But it didn't.
> What happened is that the total value for Data shrinked a bit, from 363GB to 361GB.
> 
> I did then some "btrfs balance start / -musage=5" increasing in steps of 5. I expected that "used" for Metadata decreased, but it didn't.
> Finally the value "total" for Metadata decreased (from 9GB to 8GB), which isn't completely contrary to what i've expected:
> 
> root@pc65472:~# btrfs fi df /
> Data, single: total=361.00GiB, used=343.46GiB
> System, single: total=32.00MiB, used=80.00KiB
> Metadata, single: total=8.00GiB, used=7.10GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> That would mean that after my "btrfs balance ... musage= " there is now less space for Metadata than before. Why to balance then ?
> 
> OS is Ubuntu 16.04, kernel is 4.4.0-66-generic.
> 
> What is with the most recent kernels ? Is there an automatic "btrfs balance" or do i still have to check my BTRFS regulary ?
> 
> Thanks for helping me to sort that out.
> 
> Bernd
> 
> 

Re: some principal understanding problems (balance and free space)

[Hugo Mills]

On Thu, Oct 14, 2021 at 02:57:58PM +0200, Lentes, Bernd wrote:
> Hi,
> 
> OMG. Why is BTRFS in some cases so complicated? I just expect that a FS does its job, nothing else.
> I read the wiki and Merlin's Blog, but the more i read the more i get confused.
> Please help me, i'd like to use BTRFS, especially because of the snapshot feature which i'm missing in the most others FS.
> And i like to understand what i do.
> 
> Let's take this:
> 
> root@pc65472:~# btrfs fi df /
> Data, single: total=361.00GiB, used=343.29GiB
> System, single: total=32.00MiB, used=80.00KiB
> Metadata, single: total=9.00GiB, used=7.10GiB
> root@pc65472:~#
> root@pc65472:~# btrfs fi show /
> Label: none  uuid: 3a623645-a5e1-438e-b0f3-f02520f1a2eb
>         Total devices 1 FS bytes used 350.39GiB
>         devid    1 size 420.00GiB used 372.03GiB path /dev/mapper/vg1-lv_root
> 
> What does "Data, single: total=361.00GiB, used=343.29GiB" mean ?

   Of the 361GiB currently allocated for use as data storage,
343.29GiB of actual file data is stored in it.

> Having 343GB pure data occupying 361GB ?
> From what i've read before my understanding was that 361GB are reversed for data allocation ("data will be stored there").
> And 343GB are really used for data, so 18Gb pure data can still be saved. Is that correct ?
> What if i save now 18GB, so the total and used value are equal ? Does BTRFS claim new space so that the total value is growing ?

   Yes. If you write another 20 GiB to the FS, the 361GiB in the fi df
output will go up, as will the 372.03 GiB in the fi show output.

> Or are the 18GB lost and unusable ?
> 
> I read that if Metadata is occupying more than 75% of the total value you need to react.

   Not really. If you have a fully allocated filesystem (even if it's
not full of data), then there's nowhere for new metadata chunks to be
allocated from. That's when you need to take action, by reducing the
data allocation. This is done by balancing (some of) the data.

> I did some "btrfs balance start / -dusage=5", increasing the value of dusage in steps of 5.
> I expected that i get more total space or less used space for Metadata. But it didn't.
> What happened is that the total value for Data shrinked a bit, from 363GB to 361GB.

   That's correct. Balance will compact the used data and free up any
allocation that is unused as a result.

> I did then some "btrfs balance start / -musage=5" increasing in steps of 5. I expected that "used" for Metadata decreased, but it didn't.
> Finally the value "total" for Metadata decreased (from 9GB to 8GB), which isn't completely contrary to what i've expected:

   Balance does the exact same thing to metadata -- it compacts it,
and frees up the unused allocation. However, don't balance metadata
(ever), as it exacerbates the usual out-of-space case (where metadata
is low and there's nowhere for it to expand into).

> root@pc65472:~# btrfs fi df /
> Data, single: total=361.00GiB, used=343.46GiB
> System, single: total=32.00MiB, used=80.00KiB
> Metadata, single: total=8.00GiB, used=7.10GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> That would mean that after my "btrfs balance ... musage= " there is now less space for Metadata than before. Why to balance then ?

   Don't balance metadata. :)

> OS is Ubuntu 16.04, kernel is 4.4.0-66-generic.

   This is very old, and I would strongly recommend updating to
something after about 5.5.

> What is with the most recent kernels ? Is there an automatic "btrfs balance" or do i still have to check my BTRFS regulary ?

   No, there's no automatic balance. Recent kernels will drop a chunk
if it's completely empty, but the chances of that happening without a
balance are pretty much zero.

   Hugo.

-- 
Hugo Mills             | The last man on Earth sat in a room. Suddenly, there
hugo@... carfax.org.uk | was a knock at the door.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                                        Frederic Brown

Re: some principal understanding problems (balance and free space)

[Patrik Lundquist]

On Thu, 14 Oct 2021 at 15:19, Lentes, Bernd
<bernd.lentes@helmholtz-muenchen.de> wrote:
>
> OS is Ubuntu 16.04, kernel is 4.4.0-66-generic.

At least install the HWE kernel, but you should really upgrade Ubuntu.

sudo apt-get install --install-recommends linux-generic-hwe-16.04

See https://wiki.ubuntu.com/Kernel/LTSEnablementStack

Re: some principal understanding problems (balance and free space)

[Lentes, Bernd]

[-- Attachment #1: Type: text/plain, Size: 121 bytes --]


----- On Oct 14, 2021, at 3:40 PM, Hugo Mills hugo@carfax.org.uk wrote:
>   Hugo.


Thanks, great explaination.

Bernd 

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2217 bytes --]

Re: some principal understanding problems (balance and free space)

[Lentes, Bernd]

[-- Attachment #1: Type: text/plain, Size: 143 bytes --]

----- On Oct 14, 2021, at 3:25 PM, Nikolay Borisov nborisov@suse.com wrote:

 
> Does that answer your question?

It does. Thanks a lot.

Bernd

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2217 bytes --]