some principal questions to "disk full"

some principal questions to "disk full"

[Lentes, Bernd]

[-- Attachment #1: Type: text/plain, Size: 3974 bytes --]

Hi guys,

it would be great if you can help me to sort out my understanding of BTRFS and its problems with "disk full".
I'm using BTRFS on several systems and this is now the first time that i have a "full disk".
I'd really like to understand the problem and also how to solve it.
I gave my best to read and understand https://btrfs.wiki.kernel.org/index.php/FAQ.

My setup:
Ubuntu 18.04
kernel 4.4.0-66-generic
64 bit

My disk:
root@pc65472:/data# btrfs fi show /
Label: none  uuid: 3a623645-a5e1-438e-b0f3-f02520f1a2eb
        Total devices 2 FS bytes used 350.08GiB
        devid    1 size 420.00GiB used 420.00GiB path /dev/mapper/vg1-lv_root
        devid    2 size 2.00GiB used 1.00GiB path /dev/loop0
(the loop device is from an effort of "btrfs balance" i will report later on).

Is it correct that the 420GiB shown as "used" in the line of devid 1 means "allocated" ?
Allocated means ... "not usable, reserved" ?
What is "unallocated" ? A kind of "free, usable" ?

root@pc65472:/data# btrfs fi usage /
Overall:
    Device size:                 422.00GiB
    Device allocated:            421.00GiB
    Device unallocated:            1.00GiB
    Device missing:                  0.00B
    Used:                        350.08GiB
    Free (estimated):             70.29GiB      (min: 70.29GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              512.00MiB      (used: 0.00B)

Data,single: Size:413.99GiB, Used:344.69GiB
   /dev/mapper/vg1-lv_root       413.99GiB

Metadata,single: Size:7.01GiB, Used:5.39GiB
   /dev/loop0      1.00GiB
   /dev/mapper/vg1-lv_root         6.01GiB

System,single: Size:4.00MiB, Used:80.00KiB
   /dev/mapper/vg1-lv_root         4.00MiB

Unallocated:
   /dev/loop0      1.00GiB
   /dev/mapper/vg1-lv_root           0.00B

"Device size" and "Device allocated" is slight bigger than shown with "btrfs fi show"
It seems the output of "btrfs usage /" adds the loop-device in its calculation, and "btrfs fi show" doesn't. Correct ?

I think the reason for "disk full" is the metadata:
Metadata,single: Size:7.01GiB, Used:5.39GiB
   /dev/loop0      1.00GiB
   /dev/mapper/vg1-lv_root         6.01GiB

Before i added the loop nearly all space for the metadata was occupied (5,4GiB from 6GiB). That's the problem ?
Only 600MB were free for metadata, that's roundabout the "Global Reserve", and that's, following the wiki, to little.

I deleted som old snapshots and now have enough space. I started a "btrfs balance".
My first effort with an additional device (the loop device) didn't suceed.

What can i do to prevent a "disk full" situation ?
Running regulary "btrfs balance" in a cron job ?

Is there a way to transform allocated space to unallocated space ?

        
Bernd

-- 

Bernd Lentes 
System Administrator 
Institute for Metabolism and Cell Death (MCD) 
Building 25 - office 122 
HelmholtzZentrum München 
bernd.lentes@helmholtz-muenchen.de 
phone: +49 89 3187 1241 
phone: +49 89 3187 3827 
fax: +49 89 3187 2294 
http://www.helmholtz-muenchen.de/mcd 


Public key: 

30 82 01 0a 02 82 01 01 00 b3 72 3e ce 2c 0a 6f 58 49 2c 92 23 c7 b9 c1 ff 6c 3a 53 be f7 9e e9 24 b7 49 fa 3c e8 de 28 85 2c d3 ed f7 70 03 3f 4d 82 fc cc 96 4f 18 27 1f df 25 b3 13 00 db 4b 1d ec 7f 1b cf f9 cd e8 5b 1f 11 b3 a7 48 f8 c8 37 ed 41 ff 18 9f d7 83 51 a9 bd 86 c2 32 b3 d6 2d 77 ff 32 83 92 67 9e ae ae 9c 99 ce 42 27 6f bf d8 c2 a1 54 fd 2b 6b 12 65 0e 8a 79 56 be 53 89 70 51 02 6a eb 76 b8 92 25 2d 88 aa 57 08 42 ef 57 fb fe 00 71 8e 90 ef b2 e3 22 f3 34 4f 7b f1 c4 b1 7c 2f 1d 6f bd c8 a6 a1 1f 25 f3 e4 4b 6a 23 d3 d2 fa 27 ae 97 80 a3 f0 5a c4 50 4a 45 e3 45 4d 82 9f 8b 87 90 d0 f9 92 2d a7 d2 67 53 e6 ae 1e 72 3e e9 e0 c9 d3 1c 23 e0 75 78 4a 45 60 94 f8 e3 03 0b 09 85 08 d0 6c f3 ff ce fa 50 25 d9 da 81 7b 2a dc 9e 28 8b 83 04 b4 0a 9f 37 b8 ac 58 f1 38 43 0e 72 af 02 03 01 00 01

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2217 bytes --]

Re: some principal questions to "disk full"

[Andrei Borzenkov]

On 25.05.2021 19:20, Lentes, Bernd wrote:
> Hi guys,
> 
> it would be great if you can help me to sort out my understanding of BTRFS and its problems with "disk full".
> I'm using BTRFS on several systems and this is now the first time that i have a "full disk".
> I'd really like to understand the problem and also how to solve it.
> I gave my best to read and understand https://btrfs.wiki.kernel.org/index.php/FAQ.
> 
> My setup:
> Ubuntu 18.04
> kernel 4.4.0-66-generic
> 64 bit
> 
> My disk:
> root@pc65472:/data# btrfs fi show /
> Label: none  uuid: 3a623645-a5e1-438e-b0f3-f02520f1a2eb
>         Total devices 2 FS bytes used 350.08GiB
>         devid    1 size 420.00GiB used 420.00GiB path /dev/mapper/vg1-lv_root
>         devid    2 size 2.00GiB used 1.00GiB path /dev/loop0
> (the loop device is from an effort of "btrfs balance" i will report later on).
> 
> Is it correct that the 420GiB shown as "used" in the line of devid 1 means "allocated" ?

Yes

> Allocated means ... "not usable, reserved" ?

btrfs is using two stage space management. First large areas (chunk,
block group) are allocated on device, then each chunk is subdivided in
extents as needed. You have 420iGB of block groups and inside of these
block groups 350,08GiB was used by actual data.

> What is "unallocated" ? A kind of "free, usable" ?
> 

Device space not belonging to any block group. Yes, it is "free,
usable", but so are 69.92GiB inside of allocated space.

> root@pc65472:/data# btrfs fi usage /
> Overall:
>     Device size:                 422.00GiB
>     Device allocated:            421.00GiB
>     Device unallocated:            1.00GiB
>     Device missing:                  0.00B
>     Used:                        350.08GiB
>     Free (estimated):             70.29GiB      (min: 70.29GiB)

Which is what it tells you - it is sum of unallocated space and unused
allocated space.

>     Data ratio:                       1.00
>     Metadata ratio:                   1.00
>     Global reserve:              512.00MiB      (used: 0.00B)
> 
> Data,single: Size:413.99GiB, Used:344.69GiB
>    /dev/mapper/vg1-lv_root       413.99GiB
> 
> Metadata,single: Size:7.01GiB, Used:5.39GiB
>    /dev/loop0      1.00GiB
>    /dev/mapper/vg1-lv_root         6.01GiB
> 
> System,single: Size:4.00MiB, Used:80.00KiB
>    /dev/mapper/vg1-lv_root         4.00MiB
> 
> Unallocated:
>    /dev/loop0      1.00GiB
>    /dev/mapper/vg1-lv_root           0.00B
> 
> "Device size" and "Device allocated" is slight bigger than shown with "btrfs fi show"

Both show exactly the same 422GiB and 421GiB.

> It seems the output of "btrfs usage /" adds the loop-device in its calculation, and "btrfs fi show" doesn't. Correct ?

No.

> 
> I think the reason for "disk full" is the metadata:
> Metadata,single: Size:7.01GiB, Used:5.39GiB
>    /dev/loop0      1.00GiB
>    /dev/mapper/vg1-lv_root         6.01GiB
> 
> Before i added the loop nearly all space for the metadata was occupied (5,4GiB from 6GiB). That's the problem ?

It depends, but in general everything btrfs does requires allocation of
additional metadata to complete the operation. You are certainly very
low on available metadata space.

> Only 600MB were free for metadata, that's roundabout the "Global Reserve", and that's, following the wiki, to little.
> 
> I deleted som old snapshots and now have enough space. I started a "btrfs balance".
> My first effort with an additional device (the loop device) didn't suceed.
> 

Showing actual logs with actual error is better than giving vague
description.

> What can i do to prevent a "disk full" situation ?

The only universal answer - do not fill up your disk. Seriously.

> Running regulary "btrfs balance" in a cron job ?
> 

If you are using more or less recent kernel version, you need not run
btrfs balance at all except when possibly converting profiles or adding
new device. btrfs became reasonably good at managing free space.

> Is there a way to transform allocated space to unallocated space ?
> 

Only by using balance. But that requires sufficient free space to
allocate additional metadata during balance.

Normally btrfs is using different space to store data and metadata. So
even if you have 60GiB of free space inside data block groups, it cannot
be used to allocate metadata.

You can try running "btrfs balance start -d usage=0" - it does not move
any data, just removes completely empty block groups. Also running
balance with small usage (like 1 to 5) percentage may help in compacting
block groups.

Running full balance should certainly be avoided.

Re: some principal questions to "disk full"

[Lentes, Bernd]

[-- Attachment #1: Type: text/plain, Size: 511 bytes --]



----- On May 26, 2021, at 7:36 AM, arvidjaar arvidjaar@gmail.com wrote:

> On 25.05.2021 19:20, Lentes, Bernd wrote:
 

>> What can i do to prevent a "disk full" situation ?
>> Running regulary "btrfs balance" in a cron job ?
>> 
> 
> If you are using more or less recent kernel version, you need not run
> btrfs balance at all except when possibly converting profiles or adding
> new device. btrfs became reasonably good at managing free space.

What is "more or less recent" ? Which version numbers ?

Bernd

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2217 bytes --]

Re: some principal questions to "disk full"

[Zygo Blaxell]

On Wed, May 26, 2021 at 11:41:02AM +0200, Lentes, Bernd wrote:
> 
> 
> ----- On May 26, 2021, at 7:36 AM, arvidjaar arvidjaar@gmail.com wrote:
> 
> > On 25.05.2021 19:20, Lentes, Bernd wrote:
>  
> 
> >> What can i do to prevent a "disk full" situation ?
> >> Running regulary "btrfs balance" in a cron job ?

Run

	btrfs balance start -dlimit=1 /path/to/your/fs

about once a day.  This will force the filesystem to free up some
allocated but unused space in data chunks, making them unallocated.
This will ensure you have enough unallocated space to allow you to
allocate a new metadata chunk when needed.

btrfs might allocate a new metadata chunk at any time, so it is a good
idea to ensure at least 1GB (preferably 5 or 10 GB) is always unallocated.
The filesystem does have workarounds that will let it continue to work
with very low disk space, but they can degrade performance, increase
write wear, and increase exposure to firmware bugs if your drive has them.

Never balance metadata (full balance or btrfs balance start -m) from a
maintenance script.  This will free metadata chunks and lead directly to
ENOSPC gotchas.  The kernel deals with metadata allocations fairly well
for most users--but only if you don't break it with metadata balances.

If possible, resize your filesystem to about 5GB less than the underlying
device size.  That way, if you get stuck with low space for metadata, you
can do 'btrfs fi resize 1:max /fs' to get some quick and safe emergency
space to complete a balance or a snapshot delete.  Don't forget to resize
back to 5GB less than full size when the crisis is resolved.

> > If you are using more or less recent kernel version, you need not run
> > btrfs balance at all except when possibly converting profiles or adding
> > new device. btrfs became reasonably good at managing free space.
> 
> What is "more or less recent" ? Which version numbers ?

3.18 or later.  It's certainly not a new feature.

> Bernd