Re: Removal of Device and Free Space

[Andrei Borzenkov <arvidjaar@gmail.com>]

On 14.05.2021 11:54, Christian Völker wrote:
> 
> Hi all,
> 
> I am running a three device DRBD setup (non-RAID!).
> When I do "df -h" I see loads of free space:
> 
> root@backuppc:~# df -h
> [...]
> /dev/mapper/crypt_drbd2          2,8T    1,7T  1,1T   63% /var/lib/backuppc
> 
> As written, the fs consists of three devices:
> 
> root@backuppc:~# btrfs fi sh /var/lib/backuppc/
> Label: 'backuppc'  uuid: 73b98c7b-832a-437a-a15b-6cb00734e5db
>         Total devices 3 FS bytes used 1.70TiB
>         devid    3 size 799.96GiB used 799.96GiB path dm-5
>         devid    4 size 1.07TiB used 1.07TiB path dm-4
>         devid    7 size 899.96GiB used 327.00GiB path dm-6
> 
> root@backuppc:~# btrfs fi usage /var/lib/backuppc/
> Overall:
>     Device size:                   2.73TiB
>     Device allocated:              2.61TiB
>     Device unallocated:          128.00GiB
>     Device missing:                  0.00B
>     Used:                          1.70TiB
>     Free (estimated):              1.03TiB      (min: 1.03TiB)
>     Free (statfs, df):             1.03TiB
>     Data ratio:                       1.00
>     Metadata ratio:                   1.00
>     Global reserve:              512.00MiB      (used: 0.00B)
>     Multiple profiles:                  no
> 
> Data,single: Size:2.60TiB, Used:1.69TiB (65.17%)
>    /dev/mapper/crypt_drbd2       790.93GiB
>    /dev/mapper/crypt_drbd1         1.07TiB
>    /dev/mapper/crypt_drbd3       774.96GiB
> 

That does not match the output of above. "btrfs fi sh" claims drbd3 has
327GiB allocated and "btrfs fi us" claims it is 774GiB. Not sure how it
is possible.

> Metadata,single: Size:9.00GiB, Used:3.95GiB (43.91%)
>    /dev/mapper/crypt_drbd2         6.00GiB
>    /dev/mapper/crypt_drbd1         3.00GiB
> 
> System,single: Size:32.00MiB, Used:320.00KiB (0.98%)
>    /dev/mapper/crypt_drbd2        32.00MiB
> 
> Unallocated:
>    /dev/mapper/crypt_drbd2         3.00GiB
>    /dev/mapper/crypt_drbd1         1.03MiB
>    /dev/mapper/crypt_drbd3       125.00GiB
> 
> So it tells me there is an estimated of ~1TB free. As the crypt_drbd3
> device has a size of 899G I wanted to remove the device. I expected no
> issue as "Free" shows 1.03TiB. There should still be 200GB of free space
> afterwards.
> But the removal failed after two hours:
> 
> root@backuppc:~# btrfs dev remove /dev/mapper/crypt_drbd3
> /var/lib/backuppc/
> ERROR: error removing device '/dev/mapper/crypt_drbd3': No space left on
> device
> 
> Now it looks like this:
> root@backuppc:~# btrfs fi usage /var/lib/backuppc/
> Overall:
>     Device size:                   2.73TiB
>     Device allocated:              2.17TiB
>     Device unallocated:          572.96GiB
>     Device missing:                  0.00B
>     Used:                          1.70TiB
>     Free (estimated):              1.03TiB      (min: 1.03TiB)
>     Free (statfs, df):             1.03TiB
>     Data ratio:                       1.00
>     Metadata ratio:                   1.00
>     Global reserve:              512.00MiB      (used: 0.00B)
>     Multiple profiles:                  no
> 
> Data,single: Size:2.17TiB, Used:1.69TiB (78.24%)
>    /dev/mapper/crypt_drbd2       793.93GiB
>    /dev/mapper/crypt_drbd1         1.07TiB
>    /dev/mapper/crypt_drbd3       327.00GiB
> 

And now it actually matches.

> Metadata,single: Size:9.00GiB, Used:3.89GiB (43.23%)
>    /dev/mapper/crypt_drbd2         6.00GiB
>    /dev/mapper/crypt_drbd1         3.00GiB
> 
> System,single: Size:32.00MiB, Used:288.00KiB (0.88%)
>    /dev/mapper/crypt_drbd2        32.00MiB
> 
> Unallocated:
>    /dev/mapper/crypt_drbd2         1.03MiB
>    /dev/mapper/crypt_drbd1         1.03MiB
>    /dev/mapper/crypt_drbd3       572.96GiB
> 
> 
> So some questions arise:
> 
>     Why can btrfs device remove not check in advance if there is enough
> free space available? Instead of working for hours and then failing...
> 

Because nobody implemented it? I suspect it may not be entirely trivial
in general case and any estimation you do may become obsolete very fast
(your filesystem is in use and so space that was free may suddenly
become allocated when you need it).

>     Do I have to balance my fs after the failed removal now?
> 

Yes. Device removal moves whole chunks from device to be removed to
other devices. You need as much *unallocated* space on two remaining
devices as it allocated on device you are trying to remove.

btrfs balance start -d usage=0

would be a good starting point.

>     Why is it not possible to remove the device when all information
> tell me there is enough free space available?
> 

btrfs is using two stage allocator. Free space includes both unused
allocated space and unallocated space. Only the latter can be used for
device removal.

>     What is occupying so much disk space as the data only has 1.7TB
> which should fit in 1.8TB (two) devices? (no snapshot, nothing special
> configured on btrfs). Looks like there are ~400GB allocated which are
> not from data.
> 

It is your data, how do we know?

> Just for completeness:
> 
> Debian Buster
> 
> root@backuppc:~# btrfs --version
> btrfs-progs v5.10.1
> 
> Thanks for letting me know.
> 
> 
> Greetings
> 
> /CV
> 
> 
>