Why usable space can be so different?

[Jorge Bastos <jorge.mrbastos@gmail.com>]

HI,

This is not a big deal, but mostly out of curiosity, I've noticed
before that sometimes I couldn't fill up a single device btrfs
filesystem as much as I would expect, recently I've been farming some
chia and here is a very good example, both are 8TB disks, filled up
sequentially with 100MiB chia plots, this one looks about what I would
expect:

btrfs fi usage /mnt/disk4
Overall:
    Device size:                   7.28TiB
    Device allocated:              7.28TiB
    Device unallocated:            1.04MiB
    Device missing:                  0.00B
    Used:                          7.24TiB
    Free (estimated):             34.55GiB      (min: 34.55GiB)
    Free (statfs, df):            34.55GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:7.26TiB, Used:7.22TiB (99.54%)
   /dev/md4        7.26TiB

Metadata,DUP: Size:9.50GiB, Used:8.45GiB (88.93%)
   /dev/md4       19.00GiB

System,DUP: Size:32.00MiB, Used:800.00KiB (2.44%)
   /dev/md4       64.00MiB

Unallocated:
   /dev/md4        1.04MiB




In this one, filled exactly the same way, could only fit 2 less plots:

btrfs fi usage /mnt/disk3
Overall:
    Device size:                   7.28TiB
    Device allocated:              7.28TiB
    Device unallocated:            1.04MiB
    Device missing:                  0.00B
    Used:                          7.04TiB
    Free (estimated):            239.04GiB      (min: 239.04GiB)
    Free (statfs, df):           239.04GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:7.26TiB, Used:7.03TiB (96.78%)
   /dev/md3        7.26TiB

Metadata,DUP: Size:8.51GiB, Used:7.96GiB (93.51%)
   /dev/md3       17.02GiB

System,DUP: Size:32.00MiB, Used:864.00KiB (2.64%)
   /dev/md3       64.00MiB

Unallocated:
   /dev/md3        1.04MiB




Maybe a full balance would help, but the disk was filled sequentially
with one plot (100MiB file) at a time, there shouldn't be any
fragmentation, it's as if it can't fully use the data chunks as the
other one, kernel is 5.10.21, balance with -dusage=88 doesn't relocate
any chunks, above that it fails with ENOSPC:

btrfs balance start -dusage=88 /mnt/disk3
Done, had to relocate 0 out of 7446 chunks

btrfs balance start -dusage=89 /mnt/disk3
ERROR: error during balancing '/mnt/disk3': No space left on device


Any idea what could cause this difference, also two more disks (12 and
10TB) that fall somewhere in between:

btrfs fi usage /mnt/disk1
Overall:
    Device size:                  10.91TiB
    Device allocated:             10.91TiB
    Device unallocated:            3.94GiB
    Device missing:                  0.00B
    Used:                         10.71TiB
    Free (estimated):            202.03GiB      (min: 200.06GiB)
    Free (statfs, df):           202.03GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:10.88TiB, Used:10.69TiB (98.22%)
   /dev/md1       10.88TiB

Metadata,DUP: Size:13.00GiB, Used:11.79GiB (90.74%)
   /dev/md1       26.00GiB

System,DUP: Size:32.00MiB, Used:1.25MiB (3.91%)
   /dev/md1       64.00MiB

Unallocated:
   /dev/md1        3.94GiB






btrfs fi usage /mnt/disk2
Overall:
    Device size:                   9.09TiB
    Device allocated:              9.09TiB
    Device unallocated:           13.01MiB
    Device missing:                  0.00B
    Used:                          8.93TiB
    Free (estimated):            169.67GiB      (min: 169.67GiB)
    Free (statfs, df):           169.68GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:9.07TiB, Used:8.91TiB (98.17%)
   /dev/md2        9.07TiB

Metadata,DUP: Size:11.00GiB, Used:9.98GiB (90.76%)
   /dev/md2       22.00GiB

System,DUP: Size:8.00MiB, Used:992.00KiB (12.11%)
   /dev/md2       16.00MiB

Unallocated:
   /dev/md2       13.01MiB







Thanks,
Jorge