All of lore.kernel.org
 help / color / mirror / Atom feed
* Why usable space can be so different?
@ 2021-07-29  9:47 Jorge Bastos
  2021-07-29 18:17 ` waxhead
  2021-07-30  4:47 ` Andrei Borzenkov
  0 siblings, 2 replies; 5+ messages in thread
From: Jorge Bastos @ 2021-07-29  9:47 UTC (permalink / raw)
  To: Btrfs BTRFS

HI,

This is not a big deal, but mostly out of curiosity, I've noticed
before that sometimes I couldn't fill up a single device btrfs
filesystem as much as I would expect, recently I've been farming some
chia and here is a very good example, both are 8TB disks, filled up
sequentially with 100MiB chia plots, this one looks about what I would
expect:

btrfs fi usage /mnt/disk4
Overall:
    Device size:                   7.28TiB
    Device allocated:              7.28TiB
    Device unallocated:            1.04MiB
    Device missing:                  0.00B
    Used:                          7.24TiB
    Free (estimated):             34.55GiB      (min: 34.55GiB)
    Free (statfs, df):            34.55GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:7.26TiB, Used:7.22TiB (99.54%)
   /dev/md4        7.26TiB

Metadata,DUP: Size:9.50GiB, Used:8.45GiB (88.93%)
   /dev/md4       19.00GiB

System,DUP: Size:32.00MiB, Used:800.00KiB (2.44%)
   /dev/md4       64.00MiB

Unallocated:
   /dev/md4        1.04MiB




In this one, filled exactly the same way, could only fit 2 less plots:

btrfs fi usage /mnt/disk3
Overall:
    Device size:                   7.28TiB
    Device allocated:              7.28TiB
    Device unallocated:            1.04MiB
    Device missing:                  0.00B
    Used:                          7.04TiB
    Free (estimated):            239.04GiB      (min: 239.04GiB)
    Free (statfs, df):           239.04GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:7.26TiB, Used:7.03TiB (96.78%)
   /dev/md3        7.26TiB

Metadata,DUP: Size:8.51GiB, Used:7.96GiB (93.51%)
   /dev/md3       17.02GiB

System,DUP: Size:32.00MiB, Used:864.00KiB (2.64%)
   /dev/md3       64.00MiB

Unallocated:
   /dev/md3        1.04MiB




Maybe a full balance would help, but the disk was filled sequentially
with one plot (100MiB file) at a time, there shouldn't be any
fragmentation, it's as if it can't fully use the data chunks as the
other one, kernel is 5.10.21, balance with -dusage=88 doesn't relocate
any chunks, above that it fails with ENOSPC:

btrfs balance start -dusage=88 /mnt/disk3
Done, had to relocate 0 out of 7446 chunks

btrfs balance start -dusage=89 /mnt/disk3
ERROR: error during balancing '/mnt/disk3': No space left on device


Any idea what could cause this difference, also two more disks (12 and
10TB) that fall somewhere in between:

btrfs fi usage /mnt/disk1
Overall:
    Device size:                  10.91TiB
    Device allocated:             10.91TiB
    Device unallocated:            3.94GiB
    Device missing:                  0.00B
    Used:                         10.71TiB
    Free (estimated):            202.03GiB      (min: 200.06GiB)
    Free (statfs, df):           202.03GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:10.88TiB, Used:10.69TiB (98.22%)
   /dev/md1       10.88TiB

Metadata,DUP: Size:13.00GiB, Used:11.79GiB (90.74%)
   /dev/md1       26.00GiB

System,DUP: Size:32.00MiB, Used:1.25MiB (3.91%)
   /dev/md1       64.00MiB

Unallocated:
   /dev/md1        3.94GiB






btrfs fi usage /mnt/disk2
Overall:
    Device size:                   9.09TiB
    Device allocated:              9.09TiB
    Device unallocated:           13.01MiB
    Device missing:                  0.00B
    Used:                          8.93TiB
    Free (estimated):            169.67GiB      (min: 169.67GiB)
    Free (statfs, df):           169.68GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:9.07TiB, Used:8.91TiB (98.17%)
   /dev/md2        9.07TiB

Metadata,DUP: Size:11.00GiB, Used:9.98GiB (90.76%)
   /dev/md2       22.00GiB

System,DUP: Size:8.00MiB, Used:992.00KiB (12.11%)
   /dev/md2       16.00MiB

Unallocated:
   /dev/md2       13.01MiB







Thanks,
Jorge

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Why usable space can be so different?
  2021-07-29  9:47 Why usable space can be so different? Jorge Bastos
@ 2021-07-29 18:17 ` waxhead
  2021-07-30  4:47 ` Andrei Borzenkov
  1 sibling, 0 replies; 5+ messages in thread
From: waxhead @ 2021-07-29 18:17 UTC (permalink / raw)
  To: Jorge Bastos, Btrfs BTRFS

Jorge Bastos wrote:
> HI,
> 
> This is not a big deal, but mostly out of curiosity, I've noticed
> before that sometimes I couldn't fill up a single device btrfs
> filesystem as much as I would expect, recently I've been farming some
> chia and here is a very good example, both are 8TB disks, filled up
> sequentially with 100MiB chia plots, this one looks about what I would
> expect:
>
I am just a regular BTRFS user , and this is just a wild guess - but it 
seems like the small difference in size causes data and metadata chunks 
to be allocated in a different order. E.g. you hit full data before you 
hit full metadata and vice versa.

One thing that confused me when I started to use BTRFS was the concept 
of allocated vs used disk space. I use "btrfs fi us -T /mnt" often and 
from my experience it is perfectly healthy to panic when unallocated 
(not necessarily unused) space is low. This is when I start to balance 
and shuffle stuff around so that I can have at least 2-3 gigs free for 
each device. Anything less than that is in my experience quite annoying.

I know there is a global reserve, but quite frankly I wish BTRFS did 
allocate 3-4 gigs that one might release for use with a magic 
maintenance command.


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Why usable space can be so different?
  2021-07-29  9:47 Why usable space can be so different? Jorge Bastos
  2021-07-29 18:17 ` waxhead
@ 2021-07-30  4:47 ` Andrei Borzenkov
  2021-07-30  7:12   ` Jorge Bastos
  1 sibling, 1 reply; 5+ messages in thread
From: Andrei Borzenkov @ 2021-07-30  4:47 UTC (permalink / raw)
  To: Jorge Bastos, Btrfs BTRFS

On 29.07.2021 12:47, Jorge Bastos wrote:
> HI,
> 
> This is not a big deal, but mostly out of curiosity, I've noticed
> before that sometimes I couldn't fill up a single device btrfs
> filesystem as much as I would expect, recently I've been farming some
> chia and here is a very good example, both are 8TB disks, filled up
> sequentially with 100MiB chia plots, this one looks about what I would
> expect:
> 
> btrfs fi usage /mnt/disk4
> Overall:
>     Device size:                   7.28TiB
>     Device allocated:              7.28TiB
>     Device unallocated:            1.04MiB
>     Device missing:                  0.00B
>     Used:                          7.24TiB
>     Free (estimated):             34.55GiB      (min: 34.55GiB)
>     Free (statfs, df):            34.55GiB
>     Data ratio:                       1.00
>     Metadata ratio:                   2.00
>     Global reserve:              512.00MiB      (used: 0.00B)
>     Multiple profiles:                  no
> 
> Data,single: Size:7.26TiB, Used:7.22TiB (99.54%)
>    /dev/md4        7.26TiB
> 
> Metadata,DUP: Size:9.50GiB, Used:8.45GiB (88.93%)
>    /dev/md4       19.00GiB
> 
> System,DUP: Size:32.00MiB, Used:800.00KiB (2.44%)
>    /dev/md4       64.00MiB
> 
> Unallocated:
>    /dev/md4        1.04MiB
> 
> 
> 
> 
> In this one, filled exactly the same way, could only fit 2 less plots:
> 
> btrfs fi usage /mnt/disk3
> Overall:
>     Device size:                   7.28TiB
>     Device allocated:              7.28TiB
>     Device unallocated:            1.04MiB
>     Device missing:                  0.00B
>     Used:                          7.04TiB
>     Free (estimated):            239.04GiB      (min: 239.04GiB)
>     Free (statfs, df):           239.04GiB

Is you "plot" (whatever it is) 100MiB or 100GiB? Because your first
example has 34GiB available and from your description you cannot write
anything - but I would be very surprised if you could not write another
100MiB to it.

100GiB would fit your description.

>     Data ratio:                       1.00
>     Metadata ratio:                   2.00
>     Global reserve:              512.00MiB      (used: 0.00B)
>     Multiple profiles:                  no
> 
> Data,single: Size:7.26TiB, Used:7.03TiB (96.78%)
>    /dev/md3        7.26TiB
> 
> Metadata,DUP: Size:8.51GiB, Used:7.96GiB (93.51%)
>    /dev/md3       17.02GiB
> 
> System,DUP: Size:32.00MiB, Used:864.00KiB (2.64%)
>    /dev/md3       64.00MiB
> 
> Unallocated:
>    /dev/md3        1.04MiB
> 
> 
> 
> 
> Maybe a full balance would help, but the disk was filled sequentially
> with one plot (100MiB file) at a time, there shouldn't be any
> fragmentation, it's as if it can't fully use the data chunks as the
> other one, kernel is 5.10.21, balance with -dusage=88 doesn't relocate
> any chunks, above that it fails with ENOSPC:
> 
> btrfs balance start -dusage=88 /mnt/disk3
> Done, had to relocate 0 out of 7446 chunks
> 
> btrfs balance start -dusage=89 /mnt/disk3
> ERROR: error during balancing '/mnt/disk3': No space left on device
> 
> 
> Any idea what could cause this difference, also two more disks (12 and
> 10TB) that fall somewhere in between:
> 
> btrfs fi usage /mnt/disk1
> Overall:
>     Device size:                  10.91TiB
>     Device allocated:             10.91TiB
>     Device unallocated:            3.94GiB
>     Device missing:                  0.00B
>     Used:                         10.71TiB
>     Free (estimated):            202.03GiB      (min: 200.06GiB)
>     Free (statfs, df):           202.03GiB
>     Data ratio:                       1.00
>     Metadata ratio:                   2.00
>     Global reserve:              512.00MiB      (used: 0.00B)
>     Multiple profiles:                  no
> 
> Data,single: Size:10.88TiB, Used:10.69TiB (98.22%)
>    /dev/md1       10.88TiB
> 
> Metadata,DUP: Size:13.00GiB, Used:11.79GiB (90.74%)
>    /dev/md1       26.00GiB
> 
> System,DUP: Size:32.00MiB, Used:1.25MiB (3.91%)
>    /dev/md1       64.00MiB
> 
> Unallocated:
>    /dev/md1        3.94GiB
> 
> 
> 
> 
> 
> 
> btrfs fi usage /mnt/disk2
> Overall:
>     Device size:                   9.09TiB
>     Device allocated:              9.09TiB
>     Device unallocated:           13.01MiB
>     Device missing:                  0.00B
>     Used:                          8.93TiB
>     Free (estimated):            169.67GiB      (min: 169.67GiB)
>     Free (statfs, df):           169.68GiB
>     Data ratio:                       1.00
>     Metadata ratio:                   2.00
>     Global reserve:              512.00MiB      (used: 0.00B)
>     Multiple profiles:                  no
> 
> Data,single: Size:9.07TiB, Used:8.91TiB (98.17%)
>    /dev/md2        9.07TiB
> 
> Metadata,DUP: Size:11.00GiB, Used:9.98GiB (90.76%)
>    /dev/md2       22.00GiB
> 
> System,DUP: Size:8.00MiB, Used:992.00KiB (12.11%)
>    /dev/md2       16.00MiB
> 
> Unallocated:
>    /dev/md2       13.01MiB
> 
> 
> 
> 
> 
> 
> 
> Thanks,
> Jorge
> 


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Why usable space can be so different?
  2021-07-30  4:47 ` Andrei Borzenkov
@ 2021-07-30  7:12   ` Jorge Bastos
  2021-07-30  9:11     ` Jorge Bastos
  0 siblings, 1 reply; 5+ messages in thread
From: Jorge Bastos @ 2021-07-30  7:12 UTC (permalink / raw)
  To: Andrei Borzenkov; +Cc: Btrfs BTRFS

On Fri, Jul 30, 2021 at 5:47 AM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>
>
> Is you "plot" (whatever it is) 100MiB or 100GiB? Because your first
> example has 34GiB available and from your description you cannot write
> anything - but I would be very surprised if you could not write another
> 100MiB to it.
>
> 100GiB would fit your description.
>
Yes, sorry, each file is around 100GiB.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: Why usable space can be so different?
  2021-07-30  7:12   ` Jorge Bastos
@ 2021-07-30  9:11     ` Jorge Bastos
  0 siblings, 0 replies; 5+ messages in thread
From: Jorge Bastos @ 2021-07-30  9:11 UTC (permalink / raw)
  To: Andrei Borzenkov; +Cc: Btrfs BTRFS

Was going to convert the metadata do single profile since don't need
DUP for this but stupidly forgot to free up some space first, balance
crashed and fs went read_only, and strangely it does the same even
after mounting with skip_balance:

Jul 30 09:35:09 Tower1 kernel: BTRFS info (device sdz1): using free space tree
Jul 30 09:35:09 Tower1 kernel: BTRFS info (device sdz1): has skinny extents
Jul 30 09:35:13 Tower1 kernel: BTRFS info (device sdz1): balance: resume skipped
Jul 30 09:35:32 Tower1 kernel: ------------[ cut here ]------------
Jul 30 09:35:32 Tower1 kernel: BTRFS: Transaction aborted (error -28)
Jul 30 09:35:32 Tower1 kernel: WARNING: CPU: 7 PID: 4673 at
fs/btrfs/block-group.c:2122
btrfs_create_pending_block_groups+0x176/0x1ff
Jul 30 09:35:32 Tower1 kernel: Modules linked in: xt_CHECKSUM
ipt_REJECT xt_nat ip6table_mangle ip6table_nat veth iptable_mangle tun
md_mod ipmi_devintf iptable_nat xt_MASQUERADE nf_nat wireguard
curve25519_x86_64 libcurve25519_generic libchacha20poly1305
chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libblake2s
blake2s_x86_64 libblake2s_generic libchacha ip6table_filter ip6_tables
iptable_filter ip_tables mlx4_en mlx4_core igb i2c_algo_bit sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul
crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd
cryptd glue_helper rapl ipmi_ssif intel_cstate nvme i2c_i801
intel_uncore nvme_core ahci input_leds wmi i2c_smbus i2c_core
led_class libahci acpi_ipmi acpi_power_meter ipmi_si button acpi_pad
[last unloaded: vhost_iotlb]
Jul 30 09:35:32 Tower1 kernel: CPU: 7 PID: 4673 Comm: mc Tainted: G
    W         5.10.21-Unraid #1
Jul 30 09:35:32 Tower1 kernel: Hardware name: Supermicro Super
Server/X10SRi-F, BIOS 3.3 10/28/2020
Jul 30 09:35:32 Tower1 kernel: RIP:
0010:btrfs_create_pending_block_groups+0x176/0x1ff
Jul 30 09:35:32 Tower1 kernel: Code: 45 50 48 8d b8 40 0a 00 00 e8 ae
db ff ff 84 c0 75 1a 83 fb fb 74 15 83 fb e2 74 10 89 de 48 c7 c7 da
a1 d9 81 e8 9a 65 44 00 <0f> 0b 89 d9 ba 4a 08 00 00 48 c7 c6 00 9e c2
81 48 89 ef e8 3e 91
Jul 30 09:35:32 Tower1 kernel: RSP: 0018:ffffc9000360fdd0 EFLAGS: 00010286
Jul 30 09:35:32 Tower1 kernel: RAX: 0000000000000000 RBX:
00000000ffffffe4 RCX: 0000000000000027
Jul 30 09:35:32 Tower1 kernel: RDX: 00000000ffffefff RSI:
0000000000000001 RDI: ffff88903fbd8920
Jul 30 09:35:32 Tower1 kernel: RBP: ffff88815cc28270 R08:
0000000000000000 R09: 00000000ffffefff
Jul 30 09:35:32 Tower1 kernel: R10: ffffc9000360fc00 R11:
ffffc9000360fbf8 R12: ffff888cdd644508
Jul 30 09:35:32 Tower1 kernel: R13: 0000000000000004 R14:
0000000000000000 R15: ffff8881015b6000
Jul 30 09:35:32 Tower1 kernel: FS:  000015074d988740(0000)
GS:ffff88903fbc0000(0000) knlGS:0000000000000000
Jul 30 09:35:32 Tower1 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 30 09:35:32 Tower1 kernel: CR2: 00007ffcdcb4bf18 CR3:
0000000101e96001 CR4: 00000000001706e0
Jul 30 09:35:32 Tower1 kernel: Call Trace:
Jul 30 09:35:32 Tower1 kernel: __btrfs_end_transaction+0x5b/0x150
Jul 30 09:35:32 Tower1 kernel: btrfs_unlink+0x9e/0xb6
Jul 30 09:35:32 Tower1 kernel: vfs_unlink+0x82/0xeb
Jul 30 09:35:32 Tower1 kernel: do_unlinkat+0xfe/0x1d3
Jul 30 09:35:32 Tower1 kernel: do_syscall_64+0x5d/0x6a
Jul 30 09:35:32 Tower1 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 30 09:35:32 Tower1 kernel: RIP: 0033:0x15074e246277
Jul 30 09:35:32 Tower1 kernel: Code: f0 ff ff 73 01 c3 48 8b 0d 16 6c
0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66
90 b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d e9 6b 0d
00 f7 d8 64 89 01 48
Jul 30 09:35:32 Tower1 kernel: RSP: 002b:00007ffcdcb4fd18 EFLAGS:
00000206 ORIG_RAX: 0000000000000057
Jul 30 09:35:32 Tower1 kernel: RAX: ffffffffffffffda RBX:
00000000006167a0 RCX: 000015074e246277
Jul 30 09:35:32 Tower1 kernel: RDX: 000000000058d130 RSI:
00000000ffffffff RDI: 00000000005d1e70
Jul 30 09:35:32 Tower1 kernel: RBP: 00000000006133b0 R08:
0000000000000001 R09: 0000000000000000
Jul 30 09:35:32 Tower1 kernel: R10: 0000000000000000 R11:
0000000000000206 R12: 00007ffcdcb4fd7c
Jul 30 09:35:32 Tower1 kernel: R13: 0000000000000000 R14:
00000000005d14e0 R15: 00000000006133b0
Jul 30 09:35:32 Tower1 kernel: ---[ end trace 9e1013fad580900f ]---
Jul 30 09:35:32 Tower1 kernel: BTRFS: error (device sdz1) in
btrfs_create_pending_block_groups:2122: errno=-28 No space left
Jul 30 09:35:32 Tower1 kernel: BTRFS info (device sdz1): forced readonly
Jul 30 09:36:34 Tower1 kernel: BTRFS info (device sdz1): using free space tree
Jul 30 09:36:34 Tower1 kernel: BTRFS info (device sdz1): has skinny extents
Jul 30 09:36:38 Tower1 kernel: BTRFS info (device sdz1): balance: resume skipped
Jul 30 09:37:13 Tower1 kernel: ------------[ cut here ]------------


I'm just going to re-format these with xfs, I don't really need btrfs
for this, just used it because it's my default fs.

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2021-07-30  9:11 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-29  9:47 Why usable space can be so different? Jorge Bastos
2021-07-29 18:17 ` waxhead
2021-07-30  4:47 ` Andrei Borzenkov
2021-07-30  7:12   ` Jorge Bastos
2021-07-30  9:11     ` Jorge Bastos

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.