* Why usable space can be so different?
@ 2021-07-29 9:47 Jorge Bastos
2021-07-29 18:17 ` waxhead
2021-07-30 4:47 ` Andrei Borzenkov
0 siblings, 2 replies; 5+ messages in thread
From: Jorge Bastos @ 2021-07-29 9:47 UTC (permalink / raw)
To: Btrfs BTRFS
HI,
This is not a big deal, but mostly out of curiosity, I've noticed
before that sometimes I couldn't fill up a single device btrfs
filesystem as much as I would expect, recently I've been farming some
chia and here is a very good example, both are 8TB disks, filled up
sequentially with 100MiB chia plots, this one looks about what I would
expect:
btrfs fi usage /mnt/disk4
Overall:
Device size: 7.28TiB
Device allocated: 7.28TiB
Device unallocated: 1.04MiB
Device missing: 0.00B
Used: 7.24TiB
Free (estimated): 34.55GiB (min: 34.55GiB)
Free (statfs, df): 34.55GiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no
Data,single: Size:7.26TiB, Used:7.22TiB (99.54%)
/dev/md4 7.26TiB
Metadata,DUP: Size:9.50GiB, Used:8.45GiB (88.93%)
/dev/md4 19.00GiB
System,DUP: Size:32.00MiB, Used:800.00KiB (2.44%)
/dev/md4 64.00MiB
Unallocated:
/dev/md4 1.04MiB
In this one, filled exactly the same way, could only fit 2 less plots:
btrfs fi usage /mnt/disk3
Overall:
Device size: 7.28TiB
Device allocated: 7.28TiB
Device unallocated: 1.04MiB
Device missing: 0.00B
Used: 7.04TiB
Free (estimated): 239.04GiB (min: 239.04GiB)
Free (statfs, df): 239.04GiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no
Data,single: Size:7.26TiB, Used:7.03TiB (96.78%)
/dev/md3 7.26TiB
Metadata,DUP: Size:8.51GiB, Used:7.96GiB (93.51%)
/dev/md3 17.02GiB
System,DUP: Size:32.00MiB, Used:864.00KiB (2.64%)
/dev/md3 64.00MiB
Unallocated:
/dev/md3 1.04MiB
Maybe a full balance would help, but the disk was filled sequentially
with one plot (100MiB file) at a time, there shouldn't be any
fragmentation, it's as if it can't fully use the data chunks as the
other one, kernel is 5.10.21, balance with -dusage=88 doesn't relocate
any chunks, above that it fails with ENOSPC:
btrfs balance start -dusage=88 /mnt/disk3
Done, had to relocate 0 out of 7446 chunks
btrfs balance start -dusage=89 /mnt/disk3
ERROR: error during balancing '/mnt/disk3': No space left on device
Any idea what could cause this difference, also two more disks (12 and
10TB) that fall somewhere in between:
btrfs fi usage /mnt/disk1
Overall:
Device size: 10.91TiB
Device allocated: 10.91TiB
Device unallocated: 3.94GiB
Device missing: 0.00B
Used: 10.71TiB
Free (estimated): 202.03GiB (min: 200.06GiB)
Free (statfs, df): 202.03GiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no
Data,single: Size:10.88TiB, Used:10.69TiB (98.22%)
/dev/md1 10.88TiB
Metadata,DUP: Size:13.00GiB, Used:11.79GiB (90.74%)
/dev/md1 26.00GiB
System,DUP: Size:32.00MiB, Used:1.25MiB (3.91%)
/dev/md1 64.00MiB
Unallocated:
/dev/md1 3.94GiB
btrfs fi usage /mnt/disk2
Overall:
Device size: 9.09TiB
Device allocated: 9.09TiB
Device unallocated: 13.01MiB
Device missing: 0.00B
Used: 8.93TiB
Free (estimated): 169.67GiB (min: 169.67GiB)
Free (statfs, df): 169.68GiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no
Data,single: Size:9.07TiB, Used:8.91TiB (98.17%)
/dev/md2 9.07TiB
Metadata,DUP: Size:11.00GiB, Used:9.98GiB (90.76%)
/dev/md2 22.00GiB
System,DUP: Size:8.00MiB, Used:992.00KiB (12.11%)
/dev/md2 16.00MiB
Unallocated:
/dev/md2 13.01MiB
Thanks,
Jorge
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: Why usable space can be so different?
2021-07-29 9:47 Why usable space can be so different? Jorge Bastos
@ 2021-07-29 18:17 ` waxhead
2021-07-30 4:47 ` Andrei Borzenkov
1 sibling, 0 replies; 5+ messages in thread
From: waxhead @ 2021-07-29 18:17 UTC (permalink / raw)
To: Jorge Bastos, Btrfs BTRFS
Jorge Bastos wrote:
> HI,
>
> This is not a big deal, but mostly out of curiosity, I've noticed
> before that sometimes I couldn't fill up a single device btrfs
> filesystem as much as I would expect, recently I've been farming some
> chia and here is a very good example, both are 8TB disks, filled up
> sequentially with 100MiB chia plots, this one looks about what I would
> expect:
>
I am just a regular BTRFS user , and this is just a wild guess - but it
seems like the small difference in size causes data and metadata chunks
to be allocated in a different order. E.g. you hit full data before you
hit full metadata and vice versa.
One thing that confused me when I started to use BTRFS was the concept
of allocated vs used disk space. I use "btrfs fi us -T /mnt" often and
from my experience it is perfectly healthy to panic when unallocated
(not necessarily unused) space is low. This is when I start to balance
and shuffle stuff around so that I can have at least 2-3 gigs free for
each device. Anything less than that is in my experience quite annoying.
I know there is a global reserve, but quite frankly I wish BTRFS did
allocate 3-4 gigs that one might release for use with a magic
maintenance command.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Why usable space can be so different?
2021-07-29 9:47 Why usable space can be so different? Jorge Bastos
2021-07-29 18:17 ` waxhead
@ 2021-07-30 4:47 ` Andrei Borzenkov
2021-07-30 7:12 ` Jorge Bastos
1 sibling, 1 reply; 5+ messages in thread
From: Andrei Borzenkov @ 2021-07-30 4:47 UTC (permalink / raw)
To: Jorge Bastos, Btrfs BTRFS
On 29.07.2021 12:47, Jorge Bastos wrote:
> HI,
>
> This is not a big deal, but mostly out of curiosity, I've noticed
> before that sometimes I couldn't fill up a single device btrfs
> filesystem as much as I would expect, recently I've been farming some
> chia and here is a very good example, both are 8TB disks, filled up
> sequentially with 100MiB chia plots, this one looks about what I would
> expect:
>
> btrfs fi usage /mnt/disk4
> Overall:
> Device size: 7.28TiB
> Device allocated: 7.28TiB
> Device unallocated: 1.04MiB
> Device missing: 0.00B
> Used: 7.24TiB
> Free (estimated): 34.55GiB (min: 34.55GiB)
> Free (statfs, df): 34.55GiB
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 512.00MiB (used: 0.00B)
> Multiple profiles: no
>
> Data,single: Size:7.26TiB, Used:7.22TiB (99.54%)
> /dev/md4 7.26TiB
>
> Metadata,DUP: Size:9.50GiB, Used:8.45GiB (88.93%)
> /dev/md4 19.00GiB
>
> System,DUP: Size:32.00MiB, Used:800.00KiB (2.44%)
> /dev/md4 64.00MiB
>
> Unallocated:
> /dev/md4 1.04MiB
>
>
>
>
> In this one, filled exactly the same way, could only fit 2 less plots:
>
> btrfs fi usage /mnt/disk3
> Overall:
> Device size: 7.28TiB
> Device allocated: 7.28TiB
> Device unallocated: 1.04MiB
> Device missing: 0.00B
> Used: 7.04TiB
> Free (estimated): 239.04GiB (min: 239.04GiB)
> Free (statfs, df): 239.04GiB
Is you "plot" (whatever it is) 100MiB or 100GiB? Because your first
example has 34GiB available and from your description you cannot write
anything - but I would be very surprised if you could not write another
100MiB to it.
100GiB would fit your description.
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 512.00MiB (used: 0.00B)
> Multiple profiles: no
>
> Data,single: Size:7.26TiB, Used:7.03TiB (96.78%)
> /dev/md3 7.26TiB
>
> Metadata,DUP: Size:8.51GiB, Used:7.96GiB (93.51%)
> /dev/md3 17.02GiB
>
> System,DUP: Size:32.00MiB, Used:864.00KiB (2.64%)
> /dev/md3 64.00MiB
>
> Unallocated:
> /dev/md3 1.04MiB
>
>
>
>
> Maybe a full balance would help, but the disk was filled sequentially
> with one plot (100MiB file) at a time, there shouldn't be any
> fragmentation, it's as if it can't fully use the data chunks as the
> other one, kernel is 5.10.21, balance with -dusage=88 doesn't relocate
> any chunks, above that it fails with ENOSPC:
>
> btrfs balance start -dusage=88 /mnt/disk3
> Done, had to relocate 0 out of 7446 chunks
>
> btrfs balance start -dusage=89 /mnt/disk3
> ERROR: error during balancing '/mnt/disk3': No space left on device
>
>
> Any idea what could cause this difference, also two more disks (12 and
> 10TB) that fall somewhere in between:
>
> btrfs fi usage /mnt/disk1
> Overall:
> Device size: 10.91TiB
> Device allocated: 10.91TiB
> Device unallocated: 3.94GiB
> Device missing: 0.00B
> Used: 10.71TiB
> Free (estimated): 202.03GiB (min: 200.06GiB)
> Free (statfs, df): 202.03GiB
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 512.00MiB (used: 0.00B)
> Multiple profiles: no
>
> Data,single: Size:10.88TiB, Used:10.69TiB (98.22%)
> /dev/md1 10.88TiB
>
> Metadata,DUP: Size:13.00GiB, Used:11.79GiB (90.74%)
> /dev/md1 26.00GiB
>
> System,DUP: Size:32.00MiB, Used:1.25MiB (3.91%)
> /dev/md1 64.00MiB
>
> Unallocated:
> /dev/md1 3.94GiB
>
>
>
>
>
>
> btrfs fi usage /mnt/disk2
> Overall:
> Device size: 9.09TiB
> Device allocated: 9.09TiB
> Device unallocated: 13.01MiB
> Device missing: 0.00B
> Used: 8.93TiB
> Free (estimated): 169.67GiB (min: 169.67GiB)
> Free (statfs, df): 169.68GiB
> Data ratio: 1.00
> Metadata ratio: 2.00
> Global reserve: 512.00MiB (used: 0.00B)
> Multiple profiles: no
>
> Data,single: Size:9.07TiB, Used:8.91TiB (98.17%)
> /dev/md2 9.07TiB
>
> Metadata,DUP: Size:11.00GiB, Used:9.98GiB (90.76%)
> /dev/md2 22.00GiB
>
> System,DUP: Size:8.00MiB, Used:992.00KiB (12.11%)
> /dev/md2 16.00MiB
>
> Unallocated:
> /dev/md2 13.01MiB
>
>
>
>
>
>
>
> Thanks,
> Jorge
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Why usable space can be so different?
2021-07-30 4:47 ` Andrei Borzenkov
@ 2021-07-30 7:12 ` Jorge Bastos
2021-07-30 9:11 ` Jorge Bastos
0 siblings, 1 reply; 5+ messages in thread
From: Jorge Bastos @ 2021-07-30 7:12 UTC (permalink / raw)
To: Andrei Borzenkov; +Cc: Btrfs BTRFS
On Fri, Jul 30, 2021 at 5:47 AM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
>
>
> Is you "plot" (whatever it is) 100MiB or 100GiB? Because your first
> example has 34GiB available and from your description you cannot write
> anything - but I would be very surprised if you could not write another
> 100MiB to it.
>
> 100GiB would fit your description.
>
Yes, sorry, each file is around 100GiB.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Why usable space can be so different?
2021-07-30 7:12 ` Jorge Bastos
@ 2021-07-30 9:11 ` Jorge Bastos
0 siblings, 0 replies; 5+ messages in thread
From: Jorge Bastos @ 2021-07-30 9:11 UTC (permalink / raw)
To: Andrei Borzenkov; +Cc: Btrfs BTRFS
Was going to convert the metadata do single profile since don't need
DUP for this but stupidly forgot to free up some space first, balance
crashed and fs went read_only, and strangely it does the same even
after mounting with skip_balance:
Jul 30 09:35:09 Tower1 kernel: BTRFS info (device sdz1): using free space tree
Jul 30 09:35:09 Tower1 kernel: BTRFS info (device sdz1): has skinny extents
Jul 30 09:35:13 Tower1 kernel: BTRFS info (device sdz1): balance: resume skipped
Jul 30 09:35:32 Tower1 kernel: ------------[ cut here ]------------
Jul 30 09:35:32 Tower1 kernel: BTRFS: Transaction aborted (error -28)
Jul 30 09:35:32 Tower1 kernel: WARNING: CPU: 7 PID: 4673 at
fs/btrfs/block-group.c:2122
btrfs_create_pending_block_groups+0x176/0x1ff
Jul 30 09:35:32 Tower1 kernel: Modules linked in: xt_CHECKSUM
ipt_REJECT xt_nat ip6table_mangle ip6table_nat veth iptable_mangle tun
md_mod ipmi_devintf iptable_nat xt_MASQUERADE nf_nat wireguard
curve25519_x86_64 libcurve25519_generic libchacha20poly1305
chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libblake2s
blake2s_x86_64 libblake2s_generic libchacha ip6table_filter ip6_tables
iptable_filter ip_tables mlx4_en mlx4_core igb i2c_algo_bit sb_edac
x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul
crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd
cryptd glue_helper rapl ipmi_ssif intel_cstate nvme i2c_i801
intel_uncore nvme_core ahci input_leds wmi i2c_smbus i2c_core
led_class libahci acpi_ipmi acpi_power_meter ipmi_si button acpi_pad
[last unloaded: vhost_iotlb]
Jul 30 09:35:32 Tower1 kernel: CPU: 7 PID: 4673 Comm: mc Tainted: G
W 5.10.21-Unraid #1
Jul 30 09:35:32 Tower1 kernel: Hardware name: Supermicro Super
Server/X10SRi-F, BIOS 3.3 10/28/2020
Jul 30 09:35:32 Tower1 kernel: RIP:
0010:btrfs_create_pending_block_groups+0x176/0x1ff
Jul 30 09:35:32 Tower1 kernel: Code: 45 50 48 8d b8 40 0a 00 00 e8 ae
db ff ff 84 c0 75 1a 83 fb fb 74 15 83 fb e2 74 10 89 de 48 c7 c7 da
a1 d9 81 e8 9a 65 44 00 <0f> 0b 89 d9 ba 4a 08 00 00 48 c7 c6 00 9e c2
81 48 89 ef e8 3e 91
Jul 30 09:35:32 Tower1 kernel: RSP: 0018:ffffc9000360fdd0 EFLAGS: 00010286
Jul 30 09:35:32 Tower1 kernel: RAX: 0000000000000000 RBX:
00000000ffffffe4 RCX: 0000000000000027
Jul 30 09:35:32 Tower1 kernel: RDX: 00000000ffffefff RSI:
0000000000000001 RDI: ffff88903fbd8920
Jul 30 09:35:32 Tower1 kernel: RBP: ffff88815cc28270 R08:
0000000000000000 R09: 00000000ffffefff
Jul 30 09:35:32 Tower1 kernel: R10: ffffc9000360fc00 R11:
ffffc9000360fbf8 R12: ffff888cdd644508
Jul 30 09:35:32 Tower1 kernel: R13: 0000000000000004 R14:
0000000000000000 R15: ffff8881015b6000
Jul 30 09:35:32 Tower1 kernel: FS: 000015074d988740(0000)
GS:ffff88903fbc0000(0000) knlGS:0000000000000000
Jul 30 09:35:32 Tower1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 30 09:35:32 Tower1 kernel: CR2: 00007ffcdcb4bf18 CR3:
0000000101e96001 CR4: 00000000001706e0
Jul 30 09:35:32 Tower1 kernel: Call Trace:
Jul 30 09:35:32 Tower1 kernel: __btrfs_end_transaction+0x5b/0x150
Jul 30 09:35:32 Tower1 kernel: btrfs_unlink+0x9e/0xb6
Jul 30 09:35:32 Tower1 kernel: vfs_unlink+0x82/0xeb
Jul 30 09:35:32 Tower1 kernel: do_unlinkat+0xfe/0x1d3
Jul 30 09:35:32 Tower1 kernel: do_syscall_64+0x5d/0x6a
Jul 30 09:35:32 Tower1 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
Jul 30 09:35:32 Tower1 kernel: RIP: 0033:0x15074e246277
Jul 30 09:35:32 Tower1 kernel: Code: f0 ff ff 73 01 c3 48 8b 0d 16 6c
0d 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 66
90 b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d e9 6b 0d
00 f7 d8 64 89 01 48
Jul 30 09:35:32 Tower1 kernel: RSP: 002b:00007ffcdcb4fd18 EFLAGS:
00000206 ORIG_RAX: 0000000000000057
Jul 30 09:35:32 Tower1 kernel: RAX: ffffffffffffffda RBX:
00000000006167a0 RCX: 000015074e246277
Jul 30 09:35:32 Tower1 kernel: RDX: 000000000058d130 RSI:
00000000ffffffff RDI: 00000000005d1e70
Jul 30 09:35:32 Tower1 kernel: RBP: 00000000006133b0 R08:
0000000000000001 R09: 0000000000000000
Jul 30 09:35:32 Tower1 kernel: R10: 0000000000000000 R11:
0000000000000206 R12: 00007ffcdcb4fd7c
Jul 30 09:35:32 Tower1 kernel: R13: 0000000000000000 R14:
00000000005d14e0 R15: 00000000006133b0
Jul 30 09:35:32 Tower1 kernel: ---[ end trace 9e1013fad580900f ]---
Jul 30 09:35:32 Tower1 kernel: BTRFS: error (device sdz1) in
btrfs_create_pending_block_groups:2122: errno=-28 No space left
Jul 30 09:35:32 Tower1 kernel: BTRFS info (device sdz1): forced readonly
Jul 30 09:36:34 Tower1 kernel: BTRFS info (device sdz1): using free space tree
Jul 30 09:36:34 Tower1 kernel: BTRFS info (device sdz1): has skinny extents
Jul 30 09:36:38 Tower1 kernel: BTRFS info (device sdz1): balance: resume skipped
Jul 30 09:37:13 Tower1 kernel: ------------[ cut here ]------------
I'm just going to re-format these with xfs, I don't really need btrfs
for this, just used it because it's my default fs.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-07-30 9:11 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-29 9:47 Why usable space can be so different? Jorge Bastos
2021-07-29 18:17 ` waxhead
2021-07-30 4:47 ` Andrei Borzenkov
2021-07-30 7:12 ` Jorge Bastos
2021-07-30 9:11 ` Jorge Bastos
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.