Re: ENOSPC while df shows 826.93GiB free

[Christoph Anton Mitterer <calestyo@scientia.org>]

On Tue, 2021-12-07 at 11:29 +0800, Qu Wenruo wrote:
> For other regular operations, you either got ENOSPC just like all
> other
> fses which runs out of space, or do it without problem.
> 
> Furthermore, balance in this case is not really the preferred way to
> free up space, really freeing up data is the correct way to go.

Well but to be honest... that makes btrfs kinda broke for that
particular purpose.


The software which runs on the storage and provides the data to the
experiments does in fact make sure that the space isn't fully used (per
default, it leave a gap of 4GB).

While this gap is configurable it seems a bit odd if one would have to
set it to ~1TB per fs... just to make sure that btrfs doesn't run out
of space for metadata.


And btrfs *does* show that plenty of space is left (always around 700-
800 GB)... so the application thinks it can happily continue to write,
while in fact it fails (and the cannot even start anymore as it fails
to create lock files).


My understanding was the when not using --mixed, btrfs has block groups
for data and metadata.

And it seems here that the data block groups have several 100 GB still
free, while - AFAIU you - the metadata block groups are already full.



I also wouldn't want to regularly balance (which doesn't really seem to
help that much so far)... cause it puts quite some IO load on the
systems.


So if csum data needs so much space... why can't it simply reserve e.g.
60 GB for metadata instead of just 17 GB?



If I really had to reserve ~ 1TB of storage to be unused (per 16TB fs)
just to get that working... I would need to move stuff back to ext4,
cause that's such a big loss we couldn't justify to our funding
agencies.


And we haven't had that issue with e.g. ext4 ... that seems to reserve
just enough for meta, so that we could basically fill up the fs close
to the end.



Cheers,
Chris.