repeated enospc errors during balance on a filesystem with spare room - pls advise

repeated enospc errors during balance on a filesystem with spare room - pls advise

[Ole Langbehn]

[-- Attachment #1.1: Type: text/plain, Size: 2015 bytes --]

Hi,

I have done three full balances in a row, each of them ending with an
error, telling me:

BTRFS info (device nvme1n1p1): 2 enospc errors during balance
BTRFS info (device nvme1n1p1): balance: ended with status: -28

(first balance run it was 4 enospc errors).

The filesystem has enough space to spare, though:

# btrfs fi show /
Label: none  uuid: 34ea0387-af9a-43b3-b7cc-7bdf7b37b8f1
        Total devices 1 FS bytes used 624.36GiB
        devid    1 size 931.51GiB used 627.03GiB path /dev/nvme1n1p1

# btrfs fi df /
Data, single: total=614.00GiB, used=613.72GiB
System, single: total=32.00MiB, used=112.00KiB
Metadata, single: total=13.00GiB, used=10.64GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

This is after the balances, but was about the same before the balances.
Before them, data had about 50GB diff between total and used.

The volume contains subvolumes (/ and /home) and snapshots (around 20
per subvolume, 40 total, oldest 1 month old).

My questions are:

1. why do I get enospc errors on a device that has enough spare space?
2. is this bad and if yes, how can I fix it?



A little more (noteworthy) context, if you're interested:

The reason I started the first balance was that a df on the filesystem
showed 0% free space:

# df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/nvme1n1p1 976760584 655217424 	   0 100% /
...

and a big download (chromium sources) was aborted due to "not enough
space on device".

I monitored the first balance more closely, and right after the start,
df looked normal again, showing available blocks, but during the
balance, it flip-flopped a couple of times between again showing 0
available bytes and showing the complement between actual size and used
bytes. I did not observe this behavior any more during balance 2 and 3,
but did not observe as closely.

TiA for any insights and ideas on how to proceed and a healthy start
into the new year for everyone.





[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: repeated enospc errors during balance on a filesystem with spare room - pls advise

[Ole Langbehn]

[-- Attachment #1.1: Type: text/plain, Size: 252 bytes --]

Excuse me for adding more information in a second mail:

# uname -a
Linux leo 5.4.6-gentoo #1 SMP PREEMPT Sun Dec 22 10:27:05 CET 2019
x86_64 Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz GenuineIntel GNU/Linux

# btrfs --version
btrfs-progs v5.4


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: repeated enospc errors during balance on a filesystem with spare room - pls advise

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 2972 bytes --]

On Tue, Dec 31, 2019 at 04:10:40PM +0100, Ole Langbehn wrote:
> Excuse me for adding more information in a second mail:
> 
> # uname -a
> Linux leo 5.4.6-gentoo #1 SMP PREEMPT Sun Dec 22 10:27:05 CET 2019

That is the most likely problem.  A fix for 5.4's zero-free-space issue
is still being reviewed on the mailing list.

On btrfs you can run out of space as often as you like, it won't hurt
data on the filesystem.  A lot of applications are not so robust, though,
so you may encounter problems with embedded databases and configuration
data.  The current 5.4 bug just makes the kernel think there is 0 free
space, it has nothing to do with the filesystem on disk.

Two workarounds:

	- use 5.3.18 instead of 5.4.6

	- use the 'metadata_ratio=1' mount option after balancing a few
	data block groups

In your other mail you indicated you were running a full balance.  Full
balances are never useful(*) and will make this specific situation worse.

A full balance includes a metadata balance.  The primary effect
of metadata balance is to temporarily reduce space for metadata.
Reducing metadata space causes an assortment of problems for btrfs,
only one of which is hitting the 5.4 free space bug.  For all but a few
contrived test cases, btrfs manages metadata well without interference
from balance.  Too much metadata balancing (i.e. any at all) can make
a filesystem truly run out of metadata space on disk--a condition that
is sometimes difficult to reverse.

You should run data balances to temporarily reduce space available
for data.  This will allow btrfs to use more space for metadata,
which will avoid the bad things that happen when metadata runs out.
This will also compact free space areas into large contiguous areas for
future data allocations, which will improve fragmentation and may make
the filesystem run a little faster.

In your specific case, you probably have to do the first few block groups
of data balance on a 5.3 kernel until you have enough unallocated space
to make new metadata block groups.

Going forward, run something like 'btrfs balance start -dlimit=9' every
day (adjust the limit upwards if you write more than 10GB of data per
day, downwards if you write less).  Or use 'btrfs fi usage' and make
sure that the 'unallocated' space stays above a few GB at all times,
and run data balance with larger limits if it's not.

(*) A full balance contains a metadata balance, and you should never
balance metadata except to convert from one RAID profile to another.
Older kernels have bugs that lead to overallocation of metadata, but it
is better to upgrade than to continue using the old kernels with metadata
balances.  There are some obscure corner test cases where a metadata
balance is useful, but users will not encounter these by accident.

> x86_64 Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz GenuineIntel GNU/Linux
> 
> # btrfs --version
> btrfs-progs v5.4
> 




[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: repeated enospc errors during balance on a filesystem with spare room - pls advise

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2524 bytes --]



On 2019/12/31 下午11:04, Ole Langbehn wrote:
> Hi,
> 
> I have done three full balances in a row, each of them ending with an
> error, telling me:
> 
> BTRFS info (device nvme1n1p1): 2 enospc errors during balance
> BTRFS info (device nvme1n1p1): balance: ended with status: -28
> 
> (first balance run it was 4 enospc errors).
> 
> The filesystem has enough space to spare, though:
> 
> # btrfs fi show /
> Label: none  uuid: 34ea0387-af9a-43b3-b7cc-7bdf7b37b8f1
>         Total devices 1 FS bytes used 624.36GiB
>         devid    1 size 931.51GiB used 627.03GiB path /dev/nvme1n1p1
> 
> # btrfs fi df /
> Data, single: total=614.00GiB, used=613.72GiB
> System, single: total=32.00MiB, used=112.00KiB
> Metadata, single: total=13.00GiB, used=10.64GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> This is after the balances, but was about the same before the balances.
> Before them, data had about 50GB diff between total and used.
> 
> The volume contains subvolumes (/ and /home) and snapshots (around 20
> per subvolume, 40 total, oldest 1 month old).
> 
> My questions are:
> 
> 1. why do I get enospc errors on a device that has enough spare space?

A known bug in v5.4, where extra space check in relocation doesn't match
the new metadata over-commit.

> 2. is this bad and if yes, how can I fix it?

There are patches for that:
https://patchwork.kernel.org/project/linux-btrfs/list/?series=208445

> 
> 
> 
> A little more (noteworthy) context, if you're interested:
> 
> The reason I started the first balance was that a df on the filesystem
> showed 0% free space:
> 
> # df
> Filesystem     1K-blocks      Used Available Use% Mounted on
> /dev/nvme1n1p1 976760584 655217424 	   0 100% /

Another known bug, which we also had a patch for it:
https://patchwork.kernel.org/patch/11293419/

Thanks,
Qu

> ...
> 
> and a big download (chromium sources) was aborted due to "not enough
> space on device".
> 
> I monitored the first balance more closely, and right after the start,
> df looked normal again, showing available blocks, but during the
> balance, it flip-flopped a couple of times between again showing 0
> available bytes and showing the complement between actual size and used
> bytes. I did not observe this behavior any more during balance 2 and 3,
> but did not observe as closely.
> 
> TiA for any insights and ideas on how to proceed and a healthy start
> into the new year for everyone.
> 
> 
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

Re: repeated enospc errors during balance on a filesystem with spare room - pls advise

[Cerem Cem ASLAN]

I learned a lot from previous replies. However, I had been in a very
similar situation recently and I followed
http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
so I added a pendrive to create extra space in order to be able to run
btrfs balance. The extra space was not sufficient and I lost the
ability to remove the pendrive. So I added another 1TB external disk,
then removed the pendrive with `btrfs dev remove`, run the `btrfs
balance`, then converted the profile to single/dup and then finally
removed the external disk. Cost my 3 full days to overcome the issue.
I'm posting this experience as a warning :)

https://unix.stackexchange.com/q/558363/65781

Qu Wenruo <quwenruo.btrfs@gmx.com>, 1 Oca 2020 Çar, 03:13 tarihinde şunu yazdı:
>
>
>
> On 2019/12/31 下午11:04, Ole Langbehn wrote:
> > Hi,
> >
> > I have done three full balances in a row, each of them ending with an
> > error, telling me:
> >
> > BTRFS info (device nvme1n1p1): 2 enospc errors during balance
> > BTRFS info (device nvme1n1p1): balance: ended with status: -28
> >
> > (first balance run it was 4 enospc errors).
> >
> > The filesystem has enough space to spare, though:
> >
> > # btrfs fi show /
> > Label: none  uuid: 34ea0387-af9a-43b3-b7cc-7bdf7b37b8f1
> >         Total devices 1 FS bytes used 624.36GiB
> >         devid    1 size 931.51GiB used 627.03GiB path /dev/nvme1n1p1
> >
> > # btrfs fi df /
> > Data, single: total=614.00GiB, used=613.72GiB
> > System, single: total=32.00MiB, used=112.00KiB
> > Metadata, single: total=13.00GiB, used=10.64GiB
> > GlobalReserve, single: total=512.00MiB, used=0.00B
> >
> > This is after the balances, but was about the same before the balances.
> > Before them, data had about 50GB diff between total and used.
> >
> > The volume contains subvolumes (/ and /home) and snapshots (around 20
> > per subvolume, 40 total, oldest 1 month old).
> >
> > My questions are:
> >
> > 1. why do I get enospc errors on a device that has enough spare space?
>
> A known bug in v5.4, where extra space check in relocation doesn't match
> the new metadata over-commit.
>
> > 2. is this bad and if yes, how can I fix it?
>
> There are patches for that:
> https://patchwork.kernel.org/project/linux-btrfs/list/?series=208445
>
> >
> >
> >
> > A little more (noteworthy) context, if you're interested:
> >
> > The reason I started the first balance was that a df on the filesystem
> > showed 0% free space:
> >
> > # df
> > Filesystem     1K-blocks      Used Available Use% Mounted on
> > /dev/nvme1n1p1 976760584 655217424       0 100% /
>
> Another known bug, which we also had a patch for it:
> https://patchwork.kernel.org/patch/11293419/
>
> Thanks,
> Qu
>
> > ...
> >
> > and a big download (chromium sources) was aborted due to "not enough
> > space on device".
> >
> > I monitored the first balance more closely, and right after the start,
> > df looked normal again, showing available blocks, but during the
> > balance, it flip-flopped a couple of times between again showing 0
> > available bytes and showing the complement between actual size and used
> > bytes. I did not observe this behavior any more during balance 2 and 3,
> > but did not observe as closely.
> >
> > TiA for any insights and ideas on how to proceed and a healthy start
> > into the new year for everyone.
> >
> >
> >
> >
>

Re: repeated enospc errors during balance on a filesystem with spare room - pls advise

[Swâmi Petaramesh]

Hi list,

Le 31/12/2019 à 23:58, Zygo Blaxell a écrit :
> A full balance includes a metadata balance.  The primary effect
> of metadata balance is to temporarily reduce space for metadata.
> Reducing metadata space causes an assortment of problems for btrfs,
> only one of which is hitting the 5.4 free space bug.  For all but a few
> contrived test cases, btrfs manages metadata well without interference
> from balance.  Too much metadata balancing (i.e. any at all) can make
> a filesystem truly run out of metadata space on disk--a condition that
> is sometimes difficult to reverse.

I however was hit by the "dummy zero free space condition" using 5.4 on
a machine, and the resulting filesystem (on an external HD) then still
showed 0% free using a 5.3 or a 4.15 kernel on other machines.

It however passsed "btrfs check" without any error.

The thing that fixed it and returned it to a « normal working state »
has been running a "btrfs balance -m" (on the 5.4 machine) on it.

So I'm a bit puzzled reading that metadata balance is useless when it
precisely fixed this issue on a FS here.

Kind regards (and best wishes for a happy new year).

Re: repeated enospc errors during balance on a filesystem with spare room - pls advise

[Ole Langbehn]

[-- Attachment #1.1: Type: text/plain, Size: 1555 bytes --]

Thanks for the insights (also in other responses).

On 31.12.19 23:58, Zygo Blaxell wrote:
> Two workarounds:
> 
> 	- use 5.3.18 instead of 5.4.6
> 
> 	- use the 'metadata_ratio=1' mount option after balancing a few
> 	data block groups

Switched back to 5.3.18. Since the space bug (at least the external/df
part) was solved for me before switching, I did not need to do anything
further. A full(100%) data balance succeeds successfully now.

> In your other mail you indicated you were running a full balance.  Full
> balances are never useful(*) and will make this specific situation worse.
> 
> A full balance includes a metadata balance.  The primary effect
> of metadata balance is to temporarily reduce space for metadata.
> Reducing metadata space causes an assortment of problems for btrfs,
> only one of which is hitting the 5.4 free space bug.  For all but a few
> contrived test cases, btrfs manages metadata well without interference
> from balance.  Too much metadata balancing (i.e. any at all) can make
> a filesystem truly run out of metadata space on disk--a condition that
> is sometimes difficult to reverse.

In my situation, my observation was that when I hit the space bug, I had
12GB meta reserved, 10.x used. The first balance lifted reserved space
up to 13GB and I have not seen the issue afterwards. So from practice/my
observations, it fixed the issue (also see Swamis response). I will take
into account your insights in the future, though, and try to not balance
metadata.

Cheers,

Ole


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: repeated enospc errors during balance on a filesystem with spare room - pls advise

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 4223 bytes --]

On Wed, Jan 01, 2020 at 11:05:00AM +0100, Swâmi Petaramesh wrote:
> Hi list,
> 
> Le 31/12/2019 à 23:58, Zygo Blaxell a écrit :
> > A full balance includes a metadata balance.  The primary effect
> > of metadata balance is to temporarily reduce space for metadata.
> > Reducing metadata space causes an assortment of problems for btrfs,
> > only one of which is hitting the 5.4 free space bug.  For all but a few
> > contrived test cases, btrfs manages metadata well without interference
> > from balance.  Too much metadata balancing (i.e. any at all) can make
> > a filesystem truly run out of metadata space on disk--a condition that
> > is sometimes difficult to reverse.
> 
> I however was hit by the "dummy zero free space condition" using 5.4 on
> a machine, and the resulting filesystem (on an external HD) then still
> showed 0% free using a 5.3 or a 4.15 kernel on other machines.

Interesting...so 5.4 can put the filesystem into a _persistent_ state
where the metadata allocator makes no further progress.  Two bugs in one!

I do note from the git history that metadata free space calculations
seem to have been dodgy for years.  The code that is touched from the
5.4 kernel bug fix patch dates back to v4.5, so probably all kernels
newer than 2015 have some metadata allocation bugs (kernels up to 4.4
would hit ENOSPC while reporting non-zero free space reported in df,
kernels 4.5 and later report zero free space in df first, then ENOSPC).
5.4 has other changes that make these problems more frequent, but these
are not new bugs.

The longer patch series Qu mentioned looks like it's trying to properly
solve the problem instead of using a quick guesstimate.

> It however passsed "btrfs check" without any error.

Not surprising.  "btrfs check" cannot detect kernel bugs.

"btrfs check" is concerned with on-disk data, while free space
calculations only exist in kernel memory structures.  btrfs check verifies
the metadata is correct, but check can't tell you if the kernel will read
the metadata and then just return the wrong free space number.

> The thing that fixed it and returned it to a « normal working state »
> has been running a "btrfs balance -m" (on the 5.4 machine) on it.
> 
> So I'm a bit puzzled reading that metadata balance is useless when it
> precisely fixed this issue on a FS here.

The kernel bug is triggered under very specific conditions.  Almost any
change to the filesystem could change those conditions to temporarily work
around the kernel bug.  Metadata balance is a change to the filesystem,
so sometimes it works--but it can also trigger the problem in a filesystem
that didn't have the problem before.

Other things that randomly allocate metadata, and might by chance
temporarily fix this problem without the bad side-effects of metadata
balances:

	- metadata defrag (btrfs fi defrag on a subvol without -r)

	- metadata modifications (deleting or touching files,
	creating and deleting empty directories)

	- running a data balance (allocates a lot of metadata)

	- using the metadata_ratio mount option

For 5 years our regression tests have been running on filesystems
that never run metadata balances.  We don't run metadata balances in
the test workloads or in production because 1) in production we never
have a requirement to balance metadata on any workload except for raid
conversions, and 2) balancing metadata is a pretty reliable reproducer
for the correct, non-bug metadata ENOSPC condition that a btrfs gets
into when it runs out of both metadata and unallocated space.  This is
especially bad on filesystems with less than ~250GB of free space, where
rounding to GB-sized block groups and free space fragmentation in data
block groups combine to make it very important to have a lot of slack
in metadata block groups.

We've probably been avoiding all the false-zero-free-space conditions
in kernels 4.5+ accidentally, simply as a side-effect of never running a
metadata balance.  To reproduce the 5.4 issue, I had to set up a custom
test workload which did metadata balances to force the bug to happen.

> Kind regards (and best wishes for a happy new year).
> 
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]