With Linux 5.5: Filesystem full while still 90 GiB free

With Linux 5.5: Filesystem full while still 90 GiB free

[Martin Steigerwald]

Hi!

I thought this would not happen anymore, but see yourself:

% LANG=en df -hT /daten
Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/sata-daten btrfs  400G  310G     0 100% /daten

I removed some larger files but to no avail.


However, also according to btrfs fi usage it is perfectly good:

% btrfs fi usage -T /daten
Overall:
    Device size:                 400.00GiB
    Device allocated:            311.04GiB
    Device unallocated:           88.96GiB
    Device missing:                  0.00B
    Used:                        309.50GiB
    Free (estimated):             90.16GiB      (min: 90.16GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              364.03MiB      (used: 0.00B)

                          Data      Metadata  System              
Id Path                   single    single    single   Unallocated
-- ---------------------- --------- --------- -------- -----------
 1 /dev/mapper/sata-daten 310.00GiB   1.01GiB 32.00MiB    88.96GiB
-- ---------------------- --------- --------- -------- -----------
   Total                  310.00GiB   1.01GiB 32.00MiB    88.96GiB
   Used                   308.80GiB 714.67MiB 64.00KiB


Even in a state where I think that no balance is required. I started one 
for some duration and it was able to relocate some block groups just 
fine:

[88593.639061] BTRFS info (device dm-4): relocating block group 
352409616384 flags data
[88594.076177] BTRFS info (device dm-4): found 1 extents
[88594.147171] BTRFS info (device dm-4): found 1 extents
[88594.185681] BTRFS info (device dm-4): relocating block group 
351335874560 flags data
[88597.089032] BTRFS info (device dm-4): found 559 extents
[88597.170087] BTRFS info (device dm-4): found 550 extents
[88597.206519] BTRFS info (device dm-4): relocating block group 
350262132736 flags data
[88601.850606] BTRFS info (device dm-4): found 1908 extents
[88601.988330] BTRFS info (device dm-4): found 1908 extents

I aborted it after some time ago, cause from the values above a balance 
is not required, I'd say.


Also btrfs check seems to be happy:

% btrfs check /dev/sata/daten
Opening filesystem to check...
Checking filesystem on /dev/sata/daten
UUID: […]
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 332322398208 bytes used, no error found
total csum bytes: 323723112
total tree bytes: 749453312
total fs tree bytes: 367476736
total extent tree bytes: 34422784
btree space waste bytes: 98074732
file data blocks allocated: 468393074688
 referenced 331312160768


I believe I switched this filesystem to space cache v2 some time ago. 
However looking at the mount options I am not entirely sure:

/dev/mapper/sata-daten on /daten type btrfs 
(rw,noatime,ssd,space_cache,subvolid=257,subvol=/daten)

I just cleared the space_cache:

% mount -o remount,clear_cache,space_cache=v2 /daten

% dmesg | tail -3
[89385.503291] BTRFS info (device dm-4): force clearing of disk cache
[89385.503302] BTRFS info (device dm-4): enabling free space tree
[89385.503305] BTRFS info (device dm-4): using free space tree

Still to no avail:

% LANG=en df -hT /daten
Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/sata-daten btrfs  400G  310G     0 100% /daten


There are no permanent snapshots on the filesystem. Backup scripts 
create a snapshot, but delete it after the backup is completely.

A scrub of the data, which I started just in case, reports no errors.

There is nothing unusual about the filesystem in /var/log/kern.log*


ThinkPad T520 running Debian Sid with self-compiled Linux kernel 5.5. 
BTRFS is on Samsung SSD 860 Pro 1 TB. btrfs-progs v5.4.1.


The filesystem mostly has large files I store there once and then they 
stay there. However recently I copied a bunch of root filesystems of a 
Proxmox VE cluster to them via rsync -aAHXS, so there are more smaller 
files on it now as well. There really is no frequent write activity on 
it. The generation number is just 5857. The filesystem is also quite 
new, having been created at 2018-08-11. No compression, no raid, no 
fancy features used.


So what is going on here?

Any way to recover the lost space… without redoing the filesystem from 
scratch? I can redo it easily enough, but if I can spare the time,I'd 
appreciate it.


I am really surprised that this bogus out of space thing can still 
happen. Especially on such an underutilized filesystem.

Thanks,
-- 
Martin

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Martin Raiber]

On 29.01.2020 20:33 Martin Steigerwald wrote:
> Hi!
>
> I thought this would not happen anymore, but see yourself:
>
> % LANG=en df -hT /daten
> Filesystem             Type   Size  Used Avail Use% Mounted on
> /dev/mapper/sata-daten btrfs  400G  310G     0 100% /daten
>
> I removed some larger files but to no avail.

I have the same issue since 5.4. This patch should fix it:
https://lore.kernel.org/linux-btrfs/f1f1a2ab-ed09-d841-6a93-a44a8fb2312f@gmx.com/T/
Confirm by writing to the file system. It shouldn't say that it is out
of space (only df report says zero).

As far as I know, it is unfortunately not fixed in any released kernel yet.

>
>
> However, also according to btrfs fi usage it is perfectly good:
>
> % btrfs fi usage -T /daten
> Overall:
>     Device size:                 400.00GiB
>     Device allocated:            311.04GiB
>     Device unallocated:           88.96GiB
>     Device missing:                  0.00B
>     Used:                        309.50GiB
>     Free (estimated):             90.16GiB      (min: 90.16GiB)
>     Data ratio:                       1.00
>     Metadata ratio:                   1.00
>     Global reserve:              364.03MiB      (used: 0.00B)
>
>                           Data      Metadata  System              
> Id Path                   single    single    single   Unallocated
> -- ---------------------- --------- --------- -------- -----------
>  1 /dev/mapper/sata-daten 310.00GiB   1.01GiB 32.00MiB    88.96GiB
> -- ---------------------- --------- --------- -------- -----------
>    Total                  310.00GiB   1.01GiB 32.00MiB    88.96GiB
>    Used                   308.80GiB 714.67MiB 64.00KiB
>
>
> Even in a state where I think that no balance is required. I started one 
> for some duration and it was able to relocate some block groups just 
> fine:
>
> [88593.639061] BTRFS info (device dm-4): relocating block group 
> 352409616384 flags data
> [88594.076177] BTRFS info (device dm-4): found 1 extents
> [88594.147171] BTRFS info (device dm-4): found 1 extents
> [88594.185681] BTRFS info (device dm-4): relocating block group 
> 351335874560 flags data
> [88597.089032] BTRFS info (device dm-4): found 559 extents
> [88597.170087] BTRFS info (device dm-4): found 550 extents
> [88597.206519] BTRFS info (device dm-4): relocating block group 
> 350262132736 flags data
> [88601.850606] BTRFS info (device dm-4): found 1908 extents
> [88601.988330] BTRFS info (device dm-4): found 1908 extents
>
> I aborted it after some time ago, cause from the values above a balance 
> is not required, I'd say.
>
>
> Also btrfs check seems to be happy:
>
> % btrfs check /dev/sata/daten
> Opening filesystem to check...
> Checking filesystem on /dev/sata/daten
> UUID: […]
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 332322398208 bytes used, no error found
> total csum bytes: 323723112
> total tree bytes: 749453312
> total fs tree bytes: 367476736
> total extent tree bytes: 34422784
> btree space waste bytes: 98074732
> file data blocks allocated: 468393074688
>  referenced 331312160768
>
>
> I believe I switched this filesystem to space cache v2 some time ago. 
> However looking at the mount options I am not entirely sure:
>
> /dev/mapper/sata-daten on /daten type btrfs 
> (rw,noatime,ssd,space_cache,subvolid=257,subvol=/daten)
>
> I just cleared the space_cache:
>
> % mount -o remount,clear_cache,space_cache=v2 /daten
>
> % dmesg | tail -3
> [89385.503291] BTRFS info (device dm-4): force clearing of disk cache
> [89385.503302] BTRFS info (device dm-4): enabling free space tree
> [89385.503305] BTRFS info (device dm-4): using free space tree
>
> Still to no avail:
>
> % LANG=en df -hT /daten
> Filesystem             Type   Size  Used Avail Use% Mounted on
> /dev/mapper/sata-daten btrfs  400G  310G     0 100% /daten
>
>
> There are no permanent snapshots on the filesystem. Backup scripts 
> create a snapshot, but delete it after the backup is completely.
>
> A scrub of the data, which I started just in case, reports no errors.
>
> There is nothing unusual about the filesystem in /var/log/kern.log*
>
>
> ThinkPad T520 running Debian Sid with self-compiled Linux kernel 5.5. 
> BTRFS is on Samsung SSD 860 Pro 1 TB. btrfs-progs v5.4.1.
>
>
> The filesystem mostly has large files I store there once and then they 
> stay there. However recently I copied a bunch of root filesystems of a 
> Proxmox VE cluster to them via rsync -aAHXS, so there are more smaller 
> files on it now as well. There really is no frequent write activity on 
> it. The generation number is just 5857. The filesystem is also quite 
> new, having been created at 2018-08-11. No compression, no raid, no 
> fancy features used.
>
>
> So what is going on here?
>
> Any way to recover the lost space… without redoing the filesystem from 
> scratch? I can redo it easily enough, but if I can spare the time,I'd 
> appreciate it.
>
>
> I am really surprised that this bogus out of space thing can still 
> happen. Especially on such an underutilized filesystem.
>
> Thanks,

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Martin Steigerwald]

Martin Raiber - 29.01.20, 21:04:41 CET:
> On 29.01.2020 20:33 Martin Steigerwald wrote:
> > I thought this would not happen anymore, but see yourself:
> > 
> > % LANG=en df -hT /daten
> > Filesystem             Type   Size  Used Avail Use% Mounted on
> > /dev/mapper/sata-daten btrfs  400G  310G     0 100% /daten
> > 
> > I removed some larger files but to no avail.
> 
> I have the same issue since 5.4. This patch should fix it:
> https://lore.kernel.org/linux-btrfs/f1f1a2ab-ed09-d841-6a93-a44a8fb231
> 2f@gmx.com/T/ Confirm by writing to the file system. It shouldn't say
> that it is out of space (only df report says zero).
> 
> As far as I know, it is unfortunately not fixed in any released kernel
> yet.

Indeed remaining metadata space in the one 1 GiB big metadata chunk is 
less than global reserve:

> > However, also according to btrfs fi usage it is perfectly good:
> > 
> > % btrfs fi usage -T /daten
> > 
> > Overall:
> >     Device size:                 400.00GiB
> >     Device allocated:            311.04GiB
> >     Device unallocated:           88.96GiB
> >     Device missing:                  0.00B
> >     Used:                        309.50GiB
> >     Free (estimated):             90.16GiB      (min: 90.16GiB)
> >     Data ratio:                       1.00
> >     Metadata ratio:                   1.00
> >     Global reserve:              364.03MiB      (used: 0.00B)
> >     
> >                           Data      Metadata  System
> > 
> > Id Path                   single    single    single   Unallocated
> > -- ---------------------- --------- --------- -------- -----------
> > 
> >  1 /dev/mapper/sata-daten 310.00GiB   1.01GiB 32.00MiB    88.96GiB
> > 
> > -- ---------------------- --------- --------- -------- -----------
> > 
> >    Total                  310.00GiB   1.01GiB 32.00MiB    88.96GiB
> >    Used                   308.80GiB 714.67MiB 64.00KiB

Hmmm, Dolphin file manager said out of space, but it may be cause it 
meanwhile checks for enough available space *before* initiating the copy 
or move operation.

Consequently using "mv" worked to move the files to that filesystem.

Thank you for that hint.

So if its just a cosmetic issue then I can wait for the patch to land in 
linux-stable. Or does it still need testing?

Best,
-- 
Martin

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Chris Murphy]

On Wed, Jan 29, 2020 at 2:20 PM Martin Steigerwald <martin@lichtvoll.de> wrote:
>
> So if its just a cosmetic issue then I can wait for the patch to land in
> linux-stable. Or does it still need testing?

I'm not seeing it in linux-next. A reasonable short term work around
is mount option 'metadata_ratio=1' and that's what needs more testing,
because it seems decently likely mortal users will need an easy work
around until a fix gets backported to stable. And that's gonna be a
while, me thinks.

Is that mount option sufficient? Or does it take a filtered balance?
What's the most minimal balance needed? I'm hoping -dlimit=1

I can't figure out a way to trigger this though, otherwise I'd be
doing more testing.


-- 
Chris Murphy

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Martin Steigerwald]

Chris Murphy - 29.01.20, 23:55:06 CET:
> On Wed, Jan 29, 2020 at 2:20 PM Martin Steigerwald 
<martin@lichtvoll.de> wrote:
> > So if its just a cosmetic issue then I can wait for the patch to
> > land in linux-stable. Or does it still need testing?
> 
> I'm not seeing it in linux-next. A reasonable short term work around
> is mount option 'metadata_ratio=1' and that's what needs more testing,
> because it seems decently likely mortal users will need an easy work
> around until a fix gets backported to stable. And that's gonna be a
> while, me thinks.
>
> Is that mount option sufficient? Or does it take a filtered balance?
> What's the most minimal balance needed? I'm hoping -dlimit=1

Does not make a difference. I did:

- mount -o remount,metadata_ratio=1 /daten
- touch /daten/somefile
- dd if=/dev/zero of=/daten/someotherfile bs=1M count=500
- sync
- df still reporting zero space free
 
> I can't figure out a way to trigger this though, otherwise I'd be
> doing more testing.

Sure.

I am doing the balance -dlimit=1 thing next. With metadata_ratio=0 
again.

% btrfs balance start -dlimit=1 /daten
Done, had to relocate 1 out of 312 chunks

% LANG=en df -hT /daten                   
Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten

Okay, doing with metadata_ratio=1:

% mount -o remount,metadata_ratio=1 /daten

% btrfs balance start -dlimit=1 /daten    
Done, had to relocate 1 out of 312 chunks

% LANG=en df -hT /daten                   
Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten


Okay, other suggestions? I'd like to avoid shuffling 311 GiB data around 
using a full balance.

Thanks,
-- 
Martin

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Chris Murphy]

On Thu, Jan 30, 2020 at 3:41 AM Martin Steigerwald <martin@lichtvoll.de> wrote:
>
> Chris Murphy - 29.01.20, 23:55:06 CET:
> > On Wed, Jan 29, 2020 at 2:20 PM Martin Steigerwald
> <martin@lichtvoll.de> wrote:
> > > So if its just a cosmetic issue then I can wait for the patch to
> > > land in linux-stable. Or does it still need testing?
> >
> > I'm not seeing it in linux-next. A reasonable short term work around
> > is mount option 'metadata_ratio=1' and that's what needs more testing,
> > because it seems decently likely mortal users will need an easy work
> > around until a fix gets backported to stable. And that's gonna be a
> > while, me thinks.
> >
> > Is that mount option sufficient? Or does it take a filtered balance?
> > What's the most minimal balance needed? I'm hoping -dlimit=1
>
> Does not make a difference. I did:
>
> - mount -o remount,metadata_ratio=1 /daten
> - touch /daten/somefile
> - dd if=/dev/zero of=/daten/someotherfile bs=1M count=500
> - sync
> - df still reporting zero space free
>
> > I can't figure out a way to trigger this though, otherwise I'd be
> > doing more testing.
>
> Sure.
>
> I am doing the balance -dlimit=1 thing next. With metadata_ratio=0
> again.
>
> % btrfs balance start -dlimit=1 /daten
> Done, had to relocate 1 out of 312 chunks
>
> % LANG=en df -hT /daten
> Filesystem             Type   Size  Used Avail Use% Mounted on
> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
>
> Okay, doing with metadata_ratio=1:
>
> % mount -o remount,metadata_ratio=1 /daten
>
> % btrfs balance start -dlimit=1 /daten
> Done, had to relocate 1 out of 312 chunks
>
> % LANG=en df -hT /daten
> Filesystem             Type   Size  Used Avail Use% Mounted on
> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
>
>
> Okay, other suggestions? I'd like to avoid shuffling 311 GiB data around
> using a full balance.

There's earlier anecdotal evidence that -dlimit=10 will work. But you
can just keep using -dlimit=1 and it'll balance a different block
group each time (you can confirm/deny this with the block group
address and extent count in dmesg for each balance). Count how many it
takes to get df to stop misreporting. It may be a file system specific
value.

-- 
Chris Murphy

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[David Sterba]

On Wed, Jan 29, 2020 at 03:55:06PM -0700, Chris Murphy wrote:
> On Wed, Jan 29, 2020 at 2:20 PM Martin Steigerwald <martin@lichtvoll.de> wrote:
> >
> > So if its just a cosmetic issue then I can wait for the patch to land in
> > linux-stable. Or does it still need testing?
> 
> I'm not seeing it in linux-next. A reasonable short term work around
> is mount option 'metadata_ratio=1' and that's what needs more testing,
> because it seems decently likely mortal users will need an easy work
> around until a fix gets backported to stable. And that's gonna be a
> while, me thinks.

We're looking into some fix that could be backported, as it affects a
long-term kernel (5.4).

The fix
https://lore.kernel.org/linux-btrfs/20200115034128.32889-1-wqu@suse.com/
IMHO works by accident and is not good even as a workaround, only papers
over the problem in some cases. The size of metadata over-reservation
(caused by a change in the logic that estimates the 'over-' part) adds
up to the global block reserve (that's permanent and as last resort
reserve for deletion).

In other words "we're making this larger by number A, so let's subtract
some number B". The fix is to use A.

> Is that mount option sufficient? Or does it take a filtered balance?
> What's the most minimal balance needed? I'm hoping -dlimit=1
> 
> I can't figure out a way to trigger this though, otherwise I'd be
> doing more testing.

I haven't checked but I think the suggested workarounds affect statfs as
a side effect. Also as the reservations are temporary, the numbers
change again after a sync.

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Chris Murphy]

On Thu, Jan 30, 2020 at 10:20 AM David Sterba <dsterba@suse.cz> wrote:
>
> On Wed, Jan 29, 2020 at 03:55:06PM -0700, Chris Murphy wrote:
> > On Wed, Jan 29, 2020 at 2:20 PM Martin Steigerwald <martin@lichtvoll.de> wrote:
> > >
> > > So if its just a cosmetic issue then I can wait for the patch to land in
> > > linux-stable. Or does it still need testing?
> >
> > I'm not seeing it in linux-next. A reasonable short term work around
> > is mount option 'metadata_ratio=1' and that's what needs more testing,
> > because it seems decently likely mortal users will need an easy work
> > around until a fix gets backported to stable. And that's gonna be a
> > while, me thinks.
>
> We're looking into some fix that could be backported, as it affects a
> long-term kernel (5.4).
>
> The fix
> https://lore.kernel.org/linux-btrfs/20200115034128.32889-1-wqu@suse.com/
> IMHO works by accident and is not good even as a workaround, only papers
> over the problem in some cases. The size of metadata over-reservation
> (caused by a change in the logic that estimates the 'over-' part) adds
> up to the global block reserve (that's permanent and as last resort
> reserve for deletion).
>
> In other words "we're making this larger by number A, so let's subtract
> some number B". The fix is to use A.
>
> > Is that mount option sufficient? Or does it take a filtered balance?
> > What's the most minimal balance needed? I'm hoping -dlimit=1
> >
> > I can't figure out a way to trigger this though, otherwise I'd be
> > doing more testing.
>
> I haven't checked but I think the suggested workarounds affect statfs as
> a side effect. Also as the reservations are temporary, the numbers
> change again after a sync.

Yeah I'm being careful to qualify to mortal users that any workarounds
are temporary and uncertain. I'm not even certain what the pattern is,
people with new file systems have hit it. A full balance seems to fix
it, and then soon after the problem happens again. I don't do any
balancing these days, for over a year now, so I wonder if that's why
I'm not seeing it.

But yeah a small number of people are hitting it, but it also stops
any program that does a free space check (presumably using statfs).

A more reliable/universal work around in the meantime is still useful;
in particular if it doesn't require changing mount options, or only
requires it temporarily (e.g. not added  to /etc/fstab, where it can
be forgotten for the life of that system).


-- 
Chris Murphy

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Martin Steigerwald]

Chris Murphy - 30.01.20, 20:31:40 CET:
> > > Is that mount option sufficient? Or does it take a filtered
> > > balance?
> > > What's the most minimal balance needed? I'm hoping -dlimit=1
> > > 
> > > I can't figure out a way to trigger this though, otherwise I'd be
> > > doing more testing.
> > 
> > I haven't checked but I think the suggested workarounds affect
> > statfs as a side effect. Also as the reservations are temporary,
> > the numbers change again after a sync.
> 
> Yeah I'm being careful to qualify to mortal users that any workarounds
> are temporary and uncertain. I'm not even certain what the pattern
> is, people with new file systems have hit it. A full balance seems to
> fix it, and then soon after the problem happens again. I don't do any
> balancing these days, for over a year now, so I wonder if that's why
> I'm not seeing it.
> 
> But yeah a small number of people are hitting it, but it also stops
> any program that does a free space check (presumably using statfs).
> 
> A more reliable/universal work around in the meantime is still useful;
> in particular if it doesn't require changing mount options, or only
> requires it temporarily (e.g. not added  to /etc/fstab, where it can
> be forgotten for the life of that system).

I did not balance either. Except maybe for a very short time during 
holding trainings in order to show to people how it works.

I never bought into balancing regularily.

Thanks,
-- 
Martin

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Martin Steigerwald]

Chris Murphy - 30.01.20, 17:37:42 CET:
> On Thu, Jan 30, 2020 at 3:41 AM Martin Steigerwald 
<martin@lichtvoll.de> wrote:
> > Chris Murphy - 29.01.20, 23:55:06 CET:
> > > On Wed, Jan 29, 2020 at 2:20 PM Martin Steigerwald
> > 
> > <martin@lichtvoll.de> wrote:
> > > > So if its just a cosmetic issue then I can wait for the patch to
> > > > land in linux-stable. Or does it still need testing?
> > > 
> > > I'm not seeing it in linux-next. A reasonable short term work
> > > around
> > > is mount option 'metadata_ratio=1' and that's what needs more
> > > testing, because it seems decently likely mortal users will need
> > > an easy work around until a fix gets backported to stable. And
> > > that's gonna be a while, me thinks.
> > > 
> > > Is that mount option sufficient? Or does it take a filtered
> > > balance?
> > > What's the most minimal balance needed? I'm hoping -dlimit=1
> > 
> > Does not make a difference. I did:
> > 
> > - mount -o remount,metadata_ratio=1 /daten
> > - touch /daten/somefile
> > - dd if=/dev/zero of=/daten/someotherfile bs=1M count=500
> > - sync
> > - df still reporting zero space free
> > 
> > > I can't figure out a way to trigger this though, otherwise I'd be
> > > doing more testing.
> > 
> > Sure.
> > 
> > I am doing the balance -dlimit=1 thing next. With metadata_ratio=0
> > again.
> > 
> > % btrfs balance start -dlimit=1 /daten
> > Done, had to relocate 1 out of 312 chunks
> > 
> > % LANG=en df -hT /daten
> > Filesystem             Type   Size  Used Avail Use% Mounted on
> > /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
> > 
> > Okay, doing with metadata_ratio=1:
> > 
> > % mount -o remount,metadata_ratio=1 /daten
> > 
> > % btrfs balance start -dlimit=1 /daten
> > Done, had to relocate 1 out of 312 chunks
> > 
> > % LANG=en df -hT /daten
> > Filesystem             Type   Size  Used Avail Use% Mounted on
> > /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
> > 
> > 
> > Okay, other suggestions? I'd like to avoid shuffling 311 GiB data
> > around using a full balance.
> 
> There's earlier anecdotal evidence that -dlimit=10 will work. But you
> can just keep using -dlimit=1 and it'll balance a different block
> group each time (you can confirm/deny this with the block group
> address and extent count in dmesg for each balance). Count how many it
> takes to get df to stop misreporting. It may be a file system
> specific value.

Lost the patience after 25 attempts:

date; let I=I+1; echo "Balance $I"; btrfs balance start -dlimit=1 /daten 
; LANG=en df -hT /daten
Do 30. Jan 20:59:17 CET 2020
Balance 25
Done, had to relocate 1 out of 312 chunks
Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten


Doing the -dlimit=10 balance now:

% btrfs balance start -dlimit=10 /daten ; LANG=en df -hT /daten
Done, had to relocate 10 out of 312 chunks
Filesystem             Type   Size  Used Avail Use% Mounted on
/dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten

Okay, enough of balancing for today.

I bet I just wait for a proper fix, instead of needlessly shuffling data 
around.

Thanks for the suggestions tough.

-- 
Martin

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Chris Murphy]

On Thu, Jan 30, 2020 at 1:02 PM Martin Steigerwald <martin@lichtvoll.de> wrote:
>
> Chris Murphy - 30.01.20, 17:37:42 CET:
> > On Thu, Jan 30, 2020 at 3:41 AM Martin Steigerwald
> <martin@lichtvoll.de> wrote:
> > > Chris Murphy - 29.01.20, 23:55:06 CET:
> > > > On Wed, Jan 29, 2020 at 2:20 PM Martin Steigerwald
> > >
> > > <martin@lichtvoll.de> wrote:
> > > > > So if its just a cosmetic issue then I can wait for the patch to
> > > > > land in linux-stable. Or does it still need testing?
> > > >
> > > > I'm not seeing it in linux-next. A reasonable short term work
> > > > around
> > > > is mount option 'metadata_ratio=1' and that's what needs more
> > > > testing, because it seems decently likely mortal users will need
> > > > an easy work around until a fix gets backported to stable. And
> > > > that's gonna be a while, me thinks.
> > > >
> > > > Is that mount option sufficient? Or does it take a filtered
> > > > balance?
> > > > What's the most minimal balance needed? I'm hoping -dlimit=1
> > >
> > > Does not make a difference. I did:
> > >
> > > - mount -o remount,metadata_ratio=1 /daten
> > > - touch /daten/somefile
> > > - dd if=/dev/zero of=/daten/someotherfile bs=1M count=500
> > > - sync
> > > - df still reporting zero space free
> > >
> > > > I can't figure out a way to trigger this though, otherwise I'd be
> > > > doing more testing.
> > >
> > > Sure.
> > >
> > > I am doing the balance -dlimit=1 thing next. With metadata_ratio=0
> > > again.
> > >
> > > % btrfs balance start -dlimit=1 /daten
> > > Done, had to relocate 1 out of 312 chunks
> > >
> > > % LANG=en df -hT /daten
> > > Filesystem             Type   Size  Used Avail Use% Mounted on
> > > /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
> > >
> > > Okay, doing with metadata_ratio=1:
> > >
> > > % mount -o remount,metadata_ratio=1 /daten
> > >
> > > % btrfs balance start -dlimit=1 /daten
> > > Done, had to relocate 1 out of 312 chunks
> > >
> > > % LANG=en df -hT /daten
> > > Filesystem             Type   Size  Used Avail Use% Mounted on
> > > /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
> > >
> > >
> > > Okay, other suggestions? I'd like to avoid shuffling 311 GiB data
> > > around using a full balance.
> >
> > There's earlier anecdotal evidence that -dlimit=10 will work. But you
> > can just keep using -dlimit=1 and it'll balance a different block
> > group each time (you can confirm/deny this with the block group
> > address and extent count in dmesg for each balance). Count how many it
> > takes to get df to stop misreporting. It may be a file system
> > specific value.
>
> Lost the patience after 25 attempts:
>
> date; let I=I+1; echo "Balance $I"; btrfs balance start -dlimit=1 /daten
> ; LANG=en df -hT /daten
> Do 30. Jan 20:59:17 CET 2020
> Balance 25
> Done, had to relocate 1 out of 312 chunks
> Filesystem             Type   Size  Used Avail Use% Mounted on
> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
>
>
> Doing the -dlimit=10 balance now:
>
> % btrfs balance start -dlimit=10 /daten ; LANG=en df -hT /daten
> Done, had to relocate 10 out of 312 chunks
> Filesystem             Type   Size  Used Avail Use% Mounted on
> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
>
> Okay, enough of balancing for today.
>
> I bet I just wait for a proper fix, instead of needlessly shuffling data
> around.

What about unmounting and remounting?

There is a proposed patch that David referenced in this thread, but
it's looking like it papers over the real problem. But even if so,
that'd get your file system working sooner than a proper fix, which I
think (?) needs to be demonstrated to at least cause no new
regressions in 5.6, before it'll be backported to stable.


-- 
Chris Murphy

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Josef Bacik]

On 1/30/20 3:18 PM, Chris Murphy wrote:
> On Thu, Jan 30, 2020 at 1:02 PM Martin Steigerwald <martin@lichtvoll.de> wrote:
>>
>> Chris Murphy - 30.01.20, 17:37:42 CET:
>>> On Thu, Jan 30, 2020 at 3:41 AM Martin Steigerwald
>> <martin@lichtvoll.de> wrote:
>>>> Chris Murphy - 29.01.20, 23:55:06 CET:
>>>>> On Wed, Jan 29, 2020 at 2:20 PM Martin Steigerwald
>>>>
>>>> <martin@lichtvoll.de> wrote:
>>>>>> So if its just a cosmetic issue then I can wait for the patch to
>>>>>> land in linux-stable. Or does it still need testing?
>>>>>
>>>>> I'm not seeing it in linux-next. A reasonable short term work
>>>>> around
>>>>> is mount option 'metadata_ratio=1' and that's what needs more
>>>>> testing, because it seems decently likely mortal users will need
>>>>> an easy work around until a fix gets backported to stable. And
>>>>> that's gonna be a while, me thinks.
>>>>>
>>>>> Is that mount option sufficient? Or does it take a filtered
>>>>> balance?
>>>>> What's the most minimal balance needed? I'm hoping -dlimit=1
>>>>
>>>> Does not make a difference. I did:
>>>>
>>>> - mount -o remount,metadata_ratio=1 /daten
>>>> - touch /daten/somefile
>>>> - dd if=/dev/zero of=/daten/someotherfile bs=1M count=500
>>>> - sync
>>>> - df still reporting zero space free
>>>>
>>>>> I can't figure out a way to trigger this though, otherwise I'd be
>>>>> doing more testing.
>>>>
>>>> Sure.
>>>>
>>>> I am doing the balance -dlimit=1 thing next. With metadata_ratio=0
>>>> again.
>>>>
>>>> % btrfs balance start -dlimit=1 /daten
>>>> Done, had to relocate 1 out of 312 chunks
>>>>
>>>> % LANG=en df -hT /daten
>>>> Filesystem             Type   Size  Used Avail Use% Mounted on
>>>> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
>>>>
>>>> Okay, doing with metadata_ratio=1:
>>>>
>>>> % mount -o remount,metadata_ratio=1 /daten
>>>>
>>>> % btrfs balance start -dlimit=1 /daten
>>>> Done, had to relocate 1 out of 312 chunks
>>>>
>>>> % LANG=en df -hT /daten
>>>> Filesystem             Type   Size  Used Avail Use% Mounted on
>>>> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
>>>>
>>>>
>>>> Okay, other suggestions? I'd like to avoid shuffling 311 GiB data
>>>> around using a full balance.
>>>
>>> There's earlier anecdotal evidence that -dlimit=10 will work. But you
>>> can just keep using -dlimit=1 and it'll balance a different block
>>> group each time (you can confirm/deny this with the block group
>>> address and extent count in dmesg for each balance). Count how many it
>>> takes to get df to stop misreporting. It may be a file system
>>> specific value.
>>
>> Lost the patience after 25 attempts:
>>
>> date; let I=I+1; echo "Balance $I"; btrfs balance start -dlimit=1 /daten
>> ; LANG=en df -hT /daten
>> Do 30. Jan 20:59:17 CET 2020
>> Balance 25
>> Done, had to relocate 1 out of 312 chunks
>> Filesystem             Type   Size  Used Avail Use% Mounted on
>> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
>>
>>
>> Doing the -dlimit=10 balance now:
>>
>> % btrfs balance start -dlimit=10 /daten ; LANG=en df -hT /daten
>> Done, had to relocate 10 out of 312 chunks
>> Filesystem             Type   Size  Used Avail Use% Mounted on
>> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
>>
>> Okay, enough of balancing for today.
>>
>> I bet I just wait for a proper fix, instead of needlessly shuffling data
>> around.
> 
> What about unmounting and remounting?
> 
> There is a proposed patch that David referenced in this thread, but
> it's looking like it papers over the real problem. But even if so,
> that'd get your file system working sooner than a proper fix, which I
> think (?) needs to be demonstrated to at least cause no new
> regressions in 5.6, before it'll be backported to stable.
> 
> 

The file system is fine, you don't need to balance or anything, this is purely a 
cosmetic bug.  _Always_ trust what btrfs filesystem usage tells you, and it's 
telling you that there's 88gib of unallocated space.  df is just wrong because 5 
years ago we arbitrarily decided to set b_avail to 0 if we didn't have enough 
metadata space for the whole global reserve, despite how much unallocated space 
we had left.  A recent changed means that we are more likely to not have enough 
free metadata space for the whole global reserve if there's unallocated space, 
specifically because we can use that unallocated space if we absolutely have to. 
  The fix will be to adjust the statfs() madness and then df will tell you the 
right thing (well as right as it can ever tell you anyway.)  Thanks,

Josef

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Chris Murphy]

On Thu, Jan 30, 2020 at 1:59 PM Josef Bacik <josef@toxicpanda.com> wrote:
>
> The file system is fine, you don't need to balance or anything, this is purely a
> cosmetic bug.

It's not entirely cosmetic if a program uses statfs to check free
space and then errors when it finds none. Some people are running into
that.

https://lore.kernel.org/linux-btrfs/alpine.DEB.2.21.99999.375.2001131514010.21037@trent.utfs.org/

I guess right now the most reliable work around is to revert to 5.3.18.


-- 
Chris Murphy

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Martin Steigerwald]

Chris Murphy - 30.01.20, 21:18:41 CET:
> On Thu, Jan 30, 2020 at 1:02 PM Martin Steigerwald 
<martin@lichtvoll.de> wrote:
> > Chris Murphy - 30.01.20, 17:37:42 CET:
> > > On Thu, Jan 30, 2020 at 3:41 AM Martin Steigerwald
> > 
> > <martin@lichtvoll.de> wrote:
> > > > Chris Murphy - 29.01.20, 23:55:06 CET:
> > > > > On Wed, Jan 29, 2020 at 2:20 PM Martin Steigerwald
> > > > 
> > > > <martin@lichtvoll.de> wrote:
> > > > > > So if its just a cosmetic issue then I can wait for the
> > > > > > patch to
> > > > > > land in linux-stable. Or does it still need testing?
> > > > > 
> > > > > I'm not seeing it in linux-next. A reasonable short term work
> > > > > around
> > > > > is mount option 'metadata_ratio=1' and that's what needs more
> > > > > testing, because it seems decently likely mortal users will
> > > > > need
> > > > > an easy work around until a fix gets backported to stable. And
> > > > > that's gonna be a while, me thinks.
> > > > > 
> > > > > Is that mount option sufficient? Or does it take a filtered
> > > > > balance?
> > > > > What's the most minimal balance needed? I'm hoping -dlimit=1
> > > > 
> > > > Does not make a difference. I did:
> > > > 
> > > > - mount -o remount,metadata_ratio=1 /daten
> > > > - touch /daten/somefile
> > > > - dd if=/dev/zero of=/daten/someotherfile bs=1M count=500
> > > > - sync
> > > > - df still reporting zero space free
> > > > 
> > > > > I can't figure out a way to trigger this though, otherwise I'd
> > > > > be
> > > > > doing more testing.
> > > > 
> > > > Sure.
> > > > 
> > > > I am doing the balance -dlimit=1 thing next. With
> > > > metadata_ratio=0
> > > > again.
> > > > 
> > > > % btrfs balance start -dlimit=1 /daten
> > > > Done, had to relocate 1 out of 312 chunks
> > > > 
> > > > % LANG=en df -hT /daten
> > > > Filesystem             Type   Size  Used Avail Use% Mounted on
> > > > /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
> > > > 
> > > > Okay, doing with metadata_ratio=1:
> > > > 
> > > > % mount -o remount,metadata_ratio=1 /daten
> > > > 
> > > > % btrfs balance start -dlimit=1 /daten
> > > > Done, had to relocate 1 out of 312 chunks
> > > > 
> > > > % LANG=en df -hT /daten
> > > > Filesystem             Type   Size  Used Avail Use% Mounted on
> > > > /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
> > > > 
> > > > 
> > > > Okay, other suggestions? I'd like to avoid shuffling 311 GiB
> > > > data
> > > > around using a full balance.
> > > 
> > > There's earlier anecdotal evidence that -dlimit=10 will work. But
> > > you
> > > can just keep using -dlimit=1 and it'll balance a different block
> > > group each time (you can confirm/deny this with the block group
> > > address and extent count in dmesg for each balance). Count how
> > > many it takes to get df to stop misreporting. It may be a file
> > > system specific value.
> > 
> > Lost the patience after 25 attempts:
> > 
> > date; let I=I+1; echo "Balance $I"; btrfs balance start -dlimit=1
> > /daten ; LANG=en df -hT /daten
> > Do 30. Jan 20:59:17 CET 2020
> > Balance 25
> > Done, had to relocate 1 out of 312 chunks
> > Filesystem             Type   Size  Used Avail Use% Mounted on
> > /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
> > 
> > 
> > Doing the -dlimit=10 balance now:
> > 
> > % btrfs balance start -dlimit=10 /daten ; LANG=en df -hT /daten
> > Done, had to relocate 10 out of 312 chunks
> > Filesystem             Type   Size  Used Avail Use% Mounted on
> > /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
> > 
> > Okay, enough of balancing for today.
> > 
> > I bet I just wait for a proper fix, instead of needlessly shuffling
> > data around.
> 
> What about unmounting and remounting?

Does not help.
 
> There is a proposed patch that David referenced in this thread, but
> it's looking like it papers over the real problem. But even if so,
> that'd get your file system working sooner than a proper fix, which I
> think (?) needs to be demonstrated to at least cause no new
> regressions in 5.6, before it'll be backported to stable.

I am done with re-balancing experiments.

-- 
Martin

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Martin Steigerwald]

Josef Bacik - 30.01.20, 21:59:31 CET:
> On 1/30/20 3:18 PM, Chris Murphy wrote:
> > On Thu, Jan 30, 2020 at 1:02 PM Martin Steigerwald 
<martin@lichtvoll.de> wrote:
> >> Chris Murphy - 30.01.20, 17:37:42 CET:
> >>> On Thu, Jan 30, 2020 at 3:41 AM Martin Steigerwald
> >> 
> >> <martin@lichtvoll.de> wrote:
> >>>> Chris Murphy - 29.01.20, 23:55:06 CET:
> >>>>> On Wed, Jan 29, 2020 at 2:20 PM Martin Steigerwald
> >>>> 
> >>>> <martin@lichtvoll.de> wrote:
> >>>>>> So if its just a cosmetic issue then I can wait for the patch
> >>>>>> to
> >>>>>> land in linux-stable. Or does it still need testing?
> >>>>> 
> >>>>> I'm not seeing it in linux-next. A reasonable short term work
> >>>>> around
> >>>>> is mount option 'metadata_ratio=1' and that's what needs more
> >>>>> testing, because it seems decently likely mortal users will need
> >>>>> an easy work around until a fix gets backported to stable. And
> >>>>> that's gonna be a while, me thinks.
> >>>>> 
> >>>>> Is that mount option sufficient? Or does it take a filtered
> >>>>> balance?
> >>>>> What's the most minimal balance needed? I'm hoping -dlimit=1
> >>>> 
> >>>> Does not make a difference. I did:
> >>>> 
> >>>> - mount -o remount,metadata_ratio=1 /daten
> >>>> - touch /daten/somefile
> >>>> - dd if=/dev/zero of=/daten/someotherfile bs=1M count=500
> >>>> - sync
> >>>> - df still reporting zero space free
> >>>> 
> >>>>> I can't figure out a way to trigger this though, otherwise I'd
> >>>>> be
> >>>>> doing more testing.
> >>>> 
> >>>> Sure.
> >>>> 
> >>>> I am doing the balance -dlimit=1 thing next. With
> >>>> metadata_ratio=0
> >>>> again.
> >>>> 
> >>>> % btrfs balance start -dlimit=1 /daten
> >>>> Done, had to relocate 1 out of 312 chunks
> >>>> 
> >>>> % LANG=en df -hT /daten
> >>>> Filesystem             Type   Size  Used Avail Use% Mounted on
> >>>> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
> >>>> 
> >>>> Okay, doing with metadata_ratio=1:
> >>>> 
> >>>> % mount -o remount,metadata_ratio=1 /daten
> >>>> 
> >>>> % btrfs balance start -dlimit=1 /daten
> >>>> Done, had to relocate 1 out of 312 chunks
> >>>> 
> >>>> % LANG=en df -hT /daten
> >>>> Filesystem             Type   Size  Used Avail Use% Mounted on
> >>>> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
> >>>> 
> >>>> 
> >>>> Okay, other suggestions? I'd like to avoid shuffling 311 GiB data
> >>>> around using a full balance.
> >>> 
> >>> There's earlier anecdotal evidence that -dlimit=10 will work. But
> >>> you
> >>> can just keep using -dlimit=1 and it'll balance a different block
> >>> group each time (you can confirm/deny this with the block group
> >>> address and extent count in dmesg for each balance). Count how
> >>> many it takes to get df to stop misreporting. It may be a file
> >>> system specific value.
> >> 
> >> Lost the patience after 25 attempts:
> >> 
> >> date; let I=I+1; echo "Balance $I"; btrfs balance start -dlimit=1
> >> /daten ; LANG=en df -hT /daten
> >> Do 30. Jan 20:59:17 CET 2020
> >> Balance 25
> >> Done, had to relocate 1 out of 312 chunks
> >> Filesystem             Type   Size  Used Avail Use% Mounted on
> >> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
> >> 
> >> 
> >> Doing the -dlimit=10 balance now:
> >> 
> >> % btrfs balance start -dlimit=10 /daten ; LANG=en df -hT /daten
> >> Done, had to relocate 10 out of 312 chunks
> >> Filesystem             Type   Size  Used Avail Use% Mounted on
> >> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
> >> 
> >> Okay, enough of balancing for today.
> >> 
> >> I bet I just wait for a proper fix, instead of needlessly shuffling
> >> data around.
> > 
> > What about unmounting and remounting?
> > 
> > There is a proposed patch that David referenced in this thread, but
> > it's looking like it papers over the real problem. But even if so,
> > that'd get your file system working sooner than a proper fix, which
> > I
> > think (?) needs to be demonstrated to at least cause no new
> > regressions in 5.6, before it'll be backported to stable.
> 
> The file system is fine, you don't need to balance or anything, this
> is purely a cosmetic bug.  _Always_ trust what btrfs filesystem usage
> tells you, and it's telling you that there's 88gib of unallocated
> space.  df is just wrong because 5 years ago we arbitrarily decided
> to set b_avail to 0 if we didn't have enough metadata space for the
> whole global reserve, despite how much unallocated space we had left.

Okay, that it what I got initially.

However then Chris suggested doing some balances thinking I was helping 
to test something that could help other users. I did not question 
whether the balances would make sense or not.

>  A recent changed means that we are more likely to not have enough
> free metadata space for the whole global reserve if there's
> unallocated space, specifically because we can use that unallocated
> space if we absolutely have to. The fix will be to adjust the
> statfs() madness and then df will tell you the right thing (well as
> right as it can ever tell you anyway.)  Thanks,

Works for me.

Thank you,
-- 
Martin

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Remi Gauvin]

[-- Attachment #1.1: Type: text/plain, Size: 858 bytes --]

On 2020-01-30 4:10 p.m., Martin Steigerwald wrote:

> 
> I am done with re-balancing experiments.
> 


It should be pretty easy to fix.. use the metadata_ratio=1 mount option,
then write enough to force the allocation of more data space,,

In your earlier attempt, you wrote 500MB, but from your btrfs filesystem
usage, you had over 1GB of allocated but unused space.

If you wrote and deleted, say, 20GB of zeroes, that should force the
allocation of metatada space to get you past the global reserve size
that is causing this bug,, (Assuming this bug is even impacting you.  I
was unclear from your messages if you are seeing any ill effects besides
the misreporting in df.)

Note: Make sure you don't have anything taking automatic snapshots
during the 20GB write/delete. I would create a new subvolume for it to
avoid that.





[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Martin Raiber]

On 30.01.2020 22:09 Chris Murphy wrote:
> On Thu, Jan 30, 2020 at 1:59 PM Josef Bacik <josef@toxicpanda.com> wrote:
>> The file system is fine, you don't need to balance or anything, this is purely a
>> cosmetic bug.
> It's not entirely cosmetic if a program uses statfs to check free
> space and then errors when it finds none. Some people are running into
> that.
>
> https://lore.kernel.org/linux-btrfs/alpine.DEB.2.21.99999.375.2001131514010.21037@trent.utfs.org/
>
> I guess right now the most reliable work around is to revert to 5.3.18.

Yeah, my backup system does check via statfs() if a file fits on storage
before backing it up and there is also a setting (global soft file
system quota) that allows users to configure the amount of storage the
backup system should use (e.g. 90% of available storage). In both cases
with statfs() returning zero it'll delete backups till there is "enough
free space", i.e. it'll delete all the backups it is allowed to delete
and then start reporting errors, which is obviously very problematic.

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Josef Bacik]

On 1/30/20 4:32 PM, Martin Raiber wrote:
> On 30.01.2020 22:09 Chris Murphy wrote:
>> On Thu, Jan 30, 2020 at 1:59 PM Josef Bacik <josef@toxicpanda.com> wrote:
>>> The file system is fine, you don't need to balance or anything, this is purely a
>>> cosmetic bug.
>> It's not entirely cosmetic if a program uses statfs to check free
>> space and then errors when it finds none. Some people are running into
>> that.
>>
>> https://lore.kernel.org/linux-btrfs/alpine.DEB.2.21.99999.375.2001131514010.21037@trent.utfs.org/
>>
>> I guess right now the most reliable work around is to revert to 5.3.18.
> 
> Yeah, my backup system does check via statfs() if a file fits on storage
> before backing it up and there is also a setting (global soft file
> system quota) that allows users to configure the amount of storage the
> backup system should use (e.g. 90% of available storage). In both cases
> with statfs() returning zero it'll delete backups till there is "enough
> free space", i.e. it'll delete all the backups it is allowed to delete
> and then start reporting errors, which is obviously very problematic.
> 

Yup I'm not meaning to say your application causing you problems isn't real ;). 
But balancing and such or using metadata_ratio isn't necessary, we just need to 
fix statfs for you so the application stops misbehaving.  Thanks,

Josef

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Martin Steigerwald]

Remi Gauvin - 30.01.20, 22:20:47 CET:
> On 2020-01-30 4:10 p.m., Martin Steigerwald wrote:
> > I am done with re-balancing experiments.
> 
> It should be pretty easy to fix.. use the metadata_ratio=1 mount
> option, then write enough to force the allocation of more data
> space,,
> 
> In your earlier attempt, you wrote 500MB, but from your btrfs
> filesystem usage, you had over 1GB of allocated but unused space.
> 
> If you wrote and deleted, say, 20GB of zeroes, that should force the
> allocation of metatada space to get you past the global reserve size
> that is causing this bug,, (Assuming this bug is even impacting you. 
> I was unclear from your messages if you are seeing any ill effects
> besides the misreporting in df.)

I thought more about writing a lot of little files as I expect that to 
use more metadata, but… I can just work around it by using command line 
tools instead of Dolphin to move data around. This is mostly my music, 
photos and so on filesystem, I do not change data on it very often, so 
that will most likely work just fine for me until there is a proper fix.

So do need to do any more things that could potentially age the 
filesystem. :)

-- 
Martin

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Matt Corallo]

This is a pretty critical regression for me. I have a few applications that regularly check space available and exit if they find low available space, as well as a number of applications that, eg, rsync small files, causing this bug to appear (even with many TB free). It looks like the suggested patch isn’t moving towards stable, is there some other patch we should be testing?

> On Jan 30, 2020, at 18:12, Martin Steigerwald <martin@lichtvoll.de> wrote:
> 
> Remi Gauvin - 30.01.20, 22:20:47 CET:
>>> On 2020-01-30 4:10 p.m., Martin Steigerwald wrote:
>>> I am done with re-balancing experiments.
>> 
>> It should be pretty easy to fix.. use the metadata_ratio=1 mount
>> option, then write enough to force the allocation of more data
>> space,,
>> 
>> In your earlier attempt, you wrote 500MB, but from your btrfs
>> filesystem usage, you had over 1GB of allocated but unused space.
>> 
>> If you wrote and deleted, say, 20GB of zeroes, that should force the
>> allocation of metatada space to get you past the global reserve size
>> that is causing this bug,, (Assuming this bug is even impacting you. 
>> I was unclear from your messages if you are seeing any ill effects
>> besides the misreporting in df.)
> 
> I thought more about writing a lot of little files as I expect that to 
> use more metadata, but… I can just work around it by using command line 
> tools instead of Dolphin to move data around. This is mostly my music, 
> photos and so on filesystem, I do not change data on it very often, so 
> that will most likely work just fine for me until there is a proper fix.
> 
> So do need to do any more things that could potentially age the 
> filesystem. :)
> 
> -- 
> Martin
> 
> 

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2252 bytes --]



On 2020/1/31 上午9:43, Matt Corallo wrote:
> This is a pretty critical regression for me. I have a few applications that regularly check space available and exit if they find low available space, as well as a number of applications that, eg, rsync small files, causing this bug to appear (even with many TB free). It looks like the suggested patch isn’t moving towards stable, is there some other patch we should be testing?

That mentioned patch is no longer maintained, since it was original
planned for a quick fix for v5.5, but extra concern about whether we
should report 0 available space when metadata is exhausted is the
blockage for merge.

The proper fix for next release can be found here:
https://github.com/adam900710/linux/tree/per_type_avail

I hope this time, the patchset can be merged without extra blockage.

Thanks,
Qu
> 
>> On Jan 30, 2020, at 18:12, Martin Steigerwald <martin@lichtvoll.de> wrote:
>>
>> Remi Gauvin - 30.01.20, 22:20:47 CET:
>>>> On 2020-01-30 4:10 p.m., Martin Steigerwald wrote:
>>>> I am done with re-balancing experiments.
>>>
>>> It should be pretty easy to fix.. use the metadata_ratio=1 mount
>>> option, then write enough to force the allocation of more data
>>> space,,
>>>
>>> In your earlier attempt, you wrote 500MB, but from your btrfs
>>> filesystem usage, you had over 1GB of allocated but unused space.
>>>
>>> If you wrote and deleted, say, 20GB of zeroes, that should force the
>>> allocation of metatada space to get you past the global reserve size
>>> that is causing this bug,, (Assuming this bug is even impacting you. 
>>> I was unclear from your messages if you are seeing any ill effects
>>> besides the misreporting in df.)
>>
>> I thought more about writing a lot of little files as I expect that to 
>> use more metadata, but… I can just work around it by using command line 
>> tools instead of Dolphin to move data around. This is mostly my music, 
>> photos and so on filesystem, I do not change data on it very often, so 
>> that will most likely work just fine for me until there is a proper fix.
>>
>> So do need to do any more things that could potentially age the 
>> filesystem. :)
>>
>> -- 
>> Martin
>>
>>
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Zygo Blaxell]

[-- Attachment #1: Type: text/plain, Size: 4219 bytes --]

On Thu, Jan 30, 2020 at 12:31:40PM -0700, Chris Murphy wrote:
> On Thu, Jan 30, 2020 at 10:20 AM David Sterba <dsterba@suse.cz> wrote:
> >
> > On Wed, Jan 29, 2020 at 03:55:06PM -0700, Chris Murphy wrote:
> > > On Wed, Jan 29, 2020 at 2:20 PM Martin Steigerwald <martin@lichtvoll.de> wrote:
> > > >
> > > > So if its just a cosmetic issue then I can wait for the patch to land in
> > > > linux-stable. Or does it still need testing?
> > >
> > > I'm not seeing it in linux-next. A reasonable short term work around
> > > is mount option 'metadata_ratio=1' and that's what needs more testing,
> > > because it seems decently likely mortal users will need an easy work
> > > around until a fix gets backported to stable. And that's gonna be a
> > > while, me thinks.
> >
> > We're looking into some fix that could be backported, as it affects a
> > long-term kernel (5.4).
> >
> > The fix
> > https://lore.kernel.org/linux-btrfs/20200115034128.32889-1-wqu@suse.com/
> > IMHO works by accident and is not good even as a workaround, only papers
> > over the problem in some cases. The size of metadata over-reservation
> > (caused by a change in the logic that estimates the 'over-' part) adds
> > up to the global block reserve (that's permanent and as last resort
> > reserve for deletion).
> >
> > In other words "we're making this larger by number A, so let's subtract
> > some number B". The fix is to use A.
> >
> > > Is that mount option sufficient? Or does it take a filtered balance?
> > > What's the most minimal balance needed? I'm hoping -dlimit=1
> > >
> > > I can't figure out a way to trigger this though, otherwise I'd be
> > > doing more testing.
> >
> > I haven't checked but I think the suggested workarounds affect statfs as
> > a side effect. Also as the reservations are temporary, the numbers
> > change again after a sync.
> 
> Yeah I'm being careful to qualify to mortal users that any workarounds
> are temporary and uncertain. I'm not even certain what the pattern is,
> people with new file systems have hit it. A full balance seems to fix
> it, and then soon after the problem happens again. I don't do any
> balancing these days, for over a year now, so I wonder if that's why
> I'm not seeing it.

I had to intentionally balance metadata to trigger the bug on pre-existing
test filesystems.  With new filesystems it's easy, I hit it every time
the last metadata block group is half full (assuming default BG size of
1GB and max global reserved size of 512MB).  It goes away when a new
metadata BG is allocated, then comes back again later.  Sometimes it
appears and disappears rapidly while doing a large file tree copy.

An older filesystem will have some GB of allocated but partially empty
metadata BGs, and won't hit this condition unless you run metadata balance
(which shrinks metadata), or do something that causes explosive metadata
growth.  If you normally keep a healthy amount of allocated but unused
metadata space, you probably will never hit the bug.

> But yeah a small number of people are hitting it, but it also stops
> any program that does a free space check (presumably using statfs).
> 
> A more reliable/universal work around in the meantime is still useful;
> in particular if it doesn't require changing mount options, or only
> requires it temporarily (e.g. not added  to /etc/fstab, where it can
> be forgotten for the life of that system).

You can create a GB of allocated but unused metadata space with something
like:

	btrfs sub create sub_tmp
	mkdir sub_tmp/single
	head -c 2047 /dev/urandom > sub_tmp/single/inline_file
	for x in $(seq 1 18); do 
		cp -a sub_tmp/single sub_tmp/double
		mv sub_tmp/double sub_tmp/single/$x
	done
	sync
	btrfs sub del sub_tmp

This requires the max_inline mount option to be set to the default (2048).
Random data means it works well when compression is enabled too.

Do not balance metadata until the bug is fixed.  Balancing metadata will
release the allocated but unused metadata space, possibly retriggering
the bug.

(Hmmm...the above script is also a surprisingly effective commit latency
test case...)

> 
> -- 
> Chris Murphy

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Etienne Champetier]

Le jeu. 30 janv. 2020 à 20:53, Matt Corallo <kernel@bluematt.me> a écrit :
>
> This is a pretty critical regression for me. I have a few applications that regularly check space available and exit if they find low available space, as well as a number of applications that, eg, rsync small files, causing this bug to appear (even with many TB free). It looks like the suggested patch isn’t moving towards stable, is there some other patch we should be testing?

I was migrating some data over to a new Fedora 31 server with btrfs
when dnf complained that my disk were full, despite having 2.61TiB
unallocated
Once a new kernel with a fix is available Fedora systems will need
some manual work to recover.

Running transaction test
The downloaded packages were saved in cache until the next successful
transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Transaction test error:
installing package libev-4.27-1.fc31.x86_64 needs 160KB on the / filesystem
...

# uname -r
5.4.7-200.fc31.x86_64
# LANG=C df -hT /
Filesystem                                            Type   Size
Used Avail Use% Mounted on
/dev/mapper/luks-17a54b4f-58b1-447f-a1e2-f927503936f2 btrfs  3.7T
1.1T     0 100% /

# btrfs fi usage -T /
Overall:
    Device size:           3.63TiB
    Device allocated:           1.03TiB
    Device unallocated:           2.61TiB
    Device missing:             0.00B
    Used:               1.02TiB
    Free (estimated):           2.61TiB    (min: 1.31TiB)
    Data ratio:                  1.00
    Metadata ratio:              2.00
    Global reserve:         512.00MiB    (used: 0.00B)

                                                         Data
Metadata System
Id Path                                                  RAID0
RAID1    RAID1    Unallocated
-- ----------------------------------------------------- ---------
-------- -------- -----------
 1 /dev/mapper/luks-17a54b4f-58b1-447f-a1e2-f927503936f2 522.00GiB
3.00GiB  8.00MiB     1.30TiB
 2 /dev/mapper/luks-ee555ebd-59ca-4ed8-9d68-7c3faf6e6a25 522.00GiB
3.00GiB  8.00MiB     1.30TiB
-- ----------------------------------------------------- ---------
-------- -------- -----------
   Total                                                   1.02TiB
3.00GiB  8.00MiB     2.61TiB
   Used                                                    1.02TiB
2.75GiB 96.00KiB

The good news is that I saw that I messed up my kickstart installed
and ended up with RAID0 for data

>
> > On Jan 30, 2020, at 18:12, Martin Steigerwald <martin@lichtvoll.de> wrote:
> >
> > Remi Gauvin - 30.01.20, 22:20:47 CET:
> >>> On 2020-01-30 4:10 p.m., Martin Steigerwald wrote:
> >>> I am done with re-balancing experiments.
> >>
> >> It should be pretty easy to fix.. use the metadata_ratio=1 mount
> >> option, then write enough to force the allocation of more data
> >> space,,
> >>
> >> In your earlier attempt, you wrote 500MB, but from your btrfs
> >> filesystem usage, you had over 1GB of allocated but unused space.
> >>
> >> If you wrote and deleted, say, 20GB of zeroes, that should force the
> >> allocation of metatada space to get you past the global reserve size
> >> that is causing this bug,, (Assuming this bug is even impacting you.
> >> I was unclear from your messages if you are seeing any ill effects
> >> besides the misreporting in df.)
> >
> > I thought more about writing a lot of little files as I expect that to
> > use more metadata, but… I can just work around it by using command line
> > tools instead of Dolphin to move data around. This is mostly my music,
> > photos and so on filesystem, I do not change data on it very often, so
> > that will most likely work just fine for me until there is a proper fix.
> >
> > So do need to do any more things that could potentially age the
> > filesystem. :)
> >
> > --
> > Martin
> >
> >
>

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Etienne Champetier]

Hello Qu,

Le jeu. 30 janv. 2020 à 20:57, Qu Wenruo <quwenruo.btrfs@gmx.com> a écrit :
>
>
>
> On 2020/1/31 上午9:43, Matt Corallo wrote:
> > This is a pretty critical regression for me. I have a few applications that regularly check space available and exit if they find low available space, as well as a number of applications that, eg, rsync small files, causing this bug to appear (even with many TB free). It looks like the suggested patch isn’t moving towards stable, is there some other patch we should be testing?
>
> That mentioned patch is no longer maintained, since it was original
> planned for a quick fix for v5.5, but extra concern about whether we
> should report 0 available space when metadata is exhausted is the
> blockage for merge.
>
> The proper fix for next release can be found here:
> https://github.com/adam900710/linux/tree/per_type_avail
>
> I hope this time, the patchset can be merged without extra blockage.

I see that there is a v8 of this patchset but can't find it in 5.6, is
it targetted at 5.7 now ?
https://patchwork.kernel.org/project/linux-btrfs/list/?series=245113

Thanks
Etienne

> Thanks,
> Qu
> >
> >> On Jan 30, 2020, at 18:12, Martin Steigerwald <martin@lichtvoll.de> wrote:
> >>
> >> Remi Gauvin - 30.01.20, 22:20:47 CET:
> >>>> On 2020-01-30 4:10 p.m., Martin Steigerwald wrote:
> >>>> I am done with re-balancing experiments.
> >>>
> >>> It should be pretty easy to fix.. use the metadata_ratio=1 mount
> >>> option, then write enough to force the allocation of more data
> >>> space,,
> >>>
> >>> In your earlier attempt, you wrote 500MB, but from your btrfs
> >>> filesystem usage, you had over 1GB of allocated but unused space.
> >>>
> >>> If you wrote and deleted, say, 20GB of zeroes, that should force the
> >>> allocation of metatada space to get you past the global reserve size
> >>> that is causing this bug,, (Assuming this bug is even impacting you.
> >>> I was unclear from your messages if you are seeing any ill effects
> >>> besides the misreporting in df.)
> >>
> >> I thought more about writing a lot of little files as I expect that to
> >> use more metadata, but… I can just work around it by using command line
> >> tools instead of Dolphin to move data around. This is mostly my music,
> >> photos and so on filesystem, I do not change data on it very often, so
> >> that will most likely work just fine for me until there is a proper fix.
> >>
> >> So do need to do any more things that could potentially age the
> >> filesystem. :)
> >>
> >> --
> >> Martin
> >>
> >>
> >
>

Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2835 bytes --]



On 2020/3/2 上午9:57, Etienne Champetier wrote:
> Hello Qu,
> 
> Le jeu. 30 janv. 2020 à 20:57, Qu Wenruo <quwenruo.btrfs@gmx.com> a écrit :
>>
>>
>>
>> On 2020/1/31 上午9:43, Matt Corallo wrote:
>>> This is a pretty critical regression for me. I have a few applications that regularly check space available and exit if they find low available space, as well as a number of applications that, eg, rsync small files, causing this bug to appear (even with many TB free). It looks like the suggested patch isn’t moving towards stable, is there some other patch we should be testing?
>>
>> That mentioned patch is no longer maintained, since it was original
>> planned for a quick fix for v5.5, but extra concern about whether we
>> should report 0 available space when metadata is exhausted is the
>> blockage for merge.
>>
>> The proper fix for next release can be found here:
>> https://github.com/adam900710/linux/tree/per_type_avail
>>
>> I hope this time, the patchset can be merged without extra blockage.
> 
> I see that there is a v8 of this patchset but can't find it in 5.6, is
> it targetted at 5.7 now ?
> https://patchwork.kernel.org/project/linux-btrfs/list/?series=245113

That's to be determined by David, I hope it could reach v5.7, but I
don't have the final say.

Thanks,
Qu

> 
> Thanks
> Etienne
> 
>> Thanks,
>> Qu
>>>
>>>> On Jan 30, 2020, at 18:12, Martin Steigerwald <martin@lichtvoll.de> wrote:
>>>>
>>>> Remi Gauvin - 30.01.20, 22:20:47 CET:
>>>>>> On 2020-01-30 4:10 p.m., Martin Steigerwald wrote:
>>>>>> I am done with re-balancing experiments.
>>>>>
>>>>> It should be pretty easy to fix.. use the metadata_ratio=1 mount
>>>>> option, then write enough to force the allocation of more data
>>>>> space,,
>>>>>
>>>>> In your earlier attempt, you wrote 500MB, but from your btrfs
>>>>> filesystem usage, you had over 1GB of allocated but unused space.
>>>>>
>>>>> If you wrote and deleted, say, 20GB of zeroes, that should force the
>>>>> allocation of metatada space to get you past the global reserve size
>>>>> that is causing this bug,, (Assuming this bug is even impacting you.
>>>>> I was unclear from your messages if you are seeing any ill effects
>>>>> besides the misreporting in df.)
>>>>
>>>> I thought more about writing a lot of little files as I expect that to
>>>> use more metadata, but… I can just work around it by using command line
>>>> tools instead of Dolphin to move data around. This is mostly my music,
>>>> photos and so on filesystem, I do not change data on it very often, so
>>>> that will most likely work just fine for me until there is a proper fix.
>>>>
>>>> So do need to do any more things that could potentially age the
>>>> filesystem. :)
>>>>
>>>> --
>>>> Martin
>>>>
>>>>
>>>
>>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]