Kernel 5.4 - BTRFS FS shows full with about 600 GB Free ?

Kernel 5.4 - BTRFS FS shows full with about 600 GB Free ?

[Swâmi Petaramesh]

Hi list,

After writing files and snapshots WITHOUT errors on an external BTRFS FS 
with 500+ GB of free space, using kernel 5.4.5-arch1-1, I dismount the 
FS then remount it normally, and then it says the FS has 0 space free left !

Checking the disk on another machine with

[moksha ~]# uname -r
5.4.2-1-MANJARO

And... How can this be ?

root@moksha:~# df -h /run/media/myself/MyVolume
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/luks-xxxxxxxxx-yyyy-zzzz-tttt-wwwwwww   1,9T 1,2T     0 100% 
/run/media/myself/MyVolume

root@moksha:~# btrfs fi sh
Label: 'MyVolume'  uuid: xxxxxxxxx-yyyy-zzzz-tttt-wwwwwww
         Total devices 1 FS bytes used 1.19TiB
         devid    1 size 1.82TiB used 1.20TiB path 
/dev/mapper/luks-xxxxxxxxx-yyyy-zzzz-tttt-wwwwwww

root@moksha:~# btrfs fi df /run/media/myself/MyVolume
Data, single: total=1.18TiB, used=1.18TiB
System, DUP: total=8.00MiB, used=160.00KiB
Metadata, DUP: total=7.00GiB, used=6.88GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

root@moksha:~# umount /run/media/myself/MyVolume

root@moksha:~# btrfs check /dev/mapper/luks-xxxxxxxxx-yyyy-zzzz-tttt-wwwwwww
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-xxxxxxxxx-yyyy-zzzz-tttt-wwwwwww
UUID: xxxxxxxxx-yyyy-zzzz-tttt-wwwwwww
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots

(Still running since a while, no errors...)

TIA for any help.

Kind regards.

Re: Kernel 5.4 - BTRFS FS shows full with about 600 GB Free ?

[Qu Wenruo]

On 2019/12/21 下午6:59, Swâmi Petaramesh wrote:
> Hi list,
>
> After writing files and snapshots WITHOUT errors on an external BTRFS FS
> with 500+ GB of free space, using kernel 5.4.5-arch1-1, I dismount the
> FS then remount it normally, and then it says the FS has 0 space free
> left !
>
> Checking the disk on another machine with
>
> [moksha ~]# uname -r
> 5.4.2-1-MANJARO
>
> And... How can this be ?
>
> root@moksha:~# df -h /run/media/myself/MyVolume
> Filesystem           Size  Used Avail Use% Mounted on
> /dev/mapper/luks-xxxxxxxxx-yyyy-zzzz-tttt-wwwwwww   1,9T 1,2T     0 100%
> /run/media/myself/MyVolume

A known regression introduced in v5.4.

The new metadata over-commit behavior conflicts with an existing check
in btrfs_statfs().
It is completely a runtime false behavior, and had *no* other bad effect.

If you feel like to address it with off-tree patch, there is a quick
patch to address it:
https://patchwork.kernel.org/patch/11293419/

>
> root@moksha:~# btrfs fi sh
> Label: 'MyVolume'  uuid: xxxxxxxxx-yyyy-zzzz-tttt-wwwwwww
>         Total devices 1 FS bytes used 1.19TiB
>         devid    1 size 1.82TiB used 1.20TiB path
> /dev/mapper/luks-xxxxxxxxx-yyyy-zzzz-tttt-wwwwwww
>
> root@moksha:~# btrfs fi df /run/media/myself/MyVolume
> Data, single: total=1.18TiB, used=1.18TiB
> System, DUP: total=8.00MiB, used=160.00KiB
> Metadata, DUP: total=7.00GiB, used=6.88GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> root@moksha:~# umount /run/media/myself/MyVolume
>
> root@moksha:~# btrfs check
> /dev/mapper/luks-xxxxxxxxx-yyyy-zzzz-tttt-wwwwwww
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/luks-xxxxxxxxx-yyyy-zzzz-tttt-wwwwwww
> UUID: xxxxxxxxx-yyyy-zzzz-tttt-wwwwwww
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space cache
> [4/7] checking fs roots
>
> (Still running since a while, no errors...)

Running a fsck is always a good behavior, although in this case, it
shouldn't cause any corruption.

Thanks,
Qu
>
> TIA for any help.
>
> Kind regards.
>

Re: Kernel 5.4 - BTRFS FS shows full with about 600 GB Free ?

[Swâmi Petaramesh]

Le 21/12/2019 à 13:00, Qu Wenruo a écrit :
> A known regression introduced in v5.4.
>
> The new metadata over-commit behavior conflicts with an existing check
> in btrfs_statfs().
> It is completely a runtime false behavior, and had*no*  other bad effect.

Many thanks to you for the quick explanation.

I understand « no other bad effect » as « The FS will be OK again when 
this is fixed in the kernel, I can leave the disk in my drawer in the 
mean time » ?

Well I find that makes several recent problematic regressions in 
BTRFS... 5.2... 5.4...

I'm starting worrying about this development model.

Something as serious as an « Enterprise grade filesystem » that has been 
under heavy development for about ten years is not supposed to be a toy 
that breaks every now and then and couple patches is it ?

Or it would mean that it would ever stay a toy, not an enterprise-grade 
FS...

Kind regards.

Re: Kernel 5.4 - BTRFS FS shows full with about 600 GB Free ?

[Qu Wenruo]

On 2019/12/21 下午8:07, Swâmi Petaramesh wrote:
> Le 21/12/2019 à 13:00, Qu Wenruo a écrit :
>> A known regression introduced in v5.4.
>>
>> The new metadata over-commit behavior conflicts with an existing check
>> in btrfs_statfs().
>> It is completely a runtime false behavior, and had*no*  other bad effect.
>
> Many thanks to you for the quick explanation.
>
> I understand « no other bad effect » as « The FS will be OK again when
> this is fixed in the kernel, I can leave the disk in my drawer in the
> mean time » ?

You can even use it without any problem with v5.4 kernel, except the
`df` will sometimes bother you.

>
> Well I find that makes several recent problematic regressions in
> BTRFS... 5.2... 5.4...

That's why we have LTS, and commercial company baked kernels.

Thanks,
Qu

>
> I'm starting worrying about this development model.
>
> Something as serious as an « Enterprise grade filesystem » that has been
> under heavy development for about ten years is not supposed to be a toy
> that breaks every now and then and couple patches is it ?
>
> Or it would mean that it would ever stay a toy, not an enterprise-grade
> FS...
>
> Kind regards.
>

Re: Kernel 5.4 - BTRFS FS shows full with about 600 GB Free ?

[Swâmi Petaramesh]

Le 21/12/2019 à 13:00, Qu Wenruo a écrit :
> Running a fsck is always a good behavior, although in this case, it
> shouldn't cause any corruption.

Actually looks good...

root@moksha:~# btrfs check /dev/mapper/luks-xxxxxxxxxxxxx
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-xxxxxxxxxxxxxx
UUID: yyyyyyyyyy
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 1306090934272 bytes used, no error found
total csum bytes: 1267957652
total tree bytes: 7383269376
total fs tree bytes: 5703942144
total extent tree bytes: 282771456
btree space waste bytes: 1153433142
file data blocks allocated: 1329687621632
  referenced 1430556405760

Kind regards.

Re: Kernel 5.4 - BTRFS FS shows full with about 600 GB Free ?

[Qu Wenruo]

On 2019/12/21 下午8:26, Swâmi Petaramesh wrote:
> Le 21/12/2019 à 13:16, Qu Wenruo a écrit :
>>> I understand « no other bad effect » as « The FS will be OK again when
>>> this is fixed in the kernel, I can leave the disk in my drawer in the
>>> mean time » ?
>> You can even use it without any problem with v5.4 kernel, except the
>> `df` will sometimes bother you.
>>
> Actually I discovered the issue not typing « df » but trying to copy
> files using my GUI's file manager (Cinnamon's “Nemo”).

There are other reports of tools, which utilize 'statfs()' call to
monitor available space, also get affected.

So to be more accurate, any tools utilizing 'statfs()' call would
reports false ENOSPC error.

Since you're using arch, and if you're not utilizing bleeding edge
hardware, you could try lts kernel, or at least downgrade to v5.3 and
wait for the fix to be merged and backported.

Thanks,
Qu

>
> Got bounced with « Filesystem full », so no, I cannot use the FS
> “without any problems”...
>
> But using rsync in a terminal seems to actually work...
>
> Kind regards.
>

Re: Kernel 5.4 - BTRFS FS shows full with about 600 GB Free ?

[Swâmi Petaramesh]

Hi again,

Le 21/12/2019 à 13:00, Qu Wenruo a écrit :
> A known regression introduced in v5.4.
> 
> The new metadata over-commit behavior conflicts with an existing check
> in btrfs_statfs().
> It is completely a runtime false behavior, and had*no*  other bad effect.

I'm painfully unable to understand how the same said « metadata 
over-commit » would allow, on my system, “df” to work properly on some 
filesystems and fail miserably on others.

AND show “100% full” filesystems with still more than 600 GB free... 
That would make quite much of an overcommit isn't it ?

So I wonder if there isn't something else or if the “overcommit” 
calculation is not really completely broke in the first place...

ॐ

-- 
Swâmi Petaramesh <swami@petaramesh.org> PGP 9076E32E

Re: Kernel 5.4 - BTRFS FS shows full with about 600 GB Free ?

[Qu Wenruo]

On 2019/12/22 下午3:55, Swâmi Petaramesh wrote:
> Hi again,
>
> Le 21/12/2019 à 13:00, Qu Wenruo a écrit :
>> A known regression introduced in v5.4.
>>
>> The new metadata over-commit behavior conflicts with an existing check
>> in btrfs_statfs().
>> It is completely a runtime false behavior, and had*no*  other bad effect.
>
> I'm painfully unable to understand how the same said « metadata
> over-commit » would allow, on my system, “df” to work properly on some
> filesystems and fail miserably on others.

Here comes the full explanation if you like tech details:

For v5.4, btrfs allows reserved metadata space to go beyond current
metadata free space, as long as the fs can allocate enough metadata.

E.g you have a 1T fs, with 2G metadata (SINGLE) chunks allocated.
And among all these 2G metadata chunks, you have 1G unused space.

In v5.4, btrfs is allowed to reserve extra metadata space, near (1T -
1G). Since btrfs knows we can allocate new metadata chunks for (1T -
2G). And we also have 1G free metadata space.

However, in btrfs_statfs(), there is a check to ensure we have at least
4M free space in allocated metadata chunks.
If that condition can't be met, then btrfs_statfs() return f_bavail = 0,
indicating no available space.

That check is from 2015, which is kinda OK at that time, but definitely
not works with the new over-commit behavior.

So if you have enough uncommitted metadata, that check will be triggered
and suddenly returns 0 available space, even 1 sec early you still get
tons of available space.


>
> AND show “100% full” filesystems with still more than 600 GB free...
> That would make quite much of an overcommit isn't it ?
>
> So I wonder if there isn't something else or if the “overcommit”
> calculation is not really completely broke in the first place...

It's the old check from 2015 completely broken with 2019 new behavior.

If you really want to dig into the code, please check btrfs_statfs() of
fs/btrfs/super.c.

        if (!mixed && total_free_meta - thresh < block_rsv->size)
                buf->f_bavail = 0;

Which is explained in my patch mentioned in previous reply.

Thanks,
Qu

>
> ॐ
>

Re: Kernel 5.4 - BTRFS FS shows full with about 600 GB Free ?

[Swâmi Petaramesh]

Hi again,

Le 22/12/2019 à 09:12, Qu Wenruo a écrit :
> So if you have enough uncommitted metadata, that check will be triggered
> and suddenly returns 0 available space, even 1 sec early you still get
> tons of available space.

With « uncommitted metadata » do you mean that this situation is only 
temporary and should end once all transactions are commited to disk ?

Because in the one disk on which I observe this (and which passes a full 
btrfs check with bells and whistles) it still shows 0% free after 2 days 
and several unmounts / remounts...

Furthermore I've conected it to machines using 5.3 and even 4.15 
kernels, and they *ALL* state that the free disk space is zero - even 
though I can understand if « That check is from 2015 ».

That still seems to boild down to : Once I got this error it seems I 
cannot step out of it using any reasonably recent kernel.

So I may have either to patch my kernel or wait for the patch to reach 
#Arch kernels... and hope it actually fixes it.

(I have to admit that I haven't yet fully understood the technical 
aspects of this overcommit story...)

I'm also worried that on my machines that does NOT show this problem on 
their own main filesystems, this could happen anytime soon ?

Thanks anyway very much for your help.

Kind regards.

ॐ

-- 
Swâmi Petaramesh <swami@petaramesh.org> PGP 9076E32E

Re: Kernel 5.4 - BTRFS FS shows full with about 600 GB Free ?

[Swâmi Petaramesh]

Le 22/12/2019 à 10:00, Swâmi Petaramesh a écrit :
> Because in the one disk on which I observe this (and which passes a full 
> btrfs check with bells and whistles) it still shows 0% free after 2 days 
> and several unmounts / remounts...
> 
> Furthermore I've conected it to machines using 5.3 and even 4.15 
> kernels, and they *ALL* state that the free disk space is zero - even 
> though I can understand if « That check is from 2015 ».

For the record, a « btrfs balance start -m » apparently fixed it.

ॐ

-- 
Swâmi Petaramesh <swami@petaramesh.org> PGP 9076E32E

Re: Kernel 5.4 - BTRFS FS shows full with about 600 GB Free ?

[Qu Wenruo]

On 2019/12/22 下午5:00, Swâmi Petaramesh wrote:
> Hi again,
>
> Le 22/12/2019 à 09:12, Qu Wenruo a écrit :
>> So if you have enough uncommitted metadata, that check will be triggered
>> and suddenly returns 0 available space, even 1 sec early you still get
>> tons of available space.
>
> With « uncommitted metadata » do you mean that this situation is only
> temporary and should end once all transactions are commited to disk ?

Yes. The temporary part also matches with one kind reporter's description.
So for that v5.4 temporary 0 available, it should be the case.

>
> Because in the one disk on which I observe this (and which passes a full
> btrfs check with bells and whistles) it still shows 0% free after 2 days
> and several unmounts / remounts...
>
> Furthermore I've conected it to machines using 5.3 and even 4.15
> kernels, and they *ALL* state that the free disk space is zero - even
> though I can understand if « That check is from 2015 ».

Then for older kernels, I'm afraid you're seeing a different problem.
If you feel like to try to start hacking the kernel, I could provide
some snippet to add debug output and pin down the problem.



But there is one valid behavior which may cause such 0 available space
situation.
Are you using RAID1 or RAID10 with hugely unbalanced disk size?
If so, there could be a case that btrfs can't find two devices with
enough un-allocated space to fulfill a chunk allocation.

E.g. 1T + 1T + 10T disks RAID1. You can only utilize 2T space (1T from
each smaller devices, and 2T from that larger device).
The remaining 8T from that larger device can't be utilized for RAID1.

In that case, you will still have unallocated space, but any profile
requiring two disks is unable to use it.

>
> That still seems to boild down to : Once I got this error it seems I
> cannot step out of it using any reasonably recent kernel.
>
> So I may have either to patch my kernel or wait for the patch to reach
> #Arch kernels... and hope it actually fixes it.
>
> (I have to admit that I haven't yet fully understood the technical
> aspects of this overcommit story...)
>
> I'm also worried that on my machines that does NOT show this problem on
> their own main filesystems, this could happen anytime soon ?

We need extra info to further determine the cause of the persistent 0
available space problem.
`btrfs fi usage` output would help a lot.

But still, that 2015 check can still cause 0 available space, if your
metadata available space is pretty low.

Thanks,
Qu

>
> Thanks anyway very much for your help.
>
> Kind regards.
>
> ॐ
>

Re: Kernel 5.4 - BTRFS FS shows full with about 600 GB Free ?

[Swâmi Petaramesh]

Le 22/12/2019 à 10:37, Qu Wenruo a écrit :
>>
>> With « uncommitted metadata » do you mean that this situation is only
>> temporary and should end once all transactions are commited to disk ?
> 
> Yes. The temporary part also matches with one kind reporter's description.
> So for that v5.4 temporary 0 available, it should be the case.

Unfortunately in my case it was not « temporary », the filesystem was 
idle, and the « zero free space » survived unmounts and remounts and 
showed the same using kernels 5.3 and 4.15 on other machines...

...But a « btrfs balance -m » (under kernel 5.4) fixed it in the end.

> But there is one valid behavior which may cause such 0 available space
> situation.
> Are you using RAID1 or RAID10 with hugely unbalanced disk size?

Absolutely not. Relatively fresh BTRFS FS on a single 2TB device on 
which at least 33% of the space had never been allocated.

> We need extra info to further determine the cause of the persistent 0
> available space problem.
> `btrfs fi usage` output would help a lot.

Here's all what I get AFTER the "btrfs balance -m" fixed things:

[moksha /]# uname -r
5.4.2-1-MANJARO

[moksha /]# btrfs fi sh
Label: 'MyFS'  uuid: xxxxxxxxx-yyyy-zzzz-tttt-wwwwwwwwwwww
	Total devices 1 FS bytes used 1.19TiB
	devid    1 size 1.82TiB used 1.20TiB path 
/dev/mapper/luks-aaaaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee

[moksha /]# btrfs fi df /run/media/myself/MyFS
Data, single: total=1.18TiB, used=1.18TiB
System, DUP: total=32.00MiB, used=160.00KiB
Metadata, DUP: total=8.00GiB, used=6.93GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

[moksha /]# btrfs fi us /run/media/myself/MyFS
Overall:
     Device size:		   1.82TiB
     Device allocated:		   1.20TiB
     Device unallocated:		 633.90GiB
     Device missing:		     0.00B
     Used:			   1.20TiB
     Free (estimated):		 634.26GiB	(min: 317.31GiB)
     Data ratio:			      1.00
     Metadata ratio:		      2.00
     Global reserve:		 512.00MiB	(used: 0.00B)

Data,single: Size:1.18TiB, Used:1.18TiB (99.97%)
    /dev/mapper/luks-aaaaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee	   1.18TiB

Metadata,DUP: Size:8.00GiB, Used:6.93GiB (86.63%)
    /dev/mapper/luks-aaaaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee	  16.00GiB

System,DUP: Size:32.00MiB, Used:160.00KiB (0.49%)
    /dev/mapper/luks-aaaaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee	  64.00MiB

Unallocated:
    /dev/mapper/luks-aaaaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee	 633.90GiB

[moksha /]# df -h /run/media/myself/MyFS
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/luks-aaaaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee   1,9T    1,2T 
635G  66% /run/media/myself/MyFS


ॐ
-- 
Swâmi Petaramesh <swami@petaramesh.org> PGP 9076E32E