Btrfs full balance command fails due to ENOSPC (bug 121071)

Btrfs full balance command fails due to ENOSPC (bug 121071)

[Francesco Turco]

Note: I already filed bug 121071 but perhaps I should have written to
this mailing list first.

I get the ENOSPC error when running a btrfs full balance command for my
root partition, even if it seems I have a lot of free/unallocated space.

# btrfs filesystem show /
Label: none  uuid: 27150b83-7d90-4031-8e83-581315b9a254
	Total devices 1 FS bytes used 10.79GiB
	devid    1 size 25.00GiB used 13.31GiB path /dev/mapper/Desktop-root
# btrfs filesystem df /
Data, single: total=11.00GiB, used=10.40GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.12GiB, used=392.08MiB
GlobalReserve, single: total=144.00MiB, used=0.00B
# btrfs balance start --full-balance /
ERROR: error during balancing '/': No space left on device
There may be more info in syslog - try dmesg | tail
# dmesg | tail
[29807.441930] BTRFS info (device dm-2): found 13206 extents
[29807.879845] BTRFS info (device dm-2): relocating block group
47542435840 flags 1
[29827.116083] BTRFS info (device dm-2): found 12909 extents
[29830.500110] BTRFS info (device dm-2): found 12909 extents
[29830.976485] BTRFS info (device dm-2): relocating block group
46468694016 flags 1
[29848.924188] BTRFS info (device dm-2): found 5129 extents
[29851.533076] BTRFS info (device dm-2): found 5129 extents
[29851.994787] BTRFS info (device dm-2): relocating block group
46435139584 flags 34
[29852.399460] BTRFS info (device dm-2): found 1 extents
[29852.657983] BTRFS info (device dm-2): 1 enospc errors during balance

I have successfully balanced both the boot and home partitions before.
Only root gives me problems.

Is there anything I can try? Should I run the command from a live CD? Is
this a real bug or a mistake from an unexperienced btrfs user like me?

Thanks.

-- 
Website: http://www.fturco.net/
GPG key: 6712 2364 B2FE 30E1 4791 EB82 7BB1 1F53 29DE CD34

Re: Btrfs full balance command fails due to ENOSPC (bug 121071)

[Chris Murphy]

On Mon, Jun 27, 2016 at 11:28 AM, Francesco Turco <fturco@fastmail.fm> wrote:
> Note: I already filed bug 121071 but perhaps I should have written to
> this mailing list first.

https://bugzilla.kernel.org/show_bug.cgi?id=121071

It's a good bug report.


> Is there anything I can try? Should I run the command from a live CD? Is
> this a real bug or a mistake from an unexperienced btrfs user like me?

It's a bug somewhere, not a user mistake.

If you can grab btrfs-debugfs from
https://github.com/kdave/btrfs-progs/blob/master/btrfs-debugfs

And then attach the output to the bug report it might be useful for a
developer. But really your case is an odd duck, because there's fully
14GiB unallocated, so it should be able to create a new one without
problem.

$ sudo ./btrfs-debugfs -b /



-- 
Chris Murphy

Re: Btrfs full balance command fails due to ENOSPC (bug 121071)

[Francesco Turco]

On 2016-06-27 20:18, Chris Murphy wrote:
> If you can grab btrfs-debugfs from
> https://github.com/kdave/btrfs-progs/blob/master/btrfs-debugfs
> 
> And then attach the output to the bug report it might be useful for a
> developer. But really your case is an odd duck, because there's fully
> 14GiB unallocated, so it should be able to create a new one without
> problem.
> 
> $ sudo ./btrfs-debugfs -b /

Done! Thank you, I was not aware of the existence of btrfs-debug...

-- 
Website: http://www.fturco.net/
GPG key: 6712 2364 B2FE 30E1 4791 EB82 7BB1 1F53 29DE CD34

Re: Btrfs full balance command fails due to ENOSPC (bug 121071)

[Chris Murphy]

On Mon, Jun 27, 2016 at 12:32 PM, Francesco Turco <fturco@fastmail.fm> wrote:
> On 2016-06-27 20:18, Chris Murphy wrote:
>> If you can grab btrfs-debugfs from
>> https://github.com/kdave/btrfs-progs/blob/master/btrfs-debugfs
>>
>> And then attach the output to the bug report it might be useful for a
>> developer. But really your case is an odd duck, because there's fully
>> 14GiB unallocated, so it should be able to create a new one without
>> problem.
>>
>> $ sudo ./btrfs-debugfs -b /
>
> Done! Thank you, I was not aware of the existence of btrfs-debug...

I'm not certain what the "1 enospc errors during balance' refers to.
That message happens several times, the balance operation isn't
aborted, and doesn't come with any call traces (those appear later).
Further, the btrfs-debugfs output suggests the balance worked - each
bg is continguously located after the last and they're all new bg
offset values compared to what's found in the dmesg.

This might be that obscure -28 enospc bug that affects some file
systems and hasn't been tracked down yet. If I recall correctly it's a
misleading error, and the only work around to get rid of it is migrate
to a new Btrfs file system. I don't think the file system is at any
risk in the current state, but I'm not certain as it's already an edge
case. I'd just make sure you keep suitably current backups and keep
using it.



-- 
Chris Murphy

Re: Btrfs full balance command fails due to ENOSPC (bug 121071)

[Henk Slager]

On Mon, Jun 27, 2016 at 9:24 PM, Chris Murphy <lists@colorremedies.com> wrote:
> On Mon, Jun 27, 2016 at 12:32 PM, Francesco Turco <fturco@fastmail.fm> wrote:
>> On 2016-06-27 20:18, Chris Murphy wrote:
>>> If you can grab btrfs-debugfs from
>>> https://github.com/kdave/btrfs-progs/blob/master/btrfs-debugfs
>>>
>>> And then attach the output to the bug report it might be useful for a
>>> developer. But really your case is an odd duck, because there's fully
>>> 14GiB unallocated, so it should be able to create a new one without
>>> problem.
>>>
>>> $ sudo ./btrfs-debugfs -b /
>>
>> Done! Thank you, I was not aware of the existence of btrfs-debug...
>
> I'm not certain what the "1 enospc errors during balance' refers to.
> That message happens several times, the balance operation isn't
> aborted, and doesn't come with any call traces (those appear later).
> Further, the btrfs-debugfs output suggests the balance worked - each
> bg is continguously located after the last and they're all new bg
> offset values compared to what's found in the dmesg.

btrfs-debug does not show metadata ans system chunks; the balancing
problem might come from those.
This script does show all chunks:
https://github.com/knorrie/btrfs-heatmap/blob/master/show_usage.py

You might want to use vrange or drange balance filters so that you can
just target a certain chunk and maybe that gives a hint where the
problem might be. But anyhow, the behavior experienced is a bug.

> This might be that obscure -28 enospc bug that affects some file
> systems and hasn't been tracked down yet. If I recall correctly it's a
> misleading error, and the only work around to get rid of it is migrate
> to a new Btrfs file system. I don't think the file system is at any
> risk in the current state, but I'm not certain as it's already an edge
> case. I'd just make sure you keep suitably current backups and keep
> using it.
>
>
>
> --
> Chris Murphy
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Btrfs full balance command fails due to ENOSPC (bug 121071)

[Hans van Kranenburg]

Hi!

On 06/27/2016 11:26 PM, Henk Slager wrote:
>
> btrfs-debug does not show metadata ans system chunks; the balancing
> problem might come from those.
> This script does show all chunks:
> https://github.com/knorrie/btrfs-heatmap/blob/master/show_usage.py

Since the existence of python-btrfs, it has gathered a few useful 
example scripts:

git clone https://github.com/knorrie/python-btrfs
cd python-btrfs/examples/
(get root prompt)

./show_usage.py /mountpoint <- view sorted by 'virtual' address space
./show_dev_extents.py /mountpoint <- view sorted by physical layout

The show_usage in the btrfs-heatmap repo is almost gone. I'm currently 
replacing all the proof of concept playing around stuff in there with 
dedicated png-creation code that uses the python-btrfs lib.

So, it's better to refer to the examples in python-btrfs instead. Or, 
write some others to create overviews yourself, it gets easier every day. :D

Have fun,

-- 
Hans van Kranenburg - System / Network Engineer
T +31 (0)10 2760434 | hans.van.kranenburg@mendix.com | www.mendix.com

Re: Btrfs full balance command fails due to ENOSPC (bug 121071)

[Francesco Turco]

On 2016-06-27 23:26, Henk Slager wrote:
> btrfs-debug does not show metadata ans system chunks; the balancing
> problem might come from those.
> This script does show all chunks:
> https://github.com/knorrie/btrfs-heatmap/blob/master/show_usage.py
> 
> You might want to use vrange or drange balance filters so that you can
> just target a certain chunk and maybe that gives a hint where the
> problem might be. But anyhow, the behavior experienced is a bug.

Updated the bug with the output log from your script. I simply ran it as:

./show_usage.py /

I don't know how to use vrange/drange balance filters. Can you show me
how to do that, please?

-- 
Website: http://www.fturco.net/
GPG key: 6712 2364 B2FE 30E1 4791 EB82 7BB1 1F53 29DE CD34

Re: Btrfs full balance command fails due to ENOSPC (bug 121071)

[Francesco Turco]

On 2016-06-27 23:47, Hans van Kranenburg wrote:
> Since the existence of python-btrfs, it has gathered a few useful
> example scripts:
> 
> git clone https://github.com/knorrie/python-btrfs
> cd python-btrfs/examples/
> (get root prompt)
> 
> ./show_usage.py /mountpoint <- view sorted by 'virtual' address space
> ./show_dev_extents.py /mountpoint <- view sorted by physical layout
> 
> The show_usage in the btrfs-heatmap repo is almost gone. I'm currently
> replacing all the proof of concept playing around stuff in there with
> dedicated png-creation code that uses the python-btrfs lib.

I also issued the show_dev_extents.py command as you suggested. Hope it
helps.

-- 
Website: http://www.fturco.net/
GPG key: 6712 2364 B2FE 30E1 4791 EB82 7BB1 1F53 29DE CD34

Re: Btrfs full balance command fails due to ENOSPC (bug 121071)

[Henk Slager]

On Tue, Jun 28, 2016 at 3:46 PM, Francesco Turco <fturco@fastmail.fm> wrote:
> On 2016-06-27 23:26, Henk Slager wrote:
>> btrfs-debug does not show metadata ans system chunks; the balancing
>> problem might come from those.
>> This script does show all chunks:
>> https://github.com/knorrie/btrfs-heatmap/blob/master/show_usage.py
>>
>> You might want to use vrange or drange balance filters so that you can
>> just target a certain chunk and maybe that gives a hint where the
>> problem might be. But anyhow, the behavior experienced is a bug.
>
> Updated the bug with the output log from your script. I simply ran it as:
>
> ./show_usage.py /
>
> I don't know how to use vrange/drange balance filters. Can you show me
> how to do that, please?

The original dmesg log shows that balance gets into trouble at block group
46435139584. This is SYSTEM|DUP and in the later blockgroup list
generated with Hans' py script it is not there anymore under this same
vaddr, so btrfs (or new manual balance) has managed to relocate it,
despite the enospc.

One theory I once had is that at the beginning of the disk, there were
or are small chunks of 8MiB, whereas the rest of the disk has 1G or at
least bigger chunks once the fs gets used and filled up. Those initial
small chunks tend to be system and/or metadata. If then later after
heavy use of a full balance the small chunks get relocated, there is
then unallocated space, but virtually nothing fits there if the policy
is 'first allocate big chunks'. So here the allocater mechanism could
then output an enospc, assuming that it
doesn't try exhaustively in order to keep the code simple and fast.
But it is only theory, one would need to traceback etc in such a case.
I never had such a case, so I can't prove it.

Suppose you want to relocate the metadata blockgroup (you have only
one, it is the same location in the 2 lists from the bug report)

btrfs balance start -v -mvrange=29360128..29360129 /

This 1-byte range in in the virtual adress range 29360128 ..
29360128+1G-1,[ so it will relocate the metadata blockgroup. After
succesfull balance, you will see its vaddr increased and its device
addresses (paddr) also changed.

If you want to balance based on device address and for example
relocate just 1 of the dup of the metadata:
btrfs balance start -v -mdevid=1,drange=37748736..37748737 /

All this does not solve the bug, but hopefully gives us better
understanding in cases where balance also fails and also no file
creation is possible anymore.



>
> --
> Website: http://www.fturco.net/
> GPG key: 6712 2364 B2FE 30E1 4791 EB82 7BB1 1F53 29DE CD34