btrfs full, but not full, can't rebalance

btrfs full, but not full, can't rebalance

[Rich Rauenzahn]

Running on CentOS7 ... / got full, I removed the files, but it still
thinks it is full.  I've tried following the FAQ, even adding a
loopback device during the rebalance.

# btrfs fi show /
Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
        Total devices 2 FS bytes used 24.27GiB
        devid    1 size 111.11GiB used 111.05GiB path /dev/sdf3
        devid    2 size 111.11GiB used 111.05GiB path /dev/sdg3

# btrfs fi df /
Data, RAID1: total=107.02GiB, used=22.12GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=4.05GiB, used=2.15GiB
GlobalReserve, single: total=512.00MiB, used=0.00

# btrfs balance start -v -dusage=1 /
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=1
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail

# btrfs balance start -m /
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail

What can I do?

Re: btrfs full, but not full, can't rebalance

[Donald Pearson]

Have you seen this article?

I think the interesting part for you is the "balance cannot run
because the filesystem is full" heading.

http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html

On Fri, Jul 3, 2015 at 12:32 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote:
> Running on CentOS7 ... / got full, I removed the files, but it still
> thinks it is full.  I've tried following the FAQ, even adding a
> loopback device during the rebalance.
>
> # btrfs fi show /
> Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
>         Total devices 2 FS bytes used 24.27GiB
>         devid    1 size 111.11GiB used 111.05GiB path /dev/sdf3
>         devid    2 size 111.11GiB used 111.05GiB path /dev/sdg3
>
> # btrfs fi df /
> Data, RAID1: total=107.02GiB, used=22.12GiB
> System, RAID1: total=32.00MiB, used=16.00KiB
> Metadata, RAID1: total=4.05GiB, used=2.15GiB
> GlobalReserve, single: total=512.00MiB, used=0.00
>
> # btrfs balance start -v -dusage=1 /
> Dumping filters: flags 0x1, state 0x0, force is off
>   DATA (flags 0x2): balancing, usage=1
> ERROR: error during balancing '/' - No space left on device
> There may be more info in syslog - try dmesg | tail
>
> # btrfs balance start -m /
> ERROR: error during balancing '/' - No space left on device
> There may be more info in syslog - try dmesg | tail
>
> What can I do?
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs full, but not full, can't rebalance

[Rich Rauenzahn]

Yes, I tried that -- and adding the loopback device.

# btrfs device add /dev/loop1 /
Performing full device TRIM (5.00GiB) ...

# btrfs fi show /
Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
        Total devices 3 FS bytes used 17.13GiB
        devid    1 size 111.11GiB used 111.10GiB path /dev/sdf3
        devid    2 size 111.11GiB used 111.10GiB path /dev/sdg3
        devid    3 size 5.00GiB used 0.00 path /dev/loop1

Btrfs v3.16.2

# btrfs balance start -m /
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail

# btrfs balance start -v -dusage=1 /
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=1
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail



On Thu, Jul 2, 2015 at 10:45 PM, Donald Pearson
<donaldwhpearson@gmail.com> wrote:
> Have you seen this article?
>
> I think the interesting part for you is the "balance cannot run
> because the filesystem is full" heading.
>
> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
>
> On Fri, Jul 3, 2015 at 12:32 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote:
>> Running on CentOS7 ... / got full, I removed the files, but it still
>> thinks it is full.  I've tried following the FAQ, even adding a
>> loopback device during the rebalance.
>>
>> # btrfs fi show /
>> Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
>>         Total devices 2 FS bytes used 24.27GiB
>>         devid    1 size 111.11GiB used 111.05GiB path /dev/sdf3
>>         devid    2 size 111.11GiB used 111.05GiB path /dev/sdg3
>>
>> # btrfs fi df /
>> Data, RAID1: total=107.02GiB, used=22.12GiB
>> System, RAID1: total=32.00MiB, used=16.00KiB
>> Metadata, RAID1: total=4.05GiB, used=2.15GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00
>>
>> # btrfs balance start -v -dusage=1 /
>> Dumping filters: flags 0x1, state 0x0, force is off
>>   DATA (flags 0x2): balancing, usage=1
>> ERROR: error during balancing '/' - No space left on device
>> There may be more info in syslog - try dmesg | tail
>>
>> # btrfs balance start -m /
>> ERROR: error during balancing '/' - No space left on device
>> There may be more info in syslog - try dmesg | tail
>>
>> What can I do?
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs full, but not full, can't rebalance

[Donald Pearson]

Because this is raid1 I believe you need another for that to work.

On Fri, Jul 3, 2015 at 12:57 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote:
> Yes, I tried that -- and adding the loopback device.
>
> # btrfs device add /dev/loop1 /
> Performing full device TRIM (5.00GiB) ...
>
> # btrfs fi show /
> Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
>         Total devices 3 FS bytes used 17.13GiB
>         devid    1 size 111.11GiB used 111.10GiB path /dev/sdf3
>         devid    2 size 111.11GiB used 111.10GiB path /dev/sdg3
>         devid    3 size 5.00GiB used 0.00 path /dev/loop1
>
> Btrfs v3.16.2
>
> # btrfs balance start -m /
> ERROR: error during balancing '/' - No space left on device
> There may be more info in syslog - try dmesg | tail
>
> # btrfs balance start -v -dusage=1 /
> Dumping filters: flags 0x1, state 0x0, force is off
>   DATA (flags 0x2): balancing, usage=1
> ERROR: error during balancing '/' - No space left on device
> There may be more info in syslog - try dmesg | tail
>
>
>
> On Thu, Jul 2, 2015 at 10:45 PM, Donald Pearson
> <donaldwhpearson@gmail.com> wrote:
>> Have you seen this article?
>>
>> I think the interesting part for you is the "balance cannot run
>> because the filesystem is full" heading.
>>
>> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
>>
>> On Fri, Jul 3, 2015 at 12:32 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote:
>>> Running on CentOS7 ... / got full, I removed the files, but it still
>>> thinks it is full.  I've tried following the FAQ, even adding a
>>> loopback device during the rebalance.
>>>
>>> # btrfs fi show /
>>> Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
>>>         Total devices 2 FS bytes used 24.27GiB
>>>         devid    1 size 111.11GiB used 111.05GiB path /dev/sdf3
>>>         devid    2 size 111.11GiB used 111.05GiB path /dev/sdg3
>>>
>>> # btrfs fi df /
>>> Data, RAID1: total=107.02GiB, used=22.12GiB
>>> System, RAID1: total=32.00MiB, used=16.00KiB
>>> Metadata, RAID1: total=4.05GiB, used=2.15GiB
>>> GlobalReserve, single: total=512.00MiB, used=0.00
>>>
>>> # btrfs balance start -v -dusage=1 /
>>> Dumping filters: flags 0x1, state 0x0, force is off
>>>   DATA (flags 0x2): balancing, usage=1
>>> ERROR: error during balancing '/' - No space left on device
>>> There may be more info in syslog - try dmesg | tail
>>>
>>> # btrfs balance start -m /
>>> ERROR: error during balancing '/' - No space left on device
>>> There may be more info in syslog - try dmesg | tail
>>>
>>> What can I do?
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs full, but not full, can't rebalance

[Rich Rauenzahn]

Yes -- I just figured that out as well!

Now why did it suddenly fill up?   (I still get the failure rebalancing ...)

# btrfs fi show /
Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
        Total devices 4 FS bytes used 17.12GiB
        devid    1 size 111.11GiB used 111.05GiB path /dev/sdf3
        devid    2 size 111.11GiB used 111.05GiB path /dev/sdg3
        devid    3 size 5.00GiB used 5.00GiB path /dev/loop1
        devid    4 size 5.00GiB used 5.00GiB path /dev/loop2

Btrfs v3.16.2


On Thu, Jul 2, 2015 at 11:01 PM, Donald Pearson
<donaldwhpearson@gmail.com> wrote:
> Because this is raid1 I believe you need another for that to work.
>
> On Fri, Jul 3, 2015 at 12:57 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote:
>> Yes, I tried that -- and adding the loopback device.
>>
>> # btrfs device add /dev/loop1 /
>> Performing full device TRIM (5.00GiB) ...
>>
>> # btrfs fi show /
>> Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
>>         Total devices 3 FS bytes used 17.13GiB
>>         devid    1 size 111.11GiB used 111.10GiB path /dev/sdf3
>>         devid    2 size 111.11GiB used 111.10GiB path /dev/sdg3
>>         devid    3 size 5.00GiB used 0.00 path /dev/loop1
>>
>> Btrfs v3.16.2
>>
>> # btrfs balance start -m /
>> ERROR: error during balancing '/' - No space left on device
>> There may be more info in syslog - try dmesg | tail
>>
>> # btrfs balance start -v -dusage=1 /
>> Dumping filters: flags 0x1, state 0x0, force is off
>>   DATA (flags 0x2): balancing, usage=1
>> ERROR: error during balancing '/' - No space left on device
>> There may be more info in syslog - try dmesg | tail
>>
>>
>>
>> On Thu, Jul 2, 2015 at 10:45 PM, Donald Pearson
>> <donaldwhpearson@gmail.com> wrote:
>>> Have you seen this article?
>>>
>>> I think the interesting part for you is the "balance cannot run
>>> because the filesystem is full" heading.
>>>
>>> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
>>>
>>> On Fri, Jul 3, 2015 at 12:32 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote:
>>>> Running on CentOS7 ... / got full, I removed the files, but it still
>>>> thinks it is full.  I've tried following the FAQ, even adding a
>>>> loopback device during the rebalance.
>>>>
>>>> # btrfs fi show /
>>>> Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
>>>>         Total devices 2 FS bytes used 24.27GiB
>>>>         devid    1 size 111.11GiB used 111.05GiB path /dev/sdf3
>>>>         devid    2 size 111.11GiB used 111.05GiB path /dev/sdg3
>>>>
>>>> # btrfs fi df /
>>>> Data, RAID1: total=107.02GiB, used=22.12GiB
>>>> System, RAID1: total=32.00MiB, used=16.00KiB
>>>> Metadata, RAID1: total=4.05GiB, used=2.15GiB
>>>> GlobalReserve, single: total=512.00MiB, used=0.00
>>>>
>>>> # btrfs balance start -v -dusage=1 /
>>>> Dumping filters: flags 0x1, state 0x0, force is off
>>>>   DATA (flags 0x2): balancing, usage=1
>>>> ERROR: error during balancing '/' - No space left on device
>>>> There may be more info in syslog - try dmesg | tail
>>>>
>>>> # btrfs balance start -m /
>>>> ERROR: error during balancing '/' - No space left on device
>>>> There may be more info in syslog - try dmesg | tail
>>>>
>>>> What can I do?
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs full, but not full, can't rebalance

[Donald Pearson]

what does the fi df , or btrfs fi usage show now

On Fri, Jul 3, 2015 at 1:03 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote:
> Yes -- I just figured that out as well!
>
> Now why did it suddenly fill up?   (I still get the failure rebalancing ...)
>
> # btrfs fi show /
> Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
>         Total devices 4 FS bytes used 17.12GiB
>         devid    1 size 111.11GiB used 111.05GiB path /dev/sdf3
>         devid    2 size 111.11GiB used 111.05GiB path /dev/sdg3
>         devid    3 size 5.00GiB used 5.00GiB path /dev/loop1
>         devid    4 size 5.00GiB used 5.00GiB path /dev/loop2
>
> Btrfs v3.16.2
>
>
> On Thu, Jul 2, 2015 at 11:01 PM, Donald Pearson
> <donaldwhpearson@gmail.com> wrote:
>> Because this is raid1 I believe you need another for that to work.
>>
>> On Fri, Jul 3, 2015 at 12:57 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote:
>>> Yes, I tried that -- and adding the loopback device.
>>>
>>> # btrfs device add /dev/loop1 /
>>> Performing full device TRIM (5.00GiB) ...
>>>
>>> # btrfs fi show /
>>> Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
>>>         Total devices 3 FS bytes used 17.13GiB
>>>         devid    1 size 111.11GiB used 111.10GiB path /dev/sdf3
>>>         devid    2 size 111.11GiB used 111.10GiB path /dev/sdg3
>>>         devid    3 size 5.00GiB used 0.00 path /dev/loop1
>>>
>>> Btrfs v3.16.2
>>>
>>> # btrfs balance start -m /
>>> ERROR: error during balancing '/' - No space left on device
>>> There may be more info in syslog - try dmesg | tail
>>>
>>> # btrfs balance start -v -dusage=1 /
>>> Dumping filters: flags 0x1, state 0x0, force is off
>>>   DATA (flags 0x2): balancing, usage=1
>>> ERROR: error during balancing '/' - No space left on device
>>> There may be more info in syslog - try dmesg | tail
>>>
>>>
>>>
>>> On Thu, Jul 2, 2015 at 10:45 PM, Donald Pearson
>>> <donaldwhpearson@gmail.com> wrote:
>>>> Have you seen this article?
>>>>
>>>> I think the interesting part for you is the "balance cannot run
>>>> because the filesystem is full" heading.
>>>>
>>>> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
>>>>
>>>> On Fri, Jul 3, 2015 at 12:32 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote:
>>>>> Running on CentOS7 ... / got full, I removed the files, but it still
>>>>> thinks it is full.  I've tried following the FAQ, even adding a
>>>>> loopback device during the rebalance.
>>>>>
>>>>> # btrfs fi show /
>>>>> Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
>>>>>         Total devices 2 FS bytes used 24.27GiB
>>>>>         devid    1 size 111.11GiB used 111.05GiB path /dev/sdf3
>>>>>         devid    2 size 111.11GiB used 111.05GiB path /dev/sdg3
>>>>>
>>>>> # btrfs fi df /
>>>>> Data, RAID1: total=107.02GiB, used=22.12GiB
>>>>> System, RAID1: total=32.00MiB, used=16.00KiB
>>>>> Metadata, RAID1: total=4.05GiB, used=2.15GiB
>>>>> GlobalReserve, single: total=512.00MiB, used=0.00
>>>>>
>>>>> # btrfs balance start -v -dusage=1 /
>>>>> Dumping filters: flags 0x1, state 0x0, force is off
>>>>>   DATA (flags 0x2): balancing, usage=1
>>>>> ERROR: error during balancing '/' - No space left on device
>>>>> There may be more info in syslog - try dmesg | tail
>>>>>
>>>>> # btrfs balance start -m /
>>>>> ERROR: error during balancing '/' - No space left on device
>>>>> There may be more info in syslog - try dmesg | tail
>>>>>
>>>>> What can I do?
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs full, but not full, can't rebalance

[Rich Rauenzahn]

Donald and I went offline so I will summarize where I am now:

(1) Since I'm RAID1, I had to add two loopbacks.   That makes sense!
(2) The two 5GB loopbacks filled up almost instantly and did me no good.
(3) I created two more 100 GB loopbacks (hint: use /usr/bin/truncate
not /usr/bin/dd to create the empty files!)

I then was able to make some progress with balancing with -dusage=1.
I went to -dusage=2 and it failed with enospc again.  A little while
later -dusage=2 worked.  I believe I made it up to -dusage=5 and then
higher values failed.

At this point I saw that my 'btrfs fi show' showed that I had some
headroom, so I dropped all of the loopbacks from my volume.

But I found that I still could not "touch /tmp/foo".

I then just tried running btrfs balance start.  I noticed in dmesg
that although btrfs balance was exiting with an error about space, it
was making progress on some extents.  btrfs fi show also reflected
that.   So I began a loop...

while ! btrfs balance start /; do btrfs fi show /; btrfs fi df / ; done

It is still running, but after a few iterations I am now able to touch
a file /tmp/foo successfully and the fi show seems to still be coming
down to a proper value, although occasionally it bumps up the wrong
direction:

]# while ! btrfs balance start /; do btrfs fi show /; btrfs fi df / ; done
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail
Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
        Total devices 2 FS bytes used 17.25GiB
        devid    1 size 111.11GiB used 84.03GiB path /dev/sdf3
        devid    2 size 111.11GiB used 84.03GiB path /dev/sdg3

Btrfs v3.16.2
Data, RAID1: total=41.00GiB, used=16.54GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=43.00GiB, used=724.38MiB
GlobalReserve, single: total=256.00MiB, used=336.00KiB
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail
Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
        Total devices 2 FS bytes used 17.25GiB
        devid    1 size 111.11GiB used 85.03GiB path /dev/sdf3
        devid    2 size 111.11GiB used 85.03GiB path /dev/sdg3

Btrfs v3.16.2
Data, RAID1: total=40.00GiB, used=16.54GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=45.00GiB, used=724.30MiB
GlobalReserve, single: total=256.00MiB, used=256.00KiB
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail
Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
        Total devices 2 FS bytes used 17.25GiB
        devid    1 size 111.11GiB used 86.03GiB path /dev/sdf3
        devid    2 size 111.11GiB used 86.03GiB path /dev/sdg3

Btrfs v3.16.2
Data, RAID1: total=39.00GiB, used=16.54GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=47.00GiB, used=723.78MiB
GlobalReserve, single: total=256.00MiB, used=272.00KiB

Many thanks to Donald for getting me to this point!

On Thu, Jul 2, 2015 at 10:57 PM, Rich Rauenzahn <rrauenza@gmail.com> wrote:
> Yes, I tried that -- and adding the loopback device.
>
> # btrfs device add /dev/loop1 /
> Performing full device TRIM (5.00GiB) ...
>
> # btrfs fi show /
> Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
>         Total devices 3 FS bytes used 17.13GiB
>         devid    1 size 111.11GiB used 111.10GiB path /dev/sdf3
>         devid    2 size 111.11GiB used 111.10GiB path /dev/sdg3
>         devid    3 size 5.00GiB used 0.00 path /dev/loop1
>
> Btrfs v3.16.2
>
> # btrfs balance start -m /
> ERROR: error during balancing '/' - No space left on device
> There may be more info in syslog - try dmesg | tail
>
> # btrfs balance start -v -dusage=1 /
> Dumping filters: flags 0x1, state 0x0, force is off
>   DATA (flags 0x2): balancing, usage=1
> ERROR: error during balancing '/' - No space left on device
> There may be more info in syslog - try dmesg | tail
>
>
>
> On Thu, Jul 2, 2015 at 10:45 PM, Donald Pearson
> <donaldwhpearson@gmail.com> wrote:
>> Have you seen this article?
>>
>> I think the interesting part for you is the "balance cannot run
>> because the filesystem is full" heading.
>>
>> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
>>
>> On Fri, Jul 3, 2015 at 12:32 AM, Rich Rauenzahn <rrauenza@gmail.com> wrote:
>>> Running on CentOS7 ... / got full, I removed the files, but it still
>>> thinks it is full.  I've tried following the FAQ, even adding a
>>> loopback device during the rebalance.
>>>
>>> # btrfs fi show /
>>> Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
>>>         Total devices 2 FS bytes used 24.27GiB
>>>         devid    1 size 111.11GiB used 111.05GiB path /dev/sdf3
>>>         devid    2 size 111.11GiB used 111.05GiB path /dev/sdg3
>>>
>>> # btrfs fi df /
>>> Data, RAID1: total=107.02GiB, used=22.12GiB
>>> System, RAID1: total=32.00MiB, used=16.00KiB
>>> Metadata, RAID1: total=4.05GiB, used=2.15GiB
>>> GlobalReserve, single: total=512.00MiB, used=0.00
>>>
>>> # btrfs balance start -v -dusage=1 /
>>> Dumping filters: flags 0x1, state 0x0, force is off
>>>   DATA (flags 0x2): balancing, usage=1
>>> ERROR: error during balancing '/' - No space left on device
>>> There may be more info in syslog - try dmesg | tail
>>>
>>> # btrfs balance start -m /
>>> ERROR: error during balancing '/' - No space left on device
>>> There may be more info in syslog - try dmesg | tail
>>>
>>> What can I do?
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs full, but not full, can't rebalance

[Rich Rauenzahn]

I'm still seeing periodic issues here.  The filesystem will go "full"
occasionally even though there appears to be plenty of space.  Running
another rebalance seems to fix it ... but I don't see why the system
thinks it needs to be rebalanced.

# while ! btrfs balance start /; do btrfs fi show /; btrfs fi df / ; done
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail
Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
        Total devices 2 FS bytes used 17.06GiB
        devid    1 size 111.11GiB used 22.03GiB path /dev/sdf3
        devid    2 size 111.11GiB used 22.03GiB path /dev/sdg3

Btrfs v3.16.2
Data, RAID1: total=18.00GiB, used=16.36GiB
System, RAID1: total=32.00MiB, used=16.00KiB
Metadata, RAID1: total=4.00GiB, used=711.34MiB
GlobalReserve, single: total=240.00MiB, used=128.00KiB
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail
Label: 'centos7'  uuid: 35f0ce3f-0902-47a3-8ad8-86179d1f3e3a
        Total devices 2 FS bytes used 17.06GiB
        devid    1 size 111.11GiB used 25.06GiB path /dev/sdf3
        devid    2 size 111.11GiB used 25.06GiB path /dev/sdg3

Btrfs v3.16.2
Data, RAID1: total=17.00GiB, used=16.36GiB
System, RAID1: total=64.00MiB, used=16.00KiB
Metadata, RAID1: total=8.00GiB, used=711.67MiB
GlobalReserve, single: total=240.00MiB, used=144.00KiB
[...]

For now, this is going into a cron job ...

Rich

Re: btrfs full, but not full, can't rebalance

[Chris Murphy]

What kernel version are you using?

I suggest reading the whole thread but in particular this note about
CentOS 7 and Btrfs.
http://www.spinics.net/lists/linux-btrfs/msg40850.html

So kernel 3.16 isn't necessarily too old, but I know there's been a
lot of on-going balance related work that's not in that kernel and
thus won't be in the CentOS 7.1 kernel either. And then there's also
some possibility the older btrfs-progs contributes to the problem,
even though I think most of the balance code is actually in the
kernel.

While the CentOS 7.1 kernel is going to be more stable/predictable
overall (not with respect to Btrfs but everything the kernel does):
a.) Btrfs is a tech preview
b.) To unwind the problems you've having with this volume right now,
I'd really suggest a much newer kernel.

>         Total devices 2 FS bytes used 17.06GiB
>         devid    1 size 111.11GiB used 22.03GiB path /dev/sdf3
>         devid    2 size 111.11GiB used 22.03GiB path /dev/sdg3

There's tons of unallocated space for new data or metadata chunks.

> Data, RAID1: total=18.00GiB, used=16.36GiB
> System, RAID1: total=32.00MiB, used=16.00KiB
> Metadata, RAID1: total=4.00GiB, used=711.34MiB
> GlobalReserve, single: total=240.00MiB, used=128.00KiB
> ERROR: error during balancing '/' - No space left on device

And there's tons of space in already created data and metadata chunks.
So something is wrong, and I really wouldn't persist on trying to make
this work with an older kernel and progs. So I suggest putting the
volume on a system with a 4.1 kernel and 4.1 progs, and running a
btrfs check without --repair and report back with the full output.

Once you get things fixed up, it seems going back to the CentOS 7.1
kernel is fine. But periodically you might have to revert to a newer
kernel for some maintenance, at least until the fix for the problem
makes it into a future CentOS kernel.


Chris Murphy

Re: btrfs full, but not full, can't rebalance

[Rich Rauenzahn]

Just a final note -- I'm finally back in person with the CentOS 7 server and so
booted it to the latest kernel-ml from elrepo.  It is a 4.1 kernel.

But while still remote with the older 3.10 kernel, I also tried doing
a 'mount -oremount,clear_cache /'

I can't swear it helped, but things did seem to get better
afterwards... so it may be worth a try for anyone reading this thread
with a similar problem later.  (Unless someone can definitively say
that clear_cache doesn't work with -oremount.)

Rich