Re: Balance fails with unallocated diskspace

Re: Balance fails with unallocated diskspace

[Nisse Karlsson]

Hi,
I'm having some problems with balancing and I've filed a bug at
bugzilla (https://bugzilla.kernel.org/show_bug.cgi?id=120121), but it
could also be that I've missed something even after more than a couple
of hours reading forums and mailing lists.

If someone could point me in the right direction that would be very
much appreciated.

I tried to migrate from raid1 to raid10  (btrfs balance start /files
-mconvert=raid10 -dconvert=raid10)

This failed after about 1TB with enosp errors.

I have since then managed to migrate all metadata back to raid1 but my
data is still a mix.

I've tried "balance -dusage=50" with all values from 0 to 100, also
with -dprofiles=raid1 and -dprofiles=raid10. I've also tried balance
-dconvert=raid1 (with and without -dprofiles=raid10) but I always get
enosp errors.
I've also added a small usb-drive(/dev/sdh below) to get some space
but it still gives the same error.

Since I have about 1.5TB unallocated space it's my understanding that
this shouldn't be a problem, or am I completely wrong?

The output from btrfs filesystem usage /files

Overall:
    Device size:  18.20TiB
    Device allocated:  16.67TiB
    Device unallocated:   1.53TiB
    Device missing:     0.00B
    Used:  16.67TiB
    Free (estimated): 781.62GiB (min: 781.62GiB)
    Data ratio:      2.00
    Metadata ratio:      2.00
    Global reserve: 512.00MiB (used: 0.00B)

Data,RAID1: Size:7.21TiB, Used:7.21TiB
   /dev/sdb   3.23TiB
   /dev/sdc   3.24TiB
   /dev/sdd   1.89TiB
   /dev/sde   2.32TiB
   /dev/sdf   1.42TiB
   /dev/sdg   2.33TiB

Data,RAID10: Size:1.11TiB, Used:1.11TiB
   /dev/sdb 190.05GiB
   /dev/sdc 189.34GiB
   /dev/sdd 190.05GiB
   /dev/sde 190.05GiB
   /dev/sdf 188.92GiB
   /dev/sdg 189.41GiB
   /dev/sdh   2.48GiB

Metadata,RAID1: Size:10.19GiB, Used:9.35GiB
   /dev/sdb   4.47GiB
   /dev/sdc   3.58GiB
   /dev/sdd   5.19GiB
   /dev/sde   3.07GiB
   /dev/sdf   1.34GiB
   /dev/sdg   2.72GiB

System,RAID1: Size:32.00MiB, Used:1.41MiB
   /dev/sdc  32.00MiB
   /dev/sdd  32.00MiB

Unallocated:
   /dev/sdb 222.50GiB
   /dev/sdc 219.07GiB
   /dev/sdd   1.56TiB
   /dev/sde 221.40GiB
   /dev/sdf 219.75GiB
   /dev/sdg 219.39GiB
   /dev/sdh   4.97GiB

Regards,
Nisse
>>
>>

Re: Balance fails with unallocated diskspace

[Nisse Karlsson]

I've now tried mounting with nospace_cache as suggested in another thread,
no difference.

I also tried remounting with enospc_debug, but that didn't give me more info
than before
BTRFS info (device sdb): 523 enospc errors during balance

//Nisse

Re: Balance fails with unallocated diskspace

[Duncan]

Nisse Karlsson posted on Wed, 15 Jun 2016 11:58:36 +0200 as excerpted:

> Since I have about 1.5TB unallocated space it's my understanding that
> this shouldn't be a problem, or am I completely wrong?

Quick partial answer...

It's not just unallocated space in total that matters, but also the 
number of devices with unallocated space.  To write raid10 chunks 
requires four devices.  To write raid1 chunks requires two.

However, your usage output does indicate several hundred gigs of 
unallocated space on multiple devices, so convert to raid1 shouldn't be 
ENOSPCing...  That it is would indicate a bug.  There have been various 
bugs of this sort, but I'm just a list regular, not a dev, and I'm not 
sure of the current status, whether any such bugs are currently known or 
not.

You don't mention kernel version in your post and I don't have time to 
look up the bug ATM, but you might try the latest 4.1 and 4.4 LTS kernels 
as well as the latest 4.6 and 4.7-rc kernels to see if it makes a 
difference.  If it's a problem with 4.6 and 4.7-rc then it becomes 
interesting, and if it works with 4.4 or 4.1, even more so.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Balance fails with unallocated diskspace

[Nisse Karlsson]

Thanks,

Will update the bug-report with more information.
Not sure I trust the device info in dmesg though, when running device
delete /dev/sdh /files it still refers to to sdb in dmesg. I think it
just shows the device that was used for mounting the fs.

//Nisse

2016-06-15 17:33 GMT+02:00 Chris Murphy <lists@colorremedies.com>:
>
>
> On Wed, Jun 15, 2016 at 3:58 AM, Nisse Karlsson <nisse@xcorp.at> wrote:
>>
>>
>> Unallocated:
>>    /dev/sdb 222.50GiB
>
>
> The filed bug report needs a complete dmesg attached (at least from the time
> the fs is mounted and including enospc), what mount options are used, and
> what btrfs-progs version is used.
>
> But the fact sdb reports this much free space and yet the enospc kernel
> message you report later refers to sdb is unexpected, so it's good to have
> filed a bug.
>
> You can also include as an attachment the output from btrfs-debugfs which is
> a python script not normally installed as a part of btrfs-progs but you can
> get it from here:
> https://github.com/kdave/btrfs-progs
>
> btrfs-debugfs -b <mountpoint>
>
>
>
> --
> Chris Murphy

Re: Balance fails with unallocated diskspace

[Chris Murphy]

On Wed, Jun 15, 2016 at 10:00 AM, Nisse Karlsson <nisse@xcorp.at> wrote:
> Thanks,
>
> Will update the bug-report with more information.
> Not sure I trust the device info in dmesg though, when running device
> delete /dev/sdh /files it still refers to to sdb in dmesg. I think it
> just shows the device that was used for mounting the fs.

Hmm that's not very helpful if it's not reporting the device that's
having the enospc problem...

Anyway, the debugfs output should probably make this more clear, but
also the dmesg will still be useful in that it'll report what block
group was being worked on at the time. So it'll be possible to see
that in the debugfs output along with stripe info for that bg.


-- 
Chris Murphy

Re: Balance fails with unallocated diskspace

[Nisse Karlsson]

After some more balance passes with different values of -dusage and
removal of some files (around 50GB) I finally managed to convert the
volume back to raid1.
So my problem is solved, but it still seems like bug to me.

//Nisse

2016-06-15 18:29 GMT+02:00 Chris Murphy <lists@colorremedies.com>:
> On Wed, Jun 15, 2016 at 10:00 AM, Nisse Karlsson <nisse@xcorp.at> wrote:
>> Thanks,
>>
>> Will update the bug-report with more information.
>> Not sure I trust the device info in dmesg though, when running device
>> delete /dev/sdh /files it still refers to to sdb in dmesg. I think it
>> just shows the device that was used for mounting the fs.
>
> Hmm that's not very helpful if it's not reporting the device that's
> having the enospc problem...
>
> Anyway, the debugfs output should probably make this more clear, but
> also the dmesg will still be useful in that it'll report what block
> group was being worked on at the time. So it'll be possible to see
> that in the debugfs output along with stripe info for that bg.
>
>
> --
> Chris Murphy