Balance of Raid1 pool, does not balance properly.

Balance of Raid1 pool, does not balance properly.

[Karsten Vinding]

Hello.

I have a Raid1 pool consisting of 6 drives, 3 3TB disks and 3 2TB disks.

Until yesterday it consisted of 3 2TB disks, 2 3TB disks and one 1TB disk.

I replaced the 1TB disk as the pool was close to full.

Replacement went well, and I ended up with 5 almost full disks, and 1 
3TB disk that was one third full.

So I kicked of a balance, expecting it to balance the data as evenly as 
possible on the 6 disks (btrfs balace start poolname).

The balance ran fine but I ended up with this:

Total devices 6 FS bytes used 5.66TiB
         devid    9 size 2.73TiB used 2.69TiB path /dev/sdf
         devid   10 size 1.82TiB used 1.78TiB path /dev/sdb
         devid   11 size 1.82TiB used 1.73TiB path /dev/sdc
         devid   12 size 1.82TiB used 1.73TiB path /dev/sdd
         devid   13 size 2.73TiB used 2.65TiB path /dev/sde
         devid   15 size 2.73TiB used 817.87GiB path /dev/sdg

The sixth drive sdg, is still only one third full.

How do I force BTRFS to distribute the data more evenly across the disks?

The way BTRFS has done it now, will bring problems, when I write more 
data to the array.

Re: Balance of Raid1 pool, does not balance properly.

[Duncan]

Karsten Vinding posted on Tue, 08 Jan 2019 20:40:12 +0100 as excerpted:

> Hello.
> 
> I have a Raid1 pool consisting of 6 drives, 3 3TB disks and 3 2TB disks.
> 
> Until yesterday it consisted of 3 2TB disks, 2 3TB disks and one 1TB
> disk.
> 
> I replaced the 1TB disk as the pool was close to full.
> 
> Replacement went well, and I ended up with 5 almost full disks, and 1
> 3TB disk that was one third full.
> 
> So I kicked of a balance, expecting it to balance the data as evenly as
> possible on the 6 disks (btrfs balace start poolname).
> 
> The balance ran fine but I ended up with this:
> 
> Total devices 6 FS bytes used 5.66TiB
>          devid    9 size 2.73TiB used 2.69TiB path /dev/sdf
>          devid   10 size 1.82TiB used 1.78TiB path /dev/sdb
>          devid   11 size 1.82TiB used 1.73TiB path /dev/sdc
>          devid   12 size 1.82TiB used 1.73TiB path /dev/sdd
>          devid   13 size 2.73TiB used 2.65TiB path /dev/sde
>          devid   15 size 2.73TiB used 817.87GiB path /dev/sdg
> 
> The sixth drive sdg, is still only one third full.
> 
> How do I force BTRFS to distribute the data more evenly across the
> disks?
> 
> The way BTRFS has done it now, will bring problems, when I write more
> data to the array.

After doing the btrfs replace to the larger device, did you resize to the 
full size of the larger device as noted in the btrfs-replace manpage (but 
before you do please post btrfs device usage from before, and then again 
after the resize, as below)?  I ask because that's an easy to forget step 
that you don't specifically mention doing.

If you didn't, that's your problem -- the filesystem on that device is 
still the size of the old device, and needs resized to the larger size of 
the new one, after which a balance should work as expected.

Note that there is very recently reported bug in the way btrfs filesystem 
usage reports the size in this case, adding the device slack to 
unallocated altho it can't actually be allocated by the filesystem at all 
as the filesystem size doesn't cover that space on that device.  I 
thought the bug didn't extend to show, which would indicate that you did 
the resize and just didn't mention it, but am asking as that's otherwise 
the most likely reason for the listed behavior.

I /believe/ btrfs device usage indicates the extra space in its device 
slack line, but the reporter had already increased the size by the time 
of posting and hadn't run btrfs device usage previous to that, and it was 
non-dev list regulars in the discussion that didn't know for sure and 
didn't have a replaced and as yet unresized-filesystem device to check, 
so we haven't actually verified whether it displays correctly or not yet.

Thus the request for the btrfs device usage output, to verify all that 
for both your case and the previous similar thread...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Balance of Raid1 pool, does not balance properly.

[Karsten Vinding]

Just a short answer.

I didn't use the replace command.

I added the new drive to the pool / array, and checked that it was 
registered.
Following that I removed the 1TB drive with "btrfs device delete <drive>".

As far as I know this should avoid the need to resize the new drive.

"btrfs fi us" shows all the space as available. The drive shows 1.7TB as 
unallocated.

I have started moving some data of the array to give btrfs some more 
room to move, and will follow what happens when I try a new balance 
later on.

Re: Balance of Raid1 pool, does not balance properly.

[Peter Becker]

if, before the first balance run, many blocks were filled only in
parts, a behavior like this can be happen, because btrfs aggregate the
used space in a new block.
a second full balance run, should solve this.

you didn't need to move data away. btrfs only need a few GB of
unalloceted space for balancing. (see the output of "btrfs fi us
/mnt")

In generell i prefer a small balance-script like:

btrfs balance start -musage=50 -dusage=50 /mnt
btrfs balance start -musage=90 -dusage=90 /mnt
btrfs balance start /mnt

this aggregate all blocks that are filled to less than 50%, the the
same with blocks that are filled to less than 90%.
the last line will do a full rebalance if you real need. (this moves ALL blocks)

Am Mi., 9. Jan. 2019 um 18:23 Uhr schrieb Karsten Vinding
<karstenvinding@gmail.com>:
>
> Just a short answer.
>
> I didn't use the replace command.
>
> I added the new drive to the pool / array, and checked that it was
> registered.
> Following that I removed the 1TB drive with "btrfs device delete <drive>".
>
> As far as I know this should avoid the need to resize the new drive.
>
> "btrfs fi us" shows all the space as available. The drive shows 1.7TB as
> unallocated.
>
> I have started moving some data of the array to give btrfs some more
> room to move, and will follow what happens when I try a new balance
> later on.

Re: Balance of Raid1 pool, does not balance properly.

[Karsten Vinding]

Thanks for the answer.

I had already tried "

btrfs balance start -musage=75 -dusage=75 /mnt

It only reallocated 3 blocks.

But after I had moved some data, I ran it with 90, and it realocated a lot more.

I have a new balance running, and this time it seems to be doing its job. At least data is distributed more evenly across disks.

Its not done yet, but its looking fine.

Re: Balance of Raid1 pool, does not balance properly.

[Peter Becker]

for the future you can create a little cron script to avoid this issue.
the data distribution should then be more regular, even if you delete
or overwrite a lot.

deppend on your workload use the right :

# cron.daily / cron.weekly
btrfs balance start -musage=0 -dusage=0 # releases empty blocks for
more unallocated space with can used by future balance runs
btrfs balance start -musage=25 -dusage=25
btrfs balance start -musage=50 -dusage=50


# cron.weekly / cron.monthly
btrfs balance start -musage=90 -dusage=90

A full balance run is then only needed if you add a new device to the
pool or replace a device with a bigger one.


if you are more interesseted about the block usage of your current fs
you can use this little script:

cd /tmp
wget https://raw.githubusercontent.com/kdave/btrfs-progs/master/btrfs-debugfs
chmod +x btrfs-debugfs
stats=$(sudo ./btrfs-debugfs -b /)

echo "00-49: " $(echo "$stats" | grep "usage 0.[0-4]" -c)
echo "50-79: " $(echo "$stats" | grep "usage 0.[5-7]" -c)
echo "80-89: " $(echo "$stats" | grep "usage 0.8" -c)
echo "90-99: " $(echo "$stats" | grep "usage 0.9" -c)
echo "100:   " $(echo "$stats" | grep "usage 1." -c)

The btrfs-debugfs script is from the btrfs progs source and report the
usage of each block group. The following script groups the result.

This script should take less than a few minutes to complete.

Or you can use btrfs-headmap with create a picture:
https://github.com/knorrie/btrfs-heatmap

Am Do., 10. Jan. 2019 um 19:21 Uhr schrieb Karsten Vinding
<karstenvinding@gmail.com>:
>
> Thanks for the answer.
>
> I had already tried "
>
> btrfs balance start -musage=75 -dusage=75 /mnt
>
> It only reallocated 3 blocks.
>
> But after I had moved some data, I ran it with 90, and it realocated a lot more.
>
> I have a new balance running, and this time it seems to be doing its job. At least data is distributed more evenly across disks.
>
> Its not done yet, but its looking fine.
>
>

Re: Balance of Raid1 pool, does not balance properly.

[Karsten Vinding]

Thanks for all your suggestions.


I will save your instructions for future reference and use.

The current complete balance is 90% done, and the data are now perfectly 
evenly distributed across the drives.

Vielen danke für deine hilfe :)