* Balance of Raid1 pool, does not balance properly.
@ 2019-01-08 19:40 Karsten Vinding
2019-01-09 1:08 ` Duncan
2019-01-09 17:20 ` Karsten Vinding
0 siblings, 2 replies; 7+ messages in thread
From: Karsten Vinding @ 2019-01-08 19:40 UTC (permalink / raw)
To: linux-btrfs
Hello.
I have a Raid1 pool consisting of 6 drives, 3 3TB disks and 3 2TB disks.
Until yesterday it consisted of 3 2TB disks, 2 3TB disks and one 1TB disk.
I replaced the 1TB disk as the pool was close to full.
Replacement went well, and I ended up with 5 almost full disks, and 1
3TB disk that was one third full.
So I kicked of a balance, expecting it to balance the data as evenly as
possible on the 6 disks (btrfs balace start poolname).
The balance ran fine but I ended up with this:
Total devices 6 FS bytes used 5.66TiB
devid 9 size 2.73TiB used 2.69TiB path /dev/sdf
devid 10 size 1.82TiB used 1.78TiB path /dev/sdb
devid 11 size 1.82TiB used 1.73TiB path /dev/sdc
devid 12 size 1.82TiB used 1.73TiB path /dev/sdd
devid 13 size 2.73TiB used 2.65TiB path /dev/sde
devid 15 size 2.73TiB used 817.87GiB path /dev/sdg
The sixth drive sdg, is still only one third full.
How do I force BTRFS to distribute the data more evenly across the disks?
The way BTRFS has done it now, will bring problems, when I write more
data to the array.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Balance of Raid1 pool, does not balance properly.
2019-01-08 19:40 Balance of Raid1 pool, does not balance properly Karsten Vinding
@ 2019-01-09 1:08 ` Duncan
2019-01-09 17:20 ` Karsten Vinding
1 sibling, 0 replies; 7+ messages in thread
From: Duncan @ 2019-01-09 1:08 UTC (permalink / raw)
To: linux-btrfs
Karsten Vinding posted on Tue, 08 Jan 2019 20:40:12 +0100 as excerpted:
> Hello.
>
> I have a Raid1 pool consisting of 6 drives, 3 3TB disks and 3 2TB disks.
>
> Until yesterday it consisted of 3 2TB disks, 2 3TB disks and one 1TB
> disk.
>
> I replaced the 1TB disk as the pool was close to full.
>
> Replacement went well, and I ended up with 5 almost full disks, and 1
> 3TB disk that was one third full.
>
> So I kicked of a balance, expecting it to balance the data as evenly as
> possible on the 6 disks (btrfs balace start poolname).
>
> The balance ran fine but I ended up with this:
>
> Total devices 6 FS bytes used 5.66TiB
> devid 9 size 2.73TiB used 2.69TiB path /dev/sdf
> devid 10 size 1.82TiB used 1.78TiB path /dev/sdb
> devid 11 size 1.82TiB used 1.73TiB path /dev/sdc
> devid 12 size 1.82TiB used 1.73TiB path /dev/sdd
> devid 13 size 2.73TiB used 2.65TiB path /dev/sde
> devid 15 size 2.73TiB used 817.87GiB path /dev/sdg
>
> The sixth drive sdg, is still only one third full.
>
> How do I force BTRFS to distribute the data more evenly across the
> disks?
>
> The way BTRFS has done it now, will bring problems, when I write more
> data to the array.
After doing the btrfs replace to the larger device, did you resize to the
full size of the larger device as noted in the btrfs-replace manpage (but
before you do please post btrfs device usage from before, and then again
after the resize, as below)? I ask because that's an easy to forget step
that you don't specifically mention doing.
If you didn't, that's your problem -- the filesystem on that device is
still the size of the old device, and needs resized to the larger size of
the new one, after which a balance should work as expected.
Note that there is very recently reported bug in the way btrfs filesystem
usage reports the size in this case, adding the device slack to
unallocated altho it can't actually be allocated by the filesystem at all
as the filesystem size doesn't cover that space on that device. I
thought the bug didn't extend to show, which would indicate that you did
the resize and just didn't mention it, but am asking as that's otherwise
the most likely reason for the listed behavior.
I /believe/ btrfs device usage indicates the extra space in its device
slack line, but the reporter had already increased the size by the time
of posting and hadn't run btrfs device usage previous to that, and it was
non-dev list regulars in the discussion that didn't know for sure and
didn't have a replaced and as yet unresized-filesystem device to check,
so we haven't actually verified whether it displays correctly or not yet.
Thus the request for the btrfs device usage output, to verify all that
for both your case and the previous similar thread...
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Balance of Raid1 pool, does not balance properly.
2019-01-08 19:40 Balance of Raid1 pool, does not balance properly Karsten Vinding
2019-01-09 1:08 ` Duncan
@ 2019-01-09 17:20 ` Karsten Vinding
2019-01-09 20:47 ` Peter Becker
1 sibling, 1 reply; 7+ messages in thread
From: Karsten Vinding @ 2019-01-09 17:20 UTC (permalink / raw)
To: linux-btrfs
Just a short answer.
I didn't use the replace command.
I added the new drive to the pool / array, and checked that it was
registered.
Following that I removed the 1TB drive with "btrfs device delete <drive>".
As far as I know this should avoid the need to resize the new drive.
"btrfs fi us" shows all the space as available. The drive shows 1.7TB as
unallocated.
I have started moving some data of the array to give btrfs some more
room to move, and will follow what happens when I try a new balance
later on.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Balance of Raid1 pool, does not balance properly.
2019-01-09 17:20 ` Karsten Vinding
@ 2019-01-09 20:47 ` Peter Becker
2019-01-10 18:21 ` Karsten Vinding
0 siblings, 1 reply; 7+ messages in thread
From: Peter Becker @ 2019-01-09 20:47 UTC (permalink / raw)
To: Karsten Vinding; +Cc: linux-btrfs
if, before the first balance run, many blocks were filled only in
parts, a behavior like this can be happen, because btrfs aggregate the
used space in a new block.
a second full balance run, should solve this.
you didn't need to move data away. btrfs only need a few GB of
unalloceted space for balancing. (see the output of "btrfs fi us
/mnt")
In generell i prefer a small balance-script like:
btrfs balance start -musage=50 -dusage=50 /mnt
btrfs balance start -musage=90 -dusage=90 /mnt
btrfs balance start /mnt
this aggregate all blocks that are filled to less than 50%, the the
same with blocks that are filled to less than 90%.
the last line will do a full rebalance if you real need. (this moves ALL blocks)
Am Mi., 9. Jan. 2019 um 18:23 Uhr schrieb Karsten Vinding
<karstenvinding@gmail.com>:
>
> Just a short answer.
>
> I didn't use the replace command.
>
> I added the new drive to the pool / array, and checked that it was
> registered.
> Following that I removed the 1TB drive with "btrfs device delete <drive>".
>
> As far as I know this should avoid the need to resize the new drive.
>
> "btrfs fi us" shows all the space as available. The drive shows 1.7TB as
> unallocated.
>
> I have started moving some data of the array to give btrfs some more
> room to move, and will follow what happens when I try a new balance
> later on.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Balance of Raid1 pool, does not balance properly.
2019-01-09 20:47 ` Peter Becker
@ 2019-01-10 18:21 ` Karsten Vinding
2019-01-10 18:58 ` Peter Becker
0 siblings, 1 reply; 7+ messages in thread
From: Karsten Vinding @ 2019-01-10 18:21 UTC (permalink / raw)
To: Peter Becker; +Cc: linux-btrfs
Thanks for the answer.
I had already tried "
btrfs balance start -musage=75 -dusage=75 /mnt
It only reallocated 3 blocks.
But after I had moved some data, I ran it with 90, and it realocated a lot more.
I have a new balance running, and this time it seems to be doing its job. At least data is distributed more evenly across disks.
Its not done yet, but its looking fine.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Balance of Raid1 pool, does not balance properly.
2019-01-10 18:21 ` Karsten Vinding
@ 2019-01-10 18:58 ` Peter Becker
2019-01-11 22:04 ` Karsten Vinding
0 siblings, 1 reply; 7+ messages in thread
From: Peter Becker @ 2019-01-10 18:58 UTC (permalink / raw)
To: Karsten Vinding; +Cc: linux-btrfs
for the future you can create a little cron script to avoid this issue.
the data distribution should then be more regular, even if you delete
or overwrite a lot.
deppend on your workload use the right :
# cron.daily / cron.weekly
btrfs balance start -musage=0 -dusage=0 # releases empty blocks for
more unallocated space with can used by future balance runs
btrfs balance start -musage=25 -dusage=25
btrfs balance start -musage=50 -dusage=50
# cron.weekly / cron.monthly
btrfs balance start -musage=90 -dusage=90
A full balance run is then only needed if you add a new device to the
pool or replace a device with a bigger one.
if you are more interesseted about the block usage of your current fs
you can use this little script:
cd /tmp
wget https://raw.githubusercontent.com/kdave/btrfs-progs/master/btrfs-debugfs
chmod +x btrfs-debugfs
stats=$(sudo ./btrfs-debugfs -b /)
echo "00-49: " $(echo "$stats" | grep "usage 0.[0-4]" -c)
echo "50-79: " $(echo "$stats" | grep "usage 0.[5-7]" -c)
echo "80-89: " $(echo "$stats" | grep "usage 0.8" -c)
echo "90-99: " $(echo "$stats" | grep "usage 0.9" -c)
echo "100: " $(echo "$stats" | grep "usage 1." -c)
The btrfs-debugfs script is from the btrfs progs source and report the
usage of each block group. The following script groups the result.
This script should take less than a few minutes to complete.
Or you can use btrfs-headmap with create a picture:
https://github.com/knorrie/btrfs-heatmap
Am Do., 10. Jan. 2019 um 19:21 Uhr schrieb Karsten Vinding
<karstenvinding@gmail.com>:
>
> Thanks for the answer.
>
> I had already tried "
>
> btrfs balance start -musage=75 -dusage=75 /mnt
>
> It only reallocated 3 blocks.
>
> But after I had moved some data, I ran it with 90, and it realocated a lot more.
>
> I have a new balance running, and this time it seems to be doing its job. At least data is distributed more evenly across disks.
>
> Its not done yet, but its looking fine.
>
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Balance of Raid1 pool, does not balance properly.
2019-01-10 18:58 ` Peter Becker
@ 2019-01-11 22:04 ` Karsten Vinding
0 siblings, 0 replies; 7+ messages in thread
From: Karsten Vinding @ 2019-01-11 22:04 UTC (permalink / raw)
To: Peter Becker; +Cc: linux-btrfs
Thanks for all your suggestions.
I will save your instructions for future reference and use.
The current complete balance is 90% done, and the data are now perfectly
evenly distributed across the drives.
Vielen danke für deine hilfe :)
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2019-01-11 22:04 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-01-08 19:40 Balance of Raid1 pool, does not balance properly Karsten Vinding
2019-01-09 1:08 ` Duncan
2019-01-09 17:20 ` Karsten Vinding
2019-01-09 20:47 ` Peter Becker
2019-01-10 18:21 ` Karsten Vinding
2019-01-10 18:58 ` Peter Becker
2019-01-11 22:04 ` Karsten Vinding
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).