Please review and comment, dealing with btrfs full issues

Please review and comment, dealing with btrfs full issues

[Marc MERLIN]

I've just written this new page:
http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html

First, are there problems in it?

Second, are there other FS full issues I should mention in it?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Please review and comment, dealing with btrfs full issues

[Brendan Hide]

On 05/05/14 14:16, Marc MERLIN wrote:
> I've just written this new page:
> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
>
> First, are there problems in it?
>
> Second, are there other FS full issues I should mention in it?
>
> Thanks,
> Marc
"In the case above, because the filesystem is only 55% full, I can ask 
balance to rewrite all chunks that are more than 55% full:

legolas:~# btrfs balance start -dusage=50 /mnt/btrfs_pool1"

"-dusage=50" will balance all chunks that are 50% *or less* used, not 
more. The idea is that full chunks are better left alone while emptyish 
chunks are bundled together to make new full chunks, leaving big open 
areas for new chunks. Your process is good however - just the 
explanation that needs the tweak. :)

In your last example, a full rebalance is not necessary. If you want to 
clear all unnecessary chunks you can run the balance with -dusage=80 
(636GB/800GB~=79%). That will cause a rebalance only of the data chunks 
that are 80% and less used, which would by necessity get about ~160GB 
worth chunks back out of data and available for re-use.

The issue I'm not sure of how to get through is if you can't balance 
*because* of ENOSPC errors. I'd probably start scouring the mailing list 
archives if I ever come across that.

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: Please review and comment, dealing with btrfs full issues

[Brendan Hide]

On 05/05/14 19:07, Brendan Hide wrote:
> On 05/05/14 14:16, Marc MERLIN wrote:
>> I've just written this new page:
>> http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html 
>>
>>
>> First, are there problems in it?
>>
>> Second, are there other FS full issues I should mention in it?
>>
>> Thanks,
>> Marc
> "In the case above, because the filesystem is only 55% full, I can ask 
> balance to rewrite all chunks that are more than 55% full:
>
> legolas:~# btrfs balance start -dusage=50 /mnt/btrfs_pool1"
>
> "-dusage=50" will balance all chunks that are 50% *or less* used, not 
> more. The idea is that full chunks are better left alone while 
> emptyish chunks are bundled together to make new full chunks, leaving 
> big open areas for new chunks. Your process is good however - just the 
> explanation that needs the tweak. :)
>
> In your last example, a full rebalance is not necessary. If you want 
> to clear all unnecessary chunks you can run the balance with 
> -dusage=80 (636GB/800GB~=79%). That will cause a rebalance only of the 
> data chunks that are 80% and less used, which would by necessity get 
> about ~160GB worth chunks back out of data and available for re-use.
>
> The issue I'm not sure of how to get through is if you can't balance 
> *because* of ENOSPC errors. I'd probably start scouring the mailing 
> list archives if I ever come across that.
>
Forgot this part: Also in your last example, you used "-dusage=0" and it 
balanced 91 chunks. That means you had 91 empty or very-close-to-empty 
chunks. ;)

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: Please review and comment, dealing with btrfs full issues

[Calvin Walton]

On Mon, 2014-05-05 at 19:07 +0200, Brendan Hide wrote:
> On 05/05/14 14:16, Marc MERLIN wrote:
> > I've just written this new page:
> > http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
> >
> > First, are there problems in it?
> >
> > Second, are there other FS full issues I should mention in it?
> >

> The issue I'm not sure of how to get through is if you can't balance 
> *because* of ENOSPC errors. I'd probably start scouring the mailing list 
> archives if I ever come across that.

The standard response on the mailing list for this issue is to
temporarily add an additional device to the filesystem (even e.g. a 4GB
USB flash drive is often enough) - this will add space to allocate a few
new chunks, allowing the balance to proceed. You can remove the extra
device after the balance completes.

-- 
Calvin Walton <calvin.walton@kepstin.ca>

Re: Please review and comment, dealing with btrfs full issues

[Russell Coker]

On Mon, 5 May 2014 14:36:09 Calvin Walton wrote:
> The standard response on the mailing list for this issue is to
> temporarily add an additional device to the filesystem (even e.g. a 4GB
> USB flash drive is often enough) - this will add space to allocate a few
> new chunks, allowing the balance to proceed. You can remove the extra
> device after the balance completes.

I once had a situation where adding a new device failed due to lack of space.  
That was after I had tried to balance without adding a new device and to 
remove snapshots.

http://etbe.coker.com.au/2014/04/26/btrfs-status-april-2014/

Recently kernel 3.14 allowed fixing a metadata space error that seemed to be 
impossible to solve with 3.13.  So it's possible that some of my other 
problems with a lack of metadata space could have been solved with kernel 3.14 
too.

-- 
My Main Blog         http://etbe.coker.com.au/
My Documents Blog    http://doc.coker.com.au/

Re: Please review and comment, dealing with btrfs full issues

[Marc MERLIN]

On Mon, May 05, 2014 at 07:07:29PM +0200, Brendan Hide wrote:
> "In the case above, because the filesystem is only 55% full, I can
> ask balance to rewrite all chunks that are more than 55% full:
> 
> legolas:~# btrfs balance start -dusage=50 /mnt/btrfs_pool1"
> 
> "-dusage=50" will balance all chunks that are 50% *or less* used,

Sorry, I actually meant to write 55 there.

> not more. The idea is that full chunks are better left alone while
> emptyish chunks are bundled together to make new full chunks,
> leaving big open areas for new chunks. Your process is good however
> - just the explanation that needs the tweak. :)

Mmmh, so if I'm 55% full, should I actually use -dusage=45 or 55?

> In your last example, a full rebalance is not necessary. If you want
> to clear all unnecessary chunks you can run the balance with
> -dusage=80 (636GB/800GB~=79%). That will cause a rebalance only of
> the data chunks that are 80% and less used, which would by necessity
> get about ~160GB worth chunks back out of data and available for
> re-use.

So in my case when I hit that case, I had to use dusage=0 to recover.
Anything above that just didn't work.

On Mon, May 05, 2014 at 07:09:22PM +0200, Brendan Hide wrote:
> Forgot this part: Also in your last example, you used "-dusage=0"
> and it balanced 91 chunks. That means you had 91 empty or
> very-close-to-empty chunks. ;)

Correct. That FS was very mis-balanced.

On Mon, May 05, 2014 at 02:36:09PM -0400, Calvin Walton wrote:
> The standard response on the mailing list for this issue is to
> temporarily add an additional device to the filesystem (even e.g. a 4GB
> USB flash drive is often enough) - this will add space to allocate a few
> new chunks, allowing the balance to proceed. You can remove the extra
> device after the balance completes.

I just added that tip, thank you.
 
On Tue, May 06, 2014 at 02:41:16PM +1000, Russell Coker wrote:
> Recently kernel 3.14 allowed fixing a metadata space error that seemed to be 
> impossible to solve with 3.13.  So it's possible that some of my other 
> problems with a lack of metadata space could have been solved with kernel 3.14 
> too.

Good point. I added that tip too.

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: Please review and comment, dealing with btrfs full issues

[Brendan Hide]

Hi, Marc. Inline below. :)

On 2014/05/06 02:19 PM, Marc MERLIN wrote:
> On Mon, May 05, 2014 at 07:07:29PM +0200, Brendan Hide wrote:
>> "In the case above, because the filesystem is only 55% full, I can
>> ask balance to rewrite all chunks that are more than 55% full:
>>
>> legolas:~# btrfs balance start -dusage=50 /mnt/btrfs_pool1"
>>
>> "-dusage=50" will balance all chunks that are 50% *or less* used,
> Sorry, I actually meant to write 55 there.
>
>> not more. The idea is that full chunks are better left alone while
>> emptyish chunks are bundled together to make new full chunks,
>> leaving big open areas for new chunks. Your process is good however
>> - just the explanation that needs the tweak. :)
> Mmmh, so if I'm 55% full, should I actually use -dusage=45 or 55?

As usual, it depends on what end-result you want. Paranoid rebalancing - 
always ensuring there are as many free chunks as possible - is totally 
unnecessary. There may be more good reasons to rebalance - but I'm only 
aware of two: a) to avoid ENOSPC due to running out of free chunks; and 
b) to change allocation type.

If you want all chunks either full or empty (except for that last chunk 
which will be somewhere inbetween), -dusage=55 will get you 99% there.
>> In your last example, a full rebalance is not necessary. If you want
>> to clear all unnecessary chunks you can run the balance with
>> -dusage=80 (636GB/800GB~=79%). That will cause a rebalance only of
>> the data chunks that are 80% and less used, which would by necessity
>> get about ~160GB worth chunks back out of data and available for
>> re-use.
> So in my case when I hit that case, I had to use dusage=0 to recover.
> Anything above that just didn't work.

I suspect when using more than zero the first chunk it wanted to balance 
wasn't empty - and it had nowhere to put it. Then when you did dusage=0, 
it didn't need a destination for the data. That is actually an 
interesting workaround for that case.
> On Mon, May 05, 2014 at 07:09:22PM +0200, Brendan Hide wrote:
>> Forgot this part: Also in your last example, you used "-dusage=0"
>> and it balanced 91 chunks. That means you had 91 empty or
>> very-close-to-empty chunks. ;)
> Correct. That FS was very mis-balanced.
>
> On Mon, May 05, 2014 at 02:36:09PM -0400, Calvin Walton wrote:
>> The standard response on the mailing list for this issue is to
>> temporarily add an additional device to the filesystem (even e.g. a 4GB
>> USB flash drive is often enough) - this will add space to allocate a few
>> new chunks, allowing the balance to proceed. You can remove the extra
>> device after the balance completes.
> I just added that tip, thank you.
>   
> On Tue, May 06, 2014 at 02:41:16PM +1000, Russell Coker wrote:
>> Recently kernel 3.14 allowed fixing a metadata space error that seemed to be
>> impossible to solve with 3.13.  So it's possible that some of my other
>> problems with a lack of metadata space could have been solved with kernel 3.14
>> too.
> Good point. I added that tip too.
>
> Thanks,
> Marc


-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: Please review and comment, dealing with btrfs full issues

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2750 bytes --]

On Tue, May 06, 2014 at 06:30:31PM +0200, Brendan Hide wrote:
> Hi, Marc. Inline below. :)
> 
> On 2014/05/06 02:19 PM, Marc MERLIN wrote:
> >On Mon, May 05, 2014 at 07:07:29PM +0200, Brendan Hide wrote:
> >>"In the case above, because the filesystem is only 55% full, I can
> >>ask balance to rewrite all chunks that are more than 55% full:
> >>
> >>legolas:~# btrfs balance start -dusage=50 /mnt/btrfs_pool1"
> >>
> >>"-dusage=50" will balance all chunks that are 50% *or less* used,
> >Sorry, I actually meant to write 55 there.
> >
> >>not more. The idea is that full chunks are better left alone while
> >>emptyish chunks are bundled together to make new full chunks,
> >>leaving big open areas for new chunks. Your process is good however
> >>- just the explanation that needs the tweak. :)
> >Mmmh, so if I'm 55% full, should I actually use -dusage=45 or 55?
> 
> As usual, it depends on what end-result you want. Paranoid rebalancing -
> always ensuring there are as many free chunks as possible - is totally
> unnecessary. There may be more good reasons to rebalance - but I'm only
> aware of two: a) to avoid ENOSPC due to running out of free chunks; and b)
> to change allocation type.

   c) its original reason: to redistribute the data on the FS, for
   example in the case of a new device being added or removed.

> If you want all chunks either full or empty (except for that last chunk
> which will be somewhere inbetween), -dusage=55 will get you 99% there.
> >>In your last example, a full rebalance is not necessary. If you want
> >>to clear all unnecessary chunks you can run the balance with
> >>-dusage=80 (636GB/800GB~=79%). That will cause a rebalance only of
> >>the data chunks that are 80% and less used, which would by necessity
> >>get about ~160GB worth chunks back out of data and available for
> >>re-use.
> >So in my case when I hit that case, I had to use dusage=0 to recover.
> >Anything above that just didn't work.
> 
> I suspect when using more than zero the first chunk it wanted to balance
> wasn't empty - and it had nowhere to put it. Then when you did dusage=0, it
> didn't need a destination for the data. That is actually an interesting
> workaround for that case.

   I've actually looked into implementing a "smallest=n" filter that
would taken only the n least-full chunks (by fraction) and balance
those. However, it's not entirely trivial to do efficiently with the
current filtering code.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
   --- Hail and greetings.  We are a flat-pack invasion force from ---   
                     Planet Ikea. We come in pieces.                     

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: Please review and comment, dealing with btrfs full issues

[Duncan]

Brendan Hide posted on Tue, 06 May 2014 18:30:31 +0200 as excerpted:

>> So in my case when I hit that case, I had to use dusage=0 to recover.
>> Anything above that just didn't work.
> 
> I suspect when using more than zero the first chunk it wanted to balance
> wasn't empty - and it had nowhere to put it. Then when you did dusage=0,
> it didn't need a destination for the data. That is actually an
> interesting workaround for that case.

I've actually used -Xusage=0 (where X=m or d, obviously) for exactly 
that.  If every last bit of filesystem is allocated so another chunk 
simply cannot be written in ordered to rewrite partially used chunks 
into, BUT the spread between allocated and actually used is quite high, 
there's a reasonably good chance that at least one of those allocated 
chunks is entirely empty, and -Xusage=0 allows returning it to the 
unallocated pool without actually requiring a new chunk allocation to do 
so.

With luck, that will free at least one zero-usage chunk (two for metadata 
dup, but it would both allocate and return to unallocated in pairs as so 
it balances out), allowing the user to rerun balance, this time with a 
higher -Xusage=.  


The other known valid use-case for -Xusage=0 is when freeing the 
extraneous zero-usage single-mode chunks first created by mkfs.btrfs as 
part of the mkfs process, so they don't clutter up the btrfs filesystem df 
output. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Please review and comment, dealing with btrfs full issues

[Duncan]

Brendan Hide posted on Mon, 05 May 2014 19:07:29 +0200 as excerpted:

> In your last example, a full rebalance is not necessary. If you want to
> clear all unnecessary chunks you can run the balance with -dusage=80
> (636GB/800GB~=79%). That will cause a rebalance only of the data chunks
> that are 80% and less used, which would by necessity get about ~160GB
> worth chunks back out of data and available for re-use.

I don't believe your reasoning is entirely valid.  In the pathologically 
extreme case, all data chunks are 80% plus one byte used, save for one 
that's empty, and a -dusage=80 balance will return just that one empty 
chunk (but as a consequence return real fast as there's nothing else to 
do), just the same as -dusage=0, but -dusage=81 would return the 20% 
expected, but would take as long as -dusage=100 (or simply -d).

OTOH, if all chunks but one are 100% full, with the remaining chunks 
empty, a balance with -dusage=0 will have (within one chunk, depending on 
how full it is) the same effect as -dusage=80, with both completing at 
the same fast speed and returning more chunks than expected.  But simple 
-d (or -dusage=100) would take much MUCH longer, but return nothing 
(well, within one chunk) more than the -dusage=0 would.

Actual results, therefore, depend on file AND chunk fragmentation and 
usage.  Using the autodefrag mount option should increase the likelihood 
of better results faster, at lower -Xusage= numbers, but the only way to 
be sure is to do a run with numbers you think look reasonable given the 
situation you are in and the time you're willing to wait, and then rerun 
it again if the results weren't good enough.

Your algorithm is certainly a good first approximation, but it's just 
that, an approximation.  People who want the balance over as soon as 
possible but are willing to retry if necessary, should probably try a 
lower usage= number, while those who are leaving on a trip in the morning 
and want it to run overnight, with little chance of redoing it if the 
results weren't good enough, should probably pick a higher usage= number, 
even perhaps usage=95 or the like, to recover as much space as possible, 
without wasting time rebalancing the chunks that really ARE 100% full 
already.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Please review and comment, dealing with btrfs full issues

[David Sterba]

On Tue, May 06, 2014 at 05:43:24PM +0100, Hugo Mills wrote:
> > >So in my case when I hit that case, I had to use dusage=0 to recover.
> > >Anything above that just didn't work.
> > 
> > I suspect when using more than zero the first chunk it wanted to balance
> > wasn't empty - and it had nowhere to put it. Then when you did dusage=0, it
> > didn't need a destination for the data. That is actually an interesting
> > workaround for that case.
> 
>    I've actually looked into implementing a "smallest=n" filter that
> would taken only the n least-full chunks (by fraction) and balance
> those. However, it's not entirely trivial to do efficiently with the
> current filtering code.

I've prototyped something similar, to limit the number of balanced
chunks by a number. To achieve "n least-full chunks" would be an
iterative process of increasing the usage filter and limiting the number
of chunks until the desired N is reached.

N=n
F=0
while (N > 0) {
	balance -dusage=F,limit=N
	N -= <number of balanced chunks>
	F++
}

The patch is in branch dev/balance-limit in my git repos.

We can then implement the n-least-full as a synthetic filter from
userspace.

Smallest-n balance filter (was Re: Please review and comment, dealing with btrfs full issues)

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2398 bytes --]

On Wed, May 07, 2014 at 04:09:27PM +0200, David Sterba wrote:
> On Tue, May 06, 2014 at 05:43:24PM +0100, Hugo Mills wrote:
> > > >So in my case when I hit that case, I had to use dusage=0 to recover.
> > > >Anything above that just didn't work.
> > > 
> > > I suspect when using more than zero the first chunk it wanted to balance
> > > wasn't empty - and it had nowhere to put it. Then when you did dusage=0, it
> > > didn't need a destination for the data. That is actually an interesting
> > > workaround for that case.
> > 
> >    I've actually looked into implementing a "smallest=n" filter that
> > would taken only the n least-full chunks (by fraction) and balance
> > those. However, it's not entirely trivial to do efficiently with the
> > current filtering code.
> 
> I've prototyped something similar, to limit the number of balanced
> chunks by a number. To achieve "n least-full chunks" would be an
> iterative process of increasing the usage filter and limiting the number
> of chunks until the desired N is reached.
> 
> N=n
> F=0
> while (N > 0) {
> 	balance -dusage=F,limit=N
> 	N -= <number of balanced chunks>
> 	F++
> }
> 
> The patch is in branch dev/balance-limit in my git repos.
> 
> We can then implement the n-least-full as a synthetic filter from
> userspace.

   This is inefficient, because we've got an O(m) pass through all the
chunks for every call. If we reduce the number of calls by increasing
the increment of F (F+=3, say), then we risk overbalancing, or missing
out on smaller chunks we could have balanced earlier. From a practical
point of view, it may make little difference, but the computer
scientist in me is going "ewwwww".

   The other method, for small n only, would be to construct the list
first, an O(m log n) operation for a filesystem of size m, requiring
O(n) storage, and then iterate over just those chunks. The problem
with that is the storage requirements, and keeping track of the state
of the list for restart purposes. [actually, there's probably an O(m)
algorithm to get the n smallest items, but those are a bit
complicated]

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
   --- A diverse working environment:  Di longer you vork here, di ---   
                             verse it gets.                              

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: Smallest-n balance filter (was Re: Please review and comment, dealing with btrfs full issues)

[David Sterba]

On Wed, May 07, 2014 at 03:23:01PM +0100, Hugo Mills wrote:
> > N=n
> > F=0
> > while (N > 0) {
> > 	balance -dusage=F,limit=N
> > 	N -= <number of balanced chunks>
> > 	F++
> > }
> > 
> > The patch is in branch dev/balance-limit in my git repos.
> > 
> > We can then implement the n-least-full as a synthetic filter from
> > userspace.
> 
>    This is inefficient, because we've got an O(m) pass through all the
> chunks for every call. If we reduce the number of calls by increasing
> the increment of F (F+=3, say), then we risk overbalancing, or missing
> out on smaller chunks we could have balanced earlier. From a practical
> point of view, it may make little difference, but the computer
> scientist in me is going "ewwwww".

I'm trying to find the practical way, no doubts about the inefficiencies.
The +1 increment was meant to outline the idea, I'm usually using the
sequence 0, 1, 5, 10, [etc +10].

I think we can afford some inaccuracy, I as a user would not mind if
there's some overbalancing (within a sane margin).

>    The other method, for small n only, would be to construct the list
> first, an O(m log n) operation for a filesystem of size m, requiring
> O(n) storage, and then iterate over just those chunks.

The size of filesystem matters, but the scanning phase of balance uses
in-memory structures and this should not be that bad for terabyte-sized
filesystems (ie. number of blockgoups will be "some thousands").

Possibly we can stop looking for new chunks in the first phase of
balance when there are already N candidate chunks found, and process them.

> The problem with that is the storage requirements, and keeping track
> of the state of the list for restart purposes. [actually, there's
> probably an O(m) algorithm to get the n smallest items, but those are
> a bit complicated]

If the filesystem is under load, the chunks' usage may increase or
decrease in time and as we know, balance takes time, so the
chunk-todo-list may look different when next one is about to be
processed.

But yeah, this could be a cheaper check to skip a given chunk if it's
out of the filter criteria than going through the whole list again.