ERROR: error during balancing '.' - No space left on device

ERROR: error during balancing '.' - No space left on device

[Marc MERLIN]

legolas:/mnt/btrfs_pool2# btrfs balance .
ERROR: error during balancing '.' - No space left on device
There may be more info in syslog - try dmesg | tail
[ 8454.159635] BTRFS info (device dm-1): relocating block group 288329039872 flags 1
[ 8590.167294] BTRFS info (device dm-1): relocating block group 232494465024 flags 1
[ 9200.801177] BTRFS info (device dm-1): relocating block group 85928706048 flags 1
[ 9533.830623] BTRFS info (device dm-1): 824 enospc errors during balance

But:
legolas:/mnt/btrfs_pool2# btrfs fi show `pwd`
Label: btrfs_pool2  uuid: 6afd4707-876c-46d6-9de2-21c4085b7bed
	Total devices 1 FS bytes used 646.41GiB
	devid    1 size 820.45GiB used 820.45GiB path /dev/mapper/disk2
Btrfs v3.12
legolas:/mnt/btrfs_pool2# 
Data, single: total=800.42GiB, used=636.91GiB
System, DUP: total=8.00MiB, used=92.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=10.00GiB, used=9.50GiB
Metadata, single: total=8.00MiB, used=0.00

I can't see how I'm full, and now that I can't run balance to fix
things, this is making things worse.
Kernel is 3.14.

What am I missing?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: ERROR: error during balancing '.' - No space left on device

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2063 bytes --]

On Sun, Mar 23, 2014 at 12:01:44AM -0700, Marc MERLIN wrote:
> legolas:/mnt/btrfs_pool2# btrfs balance .
> ERROR: error during balancing '.' - No space left on device
> There may be more info in syslog - try dmesg | tail
> [ 8454.159635] BTRFS info (device dm-1): relocating block group 288329039872 flags 1
> [ 8590.167294] BTRFS info (device dm-1): relocating block group 232494465024 flags 1
> [ 9200.801177] BTRFS info (device dm-1): relocating block group 85928706048 flags 1
> [ 9533.830623] BTRFS info (device dm-1): 824 enospc errors during balance
> 
> But:
> legolas:/mnt/btrfs_pool2# btrfs fi show `pwd`
> Label: btrfs_pool2  uuid: 6afd4707-876c-46d6-9de2-21c4085b7bed
> 	Total devices 1 FS bytes used 646.41GiB
> 	devid    1 size 820.45GiB used 820.45GiB path /dev/mapper/disk2
> Btrfs v3.12
> legolas:/mnt/btrfs_pool2# 
> Data, single: total=800.42GiB, used=636.91GiB
> System, DUP: total=8.00MiB, used=92.00KiB
> System, single: total=4.00MiB, used=0.00
> Metadata, DUP: total=10.00GiB, used=9.50GiB

  ^^^ This is where you're full. There's a block reserve here that
  (should be) usable for doing a balance, and it seems to be about 500
  MiB or so that's the point at which problems show up.

> Metadata, single: total=8.00MiB, used=0.00
> 
> I can't see how I'm full, and now that I can't run balance to fix
> things, this is making things worse.

   I think you probably shouldn't be doing a full balance, but a
filtered one:

# btrfs balance start -dusage=5 /mnt/btrfs_pool

which should only try to clean up chunks which have little usage (so
it's much faster to run).

> Kernel is 3.14.
> 
> What am I missing?

   Not much. We do seem to have a problem with not being able to run
balance in recent kernels under some circumstances -- you're not the
only person who's reported this kind of problem lately.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
    --- Don't worry, he's not drunk. He's like that all the time. ---    

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: ERROR: error during balancing '.' - No space left on device

[Duncan]

Marc MERLIN posted on Sun, 23 Mar 2014 00:01:44 -0700 as excerpted:

> legolas:/mnt/btrfs_pool2# btrfs balance .
> ERROR: error during balancing '.' - No space left on device

> But:
> # btrfs fi show `pwd`
> Label: btrfs_pool2  uuid: [...]
> 	Total devices 1 FS bytes used 646.41GiB
>       devid 1 size 820.45GiB used 820.45GiB path /dev/mapper/disk2
> # [btrfs fi df `pwd` (unused single metadata/system chunks omitted)]
> Data, single: total=800.42GiB, used=636.91GiB
> System, DUP: total=8.00MiB, used=92.00KiB
> Metadata, DUP: total=10.00GiB, used=9.50GiB
> 
> I can't see how I'm full, and now that I can't run balance to fix
> things, this is making things worse.
> Kernel is 3.14.
> 
> What am I missing?

When it rains, it pours.  What you're missing is that this is now the 
third thread in three days with exactly the same out-of-space-when-there-
appears-to-be-plenty problem, which is well explained and a solution 
presented, along with further discussion, on those threads.

Evidently you haven't read the others, but rather than rewrite a similar 
reply here with exactly the same explanation and fix, I'll just refer you 
to them.

First thread, subject, original poster, message-id, gmane link:

fresh btrfs filesystem, out of disk space, hundreds of gigs free

Jon Nelson <jnelson@jamponi.net>

<CAKuK5J0LyagDYro6Yq7h-p-VjbwrbeuFf9AdNwRNg0jv3bq9QQ@mail.gmail.com>

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/33640


Second thread (same problem altho he chose as a subject a btrfsck error 
that wasn't actually related to it) ...

free space inode generation (0) did not match free space cache
 generation

Hendrik Friedel <hendrik@friedels.name>

<532DD2DC.3030408@friedels.name>

http://permalink.gmane.org/gmane.comp.file-systems.btrfs/33647

Very briefly, from your btrfs fi show:

devid 1 size 820.45GiB used 820.45GiB

That's total size and total of allocated chunks.  They're equal, so 
there's no space left to allocate.  Normally you'd use a balance, but a 
normal balance won't work without some space left to write the new chunks 
it's going to copy the old data/metadata into.  There is however a 
solution, balance filters, using a command something like this (-d for 
data only since df says that's what has the extra chunks that need freed, 
usage=percent-full):

btrfs balance start -dusage=5 `pwd`

Tweak the N in usage=N as needed.

The explanation for all that is in the other threads.  If that doesn't 
work (tho it most likely will), there's other solutions available, see 
the no-space discussion in the FAQ on the wiki, again as linked in the 
other threads.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: ERROR: error during balancing '.' - No space left on device

[Marc MERLIN]

Both
legolas:/mnt/btrfs_pool2# btrfs balance start -v -dusage=5 /mnt/btrfs_pool2
legolas:/mnt/btrfs_pool2# btrfs balance start -v -dusage=0 /mnt/btrfs_pool2
failed unfortunately.

On Sun, Mar 23, 2014 at 12:26:32PM +0000, Duncan wrote:
> When it rains, it pours.  What you're missing is that this is now the 
> third thread in three days with exactly the same out-of-space-when-there-
> appears-to-be-plenty problem, which is well explained and a solution 
> presented, along with further discussion, on those threads.
> 
> Evidently you haven't read the others, but rather than rewrite a similar 
> reply here with exactly the same explanation and fix, I'll just refer you 
> to them.

Thanks. Indeed, while I spent most of yesterday dealing with 3 btrfs
filesystems, the one here that was hanging my laptop, the raid5 one that
was hanging repeatedly during balance, and then my main server were one
FS is so slow that it takes 8H to do an reflink copy or delete a backup
with 1 million inodes, I got behind on reading the list :)

Thanks for the pointers

> btrfs balance start -dusage=5 `pwd`
> 
> Tweak the N in usage=N as needed.

I had actually tried this, but it failed too:
legolas:/mnt/btrfs_pool2# btrfs balance start -v -dusage=5 /mnt/btrfs_pool2
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=5
ERROR: error during balancing '/mnt/btrfs_pool2' - No space left on device

But I now just found
https://btrfs.wiki.kernel.org/index.php/Balance_Filters
and tried -dusage=0
 
On Sun, Mar 23, 2014 at 11:47:12AM +0000, Hugo Mills wrote:
>    I think you probably shouldn't be doing a full balance, but a
> filtered one:
> 
> # btrfs balance start -dusage=5 /mnt/btrfs_pool
> 
> which should only try to clean up chunks which have little usage (so
> it's much faster to run).

Thanks for the other answer Hugo.

So, now I'm down to 
legolas:/mnt/btrfs_pool2# btrfs balance start -v -dusage=0 /mnt/btrfs_pool2
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=0
ERROR: error during balancing '/mnt/btrfs_pool2' - No space left on device

Looks like there is no good way out of this, so I'll start deleting
snapshots.

Hopefully this will be handled better in later code.

Thanks for the answer,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: ERROR: error during balancing '.' - No space left on device

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 3060 bytes --]

On Sun, Mar 23, 2014 at 09:20:00AM -0700, Marc MERLIN wrote:
> Both
> legolas:/mnt/btrfs_pool2# btrfs balance start -v -dusage=5 /mnt/btrfs_pool2
> legolas:/mnt/btrfs_pool2# btrfs balance start -v -dusage=0 /mnt/btrfs_pool2
> failed unfortunately.
> 
> On Sun, Mar 23, 2014 at 12:26:32PM +0000, Duncan wrote:
> > When it rains, it pours.  What you're missing is that this is now the 
> > third thread in three days with exactly the same out-of-space-when-there-
> > appears-to-be-plenty problem, which is well explained and a solution 
> > presented, along with further discussion, on those threads.
> > 
> > Evidently you haven't read the others, but rather than rewrite a similar 
> > reply here with exactly the same explanation and fix, I'll just refer you 
> > to them.
> 
> Thanks. Indeed, while I spent most of yesterday dealing with 3 btrfs
> filesystems, the one here that was hanging my laptop, the raid5 one that
> was hanging repeatedly during balance, and then my main server were one
> FS is so slow that it takes 8H to do an reflink copy or delete a backup
> with 1 million inodes, I got behind on reading the list :)
> 
> Thanks for the pointers
> 
> > btrfs balance start -dusage=5 `pwd`
> > 
> > Tweak the N in usage=N as needed.
> 
> I had actually tried this, but it failed too:
> legolas:/mnt/btrfs_pool2# btrfs balance start -v -dusage=5 /mnt/btrfs_pool2
> Dumping filters: flags 0x1, state 0x0, force is off
>   DATA (flags 0x2): balancing, usage=5
> ERROR: error during balancing '/mnt/btrfs_pool2' - No space left on device
> 
> But I now just found
> https://btrfs.wiki.kernel.org/index.php/Balance_Filters
> and tried -dusage=0
>  
> On Sun, Mar 23, 2014 at 11:47:12AM +0000, Hugo Mills wrote:
> >    I think you probably shouldn't be doing a full balance, but a
> > filtered one:
> > 
> > # btrfs balance start -dusage=5 /mnt/btrfs_pool
> > 
> > which should only try to clean up chunks which have little usage (so
> > it's much faster to run).
> 
> Thanks for the other answer Hugo.
> 
> So, now I'm down to 
> legolas:/mnt/btrfs_pool2# btrfs balance start -v -dusage=0 /mnt/btrfs_pool2
> Dumping filters: flags 0x1, state 0x0, force is off
>   DATA (flags 0x2): balancing, usage=0
> ERROR: error during balancing '/mnt/btrfs_pool2' - No space left on device
> 
> Looks like there is no good way out of this, so I'll start deleting
> snapshots.

   Before you do this, can you take a btrfs-image of your metadata,
and add a report to bugzilla.kernel.org? You're not the only person
who's had this problem recently, and I suspect there's something
still lurking in there that needs attention.

> Hopefully this will be handled better in later code.

   With the info you can provide from a btrfs-image... let's hope so. :)

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
    --- Anyone using a computer to generate random numbers is, of ---    
                       course,  in a state of sin.                       

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: ERROR: error during balancing '.' - No space left on device

[Marc MERLIN]

On Sun, Mar 23, 2014 at 04:28:25PM +0000, Hugo Mills wrote:
>    Before you do this, can you take a btrfs-image of your metadata,
> and add a report to bugzilla.kernel.org? You're not the only person
> who's had this problem recently, and I suspect there's something
> still lurking in there that needs attention.
> 
> > Hopefully this will be handled better in later code.
> 
>    With the info you can provide from a btrfs-image... let's hope so. :)

Mmmh, this may not be good :-/

legolas:/mnt/btrfs_pool2# btrfs-image -c 9 -t 6 /dev/mapper/disk2 /tmp/pool2.image
parent transid verify failed on 295965446144 wanted 51493 found 51495
parent transid verify failed on 295965446144 wanted 51493 found 51495
parent transid verify failed on 295965446144 wanted 51493 found 51495
parent transid verify failed on 295965446144 wanted 51493 found 51495
Ignoring transid failure
leaf parent key incorrect 295965446144
parent transid verify failed on 106205184 wanted 51468 found 51528
parent transid verify failed on 106205184 wanted 51468 found 51528
parent transid verify failed on 106205184 wanted 51468 found 51528
parent transid verify failed on 106205184 wanted 51468 found 51528
Ignoring transid failure
parent transid verify failed on 267162120192 wanted 51469 found 51528
parent transid verify failed on 267162120192 wanted 51469 found 51528
parent transid verify failed on 267162120192 wanted 51469 found 51528
parent transid verify failed on 267162120192 wanted 51469 found 51528
Ignoring transid failure
parent transid verify failed on 107315200 wanted 51468 found 51528
parent transid verify failed on 107315200 wanted 51468 found 51528
parent transid verify failed on 107315200 wanted 51468 found 51528
parent transid verify failed on 107315200 wanted 51468 found 51528
Ignoring transid failure
parent transid verify failed on 108081152 wanted 51468 found 51528
parent transid verify failed on 108081152 wanted 51468 found 51528
parent transid verify failed on 108081152 wanted 51468 found 51528
parent transid verify failed on 108081152 wanted 51468 found 51528
Ignoring transid failure
parent transid verify failed on 267162124288 wanted 51469 found 51528
parent transid verify failed on 267162124288 wanted 51469 found 51528
parent transid verify failed on 267162124288 wanted 51469 found 51528
parent transid verify failed on 267162124288 wanted 51469 found 51528
Ignoring transid failure
parent transid verify failed on 108351488 wanted 51468 found 51528
parent transid verify failed on 108351488 wanted 51468 found 51528
parent transid verify failed on 108351488 wanted 51468 found 51528
parent transid verify failed on 108351488 wanted 51468 found 51528
Ignoring transid failure
parent transid verify failed on 275021406208 wanted 51469 found 51529
parent transid verify failed on 275021406208 wanted 51469 found 51529
parent transid verify failed on 275021406208 wanted 51469 found 51529
parent transid verify failed on 275021406208 wanted 51469 found 51529
Ignoring transid failure
parent transid verify failed on 108466176 wanted 51468 found 51528
parent transid verify failed on 108466176 wanted 51468 found 51528
parent transid verify failed on 108466176 wanted 51468 found 51528
parent transid verify failed on 108466176 wanted 51468 found 51528
Ignoring transid failure
Segmentation fault

Is it a bug in the tool, or do I have real corruption?

Are there magic options I can give it to make it work around this?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: ERROR: error during balancing '.' - No space left on device

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1844 bytes --]

On Sun, Mar 23, 2014 at 10:03:14AM -0700, Marc MERLIN wrote:
> On Sun, Mar 23, 2014 at 04:28:25PM +0000, Hugo Mills wrote:
> >    Before you do this, can you take a btrfs-image of your metadata,
> > and add a report to bugzilla.kernel.org? You're not the only person
> > who's had this problem recently, and I suspect there's something
> > still lurking in there that needs attention.
> > 
> > > Hopefully this will be handled better in later code.
> > 
> >    With the info you can provide from a btrfs-image... let's hope so. :)
> 
> Mmmh, this may not be good :-/
> 
> legolas:/mnt/btrfs_pool2# btrfs-image -c 9 -t 6 /dev/mapper/disk2 /tmp/pool2.image
> parent transid verify failed on 295965446144 wanted 51493 found 51495
> parent transid verify failed on 295965446144 wanted 51493 found 51495
> parent transid verify failed on 295965446144 wanted 51493 found 51495
> parent transid verify failed on 295965446144 wanted 51493 found 51495
> Ignoring transid failure
> leaf parent key incorrect 295965446144
> parent transid verify failed on 106205184 wanted 51468 found 51528
> parent transid verify failed on 106205184 wanted 51468 found 51528
> parent transid verify failed on 106205184 wanted 51468 found 51528
> parent transid verify failed on 106205184 wanted 51468 found 51528
[snip]
> Segmentation fault
> 
> Is it a bug in the tool, or do I have real corruption?
> 
> Are there magic options I can give it to make it work around this?

   xaba on IRC has just pointed out that it looks like you're running
this on a mounted filesystem -- it needs to be unmounted for
btrfs-image to work reliably.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
         --- Unix: For controlling fungal diseases in crops. ---         

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: ERROR: error during balancing '.' - No space left on device

[Marc MERLIN]

[-- Attachment #1: Type: text/plain, Size: 818 bytes --]

On Sun, Mar 23, 2014 at 05:34:09PM +0000, Hugo Mills wrote:
>    xaba on IRC has just pointed out that it looks like you're running
> this on a mounted filesystem -- it needs to be unmounted for
> btrfs-image to work reliably.
 
Sorry, I didn't realize that, although it makes sense. btrfs-image really
should fail though:
https://bugzilla.kernel.org/show_bug.cgi?id=72791

I got the image, but it's 1.5GB (1TB filesystem), but kernel bugzilla times
out after 1% of upload (repeatedly)
Placeholder bug:
https://bugzilla.kernel.org/show_bug.cgi?id=72801

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 308 bytes --]

Re: ERROR: error during balancing '.' - No space left on device

[Marc MERLIN]

[-- Attachment #1: Type: text/plain, Size: 2533 bytes --]

On Sun, Mar 23, 2014 at 12:10:17PM -0700, Marc MERLIN wrote:
> On Sun, Mar 23, 2014 at 05:34:09PM +0000, Hugo Mills wrote:
> >    xaba on IRC has just pointed out that it looks like you're running
> > this on a mounted filesystem -- it needs to be unmounted for
> > btrfs-image to work reliably.
>  
> Sorry, I didn't realize that, although it makes sense. btrfs-image really
> should fail though:
> https://bugzilla.kernel.org/show_bug.cgi?id=72791
> 
> I got the image, but it's 1.5GB (1TB filesystem), but kernel bugzilla times
> out after 1% of upload (repeatedly)
> Placeholder bug:
> https://bugzilla.kernel.org/show_bug.cgi?id=72801

Here is the image:
http://marc.merlins.org/tmp/pool2.image

After taking that image for you, I got the filesysetm to recover:
legolas:/mnt/btrfs_pool2# for i in *weekly*; do btrfs subvolume delete $i;
done
Delete subvolume '/mnt/btrfs_pool2/home_weekly_20140309_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/home_weekly_20140316_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/root_weekly_20140309_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/root_weekly_20140316_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/tmp_weekly_20140309_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/tmp_weekly_20140316_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/usr_weekly_20140309_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/usr_weekly_20140316_00:06:01'
Delete subvolume '/mnt/btrfs_pool2/var_weekly_20140309_00:06:01'
^A Delete subvolume '/mnt/btrfs_pool2/var_weekly_20140316_00:06:01'

legolas:/mnt/btrfs_pool2# btrfs fi df .
Data, single: total=800.42GiB, used=636.91GiB
System, DUP: total=8.00MiB, used=92.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=10.00GiB, used=9.50GiB
Metadata, single: total=8.00MiB, used=0.00
legolas:/mnt/btrfs_pool2# btrfs balance start -v -dusage=0 /mnt/btrfs_pool2

Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=0
  Done, had to relocate 91 out of 823 chunks

legolas:/mnt/btrfs_pool2# btrfs fi df .
Data, single: total=709.01GiB, used=603.85GiB
System, DUP: total=8.00MiB, used=88.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=10.00GiB, used=7.39GiB
Metadata, single: total=8.00MiB, used=0.00

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 308 bytes --]