Re: No space left on device

[Hugo Mills <hugo@carfax.org.uk>]

[-- Attachment #1: Type: text/plain, Size: 3295 bytes --]

On Wed, Feb 12, 2014 at 10:51:12AM +0100, Jakob Truelsen wrote:
> Hello. I am experiencing "No space left on device" with a btrfs file
> system, yet I cannot seem to find any exhausted resource. Could some
> resource I do not know about be exhausted, or is this caused by
> something else. Below is a trace of information that might be usefull,
> please let me know if there is anything I can do to resolve the issue.
> 
> /Jakob
> 
> [jakobt@soda ~]$ uname -a
> Linux soda 3.12.8-1-ARCH #1 SMP PREEMPT Thu Jan 16 09:16:34 CET 2014
> x86_64 GNU/Linux

   Were you using this kernel when the problem happened?

> [jakobt@soda ~]$ btrfs --version
> Btrfs v3.12
> 
> [jakobt@soda ~]$ mount
> ...
> /dev/sda on /data type btrfs (rw,relatime,nospace_cache)
> 
> [jakobt@soda ~]$ df /data
> Filesystem     1K-blocks     Used Available Use% Mounted on
> /dev/sdb2       76594224 49247368  23433028  68% /
> 
> [jakobt@soda ~]$ btrfs filesystem df /data
> Data, single: total=1.82TiB, used=518.04GiB
> System, DUP: total=8.00MiB, used=204.00KiB
> System, single: total=4.00MiB, used=0.00
> Metadata, DUP: total=1.00GiB, used=767.70MiB

   ^^^ This is your problem, most likely in conjunction with all the
space on the device being allocated. Being a copy-on-write filesystem,
btrfs needs free space to make any update. If it doesn't have that
free space, you get "No space left on device". You typically need
somewhere around 0.5-1 GiB of headroom in metadata for normal
operation, so I'm surprised you got this far. :)

   The FS should normally allocate more metadata space as it needs it,
but because (I think) your data allocation has taken up all the
available space on the device, there's no way for it to add more.

> Metadata, single: total=8.00MiB, used=0.00

> [jakobt@soda ~]$ touch /data/jakobt/hat
> touch: cannot touch ‘/data/jakobt/hat’: No space left on device
> 
> [jakobt@soda ~]$ sudo btrfs fi balance start /data
> ERROR: error during balancing '/data' - No space left on device
> There may be more info in syslog - try dmesg | tail

   Try:

btrfs balance start -dusage=0 /data

which should go looking for entirely unused block groups and reclaim
those. (If you don't use the -dusage= parameter, it will try to
balance everything, which takes a long time).

> [jakobt@soda ~]$ dmesg | grep tail -n 2
> [1113177.878157] btrfs: device label Data devid 1 transid 44784 /dev/sda
> [1113507.641752] btrfs: 1866 enospc errors during balance

   Although, that said... it looks like it's tried every block group
and failed with each one, so my suggestion above may not work in this
instance.

   Let us know what happens with the balance command above anyway
(dmesg output is useful information at this point). If that doesn't
help, then we'll probably need to take a metadata image and throw it
in josef's direction, where he will start crying at having to deal
with enospc problems again. :)

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
         --- If you're not part of the solution, you're part ---         
                           of the precipiate.                            

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]