Re: No space left on device

[Jakob Truelsen <antialize@gmail.com>]

Hi and thanks for the quick reply. Have remounted the filesystem with
enospc_debug, and run the rebalance you suggested, with the trance
below. So perhaps the next step is for me to figure out how to take a
metadata image and send it to josef (perhaps with a box of tissues)

/Jakob


[jakobt@soda ~]$ mount
...
/dev/sda on /data type btrfs (rw,relatime,nospace_cache,enospc_debug)

[jakobt@soda ~]$ sudo  btrfs balance start -dusage=0 /data
ERROR: error during balancing '/data' - No space left on device
There may be more info in syslog - try dmesg | tail

[jakobt@soda ~]$ touch /data/jakobt/monkey
touch: cannot touch ‘/data/jakobt/monkey’: No space left on device

[jakobt@soda ~]$ dmesg | tail -n3
[1117530.870965] btrfs: device label Data devid 1 transid 47091 /dev/sda
[1117573.087580] btrfs: 426 enospc errors during balance
[1117642.002437] btrfs: 426 enospc errors during balance


On Wed, Feb 12, 2014 at 11:26 AM, Hugo Mills <hugo@carfax.org.uk> wrote:
> On Wed, Feb 12, 2014 at 10:51:12AM +0100, Jakob Truelsen wrote:
>> Hello. I am experiencing "No space left on device" with a btrfs file
>> system, yet I cannot seem to find any exhausted resource. Could some
>> resource I do not know about be exhausted, or is this caused by
>> something else. Below is a trace of information that might be usefull,
>> please let me know if there is anything I can do to resolve the issue.
>>
>> /Jakob
>>
>> [jakobt@soda ~]$ uname -a
>> Linux soda 3.12.8-1-ARCH #1 SMP PREEMPT Thu Jan 16 09:16:34 CET 2014
>> x86_64 GNU/Linux
>
>    Were you using this kernel when the problem happened?
>
>> [jakobt@soda ~]$ btrfs --version
>> Btrfs v3.12
>>
>> [jakobt@soda ~]$ mount
>> ...
>> /dev/sda on /data type btrfs (rw,relatime,nospace_cache)
>>
>> [jakobt@soda ~]$ df /data
>> Filesystem     1K-blocks     Used Available Use% Mounted on
>> /dev/sdb2       76594224 49247368  23433028  68% /
>>
>> [jakobt@soda ~]$ btrfs filesystem df /data
>> Data, single: total=1.82TiB, used=518.04GiB
>> System, DUP: total=8.00MiB, used=204.00KiB
>> System, single: total=4.00MiB, used=0.00
>> Metadata, DUP: total=1.00GiB, used=767.70MiB
>
>    ^^^ This is your problem, most likely in conjunction with all the
> space on the device being allocated. Being a copy-on-write filesystem,
> btrfs needs free space to make any update. If it doesn't have that
> free space, you get "No space left on device". You typically need
> somewhere around 0.5-1 GiB of headroom in metadata for normal
> operation, so I'm surprised you got this far. :)
>
>    The FS should normally allocate more metadata space as it needs it,
> but because (I think) your data allocation has taken up all the
> available space on the device, there's no way for it to add more.
>
>> Metadata, single: total=8.00MiB, used=0.00
>
>> [jakobt@soda ~]$ touch /data/jakobt/hat
>> touch: cannot touch ‘/data/jakobt/hat’: No space left on device
>>
>> [jakobt@soda ~]$ sudo btrfs fi balance start /data
>> ERROR: error during balancing '/data' - No space left on device
>> There may be more info in syslog - try dmesg | tail
>
>    Try:
>
> btrfs balance start -dusage=0 /data
>
> which should go looking for entirely unused block groups and reclaim
> those. (If you don't use the -dusage= parameter, it will try to
> balance everything, which takes a long time).
>
>> [jakobt@soda ~]$ dmesg | grep tail -n 2
>> [1113177.878157] btrfs: device label Data devid 1 transid 44784 /dev/sda
>> [1113507.641752] btrfs: 1866 enospc errors during balance
>
>    Although, that said... it looks like it's tried every block group
> and failed with each one, so my suggestion above may not work in this
> instance.
>
>    Let us know what happens with the balance command above anyway
> (dmesg output is useful information at this point). If that doesn't
> help, then we'll probably need to take a metadata image and throw it
> in josef's direction, where he will start crying at having to deal
> with enospc problems again. :)
>
>    Hugo.
>
> --
> === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
>   PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
>          --- If you're not part of the solution, you're part ---
>                            of the precipiate.