Issue: errno:28 (No space left on device)

Issue: errno:28 (No space left on device)

[Thomas]

Hello!

I'm facing a severe issue with Debian installation using BTRFS:
errno:28 (No space left on device)

After a few minutes of system usage I get this error message for any action, means there's really no disk space left.

However, the output of btrfs fi usage / reports that ~10% of disk space should be free.

I have 2 questions:
1. Why is the OS reporting an error "Issue: errno:28 (No space left on device)" whereas "btrfs fi usage" reports free space?
2. How can I solve this issue w/o the possibility of extending the partition?


THX

---------------------------------------------------------------------------------------

thomas@pc8-nb:~$ uname -a
Linux pc8-nb 4.6.1-towo.1-siduction-amd64 #1 SMP PREEMPT siduction 4.6-1 (2016-06-04) x86_64 GNU/Linux


thomas@pc8-nb:~$ sudo btrfs --version
btrfs-progs v4.5.2


thomas@pc8-nb:~$ sudo btrfs fi usage /
Overall:
     Device size:		  20.53GiB
     Device allocated:		  20.53GiB
     Device unallocated:		     0.00B
     Device missing:		     0.00B
     Used:			  18.25GiB
     Free (estimated):		   1.96GiB	(min: 1.96GiB)
     Data ratio:			      1.00
     Metadata ratio:		      1.00
     Global reserve:		 160.00MiB	(used: 400.00KiB)

Data,single: Size:19.76GiB, Used:17.81GiB
    /dev/sda8	  19.76GiB

Metadata,single: Size:776.00MiB, Used:455.73MiB
    /dev/sda8	 776.00MiB

System,single: Size:4.00MiB, Used:16.00KiB
    /dev/sda8	   4.00MiB

Unallocated:
    /dev/sda8	     0.00B


thomas@pc8-nb:~$ sudo btrfs fi show
Label: 'siduction'  uuid: dbb9433e-a84f-4e84-9200-49350b31a3f6
	Total devices 1 FS bytes used 18.25GiB
	devid    1 size 20.53GiB used 20.53GiB path /dev/sda8


thomas@pc8-nb:~$ sudo btrfs su li /
ID 257 gen 5498245 top level 5 path @
ID 258 gen 5498245 top level 5 path @home
ID 259 gen 5488356 top level 5 path @tmp
ID 260 gen 2029641 top level 5 path @var_lib_lxc
ID 261 gen 5480763 top level 5 path @var_log
ID 262 gen 5467204 top level 5 path @var_spool
ID 263 gen 5467205 top level 5 path @var_tmp
ID 290 gen 5464299 top level 257 path .snapshots
ID 291 gen 2026169 top level 258 path @home/.snapshots
ID 293 gen 2029641 top level 260 path @var_lib_lxc/.snapshots
ID 355 gen 163972 top level 290 path .snapshots/17/snapshot
ID 356 gen 167205 top level 290 path .snapshots/18/snapshot
ID 357 gen 5383634 top level 5 path @var_cache_apt_archives
ID 358 gen 5464163 top level 290 path .snapshots/19/snapshot


thomas@pc8-nb:~$ sudo journalctl | grep space
[...]
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError: [Errno 28] No space left on device
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError: [Errno 28] No space left on device
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError: [Errno 28] No space left on device
[...]
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://ftp.fau.de/debian/dists/stable/main/Contents-i386  Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://debian.mirror.lrz.de/debian/dists/stable/contrib/binary-i386/Packages.gz  Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://debian.mirror.lrz.de/debian/dists/stable/contrib/i18n/Translation-en  Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-amd64  Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-i386  Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-amd64/Packages.gz  Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-i386/Packages.gz  Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://debian.mirror.lrz.de/debian/dists/stable/non-free/i18n/Translation-en  Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://mirror.unitedcolo.de/debian/dists/stable/non-free/Contents-amd64  Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch http://ftp.fau.de/debian/dists/stable/non-free/Contents-i386  Write error - write (28: No space left on device)
[...]

Re: Issue: errno:28 (No space left on device)

[Chris Murphy]

On Tue, Aug 9, 2016 at 3:06 AM, Thomas <c.monty@web.de> wrote:

> thomas@pc8-nb:~$ uname -a
> Linux pc8-nb 4.6.1-towo.1-siduction-amd64 #1 SMP PREEMPT siduction 4.6-1
> (2016-06-04) x86_64 GNU/Linux

Try 4.5.7 or 4.7 and see if it fixes the problem, there's a related
patch in both of those that's not yet in 4.6 series. The patch mainly
permits using existing chunks for balance to consolidate extents,
rather than requiring new chunks. So you might also have to do a
filtered balance. I would probably start with -dusage=25 and see where
that gets you.


>     Device size:                  20.53GiB
>     Device allocated:             20.53GiB

>From Btrfs perspective the volume is "fully allocated" and as such
cannot create new data or metadata chunks. Whatever workload you're
doing at the time of enospc may be filling up data or metadata unused
space in existing chunks, triggering need for newly allocated chunk,
which can't happen because there's no space left for new chunks.


> Data,single: Size:19.76GiB, Used:17.81GiB
>    /dev/sda8      19.76GiB
>
> Metadata,single: Size:776.00MiB, Used:455.73MiB
>    /dev/sda8     776.00MiB

I'll guess there's some metadata centric workload, maybe with small
files that are being stored inline, that's filling up the metadata
chunks, leading to enospc.

If changing kernel versions doesn't work, then you'll need to add more
space somehow. The file system is ~90% full anyway.


-- 
Chris Murphy

Re: Issue: errno:28 (No space left on device)

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 896 bytes --]

On Tue, 9 Aug 2016 09:39:45 -0400
Noah Massey <noah.massey@gmail.com> wrote:

> Since your data and metadata profiles are both 'single', you may want
> to consider creating the filesystem with the --mixed option, which
> allows / forces BTRFS to use a single collection of data blocks for
> both data and metadata.

That also seems to be detrimental to performance, I had servers with mixed
profile FSes which went painfully slow after a couple of years (daily snapshot
creation and deletion, ~30-60 snapshots kept at all times), migrated them all
now to the regular split profile. Fresh off mkfs they are fast of course, so
not entirely certain "mixed" was the culprit, will see.

One more problem with mixed is that you can't use the leaf and nodesize of 16k,
which are accepted to be faster than the older default (and mixed's only
option) of 4k.

-- 
With respect,
Roman

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Issue: errno:28 (No space left on device)

[Josef Bacik]

On 08/09/2016 07:50 AM, Thomas wrote:
> Hello!
>
> I'm facing a severe issue with Debian installation using BTRFS:
> errno:28 (No space left on device)
>
> After a few minutes of system usage I get this error message for any action,
> means there's really no disk space left.
>
> However, the output of btrfs fi usage / reports that ~10% of disk space should
> be free.
>
> I have 2 questions:
> 1. Why is the OS reporting an error "Issue: errno:28 (No space left on device)"
> whereas "btrfs fi usage" reports free space?
> 2. How can I solve this issue w/o the possibility of extending the partition?
>

Please try linus's latest tree, it has some enospc accounting fixes that have 
recently gone in.  Thanks,

Josef

Re: Issue: errno:28 (No space left on device)

[Noah Massey]

On Tue, Aug 9, 2016 at 8:56 AM, Austin S. Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2016-08-09 07:50, Thomas wrote:
>>
>> Hello!
>
> First things first:
> Mailing lists are asynchronous.  You will almost _never_ get an immediate
> response, and will quite often not get a response for a few hours at least.
> Sending a message more than once when you don't get a response does not make
> it more likely you get a response quickly, and in fact is a good way to
> seriously annoy people and make them less willing to help you.
>>
>>
>> I'm facing a severe issue with Debian installation using BTRFS:
>> errno:28 (No space left on device)
>>
>> After a few minutes of system usage I get this error message for any
>> action, means there's really no disk space left.
>>
>> However, the output of btrfs fi usage / reports that ~10% of disk space
>> should be free.
>>
>> I have 2 questions:
>> 1. Why is the OS reporting an error "Issue: errno:28 (No space left on
>> device)" whereas "btrfs fi usage" reports free space?
>
> BTRFS allocates disk space in two stages.  In the first stage, it allocates
> large chunks that then get used only for metadata or data. Within those
> chunks, it then allocates blocks for the given type of data on-demand.  In
> your case, the filesystem has no room to allocate new chunks, but has some
> free space in existing chunks.  This shouldn't result in allocation failures
> for new files, but for currently unknown reasons it sometimes does, and this
> is a known issue that we have been unable to completely fix so far (we've
> technically 'fixed' this about 5 or 6 times already, but there appear to be
> lots of odd corner cases that haven't been found yet).
>>
>> 2. How can I solve this issue w/o the possibility of extending the
>> partition?
>
> Right now, you have three options:
> 1. Extend the partition, run a full balance, and then shrink the partition
> again.
> 2. Temporarily add another device (make sure it's at least 4GB), run a full
> balance, and then remove the device.
> 3. Recreate the filesystem from scratch and restore from a backup.
>

A quick note recreating the filesystem:
Since your data and metadata profiles are both 'single', you may want
to consider creating the filesystem with the --mixed option, which
allows / forces BTRFS to use a single collection of data blocks for
both data and metadata.
This can help with ENOMEM errors when all space is allocated, but the
allocation does not match the currently required type.

~ Noah

Re: Issue: errno:28 (No space left on device)

[Austin S. Hemmelgarn]

On 2016-08-09 07:50, Thomas wrote:
> Hello!
First things first:
Mailing lists are asynchronous.  You will almost _never_ get an 
immediate response, and will quite often not get a response for a few 
hours at least.  Sending a message more than once when you don't get a 
response does not make it more likely you get a response quickly, and in 
fact is a good way to seriously annoy people and make them less willing 
to help you.
>
> I'm facing a severe issue with Debian installation using BTRFS:
> errno:28 (No space left on device)
>
> After a few minutes of system usage I get this error message for any
> action, means there's really no disk space left.
>
> However, the output of btrfs fi usage / reports that ~10% of disk space
> should be free.
>
> I have 2 questions:
> 1. Why is the OS reporting an error "Issue: errno:28 (No space left on
> device)" whereas "btrfs fi usage" reports free space?
BTRFS allocates disk space in two stages.  In the first stage, it 
allocates large chunks that then get used only for metadata or data. 
Within those chunks, it then allocates blocks for the given type of data 
on-demand.  In your case, the filesystem has no room to allocate new 
chunks, but has some free space in existing chunks.  This shouldn't 
result in allocation failures for new files, but for currently unknown 
reasons it sometimes does, and this is a known issue that we have been 
unable to completely fix so far (we've technically 'fixed' this about 5 
or 6 times already, but there appear to be lots of odd corner cases that 
haven't been found yet).
> 2. How can I solve this issue w/o the possibility of extending the
> partition?
Right now, you have three options:
1. Extend the partition, run a full balance, and then shrink the 
partition again.
2. Temporarily add another device (make sure it's at least 4GB), run a 
full balance, and then remove the device.
3. Recreate the filesystem from scratch and restore from a backup.

Regardless of which option you choose, you either need to free up some 
disk space, or get a bigger storage device, as running within a few GB 
of a full filesystem is a known trigger for this issue.  Depending on 
how frequently you snapshot the system, just deleting some old snapshots 
may help.  Making sure to clean out the package cache after running 
updates will probably help too.
>
>
> THX
>
> ---------------------------------------------------------------------------------------
>
>
> thomas@pc8-nb:~$ uname -a
> Linux pc8-nb 4.6.1-towo.1-siduction-amd64 #1 SMP PREEMPT siduction 4.6-1
> (2016-06-04) x86_64 GNU/Linux
>
>
> thomas@pc8-nb:~$ sudo btrfs --version
> btrfs-progs v4.5.2
>
>
> thomas@pc8-nb:~$ sudo btrfs fi usage /
> Overall:
>       Device size:          20.53GiB
>       Device allocated:          20.53GiB
>       Device unallocated:             0.00B
>       Device missing:             0.00B
>       Used:              18.25GiB
>       Free (estimated):           1.96GiB    (min: 1.96GiB)
>       Data ratio:                  1.00
>       Metadata ratio:              1.00
>       Global reserve:         160.00MiB    (used: 400.00KiB)
>
> Data,single: Size:19.76GiB, Used:17.81GiB
>      /dev/sda8      19.76GiB
>
> Metadata,single: Size:776.00MiB, Used:455.73MiB
>      /dev/sda8     776.00MiB
>
> System,single: Size:4.00MiB, Used:16.00KiB
>      /dev/sda8       4.00MiB
>
> Unallocated:
>      /dev/sda8         0.00B
>
>
> thomas@pc8-nb:~$ sudo btrfs fi show
> Label: 'siduction'  uuid: dbb9433e-a84f-4e84-9200-49350b31a3f6
>     Total devices 1 FS bytes used 18.25GiB
>     devid    1 size 20.53GiB used 20.53GiB path /dev/sda8
>
>
> thomas@pc8-nb:~$ sudo btrfs su li /
> ID 257 gen 5498245 top level 5 path @
> ID 258 gen 5498245 top level 5 path @home
> ID 259 gen 5488356 top level 5 path @tmp
> ID 260 gen 2029641 top level 5 path @var_lib_lxc
> ID 261 gen 5480763 top level 5 path @var_log
> ID 262 gen 5467204 top level 5 path @var_spool
> ID 263 gen 5467205 top level 5 path @var_tmp
> ID 290 gen 5464299 top level 257 path .snapshots
> ID 291 gen 2026169 top level 258 path @home/.snapshots
> ID 293 gen 2029641 top level 260 path @var_lib_lxc/.snapshots
> ID 355 gen 163972 top level 290 path .snapshots/17/snapshot
> ID 356 gen 167205 top level 290 path .snapshots/18/snapshot
> ID 357 gen 5383634 top level 5 path @var_cache_apt_archives
> ID 358 gen 5464163 top level 290 path .snapshots/19/snapshot
>
>
> thomas@pc8-nb:~$ sudo journalctl | grep space
> [...]
> Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError:
> [Errno 28] No space left on device
> Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError:
> [Errno 28] No space left on device
> Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError:
> [Errno 28] No space left on device
> [...]
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://ftp.fau.de/debian/dists/stable/main/Contents-i386  Write error -
> write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://debian.mirror.lrz.de/debian/dists/stable/contrib/binary-i386/Packages.gz
>   Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://debian.mirror.lrz.de/debian/dists/stable/contrib/i18n/Translation-en
>   Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-amd64
> Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-i386
> Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-amd64/Packages.gz
>   Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-i386/Packages.gz
>   Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://debian.mirror.lrz.de/debian/dists/stable/non-free/i18n/Translation-en
>   Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://mirror.unitedcolo.de/debian/dists/stable/non-free/Contents-amd64
> Write error - write (28: No space left on device)
> Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch
> http://ftp.fau.de/debian/dists/stable/non-free/Contents-i386  Write
> error - write (28: No space left on device)
> [...]

Re: Issue: errno:28 (No space left on device)

[Tomasz Torcz]

On Tue, Aug 09, 2016 at 01:50:04PM +0200, Thomas wrote:
> Hello!
> 
> I'm facing a severe issue with Debian installation using BTRFS:
> errno:28 (No space left on device)

  Sending this message 3x won't make it more important.

-- 
Tomasz Torcz       ,,(...) today's high-end is tomorrow's embedded processor.''
xmpp: zdzichubg@chrome.pl                      -- Mitchell Blank on LKML

Issue: errno:28 (No space left on device)

[Thomas]

Hello!

I'm facing a severe issue with Debian installation using BTRFS:
errno:28 (No space left on device)

After a few minutes of system usage I get this error message for any 
action, means there's really no disk space left.

However, the output of btrfs fi usage / reports that ~10% of disk space 
should be free.

I have 2 questions:
1. Why is the OS reporting an error "Issue: errno:28 (No space left on 
device)" whereas "btrfs fi usage" reports free space?
2. How can I solve this issue w/o the possibility of extending the 
partition?


THX

---------------------------------------------------------------------------------------

thomas@pc8-nb:~$ uname -a
Linux pc8-nb 4.6.1-towo.1-siduction-amd64 #1 SMP PREEMPT siduction 4.6-1 
(2016-06-04) x86_64 GNU/Linux


thomas@pc8-nb:~$ sudo btrfs --version
btrfs-progs v4.5.2


thomas@pc8-nb:~$ sudo btrfs fi usage /
Overall:
       Device size:		  20.53GiB
       Device allocated:		  20.53GiB
       Device unallocated:		     0.00B
       Device missing:		     0.00B
       Used:			  18.25GiB
       Free (estimated):		   1.96GiB	(min: 1.96GiB)
       Data ratio:			      1.00
       Metadata ratio:		      1.00
       Global reserve:		 160.00MiB	(used: 400.00KiB)

Data,single: Size:19.76GiB, Used:17.81GiB
      /dev/sda8	  19.76GiB

Metadata,single: Size:776.00MiB, Used:455.73MiB
      /dev/sda8	 776.00MiB

System,single: Size:4.00MiB, Used:16.00KiB
      /dev/sda8	   4.00MiB

Unallocated:
      /dev/sda8	     0.00B


thomas@pc8-nb:~$ sudo btrfs fi show
Label: 'siduction'  uuid: dbb9433e-a84f-4e84-9200-49350b31a3f6
	Total devices 1 FS bytes used 18.25GiB
	devid    1 size 20.53GiB used 20.53GiB path /dev/sda8


thomas@pc8-nb:~$ sudo btrfs su li /
ID 257 gen 5498245 top level 5 path @
ID 258 gen 5498245 top level 5 path @home
ID 259 gen 5488356 top level 5 path @tmp
ID 260 gen 2029641 top level 5 path @var_lib_lxc
ID 261 gen 5480763 top level 5 path @var_log
ID 262 gen 5467204 top level 5 path @var_spool
ID 263 gen 5467205 top level 5 path @var_tmp
ID 290 gen 5464299 top level 257 path .snapshots
ID 291 gen 2026169 top level 258 path @home/.snapshots
ID 293 gen 2029641 top level 260 path @var_lib_lxc/.snapshots
ID 355 gen 163972 top level 290 path .snapshots/17/snapshot
ID 356 gen 167205 top level 290 path .snapshots/18/snapshot
ID 357 gen 5383634 top level 5 path @var_cache_apt_archives
ID 358 gen 5464163 top level 290 path .snapshots/19/snapshot


thomas@pc8-nb:~$ sudo journalctl | grep space
[...]
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError: 
[Errno 28] No space left on device
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError: 
[Errno 28] No space left on device
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError: 
[Errno 28] No space left on device
[...]
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://ftp.fau.de/debian/dists/stable/main/Contents-i386  Write error - 
write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://debian.mirror.lrz.de/debian/dists/stable/contrib/binary-i386/Packages.gz 
   Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://debian.mirror.lrz.de/debian/dists/stable/contrib/i18n/Translation-en 
   Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-amd64 
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-i386 
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-amd64/Packages.gz 
   Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-i386/Packages.gz 
   Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://debian.mirror.lrz.de/debian/dists/stable/non-free/i18n/Translation-en 
   Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://mirror.unitedcolo.de/debian/dists/stable/non-free/Contents-amd64 
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://ftp.fau.de/debian/dists/stable/non-free/Contents-i386  Write 
error - write (28: No space left on device)
[...]

Issue: errno:28 (No space left on device)

[Thomas]

Hello!

I'm facing a severe issue with Debian installation using BTRFS:
errno:28 (No space left on device)

After a few minutes of system usage I get this error message for any 
action, means there's really no disk space left.

However, the output of btrfs fi usage / reports that ~10% of disk space 
should be free.

I have 2 questions:
1. Why is the OS reporting an error "Issue: errno:28 (No space left on 
device)" whereas "btrfs fi usage" reports free space?
2. How can I solve this issue w/o the possibility of extending the 
partition?


THX

---------------------------------------------------------------------------------------

thomas@pc8-nb:~$ uname -a
Linux pc8-nb 4.6.1-towo.1-siduction-amd64 #1 SMP PREEMPT siduction 4.6-1 
(2016-06-04) x86_64 GNU/Linux


thomas@pc8-nb:~$ sudo btrfs --version
btrfs-progs v4.5.2


thomas@pc8-nb:~$ sudo btrfs fi usage /
Overall:
      Device size:		  20.53GiB
      Device allocated:		  20.53GiB
      Device unallocated:		     0.00B
      Device missing:		     0.00B
      Used:			  18.25GiB
      Free (estimated):		   1.96GiB	(min: 1.96GiB)
      Data ratio:			      1.00
      Metadata ratio:		      1.00
      Global reserve:		 160.00MiB	(used: 400.00KiB)

Data,single: Size:19.76GiB, Used:17.81GiB
     /dev/sda8	  19.76GiB

Metadata,single: Size:776.00MiB, Used:455.73MiB
     /dev/sda8	 776.00MiB

System,single: Size:4.00MiB, Used:16.00KiB
     /dev/sda8	   4.00MiB

Unallocated:
     /dev/sda8	     0.00B


thomas@pc8-nb:~$ sudo btrfs fi show
Label: 'siduction'  uuid: dbb9433e-a84f-4e84-9200-49350b31a3f6
	Total devices 1 FS bytes used 18.25GiB
	devid    1 size 20.53GiB used 20.53GiB path /dev/sda8


thomas@pc8-nb:~$ sudo btrfs su li /
ID 257 gen 5498245 top level 5 path @
ID 258 gen 5498245 top level 5 path @home
ID 259 gen 5488356 top level 5 path @tmp
ID 260 gen 2029641 top level 5 path @var_lib_lxc
ID 261 gen 5480763 top level 5 path @var_log
ID 262 gen 5467204 top level 5 path @var_spool
ID 263 gen 5467205 top level 5 path @var_tmp
ID 290 gen 5464299 top level 257 path .snapshots
ID 291 gen 2026169 top level 258 path @home/.snapshots
ID 293 gen 2029641 top level 260 path @var_lib_lxc/.snapshots
ID 355 gen 163972 top level 290 path .snapshots/17/snapshot
ID 356 gen 167205 top level 290 path .snapshots/18/snapshot
ID 357 gen 5383634 top level 5 path @var_cache_apt_archives
ID 358 gen 5464163 top level 290 path .snapshots/19/snapshot


thomas@pc8-nb:~$ sudo journalctl | grep space
[...]
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError: 
[Errno 28] No space left on device
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError: 
[Errno 28] No space left on device
Jun 16 17:52:39 pc8-nb unattended-upgrade-shutdown[13530]: OSError: 
[Errno 28] No space left on device
[...]
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://ftp.fau.de/debian/dists/stable/main/Contents-i386  Write error - 
write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://debian.mirror.lrz.de/debian/dists/stable/contrib/binary-i386/Packages.gz 
  Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://debian.mirror.lrz.de/debian/dists/stable/contrib/i18n/Translation-en 
  Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-amd64 
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://debian.mirror.lrz.de/debian/dists/stable/contrib/Contents-i386 
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-amd64/Packages.gz 
  Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://mirror.unitedcolo.de/debian/dists/stable/non-free/binary-i386/Packages.gz 
  Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://debian.mirror.lrz.de/debian/dists/stable/non-free/i18n/Translation-en 
  Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://mirror.unitedcolo.de/debian/dists/stable/non-free/Contents-amd64 
Write error - write (28: No space left on device)
Jul 17 04:43:02 pc8-nb cron-apt[2658]: E: Failed to fetch 
http://ftp.fau.de/debian/dists/stable/non-free/Contents-i386  Write 
error - write (28: No space left on device)
[...]