ENOSPC while creating snapshot

ENOSPC while creating snapshot

[nexus]

Hello,

I'm encountering weird enospc problem while writing to filesystem and 
creating snapshot at a same time:

Steps to reproduce:

start writing data to filesystem:

# LC_ALL=C dd if=/dev/zero of=/foobar
dd: writing to '/foobar': No space left on device
3003803+0 records in
3003802+0 records out
1537946624 bytes (1.5 GB, 1.4 GiB) copied, 10.8072 s, 142 MB/s

in other terminal create read-only (or r/w - results are the same)
# btrfs sub snap -r / /.snap1
Create a readonly snapshot of '/' in '//.snap1'

while creating this snapshot, ENOSPC is produced.

Interesting is that I can reproduce this on some filesystems but not on 
others while all of available information are the same. There IS free 
space in data, metadata and global reserve.

I've tried balancing and defragmenting with no success. I've tried 
debugging by using enospc_debug but nothing is written to dmesg or any 
system log

Does anybody have a clue what may be the cause?

# uname -a
Linux interceptor 4.4.3-1-ARCH #1 SMP PREEMPT Fri Feb 26 15:09:29 CET 
2016 x86_64 GNU/Linux

# btrfs --version
btrfs-progs v4.4.1

# btrfs fi show
Label: none  uuid: 859dec5c-850c-4660-ad99-bc87456aa309
     Total devices 1 FS bytes used 116.72GiB
     devid    1 size 200.00GiB used 121.03GiB path 
/dev/mapper/vg0-btrfsroot

# btrfs fi df /
Data, single: total=117.00GiB, used=113.68GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=4.00GiB, used=3.04GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

# dmesg | tail
[   15.618995] cfg80211:  DFS Master region: ETSI
[   15.618996] cfg80211:   (start_freq - end_freq @ bandwidth), 
(max_antenna_gain, max_eirp), (dfs_cac_time)
[   15.618998] cfg80211:   (2400000 KHz - 2483500 KHz @ 40000 KHz), 
(N/A, 2000 mBm), (N/A)
[   15.619000] cfg80211:   (5150000 KHz - 5250000 KHz @ 80000 KHz, 
200000 KHz AUTO), (N/A, 2301 mBm), (N/A)
[   15.619001] cfg80211:   (5250000 KHz - 5350000 KHz @ 80000 KHz, 
200000 KHz AUTO), (N/A, 2000 mBm), (0 s)
[   15.619002] cfg80211:   (5470000 KHz - 5725000 KHz @ 160000 KHz), 
(N/A, 2698 mBm), (0 s)
[   15.619003] cfg80211:   (57000000 KHz - 66000000 KHz @ 2160000 KHz), 
(N/A, 4000 mBm), (N/A)
[   15.792765] brcmsmac bcma0:1: brcms_ops_bss_info_changed: arp 
filtering: 1 addresses (implement)
[   20.411931] tun: Universal TUN/TAP device driver, 1.6
[   20.411938] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>

# mount | grep root
/dev/mapper/vg0-btrfsroot on / type btrfs 
(rw,noatime,nodatasum,nodatacow,ssd,discard,space_cache,enospc_debug,commit=900,subvolid=5,subvol=/)

Thanks for any insight!

Best regards,

Martin Mlynar

Re: ENOSPC while creating snapshot

[Chris Murphy]

On Fri, Mar 4, 2016 at 2:10 PM,  <nexus@smoula.net> wrote:
> Hello,
>
> I'm encountering weird enospc problem while writing to filesystem and
> creating snapshot at a same time:
>
> Steps to reproduce:
>
> start writing data to filesystem:
>
> # LC_ALL=C dd if=/dev/zero of=/foobar
> dd: writing to '/foobar': No space left on device
> 3003803+0 records in
> 3003802+0 records out
> 1537946624 bytes (1.5 GB, 1.4 GiB) copied, 10.8072 s, 142 MB/s
>
> in other terminal create read-only (or r/w - results are the same)
> # btrfs sub snap -r / /.snap1
> Create a readonly snapshot of '/' in '//.snap1'
>
> while creating this snapshot, ENOSPC is produced.


Known issue. There's a thread already about it somewhere around here
in the last 4-6 weeks.


(rw,noatime,nodatasum,nodatacow,ssd,discard,space_cache,enospc_debug,commit=900,subvolid=5,subvol=/)

Most likely unrelated but commit time of 15 minutes? Umm, OK why?


-- 
Chris Murphy

Re: ENOSPC while creating snapshot

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1728 bytes --]

On Fri, 04 Mar 2016 22:10:57 +0100
nexus@smoula.net wrote:

> Hello,
> 
> I'm encountering weird enospc problem while writing to filesystem and 
> creating snapshot at a same time:
> 
> Steps to reproduce:
> 
> start writing data to filesystem:
> 
> # LC_ALL=C dd if=/dev/zero of=/foobar
> dd: writing to '/foobar': No space left on device
> 3003803+0 records in
> 3003802+0 records out
> 1537946624 bytes (1.5 GB, 1.4 GiB) copied, 10.8072 s, 142 MB/s
> 
> in other terminal create read-only (or r/w - results are the same)
> # btrfs sub snap -r / /.snap1
> Create a readonly snapshot of '/' in '//.snap1'
> 
> while creating this snapshot, ENOSPC is produced.
> 
> Interesting is that I can reproduce this on some filesystems but not on 
> others while all of available information are the same. There IS free 
> space in data, metadata and global reserve.
> 
> I've tried balancing and defragmenting with no success. I've tried 
> debugging by using enospc_debug but nothing is written to dmesg or any 
> system log
> 
> Does anybody have a clue what may be the cause?
> 
> # uname -a
> Linux interceptor 4.4.3-1-ARCH #1 SMP PREEMPT Fri Feb 26 15:09:29 CET 
> 2016 x86_64 GNU/Linux

As you use the nodatacow mount option, this seems to be another case of
http://www.spinics.net/lists/linux-btrfs/msg51276.html
http://www.spinics.net/lists/linux-btrfs/msg51819.html

and is fixed by 
https://patchwork.kernel.org/patch/7967161/

Unfortunately the bug is known since the start of the 4.4 series and the patch
is out for 2 months, but it didn't get included into even 4.4.4 released
recently. You have to apply it by yourself and recompile the kernel.

-- 
With respect,
Roman

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: ENOSPC while creating snapshot

[Duncan]

Roman Mamedov posted on Sat, 05 Mar 2016 03:49:10 +0500 as excerpted:

> As you use the nodatacow mount option, this seems to be another case of
> http://www.spinics.net/lists/linux-btrfs/msg51276.html
> http://www.spinics.net/lists/linux-btrfs/msg51819.html
> 
> and is fixed by https://patchwork.kernel.org/patch/7967161/
> 
> Unfortunately the bug is known since the start of the 4.4 series and the
> patch is out for 2 months, but it didn't get included into even 4.4.4
> released recently. You have to apply it by yourself and recompile the
> kernel.

Tho AFAIK it should be in 4.5, which is getting close to release, so if 
you prefer to run 4.5-rcX to applying the patch yourself, that should 
work as well.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: ENOSPC while creating snapshot

[Chris Murphy]

On Fri, Mar 4, 2016 at 4:16 PM, Martin Mlynář <nexus@smoula.net> wrote:

(rw,noatime,nodatasum,nodatacow,ssd,discard,space_cache,enospc_debug,commit=900,subvolid=5,subvol=/)
>>
>> Most likely unrelated but commit time of 15 minutes? Umm, OK why?
>
>
> I'm trying to reduce writes to my ssd.

This will not reduce writes. It will only delay them. And it increases
the chance of data loss by a lot.


-- 
Chris Murphy

Re: ENOSPC while creating snapshot

[Duncan]

Chris Murphy posted on Fri, 04 Mar 2016 19:46:34 -0700 as excerpted:

> On Fri, Mar 4, 2016 at 4:16 PM, Martin Mlynář <nexus@smoula.net> wrote:

[Mount options line split/wrapped for followup]

>>>> rw,noatime,nodatasum,nodatacow,ssd,discard,space_cache,
>>>> enospc_debug,commit=900,subvolid=5,subvol=/
>>>
>>> Most likely unrelated but commit time of 15 minutes? Umm, OK why?
>>
>> I'm trying to reduce writes to my ssd.
> 
> This will not reduce writes. It will only delay them. And it increases
> the chance of data loss by a lot.

AFAIK, to the extent that temporary files are created and then deleted 
again, within that 900 seconds aka 15 minutes, it will indeed reduce 
writes.

This can be the case for the build-tmp location for people running build-
from-sources distros such as gentoo, for instance, as many packages will 
be built and tmp-installed to that build-tmp, before being quick-merged 
to the live system and the work deleted from build-tmp in well under 15 
minutes, at least on today's reasonably powerful quad-core-plus systems. 
Tho on gentoo, the recommendation for those with the memory available is 
to point that build-tmp at a tmpfs instead of a permanent-storage backed 
filesystem of any sort.

And in general, for those without the memory to support build-tmp in 
tmpfs, a 15-minute commit time isn't going to help either, because if 
they have enough memory to avoid flushing to free up memory for that full 
15 minutes, they obviously have enough memory to store all those files 
that would be in tmpfs if they'd have simply pointed build-tmp at that, 
instead.

Another use-case is laptops and other mobile systems with enough memory 
to cache the normal working set, is to power down the storage devices for 
as long as possible between powerups.  However, the heavy power usage 
there is normally on spinning up the disk and/or keeping it spinning, and 
SSDs obviously aren't subject to that.  While some small amount of power 
may still be saved by powering down the SSD, I expect it to be pretty 
small, and the writes are going to take the same amount of power no 
matter when they're done.

In either case, 15 minute commit times are rather extreme, because as has 
been pointed out, that's potentially 15 minutes of lost work should the 
system crash before those writes are completed, and losing 15 minutes 
worth of work is well beyond the acceptable risk level for most people.  
5 minutes, much more common, 10 minutes, not so common but you'll fine 
people doing it.  15 minutes, pretty rare, I expect.

The point being, yes, there are use-cases where 15 minute commit times 
makes sense.  But the given reason, a bare wish to reduce writes to the 
ssd, without support such as one of the above use-cases or something 
similar, really doesn't make sense, at least on its face.  I'll agree 
with other posters on that.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: ENOSPC while creating snapshot

[Martin Mlynář]

On 5.3.2016 06:34, Duncan wrote:
> Chris Murphy posted on Fri, 04 Mar 2016 19:46:34 -0700 as excerpted:
>
>> On Fri, Mar 4, 2016 at 4:16 PM, Martin Mlynář <nexus@smoula.net> wrote:
>
> [Mount options line split/wrapped for followup]
>
>>>>> rw,noatime,nodatasum,nodatacow,ssd,discard,space_cache,
>>>>> enospc_debug,commit=900,subvolid=5,subvol=/
>>>>
>>>> Most likely unrelated but commit time of 15 minutes? Umm, OK why?
>>>
>>> I'm trying to reduce writes to my ssd.
>>
>> This will not reduce writes. It will only delay them. And it increases
>> the chance of data loss by a lot.
>
> AFAIK, to the extent that temporary files are created and then deleted
> again, within that 900 seconds aka 15 minutes, it will indeed reduce
> writes.
>
> This can be the case for the build-tmp location for people running build-
> from-sources distros such as gentoo, for instance, as many packages will
> be built and tmp-installed to that build-tmp, before being quick-merged
> to the live system and the work deleted from build-tmp in well under 15
> minutes, at least on today's reasonably powerful quad-core-plus systems.
> Tho on gentoo, the recommendation for those with the memory available is
> to point that build-tmp at a tmpfs instead of a permanent-storage backed
> filesystem of any sort.
>
> And in general, for those without the memory to support build-tmp in
> tmpfs, a 15-minute commit time isn't going to help either, because if
> they have enough memory to avoid flushing to free up memory for that full
> 15 minutes, they obviously have enough memory to store all those files
> that would be in tmpfs if they'd have simply pointed build-tmp at that,
> instead.
>
> Another use-case is laptops and other mobile systems with enough memory
> to cache the normal working set, is to power down the storage devices for
> as long as possible between powerups.  However, the heavy power usage
> there is normally on spinning up the disk and/or keeping it spinning, and
> SSDs obviously aren't subject to that.  While some small amount of power
> may still be saved by powering down the SSD, I expect it to be pretty
> small, and the writes are going to take the same amount of power no
> matter when they're done.
>
> In either case, 15 minute commit times are rather extreme, because as has
> been pointed out, that's potentially 15 minutes of lost work should the
> system crash before those writes are completed, and losing 15 minutes
> worth of work is well beyond the acceptable risk level for most people.
> 5 minutes, much more common, 10 minutes, not so common but you'll fine
> people doing it.  15 minutes, pretty rare, I expect.
>
> The point being, yes, there are use-cases where 15 minute commit times
> makes sense.  But the given reason, a bare wish to reduce writes to the
> ssd, without support such as one of the above use-cases or something
> similar, really doesn't make sense, at least on its face.  I'll agree
> with other posters on that.
Thank you for valuable insight, all of you.

It is battery backed-up laptop so I thought it should work well. I've 
met no problems since when I've set up this few years ago. To be hones I 
even forgot I've got this set up :)

I'll lower the value, you're right, that 15 minutes are pretty extreme.

--
Martin

Re: ENOSPC while creating snapshot

[Filipe Manana]

On Sat, Mar 5, 2016 at 2:09 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> Roman Mamedov posted on Sat, 05 Mar 2016 03:49:10 +0500 as excerpted:
>
>> As you use the nodatacow mount option, this seems to be another case of
>> http://www.spinics.net/lists/linux-btrfs/msg51276.html
>> http://www.spinics.net/lists/linux-btrfs/msg51819.html
>>
>> and is fixed by https://patchwork.kernel.org/patch/7967161/
>>
>> Unfortunately the bug is known since the start of the 4.4 series and the
>> patch is out for 2 months, but it didn't get included into even 4.4.4
>> released recently. You have to apply it by yourself and recompile the
>> kernel.
>
> Tho AFAIK it should be in 4.5, which is getting close to release, so if
> you prefer to run 4.5-rcX to applying the patch yourself, that should
> work as well.

No, it's not in any 4.5-rc.
It's in the integration branch for 4.6, however.

>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Re: ENOSPC while creating snapshot

[Martin Mlynář]

On 5.3.2016 11:46, Filipe Manana wrote:
> On Sat, Mar 5, 2016 at 2:09 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>> Roman Mamedov posted on Sat, 05 Mar 2016 03:49:10 +0500 as excerpted:
>>
>>> As you use the nodatacow mount option, this seems to be another case of
>>> http://www.spinics.net/lists/linux-btrfs/msg51276.html
>>> http://www.spinics.net/lists/linux-btrfs/msg51819.html
>>>
>>> and is fixed by https://patchwork.kernel.org/patch/7967161/
>>>
>>> Unfortunately the bug is known since the start of the 4.4 series and the
>>> patch is out for 2 months, but it didn't get included into even 4.4.4
>>> released recently. You have to apply it by yourself and recompile the
>>> kernel.
>>
>> Tho AFAIK it should be in 4.5, which is getting close to release, so if
>> you prefer to run 4.5-rcX to applying the patch yourself, that should
>> work as well.
>
> No, it's not in any 4.5-rc.
> It's in the integration branch for 4.6, however.
Thank you all for your help. Now I'm on 4.5.0-rc6-mainline with pointed 
patch and issue seems to be resolved.

Thank you for your time!

--
Martin