[4.14.3] btrfs out of space error

[4.14.3] btrfs out of space error

[Ian Kumlien]

Hi,

Running a 4.14.3 kernel, this just happened, but there should have
been another 20 gigs or so available.

The filesystem seems fine after a reboot though

[1070034.614893] ------------[ cut here ]------------
[1070034.614899] WARNING: CPU: 4 PID: 18634 at fs/btrfs/inode.c:4647
btrfs_truncate_inode_items+0xea5/0xfb0
[1070034.614899] Modules linked in: mlx4_en chaoskey amdgpu mfd_core mlx4_core
[1070034.614904] CPU: 4 PID: 18634 Comm: TaskSchedulerFo Not tainted 4.14.3 #165
[1070034.614905] Hardware name: To be filled by O.E.M. To be filled by
O.E.M./SABERTOOTH 990FX, BIOS 1604 10/16/2012
[1070034.614906] task: ffff8bb0f2332c00 task.stack: ffff9630064c4000
[1070034.614907] RIP: 0010:btrfs_truncate_inode_items+0xea5/0xfb0
[1070034.614907] RSP: 0018:ffff9630064c7cc0 EFLAGS: 00010282
[1070034.614909] RAX: 0000000000000026 RBX: ffff8bb11348a070 RCX:
0000000000000000
[1070034.614909] RDX: ffff8bb13ed147b0 RSI: ffff8bb13ed0c9d8 RDI:
ffff8bb13ed0c9d8
[1070034.614910] RBP: ffff8baf61905c70 R08: 0000000000000001 R09:
00000000000005f1
[1070034.614910] R10: 0000000000001000 R11: 0000000000000000 R12:
ffff8bb1138e0800
[1070034.614911] R13: 00000000000001e6 R14: ffff8baf60610ee0 R15:
00000000000000eb
[1070034.614912] FS:  00007fad68b7f700(0000) GS:ffff8bb13ed00000(0000)
knlGS:0000000000000000
[1070034.614912] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1070034.614913] CR2: 00001c7e020d0000 CR3: 0000000180a19000 CR4:
00000000000406e0
[1070034.614913] Call Trace:
[1070034.614918]  btrfs_truncate+0x11d/0x2c0
[1070034.614919]  btrfs_setattr+0x27b/0x3c0
[1070034.614921]  notify_change+0x29f/0x430
[1070034.614923]  do_truncate+0x5e/0x90
[1070034.614925]  do_sys_ftruncate.constprop.3+0x139/0x1a0
[1070034.614927]  entry_SYSCALL_64_fastpath+0x13/0x94
[1070034.614928] RIP: 0033:0x7fadd17c945e
[1070034.614929] RSP: 002b:00007fad68b7e870 EFLAGS: 00000246 ORIG_RAX:
000000000000004d
[1070034.614930] RAX: ffffffffffffffda RBX: 000023f3339a9710 RCX:
00007fadd17c945e
[1070034.614930] RDX: 00000000000000bb RSI: 00000000000000eb RDI:
0000000000000304
[1070034.614931] RBP: 00005604376778bd R08: 0023ff6cb6a33301 R09:
00007ffd8a3c0080
[1070034.614931] R10: 0000000000000023 R11: 0000000000000246 R12:
000023f3339a96c0
[1070034.614932] R13: 0000000000000001 R14: 00007fad68b7e938 R15:
000056043dfdbaf8
[1070034.614933] Code: 48 8b 44 24 28 48 8b 40 60 f0 0f ba a8 08 ce 00
00 02 72 19 83 7c 24 58 fb 74 12 8b 74 24 58 48 c7 c7 e8 3b 22 93 e8
06 b5 c7 ff <0f> ff 8b 4c 24 58 ba 27 12 00 00 48 8b 7c 24 28 48 c7 c6
30 0b
[1070034.614950] ---[ end trace c8eff7895ddacab0 ]---
[1070034.614951] BTRFS: error (device sdb2) in
btrfs_truncate_inode_items:4647: errno=-28 No space left
[1070034.614954] BTRFS info (device sdb2): forced readonly
[1070034.616386] BTRFS error (device sdb2): pending csums is 106496

Re: [4.14.3] btrfs out of space error

[Chris Murphy]

On Thu, Dec 14, 2017 at 5:39 PM, Ian Kumlien <ian.kumlien@gmail.com> wrote:
> Hi,
>
> Running a 4.14.3 kernel, this just happened, but there should have
> been another 20 gigs or so available.
>
> The filesystem seems fine after a reboot though
>
> [1070034.614893] ------------[ cut here ]------------
> [1070034.614899] WARNING: CPU: 4 PID: 18634 at fs/btrfs/inode.c:4647
> btrfs_truncate_inode_items+0xea5/0xfb0
> [1070034.614899] Modules linked in: mlx4_en chaoskey amdgpu mfd_core mlx4_core
> [1070034.614904] CPU: 4 PID: 18634 Comm: TaskSchedulerFo Not tainted 4.14.3 #165
> [1070034.614905] Hardware name: To be filled by O.E.M. To be filled by
> O.E.M./SABERTOOTH 990FX, BIOS 1604 10/16/2012
> [1070034.614906] task: ffff8bb0f2332c00 task.stack: ffff9630064c4000
> [1070034.614907] RIP: 0010:btrfs_truncate_inode_items+0xea5/0xfb0
> [1070034.614907] RSP: 0018:ffff9630064c7cc0 EFLAGS: 00010282
> [1070034.614909] RAX: 0000000000000026 RBX: ffff8bb11348a070 RCX:
> 0000000000000000
> [1070034.614909] RDX: ffff8bb13ed147b0 RSI: ffff8bb13ed0c9d8 RDI:
> ffff8bb13ed0c9d8
> [1070034.614910] RBP: ffff8baf61905c70 R08: 0000000000000001 R09:
> 00000000000005f1
> [1070034.614910] R10: 0000000000001000 R11: 0000000000000000 R12:
> ffff8bb1138e0800
> [1070034.614911] R13: 00000000000001e6 R14: ffff8baf60610ee0 R15:
> 00000000000000eb
> [1070034.614912] FS:  00007fad68b7f700(0000) GS:ffff8bb13ed00000(0000)
> knlGS:0000000000000000
> [1070034.614912] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [1070034.614913] CR2: 00001c7e020d0000 CR3: 0000000180a19000 CR4:
> 00000000000406e0
> [1070034.614913] Call Trace:
> [1070034.614918]  btrfs_truncate+0x11d/0x2c0
> [1070034.614919]  btrfs_setattr+0x27b/0x3c0
> [1070034.614921]  notify_change+0x29f/0x430
> [1070034.614923]  do_truncate+0x5e/0x90
> [1070034.614925]  do_sys_ftruncate.constprop.3+0x139/0x1a0
> [1070034.614927]  entry_SYSCALL_64_fastpath+0x13/0x94
> [1070034.614928] RIP: 0033:0x7fadd17c945e
> [1070034.614929] RSP: 002b:00007fad68b7e870 EFLAGS: 00000246 ORIG_RAX:
> 000000000000004d
> [1070034.614930] RAX: ffffffffffffffda RBX: 000023f3339a9710 RCX:
> 00007fadd17c945e
> [1070034.614930] RDX: 00000000000000bb RSI: 00000000000000eb RDI:
> 0000000000000304
> [1070034.614931] RBP: 00005604376778bd R08: 0023ff6cb6a33301 R09:
> 00007ffd8a3c0080
> [1070034.614931] R10: 0000000000000023 R11: 0000000000000246 R12:
> 000023f3339a96c0
> [1070034.614932] R13: 0000000000000001 R14: 00007fad68b7e938 R15:
> 000056043dfdbaf8
> [1070034.614933] Code: 48 8b 44 24 28 48 8b 40 60 f0 0f ba a8 08 ce 00
> 00 02 72 19 83 7c 24 58 fb 74 12 8b 74 24 58 48 c7 c7 e8 3b 22 93 e8
> 06 b5 c7 ff <0f> ff 8b 4c 24 58 ba 27 12 00 00 48 8b 7c 24 28 48 c7 c6
> 30 0b
> [1070034.614950] ---[ end trace c8eff7895ddacab0 ]---
> [1070034.614951] BTRFS: error (device sdb2) in
> btrfs_truncate_inode_items:4647: errno=-28 No space left
> [1070034.614954] BTRFS info (device sdb2): forced readonly
> [1070034.616386] BTRFS error (device sdb2): pending csums is 106496

This last line is concerning, I interpret that as csum tree flush did
not finish before the fs was forced read only. I'm not sure what the
future consequences of that will be.

You should probably run a scrub and also an offline readonly btrfs
check and see what that turns up.

-- 
Chris Murphy

Re: [4.14.3] btrfs out of space error

[Roman Mamedov]

On Fri, 15 Dec 2017 01:39:03 +0100
Ian Kumlien <ian.kumlien@gmail.com> wrote:

> Hi,
> 
> Running a 4.14.3 kernel, this just happened, but there should have
> been another 20 gigs or so available.
> 
> The filesystem seems fine after a reboot though

What are your mount options, and can you show the output of "btrfs fi
df" and "btrfs fi us" for the filesystem? And what does 
"cat /sys/block/sdb/queue/rotational" return.

I wonder if it's the same old "ssd allocation scheme" problem, and no
balancing done in a long time or at all.

-- 
With respect,
Roman

Re: [4.14.3] btrfs out of space error

[Ian Kumlien]

On Fri, Dec 15, 2017 at 6:34 AM, Roman Mamedov <rm@romanrm.net> wrote:
> On Fri, 15 Dec 2017 01:39:03 +0100
> Ian Kumlien <ian.kumlien@gmail.com> wrote:
>
>> Hi,
>>
>> Running a 4.14.3 kernel, this just happened, but there should have
>> been another 20 gigs or so available.
>>
>> The filesystem seems fine after a reboot though
>
> What are your mount options, and can you show the output of "btrfs fi
> df" and "btrfs fi us" for the filesystem? And what does
> "cat /sys/block/sdb/queue/rotational" return.

It's a btrfs raid1 mirror of two ssd:s

mount options was:
defaults,acl,noatime,space_cache,autodefrag,compress=lzo

btrfs fi df /
Data, RAID1: total=459.25GiB, used=372.42GiB
Data, single: total=8.00MiB, used=0.00B
System, RAID1: total=8.00MiB, used=80.00KiB
System, single: total=4.00MiB, used=0.00B
Metadata, RAID1: total=6.00GiB, used=3.69GiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=512.00MiB, used=0.00B

btrfs fi us /
Overall:
    Device size: 930.54GiB
    Device allocated: 930.53GiB
    Device unallocated:   20.05MiB
    Device missing:      0.00B
    Used: 752.22GiB
    Free (estimated):   86.84GiB (min: 86.84GiB)
    Data ratio:       2.00
    Metadata ratio:       2.00
    Global reserve: 512.00MiB (used: 0.00B)

Data,single: Size:8.00MiB, Used:0.00B
   /dev/sdb2    8.00MiB

Data,RAID1: Size:459.25GiB, Used:372.42GiB
   /dev/sdb2 459.25GiB
   /dev/sdc2 459.25GiB

Metadata,single: Size:8.00MiB, Used:0.00B
   /dev/sdb2    8.00MiB

Metadata,RAID1: Size:6.00GiB, Used:3.69GiB
   /dev/sdb2    6.00GiB
   /dev/sdc2    6.00GiB

System,single: Size:4.00MiB, Used:0.00B
   /dev/sdb2    4.00MiB

System,RAID1: Size:8.00MiB, Used:80.00KiB
   /dev/sdb2    8.00MiB
   /dev/sdc2    8.00MiB

Unallocated:
   /dev/sdb2   24.00KiB
   /dev/sdc2   20.02MiB

And as expected:
cat /sys/block/sdb/queue/rotational
0

> I wonder if it's the same old "ssd allocation scheme" problem, and no
> balancing done in a long time or at all.

I had something similar happen on a laptop a while ago - took a while
before i could get it back in order
(in that case i think it was actually a oops --- it kept saying "no
space left" and switched to read only even
if you removed a lot of data, invalidated the space cache and so on)

Re: [4.14.3] btrfs out of space error

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 3202 bytes --]



On 2017年12月15日 16:36, Ian Kumlien wrote:
> On Fri, Dec 15, 2017 at 6:34 AM, Roman Mamedov <rm@romanrm.net> wrote:
>> On Fri, 15 Dec 2017 01:39:03 +0100
>> Ian Kumlien <ian.kumlien@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Running a 4.14.3 kernel, this just happened, but there should have
>>> been another 20 gigs or so available.
>>>
>>> The filesystem seems fine after a reboot though
>>
>> What are your mount options, and can you show the output of "btrfs fi
>> df" and "btrfs fi us" for the filesystem? And what does
>> "cat /sys/block/sdb/queue/rotational" return.
> 
> It's a btrfs raid1 mirror of two ssd:s
> 
> mount options was:
> defaults,acl,noatime,space_cache,autodefrag,compress=lzo
> 
> btrfs fi df /
> Data, RAID1: total=459.25GiB, used=372.42GiB
> Data, single: total=8.00MiB, used=0.00B
> System, RAID1: total=8.00MiB, used=80.00KiB
> System, single: total=4.00MiB, used=0.00B
> Metadata, RAID1: total=6.00GiB, used=3.69GiB

Both meta and data has a lot of space let.

> Metadata, single: total=8.00MiB, used=0.00B
> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> btrfs fi us /
> Overall:
>     Device size: 930.54GiB
>     Device allocated: 930.53GiB
>     Device unallocated:   20.05MiB
>     Device missing:      0.00B
>     Used: 752.22GiB
>     Free (estimated):   86.84GiB (min: 86.84GiB)
>     Data ratio:       2.00
>     Metadata ratio:       2.00
>     Global reserve: 512.00MiB (used: 0.00B)
> 
> Data,single: Size:8.00MiB, Used:0.00B
>    /dev/sdb2    8.00MiB
> 
> Data,RAID1: Size:459.25GiB, Used:372.42GiB
>    /dev/sdb2 459.25GiB
>    /dev/sdc2 459.25GiB
> 
> Metadata,single: Size:8.00MiB, Used:0.00B
>    /dev/sdb2    8.00MiB
> 
> Metadata,RAID1: Size:6.00GiB, Used:3.69GiB
>    /dev/sdb2    6.00GiB
>    /dev/sdc2    6.00GiB
> 
> System,single: Size:4.00MiB, Used:0.00B
>    /dev/sdb2    4.00MiB
> 
> System,RAID1: Size:8.00MiB, Used:80.00KiB
>    /dev/sdb2    8.00MiB
>    /dev/sdc2    8.00MiB
> 
> Unallocated:
>    /dev/sdb2   24.00KiB
>    /dev/sdc2   20.02MiB

Well, at least no new chunk can be allocated.


In v4.15, Josef introduced a new inode reservation system, which I think
it would enhance metadata related reservation.

If you're hitting the problem quite frequently, please consider to try
v4.15-rc* to see if it solves the problem.

Despite of that, there should be no damage to your fs.
(except some unwritten data in buffer)

Thanks,
Qu

> 
> And as expected:
> cat /sys/block/sdb/queue/rotational
> 0
> 
>> I wonder if it's the same old "ssd allocation scheme" problem, and no
>> balancing done in a long time or at all.
> 
> I had something similar happen on a laptop a while ago - took a while
> before i could get it back in order
> (in that case i think it was actually a oops --- it kept saying "no
> space left" and switched to read only even
> if you removed a lot of data, invalidated the space cache and so on)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]

Re: [4.14.3] btrfs out of space error

[Hans van Kranenburg]

On 12/15/2017 09:53 AM, Qu Wenruo wrote:
> 
> 
> On 2017年12月15日 16:36, Ian Kumlien wrote:
>> On Fri, Dec 15, 2017 at 6:34 AM, Roman Mamedov <rm@romanrm.net> wrote:
>>> On Fri, 15 Dec 2017 01:39:03 +0100
>>> Ian Kumlien <ian.kumlien@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Running a 4.14.3 kernel, this just happened, but there should have
>>>> been another 20 gigs or so available.
>>>>
>>>> The filesystem seems fine after a reboot though
>>>
>>> What are your mount options, and can you show the output of "btrfs fi
>>> df" and "btrfs fi us" for the filesystem? And what does
>>> "cat /sys/block/sdb/queue/rotational" return.
>>
>> It's a btrfs raid1 mirror of two ssd:s
>>
>> mount options was:
>> defaults,acl,noatime,space_cache,autodefrag,compress=lzo
>>
>> btrfs fi df /
>> Data, RAID1: total=459.25GiB, used=372.42GiB
>> Data, single: total=8.00MiB, used=0.00B
>> System, RAID1: total=8.00MiB, used=80.00KiB
>> System, single: total=4.00MiB, used=0.00B
>> Metadata, RAID1: total=6.00GiB, used=3.69GiB
> 
> Both meta and data has a lot of space let.

The real question is how fragmented that free space is.

Here's a good starting point to read up about the -o ssd behavior pre 4.14:

https://www.spinics.net/lists/linux-btrfs/msg70622.html

>> Metadata, single: total=8.00MiB, used=0.00B
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>> btrfs fi us /
>> Overall:
>>     Device size: 930.54GiB
>>     Device allocated: 930.53GiB
>>     Device unallocated:   20.05MiB
>>     Device missing:      0.00B
>>     Used: 752.22GiB
>>     Free (estimated):   86.84GiB (min: 86.84GiB)
>>     Data ratio:       2.00
>>     Metadata ratio:       2.00
>>     Global reserve: 512.00MiB (used: 0.00B)
>>
>> Data,single: Size:8.00MiB, Used:0.00B
>>    /dev/sdb2    8.00MiB
>>
>> Data,RAID1: Size:459.25GiB, Used:372.42GiB
>>    /dev/sdb2 459.25GiB
>>    /dev/sdc2 459.25GiB
>>
>> Metadata,single: Size:8.00MiB, Used:0.00B
>>    /dev/sdb2    8.00MiB
>>
>> Metadata,RAID1: Size:6.00GiB, Used:3.69GiB
>>    /dev/sdb2    6.00GiB
>>    /dev/sdc2    6.00GiB
>>
>> System,single: Size:4.00MiB, Used:0.00B
>>    /dev/sdb2    4.00MiB
>>
>> System,RAID1: Size:8.00MiB, Used:80.00KiB
>>    /dev/sdb2    8.00MiB
>>    /dev/sdc2    8.00MiB
>>
>> Unallocated:
>>    /dev/sdb2   24.00KiB
>>    /dev/sdc2   20.02MiB
> 
> Well, at least no new chunk can be allocated.

Exactly.

> In v4.15, Josef introduced a new inode reservation system, which I think
> it would enhance metadata related reservation.
> 
> If you're hitting the problem quite frequently, please consider to try
> v4.15-rc* to see if it solves the problem.
> 
> Despite of that, there should be no damage to your fs.
> (except some unwritten data in buffer)
> 
> Thanks,
> Qu
> 
>>
>> And as expected:
>> cat /sys/block/sdb/queue/rotational
>> 0
>>
>>> I wonder if it's the same old "ssd allocation scheme" problem, and no
>>> balancing done in a long time or at all.

Looks like it. So, yay, you're on 4.14 already. Now just do a full
balance of your entire filesystem, only once (data only, metadata not
needed) and then you can forget about this again.

>> I had something similar happen on a laptop a while ago - took a while
>> before i could get it back in order
>> (in that case i think it was actually a oops --- it kept saying "no
>> space left" and switched to read only even
>> if you removed a lot of data, invalidated the space cache and so on)

-- 
Hans van Kranenburg

Re: [4.14.3] btrfs out of space error

[Ian Kumlien]

On Fri, Dec 15, 2017 at 5:01 AM, Chris Murphy <lists@colorremedies.com> wrote:
> On Thu, Dec 14, 2017 at 5:39 PM, Ian Kumlien <ian.kumlien@gmail.com> wrote:
>> Hi,
>>
>> Running a 4.14.3 kernel, this just happened, but there should have
>> been another 20 gigs or so available.
>>
>> The filesystem seems fine after a reboot though
>>
>> [1070034.614893] ------------[ cut here ]------------
>> [1070034.614899] WARNING: CPU: 4 PID: 18634 at fs/btrfs/inode.c:4647
>> btrfs_truncate_inode_items+0xea5/0xfb0
>> [1070034.614899] Modules linked in: mlx4_en chaoskey amdgpu mfd_core mlx4_core
>> [1070034.614904] CPU: 4 PID: 18634 Comm: TaskSchedulerFo Not tainted 4.14.3 #165
>> [1070034.614905] Hardware name: To be filled by O.E.M. To be filled by
>> O.E.M./SABERTOOTH 990FX, BIOS 1604 10/16/2012
>> [1070034.614906] task: ffff8bb0f2332c00 task.stack: ffff9630064c4000
>> [1070034.614907] RIP: 0010:btrfs_truncate_inode_items+0xea5/0xfb0
>> [1070034.614907] RSP: 0018:ffff9630064c7cc0 EFLAGS: 00010282
>> [1070034.614909] RAX: 0000000000000026 RBX: ffff8bb11348a070 RCX:
>> 0000000000000000
>> [1070034.614909] RDX: ffff8bb13ed147b0 RSI: ffff8bb13ed0c9d8 RDI:
>> ffff8bb13ed0c9d8
>> [1070034.614910] RBP: ffff8baf61905c70 R08: 0000000000000001 R09:
>> 00000000000005f1
>> [1070034.614910] R10: 0000000000001000 R11: 0000000000000000 R12:
>> ffff8bb1138e0800
>> [1070034.614911] R13: 00000000000001e6 R14: ffff8baf60610ee0 R15:
>> 00000000000000eb
>> [1070034.614912] FS:  00007fad68b7f700(0000) GS:ffff8bb13ed00000(0000)
>> knlGS:0000000000000000
>> [1070034.614912] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [1070034.614913] CR2: 00001c7e020d0000 CR3: 0000000180a19000 CR4:
>> 00000000000406e0
>> [1070034.614913] Call Trace:
>> [1070034.614918]  btrfs_truncate+0x11d/0x2c0
>> [1070034.614919]  btrfs_setattr+0x27b/0x3c0
>> [1070034.614921]  notify_change+0x29f/0x430
>> [1070034.614923]  do_truncate+0x5e/0x90
>> [1070034.614925]  do_sys_ftruncate.constprop.3+0x139/0x1a0
>> [1070034.614927]  entry_SYSCALL_64_fastpath+0x13/0x94
>> [1070034.614928] RIP: 0033:0x7fadd17c945e
>> [1070034.614929] RSP: 002b:00007fad68b7e870 EFLAGS: 00000246 ORIG_RAX:
>> 000000000000004d
>> [1070034.614930] RAX: ffffffffffffffda RBX: 000023f3339a9710 RCX:
>> 00007fadd17c945e
>> [1070034.614930] RDX: 00000000000000bb RSI: 00000000000000eb RDI:
>> 0000000000000304
>> [1070034.614931] RBP: 00005604376778bd R08: 0023ff6cb6a33301 R09:
>> 00007ffd8a3c0080
>> [1070034.614931] R10: 0000000000000023 R11: 0000000000000246 R12:
>> 000023f3339a96c0
>> [1070034.614932] R13: 0000000000000001 R14: 00007fad68b7e938 R15:
>> 000056043dfdbaf8
>> [1070034.614933] Code: 48 8b 44 24 28 48 8b 40 60 f0 0f ba a8 08 ce 00
>> 00 02 72 19 83 7c 24 58 fb 74 12 8b 74 24 58 48 c7 c7 e8 3b 22 93 e8
>> 06 b5 c7 ff <0f> ff 8b 4c 24 58 ba 27 12 00 00 48 8b 7c 24 28 48 c7 c6
>> 30 0b
>> [1070034.614950] ---[ end trace c8eff7895ddacab0 ]---
>> [1070034.614951] BTRFS: error (device sdb2) in
>> btrfs_truncate_inode_items:4647: errno=-28 No space left
>> [1070034.614954] BTRFS info (device sdb2): forced readonly
>> [1070034.616386] BTRFS error (device sdb2): pending csums is 106496
>
> This last line is concerning, I interpret that as csum tree flush did
> not finish before the fs was forced read only. I'm not sure what the
> future consequences of that will be.
>
> You should probably run a scrub and also an offline readonly btrfs
> check and see what that turns up.

I did a full defrag and recompression of all files and it was all fine =)

> --
> Chris Murphy

Re: [4.14.3] btrfs out of space error

[Ian Kumlien]

On Fri, Dec 15, 2017 at 9:53 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> On 2017年12月15日 16:36, Ian Kumlien wrote:
>> On Fri, Dec 15, 2017 at 6:34 AM, Roman Mamedov <rm@romanrm.net> wrote:
>>> On Fri, 15 Dec 2017 01:39:03 +0100
>>> Ian Kumlien <ian.kumlien@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Running a 4.14.3 kernel, this just happened, but there should have
>>>> been another 20 gigs or so available.
>>>>
>>>> The filesystem seems fine after a reboot though
>>>
>>> What are your mount options, and can you show the output of "btrfs fi
>>> df" and "btrfs fi us" for the filesystem? And what does
>>> "cat /sys/block/sdb/queue/rotational" return.
>>
>> It's a btrfs raid1 mirror of two ssd:s

[--8<--]

(sorry if you got a reply before - dang html mails)

> Well, at least no new chunk can be allocated.
>
>
> In v4.15, Josef introduced a new inode reservation system, which I think
> it would enhance metadata related reservation.

Humm... maybe when it has been trough some rc:s ;)

> If you're hitting the problem quite frequently, please consider to try
> v4.15-rc* to see if it solves the problem.
>
> Despite of that, there should be no damage to your fs.
> (except some unwritten data in buffer)

Yeah figured as much - did a full defrag and recompress (with zstd
this time) just to play with it

> Thanks,
> Qu
>
>>
>> And as expected:
>> cat /sys/block/sdb/queue/rotational
>> 0
>>
>>> I wonder if it's the same old "ssd allocation scheme" problem, and no
>>> balancing done in a long time or at all.
>>
>> I had something similar happen on a laptop a while ago - took a while
>> before i could get it back in order
>> (in that case i think it was actually a oops --- it kept saying "no
>> space left" and switched to read only even
>> if you removed a lot of data, invalidated the space cache and so on)
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

Re: [4.14.3] btrfs out of space error

[Ian Kumlien]

On Sat, Dec 16, 2017 at 12:07 AM, Hans van Kranenburg
<hans.van.kranenburg@mendix.com> wrote:
> On 12/15/2017 09:53 AM, Qu Wenruo wrote:
>> On 2017年12月15日 16:36, Ian Kumlien wrote:
>>> On Fri, Dec 15, 2017 at 6:34 AM, Roman Mamedov <rm@romanrm.net> wrote:
>>>> On Fri, 15 Dec 2017 01:39:03 +0100
>>>> Ian Kumlien <ian.kumlien@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Running a 4.14.3 kernel, this just happened, but there should have
>>>>> been another 20 gigs or so available.
>>>>>
>>>>> The filesystem seems fine after a reboot though
>>>>
>>>> What are your mount options, and can you show the output of "btrfs fi
>>>> df" and "btrfs fi us" for the filesystem? And what does
>>>> "cat /sys/block/sdb/queue/rotational" return.
>>>
>>> It's a btrfs raid1 mirror of two ssd:s
>>>
>>> mount options was:
>>> defaults,acl,noatime,space_cache,autodefrag,compress=lzo
>>>
>>> btrfs fi df /
>>> Data, RAID1: total=459.25GiB, used=372.42GiB
>>> Data, single: total=8.00MiB, used=0.00B
>>> System, RAID1: total=8.00MiB, used=80.00KiB
>>> System, single: total=4.00MiB, used=0.00B
>>> Metadata, RAID1: total=6.00GiB, used=3.69GiB
>>
>> Both meta and data has a lot of space let.
>
> The real question is how fragmented that free space is.

Somewhat fragmented i'd say

> Here's a good starting point to read up about the -o ssd behavior pre 4.14:
>
> https://www.spinics.net/lists/linux-btrfs/msg70622.html

Will do

[--8<--]

>>> Unallocated:
>>>    /dev/sdb2   24.00KiB
>>>    /dev/sdc2   20.02MiB
>>
>> Well, at least no new chunk can be allocated.
>
> Exactly.

After running:
btrfs balance start -dusage=50 /

a few times (first one failed but freed 6 gigs) it's now:
Unallocated:
   /dev/sdb2   13.25GiB
   /dev/sdc2   13.26GiB

>> In v4.15, Josef introduced a new inode reservation system, which I think
>> it would enhance metadata related reservation.
>>
>> If you're hitting the problem quite frequently, please consider to try
>> v4.15-rc* to see if it solves the problem.
>>
>> Despite of that, there should be no damage to your fs.
>> (except some unwritten data in buffer)
>>
>> Thanks,
>> Qu
>>
>>>
>>> And as expected:
>>> cat /sys/block/sdb/queue/rotational
>>> 0
>>>
>>>> I wonder if it's the same old "ssd allocation scheme" problem, and no
>>>> balancing done in a long time or at all.
>
> Looks like it. So, yay, you're on 4.14 already. Now just do a full
> balance of your entire filesystem, only once (data only, metadata not
> needed) and then you can forget about this again.

Any special filters etc needed?

>>> I had something similar happen on a laptop a while ago - took a while
>>> before i could get it back in order
>>> (in that case i think it was actually a oops --- it kept saying "no
>>> space left" and switched to read only even
>>> if you removed a lot of data, invalidated the space cache and so on)
>
> --
> Hans van Kranenburg