* [4.14.3] btrfs out of space error @ 2017-12-15 0:39 Ian Kumlien 2017-12-15 4:01 ` Chris Murphy 2017-12-15 5:34 ` Roman Mamedov 0 siblings, 2 replies; 9+ messages in thread From: Ian Kumlien @ 2017-12-15 0:39 UTC (permalink / raw) To: Btrfs BTRFS Hi, Running a 4.14.3 kernel, this just happened, but there should have been another 20 gigs or so available. The filesystem seems fine after a reboot though [1070034.614893] ------------[ cut here ]------------ [1070034.614899] WARNING: CPU: 4 PID: 18634 at fs/btrfs/inode.c:4647 btrfs_truncate_inode_items+0xea5/0xfb0 [1070034.614899] Modules linked in: mlx4_en chaoskey amdgpu mfd_core mlx4_core [1070034.614904] CPU: 4 PID: 18634 Comm: TaskSchedulerFo Not tainted 4.14.3 #165 [1070034.614905] Hardware name: To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX, BIOS 1604 10/16/2012 [1070034.614906] task: ffff8bb0f2332c00 task.stack: ffff9630064c4000 [1070034.614907] RIP: 0010:btrfs_truncate_inode_items+0xea5/0xfb0 [1070034.614907] RSP: 0018:ffff9630064c7cc0 EFLAGS: 00010282 [1070034.614909] RAX: 0000000000000026 RBX: ffff8bb11348a070 RCX: 0000000000000000 [1070034.614909] RDX: ffff8bb13ed147b0 RSI: ffff8bb13ed0c9d8 RDI: ffff8bb13ed0c9d8 [1070034.614910] RBP: ffff8baf61905c70 R08: 0000000000000001 R09: 00000000000005f1 [1070034.614910] R10: 0000000000001000 R11: 0000000000000000 R12: ffff8bb1138e0800 [1070034.614911] R13: 00000000000001e6 R14: ffff8baf60610ee0 R15: 00000000000000eb [1070034.614912] FS: 00007fad68b7f700(0000) GS:ffff8bb13ed00000(0000) knlGS:0000000000000000 [1070034.614912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [1070034.614913] CR2: 00001c7e020d0000 CR3: 0000000180a19000 CR4: 00000000000406e0 [1070034.614913] Call Trace: [1070034.614918] btrfs_truncate+0x11d/0x2c0 [1070034.614919] btrfs_setattr+0x27b/0x3c0 [1070034.614921] notify_change+0x29f/0x430 [1070034.614923] do_truncate+0x5e/0x90 [1070034.614925] do_sys_ftruncate.constprop.3+0x139/0x1a0 [1070034.614927] entry_SYSCALL_64_fastpath+0x13/0x94 [1070034.614928] RIP: 0033:0x7fadd17c945e [1070034.614929] RSP: 002b:00007fad68b7e870 EFLAGS: 00000246 ORIG_RAX: 000000000000004d [1070034.614930] RAX: ffffffffffffffda RBX: 000023f3339a9710 RCX: 00007fadd17c945e [1070034.614930] RDX: 00000000000000bb RSI: 00000000000000eb RDI: 0000000000000304 [1070034.614931] RBP: 00005604376778bd R08: 0023ff6cb6a33301 R09: 00007ffd8a3c0080 [1070034.614931] R10: 0000000000000023 R11: 0000000000000246 R12: 000023f3339a96c0 [1070034.614932] R13: 0000000000000001 R14: 00007fad68b7e938 R15: 000056043dfdbaf8 [1070034.614933] Code: 48 8b 44 24 28 48 8b 40 60 f0 0f ba a8 08 ce 00 00 02 72 19 83 7c 24 58 fb 74 12 8b 74 24 58 48 c7 c7 e8 3b 22 93 e8 06 b5 c7 ff <0f> ff 8b 4c 24 58 ba 27 12 00 00 48 8b 7c 24 28 48 c7 c6 30 0b [1070034.614950] ---[ end trace c8eff7895ddacab0 ]--- [1070034.614951] BTRFS: error (device sdb2) in btrfs_truncate_inode_items:4647: errno=-28 No space left [1070034.614954] BTRFS info (device sdb2): forced readonly [1070034.616386] BTRFS error (device sdb2): pending csums is 106496 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [4.14.3] btrfs out of space error 2017-12-15 0:39 [4.14.3] btrfs out of space error Ian Kumlien @ 2017-12-15 4:01 ` Chris Murphy 2017-12-16 10:05 ` Ian Kumlien 2017-12-15 5:34 ` Roman Mamedov 1 sibling, 1 reply; 9+ messages in thread From: Chris Murphy @ 2017-12-15 4:01 UTC (permalink / raw) To: Ian Kumlien; +Cc: Btrfs BTRFS On Thu, Dec 14, 2017 at 5:39 PM, Ian Kumlien <ian.kumlien@gmail.com> wrote: > Hi, > > Running a 4.14.3 kernel, this just happened, but there should have > been another 20 gigs or so available. > > The filesystem seems fine after a reboot though > > [1070034.614893] ------------[ cut here ]------------ > [1070034.614899] WARNING: CPU: 4 PID: 18634 at fs/btrfs/inode.c:4647 > btrfs_truncate_inode_items+0xea5/0xfb0 > [1070034.614899] Modules linked in: mlx4_en chaoskey amdgpu mfd_core mlx4_core > [1070034.614904] CPU: 4 PID: 18634 Comm: TaskSchedulerFo Not tainted 4.14.3 #165 > [1070034.614905] Hardware name: To be filled by O.E.M. To be filled by > O.E.M./SABERTOOTH 990FX, BIOS 1604 10/16/2012 > [1070034.614906] task: ffff8bb0f2332c00 task.stack: ffff9630064c4000 > [1070034.614907] RIP: 0010:btrfs_truncate_inode_items+0xea5/0xfb0 > [1070034.614907] RSP: 0018:ffff9630064c7cc0 EFLAGS: 00010282 > [1070034.614909] RAX: 0000000000000026 RBX: ffff8bb11348a070 RCX: > 0000000000000000 > [1070034.614909] RDX: ffff8bb13ed147b0 RSI: ffff8bb13ed0c9d8 RDI: > ffff8bb13ed0c9d8 > [1070034.614910] RBP: ffff8baf61905c70 R08: 0000000000000001 R09: > 00000000000005f1 > [1070034.614910] R10: 0000000000001000 R11: 0000000000000000 R12: > ffff8bb1138e0800 > [1070034.614911] R13: 00000000000001e6 R14: ffff8baf60610ee0 R15: > 00000000000000eb > [1070034.614912] FS: 00007fad68b7f700(0000) GS:ffff8bb13ed00000(0000) > knlGS:0000000000000000 > [1070034.614912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [1070034.614913] CR2: 00001c7e020d0000 CR3: 0000000180a19000 CR4: > 00000000000406e0 > [1070034.614913] Call Trace: > [1070034.614918] btrfs_truncate+0x11d/0x2c0 > [1070034.614919] btrfs_setattr+0x27b/0x3c0 > [1070034.614921] notify_change+0x29f/0x430 > [1070034.614923] do_truncate+0x5e/0x90 > [1070034.614925] do_sys_ftruncate.constprop.3+0x139/0x1a0 > [1070034.614927] entry_SYSCALL_64_fastpath+0x13/0x94 > [1070034.614928] RIP: 0033:0x7fadd17c945e > [1070034.614929] RSP: 002b:00007fad68b7e870 EFLAGS: 00000246 ORIG_RAX: > 000000000000004d > [1070034.614930] RAX: ffffffffffffffda RBX: 000023f3339a9710 RCX: > 00007fadd17c945e > [1070034.614930] RDX: 00000000000000bb RSI: 00000000000000eb RDI: > 0000000000000304 > [1070034.614931] RBP: 00005604376778bd R08: 0023ff6cb6a33301 R09: > 00007ffd8a3c0080 > [1070034.614931] R10: 0000000000000023 R11: 0000000000000246 R12: > 000023f3339a96c0 > [1070034.614932] R13: 0000000000000001 R14: 00007fad68b7e938 R15: > 000056043dfdbaf8 > [1070034.614933] Code: 48 8b 44 24 28 48 8b 40 60 f0 0f ba a8 08 ce 00 > 00 02 72 19 83 7c 24 58 fb 74 12 8b 74 24 58 48 c7 c7 e8 3b 22 93 e8 > 06 b5 c7 ff <0f> ff 8b 4c 24 58 ba 27 12 00 00 48 8b 7c 24 28 48 c7 c6 > 30 0b > [1070034.614950] ---[ end trace c8eff7895ddacab0 ]--- > [1070034.614951] BTRFS: error (device sdb2) in > btrfs_truncate_inode_items:4647: errno=-28 No space left > [1070034.614954] BTRFS info (device sdb2): forced readonly > [1070034.616386] BTRFS error (device sdb2): pending csums is 106496 This last line is concerning, I interpret that as csum tree flush did not finish before the fs was forced read only. I'm not sure what the future consequences of that will be. You should probably run a scrub and also an offline readonly btrfs check and see what that turns up. -- Chris Murphy ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [4.14.3] btrfs out of space error 2017-12-15 4:01 ` Chris Murphy @ 2017-12-16 10:05 ` Ian Kumlien 0 siblings, 0 replies; 9+ messages in thread From: Ian Kumlien @ 2017-12-16 10:05 UTC (permalink / raw) To: Chris Murphy; +Cc: Btrfs BTRFS On Fri, Dec 15, 2017 at 5:01 AM, Chris Murphy <lists@colorremedies.com> wrote: > On Thu, Dec 14, 2017 at 5:39 PM, Ian Kumlien <ian.kumlien@gmail.com> wrote: >> Hi, >> >> Running a 4.14.3 kernel, this just happened, but there should have >> been another 20 gigs or so available. >> >> The filesystem seems fine after a reboot though >> >> [1070034.614893] ------------[ cut here ]------------ >> [1070034.614899] WARNING: CPU: 4 PID: 18634 at fs/btrfs/inode.c:4647 >> btrfs_truncate_inode_items+0xea5/0xfb0 >> [1070034.614899] Modules linked in: mlx4_en chaoskey amdgpu mfd_core mlx4_core >> [1070034.614904] CPU: 4 PID: 18634 Comm: TaskSchedulerFo Not tainted 4.14.3 #165 >> [1070034.614905] Hardware name: To be filled by O.E.M. To be filled by >> O.E.M./SABERTOOTH 990FX, BIOS 1604 10/16/2012 >> [1070034.614906] task: ffff8bb0f2332c00 task.stack: ffff9630064c4000 >> [1070034.614907] RIP: 0010:btrfs_truncate_inode_items+0xea5/0xfb0 >> [1070034.614907] RSP: 0018:ffff9630064c7cc0 EFLAGS: 00010282 >> [1070034.614909] RAX: 0000000000000026 RBX: ffff8bb11348a070 RCX: >> 0000000000000000 >> [1070034.614909] RDX: ffff8bb13ed147b0 RSI: ffff8bb13ed0c9d8 RDI: >> ffff8bb13ed0c9d8 >> [1070034.614910] RBP: ffff8baf61905c70 R08: 0000000000000001 R09: >> 00000000000005f1 >> [1070034.614910] R10: 0000000000001000 R11: 0000000000000000 R12: >> ffff8bb1138e0800 >> [1070034.614911] R13: 00000000000001e6 R14: ffff8baf60610ee0 R15: >> 00000000000000eb >> [1070034.614912] FS: 00007fad68b7f700(0000) GS:ffff8bb13ed00000(0000) >> knlGS:0000000000000000 >> [1070034.614912] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> [1070034.614913] CR2: 00001c7e020d0000 CR3: 0000000180a19000 CR4: >> 00000000000406e0 >> [1070034.614913] Call Trace: >> [1070034.614918] btrfs_truncate+0x11d/0x2c0 >> [1070034.614919] btrfs_setattr+0x27b/0x3c0 >> [1070034.614921] notify_change+0x29f/0x430 >> [1070034.614923] do_truncate+0x5e/0x90 >> [1070034.614925] do_sys_ftruncate.constprop.3+0x139/0x1a0 >> [1070034.614927] entry_SYSCALL_64_fastpath+0x13/0x94 >> [1070034.614928] RIP: 0033:0x7fadd17c945e >> [1070034.614929] RSP: 002b:00007fad68b7e870 EFLAGS: 00000246 ORIG_RAX: >> 000000000000004d >> [1070034.614930] RAX: ffffffffffffffda RBX: 000023f3339a9710 RCX: >> 00007fadd17c945e >> [1070034.614930] RDX: 00000000000000bb RSI: 00000000000000eb RDI: >> 0000000000000304 >> [1070034.614931] RBP: 00005604376778bd R08: 0023ff6cb6a33301 R09: >> 00007ffd8a3c0080 >> [1070034.614931] R10: 0000000000000023 R11: 0000000000000246 R12: >> 000023f3339a96c0 >> [1070034.614932] R13: 0000000000000001 R14: 00007fad68b7e938 R15: >> 000056043dfdbaf8 >> [1070034.614933] Code: 48 8b 44 24 28 48 8b 40 60 f0 0f ba a8 08 ce 00 >> 00 02 72 19 83 7c 24 58 fb 74 12 8b 74 24 58 48 c7 c7 e8 3b 22 93 e8 >> 06 b5 c7 ff <0f> ff 8b 4c 24 58 ba 27 12 00 00 48 8b 7c 24 28 48 c7 c6 >> 30 0b >> [1070034.614950] ---[ end trace c8eff7895ddacab0 ]--- >> [1070034.614951] BTRFS: error (device sdb2) in >> btrfs_truncate_inode_items:4647: errno=-28 No space left >> [1070034.614954] BTRFS info (device sdb2): forced readonly >> [1070034.616386] BTRFS error (device sdb2): pending csums is 106496 > > This last line is concerning, I interpret that as csum tree flush did > not finish before the fs was forced read only. I'm not sure what the > future consequences of that will be. > > You should probably run a scrub and also an offline readonly btrfs > check and see what that turns up. I did a full defrag and recompression of all files and it was all fine =) > -- > Chris Murphy ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [4.14.3] btrfs out of space error 2017-12-15 0:39 [4.14.3] btrfs out of space error Ian Kumlien 2017-12-15 4:01 ` Chris Murphy @ 2017-12-15 5:34 ` Roman Mamedov 2017-12-15 8:36 ` Ian Kumlien 1 sibling, 1 reply; 9+ messages in thread From: Roman Mamedov @ 2017-12-15 5:34 UTC (permalink / raw) To: Ian Kumlien; +Cc: Btrfs BTRFS On Fri, 15 Dec 2017 01:39:03 +0100 Ian Kumlien <ian.kumlien@gmail.com> wrote: > Hi, > > Running a 4.14.3 kernel, this just happened, but there should have > been another 20 gigs or so available. > > The filesystem seems fine after a reboot though What are your mount options, and can you show the output of "btrfs fi df" and "btrfs fi us" for the filesystem? And what does "cat /sys/block/sdb/queue/rotational" return. I wonder if it's the same old "ssd allocation scheme" problem, and no balancing done in a long time or at all. -- With respect, Roman ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [4.14.3] btrfs out of space error 2017-12-15 5:34 ` Roman Mamedov @ 2017-12-15 8:36 ` Ian Kumlien 2017-12-15 8:53 ` Qu Wenruo 0 siblings, 1 reply; 9+ messages in thread From: Ian Kumlien @ 2017-12-15 8:36 UTC (permalink / raw) To: Roman Mamedov; +Cc: Btrfs BTRFS On Fri, Dec 15, 2017 at 6:34 AM, Roman Mamedov <rm@romanrm.net> wrote: > On Fri, 15 Dec 2017 01:39:03 +0100 > Ian Kumlien <ian.kumlien@gmail.com> wrote: > >> Hi, >> >> Running a 4.14.3 kernel, this just happened, but there should have >> been another 20 gigs or so available. >> >> The filesystem seems fine after a reboot though > > What are your mount options, and can you show the output of "btrfs fi > df" and "btrfs fi us" for the filesystem? And what does > "cat /sys/block/sdb/queue/rotational" return. It's a btrfs raid1 mirror of two ssd:s mount options was: defaults,acl,noatime,space_cache,autodefrag,compress=lzo btrfs fi df / Data, RAID1: total=459.25GiB, used=372.42GiB Data, single: total=8.00MiB, used=0.00B System, RAID1: total=8.00MiB, used=80.00KiB System, single: total=4.00MiB, used=0.00B Metadata, RAID1: total=6.00GiB, used=3.69GiB Metadata, single: total=8.00MiB, used=0.00B GlobalReserve, single: total=512.00MiB, used=0.00B btrfs fi us / Overall: Device size: 930.54GiB Device allocated: 930.53GiB Device unallocated: 20.05MiB Device missing: 0.00B Used: 752.22GiB Free (estimated): 86.84GiB (min: 86.84GiB) Data ratio: 2.00 Metadata ratio: 2.00 Global reserve: 512.00MiB (used: 0.00B) Data,single: Size:8.00MiB, Used:0.00B /dev/sdb2 8.00MiB Data,RAID1: Size:459.25GiB, Used:372.42GiB /dev/sdb2 459.25GiB /dev/sdc2 459.25GiB Metadata,single: Size:8.00MiB, Used:0.00B /dev/sdb2 8.00MiB Metadata,RAID1: Size:6.00GiB, Used:3.69GiB /dev/sdb2 6.00GiB /dev/sdc2 6.00GiB System,single: Size:4.00MiB, Used:0.00B /dev/sdb2 4.00MiB System,RAID1: Size:8.00MiB, Used:80.00KiB /dev/sdb2 8.00MiB /dev/sdc2 8.00MiB Unallocated: /dev/sdb2 24.00KiB /dev/sdc2 20.02MiB And as expected: cat /sys/block/sdb/queue/rotational 0 > I wonder if it's the same old "ssd allocation scheme" problem, and no > balancing done in a long time or at all. I had something similar happen on a laptop a while ago - took a while before i could get it back in order (in that case i think it was actually a oops --- it kept saying "no space left" and switched to read only even if you removed a lot of data, invalidated the space cache and so on) ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [4.14.3] btrfs out of space error 2017-12-15 8:36 ` Ian Kumlien @ 2017-12-15 8:53 ` Qu Wenruo 2017-12-15 23:07 ` Hans van Kranenburg 2017-12-16 10:11 ` Ian Kumlien 0 siblings, 2 replies; 9+ messages in thread From: Qu Wenruo @ 2017-12-15 8:53 UTC (permalink / raw) To: Ian Kumlien, Roman Mamedov; +Cc: Btrfs BTRFS [-- Attachment #1.1: Type: text/plain, Size: 3202 bytes --] On 2017年12月15日 16:36, Ian Kumlien wrote: > On Fri, Dec 15, 2017 at 6:34 AM, Roman Mamedov <rm@romanrm.net> wrote: >> On Fri, 15 Dec 2017 01:39:03 +0100 >> Ian Kumlien <ian.kumlien@gmail.com> wrote: >> >>> Hi, >>> >>> Running a 4.14.3 kernel, this just happened, but there should have >>> been another 20 gigs or so available. >>> >>> The filesystem seems fine after a reboot though >> >> What are your mount options, and can you show the output of "btrfs fi >> df" and "btrfs fi us" for the filesystem? And what does >> "cat /sys/block/sdb/queue/rotational" return. > > It's a btrfs raid1 mirror of two ssd:s > > mount options was: > defaults,acl,noatime,space_cache,autodefrag,compress=lzo > > btrfs fi df / > Data, RAID1: total=459.25GiB, used=372.42GiB > Data, single: total=8.00MiB, used=0.00B > System, RAID1: total=8.00MiB, used=80.00KiB > System, single: total=4.00MiB, used=0.00B > Metadata, RAID1: total=6.00GiB, used=3.69GiB Both meta and data has a lot of space let. > Metadata, single: total=8.00MiB, used=0.00B > GlobalReserve, single: total=512.00MiB, used=0.00B > > btrfs fi us / > Overall: > Device size: 930.54GiB > Device allocated: 930.53GiB > Device unallocated: 20.05MiB > Device missing: 0.00B > Used: 752.22GiB > Free (estimated): 86.84GiB (min: 86.84GiB) > Data ratio: 2.00 > Metadata ratio: 2.00 > Global reserve: 512.00MiB (used: 0.00B) > > Data,single: Size:8.00MiB, Used:0.00B > /dev/sdb2 8.00MiB > > Data,RAID1: Size:459.25GiB, Used:372.42GiB > /dev/sdb2 459.25GiB > /dev/sdc2 459.25GiB > > Metadata,single: Size:8.00MiB, Used:0.00B > /dev/sdb2 8.00MiB > > Metadata,RAID1: Size:6.00GiB, Used:3.69GiB > /dev/sdb2 6.00GiB > /dev/sdc2 6.00GiB > > System,single: Size:4.00MiB, Used:0.00B > /dev/sdb2 4.00MiB > > System,RAID1: Size:8.00MiB, Used:80.00KiB > /dev/sdb2 8.00MiB > /dev/sdc2 8.00MiB > > Unallocated: > /dev/sdb2 24.00KiB > /dev/sdc2 20.02MiB Well, at least no new chunk can be allocated. In v4.15, Josef introduced a new inode reservation system, which I think it would enhance metadata related reservation. If you're hitting the problem quite frequently, please consider to try v4.15-rc* to see if it solves the problem. Despite of that, there should be no damage to your fs. (except some unwritten data in buffer) Thanks, Qu > > And as expected: > cat /sys/block/sdb/queue/rotational > 0 > >> I wonder if it's the same old "ssd allocation scheme" problem, and no >> balancing done in a long time or at all. > > I had something similar happen on a laptop a while ago - took a while > before i could get it back in order > (in that case i think it was actually a oops --- it kept saying "no > space left" and switched to read only even > if you removed a lot of data, invalidated the space cache and so on) > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 520 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [4.14.3] btrfs out of space error 2017-12-15 8:53 ` Qu Wenruo @ 2017-12-15 23:07 ` Hans van Kranenburg 2017-12-16 10:20 ` Ian Kumlien 2017-12-16 10:11 ` Ian Kumlien 1 sibling, 1 reply; 9+ messages in thread From: Hans van Kranenburg @ 2017-12-15 23:07 UTC (permalink / raw) To: Qu Wenruo, Ian Kumlien, Roman Mamedov; +Cc: Btrfs BTRFS On 12/15/2017 09:53 AM, Qu Wenruo wrote: > > > On 2017年12月15日 16:36, Ian Kumlien wrote: >> On Fri, Dec 15, 2017 at 6:34 AM, Roman Mamedov <rm@romanrm.net> wrote: >>> On Fri, 15 Dec 2017 01:39:03 +0100 >>> Ian Kumlien <ian.kumlien@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> Running a 4.14.3 kernel, this just happened, but there should have >>>> been another 20 gigs or so available. >>>> >>>> The filesystem seems fine after a reboot though >>> >>> What are your mount options, and can you show the output of "btrfs fi >>> df" and "btrfs fi us" for the filesystem? And what does >>> "cat /sys/block/sdb/queue/rotational" return. >> >> It's a btrfs raid1 mirror of two ssd:s >> >> mount options was: >> defaults,acl,noatime,space_cache,autodefrag,compress=lzo >> >> btrfs fi df / >> Data, RAID1: total=459.25GiB, used=372.42GiB >> Data, single: total=8.00MiB, used=0.00B >> System, RAID1: total=8.00MiB, used=80.00KiB >> System, single: total=4.00MiB, used=0.00B >> Metadata, RAID1: total=6.00GiB, used=3.69GiB > > Both meta and data has a lot of space let. The real question is how fragmented that free space is. Here's a good starting point to read up about the -o ssd behavior pre 4.14: https://www.spinics.net/lists/linux-btrfs/msg70622.html >> Metadata, single: total=8.00MiB, used=0.00B >> GlobalReserve, single: total=512.00MiB, used=0.00B >> >> btrfs fi us / >> Overall: >> Device size: 930.54GiB >> Device allocated: 930.53GiB >> Device unallocated: 20.05MiB >> Device missing: 0.00B >> Used: 752.22GiB >> Free (estimated): 86.84GiB (min: 86.84GiB) >> Data ratio: 2.00 >> Metadata ratio: 2.00 >> Global reserve: 512.00MiB (used: 0.00B) >> >> Data,single: Size:8.00MiB, Used:0.00B >> /dev/sdb2 8.00MiB >> >> Data,RAID1: Size:459.25GiB, Used:372.42GiB >> /dev/sdb2 459.25GiB >> /dev/sdc2 459.25GiB >> >> Metadata,single: Size:8.00MiB, Used:0.00B >> /dev/sdb2 8.00MiB >> >> Metadata,RAID1: Size:6.00GiB, Used:3.69GiB >> /dev/sdb2 6.00GiB >> /dev/sdc2 6.00GiB >> >> System,single: Size:4.00MiB, Used:0.00B >> /dev/sdb2 4.00MiB >> >> System,RAID1: Size:8.00MiB, Used:80.00KiB >> /dev/sdb2 8.00MiB >> /dev/sdc2 8.00MiB >> >> Unallocated: >> /dev/sdb2 24.00KiB >> /dev/sdc2 20.02MiB > > Well, at least no new chunk can be allocated. Exactly. > In v4.15, Josef introduced a new inode reservation system, which I think > it would enhance metadata related reservation. > > If you're hitting the problem quite frequently, please consider to try > v4.15-rc* to see if it solves the problem. > > Despite of that, there should be no damage to your fs. > (except some unwritten data in buffer) > > Thanks, > Qu > >> >> And as expected: >> cat /sys/block/sdb/queue/rotational >> 0 >> >>> I wonder if it's the same old "ssd allocation scheme" problem, and no >>> balancing done in a long time or at all. Looks like it. So, yay, you're on 4.14 already. Now just do a full balance of your entire filesystem, only once (data only, metadata not needed) and then you can forget about this again. >> I had something similar happen on a laptop a while ago - took a while >> before i could get it back in order >> (in that case i think it was actually a oops --- it kept saying "no >> space left" and switched to read only even >> if you removed a lot of data, invalidated the space cache and so on) -- Hans van Kranenburg ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [4.14.3] btrfs out of space error 2017-12-15 23:07 ` Hans van Kranenburg @ 2017-12-16 10:20 ` Ian Kumlien 0 siblings, 0 replies; 9+ messages in thread From: Ian Kumlien @ 2017-12-16 10:20 UTC (permalink / raw) To: Hans van Kranenburg; +Cc: Qu Wenruo, Roman Mamedov, Btrfs BTRFS On Sat, Dec 16, 2017 at 12:07 AM, Hans van Kranenburg <hans.van.kranenburg@mendix.com> wrote: > On 12/15/2017 09:53 AM, Qu Wenruo wrote: >> On 2017年12月15日 16:36, Ian Kumlien wrote: >>> On Fri, Dec 15, 2017 at 6:34 AM, Roman Mamedov <rm@romanrm.net> wrote: >>>> On Fri, 15 Dec 2017 01:39:03 +0100 >>>> Ian Kumlien <ian.kumlien@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> Running a 4.14.3 kernel, this just happened, but there should have >>>>> been another 20 gigs or so available. >>>>> >>>>> The filesystem seems fine after a reboot though >>>> >>>> What are your mount options, and can you show the output of "btrfs fi >>>> df" and "btrfs fi us" for the filesystem? And what does >>>> "cat /sys/block/sdb/queue/rotational" return. >>> >>> It's a btrfs raid1 mirror of two ssd:s >>> >>> mount options was: >>> defaults,acl,noatime,space_cache,autodefrag,compress=lzo >>> >>> btrfs fi df / >>> Data, RAID1: total=459.25GiB, used=372.42GiB >>> Data, single: total=8.00MiB, used=0.00B >>> System, RAID1: total=8.00MiB, used=80.00KiB >>> System, single: total=4.00MiB, used=0.00B >>> Metadata, RAID1: total=6.00GiB, used=3.69GiB >> >> Both meta and data has a lot of space let. > > The real question is how fragmented that free space is. Somewhat fragmented i'd say > Here's a good starting point to read up about the -o ssd behavior pre 4.14: > > https://www.spinics.net/lists/linux-btrfs/msg70622.html Will do [--8<--] >>> Unallocated: >>> /dev/sdb2 24.00KiB >>> /dev/sdc2 20.02MiB >> >> Well, at least no new chunk can be allocated. > > Exactly. After running: btrfs balance start -dusage=50 / a few times (first one failed but freed 6 gigs) it's now: Unallocated: /dev/sdb2 13.25GiB /dev/sdc2 13.26GiB >> In v4.15, Josef introduced a new inode reservation system, which I think >> it would enhance metadata related reservation. >> >> If you're hitting the problem quite frequently, please consider to try >> v4.15-rc* to see if it solves the problem. >> >> Despite of that, there should be no damage to your fs. >> (except some unwritten data in buffer) >> >> Thanks, >> Qu >> >>> >>> And as expected: >>> cat /sys/block/sdb/queue/rotational >>> 0 >>> >>>> I wonder if it's the same old "ssd allocation scheme" problem, and no >>>> balancing done in a long time or at all. > > Looks like it. So, yay, you're on 4.14 already. Now just do a full > balance of your entire filesystem, only once (data only, metadata not > needed) and then you can forget about this again. Any special filters etc needed? >>> I had something similar happen on a laptop a while ago - took a while >>> before i could get it back in order >>> (in that case i think it was actually a oops --- it kept saying "no >>> space left" and switched to read only even >>> if you removed a lot of data, invalidated the space cache and so on) > > -- > Hans van Kranenburg ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [4.14.3] btrfs out of space error 2017-12-15 8:53 ` Qu Wenruo 2017-12-15 23:07 ` Hans van Kranenburg @ 2017-12-16 10:11 ` Ian Kumlien 1 sibling, 0 replies; 9+ messages in thread From: Ian Kumlien @ 2017-12-16 10:11 UTC (permalink / raw) To: Qu Wenruo; +Cc: Roman Mamedov, Btrfs BTRFS On Fri, Dec 15, 2017 at 9:53 AM, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote: > On 2017年12月15日 16:36, Ian Kumlien wrote: >> On Fri, Dec 15, 2017 at 6:34 AM, Roman Mamedov <rm@romanrm.net> wrote: >>> On Fri, 15 Dec 2017 01:39:03 +0100 >>> Ian Kumlien <ian.kumlien@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> Running a 4.14.3 kernel, this just happened, but there should have >>>> been another 20 gigs or so available. >>>> >>>> The filesystem seems fine after a reboot though >>> >>> What are your mount options, and can you show the output of "btrfs fi >>> df" and "btrfs fi us" for the filesystem? And what does >>> "cat /sys/block/sdb/queue/rotational" return. >> >> It's a btrfs raid1 mirror of two ssd:s [--8<--] (sorry if you got a reply before - dang html mails) > Well, at least no new chunk can be allocated. > > > In v4.15, Josef introduced a new inode reservation system, which I think > it would enhance metadata related reservation. Humm... maybe when it has been trough some rc:s ;) > If you're hitting the problem quite frequently, please consider to try > v4.15-rc* to see if it solves the problem. > > Despite of that, there should be no damage to your fs. > (except some unwritten data in buffer) Yeah figured as much - did a full defrag and recompress (with zstd this time) just to play with it > Thanks, > Qu > >> >> And as expected: >> cat /sys/block/sdb/queue/rotational >> 0 >> >>> I wonder if it's the same old "ssd allocation scheme" problem, and no >>> balancing done in a long time or at all. >> >> I had something similar happen on a laptop a while ago - took a while >> before i could get it back in order >> (in that case i think it was actually a oops --- it kept saying "no >> space left" and switched to read only even >> if you removed a lot of data, invalidated the space cache and so on) >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2017-12-16 10:21 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-12-15 0:39 [4.14.3] btrfs out of space error Ian Kumlien 2017-12-15 4:01 ` Chris Murphy 2017-12-16 10:05 ` Ian Kumlien 2017-12-15 5:34 ` Roman Mamedov 2017-12-15 8:36 ` Ian Kumlien 2017-12-15 8:53 ` Qu Wenruo 2017-12-15 23:07 ` Hans van Kranenburg 2017-12-16 10:20 ` Ian Kumlien 2017-12-16 10:11 ` Ian Kumlien
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.