* ENOSPC on mostly empty file system
@ 2014-09-09 14:29 Arnd Bergmann
2014-09-09 15:06 ` Holger Hoffstätte
2014-09-09 18:18 ` Arnd Bergmann
0 siblings, 2 replies; 11+ messages in thread
From: Arnd Bergmann @ 2014-09-09 14:29 UTC (permalink / raw)
To: linux-btrfs; +Cc: Chris Mason
Hi Chris,
As I mentioned at the kernel summit, I have a file system that I use mostly
for storing my one kernel git tree and occasionally some build trees
(those are normally on a tmpfs), and I have again run into the problem
where the file system is only partially full (I think 18% in this case)
but I am unable to create new files.
As you recommended, I have created an image using btrfs-image and
uploaded it to google drive for inspection.
I also played around with it some more. After removing a few small
files, I could create new files with up to 20-60MB again before hitting
ENOSPC. I then did a 'make clean' in all the object directories I had
and after that could fill the file system up to 100% by creating a
62GB file using the same command (dd if=/dev/zero of=x).
While probably unrelated, I also saw a few older 'hung task' messages
with btrfs call chains in dmesg and I'm adding them here in this mail
as well. The kernel messages are ten days old, but I did not reboot
in the meantime and I never saw them again before or after that.
The kernel is the stock Ubuntu 3.16.0 image from Ubuntu, I upgrade
to that after running an old 3.11 kernel that I never rebooted.
The 82 MB image file is at
https://drive.google.com/file/d/0B_XQwQ5KlfJAWDJfS1E0TG1CLTA/edit?usp=sharing
Arnd
[407459.475639] INFO: task BrowserBlocking:3282 blocked for more than 120 seconds.
[407459.475649] Not tainted 3.16.0-10-generic #15-Ubuntu
[407459.475652] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407459.475656] BrowserBlocking D ffff88042fc94800 0 3282 2580 0x00000000
[407459.475664] ffff88040d9f7db0 0000000000000086 ffff88041215b2a0 0000000000014800
[407459.475671] ffff88040d9f7fd8 0000000000014800 ffff88041215b2a0 ffff8800bbb12af8
[407459.475677] ffff8800bbb12afc ffff88041215b2a0 00000000ffffffff ffff8800bbb12b00
[407459.475683] Call Trace:
[407459.475699] [<ffffffff8177b609>] schedule_preempt_disabled+0x29/0x70
[407459.475707] [<ffffffff8177dba5>] __mutex_lock_slowpath+0xd5/0x1f0
[407459.475714] [<ffffffff8177dcdf>] mutex_lock+0x1f/0x30
[407459.475771] [<ffffffffc0546535>] btrfs_log_inode_parent+0xd5/0x550 [btrfs]
[407459.475804] [<ffffffffc0547822>] btrfs_log_dentry_safe+0x42/0x60 [btrfs]
[407459.475834] [<ffffffffc0517d64>] btrfs_sync_file+0x184/0x300 [btrfs]
[407459.475844] [<ffffffff81210461>] do_fsync+0x51/0x80
[407459.475849] [<ffffffff812106f0>] SyS_fsync+0x10/0x20
[407459.475856] [<ffffffff8177ff2d>] system_call_fastpath+0x1a/0x1f
[407459.475861] INFO: task BrowserBlocking:3283 blocked for more than 120 seconds.
[407459.475864] Not tainted 3.16.0-10-generic #15-Ubuntu
[407459.475866] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407459.475869] BrowserBlocking D ffff88042fd54800 0 3283 2580 0x00000000
[407459.475875] ffff88040bb13db0 0000000000000086 ffff88041215ef60 0000000000014800
[407459.475881] ffff88040bb13fd8 0000000000014800 ffff88041215ef60 ffff880429b43590
[407459.475886] ffff880429b43594 ffff88041215ef60 00000000ffffffff ffff880429b43598
[407459.475892] Call Trace:
[407459.475899] [<ffffffff8177b609>] schedule_preempt_disabled+0x29/0x70
[407459.475906] [<ffffffff8177dba5>] __mutex_lock_slowpath+0xd5/0x1f0
[407459.475912] [<ffffffff8177dcdf>] mutex_lock+0x1f/0x30
[407459.475943] [<ffffffffc0546895>] btrfs_log_inode_parent+0x435/0x550 [btrfs]
[407459.475976] [<ffffffffc0547822>] btrfs_log_dentry_safe+0x42/0x60 [btrfs]
[407459.476005] [<ffffffffc0517d64>] btrfs_sync_file+0x184/0x300 [btrfs]
[407459.476012] [<ffffffff81210461>] do_fsync+0x51/0x80
[407459.476018] [<ffffffff81210713>] SyS_fdatasync+0x13/0x20
[407459.476023] [<ffffffff8177ff2d>] system_call_fastpath+0x1a/0x1f
[407459.476099] INFO: task kworker/u65:6:2090685 blocked for more than 120 seconds.
[407459.476102] Not tainted 3.16.0-10-generic #15-Ubuntu
[407459.476104] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407459.476107] kworker/u65:6 D ffff88042fd14800 0 2090685 2 0x00000000
[407459.476131] Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
[407459.476134] ffff8806189ffba0 0000000000000046 ffff880428135100 0000000000014800
[407459.476140] ffff8806189fffd8 0000000000014800 ffff880428135100 ffff8806189ffcc8
[407459.476145] ffff8806189ffcc0 7fffffffffffffff ffff880428135100 0000000000000001
[407459.476151] Call Trace:
[407459.476158] [<ffffffff8177b089>] schedule+0x29/0x70
[407459.476164] [<ffffffff8177a349>] schedule_timeout+0x229/0x2a0
[407459.476171] [<ffffffff8108b026>] ? __queue_work+0x136/0x340
[407459.476177] [<ffffffff8108b2fa>] ? __queue_delayed_work+0xaa/0x1c0
[407459.476184] [<ffffffff8177c214>] wait_for_completion+0xb4/0x1e0
[407459.476191] [<ffffffff810a5400>] ? wake_up_state+0x20/0x20
[407459.476198] [<ffffffff812096c1>] writeback_inodes_sb_nr+0x81/0xb0
[407459.476221] [<ffffffffc04efb08>] flush_space+0x458/0x4f0 [btrfs]
[407459.476243] [<ffffffffc04ef617>] ? can_overcommit+0x67/0x100 [btrfs]
[407459.476266] [<ffffffffc04efd0c>] btrfs_async_reclaim_metadata_space+0x16c/0x1e0 [btrfs]
[407459.476273] [<ffffffff8108d812>] process_one_work+0x182/0x4e0
[407459.476278] [<ffffffff8108dbdb>] worker_thread+0x6b/0x6a0
[407459.476284] [<ffffffff8108db70>] ? process_one_work+0x4e0/0x4e0
[407459.476290] [<ffffffff81094a1b>] kthread+0xdb/0x100
[407459.476297] [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407459.476302] [<ffffffff8177fe7c>] ret_from_fork+0x7c/0xb0
[407459.476308] [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407579.528238] INFO: task kworker/u65:6:2090685 blocked for more than 120 seconds.
[407579.528247] Not tainted 3.16.0-10-generic #15-Ubuntu
[407579.528251] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407579.528255] kworker/u65:6 D ffff88042fd14800 0 2090685 2 0x00000000
[407579.528308] Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
[407579.528312] ffff8806189ffba0 0000000000000046 ffff880428135100 0000000000014800
[407579.528318] ffff8806189fffd8 0000000000014800 ffff880428135100 ffff8806189ffcc8
[407579.528324] ffff8806189ffcc0 7fffffffffffffff ffff880428135100 0000000000000001
[407579.528330] Call Trace:
[407579.528345] [<ffffffff8177b089>] schedule+0x29/0x70
[407579.528352] [<ffffffff8177a349>] schedule_timeout+0x229/0x2a0
[407579.528360] [<ffffffff8108b026>] ? __queue_work+0x136/0x340
[407579.528366] [<ffffffff8108b2fa>] ? __queue_delayed_work+0xaa/0x1c0
[407579.528373] [<ffffffff8177c214>] wait_for_completion+0xb4/0x1e0
[407579.528380] [<ffffffff810a5400>] ? wake_up_state+0x20/0x20
[407579.528388] [<ffffffff812096c1>] writeback_inodes_sb_nr+0x81/0xb0
[407579.528411] [<ffffffffc04efb08>] flush_space+0x458/0x4f0 [btrfs]
[407579.528434] [<ffffffffc04ef617>] ? can_overcommit+0x67/0x100 [btrfs]
[407579.528458] [<ffffffffc04efd0c>] btrfs_async_reclaim_metadata_space+0x16c/0x1e0 [btrfs]
[407579.528464] [<ffffffff8108d812>] process_one_work+0x182/0x4e0
[407579.528470] [<ffffffff8108dbdb>] worker_thread+0x6b/0x6a0
[407579.528475] [<ffffffff8108db70>] ? process_one_work+0x4e0/0x4e0
[407579.528481] [<ffffffff81094a1b>] kthread+0xdb/0x100
[407579.528488] [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407579.528494] [<ffffffff8177fe7c>] ret_from_fork+0x7c/0xb0
[407579.528501] [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407699.580802] INFO: task kworker/u65:6:2090685 blocked for more than 120 seconds.
[407699.580811] Not tainted 3.16.0-10-generic #15-Ubuntu
[407699.580815] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407699.580818] kworker/u65:6 D ffff88042fd14800 0 2090685 2 0x00000000
[407699.580872] Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
[407699.580877] ffff8806189ffba0 0000000000000046 ffff880428135100 0000000000014800
[407699.580883] ffff8806189fffd8 0000000000014800 ffff880428135100 ffff8806189ffcc8
[407699.580889] ffff8806189ffcc0 7fffffffffffffff ffff880428135100 0000000000000001
[407699.580895] Call Trace:
[407699.580910] [<ffffffff8177b089>] schedule+0x29/0x70
[407699.580917] [<ffffffff8177a349>] schedule_timeout+0x229/0x2a0
[407699.580925] [<ffffffff8108b026>] ? __queue_work+0x136/0x340
[407699.580930] [<ffffffff8108b2fa>] ? __queue_delayed_work+0xaa/0x1c0
[407699.580938] [<ffffffff8177c214>] wait_for_completion+0xb4/0x1e0
[407699.580945] [<ffffffff810a5400>] ? wake_up_state+0x20/0x20
[407699.580952] [<ffffffff812096c1>] writeback_inodes_sb_nr+0x81/0xb0
[407699.580976] [<ffffffffc04efb08>] flush_space+0x458/0x4f0 [btrfs]
[407699.580999] [<ffffffffc04ef617>] ? can_overcommit+0x67/0x100 [btrfs]
[407699.581022] [<ffffffffc04efd0c>] btrfs_async_reclaim_metadata_space+0x16c/0x1e0 [btrfs]
[407699.581029] [<ffffffff8108d812>] process_one_work+0x182/0x4e0
[407699.581034] [<ffffffff8108dbdb>] worker_thread+0x6b/0x6a0
[407699.581040] [<ffffffff8108db70>] ? process_one_work+0x4e0/0x4e0
[407699.581046] [<ffffffff81094a1b>] kthread+0xdb/0x100
[407699.581053] [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407699.581059] [<ffffffff8177fe7c>] ret_from_fork+0x7c/0xb0
[407699.581066] [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407819.633225] INFO: task BrowserBlocking:3282 blocked for more than 120 seconds.
[407819.633234] Not tainted 3.16.0-10-generic #15-Ubuntu
[407819.633238] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407819.633242] BrowserBlocking D ffff88042fc94800 0 3282 2580 0x00000000
[407819.633250] ffff88040d9f7db0 0000000000000086 ffff88041215b2a0 0000000000014800
[407819.633256] ffff88040d9f7fd8 0000000000014800 ffff88041215b2a0 ffff8800bbb12af8
[407819.633262] ffff8800bbb12afc ffff88041215b2a0 00000000ffffffff ffff8800bbb12b00
[407819.633269] Call Trace:
[407819.633285] [<ffffffff8177b609>] schedule_preempt_disabled+0x29/0x70
[407819.633292] [<ffffffff8177dba5>] __mutex_lock_slowpath+0xd5/0x1f0
[407819.633299] [<ffffffff8177dcdf>] mutex_lock+0x1f/0x30
[407819.633357] [<ffffffffc0546535>] btrfs_log_inode_parent+0xd5/0x550 [btrfs]
[407819.633390] [<ffffffffc0547822>] btrfs_log_dentry_safe+0x42/0x60 [btrfs]
[407819.633419] [<ffffffffc0517d64>] btrfs_sync_file+0x184/0x300 [btrfs]
[407819.633429] [<ffffffff81210461>] do_fsync+0x51/0x80
[407819.633435] [<ffffffff812106f0>] SyS_fsync+0x10/0x20
[407819.633440] [<ffffffff8177ff2d>] system_call_fastpath+0x1a/0x1f
[407819.633446] INFO: task BrowserBlocking:3284 blocked for more than 120 seconds.
[407819.633449] Not tainted 3.16.0-10-generic #15-Ubuntu
[407819.633451] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407819.633454] BrowserBlocking D ffff88042fc14800 0 3284 2580 0x00000000
[407819.633460] ffff880416d4fdb0 0000000000000086 ffff88041215e540 0000000000014800
[407819.633466] ffff880416d4ffd8 0000000000014800 ffff88041215e540 ffff880429b43590
[407819.633471] ffff880429b43594 ffff88041215e540 00000000ffffffff ffff880429b43598
[407819.633477] Call Trace:
[407819.633485] [<ffffffff8177b609>] schedule_preempt_disabled+0x29/0x70
[407819.633491] [<ffffffff8177dba5>] __mutex_lock_slowpath+0xd5/0x1f0
[407819.633498] [<ffffffff8177dcdf>] mutex_lock+0x1f/0x30
[407819.633530] [<ffffffffc0546895>] btrfs_log_inode_parent+0x435/0x550 [btrfs]
[407819.633561] [<ffffffffc0547822>] btrfs_log_dentry_safe+0x42/0x60 [btrfs]
[407819.633590] [<ffffffffc0517d64>] btrfs_sync_file+0x184/0x300 [btrfs]
[407819.633597] [<ffffffff81210461>] do_fsync+0x51/0x80
[407819.633603] [<ffffffff81210713>] SyS_fdatasync+0x13/0x20
[407819.633608] [<ffffffff8177ff2d>] system_call_fastpath+0x1a/0x1f
[407819.633687] INFO: task kworker/u65:6:2090685 blocked for more than 120 seconds.
[407819.633690] Not tainted 3.16.0-10-generic #15-Ubuntu
[407819.633693] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407819.633695] kworker/u65:6 D ffff88042fd14800 0 2090685 2 0x00000000
[407819.633719] Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs]
[407819.633722] ffff8806189ffba0 0000000000000046 ffff880428135100 0000000000014800
[407819.633728] ffff8806189fffd8 0000000000014800 ffff880428135100 ffff8806189ffcc8
[407819.633734] ffff8806189ffcc0 7fffffffffffffff ffff880428135100 0000000000000001
[407819.633739] Call Trace:
[407819.633747] [<ffffffff8177b089>] schedule+0x29/0x70
[407819.633753] [<ffffffff8177a349>] schedule_timeout+0x229/0x2a0
[407819.633760] [<ffffffff8108b026>] ? __queue_work+0x136/0x340
[407819.633765] [<ffffffff8108b2fa>] ? __queue_delayed_work+0xaa/0x1c0
[407819.633772] [<ffffffff8177c214>] wait_for_completion+0xb4/0x1e0
[407819.633779] [<ffffffff810a5400>] ? wake_up_state+0x20/0x20
[407819.633786] [<ffffffff812096c1>] writeback_inodes_sb_nr+0x81/0xb0
[407819.633809] [<ffffffffc04efb08>] flush_space+0x458/0x4f0 [btrfs]
[407819.633832] [<ffffffffc04ef617>] ? can_overcommit+0x67/0x100 [btrfs]
[407819.633854] [<ffffffffc04efd0c>] btrfs_async_reclaim_metadata_space+0x16c/0x1e0 [btrfs]
[407819.633861] [<ffffffff8108d812>] process_one_work+0x182/0x4e0
[407819.633866] [<ffffffff8108dbdb>] worker_thread+0x6b/0x6a0
[407819.633872] [<ffffffff8108db70>] ? process_one_work+0x4e0/0x4e0
[407819.633878] [<ffffffff81094a1b>] kthread+0xdb/0x100
[407819.633885] [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407819.633890] [<ffffffff8177fe7c>] ret_from_fork+0x7c/0xb0
[407819.633897] [<ffffffff81094940>] ? kthread_create_on_node+0x1c0/0x1c0
[407819.633907] INFO: task find:2091978 blocked for more than 120 seconds.
[407819.633910] Not tainted 3.16.0-10-generic #15-Ubuntu
[407819.633913] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[407819.633915] find D ffff88042fc94800 0 2091978 2091977 0x00000000
[407819.633921] ffff8805c8c83ab0 0000000000000086 ffff880271929e60 0000000000014800
[407819.633926] ffff8805c8c83fd8 0000000000014800 ffff880271929e60 ffff8805c8c83bd8
[407819.633932] ffff8805c8c83bd0 7fffffffffffffff ffff880271929e60 0000000000000001
[407819.633937] Call Trace:
[407819.633944] [<ffffffff8177b089>] schedule+0x29/0x70
[407819.633950] [<ffffffff8177a349>] schedule_timeout+0x229/0x2a0
[407819.633955] [<ffffffff8108b026>] ? __queue_work+0x136/0x340
[407819.633976] [<ffffffffc04eb215>] ? block_rsv_release_bytes+0x115/0x180 [btrfs]
[407819.633981] [<ffffffff8108b2fa>] ? __queue_delayed_work+0xaa/0x1c0
[407819.633988] [<ffffffff8177c214>] wait_for_completion+0xb4/0x1e0
[407819.633994] [<ffffffff810a5400>] ? wake_up_state+0x20/0x20
[407819.633999] [<ffffffff812096c1>] writeback_inodes_sb_nr+0x81/0xb0
[407819.634023] [<ffffffffc04efb08>] flush_space+0x458/0x4f0 [btrfs]
[407819.634046] [<ffffffffc04ef617>] ? can_overcommit+0x67/0x100 [btrfs]
[407819.634068] [<ffffffffc04eff83>] reserve_metadata_bytes+0x203/0x490 [btrfs]
[407819.634075] [<ffffffff8177dcd2>] ? mutex_lock+0x12/0x30
[407819.634099] [<ffffffffc04f0786>] btrfs_block_rsv_add+0x26/0x50 [btrfs]
[407819.634126] [<ffffffffc0507e03>] start_transaction+0x453/0x590 [btrfs]
[407819.634154] [<ffffffffc0507f5b>] btrfs_start_transaction+0x1b/0x20 [btrfs]
[407819.634182] [<ffffffffc050d4a8>] btrfs_dirty_inode+0xb8/0xf0 [btrfs]
[407819.634210] [<ffffffffc050d55f>] btrfs_update_time+0x7f/0xc0 [btrfs]
[407819.634217] [<ffffffff811fb285>] update_time+0x25/0xd0
[407819.634223] [<ffffffff811fb409>] touch_atime+0xd9/0x140
[407819.634228] [<ffffffff811f3b60>] iterate_dir+0xe0/0x130
[407819.634234] [<ffffffff811f3fb9>] SyS_getdents+0x89/0x100
[407819.634238] [<ffffffff811f3c80>] ? fillonedir+0xd0/0xd0
[407819.634244] [<ffffffff8177ff2d>] system_call_fastpath+0x1a/0x1f
[408133.512919] audit: type=1400 audit(1409380699.459:52): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/cups/backend/cups-pdf" pid=2092471 comm="apparmor_parser"
[408133.512930] audit: type=1400 audit(1409380699.459:53): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/cupsd" pid=2092471 comm="apparmor_parser"
[408133.513460] audit: type=1400 audit(1409380699.459:54): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/sbin/cupsd" pid=2092471 comm="apparmor_parser"
[408299.843229] INFO: task BrowserBlocking:3282 blocked for more than 120 seconds.
[408299.843238] Not tainted 3.16.0-10-generic #15-Ubuntu
[408299.843241] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[408299.843245] BrowserBlocking D ffff88042fc14800 0 3282 2580 0x00000000
[408299.843253] ffff88040d9f7db0 0000000000000086 ffff88041215b2a0 0000000000014800
[408299.843260] ffff88040d9f7fd8 0000000000014800 ffff88041215b2a0 ffff880429b43590
[408299.843266] ffff880429b43594 ffff88041215b2a0 00000000ffffffff ffff880429b43598
[408299.843272] Call Trace:
[408299.843288] [<ffffffff8177b609>] schedule_preempt_disabled+0x29/0x70
[408299.843296] [<ffffffff8177dba5>] __mutex_lock_slowpath+0xd5/0x1f0
[408299.843303] [<ffffffff8177dcdf>] mutex_lock+0x1f/0x30
[408299.843360] [<ffffffffc0546895>] btrfs_log_inode_parent+0x435/0x550 [btrfs]
[408299.843393] [<ffffffffc0547822>] btrfs_log_dentry_safe+0x42/0x60 [btrfs]
[408299.843423] [<ffffffffc0517d64>] btrfs_sync_file+0x184/0x300 [btrfs]
[408299.843432] [<ffffffff81210461>] do_fsync+0x51/0x80
[408299.843438] [<ffffffff812106f0>] SyS_fsync+0x10/0x20
[408299.843444] [<ffffffff8177ff2d>] system_call_fastpath+0x1a/0x1f
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ENOSPC on mostly empty file system
2014-09-09 14:29 ENOSPC on mostly empty file system Arnd Bergmann
@ 2014-09-09 15:06 ` Holger Hoffstätte
2014-09-09 15:12 ` Holger Hoffstätte
2014-09-09 18:18 ` Arnd Bergmann
1 sibling, 1 reply; 11+ messages in thread
From: Holger Hoffstätte @ 2014-09-09 15:06 UTC (permalink / raw)
To: linux-btrfs
On Tue, 09 Sep 2014 16:29:05 +0200, Arnd Bergmann wrote:
> As I mentioned at the kernel summit, I have a file system that I use mostly
> for storing my one kernel git tree and occasionally some build trees
> (those are normally on a tmpfs), and I have again run into the problem
> where the file system is only partially full (I think 18% in this case)
> but I am unable to create new files.
You didn't give us any "btrfs fs show" or "btrfs fs df" outpu to look at,
so chances are that a lot of empty data chunks have made themselves
comfortable and prevent new metadata chunks from being allocated.
The usual fix is to run balance with -dusage=x, where 0 <= x <= ~20.
> The 82 MB image file is at
> https://drive.google.com/file/d/0B_XQwQ5KlfJAWDJfS1E0TG1CLTA/edit?usp=sharing
This is >780 MB..
-h
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ENOSPC on mostly empty file system
2014-09-09 15:06 ` Holger Hoffstätte
@ 2014-09-09 15:12 ` Holger Hoffstätte
0 siblings, 0 replies; 11+ messages in thread
From: Holger Hoffstätte @ 2014-09-09 15:12 UTC (permalink / raw)
To: linux-btrfs
On Tue, 09 Sep 2014 15:06:19 +0000, Holger Hoffstätte wrote:
> You didn't give us any "btrfs fs show" or "btrfs fs df" outpu to look at,
Of course "fi", not "fs"..
-h
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ENOSPC on mostly empty file system
2014-09-09 14:29 ENOSPC on mostly empty file system Arnd Bergmann
2014-09-09 15:06 ` Holger Hoffstätte
@ 2014-09-09 18:18 ` Arnd Bergmann
2014-09-09 19:49 ` Clemens Eisserer
1 sibling, 1 reply; 11+ messages in thread
From: Arnd Bergmann @ 2014-09-09 18:18 UTC (permalink / raw)
To: linux-btrfs; +Cc: Chris Mason
On Tuesday 09 September 2014 16:29:05 Arnd Bergmann wrote:
>
> I also played around with it some more. After removing a few small
> files, I could create new files with up to 20-60MB again before hitting
> ENOSPC. I then did a 'make clean' in all the object directories I had
> and after that could fill the file system up to 100% by creating a
> 62GB file using the same command (dd if=/dev/zero of=x).
Ok, one more data point: I was accidentally building randconfig kernels
in the same partition all the time, meaning I had lots of traffic with
small files using the same names all over all the time. After I got
the file system back to a usable state today, building some more randconfig
kernels broke it in the same way.
Arnd
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ENOSPC on mostly empty file system
2014-09-09 18:18 ` Arnd Bergmann
@ 2014-09-09 19:49 ` Clemens Eisserer
2014-09-09 19:56 ` Hugo Mills
2014-09-09 21:49 ` Arnd Bergmann
0 siblings, 2 replies; 11+ messages in thread
From: Clemens Eisserer @ 2014-09-09 19:49 UTC (permalink / raw)
To: Arnd Bergmann, linux-btrfs
Hi Arnd,
> Ok, one more data point:
Why don't you provide the data point you were specifically asked for,
"btrfs fi df" ;)
Regards, Clemens
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ENOSPC on mostly empty file system
2014-09-09 19:49 ` Clemens Eisserer
@ 2014-09-09 19:56 ` Hugo Mills
2014-09-09 21:49 ` Arnd Bergmann
1 sibling, 0 replies; 11+ messages in thread
From: Hugo Mills @ 2014-09-09 19:56 UTC (permalink / raw)
To: Clemens Eisserer; +Cc: Arnd Bergmann, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 619 bytes --]
On Tue, Sep 09, 2014 at 09:49:12PM +0200, Clemens Eisserer wrote:
> Hi Arnd,
>
> > Ok, one more data point:
>
> Why don't you provide the data point you were specifically asked for,
> "btrfs fi df" ;)
btrfs fi show is important as well -- it's hard to work out the
state of the FS from just one of them.
Hugo.
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- You can get more with a kind word and a two-by-four than you ---
can with just a kind word.
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ENOSPC on mostly empty file system
2014-09-09 19:49 ` Clemens Eisserer
2014-09-09 19:56 ` Hugo Mills
@ 2014-09-09 21:49 ` Arnd Bergmann
2014-09-09 21:57 ` Hugo Mills
2014-09-09 22:23 ` Holger Hoffstätte
1 sibling, 2 replies; 11+ messages in thread
From: Arnd Bergmann @ 2014-09-09 21:49 UTC (permalink / raw)
To: Clemens Eisserer; +Cc: linux-btrfs
On Tuesday 09 September 2014 21:49:12 Clemens Eisserer wrote:
> Hi Arnd,
>
> > Ok, one more data point:
>
> Why don't you provide the data point you were specifically asked for,
> "btrfs fi df" ;)
I've cleaned it up again already. At the moment, it's working fine,
with this data:
Data: total=65.11GB, used=7.41GB
System, DUP: total=8.00MB, used=12.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=1.00GB, used=799.54MB
Metadata: total=8.00MB, used=0.00
: total=184.00MB, used=0.00
I also realized that the file I uploaded was the uncompressed 800MB
version, not the compressed 82MB file, which is now at
https://drive.google.com/file/d/0B_XQwQ5KlfJAYzY5c1lJRFVQNEE/edit?usp=sharing
Sorry about that.
Ok, now I'm in the bad state again (after running a 'make allmodconfig'
kernel build:
Label: none uuid: 1d88cccb-3d0e-42d9-8252-a226dc5c2e47
Total devices 1 FS bytes used 8.79GB
devid 1 size 67.14GB used 67.14GB path /dev/sdc6
Data: total=65.11GB, used=7.99GB
System, DUP: total=8.00MB, used=12.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=1.00GB, used=821.48MB
Metadata: total=8.00MB, used=0.00
: total=200.00MB, used=0.00
Not much extra space used up. Also, a bit from my last shell session
after getting into this state:
arnd@wuerfel:~/arm-soc$ make -skj40
../arch/arm/mach-mmp/devices.c:81:21: warning: 'u2o_get' defined but not used [-Wunused-function]
static unsigned int u2o_get(void __iomem *base, unsigned int offset)
^
../arch/arm/mach-mmp/devices.c:86:13: warning: 'u2o_set' defined but not used [-Wunused-function]
static void u2o_set(void __iomem *base, unsigned int offset,
^
../arch/arm/mach-mmp/devices.c:97:13: warning: 'u2o_clear' defined but not used [-Wunused-function]
static void u2o_clear(void __iomem *base, unsigned int offset,
^
../arch/arm/mach-mmp/devices.c:108:13: warning: 'u2o_write' defined but not used [-Wunused-function]
static void u2o_write(void __iomem *base, unsigned int offset,
^
mv: cannot move 'security/selinux/ss/.tmp_conditional.o' to 'security/selinux/ss/conditional.o': No space left on device
make[4]: *** [security/selinux/ss/conditional.o] Error 1
mv: cannot move 'kernel/power/.snapshot.o.tmp' to 'kernel/power/.snapshot.o.cmd': No space left on device
make[4]: *** [kernel/power/snapshot.o] Error 1
mv: cannot move 'drivers/ata/.pata_hpt366.o.tmp' to 'drivers/ata/.pata_hpt366.o.cmd': No space left on device
make[4]: *** [drivers/ata/pata_hpt366.o] Error 1
mv: cannot move 'kernel/irq/.proc.o.tmp' to 'kernel/irq/.proc.o.cmd': No space left on device
make[4]: *** [kernel/irq/proc.o] Error 1
...
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: writing to 'x': No space left on device
416642+0 records in
416641+0 records out
213320192 bytes (213 MB) copied, 2.97814 s, 71.6 MB/s
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: opening 'x': No space left on device
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: opening 'x': No space left on device
arnd@wuerfel:~/arm-soc$ rm x
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: writing to 'x': No space left on device
366053+0 records in
366052+0 records out
187418624 bytes (187 MB) copied, 3.04426 s, 61.6 MB/s
arnd@wuerfel:~/arm-soc$ rm x
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: writing to 'x': No space left on device
363482+0 records in
363481+0 records out
186102272 bytes (186 MB) copied, 2.80644 s, 66.3 MB/s
arnd@wuerfel:~/arm-soc$ rm x
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: writing to 'x': No space left on device
366677+0 records in
366676+0 records out
187738112 bytes (188 MB) copied, 2.86939 s, 65.4 MB/s
arnd@wuerfel:~/arm-soc$ sync
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: opening 'x': No space left on device
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: opening 'x': No space left on device
arnd@wuerfel:~/arm-soc$ dd if=/dev/zero of=x
dd: opening 'x': No space left on device
I believe that there were no other processes writing to this partition
at the time.
Arnd
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ENOSPC on mostly empty file system
2014-09-09 21:49 ` Arnd Bergmann
@ 2014-09-09 21:57 ` Hugo Mills
2014-09-09 22:11 ` Roman Mamedov
2014-09-09 22:12 ` Arnd Bergmann
2014-09-09 22:23 ` Holger Hoffstätte
1 sibling, 2 replies; 11+ messages in thread
From: Hugo Mills @ 2014-09-09 21:57 UTC (permalink / raw)
To: Arnd Bergmann; +Cc: Clemens Eisserer, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1962 bytes --]
On Tue, Sep 09, 2014 at 11:49:10PM +0200, Arnd Bergmann wrote:
> Ok, now I'm in the bad state again (after running a 'make allmodconfig'
> kernel build:
>
> Label: none uuid: 1d88cccb-3d0e-42d9-8252-a226dc5c2e47
> Total devices 1 FS bytes used 8.79GB
> devid 1 size 67.14GB used 67.14GB path /dev/sdc6
All the space on the FS has been allocated to some purpose or other.
> Data: total=65.11GB, used=7.99GB
Here, you have 65 GiB allocated to data, but only 8 GiB of that
used. The FS won't automatically free up any of that (yet -- it's one
of the project ideas).
> System, DUP: total=8.00MB, used=12.00KB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=1.00GB, used=821.48MB
Here, you're running close to full with metadata -- the FS needs
some space to write new copies of metadata block in order to modify
anything. It can't get enough space to do that, because there's
nowhere for any more metadata allocation to come from (because it's
all allocated --see my first comment).
So... you need to free up some data chunks. You can do this with:
# btrfs balance start -dusage=5 /mountpoint
Take a look at the output of btrfs fi df and btrfs fi show
afterwards, and see how much the Data allocation has reduced by, and
how much unallocated space you have left afterwards. You may want to
increase the number in the above balance command to some higher value,
to free up even more chunks (it limits the balance to chunks less than
n% full -- so the command above will only touch chunks with 5% actual
data or less). This is in the FAQ.
Hugo.
> Metadata: total=8.00MB, used=0.00
> : total=200.00MB, used=0.00
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- Comic Sans goes into a bar, and the barman says, "We don't ---
serve your type here."
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ENOSPC on mostly empty file system
2014-09-09 21:57 ` Hugo Mills
@ 2014-09-09 22:11 ` Roman Mamedov
2014-09-09 22:12 ` Arnd Bergmann
1 sibling, 0 replies; 11+ messages in thread
From: Roman Mamedov @ 2014-09-09 22:11 UTC (permalink / raw)
To: Hugo Mills; +Cc: Arnd Bergmann, Clemens Eisserer, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1279 bytes --]
On Tue, 9 Sep 2014 22:57:25 +0100
Hugo Mills <hugo@carfax.org.uk> wrote:
> On Tue, Sep 09, 2014 at 11:49:10PM +0200, Arnd Bergmann wrote:
> > Ok, now I'm in the bad state again (after running a 'make allmodconfig'
> > kernel build:
> >
> > Label: none uuid: 1d88cccb-3d0e-42d9-8252-a226dc5c2e47
> > Total devices 1 FS bytes used 8.79GB
> > devid 1 size 67.14GB used 67.14GB path /dev/sdc6
>
> All the space on the FS has been allocated to some purpose or other.
>
> > Data: total=65.11GB, used=7.99GB
>
> Here, you have 65 GiB allocated to data, but only 8 GiB of that
> used. The FS won't automatically free up any of that (yet -- it's one
> of the project ideas).
If you find yourself constantly running into this situation, rather than
incorporating periodical btrfs 'balances' into your scripts, you could consider
going with the "mixed" allocation mode of the FS, using the -M option of mkfs.
Seeing as this is not the largest-sized partition in the world in the first
place, the mixed mode would allow for a more efficient and flexible space
allocation on it; at the cost of some performance penalty (you could compare
your compilation times, but I don't think it's too significant).
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ENOSPC on mostly empty file system
2014-09-09 21:57 ` Hugo Mills
2014-09-09 22:11 ` Roman Mamedov
@ 2014-09-09 22:12 ` Arnd Bergmann
1 sibling, 0 replies; 11+ messages in thread
From: Arnd Bergmann @ 2014-09-09 22:12 UTC (permalink / raw)
To: Hugo Mills; +Cc: Clemens Eisserer, linux-btrfs
On Tuesday 09 September 2014 22:57:25 Hugo Mills wrote:
> On Tue, Sep 09, 2014 at 11:49:10PM +0200, Arnd Bergmann wrote:
> > Ok, now I'm in the bad state again (after running a 'make allmodconfig'
> > kernel build:
> >
> > Label: none uuid: 1d88cccb-3d0e-42d9-8252-a226dc5c2e47
> > Total devices 1 FS bytes used 8.79GB
> > devid 1 size 67.14GB used 67.14GB path /dev/sdc6
>
> All the space on the FS has been allocated to some purpose or other.
>
> > Data: total=65.11GB, used=7.99GB
>
> Here, you have 65 GiB allocated to data, but only 8 GiB of that
> used. The FS won't automatically free up any of that (yet -- it's one
> of the project ideas).
Ok, I see.
> So... you need to free up some data chunks. You can do this with:
>
> # btrfs balance start -dusage=5 /mountpoint
>
> Take a look at the output of btrfs fi df and btrfs fi show
> afterwards, and see how much the Data allocation has reduced by, and
> how much unallocated space you have left afterwards. You may want to
> increase the number in the above balance command to some higher value,
> to free up even more chunks (it limits the balance to chunks less than
> n% full -- so the command above will only touch chunks with 5% actual
> data or less). This is in the FAQ.
This is the output afterwards:
$ time sudo btrfs fi balance start -dusage=5 /git/
Done, had to relocate 44 out of 70 chunks
real 0m22.026s
user 0m0.014s
sys 0m8.302s
$ sudo btrfs fi show /dev/sdc6
Label: none uuid: 1d88cccb-3d0e-42d9-8252-a226dc5c2e47
Total devices 1 FS bytes used 8.79GB
devid 1 size 67.14GB used 23.14GB path /dev/sdc6
$ sudo btrfs fi df /git
Data: total=21.01GB, used=7.99GB
System, DUP: total=8.00MB, used=12.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=1.05GB, used=821.32MB
Metadata: total=8.00MB, used=0.00
: total=200.00MB, used=0.00
I'm pretty sure I read the FAQ one of the last times this happened,
but I thought it would only apply to mostly full file systems.
I can usually recover by moving all the data to tmpfs and doing
a new mkfs before moving the data back, and that normally lasts
for a few months before I run out of space again.
I'll try to see how long it takes until the problem comes back
now.
Arnd
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: ENOSPC on mostly empty file system
2014-09-09 21:49 ` Arnd Bergmann
2014-09-09 21:57 ` Hugo Mills
@ 2014-09-09 22:23 ` Holger Hoffstätte
1 sibling, 0 replies; 11+ messages in thread
From: Holger Hoffstätte @ 2014-09-09 22:23 UTC (permalink / raw)
To: linux-btrfs
On Tue, 09 Sep 2014 23:49:10 +0200, Arnd Bergmann wrote:
> Label: none uuid: 1d88cccb-3d0e-42d9-8252-a226dc5c2e47
> Total devices 1 FS bytes used 8.79GB
> devid 1 size 67.14GB used 67.14GB path /dev/sdc6
^^^^^^^^^^^^^^^^^^^^ *ding ding ding*
The term "used" is confusing here and really should be called
"allocated", since it's more than the number of actually used
bytes. Note that df shows that you "have" ~66GB data and ~1GB
metadata (giving approx. ~67GB), despite the fact that of those
65GB data allocated only ~8GB are actually used.
However all these reserved data chunks are preventing the metadata
space from expanding, which is pretty full:
Metadata, DUP: total=1.00GB, used=799.54MB
Here "total" also should be "allocated/reserved" and is 1GB, but
btrfs feels it's necessary to allocate a new metadata chunk
(which would also increase "total") and fails.
Running balance with -dusage=x will garbage-collect the reserved but
mostly-empty data chunks with "up to x" percent actual usage.
-h
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2014-09-09 22:23 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-09-09 14:29 ENOSPC on mostly empty file system Arnd Bergmann
2014-09-09 15:06 ` Holger Hoffstätte
2014-09-09 15:12 ` Holger Hoffstätte
2014-09-09 18:18 ` Arnd Bergmann
2014-09-09 19:49 ` Clemens Eisserer
2014-09-09 19:56 ` Hugo Mills
2014-09-09 21:49 ` Arnd Bergmann
2014-09-09 21:57 ` Hugo Mills
2014-09-09 22:11 ` Roman Mamedov
2014-09-09 22:12 ` Arnd Bergmann
2014-09-09 22:23 ` Holger Hoffstätte
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.