BTRFS hang with 3.16-rc5

BTRFS hang with 3.16-rc5

[Martin Steigerwald]

Hi!

While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several days of
usage, with 3-16-rc5 I had a hang again. Less than a hour since booting it.

Since the hang bug I and others had with 3.15 and upto 3.16-rc2 usually
didn´t happen that quickly after boot and since backtrace looks a bit
different from what I have in memory, I post this in a new thread.
See thread "Blocked tasks on 3.15.1" for a discussion of previous hang
issues.

I lost a mail I was about to send with kmail.

merkaba:/boot> ps aux | grep D
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       162  0.0  0.0      0     0 ?        D    15:16   0:02 [kworker/u8:4]
root       166  0.0  0.0      0     0 ?        D    15:16   0:01 [kworker/u8:8]
root       694  0.0  0.0      0     0 ?        D    15:16   0:03 [btrfs-transacti]
martin    2230  0.1  0.3 407396 51868 ?        DN   15:16   0:04 /usr/local/bin/akonadi_baloo_indexer --identifier akonadi_baloo_indexer
ms        2720  0.0  0.2 323316 33936 ?        DN   15:20   0:02 /usr/local/bin/baloo_file
ms        3025  0.1  0.4 823300 66296 ?        D    15:21   0:04 /usr/bin/owncloud -session 10cec7d36b000140534103000000249840025_1405343620_420735
ms        3790  7.4  2.3 1750408 386120 ?      Dl   15:30   4:14 /usr/bin/iceweasel
root      3901  0.0  0.0      0     0 ?        D    15:31   0:01 [kworker/u8:0]
root      3956  0.0  0.0      0     0 ?        D    15:45   0:00 [kworker/u8:7]
[…]




Jul 14 16:00:13 merkaba kernel: [ 2640.529614] INFO: task kworker/u8:4:162 blocked for more than 120 seconds.
Jul 14 16:00:13 merkaba kernel: [ 2640.529627]       Tainted: G           O  3.16.0-rc5-tp520 #4
Jul 14 16:00:13 merkaba kernel: [ 2640.529631] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 14 16:00:13 merkaba kernel: [ 2640.529635] kworker/u8:4    D ffff880405cab660     0   162      2 0x00000000
Jul 14 16:00:13 merkaba kernel: [ 2640.529720] Workqueue: btrfs-flush_delalloc normal_work_helper [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.529725]  ffff8804055d3b18 0000000000000002 ffff880407f66240 ffff8804055d3fd8
Jul 14 16:00:13 merkaba kernel: [ 2640.529734]  ffff880405cab120 0000000000013080 ffff880405cab120 ffff88041e313080
Jul 14 16:00:13 merkaba kernel: [ 2640.529741]  ffff88041e5e7b68 ffff8804055d3bb0 0000000000000002 ffffffff810daffc
Jul 14 16:00:13 merkaba kernel: [ 2640.529749] Call Trace:
Jul 14 16:00:13 merkaba kernel: [ 2640.529762]  [<ffffffff810daffc>] ? wait_on_page_read+0x37/0x37
Jul 14 16:00:13 merkaba kernel: [ 2640.529772]  [<ffffffff81470c70>] schedule+0x64/0x66
Jul 14 16:00:13 merkaba kernel: [ 2640.529779]  [<ffffffff81470df7>] io_schedule+0x57/0x76
Jul 14 16:00:13 merkaba kernel: [ 2640.529785]  [<ffffffff810db005>] sleep_on_page+0x9/0xd
Jul 14 16:00:13 merkaba kernel: [ 2640.529792]  [<ffffffff814711d7>] __wait_on_bit_lock+0x41/0x85
Jul 14 16:00:13 merkaba kernel: [ 2640.529799]  [<ffffffff810db0c2>] __lock_page+0x70/0x7c
Jul 14 16:00:13 merkaba kernel: [ 2640.529809]  [<ffffffff81070f79>] ? autoremove_wake_function+0x2f/0x2f
Jul 14 16:00:13 merkaba kernel: [ 2640.529862]  [<ffffffffc0362590>] lock_page+0x1e/0x21 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.529906]  [<ffffffffc0362590>] ? lock_page+0x1e/0x21 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.529953]  [<ffffffffc036686d>] extent_write_cache_pages.isra.21.constprop.42+0x1a7/0x2d9 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.529964]  [<ffffffff81067fd7>] ? account_entity_enqueue+0x55/0x86
Jul 14 16:00:13 merkaba kernel: [ 2640.529972]  [<ffffffff810680c7>] ? __enqueue_entity+0x67/0x69
Jul 14 16:00:13 merkaba kernel: [ 2640.530016]  [<ffffffffc0366d95>] extent_writepages+0x46/0x57 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.530061]  [<ffffffffc03510a2>] ? btrfs_submit_direct+0x3ef/0x3ef [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.530103]  [<ffffffffc034f60f>] btrfs_writepages+0x23/0x25 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.530109]  [<ffffffff810e5381>] do_writepages+0x1b/0x24
Jul 14 16:00:13 merkaba kernel: [ 2640.530116]  [<ffffffff810dc6ad>] __filemap_fdatawrite_range+0x50/0x52
Jul 14 16:00:13 merkaba kernel: [ 2640.530122]  [<ffffffff810dcfd6>] filemap_flush+0x17/0x19
Jul 14 16:00:13 merkaba kernel: [ 2640.530163]  [<ffffffffc0351de9>] btrfs_run_delalloc_work+0x2e/0x64 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.530211]  [<ffffffffc0372350>] normal_work_helper+0xdf/0x224 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.530221]  [<ffffffff81052d7c>] process_one_work+0x16f/0x2b8
Jul 14 16:00:13 merkaba kernel: [ 2640.530229]  [<ffffffff81053626>] worker_thread+0x27b/0x32e
Jul 14 16:00:13 merkaba kernel: [ 2640.530238]  [<ffffffff810533ab>] ? cancel_delayed_work_sync+0x10/0x10
Jul 14 16:00:13 merkaba kernel: [ 2640.530245]  [<ffffffff81058002>] kthread+0xb2/0xba
Jul 14 16:00:13 merkaba kernel: [ 2640.530253]  [<ffffffff81470000>] ? iommu_prepare_identity_map+0x1d/0x19d
Jul 14 16:00:13 merkaba kernel: [ 2640.530260]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 14 16:00:13 merkaba kernel: [ 2640.530267]  [<ffffffff81473bec>] ret_from_fork+0x7c/0xb0
Jul 14 16:00:13 merkaba kernel: [ 2640.530274]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 14 16:00:13 merkaba kernel: [ 2640.530280] INFO: task kworker/u8:8:166 blocked for more than 120 seconds.
Jul 14 16:00:13 merkaba kernel: [ 2640.530285]       Tainted: G           O  3.16.0-rc5-tp520 #4
Jul 14 16:00:13 merkaba kernel: [ 2640.530288] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 14 16:00:13 merkaba kernel: [ 2640.530292] kworker/u8:8    D ffff880405b84ef0     0   166      2 0x00000000
Jul 14 16:00:13 merkaba kernel: [ 2640.530340] Workqueue: btrfs-flush_delalloc normal_work_helper [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.530344]  ffff880405643b18 0000000000000002 ffff880407f649b0 ffff880405643fd8
Jul 14 16:00:13 merkaba kernel: [ 2640.530351]  ffff880405b849b0 0000000000013080 ffff880405b849b0 ffff88041e293080
Jul 14 16:00:13 merkaba kernel: [ 2640.530358]  ffff88041e5d0168 ffff880405643bb0 0000000000000002 ffffffff810daffc
Jul 14 16:00:13 merkaba kernel: [ 2640.530366] Call Trace:
Jul 14 16:00:13 merkaba kernel: [ 2640.530373]  [<ffffffff810daffc>] ? wait_on_page_read+0x37/0x37
Jul 14 16:00:13 merkaba kernel: [ 2640.530381]  [<ffffffff81470c70>] schedule+0x64/0x66
Jul 14 16:00:13 merkaba kernel: [ 2640.530388]  [<ffffffff81470df7>] io_schedule+0x57/0x76
Jul 14 16:00:13 merkaba kernel: [ 2640.530394]  [<ffffffff810db005>] sleep_on_page+0x9/0xd
Jul 14 16:00:13 merkaba kernel: [ 2640.530401]  [<ffffffff814711d7>] __wait_on_bit_lock+0x41/0x85
Jul 14 16:00:13 merkaba kernel: [ 2640.530407]  [<ffffffff810db0c2>] __lock_page+0x70/0x7c
Jul 14 16:00:13 merkaba kernel: [ 2640.530415]  [<ffffffff81070f79>] ? autoremove_wake_function+0x2f/0x2f
Jul 14 16:00:13 merkaba kernel: [ 2640.530458]  [<ffffffffc0362590>] lock_page+0x1e/0x21 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.530498]  [<ffffffffc0362590>] ? lock_page+0x1e/0x21 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.530542]  [<ffffffffc036686d>] extent_write_cache_pages.isra.21.constprop.42+0x1a7/0x2d9 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.530618]  [<ffffffffc0366d95>] extent_writepages+0x46/0x57 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.530683]  [<ffffffffc03510a2>] ? btrfs_submit_direct+0x3ef/0x3ef [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.530748]  [<ffffffffc034f60f>] btrfs_writepages+0x23/0x25 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.530772]  [<ffffffff810e5381>] do_writepages+0x1b/0x24
Jul 14 16:00:13 merkaba kernel: [ 2640.530799]  [<ffffffff810dc6ad>] __filemap_fdatawrite_range+0x50/0x52
Jul 14 16:00:13 merkaba kernel: [ 2640.530829]  [<ffffffff810dcfd6>] filemap_flush+0x17/0x19
Jul 14 16:00:13 merkaba kernel: [ 2640.530890]  [<ffffffffc0351de9>] btrfs_run_delalloc_work+0x2e/0x64 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.530953]  [<ffffffffc0372350>] normal_work_helper+0xdf/0x224 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.530979]  [<ffffffff81052d7c>] process_one_work+0x16f/0x2b8
Jul 14 16:00:13 merkaba kernel: [ 2640.531000]  [<ffffffff81053626>] worker_thread+0x27b/0x32e
Jul 14 16:00:13 merkaba kernel: [ 2640.531008]  [<ffffffff810533ab>] ? cancel_delayed_work_sync+0x10/0x10
Jul 14 16:00:13 merkaba kernel: [ 2640.531015]  [<ffffffff81058002>] kthread+0xb2/0xba
Jul 14 16:00:13 merkaba kernel: [ 2640.531023]  [<ffffffff81470000>] ? iommu_prepare_identity_map+0x1d/0x19d
Jul 14 16:00:13 merkaba kernel: [ 2640.531030]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 14 16:00:13 merkaba kernel: [ 2640.531036]  [<ffffffff81473bec>] ret_from_fork+0x7c/0xb0
Jul 14 16:00:13 merkaba kernel: [ 2640.531043]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 14 16:00:13 merkaba kernel: [ 2640.531064] INFO: task btrfs-transacti:694 blocked for more than 120 seconds.
Jul 14 16:00:13 merkaba kernel: [ 2640.531069]       Tainted: G           O  3.16.0-rc5-tp520 #4
Jul 14 16:00:13 merkaba kernel: [ 2640.531072] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 14 16:00:13 merkaba kernel: [ 2640.531076] btrfs-transacti D ffff88040231b660     0   694      2 0x00000000
Jul 14 16:00:13 merkaba kernel: [ 2640.531083]  ffff880036d87c80 0000000000000002 ffff880407fc0000 ffff880036d87fd8
Jul 14 16:00:13 merkaba kernel: [ 2640.531091]  ffff88040231b120 0000000000013080 ffff88040231b120 7fffffffffffffff
Jul 14 16:00:13 merkaba kernel: [ 2640.531099]  ffff8800beee1278 0000000000000002 ffffffff81470270 ffff8800beee1270
Jul 14 16:00:13 merkaba kernel: [ 2640.531106] Call Trace:
Jul 14 16:00:13 merkaba kernel: [ 2640.531115]  [<ffffffff81470270>] ? michael_mic.part.6+0x1e/0x1e
Jul 14 16:00:13 merkaba kernel: [ 2640.531122]  [<ffffffff81470c70>] schedule+0x64/0x66
Jul 14 16:00:13 merkaba kernel: [ 2640.531129]  [<ffffffff8147029f>] schedule_timeout+0x2f/0x114
Jul 14 16:00:13 merkaba kernel: [ 2640.531137]  [<ffffffff81062c4d>] ? get_parent_ip+0xd/0x3c
Jul 14 16:00:13 merkaba kernel: [ 2640.531142]  [<ffffffff81062cf7>] ? preempt_count_add+0x7b/0x8e
Jul 14 16:00:13 merkaba kernel: [ 2640.531150]  [<ffffffff8147149d>] __wait_for_common+0x11e/0x163
Jul 14 16:00:13 merkaba kernel: [ 2640.531158]  [<ffffffff8147149d>] ? __wait_for_common+0x11e/0x163
Jul 14 16:00:13 merkaba kernel: [ 2640.531165]  [<ffffffff810647e9>] ? wake_up_state+0xd/0xd
Jul 14 16:00:13 merkaba kernel: [ 2640.531173]  [<ffffffff81471501>] wait_for_completion+0x1f/0x21
Jul 14 16:00:13 merkaba kernel: [ 2640.531218]  [<ffffffffc0359c10>] btrfs_wait_and_free_delalloc_work+0x11/0x23 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.531263]  [<ffffffffc0361b56>] btrfs_run_ordered_operations+0x1ea/0x21a [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.531331]  [<ffffffffc034d392>] btrfs_commit_transaction+0x22/0x8df [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.531385]  [<ffffffffc0349fb6>] transaction_kthread+0x107/0x1b9 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.531446]  [<ffffffffc0349eaf>] ? btrfs_cleanup_transaction+0x43a/0x43a [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.531469]  [<ffffffff81058002>] kthread+0xb2/0xba
Jul 14 16:00:13 merkaba kernel: [ 2640.531477]  [<ffffffff81470000>] ? iommu_prepare_identity_map+0x1d/0x19d
Jul 14 16:00:13 merkaba kernel: [ 2640.531484]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 14 16:00:13 merkaba kernel: [ 2640.531490]  [<ffffffff81473bec>] ret_from_fork+0x7c/0xb0
Jul 14 16:00:13 merkaba kernel: [ 2640.531497]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 14 16:00:13 merkaba kernel: [ 2640.531589] INFO: task mysqld:2758 blocked for more than 120 seconds.
Jul 14 16:00:13 merkaba kernel: [ 2640.531595]       Tainted: G           O  3.16.0-rc5-tp520 #4
Jul 14 16:00:13 merkaba kernel: [ 2640.531598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 14 16:00:13 merkaba kernel: [ 2640.531601] mysqld          D ffff88034079cef0     0  2758   2738 0x00000000
Jul 14 16:00:13 merkaba kernel: [ 2640.531625]  ffff88033d9dfbd8 0000000000000002 ffff880407f66240 ffff88033d9dffd8
Jul 14 16:00:13 merkaba kernel: [ 2640.531645]  ffff88034079c9b0 0000000000013080 ffff88034079c9b0 ffff88041e313080
Jul 14 16:00:13 merkaba kernel: [ 2640.531652]  ffff88041e5c9268 ffff88033d9dfc70 0000000000000002 ffffffff810daffc
Jul 14 16:00:13 merkaba kernel: [ 2640.531660] Call Trace:
Jul 14 16:00:13 merkaba kernel: [ 2640.531668]  [<ffffffff810daffc>] ? wait_on_page_read+0x37/0x37
Jul 14 16:00:13 merkaba kernel: [ 2640.531675]  [<ffffffff81470c70>] schedule+0x64/0x66
Jul 14 16:00:13 merkaba kernel: [ 2640.531702]  [<ffffffff81470df7>] io_schedule+0x57/0x76
Jul 14 16:00:13 merkaba kernel: [ 2640.531708]  [<ffffffff810db005>] sleep_on_page+0x9/0xd
Jul 14 16:00:13 merkaba kernel: [ 2640.531715]  [<ffffffff814711d7>] __wait_on_bit_lock+0x41/0x85
Jul 14 16:00:13 merkaba kernel: [ 2640.531721]  [<ffffffff810db0c2>] __lock_page+0x70/0x7c
Jul 14 16:00:13 merkaba kernel: [ 2640.531741]  [<ffffffff81070f79>] ? autoremove_wake_function+0x2f/0x2f
Jul 14 16:00:13 merkaba kernel: [ 2640.531787]  [<ffffffffc0362590>] lock_page+0x1e/0x21 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.531835]  [<ffffffffc0362590>] ? lock_page+0x1e/0x21 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.531880]  [<ffffffffc036686d>] extent_write_cache_pages.isra.21.constprop.42+0x1a7/0x2d9 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.531925]  [<ffffffffc0366d95>] extent_writepages+0x46/0x57 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.531967]  [<ffffffffc03510a2>] ? btrfs_submit_direct+0x3ef/0x3ef [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.532006]  [<ffffffffc034f60f>] btrfs_writepages+0x23/0x25 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.532012]  [<ffffffff810e5381>] do_writepages+0x1b/0x24
Jul 14 16:00:13 merkaba kernel: [ 2640.532018]  [<ffffffff810dc6ad>] __filemap_fdatawrite_range+0x50/0x52
Jul 14 16:00:13 merkaba kernel: [ 2640.532025]  [<ffffffff810dc714>] filemap_fdatawrite_range+0xe/0x10
Jul 14 16:00:13 merkaba kernel: [ 2640.532067]  [<ffffffffc035aa0a>] btrfs_sync_file+0x67/0x2bd [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.532074]  [<ffffffff810dc6ad>] ? __filemap_fdatawrite_range+0x50/0x52
Jul 14 16:00:13 merkaba kernel: [ 2640.532083]  [<ffffffff81155fab>] vfs_fsync_range+0x1c/0x1e
Jul 14 16:00:13 merkaba kernel: [ 2640.532090]  [<ffffffff81155fc4>] vfs_fsync+0x17/0x19
Jul 14 16:00:13 merkaba kernel: [ 2640.532104]  [<ffffffffc02aa32a>] ecryptfs_fsync+0x2f/0x34 [ecryptfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.532111]  [<ffffffff81155fab>] vfs_fsync_range+0x1c/0x1e
Jul 14 16:00:13 merkaba kernel: [ 2640.532117]  [<ffffffff81155fc4>] vfs_fsync+0x17/0x19
Jul 14 16:00:13 merkaba kernel: [ 2640.532124]  [<ffffffff811561eb>] do_fsync+0x2c/0x45
Jul 14 16:00:13 merkaba kernel: [ 2640.532132]  [<ffffffff811563d1>] SyS_fsync+0xb/0xf
Jul 14 16:00:13 merkaba kernel: [ 2640.532138]  [<ffffffff81473e8b>] tracesys+0xdd/0xe2
Jul 14 16:00:13 merkaba kernel: [ 2640.532147] INFO: task mysqld:2832 blocked for more than 120 seconds.
Jul 14 16:00:13 merkaba kernel: [ 2640.532151]       Tainted: G           O  3.16.0-rc5-tp520 #4
Jul 14 16:00:13 merkaba kernel: [ 2640.532155] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 14 16:00:13 merkaba kernel: [ 2640.532158] mysqld          D ffff88009699b660     0  2832   2738 0x00000000
Jul 14 16:00:13 merkaba kernel: [ 2640.532165]  ffff88032f9fbbd8 0000000000000002 ffffffff81a15500 ffff88032f9fbfd8
Jul 14 16:00:13 merkaba kernel: [ 2640.532173]  ffff88009699b120 0000000000013080 ffff88009699b120 ffff88041e213080
Jul 14 16:00:13 merkaba kernel: [ 2640.532180]  ffff88041e5bb468 ffff88032f9fbc70 0000000000000002 ffffffff810daffc
Jul 14 16:00:13 merkaba kernel: [ 2640.532187] Call Trace:
Jul 14 16:00:13 merkaba kernel: [ 2640.532194]  [<ffffffff810daffc>] ? wait_on_page_read+0x37/0x37
Jul 14 16:00:13 merkaba kernel: [ 2640.532201]  [<ffffffff81470c70>] schedule+0x64/0x66
Jul 14 16:00:13 merkaba kernel: [ 2640.532209]  [<ffffffff81470df7>] io_schedule+0x57/0x76
Jul 14 16:00:13 merkaba kernel: [ 2640.532214]  [<ffffffff810db005>] sleep_on_page+0x9/0xd
Jul 14 16:00:13 merkaba kernel: [ 2640.532221]  [<ffffffff814711d7>] __wait_on_bit_lock+0x41/0x85
Jul 14 16:00:13 merkaba kernel: [ 2640.532227]  [<ffffffff810db0c2>] __lock_page+0x70/0x7c
Jul 14 16:00:13 merkaba kernel: [ 2640.532236]  [<ffffffff81070f79>] ? autoremove_wake_function+0x2f/0x2f
Jul 14 16:00:13 merkaba kernel: [ 2640.532283]  [<ffffffffc0362590>] lock_page+0x1e/0x21 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.532323]  [<ffffffffc0362590>] ? lock_page+0x1e/0x21 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.532366]  [<ffffffffc036686d>] extent_write_cache_pages.isra.21.constprop.42+0x1a7/0x2d9 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.532410]  [<ffffffffc0366d95>] extent_writepages+0x46/0x57 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.532452]  [<ffffffffc03510a2>] ? btrfs_submit_direct+0x3ef/0x3ef [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.532490]  [<ffffffffc034f60f>] btrfs_writepages+0x23/0x25 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.532497]  [<ffffffff810e5381>] do_writepages+0x1b/0x24
Jul 14 16:00:13 merkaba kernel: [ 2640.532507]  [<ffffffff810dc6ad>] __filemap_fdatawrite_range+0x50/0x52
Jul 14 16:00:13 merkaba kernel: [ 2640.532513]  [<ffffffff810dc714>] filemap_fdatawrite_range+0xe/0x10
Jul 14 16:00:13 merkaba kernel: [ 2640.532577]  [<ffffffffc035aa0a>] btrfs_sync_file+0x67/0x2bd [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.532608]  [<ffffffff810dc6ad>] ? __filemap_fdatawrite_range+0x50/0x52
Jul 14 16:00:13 merkaba kernel: [ 2640.532643]  [<ffffffff81155fab>] vfs_fsync_range+0x1c/0x1e
Jul 14 16:00:13 merkaba kernel: [ 2640.532677]  [<ffffffff81155fc4>] vfs_fsync+0x17/0x19
Jul 14 16:00:13 merkaba kernel: [ 2640.532712]  [<ffffffffc02aa32a>] ecryptfs_fsync+0x2f/0x34 [ecryptfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.532741]  [<ffffffff81155fab>] vfs_fsync_range+0x1c/0x1e
Jul 14 16:00:13 merkaba kernel: [ 2640.532774]  [<ffffffff81155fc4>] vfs_fsync+0x17/0x19
Jul 14 16:00:13 merkaba kernel: [ 2640.532785]  [<ffffffff811561eb>] do_fsync+0x2c/0x45
Jul 14 16:00:13 merkaba kernel: [ 2640.532792]  [<ffffffff811563d1>] SyS_fsync+0xb/0xf
Jul 14 16:00:13 merkaba kernel: [ 2640.532798]  [<ffffffff81473e8b>] tracesys+0xdd/0xe2
Jul 14 16:00:13 merkaba kernel: [ 2640.532821] INFO: task Cache I/O:3820 blocked for more than 120 seconds.
Jul 14 16:00:13 merkaba kernel: [ 2640.532826]       Tainted: G           O  3.16.0-rc5-tp520 #4
Jul 14 16:00:13 merkaba kernel: [ 2640.532829] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 14 16:00:13 merkaba kernel: [ 2640.532833] Cache I/O       D ffff880272b08540     0  3820   2660 0x00000000
Jul 14 16:00:13 merkaba kernel: [ 2640.532840]  ffff880272aabbd8 0000000000000002 ffffffff81a15500 ffff880272aabfd8
Jul 14 16:00:13 merkaba kernel: [ 2640.532847]  ffff880272b08000 0000000000013080 ffff880272b08000 ffff88041e213080
Jul 14 16:00:13 merkaba kernel: [ 2640.532854]  ffff88041e5c5968 ffff880272aabc70 0000000000000002 ffffffff810daffc
Jul 14 16:00:13 merkaba kernel: [ 2640.532862] Call Trace:
Jul 14 16:00:13 merkaba kernel: [ 2640.532869]  [<ffffffff810daffc>] ? wait_on_page_read+0x37/0x37
Jul 14 16:00:13 merkaba kernel: [ 2640.532876]  [<ffffffff81470c70>] schedule+0x64/0x66
Jul 14 16:00:13 merkaba kernel: [ 2640.532883]  [<ffffffff81470df7>] io_schedule+0x57/0x76
Jul 14 16:00:13 merkaba kernel: [ 2640.532889]  [<ffffffff810db005>] sleep_on_page+0x9/0xd
Jul 14 16:00:13 merkaba kernel: [ 2640.532896]  [<ffffffff814711d7>] __wait_on_bit_lock+0x41/0x85
Jul 14 16:00:13 merkaba kernel: [ 2640.532902]  [<ffffffff810db0c2>] __lock_page+0x70/0x7c
Jul 14 16:00:13 merkaba kernel: [ 2640.532910]  [<ffffffff81070f79>] ? autoremove_wake_function+0x2f/0x2f
Jul 14 16:00:13 merkaba kernel: [ 2640.532953]  [<ffffffffc0362590>] lock_page+0x1e/0x21 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.533013]  [<ffffffffc0362590>] ? lock_page+0x1e/0x21 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.533082]  [<ffffffffc036686d>] extent_write_cache_pages.isra.21.constprop.42+0x1a7/0x2d9 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.533111]  [<ffffffff810e33cf>] ? rcu_read_unlock_sched_notrace+0x17/0x17
Jul 14 16:00:13 merkaba kernel: [ 2640.533117]  [<ffffffff810db5c2>] ? generic_perform_write+0x13c/0x179
Jul 14 16:00:13 merkaba kernel: [ 2640.533124]  [<ffffffff810db8a5>] ? find_get_pages_tag+0xfc/0x123
Jul 14 16:00:13 merkaba kernel: [ 2640.533166]  [<ffffffffc0366d95>] extent_writepages+0x46/0x57 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.533209]  [<ffffffffc03510a2>] ? btrfs_submit_direct+0x3ef/0x3ef [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.533263]  [<ffffffffc034f60f>] btrfs_writepages+0x23/0x25 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.533269]  [<ffffffff810e5381>] do_writepages+0x1b/0x24
Jul 14 16:00:13 merkaba kernel: [ 2640.533279]  [<ffffffff810dc6ad>] __filemap_fdatawrite_range+0x50/0x52
Jul 14 16:00:13 merkaba kernel: [ 2640.533285]  [<ffffffff810dc714>] filemap_fdatawrite_range+0xe/0x10
Jul 14 16:00:13 merkaba kernel: [ 2640.533328]  [<ffffffffc035aa27>] btrfs_sync_file+0x84/0x2bd [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.533334]  [<ffffffff810dc6ad>] ? __filemap_fdatawrite_range+0x50/0x52
Jul 14 16:00:13 merkaba kernel: [ 2640.533342]  [<ffffffff81155fab>] vfs_fsync_range+0x1c/0x1e
Jul 14 16:00:13 merkaba kernel: [ 2640.533349]  [<ffffffff81155fc4>] vfs_fsync+0x17/0x19
Jul 14 16:00:13 merkaba kernel: [ 2640.533362]  [<ffffffffc02aa32a>] ecryptfs_fsync+0x2f/0x34 [ecryptfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.533369]  [<ffffffff81155fab>] vfs_fsync_range+0x1c/0x1e
Jul 14 16:00:13 merkaba kernel: [ 2640.533375]  [<ffffffff81155fc4>] vfs_fsync+0x17/0x19
Jul 14 16:00:13 merkaba kernel: [ 2640.533382]  [<ffffffff811561eb>] do_fsync+0x2c/0x45
Jul 14 16:00:13 merkaba kernel: [ 2640.533389]  [<ffffffff811563d1>] SyS_fsync+0xb/0xf
Jul 14 16:00:13 merkaba kernel: [ 2640.533395]  [<ffffffff81473e8b>] tracesys+0xdd/0xe2
Jul 14 16:00:13 merkaba kernel: [ 2640.533405] INFO: task kworker/u8:0:3901 blocked for more than 120 seconds.
Jul 14 16:00:13 merkaba kernel: [ 2640.533409]       Tainted: G           O  3.16.0-rc5-tp520 #4
Jul 14 16:00:13 merkaba kernel: [ 2640.533413] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 14 16:00:13 merkaba kernel: [ 2640.533416] kworker/u8:0    D ffff880407f00540     0  3901      2 0x00000000
Jul 14 16:00:13 merkaba kernel: [ 2640.533466] Workqueue: btrfs-delalloc normal_work_helper [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.533470]  ffff880062757838 0000000000000002 ffff880407f649b0 ffff880062757fd8
Jul 14 16:00:13 merkaba kernel: [ 2640.533478]  ffff880407f00000 0000000000013080 ffff880407f00000 ffff88041e293080
Jul 14 16:00:13 merkaba kernel: [ 2640.533485]  ffff88041e5d2e68 ffff8800627578d0 0000000000000002 ffffffff810daffc
Jul 14 16:00:13 merkaba kernel: [ 2640.533493] Call Trace:
Jul 14 16:00:13 merkaba kernel: [ 2640.533499]  [<ffffffff810daffc>] ? wait_on_page_read+0x37/0x37
Jul 14 16:00:13 merkaba kernel: [ 2640.533507]  [<ffffffff81470c70>] schedule+0x64/0x66
Jul 14 16:00:13 merkaba kernel: [ 2640.533514]  [<ffffffff81470df7>] io_schedule+0x57/0x76
Jul 14 16:00:13 merkaba kernel: [ 2640.533537]  [<ffffffff810db005>] sleep_on_page+0x9/0xd
Jul 14 16:00:13 merkaba kernel: [ 2640.533545]  [<ffffffff814711d7>] __wait_on_bit_lock+0x41/0x85
Jul 14 16:00:13 merkaba kernel: [ 2640.533556]  [<ffffffff810db0c2>] __lock_page+0x70/0x7c
Jul 14 16:00:13 merkaba kernel: [ 2640.533588]  [<ffffffff81070f79>] ? autoremove_wake_function+0x2f/0x2f
Jul 14 16:00:13 merkaba kernel: [ 2640.533623]  [<ffffffff810db8fe>] lock_page+0x19/0x1c
Jul 14 16:00:13 merkaba kernel: [ 2640.533652]  [<ffffffff810db8fe>] ? lock_page+0x19/0x1c
Jul 14 16:00:13 merkaba kernel: [ 2640.533676]  [<ffffffff810dc02b>] pagecache_get_page+0x60/0x149
Jul 14 16:00:13 merkaba kernel: [ 2640.533726]  [<ffffffffc0380ee6>] io_ctl_prepare_pages+0x63/0x127 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.533774]  [<ffffffffc0382af8>] __load_free_space_cache+0x1ca/0x521 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.533821]  [<ffffffffc0382f30>] load_free_space_cache+0xe1/0x18f [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.533869]  [<ffffffffc033657e>] cache_block_group+0x1b9/0x330 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.533884]  [<ffffffff81070f4a>] ? finish_wait+0x5d/0x5d
Jul 14 16:00:13 merkaba kernel: [ 2640.533927]  [<ffffffffc033d249>] find_free_extent+0x3b0/0x970 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.533965]  [<ffffffffc033d985>] btrfs_reserve_extent+0x6f/0x119 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.534008]  [<ffffffffc0352b35>] cow_file_range+0x1a6/0x377 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.534054]  [<ffffffffc0366d3d>] ? extent_write_locked_range+0x10c/0x11e [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.534096]  [<ffffffffc03537d4>] submit_compressed_extents+0x100/0x3fc [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.534105]  [<ffffffff8111d44d>] ? kfree+0xf0/0x130
Jul 14 16:00:13 merkaba kernel: [ 2640.534148]  [<ffffffffc0353b52>] async_cow_submit+0x82/0x87 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.534195]  [<ffffffffc03723c4>] normal_work_helper+0x153/0x224 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.534204]  [<ffffffff81052d7c>] process_one_work+0x16f/0x2b8
Jul 14 16:00:13 merkaba kernel: [ 2640.534212]  [<ffffffff81053626>] worker_thread+0x27b/0x32e
Jul 14 16:00:13 merkaba kernel: [ 2640.534220]  [<ffffffff810533ab>] ? cancel_delayed_work_sync+0x10/0x10
Jul 14 16:00:13 merkaba kernel: [ 2640.534226]  [<ffffffff81058002>] kthread+0xb2/0xba
Jul 14 16:00:13 merkaba kernel: [ 2640.534235]  [<ffffffff81470000>] ? iommu_prepare_identity_map+0x1d/0x19d
Jul 14 16:00:13 merkaba kernel: [ 2640.534242]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 14 16:00:13 merkaba kernel: [ 2640.534248]  [<ffffffff81473bec>] ret_from_fork+0x7c/0xb0
Jul 14 16:00:13 merkaba kernel: [ 2640.534254]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 14 16:00:13 merkaba kernel: [ 2640.534262] INFO: task kworker/u8:7:3956 blocked for more than 120 seconds.
Jul 14 16:00:13 merkaba kernel: [ 2640.534266]       Tainted: G           O  3.16.0-rc5-tp520 #4
Jul 14 16:00:13 merkaba kernel: [ 2640.534269] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 14 16:00:13 merkaba kernel: [ 2640.534273] kworker/u8:7    D ffff880268063660     0  3956      2 0x00000000
Jul 14 16:00:13 merkaba kernel: [ 2640.534290] Workqueue: writeback bdi_writeback_workfn (flush-btrfs-2)
Jul 14 16:00:13 merkaba kernel: [ 2640.534299]  ffff8802470b79b8 0000000000000002 ffff880407fc0000 ffff8802470b7fd8
Jul 14 16:00:13 merkaba kernel: [ 2640.534312]  ffff880268063120 0000000000013080 ffff880268063120 ffff88041e393080
Jul 14 16:00:13 merkaba kernel: [ 2640.534323]  ffff88041e5d0a68 ffff8802470b7a50 0000000000000002 ffffffff810daffc
Jul 14 16:00:13 merkaba kernel: [ 2640.534332] Call Trace:
Jul 14 16:00:13 merkaba kernel: [ 2640.534339]  [<ffffffff810daffc>] ? wait_on_page_read+0x37/0x37
Jul 14 16:00:13 merkaba kernel: [ 2640.534351]  [<ffffffff81470c70>] schedule+0x64/0x66
Jul 14 16:00:13 merkaba kernel: [ 2640.534358]  [<ffffffff81470df7>] io_schedule+0x57/0x76
Jul 14 16:00:13 merkaba kernel: [ 2640.534363]  [<ffffffff810db005>] sleep_on_page+0x9/0xd
Jul 14 16:00:13 merkaba kernel: [ 2640.534371]  [<ffffffff814711d7>] __wait_on_bit_lock+0x41/0x85
Jul 14 16:00:13 merkaba kernel: [ 2640.534377]  [<ffffffff810db0c2>] __lock_page+0x70/0x7c
Jul 14 16:00:13 merkaba kernel: [ 2640.534384]  [<ffffffff81070f79>] ? autoremove_wake_function+0x2f/0x2f
Jul 14 16:00:13 merkaba kernel: [ 2640.534428]  [<ffffffffc0362590>] lock_page+0x1e/0x21 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.534468]  [<ffffffffc0362590>] ? lock_page+0x1e/0x21 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.534512]  [<ffffffffc036686d>] extent_write_cache_pages.isra.21.constprop.42+0x1a7/0x2d9 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.534573]  [<ffffffffc0366d95>] extent_writepages+0x46/0x57 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.534638]  [<ffffffffc03510a2>] ? btrfs_submit_direct+0x3ef/0x3ef [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.534701]  [<ffffffffc034f60f>] btrfs_writepages+0x23/0x25 [btrfs]
Jul 14 16:00:13 merkaba kernel: [ 2640.534728]  [<ffffffff810e5381>] do_writepages+0x1b/0x24
Jul 14 16:00:13 merkaba kernel: [ 2640.534762]  [<ffffffff81151d0d>] __writeback_single_inode+0x58/0x212
Jul 14 16:00:13 merkaba kernel: [ 2640.534792]  [<ffffffff81152c0f>] writeback_sb_inodes+0x1e9/0x32e
Jul 14 16:00:13 merkaba kernel: [ 2640.534806]  [<ffffffff81152dce>] __writeback_inodes_wb+0x7a/0xb0
Jul 14 16:00:13 merkaba kernel: [ 2640.534814]  [<ffffffff81152f1a>] wb_writeback+0x116/0x270
Jul 14 16:00:13 merkaba kernel: [ 2640.534821]  [<ffffffff811532c9>] bdi_writeback_workfn+0x171/0x313
Jul 14 16:00:13 merkaba kernel: [ 2640.534830]  [<ffffffff81052d7c>] process_one_work+0x16f/0x2b8
Jul 14 16:00:13 merkaba kernel: [ 2640.534838]  [<ffffffff81053626>] worker_thread+0x27b/0x32e
Jul 14 16:00:13 merkaba kernel: [ 2640.534846]  [<ffffffff810533ab>] ? cancel_delayed_work_sync+0x10/0x10
Jul 14 16:00:13 merkaba kernel: [ 2640.534853]  [<ffffffff81058002>] kthread+0xb2/0xba
Jul 14 16:00:13 merkaba kernel: [ 2640.534861]  [<ffffffff81470000>] ? iommu_prepare_identity_map+0x1d/0x19d
Jul 14 16:00:13 merkaba kernel: [ 2640.534868]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 14 16:00:13 merkaba kernel: [ 2640.534874]  [<ffffffff81473bec>] ret_from_fork+0x7c/0xb0
Jul 14 16:00:13 merkaba kernel: [ 2640.534881]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 14 16:02:13 merkaba kernel: [ 2760.554316] INFO: task kworker/u8:4:162 blocked for more than 120 seconds.
Jul 14 16:02:13 merkaba kernel: [ 2760.554328]       Tainted: G           O  3.16.0-rc5-tp520 #4
Jul 14 16:02:13 merkaba kernel: [ 2760.554332] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 14 16:02:13 merkaba kernel: [ 2760.554336] kworker/u8:4    D ffff880405cab660     0   162      2 0x00000000
Jul 14 16:02:13 merkaba kernel: [ 2760.554429] Workqueue: btrfs-flush_delalloc normal_work_helper [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.554436]  ffff8804055d3b18 0000000000000002 ffff880407f66240 ffff8804055d3fd8
Jul 14 16:02:13 merkaba kernel: [ 2760.554444]  ffff880405cab120 0000000000013080 ffff880405cab120 ffff88041e313080
Jul 14 16:02:13 merkaba kernel: [ 2760.554451]  ffff88041e5e7b68 ffff8804055d3bb0 0000000000000002 ffffffff810daffc
Jul 14 16:02:13 merkaba kernel: [ 2760.554459] Call Trace:
Jul 14 16:02:13 merkaba kernel: [ 2760.554473]  [<ffffffff810daffc>] ? wait_on_page_read+0x37/0x37
Jul 14 16:02:13 merkaba kernel: [ 2760.554483]  [<ffffffff81470c70>] schedule+0x64/0x66
Jul 14 16:02:13 merkaba kernel: [ 2760.554491]  [<ffffffff81470df7>] io_schedule+0x57/0x76
Jul 14 16:02:13 merkaba kernel: [ 2760.554497]  [<ffffffff810db005>] sleep_on_page+0x9/0xd
Jul 14 16:02:13 merkaba kernel: [ 2760.554504]  [<ffffffff814711d7>] __wait_on_bit_lock+0x41/0x85
Jul 14 16:02:13 merkaba kernel: [ 2760.554510]  [<ffffffff810db0c2>] __lock_page+0x70/0x7c
Jul 14 16:02:13 merkaba kernel: [ 2760.554521]  [<ffffffff81070f79>] ? autoremove_wake_function+0x2f/0x2f
Jul 14 16:02:13 merkaba kernel: [ 2760.554577]  [<ffffffffc0362590>] lock_page+0x1e/0x21 [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.554624]  [<ffffffffc0362590>] ? lock_page+0x1e/0x21 [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.554673]  [<ffffffffc036686d>] extent_write_cache_pages.isra.21.constprop.42+0x1a7/0x2d9 [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.554685]  [<ffffffff81067fd7>] ? account_entity_enqueue+0x55/0x86
Jul 14 16:02:13 merkaba kernel: [ 2760.554693]  [<ffffffff810680c7>] ? __enqueue_entity+0x67/0x69
Jul 14 16:02:13 merkaba kernel: [ 2760.554740]  [<ffffffffc0366d95>] extent_writepages+0x46/0x57 [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.554787]  [<ffffffffc03510a2>] ? btrfs_submit_direct+0x3ef/0x3ef [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.554832]  [<ffffffffc034f60f>] btrfs_writepages+0x23/0x25 [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.554838]  [<ffffffff810e5381>] do_writepages+0x1b/0x24
Jul 14 16:02:13 merkaba kernel: [ 2760.554845]  [<ffffffff810dc6ad>] __filemap_fdatawrite_range+0x50/0x52
Jul 14 16:02:13 merkaba kernel: [ 2760.554852]  [<ffffffff810dcfd6>] filemap_flush+0x17/0x19
Jul 14 16:02:13 merkaba kernel: [ 2760.554897]  [<ffffffffc0351de9>] btrfs_run_delalloc_work+0x2e/0x64 [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.554949]  [<ffffffffc0372350>] normal_work_helper+0xdf/0x224 [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.554959]  [<ffffffff81052d7c>] process_one_work+0x16f/0x2b8
Jul 14 16:02:13 merkaba kernel: [ 2760.554967]  [<ffffffff81053626>] worker_thread+0x27b/0x32e
Jul 14 16:02:13 merkaba kernel: [ 2760.554975]  [<ffffffff810533ab>] ? cancel_delayed_work_sync+0x10/0x10
Jul 14 16:02:13 merkaba kernel: [ 2760.554982]  [<ffffffff81058002>] kthread+0xb2/0xba
Jul 14 16:02:13 merkaba kernel: [ 2760.554991]  [<ffffffff81470000>] ? iommu_prepare_identity_map+0x1d/0x19d
Jul 14 16:02:13 merkaba kernel: [ 2760.554998]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 14 16:02:13 merkaba kernel: [ 2760.555005]  [<ffffffff81473bec>] ret_from_fork+0x7c/0xb0
Jul 14 16:02:13 merkaba kernel: [ 2760.555012]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 14 16:02:13 merkaba kernel: [ 2760.555019] INFO: task kworker/u8:8:166 blocked for more than 120 seconds.
Jul 14 16:02:13 merkaba kernel: [ 2760.555024]       Tainted: G           O  3.16.0-rc5-tp520 #4
Jul 14 16:02:13 merkaba kernel: [ 2760.555028] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 14 16:02:13 merkaba kernel: [ 2760.555031] kworker/u8:8    D ffff880405b84ef0     0   166      2 0x00000000
Jul 14 16:02:13 merkaba kernel: [ 2760.555085] Workqueue: btrfs-flush_delalloc normal_work_helper [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.555089]  ffff880405643b18 0000000000000002 ffff880407f649b0 ffff880405643fd8
Jul 14 16:02:13 merkaba kernel: [ 2760.555097]  ffff880405b849b0 0000000000013080 ffff880405b849b0 ffff88041e293080
Jul 14 16:02:13 merkaba kernel: [ 2760.555104]  ffff88041e5d0168 ffff880405643bb0 0000000000000002 ffffffff810daffc
Jul 14 16:02:13 merkaba kernel: [ 2760.555112] Call Trace:
Jul 14 16:02:13 merkaba kernel: [ 2760.555120]  [<ffffffff810daffc>] ? wait_on_page_read+0x37/0x37
Jul 14 16:02:13 merkaba kernel: [ 2760.555128]  [<ffffffff81470c70>] schedule+0x64/0x66
Jul 14 16:02:13 merkaba kernel: [ 2760.555136]  [<ffffffff81470df7>] io_schedule+0x57/0x76
Jul 14 16:02:13 merkaba kernel: [ 2760.555141]  [<ffffffff810db005>] sleep_on_page+0x9/0xd
Jul 14 16:02:13 merkaba kernel: [ 2760.555149]  [<ffffffff814711d7>] __wait_on_bit_lock+0x41/0x85
Jul 14 16:02:13 merkaba kernel: [ 2760.555155]  [<ffffffff810db0c2>] __lock_page+0x70/0x7c
Jul 14 16:02:13 merkaba kernel: [ 2760.555162]  [<ffffffff81070f79>] ? autoremove_wake_function+0x2f/0x2f
Jul 14 16:02:13 merkaba kernel: [ 2760.555246]  [<ffffffffc0362590>] lock_page+0x1e/0x21 [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.555299]  [<ffffffffc0362590>] ? lock_page+0x1e/0x21 [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.555371]  [<ffffffffc036686d>] extent_write_cache_pages.isra.21.constprop.42+0x1a7/0x2d9 [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.555435]  [<ffffffffc0366d95>] extent_writepages+0x46/0x57 [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.555497]  [<ffffffffc03510a2>] ? btrfs_submit_direct+0x3ef/0x3ef [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.555557]  [<ffffffffc034f60f>] btrfs_writepages+0x23/0x25 [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.555588]  [<ffffffff810e5381>] do_writepages+0x1b/0x24
Jul 14 16:02:13 merkaba kernel: [ 2760.555615]  [<ffffffff810dc6ad>] __filemap_fdatawrite_range+0x50/0x52
Jul 14 16:02:13 merkaba kernel: [ 2760.555645]  [<ffffffff810dcfd6>] filemap_flush+0x17/0x19
Jul 14 16:02:13 merkaba kernel: [ 2760.555691]  [<ffffffffc0351de9>] btrfs_run_delalloc_work+0x2e/0x64 [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.555738]  [<ffffffffc0372350>] normal_work_helper+0xdf/0x224 [btrfs]
Jul 14 16:02:13 merkaba kernel: [ 2760.555747]  [<ffffffff81052d7c>] process_one_work+0x16f/0x2b8
Jul 14 16:02:13 merkaba kernel: [ 2760.555755]  [<ffffffff81053626>] worker_thread+0x27b/0x32e
Jul 14 16:02:13 merkaba kernel: [ 2760.555763]  [<ffffffff810533ab>] ? cancel_delayed_work_sync+0x10/0x10
Jul 14 16:02:13 merkaba kernel: [ 2760.555770]  [<ffffffff81058002>] kthread+0xb2/0xba
Jul 14 16:02:13 merkaba kernel: [ 2760.555778]  [<ffffffff81470000>] ? iommu_prepare_identity_map+0x1d/0x19d
Jul 14 16:02:13 merkaba kernel: [ 2760.555785]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 14 16:02:13 merkaba kernel: [ 2760.555791]  [<ffffffff81473bec>] ret_from_fork+0x7c/0xb0
Jul 14 16:02:13 merkaba kernel: [ 2760.555798]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 14 16:07:16 merkaba kernel: [ 3063.762354] e1000e: eth0 NIC Link is Down
Jul 14 16:07:48 merkaba kernel: [ 3095.667377] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
Jul 14 16:07:48 merkaba kernel: [ 3095.667490] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jul 14 16:29:12 merkaba kernel: [    0.000000] Initializing cgroup subsys cpuset
Jul 14 16:29:12 merkaba kernel: [    0.000000] Initializing cgroup subsys cpu
Jul 14 16:29:12 merkaba kernel: [    0.000000] Initializing cgroup subsys cpuacct
Jul 14 16:29:12 merkaba kernel: [    0.000000] Linux version 3.16.0-rc5-tp520 (martin@merkaba) (gcc version 4.9.0 (Debian 4.9.0-10) ) #4 SMP PREEMPT Mon Jul 14 14:21:02 CEST 2014
Jul 14 16:29:12 merkaba kernel: [    0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.16.0-rc5-tp520 root=/dev/mapper/sata-debian ro rootflags=subvol=debian init=/bin/systemd resume=/dev/mapper/sata-swap

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BTRFS hang with 3.16-rc5

[Martin Steigerwald]

Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
> Hi!
> 
> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several days of
> usage, with 3-16-rc5 I had a hang again. Less than a hour since booting it.
> 
> Since the hang bug I and others had with 3.15 and upto 3.16-rc2 usually
> didn´t happen that quickly after boot and since backtrace looks a bit
> different from what I have in memory, I post this in a new thread.
> See thread "Blocked tasks on 3.15.1" for a discussion of previous hang
> issues.

Probably good to add some basic information on the filesystem:

merkaba:~> btrfs fi sh /home
Label: 'home'  uuid: […]
        Total devices 2 FS bytes used 123.20GiB
        devid    1 size 160.00GiB used 159.98GiB path /dev/mapper/msata-home
        devid    2 size 160.00GiB used 159.98GiB path /dev/dm-3

Btrfs v3.14.1
merkaba:~#1> btrfs fi df /home
Data, RAID1: total=154.95GiB, used=120.61GiB
System, RAID1: total=32.00MiB, used=48.00KiB
Metadata, RAID1: total=5.00GiB, used=2.59GiB
unknown, single: total=512.00MiB, used=0.00
merkaba:~> df -hT /home
Dateisystem            Typ   Größe Benutzt Verf. Verw% Eingehängt auf
/dev/mapper/msata-home btrfs  320G    247G   69G   79% /home

merkaba:~> file -sk /dev/sata/home
/dev/sata/home: symbolic link to `../dm-3'
merkaba:~> file -sk /dev/dm-3     
/dev/dm-3: BTRFS Filesystem label "home", sectorsize 4096, nodesize 16384, 
leafsize 16384, UUID=b96c4f72- 523-45ac-a401-f7be73dd624a, 
132303151104/343597383680 bytes used, 2 devices


I think I switched it to skinny extents, but I am not completely sure.


Dual SSD BTRFS RAID 1

No snapshots at the moment. And plenty of space free.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BTRFS hang with 3.16-rc5

[Duncan]

Martin Steigerwald posted on Mon, 14 Jul 2014 17:10:30 +0200 as excerpted:

> Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
>> Hi!
>> 
>> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several days
>> of usage, with 3-16-rc5 I had a hang again. Less than a hour since
>> booting it.
>> 
>> Since the hang bug I and others had with 3.15 and upto 3.16-rc2 usually
>> didn´t happen that quickly after boot and since backtrace looks a bit
>> different from what I have in memory, I post this in a new thread.
>> See thread "Blocked tasks on 3.15.1" for a discussion of previous hang
>> issues.
> 
> Probably good to add some basic information on the filesystem: [...]

> I think I switched it to skinny extents, but I am not completely sure.
> 
> 
> Dual SSD BTRFS RAID 1
> 
> No snapshots at the moment. And plenty of space free.

FWIW, I've been updating every few days, running two different snapshots 
between 3.16-rc4 and rc5, and now rc5 itself, and haven't noticed this 
issue.  Dual SSDs btrfs raid1, but partitioned up so the biggest 
partition is under 50 GB.

I've been doing some very heavy (gentoo) package building and shuffling 
around the last few days trying out the kde-frameworks/workspaces-5 
stuff, then deciding it's not yet ready for me, too, so I've been working 
it a bit, too.

But certainly YMMV, and I did have some nasty trouble a few weeks ago, 
which triggered a btrfs restore and then a mkfs on a couple of the 
filesystems.  But nothing at all strange with btrfs since then, or 
really, for awhile before that either.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS hang with 3.16-rc5

[Chris Mason]

On 07/14/2014 11:10 AM, Martin Steigerwald wrote:
> Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
>> Hi!
>>
>> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several days of
>> usage, with 3-16-rc5 I had a hang again. Less than a hour since booting it.
>>
>> Since the hang bug I and others had with 3.15 and upto 3.16-rc2 usually
>> didn´t happen that quickly after boot and since backtrace looks a bit
>> different from what I have in memory, I post this in a new thread.
>> See thread "Blocked tasks on 3.15.1" for a discussion of previous hang
>> issues.
> 
> Probably good to add some basic information on the filesystem:
> 
Do you have compression enabled?  I wasn't able to nail down the 3.15.1
hang before vacation attacked me, but I'm hoping to track it down today.

-chris

Re: BTRFS hang with 3.16-rc5

[Martin Steigerwald]

Am Montag, 14. Juli 2014, 16:12:22 schrieb Chris Mason:
> On 07/14/2014 11:10 AM, Martin Steigerwald wrote:
> > Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
> >> Hi!
> >> 
> >> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several days of
> >> usage, with 3-16-rc5 I had a hang again. Less than a hour since booting
> >> it.
> >> 
> >> Since the hang bug I and others had with 3.15 and upto 3.16-rc2 usually
> >> didn´t happen that quickly after boot and since backtrace looks a bit
> >> different from what I have in memory, I post this in a new thread.
> >> See thread "Blocked tasks on 3.15.1" for a discussion of previous hang
> >> issues.
> > 
> > Probably good to add some basic information on the filesystem:
> Do you have compression enabled?  I wasn't able to nail down the 3.15.1
> hang before vacation attacked me, but I'm hoping to track it down today.

Yes. I have.

It just hung again while I was playing PlaneShift.

Back to 3.16-rc4 as rc5 seems to be broke here.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BTRFS hang with 3.16-rc5

[Martin Steigerwald]

Am Montag, 14. Juli 2014, 17:51:37 schrieben Sie:
> Martin Steigerwald posted on Mon, 14 Jul 2014 17:10:30 +0200 as excerpted:
> > Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
> >> Hi!
> >> 
> >> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several days
> >> of usage, with 3-16-rc5 I had a hang again. Less than a hour since
> >> booting it.
> >> 
> >> Since the hang bug I and others had with 3.15 and upto 3.16-rc2 usually
> >> didn´t happen that quickly after boot and since backtrace looks a bit
> >> different from what I have in memory, I post this in a new thread.
> >> See thread "Blocked tasks on 3.15.1" for a discussion of previous hang
> >> issues.
> > 
> > Probably good to add some basic information on the filesystem: [...]
> > 
> > I think I switched it to skinny extents, but I am not completely sure.
> > 
> > 
> > Dual SSD BTRFS RAID 1
> > 
> > No snapshots at the moment. And plenty of space free.
> 
> FWIW, I've been updating every few days, running two different snapshots
> between 3.16-rc4 and rc5, and now rc5 itself, and haven't noticed this
> issue.  Dual SSDs btrfs raid1, but partitioned up so the biggest
> partition is under 50 GB.
> 
> I've been doing some very heavy (gentoo) package building and shuffling
> around the last few days trying out the kde-frameworks/workspaces-5
> stuff, then deciding it's not yet ready for me, too, so I've been working
> it a bit, too.
> 
> But certainly YMMV, and I did have some nasty trouble a few weeks ago,
> which triggered a btrfs restore and then a mkfs on a couple of the
> filesystems.  But nothing at all strange with btrfs since then, or
> really, for awhile before that either.

Do you have compression enabled?

I have:

/dev/mapper/msata-home /home btrfs rw,noatime,compress=lzo,ssd,space_cache 0 0


-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BTRFS hang with 3.16-rc5

[Duncan]

Martin Steigerwald posted on Tue, 15 Jul 2014 00:03:21 +0200 as excerpted:

> Am Montag, 14. Juli 2014, 17:51:37 schrieben Sie:
>> Martin Steigerwald posted on Mon, 14 Jul 2014 17:10:30 +0200 as
>> excerpted:
>> > Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
>> >> 
>> >> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several
>> >> days of usage, with 3-16-rc5 I had a hang again. Less than a hour
>> >> since booting it.
>> > 
>> > Probably good to add some basic information on the filesystem: [...]
>> > 
>> > I think I switched it to skinny extents, but I am not completely
>> > sure.

BTW, when I mount a btrfs with skinny extents, the kernel log says so.  I 
don't know if it has done that since skinny extents were introduced, but 
it certainly is now, and I think it did with 3.15 as well.

So if you look at dmesg after mount, it should say so if you have skinny 
extents.  Otherwise you don't (unless there's some sort of bug).

>> > Dual SSD BTRFS RAID 1
>> > 
>> > No snapshots at the moment. And plenty of space free.
>> 
>> FWIW, I've been updating every few days, running two different
>> snapshots between 3.16-rc4 and rc5, and now rc5 itself, and haven't
>> noticed this issue.  Dual SSDs btrfs raid1, but partitioned up so the
>> biggest partition is under 50 GB.
>> 
>> I've been doing some very heavy (gentoo) package building and shuffling
>> around the last few days trying out the kde-frameworks/workspaces-5
>> stuff, then deciding it's not yet ready for me, too, so I've been
>> working it a bit, too. I did have some nasty trouble a few weeks ago,
>> which triggered a btrfs restore and then a mkfs on a couple of the
>> filesystems.  But nothing at all strange with btrfs since then, or
>> really, for awhile before that either.
> 
> Do you have compression enabled?
> 
> I have:
> 
> /dev/mapper/msata-home /home btrfs
> rw,noatime,compress=lzo,ssd,space_cache 0 0

Yes.

But I am *NOT* doing anything with device-mapper.  I don't have it turned 
on in my kernel, either.

Also, I do run autodefrag too, so nothing should get seriously 
fragmented.  (Fragmentation isn't the big deal on SSD that it is on 
spinning rust, but I still don't want it, and I don't have any of the gig-
sized internal-write-pattern files to worry about on the SSDs, so...)

No snapshots for either of us, and I don't run quotas either.  I'm 
running skinny-metadata, 16-KiB nodes, extref (64 Ki hardlinks, default), 
and no-holes, features.  I do fresh mkfs-and-restores every few months to 
get the latest features and minimize the chance of any lingering effects 
from now-fixed bugs.

I don't particularly trust devmapper, particularly with something not 
entirely stable such as btrfs on top, so naturally that's what I'd 
suspect.  The relatively small filesystems might help, too, and I believe 
lack of quotas and snapshotting lowers the stress on the filesystem too, 
as does the fact that I don't have any of those big internal-write-
pattern files to worry about.  The combination of all of them may have a 
lot to do with my relative lack of problems compared to some of the other 
regulars.  The couple bad umounts and subsequent refusal to mounts 
necessitating a btrfs restore of two filesystems some weeks ago was the 
first actual experience with troubleshooting I had with btrfs; well, 
since I took it off the bad caps-mobo about 2.5 years ago and didn't try 
btrfs again until the SSDs I guess a year and a half ago, anyway.


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS hang with 3.16-rc5

[Chris Mason]

On 07/14/2014 05:58 PM, Martin Steigerwald wrote:
> Am Montag, 14. Juli 2014, 16:12:22 schrieb Chris Mason:
>> On 07/14/2014 11:10 AM, Martin Steigerwald wrote:
>>> Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
>>>> Hi!
>>>>
>>>> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several days of
>>>> usage, with 3-16-rc5 I had a hang again. Less than a hour since booting
>>>> it.
>>>>
>>>> Since the hang bug I and others had with 3.15 and upto 3.16-rc2 usually
>>>> didn´t happen that quickly after boot and since backtrace looks a bit
>>>> different from what I have in memory, I post this in a new thread.
>>>> See thread "Blocked tasks on 3.15.1" for a discussion of previous hang
>>>> issues.
>>>
>>> Probably good to add some basic information on the filesystem:
>> Do you have compression enabled?  I wasn't able to nail down the 3.15.1
>> hang before vacation attacked me, but I'm hoping to track it down today.
> 
> Yes. I have.
> 
> It just hung again while I was playing PlaneShift.
> 
> Back to 3.16-rc4 as rc5 seems to be broke here.

The btrfs hang you're hitting goes back to 3.15.  So 3.16-rc4 vs rc5
shouldn't be a factor.  Are you hitting other problems with 3.16?

-chris

Re: BTRFS hang with 3.16-rc5

[Martin Steigerwald]

Am Dienstag, 15. Juli 2014, 09:21:40 schrieb Chris Mason:
> On 07/14/2014 05:58 PM, Martin Steigerwald wrote:
> > Am Montag, 14. Juli 2014, 16:12:22 schrieb Chris Mason:
> >> On 07/14/2014 11:10 AM, Martin Steigerwald wrote:
> >>> Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
> >>>> Hi!
> >>>> 
> >>>> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several days
> >>>> of
> >>>> usage, with 3-16-rc5 I had a hang again. Less than a hour since booting
> >>>> it.
> >>>> 
> >>>> Since the hang bug I and others had with 3.15 and upto 3.16-rc2 usually
> >>>> didn´t happen that quickly after boot and since backtrace looks a bit
> >>>> different from what I have in memory, I post this in a new thread.
> >>>> See thread "Blocked tasks on 3.15.1" for a discussion of previous hang
> >>>> issues.
> >>> 
> >>> Probably good to add some basic information on the filesystem:
> >> Do you have compression enabled?  I wasn't able to nail down the 3.15.1
> >> hang before vacation attacked me, but I'm hoping to track it down today.
> > 
> > Yes. I have.
> > 
> > It just hung again while I was playing PlaneShift.
> > 
> > Back to 3.16-rc4 as rc5 seems to be broke here.
> 
> The btrfs hang you're hitting goes back to 3.15.  So 3.16-rc4 vs rc5
> shouldn't be a factor.  Are you hitting other problems with 3.16?

So far for this day 3.16-rc4 behaves nicely. With 3.16-rc5 I had a BTRFS hang 
twice yesterday. 3.16-rc4 before also behaved nicely for several days or well 
about a week here.

So what I see here for now is: 3.15 until 3-16-rc2 an occassional hang, 3.16-
rc3 and 3.16-rc4 okay, 3.16-rc5 two hangs in one day and then I stopped using 
it as I use this laptop for production :).

This is on ThinkPad T520 with Sandybrige and 16 GiB RAM.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BTRFS hang with 3.16-rc5

[Martin Steigerwald]

Am Dienstag, 15. Juli 2014, 09:21:40 schrieb Chris Mason:
> On 07/14/2014 05:58 PM, Martin Steigerwald wrote:
> > Am Montag, 14. Juli 2014, 16:12:22 schrieb Chris Mason:
> >> On 07/14/2014 11:10 AM, Martin Steigerwald wrote:
> >>> Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
> >>>> Hi!
> >>>> 
> >>>> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several days
> >>>> of
> >>>> usage, with 3-16-rc5 I had a hang again. Less than a hour since booting
> >>>> it.
> >>>> 
> >>>> Since the hang bug I and others had with 3.15 and upto 3.16-rc2 usually
> >>>> didn´t happen that quickly after boot and since backtrace looks a bit
> >>>> different from what I have in memory, I post this in a new thread.
> >>>> See thread "Blocked tasks on 3.15.1" for a discussion of previous hang
> >>>> issues.
> >>> 
> >>> Probably good to add some basic information on the filesystem:
> >> Do you have compression enabled?  I wasn't able to nail down the 3.15.1
> >> hang before vacation attacked me, but I'm hoping to track it down today.
> > 
> > Yes. I have.
> > 
> > It just hung again while I was playing PlaneShift.
> > 
> > Back to 3.16-rc4 as rc5 seems to be broke here.
> 
> The btrfs hang you're hitting goes back to 3.15.  So 3.16-rc4 vs rc5
> shouldn't be a factor.  Are you hitting other problems with 3.16?

On this system it is a matter.

3.16-rc5: Two hangs in one day

3.16-rc4: No hang so far with three days uptime (well with hibernation cycles 
in between)

So easy observation for me: 3.16-rc4 fine, 3.16-rc5 broke.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BTRFS hang with 3.16-rc5

[Chris Mason]

On 07/18/2014 03:51 AM, Martin Steigerwald wrote:
> Am Dienstag, 15. Juli 2014, 09:21:40 schrieb Chris Mason:
>> On 07/14/2014 05:58 PM, Martin Steigerwald wrote:
>>> Am Montag, 14. Juli 2014, 16:12:22 schrieb Chris Mason:
>>>> On 07/14/2014 11:10 AM, Martin Steigerwald wrote:
>>>>> Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
>>>>>> Hi!
>>>>>>
>>>>>> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several days
>>>>>> of
>>>>>> usage, with 3-16-rc5 I had a hang again. Less than a hour since booting
>>>>>> it.
>>>>>>
>>>>>> Since the hang bug I and others had with 3.15 and upto 3.16-rc2 usually
>>>>>> didn´t happen that quickly after boot and since backtrace looks a bit
>>>>>> different from what I have in memory, I post this in a new thread.
>>>>>> See thread "Blocked tasks on 3.15.1" for a discussion of previous hang
>>>>>> issues.
>>>>>
>>>>> Probably good to add some basic information on the filesystem:
>>>> Do you have compression enabled?  I wasn't able to nail down the 3.15.1
>>>> hang before vacation attacked me, but I'm hoping to track it down today.
>>>
>>> Yes. I have.
>>>
>>> It just hung again while I was playing PlaneShift.
>>>
>>> Back to 3.16-rc4 as rc5 seems to be broke here.
>>
>> The btrfs hang you're hitting goes back to 3.15.  So 3.16-rc4 vs rc5
>> shouldn't be a factor.  Are you hitting other problems with 3.16?
> 
> On this system it is a matter.
> 
> 3.16-rc5: Two hangs in one day
> 
> 3.16-rc4: No hang so far with three days uptime (well with hibernation cycles 
> in between)
> 
> So easy observation for me: 3.16-rc4 fine, 3.16-rc5 broke.

Can you please try this patch on rc5 and look for the printk:

diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index 3668048..8ab56df 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -8157,6 +8157,13 @@ void btrfs_destroy_inode(struct inode *inode)
 		spin_unlock(&root->fs_info->ordered_root_lock);
 	}
 
+	spin_lock(&root->fs_info->ordered_root_lock);
+	if (!list_empty(&BTRFS_I(inode)->ordered_operations)) {
+		list_del_init(&BTRFS_I(inode)->ordered_operations);
+printk(KERN_CRIT "racing inode deletion with ordered operations!!!!!!!!!!!\n");
+	}
+	spin_unlock(&root->fs_info->ordered_root_lock);
+
 	if (test_bit(BTRFS_INODE_HAS_ORPHAN_ITEM,
 		     &BTRFS_I(inode)->runtime_flags)) {
 		btrfs_info(root->fs_info, "inode %llu still on the orphan list",

Re: BTRFS hang with 3.16-rc5

[Martin Steigerwald]

Am Freitag, 18. Juli 2014, 09:36:06 schrieb Chris Mason:
> On 07/18/2014 03:51 AM, Martin Steigerwald wrote:
> > Am Dienstag, 15. Juli 2014, 09:21:40 schrieb Chris Mason:
> >> On 07/14/2014 05:58 PM, Martin Steigerwald wrote:
> >>> Am Montag, 14. Juli 2014, 16:12:22 schrieb Chris Mason:
> >>>> On 07/14/2014 11:10 AM, Martin Steigerwald wrote:
> >>>>> Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
> >>>>>> Hi!
> >>>>>> 
> >>>>>> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several
> >>>>>> days
> >>>>>> of
> >>>>>> usage, with 3-16-rc5 I had a hang again. Less than a hour since
> >>>>>> booting
> >>>>>> it.
> >>>>>> 
> >>>>>> Since the hang bug I and others had with 3.15 and upto 3.16-rc2
> >>>>>> usually
> >>>>>> didn´t happen that quickly after boot and since backtrace looks a bit
> >>>>>> different from what I have in memory, I post this in a new thread.
> >>>>>> See thread "Blocked tasks on 3.15.1" for a discussion of previous
> >>>>>> hang
> >>>>>> issues.
> >>>>> 
> >>>>> Probably good to add some basic information on the filesystem:
> >>>> Do you have compression enabled?  I wasn't able to nail down the 3.15.1
> >>>> hang before vacation attacked me, but I'm hoping to track it down
> >>>> today.
> >>> 
> >>> Yes. I have.
> >>> 
> >>> It just hung again while I was playing PlaneShift.
> >>> 
> >>> Back to 3.16-rc4 as rc5 seems to be broke here.
> >> 
> >> The btrfs hang you're hitting goes back to 3.15.  So 3.16-rc4 vs rc5
> >> shouldn't be a factor.  Are you hitting other problems with 3.16?
> > 
> > On this system it is a matter.
> > 
> > 3.16-rc5: Two hangs in one day
> > 
> > 3.16-rc4: No hang so far with three days uptime (well with hibernation
> > cycles in between)
> > 
> > So easy observation for me: 3.16-rc4 fine, 3.16-rc5 broke.
> 
> Can you please try this patch on rc5 and look for the printk:
> 
> diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
> index 3668048..8ab56df 100644
> --- a/fs/btrfs/inode.c
> +++ b/fs/btrfs/inode.c
> @@ -8157,6 +8157,13 @@ void btrfs_destroy_inode(struct inode *inode)
>  		spin_unlock(&root->fs_info->ordered_root_lock);
>  	}
> 
> +	spin_lock(&root->fs_info->ordered_root_lock);
> +	if (!list_empty(&BTRFS_I(inode)->ordered_operations)) {
> +		list_del_init(&BTRFS_I(inode)->ordered_operations);
> +printk(KERN_CRIT "racing inode deletion with ordered
> operations!!!!!!!!!!!\n"); +	}
> +	spin_unlock(&root->fs_info->ordered_root_lock);
> +
>  	if (test_bit(BTRFS_INODE_HAS_ORPHAN_ITEM,
>  		     &BTRFS_I(inode)->runtime_flags)) {
>  		btrfs_info(root->fs_info, "inode %llu still on the orphan list",

Did so and again got a hang.

No racing inodes tough:

merkaba:/boot> zgrep -i "racing inode" /var/log/syslog*
merkaba:/boot#1>

Built kernel seems right:

martin@merkaba:[…]> LANG=C grep -ir "racing inode" fs/btrfs
fs/btrfs/inode.c:printk(KERN_CRIT "racing inode deletion with ordered operations!!!!!!!!!!!\n");
Binary file fs/btrfs/inode.o matches
Binary file fs/btrfs/btrfs.o matches
Binary file fs/btrfs/btrfs.ko matches

Backtrace doesn´t seem to contain any function related to inodes.


Back to rc4 again for now.


These hangs seemed to occur first at writing several hundred MiB onto a
high speed SDHC card… yet, they persisted long after the write was finished,
upto to a point where I had to reboot cause machine hung on trying to
switch between tty7 (X11) and tty1 (for diagnosis).



Jul 19 19:29:11 merkaba kernel: [12346.692457] mmc0: new high speed SDHC card at address 0001
Jul 19 19:29:11 merkaba kernel: [12346.744276] mmcblk0: mmc0:0001 00000 29.2 GiB 
Jul 19 19:29:11 merkaba kernel: [12346.769850]  mmcblk0: p1
Jul 19 19:29:20 merkaba kernel: [12355.796267] FAT-fs (mmcblk0p1): utf8 is not a recommended IO charset for FAT filesystems, filesystem will be case se
nsitive!
Jul 19 19:29:20 merkaba kernel: [12355.805515] FAT-fs (mmcblk0p1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
Jul 19 19:33:27 merkaba kernel: [12602.162818] INFO: task btrfs-transacti:715 blocked for more than 120 seconds.
Jul 19 19:33:27 merkaba kernel: [12602.162826]       Tainted: G           O  3.16.0-rc5-tp520-btrfs-delrace+ #5
Jul 19 19:33:27 merkaba kernel: [12602.162827] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 19:33:27 merkaba kernel: [12602.162828] btrfs-transacti D ffff8800cf90e780     0   715      2 0x00000000
Jul 19 19:33:27 merkaba kernel: [12602.162834]  ffff880401ddbc80 0000000000000002 ffff880407fc0000 ffff880401ddbfd8
Jul 19 19:33:27 merkaba kernel: [12602.162836]  ffff8800cf90e240 0000000000013080 ffff8800cf90e240 7fffffffffffffff
Jul 19 19:33:27 merkaba kernel: [12602.162839]  ffff8804018efd28 0000000000000002 ffffffff81470270 ffff8804018efd20
Jul 19 19:33:27 merkaba kernel: [12602.162842] Call Trace:
Jul 19 19:33:27 merkaba kernel: [12602.162856]  [<ffffffff81470270>] ? michael_mic.part.6+0x1e/0x1e
Jul 19 19:33:27 merkaba kernel: [12602.162860]  [<ffffffff81470c70>] schedule+0x64/0x66
Jul 19 19:33:27 merkaba kernel: [12602.162862]  [<ffffffff8147029f>] schedule_timeout+0x2f/0x114
Jul 19 19:33:27 merkaba kernel: [12602.162867]  [<ffffffff81062c4d>] ? get_parent_ip+0xd/0x3c
Jul 19 19:33:27 merkaba kernel: [12602.162868]  [<ffffffff81062cf7>] ? preempt_count_add+0x7b/0x8e
Jul 19 19:33:27 merkaba kernel: [12602.162871]  [<ffffffff8147149d>] __wait_for_common+0x11e/0x163
Jul 19 19:33:27 merkaba kernel: [12602.162873]  [<ffffffff8147149d>] ? __wait_for_common+0x11e/0x163
Jul 19 19:33:27 merkaba kernel: [12602.162876]  [<ffffffff810647e9>] ? wake_up_state+0xd/0xd
Jul 19 19:33:27 merkaba kernel: [12602.162879]  [<ffffffff81471501>] wait_for_completion+0x1f/0x21
Jul 19 19:33:27 merkaba kernel: [12602.162916]  [<ffffffffc0423c16>] btrfs_wait_and_free_delalloc_work+0x11/0x23 [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.162933]  [<ffffffffc042bb55>] btrfs_run_ordered_operations+0x1ea/0x21a [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.162947]  [<ffffffffc0417392>] btrfs_commit_transaction+0x22/0x8df [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.162959]  [<ffffffffc0413fb6>] transaction_kthread+0x107/0x1b9 [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.162970]  [<ffffffffc0413eaf>] ? btrfs_cleanup_transaction+0x43a/0x43a [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.162973]  [<ffffffff81058002>] kthread+0xb2/0xba
Jul 19 19:33:27 merkaba kernel: [12602.162976]  [<ffffffff81470000>] ? iommu_prepare_identity_map+0x1d/0x19d
Jul 19 19:33:27 merkaba kernel: [12602.162978]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 19 19:33:27 merkaba kernel: [12602.162980]  [<ffffffff81473bec>] ret_from_fork+0x7c/0xb0
Jul 19 19:33:27 merkaba kernel: [12602.162982]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 19 19:33:27 merkaba kernel: [12602.162999] INFO: task QThread:2996 blocked for more than 120 seconds.
Jul 19 19:33:27 merkaba kernel: [12602.163000]       Tainted: G           O  3.16.0-rc5-tp520-btrfs-delrace+ #5
Jul 19 19:33:27 merkaba kernel: [12602.163001] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 19:33:27 merkaba kernel: [12602.163002] QThread         D ffff8803f71a6780     0  2996      1 0x00000000
Jul 19 19:33:27 merkaba kernel: [12602.163005]  ffff8802df447c18 0000000000000002 ffff880407f649b0 ffff8802df447fd8
Jul 19 19:33:27 merkaba kernel: [12602.163007]  ffff8803f71a6240 0000000000013080 ffff8803f71a6240 ffff88041e293080
Jul 19 19:33:27 merkaba kernel: [12602.163009]  ffff88041e5d5268 ffff8802df447cb0 0000000000000002 ffffffff810daffc
Jul 19 19:33:27 merkaba kernel: [12602.163012] Call Trace:
Jul 19 19:33:27 merkaba kernel: [12602.163015]  [<ffffffff810daffc>] ? wait_on_page_read+0x37/0x37
Jul 19 19:33:27 merkaba kernel: [12602.163017]  [<ffffffff81470c70>] schedule+0x64/0x66
Jul 19 19:33:27 merkaba kernel: [12602.163019]  [<ffffffff81470df7>] io_schedule+0x57/0x76
Jul 19 19:33:27 merkaba kernel: [12602.163021]  [<ffffffff810db005>] sleep_on_page+0x9/0xd
Jul 19 19:33:27 merkaba kernel: [12602.163023]  [<ffffffff814711d7>] __wait_on_bit_lock+0x41/0x85
Jul 19 19:33:27 merkaba kernel: [12602.163025]  [<ffffffff810db0c2>] __lock_page+0x70/0x7c
Jul 19 19:33:27 merkaba kernel: [12602.163030]  [<ffffffff81070f79>] ? autoremove_wake_function+0x2f/0x2f
Jul 19 19:33:27 merkaba kernel: [12602.163045]  [<ffffffffc042c58f>] lock_page+0x1e/0x21 [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.163059]  [<ffffffffc042c58f>] ? lock_page+0x1e/0x21 [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.163073]  [<ffffffffc043086c>] extent_write_cache_pages.isra.21.constprop.42+0x1a7/0x2d9 [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.163076]  [<ffffffff814716d9>] ? mutex_unlock+0x11/0x13
Jul 19 19:33:27 merkaba kernel: [12602.163089]  [<ffffffffc0425ebd>] ? btrfs_file_write_iter+0x322/0x412 [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.163091]  [<ffffffff81062c4d>] ? get_parent_ip+0xd/0x3c
Jul 19 19:33:27 merkaba kernel: [12602.163105]  [<ffffffffc0430d94>] extent_writepages+0x46/0x57 [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.163119]  [<ffffffffc041b25a>] ? btrfs_submit_direct+0x3ef/0x3ef [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.163122]  [<ffffffff81075192>] ? cpuacct_account_field+0x55/0x5f
Jul 19 19:33:27 merkaba kernel: [12602.163134]  [<ffffffffc041960f>] btrfs_writepages+0x23/0x25 [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.163136]  [<ffffffff810e5381>] do_writepages+0x1b/0x24
Jul 19 19:33:27 merkaba kernel: [12602.163138]  [<ffffffff810dc6ad>] __filemap_fdatawrite_range+0x50/0x52
Jul 19 19:33:27 merkaba kernel: [12602.163140]  [<ffffffff810dc714>] filemap_fdatawrite_range+0xe/0x10
Jul 19 19:33:27 merkaba kernel: [12602.163154]  [<ffffffffc0424a26>] btrfs_sync_file+0x84/0x2bd [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.163156]  [<ffffffff8120e159>] ? __this_cpu_preempt_check+0x13/0x15
Jul 19 19:33:27 merkaba kernel: [12602.163159]  [<ffffffff81155fab>] vfs_fsync_range+0x1c/0x1e
Jul 19 19:33:27 merkaba kernel: [12602.163161]  [<ffffffff81155fc4>] vfs_fsync+0x17/0x19
Jul 19 19:33:27 merkaba kernel: [12602.163164]  [<ffffffff811561eb>] do_fsync+0x2c/0x45
Jul 19 19:33:27 merkaba kernel: [12602.163166]  [<ffffffff811563d1>] SyS_fsync+0xb/0xf
Jul 19 19:33:27 merkaba kernel: [12602.163168]  [<ffffffff81473e8b>] tracesys+0xdd/0xe2
Jul 19 19:33:27 merkaba kernel: [12602.163193] INFO: task kworker/u8:6:3861 blocked for more than 120 seconds.
Jul 19 19:33:27 merkaba kernel: [12602.163194]       Tainted: G           O  3.16.0-rc5-tp520-btrfs-delrace+ #5
Jul 19 19:33:27 merkaba kernel: [12602.163196] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 19:33:27 merkaba kernel: [12602.163196] kworker/u8:6    D ffff880071cccef0     0  3861      2 0x00000000
Jul 19 19:33:27 merkaba kernel: [12602.163202] Workqueue: writeback bdi_writeback_workfn (flush-btrfs-3)
Jul 19 19:33:27 merkaba kernel: [12602.163204]  ffff88029ad179b8 0000000000000002 ffff880407fc6240 ffff88029ad17fd8
Jul 19 19:33:27 merkaba kernel: [12602.163206]  ffff880071ccc9b0 0000000000013080 ffff880071ccc9b0 ffff88041e293080
Jul 19 19:33:27 merkaba kernel: [12602.163209]  ffff88041e5c0268 ffff88029ad17a50 0000000000000002 ffffffff810daffc
Jul 19 19:33:27 merkaba kernel: [12602.163211] Call Trace:
Jul 19 19:33:27 merkaba kernel: [12602.163213]  [<ffffffff810daffc>] ? wait_on_page_read+0x37/0x37
Jul 19 19:33:27 merkaba kernel: [12602.163216]  [<ffffffff81470c70>] schedule+0x64/0x66
Jul 19 19:33:27 merkaba kernel: [12602.163218]  [<ffffffff81470df7>] io_schedule+0x57/0x76
Jul 19 19:33:27 merkaba kernel: [12602.163219]  [<ffffffff810db005>] sleep_on_page+0x9/0xd
Jul 19 19:33:27 merkaba kernel: [12602.163222]  [<ffffffff814711d7>] __wait_on_bit_lock+0x41/0x85
Jul 19 19:33:27 merkaba kernel: [12602.163224]  [<ffffffff810db0c2>] __lock_page+0x70/0x7c
Jul 19 19:33:27 merkaba kernel: [12602.163226]  [<ffffffff81070f79>] ? autoremove_wake_function+0x2f/0x2f
Jul 19 19:33:27 merkaba kernel: [12602.163242]  [<ffffffffc042c58f>] lock_page+0x1e/0x21 [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.163255]  [<ffffffffc042c58f>] ? lock_page+0x1e/0x21 [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.163269]  [<ffffffffc043086c>] extent_write_cache_pages.isra.21.constprop.42+0x1a7/0x2d9 [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.163284]  [<ffffffffc0430d94>] extent_writepages+0x46/0x57 [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.163296]  [<ffffffffc041b25a>] ? btrfs_submit_direct+0x3ef/0x3ef [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.163298]  [<ffffffff81062cf4>] ? preempt_count_add+0x78/0x8e
Jul 19 19:33:27 merkaba kernel: [12602.163310]  [<ffffffffc041960f>] btrfs_writepages+0x23/0x25 [btrfs]
Jul 19 19:33:27 merkaba kernel: [12602.163312]  [<ffffffff810e5381>] do_writepages+0x1b/0x24
Jul 19 19:33:27 merkaba kernel: [12602.163315]  [<ffffffff81151d0d>] __writeback_single_inode+0x58/0x212
Jul 19 19:33:27 merkaba kernel: [12602.163317]  [<ffffffff81152c0f>] writeback_sb_inodes+0x1e9/0x32e
Jul 19 19:33:27 merkaba kernel: [12602.163319]  [<ffffffff81152dce>] __writeback_inodes_wb+0x7a/0xb0
Jul 19 19:33:27 merkaba kernel: [12602.163321]  [<ffffffff81152f1a>] wb_writeback+0x116/0x270
Jul 19 19:33:27 merkaba kernel: [12602.163323]  [<ffffffff811532c9>] bdi_writeback_workfn+0x171/0x313
Jul 19 19:33:27 merkaba kernel: [12602.163327]  [<ffffffff81052d7c>] process_one_work+0x16f/0x2b8
Jul 19 19:33:27 merkaba kernel: [12602.163329]  [<ffffffff81053626>] worker_thread+0x27b/0x32e
Jul 19 19:33:27 merkaba kernel: [12602.163332]  [<ffffffff810533ab>] ? cancel_delayed_work_sync+0x10/0x10
Jul 19 19:33:27 merkaba kernel: [12602.163334]  [<ffffffff81058002>] kthread+0xb2/0xba
Jul 19 19:33:27 merkaba kernel: [12602.163336]  [<ffffffff81470000>] ? iommu_prepare_identity_map+0x1d/0x19d
Jul 19 19:33:27 merkaba kernel: [12602.163338]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 19 19:33:27 merkaba kernel: [12602.163340]  [<ffffffff81473bec>] ret_from_fork+0x7c/0xb0
Jul 19 19:33:27 merkaba kernel: [12602.163342]  [<ffffffff81057f50>] ? __kthread_parkme+0x62/0x62
Jul 19 19:33:27 merkaba kernel: [12602.163344] INFO: task kworker/u8:7:3987 blocked for more than 120 seconds.
Jul 19 19:33:27 merkaba kernel: [12602.163346]       Tainted: G           O  3.16.0-rc5-tp520-btrfs-delrace+ #5
Jul 19 19:33:27 merkaba kernel: [12602.163347] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 19 19:33:27 merkaba kernel: [12602.163348] kworker/u8:7    D ffff8803ce039dd0     0  3987      2 0x00000000


Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BTRFS hang with 3.16-rc5

[Chris Mason]

On 07/19/2014 01:59 PM, Martin Steigerwald wrote:
> Am Freitag, 18. Juli 2014, 09:36:06 schrieb Chris Mason:
>> On 07/18/2014 03:51 AM, Martin Steigerwald wrote:
>>> Am Dienstag, 15. Juli 2014, 09:21:40 schrieb Chris Mason:
>>>> On 07/14/2014 05:58 PM, Martin Steigerwald wrote:
>>>>> Am Montag, 14. Juli 2014, 16:12:22 schrieb Chris Mason:
>>>>>> On 07/14/2014 11:10 AM, Martin Steigerwald wrote:
>>>>>>> Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several
>>>>>>>> days
>>>>>>>> of
>>>>>>>> usage, with 3-16-rc5 I had a hang again. Less than a hour since
>>>>>>>> booting
>>>>>>>> it.
>>>>>>>>
>>>>>>>> Since the hang bug I and others had with 3.15 and upto 3.16-rc2
>>>>>>>> usually
>>>>>>>> didn´t happen that quickly after boot and since backtrace looks a bit
>>>>>>>> different from what I have in memory, I post this in a new thread.
>>>>>>>> See thread "Blocked tasks on 3.15.1" for a discussion of previous
>>>>>>>> hang
>>>>>>>> issues.
>>>>>>>
>>>>>>> Probably good to add some basic information on the filesystem:
>>>>>> Do you have compression enabled?  I wasn't able to nail down the 3.15.1
>>>>>> hang before vacation attacked me, but I'm hoping to track it down
>>>>>> today.
>>>>>
>>>>> Yes. I have.
>>>>>
>>>>> It just hung again while I was playing PlaneShift.
>>>>>
>>>>> Back to 3.16-rc4 as rc5 seems to be broke here.
>>>>
>>>> The btrfs hang you're hitting goes back to 3.15.  So 3.16-rc4 vs rc5
>>>> shouldn't be a factor.  Are you hitting other problems with 3.16?
>>>
>>> On this system it is a matter.
>>>
>>> 3.16-rc5: Two hangs in one day
>>>
>>> 3.16-rc4: No hang so far with three days uptime (well with hibernation
>>> cycles in between)
>>>
>>> So easy observation for me: 3.16-rc4 fine, 3.16-rc5 broke.
>>
>> Can you please try this patch on rc5 and look for the printk:
>>
>> diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
>> index 3668048..8ab56df 100644
>> --- a/fs/btrfs/inode.c
>> +++ b/fs/btrfs/inode.c
>> @@ -8157,6 +8157,13 @@ void btrfs_destroy_inode(struct inode *inode)
>>  		spin_unlock(&root->fs_info->ordered_root_lock);
>>  	}
>>
>> +	spin_lock(&root->fs_info->ordered_root_lock);
>> +	if (!list_empty(&BTRFS_I(inode)->ordered_operations)) {
>> +		list_del_init(&BTRFS_I(inode)->ordered_operations);
>> +printk(KERN_CRIT "racing inode deletion with ordered
>> operations!!!!!!!!!!!\n"); +	}
>> +	spin_unlock(&root->fs_info->ordered_root_lock);
>> +
>>  	if (test_bit(BTRFS_INODE_HAS_ORPHAN_ITEM,
>>  		     &BTRFS_I(inode)->runtime_flags)) {
>>  		btrfs_info(root->fs_info, "inode %llu still on the orphan list",
> 
> Did so and again got a hang.
> 
> No racing inodes tough:
> 
> merkaba:/boot> zgrep -i "racing inode" /var/log/syslog*
> merkaba:/boot#1>
> 
> Built kernel seems right:
> 
> martin@merkaba:[…]> LANG=C grep -ir "racing inode" fs/btrfs
> fs/btrfs/inode.c:printk(KERN_CRIT "racing inode deletion with ordered operations!!!!!!!!!!!\n");
> Binary file fs/btrfs/inode.o matches
> Binary file fs/btrfs/btrfs.o matches
> Binary file fs/btrfs/btrfs.ko matches
> 
> Backtrace doesn´t seem to contain any function related to inodes.
> 
> 
> Back to rc4 again for now.
> 
> 
> These hangs seemed to occur first at writing several hundred MiB onto a
> high speed SDHC card… yet, they persisted long after the write was finished,
> upto to a point where I had to reboot cause machine hung on trying to
> switch between tty7 (X11) and tty1 (for diagnosis).

Ok, this is definitely the same hang reported on 3.15.1.  Thanks for
giving the patch a try, I've got another long running test going this
weekend in hopes of triggering it here.

-chris

Re: BTRFS hang with 3.16-rc5

[Martin Steigerwald]

Am Samstag, 19. Juli 2014, 14:39:51 schrieb Chris Mason:
> On 07/19/2014 01:59 PM, Martin Steigerwald wrote:
> > Am Freitag, 18. Juli 2014, 09:36:06 schrieb Chris Mason:
> >> On 07/18/2014 03:51 AM, Martin Steigerwald wrote:
> >>> Am Dienstag, 15. Juli 2014, 09:21:40 schrieb Chris Mason:
> >>>> On 07/14/2014 05:58 PM, Martin Steigerwald wrote:
> >>>>> Am Montag, 14. Juli 2014, 16:12:22 schrieb Chris Mason:
> >>>>>> On 07/14/2014 11:10 AM, Martin Steigerwald wrote:
> >>>>>>> Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
> >>>>>>>> Hi!
> >>>>>>>> 
> >>>>>>>> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several
> >>>>>>>> days
> >>>>>>>> of
> >>>>>>>> usage, with 3-16-rc5 I had a hang again. Less than a hour since
> >>>>>>>> booting
> >>>>>>>> it.
> >>>>>>>> 
> >>>>>>>> Since the hang bug I and others had with 3.15 and upto 3.16-rc2
> >>>>>>>> usually
> >>>>>>>> didn´t happen that quickly after boot and since backtrace looks a
> >>>>>>>> bit
> >>>>>>>> different from what I have in memory, I post this in a new thread.
> >>>>>>>> See thread "Blocked tasks on 3.15.1" for a discussion of previous
> >>>>>>>> hang
> >>>>>>>> issues.
> >>>>>>> 
> >>>>>>> Probably good to add some basic information on the filesystem:
> >>>>>> Do you have compression enabled?  I wasn't able to nail down the
> >>>>>> 3.15.1
> >>>>>> hang before vacation attacked me, but I'm hoping to track it down
> >>>>>> today.
> >>>>> 
> >>>>> Yes. I have.
> >>>>> 
> >>>>> It just hung again while I was playing PlaneShift.
> >>>>> 
> >>>>> Back to 3.16-rc4 as rc5 seems to be broke here.
> >>>> 
> >>>> The btrfs hang you're hitting goes back to 3.15.  So 3.16-rc4 vs rc5
> >>>> shouldn't be a factor.  Are you hitting other problems with 3.16?
> >>> 
> >>> On this system it is a matter.
> >>> 
> >>> 3.16-rc5: Two hangs in one day
> >>> 
> >>> 3.16-rc4: No hang so far with three days uptime (well with hibernation
> >>> cycles in between)
> >>> 
> >>> So easy observation for me: 3.16-rc4 fine, 3.16-rc5 broke.
> >> 
> >> Can you please try this patch on rc5 and look for the printk:
> >> 
> >> diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
> >> index 3668048..8ab56df 100644
> >> --- a/fs/btrfs/inode.c
> >> +++ b/fs/btrfs/inode.c
> >> @@ -8157,6 +8157,13 @@ void btrfs_destroy_inode(struct inode *inode)
> >> 
> >>  		spin_unlock(&root->fs_info->ordered_root_lock);
> >>  	
> >>  	}
> >> 
> >> +	spin_lock(&root->fs_info->ordered_root_lock);
> >> +	if (!list_empty(&BTRFS_I(inode)->ordered_operations)) {
> >> +		list_del_init(&BTRFS_I(inode)->ordered_operations);
> >> +printk(KERN_CRIT "racing inode deletion with ordered
> >> operations!!!!!!!!!!!\n"); +	}
> >> +	spin_unlock(&root->fs_info->ordered_root_lock);
> >> +
> >> 
> >>  	if (test_bit(BTRFS_INODE_HAS_ORPHAN_ITEM,
> >>  	
> >>  		     &BTRFS_I(inode)->runtime_flags)) {
> >>  		
> >>  		btrfs_info(root->fs_info, "inode %llu still on the orphan 
list",
> > 
> > Did so and again got a hang.
> > 
> > No racing inodes tough:
> > 
> > merkaba:/boot> zgrep -i "racing inode" /var/log/syslog*
> > merkaba:/boot#1>
> > 
> > Built kernel seems right:
> > 
> > martin@merkaba:[…]> LANG=C grep -ir "racing inode" fs/btrfs
> > fs/btrfs/inode.c:printk(KERN_CRIT "racing inode deletion with ordered
> > operations!!!!!!!!!!!\n"); Binary file fs/btrfs/inode.o matches
> > Binary file fs/btrfs/btrfs.o matches
> > Binary file fs/btrfs/btrfs.ko matches
> > 
> > Backtrace doesn´t seem to contain any function related to inodes.
> > 
> > 
> > Back to rc4 again for now.
> > 
> > 
> > These hangs seemed to occur first at writing several hundred MiB onto a
> > high speed SDHC card… yet, they persisted long after the write was
> > finished, upto to a point where I had to reboot cause machine hung on
> > trying to switch between tty7 (X11) and tty1 (for diagnosis).
> 
> Ok, this is definitely the same hang reported on 3.15.1.  Thanks for
> giving the patch a try, I've got another long running test going this
> weekend in hopes of triggering it here.

I found make-kpkg (from Debian kernel-package) trigger BTRFS hang quite 
reliably with 3.14 and 3.15 at least after some update. Often during running 
objcopy commands.

Example call:

make-kpkg -j4 --rootcmd fakeroot --initrd --append-to-version -tp520-btrfs-
delrace --revision 1 linux_image

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Martin Steigerwald]

Am Dienstag, 15. Juli 2014, 17:08:27 schrieb Martin Steigerwald:
> Am Dienstag, 15. Juli 2014, 09:21:40 schrieb Chris Mason:
> > On 07/14/2014 05:58 PM, Martin Steigerwald wrote:
> > > Am Montag, 14. Juli 2014, 16:12:22 schrieb Chris Mason:
> > >> On 07/14/2014 11:10 AM, Martin Steigerwald wrote:
> > >>> Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
> > >>>> Hi!
> > >>>> 
> > >>>> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several
> > >>>> days
> > >>>> of
> > >>>> usage, with 3-16-rc5 I had a hang again. Less than a hour since
> > >>>> booting
> > >>>> it.
> > >>>> 
> > >>>> Since the hang bug I and others had with 3.15 and upto 3.16-rc2
> > >>>> usually
> > >>>> didn´t happen that quickly after boot and since backtrace looks a bit
> > >>>> different from what I have in memory, I post this in a new thread.
> > >>>> See thread "Blocked tasks on 3.15.1" for a discussion of previous
> > >>>> hang
> > >>>> issues.
> > >>> 
> > >>> Probably good to add some basic information on the filesystem:
> > >> Do you have compression enabled?  I wasn't able to nail down the 3.15.1
> > >> hang before vacation attacked me, but I'm hoping to track it down
> > >> today.
> > > 
> > > Yes. I have.
> > > 
> > > It just hung again while I was playing PlaneShift.
> > > 
> > > Back to 3.16-rc4 as rc5 seems to be broke here.
> > 
> > The btrfs hang you're hitting goes back to 3.15.  So 3.16-rc4 vs rc5
> > shouldn't be a factor.  Are you hitting other problems with 3.16?
> 
> So far for this day 3.16-rc4 behaves nicely. With 3.16-rc5 I had a BTRFS
> hang twice yesterday. 3.16-rc4 before also behaved nicely for several days
> or well about a week here.

3.16-rc4 now hung as well…

Maybe it hangs more rarely, but it hangs as well it seems.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Chris Mason]

On 07/23/2014 06:47 PM, Martin Steigerwald wrote:
> Am Dienstag, 15. Juli 2014, 17:08:27 schrieb Martin Steigerwald:
>> Am Dienstag, 15. Juli 2014, 09:21:40 schrieb Chris Mason:
>>> On 07/14/2014 05:58 PM, Martin Steigerwald wrote:
>>>> Am Montag, 14. Juli 2014, 16:12:22 schrieb Chris Mason:
>>>>> On 07/14/2014 11:10 AM, Martin Steigerwald wrote:
>>>>>> Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
>>>>>>> Hi!
>>>>>>>
>>>>>>> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several
>>>>>>> days
>>>>>>> of
>>>>>>> usage, with 3-16-rc5 I had a hang again. Less than a hour since
>>>>>>> booting
>>>>>>> it.
>>>>>>>
>>>>>>> Since the hang bug I and others had with 3.15 and upto 3.16-rc2
>>>>>>> usually
>>>>>>> didn´t happen that quickly after boot and since backtrace looks a bit
>>>>>>> different from what I have in memory, I post this in a new thread.
>>>>>>> See thread "Blocked tasks on 3.15.1" for a discussion of previous
>>>>>>> hang
>>>>>>> issues.
>>>>>>
>>>>>> Probably good to add some basic information on the filesystem:
>>>>> Do you have compression enabled?  I wasn't able to nail down the 3.15.1
>>>>> hang before vacation attacked me, but I'm hoping to track it down
>>>>> today.
>>>>
>>>> Yes. I have.
>>>>
>>>> It just hung again while I was playing PlaneShift.
>>>>
>>>> Back to 3.16-rc4 as rc5 seems to be broke here.
>>>
>>> The btrfs hang you're hitting goes back to 3.15.  So 3.16-rc4 vs rc5
>>> shouldn't be a factor.  Are you hitting other problems with 3.16?
>>
>> So far for this day 3.16-rc4 behaves nicely. With 3.16-rc5 I had a BTRFS
>> hang twice yesterday. 3.16-rc4 before also behaved nicely for several days
>> or well about a week here.
> 
> 3.16-rc4 now hung as well…

Liu Bo has a promising patch:

https://patchwork.kernel.org/patch/4618421/

Please give it a shot.  There's a second deadlock reading the free space
cache, I'm still working on that one too.

-chris

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Martin Steigerwald]

Am Donnerstag, 24. Juli 2014, 10:58:51 schrieb Chris Mason:
> On 07/23/2014 06:47 PM, Martin Steigerwald wrote:
> > Am Dienstag, 15. Juli 2014, 17:08:27 schrieb Martin Steigerwald:
> >> Am Dienstag, 15. Juli 2014, 09:21:40 schrieb Chris Mason:
> >>> On 07/14/2014 05:58 PM, Martin Steigerwald wrote:
> >>>> Am Montag, 14. Juli 2014, 16:12:22 schrieb Chris Mason:
> >>>>> On 07/14/2014 11:10 AM, Martin Steigerwald wrote:
> >>>>>> Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
> >>>>>>> Hi!
> >>>>>>> 
> >>>>>>> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several
> >>>>>>> days
> >>>>>>> of
> >>>>>>> usage, with 3-16-rc5 I had a hang again. Less than a hour since
> >>>>>>> booting
> >>>>>>> it.
> >>>>>>> 
> >>>>>>> Since the hang bug I and others had with 3.15 and upto 3.16-rc2
> >>>>>>> usually
> >>>>>>> didn´t happen that quickly after boot and since backtrace looks a
> >>>>>>> bit
> >>>>>>> different from what I have in memory, I post this in a new thread.
> >>>>>>> See thread "Blocked tasks on 3.15.1" for a discussion of previous
> >>>>>>> hang
> >>>>>>> issues.
> >>>>>> 
> >>>>>> Probably good to add some basic information on the filesystem:
> >>>>> Do you have compression enabled?  I wasn't able to nail down the
> >>>>> 3.15.1
> >>>>> hang before vacation attacked me, but I'm hoping to track it down
> >>>>> today.
> >>>> 
> >>>> Yes. I have.
> >>>> 
> >>>> It just hung again while I was playing PlaneShift.
> >>>> 
> >>>> Back to 3.16-rc4 as rc5 seems to be broke here.
> >>> 
> >>> The btrfs hang you're hitting goes back to 3.15.  So 3.16-rc4 vs rc5
> >>> shouldn't be a factor.  Are you hitting other problems with 3.16?
> >> 
> >> So far for this day 3.16-rc4 behaves nicely. With 3.16-rc5 I had a BTRFS
> >> hang twice yesterday. 3.16-rc4 before also behaved nicely for several
> >> days
> >> or well about a week here.
> > 
> > 3.16-rc4 now hung as well…
> 
> Liu Bo has a promising patch:
> 
> https://patchwork.kernel.org/patch/4618421/
> 
> Please give it a shot.  There's a second deadlock reading the free space
> cache, I'm still working on that one too.

Okay, I reverted your printk patch and applied this on on git linus git.

Lets see how this works.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Martin Steigerwald]

Am Donnerstag, 24. Juli 2014, 10:58:51 schrieb Chris Mason:
> On 07/23/2014 06:47 PM, Martin Steigerwald wrote:
> > Am Dienstag, 15. Juli 2014, 17:08:27 schrieb Martin Steigerwald:
> >> Am Dienstag, 15. Juli 2014, 09:21:40 schrieb Chris Mason:
> >>> On 07/14/2014 05:58 PM, Martin Steigerwald wrote:
> >>>> Am Montag, 14. Juli 2014, 16:12:22 schrieb Chris Mason:
> >>>>> On 07/14/2014 11:10 AM, Martin Steigerwald wrote:
> >>>>>> Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
> >>>>>>> Hi!
> >>>>>>> 
> >>>>>>> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several
> >>>>>>> days
> >>>>>>> of
> >>>>>>> usage, with 3-16-rc5 I had a hang again. Less than a hour since
> >>>>>>> booting
> >>>>>>> it.
> >>>>>>> 
> >>>>>>> Since the hang bug I and others had with 3.15 and upto 3.16-rc2
> >>>>>>> usually
> >>>>>>> didn´t happen that quickly after boot and since backtrace looks a
> >>>>>>> bit
> >>>>>>> different from what I have in memory, I post this in a new thread.
> >>>>>>> See thread "Blocked tasks on 3.15.1" for a discussion of previous
> >>>>>>> hang
> >>>>>>> issues.
> >>>>>> 
> >>>>>> Probably good to add some basic information on the filesystem:
> >>>>> Do you have compression enabled?  I wasn't able to nail down the
> >>>>> 3.15.1
> >>>>> hang before vacation attacked me, but I'm hoping to track it down
> >>>>> today.
> >>>> 
> >>>> Yes. I have.
> >>>> 
> >>>> It just hung again while I was playing PlaneShift.
> >>>> 
> >>>> Back to 3.16-rc4 as rc5 seems to be broke here.
> >>> 
> >>> The btrfs hang you're hitting goes back to 3.15.  So 3.16-rc4 vs rc5
> >>> shouldn't be a factor.  Are you hitting other problems with 3.16?
> >> 
> >> So far for this day 3.16-rc4 behaves nicely. With 3.16-rc5 I had a BTRFS
> >> hang twice yesterday. 3.16-rc4 before also behaved nicely for several
> >> days
> >> or well about a week here.
> > 
> > 3.16-rc4 now hung as well…
> 
> Liu Bo has a promising patch:
> 
> https://patchwork.kernel.org/patch/4618421/
> 
> Please give it a shot.  There's a second deadlock reading the free space
> cache, I'm still working on that one too.

Now running 3.16-rc6 + current git + this patch.

It may take some time tough cause during compiling the kernel BTRFS hung 
again, which caused loss of KDE Baloo desktop search file index and parts of a 
mail I wrote in KMail.

Since the patch mentioned ENOSPC issues but the filesystem has enough free 
space according to df I shrunk the trees with

btrfs balance start -musage=50 /home
btrfs balance start -musage=50 /home


merkaba:~> btrfs fi sh /home                   
Label: 'home'  uuid: […]
        Total devices 2 FS bytes used 124.05GiB
        devid    1 size 160.00GiB used 150.00GiB path /dev/mapper/msata-home
        devid    2 size 160.00GiB used 150.00GiB path /dev/dm-3


As I bet that the error is more likely to happen when trees occupy all space, 
it may take some time till it happens again.

Well its growing slowly already:

merkaba:~> btrfs fi df /home
Data, RAID1: total=146.97GiB, used=121.84GiB
System, RAID1: total=32.00MiB, used=48.00KiB
Metadata, RAID1: total=4.00GiB, used=2.62GiB
unknown, single: total=512.00MiB, used=0.00
merkaba:~> btrfs fi sh /home
Label: 'home'  uuid: […]
        Total devices 2 FS bytes used 124.46GiB
        devid    1 size 160.00GiB used 151.00GiB path /dev/dm-0
        devid    2 size 160.00GiB used 151.00GiB path /dev/mapper/sata-home

Btrfs v3.14.1


I wonder why ENOSPC conditions happens with that much space inside trees free. 
Were they just too fragmented?

To me

merkaba:~> LANG=C df -hT /home
Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/dm-0      btrfs  320G  249G   69G  79% /home

is a quite healthy free space margin.


Well, lets see how this goes.

I hope it can be fixed soon as it causes loss of recently saved data and 
generally locks up a machine running KDE desktop quite quickly on a BTRFS 
hang.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Chris Mason]

On 07/24/2014 02:49 PM, Martin Steigerwald wrote:
> Am Donnerstag, 24. Juli 2014, 10:58:51 schrieb Chris Mason:
>> On 07/23/2014 06:47 PM, Martin Steigerwald wrote:
>>> Am Dienstag, 15. Juli 2014, 17:08:27 schrieb Martin Steigerwald:
>>>> Am Dienstag, 15. Juli 2014, 09:21:40 schrieb Chris Mason:
>>>>> On 07/14/2014 05:58 PM, Martin Steigerwald wrote:
>>>>>> Am Montag, 14. Juli 2014, 16:12:22 schrieb Chris Mason:
>>>>>>> On 07/14/2014 11:10 AM, Martin Steigerwald wrote:
>>>>>>>> Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
>>>>>>>>> Hi!
>>>>>>>>>
>>>>>>>>> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several
>>>>>>>>> days
>>>>>>>>> of
>>>>>>>>> usage, with 3-16-rc5 I had a hang again. Less than a hour since
>>>>>>>>> booting
>>>>>>>>> it.
>>>>>>>>>
>>>>>>>>> Since the hang bug I and others had with 3.15 and upto 3.16-rc2
>>>>>>>>> usually
>>>>>>>>> didn´t happen that quickly after boot and since backtrace looks a
>>>>>>>>> bit
>>>>>>>>> different from what I have in memory, I post this in a new thread.
>>>>>>>>> See thread "Blocked tasks on 3.15.1" for a discussion of previous
>>>>>>>>> hang
>>>>>>>>> issues.
>>>>>>>>
>>>>>>>> Probably good to add some basic information on the filesystem:
>>>>>>> Do you have compression enabled?  I wasn't able to nail down the
>>>>>>> 3.15.1
>>>>>>> hang before vacation attacked me, but I'm hoping to track it down
>>>>>>> today.
>>>>>>
>>>>>> Yes. I have.
>>>>>>
>>>>>> It just hung again while I was playing PlaneShift.
>>>>>>
>>>>>> Back to 3.16-rc4 as rc5 seems to be broke here.
>>>>>
>>>>> The btrfs hang you're hitting goes back to 3.15.  So 3.16-rc4 vs rc5
>>>>> shouldn't be a factor.  Are you hitting other problems with 3.16?
>>>>
>>>> So far for this day 3.16-rc4 behaves nicely. With 3.16-rc5 I had a BTRFS
>>>> hang twice yesterday. 3.16-rc4 before also behaved nicely for several
>>>> days
>>>> or well about a week here.
>>>
>>> 3.16-rc4 now hung as well…
>>
>> Liu Bo has a promising patch:
>>
>> https://urldefense.proofpoint.com/v1/url?u=https://patchwork.kernel.org/patch/4618421/&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=6%2FL0lzzDhu0Y1hL9xm%2BQyA%3D%3D%0A&m=CJPREifRDOxlzhYeURx75h33LGU7YemJsNeLP%2FvXCv8%3D%0A&s=8fb0a70afce09530f16ea66a47d2af07966706b21281a7142d86256979013bab
>>
>> Please give it a shot.  There's a second deadlock reading the free space
>> cache, I'm still working on that one too.
> 
> Now running 3.16-rc6 + current git + this patch.
> 
> It may take some time tough cause during compiling the kernel BTRFS hung 
> again, which caused loss of KDE Baloo desktop search file index and parts of a 
> mail I wrote in KMail.
> 
> Since the patch mentioned ENOSPC issues but the filesystem has enough free 
> space according to df I shrunk the trees with

Thanks for giving it a try.  The ENOSPC mentioned here is looking for a
contiguous extent, so it's easily possible to trigger that enospc
without actually being full.

-chris

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Duncan]

Martin Steigerwald posted on Thu, 24 Jul 2014 20:49:37 +0200 as excerpted:

> It may take some time tough cause during compiling the kernel BTRFS hung
> again, which caused loss of KDE Baloo desktop search file index and
> parts of a mail I wrote in KMail.

Heh.  While I do run a kde(-lite) desktop, at least I don't have those 
problems to deal with.  As a gentooer I have the option to build kde 
without the semantic-desktop junk and I've taken that option, so no baloo 
or the like here.  And after the akonadified kmail lost one too many 
mails and I was going to need to reset the data store to retrieve them 
once again, I asked myself why I was putting up with it, after all, email 
is a decades old technology that should NOT be rocket-science any longer, 
and soon enough I was NOT putting up with it any longer, as I'd switched 
to claws-mail.  Actually, killing with fire kdepim and anything akonadi 
related was what allowed me to kill semantic-desktop as well instead of 
just run-time disabling it, since akonadi is part of the steaming pile.

And claws-mail has this nice option I didn't even know could be done on 
pop3 mail servers, too.  It downloads the mail for local reading, but 
keeps it on the server for a week (configurable) before final pop3 server-
side deletion, just in case you do crash after download and lose the 
local copy.  I've not actually had to take advantage of that feature yet, 
but it's sure nice to have, just in case. =:^)

Interesting this came up here just now, too, as there's a current xmodulo 
post about baloo and milou in kde4 and the carryover to kde-frameworks5 
and plasma5, too, with an ongoing discussion.

http://xmodulo.com/2014/07/kde-semantic-desktop-nepomuk-baloo.html

So fortunately, while I am a development version tester for of both kde 
and btrfs, the akonadi and semantic-desktop steaming-pile-of-.... is not 
something I have to worry about the stability of (or more precisely the 
lack thereof), while also testing a not yet fully stable btrfs at the 
same time.  Hopefully that'll continue to be the case in the claimed more 
modular kde-frameworks-5 era, because there's more than one way to ensure 
that I don't have to deal with that pile, and just as I suddenly found 
some other option for mail after using kmail since the kde2 era when it 
semantic-desktop-integrated without option, so my kde/plasma desktop, 
also since the kde2 era, can find itself going the same route locally, 
should it insist on going the same route globally.

Tho just as I did the akonadified kmail, I'll likely keep an open enough 
mind to try it.  <shrug>  Maybe it'll actually work this time, without 
eating up gigs of indexing space that has to be reset frequently due to 
something going wrong, to do it.  Time will tell...

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Nick Krause]

On Thu, Jul 24, 2014 at 10:32 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> Martin Steigerwald posted on Thu, 24 Jul 2014 20:49:37 +0200 as excerpted:
>
>> It may take some time tough cause during compiling the kernel BTRFS hung
>> again, which caused loss of KDE Baloo desktop search file index and
>> parts of a mail I wrote in KMail.
>
> Heh.  While I do run a kde(-lite) desktop, at least I don't have those
> problems to deal with.  As a gentooer I have the option to build kde
> without the semantic-desktop junk and I've taken that option, so no baloo
> or the like here.  And after the akonadified kmail lost one too many
> mails and I was going to need to reset the data store to retrieve them
> once again, I asked myself why I was putting up with it, after all, email
> is a decades old technology that should NOT be rocket-science any longer,
> and soon enough I was NOT putting up with it any longer, as I'd switched
> to claws-mail.  Actually, killing with fire kdepim and anything akonadi
> related was what allowed me to kill semantic-desktop as well instead of
> just run-time disabling it, since akonadi is part of the steaming pile.
>
> And claws-mail has this nice option I didn't even know could be done on
> pop3 mail servers, too.  It downloads the mail for local reading, but
> keeps it on the server for a week (configurable) before final pop3 server-
> side deletion, just in case you do crash after download and lose the
> local copy.  I've not actually had to take advantage of that feature yet,
> but it's sure nice to have, just in case. =:^)
>
> Interesting this came up here just now, too, as there's a current xmodulo
> post about baloo and milou in kde4 and the carryover to kde-frameworks5
> and plasma5, too, with an ongoing discussion.
>
> http://xmodulo.com/2014/07/kde-semantic-desktop-nepomuk-baloo.html
>
> So fortunately, while I am a development version tester for of both kde
> and btrfs, the akonadi and semantic-desktop steaming-pile-of-.... is not
> something I have to worry about the stability of (or more precisely the
> lack thereof), while also testing a not yet fully stable btrfs at the
> same time.  Hopefully that'll continue to be the case in the claimed more
> modular kde-frameworks-5 era, because there's more than one way to ensure
> that I don't have to deal with that pile, and just as I suddenly found
> some other option for mail after using kmail since the kde2 era when it
> semantic-desktop-integrated without option, so my kde/plasma desktop,
> also since the kde2 era, can find itself going the same route locally,
> should it insist on going the same route globally.
>
> Tho just as I did the akonadified kmail, I'll likely keep an open enough
> mind to try it.  <shrug>  Maybe it'll actually work this time, without
> eating up gigs of indexing space that has to be reset frequently due to
> something going wrong, to do it.  Time will tell...
>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hey Duncan and others ,
I have read this and this seems to need some working on.
If you want my help please ask , I am new to the kernel
so I may ask a dumb question or two but if that's fine with
you I have no problem helping out here. I would like
a log of printk statements leading to the hand if that's
not too much work in order for me to trace this back.
Cheers Nick

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Torbjørn]

On 07/24/2014 04:58 PM, Chris Mason wrote:
> <snip>
> Liu Bo has a promising patch:
>
> https://patchwork.kernel.org/patch/4618421/
>
> Please give it a shot.  There's a second deadlock reading the free space
> cache, I'm still working on that one too.
>
> -chris
I (as expected, my hang was with free space cache) still get hangs with 
this applied on top of 3.16-rc6.
Looking forward to the free space cache patch.

I have not been able to trigger the same hang as I had with 3.15 on any 
of the 3.16-rc6 kernels so far.

--
Torbjørn

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 5108 bytes --]

[this time, to the mailing list as well]

On Fri, Jul 25, 2014 at 09:02:44AM +0100, Hugo Mills wrote:
> On Thu, Jul 24, 2014 at 11:06:34PM -0400, Nick Krause wrote:
> > On Thu, Jul 24, 2014 at 10:32 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> [snip]
> > Hey Duncan and others ,
> > I have read this and this seems to need some working on.
> > If you want my help please ask , I am new to the kernel
> > so I may ask a dumb question or two but if that's fine with
> > you I have no problem helping out here. I would like
> > a log of printk statements leading to the hand if that's
> > not too much work in order for me to trace this back.
> 
>    Note that btrfs is complex -- there's something around 100k lines
> of code in it. My first piece of kernel work in btrfs was simply
> documenting the way that the on-disk data structures related to each
> other[1]. That on its own took me two to three weeks of solid
> full-time effort, reading the code to find where each structure was
> used and how its elements related to other structures. You can't just
> wander up and dive in without putting in the effort of learning first.
> Whilst people will help you (come over to #btrfs on Freenode for more
> real-time interaction), they can't do the basic work of sitting down
> and understanding the code in detail for you.
> 
>    Chris, who designed and wrote the filesystem, has spent the last
> couple of weeks tracking down this particular problem. Do you think
> it's appropriate to leap into the middle of the discussion on this
> subtle bug as someone with absolutely no experience in the area?
> 
>    Your first task is to reproduce the bug on your own machine. If you
> can do that, _then_ you might be able to start tracking down its
> cause. But I wouldn't recommend doing that, as (a) it's a nasty subtle
> bug, and (b) Chris seems to be close to tracking it down anyway.
> 
>    My recommendations for you, if you want to work on btrfs, are:
> 
>  * Build and install the latest kernel from Linus's git repo
> 
>  * Read and understand the user documentation [2]
> 
>  * Create one or several btrfs filesystems with different
>    configurations and learn how they work in userspace -- what are the
>    features, what are the problems you see? Actually use at least one
>    of the filesystems you created for real data in daily use (with
>    backups)
> 
>  * Build the userspace tools from git
> 
>  * Pick up one of the userspace projects from [3] and implement that.
>    If you pick the right one(s), you'll have to learn about some of
>    the internal structures of the FS anyway. Compile and test your
>    patch. If you're adding a new feature, write an automated xfstest
>    for it as well.
> 
>  * Get that patch accepted. This will probably involve a sequence of
>    revisions to it, multiple versions over a period of several weeks
>    or more, with a review process. You should also send your test to
>    xfstests and get that accepted.
> 
>  * Do the above again, until you get used to the processes involved,
>    and have demonstrated that you can work well with the other people
>    in the subsystem, and are generally producing useful and sane code.
>    It's all about trust -- can you be trusted to mostly do the right
>    thing? (So far on linux-kernel, you've rather demonstrated the
>    opposite: your intentions are good, but your execution leaves a lot
>    to be desired)
> 
>  * Use the documentation at [4], and the output of btrfs-debug-tree to
>    understand the internal structure of the FS
> 
>  * Pick up one of the smaller, more self-contained ideas from the
>    projects page [5] (say, [6] or [7]) and try to implement it. Again:
>    build, write test code, test thoroughly, submit patch for review,
>    modify as suggested by reviewers, and repeat as often as necessary
> 
>    Hugo.
> 
> [1] https://btrfs.wiki.kernel.org/index.php/Data_Structures
> [2] https://btrfs.wiki.kernel.org/index.php/Main_Page#Guides_and_usage_information
> [3] https://btrfs.wiki.kernel.org/index.php/Project_ideas#Userspace_tools_projects
> [4] https://btrfs.wiki.kernel.org/index.php/Main_Page#Developer_documentation
> [5] https://btrfs.wiki.kernel.org/index.php/Project_ideas
> [6] https://btrfs.wiki.kernel.org/index.php/Project_ideas#Cancellable_operations
> [7] https://btrfs.wiki.kernel.org/index.php/Project_ideas#Implement_new_FALLOC_FL_.2A_modes
> 
> -- 
> === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
>   PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
>   --- But people have always eaten people,  / what else is there to ---  
>          eat?  / If the Juju had meant us not to eat people / he         
>                      wouldn't have made us of meat.                      



-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
                        --- ORLY? IÄ! R'LYH! ---                        

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Martin Steigerwald]

Am Freitag, 25. Juli 2014, 02:32:17 schrieb Duncan:
> Martin Steigerwald posted on Thu, 24 Jul 2014 20:49:37 +0200 as excerpted:
> > It may take some time tough cause during compiling the kernel BTRFS hung
> > again, which caused loss of KDE Baloo desktop search file index and
> > parts of a mail I wrote in KMail.
> 
> Heh.  While I do run a kde(-lite) desktop, at least I don't have those
> problems to deal with.  As a gentooer I have the option to build kde
> without the semantic-desktop junk and I've taken that option, so no baloo
> or the like here.  And after the akonadified kmail lost one too many
> mails and I was going to need to reset the data store to retrieve them
> once again, I asked myself why I was putting up with it, after all, email
> is a decades old technology that should NOT be rocket-science any longer,
> and soon enough I was NOT putting up with it any longer, as I'd switched
> to claws-mail.  Actually, killing with fire kdepim and anything akonadi
> related was what allowed me to kill semantic-desktop as well instead of
> just run-time disabling it, since akonadi is part of the steaming pile.

Heh, I don´t want to make it a discussion about KDE, but I like to make a few 
points about it:

1) It was kmail save letter feature which saved me from *typing* all of the 
mail again. It still had a previous version open.

2) I didn´t see dataloss in KMail for months.

3) Since the recents developments (running stuff partly from Git) I am quite 
satisfied, IMAP performance / handling improved a lot. Even usable with our 
Exchange server now, and barely noticable with my own Dovecot IMAP setup.


It was a tough ride. It still haves rough edges. But it got a lot better to a 
point I am using it on all my systems again. I failed back to other mail 
clients at work for a moment cause IMAP access was barely usable.

There is still one data loss bug with filtering mails through CRM114 mailfilter 
spamfilter I didn´t dare to retry with most recent versions. But it may have 
been fixed already.

> Interesting this came up here just now, too, as there's a current xmodulo
> post about baloo and milou in kde4 and the carryover to kde-frameworks5
> and plasma5, too, with an ongoing discussion.
> 
> http://xmodulo.com/2014/07/kde-semantic-desktop-nepomuk-baloo.html

Baloo is a ton of a improvement. Only two thing I have with it at the moment 
is:

1) File indexing database is fragile in case of sudden write interruption… but 
here with BTRFS compressed extents corruption it may not even be at fault.

2) Sometimes krunner crashes while entering a search topic.

Other than that I have grown to love the marvellous mail address auto 
completion from all indexed mails, the almost instant full text mail search 
and the almost instant fulltext file search.

I suggest we continues this privately or on a KDE related list like kdepim-
users as its not really BTRFS related.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Torbjørn]

On 25. juli 2014 11:28, Liu Bo wrote:
> Hi Torbjørn,
>
> On Fri, Jul 25, 2014 at 06:51:44AM +0200, Torbjørn wrote:
>> On 07/24/2014 04:58 PM, Chris Mason wrote:
>>> <snip>
>>> Liu Bo has a promising patch:
>>>
>>> https://patchwork.kernel.org/patch/4618421/
>>>
>>> Please give it a shot.  There's a second deadlock reading the free space
>>> cache, I'm still working on that one too.
>>>
>>> -chris
>> I (as expected, my hang was with free space cache) still get hangs
>> with this applied on top of 3.16-rc6.
>> Looking forward to the free space cache patch.
>>
>> I have not been able to trigger the same hang as I had with 3.15 on
>> any of the 3.16-rc6 kernels so far.
> Seems that you can run into the hang quite stably, I have a stupid debugging
> patch here with adding a WARN_ON() to __load_free_space_inode(), would you
> please give it a shot and show us the output of dmesg?
>
> (This may make 'dmesg' a mess, but it will show who the hell holds the free
> space inode's page just at the time before hang)
>
> thanks,
> -liubo
>
>
> diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
> index 2b0a627..e364888 100644
> --- a/fs/btrfs/free-space-cache.c
> +++ b/fs/btrfs/free-space-cache.c
> @@ -350,6 +350,8 @@ static int io_ctl_prepare_pages(struct io_ctl *io_ctl,
> struct inode *inode,
>   	gfp_t mask = btrfs_alloc_write_mask(inode->i_mapping);
>   	int i;
>   
> +	WARN_ON(1);
> +
>   	for (i = 0; i < io_ctl->num_pages; i++) {
>   		page = find_or_create_page(inode->i_mapping, i, mask);
>   		if (!page) {

That patch did not apply cleanly on top of current linus master.
Not sure what branch you are at, so I added that WARN_ON(1); manually in 
free-space-cache.c.
I reverted all the other patches before manually applying this one.
If you need me to test from some other branch or with some other patch 
included, please let me know.

I'm doing a build now. Will run my rsync job and report back.

Thanks.
--
Torbjørn

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Torbjørn]

On 25. juli 2014 13:09, Torbjørn wrote:
> On 25. juli 2014 12:22, Torbjørn wrote:
>> On 25. juli 2014 11:28, Liu Bo wrote:
>>> Hi Torbjørn,
>>>
>>> On Fri, Jul 25, 2014 at 06:51:44AM +0200, Torbjørn wrote:
>>>> On 07/24/2014 04:58 PM, Chris Mason wrote:
>>>>> <snip>
>>>>> Liu Bo has a promising patch:
>>>>>
>>>>> https://patchwork.kernel.org/patch/4618421/
>>>>>
>>>>> Please give it a shot.  There's a second deadlock reading the free 
>>>>> space
>>>>> cache, I'm still working on that one too.
>>>>>
>>>>> -chris
>>>> I (as expected, my hang was with free space cache) still get hangs
>>>> with this applied on top of 3.16-rc6.
>>>> Looking forward to the free space cache patch.
>>>>
>>>> I have not been able to trigger the same hang as I had with 3.15 on
>>>> any of the 3.16-rc6 kernels so far.
>>> Seems that you can run into the hang quite stably, I have a stupid 
>>> debugging
>>> patch here with adding a WARN_ON() to __load_free_space_inode(), 
>>> would you
>>> please give it a shot and show us the output of dmesg?
>>>
>>> (This may make 'dmesg' a mess, but it will show who the hell holds 
>>> the free
>>> space inode's page just at the time before hang)
>>>
>>> thanks,
>>> -liubo
>>>
>>>
>>> diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
>>> index 2b0a627..e364888 100644
>>> --- a/fs/btrfs/free-space-cache.c
>>> +++ b/fs/btrfs/free-space-cache.c
>>> @@ -350,6 +350,8 @@ static int io_ctl_prepare_pages(struct io_ctl 
>>> *io_ctl,
>>> struct inode *inode,
>>>       gfp_t mask = btrfs_alloc_write_mask(inode->i_mapping);
>>>       int i;
>>>   +    WARN_ON(1);
>>> +
>>>       for (i = 0; i < io_ctl->num_pages; i++) {
>>>           page = find_or_create_page(inode->i_mapping, i, mask);
>>>           if (!page) {
>>
>> That patch did not apply cleanly on top of current linus master.
>> Not sure what branch you are at, so I added that WARN_ON(1); manually 
>> in free-space-cache.c.
>> I reverted all the other patches before manually applying this one.
>> If you need me to test from some other branch or with some other 
>> patch included, please let me know.
>>
>> I'm doing a build now. Will run my rsync job and report back.
>>
>> Thanks.
>> -- 
>> Torbjørn
> Back with results.
> That's output from dmesg. If you need more, I can send netconsole log 
> as well.
>
Seems like the list did not accept the huge email.
dmesg is available from: 
https://gist.github.com/anonymous/dba1f53211a1d482c3e0

--
Torbjørn

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Nick Krause]

Hey Hugo,
Thanks for the advice. I will look into this today :).
Cheers Nick

On Fri, Jul 25, 2014 at 4:02 AM, Hugo Mills <hugo@carfax.org.uk> wrote:
> On Thu, Jul 24, 2014 at 11:06:34PM -0400, Nick Krause wrote:
>> On Thu, Jul 24, 2014 at 10:32 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> [snip]
>> Hey Duncan and others ,
>> I have read this and this seems to need some working on.
>> If you want my help please ask , I am new to the kernel
>> so I may ask a dumb question or two but if that's fine with
>> you I have no problem helping out here. I would like
>> a log of printk statements leading to the hand if that's
>> not too much work in order for me to trace this back.
>
>    Note that btrfs is complex -- there's something around 100k lines
> of code in it. My first piece of kernel work in btrfs was simply
> documenting the way that the on-disk data structures related to each
> other[1]. That on its own took me two to three weeks of solid
> full-time effort, reading the code to find where each structure was
> used and how its elements related to other structures. You can't just
> wander up and dive in without putting in the effort of learning first.
> Whilst people will help you (come over to #btrfs on Freenode for more
> real-time interaction), they can't do the basic work of sitting down
> and understanding the code in detail for you.
>
>    Chris, who designed and wrote the filesystem, has spent the last
> couple of weeks tracking down this particular problem. Do you think
> it's appropriate to leap into the middle of the discussion on this
> subtle bug as someone with absolutely no experience in the area?
>
>    Your first task is to reproduce the bug on your own machine. If you
> can do that, _then_ you might be able to start tracking down its
> cause. But I wouldn't recommend doing that, as (a) it's a nasty subtle
> bug, and (b) Chris seems to be close to tracking it down anyway.
>
>    My recommendations for you, if you want to work on btrfs, are:
>
>  * Build and install the latest kernel from Linus's git repo
>
>  * Read and understand the user documentation [2]
>
>  * Create one or several btrfs filesystems with different
>    configurations and learn how they work in userspace -- what are the
>    features, what are the problems you see? Actually use at least one
>    of the filesystems you created for real data in daily use (with
>    backups)
>
>  * Build the userspace tools from git
>
>  * Pick up one of the userspace projects from [3] and implement that.
>    If you pick the right one(s), you'll have to learn about some of
>    the internal structures of the FS anyway. Compile and test your
>    patch. If you're adding a new feature, write an automated xfstest
>    for it as well.
>
>  * Get that patch accepted. This will probably involve a sequence of
>    revisions to it, multiple versions over a period of several weeks
>    or more, with a review process. You should also send your test to
>    xfstests and get that accepted.
>
>  * Do the above again, until you get used to the processes involved,
>    and have demonstrated that you can work well with the other people
>    in the subsystem, and are generally producing useful and sane code.
>    It's all about trust -- can you be trusted to mostly do the right
>    thing? (So far on linux-kernel, you've rather demonstrated the
>    opposite: your intentions are good, but your execution leaves a lot
>    to be desired)
>
>  * Use the documentation at [4], and the output of btrfs-debug-tree to
>    understand the internal structure of the FS
>
>  * Pick up one of the smaller, more self-contained ideas from the
>    projects page [5] (say, [6] or [7]) and try to implement it. Again:
>    build, write test code, test thoroughly, submit patch for review,
>    modify as suggested by reviewers, and repeat as often as necessary
>
>    Hugo.
>
> [1] https://btrfs.wiki.kernel.org/index.php/Data_Structures
> [2] https://btrfs.wiki.kernel.org/index.php/Main_Page#Guides_and_usage_information
> [3] https://btrfs.wiki.kernel.org/index.php/Project_ideas#Userspace_tools_projects
> [4] https://btrfs.wiki.kernel.org/index.php/Main_Page#Developer_documentation
> [5] https://btrfs.wiki.kernel.org/index.php/Project_ideas
> [6] https://btrfs.wiki.kernel.org/index.php/Project_ideas#Cancellable_operations
> [7] https://btrfs.wiki.kernel.org/index.php/Project_ideas#Implement_new_FALLOC_FL_.2A_modes
>
> --
> === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
>   PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
>   --- But people have always eaten people,  / what else is there to ---
>          eat?  / If the Juju had meant us not to eat people / he
>                      wouldn't have made us of meat.

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Torbjørn]

On 07/25/2014 01:37 PM, Torbjørn wrote:
> On 25. juli 2014 13:09, Torbjørn wrote:
>> On 25. juli 2014 12:22, Torbjørn wrote:
>>> On 25. juli 2014 11:28, Liu Bo wrote:
>>>> Hi Torbjørn,
>>>>
>>>> On Fri, Jul 25, 2014 at 06:51:44AM +0200, Torbjørn wrote:
>>>>> On 07/24/2014 04:58 PM, Chris Mason wrote:
>>>>>> <snip>
>>>>>> Liu Bo has a promising patch:
>>>>>>
>>>>>> https://patchwork.kernel.org/patch/4618421/
>>>>>>
>>>>>> Please give it a shot.  There's a second deadlock reading the 
>>>>>> free space
>>>>>> cache, I'm still working on that one too.
>>>>>>
>>>>>> -chris
>>>>> I (as expected, my hang was with free space cache) still get hangs
>>>>> with this applied on top of 3.16-rc6.
>>>>> Looking forward to the free space cache patch.
>>>>>
>>>>> I have not been able to trigger the same hang as I had with 3.15 on
>>>>> any of the 3.16-rc6 kernels so far.
>>>> Seems that you can run into the hang quite stably, I have a stupid 
>>>> debugging
>>>> patch here with adding a WARN_ON() to __load_free_space_inode(), 
>>>> would you
>>>> please give it a shot and show us the output of dmesg?
>>>>
>>>> (This may make 'dmesg' a mess, but it will show who the hell holds 
>>>> the free
>>>> space inode's page just at the time before hang)
>>>>
>>>> thanks,
>>>> -liubo
>>>>
>>>>
>>>> diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
>>>> index 2b0a627..e364888 100644
>>>> --- a/fs/btrfs/free-space-cache.c
>>>> +++ b/fs/btrfs/free-space-cache.c
>>>> @@ -350,6 +350,8 @@ static int io_ctl_prepare_pages(struct io_ctl 
>>>> *io_ctl,
>>>> struct inode *inode,
>>>>       gfp_t mask = btrfs_alloc_write_mask(inode->i_mapping);
>>>>       int i;
>>>>   +    WARN_ON(1);
>>>> +
>>>>       for (i = 0; i < io_ctl->num_pages; i++) {
>>>>           page = find_or_create_page(inode->i_mapping, i, mask);
>>>>           if (!page) {
>>>
>>> That patch did not apply cleanly on top of current linus master.
>>> Not sure what branch you are at, so I added that WARN_ON(1); 
>>> manually in free-space-cache.c.
>>> I reverted all the other patches before manually applying this one.
>>> If you need me to test from some other branch or with some other 
>>> patch included, please let me know.
>>>
>>> I'm doing a build now. Will run my rsync job and report back.
>>>
>>> Thanks.
>>> -- 
>>> Torbjørn
>> Back with results.
>> That's output from dmesg. If you need more, I can send netconsole log 
>> as well.
>>
> Seems like the list did not accept the huge email.
> dmesg is available from: 
> https://gist.github.com/anonymous/dba1f53211a1d482c3e0
>
> -- 
> Torbjørn
If it's of any help; I was unable to trigger the hang when running 
without compress=lzo

--
Torbjørn

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Liu Bo]

On Fri, Jul 25, 2014 at 01:37:10PM +0200, Torbjørn wrote:
> On 25. juli 2014 13:09, Torbjørn wrote:
> >On 25. juli 2014 12:22, Torbjørn wrote:
> >>On 25. juli 2014 11:28, Liu Bo wrote:
> >>>Hi Torbjørn,
> >>>
> >>>On Fri, Jul 25, 2014 at 06:51:44AM +0200, Torbjørn wrote:
> >>>>On 07/24/2014 04:58 PM, Chris Mason wrote:
> >>>>><snip>
> >>>>>Liu Bo has a promising patch:
> >>>>>
> >>>>>https://patchwork.kernel.org/patch/4618421/
> >>>>>
> >>>>>Please give it a shot.  There's a second deadlock reading
> >>>>>the free space
> >>>>>cache, I'm still working on that one too.
> >>>>>
> >>>>>-chris
> >>>>I (as expected, my hang was with free space cache) still get hangs
> >>>>with this applied on top of 3.16-rc6.
> >>>>Looking forward to the free space cache patch.
> >>>>
> >>>>I have not been able to trigger the same hang as I had with 3.15 on
> >>>>any of the 3.16-rc6 kernels so far.
> >>>Seems that you can run into the hang quite stably, I have a
> >>>stupid debugging
> >>>patch here with adding a WARN_ON() to
> >>>__load_free_space_inode(), would you
> >>>please give it a shot and show us the output of dmesg?
> >>>
> >>>(This may make 'dmesg' a mess, but it will show who the hell
> >>>holds the free
> >>>space inode's page just at the time before hang)
> >>>
> >>>thanks,
> >>>-liubo
> >>>
> >>>
> >>>diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
> >>>index 2b0a627..e364888 100644
> >>>--- a/fs/btrfs/free-space-cache.c
> >>>+++ b/fs/btrfs/free-space-cache.c
> >>>@@ -350,6 +350,8 @@ static int io_ctl_prepare_pages(struct
> >>>io_ctl *io_ctl,
> >>>struct inode *inode,
> >>>      gfp_t mask = btrfs_alloc_write_mask(inode->i_mapping);
> >>>      int i;
> >>>  +    WARN_ON(1);
> >>>+
> >>>      for (i = 0; i < io_ctl->num_pages; i++) {
> >>>          page = find_or_create_page(inode->i_mapping, i, mask);
> >>>          if (!page) {
> >>
> >>That patch did not apply cleanly on top of current linus master.
> >>Not sure what branch you are at, so I added that WARN_ON(1);
> >>manually in free-space-cache.c.
> >>I reverted all the other patches before manually applying this one.
> >>If you need me to test from some other branch or with some other
> >>patch included, please let me know.
> >>
> >>I'm doing a build now. Will run my rsync job and report back.
> >>
> >>Thanks.
> >>-- 
> >>Torbjørn
> >Back with results.
> >That's output from dmesg. If you need more, I can send netconsole
> >log as well.
> >
> Seems like the list did not accept the huge email.
> dmesg is available from:
> https://gist.github.com/anonymous/dba1f53211a1d482c3e0

This seems to be incomplete(Looks like dmesg has reached its buffer size limit),
does /var/log/message have the whole stack info?

thanks,
-liubo

> 
> --
> Torbjørn

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Torbjørn]

On 28. juli 2014 12:00, Liu Bo wrote:
<snip>
> This seems to be incomplete(Looks like dmesg has reached its buffer size limit),
> does /var/log/message have the whole stack info?
>
> thanks,
> -liubo
Hi,

Complete log was over 40MB. I uploaded everything from boot until 
"blocked for 120 seconds" started to appear.
If you want all the trailing log as well, let me know.

https://gist.github.com/anonymous/7958d8917967f727f324

--
Torbjørn

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[David Sterba]

Hugo,

On Fri, Jul 25, 2014 at 10:13:11AM +0100, Hugo Mills wrote:
> [this time, to the mailing list as well]

thank you for the writeup, I've put it on the wiki (Deverloper's FAQ) to
give an idea how to start with the development.

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Martin Steigerwald]

Am Donnerstag, 24. Juli 2014, 16:04:22 schrieb Chris Mason:
> On 07/24/2014 02:49 PM, Martin Steigerwald wrote:
> > Am Donnerstag, 24. Juli 2014, 10:58:51 schrieb Chris Mason:
> >> On 07/23/2014 06:47 PM, Martin Steigerwald wrote:
> >>> Am Dienstag, 15. Juli 2014, 17:08:27 schrieb Martin Steigerwald:
> >>>> Am Dienstag, 15. Juli 2014, 09:21:40 schrieb Chris Mason:
> >>>>> On 07/14/2014 05:58 PM, Martin Steigerwald wrote:
> >>>>>> Am Montag, 14. Juli 2014, 16:12:22 schrieb Chris Mason:
> >>>>>>> On 07/14/2014 11:10 AM, Martin Steigerwald wrote:
> >>>>>>>> Am Montag, 14. Juli 2014, 17:04:22 schrieben Sie:
> >>>>>>>>> Hi!
> >>>>>>>>> 
> >>>>>>>>> While with 3.16-rc3 and rc4 I didn´t have a BTRFS hang in several
> >>>>>>>>> days
> >>>>>>>>> of
> >>>>>>>>> usage, with 3-16-rc5 I had a hang again. Less than a hour since
> >>>>>>>>> booting
> >>>>>>>>> it.
> >>>>>>>>> 
> >>>>>>>>> Since the hang bug I and others had with 3.15 and upto 3.16-rc2
> >>>>>>>>> usually
> >>>>>>>>> didn´t happen that quickly after boot and since backtrace looks a
> >>>>>>>>> bit
> >>>>>>>>> different from what I have in memory, I post this in a new thread.
> >>>>>>>>> See thread "Blocked tasks on 3.15.1" for a discussion of previous
> >>>>>>>>> hang
> >>>>>>>>> issues.
> >>>>>>>> 
> >>>>>>>> Probably good to add some basic information on the filesystem:
> >>>>>>> Do you have compression enabled?  I wasn't able to nail down the
> >>>>>>> 3.15.1
> >>>>>>> hang before vacation attacked me, but I'm hoping to track it down
> >>>>>>> today.
> >>>>>> 
> >>>>>> Yes. I have.
> >>>>>> 
> >>>>>> It just hung again while I was playing PlaneShift.
> >>>>>> 
> >>>>>> Back to 3.16-rc4 as rc5 seems to be broke here.
> >>>>> 
> >>>>> The btrfs hang you're hitting goes back to 3.15.  So 3.16-rc4 vs rc5
> >>>>> shouldn't be a factor.  Are you hitting other problems with 3.16?
> >>>> 
> >>>> So far for this day 3.16-rc4 behaves nicely. With 3.16-rc5 I had a
> >>>> BTRFS
> >>>> hang twice yesterday. 3.16-rc4 before also behaved nicely for several
> >>>> days
> >>>> or well about a week here.
> >>> 
> >>> 3.16-rc4 now hung as well…
> >> 
> >> Liu Bo has a promising patch:
> >> 
> >> https://urldefense.proofpoint.com/v1/url?u=https://patchwork.kernel.org/p
> >> atch/4618421/&k=ZVNjlDMF0FElm4dQtryO4A%3D%3D%0A&r=6%2FL0lzzDhu0Y1hL9xm%2B
> >> QyA%3D%3D%0A&m=CJPREifRDOxlzhYeURx75h33LGU7YemJsNeLP%2FvXCv8%3D%0A&s=8fb0
> >> a70afce09530f16ea66a47d2af07966706b21281a7142d86256979013bab
> >> 
> >> Please give it a shot.  There's a second deadlock reading the free space
> >> cache, I'm still working on that one too.
> > 
> > Now running 3.16-rc6 + current git + this patch.
> > 
> > It may take some time tough cause during compiling the kernel BTRFS hung
> > again, which caused loss of KDE Baloo desktop search file index and parts
> > of a mail I wrote in KMail.
> > 
> > Since the patch mentioned ENOSPC issues but the filesystem has enough free
> > space according to df I shrunk the trees with
> 
> Thanks for giving it a try.  The ENOSPC mentioned here is looking for a
> contiguous extent, so it's easily possible to trigger that enospc
> without actually being full.

So far no lockup with 3.16.0-rc6-tp520-fixcompwrite+

But BTRFS still not used up all space for laying out trees after balance:

Label: 'home'  uuid: […]
        Total devices 2 FS bytes used 126.70GiB
        devid    1 size 160.00GiB used 156.00GiB path /dev/dm-0
        devid    2 size 160.00GiB used 156.00GiB path /dev/mapper/sata-home

May be a good sign nonetheless, thus mentioned.

Any other findings?

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Liu Bo]

On Mon, Jul 28, 2014 at 01:11:19PM +0200, Torbjørn wrote:
> On 28. juli 2014 12:00, Liu Bo wrote:
> <snip>
> >This seems to be incomplete(Looks like dmesg has reached its buffer size limit),
> >does /var/log/message have the whole stack info?
> >
> >thanks,
> >-liubo
> Hi,
> 
> Complete log was over 40MB. I uploaded everything from boot until
> "blocked for 120 seconds" started to appear.
> If you want all the trailing log as well, let me know.
> 
> https://gist.github.com/anonymous/7958d8917967f727f324

Sorry...still don't get why it's locked up, io_ctl_prepare_pages() has several
callers, and they are properly released from the code level.  And the warnings
printed in the log belong to other btrfs partitions, not the hanged btrfs one,
and we're still not able to know which one holds the free space cache inode page.

Maybe we'd better resort to a bisect between 3.14 and 3.15(I know it'd be a lot
of time though).

Here, doing rsync on compress=lzo full btrfs never hit that problem, shrug...

thanks,
-liubo

> 
> --
> Torbjørn

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Torbjørn]

On 07/29/2014 12:18 PM, Liu Bo wrote:
> On Mon, Jul 28, 2014 at 01:11:19PM +0200, Torbjørn wrote:
>> On 28. juli 2014 12:00, Liu Bo wrote:
>> <snip>
>>> This seems to be incomplete(Looks like dmesg has reached its buffer size limit),
>>> does /var/log/message have the whole stack info?
>>>
>>> thanks,
>>> -liubo
>> Hi,
>>
>> Complete log was over 40MB. I uploaded everything from boot until
>> "blocked for 120 seconds" started to appear.
>> If you want all the trailing log as well, let me know.
>>
>> https://gist.github.com/anonymous/7958d8917967f727f324
> Sorry...still don't get why it's locked up, io_ctl_prepare_pages() has several
> callers, and they are properly released from the code level.  And the warnings
> printed in the log belong to other btrfs partitions, not the hanged btrfs one,
> and we're still not able to know which one holds the free space cache inode page.
>
> Maybe we'd better resort to a bisect between 3.14 and 3.15(I know it'd be a lot
> of time though).
>
> Here, doing rsync on compress=lzo full btrfs never hit that problem, shrug...
>
> thanks,
> -liubo
>
>> --
>> Torbjørn
That's too bad.

My reproducer is not 100% guaranteed to trigger the hang, so doing a 
bisect might lead us to some innocent commit.
I have run the rsync + snapshot job several times here now, and no hang.

--
Torbjørn

Re: BTRFS hang with 3.16-rc5 (and also with 3.16-rc4)

[Liu Bo]

On Tue, Jul 29, 2014 at 05:07:31PM +0200, Torbjørn wrote:
> On 07/29/2014 12:18 PM, Liu Bo wrote:
> >On Mon, Jul 28, 2014 at 01:11:19PM +0200, Torbjørn wrote:
> >>On 28. juli 2014 12:00, Liu Bo wrote:
> >><snip>
> >>>This seems to be incomplete(Looks like dmesg has reached its buffer size limit),
> >>>does /var/log/message have the whole stack info?
> >>>
> >>>thanks,
> >>>-liubo
> >>Hi,
> >>
> >>Complete log was over 40MB. I uploaded everything from boot until
> >>"blocked for 120 seconds" started to appear.
> >>If you want all the trailing log as well, let me know.
> >>
> >>https://gist.github.com/anonymous/7958d8917967f727f324
> >Sorry...still don't get why it's locked up, io_ctl_prepare_pages() has several
> >callers, and they are properly released from the code level.  And the warnings
> >printed in the log belong to other btrfs partitions, not the hanged btrfs one,
> >and we're still not able to know which one holds the free space cache inode page.
> >
> >Maybe we'd better resort to a bisect between 3.14 and 3.15(I know it'd be a lot
> >of time though).
> >
> >Here, doing rsync on compress=lzo full btrfs never hit that problem, shrug...
> >
> >thanks,
> >-liubo
> >
> >>--
> >>Torbjørn
> That's too bad.
> 
> My reproducer is not 100% guaranteed to trigger the hang, so doing a
> bisect might lead us to some innocent commit.
> I have run the rsync + snapshot job several times here now, and no hang.

Good news! I've reproduced it with my xfstests config, will dig into it closer.

thanks,
-liubo