* mount hangs after disk crash (RAID-1)
@ 2014-05-22 2:22 Tomasz Chmielewski
2014-05-22 8:51 ` Tomasz Chmielewski
2014-05-22 9:43 ` Duncan
0 siblings, 2 replies; 9+ messages in thread
From: Tomasz Chmielewski @ 2014-05-22 2:22 UTC (permalink / raw)
To: linux-btrfs
One disk in RAID-1 crashed, so powered off, changed disk, powered
on, trying to mount degraded.
Unfortunately it hangs (running 3.14.4).
# mount -o degraded,compress=lzo,noatime /dev/sdb4 /home
(...never returns...)
iostat shows no disk activity.
Any ideas how to recover?
dmesg shows:
[ 115.521656] Btrfs loaded
[ 115.522530] BTRFS: device label home devid 2 transid 53234 /dev/sdb4
[ 115.523217] BTRFS info (device sdb4): allowing degraded mounts
[ 115.523282] BTRFS info (device sdb4): disk space caching is enabled
[ 115.624905] BTRFS: bdev (null) errs: wr 0, rd 58, flush 0, corrupt 0, gen 0
[ 144.538692] BTRFS error (device sdb4): free space inode generation (0) did not match free space cache generation (53233)
[ 360.512573] INFO: task mount:4469 blocked for more than 120 seconds.
[ 360.512632] Not tainted 3.14.4 #1
[ 360.512685] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 360.512734] mount D 0000000000000002 0 4469 4424 0x00000000
[ 360.512855] ffff880812ed9628 0000000000000082 0000041100000000 ffff880811009800
[ 360.513092] 00000000000116c0 0000000000004000 ffff880813e0b000 ffff880811009800
[ 360.513372] 0000000000000000 ffff8807c33db468 0000000000000000 000000000000b000
[ 360.513652] Call Trace:
[ 360.513746] [<ffffffffa0359dd8>] ? submit_stripe_bio+0x5b/0x5d [btrfs]
[ 360.513841] [<ffffffffa035dd83>] ? btrfs_map_bio+0x377/0x3fc [btrfs]
[ 360.513933] [<ffffffff810b1ae5>] ? find_get_pages_tag+0xe9/0x123
[ 360.514022] [<ffffffff810b1d5c>] ? __lock_page+0x63/0x63
[ 360.514112] [<ffffffff8139756e>] schedule+0x6a/0x6c
[ 360.514215] [<ffffffff813975fa>] io_schedule+0x8a/0xc8
[ 360.514302] [<ffffffff810b1d65>] sleep_on_page+0x9/0xd
[ 360.514391] [<ffffffff81397b5b>] __wait_on_bit+0x45/0x7e
[ 360.514479] [<ffffffff810b1ae5>] ? find_get_pages_tag+0xe9/0x123
[ 360.514568] [<ffffffff810b1f6d>] wait_on_page_bit+0x71/0x73
[ 360.514657] [<ffffffff81067ff2>] ? wake_atomic_t_function+0x28/0x28
[ 360.514747] [<ffffffff810b2027>] filemap_fdatawait_range+0x79/0x118
[ 360.514841] [<ffffffffa0350cac>] btrfs_wait_ordered_range+0x6e/0x120 [btrfs]
[ 360.514937] [<ffffffffa0370f8b>] __btrfs_write_out_cache+0x47a/0x6ef [btrfs]
[ 360.515030] [<ffffffffa031d921>] ? comp_keys+0x26/0x28 [btrfs]
[ 360.515123] [<ffffffffa0371485>] btrfs_write_out_cache+0x8c/0xc1 [btrfs]
[ 360.515218] [<ffffffffa035612b>] ? free_extent_buffer+0x6f/0x7c [btrfs]
[ 360.515312] [<ffffffffa032e9a0>] btrfs_write_dirty_block_groups+0x54b/0x56f [btrfs]
[ 360.515450] [<ffffffffa033a0fd>] commit_cowonly_roots+0x13f/0x21a [btrfs]
[ 360.515544] [<ffffffffa033bbf8>] btrfs_commit_transaction+0x410/0x7ff [btrfs]
[ 360.515683] [<ffffffffa037ca26>] btrfs_recover_relocation+0x2a1/0x353 [btrfs]
[ 360.515820] [<ffffffffa0339aa9>] open_ctree+0x182c/0x1b5c [btrfs]
[ 360.515910] [<ffffffff811a988e>] ? ida_get_new_above+0x178/0x192
[ 360.516001] [<ffffffffa0318059>] btrfs_mount+0x40b/0x718 [btrfs]
[ 360.516090] [<ffffffff810b8dc6>] ? __free_pages+0x1c/0x48
[ 360.516179] [<ffffffff81172056>] ? selinux_sb_copy_data+0x1a0/0x1d1
[ 360.516269] [<ffffffff810fb7fd>] mount_fs+0x64/0x14f
[ 360.516356] [<ffffffff810c839d>] ? __alloc_percpu+0xb/0xd
[ 360.516445] [<ffffffff8111076e>] vfs_kern_mount+0x61/0xec
[ 360.516533] [<ffffffff811127aa>] do_mount+0x70b/0x8af
[ 360.516621] [<ffffffff81112b88>] SyS_mount+0x80/0xb9
[ 360.516709] [<ffffffff8139a5a2>] system_call_fastpath+0x16/0x1b
[ 360.516800] INFO: task btrfs-transacti:4680 blocked for more than 120 seconds.
[ 360.516933] Not tainted 3.14.4 #1
[ 360.517018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 360.517151] btrfs-transacti D ffff8807f0da5000 0 4680 2 0x00000000
[ 360.517314] ffff8808118cfd78 0000000000000046 ffff8808118cfd00 ffff880811844800
[ 360.517594] 00000000000116c0 0000000000004000 ffff880813df9800 ffff880811844800
[ 360.517873] ffff8808118cfdb8 0000000000000292 0000000000000000 0000000000000000
[ 360.518153] Call Trace:
[ 360.518241] [<ffffffffa033c477>] ? start_transaction+0x16f/0x459 [btrfs]
[ 360.518332] [<ffffffff8139756e>] schedule+0x6a/0x6c
[ 360.518422] [<ffffffffa033b299>] wait_current_trans.isra.36+0x94/0xe2 [btrfs]
[ 360.518556] [<ffffffff81067f98>] ? __wake_up_sync+0xd/0xd
[ 360.518648] [<ffffffffa033c503>] start_transaction+0x1fb/0x459 [btrfs]
[ 360.518741] [<ffffffffa033c7ae>] btrfs_attach_transaction+0x12/0x14 [btrfs]
[ 360.518834] [<ffffffffa0339ea6>] transaction_kthread+0xcd/0x1c3 [btrfs]
[ 360.518927] [<ffffffffa0339dd9>] ? open_ctree+0x1b5c/0x1b5c [btrfs]
[ 360.519018] [<ffffffff81050d3e>] kthread+0xcd/0xd5
[ 360.519105] [<ffffffff81030000>] ? unmap_pte_range+0x51/0x96
[ 360.519194] [<ffffffff81050c71>] ? kthread_freezable_should_stop+0x43/0x43
[ 360.519284] [<ffffffff8139a4fc>] ret_from_fork+0x7c/0xb0
[ 360.519372] [<ffffffff81050c71>] ? kthread_freezable_should_stop+0x43/0x43
[ 442.318694] SysRq : Show Blocked State
[ 442.318842] task PC stack pid father
[ 442.318982] mount D 0000000000000002 0 4469 4424 0x00000000
[ 442.319171] ffff880812ed9628 0000000000000082 0000041100000000 ffff880811009800
[ 442.319494] 00000000000116c0 0000000000004000 ffff880813e0b000 ffff880811009800
[ 442.319815] 0000000000000000 ffff8807c33db468 0000000000000000 000000000000b000
[ 442.320136] Call Trace:
[ 442.320257] [<ffffffffa0359dd8>] ? submit_stripe_bio+0x5b/0x5d [btrfs]
[ 442.320382] [<ffffffffa035dd83>] ? btrfs_map_bio+0x377/0x3fc [btrfs]
[ 442.320494] [<ffffffff810b1ae5>] ? find_get_pages_tag+0xe9/0x123
[ 442.320607] [<ffffffff810b1d5c>] ? __lock_page+0x63/0x63
[ 442.320715] [<ffffffff8139756e>] schedule+0x6a/0x6c
[ 442.320820] [<ffffffff813975fa>] io_schedule+0x8a/0xc8
[ 442.320925] [<ffffffff810b1d65>] sleep_on_page+0x9/0xd
[ 442.321031] [<ffffffff81397b5b>] __wait_on_bit+0x45/0x7e
[ 442.321138] [<ffffffff810b1ae5>] ? find_get_pages_tag+0xe9/0x123
[ 442.321244] [<ffffffff810b1f6d>] wait_on_page_bit+0x71/0x73
[ 442.321352] [<ffffffff81067ff2>] ? wake_atomic_t_function+0x28/0x28
[ 442.321460] [<ffffffff810b2027>] filemap_fdatawait_range+0x79/0x118
[ 442.321580] [<ffffffffa0350cac>] btrfs_wait_ordered_range+0x6e/0x120 [btrfs]
[ 442.321701] [<ffffffffa0370f8b>] __btrfs_write_out_cache+0x47a/0x6ef [btrfs]
[ 442.321815] [<ffffffffa031d921>] ? comp_keys+0x26/0x28 [btrfs]
[ 442.321932] [<ffffffffa0371485>] btrfs_write_out_cache+0x8c/0xc1 [btrfs]
[ 442.322051] [<ffffffffa035612b>] ? free_extent_buffer+0x6f/0x7c [btrfs]
[ 442.322167] [<ffffffffa032e9a0>] btrfs_write_dirty_block_groups+0x54b/0x56f [btrfs]
[ 442.322340] [<ffffffffa033a0fd>] commit_cowonly_roots+0x13f/0x21a [btrfs]
[ 442.322458] [<ffffffffa033bbf8>] btrfs_commit_transaction+0x410/0x7ff [btrfs]
[ 442.322631] [<ffffffffa037ca26>] btrfs_recover_relocation+0x2a1/0x353 [btrfs]
[ 442.322802] [<ffffffffa0339aa9>] open_ctree+0x182c/0x1b5c [btrfs]
[ 442.322910] [<ffffffff811a988e>] ? ida_get_new_above+0x178/0x192
[ 442.323023] [<ffffffffa0318059>] btrfs_mount+0x40b/0x718 [btrfs]
[ 442.323131] [<ffffffff810b8dc6>] ? __free_pages+0x1c/0x48
[ 442.323238] [<ffffffff81172056>] ? selinux_sb_copy_data+0x1a0/0x1d1
[ 442.323346] [<ffffffff810fb7fd>] mount_fs+0x64/0x14f
[ 442.323451] [<ffffffff810c839d>] ? __alloc_percpu+0xb/0xd
[ 442.323558] [<ffffffff8111076e>] vfs_kern_mount+0x61/0xec
[ 442.323663] [<ffffffff811127aa>] do_mount+0x70b/0x8af
[ 442.325267] [<ffffffff81112b88>] SyS_mount+0x80/0xb9
[ 442.325372] [<ffffffff8139a5a2>] system_call_fastpath+0x16/0x1b
[ 442.325482] btrfs-transacti D ffff8807f0da5000 0 4680 2 0x00000000
[ 442.325671] ffff8808118cfd78 0000000000000046 ffff8808118cfd00 ffff880811844800
[ 442.325992] 00000000000116c0 0000000000004000 ffff880813df9800 ffff880811844800
[ 442.326312] ffff8808118cfdb8 0000000000000292 0000000000000000 0000000000000000
[ 442.326633] Call Trace:
[ 442.326743] [<ffffffffa033c477>] ? start_transaction+0x16f/0x459 [btrfs]
[ 442.326851] [<ffffffff8139756e>] schedule+0x6a/0x6c
[ 442.326964] [<ffffffffa033b299>] wait_current_trans.isra.36+0x94/0xe2 [btrfs]
[ 442.327126] [<ffffffff81067f98>] ? __wake_up_sync+0xd/0xd
[ 442.327239] [<ffffffffa033c503>] start_transaction+0x1fb/0x459 [btrfs]
[ 442.327355] [<ffffffffa033c7ae>] btrfs_attach_transaction+0x12/0x14 [btrfs]
[ 442.327471] [<ffffffffa0339ea6>] transaction_kthread+0xcd/0x1c3 [btrfs]
[ 442.327587] [<ffffffffa0339dd9>] ? open_ctree+0x1b5c/0x1b5c [btrfs]
[ 442.327696] [<ffffffff81050d3e>] kthread+0xcd/0xd5
[ 442.327801] [<ffffffff81030000>] ? unmap_pte_range+0x51/0x96
[ 442.327908] [<ffffffff81050c71>] ? kthread_freezable_should_stop+0x43/0x43
[ 442.328017] [<ffffffff8139a4fc>] ret_from_fork+0x7c/0xb0
[ 442.328123] [<ffffffff81050c71>] ? kthread_freezable_should_stop+0x43/0x43
[ 442.328232] Sched Debug Version: v0.11, 3.14.4 #1
[ 442.328336] ktime : 442568.889963
[ 442.328442] sched_clk : 442328.231419
[ 442.328547] cpu_clk : 442328.231449
[ 442.328653] jiffies : 4294981553
[ 442.328758] sched_clock_stable() : 1
[ 442.328861]
[ 442.328957] sysctl_sched
[ 442.329055] .sysctl_sched_latency : 24.000000
[ 442.329161] .sysctl_sched_min_granularity : 3.000000
[ 442.329266] .sysctl_sched_wakeup_granularity : 4.000000
[ 442.329372] .sysctl_sched_child_runs_first : 0
[ 442.329477] .sysctl_sched_features : 11899
[ 442.329581] .sysctl_sched_tunable_scaling : 1 (logaritmic)
[ 442.329687]
[ 442.329782] cpu#0, 3411.486 MHz
[ 442.329882] .nr_running : 0
[ 442.329984] .load : 0
[ 442.330086] .nr_switches : 193135
[ 442.330189] .nr_load_updates : 18917
[ 442.330292] .nr_uninterruptible : -977
[ 442.330395] .next_balance : 4294.981554
[ 442.330499] .curr->pid : 0
[ 442.330601] .clock : 442320.413954
[ 442.330705] .cpu_load[0] : 0
[ 442.330807] .cpu_load[1] : 0
[ 442.330909] .cpu_load[2] : 0
[ 442.331011] .cpu_load[3] : 0
[ 442.331113] .cpu_load[4] : 0
[ 442.331216] .yld_count : 13
[ 442.331318] .sched_count : 200232
[ 442.331421] .sched_goidle : 73213
[ 442.331523] .avg_idle : 864796
[ 442.331627] .ttwu_count : 130526
[ 442.331730] .ttwu_local : 109416
[ 442.331834]
[ 442.331834] cfs_rq[0]:/autogroup-26
[ 442.331989] .exec_clock : 120.820030
[ 442.332092] .MIN_vruntime : 0.000001
[ 442.332195] .min_vruntime : 119.519138
[ 442.332299] .max_vruntime : 0.000001
[ 442.332403] .spread : 0.000000
[ 442.332507] .spread0 : -9825.001370
[ 442.332611] .nr_spread_over : 0
[ 442.332713] .nr_running : 0
[ 442.332814] .load : 0
[ 442.332916] .runnable_load_avg : 0
[ 442.333018] .blocked_load_avg : 0
[ 442.333120] .tg_load_contrib : 0
[ 442.333222] .tg_runnable_contrib : 2
[ 442.333324] .tg_load_avg : 0
[ 442.333426] .tg->runnable_avg : 2
[ 442.333528] .tg->cfs_bandwidth.timer_active: 0
[ 442.333630] .throttled : 0
[ 442.333732] .throttle_count : 0
[ 442.333834] .se->exec_start : 442318.673917
[ 442.333938] .se->vruntime : 9932.611723
[ 442.334042] .se->sum_exec_runtime : 120.820030
[ 442.334146] .se->statistics.wait_start : 0.000000
[ 442.334250] .se->statistics.sleep_start : 0.000000
[ 442.334353] .se->statistics.block_start : 0.000000
[ 442.334457] .se->statistics.sleep_max : 0.000000
[ 442.334561] .se->statistics.block_max : 0.000000
[ 442.334665] .se->statistics.exec_max : 9.854141
[ 442.334769] .se->statistics.slice_max : 0.162777
[ 442.334872] .se->statistics.wait_max : 0.035742
[ 442.334976] .se->statistics.wait_sum : 0.358340
[ 442.335079] .se->statistics.wait_count : 5282
[ 442.335182] .se->load.weight : 2
[ 442.335284] .se->avg.runnable_avg_sum : 133
[ 442.335386] .se->avg.runnable_avg_period : 47866
[ 442.335489] .se->avg.load_avg_contrib : 0
[ 442.335591] .se->avg.decay_count : 421828
[ 442.335695]
[ 442.335695] cfs_rq[0]:/autogroup-6
[ 442.335850] .exec_clock : 6051.469384
[ 442.335953] .MIN_vruntime : 0.000001
[ 442.336057] .min_vruntime : 5861.283469
[ 442.336160] .max_vruntime : 0.000001
[ 442.336263] .spread : 0.000000
[ 442.336366] .spread0 : -4083.237039
[ 442.336470] .nr_spread_over : 55
[ 442.336573] .nr_running : 0
[ 442.336675] .load : 0
[ 442.336777] .runnable_load_avg : 0
[ 442.336879] .blocked_load_avg : 0
[ 442.336981] .tg_load_contrib : 0
[ 442.337083] .tg_runnable_contrib : 0
[ 442.337185] .tg_load_avg : 6
[ 442.337287] .tg->runnable_avg : 0
[ 442.337388] .tg->cfs_bandwidth.timer_active: 0
[ 442.337490] .throttled : 0
[ 442.337593] .throttle_count : 0
[ 442.337695] .se->exec_start : 442320.411912
[ 442.337799] .se->vruntime : 9944.520508
[ 442.337903] .se->sum_exec_runtime : 6051.472003
[ 442.338008] .se->statistics.wait_start : 0.000000
[ 442.338111] .se->statistics.sleep_start : 0.000000
[ 442.338215] .se->statistics.block_start : 0.000000
[ 442.338318] .se->statistics.sleep_max : 0.000000
[ 442.338421] .se->statistics.block_max : 0.000000
[ 442.338525] .se->statistics.exec_max : 9.993329
[ 442.338629] .se->statistics.slice_max : 0.140059
[ 442.338732] .se->statistics.wait_max : 0.077045
[ 442.338837] .se->statistics.wait_sum : 493.196736
[ 442.338940] .se->statistics.wait_count : 41001
[ 442.339043] .se->load.weight : 2
[ 442.339145] .se->avg.runnable_avg_sum : 121
[ 442.339247] .se->avg.runnable_avg_period : 47929
[ 442.339350] .se->avg.load_avg_contrib : 0
[ 442.339460] .se->avg.decay_count : 421830
[ 442.339572]
[ 442.339572] cfs_rq[0]:/
[ 442.339724] .exec_clock : 7755.380964
[ 442.339828] .MIN_vruntime : 0.000001
[ 442.339931] .min_vruntime : 9944.628775
[ 442.340035] .max_vruntime : 0.000001
[ 442.340138] .spread : 0.000000
[ 442.340242] .spread0 : 0.000000
[ 442.340346] .nr_spread_over : 15
[ 442.340448] .nr_running : 0
[ 442.340550] .load : 0
[ 442.340653] .runnable_load_avg : 0
[ 442.340754] .blocked_load_avg : 2
[ 442.340856] .tg_load_contrib : 2
[ 442.340958] .tg_runnable_contrib : 7
[ 442.341060] .tg_load_avg : 2
[ 442.341163] .tg->runnable_avg : 18
[ 442.341265] .tg->cfs_bandwidth.timer_active: 0
[ 442.341367] .throttled : 0
[ 442.341469] .throttle_count : 0
[ 442.341571] .avg->runnable_avg_sum : 348
[ 442.341674] .avg->runnable_avg_period : 48024
[ 442.341779]
[ 442.341779] rt_rq[0]:/
[ 442.341932] .rt_nr_running : 0
[ 442.342035] .rt_throttled : 0
[ 442.342137] .rt_time : 0.000000
[ 442.342241] .rt_runtime : 950.000000
[ 442.342346]
[ 442.342346] runnable tasks:
[ 442.342346] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 442.342346] ----------------------------------------------------------------------------------------------------------
[ 442.342750] ksoftirqd/0 3 9931.873286 940 120 9931.873286 9.380748 441453.948404 /
[ 442.343080] kworker/0:0H 5 892.634890 5 100 892.634890 0.037675 1145.285746 /
[ 442.343410] rcu_sched 7 9944.628775 9813 120 9944.628775 667.260040 441532.117991 /
[ 442.343739] rcu_bh 8 117.315808 2 120 117.315808 0.000860 0.000728 /
[ 442.344068] migration/0 9 0.000000 38 0 0.000000 602.383800 0.000876 /
[ 442.344396] watchdog/0 10 -11.785717 120 0 -11.785717 1.486157 93.388826 /
[ 442.346229] khubd 339 1893.145244 62 120 1893.145244 4.145054 34134.084693 /
[ 442.346558] fsnotify_mark 560 1789.847970 14 120 1789.847970 0.345820 32904.674520 /
[ 442.346887] kworker/0:1 587 9932.487695 10283 120 9932.487695 107.618280 440853.949530 /
[ 442.347215] kworker/0:2 675 9216.825160 8 120 9216.825160 0.076248 113974.508948 /
[ 442.347544] scsi_eh_1 843 889.953484 18 120 889.953484 0.984397 350.351017 /
[ 442.347872] scsi_eh_3 847 889.050938 3 120 889.050938 0.061990 1.339428 /
[ 442.348201] kworker/u16:5 856 904.634567 45 120 904.634567 1.639225 388.329788 /
[ 442.348530] kworker/0:1H 864 9932.102629 48754 100 9932.102629 405.424799 439787.020895 /
[ 442.348857] md2_raid1 908 9932.173256 583 120 9932.173256 19.913980 440121.454369 /
[ 442.349185] jbd2/md2-8 923 9929.333300 385 120 9929.333300 20.817376 436476.026189 /
[ 442.349516] rs:main Q:Reg 3024 5781.492937 56 120 5781.492937 6.858607 323799.016826 /autogroup-6
[ 442.349847] acpid 3157 0.892033 4 120 0.892033 0.529410 325.839215 /autogroup-10
[ 442.350178] ipv6_addrconf 3283 2816.279997 2 100 2816.279997 0.009845 0.005355 /
[ 442.350506] sshd 3359 0.532207 4 120 0.532207 0.652976 14440.055877 /autogroup-14
[ 442.350838] mysqld 4030 5848.675013 810 120 5848.675013 10.758223 402840.310308 /autogroup-6
[ 442.351169] mysqld 4051 1770.272079 27 120 1770.272079 1.613443 13749.994153 /autogroup-6
[ 442.351500] mysqld 4078 5861.283469 4400 120 5861.283469 1709.429214 398008.571608 /autogroup-6
[ 442.351830] pickup 4287 -1.064741 12 120 -1.064741 2.742415 360245.220393 /autogroup-18
[ 442.352160] sshd 4317 119.519138 5333 120 119.519138 137.837336 390966.825167 /autogroup-26
[ 442.352491] btrfs-genwork-1 4477 9886.892976 10 120 9886.892976 0.330109 273933.538729 /
[ 442.352819] btrfs-fixup-1 4480 9855.758107 5 120 9855.758107 0.075281 239871.393770 /
[ 442.353149]
[ 442.353245] cpu#1, 3411.486 MHz
[ 442.353345] .nr_running : 1
[ 442.353447] .load : 1024
[ 442.353550] .nr_switches : 62292
[ 442.353653] .nr_load_updates : 9496
[ 442.353756] .nr_uninterruptible : -364
[ 442.353859] .next_balance : 4294.981507
[ 442.353963] .curr->pid : 4319
[ 442.354066] .clock : 442318.588923
[ 442.354170] .cpu_load[0] : 0
[ 442.354272] .cpu_load[1] : 0
[ 442.354374] .cpu_load[2] : 0
[ 442.354476] .cpu_load[3] : 0
[ 442.354578] .cpu_load[4] : 0
[ 442.354680] .yld_count : 76
[ 442.354783] .sched_count : 63100
[ 442.354885] .sched_goidle : 18234
[ 442.354988] .avg_idle : 1000000
[ 442.355091] .ttwu_count : 29773
[ 442.355193] .ttwu_local : 19829
[ 442.355297]
[ 442.355297] cfs_rq[1]:/autogroup-27
[ 442.355453] .exec_clock : 81.696104
[ 442.355556] .MIN_vruntime : 0.000001
[ 442.355660] .min_vruntime : 92.535898
[ 442.355762] .max_vruntime : 0.000001
[ 442.355866] .spread : 0.000000
[ 442.355969] .spread0 : -9852.092877
[ 442.356072] .nr_spread_over : 5
[ 442.356174] .nr_running : 1
[ 442.356276] .load : 1024
[ 442.356380] .runnable_load_avg : 0
[ 442.356482] .blocked_load_avg : 0
[ 442.356583] .tg_load_contrib : 0
[ 442.356685] .tg_runnable_contrib : 0
[ 442.356788] .tg_load_avg : 0
[ 442.356891] .tg->runnable_avg : 0
[ 442.356993] .tg->cfs_bandwidth.timer_active: 0
[ 442.357095] .throttled : 0
[ 442.357197] .throttle_count : 0
[ 442.357299] .se->exec_start : 442318.588923
[ 442.357404] .se->vruntime : 3313.056370
[ 442.357507] .se->sum_exec_runtime : 81.696104
[ 442.357610] .se->statistics.wait_start : 0.000000
[ 442.357714] .se->statistics.sleep_start : 0.000000
[ 442.357817] .se->statistics.block_start : 0.000000
[ 442.357921] .se->statistics.sleep_max : 0.000000
[ 442.358024] .se->statistics.block_max : 0.000000
[ 442.358127] .se->statistics.exec_max : 4.151935
[ 442.358231] .se->statistics.slice_max : 0.000000
[ 442.358334] .se->statistics.wait_max : 0.017807
[ 442.358438] .se->statistics.wait_sum : 17.760581
[ 442.358542] .se->statistics.wait_count : 12191
[ 442.358645] .se->load.weight : 1024
[ 442.358748] .se->avg.runnable_avg_sum : 43
[ 442.358850] .se->avg.runnable_avg_period : 47852
[ 442.358953] .se->avg.load_avg_contrib : 0
[ 442.359054] .se->avg.decay_count : 0
[ 442.359158]
[ 442.359158] cfs_rq[1]:/autogroup-11
[ 442.359312] .exec_clock : 13.586455
[ 442.359418] .MIN_vruntime : 0.000001
[ 442.359538] .min_vruntime : 12.495082
[ 442.359642] .max_vruntime : 0.000001
[ 442.359746] .spread : 0.000000
[ 442.359849] .spread0 : -9932.133693
[ 442.359954] .nr_spread_over : 0
[ 442.360056] .nr_running : 0
[ 442.360158] .load : 0
[ 442.360261] .runnable_load_avg : 0
[ 442.360362] .blocked_load_avg : 0
[ 442.360465] .tg_load_contrib : 0
[ 442.360566] .tg_runnable_contrib : 0
[ 442.360668] .tg_load_avg : 11
[ 442.360770] .tg->runnable_avg : 0
[ 442.360872] .tg->cfs_bandwidth.timer_active: 0
[ 442.360974] .throttled : 0
[ 442.361075] .throttle_count : 0
[ 442.361178] .se->exec_start : 442305.421941
[ 442.361282] .se->vruntime : 3301.750096
[ 442.361386] .se->sum_exec_runtime : 13.596082
[ 442.361489] .se->statistics.wait_start : 0.000000
[ 442.361592] .se->statistics.sleep_start : 0.000000
[ 442.361696] .se->statistics.block_start : 0.000000
[ 442.361799] .se->statistics.sleep_max : 0.000000
[ 442.361902] .se->statistics.block_max : 0.000000
[ 442.362005] .se->statistics.exec_max : 0.046339
[ 442.362108] .se->statistics.slice_max : 0.023465
[ 442.362212] .se->statistics.wait_max : 0.009714
[ 442.362315] .se->statistics.wait_sum : 0.080283
[ 442.362418] .se->statistics.wait_count : 791
[ 442.362520] .se->load.weight : 2
[ 442.362621] .se->avg.runnable_avg_sum : 13
[ 442.362723] .se->avg.runnable_avg_period : 46991
[ 442.362826] .se->avg.load_avg_contrib : 0
[ 442.362928] .se->avg.decay_count : 421816
[ 442.363032]
[ 442.363032] cfs_rq[1]:/
[ 442.363185] .exec_clock : 1792.699568
[ 442.363288] .MIN_vruntime : 0.000001
[ 442.363391] .min_vruntime : 3313.735798
[ 442.363495] .max_vruntime : 0.000001
[ 442.363599] .spread : 0.000000
[ 442.363702] .spread0 : -6631.009680
[ 442.363806] .nr_spread_over : 7
[ 442.363908] .nr_running : 1
[ 442.364010] .load : 1024
[ 442.364112] .runnable_load_avg : 0
[ 442.364214] .blocked_load_avg : 0
[ 442.364315] .tg_load_contrib : 0
[ 442.364418] .tg_runnable_contrib : 1
[ 442.364519] .tg_load_avg : 3
[ 442.364621] .tg->runnable_avg : 18
[ 442.364723] .tg->cfs_bandwidth.timer_active: 0
[ 442.364826] .throttled : 0
[ 442.364928] .throttle_count : 0
[ 442.365031] .avg->runnable_avg_sum : 62
[ 442.365132] .avg->runnable_avg_period : 47701
[ 442.365235]
[ 442.365235] rt_rq[1]:/
[ 442.365387] .rt_nr_running : 0
[ 442.365496] .rt_throttled : 0
[ 442.365602] .rt_time : 0.000000
[ 442.365705] .rt_runtime : 950.000000
[ 442.365810]
[ 442.365810] runnable tasks:
[ 442.365810] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 442.365810] ----------------------------------------------------------------------------------------------------------
[ 442.366221] watchdog/1 11 -2.973630 114 0 -2.973630 1.329926 85.682243 /
[ 442.366549] migration/1 12 0.000000 41 0 0.000000 5.557719 0.000783 /
[ 442.366878] ksoftirqd/1 13 3301.496380 298 120 3301.496380 3.900868 440809.956960 /
[ 442.368705] kworker/1:0 14 -3.023870 13 120 -3.023870 0.025837 593.148821 /
[ 442.369034] kworker/1:0H 15 359.229340 6 100 359.229340 0.038046 1799.175182 /
[ 442.369366] kworker/1:1 589 3301.756800 16791 120 3301.756800 69.619205 441509.002041 /
[ 442.369696] scsi_eh_0 841 335.205607 18 120 335.205607 1.084244 350.637393 /
[ 442.370025] scsi_eh_2 845 334.171524 3 120 334.171524 0.014411 1.510574 /
[ 442.370354] kworker/u16:4 855 3301.842442 383 120 3301.842442 23.225454 440937.637564 /
[ 442.370684] kworker/1:1H 910 3201.954035 58 100 3201.954035 0.621261 147406.687293 /
[ 442.371014] vnstatd 3005 1.558586 85 120 1.558586 11.078703 404868.419760 /autogroup-7
[ 442.371345] nginx 3194 12.495082 811 120 12.495082 14.451161 405204.017547 /autogroup-11
[ 442.371676] mysqld_safe 3394 101.727991 434 120 101.727991 30.541534 458.372200 /autogroup-6
[ 442.372006] mysqld 4027 883.843182 809 120 883.843182 12.109428 402840.869767 /autogroup-6
[ 442.372337] mysqld 4083 875.137553 39714 120 875.137553 5376.267729 71672.847772 /autogroup-6
[ 442.372668] config 4233 3.076594 6 120 3.076594 3.271436 50.123413 /autogroup-16
[ 442.373000] getty 4303 0.441506 106 120 0.441506 1.670013 61.972680 /autogroup-22
[ 442.373333] R bash 4319 92.535898 359 120 92.535898 77.934814 389828.504167 /autogroup-27
[ 442.373662] bash 4424 38.770826 73 120 38.770826 44.222275 7183.090698 /autogroup-29
[ 442.373994] udevd 4475 70.155343 1 120 70.155343 0.224711 0.000000 /autogroup-4
[ 442.374325]
[ 442.374421] cpu#2, 3411.486 MHz
[ 442.374520] .nr_running : 0
[ 442.374623] .load : 0
[ 442.374725] .nr_switches : 29565
[ 442.374826] .nr_load_updates : 9147
[ 442.374929] .nr_uninterruptible : -620
[ 442.375032] .next_balance : 4294.981559
[ 442.375136] .curr->pid : 0
[ 442.375239] .clock : 442369.344644
[ 442.375343] .cpu_load[0] : 0
[ 442.375444] .cpu_load[1] : 0
[ 442.375545] .cpu_load[2] : 0
[ 442.375647] .cpu_load[3] : 0
[ 442.375748] .cpu_load[4] : 0
[ 442.375850] .yld_count : 0
[ 442.375952] .sched_count : 29689
[ 442.376055] .sched_goidle : 14224
[ 442.376158] .avg_idle : 1000000
[ 442.376261] .ttwu_count : 11458
[ 442.376363] .ttwu_local : 7275
[ 442.376466]
[ 442.376466] cfs_rq[2]:/autogroup-6
[ 442.376620] .exec_clock : 560.201135
[ 442.376724] .MIN_vruntime : 0.000001
[ 442.376827] .min_vruntime : 592.353230
[ 442.376931] .max_vruntime : 0.000001
[ 442.377035] .spread : 0.000000
[ 442.377138] .spread0 : -9352.392248
[ 442.377243] .nr_spread_over : 294
[ 442.377345] .nr_running : 0
[ 442.377448] .load : 0
[ 442.377550] .runnable_load_avg : 0
[ 442.377652] .blocked_load_avg : 0
[ 442.377753] .tg_load_contrib : 0
[ 442.377855] .tg_runnable_contrib : 0
[ 442.377958] .tg_load_avg : 0
[ 442.378059] .tg->runnable_avg : 4
[ 442.378161] .tg->cfs_bandwidth.timer_active: 0
[ 442.378263] .throttled : 0
[ 442.378365] .throttle_count : 0
[ 442.378468] .se->exec_start : 442365.468242
[ 442.378572] .se->vruntime : 2918.689109
[ 442.378676] .se->sum_exec_runtime : 560.201135
[ 442.378780] .se->statistics.wait_start : 0.000000
[ 442.378884] .se->statistics.sleep_start : 0.000000
[ 442.378988] .se->statistics.block_start : 0.000000
[ 442.379091] .se->statistics.sleep_max : 0.000000
[ 442.379194] .se->statistics.block_max : 0.000000
[ 442.379298] .se->statistics.exec_max : 2.417453
[ 442.379404] .se->statistics.slice_max : 0.015061
[ 442.379523] .se->statistics.wait_max : 0.104347
[ 442.379627] .se->statistics.wait_sum : 1.758039
[ 442.379731] .se->statistics.wait_count : 4709
[ 442.379834] .se->load.weight : 2
[ 442.379935] .se->avg.runnable_avg_sum : 17
[ 442.380038] .se->avg.runnable_avg_period : 47961
[ 442.380141] .se->avg.load_avg_contrib : 0
[ 442.380243] .se->avg.decay_count : 421873
[ 442.380346]
[ 442.380346] cfs_rq[2]:/autogroup-28
[ 442.380500] .exec_clock : 17.883599
[ 442.380603] .MIN_vruntime : 0.000001
[ 442.380706] .min_vruntime : 16.835023
[ 442.380810] .max_vruntime : 0.000001
[ 442.380912] .spread : 0.000000
[ 442.381015] .spread0 : -9928.026107
[ 442.381119] .nr_spread_over : 0
[ 442.381222] .nr_running : 0
[ 442.381324] .load : 0
[ 442.381426] .runnable_load_avg : 0
[ 442.381528] .blocked_load_avg : 0
[ 442.381630] .tg_load_contrib : 0
[ 442.381732] .tg_runnable_contrib : 0
[ 442.381835] .tg_load_avg : 0
[ 442.381936] .tg->runnable_avg : 0
[ 442.382039] .tg->cfs_bandwidth.timer_active: 0
[ 442.382140] .throttled : 0
[ 442.382242] .throttle_count : 0
[ 442.382345] .se->exec_start : 442311.004309
[ 442.382450] .se->vruntime : 2930.662936
[ 442.382554] .se->sum_exec_runtime : 17.883599
[ 442.382657] .se->statistics.wait_start : 0.000000
[ 442.382760] .se->statistics.sleep_start : 0.000000
[ 442.382863] .se->statistics.block_start : 0.000000
[ 442.382967] .se->statistics.sleep_max : 0.000000
[ 442.383071] .se->statistics.block_max : 0.000000
[ 442.383175] .se->statistics.exec_max : 1.510103
[ 442.383278] .se->statistics.slice_max : 0.000000
[ 442.383381] .se->statistics.wait_max : 0.022256
[ 442.383484] .se->statistics.wait_sum : 0.056724
[ 442.383588] .se->statistics.wait_count : 216
[ 442.383691] .se->load.weight : 2
[ 442.383792] .se->avg.runnable_avg_sum : 28
[ 442.383894] .se->avg.runnable_avg_period : 46643
[ 442.383997] .se->avg.load_avg_contrib : 0
[ 442.384099] .se->avg.decay_count : 421821
[ 442.384202]
[ 442.384202] cfs_rq[2]:/
[ 442.384356] .exec_clock : 1286.749804
[ 442.384460] .MIN_vruntime : 0.000001
[ 442.384563] .min_vruntime : 2930.662936
[ 442.384666] .max_vruntime : 0.000001
[ 442.384770] .spread : 0.000000
[ 442.384874] .spread0 : -7014.198194
[ 442.384977] .nr_spread_over : 5
[ 442.385079] .nr_running : 0
[ 442.385182] .load : 0
[ 442.385283] .runnable_load_avg : 0
[ 442.385385] .blocked_load_avg : 0
[ 442.385487] .tg_load_contrib : 0
[ 442.385589] .tg_runnable_contrib : 2
[ 442.385692] .tg_load_avg : 4
[ 442.385793] .tg->runnable_avg : 15
[ 442.385895] .tg->cfs_bandwidth.timer_active: 0
[ 442.385997] .throttled : 0
[ 442.386100] .throttle_count : 0
[ 442.386202] .avg->runnable_avg_sum : 107
[ 442.386304] .avg->runnable_avg_period : 47792
[ 442.386407]
[ 442.386407] rt_rq[2]:/
[ 442.386559] .rt_nr_running : 0
[ 442.386660] .rt_throttled : 0
[ 442.386763] .rt_time : 0.000000
[ 442.386867] .rt_runtime : 950.000000
[ 442.386972]
[ 442.386972] runnable tasks:
[ 442.386972] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 442.386972] ----------------------------------------------------------------------------------------------------------
[ 442.387378] watchdog/2 16 -5.975999 114 0 -5.975999 1.355055 72.367749 /
[ 442.387707] migration/2 17 0.000000 55 0 0.000000 5.626778 0.000808 /
[ 442.388036] ksoftirqd/2 18 2909.170278 211 120 2909.170278 2.994556 344089.003920 /
[ 442.388365] kworker/2:0 19 844.755664 14 120 844.755664 0.030489 34070.779463 /
[ 442.388693] kworker/2:0H 20 356.125675 6 100 356.125675 0.046098 1284.771571 /
[ 442.389024] kworker/2:1 591 2922.180383 3727 120 2922.180383 41.063105 441594.680493 /
[ 442.389355] kworker/2:1H 880 2855.340623 30 100 2855.340623 0.320676 147866.271845 /
[ 442.389685] md1_raid1 900 1637.880055 29 120 1637.880055 0.701836 35157.770638 /
[ 442.391507] ntpd 3222 8.940448 464 120 8.940448 27.499788 404759.032658 /autogroup-12
[ 442.391838] mysqld 4023 588.529990 816 120 588.529990 10.940580 403005.434973 /autogroup-6
[ 442.392169] mysqld 4026 585.286659 808 120 585.286659 11.844814 403340.591507 /autogroup-6
[ 442.392500] mysqld 4028 588.622413 809 120 588.622413 10.635569 403337.996701 /autogroup-6
[ 442.392830] mysqld 4050 580.800595 81 120 580.800595 1.201381 399985.008906 /autogroup-6
[ 442.393159] anvil 4211 0.149152 6 120 0.149152 1.028354 116.206947 /autogroup-16
[ 442.393491] log 4212 0.404382 6 120 0.404382 1.050234 1064.170662 /autogroup-16
[ 442.393822] qmgr 4288 1.741219 28 120 1.741219 3.370244 300265.911080 /autogroup-18
[ 442.394153] getty 4302 0.600840 111 120 0.600840 1.745011 61.804844 /autogroup-21
[ 442.394484] tmux 4423 16.835023 1985 120 16.835023 152.878474 334675.889548 /autogroup-28
[ 442.394815] btrfs-cache-1 4489 2914.065739 135 120 2914.065739 14.036423 274618.575204 /
[ 442.395144] bash 4493 1.078777 102 120 1.078777 37.488082 209025.250330 /autogroup-30
[ 442.395475] btrfs-cleaner 4679 2795.939502 3 120 2795.939502 0.200852 0.122357 /
[ 442.395804]
[ 442.395901] cpu#3, 3411.486 MHz
[ 442.396000] .nr_running : 0
[ 442.396102] .load : 0
[ 442.396204] .nr_switches : 32256
[ 442.396306] .nr_load_updates : 10284
[ 442.396410] .nr_uninterruptible : -496
[ 442.396513] .next_balance : 4294.981560
[ 442.396617] .curr->pid : 0
[ 442.396719] .clock : 442389.359621
[ 442.396822] .cpu_load[0] : 0
[ 442.396924] .cpu_load[1] : 0
[ 442.397025] .cpu_load[2] : 0
[ 442.397127] .cpu_load[3] : 0
[ 442.397228] .cpu_load[4] : 0
[ 442.397330] .yld_count : 50
[ 442.397433] .sched_count : 32402
[ 442.397536] .sched_goidle : 15346
[ 442.397639] .avg_idle : 1000000
[ 442.397742] .ttwu_count : 12834
[ 442.397845] .ttwu_local : 8789
[ 442.397948]
[ 442.397948] cfs_rq[3]:/autogroup-6
[ 442.398104] .exec_clock : 1044.684548
[ 442.398208] .MIN_vruntime : 0.000001
[ 442.398310] .min_vruntime : 1048.363630
[ 442.398414] .max_vruntime : 0.000001
[ 442.398518] .spread : 0.000000
[ 442.398622] .spread0 : -8896.497500
[ 442.398725] .nr_spread_over : 65
[ 442.398827] .nr_running : 0
[ 442.398929] .load : 0
[ 442.399031] .runnable_load_avg : 0
[ 442.399133] .blocked_load_avg : 0
[ 442.399235] .tg_load_contrib : 0
[ 442.399337] .tg_runnable_contrib : 0
[ 442.399452] .tg_load_avg : 0
[ 442.399555] .tg->runnable_avg : 2
[ 442.399657] .tg->cfs_bandwidth.timer_active: 0
[ 442.399760] .throttled : 0
[ 442.399861] .throttle_count : 0
[ 442.399964] .se->exec_start : 442365.470865
[ 442.400067] .se->vruntime : 4527.780959
[ 442.400171] .se->sum_exec_runtime : 1044.684548
[ 442.400275] .se->statistics.wait_start : 0.000000
[ 442.400378] .se->statistics.sleep_start : 0.000000
[ 442.400481] .se->statistics.block_start : 0.000000
[ 442.400584] .se->statistics.sleep_max : 0.000000
[ 442.400687] .se->statistics.block_max : 0.000000
[ 442.400791] .se->statistics.exec_max : 9.994245
[ 442.400894] .se->statistics.slice_max : 0.015489
[ 442.400998] .se->statistics.wait_max : 0.112907
[ 442.401101] .se->statistics.wait_sum : 1.549505
[ 442.401205] .se->statistics.wait_count : 2365
[ 442.401307] .se->load.weight : 2
[ 442.401410] .se->avg.runnable_avg_sum : 5
[ 442.401511] .se->avg.runnable_avg_period : 46249
[ 442.401614] .se->avg.load_avg_contrib : 0
[ 442.401716] .se->avg.decay_count : 421873
[ 442.401819]
[ 442.401819] cfs_rq[3]:/
[ 442.401972] .exec_clock : 1864.927134
[ 442.402075] .MIN_vruntime : 0.000001
[ 442.402178] .min_vruntime : 4529.306663
[ 442.402282] .max_vruntime : 0.000001
[ 442.402386] .spread : 0.000000
[ 442.402489] .spread0 : -5415.661005
[ 442.402592] .nr_spread_over : 8
[ 442.402695] .nr_running : 0
[ 442.402797] .load : 0
[ 442.402899] .runnable_load_avg : 0
[ 442.403001] .blocked_load_avg : 0
[ 442.403103] .tg_load_contrib : 0
[ 442.403206] .tg_runnable_contrib : 2
[ 442.403308] .tg_load_avg : 8
[ 442.403410] .tg->runnable_avg : 14
[ 442.403511] .tg->cfs_bandwidth.timer_active: 0
[ 442.403613] .throttled : 0
[ 442.403716] .throttle_count : 0
[ 442.403818] .avg->runnable_avg_sum : 120
[ 442.403920] .avg->runnable_avg_period : 46697
[ 442.404024]
[ 442.404024] rt_rq[3]:/
[ 442.404177] .rt_nr_running : 0
[ 442.404278] .rt_throttled : 0
[ 442.404380] .rt_time : 0.000000
[ 442.404484] .rt_runtime : 950.000000
[ 442.404587]
[ 442.404587] runnable tasks:
[ 442.404587] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 442.404587] ----------------------------------------------------------------------------------------------------------
[ 442.404990] init 1 0.224680 961 120 0.224680 513.148123 440106.934569 /autogroup-2
[ 442.405322] watchdog/3 21 -5.983216 114 0 -5.983216 1.202804 59.050855 /
[ 442.405649] migration/3 22 0.000000 38 0 0.000000 5.548407 0.000826 /
[ 442.405978] ksoftirqd/3 23 4509.315859 174 120 4509.315859 2.372825 344266.315958 /
[ 442.406307] kworker/3:0 24 892.732418 14 120 892.732418 0.038906 34057.207679 /
[ 442.406635] kworker/3:0H 25 816.234012 6 100 816.234012 0.026753 33340.454999 /
[ 442.406966] khungtaskd 482 4517.782282 5 120 4517.782282 7.084177 359803.162597 /
[ 442.407295] kworker/3:1 593 4517.316835 5510 120 4517.316835 65.055740 441595.938635 /
[ 442.407624] kworker/3:1H 1276 4264.632251 20 100 4264.632251 0.241366 85294.834423 /
[ 442.407953] rsyslogd 3026 1025.592392 16 120 1025.592392 1.437104 323803.262530 /autogroup-6
[ 442.408285] nginx 3193 9.535098 811 120 9.535098 14.637308 405204.174095 /autogroup-11
[ 442.408616] mysqld 4008 1016.506348 867 120 1016.506348 458.157069 81608.046707 /autogroup-6
[ 442.408947] mysqld 4031 1036.378910 808 120 1036.378910 11.855400 403340.264930 /autogroup-6
[ 442.409278] mysqld 4047 1048.363630 407 120 1048.363630 61.940396 401218.233637 /autogroup-6
[ 442.409609] mysqld 4049 1036.085686 81 120 1036.085686 1.474302 400195.430799 /autogroup-6
[ 442.409940] mysqld 4052 1014.859507 237 120 1014.859507 20.513955 129497.328142 /autogroup-6
[ 442.410272] master 4274 3.133218 25 120 3.133218 6.292730 360247.346775 /autogroup-18
[ 442.410604] getty 4305 0.540277 109 120 0.540277 1.685413 61.753648 /autogroup-24
[ 442.410935] mount 4469 217.192752 6526 120 217.192752 495.247509 34975.660230 /autogroup-29
[ 442.411267] btrfs-worker-1 4476 4510.314174 5 120 4510.314174 0.103229 239871.731614 /
[ 442.411597] btrfs-endio-met 4684 4513.105868 33 120 4513.105868 1.432365 240570.439858 /
[ 442.411927]
[ 442.412023] cpu#4, 3411.486 MHz
[ 442.412122] .nr_running : 0
[ 442.412224] .load : 0
[ 442.412327] .nr_switches : 6803
[ 442.412429] .nr_load_updates : 1787
[ 442.412531] .nr_uninterruptible : 518
[ 442.412634] .next_balance : 4294.981319
[ 442.412737] .curr->pid : 0
[ 442.412840] .clock : 442365.452320
[ 442.412944] .cpu_load[0] : 0
[ 442.413046] .cpu_load[1] : 0
[ 442.413148] .cpu_load[2] : 0
[ 442.413250] .cpu_load[3] : 0
[ 442.413351] .cpu_load[4] : 0
[ 442.413453] .yld_count : 0
[ 442.413555] .sched_count : 6945
[ 442.413658] .sched_goidle : 3094
[ 442.415261] .avg_idle : 1000000
[ 442.415365] .ttwu_count : 2957
[ 442.415467] .ttwu_local : 1904
[ 442.415570]
[ 442.415570] cfs_rq[4]:/autogroup-6
[ 442.415724] .exec_clock : 88.195352
[ 442.415828] .MIN_vruntime : 0.000001
[ 442.415932] .min_vruntime : 3926.498079
[ 442.416035] .max_vruntime : 0.000001
[ 442.416139] .spread : 0.000000
[ 442.416242] .spread0 : -6018.469589
[ 442.416347] .nr_spread_over : 159
[ 442.416449] .nr_running : 0
[ 442.416551] .load : 0
[ 442.416652] .runnable_load_avg : 0
[ 442.416754] .blocked_load_avg : 0
[ 442.416857] .tg_load_contrib : 0
[ 442.416960] .tg_runnable_contrib : 0
[ 442.417062] .tg_load_avg : 0
[ 442.417163] .tg->runnable_avg : 1
[ 442.417265] .tg->cfs_bandwidth.timer_active: 0
[ 442.417368] .throttled : 0
[ 442.417470] .throttle_count : 0
[ 442.417572] .se->exec_start : 442365.450096
[ 442.417676] .se->vruntime : 1643.779582
[ 442.417780] .se->sum_exec_runtime : 88.195352
[ 442.417884] .se->statistics.wait_start : 0.000000
[ 442.417987] .se->statistics.sleep_start : 0.000000
[ 442.418090] .se->statistics.block_start : 0.000000
[ 442.418194] .se->statistics.sleep_max : 0.000000
[ 442.418297] .se->statistics.block_max : 0.000000
[ 442.418400] .se->statistics.exec_max : 1.445210
[ 442.418503] .se->statistics.slice_max : 0.000000
[ 442.418606] .se->statistics.wait_max : 0.065583
[ 442.418709] .se->statistics.wait_sum : 0.194520
[ 442.418812] .se->statistics.wait_count : 1256
[ 442.418915] .se->load.weight : 2
[ 442.419016] .se->avg.runnable_avg_sum : 15
[ 442.419119] .se->avg.runnable_avg_period : 48646
[ 442.419221] .se->avg.load_avg_contrib : 0
[ 442.419323] .se->avg.decay_count : 421873
[ 442.419439]
[ 442.419439] cfs_rq[4]:/
[ 442.419592] .exec_clock : 219.636837
[ 442.419696] .MIN_vruntime : 0.000001
[ 442.419800] .min_vruntime : 1646.629753
[ 442.419903] .max_vruntime : 0.000001
[ 442.420006] .spread : 0.000000
[ 442.420110] .spread0 : -8298.449587
[ 442.420214] .nr_spread_over : 7
[ 442.420316] .nr_running : 0
[ 442.420417] .load : 0
[ 442.420519] .runnable_load_avg : 0
[ 442.420621] .blocked_load_avg : 0
[ 442.420723] .tg_load_contrib : 0
[ 442.420824] .tg_runnable_contrib : 0
[ 442.420926] .tg_load_avg : 6
[ 442.421027] .tg->runnable_avg : 12
[ 442.421130] .tg->cfs_bandwidth.timer_active: 0
[ 442.421232] .throttled : 0
[ 442.421334] .throttle_count : 0
[ 442.421436] .avg->runnable_avg_sum : 15
[ 442.421539] .avg->runnable_avg_period : 48084
[ 442.421642]
[ 442.421642] rt_rq[4]:/
[ 442.421796] .rt_nr_running : 0
[ 442.421898] .rt_throttled : 0
[ 442.422000] .rt_time : 0.000000
[ 442.422104] .rt_runtime : 950.000000
[ 442.422208]
[ 442.422208] runnable tasks:
[ 442.422208] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 442.422208] ----------------------------------------------------------------------------------------------------------
[ 442.422613] watchdog/4 26 -8.977227 114 0 -8.977227 1.407421 45.715910 /
[ 442.422941] migration/4 27 0.000000 534 0 0.000000 7.892668 0.000800 /
[ 442.423269] ksoftirqd/4 28 1634.639030 34 120 1634.639030 0.399234 301078.201192 /
[ 442.423599] kworker/4:0 29 11.099434 13 120 11.099434 0.035937 553.787808 /
[ 442.423927] kworker/4:0H 30 636.666622 6 100 636.666622 0.054768 2808.593904 /
[ 442.424258] kworker/4:1 595 1636.091646 409 120 1636.091646 4.523121 439247.259259 /
[ 442.424589] bioset 891 531.533240 2 100 531.533240 0.050708 0.027751 /
[ 442.424918] bioset 899 563.633977 2 100 563.633977 0.037556 0.017608 /
[ 442.425247] bioset 907 599.923653 2 100 599.923653 0.041427 0.036999 /
[ 442.425577] kworker/4:1H 1001 1378.730082 10 100 1378.730082 0.145252 41358.051444 /
[ 442.425906] rsyslogd 3025 3480.977057 29 120 3480.977057 1.328252 15467.321960 /autogroup-6
[ 442.426237] mdadm 3101 0.269176 1 120 0.269176 0.841264 0.000000 /autogroup-8
[ 442.426569] atd 3136 0.954776 2 120 0.954776 0.143712 0.285125 /autogroup-9
[ 442.426899] mysqld 4024 3924.483419 808 120 3924.483419 12.204679 403340.533797 /autogroup-6
[ 442.427232] bioset 4473 1429.506934 2 100 1429.506934 0.006509 0.002747 /
[ 442.427560] btrfs-endio-1 4481 1634.643810 6 120 1634.643810 0.061059 239871.284049 /
[ 442.427888] btrfs-rmw-1 4483 1634.635497 6 120 1634.635497 0.014114 239871.181403 /
[ 442.428217] btrfs-endio-rai 4484 1634.635307 6 120 1634.635307 0.012364 239871.162940 /
[ 442.428546] btrfs-endio-met 4485 1634.636449 6 120 1634.636449 0.013526 239871.147901 /
[ 442.428874] btrfs-endio-wri 4486 1634.635915 6 120 1634.635915 0.013623 239871.120772 /
[ 442.429203] btrfs-freespace 4487 1634.636106 6 120 1634.636106 0.014809 239871.054138 /
[ 442.429535] btrfs-delayed-m 4488 1634.635624 6 120 1634.635624 0.013350 239871.005792 /
[ 442.429863] btrfs-readahead 4490 1634.636001 6 120 1634.636001 0.013185 239870.915119 /
[ 442.430191] btrfs-flush_del 4491 1634.635878 6 120 1634.635878 0.013309 239870.867378 /
[ 442.430520] btrfs-qgroup-re 4492 1634.643347 5 120 1634.643347 0.021430 239870.850039 /
[ 442.430848]
[ 442.430944] cpu#5, 3411.486 MHz
[ 442.431043] .nr_running : 0
[ 442.431146] .load : 0
[ 442.431248] .nr_switches : 7645
[ 442.431350] .nr_load_updates : 2102
[ 442.431453] .nr_uninterruptible : 775
[ 442.431556] .next_balance : 4294.981369
[ 442.431659] .curr->pid : 0
[ 442.431762] .clock : 442365.450422
[ 442.431866] .cpu_load[0] : 0
[ 442.431968] .cpu_load[1] : 0
[ 442.432071] .cpu_load[2] : 0
[ 442.432172] .cpu_load[3] : 0
[ 442.432274] .cpu_load[4] : 0
[ 442.432376] .yld_count : 1
[ 442.432478] .sched_count : 7765
[ 442.432580] .sched_goidle : 3383
[ 442.432682] .avg_idle : 1000000
[ 442.432785] .ttwu_count : 3795
[ 442.432887] .ttwu_local : 2451
[ 442.432990]
[ 442.432990] cfs_rq[5]:/autogroup-6
[ 442.433144] .exec_clock : 136.183864
[ 442.433247] .MIN_vruntime : 0.000001
[ 442.433351] .min_vruntime : 4828.707667
[ 442.433454] .max_vruntime : 0.000001
[ 442.433558] .spread : 0.000000
[ 442.433661] .spread0 : -5116.371673
[ 442.433764] .nr_spread_over : 88
[ 442.433866] .nr_running : 0
[ 442.433968] .load : 0
[ 442.434070] .runnable_load_avg : 0
[ 442.434172] .blocked_load_avg : 0
[ 442.434274] .tg_load_contrib : 0
[ 442.434377] .tg_runnable_contrib : 0
[ 442.434479] .tg_load_avg : 0
[ 442.434581] .tg->runnable_avg : 1
[ 442.434682] .tg->cfs_bandwidth.timer_active: 0
[ 442.434784] .throttled : 0
[ 442.434887] .throttle_count : 0
[ 442.434988] .se->exec_start : 442365.448483
[ 442.435093] .se->vruntime : 1125.533446
[ 442.435197] .se->sum_exec_runtime : 136.183864
[ 442.435301] .se->statistics.wait_start : 0.000000
[ 442.435405] .se->statistics.sleep_start : 0.000000
[ 442.435508] .se->statistics.block_start : 0.000000
[ 442.435612] .se->statistics.sleep_max : 0.000000
[ 442.435715] .se->statistics.block_max : 0.000000
[ 442.435820] .se->statistics.exec_max : 9.986728
[ 442.435923] .se->statistics.slice_max : 0.000000
[ 442.436026] .se->statistics.wait_max : 0.033855
[ 442.436130] .se->statistics.wait_sum : 0.257970
[ 442.436234] .se->statistics.wait_count : 1474
[ 442.436336] .se->load.weight : 2
[ 442.437935] .se->avg.runnable_avg_sum : 13
[ 442.438038] .se->avg.runnable_avg_period : 48411
[ 442.438141] .se->avg.load_avg_contrib : 0
[ 442.438243] .se->avg.decay_count : 421873
[ 442.438346]
[ 442.438346] cfs_rq[5]:/
[ 442.438499] .exec_clock : 264.365610
[ 442.438603] .MIN_vruntime : 0.000001
[ 442.438706] .min_vruntime : 1128.410826
[ 442.438810] .max_vruntime : 0.000001
[ 442.438913] .spread : 0.000000
[ 442.439016] .spread0 : -8816.668514
[ 442.439121] .nr_spread_over : 2
[ 442.439222] .nr_running : 0
[ 442.439325] .load : 0
[ 442.439438] .runnable_load_avg : 0
[ 442.439540] .blocked_load_avg : 0
[ 442.439642] .tg_load_contrib : 0
[ 442.439744] .tg_runnable_contrib : 0
[ 442.439846] .tg_load_avg : 6
[ 442.439948] .tg->runnable_avg : 12
[ 442.440050] .tg->cfs_bandwidth.timer_active: 0
[ 442.440152] .throttled : 0
[ 442.440254] .throttle_count : 0
[ 442.440357] .avg->runnable_avg_sum : 13
[ 442.440459] .avg->runnable_avg_period : 48288
[ 442.440562]
[ 442.440562] rt_rq[5]:/
[ 442.440715] .rt_nr_running : 0
[ 442.440817] .rt_throttled : 0
[ 442.440918] .rt_time : 0.000000
[ 442.441022] .rt_runtime : 950.000000
[ 442.441126]
[ 442.441126] runnable tasks:
[ 442.441126] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 442.441126] ----------------------------------------------------------------------------------------------------------
[ 442.441532] watchdog/5 31 -8.984532 114 0 -8.984532 1.456254 32.394764 /
[ 442.441859] migration/5 32 0.000000 791 0 0.000000 9.046774 0.000891 /
[ 442.442187] ksoftirqd/5 33 1116.421318 42 120 1116.421318 0.373424 440091.578934 /
[ 442.442516] kworker/5:0 34 35.093325 13 120 35.093325 0.030320 540.690549 /
[ 442.442845] kworker/5:0H 35 786.861732 6 100 786.861732 0.027966 36536.941769 /
[ 442.443175] kworker/5:1 597 1117.539428 611 120 1117.539428 6.369344 439744.435645 /
[ 442.443506] rsyslogd 2993 460.368547 14 120 460.368547 2.476828 173.617609 /autogroup-6
[ 442.443836] kworker/5:1H 3039 831.850576 5 100 831.850576 0.113352 689.516336 /
[ 442.444165] nginx 3190 0.692395 1 120 0.692395 0.435221 0.000000 /autogroup-11
[ 442.444497] mysqld 4022 4826.939530 809 120 4826.939530 12.700739 403340.413374 /autogroup-6
[ 442.444828] getty 4301 0.685356 108 120 0.685356 1.944498 61.892219 /autogroup-20
[ 442.445159] udevd 4474 264.416172 1 120 264.416172 0.159301 0.000000 /autogroup-4
[ 442.445491] btrfs-transacti 4680 1116.182456 3 120 1116.182456 0.043403 29977.514675 /
[ 442.445820]
[ 442.445917] cpu#6, 3411.486 MHz
[ 442.446016] .nr_running : 0
[ 442.446117] .load : 0
[ 442.446220] .nr_switches : 12232
[ 442.446322] .nr_load_updates : 3551
[ 442.446426] .nr_uninterruptible : 562
[ 442.446529] .next_balance : 4294.981423
[ 442.446632] .curr->pid : 0
[ 442.446734] .clock : 442365.470347
[ 442.446838] .cpu_load[0] : 0
[ 442.446939] .cpu_load[1] : 0
[ 442.447041] .cpu_load[2] : 0
[ 442.447143] .cpu_load[3] : 0
[ 442.447245] .cpu_load[4] : 0
[ 442.447346] .yld_count : 1
[ 442.447448] .sched_count : 12342
[ 442.447551] .sched_goidle : 5688
[ 442.447653] .avg_idle : 1000000
[ 442.447756] .ttwu_count : 5917
[ 442.447858] .ttwu_local : 3916
[ 442.447960]
[ 442.447960] cfs_rq[6]:/autogroup-6
[ 442.448114] .exec_clock : 135.681684
[ 442.448218] .MIN_vruntime : 0.000001
[ 442.448321] .min_vruntime : 3581.900533
[ 442.448425] .max_vruntime : 0.000001
[ 442.448528] .spread : 0.000000
[ 442.448631] .spread0 : -6363.285095
[ 442.448735] .nr_spread_over : 125
[ 442.448836] .nr_running : 0
[ 442.448939] .load : 0
[ 442.449041] .runnable_load_avg : 0
[ 442.449143] .blocked_load_avg : 0
[ 442.449244] .tg_load_contrib : 0
[ 442.449348] .tg_runnable_contrib : 0
[ 442.449450] .tg_load_avg : 0
[ 442.449552] .tg->runnable_avg : 0
[ 442.449655] .tg->cfs_bandwidth.timer_active: 0
[ 442.449757] .throttled : 0
[ 442.449858] .throttle_count : 0
[ 442.449960] .se->exec_start : 442365.468019
[ 442.450065] .se->vruntime : 9033.807752
[ 442.450168] .se->sum_exec_runtime : 135.681684
[ 442.450272] .se->statistics.wait_start : 0.000000
[ 442.450375] .se->statistics.sleep_start : 0.000000
[ 442.450479] .se->statistics.block_start : 0.000000
[ 442.450582] .se->statistics.sleep_max : 0.000000
[ 442.450685] .se->statistics.block_max : 0.000000
[ 442.450788] .se->statistics.exec_max : 1.924687
[ 442.450892] .se->statistics.slice_max : 0.000000
[ 442.450996] .se->statistics.wait_max : 0.057361
[ 442.451098] .se->statistics.wait_sum : 1.004778
[ 442.451202] .se->statistics.wait_count : 1948
[ 442.451305] .se->load.weight : 2
[ 442.451406] .se->avg.runnable_avg_sum : 13
[ 442.451509] .se->avg.runnable_avg_period : 46919
[ 442.451611] .se->avg.load_avg_contrib : 0
[ 442.451713] .se->avg.decay_count : 421873
[ 442.451817]
[ 442.451817] cfs_rq[6]:/autogroup-11
[ 442.451972] .exec_clock : 15.359290
[ 442.452076] .MIN_vruntime : 0.000001
[ 442.452179] .min_vruntime : 25.073202
[ 442.452282] .max_vruntime : 0.000001
[ 442.452386] .spread : 0.000000
[ 442.452489] .spread0 : -9920.112426
[ 442.452593] .nr_spread_over : 0
[ 442.452695] .nr_running : 0
[ 442.452796] .load : 0
[ 442.452898] .runnable_load_avg : 0
[ 442.452999] .blocked_load_avg : 0
[ 442.453101] .tg_load_contrib : 0
[ 442.453203] .tg_runnable_contrib : 0
[ 442.453306] .tg_load_avg : 11
[ 442.453408] .tg->runnable_avg : 0
[ 442.453510] .tg->cfs_bandwidth.timer_active: 0
[ 442.453612] .throttled : 0
[ 442.453713] .throttle_count : 0
[ 442.453815] .se->exec_start : 442305.525383
[ 442.453919] .se->vruntime : 9021.807106
[ 442.454022] .se->sum_exec_runtime : 15.359290
[ 442.454126] .se->statistics.wait_start : 0.000000
[ 442.454229] .se->statistics.sleep_start : 0.000000
[ 442.454332] .se->statistics.block_start : 0.000000
[ 442.454434] .se->statistics.sleep_max : 0.000000
[ 442.454538] .se->statistics.block_max : 0.000000
[ 442.454641] .se->statistics.exec_max : 0.339242
[ 442.454745] .se->statistics.slice_max : 0.009873
[ 442.454849] .se->statistics.wait_max : 0.002005
[ 442.454952] .se->statistics.wait_sum : 0.002005
[ 442.455056] .se->statistics.wait_count : 826
[ 442.455158] .se->load.weight : 2
[ 442.455261] .se->avg.runnable_avg_sum : 1
[ 442.455363] .se->avg.runnable_avg_period : 47720
[ 442.455466] .se->avg.load_avg_contrib : 0
[ 442.455567] .se->avg.decay_count : 421816
[ 442.455670]
[ 442.455670] cfs_rq[6]:/
[ 442.455822] .exec_clock : 308.223493
[ 442.455925] .MIN_vruntime : 0.000001
[ 442.456029] .min_vruntime : 9033.807752
[ 442.456133] .max_vruntime : 0.000001
[ 442.456236] .spread : 0.000000
[ 442.456340] .spread0 : -911.377876
[ 442.456443] .nr_spread_over : 646
[ 442.456546] .nr_running : 0
[ 442.456648] .load : 0
[ 442.456750] .runnable_load_avg : 0
[ 442.456852] .blocked_load_avg : 0
[ 442.456954] .tg_load_contrib : 0
[ 442.457056] .tg_runnable_contrib : 0
[ 442.457158] .tg_load_avg : 7
[ 442.457260] .tg->runnable_avg : 11
[ 442.457362] .tg->cfs_bandwidth.timer_active: 0
[ 442.457464] .throttled : 0
[ 442.459064] .throttle_count : 0
[ 442.459166] .avg->runnable_avg_sum : 14
[ 442.459268] .avg->runnable_avg_period : 47717
[ 442.459373]
[ 442.459373] rt_rq[6]:/
[ 442.459525] .rt_nr_running : 0
[ 442.459627] .rt_throttled : 0
[ 442.459729] .rt_time : 0.000000
[ 442.459833] .rt_runtime : 950.000000
[ 442.459937]
[ 442.459937] runnable tasks:
[ 442.459937] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 442.459937] ----------------------------------------------------------------------------------------------------------
[ 442.460341] kthreadd 2 9004.629168 133 120 9004.629168 2.459339 149562.611708 /
[ 442.460670] watchdog/6 36 -8.979708 114 0 -8.979708 1.419955 19.064518 /
[ 442.460999] migration/6 37 0.000000 583 0 0.000000 7.676005 0.000798 /
[ 442.461327] ksoftirqd/6 38 9021.787457 60 120 9021.787457 1.056975 440747.209845 /
[ 442.461655] kworker/6:0 39 6290.796774 13 120 6290.796774 0.029128 527.622768 /
[ 442.461983] kworker/6:0H 40 8681.586920 6 100 8681.586920 0.058235 34644.619169 /
[ 442.462313] kworker/6:1 599 9021.926574 1003 120 9021.926574 11.641626 440278.129570 /
[ 442.462641] irq/44-mei_me 656 0.000000 14 49 0.000000 0.619723 0.000000 /
[ 442.462972] ext4-rsv-conver 924 8240.946018 2 100 8240.946018 0.031035 0.023098 /
[ 442.463302] kworker/6:1H 2121 8779.635829 4 100 8779.635829 0.026744 2201.095775 /
[ 442.463631] nginx 3191 11.366022 1 120 11.366022 0.414607 0.000000 /autogroup-11
[ 442.463962] nginx 3192 25.073202 811 120 25.073202 14.078193 405205.240817 /autogroup-11
[ 442.464292] mysqld 4025 3574.295754 808 120 3574.295754 11.740556 403340.542940 /autogroup-6
[ 442.464624] dovecot 4159 22.944448 18 120 22.944448 1.777530 1336.638285 /autogroup-16
[ 442.464955] getty 4304 0.971652 109 120 0.971652 2.023715 61.258946 /autogroup-23
[ 442.465286] btrfs-submit-1 4478 9019.125219 4 120 9019.125219 0.066240 239871.556343 /
[ 442.465615]
[ 442.465711] cpu#7, 3411.486 MHz
[ 442.465810] .nr_running : 0
[ 442.465913] .load : 0
[ 442.466014] .nr_switches : 13958
[ 442.466117] .nr_load_updates : 3860
[ 442.466220] .nr_uninterruptible : 604
[ 442.466323] .next_balance : 4294.981469
[ 442.466427] .curr->pid : 0
[ 442.466529] .clock : 442365.472874
[ 442.466633] .cpu_load[0] : 0
[ 442.466735] .cpu_load[1] : 0
[ 442.466837] .cpu_load[2] : 0
[ 442.466938] .cpu_load[3] : 0
[ 442.467040] .cpu_load[4] : 0
[ 442.467142] .yld_count : 0
[ 442.467244] .sched_count : 14074
[ 442.467346] .sched_goidle : 6630
[ 442.467449] .avg_idle : 1000000
[ 442.467552] .ttwu_count : 4187
[ 442.467654] .ttwu_local : 2674
[ 442.467757]
[ 442.467757] cfs_rq[7]:/autogroup-6
[ 442.467912] .exec_clock : 235.824574
[ 442.468016] .MIN_vruntime : 0.000001
[ 442.468119] .min_vruntime : 5537.389504
[ 442.468222] .max_vruntime : 0.000001
[ 442.468325] .spread : 0.000000
[ 442.468429] .spread0 : -4407.796124
[ 442.468533] .nr_spread_over : 213
[ 442.468635] .nr_running : 0
[ 442.468737] .load : 0
[ 442.468839] .runnable_load_avg : 0
[ 442.468941] .blocked_load_avg : 0
[ 442.469043] .tg_load_contrib : 0
[ 442.469145] .tg_runnable_contrib : 0
[ 442.469247] .tg_load_avg : 0
[ 442.469351] .tg->runnable_avg : 0
[ 442.469466] .tg->cfs_bandwidth.timer_active: 0
[ 442.469568] .throttled : 0
[ 442.469670] .throttle_count : 0
[ 442.469772] .se->exec_start : 442365.470857
[ 442.469876] .se->vruntime : 2130.271480
[ 442.469980] .se->sum_exec_runtime : 235.824574
[ 442.470083] .se->statistics.wait_start : 0.000000
[ 442.470186] .se->statistics.sleep_start : 0.000000
[ 442.470289] .se->statistics.block_start : 0.000000
[ 442.470392] .se->statistics.sleep_max : 0.000000
[ 442.470496] .se->statistics.block_max : 0.000000
[ 442.470598] .se->statistics.exec_max : 0.773748
[ 442.470702] .se->statistics.slice_max : 0.000000
[ 442.470805] .se->statistics.wait_max : 0.030005
[ 442.470909] .se->statistics.wait_sum : 0.323873
[ 442.471012] .se->statistics.wait_count : 2061
[ 442.471115] .se->load.weight : 2
[ 442.471218] .se->avg.runnable_avg_sum : 33
[ 442.471319] .se->avg.runnable_avg_period : 47528
[ 442.471422] .se->avg.load_avg_contrib : 0
[ 442.471524] .se->avg.decay_count : 421873
[ 442.471627]
[ 442.471627] cfs_rq[7]:/
[ 442.471780] .exec_clock : 906.945228
[ 442.471883] .MIN_vruntime : 0.000001
[ 442.471986] .min_vruntime : 2130.271480
[ 442.472090] .max_vruntime : 0.000001
[ 442.472193] .spread : 0.000000
[ 442.472297] .spread0 : -7815.020936
[ 442.472400] .nr_spread_over : 11
[ 442.472503] .nr_running : 0
[ 442.472604] .load : 0
[ 442.472706] .runnable_load_avg : 0
[ 442.472807] .blocked_load_avg : 0
[ 442.472909] .tg_load_contrib : 0
[ 442.473011] .tg_runnable_contrib : 0
[ 442.473113] .tg_load_avg : 6
[ 442.473215] .tg->runnable_avg : 10
[ 442.473317] .tg->cfs_bandwidth.timer_active: 0
[ 442.473419] .throttled : 0
[ 442.473521] .throttle_count : 0
[ 442.473623] .avg->runnable_avg_sum : 33
[ 442.473725] .avg->runnable_avg_period : 47552
[ 442.473828]
[ 442.473828] rt_rq[7]:/
[ 442.473980] .rt_nr_running : 0
[ 442.474083] .rt_throttled : 0
[ 442.474185] .rt_time : 0.000000
[ 442.474289] .rt_runtime : 950.000000
[ 442.474393]
[ 442.474393] runnable tasks:
[ 442.474393] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[ 442.474393] ----------------------------------------------------------------------------------------------------------
[ 442.474799] watchdog/7 41 -8.980665 114 0 -8.980665 1.327735 5.747945 /
[ 442.475127] migration/7 42 0.000000 619 0 0.000000 7.851498 0.000808 /
[ 442.475456] ksoftirqd/7 43 2069.587300 43 120 2069.587300 0.746714 301037.486356 /
[ 442.475784] kworker/7:0 44 206.657650 12 120 206.657650 114.455462 112.014440 /
[ 442.476112] kworker/7:0H 45 1420.076915 6 100 1420.076915 0.022407 34313.173210 /
[ 442.476440] khelper 46 10.955792 2 100 10.955792 0.004906 0.002354 /
[ 442.476769] netns 47 22.959000 2 100 22.959000 0.003532 0.001919 /
[ 442.477096] writeback 196 46.965406 2 100 46.965406 0.003584 0.001369 /
[ 442.477423] bioset 198 58.968447 2 100 58.968447 0.003349 0.001513 /
[ 442.477752] kblockd 199 70.971483 2 100 70.971483 0.003351 0.001372 /
[ 442.478081] md 346 94.978435 2 100 94.978435 0.003471 0.001431 /
[ 442.478410] kworker/7:1 447 2117.921864 628 120 2117.921864 8.406463 441029.676636 /
[ 442.478738] kswapd0 492 586.012693 3 120 586.012693 0.018964 93.205724 /
[ 442.479068] crypto 565 501.932788 2 100 501.932788 0.006100 0.003441 /
[ 442.479398] kpsmoused 672 610.094848 2 100 610.094848 0.094154 0.030677 /
[ 442.479728] deferwq 717 634.179797 2 100 634.179797 0.034165 0.003368 /
[ 442.480058] ata_sff 839 819.025334 2 100 819.025334 0.017476 0.005194 /
[ 442.480387] scsi_tmf_0 842 843.078836 2 100 843.078836 0.024724 0.003422 /
[ 442.480715] scsi_tmf_1 844 867.114334 2 100 867.114334 0.009475 0.005052 /
[ 442.481044] scsi_tmf_2 846 891.129761 2 100 891.129761 0.008669 0.004939 /
[ 442.482873] scsi_tmf_3 848 915.166073 2 100 915.166073 0.010415 0.003941 /
[ 442.483201] scsi_eh_4 849 975.323872 3 120 975.323872 0.034251 1.170933 /
[ 442.483530] scsi_tmf_4 850 939.222591 2 100 939.222591 0.031535 0.004287 /
[ 442.483858] scsi_eh_5 851 975.326229 3 120 975.326229 0.035801 0.973535 /
[ 442.484186] scsi_tmf_5 852 963.283758 2 100 963.283758 0.032461 0.003257 /
[ 442.484514] md0_raid1 892 1638.637146 29 120 1638.637146 0.585038 35377.924283 /
[ 442.484844] udevd 1111 405.605404 262 120 405.605404 42.578740 85268.067369 /autogroup-4
[ 442.485174] kworker/7:1H 1537 1667.977458 4 100 1667.977458 0.030612 3127.813179 /
[ 442.485503] kjournald 2212 1548.457495 2 120 1548.457495 0.354049 0.040124 /
[ 442.485833] mysqld 4029 5525.404781 808 120 5525.404781 12.467328 403340.541563 /autogroup-6
[ 442.486163] mysqld 4048 5537.389504 416 120 5537.389504 134.349184 401216.810737 /autogroup-6
[ 442.486493] mysqld 4077 4399.635961 2 120 4399.635961 0.159445 415.716870 /autogroup-6
[ 442.486825] getty 4300 2.870085 105 120 2.870085 4.228305 59.767820 /autogroup-19
[ 442.487157] btrfs-delalloc- 4479 2088.040334 4 120 2088.040334 0.080839 239871.453410 /
[ 442.487487]
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mount hangs after disk crash (RAID-1)
2014-05-22 2:22 mount hangs after disk crash (RAID-1) Tomasz Chmielewski
@ 2014-05-22 8:51 ` Tomasz Chmielewski
2014-05-22 15:16 ` Tomasz Chmielewski
2014-05-22 9:43 ` Duncan
1 sibling, 1 reply; 9+ messages in thread
From: Tomasz Chmielewski @ 2014-05-22 8:51 UTC (permalink / raw)
To: linux-btrfs
On Thu, 22 May 2014 03:22:58 +0100
Tomasz Chmielewski <tch@virtall.com> wrote:
> One disk in RAID-1 crashed, so powered off, changed disk, powered
> on, trying to mount degraded.
>
> Unfortunately it hangs (running 3.14.4).
>
> # mount -o degraded,compress=lzo,noatime /dev/sdb4 /home
> (...never returns...)
>
> iostat shows no disk activity.
>
> Any ideas how to recover?
Tried with 3.15-rc6 (had to hard reboot, otherwise it won't reboot), but it hangs in the same way.
# ps aux|grep mount
root 4420 0.0 0.0 20092 2476 pts/1 D+ 04:46 0:00 mount -o degraded,compress=lzo,noatime /dev/sdb4 /home
[ 119.649346] Btrfs loaded
[ 119.650297] BTRFS: device label home devid 2 transid 53234 /dev/sdb4
[ 119.650941] BTRFS info (device sdb4): allowing degraded mounts
[ 119.651002] BTRFS info (device sdb4): disk space caching is enabled
[ 119.729178] BTRFS: bdev (null) errs: wr 0, rd 58, flush 0, corrupt 0, gen 0
[ 147.976429] BTRFS error (device sdb4): free space inode generation (0) did not match free space cache generation (53233)
[ 360.024364] INFO: task mount:4420 blocked for more than 120 seconds.
[ 360.024419] Not tainted 3.15.0-rc6 #1
[ 360.024465] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 360.024520] mount D ffff88083fad16c0 0 4420 4373 0x00000000
[ 360.024650] ffff880812c836a8 0000000000000082 ffff88003723c800 ffff880812f43060
[ 360.024851] 00000000000116c0 0000000000004000 ffff880813e0b060 ffff880812f43060
[ 360.025154] ffff88003723c800 0000000000000002 ffff880812c835f8 ffffffffa039b721
[ 360.025459] Call Trace:
[ 360.025576] [<ffffffffa039b721>] ? btrfs_bio_counter_dec+0x20/0x4d [btrfs]
[ 360.025696] [<ffffffffa03662e2>] ? btrfs_map_bio+0x3c8/0x432 [btrfs]
[ 360.025804] [<ffffffff810b1bfd>] ? find_get_pages_tag+0xe9/0x123
[ 360.025909] [<ffffffff810b1f12>] ? __lock_page+0x63/0x63
[ 360.026022] [<ffffffff8139dffc>] schedule+0x6a/0x6c
[ 360.026131] [<ffffffff8139e088>] io_schedule+0x8a/0xc8
[ 360.026234] [<ffffffff810b1f1b>] sleep_on_page+0x9/0xd
[ 360.026344] [<ffffffff8139e61f>] __wait_on_bit+0x45/0x7e
[ 360.026450] [<ffffffff810b1bfd>] ? find_get_pages_tag+0xe9/0x123
[ 360.026556] [<ffffffff810b22fc>] wait_on_page_bit+0x71/0x73
[ 360.026661] [<ffffffff810685fe>] ? wake_atomic_t_function+0x28/0x28
[ 360.026767] [<ffffffff810b23b6>] filemap_fdatawait_range+0x79/0x118
[ 360.026887] [<ffffffffa035914e>] btrfs_wait_ordered_range+0x6b/0x11d [btrfs]
[ 360.027006] [<ffffffffa0379255>] __btrfs_write_out_cache+0x47a/0x6ef [btrfs]
[ 360.027119] [<ffffffffa032598d>] ? comp_keys+0x26/0x28 [btrfs]
[ 360.027233] [<ffffffffa037974f>] btrfs_write_out_cache+0x8c/0xc1 [btrfs]
[ 360.027349] [<ffffffffa035e60a>] ? free_extent_buffer+0x6f/0x7c [btrfs]
[ 360.027465] [<ffffffffa0336883>] btrfs_write_dirty_block_groups+0x54b/0x56f [btrfs]
[ 360.027638] [<ffffffffa034216d>] commit_cowonly_roots+0x165/0x234 [btrfs]
[ 360.027757] [<ffffffffa0343c6f>] btrfs_commit_transaction+0x413/0x866 [btrfs]
[ 360.027930] [<ffffffffa0384c67>] btrfs_recover_relocation+0x2a1/0x353 [btrfs]
[ 360.028103] [<ffffffffa0341b02>] open_ctree+0x1739/0x1a7c [btrfs]
[ 360.028216] [<ffffffffa03213c6>] btrfs_mount+0x4a7/0x7b5 [btrfs]
[ 360.028319] [<ffffffff810b92f5>] ? __free_pages+0x1c/0x48
[ 360.028423] [<ffffffff81173bf4>] ? selinux_sb_copy_data+0x1a0/0x1d1
[ 360.028531] [<ffffffff810fca31>] mount_fs+0x64/0x14f
[ 360.028636] [<ffffffff810c896c>] ? __alloc_percpu+0xb/0xd
[ 360.028740] [<ffffffff81111c75>] vfs_kern_mount+0x61/0xf7
[ 360.028847] [<ffffffff81113cd7>] do_mount+0x70b/0x8af
[ 360.028955] [<ffffffff811140b5>] SyS_mount+0x80/0xb9
[ 360.029058] [<ffffffff813a10e2>] system_call_fastpath+0x16/0x1b
[ 360.029172] INFO: task btrfs-transacti:4632 blocked for more than 120 seconds.
[ 360.029328] Not tainted 3.15.0-rc6 #1
[ 360.029430] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 360.029587] btrfs-transacti D ffff88083fad16c0 0 4632 2 0x00000000
[ 360.029767] ffff880811f93d78 0000000000000046 0000000000000000 ffff880812f160c0
[ 360.030071] 00000000000116c0 0000000000004000 ffff880813e0b060 ffff880812f160c0
[ 360.030376] ffff880811f93db8 ffffffff8139de33 0000000200000000 ffff880812f160c0
[ 360.030680] Call Trace:
[ 360.030779] [<ffffffff8139de33>] ? __schedule+0x743/0x8a2
[ 360.030882] [<ffffffff81061f28>] ? enqueue_entity+0x611/0x73f
[ 360.030987] [<ffffffff8139dffc>] schedule+0x6a/0x6c
[ 360.031108] [<ffffffffa034330e>] wait_current_trans.isra.36+0x94/0xe2 [btrfs]
[ 360.031276] [<ffffffff810685a4>] ? __wake_up_sync+0xd/0xd
[ 360.031389] [<ffffffffa03445ea>] start_transaction+0x205/0x463 [btrfs]
[ 360.031503] [<ffffffffa0344895>] btrfs_attach_transaction+0x12/0x14 [btrfs]
[ 360.031623] [<ffffffffa0341f12>] transaction_kthread+0xcd/0x1c3 [btrfs]
[ 360.031737] [<ffffffffa0341e45>] ? open_ctree+0x1a7c/0x1a7c [btrfs]
[ 360.031843] [<ffffffff81050f4e>] kthread+0xcd/0xd5
[ 360.031945] [<ffffffff81050e81>] ? kthread_freezable_should_stop+0x43/0x43
[ 360.032050] [<ffffffff813a103c>] ret_from_fork+0x7c/0xb0
[ 360.032155] [<ffffffff81050e81>] ? kthread_freezable_should_stop+0x43/0x43
[ 479.960978] INFO: task mount:4420 blocked for more than 120 seconds.
[ 479.961088] Not tainted 3.15.0-rc6 #1
[ 479.961188] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 479.961348] mount D ffff88083fad16c0 0 4420 4373 0x00000000
[ 479.961527] ffff880812c836a8 0000000000000082 ffff88003723c800 ffff880812f43060
[ 479.961840] 00000000000116c0 0000000000004000 ffff880813e0b060 ffff880812f43060
[ 479.962149] ffff88003723c800 0000000000000002 ffff880812c835f8 ffffffffa039b721
[ 479.962455] Call Trace:
[ 479.962576] [<ffffffffa039b721>] ? btrfs_bio_counter_dec+0x20/0x4d [btrfs]
[ 479.962693] [<ffffffffa03662e2>] ? btrfs_map_bio+0x3c8/0x432 [btrfs]
[ 479.962801] [<ffffffff810b1bfd>] ? find_get_pages_tag+0xe9/0x123
[ 479.962907] [<ffffffff810b1f12>] ? __lock_page+0x63/0x63
[ 479.963010] [<ffffffff8139dffc>] schedule+0x6a/0x6c
[ 479.963109] [<ffffffff8139e088>] io_schedule+0x8a/0xc8
[ 479.963213] [<ffffffff810b1f1b>] sleep_on_page+0x9/0xd
[ 479.963325] [<ffffffff8139e61f>] __wait_on_bit+0x45/0x7e
[ 479.963426] [<ffffffff810b1bfd>] ? find_get_pages_tag+0xe9/0x123
[ 479.963530] [<ffffffff810b22fc>] wait_on_page_bit+0x71/0x73
[ 479.963636] [<ffffffff810685fe>] ? wake_atomic_t_function+0x28/0x28
[ 479.963743] [<ffffffff810b23b6>] filemap_fdatawait_range+0x79/0x118
[ 479.963861] [<ffffffffa035914e>] btrfs_wait_ordered_range+0x6b/0x11d [btrfs]
[ 479.963991] [<ffffffffa0379255>] __btrfs_write_out_cache+0x47a/0x6ef [btrfs]
[ 479.964109] [<ffffffffa032598d>] ? comp_keys+0x26/0x28 [btrfs]
[ 479.964227] [<ffffffffa037974f>] btrfs_write_out_cache+0x8c/0xc1 [btrfs]
[ 479.964345] [<ffffffffa035e60a>] ? free_extent_buffer+0x6f/0x7c [btrfs]
[ 479.964463] [<ffffffffa0336883>] btrfs_write_dirty_block_groups+0x54b/0x56f [btrfs]
[ 479.964634] [<ffffffffa034216d>] commit_cowonly_roots+0x165/0x234 [btrfs]
[ 479.964753] [<ffffffffa0343c6f>] btrfs_commit_transaction+0x413/0x866 [btrfs]
[ 479.964928] [<ffffffffa0384c67>] btrfs_recover_relocation+0x2a1/0x353 [btrfs]
[ 479.966618] [<ffffffffa0341b02>] open_ctree+0x1739/0x1a7c [btrfs]
[ 479.966732] [<ffffffffa03213c6>] btrfs_mount+0x4a7/0x7b5 [btrfs]
[ 479.966837] [<ffffffff810b92f5>] ? __free_pages+0x1c/0x48
[ 479.966940] [<ffffffff81173bf4>] ? selinux_sb_copy_data+0x1a0/0x1d1
[ 479.967055] [<ffffffff810fca31>] mount_fs+0x64/0x14f
[ 479.967161] [<ffffffff810c896c>] ? __alloc_percpu+0xb/0xd
[ 479.967265] [<ffffffff81111c75>] vfs_kern_mount+0x61/0xf7
[ 479.967367] [<ffffffff81113cd7>] do_mount+0x70b/0x8af
[ 479.967470] [<ffffffff811140b5>] SyS_mount+0x80/0xb9
[ 479.967576] [<ffffffff813a10e2>] system_call_fastpath+0x16/0x1b
[ 479.967684] INFO: task btrfs-transacti:4632 blocked for more than 120 seconds.
[ 479.967839] Not tainted 3.15.0-rc6 #1
[ 479.967938] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 479.968099] btrfs-transacti D ffff88083fad16c0 0 4632 2 0x00000000
[ 479.968278] ffff880811f93d78 0000000000000046 0000000000000000 ffff880812f160c0
[ 479.968585] 00000000000116c0 0000000000004000 ffff880813e0b060 ffff880812f160c0
[ 479.968896] ffff880811f93db8 ffffffff8139de33 0000000200000000 ffff880812f160c0
[ 479.969207] Call Trace:
[ 479.969306] [<ffffffff8139de33>] ? __schedule+0x743/0x8a2
[ 479.969415] [<ffffffff81061f28>] ? enqueue_entity+0x611/0x73f
[ 479.969524] [<ffffffff8139dffc>] schedule+0x6a/0x6c
[ 479.969637] [<ffffffffa034330e>] wait_current_trans.isra.36+0x94/0xe2 [btrfs]
[ 479.969799] [<ffffffff810685a4>] ? __wake_up_sync+0xd/0xd
[ 479.969910] [<ffffffffa03445ea>] start_transaction+0x205/0x463 [btrfs]
[ 479.970024] [<ffffffffa0344895>] btrfs_attach_transaction+0x12/0x14 [btrfs]
[ 479.970140] [<ffffffffa0341f12>] transaction_kthread+0xcd/0x1c3 [btrfs]
[ 479.970251] [<ffffffffa0341e45>] ? open_ctree+0x1a7c/0x1a7c [btrfs]
[ 479.970359] [<ffffffff81050f4e>] kthread+0xcd/0xd5
[ 479.970462] [<ffffffff81050e81>] ? kthread_freezable_should_stop+0x43/0x43
[ 479.970564] [<ffffffff813a103c>] ret_from_fork+0x7c/0xb0
[ 479.970666] [<ffffffff81050e81>] ? kthread_freezable_should_stop+0x43/0x43
[ 599.897604] INFO: task mount:4420 blocked for more than 120 seconds.
[ 599.897727] Not tainted 3.15.0-rc6 #1
[ 599.897813] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 599.897946] mount D ffff88083fad16c0 0 4420 4373 0x00000000
[ 599.898110] ffff880812c836a8 0000000000000082 ffff88003723c800 ffff880812f43060
[ 599.898390] 00000000000116c0 0000000000004000 ffff880813e0b060 ffff880812f43060
[ 599.898668] ffff88003723c800 0000000000000002 ffff880812c835f8 ffffffffa039b721
[ 599.898955] Call Trace:
[ 599.899043] [<ffffffffa039b721>] ? btrfs_bio_counter_dec+0x20/0x4d [btrfs]
[ 599.899133] [<ffffffffa03662e2>] ? btrfs_map_bio+0x3c8/0x432 [btrfs]
[ 599.899220] [<ffffffff810b1bfd>] ? find_get_pages_tag+0xe9/0x123
[ 599.899303] [<ffffffff810b1f12>] ? __lock_page+0x63/0x63
[ 599.899385] [<ffffffff8139dffc>] schedule+0x6a/0x6c
[ 599.899466] [<ffffffff8139e088>] io_schedule+0x8a/0xc8
[ 599.899546] [<ffffffff810b1f1b>] sleep_on_page+0x9/0xd
[ 599.899640] [<ffffffff8139e61f>] __wait_on_bit+0x45/0x7e
[ 599.899744] [<ffffffff810b1bfd>] ? find_get_pages_tag+0xe9/0x123
[ 599.899834] [<ffffffff810b22fc>] wait_on_page_bit+0x71/0x73
[ 599.899923] [<ffffffff810685fe>] ? wake_atomic_t_function+0x28/0x28
[ 599.900013] [<ffffffff810b23b6>] filemap_fdatawait_range+0x79/0x118
[ 599.900109] [<ffffffffa035914e>] btrfs_wait_ordered_range+0x6b/0x11d [btrfs]
[ 599.900205] [<ffffffffa0379255>] __btrfs_write_out_cache+0x47a/0x6ef [btrfs]
[ 599.900299] [<ffffffffa032598d>] ? comp_keys+0x26/0x28 [btrfs]
[ 599.900392] [<ffffffffa037974f>] btrfs_write_out_cache+0x8c/0xc1 [btrfs]
[ 599.900488] [<ffffffffa035e60a>] ? free_extent_buffer+0x6f/0x7c [btrfs]
[ 599.900582] [<ffffffffa0336883>] btrfs_write_dirty_block_groups+0x54b/0x56f [btrfs]
[ 599.900722] [<ffffffffa034216d>] commit_cowonly_roots+0x165/0x234 [btrfs]
[ 599.900816] [<ffffffffa0343c6f>] btrfs_commit_transaction+0x413/0x866 [btrfs]
[ 599.900955] [<ffffffffa0384c67>] btrfs_recover_relocation+0x2a1/0x353 [btrfs]
[ 599.901093] [<ffffffffa0341b02>] open_ctree+0x1739/0x1a7c [btrfs]
[ 599.901185] [<ffffffffa03213c6>] btrfs_mount+0x4a7/0x7b5 [btrfs]
[ 599.901275] [<ffffffff810b92f5>] ? __free_pages+0x1c/0x48
[ 599.901363] [<ffffffff81173bf4>] ? selinux_sb_copy_data+0x1a0/0x1d1
[ 599.901454] [<ffffffff810fca31>] mount_fs+0x64/0x14f
[ 599.901542] [<ffffffff810c896c>] ? __alloc_percpu+0xb/0xd
[ 599.901630] [<ffffffff81111c75>] vfs_kern_mount+0x61/0xf7
[ 599.901718] [<ffffffff81113cd7>] do_mount+0x70b/0x8af
[ 599.901821] [<ffffffff811140b5>] SyS_mount+0x80/0xb9
[ 599.901902] [<ffffffff813a10e2>] system_call_fastpath+0x16/0x1b
[ 599.901987] INFO: task btrfs-transacti:4632 blocked for more than 120 seconds.
[ 599.902112] Not tainted 3.15.0-rc6 #1
[ 599.902190] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 599.902317] btrfs-transacti D ffff88083fad16c0 0 4632 2 0x00000000
[ 599.902459] ffff880811f93d78 0000000000000046 0000000000000000 ffff880812f160c0
[ 599.902702] 00000000000116c0 0000000000004000 ffff880813e0b060 ffff880812f160c0
[ 599.902993] ffff880811f93db8 ffffffff8139de33 0000000200000000 ffff880812f160c0
[ 599.903277] Call Trace:
[ 599.903361] [<ffffffff8139de33>] ? __schedule+0x743/0x8a2
[ 599.903450] [<ffffffff81061f28>] ? enqueue_entity+0x611/0x73f
[ 599.903538] [<ffffffff8139dffc>] schedule+0x6a/0x6c
[ 599.903637] [<ffffffffa034330e>] wait_current_trans.isra.36+0x94/0xe2 [btrfs]
[ 599.903771] [<ffffffff810685a4>] ? __wake_up_sync+0xd/0xd
[ 599.903864] [<ffffffffa03445ea>] start_transaction+0x205/0x463 [btrfs]
[ 599.903958] [<ffffffffa0344895>] btrfs_attach_transaction+0x12/0x14 [btrfs]
[ 599.904051] [<ffffffffa0341f12>] transaction_kthread+0xcd/0x1c3 [btrfs]
[ 599.904145] [<ffffffffa0341e45>] ? open_ctree+0x1a7c/0x1a7c [btrfs]
[ 599.904238] [<ffffffff81050f4e>] kthread+0xcd/0xd5
[ 599.904342] [<ffffffff81050e81>] ? kthread_freezable_should_stop+0x43/0x43
[ 599.904425] [<ffffffff813a103c>] ret_from_fork+0x7c/0xb0
[ 599.904507] [<ffffffff81050e81>] ? kthread_freezable_should_stop+0x43/0x43
[ 719.834220] INFO: task mount:4420 blocked for more than 120 seconds.
[ 719.834329] Not tainted 3.15.0-rc6 #1
[ 719.834414] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 719.834548] mount D ffff88083fad16c0 0 4420 4373 0x00000000
[ 719.834712] ffff880812c836a8 0000000000000082 ffff88003723c800 ffff880812f43060
[ 719.834991] 00000000000116c0 0000000000004000 ffff880813e0b060 ffff880812f43060
[ 719.835268] ffff88003723c800 0000000000000002 ffff880812c835f8 ffffffffa039b721
[ 719.835548] Call Trace:
[ 719.835642] [<ffffffffa039b721>] ? btrfs_bio_counter_dec+0x20/0x4d [btrfs]
[ 719.835739] [<ffffffffa03662e2>] ? btrfs_map_bio+0x3c8/0x432 [btrfs]
[ 719.835857] [<ffffffff810b1bfd>] ? find_get_pages_tag+0xe9/0x123
[ 719.835950] [<ffffffff810b1f12>] ? __lock_page+0x63/0x63
[ 719.836033] [<ffffffff8139dffc>] schedule+0x6a/0x6c
[ 719.836113] [<ffffffff8139e088>] io_schedule+0x8a/0xc8
[ 719.836194] [<ffffffff810b1f1b>] sleep_on_page+0x9/0xd
[ 719.836275] [<ffffffff8139e61f>] __wait_on_bit+0x45/0x7e
[ 719.836356] [<ffffffff810b1bfd>] ? find_get_pages_tag+0xe9/0x123
[ 719.836439] [<ffffffff810b22fc>] wait_on_page_bit+0x71/0x73
[ 719.836521] [<ffffffff810685fe>] ? wake_atomic_t_function+0x28/0x28
[ 719.836604] [<ffffffff810b23b6>] filemap_fdatawait_range+0x79/0x118
[ 719.836692] [<ffffffffa035914e>] btrfs_wait_ordered_range+0x6b/0x11d [btrfs]
[ 719.836783] [<ffffffffa0379255>] __btrfs_write_out_cache+0x47a/0x6ef [btrfs]
[ 719.836897] [<ffffffffa032598d>] ? comp_keys+0x26/0x28 [btrfs]
[ 719.836999] [<ffffffffa037974f>] btrfs_write_out_cache+0x8c/0xc1 [btrfs]
[ 719.837094] [<ffffffffa035e60a>] ? free_extent_buffer+0x6f/0x7c [btrfs]
[ 719.837187] [<ffffffffa0336883>] btrfs_write_dirty_block_groups+0x54b/0x56f [btrfs]
[ 719.837327] [<ffffffffa034216d>] commit_cowonly_roots+0x165/0x234 [btrfs]
[ 719.837422] [<ffffffffa0343c6f>] btrfs_commit_transaction+0x413/0x866 [btrfs]
[ 719.837560] [<ffffffffa0384c67>] btrfs_recover_relocation+0x2a1/0x353 [btrfs]
[ 719.837697] [<ffffffffa0341b02>] open_ctree+0x1739/0x1a7c [btrfs]
[ 719.837789] [<ffffffffa03213c6>] btrfs_mount+0x4a7/0x7b5 [btrfs]
[ 719.837879] [<ffffffff810b92f5>] ? __free_pages+0x1c/0x48
[ 719.837968] [<ffffffff81173bf4>] ? selinux_sb_copy_data+0x1a0/0x1d1
[ 719.838059] [<ffffffff810fca31>] mount_fs+0x64/0x14f
[ 719.838147] [<ffffffff810c896c>] ? __alloc_percpu+0xb/0xd
[ 719.838236] [<ffffffff81111c75>] vfs_kern_mount+0x61/0xf7
[ 719.838350] [<ffffffff81113cd7>] do_mount+0x70b/0x8af
[ 719.838442] [<ffffffff811140b5>] SyS_mount+0x80/0xb9
[ 719.838522] [<ffffffff813a10e2>] system_call_fastpath+0x16/0x1b
[ 719.838606] INFO: task btrfs-transacti:4632 blocked for more than 120 seconds.
[ 719.838732] Not tainted 3.15.0-rc6 #1
[ 719.838811] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 719.838938] btrfs-transacti D ffff88083fad16c0 0 4632 2 0x00000000
[ 719.839079] ffff880811f93d78 0000000000000046 0000000000000000 ffff880812f160c0
[ 719.839323] 00000000000116c0 0000000000004000 ffff880813e0b060 ffff880812f160c0
[ 719.839619] ffff880811f93db8 ffffffff8139de33 0000000200000000 ffff880812f160c0
[ 719.839899] Call Trace:
[ 719.839983] [<ffffffff8139de33>] ? __schedule+0x743/0x8a2
[ 719.840072] [<ffffffff81061f28>] ? enqueue_entity+0x611/0x73f
[ 719.840160] [<ffffffff8139dffc>] schedule+0x6a/0x6c
[ 719.840251] [<ffffffffa034330e>] wait_current_trans.isra.36+0x94/0xe2 [btrfs]
[ 719.840384] [<ffffffff810685a4>] ? __wake_up_sync+0xd/0xd
[ 719.840476] [<ffffffffa03445ea>] start_transaction+0x205/0x463 [btrfs]
[ 719.840569] [<ffffffffa0344895>] btrfs_attach_transaction+0x12/0x14 [btrfs]
[ 719.840662] [<ffffffffa0341f12>] transaction_kthread+0xcd/0x1c3 [btrfs]
[ 719.840755] [<ffffffffa0341e45>] ? open_ctree+0x1a7c/0x1a7c [btrfs]
[ 719.840879] [<ffffffff81050f4e>] kthread+0xcd/0xd5
[ 719.842225] [<ffffffff81050e81>] ? kthread_freezable_should_stop+0x43/0x43
[ 719.842315] [<ffffffff813a103c>] ret_from_fork+0x7c/0xb0
[ 719.842403] [<ffffffff81050e81>] ? kthread_freezable_should_stop+0x43/0x43
[ 839.770843] INFO: task mount:4420 blocked for more than 120 seconds.
[ 839.770948] Not tainted 3.15.0-rc6 #1
[ 839.771043] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 839.771198] mount D ffff88083fad16c0 0 4420 4373 0x00000000
[ 839.771414] ffff880812c836a8 0000000000000082 ffff88003723c800 ffff880812f43060
[ 839.771736] 00000000000116c0 0000000000004000 ffff880813e0b060 ffff880812f43060
[ 839.772058] ffff88003723c800 0000000000000002 ffff880812c835f8 ffffffffa039b721
[ 839.772393] Call Trace:
[ 839.772513] [<ffffffffa039b721>] ? btrfs_bio_counter_dec+0x20/0x4d [btrfs]
[ 839.772629] [<ffffffffa03662e2>] ? btrfs_map_bio+0x3c8/0x432 [btrfs]
[ 839.772733] [<ffffffff810b1bfd>] ? find_get_pages_tag+0xe9/0x123
[ 839.772833] [<ffffffff810b1f12>] ? __lock_page+0x63/0x63
[ 839.772933] [<ffffffff8139dffc>] schedule+0x6a/0x6c
[ 839.773031] [<ffffffff8139e088>] io_schedule+0x8a/0xc8
[ 839.773130] [<ffffffff810b1f1b>] sleep_on_page+0x9/0xd
[ 839.773229] [<ffffffff8139e61f>] __wait_on_bit+0x45/0x7e
[ 839.773368] [<ffffffff810b1bfd>] ? find_get_pages_tag+0xe9/0x123
[ 839.773477] [<ffffffff810b22fc>] wait_on_page_bit+0x71/0x73
[ 839.773585] [<ffffffff810685fe>] ? wake_atomic_t_function+0x28/0x28
[ 839.773694] [<ffffffff810b23b6>] filemap_fdatawait_range+0x79/0x118
[ 839.773815] [<ffffffffa035914e>] btrfs_wait_ordered_range+0x6b/0x11d [btrfs]
[ 839.773936] [<ffffffffa0379255>] __btrfs_write_out_cache+0x47a/0x6ef [btrfs]
[ 839.774051] [<ffffffffa032598d>] ? comp_keys+0x26/0x28 [btrfs]
[ 839.774171] [<ffffffffa037974f>] btrfs_write_out_cache+0x8c/0xc1 [btrfs]
[ 839.774291] [<ffffffffa035e60a>] ? free_extent_buffer+0x6f/0x7c [btrfs]
[ 839.774407] [<ffffffffa0336883>] btrfs_write_dirty_block_groups+0x54b/0x56f [btrfs]
[ 839.774581] [<ffffffffa034216d>] commit_cowonly_roots+0x165/0x234 [btrfs]
[ 839.774698] [<ffffffffa0343c6f>] btrfs_commit_transaction+0x413/0x866 [btrfs]
[ 839.774887] [<ffffffffa0384c67>] btrfs_recover_relocation+0x2a1/0x353 [btrfs]
[ 839.775057] [<ffffffffa0341b02>] open_ctree+0x1739/0x1a7c [btrfs]
[ 839.775163] [<ffffffffa03213c6>] btrfs_mount+0x4a7/0x7b5 [btrfs]
[ 839.775262] [<ffffffff810b92f5>] ? __free_pages+0x1c/0x48
[ 839.775365] [<ffffffff81173bf4>] ? selinux_sb_copy_data+0x1a0/0x1d1
[ 839.775468] [<ffffffff810fca31>] mount_fs+0x64/0x14f
[ 839.775565] [<ffffffff810c896c>] ? __alloc_percpu+0xb/0xd
[ 839.775665] [<ffffffff81111c75>] vfs_kern_mount+0x61/0xf7
[ 839.775764] [<ffffffff81113cd7>] do_mount+0x70b/0x8af
[ 839.775862] [<ffffffff811140b5>] SyS_mount+0x80/0xb9
[ 839.776003] [<ffffffff813a10e2>] system_call_fastpath+0x16/0x1b
[ 839.776114] INFO: task btrfs-transacti:4632 blocked for more than 120 seconds.
[ 839.776274] Not tainted 3.15.0-rc6 #1
[ 839.776376] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 839.776537] btrfs-transacti D ffff88083fad16c0 0 4632 2 0x00000000
[ 839.776724] ffff880811f93d78 0000000000000046 0000000000000000 ffff880812f160c0
[ 839.777045] 00000000000116c0 0000000000004000 ffff880813e0b060 ffff880812f160c0
[ 839.777363] ffff880811f93db8 ffffffff8139de33 0000000200000000 ffff880812f160c0
[ 839.777684] Call Trace:
[ 839.777798] [<ffffffff8139de33>] ? __schedule+0x743/0x8a2
[ 839.777901] [<ffffffff81061f28>] ? enqueue_entity+0x611/0x73f
[ 839.778002] [<ffffffff8139dffc>] schedule+0x6a/0x6c
[ 839.778109] [<ffffffffa034330e>] wait_current_trans.isra.36+0x94/0xe2 [btrfs]
[ 839.778265] [<ffffffff810685a4>] ? __wake_up_sync+0xd/0xd
[ 839.778375] [<ffffffffa03445ea>] start_transaction+0x205/0x463 [btrfs]
[ 839.778485] [<ffffffffa0344895>] btrfs_attach_transaction+0x12/0x14 [btrfs]
[ 839.778594] [<ffffffffa0341f12>] transaction_kthread+0xcd/0x1c3 [btrfs]
[ 839.778704] [<ffffffffa0341e45>] ? open_ctree+0x1a7c/0x1a7c [btrfs]
[ 839.778806] [<ffffffff81050f4e>] kthread+0xcd/0xd5
[ 839.778947] [<ffffffff81050e81>] ? kthread_freezable_should_stop+0x43/0x43
[ 839.779055] [<ffffffff813a103c>] ret_from_fork+0x7c/0xb0
[ 839.779162] [<ffffffff81050e81>] ? kthread_freezable_should_stop+0x43/0x43
[21899.485094] SysRq : Show Blocked State
[21899.485163] task PC stack pid father
[21899.485242] mount D ffff88083fad16c0 0 4420 4373 0x00000000
[21899.485315] ffff880812c836a8 0000000000000082 ffff88003723c800 ffff880812f43060
[21899.485441] 00000000000116c0 0000000000004000 ffff880813e0b060 ffff880812f43060
[21899.485566] ffff88003723c800 0000000000000002 ffff880812c835f8 ffffffffa039b721
[21899.485691] Call Trace:
[21899.485771] [<ffffffffa039b721>] ? btrfs_bio_counter_dec+0x20/0x4d [btrfs]
[21899.485855] [<ffffffffa03662e2>] ? btrfs_map_bio+0x3c8/0x432 [btrfs]
[21899.485927] [<ffffffff810b1bfd>] ? find_get_pages_tag+0xe9/0x123
[21899.485996] [<ffffffff810b1f12>] ? __lock_page+0x63/0x63
[21899.486064] [<ffffffff8139dffc>] schedule+0x6a/0x6c
[21899.486130] [<ffffffff8139e088>] io_schedule+0x8a/0xc8
[21899.486196] [<ffffffff810b1f1b>] sleep_on_page+0x9/0xd
[21899.486262] [<ffffffff8139e61f>] __wait_on_bit+0x45/0x7e
[21899.486329] [<ffffffff810b1bfd>] ? find_get_pages_tag+0xe9/0x123
[21899.486397] [<ffffffff810b22fc>] wait_on_page_bit+0x71/0x73
[21899.486465] [<ffffffff810685fe>] ? wake_atomic_t_function+0x28/0x28
[21899.486535] [<ffffffff810b23b6>] filemap_fdatawait_range+0x79/0x118
[21899.486616] [<ffffffffa035914e>] btrfs_wait_ordered_range+0x6b/0x11d [btrfs]
[21899.486699] [<ffffffffa0379255>] __btrfs_write_out_cache+0x47a/0x6ef [btrfs]
[21899.486775] [<ffffffffa032598d>] ? comp_keys+0x26/0x28 [btrfs]
[21899.486853] [<ffffffffa037974f>] btrfs_write_out_cache+0x8c/0xc1 [btrfs]
[21899.486934] [<ffffffffa035e60a>] ? free_extent_buffer+0x6f/0x7c [btrfs]
[21899.487012] [<ffffffffa0336883>] btrfs_write_dirty_block_groups+0x54b/0x56f [btrfs]
[21899.487147] [<ffffffffa034216d>] commit_cowonly_roots+0x165/0x234 [btrfs]
[21899.487226] [<ffffffffa0343c6f>] btrfs_commit_transaction+0x413/0x866 [btrfs]
[21899.487360] [<ffffffffa0384c67>] btrfs_recover_relocation+0x2a1/0x353 [btrfs]
[21899.487492] [<ffffffffa0341b02>] open_ctree+0x1739/0x1a7c [btrfs]
[21899.487565] [<ffffffffa03213c6>] btrfs_mount+0x4a7/0x7b5 [btrfs]
[21899.487633] [<ffffffff810b92f5>] ? __free_pages+0x1c/0x48
[21899.487701] [<ffffffff81173bf4>] ? selinux_sb_copy_data+0x1a0/0x1d1
[21899.487770] [<ffffffff810fca31>] mount_fs+0x64/0x14f
[21899.487836] [<ffffffff810c896c>] ? __alloc_percpu+0xb/0xd
[21899.487903] [<ffffffff81111c75>] vfs_kern_mount+0x61/0xf7
[21899.487969] [<ffffffff81113cd7>] do_mount+0x70b/0x8af
[21899.488035] [<ffffffff811140b5>] SyS_mount+0x80/0xb9
[21899.488101] [<ffffffff813a10e2>] system_call_fastpath+0x16/0x1b
[21899.488169] btrfs-transacti D ffff88083fad16c0 0 4632 2 0x00000000
[21899.488240] ffff880811f93d78 0000000000000046 0000000000000000 ffff880812f160c0
[21899.488365] 00000000000116c0 0000000000004000 ffff880813e0b060 ffff880812f160c0
[21899.488489] ffff880811f93db8 ffffffff8139de33 0000000200000000 ffff880812f160c0
[21899.488614] Call Trace:
[21899.488675] [<ffffffff8139de33>] ? __schedule+0x743/0x8a2
[21899.488742] [<ffffffff81061f28>] ? enqueue_entity+0x611/0x73f
[21899.488810] [<ffffffff8139dffc>] schedule+0x6a/0x6c
[21899.488884] [<ffffffffa034330e>] wait_current_trans.isra.36+0x94/0xe2 [btrfs]
[21899.489006] [<ffffffff810685a4>] ? __wake_up_sync+0xd/0xd
[21899.489082] [<ffffffffa03445ea>] start_transaction+0x205/0x463 [btrfs]
[21899.489160] [<ffffffffa0344895>] btrfs_attach_transaction+0x12/0x14 [btrfs]
[21899.489238] [<ffffffffa0341f12>] transaction_kthread+0xcd/0x1c3 [btrfs]
[21899.489314] [<ffffffffa0341e45>] ? open_ctree+0x1a7c/0x1a7c [btrfs]
[21899.489385] [<ffffffff81050f4e>] kthread+0xcd/0xd5
[21899.489451] [<ffffffff81050e81>] ? kthread_freezable_should_stop+0x43/0x43
[21899.489519] [<ffffffff813a103c>] ret_from_fork+0x7c/0xb0
[21899.489586] [<ffffffff81050e81>] ? kthread_freezable_should_stop+0x43/0x43
[21899.489656] Sched Debug Version: v0.11, 3.15.0-rc6 #1
[21899.489721] ktime : 21911055.649977
[21899.489788] sched_clk : 21899489.655251
[21899.489855] cpu_clk : 21899489.655289
[21899.489922] jiffies : 4297128402
[21899.489988] sched_clock_stable() : 1
[21899.490052]
[21899.490108] sysctl_sched
[21899.490167] .sysctl_sched_latency : 24.000000
[21899.490234] .sysctl_sched_min_granularity : 3.000000
[21899.490300] .sysctl_sched_wakeup_granularity : 4.000000
[21899.490366] .sysctl_sched_child_runs_first : 0
[21899.490430] .sysctl_sched_features : 11899
[21899.490496] .sysctl_sched_tunable_scaling : 1 (logaritmic)
[21899.490562]
[21899.490620] cpu#0, 3411.454 MHz
[21899.490680] .nr_running : 0
[21899.490744] .load : 0
[21899.490807] .nr_switches : 11468946
[21899.490871] .nr_load_updates : 866697
[21899.490935] .nr_uninterruptible : -11546
[21899.490999] .next_balance : 4297.128396
[21899.491064] .curr->pid : 0
[21899.491128] .clock : 21899484.972881
[21899.491193] .cpu_load[0] : 0
[21899.491256] .cpu_load[1] : 0
[21899.491319] .cpu_load[2] : 0
[21899.491382] .cpu_load[3] : 0
[21899.491445] .cpu_load[4] : 0
[21899.491509] .yld_count : 19
[21899.491572] .sched_count : 11473794
[21899.491637] .sched_goidle : 4188455
[21899.491701] .avg_idle : 1000000
[21899.491766] .max_idle_balance_cost : 500000
[21899.491830] .ttwu_count : 9483428
[21899.491894] .ttwu_local : 7207661
[21899.491959]
[21899.491959] cfs_rq[0]:/autogroup-25
[21899.493588] .exec_clock : 434.190881
[21899.493652] .MIN_vruntime : 0.000001
[21899.493717] .min_vruntime : 432.817689
[21899.493782] .max_vruntime : 0.000001
[21899.493846] .spread : 0.000000
[21899.493910] .spread0 : -175901.363109
[21899.493975] .nr_spread_over : 0
[21899.494038] .nr_running : 0
[21899.494102] .load : 0
[21899.494165] .runnable_load_avg : 0
[21899.494228] .blocked_load_avg : 0
[21899.494291] .tg_load_contrib : 0
[21899.494354] .tg_runnable_contrib : 0
[21899.494417] .tg_load_avg : 0
[21899.494479] .tg->runnable_avg : 0
[21899.494543] .tg->cfs_bandwidth.timer_active: 0
[21899.494606] .throttled : 0
[21899.494669] .throttle_count : 0
[21899.494732] .se->exec_start : 21899484.967126
[21899.494798] .se->vruntime : 176334.180798
[21899.494863] .se->sum_exec_runtime : 434.190881
[21899.494927] .se->statistics.wait_start : 0.000000
[21899.494992] .se->statistics.sleep_start : 0.000000
[21899.495056] .se->statistics.block_start : 0.000000
[21899.495120] .se->statistics.sleep_max : 0.000000
[21899.495185] .se->statistics.block_max : 0.000000
[21899.495249] .se->statistics.exec_max : 6.710813
[21899.495314] .se->statistics.slice_max : 0.240705
[21899.495378] .se->statistics.wait_max : 0.102857
[21899.495443] .se->statistics.wait_sum : 1.889409
[21899.495507] .se->statistics.wait_count : 4889
[21899.495570] .se->load.weight : 2
[21899.495633] .se->avg.runnable_avg_sum : 58
[21899.495697] .se->avg.runnable_avg_period : 48382
[21899.495761] .se->avg.load_avg_contrib : 0
[21899.495824] .se->avg.decay_count : 20884977
[21899.495889]
[21899.495889] cfs_rq[0]:/
[21899.496003] .exec_clock : 262940.736760
[21899.496068] .MIN_vruntime : 0.000001
[21899.496132] .min_vruntime : 176334.180798
[21899.496197] .max_vruntime : 0.000001
[21899.496261] .spread : 0.000000
[21899.496326] .spread0 : 0.000000
[21899.496390] .nr_spread_over : 35
[21899.496454] .nr_running : 0
[21899.496516] .load : 0
[21899.496579] .runnable_load_avg : 0
[21899.496642] .blocked_load_avg : 0
[21899.496704] .tg_load_contrib : 0
[21899.496767] .tg_runnable_contrib : 1
[21899.496830] .tg_load_avg : 0
[21899.496894] .tg->runnable_avg : 44
[21899.496958] .tg->cfs_bandwidth.timer_active: 0
[21899.497021] .throttled : 0
[21899.497085] .throttle_count : 0
[21899.497148] .avg->runnable_avg_sum : 83
[21899.497213] .avg->runnable_avg_period : 47328
[21899.497279]
[21899.497279] rt_rq[0]:/
[21899.497393] .rt_nr_running : 0
[21899.497456] .rt_throttled : 0
[21899.497519] .rt_time : 0.000000
[21899.497583] .rt_runtime : 950.000000
[21899.497649]
[21899.497649] runnable tasks:
[21899.497649] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[21899.497649] ----------------------------------------------------------------------------------------------------------
[21899.498014] ksoftirqd/0 3 176322.164522 26211 120 176322.164522 694.113016 21897507.123650 /
[21899.498148] kworker/0:0H 5 809.251050 5 100 809.251050 0.049381 1049.540088 /
[21899.498281] rcu_bh 8 117.315906 2 120 117.315906 0.001357 0.000732 /
[21899.498414] migration/0 9 0.000000 55 0 0.000000 595.934405 0.000945 /
[21899.498548] watchdog/0 10 -11.786542 5487 0 -11.786542 78.881463 93.388016 /
[21899.498684] md 347 428.324204 2 100 428.324204 0.007888 0.003880 /
[21899.498816] kworker/0:1 590 176322.397665 255994 120 176322.397665 3646.626853 21894359.775811 /
[21899.498950] deferwq 718 445.387190 2 100 445.387190 0.030247 0.029450 /
[21899.499083] kworker/0:2 719 1575.582100 5 120 1575.582100 0.014370 33869.085959 /
[21899.499216] scsi_eh_0 847 808.228002 18 120 808.228002 0.812184 350.576852 /
[21899.499350] scsi_eh_2 851 807.502459 3 120 807.502459 0.063064 1.557453 /
[21899.499483] scsi_eh_3 853 807.507931 3 120 807.507931 0.041415 1.480934 /
[21899.499616] scsi_eh_4 855 807.500429 3 120 807.500429 0.062727 1.383249 /
[21899.499749] kworker/0:1H 870 176322.131352 3227418 100 176322.131352 73909.919684 21822939.455055 /
[21899.499883] md2_raid1 912 176319.708127 1904634 120 176319.708127 112347.908224 21779586.811747 /
[21899.500018] rsyslogd 2998 9667.377636 17 120 9667.377636 2.168560 5993625.189747 /autogroup-6
[21899.500154] rsyslogd 3023 9351.650497 25 120 9351.650497 2.853960 5649803.120979 /autogroup-6
[21899.500290] atd 3166 0.894976 8 120 0.894976 0.251211 21588604.307372 /autogroup-9
[21899.500425] cron 3167 19.716363 432 120 19.716363 28.779929 21821074.252869 /autogroup-10
[21899.500562] acpid 3200 0.761126 4 120 0.761126 0.331029 70.181766 /autogroup-11
[21899.500697] ntpd 3249 198.174713 22173 120 198.174713 733.960500 21860725.923946 /autogroup-12
[21899.500833] nginx 3381 228.838539 12011 120 228.838539 221.763641 21080722.835065 /autogroup-14
[21899.500968] mysqld 4016 17502.260268 1026 120 17502.260268 486.235187 16082465.636474 /autogroup-6
[21899.501104] mysqld 4058 19823.487946 4372 120 19823.487946 71.761763 21853520.339707 /autogroup-6
[21899.501239] mysqld 4059 932.238656 22 120 932.238656 1.114792 14053.522372 /autogroup-6
[21899.501374] mysqld 4088 19838.049940 155687 120 19838.049940 15582.597784 21839967.663937 /autogroup-6
[21899.501508] mysqld 4091 17501.988465 39947 120 17501.988465 5364.968383 16071902.529412 /autogroup-6
[21899.501643] mysqld 5066 17502.412717 106 120 17502.412717 8.815131 82120.381421 /autogroup-6
[21899.501778] getty 4312 0.426116 90 120 0.426116 2.248871 114.518681 /autogroup-23
[21899.501913] sshd 4322 432.817689 9748 120 432.817689 517.839144 21798014.813211 /autogroup-25
[21899.502054] btrfs-cleaner 4631 7919.209869 3 120 7919.209869 0.140736 0.153269 /
[21899.502191] kworker/u16:2 5052 176321.134148 2828 120 176321.134148 189.112029 6369384.863274 /
[21899.502324] pickup 5090 46.915293 67 120 46.915293 11.710485 3899371.672474 /autogroup-18
[21899.502459] kworker/u16:1 5097 176214.804945 4 120 176214.804945 0.155542 2854191.219499 /
[21899.502591]
[21899.502648] cpu#1, 3411.454 MHz
[21899.502708] .nr_running : 2
[21899.502772] .load : 2048
[21899.502835] .nr_switches : 2872731
[21899.502899] .nr_load_updates : 464658
[21899.502963] .nr_uninterruptible : 6742
[21899.503027] .next_balance : 4297.128397
[21899.503091] .curr->pid : 4324
[21899.503155] .clock : 21899485.003402
[21899.503220] .cpu_load[0] : 0
[21899.503283] .cpu_load[1] : 0
[21899.503345] .cpu_load[2] : 0
[21899.503409] .cpu_load[3] : 0
[21899.503472] .cpu_load[4] : 0
[21899.503536] .yld_count : 116
[21899.503599] .sched_count : 2877989
[21899.503663] .sched_goidle : 1329267
[21899.503727] .avg_idle : 1000000
[21899.503791] .max_idle_balance_cost : 500000
[21899.503855] .ttwu_count : 480332
[21899.503919] .ttwu_local : 431163
[21899.503983]
[21899.503983] cfs_rq[1]:/autogroup-26
[21899.504099] .exec_clock : 12.255692
[21899.504164] .MIN_vruntime : 0.000001
[21899.504229] .min_vruntime : 10.745539
[21899.504294] .max_vruntime : 0.000001
[21899.504358] .spread : 0.000000
[21899.504422] .spread0 : -176323.435259
[21899.504488] .nr_spread_over : 0
[21899.504550] .nr_running : 1
[21899.504614] .load : 1024
[21899.504677] .runnable_load_avg : 0
[21899.504741] .blocked_load_avg : 0
[21899.504804] .tg_load_contrib : 0
[21899.504868] .tg_runnable_contrib : 0
[21899.504932] .tg_load_avg : 0
[21899.504995] .tg->runnable_avg : 0
[21899.505059] .tg->cfs_bandwidth.timer_active: 0
[21899.505122] .throttled : 0
[21899.506697] .throttle_count : 0
[21899.506760] .se->exec_start : 21899485.003402
[21899.506825] .se->vruntime : 75576.305484
[21899.506890] .se->sum_exec_runtime : 12.299428
[21899.506955] .se->statistics.wait_start : 0.000000
[21899.507020] .se->statistics.sleep_start : 0.000000
[21899.507084] .se->statistics.block_start : 0.000000
[21899.507148] .se->statistics.sleep_max : 0.000000
[21899.507213] .se->statistics.block_max : 0.000000
[21899.507277] .se->statistics.exec_max : 2.415129
[21899.507341] .se->statistics.slice_max : 0.000000
[21899.507405] .se->statistics.wait_max : 0.011789
[21899.507469] .se->statistics.wait_sum : 0.059870
[21899.507532] .se->statistics.wait_count : 30
[21899.507596] .se->load.weight : 1024
[21899.507659] .se->avg.runnable_avg_sum : 0
[21899.507723] .se->avg.runnable_avg_period : 48352
[21899.507786] .se->avg.load_avg_contrib : 0
[21899.507849] .se->avg.decay_count : 0
[21899.507913]
[21899.507913] cfs_rq[1]:/autogroup-27
[21899.508029] .exec_clock : 1075.434446
[21899.508093] .MIN_vruntime : 0.000001
[21899.508157] .min_vruntime : 1074.385870
[21899.508222] .max_vruntime : 0.000001
[21899.508286] .spread : 0.000000
[21899.508351] .spread0 : -175259.794928
[21899.508416] .nr_spread_over : 0
[21899.508480] .nr_running : 0
[21899.508542] .load : 0
[21899.508606] .runnable_load_avg : 0
[21899.508668] .blocked_load_avg : 0
[21899.508730] .tg_load_contrib : 0
[21899.508793] .tg_runnable_contrib : 0
[21899.508856] .tg_load_avg : 0
[21899.508919] .tg->runnable_avg : 0
[21899.508983] .tg->cfs_bandwidth.timer_active: 0
[21899.509047] .throttled : 0
[21899.509110] .throttle_count : 0
[21899.509174] .se->exec_start : 21899387.327955
[21899.509239] .se->vruntime : 75582.028559
[21899.509304] .se->sum_exec_runtime : 1075.463253
[21899.509369] .se->statistics.wait_start : 0.000000
[21899.509433] .se->statistics.sleep_start : 0.000000
[21899.509503] .se->statistics.block_start : 0.000000
[21899.509563] .se->statistics.sleep_max : 0.000000
[21899.509633] .se->statistics.block_max : 0.000000
[21899.509698] .se->statistics.exec_max : 0.434543
[21899.509762] .se->statistics.slice_max : 0.104413
[21899.509826] .se->statistics.wait_max : 0.031711
[21899.509891] .se->statistics.wait_sum : 1.487240
[21899.509955] .se->statistics.wait_count : 13162
[21899.510019] .se->load.weight : 2
[21899.510082] .se->avg.runnable_avg_sum : 42
[21899.510145] .se->avg.runnable_avg_period : 46355
[21899.510208] .se->avg.load_avg_contrib : 0
[21899.510272] .se->avg.decay_count : 20884884
[21899.510336]
[21899.510336] cfs_rq[1]:/
[21899.510451] .exec_clock : 85092.143374
[21899.510515] .MIN_vruntime : 75577.012695
[21899.510580] .min_vruntime : 75582.028559
[21899.510645] .max_vruntime : 75577.012695
[21899.510709] .spread : 0.000000
[21899.510773] .spread0 : -100752.152239
[21899.510839] .nr_spread_over : 6
[21899.510902] .nr_running : 2
[21899.510965] .load : 2048
[21899.511028] .runnable_load_avg : 0
[21899.511090] .blocked_load_avg : 0
[21899.511153] .tg_load_contrib : 0
[21899.511216] .tg_runnable_contrib : 1
[21899.511280] .tg_load_avg : 0
[21899.511343] .tg->runnable_avg : 48
[21899.511407] .tg->cfs_bandwidth.timer_active: 0
[21899.511470] .throttled : 0
[21899.511534] .throttle_count : 0
[21899.511598] .avg->runnable_avg_sum : 68
[21899.511661] .avg->runnable_avg_period : 47099
[21899.511726]
[21899.511726] rt_rq[1]:/
[21899.511839] .rt_nr_running : 0
[21899.511902] .rt_throttled : 0
[21899.511965] .rt_time : 0.000000
[21899.512029] .rt_runtime : 950.000000
[21899.512094]
[21899.512094] runnable tasks:
[21899.512094] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[21899.512094] ----------------------------------------------------------------------------------------------------------
[21899.512460] rcu_sched 7 75581.549156 156612 120 75581.549156 9417.078958 21889713.120356 /
[21899.512593] watchdog/1 11 -2.971829 5481 0 -2.971829 73.004291 85.681641 /
[21899.512727] migration/1 12 0.000000 80 0 0.000000 11.452365 0.001061 /
[21899.512859] ksoftirqd/1 13 75529.508663 881 120 75529.508663 19.964746 21773631.112731 /
[21899.512992] kworker/1:0 14 0.796806 12 120 0.796806 0.024592 576.481603 /
[21899.513125] kworker/1:0H 15 433.498873 6 100 433.498873 0.050749 33667.062153 /
[21899.513260] kworker/1:1 592 75577.012695 82166 120 75577.012695 1323.635024 21897375.643395 /
[21899.513393] scsi_eh_1 849 304.651175 18 120 304.651175 1.423886 350.363177 /
[21899.513527] jbd2/md2-8 927 75569.532756 16914 120 75569.532756 804.662400 21892284.700017 /
[21899.513661] kworker/1:1H 1159 75543.537912 207451 100 75543.537912 3501.628429 21794041.515434 /
[21899.513794] rs:main Q:Reg 3021 2610.794736 248 120 2610.794736 47.436735 21581452.742463 /autogroup-6
[21899.513929] sshd 3328 -1.011224 4 120 -1.011224 0.606253 64176.198279 /autogroup-13
[21899.514064] nginx 3380 374.590033 31727 120 374.590033 553.045003 21860502.309871 /autogroup-14
[21899.514198] nginx 3382 374.753157 43707 120 374.753157 763.565837 21860162.615409 /autogroup-14
[21899.514333] mysqld_safe 3402 122.818333 438 120 122.818333 30.032898 474.108745 /autogroup-6
[21899.514469] mysqld 4033 2620.376893 43761 120 2620.376893 596.642937 21858734.950881 /autogroup-6
[21899.514605] mysqld 4060 659.291830 240 120 659.291830 19.399793 129483.960055 /autogroup-6
[21899.514740] getty 4308 2.510700 99 120 2.510700 4.489629 113.428154 /autogroup-19
[21899.514876] R bash 4324 8.339001 206 120 8.339001 57.946482 21796961.826617 /autogroup-26
[21899.515012] tmux 4372 1074.385870 66560 120 1074.385870 5591.476659 21787607.137065 /autogroup-27
[21899.515147] kworker/u17:0 4568 2308.940644 3 100 2308.940644 0.024287 0.015868 /
[21899.515282]
[21899.515339] cpu#2, 3411.454 MHz
[21899.515399] .nr_running : 0
[21899.515462] .load : 0
[21899.515525] .nr_switches : 2424905
[21899.515589] .nr_load_updates : 451974
[21899.515653] .nr_uninterruptible : 1796
[21899.515716] .next_balance : 4297.128381
[21899.515780] .curr->pid : 0
[21899.515843] .clock : 21899289.240972
[21899.515908] .cpu_load[0] : 0
[21899.515971] .cpu_load[1] : 0
[21899.516034] .cpu_load[2] : 0
[21899.516097] .cpu_load[3] : 0
[21899.516159] .cpu_load[4] : 0
[21899.516222] .yld_count : 0
[21899.516285] .sched_count : 2430075
[21899.516348] .sched_goidle : 1169020
[21899.516412] .avg_idle : 1000000
[21899.516476] .max_idle_balance_cost : 500000
[21899.516540] .ttwu_count : 343117
[21899.516603] .ttwu_local : 300240
[21899.516667]
[21899.516667] cfs_rq[2]:/autogroup-14
[21899.516783] .exec_clock : 403.708732
[21899.516847] .MIN_vruntime : 0.000001
[21899.516912] .min_vruntime : 287.377673
[21899.516976] .max_vruntime : 0.000001
[21899.517040] .spread : 0.000000
[21899.517103] .spread0 : -176046.803125
[21899.517169] .nr_spread_over : 0
[21899.517231] .nr_running : 0
[21899.517294] .load : 0
[21899.517357] .runnable_load_avg : 0
[21899.517420] .blocked_load_avg : 0
[21899.517483] .tg_load_contrib : 0
[21899.517546] .tg_runnable_contrib : 0
[21899.517609] .tg_load_avg : 0
[21899.517672] .tg->runnable_avg : 0
[21899.517735] .tg->cfs_bandwidth.timer_active: 0
[21899.517797] .throttled : 0
[21899.517860] .throttle_count : 0
[21899.517923] .se->exec_start : 21899289.234443
[21899.517988] .se->vruntime : 64875.180643
[21899.518053] .se->sum_exec_runtime : 403.873293
[21899.518117] .se->statistics.wait_start : 0.000000
[21899.518181] .se->statistics.sleep_start : 0.000000
[21899.519757] .se->statistics.block_start : 0.000000
[21899.519821] .se->statistics.sleep_max : 0.000000
[21899.519886] .se->statistics.block_max : 0.000000
[21899.519949] .se->statistics.exec_max : 0.123187
[21899.520013] .se->statistics.slice_max : 0.026523
[21899.520077] .se->statistics.wait_max : 0.095209
[21899.520141] .se->statistics.wait_sum : 4.402436
[21899.520205] .se->statistics.wait_count : 23460
[21899.520268] .se->load.weight : 2
[21899.520331] .se->avg.runnable_avg_sum : 13
[21899.520394] .se->avg.runnable_avg_period : 47792
[21899.520458] .se->avg.load_avg_contrib : 0
[21899.520521] .se->avg.decay_count : 20884790
[21899.520585]
[21899.520585] cfs_rq[2]:/autogroup-6
[21899.520701] .exec_clock : 4797.463344
[21899.520766] .MIN_vruntime : 0.000001
[21899.520830] .min_vruntime : 2990.173771
[21899.520895] .max_vruntime : 0.000001
[21899.520960] .spread : 0.000000
[21899.521024] .spread0 : -173344.007027
[21899.521089] .nr_spread_over : 90
[21899.521152] .nr_running : 0
[21899.521215] .load : 0
[21899.521278] .runnable_load_avg : 0
[21899.521341] .blocked_load_avg : 0
[21899.521405] .tg_load_contrib : 0
[21899.521468] .tg_runnable_contrib : 0
[21899.521531] .tg_load_avg : 0
[21899.521594] .tg->runnable_avg : 3
[21899.521657] .tg->cfs_bandwidth.timer_active: 0
[21899.521720] .throttled : 0
[21899.521782] .throttle_count : 0
[21899.521845] .se->exec_start : 21899203.385231
[21899.521911] .se->vruntime : 64887.070968
[21899.521975] .se->sum_exec_runtime : 4798.115180
[21899.522040] .se->statistics.wait_start : 0.000000
[21899.522104] .se->statistics.sleep_start : 0.000000
[21899.522169] .se->statistics.block_start : 0.000000
[21899.522233] .se->statistics.sleep_max : 0.000000
[21899.522298] .se->statistics.block_max : 0.000000
[21899.522362] .se->statistics.exec_max : 9.984601
[21899.522427] .se->statistics.slice_max : 0.358968
[21899.522491] .se->statistics.wait_max : 0.220493
[21899.522555] .se->statistics.wait_sum : 19.612568
[21899.522620] .se->statistics.wait_count : 101892
[21899.522683] .se->load.weight : 2
[21899.522746] .se->avg.runnable_avg_sum : 1
[21899.522810] .se->avg.runnable_avg_period : 47201
[21899.522873] .se->avg.load_avg_contrib : 0
[21899.522936] .se->avg.decay_count : 20884708
[21899.523000]
[21899.523000] cfs_rq[2]:/
[21899.523114] .exec_clock : 70951.067886
[21899.523179] .MIN_vruntime : 0.000001
[21899.523243] .min_vruntime : 64887.070968
[21899.523308] .max_vruntime : 0.000001
[21899.523371] .spread : 0.000000
[21899.523436] .spread0 : -111447.109830
[21899.523500] .nr_spread_over : 5
[21899.523562] .nr_running : 0
[21899.523625] .load : 0
[21899.523688] .runnable_load_avg : 0
[21899.523752] .blocked_load_avg : 0
[21899.523815] .tg_load_contrib : 0
[21899.523879] .tg_runnable_contrib : 41
[21899.523943] .tg_load_avg : 0
[21899.524006] .tg->runnable_avg : 47
[21899.524071] .tg->cfs_bandwidth.timer_active: 0
[21899.524134] .throttled : 0
[21899.524198] .throttle_count : 0
[21899.524267] .avg->runnable_avg_sum : 1928
[21899.524330] .avg->runnable_avg_period : 47230
[21899.524395]
[21899.524395] rt_rq[2]:/
[21899.524509] .rt_nr_running : 0
[21899.524572] .rt_throttled : 0
[21899.524635] .rt_time : 0.000000
[21899.524700] .rt_runtime : 950.000000
[21899.524765]
[21899.524765] runnable tasks:
[21899.524765] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[21899.524765] ----------------------------------------------------------------------------------------------------------
[21899.525132] init 1 23.595661 4610 120 23.595661 616.234479 21897940.187047 /autogroup-2
[21899.525266] kthreadd 2 64811.539667 139 120 64811.539667 2.923312 21749449.471237 /
[21899.525400] watchdog/2 16 -5.975085 5481 0 -5.975085 71.688752 72.366594 /
[21899.525534] migration/2 17 0.000000 63 0 0.000000 5.806137 0.000964 /
[21899.525666] ksoftirqd/2 18 64873.119229 778 120 64873.119229 17.952324 21883213.134744 /
[21899.525799] kworker/2:0 19 -5.020887 13 120 -5.020887 0.028730 568.777360 /
[21899.525932] kworker/2:0H 20 358.098517 6 100 358.098517 0.070725 2626.180931 /
[21899.526066] kworker/2:1 594 64875.443491 79034 120 64875.443491 1240.181080 21897254.704480 /
[21899.526200] kworker/2:1H 955 61875.368559 76761 100 61875.368559 1274.600575 11196616.413840 /
[21899.526334] rsyslogd 3022 2893.672895 232 120 2893.672895 14.042786 21581486.275518 /autogroup-6
[21899.526470] vnstatd 3001 228.013859 4432 120 228.013859 565.645704 21860411.835162 /autogroup-7
[21899.526605] mdadm 3137 0.044246 23 120 0.044246 5.028579 21642417.577079 /autogroup-8
[21899.526741] nginx 3383 287.377673 43723 120 287.377673 759.382769 21860350.601922 /autogroup-14
[21899.526877] mysqld 4038 2978.492075 43752 120 2978.492075 581.848348 21858719.178052 /autogroup-6
[21899.527013] dovecot 4167 20.656730 50 120 20.656730 4.753497 10113792.206034 /autogroup-16
[21899.527148] getty 4313 1.305357 96 120 1.305357 2.549224 114.426945 /autogroup-24
[21899.527285]
[21899.527342] cpu#3, 3411.454 MHz
[21899.527402] .nr_running : 0
[21899.527465] .load : 0
[21899.527528] .nr_switches : 1115803
[21899.527592] .nr_load_updates : 346081
[21899.527656] .nr_uninterruptible : -130
[21899.527720] .next_balance : 4297.128395
[21899.527785] .curr->pid : 0
[21899.527848] .clock : 21899524.908508
[21899.527913] .cpu_load[0] : 0
[21899.527977] .cpu_load[1] : 0
[21899.528039] .cpu_load[2] : 0
[21899.528103] .cpu_load[3] : 0
[21899.528165] .cpu_load[4] : 0
[21899.528228] .yld_count : 0
[21899.528291] .sched_count : 1120937
[21899.528355] .sched_goidle : 542155
[21899.528419] .avg_idle : 1000000
[21899.528483] .max_idle_balance_cost : 500000
[21899.528546] .ttwu_count : 357245
[21899.528610] .ttwu_local : 311737
[21899.528674]
[21899.528674] cfs_rq[3]:/autogroup-6
[21899.528790] .exec_clock : 4287.632283
[21899.528856] .MIN_vruntime : 0.000001
[21899.528920] .min_vruntime : 2528.524559
[21899.528984] .max_vruntime : 0.000001
[21899.529048] .spread : 0.000000
[21899.529112] .spread0 : -173805.656239
[21899.529177] .nr_spread_over : 166
[21899.529240] .nr_running : 0
[21899.529302] .load : 0
[21899.529366] .runnable_load_avg : 0
[21899.529429] .blocked_load_avg : 0
[21899.529493] .tg_load_contrib : 0
[21899.529556] .tg_runnable_contrib : 2
[21899.529620] .tg_load_avg : 0
[21899.529683] .tg->runnable_avg : 3
[21899.529746] .tg->cfs_bandwidth.timer_active: 0
[21899.529809] .throttled : 0
[21899.529872] .throttle_count : 0
[21899.529935] .se->exec_start : 21899524.902076
[21899.530000] .se->vruntime : 19863.836093
[21899.530065] .se->sum_exec_runtime : 4289.053828
[21899.530129] .se->statistics.wait_start : 0.000000
[21899.530193] .se->statistics.sleep_start : 0.000000
[21899.530257] .se->statistics.block_start : 0.000000
[21899.530321] .se->statistics.sleep_max : 0.000000
[21899.530386] .se->statistics.block_max : 0.000000
[21899.530450] .se->statistics.exec_max : 9.996177
[21899.530514] .se->statistics.slice_max : 0.211223
[21899.530578] .se->statistics.wait_max : 0.185806
[21899.530643] .se->statistics.wait_sum : 13.428816
[21899.530707] .se->statistics.wait_count : 114676
[21899.530770] .se->load.weight : 2
[21899.530833] .se->avg.runnable_avg_sum : 116
[21899.530896] .se->avg.runnable_avg_period : 46970
[21899.530960] .se->avg.load_avg_contrib : 0
[21899.531022] .se->avg.decay_count : 20885015
[21899.531086]
[21899.531086] cfs_rq[3]:/
[21899.531200] .exec_clock : 25043.808837
[21899.532776] .MIN_vruntime : 0.000001
[21899.532841] .min_vruntime : 19863.836093
[21899.532907] .max_vruntime : 0.000001
[21899.532970] .spread : 0.000000
[21899.533035] .spread0 : -156470.344705
[21899.533100] .nr_spread_over : 5
[21899.533162] .nr_running : 0
[21899.533225] .load : 0
[21899.533288] .runnable_load_avg : 0
[21899.533352] .blocked_load_avg : 0
[21899.533415] .tg_load_contrib : 0
[21899.533479] .tg_runnable_contrib : 2
[21899.533543] .tg_load_avg : 0
[21899.533606] .tg->runnable_avg : 46
[21899.533670] .tg->cfs_bandwidth.timer_active: 0
[21899.533733] .throttled : 0
[21899.533797] .throttle_count : 0
[21899.533860] .avg->runnable_avg_sum : 119
[21899.533924] .avg->runnable_avg_period : 47453
[21899.533989]
[21899.533989] rt_rq[3]:/
[21899.534102] .rt_nr_running : 0
[21899.534165] .rt_throttled : 0
[21899.534229] .rt_time : 0.000000
[21899.534293] .rt_runtime : 950.000000
[21899.534358]
[21899.534358] runnable tasks:
[21899.534358] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[21899.534358] ----------------------------------------------------------------------------------------------------------
[21899.534725] watchdog/3 21 -7.045804 3 120 -7.045804 0.003826 0.001433 /
[21899.534858] migration/3 22 0.000000 39 0 0.000000 5.577471 0.000861 /
[21899.534991] ksoftirqd/3 23 19828.580802 370 120 19828.580802 7.783530 21784352.586413 /
[21899.535123] kworker/3:0 24 -10.014423 13 120 -10.014423 0.035173 555.691063 /
[21899.535256] kworker/3:0H 25 415.709424 6 100 415.709424 0.045434 1623.334982 /
[21899.535391] khungtaskd 143 19834.190218 184 120 19834.190218 39.123599 21828417.861350 /
[21899.535524] khubd 341 1006.219177 62 120 1006.219177 3.533607 34705.301678 /
[21899.535657] kworker/3:1 596 19851.700559 94183 120 19851.700559 1526.794281 21897090.224599 /
[21899.535791] scsi_eh_5 857 394.018847 3 120 394.018847 0.041174 1.250220 /
[21899.535925] md1_raid1 904 19822.012091 63 120 19822.012091 2.146865 21678439.110013 /
[21899.536059] kworker/3:1H 913 17613.807216 29829 100 17613.807216 459.164739 8659751.435858 /
[21899.536193] mysqld 4032 2516.537997 43747 120 2516.537997 583.531066 21858584.163485 /autogroup-6
[21899.536327] mysqld 4035 2526.087749 43758 120 2526.087749 648.741466 21858675.375847 /autogroup-6
[21899.536462] mysqld 4037 2516.557278 43750 120 2516.557278 633.393731 21858689.775832 /autogroup-6
[21899.536598] mysqld 4040 2524.016723 43754 120 2524.016723 580.545502 21859041.975487 /autogroup-6
[21899.536733] mysqld 4055 2524.511288 22109 120 2524.511288 3284.930419 21854644.285624 /autogroup-6
[21899.536868] mysqld 4056 2528.524559 22408 120 2528.524559 6379.210179 21850610.374232 /autogroup-6
[21899.537003] mysqld 4057 2515.634414 4376 120 2515.634414 79.041349 21853393.637059 /autogroup-6
[21899.537139] master 4282 22.986731 1159 120 22.986731 69.926946 21844197.703090 /autogroup-18
[21899.537276] qmgr 4307 13.308744 161 120 13.308744 23.300543 21590194.795219 /autogroup-18
[21899.537411] getty 4310 1.190649 100 120 1.190649 2.444583 115.312522 /autogroup-21
[21899.537547] bash 4373 28.083046 109 120 28.083046 41.494043 12565.890293 /autogroup-28
[21899.537682] mount 4420 142.405537 6520 120 142.405537 505.231069 33673.941201 /autogroup-28
[21899.537819] bash 4585 19.900318 122 120 19.900318 51.388786 16042563.639159 /autogroup-29
[21899.537954] btrfs-transacti 4632 2871.297967 3 120 2871.297967 0.071315 29976.238403 /
[21899.538087]
[21899.538145] cpu#4, 3411.454 MHz
[21899.538204] .nr_running : 0
[21899.538267] .load : 0
[21899.538330] .nr_switches : 150437
[21899.538393] .nr_load_updates : 62170
[21899.538457] .nr_uninterruptible : 781
[21899.538521] .next_balance : 4297.128118
[21899.538585] .curr->pid : 0
[21899.538648] .clock : 21899240.642168
[21899.538713] .cpu_load[0] : 0
[21899.538776] .cpu_load[1] : 0
[21899.538839] .cpu_load[2] : 0
[21899.538903] .cpu_load[3] : 0
[21899.538966] .cpu_load[4] : 0
[21899.539029] .yld_count : 0
[21899.539093] .sched_count : 155900
[21899.539158] .sched_goidle : 78244
[21899.539222] .avg_idle : 1000000
[21899.539286] .max_idle_balance_cost : 500000
[21899.539350] .ttwu_count : 57417
[21899.539413] .ttwu_local : 54751
[21899.539478]
[21899.539478] cfs_rq[4]:/autogroup-6
[21899.539594] .exec_clock : 806.671076
[21899.539658] .MIN_vruntime : 0.000001
[21899.539723] .min_vruntime : 4372.239841
[21899.539788] .max_vruntime : 0.000001
[21899.539851] .spread : 0.000000
[21899.539915] .spread0 : -171961.940957
[21899.539980] .nr_spread_over : 121
[21899.540043] .nr_running : 0
[21899.540106] .load : 0
[21899.540169] .runnable_load_avg : 0
[21899.540232] .blocked_load_avg : 0
[21899.540295] .tg_load_contrib : 0
[21899.540358] .tg_runnable_contrib : 0
[21899.540421] .tg_load_avg : 0
[21899.540484] .tg->runnable_avg : 3
[21899.540547] .tg->cfs_bandwidth.timer_active: 0
[21899.540609] .throttled : 0
[21899.540672] .throttle_count : 0
[21899.540736] .se->exec_start : 21899240.635686
[21899.540801] .se->vruntime : 6179.601864
[21899.540865] .se->sum_exec_runtime : 806.970943
[21899.540930] .se->statistics.wait_start : 0.000000
[21899.540993] .se->statistics.sleep_start : 0.000000
[21899.541058] .se->statistics.block_start : 0.000000
[21899.541121] .se->statistics.sleep_max : 0.000000
[21899.541186] .se->statistics.block_max : 0.000000
[21899.541250] .se->statistics.exec_max : 9.988278
[21899.541314] .se->statistics.slice_max : 0.018898
[21899.541378] .se->statistics.wait_max : 0.032481
[21899.541441] .se->statistics.wait_sum : 0.699690
[21899.541505] .se->statistics.wait_count : 34293
[21899.541569] .se->load.weight : 2
[21899.541631] .se->avg.runnable_avg_sum : 13
[21899.541694] .se->avg.runnable_avg_period : 47814
[21899.541758] .se->avg.load_avg_contrib : 0
[21899.541821] .se->avg.decay_count : 20884744
[21899.541885]
[21899.541885] cfs_rq[4]:/
[21899.541999] .exec_clock : 2419.310191
[21899.542064] .MIN_vruntime : 0.000001
[21899.542127] .min_vruntime : 6179.601864
[21899.542192] .max_vruntime : 0.000001
[21899.542256] .spread : 0.000000
[21899.542319] .spread0 : -170154.578934
[21899.542385] .nr_spread_over : 240
[21899.542449] .nr_running : 0
[21899.542512] .load : 0
[21899.542574] .runnable_load_avg : 0
[21899.542637] .blocked_load_avg : 0
[21899.542700] .tg_load_contrib : 0
[21899.542763] .tg_runnable_contrib : 0
[21899.542826] .tg_load_avg : 0
[21899.542890] .tg->runnable_avg : 46
[21899.542953] .tg->cfs_bandwidth.timer_active: 0
[21899.543017] .throttled : 0
[21899.543080] .throttle_count : 0
[21899.543144] .avg->runnable_avg_sum : 13
[21899.543208] .avg->runnable_avg_period : 48174
[21899.543273]
[21899.543273] rt_rq[4]:/
[21899.543386] .rt_nr_running : 0
[21899.543448] .rt_throttled : 0
[21899.543512] .rt_time : 0.000000
[21899.543576] .rt_runtime : 950.000000
[21899.543641]
[21899.543641] runnable tasks:
[21899.543641] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[21899.543641] ----------------------------------------------------------------------------------------------------------
[21899.544007] watchdog/4 26 -8.970995 5481 0 -8.970995 71.917391 45.701610 /
[21899.544139] migration/4 27 0.000000 651 0 0.000000 8.685849 0.000871 /
[21899.544273] ksoftirqd/4 28 6166.457109 292 120 6166.457109 4.968426 21858574.037354 /
[21899.545913] kworker/4:0 29 2901.103537 13 120 2901.103537 0.040330 542.596863 /
[21899.546046] kworker/4:0H 30 3874.244415 6 100 3874.244415 0.031718 33941.440802 /
[21899.546180] writeback 144 18.960406 2 100 18.960406 0.004398 0.002997 /
[21899.546312] bioset 150 30.964087 2 100 30.964087 0.004239 0.003120 /
[21899.546445] kblockd 154 42.968506 2 100 42.968506 0.004960 0.000928 /
[21899.546578] kworker/4:1 598 6167.540594 10559 120 6167.540594 208.835109 21895693.692777 /
[21899.546711] ata_sff 827 3073.515378 2 100 3073.515378 0.010434 0.002782 /
[21899.546845] md0_raid1 896 6161.452569 62 120 6161.452569 1.495908 21678660.483939 /
[21899.546979] udevd 1116 200.356223 268 120 200.356223 42.017921 88950.271269 /autogroup-4
[21899.547114] kworker/4:1H 1291 5785.001120 2825 100 5785.001120 47.406818 5596191.423059 /
[21899.547248] mysqld 4041 4372.239841 43754 120 4372.239841 628.162144 21858708.112380 /autogroup-6
[21899.547384] anvil 4219 -8.530207 20 120 -8.530207 1.994961 10113412.385971 /autogroup-16
[21899.547520] getty 4311 0.269441 99 120 0.269441 1.810545 115.554102 /autogroup-22
[21899.547657]
[21899.547715] cpu#5, 3411.454 MHz
[21899.547775] .nr_running : 0
[21899.547838] .load : 0
[21899.547900] .nr_switches : 237157
[21899.547964] .nr_load_updates : 112164
[21899.548027] .nr_uninterruptible : 838
[21899.548090] .next_balance : 4297.128340
[21899.548155] .curr->pid : 0
[21899.548218] .clock : 21899240.646096
[21899.548283] .cpu_load[0] : 0
[21899.548345] .cpu_load[1] : 0
[21899.548408] .cpu_load[2] : 0
[21899.548472] .cpu_load[3] : 0
[21899.548535] .cpu_load[4] : 0
[21899.548599] .yld_count : 0
[21899.548663] .sched_count : 242504
[21899.548727] .sched_goidle : 122273
[21899.548791] .avg_idle : 1000000
[21899.548855] .max_idle_balance_cost : 500000
[21899.548919] .ttwu_count : 119587
[21899.548983] .ttwu_local : 105312
[21899.549047]
[21899.549047] cfs_rq[5]:/autogroup-6
[21899.549163] .exec_clock : 483.170584
[21899.549228] .MIN_vruntime : 0.000001
[21899.549292] .min_vruntime : 5266.924640
[21899.549357] .max_vruntime : 0.000001
[21899.549421] .spread : 0.000000
[21899.549485] .spread0 : -171067.256158
[21899.549550] .nr_spread_over : 200
[21899.549613] .nr_running : 0
[21899.549676] .load : 0
[21899.549738] .runnable_load_avg : 0
[21899.549802] .blocked_load_avg : 0
[21899.549864] .tg_load_contrib : 0
[21899.549928] .tg_runnable_contrib : 0
[21899.549990] .tg_load_avg : 0
[21899.550053] .tg->runnable_avg : 3
[21899.550116] .tg->cfs_bandwidth.timer_active: 0
[21899.550178] .throttled : 0
[21899.550242] .throttle_count : 0
[21899.550305] .se->exec_start : 21899240.637529
[21899.550370] .se->vruntime : 7803.382091
[21899.550435] .se->sum_exec_runtime : 483.339156
[21899.550499] .se->statistics.wait_start : 0.000000
[21899.550563] .se->statistics.sleep_start : 0.000000
[21899.550627] .se->statistics.block_start : 0.000000
[21899.550691] .se->statistics.sleep_max : 0.000000
[21899.550755] .se->statistics.block_max : 0.000000
[21899.550819] .se->statistics.exec_max : 2.051977
[21899.550883] .se->statistics.slice_max : 0.000000
[21899.550947] .se->statistics.wait_max : 0.050909
[21899.551011] .se->statistics.wait_sum : 0.408472
[21899.551075] .se->statistics.wait_count : 28927
[21899.551138] .se->load.weight : 2
[21899.551201] .se->avg.runnable_avg_sum : 15
[21899.551264] .se->avg.runnable_avg_period : 48679
[21899.551327] .se->avg.load_avg_contrib : 0
[21899.551391] .se->avg.decay_count : 20884744
[21899.551455]
[21899.551455] cfs_rq[5]:/
[21899.551569] .exec_clock : 4156.415344
[21899.551634] .MIN_vruntime : 0.000001
[21899.551698] .min_vruntime : 7803.382091
[21899.551763] .max_vruntime : 0.000001
[21899.551827] .spread : 0.000000
[21899.551892] .spread0 : -168530.798707
[21899.551957] .nr_spread_over : 299
[21899.552020] .nr_running : 0
[21899.552083] .load : 0
[21899.552147] .runnable_load_avg : 0
[21899.552210] .blocked_load_avg : 0
[21899.552273] .tg_load_contrib : 0
[21899.552336] .tg_runnable_contrib : 0
[21899.552398] .tg_load_avg : 0
[21899.552461] .tg->runnable_avg : 46
[21899.552524] .tg->cfs_bandwidth.timer_active: 0
[21899.552587] .throttled : 0
[21899.552651] .throttle_count : 0
[21899.552715] .avg->runnable_avg_sum : 15
[21899.552778] .avg->runnable_avg_period : 47863
[21899.552842]
[21899.552842] rt_rq[5]:/
[21899.552956] .rt_nr_running : 0
[21899.553018] .rt_throttled : 0
[21899.553081] .rt_time : 0.000000
[21899.553145] .rt_runtime : 950.000000
[21899.553210]
[21899.553210] runnable tasks:
[21899.553210] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[21899.553210] ----------------------------------------------------------------------------------------------------------
[21899.553578] watchdog/5 31 -8.976407 5481 0 -8.976407 72.323285 32.380679 /
[21899.553710] migration/5 32 0.000000 817 0 0.000000 9.402880 0.000883 /
[21899.553843] ksoftirqd/5 33 7791.396232 123 120 7791.396232 122.942133 21898428.213472 /
[21899.553976] kworker/5:0 34 2244.723326 12 120 2244.723326 0.041102 121.897143 /
[21899.554109] kworker/5:0H 35 4568.760446 6 100 4568.760446 0.099904 2646.190175 /
[21899.554244] kworker/5:1 449 7791.425432 52203 120 7791.425432 869.248715 21897558.028766 /
[21899.554382] kworker/5:1H 1006 6052.591935 1361 100 6052.591935 25.067913 4906525.698277 /
[21899.554519] mysqld 4036 5266.924640 43758 120 5266.924640 615.632628 21858722.860926 /autogroup-6
[21899.554656] udevd 4564 350.224193 2 120 350.224193 0.340662 0.146965 /autogroup-4
[21899.554791]
[21899.554849] cpu#6, 3411.454 MHz
[21899.554908] .nr_running : 0
[21899.554971] .load : 0
[21899.555034] .nr_switches : 155890
[21899.555098] .nr_load_updates : 73983
[21899.555162] .nr_uninterruptible : 734
[21899.555225] .next_balance : 4297.128218
[21899.555290] .curr->pid : 0
[21899.555353] .clock : 21899240.642586
[21899.555418] .cpu_load[0] : 0
[21899.555481] .cpu_load[1] : 0
[21899.555544] .cpu_load[2] : 0
[21899.555607] .cpu_load[3] : 0
[21899.555670] .cpu_load[4] : 0
[21899.555733] .yld_count : 1
[21899.555796] .sched_count : 161344
[21899.555859] .sched_goidle : 81959
[21899.555923] .avg_idle : 1000000
[21899.555987] .max_idle_balance_cost : 500000
[21899.556050] .ttwu_count : 69928
[21899.556114] .ttwu_local : 67797
[21899.556177]
[21899.556177] cfs_rq[6]:/autogroup-6
[21899.556293] .exec_clock : 875.872232
[21899.556358] .MIN_vruntime : 0.000001
[21899.556422] .min_vruntime : 5901.481253
[21899.556487] .max_vruntime : 0.000001
[21899.556551] .spread : 0.000000
[21899.556615] .spread0 : -170432.699545
[21899.556680] .nr_spread_over : 126
[21899.556742] .nr_running : 0
[21899.556805] .load : 0
[21899.556868] .runnable_load_avg : 0
[21899.556931] .blocked_load_avg : 0
[21899.556995] .tg_load_contrib : 0
[21899.557058] .tg_runnable_contrib : 0
[21899.557120] .tg_load_avg : 2
[21899.557183] .tg->runnable_avg : 2
[21899.557246] .tg->cfs_bandwidth.timer_active: 0
[21899.557308] .throttled : 0
[21899.558884] .throttle_count : 0
[21899.558947] .se->exec_start : 21899240.635841
[21899.559013] .se->vruntime : 4173.876078
[21899.559077] .se->sum_exec_runtime : 876.451216
[21899.559142] .se->statistics.wait_start : 0.000000
[21899.559206] .se->statistics.sleep_start : 0.000000
[21899.559269] .se->statistics.block_start : 0.000000
[21899.559334] .se->statistics.sleep_max : 0.000000
[21899.559398] .se->statistics.block_max : 0.000000
[21899.559462] .se->statistics.exec_max : 8.535287
[21899.559526] .se->statistics.slice_max : 0.000000
[21899.559590] .se->statistics.wait_max : 0.039212
[21899.559655] .se->statistics.wait_sum : 0.689800
[21899.559718] .se->statistics.wait_count : 53793
[21899.559782] .se->load.weight : 2
[21899.559844] .se->avg.runnable_avg_sum : 13
[21899.559907] .se->avg.runnable_avg_period : 48391
[21899.559970] .se->avg.load_avg_contrib : 0
[21899.560033] .se->avg.decay_count : 20884744
[21899.560098]
[21899.560098] cfs_rq[6]:/
[21899.560212] .exec_clock : 1830.575196
[21899.560276] .MIN_vruntime : 0.000001
[21899.560340] .min_vruntime : 4173.876078
[21899.560405] .max_vruntime : 0.000001
[21899.560469] .spread : 0.000000
[21899.560533] .spread0 : -172160.304720
[21899.560598] .nr_spread_over : 102
[21899.560662] .nr_running : 0
[21899.560724] .load : 0
[21899.560787] .runnable_load_avg : 0
[21899.560849] .blocked_load_avg : 0
[21899.560912] .tg_load_contrib : 0
[21899.560976] .tg_runnable_contrib : 0
[21899.561039] .tg_load_avg : 0
[21899.561103] .tg->runnable_avg : 47
[21899.561167] .tg->cfs_bandwidth.timer_active: 0
[21899.561230] .throttled : 0
[21899.561293] .throttle_count : 0
[21899.561357] .avg->runnable_avg_sum : 13
[21899.561421] .avg->runnable_avg_period : 48564
[21899.561485]
[21899.561485] rt_rq[6]:/
[21899.561599] .rt_nr_running : 0
[21899.561661] .rt_throttled : 0
[21899.561724] .rt_time : 0.000000
[21899.561789] .rt_runtime : 950.000000
[21899.561854]
[21899.561854] runnable tasks:
[21899.561854] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[21899.561854] ----------------------------------------------------------------------------------------------------------
[21899.562222] watchdog/6 36 -8.977275 5481 0 -8.977275 69.634621 19.060411 /
[21899.562354] migration/6 37 0.000000 663 0 0.000000 8.840079 0.000910 /
[21899.562487] ksoftirqd/6 38 4160.799441 79 120 4160.799441 1.483681 21858550.861505 /
[21899.562620] kworker/6:0 39 1147.724307 13 120 1147.724307 0.058340 516.327171 /
[21899.562753] kworker/6:0H 40 1883.005876 6 100 1883.005876 0.071140 33735.933716 /
[21899.562888] kworker/6:1 601 4161.845745 6878 120 4161.845745 149.237673 21896776.774763 /
[21899.563021] acpi_thermal_pm 606 1167.732140 2 100 1167.732140 0.010717 0.002847 /
[21899.563154] irq/44-mei_me 660 0.000000 14 49 0.000000 0.579329 0.000000 /
[21899.563287] kpsmoused 674 1191.763707 2 100 1191.763707 0.027833 0.007182 /
[21899.563421] scsi_tmf_0 848 1418.027634 2 100 1418.027634 0.033926 0.004042 /
[21899.563554] scsi_tmf_1 850 1442.097401 2 100 1442.097401 0.037671 0.003357 /
[21899.563687] scsi_tmf_2 852 1466.170799 2 100 1466.170799 0.039337 0.003358 /
[21899.563820] scsi_tmf_3 854 1490.185113 2 100 1490.185113 0.007055 0.003510 /
[21899.563953] scsi_tmf_4 856 1514.230079 2 100 1514.230079 0.009111 0.003775 /
[21899.564085] scsi_tmf_5 858 1538.272533 2 100 1538.272533 0.008466 0.003968 /
[21899.564218] bioset 895 1642.662840 2 100 1642.662840 0.025913 0.005468 /
[21899.564351] bioset 903 1674.969625 2 100 1674.969625 0.034134 0.026656 /
[21899.564484] bioset 911 1707.296862 2 100 1707.296862 0.007876 0.020316 /
[21899.564618] kworker/6:1H 1207 3658.246975 1492 100 3658.246975 23.691113 5564301.725731 /
[21899.564751] kjournald 2218 2276.164886 2 120 2276.164886 1.593018 0.027832 /
[21899.564884] ipv6_addrconf 3277 2409.399382 2 100 2409.399382 0.019128 0.017320 /
[21899.565017] mysqld 4034 5901.481253 43757 120 5901.481253 655.002338 21858683.280223 /autogroup-6
[21899.565154] mysqld 4085 3553.246418 2 120 3553.246418 0.145701 769.621025 /autogroup-6
[21899.565290] bioset 4565 2620.208328 2 100 2620.208328 0.023300 0.008634 /
[21899.565423] udevd 4566 331.552504 2 120 331.552504 0.210461 0.037267 /autogroup-4
[21899.565559] btrfs-delalloc 4570 2632.221380 2 100 2632.221380 0.014551 0.007010 /
[21899.565692] btrfs-flush_del 4571 2644.229124 2 100 2644.229124 0.008615 0.004697 /
[21899.565824] btrfs-cache 4572 2656.236301 2 100 2656.236301 0.008068 0.004642 /
[21899.565957] btrfs-submit 4573 2668.243488 2 100 2668.243488 0.007991 0.002886 /
[21899.566090] btrfs-fixup 4574 2680.250506 2 100 2680.250506 0.007965 0.002899 /
[21899.566223] btrfs-endio 4575 2692.258057 2 100 2692.258057 0.008370 0.002817 /
[21899.566356] btrfs-endio-met 4576 2704.269234 2 100 2704.269234 0.012289 0.022302 /
[21899.566490] btrfs-freespace 4581 2716.277121 2 100 2716.277121 0.008360 0.083971 /
[21899.566622] btrfs-delayed-m 4582 2728.280529 2 100 2728.280529 0.003811 0.001952 /
[21899.566755] btrfs-readahead 4583 2740.286195 2 100 2740.286195 0.006497 0.050178 /
[21899.566887] btrfs-qgroup-re 4584 2752.291591 2 100 2752.291591 0.006395 0.017750 /
[21899.567020]
[21899.567077] cpu#7, 3411.454 MHz
[21899.567137] .nr_running : 0
[21899.567201] .load : 0
[21899.567263] .nr_switches : 136354
[21899.567327] .nr_load_updates : 65971
[21899.567390] .nr_uninterruptible : 787
[21899.567453] .next_balance : 4297.128268
[21899.567518] .curr->pid : 0
[21899.567580] .clock : 21899240.648277
[21899.567645] .cpu_load[0] : 0
[21899.567708] .cpu_load[1] : 0
[21899.567771] .cpu_load[2] : 0
[21899.567834] .cpu_load[3] : 0
[21899.567897] .cpu_load[4] : 0
[21899.567961] .yld_count : 1
[21899.568024] .sched_count : 141827
[21899.568089] .sched_goidle : 72227
[21899.568153] .avg_idle : 1000000
[21899.568217] .max_idle_balance_cost : 500000
[21899.568281] .ttwu_count : 61617
[21899.568344] .ttwu_local : 59740
[21899.568408]
[21899.568408] cfs_rq[7]:/autogroup-6
[21899.568524] .exec_clock : 1040.550761
[21899.568589] .MIN_vruntime : 0.000001
[21899.568653] .min_vruntime : 4769.735382
[21899.568718] .max_vruntime : 0.000001
[21899.568782] .spread : 0.000000
[21899.568846] .spread0 : -171564.445416
[21899.568911] .nr_spread_over : 136
[21899.568974] .nr_running : 0
[21899.569038] .load : 0
[21899.569101] .runnable_load_avg : 0
[21899.569164] .blocked_load_avg : 0
[21899.569227] .tg_load_contrib : 0
[21899.569290] .tg_runnable_contrib : 0
[21899.569353] .tg_load_avg : 2
[21899.569416] .tg->runnable_avg : 2
[21899.569479] .tg->cfs_bandwidth.timer_active: 0
[21899.569541] .throttled : 0
[21899.569604] .throttle_count : 0
[21899.569668] .se->exec_start : 21899240.640183
[21899.569733] .se->vruntime : 3394.675397
[21899.569798] .se->sum_exec_runtime : 1040.608571
[21899.569862] .se->statistics.wait_start : 0.000000
[21899.569927] .se->statistics.sleep_start : 0.000000
[21899.569991] .se->statistics.block_start : 0.000000
[21899.570056] .se->statistics.sleep_max : 0.000000
[21899.570121] .se->statistics.block_max : 0.000000
[21899.570186] .se->statistics.exec_max : 1.179599
[21899.570250] .se->statistics.slice_max : 0.042319
[21899.570314] .se->statistics.wait_max : 0.029437
[21899.570378] .se->statistics.wait_sum : 2.588588
[21899.570442] .se->statistics.wait_count : 27648
[21899.572015] .se->load.weight : 2
[21899.572077] .se->avg.runnable_avg_sum : 16
[21899.572140] .se->avg.runnable_avg_period : 48416
[21899.572204] .se->avg.load_avg_contrib : 0
[21899.572268] .se->avg.decay_count : 20884744
[21899.572332]
[21899.572332] cfs_rq[7]:/
[21899.572447] .exec_clock : 2153.634461
[21899.572519] .MIN_vruntime : 0.000001
[21899.572579] .min_vruntime : 3401.849899
[21899.572640] .max_vruntime : 0.000001
[21899.572703] .spread : 0.000000
[21899.572779] .spread0 : -172932.330899
[21899.572844] .nr_spread_over : 18
[21899.572907] .nr_running : 0
[21899.572969] .load : 0
[21899.573032] .runnable_load_avg : 0
[21899.573094] .blocked_load_avg : 0
[21899.573157] .tg_load_contrib : 0
[21899.573220] .tg_runnable_contrib : 0
[21899.573283] .tg_load_avg : 0
[21899.573345] .tg->runnable_avg : 52
[21899.573408] .tg->cfs_bandwidth.timer_active: 0
[21899.573471] .throttled : 0
[21899.573534] .throttle_count : 0
[21899.573597] .avg->runnable_avg_sum : 16
[21899.573660] .avg->runnable_avg_period : 48390
[21899.573724]
[21899.573724] rt_rq[7]:/
[21899.573838] .rt_nr_running : 0
[21899.573901] .rt_throttled : 0
[21899.573963] .rt_time : 0.000000
[21899.574027] .rt_runtime : 950.000000
[21899.574092]
[21899.574092] runnable tasks:
[21899.574092] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[21899.574092] ----------------------------------------------------------------------------------------------------------
[21899.574460] watchdog/7 41 -8.977606 5481 0 -8.977606 60.288612 5.744928 /
[21899.574594] migration/7 42 0.000000 761 0 0.000000 9.528422 0.000899 /
[21899.574728] ksoftirqd/7 43 3318.329118 74 120 3318.329118 1.307871 19890767.930118 /
[21899.574861] kworker/7:0 44 215.370686 13 120 215.370686 0.028703 503.264907 /
[21899.574994] kworker/7:0H 45 1918.187361 6 100 1918.187361 0.077636 176634.318153 /
[21899.575128] khelper 46 10.956305 2 100 10.956305 0.005556 0.003187 /
[21899.575261] netns 47 22.959595 2 100 22.959595 0.003603 0.002525 /
[21899.575394] crypto 152 107.170217 2 100 107.170217 0.003659 0.001423 /
[21899.575527] kswapd0 495 768.598077 3 120 768.598077 0.032157 93.474814 /
[21899.575661] fsnotify_mark 563 1222.126134 13 120 1222.126134 0.242822 33472.286981 /
[21899.575794] kworker/7:1 603 3391.090645 7447 120 3391.090645 160.117776 21897266.283828 /
[21899.575929] ext4-rsv-conver 928 968.658291 2 100 968.658291 0.060681 0.034484 /
[21899.576064] nginx 3379 140.997826 2 120 140.997826 0.532104 5991983.688812 /autogroup-14
[21899.576200] mysqld 4039 4769.735382 43754 120 4769.735382 613.989370 21858706.669425 /autogroup-6
[21899.576335] log 4220 -1.827792 23 120 -1.827792 2.455380 10113412.065845 /autogroup-16
[21899.576470] getty 4309 0.016081 103 120 0.016081 1.715809 116.202681 /autogroup-20
[21899.576607] btrfs-worker 4567 1673.179661 2 100 1673.179661 0.032876 0.005095 /
[21899.576739] btrfs-worker-hi 4569 1697.204382 2 100 1697.204382 0.004792 0.002620 /
[21899.576874] btrfs-endio-met 4577 1709.307326 2 100 1709.307326 0.108350 0.025991 /
[21899.577007] btrfs-endio-rai 4578 1721.393543 2 100 1721.393543 0.087643 0.037782 /
[21899.577139] btrfs-rmw 4579 1733.495644 2 100 1733.495644 0.105179 0.033797 /
[21899.577273] btrfs-endio-wri 4580 1745.616606 2 100 1745.616606 0.122362 0.033939 /
[21899.577406] kworker/7:1H 4635 2882.934932 1204 100 2882.934932 22.740523 5499142.092498 /
[21899.577539] kworker/u16:0 5139 3401.849899 2 120 3401.849899 0.090069 0.052751 /
[21899.577672]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mount hangs after disk crash (RAID-1)
2014-05-22 2:22 mount hangs after disk crash (RAID-1) Tomasz Chmielewski
2014-05-22 8:51 ` Tomasz Chmielewski
@ 2014-05-22 9:43 ` Duncan
2014-05-22 16:27 ` Chris Murphy
1 sibling, 1 reply; 9+ messages in thread
From: Duncan @ 2014-05-22 9:43 UTC (permalink / raw)
To: linux-btrfs
Tomasz Chmielewski posted on Thu, 22 May 2014 03:22:58 +0100 as excerpted:
> One disk in RAID-1 crashed, so powered off, changed disk, powered on,
> trying to mount degraded.
>
> Unfortunately it hangs (running 3.14.4).
>
> # mount -o degraded,compress=lzo,noatime /dev/sdb4 /home (...never
> returns...)
1) Just to be sure, btrfs raid1, not btrfs on md/raid1 or the like,
correct?
2) Btrfs raid1 for both data and metadata, correct?
3) How many devices in the btrfs raid1?
4) Did you try mounting read-only?
Note that unlike md/raid1, btrfs raid1 won't mount writable with only a
single device. You must have at least two devices to mount writable, tho
a formerly two-device raid1 with a device missing should mount read-only.
However, ordinarily you can't add a device to a filesystem unless it's
mounted writable, and if it won't go writable as you have only the one
device, so you can't add another device... I'm actually not sure how
that's supposed to work, as the requiring two devices to mount writable
bit is fairly new.
In the worst case, however, you could create a new single-device
filesystem with the new device and mount it, mount the old one read-only
and copy everything over, then unmount the old one and add it to the new
one, then do a balance-convert to raid1 mode again. That's actually how
I've been planning to handle it if I lose a device from my paired-device
btrfs raid1 and it goes read-only, so I can't add the replacement
device. Seems there should be an easier way, but...
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mount hangs after disk crash (RAID-1)
2014-05-22 8:51 ` Tomasz Chmielewski
@ 2014-05-22 15:16 ` Tomasz Chmielewski
2014-05-22 17:50 ` Tomasz Chmielewski
0 siblings, 1 reply; 9+ messages in thread
From: Tomasz Chmielewski @ 2014-05-22 15:16 UTC (permalink / raw)
To: linux-btrfs
> > One disk in RAID-1 crashed, so powered off, changed disk, powered
> > on, trying to mount degraded.
> >
> > Unfortunately it hangs (running 3.14.4).
> >
> > # mount -o degraded,compress=lzo,noatime /dev/sdb4 /home (...never
> > returns...)
>
> 1) Just to be sure, btrfs raid1, not btrfs on md/raid1 or the like,
> correct?
Correct, it's btrfs raid1.
> 2) Btrfs raid1 for both data and metadata, correct?
raid1 for both data and metadata.
> 3) How many devices in the btrfs raid1?
There were 2 devices.
> 4) Did you try mounting read-only?
Tried also mounting with nospace_cache, recover, ro, but all hang in a
similar way.
> Note that unlike md/raid1, btrfs raid1 won't mount writable with only
> a single device. You must have at least two devices to mount
> writable, tho a formerly two-device raid1 with a device missing
> should mount read-only.
That's not entirely correct - it will mount with -o degraded option.
However it hangs here...
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mount hangs after disk crash (RAID-1)
2014-05-22 9:43 ` Duncan
@ 2014-05-22 16:27 ` Chris Murphy
0 siblings, 0 replies; 9+ messages in thread
From: Chris Murphy @ 2014-05-22 16:27 UTC (permalink / raw)
To: Btrfs BTRFS
On May 22, 2014, at 3:43 AM, Duncan <1i5t5.duncan@cox.net> wrote:
>
> Note that unlike md/raid1, btrfs raid1 won't mount writable with only a
> single device. You must have at least two devices to mount writable, tho
> a formerly two-device raid1 with a device missing should mount read-only.
No, a single device from a two device Btrfs raid1 is normally mountable degraded,rw. I did this just a week ago when bug "kernel BUG at fs/btrfs/locking.c when mounting with previously missing device". At one point when reproducing it, I was running degraded,rw with a single device for several hours without problems.
Try -o recovery,degraded
I would drop the other options for now, since they aren't necessary to recover from a device failure.
Chris Murphy
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mount hangs after disk crash (RAID-1)
2014-05-22 15:16 ` Tomasz Chmielewski
@ 2014-05-22 17:50 ` Tomasz Chmielewski
2014-05-22 18:20 ` Chris Murphy
` (2 more replies)
0 siblings, 3 replies; 9+ messages in thread
From: Tomasz Chmielewski @ 2014-05-22 17:50 UTC (permalink / raw)
To: linux-btrfs
> Try -o recovery,degraded
>
> I would drop the other options for now, since they aren't necessary
> to recover from a \ device failure.
Yes I've tried that as well, and it ends in the similar "hang" - high
IO for a while, then no IO at all, mount does not return.
It *does* mount as ro,degraded, but then, it's not possible to add a
disk and recover to a functioning RAID-1.
Also, when I try to remount rw, the mount command hangs as well.
Is there anything else I can try?
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mount hangs after disk crash (RAID-1)
2014-05-22 17:50 ` Tomasz Chmielewski
@ 2014-05-22 18:20 ` Chris Murphy
2014-05-22 21:06 ` Duncan
2014-05-22 22:07 ` Chris Murphy
2 siblings, 0 replies; 9+ messages in thread
From: Chris Murphy @ 2014-05-22 18:20 UTC (permalink / raw)
To: Btrfs BTRFS
On May 22, 2014, at 11:50 AM, Tomasz Chmielewski <tch@virtall.com> wrote:
>> Try -o recovery,degraded
>>
>> I would drop the other options for now, since they aren't necessary
>> to recover from a \ device failure.
>
> Yes I've tried that as well, and it ends in the similar "hang" - high
> IO for a while, then no IO at all, mount does not return.
>
>
> It *does* mount as ro,degraded, but then, it's not possible to add a
> disk and recover to a functioning RAID-1.
> Also, when I try to remount rw, the mount command hangs as well.
>
> Is there anything else I can try?
It sounds like it's able to read the file system but in the course of repairing whatever problems it has, it's getting stuck.
I'd mount it ro and make sure its backup is updated. Then take a btrfs image btrfs-image -c 9 -t <cores> in case a dev wants to look at the fs at some point it its current state.
Then you can try some other things if you want: btrfs-next is one direction, and also regression testing with older kernels is also reasonable.
Last you could try btrfs-zero-log, but this doesn't look like the typical case for it, and probably will make things worse so it's not the next thing to try. Also I don't expect btrfs check or even --repair to work. I've not yet had any luck with it working on degraded volumes, and even rejoined volumes (no longer degraded) but not yet manually balanced the btrfs check crashes. After a manual balance then the btrfs check is OK.
Chris Murphy
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mount hangs after disk crash (RAID-1)
2014-05-22 17:50 ` Tomasz Chmielewski
2014-05-22 18:20 ` Chris Murphy
@ 2014-05-22 21:06 ` Duncan
2014-05-22 22:07 ` Chris Murphy
2 siblings, 0 replies; 9+ messages in thread
From: Duncan @ 2014-05-22 21:06 UTC (permalink / raw)
To: linux-btrfs
Tomasz Chmielewski posted on Thu, 22 May 2014 18:50:25 +0100 as excerpted:
> It *does* mount as ro,degraded, but then, it's not possible to add a
> disk and recover to a functioning RAID-1.
> Also, when I try to remount rw, the mount command hangs as well.
>
> Is there anything else I can try?
It's a long shot, but in addition to Chris's excellent suggestions, try
the skip_balance mount option. In the outside case that it's just a
resumed balance that's triggering the hang, that should allow the rw
mount to succeed as it kills the balance that would otherwise be
immediately resumed as soon as the filesystem goes writable, possibly
before the mount returns.
Otherwise, I'd go with Chris's suggestion, mount read-only and ensure
your backups are current, then go experimental on it, since there's
nothing further to lose at that point anyway.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mount hangs after disk crash (RAID-1)
2014-05-22 17:50 ` Tomasz Chmielewski
2014-05-22 18:20 ` Chris Murphy
2014-05-22 21:06 ` Duncan
@ 2014-05-22 22:07 ` Chris Murphy
2 siblings, 0 replies; 9+ messages in thread
From: Chris Murphy @ 2014-05-22 22:07 UTC (permalink / raw)
To: Btrfs BTRFS
On May 22, 2014, at 11:50 AM, Tomasz Chmielewski <tch@virtall.com> wrote:
>> Try -o recovery,degraded
>>
>> I would drop the other options for now, since they aren't necessary
>> to recover from a \ device failure.
>
> Yes I've tried that as well, and it ends in the similar "hang" - high
> IO for a while, then no IO at all, mount does not return.
>
>
> It *does* mount as ro,degraded, but then, it's not possible to add a
> disk and recover to a functioning RAID-1.
> Also, when I try to remount rw, the mount command hangs as well.
>
> Is there anything else I can try?
How much metadata is there on the volume? And how much memory does the system have? This might also be a case where it might take hours, or longer, for it sort things out, and isn't actually permanently hung up. Worth a shot to leave it overnight and see what happens.
I agree trying skip_balance is worth a shot also.
Chris Murphy
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2014-05-22 22:08 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-05-22 2:22 mount hangs after disk crash (RAID-1) Tomasz Chmielewski
2014-05-22 8:51 ` Tomasz Chmielewski
2014-05-22 15:16 ` Tomasz Chmielewski
2014-05-22 17:50 ` Tomasz Chmielewski
2014-05-22 18:20 ` Chris Murphy
2014-05-22 21:06 ` Duncan
2014-05-22 22:07 ` Chris Murphy
2014-05-22 9:43 ` Duncan
2014-05-22 16:27 ` Chris Murphy
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).