* making snapshot kills RAID-1 rebuild?
@ 2014-03-28 3:08 Tomasz Chmielewski
2014-03-28 12:25 ` Tomasz Chmielewski
0 siblings, 1 reply; 2+ messages in thread
From: Tomasz Chmielewski @ 2014-03-28 3:08 UTC (permalink / raw)
To: linux-btrfs
One of drives died in my server and was replaced.
Some time after running "btrfs device delete missing ...", a read only
snapshot was made. Since then, "btrfs subvolume snapshot -r ..."
process still didn't return, and there are no more reads or writes to
any of drives in this RAID-1 filesystem.
Kernel is 3.14.0-rc8.
[22530.276924] SysRq : Show Blocked State
[22530.276987] task PC stack pid father
[22530.277074] btrfs-flush_del D ffff8807de54f000 0 5615 2 0x00000000
[22530.277141] ffff8807dec51d88 0000000000000046 00000000000116c0 ffff8807f1d8e000
[22530.277256] 00000000000116c0 0000000000004000 ffff8807f4031800 ffff8807f1d8e000
[22530.277371] ffff8807dec51cc8 ffff880786aed490 0000000000000001 ffff8807f1d8e000
[22530.277487] Call Trace:
[22530.277570] [<ffffffffa02df97d>] ? extent_writepages+0x52/0x57 [btrfs]
[22530.277648] [<ffffffffa02c8c56>] ? btrfs_fiemap+0x4c/0x4c [btrfs]
[22530.277721] [<ffffffffa02c7dbd>] ? btrfs_writepages+0x23/0x25 [btrfs]
[22530.277787] [<ffffffff810badc0>] ? do_writepages+0x19/0x27
[22530.277850] [<ffffffff8139708e>] schedule+0x6a/0x6c
[22530.277924] [<ffffffffa02da936>] btrfs_start_ordered_extent+0x77/0x8f [btrfs]
[22530.278037] [<ffffffff81067eec>] ? __wake_up_sync+0xd/0xd
[22530.278110] [<ffffffffa02da970>] btrfs_run_ordered_extent_work+0x22/0x32 [btrfs]
[22530.278236] [<ffffffffa02eb649>] worker_loop+0x15e/0x495 [btrfs]
[22530.278310] [<ffffffffa02eb4eb>] ? btrfs_queue_worker+0x269/0x269 [btrfs]
[22530.278374] [<ffffffff81050c92>] kthread+0xcd/0xd5
[22530.278436] [<ffffffff81050bc5>] ? kthread_freezable_should_stop+0x43/0x43
[22530.278499] [<ffffffff8139a03c>] ret_from_fork+0x7c/0xb0
[22530.278561] [<ffffffff81050bc5>] ? kthread_freezable_should_stop+0x43/0x43
[22530.278626] btrfs D ffff8807c4600000 0 7182 5619 0x00000000
[22530.278691] ffff8807c613d8b8 0000000000000082 ffff88081fa40000 ffff8807c4600000
[22530.278805] 00000000000116c0 0000000000004000 ffff8807f4030000 ffff8807c4600000
[22530.278920] ffff8807c613d888 00000000ffffffff ffff88009e886068 ffff88009e886068
[22530.279035] Call Trace:
[22530.279092] [<ffffffff8105ec0b>] ? __enqueue_entity+0x64/0x66
[22530.279155] [<ffffffff81061bc9>] ? enqueue_entity+0x611/0x73f
[22530.279218] [<ffffffff811af13b>] ? string.isra.4+0x3d/0xa4
[22530.279281] [<ffffffff810620b8>] ? enqueue_task_fair+0x3c1/0x42b
[22530.279344] [<ffffffff8139708e>] schedule+0x6a/0x6c
[22530.279404] [<ffffffff813961ec>] schedule_timeout+0x24/0x1b9
[22530.279466] [<ffffffff81058711>] ? ttwu_do_activate.constprop.75+0x57/0x5c
[22530.279530] [<ffffffff8105acf6>] ? try_to_wake_up+0x211/0x222
[22530.279592] [<ffffffff81397839>] wait_for_common+0x10d/0x13e
[22530.279653] [<ffffffff8105ad07>] ? try_to_wake_up+0x222/0x222
[22530.279715] [<ffffffff81397882>] wait_for_completion+0x18/0x1a
[22530.279789] [<ffffffffa02da472>] btrfs_wait_ordered_extents+0x1a2/0x215 [btrfs]
[22530.279912] [<ffffffffa02da5c1>] btrfs_wait_ordered_roots+0xdc/0x16c [btrfs]
[22530.279989] [<ffffffffa030662d>] btrfs_relocate_block_group+0x145/0x275 [btrfs]
[22530.280114] [<ffffffffa02e3e0a>] btrfs_relocate_chunk.isra.65+0x58/0x60e [btrfs]
[22530.280237] [<ffffffffa02f27c3>] ? btrfs_set_lock_blocking_rw+0x89/0xb2 [btrfs]
[22530.280356] [<ffffffffa02a5a2a>] ? btrfs_set_path_blocking+0x23/0x54 [btrfs]
[22530.280427] [<ffffffffa02aa477>] ? btrfs_search_slot+0x783/0x7dd [btrfs]
[22530.280502] [<ffffffffa02e0103>] ? free_extent_buffer+0x6f/0x7c [btrfs]
[22530.280576] [<ffffffffa02e60f9>] btrfs_shrink_device+0x23c/0x3a5 [btrfs]
[22530.280651] [<ffffffffa02e8673>] btrfs_rm_device+0x29e/0x6a7 [btrfs]
[22530.280724] [<ffffffffa02f05f9>] btrfs_ioctl+0xa48/0x25d3 [btrfs]
[22530.280789] [<ffffffff810ce1df>] ? handle_mm_fault+0x24f/0x965
[22530.280852] [<ffffffff8116e386>] ? avc_has_perm+0x2e/0xf2
[22530.280914] [<ffffffff811071a9>] do_vfs_ioctl+0x3f2/0x43c
[22530.280976] [<ffffffff81107241>] SyS_ioctl+0x4e/0x7d
[22530.281036] [<ffffffff8102f7fd>] ? do_page_fault+0xc/0xf
[22530.281098] [<ffffffff8139a0e2>] system_call_fastpath+0x16/0x1b
[22530.281158] rsync D 0000000000000002 0 7233 7217 0x00000000
[22530.281224] ffff8807bb69fb78 0000000000000086 ffff8807bb69fa88 ffff8807c4ce1800
[22530.281338] 00000000000116c0 0000000000004000 ffff8807f4031800 ffff8807c4ce1800
[22530.281453] ffff88081fdd9d00 0000000000000046 0000000000000000 ffff8807f2071efa
[22530.281568] Call Trace:
[22530.281624] [<ffffffff8105ad46>] ? wake_up_process+0x30/0x34
[22530.281695] [<ffffffffa02eb4df>] ? btrfs_queue_worker+0x25d/0x269 [btrfs]
[22530.281760] [<ffffffff810b1c94>] ? __lock_page+0x63/0x63
[22530.281822] [<ffffffff8139708e>] schedule+0x6a/0x6c
[22530.281882] [<ffffffff8139711a>] io_schedule+0x8a/0xc8
[22530.281944] [<ffffffff810b1c9d>] sleep_on_page+0x9/0xd
[22530.282005] [<ffffffff8139767b>] __wait_on_bit+0x45/0x7e
[22530.282066] [<ffffffff810b1a1d>] ? find_get_pages_tag+0xe9/0x123
[22530.282129] [<ffffffff810b1ea5>] wait_on_page_bit+0x71/0x73
[22530.282191] [<ffffffff81067f46>] ? wake_atomic_t_function+0x28/0x28
[22530.282254] [<ffffffff810b1f5f>] filemap_fdatawait_range+0x79/0x118
[22530.282328] [<ffffffffa02c8c56>] ? btrfs_fiemap+0x4c/0x4c [btrfs]
[22530.282400] [<ffffffffa02c7dbd>] ? btrfs_writepages+0x23/0x25 [btrfs]
[22530.282464] [<ffffffff810badc0>] ? do_writepages+0x19/0x27
[22530.282526] [<ffffffff810b27c3>] ? __filemap_fdatawrite_range+0x4e/0x50
[22530.282602] [<ffffffffa02dac84>] btrfs_wait_ordered_range+0x6e/0x120 [btrfs]
[22530.282667] [<ffffffff810e8c8e>] ? kmem_cache_free+0x178/0x18e
[22530.282740] [<ffffffffa02cdeac>] btrfs_truncate+0x43/0x231 [btrfs]
[22530.282812] [<ffffffffa02ce972>] btrfs_setattr+0x1bf/0x2ad [btrfs]
[22530.282875] [<ffffffff8110e239>] notify_change+0x1ed/0x2d6
[22530.282937] [<ffffffff810f6d5c>] do_truncate+0x62/0x87
[22530.282998] [<ffffffff810f7076>] do_sys_ftruncate.constprop.15+0x114/0x128
[22530.283061] [<ffffffff810f70ac>] SyS_ftruncate+0x9/0xb
[22530.283122] [<ffffffff8139a0e2>] system_call_fastpath+0x16/0x1b
[22530.283184] btrfs-flush_del D ffff88068a9b4180 0 30924 2 0x00000000
[22530.283248] ffff8806729e7d88 0000000000000046 000000010011a3e7 ffff88009e886000
[22530.283363] 00000000000116c0 0000000000004000 ffff8807f4036000 ffff88009e886000
[22530.283478] 0000000000000000 ffff880735d5eeb0 0000000000000001 ffff88009e886000
[22530.283593] Call Trace:
[22530.283660] [<ffffffffa02df97d>] ? extent_writepages+0x52/0x57 [btrfs]
[22530.283734] [<ffffffffa02c8c56>] ? btrfs_fiemap+0x4c/0x4c [btrfs]
[22530.283805] [<ffffffffa02c7dbd>] ? btrfs_writepages+0x23/0x25 [btrfs]
[22530.283868] [<ffffffff810badc0>] ? do_writepages+0x19/0x27
[22530.283930] [<ffffffff8139708e>] schedule+0x6a/0x6c
[22530.284001] [<ffffffffa02da936>] btrfs_start_ordered_extent+0x77/0x8f [btrfs]
[22530.284114] [<ffffffff81067eec>] ? __wake_up_sync+0xd/0xd
[22530.284185] [<ffffffffa02da970>] btrfs_run_ordered_extent_work+0x22/0x32 [btrfs]
[22530.284309] [<ffffffffa02eb649>] worker_loop+0x15e/0x495 [btrfs]
[22530.284381] [<ffffffffa02eb4eb>] ? btrfs_queue_worker+0x269/0x269 [btrfs]
[22530.284445] [<ffffffff81050c92>] kthread+0xcd/0xd5
[22530.284506] [<ffffffff81050bc5>] ? kthread_freezable_should_stop+0x43/0x43
[22530.284570] [<ffffffff8139a03c>] ret_from_fork+0x7c/0xb0
[22530.284631] [<ffffffff81050bc5>] ? kthread_freezable_should_stop+0x43/0x43
[22530.284695] btrfs D ffff8807f2034800 0 32297 30191 0x00000000
[22530.284759] ffff88009c3d99e8 0000000000000082 ffff88009c3d98e8 ffff8807f2034800
[22530.286266] 00000000000116c0 0000000000004000 ffff8807f4031800 ffff8807f2034800
[22530.286381] ffff88009c3d9948 ffffffff81056718 000000000000000a 0000000000000000
[22530.286495] Call Trace:
[22530.286552] [<ffffffff81056718>] ? task_rq_lock+0x48/0x7e
[22530.286614] [<ffffffff8105ec0b>] ? __enqueue_entity+0x64/0x66
[22530.286676] [<ffffffff81061bc9>] ? enqueue_entity+0x611/0x73f
[22530.286738] [<ffffffff811af13b>] ? string.isra.4+0x3d/0xa4
[22530.286800] [<ffffffff810620b8>] ? enqueue_task_fair+0x3c1/0x42b
[22530.286863] [<ffffffff8139708e>] schedule+0x6a/0x6c
[22530.286923] [<ffffffff813961ec>] schedule_timeout+0x24/0x1b9
[22530.286985] [<ffffffff81058603>] ? ttwu_do_wakeup+0x12/0x7f
[22530.287046] [<ffffffff81058711>] ? ttwu_do_activate.constprop.75+0x57/0x5c
[22530.287109] [<ffffffff8105acf6>] ? try_to_wake_up+0x211/0x222
[22530.287171] [<ffffffff81397839>] wait_for_common+0x10d/0x13e
[22530.287233] [<ffffffff8105ad07>] ? try_to_wake_up+0x222/0x222
[22530.287294] [<ffffffff81397882>] wait_for_completion+0x18/0x1a
[22530.287369] [<ffffffffa02d2853>] btrfs_wait_and_free_delalloc_work+0x13/0x26 [btrfs]
[22530.287492] [<ffffffffa02d2a08>] __start_delalloc_inodes+0x1a2/0x24f [btrfs]
[22530.287566] [<ffffffffa02d2ade>] btrfs_start_delalloc_inodes+0x29/0xf3 [btrfs]
[22530.287688] [<ffffffffa02ed84e>] btrfs_mksubvol.isra.63+0x119/0x372 [btrfs]
[22530.287762] [<ffffffffa02edbf6>] btrfs_ioctl_snap_create_transid+0x14f/0x180 [btrfs]
[22530.287875] [<ffffffff810eab00>] ? __kmalloc_track_caller+0x45/0x158
[22530.287946] [<ffffffffa02edd55>] btrfs_ioctl_snap_create_v2+0xc2/0x115 [btrfs]
[22530.288059] [<ffffffff810cabaf>] ? __do_fault+0x3ce/0x408
[22530.288130] [<ffffffffa02f025b>] btrfs_ioctl+0x6aa/0x25d3 [btrfs]
[22530.288194] [<ffffffff810ce1df>] ? handle_mm_fault+0x24f/0x965
[22530.288256] [<ffffffff8116e386>] ? avc_has_perm+0x2e/0xf2
[22530.288318] [<ffffffff811071a9>] do_vfs_ioctl+0x3f2/0x43c
[22530.288380] [<ffffffff81107241>] SyS_ioctl+0x4e/0x7d
[22530.288440] [<ffffffff8102f7fd>] ? do_page_fault+0xc/0xf
[22530.288501] [<ffffffff8139a0e2>] system_call_fastpath+0x16/0x1b
[22530.288564] Sched Debug Version: v0.11, 3.14.0-rc8 #1
[22530.288623] ktime : 22543566.176065
[22530.288684] sched_clk : 22530288.562574
[22530.288746] cpu_clk : 22530288.562607
[22530.288807] jiffies : 4297191652
[22530.288867] sched_clock_stable() : 1
[22530.288926]
[22530.288978] sysctl_sched
[22530.289033] .sysctl_sched_latency : 24.000000
[22530.289094] .sysctl_sched_min_granularity : 3.000000
[22530.289154] .sysctl_sched_wakeup_granularity : 4.000000
[22530.289215] .sysctl_sched_child_runs_first : 0
[22530.289274] .sysctl_sched_features : 11899
[22530.289335] .sysctl_sched_tunable_scaling : 1 (logaritmic)
[22530.289396]
[22530.289448] cpu#0, 3400.082 MHz
[22530.289503] .nr_running : 0
[22530.289561] .load : 0
[22530.289619] .nr_switches : 3507722
[22530.289678] .nr_load_updates : 805575
[22530.289736] .nr_uninterruptible : -11017
[22530.289795] .next_balance : 4297.191615
[22530.289854] .curr->pid : 0
[22530.289912] .clock : 22530276.914581
[22530.289972] .cpu_load[0] : 0
[22530.290030] .cpu_load[1] : 0
[22530.290087] .cpu_load[2] : 0
[22530.290145] .cpu_load[3] : 0
[22530.290204] .cpu_load[4] : 0
[22530.290261] .yld_count : 0
[22530.290319] .sched_count : 3973365
[22530.290378] .sched_goidle : 1572201
[22530.290436] .avg_idle : 921830
[22530.290495] .ttwu_count : 2358296
[22530.290554] .ttwu_local : 1598952
[22530.290613]
[22530.290613] cfs_rq[0]:/autogroup-157
[22530.290721] .exec_clock : 809.182104
[22530.290780] .MIN_vruntime : 0.000001
[22530.290840] .min_vruntime : 807.046150
[22530.290898] .max_vruntime : 0.000001
[22530.290957] .spread : 0.000000
[22530.291016] .spread0 : -248107.038102
[22530.291076] .nr_spread_over : 0
[22530.291134] .nr_running : 0
[22530.291191] .load : 0
[22530.291249] .runnable_load_avg : 0
[22530.291307] .blocked_load_avg : 0
[22530.291365] .tg_load_contrib : 0
[22530.291422] .tg_runnable_contrib : 0
[22530.291480] .tg_load_avg : 0
[22530.291538] .tg->runnable_avg : 0
[22530.291595] .tg->cfs_bandwidth.timer_active: 0
[22530.291654] .throttled : 0
[22530.291711] .throttle_count : 0
[22530.291770] .se->exec_start : 22530276.912362
[22530.291830] .se->vruntime : 248902.998339
[22530.291890] .se->sum_exec_runtime : 809.199506
[22530.291949] .se->statistics.wait_start : 0.000000
[22530.292008] .se->statistics.sleep_start : 0.000000
[22530.292067] .se->statistics.block_start : 0.000000
[22530.292126] .se->statistics.sleep_max : 0.000000
[22530.292185] .se->statistics.block_max : 0.000000
[22530.292244] .se->statistics.exec_max : 7.542286
[22530.292303] .se->statistics.slice_max : 0.134499
[22530.292362] .se->statistics.wait_max : 2.275916
[22530.292422] .se->statistics.wait_sum : 4.847488
[22530.292481] .se->statistics.wait_count : 8362
[22530.292539] .se->load.weight : 2
[22530.292597] .se->avg.runnable_avg_sum : 131
[22530.292655] .se->avg.runnable_avg_period : 47335
[22530.292713] .se->avg.load_avg_contrib : 0
[22530.292771] .se->avg.decay_count : 21486547
[22530.292830]
[22530.292830] cfs_rq[0]:/
[22530.292935] .exec_clock : 258014.906633
[22530.292995] .MIN_vruntime : 0.000001
[22530.293053] .min_vruntime : 248914.084252
[22530.293113] .max_vruntime : 0.000001
[22530.293171] .spread : 0.000000
[22530.293230] .spread0 : 0.000000
[22530.293289] .nr_spread_over : 655
[22530.293347] .nr_running : 0
[22530.293404] .load : 0
[22530.293462] .runnable_load_avg : 0
[22530.293520] .blocked_load_avg : 0
[22530.293577] .tg_load_contrib : 0
[22530.293635] .tg_runnable_contrib : 2
[22530.293693] .tg_load_avg : 0
[22530.293750] .tg->runnable_avg : 12
[22530.293808] .tg->cfs_bandwidth.timer_active: 0
[22530.293866] .throttled : 0
[22530.293924] .throttle_count : 0
[22530.293981] .avg->runnable_avg_sum : 137
[22530.294039] .avg->runnable_avg_period : 47365
[22530.294100]
[22530.294100] rt_rq[0]:/
[22530.294204] .rt_nr_running : 0
[22530.294262] .rt_throttled : 0
[22530.294320] .rt_time : 0.000000
[22530.294379] .rt_runtime : 950.000000
[22530.294439]
[22530.294439] runnable tasks:
[22530.294439] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[22530.294439] ----------------------------------------------------------------------------------------------------------
[22530.294775] ksoftirqd/0 3 248892.504894 20815 120 248892.504894 667.383969 22497309.102745 /
[22530.294898] kworker/0:0H 5 864.972294 5 100 864.972294 0.025760 1076.756375 /
[22530.295021] migration/0 9 0.000000 170 0 0.000000 456.738855 0.000863 /
[22530.295143] watchdog/0 10 -11.811076 5645 0 -11.811076 63.673987 95.067330 /
[22530.295269] kworker/0:2 700 23319.696806 18 120 23319.696806 0.066300 1208699.947715 /
[22530.295391] scsi_eh_2 878 861.043683 18 120 861.043683 2.567128 361.349574 /
[22530.295513] kworker/0:1H 900 248901.202086 339756 100 248901.202086 3451.660543 22521748.843741 /
[22530.295636] md2_raid1 953 248901.239713 23372 120 248901.239713 720.917596 22524405.501598 /
[22530.295758] jbd2/md2-8 967 248900.964661 14480 120 248900.964661 711.234331 22522749.431203 /
[22530.295881] btrfs-rmw-1 2281 248847.482316 228 120 248847.482316 4.048357 22426773.396194 /
[22530.296003] btrfs-freespace 2285 248857.343412 223 120 248857.343412 9.639232 22461638.651443 /
[22530.296126] btrfs-readahead 2288 248847.485369 215 120 248847.485369 3.141312 22426784.100990 /
[22530.296249] sshd 2754 0.380031 5 120 0.380031 0.996694 593475.387857 /autogroup-6
[22530.296374] rsync 3750 22.677844 12 139 22.677844 1.578214 291.920835 /autogroup-33
[22530.296505] getty 4177 0.464289 95 120 0.464289 1.899354 52.176395 /autogroup-65
[22530.296630] sshd 4874 807.046150 11493 120 807.046150 1084.400149 22062813.800188 /autogroup-157
[22530.296754] btrfs-endio-met 5609 248846.872432 229 120 248846.872432 3.033588 21827114.799170 /
[22530.296876] btrfs-cache-1 5613 248797.550230 7194 120 248797.550230 558.578685 21802633.979139 /
[22530.298387] sshd 5617 486.720744 4793 120 486.720744 531.200521 21887185.058173 /autogroup-246
[22530.298512] kworker/0:1 6394 248902.102587 368771 120 248902.102587 4552.184312 21613790.232133 /
[22530.298634] bash 7123 5.892870 115 120 5.892870 44.700616 992.878781 /autogroup-429
[22530.298759] snapbackup-run- 30189 0.270167 2 120 0.270167 0.874786 0.006102 /autogroup-3415
[22530.298883] snapbackup.sh 30191 1412.240353 41 120 1412.240353 5.733957 1039790.246669 /autogroup-3415
[22530.299008] btrfs-endio-met 31577 248899.815837 74281 120 248899.815837 1531.255384 10478537.720783 /
[22530.299130] btrfs-endio-4 32272 248844.719242 598 120 248844.719242 239.272278 10111112.622058 /
[22530.299251] pickup 12729 30.098547 76 120 30.098547 8.044460 4441773.013360 /autogroup-58
[22530.299375] bash 13443 22.321558 143 120 22.321558 48.000218 4327022.584033 /autogroup-5431
[22530.299500]
[22530.299552] cpu#1, 3400.082 MHz
[22530.299607] .nr_running : 1
[22530.299665] .load : 1024
[22530.299723] .nr_switches : 2074053
[22530.299782] .nr_load_updates : 520113
[22530.299841] .nr_uninterruptible : -3039
[22530.299900] .next_balance : 4297.191616
[22530.299960] .curr->pid : 4876
[22530.300018] .clock : 22530276.841766
[22530.300078] .cpu_load[0] : 0
[22530.300136] .cpu_load[1] : 0
[22530.300193] .cpu_load[2] : 0
[22530.300251] .cpu_load[3] : 0
[22530.300308] .cpu_load[4] : 0
[22530.300367] .yld_count : 0
[22530.300424] .sched_count : 2138399
[22530.300483] .sched_goidle : 974120
[22530.300542] .avg_idle : 1000000
[22530.300601] .ttwu_count : 931846
[22530.300659] .ttwu_local : 526811
[22530.300717]
[22530.300717] cfs_rq[1]:/autogroup-158
[22530.300824] .exec_clock : 667.073283
[22530.300883] .MIN_vruntime : 0.000001
[22530.300943] .min_vruntime : 632.314349
[22530.301002] .max_vruntime : 0.000001
[22530.301061] .spread : 0.000000
[22530.301120] .spread0 : -248281.769903
[22530.301179] .nr_spread_over : 75
[22530.301238] .nr_running : 1
[22530.301295] .load : 1024
[22530.301353] .runnable_load_avg : 7
[22530.301412] .blocked_load_avg : 0
[22530.301469] .tg_load_contrib : 0
[22530.301527] .tg_runnable_contrib : 0
[22530.301584] .tg_load_avg : 0
[22530.301642] .tg->runnable_avg : 0
[22530.301700] .tg->cfs_bandwidth.timer_active: 0
[22530.301757] .throttled : 0
[22530.301815] .throttle_count : 0
[22530.301873] .se->exec_start : 22530276.841766
[22530.301933] .se->vruntime : 142845.101369
[22530.301993] .se->sum_exec_runtime : 667.073283
[22530.302053] .se->statistics.wait_start : 0.000000
[22530.302112] .se->statistics.sleep_start : 0.000000
[22530.302171] .se->statistics.block_start : 0.000000
[22530.302231] .se->statistics.sleep_max : 0.000000
[22530.302290] .se->statistics.block_max : 0.000000
[22530.302349] .se->statistics.exec_max : 74.440664
[22530.302408] .se->statistics.slice_max : 0.028242
[22530.302466] .se->statistics.wait_max : 0.064807
[22530.302526] .se->statistics.wait_sum : 7.245129
[22530.302585] .se->statistics.wait_count : 5828
[22530.302643] .se->load.weight : 1024
[22530.302701] .se->avg.runnable_avg_sum : 382
[22530.302759] .se->avg.runnable_avg_period : 49064
[22530.302817] .se->avg.load_avg_contrib : 0
[22530.302875] .se->avg.decay_count : 0
[22530.302933]
[22530.302933] cfs_rq[1]:/autogroup-34
[22530.303039] .exec_clock : 227.081107
[22530.303099] .MIN_vruntime : 0.000001
[22530.303158] .min_vruntime : 226.032531
[22530.303217] .max_vruntime : 0.000001
[22530.303276] .spread : 0.000000
[22530.303334] .spread0 : -248688.051721
[22530.303394] .nr_spread_over : 0
[22530.303452] .nr_running : 0
[22530.303509] .load : 0
[22530.303568] .runnable_load_avg : 0
[22530.303625] .blocked_load_avg : 0
[22530.303683] .tg_load_contrib : 0
[22530.303740] .tg_runnable_contrib : 0
[22530.303798] .tg_load_avg : 0
[22530.303856] .tg->runnable_avg : 0
[22530.303913] .tg->cfs_bandwidth.timer_active: 0
[22530.303971] .throttled : 0
[22530.304029] .throttle_count : 0
[22530.304087] .se->exec_start : 22530121.083904
[22530.304147] .se->vruntime : 142840.707424
[22530.304206] .se->sum_exec_runtime : 227.090067
[22530.304266] .se->statistics.wait_start : 0.000000
[22530.304325] .se->statistics.sleep_start : 0.000000
[22530.304384] .se->statistics.block_start : 0.000000
[22530.304443] .se->statistics.sleep_max : 0.000000
[22530.304502] .se->statistics.block_max : 0.000000
[22530.304561] .se->statistics.exec_max : 0.226787
[22530.304620] .se->statistics.slice_max : 0.039417
[22530.304679] .se->statistics.wait_max : 0.010746
[22530.304738] .se->statistics.wait_sum : 0.063309
[22530.304797] .se->statistics.wait_count : 6988
[22530.304855] .se->load.weight : 2
[22530.304913] .se->avg.runnable_avg_sum : 26
[22530.304970] .se->avg.runnable_avg_period : 48661
[22530.305030] .se->avg.load_avg_contrib : 0
[22530.305087] .se->avg.decay_count : 21486398
[22530.305146]
[22530.305146] cfs_rq[1]:/
[22530.305250] .exec_clock : 150878.668869
[22530.305309] .MIN_vruntime : 0.000001
[22530.305368] .min_vruntime : 142852.680775
[22530.305428] .max_vruntime : 0.000001
[22530.305486] .spread : 0.000000
[22530.305546] .spread0 : -106061.403477
[22530.305605] .nr_spread_over : 547
[22530.305663] .nr_running : 1
[22530.305721] .load : 1024
[22530.305779] .runnable_load_avg : 0
[22530.305837] .blocked_load_avg : 0
[22530.305895] .tg_load_contrib : 0
[22530.305953] .tg_runnable_contrib : 8
[22530.306011] .tg_load_avg : 0
[22530.306068] .tg->runnable_avg : 12
[22530.306127] .tg->cfs_bandwidth.timer_active: 0
[22530.306184] .throttled : 0
[22530.306242] .throttle_count : 0
[22530.306300] .avg->runnable_avg_sum : 384
[22530.306358] .avg->runnable_avg_period : 47988
[22530.306417]
[22530.306417] rt_rq[1]:/
[22530.306521] .rt_nr_running : 0
[22530.306579] .rt_throttled : 0
[22530.306637] .rt_time : 0.000000
[22530.306696] .rt_runtime : 950.000000
[22530.306756]
[22530.306756] runnable tasks:
[22530.306756] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[22530.306756] ----------------------------------------------------------------------------------------------------------
[22530.307091] watchdog/1 11 -2.981856 5639 0 -2.981856 62.498815 87.222280 /
[22530.307213] migration/1 12 0.000000 121 0 0.000000 5.988127 0.000785 /
[22530.307335] ksoftirqd/1 13 142839.223283 27668 120 142839.223283 668.479767 22523927.005548 /
[22530.307456] kworker/1:0 14 -7.015031 13 120 -7.015031 0.034470 491.515728 /
[22530.307578] kworker/1:0H 15 345.641656 6 100 345.641656 0.018471 1079.161334 /
[22530.307701] khubd 361 1283.666783 62 120 1283.666783 3.049676 34042.847451 /
[22530.307823] khungtaskd 507 142808.352161 189 120 142808.352161 35.908898 22426745.793827 /
[22530.307946] kworker/1:1 614 142840.704806 366779 120 142840.704806 4491.144543 22522298.864669 /
[22530.308069] scsi_eh_5 884 331.894030 18 120 331.894030 2.630248 350.867278 /
[22530.308191] kworker/1:1H 913 142625.552716 42328 100 142625.552716 276.438462 22382513.092065 /
[22530.308315] btrfs-endio-wri 2284 142819.804790 237 120 142819.804790 4.290606 22426772.955294 /
[22530.308437] vnstatd 3028 229.218992 4640 120 229.218992 640.155000 22477878.279644 /autogroup-9
[22530.308561] rsyslogd 3124 567.970367 11069 120 567.970367 599.293460 22479710.563717 /autogroup-8
[22530.308685] munin-node 3170 644.834766 11484 120 644.834766 861.122874 22480166.953224 /autogroup-11
[22530.308810] apache2 3354 104.205511 22473 120 104.205511 452.575770 22480263.182606 /autogroup-24
[22530.308933] apache2 3397 15.008506 495 120 15.008506 8.273786 5744379.429239 /autogroup-24
[22530.310446] apache2 3398 21.269230 140 120 21.269230 2.618367 7109168.015486 /autogroup-24
[22530.310570] apache2 3399 19.891432 4 120 19.891432 0.387395 7109698.568698 /autogroup-24
[22530.310693] apache2 3400 19.890045 4 120 19.890045 0.416336 7110378.554222 /autogroup-24
[22530.310817] ntpd 3758 226.032531 23826 120 226.032531 827.040618 22479914.778204 /autogroup-34
[22530.310941] getty 4174 0.839315 102 120 0.839315 2.094155 53.704563 /autogroup-62
[22530.311065] R bash 4876 632.314349 1890 120 632.314349 392.261830 22063145.517377 /autogroup-158
[22530.311189] btrfs-fixup-1 5604 142819.200286 211 120 142819.200286 4.891033 21827123.714318 /
[22530.311311] btrfs-qgroup-re 5616 142819.228435 218 120 142819.228435 3.132755 21827124.482051 /
[22530.311433] apache2 6387 31.587596 4 120 31.587596 0.570381 6281498.461094 /autogroup-24
[22530.311557] btrfs 7182 47316.399566 921574 120 47316.399566 232067.852203 10498065.741354 /autogroup-247
[22530.311682] rsync 7217 3510.806718 62358 120 3510.806718 3073.485160 21219003.983830 /autogroup-429
[22530.311806] btrfs-delayed-m 7241 142817.056230 50436 120 142817.056230 2779.348392 21162744.974431 /
[22530.311928] btrfs-endio-wri 31606 142830.526736 181 120 142830.526736 12.730314 10333643.192282 /
[22530.312051] kworker/u16:2 12808 142839.826554 2449 120 142839.826554 273.465604 4404294.347679 /
[22530.312173]
[22530.312225] cpu#2, 3400.082 MHz
[22530.312281] .nr_running : 0
[22530.312338] .load : 0
[22530.312396] .nr_switches : 2166478
[22530.312455] .nr_load_updates : 529639
[22530.312513] .nr_uninterruptible : -6330
[22530.312572] .next_balance : 4297.191617
[22530.312631] .curr->pid : 0
[22530.312689] .clock : 22529912.634257
[22530.312748] .cpu_load[0] : 0
[22530.312807] .cpu_load[1] : 0
[22530.312864] .cpu_load[2] : 0
[22530.312922] .cpu_load[3] : 0
[22530.312980] .cpu_load[4] : 0
[22530.313037] .yld_count : 0
[22530.313095] .sched_count : 2254291
[22530.313154] .sched_goidle : 1006282
[22530.313213] .avg_idle : 1000000
[22530.313272] .ttwu_count : 1044976
[22530.313330] .ttwu_local : 609429
[22530.313389]
[22530.313389] cfs_rq[2]:/autogroup-428
[22530.313496] .exec_clock : 1730.310039
[22530.313556] .MIN_vruntime : 0.000001
[22530.313614] .min_vruntime : 1729.261463
[22530.313674] .max_vruntime : 0.000001
[22530.313733] .spread : 0.000000
[22530.313791] .spread0 : -247184.822789
[22530.313851] .nr_spread_over : 0
[22530.313909] .nr_running : 0
[22530.313966] .load : 0
[22530.314024] .runnable_load_avg : 0
[22530.314082] .blocked_load_avg : 0
[22530.314140] .tg_load_contrib : 0
[22530.314198] .tg_runnable_contrib : 1
[22530.314256] .tg_load_avg : 0
[22530.314313] .tg->runnable_avg : 1
[22530.314371] .tg->cfs_bandwidth.timer_active: 0
[22530.314429] .throttled : 0
[22530.314486] .throttle_count : 0
[22530.314545] .se->exec_start : 22529878.504574
[22530.314605] .se->vruntime : 120575.111665
[22530.314664] .se->sum_exec_runtime : 1730.321837
[22530.314725] .se->statistics.wait_start : 0.000000
[22530.314783] .se->statistics.sleep_start : 0.000000
[22530.314843] .se->statistics.block_start : 0.000000
[22530.314902] .se->statistics.sleep_max : 0.000000
[22530.314960] .se->statistics.block_max : 0.000000
[22530.315020] .se->statistics.exec_max : 7.676439
[22530.315078] .se->statistics.slice_max : 0.120501
[22530.315137] .se->statistics.wait_max : 0.039686
[22530.315196] .se->statistics.wait_sum : 2.301828
[22530.315255] .se->statistics.wait_count : 25524
[22530.315314] .se->load.weight : 2
[22530.315371] .se->avg.runnable_avg_sum : 49
[22530.315429] .se->avg.runnable_avg_period : 46613
[22530.315487] .se->avg.load_avg_contrib : 0
[22530.315545] .se->avg.decay_count : 21486167
[22530.315605]
[22530.315605] cfs_rq[2]:/
[22530.315710] .exec_clock : 125325.860514
[22530.315769] .MIN_vruntime : 0.000001
[22530.315828] .min_vruntime : 120575.111665
[22530.315888] .max_vruntime : 0.000001
[22530.315947] .spread : 0.000000
[22530.316006] .spread0 : -128338.972587
[22530.316065] .nr_spread_over : 555
[22530.316123] .nr_running : 0
[22530.316181] .load : 0
[22530.316238] .runnable_load_avg : 0
[22530.316297] .blocked_load_avg : 0
[22530.316354] .tg_load_contrib : 0
[22530.316412] .tg_runnable_contrib : 2
[22530.316470] .tg_load_avg : 0
[22530.316527] .tg->runnable_avg : 11
[22530.316586] .tg->cfs_bandwidth.timer_active: 0
[22530.316643] .throttled : 0
[22530.316701] .throttle_count : 0
[22530.316759] .avg->runnable_avg_sum : 106
[22530.316817] .avg->runnable_avg_period : 46717
[22530.316876]
[22530.316876] rt_rq[2]:/
[22530.316980] .rt_nr_running : 0
[22530.317039] .rt_throttled : 0
[22530.317096] .rt_time : 0.000000
[22530.317155] .rt_runtime : 950.000000
[22530.317214]
[22530.317214] runnable tasks:
[22530.317214] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[22530.317214] ----------------------------------------------------------------------------------------------------------
[22530.317549] init 1 34.237478 5429 120 34.237478 536.506774 22528629.359900 /autogroup-2
[22530.317673] rcu_bh 8 3988.124030 5 120 3988.124030 0.068102 309972.486503 /
[22530.317796] watchdog/2 16 -5.977800 5639 0 -5.977800 59.835958 73.610920 /
[22530.317919] migration/2 17 0.000000 101 0 0.000000 5.849580 0.001040 /
[22530.318041] ksoftirqd/2 18 120549.467864 25959 120 120549.467864 480.198915 22497498.713614 /
[22530.318163] kworker/2:0 19 5305.173951 16 120 5305.173951 0.050614 1105947.617632 /
[22530.318284] kworker/2:0H 20 337.907610 6 100 337.907610 0.046118 1086.075641 /
[22530.318408] fsnotify_mark 585 4667.309172 26 120 4667.309172 0.688965 908665.309575 /
[22530.318532] kworker/2:1H 919 120074.406304 72690 100 120074.406304 369.889288 22378951.531283 /
[22530.318654] md1_raid1 945 119752.731680 102 120 119752.731680 2.490640 22036918.363093 /
[22530.318776] btrfs-genwork-1 2275 120551.049828 203 120 120551.049828 6.264482 22466969.166099 /
[22530.318898] btrfs-fixup-1 2278 120536.016930 216 120 120536.016930 2.858137 22426805.064861 /
[22530.319020] btrfs-endio-1 2279 120536.035313 221 120 120536.035313 6.143953 22426761.562380 /
[22530.319142] btrfs-endio-rai 2282 120536.036506 232 120 120536.036506 3.889816 22426772.944779 /
[22530.319264] btrfs-endio-met 2283 120536.009812 218 120 120536.009812 5.803848 22426761.911583 /
[22530.319386] btrfs-delayed-m 2286 120536.017145 202 120 120536.017145 3.258624 22426794.598019 /
[22530.319508] btrfs-transacti 2322 120557.306471 27242 120 120557.306471 2611.705508 22466979.009431 /
[22530.319630] rs:main Q:Reg 3123 526.436176 11582 120 526.436176 1376.828510 22478929.226354 /autogroup-8
[22530.319754] rsyslogd 3125 441.204205 52 120 441.204205 4.037016 17973802.534553 /autogroup-8
[22530.319878] apache2 3396 69.461288 1622 120 69.461288 24.001873 22476889.066979 /autogroup-24
[22530.320003] proftpd 4129 37.816560 1661 120 37.816560 103.209346 22455891.824322 /autogroup-60
[22530.320127] getty 4173 2.702581 100 120 2.702581 4.247579 51.857295 /autogroup-61
[22530.320251] btrfs-submit-1 5602 120545.766203 14061 120 120545.766203 303.818058 21854275.486264 /
[22530.320374] btrfs-endio-rai 5608 120535.321126 227 120 120535.321126 3.615731 21827114.615043 /
[22530.320496] btrfs-freespace 5611 120534.481049 8846 120 120534.481049 507.894424 21824057.505415 /
[22530.320618] btrfs-readahead 5614 120535.340917 212 120 120535.340917 4.378350 21827123.829935 /
[22530.320740] btrfs-flush_del 5615 69196.236094 112 120 69196.236094 3.632515 11219175.607685 /
[22530.320861] bash 5619 497.310559 196 120 497.310559 61.584625 480244.525172 /autogroup-247
[22530.320986] kworker/2:2 6368 120563.162736 370073 120 120563.162736 4337.359140 21719278.468694 /
[22530.322493] tmux 7122 1729.261463 76946 120 1729.261463 5588.366093 21417091.462972 /autogroup-428
[22530.322618] ssh 7218 3902.067624 39107 120 3902.067624 5071.252249 1409659.629578 /autogroup-429
[22530.322742] rsync 7233 3906.907543 134807 120 3906.907543 12511.401135 1393899.260464 /autogroup-429
[22530.322866] btrfs-worker-2 9957 120545.791230 3529 120 120545.791230 171.649767 20183115.892225 /
[22530.322988] sh 30188 0.685951 2 120 0.685951 0.901134 0.006914 /autogroup-3415
[22530.323113] btrfs 32297 1967.734855 3 120 1967.734855 0.723562 0.069948 /autogroup-3415
[22530.323238]
[22530.323290] cpu#3, 3400.082 MHz
[22530.323345] .nr_running : 0
[22530.323402] .load : 0
[22530.323460] .nr_switches : 2198261
[22530.323518] .nr_load_updates : 520761
[22530.323577] .nr_uninterruptible : -6008
[22530.323636] .next_balance : 4297.191655
[22530.323695] .curr->pid : 0
[22530.323753] .clock : 22530292.421592
[22530.323824] .cpu_load[0] : 0
[22530.323882] .cpu_load[1] : 0
[22530.323940] .cpu_load[2] : 0
[22530.323997] .cpu_load[3] : 0
[22530.324055] .cpu_load[4] : 0
[22530.324113] .yld_count : 0
[22530.324171] .sched_count : 2243914
[22530.324229] .sched_goidle : 1033185
[22530.324288] .avg_idle : 1000000
[22530.324347] .ttwu_count : 933876
[22530.324405] .ttwu_local : 524476
[22530.324464]
[22530.324464] cfs_rq[3]:/
[22530.324568] .exec_clock : 143341.067012
[22530.324628] .MIN_vruntime : 0.000001
[22530.324687] .min_vruntime : 136633.722718
[22530.324747] .max_vruntime : 0.000001
[22530.324805] .spread : 0.000000
[22530.324865] .spread0 : -112280.361534
[22530.324924] .nr_spread_over : 612
[22530.324982] .nr_running : 0
[22530.325040] .load : 0
[22530.325097] .runnable_load_avg : 0
[22530.325155] .blocked_load_avg : 0
[22530.325213] .tg_load_contrib : 0
[22530.325271] .tg_runnable_contrib : 0
[22530.325329] .tg_load_avg : 0
[22530.325386] .tg->runnable_avg : 10
[22530.325444] .tg->cfs_bandwidth.timer_active: 0
[22530.325502] .throttled : 0
[22530.325559] .throttle_count : 0
[22530.325617] .avg->runnable_avg_sum : 15
[22530.325674] .avg->runnable_avg_period : 48630
[22530.325733]
[22530.325733] rt_rq[3]:/
[22530.325837] .rt_nr_running : 0
[22530.325895] .rt_throttled : 0
[22530.325953] .rt_time : 0.000000
[22530.326012] .rt_runtime : 950.000000
[22530.326072]
[22530.326072] runnable tasks:
[22530.326072] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[22530.326072] ----------------------------------------------------------------------------------------------------------
[22530.326408] rcu_sched 7 136633.722718 558259 120 136633.722718 23574.833814 22505282.071390 /
[22530.326531] watchdog/3 21 -5.983471 5639 0 -5.983471 58.162928 60.002758 /
[22530.326653] migration/3 22 0.000000 111 0 0.000000 5.940465 0.000776 /
[22530.326774] ksoftirqd/3 23 136620.302472 28018 120 136620.302472 599.994556 22523974.798399 /
[22530.326895] kworker/3:0 24 3861.435559 17 120 3861.435559 0.048979 335090.635899 /
[22530.327017] kworker/3:0H 25 353.943497 6 100 353.943497 0.018093 1042.114442 /
[22530.327140] kpsmoused 697 11.104349 2 100 11.104349 0.006361 0.011634 /
[22530.327263] kworker/3:1H 905 136590.411032 31449 100 136590.411032 224.880863 22451347.174667 /
[22530.327386] md0_raid1 937 131267.895657 94 120 131267.895657 2.197444 22037123.463863 /
[22530.327508] kworker/3:2 1539 136621.734608 384567 120 136621.734608 4665.780706 22488882.816093 /
[22530.327629] btrfs-worker-1 2274 136601.460396 210 120 136601.460396 3.313660 22426805.267226 /
[22530.327751] btrfs-submit-1 2276 136601.413588 216 120 136601.413588 2.942803 22426785.493436 /
[22530.327873] btrfs-endio-met 2280 136618.934809 32323 120 136618.934809 924.387644 22483022.539249 /
[22530.327996] btrfs-qgroup-re 2290 136601.391933 213 120 136601.391933 3.907423 22426793.545283 /
[22530.328118] acpid 3152 0.951311 4 120 0.951311 0.391560 13.197935 /autogroup-10
[22530.328243] master 4110 13.291707 1121 120 13.291707 62.893352 22443719.990334 /autogroup-58
[22530.328367] getty 4176 0.828660 101 120 0.828660 2.103876 53.283951 /autogroup-64
[22530.328491] btrfs-genwork-1 5601 136601.604300 317 120 136601.604300 8.769193 21829539.500055 /
[22530.328613] btrfs-delalloc- 5603 136600.775528 227 120 136600.775528 3.582197 21827115.345843 /
[22530.328735] btrfs-rmw-1 5607 136600.774237 211 120 136600.774237 5.557399 21827133.450132 /
[22530.328857] btrfs-transacti 5688 136619.720388 409241 120 136619.720388 44090.424547 21806613.820149 /
[22530.328980] snapbackup.sh 7168 160.007081 28 120 160.007081 5.274723 155447.886152 /autogroup-429
[22530.329105] kworker/u16:1 29506 115523.145051 3172 120 115523.145051 415.106587 7127902.742599 /
[22530.329228]
[22530.329280] cpu#4, 3400.082 MHz
[22530.329336] .nr_running : 0
[22530.329392] .load : 0
[22530.329450] .nr_switches : 525526
[22530.329509] .nr_load_updates : 151911
[22530.329567] .nr_uninterruptible : 6760
[22530.329626] .next_balance : 4297.191324
[22530.329685] .curr->pid : 0
[22530.329743] .clock : 22530292.431696
[22530.329802] .cpu_load[0] : 0
[22530.329860] .cpu_load[1] : 0
[22530.329918] .cpu_load[2] : 0
[22530.329975] .cpu_load[3] : 0
[22530.330033] .cpu_load[4] : 0
[22530.330091] .yld_count : 0
[22530.330149] .sched_count : 547366
[22530.330208] .sched_goidle : 242892
[22530.330266] .avg_idle : 790364
[22530.330325] .ttwu_count : 272667
[22530.330383] .ttwu_local : 187575
[22530.330442]
[22530.330442] cfs_rq[4]:/
[22530.330546] .exec_clock : 85501.442258
[22530.330606] .MIN_vruntime : 0.000001
[22530.330665] .min_vruntime : 106496.936449
[22530.330724] .max_vruntime : 0.000001
[22530.330784] .spread : 0.000000
[22530.330842] .spread0 : -142417.147803
[22530.330902] .nr_spread_over : 542
[22530.330960] .nr_running : 0
[22530.331018] .load : 0
[22530.331075] .runnable_load_avg : 0
[22530.331133] .blocked_load_avg : 0
[22530.331191] .tg_load_contrib : 0
[22530.331248] .tg_runnable_contrib : 0
[22530.331306] .tg_load_avg : 0
[22530.331364] .tg->runnable_avg : 10
[22530.331421] .tg->cfs_bandwidth.timer_active: 0
[22530.331479] .throttled : 0
[22530.331537] .throttle_count : 0
[22530.331595] .avg->runnable_avg_sum : 0
[22530.331653] .avg->runnable_avg_period : 48073
[22530.331711]
[22530.331711] rt_rq[4]:/
[22530.331816] .rt_nr_running : 0
[22530.331874] .rt_throttled : 0
[22530.331931] .rt_time : 0.000000
[22530.331990] .rt_runtime : 950.000000
[22530.332050]
[22530.332050] runnable tasks:
[22530.332050] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[22530.332050] ----------------------------------------------------------------------------------------------------------
[22530.332387] watchdog/4 26 -8.982866 5639 0 -8.982866 64.971744 46.504233 /
[22530.332520] migration/4 27 0.000000 6621 0 0.000000 52.733681 0.000795 /
[22530.332642] ksoftirqd/4 28 106484.923125 5127 120 106484.923125 139.227938 22387369.791882 /
[22530.332764] kworker/4:0 29 5580.634468 23 120 5580.634468 0.108432 1094087.316362 /
[22530.332886] kworker/4:0H 30 834.814843 6 100 834.814843 0.021660 1038.532072 /
[22530.333010] deferwq 742 46.995491 2 100 46.995491 0.007740 0.003205 /
[22530.334522] scsi_eh_0 874 774.779565 3 120 774.779565 0.060958 1.843085 /
[22530.334645] scsi_tmf_0 875 666.600701 2 100 666.600701 0.067913 0.034645 /
[22530.334767] scsi_eh_1 876 774.778668 3 120 774.778668 0.062528 1.559232 /
[22530.334889] scsi_tmf_1 877 690.724569 2 100 690.724569 0.068288 0.033577 /
[22530.335010] scsi_tmf_2 879 714.738150 2 100 714.738150 0.006443 0.023105 /
[22530.335132] scsi_eh_3 880 801.449170 18 120 801.449170 2.667117 361.131087 /
[22530.335254] scsi_tmf_3 881 738.751023 2 100 738.751023 0.006412 0.018547 /
[22530.335374] scsi_tmf_4 883 762.761788 2 100 762.761788 0.005613 0.003612 /
[22530.335496] scsi_tmf_5 885 786.773061 2 100 786.773061 0.005778 0.004149 /
[22530.335618] kworker/4:1H 914 106484.944641 38243 100 106484.944641 471.666864 22525255.256452 /
[22530.335740] bioset 936 922.349057 2 100 922.349057 0.008381 0.003991 /
[22530.335862] bioset 944 946.476113 2 100 946.476113 0.006653 0.004733 /
[22530.335983] bioset 952 982.623904 2 100 982.623904 0.006756 0.004473 /
[22530.336106] ext4-rsv-conver 968 1025.295960 2 100 1025.295960 0.026530 0.006667 /
[22530.336228] udevd 1156 516.641658 346 120 516.641658 44.914580 876083.215828 /autogroup-4
[22530.336351] btrfs-delalloc- 2277 106484.947468 226 120 106484.947468 3.169508 22426784.731754 /
[22530.336473] btrfs-cache-1 2287 106484.946181 218 120 106484.946181 3.569869 22426794.643863 /
[22530.336596] rsyslogd 3050 147.469561 16 120 147.469561 2.382207 130.506899 /autogroup-8
[22530.336721] getty 4175 0.110367 93 120 0.110367 2.029572 53.375444 /autogroup-63
[22530.336845] kworker/4:2 6032 106485.763504 75411 120 106485.763504 939.712241 21732007.281605 /
[22530.336968] cron 30183 1625.469001 1 120 1625.469001 1.311734 0.000000 /autogroup-27
[22530.337092]
[22530.337145] cpu#5, 3400.082 MHz
[22530.337199] .nr_running : 0
[22530.337257] .load : 0
[22530.337315] .nr_switches : 397804
[22530.337373] .nr_load_updates : 109834
[22530.337432] .nr_uninterruptible : 6455
[22530.337490] .next_balance : 4297.191355
[22530.337550] .curr->pid : 0
[22530.337608] .clock : 22527294.192330
[22530.337668] .cpu_load[0] : 0
[22530.337725] .cpu_load[1] : 0
[22530.337783] .cpu_load[2] : 0
[22530.337840] .cpu_load[3] : 0
[22530.337899] .cpu_load[4] : 0
[22530.337956] .yld_count : 0
[22530.338014] .sched_count : 413463
[22530.338073] .sched_goidle : 186876
[22530.338131] .avg_idle : 1000000
[22530.338190] .ttwu_count : 198515
[22530.338248] .ttwu_local : 114781
[22530.338307]
[22530.338307] cfs_rq[5]:/
[22530.338413] .exec_clock : 53206.511827
[22530.338472] .MIN_vruntime : 0.000001
[22530.338531] .min_vruntime : 77129.534916
[22530.338591] .max_vruntime : 0.000001
[22530.338649] .spread : 0.000000
[22530.338709] .spread0 : -171784.549336
[22530.338768] .nr_spread_over : 634
[22530.338826] .nr_running : 0
[22530.338884] .load : 0
[22530.338942] .runnable_load_avg : 0
[22530.338999] .blocked_load_avg : 0
[22530.339057] .tg_load_contrib : 0
[22530.339114] .tg_runnable_contrib : 0
[22530.339172] .tg_load_avg : 0
[22530.339229] .tg->runnable_avg : 11
[22530.339288] .tg->cfs_bandwidth.timer_active: 0
[22530.339345] .throttled : 0
[22530.339403] .throttle_count : 0
[22530.339461] .avg->runnable_avg_sum : 20
[22530.339518] .avg->runnable_avg_period : 47195
[22530.339577]
[22530.339577] rt_rq[5]:/
[22530.339681] .rt_nr_running : 0
[22530.339740] .rt_throttled : 0
[22530.339797] .rt_time : 0.000000
[22530.339857] .rt_runtime : 950.000000
[22530.339916]
[22530.339916] runnable tasks:
[22530.339916] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[22530.339916] ----------------------------------------------------------------------------------------------------------
[22530.340253] watchdog/5 31 -8.984038 5639 0 -8.984038 68.771450 32.993025 /
[22530.340374] migration/5 32 0.000000 6362 0 0.000000 50.937131 0.000878 /
[22530.340496] ksoftirqd/5 33 77117.545395 4450 120 77117.545395 83.577565 22497932.592565 /
[22530.340618] kworker/5:0 34 4234.045902 18 120 4234.045902 0.067026 1003827.021883 /
[22530.340740] kworker/5:0H 35 413.646668 6 100 413.646668 0.023930 2681.599906 /
[22530.340864] scsi_eh_4 882 258.541557 18 120 258.541557 2.743332 350.975744 /
[22530.340987] kworker/5:1H 994 77117.492193 9004 100 77117.492193 77.541909 22448691.875804 /
[22530.341110] btrfs-flush_del 2289 77117.514861 219 120 77117.514861 2.993641 22426794.380162 /
[22530.341232] btrfs-cleaner 2321 77117.598249 1112 120 77117.598249 44.314008 22469553.164811 /
[22530.341354] udevd 2613 374.649613 16 120 374.649613 3.226625 862493.064659 /autogroup-4
[22530.341478] cron 3427 2199.374500 954 120 2199.374500 108.230518 22448993.764789 /autogroup-27
[22530.341602] dbus-daemon 3498 0.978914 1 120 0.978914 0.141902 0.000000 /autogroup-29
[22530.341726] mdadm 3655 3.182464 23 120 3.182464 4.294591 21989232.539056 /autogroup-32
[22530.341851] kworker/5:2 5705 77118.374143 68658 120 77118.374143 831.022717 21824263.004596 /
[22530.341975] btrfs-flush_del 30924 48607.753771 2 120 48607.753771 0.026860 0.004432 /
[22530.342097]
[22530.342150] cpu#6, 3400.082 MHz
[22530.342205] .nr_running : 0
[22530.342263] .load : 0
[22530.342320] .nr_switches : 451998
[22530.342379] .nr_load_updates : 130538
[22530.342446] .nr_uninterruptible : 6316
[22530.342507] .next_balance : 4297.191405
[22530.342567] .curr->pid : 0
[22530.342624] .clock : 22527793.894898
[22530.342684] .cpu_load[0] : 0
[22530.342743] .cpu_load[1] : 0
[22530.342800] .cpu_load[2] : 0
[22530.342858] .cpu_load[3] : 0
[22530.342915] .cpu_load[4] : 0
[22530.342973] .yld_count : 0
[22530.343031] .sched_count : 465739
[22530.343090] .sched_goidle : 212427
[22530.343148] .avg_idle : 1000000
[22530.343207] .ttwu_count : 223131
[22530.343265] .ttwu_local : 137451
[22530.343323]
[22530.343323] cfs_rq[6]:/
[22530.343429] .exec_clock : 52045.486977
[22530.343488] .MIN_vruntime : 0.000001
[22530.343547] .min_vruntime : 87712.801268
[22530.343607] .max_vruntime : 0.000001
[22530.343666] .spread : 0.000000
[22530.343725] .spread0 : -161201.369733
[22530.343784] .nr_spread_over : 1304
[22530.343842] .nr_running : 0
[22530.343900] .load : 0
[22530.343958] .runnable_load_avg : 0
[22530.344016] .blocked_load_avg : 0
[22530.344073] .tg_load_contrib : 0
[22530.344131] .tg_runnable_contrib : 0
[22530.344189] .tg_load_avg : 1
[22530.344246] .tg->runnable_avg : 12
[22530.344305] .tg->cfs_bandwidth.timer_active: 0
[22530.344362] .throttled : 0
[22530.344420] .throttle_count : 0
[22530.344478] .avg->runnable_avg_sum : 16
[22530.344536] .avg->runnable_avg_period : 48042
[22530.344595]
[22530.344595] rt_rq[6]:/
[22530.344699] .rt_nr_running : 0
[22530.344756] .rt_throttled : 0
[22530.344815] .rt_time : 0.000000
[22530.344874] .rt_runtime : 950.000000
[22530.344933]
[22530.344933] runnable tasks:
[22530.344933] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[22530.344933] ----------------------------------------------------------------------------------------------------------
[22530.346657] watchdog/6 36 -8.979519 5639 0 -8.979519 51.042327 19.375885 /
[22530.346779] migration/6 37 0.000000 6278 0 0.000000 48.805490 0.000779 /
[22530.346901] ksoftirqd/6 38 87701.000541 8090 120 87701.000541 80.922139 22524301.674997 /
[22530.347023] kworker/6:0 39 14238.774211 16 120 14238.774211 0.053544 1105053.883586 /
[22530.347144] kworker/6:0H 40 8366.913807 6 100 8366.913807 0.086353 2062.303268 /
[22530.347267] irq/41-mei_me 684 0.000000 17 49 0.000000 0.319436 0.000000 /
[22530.347388] ata_sff 871 8330.570706 2 100 8330.570706 0.035098 0.006128 /
[22530.347511] kworker/6:1H 972 87700.771842 8055 100 87700.771842 59.741155 22407960.884540 /
[22530.347636] qmgr 4133 1.183635 170 120 1.183635 20.886187 22191616.793959 /autogroup-58
[22530.347760] kworker/6:2 6370 87702.251718 84127 120 87702.251718 1009.772712 21721970.548519 /
[22530.347884]
[22530.347936] cpu#7, 3400.082 MHz
[22530.347991] .nr_running : 0
[22530.348049] .load : 0
[22530.348107] .nr_switches : 428809
[22530.348165] .nr_load_updates : 109047
[22530.348224] .nr_uninterruptible : 6868
[22530.348282] .next_balance : 4297.191455
[22530.348341] .curr->pid : 0
[22530.348400] .clock : 22528293.606579
[22530.348459] .cpu_load[0] : 0
[22530.348517] .cpu_load[1] : 0
[22530.348575] .cpu_load[2] : 0
[22530.348632] .cpu_load[3] : 0
[22530.348690] .cpu_load[4] : 0
[22530.348748] .yld_count : 0
[22530.348806] .sched_count : 441811
[22530.348864] .sched_goidle : 203344
[22530.348923] .avg_idle : 1000000
[22530.348981] .ttwu_count : 206180
[22530.349039] .ttwu_local : 110169
[22530.349099]
[22530.349099] cfs_rq[7]:/
[22530.349203] .exec_clock : 47934.948538
[22530.349263] .MIN_vruntime : 0.000001
[22530.349322] .min_vruntime : 74031.661277
[22530.349381] .max_vruntime : 0.000001
[22530.349440] .spread : 0.000000
[22530.349499] .spread0 : -174882.509724
[22530.349559] .nr_spread_over : 538
[22530.349616] .nr_running : 0
[22530.349674] .load : 0
[22530.349731] .runnable_load_avg : 0
[22530.349789] .blocked_load_avg : 0
[22530.349847] .tg_load_contrib : 0
[22530.349904] .tg_runnable_contrib : 0
[22530.349962] .tg_load_avg : 1
[22530.350020] .tg->runnable_avg : 12
[22530.350077] .tg->cfs_bandwidth.timer_active: 0
[22530.350135] .throttled : 0
[22530.350193] .throttle_count : 0
[22530.350250] .avg->runnable_avg_sum : 20
[22530.350308] .avg->runnable_avg_period : 48445
[22530.350368]
[22530.350368] rt_rq[7]:/
[22530.350472] .rt_nr_running : 0
[22530.350530] .rt_throttled : 0
[22530.350587] .rt_time : 0.000000
[22530.350646] .rt_runtime : 950.000000
[22530.350706]
[22530.350706] runnable tasks:
[22530.350706] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[22530.350706] ----------------------------------------------------------------------------------------------------------
[22530.351040] kthreadd 2 57305.658158 255 120 57305.658158 5.101684 18122186.129762 /
[22530.351163] watchdog/7 41 -8.987511 5639 0 -8.987511 44.385994 5.780484 /
[22530.351284] migration/7 42 0.000000 6739 0 0.000000 53.655370 0.000762 /
[22530.351406] ksoftirqd/7 43 74019.493271 4729 120 74019.493271 70.464090 22497714.668876 /
[22530.351528] kworker/7:0 44 2364.871277 14 120 2364.871277 0.038779 700944.961679 /
[22530.351649] kworker/7:0H 45 742.905544 6 100 742.905544 0.029399 983.929453 /
[22530.351771] khelper 46 10.956024 2 100 10.956024 0.005379 0.002394 /
[22530.351893] netns 47 22.959366 2 100 22.959366 0.003651 0.001915 /
[22530.352015] writeback 196 46.966222 2 100 46.966222 0.003949 0.001397 /
[22530.352137] bioset 198 58.969483 2 100 58.969483 0.003554 0.001482 /
[22530.352259] kblockd 199 70.972663 2 100 70.972663 0.003473 0.001477 /
[22530.352380] md 368 94.979536 2 100 94.979536 0.003469 0.001485 /
[22530.352514] kswapd0 517 215.094515 3 120 215.094515 0.016066 96.853648 /
[22530.352636] crypto 590 155.024795 2 100 155.024795 0.007868 0.002543 /
[22530.352758] kworker/7:1 626 74020.559054 71410 120 74020.559054 875.435981 22526134.757772 /
[22530.352881] kworker/7:1H 901 73942.944698 6029 100 73942.944698 50.935561 22381894.309503 /
[22530.353004] jbd2/md1-8 2266 1463.973239 2 120 1463.973239 0.074372 0.045978 /
[22530.353126] ext4-rsv-conver 2267 1476.021115 2 100 1476.021115 0.049257 0.024787 /
[22530.353248] bioset 2273 1500.080675 2 100 1500.080675 0.025348 0.027003 /
[22530.353372] getty 4178 0.790927 100 120 0.790927 2.124261 52.920618 /autogroup-66
[22530.353496] btrfs-cleaner 5687 74019.671496 621 120 74019.671496 17.130059 21850927.888824 /
[22530.353618] udevd 6366 506.784114 5 120 506.784114 1.277593 3.684182 /autogroup-4
[22530.353744]
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: making snapshot kills RAID-1 rebuild?
2014-03-28 3:08 making snapshot kills RAID-1 rebuild? Tomasz Chmielewski
@ 2014-03-28 12:25 ` Tomasz Chmielewski
0 siblings, 0 replies; 2+ messages in thread
From: Tomasz Chmielewski @ 2014-03-28 12:25 UTC (permalink / raw)
To: linux-btrfs
On Fri, 28 Mar 2014 04:08:49 +0100
Tomasz Chmielewski <tch@virtall.com> wrote:
> One of drives died in my server and was replaced.
>
> Some time after running "btrfs device delete missing ...", a read only
> snapshot was made. Since then, "btrfs subvolume snapshot -r ..."
> process still didn't return, and there are no more reads or writes to
> any of drives in this RAID-1 filesystem.
>
> Kernel is 3.14.0-rc8.
I had to reboot with echo b > /proc/sysrq-trigger, since it wasn't doing anything.
After it booted again, I started "btrfs device delete missing ...", but it hanged again.
There were a few of these in dmesg, but it's silent from now on.
No more read or write to any of the drives in btrfs filesystem (as seen with iostat -m).
I'm pasting echo w > /proc/sysrq-trigger output below.
[ 1679.588232] INFO: task btrfs:4421 blocked for more than 120 seconds.
[ 1679.588330] Not tainted 3.14.0-rc8 #1
[ 1679.588405] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1679.588523] btrfs D 0000000000000002 0 4421 4347 0x00000000
[ 1679.588665] ffff8807d7ce17d8 0000000000000082 ffffffffffffffe5 ffff8807f27de000
[ 1679.590016] 00000000000116c0 0000000000004000 ffffffff81610450 ffff8807f27de000
[ 1679.590258] 00000000ffffff01 ffff8807d7ce17b8 ffffffff811aefcd ffff8807d7ce174e
[ 1679.590501] Call Trace:
[ 1679.590578] [<ffffffff811aefcd>] ? number.isra.1+0x128/0x238
[ 1679.590657] [<ffffffff81079f6b>] ? ktime_get_ts+0x50/0xb7
[ 1679.590735] [<ffffffff810a301f>] ? delayacct_end+0x77/0x82
[ 1679.590814] [<ffffffff810b1c94>] ? __lock_page+0x63/0x63
[ 1679.590893] [<ffffffff8139708e>] schedule+0x6a/0x6c
[ 1679.590969] [<ffffffff8139711a>] io_schedule+0x8a/0xc8
[ 1679.591047] [<ffffffff810b1c9d>] sleep_on_page+0x9/0xd
[ 1679.591124] [<ffffffff8139767b>] __wait_on_bit+0x45/0x7e
[ 1679.591201] [<ffffffff810b1a1d>] ? find_get_pages_tag+0xe9/0x123
[ 1679.591280] [<ffffffff810b1ea5>] wait_on_page_bit+0x71/0x73
[ 1679.591358] [<ffffffff81067f46>] ? wake_atomic_t_function+0x28/0x28
[ 1679.591447] [<ffffffffa02d056e>] extent_write_cache_pages.isra.30.constprop.46+0x181/0x2f6 [btrfs]
[ 1679.591573] [<ffffffffa02d0971>] extent_writepages+0x46/0x57 [btrfs]
[ 1679.591657] [<ffffffffa02b9c56>] ? btrfs_fiemap+0x4c/0x4c [btrfs]
[ 1679.591736] [<ffffffff81392976>] ? printk+0x48/0x4a
[ 1679.591816] [<ffffffffa02b8dbd>] btrfs_writepages+0x23/0x25 [btrfs]
[ 1679.591896] [<ffffffff810badc0>] do_writepages+0x19/0x27
[ 1679.591973] [<ffffffff810b27c3>] __filemap_fdatawrite_range+0x4e/0x50
[ 1679.592052] [<ffffffff810b2d67>] filemap_fdatawrite_range+0xe/0x10
[ 1679.592136] [<ffffffffa02cbc5f>] btrfs_wait_ordered_range+0x49/0x120 [btrfs]
[ 1679.592221] [<ffffffffa02f7691>] btrfs_relocate_block_group+0x1a9/0x275 [btrfs]
[ 1679.592344] [<ffffffffa02d4e0a>] btrfs_relocate_chunk.isra.65+0x58/0x60e [btrfs]
[ 1679.592468] [<ffffffffa02e37c3>] ? btrfs_set_lock_blocking_rw+0x89/0xb2 [btrfs]
[ 1679.592588] [<ffffffffa0296a2a>] ? btrfs_set_path_blocking+0x23/0x54 [btrfs]
[ 1679.592670] [<ffffffffa029b477>] ? btrfs_search_slot+0x783/0x7dd [btrfs]
[ 1679.592754] [<ffffffffa02d1103>] ? free_extent_buffer+0x6f/0x7c [btrfs]
[ 1679.592837] [<ffffffffa02d70f9>] btrfs_shrink_device+0x23c/0x3a5 [btrfs]
[ 1679.592921] [<ffffffffa02d9673>] btrfs_rm_device+0x29e/0x6a7 [btrfs]
[ 1679.593004] [<ffffffffa02e15f9>] btrfs_ioctl+0xa48/0x25d3 [btrfs]
[ 1679.593084] [<ffffffff810ce1df>] ? handle_mm_fault+0x24f/0x965
[ 1679.593162] [<ffffffff8116e386>] ? avc_has_perm+0x2e/0xf2
[ 1679.593241] [<ffffffff811071a9>] do_vfs_ioctl+0x3f2/0x43c
[ 1679.593318] [<ffffffff81107241>] SyS_ioctl+0x4e/0x7d
[ 1679.593396] [<ffffffff8102f7fd>] ? do_page_fault+0xc/0xf
[ 1679.593474] [<ffffffff8139a0e2>] system_call_fastpath+0x16/0x1b
[32283.486714] SysRq : Show Blocked State
[32283.486762] task PC stack pid father
[32283.486815] kworker/2:1 D 0000000000000002 0 616 2 0x00000000
[32283.486869] Workqueue: events do_sync_work
[32283.486914] ffff8807f19f7b08 0000000000000046 0000000000000000 ffff8807f3f99800
[32283.487003] 00000000000116c0 0000000000004000 ffff8807f4031800 ffff8807f3f99800
[32283.487092] ffff8807f19f7a78 ffffffff8106138a ffff8807f19f7a68 ffff88081fa916c0
[32283.487181] Call Trace:
[32283.487228] [<ffffffff8106138a>] ? put_prev_task_fair+0x60/0x28e
[32283.487277] [<ffffffff81396e8f>] ? __schedule+0x74f/0x8ae
[32283.487324] [<ffffffff81058603>] ? ttwu_do_wakeup+0x12/0x7f
[32283.487372] [<ffffffff810b1c94>] ? __lock_page+0x63/0x63
[32283.487419] [<ffffffff8139708e>] schedule+0x6a/0x6c
[32283.487466] [<ffffffff8139711a>] io_schedule+0x8a/0xc8
[32283.487512] [<ffffffff810b1c9d>] sleep_on_page+0x9/0xd
[32283.487559] [<ffffffff8139767b>] __wait_on_bit+0x45/0x7e
[32283.487606] [<ffffffff810b1a1d>] ? find_get_pages_tag+0xe9/0x123
[32283.487654] [<ffffffff810b1ea5>] wait_on_page_bit+0x71/0x73
[32283.487701] [<ffffffff81067f46>] ? wake_atomic_t_function+0x28/0x28
[32283.487749] [<ffffffff810b1f5f>] filemap_fdatawait_range+0x79/0x118
[32283.487798] [<ffffffff810b201a>] filemap_fdatawait+0x1c/0x1e
[32283.487845] [<ffffffff81117124>] sync_inodes_sb+0x148/0x191
[32283.487893] [<ffffffff8111abb0>] ? fdatawrite_one_bdev+0x13/0x13
[32283.487940] [<ffffffff8111abbf>] sync_inodes_one_sb+0xf/0x11
[32283.487989] [<ffffffff810fb1ce>] iterate_supers+0x66/0xb6
[32283.488036] [<ffffffff8111ab34>] do_sync_work+0x23/0x8c
[32283.488083] [<ffffffff8104b991>] process_one_work+0x1be/0x2f4
[32283.488130] [<ffffffff8104bf2a>] worker_thread+0x1cd/0x2c4
[32283.488177] [<ffffffff8104bd5d>] ? rescuer_thread+0x26c/0x26c
[32283.488225] [<ffffffff81050c92>] kthread+0xcd/0xd5
[32283.488271] [<ffffffff81050bc5>] ? kthread_freezable_should_stop+0x43/0x43
[32283.488320] [<ffffffff8139a03c>] ret_from_fork+0x7c/0xb0
[32283.488367] [<ffffffff81050bc5>] ? kthread_freezable_should_stop+0x43/0x43
[32283.488426] btrfs D 0000000000000002 0 4421 4347 0x00000000
[32283.488475] ffff8807d7ce17d8 0000000000000082 ffffffffffffffe5 ffff8807f27de000
[32283.488564] 00000000000116c0 0000000000004000 ffffffff81610450 ffff8807f27de000
[32283.488653] 00000000ffffff01 ffff8807d7ce17b8 ffffffff811aefcd ffff8807d7ce174e
[32283.488742] Call Trace:
[32283.488786] [<ffffffff811aefcd>] ? number.isra.1+0x128/0x238
[32283.488834] [<ffffffff81079f6b>] ? ktime_get_ts+0x50/0xb7
[32283.488882] [<ffffffff810a301f>] ? delayacct_end+0x77/0x82
[32283.488929] [<ffffffff810b1c94>] ? __lock_page+0x63/0x63
[32283.488976] [<ffffffff8139708e>] schedule+0x6a/0x6c
[32283.489022] [<ffffffff8139711a>] io_schedule+0x8a/0xc8
[32283.489069] [<ffffffff810b1c9d>] sleep_on_page+0x9/0xd
[32283.489115] [<ffffffff8139767b>] __wait_on_bit+0x45/0x7e
[32283.489162] [<ffffffff810b1a1d>] ? find_get_pages_tag+0xe9/0x123
[32283.489210] [<ffffffff810b1ea5>] wait_on_page_bit+0x71/0x73
[32283.489257] [<ffffffff81067f46>] ? wake_atomic_t_function+0x28/0x28
[32283.489316] [<ffffffffa02d056e>] extent_write_cache_pages.isra.30.constprop.46+0x181/0x2f6 [btrfs]
[32283.489413] [<ffffffffa02d0971>] extent_writepages+0x46/0x57 [btrfs]
[32283.489466] [<ffffffffa02b9c56>] ? btrfs_fiemap+0x4c/0x4c [btrfs]
[32283.489515] [<ffffffff81392976>] ? printk+0x48/0x4a
[32283.489565] [<ffffffffa02b8dbd>] btrfs_writepages+0x23/0x25 [btrfs]
[32283.489614] [<ffffffff810badc0>] do_writepages+0x19/0x27
[32283.489661] [<ffffffff810b27c3>] __filemap_fdatawrite_range+0x4e/0x50
[32283.489710] [<ffffffff810b2d67>] filemap_fdatawrite_range+0xe/0x10
[32283.489763] [<ffffffffa02cbc5f>] btrfs_wait_ordered_range+0x49/0x120 [btrfs]
[32283.489818] [<ffffffffa02f7691>] btrfs_relocate_block_group+0x1a9/0x275 [btrfs]
[32283.489910] [<ffffffffa02d4e0a>] btrfs_relocate_chunk.isra.65+0x58/0x60e [btrfs]
[32283.490003] [<ffffffffa02e37c3>] ? btrfs_set_lock_blocking_rw+0x89/0xb2 [btrfs]
[32283.490094] [<ffffffffa0296a2a>] ? btrfs_set_path_blocking+0x23/0x54 [btrfs]
[32283.490145] [<ffffffffa029b477>] ? btrfs_search_slot+0x783/0x7dd [btrfs]
[32283.490198] [<ffffffffa02d1103>] ? free_extent_buffer+0x6f/0x7c [btrfs]
[32283.490251] [<ffffffffa02d70f9>] btrfs_shrink_device+0x23c/0x3a5 [btrfs]
[32283.490305] [<ffffffffa02d9673>] btrfs_rm_device+0x29e/0x6a7 [btrfs]
[32283.490357] [<ffffffffa02e15f9>] btrfs_ioctl+0xa48/0x25d3 [btrfs]
[32283.490406] [<ffffffff810ce1df>] ? handle_mm_fault+0x24f/0x965
[32283.490454] [<ffffffff8116e386>] ? avc_has_perm+0x2e/0xf2
[32283.490502] [<ffffffff811071a9>] do_vfs_ioctl+0x3f2/0x43c
[32283.490549] [<ffffffff81107241>] SyS_ioctl+0x4e/0x7d
[32283.490595] [<ffffffff8102f7fd>] ? do_page_fault+0xc/0xf
[32283.490642] [<ffffffff8139a0e2>] system_call_fastpath+0x16/0x1b
[32283.490690] Sched Debug Version: v0.11, 3.14.0-rc8 #1
[32283.490736] ktime : 32302517.957122
[32283.490783] sched_clk : 32283490.689767
[32283.490831] cpu_clk : 32283490.689784
[32283.490878] jiffies : 4298167548
[32283.490925] sched_clock_stable() : 1
[32283.490970]
[32283.491011] sysctl_sched
[32283.491054] .sysctl_sched_latency : 24.000000
[32283.491101] .sysctl_sched_min_granularity : 3.000000
[32283.491148] .sysctl_sched_wakeup_granularity : 4.000000
[32283.491195] .sysctl_sched_child_runs_first : 0
[32283.491241] .sysctl_sched_features : 11899
[32283.491288] .sysctl_sched_tunable_scaling : 1 (logaritmic)
[32283.491335]
[32283.491376] cpu#0, 3400.125 MHz
[32283.491420] .nr_running : 0
[32283.491465] .load : 0
[32283.491511] .nr_switches : 2351792
[32283.491557] .nr_load_updates : 499416
[32283.491602] .nr_uninterruptible : -19842
[32283.491648] .next_balance : 4298.167549
[32283.491694] .curr->pid : 0
[32283.491739] .clock : 32283486.618013
[32283.491785] .cpu_load[0] : 0
[32283.491830] .cpu_load[1] : 0
[32283.491875] .cpu_load[2] : 0
[32283.491920] .cpu_load[3] : 0
[32283.491965] .cpu_load[4] : 0
[32283.492010] .yld_count : 0
[32283.492055] .sched_count : 2444199
[32283.492101] .sched_goidle : 988849
[32283.492146] .avg_idle : 1000000
[32283.492192] .ttwu_count : 1749279
[32283.492238] .ttwu_local : 1230883
[32283.492284]
[32283.492284] cfs_rq[0]:/autogroup-67
[32283.492368] .exec_clock : 591.332769
[32283.492414] .MIN_vruntime : 0.000001
[32283.492460] .min_vruntime : 589.924691
[32283.492506] .max_vruntime : 0.000001
[32283.492552] .spread : 0.000000
[32283.492598] .spread0 : -142986.269264
[32283.492644] .nr_spread_over : 0
[32283.492689] .nr_running : 0
[32283.492734] .load : 0
[32283.492779] .runnable_load_avg : 0
[32283.492824] .blocked_load_avg : 0
[32283.492869] .tg_load_contrib : 0
[32283.492914] .tg_runnable_contrib : 0
[32283.492959] .tg_load_avg : 0
[32283.493003] .tg->runnable_avg : 0
[32283.494151] .tg->cfs_bandwidth.timer_active: 0
[32283.494196] .throttled : 0
[32283.494241] .throttle_count : 0
[32283.494286] .se->exec_start : 32283486.602908
[32283.494333] .se->vruntime : 143564.997569
[32283.494379] .se->sum_exec_runtime : 591.332769
[32283.494425] .se->statistics.wait_start : 0.000000
[32283.494471] .se->statistics.sleep_start : 0.000000
[32283.494516] .se->statistics.block_start : 0.000000
[32283.494562] .se->statistics.sleep_max : 0.000000
[32283.494608] .se->statistics.block_max : 0.000000
[32283.494653] .se->statistics.exec_max : 3.460545
[32283.494699] .se->statistics.slice_max : 0.085291
[32283.494745] .se->statistics.wait_max : 0.065607
[32283.494791] .se->statistics.wait_sum : 0.916045
[32283.494836] .se->statistics.wait_count : 6915
[32283.494881] .se->load.weight : 2
[32283.494926] .se->avg.runnable_avg_sum : 47
[32283.494971] .se->avg.runnable_avg_period : 47872
[32283.495017] .se->avg.load_avg_contrib : 0
[32283.495061] .se->avg.decay_count : 30787933
[32283.495107]
[32283.495107] cfs_rq[0]:/
[32283.495190] .exec_clock : 142312.443335
[32283.495236] .MIN_vruntime : 0.000001
[32283.495282] .min_vruntime : 143576.193955
[32283.495328] .max_vruntime : 0.000001
[32283.495374] .spread : 0.000000
[32283.495419] .spread0 : 0.000000
[32283.495465] .nr_spread_over : 811
[32283.495510] .nr_running : 0
[32283.495555] .load : 0
[32283.495600] .runnable_load_avg : 0
[32283.495645] .blocked_load_avg : 0
[32283.495690] .tg_load_contrib : 0
[32283.495735] .tg_runnable_contrib : 1
[32283.495779] .tg_load_avg : 0
[32283.495824] .tg->runnable_avg : 2
[32283.495869] .tg->cfs_bandwidth.timer_active: 0
[32283.495914] .throttled : 0
[32283.495959] .throttle_count : 0
[32283.496004] .avg->runnable_avg_sum : 73
[32283.496049] .avg->runnable_avg_period : 47226
[32283.496096]
[32283.496096] rt_rq[0]:/
[32283.496178] .rt_nr_running : 0
[32283.496223] .rt_throttled : 0
[32283.496268] .rt_time : 0.000000
[32283.496314] .rt_runtime : 950.000000
[32283.496360]
[32283.496360] runnable tasks:
[32283.496360] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[32283.496360] ----------------------------------------------------------------------------------------------------------
[32283.496624] init 1 40.682537 7373 120 40.682537 576.650618 32277970.085462 /autogroup-2
[32283.496720] kthreadd 2 143541.151353 182 120 143541.151353 3.448481 32164963.455002 /
[32283.496814] ksoftirqd/0 3 143564.028158 22734 120 143564.028158 355.083774 32281642.744247 /
[32283.496907] kworker/0:0H 5 884.259352 5 100 884.259352 0.040041 1079.407522 /
[32283.497002] migration/0 9 0.000000 171 0 0.000000 409.770651 0.000816 /
[32283.497095] watchdog/0 10 -11.811848 8085 0 -11.811848 94.032248 95.061989 /
[32283.497193] kworker/0:1 472 143567.427579 318731 120 143567.427579 3712.284468 32276788.436246 /
[32283.497287] kworker/0:2 700 3224.909050 7 120 3224.909050 0.026753 45933.935909 /
[32283.497382] scsi_eh_1 883 879.817283 3 120 879.817283 0.070697 1.699335 /
[32283.497476] scsi_eh_4 889 884.583224 18 120 884.583224 4.780367 350.894229 /
[32283.497570] kworker/u16:3 894 138632.780043 227 120 138632.780043 23.058594 30238432.309806 /
[32283.497664] kworker/0:1H 904 143564.016992 429911 100 143564.016992 4194.301245 32276349.829423 /
[32283.497759] md2_raid1 964 143564.229422 245280 120 143564.229422 10318.033482 32268489.872800 /
[32283.497853] jbd2/md2-8 980 143564.298666 20498 120 143564.298666 1119.724277 32278521.110560 /
[32283.497948] btrfs-endio-rai 2290 143543.620210 324 120 143543.620210 3.905287 32141051.607006 /
[32283.498042] btrfs-endio-wri 2292 143543.547257 305 120 143543.547257 4.229202 32141071.333501 /
[32283.498137] btrfs-readahead 2296 143543.547503 309 120 143543.547503 7.399807 32141038.478482 /
[32283.498231] btrfs-transacti 2330 143558.828362 28858 120 143558.828362 2673.622812 32206631.221262 /
[32283.498325] sshd 2779 0.478108 5 120 0.478108 0.931674 113390.095510 /autogroup-6
[32283.498420] rs:main Q:Reg 3148 537.928333 16152 120 537.928333 1775.128528 32227290.619330 /autogroup-8
[32283.498515] apache2 3379 100.068386 32221 120 100.068386 558.803231 32232555.121792 /autogroup-24
[32283.498611] rsync 4165 80.310540 13 139 80.310540 1.438598 52.067742 /autogroup-58
[32283.498707] getty 4198 0.823437 104 120 0.823437 4.053598 177.159436 /autogroup-61
[32283.498802] sshd 4204 589.924691 7366 120 589.924691 649.155810 32230483.926040 /autogroup-67
[32283.498897] btrfs-worker-1 4280 143543.416180 288 120 143543.416180 5.475678 32021147.594450 /
[32283.498992] btrfs-endio-met 4289 143543.373343 313 120 143543.373343 7.774000 32021123.936752 /
[32283.499086] btrfs-cache-1 4293 143540.929978 587 120 143540.929978 33.686287 32008617.769624 /
[32283.499179] btrfs-flush_del 4295 143543.345209 305 120 143543.345209 6.617744 32021145.415637 /
[32283.499274] tmux 4346 1162.913967 96969 120 1162.913967 6221.681365 32111899.717013 /autogroup-73
[32283.499369] btrfs 4421 973.871415 63843 120 973.871415 7302.768461 468569.854836 /autogroup-74
[32283.499464] pickup 7050 44.126339 40 120 44.126339 7.302359 2280927.079801 /autogroup-51
[32283.499560]
[32283.499601] cpu#1, 3400.125 MHz
[32283.499644] .nr_running : 0
[32283.499689] .load : 0
[32283.499734] .nr_switches : 1188730
[32283.499779] .nr_load_updates : 415952
[32283.499825] .nr_uninterruptible : -209
[32283.499870] .next_balance : 4298.167507
[32283.499916] .curr->pid : 0
[32283.499961] .clock : 32283063.014670
[32283.500007] .cpu_load[0] : 0
[32283.500052] .cpu_load[1] : 0
[32283.500097] .cpu_load[2] : 0
[32283.500142] .cpu_load[3] : 0
[32283.500187] .cpu_load[4] : 0
[32283.500232] .yld_count : 0
[32283.500277] .sched_count : 1206276
[32283.500322] .sched_goidle : 556410
[32283.500367] .avg_idle : 1000000
[32283.500413] .ttwu_count : 504935
[32283.500458] .ttwu_local : 366928
[32283.500504]
[32283.500504] cfs_rq[1]:/
[32283.500587] .exec_clock : 146910.069109
[32283.500633] .MIN_vruntime : 0.000001
[32283.500679] .min_vruntime : 151914.079490
[32283.500725] .max_vruntime : 0.000001
[32283.500771] .spread : 0.000000
[32283.500816] .spread0 : 8337.885535
[32283.500863] .nr_spread_over : 938
[32283.500908] .nr_running : 0
[32283.500953] .load : 0
[32283.500998] .runnable_load_avg : 0
[32283.501042] .blocked_load_avg : 0
[32283.501087] .tg_load_contrib : 0
[32283.501132] .tg_runnable_contrib : 1
[32283.501177] .tg_load_avg : 0
[32283.501222] .tg->runnable_avg : 2
[32283.501267] .tg->cfs_bandwidth.timer_active: 0
[32283.501312] .throttled : 0
[32283.501357] .throttle_count : 0
[32283.501402] .avg->runnable_avg_sum : 79
[32283.501447] .avg->runnable_avg_period : 48047
[32283.501493]
[32283.501493] rt_rq[1]:/
[32283.501575] .rt_nr_running : 0
[32283.501620] .rt_throttled : 0
[32283.501665] .rt_time : 0.000000
[32283.501711] .rt_runtime : 950.000000
[32283.501758]
[32283.501758] runnable tasks:
[32283.501758] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[32283.501758] ----------------------------------------------------------------------------------------------------------
[32283.502022] rcu_bh 8 3899.164533 5 120 3899.164533 0.100037 309315.903332 /
[32283.502116] watchdog/1 11 -2.981589 8079 0 -2.981589 88.260190 87.221202 /
[32283.502210] migration/1 12 0.000000 122 0 0.000000 5.980029 0.000823 /
[32283.502303] ksoftirqd/1 13 151901.505416 15381 120 151901.505416 207.640238 32277430.690243 /
[32283.503501] kworker/1:0 14 -7.015236 13 120 -7.015236 0.034237 521.509860 /
[32283.503594] kworker/1:0H 15 415.341186 6 100 415.341186 0.018546 1103.934040 /
[32283.503691] kworker/1:1 614 151902.093052 235361 120 151902.093052 2878.237707 32277331.627567 /
[32283.503786] kworker/1:1H 920 151857.212767 41668 100 151857.212767 327.102124 32252855.378010 /
[32283.503880] md1_raid1 955 151342.671848 68 120 151342.671848 1.602410 31832720.667613 /
[32283.503975] btrfs-worker-1 2282 151803.563595 272 120 151803.563595 10.988558 32141026.909135 /
[32283.504069] btrfs-submit-1 2284 151803.546606 294 120 151803.546606 6.576644 32141039.557915 /
[32283.504162] btrfs-delalloc- 2285 151803.480152 293 120 151803.480152 4.557906 32141071.731177 /
[32283.504256] btrfs-fixup-1 2286 151803.480910 317 120 151803.480910 4.003860 32141061.267993 /
[32283.504350] btrfs-endio-met 2288 151853.530618 32969 120 151853.530618 919.414972 32192044.398512 /
[32283.504444] btrfs-freespace 2293 151808.295609 309 120 151808.295609 12.869967 32171546.336579 /
[32283.504538] btrfs-cache-1 2295 151803.481096 325 120 151803.481096 3.426302 32141061.389559 /
[32283.504632] vnstatd 3051 251.694558 6669 120 251.694558 915.920274 32229295.679743 /autogroup-9
[32283.504728] apache2 3419 64.701658 651 120 64.701658 9.789388 30529347.032959 /autogroup-24
[32283.504823] apache2 3420 64.706192 118 120 64.706192 2.332971 30898346.965010 /autogroup-24
[32283.504918] apache2 3421 65.584472 138 120 65.584472 2.291807 32228874.553685 /autogroup-24
[32283.505013] apache2 3422 28.546967 735 120 28.546967 10.280688 14533508.017732 /autogroup-24
[32283.505108] apache2 3423 24.238810 152 120 24.238810 2.436517 16013597.986806 /autogroup-24
[32283.505203] proftpd 4002 23.257416 2312 120 23.257416 129.145742 32204818.937717 /autogroup-50
[32283.505298] master 4004 32.236058 1701 120 32.236058 87.929293 32228928.112103 /autogroup-51
[32283.505394] getty 4200 0.374777 94 120 0.374777 1.788001 179.155154 /autogroup-63
[32283.505490] btrfs-delalloc- 4283 151803.349716 300 120 151803.349716 7.134004 32021115.333485 /
[32283.505584] btrfs-endio-rai 4288 151803.372809 309 120 151803.372809 5.802438 32021126.105720 /
[32283.505678] btrfs-freespace 4291 151858.166454 612 120 151858.166454 35.290287 32104009.751491 /
[32283.505772] sshd 4297 19.511357 6626 120 19.511357 591.679085 32117289.554915 /autogroup-71
[32283.505867] btrfs-endio-2 5123 151860.344716 2482 120 151860.344716 784.337874 31730623.080403 /
[32283.505961] btrfs-endio-wri 5766 151855.218546 3028 120 151855.218546 84.694781 31641739.697827 /
[32283.506055] apache2 18600 64.716270 1454 120 64.716270 20.319748 24078956.018902 /autogroup-24
[32283.506150]
[32283.506191] cpu#2, 3400.125 MHz
[32283.506234] .nr_running : 2
[32283.506279] .load : 2048
[32283.506325] .nr_switches : 980389
[32283.506370] .nr_load_updates : 372164
[32283.506416] .nr_uninterruptible : -8155
[32283.506461] .next_balance : 4298.167508
[32283.506507] .curr->pid : 4206
[32283.506552] .clock : 32283486.645370
[32283.506599] .cpu_load[0] : 0
[32283.506644] .cpu_load[1] : 0
[32283.506689] .cpu_load[2] : 0
[32283.506734] .cpu_load[3] : 0
[32283.506779] .cpu_load[4] : 0
[32283.506824] .yld_count : 0
[32283.506869] .sched_count : 995718
[32283.506914] .sched_goidle : 466130
[32283.506960] .avg_idle : 1000000
[32283.507006] .ttwu_count : 444530
[32283.507051] .ttwu_local : 310478
[32283.507097]
[32283.507097] cfs_rq[2]:/autogroup-68
[32283.507181] .exec_clock : 213.346048
[32283.507226] .MIN_vruntime : 0.000001
[32283.507272] .min_vruntime : 193.085270
[32283.507318] .max_vruntime : 0.000001
[32283.507364] .spread : 0.000000
[32283.507410] .spread0 : -143383.108685
[32283.507456] .nr_spread_over : 3
[32283.507501] .nr_running : 1
[32283.507546] .load : 1024
[32283.507591] .runnable_load_avg : 0
[32283.507636] .blocked_load_avg : 0
[32283.507681] .tg_load_contrib : 0
[32283.507726] .tg_runnable_contrib : 0
[32283.507771] .tg_load_avg : 0
[32283.507816] .tg->runnable_avg : 0
[32283.507861] .tg->cfs_bandwidth.timer_active: 0
[32283.507906] .throttled : 0
[32283.507951] .throttle_count : 0
[32283.507996] .se->exec_start : 32283486.645370
[32283.508042] .se->vruntime : 113093.038229
[32283.508089] .se->sum_exec_runtime : 213.374611
[32283.508135] .se->statistics.wait_start : 0.000000
[32283.508180] .se->statistics.sleep_start : 0.000000
[32283.508226] .se->statistics.block_start : 0.000000
[32283.508272] .se->statistics.sleep_max : 0.000000
[32283.508318] .se->statistics.block_max : 0.000000
[32283.508364] .se->statistics.exec_max : 6.131485
[32283.508409] .se->statistics.slice_max : 0.000000
[32283.508455] .se->statistics.wait_max : 0.032102
[32283.508501] .se->statistics.wait_sum : 0.393546
[32283.508547] .se->statistics.wait_count : 2060
[32283.508592] .se->load.weight : 1024
[32283.508637] .se->avg.runnable_avg_sum : 2
[32283.508682] .se->avg.runnable_avg_period : 47524
[32283.508727] .se->avg.load_avg_contrib : 0
[32283.508773] .se->avg.decay_count : 30787693
[32283.508818]
[32283.508818] cfs_rq[2]:/
[32283.508901] .exec_clock : 110600.507307
[32283.508948] .MIN_vruntime : 113091.481479
[32283.508995] .min_vruntime : 113103.464545
[32283.509041] .max_vruntime : 113091.481479
[32283.509088] .spread : 0.000000
[32283.509133] .spread0 : -30472.729410
[32283.509180] .nr_spread_over : 640
[32283.509225] .nr_running : 2
[32283.509269] .load : 2048
[32283.509315] .runnable_load_avg : 0
[32283.509360] .blocked_load_avg : 0
[32283.509405] .tg_load_contrib : 0
[32283.509450] .tg_runnable_contrib : 0
[32283.509494] .tg_load_avg : 0
[32283.509539] .tg->runnable_avg : 2
[32283.509584] .tg->cfs_bandwidth.timer_active: 0
[32283.509629] .throttled : 0
[32283.509674] .throttle_count : 0
[32283.509719] .avg->runnable_avg_sum : 29
[32283.509764] .avg->runnable_avg_period : 47748
[32283.509810]
[32283.509810] rt_rq[2]:/
[32283.509893] .rt_nr_running : 0
[32283.509937] .rt_throttled : 0
[32283.509982] .rt_time : 0.000000
[32283.510028] .rt_runtime : 950.000000
[32283.510074]
[32283.510074] runnable tasks:
[32283.510074] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[32283.510074] ----------------------------------------------------------------------------------------------------------
[32283.510339] watchdog/2 16 -5.976134 8079 0 -5.976134 83.678348 73.609886 /
[32283.510432] migration/2 17 0.000000 129 0 0.000000 5.986655 0.000870 /
[32283.510526] ksoftirqd/2 18 113090.349583 14762 120 113090.349583 222.303526 32277411.965385 /
[32283.510620] kworker/2:0 19 113069.842366 15 120 113069.842366 0.066584 32164851.663225 /
[32283.510714] kworker/2:0H 20 437.029079 6 100 437.029079 0.020636 1386.407928 /
[32283.510810] fsnotify_mark 585 909.588176 12 120 909.588176 0.210283 33514.966304 /
[32283.510904] kworker/2:1 616 113072.368183 207681 120 113072.368183 2536.222737 32159783.267151 /
[32283.510999] scsi_eh_5 891 398.450061 18 120 398.450061 4.592000 351.095693 /
[32283.511094] kworker/2:1H 958 113048.777920 23909 100 113048.777920 176.595381 32045441.541633 /
[32283.511189] btrfs-endio-1 2287 113072.127209 303 120 113072.127209 3.421534 32141062.412231 /
[32283.511284] rsyslogd 3150 614.993634 50 120 614.993634 4.193894 32116274.387088 /autogroup-8
[32283.511379] cron 3553 657.199838 1364 120 657.199838 154.174852 32227711.805108 /autogroup-30
[32283.511475] qmgr 4126 13.703100 220 120 13.703100 27.939923 32083889.654561 /autogroup-51
[32283.511570] getty 4203 0.685349 100 120 0.685349 2.048516 178.600809 /autogroup-66
[32283.511666] R bash 4206 193.085270 515 120 193.085270 234.908746 32230422.081261 /autogroup-68
[32283.511761] btrfs-fixup-1 4284 113071.976753 311 120 113071.976753 8.474176 32021134.121645 /
[32283.512959] btrfs-readahead 4294 113072.044986 320 120 113072.044986 9.286937 32021111.925717 /
[32283.513053] btrfs-qgroup-re 4296 113072.004585 296 120 113072.004585 4.992782 32021136.912318 /
[32283.513148] kworker/2:2 11777 113091.481479 311 120 113091.481479 3.596261 118498.234539 /
[32283.513241]
[32283.513283] cpu#3, 3400.125 MHz
[32283.513326] .nr_running : 0
[32283.513370] .load : 0
[32283.513415] .nr_switches : 961397
[32283.513461] .nr_load_updates : 372289
[32283.513506] .nr_uninterruptible : -7751
[32283.513552] .next_balance : 4298.167457
[32283.513598] .curr->pid : 0
[32283.513643] .clock : 32283226.903242
[32283.513689] .cpu_load[0] : 0
[32283.513734] .cpu_load[1] : 0
[32283.513779] .cpu_load[2] : 0
[32283.513824] .cpu_load[3] : 0
[32283.513869] .cpu_load[4] : 0
[32283.513914] .yld_count : 0
[32283.513959] .sched_count : 975384
[32283.514004] .sched_goidle : 459589
[32283.514050] .avg_idle : 1000000
[32283.514095] .ttwu_count : 436893
[32283.514140] .ttwu_local : 306163
[32283.514186]
[32283.514186] cfs_rq[3]:/autogroup-34
[32283.514270] .exec_clock : 263.555831
[32283.514316] .MIN_vruntime : 0.000001
[32283.514361] .min_vruntime : 262.507255
[32283.514407] .max_vruntime : 0.000001
[32283.514453] .spread : 0.000000
[32283.514499] .spread0 : -143313.686700
[32283.514545] .nr_spread_over : 0
[32283.514590] .nr_running : 0
[32283.514635] .load : 0
[32283.514680] .runnable_load_avg : 0
[32283.514725] .blocked_load_avg : 0
[32283.514769] .tg_load_contrib : 0
[32283.514814] .tg_runnable_contrib : 0
[32283.514859] .tg_load_avg : 0
[32283.514904] .tg->runnable_avg : 0
[32283.514949] .tg->cfs_bandwidth.timer_active: 0
[32283.514994] .throttled : 0
[32283.515039] .throttle_count : 0
[32283.515084] .se->exec_start : 32283226.901241
[32283.515130] .se->vruntime : 112741.576667
[32283.515176] .se->sum_exec_runtime : 263.564303
[32283.515222] .se->statistics.wait_start : 0.000000
[32283.515268] .se->statistics.sleep_start : 0.000000
[32283.515314] .se->statistics.block_start : 0.000000
[32283.515359] .se->statistics.sleep_max : 0.000000
[32283.515405] .se->statistics.block_max : 0.000000
[32283.515451] .se->statistics.exec_max : 0.183945
[32283.515497] .se->statistics.slice_max : 0.154764
[32283.515542] .se->statistics.wait_max : 0.025195
[32283.515588] .se->statistics.wait_sum : 0.076536
[32283.515634] .se->statistics.wait_count : 8044
[32283.515679] .se->load.weight : 2
[32283.515724] .se->avg.runnable_avg_sum : 26
[32283.515769] .se->avg.runnable_avg_period : 47838
[32283.515815] .se->avg.load_avg_contrib : 0
[32283.515860] .se->avg.decay_count : 30787685
[32283.515905]
[32283.515905] cfs_rq[3]:/
[32283.515988] .exec_clock : 108595.295168
[32283.516034] .MIN_vruntime : 0.000001
[32283.516080] .min_vruntime : 112753.549612
[32283.516126] .max_vruntime : 0.000001
[32283.516172] .spread : 0.000000
[32283.516217] .spread0 : -30822.644343
[32283.516264] .nr_spread_over : 869
[32283.516309] .nr_running : 0
[32283.516354] .load : 0
[32283.516399] .runnable_load_avg : 0
[32283.516444] .blocked_load_avg : 0
[32283.516488] .tg_load_contrib : 0
[32283.516533] .tg_runnable_contrib : 0
[32283.516578] .tg_load_avg : 0
[32283.516623] .tg->runnable_avg : 2
[32283.516668] .tg->cfs_bandwidth.timer_active: 0
[32283.516713] .throttled : 0
[32283.516758] .throttle_count : 0
[32283.516803] .avg->runnable_avg_sum : 29
[32283.516848] .avg->runnable_avg_period : 47652
[32283.516893]
[32283.516893] rt_rq[3]:/
[32283.516976] .rt_nr_running : 0
[32283.517021] .rt_throttled : 0
[32283.517066] .rt_time : 0.000000
[32283.517111] .rt_runtime : 950.000000
[32283.517157]
[32283.517157] runnable tasks:
[32283.517157] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[32283.517157] ----------------------------------------------------------------------------------------------------------
[32283.517421] kworker/u16:0 6 112741.857252 22476 120 112741.857252 2662.879663 32279383.634974 /
[32283.517515] rcu_sched 7 112753.549612 401388 120 112753.549612 16138.442385 32266207.495256 /
[32283.517610] watchdog/3 21 -5.984023 8079 0 -5.984023 83.314463 60.007439 /
[32283.517703] migration/3 22 0.000000 129 0 0.000000 6.005818 0.000772 /
[32283.517797] ksoftirqd/3 23 112740.610553 14042 120 112740.610553 215.876830 32277407.054210 /
[32283.517891] kworker/3:0 24 -10.012254 13 120 -10.012254 0.037350 494.644960 /
[32283.517985] kworker/3:0H 25 408.677454 6 100 408.677454 0.018433 1081.429789 /
[32283.518080] khubd 361 958.759023 62 120 958.759023 3.234725 34462.120827 /
[32283.518175] khungtaskd 507 112737.981745 271 120 112737.981745 57.065705 32260917.298827 /
[32283.518269] kworker/3:1 618 112741.477495 211230 120 112741.477495 2636.920463 32277272.532502 /
[32283.518364] kworker/3:1H 929 112540.462721 18243 100 112540.462721 136.018491 32041958.809618 /
[32283.518459] btrfs-genwork-1 2283 112729.685423 280 120 112729.685423 9.370836 32181134.340928 /
[32283.518553] btrfs-delayed-m 2294 112722.240962 297 120 112722.240962 4.242313 32141061.461401 /
[32283.518647] btrfs-qgroup-re 2298 112722.238647 314 120 112722.238647 3.253944 32141051.877656 /
[32283.518742] rsyslogd 3149 666.642109 15446 120 666.642109 676.097200 32228396.525211 /autogroup-8
[32283.518837] munin-node 3195 932.443614 16411 120 932.443614 1011.380419 32232801.342877 /autogroup-11
[32283.518933] mdadm 3660 0.324629 33 120 0.324629 6.062281 31784203.184274 /autogroup-32
[32283.519028] ntpd 3774 262.507255 33985 120 262.507255 1094.185886 32231899.345399 /autogroup-34
[32283.519124] getty 4202 0.603945 103 120 0.603945 1.702846 178.985929 /autogroup-65
[32283.519219] btrfs-genwork-1 4281 112740.490691 289 120 112740.490691 11.974548 32121203.738013 /
[32283.519313] btrfs-endio-met 4286 112737.805418 81828 120 112737.805418 2474.842227 32100882.267865 /
[32283.519407] btrfs-rmw-1 4287 112722.098007 308 120 112722.098007 6.621868 32021125.219619 /
[32283.519501] btrfs-delayed-m 4292 112722.117885 308 120 112722.117885 5.587984 32021136.787979 /
[32283.519595] bash 4299 30.763546 49 120 30.763546 42.164339 3188.802764 /autogroup-72
[32283.519690] tmux 4344 30.801147 10 120 30.801147 1.505874 27.671527 /autogroup-72
[32283.519785] bash 4347 13.476300 101 120 13.476300 38.006035 59155.131455 /autogroup-74
[32283.519880] btrfs-transacti 4415 112738.649793 13902 120 112738.649793 1461.598972 32054465.350542 /
[32283.519975]
[32283.520016] cpu#4, 3400.125 MHz
[32283.520060] .nr_running : 0
[32283.520105] .load : 0
[32283.520150] .nr_switches : 796578
[32283.520195] .nr_load_updates : 144766
[32283.520241] .nr_uninterruptible : 8489
[32283.520286] .next_balance : 4298.167305
[32283.520333] .curr->pid : 0
[32283.520378] .clock : 32282463.708491
[32283.520424] .cpu_load[0] : 0
[32283.520469] .cpu_load[1] : 0
[32283.520514] .cpu_load[2] : 0
[32283.520559] .cpu_load[3] : 0
[32283.520604] .cpu_load[4] : 0
[32283.520649] .yld_count : 0
[32283.520694] .sched_count : 811213
[32283.520739] .sched_goidle : 335831
[32283.520785] .avg_idle : 886040
[32283.520831] .ttwu_count : 294799
[32283.520876] .ttwu_local : 236938
[32283.520922]
[32283.520922] cfs_rq[4]:/
[32283.521004] .exec_clock : 80164.617933
[32283.521051] .MIN_vruntime : 0.000001
[32283.521096] .min_vruntime : 113473.305415
[32283.522248] .max_vruntime : 0.000001
[32283.522293] .spread : 0.000000
[32283.522339] .spread0 : -30102.888540
[32283.522385] .nr_spread_over : 639
[32283.522430] .nr_running : 0
[32283.522475] .load : 0
[32283.522520] .runnable_load_avg : 0
[32283.522565] .blocked_load_avg : 0
[32283.522610] .tg_load_contrib : 0
[32283.522654] .tg_runnable_contrib : 0
[32283.522699] .tg_load_avg : 0
[32283.522744] .tg->runnable_avg : 1
[32283.522789] .tg->cfs_bandwidth.timer_active: 0
[32283.522834] .throttled : 0
[32283.522879] .throttle_count : 0
[32283.522923] .avg->runnable_avg_sum : 41
[32283.522968] .avg->runnable_avg_period : 47607
[32283.523014]
[32283.523014] rt_rq[4]:/
[32283.523096] .rt_nr_running : 0
[32283.523141] .rt_throttled : 0
[32283.523186] .rt_time : 0.000000
[32283.523232] .rt_runtime : 950.000000
[32283.523278]
[32283.523278] runnable tasks:
[32283.523278] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[32283.523278] ----------------------------------------------------------------------------------------------------------
[32283.523546] watchdog/4 26 -8.981900 8079 0 -8.981900 95.206040 46.507094 /
[32283.523640] migration/4 27 0.000000 8432 0 0.000000 66.567115 0.000781 /
[32283.523734] ksoftirqd/4 28 113461.143700 5381 120 113461.143700 55.866242 32277379.941730 /
[32283.523827] kworker/4:0 29 5624.395428 19 120 5624.395428 0.049729 302229.257972 /
[32283.523921] kworker/4:0H 30 845.888661 6 100 845.888661 0.026450 1035.111079 /
[32283.524017] deferwq 742 34.988067 2 100 34.988067 0.006138 0.003138 /
[32283.524111] scsi_eh_0 881 845.828887 3 120 845.828887 0.099703 1.933529 /
[32283.524204] scsi_eh_2 885 850.308540 18 120 850.308540 4.506782 351.764753 /
[32283.524299] scsi_eh_3 887 850.617992 18 120 850.617992 4.764073 351.578304 /
[32283.524393] kworker/4:1H 909 113461.307677 107004 100 113461.307677 1087.970430 32280053.666149 /
[32283.524487] bioset 954 984.872025 2 100 984.872025 0.008022 0.002768 /
[32283.524581] kworker/4:2 966 113463.116982 65115 120 113463.116982 786.417842 32277753.440264 /
[32283.524675] udevd 1334 216.142475 6 120 216.142475 0.848997 121725.576908 /autogroup-4
[32283.524772] rsyslogd 3069 1493.821246 17 120 1493.821246 2.120170 10703328.387513 /autogroup-8
[32283.524867] acpid 3177 0.723133 12 120 0.723133 0.585163 8861946.286186 /autogroup-10
[32283.524964] btrfs-submit-1 4282 113461.126557 301 120 113461.126557 5.452759 32021137.033922 /
[32283.525059]
[32283.525100] cpu#5, 3400.125 MHz
[32283.525143] .nr_running : 0
[32283.525188] .load : 0
[32283.525233] .nr_switches : 371913
[32283.525278] .nr_load_updates : 116195
[32283.525324] .nr_uninterruptible : 9119
[32283.525369] .next_balance : 4298.167355
[32283.525415] .curr->pid : 0
[32283.525460] .clock : 32281543.929250
[32283.525507] .cpu_load[0] : 0
[32283.525551] .cpu_load[1] : 0
[32283.525596] .cpu_load[2] : 0
[32283.525641] .cpu_load[3] : 0
[32283.525686] .cpu_load[4] : 0
[32283.525731] .yld_count : 0
[32283.525776] .sched_count : 381980
[32283.525821] .sched_goidle : 168909
[32283.525867] .avg_idle : 1000000
[32283.525912] .ttwu_count : 180902
[32283.525958] .ttwu_local : 136240
[32283.526003]
[32283.526003] cfs_rq[5]:/
[32283.526086] .exec_clock : 86484.457943
[32283.526132] .MIN_vruntime : 0.000001
[32283.526178] .min_vruntime : 123224.597554
[32283.526224] .max_vruntime : 0.000001
[32283.526270] .spread : 0.000000
[32283.526316] .spread0 : -20351.596401
[32283.526362] .nr_spread_over : 902
[32283.526407] .nr_running : 0
[32283.526452] .load : 0
[32283.526497] .runnable_load_avg : 0
[32283.526542] .blocked_load_avg : 0
[32283.526587] .tg_load_contrib : 0
[32283.526632] .tg_runnable_contrib : 0
[32283.526677] .tg_load_avg : 0
[32283.526722] .tg->runnable_avg : 1
[32283.526767] .tg->cfs_bandwidth.timer_active: 0
[32283.526812] .throttled : 0
[32283.526856] .throttle_count : 0
[32283.526901] .avg->runnable_avg_sum : 25
[32283.526946] .avg->runnable_avg_period : 46792
[32283.526992]
[32283.526992] rt_rq[5]:/
[32283.527075] .rt_nr_running : 0
[32283.527119] .rt_throttled : 0
[32283.527165] .rt_time : 0.000000
[32283.527210] .rt_runtime : 950.000000
[32283.527257]
[32283.527257] runnable tasks:
[32283.527257] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[32283.527257] ----------------------------------------------------------------------------------------------------------
[32283.527522] watchdog/5 31 -8.983613 8079 0 -8.983613 95.673587 32.996614 /
[32283.527615] migration/5 32 0.000000 9145 0 0.000000 74.036671 0.000833 /
[32283.527709] ksoftirqd/5 33 123200.560000 6327 120 123200.560000 63.225330 32041944.161169 /
[32283.527803] kworker/5:0 34 1978.177009 17 120 1978.177009 0.055700 335485.719700 /
[32283.527897] kworker/5:0H 35 454.347326 6 100 454.347326 0.025860 1053.906833 /
[32283.527993] kworker/5:1H 927 123212.598482 21548 100 123212.598482 181.058362 32252994.301326 /
[32283.528087] md0_raid1 946 123164.175974 68 120 123164.175974 1.552548 31832925.034896 /
[32283.528181] udevd 1167 581.159448 282 120 581.159448 41.532151 122076.122937 /autogroup-4
[32283.528276] kworker/5:2 1550 123213.041597 74113 120 123213.041597 891.377376 32245200.998054 /
[32283.528371] btrfs-rmw-1 2289 123212.609832 344 120 123212.609832 3.535259 32141051.616685 /
[32283.528465] btrfs-cleaner 2329 123212.708238 1437 120 123212.708238 58.310167 32209255.604976 /
[32283.528559] dbus-daemon 3571 0.977291 1 120 0.977291 0.156215 0.000000 /autogroup-31
[32283.528655] getty 4201 0.059634 104 120 0.059634 1.878105 179.087701 /autogroup-64
[32283.528751] btrfs-cleaner 4414 123212.750982 1060 120 123212.750982 33.849439 32055902.765421 /
[32283.528846] apache2 8396 139.401924 1 120 139.401924 0.345190 0.000000 /autogroup-24
[32283.528941]
[32283.528982] cpu#6, 3400.125 MHz
[32283.529025] .nr_running : 0
[32283.529070] .load : 0
[32283.529115] .nr_switches : 324724
[32283.529160] .nr_load_updates : 104201
[32283.529206] .nr_uninterruptible : 8851
[32283.529251] .next_balance : 4298.167405
[32283.529297] .curr->pid : 0
[32283.529343] .clock : 32282043.635017
[32283.529389] .cpu_load[0] : 0
[32283.529434] .cpu_load[1] : 0
[32283.529479] .cpu_load[2] : 0
[32283.529524] .cpu_load[3] : 0
[32283.529569] .cpu_load[4] : 0
[32283.529614] .yld_count : 0
[32283.529659] .sched_count : 333991
[32283.529704] .sched_goidle : 147586
[32283.529750] .avg_idle : 1000000
[32283.529795] .ttwu_count : 158527
[32283.529841] .ttwu_local : 118934
[32283.529886]
[32283.529886] cfs_rq[6]:/
[32283.529969] .exec_clock : 77778.824548
[32283.530015] .MIN_vruntime : 0.000001
[32283.530061] .min_vruntime : 124784.650522
[32283.530107] .max_vruntime : 0.000001
[32283.530153] .spread : 0.000000
[32283.530199] .spread0 : -18791.543433
[32283.530245] .nr_spread_over : 1477
[32283.530290] .nr_running : 0
[32283.530335] .load : 0
[32283.530380] .runnable_load_avg : 0
[32283.530425] .blocked_load_avg : 0
[32283.531574] .tg_load_contrib : 0
[32283.531618] .tg_runnable_contrib : 0
[32283.531664] .tg_load_avg : 0
[32283.531709] .tg->runnable_avg : 1
[32283.531754] .tg->cfs_bandwidth.timer_active: 0
[32283.531798] .throttled : 0
[32283.531843] .throttle_count : 0
[32283.531888] .avg->runnable_avg_sum : 20
[32283.531933] .avg->runnable_avg_period : 48411
[32283.531979]
[32283.531979] rt_rq[6]:/
[32283.532061] .rt_nr_running : 0
[32283.532106] .rt_throttled : 0
[32283.532151] .rt_time : 0.000000
[32283.532197] .rt_runtime : 950.000000
[32283.532243]
[32283.532243] runnable tasks:
[32283.532243] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[32283.532243] ----------------------------------------------------------------------------------------------------------
[32283.532508] watchdog/6 36 -8.978765 8079 0 -8.978765 72.822265 19.378491 /
[32283.532601] migration/6 37 0.000000 8881 0 0.000000 71.142707 0.000779 /
[32283.532695] ksoftirqd/6 38 124772.355785 5604 120 124772.355785 57.415779 32046936.617701 /
[32283.532789] kworker/6:0 39 6626.693362 13 120 6626.693362 0.028516 454.512566 /
[32283.532882] kworker/6:0H 40 8261.664557 6 100 8261.664557 0.029355 1040.620134 /
[32283.532977] kworker/6:1 624 124773.501683 66127 120 124773.501683 785.484986 32279768.003101 /
[32283.533071] irq/41-mei_me 684 0.000000 17 49 0.000000 0.322865 0.000000 /
[32283.533167] kworker/6:1H 928 123857.683366 15152 100 123857.683366 133.110757 32044853.607961 /
[32283.533261] bioset 963 8333.921569 2 100 8333.921569 0.030540 0.015219 /
[32283.533356] btrfs-endio-met 2291 124772.607721 312 120 124772.607721 4.378317 32141051.350244 /
[32283.533450] btrfs-flush_del 2297 124772.604081 309 120 124772.604081 3.702297 32141061.761262 /
[32283.533544] udevd 2638 397.417571 1 120 397.417571 0.178130 0.000000 /autogroup-4
[32283.533642]
[32283.533683] cpu#7, 3400.125 MHz
[32283.533726] .nr_running : 0
[32283.533771] .load : 0
[32283.533816] .nr_switches : 322365
[32283.533861] .nr_load_updates : 104913
[32283.533907] .nr_uninterruptible : 9500
[32283.533952] .next_balance : 4298.167455
[32283.533998] .curr->pid : 0
[32283.534043] .clock : 32282543.340939
[32283.534090] .cpu_load[0] : 0
[32283.534135] .cpu_load[1] : 0
[32283.534180] .cpu_load[2] : 0
[32283.534225] .cpu_load[3] : 0
[32283.534270] .cpu_load[4] : 0
[32283.534315] .yld_count : 0
[32283.534359] .sched_count : 331851
[32283.534405] .sched_goidle : 148490
[32283.534450] .avg_idle : 1000000
[32283.534496] .ttwu_count : 156339
[32283.534541] .ttwu_local : 116278
[32283.534587]
[32283.534587] cfs_rq[7]:/
[32283.534670] .exec_clock : 63829.596590
[32283.534716] .MIN_vruntime : 0.000001
[32283.534761] .min_vruntime : 104829.817308
[32283.534808] .max_vruntime : 0.000001
[32283.534853] .spread : 0.000000
[32283.534899] .spread0 : -38746.376647
[32283.534945] .nr_spread_over : 958
[32283.534990] .nr_running : 0
[32283.535035] .load : 0
[32283.535080] .runnable_load_avg : 0
[32283.535125] .blocked_load_avg : 0
[32283.535170] .tg_load_contrib : 0
[32283.535215] .tg_runnable_contrib : 0
[32283.535260] .tg_load_avg : 0
[32283.535305] .tg->runnable_avg : 1
[32283.535350] .tg->cfs_bandwidth.timer_active: 0
[32283.535395] .throttled : 0
[32283.535440] .throttle_count : 0
[32283.535485] .avg->runnable_avg_sum : 22
[32283.535530] .avg->runnable_avg_period : 47853
[32283.535576]
[32283.535576] rt_rq[7]:/
[32283.535659] .rt_nr_running : 0
[32283.535704] .rt_throttled : 0
[32283.535749] .rt_time : 0.000000
[32283.535794] .rt_runtime : 950.000000
[32283.535841]
[32283.535841] runnable tasks:
[32283.535841] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[32283.535841] ----------------------------------------------------------------------------------------------------------
[32283.536106] watchdog/7 41 -8.980460 8079 0 -8.980460 61.102193 5.774683 /
[32283.536199] migration/7 42 0.000000 9550 0 0.000000 75.429195 0.000766 /
[32283.536293] ksoftirqd/7 43 104817.846671 4873 120 104817.846671 49.802593 32277564.697166 /
[32283.536387] kworker/7:0 44 215.098470 13 120 215.098470 0.035739 441.053790 /
[32283.536480] kworker/7:0H 45 594.137656 6 100 594.137656 0.024059 1001.557112 /
[32283.536574] khelper 46 10.956353 2 100 10.956353 0.005489 0.002367 /
[32283.536668] netns 47 22.959676 2 100 22.959676 0.003625 0.001963 /
[32283.536762] writeback 196 46.966466 2 100 46.966466 0.003836 0.001415 /
[32283.536856] bioset 198 58.969736 2 100 58.969736 0.003569 0.001472 /
[32283.536949] kblockd 199 70.972900 2 100 70.972900 0.003443 0.001370 /
[32283.537043] md 368 94.980182 2 100 94.980182 0.003410 0.001489 /
[32283.537137] kswapd0 517 215.099636 3 120 215.099636 0.020930 96.830888 /
[32283.537231] crypto 590 155.027174 2 100 155.027174 0.007760 0.002566 /
[32283.537325] kworker/7:1 626 104819.600303 65860 120 104819.600303 796.672835 32280293.735473 /
[32283.537419] kpsmoused 697 239.100497 2 100 239.100497 0.009007 0.006247 /
[32283.537513] ata_sff 878 437.761925 2 100 437.761925 0.009202 0.068803 /
[32283.537607] scsi_tmf_0 882 449.793617 2 100 449.793617 0.033058 0.022789 /
[32283.537701] scsi_tmf_1 884 473.885027 2 100 473.885027 0.031235 0.025914 /
[32283.537795] scsi_tmf_2 886 497.990516 2 100 497.990516 0.033468 0.023583 /
[32283.537889] scsi_tmf_3 888 522.051821 2 100 522.051821 0.038183 0.004172 /
[32283.537983] scsi_tmf_4 890 546.064187 2 100 546.064187 0.005894 0.004740 /
[32283.538077] scsi_tmf_5 892 570.075913 2 100 570.075913 0.006126 0.004100 /
[32283.538170] kworker/7:1H 919 104817.675019 12387 100 104817.675019 109.019366 32042117.133959 /
[32283.538265] bioset 945 641.086597 2 100 641.086597 0.005865 0.003763 /
[32283.538359] ext4-rsv-conver 981 678.326386 2 100 678.326386 0.012128 0.005728 /
[32283.538453] jbd2/md1-8 2274 1217.197271 2 120 1217.197271 0.028998 0.003545 /
[32283.538547] ext4-rsv-conver 2275 1229.234322 2 100 1229.234322 0.037719 0.015602 /
[32283.538640] bioset 2281 1253.276231 2 100 1253.276231 0.009828 0.022100 /
[32283.538736] getty 4199 0.389559 103 120 0.389559 1.955323 179.288425 /autogroup-62
[32283.538833]
--
Tomasz Chmielewski
http://wpkg.org
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2014-03-28 12:25 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-03-28 3:08 making snapshot kills RAID-1 rebuild? Tomasz Chmielewski
2014-03-28 12:25 ` Tomasz Chmielewski
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.