* [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps
@ 2015-06-15 7:02 Huang Ying
2015-06-15 17:52 ` Thomas Gleixner
0 siblings, 1 reply; 6+ messages in thread
From: Huang Ying @ 2015-06-15 7:02 UTC (permalink / raw)
To: Jiang Liu; +Cc: Thomas Gleixner, LKML, LKP ML
[-- Attachment #1: Type: text/plain, Size: 144987 bytes --]
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts")
testcase/path_params/tbox_group: netperf/performance-300s-200%-TCP_RR/brickland3
3cb96f0c97330834 52f518a3a7c2f80551a38d38be
---------------- --------------------------
%stddev %change %stddev
\ | \
1444 ± 1% -30.5% 1004 ± 1% netperf.Throughput_tps
899.43 ± 3% -91.6% 75.48 ± 3% netperf.time.user_time
10202 ± 0% -92.4% 778 ± 1% netperf.time.system_time
3565 ± 4% -92.5% 268 ± 3% netperf.time.percent_of_cpu_this_job_got
98265 ± 3% -31.1% 67662 ± 4% netperf.time.minor_page_faults
25540 ± 11% -88.3% 2991 ± 16% netperf.time.involuntary_context_switches
69135723 ± 1% -30.0% 48364754 ± 1% netperf.time.voluntary_context_switches
37347 ± 4% +13.1% 42256 ± 3% uptime.idle
44 ± 5% -93.2% 3 ± 0% vmstat.procs.r
407528 ± 3% -26.3% 300511 ± 4% vmstat.system.cs
446620 ± 4% -92.6% 33083 ± 6% vmstat.system.in
178635 ± 1% -14.6% 152519 ± 0% meminfo.Active
82997 ± 2% -29.5% 58518 ± 1% meminfo.Active(anon)
36056 ± 3% -63.2% 13281 ± 0% meminfo.Shmem
496435 ± 3% +19.4% 592640 ± 2% numa-meminfo.node0.MemUsed
480933 ± 3% -7.3% 445927 ± 2% numa-meminfo.node2.MemUsed
3997 ± 1% +12.1% 4482 ± 3% slabinfo.kmalloc-2048.active_objs
4183 ± 1% +12.2% 4694 ± 3% slabinfo.kmalloc-2048.num_objs
313 ± 15% -78.0% 69 ± 30% slabinfo.tw_sock_TCP.active_objs
313 ± 15% -78.0% 69 ± 30% slabinfo.tw_sock_TCP.num_objs
1016370 ± 1% -90.2% 99283 ± 1% softirqs.HRTIMER
1556902 ± 0% -84.0% 248959 ± 2% softirqs.RCU
3471142 ± 1% -92.3% 267496 ± 3% softirqs.SCHED
4270170 ± 1% -82.2% 758706 ± 2% softirqs.TIMER
2.347e+09 ± 6% -63.5% 8.569e+08 ± 39% cpuidle.C1-IVT-4S.time
22319046 ± 3% +62.6% 36299214 ± 3% cpuidle.C1-IVT-4S.usage
1.591e+10 ± 9% +128.3% 3.632e+10 ± 3% cpuidle.C6-IVT-4S.time
7 ± 21% +514.0% 44 ± 31% cpuidle.POLL.usage
25540 ± 11% -88.3% 2991 ± 16% time.involuntary_context_switches
98265 ± 3% -31.1% 67662 ± 4% time.minor_page_faults
3565 ± 4% -92.5% 268 ± 3% time.percent_of_cpu_this_job_got
10202 ± 0% -92.4% 778 ± 1% time.system_time
899.43 ± 3% -91.6% 75.48 ± 3% time.user_time
69135723 ± 1% -30.0% 48364754 ± 1% time.voluntary_context_switches
268764 ± 14% +408.8% 1367464 ± 3% numa-numastat.node0.numa_hit
268758 ± 14% +408.8% 1367424 ± 3% numa-numastat.node0.local_node
438574 ± 26% -45.0% 241074 ± 2% numa-numastat.node1.local_node
438574 ± 26% -45.0% 241076 ± 2% numa-numastat.node1.numa_hit
333989 ± 23% -27.9% 240841 ± 3% numa-numastat.node2.local_node
333989 ± 23% -27.9% 240844 ± 3% numa-numastat.node2.numa_hit
381233 ± 26% -35.6% 245556 ± 3% numa-numastat.node3.local_node
381238 ± 26% -35.6% 245561 ± 3% numa-numastat.node3.numa_hit
482 ± 4% +11.9% 540 ± 1% numa-vmstat.node0.nr_alloc_batch
251510 ± 9% +212.0% 784815 ± 8% numa-vmstat.node0.numa_local
289771 ± 8% +184.0% 823025 ± 7% numa-vmstat.node0.numa_hit
454081 ± 12% -25.4% 338719 ± 4% numa-vmstat.node1.numa_hit
521 ± 1% +13.5% 592 ± 3% numa-vmstat.node1.nr_alloc_batch
395201 ± 14% -28.9% 281018 ± 4% numa-vmstat.node1.numa_local
529 ± 3% +10.1% 583 ± 1% numa-vmstat.node2.nr_alloc_batch
525 ± 6% +12.8% 593 ± 1% numa-vmstat.node3.nr_alloc_batch
51.25 ± 3% -92.9% 3.65 ± 3% turbostat.%Busy
1635 ± 3% -93.0% 114 ± 3% turbostat.Avg_MHz
45.20 ± 4% -70.1% 13.52 ± 21% turbostat.CPU%c1
357.80 ± 3% -71.5% 101.83 ± 6% turbostat.CorWatt
53 ± 2% -21.2% 42 ± 6% turbostat.CoreTmp
53 ± 1% -20.7% 42 ± 6% turbostat.PkgTmp
424 ± 2% -61.0% 165 ± 3% turbostat.PkgWatt
62.78 ± 1% -8.9% 57.17 ± 3% turbostat.RAMWatt
20746 ± 2% -29.5% 14628 ± 1% proc-vmstat.nr_active_anon
2024 ± 1% +13.4% 2295 ± 0% proc-vmstat.nr_alloc_batch
9013 ± 3% -63.2% 3319 ± 0% proc-vmstat.nr_shmem
49576 ± 5% -60.3% 19667 ± 13% proc-vmstat.numa_hint_faults
44013 ± 5% -64.6% 15598 ± 16% proc-vmstat.numa_hint_faults_local
1420966 ± 2% +47.3% 2092933 ± 1% proc-vmstat.numa_hit
1420955 ± 2% +47.3% 2092883 ± 1% proc-vmstat.numa_local
53703 ± 5% -57.9% 22601 ± 13% proc-vmstat.numa_pte_updates
8470 ± 1% -81.8% 1543 ± 3% proc-vmstat.pgactivate
162499 ± 5% +90.3% 309245 ± 5% proc-vmstat.pgalloc_dma32
5918865 ± 2% +85.7% 10992818 ± 3% proc-vmstat.pgalloc_normal
6085763 ± 2% +86.0% 11317294 ± 3% proc-vmstat.pgfree
2579 ± 5% -79.4% 532 ± 2% latency_stats.avg.do_wait.SyS_wait4.system_call_fastpath
858 ± 5% +60.1% 1375 ± 4% latency_stats.avg.cifs_user_readv.[cifs].cifs_strict_readv.[cifs].__vfs_read.vfs_read.SyS_read.system_call_fastpath
2337 ± 4% -79.5% 479 ± 2% latency_stats.avg.pipe_wait.pipe_read.__vfs_read.vfs_read.SyS_read.system_call_fastpath
875 ± 7% +74.6% 1528 ± 10% latency_stats.avg.wait_for_response.[cifs].SendReceive.[cifs].CIFSPOSIXCreate.[cifs].cifs_posix_open.[cifs].cifs_open.[cifs].do_dentry_open.vfs_open.do_last.path_openat.do_filp_open.do_sys_open.SyS_openat
455 ± 1% +108.8% 951 ± 1% latency_stats.avg.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath
541 ± 14% +137.6% 1287 ± 4% latency_stats.avg.wait_for_response.[cifs].SendReceive.[cifs].CIFSSMBUnixQPathInfo.[cifs].cifs_get_inode_info_unix.[cifs].cifs_revalidate_dentry_attr.[cifs].cifs_revalidate_dentry.[cifs].cifs_d_revalidate.[cifs].lookup_fast.do_last.path_openat.do_filp_open.do_sys_open
655 ± 20% +96.9% 1291 ± 9% latency_stats.avg.poll_schedule_timeout.do_select.core_sys_select.SyS_select.system_call_fastpath
69119790 ± 1% -30.0% 48351473 ± 1% latency_stats.hits.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath
443 ± 3% -10.0% 399 ± 2% latency_stats.hits.poll_schedule_timeout.do_select.core_sys_select.SyS_select.system_call_fastpath
190 ± 6% -95.8% 8 ± 38% latency_stats.hits.stop_one_cpu.sched_exec.do_execveat_common.SyS_execve.return_from_execve
2157 ± 23% +66.0% 3581 ± 22% latency_stats.max.cifs_user_readv.[cifs].cifs_strict_readv.[cifs].__vfs_read.vfs_read.SyS_read.system_call_fastpath
4912 ± 0% +1.6% 4992 ± 0% latency_stats.max.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath
3.151e+10 ± 0% +46.1% 4.603e+10 ± 0% latency_stats.sum.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath
16780629 ± 2% -78.9% 3539898 ± 4% latency_stats.sum.do_wait.SyS_wait4.system_call_fastpath
25402986 ± 2% -78.5% 5473439 ± 4% latency_stats.sum.pipe_wait.pipe_read.__vfs_read.vfs_read.SyS_read.system_call_fastpath
12072 ± 31% +112.0% 25588 ± 10% latency_stats.sum.wait_for_response.[cifs].SendReceive.[cifs].CIFSSMBUnixQPathInfo.[cifs].cifs_get_inode_info_unix.[cifs].cifs_revalidate_dentry_attr.[cifs].cifs_revalidate_dentry.[cifs].cifs_d_revalidate.[cifs].lookup_fast.do_last.path_openat.do_filp_open.do_sys_open
290547 ± 19% +77.5% 515581 ± 9% latency_stats.sum.poll_schedule_timeout.do_select.core_sys_select.SyS_select.system_call_fastpath
24407 ± 9% +74.1% 42503 ± 9% latency_stats.sum.wait_for_response.[cifs].SendReceive.[cifs].CIFSPOSIXCreate.[cifs].cifs_posix_open.[cifs].cifs_open.[cifs].do_dentry_open.vfs_open.do_last.path_openat.do_filp_open.do_sys_open.SyS_openat
245753 ± 23% +44.5% 355113 ± 9% latency_stats.sum.cifs_user_readv.[cifs].cifs_strict_readv.[cifs].__vfs_read.vfs_read.SyS_read.system_call_fastpath
17.85 ± 4% -16.4% 14.92 ± 2% perf-profile.cpu-cycles.ip_rcv.__netif_receive_skb_core.__netif_receive_skb.netif_receive_skb_internal.napi_gro_receive
0.97 ± 4% -85.4% 0.14 ± 32% perf-profile.cpu-cycles.ktime_get.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle
2.19 ± 8% -93.6% 0.14 ± 10% perf-profile.cpu-cycles._raw_spin_lock.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common
3.88 ± 9% -94.1% 0.23 ± 21% perf-profile.cpu-cycles.load_balance.pick_next_task_fair.__schedule.schedule.schedule_timeout
6.29 ± 7% -96.7% 0.21 ± 40% perf-profile.cpu-cycles._raw_spin_lock_irqsave.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.enqueue_task
0.94 ± 5% -90.7% 0.09 ± 13% perf-profile.cpu-cycles.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__schedule.schedule
0.94 ± 5% -90.6% 0.09 ± 13% perf-profile.cpu-cycles.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__schedule
4.19 ± 2% +61.7% 6.77 ± 3% perf-profile.cpu-cycles.ktime_get.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry.start_secondary
15.10 ± 3% -27.5% 10.94 ± 1% perf-profile.cpu-cycles.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.sys_recvfrom.system_call_fastpath
15.19 ± 3% -27.2% 11.07 ± 1% perf-profile.cpu-cycles.sock_recvmsg.SYSC_recvfrom.sys_recvfrom.system_call_fastpath
15.35 ± 3% -27.0% 11.20 ± 1% perf-profile.cpu-cycles.SYSC_recvfrom.sys_recvfrom.system_call_fastpath
15.44 ± 3% -27.0% 11.27 ± 1% perf-profile.cpu-cycles.sys_recvfrom.system_call_fastpath
14.72 ± 3% -26.5% 10.82 ± 1% perf-profile.cpu-cycles.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.sys_recvfrom
1.57 ± 3% +33.4% 2.10 ± 4% perf-profile.cpu-cycles.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg
15.03 ± 4% -25.0% 11.27 ± 2% perf-profile.cpu-cycles.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key
23.41 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.ret_from_intr.cpu_startup_entry.start_secondary
1.19 ± 12% -80.5% 0.23 ± 19% perf-profile.cpu-cycles.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__run_hrtimer
4.24 ± 2% +58.3% 6.72 ± 3% perf-profile.cpu-cycles.read_hpet.ktime_get.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry
15.05 ± 4% -24.9% 11.30 ± 2% perf-profile.cpu-cycles.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.tcp_prequeue
2.81 ± 9% -90.6% 0.26 ± 20% perf-profile.cpu-cycles.find_busiest_group.load_balance.pick_next_task_fair.__schedule.schedule
9.12 ± 4% -52.1% 4.36 ± 4% perf-profile.cpu-cycles.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom
0.46 ± 2% +127.4% 1.04 ± 6% perf-profile.cpu-cycles.pick_next_task_fair.__schedule.schedule.schedule_preempt_disabled.cpu_startup_entry
2.52 ± 2% -62.9% 0.94 ± 31% perf-profile.cpu-cycles.__run_hrtimer.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
1.58 ± 8% -81.7% 0.29 ± 21% perf-profile.cpu-cycles.tick_sched_handle.isra.18.tick_sched_timer.__run_hrtimer.hrtimer_interrupt.local_apic_timer_interrupt
2.49 ± 1% -64.0% 0.90 ± 33% perf-profile.cpu-cycles.tick_sched_timer.__run_hrtimer.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt
15.06 ± 4% -24.9% 11.32 ± 2% perf-profile.cpu-cycles.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv
1.35 ± 3% +32.3% 1.79 ± 8% perf-profile.cpu-cycles.deactivate_task.__schedule.schedule.schedule_timeout.sk_wait_data
23.23 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.do_IRQ.ret_from_intr.cpu_startup_entry.start_secondary
5.90 ± 2% -23.8% 4.50 ± 6% perf-profile.cpu-cycles.read_hpet.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle
1.72 ± 8% -83.2% 0.29 ± 21% perf-profile.cpu-cycles.update_process_times.tick_sched_handle.tick_sched_timer.__run_hrtimer.hrtimer_interrupt
1.55 ± 6% +48.4% 2.31 ± 5% perf-profile.cpu-cycles.dev_queue_xmit_sk.ip_finish_output.ip_output.ip_local_out_sk.ip_queue_xmit
1.35 ± 3% +30.7% 1.76 ± 7% perf-profile.cpu-cycles.dequeue_task.deactivate_task.__schedule.schedule.schedule_timeout
15.75 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_exit.do_IRQ.ret_from_intr.cpu_startup_entry.start_secondary
1.54 ± 5% +48.3% 2.28 ± 5% perf-profile.cpu-cycles.__dev_queue_xmit.dev_queue_xmit_sk.ip_finish_output.ip_output.ip_local_out_sk
3.70 ± 1% +49.0% 5.51 ± 1% perf-profile.cpu-cycles.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg.sock_recvmsg
72.13 ± 0% -24.2% 54.66 ± 0% perf-profile.cpu-cycles.cpu_startup_entry.start_secondary
8.28 ± 5% -61.0% 3.23 ± 5% perf-profile.cpu-cycles.schedule.schedule_timeout.sk_wait_data.tcp_recvmsg.inet_recvmsg
3.44 ± 3% +51.2% 5.19 ± 1% perf-profile.cpu-cycles.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg
2.75 ± 9% -91.1% 0.24 ± 20% perf-profile.cpu-cycles.update_sd_lb_stats.find_busiest_group.load_balance.pick_next_task_fair.__schedule
8.36 ± 5% -61.0% 3.26 ± 5% perf-profile.cpu-cycles.schedule_timeout.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg
0.33 ± 29% +218.5% 1.06 ± 10% perf-profile.cpu-cycles.sk_stream_alloc_skb.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto
15.13 ± 4% -24.2% 11.46 ± 2% perf-profile.cpu-cycles.__wake_up_common.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish
72.31 ± 0% -24.2% 54.83 ± 0% perf-profile.cpu-cycles.start_secondary
14.20 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.__do_softirq.irq_exit.do_IRQ.ret_from_intr.cpu_startup_entry
1.30 ± 3% +24.4% 1.62 ± 8% perf-profile.cpu-cycles.dequeue_task_fair.dequeue_task.deactivate_task.__schedule.schedule
1.34 ± 5% +30.0% 1.74 ± 4% perf-profile.cpu-cycles.sch_direct_xmit.__dev_queue_xmit.dev_queue_xmit_sk.ip_finish_output.ip_output
8.93 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.ret_from_intr.cpuidle_enter.cpu_startup_entry.start_secondary
15.18 ± 1% +26.1% 19.14 ± 3% perf-profile.cpu-cycles.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
4.09 ± 0% +9.8% 4.49 ± 4% perf-profile.cpu-cycles.ktime_get.clockevents_program_event.tick_program_event.__remove_hrtimer.__hrtimer_start_range_ns
8.08 ± 5% -62.1% 3.06 ± 6% perf-profile.cpu-cycles.__schedule.schedule.schedule_timeout.sk_wait_data.tcp_recvmsg
3.87 ± 2% +12.2% 4.35 ± 7% perf-profile.cpu-cycles.read_hpet.ktime_get.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry
5.80 ± 1% -12.5% 5.07 ± 5% perf-profile.cpu-cycles.ktime_get_with_offset.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll
0.36 ± 4% +188.3% 1.02 ± 5% perf-profile.cpu-cycles.dev_hard_start_xmit.sch_direct_xmit.__dev_queue_xmit.dev_queue_xmit_sk.ip_finish_output
4.72 ± 3% +186.8% 13.52 ± 6% perf-profile.cpu-cycles.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry.start_secondary
10.92 ± 4% -11.1% 9.71 ± 4% perf-profile.cpu-cycles.enqueue_entity.enqueue_task_fair.enqueue_task.activate_task.ttwu_do_activate
2.95 ± 3% -18.6% 2.40 ± 3% perf-profile.cpu-cycles.__schedule.schedule.schedule_preempt_disabled.cpu_startup_entry.start_secondary
3.83 ± 2% +14.0% 4.36 ± 7% perf-profile.cpu-cycles.ktime_get.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry.start_secondary
8.92 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.do_IRQ.ret_from_intr.cpuidle_enter.cpu_startup_entry.start_secondary
7.38 ± 1% +13.6% 8.38 ± 2% perf-profile.cpu-cycles.sock_sendmsg.SYSC_sendto.sys_sendto.system_call_fastpath
11.19 ± 4% -11.1% 9.95 ± 2% perf-profile.cpu-cycles.ttwu_do_activate.constprop.88.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common
15.76 ± 4% -20.9% 12.46 ± 3% perf-profile.cpu-cycles.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
7.73 ± 2% +14.4% 8.85 ± 2% perf-profile.cpu-cycles.SYSC_sendto.sys_sendto.system_call_fastpath
13.61 ± 1% +12.6% 15.31 ± 3% perf-profile.cpu-cycles.tick_nohz_idle_enter.cpu_startup_entry.start_secondary
2.97 ± 2% +14.2% 3.40 ± 4% perf-profile.cpu-cycles.ip_local_out_sk.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
7.79 ± 2% +14.7% 8.93 ± 2% perf-profile.cpu-cycles.sys_sendto.system_call_fastpath
4.63 ± 1% +24.6% 5.77 ± 1% perf-profile.cpu-cycles.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom
2.87 ± 2% +13.8% 3.26 ± 3% perf-profile.cpu-cycles.ip_output.ip_local_out_sk.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit
6.59 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_exit.do_IRQ.ret_from_intr.cpuidle_enter.cpu_startup_entry
3.04 ± 3% -17.9% 2.49 ± 3% perf-profile.cpu-cycles.schedule.schedule_preempt_disabled.cpu_startup_entry.start_secondary
11.17 ± 4% -18.6% 9.09 ± 2% perf-profile.cpu-cycles.enqueue_task_fair.enqueue_task.activate_task.ttwu_do_activate.try_to_wake_up
23.39 ± 2% -12.3% 20.51 ± 1% perf-profile.cpu-cycles.system_call_fastpath
7.03 ± 1% +16.0% 8.15 ± 2% perf-profile.cpu-cycles.inet_sendmsg.sock_sendmsg.SYSC_sendto.sys_sendto.system_call_fastpath
4.16 ± 0% +14.6% 4.76 ± 4% perf-profile.cpu-cycles.tick_program_event.__remove_hrtimer.__hrtimer_start_range_ns.hrtimer_start.tick_nohz_stop_sched_tick
4.15 ± 0% +14.7% 4.76 ± 4% perf-profile.cpu-cycles.clockevents_program_event.tick_program_event.__remove_hrtimer.__hrtimer_start_range_ns.hrtimer_start
3.05 ± 3% -16.3% 2.55 ± 3% perf-profile.cpu-cycles.schedule_preempt_disabled.cpu_startup_entry.start_secondary
6.33 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.__do_softirq.irq_exit.do_IRQ.ret_from_intr.cpuidle_enter
6.78 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr.cpu_startup_entry
6.75 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_enter.do_IRQ.ret_from_intr.cpu_startup_entry.start_secondary
4.73 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_irq_enter.irq_enter
10.95 ± 5% -13.5% 9.47 ± 2% perf-profile.cpu-cycles.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function.autoremove_wake_function
4.62 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_irq_enter
4.21 ± 1% +21.2% 5.10 ± 4% perf-profile.cpu-cycles.ktime_get.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
1.29 ± 4% +21.1% 1.57 ± 3% perf-profile.cpu-cycles.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process
9.15 ± 1% +16.1% 10.63 ± 2% perf-profile.cpu-cycles.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry.start_secondary
10.94 ± 5% -13.5% 9.46 ± 2% perf-profile.cpu-cycles.enqueue_task.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function
0.17 ± 35% +479.8% 0.96 ± 10% perf-profile.cpu-cycles.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg.inet_sendmsg.sock_sendmsg
4.32 ± 9% -88.6% 0.49 ± 13% perf-profile.cpu-cycles.pick_next_task_fair.__schedule.schedule.schedule_timeout.sk_wait_data
4.54 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_nohz_stop_idle.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr
16.82 ± 4% -15.0% 14.30 ± 3% perf-profile.cpu-cycles.ip_local_deliver.ip_rcv_finish.ip_rcv.__netif_receive_skb_core.__netif_receive_skb
16.81 ± 4% -15.0% 14.28 ± 3% perf-profile.cpu-cycles.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish.ip_rcv.__netif_receive_skb_core
4.28 ± 2% -14.4% 3.66 ± 6% perf-profile.cpu-cycles.ktime_get.clockevents_program_event.tick_program_event.__remove_hrtimer.hrtimer_try_to_cancel
24.19 ± 3% -15.8% 20.36 ± 3% perf-profile.cpu-cycles.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll.net_rx_action.__do_softirq
23.93 ± 3% -15.8% 20.15 ± 3% perf-profile.cpu-cycles.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll.net_rx_action
4.03 ± 1% +15.2% 4.64 ± 3% perf-profile.cpu-cycles.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_push.tcp_sendmsg
9.82 ± 4% -14.5% 8.40 ± 3% perf-profile.cpu-cycles.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.enqueue_task.activate_task
13.12 ± 1% +15.9% 15.21 ± 3% perf-profile.cpu-cycles.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry.start_secondary
6.71 ± 1% +18.5% 7.95 ± 2% perf-profile.cpu-cycles.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto.sys_sendto
8.57 ± 1% +17.2% 10.05 ± 3% perf-profile.cpu-cycles.hrtimer_start.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry
16.71 ± 4% -16.1% 14.02 ± 3% perf-profile.cpu-cycles.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish.ip_rcv
0.14 ± 15% +573.8% 0.94 ± 9% perf-profile.cpu-cycles.tcp_send_delayed_ack.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process
0.10 ± 23% +830.0% 0.93 ± 5% perf-profile.cpu-cycles.ixgbe_xmit_frame.dev_hard_start_xmit.sch_direct_xmit.__dev_queue_xmit.dev_queue_xmit_sk
0.08 ± 6% +1213.0% 1.01 ± 4% perf-profile.cpu-cycles.is_module_text_address.print_context_stack.dump_trace.save_stack_trace_tsk.__account_scheduler_latency
4.48 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_irq_enter.irq_enter.do_IRQ
4.57 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.ktime_get.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr
8.57 ± 1% +16.9% 10.02 ± 3% perf-profile.cpu-cycles.__hrtimer_start_range_ns.hrtimer_start.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.tick_nohz_idle_enter
17.72 ± 4% -16.3% 14.84 ± 2% perf-profile.cpu-cycles.ip_rcv_finish.ip_rcv.__netif_receive_skb_core.__netif_receive_skb.netif_receive_skb_internal
4.57 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.read_hpet.ktime_get.tick_irq_enter.irq_enter.do_IRQ
0.96 ± 4% -85.4% 0.14 ± 35% perf-profile.cpu-cycles.read_hpet.ktime_get.sched_clock_tick.scheduler_tick.update_process_times
18.03 ± 4% -16.6% 15.03 ± 2% perf-profile.cpu-cycles.__netif_receive_skb.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll
4.99 ± 1% +17.0% 5.84 ± 2% perf-profile.cpu-cycles.tcp_push.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto
18.00 ± 4% -16.7% 15.00 ± 2% perf-profile.cpu-cycles.__netif_receive_skb_core.__netif_receive_skb.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq
3.16 ± 1% +20.8% 3.82 ± 3% perf-profile.cpu-cycles.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_push
4.87 ± 2% +18.7% 5.78 ± 2% perf-profile.cpu-cycles.__tcp_push_pending_frames.tcp_push.tcp_sendmsg.inet_sendmsg.sock_sendmsg
2.36 ± 1% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_enter.do_IRQ.ret_from_intr.cpuidle_enter.cpu_startup_entry
0.23 ± 6% +443.4% 1.23 ± 3% perf-profile.cpu-cycles.is_module_text_address.__kernel_text_address.print_context_stack.dump_trace.save_stack_trace_tsk
0.47 ± 4% +684.2% 3.65 ± 3% perf-profile.cpu-cycles.__kernel_text_address.print_context_stack.dump_trace.save_stack_trace_tsk.__account_scheduler_latency
1.37 ± 4% +239.8% 4.65 ± 7% perf-profile.cpu-cycles.tick_nohz_stop_idle.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
1.34 ± 4% +243.2% 4.61 ± 7% perf-profile.cpu-cycles.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_nohz_idle_exit.cpu_startup_entry
0.43 ± 7% +1444.2% 6.64 ± 15% perf-profile.cpu-cycles.intel_idle.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry.start_secondary
0.54 ± 7% +3863.4% 21.27 ± 3% perf-profile.cpu-cycles.ixgbe_clean_rx_irq.ixgbe_poll.net_rx_action.__do_softirq.run_ksoftirqd
0.54 ± 7% +4049.4% 22.41 ± 2% perf-profile.cpu-cycles.ixgbe_poll.net_rx_action.__do_softirq.run_ksoftirqd.smpboot_thread_fn
0.55 ± 6% +3975.2% 22.48 ± 2% perf-profile.cpu-cycles.net_rx_action.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread
1.28 ± 3% +246.7% 4.43 ± 7% perf-profile.cpu-cycles.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_nohz_idle_exit
4.21 ± 0% +18.0% 4.97 ± 3% perf-profile.cpu-cycles.__remove_hrtimer.__hrtimer_start_range_ns.hrtimer_start.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter
5.58 ± 1% -15.9% 4.69 ± 3% perf-profile.cpu-cycles.read_hpet.ktime_get_with_offset.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq
4.77 ± 1% +19.6% 5.70 ± 2% perf-profile.cpu-cycles.tcp_write_xmit.__tcp_push_pending_frames.tcp_push.tcp_sendmsg.inet_sendmsg
1.33 ± 4% +248.1% 4.62 ± 7% perf-profile.cpu-cycles.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
1.92 ± 2% +283.1% 7.36 ± 2% perf-profile.cpu-cycles.dump_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
1.93 ± 2% +285.6% 7.46 ± 2% perf-profile.cpu-cycles.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.enqueue_task
2.20 ± 3% +19.0% 2.61 ± 4% perf-profile.cpu-cycles.ip_finish_output.ip_output.ip_local_out_sk.ip_queue_xmit.tcp_transmit_skb
16.33 ± 4% -17.9% 13.40 ± 3% perf-profile.cpu-cycles.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish
0.99 ± 4% +2179.8% 22.61 ± 2% perf-profile.cpu-cycles.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
0.97 ± 4% +2217.9% 22.60 ± 2% perf-profile.cpu-cycles.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
1.13 ± 3% +1906.2% 22.67 ± 2% perf-profile.cpu-cycles.smpboot_thread_fn.kthread.ret_from_fork
1.15 ± 2% +1878.6% 22.82 ± 2% perf-profile.cpu-cycles.kthread.ret_from_fork
1.02 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.x86_64_start_reservations.x86_64_start_kernel
1.15 ± 2% +1879.0% 22.82 ± 2% perf-profile.cpu-cycles.ret_from_fork
2.34 ± 1% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr.cpuidle_enter
1.62 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit.do_IRQ.ret_from_intr
1.61 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.read_hpet.ktime_get.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit
1.56 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_nohz_irq_exit.irq_exit.do_IRQ.ret_from_intr.cpu_startup_entry
1.48 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.ktime_get.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit.do_IRQ
0.21 ± 12% +351.6% 0.96 ± 8% perf-profile.cpu-cycles.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg
1.72 ± 2% +301.4% 6.90 ± 2% perf-profile.cpu-cycles.print_context_stack.dump_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity
1.02 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.x86_64_start_kernel
1.02 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
1.02 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.start_kernel.x86_64_start_reservations.x86_64_start_kernel
4.21 ± 1% +20.6% 5.08 ± 4% perf-profile.cpu-cycles.read_hpet.ktime_get.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
528 ± 6% +61.7% 854 ± 1% sched_debug.cfs_rq[0]:/.tg_runnable_contrib
24162 ± 6% +62.3% 39214 ± 1% sched_debug.cfs_rq[0]:/.avg->runnable_avg_sum
4882658 ± 9% -96.6% 167048 ± 8% sched_debug.cfs_rq[0]:/.min_vruntime
37303 ± 1% -89.7% 3826 ± 4% sched_debug.cfs_rq[0]:/.tg->runnable_avg
77353 ± 9% +81.9% 140694 ± 7% sched_debug.cfs_rq[0]:/.exec_clock
38349 ± 2% -89.4% 4070 ± 3% sched_debug.cfs_rq[100]:/.tg->runnable_avg
38365 ± 2% -89.4% 4071 ± 3% sched_debug.cfs_rq[101]:/.tg->runnable_avg
38369 ± 2% -89.4% 4074 ± 3% sched_debug.cfs_rq[102]:/.tg->runnable_avg
38382 ± 2% -89.4% 4075 ± 3% sched_debug.cfs_rq[103]:/.tg->runnable_avg
38383 ± 2% -89.4% 4077 ± 3% sched_debug.cfs_rq[104]:/.tg->runnable_avg
38391 ± 2% -89.4% 4079 ± 3% sched_debug.cfs_rq[105]:/.tg->runnable_avg
38388 ± 2% -89.4% 4083 ± 3% sched_debug.cfs_rq[106]:/.tg->runnable_avg
38402 ± 2% -89.4% 4084 ± 3% sched_debug.cfs_rq[107]:/.tg->runnable_avg
38401 ± 2% -89.4% 4087 ± 3% sched_debug.cfs_rq[108]:/.tg->runnable_avg
38413 ± 2% -89.4% 4088 ± 3% sched_debug.cfs_rq[109]:/.tg->runnable_avg
548 ± 3% -93.4% 36 ± 22% sched_debug.cfs_rq[10]:/.tg_runnable_contrib
37450 ± 1% -89.7% 3851 ± 4% sched_debug.cfs_rq[10]:/.tg->runnable_avg
5772463 ± 8% -98.4% 93465 ± 22% sched_debug.cfs_rq[10]:/.min_vruntime
76594 ± 8% -91.7% 6366 ± 15% sched_debug.cfs_rq[10]:/.exec_clock
25036 ± 3% -93.3% 1688 ± 20% sched_debug.cfs_rq[10]:/.avg->runnable_avg_sum
38410 ± 2% -89.4% 4089 ± 3% sched_debug.cfs_rq[110]:/.tg->runnable_avg
38417 ± 2% -89.4% 4088 ± 3% sched_debug.cfs_rq[111]:/.tg->runnable_avg
38435 ± 2% -89.4% 4091 ± 3% sched_debug.cfs_rq[112]:/.tg->runnable_avg
38435 ± 2% -89.4% 4092 ± 3% sched_debug.cfs_rq[113]:/.tg->runnable_avg
38428 ± 2% -89.3% 4095 ± 3% sched_debug.cfs_rq[114]:/.tg->runnable_avg
38439 ± 1% -89.3% 4096 ± 3% sched_debug.cfs_rq[115]:/.tg->runnable_avg
38445 ± 2% -89.3% 4095 ± 3% sched_debug.cfs_rq[116]:/.tg->runnable_avg
38449 ± 2% -89.3% 4097 ± 3% sched_debug.cfs_rq[117]:/.tg->runnable_avg
38461 ± 2% -89.3% 4100 ± 3% sched_debug.cfs_rq[118]:/.tg->runnable_avg
38460 ± 1% -89.3% 4102 ± 3% sched_debug.cfs_rq[119]:/.tg->runnable_avg
76608 ± 7% -91.7% 6356 ± 16% sched_debug.cfs_rq[11]:/.exec_clock
546 ± 2% -93.5% 35 ± 25% sched_debug.cfs_rq[11]:/.tg_runnable_contrib
37465 ± 1% -89.7% 3852 ± 4% sched_debug.cfs_rq[11]:/.tg->runnable_avg
24985 ± 2% -93.3% 1664 ± 24% sched_debug.cfs_rq[11]:/.avg->runnable_avg_sum
5775629 ± 7% -98.4% 91917 ± 12% sched_debug.cfs_rq[11]:/.min_vruntime
76166 ± 8% -92.2% 5923 ± 22% sched_debug.cfs_rq[12]:/.exec_clock
5756491 ± 8% -98.5% 86026 ± 20% sched_debug.cfs_rq[12]:/.min_vruntime
5 ± 31% -100.0% 0 ± 0% sched_debug.cfs_rq[12]:/.runnable_load_avg
37492 ± 1% -89.7% 3853 ± 4% sched_debug.cfs_rq[12]:/.tg->runnable_avg
75892 ± 8% -91.8% 6211 ± 19% sched_debug.cfs_rq[13]:/.exec_clock
37506 ± 1% -89.7% 3856 ± 4% sched_debug.cfs_rq[13]:/.tg->runnable_avg
5727275 ± 7% -98.4% 93734 ± 18% sched_debug.cfs_rq[13]:/.min_vruntime
37528 ± 1% -89.7% 3857 ± 4% sched_debug.cfs_rq[14]:/.tg->runnable_avg
25080 ± 5% -94.4% 1394 ± 43% sched_debug.cfs_rq[14]:/.avg->runnable_avg_sum
75791 ± 8% -92.2% 5907 ± 24% sched_debug.cfs_rq[14]:/.exec_clock
550 ± 4% -94.6% 29 ± 43% sched_debug.cfs_rq[14]:/.tg_runnable_contrib
6 ± 34% -100.0% 0 ± 0% sched_debug.cfs_rq[14]:/.runnable_load_avg
5717919 ± 7% -98.4% 90360 ± 22% sched_debug.cfs_rq[14]:/.min_vruntime
37554 ± 1% -89.7% 3860 ± 4% sched_debug.cfs_rq[15]:/.tg->runnable_avg
71299 ± 11% -96.4% 2537 ± 18% sched_debug.cfs_rq[15]:/.exec_clock
5471648 ± 11% -99.1% 50259 ± 22% sched_debug.cfs_rq[15]:/.min_vruntime
71273 ± 12% -96.7% 2353 ± 10% sched_debug.cfs_rq[16]:/.exec_clock
5469742 ± 11% -99.0% 53306 ± 24% sched_debug.cfs_rq[16]:/.min_vruntime
37563 ± 1% -89.7% 3864 ± 4% sched_debug.cfs_rq[16]:/.tg->runnable_avg
2807093.70 ± 36% -100.0% 0.00 ± 0% sched_debug.cfs_rq[17]:/.MIN_vruntime
6 ± 31% -100.0% 0 ± 0% sched_debug.cfs_rq[17]:/.runnable_load_avg
37568 ± 1% -89.7% 3865 ± 4% sched_debug.cfs_rq[17]:/.tg->runnable_avg
71654 ± 12% -96.9% 2229 ± 14% sched_debug.cfs_rq[17]:/.exec_clock
5474236 ± 11% -99.1% 50364 ± 25% sched_debug.cfs_rq[17]:/.min_vruntime
2807093.70 ± 36% -100.0% 0.00 ± 0% sched_debug.cfs_rq[17]:/.max_vruntime
37574 ± 1% -89.7% 3866 ± 4% sched_debug.cfs_rq[18]:/.tg->runnable_avg
2611368.18 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[18]:/.max_vruntime
2611367.85 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[18]:/.MIN_vruntime
5469964 ± 11% -99.2% 42263 ± 36% sched_debug.cfs_rq[18]:/.min_vruntime
4 ± 28% -100.0% 0 ± 0% sched_debug.cfs_rq[19]:/.runnable_load_avg
5460827 ± 11% -99.2% 46250 ± 33% sched_debug.cfs_rq[19]:/.min_vruntime
37603 ± 1% -89.7% 3871 ± 4% sched_debug.cfs_rq[19]:/.tg->runnable_avg
5931126 ± 8% -86.0% 829332 ± 28% sched_debug.cfs_rq[1]:/.min_vruntime
37333 ± 1% -89.7% 3834 ± 4% sched_debug.cfs_rq[1]:/.tg->runnable_avg
1048433 ± 16% -36.8% 662283 ± 34% sched_debug.cfs_rq[1]:/.spread0
5501605 ± 11% -99.1% 46820 ± 29% sched_debug.cfs_rq[20]:/.min_vruntime
71374 ± 11% -97.4% 1825 ± 22% sched_debug.cfs_rq[20]:/.exec_clock
37630 ± 1% -89.7% 3876 ± 4% sched_debug.cfs_rq[20]:/.tg->runnable_avg
4 ± 49% -100.0% 0 ± 0% sched_debug.cfs_rq[20]:/.runnable_load_avg
5435674 ± 11% -99.1% 48697 ± 31% sched_debug.cfs_rq[21]:/.min_vruntime
37645 ± 1% -89.7% 3875 ± 4% sched_debug.cfs_rq[21]:/.tg->runnable_avg
7 ± 40% -100.0% 0 ± 0% sched_debug.cfs_rq[21]:/.load
122 ± 48% -100.0% 0 ± 0% sched_debug.cfs_rq[21]:/.utilization_load_avg
70205 ± 12% -97.3% 1867 ± 36% sched_debug.cfs_rq[21]:/.exec_clock
7 ± 44% -100.0% 0 ± 0% sched_debug.cfs_rq[22]:/.load
37661 ± 1% -89.7% 3876 ± 4% sched_debug.cfs_rq[22]:/.tg->runnable_avg
5491828 ± 11% -99.2% 45143 ± 26% sched_debug.cfs_rq[22]:/.min_vruntime
6 ± 32% -100.0% 0 ± 0% sched_debug.cfs_rq[23]:/.load
4 ± 35% -100.0% 0 ± 0% sched_debug.cfs_rq[23]:/.runnable_load_avg
90 ± 40% -100.0% 0 ± 0% sched_debug.cfs_rq[23]:/.utilization_load_avg
37652 ± 1% -89.7% 3879 ± 4% sched_debug.cfs_rq[23]:/.tg->runnable_avg
5499468 ± 11% -99.2% 44829 ± 30% sched_debug.cfs_rq[23]:/.min_vruntime
37651 ± 1% -89.7% 3882 ± 4% sched_debug.cfs_rq[24]:/.tg->runnable_avg
7 ± 47% -100.0% 0 ± 0% sched_debug.cfs_rq[24]:/.load
2002338.86 ± 45% -100.0% 0.00 ± 0% sched_debug.cfs_rq[24]:/.MIN_vruntime
5458480 ± 11% -99.2% 45851 ± 22% sched_debug.cfs_rq[24]:/.min_vruntime
2002338.86 ± 45% -100.0% 0.00 ± 0% sched_debug.cfs_rq[24]:/.max_vruntime
499 ± 11% -98.4% 7 ± 49% sched_debug.cfs_rq[24]:/.tg_runnable_contrib
22791 ± 11% -98.3% 393 ± 47% sched_debug.cfs_rq[24]:/.avg->runnable_avg_sum
5501548 ± 11% -99.2% 42017 ± 39% sched_debug.cfs_rq[25]:/.min_vruntime
37675 ± 1% -89.7% 3883 ± 4% sched_debug.cfs_rq[25]:/.tg->runnable_avg
37701 ± 1% -89.7% 3887 ± 4% sched_debug.cfs_rq[26]:/.tg->runnable_avg
5460570 ± 11% -99.2% 42120 ± 20% sched_debug.cfs_rq[26]:/.min_vruntime
72021 ± 8% -97.9% 1500 ± 39% sched_debug.cfs_rq[26]:/.exec_clock
5443461 ± 11% -99.3% 39281 ± 30% sched_debug.cfs_rq[27]:/.min_vruntime
37720 ± 1% -89.7% 3890 ± 4% sched_debug.cfs_rq[27]:/.tg->runnable_avg
70407 ± 11% -97.6% 1657 ± 41% sched_debug.cfs_rq[27]:/.exec_clock
37737 ± 1% -89.7% 3890 ± 4% sched_debug.cfs_rq[28]:/.tg->runnable_avg
71813 ± 12% -98.0% 1424 ± 44% sched_debug.cfs_rq[28]:/.exec_clock
5482457 ± 11% -99.2% 42096 ± 24% sched_debug.cfs_rq[28]:/.min_vruntime
5478035 ± 11% -99.2% 41368 ± 31% sched_debug.cfs_rq[29]:/.min_vruntime
37748 ± 1% -89.7% 3893 ± 4% sched_debug.cfs_rq[29]:/.tg->runnable_avg
5924938 ± 8% -86.5% 799517 ± 29% sched_debug.cfs_rq[2]:/.min_vruntime
37344 ± 1% -89.7% 3839 ± 4% sched_debug.cfs_rq[2]:/.tg->runnable_avg
1042225 ± 20% -39.3% 632468 ± 35% sched_debug.cfs_rq[2]:/.spread0
37765 ± 1% -89.7% 3895 ± 4% sched_debug.cfs_rq[30]:/.tg->runnable_avg
2514014.52 ± 44% -100.0% 0.00 ± 0% sched_debug.cfs_rq[30]:/.max_vruntime
5791091 ± 11% -99.1% 49961 ± 26% sched_debug.cfs_rq[30]:/.min_vruntime
2514014.52 ± 44% -100.0% 0.00 ± 0% sched_debug.cfs_rq[30]:/.MIN_vruntime
76335 ± 12% -97.0% 2285 ± 16% sched_debug.cfs_rq[30]:/.exec_clock
76769 ± 11% -97.4% 2026 ± 16% sched_debug.cfs_rq[31]:/.exec_clock
2370182.85 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[31]:/.MIN_vruntime
5806067 ± 11% -99.2% 46410 ± 22% sched_debug.cfs_rq[31]:/.min_vruntime
2370183.19 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[31]:/.max_vruntime
37783 ± 1% -89.7% 3903 ± 4% sched_debug.cfs_rq[31]:/.tg->runnable_avg
77107 ± 12% -97.4% 1980 ± 30% sched_debug.cfs_rq[32]:/.exec_clock
5865705 ± 12% -99.2% 45792 ± 32% sched_debug.cfs_rq[32]:/.min_vruntime
37808 ± 1% -89.7% 3909 ± 4% sched_debug.cfs_rq[32]:/.tg->runnable_avg
24950 ± 9% -96.2% 943 ± 23% sched_debug.cfs_rq[33]:/.avg->runnable_avg_sum
37805 ± 1% -89.6% 3914 ± 4% sched_debug.cfs_rq[33]:/.tg->runnable_avg
5827248 ± 12% -99.2% 44413 ± 27% sched_debug.cfs_rq[33]:/.min_vruntime
76627 ± 13% -97.6% 1812 ± 27% sched_debug.cfs_rq[33]:/.exec_clock
546 ± 8% -96.4% 19 ± 22% sched_debug.cfs_rq[33]:/.tg_runnable_contrib
37804 ± 1% -89.6% 3915 ± 4% sched_debug.cfs_rq[34]:/.tg->runnable_avg
76396 ± 12% -97.7% 1738 ± 29% sched_debug.cfs_rq[34]:/.exec_clock
2052042.52 ± 30% -100.0% 0.00 ± 0% sched_debug.cfs_rq[34]:/.MIN_vruntime
5789564 ± 11% -99.2% 43627 ± 23% sched_debug.cfs_rq[34]:/.min_vruntime
2052042.52 ± 30% -100.0% 0.00 ± 0% sched_debug.cfs_rq[34]:/.max_vruntime
37810 ± 1% -89.6% 3915 ± 4% sched_debug.cfs_rq[35]:/.tg->runnable_avg
5793549 ± 11% -99.3% 40722 ± 15% sched_debug.cfs_rq[35]:/.min_vruntime
76584 ± 12% -97.8% 1682 ± 27% sched_debug.cfs_rq[35]:/.exec_clock
553 ± 8% -98.9% 6 ± 36% sched_debug.cfs_rq[36]:/.tg_runnable_contrib
37800 ± 1% -89.6% 3917 ± 4% sched_debug.cfs_rq[36]:/.tg->runnable_avg
25249 ± 8% -98.7% 330 ± 28% sched_debug.cfs_rq[36]:/.avg->runnable_avg_sum
76722 ± 13% -97.9% 1576 ± 41% sched_debug.cfs_rq[36]:/.exec_clock
5843915 ± 12% -99.3% 40252 ± 43% sched_debug.cfs_rq[36]:/.min_vruntime
75923 ± 12% -97.7% 1718 ± 38% sched_debug.cfs_rq[37]:/.exec_clock
5768738 ± 11% -99.3% 42192 ± 39% sched_debug.cfs_rq[37]:/.min_vruntime
535 ± 9% -98.8% 6 ± 28% sched_debug.cfs_rq[37]:/.tg_runnable_contrib
37811 ± 1% -89.6% 3924 ± 4% sched_debug.cfs_rq[37]:/.tg->runnable_avg
24479 ± 9% -98.7% 316 ± 26% sched_debug.cfs_rq[37]:/.avg->runnable_avg_sum
7 ± 32% -100.0% 0 ± 0% sched_debug.cfs_rq[38]:/.load
37833 ± 1% -89.6% 3925 ± 4% sched_debug.cfs_rq[38]:/.tg->runnable_avg
5 ± 30% -100.0% 0 ± 0% sched_debug.cfs_rq[38]:/.runnable_load_avg
5826004 ± 12% -99.3% 43456 ± 44% sched_debug.cfs_rq[38]:/.min_vruntime
544 ± 10% -98.4% 8 ± 27% sched_debug.cfs_rq[39]:/.tg_runnable_contrib
6 ± 21% -100.0% 0 ± 0% sched_debug.cfs_rq[39]:/.runnable_load_avg
76688 ± 13% -98.2% 1415 ± 35% sched_debug.cfs_rq[39]:/.exec_clock
37837 ± 1% -89.6% 3925 ± 4% sched_debug.cfs_rq[39]:/.tg->runnable_avg
24864 ± 9% -98.2% 454 ± 23% sched_debug.cfs_rq[39]:/.avg->runnable_avg_sum
5835422 ± 12% -99.4% 36160 ± 31% sched_debug.cfs_rq[39]:/.min_vruntime
26232 ± 5% -22.4% 20344 ± 8% sched_debug.cfs_rq[3]:/.avg->runnable_avg_sum
574 ± 6% -22.5% 445 ± 8% sched_debug.cfs_rq[3]:/.tg_runnable_contrib
37364 ± 1% -89.7% 3840 ± 4% sched_debug.cfs_rq[3]:/.tg->runnable_avg
9 ± 49% +335.2% 39 ± 44% sched_debug.cfs_rq[3]:/.load
5891140 ± 8% -87.4% 740621 ± 29% sched_debug.cfs_rq[3]:/.min_vruntime
1008386 ± 18% -43.1% 573573 ± 35% sched_debug.cfs_rq[3]:/.spread0
76125 ± 12% -98.0% 1494 ± 47% sched_debug.cfs_rq[40]:/.exec_clock
37838 ± 1% -89.6% 3926 ± 4% sched_debug.cfs_rq[40]:/.tg->runnable_avg
5784431 ± 11% -99.3% 38831 ± 36% sched_debug.cfs_rq[40]:/.min_vruntime
76346 ± 13% -98.1% 1455 ± 41% sched_debug.cfs_rq[41]:/.exec_clock
2279884.43 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[41]:/.MIN_vruntime
5802087 ± 12% -99.3% 40195 ± 34% sched_debug.cfs_rq[41]:/.min_vruntime
5 ± 29% -100.0% 0 ± 0% sched_debug.cfs_rq[41]:/.runnable_load_avg
2279884.76 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[41]:/.max_vruntime
37855 ± 1% -89.6% 3929 ± 4% sched_debug.cfs_rq[41]:/.tg->runnable_avg
134 ± 42% -100.0% 0 ± 0% sched_debug.cfs_rq[41]:/.utilization_load_avg
37858 ± 1% -89.6% 3932 ± 4% sched_debug.cfs_rq[42]:/.tg->runnable_avg
5839210 ± 12% -99.3% 39152 ± 47% sched_debug.cfs_rq[42]:/.min_vruntime
76766 ± 13% -98.2% 1387 ± 46% sched_debug.cfs_rq[42]:/.exec_clock
5 ± 20% -100.0% 0 ± 0% sched_debug.cfs_rq[42]:/.runnable_load_avg
76865 ± 13% -98.1% 1491 ± 35% sched_debug.cfs_rq[43]:/.exec_clock
5 ± 34% -100.0% 0 ± 0% sched_debug.cfs_rq[43]:/.runnable_load_avg
7 ± 44% -100.0% 0 ± 0% sched_debug.cfs_rq[43]:/.load
3274498.62 ± 42% -100.0% 0.00 ± 0% sched_debug.cfs_rq[43]:/.max_vruntime
5845980 ± 12% -99.3% 42925 ± 28% sched_debug.cfs_rq[43]:/.min_vruntime
3274498.62 ± 42% -100.0% 0.00 ± 0% sched_debug.cfs_rq[43]:/.MIN_vruntime
37884 ± 1% -89.6% 3934 ± 4% sched_debug.cfs_rq[43]:/.tg->runnable_avg
76461 ± 13% -97.9% 1582 ± 36% sched_debug.cfs_rq[44]:/.exec_clock
5801262 ± 12% -99.3% 39028 ± 29% sched_debug.cfs_rq[44]:/.min_vruntime
37904 ± 1% -89.6% 3939 ± 4% sched_debug.cfs_rq[44]:/.tg->runnable_avg
37919 ± 1% -89.6% 3945 ± 4% sched_debug.cfs_rq[45]:/.tg->runnable_avg
75128 ± 8% -96.8% 2372 ± 25% sched_debug.cfs_rq[45]:/.exec_clock
5733040 ± 8% -99.2% 47682 ± 38% sched_debug.cfs_rq[45]:/.min_vruntime
74823 ± 8% -97.4% 1957 ± 15% sched_debug.cfs_rq[46]:/.exec_clock
2181820.38 ± 44% -100.0% 0.00 ± 0% sched_debug.cfs_rq[46]:/.MIN_vruntime
5706201 ± 7% -99.2% 45069 ± 29% sched_debug.cfs_rq[46]:/.min_vruntime
37931 ± 1% -89.6% 3949 ± 4% sched_debug.cfs_rq[46]:/.tg->runnable_avg
2181820.38 ± 44% -100.0% 0.00 ± 0% sched_debug.cfs_rq[46]:/.max_vruntime
37937 ± 1% -89.6% 3952 ± 4% sched_debug.cfs_rq[47]:/.tg->runnable_avg
75941 ± 7% -97.7% 1740 ± 26% sched_debug.cfs_rq[47]:/.exec_clock
5795769 ± 7% -99.3% 43188 ± 29% sched_debug.cfs_rq[47]:/.min_vruntime
74784 ± 7% -97.9% 1582 ± 27% sched_debug.cfs_rq[48]:/.exec_clock
5703468 ± 7% -99.3% 39942 ± 31% sched_debug.cfs_rq[48]:/.min_vruntime
37947 ± 1% -89.6% 3954 ± 4% sched_debug.cfs_rq[48]:/.tg->runnable_avg
5 ± 33% -100.0% 0 ± 0% sched_debug.cfs_rq[49]:/.runnable_load_avg
75045 ± 7% -97.9% 1592 ± 23% sched_debug.cfs_rq[49]:/.exec_clock
2349144.49 ± 34% -100.0% 0.00 ± 0% sched_debug.cfs_rq[49]:/.MIN_vruntime
5717654 ± 7% -99.3% 41062 ± 37% sched_debug.cfs_rq[49]:/.min_vruntime
2349144.49 ± 34% -100.0% 0.00 ± 0% sched_debug.cfs_rq[49]:/.max_vruntime
37976 ± 1% -89.6% 3956 ± 4% sched_debug.cfs_rq[49]:/.tg->runnable_avg
26117 ± 3% -50.7% 12870 ± 7% sched_debug.cfs_rq[4]:/.avg->runnable_avg_sum
37364 ± 1% -89.7% 3843 ± 4% sched_debug.cfs_rq[4]:/.tg->runnable_avg
78451 ± 8% -47.9% 40905 ± 20% sched_debug.cfs_rq[4]:/.exec_clock
571 ± 4% -50.5% 282 ± 7% sched_debug.cfs_rq[4]:/.tg_runnable_contrib
5828333 ± 8% -91.7% 486365 ± 28% sched_debug.cfs_rq[4]:/.min_vruntime
945560 ± 28% -66.2% 319316 ± 41% sched_debug.cfs_rq[4]:/.spread0
92 ± 24% -100.0% 0 ± 0% sched_debug.cfs_rq[50]:/.utilization_load_avg
75109 ± 7% -97.7% 1699 ± 37% sched_debug.cfs_rq[50]:/.exec_clock
6 ± 49% -100.0% 0 ± 0% sched_debug.cfs_rq[50]:/.load
5726100 ± 6% -99.2% 45365 ± 30% sched_debug.cfs_rq[50]:/.min_vruntime
543 ± 11% -98.9% 6 ± 49% sched_debug.cfs_rq[50]:/.tg_runnable_contrib
24764 ± 11% -98.7% 328 ± 39% sched_debug.cfs_rq[50]:/.avg->runnable_avg_sum
37997 ± 1% -89.6% 3959 ± 4% sched_debug.cfs_rq[50]:/.tg->runnable_avg
4 ± 34% -100.0% 0 ± 0% sched_debug.cfs_rq[50]:/.runnable_load_avg
5701479 ± 6% -99.3% 37520 ± 36% sched_debug.cfs_rq[51]:/.min_vruntime
38008 ± 1% -89.6% 3959 ± 4% sched_debug.cfs_rq[51]:/.tg->runnable_avg
74668 ± 7% -98.1% 1411 ± 41% sched_debug.cfs_rq[51]:/.exec_clock
25473 ± 12% -98.4% 408 ± 48% sched_debug.cfs_rq[52]:/.avg->runnable_avg_sum
75465 ± 8% -98.1% 1450 ± 23% sched_debug.cfs_rq[52]:/.exec_clock
3567331.44 ± 29% -100.0% 0.00 ± 0% sched_debug.cfs_rq[52]:/.MIN_vruntime
5767898 ± 7% -99.3% 38845 ± 30% sched_debug.cfs_rq[52]:/.min_vruntime
8 ± 41% -100.0% 0 ± 0% sched_debug.cfs_rq[52]:/.runnable_load_avg
3567331.77 ± 29% -100.0% 0.00 ± 0% sched_debug.cfs_rq[52]:/.max_vruntime
38019 ± 1% -89.6% 3962 ± 4% sched_debug.cfs_rq[52]:/.tg->runnable_avg
38036 ± 2% -89.6% 3966 ± 4% sched_debug.cfs_rq[53]:/.tg->runnable_avg
75493 ± 8% -98.0% 1520 ± 30% sched_debug.cfs_rq[53]:/.exec_clock
2564844.76 ± 34% -100.0% 0.00 ± 0% sched_debug.cfs_rq[53]:/.max_vruntime
5768160 ± 7% -99.2% 44245 ± 35% sched_debug.cfs_rq[53]:/.min_vruntime
2564844.76 ± 34% -100.0% 0.00 ± 0% sched_debug.cfs_rq[53]:/.MIN_vruntime
75523 ± 9% -98.1% 1443 ± 40% sched_debug.cfs_rq[54]:/.exec_clock
38055 ± 2% -89.6% 3969 ± 4% sched_debug.cfs_rq[54]:/.tg->runnable_avg
5757595 ± 8% -99.3% 41312 ± 28% sched_debug.cfs_rq[54]:/.min_vruntime
5731615 ± 7% -99.3% 38668 ± 27% sched_debug.cfs_rq[55]:/.min_vruntime
75160 ± 8% -98.0% 1473 ± 39% sched_debug.cfs_rq[55]:/.exec_clock
38085 ± 2% -89.6% 3971 ± 4% sched_debug.cfs_rq[55]:/.tg->runnable_avg
552 ± 13% -99.0% 5 ± 49% sched_debug.cfs_rq[56]:/.tg_runnable_contrib
5759032 ± 7% -99.3% 37467 ± 32% sched_debug.cfs_rq[56]:/.min_vruntime
38101 ± 2% -89.6% 3980 ± 4% sched_debug.cfs_rq[56]:/.tg->runnable_avg
25129 ± 13% -98.8% 291 ± 44% sched_debug.cfs_rq[56]:/.avg->runnable_avg_sum
75483 ± 8% -98.1% 1421 ± 48% sched_debug.cfs_rq[56]:/.exec_clock
5 ± 29% -100.0% 0 ± 0% sched_debug.cfs_rq[57]:/.runnable_load_avg
74873 ± 8% -97.9% 1582 ± 49% sched_debug.cfs_rq[57]:/.exec_clock
5707036 ± 7% -99.2% 43462 ± 45% sched_debug.cfs_rq[57]:/.min_vruntime
38113 ± 2% -89.6% 3981 ± 4% sched_debug.cfs_rq[57]:/.tg->runnable_avg
2056049.76 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[58]:/.MIN_vruntime
2056049.76 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[58]:/.max_vruntime
5638172 ± 8% -99.3% 40579 ± 41% sched_debug.cfs_rq[58]:/.min_vruntime
38110 ± 1% -89.6% 3979 ± 4% sched_debug.cfs_rq[58]:/.tg->runnable_avg
74655 ± 8% -98.3% 1291 ± 34% sched_debug.cfs_rq[59]:/.exec_clock
9 ± 48% -100.0% 0 ± 0% sched_debug.cfs_rq[59]:/.load
38125 ± 1% -89.6% 3980 ± 4% sched_debug.cfs_rq[59]:/.tg->runnable_avg
5687789 ± 7% -99.3% 37622 ± 41% sched_debug.cfs_rq[59]:/.min_vruntime
569 ± 5% -89.4% 60 ± 18% sched_debug.cfs_rq[5]:/.tg_runnable_contrib
78491 ± 9% -87.5% 9836 ± 6% sched_debug.cfs_rq[5]:/.exec_clock
37386 ± 1% -89.7% 3845 ± 4% sched_debug.cfs_rq[5]:/.tg->runnable_avg
26026 ± 6% -89.3% 2782 ± 17% sched_debug.cfs_rq[5]:/.avg->runnable_avg_sum
5868941 ± 8% -97.8% 132022 ± 13% sched_debug.cfs_rq[5]:/.min_vruntime
50960 ± 9% -99.1% 438 ± 44% sched_debug.cfs_rq[60]:/.exec_clock
38133 ± 1% -89.6% 3981 ± 4% sched_debug.cfs_rq[60]:/.tg->runnable_avg
51720 ± 14% -97.9% 1072 ± 40% sched_debug.cfs_rq[61]:/.exec_clock
4337125 ± 13% -99.2% 35916 ± 38% sched_debug.cfs_rq[61]:/.min_vruntime
38160 ± 1% -89.6% 3984 ± 4% sched_debug.cfs_rq[61]:/.tg->runnable_avg
3 ± 39% -100.0% 0 ± 0% sched_debug.cfs_rq[62]:/.load
52202 ± 13% -98.8% 626 ± 34% sched_debug.cfs_rq[62]:/.exec_clock
38148 ± 1% -89.5% 3986 ± 4% sched_debug.cfs_rq[62]:/.tg->runnable_avg
14412 ± 18% -98.6% 194 ± 48% sched_debug.cfs_rq[62]:/.avg->runnable_avg_sum
2 ± 40% -100.0% 0 ± 0% sched_debug.cfs_rq[62]:/.runnable_load_avg
38154 ± 2% -89.6% 3986 ± 4% sched_debug.cfs_rq[63]:/.tg->runnable_avg
1268924 ± 47% -97.9% 26358 ± 34% sched_debug.cfs_rq[64]:/.min_vruntime
64 ± 42% -95.8% 2 ± 41% sched_debug.cfs_rq[64]:/.tg_runnable_contrib
38173 ± 2% -89.6% 3988 ± 4% sched_debug.cfs_rq[64]:/.tg->runnable_avg
2993 ± 41% -95.1% 148 ± 36% sched_debug.cfs_rq[64]:/.avg->runnable_avg_sum
38152 ± 2% -89.6% 3985 ± 4% sched_debug.cfs_rq[65]:/.tg->runnable_avg
7 ± 47% -100.0% 0 ± 0% sched_debug.cfs_rq[65]:/.nr_spread_over
38162 ± 2% -89.5% 3989 ± 3% sched_debug.cfs_rq[66]:/.tg->runnable_avg
10 ± 30% -100.0% 0 ± 0% sched_debug.cfs_rq[66]:/.nr_spread_over
15875 ± 49% -95.7% 680 ± 31% sched_debug.cfs_rq[66]:/.exec_clock
38175 ± 2% -89.5% 3990 ± 3% sched_debug.cfs_rq[67]:/.tg->runnable_avg
38206 ± 2% -89.6% 3991 ± 3% sched_debug.cfs_rq[68]:/.tg->runnable_avg
38221 ± 2% -89.6% 3992 ± 3% sched_debug.cfs_rq[69]:/.tg->runnable_avg
15369 ± 47% -94.7% 812 ± 43% sched_debug.cfs_rq[69]:/.exec_clock
25441 ± 4% -91.8% 2076 ± 13% sched_debug.cfs_rq[6]:/.avg->runnable_avg_sum
557 ± 4% -92.0% 44 ± 13% sched_debug.cfs_rq[6]:/.tg_runnable_contrib
6 ± 45% -100.0% 0 ± 0% sched_debug.cfs_rq[6]:/.load
5815571 ± 7% -98.1% 110120 ± 10% sched_debug.cfs_rq[6]:/.min_vruntime
77709 ± 8% -89.8% 7891 ± 9% sched_debug.cfs_rq[6]:/.exec_clock
5 ± 33% -100.0% 0 ± 0% sched_debug.cfs_rq[6]:/.runnable_load_avg
37407 ± 1% -89.7% 3849 ± 4% sched_debug.cfs_rq[6]:/.tg->runnable_avg
38225 ± 2% -89.5% 4001 ± 4% sched_debug.cfs_rq[70]:/.tg->runnable_avg
38235 ± 2% -89.5% 4005 ± 4% sched_debug.cfs_rq[71]:/.tg->runnable_avg
38238 ± 2% -89.5% 4007 ± 4% sched_debug.cfs_rq[72]:/.tg->runnable_avg
38253 ± 2% -89.5% 4007 ± 4% sched_debug.cfs_rq[73]:/.tg->runnable_avg
9 ± 27% -100.0% 0 ± 0% sched_debug.cfs_rq[73]:/.nr_spread_over
7 ± 45% -100.0% 0 ± 0% sched_debug.cfs_rq[74]:/.nr_spread_over
38240 ± 2% -89.5% 4012 ± 4% sched_debug.cfs_rq[74]:/.tg->runnable_avg
38235 ± 2% -89.5% 4010 ± 4% sched_debug.cfs_rq[75]:/.tg->runnable_avg
38233 ± 2% -89.5% 4010 ± 4% sched_debug.cfs_rq[76]:/.tg->runnable_avg
38239 ± 2% -89.5% 4012 ± 4% sched_debug.cfs_rq[77]:/.tg->runnable_avg
38260 ± 2% -89.5% 4015 ± 4% sched_debug.cfs_rq[78]:/.tg->runnable_avg
38284 ± 2% -89.5% 4019 ± 3% sched_debug.cfs_rq[79]:/.tg->runnable_avg
25894 ± 3% -92.4% 1956 ± 12% sched_debug.cfs_rq[7]:/.avg->runnable_avg_sum
78030 ± 8% -90.8% 7193 ± 8% sched_debug.cfs_rq[7]:/.exec_clock
5863768 ± 7% -98.2% 105821 ± 12% sched_debug.cfs_rq[7]:/.min_vruntime
566 ± 3% -92.6% 42 ± 12% sched_debug.cfs_rq[7]:/.tg_runnable_contrib
37430 ± 1% -89.7% 3850 ± 4% sched_debug.cfs_rq[7]:/.tg->runnable_avg
38292 ± 2% -89.5% 4021 ± 3% sched_debug.cfs_rq[80]:/.tg->runnable_avg
38300 ± 2% -89.5% 4022 ± 3% sched_debug.cfs_rq[81]:/.tg->runnable_avg
38303 ± 2% -89.5% 4025 ± 3% sched_debug.cfs_rq[82]:/.tg->runnable_avg
38286 ± 2% -89.5% 4028 ± 3% sched_debug.cfs_rq[83]:/.tg->runnable_avg
38294 ± 2% -89.5% 4032 ± 3% sched_debug.cfs_rq[84]:/.tg->runnable_avg
38304 ± 2% -89.5% 4034 ± 3% sched_debug.cfs_rq[85]:/.tg->runnable_avg
38321 ± 2% -89.5% 4035 ± 3% sched_debug.cfs_rq[86]:/.tg->runnable_avg
38326 ± 2% -89.5% 4039 ± 3% sched_debug.cfs_rq[87]:/.tg->runnable_avg
38322 ± 2% -89.5% 4040 ± 3% sched_debug.cfs_rq[88]:/.tg->runnable_avg
38313 ± 2% -89.4% 4044 ± 3% sched_debug.cfs_rq[89]:/.tg->runnable_avg
37445 ± 1% -89.7% 3853 ± 4% sched_debug.cfs_rq[8]:/.tg->runnable_avg
553 ± 4% -92.0% 44 ± 5% sched_debug.cfs_rq[8]:/.tg_runnable_contrib
25270 ± 4% -91.8% 2076 ± 5% sched_debug.cfs_rq[8]:/.avg->runnable_avg_sum
77680 ± 8% -91.2% 6799 ± 6% sched_debug.cfs_rq[8]:/.exec_clock
5833584 ± 8% -98.3% 99293 ± 11% sched_debug.cfs_rq[8]:/.min_vruntime
38307 ± 2% -89.4% 4044 ± 3% sched_debug.cfs_rq[90]:/.tg->runnable_avg
38316 ± 2% -89.4% 4047 ± 3% sched_debug.cfs_rq[91]:/.tg->runnable_avg
4788 ± 48% -95.1% 235 ± 35% sched_debug.cfs_rq[91]:/.avg->runnable_avg_sum
103 ± 49% -95.8% 4 ± 39% sched_debug.cfs_rq[91]:/.tg_runnable_contrib
38320 ± 2% -89.4% 4051 ± 3% sched_debug.cfs_rq[92]:/.tg->runnable_avg
38332 ± 2% -89.4% 4052 ± 3% sched_debug.cfs_rq[93]:/.tg->runnable_avg
38341 ± 2% -89.4% 4053 ± 3% sched_debug.cfs_rq[94]:/.tg->runnable_avg
38329 ± 2% -89.4% 4056 ± 3% sched_debug.cfs_rq[95]:/.tg->runnable_avg
38315 ± 2% -89.4% 4060 ± 3% sched_debug.cfs_rq[96]:/.tg->runnable_avg
38318 ± 2% -89.4% 4061 ± 3% sched_debug.cfs_rq[97]:/.tg->runnable_avg
38321 ± 2% -89.4% 4064 ± 3% sched_debug.cfs_rq[98]:/.tg->runnable_avg
38326 ± 2% -89.4% 4066 ± 3% sched_debug.cfs_rq[99]:/.tg->runnable_avg
2599395.82 ± 28% -100.0% 0.00 ± 0% sched_debug.cfs_rq[9]:/.MIN_vruntime
37435 ± 1% -89.7% 3849 ± 4% sched_debug.cfs_rq[9]:/.tg->runnable_avg
7 ± 45% -100.0% 0 ± 0% sched_debug.cfs_rq[9]:/.runnable_load_avg
25086 ± 4% -93.1% 1741 ± 14% sched_debug.cfs_rq[9]:/.avg->runnable_avg_sum
76760 ± 8% -91.3% 6708 ± 10% sched_debug.cfs_rq[9]:/.exec_clock
2599396.15 ± 28% -100.0% 0.00 ± 0% sched_debug.cfs_rq[9]:/.max_vruntime
550 ± 4% -93.2% 37 ± 15% sched_debug.cfs_rq[9]:/.tg_runnable_contrib
5775897 ± 7% -98.3% 99243 ± 18% sched_debug.cfs_rq[9]:/.min_vruntime
7 ± 39% +11559.1% 855 ± 0% sched_debug.cpu#0.cpu_load[3]
8 ± 35% +9767.3% 855 ± 0% sched_debug.cpu#0.cpu_load[2]
10 ± 25% +8443.3% 854 ± 0% sched_debug.cpu#0.cpu_load[1]
6 ± 39% +14133.3% 854 ± 0% sched_debug.cpu#0.cpu_load[4]
866410 ± 20% -42.2% 500354 ± 0% sched_debug.cpu#0.max_idle_balance_cost
401816 ± 6% +5302.7% 21709084 ± 9% sched_debug.cpu#0.ttwu_count
312394 ± 14% +1631.2% 5408220 ± 18% sched_debug.cpu#1.sched_goidle
177603 ± 24% -87.5% 22288 ± 17% sched_debug.cpu#1.ttwu_local
303227 ± 7% -46.2% 163110 ± 12% sched_debug.cpu#1.avg_idle
695685 ± 12% +1457.9% 10837841 ± 18% sched_debug.cpu#1.nr_switches
699186 ± 12% +1450.1% 10837966 ± 18% sched_debug.cpu#1.sched_count
461875 ± 8% -94.9% 23602 ± 15% sched_debug.cpu#1.ttwu_count
262569 ± 5% +240.9% 895031 ± 14% sched_debug.cpu#10.avg_idle
145636 ± 7% -30.0% 101980 ± 10% sched_debug.cpu#10.nr_load_updates
4 ± 36% -100.0% 0 ± 0% sched_debug.cpu#10.cpu_load[1]
3 ± 42% -100.0% 0 ± 0% sched_debug.cpu#10.cpu_load[3]
3 ± 34% -100.0% 0 ± 0% sched_debug.cpu#10.cpu_load[2]
534226 ± 10% -99.7% 1612 ± 29% sched_debug.cpu#10.ttwu_count
182313 ± 10% -99.3% 1351 ± 29% sched_debug.cpu#10.ttwu_local
912692 ± 29% -45.2% 500000 ± 0% sched_debug.cpu#100.max_idle_balance_cost
890003 ± 25% -43.5% 502419 ± 0% sched_debug.cpu#101.max_idle_balance_cost
895162 ± 26% -42.0% 518876 ± 8% sched_debug.cpu#102.max_idle_balance_cost
1279055 ± 16% -18.8% 1038141 ± 8% sched_debug.cpu#102.avg_idle
1210646 ± 23% -19.2% 978478 ± 1% sched_debug.cpu#103.avg_idle
873261 ± 25% -42.7% 500000 ± 0% sched_debug.cpu#103.max_idle_balance_cost
864952 ± 24% -42.2% 500000 ± 0% sched_debug.cpu#104.max_idle_balance_cost
887413 ± 31% -43.7% 500000 ± 0% sched_debug.cpu#105.max_idle_balance_cost
854103 ± 27% -40.9% 504735 ± 2% sched_debug.cpu#106.max_idle_balance_cost
864081 ± 28% -42.0% 501124 ± 0% sched_debug.cpu#107.max_idle_balance_cost
898657 ± 28% -42.5% 516711 ± 6% sched_debug.cpu#108.max_idle_balance_cost
3 ± 42% -100.0% 0 ± 0% sched_debug.cpu#11.cpu_load[2]
543726 ± 4% -99.7% 1760 ± 32% sched_debug.cpu#11.ttwu_count
145709 ± 7% -30.3% 101564 ± 9% sched_debug.cpu#11.nr_load_updates
270996 ± 1% +229.9% 894099 ± 14% sched_debug.cpu#11.avg_idle
2 ± 37% -100.0% 0 ± 0% sched_debug.cpu#11.cpu_load[3]
185681 ± 10% -99.2% 1455 ± 30% sched_debug.cpu#11.ttwu_local
873654 ± 27% -42.5% 502080 ± 0% sched_debug.cpu#110.max_idle_balance_cost
858843 ± 30% -41.1% 505597 ± 1% sched_debug.cpu#111.max_idle_balance_cost
872547 ± 28% -42.7% 500133 ± 0% sched_debug.cpu#112.max_idle_balance_cost
850260 ± 28% -40.6% 505082 ± 2% sched_debug.cpu#113.max_idle_balance_cost
960366 ± 23% -47.8% 500984 ± 0% sched_debug.cpu#114.max_idle_balance_cost
869532 ± 27% -42.2% 502156 ± 0% sched_debug.cpu#115.max_idle_balance_cost
884039 ± 27% -42.9% 505044 ± 1% sched_debug.cpu#116.max_idle_balance_cost
855531 ± 30% -41.2% 502901 ± 1% sched_debug.cpu#117.max_idle_balance_cost
844778 ± 29% -40.2% 505544 ± 1% sched_debug.cpu#118.max_idle_balance_cost
867345 ± 27% -41.1% 510969 ± 4% sched_debug.cpu#119.max_idle_balance_cost
145505 ± 8% -34.3% 95650 ± 15% sched_debug.cpu#12.nr_load_updates
5 ± 44% -100.0% 0 ± 0% sched_debug.cpu#12.cpu_load[1]
186685 ± 11% -99.3% 1223 ± 33% sched_debug.cpu#12.ttwu_local
4 ± 49% -100.0% 0 ± 0% sched_debug.cpu#12.cpu_load[2]
280891 ± 3% +222.5% 905991 ± 14% sched_debug.cpu#12.avg_idle
3 ± 39% -100.0% 0 ± 0% sched_debug.cpu#12.cpu_load[3]
187388 ± 11% -99.3% 1223 ± 30% sched_debug.cpu#13.ttwu_local
145475 ± 8% -31.9% 99013 ± 11% sched_debug.cpu#13.nr_load_updates
546320 ± 8% -99.7% 1508 ± 21% sched_debug.cpu#13.ttwu_count
3 ± 27% -100.0% 0 ± 0% sched_debug.cpu#13.cpu_load[1]
282818 ± 5% +212.5% 883684 ± 13% sched_debug.cpu#13.avg_idle
145427 ± 7% -34.5% 95288 ± 17% sched_debug.cpu#14.nr_load_updates
4 ± 41% -100.0% 0 ± 0% sched_debug.cpu#14.cpu_load[3]
283614 ± 6% +204.3% 862952 ± 17% sched_debug.cpu#14.avg_idle
548763 ± 10% -99.7% 1508 ± 26% sched_debug.cpu#14.ttwu_count
187787 ± 10% -99.3% 1258 ± 24% sched_debug.cpu#14.ttwu_local
4 ± 46% -100.0% 0 ± 0% sched_debug.cpu#14.cpu_load[2]
910931 ± 10% -90.9% 83254 ± 28% sched_debug.cpu#15.sched_count
401751 ± 10% -89.7% 41393 ± 28% sched_debug.cpu#15.sched_goidle
532305 ± 9% -99.3% 3824 ± 15% sched_debug.cpu#15.ttwu_count
227026 ± 15% -98.7% 2840 ± 12% sched_debug.cpu#15.ttwu_local
2 ± 46% -100.0% 0 ± 0% sched_debug.cpu#15.cpu_load[2]
907583 ± 10% -90.8% 83244 ± 28% sched_debug.cpu#15.nr_switches
147307 ± 8% -64.1% 52925 ± 18% sched_debug.cpu#15.nr_load_updates
271896 ± 28% +268.2% 1001194 ± 0% sched_debug.cpu#15.avg_idle
228547 ± 14% -99.6% 1006 ± 46% sched_debug.cpu#16.ttwu_local
146688 ± 7% -64.8% 51642 ± 18% sched_debug.cpu#16.nr_load_updates
538118 ± 10% -99.7% 1836 ± 36% sched_debug.cpu#16.ttwu_count
401589 ± 9% -89.9% 40532 ± 27% sched_debug.cpu#16.sched_goidle
912282 ± 10% -91.1% 81551 ± 27% sched_debug.cpu#16.sched_count
274129 ± 10% +258.6% 983056 ± 2% sched_debug.cpu#16.avg_idle
909937 ± 10% -91.0% 81542 ± 27% sched_debug.cpu#16.nr_switches
399818 ± 10% -90.8% 36949 ± 41% sched_debug.cpu#17.sched_goidle
251250 ± 33% +290.1% 980211 ± 2% sched_debug.cpu#17.avg_idle
146193 ± 8% -67.5% 47493 ± 30% sched_debug.cpu#17.nr_load_updates
907652 ± 10% -91.8% 74306 ± 41% sched_debug.cpu#17.sched_count
905857 ± 10% -91.8% 74297 ± 41% sched_debug.cpu#17.nr_switches
535902 ± 9% -99.7% 1645 ± 28% sched_debug.cpu#17.ttwu_count
535263 ± 9% -99.6% 2125 ± 33% sched_debug.cpu#18.ttwu_count
904084 ± 11% -93.0% 62907 ± 44% sched_debug.cpu#18.nr_switches
274939 ± 9% +266.2% 1006769 ± 2% sched_debug.cpu#18.avg_idle
226681 ± 14% -99.5% 1123 ± 40% sched_debug.cpu#18.ttwu_local
905555 ± 11% -93.1% 62916 ± 43% sched_debug.cpu#18.sched_count
146165 ± 7% -71.0% 42321 ± 30% sched_debug.cpu#18.nr_load_updates
399028 ± 10% -92.2% 31022 ± 44% sched_debug.cpu#18.sched_goidle
146308 ± 8% -71.5% 41686 ± 35% sched_debug.cpu#19.nr_load_updates
529715 ± 9% -99.6% 2212 ± 43% sched_debug.cpu#19.ttwu_count
296225 ± 10% +232.9% 986018 ± 1% sched_debug.cpu#19.avg_idle
3 ± 38% +605.3% 22 ± 29% sched_debug.cpu#2.cpu_load[2]
683586 ± 9% +1403.7% 10278920 ± 19% sched_debug.cpu#2.sched_count
3 ± 41% +460.9% 21 ± 46% sched_debug.cpu#2.cpu_load[1]
420236 ± 7% -95.2% 20376 ± 17% sched_debug.cpu#2.ttwu_count
154717 ± 17% -87.2% 19810 ± 18% sched_debug.cpu#2.ttwu_local
680385 ± 9% +1410.7% 10278585 ± 19% sched_debug.cpu#2.nr_switches
300978 ± 12% -50.5% 149090 ± 40% sched_debug.cpu#2.avg_idle
300267 ± 10% +1608.3% 5129477 ± 19% sched_debug.cpu#2.sched_goidle
226744 ± 15% -99.7% 791 ± 12% sched_debug.cpu#20.ttwu_local
2 ± 41% -100.0% 0 ± 0% sched_debug.cpu#20.cpu_load[3]
539626 ± 11% -99.7% 1714 ± 13% sched_debug.cpu#20.ttwu_count
3 ± 48% -100.0% 0 ± 0% sched_debug.cpu#20.cpu_load[1]
264020 ± 16% +277.1% 995578 ± 0% sched_debug.cpu#20.avg_idle
145629 ± 7% -72.8% 39540 ± 38% sched_debug.cpu#20.nr_load_updates
525878 ± 11% -99.6% 1905 ± 16% sched_debug.cpu#21.ttwu_count
145597 ± 8% -72.4% 40198 ± 40% sched_debug.cpu#21.nr_load_updates
6 ± 40% -100.0% 0 ± 0% sched_debug.cpu#21.load
226311 ± 17% -99.6% 954 ± 17% sched_debug.cpu#21.ttwu_local
2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#21.cpu_load[3]
282162 ± 10% +268.8% 1040716 ± 4% sched_debug.cpu#21.avg_idle
2 ± 40% -100.0% 0 ± 0% sched_debug.cpu#21.cpu_load[2]
4 ± 40% -100.0% 0 ± 0% sched_debug.cpu#22.cpu_load[1]
3 ± 34% -100.0% 0 ± 0% sched_debug.cpu#22.cpu_load[2]
145443 ± 7% -73.8% 38078 ± 39% sched_debug.cpu#22.nr_load_updates
2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#22.cpu_load[3]
547461 ± 10% -99.6% 2080 ± 41% sched_debug.cpu#22.ttwu_count
293095 ± 9% +242.6% 1004209 ± 2% sched_debug.cpu#22.avg_idle
538395 ± 10% -99.7% 1761 ± 30% sched_debug.cpu#23.ttwu_count
145290 ± 8% -77.3% 33007 ± 45% sched_debug.cpu#23.nr_load_updates
312631 ± 11% +217.5% 992648 ± 1% sched_debug.cpu#23.avg_idle
5 ± 42% -100.0% 0 ± 0% sched_debug.cpu#23.load
274102 ± 9% +261.8% 991658 ± 1% sched_debug.cpu#24.avg_idle
145062 ± 8% -75.6% 35376 ± 39% sched_debug.cpu#24.nr_load_updates
305640 ± 8% +215.8% 965318 ± 7% sched_debug.cpu#25.avg_idle
554385 ± 4% -99.6% 2368 ± 47% sched_debug.cpu#25.ttwu_count
146404 ± 6% -77.2% 33321 ± 47% sched_debug.cpu#25.nr_load_updates
562613 ± 11% -99.6% 2233 ± 32% sched_debug.cpu#26.ttwu_count
2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#26.cpu_load[2]
291842 ± 8% +239.5% 990850 ± 4% sched_debug.cpu#26.avg_idle
3 ± 37% -100.0% 0 ± 0% sched_debug.cpu#27.cpu_load[2]
4 ± 44% -100.0% 0 ± 0% sched_debug.cpu#27.cpu_load[1]
296953 ± 8% +234.6% 993660 ± 3% sched_debug.cpu#27.avg_idle
227387 ± 14% -99.6% 875 ± 46% sched_debug.cpu#28.ttwu_local
541565 ± 9% -99.7% 1729 ± 35% sched_debug.cpu#28.ttwu_count
287000 ± 9% +247.4% 996956 ± 1% sched_debug.cpu#28.avg_idle
292577 ± 5% +240.4% 995812 ± 9% sched_debug.cpu#29.avg_idle
2 ± 49% -100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[0]
2 ± 31% -100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[1]
3 ± 27% -100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[2]
538059 ± 11% -99.6% 1942 ± 46% sched_debug.cpu#29.ttwu_count
2 ± 40% -100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[3]
858348 ± 11% +988.5% 9343383 ± 19% sched_debug.cpu#3.nr_switches
583644 ± 14% -96.9% 18374 ± 19% sched_debug.cpu#3.ttwu_count
188558 ± 11% -90.6% 17658 ± 20% sched_debug.cpu#3.ttwu_local
861080 ± 11% +985.1% 9343434 ± 19% sched_debug.cpu#3.sched_count
3 ± 28% +325.0% 14 ± 46% sched_debug.cpu#3.cpu_load[2]
283363 ± 7% -36.9% 178917 ± 5% sched_debug.cpu#3.avg_idle
3 ± 27% +366.7% 14 ± 35% sched_debug.cpu#3.cpu_load[3]
365993 ± 11% +1174.1% 4662957 ± 19% sched_debug.cpu#3.sched_goidle
272296 ± 5% +263.0% 988390 ± 3% sched_debug.cpu#30.avg_idle
886491 ± 7% -90.7% 82339 ± 23% sched_debug.cpu#30.nr_switches
193322 ± 6% -99.6% 752 ± 14% sched_debug.cpu#30.ttwu_local
890539 ± 7% -90.7% 82392 ± 23% sched_debug.cpu#30.sched_count
145061 ± 7% -66.4% 48735 ± 20% sched_debug.cpu#30.nr_load_updates
385340 ± 7% -89.4% 40937 ± 23% sched_debug.cpu#30.sched_goidle
558855 ± 11% -99.7% 1486 ± 14% sched_debug.cpu#30.ttwu_count
607204 ± 10% -99.7% 1669 ± 28% sched_debug.cpu#31.ttwu_count
948582 ± 13% -92.4% 72412 ± 23% sched_debug.cpu#31.sched_count
147499 ± 6% -70.2% 44027 ± 20% sched_debug.cpu#31.nr_load_updates
231115 ± 34% -99.7% 707 ± 17% sched_debug.cpu#31.ttwu_local
414152 ± 16% -91.3% 35947 ± 23% sched_debug.cpu#31.sched_goidle
945517 ± 13% -92.3% 72360 ± 23% sched_debug.cpu#31.nr_switches
269881 ± 11% +260.8% 973666 ± 4% sched_debug.cpu#31.avg_idle
146258 ± 6% -72.3% 40536 ± 31% sched_debug.cpu#32.nr_load_updates
212916 ± 19% -99.7% 682 ± 15% sched_debug.cpu#32.ttwu_local
616801 ± 13% -99.8% 1439 ± 11% sched_debug.cpu#32.ttwu_count
886655 ± 10% -92.6% 65901 ± 36% sched_debug.cpu#32.sched_count
884019 ± 10% -92.6% 65851 ± 36% sched_debug.cpu#32.nr_switches
287282 ± 4% +221.9% 924747 ± 3% sched_debug.cpu#32.avg_idle
386416 ± 11% -91.5% 32707 ± 36% sched_debug.cpu#32.sched_goidle
381805 ± 6% -92.2% 29619 ± 35% sched_debug.cpu#33.sched_goidle
882037 ± 6% -93.2% 59641 ± 35% sched_debug.cpu#33.sched_count
264381 ± 4% +251.9% 930329 ± 7% sched_debug.cpu#33.avg_idle
580711 ± 13% -99.8% 1425 ± 8% sched_debug.cpu#33.ttwu_count
2 ± 37% -100.0% 0 ± 0% sched_debug.cpu#33.cpu_load[2]
198746 ± 7% -99.7% 677 ± 13% sched_debug.cpu#33.ttwu_local
879687 ± 6% -93.2% 59612 ± 35% sched_debug.cpu#33.nr_switches
145237 ± 7% -73.9% 37912 ± 30% sched_debug.cpu#33.nr_load_updates
558432 ± 12% -99.7% 1937 ± 40% sched_debug.cpu#34.ttwu_count
271737 ± 6% +267.3% 998140 ± 8% sched_debug.cpu#34.avg_idle
144986 ± 7% -74.4% 37118 ± 42% sched_debug.cpu#34.nr_load_updates
192853 ± 6% -99.6% 781 ± 15% sched_debug.cpu#34.ttwu_local
255816 ± 24% +287.7% 991833 ± 1% sched_debug.cpu#35.avg_idle
144906 ± 7% -74.4% 37168 ± 35% sched_debug.cpu#35.nr_load_updates
554138 ± 12% -99.7% 1813 ± 9% sched_debug.cpu#35.ttwu_count
379077 ± 6% -92.2% 29594 ± 42% sched_debug.cpu#35.sched_goidle
879048 ± 7% -93.2% 59608 ± 42% sched_debug.cpu#35.sched_count
876729 ± 7% -93.2% 59559 ± 42% sched_debug.cpu#35.nr_switches
192396 ± 6% -99.6% 796 ± 19% sched_debug.cpu#35.ttwu_local
144811 ± 7% -76.8% 33645 ± 44% sched_debug.cpu#36.nr_load_updates
198260 ± 3% -99.7% 681 ± 15% sched_debug.cpu#36.ttwu_local
3 ± 38% -100.0% 0 ± 0% sched_debug.cpu#36.cpu_load[2]
608120 ± 19% -99.7% 1607 ± 10% sched_debug.cpu#36.ttwu_count
272324 ± 10% +271.7% 1012309 ± 4% sched_debug.cpu#36.avg_idle
4 ± 47% -100.0% 0 ± 0% sched_debug.cpu#36.cpu_load[0]
3 ± 48% -100.0% 0 ± 0% sched_debug.cpu#36.cpu_load[1]
144880 ± 8% -75.5% 35491 ± 42% sched_debug.cpu#37.nr_load_updates
551377 ± 10% -99.7% 1487 ± 10% sched_debug.cpu#37.ttwu_count
193906 ± 7% -99.6% 680 ± 18% sched_debug.cpu#37.ttwu_local
277677 ± 3% +256.2% 989065 ± 1% sched_debug.cpu#37.avg_idle
4 ± 49% -100.0% 0 ± 0% sched_debug.cpu#38.cpu_load[1]
271911 ± 6% +259.7% 978143 ± 1% sched_debug.cpu#38.avg_idle
4 ± 37% -100.0% 0 ± 0% sched_debug.cpu#38.cpu_load[2]
588979 ± 20% -99.7% 1876 ± 49% sched_debug.cpu#38.ttwu_count
195832 ± 6% -99.7% 662 ± 13% sched_debug.cpu#38.ttwu_local
3 ± 42% -100.0% 0 ± 0% sched_debug.cpu#38.cpu_load[3]
859499 ± 7% -94.2% 49501 ± 48% sched_debug.cpu#39.nr_switches
197080 ± 5% -99.7% 616 ± 14% sched_debug.cpu#39.ttwu_local
144875 ± 8% -78.5% 31180 ± 38% sched_debug.cpu#39.nr_load_updates
861583 ± 7% -94.2% 50175 ± 46% sched_debug.cpu#39.sched_count
372809 ± 7% -93.4% 24588 ± 48% sched_debug.cpu#39.sched_goidle
585682 ± 15% -99.8% 1422 ± 11% sched_debug.cpu#39.ttwu_count
254598 ± 28% +291.0% 995522 ± 0% sched_debug.cpu#39.avg_idle
384496 ± 7% +624.1% 2784133 ± 20% sched_debug.cpu#4.sched_goidle
903009 ± 6% +517.8% 5579241 ± 20% sched_debug.cpu#4.sched_count
900237 ± 6% +519.7% 5579190 ± 20% sched_debug.cpu#4.nr_switches
556589 ± 9% -97.9% 11720 ± 21% sched_debug.cpu#4.ttwu_count
193650 ± 19% -94.2% 11155 ± 20% sched_debug.cpu#4.ttwu_local
279229 ± 6% +261.3% 1008863 ± 1% sched_debug.cpu#40.avg_idle
564514 ± 13% -99.8% 1408 ± 18% sched_debug.cpu#40.ttwu_count
2 ± 37% -100.0% 0 ± 0% sched_debug.cpu#40.cpu_load[0]
2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#40.cpu_load[4]
196449 ± 8% -99.7% 599 ± 25% sched_debug.cpu#40.ttwu_local
195020 ± 4% -99.7% 665 ± 32% sched_debug.cpu#41.ttwu_local
569391 ± 15% -99.7% 1443 ± 22% sched_debug.cpu#41.ttwu_count
250174 ± 24% +299.0% 998144 ± 0% sched_debug.cpu#41.avg_idle
7 ± 46% -100.0% 0 ± 0% sched_debug.cpu#41.load
271444 ± 4% +267.2% 996765 ± 3% sched_debug.cpu#42.avg_idle
3 ± 35% -100.0% 0 ± 0% sched_debug.cpu#42.cpu_load[2]
4 ± 43% -100.0% 0 ± 0% sched_debug.cpu#42.cpu_load[1]
3 ± 27% -100.0% 0 ± 0% sched_debug.cpu#42.cpu_load[3]
594265 ± 15% -99.8% 1417 ± 26% sched_debug.cpu#42.ttwu_count
198649 ± 7% -99.7% 578 ± 29% sched_debug.cpu#42.ttwu_local
270822 ± 9% +262.7% 982391 ± 2% sched_debug.cpu#43.avg_idle
3 ± 14% -100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[2]
595801 ± 15% -99.8% 1474 ± 35% sched_debug.cpu#43.ttwu_count
2 ± 24% -100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[3]
199411 ± 4% -99.7% 613 ± 22% sched_debug.cpu#43.ttwu_local
4 ± 25% -100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[1]
5 ± 23% -100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[0]
144742 ± 8% -79.2% 30152 ± 47% sched_debug.cpu#43.nr_load_updates
276311 ± 3% +258.0% 989070 ± 2% sched_debug.cpu#44.avg_idle
582806 ± 15% -99.8% 1362 ± 30% sched_debug.cpu#44.ttwu_count
144761 ± 8% -80.1% 28756 ± 45% sched_debug.cpu#44.nr_load_updates
144710 ± 7% -67.6% 46849 ± 27% sched_debug.cpu#45.nr_load_updates
851364 ± 13% -90.0% 85464 ± 31% sched_debug.cpu#45.sched_count
847235 ± 13% -89.9% 85423 ± 31% sched_debug.cpu#45.nr_switches
372059 ± 14% -88.6% 42469 ± 31% sched_debug.cpu#45.sched_goidle
203310 ± 19% -99.6% 843 ± 17% sched_debug.cpu#45.ttwu_local
281126 ± 7% +253.4% 993587 ± 5% sched_debug.cpu#45.avg_idle
573414 ± 12% -99.7% 1515 ± 21% sched_debug.cpu#45.ttwu_count
861853 ± 12% -92.4% 65635 ± 43% sched_debug.cpu#46.nr_switches
273192 ± 7% +268.5% 1006815 ± 2% sched_debug.cpu#46.avg_idle
864903 ± 12% -92.4% 65678 ± 43% sched_debug.cpu#46.sched_count
376696 ± 13% -91.3% 32592 ± 43% sched_debug.cpu#46.sched_goidle
552783 ± 9% -99.7% 1450 ± 14% sched_debug.cpu#46.ttwu_count
201314 ± 20% -99.6% 706 ± 19% sched_debug.cpu#46.ttwu_local
144703 ± 7% -74.6% 36762 ± 34% sched_debug.cpu#46.nr_load_updates
589 ± 34% -100.0% 0 ± 0% sched_debug.cpu#46.curr->pid
584555 ± 7% -99.7% 1599 ± 16% sched_debug.cpu#47.ttwu_count
203884 ± 19% -99.6% 723 ± 23% sched_debug.cpu#47.ttwu_local
144799 ± 7% -76.4% 34218 ± 46% sched_debug.cpu#47.nr_load_updates
274538 ± 6% +260.1% 988726 ± 1% sched_debug.cpu#47.avg_idle
3 ± 35% -100.0% 0 ± 0% sched_debug.cpu#48.cpu_load[1]
144673 ± 8% -78.2% 31538 ± 48% sched_debug.cpu#48.nr_load_updates
543724 ± 4% -99.7% 1672 ± 18% sched_debug.cpu#48.ttwu_count
199514 ± 22% -99.7% 684 ± 34% sched_debug.cpu#48.ttwu_local
2 ± 37% -100.0% 0 ± 0% sched_debug.cpu#48.cpu_load[2]
3 ± 44% -100.0% 0 ± 0% sched_debug.cpu#48.cpu_load[0]
280359 ± 8% +263.5% 1019216 ± 4% sched_debug.cpu#48.avg_idle
564757 ± 6% -99.7% 1635 ± 18% sched_debug.cpu#49.ttwu_count
203456 ± 21% -99.7% 710 ± 29% sched_debug.cpu#49.ttwu_local
144637 ± 7% -78.9% 30527 ± 47% sched_debug.cpu#49.nr_load_updates
302435 ± 19% +233.8% 1009483 ± 3% sched_debug.cpu#49.avg_idle
7 ± 35% -100.0% 0 ± 0% sched_debug.cpu#5.load
576004 ± 11% -99.5% 2984 ± 18% sched_debug.cpu#5.ttwu_count
184101 ± 10% -98.6% 2497 ± 17% sched_debug.cpu#5.ttwu_local
256732 ± 5% +131.1% 593404 ± 27% sched_debug.cpu#5.avg_idle
2 ± 49% -100.0% 0 ± 0% sched_debug.cpu#50.cpu_load[3]
200344 ± 20% -99.7% 688 ± 28% sched_debug.cpu#50.ttwu_local
2 ± 31% -100.0% 0 ± 0% sched_debug.cpu#50.cpu_load[2]
563146 ± 5% -99.7% 1908 ± 44% sched_debug.cpu#50.ttwu_count
2 ± 41% -100.0% 0 ± 0% sched_debug.cpu#50.cpu_load[1]
244904 ± 22% +302.4% 985599 ± 1% sched_debug.cpu#50.avg_idle
144490 ± 7% -79.9% 29035 ± 47% sched_debug.cpu#51.nr_load_updates
273633 ± 7% +265.2% 999259 ± 2% sched_debug.cpu#51.avg_idle
555030 ± 5% -99.7% 1398 ± 23% sched_debug.cpu#51.ttwu_count
202800 ± 21% -99.7% 617 ± 28% sched_debug.cpu#51.ttwu_local
204402 ± 20% -99.7% 672 ± 35% sched_debug.cpu#52.ttwu_local
6 ± 37% -100.0% 0 ± 0% sched_debug.cpu#52.cpu_load[2]
266105 ± 8% +268.6% 980753 ± 3% sched_debug.cpu#52.avg_idle
8 ± 40% -100.0% 0 ± 0% sched_debug.cpu#52.cpu_load[1]
4 ± 33% -100.0% 0 ± 0% sched_debug.cpu#52.cpu_load[3]
590313 ± 10% -99.8% 1394 ± 15% sched_debug.cpu#52.ttwu_count
246340 ± 20% +293.7% 969729 ± 5% sched_debug.cpu#53.avg_idle
580316 ± 10% -99.7% 1479 ± 17% sched_debug.cpu#53.ttwu_count
202868 ± 20% -99.7% 559 ± 24% sched_debug.cpu#53.ttwu_local
275898 ± 8% +251.8% 970687 ± 6% sched_debug.cpu#54.avg_idle
580835 ± 12% -99.8% 1420 ± 27% sched_debug.cpu#54.ttwu_count
203860 ± 21% -99.7% 669 ± 25% sched_debug.cpu#54.ttwu_local
561511 ± 12% -99.8% 1320 ± 32% sched_debug.cpu#55.ttwu_count
199859 ± 21% -99.7% 615 ± 44% sched_debug.cpu#55.ttwu_local
269166 ± 3% +267.9% 990163 ± 1% sched_debug.cpu#55.avg_idle
592219 ± 10% -99.8% 1394 ± 20% sched_debug.cpu#56.ttwu_count
203190 ± 20% -99.7% 622 ± 21% sched_debug.cpu#56.ttwu_local
285148 ± 8% +244.9% 983467 ± 2% sched_debug.cpu#56.avg_idle
289395 ± 8% +243.8% 994802 ± 1% sched_debug.cpu#57.avg_idle
201174 ± 22% -99.7% 588 ± 32% sched_debug.cpu#57.ttwu_local
561875 ± 9% -99.8% 1333 ± 32% sched_debug.cpu#57.ttwu_count
532410 ± 13% -99.8% 1321 ± 14% sched_debug.cpu#58.ttwu_count
199150 ± 23% -99.7% 623 ± 30% sched_debug.cpu#58.ttwu_local
281144 ± 9% +248.6% 979942 ± 4% sched_debug.cpu#58.avg_idle
206191 ± 20% -99.7% 576 ± 34% sched_debug.cpu#59.ttwu_local
290618 ± 4% +237.8% 981621 ± 2% sched_debug.cpu#59.avg_idle
568974 ± 8% -99.8% 1378 ± 21% sched_debug.cpu#59.ttwu_count
145823 ± 7% -17.5% 120332 ± 3% sched_debug.cpu#6.nr_load_updates
179817 ± 10% -99.1% 1693 ± 19% sched_debug.cpu#6.ttwu_local
546345 ± 8% -99.6% 2115 ± 21% sched_debug.cpu#6.ttwu_count
278607 ± 3% +199.1% 833241 ± 18% sched_debug.cpu#6.avg_idle
2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#6.cpu_load[3]
851921 ± 8% -40.4% 508151 ± 46% sched_debug.cpu#6.nr_switches
854405 ± 8% -40.5% 508206 ± 46% sched_debug.cpu#6.sched_count
1 ± 35% -100.0% 0 ± 0% sched_debug.cpu#6.cpu_load[4]
137215 ± 8% -92.7% 9982 ± 37% sched_debug.cpu#60.nr_load_updates
916934 ± 7% -39.4% 555767 ± 14% sched_debug.cpu#60.max_idle_balance_cost
496648 ± 21% +124.0% 1112535 ± 14% sched_debug.cpu#60.avg_idle
523174 ± 17% +90.8% 997981 ± 1% sched_debug.cpu#61.avg_idle
207882 ± 14% -99.9% 283 ± 41% sched_debug.cpu#61.ttwu_local
925690 ± 4% -45.3% 506045 ± 2% sched_debug.cpu#61.max_idle_balance_cost
303435 ± 20% -99.8% 571 ± 21% sched_debug.cpu#61.ttwu_count
203648 ± 13% -99.9% 158 ± 29% sched_debug.cpu#62.ttwu_local
904843 ± 5% -44.4% 502645 ± 1% sched_debug.cpu#62.max_idle_balance_cost
305661 ± 24% -99.9% 261 ± 30% sched_debug.cpu#62.ttwu_count
529 ± 32% -100.0% 0 ± 0% sched_debug.cpu#62.curr->pid
1080713 ± 10% -53.7% 500459 ± 0% sched_debug.cpu#63.max_idle_balance_cost
1699969 ± 10% -41.0% 1003536 ± 2% sched_debug.cpu#64.avg_idle
1096675 ± 9% -54.0% 504808 ± 2% sched_debug.cpu#64.max_idle_balance_cost
2374 ± 48% -91.1% 211 ± 27% sched_debug.cpu#64.ttwu_count
1073352 ± 9% -51.1% 524554 ± 7% sched_debug.cpu#65.max_idle_balance_cost
1599502 ± 9% -34.4% 1049932 ± 7% sched_debug.cpu#65.avg_idle
39322 ± 42% -72.4% 10848 ± 40% sched_debug.cpu#66.nr_load_updates
1409179 ± 19% -27.9% 1015680 ± 3% sched_debug.cpu#66.avg_idle
1076703 ± 7% -52.6% 510769 ± 4% sched_debug.cpu#66.max_idle_balance_cost
1086785 ± 9% -53.0% 510290 ± 4% sched_debug.cpu#67.max_idle_balance_cost
1411451 ± 16% -29.9% 989837 ± 2% sched_debug.cpu#68.avg_idle
38684 ± 43% -66.9% 12790 ± 34% sched_debug.cpu#68.nr_load_updates
1108217 ± 7% -54.9% 500229 ± 0% sched_debug.cpu#68.max_idle_balance_cost
1077891 ± 7% -47.7% 563733 ± 22% sched_debug.cpu#69.max_idle_balance_cost
566207 ± 7% -99.7% 1748 ± 21% sched_debug.cpu#7.ttwu_count
262075 ± 9% +236.5% 881956 ± 13% sched_debug.cpu#7.avg_idle
838568 ± 9% -53.9% 386340 ± 44% sched_debug.cpu#7.sched_count
357789 ± 9% -46.2% 192547 ± 44% sched_debug.cpu#7.sched_goidle
836222 ± 9% -53.8% 386289 ± 44% sched_debug.cpu#7.nr_switches
145736 ± 7% -22.7% 112707 ± 3% sched_debug.cpu#7.nr_load_updates
182859 ± 10% -99.2% 1443 ± 23% sched_debug.cpu#7.ttwu_local
1073432 ± 10% -51.3% 522486 ± 8% sched_debug.cpu#70.max_idle_balance_cost
1530831 ± 15% -35.2% 992013 ± 1% sched_debug.cpu#71.avg_idle
1054705 ± 6% -52.6% 500000 ± 0% sched_debug.cpu#71.max_idle_balance_cost
1069098 ± 9% -50.7% 527311 ± 9% sched_debug.cpu#72.max_idle_balance_cost
38445 ± 45% -61.5% 14811 ± 40% sched_debug.cpu#72.nr_load_updates
1356913 ± 13% -22.4% 1053465 ± 9% sched_debug.cpu#72.avg_idle
1332336 ± 10% -26.8% 975029 ± 6% sched_debug.cpu#73.avg_idle
1058495 ± 8% -51.8% 510063 ± 3% sched_debug.cpu#73.max_idle_balance_cost
2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#73.cpu_load[1]
1067890 ± 8% -52.7% 504615 ± 1% sched_debug.cpu#74.max_idle_balance_cost
682497 ± 27% -26.7% 500478 ± 0% sched_debug.cpu#76.max_idle_balance_cost
693411 ± 28% -27.4% 503241 ± 1% sched_debug.cpu#77.max_idle_balance_cost
734302 ± 27% -30.9% 507513 ± 2% sched_debug.cpu#78.max_idle_balance_cost
681179 ± 28% -21.9% 531938 ± 10% sched_debug.cpu#79.max_idle_balance_cost
540200 ± 8% -99.7% 1700 ± 13% sched_debug.cpu#8.ttwu_count
178504 ± 11% -99.2% 1373 ± 19% sched_debug.cpu#8.ttwu_local
145742 ± 7% -25.5% 108570 ± 3% sched_debug.cpu#8.nr_load_updates
361282 ± 8% -51.3% 176052 ± 43% sched_debug.cpu#8.sched_goidle
276970 ± 8% +230.5% 915429 ± 10% sched_debug.cpu#8.avg_idle
846491 ± 7% -58.3% 353331 ± 43% sched_debug.cpu#8.sched_count
843911 ± 7% -58.1% 353246 ± 43% sched_debug.cpu#8.nr_switches
1359 ± 48% -59.4% 552 ± 9% sched_debug.cpu#81.ttwu_count
682840 ± 27% -26.2% 503819 ± 1% sched_debug.cpu#81.max_idle_balance_cost
680252 ± 27% -24.4% 513984 ± 4% sched_debug.cpu#82.max_idle_balance_cost
688877 ± 33% -26.0% 509483 ± 3% sched_debug.cpu#83.max_idle_balance_cost
670276 ± 22% -24.9% 503665 ± 1% sched_debug.cpu#85.max_idle_balance_cost
680305 ± 22% -25.8% 504652 ± 1% sched_debug.cpu#86.max_idle_balance_cost
1390 ± 44% -65.2% 483 ± 6% sched_debug.cpu#87.ttwu_count
711200 ± 26% -29.2% 503212 ± 1% sched_debug.cpu#88.max_idle_balance_cost
649693 ± 26% -22.7% 502075 ± 0% sched_debug.cpu#89.max_idle_balance_cost
531933 ± 7% -99.7% 1700 ± 30% sched_debug.cpu#9.ttwu_count
180813 ± 10% -99.2% 1424 ± 27% sched_debug.cpu#9.ttwu_local
855358 ± 8% -58.0% 359635 ± 48% sched_debug.cpu#9.nr_switches
857752 ± 8% -58.1% 359679 ± 48% sched_debug.cpu#9.sched_count
366428 ± 8% -51.1% 179269 ± 48% sched_debug.cpu#9.sched_goidle
255284 ± 21% +256.0% 908885 ± 12% sched_debug.cpu#9.avg_idle
145661 ± 7% -28.0% 104899 ± 5% sched_debug.cpu#9.nr_load_updates
3 ± 33% -100.0% 0 ± 0% sched_debug.cpu#9.cpu_load[2]
2 ± 38% -100.0% 0 ± 0% sched_debug.cpu#9.cpu_load[3]
3 ± 40% -100.0% 0 ± 0% sched_debug.cpu#9.cpu_load[1]
880077 ± 25% -43.2% 500000 ± 0% sched_debug.cpu#90.max_idle_balance_cost
1270327 ± 11% -21.3% 1000000 ± 0% sched_debug.cpu#91.avg_idle
886286 ± 22% -43.6% 500000 ± 0% sched_debug.cpu#91.max_idle_balance_cost
861842 ± 25% -42.0% 500000 ± 0% sched_debug.cpu#92.max_idle_balance_cost
877600 ± 26% -37.3% 550542 ± 20% sched_debug.cpu#93.max_idle_balance_cost
872812 ± 25% -42.7% 500149 ± 0% sched_debug.cpu#94.max_idle_balance_cost
880573 ± 26% -43.0% 501897 ± 0% sched_debug.cpu#95.max_idle_balance_cost
898390 ± 24% -43.6% 506730 ± 2% sched_debug.cpu#96.max_idle_balance_cost
1233750 ± 15% -17.9% 1013461 ± 2% sched_debug.cpu#96.avg_idle
1279764 ± 17% -22.4% 993620 ± 1% sched_debug.cpu#97.avg_idle
887849 ± 23% -43.7% 500000 ± 0% sched_debug.cpu#97.max_idle_balance_cost
881541 ± 22% -43.0% 502208 ± 0% sched_debug.cpu#98.max_idle_balance_cost
883063 ± 25% -43.1% 502857 ± 1% sched_debug.cpu#99.max_idle_balance_cost
brickland3: Brickland Ivy Bridge-EX
Memory: 512G
stderr.netperf:send_omni:recv_data_failed:No_route_to_host
1 ++----------------*-----*--------*------*-----------------------------*
| : : : : |
| : : :: : |
0.8 ++ : : : : : : : : :|
| : : : : : : : : :|
| : : : : : : : : :|
0.6 ++ : : : : : : : : :|
| : : : : : : : : :|
0.4 ++ : : : : : : : : : |
| : : : : : : : : : |
| : : : : : : : : : |
0.2 ++ : : : : : : : : : |
| : : : : :: : : |
| : : : : : : : |
0 *+-*--*--*--*--*-----*-----*--*------*-----*--*--*--*--*--*--*--*--*--+
last_state.booting
1 ++------------------------------------------------------*-------------*
| : |
| :: |
0.8 ++ : : :|
| : : :|
| : : :|
0.6 ++ : : :|
| : : :|
0.4 ++ : : : |
| : : : |
| : : : |
0.2 ++ : : : |
| : : : |
| : : : |
0 *+--*--*---*--*---*--*---*--*---*--*---*--*---*--*---*------*--*---*--+
softirqs.TIMER
6e+06 ++------------------------------------------------------------------+
| |
5e+06 ++ * |
| .*. +: |
|.* *.*.*. .* * * *.* *.* *.* : .*.*.*.*.*.*.*
4e+06 *+ * : : : : : : : * : : * |
| : : :: : : : : : : : : |
3e+06 ++ : : : : : : : : : :: : : : |
| : : : : : : : : : : : : :: |
2e+06 ++ : : : : : : : : : : : : :: |
| : : : : : : : : : : :: |
| *: : * : : : : * :: |
1e+06 O+ O O O O O O O O O : O O : O O O OO O O O O |
| O O O* O * : : O : : O * |
0 ++----------O------------O---O-*-O-----*-O--------------------------+
softirqs.NET_RX
7e+07 ++------------------------------------------------------------------+
| |
6e+07 *+*.*.*.*.* * * * *.* *.* * *.*.* *.*.*.*.*.*.*.*
| : :: : : : : : : : : : : |
5e+07 ++ : :: : : : : : : : : : : |
| * : :: :: : : : : :: : : : |
4e+07 ++ : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : |
3e+07 ++ : : : : : : : : : : : : :: |
| : : : : : : : : : : : :: |
2e+07 ++ : : : : : : : : : : :: |
| : : :: * : : : : : :: |
1e+07 ++ *: : : : : : * : |
| : : : : : : : |
0 O+O-O-O-O-O-O-O-OO-O-O-O-O-O-O-O-O-O-O-O-O-O-O-O-O-OO-O-O-O-O-------+
softirqs.SCHED
4.5e+06 ++-----------------------------------------------*----------------+
| :: |
4e+06 ++ .*. * * :: |
3.5e+06 ++* *.* : * : * *.* *.* * *.* : *.**.*.*.*.*
* : +: : : : : : : : : : + |
3e+06 ++ * : :: :: : : : : : : : * |
2.5e+06 ++ : : : : : : : : : :: : : : |
| : : : : : : : : : :: : : : |
2e+06 ++ : : : : :: : : : :: : : : |
1.5e+06 ++ : : : : : : : : : : : : : |
| : : : : : : : : : : : : |
1e+06 ++ : : :: * : : : : : :: |
500000 ++ * : : : : : : * : |
O O OO O +O O O O O : O O :O O O O O O O OO |
0 ++O---O----O---O-*-O-*---O--O-*-O---O-*-O--O-------*--------------+
softirqs.RCU
1.8e+06 ++-----------------------------------------------*----------------+
| .** * .* .* +: .*. |
1.6e+06 *+ .*.* + .* : * * : * : * *.* : * **.*.*.*.*
1.4e+06 ++* * : : : : : : : : : : : |
| : :: :: : : : : : : : : |
1.2e+06 ++ : : : : : : : : : :: : : : |
1e+06 ++ : : : : : : : : : :: : : : |
| : : : : :: : : : :: : : : |
800000 ++ : : : : : : : : :: : : : |
600000 ++ : : : : : : : : : : : : |
| : : :: * : : : : : :: |
400000 ++ * : : : : : : * : |
200000 O+ O OO O +O O O O O : O O :O O O O O O OO |
| O O * * : : O : : O O * |
0 ++----O----O-------O-----O--O-*-O-----*-O-------------------------+
cpuidle.C1-IVT-4S.usage
4e+07 ++----------------------------------------------------------------+
O O OO O O O O O O O O O O O O O O |
3.5e+07 ++ O O O |
3e+07 ++ |
| |
2.5e+07 ++ |
*.*.*.*.* * * * *.* *.* * *.*. *. .**.*. .*.*
2e+07 ++ *. +: : : : : : : : : * : * * |
| * : :: :: : : : : :: : : : |
1.5e+07 ++ : : : : : : : : : :: : : : |
1e+07 ++ : : : : :: : : : :: : : : |
| : : : : : : : : : : : : : |
5e+06 ++ : : :: * : : : : : : : |
| *.: : : : : : * : |
0 ++O---O----O---O-*-O-*---O--O-*-O---O-*-O--O-------*--------------+
cpuidle.C6-IVT-4S.time
1.2e+11 ++----------------------------------------------------------------+
| |
1e+11 ++ * * |
| : : |
| : : * |
8e+10 ++ : : :: : |
| : : :: :: |
6e+10 ++ : : * : : : : |
| : : : : : : : |
4e+10 ++ : : : : : : : : |
O O OO O: :O O O O O O O O: O O O O O O OO |
| : : : : : : : |
2e+10 *+*.*.*.** * :.*.*.* *.* *.* * *.*.* *.*.**.*.*.*.*
| O * O + + + : O + + |
0 ++O--------O---O---------O--O-*-O---O-*-O----------*--------------+
turbostat.Avg_MHz
1800 ++-------------------------------------------------*-----*-----------+
|.*. .*.*.* * * * .* .* * *. +: + *.*. .*.*.|
1600 *+ * : : : : * : * : : : * : * * *
1400 ++ : : : : : : : : : : : : |
| : : : : :: : : : : :: : : : |
1200 ++ : : : : : : : : : : : : : : : : |
1000 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
800 ++ : : : : : : : : : : : : : : : : |
600 ++ : : : : : : : : : : : : : : : |
| : : : : : :: : : : : :: :: |
400 ++ * : : : : : : : : : : |
200 ++ : : : : : : : : : : |
O O O O O *.O O O * O O :O O : O * O O O O O O O O |
0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-----------------------+
turbostat.%Busy
60 ++---------------------------------------------------------------------+
| .*.*.* * * .*. |
50 *+*.* : * : * .* .* * *. +: * *.*. .*.*.|
| : : : : * : * : : : * : : * *
| : : : : : : : : : : : : |
40 ++ : : : : : :: : : : : :: : : : |
| : : : : : : : : : : : : : : : : |
30 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
20 ++ : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : :: |
| * : : :: : : : : : : : |
10 ++ : : : : : : : : : : |
O O O O O *.O O O * O O : O O : O * O O O O O O O O |
0 ++O---O-----O---O-*-O-*---O---O-*-O----O-*-O---O-----------------------+
turbostat.CPU%c1
50 *+-------------------------------------------------------*-------------+
45 ++* *.*.* * * * .* .* * *. : *.*.*. .*.*.|
| + + : : : : * : * : : : *.* : * *
40 ++ * : : : : : : : : : : : : |
35 ++ : : : : :: : : : : :: : : : |
| : : : : : : : : : : : : : : : : |
30 ++ : : : : : : : : : : : : : : : : |
25 ++ : : : : : : : : : : : : : : : : |
20 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : O : : : : : O |
15 O+ O O O : O : O: :O:O :: : : : : :: O O |
10 ++ * : : : : O O : O : O : O O O * O O |
| : : : : : : : : : |
5 ++ *.* : * : : O : : O |
0 ++O---O-----O---O---O-*---O---O-*-O------*-O---------------------------+
turbostat.CPU%c6
100 ++O---O---------O---O-*-----------------------O-----------------------+
90 ++ .* : * O * |
O O O O O * O O O : O O O O : O O O * O O |
80 ++ * : : : : O : O O O |
70 ++ : : : : : :: : : |
| : : : : : : : : : :: |
60 ++ : : : : : : : : : : : : |
50 ++ : : : : : : : : : : : : |
40 ++ : : : : : : : : : : : : |
| : : : : : : : : : : : : |
30 ++ : : : : : : : : : : : : |
20 ++ : : : : : : : : : : |
| : : : : : : : : : |
10 ++ .*. : : : : *. .*. : :.*.: : .*. .*
0 *+*---*-*-*-O-*-----*---*-O---O-*-O---*-*-O-*---*---*---*-*-*-*---*-*-+
turbostat.CoreTmp
60 ++---------------------------------------------------------------------+
*.*.*.*.*.* * * * .* .* *.*.* .*.*. |
50 ++ : : : : * : * : * : : *.* *.*.*.*
O O O : :O: O: :O: : : : O : : :O : : |
| O: : : : : :O: :O : O : : O O: : O O:O O O O |
40 ++ : : : : : : : : : :: : * |
| O O * O.* O * : : : O: : O O |
30 ++ * * : : : : * |
| : : : : |
20 ++ : : : : |
| : : : : |
| : : : : |
10 ++ : : : : |
| : : : : |
0 ++----------O-------------O---O-*-O------*-O---------------------------+
turbostat.PkgTmp
60 ++---------------------------------------------------------------------+
*.*.*.*.*.* * * * .* .* *.*.* .*.*. .*
50 ++ : : : : * : * : * : : * *.*.*.* |
| O O : : : O: :O: : : : O : : :O : :O |
O O: :O: : : :O: :O : O : : O O: : O O: O O O |
40 ++ : : : : : : : : : :: : * |
| O O * O.* O * : : : O: : O O |
30 ++ * * : : : : * |
| : : : : |
20 ++ : : : : |
| : : : : |
| : : : : |
10 ++ : : : : |
| : : : : |
0 ++----------O-------------O---O-*-O------*-O---------------------------+
turbostat.Pkg%pc2
70 ++---------------------------------------------------------------------+
| O |
60 ++ O O O O |
O O O O O O O |
50 ++ O O O O O O O |
| O |
40 ++ O |
| O O O O * O |
30 ++ .* : O * |
| * * : : : * * : |
20 ++ : : : : : :: : : : |
| : : : : : : : : : : : : |
10 ++ : : : : : : : : : : : : |
| .*. : : : : *. .*. : :.*.: : .*. |
0 *+*---*-*-*-O-*-----*---*-O---O-*-O----*-*-O-*---*---*---*-*-*-*---*-*-*
turbostat.PkgWatt
450 *+*---*-*-*---------*---*-----*-------*-------------------*-------*---+
| * : * : : * : * : * *.*.* * *.*.* *.*
400 ++ : : : : : : : : : : : : |
350 ++ : : :: :: : : : : : : : : |
| : : : : : : : : : : : :: : : : |
300 ++ : : : : : : : : : : : : : : : : |
250 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
200 ++ : : : : : : : : : : : : : : |
150 O+ O O O : O : O: O:O ::O :O :O :O :O: :O O O O O O O O |
| * : : : : : : : : : * |
100 ++ *.: : : : : O : : : |
50 ++O O O * O * * : : : : O |
| : : : : |
0 ++----------O-------------O---O-*-O-----*-O---------------------------+
turbostat.CorWatt
400 ++--------------------------------------------------------------------+
*.*. .*.*.* * * * .* .* * *. .* *.*.*.*. .*.*.|
350 ++ * : : : : * : * : : : * : : * *
300 ++ : : : : : : : : : : : : |
| : : : :: :: : : : : :: : : : |
250 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
200 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
150 ++ : : : : : : : : : : : : : : |
100 O+ O O : O : O: O: : : :O : :O : O O O |
| O * : : : O : O O : O : : O O O * O O |
50 ++ *.: : * : : O : : * |
| O O O * O * : : : : O |
0 ++----------O-------------O---O-*-O-----*-O---------------------------+
netperf.Throughput_tps
1600 ++-------------------------------------------------------------------+
*.*.*.*.*.* * * * *.* *.* *.*.* *.*.*. .*.*. .*
1400 ++ : : : : : : : : * : : : * * |
1200 ++ : : : : : : : : : : : : |
| : : : : : : : : : : : : |
1000 O+ O O O: :O: O: :O:O: :O :O O :O O: :O O O:O:O O O O |
| : : : : : : : : : : : : : : : : |
800 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
600 ++ : : : : : : : : : : : : : : : : |
400 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : |
200 ++ : : : : : : : : : : : |
| : : : : : : : : : : : |
0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+
netperf.time.user_time
1000 ++*-*----------------------------------------------------------------+
900 ++ : * *.*.*. .*.*.|
* : .*.*.* *. *. : : *.* *
800 ++ :.* : * * : * : * : : |
700 ++ * : : : : : : : :: : |
| : : : : : : : : : *.*.* : |
600 ++ : : : : : : : : : : : : : : |
500 ++ : : : : : : : : : : : : : : |
400 ++ : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : |
300 ++ : : : : : : : : : :: : : |
200 ++ : : : : : : : : : : :: |
| * : : * : : : : * : |
100 O+ O O O O +O O O O O :O O : O O O O O O O O O |
0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+
netperf.time.system_time
12000 ++------------------------------------------------------------------+
| .*. * .* |
10000 ++*.* *.* * : * *.* *.* *.* : .*.*.*.*.*.*.*
* + +: : : : : : : * : : * |
| * : : : : : : : : : : : |
8000 ++ : : : :: : : : : : : : : |
| : : : : : : : : : : : : : : |
6000 ++ : : : : : : : : : : : : : : |
| : : : : : : : : : : : : :: |
4000 ++ : : : : : : : : : : : :: |
| : : : : : : : : : : :: |
| :: :: * : : : : : :: |
2000 ++ *: : : : : : * : |
O O O O O O O O O O : O O : O O O OO O O O O |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
netperf.time.percent_of_cpu_this_job_got
4000 ++-------------------------------------------------*-----------------+
|.*. .*.*.* * * * .* .* *. +: .*.*.*. .*.*.|
3500 *+ * : : : : * : * : * : * : * * *
3000 ++ : : : : : : : : : : : : |
| : : : : :: : : : : : : : : |
2500 ++ : : : : : : : : : : : :: : : : |
| : : : : : : : : : : : : : : : : |
2000 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
1500 ++ : : : : : : : : : : : : : : : |
1000 ++ : : : : : :: : : : : : : : : |
| * : : : : : : : : : :: |
500 ++ : : : : : : : : : : |
O O O O O *.O O O * O O :O O : O * O O O O O O O O |
0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+
netperf.time.voluntary_context_switches
7e+07 *+*-*-*-*-*---*----*---*---*-*-----*-*---------*-*-*--*-*-*-*-*-*-*-*
| : :: : : : : : : * : : : |
6e+07 ++ : :: : : : : : : : : : : |
| * : : :: : : : : : : : : |
5e+07 O+ O O O O: O: :O:O: :O :O :O :O O: :O O OO: O O O |
| : : : : : : : : : : : : : :O |
4e+07 ++ : : : : : : : : : : : : : : |
| : : : : : : : : : : : : :: |
3e+07 ++ : : : : : : : : : : : : :: |
| : : : : : : : : : : :: |
2e+07 ++ : : : : : : : : : : :: |
| :: : * : : : : * : |
1e+07 ++ *: : : : : : : |
| : : : : : : : |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
netperf.time.involuntary_context_switches
35000 ++------------------------------------------------------------------+
| *. |
30000 ++ * + * .* |
| + : * * * : * : * *
25000 ++ * : .* * : : *.* *.* : : : : + + +|
| : * + +: : :: : : : : * : : : * *.* |
20000 ++ : * : : : : : : : : : : : : : |
*.* : : : : : : : : : :: : : * |
15000 ++ : : : : : : : : : : : : : : |
| : : : : :: : : : : : : :: |
10000 ++ : : : : : : : : : :: :: |
| : : : : * : : : : : :: |
5000 ++ *: : : : : : * ::O |
O O O O O O O O O O : O O : O O O OO O O O |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
time.user_time
1000 ++*-*----------------------------------------------------------------+
900 ++ : * *.*.*. .*.*.|
* : .*.*.* *. *. : : *.* *
800 ++ :.* : * * : * : * : : |
700 ++ * : : : : : : : :: : |
| : : : : : : : : : *.*.* : |
600 ++ : : : : : : : : : : : : : : |
500 ++ : : : : : : : : : : : : : : |
400 ++ : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : |
300 ++ : : : : : : : : : :: : : |
200 ++ : : : : : : : : : : :: |
| * : : * : : : : * : |
100 O+ O O O O +O O O O O :O O : O O O O O O O O O |
0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+
time.system_time
12000 ++------------------------------------------------------------------+
| .*. * .* |
10000 ++*.* *.* * : * *.* *.* *.* : .*.*.*.*.*.*.*
* + +: : : : : : : * : : * |
| * : : : : : : : : : : : |
8000 ++ : : : :: : : : : : : : : |
| : : : : : : : : : : : : : : |
6000 ++ : : : : : : : : : : : : : : |
| : : : : : : : : : : : : :: |
4000 ++ : : : : : : : : : : : :: |
| : : : : : : : : : : :: |
| :: :: * : : : : : :: |
2000 ++ *: : : : : : * : |
O O O O O O O O O O : O O : O O O OO O O O O |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
time.percent_of_cpu_this_job_got
4000 ++-------------------------------------------------*-----------------+
|.*. .*.*.* * * * .* .* *. +: .*.*.*. .*.*.|
3500 *+ * : : : : * : * : * : * : * * *
3000 ++ : : : : : : : : : : : : |
| : : : : :: : : : : : : : : |
2500 ++ : : : : : : : : : : : :: : : : |
| : : : : : : : : : : : : : : : : |
2000 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
1500 ++ : : : : : : : : : : : : : : : |
1000 ++ : : : : : :: : : : : : : : : |
| * : : : : : : : : : :: |
500 ++ : : : : : : : : : : |
O O O O O *.O O O * O O :O O : O * O O O O O O O O |
0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+
time.voluntary_context_switches
7e+07 *+*-*-*-*-*---*----*---*---*-*-----*-*---------*-*-*--*-*-*-*-*-*-*-*
| : :: : : : : : : * : : : |
6e+07 ++ : :: : : : : : : : : : : |
| * : : :: : : : : : : : : |
5e+07 O+ O O O O: O: :O:O: :O :O :O :O O: :O O OO: O O O |
| : : : : : : : : : : : : : :O |
4e+07 ++ : : : : : : : : : : : : : : |
| : : : : : : : : : : : : :: |
3e+07 ++ : : : : : : : : : : : : :: |
| : : : : : : : : : : :: |
2e+07 ++ : : : : : : : : : : :: |
| :: : * : : : : * : |
1e+07 ++ *: : : : : : : |
| : : : : : : : |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
time.involuntary_context_switches
35000 ++------------------------------------------------------------------+
| *. |
30000 ++ * + * .* |
| + : * * * : * : * *
25000 ++ * : .* * : : *.* *.* : : : : + + +|
| : * + +: : :: : : : : * : : : * *.* |
20000 ++ : * : : : : : : : : : : : : : |
*.* : : : : : : : : : :: : : * |
15000 ++ : : : : : : : : : : : : : : |
| : : : : :: : : : : : : :: |
10000 ++ : : : : : : : : : :: :: |
| : : : : * : : : : : :: |
5000 ++ *: : : : : : * ::O |
O O O O O O O O O O : O O : O O O OO O O O |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
vmstat.procs.r
50 ++----*---*------------------------------------------------------------+
| + * : * * * * * *. .* .*.*.* *. |
*.*.* : : : : +: +: * : * : * + + *.*
40 ++ : : : : * : * : : : : : * |
| : : : : :: : : : : : : : : |
| : : : : : : : : : : : :: : : : |
30 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
20 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : |
| : : : : : :: : : : : : : :: |
10 ++ * : : : : : : : : : : |
| : : : : : : : : : : |
O O O O O *.O O O * O O : O O : O * O O O O O O O O |
0 ++O---O-----O---O-*-O-*---O---O-*-O----O-*-O---O-----------------------+
vmstat.system.in
500000 ++-----------------------------------------------------------------+
450000 *+*. .*.*.* * * * .* .* * *. .* *.**.*. .*.*.|
| * : : : : * : * : : : * : : * *
400000 ++ : : : : : : : : : : : : |
350000 ++ : : : : :: : : : : :: : : : |
| : : : : : : : : : : : : : : : : |
300000 ++ :: : : : : : : : : : : : : : : |
250000 ++ :: : : : : : : : : : : : : : : |
200000 ++ :: : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : |
150000 ++ : : : : : :: : : : : :: : : |
100000 ++ * : : : : : : : : : : |
| : : : : : : : : : : |
50000 O+ O O O O *.O O O * O O :O O : O * O O O O O OO O |
0 ++O---O----O---O-*-O-*---O---O-*-O--O-*-O---O-------*--------------+
numa-numastat.node0.numa_hit
1.6e+06 ++----------------------------------------------------------------+
| O O O O O |
1.4e+06 O+ O O O O O O O O O O O O OO |
| O |
1.2e+06 ++ |
1e+06 ++ |
| |
800000 ++ |
| * * |
600000 ++ :: : |
400000 ++ : : : : |
| .*. : :.* : *. *. .*. .*.*. |
200000 *+*.* ** * : *. .* * : * * *.*.* *.*.** *.*
| : + * + : + : + + |
0 ++O---O----O---O-*-O-----O--O-*-O---O-*-O--O-------*--------------+
numa-numastat.node0.local_node
1.6e+06 ++----------------------------------------------------------------+
| O O O O O |
1.4e+06 O+ O O O O O O O O O O O O OO |
| O |
1.2e+06 ++ |
1e+06 ++ |
| |
800000 ++ |
| * * |
600000 ++ :: : |
400000 ++ : : : : |
| .*. : :.* : *. *. .*. .*.*. |
200000 *+*.* ** * : *. .* * : * * *.*.* *.*.** *.*
| : + * + : + : + + |
0 ++O---O----O---O-*-O-----O--O-*-O---O-*-O--O-------*--------------+
numa-vmstat.node0.numa_hit
1e+06 ++-----------------------------------------------------------------+
900000 ++ O |
| O O O O OO |
800000 O+ O O O O O O O O O O O O |
700000 ++ O |
| |
600000 ++ |
500000 ++ * * |
400000 ++ :: : : |
| .*. : : : : .*. .*. |
300000 *+*.* *.* *.*. .*. .* *.* *.* * *.*.* *.**.*.* *.*
200000 ++ O * O * : : : : + + |
| O O : : O: : O * |
100000 ++ : : : : |
0 ++---------O-------------O---O-*-O----*-O--------------------------+
numa-vmstat.node0.numa_local
900000 ++------------------------------------------------O----------------+
| O O O O OO |
800000 O+ O O O O O O O O O |
700000 ++ O O O O |
| |
600000 ++ |
500000 ++ |
| * * |
400000 ++ :: : : |
300000 ++ : : : : .* .*. |
|.*.*.*.*.* *.* *. .* *.* *.* * + .*.* *.**.*.* *.*
200000 *+ + + * : : : : * + + |
100000 ++O O O * O : : O: : O * |
| : : : : |
0 ++---------O-------------O---O-*-O----*-O--------------------------+
proc-vmstat.numa_hit
3e+06 ++----------------------------------------------------------------+
| * |
2.5e+06 ++ : |
| : : |
O O OO: :O O O O *O O O O O O O O O OO |
2e+06 ++ : : :: O O |
| : : : : *. |
1.5e+06 ++ .*.*.** *. * * *.* *.* + *.*.* .*.* .*.*. .*
*.* * : : : : : * : * * * |
1e+06 ++ : : : : : : : : : : |
| : : : : : : : : : : |
| : : : : : : : : : |
500000 ++ : * : : : : : : |
| O * O : : : : O : |
0 ++O--------O---O---------O--O-*-O---O-*-O----------*--------------+
proc-vmstat.numa_local
3e+06 ++----------------------------------------------------------------+
| * |
2.5e+06 ++ : |
| : : |
O O OO: :O O O O *O O O O O O O O O OO |
2e+06 ++ : : :: O O |
| : : : : *. |
1.5e+06 ++ .*.*.** *. * * *.* *.* + *.*.* .*.* .*.*. .*
*.* * : : : : : * : * * * |
1e+06 ++ : : : : : : : : : : |
| : : : : : : : : : : |
| : : : : : : : : : |
500000 ++ : * : : : : : : |
| O * O : : : : O : |
0 ++O--------O---O---------O--O-*-O---O-*-O----------*--------------+
proc-vmstat.pgalloc_dma32
350000 ++-----------------------------------------------------------------+
O O O O O O O O O O O O |
300000 ++ O O O O O O O O |
| O |
250000 ++ |
| |
200000 ++ * * |
*. + + + *.* * * .* .* * *.*.* .**. .*.*
150000 ++*.* * : : :: * : * : : : : * *.*.* |
| : : : : : + : : : : : : : : |
100000 ++ : : : : * : : : : : : : : |
| : : : : : : : : : : : |
50000 ++ * : : : : : : : * : : |
| +: : : : : : : |
0 ++O---O----O---O-*-O-*---O---O-*-O--O-*-O---O-------*--------------+
proc-vmstat.pgalloc_normal
1.2e+07 ++----------------------------------------------------------------+
O O O O O O O O O O O O O O O O |
1e+07 ++ O O O O O |
| |
| |
8e+06 ++ |
| .*. |
6e+06 *+*.*.*.** * * * *.* *.* *.*.* *.*.**.*.*.*.*
| : : :: : : : : * : : : |
4e+06 ++ : : : : :: : : : :: : : : |
| : : : : * : : : :: : : : |
| : : : : : : : : * : : |
2e+06 ++ * : : : : : : : : : |
| +: : : : : : : |
0 ++O---O----O---O-*-O-*---O--O-*-O---O-*-O--O-------*--------------+
proc-vmstat.pgfree
1.2e+07 ++------O------------O----------------O--O-----O-O-----O-O--------+
O O O O O O O O O O O O |
1e+07 ++ O |
| |
| |
8e+06 ++ |
| .*. .*. |
6e+06 *+*.* ** * * * *.* *.* * *.*.* *.*.**.*.*.*.*
| : : :: : : : : :: : : : |
4e+06 ++ : : : : :: : : : :: : : : |
| : : : : * : : : : :: : : |
| * : : : : : : : * : : |
2e+06 ++ : : : : : : : : : |
| :: * : : : : : |
0 ++O---O----O---O-*-O-----O--O-*-O---O-*-O--O-------*--------------+
proc-vmstat.numa_hint_faults
70000 ++------------------------------------------------------------------+
| |
60000 ++ * * * |
| +: +: .*.* :+ .*. |
50000 ++ .* *. * : * : * : : *.* *.*.*
|.* + + *.*. * * : : : : : : : |
40000 *+ * * : :: : : : : * : : * |
| : : :: : : : : :: : : : |
30000 ++ : : : : : : : : :: : : : |
| : : : : * : : : : : ::O |
20000 ++ O O O *O: :O:O : :O : : * O O OO: O O |
O O : : : : O : O :O :: O |
10000 ++ : : O : : : : :: |
| : : : : : : : |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
proc-vmstat.numa_hint_faults_local
60000 ++------------------------------------------------------------------+
| * * * |
50000 ++ +: +: :: |
| .* * : * : .*.* : : .*. |
| * + .*. * * : : : : * : : *.* *.*.*
40000 ++ * *.*.* : : : : : : : : : |
* : : :: : : : : * : : * |
30000 ++ : : : : : : : : : : : : : |
| : : : : : : : : : :: : : : |
20000 ++ : : : : : : : : : : : :: |
| O O O *O: :O:O * : :O : : : O O:O O O |
O O : : : : O : : O :O * O O:: O |
10000 ++ : : O : : : :: |
| : : : : : : : |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
[*] bisect-good sample
[O] bisect-bad sample
To reproduce:
apt-get install ruby
git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
cd lkp-tests
bin/setup-local job.yaml # the job file attached in this email
bin/run-local job.yaml
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Ying Huang
[-- Attachment #2: job.yaml --]
[-- Type: text/plain, Size: 3379 bytes --]
---
LKP_SERVER: inn
LKP_CGI_PORT: 80
testcase: netperf
default-monitors:
wait: pre-test
uptime:
iostat:
vmstat:
numa-numastat:
numa-vmstat:
numa-meminfo:
proc-vmstat:
proc-stat:
interval: 10
meminfo:
slabinfo:
interrupts:
lock_stat:
latency_stats:
softirqs:
bdi_dev_mapping:
diskstats:
nfsstat:
cpuidle:
cpufreq-stats:
turbostat:
pmeter:
sched_debug:
interval: 60
default-watchdogs:
watch-oom:
watchdog:
cpufreq_governor: performance
model: Westmere-EX
nr_cpu: 80
memory: 128G
hdd_partitions:
swap_partitions:
pmeter_server: lkp-st01
pmeter_device: yokogawa-wt310
commit: 0eab523874ab9850ec14f68751758dfb275361b9
nr_hdd_partitions: 0
rootfs_partition:
runtime: 300s
nr_threads: 200%
perf-profile:
freq: 800
cluster: cs-lkp-wsx02
if role server:
netserver:
if role client:
netperf:
test: TCP_RR
testbox: brickland3
tbox_group: brickland3
kconfig: x86_64-rhel
enqueue_time: 2015-06-13 01:32:11.981026644 +08:00
user: lkp
queue: cs
compiler: gcc-4.9
head_commit: 54896f27dd5c7e83c37c85eb1093f14543af2bfa
base_commit: d4a4f75cd8f29cd9464a5a32e9224a91571d6649
branch: next/master
rootfs: debian-x86_64-2015-02-07.cgz
result_root: "/result/netperf/performance-300s-200%-TCP_RR/brickland3/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/0eab523874ab9850ec14f68751758dfb275361b9/0"
direct_macs: a0:36:9f:18:d6:aa
direct_ips: 10.167.0.2
job_file: "/lkp/scheduled/brickland3/cs_netperf-performance-300s-200%-TCP_RR-debian-x86_64-2015-02-07.cgz-x86_64-rhel-0eab523874ab9850ec14f68751758dfb275361b9-0-20150613-5146-1kuhia6.yaml"
max_uptime: 1500
initrd: "/osimage/debian/debian-x86_64-2015-02-07.cgz"
bootloader_append:
- root=/dev/ram0
- user=lkp
- job=/lkp/scheduled/brickland3/cs_netperf-performance-300s-200%-TCP_RR-debian-x86_64-2015-02-07.cgz-x86_64-rhel-0eab523874ab9850ec14f68751758dfb275361b9-0-20150613-5146-1kuhia6.yaml
- ARCH=x86_64
- kconfig=x86_64-rhel
- branch=next/master
- commit=0eab523874ab9850ec14f68751758dfb275361b9
- BOOT_IMAGE=/pkg/linux/x86_64-rhel/gcc-4.9/0eab523874ab9850ec14f68751758dfb275361b9/vmlinuz-4.1.0-rc7-07770-g0eab523
- max_uptime=1500
- RESULT_ROOT=/result/netperf/performance-300s-200%-TCP_RR/brickland3/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/0eab523874ab9850ec14f68751758dfb275361b9/0
- LKP_SERVER=inn
- |2-
earlyprintk=ttyS0,115200 systemd.log_level=err
debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100
panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0
console=ttyS0,115200 console=tty0 vga=normal
rw
lkp_initrd: "/lkp/lkp/lkp-x86_64.cgz"
modules_initrd: "/pkg/linux/x86_64-rhel/gcc-4.9/0eab523874ab9850ec14f68751758dfb275361b9/modules.cgz"
bm_initrd: "/osimage/deps/debian-x86_64-2015-02-07.cgz/lkp.cgz,/osimage/deps/debian-x86_64-2015-02-07.cgz/run-ipconfig.cgz,/osimage/deps/debian-x86_64-2015-02-07.cgz/turbostat.cgz,/lkp/benchmarks/turbostat.cgz,/osimage/deps/debian-x86_64-2015-02-07.cgz/netperf.cgz,/lkp/benchmarks/netperf.cgz"
node_roles: client
kernel: "/pkg/linux/x86_64-rhel/gcc-4.9/0eab523874ab9850ec14f68751758dfb275361b9/vmlinuz-4.1.0-rc7-07770-g0eab523"
dequeue_time: 2015-06-13 01:32:30.522650807 +08:00
job_state: finished
loadavg: 2.33 6.80 4.32 1/1045 12657
start_time: '1434130591'
end_time: '1434130922'
version: "/lkp/lkp/.src-20150612-214517"
[-- Attachment #3: reproduce --]
[-- Type: text/plain, Size: 9476 bytes --]
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu10/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu100/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu101/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu102/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu103/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu104/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu105/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu106/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu107/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu108/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu109/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu11/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu110/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu111/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu112/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu113/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu114/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu115/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu116/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu117/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu118/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu119/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu12/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu13/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu14/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu15/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu16/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu17/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu18/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu19/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu20/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu21/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu22/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu23/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu24/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu25/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu26/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu27/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu28/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu29/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu30/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu31/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu32/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu33/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu34/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu35/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu36/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu37/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu38/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu39/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu40/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu41/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu42/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu43/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu44/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu45/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu46/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu47/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu48/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu49/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu5/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu50/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu51/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu52/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu53/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu54/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu55/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu56/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu57/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu58/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu59/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu6/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu60/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu61/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu62/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu63/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu64/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu65/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu66/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu67/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu68/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu69/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu7/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu70/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu71/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu72/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu73/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu74/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu75/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu76/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu77/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu78/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu79/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu8/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu80/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu81/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu82/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu83/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu84/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu85/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu86/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu87/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu88/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu89/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu9/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu90/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu91/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu92/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu93/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu94/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu95/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu96/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu97/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu98/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu99/cpufreq/scaling_governor
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps
2015-06-15 7:02 [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps Huang Ying
@ 2015-06-15 17:52 ` Thomas Gleixner
2015-06-16 2:09 ` Jiang Liu
0 siblings, 1 reply; 6+ messages in thread
From: Thomas Gleixner @ 2015-06-15 17:52 UTC (permalink / raw)
To: Huang Ying; +Cc: Jiang Liu, LKML, LKP ML
On Mon, 15 Jun 2015, Huang Ying wrote:
> FYI, we noticed the below changes on
>
> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
> commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts")
>
I really appreciate this testing effort, but the information provided
is not really helpful.
I asked this before. Can you pretty please, upload ALL relevant
information (.config, full dmesg, below stats, /proc/interrupts ...)
to some place where everyone interested can download them?
Then themail contains a useful link instead of 200k waste of network bandwidth.
Thanks,
tglx
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps
2015-06-15 17:52 ` Thomas Gleixner
@ 2015-06-16 2:09 ` Jiang Liu
2015-06-16 5:41 ` Huang Ying
2015-06-16 6:13 ` Thomas Gleixner
0 siblings, 2 replies; 6+ messages in thread
From: Jiang Liu @ 2015-06-16 2:09 UTC (permalink / raw)
To: Thomas Gleixner, Huang Ying; +Cc: LKML, LKP ML
On 2015/6/16 1:52, Thomas Gleixner wrote:
> On Mon, 15 Jun 2015, Huang Ying wrote:
>
>> FYI, we noticed the below changes on
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
>> commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts")
>>
>
> I really appreciate this testing effort, but the information provided
> is not really helpful.
>
> I asked this before. Can you pretty please, upload ALL relevant
> information (.config, full dmesg, below stats, /proc/interrupts ...)
> to some place where everyone interested can download them?
>
> Then themail contains a useful link instead of 200k waste of network bandwidth.
Hi Ying and Thomas,
I guess this report discloses a regression in hierarchy
irqdomain, and which should have been fixed by the patch posted at:
lkml.org/lkml/2015/6/1/80
The root cause is that, with hierarchy irqdomain enabled,
there are multiple irq_data associated with one irq. And function
irq_move_irq() on x86 uses a wrong copy of irq_data to check
whether there's pending irq migration operation. So all irq migration
/set_affinity operations will get pending for ever. This may
affect network performance due to interrupt load balance issue.
And the patch set posted at
www.gossamer-threads.com/lists/linux/kernel/2185533
should have solved all such regressions.
Thanks!
Gerry
>
> Thanks,
>
> tglx
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps
2015-06-16 2:09 ` Jiang Liu
@ 2015-06-16 5:41 ` Huang Ying
2015-06-16 5:51 ` Jiang Liu
2015-06-16 6:13 ` Thomas Gleixner
1 sibling, 1 reply; 6+ messages in thread
From: Huang Ying @ 2015-06-16 5:41 UTC (permalink / raw)
To: Jiang Liu; +Cc: Thomas Gleixner, LKML, LKP ML
Hi, Gerry,
On Tue, 2015-06-16 at 10:09 +0800, Jiang Liu wrote:
> On 2015/6/16 1:52, Thomas Gleixner wrote:
> > On Mon, 15 Jun 2015, Huang Ying wrote:
> >
> >> FYI, we noticed the below changes on
> >>
> >> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
> >> commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts")
> >>
> >
> > I really appreciate this testing effort, but the information provided
> > is not really helpful.
> >
> > I asked this before. Can you pretty please, upload ALL relevant
> > information (.config, full dmesg, below stats, /proc/interrupts ...)
> > to some place where everyone interested can download them?
> >
> > Then themail contains a useful link instead of 200k waste of network bandwidth.
>
> Hi Ying and Thomas,
> I guess this report discloses a regression in hierarchy
> irqdomain, and which should have been fixed by the patch posted at:
> lkml.org/lkml/2015/6/1/80
> The root cause is that, with hierarchy irqdomain enabled,
> there are multiple irq_data associated with one irq. And function
> irq_move_irq() on x86 uses a wrong copy of irq_data to check
> whether there's pending irq migration operation. So all irq migration
> /set_affinity operations will get pending for ever. This may
> affect network performance due to interrupt load balance issue.
> And the patch set posted at
> www.gossamer-threads.com/lists/linux/kernel/2185533
> should have solved all such regressions.
Thanks for your information. Do you have a git tree for this patchset?
Best Regards,
Huang, Ying
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps
2015-06-16 5:41 ` Huang Ying
@ 2015-06-16 5:51 ` Jiang Liu
0 siblings, 0 replies; 6+ messages in thread
From: Jiang Liu @ 2015-06-16 5:51 UTC (permalink / raw)
To: Huang Ying; +Cc: Thomas Gleixner, LKML, LKP ML
On 2015/6/16 13:41, Huang Ying wrote:
> Hi, Gerry,
>
> On Tue, 2015-06-16 at 10:09 +0800, Jiang Liu wrote:
>> On 2015/6/16 1:52, Thomas Gleixner wrote:
>>> On Mon, 15 Jun 2015, Huang Ying wrote:
>> lkml.org/lkml/2015/6/1/80
>> The root cause is that, with hierarchy irqdomain enabled,
>> there are multiple irq_data associated with one irq. And function
>> irq_move_irq() on x86 uses a wrong copy of irq_data to check
>> whether there's pending irq migration operation. So all irq migration
>> /set_affinity operations will get pending for ever. This may
>> affect network performance due to interrupt load balance issue.
>> And the patch set posted at
>> www.gossamer-threads.com/lists/linux/kernel/2185533
>> should have solved all such regressions.
>
> Thanks for your information. Do you have a git tree for this patchset?
Hi Ying,
You may access the branch at:
https://github.com/jiangliu/linux test/irq_data_v3
or view at url:
https://github.com/jiangliu/linux/tree/test/irq_data_v3
Thanks!
Gerry
>
> Best Regards,
> Huang, Ying
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps
2015-06-16 2:09 ` Jiang Liu
2015-06-16 5:41 ` Huang Ying
@ 2015-06-16 6:13 ` Thomas Gleixner
1 sibling, 0 replies; 6+ messages in thread
From: Thomas Gleixner @ 2015-06-16 6:13 UTC (permalink / raw)
To: Jiang Liu; +Cc: Huang Ying, LKML, LKP ML
On Tue, 16 Jun 2015, Jiang Liu wrote:
> On 2015/6/16 1:52, Thomas Gleixner wrote:
> > On Mon, 15 Jun 2015, Huang Ying wrote:
> >
> >> FYI, we noticed the below changes on
> >>
> >> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
> >> commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts")
> >>
> >
> > I really appreciate this testing effort, but the information provided
> > is not really helpful.
> >
> > I asked this before. Can you pretty please, upload ALL relevant
> > information (.config, full dmesg, below stats, /proc/interrupts ...)
> > to some place where everyone interested can download them?
> >
> > Then themail contains a useful link instead of 200k waste of network bandwidth.
>
> Hi Ying and Thomas,
> I guess this report discloses a regression in hierarchy
> irqdomain, and which should have been fixed by the patch posted at:
> lkml.org/lkml/2015/6/1/80
> The root cause is that, with hierarchy irqdomain enabled,
> there are multiple irq_data associated with one irq. And function
> irq_move_irq() on x86 uses a wrong copy of irq_data to check
> whether there's pending irq migration operation. So all irq migration
> /set_affinity operations will get pending for ever. This may
> affect network performance due to interrupt load balance issue.
> And the patch set posted at
> www.gossamer-threads.com/lists/linux/kernel/2185533
> should have solved all such regressions.
Next time, please send fixes which need to be applied right away
seperate from patch series which touch the world and some more.
Thanks,
tglx
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2015-06-16 6:13 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-06-15 7:02 [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps Huang Ying
2015-06-15 17:52 ` Thomas Gleixner
2015-06-16 2:09 ` Jiang Liu
2015-06-16 5:41 ` Huang Ying
2015-06-16 5:51 ` Jiang Liu
2015-06-16 6:13 ` Thomas Gleixner
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).