Greeting, FYI, we noticed a 9.4% improvement of netperf.Throughput_Mbps due to commit: commit: f6005b0d154f930116d3634a386723b64d744b53 ("[RFC PATCH v2] sched/fair: select idle cpu from idle cpumask in sched domain") url: https://github.com/0day-ci/linux/commits/Aubrey-Li/sched-fair-select-idle-cpu-from-idle-cpumask-in-sched-domain/20200916-123413 base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 848785df48835eefebe0c4eb5da7690690b0a8b7 in testcase: netperf on test machine: 16 threads Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz with 32G memory with following parameters: ip: ipv4 runtime: 300s nr_threads: 50% cluster: cs-localhost send_size: 5K test: TCP_SENDFILE cpufreq_governor: performance ucode: 0xd6 test-description: Netperf is a benchmark that can be use to measure various aspect of networking performance. test-url: http://www.netperf.org/netperf/ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/send_size/tbox_group/test/testcase/ucode: cs-localhost/gcc-9/performance/ipv4/x86_64-rhel-8.3/50%/debian-10.4-x86_64-20200603.cgz/300s/5K/lkp-cfl-e1/TCP_SENDFILE/netperf/0xd6 commit: 848785df48 ("sched/topology: Move sd_flag_debug out of #ifdef CONFIG_SYSCTL") f6005b0d15 ("sched/fair: select idle cpu from idle cpumask in sched domain") 848785df48835eef f6005b0d154f930116d3634a386 ---------------- --------------------------- %stddev %change %stddev \ | \ 23570 +9.4% 25777 netperf.Throughput_Mbps 188561 +9.4% 206218 netperf.Throughput_total_Mbps 137877 ± 17% +228.2% 452511 ± 3% netperf.time.involuntary_context_switches 115.09 +24.4% 143.19 ± 3% netperf.time.user_time 9503 ± 3% -22.3% 7381 netperf.time.voluntary_context_switches 1.381e+09 +9.4% 1.51e+09 netperf.workload 16162 +6.0% 17137 ± 5% meminfo.VmallocUsed 3.11 +0.6 3.75 ± 2% mpstat.cpu.all.usr% 9051 ± 2% +17.3% 10617 ± 6% slabinfo.anon_vma.active_objs 9173 ± 2% +15.7% 10617 ± 6% slabinfo.anon_vma.num_objs 850734 +4.6% 889611 vmstat.system.cs 35634 -4.9% 33900 vmstat.system.in 7153 -4.2% 6855 ± 3% proc-vmstat.nr_shmem 352511 -1.2% 348406 proc-vmstat.numa_hit 352511 -1.2% 348406 proc-vmstat.numa_local 406378 -2.5% 396360 proc-vmstat.pgalloc_normal 387604 -2.5% 377819 proc-vmstat.pgfree 3.49e+08 -9.9% 3.145e+08 cpuidle.C1.time 391753 ± 22% -93.9% 23878 ± 36% cpuidle.C1E.usage 2954769 ± 10% -75.1% 735822 ± 62% cpuidle.C3.time 222592 ± 16% -93.0% 15492 ± 64% cpuidle.C3.usage 41229831 ± 7% -20.3% 32851748 ± 3% cpuidle.C6.time 166376 ± 7% -60.6% 65513 ± 20% cpuidle.C6.usage 7.21 ± 48% +98.3% 14.29 ± 7% sched_debug.cfs_rq:/.load_avg.min 405.52 ± 3% -13.1% 352.23 ± 5% sched_debug.cfs_rq:/.runnable_avg.stddev 47371 ± 53% +99.2% 94370 ± 20% sched_debug.cfs_rq:/.spread0.max 403.79 ± 3% -13.2% 350.37 ± 5% sched_debug.cfs_rq:/.util_avg.stddev 517032 ± 9% -48.5% 266470 ± 10% sched_debug.cpu.avg_idle.max 136032 ± 11% -46.3% 73046 ± 10% sched_debug.cpu.avg_idle.stddev 0.44 ± 8% +64.5% 0.72 ± 7% sched_debug.cpu.clock.stddev 1120 ± 10% +34.6% 1508 ± 18% sched_debug.cpu.clock_task.stddev -11.38 -33.0% -7.62 sched_debug.cpu.nr_uninterruptible.min 6.25 ± 11% -23.1% 4.81 ± 13% sched_debug.cpu.nr_uninterruptible.stddev 13859390 ± 2% +15.9% 16063226 ± 2% softirqs.CPU0.NET_RX 11807 ± 5% +25.8% 14858 ± 11% softirqs.CPU0.RCU 13544571 ± 3% +15.8% 15687040 ± 7% softirqs.CPU1.NET_RX 47538 ± 4% -8.2% 43638 ± 5% softirqs.CPU10.SCHED 13974031 ± 4% +10.4% 15430208 ± 4% softirqs.CPU11.NET_RX 10919 ± 2% +14.5% 12503 ± 3% softirqs.CPU11.RCU 10993 ± 5% +9.2% 12002 ± 7% softirqs.CPU12.RCU 19974 +10.0% 21969 ± 5% softirqs.CPU12.TIMER 14007148 ± 2% +10.0% 15410848 softirqs.CPU13.NET_RX 10527 ± 5% +14.4% 12046 ± 4% softirqs.CPU13.RCU 13584339 +19.6% 16252320 ± 3% softirqs.CPU14.NET_RX 13514272 +18.2% 15968053 softirqs.CPU15.NET_RX 10914 ± 5% +13.0% 12333 ± 4% softirqs.CPU15.RCU 44846 ± 2% +9.5% 49085 ± 3% softirqs.CPU15.SCHED 13987884 ± 3% +17.2% 16397034 ± 2% softirqs.CPU2.NET_RX 13617061 +19.2% 16231527 ± 4% softirqs.CPU3.NET_RX 13602061 +20.3% 16359396 ± 3% softirqs.CPU4.NET_RX 43942 ± 2% +14.3% 50219 ± 5% softirqs.CPU4.SCHED 13990955 +17.4% 16418436 softirqs.CPU5.NET_RX 11002 ± 2% +11.7% 12285 ± 4% softirqs.CPU5.RCU 46890 ± 4% +6.6% 49982 ± 3% softirqs.CPU5.SCHED 13689675 +11.5% 15267728 ± 5% softirqs.CPU6.NET_RX 10593 ± 6% +16.2% 12308 ± 5% softirqs.CPU6.RCU 13804464 ± 3% +12.0% 15461285 ± 2% softirqs.CPU7.NET_RX 11124 ± 3% +12.9% 12563 ± 5% softirqs.CPU7.RCU 13858925 ± 2% +11.5% 15445780 ± 2% softirqs.CPU8.NET_RX 14197655 ± 2% +11.5% 15828958 ± 6% softirqs.CPU9.NET_RX 10786 ± 4% +21.9% 13146 ± 12% softirqs.CPU9.RCU 2.216e+08 +14.1% 2.529e+08 softirqs.NET_RX 182400 ± 2% +12.4% 204933 ± 2% softirqs.RCU 108343 ± 4% -86.8% 14289 ± 21% interrupts.CAL:Function_call_interrupts 5225 ± 11% -89.0% 577.00 ± 18% interrupts.CPU0.CAL:Function_call_interrupts 40158 ± 10% -59.6% 16239 ± 37% interrupts.CPU0.RES:Rescheduling_interrupts 5009 ± 12% -74.1% 1297 ± 64% interrupts.CPU1.CAL:Function_call_interrupts 40241 ± 14% -57.9% 16954 ± 25% interrupts.CPU1.RES:Rescheduling_interrupts 6324 ± 10% -90.4% 610.00 ± 26% interrupts.CPU10.CAL:Function_call_interrupts 44293 ± 8% -65.6% 15225 ± 35% interrupts.CPU10.RES:Rescheduling_interrupts 5854 ± 6% -88.5% 674.25 ± 9% interrupts.CPU11.CAL:Function_call_interrupts 42110 ± 6% -67.8% 13554 ± 29% interrupts.CPU11.RES:Rescheduling_interrupts 6600 ± 12% -90.5% 630.00 ± 19% interrupts.CPU12.CAL:Function_call_interrupts 44102 ± 6% -70.9% 12848 ± 28% interrupts.CPU12.RES:Rescheduling_interrupts 6835 ± 8% -90.6% 643.75 ± 10% interrupts.CPU13.CAL:Function_call_interrupts 40836 ± 2% -66.7% 13594 ± 27% interrupts.CPU13.RES:Rescheduling_interrupts 8009 ± 11% -85.5% 1165 ± 20% interrupts.CPU14.CAL:Function_call_interrupts 40088 ± 6% -57.9% 16894 ± 27% interrupts.CPU14.RES:Rescheduling_interrupts 6977 ± 10% -87.8% 850.75 ± 20% interrupts.CPU15.CAL:Function_call_interrupts 39529 ± 9% -40.9% 23375 ± 13% interrupts.CPU15.RES:Rescheduling_interrupts 6456 ± 19% -85.2% 954.75 ± 36% interrupts.CPU2.CAL:Function_call_interrupts 43543 ± 8% -58.7% 18003 ± 41% interrupts.CPU2.RES:Rescheduling_interrupts 6532 ± 12% -87.6% 811.50 ± 4% interrupts.CPU3.CAL:Function_call_interrupts 40933 ± 3% -60.7% 16085 ± 18% interrupts.CPU3.RES:Rescheduling_interrupts 7241 ± 13% -87.0% 943.25 ± 31% interrupts.CPU4.CAL:Function_call_interrupts 40382 ± 10% -60.7% 15873 ± 16% interrupts.CPU4.RES:Rescheduling_interrupts 7738 ± 8% -83.8% 1250 ± 73% interrupts.CPU5.CAL:Function_call_interrupts 41134 ± 7% -58.4% 17109 ± 27% interrupts.CPU5.RES:Rescheduling_interrupts 8415 ± 7% -89.7% 870.50 ± 27% interrupts.CPU6.CAL:Function_call_interrupts 40327 ± 8% -68.9% 12556 ± 7% interrupts.CPU6.RES:Rescheduling_interrupts 8724 ± 24% -87.0% 1134 ± 25% interrupts.CPU7.CAL:Function_call_interrupts 40562 ± 11% -34.3% 26634 ± 7% interrupts.CPU7.RES:Rescheduling_interrupts 6132 ± 12% -88.6% 698.75 ± 43% interrupts.CPU8.CAL:Function_call_interrupts 40998 ± 6% -58.3% 17089 ± 48% interrupts.CPU8.RES:Rescheduling_interrupts 6266 ± 5% -81.2% 1177 ± 46% interrupts.CPU9.CAL:Function_call_interrupts 46342 ± 6% -61.2% 17976 ± 18% interrupts.CPU9.RES:Rescheduling_interrupts 665585 ± 5% -59.4% 270012 ± 6% interrupts.RES:Rescheduling_interrupts 150.50 ± 10% +420.4% 783.25 ± 4% interrupts.TLB:TLB_shootdowns 58.99 -29.7% 41.46 ± 2% perf-stat.i.MPKI 8.663e+09 +11.6% 9.666e+09 perf-stat.i.branch-instructions 1.75 +0.0 1.79 perf-stat.i.branch-miss-rate% 1.507e+08 +14.1% 1.72e+08 perf-stat.i.branch-misses 479359 ± 2% -13.4% 415362 ± 3% perf-stat.i.cache-misses 2.604e+09 -21.5% 2.045e+09 ± 2% perf-stat.i.cache-references 856628 +4.6% 895798 perf-stat.i.context-switches 1.30 -10.2% 1.17 perf-stat.i.cpi 316.38 -60.8% 124.05 perf-stat.i.cpu-migrations 267648 ± 2% +20.6% 322661 ± 3% perf-stat.i.cycles-between-cache-misses 0.00 ± 12% -0.0 0.00 ± 11% perf-stat.i.dTLB-load-miss-rate% 214865 ± 13% -75.2% 53309 ± 7% perf-stat.i.dTLB-load-misses 1.349e+10 +11.8% 1.509e+10 perf-stat.i.dTLB-loads 0.00 ± 10% -0.0 0.00 ± 9% perf-stat.i.dTLB-store-miss-rate% 32614 ± 10% -70.4% 9653 perf-stat.i.dTLB-store-misses 7.377e+09 +11.9% 8.255e+09 perf-stat.i.dTLB-stores 1.216e+08 +16.8% 1.42e+08 perf-stat.i.iTLB-load-misses 8111370 +9.6% 8892931 perf-stat.i.iTLB-loads 4.401e+10 +11.7% 4.917e+10 perf-stat.i.instructions 0.77 +11.3% 0.85 perf-stat.i.ipc 0.23 ± 19% -66.9% 0.07 ± 12% perf-stat.i.major-faults 0.79 +59.7% 1.26 perf-stat.i.metric.K/sec 2009 +9.1% 2191 perf-stat.i.metric.M/sec 55459 ± 3% -7.9% 51104 ± 2% perf-stat.i.node-stores 59.16 -29.7% 41.59 ± 2% perf-stat.overall.MPKI 1.74 +0.0 1.78 perf-stat.overall.branch-miss-rate% 1.31 -10.2% 1.17 perf-stat.overall.cpi 119750 ± 3% +15.7% 138585 ± 3% perf-stat.overall.cycles-between-cache-misses 0.00 ± 13% -0.0 0.00 ± 7% perf-stat.overall.dTLB-load-miss-rate% 0.00 ± 10% -0.0 0.00 ± 2% perf-stat.overall.dTLB-store-miss-rate% 361.99 -4.3% 346.27 perf-stat.overall.instructions-per-iTLB-miss 0.77 +11.4% 0.85 perf-stat.overall.ipc 9586 +2.1% 9785 perf-stat.overall.path-length 8.634e+09 +11.6% 9.633e+09 perf-stat.ps.branch-instructions 1.502e+08 +14.1% 1.714e+08 perf-stat.ps.branch-misses 478507 ± 2% -13.3% 414853 ± 3% perf-stat.ps.cache-misses 2.595e+09 -21.5% 2.038e+09 ± 2% perf-stat.ps.cache-references 853745 +4.6% 892783 perf-stat.ps.context-switches 315.33 -60.8% 123.65 perf-stat.ps.cpu-migrations 214187 ± 13% -75.2% 53184 ± 7% perf-stat.ps.dTLB-load-misses 1.345e+10 +11.8% 1.504e+10 perf-stat.ps.dTLB-loads 32508 ± 10% -70.4% 9627 perf-stat.ps.dTLB-store-misses 7.352e+09 +11.9% 8.227e+09 perf-stat.ps.dTLB-stores 1.212e+08 +16.8% 1.415e+08 perf-stat.ps.iTLB-load-misses 8084054 +9.6% 8862948 perf-stat.ps.iTLB-loads 4.387e+10 +11.7% 4.9e+10 perf-stat.ps.instructions 0.23 ± 19% -66.9% 0.07 ± 12% perf-stat.ps.major-faults 55284 ± 3% -7.8% 50958 ± 2% perf-stat.ps.node-stores 1.324e+13 +11.6% 1.478e+13 perf-stat.total.instructions 26.06 -2.1 23.93 ± 2% perf-profile.calltrace.cycles-pp.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage 26.78 -2.0 24.80 ± 2% perf-profile.calltrace.cycles-pp.inet_sendpage.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe 27.03 -1.9 25.10 ± 3% perf-profile.calltrace.cycles-pp.kernel_sendpage.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe 27.33 -1.9 25.45 ± 3% perf-profile.calltrace.cycles-pp.sock_sendpage.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe.direct_splice_actor 27.85 -1.8 26.08 ± 3% perf-profile.calltrace.cycles-pp.pipe_to_sendpage.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor 29.98 -1.7 28.30 ± 3% perf-profile.calltrace.cycles-pp.__splice_from_pipe.splice_from_pipe.direct_splice_actor.splice_direct_to_actor.do_splice_direct 30.56 -1.6 28.95 ± 3% perf-profile.calltrace.cycles-pp.splice_from_pipe.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile 30.89 -1.5 29.36 ± 3% perf-profile.calltrace.cycles-pp.direct_splice_actor.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64 13.55 -1.3 12.29 ± 2% perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv 13.78 -1.2 12.59 ± 2% perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core 13.86 -1.2 12.70 ± 2% perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog 13.99 -1.1 12.90 ± 2% perf-profile.calltrace.cycles-pp.ip_local_deliver.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action 10.37 -1.1 9.31 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 6.76 ± 2% -1.0 5.72 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage 7.53 -1.0 6.51 ± 3% perf-profile.calltrace.cycles-pp.lock_sock_nested.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage 14.59 -1.0 13.63 ± 2% perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start 10.46 -0.9 9.51 perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver 6.07 ± 2% -0.9 5.16 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_bh.lock_sock_nested.tcp_sendpage.inet_sendpage 2.39 ± 2% -0.9 1.50 ± 5% perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.generic_file_buffered_read.generic_file_splice_read.splice_direct_to_actor 14.20 -0.9 13.33 ± 2% perf-profile.calltrace.cycles-pp.secondary_startup_64 2.56 ± 2% -0.8 1.74 ± 5% perf-profile.calltrace.cycles-pp.pagecache_get_page.generic_file_buffered_read.generic_file_splice_read.splice_direct_to_actor.do_splice_direct 6.64 -0.8 5.85 perf-profile.calltrace.cycles-pp.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage.sock_sendpage 5.52 -0.8 4.74 perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage 15.19 -0.7 14.45 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.net_rx_action.__softirqentry_text_start.asm_call_on_stack 7.30 ± 3% -0.7 6.59 ± 3% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 15.73 -0.6 15.10 perf-profile.calltrace.cycles-pp.process_backlog.net_rx_action.__softirqentry_text_start.asm_call_on_stack.do_softirq_own_stack 16.20 -0.6 15.64 perf-profile.calltrace.cycles-pp.net_rx_action.__softirqentry_text_start.asm_call_on_stack.do_softirq_own_stack.do_softirq 17.15 -0.5 16.61 perf-profile.calltrace.cycles-pp.__softirqentry_text_start.asm_call_on_stack.do_softirq_own_stack.do_softirq.__local_bh_enable_ip 2.16 -0.5 1.64 ± 3% perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu 17.23 -0.5 16.72 perf-profile.calltrace.cycles-pp.asm_call_on_stack.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2 4.05 -0.5 3.54 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock 1.90 ± 2% -0.5 1.41 ± 3% perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg 4.07 -0.5 3.58 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock 17.31 -0.5 16.84 perf-profile.calltrace.cycles-pp.do_softirq_own_stack.do_softirq.__local_bh_enable_ip.ip_finish_output2.ip_output 2.02 ± 2% -0.5 1.56 ± 3% perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg 2.02 -0.5 1.57 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv 0.74 ± 5% -0.5 0.29 ±100% perf-profile.calltrace.cycles-pp.__inet_lookup_established.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver 2.23 ± 3% -0.4 1.79 ± 2% perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv 3.07 -0.4 2.64 ± 3% perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu 1.19 ± 4% -0.4 0.76 ± 4% perf-profile.calltrace.cycles-pp.__kfree_skb.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv 1.89 -0.4 1.50 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established.tcp_v4_do_rcv 5.98 -0.4 5.59 ± 3% perf-profile.calltrace.cycles-pp.generic_file_buffered_read.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile 6.38 -0.4 5.99 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established.tcp_v4_do_rcv 1.81 -0.4 1.43 ± 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_rcv_established 3.84 -0.4 3.47 perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendpage.inet_sendpage 3.80 -0.3 3.46 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendpage 4.96 -0.3 4.66 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_established 2.83 ± 2% -0.3 2.57 ± 2% perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 1.09 ± 2% -0.3 0.83 ± 6% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 1.06 ± 2% -0.3 0.80 ± 6% perf-profile.calltrace.cycles-pp.__sched_text_start.schedule_idle.do_idle.cpu_startup_entry.start_secondary 6.40 -0.2 6.17 ± 2% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit 6.96 -0.2 6.73 perf-profile.calltrace.cycles-pp.ip_output.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames 1.36 ± 3% -0.2 1.16 perf-profile.calltrace.cycles-pp.tcp_current_mss.tcp_send_mss.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage 1.68 ± 2% -0.2 1.51 perf-profile.calltrace.cycles-pp.tcp_send_mss.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage 2.30 -0.2 2.15 ± 2% perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 0.86 ± 2% -0.1 0.75 perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock 2.12 ± 2% -0.1 2.03 perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg.inet_recvmsg 0.72 ± 3% -0.1 0.63 ± 4% perf-profile.calltrace.cycles-pp._raw_spin_lock_bh.release_sock.tcp_sendpage.inet_sendpage.kernel_sendpage 0.85 ± 5% -0.1 0.77 ± 2% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout 1.00 ± 4% -0.1 0.92 ± 3% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_timeout.wait_woken 0.57 ± 4% +0.0 0.61 ± 3% perf-profile.calltrace.cycles-pp.security_file_permission.do_splice_direct.do_sendfile.__x64_sys_sendfile64.do_syscall_64 0.58 ± 2% +0.1 0.64 ± 3% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe 0.64 +0.1 0.71 ± 3% perf-profile.calltrace.cycles-pp.security_file_permission.do_splice_to.splice_direct_to_actor.do_splice_direct.do_sendfile 0.55 +0.1 0.63 ± 6% perf-profile.calltrace.cycles-pp.touch_atime.generic_file_splice_read.splice_direct_to_actor.do_splice_direct.do_sendfile 0.81 ± 2% +0.1 0.89 ± 3% perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg 0.87 +0.1 0.96 ± 2% perf-profile.calltrace.cycles-pp.common_file_perm.security_file_permission.do_sendfile.__x64_sys_sendfile64.do_syscall_64 0.93 ± 2% +0.1 1.02 ± 2% perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg 0.63 +0.1 0.73 ± 7% perf-profile.calltrace.cycles-pp.touch_atime.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64 1.25 ± 2% +0.1 1.35 ± 3% perf-profile.calltrace.cycles-pp.sk_stream_alloc_skb.do_tcp_sendpages.tcp_sendpage_locked.tcp_sendpage.inet_sendpage 1.10 +0.1 1.20 ± 3% perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom 0.55 +0.1 0.65 ± 3% perf-profile.calltrace.cycles-pp.__fget_light.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.08 +0.1 1.19 ± 2% perf-profile.calltrace.cycles-pp.do_splice_to.splice_direct_to_actor.do_splice_direct.do_sendfile.__x64_sys_sendfile64 1.26 +0.1 1.37 ± 2% perf-profile.calltrace.cycles-pp.release_sock.tcp_recvmsg.inet_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 1.16 +0.1 1.28 perf-profile.calltrace.cycles-pp.security_file_permission.do_sendfile.__x64_sys_sendfile64.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.17 +0.2 1.34 ± 4% perf-profile.calltrace.cycles-pp.copy_page_to_iter.generic_file_buffered_read.generic_file_splice_read.splice_direct_to_actor.do_splice_direct 0.38 ± 57% +0.2 0.56 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64 1.46 +0.3 1.73 perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit 2.31 +0.4 2.67 ± 2% perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.__ip_queue_xmit.__tcp_transmit_skb 1.20 ± 18% +0.4 1.58 perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output 0.00 +0.5 0.52 ± 4% perf-profile.calltrace.cycles-pp.netif_rx_internal.netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit 0.00 +0.5 0.55 ± 8% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 0.00 +0.6 0.55 ± 5% perf-profile.calltrace.cycles-pp.netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2 0.00 +0.6 0.56 perf-profile.calltrace.cycles-pp.__might_fault._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg 0.00 +0.6 0.57 ± 5% perf-profile.calltrace.cycles-pp.__virt_addr_valid.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter 6.74 +0.7 7.45 ± 3% perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg.inet_recvmsg 26.12 -2.1 24.04 ± 2% perf-profile.children.cycles-pp.tcp_sendpage 26.83 -2.0 24.86 ± 2% perf-profile.children.cycles-pp.inet_sendpage 27.08 -1.9 25.15 ± 3% perf-profile.children.cycles-pp.kernel_sendpage 27.39 -1.9 25.50 ± 3% perf-profile.children.cycles-pp.sock_sendpage 27.87 -1.8 26.10 ± 3% perf-profile.children.cycles-pp.pipe_to_sendpage 30.03 -1.7 28.36 ± 3% perf-profile.children.cycles-pp.__splice_from_pipe 30.60 -1.6 28.99 ± 3% perf-profile.children.cycles-pp.splice_from_pipe 30.95 -1.5 29.42 ± 3% perf-profile.children.cycles-pp.direct_splice_actor 15.00 -1.3 13.71 perf-profile.children.cycles-pp.tcp_rcv_established 13.59 -1.3 12.33 ± 2% perf-profile.children.cycles-pp.tcp_v4_rcv 15.24 -1.2 14.03 perf-profile.children.cycles-pp.tcp_v4_do_rcv 13.79 -1.2 12.61 ± 2% perf-profile.children.cycles-pp.ip_protocol_deliver_rcu 13.87 -1.1 12.72 ± 2% perf-profile.children.cycles-pp.ip_local_deliver_finish 7.84 -1.1 6.74 ± 2% perf-profile.children.cycles-pp._raw_spin_lock_bh 14.00 -1.1 12.92 ± 2% perf-profile.children.cycles-pp.ip_local_deliver 7.95 -1.0 6.91 ± 3% perf-profile.children.cycles-pp.lock_sock_nested 6.17 ± 2% -1.0 5.21 ± 2% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 14.61 -0.9 13.66 ± 2% perf-profile.children.cycles-pp.ip_rcv 14.19 -0.9 13.30 ± 2% perf-profile.children.cycles-pp.do_idle 2.42 ± 2% -0.9 1.55 ± 5% perf-profile.children.cycles-pp.find_get_entry 14.20 -0.9 13.33 ± 2% perf-profile.children.cycles-pp.secondary_startup_64 14.20 -0.9 13.33 ± 2% perf-profile.children.cycles-pp.cpu_startup_entry 2.58 ± 2% -0.8 1.76 ± 5% perf-profile.children.cycles-pp.pagecache_get_page 15.21 -0.7 14.47 ± 2% perf-profile.children.cycles-pp.__netif_receive_skb_one_core 7.46 -0.7 6.76 perf-profile.children.cycles-pp.intel_idle 6.63 -0.7 5.95 perf-profile.children.cycles-pp.__release_sock 2.24 ± 3% -0.7 1.57 ± 3% perf-profile.children.cycles-pp.__kfree_skb 7.97 -0.7 7.31 perf-profile.children.cycles-pp.release_sock 11.05 -0.7 10.39 ± 2% perf-profile.children.cycles-pp.cpuidle_enter_state 11.05 -0.6 10.40 ± 2% perf-profile.children.cycles-pp.cpuidle_enter 15.75 -0.6 15.12 perf-profile.children.cycles-pp.process_backlog 2.85 ± 2% -0.6 2.29 ± 2% perf-profile.children.cycles-pp.tcp_clean_rtx_queue 16.25 -0.6 15.70 perf-profile.children.cycles-pp.net_rx_action 17.53 -0.5 16.99 perf-profile.children.cycles-pp.asm_call_on_stack 1.18 ± 5% -0.5 0.64 ± 4% perf-profile.children.cycles-pp.skb_release_data 17.19 -0.5 16.65 perf-profile.children.cycles-pp.__softirqentry_text_start 2.27 -0.5 1.74 ± 2% perf-profile.children.cycles-pp.sock_def_readable 3.96 -0.5 3.43 ± 2% perf-profile.children.cycles-pp.tcp_ack 17.35 -0.5 16.88 perf-profile.children.cycles-pp.do_softirq_own_stack 1.96 ± 2% -0.5 1.49 ± 3% perf-profile.children.cycles-pp.__check_object_size 2.03 -0.5 1.57 ± 3% perf-profile.children.cycles-pp.__wake_up_common_lock 2.04 ± 2% -0.5 1.59 ± 3% perf-profile.children.cycles-pp.simple_copy_to_iter 12.41 -0.4 11.96 perf-profile.children.cycles-pp.tcp_write_xmit 12.49 -0.4 12.08 perf-profile.children.cycles-pp.__tcp_push_pending_frames 1.90 -0.4 1.50 ± 3% perf-profile.children.cycles-pp.__wake_up_common 1.82 -0.4 1.44 ± 3% perf-profile.children.cycles-pp.try_to_wake_up 6.04 -0.4 5.67 ± 3% perf-profile.children.cycles-pp.generic_file_buffered_read 3.09 -0.3 2.76 perf-profile.children.cycles-pp.__sched_text_start 1.17 ± 2% -0.3 0.88 ± 2% perf-profile.children.cycles-pp.schedule_idle 0.96 ± 3% -0.3 0.67 ± 7% perf-profile.children.cycles-pp.__inet_lookup_established 0.50 ± 2% -0.3 0.22 ± 24% perf-profile.children.cycles-pp.dst_release 2.83 ± 2% -0.3 2.58 ± 2% perf-profile.children.cycles-pp.sk_wait_data 0.47 ± 5% -0.2 0.23 ± 3% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.91 ± 3% -0.2 0.70 ± 2% perf-profile.children.cycles-pp._raw_spin_lock 1.49 ± 2% -0.2 1.30 ± 2% perf-profile.children.cycles-pp.tcp_current_mss 0.39 ± 9% -0.2 0.21 ± 12% perf-profile.children.cycles-pp.tcp_rate_check_app_limited 1.71 ± 2% -0.2 1.53 perf-profile.children.cycles-pp.tcp_send_mss 0.74 ± 2% -0.2 0.57 ± 2% perf-profile.children.cycles-pp.pick_next_task_fair 0.72 ± 3% -0.2 0.55 perf-profile.children.cycles-pp.update_load_avg 0.48 ± 2% -0.2 0.33 ± 2% perf-profile.children.cycles-pp.__slab_free 2.30 -0.2 2.15 ± 2% perf-profile.children.cycles-pp.wait_woken 0.45 ± 3% -0.2 0.29 ± 3% perf-profile.children.cycles-pp.set_next_entity 0.33 ± 5% -0.1 0.19 ± 8% perf-profile.children.cycles-pp.__ksize 0.43 -0.1 0.30 ± 4% perf-profile.children.cycles-pp.__skb_clone 0.87 ± 2% -0.1 0.76 ± 3% perf-profile.children.cycles-pp.enqueue_task_fair 0.50 ± 4% -0.1 0.40 ± 10% perf-profile.children.cycles-pp.skb_release_all 0.47 ± 4% -0.1 0.37 ± 10% perf-profile.children.cycles-pp.skb_release_head_state 0.90 ± 2% -0.1 0.80 ± 4% perf-profile.children.cycles-pp.ttwu_do_activate 0.33 ± 3% -0.1 0.23 ± 6% perf-profile.children.cycles-pp.update_rq_clock 1.00 ± 4% -0.1 0.92 ± 3% perf-profile.children.cycles-pp.dequeue_task_fair 2.12 ± 2% -0.1 2.04 perf-profile.children.cycles-pp.schedule_timeout 0.86 ± 5% -0.1 0.78 ± 2% perf-profile.children.cycles-pp.dequeue_entity 0.34 ± 2% -0.1 0.26 ± 9% perf-profile.children.cycles-pp.tcp_add_backlog 0.30 ± 6% -0.1 0.22 ± 4% perf-profile.children.cycles-pp.select_task_rq_fair 0.16 ± 7% -0.1 0.08 ± 8% perf-profile.children.cycles-pp.select_idle_sibling 0.11 ± 3% -0.1 0.04 ± 57% perf-profile.children.cycles-pp.remove_wait_queue 0.23 ± 3% -0.1 0.17 ± 9% perf-profile.children.cycles-pp.__update_load_avg_se 0.17 ± 7% -0.1 0.10 ± 8% perf-profile.children.cycles-pp.tcp_mtu_probe 0.23 ± 3% -0.1 0.17 ± 11% perf-profile.children.cycles-pp.sock_put 0.16 ± 10% -0.1 0.10 ± 14% perf-profile.children.cycles-pp.tcp_check_space 0.18 ± 6% -0.1 0.12 ± 6% perf-profile.children.cycles-pp.skb_clone 0.24 ± 9% -0.1 0.18 ± 7% perf-profile.children.cycles-pp.ip_send_check 0.42 ± 5% -0.1 0.37 ± 2% perf-profile.children.cycles-pp.update_curr 0.45 -0.0 0.40 perf-profile.children.cycles-pp.kmem_cache_free 0.30 ± 6% -0.0 0.25 ± 7% perf-profile.children.cycles-pp.__ip_local_out 0.32 ± 4% -0.0 0.28 ± 6% perf-profile.children.cycles-pp.ip_local_out 0.16 ± 7% -0.0 0.11 ± 7% perf-profile.children.cycles-pp.tick_nohz_idle_exit 0.10 ± 8% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.pick_next_entity 0.13 ± 5% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.__list_del_entry_valid 0.17 ± 4% -0.0 0.14 ± 8% perf-profile.children.cycles-pp.skb_entail 0.10 ± 4% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.resched_curr 0.10 ± 4% -0.0 0.08 ± 10% perf-profile.children.cycles-pp.lock_timer_base 0.17 ± 3% -0.0 0.14 ± 3% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.15 ± 3% -0.0 0.12 ± 8% perf-profile.children.cycles-pp.tcp_update_skb_after_send 0.09 ± 7% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.reweight_entity 0.09 ± 5% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.newidle_balance 0.07 -0.0 0.06 perf-profile.children.cycles-pp.rcu_idle_exit 0.09 +0.0 0.11 ± 4% perf-profile.children.cycles-pp.switch_fpu_return 0.14 +0.0 0.16 ± 7% perf-profile.children.cycles-pp.ip_rcv_finish_core 0.07 ± 12% +0.0 0.09 ± 7% perf-profile.children.cycles-pp.tcp_v4_inbound_md5_hash 0.11 ± 7% +0.0 0.13 ± 8% perf-profile.children.cycles-pp.tcp_tx_timestamp 0.10 ± 5% +0.0 0.12 perf-profile.children.cycles-pp.validate_xmit_xfrm 0.12 ± 5% +0.0 0.15 ± 7% perf-profile.children.cycles-pp.check_stack_object 0.12 +0.0 0.15 ± 7% perf-profile.children.cycles-pp.sock_rfree 0.08 ± 6% +0.0 0.10 ± 8% perf-profile.children.cycles-pp.__tcp_v4_send_check 0.24 ± 3% +0.0 0.27 ± 4% perf-profile.children.cycles-pp.apparmor_file_permission 0.07 ± 10% +0.0 0.10 ± 12% perf-profile.children.cycles-pp.menu_reflect 0.21 ± 6% +0.0 0.24 ± 4% perf-profile.children.cycles-pp.__switch_to_asm 0.27 ± 5% +0.0 0.30 ± 4% perf-profile.children.cycles-pp.ip_rcv_core 0.12 ± 8% +0.0 0.15 ± 5% perf-profile.children.cycles-pp.__kfree_skb_flush 0.06 ± 7% +0.0 0.09 ± 13% perf-profile.children.cycles-pp.skb_network_protocol 0.08 ± 8% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.tcp_stream_memory_free 0.07 ± 6% +0.0 0.10 ± 4% perf-profile.children.cycles-pp.skb_clone_tx_timestamp 0.20 ± 7% +0.0 0.24 ± 5% perf-profile.children.cycles-pp.syscall_enter_from_user_mode 0.16 ± 2% +0.0 0.20 ± 8% perf-profile.children.cycles-pp.tcp_push 0.10 ± 8% +0.0 0.14 ± 8% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.45 ± 3% +0.0 0.49 ± 2% perf-profile.children.cycles-pp.enqueue_to_backlog 0.36 ± 4% +0.0 0.40 ± 2% perf-profile.children.cycles-pp.__put_user_8 0.10 ± 7% +0.0 0.14 ± 5% perf-profile.children.cycles-pp.bictcp_cwnd_event 0.11 ± 13% +0.0 0.15 ± 7% perf-profile.children.cycles-pp.do_splice_from 0.08 ± 13% +0.0 0.12 ± 8% perf-profile.children.cycles-pp.__next_timer_interrupt 0.33 ± 3% +0.0 0.38 ± 5% perf-profile.children.cycles-pp.tcp_established_options 0.30 ± 6% +0.0 0.34 ± 4% perf-profile.children.cycles-pp.tick_nohz_next_event 0.15 ± 3% +0.0 0.20 ± 9% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64 0.11 ± 9% +0.0 0.16 ± 5% perf-profile.children.cycles-pp.bictcp_acked 0.22 +0.0 0.27 ± 6% perf-profile.children.cycles-pp.__switch_to 0.20 ± 2% +0.0 0.25 ± 3% perf-profile.children.cycles-pp.tcp_release_cb 0.03 ±100% +0.0 0.07 ± 14% perf-profile.children.cycles-pp.__calc_delta 0.22 ± 5% +0.1 0.27 ± 6% perf-profile.children.cycles-pp.sanity 0.41 ± 2% +0.1 0.46 ± 4% perf-profile.children.cycles-pp.aa_file_perm 0.20 ± 2% +0.1 0.25 ± 10% perf-profile.children.cycles-pp.inet_ehashfn 0.01 ±173% +0.1 0.07 ± 13% perf-profile.children.cycles-pp.tcp_rate_skb_delivered 0.09 ± 4% +0.1 0.15 ± 7% perf-profile.children.cycles-pp.tcp_v4_fill_cb 0.20 ± 10% +0.1 0.25 ± 3% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.detach_if_pending 0.21 ± 2% +0.1 0.26 ± 6% perf-profile.children.cycles-pp.ip_rcv_finish 0.02 ±173% +0.1 0.08 ± 5% perf-profile.children.cycles-pp.apparmor_socket_sock_rcv_skb 0.51 ± 4% +0.1 0.57 perf-profile.children.cycles-pp.entry_SYSCALL_64 0.17 ± 6% +0.1 0.23 ± 13% perf-profile.children.cycles-pp.netif_skb_features 0.00 +0.1 0.06 ± 6% perf-profile.children.cycles-pp.update_irq_load_avg 0.35 ± 3% +0.1 0.41 ± 5% perf-profile.children.cycles-pp.fsnotify 0.23 ± 4% +0.1 0.30 perf-profile.children.cycles-pp.mark_page_accessed 0.18 ± 4% +0.1 0.25 ± 7% perf-profile.children.cycles-pp.sock_recvmsg 0.59 +0.1 0.66 ± 2% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 0.27 ± 10% +0.1 0.33 ± 5% perf-profile.children.cycles-pp.kfree 0.12 ± 10% +0.1 0.19 ± 6% perf-profile.children.cycles-pp.aa_sk_perm 0.45 ± 2% +0.1 0.52 ± 4% perf-profile.children.cycles-pp.__x86_indirect_thunk_rax 0.48 ± 6% +0.1 0.56 ± 3% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 0.13 ± 11% +0.1 0.20 ± 3% perf-profile.children.cycles-pp.security_sock_rcv_skb 0.15 ± 6% +0.1 0.22 ± 9% perf-profile.children.cycles-pp.nf_hook_slow 0.15 ± 7% +0.1 0.22 ± 7% perf-profile.children.cycles-pp.security_socket_recvmsg 0.01 ±173% +0.1 0.09 ± 12% perf-profile.children.cycles-pp._find_next_bit 0.47 ± 5% +0.1 0.55 ± 7% perf-profile.children.cycles-pp.xas_load 0.07 ± 6% +0.1 0.15 ± 9% perf-profile.children.cycles-pp.set_next_task_idle 0.22 ± 5% +0.1 0.30 ± 5% perf-profile.children.cycles-pp.__sk_dst_check 0.49 ± 6% +0.1 0.58 ± 4% perf-profile.children.cycles-pp.tcp_mstamp_refresh 0.93 ± 3% +0.1 1.02 ± 2% perf-profile.children.cycles-pp.netif_rx_internal 0.06 ± 7% +0.1 0.15 ± 7% perf-profile.children.cycles-pp.__update_idle_core 0.08 ± 8% +0.1 0.17 ± 8% perf-profile.children.cycles-pp.pick_next_task_idle 0.96 ± 3% +0.1 1.06 ± 3% perf-profile.children.cycles-pp.netif_rx 0.73 +0.1 0.83 perf-profile.children.cycles-pp.__kmalloc_reserve 0.75 ± 3% +0.1 0.85 ± 5% perf-profile.children.cycles-pp.__tcp_send_ack 1.10 +0.1 1.20 ± 2% perf-profile.children.cycles-pp.do_splice_to 1.25 ± 2% +0.1 1.36 ± 4% perf-profile.children.cycles-pp.sk_stream_alloc_skb 0.76 ± 2% +0.1 0.87 ± 5% perf-profile.children.cycles-pp.current_time 0.66 +0.1 0.77 ± 2% perf-profile.children.cycles-pp.__kmalloc_node_track_caller 0.48 ± 5% +0.1 0.59 ± 4% perf-profile.children.cycles-pp.kmem_cache_alloc_node 1.89 ± 2% +0.1 2.01 ± 2% perf-profile.children.cycles-pp.__alloc_skb 0.77 ± 2% +0.1 0.90 ± 2% perf-profile.children.cycles-pp.__fget_light 0.45 ± 5% +0.1 0.57 ± 5% perf-profile.children.cycles-pp.__virt_addr_valid 0.74 +0.1 0.87 ± 2% perf-profile.children.cycles-pp.__might_sleep 0.48 ± 2% +0.1 0.61 ± 7% perf-profile.children.cycles-pp.__netif_receive_skb_core 0.51 ± 3% +0.1 0.64 ± 3% perf-profile.children.cycles-pp.sk_filter_trim_cap 0.28 ± 2% +0.1 0.42 ± 5% perf-profile.children.cycles-pp.ipv4_dst_check 0.38 ± 6% +0.1 0.52 ± 8% perf-profile.children.cycles-pp.validate_xmit_skb 1.24 ± 2% +0.2 1.39 ± 2% perf-profile.children.cycles-pp.ktime_get 1.19 +0.2 1.35 ± 4% perf-profile.children.cycles-pp.copy_page_to_iter 1.56 ± 2% +0.2 1.73 perf-profile.children.cycles-pp.___might_sleep 0.99 ± 2% +0.2 1.16 ± 2% perf-profile.children.cycles-pp.read_tsc 1.79 ± 2% +0.2 1.97 perf-profile.children.cycles-pp.common_file_perm 0.89 +0.2 1.08 ± 4% perf-profile.children.cycles-pp.syscall_return_via_sysret 1.27 +0.2 1.47 ± 3% perf-profile.children.cycles-pp.atime_needs_update 2.39 +0.2 2.62 perf-profile.children.cycles-pp.security_file_permission 1.79 +0.3 2.04 perf-profile.children.cycles-pp.loopback_xmit 1.62 +0.3 1.88 ± 5% perf-profile.children.cycles-pp.touch_atime 1.08 ± 2% +0.3 1.35 perf-profile.children.cycles-pp.__might_fault 1.95 +0.3 2.24 perf-profile.children.cycles-pp.dev_hard_start_xmit 3.09 +0.4 3.45 perf-profile.children.cycles-pp.__dev_queue_xmit 83.83 +0.6 84.40 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 6.79 +0.7 7.50 ± 3% perf-profile.children.cycles-pp._copy_to_iter 6.12 ± 2% -1.0 5.14 ± 2% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 1.91 ± 3% -1.0 0.96 ± 4% perf-profile.self.cycles-pp.find_get_entry 7.46 -0.7 6.76 perf-profile.self.cycles-pp.intel_idle 1.40 ± 3% -0.6 0.80 ± 5% perf-profile.self.cycles-pp.__check_object_size 1.18 ± 5% -0.5 0.64 ± 4% perf-profile.self.cycles-pp.skb_release_data 0.76 ± 4% -0.3 0.43 ± 7% perf-profile.self.cycles-pp.__inet_lookup_established 0.49 ± 3% -0.3 0.21 ± 27% perf-profile.self.cycles-pp.dst_release 0.46 ± 5% -0.2 0.22 ± 4% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 2.27 ± 2% -0.2 2.07 ± 3% perf-profile.self.cycles-pp.do_tcp_sendpages 0.86 ± 3% -0.2 0.67 ± 2% perf-profile.self.cycles-pp._raw_spin_lock 0.39 ± 8% -0.2 0.20 ± 12% perf-profile.self.cycles-pp.tcp_rate_check_app_limited 1.69 -0.2 1.51 ± 2% perf-profile.self.cycles-pp._raw_spin_lock_bh 0.47 ± 2% -0.1 0.32 ± 2% perf-profile.self.cycles-pp.__slab_free 0.33 ± 5% -0.1 0.19 ± 8% perf-profile.self.cycles-pp.__ksize 0.34 ± 2% -0.1 0.21 ± 8% perf-profile.self.cycles-pp.__skb_clone 0.32 ± 2% -0.1 0.21 ± 8% perf-profile.self.cycles-pp.tcp_sendpage_locked 0.21 ± 6% -0.1 0.10 ± 8% perf-profile.self.cycles-pp.update_rq_clock 0.19 ± 5% -0.1 0.09 ± 5% perf-profile.self.cycles-pp.set_next_entity 0.60 ± 3% -0.1 0.50 ± 8% perf-profile.self.cycles-pp.__dev_queue_xmit 0.55 ± 2% -0.1 0.46 ± 4% perf-profile.self.cycles-pp.tcp_clean_rtx_queue 0.25 ± 7% -0.1 0.17 ± 12% perf-profile.self.cycles-pp.update_curr 0.33 ± 2% -0.1 0.24 ± 5% perf-profile.self.cycles-pp.update_load_avg 1.31 ± 3% -0.1 1.24 ± 3% perf-profile.self.cycles-pp.tcp_write_xmit 0.23 ± 5% -0.1 0.16 ± 9% perf-profile.self.cycles-pp.__update_load_avg_se 0.16 ± 7% -0.1 0.09 ± 15% perf-profile.self.cycles-pp.tcp_check_space 0.15 ± 7% -0.1 0.09 ± 4% perf-profile.self.cycles-pp.tcp_mtu_probe 0.18 ± 6% -0.1 0.11 ± 11% perf-profile.self.cycles-pp.wait_woken 0.11 ± 7% -0.1 0.04 ± 59% perf-profile.self.cycles-pp.__release_sock 0.24 ± 6% -0.1 0.17 ± 9% perf-profile.self.cycles-pp.sock_def_readable 0.75 -0.1 0.68 ± 4% perf-profile.self.cycles-pp.__skb_datagram_iter 0.18 ± 6% -0.1 0.12 ± 10% perf-profile.self.cycles-pp.skb_clone 0.22 ± 3% -0.1 0.16 ± 14% perf-profile.self.cycles-pp.sock_put 0.45 ± 2% -0.1 0.39 ± 3% perf-profile.self.cycles-pp.__sched_text_start 0.16 ± 9% -0.1 0.10 ± 14% perf-profile.self.cycles-pp.enqueue_task_fair 0.23 ± 7% -0.1 0.18 ± 6% perf-profile.self.cycles-pp.ip_send_check 0.12 ± 6% -0.0 0.08 ± 8% perf-profile.self.cycles-pp.ip_finish_output 0.47 ± 4% -0.0 0.42 ± 5% perf-profile.self.cycles-pp.tcp_rcv_established 0.12 ± 8% -0.0 0.08 ± 5% perf-profile.self.cycles-pp.schedule_timeout 0.11 ± 7% -0.0 0.08 ± 5% perf-profile.self.cycles-pp.__list_del_entry_valid 0.16 ± 7% -0.0 0.13 ± 6% perf-profile.self.cycles-pp.irqtime_account_irq 0.08 ± 14% -0.0 0.04 ± 58% perf-profile.self.cycles-pp.tcp_rcv_space_adjust 0.08 ± 5% -0.0 0.05 perf-profile.self.cycles-pp.pick_next_entity 0.10 ± 4% -0.0 0.07 ± 12% perf-profile.self.cycles-pp.resched_curr 0.16 ± 5% -0.0 0.14 ± 3% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.09 ± 7% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.reweight_entity 0.15 ± 3% -0.0 0.12 ± 8% perf-profile.self.cycles-pp.skb_entail 0.08 ± 5% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.newidle_balance 0.07 ± 6% +0.0 0.08 ± 5% perf-profile.self.cycles-pp.skb_release_head_state 0.05 ± 8% +0.0 0.07 ± 6% perf-profile.self.cycles-pp.tick_nohz_next_event 0.11 ± 6% +0.0 0.12 ± 4% perf-profile.self.cycles-pp.__sock_wfree 0.06 ± 11% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.tcp_mstamp_refresh 0.10 ± 4% +0.0 0.12 ± 6% perf-profile.self.cycles-pp.generic_splice_sendpage 0.21 ± 2% +0.0 0.23 ± 4% perf-profile.self.cycles-pp.apparmor_file_permission 0.15 ± 5% +0.0 0.17 ± 6% perf-profile.self.cycles-pp.direct_splice_actor 0.08 ± 6% +0.0 0.10 ± 4% perf-profile.self.cycles-pp.ip_rcv_finish 0.13 +0.0 0.15 ± 7% perf-profile.self.cycles-pp.ip_rcv_finish_core 0.06 ± 9% +0.0 0.08 ± 8% perf-profile.self.cycles-pp.__sk_dst_check 0.09 ± 7% +0.0 0.12 ± 9% perf-profile.self.cycles-pp.asm_call_on_stack 0.09 ± 8% +0.0 0.12 perf-profile.self.cycles-pp.validate_xmit_xfrm 0.14 ± 3% +0.0 0.17 ± 4% perf-profile.self.cycles-pp.do_splice_to 0.10 ± 4% +0.0 0.12 ± 4% perf-profile.self.cycles-pp.__x64_sys_recvfrom 0.07 ± 10% +0.0 0.10 ± 4% perf-profile.self.cycles-pp.__tcp_v4_send_check 0.05 ± 9% +0.0 0.08 ± 10% perf-profile.self.cycles-pp.skb_network_protocol 0.04 ± 58% +0.0 0.07 ± 6% perf-profile.self.cycles-pp.tcp_v4_inbound_md5_hash 0.21 ± 6% +0.0 0.24 ± 3% perf-profile.self.cycles-pp.__switch_to_asm 0.35 ± 3% +0.0 0.38 ± 6% perf-profile.self.cycles-pp.security_file_permission 0.08 ± 14% +0.0 0.11 ± 7% perf-profile.self.cycles-pp.do_softirq_own_stack 0.09 ± 11% +0.0 0.12 perf-profile.self.cycles-pp.__tcp_push_pending_frames 0.20 ± 7% +0.0 0.23 ± 4% perf-profile.self.cycles-pp.syscall_enter_from_user_mode 0.12 ± 5% +0.0 0.15 ± 5% perf-profile.self.cycles-pp.tcp_add_backlog 0.11 ± 4% +0.0 0.15 ± 7% perf-profile.self.cycles-pp.sock_rfree 0.10 ± 11% +0.0 0.13 ± 5% perf-profile.self.cycles-pp.bictcp_cwnd_event 0.08 ± 5% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.__kfree_skb_flush 0.23 ± 3% +0.0 0.27 ± 5% perf-profile.self.cycles-pp._cond_resched 0.08 ± 8% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.tcp_stream_memory_free 0.13 ± 3% +0.0 0.16 ± 9% perf-profile.self.cycles-pp.copyout 0.06 +0.0 0.10 ± 5% perf-profile.self.cycles-pp.skb_clone_tx_timestamp 0.26 ± 4% +0.0 0.30 ± 5% perf-profile.self.cycles-pp.ip_rcv_core 0.26 ± 3% +0.0 0.30 ± 5% perf-profile.self.cycles-pp.enqueue_to_backlog 0.24 ± 3% +0.0 0.28 ± 7% perf-profile.self.cycles-pp.do_splice_direct 0.21 ± 4% +0.0 0.25 ± 9% perf-profile.self.cycles-pp.syscall_exit_to_user_mode 0.16 ± 10% +0.0 0.20 ± 3% perf-profile.self.cycles-pp.dev_hard_start_xmit 0.15 ± 3% +0.0 0.19 ± 8% perf-profile.self.cycles-pp.tcp_push 0.13 ± 6% +0.0 0.17 ± 3% perf-profile.self.cycles-pp.tcp_v4_do_rcv 0.26 ± 5% +0.0 0.30 ± 5% perf-profile.self.cycles-pp.xas_load 0.36 ± 3% +0.0 0.40 ± 3% perf-profile.self.cycles-pp.__put_user_8 0.32 ± 5% +0.0 0.36 ± 4% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax 0.28 ± 3% +0.0 0.32 ± 6% perf-profile.self.cycles-pp.timestamp_truncate 0.22 +0.0 0.26 ± 7% perf-profile.self.cycles-pp.__switch_to 0.31 ± 6% +0.0 0.35 ± 3% perf-profile.self.cycles-pp.sock_sendpage 0.30 ± 4% +0.0 0.34 ± 4% perf-profile.self.cycles-pp.tcp_established_options 0.19 ± 3% +0.0 0.23 ± 8% perf-profile.self.cycles-pp.inet_ehashfn 0.01 ±173% +0.0 0.05 ± 9% perf-profile.self.cycles-pp.__hrtimer_next_event_base 0.10 ± 15% +0.0 0.14 ± 5% perf-profile.self.cycles-pp.do_splice_from 0.38 +0.0 0.42 perf-profile.self.cycles-pp.__ip_queue_xmit 0.08 ± 14% +0.0 0.12 ± 19% perf-profile.self.cycles-pp.nf_hook_slow 0.14 ± 3% +0.0 0.18 ± 11% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64 0.08 ± 5% +0.0 0.13 ± 8% perf-profile.self.cycles-pp.sk_wait_data 0.07 ± 7% +0.0 0.11 perf-profile.self.cycles-pp.security_sock_rcv_skb 0.07 ± 7% +0.0 0.11 ± 11% perf-profile.self.cycles-pp.ip_local_deliver_finish 0.21 ± 7% +0.0 0.26 ± 6% perf-profile.self.cycles-pp.sanity 0.11 ± 7% +0.0 0.15 ± 3% perf-profile.self.cycles-pp.bictcp_acked 0.03 ±100% +0.0 0.07 ± 11% perf-profile.self.cycles-pp.__tcp_send_ack 0.22 ± 8% +0.0 0.27 ± 11% perf-profile.self.cycles-pp.kernel_sendpage 0.38 ± 6% +0.0 0.43 ± 3% perf-profile.self.cycles-pp.do_syscall_64 0.07 ± 7% +0.0 0.11 ± 9% perf-profile.self.cycles-pp.pick_next_task_fair 0.01 ±173% +0.1 0.06 ± 13% perf-profile.self.cycles-pp.__update_idle_core 0.01 ±173% +0.1 0.06 ± 6% perf-profile.self.cycles-pp.tcp_rate_skb_delivered 0.38 ± 2% +0.1 0.43 ± 4% perf-profile.self.cycles-pp.aa_file_perm 0.14 ± 6% +0.1 0.19 ± 4% perf-profile.self.cycles-pp.ip_rcv 0.09 ± 7% +0.1 0.14 ± 9% perf-profile.self.cycles-pp.tcp_v4_fill_cb 0.01 ±173% +0.1 0.07 ± 13% perf-profile.self.cycles-pp.__next_timer_interrupt 0.18 +0.1 0.23 ± 4% perf-profile.self.cycles-pp.tcp_release_cb 0.00 +0.1 0.05 ± 9% perf-profile.self.cycles-pp.detach_if_pending 0.51 ± 4% +0.1 0.56 perf-profile.self.cycles-pp.entry_SYSCALL_64 0.17 ± 3% +0.1 0.22 ± 3% perf-profile.self.cycles-pp.pagecache_get_page 0.12 ± 8% +0.1 0.18 ± 6% perf-profile.self.cycles-pp.sk_filter_trim_cap 0.14 ± 5% +0.1 0.20 ± 4% perf-profile.self.cycles-pp.ip_local_deliver 0.24 ± 6% +0.1 0.30 ± 2% perf-profile.self.cycles-pp.ip_output 0.12 ± 6% +0.1 0.18 ± 8% perf-profile.self.cycles-pp.ip_protocol_deliver_rcu 0.50 ± 4% +0.1 0.56 ± 2% perf-profile.self.cycles-pp.splice_from_pipe 0.26 ± 9% +0.1 0.33 ± 4% perf-profile.self.cycles-pp.kfree 0.00 +0.1 0.06 ± 6% perf-profile.self.cycles-pp.update_irq_load_avg 0.08 ± 15% +0.1 0.15 ± 10% perf-profile.self.cycles-pp.aa_sk_perm 0.21 ± 6% +0.1 0.28 ± 3% perf-profile.self.cycles-pp.mark_page_accessed 0.06 ± 14% +0.1 0.12 ± 6% perf-profile.self.cycles-pp.sk_stream_alloc_skb 0.34 ± 4% +0.1 0.40 ± 5% perf-profile.self.cycles-pp.fsnotify 0.01 ±173% +0.1 0.08 ± 8% perf-profile.self.cycles-pp._find_next_bit 0.54 +0.1 0.61 ± 2% perf-profile.self.cycles-pp.__local_bh_enable_ip 0.52 ± 3% +0.1 0.59 ± 4% perf-profile.self.cycles-pp.generic_file_splice_read 0.19 ± 8% +0.1 0.26 ± 4% perf-profile.self.cycles-pp.validate_xmit_skb 0.13 ± 3% +0.1 0.21 ± 3% perf-profile.self.cycles-pp.__netif_receive_skb_one_core 0.33 ± 5% +0.1 0.41 ± 5% perf-profile.self.cycles-pp.process_backlog 0.71 ± 2% +0.1 0.80 perf-profile.self.cycles-pp.__splice_from_pipe 0.34 +0.1 0.43 ± 5% perf-profile.self.cycles-pp.__might_fault 0.52 ± 4% +0.1 0.61 ± 3% perf-profile.self.cycles-pp.atime_needs_update 0.48 ± 3% +0.1 0.57 ± 2% perf-profile.self.cycles-pp.tcp_sendpage 0.50 ± 2% +0.1 0.59 ± 6% perf-profile.self.cycles-pp.splice_direct_to_actor 0.47 ± 2% +0.1 0.57 ± 3% perf-profile.self.cycles-pp.__kmalloc_node_track_caller 0.66 +0.1 0.77 ± 3% perf-profile.self.cycles-pp.__might_sleep 0.97 +0.1 1.08 ± 4% perf-profile.self.cycles-pp.copy_page_to_iter 0.58 ± 5% +0.1 0.69 ± 6% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 1.18 ± 2% +0.1 1.29 ± 8% perf-profile.self.cycles-pp.__tcp_transmit_skb 0.46 ± 7% +0.1 0.57 ± 6% perf-profile.self.cycles-pp.pipe_to_sendpage 0.43 ± 4% +0.1 0.55 ± 5% perf-profile.self.cycles-pp.__virt_addr_valid 0.76 ± 2% +0.1 0.88 ± 2% perf-profile.self.cycles-pp.__fget_light 1.25 +0.1 1.37 perf-profile.self.cycles-pp.generic_file_buffered_read 0.47 +0.1 0.60 ± 6% perf-profile.self.cycles-pp.__netif_receive_skb_core 1.40 ± 2% +0.1 1.52 ± 2% perf-profile.self.cycles-pp.common_file_perm 0.57 ± 8% +0.1 0.70 ± 10% perf-profile.self.cycles-pp.ip_finish_output2 0.37 ± 5% +0.1 0.49 ± 5% perf-profile.self.cycles-pp.kmem_cache_alloc_node 0.26 ± 5% +0.1 0.40 ± 5% perf-profile.self.cycles-pp.ipv4_dst_check 0.54 ± 8% +0.1 0.69 ± 4% perf-profile.self.cycles-pp.inet_sendpage 0.97 ± 2% +0.2 1.12 perf-profile.self.cycles-pp.read_tsc 1.15 ± 2% +0.2 1.31 perf-profile.self.cycles-pp.do_sendfile 1.51 ± 2% +0.2 1.68 perf-profile.self.cycles-pp.___might_sleep 0.58 ± 3% +0.2 0.75 ± 4% perf-profile.self.cycles-pp._copy_to_iter 0.88 +0.2 1.07 ± 4% perf-profile.self.cycles-pp.syscall_return_via_sysret netperf.Throughput_Mbps 26500 +-------------------------------------------------------------------+ | | 26000 |-O O O O O O O O | | O O O O O O | 25500 |-+ O O | | | 25000 |-+ | | | 24500 |-+ | | | 24000 |-+ | |.+.+.+..+.+. .+..+.+.+.+ +.+.+..+.+.+.+. .+.+.. .+. | 23500 |-+ +.+.+.+.+ + + +.+ + +.+.| | + | 23000 +-------------------------------------------------------------------+ netperf.Throughput_total_Mbps 215000 +------------------------------------------------------------------+ | | 210000 |-+ O O | | O O O O O O | | O O O O O O O O | 205000 |-+ O O | | | 200000 |-+ | | | 195000 |-+ | | | | .+.+. .+. .+. .+ | 190000 |.+ +.+..+. .+.+ +.+.+.+.. +.+.+.+ +.+.+.+.+. + .+.+. .| | +.+ + + + | 185000 +------------------------------------------------------------------+ netperf.workload 1.56e+09 +----------------------------------------------------------------+ 1.54e+09 |-+ O O | | O O | 1.52e+09 |-O O O O O O O O O O O | 1.5e+09 |-+ O | | O O | 1.48e+09 |-+ | 1.46e+09 |-+ | 1.44e+09 |-+ | | | 1.42e+09 |-+ | 1.4e+09 |-+.+.+. .+. .+ | |.+ +.+. .+.+.+.+.+.+.+.. +.+.+.+ +.+.+.+.+ + .+.+. .| 1.38e+09 |-+ +.+.+ + + + | 1.36e+09 +----------------------------------------------------------------+ netperf.time.involuntary_context_switches 500000 +------------------------------------------------------------------+ | O O O O O O O | 450000 |-O O O O O O O O O | 400000 |-+ O O | | | 350000 |-+ | 300000 |-+ | | | 250000 |-+ | 200000 |-+ + | | +. + : + .+. +. | 150000 |-+.+.+.+. + +.+. .+.+ : .. + .+. .+ +. .+. + +. | 100000 |.+ +..+ + +.+ + + + +.+.. .+ +.| | + | 50000 +------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen