Greeting, FYI, we noticed a -37.5% regression of fio.read_bw_MBps due to commit: commit: 6942795656e5c8906b059622a23352a2e3c38d06 ("sched/numa: Replace runnable_load_avg by load_avg") https://github.com/0day-ci/linux/commits/UPDATE-20200221-072833/Vincent-Guittot/remove-runnable_load_avg-and-improve-group_classify/20200217-172341 in testcase: fio-basic on test machine: 96 threads Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 256G memory with following parameters: disk: 2pmem fs: ext4 mount_option: dax runtime: 200s nr_task: 50% time_based: tb rw: read bs: 2M ioengine: libaio test_size: 200G cpufreq_governor: performance ucode: 0x500002c test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. test-url: https://github.com/axboe/fio If you fix the issue, kindly add following tag Reported-by: kernel test robot Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/mount_option/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/time_based/ucode: 2M/gcc-7/performance/2pmem/ext4/libaio/x86_64-rhel-7.6/dax/50%/debian-x86_64-20191114.cgz/200s/read/lkp-csl-2sp6/200G/fio-basic/tb/0x500002c commit: 1450cfa2e9 ("sched/fair: Reorder enqueue/dequeue_task_fair path") 6942795656 ("sched/numa: Replace runnable_load_avg by load_avg") 1450cfa2e91bb42b 6942795656e5c8906b059622a23 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:stack_recursion %stddev %change %stddev \ | \ 98.24 -64.9 33.30 ± 7% fio.latency_100ms% 0.11 ± 23% +9.8 9.92 ± 26% fio.latency_20ms% 0.87 ± 13% +32.5 33.34 ± 17% fio.latency_250ms% 0.02 ± 72% +7.9 7.95 ± 29% fio.latency_500ms% 0.75 ± 61% +14.7 15.48 ± 26% fio.latency_50ms% 44572 -37.5% 27854 ± 11% fio.read_bw_MBps 67633152 +218.6% 2.155e+08 ± 17% fio.read_clat_90%_us 68419584 +331.8% 2.954e+08 ± 12% fio.read_clat_95%_us 90701824 ± 13% +297.7% 3.607e+08 ± 7% fio.read_clat_99%_us 66747576 +61.9% 1.081e+08 ± 10% fio.read_clat_mean_us 8444209 ± 8% +864.2% 81414965 ± 11% fio.read_clat_stddev 22286 -37.5% 13927 ± 11% fio.read_iops 2137474 +61.9% 3460053 ± 10% fio.read_slat_mean_us 281650 ± 7% +834.8% 2632906 ± 11% fio.read_slat_stddev 12498 +9.4% 13672 ± 2% fio.time.involuntary_context_switches 173045 +127.7% 394079 ± 10% fio.time.minor_page_faults 4457312 -37.5% 2785480 ± 11% fio.workload 7639552 ± 11% -17.4% 6307328 meminfo.DirectMap2M 2125 ± 5% -21.8% 1661 ± 22% slabinfo.task_group.active_objs 2125 ± 5% -21.8% 1661 ± 22% slabinfo.task_group.num_objs 18608329 ± 4% -24.2% 14108299 ± 25% cpuidle.C1E.usage 949981 ± 97% +476.5% 5477084 ± 65% cpuidle.C6.usage 15944 ± 4% +187.3% 45809 ± 29% cpuidle.POLL.time 3274 ± 3% +359.7% 15051 ± 33% cpuidle.POLL.usage 286841 ± 21% -33.4% 191119 ± 7% numa-numastat.node0.local_node 10892 ± 62% +98.8% 21653 ± 24% numa-numastat.node0.other_node 341302 ± 17% +24.7% 425656 ± 3% numa-numastat.node1.local_node 361678 ± 15% +20.3% 435249 ± 3% numa-numastat.node1.numa_hit 18606376 ± 4% -24.2% 14106593 ± 25% turbostat.C1E 930856 ± 99% +485.9% 5454072 ± 65% turbostat.C6 284.83 -9.1% 258.93 turbostat.PkgWatt 194.88 -10.1% 175.20 ± 2% turbostat.RAMWatt 23333 ± 5% +27.5% 29740 ± 4% sched_debug.cfs_rq:/.exec_clock.stddev 1031 +8.4% 1117 ± 5% sched_debug.cfs_rq:/.load_avg.max 24291 ± 5% +28.7% 31264 ± 5% sched_debug.cfs_rq:/.min_vruntime.stddev 24291 ± 5% +28.7% 31265 ± 5% sched_debug.cfs_rq:/.spread0.stddev 228.36 ± 3% -12.7% 199.47 ± 8% sched_debug.cfs_rq:/.util_est_enqueued.avg 880.25 -42.1% 509.31 sched_debug.cfs_rq:/.util_est_enqueued.max 213.90 ± 3% -23.1% 164.53 ± 3% sched_debug.cfs_rq:/.util_est_enqueued.stddev 504436 +15.4% 581941 ± 13% sched_debug.cpu.max_idle_balance_cost.max 450.46 ±173% +1967.4% 9312 ± 92% sched_debug.cpu.max_idle_balance_cost.stddev 0.00 ± 3% +33.3% 0.00 ± 32% sched_debug.cpu.next_balance.stddev 711.24 ± 3% +25.5% 892.49 ± 14% sched_debug.cpu.ttwu_count.stddev 45086 -2.1% 44132 proc-vmstat.nr_slab_unreclaimable 45832 ± 15% +473.9% 263038 ± 16% proc-vmstat.numa_hint_faults 26159 ± 20% +872.2% 254309 ± 15% proc-vmstat.numa_hint_faults_local 684142 -1.6% 673291 proc-vmstat.numa_hit 19359 ± 7% +152.7% 48919 ± 8% proc-vmstat.numa_huge_pte_updates 652862 -1.7% 642040 proc-vmstat.numa_local 393344 ± 5% -81.8% 71659 ± 36% proc-vmstat.numa_pages_migrated 9945716 ± 7% +154.0% 25265990 ± 9% proc-vmstat.numa_pte_updates 18088 ± 6% -15.8% 15223 ± 12% proc-vmstat.pgactivate 1869788 -17.0% 1551006 proc-vmstat.pgalloc_normal 827276 +26.7% 1048183 ± 4% proc-vmstat.pgfault 393344 ± 5% -81.8% 71659 ± 36% proc-vmstat.pgmigrate_success 419472 ± 3% +73.9% 729478 ± 4% numa-vmstat.node0.nr_active_anon 419357 ± 3% +74.0% 729487 ± 4% numa-vmstat.node0.nr_anon_pages 774.50 ± 2% +76.0% 1363 ± 4% numa-vmstat.node0.nr_anon_transparent_hugepages 419480 ± 3% +73.9% 729478 ± 4% numa-vmstat.node0.nr_zone_active_anon 12236 ± 56% +375.3% 58162 ±103% numa-vmstat.node0.numa_other 433520 ± 2% -71.7% 122601 ± 28% numa-vmstat.node1.nr_active_anon 424390 ± 3% -73.2% 113726 ± 30% numa-vmstat.node1.nr_anon_pages 778.00 ± 2% -75.6% 189.75 ± 34% numa-vmstat.node1.nr_anon_transparent_hugepages 55331 ± 91% -94.9% 2798 ± 31% numa-vmstat.node1.nr_inactive_anon 7223 ± 8% -13.2% 6266 ± 3% numa-vmstat.node1.nr_kernel_stack 55498 ± 91% -93.5% 3634 ± 19% numa-vmstat.node1.nr_mapped 1499 ± 89% -74.4% 383.25 ± 29% numa-vmstat.node1.nr_page_table_pages 64513 ± 77% -81.7% 11805 ± 13% numa-vmstat.node1.nr_shmem 433520 ± 2% -71.7% 122601 ± 28% numa-vmstat.node1.nr_zone_active_anon 55331 ± 91% -94.9% 2798 ± 31% numa-vmstat.node1.nr_zone_inactive_anon 1680276 ± 3% +74.3% 2928273 ± 4% numa-meminfo.node0.Active 1673650 ± 3% +74.6% 2921940 ± 4% numa-meminfo.node0.Active(anon) 1585126 ± 3% +76.4% 2796516 ± 4% numa-meminfo.node0.AnonHugePages 1673220 ± 3% +74.6% 2921967 ± 4% numa-meminfo.node0.AnonPages 4473331 ± 5% +34.7% 6026726 ± 2% numa-meminfo.node0.MemUsed 1731411 ± 3% -71.6% 491655 ± 28% numa-meminfo.node1.Active 1730540 ± 3% -71.7% 490490 ± 28% numa-meminfo.node1.Active(anon) 1592852 ± 2% -75.5% 389474 ± 34% numa-meminfo.node1.AnonHugePages 1693936 ± 3% -73.1% 454931 ± 30% numa-meminfo.node1.AnonPages 220820 ± 91% -94.9% 11305 ± 31% numa-meminfo.node1.Inactive 220288 ± 91% -94.9% 11149 ± 30% numa-meminfo.node1.Inactive(anon) 7222 ± 8% -13.3% 6264 ± 3% numa-meminfo.node1.KernelStack 220659 ± 91% -93.6% 14223 ± 18% numa-meminfo.node1.Mapped 4720887 ± 5% -32.9% 3168169 ± 3% numa-meminfo.node1.MemUsed 5998 ± 89% -74.4% 1534 ± 28% numa-meminfo.node1.PageTables 257078 ± 77% -81.6% 47231 ± 13% numa-meminfo.node1.Shmem 35.47 ± 17% -7.8 27.63 ± 19% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 35.05 ± 17% -7.6 27.49 ± 18% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 35.06 ± 17% -7.6 27.51 ± 18% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 36.16 ± 16% -7.5 28.65 ± 18% perf-profile.calltrace.cycles-pp.secondary_startup_64 35.33 ± 16% -7.4 27.90 ± 18% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 35.33 ± 16% -7.4 27.90 ± 18% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 35.33 ± 16% -7.4 27.90 ± 18% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 35.47 ± 17% -7.7 27.73 ± 18% perf-profile.children.cycles-pp.intel_idle 35.90 ± 17% -7.6 28.25 ± 18% perf-profile.children.cycles-pp.cpuidle_enter 35.89 ± 17% -7.6 28.24 ± 18% perf-profile.children.cycles-pp.cpuidle_enter_state 36.16 ± 16% -7.5 28.65 ± 18% perf-profile.children.cycles-pp.do_idle 36.16 ± 16% -7.5 28.65 ± 18% perf-profile.children.cycles-pp.secondary_startup_64 36.16 ± 16% -7.5 28.65 ± 18% perf-profile.children.cycles-pp.cpu_startup_entry 35.33 ± 16% -7.4 27.90 ± 18% perf-profile.children.cycles-pp.start_secondary 0.06 ± 14% +0.0 0.08 ± 10% perf-profile.children.cycles-pp.get_io_u 0.04 ± 58% +0.1 0.14 ± 15% perf-profile.children.cycles-pp.native_irq_return_iret 0.23 ± 20% +0.1 0.35 ± 18% perf-profile.children.cycles-pp.scheduler_tick 35.47 ± 17% -7.8 27.71 ± 18% perf-profile.self.cycles-pp.intel_idle 0.06 ± 14% +0.0 0.08 ± 10% perf-profile.self.cycles-pp.get_io_u 0.04 ± 58% +0.1 0.14 ± 15% perf-profile.self.cycles-pp.native_irq_return_iret 42.56 -1.6% 41.89 perf-stat.i.MPKI 5.983e+09 -36.2% 3.815e+09 ± 10% perf-stat.i.branch-instructions 0.09 +0.0 0.13 ± 5% perf-stat.i.branch-miss-rate% 4931406 -6.1% 4630438 perf-stat.i.branch-misses 94.72 -0.8 93.88 perf-stat.i.cache-miss-rate% 1.454e+09 -38.1% 8.999e+08 ± 11% perf-stat.i.cache-misses 1.528e+09 -37.5% 9.546e+08 ± 11% perf-stat.i.cache-references 3.79 +62.6% 6.16 ± 10% perf-stat.i.cpi 99.92 +4.4% 104.35 perf-stat.i.cpu-migrations 104.53 +57.2% 164.30 ± 9% perf-stat.i.cycles-between-cache-misses 0.00 ± 24% +0.0 0.00 ± 19% perf-stat.i.dTLB-load-miss-rate% 6.054e+09 -36.1% 3.868e+09 ± 10% perf-stat.i.dTLB-loads 0.00 ± 8% +0.0 0.00 ± 6% perf-stat.i.dTLB-store-miss-rate% 37244 ± 16% +27.9% 47636 ± 7% perf-stat.i.dTLB-store-misses 5.943e+09 -36.6% 3.767e+09 ± 10% perf-stat.i.dTLB-stores 44.62 +4.0 48.59 perf-stat.i.iTLB-load-miss-rate% 1751240 ± 2% -14.5% 1496904 ± 4% perf-stat.i.iTLB-load-misses 2173785 -27.0% 1587288 ± 4% perf-stat.i.iTLB-loads 3.57e+10 -36.4% 2.27e+10 ± 10% perf-stat.i.instructions 20430 ± 2% -25.4% 15243 ± 6% perf-stat.i.instructions-per-iTLB-miss 0.27 -36.0% 0.17 ± 10% perf-stat.i.ipc 3552 +32.2% 4697 ± 4% perf-stat.i.minor-faults 46.28 ± 19% +10.3 56.62 ± 6% perf-stat.i.node-load-miss-rate% 67879041 ± 18% -47.8% 35408124 ± 16% perf-stat.i.node-loads 0.65 ± 4% +34.1 34.79 ± 12% perf-stat.i.node-store-miss-rate% 1231258 ± 17% +6328.0% 79144992 ± 5% perf-stat.i.node-store-misses 3.66e+08 -55.9% 1.616e+08 ± 14% perf-stat.i.node-stores 3552 +32.2% 4697 ± 4% perf-stat.i.page-faults 42.80 -1.7% 42.06 perf-stat.overall.MPKI 0.08 +0.0 0.12 ± 8% perf-stat.overall.branch-miss-rate% 95.15 -0.9 94.26 perf-stat.overall.cache-miss-rate% 3.79 +59.5% 6.05 ± 10% perf-stat.overall.cpi 93.08 +63.9% 152.59 ± 10% perf-stat.overall.cycles-between-cache-misses 0.00 ± 26% +0.0 0.00 ± 6% perf-stat.overall.dTLB-load-miss-rate% 0.00 ± 16% +0.0 0.00 ± 16% perf-stat.overall.dTLB-store-miss-rate% 44.61 +3.9 48.51 perf-stat.overall.iTLB-load-miss-rate% 20415 ± 2% -26.1% 15095 ± 6% perf-stat.overall.instructions-per-iTLB-miss 0.26 -36.6% 0.17 ± 10% perf-stat.overall.ipc 46.12 ± 19% +10.1 56.26 ± 6% perf-stat.overall.node-load-miss-rate% 0.34 ± 17% +33.2 33.58 ± 12% perf-stat.overall.node-store-miss-rate% 1606253 +1.4% 1628237 perf-stat.overall.path-length 5.954e+09 -36.4% 3.784e+09 ± 10% perf-stat.ps.branch-instructions 4924598 -7.7% 4547566 perf-stat.ps.branch-misses 1.447e+09 -38.3% 8.931e+08 ± 11% perf-stat.ps.cache-misses 1.52e+09 -37.7% 9.472e+08 ± 11% perf-stat.ps.cache-references 98.88 +4.9% 103.73 perf-stat.ps.cpu-migrations 6.024e+09 -36.3% 3.837e+09 ± 10% perf-stat.ps.dTLB-loads 36708 ± 16% +26.7% 46499 ± 7% perf-stat.ps.dTLB-store-misses 5.914e+09 -36.8% 3.737e+09 ± 10% perf-stat.ps.dTLB-stores 1741292 ± 2% -14.6% 1487851 ± 4% perf-stat.ps.iTLB-load-misses 2161209 -26.9% 1579771 ± 4% perf-stat.ps.iTLB-loads 3.552e+10 -36.6% 2.251e+10 ± 10% perf-stat.ps.instructions 3479 +31.1% 4562 ± 4% perf-stat.ps.minor-faults 67539545 ± 18% -48.1% 35041905 ± 16% perf-stat.ps.node-loads 1241582 ± 17% +6292.6% 79369525 ± 5% perf-stat.ps.node-store-misses 3.642e+08 -56.2% 1.594e+08 ± 14% perf-stat.ps.node-stores 3479 +31.1% 4562 ± 4% perf-stat.ps.page-faults 7.16e+12 -36.7% 4.533e+12 ± 10% perf-stat.total.instructions 87848 ± 2% +13.1% 99392 softirqs.CPU0.RCU 77399 ± 5% +16.8% 90404 ± 3% softirqs.CPU1.RCU 20680 ± 10% -38.0% 12829 ± 11% softirqs.CPU12.SCHED 63366 ± 9% +34.6% 85282 ± 6% softirqs.CPU2.RCU 14411 ± 17% +42.3% 20503 ± 14% softirqs.CPU2.SCHED 71375 ± 2% -31.7% 48730 ± 8% softirqs.CPU24.RCU 18288 ± 6% -70.8% 5340 ± 24% softirqs.CPU24.SCHED 13435 ± 25% -48.2% 6954 ± 14% softirqs.CPU25.SCHED 75740 ± 4% -24.8% 56985 ± 7% softirqs.CPU26.RCU 21066 ± 6% -58.0% 8842 ± 12% softirqs.CPU26.SCHED 55879 ± 11% +21.2% 67724 ± 7% softirqs.CPU29.RCU 72914 ± 5% +7.4% 78320 ± 4% softirqs.CPU30.TIMER 73132 ± 5% +12.4% 82220 ± 5% softirqs.CPU32.TIMER 57542 ± 19% +29.8% 74664 ± 4% softirqs.CPU33.RCU 10736 ± 47% +77.5% 19062 ± 5% softirqs.CPU33.SCHED 73232 ± 3% +11.8% 81879 ± 6% softirqs.CPU33.TIMER 73807 ± 7% +17.5% 86690 ± 7% softirqs.CPU34.TIMER 67653 ± 6% +15.3% 78030 ± 3% softirqs.CPU36.RCU 15841 ± 17% +25.8% 19930 ± 7% softirqs.CPU36.SCHED 72518 ± 6% +17.9% 85512 ± 11% softirqs.CPU36.TIMER 52732 ± 22% +43.5% 75670 ± 9% softirqs.CPU37.RCU 9735 ± 46% +87.2% 18226 ± 18% softirqs.CPU37.SCHED 56946 ± 9% +30.4% 74270 ± 8% softirqs.CPU42.RCU 11172 ± 20% +65.0% 18438 ± 18% softirqs.CPU42.SCHED 74163 ± 5% +17.4% 87089 ± 7% softirqs.CPU42.TIMER 67216 ± 13% +15.6% 77708 ± 3% softirqs.CPU43.RCU 15988 ± 23% +28.6% 20568 ± 7% softirqs.CPU43.SCHED 72190 ± 4% +17.0% 84445 ± 7% softirqs.CPU43.TIMER 71178 ± 5% +13.4% 80709 ± 10% softirqs.CPU44.TIMER 71448 ± 3% +19.6% 85485 ± 8% softirqs.CPU45.TIMER 72100 ± 6% +16.4% 83936 ± 8% softirqs.CPU46.TIMER 63991 ± 10% +16.0% 74252 ± 5% softirqs.CPU47.RCU 9100 ± 26% +54.8% 14086 ± 13% softirqs.CPU47.SCHED 74711 ± 4% +15.1% 86028 ± 9% softirqs.CPU47.TIMER 42540 ± 4% +15.2% 49011 softirqs.CPU48.RCU 70569 ± 7% -20.5% 56123 ± 4% softirqs.CPU50.RCU 16387 ± 16% -56.3% 7154 ± 16% softirqs.CPU50.SCHED 9666 ± 15% -29.0% 6867 ± 26% softirqs.CPU51.SCHED 71206 ± 3% -25.2% 53245 ± 3% softirqs.CPU54.RCU 17130 ± 9% -64.3% 6114 ± 12% softirqs.CPU54.SCHED 9188 ± 23% -43.1% 5225 ± 17% softirqs.CPU55.SCHED 11532 ± 37% -50.6% 5701 ± 23% softirqs.CPU56.SCHED 67349 ± 8% -25.9% 49908 ± 3% softirqs.CPU58.RCU 15473 ± 20% -65.2% 5382 ± 11% softirqs.CPU58.SCHED 60278 ± 5% +25.4% 75582 ± 10% softirqs.CPU6.RCU 12769 ± 11% +27.4% 16268 ± 23% softirqs.CPU6.SCHED 9234 ± 26% -43.5% 5213 ± 6% softirqs.CPU60.SCHED 63277 ± 9% -21.4% 49708 softirqs.CPU61.RCU 13706 ± 22% -63.6% 4989 ± 6% softirqs.CPU61.SCHED 61652 ± 15% -20.6% 48935 ± 3% softirqs.CPU64.RCU 14778 ± 35% -70.0% 4438 ± 5% softirqs.CPU64.SCHED 10062 ± 68% -54.9% 4538 ± 13% softirqs.CPU65.SCHED 67805 ± 17% -31.0% 46752 softirqs.CPU66.RCU 18786 ± 30% -79.0% 3947 softirqs.CPU66.SCHED 67465 ± 14% -28.6% 48149 ± 3% softirqs.CPU67.RCU 17232 ± 29% -75.7% 4179 ± 15% softirqs.CPU67.SCHED 65598 ± 15% -26.5% 48222 ± 4% softirqs.CPU68.RCU 16849 ± 36% -75.7% 4092 ± 14% softirqs.CPU68.SCHED 57812 ± 12% -19.6% 46482 ± 3% softirqs.CPU70.RCU 12805 ± 30% -70.9% 3728 ± 14% softirqs.CPU70.SCHED 14209 ± 50% -74.9% 3570 ± 5% softirqs.CPU71.SCHED 59819 +59.6% 95469 softirqs.CPU72.RCU 12222 ± 7% +103.0% 24811 ± 5% softirqs.CPU72.SCHED 67645 ± 10% +37.2% 92826 ± 3% softirqs.CPU73.RCU 16869 ± 20% +40.1% 23636 ± 6% softirqs.CPU73.SCHED 52613 ± 10% +77.5% 93403 ± 5% softirqs.CPU74.RCU 9444 ± 19% +156.0% 24173 ± 9% softirqs.CPU74.SCHED 62708 ± 18% +52.1% 95389 softirqs.CPU75.RCU 14321 ± 36% +70.8% 24455 ± 6% softirqs.CPU75.SCHED 71060 ± 11% +28.0% 90963 softirqs.CPU77.RCU 71421 ± 3% +9.2% 77978 ± 5% softirqs.CPU77.TIMER 65662 ± 5% +40.3% 92151 softirqs.CPU78.RCU 16183 ± 15% +43.8% 23267 ± 4% softirqs.CPU78.SCHED 71822 ± 5% +7.2% 77025 ± 4% softirqs.CPU78.TIMER 63433 ± 5% +39.0% 88177 ± 9% softirqs.CPU79.RCU 15151 ± 11% +43.9% 21807 ± 15% softirqs.CPU79.SCHED 67711 ± 8% +29.6% 87778 ± 3% softirqs.CPU80.RCU 16505 ± 14% +47.8% 24401 ± 3% softirqs.CPU80.SCHED 72148 ± 4% +11.9% 80767 ± 3% softirqs.CPU80.TIMER 70572 ± 5% +16.9% 82478 ± 5% softirqs.CPU81.TIMER 72939 ± 5% +18.5% 86441 ± 8% softirqs.CPU82.TIMER 67040 ± 7% +21.0% 81095 ± 5% softirqs.CPU83.RCU 16105 ± 17% +45.3% 23396 ± 8% softirqs.CPU83.SCHED 72257 ± 2% +14.8% 82978 ± 8% softirqs.CPU83.TIMER 62802 ± 12% +34.0% 84185 ± 7% softirqs.CPU84.RCU 14216 ± 22% +61.7% 22988 ± 11% softirqs.CPU84.SCHED 72891 ± 5% +17.2% 85435 ± 10% softirqs.CPU84.TIMER 71482 ± 5% +17.1% 83677 ± 9% softirqs.CPU87.TIMER 73030 ± 6% +15.4% 84305 ± 10% softirqs.CPU89.TIMER 71813 ± 8% +19.6% 85872 ± 3% softirqs.CPU90.RCU 19033 ± 12% +27.5% 24258 softirqs.CPU90.SCHED 71728 ± 5% +21.6% 87204 ± 6% softirqs.CPU90.TIMER 62569 ± 12% +23.2% 77080 softirqs.CPU91.RCU 14100 ± 26% +52.4% 21492 ± 4% softirqs.CPU91.SCHED 72287 ± 6% +16.8% 84404 ± 6% softirqs.CPU91.TIMER 58336 ± 13% +40.3% 81856 ± 12% softirqs.CPU92.RCU 12005 ± 27% +84.9% 22195 ± 17% softirqs.CPU92.SCHED 62597 ± 5% +31.8% 82526 ± 7% softirqs.CPU93.RCU 13640 ± 9% +67.7% 22870 ± 10% softirqs.CPU93.SCHED 72926 ± 5% +17.8% 85913 ± 6% softirqs.CPU93.TIMER 59097 ± 16% +33.6% 78962 ± 5% softirqs.CPU94.RCU 12406 ± 40% +57.1% 19486 ± 9% softirqs.CPU94.SCHED 72443 ± 3% +15.9% 83989 ± 8% softirqs.CPU94.TIMER 64865 ± 9% +26.9% 82333 ± 7% softirqs.CPU95.RCU 15528 ± 21% +42.5% 22123 ± 10% softirqs.CPU95.SCHED 2.50 ±173% +6150.0% 156.25 ± 61% interrupts.62:PCI-MSI.31981595-edge.i40e-eth0-TxRx-26 0.00 +2.6e+103% 25.75 ±166% interrupts.93:PCI-MSI.31981626-edge.i40e-eth0-TxRx-57 4613 ± 24% -26.3% 3397 ± 33% interrupts.CPU0.NMI:Non-maskable_interrupts 4613 ± 24% -26.3% 3397 ± 33% interrupts.CPU0.PMI:Performance_monitoring_interrupts 7153 ± 11% -26.9% 5226 ± 21% interrupts.CPU11.NMI:Non-maskable_interrupts 7153 ± 11% -26.9% 5226 ± 21% interrupts.CPU11.PMI:Performance_monitoring_interrupts 4589 ± 25% -29.9% 3217 ± 35% interrupts.CPU12.NMI:Non-maskable_interrupts 4589 ± 25% -29.9% 3217 ± 35% interrupts.CPU12.PMI:Performance_monitoring_interrupts 72.50 ± 12% +85.2% 134.25 ± 6% interrupts.CPU12.RES:Rescheduling_interrupts 90.50 ± 23% +40.6% 127.25 ± 12% interrupts.CPU13.RES:Rescheduling_interrupts 56.75 ± 14% +81.5% 103.00 ± 45% interrupts.CPU16.TLB:TLB_shootdowns 115.50 ± 10% +162.3% 303.00 ± 49% interrupts.CPU2.RES:Rescheduling_interrupts 94.75 ± 20% +45.9% 138.25 ± 5% interrupts.CPU22.RES:Rescheduling_interrupts 45.25 ± 21% +86.7% 84.50 ± 29% interrupts.CPU22.TLB:TLB_shootdowns 105.75 ± 6% +79.0% 189.25 ± 8% interrupts.CPU24.RES:Rescheduling_interrupts 482.50 ± 4% -6.2% 452.75 interrupts.CPU25.CAL:Function_call_interrupts 7319 ± 8% -25.0% 5487 ± 7% interrupts.CPU25.NMI:Non-maskable_interrupts 7319 ± 8% -25.0% 5487 ± 7% interrupts.CPU25.PMI:Performance_monitoring_interrupts 135.25 ± 7% +31.4% 177.75 ± 3% interrupts.CPU25.RES:Rescheduling_interrupts 392.00 ± 64% -65.2% 136.50 ± 98% interrupts.CPU25.TLB:TLB_shootdowns 81.00 ± 8% +100.3% 162.25 ± 12% interrupts.CPU26.RES:Rescheduling_interrupts 346.00 ± 60% -81.8% 63.00 ± 94% interrupts.CPU27.TLB:TLB_shootdowns 490.00 ± 5% -7.6% 453.00 interrupts.CPU29.CAL:Function_call_interrupts 6654 ± 15% -29.3% 4702 ± 37% interrupts.CPU30.NMI:Non-maskable_interrupts 6654 ± 15% -29.3% 4702 ± 37% interrupts.CPU30.PMI:Performance_monitoring_interrupts 134.50 ± 23% -40.1% 80.50 ± 6% interrupts.CPU33.RES:Rescheduling_interrupts 124.25 ± 21% -51.3% 60.50 ± 9% interrupts.CPU34.RES:Rescheduling_interrupts 200.00 ±103% -87.9% 24.25 ± 53% interrupts.CPU35.TLB:TLB_shootdowns 72.75 ± 44% -77.3% 16.50 ± 87% interrupts.CPU41.TLB:TLB_shootdowns 103.25 ± 23% -30.0% 72.25 ± 33% interrupts.CPU43.RES:Rescheduling_interrupts 207.25 ± 97% -86.5% 28.00 ± 38% interrupts.CPU43.TLB:TLB_shootdowns 75.50 ± 52% -67.2% 24.75 ± 73% interrupts.CPU44.TLB:TLB_shootdowns 217.50 ± 90% -91.6% 18.25 ± 11% interrupts.CPU46.TLB:TLB_shootdowns 7612 -68.2% 2423 ± 65% interrupts.CPU47.NMI:Non-maskable_interrupts 7612 -68.2% 2423 ± 65% interrupts.CPU47.PMI:Performance_monitoring_interrupts 33.00 ± 67% +126.5% 74.75 ± 23% interrupts.CPU5.TLB:TLB_shootdowns 105.25 ± 20% +58.0% 166.25 ± 9% interrupts.CPU50.RES:Rescheduling_interrupts 108.50 ± 13% +43.3% 155.50 ± 3% interrupts.CPU54.RES:Rescheduling_interrupts 141.25 ± 6% +14.5% 161.75 ± 4% interrupts.CPU55.RES:Rescheduling_interrupts 6918 ± 12% -27.5% 5014 ± 32% interrupts.CPU56.NMI:Non-maskable_interrupts 6918 ± 12% -27.5% 5014 ± 32% interrupts.CPU56.PMI:Performance_monitoring_interrupts 109.75 ± 20% +49.4% 164.00 ± 3% interrupts.CPU58.RES:Rescheduling_interrupts 138.00 ± 9% +21.7% 168.00 ± 5% interrupts.CPU60.RES:Rescheduling_interrupts 113.00 ± 11% +46.9% 166.00 ± 4% interrupts.CPU61.RES:Rescheduling_interrupts 100.00 ± 34% +67.8% 167.75 interrupts.CPU64.RES:Rescheduling_interrupts 93.75 ± 25% +80.0% 168.75 ± 3% interrupts.CPU66.RES:Rescheduling_interrupts 111.00 ± 23% +52.0% 168.75 interrupts.CPU67.RES:Rescheduling_interrupts 111.00 ± 23% +54.1% 171.00 ± 2% interrupts.CPU68.RES:Rescheduling_interrupts 124.75 ± 9% +37.1% 171.00 ± 2% interrupts.CPU70.RES:Rescheduling_interrupts 114.50 ± 32% +55.7% 178.25 ± 5% interrupts.CPU71.RES:Rescheduling_interrupts 131.75 ± 16% -74.2% 34.00 ± 48% interrupts.CPU72.RES:Rescheduling_interrupts 81.25 ± 58% -80.0% 16.25 ± 59% interrupts.CPU72.TLB:TLB_shootdowns 98.50 ± 19% -58.4% 41.00 ± 38% interrupts.CPU73.RES:Rescheduling_interrupts 209.75 ± 96% -88.3% 24.50 ± 65% interrupts.CPU73.TLB:TLB_shootdowns 6712 ± 24% -41.9% 3897 ± 35% interrupts.CPU74.NMI:Non-maskable_interrupts 6712 ± 24% -41.9% 3897 ± 35% interrupts.CPU74.PMI:Performance_monitoring_interrupts 152.00 ± 7% -71.4% 43.50 ± 54% interrupts.CPU74.RES:Rescheduling_interrupts 211.25 ± 91% -93.7% 13.25 ± 32% interrupts.CPU74.TLB:TLB_shootdowns 6104 ± 24% -51.9% 2938 ± 20% interrupts.CPU75.NMI:Non-maskable_interrupts 6104 ± 24% -51.9% 2938 ± 20% interrupts.CPU75.PMI:Performance_monitoring_interrupts 119.50 ± 22% -74.7% 30.25 ± 55% interrupts.CPU75.RES:Rescheduling_interrupts 103.75 ± 5% -44.3% 57.75 ± 52% interrupts.CPU76.RES:Rescheduling_interrupts 90.25 ± 54% -59.0% 37.00 ± 33% interrupts.CPU76.TLB:TLB_shootdowns 5961 ± 27% -54.1% 2733 ± 49% interrupts.CPU77.NMI:Non-maskable_interrupts 5961 ± 27% -54.1% 2733 ± 49% interrupts.CPU77.PMI:Performance_monitoring_interrupts 96.75 ± 45% -58.7% 40.00 ± 69% interrupts.CPU77.TLB:TLB_shootdowns 5637 ± 26% -63.5% 2057 ± 27% interrupts.CPU78.NMI:Non-maskable_interrupts 5637 ± 26% -63.5% 2057 ± 27% interrupts.CPU78.PMI:Performance_monitoring_interrupts 119.00 ± 23% -59.2% 48.50 ± 13% interrupts.CPU78.RES:Rescheduling_interrupts 6907 ± 12% -64.2% 2474 ± 43% interrupts.CPU79.NMI:Non-maskable_interrupts 6907 ± 12% -64.2% 2474 ± 43% interrupts.CPU79.PMI:Performance_monitoring_interrupts 119.50 ± 12% -53.8% 55.25 ± 33% interrupts.CPU79.RES:Rescheduling_interrupts 91.75 ± 40% -67.0% 30.25 ± 28% interrupts.CPU79.TLB:TLB_shootdowns 86.00 ± 35% +56.7% 134.75 ± 18% interrupts.CPU8.RES:Rescheduling_interrupts 6728 ± 25% -48.5% 3462 ± 52% interrupts.CPU80.NMI:Non-maskable_interrupts 6728 ± 25% -48.5% 3462 ± 52% interrupts.CPU80.PMI:Performance_monitoring_interrupts 100.50 ± 16% -40.8% 59.50 ± 16% interrupts.CPU80.RES:Rescheduling_interrupts 84.25 ± 50% -68.2% 26.75 ± 13% interrupts.CPU82.TLB:TLB_shootdowns 521.75 ± 4% -16.1% 437.75 ± 18% interrupts.CPU83.CAL:Function_call_interrupts 127.50 ± 45% -57.3% 54.50 ± 38% interrupts.CPU83.RES:Rescheduling_interrupts 82.00 ± 51% -61.6% 31.50 ± 28% interrupts.CPU83.TLB:TLB_shootdowns 7620 -68.2% 2425 ± 40% interrupts.CPU84.NMI:Non-maskable_interrupts 7620 -68.2% 2425 ± 40% interrupts.CPU84.PMI:Performance_monitoring_interrupts 116.00 ± 17% -52.6% 55.00 ± 16% interrupts.CPU84.RES:Rescheduling_interrupts 522.50 ± 5% -6.4% 489.25 interrupts.CPU85.CAL:Function_call_interrupts 84.25 ± 53% -65.6% 29.00 ± 43% interrupts.CPU85.TLB:TLB_shootdowns 523.00 ± 5% -7.2% 485.50 interrupts.CPU86.CAL:Function_call_interrupts 74.50 ± 52% -68.5% 23.50 ± 25% interrupts.CPU86.TLB:TLB_shootdowns 88.00 ± 53% -71.6% 25.00 ± 33% interrupts.CPU89.TLB:TLB_shootdowns 107.75 ± 24% -53.4% 50.25 ± 10% interrupts.CPU90.RES:Rescheduling_interrupts 218.25 ± 86% -85.0% 32.75 ± 38% interrupts.CPU90.TLB:TLB_shootdowns 127.00 ± 12% -33.1% 85.00 ± 25% interrupts.CPU91.RES:Rescheduling_interrupts 81.75 ± 50% -65.4% 28.25 ± 18% interrupts.CPU91.TLB:TLB_shootdowns 6328 ± 12% -52.9% 2983 ± 40% interrupts.CPU92.NMI:Non-maskable_interrupts 6328 ± 12% -52.9% 2983 ± 40% interrupts.CPU92.PMI:Performance_monitoring_interrupts 125.25 ± 16% -51.9% 60.25 ± 31% interrupts.CPU92.RES:Rescheduling_interrupts 6390 ± 21% -50.9% 3134 ± 45% interrupts.CPU93.NMI:Non-maskable_interrupts 6390 ± 21% -50.9% 3134 ± 45% interrupts.CPU93.PMI:Performance_monitoring_interrupts 137.00 ± 9% -55.7% 60.75 ± 23% interrupts.CPU93.RES:Rescheduling_interrupts 85.50 ± 56% -65.5% 29.50 ± 10% interrupts.CPU93.TLB:TLB_shootdowns 531.75 ± 6% -7.9% 489.75 interrupts.CPU94.CAL:Function_call_interrupts 127.25 ± 26% -52.1% 61.00 ± 41% interrupts.CPU94.RES:Rescheduling_interrupts 89.00 ± 57% -67.1% 29.25 ± 5% interrupts.CPU94.TLB:TLB_shootdowns 531.00 ± 7% -8.2% 487.50 ± 2% interrupts.CPU95.CAL:Function_call_interrupts 4741 ± 24% -52.3% 2260 ± 74% interrupts.CPU95.NMI:Non-maskable_interrupts 4741 ± 24% -52.3% 2260 ± 74% interrupts.CPU95.PMI:Performance_monitoring_interrupts 342.00 ± 45% -77.8% 76.00 ± 65% interrupts.CPU95.TLB:TLB_shootdowns 11178 ± 24% -48.7% 5734 ± 26% interrupts.TLB:TLB_shootdowns fio.read_bw_MBps 45000 +-------------------------------------------------------------------+ | | 40000 |-+ | 35000 |-+ O O O O | | O O O O O O O O O O O O O O | 30000 |-+ O O O O | 25000 |-+ O O O O O | | O | 20000 |-+ | 15000 |-+ | | | 10000 |-+ | 5000 |-+ | | | 0 +-------------------------------------------------------------------+ fio.read_iops 25000 +-------------------------------------------------------------------+ | .+.+. .+. .+ | |.+..+.+.+.+..+.+.+.+..+.+.+.+. +..+ +.+..+ | 20000 |-+ | | O O O O | | O O O O O O O O O O O O O O | 15000 |-+ O O O O | | O O O O O | 10000 |-+ O | | | | | 5000 |-+ | | | | | 0 +-------------------------------------------------------------------+ fio.read_clat_mean_us 1.4e+08 +-----------------------------------------------------------------+ | O | 1.2e+08 |-+ O | | O O O O O | 1e+08 |-+ O O O O O O O O O O | | O O O O O O O | 8e+07 |-+ O O O O | |.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+ | 6e+07 |-+ | | | 4e+07 |-+ | | | 2e+07 |-+ | | | 0 +-----------------------------------------------------------------+ fio.read_clat_stddev 1e+08 +-------------------------------------------------------------------+ 9e+07 |-+ O O O | | O O O | 8e+07 |-+ O O O O O O | 7e+07 |-+ O O O O O | | O O O | 6e+07 |-O O O O O O | 5e+07 |-+ O O | 4e+07 |-+ | | | 3e+07 |-+ | 2e+07 |-+ | | .+. .+. .+. | 1e+07 |.+. +.+.+..+.+.+.+. + +..+.+.+..+.+.+.+..+.+ | 0 +-------------------------------------------------------------------+ fio.read_clat_90__us 3e+08 +-----------------------------------------------------------------+ | O | 2.5e+08 |-+ | | O O O | | | 2e+08 |-+ O O O O O O O | | O O O O O O O O | 1.5e+08 |-O O O O O O O O | | | 1e+08 |-+ | | | |.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+ | 5e+07 |-+ | | | 0 +-----------------------------------------------------------------+ fio.read_clat_99__us 4e+08 +-----------------------------------------------------------------+ | O O O O O O O O O | 3.5e+08 |-+ O O O O O O | 3e+08 |-+ O O O | | O O O O O | 2.5e+08 |-O O O | | | 2e+08 |-+ | | | 1.5e+08 |-+ | 1e+08 |.+.+.. .+. .+. .+.. .+.+. .+. | | +.+.+.+.+. + + + +.+..+.+.+ + | 5e+07 |-+ | | | 0 +-----------------------------------------------------------------+ fio.read_slat_mean_us 4.5e+06 +-----------------------------------------------------------------+ | O | 4e+06 |-+ O | 3.5e+06 |-+ O O O O O | | O O O O O O O O O O | 3e+06 |-O O O O O O O | 2.5e+06 |-+ O O O O | |.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+ | 2e+06 |-+ | 1.5e+06 |-+ | | | 1e+06 |-+ | 500000 |-+ | | | 0 +-----------------------------------------------------------------+ fio.read_slat_stddev 3.5e+06 +-----------------------------------------------------------------+ | | 3e+06 |-+ O O O O | | O O O O O | 2.5e+06 |-+ O O O | | O O O O O O | 2e+06 |-+ O O O | | O O O O O O O | 1.5e+06 |-+ | | | 1e+06 |-+ | | | 500000 |-+ | |.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.+.+.+ | 0 +-----------------------------------------------------------------+ fio.latency_100ms_ 100 +---------------------------------------------------------------------+ 90 |-+ + +. + + | | | 80 |-+ | 70 |-+ | | | 60 |-+ | 50 |-+ | 40 |-O O O O O O O O | | O O O O O O O O O O O O O O | 30 |-+ O O O O O O | 20 |-+ | | | 10 |-+ | 0 +---------------------------------------------------------------------+ fio.latency_250ms_ 45 +----------------------------------------------------------------------+ | O | 40 |-+ O | 35 |-+ O O | | O O O O O O O O O O | 30 |-+ O O O O O O O O O O O O | 25 |-+ O O | | | 20 |-+ | 15 |-+ | | | 10 |-+ | 5 |-+ | | .+. | 0 +----------------------------------------------------------------------+ fio.workload 4.5e+06 +-----------------------------------------------------------------+ | | 4e+06 |-+ | 3.5e+06 |-+ O O O O | | O O O O O O O O O O O O O O | 3e+06 |-+ O O O O | 2.5e+06 |-+ O O O O O | | O | 2e+06 |-+ | 1.5e+06 |-+ | | | 1e+06 |-+ | 500000 |-+ | | | 0 +-----------------------------------------------------------------+ fio.time.minor_page_faults 500000 +------------------------------------------------------------------+ 450000 |-+ O O | | O O O O | 400000 |-+ O O O O O O O O | 350000 |-O O O O O O O O O O O O | | O O | 300000 |-+ | 250000 |-+ | 200000 |-+ | |.+.+..+.+.+.+..+.+.+.+..+.+.+.+..+.+.+.+.+..+.+.+ | 150000 |-+ | 100000 |-+ | | | 50000 |-+ | 0 +------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen