Greeting, FYI, we noticed a -9.3% regression of fio.write_bw_MBps due to commit: commit: b91f4a426e30203b622094e01f64f38034f02d7f ("[PATCH 2/2] blk-mq: always call into the scheduler in blk_mq_make_request()") url: https://github.com/0day-ci/linux/commits/Hannes-Reinecke/blk-mq-fixup-request-re-insert-in-blk_mq_try_issue_list_directly/20190919-174642 in testcase: fio-basic on test machine: 72 threads Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz with 192G memory with following parameters: runtime: 300s disk: 1HDD fs: ext4 nr_task: 100% test_size: 128G rw: write bs: 4k ioengine: sync ucode: 0x200005e cpufreq_governor: performance test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. test-url: https://github.com/axboe/fio In addition to that, the commit also has significant impact on the following tests: +------------------+----------------------------------------------------------------------+ | testcase: change | fio-basic: fio.write_bw_MBps -10.9% regression | | test machine | 72 threads Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz with 192G memory | | test parameters | bs=4k | | | cpufreq_governor=performance | | | disk=1HDD | | | fs=btrfs | | | ioengine=sync | | | nr_task=100% | | | runtime=300s | | | rw=write | | | test_size=128G | | | ucode=0x200005e | +------------------+----------------------------------------------------------------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode: 4k/gcc-7/performance/1HDD/ext4/sync/x86_64-rhel-7.6/100%/debian-x86_64-2019-05-14.cgz/300s/write/lkp-skl-2sp8/128G/fio-basic/0x200005e commit: ff08148702 ("blk-mq: fixup request re-insert in blk_mq_try_issue_list_directly()") b91f4a426e ("blk-mq: always call into the scheduler in blk_mq_make_request()") ff0814870258907f b91f4a426e30203b622094e01f6 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 1:4 -25% :4 dmesg.WARNING:at#for_ip_interrupt_entry/0x %stddev %change %stddev \ | \ 6.62 ± 39% -5.1 1.48 ± 11% fio.latency_100us% 13.70 ± 13% +5.2 18.86 ± 9% fio.latency_20us% 0.07 ± 2% +0.0 0.08 ± 4% fio.latency_250ms% 9.82 ± 37% -4.2 5.59 ± 40% fio.latency_4us% 0.26 ± 48% -0.2 0.03 ± 8% fio.latency_500us% 0.02 ± 8% +0.0 0.02 ± 7% fio.latency_50ms% 23.32 ± 25% +11.2 34.56 fio.latency_50us% 501.83 +5.9% 531.37 fio.time.elapsed_time 501.83 +5.9% 531.37 fio.time.elapsed_time.max 1.694e+08 -9.3% 1.537e+08 fio.time.file_system_outputs 6683 ± 15% -30.8% 4627 ± 3% fio.time.involuntary_context_switches 116.50 ± 18% -41.6% 68.00 ± 2% fio.time.percent_of_cpu_this_job_got 566.73 ± 19% -40.1% 339.73 ± 2% fio.time.system_time 21173558 -9.3% 19213732 fio.workload 275.63 -9.3% 250.12 fio.write_bw_MBps 71360 ± 28% -53.0% 33504 ± 2% fio.write_clat_90%_us 120512 ± 36% -65.7% 41344 ± 2% fio.write_clat_95%_us 62652416 +8.2% 67764224 ± 5% fio.write_clat_99%_us 1018693 +10.2% 1122642 fio.write_clat_mean_us 9332015 +6.2% 9912654 ± 2% fio.write_clat_stddev 70561 -9.3% 64030 fio.write_iops 716.00 ± 4% -15.6% 604.00 ± 9% slabinfo.uid_cache.num_objs 159147 ± 6% +23.5% 196570 ± 22% softirqs.CPU12.TIMER 0.03 ± 5% -0.0 0.02 ± 2% mpstat.cpu.all.soft% 1.40 ± 22% -0.7 0.73 mpstat.cpu.all.sys% 11287932 -9.4% 10232075 ± 3% numa-numastat.node1.local_node 11296294 -9.3% 10240386 ± 3% numa-numastat.node1.numa_hit 82.00 ± 11% -25.6% 61.00 turbostat.Avg_MHz 4.86 ± 5% -0.7 4.19 ± 3% turbostat.Busy% 1691 ± 6% -13.6% 1462 ± 4% turbostat.Bzy_MHz 46.51 +4.6% 48.67 iostat.cpu.idle 52.00 -2.9% 50.51 iostat.cpu.iowait 1.42 ± 21% -46.8% 0.76 iostat.cpu.system 440.26 ±172% -99.3% 2.89 ± 57% iostat.sdb.r_await.max 942.75 ± 15% -44.2% 525.75 numa-meminfo.node0.Mlocked 35989 ± 2% +142.8% 87375 numa-meminfo.node0.Writeback 943.50 ± 15% -59.2% 384.75 numa-meminfo.node1.Mlocked 35665 ± 3% +137.9% 84863 ± 3% numa-meminfo.node1.Writeback 5480 ± 4% +6.3% 5823 ± 5% meminfo.Buffers 7040000 ± 6% -8.0% 6476288 meminfo.DirectMap2M 1886 -51.8% 909.25 meminfo.Mlocked 71533 +141.4% 172651 ± 2% meminfo.Writeback 177731 -13.9% 153014 meminfo.max_used_kB 235.50 ± 15% -44.4% 131.00 numa-vmstat.node0.nr_mlock 9207 ± 3% +138.0% 21912 numa-vmstat.node0.nr_writeback 235.50 ± 15% -59.4% 95.50 numa-vmstat.node1.nr_mlock 8697 ± 3% +143.3% 21161 ± 3% numa-vmstat.node1.nr_writeback 5397979 ± 3% -10.5% 4831386 ± 4% numa-vmstat.node1.nr_written 46.25 +3.8% 48.00 vmstat.cpu.id 51.25 -2.4% 50.00 vmstat.cpu.wa 167970 -14.1% 144239 vmstat.io.bo 5469 ± 4% +6.2% 5810 ± 5% vmstat.memory.buff 3079 ± 2% -7.0% 2864 vmstat.system.cs 135.35 ±125% -100.0% 0.00 ± 3% sched_debug.cfs_rq:/.MIN_vruntime.stddev 6969 ± 19% -39.3% 4233 ± 2% sched_debug.cfs_rq:/.exec_clock.avg 7965 ± 16% -33.0% 5335 sched_debug.cfs_rq:/.exec_clock.max 6505 ± 20% -42.4% 3749 ± 2% sched_debug.cfs_rq:/.exec_clock.min 135.35 ±125% -100.0% 0.00 ± 3% sched_debug.cfs_rq:/.max_vruntime.stddev 58920 ± 36% -25.0% 44204 ± 37% sched_debug.cpu.nr_switches.max 7912 ± 21% -22.8% 6109 ± 21% sched_debug.cpu.nr_switches.stddev 7728 ± 19% -21.8% 6040 ± 17% sched_debug.cpu.sched_count.stddev 26049 ± 36% -25.4% 19442 ± 36% sched_debug.cpu.sched_goidle.max 3614 ± 19% -22.7% 2793 ± 20% sched_debug.cpu.sched_goidle.stddev 2.625e+08 ± 2% -10.0% 2.362e+08 ± 2% perf-stat.i.branch-instructions 1.85 +0.1 1.91 perf-stat.i.branch-miss-rate% 3074 ± 2% -7.1% 2857 ± 2% perf-stat.i.context-switches 5.331e+09 ± 11% -28.5% 3.812e+09 perf-stat.i.cpu-cycles 0.03 ± 3% -0.0 0.03 ± 4% perf-stat.i.dTLB-load-miss-rate% 3.432e+08 -9.8% 3.095e+08 perf-stat.i.dTLB-loads 1.875e+08 -6.7% 1.75e+08 perf-stat.i.dTLB-stores 1.28e+09 ± 2% -9.4% 1.159e+09 ± 2% perf-stat.i.instructions 79.86 +1.0 80.89 perf-stat.i.node-load-miss-rate% 1.60 ± 2% +0.1 1.70 perf-stat.overall.branch-miss-rate% 4.16 ± 10% -20.9% 3.29 ± 2% perf-stat.overall.cpi 1756 ± 7% -21.9% 1371 ± 3% perf-stat.overall.cycles-between-cache-misses 0.24 ± 12% +24.8% 0.30 ± 2% perf-stat.overall.ipc 2.62e+08 ± 2% -10.0% 2.358e+08 ± 2% perf-stat.ps.branch-instructions 3068 ± 2% -7.0% 2851 ± 2% perf-stat.ps.context-switches 5.323e+09 ± 11% -28.4% 3.809e+09 perf-stat.ps.cpu-cycles 3.426e+08 -9.8% 3.09e+08 perf-stat.ps.dTLB-loads 1.872e+08 -6.7% 1.747e+08 perf-stat.ps.dTLB-stores 1.278e+09 ± 2% -9.4% 1.157e+09 ± 2% perf-stat.ps.instructions 6.427e+11 ± 2% -4.3% 6.153e+11 ± 2% perf-stat.total.instructions 0.75 ±173% +6700.0% 51.00 ±151% interrupts.43:PCI-MSI.31981576-edge.i40e-eth0-TxRx-7 396.00 ± 28% +76.3% 698.25 ± 45% interrupts.CPU0.RES:Rescheduling_interrupts 623.75 ± 57% -90.1% 61.50 ± 15% interrupts.CPU27.RES:Rescheduling_interrupts 151.00 ± 30% +45.9% 220.25 ± 11% interrupts.CPU45.NMI:Non-maskable_interrupts 151.00 ± 30% +45.9% 220.25 ± 11% interrupts.CPU45.PMI:Performance_monitoring_interrupts 80.00 ± 86% +104.4% 163.50 ± 36% interrupts.CPU45.RES:Rescheduling_interrupts 167.75 ± 30% +46.8% 246.25 ± 15% interrupts.CPU54.NMI:Non-maskable_interrupts 167.75 ± 30% +46.8% 246.25 ± 15% interrupts.CPU54.PMI:Performance_monitoring_interrupts 166.50 ± 27% +44.4% 240.50 ± 3% interrupts.CPU57.NMI:Non-maskable_interrupts 166.50 ± 27% +44.4% 240.50 ± 3% interrupts.CPU57.PMI:Performance_monitoring_interrupts 182.25 ± 18% +33.7% 243.75 ± 8% interrupts.CPU58.NMI:Non-maskable_interrupts 182.25 ± 18% +33.7% 243.75 ± 8% interrupts.CPU58.PMI:Performance_monitoring_interrupts 59.25 ± 60% +83.5% 108.75 ± 46% interrupts.CPU60.RES:Rescheduling_interrupts 179.75 ± 19% +61.9% 291.00 ± 25% interrupts.CPU62.NMI:Non-maskable_interrupts 179.75 ± 19% +61.9% 291.00 ± 25% interrupts.CPU62.PMI:Performance_monitoring_interrupts 4979 ± 5% +9.6% 5457 ± 7% interrupts.CPU65.CAL:Function_call_interrupts 4868 +25.0% 6085 ± 16% interrupts.CPU66.CAL:Function_call_interrupts 164.50 ± 28% +35.1% 222.25 ± 3% interrupts.CPU69.NMI:Non-maskable_interrupts 164.50 ± 28% +35.1% 222.25 ± 3% interrupts.CPU69.PMI:Performance_monitoring_interrupts 0.75 ±173% +6633.3% 50.50 ±153% interrupts.CPU7.43:PCI-MSI.31981576-edge.i40e-eth0-TxRx-7 338.25 ± 59% -77.9% 74.75 ± 36% interrupts.CPU7.RES:Rescheduling_interrupts 4705 ± 18% +36.7% 6432 ± 21% interrupts.CPU70.CAL:Function_call_interrupts 21175431 -9.3% 19215476 proc-vmstat.nr_dirtied 6559304 -1.9% 6432094 proc-vmstat.nr_dirty 17335665 -6.4% 16220070 proc-vmstat.nr_file_pages 31132550 +3.7% 32279732 proc-vmstat.nr_free_pages 65684 -5.1% 62356 proc-vmstat.nr_inactive_anon 16993871 -6.5% 15880980 proc-vmstat.nr_inactive_file 67545 -5.0% 64178 proc-vmstat.nr_mapped 471.25 -51.7% 227.50 proc-vmstat.nr_mlock 2160 -2.5% 2107 proc-vmstat.nr_page_table_pages 67596 -4.8% 64379 proc-vmstat.nr_shmem 491926 -6.3% 460880 proc-vmstat.nr_slab_reclaimable 17971 +139.9% 43107 proc-vmstat.nr_writeback 21175431 -9.3% 19215476 proc-vmstat.nr_written 65684 -5.1% 62356 proc-vmstat.nr_zone_inactive_anon 16993871 -6.5% 15880980 proc-vmstat.nr_zone_inactive_file 6578500 -1.6% 6475915 proc-vmstat.nr_zone_write_pending 222873 ± 60% +83.2% 408201 ± 13% proc-vmstat.numa_foreign 1201 ± 4% +24.9% 1500 ± 25% proc-vmstat.numa_hint_faults 22710959 -9.4% 20577032 proc-vmstat.numa_hit 22687436 -9.4% 20553544 proc-vmstat.numa_local 222873 ± 60% +83.2% 408201 ± 13% proc-vmstat.numa_miss 246396 ± 55% +75.2% 431690 ± 12% proc-vmstat.numa_other 23046534 -8.5% 21092140 proc-vmstat.pgalloc_normal 1398640 +5.2% 1471305 proc-vmstat.pgfault 23033875 -8.5% 21079937 proc-vmstat.pgfree 84865276 -9.2% 77023176 proc-vmstat.pgpgout 12.13 ± 5% -1.8 10.29 ± 4% perf-profile.calltrace.cycles-pp.__GI___libc_write 11.65 ± 6% -1.7 9.99 ± 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_write 11.63 ± 6% -1.7 9.98 ± 5% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write 10.60 ± 7% -1.6 9.00 ± 5% perf-profile.calltrace.cycles-pp.__generic_file_write_iter.ext4_file_write_iter.new_sync_write.vfs_write.ksys_write 11.17 ± 7% -1.6 9.58 ± 4% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write 10.78 ± 7% -1.6 9.20 ± 5% perf-profile.calltrace.cycles-pp.ext4_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64 11.09 ± 7% -1.6 9.53 ± 5% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write 10.82 ± 7% -1.5 9.28 ± 5% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 9.93 ± 8% -1.4 8.56 ± 6% perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter.new_sync_write.vfs_write 2.43 ± 9% -0.6 1.82 ± 11% perf-profile.calltrace.cycles-pp.ext4_da_write_end.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter.new_sync_write 1.11 ± 15% -0.4 0.69 ± 15% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.__generic_file_write_iter 1.16 ± 15% -0.4 0.76 ± 4% perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter.new_sync_write 1.12 ± 15% -0.4 0.75 ± 3% perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter 1.19 ± 21% -0.2 0.98 ± 5% perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.ext4_da_write_begin.generic_perform_write.__generic_file_write_iter 0.79 ± 14% -0.2 0.59 ± 9% perf-profile.calltrace.cycles-pp.ext4_block_write_begin.ext4_da_write_begin.generic_perform_write.__generic_file_write_iter.ext4_file_write_iter 1.37 ± 8% +0.2 1.54 ± 4% perf-profile.calltrace.cycles-pp.__next_timer_interrupt.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select 1.38 ± 6% +0.2 1.59 ± 3% perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 2.29 ± 6% +0.2 2.50 ± 2% perf-profile.calltrace.cycles-pp.get_next_timer_interrupt.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle 82.51 +1.6 84.06 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 82.51 +1.6 84.06 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 82.45 +1.6 84.01 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 83.41 +1.7 85.13 perf-profile.calltrace.cycles-pp.secondary_startup_64 12.16 ± 5% -1.8 10.33 ± 4% perf-profile.children.cycles-pp.__GI___libc_write 13.18 ± 6% -1.6 11.57 ± 4% perf-profile.children.cycles-pp.do_syscall_64 13.20 ± 6% -1.6 11.60 ± 4% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 10.60 ± 7% -1.6 9.00 ± 5% perf-profile.children.cycles-pp.__generic_file_write_iter 10.78 ± 7% -1.6 9.20 ± 5% perf-profile.children.cycles-pp.ext4_file_write_iter 11.21 ± 7% -1.6 9.63 ± 5% perf-profile.children.cycles-pp.ksys_write 11.12 ± 7% -1.6 9.57 ± 5% perf-profile.children.cycles-pp.vfs_write 10.85 ± 7% -1.5 9.32 ± 5% perf-profile.children.cycles-pp.new_sync_write 9.93 ± 8% -1.4 8.56 ± 6% perf-profile.children.cycles-pp.generic_perform_write 2.44 ± 8% -0.6 1.83 ± 11% perf-profile.children.cycles-pp.ext4_da_write_end 1.38 ± 21% -0.5 0.87 ± 15% perf-profile.children.cycles-pp.__ext4_journal_stop 1.35 ± 21% -0.5 0.85 ± 15% perf-profile.children.cycles-pp.jbd2_journal_stop 1.12 ± 15% -0.4 0.71 ± 14% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 1.16 ± 15% -0.4 0.76 ± 4% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic 1.12 ± 15% -0.4 0.75 ± 4% perf-profile.children.cycles-pp.copyin 1.20 ± 20% -0.2 0.99 ± 5% perf-profile.children.cycles-pp.pagecache_get_page 0.79 ± 14% -0.2 0.59 ± 9% perf-profile.children.cycles-pp.ext4_block_write_begin 0.48 ± 13% -0.1 0.34 ± 13% perf-profile.children.cycles-pp.ext4_da_get_block_prep 0.32 ± 28% -0.1 0.18 ± 14% perf-profile.children.cycles-pp.ext4_es_lookup_extent 0.28 ± 21% -0.1 0.18 ± 14% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.10 ± 18% -0.0 0.07 ± 5% perf-profile.children.cycles-pp.__note_gp_changes 0.09 ± 8% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.__update_load_avg_se 0.04 ± 58% +0.0 0.08 ± 15% perf-profile.children.cycles-pp.__accumulate_pelt_segments 0.12 ± 8% +0.0 0.17 ± 12% perf-profile.children.cycles-pp.__hrtimer_get_next_event 0.01 ±173% +0.1 0.07 ± 30% perf-profile.children.cycles-pp.filemap_map_pages 0.01 ±173% +0.1 0.07 ± 30% perf-profile.children.cycles-pp.rcu_gp_kthread 0.00 +0.1 0.06 ± 14% perf-profile.children.cycles-pp.autoremove_wake_function 0.03 ±105% +0.1 0.09 ± 27% perf-profile.children.cycles-pp.kjournald2 0.03 ±105% +0.1 0.09 ± 27% perf-profile.children.cycles-pp.jbd2_journal_commit_transaction 0.00 +0.1 0.06 ± 20% perf-profile.children.cycles-pp.sched_clock_idle_wakeup_event 0.00 +0.1 0.07 ± 13% perf-profile.children.cycles-pp.__wake_up_common 0.15 ± 24% +0.1 0.21 ± 11% perf-profile.children.cycles-pp.fb_flashcursor 0.15 ± 24% +0.1 0.21 ± 11% perf-profile.children.cycles-pp.bit_cursor 0.15 ± 24% +0.1 0.21 ± 11% perf-profile.children.cycles-pp.soft_cursor 0.14 ± 26% +0.1 0.21 ± 11% perf-profile.children.cycles-pp.memcpy_toio 0.14 ± 26% +0.1 0.21 ± 11% perf-profile.children.cycles-pp.ast_dirty_update 0.45 ± 7% +0.1 0.54 ± 8% perf-profile.children.cycles-pp.__hrtimer_next_event_base 0.74 ± 6% +0.1 0.83 ± 5% perf-profile.children.cycles-pp._raw_spin_trylock 1.45 ± 7% +0.2 1.63 ± 2% perf-profile.children.cycles-pp.native_irq_return_iret 1.56 ± 8% +0.2 1.78 ± 3% perf-profile.children.cycles-pp.__next_timer_interrupt 2.33 ± 6% +0.2 2.55 ± 3% perf-profile.children.cycles-pp.get_next_timer_interrupt 1.41 ± 6% +0.2 1.63 ± 3% perf-profile.children.cycles-pp.rcu_sched_clock_irq 82.51 +1.6 84.06 perf-profile.children.cycles-pp.start_secondary 83.41 +1.7 85.13 perf-profile.children.cycles-pp.secondary_startup_64 83.41 +1.7 85.13 perf-profile.children.cycles-pp.cpu_startup_entry 83.51 +1.7 85.24 perf-profile.children.cycles-pp.do_idle 0.93 ± 11% -0.2 0.70 ± 14% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.28 ± 21% -0.1 0.18 ± 14% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.27 ± 21% -0.1 0.19 ± 22% perf-profile.self.cycles-pp.tsc_verify_tsc_adjust 0.19 ± 8% -0.1 0.13 ± 18% perf-profile.self.cycles-pp.update_blocked_averages 0.15 ± 17% -0.0 0.10 ± 8% perf-profile.self.cycles-pp.__pagevec_lru_add_fn 0.08 ± 15% +0.0 0.11 ± 6% perf-profile.self.cycles-pp.__update_load_avg_se 0.04 ± 57% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.__accumulate_pelt_segments 0.14 ± 26% +0.1 0.21 ± 10% perf-profile.self.cycles-pp.memcpy_toio 0.01 ±173% +0.1 0.09 ± 27% perf-profile.self.cycles-pp.intel_pmu_disable_all 0.39 ± 4% +0.1 0.48 ± 8% perf-profile.self.cycles-pp.__hrtimer_next_event_base 1.17 ± 6% +0.1 1.31 ± 4% perf-profile.self.cycles-pp.rcu_sched_clock_irq 1.44 ± 8% +0.2 1.62 perf-profile.self.cycles-pp.native_irq_return_iret 2.63 +0.3 2.94 ± 4% perf-profile.self.cycles-pp.cpuidle_enter_state fio.write_bw_MBps 280 +-+-------------------------------------------------------------------+ | :+ + + + :+ + :+ + .++.+.+ .+. | 275 +-++ + + + +.++.++ ++.++.| | | 270 +-+ | | | 265 +-+ | | | 260 +-+ | | | 255 +-+ | O OO O OO | 250 +-+ OO OO O OO OO OO | | | 245 +-+-------------------------------------------------------------------+ fio.write_iops 72000 +-+-----------------------------------------------------------------+ |.+ ++.++.++. +.+.++.++.++.++.+ +.++.+ ++. +. | 71000 +-+:+ + :+ :+ + ++.+.++.++.++.+ .++.| 70000 +-++ + + + | | | 69000 +-+ | 68000 +-+ | | | 67000 +-+ | 66000 +-+ | | | 65000 +-+ | 64000 O-OO OO OO OO OO O O | | O O O O | 63000 +-+-----------------------------------------------------------------+ fio.write_clat_mean_us 1.14e+06 +-+--------------------------------------------------------------+ | O O | 1.12e+06 +O+OO OO OO OO OO O OO | O | 1.1e+06 +-+ | | | 1.08e+06 +-+ | | | 1.06e+06 +-+ | | | 1.04e+06 +-+ | | | 1.02e+06 +-++ + + .+ .+ +.++.++.++.++.+| |+ :.++.+ .++. .+ .+ .+ .+ + + .+ + + + +.+ | 1e+06 +-+--------------------------------------------------------------+ fio.workload 2.15e+07 +-+--------------------------------------------------------------+ | + : + + + :+ + :+ +. +.++.+ | | + + + + +.++.++.++.++.+| 2.1e+07 +-+ | | | | | 2.05e+07 +-+ | | | 2e+07 +-+ | | | | | 1.95e+07 +-+ | OO OO O OO | | O OO OO OO OO O | 1.9e+07 +-+--------------------------------------------------------------+ fio.time.elapsed_time 535 +-+-------------------------------------------------------------------+ | O O OO OO OO | 530 O-OO OO O O O | 525 +-+ O | | | 520 +-+ | 515 +-+ | | | 510 +-+ | 505 +-+ | | +. +. +. +. .+ .++.++.+.++.+ | 500 +-+ ++.+.+ .++.+ +.++.+ +.+ ++.+ + +.| 495 +-+ +.+ .+.++.++.++.+ | | + | 490 +-+-------------------------------------------------------------------+ fio.time.elapsed_time.max 535 +-+-------------------------------------------------------------------+ | O O OO OO OO | 530 O-OO OO O O O | 525 +-+ O | | | 520 +-+ | 515 +-+ | | | 510 +-+ | 505 +-+ | | +. +. +. +. .+ .++.++.+.++.+ | 500 +-+ ++.+.+ .++.+ +.++.+ +.+ ++.+ + +.| 495 +-+ +.+ .+.++.++.++.+ | | + | 490 +-+-------------------------------------------------------------------+ fio.time.file_system_outputs 1.72e+08 +-+--------------------------------------------------------------+ 1.7e+08 +-+ : + + + :+ + :+ +. +.++.+ | | + + + + +.++.++.++.++.+| 1.68e+08 +-+ | 1.66e+08 +-+ | | | 1.64e+08 +-+ | 1.62e+08 +-+ | 1.6e+08 +-+ | | | 1.58e+08 +-+ | 1.56e+08 +-+ | OO OO O OO | 1.54e+08 +-+ O OO OO OO OO O | 1.52e+08 +-+--------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-skl-2sp8: 72 threads Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz with 192G memory ========================================================================================= bs/compiler/cpufreq_governor/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode: 4k/gcc-7/performance/1HDD/btrfs/sync/x86_64-rhel-7.6/100%/debian-x86_64-2019-05-14.cgz/300s/write/lkp-skl-2sp8/128G/fio-basic/0x200005e commit: ff08148702 ("blk-mq: fixup request re-insert in blk_mq_try_issue_list_directly()") b91f4a426e ("blk-mq: always call into the scheduler in blk_mq_make_request()") ff0814870258907f b91f4a426e30203b622094e01f6 ---------------- --------------------------- %stddev %change %stddev \ | \ 38.22 +4.2 42.42 fio.latency_100us% 0.01 ± 29% +0.0 0.02 ± 8% fio.latency_10ms% 33.42 ± 4% -5.9 27.50 ± 9% fio.latency_10us% 0.02 ± 17% +0.0 0.04 ± 4% fio.latency_20ms% 10.11 ± 13% +6.6 16.76 ± 14% fio.latency_20us% 0.05 ± 5% +0.0 0.07 ± 8% fio.latency_250ms% 13.88 ± 18% -7.1 6.77 ± 26% fio.latency_4us% 0.04 ± 12% -0.0 0.02 ± 35% fio.latency_500us% 498.84 +7.9% 538.26 fio.time.elapsed_time 498.84 +7.9% 538.26 fio.time.elapsed_time.max 1.723e+08 -10.9% 1.535e+08 fio.time.file_system_outputs 139.50 ± 2% -8.1% 128.25 ± 3% fio.time.percent_of_cpu_this_job_got 349028 -7.4% 323192 ± 3% fio.time.voluntary_context_switches 21535431 -10.9% 19191998 fio.workload 280.35 -10.9% 249.83 fio.write_bw_MBps 62914560 +10.1% 69271552 ± 4% fio.write_clat_99%_us 1001578 +12.2% 1124073 fio.write_clat_mean_us 8442623 +12.1% 9463511 fio.write_clat_stddev 71769 -10.9% 63957 fio.write_iops 54452 -11.8% 48021 meminfo.Active(file) 1950699 -21.3% 1535926 meminfo.Dirty 177986 -17.2% 147322 meminfo.max_used_kB 8292 ±113% +375.5% 39434 ± 85% numa-numastat.node0.other_node 11529753 ± 3% -11.5% 10202247 ± 2% numa-numastat.node1.local_node 11544959 ± 3% -11.6% 10210605 ± 2% numa-numastat.node1.numa_hit 57.50 +3.9% 59.75 vmstat.cpu.id 172275 -17.5% 142207 vmstat.io.bo 3935 -14.7% 3356 ± 2% vmstat.system.cs 3990 ± 2% +20.5% 4808 ± 3% slabinfo.kmalloc-8k.active_objs 1052 +20.6% 1269 ± 3% slabinfo.kmalloc-8k.active_slabs 4209 +20.6% 5078 ± 3% slabinfo.kmalloc-8k.num_objs 1052 +20.6% 1269 ± 3% slabinfo.kmalloc-8k.num_slabs 993762 ± 4% -25.2% 743653 ± 7% numa-meminfo.node0.Dirty 8242 ± 86% +2124.3% 183341 ± 52% numa-meminfo.node0.Inactive(anon) 16097 ± 42% +1053.1% 185613 ± 50% numa-meminfo.node0.Shmem 946941 ± 4% -17.3% 783272 ± 3% numa-meminfo.node1.Dirty 256213 ± 2% -75.5% 62683 ±153% numa-meminfo.node1.Inactive(anon) 112316 ± 4% -6.7% 104793 ± 3% numa-meminfo.node1.KReclaimable 258811 -73.7% 68027 ±143% numa-meminfo.node1.Mapped 112316 ± 4% -6.7% 104793 ± 3% numa-meminfo.node1.SReclaimable 259933 ± 3% -73.7% 68310 ±136% numa-meminfo.node1.Shmem 57.87 +4.3% 60.34 ± 2% iostat.cpu.idle 40.31 ± 2% -5.7% 38.01 ± 3% iostat.cpu.iowait 1.77 ± 3% -9.1% 1.61 ± 2% iostat.cpu.system 44.25 ±100% -100.0% 0.00 iostat.sda.avgqu-sz.max 287.37 ±100% -100.0% 0.00 iostat.sda.await.max 1.41 ±100% -100.0% 0.00 iostat.sda.r_await.max 0.60 ±100% -100.0% 0.00 iostat.sda.svctm.max 287.37 ±100% -100.0% 0.00 iostat.sda.w_await.max 196.93 ±100% +142.4% 477.30 iostat.sdb.await 196.94 ±100% +142.4% 477.31 iostat.sdb.w_await 269628 +5.7% 285063 ± 5% sched_debug.cpu.clock.avg 269632 +5.7% 285066 ± 5% sched_debug.cpu.clock.max 269625 +5.7% 285059 ± 5% sched_debug.cpu.clock.min 269628 +5.7% 285063 ± 5% sched_debug.cpu.clock_task.avg 269632 +5.7% 285066 ± 5% sched_debug.cpu.clock_task.max 269625 +5.7% 285059 ± 5% sched_debug.cpu.clock_task.min 1037 ± 2% +6.9% 1108 ± 4% sched_debug.cpu.curr->pid.stddev 20423 +27.9% 26120 ± 6% sched_debug.cpu.sched_count.avg 269625 +5.7% 285059 ± 5% sched_debug.cpu_clk 266935 +5.8% 282369 ± 5% sched_debug.ktime 269973 +5.7% 285400 ± 5% sched_debug.sched_clk 248672 ± 4% -25.1% 186254 ± 7% numa-vmstat.node0.nr_dirty 2063 ± 85% +2119.7% 45797 ± 52% numa-vmstat.node0.nr_inactive_anon 4027 ± 42% +1051.4% 46365 ± 50% numa-vmstat.node0.nr_shmem 2063 ± 85% +2119.7% 45797 ± 52% numa-vmstat.node0.nr_zone_inactive_anon 8708 ±105% +237.7% 29405 ± 54% numa-vmstat.node0.numa_other 236734 ± 4% -17.3% 195893 ± 3% numa-vmstat.node1.nr_dirty 64096 ± 2% -75.5% 15697 ±153% numa-vmstat.node1.nr_inactive_anon 64841 -73.7% 17067 ±143% numa-vmstat.node1.nr_mapped 65026 ± 3% -73.7% 17104 ±136% numa-vmstat.node1.nr_shmem 28075 ± 4% -6.7% 26189 ± 3% numa-vmstat.node1.nr_slab_reclaimable 5470165 -13.0% 4761134 ± 4% numa-vmstat.node1.nr_written 64096 ± 2% -75.5% 15697 ±153% numa-vmstat.node1.nr_zone_inactive_anon 9754289 ± 2% -9.5% 8829087 ± 3% numa-vmstat.node1.numa_hit 9570245 ± 2% -9.6% 8651021 ± 3% numa-vmstat.node1.numa_local 37089 -9.7% 33482 softirqs.BLOCK 63404 +12.2% 71138 ± 2% softirqs.CPU24.SCHED 160959 ± 6% +47.3% 237139 ± 18% softirqs.CPU24.TIMER 113209 ± 2% -9.2% 102824 ± 4% softirqs.CPU26.RCU 160213 ± 7% +10.1% 176471 ± 4% softirqs.CPU36.TIMER 161611 ± 10% +10.8% 179104 ± 4% softirqs.CPU38.TIMER 158202 ± 8% +14.9% 181805 ± 7% softirqs.CPU41.TIMER 161394 ± 8% +9.8% 177208 ± 4% softirqs.CPU44.TIMER 61191 ± 2% +9.7% 67142 ± 2% softirqs.CPU45.SCHED 158166 ± 8% +16.4% 184035 ± 10% softirqs.CPU45.TIMER 158609 ± 8% +11.0% 176046 ± 4% softirqs.CPU48.TIMER 159505 ± 8% +10.2% 175812 ± 4% softirqs.CPU49.TIMER 163999 ± 8% +11.0% 182032 ± 7% softirqs.CPU5.TIMER 159346 ± 8% +10.9% 176725 ± 4% softirqs.CPU51.TIMER 159810 ± 8% +10.5% 176531 ± 4% softirqs.CPU52.TIMER 62252 +10.2% 68624 ± 5% softirqs.CPU60.SCHED 153923 ± 5% +32.2% 203464 ± 25% softirqs.CPU60.TIMER 162719 ± 7% +14.9% 186976 ± 11% softirqs.CPU9.TIMER 0.00 +1.2 1.21 ± 43% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__btrfs_block_rsv_release.btrfs_inode_rsv_release.btrfs_buffered_write 0.18 ±173% +1.2 1.40 ± 42% perf-profile.calltrace.cycles-pp._raw_spin_lock.__btrfs_block_rsv_release.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter 0.00 +1.2 1.23 ± 45% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write 0.20 ±173% +1.2 1.43 ± 42% perf-profile.calltrace.cycles-pp.__btrfs_block_rsv_release.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write 0.20 ±173% +1.2 1.45 ± 41% perf-profile.calltrace.cycles-pp.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write 0.14 ±173% +1.3 1.39 ± 45% perf-profile.calltrace.cycles-pp._raw_spin_lock.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter 0.65 ± 64% +1.6 2.27 ± 38% perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write 0.21 ±173% +1.7 1.88 ± 39% perf-profile.calltrace.cycles-pp.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write 0.00 +0.1 0.06 ± 11% perf-profile.children.cycles-pp.tick_nohz_idle_got_tick 0.13 ± 62% +0.3 0.39 ± 37% perf-profile.children.cycles-pp.can_overcommit 0.30 ± 97% +1.1 1.43 ± 42% perf-profile.children.cycles-pp.__btrfs_block_rsv_release 0.31 ± 96% +1.1 1.45 ± 41% perf-profile.children.cycles-pp.btrfs_inode_rsv_release 0.41 ± 68% +1.5 1.88 ± 39% perf-profile.children.cycles-pp.btrfs_reserve_metadata_bytes 0.67 ± 57% +1.6 2.27 ± 38% perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata 0.63 ± 71% +2.3 2.96 ± 32% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 1.41 ± 57% +2.7 4.14 ± 31% perf-profile.children.cycles-pp._raw_spin_lock 0.04 ± 58% +0.1 0.10 ± 49% perf-profile.self.cycles-pp.btrfs_reserve_metadata_bytes 0.03 ±100% +0.1 0.10 ± 27% perf-profile.self.cycles-pp.can_overcommit 0.63 ± 72% +2.3 2.95 ± 32% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 4.329e+08 ± 47% -33.5% 2.88e+08 ± 4% perf-stat.i.branch-instructions 3936 -14.8% 3352 ± 2% perf-stat.i.context-switches 6.07 ± 15% +38.8% 8.43 ± 16% perf-stat.i.cpu-migrations 5.515e+08 ± 37% -30.1% 3.856e+08 perf-stat.i.dTLB-loads 2.271e+08 ± 2% -10.8% 2.026e+08 perf-stat.i.dTLB-stores 1.978e+09 ± 37% -27.9% 1.426e+09 ± 3% perf-stat.i.instructions 539679 ± 3% -10.8% 481609 ± 3% perf-stat.i.node-load-misses 279956 ± 3% -14.1% 240383 ± 3% perf-stat.i.node-loads 94478 ± 23% -37.7% 58900 ± 7% perf-stat.i.node-stores 67.13 ± 14% +11.7 78.86 perf-stat.overall.node-store-miss-rate% 4.321e+08 ± 47% -33.5% 2.874e+08 ± 4% perf-stat.ps.branch-instructions 3928 -14.8% 3346 ± 2% perf-stat.ps.context-switches 6.07 ± 15% +38.7% 8.41 ± 16% perf-stat.ps.cpu-migrations 5.505e+08 ± 38% -30.1% 3.847e+08 perf-stat.ps.dTLB-loads 2.267e+08 ± 2% -10.9% 2.021e+08 perf-stat.ps.dTLB-stores 1.974e+09 ± 37% -28.0% 1.423e+09 ± 3% perf-stat.ps.instructions 538662 ± 3% -10.8% 480433 ± 3% perf-stat.ps.node-load-misses 279289 ± 3% -14.1% 239791 ± 3% perf-stat.ps.node-loads 94348 ± 23% -37.7% 58757 ± 7% perf-stat.ps.node-stores 13621 -12.0% 11980 proc-vmstat.nr_active_file 21578525 -10.9% 19231325 proc-vmstat.nr_dirtied 486311 -21.3% 382498 proc-vmstat.nr_dirty 17619220 -7.7% 16254004 proc-vmstat.nr_file_pages 31271641 +4.4% 32640654 proc-vmstat.nr_free_pages 66113 -7.0% 61506 proc-vmstat.nr_inactive_anon 17266628 -7.9% 15907732 proc-vmstat.nr_inactive_file 67965 -6.8% 63323 proc-vmstat.nr_mapped 2190 -4.6% 2089 proc-vmstat.nr_page_table_pages 69007 ± 2% -8.0% 63480 proc-vmstat.nr_shmem 56508 -5.5% 53422 proc-vmstat.nr_slab_reclaimable 6141712 -1.3% 6064303 proc-vmstat.nr_writeback 21578549 -10.9% 19231357 proc-vmstat.nr_written 13621 -12.0% 11980 proc-vmstat.nr_zone_active_file 66113 -7.0% 61506 proc-vmstat.nr_zone_inactive_anon 17266628 -7.9% 15907732 proc-vmstat.nr_zone_inactive_file 6628866 -2.7% 6447374 proc-vmstat.nr_zone_write_pending 22781579 -10.1% 20474832 proc-vmstat.numa_hit 22758080 -10.1% 20451301 proc-vmstat.numa_local 11840 ± 22% -17.0% 9824 ± 3% proc-vmstat.pgactivate 22943031 -9.9% 20668524 proc-vmstat.pgalloc_normal 1392625 +7.0% 1490792 proc-vmstat.pgfault 22469883 -8.1% 20650344 proc-vmstat.pgfree 86439201 -10.9% 77036052 proc-vmstat.pgpgout 73214 -9.9% 65995 interrupts.272:PCI-MSI.376832-edge.ahci[0000:00:17.0] 0.00 +1.3e+104% 130.75 ±153% interrupts.41:PCI-MSI.31981574-edge.i40e-eth0-TxRx-5 0.75 ±173% +5466.7% 41.75 ±149% interrupts.49:PCI-MSI.31981582-edge.i40e-eth0-TxRx-13 0.25 ±173% +1.5e+05% 369.00 ±170% interrupts.69:PCI-MSI.31981602-edge.i40e-eth0-TxRx-33 2.00 ±173% +7862.5% 159.25 ±135% interrupts.94:PCI-MSI.31981627-edge.i40e-eth0-TxRx-58 4558 ± 8% +14.1% 5199 interrupts.CPU0.CAL:Function_call_interrupts 4799 ± 2% +10.0% 5278 interrupts.CPU1.CAL:Function_call_interrupts 1172 ±131% -84.1% 186.25 ± 9% interrupts.CPU1.NMI:Non-maskable_interrupts 1172 ±131% -84.1% 186.25 ± 9% interrupts.CPU1.PMI:Performance_monitoring_interrupts 4787 ± 2% +10.2% 5273 interrupts.CPU10.CAL:Function_call_interrupts 51.00 ± 29% +255.4% 181.25 ± 91% interrupts.CPU10.RES:Rescheduling_interrupts 4790 ± 2% +10.0% 5270 interrupts.CPU11.CAL:Function_call_interrupts 4781 ± 2% +10.0% 5261 interrupts.CPU12.CAL:Function_call_interrupts 0.75 ±173% +5400.0% 41.25 ±152% interrupts.CPU13.49:PCI-MSI.31981582-edge.i40e-eth0-TxRx-13 4799 ± 2% +9.7% 5267 interrupts.CPU13.CAL:Function_call_interrupts 4776 ± 2% +9.1% 5209 ± 3% interrupts.CPU14.CAL:Function_call_interrupts 4782 ± 2% +10.1% 5263 interrupts.CPU15.CAL:Function_call_interrupts 4770 ± 2% +10.2% 5255 interrupts.CPU16.CAL:Function_call_interrupts 4803 ± 2% +15.3% 5539 ± 4% interrupts.CPU18.CAL:Function_call_interrupts 4795 ± 2% +10.1% 5277 interrupts.CPU2.CAL:Function_call_interrupts 849.75 ±130% -82.1% 152.50 ± 26% interrupts.CPU23.NMI:Non-maskable_interrupts 849.75 ±130% -82.1% 152.50 ± 26% interrupts.CPU23.PMI:Performance_monitoring_interrupts 182.50 ± 82% -74.1% 47.25 ± 31% interrupts.CPU25.RES:Rescheduling_interrupts 871.75 ±123% -79.1% 182.00 ± 6% interrupts.CPU26.NMI:Non-maskable_interrupts 871.75 ±123% -79.1% 182.00 ± 6% interrupts.CPU26.PMI:Performance_monitoring_interrupts 164.75 ±112% -99.5% 0.75 ±110% interrupts.CPU29.65:PCI-MSI.31981598-edge.i40e-eth0-TxRx-29 4789 ± 2% +10.2% 5278 interrupts.CPU3.CAL:Function_call_interrupts 94.25 ± 57% -58.6% 39.00 ± 35% interrupts.CPU30.RES:Rescheduling_interrupts 4304 ± 20% +26.5% 5445 ± 3% interrupts.CPU34.CAL:Function_call_interrupts 32.00 ± 34% +1215.6% 421.00 ±128% interrupts.CPU36.RES:Rescheduling_interrupts 3705 ± 48% +40.4% 5201 interrupts.CPU39.CAL:Function_call_interrupts 4791 ± 2% +10.1% 5277 interrupts.CPU4.CAL:Function_call_interrupts 771.75 ±106% -75.5% 188.75 ± 19% interrupts.CPU40.NMI:Non-maskable_interrupts 771.75 ±106% -75.5% 188.75 ± 19% interrupts.CPU40.PMI:Performance_monitoring_interrupts 21.25 ± 53% +251.8% 74.75 ± 68% interrupts.CPU44.RES:Rescheduling_interrupts 1170 ±132% -85.0% 175.75 ± 36% interrupts.CPU49.NMI:Non-maskable_interrupts 1170 ±132% -85.0% 175.75 ± 36% interrupts.CPU49.PMI:Performance_monitoring_interrupts 152.25 ±126% +235.6% 511.00 ± 94% interrupts.CPU5.RES:Rescheduling_interrupts 27.25 ± 64% +200.0% 81.75 ± 32% interrupts.CPU53.RES:Rescheduling_interrupts 211.50 ± 16% +29.8% 274.50 ± 23% interrupts.CPU57.NMI:Non-maskable_interrupts 211.50 ± 16% +29.8% 274.50 ± 23% interrupts.CPU57.PMI:Performance_monitoring_interrupts 1.75 ±173% +8957.1% 158.50 ±135% interrupts.CPU58.94:PCI-MSI.31981627-edge.i40e-eth0-TxRx-58 192.25 ± 88% -81.5% 35.50 ± 48% interrupts.CPU59.RES:Rescheduling_interrupts 4796 ± 2% +10.4% 5296 interrupts.CPU6.CAL:Function_call_interrupts 226.00 ±129% -87.6% 28.00 ± 91% interrupts.CPU65.RES:Rescheduling_interrupts 136.25 ± 73% -62.0% 51.75 ± 51% interrupts.CPU66.RES:Rescheduling_interrupts 4799 ± 2% +10.0% 5278 interrupts.CPU7.CAL:Function_call_interrupts 4794 ± 2% +11.9% 5363 ± 4% interrupts.CPU70.CAL:Function_call_interrupts 150.00 ± 48% -72.7% 41.00 ± 58% interrupts.CPU71.RES:Rescheduling_interrupts 4799 ± 2% +9.4% 5248 ± 2% interrupts.CPU8.CAL:Function_call_interrupts 4795 ± 2% +9.9% 5269 ± 2% interrupts.CPU9.CAL:Function_call_interrupts Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen