Greeting, FYI, we noticed a -2.6% regression of will-it-scale.per_process_ops due to commit: commit: 77e89afc25f30abd56e76a809ee2884d7c1b63ce ("PCI/MSI: Protect msi_desc::masked for multi-MSI") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: will-it-scale on test machine: 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz with 256G memory with following parameters: nr_task: 100% mode: process test: lseek2 cpufreq_governor: performance ucode: 0xd000280 test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two. test-url: https://github.com/antonblanchard/will-it-scale If you fix the issue, kindly add following tag Reported-by: kernel test robot Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run bin/lkp run generated-yaml-file ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/x86_64-rhel-8.3/process/100%/debian-10.4-x86_64-20200603.cgz/lkp-icl-2sp2/lseek2/will-it-scale/0xd000280 commit: d28d4ad2a1 ("PCI/MSI: Use msi_mask_irq() in pci_msi_shutdown()") 77e89afc25 ("PCI/MSI: Protect msi_desc::masked for multi-MSI") d28d4ad2a1aef274 77e89afc25f30abd56e76a809ee ---------------- --------------------------- %stddev %change %stddev \ | \ 1.073e+09 -2.6% 1.045e+09 will-it-scale.128.processes 8381828 -2.6% 8165037 will-it-scale.per_process_ops 1.073e+09 -2.6% 1.045e+09 will-it-scale.workload 698.88 ±105% -98.8% 8.15 ± 49% perf-sched.wait_time.avg.ms.preempt_schedule_common.__cond_resched.generic_perform_write.__generic_file_write_iter.generic_file_write_iter 1706 ± 34% -42.2% 986.00 ± 24% interrupts.CPU1.CAL:Function_call_interrupts 776.00 ± 3% +41.7% 1099 ± 32% interrupts.CPU122.CAL:Function_call_interrupts 343.00 ± 2% +87.4% 642.83 ± 37% interrupts.CPU122.RES:Rescheduling_interrupts 1673 ± 70% -51.8% 807.00 ± 6% interrupts.CPU19.CAL:Function_call_interrupts 514.67 ± 15% -18.1% 421.33 ± 12% interrupts.CPU75.RES:Rescheduling_interrupts 392.17 ± 4% +122.0% 870.50 ±112% interrupts.CPU79.RES:Rescheduling_interrupts 449.83 -11.2% 399.50 ± 8% interrupts.TLB:TLB_shootdowns 6.742e+10 -2.6% 6.568e+10 perf-stat.i.branch-instructions 0.04 ± 6% -0.0 0.03 ± 6% perf-stat.i.branch-miss-rate% 23853919 ± 6% -19.3% 19238556 ± 6% perf-stat.i.branch-misses 0.96 +2.6% 0.99 perf-stat.i.cpi 1.028e+11 -2.6% 1.001e+11 perf-stat.i.dTLB-loads 0.00 -0.0 0.00 perf-stat.i.dTLB-store-miss-rate% 475774 -9.0% 433089 perf-stat.i.dTLB-store-misses 6.959e+10 -2.6% 6.779e+10 perf-stat.i.dTLB-stores 3.414e+11 -2.6% 3.326e+11 perf-stat.i.instructions 1.04 -2.6% 1.01 perf-stat.i.ipc 1873 -2.6% 1824 perf-stat.i.metric.M/sec 0.02 +2.9% 0.02 perf-stat.overall.MPKI 0.04 ± 6% -0.0 0.03 ± 6% perf-stat.overall.branch-miss-rate% 0.96 +2.7% 0.99 perf-stat.overall.cpi 0.00 -0.0 0.00 perf-stat.overall.dTLB-store-miss-rate% 1.04 -2.6% 1.01 perf-stat.overall.ipc 6.719e+10 -2.6% 6.546e+10 perf-stat.ps.branch-instructions 23787003 ± 6% -19.4% 19181009 ± 6% perf-stat.ps.branch-misses 1.024e+11 -2.6% 9.977e+10 perf-stat.ps.dTLB-loads 474525 -9.0% 431924 perf-stat.ps.dTLB-store-misses 6.935e+10 -2.6% 6.757e+10 perf-stat.ps.dTLB-stores 3.402e+11 -2.6% 3.315e+11 perf-stat.ps.instructions 1.029e+14 -2.7% 1.001e+14 perf-stat.total.instructions 39.43 -0.9 38.58 perf-profile.calltrace.cycles-pp.__entry_text_start.llseek 4.11 ± 2% -0.8 3.29 ± 9% perf-profile.calltrace.cycles-pp.testcase 2.42 -0.4 2.04 ± 2% perf-profile.calltrace.cycles-pp.lseek@plt 6.07 -0.4 5.69 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.llseek 2.66 -0.1 2.58 perf-profile.calltrace.cycles-pp.__x64_sys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe.llseek 1.52 -0.1 1.46 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.llseek 10.12 +0.2 10.33 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.llseek 3.76 +0.3 4.10 ± 2% perf-profile.calltrace.cycles-pp.generic_file_llseek_size.ksys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe.llseek 4.56 +0.3 4.90 ± 4% perf-profile.calltrace.cycles-pp.shmem_file_llseek.ksys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe.llseek 4.55 +0.4 4.98 ± 4% perf-profile.calltrace.cycles-pp.syscall_enter_from_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.llseek 95.78 +0.5 96.27 perf-profile.calltrace.cycles-pp.llseek 1.64 ± 11% +0.5 2.14 ± 2% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.llseek 5.98 +0.7 6.71 ± 3% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe 7.87 ± 3% +0.9 8.75 ± 2% perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe.llseek 20.31 +2.2 22.53 perf-profile.calltrace.cycles-pp.ksys_lseek.do_syscall_64.entry_SYSCALL_64_after_hwframe.llseek 46.00 +2.6 48.63 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.llseek 38.81 +3.1 41.92 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.llseek 23.00 -1.0 22.02 perf-profile.children.cycles-pp.syscall_return_via_sysret 4.63 -0.8 3.86 ± 4% perf-profile.children.cycles-pp.testcase 22.32 -0.4 21.94 perf-profile.children.cycles-pp.__entry_text_start 2.00 -0.3 1.66 ± 3% perf-profile.children.cycles-pp.lseek@plt 1.64 -0.1 1.56 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack 2.12 -0.1 2.05 perf-profile.children.cycles-pp.__x64_sys_lseek 3.72 +0.4 4.08 ± 2% perf-profile.children.cycles-pp.generic_file_llseek_size 4.50 +0.4 4.89 ± 4% perf-profile.children.cycles-pp.shmem_file_llseek 95.89 +0.6 96.52 perf-profile.children.cycles-pp.llseek 5.95 +0.7 6.69 ± 3% perf-profile.children.cycles-pp.__fget_light 4.57 +0.8 5.39 perf-profile.children.cycles-pp.syscall_enter_from_user_mode 8.12 +0.9 8.97 ± 2% perf-profile.children.cycles-pp.__fdget_pos 20.98 +1.9 22.85 perf-profile.children.cycles-pp.ksys_lseek 46.20 +2.5 48.73 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 40.21 +2.7 42.88 perf-profile.children.cycles-pp.do_syscall_64 23.00 -1.0 22.02 perf-profile.self.cycles-pp.syscall_return_via_sysret 19.08 -0.8 18.31 perf-profile.self.cycles-pp.llseek 3.93 ± 2% -0.6 3.30 ± 4% perf-profile.self.cycles-pp.testcase 6.54 -0.2 6.35 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.78 ± 3% -0.2 0.63 ± 3% perf-profile.self.cycles-pp.lseek@plt 1.61 -0.1 1.52 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack 1.06 -0.1 1.00 perf-profile.self.cycles-pp.__x64_sys_lseek 3.11 +0.1 3.17 perf-profile.self.cycles-pp.syscall_exit_to_user_mode 2.17 +0.2 2.34 perf-profile.self.cycles-pp.__fdget_pos 3.16 +0.3 3.48 ± 2% perf-profile.self.cycles-pp.generic_file_llseek_size 5.74 +0.3 6.07 perf-profile.self.cycles-pp.ksys_lseek 3.88 +0.4 4.26 ± 5% perf-profile.self.cycles-pp.shmem_file_llseek 5.39 +0.7 6.08 ± 3% perf-profile.self.cycles-pp.__fget_light 3.84 +0.9 4.70 perf-profile.self.cycles-pp.syscall_enter_from_user_mode will-it-scale.128.processes 1.09e+09 +---------------------------------------------------------------+ 1.085e+09 |-+.+. .+ + +.+. +. .+. .+. .+. .+ | |.+ + : :+ + + + : +.+.+ + +.+ + : | 1.08e+09 |-+ : : + + + + : : +.| 1.075e+09 |-+ :: + +.+ :.+ : | | + + + : | 1.07e+09 |-+ +.+ | 1.065e+09 |-+ | 1.06e+09 |-+ | | | 1.055e+09 |-+ | 1.05e+09 |-+ O O O O O O O O O O | | O O O O O O O O O O O O | 1.045e+09 |-+ O O | 1.04e+09 +---------------------------------------------------------------+ will-it-scale.per_process_ops 8.5e+06 +----------------------------------------------------------------+ |.+ + : + : +.+ + +..+.+.+ + +.+ + : | 8.45e+06 |-+ : : + : + + + + : | | :: + + +.+ : +.| 8.4e+06 |-+ + +.+. + | 8.35e+06 |-+ +.+ | | | 8.3e+06 |-+ | | | 8.25e+06 |-+ | 8.2e+06 |-+ O O O O O O O O | | O O O O O O O O O O O O | 8.15e+06 |-+ O O O O | | O | 8.1e+06 +----------------------------------------------------------------+ will-it-scale.workload 1.09e+09 +---------------------------------------------------------------+ 1.085e+09 |-+.+. .+ + +.+. +. .+. .+. .+. .+ | |.+ + : :+ + + + : +.+.+ + +.+ + : | 1.08e+09 |-+ : : + + + + : : +.| 1.075e+09 |-+ :: + +.+ :.+ : | | + + + : | 1.07e+09 |-+ +.+ | 1.065e+09 |-+ | 1.06e+09 |-+ | | | 1.055e+09 |-+ | 1.05e+09 |-+ O O O O O O O O O O | | O O O O O O O O O O O O | 1.045e+09 |-+ O O | 1.04e+09 +---------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. --- 0DAY/LKP+ Test Infrastructure Open Source Technology Center https://lists.01.org/hyperkitty/list/lkp@lists.01.org Intel Corporation Thanks, Oliver Sang