Greeting, FYI, we noticed a -11.8% regression of aim7.jobs-per-min due to commit: commit: 05c2c00f3769abb9e323fcaca70d2de0b48af7ba ("ext4: protect superblock modifications with a buffer lock") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: aim7 on test machine: 88 threads Intel(R) Xeon(R) Gold 6238M CPU @ 2.10GHz with 128G memory with following parameters: disk: 4BRD_12G md: RAID1 fs: ext4 test: creat-clo load: 1000 cpufreq_governor: performance ucode: 0x5003006 test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system. test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/ In addition to that, the commit also has significant impact on the following tests: +------------------+---------------------------------------------------------------------------+ | testcase: change | aim9: aim9.udp_test.ops_per_sec 2.1% improvement | | test machine | 256 threads Intel(R) Genuine Intel(R) CPU 0000 @ 1.30GHz with 112G memory | | test parameters | cpufreq_governor=performance | | | test=all | | | testtime=5s | | | ucode=0xffff0190 | +------------------+---------------------------------------------------------------------------+ If you fix the issue, kindly add following tag Reported-by: kernel test robot Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp split-job --compatible job.yaml bin/lkp run compatible-job.yaml ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase/ucode: gcc-9/performance/4BRD_12G/ext4/x86_64-rhel-8.3/1000/RAID1/debian-10.4-x86_64-20200603.cgz/lkp-csl-2sp9/creat-clo/aim7/0x5003006 commit: 4392fbc4ba ("ext4: drop sync argument of ext4_commit_super()") 05c2c00f37 ("ext4: protect superblock modifications with a buffer lock") 4392fbc4bab57db3 05c2c00f3769abb9e323fcaca70 ---------------- --------------------------- %stddev %change %stddev \ | \ 13342 -11.8% 11771 ± 2% aim7.jobs-per-min 449.81 +13.4% 510.10 ± 2% aim7.time.elapsed_time 449.81 +13.4% 510.10 ± 2% aim7.time.elapsed_time.max 1246834 ± 2% +13.8% 1418574 ± 3% aim7.time.involuntary_context_switches 204562 ± 4% +12.2% 229478 ± 3% aim7.time.minor_page_faults 35777 +12.9% 40390 ± 2% aim7.time.system_time 6270530 ± 3% +34.5% 8435526 ± 2% aim7.time.voluntary_context_switches 8686688 ± 12% +24.1% 10784294 ± 4% cpuidle.C1E.usage 8.74 +4.5% 9.14 iostat.cpu.idle 0.02 ± 12% -0.0 0.02 ± 9% mpstat.cpu.all.iowait% 6988 ± 11% +43.1% 9999 ± 9% numa-meminfo.node0.Active(file) 1440589 ± 7% +7.8% 1552890 ± 6% numa-numastat.node1.numa_hit 1747 ± 11% +43.0% 2499 ± 9% numa-vmstat.node0.nr_active_file 1747 ± 11% +43.0% 2499 ± 9% numa-vmstat.node0.nr_zone_active_file 481.09 +12.6% 541.71 ± 2% uptime.boot 5541 +10.8% 6140 uptime.idle 42743 -10.6% 38207 slabinfo.filp.active_objs 1430 -10.0% 1287 slabinfo.filp.active_slabs 45785 -10.0% 41214 slabinfo.filp.num_objs 1430 -10.0% 1287 slabinfo.filp.num_slabs 91.00 -1.1% 90.00 vmstat.cpu.sy 2850 ± 2% +14.5% 3263 vmstat.io.bo 13739 ± 3% +29.2% 17754 ± 2% vmstat.memory.buff 25822 ± 2% +14.8% 29645 vmstat.system.cs 452981 +11.7% 505788 ± 3% meminfo.Active 439264 +11.1% 488079 ± 3% meminfo.Active(anon) 13715 ± 3% +29.1% 17708 ± 2% meminfo.Active(file) 13713 ± 3% +29.2% 17723 ± 2% meminfo.Buffers 459294 +10.6% 507955 ± 3% meminfo.Shmem 109825 +11.1% 121996 ± 3% proc-vmstat.nr_active_anon 3428 ± 3% +29.1% 4426 ± 2% proc-vmstat.nr_active_file 1427 +13.3% 1618 ± 2% proc-vmstat.nr_dirtied 376669 +3.5% 389792 proc-vmstat.nr_file_pages 114820 +10.6% 126987 ± 3% proc-vmstat.nr_shmem 1372 +15.1% 1579 ± 4% proc-vmstat.nr_written 109825 +11.1% 121996 ± 3% proc-vmstat.nr_zone_active_anon 3428 ± 3% +29.1% 4426 ± 2% proc-vmstat.nr_zone_active_file 151153 ± 5% +16.9% 176679 ± 3% proc-vmstat.numa_hint_faults 106816 ± 7% +17.4% 125437 ± 4% proc-vmstat.numa_hint_faults_local 2915186 +5.4% 3071445 proc-vmstat.numa_hit 2835807 +5.4% 2988976 proc-vmstat.numa_local 129566 ± 2% +7.4% 139212 ± 4% proc-vmstat.pgactivate 4233617 +2.8% 4353951 proc-vmstat.pgalloc_normal 1622353 +12.0% 1816326 proc-vmstat.pgfault 3863258 +2.8% 3970690 proc-vmstat.pgfree 1290723 ± 3% +29.8% 1675288 ± 2% proc-vmstat.pgpgout 86298 +10.3% 95227 ± 3% proc-vmstat.pgreuse 189.67 ± 5% -9.6% 171.45 ± 5% sched_debug.cfs_rq:/.load_avg.stddev 15090675 +13.9% 17181333 sched_debug.cfs_rq:/.min_vruntime.avg 15410230 +14.0% 17563602 sched_debug.cfs_rq:/.min_vruntime.max 14179241 +13.8% 16142220 sched_debug.cfs_rq:/.min_vruntime.min 909405 ± 5% +13.7% 1034081 ± 6% sched_debug.cfs_rq:/.spread0.avg 1229377 ± 4% +15.2% 1416676 ± 3% sched_debug.cfs_rq:/.spread0.max 240213 +12.5% 270308 sched_debug.cpu.clock.avg 240225 +12.5% 270320 sched_debug.cpu.clock.max 240200 +12.5% 270295 sched_debug.cpu.clock.min 237580 +12.5% 267317 sched_debug.cpu.clock_task.avg 237797 +12.5% 267523 sched_debug.cpu.clock_task.max 232767 +12.8% 262508 sched_debug.cpu.clock_task.min 64022 ± 2% +31.4% 84141 sched_debug.cpu.nr_switches.avg 77073 +28.8% 99292 ± 3% sched_debug.cpu.nr_switches.max 60980 ± 2% +32.5% 80775 sched_debug.cpu.nr_switches.min 1399 ± 8% +23.7% 1731 ± 4% sched_debug.cpu.nr_uninterruptible.max 248.99 ± 6% +21.4% 302.15 ± 9% sched_debug.cpu.nr_uninterruptible.stddev 240200 +12.5% 270295 sched_debug.cpu_clk 239703 +12.6% 269797 sched_debug.ktime 240543 +12.5% 270629 sched_debug.sched_clk 1.221e+10 -1.1% 1.208e+10 perf-stat.i.branch-instructions 0.22 ± 23% -0.0 0.17 perf-stat.i.branch-miss-rate% 14793770 -6.0% 13912167 perf-stat.i.branch-misses 25953 ± 2% +14.8% 29803 perf-stat.i.context-switches 4869 +13.6% 5530 perf-stat.i.cpu-migrations 1.64e+10 -1.3% 1.62e+10 perf-stat.i.dTLB-loads 1.075e+09 -9.9% 9.682e+08 ± 2% perf-stat.i.dTLB-stores 430893 ± 2% +11.6% 480826 perf-stat.i.iTLB-loads 6.072e+10 -1.1% 6.005e+10 perf-stat.i.instructions 338.69 -1.5% 333.53 perf-stat.i.metric.M/sec 6039682 -7.8% 5565765 ± 2% perf-stat.i.node-store-misses 96902 ± 2% -4.0% 93045 ± 2% perf-stat.i.node-stores 0.12 -0.0 0.12 perf-stat.overall.branch-miss-rate% 1.219e+10 -1.0% 1.206e+10 perf-stat.ps.branch-instructions 14740466 -5.9% 13872099 perf-stat.ps.branch-misses 25873 ± 2% +14.9% 29722 perf-stat.ps.context-switches 4856 +13.6% 5517 perf-stat.ps.cpu-migrations 1.637e+10 -1.2% 1.617e+10 perf-stat.ps.dTLB-loads 1.072e+09 -9.9% 9.665e+08 ± 2% perf-stat.ps.dTLB-stores 428747 ± 2% +11.6% 478487 perf-stat.ps.iTLB-loads 6.061e+10 -1.1% 5.996e+10 perf-stat.ps.instructions 6028729 -7.8% 5557348 ± 2% perf-stat.ps.node-store-misses 96841 -4.1% 92846 ± 3% perf-stat.ps.node-stores 2.734e+13 +12.2% 3.066e+13 ± 2% perf-stat.total.instructions 1.54 ± 72% -86.8% 0.20 ±221% perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 0.17 ± 19% -47.8% 0.09 ± 22% perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.__ext4_journal_get_write_access.ext4_orphan_add.ext4_truncate 1.57 ± 11% +18.6% 1.86 ± 5% perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.__getblk_gfp.__ext4_get_inode_loc.ext4_get_inode_loc 1.45 ± 37% +126.7% 3.29 ± 34% perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.jbd2__journal_start.__ext4_journal_start_sb 0.59 ± 34% +58.4% 0.93 ± 24% perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate 3.16 ± 68% -80.8% 0.61 ±223% perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 8.52 ± 28% +80.5% 15.38 ± 25% perf-sched.sch_delay.max.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.__alloc_file.alloc_empty_file 162500 ± 9% +25.1% 203298 ± 10% perf-sched.total_wait_and_delay.count.ms 116.08 ± 9% -21.7% 90.91 ± 13% perf-sched.wait_and_delay.avg.ms.preempt_schedule_common._cond_resched.__ext4_handle_dirty_metadata.ext4_do_update_inode.ext4_mark_iloc_dirty 134.71 ± 9% -24.4% 101.80 ± 7% perf-sched.wait_and_delay.avg.ms.preempt_schedule_common._cond_resched.__getblk_gfp.__ext4_get_inode_loc.ext4_get_inode_loc 145.16 ± 21% -42.0% 84.13 ± 46% perf-sched.wait_and_delay.avg.ms.preempt_schedule_common._cond_resched.__mutex_lock.isra.0 153.37 ± 30% -68.2% 48.78 ±103% perf-sched.wait_and_delay.avg.ms.preempt_schedule_common._cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_truncate 10.19 ± 5% -84.0% 1.62 ±223% perf-sched.wait_and_delay.avg.ms.preempt_schedule_common._cond_resched.stop_one_cpu.__set_cpus_allowed_ptr.sched_setaffinity 76.08 ± 11% -23.6% 58.16 ± 12% perf-sched.wait_and_delay.avg.ms.wait_transaction_locked.add_transaction_credits.start_this_handle.jbd2__journal_start 69.33 ± 14% +36.5% 94.67 ± 13% perf-sched.wait_and_delay.count.jbd2_journal_commit_transaction.kjournald2.kthread.ret_from_fork 890.17 ± 5% -30.8% 615.83 ± 4% perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.__ext4_handle_dirty_metadata.ext4_do_update_inode.ext4_mark_iloc_dirty 3176 ± 15% +43.3% 4551 ± 17% perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.__ext4_mark_inode_dirty.ext4_dirty_inode.__mark_inode_dirty 349.17 ± 5% -17.3% 288.83 ± 5% perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.__getblk_gfp.__ext4_get_inode_loc.ext4_get_inode_loc 654.17 ± 73% +108.3% 1362 ± 18% perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.kmem_cache_alloc.start_this_handle.jbd2__journal_start 953.33 ± 3% -84.6% 146.67 ±223% perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.stop_one_cpu.__set_cpus_allowed_ptr.sched_setaffinity 35615 ± 10% +26.7% 45108 ± 10% perf-sched.wait_and_delay.count.schedule_preempt_disabled.__mutex_lock.isra.0.ext4_orphan_add 54444 ± 15% +44.7% 78784 ± 17% perf-sched.wait_and_delay.count.wait_transaction_locked.add_transaction_credits.start_this_handle.jbd2__journal_start 749.81 ± 5% -18.6% 610.38 ± 3% perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.__ext4_handle_dirty_metadata.ext4_do_update_inode.ext4_mark_iloc_dirty 751.99 ± 7% -21.6% 589.37 ± 5% perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.__getblk_gfp.__ext4_get_inode_loc.ext4_get_inode_loc 545.93 ± 17% -31.5% 374.21 ± 24% perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.__kmalloc.ext4_ext_remove_space.ext4_ext_truncate 619.18 ± 17% -45.6% 337.10 ± 49% perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.__mutex_lock.isra.0 573.86 ± 20% -64.0% 206.68 ±104% perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_truncate 1067 ± 12% -84.4% 166.83 ±223% perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.stop_one_cpu.__set_cpus_allowed_ptr.sched_setaffinity 650.19 ± 8% -13.1% 564.84 ± 3% perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate 676.68 ± 9% -17.8% 556.38 ± 3% perf-sched.wait_and_delay.max.ms.preempt_schedule_common._cond_resched.wait_for_completion.stop_two_cpus.migrate_swap 745.90 ± 4% -18.2% 609.91 ± 2% perf-sched.wait_and_delay.max.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2 775.20 ± 6% -18.9% 628.79 perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.__mutex_lock.isra.0.ext4_orphan_add 770.37 ± 6% -18.9% 624.81 perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.__mutex_lock.isra.0.ext4_orphan_del 114.49 ± 9% -22.1% 89.15 ± 13% perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.__ext4_handle_dirty_metadata.ext4_do_update_inode.ext4_mark_iloc_dirty 133.14 ± 9% -24.9% 99.94 ± 7% perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.__getblk_gfp.__ext4_get_inode_loc.ext4_get_inode_loc 142.91 ± 22% -37.9% 88.74 ± 26% perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.__mutex_lock.isra.0 120.05 ± 39% -61.7% 46.00 ± 96% perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.down_write.ext4_truncate.ext4_setattr 151.19 ± 30% -57.1% 64.88 ± 53% perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_truncate 4.55 ± 4% +12.8% 5.13 ± 3% perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.jbd2_journal_commit_transaction.kjournald2.kthread 74.36 ± 11% -24.3% 56.28 ± 13% perf-sched.wait_time.avg.ms.wait_transaction_locked.add_transaction_credits.start_this_handle.jbd2__journal_start 749.64 ± 5% -18.6% 610.37 ± 3% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.__ext4_handle_dirty_metadata.ext4_do_update_inode.ext4_mark_iloc_dirty 751.98 ± 7% -21.6% 589.37 ± 5% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.__getblk_gfp.__ext4_get_inode_loc.ext4_get_inode_loc 545.92 ± 17% -32.0% 371.40 ± 24% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.__kmalloc.ext4_ext_remove_space.ext4_ext_truncate 619.17 ± 17% -38.7% 379.32 ± 23% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.__mutex_lock.isra.0 571.69 ± 20% -52.1% 273.60 ± 56% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.ext4_journal_check_start.__ext4_journal_start_sb.ext4_truncate 650.17 ± 8% -13.1% 564.83 ± 3% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.stop_one_cpu.migrate_task_to.task_numa_migrate 676.67 ± 9% -17.8% 556.37 ± 3% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.wait_for_completion.stop_two_cpus.migrate_swap 745.40 ± 4% -18.3% 609.29 ± 2% perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.path_openat.do_filp_open.do_sys_openat2 769.37 ± 6% -19.0% 622.87 ± 2% perf-sched.wait_time.max.ms.schedule_preempt_disabled.__mutex_lock.isra.0.ext4_orphan_add 769.35 ± 6% -19.1% 622.61 ± 2% perf-sched.wait_time.max.ms.schedule_preempt_disabled.__mutex_lock.isra.0.ext4_orphan_del 0.59 ± 4% -0.2 0.35 ± 70% perf-profile.calltrace.cycles-pp.ext4_ext_truncate.ext4_truncate.ext4_setattr.notify_change.do_truncate 0.81 ± 3% -0.1 0.70 ± 4% perf-profile.children.cycles-pp.ext4_reserve_inode_write 0.89 ± 3% -0.1 0.78 ± 3% perf-profile.children.cycles-pp.__ext4_mark_inode_dirty 1.00 ± 2% -0.1 0.89 ± 3% perf-profile.children.cycles-pp.ext4_mark_iloc_dirty 0.59 ± 4% -0.1 0.52 ± 4% perf-profile.children.cycles-pp.ext4_ext_truncate 0.65 ± 2% -0.1 0.59 ± 4% perf-profile.children.cycles-pp.ext4_do_update_inode 0.32 ± 2% -0.0 0.28 ± 2% perf-profile.children.cycles-pp.__mutex_unlock_slowpath 0.35 ± 4% -0.0 0.30 ± 4% perf-profile.children.cycles-pp.ext4_ext_remove_space 0.21 ± 3% -0.0 0.17 ± 4% perf-profile.children.cycles-pp.__getblk_gfp 0.15 ± 13% -0.0 0.11 ± 15% perf-profile.children.cycles-pp.run_ksoftirqd 0.18 ± 3% -0.0 0.14 ± 5% perf-profile.children.cycles-pp.crc32c_pcl_intel_update 0.15 ± 11% -0.0 0.12 ± 11% perf-profile.children.cycles-pp.smpboot_thread_fn 0.18 ± 2% -0.0 0.15 ± 4% perf-profile.children.cycles-pp.__find_get_block 0.12 ± 4% -0.0 0.09 ± 5% perf-profile.children.cycles-pp.mutex_unlock 0.15 ± 3% -0.0 0.12 ± 5% perf-profile.children.cycles-pp.ext4_inode_csum_set 0.24 ± 3% -0.0 0.21 ± 3% perf-profile.children.cycles-pp.__close 0.10 ± 9% -0.0 0.08 ± 12% perf-profile.children.cycles-pp.osq_unlock 0.20 ± 3% -0.0 0.18 ± 3% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 0.14 ± 3% -0.0 0.12 ± 3% perf-profile.children.cycles-pp.wake_up_q 0.10 ± 4% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.cap_inode_need_killpriv 0.12 ± 4% -0.0 0.11 ± 6% perf-profile.children.cycles-pp.mutex_lock 0.08 -0.0 0.06 ± 7% perf-profile.children.cycles-pp.ext4_xattr_get 0.11 -0.0 0.09 ± 5% perf-profile.children.cycles-pp.dentry_needs_remove_privs 0.10 -0.0 0.08 ± 5% perf-profile.children.cycles-pp.__vfs_getxattr 0.11 ± 3% -0.0 0.10 ± 3% perf-profile.children.cycles-pp.alloc_empty_file 0.11 -0.0 0.10 ± 4% perf-profile.children.cycles-pp.__alloc_file 0.18 ± 6% +0.0 0.20 ± 4% perf-profile.children.cycles-pp.do_softirq_own_stack 0.19 ± 6% +0.0 0.21 ± 4% perf-profile.children.cycles-pp.irq_exit_rcu 0.03 ±100% +0.0 0.07 ± 7% perf-profile.children.cycles-pp.ktime_get_update_offsets_now 0.06 ± 45% +0.0 0.10 ± 11% perf-profile.children.cycles-pp.ktime_get 0.69 +0.1 0.75 perf-profile.children.cycles-pp.mutex_spin_on_owner 0.35 ± 6% +0.1 0.42 ± 6% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.30 ± 5% +0.1 0.38 ± 8% perf-profile.children.cycles-pp.tick_sched_timer 0.47 ± 7% +0.1 0.57 ± 7% perf-profile.children.cycles-pp.hrtimer_interrupt 0.47 ± 7% +0.1 0.58 ± 7% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 0.68 ± 6% +0.1 0.80 ± 4% perf-profile.children.cycles-pp.asm_call_sysvec_on_stack 0.68 ± 6% +0.1 0.81 ± 4% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 0.72 ± 7% +0.1 0.85 ± 4% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 0.24 -0.0 0.19 ± 3% perf-profile.self.cycles-pp.ext4_reserve_inode_write 0.29 ± 4% -0.0 0.24 ± 2% perf-profile.self.cycles-pp.ext4_mark_iloc_dirty 0.32 ± 3% -0.0 0.27 ± 4% perf-profile.self.cycles-pp.__mutex_lock 0.07 ± 7% -0.0 0.03 ±100% perf-profile.self.cycles-pp.ext4_inode_csum_set 0.12 ± 4% -0.0 0.09 ± 5% perf-profile.self.cycles-pp.mutex_unlock 0.12 ± 5% -0.0 0.10 ± 7% perf-profile.self.cycles-pp.mutex_lock 0.10 ± 6% -0.0 0.08 ± 12% perf-profile.self.cycles-pp.osq_unlock 0.12 -0.0 0.10 perf-profile.self.cycles-pp._raw_spin_lock 0.09 ± 7% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.__find_get_block 0.07 ± 5% -0.0 0.05 ± 8% perf-profile.self.cycles-pp.ext4_xattr_get 0.09 -0.0 0.08 ± 6% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.05 ± 45% +0.0 0.09 ± 12% perf-profile.self.cycles-pp.ktime_get 0.02 ±141% +0.0 0.06 perf-profile.self.cycles-pp.ktime_get_update_offsets_now 0.68 +0.1 0.74 ± 2% perf-profile.self.cycles-pp.mutex_spin_on_owner 0.02 ±141% +0.1 0.08 ± 8% perf-profile.self.cycles-pp.ext4_orphan_add 20823 ± 2% +14.2% 23775 ± 3% softirqs.CPU0.SCHED 17539 ± 2% +22.4% 21460 ± 5% softirqs.CPU1.SCHED 17122 +19.9% 20527 softirqs.CPU10.SCHED 17055 ± 2% +19.8% 20433 softirqs.CPU11.SCHED 16919 +21.4% 20538 softirqs.CPU12.SCHED 17013 ± 2% +20.6% 20510 ± 2% softirqs.CPU13.SCHED 17119 +19.8% 20501 softirqs.CPU14.SCHED 17158 +19.1% 20431 ± 2% softirqs.CPU15.SCHED 17223 +18.7% 20443 softirqs.CPU16.SCHED 16969 ± 2% +19.7% 20305 softirqs.CPU17.SCHED 16997 +20.3% 20441 softirqs.CPU18.SCHED 17126 +19.1% 20399 softirqs.CPU19.SCHED 17544 ± 3% +18.5% 20796 ± 5% softirqs.CPU2.SCHED 17207 +19.1% 20489 softirqs.CPU20.SCHED 17215 +17.4% 20219 softirqs.CPU21.SCHED 16574 +19.2% 19758 ± 3% softirqs.CPU22.SCHED 16901 ± 2% +17.7% 19893 ± 2% softirqs.CPU23.SCHED 16901 ± 3% +19.4% 20180 ± 2% softirqs.CPU24.SCHED 16818 +19.5% 20099 ± 2% softirqs.CPU25.SCHED 16892 +18.6% 20039 softirqs.CPU26.SCHED 16839 +20.0% 20205 softirqs.CPU27.SCHED 16926 +18.7% 20091 softirqs.CPU28.SCHED 16762 ± 2% +20.1% 20125 softirqs.CPU29.SCHED 17050 ± 2% +19.7% 20410 ± 2% softirqs.CPU3.SCHED 16961 +19.7% 20304 ± 2% softirqs.CPU30.SCHED 17013 +19.0% 20238 ± 2% softirqs.CPU31.SCHED 16836 +22.4% 20599 ± 3% softirqs.CPU32.SCHED 17160 +17.5% 20164 softirqs.CPU33.SCHED 16658 ± 2% +21.7% 20268 ± 2% softirqs.CPU34.SCHED 16980 ± 2% +18.2% 20069 ± 3% softirqs.CPU35.SCHED 16720 +20.0% 20067 ± 2% softirqs.CPU36.SCHED 17043 +19.3% 20338 softirqs.CPU37.SCHED 16935 ± 2% +20.0% 20321 softirqs.CPU38.SCHED 17031 +19.5% 20349 softirqs.CPU39.SCHED 17457 ± 3% +17.1% 20438 softirqs.CPU4.SCHED 16977 +20.1% 20393 ± 2% softirqs.CPU40.SCHED 16873 +20.7% 20369 softirqs.CPU41.SCHED 17216 ± 2% +17.6% 20238 ± 2% softirqs.CPU42.SCHED 17045 ± 2% +19.3% 20342 ± 2% softirqs.CPU43.SCHED 16175 ± 3% +20.4% 19473 ± 2% softirqs.CPU44.SCHED 16913 ± 2% +18.6% 20051 softirqs.CPU45.SCHED 16989 ± 2% +17.6% 19980 softirqs.CPU46.SCHED 16774 ± 2% +18.9% 19949 softirqs.CPU47.SCHED 16922 ± 2% +19.1% 20162 softirqs.CPU48.SCHED 16933 ± 3% +19.8% 20286 softirqs.CPU49.SCHED 17135 ± 3% +20.1% 20585 softirqs.CPU5.SCHED 16998 ± 2% +16.8% 19847 softirqs.CPU50.SCHED 16815 ± 2% +19.4% 20077 softirqs.CPU51.SCHED 16962 +18.6% 20111 softirqs.CPU52.SCHED 16853 ± 2% +20.0% 20224 softirqs.CPU53.SCHED 17002 +19.5% 20320 softirqs.CPU54.SCHED 17025 ± 2% +18.8% 20228 ± 2% softirqs.CPU55.SCHED 16851 +19.8% 20190 softirqs.CPU56.SCHED 17018 +18.3% 20132 softirqs.CPU57.SCHED 16949 ± 2% +19.8% 20308 ± 2% softirqs.CPU58.SCHED 17189 +18.0% 20287 softirqs.CPU59.SCHED 16891 +19.2% 20135 softirqs.CPU6.SCHED 16989 +20.0% 20387 softirqs.CPU60.SCHED 17075 +17.7% 20102 ± 2% softirqs.CPU61.SCHED 16938 ± 2% +19.8% 20285 softirqs.CPU62.SCHED 16854 ± 2% +19.2% 20094 ± 2% softirqs.CPU63.SCHED 17063 +18.5% 20216 softirqs.CPU64.SCHED 17048 +17.2% 19984 softirqs.CPU65.SCHED 16870 +19.4% 20138 softirqs.CPU66.SCHED 16817 +20.6% 20274 ± 2% softirqs.CPU67.SCHED 16868 ± 2% +20.1% 20256 softirqs.CPU68.SCHED 16890 +19.7% 20212 ± 2% softirqs.CPU69.SCHED 16842 +20.2% 20252 softirqs.CPU7.SCHED 17011 +19.8% 20375 softirqs.CPU70.SCHED 17048 +20.1% 20478 ± 2% softirqs.CPU71.SCHED 16881 ± 2% +19.6% 20194 softirqs.CPU72.SCHED 16931 +19.9% 20294 softirqs.CPU73.SCHED 17102 +18.7% 20308 softirqs.CPU74.SCHED 16926 ± 2% +19.4% 20206 softirqs.CPU75.SCHED 16757 ± 2% +21.6% 20380 ± 2% softirqs.CPU76.SCHED 17052 +19.8% 20420 softirqs.CPU77.SCHED 16998 +19.8% 20368 softirqs.CPU78.SCHED 17209 ± 2% +19.3% 20533 softirqs.CPU79.SCHED 17084 ± 2% +18.6% 20260 softirqs.CPU8.SCHED 16973 ± 2% +20.9% 20525 ± 2% softirqs.CPU80.SCHED 17198 +18.9% 20457 ± 2% softirqs.CPU81.SCHED 16907 ± 2% +20.8% 20419 ± 2% softirqs.CPU82.SCHED 17023 +19.3% 20316 ± 2% softirqs.CPU83.SCHED 16935 +18.8% 20116 ± 2% softirqs.CPU84.SCHED 17084 +18.8% 20297 softirqs.CPU85.SCHED 17086 +19.2% 20363 softirqs.CPU86.SCHED 16995 ± 3% +19.7% 20350 ± 2% softirqs.CPU87.SCHED 17100 ± 2% +19.4% 20420 softirqs.CPU9.SCHED 1498602 +19.3% 1788245 softirqs.SCHED 80372 +12.5% 90439 ± 2% softirqs.TIMER 902.67 +13.6% 1025 ± 2% interrupts.9:IO-APIC.9-fasteoi.acpi 1556804 ± 4% +35.0% 2102423 ± 3% interrupts.CAL:Function_call_interrupts 17978 ± 4% +27.2% 22867 ± 8% interrupts.CPU0.CAL:Function_call_interrupts 902210 +13.5% 1024347 ± 2% interrupts.CPU0.LOC:Local_timer_interrupts 3865 ± 3% +22.0% 4714 ± 3% interrupts.CPU0.RES:Rescheduling_interrupts 902.67 +13.6% 1025 ± 2% interrupts.CPU1.9:IO-APIC.9-fasteoi.acpi 17763 ± 7% +27.9% 22718 ± 8% interrupts.CPU1.CAL:Function_call_interrupts 902064 +13.5% 1024088 ± 2% interrupts.CPU1.LOC:Local_timer_interrupts 3156 ± 3% +22.8% 3874 ± 5% interrupts.CPU1.RES:Rescheduling_interrupts 16929 ± 5% +27.5% 21592 ± 6% interrupts.CPU10.CAL:Function_call_interrupts 902163 +13.5% 1024115 ± 2% interrupts.CPU10.LOC:Local_timer_interrupts 3174 ± 4% +20.5% 3825 ± 3% interrupts.CPU10.RES:Rescheduling_interrupts 17099 ± 4% +25.8% 21503 ± 7% interrupts.CPU11.CAL:Function_call_interrupts 902105 +13.5% 1024101 ± 2% interrupts.CPU11.LOC:Local_timer_interrupts 3166 ± 3% +22.1% 3865 ± 3% interrupts.CPU11.RES:Rescheduling_interrupts 16935 ± 5% +27.2% 21542 ± 7% interrupts.CPU12.CAL:Function_call_interrupts 902122 +13.5% 1024089 ± 2% interrupts.CPU12.LOC:Local_timer_interrupts 3134 ± 4% +22.4% 3837 ± 5% interrupts.CPU12.RES:Rescheduling_interrupts 17107 ± 5% +25.2% 21414 ± 6% interrupts.CPU13.CAL:Function_call_interrupts 902138 +13.5% 1024071 ± 2% interrupts.CPU13.LOC:Local_timer_interrupts 3154 ± 4% +21.3% 3825 ± 5% interrupts.CPU13.RES:Rescheduling_interrupts 16949 ± 4% +28.0% 21692 ± 8% interrupts.CPU14.CAL:Function_call_interrupts 902142 +13.5% 1024014 ± 2% interrupts.CPU14.LOC:Local_timer_interrupts 3154 ± 2% +20.8% 3809 ± 5% interrupts.CPU14.RES:Rescheduling_interrupts 16971 ± 6% +26.5% 21460 ± 7% interrupts.CPU15.CAL:Function_call_interrupts 902125 +13.5% 1024008 ± 2% interrupts.CPU15.LOC:Local_timer_interrupts 3159 ± 3% +21.7% 3847 ± 5% interrupts.CPU15.RES:Rescheduling_interrupts 16914 ± 4% +26.9% 21469 ± 6% interrupts.CPU16.CAL:Function_call_interrupts 902263 +13.5% 1024072 ± 2% interrupts.CPU16.LOC:Local_timer_interrupts 3176 ± 4% +20.6% 3831 ± 3% interrupts.CPU16.RES:Rescheduling_interrupts 16959 ± 5% +27.7% 21659 ± 7% interrupts.CPU17.CAL:Function_call_interrupts 902131 +13.5% 1024076 ± 2% interrupts.CPU17.LOC:Local_timer_interrupts 3164 ± 3% +21.1% 3834 ± 5% interrupts.CPU17.RES:Rescheduling_interrupts 16921 ± 5% +27.9% 21647 ± 7% interrupts.CPU18.CAL:Function_call_interrupts 901992 +13.5% 1024059 ± 2% interrupts.CPU18.LOC:Local_timer_interrupts 3114 ± 3% +23.7% 3851 ± 3% interrupts.CPU18.RES:Rescheduling_interrupts 16953 ± 5% +27.8% 21669 ± 7% interrupts.CPU19.CAL:Function_call_interrupts 902113 +13.5% 1024021 ± 2% interrupts.CPU19.LOC:Local_timer_interrupts 3186 ± 2% +20.8% 3850 ± 4% interrupts.CPU19.RES:Rescheduling_interrupts 17479 ± 5% +32.7% 23186 ± 3% interrupts.CPU2.CAL:Function_call_interrupts 902148 +13.5% 1024034 ± 2% interrupts.CPU2.LOC:Local_timer_interrupts 3163 ± 3% +21.2% 3835 ± 3% interrupts.CPU2.RES:Rescheduling_interrupts 16875 ± 5% +27.8% 21575 ± 7% interrupts.CPU20.CAL:Function_call_interrupts 902126 +13.5% 1024077 ± 2% interrupts.CPU20.LOC:Local_timer_interrupts 3125 ± 3% +23.9% 3874 ± 3% interrupts.CPU20.RES:Rescheduling_interrupts 16824 ± 4% +27.8% 21502 ± 7% interrupts.CPU21.CAL:Function_call_interrupts 902123 +13.5% 1024032 ± 2% interrupts.CPU21.LOC:Local_timer_interrupts 3119 ± 2% +23.8% 3861 ± 4% interrupts.CPU21.RES:Rescheduling_interrupts 18853 ± 10% +42.1% 26784 ± 7% interrupts.CPU22.CAL:Function_call_interrupts 902129 +13.5% 1024053 ± 2% interrupts.CPU22.LOC:Local_timer_interrupts 3076 ± 2% +15.9% 3564 ± 3% interrupts.CPU22.RES:Rescheduling_interrupts 18904 ± 10% +41.5% 26758 ± 7% interrupts.CPU23.CAL:Function_call_interrupts 902141 +13.5% 1024100 ± 2% interrupts.CPU23.LOC:Local_timer_interrupts 3055 ± 2% +13.8% 3478 ± 2% interrupts.CPU23.RES:Rescheduling_interrupts 18697 ± 10% +42.0% 26548 ± 7% interrupts.CPU24.CAL:Function_call_interrupts 902132 +13.5% 1024051 ± 2% interrupts.CPU24.LOC:Local_timer_interrupts 2996 ± 3% +15.3% 3454 ± 2% interrupts.CPU24.RES:Rescheduling_interrupts 18610 ± 10% +42.4% 26498 ± 7% interrupts.CPU25.CAL:Function_call_interrupts 902099 +13.5% 1024078 ± 2% interrupts.CPU25.LOC:Local_timer_interrupts 3061 ± 3% +15.4% 3531 ± 2% interrupts.CPU25.RES:Rescheduling_interrupts 18506 ± 10% +42.8% 26417 ± 7% interrupts.CPU26.CAL:Function_call_interrupts 902116 +13.5% 1024074 ± 2% interrupts.CPU26.LOC:Local_timer_interrupts 3016 ± 3% +15.6% 3488 ± 2% interrupts.CPU26.RES:Rescheduling_interrupts 18712 ± 11% +40.1% 26212 ± 7% interrupts.CPU27.CAL:Function_call_interrupts 902131 +13.5% 1023984 ± 2% interrupts.CPU27.LOC:Local_timer_interrupts 3032 ± 3% +16.4% 3528 ± 2% interrupts.CPU27.RES:Rescheduling_interrupts 18533 ± 10% +41.4% 26214 ± 7% interrupts.CPU28.CAL:Function_call_interrupts 902108 +13.5% 1024107 ± 2% interrupts.CPU28.LOC:Local_timer_interrupts 3022 ± 2% +17.1% 3539 ± 2% interrupts.CPU28.RES:Rescheduling_interrupts 18483 ± 10% +41.9% 26234 ± 7% interrupts.CPU29.CAL:Function_call_interrupts 902262 +13.5% 1024052 ± 2% interrupts.CPU29.LOC:Local_timer_interrupts 3034 +16.4% 3533 ± 2% interrupts.CPU29.RES:Rescheduling_interrupts 17359 ± 5% +29.3% 22454 ± 7% interrupts.CPU3.CAL:Function_call_interrupts 902230 +13.5% 1024080 ± 2% interrupts.CPU3.LOC:Local_timer_interrupts 3160 ± 4% +22.2% 3863 ± 3% interrupts.CPU3.RES:Rescheduling_interrupts 18526 ± 10% +42.2% 26340 ± 7% interrupts.CPU30.CAL:Function_call_interrupts 902146 +13.5% 1024088 ± 2% interrupts.CPU30.LOC:Local_timer_interrupts 3003 +16.6% 3503 ± 2% interrupts.CPU30.RES:Rescheduling_interrupts 18417 ± 10% +42.1% 26162 ± 8% interrupts.CPU31.CAL:Function_call_interrupts 902108 +13.5% 1024056 ± 2% interrupts.CPU31.LOC:Local_timer_interrupts 2992 ± 3% +17.2% 3506 ± 2% interrupts.CPU31.RES:Rescheduling_interrupts 18324 ± 10% +41.6% 25944 ± 7% interrupts.CPU32.CAL:Function_call_interrupts 902092 +13.5% 1024078 ± 2% interrupts.CPU32.LOC:Local_timer_interrupts 3037 ± 3% +15.2% 3498 ± 2% interrupts.CPU32.RES:Rescheduling_interrupts 18353 ± 11% +42.0% 26057 ± 7% interrupts.CPU33.CAL:Function_call_interrupts 902085 +13.5% 1024101 ± 2% interrupts.CPU33.LOC:Local_timer_interrupts 3055 ± 4% +14.6% 3502 ± 2% interrupts.CPU33.RES:Rescheduling_interrupts 18376 ± 10% +41.2% 25953 ± 7% interrupts.CPU34.CAL:Function_call_interrupts 902095 +13.5% 1024078 ± 2% interrupts.CPU34.LOC:Local_timer_interrupts 3015 ± 2% +14.4% 3448 ± 3% interrupts.CPU34.RES:Rescheduling_interrupts 18232 ± 10% +42.5% 25986 ± 7% interrupts.CPU35.CAL:Function_call_interrupts 902197 +13.5% 1024042 ± 2% interrupts.CPU35.LOC:Local_timer_interrupts 2990 ± 3% +16.6% 3487 ± 2% interrupts.CPU35.RES:Rescheduling_interrupts 18314 ± 10% +41.9% 25991 ± 7% interrupts.CPU36.CAL:Function_call_interrupts 902046 +13.5% 1024078 ± 2% interrupts.CPU36.LOC:Local_timer_interrupts 3002 ± 2% +15.9% 3479 interrupts.CPU36.RES:Rescheduling_interrupts 18369 ± 10% +40.6% 25829 ± 7% interrupts.CPU37.CAL:Function_call_interrupts 902117 +13.5% 1024026 ± 2% interrupts.CPU37.LOC:Local_timer_interrupts 2999 ± 2% +16.4% 3490 ± 2% interrupts.CPU37.RES:Rescheduling_interrupts 18331 ± 10% +41.5% 25932 ± 7% interrupts.CPU38.CAL:Function_call_interrupts 902118 +13.5% 1024085 ± 2% interrupts.CPU38.LOC:Local_timer_interrupts 2995 ± 3% +16.0% 3474 ± 4% interrupts.CPU38.RES:Rescheduling_interrupts 18313 ± 10% +41.6% 25935 ± 7% interrupts.CPU39.CAL:Function_call_interrupts 902082 +13.5% 1024103 ± 2% interrupts.CPU39.LOC:Local_timer_interrupts 2973 ± 2% +18.1% 3510 interrupts.CPU39.RES:Rescheduling_interrupts 18089 ± 10% +20.9% 21864 ± 6% interrupts.CPU4.CAL:Function_call_interrupts 902120 +13.5% 1024041 ± 2% interrupts.CPU4.LOC:Local_timer_interrupts 3126 ± 2% +22.9% 3842 ± 3% interrupts.CPU4.RES:Rescheduling_interrupts 18303 ± 10% +42.3% 26045 ± 7% interrupts.CPU40.CAL:Function_call_interrupts 902120 +13.5% 1024097 ± 2% interrupts.CPU40.LOC:Local_timer_interrupts 3019 ± 2% +14.7% 3462 ± 3% interrupts.CPU40.RES:Rescheduling_interrupts 18288 ± 10% +42.8% 26107 ± 7% interrupts.CPU41.CAL:Function_call_interrupts 902064 +13.5% 1024075 ± 2% interrupts.CPU41.LOC:Local_timer_interrupts 2992 +16.6% 3489 ± 2% interrupts.CPU41.RES:Rescheduling_interrupts 18169 ± 10% +43.3% 26042 ± 7% interrupts.CPU42.CAL:Function_call_interrupts 902151 +13.5% 1024091 ± 2% interrupts.CPU42.LOC:Local_timer_interrupts 3010 ± 2% +16.7% 3512 ± 2% interrupts.CPU42.RES:Rescheduling_interrupts 18161 ± 11% +42.4% 25871 ± 7% interrupts.CPU43.CAL:Function_call_interrupts 902074 +13.5% 1024076 ± 2% interrupts.CPU43.LOC:Local_timer_interrupts 2946 ± 2% +18.1% 3478 ± 2% interrupts.CPU43.RES:Rescheduling_interrupts 16873 ± 4% +26.9% 21415 ± 6% interrupts.CPU44.CAL:Function_call_interrupts 902175 +13.5% 1024065 ± 2% interrupts.CPU44.LOC:Local_timer_interrupts 3067 ± 4% +21.2% 3719 ± 5% interrupts.CPU44.RES:Rescheduling_interrupts 16915 ± 5% +28.3% 21704 ± 6% interrupts.CPU45.CAL:Function_call_interrupts 902101 +13.5% 1024072 ± 2% interrupts.CPU45.LOC:Local_timer_interrupts 3056 ± 3% +22.2% 3734 ± 3% interrupts.CPU45.RES:Rescheduling_interrupts 16894 ± 4% +27.6% 21553 ± 7% interrupts.CPU46.CAL:Function_call_interrupts 902132 +13.5% 1024049 ± 2% interrupts.CPU46.LOC:Local_timer_interrupts 3069 ± 2% +22.9% 3773 ± 4% interrupts.CPU46.RES:Rescheduling_interrupts 16781 ± 5% +27.6% 21408 ± 7% interrupts.CPU47.CAL:Function_call_interrupts 902147 +13.5% 1024088 ± 2% interrupts.CPU47.LOC:Local_timer_interrupts 3086 ± 3% +20.9% 3732 ± 4% interrupts.CPU47.RES:Rescheduling_interrupts 16892 ± 5% +27.3% 21496 ± 7% interrupts.CPU48.CAL:Function_call_interrupts 902150 +13.5% 1024084 ± 2% interrupts.CPU48.LOC:Local_timer_interrupts 3123 ± 4% +19.1% 3721 ± 4% interrupts.CPU48.RES:Rescheduling_interrupts 16686 ± 4% +27.9% 21340 ± 7% interrupts.CPU49.CAL:Function_call_interrupts 902113 +13.5% 1024070 ± 2% interrupts.CPU49.LOC:Local_timer_interrupts 3039 ± 4% +22.8% 3733 ± 4% interrupts.CPU49.RES:Rescheduling_interrupts 17142 ± 5% +29.2% 22150 ± 7% interrupts.CPU5.CAL:Function_call_interrupts 902296 +13.5% 1024251 ± 2% interrupts.CPU5.LOC:Local_timer_interrupts 3178 ± 3% +21.7% 3866 ± 3% interrupts.CPU5.RES:Rescheduling_interrupts 16894 ± 4% +27.4% 21529 ± 6% interrupts.CPU50.CAL:Function_call_interrupts 902077 +13.5% 1024082 ± 2% interrupts.CPU50.LOC:Local_timer_interrupts 3106 ± 4% +21.0% 3759 ± 3% interrupts.CPU50.RES:Rescheduling_interrupts 16842 ± 4% +28.0% 21565 ± 6% interrupts.CPU51.CAL:Function_call_interrupts 902143 +13.5% 1024096 ± 2% interrupts.CPU51.LOC:Local_timer_interrupts 3106 ± 3% +20.1% 3730 ± 3% interrupts.CPU51.RES:Rescheduling_interrupts 16754 ± 5% +28.8% 21571 ± 7% interrupts.CPU52.CAL:Function_call_interrupts 902098 +13.5% 1024069 ± 2% interrupts.CPU52.LOC:Local_timer_interrupts 3050 ± 2% +21.8% 3715 ± 4% interrupts.CPU52.RES:Rescheduling_interrupts 16868 ± 5% +27.3% 21481 ± 6% interrupts.CPU53.CAL:Function_call_interrupts 902150 +13.5% 1024062 ± 2% interrupts.CPU53.LOC:Local_timer_interrupts 3054 ± 3% +22.1% 3729 ± 4% interrupts.CPU53.RES:Rescheduling_interrupts 16759 ± 5% +27.2% 21311 ± 7% interrupts.CPU54.CAL:Function_call_interrupts 902146 +13.5% 1024081 ± 2% interrupts.CPU54.LOC:Local_timer_interrupts 3061 ± 4% +23.5% 3780 ± 5% interrupts.CPU54.RES:Rescheduling_interrupts 16893 ± 4% +26.4% 21358 ± 7% interrupts.CPU55.CAL:Function_call_interrupts 902113 +13.5% 1024081 ± 2% interrupts.CPU55.LOC:Local_timer_interrupts 3073 ± 3% +21.8% 3742 ± 5% interrupts.CPU55.RES:Rescheduling_interrupts 16820 ± 4% +26.6% 21288 ± 7% interrupts.CPU56.CAL:Function_call_interrupts 902136 +13.5% 1024085 ± 2% interrupts.CPU56.LOC:Local_timer_interrupts 3052 ± 4% +22.3% 3734 ± 5% interrupts.CPU56.RES:Rescheduling_interrupts 16837 ± 4% +26.6% 21324 ± 7% interrupts.CPU57.CAL:Function_call_interrupts 902135 +13.5% 1024226 ± 2% interrupts.CPU57.LOC:Local_timer_interrupts 3075 ± 3% +21.1% 3723 ± 3% interrupts.CPU57.RES:Rescheduling_interrupts 16851 ± 5% +26.7% 21359 ± 7% interrupts.CPU58.CAL:Function_call_interrupts 902145 +13.5% 1024084 ± 2% interrupts.CPU58.LOC:Local_timer_interrupts 3067 ± 3% +22.0% 3742 ± 4% interrupts.CPU58.RES:Rescheduling_interrupts 16813 ± 4% +27.8% 21480 ± 7% interrupts.CPU59.CAL:Function_call_interrupts 902135 +13.5% 1024105 ± 2% interrupts.CPU59.LOC:Local_timer_interrupts 3075 ± 3% +23.1% 3784 ± 5% interrupts.CPU59.RES:Rescheduling_interrupts 17318 ± 4% +25.8% 21778 ± 6% interrupts.CPU6.CAL:Function_call_interrupts 902058 +13.5% 1024091 ± 2% interrupts.CPU6.LOC:Local_timer_interrupts 3167 ± 4% +22.1% 3866 ± 4% interrupts.CPU6.RES:Rescheduling_interrupts 16910 ± 4% +26.7% 21430 ± 7% interrupts.CPU60.CAL:Function_call_interrupts 902154 +13.5% 1024028 ± 2% interrupts.CPU60.LOC:Local_timer_interrupts 3097 ± 3% +21.4% 3760 ± 4% interrupts.CPU60.RES:Rescheduling_interrupts 16900 ± 4% +27.5% 21555 ± 7% interrupts.CPU61.CAL:Function_call_interrupts 902139 +13.5% 1024089 ± 2% interrupts.CPU61.LOC:Local_timer_interrupts 3063 ± 4% +22.0% 3738 ± 3% interrupts.CPU61.RES:Rescheduling_interrupts 16991 ± 4% +26.0% 21414 ± 7% interrupts.CPU62.CAL:Function_call_interrupts 902132 +13.5% 1024089 ± 2% interrupts.CPU62.LOC:Local_timer_interrupts 3125 ± 3% +19.7% 3742 ± 3% interrupts.CPU62.RES:Rescheduling_interrupts 16967 ± 5% +26.9% 21537 ± 6% interrupts.CPU63.CAL:Function_call_interrupts 902111 +13.5% 1024071 ± 2% interrupts.CPU63.LOC:Local_timer_interrupts 3110 ± 3% +22.1% 3799 ± 4% interrupts.CPU63.RES:Rescheduling_interrupts 16832 ± 4% +28.1% 21557 ± 7% interrupts.CPU64.CAL:Function_call_interrupts 902165 +13.5% 1024037 ± 2% interrupts.CPU64.LOC:Local_timer_interrupts 3095 ± 3% +22.5% 3791 ± 3% interrupts.CPU64.RES:Rescheduling_interrupts 16887 ± 5% +27.7% 21563 ± 7% interrupts.CPU65.CAL:Function_call_interrupts 902159 +13.5% 1024083 ± 2% interrupts.CPU65.LOC:Local_timer_interrupts 3079 ± 4% +23.5% 3802 ± 4% interrupts.CPU65.RES:Rescheduling_interrupts 18236 ± 10% +43.5% 26177 ± 8% interrupts.CPU66.CAL:Function_call_interrupts 902097 +13.5% 1023991 ± 2% interrupts.CPU66.LOC:Local_timer_interrupts 2952 ± 3% +16.0% 3423 ± 3% interrupts.CPU66.RES:Rescheduling_interrupts 18440 ± 11% +41.6% 26117 ± 7% interrupts.CPU67.CAL:Function_call_interrupts 902050 +13.5% 1024090 ± 2% interrupts.CPU67.LOC:Local_timer_interrupts 2948 ± 2% +16.8% 3443 ± 3% interrupts.CPU67.RES:Rescheduling_interrupts 18284 ± 10% +43.0% 26148 ± 7% interrupts.CPU68.CAL:Function_call_interrupts 902119 +13.5% 1024096 ± 2% interrupts.CPU68.LOC:Local_timer_interrupts 2960 ± 2% +17.7% 3485 ± 3% interrupts.CPU68.RES:Rescheduling_interrupts 18357 ± 10% +42.0% 26070 ± 7% interrupts.CPU69.CAL:Function_call_interrupts 902096 +13.5% 1024089 ± 2% interrupts.CPU69.LOC:Local_timer_interrupts 2977 +16.1% 3455 ± 5% interrupts.CPU69.RES:Rescheduling_interrupts 16984 ± 5% +28.6% 21843 ± 6% interrupts.CPU7.CAL:Function_call_interrupts 902119 +13.5% 1024116 ± 2% interrupts.CPU7.LOC:Local_timer_interrupts 3122 ± 2% +22.8% 3835 ± 4% interrupts.CPU7.RES:Rescheduling_interrupts 18176 ± 10% +43.4% 26056 ± 7% interrupts.CPU70.CAL:Function_call_interrupts 902106 +13.5% 1024063 ± 2% interrupts.CPU70.LOC:Local_timer_interrupts 2937 ± 2% +18.5% 3479 ± 2% interrupts.CPU70.RES:Rescheduling_interrupts 18320 ± 10% +42.2% 26047 ± 8% interrupts.CPU71.CAL:Function_call_interrupts 902142 +13.5% 1024123 ± 2% interrupts.CPU71.LOC:Local_timer_interrupts 2999 ± 3% +16.3% 3487 ± 3% interrupts.CPU71.RES:Rescheduling_interrupts 18410 ± 10% +42.0% 26143 ± 7% interrupts.CPU72.CAL:Function_call_interrupts 902129 +13.5% 1024094 ± 2% interrupts.CPU72.LOC:Local_timer_interrupts 2965 ± 3% +17.6% 3488 ± 4% interrupts.CPU72.RES:Rescheduling_interrupts 18305 ± 10% +42.5% 26085 ± 8% interrupts.CPU73.CAL:Function_call_interrupts 902133 +13.5% 1024041 ± 2% interrupts.CPU73.LOC:Local_timer_interrupts 2977 ± 3% +16.6% 3471 ± 3% interrupts.CPU73.RES:Rescheduling_interrupts 18344 ± 10% +42.6% 26152 ± 7% interrupts.CPU74.CAL:Function_call_interrupts 902128 +13.5% 1024065 ± 2% interrupts.CPU74.LOC:Local_timer_interrupts 3002 ± 2% +17.0% 3511 ± 3% interrupts.CPU74.RES:Rescheduling_interrupts 18310 ± 10% +41.9% 25976 ± 7% interrupts.CPU75.CAL:Function_call_interrupts 902143 +13.5% 1024072 ± 2% interrupts.CPU75.LOC:Local_timer_interrupts 2922 ± 3% +18.5% 3463 ± 3% interrupts.CPU75.RES:Rescheduling_interrupts 18275 ± 10% +41.6% 25883 ± 7% interrupts.CPU76.CAL:Function_call_interrupts 902099 +13.5% 1024098 ± 2% interrupts.CPU76.LOC:Local_timer_interrupts 2921 ± 4% +18.3% 3457 ± 2% interrupts.CPU76.RES:Rescheduling_interrupts 18330 ± 10% +42.1% 26054 ± 7% interrupts.CPU77.CAL:Function_call_interrupts 902158 +13.5% 1024103 ± 2% interrupts.CPU77.LOC:Local_timer_interrupts 2960 ± 3% +16.4% 3447 ± 3% interrupts.CPU77.RES:Rescheduling_interrupts 18199 ± 10% +43.9% 26183 ± 8% interrupts.CPU78.CAL:Function_call_interrupts 902127 +13.5% 1024113 ± 2% interrupts.CPU78.LOC:Local_timer_interrupts 2936 ± 3% +17.8% 3460 ± 2% interrupts.CPU78.RES:Rescheduling_interrupts 18302 ± 9% +42.2% 26018 ± 7% interrupts.CPU79.CAL:Function_call_interrupts 902128 +13.5% 1024093 ± 2% interrupts.CPU79.LOC:Local_timer_interrupts 2933 ± 3% +16.1% 3406 ± 2% interrupts.CPU79.RES:Rescheduling_interrupts 17257 ± 4% +26.1% 21766 ± 6% interrupts.CPU8.CAL:Function_call_interrupts 902064 +13.5% 1024249 ± 2% interrupts.CPU8.LOC:Local_timer_interrupts 3148 ± 4% +22.8% 3867 ± 4% interrupts.CPU8.RES:Rescheduling_interrupts 18351 ± 10% +41.6% 25983 ± 8% interrupts.CPU80.CAL:Function_call_interrupts 902119 +13.5% 1024104 ± 2% interrupts.CPU80.LOC:Local_timer_interrupts 2950 ± 2% +15.0% 3393 ± 2% interrupts.CPU80.RES:Rescheduling_interrupts 18351 ± 10% +42.5% 26147 ± 8% interrupts.CPU81.CAL:Function_call_interrupts 902138 +13.5% 1024087 ± 2% interrupts.CPU81.LOC:Local_timer_interrupts 2952 ± 2% +16.0% 3424 ± 2% interrupts.CPU81.RES:Rescheduling_interrupts 18259 ± 10% +42.4% 26004 ± 7% interrupts.CPU82.CAL:Function_call_interrupts 902149 +13.5% 1024272 ± 2% interrupts.CPU82.LOC:Local_timer_interrupts 2931 ± 2% +17.0% 3429 ± 2% interrupts.CPU82.RES:Rescheduling_interrupts 18210 ± 10% +42.8% 26012 ± 8% interrupts.CPU83.CAL:Function_call_interrupts 902111 +13.5% 1024099 ± 2% interrupts.CPU83.LOC:Local_timer_interrupts 2933 ± 2% +16.6% 3419 ± 2% interrupts.CPU83.RES:Rescheduling_interrupts 18085 ± 10% +44.2% 26086 ± 8% interrupts.CPU84.CAL:Function_call_interrupts 902118 +13.5% 1024109 ± 2% interrupts.CPU84.LOC:Local_timer_interrupts 2981 ± 2% +13.9% 3396 ± 2% interrupts.CPU84.RES:Rescheduling_interrupts 18294 ± 10% +43.0% 26167 ± 7% interrupts.CPU85.CAL:Function_call_interrupts 902135 +13.5% 1024098 ± 2% interrupts.CPU85.LOC:Local_timer_interrupts 2937 ± 2% +16.8% 3430 ± 3% interrupts.CPU85.RES:Rescheduling_interrupts 18253 ± 10% +42.7% 26056 ± 8% interrupts.CPU86.CAL:Function_call_interrupts 902302 +13.5% 1024102 ± 2% interrupts.CPU86.LOC:Local_timer_interrupts 2999 ± 2% +13.5% 3403 ± 3% interrupts.CPU86.RES:Rescheduling_interrupts 18248 ± 10% +41.5% 25817 ± 8% interrupts.CPU87.CAL:Function_call_interrupts 902127 +13.5% 1024064 ± 2% interrupts.CPU87.LOC:Local_timer_interrupts 2954 ± 2% +14.3% 3377 ± 2% interrupts.CPU87.RES:Rescheduling_interrupts 16991 ± 4% +26.9% 21557 ± 6% interrupts.CPU9.CAL:Function_call_interrupts 902162 +13.5% 1024110 ± 2% interrupts.CPU9.LOC:Local_timer_interrupts 3164 ± 3% +22.5% 3876 ± 5% interrupts.CPU9.RES:Rescheduling_interrupts 79387471 +13.5% 90119618 ± 2% interrupts.LOC:Local_timer_interrupts 269188 +19.2% 320766 ± 2% interrupts.RES:Rescheduling_interrupts aim7.jobs-per-min 14000 +-------------------------------------------------------------------+ | + | 13500 |-+ + + ++ + +.+ + + + .+ | | .+ + + : :: : : :: : :.+. + + : .++ + +.| |. +.+ + : : :: : :: : + ++.+. + +. + + :: .+ + | 13000 |-+ + + :: + + + + :: + | | : + : : | 12500 |-+ + :: | | :: | 12000 |-+ O O O O O O O + | | O O O O O O | | O O O O O O O O | 11500 |-+O O O O O | | O O | 11000 +-------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample *************************************************************************************************** lkp-knl-f1: 256 threads Intel(R) Genuine Intel(R) CPU 0000 @ 1.30GHz with 112G memory ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/testtime/ucode: gcc-9/performance/x86_64-rhel-8.3/debian-10.4-x86_64-20200603.cgz/lkp-knl-f1/all/aim9/5s/0xffff0190 commit: 4392fbc4ba ("ext4: drop sync argument of ext4_commit_super()") 05c2c00f37 ("ext4: protect superblock modifications with a buffer lock") 4392fbc4bab57db3 05c2c00f3769abb9e323fcaca70 ---------------- --------------------------- fail:runs %reproduction fail:runs | | | :5 20% 1:6 dmesg.WARNING:missing_R10_value_at__fsnotify_parent/0x 0:5 4% 0:6 perf-profile.children.cycles-pp.error_return 2:5 8% 2:6 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 74384 +3.4% 76883 aim9.dgram_pipe.ops_per_sec 121865 +2.6% 125020 aim9.disk_cp.ops_per_sec 198997 +1.9% 202688 aim9.disk_wrt.ops_per_sec 52029 -3.9% 49990 aim9.link_test.ops_per_sec 95645 +7.5% 102865 aim9.signal_test.ops_per_sec 80028 +5.0% 84023 aim9.stream_pipe.ops_per_sec 101888 +3.9% 105872 aim9.sync_disk_rw.ops_per_sec 17794 +1.5% 18069 aim9.tcp_test.ops_per_sec 35329 +2.1% 36070 aim9.udp_test.ops_per_sec 84505 +1.3% 85562 proc-vmstat.pgactivate 1.037e+08 -6.0% 97479544 ± 6% perf-stat.i.cache-references 1.034e+08 -6.0% 97161961 ± 6% perf-stat.ps.cache-references 412.50 ± 3% +25.8% 519.03 ± 9% sched_debug.cfs_rq:/.util_est_enqueued.max 45.99 ± 10% +31.6% 60.51 ± 11% sched_debug.cfs_rq:/.util_est_enqueued.stddev 20873 ± 13% -16.2% 17493 ± 6% softirqs.CPU134.RCU 10054 ± 56% -81.3% 1880 ±126% softirqs.CPU14.NET_RX 0.05 ± 6% +45.0% 0.08 ± 28% perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 0.04 ± 4% +22.3% 0.05 ± 10% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.01 ±123% +153.5% 0.04 ± 24% perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all 0.09 ± 18% +82.9% 0.17 ± 52% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.01 ±123% +175.8% 0.04 ± 12% perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all 141.15 ± 21% -41.1% 83.12 ± 41% perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 0.18 ± 40% +70.5% 0.31 ± 41% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.dput.path_put.set_fs_pwd 0.23 ± 33% +59.8% 0.37 ± 26% perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.kmem_cache_alloc.getname_flags.__x64_sys_unlink 141.12 ± 21% -41.1% 83.09 ± 41% perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 305.60 ± 29% -43.3% 173.17 ± 21% interrupts.CPU114.NMI:Non-maskable_interrupts 305.60 ± 29% -43.3% 173.17 ± 21% interrupts.CPU114.PMI:Performance_monitoring_interrupts 335.00 ± 20% -36.6% 212.33 ± 32% interrupts.CPU239.NMI:Non-maskable_interrupts 335.00 ± 20% -36.6% 212.33 ± 32% interrupts.CPU239.PMI:Performance_monitoring_interrupts 314.80 ± 32% -37.0% 198.33 ± 46% interrupts.CPU251.NMI:Non-maskable_interrupts 314.80 ± 32% -37.0% 198.33 ± 46% interrupts.CPU251.PMI:Performance_monitoring_interrupts 23.20 ± 44% +117.7% 50.50 ± 57% interrupts.CPU53.RES:Rescheduling_interrupts 1417 ± 10% +24.6% 1766 ± 13% interrupts.CPU6.CAL:Function_call_interrupts 30.40 ± 59% +198.8% 90.83 ± 23% interrupts.CPU6.RES:Rescheduling_interrupts 1260 ± 5% +9.3% 1377 ± 7% interrupts.CPU7.CAL:Function_call_interrupts 253.40 ± 29% +46.7% 371.83 ± 27% interrupts.CPU90.NMI:Non-maskable_interrupts 253.40 ± 29% +46.7% 371.83 ± 27% interrupts.CPU90.PMI:Performance_monitoring_interrupts 1.07 ± 2% -0.2 0.86 ± 24% perf-profile.calltrace.cycles-pp.irqtime_account_irq.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 0.75 ± 4% +0.3 1.03 ± 34% perf-profile.calltrace.cycles-pp.timerqueue_add.enqueue_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.90 ± 5% +0.3 1.21 ± 33% perf-profile.calltrace.cycles-pp.enqueue_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack 2.40 ± 6% +0.4 2.76 ± 9% perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt 1.42 ± 7% +0.4 1.84 ± 14% perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack 0.33 ± 7% -0.1 0.25 ± 11% perf-profile.children.cycles-pp.irq_work_needs_cpu 0.09 ± 13% +0.0 0.11 ± 8% perf-profile.children.cycles-pp.hrtimer_run_queues 0.09 ± 9% +0.0 0.12 ± 19% perf-profile.children.cycles-pp.__set_cpus_allowed_ptr 0.06 ± 12% +0.0 0.09 ± 13% perf-profile.children.cycles-pp.irq_work_single 0.06 ± 12% +0.0 0.09 ± 13% perf-profile.children.cycles-pp.asm_sysvec_irq_work 0.06 ± 12% +0.0 0.09 ± 13% perf-profile.children.cycles-pp.sysvec_irq_work 0.06 ± 12% +0.0 0.09 ± 13% perf-profile.children.cycles-pp.__sysvec_irq_work 0.06 ± 12% +0.0 0.09 ± 13% perf-profile.children.cycles-pp.irq_work_run 0.11 ± 8% +0.0 0.15 ± 22% perf-profile.children.cycles-pp.__x64_sys_sched_setaffinity 0.14 ± 5% +0.1 0.19 ± 17% perf-profile.children.cycles-pp.irq_work_run_list 0.29 ± 6% +0.1 0.39 ± 28% perf-profile.children.cycles-pp.rb_insert_color 0.77 ± 4% +0.3 1.07 ± 34% perf-profile.children.cycles-pp.timerqueue_add 0.92 ± 5% +0.3 1.23 ± 32% perf-profile.children.cycles-pp.enqueue_hrtimer 2.45 ± 6% +0.4 2.82 ± 9% perf-profile.children.cycles-pp.clockevents_program_event 4.23 ± 2% +0.4 4.61 ± 7% perf-profile.children.cycles-pp.scheduler_tick 0.97 ± 2% -0.1 0.83 ± 15% perf-profile.self.cycles-pp.irqtime_account_irq 0.75 ± 4% -0.1 0.67 ± 6% perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt 0.30 ± 8% -0.1 0.23 ± 15% perf-profile.self.cycles-pp.irq_work_needs_cpu 0.19 ± 3% +0.0 0.21 ± 6% perf-profile.self.cycles-pp.trigger_load_balance 0.09 ± 13% +0.0 0.11 ± 8% perf-profile.self.cycles-pp.hrtimer_run_queues 0.02 ±122% +0.0 0.06 ± 14% perf-profile.self.cycles-pp.io_serial_in 0.14 ± 11% +0.0 0.19 ± 22% perf-profile.self.cycles-pp.tick_sched_handle 0.27 ± 4% +0.1 0.37 ± 27% perf-profile.self.cycles-pp.rb_insert_color Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Oliver Sang