Hi Robot, thanks for your testing! On 7/9/22 10:42 PM, kernel test robot Wrote: > > > Greeting, > > FYI, we noticed a -11.7% regression of phoronix-test-suite.fio.SequentialWrite.IO_uring.Yes.No.4KB.DefaultTestDirectory.mb_s due to commit: > > > commit: 32fe13cd7aa184ed349d698ebf6f420fa426dd73 ("[PATCH v4 7/7] sched/fair: de-entropy for SIS filter") > url: https://github.com/intel-lab-lkp/linux/commits/Abel-Wu/sched-fair-improve-scan-efficiency-of-SIS/20220619-200743 > base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git f3dd3f674555bd9455c5ae7fafce0696bd9931b3 > patch link: https://lore.kernel.org/lkml/20220619120451.95251-8-wuyun.abel@bytedance.com > > in testcase: phoronix-test-suite > on test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz with 512G memory Does SNC enabled? > with following parameters: > > test: fio-1.14.1 > option_a: Sequential Write > option_b: IO_uring > option_c: Yes > option_d: No > option_e: 4KB > option_f: Default Test Directory > cpufreq_governor: performance > ucode: 0x500320a > > test-description: The Phoronix Test Suite is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added. > test-url: http://www.phoronix-test-suite.com/ > > In addition to that, the commit also has significant impact on the following tests: > > +------------------+-------------------------------------------------------------------------------------+ > | testcase: change | stress-ng: stress-ng.vm-rw.ops_per_sec 113.5% improvement | > | test machine | 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz with 128G memory | > | test parameters | class=memory | > | | cpufreq_governor=performance | > | | nr_threads=100% | > | | test=vm-rw | > | | testtime=60s | > | | ucode=0xd000331 | > +------------------+-------------------------------------------------------------------------------------+ > > > If you fix the issue, kindly add following tag > Reported-by: kernel test robot > > > Details are as below: > --------------------------------------------------------------------------------------------------> > > > To reproduce: > > git clone https://github.com/intel/lkp-tests.git > cd lkp-tests > sudo bin/lkp install job.yaml # job file is attached in this email > bin/lkp split-job --compatible job.yaml # generate the yaml file for lkp run > sudo bin/lkp run generated-yaml-file > > # if come across any failure that blocks the test, > # please remove ~/.lkp and /lkp dir to run from a clean state. > > ========================================================================================= > compiler/cpufreq_governor/kconfig/option_a/option_b/option_c/option_d/option_e/option_f/rootfs/tbox_group/test/testcase/ucode: > gcc-11/performance/x86_64-rhel-8.3/Sequential Write/IO_uring/Yes/No/4KB/Default Test Directory/debian-x86_64-phoronix/lkp-csl-2sp7/fio-1.14.1/phoronix-test-suite/0x500320a > > commit: > fcc108377a ("sched/fair: skip busy cores in SIS search") > 32fe13cd7a ("sched/fair: de-entropy for SIS filter") Does the 5th patch applied? It's also important to bail out early if the system is busy enough that idle cpus can hardly exist. > > fcc108377a7cf79c 32fe13cd7aa184ed349d698ebf6 > ---------------- --------------------------- > %stddev %change %stddev > \ | \ > 166666 -11.6% 147277 ± 3% phoronix-test-suite.fio.SequentialWrite.IO_uring.Yes.No.4KB.DefaultTestDirectory.iops > 651.00 -11.7% 574.83 ± 3% phoronix-test-suite.fio.SequentialWrite.IO_uring.Yes.No.4KB.DefaultTestDirectory.mb_s > 3145 ± 5% -18.4% 2565 ± 12% meminfo.Writeback > 0.19 ± 4% -0.0 0.17 ± 2% mpstat.cpu.all.iowait% > 2228 ± 33% -37.5% 1392 ± 21% numa-meminfo.node0.Writeback > 553.33 ± 37% -35.9% 354.83 ± 18% numa-vmstat.node0.nr_writeback I will try to reproduce the test to see why there is such a big change. > 445604 ± 4% -12.5% 390116 ± 4% vmstat.io.bo > 14697101 ± 3% -11.0% 13074497 ± 4% perf-stat.i.cache-misses > 9447 ± 8% -37.6% 5890 ± 5% perf-stat.i.cpu-migrations > 5125 ± 6% +12.9% 5786 ± 6% perf-stat.i.instructions-per-iTLB-miss > 2330431 ± 4% -11.4% 2064845 ± 4% perf-stat.i.node-loads > 2.55 ±104% -1.6 0.96 ± 14% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle > 2.62 ±102% -1.6 0.99 ± 14% perf-profile.children.cycles-pp.poll_idle > 0.82 ± 23% -0.3 0.53 ± 23% perf-profile.children.cycles-pp.asm_sysvec_call_function_single > 0.74 ± 23% -0.3 0.46 ± 23% perf-profile.children.cycles-pp.sysvec_call_function_single > 0.69 ± 24% -0.3 0.44 ± 24% perf-profile.children.cycles-pp.__sysvec_call_function_single > 0.38 ± 10% -0.1 0.28 ± 18% perf-profile.children.cycles-pp.__perf_event_header__init_id > 0.16 ± 13% -0.0 0.11 ± 22% perf-profile.children.cycles-pp.__task_pid_nr_ns > 2.10 ±108% -1.3 0.79 ± 11% perf-profile.self.cycles-pp.poll_idle > 0.16 ± 13% -0.0 0.11 ± 22% perf-profile.self.cycles-pp.__task_pid_nr_ns > > > *************************************************************************************************** > lkp-icl-2sp6: 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz with 128G memory > ========================================================================================= > class/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode: > memory/gcc-11/performance/x86_64-rhel-8.3/100%/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp6/vm-rw/stress-ng/60s/0xd000331 > > commit: > fcc108377a ("sched/fair: skip busy cores in SIS search") > 32fe13cd7a ("sched/fair: de-entropy for SIS filter") > > fcc108377a7cf79c 32fe13cd7aa184ed349d698ebf6 > ---------------- --------------------------- > %stddev %change %stddev > \ | \ > 7328835 ± 17% +3441.0% 2.595e+08 ± 12% stress-ng.time.involuntary_context_switches It's really horrible.. > 123165 ± 3% -14.1% 105742 ± 2% stress-ng.time.minor_page_faults > 8940 +32.8% 11872 ± 2% stress-ng.time.percent_of_cpu_this_job_got > 5268 +33.4% 7027 ± 2% stress-ng.time.system_time > 278.70 +21.5% 338.70 ± 2% stress-ng.time.user_time > 2.554e+08 +13.3% 2.894e+08 stress-ng.time.voluntary_context_switches > 1.283e+08 +113.5% 2.74e+08 ± 6% stress-ng.vm-rw.ops > 2139049 +113.5% 4567054 ± 6% stress-ng.vm-rw.ops_per_sec > 39411 ± 34% +56.3% 61612 ± 24% numa-meminfo.node1.Mapped > 5013 -22.5% 3883 ± 4% uptime.idle > 1.798e+09 -60.3% 7.135e+08 ± 21% cpuidle..time > 1.701e+08 -87.3% 21598951 ± 90% cpuidle..usage > 75821 ± 2% -11.6% 67063 ± 5% meminfo.Active > 75821 ± 2% -11.6% 67063 ± 5% meminfo.Active(anon) > 81710 ± 2% +20.1% 98158 ± 3% meminfo.Mapped > 26.00 -59.6% 10.50 ± 18% vmstat.cpu.id > 112.00 +10.9% 124.17 vmstat.procs.r > 6561639 +31.6% 8634043 ± 2% vmstat.system.cs > 990604 -62.4% 372118 ± 18% vmstat.system.in > 24.13 -16.1 8.03 ± 23% mpstat.cpu.all.idle% This indicates that the SIS scan efficiency is largely improved, which is in line with our expectations. > 2.71 -1.6 1.11 ± 10% mpstat.cpu.all.irq% > 0.17 ± 6% -0.1 0.06 ± 30% mpstat.cpu.all.soft% > 69.33 +17.4 86.71 ± 2% mpstat.cpu.all.sys% > 3.66 +0.4 4.09 mpstat.cpu.all.usr% > 2.024e+09 +93.3% 3.912e+09 ± 16% numa-vmstat.node0.nr_foll_pin_acquired > 2.024e+09 +93.3% 3.912e+09 ± 16% numa-vmstat.node0.nr_foll_pin_released > 2.043e+09 ± 2% +119.0% 4.473e+09 numa-vmstat.node1.nr_foll_pin_acquired > 2.043e+09 ± 2% +119.0% 4.473e+09 numa-vmstat.node1.nr_foll_pin_released > 9865 ± 34% +54.1% 15201 ± 23% numa-vmstat.node1.nr_mapped > 18954 ± 2% -11.5% 16767 ± 5% proc-vmstat.nr_active_anon > 4.062e+09 +107.3% 8.419e+09 ± 7% proc-vmstat.nr_foll_pin_acquired > 4.062e+09 +107.3% 8.419e+09 ± 7% proc-vmstat.nr_foll_pin_released > 87380 +5.3% 92039 proc-vmstat.nr_inactive_anon > 24453 -3.2% 23658 proc-vmstat.nr_kernel_stack > 20437 ± 2% +19.6% 24443 ± 3% proc-vmstat.nr_mapped > 18954 ± 2% -11.5% 16767 ± 5% proc-vmstat.nr_zone_active_anon > 87380 +5.3% 92039 proc-vmstat.nr_zone_inactive_anon > 108777 ± 4% -17.2% 90014 proc-vmstat.numa_hint_faults > 96756 ± 6% -17.6% 79691 ± 2% proc-vmstat.numa_hint_faults_local > 490607 -4.4% 469155 proc-vmstat.pgfault > 80.85 +10.9 91.75 turbostat.Busy% > 3221 -5.0% 3060 turbostat.Bzy_MHz > 77259218 ± 3% -87.0% 10057388 ± 92% turbostat.C1 > 6.74 ± 2% -5.9 0.85 ± 90% turbostat.C1% > 92212921 -87.8% 11243535 ± 91% turbostat.C1E > 12.00 ± 22% -6.6 5.42 ± 57% turbostat.C1E% and this. > 16.39 ± 16% -62.0% 6.24 ± 55% turbostat.CPU%c1 > 0.16 ± 3% +74.7% 0.29 ± 6% turbostat.IPC > 65322725 -62.5% 24502370 ± 18% turbostat.IRQ > 339708 -86.5% 45941 ± 88% turbostat.POLL > 0.05 -0.0 0.01 ± 82% turbostat.POLL% > 165121 ± 23% -100.0% 39.19 ±101% sched_debug.cfs_rq:/.MIN_vruntime.avg > 2462709 -99.9% 3407 ±102% sched_debug.cfs_rq:/.MIN_vruntime.max > 607348 ± 11% -99.9% 348.57 ±100% sched_debug.cfs_rq:/.MIN_vruntime.stddev > 0.56 ± 4% +11.8% 0.62 ± 3% sched_debug.cfs_rq:/.h_nr_running.avg > 2.58 ± 13% -38.7% 1.58 ± 11% sched_debug.cfs_rq:/.h_nr_running.max > 0.54 ± 9% -39.7% 0.33 ± 6% sched_debug.cfs_rq:/.h_nr_running.stddev > 165121 ± 23% -100.0% 39.19 ±101% sched_debug.cfs_rq:/.max_vruntime.avg > 2462709 -99.9% 3407 ±102% sched_debug.cfs_rq:/.max_vruntime.max > 607348 ± 11% -99.9% 348.57 ±100% sched_debug.cfs_rq:/.max_vruntime.stddev > 2439879 +43.2% 3493834 ± 4% sched_debug.cfs_rq:/.min_vruntime.avg > 2485561 +49.1% 3705888 sched_debug.cfs_rq:/.min_vruntime.max > 2129935 +34.5% 2865147 ± 2% sched_debug.cfs_rq:/.min_vruntime.min > 35480 ± 17% +324.2% 150497 ± 59% sched_debug.cfs_rq:/.min_vruntime.stddev > 0.43 ± 3% +27.9% 0.55 sched_debug.cfs_rq:/.nr_running.avg > 0.35 ± 5% -57.2% 0.15 ± 4% sched_debug.cfs_rq:/.nr_running.stddev > 2186 ± 15% -27.9% 1575 ± 11% sched_debug.cfs_rq:/.runnable_avg.max > 152.08 ± 6% +134.5% 356.58 ± 31% sched_debug.cfs_rq:/.runnable_avg.min > 399.32 ± 4% -50.5% 197.69 ± 8% sched_debug.cfs_rq:/.runnable_avg.stddev > 25106 ± 50% +1121.1% 306577 ± 66% sched_debug.cfs_rq:/.spread0.max > 35510 ± 17% +323.3% 150305 ± 59% sched_debug.cfs_rq:/.spread0.stddev > 545.95 ± 3% +16.4% 635.59 sched_debug.cfs_rq:/.util_avg.avg > 1726 ± 15% -26.7% 1266 ± 14% sched_debug.cfs_rq:/.util_avg.max > 154.67 ± 2% +112.9% 329.33 ± 30% sched_debug.cfs_rq:/.util_avg.min > 317.35 ± 4% -43.1% 180.53 ± 10% sched_debug.cfs_rq:/.util_avg.stddev > 192.70 ± 6% +104.5% 393.98 ± 7% sched_debug.cfs_rq:/.util_est_enqueued.avg > 5359 ± 4% -26.1% 3958 ± 8% sched_debug.cpu.avg_idle.min > 4.69 ± 7% +136.0% 11.07 ± 5% sched_debug.cpu.clock.stddev > 2380 ± 4% +31.0% 3117 sched_debug.cpu.curr->pid.avg > 1818 ± 3% -65.9% 620.26 ± 8% sched_debug.cpu.curr->pid.stddev > 0.00 ± 8% +59.7% 0.00 ± 10% sched_debug.cpu.next_balance.stddev > 2.58 ± 17% -41.9% 1.50 sched_debug.cpu.nr_running.max > 0.52 ± 9% -43.2% 0.29 ± 5% sched_debug.cpu.nr_running.stddev > 1610935 +31.3% 2115112 ± 2% sched_debug.cpu.nr_switches.avg > 1661619 +34.5% 2234069 sched_debug.cpu.nr_switches.max > 1415677 ± 3% +20.3% 1702445 sched_debug.cpu.nr_switches.min > 30576 ± 26% +151.6% 76923 ± 37% sched_debug.cpu.nr_switches.stddev > 25.47 -91.3% 2.21 ± 69% perf-stat.i.MPKI > 3.342e+10 +84.7% 6.172e+10 ± 5% perf-stat.i.branch-instructions > 0.58 -0.3 0.33 ± 5% perf-stat.i.branch-miss-rate% > 1.667e+08 -13.2% 1.448e+08 ± 2% perf-stat.i.branch-misses > 0.63 ± 17% +4.8 5.42 ± 39% perf-stat.i.cache-miss-rate% > 18939524 ± 4% -46.6% 10109353 ± 18% perf-stat.i.cache-misses > 4.422e+09 -87.1% 5.724e+08 ± 77% perf-stat.i.cache-references > 6897069 +30.8% 9023752 ± 2% perf-stat.i.context-switches > 2.04 -43.3% 1.16 ± 5% perf-stat.i.cpi > 3.523e+11 +3.8% 3.656e+11 perf-stat.i.cpu-cycles > 2322589 -86.6% 310934 ± 93% perf-stat.i.cpu-migrations > 18560 ± 4% +113.2% 39578 ± 15% perf-stat.i.cycles-between-cache-misses > 0.20 -0.2 0.02 ± 70% perf-stat.i.dTLB-load-miss-rate% > 85472762 -87.2% 10962661 ± 82% perf-stat.i.dTLB-load-misses > 4.266e+10 +83.8% 7.841e+10 ± 5% perf-stat.i.dTLB-loads > 0.10 ± 4% -0.1 0.01 ± 72% perf-stat.i.dTLB-store-miss-rate% > 25396322 ± 4% -86.5% 3437369 ± 90% perf-stat.i.dTLB-store-misses > 2.483e+10 +85.2% 4.598e+10 ± 5% perf-stat.i.dTLB-stores > 1.699e+11 +85.8% 3.157e+11 ± 5% perf-stat.i.instructions > 0.50 +73.4% 0.87 ± 4% perf-stat.i.ipc > 2.75 +3.8% 2.86 perf-stat.i.metric.GHz > 822.90 +77.2% 1458 ± 5% perf-stat.i.metric.M/sec > 5691 -3.4% 5500 perf-stat.i.minor-faults > 91.09 +4.6 95.71 perf-stat.i.node-load-miss-rate% > 334087 ± 17% -67.4% 109033 ± 18% perf-stat.i.node-loads > 70.09 +17.6 87.68 ± 6% perf-stat.i.node-store-miss-rate% > 1559730 ± 5% -64.9% 548115 ± 56% perf-stat.i.node-stores > 5704 -3.3% 5513 perf-stat.i.page-faults > 26.03 -92.7% 1.89 ± 83% perf-stat.overall.MPKI > 0.50 -0.3 0.24 ± 8% perf-stat.overall.branch-miss-rate% > 0.43 ± 3% +2.5 2.91 ± 60% perf-stat.overall.cache-miss-rate% > 2.08 -44.0% 1.16 ± 5% perf-stat.overall.cpi > 18664 ± 4% +100.4% 37402 ± 16% perf-stat.overall.cycles-between-cache-misses > 0.20 -0.2 0.01 ± 87% perf-stat.overall.dTLB-load-miss-rate% > 0.10 ± 4% -0.1 0.01 ± 96% perf-stat.overall.dTLB-store-miss-rate% > 0.48 +79.1% 0.86 ± 4% perf-stat.overall.ipc > 91.02 +5.1 96.07 perf-stat.overall.node-load-miss-rate% > 70.91 +17.6 88.54 ± 6% perf-stat.overall.node-store-miss-rate% > 3.289e+10 +85.0% 6.085e+10 ± 5% perf-stat.ps.branch-instructions > 1.641e+08 -13.1% 1.425e+08 ± 2% perf-stat.ps.branch-misses > 18633656 ± 4% -46.7% 9931368 ± 18% perf-stat.ps.cache-misses > 4.354e+09 -87.1% 5.613e+08 ± 77% perf-stat.ps.cache-references > 6788892 +31.0% 8894592 ± 2% perf-stat.ps.context-switches > 3.47e+11 +3.9% 3.604e+11 perf-stat.ps.cpu-cycles > 2286778 -86.7% 304327 ± 94% perf-stat.ps.cpu-migrations > 84173329 -87.2% 10770448 ± 82% perf-stat.ps.dTLB-load-misses > 4.198e+10 +84.1% 7.73e+10 ± 5% perf-stat.ps.dTLB-loads > 25001705 ± 4% -86.5% 3364501 ± 91% perf-stat.ps.dTLB-store-misses > 2.444e+10 +85.5% 4.533e+10 ± 5% perf-stat.ps.dTLB-stores > 1.673e+11 +86.1% 3.112e+11 ± 5% perf-stat.ps.instructions > 12.40 -1.5% 12.22 perf-stat.ps.major-faults > 5543 -3.9% 5329 perf-stat.ps.minor-faults > 332272 ± 17% -66.0% 112911 ± 16% perf-stat.ps.node-loads > 1533930 ± 5% -65.2% 534337 ± 57% perf-stat.ps.node-stores > 5556 -3.9% 5341 perf-stat.ps.page-faults > 1.065e+13 +86.7% 1.988e+13 ± 5% perf-stat.total.instructions > 18.10 -16.2 1.91 ±142% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify > 17.94 -16.1 1.88 ±142% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify > 17.93 -16.1 1.88 ±142% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify > 17.90 -16.0 1.88 ±142% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify > 15.83 -8.0 7.86 ± 20% perf-profile.calltrace.cycles-pp.read > 13.06 -8.0 5.11 ± 30% perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64 > 13.21 -7.9 5.30 ± 29% perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe > 13.58 -7.9 5.68 ± 27% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read > 14.72 -7.9 6.86 ± 22% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read > 14.51 -7.8 6.73 ± 22% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read > 13.77 -7.5 6.23 ± 23% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read > 9.90 -7.1 2.83 ± 47% perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.new_sync_read.vfs_read > 9.96 -7.0 2.92 ± 45% perf-profile.calltrace.cycles-pp.schedule.pipe_read.new_sync_read.vfs_read.ksys_read > 7.84 -6.9 0.94 ±142% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify > 7.44 -6.8 0.63 ±142% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify > 7.07 -6.2 0.85 ±142% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary > 7.03 -6.2 0.84 ±142% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry > 8.60 -5.3 3.30 ± 44% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write > 8.75 -5.3 3.49 ± 41% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write > 10.84 -5.3 5.58 ± 27% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write > 10.53 -5.2 5.29 ± 28% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write > 6.70 -5.2 1.49 ± 62% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_read.new_sync_read > 10.02 -5.2 4.83 ± 30% perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64 > 5.64 -5.2 0.48 ±142% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary > 10.08 -5.1 4.98 ± 29% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe > 8.21 -5.1 3.14 ± 44% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write > 8.25 -5.1 3.20 ± 44% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.new_sync_write > 5.11 -4.7 0.40 ±141% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle.cpu_startup_entry > 5.07 -4.7 0.40 ±141% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.sched_ttwu_pending.flush_smp_call_function_queue.do_idle > 5.19 -4.6 0.55 ±141% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.pipe_read > 20.28 -4.6 15.69 ± 5% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.process_vm_rw_single_vec.process_vm_rw_core > 20.55 -4.3 16.22 ± 5% perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw > 10.91 -3.7 7.17 ± 17% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write > 24.78 -3.7 21.12 ± 2% perf-profile.calltrace.cycles-pp.copy_page_to_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_readv > 10.95 -3.6 7.30 ± 16% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write > 11.15 -3.0 8.18 ± 13% perf-profile.calltrace.cycles-pp.write > 1.03 ± 4% -0.4 0.62 ± 14% perf-profile.calltrace.cycles-pp.stress_vm_child > 0.76 ± 4% +0.5 1.22 ± 10% perf-profile.calltrace.cycles-pp.stress_vm_rw > 0.63 +0.5 1.15 ± 23% perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common > 0.53 ± 2% +0.5 1.06 ± 9% perf-profile.calltrace.cycles-pp.__might_fault.copy_page_to_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw > 0.64 +0.5 1.18 ± 22% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock > 0.00 +0.6 0.60 ± 7% perf-profile.calltrace.cycles-pp.__might_resched.copy_page_to_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw > 0.00 +1.0 0.96 ± 15% perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode > 0.00 +1.0 1.02 ± 17% perf-profile.calltrace.cycles-pp.mod_node_page_state.gup_put_folio.unpin_user_pages.process_vm_rw_single_vec.process_vm_rw_core > 0.00 +1.0 1.02 ± 10% perf-profile.calltrace.cycles-pp.__might_fault.copy_page_from_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw > 0.00 +1.0 1.03 ± 15% perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 > 0.00 +1.0 1.04 ± 17% perf-profile.calltrace.cycles-pp.mod_node_page_state.gup_put_folio.unpin_user_pages_dirty_lock.process_vm_rw_single_vec.process_vm_rw_core > 0.00 +1.1 1.10 ± 16% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe > 1.24 ± 2% +1.4 2.60 ± 11% perf-profile.calltrace.cycles-pp._raw_spin_lock.follow_page_pte.__get_user_pages.__get_user_pages_remote.process_vm_rw_single_vec > 0.00 +1.4 1.40 ± 16% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.write > 0.00 +1.5 1.46 ± 16% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.write > 1.04 +1.5 2.54 ± 14% perf-profile.calltrace.cycles-pp.gup_put_folio.unpin_user_pages.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw > 0.99 ± 2% +1.6 2.57 ± 14% perf-profile.calltrace.cycles-pp.gup_put_folio.unpin_user_pages_dirty_lock.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw > 0.00 +1.7 1.69 ± 11% perf-profile.calltrace.cycles-pp.follow_pud_mask.__get_user_pages.__get_user_pages_remote.process_vm_rw_single_vec.process_vm_rw_core > 1.35 +1.8 3.20 ± 14% perf-profile.calltrace.cycles-pp.unpin_user_pages.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_readv > 0.00 +2.0 1.97 ± 10% perf-profile.calltrace.cycles-pp.follow_page_mask.__get_user_pages.__get_user_pages_remote.process_vm_rw_single_vec.process_vm_rw_core > 1.27 ± 2% +2.0 3.30 ± 14% perf-profile.calltrace.cycles-pp.unpin_user_pages_dirty_lock.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_writev > 0.00 +2.1 2.12 ± 18% perf-profile.calltrace.cycles-pp.mod_node_page_state.try_grab_page.follow_page_pte.__get_user_pages.__get_user_pages_remote > 0.00 +2.3 2.30 ± 11% perf-profile.calltrace.cycles-pp.follow_pmd_mask.__get_user_pages.__get_user_pages_remote.process_vm_rw_single_vec.process_vm_rw_core > 3.16 ± 2% +2.4 5.51 ± 11% perf-profile.calltrace.cycles-pp.try_grab_page.follow_page_pte.__get_user_pages.__get_user_pages_remote.process_vm_rw_single_vec > 32.57 +5.2 37.78 ± 3% perf-profile.calltrace.cycles-pp.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_readv.do_syscall_64 > 33.67 +5.6 39.24 ± 3% perf-profile.calltrace.cycles-pp.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_readv.do_syscall_64.entry_SYSCALL_64_after_hwframe > 6.24 +6.1 12.34 ± 10% perf-profile.calltrace.cycles-pp.__get_user_pages_remote.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_readv > 34.23 +6.2 40.39 ± 3% perf-profile.calltrace.cycles-pp.process_vm_rw.__x64_sys_process_vm_readv.do_syscall_64.entry_SYSCALL_64_after_hwframe.process_vm_readv > 34.22 +6.2 40.42 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_process_vm_readv.do_syscall_64.entry_SYSCALL_64_after_hwframe.process_vm_readv > 34.39 +6.3 40.68 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.process_vm_readv > 34.49 +6.4 40.88 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.process_vm_readv > 34.87 +6.6 41.43 ± 3% perf-profile.calltrace.cycles-pp.process_vm_readv > 6.26 +6.6 12.83 ± 11% perf-profile.calltrace.cycles-pp.follow_page_pte.__get_user_pages.__get_user_pages_remote.process_vm_rw_single_vec.process_vm_rw_core > 7.43 +6.6 14.02 ± 8% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.copy_page_from_iter.process_vm_rw_single_vec.process_vm_rw_core > 7.70 +6.9 14.64 ± 8% perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw > 4.80 ± 2% +7.2 11.95 ± 12% perf-profile.calltrace.cycles-pp.__get_user_pages_remote.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_writev > 9.52 +9.3 18.86 ± 8% perf-profile.calltrace.cycles-pp.copy_page_from_iter.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_writev > 10.80 +12.9 23.71 ± 11% perf-profile.calltrace.cycles-pp.__get_user_pages.__get_user_pages_remote.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw > 16.02 ± 2% +19.0 35.05 ± 10% perf-profile.calltrace.cycles-pp.process_vm_rw_single_vec.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_writev.do_syscall_64 > 16.70 ± 2% +19.7 36.44 ± 10% perf-profile.calltrace.cycles-pp.process_vm_rw_core.process_vm_rw.__x64_sys_process_vm_writev.do_syscall_64.entry_SYSCALL_64_after_hwframe > 17.15 ± 2% +20.4 37.51 ± 10% perf-profile.calltrace.cycles-pp.process_vm_rw.__x64_sys_process_vm_writev.do_syscall_64.entry_SYSCALL_64_after_hwframe.process_vm_writev > 17.17 ± 2% +20.4 37.55 ± 10% perf-profile.calltrace.cycles-pp.__x64_sys_process_vm_writev.do_syscall_64.entry_SYSCALL_64_after_hwframe.process_vm_writev > 17.26 ± 2% +20.5 37.72 ± 10% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.process_vm_writev > 17.31 ± 2% +20.5 37.83 ± 10% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.process_vm_writev > 17.60 ± 2% +20.8 38.42 ± 10% perf-profile.calltrace.cycles-pp.process_vm_writev > 18.10 -16.2 1.92 ±141% perf-profile.children.cycles-pp.secondary_startup_64_no_verify > 18.10 -16.2 1.92 ±141% perf-profile.children.cycles-pp.cpu_startup_entry > 18.08 -16.2 1.92 ±141% perf-profile.children.cycles-pp.do_idle > 17.94 -16.0 1.89 ±141% perf-profile.children.cycles-pp.start_secondary > 16.00 -8.0 8.02 ± 19% perf-profile.children.cycles-pp.read > 12.02 -7.9 4.10 ± 37% perf-profile.children.cycles-pp.__schedule > 13.10 -7.9 5.18 ± 30% perf-profile.children.cycles-pp.pipe_read > 13.22 -7.9 5.31 ± 29% perf-profile.children.cycles-pp.new_sync_read > 13.60 -7.9 5.70 ± 27% perf-profile.children.cycles-pp.vfs_read > 13.78 -7.5 6.25 ± 23% perf-profile.children.cycles-pp.ksys_read > 7.62 -7.0 0.65 ±142% perf-profile.children.cycles-pp.flush_smp_call_function_queue > 7.92 -7.0 0.96 ±142% perf-profile.children.cycles-pp.cpuidle_idle_call > 7.18 -6.6 0.59 ±142% perf-profile.children.cycles-pp.sched_ttwu_pending > 7.14 -6.3 0.86 ±142% perf-profile.children.cycles-pp.cpuidle_enter > 7.24 -6.3 0.98 ±124% perf-profile.children.cycles-pp.update_cfs_group > 7.12 -6.3 0.86 ±142% perf-profile.children.cycles-pp.cpuidle_enter_state > 9.97 -6.0 3.96 ± 29% perf-profile.children.cycles-pp.schedule > 6.52 -5.7 0.79 ±142% perf-profile.children.cycles-pp.mwait_idle_with_hints > 7.11 -5.4 1.69 ± 57% perf-profile.children.cycles-pp.ttwu_do_activate > 7.07 -5.4 1.66 ± 58% perf-profile.children.cycles-pp.enqueue_task_fair > 8.61 -5.3 3.30 ± 44% perf-profile.children.cycles-pp.__wake_up_common > 10.86 -5.2 5.61 ± 26% perf-profile.children.cycles-pp.ksys_write > 8.76 -5.2 3.51 ± 41% perf-profile.children.cycles-pp.__wake_up_common_lock > 10.55 -5.2 5.32 ± 28% perf-profile.children.cycles-pp.vfs_write > 6.72 -5.2 1.50 ± 61% perf-profile.children.cycles-pp.dequeue_task_fair > 10.04 -5.2 4.88 ± 30% perf-profile.children.cycles-pp.pipe_write > 10.10 -5.1 5.00 ± 29% perf-profile.children.cycles-pp.new_sync_write > 8.26 -5.1 3.20 ± 44% perf-profile.children.cycles-pp.autoremove_wake_function > 8.22 -5.0 3.17 ± 44% perf-profile.children.cycles-pp.try_to_wake_up > 5.62 -4.7 0.94 ± 62% perf-profile.children.cycles-pp.enqueue_entity > 21.11 -4.4 16.69 ± 5% perf-profile.children.cycles-pp.copyout > 5.21 -4.4 0.84 ± 70% perf-profile.children.cycles-pp.dequeue_entity > 5.50 -4.3 1.25 ± 48% perf-profile.children.cycles-pp.update_load_avg > 25.20 -3.0 22.21 ± 2% perf-profile.children.cycles-pp.copy_page_to_iter > 11.21 -2.9 8.33 ± 13% perf-profile.children.cycles-pp.write > 3.29 -2.5 0.83 ± 78% perf-profile.children.cycles-pp.select_task_rq > 3.22 -2.4 0.78 ± 83% perf-profile.children.cycles-pp.select_task_rq_fair > 2.78 -2.2 0.62 ± 93% perf-profile.children.cycles-pp.select_idle_sibling > 1.80 ± 2% -1.5 0.29 ±137% perf-profile.children.cycles-pp.available_idle_cpu > 0.87 -0.6 0.22 ± 57% perf-profile.children.cycles-pp.finish_task_switch > 0.98 -0.5 0.45 ± 23% perf-profile.children.cycles-pp._raw_spin_lock_irqsave > 0.94 -0.5 0.41 ± 27% perf-profile.children.cycles-pp.prepare_to_wait_event > 0.60 ± 2% -0.5 0.13 ± 76% perf-profile.children.cycles-pp.switch_mm_irqs_off > 1.04 ± 4% -0.4 0.63 ± 14% perf-profile.children.cycles-pp.stress_vm_child > 0.58 -0.3 0.29 ± 28% perf-profile.children.cycles-pp.update_rq_clock > 0.69 -0.3 0.40 ± 19% perf-profile.children.cycles-pp.prepare_task_switch > 0.89 ± 3% -0.3 0.62 ± 13% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt > 0.58 -0.3 0.31 ± 21% perf-profile.children.cycles-pp.__switch_to_asm > 0.52 -0.2 0.28 ± 21% perf-profile.children.cycles-pp.___perf_sw_event > 0.76 ± 4% -0.2 0.56 ± 12% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt > 0.28 ± 3% -0.2 0.09 ± 52% perf-profile.children.cycles-pp._find_next_bit > 0.50 -0.2 0.33 ± 13% perf-profile.children.cycles-pp.security_file_permission > 0.26 ± 2% -0.2 0.09 ± 41% perf-profile.children.cycles-pp.task_tick_fair > 0.24 ± 6% -0.2 0.08 ± 57% perf-profile.children.cycles-pp.__irq_exit_rcu > 0.48 -0.2 0.32 ± 5% perf-profile.children.cycles-pp.set_next_entity > 0.42 ± 4% -0.1 0.27 ± 13% perf-profile.children.cycles-pp.__hrtimer_run_queues > 0.30 ± 3% -0.1 0.16 ± 18% perf-profile.children.cycles-pp.scheduler_tick > 0.37 ± 4% -0.1 0.24 ± 13% perf-profile.children.cycles-pp.tick_sched_timer > 0.35 ± 4% -0.1 0.22 ± 14% perf-profile.children.cycles-pp.tick_sched_handle > 0.34 ± 4% -0.1 0.22 ± 15% perf-profile.children.cycles-pp.update_process_times > 0.25 ± 3% -0.1 0.12 ± 40% perf-profile.children.cycles-pp.find_vma > 0.19 ± 7% -0.1 0.07 ± 56% perf-profile.children.cycles-pp.__softirqentry_text_start > 0.68 -0.1 0.56 ± 5% perf-profile.children.cycles-pp.mutex_lock > 0.26 ± 3% -0.1 0.14 ± 34% perf-profile.children.cycles-pp.find_extend_vma > 0.38 ± 2% -0.1 0.26 ± 10% perf-profile.children.cycles-pp.apparmor_file_permission > 0.37 -0.1 0.27 ± 9% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq > 0.34 -0.1 0.25 ± 10% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template > 0.24 ± 3% -0.1 0.16 ± 9% perf-profile.children.cycles-pp.sched_clock_cpu > 0.17 ± 4% -0.1 0.09 ± 33% perf-profile.children.cycles-pp.vmacache_find > 0.64 -0.1 0.56 ± 6% perf-profile.children.cycles-pp.switch_fpu_return > 0.53 ± 4% -0.1 0.46 ± 8% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt > 0.53 ± 4% -0.1 0.46 ± 8% perf-profile.children.cycles-pp.hrtimer_interrupt > 0.19 ± 2% -0.1 0.13 ± 9% perf-profile.children.cycles-pp.native_sched_clock > 0.57 -0.1 0.50 ± 5% perf-profile.children.cycles-pp.__switch_to > 0.27 -0.0 0.22 ± 2% perf-profile.children.cycles-pp.mutex_unlock > 0.10 ± 5% -0.0 0.06 ± 17% perf-profile.children.cycles-pp.anon_pipe_buf_release > 0.13 ± 2% -0.0 0.10 ± 9% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore > 0.17 ± 2% -0.0 0.14 ± 8% perf-profile.children.cycles-pp.perf_tp_event > 0.08 ± 4% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.__list_add_valid > 0.10 -0.0 0.09 perf-profile.children.cycles-pp.finish_wait > 0.14 +0.0 0.16 ± 3% perf-profile.children.cycles-pp.atime_needs_update > 0.12 ± 4% +0.0 0.15 ± 2% perf-profile.children.cycles-pp.file_update_time > 0.06 ± 6% +0.0 0.09 ± 7% perf-profile.children.cycles-pp.__rdgsbase_inactive > 0.10 ± 6% +0.0 0.13 ± 5% perf-profile.children.cycles-pp.__wrgsbase_inactive > 0.75 +0.0 0.79 perf-profile.children.cycles-pp.pick_next_task_fair > 0.05 +0.0 0.09 ± 5% perf-profile.children.cycles-pp.pick_next_entity > 0.19 ± 2% +0.0 0.23 ± 5% perf-profile.children.cycles-pp.down_read_killable > 0.02 ±141% +0.0 0.06 ± 11% perf-profile.children.cycles-pp.perf_trace_sched_switch > 0.05 +0.0 0.10 ± 6% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack > 0.02 ± 99% +0.1 0.08 ± 6% perf-profile.children.cycles-pp.kmalloc_slab > 0.14 ± 2% +0.1 0.20 perf-profile.children.cycles-pp.down_read > 0.00 +0.1 0.06 ± 8% perf-profile.children.cycles-pp.resched_curr > 0.02 ± 99% +0.1 0.08 ± 8% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime > 0.16 ± 3% +0.1 0.21 ± 2% perf-profile.children.cycles-pp.mmput > 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64 > 0.13 ± 2% +0.1 0.19 ± 6% perf-profile.children.cycles-pp.get_task_mm > 0.00 +0.1 0.07 ± 7% perf-profile.children.cycles-pp.idr_find > 0.24 ± 2% +0.1 0.31 ± 4% perf-profile.children.cycles-pp.__update_load_avg_se > 0.02 ±141% +0.1 0.09 ± 10% perf-profile.children.cycles-pp.__calc_delta > 0.66 +0.1 0.73 ± 2% perf-profile.children.cycles-pp.update_curr > 0.04 ± 44% +0.1 0.12 ± 9% perf-profile.children.cycles-pp.memcg_slab_free_hook > 0.00 +0.1 0.07 ± 18% perf-profile.children.cycles-pp.cpumask_next_and > 0.00 +0.1 0.08 ± 10% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare > 0.08 ± 4% +0.1 0.16 ± 10% perf-profile.children.cycles-pp.up_read > 0.07 ± 11% +0.1 0.15 ± 8% perf-profile.children.cycles-pp.clockevents_program_event > 0.13 +0.1 0.22 ± 3% perf-profile.children.cycles-pp.ttwu_do_wakeup > 0.06 ± 6% +0.1 0.15 ± 9% perf-profile.children.cycles-pp.current_time > 0.00 +0.1 0.09 ± 10% perf-profile.children.cycles-pp.check_stack_object > 0.11 ± 4% +0.1 0.20 ± 4% perf-profile.children.cycles-pp.check_preempt_curr > 0.15 ± 4% +0.1 0.26 ± 6% perf-profile.children.cycles-pp.os_xsave > 0.12 ± 4% +0.1 0.23 ± 9% perf-profile.children.cycles-pp.syscall_enter_from_user_mode > 0.16 ± 3% +0.1 0.27 ± 9% perf-profile.children.cycles-pp.reweight_entity > 0.61 ± 2% +0.1 0.75 ± 2% perf-profile.children.cycles-pp.find_get_task_by_vpid > 0.19 ± 5% +0.1 0.33 ± 6% perf-profile.children.cycles-pp.__radix_tree_lookup > 0.10 ± 3% +0.2 0.25 ± 10% perf-profile.children.cycles-pp.__check_object_size > 0.13 ± 3% +0.2 0.29 ± 9% perf-profile.children.cycles-pp.syscall_return_via_sysret > 0.11 ± 6% +0.2 0.26 ± 11% perf-profile.children.cycles-pp.follow_huge_addr > 0.37 ± 2% +0.2 0.53 ± 6% perf-profile.children.cycles-pp.mm_access > 0.00 +0.2 0.17 ± 9% perf-profile.children.cycles-pp.check_preempt_wakeup > 0.00 +0.2 0.17 ± 14% perf-profile.children.cycles-pp.put_prev_entity > 0.15 ± 3% +0.2 0.39 ± 12% perf-profile.children.cycles-pp.pud_huge > 0.14 ± 3% +0.2 0.39 ± 11% perf-profile.children.cycles-pp.mark_page_accessed > 0.19 ± 3% +0.3 0.44 ± 9% perf-profile.children.cycles-pp.kfree > 0.19 ± 3% +0.3 0.50 ± 12% perf-profile.children.cycles-pp.pmd_huge > 0.31 ± 3% +0.3 0.65 ± 9% perf-profile.children.cycles-pp.__kmalloc > 0.28 ± 3% +0.4 0.67 ± 10% perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack > 0.26 ± 3% +0.4 0.70 ± 12% perf-profile.children.cycles-pp.vm_normal_page > 0.28 ± 2% +0.5 0.73 ± 11% perf-profile.children.cycles-pp.folio_mark_accessed > 0.40 ± 2% +0.5 0.86 ± 10% perf-profile.children.cycles-pp.__entry_text_start > 0.77 ± 4% +0.5 1.24 ± 10% perf-profile.children.cycles-pp.stress_vm_rw > 2.54 +0.6 3.10 ± 4% perf-profile.children.cycles-pp._raw_spin_lock > 0.44 ± 3% +0.6 1.02 ± 10% perf-profile.children.cycles-pp.__import_iovec > 0.46 ± 3% +0.6 1.05 ± 10% perf-profile.children.cycles-pp.import_iovec > 0.57 ± 2% +0.7 1.26 ± 9% perf-profile.children.cycles-pp._copy_from_user > 0.47 ± 3% +0.7 1.18 ± 11% perf-profile.children.cycles-pp.rcu_all_qs > 1.72 +0.9 2.58 ± 6% perf-profile.children.cycles-pp.__cond_resched > 0.75 ± 2% +1.0 1.71 ± 9% perf-profile.children.cycles-pp.iovec_from_user > 0.00 +1.1 1.11 ± 16% perf-profile.children.cycles-pp.exit_to_user_mode_loop > 0.98 ± 2% +1.1 2.10 ± 10% perf-profile.children.cycles-pp.__might_sleep > 0.70 +1.1 1.84 ± 9% perf-profile.children.cycles-pp.exit_to_user_mode_prepare > 0.76 ± 3% +1.2 1.95 ± 11% perf-profile.children.cycles-pp.follow_pud_mask > 0.79 +1.3 2.06 ± 9% perf-profile.children.cycles-pp.syscall_exit_to_user_mode > 0.93 ± 2% +1.3 2.22 ± 10% perf-profile.children.cycles-pp.follow_page_mask > 1.30 ± 2% +1.5 2.80 ± 9% perf-profile.children.cycles-pp.__might_fault > 1.03 ± 2% +1.6 2.67 ± 11% perf-profile.children.cycles-pp.follow_pmd_mask > 2.83 +1.7 4.55 ± 7% perf-profile.children.cycles-pp.__might_resched > 1.37 +1.9 3.26 ± 14% perf-profile.children.cycles-pp.unpin_user_pages > 1.30 ± 2% +2.1 3.38 ± 14% perf-profile.children.cycles-pp.unpin_user_pages_dirty_lock > 3.29 ± 2% +2.6 5.87 ± 11% perf-profile.children.cycles-pp.try_grab_page > 29.19 +2.6 31.78 perf-profile.children.cycles-pp.copy_user_enhanced_fast_string > 1.40 +3.0 4.42 ± 17% perf-profile.children.cycles-pp.mod_node_page_state > 2.13 ± 2% +3.2 5.35 ± 14% perf-profile.children.cycles-pp.gup_put_folio > 34.26 +6.2 40.45 ± 3% perf-profile.children.cycles-pp.__x64_sys_process_vm_readv > 34.96 +6.7 41.64 ± 3% perf-profile.children.cycles-pp.process_vm_readv > 8.05 +7.0 15.10 ± 8% perf-profile.children.cycles-pp.copyin > 6.54 +7.1 13.60 ± 11% perf-profile.children.cycles-pp.follow_page_pte > 10.06 +9.7 19.72 ± 8% perf-profile.children.cycles-pp.copy_page_from_iter > 10.99 +13.2 24.22 ± 11% perf-profile.children.cycles-pp.__get_user_pages > 11.04 +13.3 24.31 ± 11% perf-profile.children.cycles-pp.__get_user_pages_remote > 77.15 +15.3 92.46 ± 2% perf-profile.children.cycles-pp.do_syscall_64 > 77.51 +15.4 92.94 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe > 17.18 ± 2% +20.4 37.56 ± 10% perf-profile.children.cycles-pp.__x64_sys_process_vm_writev > 17.69 ± 2% +20.9 38.64 ± 10% perf-profile.children.cycles-pp.process_vm_writev > 48.65 +24.3 72.98 ± 6% perf-profile.children.cycles-pp.process_vm_rw_single_vec > 50.39 +25.4 75.74 ± 6% perf-profile.children.cycles-pp.process_vm_rw_core > 51.40 +26.5 77.94 ± 6% perf-profile.children.cycles-pp.process_vm_rw > 7.23 -6.3 0.97 ±125% perf-profile.self.cycles-pp.update_cfs_group > 6.42 -5.6 0.78 ±142% perf-profile.self.cycles-pp.mwait_idle_with_hints > 4.82 -4.2 0.67 ± 85% perf-profile.self.cycles-pp.update_load_avg > 1.78 ± 2% -1.5 0.29 ±137% perf-profile.self.cycles-pp.available_idle_cpu > 0.96 -0.5 0.42 ± 26% perf-profile.self.cycles-pp._raw_spin_lock_irqsave > 0.98 ± 2% -0.5 0.45 ± 28% perf-profile.self.cycles-pp.__schedule > 0.59 ± 2% -0.5 0.13 ± 78% perf-profile.self.cycles-pp.switch_mm_irqs_off > 0.97 -0.4 0.61 ± 15% perf-profile.self.cycles-pp.stress_vm_child > 0.45 ± 2% -0.3 0.14 ± 58% perf-profile.self.cycles-pp.update_rq_clock > 0.58 -0.3 0.30 ± 22% perf-profile.self.cycles-pp.__switch_to_asm > 0.58 -0.3 0.33 ± 17% perf-profile.self.cycles-pp.pipe_read > 0.35 ± 3% -0.2 0.10 ± 59% perf-profile.self.cycles-pp.__wake_up_common > 0.47 -0.2 0.24 ± 23% perf-profile.self.cycles-pp.___perf_sw_event > 0.32 ± 2% -0.2 0.13 ± 39% perf-profile.self.cycles-pp.finish_task_switch > 0.38 ± 2% -0.2 0.21 ± 20% perf-profile.self.cycles-pp.prepare_to_wait_event > 0.25 ± 2% -0.2 0.08 ± 53% perf-profile.self.cycles-pp._find_next_bit > 0.31 ± 3% -0.2 0.14 ± 27% perf-profile.self.cycles-pp.enqueue_entity > 0.34 ± 15% -0.2 0.18 ± 12% perf-profile.self.cycles-pp.read > 0.26 ± 2% -0.2 0.11 ± 20% perf-profile.self.cycles-pp.try_to_wake_up > 0.29 ± 3% -0.2 0.13 ± 30% perf-profile.self.cycles-pp.prepare_task_switch > 0.45 ± 2% -0.1 0.31 ± 9% perf-profile.self.cycles-pp.mutex_lock > 0.26 -0.1 0.16 ± 14% perf-profile.self.cycles-pp.apparmor_file_permission > 0.13 ± 2% -0.1 0.03 ±103% perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template > 0.35 -0.1 0.26 ± 11% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq > 0.16 ± 3% -0.1 0.08 ± 22% perf-profile.self.cycles-pp.dequeue_entity > 0.56 -0.1 0.48 ± 5% perf-profile.self.cycles-pp.__switch_to > 0.15 ± 2% -0.1 0.08 ± 37% perf-profile.self.cycles-pp.vmacache_find > 0.18 ± 2% -0.1 0.11 ± 25% perf-profile.self.cycles-pp.select_idle_sibling > 0.19 ± 3% -0.1 0.13 ± 10% perf-profile.self.cycles-pp.native_sched_clock > 0.13 ± 3% -0.1 0.07 ± 17% perf-profile.self.cycles-pp.security_file_permission > 0.19 -0.1 0.13 ± 18% perf-profile.self.cycles-pp.enqueue_task_fair > 0.35 ± 2% -0.1 0.30 ± 8% perf-profile.self.cycles-pp.update_curr > 0.15 ± 2% -0.1 0.10 ± 12% perf-profile.self.cycles-pp.dequeue_task_fair > 0.26 -0.0 0.21 ± 2% perf-profile.self.cycles-pp.mutex_unlock > 0.38 -0.0 0.34 ± 3% perf-profile.self.cycles-pp.find_get_task_by_vpid > 0.09 ± 4% -0.0 0.06 ± 13% perf-profile.self.cycles-pp.anon_pipe_buf_release > 0.11 ± 4% -0.0 0.08 ± 10% perf-profile.self.cycles-pp.atime_needs_update > 0.21 ± 2% -0.0 0.19 ± 3% perf-profile.self.cycles-pp.vfs_read > 0.17 ± 2% -0.0 0.14 ± 3% perf-profile.self.cycles-pp.switch_fpu_return > 0.11 ± 4% -0.0 0.09 ± 10% perf-profile.self.cycles-pp.aa_file_perm > 0.08 ± 6% -0.0 0.06 ± 13% perf-profile.self.cycles-pp.select_task_rq > 0.05 +0.0 0.06 perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore > 0.06 +0.0 0.07 perf-profile.self.cycles-pp.set_next_entity > 0.07 ± 5% +0.0 0.09 perf-profile.self.cycles-pp.get_task_mm > 0.06 ± 9% +0.0 0.08 perf-profile.self.cycles-pp.__get_user_pages_remote > 0.06 ± 6% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.__rdgsbase_inactive > 0.10 ± 5% +0.0 0.13 ± 4% perf-profile.self.cycles-pp.__wrgsbase_inactive > 0.11 +0.0 0.15 ± 2% perf-profile.self.cycles-pp.pick_next_task_fair > 0.09 ± 9% +0.0 0.13 ± 8% perf-profile.self.cycles-pp.ktime_get > 0.03 ± 70% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.pick_next_entity > 0.08 ± 6% +0.1 0.13 ± 7% perf-profile.self.cycles-pp.vfs_write > 0.00 +0.1 0.06 ± 9% perf-profile.self.cycles-pp.resched_curr > 0.01 ±223% +0.1 0.06 ± 11% perf-profile.self.cycles-pp.perf_trace_sched_switch > 0.00 +0.1 0.06 ± 8% perf-profile.self.cycles-pp.put_prev_entity > 0.00 +0.1 0.06 ± 8% perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare > 0.00 +0.1 0.06 ± 8% perf-profile.self.cycles-pp.idr_find > 0.22 ± 2% +0.1 0.28 ± 4% perf-profile.self.cycles-pp.__update_load_avg_se > 0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.ksys_write > 0.00 +0.1 0.06 ± 14% perf-profile.self.cycles-pp.__wake_up_common_lock > 0.00 +0.1 0.06 ± 7% perf-profile.self.cycles-pp.kmalloc_slab > 0.10 ± 6% +0.1 0.16 ± 6% perf-profile.self.cycles-pp.write > 0.00 +0.1 0.07 ± 15% perf-profile.self.cycles-pp.check_preempt_wakeup > 0.08 ± 6% +0.1 0.15 ± 9% perf-profile.self.cycles-pp.up_read > 0.01 ±223% +0.1 0.08 ± 12% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime > 0.01 ±223% +0.1 0.08 ± 10% perf-profile.self.cycles-pp.__calc_delta > 0.00 +0.1 0.08 ± 10% perf-profile.self.cycles-pp.check_stack_object > 0.00 +0.1 0.08 ± 20% perf-profile.self.cycles-pp.exit_to_user_mode_loop > 0.02 ±141% +0.1 0.09 ± 10% perf-profile.self.cycles-pp.new_sync_write > 0.02 ±141% +0.1 0.09 ± 10% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack > 0.06 ± 8% +0.1 0.14 ± 11% perf-profile.self.cycles-pp.follow_huge_addr > 0.10 ± 4% +0.1 0.19 ± 10% perf-profile.self.cycles-pp.syscall_enter_from_user_mode > 0.00 +0.1 0.09 ± 12% perf-profile.self.cycles-pp.current_time > 0.06 +0.1 0.15 ± 11% perf-profile.self.cycles-pp.__import_iovec > 0.05 ± 8% +0.1 0.14 ± 13% perf-profile.self.cycles-pp.__check_object_size > 0.03 ± 70% +0.1 0.14 ± 15% perf-profile.self.cycles-pp.syscall_exit_to_user_mode > 0.19 ± 3% +0.1 0.30 ± 5% perf-profile.self.cycles-pp.process_vm_readv > 0.15 ± 3% +0.1 0.25 ± 5% perf-profile.self.cycles-pp.os_xsave > 0.00 +0.1 0.11 ± 9% perf-profile.self.cycles-pp.memcg_slab_free_hook > 0.07 ± 5% +0.1 0.18 ± 10% perf-profile.self.cycles-pp._copy_from_user > 0.15 ± 2% +0.1 0.26 ± 6% perf-profile.self.cycles-pp.pipe_write > 0.05 +0.1 0.16 ± 12% perf-profile.self.cycles-pp.exit_to_user_mode_prepare > 0.12 ± 3% +0.1 0.24 ± 8% perf-profile.self.cycles-pp.__entry_text_start > 0.00 +0.1 0.12 ± 14% perf-profile.self.cycles-pp.schedule > 0.14 ± 12% +0.1 0.26 ± 5% perf-profile.self.cycles-pp.process_vm_rw > 0.38 +0.1 0.51 ± 6% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe > 0.19 ± 7% +0.1 0.32 ± 6% perf-profile.self.cycles-pp.__radix_tree_lookup > 0.09 ± 4% +0.1 0.23 ± 10% perf-profile.self.cycles-pp.iovec_from_user > 0.09 +0.1 0.24 ± 11% perf-profile.self.cycles-pp.mark_page_accessed > 0.14 ± 3% +0.2 0.29 ± 9% perf-profile.self.cycles-pp.process_vm_rw_core > 0.13 ± 2% +0.2 0.28 ± 9% perf-profile.self.cycles-pp.syscall_return_via_sysret > 0.13 ± 2% +0.2 0.29 ± 10% perf-profile.self.cycles-pp.process_vm_writev > 0.10 ± 3% +0.2 0.26 ± 13% perf-profile.self.cycles-pp.pud_huge > 0.09 ± 5% +0.2 0.26 ± 13% perf-profile.self.cycles-pp.pmd_huge > 0.14 ± 3% +0.2 0.31 ± 11% perf-profile.self.cycles-pp.copyout > 0.14 ± 4% +0.2 0.31 ± 10% perf-profile.self.cycles-pp.kfree > 0.17 ± 3% +0.2 0.35 ± 9% perf-profile.self.cycles-pp.do_syscall_64 > 0.18 ± 3% +0.2 0.36 ± 9% perf-profile.self.cycles-pp.__kmalloc > 0.18 ± 2% +0.3 0.45 ± 10% perf-profile.self.cycles-pp.copyin > 0.24 ± 3% +0.3 0.50 ± 9% perf-profile.self.cycles-pp.__might_fault > 0.11 ± 5% +0.3 0.41 ± 22% perf-profile.self.cycles-pp.ksys_read > 0.30 ± 3% +0.3 0.65 ± 12% perf-profile.self.cycles-pp.unpin_user_pages > 0.22 ± 3% +0.4 0.58 ± 12% perf-profile.self.cycles-pp.vm_normal_page > 0.26 ± 4% +0.4 0.62 ± 11% perf-profile.self.cycles-pp.rcu_all_qs > 0.28 ± 2% +0.4 0.65 ± 10% perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack > 0.23 ± 3% +0.4 0.61 ± 12% perf-profile.self.cycles-pp.folio_mark_accessed > 0.28 ± 4% +0.5 0.73 ± 13% perf-profile.self.cycles-pp.unpin_user_pages_dirty_lock > 0.67 ± 4% +0.5 1.15 ± 12% perf-profile.self.cycles-pp.stress_vm_rw > 0.56 ± 2% +0.5 1.06 ± 8% perf-profile.self.cycles-pp.process_vm_rw_single_vec > 0.91 +0.6 1.55 ± 8% perf-profile.self.cycles-pp.__cond_resched > 0.81 ± 2% +0.9 1.72 ± 10% perf-profile.self.cycles-pp.__might_sleep > 0.61 ± 3% +0.9 1.55 ± 11% perf-profile.self.cycles-pp.follow_pud_mask > 1.94 +0.9 2.89 ± 7% perf-profile.self.cycles-pp._raw_spin_lock > 1.60 +1.0 2.57 ± 7% perf-profile.self.cycles-pp.copy_page_to_iter > 2.56 ± 2% +1.0 3.61 ± 7% perf-profile.self.cycles-pp.try_grab_page > 0.88 ± 2% +1.1 1.98 ± 10% perf-profile.self.cycles-pp.copy_page_from_iter > 0.82 ± 3% +1.1 1.95 ± 11% perf-profile.self.cycles-pp.follow_page_mask > 2.60 +1.4 3.98 ± 6% perf-profile.self.cycles-pp.__might_resched > 0.88 ± 2% +1.4 2.27 ± 11% perf-profile.self.cycles-pp.follow_pmd_mask > 0.88 ± 2% +1.4 2.28 ± 11% perf-profile.self.cycles-pp.__get_user_pages > 1.43 ± 2% +1.7 3.15 ± 12% perf-profile.self.cycles-pp.gup_put_folio > 1.50 ± 3% +2.3 3.79 ± 11% perf-profile.self.cycles-pp.follow_page_pte > 28.89 +2.5 31.42 perf-profile.self.cycles-pp.copy_user_enhanced_fast_string > 1.26 +2.8 4.04 ± 18% perf-profile.self.cycles-pp.mod_node_page_state > > > > > > Disclaimer: > Results have been estimated based on internal Intel analysis and are provided > for informational purposes only. Any difference in system hardware or software > design or configuration may affect actual performance. > >