Greeting, FYI, we noticed a -98.3% regression of fio.write_iops due to commit: commit: e7a8dd2d9537a7ec5aeadb002fc4934be1a396eb ("[PATCH v3] btrfs: Make btrfs_direct_write atomic with respect to inode_lock") url: https://github.com/0day-ci/linux/commits/Goldwyn-Rodrigues/btrfs-Make-btrfs_direct_write-atomic-with-respect-to-inode_lock/20201219-001114 base: https://git.kernel.org/cgit/linux/kernel/git/kdave/linux.git for-next in testcase: fio-basic on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory with following parameters: runtime: 300s disk: 1HDD fs: btrfs nr_task: 100% test_size: 128G rw: write bs: 4k ioengine: sync direct: direct cpufreq_governor: performance ucode: 0x42e test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. test-url: https://github.com/axboe/fio If you fix the issue, kindly add following tag Reported-by: kernel test robot Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= bs/compiler/cpufreq_governor/direct/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode: 4k/gcc-9/performance/direct/1HDD/btrfs/sync/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/300s/write/lkp-ivb-2ep1/128G/fio-basic/0x42e commit: 97847e0652 ("Merge branch 'for-next-next-v5.10-20201211' into for-next-20201211") e7a8dd2d95 ("btrfs: Make btrfs_direct_write atomic with respect to inode_lock") 97847e06525b51ea e7a8dd2d9537a7ec5aeadb002fc ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 0:4 -4% 0:4 perf-profile.children.cycles-pp.error_return 0:4 5% 0:4 perf-profile.children.cycles-pp.error_entry 0:4 5% 0:4 perf-profile.self.cycles-pp.error_entry %stddev %change %stddev \ | \ 0.02 ± 3% +0.1 0.12 ± 13% fio.latency_1000ms% 0.01 -0.0 0.00 fio.latency_1000us% 0.96 ± 13% -0.8 0.19 ±101% fio.latency_100ms% 31.35 -31.4 0.00 fio.latency_10ms% 8.28 ± 2% -8.3 0.00 fio.latency_20ms% 0.04 ± 29% -0.0 0.02 ±100% fio.latency_250ms% 0.01 -0.0 0.00 fio.latency_2ms% 54.35 -54.3 0.00 fio.latency_4ms% 0.02 ± 10% +99.6 99.66 fio.latency_500ms% 0.01 -0.0 0.00 fio.latency_500us% 4.97 -5.0 0.00 fio.latency_50ms% 0.01 -0.0 0.00 fio.latency_750us% 17301324 -93.9% 1062134 ± 2% fio.time.file_system_outputs 316.50 ± 45% +583.7% 2164 ± 14% fio.time.involuntary_context_switches 23.75 -63.2% 8.75 ± 4% fio.time.percent_of_cpu_this_job_got 62.17 -58.4% 25.87 ± 2% fio.time.system_time 10.95 ± 2% -83.7% 1.78 ± 9% fio.time.user_time 2194671 -93.0% 153935 fio.time.voluntary_context_switches 2162665 -98.3% 36777 fio.workload 28.16 -98.3% 0.48 fio.write_bw_MBps 12156928 +3384.6% 4.236e+08 fio.write_clat_90%_us 23855104 ± 2% +1715.4% 4.331e+08 fio.write_clat_95%_us 50724864 ± 4% +797.2% 4.551e+08 fio.write_clat_99%_us 6657995 +5784.2% 3.918e+08 fio.write_clat_mean_us 15787433 ± 2% +110.1% 33164741 ± 9% fio.write_clat_stddev 7208 -98.3% 122.53 fio.write_iops 10477 +39.4% 14609 uptime.idle 18.03 ± 15% -25.7% 13.39 boot-time.dhcp 1017 ± 11% -23.4% 779.50 boot-time.idle 64.67 +45.5% 94.08 iostat.cpu.idle 32.22 ± 2% -92.9% 2.28 ± 3% iostat.cpu.iowait 460106 ± 33% +56.7% 720959 ± 15% numa-numastat.node0.local_node 461015 ± 33% +61.1% 742788 ± 15% numa-numastat.node0.numa_hit 598719 ± 25% -43.3% 339612 ± 34% numa-numastat.node1.local_node 639421 ± 24% -43.4% 361729 ± 31% numa-numastat.node1.numa_hit 61123 ± 15% -82.4% 10785 meminfo.Active(anon) 10906 ± 2% +490.4% 64393 ± 15% meminfo.Active(file) 1678 -85.8% 238.00 ± 14% meminfo.Dirty 31700 +351.3% 143073 ± 5% meminfo.Inactive(file) 283671 -52.2% 135570 meminfo.SUnreclaim 359419 -41.1% 211861 meminfo.Slab 64.47 +29.6 94.08 mpstat.cpu.all.idle% 32.42 ± 2% -30.1 2.29 ± 3% mpstat.cpu.all.iowait% 2.04 ± 9% +1.0 3.08 ± 12% mpstat.cpu.all.irq% 0.23 ± 6% -0.0 0.19 ± 2% mpstat.cpu.all.soft% 0.66 ± 2% -0.4 0.25 ± 3% mpstat.cpu.all.sys% 0.17 -0.1 0.11 ± 2% mpstat.cpu.all.usr% 64.25 +45.1% 93.25 vmstat.cpu.id 31.75 ± 2% -95.3% 1.50 ± 33% vmstat.cpu.wa 28682 -89.7% 2962 ± 2% vmstat.io.bo 46.50 -98.4% 0.75 ± 57% vmstat.procs.b 38969 -92.5% 2923 vmstat.system.cs 100537 -4.5% 96061 vmstat.system.in 1.306e+09 ± 7% -99.1% 11707061 ± 18% cpuidle.C1.time 5821744 ± 7% -96.9% 182330 ± 10% cpuidle.C1.usage 6.258e+09 ± 9% -59.8% 2.514e+09 ± 9% cpuidle.C1E.time 18768783 ± 4% -29.4% 13255968 ± 26% cpuidle.C1E.usage 6.397e+09 ± 9% +77.6% 1.136e+10 ± 2% cpuidle.C6.time 9966359 ± 7% +63.3% 16279745 ± 22% cpuidle.C6.usage 5156525 ± 14% -99.2% 40443 ± 13% cpuidle.POLL.time 3217284 ± 17% -99.4% 19436 ± 12% cpuidle.POLL.usage 12395 ± 7% +201.0% 37310 ± 23% numa-meminfo.node0.Active 10314 ± 6% +235.7% 34621 ± 21% numa-meminfo.node0.Active(file) 1636 -91.5% 138.75 ± 16% numa-meminfo.node0.Dirty 24039 ± 54% +264.0% 87496 ± 4% numa-meminfo.node0.Inactive(file) 219811 ± 4% -68.3% 69676 ± 4% numa-meminfo.node0.SUnreclaim 258497 ± 4% -57.4% 110004 ± 6% numa-meminfo.node0.Slab 59798 ± 14% -36.6% 37897 ± 6% numa-meminfo.node1.Active 59211 ± 15% -86.3% 8123 ± 15% numa-meminfo.node1.Active(anon) 587.00 ± 72% +4972.2% 29774 ± 11% numa-meminfo.node1.Active(file) 183687 ± 20% -31.4% 126043 ± 15% numa-meminfo.node1.AnonPages.max 7665 ±171% +624.8% 55557 ± 9% numa-meminfo.node1.Inactive(file) 139520 ±120% -81.7% 25573 ± 4% sched_debug.cfs_rq:/.load.avg 50.14 ± 19% -21.8% 39.22 ± 7% sched_debug.cfs_rq:/.load_avg.avg 0.11 ± 6% -22.2% 0.09 ± 7% sched_debug.cfs_rq:/.nr_running.avg 0.29 ± 5% -15.0% 0.25 sched_debug.cfs_rq:/.nr_running.stddev 122.82 ± 4% -16.2% 102.93 ± 4% sched_debug.cfs_rq:/.runnable_avg.avg 122.46 ± 4% -16.5% 102.27 ± 5% sched_debug.cfs_rq:/.util_avg.avg 101814 ± 11% +277.7% 384526 ± 27% sched_debug.cpu.avg_idle.min 220884 ± 5% -37.6% 137900 ± 13% sched_debug.cpu.avg_idle.stddev 1.10 ± 4% +17.5% 1.30 ± 3% sched_debug.cpu.clock.stddev 1669 ± 4% -63.3% 612.98 ± 6% sched_debug.cpu.clock_task.stddev 5275 ± 6% +10.7% 5839 sched_debug.cpu.curr->pid.max 106300 ± 10% -89.3% 11398 sched_debug.cpu.nr_switches.avg 1209385 ± 11% -97.4% 31808 ± 14% sched_debug.cpu.nr_switches.max 192228 ± 11% -97.3% 5222 ± 18% sched_debug.cpu.nr_switches.stddev -251.12 -86.0% -35.25 sched_debug.cpu.nr_uninterruptible.min 40.50 ± 14% -64.2% 14.48 ± 24% sched_debug.cpu.nr_uninterruptible.stddev 2577 ± 6% +235.7% 8653 ± 21% numa-vmstat.node0.nr_active_file 10194 ± 42% +216.4% 32253 ± 5% numa-vmstat.node0.nr_dirtied 409.50 -91.6% 34.50 ± 15% numa-vmstat.node0.nr_dirty 6009 ± 54% +264.0% 21873 ± 4% numa-vmstat.node0.nr_inactive_file 54951 ± 4% -68.3% 17418 ± 4% numa-vmstat.node0.nr_slab_unreclaimable 9778 ± 43% +229.6% 32228 ± 5% numa-vmstat.node0.nr_written 2577 ± 6% +235.7% 8653 ± 21% numa-vmstat.node0.nr_zone_active_file 6009 ± 54% +264.0% 21873 ± 4% numa-vmstat.node0.nr_zone_inactive_file 414.75 ± 2% -92.5% 31.00 ± 13% numa-vmstat.node0.nr_zone_write_pending 14804 ± 15% -86.3% 2027 ± 15% numa-vmstat.node1.nr_active_anon 146.75 ± 72% +4970.4% 7440 ± 10% numa-vmstat.node1.nr_active_file 2882 ±147% +676.1% 22371 ± 5% numa-vmstat.node1.nr_dirtied 1915 ±171% +625.1% 13888 ± 9% numa-vmstat.node1.nr_inactive_file 2871 ±147% +678.4% 22351 ± 5% numa-vmstat.node1.nr_written 14804 ± 15% -86.3% 2027 ± 15% numa-vmstat.node1.nr_zone_active_anon 146.75 ± 72% +4970.4% 7440 ± 10% numa-vmstat.node1.nr_zone_active_file 1915 ±171% +625.1% 13888 ± 9% numa-vmstat.node1.nr_zone_inactive_file 15380 ± 15% -82.4% 2703 proc-vmstat.nr_active_anon 2725 ± 2% +490.4% 16094 ± 16% proc-vmstat.nr_active_file 17429 +471.3% 99580 ± 2% proc-vmstat.nr_dirtied 416.75 -86.0% 58.25 ± 13% proc-vmstat.nr_dirty 567440 +4.9% 595244 proc-vmstat.nr_file_pages 7926 +351.0% 35749 ± 5% proc-vmstat.nr_inactive_file 12783 +1.7% 13005 proc-vmstat.nr_kernel_stack 297875 -4.5% 284331 proc-vmstat.nr_shmem 70910 -52.2% 33895 proc-vmstat.nr_slab_unreclaimable 16905 +488.8% 99538 ± 2% proc-vmstat.nr_written 15380 ± 15% -82.4% 2703 proc-vmstat.nr_zone_active_anon 2725 ± 2% +490.4% 16094 ± 16% proc-vmstat.nr_zone_active_file 7926 +351.0% 35749 ± 5% proc-vmstat.nr_zone_inactive_file 424.75 -87.9% 51.50 ± 9% proc-vmstat.nr_zone_write_pending 42826 ± 14% -40.5% 25476 ± 3% proc-vmstat.pgactivate 1239967 -3.7% 1194026 proc-vmstat.pgalloc_normal 8740013 -89.7% 904228 ± 2% proc-vmstat.pgpgout 53472 ± 3% +5.6% 56482 proc-vmstat.pgreuse 158.00 ± 12% +348.6% 708.75 ± 2% slabinfo.biovec-128.active_objs 158.00 ± 12% +348.6% 708.75 ± 2% slabinfo.biovec-128.num_objs 374.25 ± 8% +258.8% 1342 ± 2% slabinfo.biovec-64.active_objs 374.25 ± 8% +258.8% 1342 ± 2% slabinfo.biovec-64.num_objs 163.50 ± 13% +227.8% 536.00 ± 7% slabinfo.biovec-max.active_objs 204.75 ± 13% +168.7% 550.25 ± 7% slabinfo.biovec-max.num_objs 1297 ± 14% -34.9% 845.25 ± 6% slabinfo.btrfs_delayed_tree_ref.active_objs 1297 ± 14% -34.9% 845.25 ± 6% slabinfo.btrfs_delayed_tree_ref.num_objs 2006 ± 13% +59.9% 3209 slabinfo.btrfs_extent_buffer.active_objs 2006 ± 13% +59.9% 3209 slabinfo.btrfs_extent_buffer.num_objs 336.00 ± 23% -31.2% 231.00 ± 15% slabinfo.kmalloc-rcl-192.active_objs 336.00 ± 23% -31.2% 231.00 ± 15% slabinfo.kmalloc-rcl-192.num_objs 1079929 -99.6% 4015 slabinfo.pid_namespace.active_objs 19284 -99.6% 71.25 slabinfo.pid_namespace.active_slabs 1079934 -99.6% 4015 slabinfo.pid_namespace.num_objs 19284 -99.6% 71.25 slabinfo.pid_namespace.num_slabs 681.25 ± 10% +233.8% 2274 ± 12% slabinfo.skbuff_fclone_cache.active_objs 681.25 ± 10% +233.8% 2274 ± 12% slabinfo.skbuff_fclone_cache.num_objs 705679 ± 2% -96.8% 22805 softirqs.BLOCK 56594 ± 12% -36.3% 36023 ± 19% softirqs.CPU0.RCU 61813 ± 12% -41.2% 36360 ± 13% softirqs.CPU1.RCU 61170 ± 12% -41.9% 35511 ± 4% softirqs.CPU10.RCU 59336 ± 10% -37.7% 36972 ± 5% softirqs.CPU11.RCU 58606 ± 14% -36.7% 37114 ± 17% softirqs.CPU2.RCU 54177 ± 12% -40.1% 32471 ± 12% softirqs.CPU24.RCU 55834 ± 11% -36.9% 35230 ± 10% softirqs.CPU25.RCU 53216 ± 9% -31.9% 36262 ± 15% softirqs.CPU26.RCU 53744 ± 12% -35.7% 34533 ± 16% softirqs.CPU27.RCU 51611 ± 8% -33.8% 34181 ± 13% softirqs.CPU28.RCU 53726 ± 7% -33.5% 35754 ± 9% softirqs.CPU29.RCU 61015 ± 14% -41.3% 35804 ± 18% softirqs.CPU3.RCU 54715 ± 12% -37.5% 34188 ± 3% softirqs.CPU30.RCU 51435 ± 10% -33.5% 34224 ± 7% softirqs.CPU31.RCU 52697 ± 8% -38.8% 32238 ± 5% softirqs.CPU32.RCU 27579 ± 7% +49.8% 41304 ± 3% softirqs.CPU32.SCHED 48705 ± 9% -32.2% 33031 ± 4% softirqs.CPU33.RCU 48234 ± 8% -29.0% 34223 ± 3% softirqs.CPU34.RCU 49281 ± 10% -29.7% 34654 ± 3% softirqs.CPU35.RCU 54023 ± 8% -35.2% 34993 ± 19% softirqs.CPU4.RCU 55222 ± 26% -34.4% 36234 ± 19% softirqs.CPU40.RCU 59824 ± 8% -37.8% 37225 ± 14% softirqs.CPU5.RCU 62723 ± 13% -41.6% 36615 ± 9% softirqs.CPU6.RCU 56745 ± 13% -33.6% 37668 ± 11% softirqs.CPU7.RCU 685836 -98.4% 10922 ± 2% softirqs.CPU8.BLOCK 63989 ± 9% -41.2% 37647 ± 5% softirqs.CPU8.RCU 693875 -96.6% 23342 softirqs.CPU8.TASKLET 60701 ± 10% -41.2% 35709 ± 9% softirqs.CPU9.RCU 2511282 ± 18% -31.4% 1721969 ± 10% softirqs.RCU 694098 -96.6% 23565 softirqs.TASKLET 2089 ± 62% -80.3% 410.75 ± 91% interrupts.41:PCI-MSI.2621447-edge.eth0-TxRx-6 1385835 -96.8% 44728 interrupts.73:PCI-MSI.2097152-edge.isci-msix 774.25 ± 60% -87.9% 94.00 ± 35% interrupts.CPU1.RES:Rescheduling_interrupts 1167 ± 14% -85.9% 164.50 ± 63% interrupts.CPU10.RES:Rescheduling_interrupts 1041 ± 18% -82.2% 185.25 ± 48% interrupts.CPU11.RES:Rescheduling_interrupts 188.00 ± 23% +32.0% 248.25 ± 26% interrupts.CPU14.NMI:Non-maskable_interrupts 188.00 ± 23% +32.0% 248.25 ± 26% interrupts.CPU14.PMI:Performance_monitoring_interrupts 113.75 ± 29% -44.2% 63.50 ± 15% interrupts.CPU16.RES:Rescheduling_interrupts 470.50 ± 38% -83.7% 76.50 ± 39% interrupts.CPU2.RES:Rescheduling_interrupts 880.75 ± 24% -87.1% 113.75 ± 54% interrupts.CPU24.RES:Rescheduling_interrupts 320.50 ± 27% -37.6% 200.00 ± 28% interrupts.CPU25.NMI:Non-maskable_interrupts 320.50 ± 27% -37.6% 200.00 ± 28% interrupts.CPU25.PMI:Performance_monitoring_interrupts 640.00 ± 28% -85.5% 92.50 ± 59% interrupts.CPU25.RES:Rescheduling_interrupts 572.00 ± 30% -85.7% 81.75 ± 42% interrupts.CPU26.RES:Rescheduling_interrupts 582.25 ± 25% -85.1% 86.50 ± 40% interrupts.CPU27.RES:Rescheduling_interrupts 474.00 ± 20% -82.4% 83.25 ± 24% interrupts.CPU28.RES:Rescheduling_interrupts 375.00 ± 13% -74.7% 95.00 ± 29% interrupts.CPU29.RES:Rescheduling_interrupts 527.50 ± 46% -82.2% 93.75 ± 46% interrupts.CPU3.RES:Rescheduling_interrupts 949.00 ± 10% +56.0% 1480 ± 37% interrupts.CPU30.CAL:Function_call_interrupts 447.75 ± 30% -70.2% 133.50 ± 48% interrupts.CPU30.RES:Rescheduling_interrupts 892.25 ± 2% +26.9% 1132 ± 15% interrupts.CPU31.CAL:Function_call_interrupts 2089 ± 62% -80.3% 410.75 ± 91% interrupts.CPU32.41:PCI-MSI.2621447-edge.eth0-TxRx-6 968.00 ± 10% +94.1% 1878 ± 59% interrupts.CPU32.CAL:Function_call_interrupts 826.50 ± 28% -71.7% 233.75 ± 31% interrupts.CPU32.NMI:Non-maskable_interrupts 826.50 ± 28% -71.7% 233.75 ± 31% interrupts.CPU32.PMI:Performance_monitoring_interrupts 656.25 ± 15% -75.5% 160.75 ± 64% interrupts.CPU32.RES:Rescheduling_interrupts 558.25 ± 32% -75.0% 139.50 ± 55% interrupts.CPU33.RES:Rescheduling_interrupts 447.25 ± 16% -61.8% 171.00 ± 47% interrupts.CPU34.RES:Rescheduling_interrupts 504.50 ± 33% -81.9% 91.25 ± 32% interrupts.CPU35.RES:Rescheduling_interrupts 301.50 ± 40% -77.9% 66.75 ± 62% interrupts.CPU4.RES:Rescheduling_interrupts 315.25 ± 43% -65.9% 107.50 ± 81% interrupts.CPU5.RES:Rescheduling_interrupts 491.75 ± 57% -86.3% 67.25 ± 35% interrupts.CPU6.RES:Rescheduling_interrupts 1385835 -96.8% 44728 interrupts.CPU8.73:PCI-MSI.2097152-edge.isci-msix 916.25 ± 9% +26.0% 1154 ± 18% interrupts.CPU8.CAL:Function_call_interrupts 827.25 ± 38% -75.0% 206.50 ± 39% interrupts.CPU8.NMI:Non-maskable_interrupts 827.25 ± 38% -75.0% 206.50 ± 39% interrupts.CPU8.PMI:Performance_monitoring_interrupts 906.25 ± 2% +53.9% 1395 ± 47% interrupts.CPU9.CAL:Function_call_interrupts 1107 ± 10% -87.6% 137.50 ± 49% interrupts.CPU9.RES:Rescheduling_interrupts 13410 ± 5% -16.0% 11263 ± 7% interrupts.NMI:Non-maskable_interrupts 13410 ± 5% -16.0% 11263 ± 7% interrupts.PMI:Performance_monitoring_interrupts 15515 ± 16% -69.0% 4813 ± 14% interrupts.RES:Rescheduling_interrupts 0.02 ± 49% +181.5% 0.05 ± 72% perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 0.01 ± 8% +35.1% 0.01 ± 8% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.03 ± 59% -100.0% 0.00 perf-sched.sch_delay.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 0.01 ± 14% +287.0% 0.02 ± 16% perf-sched.sch_delay.avg.ms.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write 0.01 ± 57% -58.8% 0.00 ± 14% perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.smpboot_thread_fn.kthread.ret_from_fork 0.00 ±118% +350.0% 0.02 ± 68% perf-sched.sch_delay.avg.ms.rwsem_down_read_slowpath.__btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot 0.00 ±100% +412.5% 0.01 ± 21% perf-sched.sch_delay.avg.ms.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot 0.01 ± 49% +86.5% 0.02 ± 8% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select 0.00 ± 57% +733.3% 0.01 ± 18% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 0.02 ± 9% +37.7% 0.02 ± 10% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.01 ± 7% +73.1% 0.01 ± 3% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork 0.03 ± 34% +72.1% 0.06 ± 18% perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.03 ± 59% -100.0% 0.00 perf-sched.sch_delay.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 0.03 ±110% +197.4% 0.08 ± 32% perf-sched.sch_delay.max.ms.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.filemap_fdatawait_range 0.01 ±110% +274.4% 0.04 ± 64% perf-sched.sch_delay.max.ms.rwsem_down_read_slowpath.__btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot 0.03 ± 60% +91.0% 0.06 ± 10% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_select 0.02 ± 31% +130.6% 0.04 ± 12% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 0.04 ± 18% +26.8% 0.05 ± 7% perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.01 ± 11% +125.0% 0.01 ± 3% perf-sched.total_sch_delay.average.ms 10.12 ± 21% +1307.5% 142.39 perf-sched.total_wait_and_delay.average.ms 172750 ± 17% -91.4% 14933 ± 2% perf-sched.total_wait_and_delay.count.ms 5885 ± 25% +56.9% 9234 ± 11% perf-sched.total_wait_and_delay.max.ms 10.11 ± 21% +1308.2% 142.38 perf-sched.total_wait_time.average.ms 5885 ± 25% +56.9% 9234 ± 11% perf-sched.total_wait_time.max.ms 8.01 ± 22% +1181.9% 102.62 ± 62% perf-sched.wait_and_delay.avg.ms.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write 1.02 ±173% +314.1% 4.24 ± 3% perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork 232.13 ± 18% +103.1% 471.43 ± 36% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 4.54 ± 6% -8.7% 4.14 ± 4% perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 393.96 ± 14% +23.3% 485.81 ± 5% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork 4.09 ± 26% +6387.8% 265.12 ± 2% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork 61985 ± 18% -98.0% 1211 ± 2% perf-sched.wait_and_delay.count.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write 2.00 ±173% +4850.0% 99.00 ± 18% perf-sched.wait_and_delay.count.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.filemap_fdatawait_range 83.50 ± 16% -71.0% 24.25 ± 6% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 1547 ± 7% +16.4% 1801 ± 3% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 102506 ± 17% -97.7% 2395 ± 7% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork 943.35 ± 8% -81.1% 178.11 ± 56% perf-sched.wait_and_delay.max.ms.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write 1.26 ±173% +304.5% 5.11 perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork 1589 ± 52% +190.9% 4624 ± 25% perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork 0.00 ±173% +1.2e+05% 5.20 ± 73% perf-sched.wait_time.avg.ms.btrfs_commit_transaction.transaction_kthread.kthread.ret_from_fork 3.92 ± 11% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 8.00 ± 22% +1182.5% 102.60 ± 62% perf-sched.wait_time.avg.ms.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write 0.28 ±155% +24627.1% 69.30 ±106% perf-sched.wait_time.avg.ms.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot 232.13 ± 18% +103.1% 471.42 ± 36% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_sys_poll 4.53 ± 6% -8.9% 4.13 ± 4% perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork 393.96 ± 14% +23.3% 485.80 ± 5% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork 4.08 ± 26% +6398.3% 265.11 ± 2% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork 0.00 ±173% +4e+05% 17.17 ±107% perf-sched.wait_time.max.ms.btrfs_commit_transaction.transaction_kthread.kthread.ret_from_fork 3.92 ± 11% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown] 943.35 ± 8% -81.1% 178.04 ± 56% perf-sched.wait_time.max.ms.io_schedule.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write 6.02 ±172% +2236.8% 140.75 ± 90% perf-sched.wait_time.max.ms.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot 0.01 ± 86% +7423.8% 0.79 ±167% perf-sched.wait_time.max.ms.schedule_timeout.io_schedule_timeout.wait_for_completion_io.write_all_supers 1589 ± 52% +190.9% 4624 ± 25% perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork 44.03 +20.3% 52.96 ± 12% perf-stat.i.MPKI 2.589e+08 -36.7% 1.639e+08 perf-stat.i.branch-instructions 5.01 +6.7 11.71 ± 7% perf-stat.i.branch-miss-rate% 12714258 +31.8% 16758440 ± 7% perf-stat.i.branch-misses 9.35 ± 2% +14.7 24.09 ± 3% perf-stat.i.cache-miss-rate% 4375264 +81.1% 7923317 ± 18% perf-stat.i.cache-misses 48162798 -32.2% 32638947 ± 14% perf-stat.i.cache-references 39282 -92.6% 2916 perf-stat.i.context-switches 4.10 +83.9% 7.54 ± 6% perf-stat.i.cpi 68.61 ± 2% +64.8% 113.06 perf-stat.i.cpu-migrations 1061 ± 3% -42.2% 613.59 ± 8% perf-stat.i.cycles-between-cache-misses 1.23 ± 2% -0.2 1.06 ± 8% perf-stat.i.dTLB-load-miss-rate% 3992778 ± 2% -47.7% 2089367 ± 11% perf-stat.i.dTLB-load-misses 3.459e+08 -35.4% 2.234e+08 ± 2% perf-stat.i.dTLB-loads 0.17 ± 2% +0.0 0.20 ± 4% perf-stat.i.dTLB-store-miss-rate% 70.32 +21.5 91.83 perf-stat.i.iTLB-load-miss-rate% 1985999 ± 2% -39.6% 1199619 ± 2% perf-stat.i.iTLB-load-misses 854405 -87.4% 107547 ± 14% perf-stat.i.iTLB-loads 1.254e+09 -36.9% 7.911e+08 perf-stat.i.instructions 0.26 -41.9% 0.15 ± 6% perf-stat.i.ipc 1.34 ± 7% -58.0% 0.56 ± 31% perf-stat.i.metric.K/sec 19.36 -25.9% 14.35 ± 6% perf-stat.i.metric.M/sec 2893 -2.0% 2835 perf-stat.i.minor-faults 197998 ± 19% -32.9% 132808 ± 3% perf-stat.i.node-load-misses 228908 ± 17% -31.8% 156142 ± 3% perf-stat.i.node-loads 236295 ± 7% -17.5% 194871 ± 6% perf-stat.i.node-stores 3820 -1.7% 3754 perf-stat.i.page-faults 4.91 +5.3 10.21 ± 7% perf-stat.overall.branch-miss-rate% 9.09 ± 2% +15.1 24.16 ± 3% perf-stat.overall.cache-miss-rate% 3.65 +65.6% 6.04 ± 7% perf-stat.overall.cpi 1045 ± 3% -41.3% 613.83 ± 8% perf-stat.overall.cycles-between-cache-misses 1.14 ± 2% -0.2 0.92 ± 8% perf-stat.overall.dTLB-load-miss-rate% 0.17 ± 3% +0.0 0.19 ± 4% perf-stat.overall.dTLB-store-miss-rate% 69.91 +21.9 91.81 perf-stat.overall.iTLB-load-miss-rate% 0.27 -39.3% 0.17 ± 6% perf-stat.overall.ipc 175094 +3617.1% 6508495 perf-stat.overall.path-length 2.583e+08 -36.7% 1.636e+08 perf-stat.ps.branch-instructions 12678701 +31.8% 16711512 ± 7% perf-stat.ps.branch-misses 4361806 +81.1% 7898239 ± 18% perf-stat.ps.cache-misses 48005640 -32.2% 32532931 ± 14% perf-stat.ps.cache-references 39151 -92.6% 2907 perf-stat.ps.context-switches 68.42 ± 2% +64.7% 112.70 perf-stat.ps.cpu-migrations 3979673 ± 2% -47.7% 2082707 ± 11% perf-stat.ps.dTLB-load-misses 3.449e+08 -35.4% 2.229e+08 ± 2% perf-stat.ps.dTLB-loads 1979429 ± 2% -39.6% 1195674 ± 2% perf-stat.ps.iTLB-load-misses 851561 -87.4% 107209 ± 14% perf-stat.ps.iTLB-loads 1.251e+09 -36.9% 7.896e+08 perf-stat.ps.instructions 2884 -2.0% 2827 perf-stat.ps.minor-faults 197548 ± 19% -33.0% 132439 ± 3% perf-stat.ps.node-load-misses 228425 ± 17% -31.8% 155760 ± 3% perf-stat.ps.node-loads 236208 ± 7% -17.3% 195231 ± 6% perf-stat.ps.node-stores 3.787e+11 -36.8% 2.393e+11 perf-stat.total.instructions 10.21 ± 20% -9.9 0.36 ±100% perf-profile.calltrace.cycles-pp.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write 8.66 ± 23% -8.5 0.17 ±173% perf-profile.calltrace.cycles-pp.iomap_apply.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write.vfs_write 11.16 ± 19% -7.3 3.87 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 11.02 ± 19% -7.2 3.84 ± 7% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 10.77 ± 19% -7.2 3.59 ± 6% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 10.72 ± 19% -7.1 3.59 ± 6% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 10.54 ± 19% -6.9 3.59 ± 6% perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 10.51 ± 20% -6.9 3.59 ± 6% perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64 6.28 ± 31% -6.1 0.14 ±173% perf-profile.calltrace.cycles-pp.btrfs_dio_iomap_begin.iomap_apply.__iomap_dio_rw.btrfs_file_write_iter.new_sync_write 6.48 ± 12% -4.6 1.87 ± 23% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork 6.63 ± 12% -4.5 2.11 ± 22% perf-profile.calltrace.cycles-pp.ret_from_fork 6.63 ± 12% -4.5 2.11 ± 22% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork 5.90 ± 11% -4.1 1.82 ± 23% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork 0.91 ± 8% +0.3 1.17 ± 5% perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack 1.16 ± 12% +0.3 1.43 ± 6% perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry 0.64 ± 9% +0.3 0.93 ± 8% perf-profile.calltrace.cycles-pp.update_blocked_averages.run_rebalance_domains.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack 0.99 ± 7% +0.3 1.32 ± 12% perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 0.65 ± 10% +0.3 0.98 ± 11% perf-profile.calltrace.cycles-pp.run_rebalance_domains.__softirqentry_text_start.asm_call_sysvec_on_stack.do_softirq_own_stack.irq_exit_rcu 0.26 ±100% +0.4 0.68 ± 14% perf-profile.calltrace.cycles-pp.cpuidle_governor_latency_req.menu_select.do_idle.cpu_startup_entry.start_secondary 1.29 ± 6% +0.4 1.71 ± 14% perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 1.56 ± 9% +0.4 1.99 ± 3% perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 1.75 ± 10% +0.5 2.27 ± 5% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary 0.67 ± 65% +0.5 1.19 ± 8% perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.sysvec_irq_work.asm_sysvec_irq_work.cpuidle_enter_state.cpuidle_enter 0.67 ± 65% +0.5 1.19 ± 8% perf-profile.calltrace.cycles-pp.asm_sysvec_irq_work.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 0.67 ± 65% +0.5 1.19 ± 8% perf-profile.calltrace.cycles-pp.sysvec_irq_work.asm_sysvec_irq_work.cpuidle_enter_state.cpuidle_enter.do_idle 0.67 ± 65% +0.5 1.19 ± 8% perf-profile.calltrace.cycles-pp.__sysvec_irq_work.asm_call_sysvec_on_stack.sysvec_irq_work.asm_sysvec_irq_work.cpuidle_enter_state 0.00 +0.6 0.60 ± 6% perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file 0.00 +0.6 0.60 ± 7% perf-profile.calltrace.cycles-pp.timerqueue_del.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 0.14 ±173% +0.6 0.74 ± 7% perf-profile.calltrace.cycles-pp.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack 0.00 +0.6 0.62 ± 13% perf-profile.calltrace.cycles-pp.enqueue_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack 0.00 +0.6 0.62 ± 5% perf-profile.calltrace.cycles-pp.__mutex_lock.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter 0.80 ± 58% +0.6 1.45 ± 17% perf-profile.calltrace.cycles-pp.wait_for_xmitr.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock 0.80 ± 58% +0.6 1.45 ± 17% perf-profile.calltrace.cycles-pp.serial8250_console_putchar.uart_console_write.serial8250_console_write.console_unlock.vprintk_emit 0.15 ±173% +0.7 0.83 ± 12% perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues 0.00 +0.7 0.72 ± 5% perf-profile.calltrace.cycles-pp.wait_log_commit.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.new_sync_write 0.00 +0.7 0.73 ± 20% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node 0.00 +0.8 0.77 ± 18% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot 0.00 +0.8 0.77 ± 18% perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent 0.00 +0.8 0.77 ± 18% perf-profile.calltrace.cycles-pp.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode 0.00 +0.8 0.77 ± 18% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_insert_empty_items 2.60 ± 14% +0.9 3.48 ± 10% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 0.00 +0.9 0.90 ± 5% perf-profile.calltrace.cycles-pp.btrfs_sync_log.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write 2.73 ± 7% +1.1 3.81 ± 4% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 2.92 ± 7% +1.2 4.08 ± 4% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt 4.36 ± 15% +1.2 5.60 ± 9% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 1.03 ± 13% +1.2 2.28 ± 33% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel 1.05 ± 13% +1.3 2.31 ± 33% perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify 3.44 ± 8% +1.3 4.72 ± 4% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack 1.15 ± 15% +1.4 2.54 ± 32% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify 1.15 ± 15% +1.4 2.54 ± 32% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_kernel.secondary_startup_64_no_verify 1.15 ± 15% +1.4 2.54 ± 32% perf-profile.calltrace.cycles-pp.start_kernel.secondary_startup_64_no_verify 0.00 +1.7 1.68 ± 20% perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe 0.00 +1.7 1.68 ± 20% perf-profile.calltrace.cycles-pp.btrfs_insert_empty_items.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file 0.00 +1.9 1.87 ± 18% perf-profile.calltrace.cycles-pp.btrfs_log_inode.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file.btrfs_file_write_iter 0.00 +2.0 2.02 ± 17% perf-profile.calltrace.cycles-pp.btrfs_log_inode_parent.btrfs_log_dentry_safe.btrfs_sync_file.btrfs_file_write_iter.new_sync_write 0.00 +2.0 2.03 ± 17% perf-profile.calltrace.cycles-pp.btrfs_log_dentry_safe.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write 5.73 ± 7% +2.3 8.03 ± 3% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt 8.28 ± 10% +2.5 10.75 ± 4% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 8.66 ± 9% +2.7 11.31 ± 4% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state 8.79 ± 9% +2.7 11.50 ± 4% perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter 0.00 +3.0 3.00 ± 11% perf-profile.calltrace.cycles-pp.btrfs_sync_file.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write 13.50 ± 9% +4.3 17.84 ± 7% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle 15.39 ± 9% +4.7 20.10 ± 7% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 51.90 +8.8 60.72 ± 2% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry 72.26 ± 2% +9.9 82.17 perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 71.31 ± 2% +9.9 81.25 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary 79.02 ± 3% +10.8 89.79 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 79.10 ± 3% +10.8 89.91 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 79.10 ± 3% +10.8 89.91 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 80.25 ± 2% +12.2 92.45 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 10.22 ± 20% -9.6 0.58 ± 26% perf-profile.children.cycles-pp.__iomap_dio_rw 8.66 ± 23% -8.2 0.45 ± 32% perf-profile.children.cycles-pp.iomap_apply 10.79 ± 19% -7.2 3.62 ± 6% perf-profile.children.cycles-pp.ksys_write 12.01 ± 17% -7.1 4.86 ± 8% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 10.74 ± 19% -7.1 3.62 ± 6% perf-profile.children.cycles-pp.vfs_write 11.77 ± 17% -7.0 4.73 ± 8% perf-profile.children.cycles-pp.do_syscall_64 10.56 ± 19% -6.9 3.62 ± 6% perf-profile.children.cycles-pp.new_sync_write 10.52 ± 20% -6.9 3.59 ± 6% perf-profile.children.cycles-pp.btrfs_file_write_iter 6.28 ± 31% -5.9 0.38 ± 36% perf-profile.children.cycles-pp.btrfs_dio_iomap_begin 6.48 ± 12% -4.6 1.87 ± 23% perf-profile.children.cycles-pp.worker_thread 6.63 ± 12% -4.5 2.11 ± 22% perf-profile.children.cycles-pp.kthread 6.63 ± 12% -4.5 2.12 ± 22% perf-profile.children.cycles-pp.ret_from_fork 5.91 ± 11% -4.1 1.82 ± 23% perf-profile.children.cycles-pp.process_one_work 3.45 ± 24% -3.3 0.14 ± 55% perf-profile.children.cycles-pp.btrfs_work_helper 3.43 ± 24% -3.3 0.14 ± 55% perf-profile.children.cycles-pp.btrfs_finish_ordered_io 2.77 ± 43% -2.7 0.04 ±106% perf-profile.children.cycles-pp.asm_common_interrupt 2.77 ± 43% -2.7 0.04 ±106% perf-profile.children.cycles-pp.common_interrupt 2.33 ± 8% -2.3 0.04 ±107% perf-profile.children.cycles-pp.can_nocow_extent 2.21 ± 12% -2.1 0.06 ± 67% perf-profile.children.cycles-pp.iomap_dio_bio_actor 2.01 ± 42% -1.9 0.08 ± 17% perf-profile.children.cycles-pp.blk_done_softirq 1.95 ± 42% -1.9 0.07 ± 17% perf-profile.children.cycles-pp.scsi_io_completion 1.95 ± 42% -1.9 0.07 ± 17% perf-profile.children.cycles-pp.scsi_end_request 2.13 ± 94% -1.9 0.27 ± 49% perf-profile.children.cycles-pp.btrfs_delalloc_reserve_space 2.17 ± 12% -1.8 0.37 ± 27% perf-profile.children.cycles-pp.__schedule 2.51 ± 96% -1.7 0.77 ± 25% perf-profile.children.cycles-pp._raw_spin_lock 2.00 ±100% -1.7 0.26 ± 51% perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata 1.92 ±104% -1.6 0.29 ± 47% perf-profile.children.cycles-pp.__reserve_bytes 1.68 ± 13% -1.6 0.05 ± 62% perf-profile.children.cycles-pp.iomap_dio_submit_bio 1.90 ±105% -1.6 0.28 ± 48% perf-profile.children.cycles-pp.btrfs_reserve_metadata_bytes 1.66 ± 12% -1.6 0.05 ± 62% perf-profile.children.cycles-pp.btrfs_submit_direct 1.43 ± 41% -1.4 0.04 ±100% perf-profile.children.cycles-pp.blk_update_request 1.35 ± 41% -1.3 0.03 ±100% perf-profile.children.cycles-pp.btrfs_end_bio 1.50 ± 13% -1.3 0.24 ± 17% perf-profile.children.cycles-pp.schedule 1.11 ± 12% -1.1 0.04 ±100% perf-profile.children.cycles-pp.btrfs_map_bio 0.97 ± 18% -0.9 0.03 ±100% perf-profile.children.cycles-pp.io_schedule 0.98 ± 32% -0.9 0.09 ± 16% perf-profile.children.cycles-pp.__blk_mq_run_hw_queue 0.97 ± 32% -0.9 0.09 ± 16% perf-profile.children.cycles-pp.blk_mq_sched_dispatch_requests 0.96 ± 33% -0.9 0.09 ± 16% perf-profile.children.cycles-pp.__blk_mq_sched_dispatch_requests 0.92 ± 33% -0.8 0.09 ± 21% perf-profile.children.cycles-pp.__blk_mq_do_dispatch_sched 0.89 ± 4% -0.8 0.05 ± 61% perf-profile.children.cycles-pp.read_block_for_search 0.94 ± 32% -0.8 0.15 ± 39% perf-profile.children.cycles-pp.try_to_wake_up 0.85 ± 16% -0.7 0.16 ± 67% perf-profile.children.cycles-pp.btrfs_csum_file_blocks 0.69 ± 34% -0.6 0.06 ± 17% perf-profile.children.cycles-pp.blk_mq_dispatch_rq_list 0.73 ± 9% -0.6 0.14 ± 47% perf-profile.children.cycles-pp.schedule_idle 0.65 ± 33% -0.6 0.06 ± 17% perf-profile.children.cycles-pp.scsi_queue_rq 0.65 ± 30% -0.5 0.12 ± 33% perf-profile.children.cycles-pp.pick_next_task_fair 0.56 ± 10% -0.5 0.09 ± 13% perf-profile.children.cycles-pp.kmem_cache_alloc 0.42 ± 32% -0.4 0.04 ± 58% perf-profile.children.cycles-pp.sas_queuecommand 0.39 ± 33% -0.4 0.03 ±100% perf-profile.children.cycles-pp.__ata_scsi_queuecmd 0.43 ± 8% -0.4 0.07 ± 22% perf-profile.children.cycles-pp.dequeue_task_fair 0.43 ± 18% -0.3 0.09 ± 28% perf-profile.children.cycles-pp.blk_finish_plug 0.42 ± 19% -0.3 0.09 ± 28% perf-profile.children.cycles-pp.blk_flush_plug_list 0.42 ± 18% -0.3 0.09 ± 28% perf-profile.children.cycles-pp.blk_mq_flush_plug_list 0.40 ± 22% -0.3 0.09 ± 28% perf-profile.children.cycles-pp.blk_mq_sched_insert_requests 0.38 ± 25% -0.3 0.08 ± 30% perf-profile.children.cycles-pp.ttwu_do_activate 0.37 ± 40% -0.3 0.07 ± 20% perf-profile.children.cycles-pp.newidle_balance 0.34 ± 26% -0.3 0.05 ± 60% perf-profile.children.cycles-pp.kmem_cache_free 0.35 ± 6% -0.3 0.07 ± 21% perf-profile.children.cycles-pp.dequeue_entity 0.33 ± 22% -0.2 0.08 ± 13% perf-profile.children.cycles-pp.enqueue_task_fair 0.36 ± 24% -0.2 0.16 ± 9% perf-profile.children.cycles-pp.update_load_avg 0.26 ± 23% -0.2 0.07 ± 7% perf-profile.children.cycles-pp.enqueue_entity 0.29 ± 22% -0.2 0.12 ± 81% perf-profile.children.cycles-pp.btrfs_lookup_csum 0.18 ± 20% -0.2 0.03 ±100% perf-profile.children.cycles-pp.set_next_entity 0.12 ± 22% -0.1 0.03 ±100% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.21 ± 9% -0.1 0.13 ± 31% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 0.12 ± 15% -0.1 0.05 ± 67% perf-profile.children.cycles-pp.__wake_up_common_lock 0.11 ± 20% -0.1 0.04 ± 58% perf-profile.children.cycles-pp.__update_load_avg_se 0.15 ± 20% -0.1 0.08 ± 13% perf-profile.children.cycles-pp.__blk_mq_delay_run_hw_queue 0.79 ± 4% -0.1 0.73 ± 4% perf-profile.children.cycles-pp.read_tsc 0.12 ± 23% -0.0 0.08 ± 30% perf-profile.children.cycles-pp.read 0.07 ± 7% +0.0 0.10 ± 10% perf-profile.children.cycles-pp.path_openat 0.09 ± 16% +0.0 0.12 ± 13% perf-profile.children.cycles-pp.rcu_nmi_exit 0.07 ± 7% +0.0 0.10 ± 12% perf-profile.children.cycles-pp.do_filp_open 0.07 ± 19% +0.0 0.11 ± 23% perf-profile.children.cycles-pp.rcu_segcblist_ready_cbs 0.08 ± 14% +0.0 0.12 ± 6% perf-profile.children.cycles-pp.tick_program_event 0.01 ±173% +0.0 0.06 ± 11% perf-profile.children.cycles-pp.run_timer_softirq 0.05 ± 62% +0.1 0.10 ± 18% perf-profile.children.cycles-pp.perf_pmu_disable 0.01 ±173% +0.1 0.07 ± 26% perf-profile.children.cycles-pp.cpuidle_get_cpu_driver 0.03 ±100% +0.1 0.09 ± 23% perf-profile.children.cycles-pp.tick_nohz_idle_retain_tick 0.21 ± 10% +0.1 0.27 ± 9% perf-profile.children.cycles-pp.menu_reflect 0.03 ±102% +0.1 0.09 ± 24% perf-profile.children.cycles-pp.rcu_is_cpu_rrupt_from_idle 0.00 +0.1 0.07 ± 23% perf-profile.children.cycles-pp.alloc_tree_block_no_bg_flush 0.00 +0.1 0.07 ± 23% perf-profile.children.cycles-pp.btrfs_alloc_tree_block 0.19 ± 12% +0.1 0.26 ± 11% perf-profile.children.cycles-pp.rb_next 0.16 ± 14% +0.1 0.23 ± 10% perf-profile.children.cycles-pp.irqentry_exit 0.07 ± 11% +0.1 0.14 ± 25% perf-profile.children.cycles-pp.rcu_needs_cpu 0.03 ±102% +0.1 0.10 ± 30% perf-profile.children.cycles-pp.rwsem_spin_on_owner 0.01 ±173% +0.1 0.09 ± 45% perf-profile.children.cycles-pp.leave_mm 0.14 ± 19% +0.1 0.22 ± 16% perf-profile.children.cycles-pp.pm_qos_read_value 0.00 +0.1 0.08 ± 37% perf-profile.children.cycles-pp.cpu_latency_qos_limit 0.22 ± 18% +0.1 0.30 ± 14% perf-profile.children.cycles-pp.get_cpu_device 0.02 ±173% +0.1 0.10 ± 35% perf-profile.children.cycles-pp.sched_idle_set_state 0.07 ± 69% +0.1 0.15 ± 19% perf-profile.children.cycles-pp.hrtimer_forward 0.14 ± 10% +0.1 0.23 ± 14% perf-profile.children.cycles-pp.perf_event_task_tick 0.10 ± 15% +0.1 0.19 ± 9% perf-profile.children.cycles-pp.x86_pmu_disable 0.08 ± 67% +0.1 0.17 ± 15% perf-profile.children.cycles-pp.cpuidle_reflect 0.00 +0.1 0.09 ± 27% perf-profile.children.cycles-pp.check_leaf 0.07 ± 58% +0.1 0.17 ± 26% perf-profile.children.cycles-pp.run_local_timers 0.00 +0.1 0.10 ± 27% perf-profile.children.cycles-pp.btrfs_cow_block 0.00 +0.1 0.10 ± 27% perf-profile.children.cycles-pp.__btrfs_cow_block 0.00 +0.1 0.10 ± 25% perf-profile.children.cycles-pp.btree_csum_one_bio 0.17 ± 14% +0.1 0.27 ± 33% perf-profile.children.cycles-pp.__hrtimer_get_next_event 0.00 +0.1 0.10 ± 35% perf-profile.children.cycles-pp.write_one_eb 0.00 +0.1 0.10 ± 27% perf-profile.children.cycles-pp.btrfs_submit_metadata_bio 0.43 ± 13% +0.1 0.53 ± 5% perf-profile.children.cycles-pp.update_irq_load_avg 0.10 ± 18% +0.1 0.21 ± 14% perf-profile.children.cycles-pp.cpuidle_not_available 0.00 +0.1 0.11 ± 23% perf-profile.children.cycles-pp.submit_one_bio 0.16 ± 35% +0.1 0.27 ± 25% perf-profile.children.cycles-pp.wait_consider_task 0.16 ± 19% +0.1 0.28 ± 24% perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu 0.18 ± 32% +0.1 0.30 ± 19% perf-profile.children.cycles-pp.irqtime_account_process_tick 0.00 +0.1 0.12 ± 23% perf-profile.children.cycles-pp.btrfs_write_marked_extents 0.00 +0.1 0.12 ± 23% perf-profile.children.cycles-pp.btree_write_cache_pages 0.00 +0.1 0.13 ± 17% perf-profile.children.cycles-pp.do_writepages 0.00 +0.1 0.14 ± 19% perf-profile.children.cycles-pp.__filemap_fdatawrite_range 1.08 ± 11% +0.2 1.23 ± 7% perf-profile.children.cycles-pp.io_serial_in 0.39 ± 10% +0.2 0.55 ± 3% perf-profile.children.cycles-pp.arch_scale_freq_tick 0.00 +0.2 0.17 ± 55% perf-profile.children.cycles-pp.btrfs_log_changed_extents 0.42 ± 5% +0.2 0.63 ± 6% perf-profile.children.cycles-pp.timerqueue_del 0.32 ± 14% +0.2 0.54 ± 11% perf-profile.children.cycles-pp.timerqueue_add 0.72 ± 11% +0.2 0.95 ± 8% perf-profile.children.cycles-pp.update_blocked_averages 0.49 ± 8% +0.2 0.72 ± 13% perf-profile.children.cycles-pp.cpuidle_governor_latency_req 0.52 ± 6% +0.2 0.76 ± 7% perf-profile.children.cycles-pp.__remove_hrtimer 0.38 ± 16% +0.3 0.64 ± 13% perf-profile.children.cycles-pp.enqueue_hrtimer 0.96 ± 7% +0.3 1.22 ± 5% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.68 ± 11% +0.3 1.00 ± 11% perf-profile.children.cycles-pp.run_rebalance_domains 0.53 ± 11% +0.3 0.86 ± 13% perf-profile.children.cycles-pp.rcu_sched_clock_irq 1.22 ± 10% +0.3 1.56 ± 6% perf-profile.children.cycles-pp.tick_nohz_next_event 1.03 ± 6% +0.4 1.39 ± 13% perf-profile.children.cycles-pp.tick_irq_enter 1.38 ± 5% +0.4 1.78 ± 14% perf-profile.children.cycles-pp.irq_enter_rcu 1.64 ± 9% +0.4 2.08 ± 3% perf-profile.children.cycles-pp.scheduler_tick 1.81 ± 11% +0.6 2.36 ± 5% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 0.23 ± 18% +0.6 0.83 ± 36% perf-profile.children.cycles-pp.__btrfs_tree_read_lock 0.00 +0.7 0.72 ± 5% perf-profile.children.cycles-pp.wait_log_commit 0.06 ± 74% +0.8 0.81 ± 37% perf-profile.children.cycles-pp.rwsem_down_read_slowpath 0.00 +0.8 0.76 ± 3% perf-profile.children.cycles-pp.__mutex_lock 0.03 ±100% +0.8 0.79 ± 14% perf-profile.children.cycles-pp.__btrfs_tree_lock 0.03 ±100% +0.8 0.79 ± 14% perf-profile.children.cycles-pp.btrfs_lock_root_node 0.00 +0.8 0.79 ± 14% perf-profile.children.cycles-pp.rwsem_down_write_slowpath 0.00 +0.9 0.90 ± 5% perf-profile.children.cycles-pp.btrfs_sync_log 2.83 ± 6% +1.1 3.93 ± 3% perf-profile.children.cycles-pp.update_process_times 3.01 ± 6% +1.2 4.20 ± 4% perf-profile.children.cycles-pp.tick_sched_handle 3.54 ± 7% +1.3 4.85 ± 3% perf-profile.children.cycles-pp.tick_sched_timer 4.44 ± 14% +1.3 5.78 ± 10% perf-profile.children.cycles-pp.menu_select 1.15 ± 15% +1.4 2.54 ± 32% perf-profile.children.cycles-pp.start_kernel 0.08 ± 78% +1.5 1.60 ± 20% perf-profile.children.cycles-pp.rwsem_optimistic_spin 0.00 +1.7 1.71 ± 19% perf-profile.children.cycles-pp.btrfs_insert_empty_items 0.00 +1.9 1.87 ± 18% perf-profile.children.cycles-pp.btrfs_log_inode 0.00 +2.0 2.02 ± 17% perf-profile.children.cycles-pp.btrfs_log_inode_parent 0.00 +2.0 2.03 ± 17% perf-profile.children.cycles-pp.btrfs_log_dentry_safe 0.04 ±115% +2.2 2.23 ± 14% perf-profile.children.cycles-pp.osq_lock 5.92 ± 6% +2.3 8.26 ± 3% perf-profile.children.cycles-pp.__hrtimer_run_queues 8.53 ± 9% +2.5 11.05 ± 4% perf-profile.children.cycles-pp.hrtimer_interrupt 8.91 ± 9% +2.7 11.63 ± 5% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 0.00 +3.0 3.00 ± 11% perf-profile.children.cycles-pp.btrfs_sync_file 13.89 ± 9% +4.2 18.09 ± 6% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 15.08 ± 9% +4.4 19.51 ± 6% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 51.91 +8.8 60.72 ± 2% perf-profile.children.cycles-pp.intel_idle 79.10 ± 3% +10.8 89.91 perf-profile.children.cycles-pp.start_secondary 73.22 ± 2% +11.2 84.38 perf-profile.children.cycles-pp.cpuidle_enter_state 73.32 ± 2% +11.2 84.49 perf-profile.children.cycles-pp.cpuidle_enter 80.25 ± 2% +12.2 92.45 perf-profile.children.cycles-pp.secondary_startup_64_no_verify 80.25 ± 2% +12.2 92.45 perf-profile.children.cycles-pp.cpu_startup_entry 80.23 ± 2% +12.2 92.45 perf-profile.children.cycles-pp.do_idle 0.97 ± 2% -0.5 0.44 ± 19% perf-profile.self.cycles-pp._raw_spin_lock 0.36 ± 16% -0.3 0.05 ± 58% perf-profile.self.cycles-pp.__schedule 0.35 ± 8% -0.3 0.07 ± 13% perf-profile.self.cycles-pp.kmem_cache_alloc 0.12 ± 22% -0.1 0.03 ±100% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.15 ± 25% -0.1 0.07 ± 7% perf-profile.self.cycles-pp.update_load_avg 0.11 ± 20% -0.1 0.04 ± 58% perf-profile.self.cycles-pp.__update_load_avg_se 0.18 ± 14% -0.1 0.12 ± 32% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.08 ± 17% +0.0 0.10 ± 8% perf-profile.self.cycles-pp.cpuidle_enter 0.07 ± 15% +0.0 0.10 ± 21% perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit 0.07 ± 15% +0.0 0.11 ± 11% perf-profile.self.cycles-pp.rcu_nmi_exit 0.01 ±173% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.arch_cpu_idle_enter 0.04 ± 58% +0.0 0.08 ± 33% perf-profile.self.cycles-pp.perf_pmu_disable 0.10 ± 18% +0.0 0.14 ± 18% perf-profile.self.cycles-pp.irq_enter_rcu 0.12 ± 6% +0.0 0.17 ± 10% perf-profile.self.cycles-pp.menu_reflect 0.08 ± 14% +0.0 0.12 ± 6% perf-profile.self.cycles-pp.tick_program_event 0.03 ±100% +0.0 0.07 ± 27% perf-profile.self.cycles-pp.rcu_is_cpu_rrupt_from_idle 0.15 ± 18% +0.0 0.20 ± 7% perf-profile.self.cycles-pp.asm_call_sysvec_on_stack 0.06 ± 14% +0.1 0.11 ± 23% perf-profile.self.cycles-pp.rcu_segcblist_ready_cbs 0.01 ±173% +0.1 0.07 ± 26% perf-profile.self.cycles-pp.cpuidle_get_cpu_driver 0.20 ± 17% +0.1 0.27 ± 10% perf-profile.self.cycles-pp.get_next_timer_interrupt 0.10 ± 23% +0.1 0.17 ± 25% perf-profile.self.cycles-pp.__hrtimer_get_next_event 0.03 ±102% +0.1 0.10 ± 30% perf-profile.self.cycles-pp.rwsem_spin_on_owner 0.01 ±173% +0.1 0.08 ± 37% perf-profile.self.cycles-pp.sched_idle_set_state 0.22 ± 17% +0.1 0.29 ± 12% perf-profile.self.cycles-pp.scheduler_tick 0.17 ± 14% +0.1 0.24 ± 14% perf-profile.self.cycles-pp.rb_next 0.00 +0.1 0.07 ± 24% perf-profile.self.cycles-pp.tick_nohz_idle_retain_tick 0.13 ± 15% +0.1 0.21 ± 7% perf-profile.self.cycles-pp.irq_exit_rcu 0.00 +0.1 0.08 ± 37% perf-profile.self.cycles-pp.cpu_latency_qos_limit 0.22 ± 18% +0.1 0.30 ± 14% perf-profile.self.cycles-pp.get_cpu_device 0.10 ± 30% +0.1 0.19 ± 21% perf-profile.self.cycles-pp.tick_nohz_get_sleep_length 0.14 ± 10% +0.1 0.23 ± 14% perf-profile.self.cycles-pp.perf_event_task_tick 0.11 ± 26% +0.1 0.20 ± 12% perf-profile.self.cycles-pp.clockevents_program_event 0.09 ± 20% +0.1 0.18 ± 10% perf-profile.self.cycles-pp.x86_pmu_disable 0.05 ±102% +0.1 0.14 ± 16% perf-profile.self.cycles-pp.hrtimer_forward 0.10 ± 18% +0.1 0.20 ± 18% perf-profile.self.cycles-pp.cpuidle_not_available 0.05 ± 60% +0.1 0.14 ± 23% perf-profile.self.cycles-pp.run_local_timers 0.24 ± 8% +0.1 0.35 ± 19% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler 0.16 ± 18% +0.1 0.28 ± 24% perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu 0.17 ± 26% +0.1 0.30 ± 21% perf-profile.self.cycles-pp.irqtime_account_process_tick 0.40 ± 7% +0.1 0.55 ± 17% perf-profile.self.cycles-pp.tick_nohz_next_event 0.24 ± 10% +0.1 0.39 ± 6% perf-profile.self.cycles-pp.timerqueue_add 0.39 ± 10% +0.2 0.55 ± 3% perf-profile.self.cycles-pp.arch_scale_freq_tick 0.32 ± 12% +0.2 0.49 ± 9% perf-profile.self.cycles-pp.__hrtimer_run_queues 0.28 ± 13% +0.2 0.45 ± 9% perf-profile.self.cycles-pp.update_blocked_averages 0.40 ± 14% +0.2 0.65 ± 19% perf-profile.self.cycles-pp.rcu_sched_clock_irq 0.98 ± 10% +0.3 1.23 ± 7% perf-profile.self.cycles-pp.io_serial_in 0.04 ±113% +2.2 2.21 ± 14% perf-profile.self.cycles-pp.osq_lock 51.91 +8.8 60.71 ± 2% perf-profile.self.cycles-pp.intel_idle fio.write_bw_MBps 30 +----------------------------------------------------------------------+ |..+..+..+.+. +..+. : +. +..+..+ +..+..+..+..+..+.+..+..+..| 25 |-+ : : : : | | : : : : | | : : : : | 20 |-+ : : : : | | : : : : | 15 |-+ : : : : | | : : : : | 10 |-+ : : : : | | : : : : | | :: :: | 5 |-+ : : | | : : | 0 +----------------------------------------------------------------------+ fio.write_iops 8000 +--------------------------------------------------------------------+ |..+..+.+..+..+..+.+..+ +.+..+..+..+ +..+..+.+..+..+..+.+.. .| 7000 |-+ : : : : +. | 6000 |-+ : : : : | | : : : : | 5000 |-+ : : : : | | : : : : | 4000 |-+ : : : : | | : : : : | 3000 |-+ : : : : | 2000 |-+ : : : : | | : : :: | 1000 |-+ : : | | : : | 0 +--------------------------------------------------------------------+ fio.write_clat_mean_us 4.5e+08 +-----------------------------------------------------------------+ | O | 4e+08 |-+O O O O O O O O O O O O O O O O O O | 3.5e+08 |-+ | | | 3e+08 |-+ | 2.5e+08 |-+ | | | 2e+08 |-+ | 1.5e+08 |-+ | | | 1e+08 |-+ | 5e+07 |-+ | | | 0 +-----------------------------------------------------------------+ fio.write_clat_stddev 4.5e+07 +-----------------------------------------------------------------+ | O O O | 4e+07 |-+ O O | 3.5e+07 |-+ O O O O O O | | O | 3e+07 |-+O O O O O O O | 2.5e+07 |-+ | | | 2e+07 |-+ | 1.5e+07 |..+.+..+..+.+..+.+..+ +..+..+.+..+ +..+..+.+..+.+..+..+.+..| | : : : : | 1e+07 |-+ : : : : | 5e+06 |-+ : : : : | | :: :: | 0 +-----------------------------------------------------------------+ fio.write_clat_90__us 4.5e+08 +-----------------------------------------------------------------+ | O O O O O O O O O O O O O | 4e+08 |-+ | 3.5e+08 |-+ | | | 3e+08 |-+ | 2.5e+08 |-+ | | | 2e+08 |-+ | 1.5e+08 |-+ | | | 1e+08 |-+ | 5e+07 |-+ | | | 0 +-----------------------------------------------------------------+ fio.write_clat_95__us 5e+08 +-----------------------------------------------------------------+ 4.5e+08 |-+ O O | | O O O O O O O O O O O O O O O O O | 4e+08 |-+ | 3.5e+08 |-+ | | | 3e+08 |-+ | 2.5e+08 |-+ | 2e+08 |-+ | | | 1.5e+08 |-+ | 1e+08 |-+ | | | 5e+07 |..+.+..+..+.+..+.+..+.. .+..+..+.+..+.. .+..+..+.+..+.+..+..+.+..| 0 +-----------------------------------------------------------------+ fio.write_clat_99__us 5e+08 +-----------------------------------------------------------------+ 4.5e+08 |-+O O O O O O O O O O O O O O O O O | | O O | 4e+08 |-+ | 3.5e+08 |-+ | | | 3e+08 |-+ | 2.5e+08 |-+ | 2e+08 |-+ | | | 1.5e+08 |-+ | 1e+08 |-+ | | .+..| 5e+07 |..+.+..+..+.+..+.+..+.. .+..+..+.+..+.. .+..+..+.+..+.+..+..+ | 0 +-----------------------------------------------------------------+ fio.latency_500ms_ 100 +---------------------------------------------------------------------+ 90 |-+ | | | 80 |-+ | 70 |-+ | | | 60 |-+ | 50 |-+ | 40 |-+ | | | 30 |-+ | 20 |-+ | | | 10 |-+ | 0 +---------------------------------------------------------------------+ fio.workload 2.5e+06 +-----------------------------------------------------------------+ | .+.. | |..+.+..+..+.+..+.+..+ +. +.+..+ +..+..+.+..+.+..+..+.+..| 2e+06 |-+ : : : : | | : : : : | | : : : : | 1.5e+06 |-+ : : : : | | : : : : | 1e+06 |-+ : : : : | | : : : : | | : : : : | 500000 |-+ : : : : | | : : | | : : | 0 +-----------------------------------------------------------------+ fio.time.user_time 16 +----------------------------------------------------------------------+ | : : | 14 |-+ : : | 12 |-+ +.. +.. : : | | +. .. +.. .+ .. + : +.. .+.. .+.. .+..+.. | 10 |.. .. + +. : + + : +. +. + +..| | +..+ : : : : | 8 |-+ : : : : | | : : : : | 6 |-+ : : : : | 4 |-+ : : : : | | : : : : | 2 |-+O O O O :: O O O :: O | | O O O : O O : O O O O | 0 +----------------------------------------------------------------------+ fio.time.system_time 70 +----------------------------------------------------------------------+ | .+.. +..+.. + .+..+.. | 60 |.+ +..+.+..+..+..+..+ : +.. ..: +..+..+..+..+..+ +..| | : : + : : | 50 |-+ : : : : | | : : : : | 40 |-+ : : : : | | : : : : | 30 |-+ : : : : | | O O : : O O :O:O O O O O | 20 |-+O O O O : : O : : | | O :: O O : : | 10 |-+ : : | | : : | 0 +----------------------------------------------------------------------+ fio.time.percent_of_cpu_this_job_got 30 +----------------------------------------------------------------------+ | | 25 |-+ .+.. .+ | |..+..+..+.+..+..+..+..+ +. +..+. : +..+..+..+..+..+.+..+..+..| | : : : : | 20 |-+ : : : : | | : : : : | 15 |-+ : : : : | | : : : : | 10 |-+ : : : : | | O O : : O O :O:O O O O O | | O O O O O :O:O O O : : | 5 |-+ :: : | | : : | 0 +----------------------------------------------------------------------+ fio.time.voluntary_context_switches 2.5e+06 +-----------------------------------------------------------------+ |..+.+..+.. .+..+. .+ .+.. .+ +..+..+.+..+.+.. | | + +. : +. +.+. : : +..+.+..| 2e+06 |-+ : : : : | | : : : : | | : : : : | 1.5e+06 |-+ : : : : | | : : : : | 1e+06 |-+ : : : : | | : : : : | | : : : : | 500000 |-+ : : :: | | : : | | O O O O O O O : O O O O O : O O O O O | 0 +-----------------------------------------------------------------+ fio.time.involuntary_context_switches 3000 +--------------------------------------------------------------------+ | | 2500 |-+ O O O O | | O O O O | | O O O | 2000 |-+ O O O O | | O O O O | 1500 |-+ | | | 1000 |-+ | | | | +.. +.. | 500 |-.+..+.+.. +. .. + .+..| |. +..+..+.+..+.. .. +..+..+..+. .+ + +..+..+.+. | 0 +--------------------------------------------------------------------+ fio.time.file_system_outputs 1.8e+07 +-----------------------------------------------------------------+ | + +. : : +.+. : : +..| 1.6e+07 |-+ : : : : | 1.4e+07 |-+ : : : : | | : : : : | 1.2e+07 |-+ : : : : | 1e+07 |-+ : : : : | | : : : : | 8e+06 |-+ : : : : | 6e+06 |-+ : : : : | | : : : : | 4e+06 |-+ :: :: | 2e+06 |-+ : : | | O O O O O O O : O O O O O : O O O O O | 0 +-----------------------------------------------------------------+ perf-sched.total_wait_time.average.ms 160 +---------------------------------------------------------------------+ | O O | 140 |-+ O O O O O O | 120 |-+ O O O O O | | O O O O O | 100 |-+ O | | | 80 |-+ | | | 60 |-+ | 40 |-+ | | | 20 |-+ | |..+..+.+..+..+..+..+.+.. .+..+..+.+..+.. .+..+.+..+..+..+..+.+..+..| 0 +---------------------------------------------------------------------+ perf-sched.total_wait_and_delay.count.ms 250000 +------------------------------------------------------------------+ | | | .+.. .+..+.. .+ +.+.. +..+ .+.. | 200000 |..+.+. + + : : .. : .+. .+..+ +.. | | : : + : +. +. + :| | : : : : : :| 150000 |-+ : : : : : : | | : : : : + | 100000 |-+ : : : : | | : : : : | | : : : : | 50000 |-+ : : :: | | :: :: | | O O O O O O O : O O O O O : O O O O O | 0 +------------------------------------------------------------------+ perf-sched.total_wait_and_delay.average.ms 160 +---------------------------------------------------------------------+ | O O | 140 |-+ O O O O O O | 120 |-+ O O O O O | | O O O O O | 100 |-+ O | | | 80 |-+ | | | 60 |-+ | 40 |-+ | | | 20 |-+ | |..+..+.+..+..+..+..+.+.. .+..+..+.+..+.. .+..+.+..+..+..+..+.+..+..| 0 +---------------------------------------------------------------------+ perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork 300 +---------------------------------------------------------------------+ | O O O O O O | 250 |-+O O O O O O O O O O O O O | | | | | 200 |-+ | | | 150 |-+ | | | 100 |-+ | | | | | 50 |-+ | | | 0 +---------------------------------------------------------------------+ perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork 140000 +------------------------------------------------------------------+ | +.+.. | 120000 |-.+.+..+..+.+..+.. .+ : .+..+ .+. .+..+.+.. | |. + : : +. : +. +. +.. :| 100000 |-+ : : : : + :| | : : : : : : | 80000 |-+ : : : : : : | | : : : : + | 60000 |-+ : : : : | | : : : : | 40000 |-+ : : : : | | : : :: | 20000 |-+ : :: | | : : | 0 +------------------------------------------------------------------+ perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork 300 +---------------------------------------------------------------------+ | O O O O O O | 250 |-+O O O O O O O O O O O O O | | | | | 200 |-+ | | | 150 |-+ | | | 100 |-+ | | | | | 50 |-+ | | | 0 +---------------------------------------------------------------------+ fio.latency_2ms_ 0.01 +-------------------------------------------------------------------+ | : : : : | | : : : : | 0.008 |-+ : : : : | | : : : : | | : : : : | 0.006 |-+ : : : : | | : : : : | 0.004 |-+ : : : : | | : : : : | | : : : : | 0.002 |-+ :: : : | | : : | | : : | 0 +-------------------------------------------------------------------+ fio.latency_4ms_ 60 +----------------------------------------------------------------------+ |..+..+..+.+..+..+..+..+ +..+..+..+..+ +..+..+..+..+..+.+..+..+..| 50 |-+ : : : : | | : : : : | | : : : : | 40 |-+ : : : : | | : : : : | 30 |-+ : : : : | | : : : : | 20 |-+ : : : : | | : : : : | | :: :: | 10 |-+ : : | | : : | 0 +----------------------------------------------------------------------+ fio.latency_10ms_ 35 +----------------------------------------------------------------------+ | .+..+..+.+..+..+..+..+ +..+.. .+ .+.+..+.. .| 30 |.+ : : +..+. : +..+..+..+..+. +. | | : : : : | 25 |-+ : : : : | | : : : : | 20 |-+ : : : : | | : : : : | 15 |-+ : : : : | | : : : : | 10 |-+ : : : : | | :: : : | 5 |-+ : : | | : : | 0 +----------------------------------------------------------------------+ fio.latency_20ms_ 9 +-----------------------------------------------------------------------+ |. .+.. .+.. .+..+ +.. .+.+..+ +. +. +.+..+..+.. .| 8 |-+ +. + +. : : +. : : +. | 7 |-+ : : : : | | : : : : | 6 |-+ : : : : | 5 |-+ : : : : | | : : : : | 4 |-+ : : : : | 3 |-+ : : : : | | : : : : | 2 |-+ : : : : | 1 |-+ : : | | : : | 0 +-----------------------------------------------------------------------+ fio.latency_50ms_ 6 +-----------------------------------------------------------------------+ | +.. .+.. | 5 |..+..+..+..+.+..+..+.. +..+.. .+..+ : +. +..+.+..+.. .+..| | + : + : : +. | | : : : : | 4 |-+ : : : : | | : : : : | 3 |-+ : : : : | | : : : : | 2 |-+ : : : : | | : : : : | | : : : : | 1 |-+ : : : | | : : | 0 +-----------------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Oliver Sang