Greeting, FYI, we noticed a 40.1% improvement of fsmark.files_per_sec due to commit: commit: 853137cef46ccc490e6fd4b160a1c252d6459842 ("f2fs: fix performance issue observed with multi-thread sequential read") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master in testcase: fsmark on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory with following parameters: iterations: 1x nr_threads: 64t disk: 1BRD_48G fs: f2fs filesize: 4M test_size: 40G sync_method: fsyncBeforeClose ucode: 0x42d cpufreq_governor: performance test-description: The fsmark is a file system benchmark to test synchronous write workloads, for example, mail servers workload. test-url: https://sourceforge.net/projects/fsmark/ Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone https://github.com/intel/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/disk/filesize/fs/iterations/kconfig/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase/ucode: gcc-7/performance/1BRD_48G/4M/f2fs/1x/x86_64-rhel-7.2/64t/debian-x86_64-2018-04-03.cgz/fsyncBeforeClose/ivb44/40G/fsmark/0x42d commit: dda9f4b9ca ("f2fs: fix to skip verifying block address for non-regular inode") 853137cef4 ("f2fs: fix performance issue observed with multi-thread sequential read") dda9f4b9cac6bdd2 853137cef46ccc490e6fd4b160 ---------------- -------------------------- fail:runs %reproduction fail:runs | | | :4 25% 1:4 dmesg.WARNING:at#for_ip_ret_from_intr/0x 4:4 -100% :4 dmesg.WARNING:stack_going_in_the_wrong_direction?ip=rwsem_down_write_failed/0x 1:4 -25% :4 kmsg.pstore:crypto_comp_decompress_failed,ret= 1:4 -25% :4 kmsg.pstore:decompression_failed %stddev %change %stddev \ | \ 1292197 -19.9% 1034902 fsmark.app_overhead 265.67 +40.1% 372.30 ± 3% fsmark.files_per_sec 41.39 -24.2% 31.38 ± 2% fsmark.time.elapsed_time 41.39 -24.2% 31.38 ± 2% fsmark.time.elapsed_time.max 27468 -99.3% 205.75 ± 9% fsmark.time.involuntary_context_switches 79094 -6.3% 74088 fsmark.time.minor_page_faults 3837 -91.5% 324.50 ± 5% fsmark.time.percent_of_cpu_this_job_got 1587 -93.6% 101.14 ± 8% fsmark.time.system_time 1235762 ± 2% -98.9% 13888 ± 6% fsmark.time.voluntary_context_switches 59558 -6.5% 55713 interrupts.CAL:Function_call_interrupts 23.54 ± 9% +67.9 91.48 mpstat.cpu.idle% 0.04 ± 3% -0.0 0.00 ± 23% mpstat.cpu.iowait% 74.31 ± 2% -67.9 6.39 ± 7% mpstat.cpu.sys% 277783 ± 3% -35.8% 178384 softirqs.RCU 283065 ± 4% -7.4% 262224 ± 6% softirqs.SCHED 1134400 ± 7% -14.5% 969409 ± 5% softirqs.TIMER 920993 ± 2% +28.3% 1181660 ± 5% vmstat.io.bo 38.50 ± 7% -89.0% 4.25 ± 10% vmstat.procs.r 52008 ± 2% -95.5% 2337 ± 4% vmstat.system.cs 57273 -11.9% 50430 vmstat.system.in 276.98 ± 14% -18.5% 225.62 ± 19% sched_debug.cfs_rq:/.load_avg.stddev 5303 ± 16% -25.6% 3943 ± 26% sched_debug.cfs_rq:/.min_vruntime.avg 9599 ±105% -66.0% 3268 ± 8% sched_debug.cfs_rq:/.runnable_weight.avg 42652 ±144% -84.1% 6788 ± 4% sched_debug.cfs_rq:/.runnable_weight.stddev 49.18 ± 26% -41.0% 29.03 ± 21% sched_debug.cfs_rq:/.util_est_enqueued.avg 191284 ± 10% +17.9% 225588 ± 5% meminfo.DirectMap4k 158559 ± 2% +51.5% 240138 ± 3% meminfo.Dirty 225168 ±159% -93.2% 15267 meminfo.Inactive(anon) 27200 -10.4% 24381 meminfo.Mapped 412653 ± 2% -59.4% 167408 ± 2% meminfo.SReclaimable 305084 ±147% -93.6% 19584 meminfo.Shmem 523258 -48.4% 270220 meminfo.Slab 151.00 ± 79% +830.1% 1404 ± 2% meminfo.Writeback 52991739 ± 3% -95.7% 2255645 ± 11% cpuidle.C1.time 533066 ± 3% -94.7% 28389 ± 17% cpuidle.C1.usage 32907668 ± 2% -98.4% 537881 ± 4% cpuidle.C1E.time 243722 ± 2% -98.5% 3607 cpuidle.C1E.usage 54974737 ± 2% -97.7% 1264852 ± 2% cpuidle.C3.time 316006 ± 2% -98.0% 6351 cpuidle.C3.usage 3.284e+08 ± 15% +348.6% 1.473e+09 ± 5% cpuidle.C6.time 475703 ± 12% +229.0% 1565046 ± 6% cpuidle.C6.usage 53413 ± 5% -96.2% 2007 ± 22% cpuidle.POLL.time 1051 ± 4% -95.0% 52.75 ± 15% cpuidle.POLL.usage 217102 ± 2% -26.3% 159989 ± 22% numa-meminfo.node0.Active 196552 ± 2% -31.7% 134228 ± 28% numa-meminfo.node0.Active(anon) 20549 +25.4% 25761 ± 11% numa-meminfo.node0.Active(file) 78355 ± 2% +49.7% 117324 ± 3% numa-meminfo.node0.Dirty 6462 ± 9% +16.6% 7532 ± 3% numa-meminfo.node0.KernelStack 2154 ± 17% +63.8% 3529 ± 19% numa-meminfo.node0.PageTables 206265 -54.4% 94031 ± 10% numa-meminfo.node0.SReclaimable 260986 -40.5% 155389 ± 7% numa-meminfo.node0.Slab 80164 ± 2% +52.9% 122541 ± 7% numa-meminfo.node1.Dirty 106012 ±168% -99.1% 943.50 ± 81% numa-meminfo.node1.Inactive(anon) 5978 ± 10% -19.4% 4815 ± 5% numa-meminfo.node1.KernelStack 11366 -13.2% 9870 numa-meminfo.node1.Mapped 3275 ± 12% -44.5% 1816 ± 35% numa-meminfo.node1.PageTables 207093 ± 3% -64.6% 73366 ± 11% numa-meminfo.node1.SReclaimable 55885 ± 4% -25.8% 41453 ± 8% numa-meminfo.node1.SUnreclaim 262979 -56.3% 114819 ± 9% numa-meminfo.node1.Slab 2332 ± 2% -86.7% 309.25 ± 4% turbostat.Avg_MHz 77.98 ± 2% -68.1 9.89 ± 4% turbostat.Busy% 526510 ± 3% -95.7% 22547 ± 17% turbostat.C1 2.50 ± 4% -2.4 0.12 ± 8% turbostat.C1% 243631 ± 2% -98.6% 3530 ± 2% turbostat.C1E 1.55 ± 2% -1.5 0.03 ± 13% turbostat.C1E% 315854 ± 2% -98.0% 6228 turbostat.C3 2.60 ± 2% -2.5 0.08 ± 5% turbostat.C3% 473755 ± 12% +230.0% 1563267 ± 6% turbostat.C6 15.40 ± 13% +74.4 89.83 turbostat.C6% 13.39 ± 4% +146.9% 33.08 turbostat.CPU%c1 0.49 ± 5% -95.9% 0.02 turbostat.CPU%c3 8.13 ± 18% +601.5% 57.02 turbostat.CPU%c6 152.73 -40.7% 90.60 turbostat.CorWatt 2578350 ± 2% -31.6% 1764094 ± 5% turbostat.IRQ 182.65 -34.2% 120.19 turbostat.PkgWatt 10.95 -6.3% 10.26 ± 2% turbostat.RAMWatt 49108 ± 2% -31.7% 33562 ± 28% numa-vmstat.node0.nr_active_anon 5168 ± 2% +24.4% 6429 ± 11% numa-vmstat.node0.nr_active_file 19426 ± 3% +50.2% 29174 ± 3% numa-vmstat.node0.nr_dirty 6462 ± 9% +16.6% 7534 ± 3% numa-vmstat.node0.nr_kernel_stack 538.25 ± 17% +64.1% 883.50 ± 18% numa-vmstat.node0.nr_page_table_pages 51684 ± 2% -54.4% 23550 ± 10% numa-vmstat.node0.nr_slab_reclaimable 17.75 ± 59% +752.1% 151.25 ± 14% numa-vmstat.node0.nr_writeback 49109 ± 2% -31.7% 33563 ± 28% numa-vmstat.node0.nr_zone_active_anon 5164 ± 2% +24.6% 6432 ± 11% numa-vmstat.node0.nr_zone_active_file 18336 ± 3% +59.9% 29314 ± 3% numa-vmstat.node0.nr_zone_write_pending 20180 ± 2% +51.6% 30586 ± 6% numa-vmstat.node1.nr_dirty 26503 ±168% -99.1% 235.25 ± 82% numa-vmstat.node1.nr_inactive_anon 5987 ± 11% -19.4% 4825 ± 5% numa-vmstat.node1.nr_kernel_stack 2970 ± 2% -15.9% 2498 ± 2% numa-vmstat.node1.nr_mapped 820.75 ± 12% -44.8% 452.75 ± 35% numa-vmstat.node1.nr_page_table_pages 51894 ± 3% -64.6% 18375 ± 11% numa-vmstat.node1.nr_slab_reclaimable 13978 ± 4% -25.9% 10363 ± 8% numa-vmstat.node1.nr_slab_unreclaimable 13.75 ±110% +1201.8% 179.00 ± 9% numa-vmstat.node1.nr_writeback 26503 ±168% -99.1% 235.25 ± 82% numa-vmstat.node1.nr_zone_inactive_anon 19082 ± 2% +61.0% 30724 ± 6% numa-vmstat.node1.nr_zone_write_pending 1363 ± 2% -28.5% 975.25 ± 20% slabinfo.UNIX.active_objs 1363 ± 2% -28.5% 975.25 ± 20% slabinfo.UNIX.num_objs 6207 ± 4% -9.2% 5639 slabinfo.cred_jar.active_objs 6207 ± 4% -9.2% 5639 slabinfo.cred_jar.num_objs 4071714 -99.9% 5808 ± 6% slabinfo.ext4_io_end.active_objs 63623 -99.9% 91.50 ± 6% slabinfo.ext4_io_end.active_slabs 4071916 -99.9% 5886 ± 6% slabinfo.ext4_io_end.num_objs 63623 -99.9% 91.50 ± 6% slabinfo.ext4_io_end.num_slabs 600.00 ± 10% -18.3% 490.00 ± 3% slabinfo.file_lock_cache.active_objs 600.00 ± 10% -18.3% 490.00 ± 3% slabinfo.file_lock_cache.num_objs 850.50 ± 8% -21.0% 672.00 ± 6% slabinfo.mnt_cache.active_objs 850.50 ± 8% -21.0% 672.00 ± 6% slabinfo.mnt_cache.num_objs 1139 ± 2% -22.1% 887.75 ± 14% slabinfo.nsproxy.active_objs 1139 ± 2% -22.1% 887.75 ± 14% slabinfo.nsproxy.num_objs 2990 -21.8% 2339 ± 16% slabinfo.pool_workqueue.active_objs 2991 -21.8% 2339 ± 16% slabinfo.pool_workqueue.num_objs 404.25 ± 14% -24.0% 307.25 ± 11% slabinfo.secpath_cache.active_objs 404.25 ± 14% -24.0% 307.25 ± 11% slabinfo.secpath_cache.num_objs 890.00 ± 3% -21.1% 702.00 ± 12% slabinfo.skbuff_fclone_cache.active_objs 890.00 ± 3% -21.1% 702.00 ± 12% slabinfo.skbuff_fclone_cache.num_objs 2565 ± 2% -21.6% 2011 ± 12% slabinfo.sock_inode_cache.active_objs 2565 ± 2% -21.6% 2011 ± 12% slabinfo.sock_inode_cache.num_objs 343.00 ± 2% +12.6% 386.25 ± 4% proc-vmstat.kswapd_low_wmark_hit_quickly 95521 ± 25% -18.9% 77469 ± 2% proc-vmstat.nr_active_anon 39667 +51.1% 59943 ± 3% proc-vmstat.nr_dirty 56291 ±159% -93.2% 3816 proc-vmstat.nr_inactive_anon 6927 -10.1% 6227 proc-vmstat.nr_mapped 1356 -1.5% 1336 proc-vmstat.nr_page_table_pages 76271 ±147% -93.6% 4895 proc-vmstat.nr_shmem 103211 ± 2% -59.4% 41881 ± 2% proc-vmstat.nr_slab_reclaimable 27650 ± 6% -7.0% 25702 proc-vmstat.nr_slab_unreclaimable 25.75 ±115% +1242.7% 345.75 ± 4% proc-vmstat.nr_writeback 95523 ± 25% -18.9% 77470 ± 2% proc-vmstat.nr_zone_active_anon 56291 ±159% -93.2% 3816 proc-vmstat.nr_zone_inactive_anon 37490 +60.6% 60220 ± 3% proc-vmstat.nr_zone_write_pending 565177 ± 9% +1112.8% 6854385 ± 37% proc-vmstat.numa_foreign 20889636 -31.0% 14423625 ± 17% proc-vmstat.numa_hit 20880137 -31.0% 14414150 ± 17% proc-vmstat.numa_local 565177 ± 9% +1112.8% 6854385 ± 37% proc-vmstat.numa_miss 574675 ± 9% +1094.4% 6863860 ± 37% proc-vmstat.numa_other 21071 ± 3% -43.4% 11934 proc-vmstat.pgactivate 239238 -12.6% 209196 ± 3% proc-vmstat.pgfault 10891256 -1.6% 10721631 proc-vmstat.pgfree 5349612 -1.4% 5272402 proc-vmstat.pgscan_kswapd 5339340 -1.4% 5262072 proc-vmstat.pgsteal_kswapd 11982464 -99.7% 34464 ± 8% proc-vmstat.slabs_scanned 5.343e+11 -85.4% 7.817e+10 ± 25% perf-stat.branch-instructions 0.59 ± 8% +2.3 2.93 ± 48% perf-stat.branch-miss-rate% 51.17 +11.5 62.62 ± 2% perf-stat.cache-miss-rate% 2.179e+09 -20.2% 1.739e+09 ± 8% perf-stat.cache-misses 4.259e+09 -34.9% 2.774e+09 ± 6% perf-stat.cache-references 2370742 ± 2% -96.8% 77021 ± 3% perf-stat.context-switches 1.75 -20.5% 1.39 ± 14% perf-stat.cpi 4.909e+12 -89.9% 4.971e+11 ± 12% perf-stat.cpu-cycles 183130 -99.7% 629.75 ± 15% perf-stat.cpu-migrations 0.10 ± 20% +0.2 0.29 ± 21% perf-stat.dTLB-load-miss-rate% 8.137e+08 ± 20% -66.1% 2.762e+08 ± 2% perf-stat.dTLB-load-misses 8.484e+11 -88.3% 9.894e+10 ± 21% perf-stat.dTLB-loads 0.10 ± 20% -0.0 0.05 ± 12% perf-stat.dTLB-store-miss-rate% 46384489 ± 20% -50.9% 22757656 ± 7% perf-stat.dTLB-store-misses 56.39 +26.7 83.13 perf-stat.iTLB-load-miss-rate% 51241211 ± 3% -41.0% 30213199 ± 5% perf-stat.iTLB-load-misses 39625929 ± 3% -84.5% 6141009 ± 8% perf-stat.iTLB-loads 2.8e+12 -86.8% 3.707e+11 ± 25% perf-stat.instructions 54709 ± 3% -77.7% 12227 ± 24% perf-stat.instructions-per-iTLB-miss 0.57 +28.6% 0.73 ± 14% perf-stat.ipc 229749 -12.8% 200436 ± 3% perf-stat.minor-faults 39.56 -14.9 24.69 ± 22% perf-stat.node-load-miss-rate% 6.3e+08 -77.3% 1.431e+08 ± 27% perf-stat.node-load-misses 9.628e+08 -55.3% 4.3e+08 ± 7% perf-stat.node-loads 229759 -12.8% 200443 ± 3% perf-stat.page-faults 90.78 -85.3 5.44 ± 10% perf-profile.calltrace.cycles-pp.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page.__write_data_page.f2fs_write_cache_pages 91.40 -85.3 6.13 ± 10% perf-profile.calltrace.cycles-pp.f2fs_outplace_write_data.f2fs_do_write_data_page.__write_data_page.f2fs_write_cache_pages.f2fs_write_data_pages 92.24 -84.5 7.76 ± 10% perf-profile.calltrace.cycles-pp.f2fs_do_write_data_page.__write_data_page.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages 92.49 -84.3 8.21 ± 10% perf-profile.calltrace.cycles-pp.__write_data_page.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range 92.80 -80.0 12.78 ± 9% perf-profile.calltrace.cycles-pp.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range 92.80 -55.5 37.32 ± 6% perf-profile.calltrace.cycles-pp.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.f2fs_do_sync_file 92.80 -55.5 37.32 ± 6% perf-profile.calltrace.cycles-pp.file_write_and_wait_range.f2fs_do_sync_file.do_fsync.__x64_sys_fsync.do_syscall_64 92.80 -55.5 37.32 ± 6% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.f2fs_do_sync_file.do_fsync.__x64_sys_fsync 92.80 -55.5 37.32 ± 6% perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.f2fs_do_sync_file.do_fsync 92.90 -55.4 37.47 ± 5% perf-profile.calltrace.cycles-pp.do_fsync.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe 92.90 -55.4 37.47 ± 5% perf-profile.calltrace.cycles-pp.f2fs_do_sync_file.do_fsync.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe 92.90 -55.4 37.47 ± 5% perf-profile.calltrace.cycles-pp.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe 55.19 -55.2 0.00 perf-profile.calltrace.cycles-pp.down_write.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page 55.16 -55.2 0.00 perf-profile.calltrace.cycles-pp.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.f2fs_submit_page_write.do_write_page 55.16 -55.2 0.00 perf-profile.calltrace.cycles-pp.call_rwsem_down_write_failed.down_write.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data 53.73 -53.7 0.00 perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_failed.call_rwsem_down_write_failed.down_write.f2fs_submit_page_write 56.61 -52.2 4.45 ± 9% perf-profile.calltrace.cycles-pp.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page.__write_data_page 94.91 -47.8 47.13 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 94.91 -47.7 47.17 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 33.80 -32.9 0.88 ± 17% perf-profile.calltrace.cycles-pp.f2fs_allocate_data_block.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page.__write_data_page 32.79 -32.8 0.00 perf-profile.calltrace.cycles-pp.__mutex_lock.f2fs_allocate_data_block.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page 31.91 -31.9 0.00 perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.f2fs_allocate_data_block.do_write_page.f2fs_outplace_write_data 0.00 +0.6 0.59 ± 9% perf-profile.calltrace.cycles-pp.__test_set_page_writeback.f2fs_do_write_data_page.__write_data_page.f2fs_write_cache_pages.f2fs_write_data_pages 0.00 +0.6 0.60 ± 11% perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt 0.00 +0.7 0.68 ± 15% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.pagecache_get_page.f2fs_write_begin.generic_perform_write.__generic_file_write_iter 0.00 +0.8 0.79 ± 16% perf-profile.calltrace.cycles-pp.f2fs_map_blocks.f2fs_preallocate_blocks.f2fs_file_write_iter.__vfs_write.vfs_write 0.00 +0.8 0.80 ± 8% perf-profile.calltrace.cycles-pp.rebalance_domains.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt 0.00 +0.8 0.81 ± 15% perf-profile.calltrace.cycles-pp.f2fs_preallocate_blocks.f2fs_file_write_iter.__vfs_write.vfs_write.ksys_write 0.00 +0.9 0.89 ± 7% perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry 0.00 +0.9 0.91 ± 17% perf-profile.calltrace.cycles-pp.f2fs_set_data_page_dirty.f2fs_write_end.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter 0.00 +0.9 0.95 ± 9% perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.f2fs_write_begin.generic_perform_write.__generic_file_write_iter 0.00 +1.0 1.02 ± 20% perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt 0.00 +1.1 1.13 ± 7% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary 0.00 +1.2 1.17 ± 19% perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt 0.64 ± 20% +1.3 1.92 ± 15% perf-profile.calltrace.cycles-pp.f2fs_write_end.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.__vfs_write 0.00 +1.3 1.30 ± 18% perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 0.00 +1.8 1.81 ± 11% perf-profile.calltrace.cycles-pp.pagecache_get_page.f2fs_write_begin.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter 0.00 +1.9 1.89 ± 10% perf-profile.calltrace.cycles-pp.f2fs_write_begin.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.__vfs_write 0.00 +2.0 2.03 ± 10% perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 0.00 +2.3 2.31 ± 14% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state 0.00 +2.3 2.33 ± 10% perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle 0.00 +2.3 2.33 ± 6% perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 1.09 ± 2% +2.8 3.94 ± 9% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.__submit_merged_bio.f2fs_submit_page_write.do_write_page 1.09 ± 2% +2.8 3.94 ± 9% perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.__submit_merged_bio.f2fs_submit_page_write 1.09 ± 2% +2.9 3.94 ± 9% perf-profile.calltrace.cycles-pp.submit_bio.__submit_merged_bio.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data 1.09 ± 2% +2.9 3.94 ± 9% perf-profile.calltrace.cycles-pp.__submit_merged_bio.f2fs_submit_page_write.do_write_page.f2fs_outplace_write_data.f2fs_do_write_data_page 0.00 +3.0 3.01 ± 12% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle 0.00 +3.1 3.11 ± 10% perf-profile.calltrace.cycles-pp.clear_page_erms.get_page_from_freelist.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec 0.00 +3.5 3.51 ± 8% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec.brd_make_request 0.00 +3.6 3.58 ± 8% perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.brd_insert_page.brd_do_bvec.brd_make_request.generic_make_request 0.13 ±173% +3.7 3.82 ± 8% perf-profile.calltrace.cycles-pp.iov_iter_copy_from_user_atomic.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.__vfs_write 0.00 +3.8 3.77 ± 8% perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.__generic_file_write_iter 0.00 +3.8 3.78 ± 8% perf-profile.calltrace.cycles-pp.copyin.iov_iter_copy_from_user_atomic.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter 0.00 +3.9 3.87 ± 7% perf-profile.calltrace.cycles-pp.submit_bio.__submit_merged_bio.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages 0.00 +3.9 3.87 ± 7% perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.__submit_merged_bio.__submit_merged_write_cond.f2fs_write_cache_pages 0.00 +3.9 3.87 ± 7% perf-profile.calltrace.cycles-pp.__submit_merged_bio.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages 0.00 +3.9 3.87 ± 7% perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.submit_bio.__submit_merged_bio.__submit_merged_write_cond 0.00 +3.9 3.88 ± 7% perf-profile.calltrace.cycles-pp.__submit_merged_write_cond.f2fs_write_cache_pages.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range 0.55 ± 6% +4.3 4.81 ± 9% perf-profile.calltrace.cycles-pp.brd_insert_page.brd_do_bvec.brd_make_request.generic_make_request.submit_bio 0.00 +5.9 5.94 ± 10% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry 0.00 +6.0 5.96 ± 10% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 0.92 ± 3% +6.2 7.14 ± 8% perf-profile.calltrace.cycles-pp.brd_do_bvec.brd_make_request.generic_make_request.submit_bio.__submit_merged_bio 1.50 ± 10% +6.4 7.90 ± 10% perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.f2fs_file_write_iter.__vfs_write.vfs_write 1.51 ± 10% +6.4 7.95 ± 10% perf-profile.calltrace.cycles-pp.__generic_file_write_iter.f2fs_file_write_iter.__vfs_write.vfs_write.ksys_write 1.84 ± 11% +7.1 8.93 ± 10% perf-profile.calltrace.cycles-pp.f2fs_file_write_iter.__vfs_write.vfs_write.ksys_write.do_syscall_64 1.85 ± 11% +7.1 8.95 ± 10% perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.87 ± 11% +7.2 9.08 ± 10% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.88 ± 11% +7.3 9.13 ± 10% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +12.0 12.04 ± 28% perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range 0.00 +12.4 12.45 ± 4% perf-profile.calltrace.cycles-pp.mutex_spin_on_owner.__mutex_lock.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range 0.00 +24.5 24.50 ± 12% perf-profile.calltrace.cycles-pp.__mutex_lock.f2fs_write_data_pages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range 3.82 ± 8% +32.8 36.61 ± 4% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary 3.95 ± 11% +39.3 43.22 ± 5% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 4.22 ± 11% +42.2 46.47 ± 5% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64 4.22 ± 11% +42.2 46.47 ± 5% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64 4.22 ± 11% +42.2 46.47 ± 5% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64 4.22 ± 11% +43.1 47.28 ± 4% perf-profile.calltrace.cycles-pp.secondary_startup_64 90.80 -85.3 5.46 ± 10% perf-profile.children.cycles-pp.do_write_page 91.41 -85.3 6.14 ± 10% perf-profile.children.cycles-pp.f2fs_outplace_write_data 92.24 -84.5 7.78 ± 10% perf-profile.children.cycles-pp.f2fs_do_write_data_page 92.50 -84.3 8.23 ± 10% perf-profile.children.cycles-pp.__write_data_page 92.80 -80.0 12.79 ± 9% perf-profile.children.cycles-pp.f2fs_write_cache_pages 85.66 -73.6 12.05 ± 28% perf-profile.children.cycles-pp.osq_lock 92.80 -55.5 37.32 ± 6% perf-profile.children.cycles-pp.f2fs_write_data_pages 92.80 -55.5 37.32 ± 6% perf-profile.children.cycles-pp.file_write_and_wait_range 92.80 -55.5 37.32 ± 6% perf-profile.children.cycles-pp.__filemap_fdatawrite_range 92.80 -55.5 37.32 ± 6% perf-profile.children.cycles-pp.do_writepages 92.91 -55.4 37.47 ± 5% perf-profile.children.cycles-pp.f2fs_do_sync_file 92.90 -55.4 37.47 ± 5% perf-profile.children.cycles-pp.do_fsync 92.90 -55.4 37.47 ± 5% perf-profile.children.cycles-pp.__x64_sys_fsync 55.18 -55.2 0.00 perf-profile.children.cycles-pp.rwsem_down_write_failed 55.18 -55.2 0.00 perf-profile.children.cycles-pp.call_rwsem_down_write_failed 55.32 -55.1 0.23 ± 11% perf-profile.children.cycles-pp.down_write 56.61 -52.1 4.47 ± 9% perf-profile.children.cycles-pp.f2fs_submit_page_write 95.03 -47.5 47.57 ± 4% perf-profile.children.cycles-pp.do_syscall_64 95.03 -47.4 47.61 ± 4% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 33.81 -32.9 0.91 ± 16% perf-profile.children.cycles-pp.f2fs_allocate_data_block 32.79 -8.3 24.51 ± 12% perf-profile.children.cycles-pp.__mutex_lock 0.38 ± 9% -0.3 0.09 ± 11% perf-profile.children.cycles-pp.try_to_wake_up 0.49 ± 6% -0.2 0.26 ± 16% perf-profile.children.cycles-pp.up_write 0.22 ± 11% -0.2 0.07 ± 22% perf-profile.children.cycles-pp.ttwu_do_activate 0.22 ± 9% -0.1 0.07 ± 24% perf-profile.children.cycles-pp.enqueue_task_fair 0.21 ± 11% -0.1 0.07 ± 24% perf-profile.children.cycles-pp.enqueue_entity 0.17 ± 13% -0.1 0.03 ±105% perf-profile.children.cycles-pp.__account_scheduler_latency 0.14 ± 15% -0.1 0.03 ±102% perf-profile.children.cycles-pp.save_stack_trace_tsk 0.18 ± 12% -0.1 0.07 ± 11% perf-profile.children.cycles-pp.__schedule 0.32 ± 5% -0.1 0.22 ± 9% perf-profile.children.cycles-pp.f2fs_get_node_info 0.30 ± 10% -0.1 0.22 ± 14% perf-profile.children.cycles-pp.down_read 0.10 ± 8% -0.1 0.03 ±100% perf-profile.children.cycles-pp.f2fs_lookup_rb_tree_ret 0.05 +0.0 0.07 ± 34% perf-profile.children.cycles-pp.f2fs_create 0.00 +0.1 0.05 ± 9% perf-profile.children.cycles-pp.read_node_page 0.00 +0.1 0.06 ± 14% perf-profile.children.cycles-pp.hrtimer_forward 0.00 +0.1 0.06 ± 6% perf-profile.children.cycles-pp.call_cpuidle 0.00 +0.1 0.06 ± 6% perf-profile.children.cycles-pp.sched_avg_update 0.00 +0.1 0.06 ± 6% perf-profile.children.cycles-pp.find_next_and_bit 0.07 ± 17% +0.1 0.14 ± 16% perf-profile.children.cycles-pp.up_read 0.00 +0.1 0.07 ± 23% perf-profile.children.cycles-pp.rcu_eqs_exit 0.00 +0.1 0.07 ± 21% perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited 0.00 +0.1 0.07 ± 17% perf-profile.children.cycles-pp.f2fs_set_node_page_dirty 0.00 +0.1 0.07 ± 10% perf-profile.children.cycles-pp.menu_reflect 0.00 +0.1 0.07 ± 22% perf-profile.children.cycles-pp.can_stop_idle_tick 0.00 +0.1 0.07 ± 29% perf-profile.children.cycles-pp.tick_check_broadcast_expired 0.00 +0.1 0.07 ± 26% perf-profile.children.cycles-pp.cpu_load_update_active 0.00 +0.1 0.07 ± 17% perf-profile.children.cycles-pp.__update_load_avg_se 0.00 +0.1 0.07 ± 5% perf-profile.children.cycles-pp.cpumask_next_and 0.00 +0.1 0.08 ± 20% perf-profile.children.cycles-pp.update_group_capacity 0.00 +0.1 0.08 ± 34% perf-profile.children.cycles-pp.perf_event_task_tick 0.00 +0.1 0.08 ± 16% perf-profile.children.cycles-pp.__lookup_nat_cache 0.00 +0.1 0.08 ± 14% perf-profile.children.cycles-pp._raw_spin_trylock 0.00 +0.1 0.08 ± 14% perf-profile.children.cycles-pp.tag_pages_for_writeback 0.00 +0.1 0.08 ± 19% perf-profile.children.cycles-pp.pm_qos_read_value 0.00 +0.1 0.08 ± 8% perf-profile.children.cycles-pp.__hrtimer_get_next_event 0.00 +0.1 0.08 ± 8% perf-profile.children.cycles-pp.pm_qos_request 0.00 +0.1 0.08 ± 5% perf-profile.children.cycles-pp.__intel_pmu_disable_all 0.00 +0.1 0.08 ± 5% perf-profile.children.cycles-pp.rb_erase 0.00 +0.1 0.08 ± 13% perf-profile.children.cycles-pp.hrtimer_get_next_event 0.00 +0.1 0.08 ± 21% perf-profile.children.cycles-pp.ktime_get_update_offsets_now 0.00 +0.1 0.09 ± 19% perf-profile.children.cycles-pp.alloc_pages_current 0.00 +0.1 0.09 ± 17% perf-profile.children.cycles-pp.note_gp_changes 0.00 +0.1 0.09 ± 20% perf-profile.children.cycles-pp.mem_cgroup_commit_charge 0.00 +0.1 0.09 ± 21% perf-profile.children.cycles-pp.__mod_node_page_state 0.00 +0.1 0.09 ± 20% perf-profile.children.cycles-pp.update_rq_clock 0.06 ± 6% +0.1 0.15 ± 11% perf-profile.children.cycles-pp.do_sys_open 0.06 ± 11% +0.1 0.15 ± 9% perf-profile.children.cycles-pp.do_filp_open 0.06 ± 11% +0.1 0.15 ± 9% perf-profile.children.cycles-pp.path_openat 0.00 +0.1 0.09 ± 20% perf-profile.children.cycles-pp.release_pages 0.00 +0.1 0.09 ± 20% perf-profile.children.cycles-pp.update_ts_time_stats 0.00 +0.1 0.09 ± 15% perf-profile.children.cycles-pp.rcu_needs_cpu 0.01 ±173% +0.1 0.11 ± 27% perf-profile.children.cycles-pp.handle_mm_fault 0.01 ±173% +0.1 0.11 ± 25% perf-profile.children.cycles-pp.__handle_mm_fault 0.00 +0.1 0.10 ± 28% perf-profile.children.cycles-pp.account_process_tick 0.00 +0.1 0.10 ± 13% perf-profile.children.cycles-pp.rcu_irq_exit 0.00 +0.1 0.10 ± 8% perf-profile.children.cycles-pp.__radix_tree_replace 0.00 +0.1 0.10 ± 39% perf-profile.children.cycles-pp.__vfs_read 0.00 +0.1 0.10 ± 20% perf-profile.children.cycles-pp.f2fs_update_data_blkaddr 0.00 +0.1 0.10 ± 20% perf-profile.children.cycles-pp.f2fs_set_data_blkaddr 0.00 +0.1 0.10 ± 17% perf-profile.children.cycles-pp.tick_nohz_irq_exit 0.00 +0.1 0.10 ± 14% perf-profile.children.cycles-pp.rcu_idle_exit 0.00 +0.1 0.10 ± 12% perf-profile.children.cycles-pp._cond_resched 0.00 +0.1 0.10 ± 8% perf-profile.children.cycles-pp.intel_pmu_disable_all 0.00 +0.1 0.10 ± 36% perf-profile.children.cycles-pp.ksys_read 0.00 +0.1 0.10 ± 36% perf-profile.children.cycles-pp.vfs_read 0.00 +0.1 0.10 ± 10% perf-profile.children.cycles-pp.mem_cgroup_try_charge 0.01 ±173% +0.1 0.12 ± 25% perf-profile.children.cycles-pp.__do_page_fault 0.01 ±173% +0.1 0.12 ± 25% perf-profile.children.cycles-pp.page_fault 0.01 ±173% +0.1 0.12 ± 25% perf-profile.children.cycles-pp.do_page_fault 0.00 +0.1 0.11 ± 6% perf-profile.children.cycles-pp.dec_zone_page_state 0.06 ± 34% +0.1 0.17 ± 32% perf-profile.children.cycles-pp.__mark_inode_dirty 0.00 +0.1 0.11 ± 22% perf-profile.children.cycles-pp.set_page_dirty 0.00 +0.1 0.11 ± 9% perf-profile.children.cycles-pp.rcu_eqs_enter 0.00 +0.1 0.12 ± 5% perf-profile.children.cycles-pp.__list_del_entry_valid 0.00 +0.1 0.12 ± 12% perf-profile.children.cycles-pp.ktime_get_real_ts64 0.05 ± 8% +0.1 0.18 ± 2% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.00 +0.1 0.13 ± 11% perf-profile.children.cycles-pp.cpu_load_update 0.01 ±173% +0.1 0.14 ± 8% perf-profile.children.cycles-pp.radix_tree_tag_set 0.00 +0.1 0.13 perf-profile.children.cycles-pp.node_dirty_ok 0.07 ± 14% +0.1 0.21 ± 21% perf-profile.children.cycles-pp.update_load_avg 0.00 +0.1 0.14 ± 15% perf-profile.children.cycles-pp.run_posix_cpu_timers 0.07 ± 7% +0.1 0.20 ± 6% perf-profile.children.cycles-pp.radix_tree_tag_clear 0.00 +0.1 0.14 ± 16% perf-profile.children.cycles-pp.get_cpu_device 0.00 +0.1 0.14 ± 15% perf-profile.children.cycles-pp.do_gettimeofday 0.00 +0.1 0.15 ± 7% perf-profile.children.cycles-pp.__might_sleep 0.00 +0.2 0.15 ± 14% perf-profile.children.cycles-pp.native_sched_clock 0.06 ± 6% +0.2 0.22 ± 18% perf-profile.children.cycles-pp.percpu_counter_add_batch 0.01 ±173% +0.2 0.18 ± 8% perf-profile.children.cycles-pp.pagevec_lookup_range_tag 0.01 ±173% +0.2 0.18 ± 11% perf-profile.children.cycles-pp.find_get_pages_range_tag 0.00 +0.2 0.17 ± 13% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.00 +0.2 0.17 ± 12% perf-profile.children.cycles-pp.timerqueue_del 0.00 +0.2 0.17 ± 9% perf-profile.children.cycles-pp.sched_clock 0.15 ± 33% +0.2 0.33 ± 17% perf-profile.children.cycles-pp.f2fs_reserve_new_blocks 0.01 ±173% +0.2 0.19 ± 20% perf-profile.children.cycles-pp.account_page_dirtied 0.00 +0.2 0.17 ± 2% perf-profile.children.cycles-pp.iov_iter_fault_in_readable 0.00 +0.2 0.18 ± 26% perf-profile.children.cycles-pp.__radix_tree_insert 0.15 ± 63% +0.2 0.33 ± 23% perf-profile.children.cycles-pp.f2fs_mark_inode_dirty_sync 0.00 +0.2 0.18 ± 14% perf-profile.children.cycles-pp.__radix_tree_create 0.00 +0.2 0.18 ± 17% perf-profile.children.cycles-pp.page_mapping 0.00 +0.2 0.18 ± 13% perf-profile.children.cycles-pp.timerqueue_add 0.00 +0.2 0.19 ± 11% perf-profile.children.cycles-pp.__indirect_thunk_start 0.00 +0.2 0.19 ± 9% perf-profile.children.cycles-pp.__remove_hrtimer 0.19 ± 62% +0.2 0.38 ± 23% perf-profile.children.cycles-pp.f2fs_inode_dirtied 0.00 +0.2 0.19 ± 11% perf-profile.children.cycles-pp.enqueue_hrtimer 0.00 +0.2 0.20 ± 15% perf-profile.children.cycles-pp.rcu_process_callbacks 0.00 +0.2 0.20 ± 11% perf-profile.children.cycles-pp.page_cache_tree_insert 0.00 +0.2 0.20 ± 69% perf-profile.children.cycles-pp.kthread 0.00 +0.2 0.20 ± 65% perf-profile.children.cycles-pp.ret_from_fork 0.00 +0.2 0.21 ± 9% perf-profile.children.cycles-pp.__hrtimer_next_event_base 0.00 +0.2 0.21 ± 8% perf-profile.children.cycles-pp.read_tsc 0.06 ± 9% +0.2 0.27 ± 11% perf-profile.children.cycles-pp.__pagevec_lru_add_fn 0.00 +0.2 0.22 ± 10% perf-profile.children.cycles-pp.__x64_sys_gettimeofday 0.00 +0.2 0.22 ± 11% perf-profile.children.cycles-pp.sched_clock_cpu 0.00 +0.2 0.22 ± 10% perf-profile.children.cycles-pp.lapic_next_deadline 0.01 ±173% +0.2 0.23 ± 11% perf-profile.children.cycles-pp.brd_lookup_page 0.03 ±173% +0.2 0.25 ± 39% perf-profile.children.cycles-pp.delay_tsc 0.01 ±173% +0.2 0.24 ± 21% perf-profile.children.cycles-pp.radix_tree_lookup_slot 0.04 ± 57% +0.2 0.27 ± 11% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.07 ± 11% +0.2 0.31 ± 9% perf-profile.children.cycles-pp.pagevec_lru_move_fn 0.00 +0.2 0.25 ± 11% perf-profile.children.cycles-pp.native_irq_return_iret 0.00 +0.3 0.25 ± 12% perf-profile.children.cycles-pp.find_next_bit 0.09 +0.3 0.35 ± 15% perf-profile.children.cycles-pp.clear_page_dirty_for_io 0.08 ± 10% +0.3 0.34 ± 9% perf-profile.children.cycles-pp.__lru_cache_add 0.04 ± 57% +0.3 0.31 ± 6% perf-profile.children.cycles-pp.___might_sleep 0.00 +0.3 0.29 ± 9% perf-profile.children.cycles-pp.tick_irq_enter 0.00 +0.3 0.29 ± 10% perf-profile.children.cycles-pp.native_write_msr 0.00 +0.3 0.30 ± 25% perf-profile.children.cycles-pp.rcu_check_callbacks 0.06 ± 11% +0.3 0.37 ± 16% perf-profile.children.cycles-pp.find_get_entry 0.03 ±100% +0.3 0.37 ± 15% perf-profile.children.cycles-pp.__entry_trampoline_start 0.06 ± 14% +0.3 0.40 ± 8% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.00 +0.4 0.35 ± 9% perf-profile.children.cycles-pp.clockevents_program_event 0.00 +0.4 0.37 ± 13% perf-profile.children.cycles-pp.ktime_get 0.00 +0.4 0.37 ± 5% perf-profile.children.cycles-pp.run_timer_softirq 0.12 ± 6% +0.4 0.49 ± 5% perf-profile.children.cycles-pp.test_clear_page_writeback 0.12 ± 7% +0.4 0.49 ± 5% perf-profile.children.cycles-pp.end_page_writeback 0.00 +0.4 0.38 ± 7% perf-profile.children.cycles-pp.cpuidle_governor_latency_req 0.00 +0.4 0.38 ± 9% perf-profile.children.cycles-pp.irq_enter 0.12 ± 35% +0.4 0.51 ± 38% perf-profile.children.cycles-pp.io_serial_in 0.10 ± 25% +0.4 0.50 ± 19% perf-profile.children.cycles-pp.irq_work_interrupt 0.10 ± 25% +0.4 0.50 ± 19% perf-profile.children.cycles-pp.smp_irq_work_interrupt 0.10 ± 25% +0.4 0.50 ± 19% perf-profile.children.cycles-pp.irq_work_run 0.10 ± 25% +0.4 0.50 ± 19% perf-profile.children.cycles-pp.printk 0.00 +0.4 0.41 ± 13% perf-profile.children.cycles-pp.__next_timer_interrupt 0.10 ± 11% +0.4 0.53 ± 13% perf-profile.children.cycles-pp.__radix_tree_lookup 0.00 +0.4 0.44 ± 11% perf-profile.children.cycles-pp.find_busiest_group 0.15 ± 5% +0.5 0.61 ± 15% perf-profile.children.cycles-pp.__get_node_page 0.18 ± 3% +0.5 0.64 ± 7% perf-profile.children.cycles-pp.f2fs_write_end_io 0.10 ± 8% +0.5 0.56 ± 9% perf-profile.children.cycles-pp.__add_to_page_cache_locked 0.12 ± 16% +0.5 0.58 ± 18% perf-profile.children.cycles-pp.irq_work_run_list 0.12 ± 4% +0.5 0.61 ± 11% perf-profile.children.cycles-pp.scheduler_tick 0.15 ± 3% +0.5 0.64 ± 9% perf-profile.children.cycles-pp.__test_set_page_writeback 0.30 ± 15% +0.5 0.79 ± 16% perf-profile.children.cycles-pp.f2fs_map_blocks 0.22 ± 3% +0.5 0.71 ± 12% perf-profile.children.cycles-pp.f2fs_get_dnode_of_data 0.03 ±100% +0.5 0.52 ± 14% perf-profile.children.cycles-pp.perf_mux_hrtimer_handler 0.10 ± 12% +0.5 0.60 ± 13% perf-profile.children.cycles-pp.__set_page_dirty_nobuffers 0.00 +0.5 0.50 ± 16% perf-profile.children.cycles-pp.update_blocked_averages 0.30 ± 15% +0.5 0.81 ± 15% perf-profile.children.cycles-pp.f2fs_preallocate_blocks 0.40 ± 10% +0.5 0.91 ± 17% perf-profile.children.cycles-pp.f2fs_set_data_page_dirty 0.00 +0.5 0.52 ± 18% perf-profile.children.cycles-pp.run_rebalance_domains 0.17 ± 44% +0.5 0.70 ± 38% perf-profile.children.cycles-pp.vprintk_emit 0.16 ± 39% +0.5 0.70 ± 34% perf-profile.children.cycles-pp.serial8250_console_putchar 0.16 ± 39% +0.5 0.70 ± 34% perf-profile.children.cycles-pp.uart_console_write 0.16 ± 41% +0.6 0.71 ± 35% perf-profile.children.cycles-pp.wait_for_xmitr 0.16 ± 41% +0.6 0.72 ± 36% perf-profile.children.cycles-pp.serial8250_console_write 0.00 +0.6 0.56 ± 10% perf-profile.children.cycles-pp.get_next_timer_interrupt 0.18 ± 45% +0.6 0.76 ± 37% perf-profile.children.cycles-pp.console_unlock 0.00 +0.6 0.62 ± 8% perf-profile.children.cycles-pp.load_balance 0.18 ± 6% +0.8 0.97 ± 9% perf-profile.children.cycles-pp.add_to_page_cache_lru 0.00 +0.8 0.83 ± 6% perf-profile.children.cycles-pp.rebalance_domains 0.00 +0.9 0.94 ± 5% perf-profile.children.cycles-pp.tick_nohz_next_event 0.90 ± 15% +1.0 1.93 ± 14% perf-profile.children.cycles-pp._raw_spin_lock 0.17 ± 2% +1.1 1.23 ± 18% perf-profile.children.cycles-pp.update_process_times 0.01 ±173% +1.2 1.17 ± 6% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length 0.17 ± 2% +1.2 1.37 ± 17% perf-profile.children.cycles-pp.tick_sched_handle 0.64 ± 20% +1.3 1.95 ± 15% perf-profile.children.cycles-pp.f2fs_write_end 0.19 ± 3% +1.4 1.54 ± 16% perf-profile.children.cycles-pp.tick_sched_timer 0.33 ± 6% +1.6 1.94 ± 10% perf-profile.children.cycles-pp.f2fs_write_begin 0.40 ± 4% +1.9 2.34 ± 11% perf-profile.children.cycles-pp.pagecache_get_page 0.09 ± 39% +2.0 2.12 ± 8% perf-profile.children.cycles-pp.__softirqentry_text_start 0.10 ± 35% +2.3 2.42 ± 8% perf-profile.children.cycles-pp.irq_exit 0.04 ±110% +2.4 2.40 ± 6% perf-profile.children.cycles-pp.menu_select 0.27 ± 3% +2.4 2.62 ± 13% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.40 ± 5% +2.8 3.18 ± 10% perf-profile.children.cycles-pp.clear_page_erms 0.33 ± 5% +3.0 3.36 ± 11% perf-profile.children.cycles-pp.hrtimer_interrupt 0.48 ± 2% +3.3 3.79 ± 8% perf-profile.children.cycles-pp.copyin 0.49 ± 2% +3.3 3.82 ± 8% perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic 0.48 ± 2% +3.3 3.81 ± 8% perf-profile.children.cycles-pp.copy_user_enhanced_fast_string 0.58 ± 4% +3.6 4.18 ± 9% perf-profile.children.cycles-pp.get_page_from_freelist 0.61 ± 5% +3.8 4.37 ± 8% perf-profile.children.cycles-pp.__alloc_pages_nodemask 0.13 ± 6% +3.8 3.91 ± 7% perf-profile.children.cycles-pp.__submit_merged_write_cond 0.61 ± 5% +4.2 4.84 ± 9% perf-profile.children.cycles-pp.brd_insert_page 0.45 ± 11% +5.9 6.37 ± 8% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 0.46 ± 11% +5.9 6.40 ± 8% perf-profile.children.cycles-pp.apic_timer_interrupt 1.02 ± 3% +6.2 7.18 ± 8% perf-profile.children.cycles-pp.brd_do_bvec 1.53 ± 9% +6.4 7.93 ± 10% perf-profile.children.cycles-pp.generic_perform_write 1.52 ± 9% +6.4 7.95 ± 10% perf-profile.children.cycles-pp.__generic_file_write_iter 1.20 ± 2% +6.6 7.83 ± 8% perf-profile.children.cycles-pp.brd_make_request 1.21 ± 2% +6.6 7.85 ± 8% perf-profile.children.cycles-pp.submit_bio 1.21 ± 2% +6.6 7.85 ± 8% perf-profile.children.cycles-pp.generic_make_request 1.21 ± 2% +6.6 7.85 ± 8% perf-profile.children.cycles-pp.__submit_merged_bio 1.84 ± 11% +7.1 8.94 ± 10% perf-profile.children.cycles-pp.f2fs_file_write_iter 1.92 ± 10% +7.2 9.16 ± 12% perf-profile.children.cycles-pp.__vfs_write 1.94 ± 10% +7.4 9.29 ± 12% perf-profile.children.cycles-pp.vfs_write 1.95 ± 10% +7.4 9.35 ± 12% perf-profile.children.cycles-pp.ksys_write 0.64 ± 3% +11.8 12.46 ± 4% perf-profile.children.cycles-pp.mutex_spin_on_owner 3.82 ± 8% +33.5 37.34 ± 4% perf-profile.children.cycles-pp.intel_idle 3.96 ± 11% +40.2 44.20 ± 4% perf-profile.children.cycles-pp.cpuidle_enter_state 4.22 ± 11% +42.2 46.47 ± 5% perf-profile.children.cycles-pp.start_secondary 4.22 ± 11% +43.1 47.28 ± 4% perf-profile.children.cycles-pp.secondary_startup_64 4.22 ± 11% +43.1 47.28 ± 4% perf-profile.children.cycles-pp.cpu_startup_entry 4.23 ± 11% +43.1 47.35 ± 4% perf-profile.children.cycles-pp.do_idle 85.30 -73.3 12.02 ± 28% perf-profile.self.cycles-pp.osq_lock 0.14 ± 6% -0.1 0.08 ± 21% perf-profile.self.cycles-pp.mutex_lock 0.08 ± 10% -0.1 0.03 ±100% perf-profile.self.cycles-pp.f2fs_lookup_rb_tree_ret 0.00 +0.1 0.05 perf-profile.self.cycles-pp.f2fs_inode_dirtied 0.00 +0.1 0.05 perf-profile.self.cycles-pp.menu_reflect 0.00 +0.1 0.05 ± 9% perf-profile.self.cycles-pp.read_node_page 0.00 +0.1 0.06 ± 14% perf-profile.self.cycles-pp.hrtimer_forward 0.00 +0.1 0.06 perf-profile.self.cycles-pp.__hrtimer_get_next_event 0.00 +0.1 0.06 ± 20% perf-profile.self.cycles-pp.rcu_process_callbacks 0.00 +0.1 0.06 ± 6% perf-profile.self.cycles-pp.call_cpuidle 0.00 +0.1 0.06 ± 6% perf-profile.self.cycles-pp.sched_avg_update 0.00 +0.1 0.06 ± 6% perf-profile.self.cycles-pp.find_next_and_bit 0.00 +0.1 0.07 ± 17% perf-profile.self.cycles-pp.balance_dirty_pages_ratelimited 0.08 ± 10% +0.1 0.15 ± 15% perf-profile.self.cycles-pp.f2fs_update_dirty_page 0.07 ± 12% +0.1 0.14 ± 15% perf-profile.self.cycles-pp.up_read 0.00 +0.1 0.07 ± 12% perf-profile.self.cycles-pp.ktime_get_update_offsets_now 0.00 +0.1 0.07 ± 26% perf-profile.self.cycles-pp.account_page_dirtied 0.00 +0.1 0.07 ± 22% perf-profile.self.cycles-pp.can_stop_idle_tick 0.00 +0.1 0.07 ± 28% perf-profile.self.cycles-pp.ktime_get_real_ts64 0.00 +0.1 0.07 ± 29% perf-profile.self.cycles-pp.tick_check_broadcast_expired 0.00 +0.1 0.08 ± 20% perf-profile.self.cycles-pp.update_group_capacity 0.00 +0.1 0.08 ± 24% perf-profile.self.cycles-pp._cond_resched 0.00 +0.1 0.08 ± 6% perf-profile.self.cycles-pp.node_dirty_ok 0.00 +0.1 0.08 ± 34% perf-profile.self.cycles-pp.perf_event_task_tick 0.00 +0.1 0.08 ± 14% perf-profile.self.cycles-pp._raw_spin_trylock 0.00 +0.1 0.08 ± 10% perf-profile.self.cycles-pp.clockevents_program_event 0.00 +0.1 0.08 ± 19% perf-profile.self.cycles-pp.pm_qos_read_value 0.00 +0.1 0.08 ± 31% perf-profile.self.cycles-pp.add_to_page_cache_lru 0.00 +0.1 0.08 ± 23% perf-profile.self.cycles-pp.cpuidle_governor_latency_req 0.00 +0.1 0.08 ± 8% perf-profile.self.cycles-pp.pm_qos_request 0.00 +0.1 0.08 ± 5% perf-profile.self.cycles-pp.rb_erase 0.00 +0.1 0.09 ± 17% perf-profile.self.cycles-pp.release_pages 0.00 +0.1 0.09 ± 30% perf-profile.self.cycles-pp.f2fs_file_write_iter 0.00 +0.1 0.09 ± 21% perf-profile.self.cycles-pp.__mod_node_page_state 0.04 ± 57% +0.1 0.13 ± 23% perf-profile.self.cycles-pp.update_load_avg 0.00 +0.1 0.09 ± 24% perf-profile.self.cycles-pp.__add_to_page_cache_locked 0.00 +0.1 0.09 ± 22% perf-profile.self.cycles-pp.f2fs_get_dnode_of_data 0.00 +0.1 0.09 ± 15% perf-profile.self.cycles-pp.rcu_needs_cpu 0.00 +0.1 0.09 ± 14% perf-profile.self.cycles-pp.smp_apic_timer_interrupt 0.00 +0.1 0.10 ± 21% perf-profile.self.cycles-pp.f2fs_write_end_io 0.00 +0.1 0.10 ± 28% perf-profile.self.cycles-pp.account_process_tick 0.00 +0.1 0.10 ± 33% perf-profile.self.cycles-pp.__mark_inode_dirty 0.00 +0.1 0.10 ± 15% perf-profile.self.cycles-pp.hrtimer_interrupt 0.08 ± 10% +0.1 0.18 ± 10% perf-profile.self.cycles-pp.__write_data_page 0.00 +0.1 0.10 ± 10% perf-profile.self.cycles-pp.__set_page_dirty_nobuffers 0.00 +0.1 0.10 ± 21% perf-profile.self.cycles-pp.f2fs_write_cache_pages 0.00 +0.1 0.11 ± 17% perf-profile.self.cycles-pp.scheduler_tick 0.00 +0.1 0.11 ± 6% perf-profile.self.cycles-pp.dec_zone_page_state 0.00 +0.1 0.11 ± 15% perf-profile.self.cycles-pp.f2fs_map_blocks 0.00 +0.1 0.11 ± 9% perf-profile.self.cycles-pp.rcu_eqs_enter 0.04 ± 57% +0.1 0.15 ± 10% perf-profile.self.cycles-pp.radix_tree_tag_clear 0.00 +0.1 0.12 ± 7% perf-profile.self.cycles-pp.f2fs_update_extent_tree_range 0.00 +0.1 0.12 ± 10% perf-profile.self.cycles-pp.get_next_timer_interrupt 0.00 +0.1 0.12 ± 5% perf-profile.self.cycles-pp.__list_del_entry_valid 0.00 +0.1 0.12 ± 18% perf-profile.self.cycles-pp.f2fs_write_begin 0.06 ± 7% +0.1 0.18 ± 18% perf-profile.self.cycles-pp.percpu_counter_add_batch 0.06 ± 11% +0.1 0.18 ± 19% perf-profile.self.cycles-pp.f2fs_do_write_data_page 0.00 +0.1 0.13 ± 11% perf-profile.self.cycles-pp.cpu_load_update 0.01 ±173% +0.1 0.14 ± 8% perf-profile.self.cycles-pp.radix_tree_tag_set 0.00 +0.1 0.13 ± 13% perf-profile.self.cycles-pp.load_balance 0.00 +0.1 0.14 ± 15% perf-profile.self.cycles-pp.run_posix_cpu_timers 0.00 +0.1 0.14 ± 16% perf-profile.self.cycles-pp.get_cpu_device 0.00 +0.1 0.14 ± 9% perf-profile.self.cycles-pp.find_get_entry 0.00 +0.1 0.14 ± 19% perf-profile.self.cycles-pp.rebalance_domains 0.00 +0.1 0.14 ± 10% perf-profile.self.cycles-pp.__alloc_pages_nodemask 0.00 +0.1 0.14 ± 12% perf-profile.self.cycles-pp.timerqueue_add 0.00 +0.1 0.14 ± 18% perf-profile.self.cycles-pp.__radix_tree_create 0.00 +0.1 0.15 ± 7% perf-profile.self.cycles-pp.__might_sleep 0.07 ± 7% +0.1 0.21 ± 19% perf-profile.self.cycles-pp.clear_page_dirty_for_io 0.00 +0.1 0.15 ± 20% perf-profile.self.cycles-pp.perf_mux_hrtimer_handler 0.00 +0.2 0.15 ± 15% perf-profile.self.cycles-pp.do_syscall_64 0.00 +0.2 0.15 ± 14% perf-profile.self.cycles-pp.native_sched_clock 0.00 +0.2 0.16 ± 5% perf-profile.self.cycles-pp.irq_exit 0.00 +0.2 0.16 ± 14% perf-profile.self.cycles-pp.find_get_pages_range_tag 0.07 ± 5% +0.2 0.23 ± 20% perf-profile.self.cycles-pp.f2fs_submit_page_write 0.00 +0.2 0.17 ± 21% perf-profile.self.cycles-pp.pagecache_get_page 0.00 +0.2 0.17 ± 2% perf-profile.self.cycles-pp.iov_iter_fault_in_readable 0.00 +0.2 0.18 ± 2% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.00 +0.2 0.18 ± 9% perf-profile.self.cycles-pp.__hrtimer_run_queues 0.00 +0.2 0.18 ± 13% perf-profile.self.cycles-pp.test_clear_page_writeback 0.06 ± 6% +0.2 0.24 ± 17% perf-profile.self.cycles-pp.up_write 0.03 ±100% +0.2 0.21 ± 8% perf-profile.self.cycles-pp.__pagevec_lru_add_fn 0.00 +0.2 0.18 ± 10% perf-profile.self.cycles-pp.__indirect_thunk_start 0.00 +0.2 0.18 ± 17% perf-profile.self.cycles-pp.page_mapping 0.04 ± 58% +0.2 0.23 ± 13% perf-profile.self.cycles-pp.__test_set_page_writeback 0.00 +0.2 0.21 ± 8% perf-profile.self.cycles-pp.__hrtimer_next_event_base 0.00 +0.2 0.21 ± 9% perf-profile.self.cycles-pp.read_tsc 0.00 +0.2 0.22 ± 13% perf-profile.self.cycles-pp.__next_timer_interrupt 0.00 +0.2 0.22 ± 3% perf-profile.self.cycles-pp.tick_nohz_next_event 0.00 +0.2 0.22 ± 10% perf-profile.self.cycles-pp.__softirqentry_text_start 0.00 +0.2 0.22 ± 19% perf-profile.self.cycles-pp.ktime_get 0.03 ±173% +0.2 0.25 ± 39% perf-profile.self.cycles-pp.delay_tsc 0.10 ± 4% +0.2 0.34 ± 13% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.00 +0.2 0.25 ± 11% perf-profile.self.cycles-pp.native_irq_return_iret 0.00 +0.3 0.25 ± 12% perf-profile.self.cycles-pp.find_next_bit 0.00 +0.3 0.26 ± 4% perf-profile.self.cycles-pp.do_idle 0.00 +0.3 0.26 ± 25% perf-profile.self.cycles-pp.rcu_check_callbacks 0.00 +0.3 0.27 ± 6% perf-profile.self.cycles-pp.run_timer_softirq 0.00 +0.3 0.27 ± 12% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.04 ± 57% +0.3 0.31 ± 6% perf-profile.self.cycles-pp.___might_sleep 0.00 +0.3 0.29 ± 10% perf-profile.self.cycles-pp.native_write_msr 0.00 +0.3 0.29 ± 17% perf-profile.self.cycles-pp.update_blocked_averages 0.00 +0.3 0.30 ± 11% perf-profile.self.cycles-pp.find_busiest_group 0.03 ±100% +0.3 0.37 ± 15% perf-profile.self.cycles-pp.__entry_trampoline_start 0.06 ± 14% +0.3 0.40 ± 8% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.12 ± 35% +0.4 0.51 ± 38% perf-profile.self.cycles-pp.io_serial_in 0.10 ± 11% +0.4 0.52 ± 12% perf-profile.self.cycles-pp.__radix_tree_lookup 0.00 +0.4 0.44 ± 10% perf-profile.self.cycles-pp.cpuidle_enter_state 0.11 ± 4% +0.5 0.66 ± 7% perf-profile.self.cycles-pp.get_page_from_freelist 0.10 ± 8% +0.8 0.91 ± 14% perf-profile.self.cycles-pp.f2fs_write_end 0.01 ±173% +0.9 0.87 ± 8% perf-profile.self.cycles-pp.menu_select 0.39 ± 3% +1.5 1.90 ± 14% perf-profile.self.cycles-pp._raw_spin_lock 0.37 +1.7 2.09 ± 6% perf-profile.self.cycles-pp.brd_do_bvec 0.40 ± 5% +2.7 3.14 ± 11% perf-profile.self.cycles-pp.clear_page_erms 0.48 ± 2% +3.3 3.79 ± 8% perf-profile.self.cycles-pp.copy_user_enhanced_fast_string 0.64 ± 3% +11.7 12.36 ± 5% perf-profile.self.cycles-pp.mutex_spin_on_owner 3.82 ± 8% +33.5 37.29 ± 4% perf-profile.self.cycles-pp.intel_idle fsmark.time.system_time 1800 +-+------------------------------------------------------------------+ |.+.+ | 1600 +-+ : +.++.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.| 1400 +-+ : : | | : : | 1200 +-+ : : | 1000 +-+ : : | | : : | 800 +-+ : : | 600 +-+ : : | | : : | 400 +-+ : : | 200 +-+ : : | O O O : : O OO O O O O O O O O O O OO O O O | 0 +-+---O-O------------------------------------------------------------+ fsmark.time.percent_of_cpu_this_job_got 4000 +-+------------------------------------------------------------------+ |.+ : + +.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+ +.+ + | 3500 +-+ : : | 3000 +-+ : : | | : : | 2500 +-+ : : | | : : | 2000 +-+ : : | | : : | 1500 +-+ : : | 1000 +-+ : : | | : : | 500 +-+ : : | O O O O.O O OO O O O O O O O O O O OO O O O | 0 +-+------------------------------------------------------------------+ fsmark.time.elapsed_time 45 +-+--------------------------------------------------------------------+ |.+.+ .+.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.| 40 +-+ : +.+ | 35 +-+ : : O O O | O O O O O O O O O O O O O OO O O O | 30 +-+ : : | 25 +-+ : : | | : : | 20 +-+ : : | 15 +-+ : : | | : : | 10 +-+ : : | 5 +-+ : : | | : : | 0 +-+---O-O--------------------------------------------------------------+ fsmark.time.elapsed_time.max 45 +-+--------------------------------------------------------------------+ |.+.+ .+.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.| 40 +-+ : +.+ | 35 +-+ : : O O O | O O O O O O O O O O O O O OO O O O | 30 +-+ : : | 25 +-+ : : | | : : | 20 +-+ : : | 15 +-+ : : | | : : | 10 +-+ : : | 5 +-+ : : | | : : | 0 +-+---O-O--------------------------------------------------------------+ fsmark.time.minor_page_faults 92000 +-+-----------------------------------------------------------------+ 90000 +-+.+ | | : +.+ | 88000 +-+ : : : | 86000 +-+ : : : | 84000 +-+ : : : | 82000 +-+ : : : .+ | | : : : +. .+.++.+ + .+. .+. + +. | 80000 +-+ : : +. + +.+ + +.+.+.++.+ +.+.+.+. .+ + + +.| 78000 +-+ : : + + + | 76000 O-+ : : | 74000 +-O O : :O O O O O O O O O O O O O | | : : O O O O O | 72000 +-+ O.O | 70000 +-+-----------------------------------------------------------------+ fsmark.time.voluntary_context_switches 1.6e+06 +-+---------------------------------------------------------------+ | +.+.+.+. + | 1.4e+06 +-+ + +.+ + .+.+. .+ .+. | | + + + +.+. .+. .++. .+.+ ++.+.| 1.2e+06 +-+.+ +.+.+.+ + + + | 1e+06 +-+ : : | | : : | 800000 +-+ : : | | : : | 600000 +-+ : : | 400000 +-+ : : | | : : | 200000 +-+ :: | | :: | 0 O-O-O-OO-O-O-O-O-OO-O-O-O-O-OO-O-O-O-O-OO-------------------------+ fsmark.time.involuntary_context_switches 30000 +-+-----------------------------------------------------------------+ | +. .+.+.+.+.++.+.+.+.| 25000 +-+.+ +.+.+.+ : +.+.+ | | : : + +. + + | | : : + : +. + + + | 20000 +-+ : : + +.+ : + + | | : : +.+.+.+ + : | 15000 +-+ : : + | | : : | 10000 +-+ : : | | : : | | : : | 5000 +-+ : : | | : : | 0 O-O-O-O-OO-O-O-O-O-O-O-O-OO-O-O-O-O-O-O-O-O-------------------------+ fsmark.files_per_sec 400 +-+-------------------------------------------------------------------+ O O O O O O O O O O O O O O | 350 +-+ O O O O O O | 300 +-+ O | | +.+. | 250 +-+.+ : +.+.++.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.++.+.+.+.+.+.+.+.| | : : | 200 +-+ : : | | : : | 150 +-+ : : | 100 +-+ : : | | : : | 50 +-+ : : | | : : | 0 +-+---O-O-------------------------------------------------------------+ [*] bisect-good sample [O] bisect-bad sample Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Rong Chen