All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Huang\, Ying" <ying.huang@intel.com>
To: "Huang\, Ying" <ying.huang@intel.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
	Dave Chinner <david@fromorbit.com>,
	LKML <linux-kernel@vger.kernel.org>,
	Bob Peterson <rpeterso@redhat.com>,
	Wu Fengguang <fengguang.wu@intel.com>, LKP <lkp@01.org>,
	Christoph Hellwig <hch@lst.de>
Subject: Re: [LKP] [lkp] [xfs] 68a9f5e700: aim7.jobs-per-min -13.6% regression
Date: Wed, 10 Aug 2016 17:11:16 -0700	[thread overview]
Message-ID: <87a8gk17x7.fsf@yhuang-mobile.sh.intel.com> (raw)
In-Reply-To: <87eg5w18iu.fsf@yhuang-mobile.sh.intel.com> (Ying Huang's message of "Wed, 10 Aug 2016 16:58:17 -0700")

"Huang, Ying" <ying.huang@intel.com> writes:

> Hi, Linus,
>
> Linus Torvalds <torvalds@linux-foundation.org> writes:
>
>> On Wed, Aug 10, 2016 at 4:08 PM, Dave Chinner <david@fromorbit.com> wrote:
>>>
>>> That, to me, says there's a change in lock contention behaviour in
>>> the workload (which we know aim7 is good at exposing). i.e. the
>>> iomap change shifted contention from a sleeping lock to a spinning
>>> lock, or maybe we now trigger optimistic spinning behaviour on a
>>> lock we previously didn't spin on at all.
>>
>> Hmm. Possibly. I reacted to the lower cpu load number, but yeah, I
>> could easily imagine some locking primitive difference too.
>>
>>> We really need instruction level perf profiles to understand
>>> this - I don't have a machine with this many cpu cores available
>>> locally, so I'm not sure I'm going to be able to make any progress
>>> tracking it down in the short term. Maybe the lkp team has more
>>> in-depth cpu usage profiles they can share?
>>
>> Yeah, I've occasionally wanted to see some kind of "top-25 kernel
>> functions in the profile" thing. That said, when the load isn't all
>> that familiar, the profiles usually are not all that easy to make
>> sense of either. But comparing the before and after state might give
>> us clues.
>
> I have started perf-profile data collection, will send out the
> comparison result soon.

Here is the comparison result with perf-profile data.

=========================================================================================
compiler/cpufreq_governor/debug-setup/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase:
  gcc-6/performance/profile/1BRD_48G/xfs/x86_64-rhel/3000/debian-x86_64-2015-02-07.cgz/ivb44/disk_wrt/aim7

commit: 
  f0c6bcba74ac51cb77aadb33ad35cb2dc1ad1506
  68a9f5e7007c1afa2cf6830b690a90d0187c0684

f0c6bcba74ac51cb 68a9f5e7007c1afa2cf6830b69 
---------------- -------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    484435 ±  0%     -13.3%     420004 ±  0%  aim7.jobs-per-min
     37.37 ±  0%     +15.3%      43.09 ±  0%  aim7.time.elapsed_time
     37.37 ±  0%     +15.3%      43.09 ±  0%  aim7.time.elapsed_time.max
      6491 ±  3%     +30.8%       8491 ±  0%  aim7.time.involuntary_context_switches
    376.89 ±  0%     +28.4%     484.11 ±  0%  aim7.time.system_time
    430512 ±  0%     -20.1%     343838 ±  0%  aim7.time.voluntary_context_switches
     26816 ±  8%     +10.2%      29542 ±  1%  interrupts.CAL:Function_call_interrupts
    125122 ± 10%     -10.7%     111758 ± 12%  softirqs.SCHED
     24772 ±  0%     -28.6%      17675 ±  0%  vmstat.system.cs
     53477 ±  2%      +5.6%      56453 ±  0%  vmstat.system.in
     15627 ±  0%     +27.7%      19956 ±  1%  meminfo.Active(file)
     16103 ±  3%     +14.3%      18405 ±  8%  meminfo.AnonHugePages
    132898 ±  9%     +15.4%     153380 ±  1%  meminfo.DirectMap4k
     13777 ±  5%     +43.1%      19709 ±  0%  meminfo.Shmem
      3906 ±  0%     +28.8%       5032 ±  2%  proc-vmstat.nr_active_file
    919.33 ±  5%     +14.8%       1055 ±  8%  proc-vmstat.nr_dirty
      3444 ±  5%     +41.8%       4884 ±  0%  proc-vmstat.nr_shmem
      4092 ± 14%     +61.2%       6595 ±  1%  proc-vmstat.pgactivate
      1975 ± 15%     +63.2%       3224 ± 17%  slabinfo.scsi_data_buffer.active_objs
      1975 ± 15%     +63.2%       3224 ± 17%  slabinfo.scsi_data_buffer.num_objs
    464.33 ± 15%     +63.3%     758.33 ± 17%  slabinfo.xfs_efd_item.active_objs
    464.33 ± 15%     +63.3%     758.33 ± 17%  slabinfo.xfs_efd_item.num_objs
   1724300 ± 27%     -40.5%    1025538 ±  1%  sched_debug.cfs_rq:/.load.max
     96.36 ±  3%     +18.6%     114.32 ± 15%  sched_debug.cfs_rq:/.util_avg.stddev
   1724300 ± 27%     -40.5%    1025538 ±  1%  sched_debug.cpu.load.max
      2887 ± 30%     -28.2%       2073 ± 48%  sched_debug.cpu.nr_load_updates.min
      7.66 ± 20%     -24.9%       5.75 ± 15%  sched_debug.cpu.nr_uninterruptible.stddev
     37.37 ±  0%     +15.3%      43.09 ±  0%  time.elapsed_time
     37.37 ±  0%     +15.3%      43.09 ±  0%  time.elapsed_time.max
      6491 ±  3%     +30.8%       8491 ±  0%  time.involuntary_context_switches
      1037 ±  0%     +10.8%       1148 ±  0%  time.percent_of_cpu_this_job_got
    376.89 ±  0%     +28.4%     484.11 ±  0%  time.system_time
    430512 ±  0%     -20.1%     343838 ±  0%  time.voluntary_context_switches
     24.18 ±  0%      +9.0%      26.35 ±  0%  turbostat.%Busy
    686.00 ±  0%      +9.5%     751.00 ±  0%  turbostat.Avg_MHz
      0.28 ±  0%     -25.0%       0.21 ±  0%  turbostat.CPU%c3
     93.33 ±  1%      +3.0%      96.15 ±  0%  turbostat.CorWatt
    124.61 ±  0%      +2.1%     127.17 ±  0%  turbostat.PkgWatt
      4.74 ±  0%      -2.7%       4.61 ±  1%  turbostat.RAMWatt
      7723 ±  0%     +32.6%      10238 ±  5%  numa-meminfo.node0.Active(file)
      1589 ± 17%     +45.5%       2313 ± 24%  numa-meminfo.node0.Dirty
     56052 ±  3%     +58.2%      88666 ± 17%  numa-meminfo.node1.Active
     48142 ±  4%     +64.0%      78943 ± 19%  numa-meminfo.node1.Active(anon)
      7908 ±  1%     +22.9%       9722 ±  3%  numa-meminfo.node1.Active(file)
     46721 ±  3%     +55.9%      72837 ± 24%  numa-meminfo.node1.AnonPages
      4789 ± 69%    +102.3%       9687 ±  9%  numa-meminfo.node1.Shmem
  52991525 ±  1%     -19.4%   42687208 ±  0%  cpuidle.C1-IVT.time
    319584 ±  1%     -26.5%     234868 ±  1%  cpuidle.C1-IVT.usage
   3468808 ±  2%     -19.8%    2783341 ±  3%  cpuidle.C1E-IVT.time
     46760 ±  0%     -22.4%      36298 ±  0%  cpuidle.C1E-IVT.usage
  12590471 ±  0%     -22.3%    9788585 ±  1%  cpuidle.C3-IVT.time
     79965 ±  0%     -19.0%      64749 ±  0%  cpuidle.C3-IVT.usage
   1.3e+09 ±  0%     +13.3%  1.473e+09 ±  0%  cpuidle.C6-IVT.time
    352.33 ±  8%     -24.7%     265.33 ±  1%  cpuidle.POLL.usage
      1930 ±  0%     +33.9%       2585 ±  3%  numa-vmstat.node0.nr_active_file
      4468 ±  7%      -8.5%       4089 ±  5%  numa-vmstat.node0.nr_alloc_batch
    466.67 ±  4%     +29.3%     603.33 ± 14%  numa-vmstat.node0.nr_dirty
     12026 ±  4%     +64.1%      19734 ± 20%  numa-vmstat.node1.nr_active_anon
      1977 ±  1%     +23.6%       2444 ±  1%  numa-vmstat.node1.nr_active_file
      3809 ±  6%     +16.1%       4422 ±  4%  numa-vmstat.node1.nr_alloc_batch
     11671 ±  3%     +55.9%      18197 ± 24%  numa-vmstat.node1.nr_anon_pages
      1197 ± 69%    +102.3%       2422 ±  9%  numa-vmstat.node1.nr_shmem
    456.33 ± 57%     -75.6%     111.33 ± 86%  numa-vmstat.node1.nr_written
 2.658e+11 ±  4%     +24.7%  3.316e+11 ±  2%  perf-stat.branch-instructions
      0.41 ±  1%      -9.1%       0.37 ±  1%  perf-stat.branch-miss-rate
  1.09e+09 ±  3%     +13.4%  1.237e+09 ±  1%  perf-stat.branch-misses
    981138 ±  0%     -18.1%     803696 ±  0%  perf-stat.context-switches
 1.511e+12 ±  5%     +23.4%  1.864e+12 ±  3%  perf-stat.cpu-cycles
    102600 ±  1%      -7.3%      95075 ±  1%  perf-stat.cpu-migrations
      0.26 ± 12%     -30.8%       0.18 ± 10%  perf-stat.dTLB-load-miss-rate
 3.164e+11 ±  1%     +39.9%  4.426e+11 ±  4%  perf-stat.dTLB-loads
      0.03 ± 26%     -41.3%       0.02 ± 13%  perf-stat.dTLB-store-miss-rate
 2.247e+11 ±  6%     +26.4%  2.839e+11 ±  2%  perf-stat.dTLB-stores
  1.49e+12 ±  4%     +30.1%  1.939e+12 ±  2%  perf-stat.instructions
     43348 ±  2%     +34.2%      58161 ± 12%  perf-stat.instructions-per-iTLB-miss
      0.99 ±  0%      +5.5%       1.04 ±  0%  perf-stat.ipc
    262799 ±  0%      +4.4%     274251 ±  1%  perf-stat.minor-faults
     34.12 ±  1%      +2.1%      34.83 ±  0%  perf-stat.node-load-miss-rate
  46476754 ±  2%      +4.6%   48601269 ±  1%  perf-stat.node-load-misses
      9.96 ±  0%     +13.4%      11.30 ±  0%  perf-stat.node-store-miss-rate
  24460859 ±  1%     +14.4%   27971097 ±  1%  perf-stat.node-store-misses
    262780 ±  0%      +4.4%     274227 ±  1%  perf-stat.page-faults
     11.31 ±  1%     -18.1%       9.27 ±  0%  perf-profile.cycles-pp.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       1.68 ±  1%  perf-profile.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      1.80 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin
      2.55 ±  3%     -14.2%       2.19 ±  2%  perf-profile.cycles-pp.__alloc_pages_nodemask.alloc_pages_current.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin
      0.00 ± -1%      +Inf%       4.45 ±  1%  perf-profile.cycles-pp.__block_commit_write.isra.24.block_write_end.generic_write_end.iomap_write_actor.iomap_apply
      5.93 ±  0%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__block_commit_write.isra.24.block_write_end.generic_write_end.xfs_vm_write_end.generic_perform_write
     13.71 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__block_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter
     10.36 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write
      0.00 ± -1%      +Inf%       3.64 ±  0%  perf-profile.cycles-pp.__block_write_begin_int.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.04 ±  2%     -18.9%       0.84 ±  1%  perf-profile.cycles-pp.__delete_from_page_cache.delete_from_page_cache.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final
     11.24 ±  2%     -18.1%       9.21 ±  0%  perf-profile.cycles-pp.__dentry_kill.dput.__fput.____fput.task_work_run
     11.31 ±  2%     -18.1%       9.26 ±  0%  perf-profile.cycles-pp.__fput.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath
      0.00 ± -1%      +Inf%       1.09 ±  2%  perf-profile.cycles-pp.__mark_inode_dirty.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.32 ±  4%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__mark_inode_dirty.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write
      0.00 ± -1%      +Inf%       2.68 ±  2%  perf-profile.cycles-pp.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      3.04 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write
      1.00 ±  1%     -18.0%       0.82 ±  1%  perf-profile.cycles-pp.__radix_tree_lookup.radix_tree_lookup_slot.find_get_entry.pagecache_get_page.grab_cache_page_write_begin
      1.12 ±  2%     -17.6%       0.92 ±  4%  perf-profile.cycles-pp.__sb_start_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath
      1.38 ±  2%     -13.3%       1.19 ±  1%  perf-profile.cycles-pp.__set_page_dirty.mark_buffer_dirty.__block_commit_write.isra.24.block_write_end.generic_write_end
     54.10 ±  1%     +13.1%      61.20 ±  0%  perf-profile.cycles-pp.__vfs_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath
      6.34 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin
      0.00 ± -1%      +Inf%       3.69 ±  1%  perf-profile.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      4.02 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write
      0.98 ±  5%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.alloc_page_buffers.create_empty_buffers.create_page_buffers.__block_write_begin_int.__block_write_begin
      0.00 ± -1%      +Inf%       2.56 ±  2%  perf-profile.cycles-pp.alloc_pages_current.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      2.91 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.alloc_pages_current.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin
      3.42 ±  0%     -20.9%       2.71 ±  2%  perf-profile.cycles-pp.block_invalidatepage.xfs_vm_invalidatepage.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final
      0.00 ± -1%      +Inf%       4.69 ±  0%  perf-profile.cycles-pp.block_write_end.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      6.24 ±  0%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.block_write_end.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write
     19.18 ±  5%      -9.3%      17.40 ±  0%  perf-profile.cycles-pp.call_cpuidle.cpu_startup_entry.start_secondary
      0.94 ±  4%     -19.8%       0.76 ±  0%  perf-profile.cycles-pp.cancel_dirty_page.try_to_free_buffers.xfs_vm_releasepage.try_to_release_page.block_invalidatepage
      3.95 ±  2%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.copy_user_enhanced_fast_string.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write
      0.00 ± -1%      +Inf%       3.22 ±  0%  perf-profile.cycles-pp.copy_user_enhanced_fast_string.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
     19.75 ±  5%      -9.8%      17.81 ±  0%  perf-profile.cycles-pp.cpu_startup_entry.start_secondary
     19.18 ±  5%      -9.3%      17.40 ±  0%  perf-profile.cycles-pp.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary
     18.45 ±  5%      -9.2%      16.75 ±  0%  perf-profile.cycles-pp.cpuidle_enter_state.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary
      1.44 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.create_empty_buffers.create_page_buffers.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin
      0.00 ± -1%      +Inf%       1.18 ±  1%  perf-profile.cycles-pp.create_empty_buffers.create_page_buffers.__block_write_begin_int.iomap_write_begin.iomap_write_actor
      1.86 ±  2%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.create_page_buffers.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin.generic_perform_write
      0.00 ± -1%      +Inf%       1.53 ±  1%  perf-profile.cycles-pp.create_page_buffers.__block_write_begin_int.iomap_write_begin.iomap_write_actor.iomap_apply
      1.74 ±  2%     -19.9%       1.40 ±  3%  perf-profile.cycles-pp.delete_from_page_cache.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final.evict
      1.27 ±  0%     -22.5%       0.99 ±  4%  perf-profile.cycles-pp.destroy_inode.evict.iput.__dentry_kill.dput
      2.61 ±  1%     -24.3%       1.98 ±  1%  perf-profile.cycles-pp.do_filp_open.do_sys_open.sys_creat.entry_SYSCALL_64_fastpath
      2.66 ±  1%     -24.3%       2.01 ±  1%  perf-profile.cycles-pp.do_sys_open.sys_creat.entry_SYSCALL_64_fastpath
      1.79 ±  2%     -28.2%       1.28 ±  3%  perf-profile.cycles-pp.do_unlinkat.sys_unlink.entry_SYSCALL_64_fastpath
      1.07 ±  3%     -23.3%       0.82 ±  3%  perf-profile.cycles-pp.down_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write
      1.01 ±  3%     -17.9%       0.83 ±  2%  perf-profile.cycles-pp.down_write.xfs_ilock.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write
     11.26 ±  2%     -18.1%       9.23 ±  0%  perf-profile.cycles-pp.dput.__fput.____fput.task_work_run.exit_to_usermode_loop
     11.21 ±  2%     -18.1%       9.18 ±  0%  perf-profile.cycles-pp.evict.iput.__dentry_kill.dput.__fput
     11.34 ±  2%     -18.1%       9.29 ±  0%  perf-profile.cycles-pp.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       1.55 ±  3%  perf-profile.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      1.83 ±  2%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write
     43.95 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write
      0.00 ± -1%      +Inf%       7.91 ±  1%  perf-profile.cycles-pp.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
     10.68 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter
      1.91 ±  3%     -16.4%       1.59 ±  1%  perf-profile.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_current.__page_cache_alloc.pagecache_get_page
      0.00 ± -1%      +Inf%       9.85 ±  0%  perf-profile.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
     10.96 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter
      0.00 ± -1%      +Inf%      52.29 ±  0%  perf-profile.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write
      0.00 ± -1%      +Inf%      52.94 ±  0%  perf-profile.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write
      0.00 ± -1%      +Inf%      34.35 ±  0%  perf-profile.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter
      0.00 ± -1%      +Inf%      16.48 ±  0%  perf-profile.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
     11.22 ±  2%     -18.1%       9.19 ±  0%  perf-profile.cycles-pp.iput.__dentry_kill.dput.__fput.____fput
      0.00 ± -1%      +Inf%       1.55 ±  1%  perf-profile.cycles-pp.lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      1.72 ±  2%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin
      0.00 ± -1%      +Inf%       2.78 ±  0%  perf-profile.cycles-pp.mark_buffer_dirty.__block_commit_write.isra.24.block_write_end.generic_write_end.iomap_write_actor
      3.39 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.mark_buffer_dirty.__block_commit_write.isra.24.block_write_end.generic_write_end.xfs_vm_write_end
      0.00 ± -1%      +Inf%       3.44 ±  1%  perf-profile.cycles-pp.mark_page_accessed.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      3.03 ±  0%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.memset_erms.__block_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write
      0.00 ± -1%      +Inf%       2.43 ±  0%  perf-profile.cycles-pp.memset_erms.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      0.00 ± -1%      +Inf%       9.25 ±  0%  perf-profile.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
     10.37 ±  2%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write
      2.58 ±  1%     -24.1%       1.96 ±  0%  perf-profile.cycles-pp.path_openat.do_filp_open.do_sys_open.sys_creat.entry_SYSCALL_64_fastpath
      1.17 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.radix_tree_lookup_slot.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin
      2.06 ±  3%     -22.5%       1.60 ±  2%  perf-profile.cycles-pp.rw_verify_area.vfs_write.sys_write.entry_SYSCALL_64_fastpath
      1.79 ±  3%     -22.2%       1.39 ±  0%  perf-profile.cycles-pp.security_file_permission.rw_verify_area.vfs_write.sys_write.entry_SYSCALL_64_fastpath
      1.32 ±  4%     -21.4%       1.04 ±  0%  perf-profile.cycles-pp.selinux_file_permission.security_file_permission.rw_verify_area.vfs_write.sys_write
     19.79 ±  5%      -9.9%      17.84 ±  0%  perf-profile.cycles-pp.start_secondary
      2.67 ±  1%     -24.2%       2.02 ±  1%  perf-profile.cycles-pp.sys_creat.entry_SYSCALL_64_fastpath
      1.79 ±  3%     -27.9%       1.29 ±  3%  perf-profile.cycles-pp.sys_unlink.entry_SYSCALL_64_fastpath
     60.98 ±  1%      +9.5%      66.76 ±  0%  perf-profile.cycles-pp.sys_write.entry_SYSCALL_64_fastpath
     11.34 ±  1%     -18.1%       9.29 ±  0%  perf-profile.cycles-pp.syscall_return_slowpath.entry_SYSCALL_64_fastpath
     11.32 ±  1%     -18.0%       9.28 ±  0%  perf-profile.cycles-pp.task_work_run.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath
      5.96 ±  1%     -20.0%       4.77 ±  0%  perf-profile.cycles-pp.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final.evict.iput
      9.89 ±  2%     -17.4%       8.17 ±  0%  perf-profile.cycles-pp.truncate_inode_pages_final.evict.iput.__dentry_kill.dput
      9.87 ±  2%     -17.5%       8.15 ±  0%  perf-profile.cycles-pp.truncate_inode_pages_range.truncate_inode_pages_final.evict.iput.__dentry_kill
      2.07 ±  1%     -20.4%       1.65 ±  2%  perf-profile.cycles-pp.try_to_free_buffers.xfs_vm_releasepage.try_to_release_page.block_invalidatepage.xfs_vm_invalidatepage
      2.40 ±  1%     -21.0%       1.89 ±  2%  perf-profile.cycles-pp.try_to_release_page.block_invalidatepage.xfs_vm_invalidatepage.truncate_inode_page.truncate_inode_pages_range
      0.00 ± -1%      +Inf%       1.36 ±  1%  perf-profile.cycles-pp.unlock_page.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.72 ±  4%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.unlock_page.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write
     59.63 ±  1%     +10.2%      65.72 ±  0%  perf-profile.cycles-pp.vfs_write.sys_write.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       1.52 ±  2%  perf-profile.cycles-pp.workingset_activation.mark_page_accessed.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      0.00 ± -1%      +Inf%       1.73 ±  1%  perf-profile.cycles-pp.xfs_bmap_search_extents.xfs_bmapi_delay.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply
      0.00 ± -1%      +Inf%       1.97 ±  2%  perf-profile.cycles-pp.xfs_bmap_search_extents.xfs_bmapi_read.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write
      0.00 ± -1%      +Inf%       1.61 ±  2%  perf-profile.cycles-pp.xfs_bmap_search_extents.xfs_bmapi_read.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay.xfs_file_iomap_begin
      0.00 ± -1%      +Inf%       1.24 ±  2%  perf-profile.cycles-pp.xfs_bmap_search_multi_extents.xfs_bmap_search_extents.xfs_bmapi_delay.xfs_iomap_write_delay.xfs_file_iomap_begin
      0.00 ± -1%      +Inf%       1.46 ±  1%  perf-profile.cycles-pp.xfs_bmap_search_multi_extents.xfs_bmap_search_extents.xfs_bmapi_read.xfs_file_iomap_begin.iomap_apply
      0.00 ± -1%      +Inf%       1.21 ±  2%  perf-profile.cycles-pp.xfs_bmap_search_multi_extents.xfs_bmap_search_extents.xfs_bmapi_read.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay
      1.25 ±  0%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.xfs_bmapi_delay.xfs_iomap_write_delay.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int
      0.00 ± -1%      +Inf%       3.06 ±  1%  perf-profile.cycles-pp.xfs_bmapi_delay.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write
      1.04 ±  0%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.xfs_bmapi_read.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int.__block_write_begin
      0.00 ± -1%      +Inf%       3.04 ±  1%  perf-profile.cycles-pp.xfs_bmapi_read.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      0.00 ± -1%      +Inf%       3.05 ±  1%  perf-profile.cycles-pp.xfs_bmapi_read.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply
      1.32 ±  2%     -21.5%       1.04 ±  1%  perf-profile.cycles-pp.xfs_create.xfs_generic_create.xfs_vn_mknod.xfs_vn_create.path_openat
     51.83 ±  1%     +14.3%      59.25 ±  0%  perf-profile.cycles-pp.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write.sys_write
      0.00 ± -1%      +Inf%      16.05 ±  0%  perf-profile.cycles-pp.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter
     53.16 ±  1%     +13.6%      60.40 ±  0%  perf-profile.cycles-pp.xfs_file_write_iter.__vfs_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath
      1.24 ±  1%     -23.1%       0.95 ±  4%  perf-profile.cycles-pp.xfs_fs_destroy_inode.destroy_inode.evict.iput.__dentry_kill
      1.42 ±  2%     -21.2%       1.12 ±  1%  perf-profile.cycles-pp.xfs_generic_create.xfs_vn_mknod.xfs_vn_create.path_openat.do_filp_open
      6.46 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.xfs_get_blocks.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin.generic_perform_write
      1.29 ±  3%     -18.9%       1.04 ±  1%  perf-profile.cycles-pp.xfs_ilock.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write
      0.00 ± -1%      +Inf%       1.14 ±  3%  perf-profile.cycles-pp.xfs_ilock.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      1.21 ±  1%     -23.4%       0.93 ±  4%  perf-profile.cycles-pp.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.evict.iput
      1.23 ±  4%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.xfs_iomap_eof_want_preallocate.constprop.6.xfs_iomap_write_delay.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int
      0.00 ± -1%      +Inf%       4.14 ±  0%  perf-profile.cycles-pp.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write
      3.28 ±  2%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.xfs_iomap_write_delay.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int.__block_write_begin
      0.00 ± -1%      +Inf%       9.08 ±  0%  perf-profile.cycles-pp.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      3.54 ±  0%     -20.8%       2.81 ±  1%  perf-profile.cycles-pp.xfs_vm_invalidatepage.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final.evict
      2.35 ±  1%     -21.0%       1.86 ±  1%  perf-profile.cycles-pp.xfs_vm_releasepage.try_to_release_page.block_invalidatepage.xfs_vm_invalidatepage.truncate_inode_page
     25.10 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write
     11.03 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write
      1.42 ±  2%     -20.7%       1.13 ±  1%  perf-profile.cycles-pp.xfs_vn_create.path_openat.do_filp_open.do_sys_open.sys_creat
      1.42 ±  2%     -20.5%       1.13 ±  1%  perf-profile.cycles-pp.xfs_vn_mknod.xfs_vn_create.path_openat.do_filp_open.do_sys_open
      2.27 ±  1%     -10.6%       2.03 ±  0%  perf-profile.func.cycles-pp.___might_sleep
      2.49 ±  0%     -34.5%       1.63 ±  1%  perf-profile.func.cycles-pp.__block_commit_write.isra.24
      1.51 ±  2%     +15.4%       1.75 ±  1%  perf-profile.func.cycles-pp.__block_write_begin_int
      1.79 ±  4%     -16.8%       1.49 ±  1%  perf-profile.func.cycles-pp.__mark_inode_dirty
      1.32 ±  0%     -16.4%       1.10 ±  1%  perf-profile.func.cycles-pp.__radix_tree_lookup
      1.08 ±  2%    -100.0%       0.00 ± -1%  perf-profile.func.cycles-pp.__xfs_get_blocks
      1.16 ±  0%     -18.1%       0.95 ±  1%  perf-profile.func.cycles-pp._raw_spin_lock
      3.96 ±  2%     -18.4%       3.23 ±  0%  perf-profile.func.cycles-pp.copy_user_enhanced_fast_string
      1.41 ±  3%     -20.6%       1.12 ±  3%  perf-profile.func.cycles-pp.entry_SYSCALL_64_fastpath
      1.30 ±  2%    -100.0%       0.00 ± -1%  perf-profile.func.cycles-pp.generic_perform_write
      1.31 ±  2%     -46.7%       0.70 ±  0%  perf-profile.func.cycles-pp.generic_write_end
     18.43 ±  5%      -9.1%      16.76 ±  0%  perf-profile.func.cycles-pp.intel_idle
      0.00 ± -1%      +Inf%       1.12 ±  1%  perf-profile.func.cycles-pp.iomap_write_actor
      1.50 ±  1%     -20.9%       1.19 ±  1%  perf-profile.func.cycles-pp.mark_buffer_dirty
      0.00 ± -1%      +Inf%       1.91 ±  1%  perf-profile.func.cycles-pp.mark_page_accessed
      3.24 ±  0%     -19.8%       2.60 ±  0%  perf-profile.func.cycles-pp.memset_erms
      1.75 ±  2%     -18.9%       1.42 ±  1%  perf-profile.func.cycles-pp.unlock_page
      1.16 ±  1%     -21.6%       0.91 ±  1%  perf-profile.func.cycles-pp.vfs_write
      0.37 ±  2%    +243.6%       1.26 ±  2%  perf-profile.func.cycles-pp.xfs_bmap_search_extents
      0.41 ±  1%    +198.4%       1.22 ±  2%  perf-profile.func.cycles-pp.xfs_bmap_search_multi_extents
      0.70 ±  5%    +219.5%       2.24 ±  0%  perf-profile.func.cycles-pp.xfs_bmapi_read
      1.05 ±  2%     -15.6%       0.88 ±  3%  perf-profile.func.cycles-pp.xfs_file_write_iter
      0.64 ±  1%    +182.8%       1.81 ±  4%  perf-profile.func.cycles-pp.xfs_iext_bno_to_ext
      0.00 ± -1%      +Inf%       1.10 ±  3%  perf-profile.func.cycles-pp.xfs_iomap_eof_want_preallocate.constprop.8
      0.46 ±  4%    +161.6%       1.20 ±  1%  perf-profile.func.cycles-pp.xfs_iomap_write_delay

Best Regards,
Huang, Ying

WARNING: multiple messages have this Message-ID (diff)
From: Huang, Ying <ying.huang@intel.com>
To: lkp@lists.01.org
Subject: Re: [xfs] 68a9f5e700: aim7.jobs-per-min -13.6% regression
Date: Wed, 10 Aug 2016 17:11:16 -0700	[thread overview]
Message-ID: <87a8gk17x7.fsf@yhuang-mobile.sh.intel.com> (raw)
In-Reply-To: <87eg5w18iu.fsf@yhuang-mobile.sh.intel.com>

[-- Attachment #1: Type: text/plain, Size: 30493 bytes --]

"Huang, Ying" <ying.huang@intel.com> writes:

> Hi, Linus,
>
> Linus Torvalds <torvalds@linux-foundation.org> writes:
>
>> On Wed, Aug 10, 2016 at 4:08 PM, Dave Chinner <david@fromorbit.com> wrote:
>>>
>>> That, to me, says there's a change in lock contention behaviour in
>>> the workload (which we know aim7 is good at exposing). i.e. the
>>> iomap change shifted contention from a sleeping lock to a spinning
>>> lock, or maybe we now trigger optimistic spinning behaviour on a
>>> lock we previously didn't spin on at all.
>>
>> Hmm. Possibly. I reacted to the lower cpu load number, but yeah, I
>> could easily imagine some locking primitive difference too.
>>
>>> We really need instruction level perf profiles to understand
>>> this - I don't have a machine with this many cpu cores available
>>> locally, so I'm not sure I'm going to be able to make any progress
>>> tracking it down in the short term. Maybe the lkp team has more
>>> in-depth cpu usage profiles they can share?
>>
>> Yeah, I've occasionally wanted to see some kind of "top-25 kernel
>> functions in the profile" thing. That said, when the load isn't all
>> that familiar, the profiles usually are not all that easy to make
>> sense of either. But comparing the before and after state might give
>> us clues.
>
> I have started perf-profile data collection, will send out the
> comparison result soon.

Here is the comparison result with perf-profile data.

=========================================================================================
compiler/cpufreq_governor/debug-setup/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase:
  gcc-6/performance/profile/1BRD_48G/xfs/x86_64-rhel/3000/debian-x86_64-2015-02-07.cgz/ivb44/disk_wrt/aim7

commit: 
  f0c6bcba74ac51cb77aadb33ad35cb2dc1ad1506
  68a9f5e7007c1afa2cf6830b690a90d0187c0684

f0c6bcba74ac51cb 68a9f5e7007c1afa2cf6830b69 
---------------- -------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    484435 ±  0%     -13.3%     420004 ±  0%  aim7.jobs-per-min
     37.37 ±  0%     +15.3%      43.09 ±  0%  aim7.time.elapsed_time
     37.37 ±  0%     +15.3%      43.09 ±  0%  aim7.time.elapsed_time.max
      6491 ±  3%     +30.8%       8491 ±  0%  aim7.time.involuntary_context_switches
    376.89 ±  0%     +28.4%     484.11 ±  0%  aim7.time.system_time
    430512 ±  0%     -20.1%     343838 ±  0%  aim7.time.voluntary_context_switches
     26816 ±  8%     +10.2%      29542 ±  1%  interrupts.CAL:Function_call_interrupts
    125122 ± 10%     -10.7%     111758 ± 12%  softirqs.SCHED
     24772 ±  0%     -28.6%      17675 ±  0%  vmstat.system.cs
     53477 ±  2%      +5.6%      56453 ±  0%  vmstat.system.in
     15627 ±  0%     +27.7%      19956 ±  1%  meminfo.Active(file)
     16103 ±  3%     +14.3%      18405 ±  8%  meminfo.AnonHugePages
    132898 ±  9%     +15.4%     153380 ±  1%  meminfo.DirectMap4k
     13777 ±  5%     +43.1%      19709 ±  0%  meminfo.Shmem
      3906 ±  0%     +28.8%       5032 ±  2%  proc-vmstat.nr_active_file
    919.33 ±  5%     +14.8%       1055 ±  8%  proc-vmstat.nr_dirty
      3444 ±  5%     +41.8%       4884 ±  0%  proc-vmstat.nr_shmem
      4092 ± 14%     +61.2%       6595 ±  1%  proc-vmstat.pgactivate
      1975 ± 15%     +63.2%       3224 ± 17%  slabinfo.scsi_data_buffer.active_objs
      1975 ± 15%     +63.2%       3224 ± 17%  slabinfo.scsi_data_buffer.num_objs
    464.33 ± 15%     +63.3%     758.33 ± 17%  slabinfo.xfs_efd_item.active_objs
    464.33 ± 15%     +63.3%     758.33 ± 17%  slabinfo.xfs_efd_item.num_objs
   1724300 ± 27%     -40.5%    1025538 ±  1%  sched_debug.cfs_rq:/.load.max
     96.36 ±  3%     +18.6%     114.32 ± 15%  sched_debug.cfs_rq:/.util_avg.stddev
   1724300 ± 27%     -40.5%    1025538 ±  1%  sched_debug.cpu.load.max
      2887 ± 30%     -28.2%       2073 ± 48%  sched_debug.cpu.nr_load_updates.min
      7.66 ± 20%     -24.9%       5.75 ± 15%  sched_debug.cpu.nr_uninterruptible.stddev
     37.37 ±  0%     +15.3%      43.09 ±  0%  time.elapsed_time
     37.37 ±  0%     +15.3%      43.09 ±  0%  time.elapsed_time.max
      6491 ±  3%     +30.8%       8491 ±  0%  time.involuntary_context_switches
      1037 ±  0%     +10.8%       1148 ±  0%  time.percent_of_cpu_this_job_got
    376.89 ±  0%     +28.4%     484.11 ±  0%  time.system_time
    430512 ±  0%     -20.1%     343838 ±  0%  time.voluntary_context_switches
     24.18 ±  0%      +9.0%      26.35 ±  0%  turbostat.%Busy
    686.00 ±  0%      +9.5%     751.00 ±  0%  turbostat.Avg_MHz
      0.28 ±  0%     -25.0%       0.21 ±  0%  turbostat.CPU%c3
     93.33 ±  1%      +3.0%      96.15 ±  0%  turbostat.CorWatt
    124.61 ±  0%      +2.1%     127.17 ±  0%  turbostat.PkgWatt
      4.74 ±  0%      -2.7%       4.61 ±  1%  turbostat.RAMWatt
      7723 ±  0%     +32.6%      10238 ±  5%  numa-meminfo.node0.Active(file)
      1589 ± 17%     +45.5%       2313 ± 24%  numa-meminfo.node0.Dirty
     56052 ±  3%     +58.2%      88666 ± 17%  numa-meminfo.node1.Active
     48142 ±  4%     +64.0%      78943 ± 19%  numa-meminfo.node1.Active(anon)
      7908 ±  1%     +22.9%       9722 ±  3%  numa-meminfo.node1.Active(file)
     46721 ±  3%     +55.9%      72837 ± 24%  numa-meminfo.node1.AnonPages
      4789 ± 69%    +102.3%       9687 ±  9%  numa-meminfo.node1.Shmem
  52991525 ±  1%     -19.4%   42687208 ±  0%  cpuidle.C1-IVT.time
    319584 ±  1%     -26.5%     234868 ±  1%  cpuidle.C1-IVT.usage
   3468808 ±  2%     -19.8%    2783341 ±  3%  cpuidle.C1E-IVT.time
     46760 ±  0%     -22.4%      36298 ±  0%  cpuidle.C1E-IVT.usage
  12590471 ±  0%     -22.3%    9788585 ±  1%  cpuidle.C3-IVT.time
     79965 ±  0%     -19.0%      64749 ±  0%  cpuidle.C3-IVT.usage
   1.3e+09 ±  0%     +13.3%  1.473e+09 ±  0%  cpuidle.C6-IVT.time
    352.33 ±  8%     -24.7%     265.33 ±  1%  cpuidle.POLL.usage
      1930 ±  0%     +33.9%       2585 ±  3%  numa-vmstat.node0.nr_active_file
      4468 ±  7%      -8.5%       4089 ±  5%  numa-vmstat.node0.nr_alloc_batch
    466.67 ±  4%     +29.3%     603.33 ± 14%  numa-vmstat.node0.nr_dirty
     12026 ±  4%     +64.1%      19734 ± 20%  numa-vmstat.node1.nr_active_anon
      1977 ±  1%     +23.6%       2444 ±  1%  numa-vmstat.node1.nr_active_file
      3809 ±  6%     +16.1%       4422 ±  4%  numa-vmstat.node1.nr_alloc_batch
     11671 ±  3%     +55.9%      18197 ± 24%  numa-vmstat.node1.nr_anon_pages
      1197 ± 69%    +102.3%       2422 ±  9%  numa-vmstat.node1.nr_shmem
    456.33 ± 57%     -75.6%     111.33 ± 86%  numa-vmstat.node1.nr_written
 2.658e+11 ±  4%     +24.7%  3.316e+11 ±  2%  perf-stat.branch-instructions
      0.41 ±  1%      -9.1%       0.37 ±  1%  perf-stat.branch-miss-rate
  1.09e+09 ±  3%     +13.4%  1.237e+09 ±  1%  perf-stat.branch-misses
    981138 ±  0%     -18.1%     803696 ±  0%  perf-stat.context-switches
 1.511e+12 ±  5%     +23.4%  1.864e+12 ±  3%  perf-stat.cpu-cycles
    102600 ±  1%      -7.3%      95075 ±  1%  perf-stat.cpu-migrations
      0.26 ± 12%     -30.8%       0.18 ± 10%  perf-stat.dTLB-load-miss-rate
 3.164e+11 ±  1%     +39.9%  4.426e+11 ±  4%  perf-stat.dTLB-loads
      0.03 ± 26%     -41.3%       0.02 ± 13%  perf-stat.dTLB-store-miss-rate
 2.247e+11 ±  6%     +26.4%  2.839e+11 ±  2%  perf-stat.dTLB-stores
  1.49e+12 ±  4%     +30.1%  1.939e+12 ±  2%  perf-stat.instructions
     43348 ±  2%     +34.2%      58161 ± 12%  perf-stat.instructions-per-iTLB-miss
      0.99 ±  0%      +5.5%       1.04 ±  0%  perf-stat.ipc
    262799 ±  0%      +4.4%     274251 ±  1%  perf-stat.minor-faults
     34.12 ±  1%      +2.1%      34.83 ±  0%  perf-stat.node-load-miss-rate
  46476754 ±  2%      +4.6%   48601269 ±  1%  perf-stat.node-load-misses
      9.96 ±  0%     +13.4%      11.30 ±  0%  perf-stat.node-store-miss-rate
  24460859 ±  1%     +14.4%   27971097 ±  1%  perf-stat.node-store-misses
    262780 ±  0%      +4.4%     274227 ±  1%  perf-stat.page-faults
     11.31 ±  1%     -18.1%       9.27 ±  0%  perf-profile.cycles-pp.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       1.68 ±  1%  perf-profile.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      1.80 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin
      2.55 ±  3%     -14.2%       2.19 ±  2%  perf-profile.cycles-pp.__alloc_pages_nodemask.alloc_pages_current.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin
      0.00 ± -1%      +Inf%       4.45 ±  1%  perf-profile.cycles-pp.__block_commit_write.isra.24.block_write_end.generic_write_end.iomap_write_actor.iomap_apply
      5.93 ±  0%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__block_commit_write.isra.24.block_write_end.generic_write_end.xfs_vm_write_end.generic_perform_write
     13.71 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__block_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter
     10.36 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write
      0.00 ± -1%      +Inf%       3.64 ±  0%  perf-profile.cycles-pp.__block_write_begin_int.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.04 ±  2%     -18.9%       0.84 ±  1%  perf-profile.cycles-pp.__delete_from_page_cache.delete_from_page_cache.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final
     11.24 ±  2%     -18.1%       9.21 ±  0%  perf-profile.cycles-pp.__dentry_kill.dput.__fput.____fput.task_work_run
     11.31 ±  2%     -18.1%       9.26 ±  0%  perf-profile.cycles-pp.__fput.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath
      0.00 ± -1%      +Inf%       1.09 ±  2%  perf-profile.cycles-pp.__mark_inode_dirty.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.32 ±  4%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__mark_inode_dirty.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write
      0.00 ± -1%      +Inf%       2.68 ±  2%  perf-profile.cycles-pp.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      3.04 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write
      1.00 ±  1%     -18.0%       0.82 ±  1%  perf-profile.cycles-pp.__radix_tree_lookup.radix_tree_lookup_slot.find_get_entry.pagecache_get_page.grab_cache_page_write_begin
      1.12 ±  2%     -17.6%       0.92 ±  4%  perf-profile.cycles-pp.__sb_start_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath
      1.38 ±  2%     -13.3%       1.19 ±  1%  perf-profile.cycles-pp.__set_page_dirty.mark_buffer_dirty.__block_commit_write.isra.24.block_write_end.generic_write_end
     54.10 ±  1%     +13.1%      61.20 ±  0%  perf-profile.cycles-pp.__vfs_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath
      6.34 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin
      0.00 ± -1%      +Inf%       3.69 ±  1%  perf-profile.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      4.02 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write
      0.98 ±  5%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.alloc_page_buffers.create_empty_buffers.create_page_buffers.__block_write_begin_int.__block_write_begin
      0.00 ± -1%      +Inf%       2.56 ±  2%  perf-profile.cycles-pp.alloc_pages_current.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      2.91 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.alloc_pages_current.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin
      3.42 ±  0%     -20.9%       2.71 ±  2%  perf-profile.cycles-pp.block_invalidatepage.xfs_vm_invalidatepage.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final
      0.00 ± -1%      +Inf%       4.69 ±  0%  perf-profile.cycles-pp.block_write_end.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      6.24 ±  0%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.block_write_end.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write
     19.18 ±  5%      -9.3%      17.40 ±  0%  perf-profile.cycles-pp.call_cpuidle.cpu_startup_entry.start_secondary
      0.94 ±  4%     -19.8%       0.76 ±  0%  perf-profile.cycles-pp.cancel_dirty_page.try_to_free_buffers.xfs_vm_releasepage.try_to_release_page.block_invalidatepage
      3.95 ±  2%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.copy_user_enhanced_fast_string.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write
      0.00 ± -1%      +Inf%       3.22 ±  0%  perf-profile.cycles-pp.copy_user_enhanced_fast_string.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
     19.75 ±  5%      -9.8%      17.81 ±  0%  perf-profile.cycles-pp.cpu_startup_entry.start_secondary
     19.18 ±  5%      -9.3%      17.40 ±  0%  perf-profile.cycles-pp.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary
     18.45 ±  5%      -9.2%      16.75 ±  0%  perf-profile.cycles-pp.cpuidle_enter_state.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary
      1.44 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.create_empty_buffers.create_page_buffers.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin
      0.00 ± -1%      +Inf%       1.18 ±  1%  perf-profile.cycles-pp.create_empty_buffers.create_page_buffers.__block_write_begin_int.iomap_write_begin.iomap_write_actor
      1.86 ±  2%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.create_page_buffers.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin.generic_perform_write
      0.00 ± -1%      +Inf%       1.53 ±  1%  perf-profile.cycles-pp.create_page_buffers.__block_write_begin_int.iomap_write_begin.iomap_write_actor.iomap_apply
      1.74 ±  2%     -19.9%       1.40 ±  3%  perf-profile.cycles-pp.delete_from_page_cache.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final.evict
      1.27 ±  0%     -22.5%       0.99 ±  4%  perf-profile.cycles-pp.destroy_inode.evict.iput.__dentry_kill.dput
      2.61 ±  1%     -24.3%       1.98 ±  1%  perf-profile.cycles-pp.do_filp_open.do_sys_open.sys_creat.entry_SYSCALL_64_fastpath
      2.66 ±  1%     -24.3%       2.01 ±  1%  perf-profile.cycles-pp.do_sys_open.sys_creat.entry_SYSCALL_64_fastpath
      1.79 ±  2%     -28.2%       1.28 ±  3%  perf-profile.cycles-pp.do_unlinkat.sys_unlink.entry_SYSCALL_64_fastpath
      1.07 ±  3%     -23.3%       0.82 ±  3%  perf-profile.cycles-pp.down_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write
      1.01 ±  3%     -17.9%       0.83 ±  2%  perf-profile.cycles-pp.down_write.xfs_ilock.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write
     11.26 ±  2%     -18.1%       9.23 ±  0%  perf-profile.cycles-pp.dput.__fput.____fput.task_work_run.exit_to_usermode_loop
     11.21 ±  2%     -18.1%       9.18 ±  0%  perf-profile.cycles-pp.evict.iput.__dentry_kill.dput.__fput
     11.34 ±  2%     -18.1%       9.29 ±  0%  perf-profile.cycles-pp.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       1.55 ±  3%  perf-profile.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      1.83 ±  2%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write
     43.95 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write
      0.00 ± -1%      +Inf%       7.91 ±  1%  perf-profile.cycles-pp.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
     10.68 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter
      1.91 ±  3%     -16.4%       1.59 ±  1%  perf-profile.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_current.__page_cache_alloc.pagecache_get_page
      0.00 ± -1%      +Inf%       9.85 ±  0%  perf-profile.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
     10.96 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter
      0.00 ± -1%      +Inf%      52.29 ±  0%  perf-profile.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write
      0.00 ± -1%      +Inf%      52.94 ±  0%  perf-profile.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write
      0.00 ± -1%      +Inf%      34.35 ±  0%  perf-profile.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter
      0.00 ± -1%      +Inf%      16.48 ±  0%  perf-profile.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
     11.22 ±  2%     -18.1%       9.19 ±  0%  perf-profile.cycles-pp.iput.__dentry_kill.dput.__fput.____fput
      0.00 ± -1%      +Inf%       1.55 ±  1%  perf-profile.cycles-pp.lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      1.72 ±  2%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin
      0.00 ± -1%      +Inf%       2.78 ±  0%  perf-profile.cycles-pp.mark_buffer_dirty.__block_commit_write.isra.24.block_write_end.generic_write_end.iomap_write_actor
      3.39 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.mark_buffer_dirty.__block_commit_write.isra.24.block_write_end.generic_write_end.xfs_vm_write_end
      0.00 ± -1%      +Inf%       3.44 ±  1%  perf-profile.cycles-pp.mark_page_accessed.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      3.03 ±  0%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.memset_erms.__block_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write
      0.00 ± -1%      +Inf%       2.43 ±  0%  perf-profile.cycles-pp.memset_erms.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      0.00 ± -1%      +Inf%       9.25 ±  0%  perf-profile.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
     10.37 ±  2%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write
      2.58 ±  1%     -24.1%       1.96 ±  0%  perf-profile.cycles-pp.path_openat.do_filp_open.do_sys_open.sys_creat.entry_SYSCALL_64_fastpath
      1.17 ±  3%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.radix_tree_lookup_slot.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin
      2.06 ±  3%     -22.5%       1.60 ±  2%  perf-profile.cycles-pp.rw_verify_area.vfs_write.sys_write.entry_SYSCALL_64_fastpath
      1.79 ±  3%     -22.2%       1.39 ±  0%  perf-profile.cycles-pp.security_file_permission.rw_verify_area.vfs_write.sys_write.entry_SYSCALL_64_fastpath
      1.32 ±  4%     -21.4%       1.04 ±  0%  perf-profile.cycles-pp.selinux_file_permission.security_file_permission.rw_verify_area.vfs_write.sys_write
     19.79 ±  5%      -9.9%      17.84 ±  0%  perf-profile.cycles-pp.start_secondary
      2.67 ±  1%     -24.2%       2.02 ±  1%  perf-profile.cycles-pp.sys_creat.entry_SYSCALL_64_fastpath
      1.79 ±  3%     -27.9%       1.29 ±  3%  perf-profile.cycles-pp.sys_unlink.entry_SYSCALL_64_fastpath
     60.98 ±  1%      +9.5%      66.76 ±  0%  perf-profile.cycles-pp.sys_write.entry_SYSCALL_64_fastpath
     11.34 ±  1%     -18.1%       9.29 ±  0%  perf-profile.cycles-pp.syscall_return_slowpath.entry_SYSCALL_64_fastpath
     11.32 ±  1%     -18.0%       9.28 ±  0%  perf-profile.cycles-pp.task_work_run.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath
      5.96 ±  1%     -20.0%       4.77 ±  0%  perf-profile.cycles-pp.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final.evict.iput
      9.89 ±  2%     -17.4%       8.17 ±  0%  perf-profile.cycles-pp.truncate_inode_pages_final.evict.iput.__dentry_kill.dput
      9.87 ±  2%     -17.5%       8.15 ±  0%  perf-profile.cycles-pp.truncate_inode_pages_range.truncate_inode_pages_final.evict.iput.__dentry_kill
      2.07 ±  1%     -20.4%       1.65 ±  2%  perf-profile.cycles-pp.try_to_free_buffers.xfs_vm_releasepage.try_to_release_page.block_invalidatepage.xfs_vm_invalidatepage
      2.40 ±  1%     -21.0%       1.89 ±  2%  perf-profile.cycles-pp.try_to_release_page.block_invalidatepage.xfs_vm_invalidatepage.truncate_inode_page.truncate_inode_pages_range
      0.00 ± -1%      +Inf%       1.36 ±  1%  perf-profile.cycles-pp.unlock_page.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      1.72 ±  4%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.unlock_page.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write
     59.63 ±  1%     +10.2%      65.72 ±  0%  perf-profile.cycles-pp.vfs_write.sys_write.entry_SYSCALL_64_fastpath
      0.00 ± -1%      +Inf%       1.52 ±  2%  perf-profile.cycles-pp.workingset_activation.mark_page_accessed.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      0.00 ± -1%      +Inf%       1.73 ±  1%  perf-profile.cycles-pp.xfs_bmap_search_extents.xfs_bmapi_delay.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply
      0.00 ± -1%      +Inf%       1.97 ±  2%  perf-profile.cycles-pp.xfs_bmap_search_extents.xfs_bmapi_read.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write
      0.00 ± -1%      +Inf%       1.61 ±  2%  perf-profile.cycles-pp.xfs_bmap_search_extents.xfs_bmapi_read.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay.xfs_file_iomap_begin
      0.00 ± -1%      +Inf%       1.24 ±  2%  perf-profile.cycles-pp.xfs_bmap_search_multi_extents.xfs_bmap_search_extents.xfs_bmapi_delay.xfs_iomap_write_delay.xfs_file_iomap_begin
      0.00 ± -1%      +Inf%       1.46 ±  1%  perf-profile.cycles-pp.xfs_bmap_search_multi_extents.xfs_bmap_search_extents.xfs_bmapi_read.xfs_file_iomap_begin.iomap_apply
      0.00 ± -1%      +Inf%       1.21 ±  2%  perf-profile.cycles-pp.xfs_bmap_search_multi_extents.xfs_bmap_search_extents.xfs_bmapi_read.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay
      1.25 ±  0%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.xfs_bmapi_delay.xfs_iomap_write_delay.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int
      0.00 ± -1%      +Inf%       3.06 ±  1%  perf-profile.cycles-pp.xfs_bmapi_delay.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write
      1.04 ±  0%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.xfs_bmapi_read.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int.__block_write_begin
      0.00 ± -1%      +Inf%       3.04 ±  1%  perf-profile.cycles-pp.xfs_bmapi_read.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      0.00 ± -1%      +Inf%       3.05 ±  1%  perf-profile.cycles-pp.xfs_bmapi_read.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply
      1.32 ±  2%     -21.5%       1.04 ±  1%  perf-profile.cycles-pp.xfs_create.xfs_generic_create.xfs_vn_mknod.xfs_vn_create.path_openat
     51.83 ±  1%     +14.3%      59.25 ±  0%  perf-profile.cycles-pp.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write.sys_write
      0.00 ± -1%      +Inf%      16.05 ±  0%  perf-profile.cycles-pp.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter
     53.16 ±  1%     +13.6%      60.40 ±  0%  perf-profile.cycles-pp.xfs_file_write_iter.__vfs_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath
      1.24 ±  1%     -23.1%       0.95 ±  4%  perf-profile.cycles-pp.xfs_fs_destroy_inode.destroy_inode.evict.iput.__dentry_kill
      1.42 ±  2%     -21.2%       1.12 ±  1%  perf-profile.cycles-pp.xfs_generic_create.xfs_vn_mknod.xfs_vn_create.path_openat.do_filp_open
      6.46 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.xfs_get_blocks.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin.generic_perform_write
      1.29 ±  3%     -18.9%       1.04 ±  1%  perf-profile.cycles-pp.xfs_ilock.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write
      0.00 ± -1%      +Inf%       1.14 ±  3%  perf-profile.cycles-pp.xfs_ilock.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      1.21 ±  1%     -23.4%       0.93 ±  4%  perf-profile.cycles-pp.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.evict.iput
      1.23 ±  4%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.xfs_iomap_eof_want_preallocate.constprop.6.xfs_iomap_write_delay.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int
      0.00 ± -1%      +Inf%       4.14 ±  0%  perf-profile.cycles-pp.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write
      3.28 ±  2%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.xfs_iomap_write_delay.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int.__block_write_begin
      0.00 ± -1%      +Inf%       9.08 ±  0%  perf-profile.cycles-pp.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      3.54 ±  0%     -20.8%       2.81 ±  1%  perf-profile.cycles-pp.xfs_vm_invalidatepage.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final.evict
      2.35 ±  1%     -21.0%       1.86 ±  1%  perf-profile.cycles-pp.xfs_vm_releasepage.try_to_release_page.block_invalidatepage.xfs_vm_invalidatepage.truncate_inode_page
     25.10 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write
     11.03 ±  1%    -100.0%       0.00 ± -1%  perf-profile.cycles-pp.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write
      1.42 ±  2%     -20.7%       1.13 ±  1%  perf-profile.cycles-pp.xfs_vn_create.path_openat.do_filp_open.do_sys_open.sys_creat
      1.42 ±  2%     -20.5%       1.13 ±  1%  perf-profile.cycles-pp.xfs_vn_mknod.xfs_vn_create.path_openat.do_filp_open.do_sys_open
      2.27 ±  1%     -10.6%       2.03 ±  0%  perf-profile.func.cycles-pp.___might_sleep
      2.49 ±  0%     -34.5%       1.63 ±  1%  perf-profile.func.cycles-pp.__block_commit_write.isra.24
      1.51 ±  2%     +15.4%       1.75 ±  1%  perf-profile.func.cycles-pp.__block_write_begin_int
      1.79 ±  4%     -16.8%       1.49 ±  1%  perf-profile.func.cycles-pp.__mark_inode_dirty
      1.32 ±  0%     -16.4%       1.10 ±  1%  perf-profile.func.cycles-pp.__radix_tree_lookup
      1.08 ±  2%    -100.0%       0.00 ± -1%  perf-profile.func.cycles-pp.__xfs_get_blocks
      1.16 ±  0%     -18.1%       0.95 ±  1%  perf-profile.func.cycles-pp._raw_spin_lock
      3.96 ±  2%     -18.4%       3.23 ±  0%  perf-profile.func.cycles-pp.copy_user_enhanced_fast_string
      1.41 ±  3%     -20.6%       1.12 ±  3%  perf-profile.func.cycles-pp.entry_SYSCALL_64_fastpath
      1.30 ±  2%    -100.0%       0.00 ± -1%  perf-profile.func.cycles-pp.generic_perform_write
      1.31 ±  2%     -46.7%       0.70 ±  0%  perf-profile.func.cycles-pp.generic_write_end
     18.43 ±  5%      -9.1%      16.76 ±  0%  perf-profile.func.cycles-pp.intel_idle
      0.00 ± -1%      +Inf%       1.12 ±  1%  perf-profile.func.cycles-pp.iomap_write_actor
      1.50 ±  1%     -20.9%       1.19 ±  1%  perf-profile.func.cycles-pp.mark_buffer_dirty
      0.00 ± -1%      +Inf%       1.91 ±  1%  perf-profile.func.cycles-pp.mark_page_accessed
      3.24 ±  0%     -19.8%       2.60 ±  0%  perf-profile.func.cycles-pp.memset_erms
      1.75 ±  2%     -18.9%       1.42 ±  1%  perf-profile.func.cycles-pp.unlock_page
      1.16 ±  1%     -21.6%       0.91 ±  1%  perf-profile.func.cycles-pp.vfs_write
      0.37 ±  2%    +243.6%       1.26 ±  2%  perf-profile.func.cycles-pp.xfs_bmap_search_extents
      0.41 ±  1%    +198.4%       1.22 ±  2%  perf-profile.func.cycles-pp.xfs_bmap_search_multi_extents
      0.70 ±  5%    +219.5%       2.24 ±  0%  perf-profile.func.cycles-pp.xfs_bmapi_read
      1.05 ±  2%     -15.6%       0.88 ±  3%  perf-profile.func.cycles-pp.xfs_file_write_iter
      0.64 ±  1%    +182.8%       1.81 ±  4%  perf-profile.func.cycles-pp.xfs_iext_bno_to_ext
      0.00 ± -1%      +Inf%       1.10 ±  3%  perf-profile.func.cycles-pp.xfs_iomap_eof_want_preallocate.constprop.8
      0.46 ±  4%    +161.6%       1.20 ±  1%  perf-profile.func.cycles-pp.xfs_iomap_write_delay

Best Regards,
Huang, Ying

  reply	other threads:[~2016-08-11  0:11 UTC|newest]

Thread overview: 219+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-08-09 14:33 [lkp] [xfs] 68a9f5e700: aim7.jobs-per-min -13.6% regression kernel test robot
2016-08-09 14:33 ` kernel test robot
2016-08-10 18:24 ` [lkp] " Linus Torvalds
2016-08-10 18:24   ` Linus Torvalds
2016-08-10 23:08   ` [lkp] " Dave Chinner
2016-08-10 23:08     ` Dave Chinner
2016-08-10 23:51     ` [lkp] " Linus Torvalds
2016-08-10 23:51       ` Linus Torvalds
2016-08-10 23:58       ` [LKP] [lkp] " Huang, Ying
2016-08-10 23:58         ` Huang, Ying
2016-08-11  0:11         ` Huang, Ying [this message]
2016-08-11  0:11           ` Huang, Ying
2016-08-11  0:23           ` [LKP] [lkp] " Linus Torvalds
2016-08-11  0:23             ` Linus Torvalds
2016-08-11  0:33             ` [LKP] [lkp] " Huang, Ying
2016-08-11  0:33               ` Huang, Ying
2016-08-11  1:00               ` [LKP] [lkp] " Linus Torvalds
2016-08-11  1:00                 ` Linus Torvalds
2016-08-11  4:46                 ` [LKP] [lkp] " Dave Chinner
2016-08-11  4:46                   ` Dave Chinner
2016-08-15 17:22                   ` [LKP] [lkp] " Huang, Ying
2016-08-15 17:22                     ` Huang, Ying
2016-08-16  0:08                     ` [LKP] [lkp] " Dave Chinner
2016-08-16  0:08                       ` Dave Chinner
2016-08-11 15:57                 ` [LKP] [lkp] " Christoph Hellwig
2016-08-11 15:57                   ` Christoph Hellwig
2016-08-11 16:55                   ` [LKP] [lkp] " Linus Torvalds
2016-08-11 16:55                     ` Linus Torvalds
2016-08-11 17:51                     ` [LKP] [lkp] " Huang, Ying
2016-08-11 17:51                       ` Huang, Ying
2016-08-11 19:51                       ` [LKP] [lkp] " Linus Torvalds
2016-08-11 19:51                         ` Linus Torvalds
2016-08-11 20:00                         ` [LKP] [lkp] " Christoph Hellwig
2016-08-11 20:00                           ` Christoph Hellwig
2016-08-11 20:35                           ` [LKP] [lkp] " Linus Torvalds
2016-08-11 20:35                             ` Linus Torvalds
2016-08-11 22:16                             ` [LKP] [lkp] " Al Viro
2016-08-11 22:16                               ` Al Viro
2016-08-11 22:30                               ` [LKP] [lkp] " Linus Torvalds
2016-08-11 22:30                                 ` Linus Torvalds
2016-08-11 21:16                           ` [LKP] [lkp] " Huang, Ying
2016-08-11 21:16                             ` Huang, Ying
2016-08-11 21:40                             ` [LKP] [lkp] " Linus Torvalds
2016-08-11 21:40                               ` Linus Torvalds
2016-08-11 22:08                               ` [LKP] [lkp] " Christoph Hellwig
2016-08-11 22:08                                 ` Christoph Hellwig
2016-08-12  0:54                     ` [LKP] [lkp] " Dave Chinner
2016-08-12  0:54                       ` Dave Chinner
2016-08-12  2:23                       ` [LKP] [lkp] " Dave Chinner
2016-08-12  2:23                         ` Dave Chinner
2016-08-12  2:32                         ` [LKP] [lkp] " Linus Torvalds
2016-08-12  2:32                           ` Linus Torvalds
2016-08-12  2:52                         ` [LKP] [lkp] " Christoph Hellwig
2016-08-12  2:52                           ` Christoph Hellwig
2016-08-12  3:20                           ` [LKP] [lkp] " Linus Torvalds
2016-08-12  3:20                             ` Linus Torvalds
2016-08-12  4:16                             ` [LKP] [lkp] " Dave Chinner
2016-08-12  4:16                               ` Dave Chinner
2016-08-12  5:02                               ` [LKP] [lkp] " Linus Torvalds
2016-08-12  5:02                                 ` Linus Torvalds
2016-08-12  6:04                                 ` [LKP] [lkp] " Dave Chinner
2016-08-12  6:04                                   ` Dave Chinner
2016-08-12  6:29                                   ` [LKP] [lkp] " Ye Xiaolong
2016-08-12  6:29                                     ` Ye Xiaolong
2016-08-12  8:51                                     ` [LKP] [lkp] " Ye Xiaolong
2016-08-12  8:51                                       ` Ye Xiaolong
2016-08-12 10:02                                       ` [LKP] [lkp] " Dave Chinner
2016-08-12 10:02                                         ` Dave Chinner
2016-08-12 10:43                                         ` Fengguang Wu
2016-08-12 10:43                                           ` Fengguang Wu
2016-08-13  0:30                                         ` [LKP] [lkp] " Christoph Hellwig
2016-08-13  0:30                                           ` Christoph Hellwig
2016-08-13 21:48                                           ` [LKP] [lkp] " Christoph Hellwig
2016-08-13 21:48                                             ` Christoph Hellwig
2016-08-13 22:07                                             ` [LKP] [lkp] " Fengguang Wu
2016-08-13 22:07                                               ` Fengguang Wu
2016-08-13 22:15                                               ` [LKP] [lkp] " Christoph Hellwig
2016-08-13 22:15                                                 ` Christoph Hellwig
2016-08-13 22:51                                                 ` [LKP] [lkp] " Fengguang Wu
2016-08-13 22:51                                                   ` Fengguang Wu
2016-08-14 14:50                                                   ` [LKP] [lkp] " Fengguang Wu
2016-08-14 14:50                                                     ` Fengguang Wu
2016-08-14 16:17                                                     ` [LKP] [lkp] " Christoph Hellwig
2016-08-14 16:17                                                       ` Christoph Hellwig
2016-08-14 23:46                                                       ` [LKP] [lkp] " Dave Chinner
2016-08-14 23:46                                                         ` Dave Chinner
2016-08-14 23:57                                                       ` [LKP] [lkp] " Fengguang Wu
2016-08-14 23:57                                                         ` Fengguang Wu
2016-08-15 14:14                                                       ` [LKP] [lkp] " Fengguang Wu
2016-08-15 14:14                                                         ` Fengguang Wu
2016-08-15 21:22                                                         ` [LKP] [lkp] " Dave Chinner
2016-08-15 21:22                                                           ` Dave Chinner
2016-08-16 12:20                                                           ` [LKP] [lkp] " Fengguang Wu
2016-08-16 12:20                                                             ` Fengguang Wu
2016-08-15 20:30                                                       ` [LKP] [lkp] " Huang, Ying
2016-08-15 20:30                                                         ` Huang, Ying
2016-08-22 22:09                                                         ` [LKP] [lkp] " Huang, Ying
2016-08-22 22:09                                                           ` Huang, Ying
2016-09-26  6:25                                                           ` [LKP] [lkp] " Huang, Ying
2016-09-26  6:25                                                             ` Huang, Ying
2016-09-26 14:55                                                             ` [LKP] [lkp] " Christoph Hellwig
2016-09-26 14:55                                                               ` Christoph Hellwig
2016-09-27  0:52                                                               ` [LKP] [lkp] " Huang, Ying
2016-09-27  0:52                                                                 ` Huang, Ying
2016-08-16 13:25                                                       ` [LKP] [lkp] " Fengguang Wu
2016-08-16 13:25                                                         ` Fengguang Wu
2016-08-13 23:32                                           ` [LKP] [lkp] " Dave Chinner
2016-08-13 23:32                                             ` Dave Chinner
2016-08-12  2:27                       ` [LKP] [lkp] " Linus Torvalds
2016-08-12  2:27                         ` Linus Torvalds
2016-08-12  3:56                         ` [LKP] [lkp] " Dave Chinner
2016-08-12  3:56                           ` Dave Chinner
2016-08-12 18:03                           ` [LKP] [lkp] " Linus Torvalds
2016-08-12 18:03                             ` Linus Torvalds
2016-08-13 23:58                             ` [LKP] [lkp] " Fengguang Wu
2016-08-13 23:58                               ` Fengguang Wu
2016-08-15  0:48                             ` [LKP] [lkp] " Dave Chinner
2016-08-15  0:48                               ` Dave Chinner
2016-08-15  1:37                               ` [LKP] [lkp] " Linus Torvalds
2016-08-15  1:37                                 ` Linus Torvalds
2016-08-15  2:28                                 ` [LKP] [lkp] " Dave Chinner
2016-08-15  2:28                                   ` Dave Chinner
2016-08-15  2:53                                   ` [LKP] [lkp] " Linus Torvalds
2016-08-15  2:53                                     ` Linus Torvalds
2016-08-15  5:00                                     ` [LKP] [lkp] " Dave Chinner
2016-08-15  5:00                                       ` Dave Chinner
     [not found]                                       ` <CA+55aFwva2Xffai+Eqv1Jn_NGryk3YJ2i5JoHOQnbQv6qVPAsw@mail.gmail.com>
     [not found]                                         ` <CA+55aFy14nUnJQ_GdF=j8Fa9xiH70c6fY2G3q5HQ01+8z1z3qQ@mail.gmail.com>
2016-08-15  5:12                                           ` Linus Torvalds
2016-08-15 22:22                                             ` [LKP] [lkp] " Dave Chinner
2016-08-15 22:22                                               ` Dave Chinner
2016-08-15 22:42                                               ` [LKP] [lkp] " Dave Chinner
2016-08-15 22:42                                                 ` Dave Chinner
2016-08-15 23:20                                                 ` [LKP] [lkp] " Linus Torvalds
2016-08-15 23:20                                                   ` Linus Torvalds
2016-08-15 23:48                                                   ` [LKP] [lkp] " Linus Torvalds
2016-08-15 23:48                                                     ` Linus Torvalds
2016-08-16  0:44                                                     ` [LKP] [lkp] " Dave Chinner
2016-08-16  0:44                                                       ` Dave Chinner
2016-08-16 15:05                                                     ` [LKP] [lkp] " Mel Gorman
2016-08-16 15:05                                                       ` Mel Gorman
2016-08-16 17:47                                                       ` [LKP] [lkp] " Linus Torvalds
2016-08-16 17:47                                                         ` Linus Torvalds
2016-08-17 15:48                                                         ` [LKP] [lkp] " Michal Hocko
2016-08-17 15:48                                                           ` Michal Hocko
2016-08-17 16:42                                                           ` [LKP] [lkp] " Michal Hocko
2016-08-17 16:42                                                             ` Michal Hocko
2016-08-17 15:49                                                         ` [LKP] [lkp] " Mel Gorman
2016-08-17 15:49                                                           ` Mel Gorman
2016-08-18  0:45                                                           ` [LKP] [lkp] " Mel Gorman
2016-08-18  0:45                                                             ` Mel Gorman
2016-08-18  7:11                                                             ` [LKP] [lkp] " Dave Chinner
2016-08-18  7:11                                                               ` Dave Chinner
2016-08-18 13:24                                                               ` [LKP] [lkp] " Mel Gorman
2016-08-18 13:24                                                                 ` Mel Gorman
2016-08-18 17:55                                                                 ` [LKP] [lkp] " Linus Torvalds
2016-08-18 17:55                                                                   ` Linus Torvalds
2016-08-18 21:19                                                                   ` [LKP] [lkp] " Dave Chinner
2016-08-18 21:19                                                                     ` Dave Chinner
2016-08-18 22:25                                                                     ` [LKP] [lkp] " Linus Torvalds
2016-08-18 22:25                                                                       ` Linus Torvalds
2016-08-19  9:00                                                                       ` [LKP] [lkp] " Michal Hocko
2016-08-19  9:00                                                                         ` Michal Hocko
2016-08-19 10:49                                                                       ` [LKP] [lkp] " Mel Gorman
2016-08-19 10:49                                                                         ` Mel Gorman
2016-08-19 23:48                                                                         ` [LKP] [lkp] " Dave Chinner
2016-08-19 23:48                                                                           ` Dave Chinner
2016-08-20  1:08                                                                           ` [LKP] [lkp] " Linus Torvalds
2016-08-20  1:08                                                                             ` Linus Torvalds
2016-08-20 12:16                                                                           ` [LKP] [lkp] " Mel Gorman
2016-08-20 12:16                                                                             ` Mel Gorman
2016-08-19 15:08                                                               ` [LKP] [lkp] " Mel Gorman
2016-08-19 15:08                                                                 ` Mel Gorman
2016-09-01 23:32                                                                 ` [LKP] [lkp] " Dave Chinner
2016-09-01 23:32                                                                   ` Dave Chinner
2016-09-06 15:37                                                                   ` [LKP] [lkp] " Mel Gorman
2016-09-06 15:37                                                                     ` Mel Gorman
2016-09-06 15:52                                                                     ` [LKP] [lkp] " Huang, Ying
2016-09-06 15:52                                                                       ` Huang, Ying
2016-08-24 15:40                                                             ` [LKP] [lkp] " Huang, Ying
2016-08-24 15:40                                                               ` Huang, Ying
2016-08-25  9:37                                                               ` [LKP] [lkp] " Mel Gorman
2016-08-25  9:37                                                                 ` Mel Gorman
2016-08-18  2:44                                                           ` [LKP] [lkp] " Dave Chinner
2016-08-18  2:44                                                             ` Dave Chinner
2016-08-16  0:15                                                   ` [LKP] [lkp] " Linus Torvalds
2016-08-16  0:15                                                     ` Linus Torvalds
2016-08-16  0:38                                                     ` [LKP] [lkp] " Dave Chinner
2016-08-16  0:38                                                       ` Dave Chinner
2016-08-16  0:50                                                       ` [LKP] [lkp] " Linus Torvalds
2016-08-16  0:50                                                         ` Linus Torvalds
2016-08-16  0:19                                                   ` [LKP] [lkp] " Dave Chinner
2016-08-16  0:19                                                     ` Dave Chinner
2016-08-16  1:51                                                     ` [LKP] [lkp] " Linus Torvalds
2016-08-16  1:51                                                       ` Linus Torvalds
2016-08-16 22:02                                                       ` [LKP] [lkp] " Dave Chinner
2016-08-16 22:02                                                         ` Dave Chinner
2016-08-16 23:23                                                         ` [LKP] [lkp] " Linus Torvalds
2016-08-16 23:23                                                           ` Linus Torvalds
2016-08-15 23:01                                               ` [LKP] [lkp] " Linus Torvalds
2016-08-15 23:01                                                 ` Linus Torvalds
2016-08-16  0:17                                                 ` [LKP] [lkp] " Dave Chinner
2016-08-16  0:17                                                   ` Dave Chinner
2016-08-16  0:45                                                   ` [LKP] [lkp] " Linus Torvalds
2016-08-16  0:45                                                     ` Linus Torvalds
2016-08-15  5:03                                     ` [LKP] [lkp] " Ingo Molnar
2016-08-15  5:03                                       ` Ingo Molnar
2016-08-17 16:24                                       ` [LKP] [lkp] " Peter Zijlstra
2016-08-17 16:24                                         ` Peter Zijlstra
2016-08-15 12:58                             ` [LKP] [lkp] " Fengguang Wu
2016-08-15 12:58                               ` Fengguang Wu
2016-08-11  1:16               ` [LKP] [lkp] " Dave Chinner
2016-08-11  1:16                 ` Dave Chinner
2016-08-11  1:32                 ` [LKP] [lkp] " Dave Chinner
2016-08-11  1:32                   ` Dave Chinner
2016-08-11  2:36                   ` [LKP] [lkp] " Ye Xiaolong
2016-08-11  2:36                     ` Ye Xiaolong
2016-08-11  3:05                     ` [LKP] [lkp] " Dave Chinner
2016-08-11  3:05                       ` Dave Chinner
2016-08-12  1:26                 ` [LKP] [lkp] " Dave Chinner
2016-08-12  1:26                   ` Dave Chinner

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87a8gk17x7.fsf@yhuang-mobile.sh.intel.com \
    --to=ying.huang@intel.com \
    --cc=david@fromorbit.com \
    --cc=fengguang.wu@intel.com \
    --cc=hch@lst.de \
    --cc=linux-kernel@vger.kernel.org \
    --cc=lkp@01.org \
    --cc=rpeterso@redhat.com \
    --cc=torvalds@linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.