Re: [LKP] [lkp] [xfs] 68a9f5e700: aim7.jobs-per-min -13.6% regression

[Fengguang Wu <fengguang.wu@intel.com>]

Hi Christoph,

On Sun, Aug 14, 2016 at 06:17:24PM +0200, Christoph Hellwig wrote:
>Snipping the long contest:
>
>I think there are three observations here:
>
> (1) removing the mark_page_accessed (which is the only significant
>     change in the parent commit)  hurts the
>     aim7/1BRD_48G-xfs-disk_rr-3000-performance/ivb44 test.
>     I'd still rather stick to the filemap version and let the
>     VM people sort it out.  How do the numbers for this test
>     look for XFS vs say ext4 and btrfs?
> (2) lots of additional spinlock contention in the new case.  A quick
>     check shows that I fat-fingered my rewrite so that we do
>     the xfs_inode_set_eofblocks_tag call now for the pure lookup
>     case, and pretty much all new cycles come from that.
> (3) Boy, are those xfs_inode_set_eofblocks_tag calls expensive, and
>     we're already doing way to many even without my little bug above.
>
>So I've force pushed a new version of the iomap-fixes branch with
>(2) fixed, and also a little patch to xfs_inode_set_eofblocks_tag a
>lot less expensive slotted in before that.  Would be good to see
>the numbers with that.

The aim7 1BRD tests finished and there are ups and downs, with overall
performance remain flat.

99091700659f4df9  74a242ad94d13436a1644c0b45  bf4dc6e4ecc2a3d042029319bc  testcase/testparams/testbox
----------------  --------------------------  --------------------------  ---------------------------
         %stddev     %change         %stddev     %change         %stddev
             \          |                \          |                \  
    159926                      157324                      158574        GEO-MEAN aim7.jobs-per-min
     70897               5%      74137               4%      73775        aim7/1BRD_48G-xfs-creat-clo-1500-performance/ivb44
    485217 ±  3%                492431                      477533        aim7/1BRD_48G-xfs-disk_rd-9000-performance/ivb44
    360451             -19%     292980             -17%     299377        aim7/1BRD_48G-xfs-disk_rr-3000-performance/ivb44
    338114                      338410               5%     354078        aim7/1BRD_48G-xfs-disk_rw-3000-performance/ivb44
     60130 ±  5%         4%      62438               5%      62923        aim7/1BRD_48G-xfs-disk_src-3000-performance/ivb44
    403144                      397790                      410648        aim7/1BRD_48G-xfs-disk_wrt-3000-performance/ivb44
     26327                       26534                       26128        aim7/1BRD_48G-xfs-sync_disk_rw-600-performance/ivb44

The new commit bf4dc6e ("xfs: rewrite and optimize the delalloc write
path") improves the aim7/1BRD_48G-xfs-disk_rw-3000-performance/ivb44
case by 5%. Here are the detailed numbers:

aim7/1BRD_48G-xfs-disk_rw-3000-performance/ivb44

74a242ad94d13436  bf4dc6e4ecc2a3d042029319bc
----------------  --------------------------
         %stddev     %change         %stddev
             \          |                \
    338410               5%     354078        aim7.jobs-per-min
    404390               8%     435117        aim7.time.voluntary_context_switches
      2502              -4%       2396        aim7.time.maximum_resident_set_size
     15018              -9%      13701        aim7.time.involuntary_context_switches
       900             -11%        801        aim7.time.system_time
     17432              11%      19365        vmstat.system.cs
     47736 ± 19%       -24%      36087        interrupts.CAL:Function_call_interrupts
   2129646              31%    2790638        proc-vmstat.pgalloc_dma32
    379503              13%     429384        numa-meminfo.node0.Dirty
     15018              -9%      13701        time.involuntary_context_switches
       900             -11%        801        time.system_time
      1560              10%       1716        slabinfo.mnt_cache.active_objs
      1560              10%       1716        slabinfo.mnt_cache.num_objs
     61.53               -4      57.45 ±  4%  perf-profile.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.call_cpuidle.cpu_startup_entry
     61.63               -4      57.55 ±  4%  perf-profile.func.cycles-pp.intel_idle
   1007188 ± 16%       156%    2577911 ±  6%  numa-numastat.node0.numa_miss
   9662857 ±  4%       -13%    8420159 ±  3%  numa-numastat.node0.numa_foreign
   1008220 ± 16%       155%    2570630 ±  6%  numa-numastat.node1.numa_foreign
   9664033 ±  4%       -13%    8413184 ±  3%  numa-numastat.node1.numa_miss
  26519887 ±  3%        18%   31322674        cpuidle.C1-IVT.time
    122238              16%     142383        cpuidle.C1-IVT.usage
     46548              11%      51645        cpuidle.C1E-IVT.usage
  17253419              13%   19567582        cpuidle.C3-IVT.time
     86847              13%      98333        cpuidle.C3-IVT.usage
    482033 ± 12%       108%    1000665 ±  8%  numa-vmstat.node0.numa_miss
     94689              14%     107744        numa-vmstat.node0.nr_zone_write_pending
     94677              14%     107718        numa-vmstat.node0.nr_dirty
   3156643 ±  3%       -20%    2527460 ±  3%  numa-vmstat.node0.numa_foreign
    429288 ± 12%       129%     983053 ±  8%  numa-vmstat.node1.numa_foreign
   3104193 ±  3%       -19%    2510128        numa-vmstat.node1.numa_miss
      6.43 ±  5%        51%       9.70 ± 11%  turbostat.Pkg%pc2
      0.30              28%       0.38        turbostat.CPU%c3
      9.71                        9.92        turbostat.RAMWatt
       158                         154        turbostat.PkgWatt
       125              -3%        121        turbostat.CorWatt
      1141              -6%       1078        turbostat.Avg_MHz
     38.70              -6%      36.48        turbostat.%Busy
      5.03 ± 11%       -51%       2.46 ± 40%  turbostat.Pkg%pc6
      8.33 ± 48%        88%      15.67 ± 36%  sched_debug.cfs_rq:/.runnable_load_avg.max
      1947 ±  3%       -12%       1710 ±  7%  sched_debug.cfs_rq:/.spread0.stddev
      1936 ±  3%       -12%       1698 ±  8%  sched_debug.cfs_rq:/.min_vruntime.stddev
      2170 ± 10%       -14%       1863 ±  6%  sched_debug.cfs_rq:/.load_avg.max
    220926 ± 18%        37%     303192 ±  5%  sched_debug.cpu.avg_idle.stddev
      0.06 ± 13%       357%       0.28 ± 23%  sched_debug.rt_rq:/.rt_time.avg
      0.37 ± 10%       240%       1.25 ± 15%  sched_debug.rt_rq:/.rt_time.stddev
      2.54 ± 10%       160%       6.59 ± 10%  sched_debug.rt_rq:/.rt_time.max
      0.32 ± 19%        29%       0.42 ± 10%  perf-stat.dTLB-load-miss-rate
    964727               7%    1028830        perf-stat.context-switches
    176406               4%     184289        perf-stat.cpu-migrations
      0.29               4%       0.30        perf-stat.branch-miss-rate
 1.634e+09                   1.673e+09        perf-stat.node-store-misses
     23.60                       23.99        perf-stat.node-store-miss-rate
     40.01                       40.57        perf-stat.cache-miss-rate
      0.95              -8%       0.87        perf-stat.ipc
 3.203e+12              -9%  2.928e+12        perf-stat.cpu-cycles
 1.506e+09             -11%  1.345e+09        perf-stat.branch-misses
     50.64 ± 13%       -14%      43.45 ±  4%  perf-stat.iTLB-load-miss-rate
 5.285e+11             -14%  4.523e+11        perf-stat.branch-instructions
 3.042e+12             -16%  2.551e+12        perf-stat.instructions
 7.996e+11             -18%  6.584e+11        perf-stat.dTLB-loads
 5.569e+11 ±  4%       -18%  4.578e+11        perf-stat.dTLB-stores


Here are the detailed numbers for the slowed down case:

aim7/1BRD_48G-xfs-disk_rr-3000-performance/ivb44

99091700659f4df9  bf4dc6e4ecc2a3d042029319bc
----------------  --------------------------
         %stddev      change         %stddev
             \          |                \
    360451             -17%     299377        aim7.jobs-per-min
     12806             481%      74447        aim7.time.involuntary_context_switches
       755              44%       1086        aim7.time.system_time
     50.17              20%      60.36        aim7.time.elapsed_time
     50.17              20%      60.36        aim7.time.elapsed_time.max
    438148                      446012        aim7.time.voluntary_context_switches
     37798 ± 16%       780%     332583 ±  8%  interrupts.CAL:Function_call_interrupts
     78.82 ±  5%        18%      93.35 ±  5%  uptime.boot
      2847 ±  7%        11%       3160 ±  7%  uptime.idle
    147490 ±  8%        34%     197261 ±  3%  softirqs.RCU
    648159              29%     839283        softirqs.TIMER
    160830              10%     177144        softirqs.SCHED
   3845352 ±  4%        91%    7349133        numa-numastat.node0.numa_miss
   4686838 ±  5%        67%    7835640        numa-numastat.node0.numa_foreign
   3848455 ±  4%        91%    7352436        numa-numastat.node1.numa_foreign
   4689920 ±  5%        67%    7838734        numa-numastat.node1.numa_miss
     50.17              20%      60.36        time.elapsed_time.max
     12806             481%      74447        time.involuntary_context_switches
       755              44%       1086        time.system_time
     50.17              20%      60.36        time.elapsed_time
      1563              18%       1846        time.percent_of_cpu_this_job_got
     11699 ± 19%      3738%     449048        vmstat.io.bo
  18836969             -16%   15789996        vmstat.memory.free
        16              19%         19        vmstat.procs.r
     19377             459%     108364        vmstat.system.cs
     48255              11%      53537        vmstat.system.in
   2357299              25%    2951384        meminfo.Inactive(file)
   2366381              25%    2960468        meminfo.Inactive
   1575292              -9%    1429971        meminfo.Cached
  19342499             -17%   16100340        meminfo.MemFree
   1057904             -20%     842987        meminfo.Dirty
      1057              21%       1284        turbostat.Avg_MHz
     35.78              21%      43.24        turbostat.%Busy
      9.95              15%      11.47        turbostat.RAMWatt
        74 ±  5%        10%         81        turbostat.CoreTmp
        74 ±  4%        10%         81        turbostat.PkgTmp
       118               8%        128        turbostat.CorWatt
       151               7%        162        turbostat.PkgWatt
     29.06             -23%      22.39        turbostat.CPU%c6
       487 ± 89%      3e+04      26448 ± 57%  latency_stats.max.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agf.xfs_alloc_read_agf.xfs_alloc_fix_freelist.xfs_free_extent_fix_freelist.xfs_free_extent.xfs_trans_free_extent
      1823 ± 82%      2e+06    1913796 ± 38%  latency_stats.sum.down.xfs_buf_lock._xfs_buf_find.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agf.xfs_alloc_read_agf.xfs_alloc_fix_freelist.xfs_free_extent_fix_freelist.xfs_free_extent.xfs_trans_free_extent
    208475 ± 43%      1e+06    1409494 ±  5%  latency_stats.sum.wait_on_page_bit.truncate_inode_pages_range.truncate_inode_pages_final.evict.iput.dentry_unlink_inode.__dentry_kill.dput.__fput.____fput.task_work_run.exit_to_usermode_loop
      6884 ± 73%      8e+04      90790 ±  9%  latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_commit.xfs_vn_update_time.file_update_time.xfs_file_aio_write_checks.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write.SyS_write
      1598 ± 20%      3e+04      35015 ± 27%  latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.__xfs_trans_roll.xfs_trans_roll.xfs_itruncate_extents.xfs_free_eofblocks.xfs_release.xfs_file_release.__fput.____fput.task_work_run
      2006 ± 25%      3e+04      31143 ± 35%  latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.__xfs_trans_roll.xfs_trans_roll.xfs_itruncate_extents.xfs_inactive_truncate.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.evict.iput
        29 ±101%      1e+04      10214 ± 29%  latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.__xfs_trans_roll.xfs_trans_roll.xfs_defer_trans_roll.xfs_defer_finish.xfs_itruncate_extents.xfs_inactive_truncate.xfs_inactive.xfs_fs_destroy_inode.destroy_inode
      1206 ± 51%      9e+03       9919 ± 25%  latency_stats.sum.call_rwsem_down_read_failed.xfs_log_commit_cil.__xfs_trans_commit.xfs_trans_commit.xfs_vn_update_time.touch_atime.generic_file_read_iter.xfs_file_buffered_aio_read.xfs_file_read_iter.__vfs_read.vfs_read.SyS_read
  29869205 ±  4%       -10%   26804569        cpuidle.C1-IVT.time
   5737726              39%    7952214        cpuidle.C1E-IVT.time
     51141              17%      59958        cpuidle.C1E-IVT.usage
  18377551              37%   25176426        cpuidle.C3-IVT.time
     96067              17%     112045        cpuidle.C3-IVT.usage
   1806811              12%    2024041        cpuidle.C6-IVT.usage
   1104420 ± 36%       204%    3361085 ± 27%  cpuidle.POLL.time
       281 ± 10%        20%        338        cpuidle.POLL.usage
      5.61 ± 11%       -0.5       5.12 ± 18%  perf-profile.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter.call_cpuidle
      5.85 ±  6%       -0.8       5.06 ± 15%  perf-profile.cycles-pp.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter
      6.32 ±  6%       -0.9       5.42 ± 15%  perf-profile.cycles-pp.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter.call_cpuidle
     15.77 ±  8%         -2      13.83 ± 17%  perf-profile.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter.call_cpuidle.cpu_startup_entry
     16.04 ±  8%         -2      14.01 ± 15%  perf-profile.cycles-pp.apic_timer_interrupt.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary
     60.25 ±  4%         -7      53.03 ±  7%  perf-profile.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.call_cpuidle.cpu_startup_entry
     60.41 ±  4%         -7      53.12 ±  7%  perf-profile.func.cycles-pp.intel_idle
   1174104              22%    1436859        numa-meminfo.node0.Inactive
   1167471              22%    1428271        numa-meminfo.node0.Inactive(file)
    770811              -9%     698147        numa-meminfo.node0.FilePages
  20707294             -12%   18281509 ±  6%  numa-meminfo.node0.Active
  20613745             -12%   18180987 ±  6%  numa-meminfo.node0.Active(file)
   9676639             -17%    8003627        numa-meminfo.node0.MemFree
    509906             -22%     396192        numa-meminfo.node0.Dirty
   1189539              28%    1524697        numa-meminfo.node1.Inactive(file)
   1191989              28%    1525194        numa-meminfo.node1.Inactive
    804508             -10%     727067        numa-meminfo.node1.FilePages
   9654540             -16%    8077810        numa-meminfo.node1.MemFree
    547956             -19%     441933        numa-meminfo.node1.Dirty
       396 ± 12%       485%       2320 ± 37%  slabinfo.bio-1.num_objs
       396 ± 12%       481%       2303 ± 37%  slabinfo.bio-1.active_objs
        73             140%        176 ± 14%  slabinfo.kmalloc-128.active_slabs
        73             140%        176 ± 14%  slabinfo.kmalloc-128.num_slabs
      4734              94%       9171 ± 11%  slabinfo.kmalloc-128.num_objs
      4734              88%       8917 ± 13%  slabinfo.kmalloc-128.active_objs
     16238             -10%      14552 ±  3%  slabinfo.kmalloc-256.active_objs
     17189             -13%      15033 ±  3%  slabinfo.kmalloc-256.num_objs
     20651              96%      40387 ± 17%  slabinfo.radix_tree_node.active_objs
       398              91%        761 ± 17%  slabinfo.radix_tree_node.active_slabs
       398              91%        761 ± 17%  slabinfo.radix_tree_node.num_slabs
     22313              91%      42650 ± 17%  slabinfo.radix_tree_node.num_objs
        32             638%        236 ± 28%  slabinfo.xfs_efd_item.active_slabs
        32             638%        236 ± 28%  slabinfo.xfs_efd_item.num_slabs
      1295             281%       4934 ± 23%  slabinfo.xfs_efd_item.num_objs
      1295             280%       4923 ± 23%  slabinfo.xfs_efd_item.active_objs
      1661              81%       3000 ± 42%  slabinfo.xfs_log_ticket.num_objs
      1661              78%       2952 ± 42%  slabinfo.xfs_log_ticket.active_objs
      2617              49%       3905 ± 30%  slabinfo.xfs_trans.num_objs
      2617              48%       3870 ± 31%  slabinfo.xfs_trans.active_objs
   1015933             567%    6779099        perf-stat.context-switches
 4.864e+08             126%  1.101e+09        perf-stat.node-load-misses
 1.179e+09             103%  2.399e+09        perf-stat.node-loads
      0.06 ± 34%        92%       0.12 ± 11%  perf-stat.dTLB-store-miss-rate
 2.985e+08 ± 32%        86%  5.542e+08 ± 11%  perf-stat.dTLB-store-misses
 2.551e+09 ± 15%        81%  4.625e+09 ± 13%  perf-stat.dTLB-load-misses
      0.39 ± 14%        66%       0.65 ± 13%  perf-stat.dTLB-load-miss-rate
  1.26e+09              60%  2.019e+09        perf-stat.node-store-misses
  46072661 ± 27%        49%   68472915        perf-stat.iTLB-loads
 2.738e+12 ±  4%        43%  3.916e+12        perf-stat.cpu-cycles
     21.48              32%      28.35        perf-stat.node-store-miss-rate
 1.612e+10 ±  3%        28%  2.066e+10        perf-stat.cache-references
 1.669e+09 ±  3%        24%  2.063e+09        perf-stat.branch-misses
 6.816e+09 ±  3%        20%  8.179e+09        perf-stat.cache-misses
    177699              18%     209145        perf-stat.cpu-migrations
      0.39              13%       0.44        perf-stat.branch-miss-rate
 4.606e+09              11%  5.102e+09        perf-stat.node-stores
 4.329e+11 ±  4%         9%  4.727e+11        perf-stat.branch-instructions
 6.458e+11               9%  7.046e+11        perf-stat.dTLB-loads
     29.19               8%      31.45        perf-stat.node-load-miss-rate
    286173               8%     308115        perf-stat.page-faults
    286191               8%     308109        perf-stat.minor-faults
  45084934               4%   47073719        perf-stat.iTLB-load-misses
     42.28              -6%      39.58        perf-stat.cache-miss-rate
     50.62 ± 16%       -19%      40.75        perf-stat.iTLB-load-miss-rate
      0.89             -28%       0.64        perf-stat.ipc
         2 ± 36%     4e+07%     970191        proc-vmstat.pgrotated
       150 ± 21%     1e+07%   15356485 ±  3%  proc-vmstat.nr_vmscan_immediate_reclaim
     76823 ± 35%     56899%   43788651        proc-vmstat.pgscan_direct
    153407 ± 19%      4483%    7031431        proc-vmstat.nr_written
    619699 ± 19%      4441%   28139689        proc-vmstat.pgpgout
   5342421            1061%   62050709        proc-vmstat.pgactivate
        47 ± 25%       354%        217        proc-vmstat.nr_pages_scanned
   8542963 ±  3%        78%   15182914        proc-vmstat.numa_miss
   8542963 ±  3%        78%   15182715        proc-vmstat.numa_foreign
   2820568              31%    3699073        proc-vmstat.pgalloc_dma32
    589234              25%     738160        proc-vmstat.nr_zone_inactive_file
    589240              25%     738155        proc-vmstat.nr_inactive_file
  61347830              13%   69522958        proc-vmstat.pgfree
    393711              -9%     356981        proc-vmstat.nr_file_pages
   4831749             -17%    4020131        proc-vmstat.nr_free_pages
  61252784             -18%   50183773        proc-vmstat.pgrefill
  61245420             -18%   50176301        proc-vmstat.pgdeactivate
    264397             -20%     210222        proc-vmstat.nr_zone_write_pending
    264367             -20%     210188        proc-vmstat.nr_dirty
  60420248             -39%   36646178        proc-vmstat.pgscan_kswapd
  60373976             -44%   33735064        proc-vmstat.pgsteal_kswapd
      1753             -98%         43 ± 18%  proc-vmstat.pageoutrun
      1095             -98%         25 ± 17%  proc-vmstat.kswapd_low_wmark_hit_quickly
       656 ±  3%       -98%         15 ± 24%  proc-vmstat.kswapd_high_wmark_hit_quickly
         0                     1136221        numa-vmstat.node0.workingset_refault
         0                     1136221        numa-vmstat.node0.workingset_activate
        23 ± 45%     1e+07%    2756907        numa-vmstat.node0.nr_vmscan_immediate_reclaim
     37618 ± 24%      3234%    1254165        numa-vmstat.node0.nr_written
   1346538 ±  4%       104%    2748439        numa-vmstat.node0.numa_miss
   1577620 ±  5%        80%    2842882        numa-vmstat.node0.numa_foreign
    291242              23%     357407        numa-vmstat.node0.nr_inactive_file
    291237              23%     357390        numa-vmstat.node0.nr_zone_inactive_file
  13961935              12%   15577331        numa-vmstat.node0.numa_local
  13961938              12%   15577332        numa-vmstat.node0.numa_hit
     39831              10%      43768        numa-vmstat.node0.nr_unevictable
     39831              10%      43768        numa-vmstat.node0.nr_zone_unevictable
    193467             -10%     174639        numa-vmstat.node0.nr_file_pages
   5147212             -12%    4542321 ±  6%  numa-vmstat.node0.nr_active_file
   5147237             -12%    4542325 ±  6%  numa-vmstat.node0.nr_zone_active_file
   2426129             -17%    2008637        numa-vmstat.node0.nr_free_pages
    128285             -23%      99206        numa-vmstat.node0.nr_zone_write_pending
    128259             -23%      99183        numa-vmstat.node0.nr_dirty
         0                     1190594        numa-vmstat.node1.workingset_refault
         0                     1190594        numa-vmstat.node1.workingset_activate
        21 ± 36%     1e+07%    3120425 ±  4%  numa-vmstat.node1.nr_vmscan_immediate_reclaim
     38541 ± 26%      3336%    1324185        numa-vmstat.node1.nr_written
   1316819 ±  4%       105%    2699075        numa-vmstat.node1.numa_foreign
   1547929 ±  4%        80%    2793491        numa-vmstat.node1.numa_miss
    296714              28%     381124        numa-vmstat.node1.nr_zone_inactive_file
    296714              28%     381123        numa-vmstat.node1.nr_inactive_file
  14311131              10%   15750908        numa-vmstat.node1.numa_hit
  14311130              10%   15750905        numa-vmstat.node1.numa_local
    201164             -10%     181742        numa-vmstat.node1.nr_file_pages
   2422825             -16%    2027750        numa-vmstat.node1.nr_free_pages
    137069             -19%     110501        numa-vmstat.node1.nr_zone_write_pending
    137069             -19%     110497        numa-vmstat.node1.nr_dirty
       737 ± 29%     27349%     202387        sched_debug.cfs_rq:/.min_vruntime.min
      3637 ± 20%      7919%     291675        sched_debug.cfs_rq:/.min_vruntime.avg
     11.00 ± 44%      4892%     549.17 ±  9%  sched_debug.cfs_rq:/.runnable_load_avg.max
      2.12 ± 36%      4853%     105.12 ±  5%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
      1885 ±  6%      4189%      80870        sched_debug.cfs_rq:/.min_vruntime.stddev
      1896 ±  6%      4166%      80895        sched_debug.cfs_rq:/.spread0.stddev
     10774 ± 13%      4113%     453925        sched_debug.cfs_rq:/.min_vruntime.max
      1.02 ± 19%      2630%      27.72 ±  7%  sched_debug.cfs_rq:/.runnable_load_avg.avg
     63060 ± 45%       776%     552157        sched_debug.cfs_rq:/.load.max
     14442 ± 21%       590%      99615 ± 14%  sched_debug.cfs_rq:/.load.stddev
      8397 ±  9%       309%      34370 ± 12%  sched_debug.cfs_rq:/.load.avg
     46.02 ± 24%       176%     126.96 ±  6%  sched_debug.cfs_rq:/.util_avg.stddev
       817              19%        974 ±  3%  sched_debug.cfs_rq:/.util_avg.max
       721             -17%        600 ±  3%  sched_debug.cfs_rq:/.util_avg.avg
       595 ± 11%       -38%        371 ±  7%  sched_debug.cfs_rq:/.util_avg.min
      1484 ± 20%       -47%        792 ±  5%  sched_debug.cfs_rq:/.load_avg.min
      1798 ±  4%       -50%        903 ±  5%  sched_debug.cfs_rq:/.load_avg.avg
       322 ±  8%      7726%      25239 ±  8%  sched_debug.cpu.nr_switches.min
       969            7238%      71158        sched_debug.cpu.nr_switches.avg
      2.23 ± 40%      4650%     106.14 ±  4%  sched_debug.cpu.cpu_load[0].stddev
       943 ±  4%      3475%      33730 ±  3%  sched_debug.cpu.nr_switches.stddev
      0.87 ± 25%      3057%      27.46 ±  7%  sched_debug.cpu.cpu_load[0].avg
      5.43 ± 13%      2232%     126.61        sched_debug.cpu.nr_uninterruptible.stddev
      6131 ±  3%      2028%     130453        sched_debug.cpu.nr_switches.max
      1.58 ± 29%      1852%      30.90 ±  4%  sched_debug.cpu.cpu_load[4].avg
      2.00 ± 49%      1422%      30.44 ±  5%  sched_debug.cpu.cpu_load[3].avg
     63060 ± 45%      1053%     726920 ± 32%  sched_debug.cpu.load.max
     21.25 ± 44%       777%     186.33 ±  7%  sched_debug.cpu.nr_uninterruptible.max
     14419 ± 21%       731%     119865 ± 31%  sched_debug.cpu.load.stddev
      3586             381%      17262        sched_debug.cpu.nr_load_updates.min
      8286 ±  8%       364%      38414 ± 17%  sched_debug.cpu.load.avg
      5444             303%      21956        sched_debug.cpu.nr_load_updates.avg
      1156             231%       3827        sched_debug.cpu.nr_load_updates.stddev
      8603 ±  4%       222%      27662        sched_debug.cpu.nr_load_updates.max
      1410             165%       3735        sched_debug.cpu.curr->pid.max
     28742 ± 15%       120%      63101 ±  7%  sched_debug.cpu.clock.min
     28742 ± 15%       120%      63101 ±  7%  sched_debug.cpu.clock_task.min
     28748 ± 15%       120%      63107 ±  7%  sched_debug.cpu.clock.avg
     28748 ± 15%       120%      63107 ±  7%  sched_debug.cpu.clock_task.avg
     28751 ± 15%       120%      63113 ±  7%  sched_debug.cpu.clock.max
     28751 ± 15%       120%      63113 ±  7%  sched_debug.cpu.clock_task.max
       442 ± 11%        93%        854 ± 15%  sched_debug.cpu.curr->pid.avg
       618 ±  3%        72%       1065 ±  4%  sched_debug.cpu.curr->pid.stddev
      1.88 ± 11%        50%       2.83 ±  8%  sched_debug.cpu.clock.stddev
      1.88 ± 11%        50%       2.83 ±  8%  sched_debug.cpu.clock_task.stddev
      5.22 ±  9%       -55%       2.34 ± 23%  sched_debug.rt_rq:/.rt_time.max
      0.85             -55%       0.38 ± 28%  sched_debug.rt_rq:/.rt_time.stddev
      0.17             -56%       0.07 ± 33%  sched_debug.rt_rq:/.rt_time.avg
     27633 ± 16%       124%      61980 ±  8%  sched_debug.ktime
     28745 ± 15%       120%      63102 ±  7%  sched_debug.sched_clk
     28745 ± 15%       120%      63102 ±  7%  sched_debug.cpu_clk

Thanks,
Fengguang