All of lore.kernel.org
 help / color / mirror / Atom feed
* [locking/rwsem] 00f3c5a3df: will-it-scale.per_thread_ops -5.1% regression
@ 2019-09-05  7:52 kernel test robot
  0 siblings, 0 replies; only message in thread
From: kernel test robot @ 2019-09-05  7:52 UTC (permalink / raw)
  To: lkp

[-- Attachment #1: Type: text/plain, Size: 592953 bytes --]

Greeting,

FYI, we noticed a -5.1% regression of will-it-scale.per_thread_ops due to commit:


commit: 00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f ("locking/rwsem: Always release wait_lock before waking up tasks")
https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux.git master

in testcase: will-it-scale
on test machine: 96 threads Intel(R) Xeon(R) CPU @ 2.30GHz with 128G memory
with following parameters:

	nr_task: 50%
	mode: thread
	test: mmap2
	ucode: 0x400001c
	cpufreq_governor: performance

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale

In addition to that, the commit also has significant impact on the following tests:

+------------------+------------------------------------------------------------------------+
| testcase: change | reaim: boot-time.boot 2.6% regression                                  |
| test machine     | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory |
| test parameters  | cpufreq_governor=performance                                           |
|                  | iterations=30                                                          |
|                  | nr_task=1600%                                                          |
|                  | test=new_dbase                                                         |
|                  | ucode=0x14                                                             |
+------------------+------------------------------------------------------------------------+
| testcase: change | aim7:                                                                  |
| test machine     | 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory  |
| test parameters  | cpufreq_governor=performance                                           |
|                  | disk=4BRD_12G                                                          |
|                  | fs=ext4                                                                |
|                  | load=3000                                                              |
|                  | md=RAID1                                                               |
|                  | test=disk_src                                                          |
+------------------+------------------------------------------------------------------------+
| testcase: change | aim7:                                                                  |
| test machine     | 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory  |
| test parameters  | cpufreq_governor=performance                                           |
|                  | disk=4BRD_12G                                                          |
|                  | fs=xfs                                                                 |
|                  | load=3000                                                              |
|                  | md=RAID1                                                               |
|                  | test=disk_wrt                                                          |
+------------------+------------------------------------------------------------------------+
| testcase: change | aim7: iostat.md0.wkB/s 56.8% improvement                               |
| test machine     | 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory  |
| test parameters  | cpufreq_governor=performance                                           |
|                  | disk=4BRD_12G                                                          |
|                  | fs=xfs                                                                 |
|                  | load=3000                                                              |
|                  | md=RAID1                                                               |
|                  | test=disk_wrt                                                          |
+------------------+------------------------------------------------------------------------+
| testcase: change | reaim: reaim.child_utime -1.6% undefined                               |
| test machine     | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory |
| test parameters  | cpufreq_governor=performance                                           |
|                  | iterations=20                                                          |
|                  | nr_task=1600%                                                          |
|                  | test=aim9                                                              |
|                  | ucode=0x14                                                             |
+------------------+------------------------------------------------------------------------+
| testcase: change | unixbench:                                                             |
| test machine     | 192 threads Skylake-SP with 256G memory                                |
| test parameters  | cpufreq_governor=performance                                           |
|                  | nr_task=30%                                                            |
|                  | runtime=300s                                                           |
|                  | test=shell8                                                            |
+------------------+------------------------------------------------------------------------+
| testcase: change | will-it-scale:                                                         |
| test machine     | 192 threads Skylake-SP with 256G memory                                |
| test parameters  | cpufreq_governor=performance                                           |
|                  | mode=process                                                           |
|                  | nr_task=100%                                                           |
|                  | test=pwrite2                                                           |
+------------------+------------------------------------------------------------------------+
| testcase: change | reaim: boot-time.boot -4.1% improvement                                |
| test machine     | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory |
| test parameters  | cpufreq_governor=performance                                           |
|                  | iterations=10                                                          |
|                  | nr_task=1600%                                                          |
|                  | test=custom                                                            |
|                  | ucode=0x14                                                             |
+------------------+------------------------------------------------------------------------+
| testcase: change | aim7:                                                                  |
| test machine     | 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 496G memory |
| test parameters  | cpufreq_governor=performance                                           |
|                  | load=2000                                                              |
|                  | test=fork_test                                                         |
|                  | ucode=0xb000036                                                        |
+------------------+------------------------------------------------------------------------+
| testcase: change | aim7:                                                                  |
| test machine     | 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 496G memory |
| test parameters  | cpufreq_governor=performance                                           |
|                  | load=8000                                                              |
|                  | test=misc_rtns_1                                                       |
|                  | ucode=0xb000036                                                        |
+------------------+------------------------------------------------------------------------+
| testcase: change | will-it-scale:                                                         |
| test machine     | 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory   |
| test parameters  | cpufreq_governor=performance                                           |
|                  | mode=thread                                                            |
|                  | nr_task=100%                                                           |
|                  | test=unlink1                                                           |
+------------------+------------------------------------------------------------------------+
| testcase: change | reaim: reaim.child_systime 23.3% undefined                             |
| test machine     | 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory  |
| test parameters  | cpufreq_governor=performance                                           |
|                  | nr_task=1600%                                                          |
|                  | runtime=300s                                                           |
|                  | test=alltests                                                          |
|                  | ucode=0x43                                                             |
+------------------+------------------------------------------------------------------------+
| testcase: change | reaim: reaim.child_systime -2.7% undefined                             |
| test machine     | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory   |
| test parameters  | cpufreq_governor=performance                                           |
|                  | nr_task=900t                                                           |
|                  | runtime=300s                                                           |
|                  | test=disk_cp                                                           |
|                  | ucode=0x42e                                                            |
+------------------+------------------------------------------------------------------------+
| testcase: change | reaim:                                                                 |
| test machine     | 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory  |
| test parameters  | cpufreq_governor=performance                                           |
|                  | nr_task=1600%                                                          |
|                  | runtime=300s                                                           |
|                  | test=new_dbase                                                         |
|                  | ucode=0x43                                                             |
+------------------+------------------------------------------------------------------------+
| testcase: change | reaim:                                                                 |
| test machine     | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory   |
| test parameters  | cpufreq_governor=performance                                           |
|                  | nr_task=300t                                                           |
|                  | runtime=300s                                                           |
|                  | test=disk_wrt                                                          |
|                  | ucode=0x42e                                                            |
+------------------+------------------------------------------------------------------------+
| testcase: change | vm-scalability: boot-time.boot 2.0% regression                         |
| test machine     | 160 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 256G memory |
| test parameters  | cpufreq_governor=performance                                           |
|                  | runtime=300s                                                           |
|                  | test=small-allocs                                                      |
|                  | ucode=0xb000036                                                        |
+------------------+------------------------------------------------------------------------+
| testcase: change | reaim: boot-time.boot 3.3% regression                                  |
| test machine     | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory |
| test parameters  | cpufreq_governor=performance                                           |
|                  | iterations=30                                                          |
|                  | nr_task=1600%                                                          |
|                  | test=dbase                                                             |
|                  | ucode=0x14                                                             |
+------------------+------------------------------------------------------------------------+
| testcase: change | aim7:                                                                  |
| test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory  |
| test parameters  | cpufreq_governor=performance                                           |
|                  | load=2000                                                              |
|                  | test=fork_test                                                         |
|                  | ucode=0xb000036                                                        |
+------------------+------------------------------------------------------------------------+
| testcase: change | aim7:                                                                  |
| test machine     | 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 496G memory |
| test parameters  | cpufreq_governor=performance                                           |
|                  | load=8000                                                              |
|                  | test=disk_src                                                          |
|                  | ucode=0xb000036                                                        |
+------------------+------------------------------------------------------------------------+
| testcase: change | reaim:                                                                 |
| test machine     | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory   |
| test parameters  | cpufreq_governor=performance                                           |
|                  | nr_task=7000t                                                          |
|                  | runtime=300s                                                           |
|                  | test=shared_memory                                                     |
|                  | ucode=0x42e                                                            |
+------------------+------------------------------------------------------------------------+
| testcase: change | reaim:                                                                 |
| test machine     | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory   |
| test parameters  | cpufreq_governor=performance                                           |
|                  | nr_task=400t                                                           |
|                  | runtime=300s                                                           |
|                  | test=disk_src                                                          |
|                  | ucode=0x42e                                                            |
+------------------+------------------------------------------------------------------------+
| testcase: change | fxmark: iostat.sda.wkB/s 1.8% improvement                              |
| test machine     | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory   |
| test parameters  | cpufreq_governor=performance                                           |
|                  | directio=directio                                                      |
|                  | disk=1HDD                                                              |
|                  | fstype=xfs                                                             |
|                  | media=hdd                                                              |
|                  | test=MWUL                                                              |
|                  | ucode=0x42e                                                            |
+------------------+------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@intel.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.2-clear/thread/50%/clear-ota-25590-x86_64-2018-10-18.cgz/lkp-csl-2sp4/mmap2/will-it-scale/0x400001c

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
      5762            -5.1%       5467        will-it-scale.per_thread_ops
    176.00            +1.3%     178.25        will-it-scale.time.percent_of_cpu_this_job_got
    521.49            +1.2%     527.84        will-it-scale.time.system_time
  56180812            -2.1%   55004407        will-it-scale.time.voluntary_context_switches
    276626            -5.1%     262480        will-it-scale.workload
      6836            -1.8%       6710        boot-time.idle
   4584795           -12.7%    4004010        meminfo.Committed_AS
      1501 ±  5%     -15.4%       1270 ±  5%  slabinfo.task_group.active_objs
      1501 ±  5%     -15.4%       1270 ±  5%  slabinfo.task_group.num_objs
    371197            -4.5%     354429 ±  2%  vmstat.system.cs
    206566            -1.9%     202660        vmstat.system.in
  4.91e+08 ± 11%     -23.4%  3.762e+08 ±  4%  cpuidle.C1.time
   5335198 ±  3%      -9.2%    4845346 ±  4%  cpuidle.C1.usage
   9660501 ±  2%     -12.4%    8459367 ±  6%  cpuidle.POLL.usage
   5332157 ±  3%      -9.2%    4843565 ±  4%  turbostat.C1
      1.69 ± 11%      -0.4        1.26 ±  3%  turbostat.C1%
      0.39 ±  5%    +184.1%       1.12 ± 59%  turbostat.CPU%c6
   2637533 ±  8%     +18.5%    3124786 ±  5%  numa-vmstat.node0.numa_hit
   2621013 ±  8%     +18.8%    3114498 ±  5%  numa-vmstat.node0.numa_local
   2195535 ± 12%     -19.2%    1773999 ± 10%  numa-vmstat.node1.numa_hit
    986533 ± 27%     -43.4%     558697 ± 33%  numa-vmstat.node1.numa_local
    258.33 ± 16%     +27.8%     330.25 ±  8%  sched_debug.cfs_rq:/.runnable_load_avg.max
    256.92 ± 17%     +28.5%     330.25 ±  8%  sched_debug.cpu.cpu_load[0].max
    276147 ± 30%     +41.1%     389714 ±  7%  sched_debug.cpu.load.max
     59345 ±  3%      +9.7%      65073 ±  9%  sched_debug.cpu.max_idle_balance_cost.avg
      2.60 ± 16%     -17.3%       2.15 ± 13%  sched_debug.cpu.nr_uninterruptible.stddev
      8182 ±133%    +541.6%      52501 ± 50%  interrupts.CPU10.RES:Rescheduling_interrupts
     72730 ± 18%     -79.8%      14676 ± 60%  interrupts.CPU30.RES:Rescheduling_interrupts
     68164 ± 32%     -56.6%      29579 ± 55%  interrupts.CPU49.RES:Rescheduling_interrupts
    115055 ± 36%     -71.6%      32664 ± 49%  interrupts.CPU52.RES:Rescheduling_interrupts
     29073 ± 71%    +150.2%      72747 ± 43%  interrupts.CPU54.RES:Rescheduling_interrupts
     54063 ± 38%     -61.0%      21064 ± 54%  interrupts.CPU67.RES:Rescheduling_interrupts
     82518 ± 29%     -69.0%      25542 ± 76%  interrupts.CPU73.RES:Rescheduling_interrupts
      2782 ± 31%     +39.5%       3880 ±  8%  interrupts.CPU77.CAL:Function_call_interrupts
     24230 ± 22%     +71.2%      41476 ± 33%  interrupts.CPU79.RES:Rescheduling_interrupts
      3298 ± 11%     +18.4%       3906 ±  8%  interrupts.CPU85.CAL:Function_call_interrupts
      3200 ±  6%     +22.0%       3903 ±  8%  interrupts.CPU86.CAL:Function_call_interrupts
    641.50 ± 27%     +37.2%     880.00 ± 11%  interrupts.CPU89.NMI:Non-maskable_interrupts
    641.50 ± 27%     +37.2%     880.00 ± 11%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
      3241 ± 13%     +20.3%       3900 ±  8%  interrupts.CPU92.CAL:Function_call_interrupts
    674.00 ± 20%     +25.2%     844.00 ±  7%  interrupts.CPU95.NMI:Non-maskable_interrupts
    674.00 ± 20%     +25.2%     844.00 ±  7%  interrupts.CPU95.PMI:Performance_monitoring_interrupts
   4571304 ±  9%     -23.3%    3507482 ±  5%  interrupts.RES:Rescheduling_interrupts
    306.75 ±  8%     +16.7%     358.00 ±  5%  interrupts.TLB:TLB_shootdowns
     10.88           +13.6%      12.36 ±  9%  perf-stat.i.MPKI
 1.174e+09            -3.6%  1.131e+09 ±  2%  perf-stat.i.branch-instructions
    375713            -4.2%     359844 ±  2%  perf-stat.i.context-switches
      1.88            +9.1%       2.05 ±  7%  perf-stat.i.cpi
 1.466e+09            -3.6%  1.413e+09 ±  2%  perf-stat.i.dTLB-loads
 8.112e+08            -4.2%  7.773e+08 ±  2%  perf-stat.i.dTLB-stores
     71.13            -1.2       69.98        perf-stat.i.iTLB-load-miss-rate%
 5.361e+09            -3.8%  5.159e+09 ±  2%  perf-stat.i.instructions
      0.53            -3.9%       0.51 ±  2%  perf-stat.i.ipc
      1.86            +3.7%       1.93 ±  2%  perf-stat.overall.cpi
     71.15            -0.8       70.34        perf-stat.overall.iTLB-load-miss-rate%
      0.54            -3.5%       0.52 ±  2%  perf-stat.overall.ipc
   5789520            +3.8%    6012086        perf-stat.overall.path-length
  1.17e+09            -3.6%  1.128e+09 ±  2%  perf-stat.ps.branch-instructions
    374455            -4.2%     358648 ±  2%  perf-stat.ps.context-switches
 1.461e+09            -3.6%  1.408e+09 ±  2%  perf-stat.ps.dTLB-loads
 8.085e+08            -4.2%  7.747e+08 ±  2%  perf-stat.ps.dTLB-stores
 5.343e+09            -3.8%  5.142e+09 ±  2%  perf-stat.ps.instructions
    100352 ±  2%     +13.0%     113434 ±  6%  softirqs.CPU10.TIMER
    101777 ±  3%     +11.5%     113486 ±  5%  softirqs.CPU11.TIMER
    100055 ±  3%     +10.0%     110075 ±  5%  softirqs.CPU13.TIMER
    101021 ±  3%     +32.8%     134115 ± 16%  softirqs.CPU15.TIMER
     99491 ±  2%     +13.5%     112882 ±  9%  softirqs.CPU16.TIMER
    100003 ±  2%      +9.9%     109934 ±  5%  softirqs.CPU17.TIMER
     37527 ±  2%     +15.4%      43321 ± 12%  softirqs.CPU19.SCHED
    103667 ±  4%     +11.6%     115707 ±  4%  softirqs.CPU2.TIMER
    101609 ±  4%      +9.0%     110781 ±  7%  softirqs.CPU23.TIMER
     97414 ±  2%     +12.8%     109926 ±  7%  softirqs.CPU25.TIMER
     99800 ±  5%     +10.8%     110574 ±  8%  softirqs.CPU26.TIMER
     99683 ±  5%     +13.5%     113100 ±  6%  softirqs.CPU27.TIMER
     97869 ±  2%     +10.8%     108412 ±  5%  softirqs.CPU30.TIMER
     98332 ±  3%     +14.8%     112847 ±  7%  softirqs.CPU33.TIMER
     37531 ±  2%     +12.7%      42283 ± 10%  softirqs.CPU34.SCHED
     99845 ±  6%     +11.6%     111447 ±  6%  softirqs.CPU34.TIMER
    100067 ±  6%     +12.8%     112851 ±  6%  softirqs.CPU35.TIMER
     98909 ±  4%     +11.5%     110324 ±  6%  softirqs.CPU36.TIMER
     99760 ±  4%     +11.1%     110797 ±  6%  softirqs.CPU37.TIMER
     99036 ±  2%     +12.8%     111684 ±  6%  softirqs.CPU38.TIMER
    100075 ±  2%     +11.4%     111475 ±  4%  softirqs.CPU4.TIMER
    100876 ±  6%     +12.9%     113938 ±  7%  softirqs.CPU41.TIMER
     98943 ±  5%     +12.1%     110934 ±  6%  softirqs.CPU42.TIMER
     37554 ±  2%      +9.8%      41249 ±  4%  softirqs.CPU43.SCHED
    100322 ±  5%     +13.3%     113660 ±  8%  softirqs.CPU43.TIMER
     39193 ±  6%     +11.8%      43810 ± 11%  softirqs.CPU44.SCHED
     37486 ±  2%     +12.0%      41992 ±  7%  softirqs.CPU45.SCHED
     99330 ±  5%     +14.4%     113605 ±  5%  softirqs.CPU45.TIMER
    100692 ±  5%     +12.0%     112743 ±  5%  softirqs.CPU46.TIMER
     37703 ±  3%      +6.7%      40218 ±  3%  softirqs.CPU68.SCHED
     98362 ±  2%     +10.5%     108699 ±  6%  softirqs.CPU68.TIMER
     36569 ±  4%     +10.3%      40344 ±  6%  softirqs.CPU74.SCHED
     94622 ±  2%     +11.3%     105361 ±  5%  softirqs.CPU76.TIMER
     35949 ±  3%      +9.7%      39444 ±  3%  softirqs.CPU78.SCHED
     96464           +10.2%     106340 ±  4%  softirqs.CPU80.TIMER
     95793 ±  4%      +9.2%     104568 ±  3%  softirqs.CPU88.TIMER
     96624 ±  2%      +7.8%     104154 ±  5%  softirqs.CPU89.TIMER
    101648 ±  3%     +10.5%     112304 ±  6%  softirqs.CPU9.TIMER
     36172 ±  3%      +8.0%      39083 ±  6%  softirqs.CPU91.SCHED
     97133 ±  4%     +11.4%     108210 ±  7%  softirqs.CPU92.TIMER
     35978 ±  3%     +11.3%      40060 ±  4%  softirqs.CPU93.SCHED
     97301 ±  3%     +20.1%     116847 ± 19%  softirqs.CPU93.TIMER
     95831 ±  2%      +9.4%     104867 ±  4%  softirqs.CPU95.TIMER
     66.97 ±  2%      -2.0       65.01        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     67.56 ±  2%      -1.9       65.61        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     77.08            -1.9       75.15        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     77.07            -1.9       75.15        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     77.02            -1.9       75.10        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     78.27            -1.8       76.45        perf-profile.calltrace.cycles-pp.secondary_startup_64
     59.20            -1.7       57.45        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      1.64            -0.2        1.40 ±  4%  perf-profile.calltrace.cycles-pp.schedule.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
      1.57 ±  2%      -0.2        1.34 ±  4%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.rwsem_down_write_slowpath.down_write_killable.__vm_munmap
      1.18 ±  4%      -0.1        1.06 ±  5%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.56 ±  4%      +0.0        0.60 ±  4%  perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
      0.82 ±  2%      +0.1        0.91 ±  6%  perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      0.80 ±  6%      +0.1        0.90 ±  7%  perf-profile.calltrace.cycles-pp.find_vma.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
      1.12 ±  3%      +0.1        1.24 ±  2%  perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
      1.39            +0.1        1.53 ±  2%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
      0.86 ±  9%      +0.2        1.01 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
      1.52            +0.2        1.68 ±  2%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle
      1.54            +0.2        1.70 ±  3%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
      1.54            +0.2        1.71 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
      0.41 ± 58%      +0.2        0.62        perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff
      0.66 ±  8%      +0.2        0.89 ±  5%  perf-profile.calltrace.cycles-pp.unmapped_area_topdown.arch_get_unmapped_area_topdown.shmem_get_unmapped_area.get_unmapped_area.do_mmap
      2.28 ±  3%      +0.2        2.51 ±  2%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.77 ±  9%      +0.3        1.03 ±  3%  perf-profile.calltrace.cycles-pp.arch_get_unmapped_area_topdown.shmem_get_unmapped_area.get_unmapped_area.do_mmap.vm_mmap_pgoff
      0.88 ±  9%      +0.3        1.13 ±  2%  perf-profile.calltrace.cycles-pp.shmem_get_unmapped_area.get_unmapped_area.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
      1.13 ±  4%      +0.3        1.39 ±  2%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff
      1.17 ±  5%      +0.3        1.44 ±  2%  perf-profile.calltrace.cycles-pp.schedule.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff
      0.98 ±  7%      +0.3        1.27 ±  2%  perf-profile.calltrace.cycles-pp.get_unmapped_area.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      3.32 ±  4%      +0.3        3.65 ±  2%  perf-profile.calltrace.cycles-pp.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.13 ±173%      +0.4        0.52 ±  2%  perf-profile.calltrace.cycles-pp.vma_link.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
      3.63 ±  4%      +0.4        4.08        perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.6        0.55 ±  3%  perf-profile.calltrace.cycles-pp.unmap_vmas.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
      2.87 ±  4%      +0.7        3.58 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      3.10 ±  4%      +0.7        3.83 ±  2%  perf-profile.calltrace.cycles-pp.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.07 ±  4%      +1.3       11.35 ±  2%  perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.22 ±  4%      +1.3       11.51 ±  2%  perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
     20.46 ±  4%      +1.8       22.31 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     20.40 ±  4%      +1.8       22.24 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     77.08            -1.9       75.15        perf-profile.children.cycles-pp.start_secondary
     68.72            -1.8       66.88        perf-profile.children.cycles-pp.cpuidle_enter
     68.69            -1.8       66.86        perf-profile.children.cycles-pp.cpuidle_enter_state
     78.27            -1.8       76.45        perf-profile.children.cycles-pp.secondary_startup_64
     78.27            -1.8       76.45        perf-profile.children.cycles-pp.cpu_startup_entry
     78.29            -1.8       76.48        perf-profile.children.cycles-pp.do_idle
     59.20            -1.7       57.46        perf-profile.children.cycles-pp.intel_idle
      0.64 ± 10%      -0.2        0.47 ± 28%  perf-profile.children.cycles-pp.tick_nohz_irq_exit
      1.18 ±  4%      -0.1        1.06 ±  5%  perf-profile.children.cycles-pp.poll_idle
      0.47 ± 10%      -0.1        0.39 ± 13%  perf-profile.children.cycles-pp.reschedule_interrupt
      0.41 ± 11%      -0.1        0.33 ± 13%  perf-profile.children.cycles-pp.scheduler_ipi
      0.12 ±  6%      -0.0        0.10 ±  8%  perf-profile.children.cycles-pp.run_timer_softirq
      0.26            -0.0        0.23 ±  6%  perf-profile.children.cycles-pp.rcu_eqs_enter
      0.36            -0.0        0.34 ±  4%  perf-profile.children.cycles-pp.irq_enter
      0.09 ±  5%      -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.kfree
      0.07 ±  7%      -0.0        0.05        perf-profile.children.cycles-pp.tlb_finish_mmu
      0.07 ± 16%      +0.0        0.09 ±  5%  perf-profile.children.cycles-pp.rcu_all_qs
      0.06 ± 14%      +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.15 ±  7%      +0.0        0.17 ±  4%  perf-profile.children.cycles-pp.fput_many
      0.11 ±  6%      +0.0        0.13 ± 12%  perf-profile.children.cycles-pp.vma_merge
      0.39 ±  3%      +0.0        0.41 ±  2%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.61 ±  2%      +0.0        0.64        perf-profile.children.cycles-pp.native_write_msr
      0.32 ±  2%      +0.0        0.35 ±  3%  perf-profile.children.cycles-pp.vma_compute_subtree_gap
      0.19 ±  7%      +0.0        0.22 ±  4%  perf-profile.children.cycles-pp.__vm_enough_memory
      0.48 ±  5%      +0.0        0.52        perf-profile.children.cycles-pp.vma_link
      0.36 ±  4%      +0.0        0.40 ±  4%  perf-profile.children.cycles-pp.find_next_bit
      0.13 ± 14%      +0.0        0.18 ±  2%  perf-profile.children.cycles-pp._cond_resched
      0.18 ± 12%      +0.0        0.23 ±  2%  perf-profile.children.cycles-pp.___might_sleep
      1.02 ±  3%      +0.1        1.12 ±  3%  perf-profile.children.cycles-pp.rwsem_mark_wake
      0.80 ±  6%      +0.1        0.91 ±  7%  perf-profile.children.cycles-pp.find_vma
      1.13 ±  3%      +0.1        1.25 ±  2%  perf-profile.children.cycles-pp.unmap_region
      0.67 ±  9%      +0.1        0.78        perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.44 ±  4%      +0.1        0.57 ±  2%  perf-profile.children.cycles-pp.unmap_vmas
      0.38 ±  6%      +0.1        0.51 ±  4%  perf-profile.children.cycles-pp.unmap_page_range
      0.20 ± 35%      +0.2        0.36 ±  6%  perf-profile.children.cycles-pp.run_local_timers
      0.66 ±  8%      +0.2        0.89 ±  5%  perf-profile.children.cycles-pp.unmapped_area_topdown
      1.39 ±  8%      +0.2        1.64        perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.78 ±  8%      +0.3        1.04 ±  3%  perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
      0.88 ±  9%      +0.3        1.13 ±  2%  perf-profile.children.cycles-pp.shmem_get_unmapped_area
      0.99 ±  7%      +0.3        1.27 ±  2%  perf-profile.children.cycles-pp.get_unmapped_area
      3.34 ±  4%      +0.3        3.67 ±  2%  perf-profile.children.cycles-pp.__do_munmap
      3.64 ±  4%      +0.5        4.09        perf-profile.children.cycles-pp.do_mmap
      6.32 ±  4%      +0.8        7.07 ±  2%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      6.81 ±  4%      +0.8        7.62 ±  2%  perf-profile.children.cycles-pp.down_write_killable
     10.07 ±  4%      +1.3       11.35 ±  2%  perf-profile.children.cycles-pp.vm_mmap_pgoff
     10.22 ±  4%      +1.3       11.51 ±  2%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
     20.46 ±  4%      +1.8       22.31 ±  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     20.41 ±  4%      +1.8       22.26 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
     59.19            -1.7       57.44        perf-profile.self.cycles-pp.intel_idle
      1.14 ±  3%      -0.1        1.03 ±  5%  perf-profile.self.cycles-pp.poll_idle
      0.15 ±  5%      -0.0        0.12 ±  8%  perf-profile.self.cycles-pp.call_cpuidle
      0.06 ± 14%      +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.11 ±  6%      +0.0        0.13 ± 12%  perf-profile.self.cycles-pp.vma_merge
      0.04 ± 57%      +0.0        0.06 ±  6%  perf-profile.self.cycles-pp.rcu_all_qs
      0.61 ±  2%      +0.0        0.64        perf-profile.self.cycles-pp.native_write_msr
      0.30 ±  2%      +0.0        0.34 ±  2%  perf-profile.self.cycles-pp.vma_compute_subtree_gap
      0.28 ±  5%      +0.0        0.32 ±  3%  perf-profile.self.cycles-pp.__do_munmap
      0.17 ± 13%      +0.0        0.21 ±  3%  perf-profile.self.cycles-pp.___might_sleep
      0.38 ±  6%      +0.0        0.42        perf-profile.self.cycles-pp.mmap_region
      0.12 ±  9%      +0.0        0.16 ± 22%  perf-profile.self.cycles-pp.__account_scheduler_latency
      0.44 ±  6%      +0.0        0.48 ±  5%  perf-profile.self.cycles-pp.down_write_killable
      0.23 ±  9%      +0.0        0.28 ±  4%  perf-profile.self.cycles-pp.rwsem_mark_wake
      0.22 ±  6%      +0.1        0.29 ±  5%  perf-profile.self.cycles-pp.unmap_page_range
      0.65 ±  7%      +0.1        0.74 ±  8%  perf-profile.self.cycles-pp.find_vma
      0.37 ±  9%      +0.1        0.47 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.74 ±  6%      +0.1        0.89 ±  4%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.20 ± 35%      +0.2        0.35 ±  8%  perf-profile.self.cycles-pp.run_local_timers
      0.66 ±  8%      +0.2        0.88 ±  5%  perf-profile.self.cycles-pp.unmapped_area_topdown
      1.39 ±  8%      +0.2        1.63        perf-profile.self.cycles-pp.rwsem_spin_on_owner


                                                                                
                           will-it-scale.per_thread_ops                         
                                                                                
  7000 +-+------------------------------------------------------------------+   
       |                                                                    |   
  6000 +-++.+..+.+..+.+..+.+..+..+.+..+.+..+.+..+.+..+..+.+..+.+..          |   
       |  :                                                  O O  O.O  O O  O   
  5000 +-+:                                                                 |   
       O :  O    O  O O  O O  O  O O  O O  O    O    O  O                   |   
  4000 +-+O    O                             O    O       O                 |   
       | :                                                                  |   
  3000 +-+                                                                  |   
       |:                                                                   |   
  2000 +-+                                                                  |   
       |:                                                                   |   
  1000 +-+                                                                  |   
       |                                                                    |   
     0 +-+------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-hsw-4ex1: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory
=========================================================================================
compiler/cpufreq_governor/iterations/kconfig/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/30/x86_64-rhel-7.6/1600%/debian-x86_64-2019-05-14.cgz/lkp-hsw-4ex1/new_dbase/reaim/0x14

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :2     dmesg.WARNING:at_ip_do_semtimedop/0x
           :4           50%           2:2     dmesg.WARNING:at_ip_perf_event_mmap_output/0x
          1:4          -14%           1:2     perf-profile.children.cycles-pp.error_entry
          1:4          -11%           0:2     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
   1046913            -1.5%    1031046        reaim.max_jobs_per_min
    193655 ± 19%     -18.9%     157005 ±  2%  softirqs.CPU43.TIMER
      5.00 ± 61%    +220.0%      16.00 ± 18%  vmstat.procs.b
      1.29 ± 70%      +1.2        2.51 ±  4%  turbostat.C1E%
     66.48            +1.2%      67.31        turbostat.RAMWatt
     37.58 ±  2%      +2.6%      38.57        boot-time.boot
     23.94            +0.7%      24.11        boot-time.dhcp
      4655 ±  3%      +3.2%       4801        boot-time.idle
   9479168 ± 12%     +27.5%   12090368 ±  8%  meminfo.DirectMap2M
      1601 ± 28%     -50.9%     787.00        meminfo.Mlocked
    114587 ±  7%     +13.5%     130108        meminfo.Shmem
      4681 ± 11%     +17.6%       5507 ±  7%  slabinfo.kmalloc-rcl-64.active_objs
      4681 ± 11%     +17.6%       5507 ±  7%  slabinfo.kmalloc-rcl-64.num_objs
      1076 ±  7%     -15.7%     908.00        slabinfo.xfrm_state.active_objs
      1076 ±  7%     -15.7%     908.00        slabinfo.xfrm_state.num_objs
    294089 ±  8%      +7.1%     315115 ±  8%  numa-meminfo.node1.FilePages
      1741 ± 55%     +80.2%       3139 ±  3%  numa-meminfo.node1.Inactive
      1559 ± 61%     +88.6%       2939 ±  3%  numa-meminfo.node1.Inactive(anon)
     20963 ± 16%     +65.1%      34612 ± 29%  numa-meminfo.node1.KReclaimable
     20963 ± 16%     +65.1%      34612 ± 29%  numa-meminfo.node1.SReclaimable
     21840 ±132%    +132.2%      50711 ± 86%  numa-meminfo.node1.Shmem
    446783 ± 17%     -24.6%     337072 ±  3%  numa-meminfo.node3.AnonPages
    110.50 ± 32%     -50.2%      55.00 ± 29%  numa-vmstat.node0.nr_mlock
     73537 ±  8%      +7.0%      78692 ±  8%  numa-vmstat.node1.nr_file_pages
    388.50 ± 61%     +89.6%     736.50 ±  3%  numa-vmstat.node1.nr_inactive_anon
      1668 ±  3%      +9.5%       1827 ±  7%  numa-vmstat.node1.nr_mapped
    102.75 ± 44%     -61.1%      40.00 ±  2%  numa-vmstat.node1.nr_mlock
      5475 ±132%    +130.0%      12591 ± 86%  numa-vmstat.node1.nr_shmem
      5242 ± 16%     +65.1%       8653 ± 29%  numa-vmstat.node1.nr_slab_reclaimable
    388.50 ± 61%     +89.6%     736.50 ±  3%  numa-vmstat.node1.nr_zone_inactive_anon
     83.50 ± 28%     -52.7%      39.50        numa-vmstat.node2.nr_mlock
    112398 ± 16%     -24.1%      85258 ±  4%  numa-vmstat.node3.nr_anon_pages
    284891            +1.3%     288657        proc-vmstat.nr_file_pages
      5637            +2.4%       5770        proc-vmstat.nr_inactive_anon
    401.25 ± 28%     -52.0%     192.50        proc-vmstat.nr_mlock
     28688 ±  7%     +13.3%      32516 ±  2%  proc-vmstat.nr_shmem
      5637            +2.4%       5770        proc-vmstat.nr_zone_inactive_anon
      7134 ± 61%     +78.7%      12750 ±  6%  proc-vmstat.numa_hint_faults
 1.194e+09            +0.7%  1.203e+09        proc-vmstat.numa_hit
 1.194e+09            +0.7%  1.203e+09        proc-vmstat.numa_local
     54546 ±  2%      +7.2%      58468 ±  3%  proc-vmstat.pgactivate
 1.204e+09            +0.7%  1.212e+09        proc-vmstat.pgalloc_normal
 1.203e+09            +0.7%  1.212e+09        proc-vmstat.pgfree
    571.58 ±  5%     +15.9%     662.44 ±  2%  sched_debug.cfs_rq:/.load_avg.max
      0.42 ± 29%    +100.0%       0.83 ±  6%  sched_debug.cfs_rq:/.load_avg.min
      0.52 ± 15%     +24.0%       0.65 ±  4%  sched_debug.cfs_rq:/.nr_running.avg
    628.88 ± 13%     +19.7%     752.45 ±  4%  sched_debug.cfs_rq:/.util_avg.avg
      1383 ±  9%     +11.5%       1542 ±  6%  sched_debug.cfs_rq:/.util_avg.max
    248.75 ± 15%     +33.8%     332.72 ±  8%  sched_debug.cfs_rq:/.util_avg.min
    119.07 ± 27%     +35.0%     160.71 ± 10%  sched_debug.cfs_rq:/.util_est_enqueued.avg
      0.06 ±173%    +500.0%       0.33 ± 33%  sched_debug.cfs_rq:/.util_est_enqueued.min
    146725 ± 15%     +34.6%     197510 ±  3%  sched_debug.cpu.avg_idle.min
     85.85 ± 10%     +25.7%     107.90 ± 10%  sched_debug.cpu.clock.stddev
     85.85 ± 10%     +25.7%     107.90 ± 10%  sched_debug.cpu.clock_task.stddev
     25195 ± 12%     +18.2%      29793 ±  5%  sched_debug.cpu.curr->pid.stddev
      0.00 ±  9%     +22.5%       0.00 ±  9%  sched_debug.cpu.next_balance.stddev
     10.13 ±  2%     +13.9%      11.54 ±  5%  sched_debug.cpu.nr_uninterruptible.avg
  15162686 ± 75%    +707.2%  1.224e+08 ± 79%  syscalls.sys_brk.max
 1.345e+08 ± 10%  +2.6e+07   1.602e+08 ±  3%  syscalls.sys_brk.noise.100%
 1.471e+08 ± 12%  +2.7e+07   1.741e+08 ±  2%  syscalls.sys_brk.noise.2%
 1.437e+08 ± 11%  +2.7e+07   1.706e+08 ±  2%  syscalls.sys_brk.noise.25%
 1.467e+08 ± 12%  +2.7e+07   1.736e+08 ±  2%  syscalls.sys_brk.noise.5%
 1.397e+08 ± 11%  +2.7e+07   1.663e+08 ±  2%  syscalls.sys_brk.noise.50%
 1.367e+08 ± 11%  +2.6e+07   1.626e+08 ±  2%  syscalls.sys_brk.noise.75%
 6.435e+08 ± 78%    +108.2%  1.339e+09        syscalls.sys_mmap.max
    320983 ±138%   +5348.2%   17487893 ± 13%  syscalls.sys_newstat.max
      3342 ±101%    +130.9%       7718        syscalls.sys_newstat.med
 1.073e+08 ±100%  +1.2e+08   2.271e+08 ±  3%  syscalls.sys_newstat.noise.2%
  1.01e+08 ±100%  +1.1e+08   2.106e+08 ±  3%  syscalls.sys_newstat.noise.25%
 1.071e+08 ±100%  +1.2e+08   2.263e+08 ±  3%  syscalls.sys_newstat.noise.5%
  87529237 ±100%  +9.3e+07   1.802e+08 ±  3%  syscalls.sys_newstat.noise.50%
     23.08 ±  4%     -12.8%      20.13 ±  3%  perf-stat.i.MPKI
      1.38            +0.1        1.45        perf-stat.i.cache-miss-rate%
     84942 ±  2%      +3.4%      87812        perf-stat.i.context-switches
      1.89 ±  7%     -17.3%       1.56        perf-stat.i.cpi
      8139 ± 11%     -24.0%       6183 ±  2%  perf-stat.i.cycles-between-cache-misses
      0.28 ± 12%      -0.1        0.21 ±  6%  perf-stat.i.dTLB-load-miss-rate%
      0.14            -0.0        0.13 ±  2%  perf-stat.i.dTLB-store-miss-rate%
     58.16 ±  2%      -2.2       55.97        perf-stat.i.iTLB-load-miss-rate%
  10359697 ±  3%      +6.6%   11042136 ±  3%  perf-stat.i.iTLB-loads
      0.79            +3.3%       0.82        perf-stat.i.ipc
   6896869 ±  2%      +2.9%    7098370        perf-stat.i.node-loads
   8004284 ±  3%      +4.2%    8341263        perf-stat.i.node-store-misses
   7417146 ±  5%      +8.7%    8061221 ±  3%  perf-stat.i.node-stores
      1.42            +0.0        1.44        perf-stat.overall.cache-miss-rate%
      4567            -1.6%       4493        perf-stat.overall.cycles-between-cache-misses
  10320774 ±  3%      +6.5%   10994114 ±  3%  perf-stat.ps.iTLB-loads
   6871230 ±  2%      +2.9%    7067176        perf-stat.ps.node-loads
   7969678 ±  3%      +4.1%    8297859        perf-stat.ps.node-store-misses
   7413339 ±  5%      +8.7%    8057926 ±  3%  perf-stat.ps.node-stores
      1296 ±106%     -67.7%     418.50 ±  2%  interrupts.45:PCI-MSI.1572864-edge.eth0-TxRx-0
    440.50 ± 59%    +324.0%       1867 ± 36%  interrupts.48:PCI-MSI.1572867-edge.eth0-TxRx-3
    266.50 ±  5%     +51.4%     403.50 ± 28%  interrupts.54:PCI-MSI.1572873-edge.eth0-TxRx-9
    279.25 ±  8%     +62.4%     453.50 ± 26%  interrupts.60:PCI-MSI.1572879-edge.eth0-TxRx-15
    279.25 ±  6%     -11.2%     248.00        interrupts.63:PCI-MSI.1572882-edge.eth0-TxRx-18
    929442 ±  2%      +3.4%     960727        interrupts.CAL:Function_call_interrupts
      1296 ±106%     -67.7%     418.50 ±  2%  interrupts.CPU0.45:PCI-MSI.1572864-edge.eth0-TxRx-0
      1788 ±  3%     +13.5%       2030 ±  2%  interrupts.CPU0.TLB:TLB_shootdowns
      1773 ±  2%     +21.2%       2149 ±  2%  interrupts.CPU1.TLB:TLB_shootdowns
      6356 ±  5%      +7.1%       6808 ±  3%  interrupts.CPU10.CAL:Function_call_interrupts
    677.00 ±172%    +408.1%       3440        interrupts.CPU10.NMI:Non-maskable_interrupts
    677.00 ±172%    +408.1%       3440        interrupts.CPU10.PMI:Performance_monitoring_interrupts
      1766           +22.5%       2162        interrupts.CPU10.TLB:TLB_shootdowns
      1834 ±  4%     +15.6%       2120 ±  3%  interrupts.CPU100.TLB:TLB_shootdowns
      1539 ± 76%    +235.8%       5168 ± 30%  interrupts.CPU101.NMI:Non-maskable_interrupts
      1539 ± 76%    +235.8%       5168 ± 30%  interrupts.CPU101.PMI:Performance_monitoring_interrupts
      1810 ±  3%     +15.4%       2088 ±  3%  interrupts.CPU101.TLB:TLB_shootdowns
      1842 ±  8%     +11.8%       2060 ±  6%  interrupts.CPU103.TLB:TLB_shootdowns
      1796 ±  4%     +13.0%       2029 ±  3%  interrupts.CPU104.TLB:TLB_shootdowns
      1835 ±  4%     +16.2%       2132        interrupts.CPU105.TLB:TLB_shootdowns
      1848 ±  5%     +14.7%       2120 ±  2%  interrupts.CPU106.TLB:TLB_shootdowns
      1834 ±  2%     +15.0%       2110        interrupts.CPU107.TLB:TLB_shootdowns
      1825 ±  3%     +14.5%       2090        interrupts.CPU108.TLB:TLB_shootdowns
      1844 ±  4%     +14.0%       2103 ±  2%  interrupts.CPU109.TLB:TLB_shootdowns
      6380 ±  6%      +8.9%       6950 ±  2%  interrupts.CPU11.CAL:Function_call_interrupts
    694.50 ±166%    +633.8%       5096 ± 32%  interrupts.CPU11.NMI:Non-maskable_interrupts
    694.50 ±166%    +633.8%       5096 ± 32%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
      1828 ±  4%     +15.1%       2105 ±  4%  interrupts.CPU11.TLB:TLB_shootdowns
      1819 ±  5%     +18.0%       2147 ±  6%  interrupts.CPU110.TLB:TLB_shootdowns
      3634 ± 58%     -99.0%      36.00 ± 97%  interrupts.CPU111.NMI:Non-maskable_interrupts
      3634 ± 58%     -99.0%      36.00 ± 97%  interrupts.CPU111.PMI:Performance_monitoring_interrupts
      1876 ±  8%     +13.8%       2135 ±  5%  interrupts.CPU111.TLB:TLB_shootdowns
      5937 ± 12%    -100.0%       0.50 ±100%  interrupts.CPU112.NMI:Non-maskable_interrupts
      5937 ± 12%    -100.0%       0.50 ±100%  interrupts.CPU112.PMI:Performance_monitoring_interrupts
      1835 ±  6%     +13.7%       2087 ±  4%  interrupts.CPU112.TLB:TLB_shootdowns
      5033 ± 18%     -33.0%       3374 ±  2%  interrupts.CPU113.NMI:Non-maskable_interrupts
      5033 ± 18%     -33.0%       3374 ±  2%  interrupts.CPU113.PMI:Performance_monitoring_interrupts
      1798 ±  4%     +17.8%       2118 ±  4%  interrupts.CPU113.TLB:TLB_shootdowns
      1818 ±  3%     +19.9%       2180 ±  3%  interrupts.CPU114.TLB:TLB_shootdowns
      1787 ±  3%     +17.1%       2093 ±  4%  interrupts.CPU115.TLB:TLB_shootdowns
      5083 ± 28%     +35.1%       6868 ±  3%  interrupts.CPU116.NMI:Non-maskable_interrupts
      5083 ± 28%     +35.1%       6868 ±  3%  interrupts.CPU116.PMI:Performance_monitoring_interrupts
      2939 ± 60%    +134.2%       6885 ±  2%  interrupts.CPU117.NMI:Non-maskable_interrupts
      2939 ± 60%    +134.2%       6885 ±  2%  interrupts.CPU117.PMI:Performance_monitoring_interrupts
      1777 ±  2%     +18.4%       2104        interrupts.CPU117.TLB:TLB_shootdowns
      1799 ±  5%     +22.8%       2209 ±  3%  interrupts.CPU118.TLB:TLB_shootdowns
      1861 ±  3%     +17.8%       2193        interrupts.CPU119.TLB:TLB_shootdowns
      1895 ±  7%     +11.9%       2120 ±  2%  interrupts.CPU12.TLB:TLB_shootdowns
      1846 ±  3%     +19.5%       2206 ±  4%  interrupts.CPU120.TLB:TLB_shootdowns
      1806 ±  5%     +18.4%       2139        interrupts.CPU121.TLB:TLB_shootdowns
      1856 ±  6%     +15.7%       2148 ±  2%  interrupts.CPU122.TLB:TLB_shootdowns
      1768 ±  4%     +21.7%       2152 ±  2%  interrupts.CPU123.TLB:TLB_shootdowns
      1821 ±  3%     +13.5%       2067 ±  4%  interrupts.CPU124.TLB:TLB_shootdowns
    733.75 ±168%    +354.2%       3333 ± 97%  interrupts.CPU125.NMI:Non-maskable_interrupts
    733.75 ±168%    +354.2%       3333 ± 97%  interrupts.CPU125.PMI:Performance_monitoring_interrupts
      1828 ±  3%     +14.4%       2091        interrupts.CPU125.TLB:TLB_shootdowns
      1806 ±  4%     +12.1%       2024 ±  4%  interrupts.CPU126.TLB:TLB_shootdowns
      1359 ±172%    +279.4%       5156 ± 30%  interrupts.CPU127.NMI:Non-maskable_interrupts
      1359 ±172%    +279.4%       5156 ± 30%  interrupts.CPU127.PMI:Performance_monitoring_interrupts
      1834 ±  5%     +15.9%       2125 ±  3%  interrupts.CPU127.TLB:TLB_shootdowns
      1810 ±  3%     +14.8%       2078        interrupts.CPU128.TLB:TLB_shootdowns
      1841 ±  5%     +14.3%       2105        interrupts.CPU129.TLB:TLB_shootdowns
      5913 ± 12%     -70.4%       1748 ± 99%  interrupts.CPU13.NMI:Non-maskable_interrupts
      5913 ± 12%     -70.4%       1748 ± 99%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
      1778 ±  4%     +18.9%       2114        interrupts.CPU13.TLB:TLB_shootdowns
      1861 ±  6%     +11.7%       2080 ±  6%  interrupts.CPU130.TLB:TLB_shootdowns
      1768           +25.3%       2216 ±  2%  interrupts.CPU131.TLB:TLB_shootdowns
      1810 ±  3%     +16.9%       2115 ±  4%  interrupts.CPU132.TLB:TLB_shootdowns
      2343 ± 87%    +119.8%       5152 ± 30%  interrupts.CPU133.NMI:Non-maskable_interrupts
      2343 ± 87%    +119.8%       5152 ± 30%  interrupts.CPU133.PMI:Performance_monitoring_interrupts
      1842 ±  5%     +14.8%       2115 ±  3%  interrupts.CPU133.TLB:TLB_shootdowns
      1866 ±  6%     +12.0%       2090 ±  3%  interrupts.CPU134.TLB:TLB_shootdowns
      1829 ±  3%     +15.5%       2113 ±  2%  interrupts.CPU135.TLB:TLB_shootdowns
      1898 ±  5%     +13.8%       2159 ±  3%  interrupts.CPU136.TLB:TLB_shootdowns
      1778 ±  3%     +24.8%       2219 ±  6%  interrupts.CPU137.TLB:TLB_shootdowns
      1887 ±  5%     +13.3%       2138 ±  3%  interrupts.CPU138.TLB:TLB_shootdowns
      9349 ±  2%     +10.8%      10359        interrupts.CPU139.RES:Rescheduling_interrupts
      1824 ±  3%     +21.0%       2206 ±  2%  interrupts.CPU139.TLB:TLB_shootdowns
      1815 ±  3%     +19.0%       2161 ±  3%  interrupts.CPU14.TLB:TLB_shootdowns
      1815 ±  6%     +18.1%       2143 ±  4%  interrupts.CPU140.TLB:TLB_shootdowns
      1815 ±  3%     +16.3%       2110 ±  3%  interrupts.CPU141.TLB:TLB_shootdowns
      1843 ±  5%     +16.4%       2145        interrupts.CPU142.TLB:TLB_shootdowns
      9028            +8.2%       9767        interrupts.CPU143.RES:Rescheduling_interrupts
      1782 ±  3%     +20.4%       2147 ±  2%  interrupts.CPU143.TLB:TLB_shootdowns
    279.25 ±  8%     +62.4%     453.50 ± 26%  interrupts.CPU15.60:PCI-MSI.1572879-edge.eth0-TxRx-15
      6215 ±  2%     +12.7%       7006        interrupts.CPU15.CAL:Function_call_interrupts
      1608 ±141%    +328.5%       6893        interrupts.CPU15.NMI:Non-maskable_interrupts
      1608 ±141%    +328.5%       6893        interrupts.CPU15.PMI:Performance_monitoring_interrupts
      1885 ±  7%     +12.8%       2128 ±  3%  interrupts.CPU15.TLB:TLB_shootdowns
      1784 ±  2%     +21.3%       2165 ±  2%  interrupts.CPU16.TLB:TLB_shootdowns
      6329 ±  5%      +9.3%       6917 ±  2%  interrupts.CPU17.CAL:Function_call_interrupts
      1808 ±  3%     +17.2%       2120 ±  2%  interrupts.CPU17.TLB:TLB_shootdowns
    279.25 ±  6%     -11.2%     248.00        interrupts.CPU18.63:PCI-MSI.1572882-edge.eth0-TxRx-18
      1783 ±  5%     +19.8%       2136 ±  6%  interrupts.CPU18.TLB:TLB_shootdowns
      6202 ±  3%      +7.9%       6692 ±  3%  interrupts.CPU19.CAL:Function_call_interrupts
      1314 ±173%    +165.9%       3495 ± 99%  interrupts.CPU19.NMI:Non-maskable_interrupts
      1314 ±173%    +165.9%       3495 ± 99%  interrupts.CPU19.PMI:Performance_monitoring_interrupts
      1814 ±  7%     +19.1%       2161 ±  3%  interrupts.CPU19.TLB:TLB_shootdowns
      6427 ±  5%      +5.9%       6806 ±  3%  interrupts.CPU2.CAL:Function_call_interrupts
    697.75 ±162%    +875.9%       6809 ±  2%  interrupts.CPU2.NMI:Non-maskable_interrupts
    697.75 ±162%    +875.9%       6809 ±  2%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      1824 ±  4%     +17.3%       2141 ±  5%  interrupts.CPU2.TLB:TLB_shootdowns
      1843 ±  6%     +16.6%       2149 ±  3%  interrupts.CPU20.TLB:TLB_shootdowns
      3475 ± 65%    +100.0%       6949 ±  2%  interrupts.CPU21.NMI:Non-maskable_interrupts
      3475 ± 65%    +100.0%       6949 ±  2%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
      1813 ±  4%     +18.7%       2153 ±  2%  interrupts.CPU21.TLB:TLB_shootdowns
      1859 ±  3%     +14.3%       2124        interrupts.CPU22.TLB:TLB_shootdowns
      6549 ±  3%      +6.3%       6962 ±  2%  interrupts.CPU23.CAL:Function_call_interrupts
      2528 ± 63%    +170.2%       6833 ±  2%  interrupts.CPU23.NMI:Non-maskable_interrupts
      2528 ± 63%    +170.2%       6833 ±  2%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
      1837 ±  6%     +17.0%       2150 ±  4%  interrupts.CPU23.TLB:TLB_shootdowns
      1808 ±  3%     +15.7%       2093 ±  4%  interrupts.CPU24.TLB:TLB_shootdowns
      1816 ±  4%     +18.0%       2142 ±  2%  interrupts.CPU25.TLB:TLB_shootdowns
      1812 ±  5%     +17.0%       2120        interrupts.CPU26.TLB:TLB_shootdowns
      1769 ±  3%     +19.5%       2115 ±  7%  interrupts.CPU27.TLB:TLB_shootdowns
      1862           +12.5%       2094        interrupts.CPU28.TLB:TLB_shootdowns
      1838 ±  5%     +11.9%       2057 ±  5%  interrupts.CPU29.TLB:TLB_shootdowns
    440.50 ± 59%    +324.0%       1867 ± 36%  interrupts.CPU3.48:PCI-MSI.1572867-edge.eth0-TxRx-3
      6171 ±  2%      +9.6%       6764 ±  5%  interrupts.CPU3.CAL:Function_call_interrupts
      1280 ±173%    +302.0%       5146 ± 32%  interrupts.CPU3.NMI:Non-maskable_interrupts
      1280 ±173%    +302.0%       5146 ± 32%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
      1769 ±  3%     +20.8%       2137 ±  4%  interrupts.CPU3.TLB:TLB_shootdowns
    905.25 ±169%    +471.1%       5169 ± 34%  interrupts.CPU30.NMI:Non-maskable_interrupts
    905.25 ±169%    +471.1%       5169 ± 34%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
      1798 ±  5%     +19.2%       2145 ±  4%  interrupts.CPU31.TLB:TLB_shootdowns
      1821 ±  3%     +19.5%       2176 ±  5%  interrupts.CPU33.TLB:TLB_shootdowns
      6256 ±  3%     +12.4%       7031        interrupts.CPU34.CAL:Function_call_interrupts
    252.00 ±172%   +1926.4%       5106 ± 31%  interrupts.CPU34.NMI:Non-maskable_interrupts
    252.00 ±172%   +1926.4%       5106 ± 31%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
      1877 ±  5%     +14.3%       2145 ±  8%  interrupts.CPU34.TLB:TLB_shootdowns
      1804 ±  6%     +17.5%       2120 ±  7%  interrupts.CPU35.TLB:TLB_shootdowns
      6188 ±  4%     +11.6%       6909        interrupts.CPU36.CAL:Function_call_interrupts
      1347 ±173%    +280.4%       5126 ± 36%  interrupts.CPU36.NMI:Non-maskable_interrupts
      1347 ±173%    +280.4%       5126 ± 36%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
      1751 ±  4%     +19.7%       2096 ±  3%  interrupts.CPU36.TLB:TLB_shootdowns
      1772 ±  4%     +18.1%       2093 ±  5%  interrupts.CPU37.TLB:TLB_shootdowns
      1875 ±  3%     +12.6%       2111        interrupts.CPU38.TLB:TLB_shootdowns
    900.75 ±172%    +675.1%       6981 ±  3%  interrupts.CPU39.NMI:Non-maskable_interrupts
    900.75 ±172%    +675.1%       6981 ±  3%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
      1818 ±  4%     +15.9%       2106 ±  3%  interrupts.CPU39.TLB:TLB_shootdowns
      1827 ±  6%     +17.2%       2142 ±  3%  interrupts.CPU4.TLB:TLB_shootdowns
      6197 ±  2%     +13.4%       7027        interrupts.CPU40.CAL:Function_call_interrupts
      1.25 ± 34%  +5.4e+05%       6793 ±  3%  interrupts.CPU40.NMI:Non-maskable_interrupts
      1.25 ± 34%  +5.4e+05%       6793 ±  3%  interrupts.CPU40.PMI:Performance_monitoring_interrupts
      1811 ±  3%     +22.0%       2210 ±  2%  interrupts.CPU40.TLB:TLB_shootdowns
      1372 ±168%    +406.5%       6951        interrupts.CPU41.NMI:Non-maskable_interrupts
      1372 ±168%    +406.5%       6951        interrupts.CPU41.PMI:Performance_monitoring_interrupts
      1827 ±  6%     +13.9%       2081 ±  4%  interrupts.CPU41.TLB:TLB_shootdowns
      2099 ±108%    +227.5%       6874 ±  2%  interrupts.CPU42.NMI:Non-maskable_interrupts
      2099 ±108%    +227.5%       6874 ±  2%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
      1824 ±  5%     +14.7%       2091 ±  4%  interrupts.CPU42.TLB:TLB_shootdowns
      2912 ± 92%    +139.0%       6960 ±  2%  interrupts.CPU43.NMI:Non-maskable_interrupts
      2912 ± 92%    +139.0%       6960 ±  2%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
      1775 ±  2%     +20.1%       2133 ±  2%  interrupts.CPU43.TLB:TLB_shootdowns
      1815 ±  4%     +13.4%       2058        interrupts.CPU44.TLB:TLB_shootdowns
      1803 ±  2%     +19.9%       2162 ±  4%  interrupts.CPU45.TLB:TLB_shootdowns
    722.00 ±149%    +856.6%       6906 ±  2%  interrupts.CPU46.NMI:Non-maskable_interrupts
    722.00 ±149%    +856.6%       6906 ±  2%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
      1821 ±  2%     +20.6%       2196 ±  2%  interrupts.CPU46.TLB:TLB_shootdowns
      1350 ±173%    +403.6%       6802 ±  3%  interrupts.CPU47.NMI:Non-maskable_interrupts
      1350 ±173%    +403.6%       6802 ±  3%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
      1784 ±  5%     +16.2%       2073 ±  3%  interrupts.CPU47.TLB:TLB_shootdowns
      1.00 ± 70%  +3.4e+05%       3397 ± 93%  interrupts.CPU48.NMI:Non-maskable_interrupts
      1.00 ± 70%  +3.4e+05%       3397 ± 93%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
      1919 ±  8%     +10.4%       2119 ±  4%  interrupts.CPU48.TLB:TLB_shootdowns
      6184            +8.5%       6707 ±  3%  interrupts.CPU49.CAL:Function_call_interrupts
      6.50 ±155%  +1.1e+05%       6876        interrupts.CPU49.NMI:Non-maskable_interrupts
      6.50 ±155%  +1.1e+05%       6876        interrupts.CPU49.PMI:Performance_monitoring_interrupts
      1774           +20.6%       2139 ±  4%  interrupts.CPU49.TLB:TLB_shootdowns
      1809 ±  2%     +15.6%       2091 ±  2%  interrupts.CPU5.TLB:TLB_shootdowns
      6218            +8.9%       6771 ±  4%  interrupts.CPU50.CAL:Function_call_interrupts
      4.25 ± 87%  +1.6e+05%       6759 ±  3%  interrupts.CPU50.NMI:Non-maskable_interrupts
      4.25 ± 87%  +1.6e+05%       6759 ±  3%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
      1802 ±  3%     +19.5%       2154 ±  2%  interrupts.CPU50.TLB:TLB_shootdowns
      1885 ±  2%     +10.6%       2085        interrupts.CPU51.TLB:TLB_shootdowns
      1818 ±  3%     +20.2%       2187        interrupts.CPU52.TLB:TLB_shootdowns
      1839 ±  6%     +14.0%       2096 ±  8%  interrupts.CPU53.TLB:TLB_shootdowns
      6314 ±  4%      +5.6%       6669 ±  3%  interrupts.CPU54.CAL:Function_call_interrupts
      1735 ±  4%     +22.1%       2118 ±  2%  interrupts.CPU54.TLB:TLB_shootdowns
      4565 ± 59%    -100.0%       0.00        interrupts.CPU55.NMI:Non-maskable_interrupts
      4565 ± 59%    -100.0%       0.00        interrupts.CPU55.PMI:Performance_monitoring_interrupts
      1792 ±  4%     +16.9%       2095 ±  3%  interrupts.CPU55.TLB:TLB_shootdowns
     10308 ±  2%      +8.8%      11220 ±  2%  interrupts.CPU56.RES:Rescheduling_interrupts
      1848 ±  6%     +13.7%       2102 ±  2%  interrupts.CPU56.TLB:TLB_shootdowns
      6284 ±  4%      +9.1%       6854        interrupts.CPU57.CAL:Function_call_interrupts
      1791 ±131%    +283.6%       6871        interrupts.CPU57.NMI:Non-maskable_interrupts
      1791 ±131%    +283.6%       6871        interrupts.CPU57.PMI:Performance_monitoring_interrupts
     10680 ±  4%     +11.1%      11866 ±  2%  interrupts.CPU57.RES:Rescheduling_interrupts
      1776           +19.1%       2116 ±  3%  interrupts.CPU57.TLB:TLB_shootdowns
      1849 ±  3%     +11.7%       2066 ±  2%  interrupts.CPU58.TLB:TLB_shootdowns
      1774 ±  6%     +27.3%       2258 ±  4%  interrupts.CPU59.TLB:TLB_shootdowns
      1799 ±  4%     +24.7%       2243 ±  7%  interrupts.CPU6.TLB:TLB_shootdowns
      1839 ±  5%     +14.6%       2107        interrupts.CPU60.TLB:TLB_shootdowns
      1819 ±  3%     +11.4%       2027 ±  2%  interrupts.CPU61.TLB:TLB_shootdowns
      9953 ±  2%     +14.5%      11400 ±  8%  interrupts.CPU62.RES:Rescheduling_interrupts
      1839 ±  2%     +15.4%       2123        interrupts.CPU62.TLB:TLB_shootdowns
      1840 ±  3%     +13.8%       2093 ±  2%  interrupts.CPU63.TLB:TLB_shootdowns
      1829 ±  3%     +20.9%       2212 ±  4%  interrupts.CPU64.TLB:TLB_shootdowns
      1802 ±  2%     +16.6%       2100        interrupts.CPU65.TLB:TLB_shootdowns
      1850 ±  4%     +10.8%       2050 ±  7%  interrupts.CPU66.TLB:TLB_shootdowns
      1816 ±  5%     +13.6%       2064 ±  4%  interrupts.CPU67.TLB:TLB_shootdowns
      1841 ±  4%     +12.9%       2079 ±  2%  interrupts.CPU68.TLB:TLB_shootdowns
      6374 ±  2%      +9.4%       6974        interrupts.CPU7.CAL:Function_call_interrupts
    789.25 ±153%    +531.0%       4980 ± 31%  interrupts.CPU7.NMI:Non-maskable_interrupts
    789.25 ±153%    +531.0%       4980 ± 31%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
      1886 ±  6%     +12.4%       2119 ±  3%  interrupts.CPU7.TLB:TLB_shootdowns
      1841 ±  7%     +13.0%       2080        interrupts.CPU70.TLB:TLB_shootdowns
      1781 ±  3%     +25.0%       2227        interrupts.CPU71.TLB:TLB_shootdowns
      1805 ±  3%     +17.9%       2129 ±  3%  interrupts.CPU72.TLB:TLB_shootdowns
      1844 ±  5%     +16.8%       2153        interrupts.CPU73.TLB:TLB_shootdowns
      1848 ±  5%     +16.2%       2148 ±  2%  interrupts.CPU74.TLB:TLB_shootdowns
      1850 ±  3%     +15.4%       2136 ±  5%  interrupts.CPU75.TLB:TLB_shootdowns
      1872 ±  4%     +18.0%       2210        interrupts.CPU76.TLB:TLB_shootdowns
      3894 ± 61%     +75.1%       6818 ±  2%  interrupts.CPU77.NMI:Non-maskable_interrupts
      3894 ± 61%     +75.1%       6818 ±  2%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
      1778 ±  3%     +18.2%       2102 ±  3%  interrupts.CPU77.TLB:TLB_shootdowns
      4300 ± 24%     +58.6%       6819        interrupts.CPU78.NMI:Non-maskable_interrupts
      4300 ± 24%     +58.6%       6819        interrupts.CPU78.PMI:Performance_monitoring_interrupts
      1799           +18.2%       2127 ±  4%  interrupts.CPU78.TLB:TLB_shootdowns
      4562 ± 41%     -98.5%      69.50 ± 51%  interrupts.CPU79.NMI:Non-maskable_interrupts
      4562 ± 41%     -98.5%      69.50 ± 51%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
      1783 ±  3%     +20.7%       2152        interrupts.CPU79.TLB:TLB_shootdowns
      1843           +14.2%       2106        interrupts.CPU8.TLB:TLB_shootdowns
      1769 ±  2%     +19.6%       2115        interrupts.CPU80.TLB:TLB_shootdowns
      1828 ±  5%     +14.9%       2101        interrupts.CPU81.TLB:TLB_shootdowns
      1844           +15.9%       2137 ±  4%  interrupts.CPU82.TLB:TLB_shootdowns
      1793 ±  5%     +19.4%       2141        interrupts.CPU83.TLB:TLB_shootdowns
     47.25 ±120%    +704.2%     380.00 ± 80%  interrupts.CPU84.NMI:Non-maskable_interrupts
     47.25 ±120%    +704.2%     380.00 ± 80%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
      1834 ±  7%     +19.1%       2184        interrupts.CPU85.TLB:TLB_shootdowns
      1855 ±  3%     +15.1%       2135 ±  3%  interrupts.CPU86.TLB:TLB_shootdowns
      3879 ± 69%    -100.0%       1.50 ± 33%  interrupts.CPU87.NMI:Non-maskable_interrupts
      3879 ± 69%    -100.0%       1.50 ± 33%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
      1852 ±  2%     +15.7%       2143 ±  4%  interrupts.CPU87.TLB:TLB_shootdowns
      1790 ±  5%     +19.1%       2131 ±  4%  interrupts.CPU88.TLB:TLB_shootdowns
      1848 ±  6%     +18.1%       2183 ±  4%  interrupts.CPU89.TLB:TLB_shootdowns
    266.50 ±  5%     +51.4%     403.50 ± 28%  interrupts.CPU9.54:PCI-MSI.1572873-edge.eth0-TxRx-9
      1816 ±  3%     +19.1%       2162 ±  2%  interrupts.CPU9.TLB:TLB_shootdowns
      1842 ±  4%     +15.4%       2125 ±  5%  interrupts.CPU90.TLB:TLB_shootdowns
      1795 ±  4%     +21.1%       2174        interrupts.CPU91.TLB:TLB_shootdowns
      1804 ±  4%     +17.5%       2120        interrupts.CPU92.TLB:TLB_shootdowns
      1791 ±  5%     +23.8%       2218        interrupts.CPU93.TLB:TLB_shootdowns
      1874 ±  6%     +14.3%       2142 ±  3%  interrupts.CPU94.TLB:TLB_shootdowns
      5182 ± 29%    -100.0%       1.00        interrupts.CPU95.NMI:Non-maskable_interrupts
      5182 ± 29%    -100.0%       1.00        interrupts.CPU95.PMI:Performance_monitoring_interrupts
      1841 ±  5%     +16.0%       2135 ±  4%  interrupts.CPU95.TLB:TLB_shootdowns
      1821 ±  4%     +15.8%       2109 ±  3%  interrupts.CPU96.TLB:TLB_shootdowns
      1789 ±  2%     +19.6%       2140        interrupts.CPU97.TLB:TLB_shootdowns
      1789 ±  4%     +22.3%       2188 ±  3%  interrupts.CPU98.TLB:TLB_shootdowns
      1797 ±  5%     +19.4%       2146 ±  3%  interrupts.CPU99.TLB:TLB_shootdowns
    101.25 ± 23%     +46.7%     148.50 ±  7%  interrupts.IWI:IRQ_work_interrupts
    401089 ± 11%     +20.2%     481954 ±  6%  interrupts.NMI:Non-maskable_interrupts
    401089 ± 11%     +20.2%     481954 ±  6%  interrupts.PMI:Performance_monitoring_interrupts
    262999 ±  3%     +16.5%     306361 ±  3%  interrupts.TLB:TLB_shootdowns
      9.50 ±103%      -9.5        0.00        perf-profile.calltrace.cycles-pp.sieve
      0.51 ± 60%      +0.3        0.81        perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit
      0.55 ± 61%      +0.3        0.89        perf-profile.calltrace.cycles-pp.filemap_map_pages.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
      0.32 ±100%      +0.4        0.68 ±  3%  perf-profile.calltrace.cycles-pp.tlb_finish_mmu.exit_mmap.mmput.do_exit.do_group_exit
      0.31 ±100%      +0.4        0.68 ±  3%  perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput.do_exit
      1.00 ± 31%      +0.4        1.38 ±  6%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.unlink_file_vma.free_pgtables.unmap_region
      1.02 ± 31%      +0.4        1.41 ±  7%  perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.unmap_region.__do_munmap.mmap_region
      1.02 ± 31%      +0.4        1.40 ±  7%  perf-profile.calltrace.cycles-pp.down_write.unlink_file_vma.free_pgtables.unmap_region.__do_munmap
      0.37 ±100%      +0.4        0.77 ±  2%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit
      0.14 ±173%      +0.4        0.54 ±  3%  perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.exit_mmap.mmput
      0.32 ±100%      +0.4        0.73 ±  4%  perf-profile.calltrace.cycles-pp.clear_page_erms.prep_new_page.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma
      0.14 ±173%      +0.4        0.55 ±  2%  perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap
      0.36 ±100%      +0.4        0.79        perf-profile.calltrace.cycles-pp.native_flush_tlb_one_user.flush_tlb_func_common.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu
      1.10 ± 31%      +0.4        1.52 ±  7%  perf-profile.calltrace.cycles-pp.free_pgtables.unmap_region.__do_munmap.mmap_region.do_mmap
      0.37 ±100%      +0.4        0.80        perf-profile.calltrace.cycles-pp.flush_tlb_func_common.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region
      0.34 ±100%      +0.4        0.77 ±  3%  perf-profile.calltrace.cycles-pp.prep_new_page.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page
      1.14 ± 31%      +0.4        1.58 ±  6%  perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.mmap_region.do_mmap.vm_mmap_pgoff
      0.61 ± 63%      +0.4        1.06 ±  3%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page.__handle_mm_fault
      0.35 ±100%      +0.4        0.79 ±  3%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.__vma_adjust.__split_vma.__do_munmap
      0.38 ±100%      +0.4        0.83        perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap
      0.35 ±100%      +0.4        0.80 ±  3%  perf-profile.calltrace.cycles-pp.down_write.__vma_adjust.__split_vma.__do_munmap.mmap_region
      0.96 ± 38%      +0.5        1.44        perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__x64_sys_brk
      0.41 ±100%      +0.5        0.90 ±  3%  perf-profile.calltrace.cycles-pp.__split_vma.__do_munmap.mmap_region.do_mmap.vm_mmap_pgoff
      0.40 ±100%      +0.5        0.89 ±  4%  perf-profile.calltrace.cycles-pp.__vma_adjust.__split_vma.__do_munmap.mmap_region.do_mmap
      0.97 ± 39%      +0.5        1.46        perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64
      0.79 ± 64%      +0.5        1.32        perf-profile.calltrace.cycles-pp.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode
      0.76 ± 63%      +0.5        1.31 ±  3%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page.__handle_mm_fault.handle_mm_fault
      0.83 ± 63%      +0.6        1.43        perf-profile.calltrace.cycles-pp.alloc_pages_vma.do_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
      1.26 ± 38%      +0.6        1.90        perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.93 ± 66%      +0.7        1.64        perf-profile.calltrace.cycles-pp.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.77 ± 31%      +0.7        2.50 ±  5%  perf-profile.calltrace.cycles-pp.__do_munmap.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
      1.98 ± 25%      +0.8        2.74        perf-profile.calltrace.cycles-pp.do_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
      1.18 ± 67%      +0.9        2.09        perf-profile.calltrace.cycles-pp.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.50 ± 21%      +1.1        4.61        perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      3.34 ± 34%      +1.5        4.86        perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      2.51 ± 49%      +1.6        4.06        perf-profile.calltrace.cycles-pp.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.87 ± 35%      +1.8        5.63        perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
      4.00 ± 35%      +1.8        5.81        perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
      4.04 ± 35%      +1.8        5.87        perf-profile.calltrace.cycles-pp.page_fault
      5.13 ± 31%      +2.0        7.17        perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      5.16 ± 31%      +2.0        7.21        perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.18 ± 31%      +2.1        7.23        perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.19 ± 31%      +2.1        7.25        perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.34 ± 30%      +2.1        7.41 ±  3%  perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.mmput.do_exit
      5.58 ± 30%      +2.1        7.73 ±  3%  perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit
      6.82 ± 29%      +2.5        9.36 ±  2%  perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      6.81 ± 29%      +2.5        9.35 ±  2%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
      7.25 ± 28%      +2.7        9.93 ±  2%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.25 ± 28%      +2.7        9.93 ±  2%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.25 ± 28%      +2.7        9.93 ±  2%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.01 ± 33%      +3.3       11.32 ±  2%  perf-profile.calltrace.cycles-pp.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.12 ± 33%      +3.4       11.47 ±  2%  perf-profile.calltrace.cycles-pp._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.72 ± 34%      +3.4       12.12 ±  2%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.dup_mm.copy_process
      8.99 ± 33%      +3.5       12.52 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.dup_mm.copy_process._do_fork
      9.03 ± 33%      +3.5       12.58 ±  2%  perf-profile.calltrace.cycles-pp.down_write.dup_mm.copy_process._do_fork.do_syscall_64
     10.29 ± 33%      +4.0       14.31 ±  2%  perf-profile.calltrace.cycles-pp.dup_mm.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
     33.92 ± 38%     +15.3       49.23        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     33.96 ± 38%     +15.3       49.30        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      9.51 ±103%      -9.5        0.00        perf-profile.children.cycles-pp.sieve
      0.07 ± 19%      +0.0        0.09        perf-profile.children.cycles-pp.strlcpy
      0.06 ± 20%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.__do_fault
      0.06 ± 20%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.__tlb_remove_page_size
      0.10 ± 12%      +0.0        0.12 ±  4%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.07 ± 17%      +0.0        0.09 ± 11%  perf-profile.children.cycles-pp.__list_add_valid
      0.07 ± 17%      +0.0        0.09        perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.07 ± 20%      +0.0        0.10        perf-profile.children.cycles-pp.__check_object_size
      0.04 ± 58%      +0.0        0.07        perf-profile.children.cycles-pp.mem_cgroup_update_lru_size
      0.09 ± 24%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.down_read
      0.08 ± 19%      +0.0        0.11 ±  9%  perf-profile.children.cycles-pp.strncpy_from_user
      0.11 ± 21%      +0.0        0.14 ± 11%  perf-profile.children.cycles-pp.new_sync_read
      0.08 ± 22%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.finish_task_switch
      0.08 ± 21%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.down_write_killable
      0.07 ± 20%      +0.0        0.10 ± 14%  perf-profile.children.cycles-pp.do_notify_parent
      0.08 ± 23%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.try_charge
      0.09 ± 20%      +0.0        0.12 ±  4%  perf-profile.children.cycles-pp.__fxstat64
      0.08 ± 23%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_list
      0.07 ± 26%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.find_get_entry
      0.05 ± 60%      +0.0        0.08        perf-profile.children.cycles-pp.free_pgd_range
      0.08 ± 26%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.__count_memcg_events
      0.05 ± 61%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.d_alloc_parallel
      0.03 ±100%      +0.0        0.06        perf-profile.children.cycles-pp.inode_permission
      0.03 ±100%      +0.0        0.06        perf-profile.children.cycles-pp.move_queued_task
      0.11 ± 15%      +0.0        0.14 ±  7%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.11 ± 20%      +0.0        0.15 ±  3%  perf-profile.children.cycles-pp.page_add_file_rmap
      0.10 ± 25%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.memcpy_erms
      0.10 ± 28%      +0.0        0.13 ±  7%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.04 ± 59%      +0.0        0.08        perf-profile.children.cycles-pp.shift_arg_pages
      0.10 ± 23%      +0.0        0.14 ±  7%  perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
      0.11 ± 22%      +0.0        0.15 ±  3%  perf-profile.children.cycles-pp.do_open_execat
      0.10 ± 23%      +0.0        0.14 ±  7%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.03 ±100%      +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.free_p4d_range
      0.03 ±100%      +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.12 ± 27%      +0.0        0.15 ±  3%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.12 ± 26%      +0.0        0.16 ±  6%  perf-profile.children.cycles-pp.dput
      0.11 ± 28%      +0.0        0.15 ±  3%  perf-profile.children.cycles-pp.__d_lookup_rcu
      0.10 ± 25%      +0.0        0.14        perf-profile.children.cycles-pp.time
      0.10 ± 17%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.__x64_sys_munmap
      0.06 ± 58%      +0.0        0.10        perf-profile.children.cycles-pp.cap_vm_enough_memory
      0.11 ± 28%      +0.0        0.15 ±  6%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.11 ± 23%      +0.0        0.15        perf-profile.children.cycles-pp.__might_sleep
      0.07 ± 24%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.lookup_slow
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.__ctype_init
      0.09 ± 29%      +0.0        0.13        perf-profile.children.cycles-pp.setup_arg_pages
      0.03 ±100%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp._IO_file_open
      0.12 ± 24%      +0.0        0.17 ±  5%  perf-profile.children.cycles-pp.do_task_dead
      0.10 ± 23%      +0.0        0.15        perf-profile.children.cycles-pp.__rb_insert_augmented
      0.05 ± 62%      +0.0        0.10        perf-profile.children.cycles-pp.__lookup_slow
      0.01 ±173%      +0.0        0.06 ± 16%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.01 ±173%      +0.0        0.06        perf-profile.children.cycles-pp.khugepaged_enter_vma_merge
      0.15 ± 21%      +0.0        0.20        perf-profile.children.cycles-pp.copy_user_highpage
      0.15 ± 21%      +0.0        0.20        perf-profile.children.cycles-pp.copy_page
      0.03 ±100%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__wake_up_common
      0.11 ± 25%      +0.0        0.15 ±  3%  perf-profile.children.cycles-pp.refcount_dec_not_one
      0.04 ±100%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.rcu_all_qs
      0.05 ± 61%      +0.1        0.10 ± 19%  perf-profile.children.cycles-pp.__vma_link_rb
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.filemap_fault
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.mark_page_accessed
      0.00            +0.1        0.05        perf-profile.children.cycles-pp._exit
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.memmem
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.perf_exclude_event
      0.14 ± 27%      +0.1        0.19 ±  5%  perf-profile.children.cycles-pp._cond_resched
      0.14 ± 26%      +0.1        0.20 ±  2%  perf-profile.children.cycles-pp.selinux_inode_permission
      0.09 ± 31%      +0.1        0.15 ± 31%  perf-profile.children.cycles-pp.__generic_file_write_iter
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.__slab_alloc
      0.15 ± 24%      +0.1        0.20 ±  4%  perf-profile.children.cycles-pp.update_curr
      0.10 ± 29%      +0.1        0.15 ±  6%  perf-profile.children.cycles-pp.free_uid
      0.10 ± 29%      +0.1        0.15 ±  6%  perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave
      0.14 ± 26%      +0.1        0.20 ± 23%  perf-profile.children.cycles-pp.new_sync_write
      0.10 ± 32%      +0.1        0.15 ± 29%  perf-profile.children.cycles-pp.generic_file_write_iter
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.__mod_zone_page_state
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.___slab_alloc
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.uncharge_batch
      0.16 ± 27%      +0.1        0.22 ± 20%  perf-profile.children.cycles-pp.vfs_write
      0.14 ± 23%      +0.1        0.20        perf-profile.children.cycles-pp.do_faccessat
      0.13 ± 25%      +0.1        0.19        perf-profile.children.cycles-pp.remove_vma
      0.12 ± 24%      +0.1        0.18        perf-profile.children.cycles-pp.strnlen_user
      0.12 ± 22%      +0.1        0.17 ±  2%  perf-profile.children.cycles-pp.mem_cgroup_commit_charge
      0.15 ± 29%      +0.1        0.21 ±  4%  perf-profile.children.cycles-pp.security_inode_permission
      0.16 ± 29%      +0.1        0.21 ±  2%  perf-profile.children.cycles-pp.anon_vma_interval_tree_insert
      0.17 ± 28%      +0.1        0.23 ±  2%  perf-profile.children.cycles-pp.__clear_user
      0.15 ± 27%      +0.1        0.21 ±  4%  perf-profile.children.cycles-pp.wake_up_new_task
      0.12 ± 22%      +0.1        0.18 ± 11%  perf-profile.children.cycles-pp.anon_vma_interval_tree_remove
      0.00            +0.1        0.06        perf-profile.children.cycles-pp.unmap_single_vma
      0.17 ± 26%      +0.1        0.23        perf-profile.children.cycles-pp.lookup_fast
      0.16 ± 27%      +0.1        0.22 ±  6%  perf-profile.children.cycles-pp.sched_exec
      0.12 ± 24%      +0.1        0.18 ±  5%  perf-profile.children.cycles-pp.perf_event_mmap_output
      0.16 ± 25%      +0.1        0.23 ± 20%  perf-profile.children.cycles-pp.ksys_write
      0.20 ± 26%      +0.1        0.26        perf-profile.children.cycles-pp.free_pages_and_swap_cache
      0.20 ± 16%      +0.1        0.27 ±  5%  perf-profile.children.cycles-pp.lru_add_drain_cpu
      0.20 ± 16%      +0.1        0.27 ±  5%  perf-profile.children.cycles-pp.lru_add_drain
      0.24 ± 19%      +0.1        0.31 ±  9%  perf-profile.children.cycles-pp.free_unref_page_list
      0.20 ± 29%      +0.1        0.27        perf-profile.children.cycles-pp.enqueue_entity
      0.18 ± 25%      +0.1        0.25        perf-profile.children.cycles-pp.___might_sleep
      0.25 ± 20%      +0.1        0.33 ±  3%  perf-profile.children.cycles-pp.get_unmapped_area
      0.21 ± 28%      +0.1        0.30 ±  5%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.23 ± 29%      +0.1        0.31 ±  3%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.19 ± 31%      +0.1        0.28 ±  3%  perf-profile.children.cycles-pp.smpboot_thread_fn
      0.31 ± 20%      +0.1        0.41 ±  2%  perf-profile.children.cycles-pp.wp_page_copy
      0.23 ± 24%      +0.1        0.33        perf-profile.children.cycles-pp.alloc_set_pte
      0.29 ± 28%      +0.1        0.39        perf-profile.children.cycles-pp.mmap64
      0.25 ± 24%      +0.1        0.35        perf-profile.children.cycles-pp.walk_component
      0.31 ± 25%      +0.1        0.42 ±  3%  perf-profile.children.cycles-pp.find_vma
      0.34 ± 23%      +0.1        0.45        perf-profile.children.cycles-pp.link_path_walk
      0.17 ± 25%      +0.1        0.28 ±  3%  perf-profile.children.cycles-pp.put_cred_rcu
      0.31 ± 30%      +0.1        0.42 ±  4%  perf-profile.children.cycles-pp._dl_addr
      0.29 ± 25%      +0.1        0.41 ±  3%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.30 ± 26%      +0.1        0.42 ±  3%  perf-profile.children.cycles-pp.copy_strings
      0.39 ± 18%      +0.1        0.50 ±  2%  perf-profile.children.cycles-pp.do_wp_page
      0.31 ± 19%      +0.1        0.42 ±  3%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.31 ± 23%      +0.1        0.43 ±  4%  perf-profile.children.cycles-pp.perf_iterate_sb
      0.36 ± 21%      +0.1        0.48 ±  8%  perf-profile.children.cycles-pp.iterate_supers
      0.46 ± 19%      +0.1        0.58 ±  6%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.38 ± 22%      +0.1        0.50 ±  8%  perf-profile.children.cycles-pp.__x64_sys_sync
      0.38 ± 22%      +0.1        0.50 ±  8%  perf-profile.children.cycles-pp.ksys_sync
      0.39 ± 17%      +0.1        0.52 ±  2%  perf-profile.children.cycles-pp.__lru_cache_add
      0.38 ± 23%      +0.1        0.51 ±  7%  perf-profile.children.cycles-pp.sync
      0.35 ± 29%      +0.1        0.49 ±  2%  perf-profile.children.cycles-pp.vma_interval_tree_insert
      0.47 ± 22%      +0.1        0.61 ±  4%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.36 ± 21%      +0.1        0.51 ±  6%  perf-profile.children.cycles-pp.vma_merge
      0.33 ± 27%      +0.2        0.48 ±  2%  perf-profile.children.cycles-pp.rcu_core
      0.43 ± 27%      +0.2        0.58        perf-profile.children.cycles-pp.copy_p4d_range
      0.44 ± 28%      +0.2        0.60        perf-profile.children.cycles-pp.copy_page_range
      0.54 ± 15%      +0.2        0.71        perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.57 ± 22%      +0.2        0.77        perf-profile.children.cycles-pp.perf_event_mmap
      0.61 ± 27%      +0.2        0.81 ±  6%  perf-profile.children.cycles-pp.load_balance
      0.68 ± 21%      +0.2        0.89        perf-profile.children.cycles-pp.native_flush_tlb_one_user
      0.57 ± 29%      +0.2        0.78 ±  3%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.70 ± 20%      +0.2        0.92        perf-profile.children.cycles-pp.flush_tlb_func_common
      0.76 ± 20%      +0.2        1.00 ±  2%  perf-profile.children.cycles-pp.flush_tlb_mm_range
      0.64 ± 27%      +0.2        0.89 ±  9%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.60 ± 26%      +0.3        0.85 ± 12%  perf-profile.children.cycles-pp.do_sys_open
      0.79 ± 20%      +0.3        1.06 ±  3%  perf-profile.children.cycles-pp.clear_page_erms
      0.63 ± 26%      +0.3        0.89 ± 12%  perf-profile.children.cycles-pp.do_filp_open
      0.62 ± 27%      +0.3        0.88 ± 12%  perf-profile.children.cycles-pp.path_openat
      0.68 ± 27%      +0.3        0.95 ±  8%  perf-profile.children.cycles-pp.schedule
      0.79 ± 25%      +0.3        1.06 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.84 ± 20%      +0.3        1.11 ±  3%  perf-profile.children.cycles-pp.prep_new_page
      0.77 ± 33%      +0.3        1.08 ±  5%  perf-profile.children.cycles-pp.rwsem_wake
      1.09 ± 22%      +0.3        1.40        perf-profile.children.cycles-pp.prepare_exit_to_usermode
      1.15 ± 20%      +0.3        1.46        perf-profile.children.cycles-pp.release_pages
      0.70 ± 32%      +0.3        1.05 ±  4%  perf-profile.children.cycles-pp.wake_up_q
      0.94 ± 27%      +0.4        1.29 ±  8%  perf-profile.children.cycles-pp.__schedule
      1.22 ± 19%      +0.4        1.57        perf-profile.children.cycles-pp.alloc_pages_vma
      1.22 ± 21%      +0.4        1.59 ±  3%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.76 ± 31%      +0.4        1.13 ±  3%  perf-profile.children.cycles-pp.try_to_wake_up
      1.29 ± 20%      +0.4        1.69        perf-profile.children.cycles-pp.do_brk_flags
      1.19 ± 26%      +0.4        1.60        perf-profile.children.cycles-pp.filemap_map_pages
      1.46 ± 21%      +0.4        1.91 ±  2%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      1.27 ± 24%      +0.4        1.71        perf-profile.children.cycles-pp.unmap_page_range
      1.33 ± 24%      +0.5        1.81        perf-profile.children.cycles-pp.unmap_vmas
      1.98 ± 21%      +0.6        2.54        perf-profile.children.cycles-pp.tlb_finish_mmu
      1.95 ± 21%      +0.6        2.52        perf-profile.children.cycles-pp.tlb_flush_mmu
      2.19 ± 19%      +0.7        2.85        perf-profile.children.cycles-pp.do_anonymous_page
      2.88 ± 24%      +0.9        3.78 ±  3%  perf-profile.children.cycles-pp.unmap_region
      3.15 ± 20%      +1.0        4.11        perf-profile.children.cycles-pp.__x64_sys_brk
      3.91 ± 25%      +1.3        5.20 ±  2%  perf-profile.children.cycles-pp.__do_munmap
      4.39 ± 21%      +1.4        5.79        perf-profile.children.cycles-pp.__handle_mm_fault
      4.61 ± 21%      +1.5        6.08        perf-profile.children.cycles-pp.handle_mm_fault
      4.53 ± 31%      +1.5        6.04 ±  3%  perf-profile.children.cycles-pp.__vma_adjust
      5.54 ± 19%      +1.6        7.16        perf-profile.children.cycles-pp.page_fault
      5.20 ± 22%      +1.6        6.85        perf-profile.children.cycles-pp.__do_page_fault
      5.35 ± 22%      +1.7        7.04        perf-profile.children.cycles-pp.do_page_fault
      5.42 ± 31%      +2.2        7.58        perf-profile.children.cycles-pp.ksys_mmap_pgoff
      7.28 ± 28%      +2.7        9.98 ±  2%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      7.28 ± 28%      +2.7        9.98 ±  2%  perf-profile.children.cycles-pp.do_group_exit
      7.28 ± 28%      +2.7        9.98 ±  2%  perf-profile.children.cycles-pp.do_exit
     10.30 ± 33%      +4.0       14.31 ±  2%  perf-profile.children.cycles-pp.dup_mm
     10.75 ± 33%      +4.1       14.87 ±  2%  perf-profile.children.cycles-pp.copy_process
     10.93 ± 33%      +4.2       15.12 ±  2%  perf-profile.children.cycles-pp._do_fork
     29.33 ± 31%      +9.7       39.08        perf-profile.children.cycles-pp.rwsem_down_write_slowpath
     29.57 ± 31%      +9.9       39.42        perf-profile.children.cycles-pp.down_write
     45.34 ± 29%     +15.0       60.32        perf-profile.children.cycles-pp.do_syscall_64
     45.39 ± 29%     +15.0       60.38        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
      9.39 ±103%      -9.4        0.00        perf-profile.self.cycles-pp.sieve
      0.08 ± 19%      +0.0        0.10 ± 10%  perf-profile.self.cycles-pp.sync_regs
      0.09 ± 21%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.page_add_file_rmap
      0.08 ± 19%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.unlock_page
      0.08 ± 16%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.do_brk_flags
      0.05 ± 58%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.__mod_memcg_state
      0.07 ± 23%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.unlink_anon_vmas
      0.08 ± 24%      +0.0        0.11        perf-profile.self.cycles-pp.__count_memcg_events
      0.08 ± 24%      +0.0        0.11 ±  9%  perf-profile.self.cycles-pp.perf_event_mmap_output
      0.09 ± 20%      +0.0        0.12 ±  4%  perf-profile.self.cycles-pp.update_curr
      0.10 ± 23%      +0.0        0.13 ±  7%  perf-profile.self.cycles-pp.memcpy_erms
      0.05 ± 60%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.free_pcppages_bulk
      0.09 ± 29%      +0.0        0.12 ± 12%  perf-profile.self.cycles-pp.vma_compute_subtree_gap
      0.03 ±100%      +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.rcu_all_qs
      0.10 ± 27%      +0.0        0.14        perf-profile.self.cycles-pp.__might_sleep
      0.03 ±100%      +0.0        0.07        perf-profile.self.cycles-pp.dup_mm
      0.13 ± 26%      +0.0        0.17 ±  5%  perf-profile.self.cycles-pp.vma_interval_tree_remove
      0.04 ± 58%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.down_write_killable
      0.09 ± 25%      +0.0        0.14 ±  3%  perf-profile.self.cycles-pp.strnlen_user
      0.09 ± 22%      +0.0        0.14 ±  3%  perf-profile.self.cycles-pp.__rb_insert_augmented
      0.16 ± 18%      +0.0        0.21 ±  2%  perf-profile.self.cycles-pp.perf_iterate_sb
      0.14 ± 23%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.perf_event_mmap
      0.15 ± 20%      +0.0        0.20 ±  2%  perf-profile.self.cycles-pp.copy_page
      0.19 ± 20%      +0.0        0.24 ±  4%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.05 ± 62%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.put_cred_rcu
      0.13 ± 18%      +0.0        0.18        perf-profile.self.cycles-pp.__vma_adjust
      0.11 ± 25%      +0.0        0.15 ±  3%  perf-profile.self.cycles-pp.refcount_dec_not_one
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.___slab_alloc
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.pmd_devmap_trans_unstable
      0.00            +0.1        0.05        perf-profile.self.cycles-pp._vm_normal_page
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.mark_page_accessed
      0.11 ± 24%      +0.1        0.16 ±  9%  perf-profile.self.cycles-pp.anon_vma_interval_tree_remove
      0.20 ± 20%      +0.1        0.25 ±  4%  perf-profile.self.cycles-pp.ftrace_syscall_enter
      0.15 ± 23%      +0.1        0.20        perf-profile.self.cycles-pp.find_vma
      0.15 ± 25%      +0.1        0.21 ±  2%  perf-profile.self.cycles-pp.__slab_free
      0.12 ± 17%      +0.1        0.17 ±  2%  perf-profile.self.cycles-pp.handle_mm_fault
      0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.__mod_zone_page_state
      0.17 ± 20%      +0.1        0.23 ±  2%  perf-profile.self.cycles-pp.__do_page_fault
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.unmap_single_vma
      0.15 ± 31%      +0.1        0.21 ±  2%  perf-profile.self.cycles-pp.anon_vma_interval_tree_insert
      0.19 ± 20%      +0.1        0.26        perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.17 ± 26%      +0.1        0.24        perf-profile.self.cycles-pp.___might_sleep
      0.18 ± 24%      +0.1        0.25        perf-profile.self.cycles-pp.down_write
      0.13 ± 28%      +0.1        0.20 ±  4%  perf-profile.self.cycles-pp.try_to_wake_up
      0.21 ± 25%      +0.1        0.29        perf-profile.self.cycles-pp.select_task_rq_fair
      0.23 ± 27%      +0.1        0.33        perf-profile.self.cycles-pp.copy_p4d_range
      0.28 ± 19%      +0.1        0.38        perf-profile.self.cycles-pp.__handle_mm_fault
      0.29 ± 30%      +0.1        0.41 ±  3%  perf-profile.self.cycles-pp._dl_addr
      0.35 ± 29%      +0.1        0.48 ±  2%  perf-profile.self.cycles-pp.vma_interval_tree_insert
      0.45 ± 21%      +0.1        0.57 ±  6%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.37 ± 23%      +0.1        0.51        perf-profile.self.cycles-pp._raw_spin_lock
      0.37 ± 27%      +0.1        0.51 ±  8%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.47 ± 22%      +0.1        0.61 ±  4%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.45 ± 28%      +0.2        0.62 ±  3%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.55 ± 23%      +0.2        0.73        perf-profile.self.cycles-pp.release_pages
      0.68 ± 21%      +0.2        0.89        perf-profile.self.cycles-pp.native_flush_tlb_one_user
      0.79 ± 21%      +0.3        1.04 ±  3%  perf-profile.self.cycles-pp.clear_page_erms
      0.72 ± 30%      +0.3        0.98 ±  3%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.70 ± 26%      +0.3        0.97        perf-profile.self.cycles-pp.unmap_page_range
      0.82 ± 27%      +0.3        1.10        perf-profile.self.cycles-pp.filemap_map_pages
      1.04 ± 22%      +0.3        1.35        perf-profile.self.cycles-pp.prepare_exit_to_usermode
      1.28 ± 22%      +0.4        1.67        perf-profile.self.cycles-pp.do_syscall_64



***************************************************************************************************
lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
  gcc-7/performance/4BRD_12G/ext4/x86_64-rhel-7.6/3000/RAID1/debian-x86_64-2019-05-14.cgz/lkp-ivb-ep01/disk_src/aim7

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:2     dmesg.WARNING:at#for_ip_interrupt_entry/0x
          2:4          -50%            :2     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
         %stddev     %change         %stddev
             \          |                \  
      2930 ±  2%      -4.4%       2802        aim7.time.maximum_resident_set_size
    309.25 ±173%    +304.2%       1250        meminfo.Mlocked
      0.03 ± 31%      +0.0        0.04 ± 25%  mpstat.cpu.all.iowait%
      2.54 ± 76%      -2.0        0.53 ±  5%  turbostat.C1%
   1468703 ± 75%    +176.4%    4059526 ± 62%  turbostat.C3
      5.75 ± 93%      +8.9       14.62 ± 72%  turbostat.C3%
 4.132e+08 ± 95%    +160.6%  1.077e+09 ± 71%  cpuidle.C3.time
   1469107 ± 75%    +176.4%    4059905 ± 62%  cpuidle.C3.usage
    106863 ±  3%     +13.0%     120776        cpuidle.POLL.time
      6635 ±  5%     +12.3%       7451 ±  4%  cpuidle.POLL.usage
    109506            +1.1%     110712        proc-vmstat.nr_active_anon
    107428            +1.0%     108475        proc-vmstat.nr_anon_pages
     77.25 ±173%    +303.9%     312.00        proc-vmstat.nr_mlock
    109506            +1.1%     110712        proc-vmstat.nr_zone_active_anon
      1778 ±  5%     -79.3%     368.50 ± 50%  numa-meminfo.node0.Active(file)
    356.25 ± 26%    +399.8%       1780 ±  9%  numa-meminfo.node1.Active(file)
     55033 ± 37%     +93.4%     106429 ± 15%  numa-meminfo.node1.AnonHugePages
    214912 ± 14%     +15.9%     249074 ±  7%  numa-meminfo.node1.AnonPages
   1237733 ± 12%     +52.8%    1890859 ± 29%  numa-meminfo.node1.MemUsed
      5280 ± 15%     +16.2%       6133 ± 10%  slabinfo.dmaengine-unmap-16.active_objs
      5283 ± 15%     +16.2%       6140 ± 10%  slabinfo.dmaengine-unmap-16.num_objs
     20271 ±  6%      +8.8%      22058 ±  2%  slabinfo.kmalloc-32.active_objs
    191.25 ± 35%     +49.0%     285.00 ± 10%  slabinfo.request_queue.num_objs
      1732 ± 10%     +19.2%       2064        slabinfo.skbuff_head_cache.active_objs
      1840 ± 10%     +19.1%       2192        slabinfo.skbuff_head_cache.num_objs
      0.75 ±  3%      -0.1        0.65 ±  3%  perf-stat.i.dTLB-load-miss-rate%
  17735473 ± 11%     -19.1%   14349602 ±  4%  perf-stat.i.dTLB-load-misses
      0.19 ±  4%      -0.0        0.15 ±  3%  perf-stat.i.dTLB-store-miss-rate%
   1897982 ±  4%     -20.9%    1500903 ±  3%  perf-stat.i.dTLB-store-misses
   2188000 ±  3%      -4.4%    2091270        perf-stat.i.iTLB-load-misses
      0.74 ±  3%      -0.1        0.64 ±  4%  perf-stat.overall.dTLB-load-miss-rate%
      0.19 ±  5%      -0.0        0.15 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
  17630676 ± 11%     -19.1%   14270976 ±  4%  perf-stat.ps.dTLB-load-misses
   1886940 ±  4%     -20.9%    1492696 ±  3%  perf-stat.ps.dTLB-store-misses
    434.30 ± 16%     +31.3%     570.43 ±  6%  sched_debug.cfs_rq:/.exec_clock.stddev
     20136 ±  5%     +17.9%      23747 ±  7%  sched_debug.cfs_rq:/.load.avg
      2.00 ± 40%     +75.0%       3.50 ±  7%  sched_debug.cfs_rq:/.load_avg.min
      0.31 ± 12%     +14.2%       0.36 ±  8%  sched_debug.cfs_rq:/.nr_running.avg
      2.06 ±  5%     +33.3%       2.75 ±  9%  sched_debug.cfs_rq:/.nr_spread_over.max
      7.64 ±  8%     +11.2%       8.50 ±  5%  sched_debug.cfs_rq:/.runnable_load_avg.avg
    142834 ± 14%     +14.9%     164118 ±  4%  sched_debug.cfs_rq:/.spread0.max
    151.79 ± 18%     +21.5%     184.50 ±  5%  sched_debug.cfs_rq:/.util_avg.min
      9.16 ±  4%     -11.3%       8.12 ±  2%  sched_debug.cpu.cpu_load[1].avg
      2364 ± 10%     +25.7%       2971 ±  9%  sched_debug.cpu.nr_switches.stddev
     -0.23         -5554.5%      12.50 ± 20%  sched_debug.cpu.nr_uninterruptible.min
    126.66 ± 16%     +17.4%     148.73        sched_debug.cpu.ttwu_local.stddev
    176.50 ± 61%     +46.2%     258.00 ± 46%  interrupts.42:IR-PCI-MSI.524292-edge.eth0-TxRx-3
    135.50 ± 26%     +60.5%     217.50 ±  5%  interrupts.43:IR-PCI-MSI.524293-edge.eth0-TxRx-4
    784.00 ± 86%     -85.9%     110.50 ±  2%  interrupts.45:IR-PCI-MSI.524295-edge.eth0-TxRx-6
    319.50 ± 45%     -68.7%     100.00 ±  5%  interrupts.46:IR-PCI-MSI.524296-edge.eth0-TxRx-7
      2847 ± 12%     +18.4%       3371 ±  5%  interrupts.CPU0.RES:Rescheduling_interrupts
      1236 ±  3%    -100.0%       0.00        interrupts.CPU12.10:IR-IO-APIC.10-edge.ipmi_si
      2600 ±  2%     +10.0%       2861 ±  3%  interrupts.CPU18.RES:Rescheduling_interrupts
    477.75 ± 96%    +108.3%     995.00 ±  9%  interrupts.CPU21.NMI:Non-maskable_interrupts
    477.75 ± 96%    +108.3%     995.00 ±  9%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
    176.50 ± 61%     +46.2%     258.00 ± 46%  interrupts.CPU36.42:IR-PCI-MSI.524292-edge.eth0-TxRx-3
    135.50 ± 26%     +60.5%     217.50 ±  5%  interrupts.CPU38.43:IR-PCI-MSI.524293-edge.eth0-TxRx-4
    784.00 ± 86%     -85.9%     110.50 ±  2%  interrupts.CPU4.45:IR-PCI-MSI.524295-edge.eth0-TxRx-6
    319.50 ± 45%     -68.7%     100.00 ±  5%  interrupts.CPU6.46:IR-PCI-MSI.524296-edge.eth0-TxRx-7
    444.50 ±  5%     -79.3%      92.00 ± 50%  numa-vmstat.node0.nr_active_file
     17870           -97.4%     462.00 ± 40%  numa-vmstat.node0.nr_dirtied
     44.50 ±173%    +250.6%     156.00 ± 16%  numa-vmstat.node0.nr_mlock
     17843           -97.6%     432.50 ± 41%  numa-vmstat.node0.nr_written
    444.50 ±  5%     -79.3%      92.00 ± 50%  numa-vmstat.node0.nr_zone_active_file
    831590 ±  3%     -33.4%     553775 ± 22%  numa-vmstat.node0.numa_hit
    819665 ±  4%     -39.6%     494911 ± 15%  numa-vmstat.node0.numa_local
     11924 ±108%    +393.6%      58863 ± 77%  numa-vmstat.node0.numa_other
     88.75 ± 26%    +400.8%     444.50 ±  9%  numa-vmstat.node1.nr_active_file
     53723 ± 14%     +15.9%      62267 ±  7%  numa-vmstat.node1.nr_anon_pages
    694.50 ± 28%   +2499.2%      18051        numa-vmstat.node1.nr_dirtied
     32.75 ±173%    +374.8%     155.50 ± 15%  numa-vmstat.node1.nr_mlock
    665.75 ± 29%   +2605.9%      18014        numa-vmstat.node1.nr_written
     88.75 ± 26%    +400.8%     444.50 ±  9%  numa-vmstat.node1.nr_zone_active_file
    510127 ±  7%     +55.2%     791681 ± 15%  numa-vmstat.node1.numa_hit
    343005 ± 12%     +95.7%     671266 ± 11%  numa-vmstat.node1.numa_local
     52986            +7.8%      57135 ±  2%  softirqs.CPU0.RCU
     54275            +8.0%      58613 ±  4%  softirqs.CPU1.RCU
     28664 ±  3%     +10.6%      31701 ±  2%  softirqs.CPU1.SCHED
     74527 ±  5%     -17.7%      61372 ±  4%  softirqs.CPU1.TIMER
     52906           +11.4%      58960        softirqs.CPU11.RCU
     52231           +12.0%      58488 ±  6%  softirqs.CPU13.RCU
     28678 ±  3%      +9.8%      31491 ±  4%  softirqs.CPU13.SCHED
     52884 ±  2%     +11.4%      58899        softirqs.CPU18.RCU
     52822            +9.9%      58040 ±  2%  softirqs.CPU19.RCU
     52881 ±  2%      +9.9%      58100 ±  4%  softirqs.CPU2.RCU
     52515 ±  2%      +9.2%      57356 ±  2%  softirqs.CPU20.RCU
     54018           +14.4%      61810 ±  4%  softirqs.CPU21.RCU
     28115           +12.9%      31748        softirqs.CPU21.SCHED
     54773 ±  2%      +6.5%      58316 ±  3%  softirqs.CPU22.RCU
     53536           +11.4%      59660 ±  5%  softirqs.CPU23.RCU
     29616 ±  5%      +7.1%      31705 ±  5%  softirqs.CPU23.SCHED
     27797            +9.0%      30292 ±  2%  softirqs.CPU24.SCHED
     55008            +8.5%      59698 ±  4%  softirqs.CPU25.RCU
     55186 ±  2%      +7.3%      59236 ±  4%  softirqs.CPU27.RCU
     54285            +9.1%      59232 ±  5%  softirqs.CPU29.RCU
     53456            +8.1%      57762 ±  3%  softirqs.CPU3.RCU
     53396           +10.8%      59144 ±  3%  softirqs.CPU31.RCU
     54044           +10.2%      59576 ±  5%  softirqs.CPU5.RCU
      1.22 ±  2%      -0.1        1.11 ±  4%  perf-profile.calltrace.cycles-pp.ext4_orphan_add.ext4_unlink.vfs_unlink.do_unlinkat.do_syscall_64
      0.65 ±  6%      +0.1        0.73 ±  3%  perf-profile.calltrace.cycles-pp.crc32c_pcl_intel_update.creat
      1.16 ±  9%      +0.1        1.29        perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.00 ± 10%      +0.2        1.16        perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      4.58 ±  3%      +0.2        4.78 ±  2%  perf-profile.calltrace.cycles-pp.ext4_evict_inode.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.89 ±  3%      +0.2        5.08 ±  2%  perf-profile.calltrace.cycles-pp.evict.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      0.47 ± 57%      +0.2        0.71 ±  3%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.13 ±173%      +0.5        0.60 ±  6%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.00            +0.5        0.54 ±  6%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.24 ± 22%      -0.1        0.17        perf-profile.children.cycles-pp.mutex_lock
      0.39 ±  7%      -0.1        0.33        perf-profile.children.cycles-pp.dput
      0.12 ± 13%      -0.1        0.07 ±  7%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.41 ±  3%      -0.0        0.37 ±  2%  perf-profile.children.cycles-pp.inode_permission
      0.18 ±  8%      -0.0        0.15 ± 10%  perf-profile.children.cycles-pp.may_delete
      0.09 ± 17%      -0.0        0.06        perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.12 ± 12%      -0.0        0.10 ±  5%  perf-profile.children.cycles-pp.find_inode_bit
      0.11 ±  7%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.find_next_zero_bit
      0.05 ±  8%      +0.0        0.07        perf-profile.children.cycles-pp.open64
      0.09 ± 12%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.set_next_entity
      0.14 ± 10%      +0.0        0.16 ±  6%  perf-profile.children.cycles-pp.mark_page_accessed
      0.06 ± 14%      +0.0        0.08        perf-profile.children.cycles-pp.generic_file_read_iter
      0.18 ±  8%      +0.0        0.20        perf-profile.children.cycles-pp.switch_fpu_return
      0.08 ±  8%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.fsnotify_grab_connector
      0.07 ± 14%      +0.0        0.10 ± 10%  perf-profile.children.cycles-pp.security_file_alloc
      0.12 ± 19%      +0.0        0.15 ± 17%  perf-profile.children.cycles-pp.vfs_read
      0.10 ±  8%      +0.0        0.13 ±  7%  perf-profile.children.cycles-pp.run_timer_softirq
      0.07 ± 15%      +0.0        0.10 ± 10%  perf-profile.children.cycles-pp.note_gp_changes
      0.13 ± 16%      +0.0        0.16 ±  9%  perf-profile.children.cycles-pp.ksys_read
      0.04 ± 58%      +0.0        0.07 ± 14%  perf-profile.children.cycles-pp.__ext4_handle_dirty_super
      0.04 ± 60%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.update_cfs_rq_h_load
      0.08 ±  6%      +0.0        0.11 ±  9%  perf-profile.children.cycles-pp.new_sync_read
      0.12 ± 17%      +0.0        0.16 ± 12%  perf-profile.children.cycles-pp.rcu_all_qs
      0.13 ± 10%      +0.0        0.17 ±  3%  perf-profile.children.cycles-pp.ext4_release_file
      0.04 ± 59%      +0.0        0.08 ± 12%  perf-profile.children.cycles-pp.__srcu_read_lock
      0.16 ±  6%      +0.0        0.20 ±  4%  perf-profile.children.cycles-pp.avc_has_perm
      0.29 ±  5%      +0.0        0.33        perf-profile.children.cycles-pp.d_lookup
      0.44 ±  5%      +0.0        0.48 ±  2%  perf-profile.children.cycles-pp.__fput
      0.05 ± 61%      +0.0        0.09 ± 11%  perf-profile.children.cycles-pp.__kernel_text_address
      0.11 ± 17%      +0.0        0.15 ± 13%  perf-profile.children.cycles-pp.ext4_inode_attach_jinode
      0.19 ± 11%      +0.0        0.23 ±  2%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.03 ±100%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.kernel_text_address
      0.03 ±102%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.15 ±  5%      +0.0        0.20 ±  7%  perf-profile.children.cycles-pp.read
      0.01 ±173%      +0.0        0.06        perf-profile.children.cycles-pp.check_preempt_curr
      0.22 ± 14%      +0.0        0.27 ±  3%  perf-profile.children.cycles-pp.update_blocked_averages
      0.20 ± 19%      +0.0        0.24 ±  6%  perf-profile.children.cycles-pp.__alloc_file
      0.17 ± 15%      +0.0        0.21 ±  2%  perf-profile.children.cycles-pp._raw_spin_trylock
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.native_apic_mem_write
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.__note_gp_changes
      0.23 ± 15%      +0.1        0.28 ±  3%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.30 ± 14%      +0.1        0.35 ±  4%  perf-profile.children.cycles-pp.clockevents_program_event
      0.27 ±  7%      +0.1        0.33        perf-profile.children.cycles-pp.__d_lookup
      0.23 ± 15%      +0.1        0.28 ±  3%  perf-profile.children.cycles-pp.alloc_empty_file
      0.38 ± 11%      +0.1        0.45 ±  4%  perf-profile.children.cycles-pp.jbd2_journal_stop
      0.52 ±  6%      +0.1        0.59 ±  5%  perf-profile.children.cycles-pp.rcu_core
      0.41 ±  7%      +0.1        0.49 ±  7%  perf-profile.children.cycles-pp.__ext4_journal_stop
      0.14 ± 11%      +0.1        0.22 ± 28%  perf-profile.children.cycles-pp.__virt_addr_valid
      0.50 ±  4%      +0.1        0.60 ±  8%  perf-profile.children.cycles-pp.ext4_map_blocks
      0.72 ±  7%      +0.1        0.82 ±  7%  perf-profile.children.cycles-pp.getname_flags
      0.47 ±  7%      +0.1        0.57 ±  7%  perf-profile.children.cycles-pp.strncpy_from_user
      0.33 ±  6%      +0.1        0.43 ± 12%  perf-profile.children.cycles-pp.ext4_es_lookup_extent
      0.23 ± 35%      +0.1        0.35 ± 28%  perf-profile.children.cycles-pp.io_serial_in
      0.89 ±  9%      +0.1        1.01 ±  7%  perf-profile.children.cycles-pp.tick_sched_handle
      1.03 ±  9%      +0.1        1.16 ±  5%  perf-profile.children.cycles-pp.tick_sched_timer
      1.38 ±  6%      +0.1        1.52        perf-profile.children.cycles-pp.irq_exit
      1.21 ±  8%      +0.2        1.38 ±  2%  perf-profile.children.cycles-pp.__softirqentry_text_start
      1.55 ±  5%      +0.2        1.73        perf-profile.children.cycles-pp.crc32c_pcl_intel_update
      4.59 ±  3%      +0.2        4.79 ±  2%  perf-profile.children.cycles-pp.ext4_evict_inode
      4.89 ±  3%      +0.2        5.08 ±  2%  perf-profile.children.cycles-pp.evict
      4.23 ±  4%      +0.2        4.46        perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      4.49 ±  4%      +0.2        4.72        perf-profile.children.cycles-pp.apic_timer_interrupt
      0.20 ± 18%      -0.0        0.15 ±  3%  perf-profile.self.cycles-pp.mutex_lock
      0.07 ± 12%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.13 ±  6%      -0.0        0.10 ± 10%  perf-profile.self.cycles-pp.jbd2__journal_start
      0.09 ± 17%      -0.0        0.06        perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.09 ±  9%      -0.0        0.06        perf-profile.self.cycles-pp.inode_init_always
      0.11 ±  7%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.find_next_zero_bit
      0.06 ±  6%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.ext4_alloc_inode
      0.13 ± 10%      +0.0        0.15 ±  3%  perf-profile.self.cycles-pp.__virt_addr_valid
      0.13 ± 12%      +0.0        0.16 ±  6%  perf-profile.self.cycles-pp.mark_page_accessed
      0.07 ± 11%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.update_blocked_averages
      0.11 ± 14%      +0.0        0.15 ± 10%  perf-profile.self.cycles-pp.ext4_mark_inode_dirty
      0.04 ± 60%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.update_cfs_rq_h_load
      0.04 ± 59%      +0.0        0.08 ± 12%  perf-profile.self.cycles-pp.__srcu_read_lock
      0.22 ± 10%      +0.0        0.26 ±  7%  perf-profile.self.cycles-pp.strncpy_from_user
      0.16 ± 12%      +0.0        0.20 ±  7%  perf-profile.self.cycles-pp.ktime_get
      0.15 ± 10%      +0.0        0.20 ±  2%  perf-profile.self.cycles-pp.avc_has_perm
      0.11 ± 14%      +0.0        0.15 ± 16%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.32 ±  9%      +0.0        0.37 ±  2%  perf-profile.self.cycles-pp.jbd2_journal_stop
      0.16 ± 14%      +0.0        0.21        perf-profile.self.cycles-pp._raw_spin_trylock
      0.21 ± 12%      +0.0        0.26 ±  3%  perf-profile.self.cycles-pp.__d_lookup
      0.13 ± 17%      +0.0        0.18 ± 16%  perf-profile.self.cycles-pp.__schedule
      0.03 ±100%      +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.__getblk_gfp
      0.24 ±  6%      +0.1        0.30 ±  8%  perf-profile.self.cycles-pp.find_get_entry
      0.10 ± 22%      +0.1        0.15 ± 35%  perf-profile.self.cycles-pp.add_dirent_to_buf
      0.21 ±  8%      +0.1        0.32 ± 15%  perf-profile.self.cycles-pp.ext4_es_lookup_extent
      1.21 ±  4%      +0.2        1.40        perf-profile.self.cycles-pp.crc32c_pcl_intel_update



***************************************************************************************************
lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
  gcc-7/performance/4BRD_12G/xfs/x86_64-rhel-7.6/3000/RAID1/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/disk_wrt/aim7

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :2     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
     63.75            +2.0%      65.00        turbostat.PkgTmp
    280.00 ± 12%     +37.1%     384.00 ±  8%  slabinfo.biovec-64.active_objs
    280.00 ± 12%     +37.1%     384.00 ±  8%  slabinfo.biovec-64.num_objs
     20468            +0.6%      20586        perf-stat.i.context-switches
      2875 ±  4%      +5.8%       3043        perf-stat.overall.instructions-per-iTLB-miss
      0.74            +0.8%       0.74        perf-stat.overall.ipc
     20280            +0.6%      20397        perf-stat.ps.context-switches
    124.21 ± 11%     +21.6%     150.99        sched_debug.cfs_rq:/.load_avg.stddev
     87668 ± 31%     +30.1%     114018 ±  9%  sched_debug.cfs_rq:/.spread0.max
     47.30 ±  9%     -31.3%      32.48 ±  3%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    884.66 ± 20%     +28.0%       1132 ±  7%  sched_debug.cpu.curr->pid.avg
      1213 ± 11%     +20.2%       1458        sched_debug.cpu.curr->pid.stddev
    122.38 ±  5%     +13.8%     139.25        sched_debug.cpu.ttwu_local.min
      1870            -1.5%       1843        proc-vmstat.nr_dirty
      2284            -1.8%       2242        proc-vmstat.nr_inactive_file
      2284            -1.8%       2242        proc-vmstat.nr_zone_inactive_file
      1673 ±  2%      -4.9%       1591        proc-vmstat.nr_zone_write_pending
    562.25 ±158%    +366.7%       2624 ± 19%  proc-vmstat.numa_pages_migrated
      1286 ±130%   +2496.0%      33384        proc-vmstat.numa_pte_updates
    562.25 ±158%    +366.7%       2624 ± 19%  proc-vmstat.pgmigrate_success
     50132 ± 12%     +26.4%      63389 ±  7%  softirqs.CPU1.TIMER
     19663 ± 10%     +12.7%      22151 ±  8%  softirqs.CPU12.RCU
     18448 ±  2%      +8.3%      19976 ±  3%  softirqs.CPU18.RCU
     49885 ±  5%      -8.7%      45564        softirqs.CPU20.TIMER
     18240 ±  4%     +14.6%      20897 ±  8%  softirqs.CPU22.RCU
     18703 ±  2%      +8.1%      20219 ±  4%  softirqs.CPU24.RCU
     16173            +7.7%      17418 ±  4%  softirqs.CPU25.SCHED
    161721 ± 24%     +34.3%     217174 ±  2%  numa-meminfo.node0.Active
    161373 ± 24%     +33.7%     215820 ±  2%  numa-meminfo.node0.Active(anon)
    348.00 ±166%    +289.1%       1354        numa-meminfo.node0.Active(file)
    156218 ± 24%     +33.7%     208918 ±  2%  numa-meminfo.node0.AnonPages
     12329 ± 41%     +35.8%      16737 ± 10%  numa-meminfo.node1.Inactive(anon)
      4700 ± 14%     -24.2%       3561        numa-meminfo.node1.Inactive(file)
     52814 ± 87%     -96.4%       1915 ±  9%  numa-meminfo.node1.PageTables
     82374 ± 46%     -47.9%      42926        numa-meminfo.node1.SUnreclaim
     40389 ± 24%     +33.9%      54061 ±  2%  numa-vmstat.node0.nr_active_anon
     87.00 ±166%    +289.1%     338.50        numa-vmstat.node0.nr_active_file
     39099 ± 24%     +33.9%      52339 ±  2%  numa-vmstat.node0.nr_anon_pages
      1076 ± 16%     +28.0%       1377 ±  2%  numa-vmstat.node0.nr_inactive_file
     40389 ± 24%     +33.9%      54061 ±  2%  numa-vmstat.node0.nr_zone_active_anon
     87.00 ±166%    +289.1%     338.50        numa-vmstat.node0.nr_zone_active_file
      1076 ± 15%     +27.5%       1372 ±  2%  numa-vmstat.node0.nr_zone_inactive_file
     41725 ±129%     +78.0%      74289 ± 82%  numa-vmstat.node0.numa_other
      3083 ± 41%     +35.7%       4184 ± 10%  numa-vmstat.node1.nr_inactive_anon
      1183 ± 12%     -22.4%     918.50 ±  5%  numa-vmstat.node1.nr_inactive_file
     13205 ± 87%     -96.4%     479.00 ±  9%  numa-vmstat.node1.nr_page_table_pages
     20593 ± 46%     -47.9%      10739        numa-vmstat.node1.nr_slab_unreclaimable
      3083 ± 41%     +35.7%       4184 ± 10%  numa-vmstat.node1.nr_zone_inactive_anon
      1181 ± 12%     -22.4%     917.50 ±  4%  numa-vmstat.node1.nr_zone_inactive_file
     67.00 ±  9%     +50.7%     101.00 ± 28%  interrupts.44:IR-PCI-MSI.524294-edge.eth0-TxRx-5
    305.50 ±144%    +435.4%       1635 ± 31%  interrupts.CPU11.NMI:Non-maskable_interrupts
    305.50 ±144%    +435.4%       1635 ± 31%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
    845.00 ± 17%     -27.0%     617.00 ±  7%  interrupts.CPU12.RES:Rescheduling_interrupts
    973.75 ± 17%     -28.4%     697.00 ±  4%  interrupts.CPU14.RES:Rescheduling_interrupts
    706.75 ± 21%     +20.1%     849.00 ± 14%  interrupts.CPU16.RES:Rescheduling_interrupts
      1656 ± 32%     -67.1%     544.50 ± 99%  interrupts.CPU18.NMI:Non-maskable_interrupts
      1656 ± 32%     -67.1%     544.50 ± 99%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
     67.00 ±  9%     +50.7%     101.00 ± 28%  interrupts.CPU2.44:IR-PCI-MSI.524294-edge.eth0-TxRx-5
      1195 ± 12%     +16.7%       1395 ± 12%  interrupts.CPU2.CAL:Function_call_interrupts
     95.75 ±173%   +1649.9%       1675 ± 32%  interrupts.CPU2.NMI:Non-maskable_interrupts
     95.75 ±173%   +1649.9%       1675 ± 32%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      1197 ± 11%     +10.7%       1325 ±  8%  interrupts.CPU20.CAL:Function_call_interrupts
      1633 ± 33%     -69.9%     492.00 ± 92%  interrupts.CPU22.NMI:Non-maskable_interrupts
      1633 ± 33%     -69.9%     492.00 ± 92%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
    860.75 ± 13%     -32.7%     579.50 ± 12%  interrupts.CPU22.RES:Rescheduling_interrupts
      1652 ± 33%     -95.2%      79.00 ±100%  interrupts.CPU24.NMI:Non-maskable_interrupts
      1652 ± 33%     -95.2%      79.00 ±100%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
      1193 ±  9%     +11.6%       1331 ± 10%  interrupts.CPU28.CAL:Function_call_interrupts
    636.25 ± 16%     +30.4%     829.50 ± 10%  interrupts.CPU30.RES:Rescheduling_interrupts
    784.00 ±  8%     +20.5%     944.50        interrupts.CPU31.RES:Rescheduling_interrupts
      1174 ± 12%     +19.5%       1403 ± 13%  interrupts.CPU32.CAL:Function_call_interrupts
      7.75 ±173%  +14248.4%       1112        interrupts.CPU32.NMI:Non-maskable_interrupts
      7.75 ±173%  +14248.4%       1112        interrupts.CPU32.PMI:Performance_monitoring_interrupts
    276.00 ±172%    +211.1%     858.50 ± 95%  interrupts.CPU33.NMI:Non-maskable_interrupts
    276.00 ±172%    +211.1%     858.50 ± 95%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
    280.00 ±166%    +320.4%       1177 ± 85%  interrupts.CPU34.NMI:Non-maskable_interrupts
    280.00 ±166%    +320.4%       1177 ± 85%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
     83.00 ±172%    +608.4%     588.00 ± 80%  interrupts.CPU38.NMI:Non-maskable_interrupts
     83.00 ±172%    +608.4%     588.00 ± 80%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
    572.75 ± 18%     +22.9%     704.00 ±  5%  interrupts.CPU38.RES:Rescheduling_interrupts
    278.25 ±173%    +490.7%       1643 ± 31%  interrupts.CPU4.NMI:Non-maskable_interrupts
    278.25 ±173%    +490.7%       1643 ± 31%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
    663.50 ± 14%     +24.6%     826.50 ± 12%  interrupts.CPU4.RES:Rescheduling_interrupts
      1306 ± 62%     -46.5%     699.00        interrupts.CPU7.RES:Rescheduling_interrupts
      1090 ± 70%     +53.6%       1674 ± 31%  interrupts.CPU9.NMI:Non-maskable_interrupts
      1090 ± 70%     +53.6%       1674 ± 31%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
     42.50            -1.0       41.50        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     42.85            -1.0       41.87        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     36.02            -0.8       35.22        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     34.94            -0.8       34.18        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     30.66            -0.7       29.95        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     28.84            -0.7       28.16        perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     23.88            -0.7       23.21        perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
     23.66            -0.6       23.02        perf-profile.calltrace.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write.vfs_write
      1.48 ±  2%      -0.1        1.39 ±  3%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      0.55 ±  3%      +0.1        0.60 ±  7%  perf-profile.calltrace.cycles-pp.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat.do_filp_open
      0.54 ±  2%      +0.1        0.60 ±  7%  perf-profile.calltrace.cycles-pp.xfs_ialloc.xfs_dir_ialloc.xfs_create.xfs_generic_create.path_openat
      0.72 ±  4%      +0.1        0.79 ±  4%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.72 ±  5%      +0.1        0.80 ±  4%  perf-profile.calltrace.cycles-pp.down_write.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      1.31 ±  2%      +0.1        1.39 ±  3%  perf-profile.calltrace.cycles-pp.xfs_create.xfs_generic_create.path_openat.do_filp_open.do_sys_open
      0.59 ±  2%      +0.1        0.67 ±  4%  perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_slowpath.down_write.do_unlinkat.do_syscall_64
      0.90 ±  8%      +0.1        1.02        perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      2.34 ±  4%      +0.1        2.47        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.unlink
      2.34 ±  4%      +0.1        2.47        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      2.74 ±  2%      +0.1        2.87        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.write
      1.73 ±  2%      +0.1        1.86 ±  4%  perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      2.30 ±  4%      +0.1        2.44        perf-profile.calltrace.cycles-pp.do_unlinkat.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink
      2.39 ±  3%      +0.2        2.54        perf-profile.calltrace.cycles-pp.unlink
      3.35 ±  3%      +0.2        3.59        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.write
     36.04            -0.8       35.23        perf-profile.children.cycles-pp.ksys_write
     52.54            -0.8       51.76        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     52.20            -0.8       51.43        perf-profile.children.cycles-pp.do_syscall_64
     34.98            -0.8       34.22        perf-profile.children.cycles-pp.vfs_write
     30.68            -0.7       29.97        perf-profile.children.cycles-pp.new_sync_write
     28.89            -0.7       28.20        perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
     23.89            -0.7       23.23        perf-profile.children.cycles-pp.iomap_file_buffered_write
     23.71            -0.7       23.05        perf-profile.children.cycles-pp.iomap_apply
      0.37 ±  5%      -0.1        0.28 ±  3%  perf-profile.children.cycles-pp.node_dirty_ok
      0.14 ± 16%      -0.1        0.04 ±100%  perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
      1.51 ±  2%      -0.1        1.43 ±  2%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.56 ±  4%      -0.1        0.48        perf-profile.children.cycles-pp._cond_resched
      0.28 ± 10%      -0.0        0.23 ±  4%  perf-profile.children.cycles-pp.rcu_all_qs
      0.16 ± 13%      -0.0        0.13        perf-profile.children.cycles-pp.__xfs_btree_check_sblock
      0.19 ±  2%      -0.0        0.17 ±  3%  perf-profile.children.cycles-pp.xfs_da3_node_read
      0.15 ± 10%      -0.0        0.12 ±  4%  perf-profile.children.cycles-pp.xfs_get_extsz_hint
      0.10 ±  5%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.orc_find
      0.11 ±  4%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.15 ±  7%      +0.0        0.17        perf-profile.children.cycles-pp.xfs_btree_increment
      0.06 ± 20%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.xfs_btree_read_buf_block
      0.09 ± 20%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.xfs_btree_lookup_get_block
      0.04 ± 58%      +0.0        0.07        perf-profile.children.cycles-pp.xas_find
      0.26 ±  8%      +0.0        0.29        perf-profile.children.cycles-pp.iov_iter_advance
      0.16 ±  5%      +0.0        0.20 ±  2%  perf-profile.children.cycles-pp.xfs_dir_ino_validate
      0.04 ± 58%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__x86_indirect_thunk_r15
      0.14 ±  9%      +0.0        0.18 ±  5%  perf-profile.children.cycles-pp.xfs_buf_item_format
      0.08 ± 19%      +0.0        0.12 ±  8%  perf-profile.children.cycles-pp._xfs_trans_bjoin
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.__inc_numa_state
      0.03 ±100%      +0.0        0.07        perf-profile.children.cycles-pp.xfs_trans_free
      0.06 ± 67%      +0.0        0.10 ± 14%  perf-profile.children.cycles-pp.xfs_find_daxdev_for_inode
      0.01 ±173%      +0.0        0.06 ± 16%  perf-profile.children.cycles-pp.__unwind_start
      0.55 ±  3%      +0.1        0.60 ±  7%  perf-profile.children.cycles-pp.xfs_dir_ialloc
      0.34 ±  6%      +0.1        0.40 ± 10%  perf-profile.children.cycles-pp.__generic_write_end
      0.54 ±  2%      +0.1        0.60 ±  7%  perf-profile.children.cycles-pp.xfs_ialloc
      0.47 ±  3%      +0.1        0.54 ±  7%  perf-profile.children.cycles-pp.xfs_dialloc
      1.31 ±  2%      +0.1        1.39 ±  3%  perf-profile.children.cycles-pp.xfs_create
      1.01 ±  4%      +0.1        1.09 ±  3%  perf-profile.children.cycles-pp.___might_sleep
      0.95 ±  4%      +0.1        1.03 ±  4%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.53 ± 12%      +0.1        0.62 ±  6%  perf-profile.children.cycles-pp.__xfs_dir3_data_check
      0.53 ± 12%      +0.1        0.62 ±  5%  perf-profile.children.cycles-pp.xfs_dir3_data_check
      2.31 ±  4%      +0.1        2.44        perf-profile.children.cycles-pp.do_unlinkat
      2.40 ±  3%      +0.2        2.55        perf-profile.children.cycles-pp.unlink
      3.07            +0.2        3.25        perf-profile.children.cycles-pp.syscall_return_via_sysret
      3.39 ±  3%      +0.2        3.63        perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.99 ±  4%      -0.1        0.86        perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
      0.14 ± 16%      -0.1        0.04 ±100%  perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
      0.18 ± 30%      -0.1        0.09 ± 11%  perf-profile.self.cycles-pp.lock_page_memcg
      0.47 ±  4%      -0.1        0.40 ±  3%  perf-profile.self.cycles-pp.iomap_write_end
      0.25 ±  7%      -0.1        0.19 ±  5%  perf-profile.self.cycles-pp.node_dirty_ok
      0.43 ±  7%      -0.0        0.38        perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.27 ±  4%      -0.0        0.22        perf-profile.self.cycles-pp.xfs_bmbt_to_iomap
      0.11 ±  9%      -0.0        0.08 ± 12%  perf-profile.self.cycles-pp.node_page_state
      0.26            -0.0        0.23 ±  2%  perf-profile.self.cycles-pp.file_update_time
      0.11 ± 12%      -0.0        0.08 ±  5%  perf-profile.self.cycles-pp.xfs_get_extsz_hint
      0.08            -0.0        0.06 ± 16%  perf-profile.self.cycles-pp.xfs_agino_range
      0.08 ±  5%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.mem_cgroup_try_charge
      0.08 ± 10%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.truncate_cleanup_page
      0.09 ± 12%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.xas_create
      0.06 ± 17%      +0.0        0.09 ± 17%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.04 ± 58%      +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.free_unref_page_list
      0.15 ±  5%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.__fdget_pos
      0.07 ± 17%      +0.0        0.10 ± 14%  perf-profile.self.cycles-pp.memcpy_erms
      0.17 ±  4%      +0.0        0.21 ±  2%  perf-profile.self.cycles-pp.__mod_lruvec_state
      0.31 ±  7%      +0.0        0.35        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.27 ±  7%      +0.0        0.31 ±  6%  perf-profile.self.cycles-pp.__add_to_page_cache_locked
      0.26 ± 10%      +0.0        0.30 ±  3%  perf-profile.self.cycles-pp.free_pcppages_bulk
      0.05 ± 74%      +0.0        0.10 ± 19%  perf-profile.self.cycles-pp.xfs_find_daxdev_for_inode
      0.01 ±173%      +0.0        0.06 ± 16%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.xfs_buf_item_init
      0.03 ±100%      +0.1        0.08 ±  6%  perf-profile.self.cycles-pp.__x86_indirect_thunk_r15
      0.19 ± 27%      +0.1        0.24 ±  4%  perf-profile.self.cycles-pp.__sb_end_write
      0.12 ± 16%      +0.1        0.18 ±  2%  perf-profile.self.cycles-pp.mem_cgroup_commit_charge
      0.97 ±  3%      +0.1        1.04 ±  4%  perf-profile.self.cycles-pp.___might_sleep
      0.61 ±  9%      +0.1        0.69 ±  5%  perf-profile.self.cycles-pp.xas_load
      0.51 ±  3%      +0.1        0.59 ±  2%  perf-profile.self.cycles-pp.write
      0.91 ±  3%      +0.1        1.00 ±  4%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.26 ±  9%      +0.1        0.36 ±  6%  perf-profile.self.cycles-pp.__alloc_pages_nodemask
      3.06            +0.2        3.24        perf-profile.self.cycles-pp.syscall_return_via_sysret
      3.03 ±  3%      +0.2        3.25        perf-profile.self.cycles-pp.entry_SYSCALL_64



***************************************************************************************************
lkp-ivb-ep01: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
  gcc-7/performance/4BRD_12G/xfs/x86_64-rhel-7.6/3000/RAID1/debian-x86_64-2019-05-14.cgz/lkp-ivb-ep01/disk_wrt/aim7

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      1304            -3.1%       1263        aim7.time.system_time
      5.18 ±  4%      +0.7        5.91        mpstat.cpu.all.usr%
     69.05            -5.1%      65.53        iostat.cpu.idle
     25.78 ±  3%     +10.8%      28.56        iostat.cpu.system
      5.11 ±  4%     +14.2%       5.83        iostat.cpu.user
      1024 ± 54%     +56.8%       1607 ±  5%  iostat.md0.wkB/s
     68.50            -5.1%      65.00        vmstat.cpu.id
      1024 ± 54%     +56.7%       1604 ±  5%  vmstat.io.bo
      4.00          +500.0%      24.00        vmstat.memory.buff
     17890 ±  3%     +13.5%      20313        vmstat.system.cs
      1101 ±  2%      +7.3%       1181 ±  2%  slabinfo.UNIX.active_objs
      1101 ±  2%      +7.3%       1181 ±  2%  slabinfo.UNIX.num_objs
      4346 ±  4%     +11.5%       4846 ±  2%  slabinfo.kmalloc-1k.active_objs
      4378 ±  4%     +11.0%       4859 ±  2%  slabinfo.kmalloc-1k.num_objs
      1952 ± 18%     +31.1%       2560        slabinfo.skbuff_head_cache.active_objs
      2040 ± 17%     +31.8%       2688        slabinfo.skbuff_head_cache.num_objs
    194684 ±  7%     -49.0%      99382        meminfo.AnonHugePages
   2519569            +9.0%    2747468        meminfo.Committed_AS
   6016512           +25.8%    7570432 ±  6%  meminfo.DirectMap2M
      6647 ±  7%     +11.7%       7424 ±  2%  meminfo.Dirty
      8149 ±  5%     +11.5%       9090 ±  2%  meminfo.Inactive(file)
      1977 ± 11%    -100.0%       0.00        meminfo.Mlocked
     20229 ±  6%     +15.5%      23363        meminfo.max_used_kB
    167735 ± 28%     +20.1%     201529 ± 28%  numa-meminfo.node0.Active(anon)
    106332 ± 16%     -61.9%      40537 ± 27%  numa-meminfo.node0.AnonHugePages
    162893 ± 28%     +22.3%     199190 ± 29%  numa-meminfo.node0.AnonPages
      8638 ± 28%     +59.4%      13773 ± 16%  numa-meminfo.node0.Inactive
      4576 ± 56%    +110.7%       9640 ± 30%  numa-meminfo.node0.Inactive(anon)
      1140 ± 10%    -100.0%       0.00        numa-meminfo.node0.Mlocked
     88219 ± 15%     -33.5%      58632 ± 17%  numa-meminfo.node1.AnonHugePages
      3251 ±  6%     +10.0%       3575 ±  2%  numa-meminfo.node1.Dirty
      4053 ± 18%     +18.0%       4781 ± 14%  numa-meminfo.node1.Inactive(file)
    407.00 ±  3%     +11.2%     452.50        turbostat.Avg_MHz
      0.86 ±112%    +176.5%       2.38 ± 82%  turbostat.CPU%c3
     33.81 ±  3%     +17.8%      39.83        turbostat.CorWatt
     51.75 ±  5%     +23.7%      64.00        turbostat.CoreTmp
     10.75 ± 34%     -57.4%       4.58        turbostat.Pkg%pc2
      0.03 ±173%    +483.3%       0.18 ± 94%  turbostat.Pkg%pc3
      2.94 ± 39%     -90.3%       0.29 ± 96%  turbostat.Pkg%pc6
     52.00 ±  5%     +23.1%      64.00        turbostat.PkgTmp
     59.26           +11.4%      66.03        turbostat.PkgWatt
     10206 ±  3%     -13.0%       8884        turbostat.SMI
    337.75            +1.0%     341.00        proc-vmstat.nr_active_file
     94.25 ±  7%     -49.1%      48.00        proc-vmstat.nr_anon_transparent_hugepages
      1610 ±  5%     +12.8%       1816        proc-vmstat.nr_dirty
      1979 ±  4%     +12.2%       2220        proc-vmstat.nr_inactive_file
     48494 ±  3%     +12.0%      54334        proc-vmstat.nr_kernel_stack
    494.75 ± 11%    -100.0%       0.00        proc-vmstat.nr_mlock
     39652            +5.8%      41942        proc-vmstat.nr_slab_unreclaimable
     28998 ± 58%     +44.2%      41804 ±  4%  proc-vmstat.nr_written
    337.75            +1.0%     341.00        proc-vmstat.nr_zone_active_file
      1978 ±  4%     +12.2%       2220        proc-vmstat.nr_zone_inactive_file
      1419 ±  7%     +11.7%       1586 ±  3%  proc-vmstat.nr_zone_write_pending
     23827 ±  6%     -94.6%       1292 ± 99%  proc-vmstat.numa_pte_updates
    483395            -9.5%     437383        proc-vmstat.pgfault
      1724 ±  2%      -2.9%       1675        proc-vmstat.unevictable_pgs_culled
     40697 ± 28%     +22.3%      49761 ± 29%  numa-vmstat.node0.nr_anon_pages
  19766117 ±  3%     -12.6%   17268001        numa-vmstat.node0.nr_dirtied
    820.75 ±  3%     +13.6%     932.00        numa-vmstat.node0.nr_dirty
      1146 ± 56%    +110.2%       2409 ± 30%  numa-vmstat.node0.nr_inactive_anon
    285.50 ± 10%    -100.0%       0.00        numa-vmstat.node0.nr_mlock
      1146 ± 56%    +110.2%       2409 ± 30%  numa-vmstat.node0.nr_zone_inactive_anon
    732.25 ±  2%     +11.4%     816.00        numa-vmstat.node0.nr_zone_write_pending
  20278495 ±  3%     -12.6%   17727979        numa-vmstat.node0.numa_hit
  20276149 ±  3%     -12.9%   17656262        numa-vmstat.node0.numa_local
      2353 ± 90%   +2947.3%      71725 ± 89%  numa-vmstat.node0.numa_other
  19859884 ±  3%     -13.8%   17124582        numa-vmstat.node1.nr_dirtied
    817.25 ±  4%     +11.3%     909.50        numa-vmstat.node1.nr_dirty
      1006 ± 17%     +19.1%       1198 ± 11%  numa-vmstat.node1.nr_inactive_file
    208.75 ± 11%    -100.0%       0.00        numa-vmstat.node1.nr_mlock
      1006 ± 17%     +19.3%       1201 ± 10%  numa-vmstat.node1.nr_zone_inactive_file
    723.25 ±  3%     +11.1%     803.50        numa-vmstat.node1.nr_zone_write_pending
  20396643 ±  3%     -13.3%   17691336        numa-vmstat.node1.numa_hit
  20225241 ±  3%     -13.0%   17589172        numa-vmstat.node1.numa_local
     52229 ±  7%     -13.5%      45200        interrupts.CAL:Function_call_interrupts
      1037 ± 10%     +61.8%       1678 ±  5%  interrupts.CPU0.RES:Rescheduling_interrupts
      1301 ±  6%     -17.5%       1073        interrupts.CPU10.CAL:Function_call_interrupts
      1289 ±  2%     -15.7%       1087 ±  8%  interrupts.CPU12.CAL:Function_call_interrupts
      1322 ±  7%     -14.5%       1131 ±  4%  interrupts.CPU19.CAL:Function_call_interrupts
     17.00 ±169%   +6250.0%       1079 ±  2%  interrupts.CPU2.NMI:Non-maskable_interrupts
     17.00 ±169%   +6250.0%       1079 ±  2%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      1923 ± 23%     -73.2%     515.00 ± 99%  interrupts.CPU21.NMI:Non-maskable_interrupts
      1923 ± 23%     -73.2%     515.00 ± 99%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
      1350 ±  5%     -27.6%     978.00 ±  6%  interrupts.CPU22.CAL:Function_call_interrupts
      1959 ± 24%     -91.7%     162.50 ± 99%  interrupts.CPU22.NMI:Non-maskable_interrupts
      1959 ± 24%     -91.7%     162.50 ± 99%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
      1046 ± 21%     -41.0%     617.50 ±  2%  interrupts.CPU22.RES:Rescheduling_interrupts
      1341 ±  8%     -18.0%       1100        interrupts.CPU23.CAL:Function_call_interrupts
    880.75 ± 12%     -29.0%     625.50        interrupts.CPU23.RES:Rescheduling_interrupts
      1323 ± 10%     -15.9%       1112        interrupts.CPU25.CAL:Function_call_interrupts
    872.75 ± 12%     -29.5%     615.00 ±  2%  interrupts.CPU25.RES:Rescheduling_interrupts
    347.75 ±133%    +210.7%       1080 ±  5%  interrupts.CPU30.NMI:Non-maskable_interrupts
    347.75 ±133%    +210.7%       1080 ±  5%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
      1316 ± 10%     -15.8%       1108        interrupts.CPU31.CAL:Function_call_interrupts
    721.75 ±  5%     -14.6%     616.50 ±  3%  interrupts.CPU31.RES:Rescheduling_interrupts
      1304 ± 15%     -21.7%       1021 ±  2%  interrupts.CPU36.CAL:Function_call_interrupts
    711.75 ± 13%    +162.6%       1869 ± 54%  interrupts.CPU5.RES:Rescheduling_interrupts
      1331 ±  7%     -20.3%       1061 ±  2%  interrupts.CPU6.CAL:Function_call_interrupts
    802.75 ± 11%     -21.8%     628.00        interrupts.CPU6.RES:Rescheduling_interrupts
      1325 ±  5%     -14.2%       1138 ±  4%  interrupts.CPU7.CAL:Function_call_interrupts
      1314 ±  6%     -15.8%       1106        interrupts.CPU9.CAL:Function_call_interrupts
      9961           +11.7%      11124        sched_debug.cfs_rq:/.exec_clock.max
    162.13 ± 11%    +102.1%     327.63 ±  7%  sched_debug.cfs_rq:/.exec_clock.stddev
    237563 ±  5%      +6.4%     252874 ±  3%  sched_debug.cfs_rq:/.min_vruntime.max
      0.39 ±  7%     -14.5%       0.33        sched_debug.cfs_rq:/.nr_running.avg
     12.54          +148.2%      31.13 ± 21%  sched_debug.cfs_rq:/.removed.load_avg.avg
     78.34           +52.8%     119.68 ± 11%  sched_debug.cfs_rq:/.removed.load_avg.stddev
    577.48          +149.3%       1439 ± 21%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
      3606           +53.4%       5533 ± 11%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
      3.67 ± 28%    +203.5%      11.12 ± 16%  sched_debug.cfs_rq:/.removed.util_avg.avg
    146.62 ± 28%     +43.4%     210.25 ±  9%  sched_debug.cfs_rq:/.removed.util_avg.max
     22.89 ± 28%     +91.2%      43.76 ±  4%  sched_debug.cfs_rq:/.removed.util_avg.stddev
     10.87 ±  9%     +69.8%      18.47 ± 39%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
     39019          +663.6%     297940 ± 86%  sched_debug.cfs_rq:/.runnable_weight.max
     17929          +218.3%      57063 ± 68%  sched_debug.cfs_rq:/.runnable_weight.stddev
     60838 ± 26%     +49.2%      90778 ± 11%  sched_debug.cfs_rq:/.spread0.max
    371.50 ± 21%     +31.5%     488.50 ± 10%  sched_debug.cfs_rq:/.util_est_enqueued.max
    477488 ± 75%     -84.5%      74077 ±  2%  sched_debug.cpu.clock.avg
    477508 ± 75%     -84.5%      74095 ±  2%  sched_debug.cpu.clock.max
    477467 ± 75%     -84.5%      74058 ±  2%  sched_debug.cpu.clock.min
    477488 ± 75%     -84.5%      74077 ±  2%  sched_debug.cpu.clock_task.avg
    477508 ± 75%     -84.5%      74095 ±  2%  sched_debug.cpu.clock_task.max
    477467 ± 75%     -84.5%      74058 ±  2%  sched_debug.cpu.clock_task.min
      7.81 ±  8%     +19.0%       9.29 ±  5%  sched_debug.cpu.cpu_load[0].avg
      8.75 ±  7%     +18.3%      10.34 ±  6%  sched_debug.cpu.cpu_load[1].avg
      9.42 ±  7%     +18.1%      11.12 ±  2%  sched_debug.cpu.cpu_load[2].avg
     43.38 ± 11%     +91.4%      83.00 ± 43%  sched_debug.cpu.cpu_load[2].max
      8.80 ±  9%     +67.0%      14.70 ± 35%  sched_debug.cpu.cpu_load[2].stddev
      9.72 ±  9%     +23.6%      12.01        sched_debug.cpu.cpu_load[3].avg
     46.62 ± 18%     +71.6%      80.00 ± 19%  sched_debug.cpu.cpu_load[3].max
      8.60 ± 14%     +69.9%      14.61 ± 17%  sched_debug.cpu.cpu_load[3].stddev
      8.47 ± 10%     +61.0%      13.64 ±  7%  sched_debug.cpu.cpu_load[4].avg
     41.12 ± 15%    +336.5%     179.50 ± 12%  sched_debug.cpu.cpu_load[4].max
      8.28 ± 13%    +248.2%      28.83 ± 12%  sched_debug.cpu.cpu_load[4].stddev
     42318 ±  9%     -11.5%      37470        sched_debug.cpu.nr_load_updates.avg
     39974 ±  9%     -13.0%      34783        sched_debug.cpu.nr_load_updates.min
      0.38 ±  8%     -16.4%       0.32        sched_debug.cpu.nr_running.avg
     15.56 ±  3%     +15.8%      18.02 ±  4%  sched_debug.cpu.nr_uninterruptible.stddev
      8499            +6.5%       9050 ±  4%  sched_debug.cpu.ttwu_count.max
    455.17 ±  8%     +14.3%     520.19 ±  5%  sched_debug.cpu.ttwu_count.stddev
    121.12 ±  3%      +7.3%     130.00 ±  4%  sched_debug.cpu.ttwu_local.min
    477466 ± 75%     -84.5%      74058 ±  2%  sched_debug.cpu_clk
    473896 ± 75%     -85.1%      70485 ±  2%  sched_debug.ktime
    477908 ± 74%     -84.4%      74502 ±  2%  sched_debug.sched_clk
     18.97 ± 14%     -39.7%      11.43 ±  8%  perf-stat.i.MPKI
 2.325e+09 ±  3%     +14.8%  2.669e+09        perf-stat.i.branch-instructions
      3.65 ±  9%      -1.2        2.47 ±  3%  perf-stat.i.branch-miss-rate%
  40992858 ±  4%     +17.0%   47962008        perf-stat.i.branch-misses
      9.45 ± 14%      -2.3        7.17        perf-stat.i.cache-miss-rate%
   5752408 ±  2%      +7.9%    6205826 ±  3%  perf-stat.i.cache-misses
  99449480 ±  4%     +12.5%  1.119e+08 ±  2%  perf-stat.i.cache-references
     18273 ±  3%     +12.5%      20561        perf-stat.i.context-switches
      2.48 ±  8%     -31.7%       1.70 ±  3%  perf-stat.i.cpi
 1.629e+10 ±  4%     +11.1%  1.809e+10        perf-stat.i.cpu-cycles
    945.35 ±  4%      +9.1%       1031        perf-stat.i.cpu-migrations
      2944 ±  3%      +4.2%       3067        perf-stat.i.cycles-between-cache-misses
 3.591e+09 ±  3%     +14.3%  4.102e+09        perf-stat.i.dTLB-loads
   7154623 ±  8%     +13.3%    8105429        perf-stat.i.dTLB-store-misses
 2.473e+09 ±  3%     +14.4%   2.83e+09        perf-stat.i.dTLB-stores
     82.21 ±  2%      -3.2       79.06        perf-stat.i.iTLB-load-miss-rate%
   1050734 ± 12%     +15.3%    1210985        perf-stat.i.iTLB-loads
 1.172e+10 ±  3%     +14.8%  1.345e+10        perf-stat.i.instructions
      2449           +27.2%       3114        perf-stat.i.instructions-per-iTLB-miss
      0.62 ±  2%     +14.3%       0.71        perf-stat.i.ipc
   1295155 ±  7%     +17.5%    1522168 ±  3%  perf-stat.i.node-load-misses
   1692798 ±  5%     +17.0%    1980114        perf-stat.i.node-loads
    761603 ±  8%     +15.8%     882204        perf-stat.i.node-store-misses
   1197239 ±  9%     +17.3%    1403801        perf-stat.i.node-stores
      1.76            +0.0        1.80        perf-stat.overall.branch-miss-rate%
      1.39            -3.2%       1.35        perf-stat.overall.cpi
     79.96            -1.9       78.04        perf-stat.overall.iTLB-load-miss-rate%
      2811 ±  2%     +11.2%       3125        perf-stat.overall.instructions-per-iTLB-miss
      0.72            +3.3%       0.74        perf-stat.overall.ipc
 2.306e+09 ±  3%     +14.7%  2.645e+09        perf-stat.ps.branch-instructions
  40663464 ±  4%     +16.9%   47517919        perf-stat.ps.branch-misses
   5707130 ±  2%      +7.7%    6149393 ±  3%  perf-stat.ps.cache-misses
  98651423 ±  4%     +12.4%  1.109e+08 ±  2%  perf-stat.ps.cache-references
     18126 ±  3%     +12.4%      20371        perf-stat.ps.context-switches
 1.616e+10 ±  4%     +10.9%  1.793e+10        perf-stat.ps.cpu-cycles
    937.88 ±  4%      +9.0%       1022        perf-stat.ps.cpu-migrations
 3.562e+09 ±  3%     +14.1%  4.064e+09        perf-stat.ps.dTLB-loads
   7096976 ±  8%     +13.2%    8030377        perf-stat.ps.dTLB-store-misses
 2.454e+09 ±  3%     +14.3%  2.804e+09        perf-stat.ps.dTLB-stores
   1042256 ± 12%     +15.1%    1199770        perf-stat.ps.iTLB-loads
 1.162e+10 ±  3%     +14.6%  1.332e+10        perf-stat.ps.instructions
   1284746 ±  7%     +17.4%    1508066 ±  3%  perf-stat.ps.node-load-misses
   1679220 ±  5%     +16.8%    1961768        perf-stat.ps.node-loads
    755495 ±  8%     +15.7%     874037        perf-stat.ps.node-store-misses
   1187940 ±  9%     +17.1%    1390847        perf-stat.ps.node-stores
     54528 ±  2%     -22.6%      42229 ±  6%  softirqs.CPU0.TIMER
     19067 ±  4%     -10.6%      17047        softirqs.CPU1.SCHED
     53953           -17.8%      44323 ±  7%  softirqs.CPU1.TIMER
     18615 ±  5%      -9.4%      16866        softirqs.CPU10.SCHED
     18576 ±  3%      -7.5%      17176        softirqs.CPU11.SCHED
     49349 ±  6%     -15.8%      41544 ±  7%  softirqs.CPU11.TIMER
     18891 ±  5%     -10.9%      16837 ±  2%  softirqs.CPU12.SCHED
     57456 ± 14%     -28.2%      41276 ±  4%  softirqs.CPU12.TIMER
     18784 ±  2%     -10.1%      16884        softirqs.CPU13.SCHED
     49424 ±  5%     -16.5%      41289 ±  9%  softirqs.CPU13.TIMER
     19223 ±  2%     -11.2%      17070        softirqs.CPU14.SCHED
     51678 ±  8%     -19.0%      41856 ±  6%  softirqs.CPU14.TIMER
     19010 ±  2%      -9.7%      17160        softirqs.CPU15.SCHED
     19306 ±  2%     -12.1%      16971        softirqs.CPU16.SCHED
     52068 ±  8%     -20.7%      41291 ±  7%  softirqs.CPU16.TIMER
     19982 ±  7%      +7.6%      21503 ±  5%  softirqs.CPU17.RCU
     18737 ±  3%      -8.4%      17166        softirqs.CPU17.SCHED
     19111 ±  3%     -11.8%      16858 ±  2%  softirqs.CPU18.SCHED
     18896 ±  4%      -9.2%      17159        softirqs.CPU19.SCHED
     52848 ±  5%     -21.2%      41638 ±  5%  softirqs.CPU19.TIMER
     19445 ±  2%     -11.8%      17151        softirqs.CPU2.SCHED
     53871 ± 11%     -23.9%      40994 ±  7%  softirqs.CPU2.TIMER
     52506 ± 12%     -21.1%      41435 ±  6%  softirqs.CPU20.TIMER
     20471 ±  6%      +7.7%      22044 ±  2%  softirqs.CPU21.RCU
     18435 ±  4%     -10.0%      16590        softirqs.CPU22.SCHED
     49458 ±  3%     -18.2%      40465 ±  4%  softirqs.CPU22.TIMER
     20835 ±  5%     +15.4%      24044 ±  8%  softirqs.CPU23.RCU
     18719 ±  4%      -9.2%      16999        softirqs.CPU24.SCHED
     49855 ±  4%     -16.3%      41723 ±  6%  softirqs.CPU24.TIMER
     49582 ±  5%     -15.7%      41774 ±  5%  softirqs.CPU25.TIMER
     18825 ±  4%      -9.7%      16996        softirqs.CPU26.SCHED
     50142 ±  4%     -17.3%      41468 ±  6%  softirqs.CPU26.TIMER
     49521 ±  6%     -16.6%      41297 ±  6%  softirqs.CPU27.TIMER
     19061 ±  4%     -11.3%      16913        softirqs.CPU28.SCHED
     56138 ± 16%     -26.7%      41134 ±  7%  softirqs.CPU28.TIMER
     49305 ±  7%     -16.3%      41278 ±  7%  softirqs.CPU29.TIMER
     50274 ±  6%     -17.8%      41305 ±  5%  softirqs.CPU3.TIMER
     19289 ±  5%     -13.7%      16652        softirqs.CPU30.SCHED
     19318 ±  7%     -12.3%      16948        softirqs.CPU32.SCHED
     18563 ±  3%      -8.1%      17055        softirqs.CPU33.SCHED
     19555 ±  3%     -13.5%      16910        softirqs.CPU34.SCHED
     54208 ±  9%     -23.6%      41413 ±  7%  softirqs.CPU34.TIMER
     18777 ±  3%      -9.6%      16972        softirqs.CPU35.SCHED
     18985 ±  3%     -10.1%      17066        softirqs.CPU36.SCHED
     52365 ±  9%     -19.2%      42299 ±  8%  softirqs.CPU36.TIMER
     18686 ±  4%      -8.9%      17030        softirqs.CPU37.SCHED
     51287 ±  5%     -17.9%      42102 ±  7%  softirqs.CPU37.TIMER
     19511 ±  2%     -14.0%      16783        softirqs.CPU38.SCHED
     58324 ± 11%     -29.9%      40875 ±  8%  softirqs.CPU38.TIMER
     18670 ±  3%      -8.6%      17072        softirqs.CPU39.SCHED
     52113 ±  4%     -19.7%      41828 ±  7%  softirqs.CPU39.TIMER
     49903 ±  5%     -16.3%      41768 ±  8%  softirqs.CPU4.TIMER
     49614 ±  6%     -16.3%      41528 ±  5%  softirqs.CPU5.TIMER
     18943 ±  6%      -9.5%      17144        softirqs.CPU6.SCHED
     49359 ±  5%     -15.6%      41661 ±  7%  softirqs.CPU6.TIMER
     49260 ±  6%     -14.8%      41974 ±  4%  softirqs.CPU7.TIMER
     19050 ±  5%     -10.9%      16982        softirqs.CPU8.SCHED
     18876 ±  3%      -8.1%      17339        softirqs.CPU9.SCHED
    760956 ±  3%      -9.5%     688784        softirqs.SCHED
   2064576 ±  3%     -17.5%    1704059 ±  6%  softirqs.TIMER
      9.99            -0.8        9.21 ±  3%  perf-profile.calltrace.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      7.40            -0.7        6.73 ±  4%  perf-profile.calltrace.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
      7.11            -0.7        6.45 ±  3%  perf-profile.calltrace.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
      1.78 ±  3%      -0.3        1.44 ±  5%  perf-profile.calltrace.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
      2.94 ±  2%      -0.3        2.60 ±  4%  perf-profile.calltrace.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      0.55            -0.3        0.27 ±100%  perf-profile.calltrace.cycles-pp.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_fs_destroy_inode.destroy_inode
      0.87 ± 16%      -0.2        0.62 ±  2%  perf-profile.calltrace.cycles-pp.__sb_start_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.50 ±  2%      -0.2        1.34 ±  2%  perf-profile.calltrace.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
      0.82 ±  5%      -0.1        0.74        perf-profile.calltrace.cycles-pp.xfs_file_iomap_end.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.new_sync_write
      0.60            +0.0        0.61        perf-profile.calltrace.cycles-pp.rwsem_spin_on_owner.rwsem_down_write_slowpath.down_write.do_unlinkat.do_syscall_64
      0.44 ± 59%      +0.2        0.68 ± 10%  perf-profile.calltrace.cycles-pp.xfs_iext_lookup_extent.xfs_file_iomap_begin_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write
      0.14 ±173%      +0.4        0.58 ± 13%  perf-profile.calltrace.cycles-pp.iov_iter_fault_in_readable.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
      1.12 ±  8%      +0.5        1.64 ±  3%  perf-profile.calltrace.cycles-pp.xfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     10.03            -0.8        9.25 ±  3%  perf-profile.children.cycles-pp.iomap_write_begin
      7.43            -0.7        6.76 ±  4%  perf-profile.children.cycles-pp.grab_cache_page_write_begin
      7.14            -0.7        6.48 ±  4%  perf-profile.children.cycles-pp.pagecache_get_page
      1.82 ±  3%      -0.4        1.46 ±  5%  perf-profile.children.cycles-pp.__add_to_page_cache_locked
      2.95 ±  2%      -0.3        2.60 ±  4%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      0.95 ± 14%      -0.3        0.68 ±  5%  perf-profile.children.cycles-pp.__sb_start_write
      0.97 ±  3%      -0.2        0.78 ±  9%  perf-profile.children.cycles-pp.xas_load
      1.52            -0.2        1.36 ±  2%  perf-profile.children.cycles-pp.find_get_entry
      0.93 ±  5%      -0.1        0.80 ±  4%  perf-profile.children.cycles-pp.xfs_iunlock
      0.43 ±  4%      -0.1        0.33 ±  9%  perf-profile.children.cycles-pp.xfs_errortag_test
      0.67 ±  2%      -0.1        0.56 ±  3%  perf-profile.children.cycles-pp.up_write
      0.38 ±  8%      -0.1        0.29 ±  6%  perf-profile.children.cycles-pp.xas_start
      0.36 ± 14%      -0.1        0.27 ±  7%  perf-profile.children.cycles-pp.node_dirty_ok
      0.86 ±  6%      -0.1        0.78        perf-profile.children.cycles-pp.xfs_file_iomap_end
      1.06 ±  3%      -0.1        0.97        perf-profile.children.cycles-pp.___might_sleep
      0.49 ±  4%      -0.1        0.41        perf-profile.children.cycles-pp.xfs_trans_read_buf_map
      0.45 ± 16%      -0.1        0.39        perf-profile.children.cycles-pp.scheduler_tick
      0.36 ± 15%      -0.1        0.30 ±  5%  perf-profile.children.cycles-pp.xfs_trans_alloc
      0.21 ±  5%      -0.1        0.15 ± 10%  perf-profile.children.cycles-pp.workingset_update_node
      0.35 ±  4%      -0.1        0.29        perf-profile.children.cycles-pp.xfs_buf_read_map
      0.23 ± 16%      -0.1        0.17 ±  8%  perf-profile.children.cycles-pp.xfs_log_reserve
      0.19 ±  9%      -0.1        0.13        perf-profile.children.cycles-pp.lock_page_memcg
      0.55            -0.0        0.50 ±  6%  perf-profile.children.cycles-pp.xfs_ifree
      0.08 ±  5%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.33 ±  3%      -0.0        0.28 ±  3%  perf-profile.children.cycles-pp.xfs_buf_get_map
      0.07 ± 10%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.24 ± 20%      -0.0        0.20        perf-profile.children.cycles-pp.task_tick_fair
      0.31            -0.0        0.27        perf-profile.children.cycles-pp.xfs_buf_find
      0.27 ±  4%      -0.0        0.23 ±  4%  perf-profile.children.cycles-pp.xfs_read_agi
      0.12 ± 15%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.xas_create
      0.09 ± 16%      -0.0        0.06 ±  9%  perf-profile.children.cycles-pp.memcg_check_events
      0.14 ± 12%      -0.0        0.11        perf-profile.children.cycles-pp.update_load_avg
      0.08 ±  8%      -0.0        0.06 ±  9%  perf-profile.children.cycles-pp.__x86_indirect_thunk_r10
      0.35 ±  3%      -0.0        0.33        perf-profile.children.cycles-pp.xfs_dir_createname
      0.08 ± 11%      -0.0        0.06 ±  9%  perf-profile.children.cycles-pp.xfs_btree_read_buf_block
      0.07 ±  5%      -0.0        0.06        perf-profile.children.cycles-pp.xfs_dir3_data_get_ftype
      0.03 ±100%      +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.xfs_perag_get
      0.04 ± 60%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.uncharge_page
      0.09 ± 37%      +0.0        0.13        perf-profile.children.cycles-pp.ret_from_fork
      0.08 ± 35%      +0.0        0.13        perf-profile.children.cycles-pp.kthread
      0.49 ±  7%      +0.0        0.54 ±  2%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.__generic_file_write_iter
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.free_unref_page_prepare
      0.22 ± 12%      +0.1        0.28 ±  3%  perf-profile.children.cycles-pp.xfs_bmapi_reserve_delalloc
      1.29 ±  6%      +0.1        1.36 ±  5%  perf-profile.children.cycles-pp.iomap_set_range_uptodate
      0.34 ±  4%      +0.1        0.42        perf-profile.children.cycles-pp.__fsnotify_parent
      0.11 ± 10%      +0.1        0.19 ±  5%  perf-profile.children.cycles-pp.delay_tsc
      0.64 ±  9%      +0.1        0.72        perf-profile.children.cycles-pp._raw_spin_lock
      0.28 ± 19%      +0.1        0.37        perf-profile.children.cycles-pp.uart_console_write
      0.49 ±  6%      +0.1        0.58 ± 13%  perf-profile.children.cycles-pp.iov_iter_fault_in_readable
      0.28 ± 19%      +0.1        0.37        perf-profile.children.cycles-pp.serial8250_console_putchar
      0.58 ± 12%      +0.1        0.71 ±  8%  perf-profile.children.cycles-pp.xfs_iext_lookup_extent
      1.13 ±  8%      +0.5        1.64 ±  3%  perf-profile.children.cycles-pp.xfs_file_write_iter
      0.99 ±  3%      -0.2        0.83 ±  4%  perf-profile.self.cycles-pp.xfs_file_buffered_aio_write
      0.40 ± 14%      -0.1        0.26 ±  3%  perf-profile.self.cycles-pp.security_file_permission
      0.59 ±  3%      -0.1        0.48 ± 12%  perf-profile.self.cycles-pp.xas_load
      0.40 ±  6%      -0.1        0.29 ±  8%  perf-profile.self.cycles-pp.xfs_errortag_test
      0.60 ±  3%      -0.1        0.51        perf-profile.self.cycles-pp.up_write
      1.02 ±  2%      -0.1        0.94        perf-profile.self.cycles-pp.___might_sleep
      0.31 ± 12%      -0.1        0.23        perf-profile.self.cycles-pp.__add_to_page_cache_locked
      0.30 ±  4%      -0.1        0.22 ±  6%  perf-profile.self.cycles-pp.file_update_time
      0.19 ±  6%      -0.1        0.12 ±  8%  perf-profile.self.cycles-pp.workingset_update_node
      0.48 ±  3%      -0.1        0.42 ±  6%  perf-profile.self.cycles-pp.xfs_file_iomap_begin
      0.29 ±  8%      -0.1        0.23 ±  4%  perf-profile.self.cycles-pp.xfs_fs_dirty_inode
      0.17 ± 12%      -0.1        0.11 ±  4%  perf-profile.self.cycles-pp.lock_page_memcg
      0.07 ± 11%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.14 ±  7%      -0.0        0.10 ± 15%  perf-profile.self.cycles-pp.mem_cgroup_commit_charge
      0.10 ± 18%      -0.0        0.06        perf-profile.self.cycles-pp.xas_create
      0.15 ±  7%      -0.0        0.12 ±  8%  perf-profile.self.cycles-pp.__fdget_pos
      0.09 ± 21%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.mem_cgroup_try_charge
      0.07 ± 12%      -0.0        0.05        perf-profile.self.cycles-pp.__percpu_counter_sum
      0.07            -0.0        0.06        perf-profile.self.cycles-pp.xfs_dir3_data_get_ftype
      0.06 ± 17%      +0.0        0.08 ± 12%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.36 ±  4%      +0.0        0.39 ±  2%  perf-profile.self.cycles-pp.xas_store
      0.05 ± 58%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.xfs_agino_range
      0.47 ±  3%      +0.0        0.51        perf-profile.self.cycles-pp.__list_del_entry_valid
      0.25 ± 11%      +0.0        0.29 ±  6%  perf-profile.self.cycles-pp.__alloc_pages_nodemask
      0.04 ± 58%      +0.0        0.08 ± 12%  perf-profile.self.cycles-pp.uncharge_page
      0.42 ±  8%      +0.0        0.46        perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.41 ±  4%      +0.0        0.46        perf-profile.self.cycles-pp.__mark_inode_dirty
      0.32 ±  5%      +0.0        0.36 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.11 ± 10%      +0.0        0.15 ±  6%  perf-profile.self.cycles-pp.delay_tsc
      0.08 ± 23%      +0.1        0.13 ±  7%  perf-profile.self.cycles-pp.__lru_cache_add
      0.47 ±  6%      +0.1        0.53 ±  4%  perf-profile.self.cycles-pp.iov_iter_fault_in_readable
      0.41 ±  3%      +0.1        0.47 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock
      0.31 ± 10%      +0.1        0.38 ±  5%  perf-profile.self.cycles-pp.new_sync_write
      0.32 ±  4%      +0.1        0.40 ±  3%  perf-profile.self.cycles-pp.__fsnotify_parent
      0.57 ± 13%      +0.1        0.66 ±  6%  perf-profile.self.cycles-pp.xfs_iext_lookup_extent
      0.55 ±  8%      +0.1        0.65 ±  5%  perf-profile.self.cycles-pp.iomap_write_actor
      1.08 ±  9%      +0.5        1.56 ±  2%  perf-profile.self.cycles-pp.xfs_file_write_iter



***************************************************************************************************
lkp-hsw-4ex1: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory
=========================================================================================
compiler/cpufreq_governor/iterations/kconfig/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/20/x86_64-rhel-7.6/1600%/debian-x86_64-2019-05-14.cgz/lkp-hsw-4ex1/aim9/reaim/0x14

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :2     dmesg.WARNING:at#for_ip_interrupt_entry/0x
          0:4           -4%           0:2     perf-profile.children.cycles-pp.error_entry
          0:4           -3%           0:2     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
      1589            -1.6%       1564        reaim.child_utime
     92.73            +3.1%      95.62        reaim.jti
      6.68 ± 20%     -39.9%       4.02 ±  4%  reaim.std_dev_percent
      2.29 ± 25%     -37.7%       1.43 ±  2%  reaim.std_dev_time
   9735811 ±  5%     -20.4%    7751707        reaim.time.involuntary_context_switches
      4456            +2.0%       4544        reaim.time.maximum_resident_set_size
 9.237e+08            +1.4%  9.371e+08        reaim.time.minor_page_faults
     31822            -1.6%      31324        reaim.time.user_time
  39972125 ±  3%      +7.1%   42816711        reaim.time.voluntary_context_switches
      5363 ±  9%     -12.5%       4693        boot-time.idle
     18335 ± 68%     -98.1%     355.00        numa-numastat.node0.other_node
    866.09 ±  3%      +4.9%     908.47        pmeter.Average_Active_Power
    374417 ±  9%     -25.0%     280892 ±  5%  meminfo.DirectMap4k
    244169 ±  3%      +9.3%     266898 ±  2%  meminfo.PageTables
      0.08 ± 16%      -0.0        0.04 ±  2%  mpstat.cpu.all.iowait%
      0.02 ± 10%      +0.0        0.02 ±  8%  mpstat.cpu.all.soft%
      2.67 ± 17%     +68.8%       4.50 ± 11%  vmstat.procs.b
    102516 ±  5%     +15.0%     117851        vmstat.system.cs
    306586 ±  9%     -14.3%     262885 ±  2%  softirqs.CPU133.TIMER
    301177 ± 10%     -14.4%     257932 ±  2%  softirqs.CPU134.TIMER
    348974 ± 20%     -25.6%     259770 ±  2%  softirqs.CPU61.TIMER
    299486 ±  9%     -15.4%     253362        softirqs.CPU64.TIMER
     80298 ±  3%      +9.2%      87708 ±  3%  softirqs.CPU94.NET_RX
     27.45 ±  4%      -5.2       22.24 ± 11%  turbostat.C6%
      0.85 ±107%    +292.7%       3.32 ± 90%  turbostat.CPU%c3
     16.26 ±  4%     -43.9%       9.12 ± 34%  turbostat.CPU%c6
     52.33 ±  4%     +11.8%      58.50        turbostat.CoreTmp
     11.62 ±  7%     -30.6%       8.06 ±  2%  turbostat.Pkg%pc2
      0.59 ± 17%     -61.0%       0.23 ±100%  turbostat.Pkg%pc6
     56.67 ±  3%      +8.5%      61.50        turbostat.PkgTmp
    496.32 ±  3%      +6.4%     528.04        turbostat.PkgWatt
     72.66            +4.6%      76.01        turbostat.RAMWatt
     10083 ±  3%     +50.5%      15171 ± 28%  numa-vmstat.node0.nr_slab_reclaimable
     19593 ± 64%     -93.1%       1353 ± 30%  numa-vmstat.node0.numa_other
    126905 ±  7%     -14.4%     108652        numa-vmstat.node1.nr_file_pages
     22060 ± 13%     +42.3%      31389 ± 26%  numa-vmstat.node1.nr_kernel_stack
     11946 ± 13%     +48.1%      17697 ± 25%  numa-vmstat.node1.nr_page_table_pages
     62212 ±  8%     -32.9%      41751        numa-vmstat.node1.nr_shmem
     51065 ± 15%     -18.8%      41449 ±  4%  numa-vmstat.node2.nr_shmem
    152702 ± 12%     +20.5%     184056 ±  6%  numa-vmstat.node3.nr_active_anon
     95312 ± 15%     +34.3%     128000        numa-vmstat.node3.nr_anon_pages
     47.00 ± 51%    +124.5%     105.50 ± 22%  numa-vmstat.node3.nr_dirtied
     47.00 ± 51%    +124.5%     105.50 ± 22%  numa-vmstat.node3.nr_written
    152702 ± 12%     +20.5%     184051 ±  6%  numa-vmstat.node3.nr_zone_active_anon
  17552894        +1.7e+06    19268273        syscalls.sys_close.noise.2%
  15759501 ±  2%  +1.9e+06    17627007        syscalls.sys_close.noise.25%
  17442873        +1.7e+06    19155505        syscalls.sys_close.noise.5%
  13169085 ±  2%  +1.9e+06    15084330        syscalls.sys_close.noise.50%
  10857762 ±  5%  +1.1e+06    11964287 ±  3%  syscalls.sys_close.noise.75%
 1.219e+09 ± 93%     -93.4%   80431405 ± 21%  syscalls.sys_read.max
 2.569e+09 ±123%  -2.3e+09   2.578e+08 ± 10%  syscalls.sys_read.noise.100%
 2.596e+09 ±122%  -2.3e+09    2.88e+08 ±  9%  syscalls.sys_read.noise.2%
 2.591e+09 ±122%  -2.3e+09   2.817e+08 ± 10%  syscalls.sys_read.noise.25%
 2.596e+09 ±122%  -2.3e+09   2.878e+08 ±  9%  syscalls.sys_read.noise.5%
 2.581e+09 ±123%  -2.3e+09    2.71e+08 ± 10%  syscalls.sys_read.noise.50%
 2.574e+09 ±123%  -2.3e+09   2.622e+08 ± 10%  syscalls.sys_read.noise.75%
    399915 ±  3%      +9.9%     439446        proc-vmstat.nr_anon_pages
    176.67 ±  3%      +6.1%     187.50        proc-vmstat.nr_dirtied
    487035 ±  5%      -6.0%     457770        proc-vmstat.nr_file_pages
    110310 ±  3%      +7.8%     118870        proc-vmstat.nr_kernel_stack
      6819 ±  2%      +3.8%       7080        proc-vmstat.nr_mapped
     60748 ±  3%      +9.9%      66752 ±  2%  proc-vmstat.nr_page_table_pages
    231076 ± 11%     -12.5%     202112        proc-vmstat.nr_shmem
    118658            +2.4%     121531        proc-vmstat.nr_slab_unreclaimable
    175.33 ±  3%      +7.2%     188.00        proc-vmstat.nr_written
    226291 ± 21%     -32.6%     152628 ±  3%  proc-vmstat.numa_hint_faults
    175326 ± 20%     -30.7%     121440        proc-vmstat.numa_pages_migrated
 9.266e+08            +1.4%  9.397e+08        proc-vmstat.pgfault
    175326 ± 20%     -30.7%     121440        proc-vmstat.pgmigrate_success
     40337 ±  3%     +50.6%      60759 ± 28%  numa-meminfo.node0.KReclaimable
      8745 ± 28%     +26.4%      11057 ±  3%  numa-meminfo.node0.Mapped
     40337 ±  3%     +50.6%      60759 ± 28%  numa-meminfo.node0.SReclaimable
     31869 ± 79%    -100.0%       0.00        numa-meminfo.node1.AnonHugePages
    508265 ±  7%     -14.6%     434111        numa-meminfo.node1.FilePages
     22056 ± 13%     +41.2%      31138 ± 25%  numa-meminfo.node1.KernelStack
     47686 ± 13%     +47.1%      70141 ± 25%  numa-meminfo.node1.PageTables
    249498 ±  8%     -33.3%     166508        numa-meminfo.node1.Shmem
    204458 ± 13%     -19.1%     165313 ±  5%  numa-meminfo.node2.Shmem
    613353 ± 12%     +19.6%     733539 ±  6%  numa-meminfo.node3.Active
    613302 ± 12%     +19.6%     733387 ±  6%  numa-meminfo.node3.Active(anon)
     48847 ± 73%    +189.7%     141490 ±  2%  numa-meminfo.node3.AnonHugePages
    382035 ± 15%     +33.5%     510087        numa-meminfo.node3.AnonPages
      5411            +7.5%       5817 ±  4%  numa-meminfo.node3.Mapped
    171762 ± 27%     +41.6%     243282 ±  2%  slabinfo.Acpi-Parse.active_objs
      2382 ± 27%     +41.9%       3380 ±  2%  slabinfo.Acpi-Parse.active_slabs
    173961 ± 27%     +41.9%     246813 ±  2%  slabinfo.Acpi-Parse.num_objs
      2382 ± 27%     +41.9%       3380 ±  2%  slabinfo.Acpi-Parse.num_slabs
    987.67 ±  2%     +19.1%       1176        slabinfo.biovec-128.active_objs
    987.67 ±  2%     +19.1%       1176        slabinfo.biovec-128.num_objs
      1358 ±  8%     -14.9%       1155        slabinfo.kmalloc-rcl-96.active_objs
      1358 ±  8%     -14.9%       1155        slabinfo.kmalloc-rcl-96.num_objs
    191139 ±  8%     -28.7%     136191 ±  6%  slabinfo.lsm_file_cache.active_objs
    746.00 ±  8%     -28.8%     531.50 ±  6%  slabinfo.lsm_file_cache.active_slabs
    191139 ±  8%     -28.7%     136191 ±  6%  slabinfo.lsm_file_cache.num_objs
    746.00 ±  8%     -28.8%     531.50 ±  6%  slabinfo.lsm_file_cache.num_slabs
      1638 ±  7%     +15.0%       1883 ±  6%  slabinfo.nfs_read_data.active_objs
      1638 ±  7%     +15.0%       1883 ±  6%  slabinfo.nfs_read_data.num_objs
    767.69 ± 15%     +28.1%     983.45 ±  4%  sched_debug.cfs_rq:/.exec_clock.stddev
      0.31 ±  5%     +19.5%       0.37        sched_debug.cfs_rq:/.nr_running.stddev
     13.37 ±  8%     -16.3%      11.19 ±  3%  sched_debug.cfs_rq:/.removed.load_avg.avg
    716.88 ± 16%     +19.6%     857.73 ±  3%  sched_debug.cfs_rq:/.removed.load_avg.max
    611.11 ±  7%     -15.6%     515.87 ±  3%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
     33176 ± 16%     +19.6%      39678 ±  3%  sched_debug.cfs_rq:/.removed.runnable_sum.max
      2.24 ± 21%     -44.5%       1.24 ± 14%  sched_debug.cfs_rq:/.removed.util_avg.avg
     16.35 ± 12%     -29.0%      11.62 ± 16%  sched_debug.cfs_rq:/.removed.util_avg.stddev
      5.39 ±  9%     -32.7%       3.63        sched_debug.cfs_rq:/.runnable_load_avg.avg
    369.63 ± 29%     -44.1%     206.73 ±  8%  sched_debug.cfs_rq:/.runnable_load_avg.max
     36.03 ± 19%     -41.6%      21.04        sched_debug.cfs_rq:/.runnable_load_avg.stddev
    225.53 ±  4%     +34.9%     304.13 ±  3%  sched_debug.cfs_rq:/.util_avg.min
    228.37 ±  5%     -12.8%     199.11        sched_debug.cfs_rq:/.util_avg.stddev
     66833 ± 10%     +72.7%     115413 ±  2%  sched_debug.cpu.avg_idle.min
      5.81 ± 22%     -35.2%       3.76 ±  8%  sched_debug.cpu.cpu_load[0].avg
    445.72 ± 16%     -25.6%     331.77 ±  4%  sched_debug.cpu.cpu_load[0].max
     42.06 ± 20%     -32.8%      28.27 ±  5%  sched_debug.cpu.cpu_load[0].stddev
      5.69 ± 19%     -35.5%       3.67 ±  5%  sched_debug.cpu.cpu_load[1].avg
    337.12 ± 10%     -25.8%     250.03 ±  5%  sched_debug.cpu.cpu_load[1].max
     32.57 ± 14%     -32.0%      22.16 ±  3%  sched_debug.cpu.cpu_load[1].stddev
      5.52 ± 16%     -34.0%       3.65        sched_debug.cpu.cpu_load[2].avg
    257.90 ±  6%     -22.0%     201.17 ± 12%  sched_debug.cpu.cpu_load[2].max
     26.25 ±  9%     -27.9%      18.93 ±  8%  sched_debug.cpu.cpu_load[2].stddev
      5.15 ± 14%     -33.3%       3.43        sched_debug.cpu.cpu_load[3].avg
     19.84 ±  7%     -23.9%      15.10 ± 11%  sched_debug.cpu.cpu_load[3].stddev
      4.52 ± 14%     -32.9%       3.03 ±  2%  sched_debug.cpu.cpu_load[4].avg
     28161 ±  5%      +9.7%      30894 ±  4%  sched_debug.cpu.curr->pid.stddev
    597.66 ± 39%    +150.9%       1499 ± 26%  sched_debug.cpu.max_idle_balance_cost.stddev
      6.97 ±  5%     +19.7%       8.35        sched_debug.cpu.nr_uninterruptible.avg
    385919 ±  5%      +9.4%     422383 ±  6%  sched_debug.cpu.sched_count.max
    104453 ±  7%     +21.8%     127211        sched_debug.cpu.sched_goidle.avg
    110916 ±  7%     +19.5%     132580        sched_debug.cpu.sched_goidle.max
    100330 ±  7%     +22.9%     123301        sched_debug.cpu.sched_goidle.min
    157482 ±  4%     +13.6%     178917        sched_debug.cpu.ttwu_count.avg
    163535 ±  4%     +13.5%     185634        sched_debug.cpu.ttwu_count.max
    152786 ±  5%     +14.1%     174393        sched_debug.cpu.ttwu_count.min
    673.39 ±  6%     -20.9%     532.74 ±  5%  sched_debug.cpu.ttwu_local.stddev
     15.49 ± 17%     -32.2%      10.50 ±  5%  perf-stat.i.MPKI
 2.531e+10 ±  7%      +6.3%  2.692e+10        perf-stat.i.branch-instructions
      2.16 ± 19%      -0.7        1.46 ±  4%  perf-stat.i.branch-miss-rate%
 2.407e+08 ±  7%      +6.3%  2.559e+08        perf-stat.i.branch-misses
      7.10 ±  3%      +1.1        8.17        perf-stat.i.cache-miss-rate%
  88290679 ±  7%     +13.3%      1e+08        perf-stat.i.cache-misses
 1.015e+09 ±  7%      +7.1%  1.087e+09        perf-stat.i.cache-references
    103162 ±  5%     +14.9%     118526        perf-stat.i.context-switches
      2.68 ± 13%     -23.2%       2.05        perf-stat.i.cpi
     35426 ±  6%     +15.7%      40979        perf-stat.i.cpu-migrations
     10332 ± 27%     -45.3%       5653 ±  4%  perf-stat.i.cycles-between-cache-misses
      0.39 ±  7%      -0.1        0.31 ± 12%  perf-stat.i.dTLB-load-miss-rate%
 4.634e+10 ±  7%      +6.3%  4.926e+10        perf-stat.i.dTLB-loads
      0.12 ±  3%      -0.0        0.11        perf-stat.i.dTLB-store-miss-rate%
   7036966 ±  7%      +8.8%    7657668        perf-stat.i.iTLB-loads
 1.595e+11 ±  7%      +6.3%  1.695e+11        perf-stat.i.instructions
      7596           +11.9%       8503        perf-stat.i.instructions-per-iTLB-miss
      0.53 ±  2%      +7.9%       0.57        perf-stat.i.ipc
  50913654 ±  7%     +15.4%   58779174        perf-stat.i.node-load-misses
  16212935 ±  7%     +11.2%   18030635        perf-stat.i.node-loads
  10815686 ±  7%     +10.5%   11953107        perf-stat.i.node-store-misses
   9536170 ±  7%      +7.9%   10287859        perf-stat.i.node-stores
      8.69            +0.5        9.20        perf-stat.overall.cache-miss-rate%
      1.68            -1.8%       1.65        perf-stat.overall.cpi
      3030            -7.9%       2789        perf-stat.overall.cycles-between-cache-misses
      0.11 ±  3%      -0.0        0.10        perf-stat.overall.dTLB-store-miss-rate%
     72.17            -1.6       70.56        perf-stat.overall.iTLB-load-miss-rate%
      8739 ±  2%      +5.7%       9236        perf-stat.overall.instructions-per-iTLB-miss
      0.60            +1.9%       0.61        perf-stat.overall.ipc
     75.84            +0.7       76.52        perf-stat.overall.node-load-miss-rate%
     53.12            +0.6       53.73        perf-stat.overall.node-store-miss-rate%
 2.523e+10 ±  7%      +6.4%  2.685e+10        perf-stat.ps.branch-instructions
   2.4e+08 ±  7%      +6.4%  2.553e+08        perf-stat.ps.branch-misses
  87961660 ±  7%     +13.4%   99747792        perf-stat.ps.cache-misses
 1.012e+09 ±  7%      +7.2%  1.084e+09        perf-stat.ps.cache-references
    102771 ±  5%     +15.0%     118185        perf-stat.ps.context-switches
     35287 ±  6%     +15.8%      40858        perf-stat.ps.cpu-migrations
 4.619e+10 ±  7%      +6.4%  4.914e+10        perf-stat.ps.dTLB-loads
   7013138 ±  7%      +8.9%    7637838        perf-stat.ps.iTLB-loads
  1.59e+11 ±  7%      +6.4%  1.691e+11        perf-stat.ps.instructions
  50716358 ±  7%     +15.6%   58608142        perf-stat.ps.node-load-misses
  16157092 ±  7%     +11.3%   17983416        perf-stat.ps.node-loads
  10773759 ±  7%     +10.6%   11918485        perf-stat.ps.node-store-misses
   9505149 ±  7%      +8.0%   10262547        perf-stat.ps.node-stores
      1.62 ± 36%      -1.3        0.28 ±100%  perf-profile.calltrace.cycles-pp.ftrace_syscall_enter.syscall_trace_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.96 ± 30%      -1.2        0.75 ±  5%  perf-profile.calltrace.cycles-pp.syscall_trace_enter.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.10 ± 16%      -1.2        3.92        perf-profile.calltrace.cycles-pp.__generic_file_write_iter.generic_file_write_iter.new_sync_write.vfs_write.ksys_write
      4.84 ± 16%      -1.1        3.72        perf-profile.calltrace.cycles-pp.generic_perform_write.__generic_file_write_iter.generic_file_write_iter.new_sync_write.vfs_write
      1.56 ± 33%      -1.0        0.59        perf-profile.calltrace.cycles-pp.syscall_slow_exit_work.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.36 ± 38%      +0.6        1.96        perf-profile.calltrace.cycles-pp.filemap_map_pages.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
      0.00            +0.6        0.65 ±  9%  perf-profile.calltrace.cycles-pp.down_write.__vma_adjust.__split_vma.__do_munmap.mmap_region
      0.00            +0.6        0.65 ±  9%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.__vma_adjust.__split_vma.__do_munmap
      0.00            +0.7        0.69 ±  8%  perf-profile.calltrace.cycles-pp.__vma_adjust.__split_vma.__do_munmap.mmap_region.do_mmap
      0.00            +0.7        0.69 ±  8%  perf-profile.calltrace.cycles-pp.__split_vma.__do_munmap.mmap_region.do_mmap.vm_mmap_pgoff
      2.64 ± 33%      +0.7        3.33 ±  2%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      2.71 ± 33%      +0.7        3.42 ±  2%  perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      0.36 ± 70%      +0.9        1.28 ±  8%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.unlink_file_vma.free_pgtables.unmap_region
      0.36 ± 70%      +0.9        1.29 ±  8%  perf-profile.calltrace.cycles-pp.down_write.unlink_file_vma.free_pgtables.unmap_region.__do_munmap
      0.36 ± 70%      +0.9        1.30 ±  8%  perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.unmap_region.__do_munmap.mmap_region
      0.38 ± 70%      +1.0        1.34 ±  7%  perf-profile.calltrace.cycles-pp.free_pgtables.unmap_region.__do_munmap.mmap_region.do_mmap
      0.42 ± 70%      +1.0        1.39 ±  6%  perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.mmap_region.do_mmap.vm_mmap_pgoff
      1.12 ± 34%      +1.0        2.13 ±  2%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.vma_link.mmap_region
      2.79 ± 30%      +1.1        3.84 ±  4%  perf-profile.calltrace.cycles-pp.flush_old_exec.load_elf_binary.search_binary_handler.__do_execve_file.__x64_sys_execve
      2.77 ± 30%      +1.1        3.83 ±  4%  perf-profile.calltrace.cycles-pp.mmput.flush_old_exec.load_elf_binary.search_binary_handler.__do_execve_file
      1.16 ± 35%      +1.1        2.21 ±  2%  perf-profile.calltrace.cycles-pp.down_write.vma_link.mmap_region.do_mmap.vm_mmap_pgoff
      1.15 ± 34%      +1.1        2.21 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.vma_link.mmap_region.do_mmap
      1.29 ± 34%      +1.1        2.37 ±  2%  perf-profile.calltrace.cycles-pp.vma_link.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
      1.08 ± 39%      +1.3        2.43 ±  6%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.__vma_adjust.__split_vma.mprotect_fixup
      1.10 ± 38%      +1.4        2.45 ±  6%  perf-profile.calltrace.cycles-pp.down_write.__vma_adjust.__split_vma.mprotect_fixup.do_mprotect_pkey
      1.29 ± 37%      +1.4        2.69 ±  6%  perf-profile.calltrace.cycles-pp.__split_vma.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64
      1.27 ± 37%      +1.4        2.67 ±  6%  perf-profile.calltrace.cycles-pp.__vma_adjust.__split_vma.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect
      1.36 ± 37%      +1.4        2.77 ±  6%  perf-profile.calltrace.cycles-pp.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.36 ± 37%      +1.4        2.77 ±  6%  perf-profile.calltrace.cycles-pp.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.34 ± 37%      +1.4        2.75 ±  6%  perf-profile.calltrace.cycles-pp.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.96 ± 37%      +1.4        4.39 ±  4%  perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary
      0.64 ± 70%      +1.4        2.09 ±  7%  perf-profile.calltrace.cycles-pp.__do_munmap.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
      3.32 ± 37%      +1.5        4.79 ±  3%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler
      2.58 ± 31%      +1.5        4.11 ±  5%  perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.mmput.flush_old_exec
      0.87 ± 71%      +2.1        2.96 ±  7%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.__vma_adjust.__split_vma
      2.20 ± 37%      +2.4        4.62 ±  4%  perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.16 ± 37%      +2.4        4.59 ±  4%  perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.19 ± 37%      +2.4        4.62 ±  4%  perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.12 ± 37%      +2.4        4.55 ±  4%  perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
     10.81 ± 51%      +8.5       19.27 ±  4%  perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit
     12.55 ± 49%      +8.7       21.27 ±  3%  perf-profile.calltrace.cycles-pp.dup_mm.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.39 ± 48%      +8.8       22.17 ±  2%  perf-profile.calltrace.cycles-pp.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.62 ± 48%      +8.8       22.43 ±  2%  perf-profile.calltrace.cycles-pp._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.40 ± 39%      +9.0       17.41 ±  5%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.dup_mm.copy_process
     12.77 ± 51%      +9.2       21.96 ±  3%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
     12.78 ± 51%      +9.2       21.98 ±  3%  perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      8.56 ± 40%      +9.3       17.88 ±  4%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.dup_mm.copy_process._do_fork
      8.63 ± 40%      +9.4       17.98 ±  4%  perf-profile.calltrace.cycles-pp.down_write.dup_mm.copy_process._do_fork.do_syscall_64
     13.46 ± 51%      +9.4       22.87 ±  2%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.46 ± 51%      +9.4       22.87 ±  2%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.46 ± 51%      +9.4       22.88 ±  2%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.45 ± 44%      +9.6       18.02 ±  4%  perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.mmput.do_exit
     10.85 ± 40%     +11.0       21.84 ±  5%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.unlink_file_vma.free_pgtables.exit_mmap
     10.94 ± 40%     +11.0       21.97 ±  5%  perf-profile.calltrace.cycles-pp.down_write.unlink_file_vma.free_pgtables.exit_mmap.mmput
     10.75 ± 41%     +11.7       22.43 ±  5%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.unlink_file_vma.free_pgtables
      6.37 ± 66%      -4.6        1.77 ±  2%  perf-profile.children.cycles-pp.apic_timer_interrupt
      6.00 ± 66%      -4.3        1.71 ±  2%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      4.04 ± 68%      -2.9        1.12        perf-profile.children.cycles-pp.hrtimer_interrupt
      2.89 ± 59%      -1.9        0.99        perf-profile.children.cycles-pp.__hrtimer_run_queues
      1.89 ± 91%      -1.8        0.07 ±  7%  perf-profile.children.cycles-pp.menu_select
      1.97 ± 30%      -1.2        0.76 ±  4%  perf-profile.children.cycles-pp.syscall_trace_enter
      4.86 ± 16%      -1.1        3.74        perf-profile.children.cycles-pp.generic_perform_write
      1.64 ± 35%      -1.1        0.53 ±  9%  perf-profile.children.cycles-pp.ftrace_syscall_enter
      1.22 ± 58%      -1.1        0.17 ±  5%  perf-profile.children.cycles-pp.trace_buffer_lock_reserve
      1.89 ± 48%      -1.0        0.86        perf-profile.children.cycles-pp.tick_sched_timer
      1.57 ± 32%      -1.0        0.60        perf-profile.children.cycles-pp.syscall_slow_exit_work
      1.39 ± 37%      -0.9        0.46 ±  2%  perf-profile.children.cycles-pp.ftrace_syscall_exit
      1.71 ± 44%      -0.9        0.84        perf-profile.children.cycles-pp.tick_sched_handle
      0.97 ± 61%      -0.9        0.11 ±  9%  perf-profile.children.cycles-pp.ring_buffer_lock_reserve
      0.86 ± 95%      -0.8        0.06 ±  9%  perf-profile.children.cycles-pp.ktime_get
      1.59 ± 40%      -0.8        0.83        perf-profile.children.cycles-pp.update_process_times
      1.30 ± 48%      -0.8        0.54 ± 10%  perf-profile.children.cycles-pp.irq_exit
      0.70 ± 91%      -0.6        0.07 ±  7%  perf-profile.children.cycles-pp.clockevents_program_event
      1.08 ± 31%      -0.5        0.59 ±  8%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.41 ± 83%      -0.4        0.03 ±100%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      1.08 ± 20%      -0.3        0.73 ±  2%  perf-profile.children.cycles-pp.scheduler_tick
      0.32 ± 82%      -0.3        0.03 ±100%  perf-profile.children.cycles-pp.native_write_msr
      0.33 ± 67%      -0.3        0.06 ± 16%  perf-profile.children.cycles-pp.rebalance_domains
      0.41 ± 37%      -0.2        0.21 ±  4%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.60 ± 17%      -0.2        0.41 ±  7%  perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      0.38 ± 26%      -0.2        0.20 ±  2%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.51 ± 15%      -0.2        0.34 ±  8%  perf-profile.children.cycles-pp.copyin
      0.24 ± 35%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp.memcpy_erms
      0.30 ± 33%      -0.1        0.18        perf-profile.children.cycles-pp.elf_map
      0.15 ± 45%      -0.1        0.04 ±100%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.41 ± 14%      -0.1        0.30 ±  4%  perf-profile.children.cycles-pp.shmem_alloc_page
      0.18 ± 44%      -0.1        0.08        perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.15 ± 35%      -0.1        0.07 ± 28%  perf-profile.children.cycles-pp.update_blocked_averages
      0.23 ±  9%      -0.1        0.17 ±  2%  perf-profile.children.cycles-pp.get_unmapped_area
      0.14 ± 22%      -0.1        0.09 ± 17%  perf-profile.children.cycles-pp.update_rq_clock
      0.15 ± 10%      -0.1        0.10 ± 10%  perf-profile.children.cycles-pp.vfs_statx
      0.18 ± 14%      -0.0        0.14 ±  3%  perf-profile.children.cycles-pp.__x64_sys_getdents
      0.18 ± 12%      -0.0        0.14 ±  3%  perf-profile.children.cycles-pp.iterate_dir
      0.13 ±  6%      -0.0        0.10 ±  5%  perf-profile.children.cycles-pp.__do_sys_newstat
      0.12 ±  6%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.sock_write_iter
      0.06            -0.0        0.03 ±100%  perf-profile.children.cycles-pp.__ip_queue_xmit
      0.06            -0.0        0.03 ±100%  perf-profile.children.cycles-pp.sock_read_iter
      0.12 ± 10%      -0.0        0.08 ±  5%  perf-profile.children.cycles-pp.sock_sendmsg
      0.08            -0.0        0.05        perf-profile.children.cycles-pp.__x64_sys_kill
      0.08 ±  5%      -0.0        0.06 ±  9%  perf-profile.children.cycles-pp.iov_iter_advance
      0.09 ± 10%      -0.0        0.06        perf-profile.children.cycles-pp.__set_page_dirty_no_writeback
      0.08 ±  6%      -0.0        0.05        perf-profile.children.cycles-pp.__x64_sys_write
      0.10 ± 12%      -0.0        0.07        perf-profile.children.cycles-pp.do_signal
      0.09 ± 13%      -0.0        0.07        perf-profile.children.cycles-pp.security_mmap_addr
      0.09 ±  9%      -0.0        0.07        perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
      0.08 ± 17%      -0.0        0.06        perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.08 ±  6%      -0.0        0.06        perf-profile.children.cycles-pp.ip_output
      0.07 ±  7%      -0.0        0.05        perf-profile.children.cycles-pp.__tcp_transmit_skb
      0.04 ± 71%      +0.0        0.07        perf-profile.children.cycles-pp.pipe_read
      0.02 ±141%      +0.0        0.06        perf-profile.children.cycles-pp.pipe_wait
      0.07 ± 70%      +0.0        0.12        perf-profile.children.cycles-pp.osq_unlock
      0.07 ± 71%      +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.dequeue_entity
      0.15 ± 35%      +0.1        0.22 ±  6%  perf-profile.children.cycles-pp.__pte_alloc
      0.13 ± 39%      +0.1        0.19 ±  5%  perf-profile.children.cycles-pp.put_cred_rcu
      0.07 ± 72%      +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.schedule_idle
      0.13 ± 73%      +0.1        0.27 ±  5%  perf-profile.children.cycles-pp.do_task_dead
      0.50 ±  5%      +0.1        0.63 ±  4%  perf-profile.children.cycles-pp.load_balance
      0.33 ±  8%      +0.2        0.49 ±  4%  perf-profile.children.cycles-pp.find_busiest_group
      0.31 ± 12%      +0.2        0.48 ±  4%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.72 ± 26%      +0.2        0.95 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.41 ± 37%      +0.2        0.64 ±  2%  perf-profile.children.cycles-pp.schedule
      0.37 ± 49%      +0.3        0.69 ±  3%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.65 ± 42%      +0.4        1.05 ±  3%  perf-profile.children.cycles-pp.__schedule
      0.34 ± 62%      +0.5        0.79        perf-profile.children.cycles-pp.wake_up_q
      0.42 ± 59%      +0.5        0.88        perf-profile.children.cycles-pp.rwsem_wake
      0.51 ± 53%      +0.5        0.99        perf-profile.children.cycles-pp.try_to_wake_up
      1.39 ± 37%      +0.6        1.99        perf-profile.children.cycles-pp.filemap_map_pages
      0.79 ± 35%      +0.8        1.63 ±  4%  perf-profile.children.cycles-pp.unmap_region
      1.43 ± 33%      +1.0        2.42 ±  2%  perf-profile.children.cycles-pp.vma_link
      1.22 ± 34%      +1.3        2.47 ±  5%  perf-profile.children.cycles-pp.__do_munmap
      3.51 ± 30%      +1.3        4.80 ±  3%  perf-profile.children.cycles-pp.flush_old_exec
      1.36 ± 37%      +1.4        2.77 ±  6%  perf-profile.children.cycles-pp.__x64_sys_mprotect
      1.36 ± 37%      +1.4        2.77 ±  6%  perf-profile.children.cycles-pp.do_mprotect_pkey
      1.35 ± 37%      +1.4        2.77 ±  6%  perf-profile.children.cycles-pp.mprotect_fixup
      1.64 ± 37%      +1.8        3.47 ±  6%  perf-profile.children.cycles-pp.__split_vma
      1.81 ± 35%      +1.8        3.65 ±  5%  perf-profile.children.cycles-pp.__vma_adjust
      2.37 ± 35%      +2.4        4.72 ±  4%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      2.28 ± 36%      +2.4        4.64 ±  4%  perf-profile.children.cycles-pp.mmap_region
      2.32 ± 36%      +2.4        4.69 ±  4%  perf-profile.children.cycles-pp.do_mmap
      2.20 ± 37%      +2.4        4.62 ±  4%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
     12.55 ± 49%      +8.7       21.27 ±  3%  perf-profile.children.cycles-pp.dup_mm
     13.39 ± 48%      +8.8       22.17 ±  2%  perf-profile.children.cycles-pp.copy_process
     13.62 ± 48%      +8.8       22.43 ±  2%  perf-profile.children.cycles-pp._do_fork
     13.46 ± 51%      +9.4       22.87 ±  2%  perf-profile.children.cycles-pp.do_group_exit
     13.46 ± 51%      +9.4       22.87 ±  2%  perf-profile.children.cycles-pp.do_exit
     13.46 ± 51%      +9.4       22.88 ±  2%  perf-profile.children.cycles-pp.__x64_sys_exit_group
     16.26 ± 46%     +10.5       26.75 ±  3%  perf-profile.children.cycles-pp.exit_mmap
     16.27 ± 46%     +10.5       26.77 ±  3%  perf-profile.children.cycles-pp.mmput
     14.39 ± 46%     +10.6       25.03 ±  4%  perf-profile.children.cycles-pp.free_pgtables
     11.62 ± 38%     +11.8       23.45 ±  5%  perf-profile.children.cycles-pp.unlink_file_vma
     26.18 ± 46%     +19.5       45.70 ±  5%  perf-profile.children.cycles-pp.osq_lock
     27.24 ± 46%     +20.3       47.53 ±  4%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
     27.66 ± 46%     +20.4       48.02 ±  4%  perf-profile.children.cycles-pp.down_write
      0.32 ± 82%      -0.3        0.03 ±100%  perf-profile.self.cycles-pp.native_write_msr
      0.34 ± 43%      -0.2        0.10 ± 10%  perf-profile.self.cycles-pp.ring_buffer_lock_reserve
      0.66 ±  3%      -0.2        0.44 ± 11%  perf-profile.self.cycles-pp.ftrace_syscall_enter
      0.41 ± 37%      -0.2        0.21 ±  4%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.34 ± 26%      -0.2        0.17 ±  5%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.50 ± 15%      -0.1        0.35        perf-profile.self.cycles-pp.ftrace_syscall_exit
      0.23 ± 37%      -0.1        0.10        perf-profile.self.cycles-pp.memcpy_erms
      0.32 ± 11%      -0.1        0.22 ±  6%  perf-profile.self.cycles-pp.syscall_trace_enter
      0.24 ±  7%      -0.1        0.18 ± 11%  perf-profile.self.cycles-pp.get_page_from_freelist
      0.11 ± 25%      -0.0        0.07        perf-profile.self.cycles-pp.trace_buffer_lock_reserve
      0.13 ± 13%      -0.0        0.10 ±  5%  perf-profile.self.cycles-pp.generic_file_write_iter
      0.13 ± 12%      -0.0        0.10        perf-profile.self.cycles-pp.generic_perform_write
      0.08 ± 10%      -0.0        0.05        perf-profile.self.cycles-pp.__set_page_dirty_no_writeback
      0.08 ± 10%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.ksys_write
      0.08 ± 10%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.__x64_sys_brk
      0.08 ±  6%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.iov_iter_advance
      0.09 ± 10%      -0.0        0.07 ±  7%  perf-profile.self.cycles-pp.shmem_write_begin
      0.07 ± 11%      -0.0        0.05        perf-profile.self.cycles-pp.__fdget_pos
      0.07 ± 11%      -0.0        0.05        perf-profile.self.cycles-pp.file_update_time
      0.06 ±  7%      -0.0        0.05        perf-profile.self.cycles-pp.vma_merge
      0.03 ± 70%      +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.put_cred_rcu
      0.04 ± 71%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.vm_area_dup
      0.07 ± 71%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.osq_unlock
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.__anon_vma_interval_tree_augment_rotate
      0.05 ± 74%      +0.1        0.11        perf-profile.self.cycles-pp.try_to_wake_up
      0.24 ± 17%      +0.1        0.38 ±  5%  perf-profile.self.cycles-pp.update_sd_lb_stats
      0.39 ± 42%      +0.3        0.69        perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.98 ± 39%      +0.5        1.48        perf-profile.self.cycles-pp.filemap_map_pages
     25.86 ± 46%     +19.3       45.15 ±  5%  perf-profile.self.cycles-pp.osq_lock
    739.00 ± 42%     -39.4%     448.00 ±  2%  interrupts.51:PCI-MSI.1572870-edge.eth0-TxRx-6
    477.67 ±  9%    +245.7%       1651 ±  7%  interrupts.53:PCI-MSI.1572872-edge.eth0-TxRx-8
    683.00 ± 32%     -33.6%     453.50        interrupts.60:PCI-MSI.1572879-edge.eth0-TxRx-15
    499.67 ±  9%     -13.4%     432.50        interrupts.72:PCI-MSI.1572891-edge.eth0-TxRx-27
     24632 ±  6%     -12.4%      21588 ±  2%  interrupts.CPU0.RES:Rescheduling_interrupts
     22338 ±  4%     -12.9%      19463        interrupts.CPU1.RES:Rescheduling_interrupts
      2217 ±  9%     +12.7%       2498 ±  2%  interrupts.CPU1.TLB:TLB_shootdowns
     21081 ±  3%     -17.2%      17455 ±  2%  interrupts.CPU10.RES:Rescheduling_interrupts
      2023 ± 11%     +14.2%       2311 ±  4%  interrupts.CPU10.TLB:TLB_shootdowns
     10088 ±  3%     -10.4%       9042 ±  2%  interrupts.CPU100.CAL:Function_call_interrupts
      2940 ± 84%     -99.5%      15.00 ±100%  interrupts.CPU100.NMI:Non-maskable_interrupts
      2940 ± 84%     -99.5%      15.00 ±100%  interrupts.CPU100.PMI:Performance_monitoring_interrupts
     21754           -18.6%      17712 ±  3%  interrupts.CPU100.RES:Rescheduling_interrupts
      2052 ± 11%     +17.6%       2413 ±  2%  interrupts.CPU100.TLB:TLB_shootdowns
      1161 ± 40%     -77.6%     260.00 ±100%  interrupts.CPU101.NMI:Non-maskable_interrupts
      1161 ± 40%     -77.6%     260.00 ±100%  interrupts.CPU101.PMI:Performance_monitoring_interrupts
     20386           -28.0%      14672        interrupts.CPU101.RES:Rescheduling_interrupts
      2093 ± 13%     +16.0%       2428 ±  2%  interrupts.CPU101.TLB:TLB_shootdowns
     20075           -23.1%      15434        interrupts.CPU102.RES:Rescheduling_interrupts
      1991 ±  9%     +20.3%       2396        interrupts.CPU102.TLB:TLB_shootdowns
     20283 ±  4%     -20.8%      16055        interrupts.CPU103.RES:Rescheduling_interrupts
     20393           -23.8%      15537        interrupts.CPU104.RES:Rescheduling_interrupts
     20123           -23.5%      15401 ±  3%  interrupts.CPU105.RES:Rescheduling_interrupts
      2049 ±  9%     +17.4%       2405 ±  2%  interrupts.CPU105.TLB:TLB_shootdowns
     20391 ±  2%     -25.1%      15272 ±  5%  interrupts.CPU106.RES:Rescheduling_interrupts
      2079 ±  6%     +18.2%       2457 ±  2%  interrupts.CPU106.TLB:TLB_shootdowns
     19707           -22.7%      15239 ±  5%  interrupts.CPU107.RES:Rescheduling_interrupts
      2143 ±  7%     +16.1%       2487 ±  3%  interrupts.CPU107.TLB:TLB_shootdowns
      3426 ± 68%     -99.7%      11.00 ±100%  interrupts.CPU108.NMI:Non-maskable_interrupts
      3426 ± 68%     -99.7%      11.00 ±100%  interrupts.CPU108.PMI:Performance_monitoring_interrupts
     21621           -18.6%      17600        interrupts.CPU108.RES:Rescheduling_interrupts
      1947 ± 10%     +19.0%       2318        interrupts.CPU108.TLB:TLB_shootdowns
      2418 ±121%    +195.3%       7141        interrupts.CPU109.NMI:Non-maskable_interrupts
      2418 ±121%    +195.3%       7141        interrupts.CPU109.PMI:Performance_monitoring_interrupts
     21399 ±  2%     -13.9%      18430 ±  2%  interrupts.CPU109.RES:Rescheduling_interrupts
      1978 ±  9%     +25.1%       2474        interrupts.CPU109.TLB:TLB_shootdowns
     20465 ±  2%     -22.3%      15895        interrupts.CPU11.RES:Rescheduling_interrupts
     21345 ±  2%     -18.3%      17435 ±  4%  interrupts.CPU110.RES:Rescheduling_interrupts
      1971 ±  8%     +18.6%       2338 ±  4%  interrupts.CPU110.TLB:TLB_shootdowns
     20086 ±  3%     -19.1%      16258 ±  2%  interrupts.CPU111.RES:Rescheduling_interrupts
      2047 ±  5%     +17.2%       2399 ± 13%  interrupts.CPU111.TLB:TLB_shootdowns
      1472 ±109%    +380.9%       7082 ±  2%  interrupts.CPU112.NMI:Non-maskable_interrupts
      1472 ±109%    +380.9%       7082 ±  2%  interrupts.CPU112.PMI:Performance_monitoring_interrupts
     20424           -20.0%      16331        interrupts.CPU112.RES:Rescheduling_interrupts
      2094 ± 11%     +12.6%       2358 ±  3%  interrupts.CPU112.TLB:TLB_shootdowns
     20019           -21.5%      15722 ±  4%  interrupts.CPU113.RES:Rescheduling_interrupts
      2022 ± 13%     +17.1%       2368 ±  3%  interrupts.CPU113.TLB:TLB_shootdowns
     20327           -19.4%      16374 ±  2%  interrupts.CPU114.RES:Rescheduling_interrupts
      2022 ± 12%     +19.0%       2407        interrupts.CPU114.TLB:TLB_shootdowns
     20429 ±  2%     -21.3%      16079 ±  5%  interrupts.CPU115.RES:Rescheduling_interrupts
      2091 ±  9%     +15.5%       2415        interrupts.CPU115.TLB:TLB_shootdowns
     20389 ±  2%     -13.6%      17614        interrupts.CPU116.RES:Rescheduling_interrupts
      2041 ± 11%     +17.8%       2405 ±  2%  interrupts.CPU116.TLB:TLB_shootdowns
     20250 ±  2%     -18.7%      16460 ±  2%  interrupts.CPU117.RES:Rescheduling_interrupts
     20035           -20.8%      15878        interrupts.CPU118.RES:Rescheduling_interrupts
      1960 ±  8%     +16.7%       2288 ±  7%  interrupts.CPU118.TLB:TLB_shootdowns
     19647 ±  2%     -21.1%      15504 ±  5%  interrupts.CPU119.RES:Rescheduling_interrupts
      2159 ±  6%      +7.4%       2319 ±  2%  interrupts.CPU119.TLB:TLB_shootdowns
      2359 ±123%    +200.0%       7078        interrupts.CPU12.NMI:Non-maskable_interrupts
      2359 ±123%    +200.0%       7078        interrupts.CPU12.PMI:Performance_monitoring_interrupts
     20907 ±  3%     -13.9%      17995        interrupts.CPU12.RES:Rescheduling_interrupts
      3773 ± 54%     +87.1%       7060        interrupts.CPU120.NMI:Non-maskable_interrupts
      3773 ± 54%     +87.1%       7060        interrupts.CPU120.PMI:Performance_monitoring_interrupts
     20164           -19.9%      16159        interrupts.CPU120.RES:Rescheduling_interrupts
      2107 ± 10%     +14.7%       2417 ±  4%  interrupts.CPU120.TLB:TLB_shootdowns
      1520 ± 87%    +365.3%       7076        interrupts.CPU121.NMI:Non-maskable_interrupts
      1520 ± 87%    +365.3%       7076        interrupts.CPU121.PMI:Performance_monitoring_interrupts
     19674 ±  3%     -17.3%      16266        interrupts.CPU121.RES:Rescheduling_interrupts
      1950 ± 11%     +23.2%       2403 ±  2%  interrupts.CPU121.TLB:TLB_shootdowns
    264.67 ±100%   +1873.2%       5222 ± 30%  interrupts.CPU122.NMI:Non-maskable_interrupts
    264.67 ±100%   +1873.2%       5222 ± 30%  interrupts.CPU122.PMI:Performance_monitoring_interrupts
     19613 ±  2%     -17.5%      16190        interrupts.CPU122.RES:Rescheduling_interrupts
      2100 ± 10%     +20.5%       2530 ± 10%  interrupts.CPU122.TLB:TLB_shootdowns
      1136 ± 54%     -93.0%      80.00 ±100%  interrupts.CPU123.NMI:Non-maskable_interrupts
      1136 ± 54%     -93.0%      80.00 ±100%  interrupts.CPU123.PMI:Performance_monitoring_interrupts
     20297           -26.2%      14973        interrupts.CPU123.RES:Rescheduling_interrupts
     20612 ±  3%     -19.7%      16553 ±  2%  interrupts.CPU124.RES:Rescheduling_interrupts
      2179 ± 39%    +142.7%       5289 ± 31%  interrupts.CPU125.NMI:Non-maskable_interrupts
      2179 ± 39%    +142.7%       5289 ± 31%  interrupts.CPU125.PMI:Performance_monitoring_interrupts
     20258           -20.6%      16085        interrupts.CPU125.RES:Rescheduling_interrupts
      2161 ± 39%    +225.4%       7032        interrupts.CPU126.NMI:Non-maskable_interrupts
      2161 ± 39%    +225.4%       7032        interrupts.CPU126.PMI:Performance_monitoring_interrupts
     22110 ±  3%     -13.9%      19029        interrupts.CPU126.RES:Rescheduling_interrupts
      2050 ±  9%     +16.7%       2391 ±  5%  interrupts.CPU126.TLB:TLB_shootdowns
     21262           -12.1%      18698        interrupts.CPU127.RES:Rescheduling_interrupts
      2097 ± 10%     +14.6%       2404        interrupts.CPU127.TLB:TLB_shootdowns
     21213 ±  3%     -15.8%      17857 ±  2%  interrupts.CPU128.RES:Rescheduling_interrupts
      1999 ± 12%     +21.0%       2418 ±  4%  interrupts.CPU128.TLB:TLB_shootdowns
     20900 ±  2%     -20.3%      16649 ±  4%  interrupts.CPU129.RES:Rescheduling_interrupts
      2038 ± 12%     +11.9%       2280 ±  4%  interrupts.CPU129.TLB:TLB_shootdowns
      2670 ±106%     +99.1%       5316 ± 35%  interrupts.CPU13.NMI:Non-maskable_interrupts
      2670 ±106%     +99.1%       5316 ± 35%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
     20950 ±  3%     -14.5%      17911        interrupts.CPU13.RES:Rescheduling_interrupts
      2117 ± 11%      +9.3%       2315 ±  7%  interrupts.CPU13.TLB:TLB_shootdowns
      1031 ± 73%    +411.8%       5280 ± 31%  interrupts.CPU130.NMI:Non-maskable_interrupts
      1031 ± 73%    +411.8%       5280 ± 31%  interrupts.CPU130.PMI:Performance_monitoring_interrupts
     20559 ±  2%     -19.7%      16516        interrupts.CPU130.RES:Rescheduling_interrupts
      1991 ±  9%     +16.8%       2326        interrupts.CPU130.TLB:TLB_shootdowns
      2630 ±108%    +100.7%       5279 ± 30%  interrupts.CPU131.NMI:Non-maskable_interrupts
      2630 ±108%    +100.7%       5279 ± 30%  interrupts.CPU131.PMI:Performance_monitoring_interrupts
     20323 ±  2%     -17.9%      16683        interrupts.CPU131.RES:Rescheduling_interrupts
      2094 ± 10%     +12.0%       2347 ±  4%  interrupts.CPU131.TLB:TLB_shootdowns
      2703 ± 38%    +157.0%       6949        interrupts.CPU132.NMI:Non-maskable_interrupts
      2703 ± 38%    +157.0%       6949        interrupts.CPU132.PMI:Performance_monitoring_interrupts
     21400 ±  2%     -11.1%      19014        interrupts.CPU132.RES:Rescheduling_interrupts
      2116 ±  8%     +13.5%       2402 ±  9%  interrupts.CPU132.TLB:TLB_shootdowns
     20862 ±  2%     -21.9%      16294 ±  2%  interrupts.CPU133.RES:Rescheduling_interrupts
      2036 ±  8%     +12.1%       2283 ±  6%  interrupts.CPU133.TLB:TLB_shootdowns
      1662 ± 77%    +215.9%       5252 ± 31%  interrupts.CPU134.NMI:Non-maskable_interrupts
      1662 ± 77%    +215.9%       5252 ± 31%  interrupts.CPU134.PMI:Performance_monitoring_interrupts
     20319           -19.9%      16272        interrupts.CPU134.RES:Rescheduling_interrupts
      2045 ± 10%     +10.0%       2249 ±  5%  interrupts.CPU134.TLB:TLB_shootdowns
     20071 ±  2%     -19.9%      16068 ±  4%  interrupts.CPU135.RES:Rescheduling_interrupts
      2031 ± 10%     +12.1%       2276 ±  5%  interrupts.CPU135.TLB:TLB_shootdowns
     20438           -19.5%      16448 ±  3%  interrupts.CPU136.RES:Rescheduling_interrupts
      2045 ± 10%     +23.0%       2515        interrupts.CPU136.TLB:TLB_shootdowns
    788.00 ± 72%    +797.5%       7072 ±  2%  interrupts.CPU137.NMI:Non-maskable_interrupts
    788.00 ± 72%    +797.5%       7072 ±  2%  interrupts.CPU137.PMI:Performance_monitoring_interrupts
     20227           -18.2%      16546        interrupts.CPU137.RES:Rescheduling_interrupts
      2074 ± 10%     +12.9%       2341 ±  4%  interrupts.CPU137.TLB:TLB_shootdowns
      1653 ± 90%    +332.2%       7147 ±  2%  interrupts.CPU138.NMI:Non-maskable_interrupts
      1653 ± 90%    +332.2%       7147 ±  2%  interrupts.CPU138.PMI:Performance_monitoring_interrupts
     20251 ±  2%     -18.0%      16607        interrupts.CPU138.RES:Rescheduling_interrupts
     19889 ±  2%     -22.3%      15451        interrupts.CPU139.RES:Rescheduling_interrupts
      2040 ±  7%     +19.8%       2444 ±  4%  interrupts.CPU139.TLB:TLB_shootdowns
      3569 ± 67%    -100.0%       0.50 ±100%  interrupts.CPU14.NMI:Non-maskable_interrupts
      3569 ± 67%    -100.0%       0.50 ±100%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
     20557 ±  2%     -19.9%      16471        interrupts.CPU14.RES:Rescheduling_interrupts
    482.00 ±122%   +1360.9%       7041 ±  2%  interrupts.CPU140.NMI:Non-maskable_interrupts
    482.00 ±122%   +1360.9%       7041 ±  2%  interrupts.CPU140.PMI:Performance_monitoring_interrupts
     21150           -15.1%      17964 ±  2%  interrupts.CPU140.RES:Rescheduling_interrupts
    872.33 ± 69%    +505.7%       5283 ± 31%  interrupts.CPU141.NMI:Non-maskable_interrupts
    872.33 ± 69%    +505.7%       5283 ± 31%  interrupts.CPU141.PMI:Performance_monitoring_interrupts
     20878 ±  3%     -12.5%      18258        interrupts.CPU141.RES:Rescheduling_interrupts
      1889 ± 10%     +23.5%       2333 ±  4%  interrupts.CPU141.TLB:TLB_shootdowns
      2737 ± 37%     +93.6%       5300 ± 32%  interrupts.CPU142.NMI:Non-maskable_interrupts
      2737 ± 37%     +93.6%       5300 ± 32%  interrupts.CPU142.PMI:Performance_monitoring_interrupts
     20634           -20.8%      16339        interrupts.CPU142.RES:Rescheduling_interrupts
      2004 ± 12%     +24.1%       2487 ±  2%  interrupts.CPU142.TLB:TLB_shootdowns
      2756 ± 38%     -98.3%      47.00 ± 23%  interrupts.CPU143.NMI:Non-maskable_interrupts
      2756 ± 38%     -98.3%      47.00 ± 23%  interrupts.CPU143.PMI:Performance_monitoring_interrupts
     19788 ±  2%     -25.4%      14763        interrupts.CPU143.RES:Rescheduling_interrupts
      2147 ± 10%     +17.3%       2518        interrupts.CPU143.TLB:TLB_shootdowns
    683.00 ± 32%     -33.6%     453.50        interrupts.CPU15.60:PCI-MSI.1572879-edge.eth0-TxRx-15
     20730 ±  2%     -18.7%      16860        interrupts.CPU15.RES:Rescheduling_interrupts
     20331 ±  2%     -23.6%      15523        interrupts.CPU16.RES:Rescheduling_interrupts
      1949 ± 14%     +23.5%       2407        interrupts.CPU16.TLB:TLB_shootdowns
     20198 ±  3%     -20.5%      16063        interrupts.CPU17.RES:Rescheduling_interrupts
      2090 ± 12%     +15.2%       2409 ±  2%  interrupts.CPU17.TLB:TLB_shootdowns
     22240 ±  5%     -16.8%      18511 ±  3%  interrupts.CPU19.RES:Rescheduling_interrupts
      1074 ±141%    +395.0%       5316 ± 34%  interrupts.CPU2.NMI:Non-maskable_interrupts
      1074 ±141%    +395.0%       5316 ± 34%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
     21389 ±  3%      -8.4%      19582        interrupts.CPU2.RES:Rescheduling_interrupts
     21922 ±  2%     -16.9%      18226 ±  2%  interrupts.CPU20.RES:Rescheduling_interrupts
     22730 ±  4%     -17.2%      18809 ±  5%  interrupts.CPU21.RES:Rescheduling_interrupts
     21374 ±  3%     -16.8%      17777 ±  2%  interrupts.CPU22.RES:Rescheduling_interrupts
      2034 ±  9%     +16.3%       2365 ±  3%  interrupts.CPU22.TLB:TLB_shootdowns
      2148 ±141%    +144.1%       5244 ± 35%  interrupts.CPU23.NMI:Non-maskable_interrupts
      2148 ±141%    +144.1%       5244 ± 35%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
     20429 ±  4%     -14.3%      17512        interrupts.CPU23.RES:Rescheduling_interrupts
      1964 ±  9%     +21.5%       2386 ±  2%  interrupts.CPU23.TLB:TLB_shootdowns
     20429           -15.6%      17238 ±  2%  interrupts.CPU24.RES:Rescheduling_interrupts
      1925 ± 12%     +25.1%       2408 ±  2%  interrupts.CPU24.TLB:TLB_shootdowns
     20575           -16.3%      17216 ±  5%  interrupts.CPU25.RES:Rescheduling_interrupts
      2015 ± 10%     +19.3%       2405        interrupts.CPU25.TLB:TLB_shootdowns
     20893 ±  3%     -18.9%      16938 ±  5%  interrupts.CPU26.RES:Rescheduling_interrupts
      1972 ±  9%     +19.5%       2356 ±  4%  interrupts.CPU26.TLB:TLB_shootdowns
    499.67 ±  9%     -13.4%     432.50        interrupts.CPU27.72:PCI-MSI.1572891-edge.eth0-TxRx-27
      1707 ± 79%    +205.5%       5217 ± 31%  interrupts.CPU27.NMI:Non-maskable_interrupts
      1707 ± 79%    +205.5%       5217 ± 31%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
     20676 ±  3%     -17.7%      17015        interrupts.CPU27.RES:Rescheduling_interrupts
    612.67 ±136%    +751.8%       5219 ± 32%  interrupts.CPU28.NMI:Non-maskable_interrupts
    612.67 ±136%    +751.8%       5219 ± 32%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
      2018 ± 14%     +19.1%       2404 ±  2%  interrupts.CPU28.TLB:TLB_shootdowns
      2331 ±130%    +129.0%       5340 ± 34%  interrupts.CPU29.NMI:Non-maskable_interrupts
      2331 ±130%    +129.0%       5340 ± 34%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
      2056 ± 10%     +16.1%       2386 ±  3%  interrupts.CPU29.TLB:TLB_shootdowns
      0.67 ±141%  +5.7e+05%       3796 ± 86%  interrupts.CPU3.NMI:Non-maskable_interrupts
      0.67 ±141%  +5.7e+05%       3796 ± 86%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
     21956           -14.3%      18823        interrupts.CPU3.RES:Rescheduling_interrupts
      2066 ± 12%     +14.1%       2356 ±  4%  interrupts.CPU3.TLB:TLB_shootdowns
     20772 ±  4%     -21.9%      16220 ±  5%  interrupts.CPU30.RES:Rescheduling_interrupts
      2015 ± 12%     +18.5%       2388 ±  6%  interrupts.CPU30.TLB:TLB_shootdowns
     20566           -21.4%      16163        interrupts.CPU31.RES:Rescheduling_interrupts
     20370 ±  3%     -19.6%      16372 ±  2%  interrupts.CPU32.RES:Rescheduling_interrupts
     21077 ±  4%     -19.6%      16949        interrupts.CPU33.RES:Rescheduling_interrupts
     20812           -20.3%      16588 ±  2%  interrupts.CPU34.RES:Rescheduling_interrupts
      2005 ± 15%     +18.7%       2381 ±  3%  interrupts.CPU34.TLB:TLB_shootdowns
     20487 ±  3%     -22.1%      15957 ±  2%  interrupts.CPU35.RES:Rescheduling_interrupts
      2015 ±  7%     +16.9%       2357        interrupts.CPU35.TLB:TLB_shootdowns
    948.67 ± 75%    +648.6%       7101 ±  2%  interrupts.CPU36.NMI:Non-maskable_interrupts
    948.67 ± 75%    +648.6%       7101 ±  2%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
      1991 ± 12%     +21.4%       2417 ±  8%  interrupts.CPU36.TLB:TLB_shootdowns
     21592           -16.4%      18055        interrupts.CPU37.RES:Rescheduling_interrupts
      2080 ± 10%     +15.8%       2408 ±  2%  interrupts.CPU37.TLB:TLB_shootdowns
      2071 ± 11%     +15.3%       2389 ±  4%  interrupts.CPU38.TLB:TLB_shootdowns
     21635 ±  2%     -14.2%      18572 ±  3%  interrupts.CPU39.RES:Rescheduling_interrupts
      2047 ±  8%     +17.7%       2409 ±  5%  interrupts.CPU39.TLB:TLB_shootdowns
     22078 ±  6%     -15.4%      18681        interrupts.CPU4.RES:Rescheduling_interrupts
      1926 ± 12%     +18.8%       2288        interrupts.CPU4.TLB:TLB_shootdowns
     20553           -20.6%      16318        interrupts.CPU40.RES:Rescheduling_interrupts
      1973 ± 15%     +21.2%       2391 ±  4%  interrupts.CPU40.TLB:TLB_shootdowns
     20719 ±  2%     -18.5%      16878 ±  7%  interrupts.CPU41.RES:Rescheduling_interrupts
      1942 ± 13%     +21.7%       2364 ±  3%  interrupts.CPU41.TLB:TLB_shootdowns
     21622 ±  4%     -21.1%      17064 ±  4%  interrupts.CPU42.RES:Rescheduling_interrupts
     21437 ±  2%     -20.5%      17049 ±  6%  interrupts.CPU43.RES:Rescheduling_interrupts
      1996 ± 10%     +24.1%       2477 ±  2%  interrupts.CPU43.TLB:TLB_shootdowns
     21385           -16.5%      17866        interrupts.CPU44.RES:Rescheduling_interrupts
     21222 ±  3%     -17.4%      17519 ±  5%  interrupts.CPU45.RES:Rescheduling_interrupts
     21934           -21.4%      17233        interrupts.CPU46.RES:Rescheduling_interrupts
      2117 ±  7%     +15.4%       2443 ±  5%  interrupts.CPU46.TLB:TLB_shootdowns
     20216 ±  2%     -19.0%      16366 ±  2%  interrupts.CPU47.RES:Rescheduling_interrupts
     19863           -21.9%      15513        interrupts.CPU48.RES:Rescheduling_interrupts
      2055 ±  8%     +16.7%       2398 ±  2%  interrupts.CPU48.TLB:TLB_shootdowns
     20442 ±  3%     -22.6%      15830        interrupts.CPU49.RES:Rescheduling_interrupts
     21033 ±  4%     -15.7%      17732        interrupts.CPU5.RES:Rescheduling_interrupts
     20830 ±  2%     -23.9%      15855        interrupts.CPU50.RES:Rescheduling_interrupts
      2060 ±  9%     +13.6%       2341 ±  2%  interrupts.CPU50.TLB:TLB_shootdowns
      1199 ±120%    +201.4%       3615        interrupts.CPU51.NMI:Non-maskable_interrupts
      1199 ±120%    +201.4%       3615        interrupts.CPU51.PMI:Performance_monitoring_interrupts
     20247 ±  4%     -16.4%      16927        interrupts.CPU51.RES:Rescheduling_interrupts
      2070 ± 10%     +17.6%       2434 ±  7%  interrupts.CPU51.TLB:TLB_shootdowns
      1618 ± 91%    +227.9%       5307 ± 30%  interrupts.CPU52.NMI:Non-maskable_interrupts
      1618 ± 91%    +227.9%       5307 ± 30%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
     21352           -17.4%      17631        interrupts.CPU52.RES:Rescheduling_interrupts
     20631           -17.6%      17002        interrupts.CPU53.RES:Rescheduling_interrupts
      2143 ±  6%      +8.8%       2333 ±  4%  interrupts.CPU53.TLB:TLB_shootdowns
     23539 ±  2%     -14.1%      20216 ±  3%  interrupts.CPU54.RES:Rescheduling_interrupts
      2037 ±  9%     +11.4%       2269 ±  4%  interrupts.CPU54.TLB:TLB_shootdowns
     22685           -19.3%      18306 ±  4%  interrupts.CPU55.RES:Rescheduling_interrupts
      2107 ± 10%      +9.0%       2296 ±  5%  interrupts.CPU55.TLB:TLB_shootdowns
     21677           -15.8%      18247 ±  4%  interrupts.CPU56.RES:Rescheduling_interrupts
      2008 ±  9%     +17.3%       2355 ±  7%  interrupts.CPU56.TLB:TLB_shootdowns
     21908 ±  2%     -15.3%      18548        interrupts.CPU57.RES:Rescheduling_interrupts
      2047 ± 12%     +13.4%       2321 ±  7%  interrupts.CPU57.TLB:TLB_shootdowns
      2270 ±130%     -99.9%       1.50 ± 33%  interrupts.CPU58.NMI:Non-maskable_interrupts
      2270 ±130%     -99.9%       1.50 ± 33%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
     20646           -19.4%      16650        interrupts.CPU58.RES:Rescheduling_interrupts
     21046           -21.5%      16522 ±  2%  interrupts.CPU59.RES:Rescheduling_interrupts
      1992 ± 14%     +22.0%       2430        interrupts.CPU59.TLB:TLB_shootdowns
    739.00 ± 42%     -39.4%     448.00 ±  2%  interrupts.CPU6.51:PCI-MSI.1572870-edge.eth0-TxRx-6
    919.33 ±141%     +86.4%       1713 ± 99%  interrupts.CPU6.NMI:Non-maskable_interrupts
    919.33 ±141%     +86.4%       1713 ± 99%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
      1926 ±  9%     +16.8%       2251 ±  5%  interrupts.CPU6.TLB:TLB_shootdowns
     21628 ±  2%     -14.4%      18524        interrupts.CPU60.RES:Rescheduling_interrupts
      2054 ± 12%     +17.5%       2413        interrupts.CPU60.TLB:TLB_shootdowns
      1723 ± 82%    +311.3%       7088 ±  2%  interrupts.CPU61.NMI:Non-maskable_interrupts
      1723 ± 82%    +311.3%       7088 ±  2%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
     21809           -14.8%      18586        interrupts.CPU61.RES:Rescheduling_interrupts
      2111 ± 10%     +15.6%       2440 ±  3%  interrupts.CPU61.TLB:TLB_shootdowns
     20756           -23.3%      15923 ±  2%  interrupts.CPU62.RES:Rescheduling_interrupts
     21028           -19.4%      16955 ±  2%  interrupts.CPU63.RES:Rescheduling_interrupts
      2187 ±  4%     +11.1%       2430 ±  7%  interrupts.CPU63.TLB:TLB_shootdowns
     20854 ±  2%     -15.7%      17577        interrupts.CPU64.RES:Rescheduling_interrupts
      2076 ±  8%     +10.7%       2297 ±  5%  interrupts.CPU64.TLB:TLB_shootdowns
     20753           -22.6%      16067        interrupts.CPU65.RES:Rescheduling_interrupts
     20470 ±  2%     -23.3%      15699        interrupts.CPU66.RES:Rescheduling_interrupts
      2014 ± 11%     +15.1%       2319        interrupts.CPU66.TLB:TLB_shootdowns
      1691 ± 79%    +211.8%       5272 ± 31%  interrupts.CPU67.NMI:Non-maskable_interrupts
      1691 ± 79%    +211.8%       5272 ± 31%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
     21412 ±  2%     -15.3%      18129        interrupts.CPU67.RES:Rescheduling_interrupts
      2057 ± 10%     +16.8%       2404 ±  2%  interrupts.CPU67.TLB:TLB_shootdowns
     21473 ±  2%     -16.0%      18042 ±  2%  interrupts.CPU68.RES:Rescheduling_interrupts
      2056 ± 11%     +16.3%       2391 ±  3%  interrupts.CPU68.TLB:TLB_shootdowns
     21768 ±  3%     -19.6%      17502        interrupts.CPU69.RES:Rescheduling_interrupts
      2102 ±  7%     +15.1%       2419 ±  2%  interrupts.CPU69.TLB:TLB_shootdowns
      2637 ±103%    -100.0%       0.00        interrupts.CPU7.NMI:Non-maskable_interrupts
      2637 ±103%    -100.0%       0.00        interrupts.CPU7.PMI:Performance_monitoring_interrupts
     21803           -20.8%      17265        interrupts.CPU7.RES:Rescheduling_interrupts
      2027 ± 10%     +25.0%       2534        interrupts.CPU7.TLB:TLB_shootdowns
     20582           -20.4%      16392        interrupts.CPU70.RES:Rescheduling_interrupts
      1253 ±141%    +465.2%       7086 ±  2%  interrupts.CPU71.NMI:Non-maskable_interrupts
      1253 ±141%    +465.2%       7086 ±  2%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
      1417 ± 93%    +396.7%       7042        interrupts.CPU72.NMI:Non-maskable_interrupts
      1417 ± 93%    +396.7%       7042        interrupts.CPU72.PMI:Performance_monitoring_interrupts
      1976 ±  9%     +20.7%       2385        interrupts.CPU72.TLB:TLB_shootdowns
     21584 ±  2%     -14.4%      18475        interrupts.CPU73.RES:Rescheduling_interrupts
      1936 ± 11%     +22.0%       2362 ±  2%  interrupts.CPU73.TLB:TLB_shootdowns
     21004 ±  2%     -20.2%      16753        interrupts.CPU74.RES:Rescheduling_interrupts
      2068 ±  6%     +16.7%       2415        interrupts.CPU74.TLB:TLB_shootdowns
     21152           -19.9%      16943 ±  5%  interrupts.CPU75.RES:Rescheduling_interrupts
      2022 ± 12%     +12.2%       2268 ±  4%  interrupts.CPU75.TLB:TLB_shootdowns
      2334 ±119%    +124.7%       5244 ± 31%  interrupts.CPU76.NMI:Non-maskable_interrupts
      2334 ±119%    +124.7%       5244 ± 31%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
     21464           -13.7%      18515        interrupts.CPU76.RES:Rescheduling_interrupts
      1645 ± 75%    +330.1%       7078        interrupts.CPU77.NMI:Non-maskable_interrupts
      1645 ± 75%    +330.1%       7078        interrupts.CPU77.PMI:Performance_monitoring_interrupts
     21447 ±  2%     -18.9%      17392        interrupts.CPU77.RES:Rescheduling_interrupts
     20921 ±  5%     -16.2%      17536        interrupts.CPU78.RES:Rescheduling_interrupts
      1969 ± 10%     +21.8%       2398 ± 11%  interrupts.CPU78.TLB:TLB_shootdowns
    593.33 ±141%    +789.1%       5275 ± 31%  interrupts.CPU79.NMI:Non-maskable_interrupts
    593.33 ±141%    +789.1%       5275 ± 31%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
     20456 ±  3%     -14.8%      17424 ±  2%  interrupts.CPU79.RES:Rescheduling_interrupts
      2007 ±  5%     +18.2%       2373 ±  5%  interrupts.CPU79.TLB:TLB_shootdowns
    477.67 ±  9%    +245.7%       1651 ±  7%  interrupts.CPU8.53:PCI-MSI.1572872-edge.eth0-TxRx-8
      2284 ± 65%    +134.0%       5346 ± 34%  interrupts.CPU8.NMI:Non-maskable_interrupts
      2284 ± 65%    +134.0%       5346 ± 34%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
     20918 ±  4%     -16.7%      17427 ±  4%  interrupts.CPU8.RES:Rescheduling_interrupts
     20186 ±  2%     -20.3%      16087 ±  5%  interrupts.CPU80.RES:Rescheduling_interrupts
      2751 ± 38%     -99.8%       5.50 ± 81%  interrupts.CPU81.NMI:Non-maskable_interrupts
      2751 ± 38%     -99.8%       5.50 ± 81%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
     20462           -25.4%      15275        interrupts.CPU81.RES:Rescheduling_interrupts
      2053 ± 10%     +22.0%       2505 ±  4%  interrupts.CPU81.TLB:TLB_shootdowns
     20707 ±  3%     -19.2%      16737        interrupts.CPU82.RES:Rescheduling_interrupts
      3209 ± 71%    +121.7%       7115 ±  2%  interrupts.CPU83.NMI:Non-maskable_interrupts
      3209 ± 71%    +121.7%       7115 ±  2%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
     20304           -21.3%      15973        interrupts.CPU83.RES:Rescheduling_interrupts
      2031 ± 13%     +15.0%       2337 ±  6%  interrupts.CPU83.TLB:TLB_shootdowns
      2143 ± 38%     -95.8%      91.00 ± 86%  interrupts.CPU84.NMI:Non-maskable_interrupts
      2143 ± 38%     -95.8%      91.00 ± 86%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
     21467 ±  2%     -22.0%      16750        interrupts.CPU84.RES:Rescheduling_interrupts
      2054 ± 14%     +17.9%       2421 ±  4%  interrupts.CPU84.TLB:TLB_shootdowns
     20823 ±  2%     -22.3%      16175        interrupts.CPU85.RES:Rescheduling_interrupts
      2156 ± 12%     +18.4%       2553 ± 16%  interrupts.CPU85.TLB:TLB_shootdowns
      1525 ± 16%    +247.0%       5292 ± 31%  interrupts.CPU86.NMI:Non-maskable_interrupts
      1525 ± 16%    +247.0%       5292 ± 31%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
     20385           -18.1%      16698        interrupts.CPU86.RES:Rescheduling_interrupts
     20292           -21.2%      15997 ±  4%  interrupts.CPU87.RES:Rescheduling_interrupts
      2020 ± 16%     +24.2%       2510 ±  3%  interrupts.CPU87.TLB:TLB_shootdowns
      1071 ± 73%    +402.3%       5380 ± 34%  interrupts.CPU88.NMI:Non-maskable_interrupts
      1071 ± 73%    +402.3%       5380 ± 34%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
     20153 ±  2%     -17.3%      16669        interrupts.CPU88.RES:Rescheduling_interrupts
      2125 ± 12%     +14.3%       2429 ±  4%  interrupts.CPU88.TLB:TLB_shootdowns
     20208           -24.7%      15207 ±  8%  interrupts.CPU89.RES:Rescheduling_interrupts
      2167 ± 12%     +13.9%       2468 ±  3%  interrupts.CPU89.TLB:TLB_shootdowns
      2132 ±141%    +149.4%       5318 ± 35%  interrupts.CPU9.NMI:Non-maskable_interrupts
      2132 ±141%    +149.4%       5318 ± 35%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
     20990 ±  3%     -15.7%      17687        interrupts.CPU9.RES:Rescheduling_interrupts
      2037 ± 13%     +13.3%       2308 ±  6%  interrupts.CPU9.TLB:TLB_shootdowns
      3629 ± 57%     +46.6%       5321 ± 31%  interrupts.CPU90.NMI:Non-maskable_interrupts
      3629 ± 57%     +46.6%       5321 ± 31%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
     22658 ±  3%     -16.2%      18998        interrupts.CPU90.RES:Rescheduling_interrupts
      2055 ± 12%     +16.1%       2386 ±  8%  interrupts.CPU90.TLB:TLB_shootdowns
      1761 ± 89%    +201.8%       5314 ± 34%  interrupts.CPU91.NMI:Non-maskable_interrupts
      1761 ± 89%    +201.8%       5314 ± 34%  interrupts.CPU91.PMI:Performance_monitoring_interrupts
     21470 ±  2%     -14.4%      18381        interrupts.CPU91.RES:Rescheduling_interrupts
      2008 ± 12%     +18.0%       2371 ±  5%  interrupts.CPU91.TLB:TLB_shootdowns
     20890 ±  3%     -18.1%      17101 ±  2%  interrupts.CPU92.RES:Rescheduling_interrupts
      2194 ± 11%     +11.9%       2456 ±  4%  interrupts.CPU92.TLB:TLB_shootdowns
     22379           -15.5%      18910 ±  4%  interrupts.CPU93.RES:Rescheduling_interrupts
      1974 ±  8%     +18.9%       2346 ±  2%  interrupts.CPU93.TLB:TLB_shootdowns
     21107           -20.7%      16740 ±  6%  interrupts.CPU94.RES:Rescheduling_interrupts
     20073           -20.6%      15938        interrupts.CPU95.RES:Rescheduling_interrupts
      2011 ± 11%     +18.4%       2382 ±  2%  interrupts.CPU95.TLB:TLB_shootdowns
     20381           -18.5%      16618 ±  4%  interrupts.CPU96.RES:Rescheduling_interrupts
     20631           -20.2%      16466 ±  3%  interrupts.CPU97.RES:Rescheduling_interrupts
      2062 ±  6%     +13.8%       2347        interrupts.CPU97.TLB:TLB_shootdowns
     20499           -25.1%      15362 ±  2%  interrupts.CPU98.RES:Rescheduling_interrupts
      1521 ± 85%    -100.0%       0.50 ±100%  interrupts.CPU99.NMI:Non-maskable_interrupts
      1521 ± 85%    -100.0%       0.50 ±100%  interrupts.CPU99.PMI:Performance_monitoring_interrupts
     20320           -26.1%      15026        interrupts.CPU99.RES:Rescheduling_interrupts
   3022635           -18.3%    2470301        interrupts.RES:Rescheduling_interrupts
    296038 ± 10%     +15.1%     340639 ±  3%  interrupts.TLB:TLB_shootdowns



***************************************************************************************************
lkp-skl-4sp1: 192 threads Skylake-SP with 256G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase:
  gcc-7/performance/x86_64-rhel-7.6/30%/debian-x86_64-2019-05-14.cgz/300s/lkp-skl-4sp1/shell8/unixbench

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          0:4           45%           2:2     perf-profile.calltrace.cycles-pp.error_entry
          2:4           -2%           2:2     perf-profile.children.cycles-pp.error_entry
          1:4           20%           2:2     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
 4.215e+09 ± 33%     +43.4%  6.046e+09 ± 31%  cpuidle.C1E.time
      0.00 ±135%      +0.0        0.01 ± 80%  mpstat.cpu.all.soft%
      1047 ±155%     -99.5%       5.00        proc-vmstat.numa_hint_faults_local
     30037 ± 45%     -96.4%       1092 ± 64%  proc-vmstat.numa_pte_updates
     32.20 ± 31%     +13.2       45.39 ± 28%  turbostat.C1E%
     62.11 ±  8%      +9.8%      68.22 ±  7%  turbostat.CPU%c1
    516.50 ±  9%     +17.3%     606.00 ±  5%  slabinfo.skbuff_fclone_cache.active_objs
    516.50 ±  9%     +17.3%     606.00 ±  5%  slabinfo.skbuff_fclone_cache.num_objs
     26982 ±  4%      -7.6%      24934        slabinfo.vmap_area.active_objs
     63208 ±  5%      +7.2%      67766 ±  4%  numa-vmstat.node0.nr_file_pages
     13378 ±  9%     -14.6%      11428 ±  2%  numa-vmstat.node1.nr_kernel_stack
      6558 ± 28%     -29.8%       4601        numa-vmstat.node1.nr_page_table_pages
      5421 ± 11%     +20.5%       6535 ±  3%  numa-vmstat.node3.nr_page_table_pages
    272.75 ± 34%    +191.3%     794.50 ± 42%  numa-vmstat.node3.nr_shmem
      6026 ± 23%     +30.4%       7856 ± 17%  numa-vmstat.node3.nr_slab_reclaimable
      3.11 ±  6%      -0.2        2.92        perf-stat.i.branch-miss-rate%
     19.24 ±  6%      +2.3       21.55 ±  8%  perf-stat.i.cache-miss-rate%
     30.43 ±  3%      -1.4       29.05        perf-stat.i.iTLB-load-miss-rate%
     90.26            +0.5       90.81        perf-stat.i.node-load-miss-rate%
     20.05 ±  6%      +2.4       22.49 ± 10%  perf-stat.overall.cache-miss-rate%
     78.86            +0.9       79.75        perf-stat.overall.node-store-miss-rate%
     49116 ± 58%     -98.8%     612.50 ± 58%  numa-meminfo.node0.AnonHugePages
    252747 ±  5%      +7.3%     271178 ±  4%  numa-meminfo.node0.FilePages
      3256 ± 40%     -67.0%       1075 ± 13%  numa-meminfo.node1.Inactive
    829650 ± 10%     -21.7%     649923        numa-meminfo.node1.MemUsed
     26061 ± 29%     -29.2%      18456        numa-meminfo.node1.PageTables
      5095 ±168%    +311.5%      20968 ± 85%  numa-meminfo.node2.AnonHugePages
     24108 ± 23%     +30.3%      31425 ± 17%  numa-meminfo.node3.KReclaimable
     21780 ± 11%     +21.2%      26407 ±  2%  numa-meminfo.node3.PageTables
     24108 ± 23%     +30.3%      31425 ± 17%  numa-meminfo.node3.SReclaimable
      1094 ± 34%    +190.6%       3179 ± 42%  numa-meminfo.node3.Shmem
    100972 ±  8%      +9.4%     110438 ±  6%  numa-meminfo.node3.Slab
      2534 ±100%    +203.6%       7695 ± 29%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    486638 ±100%    +105.9%    1001882 ±  3%  sched_debug.cfs_rq:/.MIN_vruntime.max
     35028 ±100%    +145.6%      86026 ± 13%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
      2534 ±100%    +203.6%       7695 ± 29%  sched_debug.cfs_rq:/.max_vruntime.avg
    486638 ±100%    +105.9%    1001882 ±  3%  sched_debug.cfs_rq:/.max_vruntime.max
     35028 ±100%    +145.6%      86026 ± 13%  sched_debug.cfs_rq:/.max_vruntime.stddev
      0.29 ± 15%     +16.7%       0.34 ±  5%  sched_debug.cfs_rq:/.nr_spread_over.stddev
     10223 ±107%    +180.5%      28676 ± 46%  sched_debug.cfs_rq:/.spread0.avg
     82274 ± 16%     +26.7%     104241 ±  6%  sched_debug.cfs_rq:/.spread0.max
    -41611           -39.3%     -25237        sched_debug.cfs_rq:/.spread0.min
      1.47 ± 72%     -54.7%       0.67 ±  2%  sched_debug.cpu.cpu_load[0].avg
     11.54 ±130%     -78.0%       2.54 ±  3%  sched_debug.cpu.cpu_load[0].stddev
     60.88 ± 26%     +38.8%      84.50 ±  8%  sched_debug.cpu.cpu_load[3].max
      9596 ± 20%     +26.1%      12101 ± 14%  sched_debug.cpu.curr->pid.avg
    280526 ± 91%     -92.8%      20294        sched_debug.cpu.load.max
     50817 ±  6%     +17.6%      59740 ±  6%  sched_debug.cpu.nr_load_updates.max
      1454 ± 11%     +34.6%       1957 ±  7%  sched_debug.cpu.nr_load_updates.stddev
     14922 ±  6%     +12.0%      16718        softirqs.CPU0.SCHED
     17109 ±  2%     +17.3%      20074 ± 11%  softirqs.CPU10.RCU
     16556 ±  2%     +10.4%      18285 ±  4%  softirqs.CPU109.RCU
     10912 ±  4%      +6.6%      11636 ±  3%  softirqs.CPU109.SCHED
     26380 ±  5%     +19.5%      31531 ± 13%  softirqs.CPU109.TIMER
     17052 ±  4%     +34.2%      22878 ± 26%  softirqs.CPU11.RCU
     10926 ±  4%     +27.8%      13963 ± 12%  softirqs.CPU11.SCHED
     27695 ±  5%     +44.6%      40059 ± 23%  softirqs.CPU11.TIMER
     17034            +6.5%      18135 ±  3%  softirqs.CPU119.RCU
     10905 ±  3%      +7.3%      11705 ±  5%  softirqs.CPU119.SCHED
     10900 ±  5%      +5.2%      11462 ±  5%  softirqs.CPU12.SCHED
     10705 ±  3%      +4.6%      11195 ±  4%  softirqs.CPU121.SCHED
     10596 ±  4%      +5.1%      11139 ±  5%  softirqs.CPU138.SCHED
     16059 ±  7%     +14.7%      18424 ± 10%  softirqs.CPU141.RCU
     10472 ±  5%      +5.5%      11044 ±  5%  softirqs.CPU143.SCHED
     10663 ±  4%      +3.7%      11059 ±  4%  softirqs.CPU159.SCHED
     10779 ±  5%     +13.0%      12177 ±  4%  softirqs.CPU180.SCHED
     17166 ±  2%     +10.7%      19002 ±  5%  softirqs.CPU19.RCU
     27980 ±  5%     +15.6%      32334 ±  2%  softirqs.CPU2.TIMER
     16568 ±  3%     +10.7%      18348 ±  3%  softirqs.CPU24.RCU
     16860 ±  2%     +14.6%      19322 ± 10%  softirqs.CPU25.RCU
     10843 ±  2%      +5.7%      11459 ±  3%  softirqs.CPU38.SCHED
     10689 ±  3%      +4.4%      11163 ±  5%  softirqs.CPU46.SCHED
     11435 ± 12%     +16.7%      13347 ± 13%  softirqs.CPU55.SCHED
     28041 ± 11%     +13.1%      31725 ± 10%  softirqs.CPU55.TIMER
     10694 ±  4%     +10.6%      11832 ±  3%  softirqs.CPU61.SCHED
     11175 ±  6%     +10.5%      12354 ±  8%  softirqs.CPU72.SCHED
      0.72 ±115%      +1.1        1.85 ± 28%  perf-profile.calltrace.cycles-pp.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.72 ±115%      +1.1        1.85 ± 28%  perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.72 ±115%      +1.1        1.85 ± 28%  perf-profile.calltrace.cycles-pp.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.72 ±115%      +1.1        1.85 ± 28%  perf-profile.calltrace.cycles-pp.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.47 ±100%      +1.4        1.85 ± 28%  perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      0.43 ±173%      +1.4        1.85 ± 28%  perf-profile.calltrace.cycles-pp.do_dentry_open.path_openat.do_filp_open.do_sys_open.do_syscall_64
      0.00            +1.9        1.85 ± 28%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
     44.74 ± 11%      +8.3       53.00 ±  5%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     47.19 ± 14%     +10.0       57.23        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     47.15 ± 14%     +11.3       58.42        perf-profile.calltrace.cycles-pp.secondary_startup_64
     47.15 ± 14%     +11.3       58.42        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     47.15 ± 14%     +11.3       58.42        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     47.15 ± 14%     +11.3       58.42        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     46.93 ± 14%     +11.5       58.42        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     40.50 ± 19%     -10.9       29.62 ±  3%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     40.29 ± 19%     -10.7       29.62 ±  3%  perf-profile.children.cycles-pp.do_syscall_64
      2.91 ± 47%      -2.2        0.67 ±100%  perf-profile.children.cycles-pp.mutex_lock
      0.72 ±115%      +1.1        1.85 ± 28%  perf-profile.children.cycles-pp.__x64_sys_mprotect
      0.72 ±115%      +1.1        1.85 ± 28%  perf-profile.children.cycles-pp.do_mmap
      0.72 ±115%      +1.1        1.85 ± 28%  perf-profile.children.cycles-pp.do_mprotect_pkey
      0.72 ±115%      +1.1        1.85 ± 28%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      0.72 ±115%      +1.1        1.85 ± 28%  perf-profile.children.cycles-pp.mprotect_fixup
      0.72 ±115%      +1.1        1.85 ± 28%  perf-profile.children.cycles-pp.vm_mmap_pgoff
      0.47 ±100%      +1.4        1.85 ± 28%  perf-profile.children.cycles-pp.mmap_region
      0.43 ±173%      +1.4        1.85 ± 28%  perf-profile.children.cycles-pp.do_dentry_open
      0.00            +1.9        1.85 ± 28%  perf-profile.children.cycles-pp.clockevents_program_event
      0.00            +3.7        3.71 ± 28%  perf-profile.children.cycles-pp._raw_spin_lock
     44.74 ± 11%      +8.3       53.00 ±  5%  perf-profile.children.cycles-pp.intel_idle
     47.15 ± 14%     +11.3       58.42        perf-profile.children.cycles-pp.secondary_startup_64
     47.15 ± 14%     +11.3       58.42        perf-profile.children.cycles-pp.start_secondary
     47.15 ± 14%     +11.3       58.42        perf-profile.children.cycles-pp.cpu_startup_entry
     47.15 ± 14%     +11.3       58.42        perf-profile.children.cycles-pp.do_idle
     46.93 ± 14%     +11.5       58.42        perf-profile.children.cycles-pp.cpuidle_enter
     46.93 ± 14%     +11.5       58.42        perf-profile.children.cycles-pp.cpuidle_enter_state
      2.91 ± 47%      -2.2        0.67 ±100%  perf-profile.self.cycles-pp.mutex_lock
      0.00            +3.7        3.71 ± 28%  perf-profile.self.cycles-pp._raw_spin_lock
     44.74 ± 11%      +8.3       53.00 ±  5%  perf-profile.self.cycles-pp.intel_idle
      1846 ±  5%     +11.6%       2060 ±  7%  interrupts.CPU0.TLB:TLB_shootdowns
      2000 ±  2%     +10.2%       2205 ±  7%  interrupts.CPU1.TLB:TLB_shootdowns
      2086 ±  4%      +9.8%       2291 ±  8%  interrupts.CPU100.TLB:TLB_shootdowns
      2030 ±  5%     +12.5%       2285 ±  8%  interrupts.CPU103.TLB:TLB_shootdowns
    746.75 ±  8%     -14.1%     641.50 ±  3%  interrupts.CPU104.RES:Rescheduling_interrupts
      1989 ±  5%     +13.6%       2259 ±  4%  interrupts.CPU107.TLB:TLB_shootdowns
      2014 ±  5%     +11.6%       2247 ±  3%  interrupts.CPU108.TLB:TLB_shootdowns
      2024 ±  4%      +9.7%       2222 ±  4%  interrupts.CPU109.TLB:TLB_shootdowns
      2008 ±  2%     +16.0%       2330 ±  6%  interrupts.CPU11.TLB:TLB_shootdowns
      2013 ±  4%      +9.9%       2212 ±  7%  interrupts.CPU113.TLB:TLB_shootdowns
      1967 ±  5%     +14.0%       2243 ±  4%  interrupts.CPU114.TLB:TLB_shootdowns
      1989 ±  5%      +9.6%       2180 ±  8%  interrupts.CPU117.TLB:TLB_shootdowns
      2015 ±  2%     +10.0%       2216 ±  4%  interrupts.CPU118.TLB:TLB_shootdowns
      2006 ±  5%     +14.0%       2286 ±  4%  interrupts.CPU119.TLB:TLB_shootdowns
    899.25 ±  4%     +11.0%     998.00 ±  3%  interrupts.CPU120.RES:Rescheduling_interrupts
      1881 ±  6%     +12.9%       2124 ±  8%  interrupts.CPU120.TLB:TLB_shootdowns
      2070 ±  6%      +7.9%       2233 ±  6%  interrupts.CPU121.TLB:TLB_shootdowns
    134762 ±  4%     +16.7%     157232 ±  4%  interrupts.CPU128.LOC:Local_timer_interrupts
      1999 ±  4%     +12.2%       2244 ± 10%  interrupts.CPU128.TLB:TLB_shootdowns
      2039 ±  8%      +8.4%       2212 ±  6%  interrupts.CPU129.TLB:TLB_shootdowns
      2019 ±  5%     +17.3%       2369 ±  7%  interrupts.CPU13.TLB:TLB_shootdowns
    133696 ±  4%      +6.1%     141813 ±  4%  interrupts.CPU130.LOC:Local_timer_interrupts
      2071 ±  7%      +8.0%       2236 ±  8%  interrupts.CPU131.TLB:TLB_shootdowns
      2063 ±  4%     +10.0%       2270 ±  6%  interrupts.CPU135.TLB:TLB_shootdowns
      2033 ±  5%     +11.5%       2266 ±  6%  interrupts.CPU136.TLB:TLB_shootdowns
    132973 ±  4%      +6.2%     141217 ±  4%  interrupts.CPU139.LOC:Local_timer_interrupts
      2041 ±  6%     +10.9%       2265 ±  9%  interrupts.CPU140.TLB:TLB_shootdowns
      2036 ±  5%     +10.8%       2255 ±  8%  interrupts.CPU141.TLB:TLB_shootdowns
      2023 ±  7%     +11.9%       2263 ±  5%  interrupts.CPU143.TLB:TLB_shootdowns
      2030 ±  6%     +10.0%       2233 ±  9%  interrupts.CPU145.TLB:TLB_shootdowns
      1991 ±  4%     +18.5%       2360 ±  5%  interrupts.CPU159.TLB:TLB_shootdowns
      2091 ±  5%      +7.9%       2256 ±  7%  interrupts.CPU160.TLB:TLB_shootdowns
      2070 ±  4%     +10.6%       2290 ±  5%  interrupts.CPU161.TLB:TLB_shootdowns
      2064 ±  7%     +11.9%       2310 ±  5%  interrupts.CPU162.TLB:TLB_shootdowns
      2062 ±  5%     +13.2%       2336 ±  7%  interrupts.CPU163.TLB:TLB_shootdowns
      2064 ±  8%     +12.7%       2326 ± 10%  interrupts.CPU165.TLB:TLB_shootdowns
      2033 ±  6%     +13.2%       2302 ±  9%  interrupts.CPU166.TLB:TLB_shootdowns
      2088 ±  6%      +9.7%       2292 ±  5%  interrupts.CPU167.TLB:TLB_shootdowns
      2004 ±  6%      +9.8%       2201 ± 10%  interrupts.CPU174.TLB:TLB_shootdowns
      2047 ±  2%     +12.2%       2297 ±  9%  interrupts.CPU176.TLB:TLB_shootdowns
    763.00 ±  6%     -10.4%     683.50        interrupts.CPU181.RES:Rescheduling_interrupts
      2087 ±  7%      +7.5%       2244 ±  7%  interrupts.CPU182.TLB:TLB_shootdowns
      2101 ±  3%     +11.5%       2342 ±  9%  interrupts.CPU186.TLB:TLB_shootdowns
      2009 ±  4%     +14.4%       2299 ±  9%  interrupts.CPU187.TLB:TLB_shootdowns
      2073 ±  2%      +6.2%       2201 ±  6%  interrupts.CPU189.TLB:TLB_shootdowns
      2070 ±  9%      +8.5%       2246 ±  6%  interrupts.CPU190.TLB:TLB_shootdowns
      1995 ±  4%     +11.4%       2223 ±  5%  interrupts.CPU23.TLB:TLB_shootdowns
    890.25 ±  5%      +8.4%     965.00 ±  4%  interrupts.CPU24.RES:Rescheduling_interrupts
    133004 ±  4%      +6.7%     141981 ±  5%  interrupts.CPU35.LOC:Local_timer_interrupts
      2057 ±  6%      +9.7%       2257 ±  4%  interrupts.CPU36.TLB:TLB_shootdowns
      2033 ±  3%     +10.0%       2237 ±  7%  interrupts.CPU37.TLB:TLB_shootdowns
    133340 ±  5%      +5.9%     141238 ±  4%  interrupts.CPU42.LOC:Local_timer_interrupts
    134080 ±  5%      +6.2%     142377 ±  5%  interrupts.CPU44.LOC:Local_timer_interrupts
      2075 ±  2%     +12.6%       2335 ±  6%  interrupts.CPU44.TLB:TLB_shootdowns
      2018 ±  4%     +16.7%       2355 ±  9%  interrupts.CPU45.TLB:TLB_shootdowns
      2029 ±  4%     +16.1%       2356 ±  3%  interrupts.CPU5.TLB:TLB_shootdowns
      2125 ±  4%      +8.0%       2296 ±  4%  interrupts.CPU50.TLB:TLB_shootdowns
      2054            +8.6%       2231 ±  7%  interrupts.CPU54.TLB:TLB_shootdowns
      2043 ±  5%     +12.9%       2306 ±  9%  interrupts.CPU56.TLB:TLB_shootdowns
      2098 ±  5%      +8.4%       2274 ±  2%  interrupts.CPU61.TLB:TLB_shootdowns
      2027 ±  2%     +13.7%       2306 ±  7%  interrupts.CPU63.TLB:TLB_shootdowns
      2018 ±  9%     +13.1%       2283 ±  7%  interrupts.CPU66.TLB:TLB_shootdowns
    772.25 ±  6%     -10.1%     694.50        interrupts.CPU69.RES:Rescheduling_interrupts
      2050 ±  5%     +12.8%       2312 ± 10%  interrupts.CPU70.TLB:TLB_shootdowns
      2114 ±  3%      +8.5%       2295 ±  6%  interrupts.CPU74.TLB:TLB_shootdowns
      2179 ±  6%      +8.4%       2361 ±  5%  interrupts.CPU86.TLB:TLB_shootdowns
      2043 ±  5%     +11.3%       2273 ±  7%  interrupts.CPU9.TLB:TLB_shootdowns
      2078 ±  2%     +10.8%       2301 ±  7%  interrupts.CPU90.TLB:TLB_shootdowns
      2036 ±  4%      +8.2%       2204 ±  6%  interrupts.CPU92.TLB:TLB_shootdowns
      2063 ±  8%     +11.0%       2289 ±  9%  interrupts.CPU93.TLB:TLB_shootdowns
      2008 ±  5%     +10.7%       2223 ±  6%  interrupts.CPU96.TLB:TLB_shootdowns
      1985 ±  6%     +11.7%       2218 ±  7%  interrupts.CPU98.TLB:TLB_shootdowns
    395318 ±  4%      +8.3%     428205 ±  8%  interrupts.TLB:TLB_shootdowns



***************************************************************************************************
lkp-skl-4sp1: 192 threads Skylake-SP with 256G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
  gcc-7/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-2019-05-14.cgz/lkp-skl-4sp1/pwrite2/will-it-scale

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    407.25 ±  5%     -21.8%     318.50 ± 16%  kthread_noise.total_time
      0.00 ±  6%      +0.0        0.00 ± 61%  mpstat.cpu.all.soft%
      0.05 ±  3%      +0.0        0.06 ±  9%  mpstat.cpu.all.usr%
      6.41 ± 12%     -36.3%       4.08 ± 26%  irq_exception_noise.__do_page_fault.95th
     11.54 ±  5%     -30.4%       8.02 ±  6%  irq_exception_noise.__do_page_fault.99th
    201126 ±  4%     +35.7%     272858 ± 24%  irq_exception_noise.softirq_time
     25808            +1.6%      26225        proc-vmstat.nr_slab_reclaimable
    472.50 ± 56%    +628.3%       3441 ± 80%  proc-vmstat.numa_hint_faults_local
   1555625            -2.4%    1517554        proc-vmstat.numa_hit
   1410822 ±  2%      -2.7%    1372898        proc-vmstat.numa_local
      2706 ±  7%     +13.1%       3060 ±  8%  proc-vmstat.pgactivate
     67.00           +28.4%      86.00 ± 19%  turbostat.Avg_MHz
      4.69            +1.2        5.85 ± 18%  turbostat.Busy%
      0.09 ± 28%   +1883.3%       1.79 ± 92%  turbostat.Pkg%pc2
     37.00            +5.4%      39.00        turbostat.PkgTmp
    191.96            +4.7%     200.91 ±  3%  turbostat.PkgWatt
     52.35           +17.2%      61.37 ± 12%  turbostat.RAMWatt
     16717 ±  5%     +11.2%      18597        slabinfo.cred_jar.active_objs
     16717 ±  5%     +11.2%      18597        slabinfo.cred_jar.num_objs
      4493 ±  3%     -10.8%       4007 ±  2%  slabinfo.eventpoll_pwq.active_objs
      4493 ±  3%     -10.8%       4007 ±  2%  slabinfo.eventpoll_pwq.num_objs
     51900 ±  4%      +6.3%      55166 ±  3%  slabinfo.filp.active_objs
     52293 ±  4%      +6.9%      55879 ±  3%  slabinfo.filp.num_objs
      1299 ±  6%     +11.8%       1453 ±  4%  slabinfo.task_group.active_objs
      1299 ±  6%     +11.8%       1453 ±  4%  slabinfo.task_group.num_objs
     10619 ± 85%     -95.4%     485.50 ± 63%  numa-meminfo.node1.Inactive(anon)
      1041 ±112%    +892.0%      10332 ± 75%  numa-meminfo.node2.Inactive
    320.75 ± 34%   +3121.2%      10332 ± 75%  numa-meminfo.node2.Inactive(anon)
     15907 ± 21%     +60.7%      25570 ± 17%  numa-meminfo.node2.KReclaimable
      6626 ±  6%     +15.9%       7683 ±  7%  numa-meminfo.node2.KernelStack
      8908 ±  8%     +53.3%      13657 ± 31%  numa-meminfo.node2.PageTables
     15907 ± 21%     +60.7%      25570 ± 17%  numa-meminfo.node2.SReclaimable
     53362 ± 10%     +25.5%      66971        numa-meminfo.node2.SUnreclaim
    927.75 ± 95%   +1203.5%      12093 ± 53%  numa-meminfo.node2.Shmem
     69269 ± 12%     +33.6%      92541 ±  5%  numa-meminfo.node2.Slab
      2654 ± 85%     -95.4%     121.00 ± 63%  numa-vmstat.node1.nr_inactive_anon
      2161 ± 23%     -40.1%       1294        numa-vmstat.node1.nr_mapped
      2654 ± 85%     -95.4%     121.00 ± 63%  numa-vmstat.node1.nr_zone_inactive_anon
     79.75 ± 34%   +3138.2%       2582 ± 75%  numa-vmstat.node2.nr_inactive_anon
      6622 ±  6%     +15.9%       7676 ±  7%  numa-vmstat.node2.nr_kernel_stack
      1465 ± 21%     +38.6%       2030 ± 30%  numa-vmstat.node2.nr_mapped
      2224 ±  8%     +53.4%       3412 ± 30%  numa-vmstat.node2.nr_page_table_pages
    231.25 ± 95%   +1207.0%       3022 ± 53%  numa-vmstat.node2.nr_shmem
      3976 ± 21%     +60.8%       6392 ± 17%  numa-vmstat.node2.nr_slab_reclaimable
     13340 ± 10%     +25.5%      16744        numa-vmstat.node2.nr_slab_unreclaimable
     79.75 ± 34%   +3138.2%       2582 ± 75%  numa-vmstat.node2.nr_zone_inactive_anon
    465714 ± 32%     +62.3%     755809 ± 36%  numa-vmstat.node2.numa_hit
    336044 ± 46%     +90.6%     640510 ± 46%  numa-vmstat.node2.numa_local
      2005 ±  2%     +11.6%       2238 ± 10%  sched_debug.cfs_rq:/.exec_clock.avg
     14.00 ±  4%     -12.0%      12.31        sched_debug.cpu.clock.stddev
     14.00 ±  4%     -12.0%      12.31        sched_debug.cpu.clock_task.stddev
      1.37 ±  4%     +17.5%       1.61 ±  4%  sched_debug.cpu.cpu_load[1].avg
    199.10 ±  2%     +24.5%     247.97 ±  3%  sched_debug.cpu.cpu_load[1].max
     14.80 ±  2%     +21.4%      17.97 ±  4%  sched_debug.cpu.cpu_load[1].stddev
      1.31 ±  5%     +11.0%       1.46 ±  2%  sched_debug.cpu.cpu_load[2].avg
    195.25 ±  2%     +13.3%     221.23        sched_debug.cpu.cpu_load[2].max
     14.48 ±  2%     +10.9%      16.06        sched_debug.cpu.cpu_load[2].stddev
      5483 ± 15%    +201.0%      16507 ± 60%  sched_debug.cpu.max_idle_balance_cost.stddev
      0.00 ±  8%     -11.9%       0.00        sched_debug.cpu.next_balance.stddev
    130966           +11.5%     146000 ± 10%  sched_debug.cpu.nr_load_updates.avg
    139797           +11.3%     155637 ±  9%  sched_debug.cpu.nr_load_updates.max
    127184           +12.5%     143112 ±  9%  sched_debug.cpu.nr_load_updates.min
      2.20            +3.6        5.82 ± 61%  perf-stat.i.branch-miss-rate%
  17065871          +128.2%   38940396 ± 54%  perf-stat.i.branch-misses
  15171379 ±  4%     -11.5%   13428822 ±  5%  perf-stat.i.cache-misses
      3.61           +34.7%       4.87 ± 23%  perf-stat.i.cpi
 1.207e+10           +26.4%  1.525e+10 ± 18%  perf-stat.i.cpu-cycles
    873.59 ± 11%     +32.5%       1157 ± 22%  perf-stat.i.cycles-between-cache-misses
      0.00 ±  4%      +0.1        0.09 ± 94%  perf-stat.i.dTLB-store-miss-rate%
     24641 ±  3%   +1481.0%     389570 ± 92%  perf-stat.i.dTLB-store-misses
      0.28           -21.4%       0.22 ± 23%  perf-stat.i.ipc
     33574 ±  6%     +17.3%      39391 ±  9%  perf-stat.i.node-loads
   1501402 ± 11%     -16.1%    1260418 ±  4%  perf-stat.i.node-store-misses
      2.35            +3.5        5.82 ± 58%  perf-stat.overall.branch-miss-rate%
      3.54           +33.5%       4.72 ± 23%  perf-stat.overall.cpi
    798.25 ±  5%     +44.0%       1149 ± 23%  perf-stat.overall.cycles-between-cache-misses
      0.00 ±  3%      +0.1        0.08 ± 93%  perf-stat.overall.dTLB-store-miss-rate%
      0.28           -20.8%       0.22 ± 23%  perf-stat.overall.ipc
  17098757          +127.2%   38846297 ± 54%  perf-stat.ps.branch-misses
  15119630 ±  4%     -11.5%   13383715 ±  5%  perf-stat.ps.cache-misses
 1.204e+10           +26.3%   1.52e+10 ± 17%  perf-stat.ps.cpu-cycles
     24654 ±  3%   +1475.3%     388374 ± 92%  perf-stat.ps.dTLB-store-misses
     33486 ±  6%     +17.3%      39294 ±  8%  perf-stat.ps.node-loads
   1496284 ± 11%     -16.1%    1256127 ±  4%  perf-stat.ps.node-store-misses
     19.96 ± 21%      -7.3       12.65 ± 10%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_pwrite64.do_syscall_64.entry_SYSCALL_64_after_hwframe
     20.20 ± 21%      -7.3       12.89 ±  9%  perf-profile.calltrace.cycles-pp.vfs_write.ksys_pwrite64.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_pwrite
     20.25 ± 21%      -7.3       12.95 ±  9%  perf-profile.calltrace.cycles-pp.ksys_pwrite64.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_pwrite
     20.35 ± 21%      -7.3       13.05 ±  9%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_pwrite
     20.36 ± 21%      -7.3       13.07 ±  9%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_pwrite
     19.85 ± 21%      -7.3       12.56 ± 10%  perf-profile.calltrace.cycles-pp.generic_file_write_iter.new_sync_write.vfs_write.ksys_pwrite64.do_syscall_64
     20.92 ± 20%      -7.3       13.66 ±  8%  perf-profile.calltrace.cycles-pp.__libc_pwrite
     14.24 ± 31%      -6.7        7.52 ±  9%  perf-profile.calltrace.cycles-pp.down_write.generic_file_write_iter.new_sync_write.vfs_write.ksys_pwrite64
     13.94 ± 32%      -6.7        7.26 ±  8%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.generic_file_write_iter.new_sync_write.vfs_write
      0.54 ±  4%      +0.0        0.58 ±  2%  perf-profile.calltrace.cycles-pp.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task
      1.04 ± 12%      +0.5        1.50        perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.00            +0.5        0.53 ±  3%  perf-profile.calltrace.cycles-pp.arch_stack_walk.stack_trace_save_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
      1.23 ± 11%      +0.7        1.88 ±  8%  perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      3.32 ± 14%      +0.7        4.05 ±  9%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.06 ±  9%      +0.8        1.86        perf-profile.calltrace.cycles-pp.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.00            +0.9        0.93 ± 11%  perf-profile.calltrace.cycles-pp.ktime_get.perf_mux_hrtimer_handler.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      4.92 ± 12%      +2.2        7.08 ± 12%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      9.38 ± 14%      +2.8       12.21 ± 10%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
     20.57 ± 10%      +3.0       23.55 ±  8%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     17.96 ± 10%      +3.0       21.00 ±  9%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
     43.67 ±  5%      +5.1       48.73 ±  7%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     77.15 ±  7%      +8.3       85.41        perf-profile.calltrace.cycles-pp.secondary_startup_64
     76.62 ±  7%      +8.4       85.03        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     76.62 ±  7%      +8.4       85.03        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     76.58 ±  7%      +8.4       85.00        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     64.90 ±  6%      +8.9       73.77 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     64.56 ±  6%      +9.0       73.60 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     21.55 ± 26%      -8.2       13.31 ±  9%  perf-profile.children.cycles-pp.do_syscall_64
     21.57 ± 26%      -8.2       13.33 ±  9%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     19.97 ± 21%      -7.3       12.66 ± 10%  perf-profile.children.cycles-pp.new_sync_write
     20.25 ± 21%      -7.3       12.95 ±  9%  perf-profile.children.cycles-pp.ksys_pwrite64
     20.21 ± 21%      -7.3       12.91 ±  9%  perf-profile.children.cycles-pp.vfs_write
     19.85 ± 21%      -7.3       12.56 ± 10%  perf-profile.children.cycles-pp.generic_file_write_iter
     20.96 ± 20%      -7.3       13.70 ±  8%  perf-profile.children.cycles-pp.__libc_pwrite
     14.29 ± 32%      -6.8        7.52 ±  9%  perf-profile.children.cycles-pp.down_write
     14.02 ± 32%      -6.7        7.29 ±  8%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      2.11 ± 35%      -0.7        1.39 ± 15%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.47 ± 17%      -0.2        0.30 ±  4%  perf-profile.children.cycles-pp.arch_cpu_idle_enter
      0.41 ± 19%      -0.2        0.25 ±  4%  perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
      0.53 ±  8%      -0.1        0.41 ± 14%  perf-profile.children.cycles-pp.update_load_avg
      0.22 ± 11%      -0.1        0.15 ± 20%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.47 ±  3%      -0.1        0.42 ±  6%  perf-profile.children.cycles-pp.run_timer_softirq
      0.10 ±  8%      +0.0        0.13 ±  7%  perf-profile.children.cycles-pp.__orc_find
      0.06 ± 11%      +0.0        0.09 ± 11%  perf-profile.children.cycles-pp.get_cpu_device
      0.14 ±  6%      +0.0        0.17 ±  5%  perf-profile.children.cycles-pp.rcu_irq_exit
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.update_rt_rq_load_avg
      0.08 ± 12%      +0.0        0.12 ± 19%  perf-profile.children.cycles-pp.intel_pmu_disable_all
      0.00            +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.rcu_needs_cpu
      0.58 ±  4%      +0.1        0.67 ±  8%  perf-profile.children.cycles-pp.sched_clock
      0.41 ±  6%      +0.1        0.49 ± 11%  perf-profile.children.cycles-pp.unlock_page
      0.56 ±  4%      +0.1        0.65 ±  9%  perf-profile.children.cycles-pp.native_sched_clock
      0.05 ±  8%      +0.1        0.15 ± 58%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.53 ±  5%      +0.1        0.66 ± 15%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.11 ±  4%      +0.2        0.27 ± 32%  perf-profile.children.cycles-pp.rcu_irq_enter
      0.03 ±100%      +0.2        0.18 ± 67%  perf-profile.children.cycles-pp.hrtimer_forward
      0.06 ± 14%      +0.3        0.33 ± 78%  perf-profile.children.cycles-pp.tick_check_broadcast_expired
      1.06 ± 12%      +0.5        1.52        perf-profile.children.cycles-pp.tick_irq_enter
      1.26 ± 11%      +0.7        1.93 ±  8%  perf-profile.children.cycles-pp.irq_enter
      1.09 ±  9%      +0.8        1.92        perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      5.15 ± 10%      +2.1        7.21 ± 12%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      9.72 ± 12%      +2.7       12.41 ± 10%  perf-profile.children.cycles-pp.hrtimer_interrupt
     18.39 ± 10%      +2.9       21.27 ±  9%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
     19.78 ± 10%      +2.9       22.66 ±  8%  perf-profile.children.cycles-pp.apic_timer_interrupt
     43.98 ±  4%      +5.0       48.98 ±  7%  perf-profile.children.cycles-pp.intel_idle
     77.21 ±  7%      +8.2       85.46        perf-profile.children.cycles-pp.do_idle
     77.15 ±  7%      +8.3       85.41        perf-profile.children.cycles-pp.secondary_startup_64
     77.15 ±  7%      +8.3       85.41        perf-profile.children.cycles-pp.cpu_startup_entry
     76.62 ±  7%      +8.4       85.03        perf-profile.children.cycles-pp.start_secondary
     65.36 ±  6%      +8.7       74.10 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter
     65.33 ±  6%      +8.8       74.08 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter_state
      0.39 ± 19%      -0.2        0.20        perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
      0.47 ±  9%      -0.1        0.32 ± 10%  perf-profile.self.cycles-pp.__softirqentry_text_start
      0.23 ±  4%      -0.1        0.14 ± 42%  perf-profile.self.cycles-pp.shmem_write_end
      0.20 ± 10%      -0.1        0.14 ± 21%  perf-profile.self.cycles-pp.__update_load_avg_se
      0.22 ± 10%      -0.1        0.16 ±  6%  perf-profile.self.cycles-pp.trace_event_raw_event_softirq
      0.07 ± 17%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.intel_pmu_disable_all
      0.04 ± 58%      +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.hrtimer_next_event_without
      0.10 ±  8%      +0.0        0.12 ± 12%  perf-profile.self.cycles-pp.__orc_find
      0.23            +0.0        0.26 ±  3%  perf-profile.self.cycles-pp.lapic_next_event
      0.13 ±  5%      +0.0        0.17 ±  5%  perf-profile.self.cycles-pp.rcu_irq_exit
      0.04 ± 59%      +0.0        0.09 ± 11%  perf-profile.self.cycles-pp.get_cpu_device
      0.28 ± 10%      +0.1        0.35 ± 17%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.00            +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.rcu_needs_cpu
      0.40 ±  5%      +0.1        0.48 ± 11%  perf-profile.self.cycles-pp.unlock_page
      0.05 ±  8%      +0.1        0.15 ± 58%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.07 ± 11%      +0.1        0.17 ± 41%  perf-profile.self.cycles-pp.scheduler_tick
      0.15 ± 17%      +0.1        0.24 ± 26%  perf-profile.self.cycles-pp.perf_mux_hrtimer_handler
      0.01 ±173%      +0.1        0.11 ± 54%  perf-profile.self.cycles-pp.timerqueue_del
      0.01 ±173%      +0.1        0.12 ± 47%  perf-profile.self.cycles-pp.irq_enter
      0.01 ±173%      +0.1        0.12 ± 58%  perf-profile.self.cycles-pp.sched_clock
      0.13 ±  5%      +0.1        0.27 ± 47%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.00            +0.1        0.14 ± 44%  perf-profile.self.cycles-pp.rcu_irq_enter
      0.00            +0.2        0.17 ± 65%  perf-profile.self.cycles-pp.hrtimer_forward
      0.04 ± 58%      +0.3        0.32 ± 81%  perf-profile.self.cycles-pp.tick_check_broadcast_expired
     43.97 ±  4%      +4.9       48.83 ±  7%  perf-profile.self.cycles-pp.intel_idle
     31419 ± 15%     -24.2%      23802        softirqs.CPU1.RCU
     30103 ± 22%     -25.2%      22515 ±  3%  softirqs.CPU100.RCU
     30145 ± 22%     -25.7%      22383 ±  3%  softirqs.CPU101.RCU
     29952 ± 21%     -24.9%      22503 ±  3%  softirqs.CPU102.RCU
     29935 ± 22%     -25.4%      22342 ±  4%  softirqs.CPU103.RCU
     30697 ± 19%     -27.5%      22263 ±  4%  softirqs.CPU104.RCU
     29884 ± 22%     -22.8%      23067        softirqs.CPU105.RCU
     30351 ± 22%     -26.2%      22407 ±  4%  softirqs.CPU106.RCU
     30122 ± 22%     -25.9%      22317 ±  3%  softirqs.CPU107.RCU
     30203 ± 21%     -26.8%      22111 ±  5%  softirqs.CPU108.RCU
     29667 ± 22%     -25.0%      22235 ±  3%  softirqs.CPU109.RCU
     30310 ± 25%     -26.3%      22324 ±  2%  softirqs.CPU11.RCU
     29722 ± 22%     -25.9%      22034 ±  3%  softirqs.CPU110.RCU
     30799 ± 19%     -28.2%      22126 ±  3%  softirqs.CPU111.RCU
     29287 ± 22%     -27.6%      21205 ±  5%  softirqs.CPU112.RCU
     29356 ± 23%     -27.8%      21186 ±  7%  softirqs.CPU113.RCU
     29350 ± 23%     -29.4%      20727 ±  9%  softirqs.CPU114.RCU
     29271 ± 22%     -29.2%      20733 ± 10%  softirqs.CPU115.RCU
     29889 ± 22%     -28.7%      21315 ±  6%  softirqs.CPU116.RCU
     29521 ± 24%     -28.1%      21217 ±  6%  softirqs.CPU117.RCU
     29332 ± 22%     -27.3%      21338 ±  6%  softirqs.CPU118.RCU
     29485 ± 22%     -27.9%      21262 ±  6%  softirqs.CPU119.RCU
     29350 ± 21%     -24.5%      22155 ±  3%  softirqs.CPU12.RCU
     29902 ± 26%     -26.4%      21994        softirqs.CPU129.RCU
     29654 ± 21%     -25.6%      22076 ±  2%  softirqs.CPU13.RCU
     29163 ± 21%     -28.0%      21003 ±  8%  softirqs.CPU132.RCU
     28891 ± 21%     -25.2%      21622 ±  2%  softirqs.CPU141.RCU
     29024 ± 25%     -27.7%      20971 ±  3%  softirqs.CPU146.RCU
     29658 ± 24%     -26.6%      21768 ±  2%  softirqs.CPU149.RCU
     29962 ± 19%     -24.4%      22650        softirqs.CPU15.RCU
     29597 ± 23%     -26.4%      21786 ±  2%  softirqs.CPU153.RCU
     29312 ± 24%     -26.4%      21562 ±  3%  softirqs.CPU154.RCU
     29587 ± 23%     -26.7%      21700 ±  2%  softirqs.CPU155.RCU
     31189 ± 18%     -29.1%      22113 ±  4%  softirqs.CPU16.RCU
     28949 ± 24%     -28.6%      20677 ±  3%  softirqs.CPU160.RCU
     29190 ± 23%     -29.5%      20579 ±  4%  softirqs.CPU161.RCU
     29015 ± 25%     -32.1%      19710 ±  6%  softirqs.CPU162.RCU
     28928 ± 24%     -28.8%      20599 ±  3%  softirqs.CPU164.RCU
     28777 ± 24%     -29.7%      20239 ±  4%  softirqs.CPU165.RCU
     28013 ± 23%     -29.2%      19826 ±  8%  softirqs.CPU168.RCU
     28886 ± 23%     -28.7%      20589 ±  7%  softirqs.CPU169.RCU
     28894 ± 23%     -29.3%      20436 ±  9%  softirqs.CPU170.RCU
     29063 ± 23%     -29.4%      20528 ±  7%  softirqs.CPU172.RCU
     29041 ± 21%     -29.7%      20408 ±  7%  softirqs.CPU174.RCU
     29317 ± 22%     -30.1%      20500 ±  7%  softirqs.CPU175.RCU
     28672 ± 23%     -25.0%      21504 ±  3%  softirqs.CPU178.RCU
     28472 ± 23%     -27.3%      20700 ±  5%  softirqs.CPU181.RCU
     28458 ± 23%     -25.6%      21161 ±  3%  softirqs.CPU182.RCU
     28616 ± 23%     -26.5%      21035 ±  4%  softirqs.CPU183.RCU
     28791 ± 23%     -25.9%      21339 ±  3%  softirqs.CPU184.RCU
     29064 ± 22%     -26.4%      21381 ±  4%  softirqs.CPU185.RCU
     28681 ± 23%     -27.2%      20885 ±  4%  softirqs.CPU186.RCU
     28700 ± 23%     -26.3%      21163 ±  3%  softirqs.CPU187.RCU
     28551 ± 23%     -26.5%      20974        softirqs.CPU188.RCU
     28521 ± 23%     -25.6%      21214 ±  2%  softirqs.CPU189.RCU
     29479 ± 22%     -27.4%      21403 ±  7%  softirqs.CPU19.RCU
     28570 ± 24%     -25.6%      21255 ±  3%  softirqs.CPU190.RCU
     28954 ± 22%     -26.3%      21340 ±  3%  softirqs.CPU191.RCU
     29496 ± 22%     -26.8%      21604 ±  6%  softirqs.CPU20.RCU
     29924 ± 23%     -28.0%      21549 ±  6%  softirqs.CPU21.RCU
     29499 ± 21%     -26.2%      21768 ±  6%  softirqs.CPU23.RCU
     29133 ± 20%     -25.2%      21795        softirqs.CPU33.RCU
    103342            +9.2%     112860        softirqs.CPU33.TIMER
     29694 ± 21%     -24.0%      22576 ±  3%  softirqs.CPU4.RCU
     29369 ± 19%     -23.8%      22377 ±  2%  softirqs.CPU45.RCU
     30046 ± 23%     -28.7%      21416 ±  6%  softirqs.CPU49.RCU
     29434 ± 22%     -24.1%      22330 ±  2%  softirqs.CPU5.RCU
     29605 ± 23%     -26.1%      21874 ±  2%  softirqs.CPU54.RCU
     29537 ± 22%     -24.8%      22222        softirqs.CPU55.RCU
     29138 ± 23%     -24.1%      22112        softirqs.CPU56.RCU
     29665 ± 23%     -25.5%      22090        softirqs.CPU57.RCU
     29672 ± 23%     -25.6%      22074 ±  2%  softirqs.CPU58.RCU
     29388 ± 21%     -23.8%      22407 ±  3%  softirqs.CPU6.RCU
     29921 ± 23%     -27.7%      21641 ±  4%  softirqs.CPU60.RCU
     29906 ± 24%     -27.5%      21696 ±  2%  softirqs.CPU61.RCU
     29619 ± 23%     -25.5%      22072 ±  2%  softirqs.CPU62.RCU
     29673 ± 23%     -25.7%      22053 ±  2%  softirqs.CPU63.RCU
     29114 ± 22%     -24.4%      21998 ±  2%  softirqs.CPU64.RCU
     29106 ± 23%     -28.4%      20827 ±  2%  softirqs.CPU65.RCU
     28473 ± 22%     -29.9%      19972 ±  4%  softirqs.CPU66.RCU
     28763 ± 24%     -27.8%      20761 ±  2%  softirqs.CPU67.RCU
     28660 ± 23%     -27.2%      20876 ±  2%  softirqs.CPU68.RCU
     28962 ± 23%     -29.5%      20431 ±  3%  softirqs.CPU69.RCU
     28894 ± 24%     -27.2%      21033 ±  2%  softirqs.CPU70.RCU
     29887 ± 28%     -28.4%      21401        softirqs.CPU71.RCU
     28072 ± 20%     -28.8%      19980 ±  7%  softirqs.CPU72.RCU
     28418 ± 23%     -27.7%      20550 ±  6%  softirqs.CPU73.RCU
     28601 ± 22%     -29.0%      20316 ±  8%  softirqs.CPU74.RCU
     28833 ± 22%     -28.6%      20575 ±  7%  softirqs.CPU75.RCU
     28743 ± 23%     -28.0%      20690 ±  7%  softirqs.CPU76.RCU
    102826 ±  3%     +30.7%     134349 ± 23%  softirqs.CPU77.TIMER
     28531 ± 23%     -28.6%      20368 ±  6%  softirqs.CPU78.RCU
     28899 ± 21%     -29.2%      20464 ±  6%  softirqs.CPU79.RCU
     28775 ± 23%     -26.4%      21190 ±  3%  softirqs.CPU80.RCU
     28592 ± 23%     -26.5%      21007 ±  2%  softirqs.CPU81.RCU
     29934 ± 24%     -30.4%      20825 ±  6%  softirqs.CPU84.RCU
     28802 ± 22%     -27.2%      20978 ±  5%  softirqs.CPU85.RCU
     28764 ± 22%     -25.3%      21500 ±  2%  softirqs.CPU86.RCU
     28839 ± 22%     -26.5%      21187 ±  3%  softirqs.CPU87.RCU
     28545 ± 22%     -24.4%      21574 ±  2%  softirqs.CPU88.RCU
     29941 ± 24%     -28.0%      21557 ±  3%  softirqs.CPU89.RCU
     29404 ± 22%     -23.5%      22508 ±  2%  softirqs.CPU9.RCU
     29032 ± 22%     -26.8%      21248 ±  5%  softirqs.CPU90.RCU
     29070 ± 22%     -26.8%      21273 ±  2%  softirqs.CPU91.RCU
     28791 ± 22%     -25.1%      21564 ±  2%  softirqs.CPU92.RCU
     28534 ± 22%     -24.3%      21596 ±  2%  softirqs.CPU93.RCU
     28411 ± 22%     -23.7%      21689 ±  2%  softirqs.CPU94.RCU
     28526 ± 22%     -24.2%      21609 ±  3%  softirqs.CPU95.RCU
     31022 ± 19%     -30.6%      21526 ±  7%  softirqs.CPU96.RCU
     29709 ± 22%     -29.4%      20981 ±  9%  softirqs.CPU97.RCU
     29889 ± 22%     -25.8%      22190 ±  4%  softirqs.CPU99.RCU
   5598226 ± 22%     -26.0%    4141614 ±  4%  softirqs.RCU
    210.75 ± 20%    +128.2%     481.00 ± 35%  interrupts.34:PCI-MSI.26738691-edge.eth0-TxRx-2
   1224085 ±  7%      -7.2%    1135614        interrupts.CAL:Function_call_interrupts
      7650 ±  5%     -50.3%       3803 ± 47%  interrupts.CPU1.RES:Rescheduling_interrupts
    210.75 ± 20%    +128.2%     481.00 ± 35%  interrupts.CPU10.34:PCI-MSI.26738691-edge.eth0-TxRx-2
    514.50 ±  7%     -31.5%     352.50 ± 18%  interrupts.CPU10.NMI:Non-maskable_interrupts
    514.50 ±  7%     -31.5%     352.50 ± 18%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
    369.75 ± 31%     +41.7%     524.00 ±  9%  interrupts.CPU100.NMI:Non-maskable_interrupts
    369.75 ± 31%     +41.7%     524.00 ±  9%  interrupts.CPU100.PMI:Performance_monitoring_interrupts
    337.75 ± 31%     +49.8%     506.00 ± 12%  interrupts.CPU101.NMI:Non-maskable_interrupts
    337.75 ± 31%     +49.8%     506.00 ± 12%  interrupts.CPU101.PMI:Performance_monitoring_interrupts
      6985 ±  5%     -41.6%       4081 ± 47%  interrupts.CPU101.RES:Rescheduling_interrupts
    292.75 ± 29%     +74.0%     509.50 ± 10%  interrupts.CPU102.NMI:Non-maskable_interrupts
    292.75 ± 29%     +74.0%     509.50 ± 10%  interrupts.CPU102.PMI:Performance_monitoring_interrupts
      7501 ± 11%     -43.2%       4259 ± 46%  interrupts.CPU102.RES:Rescheduling_interrupts
    360.25 ± 33%     +46.8%     529.00 ± 10%  interrupts.CPU103.NMI:Non-maskable_interrupts
    360.25 ± 33%     +46.8%     529.00 ± 10%  interrupts.CPU103.PMI:Performance_monitoring_interrupts
    342.25 ± 27%     +47.0%     503.00 ± 10%  interrupts.CPU104.NMI:Non-maskable_interrupts
    342.25 ± 27%     +47.0%     503.00 ± 10%  interrupts.CPU104.PMI:Performance_monitoring_interrupts
      7726 ±  7%     -43.0%       4402 ± 46%  interrupts.CPU104.RES:Rescheduling_interrupts
    350.50 ± 32%     +54.9%     543.00 ± 10%  interrupts.CPU106.NMI:Non-maskable_interrupts
    350.50 ± 32%     +54.9%     543.00 ± 10%  interrupts.CPU106.PMI:Performance_monitoring_interrupts
      7558 ±  8%     -45.0%       4155 ± 54%  interrupts.CPU106.RES:Rescheduling_interrupts
    366.25 ± 32%     +49.9%     549.00 ± 13%  interrupts.CPU107.NMI:Non-maskable_interrupts
    366.25 ± 32%     +49.9%     549.00 ± 13%  interrupts.CPU107.PMI:Performance_monitoring_interrupts
      7650 ±  5%     -43.8%       4300 ± 51%  interrupts.CPU108.RES:Rescheduling_interrupts
    338.00 ± 34%     +62.9%     550.50 ±  3%  interrupts.CPU109.NMI:Non-maskable_interrupts
    338.00 ± 34%     +62.9%     550.50 ±  3%  interrupts.CPU109.PMI:Performance_monitoring_interrupts
      7347 ±  4%     -45.4%       4015 ± 41%  interrupts.CPU109.RES:Rescheduling_interrupts
      7213 ±  6%     -38.5%       4438 ± 46%  interrupts.CPU110.RES:Rescheduling_interrupts
    340.00 ± 30%     +56.3%     531.50 ±  9%  interrupts.CPU111.NMI:Non-maskable_interrupts
    340.00 ± 30%     +56.3%     531.50 ±  9%  interrupts.CPU111.PMI:Performance_monitoring_interrupts
      7143 ±  3%     -39.0%       4360 ± 49%  interrupts.CPU112.RES:Rescheduling_interrupts
    368.25 ± 27%     +49.9%     552.00 ±  4%  interrupts.CPU113.NMI:Non-maskable_interrupts
    368.25 ± 27%     +49.9%     552.00 ±  4%  interrupts.CPU113.PMI:Performance_monitoring_interrupts
    369.50 ± 27%     +41.7%     523.50 ± 10%  interrupts.CPU114.NMI:Non-maskable_interrupts
    369.50 ± 27%     +41.7%     523.50 ± 10%  interrupts.CPU114.PMI:Performance_monitoring_interrupts
    349.25 ± 30%     +47.9%     516.50 ± 14%  interrupts.CPU116.NMI:Non-maskable_interrupts
    349.25 ± 30%     +47.9%     516.50 ± 14%  interrupts.CPU116.PMI:Performance_monitoring_interrupts
      7127 ±  2%     -38.7%       4370 ± 49%  interrupts.CPU116.RES:Rescheduling_interrupts
    347.50 ± 34%     +47.5%     512.50 ±  7%  interrupts.CPU117.NMI:Non-maskable_interrupts
    347.50 ± 34%     +47.5%     512.50 ±  7%  interrupts.CPU117.PMI:Performance_monitoring_interrupts
      6399 ±  7%      -8.2%       5877        interrupts.CPU118.CAL:Function_call_interrupts
    347.25 ± 29%     +55.8%     541.00 ±  9%  interrupts.CPU118.NMI:Non-maskable_interrupts
    347.25 ± 29%     +55.8%     541.00 ±  9%  interrupts.CPU118.PMI:Performance_monitoring_interrupts
      8105 ±  5%     -44.4%       4504 ± 49%  interrupts.CPU118.RES:Rescheduling_interrupts
      6419 ±  7%      -7.9%       5909        interrupts.CPU119.CAL:Function_call_interrupts
    404.25 ± 21%     +31.5%     531.50 ±  9%  interrupts.CPU119.NMI:Non-maskable_interrupts
    404.25 ± 21%     +31.5%     531.50 ±  9%  interrupts.CPU119.PMI:Performance_monitoring_interrupts
      6957 ±  8%     -39.9%       4181 ± 48%  interrupts.CPU119.RES:Rescheduling_interrupts
      6403 ±  7%      -7.6%       5915        interrupts.CPU12.CAL:Function_call_interrupts
      6524 ± 14%     -50.9%       3201 ± 54%  interrupts.CPU120.RES:Rescheduling_interrupts
    409.50 ± 23%     +31.5%     538.50 ±  3%  interrupts.CPU121.NMI:Non-maskable_interrupts
    409.50 ± 23%     +31.5%     538.50 ±  3%  interrupts.CPU121.PMI:Performance_monitoring_interrupts
      6429 ±  6%      -8.0%       5917        interrupts.CPU125.CAL:Function_call_interrupts
      6424 ±  6%      -7.7%       5930        interrupts.CPU127.CAL:Function_call_interrupts
      6418 ±  6%      -7.9%       5913        interrupts.CPU128.CAL:Function_call_interrupts
      6413 ±  6%      -7.5%       5934        interrupts.CPU129.CAL:Function_call_interrupts
      6409 ±  6%      -8.5%       5862        interrupts.CPU130.CAL:Function_call_interrupts
      7341 ±  6%     -42.7%       4204 ± 48%  interrupts.CPU130.RES:Rescheduling_interrupts
      6409 ±  6%      -7.4%       5937        interrupts.CPU131.CAL:Function_call_interrupts
      6413 ±  6%      -7.6%       5928        interrupts.CPU132.CAL:Function_call_interrupts
    385.50 ± 18%     +33.7%     515.50 ± 12%  interrupts.CPU133.NMI:Non-maskable_interrupts
    385.50 ± 18%     +33.7%     515.50 ± 12%  interrupts.CPU133.PMI:Performance_monitoring_interrupts
    403.25 ± 29%     +33.4%     538.00 ± 14%  interrupts.CPU135.NMI:Non-maskable_interrupts
    403.25 ± 29%     +33.4%     538.00 ± 14%  interrupts.CPU135.PMI:Performance_monitoring_interrupts
      7641 ±  2%     -39.0%       4662 ± 51%  interrupts.CPU137.RES:Rescheduling_interrupts
    402.00 ± 29%     +29.7%     521.50 ± 15%  interrupts.CPU138.NMI:Non-maskable_interrupts
    402.00 ± 29%     +29.7%     521.50 ± 15%  interrupts.CPU138.PMI:Performance_monitoring_interrupts
    349.75 ± 32%     +45.8%     510.00 ±  7%  interrupts.CPU139.NMI:Non-maskable_interrupts
    349.75 ± 32%     +45.8%     510.00 ±  7%  interrupts.CPU139.PMI:Performance_monitoring_interrupts
      7371           -39.1%       4487 ± 44%  interrupts.CPU14.RES:Rescheduling_interrupts
    342.50 ± 33%     +55.5%     532.50 ± 11%  interrupts.CPU142.NMI:Non-maskable_interrupts
    342.50 ± 33%     +55.5%     532.50 ± 11%  interrupts.CPU142.PMI:Performance_monitoring_interrupts
      6402 ±  6%      -7.5%       5920        interrupts.CPU144.CAL:Function_call_interrupts
      6442 ±  6%      -8.0%       5929        interrupts.CPU145.CAL:Function_call_interrupts
    333.00 ± 28%     +46.5%     488.00 ± 10%  interrupts.CPU146.NMI:Non-maskable_interrupts
    333.00 ± 28%     +46.5%     488.00 ± 10%  interrupts.CPU146.PMI:Performance_monitoring_interrupts
    351.25 ± 32%     +46.0%     513.00 ±  3%  interrupts.CPU147.NMI:Non-maskable_interrupts
    351.25 ± 32%     +46.0%     513.00 ±  3%  interrupts.CPU147.PMI:Performance_monitoring_interrupts
    348.50 ± 31%     +55.2%     541.00 ± 12%  interrupts.CPU148.NMI:Non-maskable_interrupts
    348.50 ± 31%     +55.2%     541.00 ± 12%  interrupts.CPU148.PMI:Performance_monitoring_interrupts
    350.75 ± 33%     +52.8%     536.00 ± 15%  interrupts.CPU149.NMI:Non-maskable_interrupts
    350.75 ± 33%     +52.8%     536.00 ± 15%  interrupts.CPU149.PMI:Performance_monitoring_interrupts
      7512 ±  2%     -45.3%       4107 ± 50%  interrupts.CPU15.RES:Rescheduling_interrupts
    337.50 ± 31%     +48.9%     502.50 ±  7%  interrupts.CPU150.NMI:Non-maskable_interrupts
    337.50 ± 31%     +48.9%     502.50 ±  7%  interrupts.CPU150.PMI:Performance_monitoring_interrupts
    349.75 ± 28%     +47.7%     516.50 ± 10%  interrupts.CPU151.NMI:Non-maskable_interrupts
    349.75 ± 28%     +47.7%     516.50 ± 10%  interrupts.CPU151.PMI:Performance_monitoring_interrupts
      7436 ±  3%     -39.5%       4502 ± 51%  interrupts.CPU151.RES:Rescheduling_interrupts
    284.75 ± 36%     +83.1%     521.50 ±  6%  interrupts.CPU155.NMI:Non-maskable_interrupts
    284.75 ± 36%     +83.1%     521.50 ±  6%  interrupts.CPU155.PMI:Performance_monitoring_interrupts
    286.25 ± 31%     +68.4%     482.00 ± 11%  interrupts.CPU156.NMI:Non-maskable_interrupts
    286.25 ± 31%     +68.4%     482.00 ± 11%  interrupts.CPU156.PMI:Performance_monitoring_interrupts
    285.25 ± 29%     +76.0%     502.00 ± 13%  interrupts.CPU157.NMI:Non-maskable_interrupts
    285.25 ± 29%     +76.0%     502.00 ± 13%  interrupts.CPU157.PMI:Performance_monitoring_interrupts
    281.75 ± 30%     +91.3%     539.00 ± 12%  interrupts.CPU158.NMI:Non-maskable_interrupts
    281.75 ± 30%     +91.3%     539.00 ± 12%  interrupts.CPU158.PMI:Performance_monitoring_interrupts
    284.25 ± 39%     +81.9%     517.00 ±  8%  interrupts.CPU159.NMI:Non-maskable_interrupts
    284.25 ± 39%     +81.9%     517.00 ±  8%  interrupts.CPU159.PMI:Performance_monitoring_interrupts
      6392 ±  7%      -7.6%       5908        interrupts.CPU16.CAL:Function_call_interrupts
    286.75 ± 29%     +81.5%     520.50 ± 12%  interrupts.CPU160.NMI:Non-maskable_interrupts
    286.75 ± 29%     +81.5%     520.50 ± 12%  interrupts.CPU160.PMI:Performance_monitoring_interrupts
    287.75 ± 29%     +80.0%     518.00 ± 13%  interrupts.CPU161.NMI:Non-maskable_interrupts
    287.75 ± 29%     +80.0%     518.00 ± 13%  interrupts.CPU161.PMI:Performance_monitoring_interrupts
    279.75 ± 30%     +88.2%     526.50 ±  5%  interrupts.CPU162.NMI:Non-maskable_interrupts
    279.75 ± 30%     +88.2%     526.50 ±  5%  interrupts.CPU162.PMI:Performance_monitoring_interrupts
      6407 ±  7%      -7.7%       5912        interrupts.CPU165.CAL:Function_call_interrupts
    412.25 ± 13%     +24.6%     513.50 ± 11%  interrupts.CPU167.NMI:Non-maskable_interrupts
    412.25 ± 13%     +24.6%     513.50 ± 11%  interrupts.CPU167.PMI:Performance_monitoring_interrupts
      7213           -35.4%       4657 ± 41%  interrupts.CPU17.RES:Rescheduling_interrupts
    354.00 ± 29%     +37.9%     488.00 ± 12%  interrupts.CPU171.NMI:Non-maskable_interrupts
    354.00 ± 29%     +37.9%     488.00 ± 12%  interrupts.CPU171.PMI:Performance_monitoring_interrupts
    343.50 ± 32%     +50.5%     517.00 ± 12%  interrupts.CPU172.NMI:Non-maskable_interrupts
    343.50 ± 32%     +50.5%     517.00 ± 12%  interrupts.CPU172.PMI:Performance_monitoring_interrupts
      6444 ±  6%      -7.8%       5939        interrupts.CPU174.CAL:Function_call_interrupts
      6413 ±  7%      -7.7%       5919        interrupts.CPU18.CAL:Function_call_interrupts
      6402 ±  7%      -7.7%       5910        interrupts.CPU19.CAL:Function_call_interrupts
      6840 ±  5%     -43.9%       3839 ± 46%  interrupts.CPU191.RES:Rescheduling_interrupts
      6397 ±  7%      -7.6%       5909        interrupts.CPU20.CAL:Function_call_interrupts
      6848 ±  8%     -41.5%       4003 ± 40%  interrupts.CPU20.RES:Rescheduling_interrupts
      6417 ±  7%      -8.0%       5903        interrupts.CPU21.CAL:Function_call_interrupts
      7703 ± 10%     -47.5%       4045 ± 47%  interrupts.CPU21.RES:Rescheduling_interrupts
      7202 ±  8%     -44.1%       4025 ± 44%  interrupts.CPU22.RES:Rescheduling_interrupts
      7280 ±  3%     -43.7%       4098 ± 44%  interrupts.CPU23.RES:Rescheduling_interrupts
      6404 ±  7%      -8.0%       5890        interrupts.CPU26.CAL:Function_call_interrupts
      6415 ±  7%     -16.1%       5384 ± 10%  interrupts.CPU3.CAL:Function_call_interrupts
    470.00 ±  6%     +16.2%     546.00 ± 12%  interrupts.CPU32.NMI:Non-maskable_interrupts
    470.00 ±  6%     +16.2%     546.00 ± 12%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
      6414 ±  6%      -7.6%       5923        interrupts.CPU35.CAL:Function_call_interrupts
      6409 ±  6%      -7.0%       5958        interrupts.CPU36.CAL:Function_call_interrupts
    465.00 ±  8%     +14.0%     530.00 ± 13%  interrupts.CPU38.NMI:Non-maskable_interrupts
    465.00 ±  8%     +14.0%     530.00 ± 13%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
      6412 ±  6%      -7.7%       5918        interrupts.CPU39.CAL:Function_call_interrupts
    462.50 ± 12%     +11.8%     517.00 ±  9%  interrupts.CPU39.NMI:Non-maskable_interrupts
    462.50 ± 12%     +11.8%     517.00 ±  9%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
      6851 ±  6%     -42.2%       3963 ± 49%  interrupts.CPU4.RES:Rescheduling_interrupts
    455.00 ±  7%     +14.6%     521.50 ± 13%  interrupts.CPU41.NMI:Non-maskable_interrupts
    455.00 ±  7%     +14.6%     521.50 ± 13%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
      6422 ±  6%      -7.3%       5951        interrupts.CPU42.CAL:Function_call_interrupts
      7549 ±  3%     -41.2%       4437 ± 50%  interrupts.CPU42.RES:Rescheduling_interrupts
      6428 ±  7%      -7.6%       5940        interrupts.CPU43.CAL:Function_call_interrupts
      6431 ±  7%      -7.7%       5938        interrupts.CPU44.CAL:Function_call_interrupts
      6376 ±  6%      -7.0%       5931        interrupts.CPU47.CAL:Function_call_interrupts
    500.00 ± 11%     -28.1%     359.50 ± 23%  interrupts.CPU49.NMI:Non-maskable_interrupts
    500.00 ± 11%     -28.1%     359.50 ± 23%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
      6415 ±  6%      -8.2%       5889        interrupts.CPU50.CAL:Function_call_interrupts
      6407 ±  7%      -9.4%       5805        interrupts.CPU51.CAL:Function_call_interrupts
    409.50 ± 22%     +39.1%     569.50 ±  5%  interrupts.CPU51.NMI:Non-maskable_interrupts
    409.50 ± 22%     +39.1%     569.50 ±  5%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
      6408 ±  7%      -8.0%       5896        interrupts.CPU52.CAL:Function_call_interrupts
    342.75 ± 33%     +58.0%     541.50 ± 10%  interrupts.CPU52.NMI:Non-maskable_interrupts
    342.75 ± 33%     +58.0%     541.50 ± 10%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
      6411 ±  7%      -7.8%       5910        interrupts.CPU53.CAL:Function_call_interrupts
    348.00 ± 35%     +50.6%     524.00 ±  7%  interrupts.CPU53.NMI:Non-maskable_interrupts
    348.00 ± 35%     +50.6%     524.00 ±  7%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
      6416 ±  6%      -7.7%       5919        interrupts.CPU54.CAL:Function_call_interrupts
    338.75 ± 31%     +51.7%     514.00 ± 12%  interrupts.CPU54.NMI:Non-maskable_interrupts
    338.75 ± 31%     +51.7%     514.00 ± 12%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
      6419 ±  7%      -8.4%       5883        interrupts.CPU55.CAL:Function_call_interrupts
    350.00 ± 34%     +53.3%     536.50 ±  8%  interrupts.CPU55.NMI:Non-maskable_interrupts
    350.00 ± 34%     +53.3%     536.50 ±  8%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
      6422 ±  6%      -7.9%       5914        interrupts.CPU56.CAL:Function_call_interrupts
    401.00 ± 21%     +34.8%     540.50 ±  8%  interrupts.CPU56.NMI:Non-maskable_interrupts
    401.00 ± 21%     +34.8%     540.50 ±  8%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
      6418 ±  7%      -7.7%       5926        interrupts.CPU57.CAL:Function_call_interrupts
    411.00 ± 21%     +30.4%     536.00 ± 10%  interrupts.CPU58.NMI:Non-maskable_interrupts
    411.00 ± 21%     +30.4%     536.00 ± 10%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
      6410 ±  6%      -7.9%       5904        interrupts.CPU59.CAL:Function_call_interrupts
    403.00 ± 24%     +25.7%     506.50 ±  8%  interrupts.CPU59.NMI:Non-maskable_interrupts
    403.00 ± 24%     +25.7%     506.50 ±  8%  interrupts.CPU59.PMI:Performance_monitoring_interrupts
      7397 ±  3%     -40.2%       4424 ± 50%  interrupts.CPU6.RES:Rescheduling_interrupts
      6403 ±  7%      -7.6%       5917        interrupts.CPU60.CAL:Function_call_interrupts
    394.50 ± 22%     +35.2%     533.50 ± 14%  interrupts.CPU60.NMI:Non-maskable_interrupts
    394.50 ± 22%     +35.2%     533.50 ± 14%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
      6406 ±  7%      -7.3%       5939        interrupts.CPU62.CAL:Function_call_interrupts
    403.25 ± 22%     +25.9%     507.50 ± 10%  interrupts.CPU62.NMI:Non-maskable_interrupts
    403.25 ± 22%     +25.9%     507.50 ± 10%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
    427.25 ± 23%     +19.4%     510.00 ±  6%  interrupts.CPU63.NMI:Non-maskable_interrupts
    427.25 ± 23%     +19.4%     510.00 ±  6%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
    457.50 ±  8%     +12.0%     512.50 ± 11%  interrupts.CPU64.NMI:Non-maskable_interrupts
    457.50 ±  8%     +12.0%     512.50 ± 11%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
    473.25 ± 11%     +13.7%     538.00 ± 11%  interrupts.CPU71.NMI:Non-maskable_interrupts
    473.25 ± 11%     +13.7%     538.00 ± 11%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
      6421 ±  6%     -16.0%       5395 ±  4%  interrupts.CPU72.CAL:Function_call_interrupts
      6311 ±  7%     -59.7%       2542 ± 56%  interrupts.CPU72.RES:Rescheduling_interrupts
      6434 ±  6%     -12.8%       5611 ±  6%  interrupts.CPU73.CAL:Function_call_interrupts
      6931 ±  6%     -47.7%       3623 ± 40%  interrupts.CPU73.RES:Rescheduling_interrupts
      6415 ±  6%      -7.3%       5945        interrupts.CPU75.CAL:Function_call_interrupts
      6427 ±  6%      -7.6%       5941        interrupts.CPU76.CAL:Function_call_interrupts
      6428 ±  6%      -7.5%       5946        interrupts.CPU78.CAL:Function_call_interrupts
      7307 ±  7%     -41.7%       4262 ± 50%  interrupts.CPU8.RES:Rescheduling_interrupts
      6428 ±  6%      -7.7%       5930        interrupts.CPU82.CAL:Function_call_interrupts
      6402 ±  6%      -7.4%       5927        interrupts.CPU86.CAL:Function_call_interrupts
    469.75 ±  8%     +12.0%     526.00 ± 10%  interrupts.CPU86.NMI:Non-maskable_interrupts
    469.75 ±  8%     +12.0%     526.00 ± 10%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
      6410 ±  6%      -7.7%       5918        interrupts.CPU87.CAL:Function_call_interrupts
      6411 ±  6%      -7.5%       5928        interrupts.CPU88.CAL:Function_call_interrupts
      6413 ±  6%      -7.7%       5922        interrupts.CPU89.CAL:Function_call_interrupts
      6417 ±  7%      -7.6%       5931        interrupts.CPU91.CAL:Function_call_interrupts
      6409 ±  7%      -7.5%       5926        interrupts.CPU94.CAL:Function_call_interrupts
      6405 ±  6%      -8.4%       5865        interrupts.CPU96.CAL:Function_call_interrupts
      8173 ±  5%     -42.6%       4691 ± 38%  interrupts.CPU96.RES:Rescheduling_interrupts
      6417 ±  7%      -7.8%       5918        interrupts.CPU97.CAL:Function_call_interrupts
      7758 ±  6%     -39.6%       4687 ± 38%  interrupts.CPU98.RES:Rescheduling_interrupts
      6987 ±  6%     -42.4%       4027 ± 28%  interrupts.CPU99.RES:Rescheduling_interrupts
     80504 ±  7%     +15.5%      92961 ±  8%  interrupts.NMI:Non-maskable_interrupts
     80504 ±  7%     +15.5%      92961 ±  8%  interrupts.PMI:Performance_monitoring_interrupts



***************************************************************************************************
lkp-hsw-4ex1: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory
=========================================================================================
compiler/cpufreq_governor/iterations/kconfig/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/10/x86_64-rhel-7.6/1600%/debian-x86_64-2019-05-14.cgz/lkp-hsw-4ex1/custom/reaim/0x14

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          0:4          -11%           0:2     perf-profile.children.cycles-pp.error_entry
          0:4          -10%           0:2     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
      2549 ±  2%      +5.2%       2683 ±  2%  reaim.child_systime
     25527 ±  2%      +5.2%      26866 ±  2%  reaim.time.system_time
    364348 ±  5%     +14.8%     418108        meminfo.DirectMap4k
     97407 ± 94%    +331.6%     420408 ± 38%  syscalls.sys_rt_sigaction.max
     18.16 ± 25%      -7.9       10.27 ± 38%  turbostat.C3%
     38.28 ±  2%      -4.1%      36.72        boot-time.boot
      4721 ±  2%      -3.7%       4548        boot-time.idle
   2156912 ±  6%     +71.9%    3707777 ± 29%  cpuidle.POLL.time
    468846 ± 11%     +47.5%     691572 ± 15%  cpuidle.POLL.usage
     62557 ±  2%      +5.7%      66113 ±  6%  numa-vmstat.node1.nr_unevictable
     62557 ±  2%      +5.7%      66113 ±  6%  numa-vmstat.node1.nr_zone_unevictable
      5732 ± 29%     +41.4%       8104 ± 27%  numa-vmstat.node3.nr_slab_reclaimable
      7305 ± 29%     +22.8%       8969 ± 30%  numa-meminfo.node1.Mapped
    250228 ±  2%      +5.7%     264457 ±  6%  numa-meminfo.node1.Unevictable
     22894 ± 29%     +41.9%      32489 ± 27%  numa-meminfo.node3.KReclaimable
      6373 ±  5%     +15.9%       7384 ± 13%  numa-meminfo.node3.Mapped
     22894 ± 29%     +41.9%      32489 ± 27%  numa-meminfo.node3.SReclaimable
     21.79 ±  4%      -7.6%      20.14        perf-stat.i.MPKI
      1.48 ± 11%      -0.2        1.25        perf-stat.i.branch-miss-rate%
      2626            +3.5%       2720        perf-stat.overall.cycles-between-cache-misses
  26402092 ±  2%      +3.9%   27418780        perf-stat.overall.path-length
 6.083e+13 ±  2%      +3.9%  6.317e+13        perf-stat.total.instructions
     42744 ±  6%      -7.8%      39414        softirqs.CPU124.NET_RX
     40812 ±  2%      +7.6%      43917 ±  5%  softirqs.CPU48.NET_RX
     38081 ±  3%      +7.8%      41070 ±  4%  softirqs.CPU59.NET_RX
     40706 ±  4%      +9.7%      44636 ±  3%  softirqs.CPU64.NET_RX
     38992 ±  2%     +16.8%      45528        softirqs.CPU83.NET_RX
     38859 ±  2%     +16.2%      45160        softirqs.CPU88.NET_RX
    115563 ±  5%     +10.6%     127765 ±  5%  slabinfo.kmalloc-32.active_objs
    903.00 ±  5%     +10.5%     998.00 ±  5%  slabinfo.kmalloc-32.active_slabs
    115656 ±  5%     +10.5%     127819 ±  5%  slabinfo.kmalloc-32.num_objs
    903.00 ±  5%     +10.5%     998.00 ±  5%  slabinfo.kmalloc-32.num_slabs
     58841 ± 18%     -23.3%      45131        slabinfo.lsm_file_cache.active_objs
     58841 ± 18%     -23.3%      45131        slabinfo.lsm_file_cache.num_objs
    888.75 ± 14%     +35.1%       1200 ±  6%  slabinfo.xfrm_state.active_objs
    888.75 ± 14%     +35.1%       1200 ±  6%  slabinfo.xfrm_state.num_objs
    117.75 ±  4%      +8.3%     127.50        proc-vmstat.nr_dirtied
    280918            +1.4%     284967        proc-vmstat.nr_file_pages
      5847            +2.9%       6019        proc-vmstat.nr_inactive_anon
    102732 ±  2%      +2.2%     105006        proc-vmstat.nr_kernel_stack
      7804 ±  2%      +4.5%       8154        proc-vmstat.nr_mapped
     25837 ±  7%     +10.4%      28533        proc-vmstat.nr_shmem
    116.75 ±  5%      +7.5%     125.50        proc-vmstat.nr_written
      5847            +2.9%       6019        proc-vmstat.nr_zone_inactive_anon
 4.568e+08            -3.4%  4.414e+08        proc-vmstat.numa_hit
 4.566e+08            -3.4%  4.413e+08        proc-vmstat.numa_local
     34509 ±  3%      +6.9%      36895        proc-vmstat.pgactivate
 4.683e+08            -3.3%  4.529e+08        proc-vmstat.pgalloc_normal
 4.682e+08            -3.3%  4.528e+08        proc-vmstat.pgfree
    673.67 ±  7%     -13.4%     583.64 ±  2%  sched_debug.cfs_rq:/.exec_clock.stddev
      0.93 ±  7%     +23.1%       1.14        sched_debug.cfs_rq:/.load_avg.min
      1.23 ±  9%     -18.4%       1.00        sched_debug.cfs_rq:/.nr_running.max
    779.44 ±  9%     +15.0%     896.36 ±  2%  sched_debug.cfs_rq:/.removed.load_avg.max
     35998 ±  9%     +15.0%      41403        sched_debug.cfs_rq:/.removed.runnable_sum.max
    314.12 ± 26%     -49.0%     160.29 ±  5%  sched_debug.cfs_rq:/.runnable_load_avg.max
    243.17 ± 17%     +20.1%     292.07 ± 11%  sched_debug.cfs_rq:/.util_avg.min
     53.89 ± 15%     +23.0%      66.26 ±  5%  sched_debug.cpu.clock.stddev
     53.89 ± 15%     +23.0%      66.26 ±  5%  sched_debug.cpu.clock_task.stddev
     23.03 ±  8%     +13.3%      26.09 ±  8%  sched_debug.cpu.cpu_load[2].stddev
    185.49 ± 18%     +23.0%     228.21 ±  4%  sched_debug.cpu.cpu_load[3].max
     18.25 ± 14%     +27.2%      23.22 ±  3%  sched_debug.cpu.cpu_load[3].stddev
      3.41 ± 12%     +14.7%       3.91        sched_debug.cpu.cpu_load[4].avg
    171.67 ± 27%     +32.5%     227.50 ± 10%  sched_debug.cpu.cpu_load[4].max
     16.29 ± 23%     +37.1%      22.33 ±  2%  sched_debug.cpu.cpu_load[4].stddev
      0.00 ±  8%     +32.6%       0.00 ±  2%  sched_debug.cpu.next_balance.stddev
      9.29 ± 12%     +22.2%      11.35        sched_debug.cpu.nr_uninterruptible.avg
      3.04 ±100%      -3.0        0.00        perf-profile.calltrace.cycles-pp.sieve
      0.52 ± 60%      +0.3        0.80 ±  7%  perf-profile.calltrace.cycles-pp.do_wp_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
      1.24 ± 18%      +0.3        1.55 ±  7%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit
      1.30 ± 18%      +0.3        1.62 ±  7%  perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit
      0.29 ±100%      +0.3        0.64 ±  5%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      0.56 ± 63%      +0.3        0.91 ±  7%  perf-profile.calltrace.cycles-pp.copy_p4d_range.copy_page_range.dup_mm.copy_process._do_fork
      0.57 ± 63%      +0.4        0.94 ±  6%  perf-profile.calltrace.cycles-pp.copy_page_range.dup_mm.copy_process._do_fork.do_syscall_64
      0.12 ±173%      +0.5        0.59 ±  4%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.16 ±173%      +0.5        0.65        perf-profile.calltrace.cycles-pp.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.16 ±173%      +0.5        0.65        perf-profile.calltrace.cycles-pp.kernel_wait4.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.38 ± 23%      +0.5        1.86 ±  4%  perf-profile.calltrace.cycles-pp.filemap_map_pages.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
      0.75 ± 38%      +0.5        1.29        perf-profile.calltrace.cycles-pp.anon_vma_fork.dup_mm.copy_process._do_fork.do_syscall_64
      0.36 ±105%      +0.6        0.94        perf-profile.calltrace.cycles-pp.anon_vma_clone.anon_vma_fork.dup_mm.copy_process._do_fork
      2.83 ± 20%      +0.6        3.48 ±  7%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      2.92 ± 20%      +0.7        3.58 ±  7%  perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      3.26 ± 19%      +0.7        3.94 ±  8%  perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
     11.68 ± 25%      +3.9       15.58 ± 12%  perf-profile.calltrace.cycles-pp.dup_mm.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.38 ± 34%      +4.0       13.43 ± 15%  perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit
     11.97 ± 27%      +4.6       16.54 ± 11%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
     11.98 ± 27%      +4.6       16.55 ± 11%  perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     12.86 ± 26%      +4.6       17.47 ± 10%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     12.86 ± 26%      +4.6       17.47 ± 10%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     12.86 ± 26%      +4.6       17.47 ± 10%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.05 ± 44%      +6.0       14.07 ± 14%  perf-profile.calltrace.cycles-pp.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.23 ± 44%      +6.1       14.34 ± 13%  perf-profile.calltrace.cycles-pp._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
     37.29 ± 20%     +11.1       48.38 ±  5%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     37.31 ± 20%     +11.1       48.41 ±  5%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      3.04 ±100%      -3.0        0.00        perf-profile.children.cycles-pp.sieve
      0.41 ± 21%      -0.2        0.22 ± 13%  perf-profile.children.cycles-pp.syscall_trace_enter
      0.20 ± 39%      -0.2        0.03 ±100%  perf-profile.children.cycles-pp.trace_buffer_lock_reserve
      0.37 ± 32%      -0.2        0.20 ±  2%  perf-profile.children.cycles-pp.menu_select
      0.33 ± 18%      -0.2        0.16        perf-profile.children.cycles-pp.syscall_slow_exit_work
      0.32 ± 24%      -0.2        0.17 ± 15%  perf-profile.children.cycles-pp.ftrace_syscall_enter
      0.28 ± 22%      -0.2        0.12 ±  4%  perf-profile.children.cycles-pp.ftrace_syscall_exit
      0.17 ± 38%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.ring_buffer_lock_reserve
      0.11 ± 29%      -0.1        0.06        perf-profile.children.cycles-pp.irq_enter
      0.07 ± 14%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.16 ± 15%      -0.0        0.11 ±  4%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.07 ± 13%      -0.0        0.05        perf-profile.children.cycles-pp.__list_add_valid
      0.14 ± 18%      +0.0        0.17 ±  8%  perf-profile.children.cycles-pp.vm_area_dup
      0.15 ±  9%      +0.0        0.18 ± 13%  perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave
      0.12 ± 18%      +0.0        0.15 ± 13%  perf-profile.children.cycles-pp.__put_user_4
      0.15 ± 11%      +0.0        0.19 ± 15%  perf-profile.children.cycles-pp.free_uid
      0.19 ± 14%      +0.0        0.23 ± 11%  perf-profile.children.cycles-pp.anon_vma_interval_tree_remove
      0.10 ± 24%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.schedule_tail
      0.25 ± 14%      +0.0        0.30 ±  8%  perf-profile.children.cycles-pp.wake_up_new_task
      0.01 ±173%      +0.0        0.06        perf-profile.children.cycles-pp.__put_task_struct
      0.15 ± 11%      +0.1        0.21 ± 12%  perf-profile.children.cycles-pp.refcount_dec_not_one
      0.04 ±103%      +0.1        0.10 ±  5%  perf-profile.children.cycles-pp.autoremove_wake_function
      0.05 ± 74%      +0.1        0.11 ±  4%  perf-profile.children.cycles-pp.pipe_wait
      0.14 ± 28%      +0.1        0.21 ±  9%  perf-profile.children.cycles-pp.__wake_up_common
      0.24 ± 14%      +0.1        0.31 ± 12%  perf-profile.children.cycles-pp.put_cred_rcu
      0.07 ± 70%      +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.pipe_read
      0.15 ± 29%      +0.1        0.23 ±  6%  perf-profile.children.cycles-pp.__wake_up_common_lock
      0.04 ±104%      +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.pipe_write
      0.48 ± 10%      +0.1        0.60 ± 15%  perf-profile.children.cycles-pp.rcu_core
      0.79 ±  9%      +0.1        0.94 ±  4%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.74 ± 15%      +0.1        0.89 ±  4%  perf-profile.children.cycles-pp.unlink_anon_vmas
      0.37 ± 19%      +0.2        0.52 ±  7%  perf-profile.children.cycles-pp.anon_vma_interval_tree_insert
      0.74 ± 18%      +0.2        0.90 ±  8%  perf-profile.children.cycles-pp.do_wp_page
      0.63 ± 20%      +0.2        0.79 ±  6%  perf-profile.children.cycles-pp.wp_page_copy
      0.74 ± 17%      +0.2        0.98 ±  2%  perf-profile.children.cycles-pp.anon_vma_clone
      0.99 ± 20%      +0.3        1.33        perf-profile.children.cycles-pp.anon_vma_fork
      2.20 ± 17%      +0.4        2.62 ±  7%  perf-profile.children.cycles-pp.filemap_map_pages
     11.68 ± 25%      +3.9       15.59 ± 12%  perf-profile.children.cycles-pp.dup_mm
     12.60 ± 23%      +3.9       16.52 ± 11%  perf-profile.children.cycles-pp.copy_process
     17.92 ± 18%      +3.9       21.86 ±  7%  perf-profile.children.cycles-pp.exit_mmap
     17.94 ± 18%      +3.9       21.88 ±  7%  perf-profile.children.cycles-pp.mmput
     12.90 ± 22%      +4.0       16.86 ± 11%  perf-profile.children.cycles-pp._do_fork
     12.90 ± 25%      +4.6       17.49 ± 10%  perf-profile.children.cycles-pp.do_exit
     12.91 ± 25%      +4.6       17.50 ± 10%  perf-profile.children.cycles-pp.__x64_sys_exit_group
     12.91 ± 25%      +4.6       17.50 ± 10%  perf-profile.children.cycles-pp.do_group_exit
     50.20 ± 10%      +5.2       55.40 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     50.16 ± 10%      +5.2       55.37 ±  4%  perf-profile.children.cycles-pp.do_syscall_64
      3.00 ±100%      -3.0        0.00        perf-profile.self.cycles-pp.sieve
      0.17 ± 25%      -0.1        0.11 ±  4%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.13 ±  9%      -0.0        0.09 ± 11%  perf-profile.self.cycles-pp.ftrace_syscall_exit
      0.06 ±  6%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.update_load_avg
      0.07 ± 11%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.put_cred_rcu
      0.13 ± 14%      +0.0        0.16        perf-profile.self.cycles-pp.unlock_page
      0.12 ± 13%      +0.0        0.15 ±  6%  perf-profile.self.cycles-pp.anon_vma_clone
      0.18 ± 14%      +0.0        0.21 ± 11%  perf-profile.self.cycles-pp.anon_vma_interval_tree_remove
      0.15 ± 11%      +0.0        0.20 ± 10%  perf-profile.self.cycles-pp.refcount_dec_not_one
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.__anon_vma_interval_tree_augment_rotate
      0.01 ±173%      +0.1        0.08 ± 33%  perf-profile.self.cycles-pp.update_blocked_averages
      0.30 ± 24%      +0.1        0.39 ± 10%  perf-profile.self.cycles-pp.copy_page
      0.37 ± 19%      +0.1        0.52 ±  6%  perf-profile.self.cycles-pp.anon_vma_interval_tree_insert
      1.56 ± 20%      +0.4        1.98 ±  8%  perf-profile.self.cycles-pp.filemap_map_pages
    315.75 ± 49%     -36.2%     201.50 ±  3%  interrupts.47:PCI-MSI.1572866-edge.eth0-TxRx-2
    209.50 ±  7%    +607.6%       1482 ± 85%  interrupts.48:PCI-MSI.1572867-edge.eth0-TxRx-3
    261.00 ± 40%     +47.3%     384.50 ± 43%  interrupts.54:PCI-MSI.1572873-edge.eth0-TxRx-9
    262.75 ± 21%     -20.5%     209.00        interrupts.60:PCI-MSI.1572879-edge.eth0-TxRx-15
    194.25 ±  8%     +15.6%     224.50 ±  9%  interrupts.70:PCI-MSI.1572889-edge.eth0-TxRx-25
    194.50 ±  8%     +20.3%     234.00 ± 13%  interrupts.72:PCI-MSI.1572891-edge.eth0-TxRx-27
    194.00 ±  8%    +100.3%     388.50 ± 48%  interrupts.76:PCI-MSI.1572895-edge.eth0-TxRx-31
     16378 ±  4%     -10.9%      14598        interrupts.CPU0.RES:Rescheduling_interrupts
    611.75 ±131%    +693.1%       4851 ±  6%  interrupts.CPU10.NMI:Non-maskable_interrupts
    611.75 ±131%    +693.1%       4851 ±  6%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      2966 ± 65%     -99.3%      22.00 ± 72%  interrupts.CPU100.NMI:Non-maskable_interrupts
      2966 ± 65%     -99.3%      22.00 ± 72%  interrupts.CPU100.PMI:Performance_monitoring_interrupts
     12853 ±  5%     -14.1%      11047        interrupts.CPU100.RES:Rescheduling_interrupts
      4472 ±  7%     -71.9%       1257 ± 97%  interrupts.CPU101.NMI:Non-maskable_interrupts
      4472 ±  7%     -71.9%       1257 ± 97%  interrupts.CPU101.PMI:Performance_monitoring_interrupts
      9931 ± 10%     -14.3%       8514        interrupts.CPU102.RES:Rescheduling_interrupts
      9671 ±  8%     -13.4%       8377        interrupts.CPU103.RES:Rescheduling_interrupts
      1674 ±106%    +184.5%       4762 ±  8%  interrupts.CPU106.NMI:Non-maskable_interrupts
      1674 ±106%    +184.5%       4762 ±  8%  interrupts.CPU106.PMI:Performance_monitoring_interrupts
     12916 ±  6%     -14.7%      11018        interrupts.CPU108.RES:Rescheduling_interrupts
     12536 ±  9%     -17.8%      10301 ±  4%  interrupts.CPU109.RES:Rescheduling_interrupts
     10666 ±  6%     -17.7%       8781        interrupts.CPU11.RES:Rescheduling_interrupts
     10791 ±  7%     -13.4%       9347        interrupts.CPU111.RES:Rescheduling_interrupts
      1536 ±121%    +209.4%       4753 ±  7%  interrupts.CPU112.NMI:Non-maskable_interrupts
      1536 ±121%    +209.4%       4753 ±  7%  interrupts.CPU112.PMI:Performance_monitoring_interrupts
      3978 ± 27%     -99.9%       3.50 ± 14%  interrupts.CPU118.NMI:Non-maskable_interrupts
      3978 ± 27%     -99.9%       3.50 ± 14%  interrupts.CPU118.PMI:Performance_monitoring_interrupts
     10560 ±  5%     -18.5%       8611 ±  6%  interrupts.CPU118.RES:Rescheduling_interrupts
     12039 ±  7%     -21.1%       9504 ±  6%  interrupts.CPU12.RES:Rescheduling_interrupts
      1123 ±169%    +329.2%       4823 ±  8%  interrupts.CPU122.NMI:Non-maskable_interrupts
      1123 ±169%    +329.2%       4823 ±  8%  interrupts.CPU122.PMI:Performance_monitoring_interrupts
      9473 ±  6%     -13.5%       8193 ±  4%  interrupts.CPU123.RES:Rescheduling_interrupts
      1023 ±173%    +124.4%       2297 ± 99%  interrupts.CPU124.NMI:Non-maskable_interrupts
      1023 ±173%    +124.4%       2297 ± 99%  interrupts.CPU124.PMI:Performance_monitoring_interrupts
      1620 ±112%    +120.1%       3565 ± 25%  interrupts.CPU125.NMI:Non-maskable_interrupts
      1620 ±112%    +120.1%       3565 ± 25%  interrupts.CPU125.PMI:Performance_monitoring_interrupts
      1000 ±169%    +183.7%       2836 ± 84%  interrupts.CPU126.NMI:Non-maskable_interrupts
      1000 ±169%    +183.7%       2836 ± 84%  interrupts.CPU126.PMI:Performance_monitoring_interrupts
     12071 ±  6%     -16.4%      10091 ±  6%  interrupts.CPU126.RES:Rescheduling_interrupts
     13431 ±  3%     -14.6%      11466 ±  4%  interrupts.CPU127.RES:Rescheduling_interrupts
     10962 ±  6%     -13.7%       9459        interrupts.CPU129.RES:Rescheduling_interrupts
     11360 ±  5%     -13.8%       9787 ±  5%  interrupts.CPU13.RES:Rescheduling_interrupts
      3423 ± 58%     +41.9%       4857 ±  6%  interrupts.CPU130.NMI:Non-maskable_interrupts
      3423 ± 58%     +41.9%       4857 ±  6%  interrupts.CPU130.PMI:Performance_monitoring_interrupts
      9669 ±  6%     -13.0%       8411 ±  4%  interrupts.CPU131.RES:Rescheduling_interrupts
     12576 ±  7%     -13.7%      10857        interrupts.CPU132.RES:Rescheduling_interrupts
      3946 ± 27%     -91.5%     334.50 ± 96%  interrupts.CPU134.NMI:Non-maskable_interrupts
      3946 ± 27%     -91.5%     334.50 ± 96%  interrupts.CPU134.PMI:Performance_monitoring_interrupts
     10151 ±  8%     -16.7%       8455 ±  4%  interrupts.CPU134.RES:Rescheduling_interrupts
      2942 ± 65%     -99.3%      21.00 ±100%  interrupts.CPU139.NMI:Non-maskable_interrupts
      2942 ± 65%     -99.3%      21.00 ±100%  interrupts.CPU139.PMI:Performance_monitoring_interrupts
      9758 ±  8%     -16.7%       8125 ±  2%  interrupts.CPU139.RES:Rescheduling_interrupts
      1224 ±172%    +203.8%       3719 ± 38%  interrupts.CPU14.NMI:Non-maskable_interrupts
      1224 ±172%    +203.8%       3719 ± 38%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
     10654 ± 10%     -14.9%       9071        interrupts.CPU14.RES:Rescheduling_interrupts
     11715 ±  7%     -15.8%       9863        interrupts.CPU140.RES:Rescheduling_interrupts
     10556 ±  5%     -15.3%       8938 ±  3%  interrupts.CPU141.RES:Rescheduling_interrupts
      2317 ± 96%    +113.1%       4939 ±  5%  interrupts.CPU143.NMI:Non-maskable_interrupts
      2317 ± 96%    +113.1%       4939 ±  5%  interrupts.CPU143.PMI:Performance_monitoring_interrupts
    262.75 ± 21%     -20.5%     209.00        interrupts.CPU15.60:PCI-MSI.1572879-edge.eth0-TxRx-15
      9911 ±  5%     -10.1%       8914        interrupts.CPU16.RES:Rescheduling_interrupts
     15340 ±  2%     -14.5%      13116 ±  7%  interrupts.CPU18.RES:Rescheduling_interrupts
     12851 ±  4%     -11.4%      11384 ±  5%  interrupts.CPU19.RES:Rescheduling_interrupts
    315.75 ± 49%     -36.2%     201.50 ±  3%  interrupts.CPU2.47:PCI-MSI.1572866-edge.eth0-TxRx-2
      0.75 ±173%  +1.7e+05%       1252 ± 76%  interrupts.CPU20.NMI:Non-maskable_interrupts
      0.75 ±173%  +1.7e+05%       1252 ± 76%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
     12010 ±  8%     -12.0%      10567 ±  2%  interrupts.CPU20.RES:Rescheduling_interrupts
     13261 ±  2%     -19.3%      10706        interrupts.CPU21.RES:Rescheduling_interrupts
    726.00 ±173%    +541.1%       4654 ±  7%  interrupts.CPU22.NMI:Non-maskable_interrupts
    726.00 ±173%    +541.1%       4654 ±  7%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
     12065 ±  6%     -15.6%      10178 ±  4%  interrupts.CPU22.RES:Rescheduling_interrupts
    194.25 ±  8%     +15.6%     224.50 ±  9%  interrupts.CPU25.70:PCI-MSI.1572889-edge.eth0-TxRx-25
     10775 ±  7%     -15.8%       9069 ±  3%  interrupts.CPU25.RES:Rescheduling_interrupts
     10707 ± 10%     -13.3%       9279        interrupts.CPU26.RES:Rescheduling_interrupts
    194.50 ±  8%     +20.3%     234.00 ± 13%  interrupts.CPU27.72:PCI-MSI.1572891-edge.eth0-TxRx-27
      6.75 ±156%  +16277.8%       1105 ± 98%  interrupts.CPU27.NMI:Non-maskable_interrupts
      6.75 ±156%  +16277.8%       1105 ± 98%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
     10577 ±  6%     -11.9%       9316 ±  2%  interrupts.CPU27.RES:Rescheduling_interrupts
      1240 ±160%    +277.1%       4676 ±  6%  interrupts.CPU28.NMI:Non-maskable_interrupts
      1240 ±160%    +277.1%       4676 ±  6%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
     16.00 ±129%  +21987.5%       3534 ± 26%  interrupts.CPU29.NMI:Non-maskable_interrupts
     16.00 ±129%  +21987.5%       3534 ± 26%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
    209.50 ±  7%    +607.6%       1482 ± 85%  interrupts.CPU3.48:PCI-MSI.1572867-edge.eth0-TxRx-3
     12626 ±  5%     -12.0%      11109 ±  3%  interrupts.CPU3.RES:Rescheduling_interrupts
    194.00 ±  8%    +100.3%     388.50 ± 48%  interrupts.CPU31.76:PCI-MSI.1572895-edge.eth0-TxRx-31
      1051 ±100%    +356.8%       4802 ±  8%  interrupts.CPU31.NMI:Non-maskable_interrupts
      1051 ±100%    +356.8%       4802 ±  8%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
      5.50 ±152%   +1700.0%      99.00 ± 69%  interrupts.CPU32.NMI:Non-maskable_interrupts
      5.50 ±152%   +1700.0%      99.00 ± 69%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
     10150 ±  7%     -15.1%       8618 ±  6%  interrupts.CPU32.RES:Rescheduling_interrupts
     10730 ±  6%     -14.0%       9231 ±  4%  interrupts.CPU33.RES:Rescheduling_interrupts
     10838 ±  5%     -17.9%       8894 ±  2%  interrupts.CPU34.RES:Rescheduling_interrupts
      9877 ±  7%     -12.8%       8611 ±  5%  interrupts.CPU35.RES:Rescheduling_interrupts
      1209 ± 90%     -99.7%       3.50 ±100%  interrupts.CPU36.NMI:Non-maskable_interrupts
      1209 ± 90%     -99.7%       3.50 ±100%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
     14558 ±  2%     -15.8%      12260 ±  4%  interrupts.CPU36.RES:Rescheduling_interrupts
     13040 ±  5%     -18.1%      10680 ±  7%  interrupts.CPU37.RES:Rescheduling_interrupts
     12539 ±  5%     -12.7%      10943        interrupts.CPU38.RES:Rescheduling_interrupts
    639.75 ±173%    +124.9%       1439 ± 80%  interrupts.CPU39.NMI:Non-maskable_interrupts
    639.75 ±173%    +124.9%       1439 ± 80%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
     12261 ±  6%     -14.5%      10481 ±  6%  interrupts.CPU39.RES:Rescheduling_interrupts
     13154 ±  4%     -18.0%      10789 ±  3%  interrupts.CPU4.RES:Rescheduling_interrupts
     11044 ±  3%     -16.1%       9267 ±  5%  interrupts.CPU40.RES:Rescheduling_interrupts
     11192 ±  5%     -18.5%       9124 ±  3%  interrupts.CPU42.RES:Rescheduling_interrupts
     11388 ±  5%     -19.5%       9171        interrupts.CPU43.RES:Rescheduling_interrupts
     12405 ±  4%     -13.0%      10791        interrupts.CPU44.RES:Rescheduling_interrupts
     12086 ±  7%     -15.6%      10203 ±  7%  interrupts.CPU45.RES:Rescheduling_interrupts
     93.75 ±155%   +5034.9%       4814 ±  6%  interrupts.CPU46.NMI:Non-maskable_interrupts
     93.75 ±155%   +5034.9%       4814 ±  6%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
     10908 ±  6%     -14.3%       9346 ±  3%  interrupts.CPU47.RES:Rescheduling_interrupts
     10191 ±  7%     -15.6%       8604 ±  6%  interrupts.CPU48.RES:Rescheduling_interrupts
     10421 ±  9%     -14.5%       8913        interrupts.CPU49.RES:Rescheduling_interrupts
     11994 ±  5%     -19.2%       9690 ±  3%  interrupts.CPU5.RES:Rescheduling_interrupts
      3398 ± 58%    -100.0%       0.00        interrupts.CPU50.NMI:Non-maskable_interrupts
      3398 ± 58%    -100.0%       0.00        interrupts.CPU50.PMI:Performance_monitoring_interrupts
     10739 ±  4%     -17.6%       8844 ±  4%  interrupts.CPU50.RES:Rescheduling_interrupts
     10198 ± 10%     -16.1%       8552 ±  3%  interrupts.CPU51.RES:Rescheduling_interrupts
     11673 ±  6%     -19.0%       9456 ±  7%  interrupts.CPU52.RES:Rescheduling_interrupts
     12088 ±  8%     -18.9%       9808 ±  5%  interrupts.CPU53.RES:Rescheduling_interrupts
     15591 ±  3%     -15.4%      13192 ±  2%  interrupts.CPU54.RES:Rescheduling_interrupts
     12738 ±  3%     -18.7%      10357 ±  2%  interrupts.CPU55.RES:Rescheduling_interrupts
     11467 ±  7%     -15.1%       9732 ±  2%  interrupts.CPU56.RES:Rescheduling_interrupts
      1174 ± 98%    +313.2%       4852 ±  8%  interrupts.CPU57.NMI:Non-maskable_interrupts
      1174 ± 98%    +313.2%       4852 ±  8%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
     11791 ±  5%      -8.2%      10822        interrupts.CPU57.RES:Rescheduling_interrupts
     10778 ±  7%     -15.6%       9094        interrupts.CPU58.RES:Rescheduling_interrupts
     10887 ±  7%     -15.7%       9183        interrupts.CPU59.RES:Rescheduling_interrupts
     11517 ±  9%     -18.4%       9395 ±  3%  interrupts.CPU6.RES:Rescheduling_interrupts
     12452 ±  4%     -12.2%      10931        interrupts.CPU61.RES:Rescheduling_interrupts
     76.75 ±173%   +6202.3%       4837 ±  6%  interrupts.CPU62.NMI:Non-maskable_interrupts
     76.75 ±173%   +6202.3%       4837 ±  6%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
      9.50 ±173%   +1194.7%     123.00 ± 86%  interrupts.CPU63.NMI:Non-maskable_interrupts
      9.50 ±173%   +1194.7%     123.00 ± 86%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
     10589 ±  8%     -17.2%       8766 ±  3%  interrupts.CPU63.RES:Rescheduling_interrupts
     11311 ±  9%     -14.8%       9641 ±  2%  interrupts.CPU64.RES:Rescheduling_interrupts
      1059 ±173%    +111.6%       2242 ± 97%  interrupts.CPU65.NMI:Non-maskable_interrupts
      1059 ±173%    +111.6%       2242 ± 97%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
      9977 ±  6%     -10.7%       8910        interrupts.CPU65.RES:Rescheduling_interrupts
     52.75 ±156%     -97.2%       1.50 ± 33%  interrupts.CPU66.NMI:Non-maskable_interrupts
     52.75 ±156%     -97.2%       1.50 ± 33%  interrupts.CPU66.PMI:Performance_monitoring_interrupts
     10196 ±  6%     -15.8%       8583 ±  4%  interrupts.CPU66.RES:Rescheduling_interrupts
      1030 ±166%    +244.1%       3545 ± 29%  interrupts.CPU67.NMI:Non-maskable_interrupts
      1030 ±166%    +244.1%       3545 ± 29%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
     12609 ±  4%     -17.6%      10389 ±  2%  interrupts.CPU68.RES:Rescheduling_interrupts
     11849 ±  7%     -16.7%       9867        interrupts.CPU69.RES:Rescheduling_interrupts
     12074 ±  6%     -19.1%       9767 ±  3%  interrupts.CPU7.RES:Rescheduling_interrupts
     10885 ±  2%     -16.4%       9097 ±  2%  interrupts.CPU70.RES:Rescheduling_interrupts
     13025 ±  9%     -23.9%       9908        interrupts.CPU71.RES:Rescheduling_interrupts
     13167 ±  4%     -15.1%      11182 ±  3%  interrupts.CPU72.RES:Rescheduling_interrupts
      1786 ±112%    +113.0%       3803 ± 37%  interrupts.CPU73.NMI:Non-maskable_interrupts
      1786 ±112%    +113.0%       3803 ± 37%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
     13026 ± 10%     -17.2%      10791 ±  3%  interrupts.CPU73.RES:Rescheduling_interrupts
     11187 ±  5%     -15.0%       9507 ±  5%  interrupts.CPU74.RES:Rescheduling_interrupts
     12505 ±  3%     -14.3%      10719 ±  2%  interrupts.CPU76.RES:Rescheduling_interrupts
     10949 ±  6%     -11.6%       9680        interrupts.CPU8.RES:Rescheduling_interrupts
     10064 ±  8%     -18.4%       8214        interrupts.CPU80.RES:Rescheduling_interrupts
      2392 ± 73%     -99.9%       3.00 ±100%  interrupts.CPU82.NMI:Non-maskable_interrupts
      2392 ± 73%     -99.9%       3.00 ±100%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
     10354 ±  8%     -16.7%       8624 ±  4%  interrupts.CPU82.RES:Rescheduling_interrupts
      1082 ±160%    +235.1%       3628 ± 28%  interrupts.CPU84.NMI:Non-maskable_interrupts
      1082 ±160%    +235.1%       3628 ± 28%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
     11224 ±  6%     -17.3%       9280        interrupts.CPU85.RES:Rescheduling_interrupts
      2319 ± 99%    +109.4%       4857 ±  6%  interrupts.CPU89.NMI:Non-maskable_interrupts
      2319 ± 99%    +109.4%       4857 ±  6%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
    261.00 ± 40%     +47.3%     384.50 ± 43%  interrupts.CPU9.54:PCI-MSI.1572873-edge.eth0-TxRx-9
    800.75 ±162%    +258.5%       2870 ± 88%  interrupts.CPU9.NMI:Non-maskable_interrupts
    800.75 ±162%    +258.5%       2870 ± 88%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
     10918 ±  6%     -16.9%       9070        interrupts.CPU9.RES:Rescheduling_interrupts
     13417 ±  9%     -19.6%      10786 ±  3%  interrupts.CPU90.RES:Rescheduling_interrupts
      2188 ± 67%     +59.6%       3492 ± 25%  interrupts.CPU91.NMI:Non-maskable_interrupts
      2188 ± 67%     +59.6%       3492 ± 25%  interrupts.CPU91.PMI:Performance_monitoring_interrupts
     11478 ±  4%     -15.5%       9703 ±  7%  interrupts.CPU92.RES:Rescheduling_interrupts
     11186 ±  7%     -13.3%       9702 ±  5%  interrupts.CPU94.RES:Rescheduling_interrupts
     10416 ±  9%     -14.6%       8892        interrupts.CPU95.RES:Rescheduling_interrupts
     10373 ±  7%     -12.4%       9091        interrupts.CPU96.RES:Rescheduling_interrupts
      4550 ±  6%     -46.3%       2445 ±  7%  interrupts.CPU97.NMI:Non-maskable_interrupts
      4550 ±  6%     -46.3%       2445 ±  7%  interrupts.CPU97.PMI:Performance_monitoring_interrupts
     10449 ±  8%     -13.9%       8993 ±  4%  interrupts.CPU97.RES:Rescheduling_interrupts
      4478 ±  7%     -71.7%       1268 ± 97%  interrupts.CPU99.NMI:Non-maskable_interrupts
      4478 ±  7%     -71.7%       1268 ± 97%  interrupts.CPU99.PMI:Performance_monitoring_interrupts
   1602258 ±  6%     -13.8%    1381405 ±  3%  interrupts.RES:Rescheduling_interrupts



***************************************************************************************************
lkp-bdw-ex1: 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 496G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/2000/debian-x86_64-2019-05-14.cgz/lkp-bdw-ex1/fork_test/aim7/0xb000036

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :2           50%           1:2     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
           :2           50%           1:2     dmesg.WARNING:stack_recursion



***************************************************************************************************
lkp-bdw-ex1: 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 496G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/8000/debian-x86_64-2019-05-14.cgz/lkp-bdw-ex1/misc_rtns_1/aim7/0xb000036

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          0:4           -5%           0:2     perf-profile.children.cycles-pp.error_entry
          0:4           -3%           0:2     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
      2288 ±  2%      +9.2%       2500        aim7.time.involuntary_context_switches
      2890            +2.9%       2974        aim7.time.maximum_resident_set_size
      0.00 ± 14%      -0.0        0.00 ±  2%  mpstat.cpu.all.iowait%
      0.01 ± 30%      +0.0        0.01 ±  6%  mpstat.cpu.all.soft%
     30.00 ±173%    +383.3%     145.00 ±  9%  proc-vmstat.nr_mlock
     94893 ±  3%      +6.4%     100935 ±  3%  proc-vmstat.nr_page_table_pages
      5058            +1.5%       5134        proc-vmstat.unevictable_pgs_culled
    252673            +6.3%     268545 ±  5%  numa-meminfo.node1.Unevictable
    380351 ± 60%     -73.7%      99869 ± 27%  numa-meminfo.node2.AnonPages
    176112 ± 88%     -90.9%      16067 ± 19%  numa-meminfo.node2.PageTables
      4988 ±  2%     +55.8%       7772 ± 31%  numa-meminfo.node3.Mapped
      4180 ±  9%     +21.1%       5062 ±  7%  slabinfo.eventpoll_pwq.active_objs
      4180 ±  9%     +21.1%       5062 ±  7%  slabinfo.eventpoll_pwq.num_objs
    541.75 ± 14%     +25.5%     680.00        slabinfo.skbuff_fclone_cache.active_objs
    541.75 ± 14%     +25.5%     680.00        slabinfo.skbuff_fclone_cache.num_objs
      6.20 ±  5%      +0.4        6.57 ±  3%  perf-stat.i.cache-miss-rate%
      0.28            +0.0        0.29        perf-stat.i.dTLB-store-miss-rate%
  12479108            -2.2%   12200011        perf-stat.i.node-load-misses
      0.29            +0.0        0.29        perf-stat.overall.dTLB-store-miss-rate%
  12329381            -2.2%   12053893        perf-stat.ps.node-load-misses
     53.00 ±  3%    +140.6%     127.50 ± 49%  interrupts.139:PCI-MSI.1574921-edge.eth3-TxRx-9
      1996 ± 10%     -16.8%       1660        interrupts.CPU160.RES:Rescheduling_interrupts
      1968 ±  9%     -20.2%       1570 ±  4%  interrupts.CPU163.RES:Rescheduling_interrupts
      1979 ± 10%     -20.4%       1575 ±  9%  interrupts.CPU56.RES:Rescheduling_interrupts
      1926 ±  8%     -14.6%       1645 ±  2%  interrupts.CPU66.RES:Rescheduling_interrupts
     53.00 ±  3%    +140.6%     127.50 ± 49%  interrupts.CPU9.139:PCI-MSI.1574921-edge.eth3-TxRx-9
      1235            +8.3%       1337 ±  6%  numa-vmstat.node1.nr_mapped
     63169            +6.3%      67134 ±  5%  numa-vmstat.node1.nr_unevictable
     63169            +6.3%      67134 ±  5%  numa-vmstat.node1.nr_zone_unevictable
     95077 ± 61%     -73.7%      24959 ± 28%  numa-vmstat.node2.nr_anon_pages
     44094 ± 88%     -90.9%       4014 ± 19%  numa-vmstat.node2.nr_page_table_pages
      1309 ± 10%     +57.2%       2058 ± 24%  numa-vmstat.node3.nr_mapped
    376579 ±  7%     +22.1%     459946 ± 12%  numa-vmstat.node3.numa_hit
    278936 ± 13%     +24.4%     347071 ± 16%  numa-vmstat.node3.numa_local
     14.31 ± 54%   +1356.5%     208.44 ± 91%  sched_debug.cfs_rq:/.load_avg.avg
    485.50 ± 10%   +4220.5%      20976 ± 97%  sched_debug.cfs_rq:/.load_avg.max
     64.90 ± 34%   +3027.9%       2030 ± 96%  sched_debug.cfs_rq:/.load_avg.stddev
     58047 ± 10%     +23.4%      71641 ±  2%  sched_debug.cfs_rq:/.min_vruntime.stddev
     37.00 ±  4%     -12.8%      32.25        sched_debug.cfs_rq:/.runnable_load_avg.max
     58045 ± 10%     +23.4%      71609 ±  2%  sched_debug.cfs_rq:/.spread0.stddev
    149.17 ± 15%     +30.3%     194.33 ±  5%  sched_debug.cfs_rq:/.util_avg.avg
      8.45 ± 34%     +53.5%      12.97 ± 10%  sched_debug.cfs_rq:/.util_est_enqueued.avg
     42.87 ± 21%     +24.7%      53.47        sched_debug.cfs_rq:/.util_est_enqueued.stddev
     37.00 ±  4%     -12.8%      32.25        sched_debug.cpu.cpu_load[0].max
      0.85 ± 10%     +22.9%       1.04        sched_debug.cpu.cpu_load[4].avg
     70.12 ± 11%     +48.0%     103.75 ± 22%  sched_debug.cpu.cpu_load[4].max
      5.40 ± 10%     +43.2%       7.73 ± 19%  sched_debug.cpu.cpu_load[4].stddev
     22047 ±  3%      +7.5%      23711 ±  7%  sched_debug.cpu.nr_switches.max
     13940 ±  4%     +15.9%      16158 ±  6%  sched_debug.cpu.ttwu_count.max
    433.88 ± 11%     +20.1%     521.25 ±  3%  sched_debug.cpu.ttwu_local.max
     37.71 ±  4%     +10.2%      41.56        sched_debug.cpu.ttwu_local.stddev
      0.49 ±104%      +0.9        1.42 ± 32%  perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      0.59 ±103%      +0.9        1.53 ± 23%  perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
      0.61 ±103%      +0.9        1.55 ± 21%  perf-profile.calltrace.cycles-pp.page_fault
      0.61 ±103%      +0.9        1.55 ± 21%  perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
      0.46 ±104%      +1.9        2.33 ± 61%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      1.05 ± 77%      -0.9        0.18 ±100%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.99 ± 87%      -0.9        0.12 ±100%  perf-profile.children.cycles-pp.load_balance
      0.88 ±103%      -0.8        0.09 ±100%  perf-profile.children.cycles-pp.find_busiest_group
      0.97 ± 87%      -0.8        0.18 ±100%  perf-profile.children.cycles-pp.native_write_msr
      0.84 ±110%      -0.8        0.09 ± 99%  perf-profile.children.cycles-pp.update_sd_lb_stats
      0.72 ±137%      -0.7        0.06 ±100%  perf-profile.children.cycles-pp.ksys_write
      0.70 ±142%      -0.7        0.04 ±100%  perf-profile.children.cycles-pp.vfs_write
      0.70 ±143%      -0.7        0.04 ±100%  perf-profile.children.cycles-pp.new_sync_write
      0.71 ±141%      -0.7        0.05 ±100%  perf-profile.children.cycles-pp.idle_cpu
      0.23 ± 60%      +0.9        1.11 ± 69%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.09 ±102%      +0.9        1.02 ± 84%  perf-profile.children.cycles-pp.lockref_get_not_dead
      0.10 ±101%      +0.9        1.04 ± 81%  perf-profile.children.cycles-pp.unlazy_walk
      0.07 ±103%      +1.0        1.02 ± 85%  perf-profile.children.cycles-pp.filp_close
      0.05 ±103%      +1.0        1.00 ± 88%  perf-profile.children.cycles-pp.__do_fault
      0.01 ±173%      +1.0        0.98 ± 92%  perf-profile.children.cycles-pp.walk_component
      0.56 ± 83%      +1.8        2.38 ± 58%  perf-profile.children.cycles-pp.handle_mm_fault
      0.52 ± 83%      +1.8        2.34 ± 61%  perf-profile.children.cycles-pp.__handle_mm_fault
      0.66 ± 86%      +1.8        2.48 ± 52%  perf-profile.children.cycles-pp.__do_page_fault
      0.68 ± 86%      +1.8        2.50 ± 50%  perf-profile.children.cycles-pp.do_page_fault
      0.10 ±100%      +1.9        2.00 ± 88%  perf-profile.children.cycles-pp.link_path_walk
      0.97 ± 87%      -0.8        0.18 ±100%  perf-profile.self.cycles-pp.native_write_msr
      0.71 ±141%      -0.7        0.05 ±100%  perf-profile.self.cycles-pp.idle_cpu
      0.12 ± 62%      +0.9        1.04 ± 80%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.09 ±102%      +0.9        1.02 ± 84%  perf-profile.self.cycles-pp.lockref_get_not_dead
     40636 ± 10%     +17.1%      47594 ±  6%  softirqs.CPU0.TIMER
     14166           +14.8%      16269 ±  2%  softirqs.CPU1.SCHED
     38044 ± 13%     +29.5%      49279 ±  9%  softirqs.CPU119.TIMER
     35559 ± 11%     +16.2%      41310 ±  4%  softirqs.CPU120.TIMER
     35455 ± 11%     +17.0%      41488 ±  2%  softirqs.CPU121.TIMER
     35068 ± 10%     +16.1%      40722 ±  5%  softirqs.CPU122.TIMER
     35803 ± 10%     +13.0%      40466 ±  5%  softirqs.CPU123.TIMER
     35825 ± 10%     +14.1%      40893 ±  3%  softirqs.CPU124.TIMER
     35520 ± 11%     +13.7%      40369 ±  4%  softirqs.CPU125.TIMER
     35467 ± 12%     +17.5%      41691 ±  8%  softirqs.CPU126.TIMER
     34958 ± 11%     +15.1%      40232 ±  4%  softirqs.CPU127.TIMER
     35118 ± 11%     +14.4%      40160 ±  4%  softirqs.CPU128.TIMER
     35132 ± 11%     +16.9%      41075 ±  6%  softirqs.CPU129.TIMER
     35102 ± 11%     +16.8%      40985 ±  3%  softirqs.CPU130.TIMER
     34985 ± 11%     +14.6%      40090 ±  4%  softirqs.CPU131.TIMER
     34851 ± 11%     +15.2%      40142 ±  4%  softirqs.CPU132.TIMER
     35377 ± 12%     +13.3%      40081 ±  4%  softirqs.CPU133.TIMER
     35590 ± 11%     +13.4%      40345 ±  4%  softirqs.CPU134.TIMER
     35790 ±  8%     +12.2%      40158 ±  4%  softirqs.CPU135.TIMER
     35199 ± 10%     +14.1%      40152 ±  4%  softirqs.CPU136.TIMER
     35621 ± 10%     +12.7%      40149 ±  4%  softirqs.CPU137.TIMER
     34932 ± 11%     +14.9%      40122 ±  4%  softirqs.CPU138.TIMER
     35099 ± 10%     +14.2%      40084 ±  4%  softirqs.CPU139.TIMER
     35072 ± 11%     +14.2%      40039 ±  4%  softirqs.CPU140.TIMER
     35675 ± 12%     +13.5%      40489 ±  5%  softirqs.CPU141.TIMER
     35764 ± 12%     +12.6%      40259 ±  4%  softirqs.CPU142.TIMER
     36980 ±  9%     +14.4%      42313 ±  4%  softirqs.CPU25.TIMER
     36698 ± 11%     +13.6%      41699 ±  4%  softirqs.CPU26.TIMER
     35995 ± 10%     +15.4%      41530 ±  4%  softirqs.CPU27.TIMER
     36891 ±  9%     +12.2%      41375 ±  4%  softirqs.CPU28.TIMER
     36396 ± 10%     +13.4%      41291 ±  4%  softirqs.CPU29.TIMER
     36554 ± 11%     +12.4%      41089 ±  4%  softirqs.CPU30.TIMER
     36128 ± 10%     +14.0%      41170 ±  4%  softirqs.CPU31.TIMER
     36231 ± 10%     +16.3%      42151 ±  6%  softirqs.CPU32.TIMER
     36349 ± 11%     +15.7%      42052 ±  6%  softirqs.CPU33.TIMER
     36689 ±  9%     +12.6%      41328 ±  4%  softirqs.CPU34.TIMER
     36109 ± 10%     +14.4%      41318 ±  4%  softirqs.CPU35.TIMER
     36038 ± 10%     +14.0%      41098 ±  3%  softirqs.CPU36.TIMER
     36379 ± 11%     +13.0%      41123 ±  4%  softirqs.CPU37.TIMER
     36229 ± 11%     +13.5%      41132 ±  4%  softirqs.CPU38.TIMER
     36601 ± 11%     +12.2%      41074 ±  3%  softirqs.CPU39.TIMER
     36723 ±  9%     +12.0%      41148 ±  3%  softirqs.CPU40.TIMER
     36789 ± 10%     +11.5%      41010 ±  3%  softirqs.CPU41.TIMER
     36238 ±  8%     +13.4%      41097 ±  3%  softirqs.CPU42.TIMER
     36335 ± 12%     +13.1%      41079 ±  3%  softirqs.CPU43.TIMER
     36114 ±  9%     +13.8%      41110 ±  4%  softirqs.CPU44.TIMER
     36427 ± 11%     +13.5%      41361 ±  4%  softirqs.CPU45.TIMER
     36312 ± 11%     +13.8%      41311 ±  4%  softirqs.CPU46.TIMER
     23726 ±  6%     +12.0%      26584 ±  3%  softirqs.CPU47.RCU
     13762 ±  2%     +12.5%      15485 ±  6%  softirqs.CPU48.SCHED
     38329 ± 11%     +30.0%      49841 ± 16%  softirqs.CPU48.TIMER



***************************************************************************************************
lkp-knm01: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
  gcc-7/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-2019-05-14.cgz/lkp-knm01/unlink1/will-it-scale

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      1119 ±  2%      +3.1%       1154        will-it-scale.time.involuntary_context_switches
    318.75 ±  5%      +9.2%     348.00 ±  3%  will-it-scale.time.percent_of_cpu_this_job_got
    928.56 ±  5%      +9.6%       1017 ±  3%  will-it-scale.time.system_time
      1.07 ±  7%      +0.2        1.26 ± 10%  mpstat.cpu.all.sys%
    275596            +1.2%     278949        proc-vmstat.nr_file_pages
     94.05            +2.6%      96.48 ±  2%  turbostat.PkgWatt
      1821 ±  7%     +14.4%       2084 ±  4%  sched_debug.cfs_rq:/.exec_clock.avg
     43331 ±  3%     +10.0%      47671 ±  3%  sched_debug.cfs_rq:/.load.stddev
    272381 ±  8%     -19.1%     220404 ±  5%  sched_debug.cfs_rq:/.min_vruntime.avg
     43257 ±  3%     +10.2%      47663 ±  3%  sched_debug.cfs_rq:/.runnable_weight.stddev
     21.49 ±  4%     -12.9%      18.71 ±  7%  sched_debug.cpu.cpu_load[0].stddev
    348.60 ±  4%     -15.0%     296.25 ±  9%  sched_debug.cpu.cpu_load[1].max
    348.25 ±  3%     -15.1%     295.83 ± 10%  sched_debug.cpu.cpu_load[2].max
     -5.76           -30.5%      -4.00        sched_debug.cpu.nr_uninterruptible.min
     62873 ±  7%     -20.3%      50079        sched_debug.cpu.sched_count.max
    152.30 ±  6%     +12.9%     171.99 ±  6%  sched_debug.cpu.ttwu_local.avg
 7.873e+08 ±  2%      +4.0%  8.191e+08        perf-stat.i.branch-instructions
      8.29 ±  7%      +1.8       10.08 ± 11%  perf-stat.i.branch-miss-rate%
  65211385 ±  8%     +26.6%   82541419 ± 12%  perf-stat.i.branch-misses
     12.91 ±  4%     +24.6%      16.08 ± 18%  perf-stat.i.cpi
  4.73e+10 ±  5%     +28.9%  6.098e+10 ± 19%  perf-stat.i.cpu-cycles
      1.80            +0.1        1.86 ±  3%  perf-stat.i.iTLB-load-miss-rate%
  67010767 ±  2%      +7.4%   71977406 ±  3%  perf-stat.i.iTLB-load-misses
 3.694e+09 ±  2%      +3.4%  3.819e+09        perf-stat.i.iTLB-loads
 3.691e+09 ±  2%      +3.4%  3.817e+09        perf-stat.i.instructions
      2527            +0.5%       2539        perf-stat.i.minor-faults
      2527            +0.5%       2540        perf-stat.i.page-faults
      8.28 ±  7%      +1.8       10.07 ± 11%  perf-stat.overall.branch-miss-rate%
     12.81 ±  4%     +24.6%      15.96 ± 18%  perf-stat.overall.cpi
      1.78            +0.1        1.85 ±  3%  perf-stat.overall.iTLB-load-miss-rate%
  36179580 ±  3%     +17.7%   42571444 ± 13%  perf-stat.overall.path-length
 7.846e+08 ±  2%      +4.0%  8.163e+08        perf-stat.ps.branch-instructions
  64994927 ±  8%     +26.6%   82258003 ± 12%  perf-stat.ps.branch-misses
 4.714e+10 ±  5%     +28.9%  6.076e+10 ± 19%  perf-stat.ps.cpu-cycles
  66779111 ±  2%      +7.4%   71724036 ±  3%  perf-stat.ps.iTLB-load-misses
 3.681e+09 ±  2%      +3.4%  3.806e+09        perf-stat.ps.iTLB-loads
 3.678e+09 ±  2%      +3.4%  3.804e+09        perf-stat.ps.instructions
      2519            +0.5%       2531        perf-stat.ps.minor-faults
      2519            +0.5%       2532        perf-stat.ps.page-faults
 1.105e+12 ±  2%      +3.8%  1.147e+12        perf-stat.total.instructions
    110711 ±  4%      +7.0%     118470 ±  5%  softirqs.CPU109.TIMER
    110627 ±  4%      +6.2%     117444 ±  5%  softirqs.CPU112.TIMER
    112989 ±  7%      +8.4%     122443 ±  5%  softirqs.CPU123.TIMER
    109980 ±  3%      +5.1%     115558 ±  4%  softirqs.CPU14.TIMER
    109702 ±  4%      +4.8%     114978 ±  4%  softirqs.CPU17.TIMER
    114458 ±  6%      +7.4%     122929 ±  4%  softirqs.CPU195.TIMER
    113939 ±  3%      +5.0%     119644 ±  5%  softirqs.CPU236.TIMER
    110976 ±  6%      +4.9%     116424 ±  5%  softirqs.CPU240.TIMER
    108141 ±  5%      +6.4%     115052 ±  7%  softirqs.CPU253.TIMER
    109346 ±  6%      +8.2%     118301 ±  2%  softirqs.CPU256.TIMER
    109601 ±  5%      +7.0%     117275 ±  4%  softirqs.CPU266.TIMER
    111080 ±  2%      +6.8%     118592 ±  5%  softirqs.CPU268.TIMER
    110244 ±  5%      +4.5%     115237 ±  6%  softirqs.CPU274.TIMER
    109345 ±  6%      +8.0%     118148 ±  8%  softirqs.CPU275.TIMER
    105278 ±  2%     +11.3%     117150 ±  8%  softirqs.CPU287.TIMER
     25916 ±  3%      +3.4%      26788 ±  4%  softirqs.CPU3.SCHED
    109550 ±  6%      +7.1%     117288 ±  7%  softirqs.CPU51.TIMER
    109353 ±  4%      +7.6%     117652 ±  6%  softirqs.CPU61.TIMER
    109018 ±  3%      +7.4%     117069 ±  6%  softirqs.CPU66.TIMER
    111093 ±  2%      +8.6%     120616 ±  8%  softirqs.CPU67.TIMER
    108126 ±  3%      +7.5%     116246 ±  6%  softirqs.CPU69.TIMER
    109627 ±  4%      +8.4%     118799 ±  6%  softirqs.CPU70.TIMER
    106067           +11.0%     117778 ±  7%  softirqs.CPU71.TIMER
    109417 ±  4%      +6.5%     116523 ±  6%  softirqs.CPU77.TIMER
    112795            +8.9%     122821 ±  7%  softirqs.CPU83.TIMER
    110905 ±  4%      +5.5%     117001 ±  5%  softirqs.CPU89.TIMER
     26108            +6.3%      27763 ±  5%  softirqs.CPU92.SCHED
      1.41 ± 11%      +0.2        1.58        perf-profile.calltrace.cycles-pp.perf_event_task_tick.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.93 ± 18%      +0.2        1.17 ±  3%  perf-profile.calltrace.cycles-pp.ktime_get.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.61 ± 12%      +0.3        1.89 ±  4%  perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
     93.17            +0.6       93.78        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     93.17            +0.6       93.78        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     93.05            +0.6       93.66        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      7.79 ±  6%      +0.7        8.52 ±  8%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      8.40 ±  7%      +0.8        9.25 ±  9%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      3.12            -0.3        2.84 ±  6%  perf-profile.children.cycles-pp.sched_clock_cpu
      1.03 ± 15%      -0.2        0.84 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock
      0.68 ± 11%      -0.1        0.62        perf-profile.children.cycles-pp.update_sd_lb_stats
      0.06 ±  6%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.__strcspn_sse42
      0.06 ±  6%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.generic_start_main
      0.06 ±  6%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.main
      0.06 ±  6%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.run_builtin
      0.13 ±  9%      -0.0        0.10 ± 10%  perf-profile.children.cycles-pp.__intel_pmu_enable_all
      0.12 ± 13%      +0.0        0.16 ±  9%  perf-profile.children.cycles-pp.retint_kernel
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.tick_nohz_idle_retain_tick
      1.68 ± 12%      +0.3        1.95 ±  5%  perf-profile.children.cycles-pp.clockevents_program_event
     93.17            +0.6       93.78        perf-profile.children.cycles-pp.start_secondary
      7.99 ±  6%      +0.7        8.64 ±  8%  perf-profile.children.cycles-pp.update_process_times
      8.59 ±  7%      +0.8        9.38 ±  9%  perf-profile.children.cycles-pp.tick_sched_handle
      3.99 ±  8%      -0.4        3.54 ±  4%  perf-profile.self.cycles-pp.__hrtimer_run_queues
      0.99 ± 15%      -0.2        0.81 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock
      0.55 ±  3%      -0.1        0.47 ±  5%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.09 ±  5%      -0.0        0.07 ±  7%  perf-profile.self.cycles-pp.irq_work_run_list
      0.16 ±  6%      +0.0        0.19        perf-profile.self.cycles-pp.lapic_next_deadline
      0.12 ± 15%      +0.0        0.15 ±  6%  perf-profile.self.cycles-pp.retint_kernel
      1198 ± 11%     -60.8%     470.50 ± 28%  interrupts.32:IR-PCI-MSI.2097155-edge.eth0-TxRx-2
    620.25 ± 12%     -43.2%     352.00 ± 25%  interrupts.33:IR-PCI-MSI.2097156-edge.eth0-TxRx-3
      1083 ± 37%     +71.1%       1853 ± 15%  interrupts.CPU112.NMI:Non-maskable_interrupts
      1083 ± 37%     +71.1%       1853 ± 15%  interrupts.CPU112.PMI:Performance_monitoring_interrupts
     62.50 ± 82%     -86.4%       8.50 ±  5%  interrupts.CPU112.TLB:TLB_shootdowns
      1137 ± 28%     +59.1%       1809 ± 14%  interrupts.CPU114.NMI:Non-maskable_interrupts
      1137 ± 28%     +59.1%       1809 ± 14%  interrupts.CPU114.PMI:Performance_monitoring_interrupts
      1137 ± 28%     +63.1%       1855 ± 13%  interrupts.CPU115.NMI:Non-maskable_interrupts
      1137 ± 28%     +63.1%       1855 ± 13%  interrupts.CPU115.PMI:Performance_monitoring_interrupts
      1196 ± 31%     +57.8%       1888 ± 16%  interrupts.CPU116.NMI:Non-maskable_interrupts
      1196 ± 31%     +57.8%       1888 ± 16%  interrupts.CPU116.PMI:Performance_monitoring_interrupts
      1198 ± 11%     -60.8%     470.50 ± 28%  interrupts.CPU12.32:IR-PCI-MSI.2097155-edge.eth0-TxRx-2
      1375 ± 23%     +43.4%       1972 ± 16%  interrupts.CPU120.NMI:Non-maskable_interrupts
      1375 ± 23%     +43.4%       1972 ± 16%  interrupts.CPU120.PMI:Performance_monitoring_interrupts
      1377 ± 24%     +45.2%       2000 ± 15%  interrupts.CPU121.NMI:Non-maskable_interrupts
      1377 ± 24%     +45.2%       2000 ± 15%  interrupts.CPU121.PMI:Performance_monitoring_interrupts
    799.75 ±  9%     +25.7%       1005 ±  6%  interrupts.CPU123.RES:Rescheduling_interrupts
    898.50 ± 17%     -25.9%     665.50 ±  9%  interrupts.CPU129.RES:Rescheduling_interrupts
    620.25 ± 12%     -43.2%     352.00 ± 25%  interrupts.CPU13.33:IR-PCI-MSI.2097156-edge.eth0-TxRx-3
    875.75 ±  7%     -28.5%     626.50 ±  9%  interrupts.CPU13.RES:Rescheduling_interrupts
      1219 ± 40%     +58.7%       1934 ± 12%  interrupts.CPU131.NMI:Non-maskable_interrupts
      1219 ± 40%     +58.7%       1934 ± 12%  interrupts.CPU131.PMI:Performance_monitoring_interrupts
      1011 ± 45%     +93.8%       1960 ± 12%  interrupts.CPU132.NMI:Non-maskable_interrupts
      1011 ± 45%     +93.8%       1960 ± 12%  interrupts.CPU132.PMI:Performance_monitoring_interrupts
      1186 ± 34%     +51.9%       1802 ± 11%  interrupts.CPU133.NMI:Non-maskable_interrupts
      1186 ± 34%     +51.9%       1802 ± 11%  interrupts.CPU133.PMI:Performance_monitoring_interrupts
      1158 ± 33%     +58.5%       1836 ± 18%  interrupts.CPU134.NMI:Non-maskable_interrupts
      1158 ± 33%     +58.5%       1836 ± 18%  interrupts.CPU134.PMI:Performance_monitoring_interrupts
      1858 ± 64%     -66.4%     625.00 ± 23%  interrupts.CPU134.RES:Rescheduling_interrupts
      1290 ± 27%     +39.1%       1794 ± 15%  interrupts.CPU135.NMI:Non-maskable_interrupts
      1290 ± 27%     +39.1%       1794 ± 15%  interrupts.CPU135.PMI:Performance_monitoring_interrupts
      1399 ± 71%     -57.9%     589.50 ±  3%  interrupts.CPU135.RES:Rescheduling_interrupts
      1173 ± 35%     +58.8%       1863 ± 16%  interrupts.CPU136.NMI:Non-maskable_interrupts
      1173 ± 35%     +58.8%       1863 ± 16%  interrupts.CPU136.PMI:Performance_monitoring_interrupts
      1342 ± 26%     +44.2%       1936 ± 17%  interrupts.CPU137.NMI:Non-maskable_interrupts
      1342 ± 26%     +44.2%       1936 ± 17%  interrupts.CPU137.PMI:Performance_monitoring_interrupts
    885.50 ± 14%     -30.3%     617.50 ±  9%  interrupts.CPU137.RES:Rescheduling_interrupts
    645.00 ± 12%     +19.1%     768.00 ±  6%  interrupts.CPU14.RES:Rescheduling_interrupts
    861.75 ±  9%     -31.5%     590.50 ±  4%  interrupts.CPU140.RES:Rescheduling_interrupts
      1142 ± 28%     +66.7%       1904 ± 14%  interrupts.CPU144.NMI:Non-maskable_interrupts
      1142 ± 28%     +66.7%       1904 ± 14%  interrupts.CPU144.PMI:Performance_monitoring_interrupts
      1386 ± 27%     +39.3%       1931 ± 13%  interrupts.CPU147.NMI:Non-maskable_interrupts
      1386 ± 27%     +39.3%       1931 ± 13%  interrupts.CPU147.PMI:Performance_monitoring_interrupts
      1392 ± 25%     +42.2%       1981 ± 14%  interrupts.CPU148.NMI:Non-maskable_interrupts
      1392 ± 25%     +42.2%       1981 ± 14%  interrupts.CPU148.PMI:Performance_monitoring_interrupts
      1383 ± 24%     +34.2%       1856 ± 11%  interrupts.CPU149.NMI:Non-maskable_interrupts
      1383 ± 24%     +34.2%       1856 ± 11%  interrupts.CPU149.PMI:Performance_monitoring_interrupts
      1365 ± 21%     +36.8%       1868 ± 14%  interrupts.CPU150.NMI:Non-maskable_interrupts
      1365 ± 21%     +36.8%       1868 ± 14%  interrupts.CPU150.PMI:Performance_monitoring_interrupts
      1386 ± 25%     +29.3%       1792 ± 11%  interrupts.CPU151.NMI:Non-maskable_interrupts
      1386 ± 25%     +29.3%       1792 ± 11%  interrupts.CPU151.PMI:Performance_monitoring_interrupts
      1381 ± 21%     +36.9%       1891 ± 13%  interrupts.CPU152.NMI:Non-maskable_interrupts
      1381 ± 21%     +36.9%       1891 ± 13%  interrupts.CPU152.PMI:Performance_monitoring_interrupts
      1348 ± 24%     +38.9%       1872 ± 14%  interrupts.CPU153.NMI:Non-maskable_interrupts
      1348 ± 24%     +38.9%       1872 ± 14%  interrupts.CPU153.PMI:Performance_monitoring_interrupts
      1435 ± 27%     +39.5%       2003 ± 13%  interrupts.CPU157.NMI:Non-maskable_interrupts
      1435 ± 27%     +39.5%       2003 ± 13%  interrupts.CPU157.PMI:Performance_monitoring_interrupts
     32.75 ± 64%    +193.1%      96.00 ± 23%  interrupts.CPU158.TLB:TLB_shootdowns
      1297 ± 26%     +36.2%       1766 ±  8%  interrupts.CPU159.NMI:Non-maskable_interrupts
      1297 ± 26%     +36.2%       1766 ±  8%  interrupts.CPU159.PMI:Performance_monitoring_interrupts
    920.00 ±  8%     -36.1%     587.50 ± 17%  interrupts.CPU159.RES:Rescheduling_interrupts
      1329 ± 25%     +39.4%       1853 ± 18%  interrupts.CPU160.NMI:Non-maskable_interrupts
      1329 ± 25%     +39.4%       1853 ± 18%  interrupts.CPU160.PMI:Performance_monitoring_interrupts
      1377 ± 26%     +37.2%       1889 ± 14%  interrupts.CPU161.NMI:Non-maskable_interrupts
      1377 ± 26%     +37.2%       1889 ± 14%  interrupts.CPU161.PMI:Performance_monitoring_interrupts
      1397 ± 20%     +36.8%       1910 ± 13%  interrupts.CPU162.NMI:Non-maskable_interrupts
      1397 ± 20%     +36.8%       1910 ± 13%  interrupts.CPU162.PMI:Performance_monitoring_interrupts
      1350 ± 23%     +33.0%       1796 ± 11%  interrupts.CPU163.NMI:Non-maskable_interrupts
      1350 ± 23%     +33.0%       1796 ± 11%  interrupts.CPU163.PMI:Performance_monitoring_interrupts
      1408 ± 26%     +43.0%       2013 ± 18%  interrupts.CPU164.NMI:Non-maskable_interrupts
      1408 ± 26%     +43.0%       2013 ± 18%  interrupts.CPU164.PMI:Performance_monitoring_interrupts
    695.25 ± 21%     +21.2%     842.50 ±  9%  interrupts.CPU17.RES:Rescheduling_interrupts
      1396 ± 25%     +30.5%       1823 ±  7%  interrupts.CPU184.NMI:Non-maskable_interrupts
      1396 ± 25%     +30.5%       1823 ±  7%  interrupts.CPU184.PMI:Performance_monitoring_interrupts
      1354 ± 25%     +36.8%       1852 ±  8%  interrupts.CPU185.NMI:Non-maskable_interrupts
      1354 ± 25%     +36.8%       1852 ±  8%  interrupts.CPU185.PMI:Performance_monitoring_interrupts
      1169 ± 27%     +62.3%       1897 ± 17%  interrupts.CPU190.NMI:Non-maskable_interrupts
      1169 ± 27%     +62.3%       1897 ± 17%  interrupts.CPU190.PMI:Performance_monitoring_interrupts
      1059 ± 21%     -27.3%     770.50        interrupts.CPU190.RES:Rescheduling_interrupts
      1124 ± 29%     +64.0%       1843 ± 17%  interrupts.CPU191.NMI:Non-maskable_interrupts
      1124 ± 29%     +64.0%       1843 ± 17%  interrupts.CPU191.PMI:Performance_monitoring_interrupts
      1159 ± 29%     +71.5%       1988 ± 17%  interrupts.CPU192.NMI:Non-maskable_interrupts
      1159 ± 29%     +71.5%       1988 ± 17%  interrupts.CPU192.PMI:Performance_monitoring_interrupts
      1223 ± 30%     +57.9%       1931 ± 12%  interrupts.CPU194.NMI:Non-maskable_interrupts
      1223 ± 30%     +57.9%       1931 ± 12%  interrupts.CPU194.PMI:Performance_monitoring_interrupts
      1234 ± 32%     +53.0%       1888 ± 11%  interrupts.CPU195.NMI:Non-maskable_interrupts
      1234 ± 32%     +53.0%       1888 ± 11%  interrupts.CPU195.PMI:Performance_monitoring_interrupts
      1173 ± 25%     +68.7%       1980 ± 11%  interrupts.CPU196.NMI:Non-maskable_interrupts
      1173 ± 25%     +68.7%       1980 ± 11%  interrupts.CPU196.PMI:Performance_monitoring_interrupts
    685.25 ± 10%     +19.0%     815.50 ± 16%  interrupts.CPU2.NMI:Non-maskable_interrupts
    685.25 ± 10%     +19.0%     815.50 ± 16%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
     79.75 ± 56%     -67.4%      26.00 ±  7%  interrupts.CPU208.TLB:TLB_shootdowns
    786.00 ±  9%     +90.1%       1494 ± 47%  interrupts.CPU220.NMI:Non-maskable_interrupts
    786.00 ±  9%     +90.1%       1494 ± 47%  interrupts.CPU220.PMI:Performance_monitoring_interrupts
    749.25 ±  4%    +100.5%       1502 ± 45%  interrupts.CPU221.NMI:Non-maskable_interrupts
    749.25 ±  4%    +100.5%       1502 ± 45%  interrupts.CPU221.PMI:Performance_monitoring_interrupts
    749.75 ±  3%    +149.4%       1870 ± 18%  interrupts.CPU223.NMI:Non-maskable_interrupts
    749.75 ±  3%    +149.4%       1870 ± 18%  interrupts.CPU223.PMI:Performance_monitoring_interrupts
    706.25 ±  3%    +156.1%       1809 ± 19%  interrupts.CPU224.NMI:Non-maskable_interrupts
    706.25 ±  3%    +156.1%       1809 ± 19%  interrupts.CPU224.PMI:Performance_monitoring_interrupts
     22.00 ± 68%    +256.8%      78.50 ± 40%  interrupts.CPU227.TLB:TLB_shootdowns
      1002 ± 37%     +45.5%       1458 ± 43%  interrupts.CPU231.NMI:Non-maskable_interrupts
      1002 ± 37%     +45.5%       1458 ± 43%  interrupts.CPU231.PMI:Performance_monitoring_interrupts
    977.50 ± 35%     +55.0%       1515 ± 46%  interrupts.CPU233.NMI:Non-maskable_interrupts
    977.50 ± 35%     +55.0%       1515 ± 46%  interrupts.CPU233.PMI:Performance_monitoring_interrupts
    970.00 ± 34%     +57.6%       1529 ± 45%  interrupts.CPU239.NMI:Non-maskable_interrupts
    970.00 ± 34%     +57.6%       1529 ± 45%  interrupts.CPU239.PMI:Performance_monitoring_interrupts
      1999 ±100%     -67.0%     659.00        interrupts.CPU239.RES:Rescheduling_interrupts
    979.75 ± 35%     +53.1%       1500 ± 43%  interrupts.CPU240.NMI:Non-maskable_interrupts
    979.75 ± 35%     +53.1%       1500 ± 43%  interrupts.CPU240.PMI:Performance_monitoring_interrupts
      1019 ± 38%     +46.7%       1495 ± 43%  interrupts.CPU241.NMI:Non-maskable_interrupts
      1019 ± 38%     +46.7%       1495 ± 43%  interrupts.CPU241.PMI:Performance_monitoring_interrupts
    687.75 ± 19%     +51.1%       1039 ±  3%  interrupts.CPU241.RES:Rescheduling_interrupts
    872.00 ± 11%     +61.3%       1406 ± 19%  interrupts.CPU243.RES:Rescheduling_interrupts
    763.00 ±  7%     -12.1%     670.50        interrupts.CPU25.RES:Rescheduling_interrupts
    933.25 ± 33%     +53.0%       1428 ± 43%  interrupts.CPU253.NMI:Non-maskable_interrupts
    933.25 ± 33%     +53.0%       1428 ± 43%  interrupts.CPU253.PMI:Performance_monitoring_interrupts
    924.75 ± 35%     +57.1%       1453 ± 46%  interrupts.CPU256.NMI:Non-maskable_interrupts
    924.75 ± 35%     +57.1%       1453 ± 46%  interrupts.CPU256.PMI:Performance_monitoring_interrupts
    938.50 ± 36%     +54.5%       1450 ± 44%  interrupts.CPU257.NMI:Non-maskable_interrupts
    938.50 ± 36%     +54.5%       1450 ± 44%  interrupts.CPU257.PMI:Performance_monitoring_interrupts
      1122 ± 29%     -43.5%     634.00 ± 13%  interrupts.CPU258.RES:Rescheduling_interrupts
      1225 ± 27%     -40.6%     727.50 ±  5%  interrupts.CPU259.RES:Rescheduling_interrupts
     41.75 ± 27%     +79.6%      75.00 ± 33%  interrupts.CPU261.TLB:TLB_shootdowns
    725.50 ±  8%     -24.5%     548.00 ± 10%  interrupts.CPU264.RES:Rescheduling_interrupts
    741.25 ±  7%    +103.8%       1511 ± 47%  interrupts.CPU268.NMI:Non-maskable_interrupts
    741.25 ±  7%    +103.8%       1511 ± 47%  interrupts.CPU268.PMI:Performance_monitoring_interrupts
    683.25 ±  7%    +113.6%       1459 ± 44%  interrupts.CPU269.NMI:Non-maskable_interrupts
    683.25 ±  7%    +113.6%       1459 ± 44%  interrupts.CPU269.PMI:Performance_monitoring_interrupts
    793.00 ± 17%     +56.6%       1242 ± 26%  interrupts.CPU271.RES:Rescheduling_interrupts
    690.50 ±  8%     +97.6%       1364 ± 44%  interrupts.CPU274.NMI:Non-maskable_interrupts
    690.50 ±  8%     +97.6%       1364 ± 44%  interrupts.CPU274.PMI:Performance_monitoring_interrupts
    690.50 ±  9%     +95.7%       1351 ± 46%  interrupts.CPU276.NMI:Non-maskable_interrupts
    690.50 ±  9%     +95.7%       1351 ± 46%  interrupts.CPU276.PMI:Performance_monitoring_interrupts
    888.75 ± 22%     -26.6%     652.00 ±  2%  interrupts.CPU278.RES:Rescheduling_interrupts
    944.25 ± 43%     +44.2%       1362 ± 45%  interrupts.CPU279.NMI:Non-maskable_interrupts
    944.25 ± 43%     +44.2%       1362 ± 45%  interrupts.CPU279.PMI:Performance_monitoring_interrupts
    594.00 ± 13%     +18.5%     704.00 ±  3%  interrupts.CPU28.RES:Rescheduling_interrupts
    812.75 ±  9%     -34.2%     534.50 ± 25%  interrupts.CPU280.RES:Rescheduling_interrupts
    724.75 ±  7%     -16.3%     606.50 ±  4%  interrupts.CPU281.RES:Rescheduling_interrupts
    660.50 ±  8%     +29.6%     856.00 ± 17%  interrupts.CPU3.NMI:Non-maskable_interrupts
    660.50 ±  8%     +29.6%     856.00 ± 17%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
    697.75 ± 15%     +64.2%       1145 ± 12%  interrupts.CPU30.NMI:Non-maskable_interrupts
    697.75 ± 15%     +64.2%       1145 ± 12%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
    694.25 ±  8%     +72.5%       1197 ± 17%  interrupts.CPU31.NMI:Non-maskable_interrupts
    694.25 ±  8%     +72.5%       1197 ± 17%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
    643.25 ± 14%     +33.2%     857.00 ± 19%  interrupts.CPU32.NMI:Non-maskable_interrupts
    643.25 ± 14%     +33.2%     857.00 ± 19%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
    861.75 ±  7%     -17.3%     712.50        interrupts.CPU32.RES:Rescheduling_interrupts
    633.75 ± 14%     +37.3%     870.00 ± 19%  interrupts.CPU33.NMI:Non-maskable_interrupts
    633.75 ± 14%     +37.3%     870.00 ± 19%  interrupts.CPU33.PMI:Performance_monitoring_interrupts
    642.75 ±  9%     +34.3%     863.00 ± 18%  interrupts.CPU34.NMI:Non-maskable_interrupts
    642.75 ±  9%     +34.3%     863.00 ± 18%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
    649.00 ±  5%     +31.9%     856.00 ± 20%  interrupts.CPU35.NMI:Non-maskable_interrupts
    649.00 ±  5%     +31.9%     856.00 ± 20%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
    644.75 ±  7%     +37.4%     886.00 ± 19%  interrupts.CPU38.NMI:Non-maskable_interrupts
    644.75 ±  7%     +37.4%     886.00 ± 19%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
    667.25 ±  6%     +30.5%     870.50 ± 21%  interrupts.CPU39.NMI:Non-maskable_interrupts
    667.25 ±  6%     +30.5%     870.50 ± 21%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
    673.75           +26.3%     851.00 ± 15%  interrupts.CPU4.NMI:Non-maskable_interrupts
    673.75           +26.3%     851.00 ± 15%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
    664.50 ± 10%     +28.7%     855.00 ± 17%  interrupts.CPU41.NMI:Non-maskable_interrupts
    664.50 ± 10%     +28.7%     855.00 ± 17%  interrupts.CPU41.PMI:Performance_monitoring_interrupts
    697.75 ± 16%     +31.4%     917.00 ± 12%  interrupts.CPU42.RES:Rescheduling_interrupts
    775.00 ±  5%     -16.1%     650.50 ±  2%  interrupts.CPU45.RES:Rescheduling_interrupts
    690.75 ±  4%     +34.4%     928.50 ± 20%  interrupts.CPU48.NMI:Non-maskable_interrupts
    690.75 ±  4%     +34.4%     928.50 ± 20%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
    761.50 ±  5%     +27.1%     967.50 ± 20%  interrupts.CPU49.NMI:Non-maskable_interrupts
    761.50 ±  5%     +27.1%     967.50 ± 20%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
    643.50 ±  3%     +32.6%     853.00 ± 22%  interrupts.CPU5.NMI:Non-maskable_interrupts
    643.50 ±  3%     +32.6%     853.00 ± 22%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
    835.75 ± 14%    +310.5%       3430 ± 67%  interrupts.CPU55.RES:Rescheduling_interrupts
    636.75 ±  5%     +34.1%     854.00 ± 19%  interrupts.CPU6.NMI:Non-maskable_interrupts
    636.75 ±  5%     +34.1%     854.00 ± 19%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
    857.75 ± 30%     +45.2%       1245 ± 15%  interrupts.CPU61.NMI:Non-maskable_interrupts
    857.75 ± 30%     +45.2%       1245 ± 15%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
      1333 ± 49%     -46.3%     716.50        interrupts.CPU62.RES:Rescheduling_interrupts
    836.50 ± 32%     +43.5%       1200 ±  8%  interrupts.CPU63.NMI:Non-maskable_interrupts
    836.50 ± 32%     +43.5%       1200 ±  8%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
      1363 ± 39%     -47.1%     721.00 ±  6%  interrupts.CPU63.RES:Rescheduling_interrupts
    842.00 ± 34%     +46.6%       1234 ± 10%  interrupts.CPU65.NMI:Non-maskable_interrupts
    842.00 ± 34%     +46.6%       1234 ± 10%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
    813.75 ± 14%     -22.6%     629.50        interrupts.CPU66.RES:Rescheduling_interrupts
     65.25 ± 71%     -72.4%      18.00 ± 38%  interrupts.CPU66.TLB:TLB_shootdowns
    646.25 ±  4%     +28.7%     831.50 ± 17%  interrupts.CPU7.NMI:Non-maskable_interrupts
    646.25 ±  4%     +28.7%     831.50 ± 17%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
    816.25 ± 13%     -23.6%     623.50 ±  4%  interrupts.CPU72.RES:Rescheduling_interrupts
    963.25 ± 28%     -33.1%     644.00 ±  9%  interrupts.CPU76.RES:Rescheduling_interrupts
     59.50 ± 53%     -65.5%      20.50 ± 31%  interrupts.CPU77.TLB:TLB_shootdowns
    856.00 ± 10%     -24.7%     644.50 ±  9%  interrupts.CPU78.RES:Rescheduling_interrupts
    639.50 ±  5%     +31.4%     840.50 ± 19%  interrupts.CPU8.NMI:Non-maskable_interrupts
    639.50 ±  5%     +31.4%     840.50 ± 19%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
    771.00 ±  4%     +91.1%       1473 ± 43%  interrupts.CPU80.NMI:Non-maskable_interrupts
    771.00 ±  4%     +91.1%       1473 ± 43%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
    767.50 ±  6%     +20.7%     926.50 ± 11%  interrupts.CPU81.NMI:Non-maskable_interrupts
    767.50 ±  6%     +20.7%     926.50 ± 11%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
    779.25 ±  4%     +14.9%     895.00 ± 12%  interrupts.CPU83.NMI:Non-maskable_interrupts
    779.25 ±  4%     +14.9%     895.00 ± 12%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
    114.25 ± 96%     -83.8%      18.50 ±  2%  interrupts.CPU84.TLB:TLB_shootdowns
    915.75 ± 24%     +48.9%       1363 ± 21%  interrupts.CPU87.NMI:Non-maskable_interrupts
    915.75 ± 24%     +48.9%       1363 ± 21%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
    720.50 ± 19%     +17.8%     849.00 ±  4%  interrupts.CPU87.RES:Rescheduling_interrupts
    952.25 ± 30%     +30.7%       1245 ± 22%  interrupts.CPU88.NMI:Non-maskable_interrupts
    952.25 ± 30%     +30.7%       1245 ± 22%  interrupts.CPU88.PMI:Performance_monitoring_interrupts
    947.25 ± 28%     +33.5%       1264 ± 20%  interrupts.CPU89.NMI:Non-maskable_interrupts
    947.25 ± 28%     +33.5%       1264 ± 20%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
    647.75 ±  7%     +28.5%     832.50 ± 19%  interrupts.CPU9.NMI:Non-maskable_interrupts
    647.75 ±  7%     +28.5%     832.50 ± 19%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
    948.50 ± 28%     +37.0%       1299 ± 21%  interrupts.CPU90.NMI:Non-maskable_interrupts
    948.50 ± 28%     +37.0%       1299 ± 21%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
    933.00 ± 31%     +39.4%       1301 ± 22%  interrupts.CPU91.NMI:Non-maskable_interrupts
    933.00 ± 31%     +39.4%       1301 ± 22%  interrupts.CPU91.PMI:Performance_monitoring_interrupts
    914.75 ± 26%    +103.3%       1859 ± 21%  interrupts.CPU92.NMI:Non-maskable_interrupts
    914.75 ± 26%    +103.3%       1859 ± 21%  interrupts.CPU92.PMI:Performance_monitoring_interrupts
    927.00 ± 30%    +105.9%       1908 ± 21%  interrupts.CPU93.NMI:Non-maskable_interrupts
    927.00 ± 30%    +105.9%       1908 ± 21%  interrupts.CPU93.PMI:Performance_monitoring_interrupts
      1009 ± 29%     +91.5%       1933 ± 21%  interrupts.CPU94.NMI:Non-maskable_interrupts
      1009 ± 29%     +91.5%       1933 ± 21%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
      1114 ± 30%     +68.9%       1882 ± 18%  interrupts.CPU95.NMI:Non-maskable_interrupts
      1114 ± 30%     +68.9%       1882 ± 18%  interrupts.CPU95.PMI:Performance_monitoring_interrupts
      1124 ± 31%     +70.6%       1918 ± 16%  interrupts.CPU96.NMI:Non-maskable_interrupts
      1124 ± 31%     +70.6%       1918 ± 16%  interrupts.CPU96.PMI:Performance_monitoring_interrupts
      1133 ± 36%     +66.8%       1889 ± 13%  interrupts.CPU97.NMI:Non-maskable_interrupts
      1133 ± 36%     +66.8%       1889 ± 13%  interrupts.CPU97.PMI:Performance_monitoring_interrupts
      3962 ±138%     -85.8%     561.50 ± 24%  interrupts.CPU99.RES:Rescheduling_interrupts



***************************************************************************************************
lkp-hsw-ep2: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/1600%/debian-x86_64-2019-05-14.cgz/300s/lkp-hsw-ep2/alltests/reaim/0x43

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:2     dmesg.WARNING:at_ip__slab_free/0x
          2:4          -37%           0:2     perf-profile.children.cycles-pp.error_entry
          1:4          -30%           0:2     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
    160.75 ±  7%     +23.3%     198.29 ± 16%  reaim.child_systime
    506927            -1.7%     498446        reaim.max_jobs_per_min
   7931060            -3.2%    7678799        reaim.time.involuntary_context_switches
      3107 ±  5%     +18.1%       3668 ± 13%  reaim.time.system_time
   7217664 ±  6%     +11.2%    8027136        meminfo.DirectMap2M
     16.52 ±  5%      +2.6       19.11 ± 10%  mpstat.cpu.all.sys%
     14534 ±  4%      -6.2%      13628        proc-vmstat.nr_shmem
      2809 ± 11%     +35.4%       3805        slabinfo.eventpoll_pwq.active_objs
      2809 ± 11%     +35.4%       3805        slabinfo.eventpoll_pwq.num_objs
     39477 ±  4%      -9.9%      35575        softirqs.CPU21.NET_RX
     39035 ±  2%      -9.3%      35393        softirqs.CPU29.NET_RX
     41045 ±  7%     -11.1%      36472        softirqs.CPU4.NET_RX
     38754 ±  3%      -9.8%      34966        softirqs.CPU64.NET_RX
 1.498e+10            +1.7%  1.523e+10        perf-stat.i.branch-instructions
      2.23 ±  2%      +0.1        2.34 ±  2%  perf-stat.i.cache-miss-rate%
     36.48            +0.8       37.33        perf-stat.i.node-store-miss-rate%
     35.22            +0.5       35.71        perf-stat.overall.node-store-miss-rate%
  16774806            +2.7%   17222201 ±  2%  perf-stat.overall.path-length
 1.493e+10            +1.7%  1.518e+10        perf-stat.ps.branch-instructions
      1777 ± 19%     +33.1%       2366 ±  5%  sched_debug.cfs_rq:/.load.min
      0.43 ± 16%     +53.8%       0.67        sched_debug.cfs_rq:/.nr_running.min
      0.17 ±  6%     -32.9%       0.12 ± 12%  sched_debug.cfs_rq:/.nr_running.stddev
      3.35 ± 27%     -85.7%       0.48 ± 50%  sched_debug.cfs_rq:/.removed.util_avg.avg
    125.62 ± 10%     -81.3%      23.50 ± 46%  sched_debug.cfs_rq:/.removed.util_avg.max
     18.92 ± 19%     -84.7%       2.89 ± 46%  sched_debug.cfs_rq:/.removed.util_avg.stddev
      0.58 ± 46%     +59.4%       0.92 ±  9%  sched_debug.cfs_rq:/.runnable_load_avg.min
      1088 ± 21%     +50.8%       1641 ± 11%  sched_debug.cfs_rq:/.runnable_weight.min
    -92558          -305.9%     190556 ±  5%  sched_debug.cfs_rq:/.spread0.avg
    511049 ± 44%     +69.5%     866184 ± 19%  sched_debug.cfs_rq:/.spread0.max
    600188 ±  6%     +13.4%     680581 ±  6%  sched_debug.cpu.avg_idle.avg
     28.76 ± 20%     +29.4%      37.21 ± 12%  sched_debug.cpu.clock.stddev
     28.76 ± 20%     +29.4%      37.21 ± 12%  sched_debug.cpu.clock_task.stddev
      2705 ±100%    +314.4%      11209 ± 39%  sched_debug.cpu.curr->pid.min
      9054 ±  9%     -21.8%       7084 ±  8%  sched_debug.cpu.curr->pid.stddev
     19617 ± 14%     -25.3%      14660 ± 13%  sched_debug.cpu.load.avg
    534995 ± 35%     -62.1%     202918 ± 77%  sched_debug.cpu.load.max
      1777 ± 19%     +33.1%       2366 ±  5%  sched_debug.cpu.load.min
     65254 ± 31%     -59.2%      26605 ± 73%  sched_debug.cpu.load.stddev
      0.43 ± 16%     +53.8%       0.67        sched_debug.cpu.nr_running.min
    278.68 ±  3%     +13.9%     317.50        sched_debug.cpu.nr_uninterruptible.max
      3.18 ± 18%      -1.6        1.56 ± 20%  perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit
      1.92 ± 49%      -1.3        0.58 ± 11%  perf-profile.calltrace.cycles-pp.down_write.unlink_file_vma.free_pgtables.exit_mmap.mmput
      2.04 ± 34%      -1.3        0.76 ±  3%  perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.mmput.do_exit
      5.01 ± 21%      -2.5        2.53 ±  6%  perf-profile.children.cycles-pp.down_write
      4.24 ± 28%      -2.3        1.96 ±  5%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      4.70 ± 14%      -2.2        2.46 ± 23%  perf-profile.children.cycles-pp.free_pgtables
      3.58 ± 34%      -2.0        1.55 ± 12%  perf-profile.children.cycles-pp.osq_lock
      2.96 ± 27%      -1.8        1.20 ±  4%  perf-profile.children.cycles-pp.unlink_file_vma
      0.78 ± 12%      -0.3        0.45 ± 25%  perf-profile.children.cycles-pp.vma_link
      0.18 ± 13%      -0.1        0.11 ± 27%  perf-profile.children.cycles-pp.rwsem_wake
      0.17 ± 15%      -0.1        0.10 ± 15%  perf-profile.children.cycles-pp.queued_read_lock_slowpath
      0.08 ± 23%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__alloc_fd
      0.07 ± 11%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.sched_clock
      0.14 ±  8%      -0.0        0.09 ± 22%  perf-profile.children.cycles-pp.wake_up_q
      0.09 ± 14%      -0.0        0.05        perf-profile.children.cycles-pp.sched_clock_cpu
      0.01 ±173%      +0.1        0.08 ± 41%  perf-profile.children.cycles-pp.memcpy_toio
      0.01 ±173%      +0.1        0.08 ± 41%  perf-profile.children.cycles-pp.fb_flashcursor
      0.01 ±173%      +0.1        0.08 ± 41%  perf-profile.children.cycles-pp.bit_cursor
      0.01 ±173%      +0.1        0.08 ± 41%  perf-profile.children.cycles-pp.soft_cursor
      0.01 ±173%      +0.1        0.08 ± 41%  perf-profile.children.cycles-pp.mga_dirty_update
      0.04 ±173%      +0.2        0.28 ±  7%  perf-profile.children.cycles-pp.start_kernel
      3.54 ± 34%      -2.0        1.54 ± 12%  perf-profile.self.cycles-pp.osq_lock
      0.27 ± 16%      -0.1        0.16 ± 31%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.07 ±  5%      -0.0        0.06 ±  9%  perf-profile.self.cycles-pp.vfprintf
      0.01 ±173%      +0.1        0.08 ± 41%  perf-profile.self.cycles-pp.memcpy_toio
    178.25 ± 14%     +21.7%     217.00 ± 15%  interrupts.49:PCI-MSI.1572878-edge.eth0-TxRx-14
    151.50            +7.9%     163.50 ±  2%  interrupts.81:PCI-MSI.1572910-edge.eth0-TxRx-46
    824.75 ±162%    +603.2%       5799 ±  3%  interrupts.CPU0.NMI:Non-maskable_interrupts
    824.75 ±162%    +603.2%       5799 ±  3%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
     38973 ±  4%      -6.1%      36586        interrupts.CPU10.RES:Rescheduling_interrupts
    178.25 ± 14%     +21.7%     217.00 ± 15%  interrupts.CPU14.49:PCI-MSI.1572878-edge.eth0-TxRx-14
      4727 ± 30%     -70.4%       1397 ± 99%  interrupts.CPU17.NMI:Non-maskable_interrupts
      4727 ± 30%     -70.4%       1397 ± 99%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
    163.50 ± 14%     +28.7%     210.50 ±  6%  interrupts.CPU21.TLB:TLB_shootdowns
      1151 ±100%    +276.3%       4331 ± 28%  interrupts.CPU31.NMI:Non-maskable_interrupts
      1151 ±100%    +276.3%       4331 ± 28%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
      4371 ± 25%     -99.3%      29.00 ± 96%  interrupts.CPU32.NMI:Non-maskable_interrupts
      4371 ± 25%     -99.3%      29.00 ± 96%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
      2828 ±  5%      +9.9%       3110 ±  2%  interrupts.CPU33.CAL:Function_call_interrupts
    173.00 ±  8%     -22.5%     134.00 ±  2%  interrupts.CPU33.TLB:TLB_shootdowns
      5807 ± 16%     -76.3%       1378 ± 99%  interrupts.CPU38.NMI:Non-maskable_interrupts
      5807 ± 16%     -76.3%       1378 ± 99%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
    206.25 ±  8%     -19.5%     166.00 ±  7%  interrupts.CPU40.TLB:TLB_shootdowns
    151.50            +7.9%     163.50 ±  2%  interrupts.CPU46.81:PCI-MSI.1572910-edge.eth0-TxRx-46
    166.50 ±  5%     +18.3%     197.00 ±  6%  interrupts.CPU5.TLB:TLB_shootdowns
      5546 ± 25%     -47.7%       2902 ±  4%  interrupts.CPU50.NMI:Non-maskable_interrupts
      5546 ± 25%     -47.7%       2902 ±  4%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
      2270 ± 53%    -100.0%       0.00        interrupts.CPU56.NMI:Non-maskable_interrupts
      2270 ± 53%    -100.0%       0.00        interrupts.CPU56.PMI:Performance_monitoring_interrupts
    142.00 ± 13%     +33.8%     190.00 ± 11%  interrupts.CPU56.TLB:TLB_shootdowns
     38592 ±  2%      +5.0%      40532 ±  3%  interrupts.CPU58.RES:Rescheduling_interrupts
    212.75 ± 26%     -31.6%     145.50 ±  7%  interrupts.CPU59.TLB:TLB_shootdowns
      2086 ± 60%    +180.0%       5841 ±  4%  interrupts.CPU60.NMI:Non-maskable_interrupts
      2086 ± 60%    +180.0%       5841 ±  4%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
     26.25 ±128%   +6105.7%       1629 ± 89%  interrupts.CPU61.NMI:Non-maskable_interrupts
     26.25 ±128%   +6105.7%       1629 ± 89%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
      4792 ± 35%     -39.1%       2918 ±  5%  interrupts.CPU63.NMI:Non-maskable_interrupts
      4792 ± 35%     -39.1%       2918 ±  5%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
    168.00 ± 16%     -23.8%     128.00 ±  5%  interrupts.CPU66.TLB:TLB_shootdowns
      3936 ± 32%     -83.3%     655.50 ±100%  interrupts.CPU67.NMI:Non-maskable_interrupts
      3936 ± 32%     -83.3%     655.50 ±100%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
      2560 ± 85%    +124.6%       5751 ±  5%  interrupts.CPU68.NMI:Non-maskable_interrupts
      2560 ± 85%    +124.6%       5751 ±  5%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
      3693 ± 21%     +58.6%       5857 ±  4%  interrupts.CPU70.NMI:Non-maskable_interrupts
      3693 ± 21%     +58.6%       5857 ±  4%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
    112.00 ±  7%      -8.9%     102.00        interrupts.IWI:IRQ_work_interrupts



***************************************************************************************************
lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/900t/debian-x86_64-2019-05-14.cgz/300s/lkp-ivb-2ep1/disk_cp/reaim/0x42e

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -15%           0:2     perf-profile.children.cycles-pp.error_entry
          0:4          -11%           0:2     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
      1116            -2.7%       1087        reaim.child_systime
    173.74            +1.0%     175.47        reaim.child_utime
      2468            -1.7%       2425        reaim.time.percent_of_cpu_this_job_got
      6702            -2.7%       6524        reaim.time.system_time
      1042            +1.0%       1052        reaim.time.user_time
  17750445            +1.1%   17946505        reaim.time.voluntary_context_switches
      3128 ± 16%     +25.7%       3930 ±  2%  numa-vmstat.node1.nr_mapped
    108987 ±  3%     +29.1%     140714 ± 21%  softirqs.CPU28.TIMER
     47.75            +2.6%      49.00        vmstat.cpu.id
 8.151e+08 ± 27%     +32.2%  1.078e+09        cpuidle.C1E.time
   5709370 ± 25%     +25.1%    7144757        cpuidle.C1E.usage
      1294 ±  5%     +13.7%       1471 ±  8%  slabinfo.pool_workqueue.active_objs
      1308 ±  4%     +12.5%       1472 ±  8%  slabinfo.pool_workqueue.num_objs
      1586            -1.8%       1558        turbostat.Avg_MHz
   5709242 ± 25%     +25.1%    7144385        turbostat.C1E
      5.38 ± 27%      +1.8        7.13        turbostat.C1E%
      0.01 ±110%    +233.3%       0.03 ± 19%  turbostat.Pkg%pc3
      5342 ±173%    +595.0%      37130 ± 28%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    256451 ±173%    +595.0%    1782246 ± 28%  sched_debug.cfs_rq:/.MIN_vruntime.max
     36628 ±173%    +595.0%     254551 ± 28%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
    477.71 ± 29%    +214.6%       1503 ± 57%  sched_debug.cfs_rq:/.load_avg.max
     89.14 ± 27%    +160.7%     232.44 ± 54%  sched_debug.cfs_rq:/.load_avg.stddev
      5342 ±173%    +595.0%      37130 ± 28%  sched_debug.cfs_rq:/.max_vruntime.avg
    256451 ±173%    +595.0%    1782246 ± 28%  sched_debug.cfs_rq:/.max_vruntime.max
     36628 ±173%    +595.0%     254551 ± 28%  sched_debug.cfs_rq:/.max_vruntime.stddev
      1.00           +16.7%       1.17        sched_debug.cfs_rq:/.nr_running.max
     33.74 ±  6%     -16.1%      28.32 ±  2%  sched_debug.cfs_rq:/.util_est_enqueued.avg
     11.40            +4.7%      11.93        perf-stat.i.MPKI
 7.907e+09            -2.3%  7.726e+09        perf-stat.i.branch-instructions
      1.74            +0.1        1.83        perf-stat.i.branch-miss-rate%
 4.285e+08            +1.2%  4.335e+08        perf-stat.i.cache-references
    139218            +1.1%     140680        perf-stat.i.context-switches
 7.603e+10            -1.9%  7.457e+10        perf-stat.i.cpu-cycles
    925.05            -2.7%     900.07        perf-stat.i.cycles-between-cache-misses
 1.161e+10            -2.5%  1.132e+10        perf-stat.i.dTLB-loads
  10864114            -1.1%   10743610        perf-stat.i.iTLB-load-misses
   1201431            +4.7%    1258341        perf-stat.i.iTLB-loads
 3.926e+10            -2.4%  3.832e+10        perf-stat.i.instructions
     10.91            +3.6%      11.31        perf-stat.overall.MPKI
      1.63            +0.1        1.68        perf-stat.overall.branch-miss-rate%
    921.98            -2.6%     898.33        perf-stat.overall.cycles-between-cache-misses
  22791180            -2.5%   22226281        perf-stat.overall.path-length
 7.882e+09            -2.3%  7.701e+09        perf-stat.ps.branch-instructions
 4.271e+08            +1.2%  4.321e+08        perf-stat.ps.cache-references
    138768            +1.1%     140229        perf-stat.ps.context-switches
 7.578e+10            -1.9%  7.434e+10        perf-stat.ps.cpu-cycles
 1.157e+10            -2.5%  1.129e+10        perf-stat.ps.dTLB-loads
  10829073            -1.1%   10709146        perf-stat.ps.iTLB-load-misses
   1197556            +4.7%    1254304        perf-stat.ps.iTLB-loads
 3.914e+10            -2.4%   3.82e+10        perf-stat.ps.instructions
 1.231e+13            -2.5%    1.2e+13        perf-stat.total.instructions
    925085 ±  3%      +5.5%     976146        interrupts.CAL:Function_call_interrupts
     11296 ±  4%      +9.2%      12340 ±  3%  interrupts.CPU0.RES:Rescheduling_interrupts
      3692 ± 32%    -100.0%       0.50 ±100%  interrupts.CPU10.NMI:Non-maskable_interrupts
      3692 ± 32%    -100.0%       0.50 ±100%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      3699 ± 32%     -96.4%     132.00 ± 64%  interrupts.CPU20.NMI:Non-maskable_interrupts
      3699 ± 32%     -96.4%     132.00 ± 64%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
    635.25 ±163%     -99.9%       0.50 ±100%  interrupts.CPU22.NMI:Non-maskable_interrupts
    635.25 ±163%     -99.9%       0.50 ±100%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
     19144 ±  2%      +9.9%      21036        interrupts.CPU23.CAL:Function_call_interrupts
    706.75 ±143%    +420.9%       3681 ± 31%  interrupts.CPU23.NMI:Non-maskable_interrupts
    706.75 ±143%    +420.9%       3681 ± 31%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
     16166           +10.2%      17809        interrupts.CPU23.TLB:TLB_shootdowns
     16484 ±  5%      +5.7%      17432 ±  4%  interrupts.CPU25.TLB:TLB_shootdowns
    654.75 ±157%    +464.7%       3697 ± 30%  interrupts.CPU27.NMI:Non-maskable_interrupts
    654.75 ±157%    +464.7%       3697 ± 30%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
     15719 ±  4%      +7.8%      16941 ±  5%  interrupts.CPU29.TLB:TLB_shootdowns
    191.75 ±152%   +1195.2%       2483 ±  4%  interrupts.CPU30.NMI:Non-maskable_interrupts
    191.75 ±152%   +1195.2%       2483 ±  4%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
      2.50 ± 82%  +1.5e+05%       3665 ± 30%  interrupts.CPU34.NMI:Non-maskable_interrupts
      2.50 ± 82%  +1.5e+05%       3665 ± 30%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
     10577 ±  2%      +5.7%      11181 ±  5%  interrupts.CPU34.RES:Rescheduling_interrupts
    671.50 ±151%    +449.5%       3690 ± 30%  interrupts.CPU44.NMI:Non-maskable_interrupts
    671.50 ±151%    +449.5%       3690 ± 30%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
     10379 ±  3%      +8.5%      11258 ±  3%  interrupts.CPU44.RES:Rescheduling_interrupts
     16081 ±  2%     +11.2%      17882 ±  2%  interrupts.CPU47.TLB:TLB_shootdowns
     18858 ±  5%      +7.0%      20175 ±  4%  interrupts.CPU5.CAL:Function_call_interrupts
     15595 ±  6%      +7.9%      16826 ±  5%  interrupts.CPU5.TLB:TLB_shootdowns
     20.85            -1.1       19.71 ±  2%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.unlink_file_vma.free_pgtables
     14.50            -0.9       13.56 ±  2%  perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.__do_execve_file.__x64_sys_execve.do_syscall_64
     14.52            -0.9       13.59 ±  2%  perf-profile.calltrace.cycles-pp.search_binary_handler.__do_execve_file.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
     12.03            -0.8       11.20        perf-profile.calltrace.cycles-pp.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler
     12.13            -0.8       11.31        perf-profile.calltrace.cycles-pp.flush_old_exec.load_elf_binary.search_binary_handler.__do_execve_file.__x64_sys_execve
     12.04            -0.8       11.22        perf-profile.calltrace.cycles-pp.mmput.flush_old_exec.load_elf_binary.search_binary_handler.__do_execve_file
     10.75            -0.7       10.01        perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary
      9.92            -0.7        9.21        perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.mmput.flush_old_exec
     24.45            -1.2       23.26 ±  3%  perf-profile.children.cycles-pp.free_pgtables
     22.85            -1.1       21.73 ±  2%  perf-profile.children.cycles-pp.unlink_file_vma
     15.96            -1.0       15.00 ±  2%  perf-profile.children.cycles-pp.__do_execve_file
     16.03            -0.9       15.09 ±  2%  perf-profile.children.cycles-pp.__x64_sys_execve
     14.52            -0.9       13.58 ±  2%  perf-profile.children.cycles-pp.search_binary_handler
     14.50            -0.9       13.56 ±  2%  perf-profile.children.cycles-pp.load_elf_binary
     12.14            -0.8       11.31        perf-profile.children.cycles-pp.flush_old_exec
      0.30 ±  2%      -0.0        0.27 ±  3%  perf-profile.children.cycles-pp.perf_iterate_sb
      0.17 ±  5%      -0.0        0.15 ±  3%  perf-profile.children.cycles-pp.unwind_next_frame
      0.10            -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.14 ±  6%      -0.0        0.11 ±  4%  perf-profile.children.cycles-pp.perf_event_mmap_output
      0.08 ± 10%      -0.0        0.06        perf-profile.children.cycles-pp.move_queued_task
      0.05 ±  9%      +0.0        0.07        perf-profile.children.cycles-pp.__x64_sys_close
      0.06 ±  7%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.fput_many
      0.04 ± 57%      +0.0        0.06        perf-profile.children.cycles-pp.lapic_next_deadline
      0.54 ±  4%      +0.0        0.56 ±  2%  perf-profile.children.cycles-pp.alloc_set_pte
      0.15 ±  8%      +0.0        0.18 ±  5%  perf-profile.children.cycles-pp.__d_lookup_rcu
      0.20 ± 10%      +0.0        0.23 ±  2%  perf-profile.children.cycles-pp.wait_consider_task
      0.10 ± 21%      +0.0        0.14 ± 14%  perf-profile.children.cycles-pp.ksys_write
      0.09 ± 23%      +0.0        0.13 ± 15%  perf-profile.children.cycles-pp.new_sync_write
      0.06 ± 68%      +0.0        0.11 ± 23%  perf-profile.children.cycles-pp.__waitpid
      0.24 ±  6%      +0.0        0.28 ±  3%  perf-profile.children.cycles-pp.lookup_fast
      0.57 ±  6%      +0.0        0.61 ±  3%  perf-profile.children.cycles-pp.anon_vma_fork
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.cp_new_stat
      0.10 ±  4%      +0.1        0.16 ± 25%  perf-profile.children.cycles-pp.ktime_get
      0.10 ±  5%      +0.1        0.16 ± 31%  perf-profile.children.cycles-pp.clockevents_program_event
      0.68 ±  4%      +0.1        0.74 ±  2%  perf-profile.children.cycles-pp.schedule
      1.11 ±  6%      +0.1        1.18 ±  2%  perf-profile.children.cycles-pp.__schedule
      0.32 ±  7%      +0.1        0.39 ± 12%  perf-profile.children.cycles-pp.__slab_free
      0.09 ± 13%      -0.0        0.07        perf-profile.self.cycles-pp.cpuidle_enter_state
      0.07 ±  6%      +0.0        0.08        perf-profile.self.cycles-pp.___slab_alloc
      0.09 ±  4%      +0.0        0.10        perf-profile.self.cycles-pp.avc_has_perm_noaudit
      0.17 ±  4%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.alloc_set_pte
      0.05            +0.0        0.07 ±  7%  perf-profile.self.cycles-pp._vm_normal_page
      0.21 ±  6%      +0.0        0.24 ±  4%  perf-profile.self.cycles-pp.anon_vma_interval_tree_insert
      0.34 ±  6%      +0.0        0.38        perf-profile.self.cycles-pp.__handle_mm_fault
      0.15 ±  7%      +0.0        0.18 ±  5%  perf-profile.self.cycles-pp.__d_lookup_rcu
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.path_openat
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.enqueue_task_fair



***************************************************************************************************
lkp-hsw-ep2: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/1600%/debian-x86_64-2019-05-14.cgz/300s/lkp-hsw-ep2/new_dbase/reaim/0x43

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :2     dmesg.WARNING:at_ip_perf_event_mmap_output/0x
          1:4          -17%           0:2     perf-profile.children.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
  10584147            +1.9%   10785891        perf-stat.i.node-load-misses
  10542342            +1.9%   10743011        perf-stat.ps.node-load-misses
      1393 ±  7%     -19.5%       1122 ±  2%  slabinfo.task_group.active_objs
      1393 ±  7%     -19.5%       1122 ±  2%  slabinfo.task_group.num_objs
     34065            +0.9%      34375        proc-vmstat.nr_kernel_stack
     13974            +2.7%      14349        proc-vmstat.nr_page_table_pages
      3127 ± 92%     +91.2%       5980 ± 34%  proc-vmstat.numa_hint_faults
    122943 ± 12%     +12.9%     138853 ± 12%  numa-vmstat.node0.nr_active_anon
     13963 ± 48%     +34.6%      18801 ± 41%  numa-vmstat.node0.nr_kernel_stack
      4857 ± 76%     +56.6%       7609 ± 57%  numa-vmstat.node0.nr_page_table_pages
    122943 ± 12%     +12.9%     138852 ± 12%  numa-vmstat.node0.nr_zone_active_anon
    494560 ± 12%     +12.6%     557012 ± 12%  numa-meminfo.node0.Active
    494224 ± 12%     +12.6%     556676 ± 12%  numa-meminfo.node0.Active(anon)
    474941 ± 14%     +13.0%     536795 ± 14%  numa-meminfo.node0.AnonPages
     13949 ± 49%     +32.9%      18544 ± 40%  numa-meminfo.node0.KernelStack
   1479782 ±  9%      +9.4%    1618507 ±  8%  numa-meminfo.node0.MemUsed
     19602 ± 76%     +52.1%      29814 ± 56%  numa-meminfo.node0.PageTables
    117010           +24.4%     145600 ± 14%  softirqs.CPU10.TIMER
     13137 ±  2%     -12.0%      11564        softirqs.CPU18.SCHED
     82501 ±  5%      +8.2%      89260 ±  8%  softirqs.CPU29.RCU
    117180           +12.4%     131653 ±  5%  softirqs.CPU35.TIMER
     12029 ±  5%      -6.1%      11294        softirqs.CPU37.SCHED
    117019           +10.1%     128814 ±  5%  softirqs.CPU46.TIMER
     74749 ±  6%      +9.2%      81658 ±  6%  softirqs.CPU60.RCU
     72124 ±  6%     +10.9%      80008 ±  9%  softirqs.CPU65.RCU
    117036 ±  2%     +20.5%     141060 ± 13%  softirqs.CPU71.TIMER
      0.10 ± 16%     +37.6%       0.14        sched_debug.cfs_rq:/.nr_running.stddev
     11.66 ±  6%     -17.7%       9.60 ±  8%  sched_debug.cfs_rq:/.runnable_load_avg.avg
     90.96 ± 19%     +18.5%     107.82 ±  9%  sched_debug.cpu.clock.stddev
     90.96 ± 19%     +18.5%     107.82 ±  9%  sched_debug.cpu.clock_task.stddev
     16220           -13.9%      13969        sched_debug.cpu.nr_switches.stddev
    321.04 ±  4%     +39.0%     446.25 ± 13%  sched_debug.cpu.nr_uninterruptible.max
    142.97           +20.7%     172.63 ±  8%  sched_debug.cpu.nr_uninterruptible.stddev
     17028 ±  2%      -9.8%      15355        sched_debug.cpu.sched_count.stddev
     21177 ±  3%     +12.2%      23770        sched_debug.cpu.ttwu_count.min
     10062 ±  2%     -11.7%       8882        sched_debug.cpu.ttwu_count.stddev
     12442           +12.0%      13935        sched_debug.cpu.ttwu_local.min
      5892 ±  3%     -12.1%       5178        sched_debug.cpu.ttwu_local.stddev
    169.25 ±  8%    +344.9%     753.00 ± 75%  interrupts.38:PCI-MSI.1572867-edge.eth0-TxRx-3
    491.50 ± 72%     -64.8%     173.00        interrupts.39:PCI-MSI.1572868-edge.eth0-TxRx-4
    299.25 ± 58%     -42.5%     172.00        interrupts.45:PCI-MSI.1572874-edge.eth0-TxRx-10
    178.50 ±  7%     +97.8%     353.00 ±  9%  interrupts.48:PCI-MSI.1572877-edge.eth0-TxRx-13
    210.75 ± 27%     +72.0%     362.50 ± 31%  interrupts.50:PCI-MSI.1572879-edge.eth0-TxRx-15
    150.00            +7.3%     161.00 ±  6%  interrupts.73:PCI-MSI.1572902-edge.eth0-TxRx-38
    150.00            +6.7%     160.00 ±  5%  interrupts.96:PCI-MSI.1572925-edge.eth0-TxRx-61
    299.25 ± 58%     -42.5%     172.00        interrupts.CPU10.45:PCI-MSI.1572874-edge.eth0-TxRx-10
    178.50 ±  7%     +97.8%     353.00 ±  9%  interrupts.CPU13.48:PCI-MSI.1572877-edge.eth0-TxRx-13
      2731 ±  8%     +11.9%       3055 ±  2%  interrupts.CPU13.CAL:Function_call_interrupts
    210.75 ± 27%     +72.0%     362.50 ± 31%  interrupts.CPU15.50:PCI-MSI.1572879-edge.eth0-TxRx-15
      2401 ±100%    -100.0%       1.00 ±100%  interrupts.CPU15.NMI:Non-maskable_interrupts
      2401 ±100%    -100.0%       1.00 ±100%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
    138.50 ± 25%     +26.4%     175.00 ±  6%  interrupts.CPU17.TLB:TLB_shootdowns
      4544 ± 57%    -100.0%       0.00        interrupts.CPU26.NMI:Non-maskable_interrupts
      4544 ± 57%    -100.0%       0.00        interrupts.CPU26.PMI:Performance_monitoring_interrupts
    169.25 ±  8%    +344.9%     753.00 ± 75%  interrupts.CPU3.38:PCI-MSI.1572867-edge.eth0-TxRx-3
    825.50 ±152%    +450.6%       4545 ± 32%  interrupts.CPU36.NMI:Non-maskable_interrupts
    825.50 ±152%    +450.6%       4545 ± 32%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
    150.00            +7.3%     161.00 ±  6%  interrupts.CPU38.73:PCI-MSI.1572902-edge.eth0-TxRx-38
    491.50 ± 72%     -64.8%     173.00        interrupts.CPU4.39:PCI-MSI.1572868-edge.eth0-TxRx-4
    176.50 ± 11%     -24.9%     132.50 ± 16%  interrupts.CPU4.TLB:TLB_shootdowns
      4580 ± 33%     -66.0%       1558 ± 95%  interrupts.CPU44.NMI:Non-maskable_interrupts
      4580 ± 33%     -66.0%       1558 ± 95%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      2732 ± 80%     -95.9%     111.50 ±  9%  interrupts.CPU47.NMI:Non-maskable_interrupts
      2732 ± 80%     -95.9%     111.50 ±  9%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
    144.75 ±  5%     +10.5%     160.00 ±  4%  interrupts.CPU56.TLB:TLB_shootdowns
      1725 ±145%    +188.6%       4981 ± 22%  interrupts.CPU58.NMI:Non-maskable_interrupts
      1725 ±145%    +188.6%       4981 ± 22%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
    145.25 ± 16%     +30.8%     190.00        interrupts.CPU58.TLB:TLB_shootdowns
      1515 ±173%    +201.8%       4573 ± 33%  interrupts.CPU6.NMI:Non-maskable_interrupts
      1515 ±173%    +201.8%       4573 ± 33%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
    150.00            +6.7%     160.00 ±  5%  interrupts.CPU61.96:PCI-MSI.1572925-edge.eth0-TxRx-61
      2881 ±  4%      +9.1%       3143        interrupts.CPU62.CAL:Function_call_interrupts
      1622 ±159%    +183.6%       4602 ± 33%  interrupts.CPU62.NMI:Non-maskable_interrupts
      1622 ±159%    +183.6%       4602 ± 33%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
      2449 ± 97%    +150.0%       6124        interrupts.CPU68.NMI:Non-maskable_interrupts
      2449 ± 97%    +150.0%       6124        interrupts.CPU68.PMI:Performance_monitoring_interrupts
      2846 ±  5%     +10.1%       3133        interrupts.CPU7.CAL:Function_call_interrupts
      3055 ± 70%     +50.6%       4601 ± 32%  interrupts.CPU7.NMI:Non-maskable_interrupts
      3055 ± 70%     +50.6%       4601 ± 32%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
     33.94 ±  6%      -3.0       30.98        perf-profile.calltrace.cycles-pp.sieve
      9.37 ±  6%      -0.9        8.44        perf-profile.calltrace.cycles-pp.string_rtns_1
      3.44 ±  6%      -0.3        3.10        perf-profile.calltrace.cycles-pp.brk
      2.53 ±  8%      -0.3        2.26 ±  3%  perf-profile.calltrace.cycles-pp.page_test
      2.72 ±  6%      -0.3        2.46        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.brk
      2.71 ±  6%      -0.3        2.45        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
      2.25 ±  6%      -0.2        2.04        perf-profile.calltrace.cycles-pp.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
      1.55 ±  7%      -0.2        1.38 ±  3%  perf-profile.calltrace.cycles-pp.page_fault.page_test
      1.55 ±  7%      -0.2        1.38 ±  3%  perf-profile.calltrace.cycles-pp.do_page_fault.page_fault.page_test
      1.48 ±  7%      -0.2        1.31 ±  4%  perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault.page_test
      1.25 ±  7%      -0.1        1.10 ±  4%  perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault.page_test
      0.89 ±  8%      -0.1        0.79        perf-profile.calltrace.cycles-pp.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
      1.16 ±  5%      -0.1        1.07        perf-profile.calltrace.cycles-pp.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
      0.77 ±  6%      -0.1        0.69        perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__x64_sys_brk
      0.78 ±  6%      -0.1        0.70        perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64
      0.41 ± 57%      +0.2        0.59 ±  2%  perf-profile.calltrace.cycles-pp.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode
      0.55 ± 57%      +0.3        0.82 ±  2%  perf-profile.calltrace.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      3.17 ±  8%      +0.3        3.45 ±  2%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      0.85 ± 21%      +0.3        1.14 ±  9%  perf-profile.calltrace.cycles-pp.__libc_fork
      1.31 ± 18%      +0.3        1.60 ±  4%  perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.59 ± 59%      +0.4        0.97 ± 11%  perf-profile.calltrace.cycles-pp.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      0.63 ± 59%      +0.4        1.01 ± 11%  perf-profile.calltrace.cycles-pp._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      0.30 ±100%      +0.4        0.69 ± 10%  perf-profile.calltrace.cycles-pp.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.63 ± 59%      +0.4        1.02 ± 11%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_fork
      0.63 ± 59%      +0.4        1.02 ± 11%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      0.13 ±173%      +0.4        0.53 ±  4%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page.__handle_mm_fault
      2.11 ± 17%      +0.4        2.52        perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      2.44 ± 17%      +0.5        2.91 ±  2%  perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
      2.58 ± 16%      +0.5        3.05 ±  2%  perf-profile.calltrace.cycles-pp.page_fault
      2.53 ± 17%      +0.5        3.00 ±  2%  perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
      1.58 ± 25%      +0.5        2.09 ± 12%  perf-profile.calltrace.cycles-pp.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.66 ± 24%      +0.5        2.17 ± 12%  perf-profile.calltrace.cycles-pp._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.33 ±105%      +0.5        0.86 ±  2%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.vma_link.mmap_region.do_mmap
      0.33 ±105%      +0.5        0.87 ±  2%  perf-profile.calltrace.cycles-pp.down_write.vma_link.mmap_region.do_mmap.vm_mmap_pgoff
      1.23 ± 39%      +0.6        1.79 ± 15%  perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.mmput.do_exit
      0.35 ±104%      +0.6        0.93 ±  2%  perf-profile.calltrace.cycles-pp.vma_link.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
      1.33 ± 37%      +0.6        1.90 ± 14%  perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit
      0.77 ± 62%      +0.6        1.38        perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      0.82 ± 62%      +0.6        1.43 ±  2%  perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.79 ± 62%      +0.6        1.41 ±  2%  perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.6        0.61 ±  2%  perf-profile.calltrace.cycles-pp.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.81 ± 62%      +0.6        1.43 ±  2%  perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.81 ± 60%      +0.6        1.44 ±  6%  perf-profile.calltrace.cycles-pp.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.20 ±173%      +0.6        0.82        perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.vma_link.mmap_region
      1.14 ± 58%      +0.7        1.79        perf-profile.calltrace.cycles-pp.__do_execve_file.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.15 ± 58%      +0.7        1.80        perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.12 ± 27%      +0.7        2.83 ±  8%  perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      2.12 ± 27%      +0.7        2.83 ±  8%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
      3.23 ± 29%      +0.7        3.94 ±  5%  perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.__do_execve_file.__x64_sys_execve.do_syscall_64
      0.90 ± 74%      +0.7        1.62 ± 37%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.dup_mm.copy_process
      3.24 ± 29%      +0.7        3.96 ±  5%  perf-profile.calltrace.cycles-pp.search_binary_handler.__do_execve_file.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.38 ± 25%      +0.7        3.12 ±  8%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.38 ± 25%      +0.7        3.12 ±  7%  perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.38 ± 25%      +0.7        3.12 ±  7%  perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.92 ± 74%      +0.8        1.68 ± 36%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.dup_mm.copy_process._do_fork
      0.93 ± 74%      +0.8        1.69 ± 36%  perf-profile.calltrace.cycles-pp.down_write.dup_mm.copy_process._do_fork.do_syscall_64
      3.83 ±  7%      +0.8        4.60 ±  3%  perf-profile.calltrace.cycles-pp.secondary_startup_64
      3.02 ± 11%      +0.8        3.79 ±  3%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      3.50 ±  9%      +0.8        4.32 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
      3.53 ±  9%      +0.8        4.36 ±  2%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      3.76 ±  8%      +0.8        4.60 ±  2%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      3.77 ±  8%      +0.8        4.60 ±  3%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      3.77 ±  8%      +0.8        4.60 ±  3%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      1.91 ± 35%      +0.8        2.75 ± 13%  perf-profile.calltrace.cycles-pp.dup_mm.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.28 ± 60%      +1.2        3.53 ± 13%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.unlink_file_vma.free_pgtables
      2.39 ± 60%      +1.3        3.67 ± 13%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.unlink_file_vma.free_pgtables.exit_mmap
      2.42 ± 60%      +1.3        3.71 ± 13%  perf-profile.calltrace.cycles-pp.down_write.unlink_file_vma.free_pgtables.exit_mmap.mmput
      8.91 ± 28%      +3.3       12.18 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.93 ± 28%      +3.3       12.20 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     33.97 ±  6%      -3.0       31.00        perf-profile.children.cycles-pp.sieve
      9.44 ±  6%      -0.9        8.49        perf-profile.children.cycles-pp.string_rtns_1
      3.47 ±  6%      -0.3        3.12        perf-profile.children.cycles-pp.brk
      2.87 ±  8%      -0.3        2.57 ±  2%  perf-profile.children.cycles-pp.page_test
      0.16 ±  8%      -0.0        0.13        perf-profile.children.cycles-pp.__libc_calloc
      0.17 ±  2%      -0.0        0.16        perf-profile.children.cycles-pp.free
      0.07 ±  6%      +0.0        0.08        perf-profile.children.cycles-pp.wait_consider_task
      0.05 ±  8%      +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.06 ±  7%      +0.0        0.07        perf-profile.children.cycles-pp.finish_task_switch
      0.05            +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.up_read
      0.10 ± 11%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.__slab_free
      0.09 ±  9%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.__x64_sys_munmap
      0.08 ± 14%      +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.enqueue_entity
      0.05 ±  8%      +0.0        0.07        perf-profile.children.cycles-pp.do_dentry_open
      0.12 ±  9%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.vmacache_find
      0.19 ±  5%      +0.0        0.21        perf-profile.children.cycles-pp.exit_to_usermode_loop
      0.08 ± 11%      +0.0        0.10 ±  5%  perf-profile.children.cycles-pp.load_balance
      0.07 ± 13%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.06 ± 11%      +0.0        0.08 ± 12%  perf-profile.children.cycles-pp.mem_cgroup_throttle_swaprate
      0.32 ±  4%      +0.0        0.34        perf-profile.children.cycles-pp.copy_p4d_range
      0.09 ± 12%      +0.0        0.11 ±  9%  perf-profile.children.cycles-pp.wake_up_q
      0.06 ±  7%      +0.0        0.08 ± 12%  perf-profile.children.cycles-pp.__tlb_remove_page_size
      0.11 ± 13%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.wait4
      0.21 ±  5%      +0.0        0.24 ±  4%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.08 ± 10%      +0.0        0.11        perf-profile.children.cycles-pp.activate_task
      0.04 ± 58%      +0.0        0.07        perf-profile.children.cycles-pp.__list_add_valid
      0.27 ±  8%      +0.0        0.30 ±  3%  perf-profile.children.cycles-pp.page_remove_rmap
      0.28 ±  5%      +0.0        0.31 ±  6%  perf-profile.children.cycles-pp.mem_cgroup_try_charge_delay
      0.15 ± 11%      +0.0        0.18 ±  2%  perf-profile.children.cycles-pp.do_wait
      0.30 ±  4%      +0.0        0.33        perf-profile.children.cycles-pp.perf_iterate_sb
      0.24 ± 11%      +0.0        0.27 ±  3%  perf-profile.children.cycles-pp.lru_add_drain_cpu
      0.11 ±  7%      +0.0        0.15 ±  3%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.21 ± 10%      +0.0        0.25        perf-profile.children.cycles-pp.anon_vma_fork
      0.08 ± 10%      +0.0        0.12 ± 25%  perf-profile.children.cycles-pp.memcpy_erms
      0.24 ±  7%      +0.0        0.28 ±  5%  perf-profile.children.cycles-pp.schedule
      0.23 ± 13%      +0.0        0.27 ±  5%  perf-profile.children.cycles-pp.lru_add_drain
      0.27 ±  8%      +0.0        0.30        perf-profile.children.cycles-pp.scheduler_tick
      0.19 ±  9%      +0.0        0.23 ±  4%  perf-profile.children.cycles-pp.anon_vma_clone
      0.33 ±  7%      +0.0        0.36 ±  4%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.16 ± 13%      +0.0        0.20 ±  4%  perf-profile.children.cycles-pp.kernel_wait4
      0.41 ±  7%      +0.0        0.46        perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.interrupt_entry
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.find_busiest_group
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.update_load_avg
      0.17 ±  6%      +0.0        0.21 ±  4%  perf-profile.children.cycles-pp.try_to_wake_up
      0.16 ± 13%      +0.0        0.21 ±  7%  perf-profile.children.cycles-pp.__do_sys_wait4
      0.11 ± 29%      +0.0        0.16 ±  9%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.update_sd_lb_stats
      0.40 ±  6%      +0.1        0.45 ±  3%  perf-profile.children.cycles-pp.update_process_times
      0.42 ±  6%      +0.1        0.47 ±  4%  perf-profile.children.cycles-pp.tick_sched_handle
      0.51 ±  6%      +0.1        0.56 ±  3%  perf-profile.children.cycles-pp.do_sys_open
      0.52 ±  6%      +0.1        0.57 ±  2%  perf-profile.children.cycles-pp.path_openat
      0.52 ±  6%      +0.1        0.58 ±  3%  perf-profile.children.cycles-pp.do_filp_open
      0.45 ±  5%      +0.1        0.52 ±  3%  perf-profile.children.cycles-pp.tick_sched_timer
      0.55 ±  5%      +0.1        0.62 ±  4%  perf-profile.children.cycles-pp.perf_event_mmap
      0.33 ±  7%      +0.1        0.40 ±  4%  perf-profile.children.cycles-pp.__schedule
      0.32 ± 16%      +0.1        0.40 ± 13%  perf-profile.children.cycles-pp._raw_spin_lock
      0.40 ± 15%      +0.1        0.48 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.62 ±  3%      +0.1        0.70        perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.59 ± 10%      +0.1        0.67 ±  5%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.80 ±  3%      +0.1        0.91        perf-profile.children.cycles-pp.hrtimer_interrupt
      0.42 ± 18%      +0.1        0.53        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      1.42 ±  7%      +0.1        1.53        perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.91 ± 10%      +0.1        1.02 ±  3%  perf-profile.children.cycles-pp.release_pages
      0.82 ±  9%      +0.1        0.95 ±  4%  perf-profile.children.cycles-pp.filemap_map_pages
      1.01 ±  8%      +0.1        1.14 ±  4%  perf-profile.children.cycles-pp.unmap_page_range
      1.07 ±  8%      +0.1        1.21 ±  3%  perf-profile.children.cycles-pp.unmap_vmas
      1.27 ±  7%      +0.2        1.42        perf-profile.children.cycles-pp.do_brk_flags
      1.57 ±  8%      +0.2        1.73 ±  2%  perf-profile.children.cycles-pp.tlb_flush_mmu
      1.59 ±  8%      +0.2        1.77 ±  2%  perf-profile.children.cycles-pp.tlb_finish_mmu
      1.19 ±  5%      +0.2        1.37 ±  3%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      1.32 ±  3%      +0.2        1.50 ±  2%  perf-profile.children.cycles-pp.apic_timer_interrupt
      1.77 ±  9%      +0.2        1.99 ±  4%  perf-profile.children.cycles-pp.unmap_region
      2.11 ± 10%      +0.3        2.38 ±  4%  perf-profile.children.cycles-pp.__do_munmap
      0.90 ± 21%      +0.3        1.18 ± 10%  perf-profile.children.cycles-pp.__libc_fork
      3.90 ±  8%      +0.3        4.23        perf-profile.children.cycles-pp.__handle_mm_fault
      4.12 ±  7%      +0.3        4.46        perf-profile.children.cycles-pp.handle_mm_fault
      3.14 ±  8%      +0.4        3.49 ±  3%  perf-profile.children.cycles-pp.__x64_sys_brk
      0.74 ± 36%      +0.5        1.20        perf-profile.children.cycles-pp.vma_link
      1.12 ± 29%      +0.5        1.62 ±  2%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      1.22 ± 29%      +0.6        1.79        perf-profile.children.cycles-pp.mmap_region
      1.31 ± 28%      +0.6        1.88        perf-profile.children.cycles-pp.do_mmap
      1.36 ± 27%      +0.6        1.93        perf-profile.children.cycles-pp.vm_mmap_pgoff
      3.32 ± 24%      +0.6        3.96 ±  5%  perf-profile.children.cycles-pp.search_binary_handler
      3.98 ± 21%      +0.6        4.62 ±  4%  perf-profile.children.cycles-pp.__do_execve_file
      4.00 ± 21%      +0.6        4.65 ±  4%  perf-profile.children.cycles-pp.__x64_sys_execve
      3.07 ± 11%      +0.7        3.79 ±  3%  perf-profile.children.cycles-pp.intel_idle
      2.40 ± 25%      +0.7        3.15 ±  7%  perf-profile.children.cycles-pp.__x64_sys_exit_group
      2.40 ± 25%      +0.7        3.15 ±  7%  perf-profile.children.cycles-pp.do_group_exit
      2.40 ± 25%      +0.8        3.15 ±  7%  perf-profile.children.cycles-pp.do_exit
      3.60 ±  8%      +0.8        4.36 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter
      3.60 ±  8%      +0.8        4.36 ±  2%  perf-profile.children.cycles-pp.cpuidle_enter_state
      2.01 ± 28%      +0.8        2.78 ± 13%  perf-profile.children.cycles-pp.dup_mm
      3.83 ±  7%      +0.8        4.60 ±  3%  perf-profile.children.cycles-pp.do_idle
      3.83 ±  7%      +0.8        4.60 ±  3%  perf-profile.children.cycles-pp.secondary_startup_64
      3.83 ±  7%      +0.8        4.60 ±  3%  perf-profile.children.cycles-pp.cpu_startup_entry
      2.31 ± 25%      +0.8        3.09 ± 12%  perf-profile.children.cycles-pp.copy_process
      2.43 ± 24%      +0.8        3.22 ± 12%  perf-profile.children.cycles-pp._do_fork
      3.77 ±  8%      +0.8        4.60 ±  3%  perf-profile.children.cycles-pp.start_secondary
      2.89 ± 39%      +1.1        4.01 ± 11%  perf-profile.children.cycles-pp.unlink_file_vma
      3.24 ± 36%      +1.1        4.38 ± 11%  perf-profile.children.cycles-pp.free_pgtables
      4.46 ± 28%      +1.3        5.72 ±  8%  perf-profile.children.cycles-pp.exit_mmap
      4.46 ± 28%      +1.3        5.73 ±  8%  perf-profile.children.cycles-pp.mmput
      5.02 ± 40%      +2.2        7.23 ± 10%  perf-profile.children.cycles-pp.osq_lock
      5.33 ± 40%      +2.4        7.70 ± 11%  perf-profile.children.cycles-pp.rwsem_down_write_slowpath
      5.50 ± 39%      +2.4        7.88 ± 10%  perf-profile.children.cycles-pp.down_write
     17.38 ± 16%      +3.4       20.80 ±  3%  perf-profile.children.cycles-pp.do_syscall_64
     17.43 ± 16%      +3.4       20.86 ±  3%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     33.64 ±  6%      -2.9       30.71        perf-profile.self.cycles-pp.sieve
      9.29 ±  6%      -0.9        8.35        perf-profile.self.cycles-pp.string_rtns_1
      0.43 ±  9%      -0.0        0.40        perf-profile.self.cycles-pp.page_test
      0.14 ±  3%      -0.0        0.11 ±  4%  perf-profile.self.cycles-pp.free
      0.09 ±  9%      -0.0        0.07        perf-profile.self.cycles-pp.memcpy
      0.08            +0.0        0.09        perf-profile.self.cycles-pp.alloc_set_pte
      0.05 ±  9%      +0.0        0.07        perf-profile.self.cycles-pp.native_write_msr
      0.12 ±  9%      +0.0        0.14 ±  3%  perf-profile.self.cycles-pp.vmacache_find
      0.10 ± 11%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.16 ± 11%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.copy_p4d_range
      0.15 ±  5%      +0.0        0.17        perf-profile.self.cycles-pp.perf_iterate_sb
      0.04 ± 58%      +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.mem_cgroup_throttle_swaprate
      0.23 ±  8%      +0.0        0.26        perf-profile.self.cycles-pp.page_remove_rmap
      0.18 ±  8%      +0.0        0.21 ±  2%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.04 ± 58%      +0.0        0.07        perf-profile.self.cycles-pp.__list_add_valid
      0.08 ± 26%      +0.0        0.12 ± 13%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.22 ±  7%      +0.0        0.26 ±  9%  perf-profile.self.cycles-pp._raw_spin_lock
      0.03 ±100%      +0.0        0.07 ±  7%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.41 ±  7%      +0.0        0.46        perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.32 ±  8%      +0.0        0.36        perf-profile.self.cycles-pp.release_pages
      0.35 ±  7%      +0.0        0.39 ±  2%  perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.interrupt_entry
      0.56 ±  8%      +0.1        0.62        perf-profile.self.cycles-pp.unmap_page_range
      0.19 ± 36%      +0.1        0.29 ± 12%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.42 ± 18%      +0.1        0.53        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      3.07 ± 11%      +0.7        3.79 ±  3%  perf-profile.self.cycles-pp.intel_idle
      4.97 ± 41%      +2.2        7.17 ± 11%  perf-profile.self.cycles-pp.osq_lock



***************************************************************************************************
lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/300t/debian-x86_64-2019-05-14.cgz/300s/lkp-ivb-2ep1/disk_wrt/reaim/0x42e

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          0:4           -4%           0:2     perf-profile.children.cycles-pp.error_entry
          0:4           -3%           0:2     perf-profile.self.cycles-pp.error_entry



***************************************************************************************************
lkp-bdw-ex2: 160 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 256G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/debian-x86_64-2019-05-14.cgz/300s/lkp-bdw-ex2/small-allocs/vm-scalability/0xb000036

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          2:4          -27%           1:2     perf-profile.calltrace.cycles-pp.error_entry.do_access
          2:4          -32%           1:2     perf-profile.children.cycles-pp.error_entry
          2:4          -26%           1:2     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
      3852            -4.9%       3664        vm-scalability.time.maximum_resident_set_size
      0.02 ±109%      +0.0        0.05 ± 63%  mpstat.cpu.all.soft%
     57.25            -2.2%      56.00        turbostat.PkgTmp
     53.13            +2.0%      54.21        boot-time.boot
      8844 ±  2%      +3.5%       9154        boot-time.idle
    117451 ±  5%     +14.1%     134069 ±  6%  softirqs.CPU191.TIMER
     20234 ±  8%      +6.3%      21507 ±  7%  softirqs.CPU88.RCU
    615729            -1.5%     606471        proc-vmstat.nr_page_table_pages
     24641            +1.4%      24974        proc-vmstat.nr_slab_reclaimable
   1632391            -1.5%    1607656        proc-vmstat.nr_slab_unreclaimable
    717417 ± 10%     +64.0%    1176379 ±  3%  numa-numastat.node2.local_node
    743836 ±  9%     +58.2%    1176412 ±  3%  numa-numastat.node2.numa_hit
     26418 ± 30%     -99.9%      37.50 ± 44%  numa-numastat.node2.other_node
     15524 ± 72%    +100.3%      31101        numa-numastat.node3.other_node
      6.28 ±  8%      +9.1%       6.86 ±  8%  perf-stat.i.MPKI
      0.45 ±  5%      +0.0        0.50 ±  3%  perf-stat.i.branch-miss-rate%
  47984336 ±  4%      +9.8%   52679218 ±  2%  perf-stat.i.branch-misses
  2.18e+08 ±  8%      +9.3%  2.383e+08 ±  8%  perf-stat.i.cache-references
   4924680 ±  6%      +7.9%    5312753 ±  7%  perf-stat.i.iTLB-load-misses
    491869 ±  5%      +5.7%     520141 ±  4%  perf-stat.i.node-loads
      3.03 ±  5%     +13.8%       3.45 ±  9%  perf-stat.overall.MPKI
      0.29            +0.0        0.32        perf-stat.overall.branch-miss-rate%
  47906630 ±  4%      +9.8%   52585388 ±  2%  perf-stat.ps.branch-misses
 2.176e+08 ±  8%      +9.3%  2.379e+08 ±  8%  perf-stat.ps.cache-references
   4917994 ±  6%      +7.8%    5303407 ±  7%  perf-stat.ps.iTLB-load-misses
    489770 ±  5%      +5.8%     518214 ±  4%  perf-stat.ps.node-loads
     17149 ± 28%     -52.0%       8235 ±  3%  sched_debug.cfs_rq:/.exec_clock.max
   1378506 ± 52%     -92.1%     108984 ±  2%  sched_debug.cfs_rq:/.min_vruntime.avg
   1439941 ± 51%     -89.9%     144995 ±  3%  sched_debug.cfs_rq:/.min_vruntime.max
   1173447 ± 53%     -94.3%      67349 ± 10%  sched_debug.cfs_rq:/.min_vruntime.min
      0.15 ± 35%     -59.2%       0.06 ± 12%  sched_debug.cfs_rq:/.nr_running.avg
      0.15 ± 33%     -57.3%       0.07        sched_debug.cfs_rq:/.nr_spread_over.avg
      1.89 ± 34%     -54.4%       0.86 ±  2%  sched_debug.cfs_rq:/.runnable_load_avg.avg
    176.85 ± 27%     -43.4%     100.17 ±  3%  sched_debug.cfs_rq:/.util_avg.avg
     94.34 ± 59%     -92.3%       7.23 ± 11%  sched_debug.cfs_rq:/.util_est_enqueued.avg
      1.21 ± 20%     -40.7%       0.71        sched_debug.cpu.cpu_load[0].avg
    450.75 ± 41%     -61.6%     173.13 ±  6%  sched_debug.cpu.curr->pid.avg
      8160 ±  9%     -19.4%       6580 ±  2%  sched_debug.cpu.curr->pid.max
    914.49 ±  6%     -13.8%     788.17 ±  3%  sched_debug.cpu.curr->pid.stddev
     72735 ±  4%    +144.1%     177555 ± 56%  sched_debug.cpu.load.max
      1077 ±  5%     +29.6%       1396 ± 22%  sched_debug.cpu.nr_load_updates.stddev
      0.14 ± 43%     -61.4%       0.05 ±  7%  sched_debug.cpu.nr_running.avg
      0.70 ±  5%      +9.7%       0.77        sched_debug.cpu.nr_uninterruptible.avg
     89157 ± 40%     -81.2%      16779 ± 92%  numa-meminfo.node0.Active
     89092 ± 40%     -81.2%      16779 ± 92%  numa-meminfo.node0.Active(anon)
     60805 ± 41%     -84.3%       9523 ±100%  numa-meminfo.node0.AnonHugePages
     83875 ± 40%     -84.5%      13028 ± 91%  numa-meminfo.node0.AnonPages
    255728 ±  4%    +108.4%     532879        numa-meminfo.node2.FilePages
      1763 ± 11%  +13493.2%     239750        numa-meminfo.node2.Inactive
      1188 ± 46%  +20035.5%     239361        numa-meminfo.node2.Inactive(anon)
     15779 ± 28%    +125.9%      35646 ±  6%  numa-meminfo.node2.KReclaimable
      5937 ± 10%     +68.2%       9985        numa-meminfo.node2.KernelStack
     15779 ± 28%    +125.9%      35646 ±  6%  numa-meminfo.node2.SReclaimable
      2215 ± 67%  +11124.7%     248655        numa-meminfo.node2.Shmem
    252853 ±  3%     +12.2%     283657        numa-meminfo.node2.Unevictable
     83953 ± 11%     +59.0%     133447 ± 20%  numa-meminfo.node3.Active
     83842 ± 11%     +59.1%     133371 ± 20%  numa-meminfo.node3.Active(anon)
     54812 ± 18%     +77.5%      97277 ± 21%  numa-meminfo.node3.AnonHugePages
     78034 ± 11%     +69.6%     132378 ± 20%  numa-meminfo.node3.AnonPages
    513368 ±  3%     -51.0%     251319        numa-meminfo.node3.FilePages
    239267           -99.3%       1632 ±  2%  numa-meminfo.node3.Inactive
    238889           -99.7%     667.50 ±  6%  numa-meminfo.node3.Inactive(anon)
      9016 ± 18%     -33.3%       6017 ±  4%  numa-meminfo.node3.KernelStack
   2964209 ±  4%      -9.6%    2678216        numa-meminfo.node3.MemUsed
    243897           -99.6%     862.00 ± 10%  numa-meminfo.node3.Shmem
     22271 ± 40%     -81.2%       4197 ± 92%  numa-vmstat.node0.nr_active_anon
     20968 ± 40%     -84.4%       3263 ± 91%  numa-vmstat.node0.nr_anon_pages
     22271 ± 40%     -81.2%       4197 ± 92%  numa-vmstat.node0.nr_zone_active_anon
     63932 ±  4%    +108.4%     133212        numa-vmstat.node2.nr_file_pages
    296.50 ± 46%  +20081.3%      59837        numa-vmstat.node2.nr_inactive_anon
      5938 ± 10%     +68.3%       9996        numa-vmstat.node2.nr_kernel_stack
      1234           +12.3%       1386        numa-vmstat.node2.nr_mapped
    553.25 ± 67%  +11134.9%      62157        numa-vmstat.node2.nr_shmem
      3944 ± 28%    +125.9%       8910 ±  6%  numa-vmstat.node2.nr_slab_reclaimable
     63213 ±  3%     +12.2%      70914        numa-vmstat.node2.nr_unevictable
    296.50 ± 46%  +20081.3%      59837        numa-vmstat.node2.nr_zone_inactive_anon
     63213 ±  3%     +12.2%      70914        numa-vmstat.node2.nr_zone_unevictable
    674596 ± 10%     +55.5%    1049184 ±  5%  numa-vmstat.node2.numa_hit
    565285 ± 13%     +70.8%     965693 ±  5%  numa-vmstat.node2.numa_local
    109311 ±  7%     -23.6%      83491        numa-vmstat.node2.numa_other
     20964 ± 11%     +59.0%      33339 ± 20%  numa-vmstat.node3.nr_active_anon
     19515 ± 11%     +69.6%      33089 ± 20%  numa-vmstat.node3.nr_anon_pages
    128296 ±  3%     -51.0%      62829        numa-vmstat.node3.nr_file_pages
     59679           -99.7%     166.50 ±  6%  numa-vmstat.node3.nr_inactive_anon
      9013 ± 18%     -33.3%       6014 ±  4%  numa-vmstat.node3.nr_kernel_stack
     60929           -99.6%     215.00 ± 10%  numa-vmstat.node3.nr_shmem
     20964 ± 11%     +59.0%      33339 ± 20%  numa-vmstat.node3.nr_zone_active_anon
     59679           -99.7%     166.50 ±  6%  numa-vmstat.node3.nr_zone_inactive_anon
     98042 ± 11%     +16.1%     113819        numa-vmstat.node3.numa_other
      2.35 ±  7%      -0.4        1.94        perf-profile.calltrace.cycles-pp.mmap64
      2.29 ±  8%      -0.4        1.89        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64
      2.29 ±  8%      -0.4        1.89        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.mmap64
      2.21 ±  7%      -0.4        1.83        perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64
      2.14 ±  8%      -0.4        1.78        perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64
      0.66 ± 10%      +0.1        0.77 ±  7%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.40 ± 57%      +0.2        0.56 ±  3%  perf-profile.calltrace.cycles-pp.wake_up_q.rwsem_wake.vma_link.mmap_region.do_mmap
      0.72 ± 22%      +0.2        0.90 ± 11%  perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      7.35            +0.2        7.56        perf-profile.calltrace.cycles-pp.do_rw_once
      1.32 ±  6%      +0.2        1.55        perf-profile.calltrace.cycles-pp.tick_nohz_next_event.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry
      1.69 ±  6%      +0.2        1.94 ±  2%  perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.do_idle.cpu_startup_entry.start_secondary
      0.49 ± 61%      +0.3        0.74 ± 10%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      1.39 ± 20%      +0.3        1.72 ± 11%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      1.53 ± 20%      +0.4        1.89 ± 11%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.14 ±173%      +0.4        0.54 ±  2%  perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.rwsem_wake.vma_link.mmap_region
      1.74 ± 18%      +0.4        2.15 ± 10%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      4.31 ±  9%      +0.6        4.92 ±  8%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      8.64 ±  6%      +0.7        9.36 ±  2%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     61.25 ±  2%      +1.3       62.53        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     61.75 ±  2%      +1.4       63.12        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     67.87 ±  2%      +1.6       69.50        perf-profile.calltrace.cycles-pp.secondary_startup_64
     67.47 ±  2%      +1.7       69.17        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     67.47 ±  2%      +1.7       69.17        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     67.41 ±  2%      +1.7       69.12        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      2.36 ±  7%      -0.4        1.96        perf-profile.children.cycles-pp.mmap64
      0.12 ± 13%      +0.0        0.15 ± 10%  perf-profile.children.cycles-pp.run_posix_cpu_timers
      0.13 ±  5%      +0.0        0.15 ±  3%  perf-profile.children.cycles-pp.rcu_idle_exit
      0.09 ±  7%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.cpu_load_update
      0.31 ±  6%      +0.0        0.34        perf-profile.children.cycles-pp._raw_spin_lock
      0.23 ±  3%      +0.0        0.26 ±  5%  perf-profile.children.cycles-pp.sched_clock
      0.08 ± 15%      +0.0        0.11 ± 18%  perf-profile.children.cycles-pp.rcu_irq_exit
      0.10 ± 21%      +0.0        0.14 ± 18%  perf-profile.children.cycles-pp.rcu_dynticks_eqs_exit
      0.04 ± 57%      +0.0        0.07 ± 14%  perf-profile.children.cycles-pp.hrtimer_forward
      0.10 ± 26%      +0.0        0.14 ± 18%  perf-profile.children.cycles-pp.rcu_irq_enter
      0.10 ± 24%      +0.0        0.14 ± 11%  perf-profile.children.cycles-pp.perf_event_task_tick
      0.03 ±100%      +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.irq_work_tick
      0.09 ± 22%      +0.0        0.13 ±  7%  perf-profile.children.cycles-pp.__hrtimer_get_next_event
      0.14 ± 22%      +0.0        0.18 ± 16%  perf-profile.children.cycles-pp.interrupt_entry
      0.30 ±  5%      +0.0        0.34        perf-profile.children.cycles-pp.sched_clock_cpu
      0.05 ± 64%      +0.1        0.11 ± 23%  perf-profile.children.cycles-pp.rb_insert_color
      0.01 ±173%      +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.tick_sched_do_timer
      0.30 ±  8%      +0.1        0.35 ±  7%  perf-profile.children.cycles-pp.find_next_bit
      0.49 ±  9%      +0.1        0.56 ± 11%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.24 ± 25%      +0.1        0.32 ± 14%  perf-profile.children.cycles-pp.__hrtimer_next_event_base
      0.20 ± 32%      +0.1        0.28 ± 17%  perf-profile.children.cycles-pp.timerqueue_add
      0.30 ± 10%      +0.1        0.39 ± 10%  perf-profile.children.cycles-pp.read_tsc
      0.23 ± 30%      +0.1        0.32 ± 15%  perf-profile.children.cycles-pp.enqueue_hrtimer
      0.61 ± 22%      +0.1        0.76 ±  9%  perf-profile.children.cycles-pp.tick_irq_enter
      0.28 ± 17%      +0.2        0.46 ± 16%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.75 ± 21%      +0.2        0.93 ±  9%  perf-profile.children.cycles-pp.irq_enter
      1.36 ±  6%      +0.2        1.58        perf-profile.children.cycles-pp.tick_nohz_next_event
      1.71 ±  5%      +0.2        1.95 ±  2%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      1.59 ± 14%      +0.3        1.86 ± 11%  perf-profile.children.cycles-pp.update_process_times
      1.73 ± 14%      +0.3        2.03 ± 11%  perf-profile.children.cycles-pp.tick_sched_handle
      1.96 ± 13%      +0.3        2.29 ± 10%  perf-profile.children.cycles-pp.tick_sched_timer
      3.23 ±  7%      +0.3        3.56 ±  8%  perf-profile.children.cycles-pp.menu_select
      3.37 ± 13%      +0.4        3.76 ± 10%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      4.67 ±  7%      +0.6        5.30 ± 10%  perf-profile.children.cycles-pp.hrtimer_interrupt
     62.12 ±  2%      +1.3       63.42        perf-profile.children.cycles-pp.cpuidle_enter
     62.10 ±  2%      +1.3       63.41        perf-profile.children.cycles-pp.cpuidle_enter_state
     67.88 ±  2%      +1.6       69.51        perf-profile.children.cycles-pp.do_idle
     67.87 ±  2%      +1.6       69.50        perf-profile.children.cycles-pp.secondary_startup_64
     67.87 ±  2%      +1.6       69.50        perf-profile.children.cycles-pp.cpu_startup_entry
     67.47 ±  2%      +1.7       69.17        perf-profile.children.cycles-pp.start_secondary
      0.29            -0.0        0.27        perf-profile.self.cycles-pp.do_syscall_64
      0.06 ±  6%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.__perf_sw_event
      0.12 ± 13%      +0.0        0.15 ± 10%  perf-profile.self.cycles-pp.run_posix_cpu_timers
      0.09 ±  7%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.cpu_load_update
      0.29 ±  6%      +0.0        0.32        perf-profile.self.cycles-pp._raw_spin_lock
      0.20 ±  2%      +0.0        0.23 ±  4%  perf-profile.self.cycles-pp.native_sched_clock
      0.10 ± 21%      +0.0        0.13 ± 15%  perf-profile.self.cycles-pp.rcu_dynticks_eqs_exit
      0.08 ± 15%      +0.0        0.11 ± 18%  perf-profile.self.cycles-pp.rcu_irq_exit
      0.04 ± 59%      +0.0        0.08 ±  6%  perf-profile.self.cycles-pp.__hrtimer_get_next_event
      0.04 ± 57%      +0.0        0.07 ± 14%  perf-profile.self.cycles-pp.hrtimer_forward
      0.03 ±100%      +0.0        0.06        perf-profile.self.cycles-pp.rcu_idle_exit
      0.14 ± 19%      +0.0        0.17 ± 17%  perf-profile.self.cycles-pp.interrupt_entry
      0.10 ± 24%      +0.0        0.14 ± 11%  perf-profile.self.cycles-pp.perf_event_task_tick
      0.26 ±  8%      +0.0        0.30 ±  6%  perf-profile.self.cycles-pp.find_next_bit
      0.06 ± 26%      +0.0        0.11        perf-profile.self.cycles-pp.lapic_next_deadline
      0.13 ± 13%      +0.1        0.18 ±  2%  perf-profile.self.cycles-pp.hrtimer_interrupt
      0.21 ± 17%      +0.1        0.27 ± 13%  perf-profile.self.cycles-pp.__hrtimer_next_event_base
      0.04 ±103%      +0.1        0.11 ± 23%  perf-profile.self.cycles-pp.rb_insert_color
      0.28 ± 10%      +0.1        0.37 ± 10%  perf-profile.self.cycles-pp.read_tsc
      0.85 ± 10%      +0.1        0.98 ±  4%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.30 ± 35%      +0.2        0.47 ±  7%  perf-profile.self.cycles-pp.tick_nohz_next_event
    519.50 ± 25%     +24.0%     644.00 ± 17%  interrupts.110:PCI-MSI.1574912-edge.eth1-TxRx-0
    196.25 ±  9%     +57.7%     309.50 ± 20%  interrupts.116:PCI-MSI.1574918-edge.eth1-TxRx-6
    310.50 ± 53%     -43.2%     176.50        interrupts.122:PCI-MSI.1574924-edge.eth1-TxRx-12
    183.25           +15.4%     211.50 ± 11%  interrupts.161:PCI-MSI.1574963-edge.eth1-TxRx-51
    180.25 ±  2%     +55.1%     279.50 ± 32%  interrupts.164:PCI-MSI.1574966-edge.eth1-TxRx-54
    519.50 ± 25%     +24.0%     644.00 ± 17%  interrupts.CPU0.110:PCI-MSI.1574912-edge.eth1-TxRx-0
      1203 ±  6%     -28.0%     866.50 ± 20%  interrupts.CPU1.NMI:Non-maskable_interrupts
      1203 ±  6%     -28.0%     866.50 ± 20%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
      5359 ± 39%     +60.5%       8604 ±  6%  interrupts.CPU10.RES:Rescheduling_interrupts
      5784 ± 37%     +58.5%       9168 ± 15%  interrupts.CPU11.RES:Rescheduling_interrupts
    909.00 ± 26%     +39.5%       1268 ±  4%  interrupts.CPU110.NMI:Non-maskable_interrupts
    909.00 ± 26%     +39.5%       1268 ±  4%  interrupts.CPU110.PMI:Performance_monitoring_interrupts
    922.25 ± 25%     +34.0%       1236        interrupts.CPU111.NMI:Non-maskable_interrupts
    922.25 ± 25%     +34.0%       1236        interrupts.CPU111.PMI:Performance_monitoring_interrupts
    922.00 ± 30%     +34.8%       1243 ±  4%  interrupts.CPU114.NMI:Non-maskable_interrupts
    922.00 ± 30%     +34.8%       1243 ±  4%  interrupts.CPU114.PMI:Performance_monitoring_interrupts
      1033 ± 24%     +19.0%       1229 ±  4%  interrupts.CPU116.NMI:Non-maskable_interrupts
      1033 ± 24%     +19.0%       1229 ±  4%  interrupts.CPU116.PMI:Performance_monitoring_interrupts
    867.50 ± 30%     +33.7%       1160 ±  7%  interrupts.CPU118.NMI:Non-maskable_interrupts
    867.50 ± 30%     +33.7%       1160 ±  7%  interrupts.CPU118.PMI:Performance_monitoring_interrupts
      6329 ± 35%     +30.5%       8260 ± 23%  interrupts.CPU119.RES:Rescheduling_interrupts
    310.50 ± 53%     -43.2%     176.50        interrupts.CPU12.122:PCI-MSI.1574924-edge.eth1-TxRx-12
      6510 ± 26%     +25.0%       8140 ± 22%  interrupts.CPU120.RES:Rescheduling_interrupts
      6814 ± 26%     +41.7%       9655        interrupts.CPU122.RES:Rescheduling_interrupts
      6239 ± 29%     +24.6%       7774 ± 23%  interrupts.CPU125.RES:Rescheduling_interrupts
      6504 ± 28%     +25.1%       8138 ± 24%  interrupts.CPU126.RES:Rescheduling_interrupts
      6206 ± 30%     +28.5%       7974 ± 22%  interrupts.CPU128.RES:Rescheduling_interrupts
      6156 ± 30%     +23.8%       7620 ± 25%  interrupts.CPU129.RES:Rescheduling_interrupts
      5867 ± 38%     +57.2%       9223 ± 22%  interrupts.CPU13.RES:Rescheduling_interrupts
      6343 ± 29%     +22.4%       7765 ± 28%  interrupts.CPU132.RES:Rescheduling_interrupts
      6263 ± 32%     +26.6%       7926 ± 27%  interrupts.CPU138.RES:Rescheduling_interrupts
      1071 ±  9%     +20.7%       1294        interrupts.CPU14.NMI:Non-maskable_interrupts
      1071 ±  9%     +20.7%       1294        interrupts.CPU14.PMI:Performance_monitoring_interrupts
      6903 ± 25%     +40.6%       9702 ±  8%  interrupts.CPU142.RES:Rescheduling_interrupts
      5677 ± 40%     +53.7%       8724 ± 30%  interrupts.CPU16.RES:Rescheduling_interrupts
    721.25 ± 24%     +39.3%       1004 ± 32%  interrupts.CPU160.NMI:Non-maskable_interrupts
    721.25 ± 24%     +39.3%       1004 ± 32%  interrupts.CPU160.PMI:Performance_monitoring_interrupts
    785.75 ± 29%     +27.8%       1004 ± 29%  interrupts.CPU161.NMI:Non-maskable_interrupts
    785.75 ± 29%     +27.8%       1004 ± 29%  interrupts.CPU161.PMI:Performance_monitoring_interrupts
    778.50 ± 34%     +49.2%       1161 ±  8%  interrupts.CPU162.NMI:Non-maskable_interrupts
    778.50 ± 34%     +49.2%       1161 ±  8%  interrupts.CPU162.PMI:Performance_monitoring_interrupts
      4841 ± 22%     +59.2%       7707 ±  9%  interrupts.CPU170.RES:Rescheduling_interrupts
      5010 ± 26%     +41.1%       7067 ± 17%  interrupts.CPU175.RES:Rescheduling_interrupts
      5333 ± 30%     +50.5%       8025 ± 11%  interrupts.CPU176.RES:Rescheduling_interrupts
      4825 ± 19%     +52.4%       7354 ± 17%  interrupts.CPU179.RES:Rescheduling_interrupts
    827.00 ± 36%     +38.0%       1141 ± 14%  interrupts.CPU180.NMI:Non-maskable_interrupts
    827.00 ± 36%     +38.0%       1141 ± 14%  interrupts.CPU180.PMI:Performance_monitoring_interrupts
      1266 ±  3%     -30.5%     880.00 ± 28%  interrupts.CPU2.NMI:Non-maskable_interrupts
      1266 ±  3%     -30.5%     880.00 ± 28%  interrupts.CPU2.PMI:Performance_monitoring_interrupts
      6759 ± 31%     +36.8%       9246 ±  5%  interrupts.CPU20.RES:Rescheduling_interrupts
      5976 ± 35%     +41.2%       8438 ± 14%  interrupts.CPU21.RES:Rescheduling_interrupts
      5601 ± 39%     +72.7%       9674 ± 14%  interrupts.CPU22.RES:Rescheduling_interrupts
      6356 ± 30%     +34.8%       8568 ± 14%  interrupts.CPU25.RES:Rescheduling_interrupts
      6356 ± 29%     +24.6%       7922 ± 22%  interrupts.CPU26.RES:Rescheduling_interrupts
      1287 ±  4%      -5.7%       1213        interrupts.CPU27.NMI:Non-maskable_interrupts
      1287 ±  4%      -5.7%       1213        interrupts.CPU27.PMI:Performance_monitoring_interrupts
      6381 ± 29%     +42.2%       9075 ± 10%  interrupts.CPU27.RES:Rescheduling_interrupts
      6168 ± 28%     +27.0%       7833 ± 23%  interrupts.CPU28.RES:Rescheduling_interrupts
      6346 ± 29%     +25.3%       7955 ± 24%  interrupts.CPU29.RES:Rescheduling_interrupts
      6413 ± 26%     +26.7%       8128 ± 23%  interrupts.CPU30.RES:Rescheduling_interrupts
      6115 ± 33%     +70.4%      10417 ±  7%  interrupts.CPU31.RES:Rescheduling_interrupts
      6226 ± 28%     +26.1%       7853 ± 25%  interrupts.CPU32.RES:Rescheduling_interrupts
      5697 ± 33%     +69.0%       9630 ± 11%  interrupts.CPU4.RES:Rescheduling_interrupts
      6328 ± 27%     +38.1%       8740 ±  8%  interrupts.CPU41.RES:Rescheduling_interrupts
      6079 ± 31%     +22.8%       7465 ± 28%  interrupts.CPU42.RES:Rescheduling_interrupts
      6296 ± 26%     +20.2%       7565 ± 25%  interrupts.CPU44.RES:Rescheduling_interrupts
      6309 ± 32%     +25.8%       7939 ± 27%  interrupts.CPU47.RES:Rescheduling_interrupts
    927.75 ± 33%     +41.3%       1310 ±  3%  interrupts.CPU48.NMI:Non-maskable_interrupts
    927.75 ± 33%     +41.3%       1310 ±  3%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
    701.75 ± 18%     +69.4%       1188 ±  7%  interrupts.CPU50.NMI:Non-maskable_interrupts
    701.75 ± 18%     +69.4%       1188 ±  7%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
    183.25           +15.4%     211.50 ± 11%  interrupts.CPU51.161:PCI-MSI.1574963-edge.eth1-TxRx-51
    180.25 ±  2%     +55.1%     279.50 ± 32%  interrupts.CPU54.164:PCI-MSI.1574966-edge.eth1-TxRx-54
      1273           -23.5%     974.00 ± 13%  interrupts.CPU54.NMI:Non-maskable_interrupts
      1273           -23.5%     974.00 ± 13%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
      1250 ±  2%     -27.2%     910.50 ±  3%  interrupts.CPU55.NMI:Non-maskable_interrupts
      1250 ±  2%     -27.2%     910.50 ±  3%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
      6200 ± 26%     +43.9%       8925 ± 22%  interrupts.CPU55.RES:Rescheduling_interrupts
      6497 ± 23%     +34.0%       8708 ± 16%  interrupts.CPU58.RES:Rescheduling_interrupts
    196.25 ±  9%     +57.7%     309.50 ± 20%  interrupts.CPU6.116:PCI-MSI.1574918-edge.eth1-TxRx-6
      1197 ± 10%     -43.8%     673.00 ±  3%  interrupts.CPU64.NMI:Non-maskable_interrupts
      1197 ± 10%     -43.8%     673.00 ±  3%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
      1286 ±  4%     -48.6%     661.50        interrupts.CPU65.NMI:Non-maskable_interrupts
      1286 ±  4%     -48.6%     661.50        interrupts.CPU65.PMI:Performance_monitoring_interrupts
      1059 ± 27%     -51.1%     518.50 ±  8%  interrupts.CPU68.NMI:Non-maskable_interrupts
      1059 ± 27%     -51.1%     518.50 ±  8%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
      1076 ± 25%     -50.0%     537.50        interrupts.CPU69.NMI:Non-maskable_interrupts
      1076 ± 25%     -50.0%     537.50        interrupts.CPU69.PMI:Performance_monitoring_interrupts
      5701 ± 21%     +40.9%       8032 ± 20%  interrupts.CPU72.RES:Rescheduling_interrupts
      5004 ± 30%     +51.4%       7576 ± 18%  interrupts.CPU73.RES:Rescheduling_interrupts
      5079 ± 34%     +53.5%       7796 ± 16%  interrupts.CPU74.RES:Rescheduling_interrupts
      4924 ± 33%     +64.4%       8098 ±  2%  interrupts.CPU75.RES:Rescheduling_interrupts
      1056 ± 24%     -44.4%     587.00 ±  2%  interrupts.CPU77.NMI:Non-maskable_interrupts
      1056 ± 24%     -44.4%     587.00 ±  2%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
      5362 ± 25%     +44.7%       7759 ± 13%  interrupts.CPU77.RES:Rescheduling_interrupts
      5660 ± 37%     +37.9%       7807 ± 20%  interrupts.CPU8.RES:Rescheduling_interrupts
      4760 ± 26%     +58.6%       7549 ± 14%  interrupts.CPU81.RES:Rescheduling_interrupts
    786.50 ± 34%     +41.9%       1116 ± 16%  interrupts.CPU83.NMI:Non-maskable_interrupts
    786.50 ± 34%     +41.9%       1116 ± 16%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
      5101 ± 29%     +58.7%       8095 ±  8%  interrupts.CPU83.RES:Rescheduling_interrupts
      5037 ± 25%     +56.7%       7895 ±  4%  interrupts.CPU86.RES:Rescheduling_interrupts
      5031 ± 24%     +33.8%       6731 ±  8%  interrupts.CPU89.RES:Rescheduling_interrupts
      5011 ± 20%     +35.5%       6789 ±  5%  interrupts.CPU90.RES:Rescheduling_interrupts
      5051 ± 21%     +43.0%       7224 ±  6%  interrupts.CPU91.RES:Rescheduling_interrupts
      5231 ± 17%     +41.0%       7373 ± 12%  interrupts.CPU93.RES:Rescheduling_interrupts
   1156218 ± 26%     +23.3%    1425107 ± 18%  interrupts.RES:Rescheduling_interrupts



***************************************************************************************************
lkp-hsw-4ex1: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory
=========================================================================================
compiler/cpufreq_governor/iterations/kconfig/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/30/x86_64-rhel-7.6/1600%/debian-x86_64-2019-05-14.cgz/lkp-hsw-4ex1/dbase/reaim/0x14

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           14%           0:2     perf-profile.calltrace.cycles-pp.error_entry
          1:4           -8%           1:2     perf-profile.children.cycles-pp.error_entry
          1:4           -7%           0:2     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
    442.37 ±  3%      -5.1%     419.99 ±  2%  reaim.child_systime
     13286 ±  3%      -5.1%      12615 ±  2%  reaim.time.system_time
    111693 ±  6%     +10.8%     123765        meminfo.Shmem
     26070 ± 22%     +39.0%      36245        numa-numastat.node3.other_node
     38.05 ±  2%      +3.3%      39.30        boot-time.boot
      4698 ±  2%      +4.2%       4897        boot-time.idle
     87.32 ±  8%     -12.7%      76.27 ±  2%  sched_debug.cpu.clock.stddev
     87.32 ±  8%     -12.7%      76.27 ±  2%  sched_debug.cpu.clock_task.stddev
     28159 ± 19%     -28.9%      20019        numa-vmstat.node1.nr_slab_unreclaimable
     98486 ± 11%     +17.2%     115461 ±  4%  numa-vmstat.node2.nr_anon_pages
     19281 ±  3%     +15.5%      22264 ± 10%  numa-vmstat.node3.nr_kernel_stack
     23454           +16.2%      27253 ±  9%  numa-vmstat.node3.nr_slab_unreclaimable
    107391 ±  5%      +9.7%     117758        numa-vmstat.node3.numa_other
    150.50 ±  3%     +14.6%     172.50 ±  3%  proc-vmstat.nr_dirtied
    283397            +1.5%     287623        proc-vmstat.nr_file_pages
     27944 ±  6%     +10.5%      30887        proc-vmstat.nr_shmem
    150.50 ±  3%     +14.3%     172.00 ±  3%  proc-vmstat.nr_written
     49856 ±  4%      +4.2%      51959        proc-vmstat.pgactivate
      5655 ± 11%     +12.7%       6376 ± 10%  numa-meminfo.node1.Mapped
    112281 ± 19%     -28.6%      80154        numa-meminfo.node1.SUnreclaim
     28102 ±117%    +196.1%      83206 ±  4%  numa-meminfo.node2.AnonHugePages
    393685 ± 11%     +17.6%     463066 ±  4%  numa-meminfo.node2.AnonPages
     19251 ±  4%     +17.0%      22515 ± 10%  numa-meminfo.node3.KernelStack
     93821           +16.6%     109402 ±  9%  numa-meminfo.node3.SUnreclaim
     90973 ±  2%      +2.5%      93275        perf-stat.i.context-switches
      1.61 ±  2%      -5.8%       1.52        perf-stat.i.cpi
     26562            +1.4%      26929        perf-stat.i.cpu-migrations
  22500284            +2.5%   23068436        perf-stat.i.dTLB-store-misses
  13732559 ±  5%      -8.0%   12628121        perf-stat.i.iTLB-load-misses
     15609 ±  3%      +8.3%      16896        perf-stat.i.instructions-per-iTLB-miss
      0.82            +1.5%       0.84        perf-stat.i.ipc
  35101206            +1.1%   35493977        perf-stat.i.node-load-misses
      0.12            +0.0        0.12        perf-stat.overall.dTLB-store-miss-rate%
     16572 ±  3%      +8.1%      17908        perf-stat.overall.instructions-per-iTLB-miss
     26438            +1.3%      26785        perf-stat.ps.cpu-migrations
  22411063            +2.5%   22965030        perf-stat.ps.dTLB-store-misses
  13678545 ±  5%      -8.1%   12571691        perf-stat.ps.iTLB-load-misses
  34938255            +1.0%   35300180        perf-stat.ps.node-load-misses
    844.25 ± 10%     +31.1%       1106 ±  3%  slabinfo.biovec-128.active_objs
    844.25 ± 10%     +31.1%       1106 ±  3%  slabinfo.biovec-128.num_objs
      1555 ± 10%     +22.4%       1904 ±  3%  slabinfo.dmaengine-unmap-16.active_objs
      1555 ± 10%     +22.4%       1904 ±  3%  slabinfo.dmaengine-unmap-16.num_objs
      3626 ±  8%     +19.3%       4325 ±  8%  slabinfo.eventpoll_pwq.active_objs
      3626 ±  8%     +19.3%       4325 ±  8%  slabinfo.eventpoll_pwq.num_objs
    697.00 ±  7%     -11.2%     619.00        slabinfo.file_lock_cache.active_objs
    697.00 ±  7%     -11.2%     619.00        slabinfo.file_lock_cache.num_objs
      4278 ±  6%     +26.2%       5399 ±  8%  slabinfo.kmalloc-rcl-64.active_objs
      4278 ±  6%     +26.2%       5399 ±  8%  slabinfo.kmalloc-rcl-64.num_objs
     88376 ± 17%     -40.0%      53057        slabinfo.lsm_file_cache.active_objs
    344.75 ± 17%     -40.0%     207.00        slabinfo.lsm_file_cache.active_slabs
     88376 ± 17%     -40.0%      53057        slabinfo.lsm_file_cache.num_objs
    344.75 ± 17%     -40.0%     207.00        slabinfo.lsm_file_cache.num_slabs
      1391 ± 13%     +23.5%       1717        slabinfo.nfs_read_data.active_objs
      1391 ± 13%     +23.5%       1717        slabinfo.nfs_read_data.num_objs
      1663 ± 11%     +40.6%       2340 ±  8%  slabinfo.skbuff_ext_cache.active_objs
      1663 ± 11%     +40.6%       2340 ±  8%  slabinfo.skbuff_ext_cache.num_objs
      1808 ± 18%     +17.6%       2126 ±  6%  slabinfo.skbuff_fclone_cache.active_objs
      1808 ± 18%     +17.6%       2126 ±  6%  slabinfo.skbuff_fclone_cache.num_objs
    937.75 ± 11%     +28.2%       1202        slabinfo.xfrm_state.active_objs
    937.75 ± 11%     +28.2%       1202        slabinfo.xfrm_state.num_objs
    356.25 ± 34%    +209.1%       1101 ± 18%  interrupts.47:PCI-MSI.1572866-edge.eth0-TxRx-2
    514.25 ± 21%     -46.3%     276.00        interrupts.48:PCI-MSI.1572867-edge.eth0-TxRx-3
    349.50 ± 45%    +124.2%     783.50 ±  6%  interrupts.52:PCI-MSI.1572871-edge.eth0-TxRx-7
    388.25 ± 44%     -34.4%     254.50 ±  2%  interrupts.57:PCI-MSI.1572876-edge.eth0-TxRx-12
    259.50 ±  3%    +105.8%     534.00 ± 50%  interrupts.60:PCI-MSI.1572879-edge.eth0-TxRx-15
      4201 ± 67%    -100.0%       1.50 ± 33%  interrupts.CPU102.NMI:Non-maskable_interrupts
      4201 ± 67%    -100.0%       1.50 ± 33%  interrupts.CPU102.PMI:Performance_monitoring_interrupts
      6610 ±  6%      +6.0%       7008 ±  5%  interrupts.CPU104.CAL:Function_call_interrupts
      2142 ±  9%      +8.7%       2328 ±  7%  interrupts.CPU107.TLB:TLB_shootdowns
      3156 ±100%    +110.9%       6658        interrupts.CPU108.NMI:Non-maskable_interrupts
      3156 ±100%    +110.9%       6658        interrupts.CPU108.PMI:Performance_monitoring_interrupts
      2160 ± 10%     +11.1%       2399 ±  5%  interrupts.CPU112.TLB:TLB_shootdowns
      1430 ±150%    +245.3%       4938 ± 33%  interrupts.CPU119.NMI:Non-maskable_interrupts
      1430 ±150%    +245.3%       4938 ± 33%  interrupts.CPU119.PMI:Performance_monitoring_interrupts
    388.25 ± 44%     -34.4%     254.50 ±  2%  interrupts.CPU12.57:PCI-MSI.1572876-edge.eth0-TxRx-12
      1678 ±160%    +304.1%       6781 ±  3%  interrupts.CPU13.NMI:Non-maskable_interrupts
      1678 ±160%    +304.1%       6781 ±  3%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
      2866 ±101%    -100.0%       0.50 ±100%  interrupts.CPU131.NMI:Non-maskable_interrupts
      2866 ±101%    -100.0%       0.50 ±100%  interrupts.CPU131.PMI:Performance_monitoring_interrupts
      1572 ±170%    +314.2%       6515        interrupts.CPU137.NMI:Non-maskable_interrupts
      1572 ±170%    +314.2%       6515        interrupts.CPU137.PMI:Performance_monitoring_interrupts
      3900 ± 65%     +67.4%       6529 ±  2%  interrupts.CPU141.NMI:Non-maskable_interrupts
      3900 ± 65%     +67.4%       6529 ±  2%  interrupts.CPU141.PMI:Performance_monitoring_interrupts
    259.50 ±  3%    +105.8%     534.00 ± 50%  interrupts.CPU15.60:PCI-MSI.1572879-edge.eth0-TxRx-15
     10701 ±  4%      -9.7%       9658        interrupts.CPU15.RES:Rescheduling_interrupts
    356.25 ± 34%    +209.1%       1101 ± 18%  interrupts.CPU2.47:PCI-MSI.1572866-edge.eth0-TxRx-2
      4498 ± 49%    -100.0%       2.00        interrupts.CPU22.NMI:Non-maskable_interrupts
      4498 ± 49%    -100.0%       2.00        interrupts.CPU22.PMI:Performance_monitoring_interrupts
      4182 ± 63%     -98.2%      75.50 ± 90%  interrupts.CPU23.NMI:Non-maskable_interrupts
      4182 ± 63%     -98.2%      75.50 ± 90%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
      2150 ± 12%     +30.9%       2814 ± 23%  interrupts.CPU26.TLB:TLB_shootdowns
    514.25 ± 21%     -46.3%     276.00        interrupts.CPU3.48:PCI-MSI.1572867-edge.eth0-TxRx-3
      2126 ±  8%      +9.1%       2319 ±  5%  interrupts.CPU37.TLB:TLB_shootdowns
      3959 ± 57%     +69.3%       6704        interrupts.CPU42.NMI:Non-maskable_interrupts
      3959 ± 57%     +69.3%       6704        interrupts.CPU42.PMI:Performance_monitoring_interrupts
      4068 ± 34%     -85.2%     603.50 ± 99%  interrupts.CPU47.NMI:Non-maskable_interrupts
      4068 ± 34%     -85.2%     603.50 ± 99%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
      6374 ±  4%      +6.3%       6776 ±  5%  interrupts.CPU48.CAL:Function_call_interrupts
      2782 ± 76%    -100.0%       0.50 ±100%  interrupts.CPU58.NMI:Non-maskable_interrupts
      2782 ± 76%    -100.0%       0.50 ±100%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
      3930 ± 68%     +70.1%       6684        interrupts.CPU60.NMI:Non-maskable_interrupts
      3930 ± 68%     +70.1%       6684        interrupts.CPU60.PMI:Performance_monitoring_interrupts
      2355 ±108%    +110.9%       4966 ± 31%  interrupts.CPU62.NMI:Non-maskable_interrupts
      2355 ±108%    +110.9%       4966 ± 31%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
      3140 ± 70%    +110.4%       6608        interrupts.CPU63.NMI:Non-maskable_interrupts
      3140 ± 70%    +110.4%       6608        interrupts.CPU63.PMI:Performance_monitoring_interrupts
      3356 ± 71%    -100.0%       0.00        interrupts.CPU65.NMI:Non-maskable_interrupts
      3356 ± 71%    -100.0%       0.00        interrupts.CPU65.PMI:Performance_monitoring_interrupts
      2113 ± 11%     +11.1%       2348 ±  6%  interrupts.CPU66.TLB:TLB_shootdowns
    349.50 ± 45%    +124.2%     783.50 ±  6%  interrupts.CPU7.52:PCI-MSI.1572871-edge.eth0-TxRx-7
      6486 ±  3%      +9.3%       7091        interrupts.CPU7.CAL:Function_call_interrupts
      1.25 ±173%  +5.3e+05%       6593        interrupts.CPU7.NMI:Non-maskable_interrupts
      1.25 ±173%  +5.3e+05%       6593        interrupts.CPU7.PMI:Performance_monitoring_interrupts
      2089 ±  9%     +11.5%       2330 ± 11%  interrupts.CPU74.TLB:TLB_shootdowns
      5708 ± 27%     -99.5%      26.00 ± 76%  interrupts.CPU79.NMI:Non-maskable_interrupts
      5708 ± 27%     -99.5%      26.00 ± 76%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
    846.25 ±168%    +316.0%       3520 ± 89%  interrupts.CPU83.NMI:Non-maskable_interrupts
    846.25 ±168%    +316.0%       3520 ± 89%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
      2117 ± 10%     +10.4%       2337 ±  6%  interrupts.CPU84.TLB:TLB_shootdowns
      3002 ± 73%    +122.1%       6667        interrupts.CPU9.NMI:Non-maskable_interrupts
      3002 ± 73%    +122.1%       6667        interrupts.CPU9.PMI:Performance_monitoring_interrupts
      2132 ±  8%      +8.8%       2320 ±  6%  interrupts.CPU93.TLB:TLB_shootdowns
      2002 ±110%    +144.9%       4904 ± 31%  interrupts.CPU94.NMI:Non-maskable_interrupts
      2002 ±110%    +144.9%       4904 ± 31%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
      2099 ±  7%     +13.1%       2375 ±  3%  interrupts.CPU96.TLB:TLB_shootdowns
      1896 ±144%    +257.9%       6786        interrupts.CPU99.NMI:Non-maskable_interrupts
      1896 ±144%    +257.9%       6786        interrupts.CPU99.PMI:Performance_monitoring_interrupts
     17.22 ± 68%     -17.2        0.00        perf-profile.calltrace.cycles-pp.sieve
      4.71 ± 68%      -4.7        0.00        perf-profile.calltrace.cycles-pp.string_rtns_1
      0.13 ±173%      +0.4        0.52 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.setlocale
      0.18 ±173%      +0.5        0.67 ±  9%  perf-profile.calltrace.cycles-pp.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      0.35 ±108%      +0.5        0.85 ± 11%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.19 ±173%      +0.5        0.70 ± 15%  perf-profile.calltrace.cycles-pp.clear_page_erms.prep_new_page.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma
      0.36 ±107%      +0.5        0.88 ± 11%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.20 ±173%      +0.5        0.74 ± 14%  perf-profile.calltrace.cycles-pp.prep_new_page.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page
      0.87 ± 38%      +0.6        1.44 ± 15%  perf-profile.calltrace.cycles-pp.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap.__x64_sys_brk
      0.20 ±173%      +0.6        0.77 ± 10%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      0.88 ± 38%      +0.6        1.46 ± 15%  perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64
      0.41 ±107%      +0.6        0.99 ± 11%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.45 ±106%      +0.6        1.04 ± 13%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page.__handle_mm_fault
      0.20 ±173%      +0.6        0.79 ± 13%  perf-profile.calltrace.cycles-pp.native_flush_tlb_one_user.flush_tlb_func_common.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu
      0.20 ±173%      +0.6        0.80 ± 13%  perf-profile.calltrace.cycles-pp.flush_tlb_func_common.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region
      0.40 ±107%      +0.6        1.00 ±  4%  perf-profile.calltrace.cycles-pp.setlocale
      0.23 ±173%      +0.6        0.84 ± 11%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.46 ±107%      +0.7        1.11 ± 11%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      0.20 ±173%      +0.7        0.86 ± 17%  perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.tlb_flush_mmu.tlb_finish_mmu.unmap_region.__do_munmap
      0.38 ±102%      +0.7        1.06 ±  7%  perf-profile.calltrace.cycles-pp.filemap_map_pages.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
      0.55 ±107%      +0.7        1.27 ± 13%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.alloc_pages_vma.do_anonymous_page.__handle_mm_fault.handle_mm_fault
      1.16 ± 39%      +0.7        1.89 ± 14%  perf-profile.calltrace.cycles-pp.unmap_region.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.59 ±107%      +0.8        1.39 ± 13%  perf-profile.calltrace.cycles-pp.alloc_pages_vma.do_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
      0.58 ±108%      +0.8        1.39 ± 11%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt
      0.54 ±107%      +0.8        1.35 ± 13%  perf-profile.calltrace.cycles-pp.prepare_exit_to_usermode.swapgs_restore_regs_and_return_to_usermode
      0.59 ±108%      +0.8        1.42 ± 11%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt
      0.64 ±110%      +1.0        1.65 ± 12%  perf-profile.calltrace.cycles-pp.do_brk_flags.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.73 ±107%      +1.1        1.82 ± 13%  perf-profile.calltrace.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      1.57 ± 51%      +1.1        2.69 ± 13%  perf-profile.calltrace.cycles-pp.do_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
      0.80 ±110%      +1.3        2.06 ± 14%  perf-profile.calltrace.cycles-pp.__do_munmap.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.92 ± 37%      +2.2        5.11 ± 17%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      1.59 ±110%      +2.5        4.06 ± 13%  perf-profile.calltrace.cycles-pp.__x64_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.56 ± 62%      +2.8        5.37 ± 17%  perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      2.96 ± 63%      +3.2        6.16 ± 16%  perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
      3.06 ± 63%      +3.3        6.33 ± 16%  perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
      3.10 ± 63%      +3.3        6.41 ± 16%  perf-profile.calltrace.cycles-pp.page_fault
     15.46 ± 36%     +16.9       32.39 ± 13%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     16.01 ± 35%     +17.2       33.22 ± 13%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     16.04 ± 35%     +17.2       33.26 ± 13%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     16.41 ± 34%     +17.4       33.85 ± 13%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     16.41 ± 34%     +17.4       33.85 ± 13%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     16.41 ± 34%     +17.4       33.85 ± 13%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     16.53 ± 34%     +17.5       34.08 ± 12%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     17.23 ± 68%     -17.2        0.00        perf-profile.children.cycles-pp.sieve
      4.75 ± 68%      -4.7        0.00        perf-profile.children.cycles-pp.string_rtns_1
      0.34 ± 24%      -0.1        0.22        perf-profile.children.cycles-pp.__xstat64
      0.08 ± 10%      +0.0        0.10 ± 10%  perf-profile.children.cycles-pp.perf_mux_hrtimer_handler
      0.07 ± 34%      +0.0        0.10 ± 14%  perf-profile.children.cycles-pp.mem_cgroup_uncharge_list
      0.11 ± 27%      +0.0        0.14 ± 14%  perf-profile.children.cycles-pp.ksys_read
      0.05 ± 63%      +0.0        0.09 ± 11%  perf-profile.children.cycles-pp.mem_cgroup_charge_statistics
      0.09 ± 33%      +0.0        0.13 ± 15%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.09 ± 26%      +0.0        0.13 ±  7%  perf-profile.children.cycles-pp.new_sync_read
      0.03 ±100%      +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.__update_load_avg_se
      0.11 ± 30%      +0.0        0.15 ± 16%  perf-profile.children.cycles-pp.trailing_symlink
      0.03 ±102%      +0.0        0.07 ± 14%  perf-profile.children.cycles-pp.page_get_link
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.schedule_tail
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.arch_dup_task_struct
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp.migration_cpu_stop
      0.10 ± 18%      +0.0        0.14 ±  7%  perf-profile.children.cycles-pp.clockevents_program_event
      0.03 ±100%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.free_pgd_range
      0.05 ± 64%      +0.0        0.10 ±  5%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.04 ±102%      +0.0        0.08 ± 12%  perf-profile.children.cycles-pp.up_read
      0.01 ±173%      +0.0        0.06 ± 16%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.01 ±173%      +0.0        0.06 ± 16%  perf-profile.children.cycles-pp.cap_mmap_addr
      0.01 ±173%      +0.0        0.06 ± 16%  perf-profile.children.cycles-pp.d_alloc
      0.01 ±173%      +0.0        0.06 ± 16%  perf-profile.children.cycles-pp.shmem_getpage_gfp
      0.03 ±102%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.avc_has_perm
      0.11 ± 28%      +0.0        0.15 ± 16%  perf-profile.children.cycles-pp.new_sync_write
      0.08 ± 44%      +0.0        0.13 ± 15%  perf-profile.children.cycles-pp.vma_compute_subtree_gap
      0.05 ± 67%      +0.0        0.10 ± 10%  perf-profile.children.cycles-pp.strlcpy
      0.02 ±173%      +0.0        0.07 ± 23%  perf-profile.children.cycles-pp.inode_permission
      0.01 ±173%      +0.0        0.06 ± 16%  perf-profile.children.cycles-pp._vm_normal_page
      0.01 ±173%      +0.0        0.06 ± 16%  perf-profile.children.cycles-pp.__getpid
      0.01 ±173%      +0.0        0.06 ± 16%  perf-profile.children.cycles-pp.security_mmap_file
      0.01 ±173%      +0.0        0.06 ± 16%  perf-profile.children.cycles-pp.d_path
      0.01 ±173%      +0.0        0.06        perf-profile.children.cycles-pp.unmap_single_vma
      0.04 ±103%      +0.0        0.09 ± 17%  perf-profile.children.cycles-pp.pgd_alloc
      0.03 ±100%      +0.0        0.08 ±  6%  perf-profile.children.cycles-pp.__vsnprintf_chk
      0.11 ± 30%      +0.0        0.16 ±  9%  perf-profile.children.cycles-pp.mem_cgroup_commit_charge
      0.12 ± 26%      +0.0        0.17 ± 14%  perf-profile.children.cycles-pp.vfs_write
      0.06 ± 67%      +0.1        0.11 ± 18%  perf-profile.children.cycles-pp.sync_regs
      0.03 ±102%      +0.1        0.08 ± 12%  perf-profile.children.cycles-pp.__wake_up_common
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.wake_q_add
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.cpumask_next_and
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.vma_interval_tree_augment_rotate
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.find_next_and_bit
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.fopen
      0.13 ± 29%      +0.1        0.18 ± 11%  perf-profile.children.cycles-pp.ksys_write
      0.07 ± 39%      +0.1        0.12 ±  4%  perf-profile.children.cycles-pp.vm_area_dup
      0.07 ± 64%      +0.1        0.12 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock_irq
      0.03 ±105%      +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.d_alloc_parallel
      0.03 ±100%      +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.04 ±103%      +0.1        0.09 ± 11%  perf-profile.children.cycles-pp.__tlb_remove_page_size
      0.08 ± 34%      +0.1        0.13 ± 30%  perf-profile.children.cycles-pp.__count_memcg_events
      0.08 ± 42%      +0.1        0.13 ±  7%  perf-profile.children.cycles-pp.create_elf_tables
      0.08 ± 42%      +0.1        0.13 ±  7%  perf-profile.children.cycles-pp.copy_strings_kernel
      0.02 ±173%      +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.__anon_vma_prepare
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.children.cycles-pp.generic_file_read_iter
      0.04 ±102%      +0.1        0.09 ± 11%  perf-profile.children.cycles-pp.__list_add_valid
      0.03 ±102%      +0.1        0.08 ±  5%  perf-profile.children.cycles-pp.osq_unlock
      0.06 ± 71%      +0.1        0.11 ±  9%  perf-profile.children.cycles-pp.__pmd_alloc
      0.04 ±101%      +0.1        0.10 ±  5%  perf-profile.children.cycles-pp.__check_object_size
      0.00            +0.1        0.06 ±  9%  perf-profile.children.cycles-pp.__cxa_atexit
      0.01 ±173%      +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.kernel_read
      0.04 ± 59%      +0.1        0.10 ± 19%  perf-profile.children.cycles-pp.__vmalloc_node_range
      0.04 ± 63%      +0.1        0.10 ± 30%  perf-profile.children.cycles-pp.generic_perform_write
      0.05 ±102%      +0.1        0.10 ± 14%  perf-profile.children.cycles-pp.mm_init
      0.03 ±100%      +0.1        0.09 ± 11%  perf-profile.children.cycles-pp.mem_cgroup_throttle_swaprate
      0.08 ± 37%      +0.1        0.14        perf-profile.children.cycles-pp.do_open_execat
      0.02 ±173%      +0.1        0.08 ±  6%  perf-profile.children.cycles-pp._IO_file_open
      0.01 ±173%      +0.1        0.07 ± 14%  perf-profile.children.cycles-pp.__ctype_init
      0.10 ± 39%      +0.1        0.16 ±  9%  perf-profile.children.cycles-pp.__might_sleep
      0.09 ± 37%      +0.1        0.15 ± 10%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      0.06 ± 60%      +0.1        0.11 ± 30%  perf-profile.children.cycles-pp.generic_file_write_iter
      0.04 ±107%      +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.do_dentry_open
      0.05 ± 64%      +0.1        0.11 ± 36%  perf-profile.children.cycles-pp.worker_thread
      0.03 ±100%      +0.1        0.09 ± 17%  perf-profile.children.cycles-pp.xas_find
      0.07 ± 39%      +0.1        0.14 ±  3%  perf-profile.children.cycles-pp.__alloc_file
      0.06 ± 70%      +0.1        0.12 ±  4%  perf-profile.children.cycles-pp.setup_arg_pages
      0.04 ±108%      +0.1        0.10 ± 14%  perf-profile.children.cycles-pp.lookup_slow
      0.03 ±105%      +0.1        0.10 ± 15%  perf-profile.children.cycles-pp.unwind_next_frame
      0.03 ±100%      +0.1        0.09 ± 11%  perf-profile.children.cycles-pp.xas_load
      0.02 ±173%      +0.1        0.08        perf-profile.children.cycles-pp._copy_from_user
      0.08 ± 34%      +0.1        0.14 ±  7%  perf-profile.children.cycles-pp.__rb_insert_augmented
      0.05 ± 62%      +0.1        0.11 ± 27%  perf-profile.children.cycles-pp.__generic_file_write_iter
      0.04 ±106%      +0.1        0.11 ±  4%  perf-profile.children.cycles-pp.change_protection
      0.08 ± 45%      +0.1        0.15 ±  3%  perf-profile.children.cycles-pp.alloc_empty_file
      0.07 ± 30%      +0.1        0.14 ±  7%  perf-profile.children.cycles-pp.update_rq_clock
      0.10 ± 42%      +0.1        0.16 ±  9%  perf-profile.children.cycles-pp.anon_vma_interval_tree_remove
      0.12 ± 40%      +0.1        0.19 ± 10%  perf-profile.children.cycles-pp.selinux_inode_permission
      0.12 ± 40%      +0.1        0.18 ± 13%  perf-profile.children.cycles-pp.__get_free_pages
      0.12 ± 43%      +0.1        0.18 ± 13%  perf-profile.children.cycles-pp.__pte_alloc
      0.13 ± 32%      +0.1        0.20 ± 15%  perf-profile.children.cycles-pp.vfs_read
      0.12 ± 30%      +0.1        0.18 ±  2%  perf-profile.children.cycles-pp.copy_user_highpage
      0.09 ± 39%      +0.1        0.16 ±  6%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.09 ± 33%      +0.1        0.16        perf-profile.children.cycles-pp.time
      0.06 ± 70%      +0.1        0.13        perf-profile.children.cycles-pp.dequeue_entity
      0.04 ±108%      +0.1        0.11 ±  9%  perf-profile.children.cycles-pp.release_task
      0.12 ± 33%      +0.1        0.18 ±  2%  perf-profile.children.cycles-pp.copy_page
      0.12 ± 38%      +0.1        0.19 ± 21%  perf-profile.children.cycles-pp.avc_has_perm_noaudit
      0.04 ±102%      +0.1        0.10 ± 14%  perf-profile.children.cycles-pp.switch_mm_irqs_off
      0.04 ±104%      +0.1        0.11 ±  4%  perf-profile.children.cycles-pp.unlock_page
      0.09 ± 40%      +0.1        0.16 ± 12%  perf-profile.children.cycles-pp.__d_lookup_rcu
      0.04 ±106%      +0.1        0.11 ±  9%  perf-profile.children.cycles-pp.rwsem_mark_wake
      0.04 ±107%      +0.1        0.11 ±  4%  perf-profile.children.cycles-pp.__strcasecmp
      0.13 ± 36%      +0.1        0.21 ± 12%  perf-profile.children.cycles-pp.security_inode_permission
      0.11 ± 38%      +0.1        0.18 ±  8%  perf-profile.children.cycles-pp.do_faccessat
      0.10 ± 47%      +0.1        0.17 ±  8%  perf-profile.children.cycles-pp.get_user_pages_remote
      0.04 ±100%      +0.1        0.11 ±  4%  perf-profile.children.cycles-pp.update_load_avg
      0.12 ± 35%      +0.1        0.20 ±  7%  perf-profile.children.cycles-pp.update_curr
      0.12 ± 40%      +0.1        0.20 ±  2%  perf-profile.children.cycles-pp.__slab_free
      0.09 ± 40%      +0.1        0.17 ±  3%  perf-profile.children.cycles-pp.do_task_dead
      0.13 ± 44%      +0.1        0.21 ± 14%  perf-profile.children.cycles-pp.wake_up_new_task
      0.04 ±108%      +0.1        0.12        perf-profile.children.cycles-pp.finish_task_switch
      0.09 ± 37%      +0.1        0.17 ± 15%  perf-profile.children.cycles-pp.page_add_file_rmap
      0.07 ± 39%      +0.1        0.15 ± 13%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.16 ± 34%      +0.1        0.24 ± 12%  perf-profile.children.cycles-pp.mem_cgroup_try_charge
      0.12 ± 37%      +0.1        0.21 ± 21%  perf-profile.children.cycles-pp.cred_has_capability
      0.14 ± 38%      +0.1        0.22 ±  6%  perf-profile.children.cycles-pp.anon_vma_interval_tree_insert
      0.06 ± 68%      +0.1        0.14 ± 14%  perf-profile.children.cycles-pp.arch_stack_walk
      0.10 ± 44%      +0.1        0.19 ±  5%  perf-profile.children.cycles-pp.remove_vma
      0.09 ± 41%      +0.1        0.17 ± 14%  perf-profile.children.cycles-pp.__account_scheduler_latency
      0.06 ± 77%      +0.1        0.15 ± 13%  perf-profile.children.cycles-pp.refcount_dec_not_one
      0.06 ± 77%      +0.1        0.15 ± 13%  perf-profile.children.cycles-pp.free_uid
      0.06 ± 77%      +0.1        0.15 ± 13%  perf-profile.children.cycles-pp.refcount_dec_and_lock_irqsave
      0.05 ±108%      +0.1        0.15 ± 10%  perf-profile.children.cycles-pp.wait_consider_task
      0.15 ± 36%      +0.1        0.25 ± 24%  perf-profile.children.cycles-pp.selinux_vm_enough_memory
      0.15 ± 38%      +0.1        0.24 ± 10%  perf-profile.children.cycles-pp.lookup_fast
      0.06 ± 70%      +0.1        0.16 ± 12%  perf-profile.children.cycles-pp.sched_ttwu_pending
      0.05 ±106%      +0.1        0.15 ± 10%  perf-profile.children.cycles-pp.queued_write_lock_slowpath
      0.15 ± 42%      +0.1        0.26 ±  7%  perf-profile.children.cycles-pp.___might_sleep
      0.04 ±103%      +0.1        0.15 ±  6%  perf-profile.children.cycles-pp.idle_cpu
      0.19 ± 41%      +0.1        0.30 ±  6%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.15 ± 40%      +0.1        0.26 ±  3%  perf-profile.children.cycles-pp.free_pages_and_swap_cache
      0.17 ± 37%      +0.1        0.28 ±  7%  perf-profile.children.cycles-pp.up_write
      0.23 ± 36%      +0.1        0.34 ±  7%  perf-profile.children.cycles-pp.get_unmapped_area
      0.22 ± 34%      +0.1        0.34 ± 13%  perf-profile.children.cycles-pp.mem_cgroup_try_charge_delay
      0.20 ± 38%      +0.1        0.32 ±  9%  perf-profile.children.cycles-pp.alloc_set_pte
      0.14 ± 44%      +0.1        0.27 ±  3%  perf-profile.children.cycles-pp.wait4
      0.29 ± 35%      +0.1        0.42 ±  8%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.18 ± 38%      +0.1        0.32 ±  7%  perf-profile.children.cycles-pp.kmem_cache_free
      0.06 ± 65%      +0.1        0.19        perf-profile.children.cycles-pp.schedule_idle
      0.24 ± 37%      +0.1        0.38 ± 14%  perf-profile.children.cycles-pp.security_vm_enough_memory_mm
      0.21 ± 38%      +0.1        0.35 ±  9%  perf-profile.children.cycles-pp.walk_component
      0.25 ± 34%      +0.1        0.39 ±  5%  perf-profile.children.cycles-pp.wp_page_copy
      0.13 ± 46%      +0.1        0.27 ± 11%  perf-profile.children.cycles-pp.put_cred_rcu
      0.27 ± 40%      +0.1        0.41 ±  9%  perf-profile.children.cycles-pp.find_vma
      0.27 ± 40%      +0.1        0.41 ±  9%  perf-profile.children.cycles-pp.unlink_anon_vmas
      0.14 ± 39%      +0.1        0.29 ± 12%  perf-profile.children.cycles-pp.ttwu_do_activate
      0.26 ± 43%      +0.2        0.41 ± 11%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.15 ± 40%      +0.2        0.31 ±  8%  perf-profile.children.cycles-pp.enqueue_entity
      0.32 ± 36%      +0.2        0.47 ± 10%  perf-profile.children.cycles-pp.vma_merge
      0.23 ± 38%      +0.2        0.38 ±  5%  perf-profile.children.cycles-pp.kthread
      0.35 ± 35%      +0.2        0.51 ± 10%  perf-profile.children.cycles-pp.__lru_cache_add
      0.26 ± 44%      +0.2        0.42 ±  8%  perf-profile.children.cycles-pp.perf_iterate_sb
      0.17 ± 39%      +0.2        0.33 ±  9%  perf-profile.children.cycles-pp.activate_task
      0.30 ± 40%      +0.2        0.46 ± 11%  perf-profile.children.cycles-pp.link_path_walk
      0.18 ± 38%      +0.2        0.35 ±  9%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.26 ± 44%      +0.2        0.44 ±  6%  perf-profile.children.cycles-pp.copy_strings
      0.26 ± 44%      +0.2        0.44 ±  9%  perf-profile.children.cycles-pp._dl_addr
      0.26 ± 39%      +0.2        0.45 ±  2%  perf-profile.children.cycles-pp.ret_from_fork
      0.23 ± 49%      +0.2        0.43        perf-profile.children.cycles-pp.do_wait
      0.38 ± 32%      +0.2        0.58 ±  9%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.46 ± 38%      +0.2        0.67 ± 14%  perf-profile.children.cycles-pp.path_lookupat
      0.23 ± 51%      +0.2        0.45        perf-profile.children.cycles-pp.kernel_wait4
      0.47 ± 37%      +0.2        0.69 ± 15%  perf-profile.children.cycles-pp.filename_lookup
      0.26 ± 41%      +0.2        0.47 ±  7%  perf-profile.children.cycles-pp.rcu_core
      0.23 ± 51%      +0.2        0.45 ±  2%  perf-profile.children.cycles-pp.__do_sys_wait4
      0.47 ± 36%      +0.2        0.69 ±  9%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.29 ± 43%      +0.2        0.52 ±  4%  perf-profile.children.cycles-pp.vma_interval_tree_insert
      0.39 ± 38%      +0.2        0.62 ±  5%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.43 ± 35%      +0.3        0.71 ± 12%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.45 ± 31%      +0.3        0.73 ± 10%  perf-profile.children.cycles-pp.do_sys_open
      0.33 ± 40%      +0.3        0.61 ±  8%  perf-profile.children.cycles-pp.copy_p4d_range
      0.46 ± 31%      +0.3        0.75 ± 10%  perf-profile.children.cycles-pp.do_filp_open
      0.46 ± 31%      +0.3        0.74 ± 11%  perf-profile.children.cycles-pp.path_openat
      0.48 ± 40%      +0.3        0.76 ±  8%  perf-profile.children.cycles-pp.perf_event_mmap
      0.34 ± 40%      +0.3        0.62 ±  8%  perf-profile.children.cycles-pp.copy_page_range
      0.41 ± 25%      +0.3        0.71 ± 10%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.72 ± 36%      +0.3        1.02 ± 12%  perf-profile.children.cycles-pp.clear_page_erms
      0.44 ± 23%      +0.3        0.76 ±  9%  perf-profile.children.cycles-pp.irq_exit
      0.57 ± 36%      +0.3        0.89 ± 12%  perf-profile.children.cycles-pp.native_flush_tlb_one_user
      0.46 ± 42%      +0.3        0.79 ±  3%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.59 ± 35%      +0.3        0.92 ± 13%  perf-profile.children.cycles-pp.flush_tlb_func_common
      0.75 ± 35%      +0.3        1.08 ± 12%  perf-profile.children.cycles-pp.prep_new_page
      0.35 ± 39%      +0.4        0.71        perf-profile.children.cycles-pp.update_sd_lb_stats
      0.84 ± 33%      +0.4        1.21 ±  6%  perf-profile.children.cycles-pp.scheduler_tick
      0.36 ± 40%      +0.4        0.73        perf-profile.children.cycles-pp.find_busiest_group
      0.64 ± 36%      +0.4        1.02 ± 16%  perf-profile.children.cycles-pp.flush_tlb_mm_range
      0.58 ± 45%      +0.4        1.00 ±  4%  perf-profile.children.cycles-pp.setlocale
      0.97 ± 30%      +0.4        1.40 ±  6%  perf-profile.children.cycles-pp.tick_sched_handle
      0.96 ± 31%      +0.4        1.39 ±  6%  perf-profile.children.cycles-pp.update_process_times
      0.46 ± 39%      +0.4        0.89        perf-profile.children.cycles-pp.load_balance
      1.01 ± 30%      +0.4        1.46 ±  6%  perf-profile.children.cycles-pp.tick_sched_timer
      1.07 ± 35%      +0.5        1.53 ± 12%  perf-profile.children.cycles-pp.alloc_pages_vma
      0.49 ± 42%      +0.5        0.96 ±  3%  perf-profile.children.cycles-pp.pick_next_task_fair
      1.21 ± 26%      +0.5        1.70 ±  6%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.55 ± 40%      +0.5        1.04 ±  5%  perf-profile.children.cycles-pp.schedule
      1.11 ± 37%      +0.5        1.60 ± 11%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.93 ± 35%      +0.5        1.43 ± 13%  perf-profile.children.cycles-pp.prepare_exit_to_usermode
      0.32 ± 34%      +0.5        0.84 ± 46%  perf-profile.children.cycles-pp.do_wp_page
      1.42 ± 25%      +0.5        1.96 ±  4%  perf-profile.children.cycles-pp.hrtimer_interrupt
      1.31 ± 37%      +0.6        1.90 ± 11%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      1.11 ± 38%      +0.6        1.71 ± 11%  perf-profile.children.cycles-pp.do_brk_flags
      0.61 ± 43%      +0.6        1.23        perf-profile.children.cycles-pp.rwsem_wake
      0.55 ± 45%      +0.6        1.20        perf-profile.children.cycles-pp.wake_up_q
      1.00 ± 44%      +0.7        1.66 ±  8%  perf-profile.children.cycles-pp.unmap_page_range
      0.61 ± 44%      +0.7        1.28 ±  3%  perf-profile.children.cycles-pp.try_to_wake_up
      1.26 ± 36%      +0.7        1.93 ± 13%  perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      1.06 ± 44%      +0.7        1.75 ±  8%  perf-profile.children.cycles-pp.unmap_vmas
      0.73 ± 40%      +0.7        1.43 ±  4%  perf-profile.children.cycles-pp.__schedule
      0.63 ± 41%      +0.8        1.45 ± 25%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.76 ± 50%      +0.8        1.59 ± 19%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      1.95 ± 23%      +0.9        2.80        perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      2.04 ± 22%      +0.9        2.90        perf-profile.children.cycles-pp.apic_timer_interrupt
      1.94 ± 35%      +0.9        2.81 ± 12%  perf-profile.children.cycles-pp.do_anonymous_page
      1.58 ± 40%      +0.9        2.45 ± 10%  perf-profile.children.cycles-pp.tlb_flush_mmu
      1.05 ± 34%      +0.9        1.93 ±  7%  perf-profile.children.cycles-pp.filemap_map_pages
      1.59 ± 41%      +0.9        2.49 ± 10%  perf-profile.children.cycles-pp.tlb_finish_mmu
      2.75 ± 36%      +1.4        4.12 ± 13%  perf-profile.children.cycles-pp.__x64_sys_brk
      3.85 ± 35%      +2.6        6.48 ± 14%  perf-profile.children.cycles-pp.__handle_mm_fault
      4.04 ± 35%      +2.7        6.79 ± 15%  perf-profile.children.cycles-pp.handle_mm_fault
      4.97 ± 31%      +2.9        7.88 ± 14%  perf-profile.children.cycles-pp.page_fault
      4.55 ± 35%      +3.0        7.55 ± 14%  perf-profile.children.cycles-pp.__do_page_fault
      4.70 ± 35%      +3.0        7.74 ± 14%  perf-profile.children.cycles-pp.do_page_fault
     15.57 ± 37%     +17.0       32.62 ± 12%  perf-profile.children.cycles-pp.intel_idle
     16.16 ± 35%     +17.3       33.48 ± 12%  perf-profile.children.cycles-pp.cpuidle_enter_state
     16.16 ± 35%     +17.3       33.48 ± 12%  perf-profile.children.cycles-pp.cpuidle_enter
     16.41 ± 34%     +17.4       33.85 ± 13%  perf-profile.children.cycles-pp.start_secondary
     16.53 ± 34%     +17.5       34.08 ± 12%  perf-profile.children.cycles-pp.do_idle
     16.53 ± 34%     +17.5       34.08 ± 12%  perf-profile.children.cycles-pp.secondary_startup_64
     16.53 ± 34%     +17.5       34.08 ± 12%  perf-profile.children.cycles-pp.cpu_startup_entry
     17.02 ± 68%     -17.0        0.00        perf-profile.self.cycles-pp.sieve
      4.65 ± 68%      -4.7        0.00        perf-profile.self.cycles-pp.string_rtns_1
      0.07 ± 34%      +0.0        0.10 ± 14%  perf-profile.self.cycles-pp.do_brk_flags
      0.07 ± 34%      +0.0        0.10 ± 14%  perf-profile.self.cycles-pp.selinux_inode_permission
      0.03 ±102%      +0.0        0.07 ± 14%  perf-profile.self.cycles-pp.__mod_node_page_state
      0.08 ± 34%      +0.0        0.12 ±  8%  perf-profile.self.cycles-pp.update_curr
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.alloc_pages_vma
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.prep_new_page
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.anon_vma_fork
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.unwind_next_frame
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.cap_mmap_addr
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.unmap_single_vma
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.queued_write_lock_slowpath
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.cap_vm_enough_memory
      0.03 ±100%      +0.0        0.07 ± 14%  perf-profile.self.cycles-pp.mem_cgroup_throttle_swaprate
      0.07 ± 34%      +0.0        0.11 ±  4%  perf-profile.self.cycles-pp.perf_event_mmap_output
      0.01 ±173%      +0.0        0.06 ± 16%  perf-profile.self.cycles-pp.do_wp_page
      0.01 ±173%      +0.0        0.06 ± 16%  perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.08 ± 34%      +0.0        0.12 ± 12%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.03 ±105%      +0.0        0.08 ± 12%  perf-profile.self.cycles-pp.__list_add_valid
      0.07 ± 26%      +0.0        0.12 ±  8%  perf-profile.self.cycles-pp.update_rq_clock
      0.06 ± 66%      +0.0        0.10 ± 14%  perf-profile.self.cycles-pp.sync_regs
      0.03 ±102%      +0.1        0.08 ± 12%  perf-profile.self.cycles-pp.mem_cgroup_try_charge
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.wake_q_add
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.wake_up_q
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.__lru_cache_add
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.vma_interval_tree_augment_rotate
      0.03 ±105%      +0.1        0.09 ± 17%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.03 ±100%      +0.1        0.08        perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
      0.04 ±102%      +0.1        0.09 ± 11%  perf-profile.self.cycles-pp.syscall_slow_exit_work
      0.03 ±102%      +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.__schedule
      0.04 ±101%      +0.1        0.10 ± 15%  perf-profile.self.cycles-pp.anon_vma_clone
      0.09 ± 41%      +0.1        0.14 ±  7%  perf-profile.self.cycles-pp.__might_sleep
      0.07 ± 72%      +0.1        0.12 ± 16%  perf-profile.self.cycles-pp.vma_compute_subtree_gap
      0.01 ±173%      +0.1        0.07 ± 14%  perf-profile.self.cycles-pp._cond_resched
      0.01 ±173%      +0.1        0.07        perf-profile.self.cycles-pp.dup_mm
      0.03 ±100%      +0.1        0.09        perf-profile.self.cycles-pp.vma_merge
      0.03 ±100%      +0.1        0.08 ±  5%  perf-profile.self.cycles-pp.osq_unlock
      0.07 ± 34%      +0.1        0.13 ± 30%  perf-profile.self.cycles-pp.__count_memcg_events
      0.01 ±173%      +0.1        0.07 ± 14%  perf-profile.self.cycles-pp.switch_mm_irqs_off
      0.11 ± 35%      +0.1        0.16 ±  9%  perf-profile.self.cycles-pp.__vma_adjust
      0.00            +0.1        0.06        perf-profile.self.cycles-pp.load_balance
      0.11 ± 43%      +0.1        0.17 ± 17%  perf-profile.self.cycles-pp.handle_mm_fault
      0.09 ± 43%      +0.1        0.15 ± 13%  perf-profile.self.cycles-pp.anon_vma_interval_tree_remove
      0.03 ±105%      +0.1        0.10 ±  5%  perf-profile.self.cycles-pp.put_cred_rcu
      0.02 ±173%      +0.1        0.08        perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.04 ±106%      +0.1        0.11 ±  4%  perf-profile.self.cycles-pp.unlock_page
      0.01 ±173%      +0.1        0.08        perf-profile.self.cycles-pp.__alloc_file
      0.09 ± 40%      +0.1        0.16 ±  9%  perf-profile.self.cycles-pp.__d_lookup_rcu
      0.12 ± 33%      +0.1        0.18 ±  2%  perf-profile.self.cycles-pp.copy_page
      0.12 ± 40%      +0.1        0.19 ±  5%  perf-profile.self.cycles-pp.__slab_free
      0.07 ± 34%      +0.1        0.14 ± 14%  perf-profile.self.cycles-pp.page_add_file_rmap
      0.06 ± 67%      +0.1        0.13 ±  7%  perf-profile.self.cycles-pp.__rb_insert_augmented
      0.11 ± 38%      +0.1        0.18 ± 18%  perf-profile.self.cycles-pp.avc_has_perm_noaudit
      0.05 ± 67%      +0.1        0.12 ± 12%  perf-profile.self.cycles-pp.kmem_cache_free
      0.17 ± 35%      +0.1        0.24 ± 10%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.12 ± 42%      +0.1        0.21 ± 12%  perf-profile.self.cycles-pp.perf_iterate_sb
      0.13 ± 40%      +0.1        0.21 ± 14%  perf-profile.self.cycles-pp.find_vma
      0.14 ± 38%      +0.1        0.22 ±  6%  perf-profile.self.cycles-pp.anon_vma_interval_tree_insert
      0.11 ± 41%      +0.1        0.19 ±  5%  perf-profile.self.cycles-pp.vma_interval_tree_remove
      0.15 ± 38%      +0.1        0.24 ± 14%  perf-profile.self.cycles-pp.__do_page_fault
      0.06 ± 77%      +0.1        0.15 ± 13%  perf-profile.self.cycles-pp.refcount_dec_not_one
      0.15 ± 40%      +0.1        0.24 ± 10%  perf-profile.self.cycles-pp.___might_sleep
      0.18 ± 42%      +0.1        0.28 ± 12%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.14 ± 41%      +0.1        0.24 ±  4%  perf-profile.self.cycles-pp.free_pages_and_swap_cache
      0.04 ±101%      +0.1        0.15 ±  3%  perf-profile.self.cycles-pp.idle_cpu
      0.14 ± 34%      +0.1        0.24 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.16 ± 35%      +0.1        0.27 ±  7%  perf-profile.self.cycles-pp.up_write
      0.15 ± 36%      +0.1        0.28 ±  3%  perf-profile.self.cycles-pp.down_write
      0.11 ± 42%      +0.1        0.23 ±  2%  perf-profile.self.cycles-pp.try_to_wake_up
      0.32 ± 36%      +0.2        0.49 ± 13%  perf-profile.self.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.24 ± 34%      +0.2        0.42 ± 10%  perf-profile.self.cycles-pp.__handle_mm_fault
      0.25 ± 45%      +0.2        0.43 ±  9%  perf-profile.self.cycles-pp._dl_addr
      0.17 ± 41%      +0.2        0.35 ±  5%  perf-profile.self.cycles-pp.copy_p4d_range
      0.31 ± 36%      +0.2        0.52        perf-profile.self.cycles-pp._raw_spin_lock
      0.35 ± 37%      +0.2        0.57 ±  9%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.39 ± 38%      +0.2        0.62 ±  6%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.28 ± 44%      +0.2        0.51 ±  3%  perf-profile.self.cycles-pp.vma_interval_tree_insert
      0.27 ± 42%      +0.3        0.54        perf-profile.self.cycles-pp.update_sd_lb_stats
      0.43 ± 34%      +0.3        0.70 ± 11%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.36 ± 42%      +0.3        0.63 ±  3%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.70 ± 36%      +0.3        1.01 ± 12%  perf-profile.self.cycles-pp.clear_page_erms
      0.57 ± 35%      +0.3        0.89 ± 12%  perf-profile.self.cycles-pp.native_flush_tlb_one_user
      0.62 ± 35%      +0.4        0.98 ±  5%  perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.56 ± 45%      +0.4        0.94 ±  8%  perf-profile.self.cycles-pp.unmap_page_range
      0.89 ± 36%      +0.5        1.38 ± 12%  perf-profile.self.cycles-pp.prepare_exit_to_usermode
      1.07 ± 39%      +0.6        1.72 ± 10%  perf-profile.self.cycles-pp.do_syscall_64
      0.74 ± 33%      +0.7        1.40 ±  6%  perf-profile.self.cycles-pp.filemap_map_pages
      0.75 ± 50%      +0.8        1.59 ± 19%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
     15.56 ± 37%     +17.0       32.59 ± 12%  perf-profile.self.cycles-pp.intel_idle



***************************************************************************************************
lkp-bdw-ep2: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/2000/debian-x86_64-2019-05-14.cgz/lkp-bdw-ep2/fork_test/aim7/0xb000036

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          0:4           -7%           0:2     perf-profile.children.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
    330032 ±  5%      +8.1%     356656 ±  5%  meminfo.DirectMap4k
      0.00 ±173%   +5700.0%       0.15 ± 51%  turbostat.Pkg%pc6
     20.03            +1.4%      20.32        turbostat.RAMWatt
      1552 ±106%    +185.7%       4436        numa-vmstat.node0.nr_inactive_anon
      1552 ±106%    +185.7%       4436        numa-vmstat.node0.nr_zone_inactive_anon
     15713 ± 57%     -60.7%       6171        numa-vmstat.node1.nr_page_table_pages
      1703 ±  8%      +9.7%       1869 ±  5%  slabinfo.dmaengine-unmap-16.active_objs
      1703 ±  8%      +9.7%       1869 ±  5%  slabinfo.dmaengine-unmap-16.num_objs
      7189            +8.8%       7820 ±  6%  slabinfo.files_cache.active_objs
      7189            +8.8%       7820 ±  6%  slabinfo.files_cache.num_objs
      0.51 ±  7%      +0.1        0.56        perf-stat.i.branch-miss-rate%
     15.18            +0.3       15.49        perf-stat.i.cache-miss-rate%
  36584641 ±  7%      +5.9%   38751965        perf-stat.i.dTLB-load-misses
     15.37            +0.3       15.69        perf-stat.overall.cache-miss-rate%
  36528779 ±  7%      +5.9%   38692724        perf-stat.ps.dTLB-load-misses
    121293            +1.8%     123486        proc-vmstat.nr_active_anon
     81294            +1.9%      82802        proc-vmstat.nr_anon_pages
     31379            -7.4%      29054 ±  3%  proc-vmstat.nr_page_table_pages
    121293            +1.8%     123486        proc-vmstat.nr_zone_active_anon
      1732 ±130%     -98.2%      31.00 ±100%  proc-vmstat.numa_hint_faults_local
      6396 ±104%    +179.6%      17884        numa-meminfo.node0.Inactive
      6239 ±105%    +185.4%      17806 ±  2%  numa-meminfo.node0.Inactive(anon)
     12314 ± 16%     +29.6%      15963        numa-meminfo.node0.Mapped
     91839 ± 25%     +28.3%     117810 ± 10%  numa-meminfo.node1.AnonHugePages
     14708 ± 12%     -23.6%      11243 ±  2%  numa-meminfo.node1.Mapped
     62843 ± 57%     -60.7%      24674        numa-meminfo.node1.PageTables
   8540823 ± 15%     -58.5%    3542528 ± 79%  sched_debug.cfs_rq:/.MIN_vruntime.max
    960.57 ±  5%     -17.2%     795.67 ±  3%  sched_debug.cfs_rq:/.exec_clock.stddev
      4.34 ±  6%      +9.3%       4.75 ±  3%  sched_debug.cfs_rq:/.load_avg.min
   8540823 ± 15%     -58.5%    3542528 ± 79%  sched_debug.cfs_rq:/.max_vruntime.max
    196336 ± 30%     +74.9%     343381 ±  7%  sched_debug.cfs_rq:/.min_vruntime.stddev
    170.62 ± 19%     +32.7%     226.38        sched_debug.cfs_rq:/.removed.util_avg.max
   -218268          -209.2%     238323 ± 11%  sched_debug.cfs_rq:/.spread0.avg
    213438 ± 59%    +314.5%     884735        sched_debug.cfs_rq:/.spread0.max
   -650962           -44.1%    -363711        sched_debug.cfs_rq:/.spread0.min
    196606 ± 30%     +74.6%     343259 ±  7%  sched_debug.cfs_rq:/.spread0.stddev
    133.29 ± 19%     +20.9%     161.21 ±  9%  sched_debug.cpu.cpu_load[2].max
     67.65 ± 10%     +10.0%      74.42 ±  6%  sched_debug.cpu.cpu_load[4].max
      5021 ±  7%     -20.8%       3978 ±  8%  sched_debug.cpu.nr_switches.stddev
    613.25 ± 39%     -35.6%     395.00        interrupts.105:IR-PCI-MSI.1574916-edge.eth1-TxRx-4
    538.00 ± 51%     +36.2%     733.00 ± 46%  interrupts.109:IR-PCI-MSI.1574920-edge.eth1-TxRx-8
    394.00 ±  6%      +6.1%     418.00 ±  5%  interrupts.111:IR-PCI-MSI.1574922-edge.eth1-TxRx-10
    345.00 ±  2%     +24.1%     428.00 ± 18%  interrupts.146:IR-PCI-MSI.1574957-edge.eth1-TxRx-45
    631.25 ±  4%     +25.1%     790.00 ±  5%  interrupts.CPU1.TLB:TLB_shootdowns
    394.00 ±  6%      +6.1%     418.00 ±  5%  interrupts.CPU10.111:IR-PCI-MSI.1574922-edge.eth1-TxRx-10
    649.25 ± 10%     +12.4%     729.50 ±  4%  interrupts.CPU12.TLB:TLB_shootdowns
    601.75 ±  4%     +32.6%     798.00 ±  7%  interrupts.CPU13.TLB:TLB_shootdowns
    629.00 ±  9%     +24.1%     780.50        interrupts.CPU14.TLB:TLB_shootdowns
    601.25 ± 16%     +27.7%     767.50 ± 15%  interrupts.CPU15.TLB:TLB_shootdowns
    635.25 ±  8%     +15.7%     735.00 ±  6%  interrupts.CPU17.TLB:TLB_shootdowns
    604.50 ± 13%     +18.3%     715.00 ±  2%  interrupts.CPU18.TLB:TLB_shootdowns
    635.25 ± 10%     +15.7%     735.00 ±  6%  interrupts.CPU19.TLB:TLB_shootdowns
    607.25 ±  8%     +27.7%     775.50        interrupts.CPU2.TLB:TLB_shootdowns
    686.50 ±  9%     +18.2%     811.50 ± 16%  interrupts.CPU21.TLB:TLB_shootdowns
      5748 ±  2%     -31.4%       3942 ± 32%  interrupts.CPU22.NMI:Non-maskable_interrupts
      5748 ±  2%     -31.4%       3942 ± 32%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
    651.25 ± 11%     +17.2%     763.50 ± 10%  interrupts.CPU22.TLB:TLB_shootdowns
      5735 ±  3%     -31.3%       3941 ± 32%  interrupts.CPU23.NMI:Non-maskable_interrupts
      5735 ±  3%     -31.3%       3941 ± 32%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
      5679 ±  2%     -30.6%       3942 ± 31%  interrupts.CPU24.NMI:Non-maskable_interrupts
      5679 ±  2%     -30.6%       3942 ± 31%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
    647.50 ±  6%     +35.5%     877.50 ±  8%  interrupts.CPU24.TLB:TLB_shootdowns
    621.00 ±  3%     +34.5%     835.50 ±  4%  interrupts.CPU26.TLB:TLB_shootdowns
    597.75 ±  9%     +26.8%     758.00 ±  5%  interrupts.CPU27.TLB:TLB_shootdowns
    587.25 ±  7%     +19.2%     700.00 ±  6%  interrupts.CPU37.TLB:TLB_shootdowns
    613.25 ± 39%     -35.6%     395.00        interrupts.CPU4.105:IR-PCI-MSI.1574916-edge.eth1-TxRx-4
    668.25 ± 11%     +19.5%     798.50 ±  4%  interrupts.CPU40.TLB:TLB_shootdowns
    646.25 ±  7%     +11.5%     720.50 ±  9%  interrupts.CPU43.TLB:TLB_shootdowns
    345.00 ±  2%     +24.1%     428.00 ± 18%  interrupts.CPU45.146:IR-PCI-MSI.1574957-edge.eth1-TxRx-45
    587.00 ± 13%     +18.2%     694.00 ± 10%  interrupts.CPU45.TLB:TLB_shootdowns
    616.75 ±  3%     +16.9%     721.00 ±  9%  interrupts.CPU49.TLB:TLB_shootdowns
    680.00 ±  9%     +24.9%     849.50 ±  2%  interrupts.CPU50.TLB:TLB_shootdowns
    661.75 ± 11%     +24.1%     821.00 ±  2%  interrupts.CPU6.TLB:TLB_shootdowns
    606.50 ± 13%     +25.2%     759.50 ±  7%  interrupts.CPU62.TLB:TLB_shootdowns
    611.25 ± 11%     +18.2%     722.50        interrupts.CPU63.TLB:TLB_shootdowns
    576.50 ± 10%     +31.7%     759.50 ±  8%  interrupts.CPU64.TLB:TLB_shootdowns
    166.25 ±173%    +304.5%     672.50        interrupts.CPU65.168:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
      5687 ±  4%     -30.2%       3969 ± 30%  interrupts.CPU66.NMI:Non-maskable_interrupts
      5687 ±  4%     -30.2%       3969 ± 30%  interrupts.CPU66.PMI:Performance_monitoring_interrupts
      5683 ±  3%     -30.7%       3938 ± 31%  interrupts.CPU67.NMI:Non-maskable_interrupts
      5683 ±  3%     -30.7%       3938 ± 31%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
      5706 ±  2%     -30.7%       3955 ± 32%  interrupts.CPU68.NMI:Non-maskable_interrupts
      5706 ±  2%     -30.7%       3955 ± 32%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
    610.50 ± 10%     +17.3%     716.00 ±  5%  interrupts.CPU68.TLB:TLB_shootdowns
      5624 ±  2%     -29.5%       3967 ± 32%  interrupts.CPU70.NMI:Non-maskable_interrupts
      5624 ±  2%     -29.5%       3967 ± 32%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
    562.75 ± 15%     +14.5%     644.50 ± 11%  interrupts.CPU71.TLB:TLB_shootdowns
      5672 ±  3%     -30.2%       3957 ± 30%  interrupts.CPU74.NMI:Non-maskable_interrupts
      5672 ±  3%     -30.2%       3957 ± 30%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
    606.00 ±  8%     +25.5%     760.50        interrupts.CPU76.TLB:TLB_shootdowns
      5739 ±  2%     -31.6%       3923 ± 30%  interrupts.CPU79.NMI:Non-maskable_interrupts
      5739 ±  2%     -31.6%       3923 ± 30%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
    538.00 ± 51%     +36.2%     733.00 ± 46%  interrupts.CPU8.109:IR-PCI-MSI.1574920-edge.eth1-TxRx-8
    655.25 ±  7%     +17.1%     767.00 ± 13%  interrupts.CPU81.TLB:TLB_shootdowns
    575.75 ± 11%     +47.8%     851.00 ±  9%  interrupts.CPU82.TLB:TLB_shootdowns
    684.75 ±  4%     +16.8%     800.00 ±  6%  interrupts.CPU84.TLB:TLB_shootdowns
     30.75 ±  2%      -3.6       27.16 ±  3%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     29.27 ±  3%      -3.6       25.70 ±  3%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     29.82 ±  2%      -3.6       26.24 ±  3%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     29.79 ±  2%      -3.6       26.21 ±  3%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     30.37 ±  2%      -3.5       26.83 ±  3%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     30.37 ±  2%      -3.5       26.83 ±  3%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     30.37 ±  2%      -3.5       26.84 ±  3%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      0.71 ±  3%      +0.1        0.84        perf-profile.calltrace.cycles-pp.do_wp_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
      0.63 ±  4%      +0.1        0.77        perf-profile.calltrace.cycles-pp.wp_page_copy.do_wp_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
      0.83 ± 10%      +0.2        1.01        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_read_lock_slowpath.do_wait.kernel_wait4.__do_sys_wait4
      0.98 ±  9%      +0.2        1.17 ±  2%  perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      0.40 ± 57%      +0.2        0.60        perf-profile.calltrace.cycles-pp.release_task.wait_consider_task.do_wait.kernel_wait4.__do_sys_wait4
      0.42 ± 57%      +0.2        0.64        perf-profile.calltrace.cycles-pp.wait_consider_task.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64
      1.92 ±  5%      +0.3        2.17 ±  3%  perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      1.95 ±  5%      +0.3        2.22 ±  3%  perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      2.12 ±  5%      +0.3        2.41 ±  3%  perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
      2.17 ±  5%      +0.3        2.46 ±  3%  perf-profile.calltrace.cycles-pp.page_fault
      2.15 ±  5%      +0.3        2.44 ±  3%  perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
      1.81 ±  8%      +0.3        2.11        perf-profile.calltrace.cycles-pp.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait
      1.99 ±  8%      +0.3        2.30        perf-profile.calltrace.cycles-pp.wait
      1.78 ±  8%      +0.3        2.09        perf-profile.calltrace.cycles-pp.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.85 ±  8%      +0.3        2.16        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.wait
      1.84 ±  8%      +0.3        2.16        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait
      1.80 ±  8%      +0.3        2.11        perf-profile.calltrace.cycles-pp.kernel_wait4.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait
      0.27 ±100%      +0.3        0.60        perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      0.28 ±100%      +0.3        0.61        perf-profile.calltrace.cycles-pp.ret_from_fork
      0.28 ±100%      +0.3        0.61        perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      0.29 ±100%      +0.3        0.64        perf-profile.calltrace.cycles-pp.__vmalloc_node_range.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.13 ±173%      +0.4        0.56        perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      0.13 ±173%      +0.4        0.56        perf-profile.calltrace.cycles-pp.__get_vm_area_node.__vmalloc_node_range.copy_process._do_fork.do_syscall_64
      0.00            +0.5        0.53        perf-profile.calltrace.cycles-pp.__vunmap.free_work.process_one_work.worker_thread.kthread
      0.00            +0.5        0.53        perf-profile.calltrace.cycles-pp.alloc_vmap_area.__get_vm_area_node.__vmalloc_node_range.copy_process._do_fork
      0.00            +0.5        0.53        perf-profile.calltrace.cycles-pp.free_work.process_one_work.worker_thread.kthread.ret_from_fork
     15.78            +0.7       16.50        perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.anon_vma_clone.anon_vma_fork
     16.55            +0.7       17.28        perf-profile.calltrace.cycles-pp.down_write.anon_vma_clone.anon_vma_fork.dup_mm.copy_process
     16.44            +0.7       17.17        perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.anon_vma_clone.anon_vma_fork.dup_mm
     20.26            +0.8       21.02        perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.unlink_anon_vmas.free_pgtables
     20.90            +0.8       21.75        perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.unlink_anon_vmas.free_pgtables.exit_mmap
     21.04            +0.8       21.88        perf-profile.calltrace.cycles-pp.down_write.unlink_anon_vmas.free_pgtables.exit_mmap.mmput
     17.93            +0.8       18.77        perf-profile.calltrace.cycles-pp.anon_vma_clone.anon_vma_fork.dup_mm.copy_process._do_fork
     25.06 ±  2%      +1.0       26.04        perf-profile.calltrace.cycles-pp.anon_vma_fork.dup_mm.copy_process._do_fork.do_syscall_64
     26.77 ±  2%      +1.1       27.86        perf-profile.calltrace.cycles-pp.dup_mm.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
     32.33            +1.3       33.63        perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__x64_sys_exit_group
     32.35            +1.3       33.65        perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
     28.27            +1.3       29.60        perf-profile.calltrace.cycles-pp.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
     28.61            +1.4       29.97        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_fork.fork_test
     28.61            +1.4       29.97        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork.fork_test
     28.59            +1.4       29.95        perf-profile.calltrace.cycles-pp._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork.fork_test
     29.46            +1.4       30.87        perf-profile.calltrace.cycles-pp.__libc_fork.fork_test
     33.46            +1.4       34.91        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     33.46            +1.4       34.91        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     33.45            +1.5       34.91        perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     33.45            +1.5       34.91        perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     33.45            +1.5       34.91        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
     29.95            +1.5       31.41        perf-profile.calltrace.cycles-pp.fork_test
     29.64 ±  3%      -3.6       26.01 ±  3%  perf-profile.children.cycles-pp.intel_idle
     30.20 ±  2%      -3.6       26.56 ±  3%  perf-profile.children.cycles-pp.cpuidle_enter
     30.19 ±  2%      -3.6       26.56 ±  3%  perf-profile.children.cycles-pp.cpuidle_enter_state
     30.75 ±  2%      -3.6       27.16 ±  3%  perf-profile.children.cycles-pp.do_idle
     30.75 ±  2%      -3.6       27.16 ±  3%  perf-profile.children.cycles-pp.secondary_startup_64
     30.75 ±  2%      -3.6       27.16 ±  3%  perf-profile.children.cycles-pp.cpu_startup_entry
     30.37 ±  2%      -3.5       26.84 ±  3%  perf-profile.children.cycles-pp.start_secondary
      0.38 ±  6%      -0.1        0.33 ±  4%  perf-profile.children.cycles-pp.start_kernel
      0.06 ±  7%      +0.0        0.07        perf-profile.children.cycles-pp.security_vm_enough_memory_mm
      0.10 ±  5%      +0.0        0.11        perf-profile.children.cycles-pp.___perf_sw_event
      0.07 ±  7%      +0.0        0.08        perf-profile.children.cycles-pp.put_task_stack
      0.09 ±  4%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.rwsem_mark_wake
      0.11 ±  7%      +0.0        0.13        perf-profile.children.cycles-pp.sched_move_task
      0.19 ±  3%      +0.0        0.21        perf-profile.children.cycles-pp.dequeue_task_fair
      0.18 ±  5%      +0.0        0.20        perf-profile.children.cycles-pp.__wake_up_common_lock
      0.06            +0.0        0.08        perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.16 ±  2%      +0.0        0.18 ±  2%  perf-profile.children.cycles-pp.dequeue_entity
      0.18 ±  6%      +0.0        0.21        perf-profile.children.cycles-pp.unlink_file_vma
      0.18 ±  6%      +0.0        0.20        perf-profile.children.cycles-pp.__wake_up_common
      0.15 ±  3%      +0.0        0.18 ±  5%  perf-profile.children.cycles-pp.flush_tlb_func_common
      0.14 ±  5%      +0.0        0.17 ±  3%  perf-profile.children.cycles-pp.native_flush_tlb_one_user
      0.22 ±  4%      +0.0        0.25 ±  4%  perf-profile.children.cycles-pp.schedule_idle
      0.18 ±  2%      +0.0        0.21 ±  4%  perf-profile.children.cycles-pp.ptep_clear_flush
      0.15 ±  2%      +0.0        0.17 ±  2%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.21 ±  3%      +0.0        0.24 ±  4%  perf-profile.children.cycles-pp.flush_tlb_mm_range
      0.14 ±  8%      +0.0        0.17 ±  3%  perf-profile.children.cycles-pp.finish_task_switch
      0.34 ±  4%      +0.0        0.37 ±  2%  perf-profile.children.cycles-pp.find_busiest_group
      0.53 ±  2%      +0.0        0.56        perf-profile.children.cycles-pp.schedule
      0.45 ±  3%      +0.0        0.48 ±  2%  perf-profile.children.cycles-pp.load_balance
      0.36 ±  6%      +0.0        0.39 ±  2%  perf-profile.children.cycles-pp.prepare_exit_to_usermode
      0.27 ±  2%      +0.0        0.30 ±  4%  perf-profile.children.cycles-pp.up_write
      0.22 ±  9%      +0.0        0.26        perf-profile.children.cycles-pp.schedule_tail
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.__list_add_valid
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.queue_work_on
      0.02 ±173%      +0.1        0.08        perf-profile.children.cycles-pp.__lru_cache_add
      0.38 ±  5%      +0.1        0.45        perf-profile.children.cycles-pp.copy_page
      0.39 ±  6%      +0.1        0.46        perf-profile.children.cycles-pp.copy_user_highpage
      0.98 ±  3%      +0.1        1.04        perf-profile.children.cycles-pp.__schedule
      0.51 ±  9%      +0.1        0.60        perf-profile.children.cycles-pp.release_task
      0.54 ±  9%      +0.1        0.64        perf-profile.children.cycles-pp.wait_consider_task
      1.04 ±  4%      +0.1        1.14        perf-profile.children.cycles-pp.rwsem_wake
      0.37 ± 19%      +0.1        0.48        perf-profile.children.cycles-pp.find_vmap_area
      0.93 ±  3%      +0.1        1.04        perf-profile.children.cycles-pp.wake_up_q
      0.41 ± 16%      +0.1        0.53        perf-profile.children.cycles-pp.alloc_vmap_area
      0.44 ± 15%      +0.1        0.56        perf-profile.children.cycles-pp.__get_vm_area_node
      0.41 ± 17%      +0.1        0.53        perf-profile.children.cycles-pp.__vunmap
      0.49 ± 14%      +0.1        0.61        perf-profile.children.cycles-pp.kthread
      0.48 ± 15%      +0.1        0.60        perf-profile.children.cycles-pp.worker_thread
      0.43 ± 15%      +0.1        0.56        perf-profile.children.cycles-pp.process_one_work
      0.41 ± 17%      +0.1        0.53        perf-profile.children.cycles-pp.free_work
      1.10 ±  5%      +0.1        1.23        perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.51 ± 14%      +0.1        0.64        perf-profile.children.cycles-pp.__vmalloc_node_range
      1.28 ±  4%      +0.1        1.42        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.08 ±  4%      +0.1        1.22        perf-profile.children.cycles-pp.try_to_wake_up
      0.93 ±  4%      +0.2        1.08 ±  2%  perf-profile.children.cycles-pp.wp_page_copy
      0.74 ± 12%      +0.2        0.91        perf-profile.children.cycles-pp.ret_from_fork
      1.05 ±  4%      +0.2        1.22 ±  2%  perf-profile.children.cycles-pp.do_wp_page
      0.98 ±  9%      +0.2        1.17 ±  2%  perf-profile.children.cycles-pp.queued_read_lock_slowpath
      1.00 ± 10%      +0.2        1.19 ±  2%  perf-profile.children.cycles-pp.queued_write_lock_slowpath
      1.08 ± 14%      +0.2        1.33        perf-profile.children.cycles-pp._raw_spin_lock
      2.01 ±  8%      +0.3        2.32        perf-profile.children.cycles-pp.wait
      1.81 ±  8%      +0.3        2.12        perf-profile.children.cycles-pp.__do_sys_wait4
      1.78 ±  8%      +0.3        2.09        perf-profile.children.cycles-pp.do_wait
      1.80 ±  8%      +0.3        2.11        perf-profile.children.cycles-pp.kernel_wait4
      3.58 ±  9%      +0.7        4.27        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     17.93            +0.9       18.78        perf-profile.children.cycles-pp.anon_vma_clone
     25.07 ±  2%      +1.0       26.04        perf-profile.children.cycles-pp.anon_vma_fork
     26.78 ±  2%      +1.1       27.87        perf-profile.children.cycles-pp.dup_mm
     32.33            +1.3       33.63        perf-profile.children.cycles-pp.exit_mmap
     32.35            +1.3       33.65        perf-profile.children.cycles-pp.mmput
     28.27            +1.3       29.61        perf-profile.children.cycles-pp.copy_process
     28.59            +1.4       29.95        perf-profile.children.cycles-pp._do_fork
     29.50            +1.4       30.92        perf-profile.children.cycles-pp.__libc_fork
     33.51            +1.5       34.97        perf-profile.children.cycles-pp.do_exit
     33.52            +1.5       34.98        perf-profile.children.cycles-pp.__x64_sys_exit_group
     33.52            +1.5       34.98        perf-profile.children.cycles-pp.do_group_exit
     30.00            +1.5       31.47        perf-profile.children.cycles-pp.fork_test
     64.03            +3.1       67.16        perf-profile.children.cycles-pp.do_syscall_64
     64.04            +3.1       67.16        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     29.63 ±  3%      -3.6       25.99 ±  3%  perf-profile.self.cycles-pp.intel_idle
      0.14 ±  5%      +0.0        0.17 ±  3%  perf-profile.self.cycles-pp.native_flush_tlb_one_user
      0.16 ± 11%      +0.0        0.19 ±  5%  perf-profile.self.cycles-pp.unlock_page
      0.27 ±  3%      +0.0        0.30 ±  3%  perf-profile.self.cycles-pp.up_write
      1.12            +0.0        1.17        perf-profile.self.cycles-pp.rwsem_down_write_slowpath
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.00            +0.1        0.05        perf-profile.self.cycles-pp.__list_add_valid
      0.38 ±  6%      +0.1        0.44        perf-profile.self.cycles-pp.copy_page
      0.92 ±  6%      +0.1        1.04        perf-profile.self.cycles-pp.rwsem_spin_on_owner
      3.56 ±  9%      +0.7        4.25        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath



***************************************************************************************************
lkp-bdw-ex1: 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 496G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/load/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/8000/debian-x86_64-2019-05-14.cgz/lkp-bdw-ex1/disk_src/aim7/0xb000036

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     38518            +1.2%      38996        proc-vmstat.nr_slab_reclaimable
    233358 ± 29%     -83.8%      37823 ±  4%  numa-numastat.node0.local_node
    233390 ± 29%     -70.5%      68854 ±  2%  numa-numastat.node0.numa_hit
     34.25 ±108%  +90501.5%      31031        numa-numastat.node0.other_node
     62521 ± 58%    +441.7%     338669        numa-numastat.node2.local_node
     93578 ± 38%    +288.4%     363493 ±  3%  numa-numastat.node2.numa_hit
     31070           -79.7%       6300 ± 98%  numa-numastat.node3.other_node
     60884 ±  6%      -8.3%      55851        slabinfo.shmem_inode_cache.active_objs
      1327 ±  6%      -8.4%       1216        slabinfo.shmem_inode_cache.active_slabs
     61092 ±  6%      -8.4%      55988        slabinfo.shmem_inode_cache.num_objs
      1327 ±  6%      -8.4%       1216        slabinfo.shmem_inode_cache.num_slabs
    771.50 ± 17%     -21.4%     606.50 ±  5%  slabinfo.skbuff_ext_cache.active_objs
    771.50 ± 17%     -21.4%     606.50 ±  5%  slabinfo.skbuff_ext_cache.num_objs
 3.357e+08            -2.1%  3.287e+08        perf-stat.i.cache-references
      6168 ± 12%     +24.6%       7686 ±  6%  perf-stat.i.cycles-between-cache-misses
      0.40            +3.1%       0.41 ±  2%  perf-stat.i.ipc
   1343568 ± 10%     +21.3%    1630157 ±  6%  perf-stat.i.node-loads
     17.33            -4.9%      16.48        perf-stat.overall.MPKI
      2.39            -3.4%       2.30        perf-stat.overall.cpi
      0.42            +3.6%       0.43        perf-stat.overall.ipc
 3.315e+08            -2.1%  3.244e+08        perf-stat.ps.cache-references
      7455            +1.4%       7560        perf-stat.ps.minor-faults
   1325962 ± 10%     +21.2%    1607636 ±  6%  perf-stat.ps.node-loads
      7454            +1.4%       7559        perf-stat.ps.page-faults
     48.50 ±  7%    +187.6%     139.50 ± 62%  interrupts.145:PCI-MSI.1574927-edge.eth3-TxRx-15
      1528 ± 18%     +67.9%       2565 ± 35%  interrupts.CPU1.RES:Rescheduling_interrupts
      1355 ±  8%     +20.5%       1632 ±  7%  interrupts.CPU11.RES:Rescheduling_interrupts
      1058 ± 48%    +154.3%       2690 ± 49%  interrupts.CPU138.RES:Rescheduling_interrupts
      1571 ± 16%     +19.7%       1881 ± 11%  interrupts.CPU144.RES:Rescheduling_interrupts
      1438 ± 17%     +18.5%       1704 ± 11%  interrupts.CPU145.RES:Rescheduling_interrupts
      1440 ± 14%     +19.7%       1724 ±  4%  interrupts.CPU149.RES:Rescheduling_interrupts
     48.50 ±  7%    +187.6%     139.50 ± 62%  interrupts.CPU15.145:PCI-MSI.1574927-edge.eth3-TxRx-15
      1412 ± 14%     +11.5%       1574 ± 11%  interrupts.CPU155.RES:Rescheduling_interrupts
      1407 ± 14%     +15.8%       1629 ± 10%  interrupts.CPU157.RES:Rescheduling_interrupts
      1334 ±  3%     +10.2%       1470        interrupts.CPU27.RES:Rescheduling_interrupts
      1393 ±  5%     +10.5%       1539 ±  7%  interrupts.CPU29.RES:Rescheduling_interrupts
      1122 ± 22%    +354.4%       5099 ± 73%  interrupts.CPU34.RES:Rescheduling_interrupts
      1132 ± 21%    +299.5%       4525 ± 67%  interrupts.CPU35.RES:Rescheduling_interrupts
      2028 ± 49%     -62.6%     758.00 ± 70%  interrupts.CPU40.RES:Rescheduling_interrupts
      1159 ± 35%    +148.2%       2878 ± 48%  interrupts.CPU45.RES:Rescheduling_interrupts
      1356 ± 16%     +19.3%       1618 ± 10%  interrupts.CPU71.RES:Rescheduling_interrupts
      1.03 ±100%      +1.5        2.56 ± 11%  perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit
      0.49 ±173%      +2.1        2.56 ± 11%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit
      0.98 ±173%      +2.7        3.71 ± 22%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
      0.98 ±173%      +2.7        3.71 ± 22%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.49 ±173%      +3.2        3.71 ± 22%  perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.49 ±173%      +3.2        3.71 ± 22%  perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.00            +3.7        3.71 ± 22%  perf-profile.calltrace.cycles-pp.seq_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.98 ±173%      +4.2        5.13 ± 11%  perf-profile.calltrace.cycles-pp.read
      1.03 ±100%      +1.5        2.56 ± 11%  perf-profile.children.cycles-pp.unmap_vmas
      0.49 ±173%      +2.1        2.56 ± 11%  perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.49 ±173%      +2.1        2.56 ± 11%  perf-profile.children.cycles-pp.unmap_page_range
      0.00            +2.6        2.56 ± 11%  perf-profile.children.cycles-pp.mutex_unlock
     18.25 ± 10%      +3.4       21.66 ±  5%  perf-profile.children.cycles-pp.do_group_exit
     18.25 ± 10%      +3.4       21.66 ±  5%  perf-profile.children.cycles-pp.do_exit
      0.98 ±173%      +4.2        5.13 ± 11%  perf-profile.children.cycles-pp.read
      0.49 ±173%      +2.1        2.56 ± 11%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.00            +2.6        2.56 ± 11%  perf-profile.self.cycles-pp.mutex_unlock
      3235 ±  3%     -59.2%       1318 ± 98%  sched_debug.cfs_rq:/.exec_clock.max
     10.08 ± 24%    +109.8%      21.14 ± 30%  sched_debug.cfs_rq:/.load_avg.avg
    440.50 ± 11%     +75.9%     774.75 ± 32%  sched_debug.cfs_rq:/.load_avg.max
     51.31 ± 21%     +92.3%      98.66 ± 34%  sched_debug.cfs_rq:/.load_avg.stddev
    416493 ±  3%     -69.9%     125500 ± 71%  sched_debug.cfs_rq:/.min_vruntime.max
      1.36 ±173%    +360.0%       6.26 ± 26%  sched_debug.cfs_rq:/.removed.load_avg.avg
    122.00 ±173%    +522.5%     759.50 ± 34%  sched_debug.cfs_rq:/.removed.load_avg.max
     12.77 ±173%    +437.4%      68.61 ± 30%  sched_debug.cfs_rq:/.removed.load_avg.stddev
     62.75 ±173%    +360.3%     288.79 ± 26%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
      5623 ±173%    +522.8%      35019 ± 34%  sched_debug.cfs_rq:/.removed.runnable_sum.max
    588.63 ±173%    +437.5%       3164 ± 30%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
      0.55 ±173%    +473.9%       3.14 ± 26%  sched_debug.cfs_rq:/.removed.util_avg.avg
     61.62 ±173%    +517.8%     380.75 ± 34%  sched_debug.cfs_rq:/.removed.util_avg.max
      5.31 ±173%    +547.6%      34.40 ± 30%  sched_debug.cfs_rq:/.removed.util_avg.stddev
      0.79 ± 14%     +24.7%       0.99 ±  8%  sched_debug.cfs_rq:/.runnable_load_avg.avg
    288696 ± 13%     -90.7%      26906 ± 96%  sched_debug.cfs_rq:/.spread0.max
    135.02 ±  3%     +32.0%     178.17 ±  9%  sched_debug.cfs_rq:/.util_avg.avg
      6.14 ± 20%     +96.1%      12.04 ± 40%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    282.25           +40.7%     397.00 ± 24%  sched_debug.cfs_rq:/.util_est_enqueued.max
     36.69 ± 11%     +72.8%      63.38 ± 33%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
    143.02 ±  6%     -65.3%      49.57 ± 83%  sched_debug.cpu.clock.stddev
    143.02 ±  6%     -65.3%      49.57 ± 83%  sched_debug.cpu.clock_task.stddev
     85.62 ± 18%     -35.2%      55.50 ±  4%  sched_debug.cpu.cpu_load[4].max
      6.55 ± 16%     -24.0%       4.98        sched_debug.cpu.cpu_load[4].stddev
      0.00 ±  6%     -61.0%       0.00 ± 62%  sched_debug.cpu.next_balance.stddev
    212392 ± 71%     -71.1%      61441 ±  5%  numa-meminfo.node0.Active
    212340 ± 71%     -71.1%      61441 ±  5%  numa-meminfo.node0.Active(anon)
     25308 ± 36%     -96.0%       1024 ±100%  numa-meminfo.node0.AnonHugePages
    203492 ± 74%     -69.7%      61607 ±  5%  numa-meminfo.node0.AnonPages
     96358 ±  8%     -66.7%      32077 ±  3%  numa-meminfo.node0.KReclaimable
   1236693 ± 41%     -36.1%     790523        numa-meminfo.node0.MemUsed
     96358 ±  8%     -66.7%      32077 ±  3%  numa-meminfo.node0.SReclaimable
    157686 ± 55%     -45.7%      85674        numa-meminfo.node0.SUnreclaim
     28625 ±  3%     -30.4%      19912        numa-meminfo.node0.Shmem
    254045 ± 36%     -53.6%     117751        numa-meminfo.node0.Slab
      1162 ± 27%     -76.0%     279.00 ± 81%  numa-meminfo.node1.Inactive
      1069 ± 25%     -82.3%     189.50 ± 72%  numa-meminfo.node1.Inactive(anon)
      1110 ± 23%     -49.7%     558.00 ±  7%  numa-meminfo.node1.Shmem
    107470 ± 48%    +329.4%     461499 ±  6%  numa-meminfo.node2.Active
    107469 ± 48%    +329.3%     461407 ±  6%  numa-meminfo.node2.Active(anon)
    107671 ± 48%    +325.7%     458377 ±  5%  numa-meminfo.node2.AnonPages
    249548            +8.9%     271751 ±  7%  numa-meminfo.node2.FilePages
     15548 ±  7%    +247.4%      54007 ± 12%  numa-meminfo.node2.KReclaimable
      5609 ±  2%   +2180.6%     127917        numa-meminfo.node2.KernelStack
    589997 ±  8%    +214.5%    1855314 ±  5%  numa-meminfo.node2.MemUsed
      2892 ± 57%   +9676.0%     282747 ±  8%  numa-meminfo.node2.PageTables
     15548 ±  7%    +247.4%      54007 ± 12%  numa-meminfo.node2.SReclaimable
     59530 ±  6%    +379.2%     285257        numa-meminfo.node2.SUnreclaim
      1011 ± 19%    +310.3%       4149 ± 53%  numa-meminfo.node2.Shmem
     75078 ±  5%    +351.9%     339265 ±  2%  numa-meminfo.node2.Slab
    248134            +7.5%     266686 ±  6%  numa-meminfo.node2.Unevictable
    578.25 ± 71%    +132.0%       1341 ± 44%  numa-meminfo.node3.Inactive(anon)
     23848 ± 39%    +116.8%      51695 ±  9%  numa-meminfo.node3.KReclaimable
    209891 ± 56%     -99.0%       2018 ± 35%  numa-meminfo.node3.PageTables
     23848 ± 39%    +116.8%      51695 ±  9%  numa-meminfo.node3.SReclaimable
      1612 ± 48%    +377.7%       7701 ± 26%  numa-meminfo.node3.Shmem
     52988 ± 71%     -70.9%      15424 ±  5%  numa-vmstat.node0.nr_active_anon
     50771 ± 74%     -69.5%      15467 ±  5%  numa-vmstat.node0.nr_anon_pages
      7156 ±  3%     -30.4%       4978        numa-vmstat.node0.nr_shmem
     24067 ±  8%     -66.7%       8025 ±  3%  numa-vmstat.node0.nr_slab_reclaimable
     39378 ± 55%     -45.5%      21444        numa-vmstat.node0.nr_slab_unreclaimable
     52988 ± 71%     -70.9%      15424 ±  5%  numa-vmstat.node0.nr_zone_active_anon
    674305 ± 19%     -23.6%     515357 ±  3%  numa-vmstat.node0.numa_hit
    672585 ± 19%     -28.0%     484049 ±  3%  numa-vmstat.node0.numa_local
      1719 ±  3%   +1720.5%      31307        numa-vmstat.node0.numa_other
    267.25 ± 25%     -82.4%      47.00 ± 72%  numa-vmstat.node1.nr_inactive_anon
      1267 ±  4%      +3.3%       1309 ±  4%  numa-vmstat.node1.nr_mapped
    277.50 ± 23%     -49.9%     139.00 ±  7%  numa-vmstat.node1.nr_shmem
    267.25 ± 25%     -82.4%      47.00 ± 72%  numa-vmstat.node1.nr_zone_inactive_anon
     26830 ± 48%    +330.4%     115485 ±  5%  numa-vmstat.node2.nr_active_anon
     26878 ± 48%    +326.7%     114684 ±  5%  numa-vmstat.node2.nr_anon_pages
     62390            +8.9%      67941 ±  7%  numa-vmstat.node2.nr_file_pages
      5612 ±  2%   +2183.0%     128119        numa-vmstat.node2.nr_kernel_stack
      1238            +6.2%       1315 ±  5%  numa-vmstat.node2.nr_mapped
    719.50 ± 57%   +9745.0%      70835 ±  8%  numa-vmstat.node2.nr_page_table_pages
    253.25 ± 19%    +311.1%       1041 ± 53%  numa-vmstat.node2.nr_shmem
      3889 ±  7%    +247.5%      13517 ± 12%  numa-vmstat.node2.nr_slab_reclaimable
     14889 ±  6%    +379.5%      71397        numa-vmstat.node2.nr_slab_unreclaimable
     62037            +7.5%      66670 ±  6%  numa-vmstat.node2.nr_unevictable
     26830 ± 48%    +330.4%     115485 ±  5%  numa-vmstat.node2.nr_zone_active_anon
     62037            +7.5%      66670 ±  6%  numa-vmstat.node2.nr_zone_unevictable
    298653 ±  7%     +95.0%     582384        numa-vmstat.node2.numa_hit
    185598 ± 11%    +155.9%     474964        numa-vmstat.node2.numa_local
    144.25 ± 71%    +132.2%     335.00 ± 45%  numa-vmstat.node3.nr_inactive_anon
     52258 ± 56%     -99.0%     504.50 ± 36%  numa-vmstat.node3.nr_page_table_pages
    403.50 ± 48%    +376.8%       1924 ± 26%  numa-vmstat.node3.nr_shmem
      5961 ± 39%    +116.9%      12927 ±  9%  numa-vmstat.node3.nr_slab_reclaimable
    144.25 ± 71%    +132.2%     335.00 ± 45%  numa-vmstat.node3.nr_zone_inactive_anon
    112785           -21.2%      88887 ±  6%  numa-vmstat.node3.numa_other
     32578 ± 10%     -13.9%      28035        softirqs.CPU10.TIMER
     31748 ±  9%     -14.3%      27201        softirqs.CPU100.TIMER
     31646 ±  8%     -12.3%      27766        softirqs.CPU101.TIMER
     31008 ±  7%     -12.7%      27078        softirqs.CPU102.TIMER
     31788 ± 10%     -14.9%      27056        softirqs.CPU103.TIMER
     31729 ±  8%     -14.7%      27073        softirqs.CPU104.TIMER
     31415 ±  7%     -13.9%      27037        softirqs.CPU105.TIMER
     31606 ± 10%     -14.8%      26924        softirqs.CPU106.TIMER
     30924 ±  8%     -12.7%      26983        softirqs.CPU107.TIMER
     31042 ±  7%     -13.2%      26952        softirqs.CPU108.TIMER
     31113 ±  6%     -12.6%      27182        softirqs.CPU109.TIMER
     32157 ±  8%     -12.8%      28041        softirqs.CPU11.TIMER
     31768 ±  7%     -15.1%      26969        softirqs.CPU110.TIMER
     31708 ±  9%     -14.8%      27018        softirqs.CPU111.TIMER
     31989 ±  6%     -15.3%      27088        softirqs.CPU112.TIMER
     32354 ±  7%     -16.6%      26968        softirqs.CPU113.TIMER
     31293 ±  6%     -14.2%      26843        softirqs.CPU114.TIMER
     31129 ±  7%     -13.7%      26870        softirqs.CPU115.TIMER
     31818 ±  5%     -14.9%      27074        softirqs.CPU117.TIMER
     31706 ±  6%     -14.6%      27067        softirqs.CPU118.TIMER
     32302 ±  8%     -12.7%      28198 ±  2%  softirqs.CPU12.TIMER
     32852 ±  9%     -14.6%      28058        softirqs.CPU13.TIMER
     28951 ±  6%      -7.4%      26808        softirqs.CPU147.TIMER
     29375 ±  7%      -7.9%      27053        softirqs.CPU148.TIMER
     32964 ±  9%     -14.8%      28077        softirqs.CPU15.TIMER
     29081 ±  6%      -7.2%      26984        softirqs.CPU150.TIMER
     29028 ±  6%      -7.5%      26853        softirqs.CPU152.TIMER
     28918 ±  7%      -7.4%      26780        softirqs.CPU158.TIMER
     29128 ±  7%      -7.9%      26840        softirqs.CPU159.TIMER
     32669 ±  9%     -13.9%      28130        softirqs.CPU16.TIMER
     28807 ±  6%      -8.1%      26482        softirqs.CPU162.TIMER
     29120 ±  7%      -8.8%      26544        softirqs.CPU164.TIMER
     33381 ±  9%     -16.0%      28032        softirqs.CPU17.TIMER
     24424 ±  3%     +11.4%      27218 ±  5%  softirqs.CPU170.RCU
     30548 ± 10%     -13.5%      26436        softirqs.CPU171.TIMER
     30675 ±  9%     -13.7%      26469        softirqs.CPU172.TIMER
     30949 ± 10%     -13.9%      26650        softirqs.CPU173.TIMER
     30559 ±  9%     -15.0%      25980        softirqs.CPU174.TIMER
     30435 ± 10%     -13.8%      26234        softirqs.CPU176.TIMER
     30427 ± 10%     -14.2%      26104        softirqs.CPU177.TIMER
     30221 ± 10%     -13.2%      26220        softirqs.CPU178.TIMER
     30497 ± 11%     -13.4%      26402        softirqs.CPU179.TIMER
     32166 ±  8%     -13.2%      27924        softirqs.CPU18.TIMER
     30188 ±  9%     -13.0%      26256        softirqs.CPU180.TIMER
     30317 ±  9%     -13.6%      26205        softirqs.CPU181.TIMER
     30256 ±  9%     -13.3%      26227        softirqs.CPU182.TIMER
     30373 ±  9%     -14.3%      26017 ±  2%  softirqs.CPU184.TIMER
     30857 ±  8%     -14.3%      26434        softirqs.CPU185.TIMER
     30779 ±  8%     -14.8%      26234        softirqs.CPU186.TIMER
     30871 ± 11%     -17.4%      25500 ±  3%  softirqs.CPU187.TIMER
     30105 ±  9%     -12.4%      26357        softirqs.CPU188.TIMER
     30180 ±  9%     -11.8%      26622        softirqs.CPU189.TIMER
     32606 ±  8%     -11.7%      28778        softirqs.CPU19.TIMER
     30348 ± 10%     -12.9%      26426        softirqs.CPU190.TIMER
     32679 ±  8%     -12.8%      28484 ±  2%  softirqs.CPU2.TIMER
     32633 ±  9%     -12.7%      28482        softirqs.CPU20.TIMER
     32846 ±  9%     -14.5%      28092        softirqs.CPU21.TIMER
     32451 ±  7%     -13.6%      28038        softirqs.CPU22.TIMER
     31888 ±  8%     -12.2%      27989        softirqs.CPU23.TIMER
     32673 ±  9%     -13.3%      28340        softirqs.CPU3.TIMER
     30578 ±  7%     -10.0%      27535        softirqs.CPU49.TIMER
     32571 ±  7%     -13.4%      28205        softirqs.CPU5.TIMER
     30578 ±  6%      -9.5%      27667        softirqs.CPU51.TIMER
     30173 ±  7%      -8.2%      27697        softirqs.CPU52.TIMER
     29841 ±  6%      -7.9%      27496        softirqs.CPU53.TIMER
     29906 ±  6%      -8.0%      27527        softirqs.CPU56.TIMER
     29967 ±  5%      -7.6%      27701        softirqs.CPU57.TIMER
     29938 ±  6%      -7.3%      27765        softirqs.CPU58.TIMER
     30465 ±  6%      -9.6%      27541        softirqs.CPU59.TIMER
     32882 ±  6%     -14.1%      28238        softirqs.CPU6.TIMER
     30412 ±  7%      -9.6%      27507        softirqs.CPU60.TIMER
     31318 ±  6%     -11.7%      27643        softirqs.CPU62.TIMER
     31305 ±  7%     -11.3%      27776        softirqs.CPU63.TIMER
     24337 ±  5%      +8.7%      26447 ±  4%  softirqs.CPU66.RCU
     30129 ±  5%      -8.3%      27630        softirqs.CPU66.TIMER
     24350 ±  4%      +8.2%      26346 ±  5%  softirqs.CPU67.RCU
     31037 ±  6%     -11.7%      27413        softirqs.CPU68.TIMER
     30792 ±  7%      -9.4%      27893        softirqs.CPU69.TIMER
     32250 ±  7%     -12.4%      28241        softirqs.CPU7.TIMER
     30980 ±  5%      -9.7%      27967        softirqs.CPU70.TIMER
     30611 ±  5%     -10.1%      27526        softirqs.CPU71.TIMER
     31699 ±  9%     -11.9%      27921 ±  2%  softirqs.CPU73.TIMER
     31857 ± 10%     -13.4%      27585        softirqs.CPU77.TIMER
     31624 ±  9%     -15.3%      26795 ±  2%  softirqs.CPU78.TIMER
     31386 ±  9%     -12.8%      27368        softirqs.CPU79.TIMER
     32576 ±  9%     -13.5%      28174        softirqs.CPU8.TIMER
     31444 ± 10%     -14.0%      27045        softirqs.CPU80.TIMER
     31232 ± 10%     -12.5%      27325        softirqs.CPU81.TIMER
     31572 ± 11%     -13.9%      27185        softirqs.CPU82.TIMER
     31527 ± 10%     -14.1%      27071        softirqs.CPU83.TIMER
     31293 ±  9%     -13.6%      27025        softirqs.CPU84.TIMER
     31232 ±  9%     -13.7%      26962        softirqs.CPU85.TIMER
     31197 ± 10%     -15.1%      26492 ±  2%  softirqs.CPU86.TIMER
     31215 ±  9%     -12.7%      27238        softirqs.CPU87.TIMER
     31372 ± 10%     -13.0%      27308        softirqs.CPU88.TIMER
     31465 ±  9%     -14.3%      26968 ±  3%  softirqs.CPU89.TIMER
     32612 ±  9%     -14.3%      27953        softirqs.CPU9.TIMER
     31643 ±  8%     -16.5%      26422 ±  3%  softirqs.CPU90.TIMER
     30989 ±  9%     -14.4%      26521 ±  2%  softirqs.CPU92.TIMER
     31242 ±  9%     -12.0%      27494        softirqs.CPU93.TIMER
     31073 ±  9%     -13.4%      26913        softirqs.CPU94.TIMER
     25373           -12.6%      22177 ±  3%  softirqs.CPU95.RCU
     33071 ±  8%     -17.2%      27398        softirqs.CPU97.TIMER
     26081 ±  6%      -9.1%      23705        softirqs.CPU98.RCU
     31449 ±  9%     -15.7%      26505        softirqs.CPU98.TIMER
     31934 ±  8%     -14.5%      27300        softirqs.CPU99.TIMER



***************************************************************************************************
lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/7000t/debian-x86_64-2019-05-14.cgz/300s/lkp-ivb-2ep1/shared_memory/reaim/0x42e

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :2     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
     45.00 ± 16%     +25.6%      56.50 ± 11%  vmstat.procs.r
 6.093e+08 ± 19%     +41.0%  8.588e+08 ± 32%  cpuidle.C1.time
   8005497 ± 11%     +34.4%   10759454 ± 28%  cpuidle.C1.usage
   8001963 ± 11%     +34.4%   10756594 ± 28%  turbostat.C1
      4.03 ± 19%      +1.6        5.67 ± 32%  turbostat.C1%
      6.89 ± 29%     +48.0%      10.20 ± 12%  turbostat.CPU%c6
      2.18 ± 26%     +47.4%       3.21 ± 23%  turbostat.Pkg%pc2
    202.00            +9.9%     222.00        proc-vmstat.nr_inactive_file
     16125            +2.1%      16469        proc-vmstat.nr_shmem
     19096            -2.9%      18541        proc-vmstat.nr_slab_reclaimable
    202.00            +9.9%     222.00        proc-vmstat.nr_zone_inactive_file
      5269 ±  5%     +10.4%       5819 ±  3%  proc-vmstat.pgactivate
    105576 ±  2%     +31.9%     139239 ± 16%  softirqs.CPU28.TIMER
      6097 ±169%     -98.4%      96.50 ±  9%  softirqs.CPU29.NET_RX
      6060 ±169%    +294.5%      23910        softirqs.CPU31.NET_RX
     81682 ±  7%     +12.4%      91797        softirqs.CPU37.RCU
    106031            +9.1%     115712 ±  5%  softirqs.CPU4.TIMER
    938.75 ±  9%     +13.4%       1064 ±  3%  slabinfo.dmaengine-unmap-16.active_objs
    938.75 ±  9%     +13.4%       1064 ±  3%  slabinfo.dmaengine-unmap-16.num_objs
    558.00           +16.7%     651.00 ±  4%  slabinfo.numa_policy.active_objs
    558.00           +16.7%     651.00 ±  4%  slabinfo.numa_policy.num_objs
    153.00           +50.0%     229.50 ± 11%  slabinfo.xfrm_dst_cache.active_objs
    153.00           +50.0%     229.50 ± 11%  slabinfo.xfrm_dst_cache.num_objs
 6.594e+09            +1.2%  6.672e+09        perf-stat.i.branch-instructions
     15.39            +0.3       15.72        perf-stat.i.cache-miss-rate%
      2.28            -3.1%       2.21        perf-stat.i.cpi
 9.788e+09            +1.0%  9.889e+09        perf-stat.i.dTLB-loads
  21054228            -4.0%   20216327        perf-stat.i.iTLB-load-misses
 3.493e+10            +1.2%  3.535e+10        perf-stat.i.instructions
      1785 ±  2%      +7.5%       1920        perf-stat.i.instructions-per-iTLB-miss
      0.45            +2.2%       0.46        perf-stat.i.ipc
     14.34            +0.2       14.56        perf-stat.overall.cache-miss-rate%
      1661            +5.3%       1749        perf-stat.overall.instructions-per-iTLB-miss
      0.46            +1.1%       0.47        perf-stat.overall.ipc
    743013            +1.5%     754422        perf-stat.overall.path-length
 6.569e+09            +1.2%  6.646e+09        perf-stat.ps.branch-instructions
 9.752e+09            +1.0%  9.852e+09        perf-stat.ps.dTLB-loads
 3.479e+10            +1.2%  3.522e+10        perf-stat.ps.instructions
 1.092e+13            +1.5%  1.109e+13        perf-stat.total.instructions
    168.00 ±  2%      +8.6%     182.50 ±  5%  interrupts.36:IR-PCI-MSI.2621442-edge.eth0-TxRx-1
    169.25 ±  3%     +11.4%     188.50 ±  3%  interrupts.39:IR-PCI-MSI.2621445-edge.eth0-TxRx-4
     12023 ±170%    +296.0%      47614        interrupts.42:IR-PCI-MSI.2621448-edge.eth0-TxRx-7
      1857 ± 57%    +157.2%       4777 ± 32%  interrupts.CPU10.NMI:Non-maskable_interrupts
      1857 ± 57%    +157.2%       4777 ± 32%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      9964 ±  8%      +9.7%      10927 ±  7%  interrupts.CPU14.RES:Rescheduling_interrupts
    708.75 ±169%    +370.2%       3332 ± 96%  interrupts.CPU15.NMI:Non-maskable_interrupts
    708.75 ±169%    +370.2%       3332 ± 96%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
      9702 ±  4%      +9.0%      10579 ±  4%  interrupts.CPU16.RES:Rescheduling_interrupts
      3535 ± 12%     -14.5%       3023        interrupts.CPU18.CAL:Function_call_interrupts
      9525 ±  5%     +10.5%      10524        interrupts.CPU19.RES:Rescheduling_interrupts
      1664 ± 68%     +91.4%       3185        interrupts.CPU20.NMI:Non-maskable_interrupts
      1664 ± 68%     +91.4%       3185        interrupts.CPU20.PMI:Performance_monitoring_interrupts
    168.00 ±  2%      +8.6%     182.50 ±  5%  interrupts.CPU25.36:IR-PCI-MSI.2621442-edge.eth0-TxRx-1
      1757 ± 58%    +172.3%       4785 ± 32%  interrupts.CPU27.NMI:Non-maskable_interrupts
      1757 ± 58%    +172.3%       4785 ± 32%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
    169.25 ±  3%     +11.4%     188.50 ±  3%  interrupts.CPU28.39:IR-PCI-MSI.2621445-edge.eth0-TxRx-4
    189.25 ±126%   +1794.1%       3584 ± 82%  interrupts.CPU28.NMI:Non-maskable_interrupts
    189.25 ±126%   +1794.1%       3584 ± 82%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
    624.75 ±172%    +409.3%       3182 ± 97%  interrupts.CPU29.NMI:Non-maskable_interrupts
    624.75 ±172%    +409.3%       3182 ± 97%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
     12023 ±170%    +296.0%      47614        interrupts.CPU31.42:IR-PCI-MSI.2621448-edge.eth0-TxRx-7
      2004 ± 58%     +59.1%       3187        interrupts.CPU36.NMI:Non-maskable_interrupts
      2004 ± 58%     +59.1%       3187        interrupts.CPU36.PMI:Performance_monitoring_interrupts
      1848 ±110%    +158.4%       4775 ± 32%  interrupts.CPU42.NMI:Non-maskable_interrupts
      1848 ±110%    +158.4%       4775 ± 32%  interrupts.CPU42.PMI:Performance_monitoring_interrupts
      9944 ±  7%     +16.6%      11595 ±  4%  interrupts.CPU42.RES:Rescheduling_interrupts
      2432 ±  4%     -99.6%      10.00 ± 90%  interrupts.CPU44.NMI:Non-maskable_interrupts
      2432 ±  4%     -99.6%      10.00 ± 90%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
    623.75 ±173%    +663.3%       4761 ± 32%  interrupts.CPU6.NMI:Non-maskable_interrupts
    623.75 ±173%    +663.3%       4761 ± 32%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
      1193 ±172%    +298.6%       4756 ± 32%  interrupts.CPU9.NMI:Non-maskable_interrupts
      1193 ±172%    +298.6%       4756 ± 32%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
      9355 ±  8%     +13.2%      10593 ±  6%  interrupts.CPU9.RES:Rescheduling_interrupts
     66.50 ± 44%     -45.9%      36.00 ±  8%  interrupts.CPU9.TLB:TLB_shootdowns
     95338 ± 13%     +24.4%     118623 ± 10%  interrupts.NMI:Non-maskable_interrupts
     95338 ± 13%     +24.4%     118623 ± 10%  interrupts.PMI:Performance_monitoring_interrupts
     18276 ± 34%     +95.4%      35703 ± 29%  sched_debug.cfs_rq:/.load.avg
    396888 ± 33%     +32.9%     527407 ±  8%  sched_debug.cfs_rq:/.load.max
     68973 ± 26%     +67.5%     115530 ± 15%  sched_debug.cfs_rq:/.load.stddev
     35.56 ± 10%     +27.5%      45.33 ±  4%  sched_debug.cfs_rq:/.load_avg.avg
    102.31 ± 14%     +38.2%     141.35 ±  4%  sched_debug.cfs_rq:/.load_avg.stddev
      6.07 ± 63%    +103.5%      12.35 ± 14%  sched_debug.cfs_rq:/.removed.load_avg.avg
     26.67 ± 59%     +64.5%      43.88 ±  6%  sched_debug.cfs_rq:/.removed.load_avg.stddev
    280.13 ± 63%    +103.3%     569.61 ± 14%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
      1230 ± 59%     +64.4%       2023 ±  6%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
      1.61 ± 58%    +188.3%       4.64        sched_debug.cfs_rq:/.removed.util_avg.avg
     47.17 ± 64%     +57.6%      74.33 ± 16%  sched_debug.cfs_rq:/.removed.util_avg.max
      8.04 ± 60%    +110.8%      16.95 ±  8%  sched_debug.cfs_rq:/.removed.util_avg.stddev
     12.11 ± 33%     +76.7%      21.40 ± 13%  sched_debug.cfs_rq:/.runnable_load_avg.avg
     51.67 ± 26%     +68.8%      87.23        sched_debug.cfs_rq:/.runnable_load_avg.stddev
     17578 ± 36%    +100.8%      35303 ± 29%  sched_debug.cfs_rq:/.runnable_weight.avg
    380029 ± 35%     +37.8%     523711 ±  9%  sched_debug.cfs_rq:/.runnable_weight.max
     66430 ± 27%     +72.8%     114787 ± 15%  sched_debug.cfs_rq:/.runnable_weight.stddev
    279.47 ± 26%     -49.7%     140.62 ±  3%  sched_debug.cfs_rq:/.util_avg.avg
      6.87 ± 42%     -82.1%       1.23 ± 16%  sched_debug.cpu.clock.stddev
      6.87 ± 42%     -82.1%       1.23 ± 16%  sched_debug.cpu.clock_task.stddev
     12.26 ± 33%     +42.1%      17.42 ±  7%  sched_debug.cpu.cpu_load[0].avg
     51.68 ± 26%     +54.0%      79.58 ±  9%  sched_debug.cpu.cpu_load[0].stddev
     12.43 ± 31%     +39.9%      17.39 ± 14%  sched_debug.cpu.cpu_load[1].avg
    312.42 ± 23%     +51.9%     474.58 ± 14%  sched_debug.cpu.cpu_load[1].max
     50.56 ± 23%     +56.7%      79.25 ± 15%  sched_debug.cpu.cpu_load[1].stddev
     11.92 ± 28%     +41.7%      16.90 ± 15%  sched_debug.cpu.cpu_load[2].avg
    325.07 ± 18%     +66.9%     542.67 ± 13%  sched_debug.cpu.cpu_load[2].max
     49.19 ± 18%     +67.7%      82.48 ± 15%  sched_debug.cpu.cpu_load[2].stddev
     11.52 ± 25%     +40.7%      16.21 ± 13%  sched_debug.cpu.cpu_load[3].avg
    335.89 ± 14%     +76.2%     591.83 ± 11%  sched_debug.cpu.cpu_load[3].max
     49.24 ± 14%     +74.6%      85.98 ± 12%  sched_debug.cpu.cpu_load[3].stddev
     11.46 ± 26%     +40.7%      16.12 ± 10%  sched_debug.cpu.cpu_load[4].avg
    355.26 ±  8%     +82.7%     649.08 ±  9%  sched_debug.cpu.cpu_load[4].max
     51.09 ±  8%     +82.0%      92.95 ±  9%  sched_debug.cpu.cpu_load[4].stddev
      5957 ± 87%     -72.0%       1667        sched_debug.cpu.curr->pid.avg
     15141 ± 19%     -28.5%      10833        sched_debug.cpu.curr->pid.stddev
     77073 ± 31%     +33.7%     103028 ±  5%  sched_debug.cpu.load.stddev
      0.18 ± 52%     -42.0%       0.10 ±  3%  sched_debug.cpu.nr_running.avg
      1.30 ±  4%     -23.1%       1.00        sched_debug.cpu.nr_running.max
      0.35 ± 10%     -18.0%       0.28        sched_debug.cpu.nr_running.stddev
      8109 ± 11%     +49.7%      12141 ± 28%  sched_debug.cpu.ttwu_local.max
    459.06 ± 14%    +101.8%     926.21 ± 44%  sched_debug.cpu.ttwu_local.stddev
      0.78 ±  3%      -0.1        0.64        perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      0.90 ±  2%      -0.1        0.77        perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
      0.91 ±  2%      -0.1        0.78        perf-profile.calltrace.cycles-pp.page_fault
      0.88 ±  3%      -0.1        0.75        perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
      0.74 ±  4%      -0.1        0.61        perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
      0.54 ±  5%      +0.1        0.60 ±  2%  perf-profile.calltrace.cycles-pp.dentry_kill.dput.__fput.task_work_run.exit_to_usermode_loop
      0.56 ±  6%      +0.1        0.64 ±  2%  perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
      0.72 ±  6%      +0.1        0.80        perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.73 ±  8%      +0.1        0.82 ±  7%  perf-profile.calltrace.cycles-pp.newseg.ipcget.__x64_sys_shmget.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.77 ±  7%      +0.1        0.86 ±  2%  perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.80 ±  7%      +0.1        0.90        perf-profile.calltrace.cycles-pp.down_write.shmctl_down.ksys_shmctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.81 ±  7%      +0.1        0.91        perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.91 ±  5%      +0.1        1.02 ±  2%  perf-profile.calltrace.cycles-pp.do_smart_wakeup_zero.do_smart_update.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.43 ±  6%      +0.1        1.54        perf-profile.calltrace.cycles-pp.shmctl_down.ksys_shmctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.43 ±  6%      +0.1        1.56        perf-profile.calltrace.cycles-pp.ksys_shmctl.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.93 ±  9%      +0.1        1.07 ±  6%  perf-profile.calltrace.cycles-pp.down_write.do_shmat.__x64_sys_shmat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.89 ±  9%      +0.1        1.04 ±  5%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.do_shmat.__x64_sys_shmat.do_syscall_64
      1.34 ±  7%      +0.2        1.50        perf-profile.calltrace.cycles-pp._copy_from_user.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.77 ±  6%      +0.2        1.94        perf-profile.calltrace.cycles-pp.do_smart_update.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.40 ± 58%      +0.2        0.59 ±  2%  perf-profile.calltrace.cycles-pp.__dentry_kill.dentry_kill.dput.__fput.task_work_run
      0.41 ± 57%      +0.2        0.62 ±  2%  perf-profile.calltrace.cycles-pp.perform_atomic_semop.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.00 ±  5%      +0.3        2.27 ±  3%  perf-profile.calltrace.cycles-pp.do_shmat.__x64_sys_shmat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.00 ±  5%      +0.3        2.28 ±  3%  perf-profile.calltrace.cycles-pp.__x64_sys_shmat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.19 ±  5%      +0.4        4.54        perf-profile.calltrace.cycles-pp.ipc_obtain_object_check.do_semtimedop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.13 ±173%      +0.4        0.52        perf-profile.calltrace.cycles-pp.do_mmap.do_shmat.__x64_sys_shmat.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.12 ± 12%      +0.4        1.50 ± 13%  perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write.shm_close.remove_vma
      1.58 ±  9%      +0.4        1.98 ± 10%  perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.shm_close.remove_vma.__do_munmap
      1.62 ±  9%      +0.4        2.04 ± 10%  perf-profile.calltrace.cycles-pp.down_write.shm_close.remove_vma.__do_munmap.ksys_shmdt
      1.67 ±  9%      +0.4        2.10 ±  9%  perf-profile.calltrace.cycles-pp.shm_close.remove_vma.__do_munmap.ksys_shmdt.do_syscall_64
      2.16 ±  8%      +0.4        2.61 ±  8%  perf-profile.calltrace.cycles-pp.remove_vma.__do_munmap.ksys_shmdt.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.69 ±  8%      +0.5        3.17 ±  7%  perf-profile.calltrace.cycles-pp.__do_munmap.ksys_shmdt.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.73 ±  8%      +0.5        3.22 ±  7%  perf-profile.calltrace.cycles-pp.ksys_shmdt.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.02 ±  6%      +0.5        6.53 ±  2%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      8.28 ± 32%      +3.6       11.89 ±  3%  perf-profile.calltrace.cycles-pp.ipcget.__x64_sys_shmget.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.29 ± 32%      +3.6       11.91 ±  3%  perf-profile.calltrace.cycles-pp.__x64_sys_shmget.do_syscall_64.entry_SYSCALL_64_after_hwframe
     49.89 ±  9%      +6.4       56.31        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     50.69 ±  9%      +6.5       57.20        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.49 ± 11%      -0.4        1.14 ±  3%  perf-profile.children.cycles-pp.apic_timer_interrupt
      1.42 ± 11%      -0.3        1.08 ±  3%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      1.04 ±  6%      -0.3        0.78        perf-profile.children.cycles-pp.page_fault
      1.01 ±  5%      -0.3        0.75        perf-profile.children.cycles-pp.__do_page_fault
      1.02 ±  6%      -0.3        0.77        perf-profile.children.cycles-pp.do_page_fault
      0.89 ±  5%      -0.2        0.65        perf-profile.children.cycles-pp.handle_mm_fault
      0.86 ±  5%      -0.2        0.61        perf-profile.children.cycles-pp.__handle_mm_fault
      0.82 ± 13%      -0.2        0.64 ±  5%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.34 ± 23%      -0.2        0.17        perf-profile.children.cycles-pp.menu_select
      0.48 ± 10%      -0.1        0.39 ±  2%  perf-profile.children.cycles-pp.irq_exit
      0.38 ± 12%      -0.1        0.32        perf-profile.children.cycles-pp.tick_sched_timer
      0.11 ± 12%      -0.1        0.05        perf-profile.children.cycles-pp.load_balance
      0.08 ± 10%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.run_rebalance_domains
      0.08 ± 11%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.update_blocked_averages
      0.10 ± 21%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.rebalance_domains
      0.10 ± 18%      -0.0        0.07        perf-profile.children.cycles-pp.swapgs_restore_regs_and_return_to_usermode
      0.07 ± 13%      -0.0        0.05        perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
      0.11 ±  3%      +0.0        0.12 ±  4%  perf-profile.children.cycles-pp.up_write
      0.11 ±  8%      +0.0        0.12 ±  8%  perf-profile.children.cycles-pp.idr_alloc_u32
      0.07 ± 10%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.idr_replace
      0.11 ± 11%      +0.0        0.13 ±  7%  perf-profile.children.cycles-pp.idr_alloc_cyclic
      0.11 ±  6%      +0.0        0.13        perf-profile.children.cycles-pp.shmem_get_unmapped_area
      0.10 ±  8%      +0.0        0.12        perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
      0.25 ±  4%      +0.0        0.27        perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      0.17 ±  4%      +0.0        0.20 ±  2%  perf-profile.children.cycles-pp.shm_destroy
      0.14 ±  5%      +0.0        0.17 ±  5%  perf-profile.children.cycles-pp.get_unmapped_area
      0.10 ± 15%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.task_tick_fair
      0.28 ±  5%      +0.0        0.32        perf-profile.children.cycles-pp.arch_stack_walk
      0.24 ±  7%      +0.0        0.28        perf-profile.children.cycles-pp.newary
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.children.cycles-pp._cond_resched
      1.04 ±  4%      +0.1        1.10 ±  2%  perf-profile.children.cycles-pp.do_smart_wakeup_zero
      0.76 ±  4%      +0.1        0.81        perf-profile.children.cycles-pp.__fput
      0.81 ±  5%      +0.1        0.86 ±  2%  perf-profile.children.cycles-pp.task_work_run
      0.86 ±  4%      +0.1        0.92        perf-profile.children.cycles-pp.exit_to_usermode_loop
      0.76 ±  2%      +0.1        0.82 ±  7%  perf-profile.children.cycles-pp.newseg
      0.32 ±  7%      +0.1        0.39        perf-profile.children.cycles-pp.ipc_addid
      1.47 ±  5%      +0.1        1.54        perf-profile.children.cycles-pp._copy_from_user
      0.35 ±  5%      +0.1        0.43 ±  6%  perf-profile.children.cycles-pp.ksys_semget
      0.57            +0.1        0.66 ±  2%  perf-profile.children.cycles-pp.perform_atomic_semop
      0.28 ±  7%      +0.1        0.39 ± 19%  perf-profile.children.cycles-pp.semctl_down
      0.37 ±  5%      +0.1        0.48 ± 14%  perf-profile.children.cycles-pp.ksys_semctl
      2.10 ±  6%      +0.2        2.27 ±  3%  perf-profile.children.cycles-pp.do_shmat
      2.10 ±  5%      +0.2        2.28 ±  3%  perf-profile.children.cycles-pp.__x64_sys_shmat
      1.74 ±  7%      +0.4        2.10 ±  9%  perf-profile.children.cycles-pp.shm_close
      2.25 ±  6%      +0.4        2.62 ±  8%  perf-profile.children.cycles-pp.remove_vma
      0.08 ± 15%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.16 ± 16%      -0.1        0.11        perf-profile.self.cycles-pp.menu_select
      0.11 ±  4%      -0.0        0.10 ±  5%  perf-profile.self.cycles-pp.wake_up_q
      0.23 ±  3%      +0.0        0.25        perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.13 ±  6%      +0.0        0.15 ±  3%  perf-profile.self.cycles-pp.unwind_next_frame
      0.16 ±  4%      +0.0        0.18 ±  5%  perf-profile.self.cycles-pp.down_write
      0.63 ±  2%      +0.0        0.66        perf-profile.self.cycles-pp.do_smart_wakeup_zero
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.ktime_get_real_seconds
      0.72 ±  4%      +0.0        0.77        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.29 ±  6%      +0.0        0.34        perf-profile.self.cycles-pp._copy_from_user
      0.56            +0.1        0.64        perf-profile.self.cycles-pp.perform_atomic_semop



***************************************************************************************************
lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.6/400t/debian-x86_64-2019-05-14.cgz/300s/lkp-ivb-2ep1/disk_src/reaim/0x42e

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:2     dmesg.WARNING:at#for_ip_interrupt_entry/0x
          2:4          -50%            :2     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
           :4           25%           1:2     dmesg.WARNING:at_ip__fsnotify_parent/0x



***************************************************************************************************
lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 64G memory
=========================================================================================
compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/ucode:
  gcc-7/performance/directio/1HDD/xfs/x86_64-rhel-7.6/hdd/debian-x86_64-2019-05-14.cgz/lkp-ivb-2ep1/MWUL/fxmark/0x42e

commit: 
  4f23dbc1e6 ("locking/rwsem: Implement lock handoff to prevent lock starvation")
  00f3c5a3df ("locking/rwsem: Always release wait_lock before waking up tasks")

4f23dbc1e657951e 00f3c5a3df2c1e3dab14d0dd2b7 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      0.18 ± 17%     +30.0%       0.23        fxmark.hdd_xfs_MWUL_12_directio.iowait_util
      1145            +0.9%       1156        boot-time.idle
    197.75            +2.9%     203.50        vmstat.io.bi
  46933823           +15.4%   54152343 ± 11%  cpuidle.C1.time
    675211            +8.7%     733632 ±  7%  cpuidle.C1.usage
    441822 ± 74%    +107.2%     915380 ± 11%  cpuidle.C1E.usage
      0.87 ±  6%     +14.6%       0.99 ±  5%  iostat.cpu.iowait
     12.52            +3.9%      13.01        iostat.cpu.system
      9725            +1.8%       9900        iostat.sda.wkB/s
    250.50 ±  7%     +50.7%     377.50 ±  4%  slabinfo.numa_policy.active_objs
      1586 ±  2%     +17.4%       1862 ± 11%  slabinfo.proc_inode_cache.active_objs
      2208 ±  3%     +17.4%       2592 ±  7%  slabinfo.proc_inode_cache.num_objs
     15301 ±  3%     +19.6%      18303 ±  4%  softirqs.BLOCK
      9332 ±  4%     +10.0%      10265 ±  5%  softirqs.CPU11.TIMER
      9806 ±  5%     +76.8%      17339 ±  3%  softirqs.CPU12.TIMER
     13390 ± 14%     +19.5%      15996        softirqs.CPU3.RCU
     10236 ±  6%     -18.3%       8364 ±  9%  softirqs.CPU40.TIMER
      5703            +2.3%       5834        proc-vmstat.nr_inactive_file
     24468            +0.8%      24675        proc-vmstat.nr_slab_unreclaimable
      5703            +2.3%       5834        proc-vmstat.nr_zone_inactive_file
      3368 ± 81%    +190.8%       9794 ±  8%  proc-vmstat.numa_pages_migrated
     20369 ± 59%    +164.1%      53802 ± 27%  proc-vmstat.numa_pte_updates
      3368 ± 81%    +190.8%       9794 ±  8%  proc-vmstat.pgmigrate_success
    221862 ± 15%     -24.2%     168110 ±  3%  perf-stat.i.dTLB-load-misses
      3966            +1.6%       4028        perf-stat.i.minor-faults
     19254 ±  7%     +12.3%      21613 ±  6%  perf-stat.i.node-load-misses
      2.21 ± 17%      +0.5        2.72        perf-stat.i.node-store-miss-rate%
     16644 ± 12%     +20.2%      20014 ±  6%  perf-stat.i.node-store-misses
      3966            +1.6%       4028        perf-stat.i.page-faults
    219511 ± 15%     -24.2%     166480 ±  3%  perf-stat.ps.dTLB-load-misses
     19050 ±  7%     +12.4%      21404 ±  6%  perf-stat.ps.node-load-misses
     16480 ± 12%     +20.3%      19830 ±  6%  perf-stat.ps.node-store-misses
     32707 ± 12%     -49.4%      16539 ±  3%  numa-meminfo.node0.Inactive
     14732 ±  3%     -88.7%       1669 ± 16%  numa-meminfo.node0.Inactive(anon)
     16642           -10.0%      14973        numa-meminfo.node0.Mapped
     14864 ±  3%     -88.0%       1778 ± 14%  numa-meminfo.node0.Shmem
    534994           +10.7%     592351        numa-meminfo.node0.Unevictable
      6988 ± 55%    +230.7%      23113        numa-meminfo.node1.Inactive
      1740 ± 32%    +749.7%      14785 ±  2%  numa-meminfo.node1.Inactive(anon)
     14910           +11.8%      16667        numa-meminfo.node1.Mapped
     37981 ±  4%     +14.0%      43311        numa-meminfo.node1.SUnreclaim
      1788 ± 32%    +730.6%      14853        numa-meminfo.node1.Shmem
    587633            -9.7%     530463        numa-meminfo.node1.Unevictable
     71788 ±  4%     +13.3%      81318        interrupts.CAL:Function_call_interrupts
      2387 ± 26%     +78.0%       4249 ± 18%  interrupts.CPU0.CAL:Function_call_interrupts
      2160 ± 18%     +62.1%       3502 ± 16%  interrupts.CPU1.CAL:Function_call_interrupts
      2464 ± 15%     +67.9%       4137 ± 14%  interrupts.CPU2.CAL:Function_call_interrupts
    161.25 ± 20%    +118.6%     352.50 ± 33%  interrupts.CPU21.RES:Rescheduling_interrupts
      1705 ± 14%     +18.9%       2028 ±  4%  interrupts.CPU22.CAL:Function_call_interrupts
      2565 ± 22%     +43.4%       3679        interrupts.CPU3.CAL:Function_call_interrupts
      1151 ± 13%     +21.7%       1401        interrupts.CPU3.RES:Rescheduling_interrupts
    760.00 ± 14%    +143.6%       1851 ± 52%  interrupts.CPU4.RES:Rescheduling_interrupts
      2400 ± 24%     +34.3%       3224 ± 14%  interrupts.CPU6.CAL:Function_call_interrupts
      2011 ±  3%     +32.5%       2665 ±  3%  interrupts.CPU7.CAL:Function_call_interrupts
     59.75 ±150%    +141.0%     144.00 ± 80%  interrupts.CPU8.RES:Rescheduling_interrupts
      3681 ±  3%     -88.7%     416.50 ± 17%  numa-vmstat.node0.nr_inactive_anon
      3714 ±  3%     -88.1%     442.00 ± 15%  numa-vmstat.node0.nr_shmem
    133747           +10.7%     148087        numa-vmstat.node0.nr_unevictable
      3681 ±  3%     -88.7%     416.50 ± 17%  numa-vmstat.node0.nr_zone_inactive_anon
    133747           +10.7%     148087        numa-vmstat.node0.nr_zone_unevictable
    566715 ±  5%     +11.1%     629665        numa-vmstat.node0.numa_local
    434.75 ± 31%    +750.1%       3696 ±  2%  numa-vmstat.node1.nr_inactive_anon
    446.75 ± 32%    +731.2%       3713        numa-vmstat.node1.nr_shmem
      9486 ±  4%     +14.2%      10831        numa-vmstat.node1.nr_slab_unreclaimable
    146908            -9.7%     132615        numa-vmstat.node1.nr_unevictable
    434.75 ± 31%    +750.1%       3696 ±  2%  numa-vmstat.node1.nr_zone_inactive_anon
    146908            -9.7%     132615        numa-vmstat.node1.nr_zone_unevictable
    308139 ± 10%     -17.1%     255441 ±  2%  numa-vmstat.node1.numa_local
     75.28 ± 22%     +88.6%     141.96 ± 12%  sched_debug.cfs_rq:/.load_avg.avg
    210.99 ± 18%     +49.2%     314.68 ±  5%  sched_debug.cfs_rq:/.load_avg.stddev
      4917 ±  6%     -16.5%       4106        sched_debug.cfs_rq:/.min_vruntime.stddev
     15.85 ±110%    +429.8%      83.99 ± 23%  sched_debug.cfs_rq:/.removed.load_avg.avg
     87.02 ±102%    +217.6%     276.43 ± 10%  sched_debug.cfs_rq:/.removed.load_avg.stddev
    725.97 ±110%    +434.6%       3881 ± 23%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
      3986 ±102%    +220.4%      12776 ± 10%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
      5.03 ±106%    +451.8%      27.76 ± 16%  sched_debug.cfs_rq:/.removed.util_avg.avg
    167.00 ±100%    +164.4%     441.50 ± 17%  sched_debug.cfs_rq:/.removed.util_avg.max
     27.99 ±100%    +237.2%      94.38        sched_debug.cfs_rq:/.removed.util_avg.stddev
     -3468           -52.9%      -1633        sched_debug.cfs_rq:/.spread0.min
      4917 ±  6%     -16.5%       4104        sched_debug.cfs_rq:/.spread0.stddev
      7.26 ± 26%     -28.6%       5.18 ±  2%  sched_debug.cpu.cpu_load[1].avg
      8.21 ± 16%     -28.6%       5.86        sched_debug.cpu.cpu_load[2].avg
      9.19 ±  6%     -20.0%       7.35        sched_debug.cpu.cpu_load[3].avg
     10.94 ±  6%     -14.5%       9.35        sched_debug.cpu.cpu_load[4].avg
      2077 ±  5%      +9.0%       2264        sched_debug.cpu.nr_switches.stddev





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


[-- Attachment #2: config-5.2.0-rc5-00067-g00f3c5a --]
[-- Type: text/plain, Size: 175558 bytes --]

#
# Automatically generated file; DO NOT EDIT.
# Linux/x86_64 5.2.0-rc5 Kernel Configuration
#

#
# Compiler: gcc-7 (Debian 7.4.0-9) 7.4.0
#
CONFIG_CC_IS_GCC=y
CONFIG_GCC_VERSION=70400
CONFIG_CLANG_VERSION=0
CONFIG_CC_HAS_ASM_GOTO=y
CONFIG_CC_HAS_WARN_MAYBE_UNINITIALIZED=y
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y
CONFIG_THREAD_INFO_IN_TASK=y

#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
# CONFIG_COMPILE_TEST is not set
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_BUILD_SALT=""
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set
# CONFIG_KERNEL_LZ4 is not set
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_CROSS_MEMORY_ATTACH=y
CONFIG_USELIB=y
CONFIG_AUDIT=y
CONFIG_HAVE_ARCH_AUDITSYSCALL=y
CONFIG_AUDITSYSCALL=y

#
# IRQ subsystem
#
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_GENERIC_IRQ_MIGRATION=y
CONFIG_IRQ_DOMAIN=y
CONFIG_IRQ_SIM=y
CONFIG_IRQ_DOMAIN_HIERARCHY=y
CONFIG_GENERIC_MSI_IRQ=y
CONFIG_GENERIC_MSI_IRQ_DOMAIN=y
CONFIG_GENERIC_IRQ_MATRIX_ALLOCATOR=y
CONFIG_GENERIC_IRQ_RESERVATION_MODE=y
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
# CONFIG_GENERIC_IRQ_DEBUGFS is not set
# end of IRQ subsystem

CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_ARCH_CLOCKSOURCE_INIT=y
CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
CONFIG_GENERIC_CMOS_UPDATE=y

#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
# CONFIG_NO_HZ_IDLE is not set
CONFIG_NO_HZ_FULL=y
CONFIG_CONTEXT_TRACKING=y
# CONFIG_CONTEXT_TRACKING_FORCE is not set
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
# end of Timers subsystem

# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_COUNT=y

#
# CPU/Task time and stats accounting
#
CONFIG_VIRT_CPU_ACCOUNTING=y
CONFIG_VIRT_CPU_ACCOUNTING_GEN=y
# CONFIG_IRQ_TIME_ACCOUNTING is not set
CONFIG_HAVE_SCHED_AVG_IRQ=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
# CONFIG_PSI is not set
# end of CPU/Task time and stats accounting

CONFIG_CPU_ISOLATION=y

#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_RCU_EXPERT is not set
CONFIG_SRCU=y
CONFIG_TREE_SRCU=y
CONFIG_TASKS_RCU=y
CONFIG_RCU_STALL_COMMON=y
CONFIG_RCU_NEED_SEGCBLIST=y
CONFIG_RCU_NOCB_CPU=y
# end of RCU Subsystem

CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
# CONFIG_IKHEADERS is not set
CONFIG_LOG_BUF_SHIFT=20
CONFIG_LOG_CPU_MAX_BUF_SHIFT=12
CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=13
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH=y
CONFIG_ARCH_SUPPORTS_INT128=y
CONFIG_NUMA_BALANCING=y
CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
CONFIG_CGROUPS=y
CONFIG_PAGE_COUNTER=y
CONFIG_MEMCG=y
CONFIG_MEMCG_SWAP=y
CONFIG_MEMCG_SWAP_ENABLED=y
CONFIG_MEMCG_KMEM=y
CONFIG_BLK_CGROUP=y
# CONFIG_DEBUG_BLK_CGROUP is not set
CONFIG_CGROUP_WRITEBACK=y
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_RDMA=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_BPF=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_SOCK_CGROUP_DATA=y
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_SCHED_AUTOGROUP=y
# CONFIG_SYSFS_DEPRECATED is not set
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_RD_LZ4=y
CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_HAVE_UID16=y
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
CONFIG_BPF=y
CONFIG_EXPERT=y
CONFIG_UID16=y
CONFIG_MULTIUSER=y
CONFIG_SGETMASK_SYSCALL=y
CONFIG_SYSFS_SYSCALL=y
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_FHANDLE=y
CONFIG_POSIX_TIMERS=y
CONFIG_PRINTK=y
CONFIG_PRINTK_NMI=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_FUTEX_PI=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_IO_URING=y
CONFIG_ADVISE_SYSCALLS=y
CONFIG_MEMBARRIER=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y
CONFIG_KALLSYMS_BASE_RELATIVE=y
CONFIG_BPF_SYSCALL=y
# CONFIG_BPF_JIT_ALWAYS_ON is not set
CONFIG_USERFAULTFD=y
CONFIG_ARCH_HAS_MEMBARRIER_SYNC_CORE=y
CONFIG_RSEQ=y
# CONFIG_DEBUG_RSEQ is not set
CONFIG_EMBEDDED=y
CONFIG_HAVE_PERF_EVENTS=y
# CONFIG_PC104 is not set

#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
# end of Kernel Performance Events And Counters

CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_SLUB_MEMCG_SYSFS_ON is not set
# CONFIG_COMPAT_BRK is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
CONFIG_SLAB_MERGE_DEFAULT=y
# CONFIG_SLAB_FREELIST_RANDOM is not set
# CONFIG_SLAB_FREELIST_HARDENED is not set
# CONFIG_SHUFFLE_PAGE_ALLOCATOR is not set
CONFIG_SLUB_CPU_PARTIAL=y
CONFIG_SYSTEM_DATA_VERIFICATION=y
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
# end of General setup

CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_MMU=y
CONFIG_ARCH_MMAP_RND_BITS_MIN=28
CONFIG_ARCH_MMAP_RND_BITS_MAX=32
CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=8
CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MAX=16
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_ARCH_HAS_FILTER_PGPROT=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y
CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
CONFIG_ZONE_DMA32=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_HAVE_INTEL_TXT=y
CONFIG_X86_64_SMP=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_PGTABLE_LEVELS=5
CONFIG_CC_HAS_SANE_STACKPROTECTOR=y

#
# Processor type and features
#
CONFIG_ZONE_DMA=y
CONFIG_SMP=y
CONFIG_X86_FEATURE_NAMES=y
CONFIG_X86_X2APIC=y
CONFIG_X86_MPPARSE=y
# CONFIG_GOLDFISH is not set
CONFIG_RETPOLINE=y
CONFIG_X86_CPU_RESCTRL=y
CONFIG_X86_EXTENDED_PLATFORM=y
# CONFIG_X86_NUMACHIP is not set
# CONFIG_X86_VSMP is not set
CONFIG_X86_UV=y
# CONFIG_X86_GOLDFISH is not set
# CONFIG_X86_INTEL_MID is not set
CONFIG_X86_INTEL_LPSS=y
# CONFIG_X86_AMD_PLATFORM_DEVICE is not set
CONFIG_IOSF_MBI=y
# CONFIG_IOSF_MBI_DEBUG is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
# CONFIG_SCHED_OMIT_FRAME_POINTER is not set
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y
CONFIG_PARAVIRT_XXL=y
# CONFIG_PARAVIRT_DEBUG is not set
CONFIG_PARAVIRT_SPINLOCKS=y
CONFIG_XEN=y
CONFIG_XEN_PV=y
CONFIG_XEN_PV_SMP=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_PVHVM_SMP=y
CONFIG_XEN_512GB=y
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
# CONFIG_XEN_PVH is not set
CONFIG_KVM_GUEST=y
# CONFIG_PVH is not set
# CONFIG_KVM_DEBUG_FS is not set
CONFIG_PARAVIRT_TIME_ACCOUNTING=y
CONFIG_PARAVIRT_CLOCK=y
# CONFIG_JAILHOUSE_GUEST is not set
# CONFIG_MK8 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_GENERIC_CPU=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
# CONFIG_PROCESSOR_SELECT is not set
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_HYGON=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
# CONFIG_CALGARY_IOMMU is not set
CONFIG_MAXSMP=y
CONFIG_NR_CPUS_RANGE_BEGIN=8192
CONFIG_NR_CPUS_RANGE_END=8192
CONFIG_NR_CPUS_DEFAULT=8192
CONFIG_NR_CPUS=8192
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
CONFIG_SCHED_MC_PRIO=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCELOG_LEGACY=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
CONFIG_X86_MCE_INJECT=m
CONFIG_X86_THERMAL_VECTOR=y

#
# Performance monitoring
#
CONFIG_PERF_EVENTS_INTEL_UNCORE=y
CONFIG_PERF_EVENTS_INTEL_RAPL=y
CONFIG_PERF_EVENTS_INTEL_CSTATE=y
# CONFIG_PERF_EVENTS_AMD_POWER is not set
# end of Performance monitoring

CONFIG_X86_16BIT=y
CONFIG_X86_ESPFIX64=y
CONFIG_X86_VSYSCALL_EMULATION=y
CONFIG_I8K=m
CONFIG_MICROCODE=y
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_X86_5LEVEL=y
CONFIG_X86_DIRECT_GBPAGES=y
# CONFIG_X86_CPA_STATISTICS is not set
CONFIG_ARCH_HAS_MEM_ENCRYPT=y
# CONFIG_AMD_MEM_ENCRYPT is not set
CONFIG_NUMA=y
CONFIG_AMD_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NODES_SPAN_OTHER_NODES=y
# CONFIG_NUMA_EMU is not set
CONFIG_NODES_SHIFT=10
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_MEMORY_PROBE=y
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_X86_PMEM_LEGACY_DEVICE=y
CONFIG_X86_PMEM_LEGACY=m
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
# CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK is not set
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_ARCH_RANDOM=y
CONFIG_X86_SMAP=y
CONFIG_X86_INTEL_UMIP=y
CONFIG_X86_INTEL_MPX=y
CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS=y
CONFIG_EFI=y
CONFIG_EFI_STUB=y
# CONFIG_EFI_MIXED is not set
CONFIG_SECCOMP=y
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
# CONFIG_KEXEC_FILE is not set
CONFIG_CRASH_DUMP=y
CONFIG_KEXEC_JUMP=y
CONFIG_PHYSICAL_START=0x1000000
CONFIG_RELOCATABLE=y
# CONFIG_RANDOMIZE_BASE is not set
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_DYNAMIC_MEMORY_LAYOUT=y
CONFIG_HOTPLUG_CPU=y
CONFIG_BOOTPARAM_HOTPLUG_CPU0=y
# CONFIG_DEBUG_HOTPLUG_CPU0 is not set
# CONFIG_COMPAT_VDSO is not set
CONFIG_LEGACY_VSYSCALL_EMULATE=y
# CONFIG_LEGACY_VSYSCALL_NONE is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_MODIFY_LDT_SYSCALL=y
CONFIG_HAVE_LIVEPATCH=y
# CONFIG_LIVEPATCH is not set
# end of Processor type and features

CONFIG_ARCH_HAS_ADD_PAGES=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
CONFIG_USE_PERCPU_NUMA_NODE_ID=y
CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK=y
CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION=y
CONFIG_ARCH_ENABLE_THP_MIGRATION=y

#
# Power management and ACPI options
#
CONFIG_ARCH_HIBERNATION_HEADER=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
# CONFIG_SUSPEND_SKIP_SYNC is not set
CONFIG_HIBERNATE_CALLBACKS=y
CONFIG_HIBERNATION=y
CONFIG_PM_STD_PARTITION=""
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
# CONFIG_PM_AUTOSLEEP is not set
# CONFIG_PM_WAKELOCKS is not set
CONFIG_PM=y
CONFIG_PM_DEBUG=y
CONFIG_PM_ADVANCED_DEBUG=y
CONFIG_PM_TEST_SUSPEND=y
CONFIG_PM_SLEEP_DEBUG=y
# CONFIG_DPM_WATCHDOG is not set
# CONFIG_PM_TRACE_RTC is not set
CONFIG_PM_CLK=y
# CONFIG_WQ_POWER_EFFICIENT_DEFAULT is not set
# CONFIG_ENERGY_MODEL is not set
CONFIG_ARCH_SUPPORTS_ACPI=y
CONFIG_ACPI=y
CONFIG_ACPI_LEGACY_TABLES_LOOKUP=y
CONFIG_ARCH_MIGHT_HAVE_ACPI_PDC=y
CONFIG_ACPI_SYSTEM_POWER_STATES_SUPPORT=y
# CONFIG_ACPI_DEBUGGER is not set
CONFIG_ACPI_SPCR_TABLE=y
CONFIG_ACPI_LPIT=y
CONFIG_ACPI_SLEEP=y
# CONFIG_ACPI_PROCFS_POWER is not set
CONFIG_ACPI_REV_OVERRIDE_POSSIBLE=y
CONFIG_ACPI_EC_DEBUGFS=m
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=y
# CONFIG_ACPI_TAD is not set
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_CPU_FREQ_PSS=y
CONFIG_ACPI_PROCESSOR_CSTATE=y
CONFIG_ACPI_PROCESSOR_IDLE=y
CONFIG_ACPI_CPPC_LIB=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_IPMI=m
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_NUMA=y
CONFIG_ARCH_HAS_ACPI_TABLE_UPGRADE=y
CONFIG_ACPI_TABLE_UPGRADE=y
CONFIG_ACPI_DEBUG=y
CONFIG_ACPI_PCI_SLOT=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_HOTPLUG_MEMORY=y
CONFIG_ACPI_HOTPLUG_IOAPIC=y
CONFIG_ACPI_SBS=m
CONFIG_ACPI_HED=y
CONFIG_ACPI_CUSTOM_METHOD=m
CONFIG_ACPI_BGRT=y
# CONFIG_ACPI_REDUCED_HARDWARE_ONLY is not set
CONFIG_ACPI_NFIT=m
# CONFIG_NFIT_SECURITY_DEBUG is not set
# CONFIG_ACPI_HMAT is not set
CONFIG_HAVE_ACPI_APEI=y
CONFIG_HAVE_ACPI_APEI_NMI=y
CONFIG_ACPI_APEI=y
CONFIG_ACPI_APEI_GHES=y
CONFIG_ACPI_APEI_PCIEAER=y
CONFIG_ACPI_APEI_MEMORY_FAILURE=y
CONFIG_ACPI_APEI_EINJ=m
CONFIG_ACPI_APEI_ERST_DEBUG=y
# CONFIG_DPTF_POWER is not set
CONFIG_ACPI_WATCHDOG=y
CONFIG_ACPI_EXTLOG=m
CONFIG_ACPI_ADXL=y
# CONFIG_PMIC_OPREGION is not set
# CONFIG_ACPI_CONFIGFS is not set
CONFIG_X86_PM_TIMER=y
CONFIG_SFI=y

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_GOV_ATTR_SET=y
CONFIG_CPU_FREQ_GOV_COMMON=y
# CONFIG_CPU_FREQ_STAT is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
# CONFIG_CPU_FREQ_GOV_SCHEDUTIL is not set

#
# CPU frequency scaling drivers
#
CONFIG_X86_INTEL_PSTATE=y
CONFIG_X86_PCC_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ_CPB=y
CONFIG_X86_POWERNOW_K8=m
CONFIG_X86_AMD_FREQ_SENSITIVITY=m
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
CONFIG_X86_P4_CLOCKMOD=m

#
# shared options
#
CONFIG_X86_SPEEDSTEP_LIB=m
# end of CPU Frequency scaling

#
# CPU Idle
#
CONFIG_CPU_IDLE=y
# CONFIG_CPU_IDLE_GOV_LADDER is not set
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_CPU_IDLE_GOV_TEO is not set
# end of CPU Idle

CONFIG_INTEL_IDLE=y
# end of Power management and ACPI options

#
# Bus options (PCI etc.)
#
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_XEN=y
CONFIG_MMCONF_FAM10H=y
# CONFIG_PCI_CNB20LE_QUIRK is not set
# CONFIG_ISA_BUS is not set
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y
# CONFIG_X86_SYSFB is not set
# end of Bus options (PCI etc.)

#
# Binary Emulations
#
CONFIG_IA32_EMULATION=y
# CONFIG_X86_X32 is not set
CONFIG_COMPAT_32=y
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
# end of Binary Emulations

CONFIG_HAVE_GENERIC_GUP=y

#
# Firmware Drivers
#
CONFIG_EDD=m
# CONFIG_EDD_OFF is not set
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_DMIID=y
CONFIG_DMI_SYSFS=y
CONFIG_DMI_SCAN_MACHINE_NON_EFI_FALLBACK=y
CONFIG_ISCSI_IBFT_FIND=y
CONFIG_ISCSI_IBFT=m
# CONFIG_FW_CFG_SYSFS is not set
# CONFIG_GOOGLE_FIRMWARE is not set

#
# EFI (Extensible Firmware Interface) Support
#
CONFIG_EFI_VARS=y
CONFIG_EFI_ESRT=y
CONFIG_EFI_VARS_PSTORE=y
CONFIG_EFI_VARS_PSTORE_DEFAULT_DISABLE=y
CONFIG_EFI_RUNTIME_MAP=y
# CONFIG_EFI_FAKE_MEMMAP is not set
CONFIG_EFI_RUNTIME_WRAPPERS=y
# CONFIG_EFI_BOOTLOADER_CONTROL is not set
# CONFIG_EFI_CAPSULE_LOADER is not set
# CONFIG_EFI_TEST is not set
# CONFIG_APPLE_PROPERTIES is not set
# CONFIG_RESET_ATTACK_MITIGATION is not set
# end of EFI (Extensible Firmware Interface) Support

CONFIG_UEFI_CPER=y
CONFIG_UEFI_CPER_X86=y
CONFIG_EFI_EARLYCON=y

#
# Tegra firmware driver
#
# end of Tegra firmware driver
# end of Firmware Drivers

CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_IRQFD=y
CONFIG_HAVE_KVM_IRQ_ROUTING=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
CONFIG_KVM_VFIO=y
CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=y
CONFIG_KVM_COMPAT=y
CONFIG_HAVE_KVM_IRQ_BYPASS=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
CONFIG_KVM_MMU_AUDIT=y
CONFIG_VHOST_NET=m
# CONFIG_VHOST_SCSI is not set
# CONFIG_VHOST_VSOCK is not set
CONFIG_VHOST=m
# CONFIG_VHOST_CROSS_ENDIAN_LEGACY is not set

#
# General architecture-dependent options
#
CONFIG_CRASH_CORE=y
CONFIG_KEXEC_CORE=y
CONFIG_HOTPLUG_SMT=y
CONFIG_OPROFILE=m
CONFIG_OPROFILE_EVENT_MULTIPLEX=y
CONFIG_HAVE_OPROFILE=y
CONFIG_OPROFILE_NMI_TIMER=y
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
# CONFIG_STATIC_KEYS_SELFTEST is not set
CONFIG_OPTPROBES=y
CONFIG_KPROBES_ON_FTRACE=y
CONFIG_UPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_ARCH_USE_BUILTIN_BSWAP=y
CONFIG_KRETPROBES=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_HAVE_FUNCTION_ERROR_INJECTION=y
CONFIG_HAVE_NMI=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_CONTIGUOUS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_ARCH_HAS_FORTIFY_SOURCE=y
CONFIG_ARCH_HAS_SET_MEMORY=y
CONFIG_ARCH_HAS_SET_DIRECT_MAP=y
CONFIG_HAVE_ARCH_THREAD_STRUCT_WHITELIST=y
CONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_RSEQ=y
CONFIG_HAVE_FUNCTION_ARG_ACCESS_API=y
CONFIG_HAVE_CLK=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_HARDLOCKUP_DETECTOR_PERF=y
CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_HAVE_ARCH_JUMP_LABEL_RELATIVE=y
CONFIG_HAVE_RCU_TABLE_FREE=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_ARCH_WANT_COMPAT_IPC_PARSE_VERSION=y
CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP_FILTER=y
CONFIG_HAVE_ARCH_STACKLEAK=y
CONFIG_HAVE_STACKPROTECTOR=y
CONFIG_CC_HAS_STACKPROTECTOR_NONE=y
CONFIG_STACKPROTECTOR=y
CONFIG_STACKPROTECTOR_STRONG=y
CONFIG_HAVE_ARCH_WITHIN_STACK_FRAMES=y
CONFIG_HAVE_CONTEXT_TRACKING=y
CONFIG_HAVE_VIRT_CPU_ACCOUNTING_GEN=y
CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
CONFIG_HAVE_MOVE_PMD=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD=y
CONFIG_HAVE_ARCH_HUGE_VMAP=y
CONFIG_HAVE_ARCH_SOFT_DIRTY=y
CONFIG_HAVE_MOD_ARCH_SPECIFIC=y
CONFIG_MODULES_USE_ELF_RELA=y
CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK=y
CONFIG_ARCH_HAS_ELF_RANDOMIZE=y
CONFIG_HAVE_ARCH_MMAP_RND_BITS=y
CONFIG_HAVE_EXIT_THREAD=y
CONFIG_ARCH_MMAP_RND_BITS=28
CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS=y
CONFIG_ARCH_MMAP_RND_COMPAT_BITS=8
CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES=y
CONFIG_HAVE_COPY_THREAD_TLS=y
CONFIG_HAVE_STACK_VALIDATION=y
CONFIG_HAVE_RELIABLE_STACKTRACE=y
CONFIG_OLD_SIGSUSPEND3=y
CONFIG_COMPAT_OLD_SIGACTION=y
CONFIG_64BIT_TIME=y
CONFIG_COMPAT_32BIT_TIME=y
CONFIG_HAVE_ARCH_VMAP_STACK=y
CONFIG_VMAP_STACK=y
CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=y
CONFIG_STRICT_KERNEL_RWX=y
CONFIG_ARCH_HAS_STRICT_MODULE_RWX=y
CONFIG_STRICT_MODULE_RWX=y
CONFIG_ARCH_HAS_REFCOUNT=y
# CONFIG_REFCOUNT_FULL is not set
CONFIG_HAVE_ARCH_PREL32_RELOCATIONS=y
CONFIG_ARCH_USE_MEMREMAP_PROT=y
# CONFIG_LOCK_EVENT_COUNTS is not set

#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
# end of GCOV-based kernel profiling

CONFIG_PLUGIN_HOSTCC="g++"
CONFIG_HAVE_GCC_PLUGINS=y
CONFIG_GCC_PLUGINS=y

#
# GCC plugins
#
# CONFIG_GCC_PLUGIN_CYC_COMPLEXITY is not set
# CONFIG_GCC_PLUGIN_LATENT_ENTROPY is not set
# CONFIG_GCC_PLUGIN_RANDSTRUCT is not set
# end of GCC plugins
# end of General architecture-dependent options

CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
# CONFIG_MODULE_SIG is not set
# CONFIG_MODULE_COMPRESS is not set
# CONFIG_TRIM_UNUSED_KSYMS is not set
CONFIG_MODULES_TREE_LOOKUP=y
CONFIG_BLOCK=y
CONFIG_BLK_SCSI_REQUEST=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_BSGLIB=y
CONFIG_BLK_DEV_INTEGRITY=y
CONFIG_BLK_DEV_ZONED=y
CONFIG_BLK_DEV_THROTTLING=y
# CONFIG_BLK_DEV_THROTTLING_LOW is not set
# CONFIG_BLK_CMDLINE_PARSER is not set
# CONFIG_BLK_WBT is not set
# CONFIG_BLK_CGROUP_IOLATENCY is not set
CONFIG_BLK_DEBUG_FS=y
CONFIG_BLK_DEBUG_FS_ZONED=y
# CONFIG_BLK_SED_OPAL is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_AIX_PARTITION is not set
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
# CONFIG_ATARI_PARTITION is not set
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
# CONFIG_LDM_PARTITION is not set
CONFIG_SGI_PARTITION=y
# CONFIG_ULTRIX_PARTITION is not set
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
# CONFIG_CMDLINE_PARTITION is not set
# end of Partition Types

CONFIG_BLOCK_COMPAT=y
CONFIG_BLK_MQ_PCI=y
CONFIG_BLK_MQ_VIRTIO=y
CONFIG_BLK_PM=y

#
# IO Schedulers
#
CONFIG_MQ_IOSCHED_DEADLINE=y
CONFIG_MQ_IOSCHED_KYBER=y
# CONFIG_IOSCHED_BFQ is not set
# end of IO Schedulers

CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_PADATA=y
CONFIG_ASN1=y
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_INLINE_READ_UNLOCK=y
CONFIG_INLINE_READ_UNLOCK_IRQ=y
CONFIG_INLINE_WRITE_UNLOCK=y
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_RWSEM_SPIN_ON_OWNER=y
CONFIG_LOCK_SPIN_ON_OWNER=y
CONFIG_ARCH_USE_QUEUED_SPINLOCKS=y
CONFIG_QUEUED_SPINLOCKS=y
CONFIG_ARCH_USE_QUEUED_RWLOCKS=y
CONFIG_QUEUED_RWLOCKS=y
CONFIG_ARCH_HAS_SYNC_CORE_BEFORE_USERMODE=y
CONFIG_ARCH_HAS_SYSCALL_WRAPPER=y
CONFIG_FREEZER=y

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_ELFCORE=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
CONFIG_BINFMT_SCRIPT=y
CONFIG_BINFMT_MISC=m
CONFIG_COREDUMP=y
# end of Executable file formats

#
# Memory Management options
#
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_NEED_MULTIPLE_NODES=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_MEMORY_ISOLATION=y
CONFIG_HAVE_BOOTMEM_INFO_NODE=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_SPARSE=y
# CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set
CONFIG_MEMORY_HOTREMOVE=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_MEMORY_BALLOON=y
CONFIG_BALLOON_COMPACTION=y
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_CONTIG_ALLOC=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_MEMORY_FAILURE=y
CONFIG_HWPOISON_INJECT=m
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set
CONFIG_ARCH_WANTS_THP_SWAP=y
CONFIG_THP_SWAP=y
CONFIG_TRANSPARENT_HUGE_PAGECACHE=y
CONFIG_CLEANCACHE=y
CONFIG_FRONTSWAP=y
CONFIG_CMA=y
# CONFIG_CMA_DEBUG is not set
# CONFIG_CMA_DEBUGFS is not set
CONFIG_CMA_AREAS=7
# CONFIG_MEM_SOFT_DIRTY is not set
CONFIG_ZSWAP=y
CONFIG_ZPOOL=y
CONFIG_ZBUD=y
# CONFIG_Z3FOLD is not set
CONFIG_ZSMALLOC=y
# CONFIG_PGTABLE_MAPPING is not set
# CONFIG_ZSMALLOC_STAT is not set
CONFIG_GENERIC_EARLY_IOREMAP=y
# CONFIG_DEFERRED_STRUCT_PAGE_INIT is not set
CONFIG_IDLE_PAGE_TRACKING=y
CONFIG_ARCH_HAS_ZONE_DEVICE=y
CONFIG_ZONE_DEVICE=y
CONFIG_ARCH_HAS_HMM_MIRROR=y
CONFIG_ARCH_HAS_HMM_DEVICE=y
CONFIG_ARCH_HAS_HMM=y
CONFIG_DEV_PAGEMAP_OPS=y
# CONFIG_HMM_MIRROR is not set
# CONFIG_DEVICE_PRIVATE is not set
# CONFIG_DEVICE_PUBLIC is not set
CONFIG_FRAME_VECTOR=y
CONFIG_ARCH_USES_HIGH_VMA_FLAGS=y
CONFIG_ARCH_HAS_PKEYS=y
# CONFIG_PERCPU_STATS is not set
# CONFIG_GUP_BENCHMARK is not set
CONFIG_ARCH_HAS_PTE_SPECIAL=y
# end of Memory Management options

CONFIG_NET=y
CONFIG_COMPAT_NETLINK_MESSAGES=y
CONFIG_NET_INGRESS=y
CONFIG_NET_EGRESS=y
CONFIG_SKB_EXTENSIONS=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_DIAG=m
CONFIG_UNIX=y
CONFIG_UNIX_SCM=y
CONFIG_UNIX_DIAG=m
# CONFIG_TLS is not set
CONFIG_XFRM=y
CONFIG_XFRM_ALGO=y
CONFIG_XFRM_USER=y
# CONFIG_XFRM_INTERFACE is not set
CONFIG_XFRM_SUB_POLICY=y
CONFIG_XFRM_MIGRATE=y
CONFIG_XFRM_STATISTICS=y
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
CONFIG_NET_KEY_MIGRATE=y
# CONFIG_XDP_SOCKETS is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_FIB_TRIE_STATS=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_ROUTE_CLASSID=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
# CONFIG_IP_PNP_BOOTP is not set
# CONFIG_IP_PNP_RARP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE_DEMUX=m
CONFIG_NET_IP_TUNNEL=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE_COMMON=y
CONFIG_IP_MROUTE=y
CONFIG_IP_MROUTE_MULTIPLE_TABLES=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
CONFIG_SYN_COOKIES=y
CONFIG_NET_IPVTI=m
CONFIG_NET_UDP_TUNNEL=m
CONFIG_NET_FOU=m
CONFIG_NET_FOU_IP_TUNNELS=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
# CONFIG_INET_ESP_OFFLOAD is not set
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
CONFIG_INET_UDP_DIAG=m
# CONFIG_INET_RAW_DIAG is not set
# CONFIG_INET_DIAG_DESTROY is not set
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_TCP_CONG_HSTCP=m
CONFIG_TCP_CONG_HYBLA=m
CONFIG_TCP_CONG_VEGAS=m
# CONFIG_TCP_CONG_NV is not set
CONFIG_TCP_CONG_SCALABLE=m
CONFIG_TCP_CONG_LP=m
CONFIG_TCP_CONG_VENO=m
CONFIG_TCP_CONG_YEAH=m
CONFIG_TCP_CONG_ILLINOIS=m
# CONFIG_TCP_CONG_DCTCP is not set
# CONFIG_TCP_CONG_CDG is not set
# CONFIG_TCP_CONG_BBR is not set
CONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_TCP_MD5SIG=y
CONFIG_IPV6=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
CONFIG_IPV6_OPTIMISTIC_DAD=y
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
# CONFIG_INET6_ESP_OFFLOAD is not set
CONFIG_INET6_IPCOMP=m
CONFIG_IPV6_MIP6=m
# CONFIG_IPV6_ILA is not set
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
CONFIG_IPV6_VTI=m
CONFIG_IPV6_SIT=m
CONFIG_IPV6_SIT_6RD=y
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
# CONFIG_IPV6_GRE is not set
CONFIG_IPV6_FOU=m
CONFIG_IPV6_FOU_TUNNEL=m
CONFIG_IPV6_MULTIPLE_TABLES=y
# CONFIG_IPV6_SUBTREES is not set
CONFIG_IPV6_MROUTE=y
CONFIG_IPV6_MROUTE_MULTIPLE_TABLES=y
CONFIG_IPV6_PIMSM_V2=y
CONFIG_IPV6_SEG6_LWTUNNEL=y
# CONFIG_IPV6_SEG6_HMAC is not set
CONFIG_IPV6_SEG6_BPF=y
CONFIG_NETLABEL=y
CONFIG_NETWORK_SECMARK=y
CONFIG_NET_PTP_CLASSIFY=y
CONFIG_NETWORK_PHY_TIMESTAMPING=y
CONFIG_NETFILTER=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=m

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_INGRESS=y
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_FAMILY_BRIDGE=y
CONFIG_NETFILTER_FAMILY_ARP=y
CONFIG_NETFILTER_NETLINK_ACCT=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NETFILTER_NETLINK_OSF=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_LOG_COMMON=m
# CONFIG_NF_LOG_NETDEV is not set
CONFIG_NETFILTER_CONNCOUNT=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_ZONES=y
CONFIG_NF_CONNTRACK_PROCFS=y
CONFIG_NF_CONNTRACK_EVENTS=y
# CONFIG_NF_CONNTRACK_TIMEOUT is not set
CONFIG_NF_CONNTRACK_TIMESTAMP=y
CONFIG_NF_CONNTRACK_LABELS=y
CONFIG_NF_CT_PROTO_DCCP=y
CONFIG_NF_CT_PROTO_GRE=y
CONFIG_NF_CT_PROTO_SCTP=y
CONFIG_NF_CT_PROTO_UDPLITE=y
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_BROADCAST=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_SNMP=m
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
# CONFIG_NETFILTER_NETLINK_GLUE_CT is not set
CONFIG_NF_NAT=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_SIP=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_REDIRECT=y
CONFIG_NF_NAT_MASQUERADE=y
CONFIG_NETFILTER_SYNPROXY=m
CONFIG_NF_TABLES=m
# CONFIG_NF_TABLES_SET is not set
# CONFIG_NF_TABLES_INET is not set
# CONFIG_NF_TABLES_NETDEV is not set
# CONFIG_NFT_NUMGEN is not set
CONFIG_NFT_CT=m
CONFIG_NFT_COUNTER=m
# CONFIG_NFT_CONNLIMIT is not set
CONFIG_NFT_LOG=m
CONFIG_NFT_LIMIT=m
# CONFIG_NFT_MASQ is not set
# CONFIG_NFT_REDIR is not set
# CONFIG_NFT_TUNNEL is not set
# CONFIG_NFT_OBJREF is not set
# CONFIG_NFT_QUEUE is not set
# CONFIG_NFT_QUOTA is not set
# CONFIG_NFT_REJECT is not set
CONFIG_NFT_COMPAT=m
CONFIG_NFT_HASH=m
# CONFIG_NFT_XFRM is not set
# CONFIG_NFT_SOCKET is not set
# CONFIG_NFT_OSF is not set
# CONFIG_NFT_TPROXY is not set
# CONFIG_NF_FLOW_TABLE is not set
CONFIG_NETFILTER_XTABLES=y

#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m
CONFIG_NETFILTER_XT_SET=m

#
# Xtables targets
#
CONFIG_NETFILTER_XT_TARGET_AUDIT=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_TARGET_CT=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_HMARK=m
CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
CONFIG_NETFILTER_XT_TARGET_LED=m
CONFIG_NETFILTER_XT_TARGET_LOG=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_NAT=m
CONFIG_NETFILTER_XT_TARGET_NETMAP=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
CONFIG_NETFILTER_XT_TARGET_RATEEST=m
CONFIG_NETFILTER_XT_TARGET_REDIRECT=m
CONFIG_NETFILTER_XT_TARGET_MASQUERADE=m
CONFIG_NETFILTER_XT_TARGET_TEE=m
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NETFILTER_XT_TARGET_SECMARK=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m

#
# Xtables matches
#
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
CONFIG_NETFILTER_XT_MATCH_BPF=m
# CONFIG_NETFILTER_XT_MATCH_CGROUP is not set
CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLABEL=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_CPU=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ECN=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
# CONFIG_NETFILTER_XT_MATCH_IPCOMP is not set
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_IPVS=m
CONFIG_NETFILTER_XT_MATCH_L2TP=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_NFACCT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_RATEEST=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_RECENT=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
# CONFIG_NETFILTER_XT_MATCH_SOCKET is not set
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NETFILTER_XT_MATCH_U32=m
# end of Core Netfilter Configuration

CONFIG_IP_SET=m
CONFIG_IP_SET_MAX=256
CONFIG_IP_SET_BITMAP_IP=m
CONFIG_IP_SET_BITMAP_IPMAC=m
CONFIG_IP_SET_BITMAP_PORT=m
CONFIG_IP_SET_HASH_IP=m
# CONFIG_IP_SET_HASH_IPMARK is not set
CONFIG_IP_SET_HASH_IPPORT=m
CONFIG_IP_SET_HASH_IPPORTIP=m
CONFIG_IP_SET_HASH_IPPORTNET=m
# CONFIG_IP_SET_HASH_IPMAC is not set
# CONFIG_IP_SET_HASH_MAC is not set
# CONFIG_IP_SET_HASH_NETPORTNET is not set
CONFIG_IP_SET_HASH_NET=m
# CONFIG_IP_SET_HASH_NETNET is not set
CONFIG_IP_SET_HASH_NETPORT=m
CONFIG_IP_SET_HASH_NETIFACE=m
CONFIG_IP_SET_LIST_SET=m
CONFIG_IP_VS=m
CONFIG_IP_VS_IPV6=y
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12

#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
CONFIG_IP_VS_PROTO_SCTP=y

#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
# CONFIG_IP_VS_FO is not set
# CONFIG_IP_VS_OVF is not set
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
# CONFIG_IP_VS_MH is not set
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m

#
# IPVS SH scheduler
#
CONFIG_IP_VS_SH_TAB_BITS=8

#
# IPVS MH scheduler
#
CONFIG_IP_VS_MH_TAB_INDEX=12

#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m
CONFIG_IP_VS_NFCT=y
CONFIG_IP_VS_PE_SIP=m

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=m
# CONFIG_NF_SOCKET_IPV4 is not set
CONFIG_NF_TPROXY_IPV4=m
# CONFIG_NF_TABLES_IPV4 is not set
# CONFIG_NF_TABLES_ARP is not set
CONFIG_NF_DUP_IPV4=m
# CONFIG_NF_LOG_ARP is not set
CONFIG_NF_LOG_IPV4=m
CONFIG_NF_REJECT_IPV4=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_RPFILTER=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_SYNPROXY=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_SECURITY=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
# end of IP: Netfilter Configuration

#
# IPv6: Netfilter Configuration
#
# CONFIG_NF_SOCKET_IPV6 is not set
CONFIG_NF_TPROXY_IPV6=m
# CONFIG_NF_TABLES_IPV6 is not set
CONFIG_NF_DUP_IPV6=m
CONFIG_NF_REJECT_IPV6=m
CONFIG_NF_LOG_IPV6=m
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_AH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_MH=m
CONFIG_IP6_NF_MATCH_RPFILTER=m
CONFIG_IP6_NF_MATCH_RT=m
# CONFIG_IP6_NF_MATCH_SRH is not set
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_TARGET_SYNPROXY=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_RAW=m
CONFIG_IP6_NF_SECURITY=m
# CONFIG_IP6_NF_NAT is not set
# end of IPv6: Netfilter Configuration

CONFIG_NF_DEFRAG_IPV6=m
# CONFIG_NF_TABLES_BRIDGE is not set
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_IP6=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_NFLOG=m
# CONFIG_BPFILTER is not set
CONFIG_IP_DCCP=m
CONFIG_INET_DCCP_DIAG=m

#
# DCCP CCIDs Configuration
#
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
CONFIG_IP_DCCP_CCID3=y
# CONFIG_IP_DCCP_CCID3_DEBUG is not set
CONFIG_IP_DCCP_TFRC_LIB=y
# end of DCCP CCIDs Configuration

#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
# end of DCCP Kernel Hacking

CONFIG_IP_SCTP=m
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_DEFAULT_COOKIE_HMAC_MD5 is not set
CONFIG_SCTP_DEFAULT_COOKIE_HMAC_SHA1=y
# CONFIG_SCTP_DEFAULT_COOKIE_HMAC_NONE is not set
CONFIG_SCTP_COOKIE_HMAC_MD5=y
CONFIG_SCTP_COOKIE_HMAC_SHA1=y
CONFIG_INET_SCTP_DIAG=m
# CONFIG_RDS is not set
CONFIG_TIPC=m
CONFIG_TIPC_MEDIA_UDP=y
CONFIG_TIPC_DIAG=m
CONFIG_ATM=m
CONFIG_ATM_CLIP=m
# CONFIG_ATM_CLIP_NO_ICMP is not set
CONFIG_ATM_LANE=m
# CONFIG_ATM_MPOA is not set
CONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set
CONFIG_L2TP=m
CONFIG_L2TP_DEBUGFS=m
CONFIG_L2TP_V3=y
CONFIG_L2TP_IP=m
CONFIG_L2TP_ETH=m
CONFIG_STP=m
CONFIG_GARP=m
CONFIG_MRP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
CONFIG_BRIDGE_VLAN_FILTERING=y
CONFIG_HAVE_NET_DSA=y
# CONFIG_NET_DSA is not set
CONFIG_VLAN_8021Q=m
CONFIG_VLAN_8021Q_GVRP=y
CONFIG_VLAN_8021Q_MVRP=y
# CONFIG_DECNET is not set
CONFIG_LLC=m
# CONFIG_LLC2 is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_PHONET is not set
# CONFIG_6LOWPAN is not set
CONFIG_IEEE802154=m
# CONFIG_IEEE802154_NL802154_EXPERIMENTAL is not set
CONFIG_IEEE802154_SOCKET=m
CONFIG_MAC802154=m
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_ATM=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_MULTIQ=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFB=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
# CONFIG_NET_SCH_CBS is not set
# CONFIG_NET_SCH_ETF is not set
# CONFIG_NET_SCH_TAPRIO is not set
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_DRR=m
CONFIG_NET_SCH_MQPRIO=m
# CONFIG_NET_SCH_SKBPRIO is not set
CONFIG_NET_SCH_CHOKE=m
CONFIG_NET_SCH_QFQ=m
CONFIG_NET_SCH_CODEL=m
CONFIG_NET_SCH_FQ_CODEL=m
# CONFIG_NET_SCH_CAKE is not set
# CONFIG_NET_SCH_FQ is not set
# CONFIG_NET_SCH_HHF is not set
# CONFIG_NET_SCH_PIE is not set
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_SCH_PLUG=m
# CONFIG_NET_SCH_DEFAULT is not set

#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_FLOW=m
CONFIG_NET_CLS_CGROUP=y
CONFIG_NET_CLS_BPF=m
# CONFIG_NET_CLS_FLOWER is not set
# CONFIG_NET_CLS_MATCHALL is not set
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
# CONFIG_NET_EMATCH_CANID is not set
CONFIG_NET_EMATCH_IPSET=m
# CONFIG_NET_EMATCH_IPT is not set
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
# CONFIG_NET_ACT_SAMPLE is not set
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_NAT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_ACT_SKBEDIT=m
CONFIG_NET_ACT_CSUM=m
# CONFIG_NET_ACT_VLAN is not set
# CONFIG_NET_ACT_BPF is not set
# CONFIG_NET_ACT_CONNMARK is not set
# CONFIG_NET_ACT_SKBMOD is not set
# CONFIG_NET_ACT_IFE is not set
# CONFIG_NET_ACT_TUNNEL_KEY is not set
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y
CONFIG_DCB=y
CONFIG_DNS_RESOLVER=m
# CONFIG_BATMAN_ADV is not set
CONFIG_OPENVSWITCH=m
CONFIG_OPENVSWITCH_GRE=m
CONFIG_OPENVSWITCH_VXLAN=m
CONFIG_VSOCKETS=m
CONFIG_VSOCKETS_DIAG=m
CONFIG_VMWARE_VMCI_VSOCKETS=m
# CONFIG_VIRTIO_VSOCKETS is not set
# CONFIG_HYPERV_VSOCKETS is not set
CONFIG_NETLINK_DIAG=m
CONFIG_MPLS=y
CONFIG_NET_MPLS_GSO=m
CONFIG_MPLS_ROUTING=m
CONFIG_MPLS_IPTUNNEL=m
CONFIG_NET_NSH=m
# CONFIG_HSR is not set
# CONFIG_NET_SWITCHDEV is not set
CONFIG_NET_L3_MASTER_DEV=y
# CONFIG_NET_NCSI is not set
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
CONFIG_CGROUP_NET_PRIO=y
CONFIG_CGROUP_NET_CLASSID=y
CONFIG_NET_RX_BUSY_POLL=y
CONFIG_BQL=y
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_NET_FLOW_LIMIT=y

#
# Network testing
#
CONFIG_NET_PKTGEN=m
CONFIG_NET_DROP_MONITOR=y
# end of Network testing
# end of Networking options

# CONFIG_HAMRADIO is not set
CONFIG_CAN=m
CONFIG_CAN_RAW=m
CONFIG_CAN_BCM=m
CONFIG_CAN_GW=m

#
# CAN Device Drivers
#
CONFIG_CAN_VCAN=m
# CONFIG_CAN_VXCAN is not set
# CONFIG_CAN_SLCAN is not set
CONFIG_CAN_DEV=m
CONFIG_CAN_CALC_BITTIMING=y
# CONFIG_CAN_C_CAN is not set
# CONFIG_CAN_CC770 is not set
# CONFIG_CAN_IFI_CANFD is not set
# CONFIG_CAN_M_CAN is not set
# CONFIG_CAN_PEAK_PCIEFD is not set
# CONFIG_CAN_SJA1000 is not set
# CONFIG_CAN_SOFTING is not set

#
# CAN SPI interfaces
#
# CONFIG_CAN_HI311X is not set
# CONFIG_CAN_MCP251X is not set
# end of CAN SPI interfaces

#
# CAN USB interfaces
#
# CONFIG_CAN_8DEV_USB is not set
# CONFIG_CAN_EMS_USB is not set
# CONFIG_CAN_ESD_USB2 is not set
# CONFIG_CAN_GS_USB is not set
# CONFIG_CAN_KVASER_USB is not set
# CONFIG_CAN_MCBA_USB is not set
# CONFIG_CAN_PEAK_USB is not set
# CONFIG_CAN_UCAN is not set
# end of CAN USB interfaces

# CONFIG_CAN_DEBUG_DEVICES is not set
# end of CAN Device Drivers

# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
# CONFIG_AF_KCM is not set
CONFIG_STREAM_PARSER=y
CONFIG_FIB_RULES=y
CONFIG_WIRELESS=y
CONFIG_WIRELESS_EXT=y
CONFIG_WEXT_CORE=y
CONFIG_WEXT_PROC=y
CONFIG_WEXT_PRIV=y
CONFIG_CFG80211=m
# CONFIG_NL80211_TESTMODE is not set
# CONFIG_CFG80211_DEVELOPER_WARNINGS is not set
# CONFIG_CFG80211_CERTIFICATION_ONUS is not set
CONFIG_CFG80211_REQUIRE_SIGNED_REGDB=y
CONFIG_CFG80211_USE_KERNEL_REGDB_KEYS=y
CONFIG_CFG80211_DEFAULT_PS=y
# CONFIG_CFG80211_DEBUGFS is not set
CONFIG_CFG80211_CRDA_SUPPORT=y
CONFIG_CFG80211_WEXT=y
CONFIG_LIB80211=m
# CONFIG_LIB80211_DEBUG is not set
CONFIG_MAC80211=m
CONFIG_MAC80211_HAS_RC=y
CONFIG_MAC80211_RC_MINSTREL=y
CONFIG_MAC80211_RC_DEFAULT_MINSTREL=y
CONFIG_MAC80211_RC_DEFAULT="minstrel_ht"
CONFIG_MAC80211_MESH=y
CONFIG_MAC80211_LEDS=y
CONFIG_MAC80211_DEBUGFS=y
# CONFIG_MAC80211_MESSAGE_TRACING is not set
# CONFIG_MAC80211_DEBUG_MENU is not set
CONFIG_MAC80211_STA_HASH_MAX_SIZE=0
# CONFIG_WIMAX is not set
CONFIG_RFKILL=m
CONFIG_RFKILL_LEDS=y
CONFIG_RFKILL_INPUT=y
# CONFIG_RFKILL_GPIO is not set
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
# CONFIG_NET_9P_XEN is not set
# CONFIG_NET_9P_DEBUG is not set
# CONFIG_CAIF is not set
# CONFIG_CEPH_LIB is not set
# CONFIG_NFC is not set
# CONFIG_PSAMPLE is not set
# CONFIG_NET_IFE is not set
CONFIG_LWTUNNEL=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_DST_CACHE=y
CONFIG_GRO_CELLS=y
CONFIG_NET_SOCK_MSG=y
CONFIG_NET_DEVLINK=y
CONFIG_FAILOVER=y
CONFIG_HAVE_EBPF_JIT=y

#
# Device Drivers
#
CONFIG_HAVE_EISA=y
# CONFIG_EISA is not set
CONFIG_HAVE_PCI=y
CONFIG_PCI=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=y
CONFIG_PCIEAER=y
CONFIG_PCIEAER_INJECT=m
CONFIG_PCIE_ECRC=y
CONFIG_PCIEASPM=y
# CONFIG_PCIEASPM_DEBUG is not set
CONFIG_PCIEASPM_DEFAULT=y
# CONFIG_PCIEASPM_POWERSAVE is not set
# CONFIG_PCIEASPM_POWER_SUPERSAVE is not set
# CONFIG_PCIEASPM_PERFORMANCE is not set
CONFIG_PCIE_PME=y
# CONFIG_PCIE_DPC is not set
# CONFIG_PCIE_PTM is not set
# CONFIG_PCIE_BW is not set
CONFIG_PCI_MSI=y
CONFIG_PCI_MSI_IRQ_DOMAIN=y
CONFIG_PCI_QUIRKS=y
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_REALLOC_ENABLE_AUTO is not set
CONFIG_PCI_STUB=y
# CONFIG_PCI_PF_STUB is not set
# CONFIG_XEN_PCIDEV_FRONTEND is not set
CONFIG_PCI_ATS=y
CONFIG_PCI_LOCKLESS_CONFIG=y
CONFIG_PCI_IOV=y
CONFIG_PCI_PRI=y
CONFIG_PCI_PASID=y
# CONFIG_PCI_P2PDMA is not set
CONFIG_PCI_LABEL=y
# CONFIG_PCI_HYPERV is not set
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_ACPI=y
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
# CONFIG_HOTPLUG_PCI_CPCI is not set
# CONFIG_HOTPLUG_PCI_SHPC is not set

#
# PCI controller drivers
#

#
# Cadence PCIe controllers support
#
# end of Cadence PCIe controllers support

# CONFIG_VMD is not set

#
# DesignWare PCI Core Support
#
# CONFIG_PCIE_DW_PLAT_HOST is not set
# CONFIG_PCI_MESON is not set
# end of DesignWare PCI Core Support
# end of PCI controller drivers

#
# PCI Endpoint
#
# CONFIG_PCI_ENDPOINT is not set
# end of PCI Endpoint

#
# PCI switch controller drivers
#
# CONFIG_PCI_SW_SWITCHTEC is not set
# end of PCI switch controller drivers

CONFIG_PCCARD=y
# CONFIG_PCMCIA is not set
CONFIG_CARDBUS=y

#
# PC-card bridges
#
CONFIG_YENTA=m
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
# CONFIG_RAPIDIO is not set

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER=y
CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y

#
# Firmware loader
#
CONFIG_FW_LOADER=y
CONFIG_EXTRA_FIRMWARE=""
CONFIG_FW_LOADER_USER_HELPER=y
CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
# end of Firmware loader

CONFIG_ALLOW_DEV_COREDUMP=y
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_DEBUG_TEST_DRIVER_REMOVE is not set
# CONFIG_TEST_ASYNC_DRIVER_PROBE is not set
CONFIG_SYS_HYPERVISOR=y
CONFIG_GENERIC_CPU_AUTOPROBE=y
CONFIG_GENERIC_CPU_VULNERABILITIES=y
CONFIG_REGMAP=y
CONFIG_REGMAP_I2C=y
CONFIG_REGMAP_SPI=y
CONFIG_DMA_SHARED_BUFFER=y
# CONFIG_DMA_FENCE_TRACE is not set
# end of Generic Driver Options

#
# Bus devices
#
# end of Bus devices

CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
# CONFIG_GNSS is not set
CONFIG_MTD=m
# CONFIG_MTD_TESTS is not set
# CONFIG_MTD_CMDLINE_PARTS is not set
# CONFIG_MTD_AR7_PARTS is not set

#
# Partition parsers
#
# CONFIG_MTD_REDBOOT_PARTS is not set
# end of Partition parsers

#
# User Modules And Translation Layers
#
CONFIG_MTD_BLKDEVS=m
CONFIG_MTD_BLOCK=m
# CONFIG_MTD_BLOCK_RO is not set
# CONFIG_FTL is not set
# CONFIG_NFTL is not set
# CONFIG_INFTL is not set
# CONFIG_RFD_FTL is not set
# CONFIG_SSFDC is not set
# CONFIG_SM_FTL is not set
# CONFIG_MTD_OOPS is not set
# CONFIG_MTD_SWAP is not set
# CONFIG_MTD_PARTITIONED_MASTER is not set

#
# RAM/ROM/Flash chip drivers
#
# CONFIG_MTD_CFI is not set
# CONFIG_MTD_JEDECPROBE is not set
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
# CONFIG_MTD_RAM is not set
# CONFIG_MTD_ROM is not set
# CONFIG_MTD_ABSENT is not set
# end of RAM/ROM/Flash chip drivers

#
# Mapping drivers for chip access
#
# CONFIG_MTD_COMPLEX_MAPPINGS is not set
# CONFIG_MTD_INTEL_VR_NOR is not set
# CONFIG_MTD_PLATRAM is not set
# end of Mapping drivers for chip access

#
# Self-contained MTD device drivers
#
# CONFIG_MTD_PMC551 is not set
# CONFIG_MTD_DATAFLASH is not set
# CONFIG_MTD_MCHP23K256 is not set
# CONFIG_MTD_SST25L is not set
# CONFIG_MTD_SLRAM is not set
# CONFIG_MTD_PHRAM is not set
# CONFIG_MTD_MTDRAM is not set
# CONFIG_MTD_BLOCK2MTD is not set

#
# Disk-On-Chip Device Drivers
#
# CONFIG_MTD_DOCG3 is not set
# end of Self-contained MTD device drivers

# CONFIG_MTD_ONENAND is not set
# CONFIG_MTD_RAW_NAND is not set
# CONFIG_MTD_SPI_NAND is not set

#
# LPDDR & LPDDR2 PCM memory drivers
#
# CONFIG_MTD_LPDDR is not set
# end of LPDDR & LPDDR2 PCM memory drivers

# CONFIG_MTD_SPI_NOR is not set
CONFIG_MTD_UBI=m
CONFIG_MTD_UBI_WL_THRESHOLD=4096
CONFIG_MTD_UBI_BEB_LIMIT=20
# CONFIG_MTD_UBI_FASTMAP is not set
# CONFIG_MTD_UBI_GLUEBI is not set
# CONFIG_MTD_UBI_BLOCK is not set
# CONFIG_OF is not set
CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_SERIAL=m
# CONFIG_PARPORT_PC_FIFO is not set
# CONFIG_PARPORT_PC_SUPERIO is not set
# CONFIG_PARPORT_AX88796 is not set
CONFIG_PARPORT_1284=y
CONFIG_PARPORT_NOT_PC=y
CONFIG_PNP=y
# CONFIG_PNP_DEBUG_MESSAGES is not set

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_NULL_BLK=m
CONFIG_BLK_DEV_NULL_BLK_FAULT_INJECTION=y
CONFIG_BLK_DEV_FD=m
CONFIG_CDROM=m
# CONFIG_PARIDE is not set
CONFIG_BLK_DEV_PCIESSD_MTIP32XX=m
# CONFIG_ZRAM is not set
# CONFIG_BLK_DEV_UMEM is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_LOOP_MIN_COUNT=0
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
# CONFIG_BLK_DEV_DRBD is not set
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SKD is not set
CONFIG_BLK_DEV_SX8=m
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=16384
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
CONFIG_ATA_OVER_ETH=m
CONFIG_XEN_BLKDEV_FRONTEND=m
# CONFIG_XEN_BLKDEV_BACKEND is not set
CONFIG_VIRTIO_BLK=y
# CONFIG_VIRTIO_BLK_SCSI is not set
# CONFIG_BLK_DEV_RBD is not set
CONFIG_BLK_DEV_RSXX=m

#
# NVME Support
#
CONFIG_NVME_CORE=m
CONFIG_BLK_DEV_NVME=m
CONFIG_NVME_MULTIPATH=y
CONFIG_NVME_FABRICS=m
# CONFIG_NVME_FC is not set
# CONFIG_NVME_TCP is not set
CONFIG_NVME_TARGET=m
CONFIG_NVME_TARGET_LOOP=m
# CONFIG_NVME_TARGET_FC is not set
# CONFIG_NVME_TARGET_TCP is not set
# end of NVME Support

#
# Misc devices
#
CONFIG_SENSORS_LIS3LV02D=m
# CONFIG_AD525X_DPOT is not set
# CONFIG_DUMMY_IRQ is not set
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
CONFIG_SGI_IOC4=m
CONFIG_TIFM_CORE=m
CONFIG_TIFM_7XX1=m
# CONFIG_ICS932S401 is not set
CONFIG_ENCLOSURE_SERVICES=m
CONFIG_SGI_XP=m
CONFIG_HP_ILO=m
CONFIG_SGI_GRU=m
# CONFIG_SGI_GRU_DEBUG is not set
CONFIG_APDS9802ALS=m
CONFIG_ISL29003=m
CONFIG_ISL29020=m
CONFIG_SENSORS_TSL2550=m
CONFIG_SENSORS_BH1770=m
CONFIG_SENSORS_APDS990X=m
# CONFIG_HMC6352 is not set
# CONFIG_DS1682 is not set
CONFIG_VMWARE_BALLOON=m
# CONFIG_USB_SWITCH_FSA9480 is not set
# CONFIG_LATTICE_ECP3_CONFIG is not set
# CONFIG_SRAM is not set
# CONFIG_PCI_ENDPOINT_TEST is not set
CONFIG_PVPANIC=y
# CONFIG_C2PORT is not set

#
# EEPROM support
#
CONFIG_EEPROM_AT24=m
# CONFIG_EEPROM_AT25 is not set
CONFIG_EEPROM_LEGACY=m
CONFIG_EEPROM_MAX6875=m
CONFIG_EEPROM_93CX6=m
# CONFIG_EEPROM_93XX46 is not set
# CONFIG_EEPROM_IDT_89HPESX is not set
# CONFIG_EEPROM_EE1004 is not set
# end of EEPROM support

CONFIG_CB710_CORE=m
# CONFIG_CB710_DEBUG is not set
CONFIG_CB710_DEBUG_ASSUMPTIONS=y

#
# Texas Instruments shared transport line discipline
#
# CONFIG_TI_ST is not set
# end of Texas Instruments shared transport line discipline

CONFIG_SENSORS_LIS3_I2C=m
CONFIG_ALTERA_STAPL=m
CONFIG_INTEL_MEI=y
CONFIG_INTEL_MEI_ME=y
# CONFIG_INTEL_MEI_TXE is not set
# CONFIG_INTEL_MEI_HDCP is not set
CONFIG_VMWARE_VMCI=m

#
# Intel MIC & related support
#

#
# Intel MIC Bus Driver
#
# CONFIG_INTEL_MIC_BUS is not set

#
# SCIF Bus Driver
#
# CONFIG_SCIF_BUS is not set

#
# VOP Bus Driver
#
# CONFIG_VOP_BUS is not set

#
# Intel MIC Host Driver
#

#
# Intel MIC Card Driver
#

#
# SCIF Driver
#

#
# Intel MIC Coprocessor State Management (COSM) Drivers
#

#
# VOP Driver
#
# end of Intel MIC & related support

# CONFIG_GENWQE is not set
# CONFIG_ECHO is not set
# CONFIG_MISC_ALCOR_PCI is not set
# CONFIG_MISC_RTSX_PCI is not set
# CONFIG_MISC_RTSX_USB is not set
# CONFIG_HABANA_AI is not set
# end of Misc devices

CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
CONFIG_CHR_DEV_ST=m
CONFIG_CHR_DEV_OSST=m
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=m
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_ENCLOSURE=m
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
CONFIG_SCSI_SAS_LIBSAS=m
CONFIG_SCSI_SAS_ATA=y
CONFIG_SCSI_SAS_HOST_SMP=y
CONFIG_SCSI_SRP_ATTRS=m
# end of SCSI Transports

CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=m
CONFIG_ISCSI_BOOT_SYSFS=m
CONFIG_SCSI_CXGB3_ISCSI=m
CONFIG_SCSI_CXGB4_ISCSI=m
CONFIG_SCSI_BNX2_ISCSI=m
CONFIG_SCSI_BNX2X_FCOE=m
CONFIG_BE2ISCSI=m
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
CONFIG_SCSI_HPSA=m
CONFIG_SCSI_3W_9XXX=m
CONFIG_SCSI_3W_SAS=m
# CONFIG_SCSI_ACARD is not set
CONFIG_SCSI_AACRAID=m
# CONFIG_SCSI_AIC7XXX is not set
CONFIG_SCSI_AIC79XX=m
CONFIG_AIC79XX_CMDS_PER_DEVICE=4
CONFIG_AIC79XX_RESET_DELAY_MS=15000
# CONFIG_AIC79XX_DEBUG_ENABLE is not set
CONFIG_AIC79XX_DEBUG_MASK=0
# CONFIG_AIC79XX_REG_PRETTY_PRINT is not set
# CONFIG_SCSI_AIC94XX is not set
CONFIG_SCSI_MVSAS=m
# CONFIG_SCSI_MVSAS_DEBUG is not set
CONFIG_SCSI_MVSAS_TASKLET=y
CONFIG_SCSI_MVUMI=m
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
CONFIG_SCSI_ARCMSR=m
# CONFIG_SCSI_ESAS2R is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
CONFIG_MEGARAID_SAS=m
CONFIG_SCSI_MPT3SAS=m
CONFIG_SCSI_MPT2SAS_MAX_SGE=128
CONFIG_SCSI_MPT3SAS_MAX_SGE=128
CONFIG_SCSI_MPT2SAS=m
# CONFIG_SCSI_SMARTPQI is not set
CONFIG_SCSI_UFSHCD=m
CONFIG_SCSI_UFSHCD_PCI=m
# CONFIG_SCSI_UFS_DWC_TC_PCI is not set
# CONFIG_SCSI_UFSHCD_PLATFORM is not set
# CONFIG_SCSI_UFS_BSG is not set
CONFIG_SCSI_HPTIOP=m
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_MYRB is not set
# CONFIG_SCSI_MYRS is not set
CONFIG_VMWARE_PVSCSI=m
# CONFIG_XEN_SCSI_FRONTEND is not set
CONFIG_HYPERV_STORAGE=m
CONFIG_LIBFC=m
CONFIG_LIBFCOE=m
CONFIG_FCOE=m
CONFIG_FCOE_FNIC=m
# CONFIG_SCSI_SNIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_GDTH is not set
CONFIG_SCSI_ISCI=m
# CONFIG_SCSI_IPS is not set
CONFIG_SCSI_INITIO=m
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
CONFIG_SCSI_STEX=m
# CONFIG_SCSI_SYM53C8XX_2 is not set
CONFIG_SCSI_IPR=m
CONFIG_SCSI_IPR_TRACE=y
CONFIG_SCSI_IPR_DUMP=y
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLA_FC=m
# CONFIG_TCM_QLA2XXX is not set
CONFIG_SCSI_QLA_ISCSI=m
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_AM53C974 is not set
# CONFIG_SCSI_WD719X is not set
CONFIG_SCSI_DEBUG=m
CONFIG_SCSI_PMCRAID=m
CONFIG_SCSI_PM8001=m
# CONFIG_SCSI_BFA_FC is not set
CONFIG_SCSI_VIRTIO=m
CONFIG_SCSI_CHELSIO_FCOE=m
CONFIG_SCSI_DH=y
CONFIG_SCSI_DH_RDAC=y
CONFIG_SCSI_DH_HP_SW=y
CONFIG_SCSI_DH_EMC=y
CONFIG_SCSI_DH_ALUA=y
# end of SCSI device support

CONFIG_ATA=m
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
# CONFIG_SATA_ZPODD is not set
CONFIG_SATA_PMP=y

#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=m
CONFIG_SATA_MOBILE_LPM_POLICY=0
CONFIG_SATA_AHCI_PLATFORM=m
# CONFIG_SATA_INIC162X is not set
CONFIG_SATA_ACARD_AHCI=m
CONFIG_SATA_SIL24=m
CONFIG_ATA_SFF=y

#
# SFF controllers with custom DMA interface
#
CONFIG_PDC_ADMA=m
CONFIG_SATA_QSTOR=m
CONFIG_SATA_SX4=m
CONFIG_ATA_BMDMA=y

#
# SATA SFF controllers with BMDMA
#
CONFIG_ATA_PIIX=m
# CONFIG_SATA_DWC is not set
CONFIG_SATA_MV=m
CONFIG_SATA_NV=m
CONFIG_SATA_PROMISE=m
CONFIG_SATA_SIL=m
CONFIG_SATA_SIS=m
CONFIG_SATA_SVW=m
CONFIG_SATA_ULI=m
CONFIG_SATA_VIA=m
CONFIG_SATA_VITESSE=m

#
# PATA SFF controllers with BMDMA
#
CONFIG_PATA_ALI=m
CONFIG_PATA_AMD=m
CONFIG_PATA_ARTOP=m
CONFIG_PATA_ATIIXP=m
CONFIG_PATA_ATP867X=m
CONFIG_PATA_CMD64X=m
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
CONFIG_PATA_HPT366=m
CONFIG_PATA_HPT37X=m
CONFIG_PATA_HPT3X2N=m
CONFIG_PATA_HPT3X3=m
# CONFIG_PATA_HPT3X3_DMA is not set
CONFIG_PATA_IT8213=m
CONFIG_PATA_IT821X=m
CONFIG_PATA_JMICRON=m
CONFIG_PATA_MARVELL=m
CONFIG_PATA_NETCELL=m
CONFIG_PATA_NINJA32=m
# CONFIG_PATA_NS87415 is not set
CONFIG_PATA_OLDPIIX=m
# CONFIG_PATA_OPTIDMA is not set
CONFIG_PATA_PDC2027X=m
CONFIG_PATA_PDC_OLD=m
# CONFIG_PATA_RADISYS is not set
CONFIG_PATA_RDC=m
CONFIG_PATA_SCH=m
CONFIG_PATA_SERVERWORKS=m
CONFIG_PATA_SIL680=m
CONFIG_PATA_SIS=m
CONFIG_PATA_TOSHIBA=m
# CONFIG_PATA_TRIFLEX is not set
CONFIG_PATA_VIA=m
# CONFIG_PATA_WINBOND is not set

#
# PIO-only SFF controllers
#
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_PLATFORM is not set
# CONFIG_PATA_RZ1000 is not set

#
# Generic fallback / legacy drivers
#
CONFIG_PATA_ACPI=m
CONFIG_ATA_GENERIC=m
# CONFIG_PATA_LEGACY is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
# CONFIG_MD_CLUSTER is not set
# CONFIG_BCACHE is not set
CONFIG_BLK_DEV_DM_BUILTIN=y
CONFIG_BLK_DEV_DM=m
CONFIG_DM_DEBUG=y
CONFIG_DM_BUFIO=m
# CONFIG_DM_DEBUG_BLOCK_MANAGER_LOCKING is not set
CONFIG_DM_BIO_PRISON=m
CONFIG_DM_PERSISTENT_DATA=m
# CONFIG_DM_UNSTRIPED is not set
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_THIN_PROVISIONING=m
CONFIG_DM_CACHE=m
CONFIG_DM_CACHE_SMQ=m
# CONFIG_DM_WRITECACHE is not set
# CONFIG_DM_ERA is not set
CONFIG_DM_MIRROR=m
CONFIG_DM_LOG_USERSPACE=m
CONFIG_DM_RAID=m
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_MULTIPATH_QL=m
CONFIG_DM_MULTIPATH_ST=m
CONFIG_DM_DELAY=m
# CONFIG_DM_DUST is not set
CONFIG_DM_UEVENT=y
CONFIG_DM_FLAKEY=m
CONFIG_DM_VERITY=m
# CONFIG_DM_VERITY_FEC is not set
CONFIG_DM_SWITCH=m
CONFIG_DM_LOG_WRITES=m
# CONFIG_DM_INTEGRITY is not set
# CONFIG_DM_ZONED is not set
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
CONFIG_TCM_PSCSI=m
# CONFIG_TCM_USER2 is not set
CONFIG_LOOPBACK_TARGET=m
CONFIG_TCM_FC=m
CONFIG_ISCSI_TARGET=m
# CONFIG_ISCSI_TARGET_CXGB4 is not set
# CONFIG_SBP_TARGET is not set
CONFIG_FUSION=y
CONFIG_FUSION_SPI=m
# CONFIG_FUSION_FC is not set
CONFIG_FUSION_SAS=m
CONFIG_FUSION_MAX_SGE=128
CONFIG_FUSION_CTL=m
CONFIG_FUSION_LOGGING=y

#
# IEEE 1394 (FireWire) support
#
CONFIG_FIREWIRE=m
CONFIG_FIREWIRE_OHCI=m
CONFIG_FIREWIRE_SBP2=m
CONFIG_FIREWIRE_NET=m
# CONFIG_FIREWIRE_NOSY is not set
# end of IEEE 1394 (FireWire) support

CONFIG_MACINTOSH_DRIVERS=y
CONFIG_MAC_EMUMOUSEBTN=y
CONFIG_NETDEVICES=y
CONFIG_MII=y
CONFIG_NET_CORE=y
CONFIG_BONDING=m
CONFIG_DUMMY=m
# CONFIG_EQUALIZER is not set
CONFIG_NET_FC=y
CONFIG_IFB=m
CONFIG_NET_TEAM=m
CONFIG_NET_TEAM_MODE_BROADCAST=m
CONFIG_NET_TEAM_MODE_ROUNDROBIN=m
CONFIG_NET_TEAM_MODE_RANDOM=m
CONFIG_NET_TEAM_MODE_ACTIVEBACKUP=m
CONFIG_NET_TEAM_MODE_LOADBALANCE=m
CONFIG_MACVLAN=m
CONFIG_MACVTAP=m
# CONFIG_IPVLAN is not set
CONFIG_VXLAN=m
# CONFIG_GENEVE is not set
# CONFIG_GTP is not set
CONFIG_MACSEC=y
CONFIG_NETCONSOLE=m
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_TUN=m
CONFIG_TAP=m
# CONFIG_TUN_VNET_CROSS_LE is not set
CONFIG_VETH=m
CONFIG_VIRTIO_NET=y
CONFIG_NLMON=m
CONFIG_NET_VRF=y
# CONFIG_ARCNET is not set
# CONFIG_ATM_DRIVERS is not set

#
# CAIF transport drivers
#

#
# Distributed Switch Architecture drivers
#
# end of Distributed Switch Architecture drivers

CONFIG_ETHERNET=y
CONFIG_MDIO=y
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_NET_VENDOR_ADAPTEC is not set
CONFIG_NET_VENDOR_AGERE=y
# CONFIG_ET131X is not set
CONFIG_NET_VENDOR_ALACRITECH=y
# CONFIG_SLICOSS is not set
# CONFIG_NET_VENDOR_ALTEON is not set
# CONFIG_ALTERA_TSE is not set
CONFIG_NET_VENDOR_AMAZON=y
# CONFIG_ENA_ETHERNET is not set
# CONFIG_NET_VENDOR_AMD is not set
CONFIG_NET_VENDOR_AQUANTIA=y
# CONFIG_AQTION is not set
CONFIG_NET_VENDOR_ARC=y
CONFIG_NET_VENDOR_ATHEROS=y
CONFIG_ATL2=m
CONFIG_ATL1=m
CONFIG_ATL1E=m
CONFIG_ATL1C=m
CONFIG_ALX=m
# CONFIG_NET_VENDOR_AURORA is not set
CONFIG_NET_VENDOR_BROADCOM=y
CONFIG_B44=m
CONFIG_B44_PCI_AUTOSELECT=y
CONFIG_B44_PCICORE_AUTOSELECT=y
CONFIG_B44_PCI=y
# CONFIG_BCMGENET is not set
CONFIG_BNX2=m
CONFIG_CNIC=m
CONFIG_TIGON3=y
CONFIG_TIGON3_HWMON=y
# CONFIG_BNX2X is not set
# CONFIG_SYSTEMPORT is not set
# CONFIG_BNXT is not set
CONFIG_NET_VENDOR_BROCADE=y
CONFIG_BNA=m
CONFIG_NET_VENDOR_CADENCE=y
# CONFIG_MACB is not set
CONFIG_NET_VENDOR_CAVIUM=y
# CONFIG_THUNDER_NIC_PF is not set
# CONFIG_THUNDER_NIC_VF is not set
# CONFIG_THUNDER_NIC_BGX is not set
# CONFIG_THUNDER_NIC_RGX is not set
CONFIG_CAVIUM_PTP=y
# CONFIG_LIQUIDIO is not set
# CONFIG_LIQUIDIO_VF is not set
CONFIG_NET_VENDOR_CHELSIO=y
# CONFIG_CHELSIO_T1 is not set
CONFIG_CHELSIO_T3=m
CONFIG_CHELSIO_T4=m
# CONFIG_CHELSIO_T4_DCB is not set
CONFIG_CHELSIO_T4VF=m
CONFIG_CHELSIO_LIB=m
CONFIG_NET_VENDOR_CISCO=y
CONFIG_ENIC=m
CONFIG_NET_VENDOR_CORTINA=y
# CONFIG_CX_ECAT is not set
CONFIG_DNET=m
CONFIG_NET_VENDOR_DEC=y
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_DE2104X_DSL=0
CONFIG_TULIP=y
# CONFIG_TULIP_MWI is not set
CONFIG_TULIP_MMIO=y
# CONFIG_TULIP_NAPI is not set
CONFIG_DE4X5=m
CONFIG_WINBOND_840=m
CONFIG_DM9102=m
CONFIG_ULI526X=m
CONFIG_PCMCIA_XIRCOM=m
# CONFIG_NET_VENDOR_DLINK is not set
CONFIG_NET_VENDOR_EMULEX=y
CONFIG_BE2NET=m
CONFIG_BE2NET_HWMON=y
CONFIG_BE2NET_BE2=y
CONFIG_BE2NET_BE3=y
CONFIG_BE2NET_LANCER=y
CONFIG_BE2NET_SKYHAWK=y
CONFIG_NET_VENDOR_EZCHIP=y
# CONFIG_NET_VENDOR_HP is not set
CONFIG_NET_VENDOR_HUAWEI=y
# CONFIG_HINIC is not set
# CONFIG_NET_VENDOR_I825XX is not set
CONFIG_NET_VENDOR_INTEL=y
# CONFIG_E100 is not set
CONFIG_E1000=y
CONFIG_E1000E=y
CONFIG_E1000E_HWTS=y
CONFIG_IGB=y
CONFIG_IGB_HWMON=y
CONFIG_IGBVF=m
CONFIG_IXGB=y
CONFIG_IXGBE=y
CONFIG_IXGBE_HWMON=y
CONFIG_IXGBE_DCB=y
CONFIG_IXGBEVF=m
CONFIG_I40E=y
# CONFIG_I40E_DCB is not set
# CONFIG_I40EVF is not set
# CONFIG_ICE is not set
# CONFIG_FM10K is not set
# CONFIG_IGC is not set
CONFIG_JME=m
CONFIG_NET_VENDOR_MARVELL=y
CONFIG_MVMDIO=m
CONFIG_SKGE=y
CONFIG_SKGE_DEBUG=y
CONFIG_SKGE_GENESIS=y
CONFIG_SKY2=m
CONFIG_SKY2_DEBUG=y
CONFIG_NET_VENDOR_MELLANOX=y
CONFIG_MLX4_EN=m
CONFIG_MLX4_EN_DCB=y
CONFIG_MLX4_CORE=m
CONFIG_MLX4_DEBUG=y
CONFIG_MLX4_CORE_GEN2=y
# CONFIG_MLX5_CORE is not set
# CONFIG_MLXSW_CORE is not set
# CONFIG_MLXFW is not set
# CONFIG_NET_VENDOR_MICREL is not set
CONFIG_NET_VENDOR_MICROCHIP=y
# CONFIG_ENC28J60 is not set
# CONFIG_ENCX24J600 is not set
# CONFIG_LAN743X is not set
CONFIG_NET_VENDOR_MICROSEMI=y
CONFIG_NET_VENDOR_MYRI=y
CONFIG_MYRI10GE=m
# CONFIG_FEALNX is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
CONFIG_NET_VENDOR_NETERION=y
# CONFIG_S2IO is not set
# CONFIG_VXGE is not set
CONFIG_NET_VENDOR_NETRONOME=y
# CONFIG_NFP is not set
CONFIG_NET_VENDOR_NI=y
# CONFIG_NI_XGE_MANAGEMENT_ENET is not set
# CONFIG_NET_VENDOR_NVIDIA is not set
CONFIG_NET_VENDOR_OKI=y
CONFIG_ETHOC=m
CONFIG_NET_VENDOR_PACKET_ENGINES=y
# CONFIG_HAMACHI is not set
CONFIG_YELLOWFIN=m
CONFIG_NET_VENDOR_QLOGIC=y
CONFIG_QLA3XXX=m
CONFIG_QLCNIC=m
CONFIG_QLCNIC_SRIOV=y
CONFIG_QLCNIC_DCB=y
CONFIG_QLCNIC_HWMON=y
CONFIG_QLGE=m
CONFIG_NETXEN_NIC=m
# CONFIG_QED is not set
CONFIG_NET_VENDOR_QUALCOMM=y
# CONFIG_QCOM_EMAC is not set
# CONFIG_RMNET is not set
# CONFIG_NET_VENDOR_RDC is not set
CONFIG_NET_VENDOR_REALTEK=y
# CONFIG_ATP is not set
CONFIG_8139CP=y
CONFIG_8139TOO=y
CONFIG_8139TOO_PIO=y
# CONFIG_8139TOO_TUNE_TWISTER is not set
CONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not set
CONFIG_R8169=y
CONFIG_NET_VENDOR_RENESAS=y
CONFIG_NET_VENDOR_ROCKER=y
CONFIG_NET_VENDOR_SAMSUNG=y
# CONFIG_SXGBE_ETH is not set
# CONFIG_NET_VENDOR_SEEQ is not set
CONFIG_NET_VENDOR_SOLARFLARE=y
CONFIG_SFC=m
CONFIG_SFC_MTD=y
CONFIG_SFC_MCDI_MON=y
CONFIG_SFC_SRIOV=y
CONFIG_SFC_MCDI_LOGGING=y
# CONFIG_SFC_FALCON is not set
# CONFIG_NET_VENDOR_SILAN is not set
# CONFIG_NET_VENDOR_SIS is not set
CONFIG_NET_VENDOR_SMSC=y
CONFIG_EPIC100=m
# CONFIG_SMSC911X is not set
CONFIG_SMSC9420=m
CONFIG_NET_VENDOR_SOCIONEXT=y
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_SUN is not set
CONFIG_NET_VENDOR_SYNOPSYS=y
# CONFIG_DWC_XLGMAC is not set
# CONFIG_NET_VENDOR_TEHUTI is not set
# CONFIG_NET_VENDOR_TI is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
CONFIG_NET_VENDOR_XILINX=y
# CONFIG_XILINX_LL_TEMAC is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_NET_SB1000 is not set
CONFIG_MDIO_DEVICE=y
CONFIG_MDIO_BUS=y
# CONFIG_MDIO_BCM_UNIMAC is not set
CONFIG_MDIO_BITBANG=m
# CONFIG_MDIO_GPIO is not set
# CONFIG_MDIO_MSCC_MIIM is not set
# CONFIG_MDIO_THUNDER is not set
CONFIG_PHYLIB=y
CONFIG_SWPHY=y
# CONFIG_LED_TRIGGER_PHY is not set

#
# MII PHY device drivers
#
CONFIG_AMD_PHY=m
# CONFIG_AQUANTIA_PHY is not set
# CONFIG_ASIX_PHY is not set
CONFIG_AT803X_PHY=m
# CONFIG_BCM7XXX_PHY is not set
CONFIG_BCM87XX_PHY=m
CONFIG_BCM_NET_PHYLIB=m
CONFIG_BROADCOM_PHY=m
CONFIG_CICADA_PHY=m
# CONFIG_CORTINA_PHY is not set
CONFIG_DAVICOM_PHY=m
# CONFIG_DP83822_PHY is not set
# CONFIG_DP83TC811_PHY is not set
# CONFIG_DP83848_PHY is not set
# CONFIG_DP83867_PHY is not set
CONFIG_FIXED_PHY=y
CONFIG_ICPLUS_PHY=m
# CONFIG_INTEL_XWAY_PHY is not set
CONFIG_LSI_ET1011C_PHY=m
CONFIG_LXT_PHY=m
CONFIG_MARVELL_PHY=m
# CONFIG_MARVELL_10G_PHY is not set
CONFIG_MICREL_PHY=m
# CONFIG_MICROCHIP_PHY is not set
# CONFIG_MICROCHIP_T1_PHY is not set
# CONFIG_MICROSEMI_PHY is not set
CONFIG_NATIONAL_PHY=m
CONFIG_QSEMI_PHY=m
CONFIG_REALTEK_PHY=y
# CONFIG_RENESAS_PHY is not set
# CONFIG_ROCKCHIP_PHY is not set
CONFIG_SMSC_PHY=m
CONFIG_STE10XP=m
# CONFIG_TERANETICS_PHY is not set
CONFIG_VITESSE_PHY=m
# CONFIG_XILINX_GMII2RGMII is not set
# CONFIG_MICREL_KS8995MA is not set
# CONFIG_PLIP is not set
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_FILTER=y
CONFIG_PPP_MPPE=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPPOATM=m
CONFIG_PPPOE=m
CONFIG_PPTP=m
CONFIG_PPPOL2TP=m
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_SLIP=m
CONFIG_SLHC=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLIP_SMART=y
# CONFIG_SLIP_MODE_SLIP6 is not set
CONFIG_USB_NET_DRIVERS=y
CONFIG_USB_CATC=y
CONFIG_USB_KAWETH=y
CONFIG_USB_PEGASUS=y
CONFIG_USB_RTL8150=y
CONFIG_USB_RTL8152=m
# CONFIG_USB_LAN78XX is not set
CONFIG_USB_USBNET=y
CONFIG_USB_NET_AX8817X=y
CONFIG_USB_NET_AX88179_178A=m
CONFIG_USB_NET_CDCETHER=y
CONFIG_USB_NET_CDC_EEM=y
CONFIG_USB_NET_CDC_NCM=m
# CONFIG_USB_NET_HUAWEI_CDC_NCM is not set
CONFIG_USB_NET_CDC_MBIM=m
CONFIG_USB_NET_DM9601=y
# CONFIG_USB_NET_SR9700 is not set
# CONFIG_USB_NET_SR9800 is not set
CONFIG_USB_NET_SMSC75XX=y
CONFIG_USB_NET_SMSC95XX=y
CONFIG_USB_NET_GL620A=y
CONFIG_USB_NET_NET1080=y
CONFIG_USB_NET_PLUSB=y
CONFIG_USB_NET_MCS7830=y
CONFIG_USB_NET_RNDIS_HOST=y
CONFIG_USB_NET_CDC_SUBSET_ENABLE=y
CONFIG_USB_NET_CDC_SUBSET=y
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
CONFIG_USB_KC2190=y
CONFIG_USB_NET_ZAURUS=y
CONFIG_USB_NET_CX82310_ETH=m
CONFIG_USB_NET_KALMIA=m
CONFIG_USB_NET_QMI_WWAN=m
CONFIG_USB_HSO=m
CONFIG_USB_NET_INT51X1=y
CONFIG_USB_IPHETH=y
CONFIG_USB_SIERRA_NET=y
CONFIG_USB_VL600=m
# CONFIG_USB_NET_CH9200 is not set
# CONFIG_USB_NET_AQC111 is not set
CONFIG_WLAN=y
# CONFIG_WIRELESS_WDS is not set
CONFIG_WLAN_VENDOR_ADMTEK=y
# CONFIG_ADM8211 is not set
CONFIG_WLAN_VENDOR_ATH=y
# CONFIG_ATH_DEBUG is not set
# CONFIG_ATH5K is not set
# CONFIG_ATH5K_PCI is not set
# CONFIG_ATH9K is not set
# CONFIG_ATH9K_HTC is not set
# CONFIG_CARL9170 is not set
# CONFIG_ATH6KL is not set
# CONFIG_AR5523 is not set
# CONFIG_WIL6210 is not set
# CONFIG_ATH10K is not set
# CONFIG_WCN36XX is not set
CONFIG_WLAN_VENDOR_ATMEL=y
# CONFIG_ATMEL is not set
# CONFIG_AT76C50X_USB is not set
CONFIG_WLAN_VENDOR_BROADCOM=y
# CONFIG_B43 is not set
# CONFIG_B43LEGACY is not set
# CONFIG_BRCMSMAC is not set
# CONFIG_BRCMFMAC is not set
CONFIG_WLAN_VENDOR_CISCO=y
# CONFIG_AIRO is not set
CONFIG_WLAN_VENDOR_INTEL=y
# CONFIG_IPW2100 is not set
# CONFIG_IPW2200 is not set
# CONFIG_IWL4965 is not set
# CONFIG_IWL3945 is not set
# CONFIG_IWLWIFI is not set
CONFIG_WLAN_VENDOR_INTERSIL=y
# CONFIG_HOSTAP is not set
# CONFIG_HERMES is not set
# CONFIG_P54_COMMON is not set
# CONFIG_PRISM54 is not set
CONFIG_WLAN_VENDOR_MARVELL=y
# CONFIG_LIBERTAS is not set
# CONFIG_LIBERTAS_THINFIRM is not set
# CONFIG_MWIFIEX is not set
# CONFIG_MWL8K is not set
CONFIG_WLAN_VENDOR_MEDIATEK=y
# CONFIG_MT7601U is not set
# CONFIG_MT76x0U is not set
# CONFIG_MT76x0E is not set
# CONFIG_MT76x2E is not set
# CONFIG_MT76x2U is not set
# CONFIG_MT7603E is not set
# CONFIG_MT7615E is not set
CONFIG_WLAN_VENDOR_RALINK=y
# CONFIG_RT2X00 is not set
CONFIG_WLAN_VENDOR_REALTEK=y
# CONFIG_RTL8180 is not set
# CONFIG_RTL8187 is not set
CONFIG_RTL_CARDS=m
# CONFIG_RTL8192CE is not set
# CONFIG_RTL8192SE is not set
# CONFIG_RTL8192DE is not set
# CONFIG_RTL8723AE is not set
# CONFIG_RTL8723BE is not set
# CONFIG_RTL8188EE is not set
# CONFIG_RTL8192EE is not set
# CONFIG_RTL8821AE is not set
# CONFIG_RTL8192CU is not set
# CONFIG_RTL8XXXU is not set
# CONFIG_RTW88 is not set
CONFIG_WLAN_VENDOR_RSI=y
# CONFIG_RSI_91X is not set
CONFIG_WLAN_VENDOR_ST=y
# CONFIG_CW1200 is not set
CONFIG_WLAN_VENDOR_TI=y
# CONFIG_WL1251 is not set
# CONFIG_WL12XX is not set
# CONFIG_WL18XX is not set
# CONFIG_WLCORE is not set
CONFIG_WLAN_VENDOR_ZYDAS=y
# CONFIG_USB_ZD1201 is not set
# CONFIG_ZD1211RW is not set
CONFIG_WLAN_VENDOR_QUANTENNA=y
# CONFIG_QTNFMAC_PCIE is not set
CONFIG_MAC80211_HWSIM=m
# CONFIG_USB_NET_RNDIS_WLAN is not set
# CONFIG_VIRT_WIFI is not set

#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#
CONFIG_WAN=y
# CONFIG_LANMEDIA is not set
CONFIG_HDLC=m
CONFIG_HDLC_RAW=m
# CONFIG_HDLC_RAW_ETH is not set
CONFIG_HDLC_CISCO=m
CONFIG_HDLC_FR=m
CONFIG_HDLC_PPP=m

#
# X.25/LAPB support is disabled
#
# CONFIG_PCI200SYN is not set
# CONFIG_WANXL is not set
# CONFIG_PC300TOO is not set
# CONFIG_FARSYNC is not set
# CONFIG_DSCC4 is not set
CONFIG_DLCI=m
CONFIG_DLCI_MAX=8
# CONFIG_SBNI is not set
CONFIG_IEEE802154_DRIVERS=m
CONFIG_IEEE802154_FAKELB=m
# CONFIG_IEEE802154_AT86RF230 is not set
# CONFIG_IEEE802154_MRF24J40 is not set
# CONFIG_IEEE802154_CC2520 is not set
# CONFIG_IEEE802154_ATUSB is not set
# CONFIG_IEEE802154_ADF7242 is not set
# CONFIG_IEEE802154_CA8210 is not set
# CONFIG_IEEE802154_MCR20A is not set
# CONFIG_IEEE802154_HWSIM is not set
CONFIG_XEN_NETDEV_FRONTEND=m
# CONFIG_XEN_NETDEV_BACKEND is not set
CONFIG_VMXNET3=m
# CONFIG_FUJITSU_ES is not set
CONFIG_HYPERV_NET=m
CONFIG_NETDEVSIM=m
CONFIG_NET_FAILOVER=y
CONFIG_ISDN=y
CONFIG_ISDN_I4L=m
CONFIG_ISDN_PPP=y
CONFIG_ISDN_PPP_VJ=y
CONFIG_ISDN_MPP=y
CONFIG_IPPP_FILTER=y
# CONFIG_ISDN_PPP_BSDCOMP is not set
CONFIG_ISDN_AUDIO=y
CONFIG_ISDN_TTY_FAX=y

#
# ISDN feature submodules
#
CONFIG_ISDN_DIVERSION=m
# end of ISDN feature submodules

#
# ISDN4Linux hardware drivers
#

#
# Passive cards
#
# CONFIG_ISDN_DRV_HISAX is not set
# end of Passive cards

CONFIG_ISDN_CAPI=m
# CONFIG_CAPI_TRACE is not set
CONFIG_ISDN_CAPI_CAPI20=m
CONFIG_ISDN_CAPI_MIDDLEWARE=y
CONFIG_ISDN_CAPI_CAPIDRV=m
# CONFIG_ISDN_CAPI_CAPIDRV_VERBOSE is not set

#
# CAPI hardware drivers
#
CONFIG_CAPI_AVM=y
CONFIG_ISDN_DRV_AVMB1_B1PCI=m
CONFIG_ISDN_DRV_AVMB1_B1PCIV4=y
CONFIG_ISDN_DRV_AVMB1_T1PCI=m
CONFIG_ISDN_DRV_AVMB1_C4=m
CONFIG_ISDN_DRV_GIGASET=m
CONFIG_GIGASET_CAPI=y
CONFIG_GIGASET_BASE=m
CONFIG_GIGASET_M105=m
CONFIG_GIGASET_M101=m
# CONFIG_GIGASET_DEBUG is not set
CONFIG_HYSDN=m
CONFIG_HYSDN_CAPI=y
CONFIG_MISDN=m
CONFIG_MISDN_DSP=m
CONFIG_MISDN_L1OIP=m

#
# mISDN hardware drivers
#
CONFIG_MISDN_HFCPCI=m
CONFIG_MISDN_HFCMULTI=m
CONFIG_MISDN_HFCUSB=m
CONFIG_MISDN_AVMFRITZ=m
CONFIG_MISDN_SPEEDFAX=m
CONFIG_MISDN_INFINEON=m
CONFIG_MISDN_W6692=m
CONFIG_MISDN_NETJET=m
CONFIG_MISDN_IPAC=m
CONFIG_MISDN_ISAR=m
CONFIG_ISDN_HDLC=m
CONFIG_NVM=y
# CONFIG_NVM_PBLK is not set

#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_LEDS=y
CONFIG_INPUT_FF_MEMLESS=y
CONFIG_INPUT_POLLDEV=m
CONFIG_INPUT_SPARSEKMAP=m
# CONFIG_INPUT_MATRIXKMAP is not set

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
# CONFIG_KEYBOARD_ADP5588 is not set
# CONFIG_KEYBOARD_ADP5589 is not set
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_QT1050 is not set
# CONFIG_KEYBOARD_QT1070 is not set
# CONFIG_KEYBOARD_QT2160 is not set
# CONFIG_KEYBOARD_DLINK_DIR685 is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_GPIO is not set
# CONFIG_KEYBOARD_GPIO_POLLED is not set
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_TCA8418 is not set
# CONFIG_KEYBOARD_MATRIX is not set
# CONFIG_KEYBOARD_LM8323 is not set
# CONFIG_KEYBOARD_LM8333 is not set
# CONFIG_KEYBOARD_MAX7359 is not set
# CONFIG_KEYBOARD_MCS is not set
# CONFIG_KEYBOARD_MPR121 is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
# CONFIG_KEYBOARD_SAMSUNG is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_TM2_TOUCHKEY is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_BYD=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_SYNAPTICS_SMBUS=y
CONFIG_MOUSE_PS2_CYPRESS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
CONFIG_MOUSE_PS2_ELANTECH=y
CONFIG_MOUSE_PS2_ELANTECH_SMBUS=y
CONFIG_MOUSE_PS2_SENTELIC=y
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
CONFIG_MOUSE_PS2_FOCALTECH=y
# CONFIG_MOUSE_PS2_VMMOUSE is not set
CONFIG_MOUSE_PS2_SMBUS=y
CONFIG_MOUSE_SERIAL=m
CONFIG_MOUSE_APPLETOUCH=m
CONFIG_MOUSE_BCM5974=m
CONFIG_MOUSE_CYAPA=m
# CONFIG_MOUSE_ELAN_I2C is not set
CONFIG_MOUSE_VSXXXAA=m
# CONFIG_MOUSE_GPIO is not set
CONFIG_MOUSE_SYNAPTICS_I2C=m
CONFIG_MOUSE_SYNAPTICS_USB=m
# CONFIG_INPUT_JOYSTICK is not set
CONFIG_INPUT_TABLET=y
CONFIG_TABLET_USB_ACECAD=m
CONFIG_TABLET_USB_AIPTEK=m
CONFIG_TABLET_USB_GTCO=m
# CONFIG_TABLET_USB_HANWANG is not set
CONFIG_TABLET_USB_KBTAB=m
# CONFIG_TABLET_USB_PEGASUS is not set
# CONFIG_TABLET_SERIAL_WACOM4 is not set
CONFIG_INPUT_TOUCHSCREEN=y
CONFIG_TOUCHSCREEN_PROPERTIES=y
# CONFIG_TOUCHSCREEN_ADS7846 is not set
# CONFIG_TOUCHSCREEN_AD7877 is not set
# CONFIG_TOUCHSCREEN_AD7879 is not set
# CONFIG_TOUCHSCREEN_ATMEL_MXT is not set
# CONFIG_TOUCHSCREEN_AUO_PIXCIR is not set
# CONFIG_TOUCHSCREEN_BU21013 is not set
# CONFIG_TOUCHSCREEN_BU21029 is not set
# CONFIG_TOUCHSCREEN_CHIPONE_ICN8505 is not set
# CONFIG_TOUCHSCREEN_CY8CTMG110 is not set
# CONFIG_TOUCHSCREEN_CYTTSP_CORE is not set
# CONFIG_TOUCHSCREEN_CYTTSP4_CORE is not set
# CONFIG_TOUCHSCREEN_DYNAPRO is not set
# CONFIG_TOUCHSCREEN_HAMPSHIRE is not set
# CONFIG_TOUCHSCREEN_EETI is not set
# CONFIG_TOUCHSCREEN_EGALAX_SERIAL is not set
# CONFIG_TOUCHSCREEN_EXC3000 is not set
# CONFIG_TOUCHSCREEN_FUJITSU is not set
# CONFIG_TOUCHSCREEN_GOODIX is not set
# CONFIG_TOUCHSCREEN_HIDEEP is not set
# CONFIG_TOUCHSCREEN_ILI210X is not set
# CONFIG_TOUCHSCREEN_S6SY761 is not set
# CONFIG_TOUCHSCREEN_GUNZE is not set
# CONFIG_TOUCHSCREEN_EKTF2127 is not set
# CONFIG_TOUCHSCREEN_ELAN is not set
# CONFIG_TOUCHSCREEN_ELO is not set
CONFIG_TOUCHSCREEN_WACOM_W8001=m
CONFIG_TOUCHSCREEN_WACOM_I2C=m
# CONFIG_TOUCHSCREEN_MAX11801 is not set
# CONFIG_TOUCHSCREEN_MCS5000 is not set
# CONFIG_TOUCHSCREEN_MMS114 is not set
# CONFIG_TOUCHSCREEN_MELFAS_MIP4 is not set
# CONFIG_TOUCHSCREEN_MTOUCH is not set
# CONFIG_TOUCHSCREEN_INEXIO is not set
# CONFIG_TOUCHSCREEN_MK712 is not set
# CONFIG_TOUCHSCREEN_PENMOUNT is not set
# CONFIG_TOUCHSCREEN_EDT_FT5X06 is not set
# CONFIG_TOUCHSCREEN_TOUCHRIGHT is not set
# CONFIG_TOUCHSCREEN_TOUCHWIN is not set
# CONFIG_TOUCHSCREEN_PIXCIR is not set
# CONFIG_TOUCHSCREEN_WDT87XX_I2C is not set
# CONFIG_TOUCHSCREEN_WM97XX is not set
# CONFIG_TOUCHSCREEN_USB_COMPOSITE is not set
# CONFIG_TOUCHSCREEN_TOUCHIT213 is not set
# CONFIG_TOUCHSCREEN_TSC_SERIO is not set
# CONFIG_TOUCHSCREEN_TSC2004 is not set
# CONFIG_TOUCHSCREEN_TSC2005 is not set
# CONFIG_TOUCHSCREEN_TSC2007 is not set
# CONFIG_TOUCHSCREEN_RM_TS is not set
# CONFIG_TOUCHSCREEN_SILEAD is not set
# CONFIG_TOUCHSCREEN_SIS_I2C is not set
# CONFIG_TOUCHSCREEN_ST1232 is not set
# CONFIG_TOUCHSCREEN_STMFTS is not set
# CONFIG_TOUCHSCREEN_SUR40 is not set
# CONFIG_TOUCHSCREEN_SURFACE3_SPI is not set
# CONFIG_TOUCHSCREEN_SX8654 is not set
# CONFIG_TOUCHSCREEN_TPS6507X is not set
# CONFIG_TOUCHSCREEN_ZET6223 is not set
# CONFIG_TOUCHSCREEN_ZFORCE is not set
# CONFIG_TOUCHSCREEN_ROHM_BU21023 is not set
# CONFIG_TOUCHSCREEN_IQS5XX is not set
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_AD714X is not set
# CONFIG_INPUT_BMA150 is not set
# CONFIG_INPUT_E3X0_BUTTON is not set
# CONFIG_INPUT_MSM_VIBRATOR is not set
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_MMA8450 is not set
CONFIG_INPUT_APANEL=m
# CONFIG_INPUT_GP2A is not set
# CONFIG_INPUT_GPIO_BEEPER is not set
# CONFIG_INPUT_GPIO_DECODER is not set
# CONFIG_INPUT_GPIO_VIBRA is not set
CONFIG_INPUT_ATLAS_BTNS=m
CONFIG_INPUT_ATI_REMOTE2=m
CONFIG_INPUT_KEYSPAN_REMOTE=m
# CONFIG_INPUT_KXTJ9 is not set
CONFIG_INPUT_POWERMATE=m
CONFIG_INPUT_YEALINK=m
CONFIG_INPUT_CM109=m
CONFIG_INPUT_UINPUT=m
# CONFIG_INPUT_PCF8574 is not set
# CONFIG_INPUT_PWM_BEEPER is not set
# CONFIG_INPUT_PWM_VIBRA is not set
# CONFIG_INPUT_GPIO_ROTARY_ENCODER is not set
# CONFIG_INPUT_ADXL34X is not set
# CONFIG_INPUT_IMS_PCU is not set
# CONFIG_INPUT_CMA3000 is not set
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=m
# CONFIG_INPUT_IDEAPAD_SLIDEBAR is not set
# CONFIG_INPUT_DRV260X_HAPTICS is not set
# CONFIG_INPUT_DRV2665_HAPTICS is not set
# CONFIG_INPUT_DRV2667_HAPTICS is not set
# CONFIG_RMI4_CORE is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_ARCH_MIGHT_HAVE_PC_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m
CONFIG_SERIO_ALTERA_PS2=m
# CONFIG_SERIO_PS2MULT is not set
CONFIG_SERIO_ARC_PS2=m
CONFIG_HYPERV_KEYBOARD=m
# CONFIG_SERIO_GPIO_PS2 is not set
# CONFIG_USERIO is not set
# CONFIG_GAMEPORT is not set
# end of Hardware I/O ports
# end of Input device support

#
# Character devices
#
CONFIG_TTY=y
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_SERIAL_NONSTANDARD=y
# CONFIG_ROCKETPORT is not set
CONFIG_CYCLADES=m
# CONFIG_CYZ_INTR is not set
CONFIG_MOXA_INTELLIO=m
CONFIG_MOXA_SMARTIO=m
CONFIG_SYNCLINK=m
CONFIG_SYNCLINKMP=m
CONFIG_SYNCLINK_GT=m
CONFIG_NOZOMI=m
# CONFIG_ISI is not set
CONFIG_N_HDLC=m
CONFIG_N_GSM=m
# CONFIG_TRACE_SINK is not set
# CONFIG_NULL_TTY is not set
CONFIG_LDISC_AUTOLOAD=y
CONFIG_DEVMEM=y
# CONFIG_DEVKMEM is not set

#
# Serial drivers
#
CONFIG_SERIAL_EARLYCON=y
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_DEPRECATED_OPTIONS is not set
CONFIG_SERIAL_8250_PNP=y
# CONFIG_SERIAL_8250_FINTEK is not set
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_DMA=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_EXAR=y
CONFIG_SERIAL_8250_NR_UARTS=32
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
# CONFIG_SERIAL_8250_DETECT_IRQ is not set
CONFIG_SERIAL_8250_RSA=y
CONFIG_SERIAL_8250_DW=y
# CONFIG_SERIAL_8250_RT288X is not set
CONFIG_SERIAL_8250_LPSS=y
CONFIG_SERIAL_8250_MID=y
# CONFIG_SERIAL_8250_MOXA is not set

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_MAX3100 is not set
# CONFIG_SERIAL_MAX310X is not set
# CONFIG_SERIAL_UARTLITE is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SERIAL_JSM=m
# CONFIG_SERIAL_SCCNXP is not set
# CONFIG_SERIAL_SC16IS7XX is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_IFX6X60 is not set
CONFIG_SERIAL_ARC=m
CONFIG_SERIAL_ARC_NR_PORTS=1
# CONFIG_SERIAL_RP2 is not set
# CONFIG_SERIAL_FSL_LPUART is not set
# end of Serial drivers

# CONFIG_SERIAL_DEV_BUS is not set
# CONFIG_TTY_PRINTK is not set
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
CONFIG_PPDEV=m
CONFIG_HVC_DRIVER=y
CONFIG_HVC_IRQ=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_IPMI_HANDLER=m
CONFIG_IPMI_DMI_DECODE=y
CONFIG_IPMI_PLAT_DATA=y
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
# CONFIG_IPMI_SSIF is not set
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_TIMERIOMEM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_HW_RANDOM_VIA=m
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_NVRAM=y
# CONFIG_APPLICOM is not set
# CONFIG_MWAVE is not set
CONFIG_RAW_DRIVER=y
CONFIG_MAX_RAW_DEVS=8192
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
# CONFIG_HPET_MMAP_DEFAULT is not set
CONFIG_HANGCHECK_TIMER=m
CONFIG_UV_MMTIMER=m
CONFIG_TCG_TPM=y
CONFIG_HW_RANDOM_TPM=y
CONFIG_TCG_TIS_CORE=y
CONFIG_TCG_TIS=y
# CONFIG_TCG_TIS_SPI is not set
# CONFIG_TCG_TIS_I2C_ATMEL is not set
# CONFIG_TCG_TIS_I2C_INFINEON is not set
# CONFIG_TCG_TIS_I2C_NUVOTON is not set
CONFIG_TCG_NSC=m
CONFIG_TCG_ATMEL=m
CONFIG_TCG_INFINEON=m
# CONFIG_TCG_XEN is not set
CONFIG_TCG_CRB=y
# CONFIG_TCG_VTPM_PROXY is not set
# CONFIG_TCG_TIS_ST33ZP24_I2C is not set
# CONFIG_TCG_TIS_ST33ZP24_SPI is not set
CONFIG_TELCLOCK=m
CONFIG_DEVPORT=y
# CONFIG_XILLYBUS is not set
# end of Character devices

# CONFIG_RANDOM_TRUST_CPU is not set

#
# I2C support
#
CONFIG_I2C=y
CONFIG_ACPI_I2C_OPREGION=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_MUX=m

#
# Multiplexer I2C Chip support
#
# CONFIG_I2C_MUX_GPIO is not set
# CONFIG_I2C_MUX_LTC4306 is not set
# CONFIG_I2C_MUX_PCA9541 is not set
# CONFIG_I2C_MUX_PCA954x is not set
# CONFIG_I2C_MUX_REG is not set
# CONFIG_I2C_MUX_MLXCPLD is not set
# end of Multiplexer I2C Chip support

CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_SMBUS=y
CONFIG_I2C_ALGOBIT=y
CONFIG_I2C_ALGOPCA=m

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD756_S4882=m
CONFIG_I2C_AMD8111=m
# CONFIG_I2C_AMD_MP2 is not set
CONFIG_I2C_I801=y
CONFIG_I2C_ISCH=m
CONFIG_I2C_ISMT=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_NFORCE2_S4985=m
# CONFIG_I2C_NVIDIA_GPU is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
CONFIG_I2C_SIS96X=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m

#
# ACPI drivers
#
CONFIG_I2C_SCMI=m

#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_CBUS_GPIO is not set
CONFIG_I2C_DESIGNWARE_CORE=m
CONFIG_I2C_DESIGNWARE_PLATFORM=m
# CONFIG_I2C_DESIGNWARE_SLAVE is not set
CONFIG_I2C_DESIGNWARE_PCI=m
# CONFIG_I2C_DESIGNWARE_BAYTRAIL is not set
# CONFIG_I2C_EMEV2 is not set
# CONFIG_I2C_GPIO is not set
# CONFIG_I2C_OCORES is not set
CONFIG_I2C_PCA_PLATFORM=m
CONFIG_I2C_SIMTEC=m
# CONFIG_I2C_XILINX is not set

#
# External I2C/SMBus adapter drivers
#
CONFIG_I2C_DIOLAN_U2C=m
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
# CONFIG_I2C_ROBOTFUZZ_OSIF is not set
# CONFIG_I2C_TAOS_EVM is not set
CONFIG_I2C_TINY_USB=m
CONFIG_I2C_VIPERBOARD=m

#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_MLXCPLD is not set
# end of I2C Hardware Bus support

CONFIG_I2C_STUB=m
# CONFIG_I2C_SLAVE is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# end of I2C support

# CONFIG_I3C is not set
CONFIG_SPI=y
# CONFIG_SPI_DEBUG is not set
CONFIG_SPI_MASTER=y
# CONFIG_SPI_MEM is not set

#
# SPI Master Controller Drivers
#
# CONFIG_SPI_ALTERA is not set
# CONFIG_SPI_AXI_SPI_ENGINE is not set
# CONFIG_SPI_BITBANG is not set
# CONFIG_SPI_BUTTERFLY is not set
# CONFIG_SPI_CADENCE is not set
CONFIG_SPI_DESIGNWARE=m
# CONFIG_SPI_DW_PCI is not set
# CONFIG_SPI_DW_MMIO is not set
# CONFIG_SPI_NXP_FLEXSPI is not set
# CONFIG_SPI_GPIO is not set
# CONFIG_SPI_LM70_LLP is not set
# CONFIG_SPI_OC_TINY is not set
CONFIG_SPI_PXA2XX=m
CONFIG_SPI_PXA2XX_PCI=m
# CONFIG_SPI_ROCKCHIP is not set
# CONFIG_SPI_SC18IS602 is not set
# CONFIG_SPI_SIFIVE is not set
# CONFIG_SPI_MXIC is not set
# CONFIG_SPI_XCOMM is not set
# CONFIG_SPI_XILINX is not set
# CONFIG_SPI_ZYNQMP_GQSPI is not set

#
# SPI Protocol Masters
#
# CONFIG_SPI_SPIDEV is not set
# CONFIG_SPI_LOOPBACK_TEST is not set
# CONFIG_SPI_TLE62X0 is not set
# CONFIG_SPI_SLAVE is not set
# CONFIG_SPMI is not set
# CONFIG_HSI is not set
CONFIG_PPS=y
# CONFIG_PPS_DEBUG is not set

#
# PPS clients support
#
# CONFIG_PPS_CLIENT_KTIMER is not set
CONFIG_PPS_CLIENT_LDISC=m
CONFIG_PPS_CLIENT_PARPORT=m
CONFIG_PPS_CLIENT_GPIO=m

#
# PPS generators support
#

#
# PTP clock support
#
CONFIG_PTP_1588_CLOCK=y
CONFIG_DP83640_PHY=m
CONFIG_PTP_1588_CLOCK_KVM=y
# end of PTP clock support

CONFIG_PINCTRL=y
CONFIG_PINMUX=y
CONFIG_PINCONF=y
CONFIG_GENERIC_PINCONF=y
# CONFIG_DEBUG_PINCTRL is not set
# CONFIG_PINCTRL_AMD is not set
# CONFIG_PINCTRL_MCP23S08 is not set
# CONFIG_PINCTRL_SX150X is not set
CONFIG_PINCTRL_BAYTRAIL=y
# CONFIG_PINCTRL_CHERRYVIEW is not set
# CONFIG_PINCTRL_BROXTON is not set
# CONFIG_PINCTRL_CANNONLAKE is not set
# CONFIG_PINCTRL_CEDARFORK is not set
# CONFIG_PINCTRL_DENVERTON is not set
# CONFIG_PINCTRL_GEMINILAKE is not set
# CONFIG_PINCTRL_ICELAKE is not set
# CONFIG_PINCTRL_LEWISBURG is not set
# CONFIG_PINCTRL_SUNRISEPOINT is not set
CONFIG_GPIOLIB=y
CONFIG_GPIOLIB_FASTPATH_LIMIT=512
CONFIG_GPIO_ACPI=y
CONFIG_GPIOLIB_IRQCHIP=y
# CONFIG_DEBUG_GPIO is not set
CONFIG_GPIO_SYSFS=y

#
# Memory mapped GPIO drivers
#
# CONFIG_GPIO_AMDPT is not set
# CONFIG_GPIO_DWAPB is not set
# CONFIG_GPIO_EXAR is not set
# CONFIG_GPIO_GENERIC_PLATFORM is not set
# CONFIG_GPIO_ICH is not set
CONFIG_GPIO_LYNXPOINT=m
# CONFIG_GPIO_MB86S7X is not set
# CONFIG_GPIO_VX855 is not set
# CONFIG_GPIO_AMD_FCH is not set
# end of Memory mapped GPIO drivers

#
# Port-mapped I/O GPIO drivers
#
# CONFIG_GPIO_F7188X is not set
# CONFIG_GPIO_IT87 is not set
# CONFIG_GPIO_SCH is not set
# CONFIG_GPIO_SCH311X is not set
# CONFIG_GPIO_WINBOND is not set
# CONFIG_GPIO_WS16C48 is not set
# end of Port-mapped I/O GPIO drivers

#
# I2C GPIO expanders
#
# CONFIG_GPIO_ADP5588 is not set
# CONFIG_GPIO_MAX7300 is not set
# CONFIG_GPIO_MAX732X is not set
# CONFIG_GPIO_PCA953X is not set
# CONFIG_GPIO_PCF857X is not set
# CONFIG_GPIO_TPIC2810 is not set
# end of I2C GPIO expanders

#
# MFD GPIO expanders
#
# end of MFD GPIO expanders

#
# PCI GPIO expanders
#
# CONFIG_GPIO_AMD8111 is not set
# CONFIG_GPIO_ML_IOH is not set
# CONFIG_GPIO_PCI_IDIO_16 is not set
# CONFIG_GPIO_PCIE_IDIO_24 is not set
# CONFIG_GPIO_RDC321X is not set
# end of PCI GPIO expanders

#
# SPI GPIO expanders
#
# CONFIG_GPIO_MAX3191X is not set
# CONFIG_GPIO_MAX7301 is not set
# CONFIG_GPIO_MC33880 is not set
# CONFIG_GPIO_PISOSR is not set
# CONFIG_GPIO_XRA1403 is not set
# end of SPI GPIO expanders

#
# USB GPIO expanders
#
# CONFIG_GPIO_VIPERBOARD is not set
# end of USB GPIO expanders

CONFIG_GPIO_MOCKUP=y
# CONFIG_W1 is not set
# CONFIG_POWER_AVS is not set
CONFIG_POWER_RESET=y
# CONFIG_POWER_RESET_RESTART is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_TEST_POWER is not set
# CONFIG_CHARGER_ADP5061 is not set
# CONFIG_BATTERY_DS2780 is not set
# CONFIG_BATTERY_DS2781 is not set
# CONFIG_BATTERY_DS2782 is not set
# CONFIG_BATTERY_SBS is not set
# CONFIG_CHARGER_SBS is not set
# CONFIG_MANAGER_SBS is not set
# CONFIG_BATTERY_BQ27XXX is not set
# CONFIG_BATTERY_MAX17040 is not set
# CONFIG_BATTERY_MAX17042 is not set
# CONFIG_CHARGER_ISP1704 is not set
# CONFIG_CHARGER_MAX8903 is not set
# CONFIG_CHARGER_LP8727 is not set
# CONFIG_CHARGER_GPIO is not set
# CONFIG_CHARGER_LT3651 is not set
# CONFIG_CHARGER_BQ2415X is not set
# CONFIG_CHARGER_BQ24190 is not set
# CONFIG_CHARGER_BQ24257 is not set
# CONFIG_CHARGER_BQ24735 is not set
# CONFIG_CHARGER_BQ25890 is not set
CONFIG_CHARGER_SMB347=m
# CONFIG_BATTERY_GAUGE_LTC2941 is not set
# CONFIG_CHARGER_RT9455 is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=m
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Native drivers
#
CONFIG_SENSORS_ABITUGURU=m
CONFIG_SENSORS_ABITUGURU3=m
# CONFIG_SENSORS_AD7314 is not set
CONFIG_SENSORS_AD7414=m
CONFIG_SENSORS_AD7418=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
CONFIG_SENSORS_ADM1029=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ADM9240=m
CONFIG_SENSORS_ADT7X10=m
# CONFIG_SENSORS_ADT7310 is not set
CONFIG_SENSORS_ADT7410=m
CONFIG_SENSORS_ADT7411=m
CONFIG_SENSORS_ADT7462=m
CONFIG_SENSORS_ADT7470=m
CONFIG_SENSORS_ADT7475=m
CONFIG_SENSORS_ASC7621=m
CONFIG_SENSORS_K8TEMP=m
CONFIG_SENSORS_K10TEMP=m
CONFIG_SENSORS_FAM15H_POWER=m
CONFIG_SENSORS_APPLESMC=m
CONFIG_SENSORS_ASB100=m
# CONFIG_SENSORS_ASPEED is not set
CONFIG_SENSORS_ATXP1=m
CONFIG_SENSORS_DS620=m
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_DELL_SMM=m
CONFIG_SENSORS_I5K_AMB=m
CONFIG_SENSORS_F71805F=m
CONFIG_SENSORS_F71882FG=m
CONFIG_SENSORS_F75375S=m
CONFIG_SENSORS_FSCHMD=m
# CONFIG_SENSORS_FTSTEUTATES is not set
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
CONFIG_SENSORS_G760A=m
# CONFIG_SENSORS_G762 is not set
# CONFIG_SENSORS_HIH6130 is not set
CONFIG_SENSORS_IBMAEM=m
CONFIG_SENSORS_IBMPEX=m
# CONFIG_SENSORS_I5500 is not set
CONFIG_SENSORS_CORETEMP=m
CONFIG_SENSORS_IT87=m
# CONFIG_SENSORS_JC42 is not set
# CONFIG_SENSORS_POWR1220 is not set
CONFIG_SENSORS_LINEAGE=m
# CONFIG_SENSORS_LTC2945 is not set
# CONFIG_SENSORS_LTC2990 is not set
CONFIG_SENSORS_LTC4151=m
CONFIG_SENSORS_LTC4215=m
# CONFIG_SENSORS_LTC4222 is not set
CONFIG_SENSORS_LTC4245=m
# CONFIG_SENSORS_LTC4260 is not set
CONFIG_SENSORS_LTC4261=m
# CONFIG_SENSORS_MAX1111 is not set
CONFIG_SENSORS_MAX16065=m
CONFIG_SENSORS_MAX1619=m
CONFIG_SENSORS_MAX1668=m
CONFIG_SENSORS_MAX197=m
# CONFIG_SENSORS_MAX31722 is not set
# CONFIG_SENSORS_MAX6621 is not set
CONFIG_SENSORS_MAX6639=m
CONFIG_SENSORS_MAX6642=m
CONFIG_SENSORS_MAX6650=m
CONFIG_SENSORS_MAX6697=m
# CONFIG_SENSORS_MAX31790 is not set
CONFIG_SENSORS_MCP3021=m
# CONFIG_SENSORS_TC654 is not set
# CONFIG_SENSORS_ADCXX is not set
CONFIG_SENSORS_LM63=m
# CONFIG_SENSORS_LM70 is not set
CONFIG_SENSORS_LM73=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
CONFIG_SENSORS_LM93=m
CONFIG_SENSORS_LM95234=m
CONFIG_SENSORS_LM95241=m
CONFIG_SENSORS_LM95245=m
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_PC87427=m
CONFIG_SENSORS_NTC_THERMISTOR=m
# CONFIG_SENSORS_NCT6683 is not set
CONFIG_SENSORS_NCT6775=m
# CONFIG_SENSORS_NCT7802 is not set
# CONFIG_SENSORS_NCT7904 is not set
# CONFIG_SENSORS_NPCM7XX is not set
CONFIG_SENSORS_PCF8591=m
CONFIG_PMBUS=m
CONFIG_SENSORS_PMBUS=m
CONFIG_SENSORS_ADM1275=m
# CONFIG_SENSORS_IBM_CFFPS is not set
# CONFIG_SENSORS_IR35221 is not set
# CONFIG_SENSORS_IR38064 is not set
# CONFIG_SENSORS_ISL68137 is not set
CONFIG_SENSORS_LM25066=m
CONFIG_SENSORS_LTC2978=m
# CONFIG_SENSORS_LTC3815 is not set
CONFIG_SENSORS_MAX16064=m
# CONFIG_SENSORS_MAX20751 is not set
# CONFIG_SENSORS_MAX31785 is not set
CONFIG_SENSORS_MAX34440=m
CONFIG_SENSORS_MAX8688=m
# CONFIG_SENSORS_TPS40422 is not set
# CONFIG_SENSORS_TPS53679 is not set
CONFIG_SENSORS_UCD9000=m
CONFIG_SENSORS_UCD9200=m
CONFIG_SENSORS_ZL6100=m
# CONFIG_SENSORS_SHT15 is not set
CONFIG_SENSORS_SHT21=m
# CONFIG_SENSORS_SHT3x is not set
# CONFIG_SENSORS_SHTC1 is not set
CONFIG_SENSORS_SIS5595=m
CONFIG_SENSORS_DME1737=m
CONFIG_SENSORS_EMC1403=m
# CONFIG_SENSORS_EMC2103 is not set
CONFIG_SENSORS_EMC6W201=m
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47M192=m
CONFIG_SENSORS_SMSC47B397=m
CONFIG_SENSORS_SCH56XX_COMMON=m
CONFIG_SENSORS_SCH5627=m
CONFIG_SENSORS_SCH5636=m
# CONFIG_SENSORS_STTS751 is not set
# CONFIG_SENSORS_SMM665 is not set
# CONFIG_SENSORS_ADC128D818 is not set
CONFIG_SENSORS_ADS1015=m
CONFIG_SENSORS_ADS7828=m
# CONFIG_SENSORS_ADS7871 is not set
CONFIG_SENSORS_AMC6821=m
CONFIG_SENSORS_INA209=m
CONFIG_SENSORS_INA2XX=m
# CONFIG_SENSORS_INA3221 is not set
# CONFIG_SENSORS_TC74 is not set
CONFIG_SENSORS_THMC50=m
CONFIG_SENSORS_TMP102=m
# CONFIG_SENSORS_TMP103 is not set
# CONFIG_SENSORS_TMP108 is not set
CONFIG_SENSORS_TMP401=m
CONFIG_SENSORS_TMP421=m
CONFIG_SENSORS_VIA_CPUTEMP=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_VT1211=m
CONFIG_SENSORS_VT8231=m
# CONFIG_SENSORS_W83773G is not set
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83793=m
CONFIG_SENSORS_W83795=m
# CONFIG_SENSORS_W83795_FANCTRL is not set
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83L786NG=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
# CONFIG_SENSORS_XGENE is not set

#
# ACPI drivers
#
CONFIG_SENSORS_ACPI_POWER=m
CONFIG_SENSORS_ATK0110=m
CONFIG_THERMAL=y
# CONFIG_THERMAL_STATISTICS is not set
CONFIG_THERMAL_EMERGENCY_POWEROFF_DELAY_MS=0
CONFIG_THERMAL_HWMON=y
CONFIG_THERMAL_WRITABLE_TRIPS=y
CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE=y
# CONFIG_THERMAL_DEFAULT_GOV_FAIR_SHARE is not set
# CONFIG_THERMAL_DEFAULT_GOV_USER_SPACE is not set
# CONFIG_THERMAL_DEFAULT_GOV_POWER_ALLOCATOR is not set
CONFIG_THERMAL_GOV_FAIR_SHARE=y
CONFIG_THERMAL_GOV_STEP_WISE=y
CONFIG_THERMAL_GOV_BANG_BANG=y
CONFIG_THERMAL_GOV_USER_SPACE=y
# CONFIG_THERMAL_GOV_POWER_ALLOCATOR is not set
# CONFIG_CLOCK_THERMAL is not set
# CONFIG_DEVFREQ_THERMAL is not set
# CONFIG_THERMAL_EMULATION is not set

#
# Intel thermal drivers
#
CONFIG_INTEL_POWERCLAMP=m
CONFIG_X86_PKG_TEMP_THERMAL=m
# CONFIG_INTEL_SOC_DTS_THERMAL is not set

#
# ACPI INT340X thermal drivers
#
# CONFIG_INT340X_THERMAL is not set
# end of ACPI INT340X thermal drivers

CONFIG_INTEL_PCH_THERMAL=m
# end of Intel thermal drivers

CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
# CONFIG_WATCHDOG_NOWAYOUT is not set
CONFIG_WATCHDOG_HANDLE_BOOT_ENABLED=y
# CONFIG_WATCHDOG_SYSFS is not set

#
# Watchdog Pretimeout Governors
#
# CONFIG_WATCHDOG_PRETIMEOUT_GOV is not set

#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
CONFIG_WDAT_WDT=m
# CONFIG_XILINX_WATCHDOG is not set
# CONFIG_ZIIRAVE_WATCHDOG is not set
# CONFIG_CADENCE_WATCHDOG is not set
# CONFIG_DW_WATCHDOG is not set
# CONFIG_MAX63XX_WATCHDOG is not set
# CONFIG_ACQUIRE_WDT is not set
# CONFIG_ADVANTECH_WDT is not set
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=m
# CONFIG_EBC_C384_WDT is not set
CONFIG_F71808E_WDT=m
CONFIG_SP5100_TCO=m
CONFIG_SBC_FITPC2_WATCHDOG=m
# CONFIG_EUROTECH_WDT is not set
CONFIG_IB700_WDT=m
CONFIG_IBMASR=m
# CONFIG_WAFER_WDT is not set
CONFIG_I6300ESB_WDT=y
CONFIG_IE6XX_WDT=m
CONFIG_ITCO_WDT=y
CONFIG_ITCO_VENDOR_SUPPORT=y
CONFIG_IT8712F_WDT=m
CONFIG_IT87_WDT=m
CONFIG_HP_WATCHDOG=m
CONFIG_HPWDT_NMI_DECODING=y
# CONFIG_SC1200_WDT is not set
# CONFIG_PC87413_WDT is not set
CONFIG_NV_TCO=m
# CONFIG_60XX_WDT is not set
# CONFIG_CPU5_WDT is not set
CONFIG_SMSC_SCH311X_WDT=m
# CONFIG_SMSC37B787_WDT is not set
# CONFIG_TQMX86_WDT is not set
CONFIG_VIA_WDT=m
CONFIG_W83627HF_WDT=m
CONFIG_W83877F_WDT=m
CONFIG_W83977F_WDT=m
CONFIG_MACHZ_WDT=m
# CONFIG_SBC_EPX_C3_WATCHDOG is not set
# CONFIG_INTEL_MEI_WDT is not set
# CONFIG_NI903X_WDT is not set
# CONFIG_NIC7018_WDT is not set
# CONFIG_MEN_A21_WDT is not set
CONFIG_XEN_WDT=m

#
# PCI-based Watchdog Cards
#
CONFIG_PCIPCWATCHDOG=m
CONFIG_WDTPCI=m

#
# USB-based Watchdog Cards
#
CONFIG_USBPCWATCHDOG=m
CONFIG_SSB_POSSIBLE=y
CONFIG_SSB=m
CONFIG_SSB_SPROM=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
CONFIG_SSB_SDIOHOST_POSSIBLE=y
CONFIG_SSB_SDIOHOST=y
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y
# CONFIG_SSB_DRIVER_GPIO is not set
CONFIG_BCMA_POSSIBLE=y
CONFIG_BCMA=m
CONFIG_BCMA_HOST_PCI_POSSIBLE=y
CONFIG_BCMA_HOST_PCI=y
# CONFIG_BCMA_HOST_SOC is not set
CONFIG_BCMA_DRIVER_PCI=y
CONFIG_BCMA_DRIVER_GMAC_CMN=y
# CONFIG_BCMA_DRIVER_GPIO is not set
# CONFIG_BCMA_DEBUG is not set

#
# Multifunction device drivers
#
CONFIG_MFD_CORE=y
# CONFIG_MFD_AS3711 is not set
# CONFIG_PMIC_ADP5520 is not set
# CONFIG_MFD_AAT2870_CORE is not set
# CONFIG_MFD_BCM590XX is not set
# CONFIG_MFD_BD9571MWV is not set
# CONFIG_MFD_AXP20X_I2C is not set
# CONFIG_MFD_CROS_EC is not set
# CONFIG_MFD_MADERA is not set
# CONFIG_PMIC_DA903X is not set
# CONFIG_MFD_DA9052_SPI is not set
# CONFIG_MFD_DA9052_I2C is not set
# CONFIG_MFD_DA9055 is not set
# CONFIG_MFD_DA9062 is not set
# CONFIG_MFD_DA9063 is not set
# CONFIG_MFD_DA9150 is not set
# CONFIG_MFD_DLN2 is not set
# CONFIG_MFD_MC13XXX_SPI is not set
# CONFIG_MFD_MC13XXX_I2C is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_HTC_I2CPLD is not set
# CONFIG_MFD_INTEL_QUARK_I2C_GPIO is not set
CONFIG_LPC_ICH=y
CONFIG_LPC_SCH=m
# CONFIG_INTEL_SOC_PMIC_CHTDC_TI is not set
# CONFIG_MFD_INTEL_LPSS_ACPI is not set
# CONFIG_MFD_INTEL_LPSS_PCI is not set
# CONFIG_MFD_JANZ_CMODIO is not set
# CONFIG_MFD_KEMPLD is not set
# CONFIG_MFD_88PM800 is not set
# CONFIG_MFD_88PM805 is not set
# CONFIG_MFD_88PM860X is not set
# CONFIG_MFD_MAX14577 is not set
# CONFIG_MFD_MAX77693 is not set
# CONFIG_MFD_MAX77843 is not set
# CONFIG_MFD_MAX8907 is not set
# CONFIG_MFD_MAX8925 is not set
# CONFIG_MFD_MAX8997 is not set
# CONFIG_MFD_MAX8998 is not set
# CONFIG_MFD_MT6397 is not set
# CONFIG_MFD_MENF21BMC is not set
# CONFIG_EZX_PCAP is not set
CONFIG_MFD_VIPERBOARD=m
# CONFIG_MFD_RETU is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_UCB1400_CORE is not set
# CONFIG_MFD_RDC321X is not set
# CONFIG_MFD_RT5033 is not set
# CONFIG_MFD_RC5T583 is not set
# CONFIG_MFD_SEC_CORE is not set
# CONFIG_MFD_SI476X_CORE is not set
CONFIG_MFD_SM501=m
# CONFIG_MFD_SM501_GPIO is not set
# CONFIG_MFD_SKY81452 is not set
# CONFIG_MFD_SMSC is not set
# CONFIG_ABX500_CORE is not set
# CONFIG_MFD_SYSCON is not set
# CONFIG_MFD_TI_AM335X_TSCADC is not set
# CONFIG_MFD_LP3943 is not set
# CONFIG_MFD_LP8788 is not set
# CONFIG_MFD_TI_LMU is not set
# CONFIG_MFD_PALMAS is not set
# CONFIG_TPS6105X is not set
# CONFIG_TPS65010 is not set
# CONFIG_TPS6507X is not set
# CONFIG_MFD_TPS65086 is not set
# CONFIG_MFD_TPS65090 is not set
# CONFIG_MFD_TI_LP873X is not set
# CONFIG_MFD_TPS6586X is not set
# CONFIG_MFD_TPS65910 is not set
# CONFIG_MFD_TPS65912_I2C is not set
# CONFIG_MFD_TPS65912_SPI is not set
# CONFIG_MFD_TPS80031 is not set
# CONFIG_TWL4030_CORE is not set
# CONFIG_TWL6040_CORE is not set
# CONFIG_MFD_WL1273_CORE is not set
# CONFIG_MFD_LM3533 is not set
# CONFIG_MFD_TQMX86 is not set
CONFIG_MFD_VX855=m
# CONFIG_MFD_ARIZONA_I2C is not set
# CONFIG_MFD_ARIZONA_SPI is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_WM831X_I2C is not set
# CONFIG_MFD_WM831X_SPI is not set
# CONFIG_MFD_WM8350_I2C is not set
# CONFIG_MFD_WM8994 is not set
# end of Multifunction device drivers

# CONFIG_REGULATOR is not set
CONFIG_RC_CORE=m
CONFIG_RC_MAP=m
CONFIG_LIRC=y
CONFIG_RC_DECODERS=y
CONFIG_IR_NEC_DECODER=m
CONFIG_IR_RC5_DECODER=m
CONFIG_IR_RC6_DECODER=m
CONFIG_IR_JVC_DECODER=m
CONFIG_IR_SONY_DECODER=m
CONFIG_IR_SANYO_DECODER=m
CONFIG_IR_SHARP_DECODER=m
CONFIG_IR_MCE_KBD_DECODER=m
CONFIG_IR_XMP_DECODER=m
# CONFIG_IR_IMON_DECODER is not set
# CONFIG_IR_RCMM_DECODER is not set
CONFIG_RC_DEVICES=y
CONFIG_RC_ATI_REMOTE=m
CONFIG_IR_ENE=m
CONFIG_IR_IMON=m
# CONFIG_IR_IMON_RAW is not set
CONFIG_IR_MCEUSB=m
CONFIG_IR_ITE_CIR=m
CONFIG_IR_FINTEK=m
CONFIG_IR_NUVOTON=m
CONFIG_IR_REDRAT3=m
CONFIG_IR_STREAMZAP=m
CONFIG_IR_WINBOND_CIR=m
# CONFIG_IR_IGORPLUGUSB is not set
CONFIG_IR_IGUANA=m
CONFIG_IR_TTUSBIR=m
CONFIG_RC_LOOPBACK=m
# CONFIG_IR_SERIAL is not set
# CONFIG_IR_SIR is not set
# CONFIG_RC_XBOX_DVD is not set
CONFIG_MEDIA_SUPPORT=m

#
# Multimedia core support
#
CONFIG_MEDIA_CAMERA_SUPPORT=y
CONFIG_MEDIA_ANALOG_TV_SUPPORT=y
CONFIG_MEDIA_DIGITAL_TV_SUPPORT=y
CONFIG_MEDIA_RADIO_SUPPORT=y
# CONFIG_MEDIA_SDR_SUPPORT is not set
# CONFIG_MEDIA_CEC_SUPPORT is not set
CONFIG_MEDIA_CONTROLLER=y
CONFIG_MEDIA_CONTROLLER_DVB=y
CONFIG_VIDEO_DEV=m
# CONFIG_VIDEO_V4L2_SUBDEV_API is not set
CONFIG_VIDEO_V4L2=m
# CONFIG_VIDEO_ADV_DEBUG is not set
# CONFIG_VIDEO_FIXED_MINOR_RANGES is not set
CONFIG_VIDEO_TUNER=m
CONFIG_VIDEOBUF_GEN=m
CONFIG_VIDEOBUF_DMA_SG=m
CONFIG_VIDEOBUF_VMALLOC=m
CONFIG_DVB_CORE=m
# CONFIG_DVB_MMAP is not set
CONFIG_DVB_NET=y
CONFIG_TTPCI_EEPROM=m
CONFIG_DVB_MAX_ADAPTERS=8
CONFIG_DVB_DYNAMIC_MINORS=y
# CONFIG_DVB_DEMUX_SECTION_LOSS_LOG is not set
# CONFIG_DVB_ULE_DEBUG is not set

#
# Media drivers
#
CONFIG_MEDIA_USB_SUPPORT=y

#
# Webcam devices
#
CONFIG_USB_VIDEO_CLASS=m
CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y
CONFIG_USB_GSPCA=m
CONFIG_USB_M5602=m
CONFIG_USB_STV06XX=m
CONFIG_USB_GL860=m
CONFIG_USB_GSPCA_BENQ=m
CONFIG_USB_GSPCA_CONEX=m
CONFIG_USB_GSPCA_CPIA1=m
# CONFIG_USB_GSPCA_DTCS033 is not set
CONFIG_USB_GSPCA_ETOMS=m
CONFIG_USB_GSPCA_FINEPIX=m
CONFIG_USB_GSPCA_JEILINJ=m
CONFIG_USB_GSPCA_JL2005BCD=m
# CONFIG_USB_GSPCA_KINECT is not set
CONFIG_USB_GSPCA_KONICA=m
CONFIG_USB_GSPCA_MARS=m
CONFIG_USB_GSPCA_MR97310A=m
CONFIG_USB_GSPCA_NW80X=m
CONFIG_USB_GSPCA_OV519=m
CONFIG_USB_GSPCA_OV534=m
CONFIG_USB_GSPCA_OV534_9=m
CONFIG_USB_GSPCA_PAC207=m
CONFIG_USB_GSPCA_PAC7302=m
CONFIG_USB_GSPCA_PAC7311=m
CONFIG_USB_GSPCA_SE401=m
CONFIG_USB_GSPCA_SN9C2028=m
CONFIG_USB_GSPCA_SN9C20X=m
CONFIG_USB_GSPCA_SONIXB=m
CONFIG_USB_GSPCA_SONIXJ=m
CONFIG_USB_GSPCA_SPCA500=m
CONFIG_USB_GSPCA_SPCA501=m
CONFIG_USB_GSPCA_SPCA505=m
CONFIG_USB_GSPCA_SPCA506=m
CONFIG_USB_GSPCA_SPCA508=m
CONFIG_USB_GSPCA_SPCA561=m
CONFIG_USB_GSPCA_SPCA1528=m
CONFIG_USB_GSPCA_SQ905=m
CONFIG_USB_GSPCA_SQ905C=m
CONFIG_USB_GSPCA_SQ930X=m
CONFIG_USB_GSPCA_STK014=m
# CONFIG_USB_GSPCA_STK1135 is not set
CONFIG_USB_GSPCA_STV0680=m
CONFIG_USB_GSPCA_SUNPLUS=m
CONFIG_USB_GSPCA_T613=m
CONFIG_USB_GSPCA_TOPRO=m
# CONFIG_USB_GSPCA_TOUPTEK is not set
CONFIG_USB_GSPCA_TV8532=m
CONFIG_USB_GSPCA_VC032X=m
CONFIG_USB_GSPCA_VICAM=m
CONFIG_USB_GSPCA_XIRLINK_CIT=m
CONFIG_USB_GSPCA_ZC3XX=m
CONFIG_USB_PWC=m
# CONFIG_USB_PWC_DEBUG is not set
CONFIG_USB_PWC_INPUT_EVDEV=y
# CONFIG_VIDEO_CPIA2 is not set
CONFIG_USB_ZR364XX=m
CONFIG_USB_STKWEBCAM=m
CONFIG_USB_S2255=m
# CONFIG_VIDEO_USBTV is not set

#
# Analog TV USB devices
#
CONFIG_VIDEO_PVRUSB2=m
CONFIG_VIDEO_PVRUSB2_SYSFS=y
CONFIG_VIDEO_PVRUSB2_DVB=y
# CONFIG_VIDEO_PVRUSB2_DEBUGIFC is not set
CONFIG_VIDEO_HDPVR=m
CONFIG_VIDEO_USBVISION=m
# CONFIG_VIDEO_STK1160_COMMON is not set
# CONFIG_VIDEO_GO7007 is not set

#
# Analog/digital TV USB devices
#
CONFIG_VIDEO_AU0828=m
CONFIG_VIDEO_AU0828_V4L2=y
# CONFIG_VIDEO_AU0828_RC is not set
CONFIG_VIDEO_CX231XX=m
CONFIG_VIDEO_CX231XX_RC=y
CONFIG_VIDEO_CX231XX_ALSA=m
CONFIG_VIDEO_CX231XX_DVB=m
CONFIG_VIDEO_TM6000=m
CONFIG_VIDEO_TM6000_ALSA=m
CONFIG_VIDEO_TM6000_DVB=m

#
# Digital TV USB devices
#
CONFIG_DVB_USB=m
# CONFIG_DVB_USB_DEBUG is not set
CONFIG_DVB_USB_DIB3000MC=m
CONFIG_DVB_USB_A800=m
CONFIG_DVB_USB_DIBUSB_MB=m
# CONFIG_DVB_USB_DIBUSB_MB_FAULTY is not set
CONFIG_DVB_USB_DIBUSB_MC=m
CONFIG_DVB_USB_DIB0700=m
CONFIG_DVB_USB_UMT_010=m
CONFIG_DVB_USB_CXUSB=m
CONFIG_DVB_USB_M920X=m
CONFIG_DVB_USB_DIGITV=m
CONFIG_DVB_USB_VP7045=m
CONFIG_DVB_USB_VP702X=m
CONFIG_DVB_USB_GP8PSK=m
CONFIG_DVB_USB_NOVA_T_USB2=m
CONFIG_DVB_USB_TTUSB2=m
CONFIG_DVB_USB_DTT200U=m
CONFIG_DVB_USB_OPERA1=m
CONFIG_DVB_USB_AF9005=m
CONFIG_DVB_USB_AF9005_REMOTE=m
CONFIG_DVB_USB_PCTV452E=m
CONFIG_DVB_USB_DW2102=m
CONFIG_DVB_USB_CINERGY_T2=m
CONFIG_DVB_USB_DTV5100=m
CONFIG_DVB_USB_AZ6027=m
CONFIG_DVB_USB_TECHNISAT_USB2=m
CONFIG_DVB_USB_V2=m
CONFIG_DVB_USB_AF9015=m
CONFIG_DVB_USB_AF9035=m
CONFIG_DVB_USB_ANYSEE=m
CONFIG_DVB_USB_AU6610=m
CONFIG_DVB_USB_AZ6007=m
CONFIG_DVB_USB_CE6230=m
CONFIG_DVB_USB_EC168=m
CONFIG_DVB_USB_GL861=m
CONFIG_DVB_USB_LME2510=m
CONFIG_DVB_USB_MXL111SF=m
CONFIG_DVB_USB_RTL28XXU=m
# CONFIG_DVB_USB_DVBSKY is not set
# CONFIG_DVB_USB_ZD1301 is not set
CONFIG_DVB_TTUSB_BUDGET=m
CONFIG_DVB_TTUSB_DEC=m
CONFIG_SMS_USB_DRV=m
CONFIG_DVB_B2C2_FLEXCOP_USB=m
# CONFIG_DVB_B2C2_FLEXCOP_USB_DEBUG is not set
# CONFIG_DVB_AS102 is not set

#
# Webcam, TV (analog/digital) USB devices
#
CONFIG_VIDEO_EM28XX=m
# CONFIG_VIDEO_EM28XX_V4L2 is not set
CONFIG_VIDEO_EM28XX_ALSA=m
CONFIG_VIDEO_EM28XX_DVB=m
CONFIG_VIDEO_EM28XX_RC=m
CONFIG_MEDIA_PCI_SUPPORT=y

#
# Media capture support
#
# CONFIG_VIDEO_MEYE is not set
# CONFIG_VIDEO_SOLO6X10 is not set
# CONFIG_VIDEO_TW5864 is not set
# CONFIG_VIDEO_TW68 is not set
# CONFIG_VIDEO_TW686X is not set

#
# Media capture/analog TV support
#
CONFIG_VIDEO_IVTV=m
# CONFIG_VIDEO_IVTV_DEPRECATED_IOCTLS is not set
# CONFIG_VIDEO_IVTV_ALSA is not set
CONFIG_VIDEO_FB_IVTV=m
# CONFIG_VIDEO_FB_IVTV_FORCE_PAT is not set
# CONFIG_VIDEO_HEXIUM_GEMINI is not set
# CONFIG_VIDEO_HEXIUM_ORION is not set
# CONFIG_VIDEO_MXB is not set
# CONFIG_VIDEO_DT3155 is not set

#
# Media capture/analog/hybrid TV support
#
CONFIG_VIDEO_CX18=m
CONFIG_VIDEO_CX18_ALSA=m
CONFIG_VIDEO_CX23885=m
CONFIG_MEDIA_ALTERA_CI=m
# CONFIG_VIDEO_CX25821 is not set
CONFIG_VIDEO_CX88=m
CONFIG_VIDEO_CX88_ALSA=m
CONFIG_VIDEO_CX88_BLACKBIRD=m
CONFIG_VIDEO_CX88_DVB=m
CONFIG_VIDEO_CX88_ENABLE_VP3054=y
CONFIG_VIDEO_CX88_VP3054=m
CONFIG_VIDEO_CX88_MPEG=m
CONFIG_VIDEO_BT848=m
CONFIG_DVB_BT8XX=m
CONFIG_VIDEO_SAA7134=m
CONFIG_VIDEO_SAA7134_ALSA=m
CONFIG_VIDEO_SAA7134_RC=y
CONFIG_VIDEO_SAA7134_DVB=m
CONFIG_VIDEO_SAA7164=m

#
# Media digital TV PCI Adapters
#
CONFIG_DVB_AV7110_IR=y
CONFIG_DVB_AV7110=m
CONFIG_DVB_AV7110_OSD=y
CONFIG_DVB_BUDGET_CORE=m
CONFIG_DVB_BUDGET=m
CONFIG_DVB_BUDGET_CI=m
CONFIG_DVB_BUDGET_AV=m
CONFIG_DVB_BUDGET_PATCH=m
CONFIG_DVB_B2C2_FLEXCOP_PCI=m
# CONFIG_DVB_B2C2_FLEXCOP_PCI_DEBUG is not set
CONFIG_DVB_PLUTO2=m
CONFIG_DVB_DM1105=m
CONFIG_DVB_PT1=m
# CONFIG_DVB_PT3 is not set
CONFIG_MANTIS_CORE=m
CONFIG_DVB_MANTIS=m
CONFIG_DVB_HOPPER=m
CONFIG_DVB_NGENE=m
CONFIG_DVB_DDBRIDGE=m
# CONFIG_DVB_DDBRIDGE_MSIENABLE is not set
# CONFIG_DVB_SMIPCIE is not set
# CONFIG_DVB_NETUP_UNIDVB is not set
# CONFIG_V4L_PLATFORM_DRIVERS is not set
# CONFIG_V4L_MEM2MEM_DRIVERS is not set
# CONFIG_V4L_TEST_DRIVERS is not set
# CONFIG_DVB_PLATFORM_DRIVERS is not set

#
# Supported MMC/SDIO adapters
#
CONFIG_SMS_SDIO_DRV=m
CONFIG_RADIO_ADAPTERS=y
CONFIG_RADIO_TEA575X=m
# CONFIG_RADIO_SI470X is not set
# CONFIG_RADIO_SI4713 is not set
# CONFIG_USB_MR800 is not set
# CONFIG_USB_DSBR is not set
# CONFIG_RADIO_MAXIRADIO is not set
# CONFIG_RADIO_SHARK is not set
# CONFIG_RADIO_SHARK2 is not set
# CONFIG_USB_KEENE is not set
# CONFIG_USB_RAREMONO is not set
# CONFIG_USB_MA901 is not set
# CONFIG_RADIO_TEA5764 is not set
# CONFIG_RADIO_SAA7706H is not set
# CONFIG_RADIO_TEF6862 is not set
# CONFIG_RADIO_WL1273 is not set

#
# Texas Instruments WL128x FM driver (ST based)
#
# end of Texas Instruments WL128x FM driver (ST based)

#
# Supported FireWire (IEEE 1394) Adapters
#
CONFIG_DVB_FIREDTV=m
CONFIG_DVB_FIREDTV_INPUT=y
CONFIG_MEDIA_COMMON_OPTIONS=y

#
# common driver options
#
CONFIG_VIDEO_CX2341X=m
CONFIG_VIDEO_TVEEPROM=m
CONFIG_CYPRESS_FIRMWARE=m
CONFIG_VIDEOBUF2_CORE=m
CONFIG_VIDEOBUF2_V4L2=m
CONFIG_VIDEOBUF2_MEMOPS=m
CONFIG_VIDEOBUF2_VMALLOC=m
CONFIG_VIDEOBUF2_DMA_SG=m
CONFIG_VIDEOBUF2_DVB=m
CONFIG_DVB_B2C2_FLEXCOP=m
CONFIG_VIDEO_SAA7146=m
CONFIG_VIDEO_SAA7146_VV=m
CONFIG_SMS_SIANO_MDTV=m
CONFIG_SMS_SIANO_RC=y
# CONFIG_SMS_SIANO_DEBUGFS is not set

#
# Media ancillary drivers (tuners, sensors, i2c, spi, frontends)
#
CONFIG_MEDIA_SUBDRV_AUTOSELECT=y
CONFIG_MEDIA_ATTACH=y
CONFIG_VIDEO_IR_I2C=m

#
# Audio decoders, processors and mixers
#
CONFIG_VIDEO_TVAUDIO=m
CONFIG_VIDEO_TDA7432=m
CONFIG_VIDEO_MSP3400=m
CONFIG_VIDEO_CS3308=m
CONFIG_VIDEO_CS5345=m
CONFIG_VIDEO_CS53L32A=m
CONFIG_VIDEO_WM8775=m
CONFIG_VIDEO_WM8739=m
CONFIG_VIDEO_VP27SMPX=m

#
# RDS decoders
#
CONFIG_VIDEO_SAA6588=m

#
# Video decoders
#
CONFIG_VIDEO_SAA711X=m

#
# Video and audio decoders
#
CONFIG_VIDEO_SAA717X=m
CONFIG_VIDEO_CX25840=m

#
# Video encoders
#
CONFIG_VIDEO_SAA7127=m

#
# Camera sensor devices
#

#
# Lens drivers
#

#
# Flash devices
#

#
# Video improvement chips
#
CONFIG_VIDEO_UPD64031A=m
CONFIG_VIDEO_UPD64083=m

#
# Audio/Video compression chips
#
CONFIG_VIDEO_SAA6752HS=m

#
# SDR tuner chips
#

#
# Miscellaneous helper chips
#
CONFIG_VIDEO_M52790=m

#
# Media SPI Adapters
#
# CONFIG_CXD2880_SPI_DRV is not set
# end of Media SPI Adapters

CONFIG_MEDIA_TUNER=m
CONFIG_MEDIA_TUNER_SIMPLE=m
CONFIG_MEDIA_TUNER_TDA18250=m
CONFIG_MEDIA_TUNER_TDA8290=m
CONFIG_MEDIA_TUNER_TDA827X=m
CONFIG_MEDIA_TUNER_TDA18271=m
CONFIG_MEDIA_TUNER_TDA9887=m
CONFIG_MEDIA_TUNER_TEA5761=m
CONFIG_MEDIA_TUNER_TEA5767=m
CONFIG_MEDIA_TUNER_MT20XX=m
CONFIG_MEDIA_TUNER_MT2060=m
CONFIG_MEDIA_TUNER_MT2063=m
CONFIG_MEDIA_TUNER_MT2266=m
CONFIG_MEDIA_TUNER_MT2131=m
CONFIG_MEDIA_TUNER_QT1010=m
CONFIG_MEDIA_TUNER_XC2028=m
CONFIG_MEDIA_TUNER_XC5000=m
CONFIG_MEDIA_TUNER_XC4000=m
CONFIG_MEDIA_TUNER_MXL5005S=m
CONFIG_MEDIA_TUNER_MXL5007T=m
CONFIG_MEDIA_TUNER_MC44S803=m
CONFIG_MEDIA_TUNER_MAX2165=m
CONFIG_MEDIA_TUNER_TDA18218=m
CONFIG_MEDIA_TUNER_FC0011=m
CONFIG_MEDIA_TUNER_FC0012=m
CONFIG_MEDIA_TUNER_FC0013=m
CONFIG_MEDIA_TUNER_TDA18212=m
CONFIG_MEDIA_TUNER_E4000=m
CONFIG_MEDIA_TUNER_FC2580=m
CONFIG_MEDIA_TUNER_M88RS6000T=m
CONFIG_MEDIA_TUNER_TUA9001=m
CONFIG_MEDIA_TUNER_SI2157=m
CONFIG_MEDIA_TUNER_IT913X=m
CONFIG_MEDIA_TUNER_R820T=m
CONFIG_MEDIA_TUNER_QM1D1C0042=m
CONFIG_MEDIA_TUNER_QM1D1B0004=m

#
# Multistandard (satellite) frontends
#
CONFIG_DVB_STB0899=m
CONFIG_DVB_STB6100=m
CONFIG_DVB_STV090x=m
CONFIG_DVB_STV0910=m
CONFIG_DVB_STV6110x=m
CONFIG_DVB_STV6111=m
CONFIG_DVB_MXL5XX=m
CONFIG_DVB_M88DS3103=m

#
# Multistandard (cable + terrestrial) frontends
#
CONFIG_DVB_DRXK=m
CONFIG_DVB_TDA18271C2DD=m
CONFIG_DVB_SI2165=m
CONFIG_DVB_MN88472=m
CONFIG_DVB_MN88473=m

#
# DVB-S (satellite) frontends
#
CONFIG_DVB_CX24110=m
CONFIG_DVB_CX24123=m
CONFIG_DVB_MT312=m
CONFIG_DVB_ZL10036=m
CONFIG_DVB_ZL10039=m
CONFIG_DVB_S5H1420=m
CONFIG_DVB_STV0288=m
CONFIG_DVB_STB6000=m
CONFIG_DVB_STV0299=m
CONFIG_DVB_STV6110=m
CONFIG_DVB_STV0900=m
CONFIG_DVB_TDA8083=m
CONFIG_DVB_TDA10086=m
CONFIG_DVB_TDA8261=m
CONFIG_DVB_VES1X93=m
CONFIG_DVB_TUNER_ITD1000=m
CONFIG_DVB_TUNER_CX24113=m
CONFIG_DVB_TDA826X=m
CONFIG_DVB_TUA6100=m
CONFIG_DVB_CX24116=m
CONFIG_DVB_CX24117=m
CONFIG_DVB_CX24120=m
CONFIG_DVB_SI21XX=m
CONFIG_DVB_TS2020=m
CONFIG_DVB_DS3000=m
CONFIG_DVB_MB86A16=m
CONFIG_DVB_TDA10071=m

#
# DVB-T (terrestrial) frontends
#
CONFIG_DVB_SP8870=m
CONFIG_DVB_SP887X=m
CONFIG_DVB_CX22700=m
CONFIG_DVB_CX22702=m
CONFIG_DVB_DRXD=m
CONFIG_DVB_L64781=m
CONFIG_DVB_TDA1004X=m
CONFIG_DVB_NXT6000=m
CONFIG_DVB_MT352=m
CONFIG_DVB_ZL10353=m
CONFIG_DVB_DIB3000MB=m
CONFIG_DVB_DIB3000MC=m
CONFIG_DVB_DIB7000M=m
CONFIG_DVB_DIB7000P=m
CONFIG_DVB_TDA10048=m
CONFIG_DVB_AF9013=m
CONFIG_DVB_EC100=m
CONFIG_DVB_STV0367=m
CONFIG_DVB_CXD2820R=m
CONFIG_DVB_CXD2841ER=m
CONFIG_DVB_RTL2830=m
CONFIG_DVB_RTL2832=m
CONFIG_DVB_SI2168=m
CONFIG_DVB_GP8PSK_FE=m

#
# DVB-C (cable) frontends
#
CONFIG_DVB_VES1820=m
CONFIG_DVB_TDA10021=m
CONFIG_DVB_TDA10023=m
CONFIG_DVB_STV0297=m

#
# ATSC (North American/Korean Terrestrial/Cable DTV) frontends
#
CONFIG_DVB_NXT200X=m
CONFIG_DVB_OR51211=m
CONFIG_DVB_OR51132=m
CONFIG_DVB_BCM3510=m
CONFIG_DVB_LGDT330X=m
CONFIG_DVB_LGDT3305=m
CONFIG_DVB_LGDT3306A=m
CONFIG_DVB_LG2160=m
CONFIG_DVB_S5H1409=m
CONFIG_DVB_AU8522=m
CONFIG_DVB_AU8522_DTV=m
CONFIG_DVB_AU8522_V4L=m
CONFIG_DVB_S5H1411=m

#
# ISDB-T (terrestrial) frontends
#
CONFIG_DVB_S921=m
CONFIG_DVB_DIB8000=m
CONFIG_DVB_MB86A20S=m

#
# ISDB-S (satellite) & ISDB-T (terrestrial) frontends
#
CONFIG_DVB_TC90522=m

#
# Digital terrestrial only tuners/PLL
#
CONFIG_DVB_PLL=m
CONFIG_DVB_TUNER_DIB0070=m
CONFIG_DVB_TUNER_DIB0090=m

#
# SEC control devices for DVB-S
#
CONFIG_DVB_DRX39XYJ=m
CONFIG_DVB_LNBH25=m
CONFIG_DVB_LNBP21=m
CONFIG_DVB_LNBP22=m
CONFIG_DVB_ISL6405=m
CONFIG_DVB_ISL6421=m
CONFIG_DVB_ISL6423=m
CONFIG_DVB_A8293=m
CONFIG_DVB_LGS8GXX=m
CONFIG_DVB_ATBM8830=m
CONFIG_DVB_TDA665x=m
CONFIG_DVB_IX2505V=m
CONFIG_DVB_M88RS2000=m
CONFIG_DVB_AF9033=m

#
# Common Interface (EN50221) controller drivers
#
CONFIG_DVB_CXD2099=m

#
# Tools to develop new frontends
#
CONFIG_DVB_DUMMY_FE=m

#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_AGP_SIS=y
CONFIG_AGP_VIA=y
CONFIG_INTEL_GTT=y
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=64
CONFIG_VGA_SWITCHEROO=y
CONFIG_DRM=m
CONFIG_DRM_MIPI_DSI=y
# CONFIG_DRM_DP_AUX_CHARDEV is not set
CONFIG_DRM_DEBUG_SELFTEST=m
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_KMS_FB_HELPER=y
CONFIG_DRM_FBDEV_EMULATION=y
CONFIG_DRM_FBDEV_OVERALLOC=100
# CONFIG_DRM_FBDEV_LEAK_PHYS_SMEM is not set
CONFIG_DRM_LOAD_EDID_FIRMWARE=y
# CONFIG_DRM_DP_CEC is not set
CONFIG_DRM_TTM=m
CONFIG_DRM_GEM_SHMEM_HELPER=y

#
# I2C encoder or helper chips
#
CONFIG_DRM_I2C_CH7006=m
CONFIG_DRM_I2C_SIL164=m
CONFIG_DRM_I2C_NXP_TDA998X=m
# CONFIG_DRM_I2C_NXP_TDA9950 is not set
# end of I2C encoder or helper chips

#
# ARM devices
#
# end of ARM devices

# CONFIG_DRM_RADEON is not set
# CONFIG_DRM_AMDGPU is not set

#
# ACP (Audio CoProcessor) Configuration
#
# end of ACP (Audio CoProcessor) Configuration

# CONFIG_DRM_NOUVEAU is not set
CONFIG_DRM_I915=m
# CONFIG_DRM_I915_ALPHA_SUPPORT is not set
CONFIG_DRM_I915_CAPTURE_ERROR=y
CONFIG_DRM_I915_COMPRESS_ERROR=y
CONFIG_DRM_I915_USERPTR=y
# CONFIG_DRM_I915_GVT is not set

#
# drm/i915 Debugging
#
# CONFIG_DRM_I915_WERROR is not set
# CONFIG_DRM_I915_DEBUG is not set
# CONFIG_DRM_I915_SW_FENCE_DEBUG_OBJECTS is not set
# CONFIG_DRM_I915_SW_FENCE_CHECK_DAG is not set
# CONFIG_DRM_I915_DEBUG_GUC is not set
# CONFIG_DRM_I915_SELFTEST is not set
# CONFIG_DRM_I915_LOW_LEVEL_TRACEPOINTS is not set
# CONFIG_DRM_I915_DEBUG_VBLANK_EVADE is not set
# CONFIG_DRM_I915_DEBUG_RUNTIME_PM is not set
# end of drm/i915 Debugging

CONFIG_DRM_VGEM=m
# CONFIG_DRM_VKMS is not set
CONFIG_DRM_VMWGFX=m
CONFIG_DRM_VMWGFX_FBCON=y
CONFIG_DRM_GMA500=m
CONFIG_DRM_GMA600=y
CONFIG_DRM_GMA3600=y
CONFIG_DRM_UDL=m
CONFIG_DRM_AST=m
CONFIG_DRM_MGAG200=m
CONFIG_DRM_CIRRUS_QEMU=m
CONFIG_DRM_QXL=m
# CONFIG_DRM_BOCHS is not set
# CONFIG_DRM_VIRTIO_GPU is not set
CONFIG_DRM_PANEL=y

#
# Display Panels
#
# CONFIG_DRM_PANEL_RASPBERRYPI_TOUCHSCREEN is not set
# end of Display Panels

CONFIG_DRM_BRIDGE=y
CONFIG_DRM_PANEL_BRIDGE=y

#
# Display Interface Bridges
#
# CONFIG_DRM_ANALOGIX_ANX78XX is not set
# end of Display Interface Bridges

# CONFIG_DRM_ETNAVIV is not set
# CONFIG_DRM_HISI_HIBMC is not set
# CONFIG_DRM_TINYDRM is not set
# CONFIG_DRM_XEN is not set
# CONFIG_DRM_VBOXVIDEO is not set
# CONFIG_DRM_LEGACY is not set
CONFIG_DRM_PANEL_ORIENTATION_QUIRKS=y
CONFIG_DRM_LIB_RANDOM=y

#
# Frame buffer Devices
#
CONFIG_FB_CMDLINE=y
CONFIG_FB_NOTIFY=y
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
CONFIG_FB_SYS_FILLRECT=m
CONFIG_FB_SYS_COPYAREA=m
CONFIG_FB_SYS_IMAGEBLIT=m
# CONFIG_FB_FOREIGN_ENDIAN is not set
CONFIG_FB_SYS_FOPS=m
CONFIG_FB_DEFERRED_IO=y
# CONFIG_FB_MODE_HELPERS is not set
CONFIG_FB_TILEBLITTING=y

#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_UVESA is not set
CONFIG_FB_VESA=y
CONFIG_FB_EFI=y
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_OPENCORES is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_I740 is not set
# CONFIG_FB_LE80578 is not set
# CONFIG_FB_INTEL is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_VIA is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_VT8623 is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_SM501 is not set
# CONFIG_FB_SMSCUFX is not set
# CONFIG_FB_UDL is not set
# CONFIG_FB_IBM_GXT4500 is not set
# CONFIG_FB_VIRTUAL is not set
# CONFIG_XEN_FBDEV_FRONTEND is not set
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
CONFIG_FB_HYPERV=m
# CONFIG_FB_SIMPLE is not set
# CONFIG_FB_SM712 is not set
# end of Frame buffer Devices

#
# Backlight & LCD device support
#
CONFIG_LCD_CLASS_DEVICE=m
# CONFIG_LCD_L4F00242T03 is not set
# CONFIG_LCD_LMS283GF05 is not set
# CONFIG_LCD_LTV350QV is not set
# CONFIG_LCD_ILI922X is not set
# CONFIG_LCD_ILI9320 is not set
# CONFIG_LCD_TDO24M is not set
# CONFIG_LCD_VGG2432A4 is not set
CONFIG_LCD_PLATFORM=m
# CONFIG_LCD_AMS369FG06 is not set
# CONFIG_LCD_LMS501KF03 is not set
# CONFIG_LCD_HX8357 is not set
# CONFIG_LCD_OTM3225A is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
# CONFIG_BACKLIGHT_GENERIC is not set
# CONFIG_BACKLIGHT_PWM is not set
CONFIG_BACKLIGHT_APPLE=m
# CONFIG_BACKLIGHT_PM8941_WLED is not set
# CONFIG_BACKLIGHT_SAHARA is not set
# CONFIG_BACKLIGHT_ADP8860 is not set
# CONFIG_BACKLIGHT_ADP8870 is not set
# CONFIG_BACKLIGHT_LM3630A is not set
# CONFIG_BACKLIGHT_LM3639 is not set
# CONFIG_BACKLIGHT_LP855X is not set
# CONFIG_BACKLIGHT_GPIO is not set
# CONFIG_BACKLIGHT_LV5207LP is not set
# CONFIG_BACKLIGHT_BD6107 is not set
# CONFIG_BACKLIGHT_ARCXCNN is not set
# end of Backlight & LCD device support

CONFIG_HDMI=y

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
# CONFIG_VGACON_SOFT_SCROLLBACK_PERSISTENT_ENABLE_BY_DEFAULT is not set
CONFIG_DUMMY_CONSOLE=y
CONFIG_DUMMY_CONSOLE_COLUMNS=80
CONFIG_DUMMY_CONSOLE_ROWS=25
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
# CONFIG_FRAMEBUFFER_CONSOLE_DEFERRED_TAKEOVER is not set
# end of Console display driver support

CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y
# end of Graphics support

CONFIG_SOUND=m
CONFIG_SOUND_OSS_CORE=y
CONFIG_SOUND_OSS_CORE_PRECLAIM=y
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_SEQ_DEVICE=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_JACK=y
CONFIG_SND_JACK_INPUT_DEV=y
CONFIG_SND_OSSEMUL=y
# CONFIG_SND_MIXER_OSS is not set
# CONFIG_SND_PCM_OSS is not set
CONFIG_SND_PCM_TIMER=y
CONFIG_SND_HRTIMER=m
CONFIG_SND_DYNAMIC_MINORS=y
CONFIG_SND_MAX_CARDS=32
# CONFIG_SND_SUPPORT_OLD_API is not set
CONFIG_SND_PROC_FS=y
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_SEQUENCER_OSS=m
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_SEQ_MIDI_EVENT=m
CONFIG_SND_SEQ_MIDI=m
CONFIG_SND_SEQ_MIDI_EMUL=m
CONFIG_SND_SEQ_VIRMIDI=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_OPL3_LIB=m
CONFIG_SND_OPL3_LIB_SEQ=m
CONFIG_SND_VX_LIB=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DRIVERS=y
CONFIG_SND_PCSP=m
CONFIG_SND_DUMMY=m
CONFIG_SND_ALOOP=m
CONFIG_SND_VIRMIDI=m
CONFIG_SND_MTPAV=m
# CONFIG_SND_MTS64 is not set
# CONFIG_SND_SERIAL_U16550 is not set
CONFIG_SND_MPU401=m
# CONFIG_SND_PORTMAN2X4 is not set
CONFIG_SND_AC97_POWER_SAVE=y
CONFIG_SND_AC97_POWER_SAVE_DEFAULT=5
CONFIG_SND_PCI=y
CONFIG_SND_AD1889=m
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
CONFIG_SND_ALI5451=m
CONFIG_SND_ASIHPI=m
CONFIG_SND_ATIIXP=m
CONFIG_SND_ATIIXP_MODEM=m
CONFIG_SND_AU8810=m
CONFIG_SND_AU8820=m
CONFIG_SND_AU8830=m
# CONFIG_SND_AW2 is not set
# CONFIG_SND_AZT3328 is not set
CONFIG_SND_BT87X=m
# CONFIG_SND_BT87X_OVERCLOCK is not set
CONFIG_SND_CA0106=m
CONFIG_SND_CMIPCI=m
CONFIG_SND_OXYGEN_LIB=m
CONFIG_SND_OXYGEN=m
# CONFIG_SND_CS4281 is not set
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
CONFIG_SND_CTXFI=m
CONFIG_SND_DARLA20=m
CONFIG_SND_GINA20=m
CONFIG_SND_LAYLA20=m
CONFIG_SND_DARLA24=m
CONFIG_SND_GINA24=m
CONFIG_SND_LAYLA24=m
CONFIG_SND_MONA=m
CONFIG_SND_MIA=m
CONFIG_SND_ECHO3G=m
CONFIG_SND_INDIGO=m
CONFIG_SND_INDIGOIO=m
CONFIG_SND_INDIGODJ=m
CONFIG_SND_INDIGOIOX=m
CONFIG_SND_INDIGODJX=m
CONFIG_SND_EMU10K1=m
CONFIG_SND_EMU10K1_SEQ=m
CONFIG_SND_EMU10K1X=m
CONFIG_SND_ENS1370=m
CONFIG_SND_ENS1371=m
# CONFIG_SND_ES1938 is not set
CONFIG_SND_ES1968=m
CONFIG_SND_ES1968_INPUT=y
CONFIG_SND_ES1968_RADIO=y
# CONFIG_SND_FM801 is not set
CONFIG_SND_HDSP=m
CONFIG_SND_HDSPM=m
CONFIG_SND_ICE1712=m
CONFIG_SND_ICE1724=m
CONFIG_SND_INTEL8X0=m
CONFIG_SND_INTEL8X0M=m
CONFIG_SND_KORG1212=m
CONFIG_SND_LOLA=m
CONFIG_SND_LX6464ES=m
CONFIG_SND_MAESTRO3=m
CONFIG_SND_MAESTRO3_INPUT=y
CONFIG_SND_MIXART=m
# CONFIG_SND_NM256 is not set
CONFIG_SND_PCXHR=m
# CONFIG_SND_RIPTIDE is not set
CONFIG_SND_RME32=m
CONFIG_SND_RME96=m
CONFIG_SND_RME9652=m
# CONFIG_SND_SONICVIBES is not set
CONFIG_SND_TRIDENT=m
CONFIG_SND_VIA82XX=m
CONFIG_SND_VIA82XX_MODEM=m
CONFIG_SND_VIRTUOSO=m
CONFIG_SND_VX222=m
# CONFIG_SND_YMFPCI is not set

#
# HD-Audio
#
CONFIG_SND_HDA=m
CONFIG_SND_HDA_INTEL=m
CONFIG_SND_HDA_HWDEP=y
# CONFIG_SND_HDA_RECONFIG is not set
CONFIG_SND_HDA_INPUT_BEEP=y
CONFIG_SND_HDA_INPUT_BEEP_MODE=0
# CONFIG_SND_HDA_PATCH_LOADER is not set
CONFIG_SND_HDA_CODEC_REALTEK=m
CONFIG_SND_HDA_CODEC_ANALOG=m
CONFIG_SND_HDA_CODEC_SIGMATEL=m
CONFIG_SND_HDA_CODEC_VIA=m
CONFIG_SND_HDA_CODEC_HDMI=m
CONFIG_SND_HDA_CODEC_CIRRUS=m
CONFIG_SND_HDA_CODEC_CONEXANT=m
CONFIG_SND_HDA_CODEC_CA0110=m
CONFIG_SND_HDA_CODEC_CA0132=m
CONFIG_SND_HDA_CODEC_CA0132_DSP=y
CONFIG_SND_HDA_CODEC_CMEDIA=m
CONFIG_SND_HDA_CODEC_SI3054=m
CONFIG_SND_HDA_GENERIC=m
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=0
# end of HD-Audio

CONFIG_SND_HDA_CORE=m
CONFIG_SND_HDA_DSP_LOADER=y
CONFIG_SND_HDA_COMPONENT=y
CONFIG_SND_HDA_I915=y
CONFIG_SND_HDA_PREALLOC_SIZE=512
CONFIG_SND_SPI=y
CONFIG_SND_USB=y
CONFIG_SND_USB_AUDIO=m
CONFIG_SND_USB_AUDIO_USE_MEDIA_CONTROLLER=y
CONFIG_SND_USB_UA101=m
CONFIG_SND_USB_USX2Y=m
CONFIG_SND_USB_CAIAQ=m
CONFIG_SND_USB_CAIAQ_INPUT=y
CONFIG_SND_USB_US122L=m
CONFIG_SND_USB_6FIRE=m
# CONFIG_SND_USB_HIFACE is not set
# CONFIG_SND_BCD2000 is not set
# CONFIG_SND_USB_POD is not set
# CONFIG_SND_USB_PODHD is not set
# CONFIG_SND_USB_TONEPORT is not set
# CONFIG_SND_USB_VARIAX is not set
CONFIG_SND_FIREWIRE=y
CONFIG_SND_FIREWIRE_LIB=m
# CONFIG_SND_DICE is not set
# CONFIG_SND_OXFW is not set
CONFIG_SND_ISIGHT=m
# CONFIG_SND_FIREWORKS is not set
# CONFIG_SND_BEBOB is not set
# CONFIG_SND_FIREWIRE_DIGI00X is not set
# CONFIG_SND_FIREWIRE_TASCAM is not set
# CONFIG_SND_FIREWIRE_MOTU is not set
# CONFIG_SND_FIREFACE is not set
# CONFIG_SND_SOC is not set
CONFIG_SND_X86=y
# CONFIG_HDMI_LPE_AUDIO is not set
CONFIG_SND_SYNTH_EMUX=m
# CONFIG_SND_XEN_FRONTEND is not set
CONFIG_AC97_BUS=m

#
# HID support
#
CONFIG_HID=y
CONFIG_HID_BATTERY_STRENGTH=y
CONFIG_HIDRAW=y
CONFIG_UHID=m
CONFIG_HID_GENERIC=y

#
# Special HID drivers
#
CONFIG_HID_A4TECH=y
# CONFIG_HID_ACCUTOUCH is not set
CONFIG_HID_ACRUX=m
# CONFIG_HID_ACRUX_FF is not set
CONFIG_HID_APPLE=y
CONFIG_HID_APPLEIR=m
# CONFIG_HID_ASUS is not set
CONFIG_HID_AUREAL=m
CONFIG_HID_BELKIN=y
# CONFIG_HID_BETOP_FF is not set
# CONFIG_HID_BIGBEN_FF is not set
CONFIG_HID_CHERRY=y
CONFIG_HID_CHICONY=y
# CONFIG_HID_CORSAIR is not set
# CONFIG_HID_COUGAR is not set
# CONFIG_HID_MACALLY is not set
CONFIG_HID_PRODIKEYS=m
# CONFIG_HID_CMEDIA is not set
# CONFIG_HID_CP2112 is not set
CONFIG_HID_CYPRESS=y
CONFIG_HID_DRAGONRISE=m
# CONFIG_DRAGONRISE_FF is not set
# CONFIG_HID_EMS_FF is not set
# CONFIG_HID_ELAN is not set
CONFIG_HID_ELECOM=m
# CONFIG_HID_ELO is not set
CONFIG_HID_EZKEY=y
# CONFIG_HID_GEMBIRD is not set
# CONFIG_HID_GFRM is not set
CONFIG_HID_HOLTEK=m
# CONFIG_HOLTEK_FF is not set
# CONFIG_HID_GT683R is not set
CONFIG_HID_KEYTOUCH=m
CONFIG_HID_KYE=m
CONFIG_HID_UCLOGIC=m
CONFIG_HID_WALTOP=m
# CONFIG_HID_VIEWSONIC is not set
CONFIG_HID_GYRATION=m
CONFIG_HID_ICADE=m
# CONFIG_HID_ITE is not set
# CONFIG_HID_JABRA is not set
CONFIG_HID_TWINHAN=m
CONFIG_HID_KENSINGTON=y
CONFIG_HID_LCPOWER=m
CONFIG_HID_LED=m
# CONFIG_HID_LENOVO is not set
CONFIG_HID_LOGITECH=y
CONFIG_HID_LOGITECH_DJ=m
CONFIG_HID_LOGITECH_HIDPP=m
# CONFIG_LOGITECH_FF is not set
# CONFIG_LOGIRUMBLEPAD2_FF is not set
# CONFIG_LOGIG940_FF is not set
# CONFIG_LOGIWHEELS_FF is not set
CONFIG_HID_MAGICMOUSE=y
# CONFIG_HID_MALTRON is not set
# CONFIG_HID_MAYFLASH is not set
# CONFIG_HID_REDRAGON is not set
CONFIG_HID_MICROSOFT=y
CONFIG_HID_MONTEREY=y
CONFIG_HID_MULTITOUCH=m
# CONFIG_HID_NTI is not set
CONFIG_HID_NTRIG=y
CONFIG_HID_ORTEK=m
CONFIG_HID_PANTHERLORD=m
# CONFIG_PANTHERLORD_FF is not set
# CONFIG_HID_PENMOUNT is not set
CONFIG_HID_PETALYNX=m
CONFIG_HID_PICOLCD=m
CONFIG_HID_PICOLCD_FB=y
CONFIG_HID_PICOLCD_BACKLIGHT=y
CONFIG_HID_PICOLCD_LCD=y
CONFIG_HID_PICOLCD_LEDS=y
CONFIG_HID_PICOLCD_CIR=y
CONFIG_HID_PLANTRONICS=y
CONFIG_HID_PRIMAX=m
# CONFIG_HID_RETRODE is not set
CONFIG_HID_ROCCAT=m
CONFIG_HID_SAITEK=m
CONFIG_HID_SAMSUNG=m
CONFIG_HID_SONY=m
# CONFIG_SONY_FF is not set
CONFIG_HID_SPEEDLINK=m
# CONFIG_HID_STEAM is not set
CONFIG_HID_STEELSERIES=m
CONFIG_HID_SUNPLUS=m
# CONFIG_HID_RMI is not set
CONFIG_HID_GREENASIA=m
# CONFIG_GREENASIA_FF is not set
CONFIG_HID_HYPERV_MOUSE=m
CONFIG_HID_SMARTJOYPLUS=m
# CONFIG_SMARTJOYPLUS_FF is not set
CONFIG_HID_TIVO=m
CONFIG_HID_TOPSEED=m
CONFIG_HID_THINGM=m
CONFIG_HID_THRUSTMASTER=m
# CONFIG_THRUSTMASTER_FF is not set
# CONFIG_HID_UDRAW_PS3 is not set
# CONFIG_HID_U2FZERO is not set
CONFIG_HID_WACOM=m
CONFIG_HID_WIIMOTE=m
# CONFIG_HID_XINMO is not set
CONFIG_HID_ZEROPLUS=m
# CONFIG_ZEROPLUS_FF is not set
CONFIG_HID_ZYDACRON=m
# CONFIG_HID_SENSOR_HUB is not set
# CONFIG_HID_ALPS is not set
# end of Special HID drivers

#
# USB HID support
#
CONFIG_USB_HID=y
CONFIG_HID_PID=y
CONFIG_USB_HIDDEV=y
# end of USB HID support

#
# I2C HID support
#
CONFIG_I2C_HID=m
# end of I2C HID support

#
# Intel ISH HID support
#
# CONFIG_INTEL_ISH_HID is not set
# end of Intel ISH HID support
# end of HID support

CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=y
CONFIG_USB_PCI=y
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y

#
# Miscellaneous USB options
#
CONFIG_USB_DEFAULT_PERSIST=y
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_OTG is not set
# CONFIG_USB_OTG_WHITELIST is not set
# CONFIG_USB_OTG_BLACKLIST_HUB is not set
# CONFIG_USB_LEDS_TRIGGER_USBPORT is not set
CONFIG_USB_AUTOSUSPEND_DELAY=2
CONFIG_USB_MON=y
CONFIG_USB_WUSB=m
CONFIG_USB_WUSB_CBAF=m
# CONFIG_USB_WUSB_CBAF_DEBUG is not set

#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_XHCI_HCD=y
# CONFIG_USB_XHCI_DBGCAP is not set
CONFIG_USB_XHCI_PCI=y
CONFIG_USB_XHCI_PLATFORM=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_USB_EHCI_PCI=y
# CONFIG_USB_EHCI_FSL is not set
# CONFIG_USB_EHCI_HCD_PLATFORM is not set
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_FOTG210_HCD is not set
# CONFIG_USB_MAX3421_HCD is not set
CONFIG_USB_OHCI_HCD=y
CONFIG_USB_OHCI_HCD_PCI=y
# CONFIG_USB_OHCI_HCD_PLATFORM is not set
CONFIG_USB_UHCI_HCD=y
# CONFIG_USB_U132_HCD is not set
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_WHCI_HCD is not set
CONFIG_USB_HWA_HCD=m
# CONFIG_USB_HCD_BCMA is not set
# CONFIG_USB_HCD_SSB is not set
# CONFIG_USB_HCD_TEST_MODE is not set

#
# USB Device Class drivers
#
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
CONFIG_USB_WDM=m
CONFIG_USB_TMC=m

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#

#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
CONFIG_USB_STORAGE_REALTEK=m
CONFIG_REALTEK_AUTOPM=y
CONFIG_USB_STORAGE_DATAFAB=m
CONFIG_USB_STORAGE_FREECOM=m
CONFIG_USB_STORAGE_ISD200=m
CONFIG_USB_STORAGE_USBAT=m
CONFIG_USB_STORAGE_SDDR09=m
CONFIG_USB_STORAGE_SDDR55=m
CONFIG_USB_STORAGE_JUMPSHOT=m
CONFIG_USB_STORAGE_ALAUDA=m
CONFIG_USB_STORAGE_ONETOUCH=m
CONFIG_USB_STORAGE_KARMA=m
CONFIG_USB_STORAGE_CYPRESS_ATACB=m
CONFIG_USB_STORAGE_ENE_UB6250=m
CONFIG_USB_UAS=m

#
# USB Imaging devices
#
CONFIG_USB_MDC800=m
CONFIG_USB_MICROTEK=m
# CONFIG_USBIP_CORE is not set
# CONFIG_USB_MUSB_HDRC is not set
CONFIG_USB_DWC3=y
# CONFIG_USB_DWC3_HOST is not set
CONFIG_USB_DWC3_GADGET=y
# CONFIG_USB_DWC3_DUAL_ROLE is not set

#
# Platform Glue Driver Support
#
CONFIG_USB_DWC3_PCI=y
CONFIG_USB_DWC3_HAPS=y
# CONFIG_USB_DWC2 is not set
# CONFIG_USB_CHIPIDEA is not set
# CONFIG_USB_ISP1760 is not set

#
# USB port drivers
#
CONFIG_USB_USS720=m
CONFIG_USB_SERIAL=y
CONFIG_USB_SERIAL_CONSOLE=y
CONFIG_USB_SERIAL_GENERIC=y
# CONFIG_USB_SERIAL_SIMPLE is not set
CONFIG_USB_SERIAL_AIRCABLE=m
CONFIG_USB_SERIAL_ARK3116=m
CONFIG_USB_SERIAL_BELKIN=m
CONFIG_USB_SERIAL_CH341=m
CONFIG_USB_SERIAL_WHITEHEAT=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
CONFIG_USB_SERIAL_CP210X=m
CONFIG_USB_SERIAL_CYPRESS_M8=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
# CONFIG_USB_SERIAL_F81232 is not set
# CONFIG_USB_SERIAL_F8153X is not set
CONFIG_USB_SERIAL_GARMIN=m
CONFIG_USB_SERIAL_IPW=m
CONFIG_USB_SERIAL_IUU=m
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
# CONFIG_USB_SERIAL_METRO is not set
CONFIG_USB_SERIAL_MOS7720=m
CONFIG_USB_SERIAL_MOS7715_PARPORT=y
CONFIG_USB_SERIAL_MOS7840=m
# CONFIG_USB_SERIAL_MXUPORT is not set
CONFIG_USB_SERIAL_NAVMAN=m
CONFIG_USB_SERIAL_PL2303=m
CONFIG_USB_SERIAL_OTI6858=m
CONFIG_USB_SERIAL_QCAUX=m
CONFIG_USB_SERIAL_QUALCOMM=m
CONFIG_USB_SERIAL_SPCP8X5=m
CONFIG_USB_SERIAL_SAFE=m
CONFIG_USB_SERIAL_SAFE_PADDED=y
CONFIG_USB_SERIAL_SIERRAWIRELESS=m
CONFIG_USB_SERIAL_SYMBOL=m
# CONFIG_USB_SERIAL_TI is not set
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_WWAN=m
CONFIG_USB_SERIAL_OPTION=m
CONFIG_USB_SERIAL_OMNINET=m
CONFIG_USB_SERIAL_OPTICON=m
CONFIG_USB_SERIAL_XSENS_MT=m
# CONFIG_USB_SERIAL_WISHBONE is not set
CONFIG_USB_SERIAL_SSU100=m
CONFIG_USB_SERIAL_QT2=m
# CONFIG_USB_SERIAL_UPD78F0730 is not set
CONFIG_USB_SERIAL_DEBUG=m

#
# USB Miscellaneous drivers
#
CONFIG_USB_EMI62=m
CONFIG_USB_EMI26=m
CONFIG_USB_ADUTUX=m
CONFIG_USB_SEVSEG=m
# CONFIG_USB_RIO500 is not set
CONFIG_USB_LEGOTOWER=m
CONFIG_USB_LCD=m
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
CONFIG_USB_IDMOUSE=m
CONFIG_USB_FTDI_ELAN=m
CONFIG_USB_APPLEDISPLAY=m
CONFIG_USB_SISUSBVGA=m
CONFIG_USB_SISUSBVGA_CON=y
CONFIG_USB_LD=m
# CONFIG_USB_TRANCEVIBRATOR is not set
CONFIG_USB_IOWARRIOR=m
# CONFIG_USB_TEST is not set
# CONFIG_USB_EHSET_TEST_FIXTURE is not set
CONFIG_USB_ISIGHTFW=m
# CONFIG_USB_YUREX is not set
CONFIG_USB_EZUSB_FX2=m
# CONFIG_USB_HUB_USB251XB is not set
CONFIG_USB_HSIC_USB3503=m
# CONFIG_USB_HSIC_USB4604 is not set
# CONFIG_USB_LINK_LAYER_TEST is not set
# CONFIG_USB_CHAOSKEY is not set
CONFIG_USB_ATM=m
CONFIG_USB_SPEEDTOUCH=m
CONFIG_USB_CXACRU=m
CONFIG_USB_UEAGLEATM=m
CONFIG_USB_XUSBATM=m

#
# USB Physical Layer drivers
#
CONFIG_USB_PHY=y
CONFIG_NOP_USB_XCEIV=y
# CONFIG_USB_GPIO_VBUS is not set
# CONFIG_USB_ISP1301 is not set
# end of USB Physical Layer drivers

CONFIG_USB_GADGET=y
# CONFIG_USB_GADGET_DEBUG is not set
# CONFIG_USB_GADGET_DEBUG_FILES is not set
# CONFIG_USB_GADGET_DEBUG_FS is not set
CONFIG_USB_GADGET_VBUS_DRAW=2
CONFIG_USB_GADGET_STORAGE_NUM_BUFFERS=2

#
# USB Peripheral Controller
#
# CONFIG_USB_FOTG210_UDC is not set
# CONFIG_USB_GR_UDC is not set
# CONFIG_USB_R8A66597 is not set
# CONFIG_USB_PXA27X is not set
# CONFIG_USB_MV_UDC is not set
# CONFIG_USB_MV_U3D is not set
# CONFIG_USB_M66592 is not set
# CONFIG_USB_BDC_UDC is not set
# CONFIG_USB_AMD5536UDC is not set
# CONFIG_USB_NET2272 is not set
# CONFIG_USB_NET2280 is not set
# CONFIG_USB_GOKU is not set
# CONFIG_USB_EG20T is not set
# CONFIG_USB_DUMMY_HCD is not set
# end of USB Peripheral Controller

CONFIG_USB_LIBCOMPOSITE=m
CONFIG_USB_F_MASS_STORAGE=m
# CONFIG_USB_CONFIGFS is not set
# CONFIG_USB_ZERO is not set
# CONFIG_USB_AUDIO is not set
# CONFIG_USB_ETH is not set
# CONFIG_USB_G_NCM is not set
# CONFIG_USB_GADGETFS is not set
# CONFIG_USB_FUNCTIONFS is not set
CONFIG_USB_MASS_STORAGE=m
# CONFIG_USB_GADGET_TARGET is not set
# CONFIG_USB_G_SERIAL is not set
# CONFIG_USB_MIDI_GADGET is not set
# CONFIG_USB_G_PRINTER is not set
# CONFIG_USB_CDC_COMPOSITE is not set
# CONFIG_USB_G_ACM_MS is not set
# CONFIG_USB_G_MULTI is not set
# CONFIG_USB_G_HID is not set
# CONFIG_USB_G_DBGP is not set
# CONFIG_USB_G_WEBCAM is not set
# CONFIG_TYPEC is not set
# CONFIG_USB_ROLE_SWITCH is not set
# CONFIG_USB_LED_TRIG is not set
# CONFIG_USB_ULPI_BUS is not set
CONFIG_UWB=m
CONFIG_UWB_HWA=m
CONFIG_UWB_WHCI=m
CONFIG_UWB_I1480U=m
CONFIG_MMC=m
CONFIG_MMC_BLOCK=m
CONFIG_MMC_BLOCK_MINORS=8
CONFIG_SDIO_UART=m
# CONFIG_MMC_TEST is not set

#
# MMC/SD/SDIO Host Controller Drivers
#
# CONFIG_MMC_DEBUG is not set
CONFIG_MMC_SDHCI=m
CONFIG_MMC_SDHCI_PCI=m
CONFIG_MMC_RICOH_MMC=y
CONFIG_MMC_SDHCI_ACPI=m
CONFIG_MMC_SDHCI_PLTFM=m
# CONFIG_MMC_SDHCI_F_SDH30 is not set
# CONFIG_MMC_WBSD is not set
CONFIG_MMC_TIFM_SD=m
# CONFIG_MMC_SPI is not set
CONFIG_MMC_CB710=m
CONFIG_MMC_VIA_SDMMC=m
CONFIG_MMC_VUB300=m
CONFIG_MMC_USHC=m
# CONFIG_MMC_USDHI6ROL0 is not set
CONFIG_MMC_CQHCI=m
# CONFIG_MMC_TOSHIBA_PCI is not set
# CONFIG_MMC_MTK is not set
# CONFIG_MMC_SDHCI_XENON is not set
CONFIG_MEMSTICK=m
# CONFIG_MEMSTICK_DEBUG is not set

#
# MemoryStick drivers
#
# CONFIG_MEMSTICK_UNSAFE_RESUME is not set
CONFIG_MSPRO_BLOCK=m
# CONFIG_MS_BLOCK is not set

#
# MemoryStick Host Controller Drivers
#
CONFIG_MEMSTICK_TIFM_MS=m
CONFIG_MEMSTICK_JMICRON_38X=m
CONFIG_MEMSTICK_R592=m
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y
# CONFIG_LEDS_CLASS_FLASH is not set
# CONFIG_LEDS_BRIGHTNESS_HW_CHANGED is not set

#
# LED drivers
#
# CONFIG_LEDS_APU is not set
CONFIG_LEDS_LM3530=m
# CONFIG_LEDS_LM3532 is not set
# CONFIG_LEDS_LM3642 is not set
# CONFIG_LEDS_PCA9532 is not set
# CONFIG_LEDS_GPIO is not set
CONFIG_LEDS_LP3944=m
# CONFIG_LEDS_LP3952 is not set
CONFIG_LEDS_LP55XX_COMMON=m
CONFIG_LEDS_LP5521=m
CONFIG_LEDS_LP5523=m
CONFIG_LEDS_LP5562=m
# CONFIG_LEDS_LP8501 is not set
CONFIG_LEDS_CLEVO_MAIL=m
# CONFIG_LEDS_PCA955X is not set
# CONFIG_LEDS_PCA963X is not set
# CONFIG_LEDS_DAC124S085 is not set
# CONFIG_LEDS_PWM is not set
# CONFIG_LEDS_BD2802 is not set
CONFIG_LEDS_INTEL_SS4200=m
# CONFIG_LEDS_TCA6507 is not set
# CONFIG_LEDS_TLC591XX is not set
# CONFIG_LEDS_LM355x is not set

#
# LED driver for blink(1) USB RGB LED is under Special HID drivers (HID_THINGM)
#
CONFIG_LEDS_BLINKM=m
# CONFIG_LEDS_MLXCPLD is not set
# CONFIG_LEDS_MLXREG is not set
# CONFIG_LEDS_USER is not set
# CONFIG_LEDS_NIC78BX is not set

#
# LED Triggers
#
CONFIG_LEDS_TRIGGERS=y
CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_ONESHOT=m
# CONFIG_LEDS_TRIGGER_DISK is not set
# CONFIG_LEDS_TRIGGER_MTD is not set
CONFIG_LEDS_TRIGGER_HEARTBEAT=m
CONFIG_LEDS_TRIGGER_BACKLIGHT=m
# CONFIG_LEDS_TRIGGER_CPU is not set
# CONFIG_LEDS_TRIGGER_ACTIVITY is not set
# CONFIG_LEDS_TRIGGER_GPIO is not set
CONFIG_LEDS_TRIGGER_DEFAULT_ON=m

#
# iptables trigger is under Netfilter config (LED target)
#
CONFIG_LEDS_TRIGGER_TRANSIENT=m
CONFIG_LEDS_TRIGGER_CAMERA=m
# CONFIG_LEDS_TRIGGER_PANIC is not set
# CONFIG_LEDS_TRIGGER_NETDEV is not set
# CONFIG_LEDS_TRIGGER_PATTERN is not set
CONFIG_LEDS_TRIGGER_AUDIO=m
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
CONFIG_EDAC_ATOMIC_SCRUB=y
CONFIG_EDAC_SUPPORT=y
CONFIG_EDAC=y
CONFIG_EDAC_LEGACY_SYSFS=y
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_DECODE_MCE=m
# CONFIG_EDAC_GHES is not set
CONFIG_EDAC_AMD64=m
# CONFIG_EDAC_AMD64_ERROR_INJECTION is not set
CONFIG_EDAC_E752X=m
CONFIG_EDAC_I82975X=m
CONFIG_EDAC_I3000=m
CONFIG_EDAC_I3200=m
# CONFIG_EDAC_IE31200 is not set
CONFIG_EDAC_X38=m
CONFIG_EDAC_I5400=m
CONFIG_EDAC_I7CORE=m
CONFIG_EDAC_I5000=m
CONFIG_EDAC_I5100=m
CONFIG_EDAC_I7300=m
CONFIG_EDAC_SBRIDGE=m
CONFIG_EDAC_SKX=m
# CONFIG_EDAC_I10NM is not set
# CONFIG_EDAC_PND2 is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_MC146818_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
# CONFIG_RTC_SYSTOHC is not set
# CONFIG_RTC_DEBUG is not set
CONFIG_RTC_NVMEM=y

#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set

#
# I2C RTC drivers
#
# CONFIG_RTC_DRV_ABB5ZES3 is not set
# CONFIG_RTC_DRV_ABEOZ9 is not set
# CONFIG_RTC_DRV_ABX80X is not set
CONFIG_RTC_DRV_DS1307=m
# CONFIG_RTC_DRV_DS1307_CENTURY is not set
CONFIG_RTC_DRV_DS1374=m
# CONFIG_RTC_DRV_DS1374_WDT is not set
CONFIG_RTC_DRV_DS1672=m
CONFIG_RTC_DRV_MAX6900=m
CONFIG_RTC_DRV_RS5C372=m
CONFIG_RTC_DRV_ISL1208=m
CONFIG_RTC_DRV_ISL12022=m
CONFIG_RTC_DRV_X1205=m
CONFIG_RTC_DRV_PCF8523=m
# CONFIG_RTC_DRV_PCF85063 is not set
# CONFIG_RTC_DRV_PCF85363 is not set
CONFIG_RTC_DRV_PCF8563=m
CONFIG_RTC_DRV_PCF8583=m
CONFIG_RTC_DRV_M41T80=m
CONFIG_RTC_DRV_M41T80_WDT=y
CONFIG_RTC_DRV_BQ32K=m
# CONFIG_RTC_DRV_S35390A is not set
CONFIG_RTC_DRV_FM3130=m
# CONFIG_RTC_DRV_RX8010 is not set
CONFIG_RTC_DRV_RX8581=m
CONFIG_RTC_DRV_RX8025=m
CONFIG_RTC_DRV_EM3027=m
# CONFIG_RTC_DRV_RV3028 is not set
# CONFIG_RTC_DRV_RV8803 is not set
# CONFIG_RTC_DRV_SD3078 is not set

#
# SPI RTC drivers
#
# CONFIG_RTC_DRV_M41T93 is not set
# CONFIG_RTC_DRV_M41T94 is not set
# CONFIG_RTC_DRV_DS1302 is not set
# CONFIG_RTC_DRV_DS1305 is not set
# CONFIG_RTC_DRV_DS1343 is not set
# CONFIG_RTC_DRV_DS1347 is not set
# CONFIG_RTC_DRV_DS1390 is not set
# CONFIG_RTC_DRV_MAX6916 is not set
# CONFIG_RTC_DRV_R9701 is not set
# CONFIG_RTC_DRV_RX4581 is not set
# CONFIG_RTC_DRV_RX6110 is not set
# CONFIG_RTC_DRV_RS5C348 is not set
# CONFIG_RTC_DRV_MAX6902 is not set
# CONFIG_RTC_DRV_PCF2123 is not set
# CONFIG_RTC_DRV_MCP795 is not set
CONFIG_RTC_I2C_AND_SPI=y

#
# SPI and I2C RTC drivers
#
CONFIG_RTC_DRV_DS3232=m
CONFIG_RTC_DRV_DS3232_HWMON=y
# CONFIG_RTC_DRV_PCF2127 is not set
CONFIG_RTC_DRV_RV3029C2=m
CONFIG_RTC_DRV_RV3029_HWMON=y

#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=y
CONFIG_RTC_DRV_DS1286=m
CONFIG_RTC_DRV_DS1511=m
CONFIG_RTC_DRV_DS1553=m
# CONFIG_RTC_DRV_DS1685_FAMILY is not set
CONFIG_RTC_DRV_DS1742=m
CONFIG_RTC_DRV_DS2404=m
CONFIG_RTC_DRV_STK17TA8=m
# CONFIG_RTC_DRV_M48T86 is not set
CONFIG_RTC_DRV_M48T35=m
CONFIG_RTC_DRV_M48T59=m
CONFIG_RTC_DRV_MSM6242=m
CONFIG_RTC_DRV_BQ4802=m
CONFIG_RTC_DRV_RP5C01=m
CONFIG_RTC_DRV_V3020=m

#
# on-CPU RTC drivers
#
# CONFIG_RTC_DRV_FTRTC010 is not set

#
# HID Sensor RTC drivers
#
CONFIG_DMADEVICES=y
# CONFIG_DMADEVICES_DEBUG is not set

#
# DMA Devices
#
CONFIG_DMA_ENGINE=y
CONFIG_DMA_VIRTUAL_CHANNELS=y
CONFIG_DMA_ACPI=y
# CONFIG_ALTERA_MSGDMA is not set
# CONFIG_INTEL_IDMA64 is not set
# CONFIG_INTEL_IOATDMA is not set
# CONFIG_QCOM_HIDMA_MGMT is not set
# CONFIG_QCOM_HIDMA is not set
CONFIG_DW_DMAC_CORE=y
CONFIG_DW_DMAC=m
CONFIG_DW_DMAC_PCI=y
CONFIG_HSU_DMA=y

#
# DMA Clients
#
CONFIG_ASYNC_TX_DMA=y
CONFIG_DMATEST=m
CONFIG_DMA_ENGINE_RAID=y

#
# DMABUF options
#
CONFIG_SYNC_FILE=y
CONFIG_SW_SYNC=y
# CONFIG_UDMABUF is not set
# end of DMABUF options

CONFIG_AUXDISPLAY=y
# CONFIG_HD44780 is not set
CONFIG_KS0108=m
CONFIG_KS0108_PORT=0x378
CONFIG_KS0108_DELAY=2
CONFIG_CFAG12864B=m
CONFIG_CFAG12864B_RATE=20
# CONFIG_IMG_ASCII_LCD is not set
# CONFIG_PARPORT_PANEL is not set
# CONFIG_CHARLCD_BL_OFF is not set
# CONFIG_CHARLCD_BL_ON is not set
CONFIG_CHARLCD_BL_FLASH=y
# CONFIG_PANEL is not set
CONFIG_UIO=m
CONFIG_UIO_CIF=m
CONFIG_UIO_PDRV_GENIRQ=m
# CONFIG_UIO_DMEM_GENIRQ is not set
CONFIG_UIO_AEC=m
CONFIG_UIO_SERCOS3=m
CONFIG_UIO_PCI_GENERIC=m
# CONFIG_UIO_NETX is not set
# CONFIG_UIO_PRUSS is not set
# CONFIG_UIO_MF624 is not set
# CONFIG_UIO_HV_GENERIC is not set
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO_VIRQFD=m
CONFIG_VFIO=m
# CONFIG_VFIO_NOIOMMU is not set
CONFIG_VFIO_PCI=m
# CONFIG_VFIO_PCI_VGA is not set
CONFIG_VFIO_PCI_MMAP=y
CONFIG_VFIO_PCI_INTX=y
CONFIG_VFIO_PCI_IGD=y
# CONFIG_VFIO_MDEV is not set
CONFIG_IRQ_BYPASS_MANAGER=m
# CONFIG_VIRT_DRIVERS is not set
CONFIG_VIRTIO=y
CONFIG_VIRTIO_MENU=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_VIRTIO_BALLOON=y
# CONFIG_VIRTIO_INPUT is not set
# CONFIG_VIRTIO_MMIO is not set

#
# Microsoft Hyper-V guest support
#
CONFIG_HYPERV=m
CONFIG_HYPERV_TSCPAGE=y
CONFIG_HYPERV_UTILS=m
CONFIG_HYPERV_BALLOON=m
# end of Microsoft Hyper-V guest support

#
# Xen driver support
#
CONFIG_XEN_BALLOON=y
# CONFIG_XEN_SELFBALLOONING is not set
# CONFIG_XEN_BALLOON_MEMORY_HOTPLUG is not set
CONFIG_XEN_SCRUB_PAGES_DEFAULT=y
CONFIG_XEN_DEV_EVTCHN=m
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=m
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
# CONFIG_XEN_GNTDEV is not set
# CONFIG_XEN_GRANT_DEV_ALLOC is not set
# CONFIG_XEN_GRANT_DMA_ALLOC is not set
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_TMEM=m
CONFIG_XEN_PCIDEV_BACKEND=m
# CONFIG_XEN_PVCALLS_FRONTEND is not set
# CONFIG_XEN_PVCALLS_BACKEND is not set
# CONFIG_XEN_SCSI_BACKEND is not set
CONFIG_XEN_PRIVCMD=m
CONFIG_XEN_ACPI_PROCESSOR=m
# CONFIG_XEN_MCE_LOG is not set
CONFIG_XEN_HAVE_PVMMU=y
CONFIG_XEN_EFI=y
CONFIG_XEN_AUTO_XLATE=y
CONFIG_XEN_ACPI=y
CONFIG_XEN_SYMS=y
CONFIG_XEN_HAVE_VPMU=y
# end of Xen driver support

CONFIG_STAGING=y
# CONFIG_PRISM2_USB is not set
# CONFIG_COMEDI is not set
# CONFIG_RTL8192U is not set
CONFIG_RTLLIB=m
CONFIG_RTLLIB_CRYPTO_CCMP=m
CONFIG_RTLLIB_CRYPTO_TKIP=m
CONFIG_RTLLIB_CRYPTO_WEP=m
CONFIG_RTL8192E=m
# CONFIG_RTL8723BS is not set
CONFIG_R8712U=m
# CONFIG_R8188EU is not set
# CONFIG_RTS5208 is not set
# CONFIG_VT6655 is not set
# CONFIG_VT6656 is not set
# CONFIG_FB_SM750 is not set

#
# Speakup console speech
#
# CONFIG_SPEAKUP is not set
# end of Speakup console speech

# CONFIG_STAGING_MEDIA is not set

#
# Android
#
# CONFIG_ASHMEM is not set
# CONFIG_ANDROID_VSOC is not set
CONFIG_ION=y
CONFIG_ION_SYSTEM_HEAP=y
# CONFIG_ION_CARVEOUT_HEAP is not set
# CONFIG_ION_CHUNK_HEAP is not set
# CONFIG_ION_CMA_HEAP is not set
# end of Android

# CONFIG_LTE_GDM724X is not set
CONFIG_FIREWIRE_SERIAL=m
CONFIG_FWTTY_MAX_TOTAL_PORTS=64
CONFIG_FWTTY_MAX_CARD_PORTS=32
# CONFIG_GS_FPGABOOT is not set
# CONFIG_UNISYSSPAR is not set
# CONFIG_FB_TFT is not set
# CONFIG_WILC1000_SDIO is not set
# CONFIG_WILC1000_SPI is not set
# CONFIG_MOST is not set
# CONFIG_KS7010 is not set
# CONFIG_GREYBUS is not set
# CONFIG_PI433 is not set

#
# Gasket devices
#
# CONFIG_STAGING_GASKET_FRAMEWORK is not set
# end of Gasket devices

# CONFIG_EROFS_FS is not set
# CONFIG_FIELDBUS_DEV is not set
# CONFIG_KPC2000 is not set
CONFIG_X86_PLATFORM_DEVICES=y
CONFIG_ACER_WMI=m
# CONFIG_ACER_WIRELESS is not set
CONFIG_ACERHDF=m
# CONFIG_ALIENWARE_WMI is not set
CONFIG_ASUS_LAPTOP=m
CONFIG_DCDBAS=m
# CONFIG_DELL_SMBIOS is not set
CONFIG_DELL_WMI_AIO=m
# CONFIG_DELL_WMI_LED is not set
# CONFIG_DELL_SMO8800 is not set
# CONFIG_DELL_RBTN is not set
CONFIG_DELL_RBU=m
CONFIG_FUJITSU_LAPTOP=m
CONFIG_FUJITSU_TABLET=m
CONFIG_AMILO_RFKILL=m
# CONFIG_GPD_POCKET_FAN is not set
CONFIG_HP_ACCEL=m
# CONFIG_HP_WIRELESS is not set
CONFIG_HP_WMI=m
# CONFIG_LG_LAPTOP is not set
CONFIG_MSI_LAPTOP=m
CONFIG_PANASONIC_LAPTOP=m
CONFIG_COMPAL_LAPTOP=m
CONFIG_SONY_LAPTOP=m
CONFIG_SONYPI_COMPAT=y
CONFIG_IDEAPAD_LAPTOP=m
# CONFIG_SURFACE3_WMI is not set
CONFIG_THINKPAD_ACPI=m
CONFIG_THINKPAD_ACPI_ALSA_SUPPORT=y
# CONFIG_THINKPAD_ACPI_DEBUGFACILITIES is not set
# CONFIG_THINKPAD_ACPI_DEBUG is not set
# CONFIG_THINKPAD_ACPI_UNSAFE_LEDS is not set
CONFIG_THINKPAD_ACPI_VIDEO=y
CONFIG_THINKPAD_ACPI_HOTKEY_POLL=y
CONFIG_SENSORS_HDAPS=m
# CONFIG_INTEL_MENLOW is not set
CONFIG_EEEPC_LAPTOP=m
CONFIG_ASUS_WMI=m
CONFIG_ASUS_NB_WMI=m
CONFIG_EEEPC_WMI=m
# CONFIG_ASUS_WIRELESS is not set
CONFIG_ACPI_WMI=m
CONFIG_WMI_BMOF=m
# CONFIG_INTEL_WMI_THUNDERBOLT is not set
CONFIG_MSI_WMI=m
# CONFIG_PEAQ_WMI is not set
CONFIG_TOPSTAR_LAPTOP=m
CONFIG_TOSHIBA_BT_RFKILL=m
# CONFIG_TOSHIBA_HAPS is not set
# CONFIG_TOSHIBA_WMI is not set
# CONFIG_INTEL_INT0002_VGPIO is not set
# CONFIG_INTEL_HID_EVENT is not set
# CONFIG_INTEL_VBTN is not set
CONFIG_INTEL_IPS=m
# CONFIG_INTEL_PMC_CORE is not set
# CONFIG_IBM_RTL is not set
CONFIG_SAMSUNG_LAPTOP=m
CONFIG_MXM_WMI=m
CONFIG_INTEL_OAKTRAIL=m
CONFIG_APPLE_GMUX=m
# CONFIG_INTEL_RST is not set
# CONFIG_INTEL_SMARTCONNECT is not set
# CONFIG_INTEL_PMC_IPC is not set
# CONFIG_SURFACE_PRO3_BUTTON is not set
# CONFIG_INTEL_PUNIT_IPC is not set
# CONFIG_MLX_PLATFORM is not set
# CONFIG_INTEL_TURBO_MAX_3 is not set
# CONFIG_I2C_MULTI_INSTANTIATE is not set
# CONFIG_INTEL_ATOMISP2_PM is not set
# CONFIG_HUAWEI_WMI is not set
# CONFIG_PCENGINES_APU2 is not set
CONFIG_PMC_ATOM=y
# CONFIG_CHROME_PLATFORMS is not set
# CONFIG_MELLANOX_PLATFORM is not set
CONFIG_CLKDEV_LOOKUP=y
CONFIG_HAVE_CLK_PREPARE=y
CONFIG_COMMON_CLK=y

#
# Common Clock Framework
#
# CONFIG_COMMON_CLK_MAX9485 is not set
# CONFIG_COMMON_CLK_SI5351 is not set
# CONFIG_COMMON_CLK_SI544 is not set
# CONFIG_COMMON_CLK_CDCE706 is not set
# CONFIG_COMMON_CLK_CS2000_CP is not set
# CONFIG_COMMON_CLK_PWM is not set
# end of Common Clock Framework

# CONFIG_HWSPINLOCK is not set

#
# Clock Source drivers
#
CONFIG_CLKEVT_I8253=y
CONFIG_I8253_LOCK=y
CONFIG_CLKBLD_I8253=y
# end of Clock Source drivers

CONFIG_MAILBOX=y
CONFIG_PCC=y
# CONFIG_ALTERA_MBOX is not set
CONFIG_IOMMU_IOVA=y
CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y

#
# Generic IOMMU Pagetable Support
#
# end of Generic IOMMU Pagetable Support

# CONFIG_IOMMU_DEBUGFS is not set
# CONFIG_IOMMU_DEFAULT_PASSTHROUGH is not set
CONFIG_AMD_IOMMU=y
CONFIG_AMD_IOMMU_V2=m
CONFIG_DMAR_TABLE=y
CONFIG_INTEL_IOMMU=y
# CONFIG_INTEL_IOMMU_SVM is not set
# CONFIG_INTEL_IOMMU_DEFAULT_ON is not set
CONFIG_INTEL_IOMMU_FLOPPY_WA=y
CONFIG_IRQ_REMAP=y
CONFIG_HYPERV_IOMMU=y

#
# Remoteproc drivers
#
# CONFIG_REMOTEPROC is not set
# end of Remoteproc drivers

#
# Rpmsg drivers
#
# CONFIG_RPMSG_QCOM_GLINK_RPM is not set
# CONFIG_RPMSG_VIRTIO is not set
# end of Rpmsg drivers

# CONFIG_SOUNDWIRE is not set

#
# SOC (System On Chip) specific Drivers
#

#
# Amlogic SoC drivers
#
# end of Amlogic SoC drivers

#
# Aspeed SoC drivers
#
# end of Aspeed SoC drivers

#
# Broadcom SoC drivers
#
# end of Broadcom SoC drivers

#
# NXP/Freescale QorIQ SoC drivers
#
# end of NXP/Freescale QorIQ SoC drivers

#
# i.MX SoC drivers
#
# end of i.MX SoC drivers

#
# IXP4xx SoC drivers
#
# CONFIG_IXP4XX_QMGR is not set
# CONFIG_IXP4XX_NPE is not set
# end of IXP4xx SoC drivers

#
# Qualcomm SoC drivers
#
# end of Qualcomm SoC drivers

# CONFIG_SOC_TI is not set

#
# Xilinx SoC drivers
#
# CONFIG_XILINX_VCU is not set
# end of Xilinx SoC drivers
# end of SOC (System On Chip) specific Drivers

CONFIG_PM_DEVFREQ=y

#
# DEVFREQ Governors
#
CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=m
# CONFIG_DEVFREQ_GOV_PERFORMANCE is not set
# CONFIG_DEVFREQ_GOV_POWERSAVE is not set
# CONFIG_DEVFREQ_GOV_USERSPACE is not set
# CONFIG_DEVFREQ_GOV_PASSIVE is not set

#
# DEVFREQ Drivers
#
# CONFIG_PM_DEVFREQ_EVENT is not set
CONFIG_EXTCON=y

#
# Extcon Device Drivers
#
# CONFIG_EXTCON_GPIO is not set
# CONFIG_EXTCON_INTEL_INT3496 is not set
# CONFIG_EXTCON_MAX3355 is not set
# CONFIG_EXTCON_PTN5150 is not set
# CONFIG_EXTCON_RT8973A is not set
# CONFIG_EXTCON_SM5502 is not set
# CONFIG_EXTCON_USB_GPIO is not set
# CONFIG_MEMORY is not set
# CONFIG_IIO is not set
CONFIG_NTB=m
# CONFIG_NTB_AMD is not set
# CONFIG_NTB_IDT is not set
# CONFIG_NTB_INTEL is not set
# CONFIG_NTB_SWITCHTEC is not set
# CONFIG_NTB_PINGPONG is not set
# CONFIG_NTB_TOOL is not set
# CONFIG_NTB_PERF is not set
# CONFIG_NTB_TRANSPORT is not set
# CONFIG_VME_BUS is not set
CONFIG_PWM=y
CONFIG_PWM_SYSFS=y
CONFIG_PWM_LPSS=m
CONFIG_PWM_LPSS_PCI=m
CONFIG_PWM_LPSS_PLATFORM=m
# CONFIG_PWM_PCA9685 is not set

#
# IRQ chip support
#
CONFIG_ARM_GIC_MAX_NR=1
# end of IRQ chip support

# CONFIG_IPACK_BUS is not set
# CONFIG_RESET_CONTROLLER is not set
# CONFIG_FMC is not set

#
# PHY Subsystem
#
CONFIG_GENERIC_PHY=y
# CONFIG_BCM_KONA_USB2_PHY is not set
# CONFIG_PHY_PXA_28NM_HSIC is not set
# CONFIG_PHY_PXA_28NM_USB2 is not set
# end of PHY Subsystem

CONFIG_POWERCAP=y
CONFIG_INTEL_RAPL=m
# CONFIG_IDLE_INJECT is not set
# CONFIG_MCB is not set

#
# Performance monitor support
#
# end of Performance monitor support

CONFIG_RAS=y
# CONFIG_RAS_CEC is not set
# CONFIG_THUNDERBOLT is not set

#
# Android
#
CONFIG_ANDROID=y
# CONFIG_ANDROID_BINDER_IPC is not set
# end of Android

CONFIG_LIBNVDIMM=m
CONFIG_BLK_DEV_PMEM=m
CONFIG_ND_BLK=m
CONFIG_ND_CLAIM=y
CONFIG_ND_BTT=m
CONFIG_BTT=y
CONFIG_ND_PFN=m
CONFIG_NVDIMM_PFN=y
CONFIG_NVDIMM_DAX=y
CONFIG_NVDIMM_KEYS=y
CONFIG_DAX_DRIVER=y
CONFIG_DAX=y
CONFIG_DEV_DAX=m
CONFIG_DEV_DAX_PMEM=m
CONFIG_DEV_DAX_KMEM=m
CONFIG_DEV_DAX_PMEM_COMPAT=m
CONFIG_NVMEM=y
CONFIG_NVMEM_SYSFS=y

#
# HW tracing support
#
CONFIG_STM=m
# CONFIG_STM_PROTO_BASIC is not set
# CONFIG_STM_PROTO_SYS_T is not set
CONFIG_STM_DUMMY=m
CONFIG_STM_SOURCE_CONSOLE=m
CONFIG_STM_SOURCE_HEARTBEAT=m
CONFIG_STM_SOURCE_FTRACE=m
CONFIG_INTEL_TH=m
CONFIG_INTEL_TH_PCI=m
# CONFIG_INTEL_TH_ACPI is not set
CONFIG_INTEL_TH_GTH=m
CONFIG_INTEL_TH_STH=m
CONFIG_INTEL_TH_MSU=m
CONFIG_INTEL_TH_PTI=m
# CONFIG_INTEL_TH_DEBUG is not set
# end of HW tracing support

# CONFIG_FPGA is not set
CONFIG_PM_OPP=y
# CONFIG_UNISYS_VISORBUS is not set
# CONFIG_SIOX is not set
# CONFIG_SLIMBUS is not set
# CONFIG_INTERCONNECT is not set
# CONFIG_COUNTER is not set
# end of Device Drivers

#
# File systems
#
CONFIG_DCACHE_WORD_ACCESS=y
CONFIG_VALIDATE_FS_PARSER=y
CONFIG_FS_IOMAP=y
# CONFIG_EXT2_FS is not set
# CONFIG_EXT3_FS is not set
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT2=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
# CONFIG_EXT4_DEBUG is not set
CONFIG_JBD2=y
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
CONFIG_XFS_FS=m
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_RT=y
CONFIG_XFS_ONLINE_SCRUB=y
CONFIG_XFS_ONLINE_REPAIR=y
CONFIG_XFS_DEBUG=y
CONFIG_XFS_ASSERT_FATAL=y
CONFIG_GFS2_FS=m
CONFIG_GFS2_FS_LOCKING_DLM=y
CONFIG_OCFS2_FS=m
CONFIG_OCFS2_FS_O2CB=m
CONFIG_OCFS2_FS_USERSPACE_CLUSTER=m
CONFIG_OCFS2_FS_STATS=y
CONFIG_OCFS2_DEBUG_MASKLOG=y
# CONFIG_OCFS2_DEBUG_FS is not set
CONFIG_BTRFS_FS=m
CONFIG_BTRFS_FS_POSIX_ACL=y
# CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set
# CONFIG_BTRFS_FS_RUN_SANITY_TESTS is not set
# CONFIG_BTRFS_DEBUG is not set
# CONFIG_BTRFS_ASSERT is not set
# CONFIG_BTRFS_FS_REF_VERIFY is not set
# CONFIG_NILFS2_FS is not set
CONFIG_F2FS_FS=m
CONFIG_F2FS_STAT_FS=y
CONFIG_F2FS_FS_XATTR=y
CONFIG_F2FS_FS_POSIX_ACL=y
# CONFIG_F2FS_FS_SECURITY is not set
# CONFIG_F2FS_CHECK_FS is not set
# CONFIG_F2FS_IO_TRACE is not set
# CONFIG_F2FS_FAULT_INJECTION is not set
CONFIG_FS_DAX=y
CONFIG_FS_DAX_PMD=y
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=y
# CONFIG_EXPORTFS_BLOCK_OPS is not set
CONFIG_FILE_LOCKING=y
CONFIG_MANDATORY_FILE_LOCKING=y
CONFIG_FS_ENCRYPTION=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
CONFIG_PRINT_QUOTA_WARNING=y
# CONFIG_QUOTA_DEBUG is not set
CONFIG_QUOTA_TREE=y
# CONFIG_QFMT_V1 is not set
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
CONFIG_QUOTACTL_COMPAT=y
CONFIG_AUTOFS4_FS=y
CONFIG_AUTOFS_FS=y
CONFIG_FUSE_FS=m
CONFIG_CUSE=m
CONFIG_OVERLAY_FS=m
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
# CONFIG_OVERLAY_FS_XINO_AUTO is not set
# CONFIG_OVERLAY_FS_METACOPY is not set

#
# Caches
#
CONFIG_FSCACHE=m
CONFIG_FSCACHE_STATS=y
# CONFIG_FSCACHE_HISTOGRAM is not set
# CONFIG_FSCACHE_DEBUG is not set
# CONFIG_FSCACHE_OBJECT_LIST is not set
CONFIG_CACHEFILES=m
# CONFIG_CACHEFILES_DEBUG is not set
# CONFIG_CACHEFILES_HISTOGRAM is not set
# end of Caches

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
# end of CD-ROM/DVD Filesystems

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="ascii"
# CONFIG_FAT_DEFAULT_UTF8 is not set
# CONFIG_NTFS_FS is not set
# end of DOS/FAT/NT Filesystems

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_VMCORE=y
# CONFIG_PROC_VMCORE_DEVICE_DUMP is not set
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_PROC_CHILDREN=y
CONFIG_KERNFS=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_MEMFD_CREATE=y
CONFIG_ARCH_HAS_GIGANTIC_PAGE=y
CONFIG_CONFIGFS_FS=y
CONFIG_EFIVAR_FS=y
# end of Pseudo filesystems

CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ORANGEFS_FS is not set
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_ECRYPT_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_JFFS2_FS is not set
# CONFIG_UBIFS_FS is not set
CONFIG_CRAMFS=m
CONFIG_CRAMFS_BLOCKDEV=y
# CONFIG_CRAMFS_MTD is not set
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_FILE_CACHE=y
# CONFIG_SQUASHFS_FILE_DIRECT is not set
CONFIG_SQUASHFS_DECOMP_SINGLE=y
# CONFIG_SQUASHFS_DECOMP_MULTI is not set
# CONFIG_SQUASHFS_DECOMP_MULTI_PERCPU is not set
CONFIG_SQUASHFS_XATTR=y
CONFIG_SQUASHFS_ZLIB=y
# CONFIG_SQUASHFS_LZ4 is not set
CONFIG_SQUASHFS_LZO=y
CONFIG_SQUASHFS_XZ=y
# CONFIG_SQUASHFS_ZSTD is not set
# CONFIG_SQUASHFS_4K_DEVBLK_SIZE is not set
# CONFIG_SQUASHFS_EMBEDDED is not set
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3
# CONFIG_VXFS_FS is not set
CONFIG_MINIX_FS=m
# CONFIG_OMFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_QNX6FS_FS is not set
# CONFIG_ROMFS_FS is not set
CONFIG_PSTORE=y
CONFIG_PSTORE_DEFLATE_COMPRESS=y
# CONFIG_PSTORE_LZO_COMPRESS is not set
# CONFIG_PSTORE_LZ4_COMPRESS is not set
# CONFIG_PSTORE_LZ4HC_COMPRESS is not set
# CONFIG_PSTORE_842_COMPRESS is not set
# CONFIG_PSTORE_ZSTD_COMPRESS is not set
CONFIG_PSTORE_COMPRESS=y
CONFIG_PSTORE_DEFLATE_COMPRESS_DEFAULT=y
CONFIG_PSTORE_COMPRESS_DEFAULT="deflate"
CONFIG_PSTORE_CONSOLE=y
CONFIG_PSTORE_PMSG=y
# CONFIG_PSTORE_FTRACE is not set
CONFIG_PSTORE_RAM=m
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
# CONFIG_NFS_V2 is not set
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=m
# CONFIG_NFS_SWAP is not set
CONFIG_NFS_V4_1=y
CONFIG_NFS_V4_2=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m
CONFIG_PNFS_FLEXFILE_LAYOUT=m
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
# CONFIG_NFS_V4_1_MIGRATION is not set
CONFIG_NFS_V4_SECURITY_LABEL=y
CONFIG_ROOT_NFS=y
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFS_DEBUG=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
# CONFIG_NFSD_BLOCKLAYOUT is not set
# CONFIG_NFSD_SCSILAYOUT is not set
# CONFIG_NFSD_FLEXFILELAYOUT is not set
CONFIG_NFSD_V4_SECURITY_LABEL=y
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_GRACE_PERIOD=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=m
CONFIG_SUNRPC_BACKCHANNEL=y
CONFIG_RPCSEC_GSS_KRB5=m
# CONFIG_CONFIG_SUNRPC_DISABLE_INSECURE_ENCTYPES is not set
CONFIG_SUNRPC_DEBUG=y
# CONFIG_CEPH_FS is not set
CONFIG_CIFS=m
# CONFIG_CIFS_STATS2 is not set
CONFIG_CIFS_ALLOW_INSECURE_LEGACY=y
CONFIG_CIFS_WEAK_PW_HASH=y
CONFIG_CIFS_UPCALL=y
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
CONFIG_CIFS_ACL=y
CONFIG_CIFS_DEBUG=y
# CONFIG_CIFS_DEBUG2 is not set
# CONFIG_CIFS_DEBUG_DUMP_KEYS is not set
CONFIG_CIFS_DFS_UPCALL=y
# CONFIG_CIFS_FSCACHE is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
CONFIG_9P_FS=y
CONFIG_9P_FS_POSIX_ACL=y
# CONFIG_9P_FS_SECURITY is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_MAC_ROMAN=m
CONFIG_NLS_MAC_CELTIC=m
CONFIG_NLS_MAC_CENTEURO=m
CONFIG_NLS_MAC_CROATIAN=m
CONFIG_NLS_MAC_CYRILLIC=m
CONFIG_NLS_MAC_GAELIC=m
CONFIG_NLS_MAC_GREEK=m
CONFIG_NLS_MAC_ICELAND=m
CONFIG_NLS_MAC_INUIT=m
CONFIG_NLS_MAC_ROMANIAN=m
CONFIG_NLS_MAC_TURKISH=m
CONFIG_NLS_UTF8=m
CONFIG_DLM=m
CONFIG_DLM_DEBUG=y
# CONFIG_UNICODE is not set
# end of File systems

#
# Security options
#
CONFIG_KEYS=y
CONFIG_KEYS_COMPAT=y
CONFIG_PERSISTENT_KEYRINGS=y
CONFIG_BIG_KEYS=y
CONFIG_TRUSTED_KEYS=y
CONFIG_ENCRYPTED_KEYS=y
# CONFIG_KEY_DH_OPERATIONS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
CONFIG_SECURITY=y
CONFIG_SECURITY_WRITABLE_HOOKS=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_PAGE_TABLE_ISOLATION=y
CONFIG_SECURITY_NETWORK_XFRM=y
CONFIG_SECURITY_PATH=y
CONFIG_INTEL_TXT=y
CONFIG_LSM_MMAP_MIN_ADDR=65535
CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR=y
# CONFIG_HARDENED_USERCOPY is not set
# CONFIG_FORTIFY_SOURCE is not set
# CONFIG_STATIC_USERMODEHELPER is not set
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SMACK is not set
# CONFIG_SECURITY_TOMOYO is not set
CONFIG_SECURITY_APPARMOR=y
CONFIG_SECURITY_APPARMOR_HASH=y
CONFIG_SECURITY_APPARMOR_HASH_DEFAULT=y
# CONFIG_SECURITY_APPARMOR_DEBUG is not set
# CONFIG_SECURITY_LOADPIN is not set
# CONFIG_SECURITY_YAMA is not set
# CONFIG_SECURITY_SAFESETID is not set
CONFIG_INTEGRITY=y
CONFIG_INTEGRITY_SIGNATURE=y
CONFIG_INTEGRITY_ASYMMETRIC_KEYS=y
CONFIG_INTEGRITY_TRUSTED_KEYRING=y
CONFIG_INTEGRITY_AUDIT=y
CONFIG_IMA=y
CONFIG_IMA_MEASURE_PCR_IDX=10
CONFIG_IMA_LSM_RULES=y
# CONFIG_IMA_TEMPLATE is not set
CONFIG_IMA_NG_TEMPLATE=y
# CONFIG_IMA_SIG_TEMPLATE is not set
CONFIG_IMA_DEFAULT_TEMPLATE="ima-ng"
CONFIG_IMA_DEFAULT_HASH_SHA1=y
# CONFIG_IMA_DEFAULT_HASH_SHA256 is not set
# CONFIG_IMA_DEFAULT_HASH_SHA512 is not set
CONFIG_IMA_DEFAULT_HASH="sha1"
# CONFIG_IMA_WRITE_POLICY is not set
# CONFIG_IMA_READ_POLICY is not set
CONFIG_IMA_APPRAISE=y
# CONFIG_IMA_ARCH_POLICY is not set
# CONFIG_IMA_APPRAISE_BUILD_POLICY is not set
CONFIG_IMA_APPRAISE_BOOTPARAM=y
CONFIG_IMA_TRUSTED_KEYRING=y
# CONFIG_IMA_BLACKLIST_KEYRING is not set
# CONFIG_IMA_LOAD_X509 is not set
CONFIG_EVM=y
CONFIG_EVM_ATTR_FSUUID=y
# CONFIG_EVM_ADD_XATTRS is not set
# CONFIG_EVM_LOAD_X509 is not set
CONFIG_DEFAULT_SECURITY_SELINUX=y
# CONFIG_DEFAULT_SECURITY_APPARMOR is not set
# CONFIG_DEFAULT_SECURITY_DAC is not set
CONFIG_LSM="yama,loadpin,safesetid,integrity,selinux,smack,tomoyo,apparmor"

#
# Kernel hardening options
#

#
# Memory initialization
#
CONFIG_INIT_STACK_NONE=y
# CONFIG_GCC_PLUGIN_STRUCTLEAK_USER is not set
# CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF is not set
# CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL is not set
# CONFIG_GCC_PLUGIN_STACKLEAK is not set
# end of Memory initialization
# end of Kernel hardening options
# end of Security options

CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_RNG_DEFAULT=y
CONFIG_CRYPTO_AKCIPHER2=y
CONFIG_CRYPTO_AKCIPHER=y
CONFIG_CRYPTO_KPP2=y
CONFIG_CRYPTO_ACOMP2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_USER=m
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_GF128MUL=y
CONFIG_CRYPTO_NULL=y
CONFIG_CRYPTO_NULL2=y
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_CRYPTO_CRYPTD=m
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_TEST=m
CONFIG_CRYPTO_SIMD=m
CONFIG_CRYPTO_GLUE_HELPER_X86=m
CONFIG_CRYPTO_ENGINE=m

#
# Public-key cryptography
#
CONFIG_CRYPTO_RSA=y
# CONFIG_CRYPTO_DH is not set
# CONFIG_CRYPTO_ECDH is not set
# CONFIG_CRYPTO_ECRDSA is not set

#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=y
# CONFIG_CRYPTO_CHACHA20POLY1305 is not set
# CONFIG_CRYPTO_AEGIS128 is not set
# CONFIG_CRYPTO_AEGIS128L is not set
# CONFIG_CRYPTO_AEGIS256 is not set
# CONFIG_CRYPTO_AEGIS128_AESNI_SSE2 is not set
# CONFIG_CRYPTO_AEGIS128L_AESNI_SSE2 is not set
# CONFIG_CRYPTO_AEGIS256_AESNI_SSE2 is not set
# CONFIG_CRYPTO_MORUS640 is not set
# CONFIG_CRYPTO_MORUS640_SSE2 is not set
# CONFIG_CRYPTO_MORUS1280 is not set
# CONFIG_CRYPTO_MORUS1280_SSE2 is not set
# CONFIG_CRYPTO_MORUS1280_AVX2 is not set
CONFIG_CRYPTO_SEQIV=y
CONFIG_CRYPTO_ECHAINIV=m

#
# Block modes
#
CONFIG_CRYPTO_CBC=y
# CONFIG_CRYPTO_CFB is not set
CONFIG_CRYPTO_CTR=y
CONFIG_CRYPTO_CTS=y
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_LRW=m
# CONFIG_CRYPTO_OFB is not set
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=y
# CONFIG_CRYPTO_KEYWRAP is not set
# CONFIG_CRYPTO_NHPOLY1305_SSE2 is not set
# CONFIG_CRYPTO_NHPOLY1305_AVX2 is not set
# CONFIG_CRYPTO_ADIANTUM is not set

#
# Hash modes
#
CONFIG_CRYPTO_CMAC=m
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_VMAC=m

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_CRC32C_INTEL=m
CONFIG_CRYPTO_CRC32=m
CONFIG_CRYPTO_CRC32_PCLMUL=m
CONFIG_CRYPTO_CRCT10DIF=y
CONFIG_CRYPTO_CRCT10DIF_PCLMUL=m
CONFIG_CRYPTO_GHASH=y
# CONFIG_CRYPTO_POLY1305 is not set
# CONFIG_CRYPTO_POLY1305_X86_64 is not set
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD128=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_RMD256=m
CONFIG_CRYPTO_RMD320=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA1_SSSE3=m
CONFIG_CRYPTO_SHA256_SSSE3=m
CONFIG_CRYPTO_SHA512_SSSE3=m
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_SHA512=y
CONFIG_CRYPTO_SHA3=m
# CONFIG_CRYPTO_SM3 is not set
# CONFIG_CRYPTO_STREEBOG is not set
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL=m

#
# Ciphers
#
CONFIG_CRYPTO_AES=y
# CONFIG_CRYPTO_AES_TI is not set
CONFIG_CRYPTO_AES_X86_64=y
CONFIG_CRYPTO_AES_NI_INTEL=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_BLOWFISH_COMMON=m
CONFIG_CRYPTO_BLOWFISH_X86_64=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAMELLIA_X86_64=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX2_X86_64=m
CONFIG_CRYPTO_CAST_COMMON=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST5_AVX_X86_64=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_CAST6_AVX_X86_64=m
CONFIG_CRYPTO_DES=m
# CONFIG_CRYPTO_DES3_EDE_X86_64 is not set
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
# CONFIG_CRYPTO_CHACHA20 is not set
# CONFIG_CRYPTO_CHACHA20_X86_64 is not set
CONFIG_CRYPTO_SEED=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_SERPENT_SSE2_X86_64=m
CONFIG_CRYPTO_SERPENT_AVX_X86_64=m
CONFIG_CRYPTO_SERPENT_AVX2_X86_64=m
# CONFIG_CRYPTO_SM4 is not set
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m
CONFIG_CRYPTO_TWOFISH_X86_64_3WAY=m
CONFIG_CRYPTO_TWOFISH_AVX_X86_64=m

#
# Compression
#
CONFIG_CRYPTO_DEFLATE=y
CONFIG_CRYPTO_LZO=y
# CONFIG_CRYPTO_842 is not set
# CONFIG_CRYPTO_LZ4 is not set
# CONFIG_CRYPTO_LZ4HC is not set
# CONFIG_CRYPTO_ZSTD is not set

#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=m
CONFIG_CRYPTO_DRBG_MENU=y
CONFIG_CRYPTO_DRBG_HMAC=y
# CONFIG_CRYPTO_DRBG_HASH is not set
# CONFIG_CRYPTO_DRBG_CTR is not set
CONFIG_CRYPTO_DRBG=y
CONFIG_CRYPTO_JITTERENTROPY=y
CONFIG_CRYPTO_USER_API=y
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CRYPTO_USER_API_SKCIPHER=y
# CONFIG_CRYPTO_USER_API_RNG is not set
# CONFIG_CRYPTO_USER_API_AEAD is not set
# CONFIG_CRYPTO_STATS is not set
CONFIG_CRYPTO_HASH_INFO=y
CONFIG_CRYPTO_HW=y
CONFIG_CRYPTO_DEV_PADLOCK=m
CONFIG_CRYPTO_DEV_PADLOCK_AES=m
CONFIG_CRYPTO_DEV_PADLOCK_SHA=m
# CONFIG_CRYPTO_DEV_CCP is not set
# CONFIG_CRYPTO_DEV_QAT_DH895xCC is not set
# CONFIG_CRYPTO_DEV_QAT_C3XXX is not set
# CONFIG_CRYPTO_DEV_QAT_C62X is not set
# CONFIG_CRYPTO_DEV_QAT_DH895xCCVF is not set
# CONFIG_CRYPTO_DEV_QAT_C3XXXVF is not set
# CONFIG_CRYPTO_DEV_QAT_C62XVF is not set
# CONFIG_CRYPTO_DEV_NITROX_CNN55XX is not set
# CONFIG_CRYPTO_DEV_CHELSIO is not set
CONFIG_CRYPTO_DEV_VIRTIO=m
CONFIG_ASYMMETRIC_KEY_TYPE=y
CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE=y
# CONFIG_ASYMMETRIC_TPM_KEY_SUBTYPE is not set
CONFIG_X509_CERTIFICATE_PARSER=y
# CONFIG_PKCS8_PRIVATE_KEY_PARSER is not set
CONFIG_PKCS7_MESSAGE_PARSER=y
# CONFIG_PKCS7_TEST_KEY is not set
# CONFIG_SIGNED_PE_FILE_VERIFICATION is not set

#
# Certificates for signature checking
#
CONFIG_SYSTEM_TRUSTED_KEYRING=y
CONFIG_SYSTEM_TRUSTED_KEYS=""
# CONFIG_SYSTEM_EXTRA_CERTIFICATE is not set
# CONFIG_SECONDARY_TRUSTED_KEYRING is not set
# CONFIG_SYSTEM_BLACKLIST_KEYRING is not set
# end of Certificates for signature checking

CONFIG_BINARY_PRINTF=y

#
# Library routines
#
CONFIG_RAID6_PQ=m
CONFIG_RAID6_PQ_BENCHMARK=y
# CONFIG_PACKING is not set
CONFIG_BITREVERSE=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_NET_UTILS=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_CORDIC=m
CONFIG_PRIME_NUMBERS=m
CONFIG_RATIONAL=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y
CONFIG_ARCH_USE_CMPXCHG_LOCKREF=y
CONFIG_ARCH_HAS_FAST_MULTIPLIER=y
CONFIG_CRC_CCITT=y
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
# CONFIG_CRC32_SELFTEST is not set
CONFIG_CRC32_SLICEBY8=y
# CONFIG_CRC32_SLICEBY4 is not set
# CONFIG_CRC32_SARWATE is not set
# CONFIG_CRC32_BIT is not set
# CONFIG_CRC64 is not set
# CONFIG_CRC4 is not set
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=y
CONFIG_CRC8=m
CONFIG_XXHASH=y
# CONFIG_RANDOM32_SELFTEST is not set
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_LZ4_DECOMPRESS=y
CONFIG_ZSTD_COMPRESS=m
CONFIG_ZSTD_DECOMPRESS=m
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_DECOMPRESS_LZ4=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_REED_SOLOMON=m
CONFIG_REED_SOLOMON_ENC8=y
CONFIG_REED_SOLOMON_DEC8=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_BTREE=y
CONFIG_INTERVAL_TREE=y
CONFIG_XARRAY_MULTI=y
CONFIG_ASSOCIATIVE_ARRAY=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT_MAP=y
CONFIG_HAS_DMA=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_DMA_DECLARE_COHERENT=y
CONFIG_SWIOTLB=y
CONFIG_DMA_CMA=y

#
# Default contiguous memory area size:
#
CONFIG_CMA_SIZE_MBYTES=200
CONFIG_CMA_SIZE_SEL_MBYTES=y
# CONFIG_CMA_SIZE_SEL_PERCENTAGE is not set
# CONFIG_CMA_SIZE_SEL_MIN is not set
# CONFIG_CMA_SIZE_SEL_MAX is not set
CONFIG_CMA_ALIGNMENT=8
# CONFIG_DMA_API_DEBUG is not set
CONFIG_SGL_ALLOC=y
CONFIG_IOMMU_HELPER=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_CPUMASK_OFFSTACK=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_GLOB=y
# CONFIG_GLOB_SELFTEST is not set
CONFIG_NLATTR=y
CONFIG_CLZ_TAB=y
# CONFIG_DDR is not set
CONFIG_IRQ_POLL=y
CONFIG_MPILIB=y
CONFIG_SIGNATURE=y
CONFIG_OID_REGISTRY=y
CONFIG_UCS2_STRING=y
CONFIG_FONT_SUPPORT=y
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_SG_POOL=y
CONFIG_ARCH_HAS_PMEM_API=y
CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE=y
CONFIG_ARCH_HAS_UACCESS_MCSAFE=y
CONFIG_ARCH_STACKWALK=y
CONFIG_SBITMAP=y
# CONFIG_STRING_SELFTEST is not set
# end of Library routines

#
# Kernel hacking
#

#
# printk and dmesg options
#
CONFIG_PRINTK_TIME=y
# CONFIG_PRINTK_CALLER is not set
CONFIG_CONSOLE_LOGLEVEL_DEFAULT=7
CONFIG_CONSOLE_LOGLEVEL_QUIET=4
CONFIG_MESSAGE_LOGLEVEL_DEFAULT=4
CONFIG_BOOT_PRINTK_DELAY=y
CONFIG_DYNAMIC_DEBUG=y
# end of printk and dmesg options

#
# Compile-time checks and compiler options
#
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_REDUCED=y
# CONFIG_DEBUG_INFO_SPLIT is not set
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_DEBUG_INFO_BTF is not set
# CONFIG_GDB_SCRIPTS is not set
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_STRIP_ASM_SYMS=y
# CONFIG_READABLE_ASM is not set
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
CONFIG_HEADERS_CHECK=y
CONFIG_OPTIMIZE_INLINING=y
CONFIG_DEBUG_SECTION_MISMATCH=y
CONFIG_SECTION_MISMATCH_WARN_ONLY=y
CONFIG_STACK_VALIDATION=y
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
# end of Compile-time checks and compiler options

CONFIG_MAGIC_SYSRQ=y
CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
CONFIG_MAGIC_SYSRQ_SERIAL=y
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_MISC=y

#
# Memory Debugging
#
# CONFIG_PAGE_EXTENSION is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_PAGE_OWNER is not set
# CONFIG_PAGE_POISONING is not set
# CONFIG_DEBUG_PAGE_REF is not set
CONFIG_DEBUG_RODATA_TEST=y
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
CONFIG_HAVE_DEBUG_KMEMLEAK=y
# CONFIG_DEBUG_KMEMLEAK is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_VM is not set
CONFIG_ARCH_HAS_DEBUG_VIRTUAL=y
# CONFIG_DEBUG_VIRTUAL is not set
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_MEMORY_NOTIFIER_ERROR_INJECT=m
# CONFIG_DEBUG_PER_CPU_MAPS is not set
CONFIG_HAVE_ARCH_KASAN=y
CONFIG_CC_HAS_KASAN_GENERIC=y
# CONFIG_KASAN is not set
CONFIG_KASAN_STACK=1
# end of Memory Debugging

CONFIG_ARCH_HAS_KCOV=y
CONFIG_CC_HAS_SANCOV_TRACE_PC=y
# CONFIG_KCOV is not set
CONFIG_DEBUG_SHIRQ=y

#
# Debug Lockups and Hangs
#
CONFIG_LOCKUP_DETECTOR=y
CONFIG_SOFTLOCKUP_DETECTOR=y
# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
CONFIG_HARDLOCKUP_DETECTOR_PERF=y
CONFIG_HARDLOCKUP_CHECK_TIMESTAMP=y
CONFIG_HARDLOCKUP_DETECTOR=y
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=y
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=1
# CONFIG_DETECT_HUNG_TASK is not set
# CONFIG_WQ_WATCHDOG is not set
# end of Debug Lockups and Hangs

CONFIG_PANIC_ON_OOPS=y
CONFIG_PANIC_ON_OOPS_VALUE=1
CONFIG_PANIC_TIMEOUT=0
CONFIG_SCHED_DEBUG=y
CONFIG_SCHED_INFO=y
CONFIG_SCHEDSTATS=y
# CONFIG_SCHED_STACK_END_CHECK is not set
# CONFIG_DEBUG_TIMEKEEPING is not set

#
# Lock Debugging (spinlocks, mutexes, etc...)
#
CONFIG_LOCK_DEBUGGING_SUPPORT=y
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_WW_MUTEX_SLOWPATH is not set
# CONFIG_DEBUG_RWSEMS is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
CONFIG_DEBUG_ATOMIC_SLEEP=y
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_LOCK_TORTURE_TEST=m
CONFIG_WW_MUTEX_SELFTEST=m
# end of Lock Debugging (spinlocks, mutexes, etc...)

CONFIG_STACKTRACE=y
# CONFIG_WARN_ALL_UNSEEDED_RANDOM is not set
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_LIST=y
# CONFIG_DEBUG_PLIST is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set

#
# RCU Debugging
#
CONFIG_TORTURE_TEST=m
CONFIG_RCU_PERF_TEST=m
CONFIG_RCU_TORTURE_TEST=m
CONFIG_RCU_CPU_STALL_TIMEOUT=60
# CONFIG_RCU_TRACE is not set
# CONFIG_RCU_EQS_DEBUG is not set
# end of RCU Debugging

# CONFIG_DEBUG_WQ_FORCE_RR_CPU is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_CPU_HOTPLUG_STATE_CONTROL is not set
CONFIG_NOTIFIER_ERROR_INJECTION=m
CONFIG_PM_NOTIFIER_ERROR_INJECT=m
# CONFIG_NETDEV_NOTIFIER_ERROR_INJECT is not set
CONFIG_FUNCTION_ERROR_INJECTION=y
CONFIG_FAULT_INJECTION=y
# CONFIG_FAILSLAB is not set
# CONFIG_FAIL_PAGE_ALLOC is not set
CONFIG_FAIL_MAKE_REQUEST=y
# CONFIG_FAIL_IO_TIMEOUT is not set
# CONFIG_FAIL_FUTEX is not set
CONFIG_FAULT_INJECTION_DEBUG_FS=y
# CONFIG_FAIL_FUNCTION is not set
# CONFIG_FAIL_MMC_REQUEST is not set
CONFIG_LATENCYTOP=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_FENTRY=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACE_CLOCK=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
# CONFIG_PREEMPTIRQ_EVENTS is not set
# CONFIG_IRQSOFF_TRACER is not set
CONFIG_SCHED_TRACER=y
# CONFIG_HWLAT_TRACER is not set
CONFIG_FTRACE_SYSCALLS=y
CONFIG_TRACER_SNAPSHOT=y
# CONFIG_TRACER_SNAPSHOT_PER_CPU_SWAP is not set
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
CONFIG_STACK_TRACER=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_KPROBE_EVENTS=y
# CONFIG_KPROBE_EVENTS_ON_NOTRACE is not set
CONFIG_UPROBE_EVENTS=y
CONFIG_BPF_EVENTS=y
CONFIG_DYNAMIC_EVENTS=y
CONFIG_PROBE_EVENTS=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_FUNCTION_PROFILER=y
# CONFIG_BPF_KPROBE_OVERRIDE is not set
CONFIG_FTRACE_MCOUNT_RECORD=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_MMIOTRACE is not set
CONFIG_TRACING_MAP=y
CONFIG_HIST_TRIGGERS=y
# CONFIG_TRACEPOINT_BENCHMARK is not set
CONFIG_RING_BUFFER_BENCHMARK=m
# CONFIG_RING_BUFFER_STARTUP_TEST is not set
# CONFIG_PREEMPTIRQ_DELAY_TEST is not set
# CONFIG_TRACE_EVAL_MAP_FILE is not set
CONFIG_PROVIDE_OHCI1394_DMA_INIT=y
CONFIG_RUNTIME_TESTING_MENU=y
# CONFIG_LKDTM is not set
# CONFIG_TEST_LIST_SORT is not set
# CONFIG_TEST_SORT is not set
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_RBTREE_TEST is not set
# CONFIG_INTERVAL_TREE_TEST is not set
# CONFIG_PERCPU_TEST is not set
CONFIG_ATOMIC64_SELFTEST=y
# CONFIG_ASYNC_RAID6_TEST is not set
# CONFIG_TEST_HEXDUMP is not set
# CONFIG_TEST_STRING_HELPERS is not set
CONFIG_TEST_STRSCPY=m
# CONFIG_TEST_KSTRTOX is not set
CONFIG_TEST_PRINTF=m
CONFIG_TEST_BITMAP=m
# CONFIG_TEST_BITFIELD is not set
# CONFIG_TEST_UUID is not set
# CONFIG_TEST_XARRAY is not set
# CONFIG_TEST_OVERFLOW is not set
# CONFIG_TEST_RHASHTABLE is not set
# CONFIG_TEST_HASH is not set
# CONFIG_TEST_IDA is not set
CONFIG_TEST_LKM=m
CONFIG_TEST_VMALLOC=m
CONFIG_TEST_USER_COPY=m
CONFIG_TEST_BPF=m
# CONFIG_FIND_BIT_BENCHMARK is not set
CONFIG_TEST_FIRMWARE=m
CONFIG_TEST_SYSCTL=m
# CONFIG_TEST_UDELAY is not set
CONFIG_TEST_STATIC_KEYS=m
CONFIG_TEST_KMOD=m
# CONFIG_TEST_MEMCAT_P is not set
# CONFIG_TEST_STACKINIT is not set
# CONFIG_MEMTEST is not set
# CONFIG_BUG_ON_DATA_CORRUPTION is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_ARCH_HAS_UBSAN_SANITIZE_ALL=y
# CONFIG_UBSAN is not set
CONFIG_UBSAN_ALIGNMENT=y
CONFIG_ARCH_HAS_DEVMEM_IS_ALLOWED=y
CONFIG_STRICT_DEVMEM=y
# CONFIG_IO_STRICT_DEVMEM is not set
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_EARLY_PRINTK_USB=y
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
CONFIG_EARLY_PRINTK_DBGP=y
# CONFIG_EARLY_PRINTK_USB_XDBC is not set
# CONFIG_X86_PTDUMP is not set
# CONFIG_EFI_PGT_DUMP is not set
# CONFIG_DEBUG_WX is not set
CONFIG_DOUBLEFAULT=y
# CONFIG_DEBUG_TLBFLUSH is not set
# CONFIG_IOMMU_DEBUG is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_X86_DECODER_SELFTEST=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_DEBUG_BOOT_PARAMS=y
# CONFIG_CPA_DEBUG is not set
# CONFIG_DEBUG_ENTRY is not set
# CONFIG_DEBUG_NMI_SELFTEST is not set
CONFIG_X86_DEBUG_FPU=y
# CONFIG_PUNIT_ATOM_DEBUG is not set
CONFIG_UNWINDER_ORC=y
# CONFIG_UNWINDER_FRAME_POINTER is not set
# CONFIG_UNWINDER_GUESS is not set
# end of Kernel hacking

[-- Attachment #3: job-script.ksh --]
[-- Type: text/plain, Size: 7437 bytes --]

#!/bin/sh

export_top_env()
{
	export suite='will-it-scale'
	export testcase='will-it-scale'
	export category='benchmark'
	export nr_task=48
	export job_origin='/lkp/lkp/.src-20190827-185517/allot/cyclic:clear:internal-eywa:master/lkp-csl-2sp4/will-it-scale-50.yaml'
	export queue_cmdline_keys='branch
commit
queue_at_least_once'
	export queue='validate'
	export testbox='lkp-csl-2sp4'
	export tbox_group='lkp-csl-2sp4'
	export submit_id='5d7045a4ddb30725d08ec528'
	export job_file='/lkp/jobs/scheduled/lkp-csl-2sp4/will-it-scale-performance-thread-50%-mmap2-ucode=0x400001c-clear--20190905-9680-1vdpqfd-3.yaml'
	export id='485472c35036bb4a402eeb957e19a671b5ac5fc3'
	export queuer_version='/lkp-src'
	export arch='x86_64'
	export model='Cascade Lake'
	export nr_node=2
	export nr_cpu=96
	export memory='128G'
	export nr_hdd_partitions=2
	export hdd_partitions='/dev/disk/by-id/ata-ST500LM021-1KJ152_W625RX03-part*'
	export nr_ssd_partitions=4
	export ssd_partitions='/dev/disk/by-id/ata-INTEL_SSDSC2BG012T4_BTHC427503001P2KGN-part*'
	export swap_partitions=
	export rootfs_partition='LABEL=LKP-ROOTFS'
	export brand='Intel(R) Xeon(R) CPU @ 2.30GHz'
	export ucode='0x400001c'
	export commit='00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f'
	export rootfs='clear-ota-25590-x86_64-2018-10-18.cgz'
	export kconfig='x86_64-rhel-7.2-clear'
	export run_on_local_disk=true
	export need_kconfig_hw='CONFIG_I40E=y'
	export compiler='gcc-7'
	export enqueue_time='2019-09-05 07:16:34 +0800'
	export _id='5d7045d2ddb30725d08ec529'
	export _rt='/result/will-it-scale/performance-thread-50%-mmap2-ucode=0x400001c/lkp-csl-2sp4/clear-ota-25590-x86_64-2018-10-18.cgz/x86_64-rhel-7.2-clear/gcc-7/00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f'
	export user='lkp'
	export head_commit='c16bf780467a13b504bae4ec11a0421417fda99a'
	export base_commit='a55aa89aab90fae7c815b0551b07be37db359d76'
	export branch='internal-eywa/master'
	export result_root='/result/will-it-scale/performance-thread-50%-mmap2-ucode=0x400001c/lkp-csl-2sp4/clear-ota-25590-x86_64-2018-10-18.cgz/x86_64-rhel-7.2-clear/gcc-7/00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f/3'
	export scheduler_version='/lkp/lkp/.src-20190904-142029'
	export LKP_SERVER='inn'
	export max_uptime=1500
	export initrd='/osimage/clear/clear-ota-25590-x86_64-2018-10-18.cgz'
	export bootloader_append='root=/dev/ram0
user=lkp
job=/lkp/jobs/scheduled/lkp-csl-2sp4/will-it-scale-performance-thread-50%-mmap2-ucode=0x400001c-clear--20190905-9680-1vdpqfd-3.yaml
ARCH=x86_64
kconfig=x86_64-rhel-7.2-clear
branch=internal-eywa/master
commit=00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f
BOOT_IMAGE=/pkg/linux/x86_64-rhel-7.2-clear/gcc-7/00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f/vmlinuz-5.2.0-rc5-00067-g00f3c5a
max_uptime=1500
RESULT_ROOT=/result/will-it-scale/performance-thread-50%-mmap2-ucode=0x400001c/lkp-csl-2sp4/clear-ota-25590-x86_64-2018-10-18.cgz/x86_64-rhel-7.2-clear/gcc-7/00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f/3
LKP_SERVER=inn
debug
apic=debug
sysrq_always_enabled
rcupdate.rcu_cpu_stall_timeout=100
net.ifnames=0
printk.devkmsg=on
panic=-1
softlockup_panic=1
nmi_watchdog=panic
oops=panic
load_ramdisk=2
prompt_ramdisk=0
drbd.minor_count=8
systemd.log_level=err
ignore_loglevel
console=tty0
earlyprintk=ttyS0,115200
console=ttyS0,115200
vga=normal
rw'
	export modules_initrd='/pkg/linux/x86_64-rhel-7.2-clear/gcc-7/00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f/modules.cgz'
	export bm_initrd='/osimage/deps/clear-ota-25590-x86_64-2018-10-18.cgz/lkp_2019-04-25.cgz,/osimage/deps/clear-ota-25590-x86_64-2018-10-18.cgz/will-it-scale_2019-06-19.cgz,/osimage/pkg/clear-ota-25590-x86_64-2018-10-18.cgz/will-it-scale-x86_64-1eef89e_2019-08-22.cgz,/osimage/deps/clear-ota-25590-x86_64-2018-10-18.cgz/mpstat_2019-04-30.cgz,/osimage/pkg/clear-ota-25590-x86_64-2018-10-18.cgz/turbostat-x86_64-d5256b2_2019-04-30.cgz,/osimage/deps/clear-ota-25590-x86_64-2018-10-18.cgz/perf_2019-04-30.cgz,/osimage/pkg/clear-ota-25590-x86_64-2018-10-18.cgz/perf-x86_64-cb6f8739fbf9_2019-05-20.cgz,/osimage/pkg/clear-ota-25590-x86_64-2018-10-18.cgz/mpstat-x86_64-git-1_2019-05-05.cgz'
	export lkp_initrd='/osimage/user/lkp/lkp-x86_64.cgz'
	export site='inn'
	export LKP_CGI_PORT=80
	export LKP_CIFS_PORT=139
	export repeat_to=4
	export bad_samples='5481
5513
5453'
	export schedule_notify_address=
	export queue_at_least_once=1
	export kernel='/pkg/linux/x86_64-rhel-7.2-clear/gcc-7/00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f/vmlinuz-5.2.0-rc5-00067-g00f3c5a'
	export dequeue_time='2019-09-05 07:25:22 +0800'
	export job_initrd='/lkp/jobs/scheduled/lkp-csl-2sp4/will-it-scale-performance-thread-50%-mmap2-ucode=0x400001c-clear--20190905-9680-1vdpqfd-3.cgz'

	[ -n "$LKP_SRC" ] ||
	export LKP_SRC=/lkp/${user:-lkp}/src
}

run_job()
{
	echo $$ > $TMP/run-job.pid

	. $LKP_SRC/lib/http.sh
	. $LKP_SRC/lib/job.sh
	. $LKP_SRC/lib/env.sh

	export_top_env

	run_setup $LKP_SRC/setup/cpufreq_governor 'performance'

	run_monitor $LKP_SRC/monitors/wrapper kmsg
	run_monitor $LKP_SRC/monitors/no-stdout/wrapper boot-time
	run_monitor $LKP_SRC/monitors/wrapper iostat
	run_monitor $LKP_SRC/monitors/wrapper heartbeat
	run_monitor $LKP_SRC/monitors/wrapper vmstat
	run_monitor $LKP_SRC/monitors/wrapper numa-numastat
	run_monitor $LKP_SRC/monitors/wrapper numa-vmstat
	run_monitor $LKP_SRC/monitors/wrapper numa-meminfo
	run_monitor $LKP_SRC/monitors/wrapper proc-vmstat
	run_monitor $LKP_SRC/monitors/wrapper proc-stat
	run_monitor $LKP_SRC/monitors/wrapper meminfo
	run_monitor $LKP_SRC/monitors/wrapper slabinfo
	run_monitor $LKP_SRC/monitors/wrapper interrupts
	run_monitor $LKP_SRC/monitors/wrapper lock_stat
	run_monitor $LKP_SRC/monitors/wrapper latency_stats
	run_monitor $LKP_SRC/monitors/wrapper softirqs
	run_monitor $LKP_SRC/monitors/one-shot/wrapper bdi_dev_mapping
	run_monitor $LKP_SRC/monitors/wrapper diskstats
	run_monitor $LKP_SRC/monitors/wrapper nfsstat
	run_monitor $LKP_SRC/monitors/wrapper cpuidle
	run_monitor $LKP_SRC/monitors/wrapper cpufreq-stats
	run_monitor $LKP_SRC/monitors/wrapper turbostat
	run_monitor $LKP_SRC/monitors/wrapper sched_debug
	run_monitor $LKP_SRC/monitors/wrapper perf-stat
	run_monitor $LKP_SRC/monitors/wrapper mpstat
	run_monitor $LKP_SRC/monitors/no-stdout/wrapper perf-profile
	run_monitor $LKP_SRC/monitors/wrapper oom-killer
	run_monitor $LKP_SRC/monitors/plain/watchdog

	run_test mode='thread' test='mmap2' $LKP_SRC/tests/wrapper will-it-scale
}

extract_stats()
{
	export stats_part_begin=
	export stats_part_end=

	$LKP_SRC/stats/wrapper will-it-scale
	$LKP_SRC/stats/wrapper kmsg
	$LKP_SRC/stats/wrapper boot-time
	$LKP_SRC/stats/wrapper iostat
	$LKP_SRC/stats/wrapper vmstat
	$LKP_SRC/stats/wrapper numa-numastat
	$LKP_SRC/stats/wrapper numa-vmstat
	$LKP_SRC/stats/wrapper numa-meminfo
	$LKP_SRC/stats/wrapper proc-vmstat
	$LKP_SRC/stats/wrapper meminfo
	$LKP_SRC/stats/wrapper slabinfo
	$LKP_SRC/stats/wrapper interrupts
	$LKP_SRC/stats/wrapper lock_stat
	$LKP_SRC/stats/wrapper latency_stats
	$LKP_SRC/stats/wrapper softirqs
	$LKP_SRC/stats/wrapper diskstats
	$LKP_SRC/stats/wrapper nfsstat
	$LKP_SRC/stats/wrapper cpuidle
	$LKP_SRC/stats/wrapper turbostat
	$LKP_SRC/stats/wrapper sched_debug
	$LKP_SRC/stats/wrapper perf-stat
	$LKP_SRC/stats/wrapper mpstat
	$LKP_SRC/stats/wrapper perf-profile

	$LKP_SRC/stats/wrapper time will-it-scale.time
	$LKP_SRC/stats/wrapper dmesg
	$LKP_SRC/stats/wrapper kmsg
	$LKP_SRC/stats/wrapper last_state
	$LKP_SRC/stats/wrapper stderr
	$LKP_SRC/stats/wrapper time
}

"$@"

[-- Attachment #4: job.yaml --]
[-- Type: text/plain, Size: 4796 bytes --]

---
suite: will-it-scale
testcase: will-it-scale
category: benchmark
nr_task: 50%
will-it-scale:
  mode: thread
  test: mmap2
job_origin: "/lkp/lkp/.src-20190827-185517/allot/cyclic:clear:internal-eywa:master/lkp-csl-2sp4/will-it-scale-50.yaml"
queue_cmdline_keys:
- branch
- commit
- queue_at_least_once
queue: bisect
testbox: lkp-csl-2sp4
tbox_group: lkp-csl-2sp4
submit_id: 5d702a49ddb3072371a9a169
job_file: "/lkp/jobs/scheduled/lkp-csl-2sp4/will-it-scale-performance-thread-50%-mmap2-ucode=0x400001c-clear-ot-20190905-9073-16oqe12-0.yaml"
id: 491cb9535d17658447f3f6810e0b2920ccf67b39
queuer_version: "/lkp-src"
arch: x86_64
model: Cascade Lake
nr_node: 2
nr_cpu: 96
memory: 128G
nr_hdd_partitions: 2
hdd_partitions: "/dev/disk/by-id/ata-ST500LM021-1KJ152_W625RX03-part*"
nr_ssd_partitions: 4
ssd_partitions: "/dev/disk/by-id/ata-INTEL_SSDSC2BG012T4_BTHC427503001P2KGN-part*"
swap_partitions: 
rootfs_partition: LABEL=LKP-ROOTFS
brand: Intel(R) Xeon(R) CPU @ 2.30GHz
ucode: '0x400001c'
kmsg: 
boot-time: 
iostat: 
heartbeat: 
vmstat: 
numa-numastat: 
numa-vmstat: 
numa-meminfo: 
proc-vmstat: 
proc-stat: 
meminfo: 
slabinfo: 
interrupts: 
lock_stat: 
latency_stats: 
softirqs: 
bdi_dev_mapping: 
diskstats: 
nfsstat: 
cpuidle: 
cpufreq-stats: 
turbostat: 
sched_debug: 
perf-stat: 
mpstat: 
perf-profile: 
cpufreq_governor: performance
commit: 00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f
rootfs: clear-ota-25590-x86_64-2018-10-18.cgz
kconfig: x86_64-rhel-7.2-clear
run_on_local_disk: true
need_kconfig_hw: CONFIG_I40E=y
compiler: gcc-7
enqueue_time: 2019-09-05 05:20:05.149041055 +08:00
_id: 5d702a49ddb3072371a9a169
_rt: "/result/will-it-scale/performance-thread-50%-mmap2-ucode=0x400001c/lkp-csl-2sp4/clear-ota-25590-x86_64-2018-10-18.cgz/x86_64-rhel-7.2-clear/gcc-7/00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f"
user: lkp
head_commit: c16bf780467a13b504bae4ec11a0421417fda99a
base_commit: a55aa89aab90fae7c815b0551b07be37db359d76
branch: internal-eywa/master
result_root: "/result/will-it-scale/performance-thread-50%-mmap2-ucode=0x400001c/lkp-csl-2sp4/clear-ota-25590-x86_64-2018-10-18.cgz/x86_64-rhel-7.2-clear/gcc-7/00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f/0"
scheduler_version: "/lkp/lkp/.src-20190904-142029"
LKP_SERVER: inn
max_uptime: 1500
initrd: "/osimage/clear/clear-ota-25590-x86_64-2018-10-18.cgz"
bootloader_append:
- root=/dev/ram0
- user=lkp
- job=/lkp/jobs/scheduled/lkp-csl-2sp4/will-it-scale-performance-thread-50%-mmap2-ucode=0x400001c-clear-ot-20190905-9073-16oqe12-0.yaml
- ARCH=x86_64
- kconfig=x86_64-rhel-7.2-clear
- branch=internal-eywa/master
- commit=00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f
- BOOT_IMAGE=/pkg/linux/x86_64-rhel-7.2-clear/gcc-7/00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f/vmlinuz-5.2.0-rc5-00067-g00f3c5a
- max_uptime=1500
- RESULT_ROOT=/result/will-it-scale/performance-thread-50%-mmap2-ucode=0x400001c/lkp-csl-2sp4/clear-ota-25590-x86_64-2018-10-18.cgz/x86_64-rhel-7.2-clear/gcc-7/00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f/0
- LKP_SERVER=inn
- debug
- apic=debug
- sysrq_always_enabled
- rcupdate.rcu_cpu_stall_timeout=100
- net.ifnames=0
- printk.devkmsg=on
- panic=-1
- softlockup_panic=1
- nmi_watchdog=panic
- oops=panic
- load_ramdisk=2
- prompt_ramdisk=0
- drbd.minor_count=8
- systemd.log_level=err
- ignore_loglevel
- console=tty0
- earlyprintk=ttyS0,115200
- console=ttyS0,115200
- vga=normal
- rw
modules_initrd: "/pkg/linux/x86_64-rhel-7.2-clear/gcc-7/00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f/modules.cgz"
bm_initrd: "/osimage/deps/clear-ota-25590-x86_64-2018-10-18.cgz/lkp_2019-04-25.cgz,/osimage/deps/clear-ota-25590-x86_64-2018-10-18.cgz/will-it-scale_2019-06-19.cgz,/osimage/pkg/clear-ota-25590-x86_64-2018-10-18.cgz/will-it-scale-x86_64-1eef89e_2019-08-22.cgz,/osimage/deps/clear-ota-25590-x86_64-2018-10-18.cgz/mpstat_2019-04-30.cgz,/osimage/pkg/clear-ota-25590-x86_64-2018-10-18.cgz/turbostat-x86_64-d5256b2_2019-04-30.cgz,/osimage/deps/clear-ota-25590-x86_64-2018-10-18.cgz/perf_2019-04-30.cgz,/osimage/pkg/clear-ota-25590-x86_64-2018-10-18.cgz/perf-x86_64-cb6f8739fbf9_2019-05-20.cgz,/osimage/pkg/clear-ota-25590-x86_64-2018-10-18.cgz/mpstat-x86_64-git-1_2019-05-05.cgz"
lkp_initrd: "/osimage/user/lkp/lkp-x86_64.cgz"
site: inn
LKP_CGI_PORT: 80
LKP_CIFS_PORT: 139
oom-killer: 
watchdog: 
repeat_to: 2
bad_samples:
- 5481
- 5513
- 5453

#! queue options
schedule_notify_address: 

#! user overrides
queue_at_least_once: 0

#! schedule options
kernel: "/pkg/linux/x86_64-rhel-7.2-clear/gcc-7/00f3c5a3df2c1e3dab14d0dd2b71f852d46be97f/vmlinuz-5.2.0-rc5-00067-g00f3c5a"
dequeue_time: 2019-09-05 06:13:28.934278473 +08:00

#! /lkp/lkp/.src-20190904-142029/include/site/inn

#! runtime status
job_state: finished
loadavg: 27.31 27.49 12.99 1/727 32646
start_time: '1567635347'
end_time: '1567635665'
version: "/lkp/lkp/.src-20190904-142052"

[-- Attachment #5: reproduce.ksh --]
[-- Type: text/plain, Size: 307 bytes --]


for cpu_dir in /sys/devices/system/cpu/cpu[0-9]*
do
	online_file="$cpu_dir"/online
	[ -f "$online_file" ] && [ "$(cat "$online_file")" -eq 0 ] && continue

	file="$cpu_dir"/cpufreq/scaling_governor
	[ -f "$file" ] && echo "performance" > "$file"
done

 "python2" "./runtest.py" "mmap2" "295" "thread" "48"

^ permalink raw reply	[flat|nested] only message in thread

only message in thread, other threads:[~2019-09-05  7:52 UTC | newest]

Thread overview: (only message) (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-09-05  7:52 [locking/rwsem] 00f3c5a3df: will-it-scale.per_thread_ops -5.1% regression kernel test robot

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.