All of lore.kernel.org
 help / color / mirror / Atom feed
* [sched/fair] 2c83362734: pft.faults_per_sec_per_cpu -41.4% regression
@ 2019-02-28  7:17 kernel test robot
  2019-02-28 11:10   ` Mel Gorman
  0 siblings, 1 reply; 4+ messages in thread
From: kernel test robot @ 2019-02-28  7:17 UTC (permalink / raw)
  To: lkp

[-- Attachment #1: Type: text/plain, Size: 357451 bytes --]

Greeting,

FYI, we noticed a -41.4% regression of pft.faults_per_sec_per_cpu due to commit:


commit: 2c83362734dad8e48ccc0710b5cd2436a0323893 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: pft
on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory
with following parameters:

	runtime: 300s
	nr_task: 50%
	cpufreq_governor: performance
	ucode: 0xb00002e

test-description: Pft is the page fault test micro benchmark.
test-url: https://github.com/gormanm/pft

In addition to that, the commit also has significant impact on the following tests:

+------------------+--------------------------------------------------------------------------+
| testcase: change | stream:                                                                  |
| test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
| test parameters  | array_size=10000000                                                      |
|                  | cpufreq_governor=performance                                             |
|                  | nr_threads=25%                                                           |
|                  | omp=true                                                                 |
|                  | ucode=0xb00002e                                                          |
+------------------+--------------------------------------------------------------------------+
| testcase: change | reaim: reaim.jobs_per_min 1.3% improvement                               |
| test machine     | 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory    |
| test parameters  | cpufreq_governor=performance                                             |
|                  | nr_job=3000                                                              |
|                  | nr_task=100%                                                             |
|                  | runtime=300s                                                             |
|                  | test=custom                                                              |
|                  | ucode=0x3d                                                               |
+------------------+--------------------------------------------------------------------------+
| testcase: change | stream: stream.add_bandwidth_MBps -32.0% regression                      |
| test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
| test parameters  | array_size=10000000                                                      |
|                  | cpufreq_governor=performance                                             |
|                  | nr_threads=50%                                                           |
|                  | omp=true                                                                 |
|                  | ucode=0xb00002e                                                          |
+------------------+--------------------------------------------------------------------------+
| testcase: change | plzip:                                                                   |
| test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
| test parameters  | cpufreq_governor=performance                                             |
|                  | nr_threads=100%                                                          |
|                  | ucode=0xb00002e                                                          |
+------------------+--------------------------------------------------------------------------+
| testcase: change | reaim: reaim.jobs_per_min -11.9% regression                              |
| test machine     | 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 512G memory   |
| test parameters  | cpufreq_governor=performance                                             |
|                  | nr_task=100%                                                             |
|                  | runtime=300s                                                             |
|                  | test=all_utime                                                           |
|                  | ucode=0xb00002e                                                          |
+------------------+--------------------------------------------------------------------------+
| testcase: change | hackbench: hackbench.throughput -7.3% regression                         |
| test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory     |
| test parameters  | cpufreq_governor=performance                                             |
|                  | ipc=pipe                                                                 |
|                  | mode=process                                                             |
|                  | nr_threads=1600%                                                         |
|                  | ucode=0xb00002e                                                          |
+------------------+--------------------------------------------------------------------------+
| testcase: change | reaim: reaim.std_dev_percent 11.4% undefined                             |
| test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
| test parameters  | cpufreq_governor=performance                                             |
|                  | nr_task=100%                                                             |
|                  | runtime=300s                                                             |
|                  | test=custom                                                              |
|                  | ucode=0x200004d                                                          |
+------------------+--------------------------------------------------------------------------+
| testcase: change | reaim: boot-time.boot 95.3% regression                                   |
| test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
| test parameters  | cpufreq_governor=performance                                             |
|                  | nr_task=100%                                                             |
|                  | runtime=300s                                                             |
|                  | test=alltests                                                            |
|                  | ucode=0x200004d                                                          |
+------------------+--------------------------------------------------------------------------+
| testcase: change | pft: pft.faults_per_sec_per_cpu -42.7% regression                        |
| test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
| test parameters  | cpufreq_governor=performance                                             |
|                  | nr_task=50%                                                              |
|                  | runtime=300s                                                             |
|                  | ucode=0x200004d                                                          |
+------------------+--------------------------------------------------------------------------+
| testcase: change | stream: stream.add_bandwidth_MBps -28.8% regression                      |
| test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
| test parameters  | array_size=50000000                                                      |
|                  | cpufreq_governor=performance                                             |
|                  | nr_threads=50%                                                           |
|                  | omp=true                                                                 |
+------------------+--------------------------------------------------------------------------+
| testcase: change | stream: stream.add_bandwidth_MBps -30.6% regression                      |
| test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
| test parameters  | array_size=10000000                                                      |
|                  | cpufreq_governor=performance                                             |
|                  | nr_threads=50%                                                           |
|                  | omp=true                                                                 |
+------------------+--------------------------------------------------------------------------+
| testcase: change | pft: pft.faults_per_sec_per_cpu -42.5% regression                        |
| test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
| test parameters  | cpufreq_governor=performance                                             |
|                  | nr_task=50%                                                              |
|                  | runtime=300s                                                             |
+------------------+--------------------------------------------------------------------------+
| testcase: change | reaim: reaim.child_systime -1.4% undefined                               |
| test machine     | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory   |
| test parameters  | cpufreq_governor=performance                                             |
|                  | iterations=30                                                            |
|                  | nr_task=1600%                                                            |
|                  | test=compute                                                             |
+------------------+--------------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.fifo.ops_per_sec 76.2% improvement                  |
| test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
| test parameters  | class=pipe                                                               |
|                  | cpufreq_governor=performance                                             |
|                  | nr_threads=100%                                                          |
|                  | testtime=1s                                                              |
+------------------+--------------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.tsearch.ops_per_sec -17.1% regression               |
| test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
| test parameters  | class=cpu                                                                |
|                  | cpufreq_governor=performance                                             |
|                  | nr_threads=100%                                                          |
|                  | testtime=1s                                                              |
+------------------+--------------------------------------------------------------------------+


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.2/50%/debian-x86_64-2018-04-03.cgz/300s/lkp-bdw-ep3/pft/0xb00002e

commit: 
  24d0c1d6e6 ("sched/fair: Do not migrate due to a sync wakeup on exit")
  2c83362734 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")

24d0c1d6e65f635b 2c83362734dad8e48ccc0710b5c 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          2:4          -50%            :4     dmesg.WARNING:stack_going_in_the_wrong_direction?ip=_cond_resched/0x
           :4           25%           1:4     dmesg.WARNING:stack_going_in_the_wrong_direction?ip=schedule_tail/0x
         %stddev     %change         %stddev
             \          |                \  
    250875           -41.4%     146900        pft.faults_per_sec_per_cpu
   7127548           -32.9%    4779214        pft.time.minor_page_faults
      3533           +14.5%       4047        pft.time.percent_of_cpu_this_job_got
     10444           +13.3%      11828        pft.time.system_time
    189.79           +84.8%     350.70 ±  5%  pft.time.user_time
    105380 ±  2%     +31.5%     138536 ±  5%  pft.time.voluntary_context_switches
   6180331 ±  9%     -60.3%    2451607 ± 34%  numa-numastat.node1.local_node
   6187616 ±  9%     -60.3%    2455998 ± 34%  numa-numastat.node1.numa_hit
     58.50            -9.8%      52.75        vmstat.cpu.id
     35.75           +14.0%      40.75        vmstat.procs.r
     59.07            -6.1       52.99        mpstat.cpu.idle%
     39.94            +5.5       45.45        mpstat.cpu.sys%
      0.93            +0.6        1.54 ±  5%  mpstat.cpu.usr%
    147799 ±  6%     +13.9%     168315 ±  5%  cpuidle.C3.usage
 1.532e+10 ±  3%     -11.9%   1.35e+10        cpuidle.C6.time
  15901515 ±  3%     -12.4%   13931328        cpuidle.C6.usage
 1.062e+08 ±  9%    +123.1%  2.369e+08 ± 14%  cpuidle.POLL.time
      2491 ±  8%     -28.0%       1793 ± 10%  slabinfo.eventpoll_epi.active_objs
      2491 ±  8%     -28.0%       1793 ± 10%  slabinfo.eventpoll_epi.num_objs
      4332 ±  7%     -27.9%       3125 ± 10%  slabinfo.eventpoll_pwq.active_objs
      4332 ±  7%     -27.9%       3125 ± 10%  slabinfo.eventpoll_pwq.num_objs
      4601 ±  2%     -16.0%       3866 ±  2%  slabinfo.mm_struct.active_objs
      4662 ±  2%     -15.7%       3930 ±  2%  slabinfo.mm_struct.num_objs
   5396205 ±  2%     +18.2%    6378310        meminfo.Active
   5333263 ±  2%     +18.4%    6316138        meminfo.Active(anon)
   2054416 ±  5%     +22.7%    2521118 ± 16%  meminfo.AnonHugePages
   5180518 ±  2%     +18.7%    6147856        meminfo.AnonPages
 4.652e+08 ±  2%     +16.4%  5.416e+08        meminfo.Committed_AS
   6863577           +13.8%    7807920        meminfo.Memused
      9381 ±  2%     +11.7%      10480 ±  7%  meminfo.PageTables
      1203           +15.0%       1383        turbostat.Avg_MHz
     43.02            +6.3       49.30        turbostat.Busy%
    147383 ±  6%     +14.1%     168181 ±  5%  turbostat.C3
  15900108 ±  3%     -12.4%   13931193        turbostat.C6
     56.94            -6.2       50.74        turbostat.C6%
     33.20           -47.1%      17.57 ±  4%  turbostat.CPU%c1
     23.69 ±  4%     +39.5%      33.05 ±  2%  turbostat.CPU%c6
    186.47            -9.2%     169.38        turbostat.PkgWatt
     18.43           -19.6%      14.82        turbostat.RAMWatt
    715561 ± 15%     +70.3%    1218740 ± 16%  numa-vmstat.node0.nr_active_anon
    701119 ± 15%     +69.9%    1190948 ± 15%  numa-vmstat.node0.nr_anon_pages
    544.25 ± 20%     +80.0%     979.50 ± 20%  numa-vmstat.node0.nr_anon_transparent_hugepages
      1240 ± 15%     +37.7%       1709 ± 16%  numa-vmstat.node0.nr_page_table_pages
    715526 ± 15%     +70.3%    1218728 ± 16%  numa-vmstat.node0.nr_zone_active_anon
    636795 ± 10%     -44.9%     350831 ± 53%  numa-vmstat.node1.nr_active_anon
    614709 ± 11%     -45.1%     337557 ± 54%  numa-vmstat.node1.nr_anon_pages
    636843 ± 11%     -44.9%     350830 ± 53%  numa-vmstat.node1.nr_zone_active_anon
   3520461 ± 10%     -41.3%    2066299 ± 34%  numa-vmstat.node1.numa_hit
   3380522 ± 10%     -42.9%    1929309 ± 37%  numa-vmstat.node1.numa_local
   2860563 ± 14%     +73.7%    4970140 ± 14%  numa-meminfo.node0.Active
   2829095 ± 14%     +74.6%    4938791 ± 14%  numa-meminfo.node0.Active(anon)
   1029787 ± 15%     +90.1%    1957511 ± 16%  numa-meminfo.node0.AnonHugePages
   2782901 ± 14%     +73.2%    4820952 ± 13%  numa-meminfo.node0.AnonPages
   3598217 ± 11%     +59.0%    5721834 ± 12%  numa-meminfo.node0.MemUsed
      4779 ± 14%     +39.3%       6658 ± 13%  numa-meminfo.node0.PageTables
   2711469 ± 11%     -48.3%    1401167 ± 53%  numa-meminfo.node1.Active
   2679996 ± 11%     -48.9%    1370343 ± 55%  numa-meminfo.node1.Active(anon)
   1104830 ± 11%     -52.3%     527490 ± 43%  numa-meminfo.node1.AnonHugePages
   2592998 ± 13%     -50.4%    1285649 ± 54%  numa-meminfo.node1.AnonPages
   3441628 ±  9%     -39.6%    2078804 ± 37%  numa-meminfo.node1.MemUsed
   1348502 ±  3%     +15.9%    1562690 ±  2%  proc-vmstat.nr_active_anon
   1313538 ±  3%     +16.0%    1523248 ±  2%  proc-vmstat.nr_anon_pages
    993.00 ±  7%     +20.4%       1195 ±  4%  proc-vmstat.nr_anon_transparent_hugepages
   1484488            -1.4%    1464054        proc-vmstat.nr_dirty_background_threshold
   2972608            -1.4%    2931689        proc-vmstat.nr_dirty_threshold
  14732846            -1.4%   14528187        proc-vmstat.nr_free_pages
      2334 ±  4%     +11.9%       2611 ±  3%  proc-vmstat.nr_page_table_pages
     39493            -2.2%      38606        proc-vmstat.nr_slab_unreclaimable
   1348499 ±  3%     +15.9%    1562687 ±  2%  proc-vmstat.nr_zone_active_anon
     11707 ± 19%     -79.6%       2390 ±105%  proc-vmstat.numa_hint_faults
      5736 ± 68%     -68.8%       1789 ±122%  proc-vmstat.numa_hint_faults_local
  12846700           -31.1%    8854558        proc-vmstat.numa_hit
    834.00 ± 15%     -55.0%     375.50 ± 29%  proc-vmstat.numa_huge_pte_updates
  12829442           -31.1%    8837365        proc-vmstat.numa_local
     29698 ± 16%     -71.4%       8480 ± 72%  proc-vmstat.numa_pages_migrated
    464744 ± 17%     -57.4%     197920 ± 31%  proc-vmstat.numa_pte_updates
 2.591e+09           -33.0%  1.736e+09        proc-vmstat.pgalloc_normal
   7958915           -29.7%    5591702        proc-vmstat.pgfault
 2.589e+09           -33.0%  1.735e+09        proc-vmstat.pgfree
     29698 ± 16%     -71.4%       8480 ± 72%  proc-vmstat.pgmigrate_success
   5041287           -33.0%    3378764        proc-vmstat.thp_deferred_split_page
   5044208           -33.0%    3379878        proc-vmstat.thp_fault_alloc
    495.50 ± 58%     -64.8%     174.50 ±  4%  interrupts.35:PCI-MSI.3145732-edge.eth0-TxRx-3
      3476 ± 10%     -14.1%       2986        interrupts.CPU1.CAL:Function_call_interrupts
     40310 ±  8%     -50.4%      20001 ± 37%  interrupts.CPU1.RES:Rescheduling_interrupts
      3458 ± 12%     -13.5%       2992 ±  2%  interrupts.CPU13.CAL:Function_call_interrupts
    495.50 ± 58%     -64.8%     174.50 ±  4%  interrupts.CPU14.35:PCI-MSI.3145732-edge.eth0-TxRx-3
    232.75 ± 37%    +199.9%     698.00 ± 59%  interrupts.CPU17.RES:Rescheduling_interrupts
    372.75 ± 61%    +226.2%       1215 ± 41%  interrupts.CPU19.RES:Rescheduling_interrupts
      3428 ± 10%     -12.0%       3016 ±  3%  interrupts.CPU2.CAL:Function_call_interrupts
     16318 ± 14%     -79.4%       3366 ± 18%  interrupts.CPU2.RES:Rescheduling_interrupts
    112.50 ± 39%    +573.1%     757.25 ± 42%  interrupts.CPU20.RES:Rescheduling_interrupts
    103.75 ± 37%   +1322.2%       1475 ± 34%  interrupts.CPU21.RES:Rescheduling_interrupts
      1046 ± 41%   +3220.0%      34735 ± 46%  interrupts.CPU22.RES:Rescheduling_interrupts
    485.25 ± 36%   +3116.5%      15608 ±125%  interrupts.CPU23.RES:Rescheduling_interrupts
    404.75 ± 48%     -81.6%      74.50 ± 55%  interrupts.CPU29.RES:Rescheduling_interrupts
     12888 ± 17%     -73.6%       3399 ± 57%  interrupts.CPU3.RES:Rescheduling_interrupts
    341.25 ± 47%     -78.7%      72.75 ± 77%  interrupts.CPU31.RES:Rescheduling_interrupts
    290.75 ± 29%     -65.9%      99.25 ± 99%  interrupts.CPU34.RES:Rescheduling_interrupts
      3520 ±  7%     -28.8%       2507 ± 30%  interrupts.CPU35.CAL:Function_call_interrupts
    238.75 ± 50%     -75.6%      58.25 ± 35%  interrupts.CPU35.RES:Rescheduling_interrupts
    285.50 ± 66%     -87.3%      36.25 ± 70%  interrupts.CPU36.RES:Rescheduling_interrupts
      3520 ±  9%     -22.8%       2716 ± 16%  interrupts.CPU37.CAL:Function_call_interrupts
    303.00 ± 55%     -81.2%      57.00 ±101%  interrupts.CPU38.RES:Rescheduling_interrupts
    261.75 ± 68%     -83.2%      44.00 ± 81%  interrupts.CPU39.RES:Rescheduling_interrupts
      9633 ±  7%     -79.9%       1935 ± 41%  interrupts.CPU4.RES:Rescheduling_interrupts
    169.75 ± 47%     -71.4%      48.50 ± 67%  interrupts.CPU41.RES:Rescheduling_interrupts
    230.25 ± 52%     -73.4%      61.25 ± 92%  interrupts.CPU42.RES:Rescheduling_interrupts
      3426 ± 10%     -11.4%       3036        interrupts.CPU6.CAL:Function_call_interrupts
      4643 ± 20%     -77.7%       1036 ± 13%  interrupts.CPU6.RES:Rescheduling_interrupts
    237.75 ± 49%     -79.7%      48.25 ± 82%  interrupts.CPU66.RES:Rescheduling_interrupts
    231.00 ± 64%     -89.0%      25.50 ± 28%  interrupts.CPU69.RES:Rescheduling_interrupts
      3432 ± 10%     -15.0%       2918 ±  4%  interrupts.CPU7.CAL:Function_call_interrupts
      4134 ± 13%     -64.2%       1481 ± 43%  interrupts.CPU7.RES:Rescheduling_interrupts
     96.75 ± 51%     -80.4%      19.00 ± 46%  interrupts.CPU75.RES:Rescheduling_interrupts
      3453 ± 11%     -12.7%       3015        interrupts.CPU8.CAL:Function_call_interrupts
     18.33 ±  5%     -13.3        5.08 ±  4%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
     19.33 ±  3%     -12.0        7.38 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     19.70 ±  3%     -11.9        7.76 ±  7%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     19.40 ±  3%     -11.9        7.50 ±  4%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     19.40 ±  3%     -11.9        7.50 ±  4%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     19.40 ±  3%     -11.9        7.50 ±  4%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      0.55 ±  4%      +0.1        0.64 ±  2%  perf-profile.calltrace.cycles-pp.clear_huge_page
      0.95            +0.1        1.06 ±  2%  perf-profile.calltrace.cycles-pp.___might_sleep.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      0.56 ±  4%      +0.3        0.84 ±  8%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.clear_page_erms.clear_huge_page
      0.62 ±  3%      +0.3        0.94 ±  8%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.clear_page_erms.clear_huge_page.do_huge_pmd_anonymous_page
      0.60 ±  4%      +0.3        0.93        perf-profile.calltrace.cycles-pp._cond_resched.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      0.63 ±  3%      +0.3        0.96 ±  8%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.clear_page_erms.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault
      1.17 ±  3%      +0.3        1.50 ±  2%  perf-profile.calltrace.cycles-pp.__free_pages_ok.release_pages.tlb_flush_mmu_free.unmap_page_range.unmap_vmas
      0.38 ± 57%      +0.4        0.76 ±  9%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.clear_page_erms
      1.24 ±  3%      +0.4        1.65        perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu_free.unmap_page_range.unmap_vmas.exit_mmap
      1.24 ±  2%      +0.4        1.66        perf-profile.calltrace.cycles-pp.tlb_flush_mmu_free.unmap_page_range.unmap_vmas.exit_mmap.mmput
      1.31 ±  2%      +0.5        1.79        perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit
      1.31 ±  2%      +0.5        1.78        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit
      1.34 ±  2%      +0.5        1.81        perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__wake_up_parent.do_syscall_64
      1.34 ±  2%      +0.5        1.81        perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__wake_up_parent
      1.35 ±  3%      +0.5        1.83        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.35 ±  3%      +0.5        1.83        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.34 ±  3%      +0.5        1.82        perf-profile.calltrace.cycles-pp.__wake_up_parent.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.34 ±  3%      +0.5        1.82        perf-profile.calltrace.cycles-pp.do_group_exit.__wake_up_parent.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.34 ±  3%      +0.5        1.82        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__wake_up_parent.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.6        0.57 ±  2%  perf-profile.calltrace.cycles-pp.___might_sleep
      0.00            +0.6        0.64 ± 10%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      0.00            +0.7        0.67 ± 10%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      0.00            +0.7        0.68 ± 10%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.80 ±  2%      +0.7        2.54 ±  2%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
      1.80 ±  2%      +0.7        2.54 ±  2%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      0.00            +0.9        0.88 ±  5%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.do_huge_pmd_anonymous_page
      0.00            +0.9        0.90 ±  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.do_huge_pmd_anonymous_page.__handle_mm_fault
      0.66 ± 62%      +1.2        1.88 ± 34%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
     69.36            +8.7       78.09        perf-profile.calltrace.cycles-pp.clear_page_erms.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
     72.75           +10.1       82.87        perf-profile.calltrace.cycles-pp.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
     74.99           +10.9       85.91        perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     74.91           +10.9       85.84        perf-profile.calltrace.cycles-pp.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
     75.01           +10.9       85.95        perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     75.10           +11.0       86.06        perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
     75.10           +11.0       86.06        perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
     75.11           +11.0       86.08        perf-profile.calltrace.cycles-pp.page_fault
    103.91           -34.9%      67.64 ±  3%  perf-stat.i.MPKI
 1.482e+09           +13.6%  1.684e+09 ±  5%  perf-stat.i.branch-instructions
   9762787           +14.7%   11201300 ±  2%  perf-stat.i.branch-misses
      7.38 ±  2%      +1.0        8.41        perf-stat.i.cache-miss-rate%
  45987730 ±  3%     -19.9%   36831773 ±  2%  perf-stat.i.cache-misses
 6.184e+08           -29.2%  4.378e+08        perf-stat.i.cache-references
 1.035e+11           +16.0%    1.2e+11        perf-stat.i.cpu-cycles
     35.49 ±  2%     -24.7%      26.73 ±  8%  perf-stat.i.cpu-migrations
   1053398           +19.7%    1260826        perf-stat.i.dTLB-load-misses
 1.728e+09           +15.0%  1.987e+09 ±  5%  perf-stat.i.dTLB-loads
      0.04 ±  3%      +0.0        0.04 ±  2%  perf-stat.i.dTLB-store-miss-rate%
    592773           -13.0%     515699        perf-stat.i.dTLB-store-misses
  1.69e+09           -22.2%  1.314e+09        perf-stat.i.dTLB-stores
     49.34 ±  2%     +24.3       73.67        perf-stat.i.iTLB-load-miss-rate%
    462159           +19.0%     550118        perf-stat.i.iTLB-load-misses
    481420 ±  3%     -56.1%     211333 ±  3%  perf-stat.i.iTLB-loads
  6.07e+09           +11.9%  6.793e+09 ±  4%  perf-stat.i.instructions
     13676            -8.6%      12500 ±  4%  perf-stat.i.instructions-per-iTLB-miss
     25994           -29.2%      18413        perf-stat.i.minor-faults
     12.62 ±  5%      -3.1        9.52 ± 10%  perf-stat.i.node-load-miss-rate%
    813594 ±  8%     -45.3%     445188 ± 10%  perf-stat.i.node-load-misses
   6036541 ±  3%     -29.9%    4234277        perf-stat.i.node-loads
      2.63 ± 15%      -2.0        0.66 ± 32%  perf-stat.i.node-store-miss-rate%
    488029 ± 37%     -68.5%     153814 ± 34%  perf-stat.i.node-store-misses
  23876394            -3.5%   23039440        perf-stat.i.node-stores
     25995           -29.2%      18414        perf-stat.i.page-faults
    101.89           -36.6%      64.58 ±  3%  perf-stat.overall.MPKI
      7.44            +1.0        8.41        perf-stat.overall.cache-miss-rate%
      2251           +44.8%       3258        perf-stat.overall.cycles-between-cache-misses
      0.04            +0.0        0.04 ±  2%  perf-stat.overall.dTLB-store-miss-rate%
     48.98 ±  2%     +23.3       72.25        perf-stat.overall.iTLB-load-miss-rate%
     11.86 ±  4%      -2.3        9.51 ± 10%  perf-stat.overall.node-load-miss-rate%
      1.99 ± 36%      -1.3        0.66 ± 33%  perf-stat.overall.node-store-miss-rate%
 1.477e+09           +13.6%  1.677e+09 ±  5%  perf-stat.ps.branch-instructions
   9696637           +14.7%   11121372 ±  2%  perf-stat.ps.branch-misses
  45824683 ±  3%     -19.9%   36699436 ±  2%  perf-stat.ps.cache-misses
 6.162e+08           -29.2%  4.362e+08        perf-stat.ps.cache-references
 1.031e+11           +16.0%  1.195e+11        perf-stat.ps.cpu-cycles
     35.37 ±  2%     -24.7%      26.64 ±  8%  perf-stat.ps.cpu-migrations
   1049561           +19.7%    1256350        perf-stat.ps.dTLB-load-misses
 1.721e+09           +15.0%  1.979e+09 ±  5%  perf-stat.ps.dTLB-loads
    590625           -13.0%     513863        perf-stat.ps.dTLB-store-misses
 1.684e+09           -22.2%   1.31e+09        perf-stat.ps.dTLB-stores
    460416           +19.1%     548246        perf-stat.ps.iTLB-load-misses
    479698 ±  3%     -56.1%     210661 ±  3%  perf-stat.ps.iTLB-loads
 6.047e+09           +11.9%  6.766e+09 ±  4%  perf-stat.ps.instructions
     25909           -29.2%      18350        perf-stat.ps.minor-faults
    810622 ±  7%     -45.3%     443783 ± 10%  perf-stat.ps.node-load-misses
   6015816 ±  3%     -29.9%    4219461        perf-stat.ps.node-loads
    486412 ± 37%     -68.5%     153252 ± 33%  perf-stat.ps.node-store-misses
  23792569            -3.5%   22959184        perf-stat.ps.node-stores
     25909           -29.2%      18349        perf-stat.ps.page-faults
 1.843e+12           +10.8%  2.043e+12 ±  4%  perf-stat.total.instructions
     40.36 ±173%    -100.0%       0.00 ±  5%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
     49736           +54.0%      76581 ±  9%  sched_debug.cfs_rq:/.exec_clock.avg
     81364 ±  8%     +74.4%     141938 ±  4%  sched_debug.cfs_rq:/.exec_clock.max
     10337 ± 14%    +266.4%      37876 ± 42%  sched_debug.cfs_rq:/.exec_clock.stddev
     11225 ± 10%     +25.9%      14136 ± 13%  sched_debug.cfs_rq:/.load.avg
     40.36 ±173%    -100.0%       0.00 ±  5%  sched_debug.cfs_rq:/.max_vruntime.stddev
   2067637           +61.7%    3344320 ±  9%  sched_debug.cfs_rq:/.min_vruntime.avg
   3187109 ±  6%     +93.2%    6157097 ±  4%  sched_debug.cfs_rq:/.min_vruntime.max
    402561 ± 13%    +311.1%    1655080 ± 42%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.42 ±  8%     +26.9%       0.53 ±  4%  sched_debug.cfs_rq:/.nr_running.avg
      1.53 ± 10%     +58.3%       2.42 ± 14%  sched_debug.cfs_rq:/.nr_spread_over.avg
      1.13 ± 24%     +63.7%       1.85 ± 19%  sched_debug.cfs_rq:/.nr_spread_over.stddev
      3.41 ± 33%     -57.7%       1.45 ±111%  sched_debug.cfs_rq:/.removed.load_avg.avg
    201.45           -58.0%      84.54 ±100%  sched_debug.cfs_rq:/.removed.load_avg.max
     25.53 ± 17%     -57.7%      10.80 ±103%  sched_debug.cfs_rq:/.removed.load_avg.stddev
    156.58 ± 33%     -57.5%      66.54 ±111%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
      9231           -57.8%       3894 ±100%  sched_debug.cfs_rq:/.removed.runnable_sum.max
      1170 ± 17%     -57.6%     496.50 ±103%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
      9.87 ±  2%     +27.1%      12.55 ±  6%  sched_debug.cfs_rq:/.runnable_load_avg.avg
     11202 ± 10%     +25.8%      14092 ± 13%  sched_debug.cfs_rq:/.runnable_weight.avg
   1717275 ± 24%    +109.4%    3595584 ± 21%  sched_debug.cfs_rq:/.spread0.max
   -313871          +414.7%   -1615516        sched_debug.cfs_rq:/.spread0.min
    402568 ± 13%    +311.1%    1655086 ± 42%  sched_debug.cfs_rq:/.spread0.stddev
    441.98 ±  8%     +19.5%     528.20 ±  7%  sched_debug.cfs_rq:/.util_avg.avg
    152317           +19.8%     182542        sched_debug.cpu.clock.avg
    152324           +19.8%     182549        sched_debug.cpu.clock.max
    152309           +19.8%     182535        sched_debug.cpu.clock.min
    152317           +19.8%     182542        sched_debug.cpu.clock_task.avg
    152324           +19.8%     182549        sched_debug.cpu.clock_task.max
    152309           +19.8%     182535        sched_debug.cpu.clock_task.min
      9.55            +7.5%      10.27 ±  4%  sched_debug.cpu.cpu_load[1].avg
     89744           +27.7%     114579        sched_debug.cpu.nr_load_updates.avg
    105109 ±  2%     +46.9%     154390 ±  7%  sched_debug.cpu.nr_load_updates.max
      7386 ± 13%    +245.2%      25495 ± 34%  sched_debug.cpu.nr_load_updates.stddev
      6176           +32.2%       8164 ±  4%  sched_debug.cpu.nr_switches.avg
     33527 ± 10%    +167.9%      89818 ± 23%  sched_debug.cpu.nr_switches.max
      5721 ±  7%    +107.1%      11850 ± 23%  sched_debug.cpu.nr_switches.stddev
      6.70 ± 19%    +122.0%      14.88 ± 40%  sched_debug.cpu.nr_uninterruptible.max
      2.62 ±  8%     +36.4%       3.58 ±  7%  sched_debug.cpu.nr_uninterruptible.stddev
      5865           +35.3%       7937 ±  4%  sched_debug.cpu.sched_count.avg
     45223 ± 23%    +103.6%      92061 ± 23%  sched_debug.cpu.sched_count.max
      6871 ± 10%     +83.8%      12631 ± 20%  sched_debug.cpu.sched_count.stddev
      2580 ±  2%     +33.7%       3449 ±  6%  sched_debug.cpu.sched_goidle.avg
     15649 ±  9%    +169.3%      42137 ± 30%  sched_debug.cpu.sched_goidle.max
      2697 ±  7%    +106.9%       5582 ± 28%  sched_debug.cpu.sched_goidle.stddev
      2494 ±  2%     +40.6%       3507 ±  4%  sched_debug.cpu.ttwu_count.avg
     20565 ± 18%    +174.4%      56434 ± 39%  sched_debug.cpu.ttwu_count.max
      3385 ±  9%    +112.6%       7196 ± 31%  sched_debug.cpu.ttwu_count.stddev
    856.38           +60.8%       1377 ±  3%  sched_debug.cpu.ttwu_local.avg
      2955 ± 13%    +147.0%       7301 ± 28%  sched_debug.cpu.ttwu_local.max
    550.87 ±  4%     +61.0%     886.64 ± 19%  sched_debug.cpu.ttwu_local.stddev
    152311           +19.8%     182536        sched_debug.cpu_clk
    152311           +19.8%     182536        sched_debug.ktime
    155210           +19.5%     185434        sched_debug.sched_clk
     30209 ±  6%     -41.7%      17601 ± 15%  softirqs.CPU1.SCHED
     94445 ±  9%     +28.2%     121092 ±  5%  softirqs.CPU1.TIMER
      8720 ± 16%     -30.8%       6038 ± 15%  softirqs.CPU10.SCHED
     91259 ±  7%     +45.3%     132597 ± 12%  softirqs.CPU10.TIMER
      9600 ± 10%     -37.6%       5988 ± 24%  softirqs.CPU11.SCHED
     93321 ±  6%     +40.7%     131338 ± 14%  softirqs.CPU11.TIMER
     93395 ±  8%     +39.0%     129851 ± 12%  softirqs.CPU12.TIMER
      9114 ± 11%     -34.2%       6001 ± 13%  softirqs.CPU13.SCHED
     92702 ±  8%     +44.7%     134175 ± 10%  softirqs.CPU13.TIMER
     98912 ±  8%     +31.0%     129598 ± 11%  softirqs.CPU14.TIMER
     91279 ±  7%     +35.5%     123650 ±  8%  softirqs.CPU15.TIMER
      3119 ±  6%    +152.6%       7878 ± 73%  softirqs.CPU16.RCU
     94832 ±  7%     +55.9%     147813 ±  9%  softirqs.CPU16.TIMER
     94741 ±  7%     +38.2%     130953 ± 13%  softirqs.CPU17.TIMER
     89710 ±  7%     +48.8%     133512 ± 12%  softirqs.CPU18.TIMER
     95107 ±  7%     +41.2%     134251 ± 11%  softirqs.CPU19.TIMER
     17569 ±  3%     -51.5%       8516 ± 17%  softirqs.CPU2.SCHED
    101951 ±  8%     +34.6%     137209 ±  7%  softirqs.CPU2.TIMER
     95682 ±  5%     +43.4%     137183 ±  8%  softirqs.CPU20.TIMER
     90560 ± 18%     +47.5%     133620 ± 10%  softirqs.CPU21.TIMER
      8858 ±  9%    +171.4%      24041 ± 37%  softirqs.CPU22.SCHED
     82796 ± 11%     -32.9%      55594 ± 22%  softirqs.CPU22.TIMER
     92560 ±  8%     -30.4%      64376 ± 14%  softirqs.CPU23.TIMER
     89890 ± 11%     -31.9%      61223 ± 18%  softirqs.CPU25.TIMER
     82728 ± 11%     -28.7%      58954 ± 21%  softirqs.CPU26.TIMER
      7585 ±  5%     +65.0%      12517 ± 50%  softirqs.CPU27.SCHED
     87967 ±  7%     -22.7%      68014 ± 20%  softirqs.CPU29.TIMER
      3847 ± 11%     +78.0%       6847 ± 48%  softirqs.CPU3.RCU
     15118 ±  6%     -51.1%       7400 ± 19%  softirqs.CPU3.SCHED
     99241 ±  8%     +33.0%     131954 ± 13%  softirqs.CPU3.TIMER
      5273 ± 70%     -52.9%       2481 ± 13%  softirqs.CPU30.RCU
     86738 ±  4%     -26.9%      63403 ± 20%  softirqs.CPU30.TIMER
     87717 ±  4%     -19.7%      70426 ± 11%  softirqs.CPU33.TIMER
     91009 ±  8%     -26.6%      66826 ± 30%  softirqs.CPU37.TIMER
     91238 ±  6%     -34.6%      59629 ± 35%  softirqs.CPU39.TIMER
     13630 ±  6%     -48.8%       6975 ± 14%  softirqs.CPU4.SCHED
     98325 ±  9%     +36.5%     134220 ± 10%  softirqs.CPU4.TIMER
      7882 ± 58%     -37.7%       4910 ± 76%  softirqs.CPU40.RCU
     91734 ±  7%     -41.1%      54044 ± 24%  softirqs.CPU40.TIMER
      7837 ± 57%     -64.0%       2820 ± 18%  softirqs.CPU42.RCU
     90928 ±  7%     -29.9%      63731 ± 23%  softirqs.CPU42.TIMER
     92238 ±  5%     -36.2%      58803 ± 27%  softirqs.CPU43.TIMER
      9727 ±  9%     -39.0%       5931 ± 17%  softirqs.CPU45.SCHED
     95857 ±  5%     +35.5%     129846 ± 10%  softirqs.CPU45.TIMER
      9358 ±  8%     -36.5%       5941 ± 21%  softirqs.CPU46.SCHED
     92711 ± 10%     +43.7%     133268 ± 10%  softirqs.CPU46.TIMER
      9920 ±  9%     -37.9%       6161 ± 22%  softirqs.CPU47.SCHED
     97785 ±  9%     +36.4%     133427 ±  9%  softirqs.CPU47.TIMER
      9119 ± 13%     -31.4%       6255 ± 18%  softirqs.CPU48.SCHED
     96161 ± 12%     +39.8%     134472 ±  9%  softirqs.CPU48.TIMER
     95489 ± 12%     +36.7%     130507 ± 10%  softirqs.CPU49.TIMER
     95172 ±  8%     +34.9%     128353 ±  5%  softirqs.CPU5.TIMER
     95302 ±  9%     +39.9%     133360 ± 11%  softirqs.CPU50.TIMER
     93574 ± 13%     +47.3%     137842 ±  8%  softirqs.CPU51.TIMER
      3339 ±  6%    +102.3%       6754 ± 50%  softirqs.CPU52.RCU
     96278 ±  5%     +35.0%     129973 ± 10%  softirqs.CPU53.TIMER
      3243 ± 11%     +95.5%       6339 ± 60%  softirqs.CPU54.RCU
      7818 ± 22%     -35.3%       5060 ± 11%  softirqs.CPU54.SCHED
     97165 ±  4%     +35.0%     131219 ± 12%  softirqs.CPU54.TIMER
     99534 ±  8%     +33.4%     132761 ± 11%  softirqs.CPU55.TIMER
     93342 ±  6%     +37.6%     128456 ± 13%  softirqs.CPU56.TIMER
      3104 ±  5%    +176.2%       8574 ± 51%  softirqs.CPU57.RCU
      8598 ± 19%     -43.3%       4871 ± 17%  softirqs.CPU57.SCHED
     96358 ±  5%     +34.3%     129405 ± 14%  softirqs.CPU57.TIMER
     90709 ±  6%     +45.1%     131643 ± 10%  softirqs.CPU58.TIMER
     94807 ±  6%     +31.0%     124229 ± 10%  softirqs.CPU59.TIMER
     10284 ±  8%     -34.9%       6694 ± 12%  softirqs.CPU6.SCHED
     94038 ± 11%     +44.9%     136217 ± 10%  softirqs.CPU6.TIMER
     95144 ±  5%     +37.2%     130541 ± 13%  softirqs.CPU60.TIMER
     95772 ±  8%     +33.9%     128220 ± 13%  softirqs.CPU61.TIMER
     96450 ±  8%     +34.0%     129241 ± 13%  softirqs.CPU62.TIMER
      2935 ±  8%    +104.4%       5999 ± 58%  softirqs.CPU63.RCU
     93061 ±  7%     +43.2%     133226 ±  9%  softirqs.CPU63.TIMER
     95785 ± 10%     +38.0%     132228 ± 10%  softirqs.CPU64.TIMER
     87927 ± 20%     +51.6%     133269 ±  8%  softirqs.CPU65.TIMER
     87499 ± 10%     -38.1%      54161 ± 26%  softirqs.CPU66.TIMER
     91805 ±  9%     -29.9%      64360 ± 25%  softirqs.CPU67.TIMER
     92714 ±  8%     -29.6%      65310 ± 23%  softirqs.CPU68.TIMER
     87340 ±  7%     -31.8%      59600 ± 28%  softirqs.CPU69.TIMER
      3716 ±  5%     +86.8%       6942 ± 61%  softirqs.CPU7.RCU
     10113 ± 11%     -41.6%       5907 ±  9%  softirqs.CPU7.SCHED
     92161 ±  9%     +43.3%     132028 ± 11%  softirqs.CPU7.TIMER
     91161 ± 17%     -27.6%      66000 ± 24%  softirqs.CPU73.TIMER
     85233 ±  9%     -19.1%      68973 ± 14%  softirqs.CPU75.TIMER
     94828 ±  2%     -32.7%      63848 ± 22%  softirqs.CPU77.TIMER
     89287 ±  7%     -29.9%      62592 ± 27%  softirqs.CPU78.TIMER
     90874 ±  9%     -24.0%      69092 ± 24%  softirqs.CPU80.TIMER
      7403 ± 58%     -68.8%       2307 ± 12%  softirqs.CPU82.RCU
     90855 ± 10%     -27.9%      65495 ± 25%  softirqs.CPU82.TIMER
     90678 ± 11%     -28.8%      64580 ± 12%  softirqs.CPU84.TIMER
     90188 ± 15%     -21.7%      70622 ±  3%  softirqs.CPU85.TIMER
      7438 ± 57%     -37.8%       4626 ± 74%  softirqs.CPU86.RCU
     89728 ± 12%     -35.7%      57665 ± 17%  softirqs.CPU86.TIMER
      5097 ± 72%     -52.3%       2430 ±  5%  softirqs.CPU87.RCU
     88927 ±  9%     -34.6%      58201 ± 25%  softirqs.CPU87.TIMER
      3488 ±  8%     +98.4%       6921 ± 45%  softirqs.CPU9.RCU
     88673 ± 12%     +43.7%     127443 ± 12%  softirqs.CPU9.TIMER


                                                                                
                             pft.faults_per_sec_per_cpu                         
                                                                                
  300000 +-+----------------------------------------------------------------+   
         |                                                                  |   
  250000 +-+.++.+.+.+.++.+   +.+.++.+.+.+.+                                 |   
         |               :   :                                              |   
         |               :   :                                              |   
  200000 +-+              :  :                                              |   
         |                : :                                               |   
  150000 O-O  O O O O OO O:O:O O OO O O   OO O O O OO O O O O OO O O O OO O O   
         |                : :                                               |   
  100000 +-+              : :                                               |   
         |                : :                                               |   
         |                 ::                                               |   
   50000 +-+               :                                                |   
         |                 :                                                |   
       0 +-+-O--------------------------O-----------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-bdw-ep4: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory


***************************************************************************************************
lkp-hsw-ep2: 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_job/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.2/3000/100%/debian-x86_64-2018-04-03.cgz/300s/lkp-hsw-ep2/custom/reaim/0x3d

commit: 
  24d0c1d6e6 ("sched/fair: Do not migrate due to a sync wakeup on exit")
  2c83362734 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")

24d0c1d6e65f635b 2c83362734dad8e48ccc0710b5c 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.WARNING:stack_going_in_the_wrong_direction?ip=sched_slice/0x
         %stddev     %change         %stddev
             \          |                \  
    389.13            -2.7%     378.77        reaim.child_systime
    540687            +1.3%     547673        reaim.jobs_per_min
      7509            +1.3%       7606        reaim.jobs_per_min_child
    544939            +1.2%     551502        reaim.max_jobs_per_min
     23.92            -1.3%      23.62        reaim.parent_time
      0.68 ±  2%     +21.6%       0.83        reaim.std_dev_percent
      0.16 ±  2%     +20.1%       0.19        reaim.std_dev_time
    311.40            -1.2%     307.71        reaim.time.elapsed_time
    311.40            -1.2%     307.71        reaim.time.elapsed_time.max
   5096562           -16.0%    4280492        reaim.time.involuntary_context_switches
      4694            -2.7%       4569        reaim.time.system_time
  21203641            +2.1%   21642097        reaim.time.voluntary_context_switches
    169548            +1.9%     172739        vmstat.system.cs
 5.805e+08           +14.5%  6.645e+08        cpuidle.C1E.time
   5649746           +14.9%    6493131        cpuidle.C1E.usage
    114430 ±  4%      -6.4%     107089        softirqs.CPU33.TIMER
    116034            -8.8%     105833 ±  2%  softirqs.CPU35.TIMER
     13118           +13.4%      14873        meminfo.PageTables
     49287 ±  5%     +21.9%      60093 ±  7%  meminfo.Shmem
      7913           +12.1%       8874        meminfo.max_used_kB
      7969 ± 42%    +115.6%      17180 ±  5%  numa-meminfo.node0.Inactive(anon)
     12094 ± 15%     +35.1%      16333 ±  3%  numa-meminfo.node0.Mapped
     10024 ± 34%     -82.7%       1739 ± 48%  numa-meminfo.node1.Inactive(anon)
      6709 ±  5%     +30.3%       8745 ±  8%  numa-meminfo.node1.PageTables
   5649886           +14.9%    6493215        turbostat.C1E
      2.58            +0.4        2.98        turbostat.C1E%
      0.29           -58.6%       0.12        turbostat.CPU%c3
      2.18 ±  2%     +10.3%       2.41 ±  2%  turbostat.Pkg%pc2
     13.25            -3.8%      12.76        turbostat.RAMWatt
      1993 ± 42%    +115.1%       4288 ±  4%  numa-vmstat.node0.nr_inactive_anon
      3125 ± 15%     +32.9%       4154 ±  2%  numa-vmstat.node0.nr_mapped
      1993 ± 42%    +115.1%       4288 ±  4%  numa-vmstat.node0.nr_zone_inactive_anon
      2498 ± 34%     -82.9%     426.75 ± 47%  numa-vmstat.node1.nr_inactive_anon
      1670 ±  5%     +31.9%       2202 ±  9%  numa-vmstat.node1.nr_page_table_pages
      2498 ± 34%     -82.9%     426.75 ± 47%  numa-vmstat.node1.nr_zone_inactive_anon
      2027 ±  2%     -21.6%       1590 ±  8%  slabinfo.eventpoll_epi.active_objs
      2027 ±  2%     -21.6%       1590 ±  8%  slabinfo.eventpoll_epi.num_objs
      3548 ±  2%     -22.0%       2769 ±  9%  slabinfo.eventpoll_pwq.active_objs
      3548 ±  2%     -22.0%       2769 ±  9%  slabinfo.eventpoll_pwq.num_objs
    662.25 ±  6%     -19.6%     532.50 ±  3%  slabinfo.file_lock_cache.active_objs
    662.25 ±  6%     -19.6%     532.50 ±  3%  slabinfo.file_lock_cache.num_objs
    933.75 ±  2%     -23.5%     714.00 ±  2%  slabinfo.names_cache.active_objs
    938.50 ±  2%     -23.9%     714.00 ±  2%  slabinfo.names_cache.num_objs
     93741            +3.5%      97068        proc-vmstat.nr_active_anon
     84153            +1.7%      85581        proc-vmstat.nr_anon_pages
    235621            +1.1%     238317        proc-vmstat.nr_file_pages
      4495            +5.3%       4732        proc-vmstat.nr_inactive_anon
     15557            +2.7%      15981        proc-vmstat.nr_kernel_stack
      6662            +5.9%       7053        proc-vmstat.nr_mapped
      3245           +13.7%       3692        proc-vmstat.nr_page_table_pages
     12326 ±  5%     +21.9%      15022 ±  7%  proc-vmstat.nr_shmem
     44278            -1.3%      43703        proc-vmstat.nr_slab_unreclaimable
     93741            +3.5%      97068        proc-vmstat.nr_zone_active_anon
      4495            +5.3%       4732        proc-vmstat.nr_zone_inactive_anon
    158417 ± 11%     -20.5%     125975 ±  3%  proc-vmstat.numa_pte_updates
      2340          -100.0%       0.00        sched_debug.cfs_rq:/.load.min
      0.74           -17.1%       0.61 ±  3%  sched_debug.cfs_rq:/.nr_running.avg
      0.17          -100.0%       0.00        sched_debug.cfs_rq:/.nr_running.min
      1.83 ± 22%    -100.0%       0.00        sched_debug.cfs_rq:/.runnable_load_avg.min
     25.56 ± 51%    +131.9%      59.28 ± 32%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
      2339          -100.0%       0.00        sched_debug.cfs_rq:/.runnable_weight.min
    741.21           -19.7%     595.16 ±  2%  sched_debug.cfs_rq:/.util_avg.avg
    144.54 ± 16%     +38.7%     200.54 ± 27%  sched_debug.cpu.cpu_load[2].max
     15.92 ± 13%     +33.3%      21.22 ± 18%  sched_debug.cpu.cpu_load[3].stddev
      3.08 ± 18%     -44.6%       1.71 ± 46%  sched_debug.cpu.cpu_load[4].min
     23537 ±  2%     -16.0%      19781 ±  4%  sched_debug.cpu.curr->pid.avg
      4236          -100.0%       0.00        sched_debug.cpu.curr->pid.min
      2340          -100.0%       0.00        sched_debug.cpu.load.min
      0.89 ±  2%     -15.7%       0.75 ±  5%  sched_debug.cpu.nr_running.avg
      0.17          -100.0%       0.00        sched_debug.cpu.nr_running.min
    128.17 ±  5%     +10.5%     141.67 ±  4%  sched_debug.cpu.nr_uninterruptible.max
   -143.46            -9.9%    -129.21        sched_debug.cpu.nr_uninterruptible.min
      7407 ±  7%     +16.7%       8643 ±  6%  sched_debug.cpu.sched_count.stddev
      0.00 ±  5%     -44.0%       0.00 ± 14%  sched_debug.rt_rq:/.rt_time.avg
      0.04 ±  7%     -42.0%       0.02 ± 57%  sched_debug.rt_rq:/.rt_time.max
      0.00 ±  8%     -36.3%       0.00 ± 40%  sched_debug.rt_rq:/.rt_time.stddev
 2.198e+10            +1.1%  2.223e+10        perf-stat.i.branch-instructions
      2.02            -0.0        1.99        perf-stat.i.branch-miss-rate%
      2.44            -0.5        1.89        perf-stat.i.cache-miss-rate%
  84674052           -20.1%   67690215        perf-stat.i.cache-misses
 3.972e+09            +1.0%  4.013e+09        perf-stat.i.cache-references
    171277            +1.8%     174409        perf-stat.i.context-switches
      4613 ±  3%     +20.8%       5574 ±  3%  perf-stat.i.cycles-between-cache-misses
 1.248e+10            +1.3%  1.264e+10        perf-stat.i.dTLB-loads
  10489856 ±  2%      -6.8%    9775230 ±  2%  perf-stat.i.dTLB-store-misses
     61.17            -0.5       60.69        perf-stat.i.iTLB-load-miss-rate%
  1.04e+11            +1.1%  1.052e+11        perf-stat.i.instructions
     79.97            -1.9       78.03        perf-stat.i.node-load-miss-rate%
  54374000           -26.7%   39866795        perf-stat.i.node-load-misses
   5262627            -5.0%    4998608        perf-stat.i.node-loads
     47.13            -1.5       45.63        perf-stat.i.node-store-miss-rate%
  11256870           -13.1%    9780571        perf-stat.i.node-store-misses
  12952272            -6.5%   12106552        perf-stat.i.node-stores
      2.13            -0.4        1.69        perf-stat.overall.cache-miss-rate%
      1806           +25.2%       2262        perf-stat.overall.cycles-between-cache-misses
      0.08 ±  2%      -0.0        0.07 ±  3%  perf-stat.overall.dTLB-store-miss-rate%
     57.16            -0.4       56.72        perf-stat.overall.iTLB-load-miss-rate%
      6126            +2.6%       6282        perf-stat.overall.instructions-per-iTLB-miss
      0.68            +1.0%       0.69        perf-stat.overall.ipc
     91.17            -2.3       88.86        perf-stat.overall.node-load-miss-rate%
     46.50            -1.8       44.68        perf-stat.overall.node-store-miss-rate%
 2.191e+10            +1.1%  2.215e+10        perf-stat.ps.branch-instructions
  84380063           -20.1%   67448069        perf-stat.ps.cache-misses
 3.959e+09            +1.0%      4e+09        perf-stat.ps.cache-references
    170674            +1.8%     173772        perf-stat.ps.context-switches
 1.244e+10            +1.3%  1.259e+10        perf-stat.ps.dTLB-loads
  10454233 ±  2%      -6.8%    9740884 ±  2%  perf-stat.ps.dTLB-store-misses
 1.037e+11            +1.1%  1.048e+11        perf-stat.ps.instructions
  54182937           -26.7%   39721849        perf-stat.ps.node-load-misses
   5244550            -5.0%    4980894        perf-stat.ps.node-loads
  11218214           -13.1%    9746115        perf-stat.ps.node-store-misses
  12908974            -6.5%   12064855        perf-stat.ps.node-stores
    252.25 ± 40%     -32.0%     171.50 ±  4%  interrupts.40:PCI-MSI.1572869-edge.eth0-TxRx-5
     10875 ±  4%     -15.8%       9161 ±  8%  interrupts.CPU0.RES:Rescheduling_interrupts
      7594 ±  2%     -23.1%       5838 ±  6%  interrupts.CPU1.RES:Rescheduling_interrupts
      7393           -35.6%       4762 ±  3%  interrupts.CPU10.RES:Rescheduling_interrupts
      7422 ± 13%     -33.1%       4965 ±  8%  interrupts.CPU11.RES:Rescheduling_interrupts
      7029 ±  2%     -35.2%       4558 ±  2%  interrupts.CPU12.RES:Rescheduling_interrupts
      7067 ±  2%     -36.0%       4526        interrupts.CPU13.RES:Rescheduling_interrupts
      7201 ±  2%     -34.6%       4709        interrupts.CPU14.RES:Rescheduling_interrupts
      7104 ±  2%     -36.1%       4537 ±  5%  interrupts.CPU15.RES:Rescheduling_interrupts
      7208           -35.5%       4646 ±  4%  interrupts.CPU16.RES:Rescheduling_interrupts
      7127           -33.9%       4709 ±  9%  interrupts.CPU17.RES:Rescheduling_interrupts
      7395 ±  3%     -36.3%       4709 ±  2%  interrupts.CPU18.RES:Rescheduling_interrupts
      7005           -34.4%       4596 ±  3%  interrupts.CPU19.RES:Rescheduling_interrupts
      7853 ±  3%     -31.2%       5406 ±  8%  interrupts.CPU2.RES:Rescheduling_interrupts
      7205           -31.2%       4958 ±  8%  interrupts.CPU20.RES:Rescheduling_interrupts
      7137           -34.3%       4686 ±  3%  interrupts.CPU21.RES:Rescheduling_interrupts
      6968           -32.7%       4691        interrupts.CPU22.RES:Rescheduling_interrupts
      6914           -30.8%       4787 ±  3%  interrupts.CPU23.RES:Rescheduling_interrupts
      6922           -32.8%       4651 ±  2%  interrupts.CPU24.RES:Rescheduling_interrupts
      7184 ±  2%     -35.8%       4615 ±  2%  interrupts.CPU25.RES:Rescheduling_interrupts
      7341 ±  2%     -36.9%       4635 ±  3%  interrupts.CPU26.RES:Rescheduling_interrupts
      7230           -34.7%       4720 ±  4%  interrupts.CPU27.RES:Rescheduling_interrupts
      7205 ±  2%     -31.4%       4939 ±  7%  interrupts.CPU28.RES:Rescheduling_interrupts
      7064           -33.9%       4667 ±  3%  interrupts.CPU29.RES:Rescheduling_interrupts
      8138 ± 11%     -36.3%       5183 ±  6%  interrupts.CPU3.RES:Rescheduling_interrupts
      7003           -33.7%       4644 ±  2%  interrupts.CPU30.RES:Rescheduling_interrupts
      7189 ±  3%     -37.2%       4512 ±  3%  interrupts.CPU31.RES:Rescheduling_interrupts
      7236 ±  4%     -35.7%       4654        interrupts.CPU32.RES:Rescheduling_interrupts
      7071           -35.4%       4571        interrupts.CPU33.RES:Rescheduling_interrupts
      7111 ±  2%     -36.1%       4540        interrupts.CPU34.RES:Rescheduling_interrupts
      7196 ±  2%     -33.4%       4791 ±  4%  interrupts.CPU35.RES:Rescheduling_interrupts
      6758           -34.0%       4462        interrupts.CPU36.RES:Rescheduling_interrupts
      6853           -35.7%       4403 ±  4%  interrupts.CPU37.RES:Rescheduling_interrupts
      7193 ±  3%     -34.8%       4691        interrupts.CPU38.RES:Rescheduling_interrupts
      7197 ±  5%     -35.2%       4660 ±  3%  interrupts.CPU39.RES:Rescheduling_interrupts
      7753           -31.2%       5333 ±  7%  interrupts.CPU4.RES:Rescheduling_interrupts
      7031           -33.0%       4711 ±  3%  interrupts.CPU40.RES:Rescheduling_interrupts
      6888           -35.7%       4431        interrupts.CPU41.RES:Rescheduling_interrupts
      6914 ±  2%     -32.8%       4644 ±  3%  interrupts.CPU42.RES:Rescheduling_interrupts
      7186 ±  6%     -37.4%       4498        interrupts.CPU43.RES:Rescheduling_interrupts
      7018 ±  2%     -36.5%       4458 ±  2%  interrupts.CPU44.RES:Rescheduling_interrupts
      7083 ±  3%     -37.4%       4430 ±  2%  interrupts.CPU45.RES:Rescheduling_interrupts
      7044 ±  3%     -35.8%       4521 ±  2%  interrupts.CPU46.RES:Rescheduling_interrupts
      6916           -34.3%       4546        interrupts.CPU47.RES:Rescheduling_interrupts
      7012 ±  3%     -36.6%       4443 ±  2%  interrupts.CPU48.RES:Rescheduling_interrupts
      6966           -35.3%       4506 ±  3%  interrupts.CPU49.RES:Rescheduling_interrupts
    252.25 ± 40%     -32.0%     171.50 ±  4%  interrupts.CPU5.40:PCI-MSI.1572869-edge.eth0-TxRx-5
      7180 ±  2%     -33.7%       4760 ±  3%  interrupts.CPU5.RES:Rescheduling_interrupts
      6969           -35.1%       4523        interrupts.CPU50.RES:Rescheduling_interrupts
      7000           -34.5%       4587        interrupts.CPU51.RES:Rescheduling_interrupts
      7039 ±  2%     -34.0%       4643 ±  3%  interrupts.CPU52.RES:Rescheduling_interrupts
      6917           -35.8%       4443 ±  2%  interrupts.CPU53.RES:Rescheduling_interrupts
      7059 ±  2%     -37.2%       4430 ±  2%  interrupts.CPU54.RES:Rescheduling_interrupts
      6918           -33.8%       4580 ±  2%  interrupts.CPU55.RES:Rescheduling_interrupts
      7187           -35.1%       4662 ±  3%  interrupts.CPU56.RES:Rescheduling_interrupts
      6875           -33.5%       4568        interrupts.CPU57.RES:Rescheduling_interrupts
      7008           -34.7%       4574 ±  3%  interrupts.CPU58.RES:Rescheduling_interrupts
      6891           -32.0%       4683 ±  3%  interrupts.CPU59.RES:Rescheduling_interrupts
      7151 ±  4%     -33.2%       4780 ±  2%  interrupts.CPU6.RES:Rescheduling_interrupts
      6903           -30.9%       4767 ±  7%  interrupts.CPU60.RES:Rescheduling_interrupts
      7063           -33.4%       4706 ±  3%  interrupts.CPU61.RES:Rescheduling_interrupts
      6855 ±  3%     -33.9%       4532 ±  4%  interrupts.CPU62.RES:Rescheduling_interrupts
      7054           -35.1%       4578 ±  2%  interrupts.CPU63.RES:Rescheduling_interrupts
      7195 ±  2%     -37.0%       4531 ±  2%  interrupts.CPU64.RES:Rescheduling_interrupts
      7040 ±  2%     -35.3%       4556 ±  4%  interrupts.CPU65.RES:Rescheduling_interrupts
      6848           -33.7%       4538 ±  2%  interrupts.CPU66.RES:Rescheduling_interrupts
      7026 ±  2%     -35.4%       4538        interrupts.CPU67.RES:Rescheduling_interrupts
      7055           -34.2%       4645 ±  2%  interrupts.CPU68.RES:Rescheduling_interrupts
      6876           -33.4%       4576        interrupts.CPU69.RES:Rescheduling_interrupts
      7598 ±  5%     -35.5%       4898 ±  2%  interrupts.CPU7.RES:Rescheduling_interrupts
      7097 ±  2%     -35.9%       4548 ±  5%  interrupts.CPU70.RES:Rescheduling_interrupts
      6982           -35.7%       4488        interrupts.CPU71.RES:Rescheduling_interrupts
      7378 ±  2%     -36.2%       4705 ±  2%  interrupts.CPU8.RES:Rescheduling_interrupts
      7349           -34.9%       4784        interrupts.CPU9.RES:Rescheduling_interrupts
     83.75 ± 11%     +52.5%     127.75 ± 18%  interrupts.IWI:IRQ_work_interrupts
    516736           -34.1%     340755        interrupts.RES:Rescheduling_interrupts
     22.25            -1.3       20.95        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     22.22            -1.3       20.93        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.69 ±  5%      -1.1        2.60 ±  3%  perf-profile.calltrace.cycles-pp.search_binary_handler.do_execveat_common.sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.69 ±  5%      -1.1        2.59 ±  3%  perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.do_execveat_common.sys_execve.do_syscall_64
      1.71 ±  4%      -0.8        0.88 ±  4%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt
      1.64 ±  4%      -0.8        0.84 ±  4%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt
      1.02            -0.8        0.27 ±100%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.25 ±  3%      -0.6        0.60 ±  9%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      3.74 ±  2%      -0.5        3.20        perf-profile.calltrace.cycles-pp.copy_page_range.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.06 ± 13%      -0.5        1.56 ±  3%  perf-profile.calltrace.cycles-pp.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common.sys_execve
      1.86 ±  3%      -0.5        1.36 ±  2%  perf-profile.calltrace.cycles-pp._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.97 ± 12%      -0.5        1.51 ±  3%  perf-profile.calltrace.cycles-pp.mmput.flush_old_exec.load_elf_binary.search_binary_handler.do_execveat_common
      1.96 ± 12%      -0.5        1.50 ±  3%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.flush_old_exec.load_elf_binary.search_binary_handler
      1.70 ±  3%      -0.4        1.27 ±  2%  perf-profile.calltrace.cycles-pp.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.29 ±  7%      -0.4        0.87 ± 16%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      1.29 ±  7%      -0.4        0.87 ± 16%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
      4.20 ±  3%      -0.4        3.80 ±  3%  perf-profile.calltrace.cycles-pp.execve
      1.21 ±  6%      -0.4        0.85 ±  2%  perf-profile.calltrace.cycles-pp.setlocale
      3.81 ±  3%      -0.4        3.46 ±  3%  perf-profile.calltrace.cycles-pp.do_execveat_common.sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      3.84 ±  3%      -0.4        3.49 ±  3%  perf-profile.calltrace.cycles-pp.sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      3.84 ±  3%      -0.3        3.50 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve
      3.84 ±  3%      -0.3        3.50 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      0.73 ±  6%      -0.2        0.54        perf-profile.calltrace.cycles-pp._dl_addr
      0.91 ±  3%      -0.2        0.75 ±  4%  perf-profile.calltrace.cycles-pp.ret_from_fork
      0.89 ±  3%      -0.1        0.74 ±  4%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      0.68 ±  4%      -0.1        0.60 ±  5%  perf-profile.calltrace.cycles-pp.__pte_alloc.copy_page_range.copy_process._do_fork.do_syscall_64
      0.72 ±  3%      +0.1        0.79 ±  4%  perf-profile.calltrace.cycles-pp.read
      0.56 ±  7%      +0.1        0.66        perf-profile.calltrace.cycles-pp.do_brk_flags.sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
      0.54 ±  6%      +0.1        0.64 ±  4%  perf-profile.calltrace.cycles-pp.kmem_cache_free.unlink_anon_vmas.free_pgtables.exit_mmap.mmput
      0.91 ±  3%      +0.1        1.03        perf-profile.calltrace.cycles-pp.anon_vma_interval_tree_insert.anon_vma_clone.anon_vma_fork.copy_process._do_fork
      0.56 ±  3%      +0.1        0.68 ±  4%  perf-profile.calltrace.cycles-pp.close
      0.70 ±  6%      +0.1        0.82 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill
      0.71 ±  6%      +0.1        0.83 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.kill
      0.96 ±  6%      +0.1        1.11 ±  3%  perf-profile.calltrace.cycles-pp.kill
      0.83 ±  3%      +0.2        0.98 ±  2%  perf-profile.calltrace.cycles-pp.free_pages_and_swap_cache.tlb_flush_mmu_free.arch_tlb_finish_mmu.tlb_finish_mmu.exit_mmap
      0.94            +0.2        1.09 ±  2%  perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.05            +0.2        1.22 ±  2%  perf-profile.calltrace.cycles-pp.vfs_write.sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      0.54 ±  4%      +0.2        0.71 ±  3%  perf-profile.calltrace.cycles-pp.tlb_finish_mmu.unmap_region.do_munmap.sys_brk.do_syscall_64
      0.54 ±  4%      +0.2        0.71 ±  3%  perf-profile.calltrace.cycles-pp.arch_tlb_finish_mmu.tlb_finish_mmu.unmap_region.do_munmap.sys_brk
      1.12            +0.2        1.29 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      1.10            +0.2        1.27 ±  3%  perf-profile.calltrace.cycles-pp.sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
      1.13            +0.2        1.31 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
      0.71 ±  6%      +0.2        0.91 ±  3%  perf-profile.calltrace.cycles-pp.unmap_region.do_munmap.sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.75 ±  5%      +0.2        0.96 ±  3%  perf-profile.calltrace.cycles-pp.do_munmap.sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
      1.86 ±  2%      +0.2        2.08 ±  3%  perf-profile.calltrace.cycles-pp.anon_vma_clone.anon_vma_fork.copy_process._do_fork.do_syscall_64
      3.38 ±  3%      +0.3        3.65        perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.do_exit.do_group_exit
      1.72 ±  2%      +0.3        1.98        perf-profile.calltrace.cycles-pp.write
      1.93 ±  2%      +0.3        2.23        perf-profile.calltrace.cycles-pp.page_remove_rmap.unmap_page_range.unmap_vmas.exit_mmap.mmput
      0.28 ±100%      +0.3        0.60 ±  4%  perf-profile.calltrace.cycles-pp.do_notify_parent.do_exit.do_group_exit.__wake_up_parent.do_syscall_64
      2.17 ±  4%      +0.3        2.49 ±  2%  perf-profile.calltrace.cycles-pp.unlink_anon_vmas.free_pgtables.exit_mmap.mmput.do_exit
      0.26 ±100%      +0.3        0.59 ±  4%  perf-profile.calltrace.cycles-pp.__slab_free.kmem_cache_free.unlink_anon_vmas.free_pgtables.exit_mmap
      1.44 ±  6%      +0.3        1.78        perf-profile.calltrace.cycles-pp.sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
      1.47 ±  6%      +0.3        1.81        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
      2.81 ±  3%      +0.4        3.16        perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu_free.arch_tlb_finish_mmu.tlb_finish_mmu.exit_mmap
      1.48 ±  6%      +0.4        1.84        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.brk
      2.75            +0.4        3.10 ±  3%  perf-profile.calltrace.cycles-pp.anon_vma_fork.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.71 ±  4%      +0.4        2.09        perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.92 ±  2%      +0.4        6.30        perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit
      1.73 ±  4%      +0.4        2.11        perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      1.86 ±  4%      +0.4        2.25        perf-profile.calltrace.cycles-pp.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      5.77            +0.4        6.16        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit
      1.89 ±  4%      +0.4        2.28        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat
      0.13 ±173%      +0.4        0.53 ±  2%  perf-profile.calltrace.cycles-pp.remove_vma.exit_mmap.mmput.do_exit.do_group_exit
      1.89 ±  4%      +0.4        2.30        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat
      0.27 ±100%      +0.4        0.69        perf-profile.calltrace.cycles-pp.wait_consider_task.do_wait.kernel_wait4.SYSC_wait4.do_syscall_64
      1.97 ±  6%      +0.4        2.40        perf-profile.calltrace.cycles-pp.brk
      2.36 ±  4%      +0.5        2.81        perf-profile.calltrace.cycles-pp.creat
      0.13 ±173%      +0.5        0.60 ±  8%  perf-profile.calltrace.cycles-pp.down_write.anon_vma_fork.copy_process._do_fork.do_syscall_64
      0.67 ±  4%      +0.5        1.16 ±  4%  perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.do_wait.kernel_wait4.SYSC_wait4.do_syscall_64
      3.65 ±  2%      +0.5        4.15        perf-profile.calltrace.cycles-pp.tlb_flush_mmu_free.arch_tlb_finish_mmu.tlb_finish_mmu.exit_mmap.mmput
      0.13 ±173%      +0.5        0.63        perf-profile.calltrace.cycles-pp.release_task.wait_consider_task.do_wait.kernel_wait4.SYSC_wait4
      3.68 ±  2%      +0.5        4.20        perf-profile.calltrace.cycles-pp.tlb_finish_mmu.exit_mmap.mmput.do_exit.do_group_exit
      3.67 ±  2%      +0.5        4.19        perf-profile.calltrace.cycles-pp.arch_tlb_finish_mmu.tlb_finish_mmu.exit_mmap.mmput.do_exit
      1.44 ±  2%      +0.7        2.13        perf-profile.calltrace.cycles-pp.do_wait.kernel_wait4.SYSC_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.45 ±  2%      +0.7        2.15        perf-profile.calltrace.cycles-pp.kernel_wait4.SYSC_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait
      1.47 ±  2%      +0.7        2.17        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.wait
      1.46 ±  2%      +0.7        2.17        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait
      1.45 ±  2%      +0.7        2.16        perf-profile.calltrace.cycles-pp.SYSC_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait
      1.69 ±  2%      +0.7        2.43        perf-profile.calltrace.cycles-pp.wait
     11.98            +0.7       12.73        perf-profile.calltrace.cycles-pp.__libc_fork
      9.99 ±  2%      +0.8       10.77        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_fork
      9.99 ±  2%      +0.8       10.77        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      9.98 ±  2%      +0.8       10.77        perf-profile.calltrace.cycles-pp._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
      9.35            +0.8       10.18        perf-profile.calltrace.cycles-pp.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_fork
     27.80            +0.8       28.64        perf-profile.calltrace.cycles-pp.secondary_startup_64
      0.00            +0.9        0.86 ±  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_read_lock_slowpath.do_wait.kernel_wait4.SYSC_wait4
     27.36            +0.9       28.29        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     27.36            +0.9       28.29        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     27.35            +0.9       28.29        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     26.38            +1.0       27.42        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     13.81            +1.2       14.97        perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__wake_up_parent
     13.84            +1.2       15.00        perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__wake_up_parent.do_syscall_64
     15.92            +1.3       17.21        perf-profile.calltrace.cycles-pp.__wake_up_parent.do_syscall_64.entry_SYSCALL_64_after_hwframe
     15.92            +1.3       17.21        perf-profile.calltrace.cycles-pp.do_group_exit.__wake_up_parent.do_syscall_64.entry_SYSCALL_64_after_hwframe
     15.92            +1.3       17.21        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__wake_up_parent.do_syscall_64.entry_SYSCALL_64_after_hwframe
     24.25            +1.3       25.56        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary



***************************************************************************************************
lkp-bdw-ep4: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
=========================================================================================
array_size/compiler/cpufreq_governor/kconfig/nr_threads/omp/rootfs/tbox_group/testcase/ucode:
  10000000/gcc-7/performance/x86_64-rhel-7.2/50%/true/debian-x86_64-2018-04-03.cgz/lkp-bdw-ep4/stream/0xb00002e

commit: 
  24d0c1d6e6 ("sched/fair: Do not migrate due to a sync wakeup on exit")
  2c83362734 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")

24d0c1d6e65f635b 2c83362734dad8e48ccc0710b5c 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     70910 ±  3%     -32.0%      48230 ±  2%  stream.add_bandwidth_MBps
     76061           -39.0%      46386 ±  3%  stream.copy_bandwidth_MBps
     67937 ±  2%     -35.7%      43683 ±  2%  stream.scale_bandwidth_MBps
     52.42           +47.9%      77.52 ±  2%  stream.time.user_time
     74940           -33.4%      49941 ±  2%  stream.triad_bandwidth_MBps
     19.95 ±  9%      +3.4       23.35 ±  2%  mpstat.cpu.usr%
   3192799 ± 31%    +124.6%    7172184 ± 16%  cpuidle.C1E.time
     23982 ±  9%     +47.3%      35332 ± 28%  cpuidle.C1E.usage
     91178 ±  8%     +43.2%     130560 ±  4%  meminfo.AnonHugePages
    669014 ± 11%     -20.3%     533101        meminfo.max_used_kB
    643.25 ± 15%    +408.3%       3269 ±129%  softirqs.CPU16.RCU
     10783 ±  4%     +13.6%      12250 ± 10%  softirqs.CPU46.TIMER
     84.00            -4.2%      80.50        vmstat.cpu.id
     15.00 ±  8%     +23.3%      18.50 ±  2%  vmstat.cpu.us
      5888            +3.3%       6081        proc-vmstat.nr_mapped
    985.50 ±  2%      +7.6%       1060 ±  2%  proc-vmstat.nr_page_table_pages
     81826            +2.9%      84193        proc-vmstat.numa_hit
     64731            +3.6%      67074        proc-vmstat.numa_local
     20069 ± 12%     +60.3%      32178 ± 32%  turbostat.C1E
      1.15 ± 31%      +1.2        2.33 ± 17%  turbostat.C1E%
     34.77 ±  4%     -24.8%      26.13 ±  2%  turbostat.CPU%c1
     30.82 ± 16%     +25.9%      38.80 ±  9%  turbostat.CPU%c6
    289644 ±  6%     +17.8%     341157 ±  7%  turbostat.IRQ
     24.77 ±  6%     -10.6%      22.14 ±  2%  turbostat.RAMWatt
    672.00 ±  9%     +46.4%     984.00 ± 22%  slabinfo.dmaengine-unmap-16.active_objs
    672.00 ±  9%     +46.4%     984.00 ± 22%  slabinfo.dmaengine-unmap-16.num_objs
      2128 ±  9%     -25.9%       1576 ±  9%  slabinfo.eventpoll_epi.active_objs
      2128 ±  9%     -25.9%       1576 ±  9%  slabinfo.eventpoll_epi.num_objs
      3724 ±  9%     -25.9%       2758 ±  9%  slabinfo.eventpoll_pwq.active_objs
      3724 ±  9%     -25.9%       2758 ±  9%  slabinfo.eventpoll_pwq.num_objs
    760.00 ±  6%     -17.1%     630.00 ±  2%  slabinfo.file_lock_cache.active_objs
    760.00 ±  6%     -17.1%     630.00 ±  2%  slabinfo.file_lock_cache.num_objs
      7771 ± 60%    +112.3%      16501        numa-meminfo.node0.Inactive(anon)
      8035 ± 58%    +111.6%      17005 ±  2%  numa-meminfo.node0.Shmem
     93367 ±  2%     +12.8%     105291        numa-meminfo.node0.Slab
     21669            +8.8%      23569 ±  4%  numa-meminfo.node1.Active(file)
      9225 ± 51%     -94.7%     493.50 ± 62%  numa-meminfo.node1.Inactive(anon)
     31028 ±  6%     -10.9%      27656        numa-meminfo.node1.SReclaimable
     62893 ±  4%     -13.6%      54313 ±  7%  numa-meminfo.node1.SUnreclaim
      9625 ± 49%     -92.8%     690.00 ± 51%  numa-meminfo.node1.Shmem
     93922           -12.7%      81969 ±  5%  numa-meminfo.node1.Slab
      1942 ± 61%    +112.4%       4125        numa-vmstat.node0.nr_inactive_anon
      2007 ± 59%    +111.7%       4250 ±  2%  numa-vmstat.node0.nr_shmem
      1942 ± 61%    +112.4%       4125        numa-vmstat.node0.nr_zone_inactive_anon
      5390            +9.3%       5892 ±  4%  numa-vmstat.node1.nr_active_file
      2305 ± 51%     -94.7%     123.25 ± 62%  numa-vmstat.node1.nr_inactive_anon
      2404 ± 49%     -92.8%     172.00 ± 51%  numa-vmstat.node1.nr_shmem
     15666 ±  4%     -13.3%      13579 ±  7%  numa-vmstat.node1.nr_slab_unreclaimable
      5390            +9.3%       5892 ±  4%  numa-vmstat.node1.nr_zone_active_file
      2305 ± 51%     -94.7%     123.25 ± 62%  numa-vmstat.node1.nr_zone_inactive_anon
     11.33 ±120%     -11.3        0.00        perf-profile.calltrace.cycles-pp.__vfs_read.vfs_read.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.83 ±140%      -9.8        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
      9.83 ±140%      -9.8        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      9.83 ±140%      -9.8        0.00        perf-profile.calltrace.cycles-pp.read
      9.83 ±140%      -9.8        0.00        perf-profile.calltrace.cycles-pp.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      9.83 ±140%      -9.8        0.00        perf-profile.calltrace.cycles-pp.vfs_read.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     20.83 ±103%      -8.3       12.50 ±173%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     20.83 ±103%      -8.3       12.50 ±173%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     20.83 ±103%      -8.3       12.50 ±173%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     20.33 ±103%      -7.8       12.50 ±173%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     20.33 ±103%      -7.8       12.50 ±173%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
     12.83 ±108%      -0.3       12.50 ±173%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     12.83 ±108%      -0.3       12.50 ±173%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     27016 ±  6%     -12.9%      23538 ±  5%  sched_debug.cfs_rq:/.min_vruntime.max
      2243 ± 40%     -47.7%       1173 ± 35%  sched_debug.cfs_rq:/.min_vruntime.min
   -692.28          +943.1%      -7221        sched_debug.cfs_rq:/.spread0.avg
     19808 ± 11%     -47.9%      10321 ± 61%  sched_debug.cfs_rq:/.spread0.max
     -5010          +140.7%     -12059        sched_debug.cfs_rq:/.spread0.min
      2.18 ±  9%     -12.5%       1.91 ±  8%  sched_debug.cpu.cpu_load[0].avg
    160.00 ± 22%     -42.2%      92.50 ± 25%  sched_debug.cpu.cpu_load[4].max
     18.25 ± 22%     -37.0%      11.50 ± 14%  sched_debug.cpu.cpu_load[4].stddev
    282.50 ± 11%     -16.2%     236.72 ±  6%  sched_debug.cpu.curr->pid.avg
    642.46 ±  4%      -7.3%     595.31        sched_debug.cpu.curr->pid.stddev
    229.75           -19.0%     186.00 ±  9%  sched_debug.cpu.nr_switches.min
      1754 ± 11%     +30.2%       2284 ±  8%  sched_debug.cpu.nr_switches.stddev
      5.25 ± 15%    +128.6%      12.00 ± 20%  sched_debug.cpu.nr_uninterruptible.max
      2.38 ± 13%     +50.4%       3.57 ± 19%  sched_debug.cpu.nr_uninterruptible.stddev
      1.56 ±171%    +602.6%      10.93 ± 52%  sched_debug.cpu.ttwu_count.avg
      9.75 ±161%   +2407.7%     244.50 ± 51%  sched_debug.cpu.ttwu_count.max
      3.53 ±167%   +1270.9%      48.37 ± 50%  sched_debug.cpu.ttwu_count.stddev
      3.14 ± 12%     +41.1%       4.43 ± 17%  perf-stat.i.cpi
      0.32 ± 13%     -24.8%       0.24 ± 12%  perf-stat.i.ipc
     14784 ±  4%     -25.8%      10970 ± 16%  perf-stat.i.minor-faults
     37.43 ± 11%     -32.1        5.36 ± 27%  perf-stat.i.node-load-miss-rate%
   1700226 ± 23%     -86.3%     233356 ±  9%  perf-stat.i.node-load-misses
   2814735 ± 20%    +183.3%    7973698 ± 61%  perf-stat.i.node-loads
     36.40 ± 15%     -34.6        1.80 ± 79%  perf-stat.i.node-store-miss-rate%
   3649361 ± 35%     -95.2%     175235 ± 54%  perf-stat.i.node-store-misses
     14790 ±  4%     -25.8%      10971 ± 16%  perf-stat.i.page-faults
      3.14 ± 12%     +45.8%       4.58 ± 20%  perf-stat.overall.cpi
      0.32 ± 13%     -29.6%       0.23 ± 20%  perf-stat.overall.ipc
     37.43 ± 11%     -33.1        4.30 ± 57%  perf-stat.overall.node-load-miss-rate%
     36.40 ± 15%     -34.8        1.58 ± 99%  perf-stat.overall.node-store-miss-rate%
      7440 ±  4%     -15.4%       6296 ±  6%  perf-stat.ps.minor-faults
    855845 ± 24%     -83.9%     138073 ± 21%  perf-stat.ps.node-load-misses
   1416749 ± 20%    +257.9%    5070429 ± 69%  perf-stat.ps.node-loads
   1837180 ± 35%     -94.4%     103194 ± 56%  perf-stat.ps.node-store-misses
      7443 ±  4%     -15.4%       6297 ±  6%  perf-stat.ps.page-faults
      2645 ±  6%     +21.1%       3205 ±  8%  interrupts.CPU0.LOC:Local_timer_interrupts
      2659 ±  5%     +19.9%       3189 ±  8%  interrupts.CPU1.LOC:Local_timer_interrupts
      2690 ±  4%     +21.8%       3278 ± 10%  interrupts.CPU10.LOC:Local_timer_interrupts
      2649 ±  5%     +22.0%       3233 ±  8%  interrupts.CPU11.LOC:Local_timer_interrupts
      2665 ±  5%     +20.0%       3198 ±  8%  interrupts.CPU12.LOC:Local_timer_interrupts
      2697 ±  7%     +18.6%       3200 ±  8%  interrupts.CPU13.LOC:Local_timer_interrupts
      2668 ±  5%     +19.6%       3192 ±  8%  interrupts.CPU14.LOC:Local_timer_interrupts
      2656 ±  5%     +21.3%       3223 ±  7%  interrupts.CPU15.LOC:Local_timer_interrupts
      2652 ±  5%     +20.6%       3199 ±  8%  interrupts.CPU16.LOC:Local_timer_interrupts
      2680 ±  4%     +20.4%       3227 ±  8%  interrupts.CPU17.LOC:Local_timer_interrupts
      2653 ±  5%     +21.5%       3224 ±  9%  interrupts.CPU18.LOC:Local_timer_interrupts
      2685 ±  5%     +20.2%       3228 ±  8%  interrupts.CPU19.LOC:Local_timer_interrupts
      2676 ±  5%     +20.7%       3229 ±  9%  interrupts.CPU2.LOC:Local_timer_interrupts
      2665 ±  5%     +20.6%       3215 ±  9%  interrupts.CPU20.LOC:Local_timer_interrupts
      2681 ±  5%     +19.3%       3198 ±  8%  interrupts.CPU21.LOC:Local_timer_interrupts
      2653 ±  5%     +20.8%       3206 ±  8%  interrupts.CPU22.LOC:Local_timer_interrupts
      2643 ±  6%     +22.8%       3245 ±  8%  interrupts.CPU23.LOC:Local_timer_interrupts
      2654 ±  5%     +21.7%       3231 ±  7%  interrupts.CPU24.LOC:Local_timer_interrupts
      2649 ±  5%     +21.8%       3226 ±  7%  interrupts.CPU25.LOC:Local_timer_interrupts
      2707 ±  7%     +20.0%       3248 ±  7%  interrupts.CPU26.LOC:Local_timer_interrupts
      2701 ±  6%     +20.2%       3247 ±  8%  interrupts.CPU27.LOC:Local_timer_interrupts
      2701 ±  6%     +18.8%       3208 ±  8%  interrupts.CPU28.LOC:Local_timer_interrupts
      2702 ±  5%     +18.4%       3201 ±  8%  interrupts.CPU29.LOC:Local_timer_interrupts
      2688 ±  6%     +18.8%       3193 ±  8%  interrupts.CPU30.LOC:Local_timer_interrupts
      2641 ±  6%     +21.5%       3208 ±  8%  interrupts.CPU31.LOC:Local_timer_interrupts
      2666 ±  4%     +20.4%       3210 ±  8%  interrupts.CPU32.LOC:Local_timer_interrupts
      2639 ±  5%     +21.4%       3204 ±  8%  interrupts.CPU33.LOC:Local_timer_interrupts
      2656 ±  5%     +22.2%       3246 ±  8%  interrupts.CPU34.LOC:Local_timer_interrupts
      2664 ±  5%     +21.8%       3245 ±  8%  interrupts.CPU35.LOC:Local_timer_interrupts
      2680 ±  6%     +20.9%       3241 ±  8%  interrupts.CPU36.LOC:Local_timer_interrupts
      2650 ±  6%     +22.4%       3245 ±  7%  interrupts.CPU37.LOC:Local_timer_interrupts
      2653 ±  6%     +20.8%       3206 ±  8%  interrupts.CPU38.LOC:Local_timer_interrupts
      2653 ±  5%     +21.6%       3226 ±  8%  interrupts.CPU39.LOC:Local_timer_interrupts
      2650 ±  5%     +23.6%       3276 ±  6%  interrupts.CPU4.LOC:Local_timer_interrupts
      2679 ±  6%     +19.6%       3204 ±  8%  interrupts.CPU40.LOC:Local_timer_interrupts
      2649 ±  5%     +20.9%       3204 ±  8%  interrupts.CPU41.LOC:Local_timer_interrupts
      2666 ±  6%     +20.2%       3204 ±  8%  interrupts.CPU42.LOC:Local_timer_interrupts
      2699 ±  6%     +18.9%       3209 ±  8%  interrupts.CPU43.LOC:Local_timer_interrupts
      2685 ±  8%     +21.2%       3254 ±  9%  interrupts.CPU44.LOC:Local_timer_interrupts
      2718 ±  8%     +18.9%       3233 ±  8%  interrupts.CPU45.LOC:Local_timer_interrupts
      2644 ±  6%     +21.0%       3200 ±  9%  interrupts.CPU46.LOC:Local_timer_interrupts
      2672 ±  5%     +19.8%       3201 ±  9%  interrupts.CPU47.LOC:Local_timer_interrupts
      2662 ±  5%     +21.0%       3220 ±  9%  interrupts.CPU48.LOC:Local_timer_interrupts
      2676 ±  5%     +20.2%       3216 ±  8%  interrupts.CPU49.LOC:Local_timer_interrupts
      2655 ±  5%     +20.5%       3200 ±  8%  interrupts.CPU5.LOC:Local_timer_interrupts
    623.75           +14.3%     712.75 ± 11%  interrupts.CPU50.CAL:Function_call_interrupts
      2672 ±  6%     +19.4%       3191 ±  8%  interrupts.CPU50.LOC:Local_timer_interrupts
      2653 ±  5%     +20.4%       3194 ±  8%  interrupts.CPU51.LOC:Local_timer_interrupts
      2640 ±  6%     +20.9%       3193 ±  8%  interrupts.CPU52.LOC:Local_timer_interrupts
      2665 ±  6%     +19.9%       3196 ±  8%  interrupts.CPU53.LOC:Local_timer_interrupts
      2657 ±  5%     +20.0%       3189 ±  8%  interrupts.CPU54.LOC:Local_timer_interrupts
      2653 ±  5%     +21.1%       3212 ±  9%  interrupts.CPU55.LOC:Local_timer_interrupts
      2659 ±  5%     +20.0%       3192 ±  8%  interrupts.CPU56.LOC:Local_timer_interrupts
    208.75 ± 59%     -84.7%      32.00 ±173%  interrupts.CPU56.TLB:TLB_shootdowns
      2657 ±  5%     +20.2%       3194 ±  8%  interrupts.CPU57.LOC:Local_timer_interrupts
      2638 ±  6%     +21.0%       3192 ±  8%  interrupts.CPU58.LOC:Local_timer_interrupts
      2667 ±  7%     +19.7%       3194 ±  8%  interrupts.CPU59.LOC:Local_timer_interrupts
      2649 ±  5%     +21.0%       3205 ±  8%  interrupts.CPU6.LOC:Local_timer_interrupts
    236.00 ± 66%     -86.2%      32.50 ±169%  interrupts.CPU6.TLB:TLB_shootdowns
      2677 ±  6%     +19.0%       3186 ±  8%  interrupts.CPU60.LOC:Local_timer_interrupts
      2674 ±  6%     +19.3%       3191 ±  8%  interrupts.CPU61.LOC:Local_timer_interrupts
      2658 ±  5%     +20.2%       3194 ±  8%  interrupts.CPU62.LOC:Local_timer_interrupts
      2666 ±  5%     +20.8%       3221 ±  9%  interrupts.CPU63.LOC:Local_timer_interrupts
      2690 ±  6%     +19.9%       3225 ± 10%  interrupts.CPU64.LOC:Local_timer_interrupts
      2653 ±  5%     +20.4%       3194 ±  8%  interrupts.CPU65.LOC:Local_timer_interrupts
    652.50 ±  9%     +21.9%     795.25 ± 18%  interrupts.CPU66.CAL:Function_call_interrupts
      2669 ±  5%     +19.7%       3196 ±  8%  interrupts.CPU66.LOC:Local_timer_interrupts
      2655 ±  5%     +20.2%       3191 ±  8%  interrupts.CPU67.LOC:Local_timer_interrupts
      2665 ±  5%     +20.4%       3207 ±  8%  interrupts.CPU68.LOC:Local_timer_interrupts
    614.50           +27.0%     780.50 ± 25%  interrupts.CPU69.CAL:Function_call_interrupts
      2666 ±  5%     +21.2%       3232 ±  9%  interrupts.CPU69.LOC:Local_timer_interrupts
      2678 ±  6%     +21.1%       3243 ±  7%  interrupts.CPU7.LOC:Local_timer_interrupts
    657.50 ±  9%     +24.1%     815.75 ± 21%  interrupts.CPU70.CAL:Function_call_interrupts
      2676 ±  6%     +19.9%       3208 ±  8%  interrupts.CPU70.LOC:Local_timer_interrupts
    657.00 ±  9%     +24.2%     815.75 ± 21%  interrupts.CPU71.CAL:Function_call_interrupts
      2674 ±  6%     +19.5%       3194 ±  8%  interrupts.CPU71.LOC:Local_timer_interrupts
      2653 ±  5%     +21.4%       3221 ±  9%  interrupts.CPU72.LOC:Local_timer_interrupts
      2670 ±  6%     +20.0%       3204 ±  8%  interrupts.CPU73.LOC:Local_timer_interrupts
      2669 ±  6%     +19.8%       3197 ±  9%  interrupts.CPU74.LOC:Local_timer_interrupts
    652.00 ±  9%     +25.3%     816.75 ± 21%  interrupts.CPU75.CAL:Function_call_interrupts
      2641 ±  5%     +21.4%       3205 ±  8%  interrupts.CPU75.LOC:Local_timer_interrupts
    625.00           +25.8%     786.25 ± 16%  interrupts.CPU76.CAL:Function_call_interrupts
      2660 ±  5%     +20.6%       3208 ±  8%  interrupts.CPU76.LOC:Local_timer_interrupts
    656.75 ±  9%     +20.6%     791.75 ± 17%  interrupts.CPU77.CAL:Function_call_interrupts
      2665 ±  5%     +19.6%       3187 ±  8%  interrupts.CPU77.LOC:Local_timer_interrupts
    657.00 ±  9%     +21.3%     796.75 ± 18%  interrupts.CPU78.CAL:Function_call_interrupts
      2666 ±  5%     +19.7%       3190 ±  8%  interrupts.CPU78.LOC:Local_timer_interrupts
      2660 ±  5%     +19.9%       3190 ±  8%  interrupts.CPU79.LOC:Local_timer_interrupts
    648.75 ±  7%     +19.2%     773.00 ± 16%  interrupts.CPU8.CAL:Function_call_interrupts
      2647 ±  5%     +21.3%       3211 ±  7%  interrupts.CPU8.LOC:Local_timer_interrupts
      2671 ±  6%     +19.5%       3191 ±  8%  interrupts.CPU80.LOC:Local_timer_interrupts
      2660 ±  5%     +20.9%       3217 ±  8%  interrupts.CPU81.LOC:Local_timer_interrupts
      2639 ±  6%     +21.6%       3209 ±  8%  interrupts.CPU82.LOC:Local_timer_interrupts
      2652 ±  5%     +21.5%       3223 ±  7%  interrupts.CPU83.LOC:Local_timer_interrupts
      2656 ±  5%     +20.5%       3201 ±  8%  interrupts.CPU84.LOC:Local_timer_interrupts
      2678 ±  4%     +19.8%       3207 ±  8%  interrupts.CPU85.LOC:Local_timer_interrupts
      2659 ±  5%     +20.7%       3209 ±  8%  interrupts.CPU86.LOC:Local_timer_interrupts
    656.25 ±  9%     +24.8%     819.00 ± 22%  interrupts.CPU87.CAL:Function_call_interrupts
      2656 ±  5%     +20.5%       3200 ±  8%  interrupts.CPU87.LOC:Local_timer_interrupts
      2655 ±  5%     +20.9%       3211 ±  8%  interrupts.CPU9.LOC:Local_timer_interrupts
    234559 ±  5%     +20.5%     282643 ±  8%  interrupts.LOC:Local_timer_interrupts



***************************************************************************************************
lkp-bdw-ep6: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.2/100%/debian-x86_64-2018-04-03.cgz/lkp-bdw-ep6/plzip/0xb00002e

commit: 
  24d0c1d6e6 ("sched/fair: Do not migrate due to a sync wakeup on exit")
  2c83362734 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")

24d0c1d6e65f635b 2c83362734dad8e48ccc0710b5c 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   4674851            -5.8%    4402603        plzip.time.minor_page_faults
    187.93            -1.7%     184.66        plzip.time.system_time
    236915            +3.3%     244744        plzip.time.voluntary_context_switches
  20188778 ±  7%     +62.1%   32719513 ± 32%  cpuidle.POLL.time
      0.00            -0.0        0.00 ± 24%  mpstat.cpu.soft%
     11424 ± 49%     -62.2%       4321 ±169%  numa-numastat.node0.other_node
      2223            -2.7%       2163        vmstat.system.cs
    199507            -6.3%     186973        vmstat.system.in
     11254           +22.5%      13784 ± 16%  numa-meminfo.node0.Mapped
     75.00          +544.0%     483.00 ± 81%  numa-meminfo.node0.Mlocked
     75.00          +544.0%     483.00 ± 81%  numa-meminfo.node0.Unevictable
     38772 ±  5%     -11.6%      34263 ± 10%  numa-meminfo.node1.SReclaimable
      2793           +25.4%       3502 ± 14%  numa-vmstat.node0.nr_mapped
     18.50 ±  2%    +550.0%     120.25 ± 82%  numa-vmstat.node0.nr_mlock
     18.50 ±  2%    +550.0%     120.25 ± 82%  numa-vmstat.node0.nr_unevictable
     18.50 ±  2%    +550.0%     120.25 ± 82%  numa-vmstat.node0.nr_zone_unevictable
     11543 ± 49%     -60.8%       4519 ±159%  numa-vmstat.node0.numa_other
      9693 ±  5%     -11.6%       8566 ± 10%  numa-vmstat.node1.nr_slab_reclaimable
   3008454            -9.0%    2737405        proc-vmstat.numa_hint_faults
   2421331            -7.5%    2240371        proc-vmstat.numa_hint_faults_local
    185494           -11.9%     163404        proc-vmstat.numa_huge_pte_updates
   5591805 ±  3%     -12.4%    4899638 ±  3%  proc-vmstat.numa_pages_migrated
  98610869           -11.8%   86933972        proc-vmstat.numa_pte_updates
   5570737            -4.9%    5295155        proc-vmstat.pgfault
   5591805 ±  3%     -12.4%    4899638 ±  3%  proc-vmstat.pgmigrate_success
      2358 ±  3%     -24.9%       1770 ± 14%  slabinfo.eventpoll_epi.active_objs
      2358 ±  3%     -24.9%       1770 ± 14%  slabinfo.eventpoll_epi.num_objs
      4127 ±  3%     -25.6%       3071 ± 15%  slabinfo.eventpoll_pwq.active_objs
      4127 ±  3%     -25.6%       3071 ± 15%  slabinfo.eventpoll_pwq.num_objs
    767.50 ±  5%     -16.6%     640.00 ± 10%  slabinfo.file_lock_cache.active_objs
    767.50 ±  5%     -16.6%     640.00 ± 10%  slabinfo.file_lock_cache.num_objs
    548.00 ± 12%     -13.4%     474.75 ± 17%  slabinfo.skbuff_fclone_cache.active_objs
    548.00 ± 12%     -13.4%     474.75 ± 17%  slabinfo.skbuff_fclone_cache.num_objs
      3424           -13.7%       2955 ±  8%  slabinfo.sock_inode_cache.active_objs
      3424           -13.7%       2955 ±  8%  slabinfo.sock_inode_cache.num_objs
      1.38            +8.7%       1.50 ±  6%  sched_debug.cfs_rq:/.nr_spread_over.avg
    123441 ± 32%     -87.5%      15420 ± 31%  sched_debug.cfs_rq:/.spread0.avg
    310956 ± 19%     -40.1%     186343 ±  7%  sched_debug.cfs_rq:/.spread0.max
     29.67 ±  5%      -7.4%      27.46 ±  5%  sched_debug.cpu.cpu_load[1].max
     53.50 ±  8%     -16.1%      44.88 ± 16%  sched_debug.cpu.cpu_load[4].max
      4.96 ±  7%     -16.0%       4.17 ± 17%  sched_debug.cpu.cpu_load[4].stddev
      3942 ± 15%     -36.1%       2519 ± 42%  sched_debug.cpu.curr->pid.min
    312.17 ±  4%     -12.6%     272.83 ±  4%  sched_debug.cpu.sched_goidle.stddev
      5485           +31.8%       7231 ± 15%  sched_debug.cpu.ttwu_count.max
    829.33 ±  6%     -10.5%     742.58 ± 12%  sched_debug.cpu.ttwu_count.min
      3789 ±  3%     +38.5%       5249 ± 18%  sched_debug.cpu.ttwu_local.max
    361.67 ± 16%     -24.7%     272.42 ±  9%  sched_debug.cpu.ttwu_local.min
     42735 ±  4%      -9.5%      38655 ±  4%  softirqs.CPU1.RCU
     43986           -11.1%      39106 ±  6%  softirqs.CPU11.RCU
    138451 ±  3%      -6.2%     129877 ±  3%  softirqs.CPU11.TIMER
     43616 ±  3%     -11.3%      38683 ±  8%  softirqs.CPU14.RCU
     46391 ±  3%     -15.0%      39420 ±  4%  softirqs.CPU2.RCU
     46679 ±  2%     -10.4%      41843 ±  2%  softirqs.CPU35.RCU
     54405 ±  9%     -13.0%      47347 ± 11%  softirqs.CPU36.RCU
     53662 ±  5%     -16.5%      44815 ±  5%  softirqs.CPU37.RCU
     51563 ± 11%     -17.6%      42505 ±  5%  softirqs.CPU39.RCU
    146287 ±  3%      -7.7%     135015 ±  3%  softirqs.CPU4.TIMER
     51810 ±  8%     -12.9%      45103 ± 11%  softirqs.CPU41.RCU
    144965 ±  3%      -7.2%     134488 ±  2%  softirqs.CPU5.TIMER
     36399 ±  2%     +19.8%      43611 ±  2%  softirqs.CPU66.RCU
     54540           -17.2%      45175 ±  2%  softirqs.CPU68.RCU
     45027           -10.8%      40167 ±  6%  softirqs.CPU7.RCU
     46747 ± 10%     -20.2%      37293 ± 10%  softirqs.CPU77.RCU
     42610 ±  2%     -10.8%      38011 ±  7%  softirqs.CPU83.RCU
     43534 ±  3%     -10.6%      38935 ±  9%  softirqs.CPU87.RCU
     45144 ±  4%      -8.2%      41426 ±  5%  softirqs.CPU9.RCU
      2207            -2.7%       2149        perf-stat.i.context-switches
    132.50            -6.8%     123.43 ±  2%  perf-stat.i.cpu-migrations
      0.04            -0.0        0.03 ±  2%  perf-stat.i.dTLB-store-miss-rate%
   2870039            -7.1%    2665015        perf-stat.i.dTLB-store-misses
     69.91            +2.8       72.76 ±  2%  perf-stat.i.iTLB-load-miss-rate%
    593399           -13.4%     513752 ±  3%  perf-stat.i.iTLB-loads
     15830            -5.4%      14979        perf-stat.i.minor-faults
      5.76 ±  6%      -1.1        4.63 ± 12%  perf-stat.i.node-load-miss-rate%
  29280168 ±  5%     -22.1%   22809166 ±  7%  perf-stat.i.node-load-misses
      5.96 ±  8%      -1.4        4.52 ± 11%  perf-stat.i.node-store-miss-rate%
   4992121 ±  7%     -27.5%    3621719 ±  7%  perf-stat.i.node-store-misses
     15830            -5.4%      14979        perf-stat.i.page-faults
      0.04            -0.0        0.03        perf-stat.overall.dTLB-store-miss-rate%
      5.46 ±  5%      -1.2        4.24 ±  7%  perf-stat.overall.node-load-miss-rate%
      5.68 ±  7%      -1.6        4.12 ±  8%  perf-stat.overall.node-store-miss-rate%
      2201            -2.6%       2143        perf-stat.ps.context-switches
    132.11            -6.8%     123.07 ±  2%  perf-stat.ps.cpu-migrations
   2861551            -7.1%    2657323        perf-stat.ps.dTLB-store-misses
    591695           -13.4%     512284 ±  3%  perf-stat.ps.iTLB-loads
     15785            -5.4%      14937        perf-stat.ps.minor-faults
  29194092 ±  5%     -22.1%   22742509 ±  7%  perf-stat.ps.node-load-misses
   4977398 ±  7%     -27.4%    3611184 ±  7%  perf-stat.ps.node-store-misses
     15785            -5.4%      14937        perf-stat.ps.page-faults
     37.67            -3.5       34.22 ±  3%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt
     36.20            -3.2       32.98 ±  3%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt
     30.64            -2.6       28.08 ±  3%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
     15.44 ±  8%      -2.2       13.23 ±  5%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
     16.59 ±  7%      -2.2       14.40 ±  4%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
     14.94 ±  8%      -2.1       12.84 ±  5%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
     23.31 ±  2%      -2.1       21.23 ±  3%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
     11.29 ±  5%      -1.7        9.60 ±  5%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      8.27 ±  2%      -1.0        7.23 ±  4%  perf-profile.calltrace.cycles-pp.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
     12.36            -0.6       11.71 ±  4%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     12.36            -0.6       11.71 ±  4%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.88            -0.6        3.31 ±  4%  perf-profile.calltrace.cycles-pp.vfs_write.sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.86 ±  2%      -0.6        3.30 ±  4%  perf-profile.calltrace.cycles-pp.pipe_write.__vfs_write.vfs_write.sys_write.do_syscall_64
      3.88            -0.6        3.32 ±  4%  perf-profile.calltrace.cycles-pp.sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.86 ±  2%      -0.6        3.31 ±  3%  perf-profile.calltrace.cycles-pp.__vfs_write.vfs_write.sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.08            -0.5        3.56 ±  2%  perf-profile.calltrace.cycles-pp.update_load_avg.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
      2.81 ±  2%      -0.4        2.39 ±  7%  perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.__vfs_write.vfs_write.sys_write
      2.62 ±  4%      -0.4        2.21 ±  6%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyin.copy_page_from_iter.pipe_write.__vfs_write
      2.62 ±  4%      -0.4        2.21 ±  6%  perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter.pipe_write.__vfs_write.vfs_write
      0.87 ±  6%      -0.4        0.47 ± 57%  perf-profile.calltrace.cycles-pp.native_apic_msr_eoi_write.smp_apic_timer_interrupt.apic_timer_interrupt
      0.83 ±  7%      -0.4        0.44 ± 58%  perf-profile.calltrace.cycles-pp.account_user_time.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      3.40 ±  3%      -0.3        3.12 ±  2%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      3.00 ±  4%      -0.3        2.71        perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      1.44 ±  2%      -0.2        1.29 ± 10%  perf-profile.calltrace.cycles-pp.update_curr.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle
      0.77 ±  3%      -0.1        0.62 ± 12%  perf-profile.calltrace.cycles-pp.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
    545.00 ± 25%     -41.9%     316.75 ± 47%  interrupts.36:IR-PCI-MSI.1572867-edge.eth0-TxRx-2
    203.50 ±  9%     +45.3%     295.75 ± 24%  interrupts.39:IR-PCI-MSI.1572870-edge.eth0-TxRx-5
  38166429           -11.0%   33961585        interrupts.CAL:Function_call_interrupts
    432615           -10.9%     385659        interrupts.CPU0.CAL:Function_call_interrupts
    434222           -11.2%     385591        interrupts.CPU1.CAL:Function_call_interrupts
    432253 ±  2%     -10.6%     386505        interrupts.CPU10.CAL:Function_call_interrupts
    432492 ±  2%     -11.0%     384996        interrupts.CPU11.CAL:Function_call_interrupts
    450925 ±  2%      -7.4%     417361 ±  2%  interrupts.CPU11.TLB:TLB_shootdowns
    431152           -10.4%     386198        interrupts.CPU12.CAL:Function_call_interrupts
      7893           -37.4%       4939 ± 34%  interrupts.CPU12.NMI:Non-maskable_interrupts
      7893           -37.4%       4939 ± 34%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
    431989 ±  2%     -10.8%     385478        interrupts.CPU13.CAL:Function_call_interrupts
    432141           -10.8%     385372        interrupts.CPU14.CAL:Function_call_interrupts
    430309           -10.2%     386216        interrupts.CPU15.CAL:Function_call_interrupts
    545.00 ± 25%     -41.9%     316.75 ± 47%  interrupts.CPU16.36:IR-PCI-MSI.1572867-edge.eth0-TxRx-2
    432466           -10.4%     387465        interrupts.CPU16.CAL:Function_call_interrupts
    432582           -11.0%     384925        interrupts.CPU17.CAL:Function_call_interrupts
    434853           -11.0%     386855        interrupts.CPU18.CAL:Function_call_interrupts
    203.50 ±  9%     +45.3%     295.75 ± 24%  interrupts.CPU19.39:IR-PCI-MSI.1572870-edge.eth0-TxRx-5
    432706           -10.6%     386950        interrupts.CPU19.CAL:Function_call_interrupts
    434387           -12.4%     380661        interrupts.CPU2.CAL:Function_call_interrupts
    453229            -9.2%     411516        interrupts.CPU2.TLB:TLB_shootdowns
    435346           -11.3%     386104        interrupts.CPU20.CAL:Function_call_interrupts
    454122            -7.9%     418229 ±  2%  interrupts.CPU20.TLB:TLB_shootdowns
    434493           -10.9%     387125        interrupts.CPU21.CAL:Function_call_interrupts
    621.00 ± 11%     +64.5%       1021 ± 20%  interrupts.CPU21.RES:Rescheduling_interrupts
    425823 ±  3%      -9.4%     385897        interrupts.CPU22.CAL:Function_call_interrupts
    433468           -11.0%     385821        interrupts.CPU23.CAL:Function_call_interrupts
    434913           -11.2%     386097        interrupts.CPU24.CAL:Function_call_interrupts
    434374           -11.2%     385692        interrupts.CPU25.CAL:Function_call_interrupts
    433527           -10.6%     387596        interrupts.CPU26.CAL:Function_call_interrupts
    434308           -11.3%     385115        interrupts.CPU27.CAL:Function_call_interrupts
    431181           -10.9%     384326        interrupts.CPU28.CAL:Function_call_interrupts
    434448           -11.5%     384490        interrupts.CPU29.CAL:Function_call_interrupts
    453474 ±  2%      -8.0%     417056        interrupts.CPU29.TLB:TLB_shootdowns
    433938           -11.1%     385867        interrupts.CPU3.CAL:Function_call_interrupts
      2739 ± 14%     -41.8%       1594 ± 27%  interrupts.CPU3.RES:Rescheduling_interrupts
    433773           -11.0%     386014        interrupts.CPU30.CAL:Function_call_interrupts
    434250           -11.1%     385917        interrupts.CPU31.CAL:Function_call_interrupts
    436173           -11.6%     385778        interrupts.CPU32.CAL:Function_call_interrupts
    455271 ±  2%      -8.0%     418794 ±  2%  interrupts.CPU32.TLB:TLB_shootdowns
    435787           -11.4%     386189        interrupts.CPU33.CAL:Function_call_interrupts
    436438           -11.5%     386321        interrupts.CPU34.CAL:Function_call_interrupts
    433917           -11.4%     384653        interrupts.CPU35.CAL:Function_call_interrupts
    434362           -10.8%     387306        interrupts.CPU36.CAL:Function_call_interrupts
    434686           -11.3%     385416        interrupts.CPU37.CAL:Function_call_interrupts
    453983 ±  2%      -7.8%     418472        interrupts.CPU37.TLB:TLB_shootdowns
    433245           -10.7%     386871        interrupts.CPU38.CAL:Function_call_interrupts
    430705           -10.0%     387585        interrupts.CPU39.CAL:Function_call_interrupts
    434411           -11.2%     385547        interrupts.CPU4.CAL:Function_call_interrupts
      1251 ±  9%     +90.3%       2380 ± 32%  interrupts.CPU4.RES:Rescheduling_interrupts
    434631           -11.1%     386349        interrupts.CPU40.CAL:Function_call_interrupts
    436453           -11.6%     385886        interrupts.CPU41.CAL:Function_call_interrupts
    436514           -11.5%     386455        interrupts.CPU42.CAL:Function_call_interrupts
    432629 ±  2%     -11.0%     385234        interrupts.CPU43.CAL:Function_call_interrupts
      1982 ± 40%     -39.4%       1201 ± 44%  interrupts.CPU43.RES:Rescheduling_interrupts
    438403           -11.0%     390257        interrupts.CPU44.CAL:Function_call_interrupts
    437673           -10.6%     391240        interrupts.CPU45.CAL:Function_call_interrupts
    436956           -10.0%     393070        interrupts.CPU46.CAL:Function_call_interrupts
    436535 ±  2%     -10.3%     391586        interrupts.CPU47.CAL:Function_call_interrupts
    436404 ±  2%     -10.3%     391331        interrupts.CPU48.CAL:Function_call_interrupts
    440924           -11.0%     392351        interrupts.CPU49.CAL:Function_call_interrupts
    455336 ±  2%      -8.0%     418988 ±  2%  interrupts.CPU49.TLB:TLB_shootdowns
    429597           -10.2%     385817        interrupts.CPU5.CAL:Function_call_interrupts
      2657 ± 16%     -57.7%       1125 ± 38%  interrupts.CPU5.RES:Rescheduling_interrupts
    432290           -10.8%     385504        interrupts.CPU50.CAL:Function_call_interrupts
    430785 ±  2%     -10.2%     386960        interrupts.CPU51.CAL:Function_call_interrupts
    433428 ±  2%     -11.0%     385627        interrupts.CPU52.CAL:Function_call_interrupts
    452795 ±  2%      -7.6%     418273        interrupts.CPU52.TLB:TLB_shootdowns
    432275           -10.8%     385758        interrupts.CPU53.CAL:Function_call_interrupts
    432401           -10.8%     385721        interrupts.CPU54.CAL:Function_call_interrupts
    433685           -11.2%     384984        interrupts.CPU55.CAL:Function_call_interrupts
    434907           -11.3%     385865        interrupts.CPU56.CAL:Function_call_interrupts
    454111 ±  2%      -7.7%     419097 ±  2%  interrupts.CPU56.TLB:TLB_shootdowns
    433831 ±  2%     -11.6%     383393        interrupts.CPU57.CAL:Function_call_interrupts
    452988 ±  2%      -8.1%     416171        interrupts.CPU57.TLB:TLB_shootdowns
    434640           -11.0%     386694        interrupts.CPU58.CAL:Function_call_interrupts
    433779           -11.1%     385790 ±  2%  interrupts.CPU59.CAL:Function_call_interrupts
    433594           -11.3%     384456 ±  2%  interrupts.CPU6.CAL:Function_call_interrupts
    433902           -11.5%     383949        interrupts.CPU60.CAL:Function_call_interrupts
    453607            -8.1%     416661        interrupts.CPU60.TLB:TLB_shootdowns
    432735           -10.7%     386644        interrupts.CPU61.CAL:Function_call_interrupts
    430100 ±  2%     -10.3%     385648        interrupts.CPU62.CAL:Function_call_interrupts
      1361 ± 17%     -46.0%     734.75 ± 21%  interrupts.CPU62.RES:Rescheduling_interrupts
    430135 ±  2%     -10.4%     385243        interrupts.CPU63.CAL:Function_call_interrupts
    433506           -11.4%     384298        interrupts.CPU64.CAL:Function_call_interrupts
    428442           -10.6%     383084        interrupts.CPU65.CAL:Function_call_interrupts
    934.00           +41.7%       1323 ±  6%  interrupts.CPU65.RES:Rescheduling_interrupts
    429851           -10.2%     385861 ±  2%  interrupts.CPU66.CAL:Function_call_interrupts
    432662           -10.6%     386768        interrupts.CPU67.CAL:Function_call_interrupts
      1137 ± 30%     -37.3%     713.25 ± 22%  interrupts.CPU67.RES:Rescheduling_interrupts
    435998           -11.2%     387345        interrupts.CPU68.CAL:Function_call_interrupts
    433910           -11.1%     385547 ±  2%  interrupts.CPU69.CAL:Function_call_interrupts
    433665           -11.1%     385376        interrupts.CPU7.CAL:Function_call_interrupts
    433964           -11.2%     385200        interrupts.CPU70.CAL:Function_call_interrupts
    434280           -11.2%     385800        interrupts.CPU71.CAL:Function_call_interrupts
    436755           -12.1%     383884        interrupts.CPU72.CAL:Function_call_interrupts
    434534           -11.1%     386259        interrupts.CPU73.CAL:Function_call_interrupts
    435838           -11.3%     386437        interrupts.CPU74.CAL:Function_call_interrupts
    849.50           +22.9%       1044 ± 15%  interrupts.CPU74.RES:Rescheduling_interrupts
    431545           -11.2%     383340        interrupts.CPU75.CAL:Function_call_interrupts
    435028           -11.6%     384709        interrupts.CPU76.CAL:Function_call_interrupts
    434623           -11.7%     383907        interrupts.CPU77.CAL:Function_call_interrupts
    432569           -11.0%     384848        interrupts.CPU78.CAL:Function_call_interrupts
    433842           -12.4%     379948 ±  3%  interrupts.CPU79.CAL:Function_call_interrupts
    453571            -9.3%     411502 ±  3%  interrupts.CPU79.TLB:TLB_shootdowns
    434907           -11.6%     384337 ±  2%  interrupts.CPU8.CAL:Function_call_interrupts
    435116           -11.4%     385431        interrupts.CPU80.CAL:Function_call_interrupts
    677.50 ± 25%     -24.1%     514.00 ± 28%  interrupts.CPU80.RES:Rescheduling_interrupts
    432192           -11.0%     384794        interrupts.CPU81.CAL:Function_call_interrupts
    433320           -10.7%     387090        interrupts.CPU82.CAL:Function_call_interrupts
      1174 ±  9%     -50.1%     586.50 ± 29%  interrupts.CPU82.RES:Rescheduling_interrupts
    435824           -11.9%     384143        interrupts.CPU83.CAL:Function_call_interrupts
    434477           -11.0%     386568        interrupts.CPU84.CAL:Function_call_interrupts
    432705           -10.4%     387511        interrupts.CPU85.CAL:Function_call_interrupts
    434758 ±  2%     -11.5%     384778        interrupts.CPU86.CAL:Function_call_interrupts
    454777 ±  2%      -7.9%     418646        interrupts.CPU86.TLB:TLB_shootdowns
    432224           -11.3%     383338        interrupts.CPU87.CAL:Function_call_interrupts
    432930           -11.2%     384537        interrupts.CPU9.CAL:Function_call_interrupts



***************************************************************************************************
lkp-bdw-ex1: 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 512G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.2/100%/debian-x86_64-2018-04-03.cgz/300s/lkp-bdw-ex1/all_utime/reaim/0xb00002e

commit: 
  24d0c1d6e6 ("sched/fair: Do not migrate due to a sync wakeup on exit")
  2c83362734 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")

24d0c1d6e65f635b 2c83362734dad8e48ccc0710b5c 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.WARNING:stack_going_in_the_wrong_direction?ip=schedule_tail/0x
         %stddev     %change         %stddev
             \          |                \  
      0.08           -10.2%       0.07        reaim.child_systime
    753802           -11.9%     663774        reaim.jobs_per_min
      3926           -11.9%       3457        reaim.jobs_per_min_child
     97.65            -3.8%      93.97        reaim.jti
    806247            -6.3%     755738        reaim.max_jobs_per_min
      1.56           +13.6%       1.78        reaim.parent_time
      1.88 ±  3%    +194.4%       5.53 ±  4%  reaim.std_dev_percent
      0.03 ±  3%    +217.2%       0.09 ±  4%  reaim.std_dev_time
    104138           +86.0%     193748        reaim.time.involuntary_context_switches
   1074884            +2.8%    1105377        reaim.time.minor_page_faults
      7721            -6.3%       7236        reaim.time.percent_of_cpu_this_job_got
     23369            -6.5%      21847        reaim.time.user_time
     66048            -1.7%      64924        reaim.time.voluntary_context_switches
   1612800            -5.7%    1521600        reaim.workload
 1.448e+08 ±  4%     +75.9%  2.548e+08 ± 27%  cpuidle.POLL.time
    449918 ±  4%     -25.3%     336278 ± 15%  numa-numastat.node1.local_node
    462355 ±  2%     -23.2%     354953 ± 14%  numa-numastat.node1.numa_hit
     59.00            +5.1%      62.00        vmstat.cpu.id
     40.00            -7.5%      37.00        vmstat.cpu.us
      2033           +18.8%       2415        vmstat.system.cs
      1092            -5.8%       1029        turbostat.Avg_MHz
     12.57           +17.7%      14.79        turbostat.CPU%c1
      2.17 ±  5%     -26.9%       1.59 ± 13%  turbostat.Pkg%pc6
     20.95            +7.4%      22.50 ±  4%  turbostat.RAMWatt
     22388 ± 34%     -62.8%       8322 ± 85%  numa-vmstat.node1.nr_active_anon
      6845 ± 15%     -38.8%       4189 ± 10%  numa-vmstat.node1.nr_slab_reclaimable
     17978 ±  3%     -26.0%      13303 ± 10%  numa-vmstat.node1.nr_slab_unreclaimable
     22388 ± 34%     -62.8%       8322 ± 85%  numa-vmstat.node1.nr_zone_active_anon
      2740 ± 62%     -95.1%     135.50 ± 77%  numa-vmstat.node3.nr_inactive_anon
      2921 ± 59%     -91.1%     259.00 ± 31%  numa-vmstat.node3.nr_shmem
      2740 ± 62%     -95.1%     135.50 ± 77%  numa-vmstat.node3.nr_zone_inactive_anon
    101194 ± 30%     -53.7%      46870 ± 62%  numa-meminfo.node1.Active
     89611 ± 34%     -62.9%      33266 ± 85%  numa-meminfo.node1.Active(anon)
    566915 ±  3%     -18.3%     463403 ±  8%  numa-meminfo.node1.MemUsed
     27379 ± 15%     -38.8%      16759 ± 10%  numa-meminfo.node1.SReclaimable
     71912 ±  3%     -26.0%      53214 ± 10%  numa-meminfo.node1.SUnreclaim
     99292 ±  5%     -29.5%      69973 ±  8%  numa-meminfo.node1.Slab
     10962 ± 62%     -95.0%     543.25 ± 77%  numa-meminfo.node3.Inactive(anon)
     11687 ± 59%     -91.1%       1037 ± 31%  numa-meminfo.node3.Shmem
     77999            -3.5%      75239        proc-vmstat.nr_slab_unreclaimable
     38308 ± 13%    +242.0%     131006 ±  2%  proc-vmstat.numa_hint_faults
    780.75 ±100%    +571.6%       5243 ± 75%  proc-vmstat.numa_hint_faults_local
   1910995            +2.1%    1950408        proc-vmstat.numa_hit
   1854894            +2.1%    1894321        proc-vmstat.numa_local
     40180 ± 21%    +204.9%     122514 ±  4%  proc-vmstat.numa_pages_migrated
    145766 ± 44%    +152.1%     367445        proc-vmstat.numa_pte_updates
   2082332            +1.5%    2113859        proc-vmstat.pgalloc_normal
   2106944            +1.5%    2139589        proc-vmstat.pgfault
   2018610            +1.6%    2050051        proc-vmstat.pgfree
     40180 ± 21%    +204.9%     122514 ±  4%  proc-vmstat.pgmigrate_success
      9772 ±  7%     +48.0%      14465 ± 12%  softirqs.CPU0.SCHED
     62268           +13.2%      70466        softirqs.CPU0.TIMER
     14263 ±  6%     +19.6%      17059 ±  7%  softirqs.CPU148.RCU
     53614           +10.2%      59094 ±  6%  softirqs.CPU148.TIMER
     15228 ±  3%     +19.8%      18246 ± 14%  softirqs.CPU37.RCU
     13686 ±  9%     +19.4%      16346 ±  7%  softirqs.CPU53.RCU
     14747 ±  9%     +15.4%      17023 ±  7%  softirqs.CPU54.RCU
     14975 ±  4%     +20.3%      18017 ± 18%  softirqs.CPU55.RCU
     14654 ±  4%     +12.7%      16510 ±  9%  softirqs.CPU59.RCU
     14608 ±  4%      +9.0%      15927 ±  8%  softirqs.CPU60.RCU
     14141           +13.6%      16067 ±  7%  softirqs.CPU61.RCU
     13788 ±  3%     +28.1%      17667 ± 19%  softirqs.CPU63.RCU
     14460 ±  7%     -16.3%      12105 ±  9%  softirqs.CPU96.RCU
      3871 ±  3%     -23.7%       2953 ±  7%  slabinfo.biovec-64.active_objs
      3871 ±  3%     -23.7%       2953 ±  7%  slabinfo.biovec-64.num_objs
      4456 ±  4%     -66.5%       1490 ±  3%  slabinfo.buffer_head.active_objs
      4456 ±  4%     -66.5%       1490 ±  3%  slabinfo.buffer_head.num_objs
      9327 ±  7%     -49.8%       4681 ± 20%  slabinfo.eventpoll_epi.active_objs
      9327 ±  7%     -49.8%       4681 ± 20%  slabinfo.eventpoll_epi.num_objs
      8161 ±  7%     -49.8%       4096 ± 20%  slabinfo.eventpoll_pwq.active_objs
      8161 ±  7%     -49.8%       4096 ± 20%  slabinfo.eventpoll_pwq.num_objs
    870.00 ± 10%     -22.0%     678.25 ±  5%  slabinfo.file_lock_cache.active_objs
    870.00 ± 10%     -22.0%     678.25 ±  5%  slabinfo.file_lock_cache.num_objs
     11826           -12.3%      10377 ±  4%  slabinfo.shmem_inode_cache.active_objs
     11826           -12.3%      10377 ±  4%  slabinfo.shmem_inode_cache.num_objs
      7919 ±  2%     -16.7%       6596 ±  2%  slabinfo.sighand_cache.active_objs
      8073 ±  2%     -18.0%       6616 ±  2%  slabinfo.sighand_cache.num_objs
     13382           -20.2%      10683        slabinfo.signal_cache.active_objs
     13441           -20.4%      10695        slabinfo.signal_cache.num_objs
      9837           +26.3%      12423        slabinfo.sigqueue.active_objs
      9837           +26.3%      12423        slabinfo.sigqueue.num_objs
      6269 ±  3%     -20.0%       5015 ±  6%  slabinfo.sock_inode_cache.active_objs
      6269 ±  3%     -20.0%       5015 ±  6%  slabinfo.sock_inode_cache.num_objs
     16.88 ± 23%      -8.9        7.94 ± 20%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
     16.79 ± 23%      -8.9        7.88 ± 20%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
     11.45 ± 34%      -7.5        3.90 ± 28%  perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
     11.06 ± 35%      -7.4        3.63 ± 29%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      9.53 ± 42%      -6.6        2.90 ± 22%  perf-profile.calltrace.cycles-pp.tick_do_update_jiffies64.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt
      9.49 ± 42%      -6.6        2.88 ± 22%  perf-profile.calltrace.cycles-pp._raw_spin_lock.tick_do_update_jiffies64.tick_irq_enter.irq_enter.smp_apic_timer_interrupt
      8.59 ± 41%      -6.2        2.44 ± 29%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.tick_do_update_jiffies64.tick_irq_enter.irq_enter
      6.58 ±  4%      -1.5        5.11 ± 11%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt
      6.38 ±  4%      -1.4        4.93 ± 11%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt
      5.84 ±  4%      -1.4        4.48 ± 12%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      4.18 ±  6%      -1.0        3.16 ± 12%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      2.96 ±  3%      -0.8        2.14 ± 12%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      2.70 ±  4%      -0.8        1.91 ± 13%  perf-profile.calltrace.cycles-pp.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt
      3.17 ±  3%      -0.8        2.40 ± 14%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      2.67 ±  4%      -0.8        1.90 ± 13%  perf-profile.calltrace.cycles-pp.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt
      1.98 ±  7%      -0.8        1.20 ± 13%  perf-profile.calltrace.cycles-pp.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      2.38 ±  5%      -0.7        1.65 ± 13%  perf-profile.calltrace.cycles-pp.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      1.97 ±  4%      -0.5        1.49 ± 12%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      1.59 ±  7%      -0.5        1.13 ± 17%  perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      1.39 ±  5%      -0.4        0.97 ± 17%  perf-profile.calltrace.cycles-pp.__tick_nohz_idle_enter.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      0.68 ±  8%      -0.4        0.29 ±100%  perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
      1.19 ±  6%      -0.4        0.81 ± 18%  perf-profile.calltrace.cycles-pp.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      0.82 ±  6%      -0.2        0.62 ± 11%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state
     86.66            +3.0       89.68        perf-profile.calltrace.cycles-pp.secondary_startup_64
     86.09            +3.1       89.20        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     86.09            +3.1       89.20        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     86.09            +3.1       89.20        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     84.14            +3.5       87.62        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      9.63 ± 15%      +5.5       15.11 ± 20%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
     56.81 ±  6%      +7.1       63.92 ±  2%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      0.00 ± 22%  +4.8e+24%      71.37 ±105%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
    612.74 ± 12%    +597.4%       4272 ±  9%  sched_debug.cfs_rq:/.exec_clock.stddev
    433198 ± 28%     -50.6%     214019 ± 58%  sched_debug.cfs_rq:/.load.max
      0.00 ± 22%  +4.8e+24%      71.37 ±105%  sched_debug.cfs_rq:/.max_vruntime.stddev
     72913 ± 11%    +344.0%     323724 ± 12%  sched_debug.cfs_rq:/.min_vruntime.stddev
    370.32 ± 23%     -45.8%     200.85 ± 64%  sched_debug.cfs_rq:/.runnable_load_avg.max
    432768 ± 28%     -52.6%     204920 ± 65%  sched_debug.cfs_rq:/.runnable_weight.max
     70393 ±132%    -862.4%    -536655        sched_debug.cfs_rq:/.spread0.avg
    333239 ± 42%     -70.2%      99452 ± 81%  sched_debug.cfs_rq:/.spread0.max
   -139349          +878.6%   -1363651        sched_debug.cfs_rq:/.spread0.min
     72941 ± 11%    +344.6%     324292 ± 12%  sched_debug.cfs_rq:/.spread0.stddev
    276.49 ± 35%     +61.9%     447.55 ± 28%  sched_debug.cfs_rq:/.util_avg.avg
    109.18 ± 13%     +77.2%     193.51 ± 19%  sched_debug.cfs_rq:/.util_avg.stddev
     24.69 ± 30%     -43.1%      14.05 ± 18%  sched_debug.cpu.clock.stddev
     24.69 ± 30%     -43.1%      14.05 ± 18%  sched_debug.cpu.clock_task.stddev
    304.28 ± 18%     -42.8%     173.94 ± 59%  sched_debug.cpu.cpu_load[3].max
     22.83 ± 18%     -40.2%      13.66 ± 54%  sched_debug.cpu.cpu_load[3].stddev
      2.88 ±  9%     +14.8%       3.30 ±  7%  sched_debug.cpu.cpu_load[4].avg
    231.47 ± 14%     -35.1%     150.19 ± 45%  sched_debug.cpu.cpu_load[4].max
     17.23 ± 14%     -31.6%      11.79 ± 43%  sched_debug.cpu.cpu_load[4].stddev
    433198 ± 28%     -50.6%     214019 ± 58%  sched_debug.cpu.load.max
     37141 ± 29%     -47.0%      19685 ± 56%  sched_debug.cpu.load.stddev
      0.00 ± 29%     -26.6%       0.00 ± 16%  sched_debug.cpu.next_balance.stddev
      1243 ±  7%    +228.3%       4082 ± 12%  sched_debug.cpu.nr_load_updates.stddev
      0.15 ± 21%     +37.7%       0.20 ± 31%  sched_debug.cpu.nr_running.stddev
    926.31 ±  7%     -15.7%     780.48 ±  4%  sched_debug.cpu.nr_switches.min
      1739 ±  8%     +24.6%       2167 ±  8%  sched_debug.cpu.nr_switches.stddev
     -6.12           +51.0%      -9.25        sched_debug.cpu.nr_uninterruptible.min
      3408 ± 12%    +120.8%       7525 ± 20%  sched_debug.cpu.sched_goidle.max
    187.60 ± 10%     -23.7%     143.10 ± 10%  sched_debug.cpu.sched_goidle.min
    501.83 ±  2%     +49.4%     749.50 ±  9%  sched_debug.cpu.sched_goidle.stddev
      4105 ± 11%    +145.9%      10094 ± 12%  sched_debug.cpu.ttwu_count.max
    591.57 ±  5%     +66.5%     985.13 ± 11%  sched_debug.cpu.ttwu_count.stddev
    177.60 ± 11%     +60.5%     285.13 ±  7%  sched_debug.cpu.ttwu_local.stddev
      0.02 ± 27%     +33.0%       0.03 ± 12%  sched_debug.rt_rq:/.rt_time.max
      0.00 ± 20%     +51.7%       0.00 ± 17%  sched_debug.rt_rq:/.rt_time.stddev
 6.249e+09            -4.9%  5.943e+09        perf-stat.i.branch-instructions
    733533 ±  4%     -12.9%     638917 ±  3%  perf-stat.i.cache-misses
      2001           +19.5%       2392        perf-stat.i.context-switches
 2.108e+11            -5.9%  1.983e+11        perf-stat.i.cpu-cycles
    124.23           +51.2%     187.86        perf-stat.i.cpu-migrations
    404084 ±  2%     -17.4%     333693 ±  3%  perf-stat.i.cycles-between-cache-misses
   4.2e+10            -5.1%  3.984e+10        perf-stat.i.dTLB-loads
      0.06 ±  3%      -0.0        0.06 ±  2%  perf-stat.i.dTLB-store-miss-rate%
    787597            -2.2%     770036        perf-stat.i.dTLB-store-misses
  2.75e+10            -5.2%  2.606e+10        perf-stat.i.dTLB-stores
     83.86            -4.9       78.94        perf-stat.i.iTLB-load-miss-rate%
    989375            +1.9%    1008403        perf-stat.i.iTLB-load-misses
    153254 ±  6%     +31.7%     201890 ±  2%  perf-stat.i.iTLB-loads
 1.251e+11            -5.3%  1.184e+11        perf-stat.i.instructions
    406474 ±  2%     -13.6%     351010        perf-stat.i.instructions-per-iTLB-miss
      6821            +1.9%       6953        perf-stat.i.minor-faults
    279978            -9.6%     253077 ±  2%  perf-stat.i.node-load-misses
      6824            +1.9%       6954        perf-stat.i.page-faults
      0.63 ±  2%      +7.1%       0.68        perf-stat.overall.MPKI
      0.26            +0.0        0.28 ±  3%  perf-stat.overall.branch-miss-rate%
      0.94 ±  6%      -0.1        0.80 ±  3%  perf-stat.overall.cache-miss-rate%
    284356 ±  4%      +8.2%     307605 ±  3%  perf-stat.overall.cycles-between-cache-misses
      0.01            +0.0        0.01        perf-stat.overall.dTLB-load-miss-rate%
      0.00            +0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
     86.63            -3.2       83.38        perf-stat.overall.iTLB-load-miss-rate%
    124752            -7.0%     115987        perf-stat.overall.instructions-per-iTLB-miss
 6.168e+09            -4.8%  5.869e+09        perf-stat.ps.branch-instructions
    734409 ±  4%     -13.1%     638302 ±  2%  perf-stat.ps.cache-misses
      1996           +19.2%       2380        perf-stat.ps.context-switches
 2.084e+11            -5.9%  1.962e+11        perf-stat.ps.cpu-cycles
    123.49           +50.6%     185.98        perf-stat.ps.cpu-migrations
 4.154e+10            -5.1%  3.942e+10        perf-stat.ps.dTLB-loads
    789367            -2.3%     771264        perf-stat.ps.dTLB-store-misses
  2.72e+10            -5.2%  2.579e+10        perf-stat.ps.dTLB-stores
    990459            +1.9%    1008890        perf-stat.ps.iTLB-load-misses
    152951 ±  6%     +31.5%     201083 ±  2%  perf-stat.ps.iTLB-loads
 1.236e+11            -5.3%   1.17e+11        perf-stat.ps.instructions
      6818            +1.9%       6949        perf-stat.ps.minor-faults
    280027            -9.8%     252575 ±  2%  perf-stat.ps.node-load-misses
      6819            +1.9%       6949        perf-stat.ps.page-faults
 3.747e+13            -5.6%  3.536e+13        perf-stat.total.instructions
    156.25 ±  4%    +253.4%     552.25 ± 78%  interrupts.132:PCI-MSI.1574914-edge.eth3-TxRx-2
      2522 ± 24%     +31.5%       3317 ±  3%  interrupts.CPU0.NMI:Non-maskable_interrupts
      2522 ± 24%     +31.5%       3317 ±  3%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
      9419 ±  4%    +114.1%      20162 ± 15%  interrupts.CPU0.RES:Rescheduling_interrupts
      2454 ± 23%     +27.4%       3126 ±  5%  interrupts.CPU10.NMI:Non-maskable_interrupts
      2454 ± 23%     +27.4%       3126 ±  5%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      2438 ± 23%     +44.2%       3516 ± 22%  interrupts.CPU11.NMI:Non-maskable_interrupts
      2438 ± 23%     +44.2%       3516 ± 22%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
      2807 ±  2%     +11.9%       3141 ±  4%  interrupts.CPU119.NMI:Non-maskable_interrupts
      2807 ±  2%     +11.9%       3141 ±  4%  interrupts.CPU119.PMI:Performance_monitoring_interrupts
      2800 ±  2%     +12.0%       3135 ±  5%  interrupts.CPU12.NMI:Non-maskable_interrupts
      2800 ±  2%     +12.0%       3135 ±  5%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
      2800           +13.9%       3190 ±  6%  interrupts.CPU120.NMI:Non-maskable_interrupts
      2800           +13.9%       3190 ±  6%  interrupts.CPU120.PMI:Performance_monitoring_interrupts
     62.25 ± 47%    +101.6%     125.50 ± 39%  interrupts.CPU121.RES:Rescheduling_interrupts
      2800 ±  2%     +12.2%       3143 ±  5%  interrupts.CPU13.NMI:Non-maskable_interrupts
      2800 ±  2%     +12.2%       3143 ±  5%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
      2770           +14.5%       3171 ±  6%  interrupts.CPU131.NMI:Non-maskable_interrupts
      2770           +14.5%       3171 ±  6%  interrupts.CPU131.PMI:Performance_monitoring_interrupts
     63.00 ± 46%    +174.2%     172.75 ± 33%  interrupts.CPU133.RES:Rescheduling_interrupts
     73.75 ± 27%   +1000.7%     811.75 ±142%  interrupts.CPU134.RES:Rescheduling_interrupts
    118.25 ± 22%     +35.7%     160.50 ± 16%  interrupts.CPU137.RES:Rescheduling_interrupts
    108.50 ± 35%    +241.0%     370.00 ± 44%  interrupts.CPU141.RES:Rescheduling_interrupts
      2791 ±  2%     +22.5%       3420 ±  5%  interrupts.CPU145.NMI:Non-maskable_interrupts
      2791 ±  2%     +22.5%       3420 ±  5%  interrupts.CPU145.PMI:Performance_monitoring_interrupts
      2776 ±  2%     +14.8%       3188 ±  4%  interrupts.CPU15.NMI:Non-maskable_interrupts
      2776 ±  2%     +14.8%       3188 ±  4%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
      2813 ±  2%     +15.5%       3248 ±  4%  interrupts.CPU16.NMI:Non-maskable_interrupts
      2813 ±  2%     +15.5%       3248 ±  4%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
    124.50 ± 40%    +134.5%     292.00 ± 38%  interrupts.CPU16.RES:Rescheduling_interrupts
      2801 ±  2%     +11.4%       3119 ±  3%  interrupts.CPU17.NMI:Non-maskable_interrupts
      2801 ±  2%     +11.4%       3119 ±  3%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
      2802 ±  2%     +17.4%       3288 ±  9%  interrupts.CPU183.NMI:Non-maskable_interrupts
      2802 ±  2%     +17.4%       3288 ±  9%  interrupts.CPU183.PMI:Performance_monitoring_interrupts
     44.00 ± 31%    +383.5%     212.75 ± 77%  interrupts.CPU183.RES:Rescheduling_interrupts
      2809           +17.5%       3301 ± 10%  interrupts.CPU189.NMI:Non-maskable_interrupts
      2809           +17.5%       3301 ± 10%  interrupts.CPU189.PMI:Performance_monitoring_interrupts
      2848 ±  3%     +11.6%       3179        interrupts.CPU19.NMI:Non-maskable_interrupts
      2848 ±  3%     +11.6%       3179        interrupts.CPU19.PMI:Performance_monitoring_interrupts
     82.00 ± 45%    +145.7%     201.50 ± 53%  interrupts.CPU190.RES:Rescheduling_interrupts
      2797 ±  2%     +25.5%       3509 ±  8%  interrupts.CPU191.NMI:Non-maskable_interrupts
      2797 ±  2%     +25.5%       3509 ±  8%  interrupts.CPU191.PMI:Performance_monitoring_interrupts
    156.25 ±  4%    +253.4%     552.25 ± 78%  interrupts.CPU2.132:PCI-MSI.1574914-edge.eth3-TxRx-2
    668.50 ± 16%    +290.0%       2607 ± 31%  interrupts.CPU2.RES:Rescheduling_interrupts
      2852           +14.2%       3256 ±  7%  interrupts.CPU23.NMI:Non-maskable_interrupts
      2852           +14.2%       3256 ±  7%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
    190.50 ± 18%    +169.8%     514.00 ± 91%  interrupts.CPU26.RES:Rescheduling_interrupts
    570.25 ± 19%    +109.6%       1195 ± 15%  interrupts.CPU3.RES:Rescheduling_interrupts
     87.25 ± 29%    +769.6%     758.75 ±119%  interrupts.CPU35.RES:Rescheduling_interrupts
     96.00 ± 27%    +182.0%     270.75 ± 33%  interrupts.CPU38.RES:Rescheduling_interrupts
      1619 ± 96%     -90.1%     160.00 ±101%  interrupts.CPU49.RES:Rescheduling_interrupts
    157.75 ± 25%     -38.5%      97.00 ± 29%  interrupts.CPU59.RES:Rescheduling_interrupts
    140.50 ± 33%     -46.6%      75.00 ± 67%  interrupts.CPU62.RES:Rescheduling_interrupts
    152.25 ± 39%    +160.8%     397.00 ± 34%  interrupts.CPU72.RES:Rescheduling_interrupts
    176.25 ± 29%     -50.1%      88.00 ± 33%  interrupts.CPU86.RES:Rescheduling_interrupts
     60.50 ± 47%    +114.0%     129.50 ± 35%  interrupts.CPU87.RES:Rescheduling_interrupts
      2799 ±  2%     +23.8%       3466 ±  7%  interrupts.CPU89.NMI:Non-maskable_interrupts
      2799 ±  2%     +23.8%       3466 ±  7%  interrupts.CPU89.PMI:Performance_monitoring_interrupts
      2813 ±  3%     +17.3%       3300 ±  8%  interrupts.CPU95.NMI:Non-maskable_interrupts
      2813 ±  3%     +17.3%       3300 ±  8%  interrupts.CPU95.PMI:Performance_monitoring_interrupts
      2834           +14.9%       3256 ±  3%  interrupts.CPU96.NMI:Non-maskable_interrupts
      2834           +14.9%       3256 ±  3%  interrupts.CPU96.PMI:Performance_monitoring_interrupts
     51108           +36.3%      69684 ±  2%  interrupts.RES:Rescheduling_interrupts



***************************************************************************************************
lkp-bdw-ep3: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory
=========================================================================================
compiler/cpufreq_governor/ipc/kconfig/mode/nr_threads/rootfs/tbox_group/testcase/ucode:
  gcc-7/performance/pipe/x86_64-rhel-7.2/process/1600%/debian-x86_64-2018-04-03.cgz/lkp-bdw-ep3/hackbench/0xb00002e

commit: 
  24d0c1d6e6 ("sched/fair: Do not migrate due to a sync wakeup on exit")
  2c83362734 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")

24d0c1d6e65f635b 2c83362734dad8e48ccc0710b5c 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:stack_going_in_the_wrong_direction?ip=sched_slice/0x
          1:4          -25%            :4     dmesg.WARNING:stack_going_in_the_wrong_direction?ip=schedule/0x
          1:4          -25%            :4     dmesg.WARNING:stack_going_in_the_wrong_direction?ip=schedule_tail/0x
         %stddev     %change         %stddev
             \          |                \  
    494809            -7.3%     458896        hackbench.throughput
    612.95            +1.8%     624.20        hackbench.time.elapsed_time
    612.95            +1.8%     624.20        hackbench.time.elapsed_time.max
 4.886e+08           +88.4%  9.206e+08 ±  2%  hackbench.time.involuntary_context_switches
  48035824            -4.2%   46000152        hackbench.time.minor_page_faults
      7176            +1.1%       7255        hackbench.time.percent_of_cpu_this_job_got
     36530            +3.8%      37917        hackbench.time.system_time
      7459            -1.2%       7372        hackbench.time.user_time
 8.862e+08           +62.7%  1.442e+09        hackbench.time.voluntary_context_switches
 2.534e+09            -4.2%  2.429e+09        hackbench.workload
      2794 ±  8%      -9.4%       2530 ±  2%  boot-time.idle
     62869 ±  8%     -27.5%      45585 ± 26%  numa-meminfo.node1.SReclaimable
     15712 ±  8%     -27.4%      11404 ± 26%  numa-vmstat.node1.nr_slab_reclaimable
     66.75            +2.6%      68.50        vmstat.cpu.sy
   2236124           +69.2%    3782640        vmstat.system.cs
    151493           +64.8%     249683        vmstat.system.in
  25739582           +28.3%   33017943 ± 17%  cpuidle.C1.time
   2600683           +89.6%    4931044 ± 21%  cpuidle.C1.usage
 1.177e+08           -23.5%   90006774 ±  3%  cpuidle.C3.time
    400081           -26.3%     294861        cpuidle.C3.usage
  28889782 ± 37%    +210.6%   89736479 ± 39%  cpuidle.POLL.time
     32231 ±  2%    +139.5%      77205 ± 21%  cpuidle.POLL.usage
      2274            +1.7%       2314        turbostat.Avg_MHz
   2598175           +89.7%    4928921 ± 21%  turbostat.C1
    399873           -26.3%     294659        turbostat.C3
      0.22            -0.1        0.16 ±  2%  turbostat.C3%
      5.81 ±  2%     -10.8%       5.18        turbostat.CPU%c1
      0.12           -33.3%       0.08 ± 15%  turbostat.CPU%c3
  93461424           +67.4%  1.564e+08 ±  2%  turbostat.IRQ
      2.65 ±  6%     +52.8%       4.05 ±  8%  turbostat.Pkg%pc2
   1541845           +14.3%    1762636        meminfo.Active
   1480497           +14.9%    1701349        meminfo.Active(anon)
   1440429 ±  2%     +11.9%    1612079        meminfo.AnonPages
  37373539 ±  2%     +16.5%   43542569        meminfo.Committed_AS
     18778           +11.2%      20883 ±  3%  meminfo.Inactive(anon)
    510694 ±  2%     +14.5%     584945        meminfo.KernelStack
   6655144           +12.7%    7497435        meminfo.Memused
   1212773           +16.3%    1410085        meminfo.PageTables
   1202917           +12.2%    1349397        meminfo.SUnreclaim
     54584 ± 12%     +64.5%      89814 ± 10%  meminfo.Shmem
   1318488           +11.2%    1465692        meminfo.Slab
     55493 ±  2%     -28.9%      39468 ±  2%  meminfo.max_used_kB
    369842 ±  2%     +14.7%     424031        proc-vmstat.nr_active_anon
    359829 ±  2%     +11.7%     401857        proc-vmstat.nr_anon_pages
   1491001            -1.4%    1470310        proc-vmstat.nr_dirty_background_threshold
   2985649            -1.4%    2944217        proc-vmstat.nr_dirty_threshold
    234629            +3.7%     243333        proc-vmstat.nr_file_pages
  14800700            -1.4%   14593509        proc-vmstat.nr_free_pages
      4698           +10.9%       5209 ±  3%  proc-vmstat.nr_inactive_anon
    510506 ±  2%     +14.3%     583465        proc-vmstat.nr_kernel_stack
      6228            +2.9%       6408        proc-vmstat.nr_mapped
    303038 ±  2%     +15.9%     351178        proc-vmstat.nr_page_table_pages
     13683 ± 13%     +63.7%      22404 ± 11%  proc-vmstat.nr_shmem
    301079 ±  2%     +12.1%     337558        proc-vmstat.nr_slab_unreclaimable
    369842 ±  2%     +14.7%     424031        proc-vmstat.nr_zone_active_anon
      4698           +10.9%       5209 ±  3%  proc-vmstat.nr_zone_inactive_anon
    403.75 ±123%   +1057.7%       4674 ±122%  proc-vmstat.numa_hint_faults
     47.25 ± 81%   +4297.4%       2077 ±112%  proc-vmstat.numa_hint_faults_local
 6.147e+08            -7.3%  5.701e+08        proc-vmstat.numa_hit
 6.147e+08            -7.3%  5.701e+08        proc-vmstat.numa_local
     11583 ± 16%     +71.6%      19875 ± 14%  proc-vmstat.pgactivate
 6.224e+08            -7.2%  5.774e+08        proc-vmstat.pgalloc_normal
  48581147            -3.7%   46773973        proc-vmstat.pgfault
 6.221e+08            -7.2%  5.773e+08        proc-vmstat.pgfree
    400811 ±  7%     -48.8%     205202 ± 51%  sched_debug.cfs_rq:/.load.max
     63923 ± 21%     -56.9%      27565 ± 49%  sched_debug.cfs_rq:/.load.stddev
    473.56 ±  9%     -34.1%     311.94 ± 18%  sched_debug.cfs_rq:/.load_avg.max
     70.01 ± 13%     -36.1%      44.72 ± 20%  sched_debug.cfs_rq:/.load_avg.stddev
  24050559           +66.1%   39942558 ±  2%  sched_debug.cfs_rq:/.min_vruntime.avg
  27186152          +150.9%   68197622 ±  4%  sched_debug.cfs_rq:/.min_vruntime.max
  22417469           -26.5%   16476834 ±  5%  sched_debug.cfs_rq:/.min_vruntime.min
    934692 ±  9%   +2357.1%   22966409 ±  6%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.42 ±  4%     -28.3%       0.30 ±  9%  sched_debug.cfs_rq:/.nr_running.stddev
     10.48 ± 24%     -37.7%       6.53 ± 11%  sched_debug.cfs_rq:/.runnable_load_avg.avg
    343.08 ± 13%     -66.5%     114.81 ± 78%  sched_debug.cfs_rq:/.runnable_load_avg.max
     43.75 ± 18%     -64.0%      15.76 ± 57%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
    399576 ±  7%     -49.9%     200210 ± 54%  sched_debug.cfs_rq:/.runnable_weight.max
     63786 ± 21%     -57.9%      26851 ± 52%  sched_debug.cfs_rq:/.runnable_weight.stddev
   3479430 ± 23%   +1081.6%   41111920 ± 35%  sched_debug.cfs_rq:/.spread0.max
    934513 ±  9%   +2358.3%   22973201 ±  6%  sched_debug.cfs_rq:/.spread0.stddev
    329009 ±  9%     +75.8%     578522 ±  9%  sched_debug.cpu.avg_idle.avg
    100.19 ± 17%    +339.0%     439.87 ± 71%  sched_debug.cpu.clock.stddev
    100.19 ± 17%    +339.0%     439.87 ± 71%  sched_debug.cpu.clock_task.stddev
    255.50 ± 31%     -43.7%     143.92 ± 61%  sched_debug.cpu.cpu_load[0].max
     31.10 ± 29%     -40.5%      18.51 ± 48%  sched_debug.cpu.cpu_load[0].stddev
    239.47 ± 25%     -49.9%     119.92 ± 31%  sched_debug.cpu.cpu_load[2].max
     29.44 ± 24%     -46.2%      15.84 ± 24%  sched_debug.cpu.cpu_load[2].stddev
    225.33 ± 23%     -50.8%     110.81 ± 30%  sched_debug.cpu.cpu_load[3].max
     27.43 ± 23%     -45.6%      14.91 ± 22%  sched_debug.cpu.cpu_load[3].stddev
    210.61 ± 21%     -45.9%     114.03 ± 30%  sched_debug.cpu.cpu_load[4].max
     25.26 ± 21%     -40.7%      14.98 ± 20%  sched_debug.cpu.cpu_load[4].stddev
     22185 ±  8%     -20.7%      17584 ± 12%  sched_debug.cpu.curr->pid.stddev
    401289 ±  7%     -38.3%     247619 ± 32%  sched_debug.cpu.load.max
     63993 ± 21%     -51.8%      30820 ± 28%  sched_debug.cpu.load.stddev
      0.00 ± 25%    +314.4%       0.00 ± 65%  sched_debug.cpu.next_balance.stddev
      2161 ±  6%     +73.4%       3748 ±  3%  sched_debug.cpu.nr_load_updates.stddev
      1.80 ± 44%   +1219.7%      23.78 ± 53%  sched_debug.cpu.nr_running.avg
     18.56 ± 32%    +447.0%     101.50 ± 63%  sched_debug.cpu.nr_running.max
      3.54 ± 40%    +673.8%      27.43 ± 58%  sched_debug.cpu.nr_running.stddev
   7777523           +69.6%   13189331        sched_debug.cpu.nr_switches.avg
   8892283          +187.0%   25524847        sched_debug.cpu.nr_switches.max
   6863956 ±  2%     -34.3%    4508011 ±  4%  sched_debug.cpu.nr_switches.min
    399499 ±  6%   +1992.9%    8360950 ±  2%  sched_debug.cpu.nr_switches.stddev
      0.25 ± 52%   +8572.4%      21.46 ± 62%  sched_debug.cpu.nr_uninterruptible.avg
    889.83 ±  4%    +243.9%       3060 ±  4%  sched_debug.cpu.nr_uninterruptible.max
   -802.22          +323.8%      -3399        sched_debug.cpu.nr_uninterruptible.min
    348.15 ±  3%    +506.4%       2111 ±  3%  sched_debug.cpu.nr_uninterruptible.stddev
    584593 ±  2%     +14.5%     669643        slabinfo.anon_vma.active_objs
     13084 ±  2%     +14.5%      14982        slabinfo.anon_vma.active_slabs
    601913 ±  2%     +14.5%     689224        slabinfo.anon_vma.num_objs
     13084 ±  2%     +14.5%      14982        slabinfo.anon_vma.num_slabs
     63008           +10.7%      69740        slabinfo.cred_jar.active_objs
      1505           +10.9%       1669        slabinfo.cred_jar.active_slabs
     63231           +10.9%      70129        slabinfo.cred_jar.num_objs
      1505           +10.9%       1669        slabinfo.cred_jar.num_slabs
      2494 ±  7%     -20.4%       1985 ± 17%  slabinfo.eventpoll_epi.active_objs
      2494 ±  7%     -20.4%       1985 ± 17%  slabinfo.eventpoll_epi.num_objs
      4365 ±  7%     -20.4%       3474 ± 17%  slabinfo.eventpoll_pwq.active_objs
      4365 ±  7%     -20.4%       3474 ± 17%  slabinfo.eventpoll_pwq.num_objs
    777.00 ±  8%     -22.1%     605.00 ± 11%  slabinfo.file_lock_cache.active_objs
    777.00 ±  8%     -22.1%     605.00 ± 11%  slabinfo.file_lock_cache.num_objs
      4667 ±  5%      -9.5%       4226 ±  2%  slabinfo.kmalloc-128.active_objs
      4667 ±  5%      -9.5%       4226 ±  2%  slabinfo.kmalloc-128.num_objs
     63873 ±  2%     +10.2%      70416 ±  3%  slabinfo.kmalloc-96.active_objs
     36770 ±  2%     +13.3%      41645        slabinfo.mm_struct.active_objs
      2448 ±  2%     +13.3%       2774        slabinfo.mm_struct.active_slabs
     39182 ±  2%     +13.3%      44397        slabinfo.mm_struct.num_objs
      2448 ±  2%     +13.3%       2774        slabinfo.mm_struct.num_slabs
   1239866 ±  2%     +15.4%    1430445        slabinfo.pid.active_objs
     23099 ±  2%     +16.9%      27003        slabinfo.pid.active_slabs
   1478380 ±  2%     +16.9%    1728215        slabinfo.pid.num_objs
     23099 ±  2%     +16.9%      27003        slabinfo.pid.num_slabs
    347835           -12.6%     304030 ±  2%  slabinfo.selinux_file_security.active_objs
      1364           -12.7%       1191 ±  3%  slabinfo.selinux_file_security.active_slabs
    349266           -12.6%     305089 ±  3%  slabinfo.selinux_file_security.num_objs
      1364           -12.7%       1191 ±  3%  slabinfo.selinux_file_security.num_slabs
     41509           +16.7%      48454        slabinfo.sighand_cache.active_objs
      2772           +17.0%       3243        slabinfo.sighand_cache.active_slabs
     41590           +17.0%      48658        slabinfo.sighand_cache.num_objs
      2772           +17.0%       3243        slabinfo.sighand_cache.num_slabs
     44791 ±  2%     +15.0%      51511        slabinfo.signal_cache.active_objs
      1498 ±  2%     +15.3%       1726        slabinfo.signal_cache.active_slabs
     44952 ±  2%     +15.3%      51811        slabinfo.signal_cache.num_objs
      1498 ±  2%     +15.3%       1726        slabinfo.signal_cache.num_slabs
     39448 ±  2%     +17.6%      46374        slabinfo.task_struct.active_objs
     13155 ±  2%     +17.7%      15484        slabinfo.task_struct.active_slabs
     39467 ±  2%     +17.7%      46452        slabinfo.task_struct.num_objs
     13155 ±  2%     +17.7%      15484        slabinfo.task_struct.num_slabs
    886246 ±  2%     +16.0%    1027892        slabinfo.vm_area_struct.active_objs
     22632 ±  2%     +15.6%      26163        slabinfo.vm_area_struct.active_slabs
    905325 ±  2%     +15.6%    1046529        slabinfo.vm_area_struct.num_objs
     22632 ±  2%     +15.6%      26163        slabinfo.vm_area_struct.num_slabs
     36.79 ±  6%     -36.8        0.00        perf-profile.calltrace.cycles-pp.__GI___libc_write
     29.71 ±  8%     -29.7        0.00        perf-profile.calltrace.cycles-pp.__GI___libc_read
     24.00 ±  6%     -24.0        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     23.48 ±  6%     -23.5        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     22.75 ±  6%     -22.8        0.00        perf-profile.calltrace.cycles-pp.sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     21.26 ±  6%     -21.3        0.00        perf-profile.calltrace.cycles-pp.vfs_write.sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     16.73 ±  8%     -16.7        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     16.22 ±  8%     -16.2        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     15.47 ±  8%     -15.5        0.00        perf-profile.calltrace.cycles-pp.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     14.40 ±  8%     -14.4        0.00        perf-profile.calltrace.cycles-pp.vfs_read.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
      3.88 ±  3%      -1.7        2.15 ±  6%  perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_write.__vfs_write.vfs_write.sys_write
      0.65 ± 11%      +0.3        0.97 ± 28%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function
      0.68 ± 11%      +0.4        1.05 ±  7%  perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.22 ±  7%      +0.4        2.65 ±  9%  perf-profile.calltrace.cycles-pp.copy_user_enhanced_fast_string.copyout.copy_page_to_iter.pipe_read.__vfs_read
      1.10 ±  8%      +0.5        1.55 ± 11%  perf-profile.calltrace.cycles-pp.touch_atime.pipe_read.__vfs_read.vfs_read.sys_read
      0.42 ± 59%      +0.6        0.97 ± 25%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.pipe_wait.pipe_read
      0.28 ±101%      +0.6        0.86 ± 25%  perf-profile.calltrace.cycles-pp.switch_mm_irqs_off.__schedule.schedule.exit_to_usermode_loop.do_syscall_64
      1.18 ± 10%      +0.6        1.79 ± 36%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_wait.pipe_read
      1.37 ± 11%      +0.6        2.00 ± 31%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
      0.00            +0.6        0.64 ±  7%  perf-profile.calltrace.cycles-pp.__lock_text_start.__wake_up_common_lock.pipe_write.__vfs_write.vfs_write
      1.47 ± 11%      +0.7        2.17 ± 30%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
      0.30 ±101%      +0.7        0.99 ± 29%  perf-profile.calltrace.cycles-pp.__switch_to
      0.00            +0.7        0.73 ± 20%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.exit_to_usermode_loop.do_syscall_64
      0.00            +0.7        0.73 ± 20%  perf-profile.calltrace.cycles-pp.cpumask_next_wrap.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function
      0.00            +0.8        0.78 ± 10%  perf-profile.calltrace.cycles-pp.__indirect_thunk_start
      0.00            +1.0        0.96 ± 10%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_stage2
      1.28 ± 15%      +1.0        2.28 ± 26%  perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.29 ± 15%      +1.0        2.29 ± 26%  perf-profile.calltrace.cycles-pp.schedule.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.32 ± 15%      +1.0        2.35 ± 26%  perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.1        1.13 ±  8%  perf-profile.calltrace.cycles-pp.__fdget_pos.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.1        1.15 ± 11%  perf-profile.calltrace.cycles-pp.__fdget_pos.sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.71 ±  9%      +1.5        5.17 ± 35%  perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_wait.pipe_read.__vfs_read
      3.80 ±  9%      +1.5        5.28 ± 35%  perf-profile.calltrace.cycles-pp.schedule.pipe_wait.pipe_read.__vfs_read.vfs_read
      4.22 ±  9%      +1.7        5.91 ± 35%  perf-profile.calltrace.cycles-pp.pipe_wait.pipe_read.__vfs_read.vfs_read.sys_read
      2.54 ± 18%      +1.8        4.29 ± 34%  perf-profile.calltrace.cycles-pp.idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function
      4.11 ±  9%      +2.2        6.30 ± 32%  perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock
     14.37            +2.4       16.75 ±  5%  perf-profile.calltrace.cycles-pp.pipe_read.__vfs_read.vfs_read.sys_read.do_syscall_64
      3.42 ± 17%      +2.5        5.88 ± 31%  perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.autoremove_wake_function.__wake_up_common
     15.21            +2.5       17.68 ±  4%  perf-profile.calltrace.cycles-pp.__vfs_read.vfs_read.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      9.46 ±  8%      +2.5       11.97 ± 28%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.__vfs_write.vfs_write.sys_write
      7.24 ± 11%      +2.8       10.03 ± 33%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write
      7.63 ± 10%      +2.8       10.43 ± 33%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.pipe_write.__vfs_write.vfs_write
      7.29 ± 10%      +2.8       10.12 ± 33%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.pipe_write.__vfs_write
      1.45 ± 29%      +8.8       10.29 ± 14%  perf-profile.calltrace.cycles-pp._entry_trampoline
      1.54 ± 27%      +9.0       10.56 ± 14%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
      6.32 ± 18%     +17.4       23.75        perf-profile.calltrace.cycles-pp.vfs_read.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.63 ± 19%     +18.7       25.34        perf-profile.calltrace.cycles-pp.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.10 ± 17%     +20.3       30.45 ±  3%  perf-profile.calltrace.cycles-pp.vfs_write.sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     10.36 ± 17%     +21.7       32.02 ±  2%  perf-profile.calltrace.cycles-pp.sys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     19.46 ± 17%     +42.8       62.30 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
     19.69 ± 17%     +44.0       63.67 ±  2%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
     17.10           -21.3%      13.46 ±  2%  perf-stat.i.MPKI
 4.035e+10           -17.5%   3.33e+10 ±  3%  perf-stat.i.branch-instructions
      2.14            -0.2        1.95        perf-stat.i.branch-miss-rate%
 7.194e+08           -18.6%  5.855e+08 ±  3%  perf-stat.i.branch-misses
     12.70            -1.4       11.33        perf-stat.i.cache-miss-rate%
 1.014e+08           -11.3%   89967376 ±  3%  perf-stat.i.cache-misses
   3519724           +39.3%    4901677 ±  3%  perf-stat.i.context-switches
      2.06 ±  2%      -6.7%       1.92        perf-stat.i.cpi
 3.139e+11           -16.5%  2.622e+11 ±  3%  perf-stat.i.cpu-cycles
    113205 ±  2%     +72.7%     195554 ±  5%  perf-stat.i.cpu-migrations
      0.47            +0.1        0.53 ±  3%  perf-stat.i.dTLB-load-miss-rate%
 1.716e+08           +16.6%  2.001e+08 ±  5%  perf-stat.i.dTLB-load-misses
 6.393e+10           -18.1%  5.234e+10 ±  3%  perf-stat.i.dTLB-loads
   8147741           +33.5%   10877323 ±  3%  perf-stat.i.dTLB-store-misses
 4.046e+10           -18.6%  3.294e+10 ±  3%  perf-stat.i.dTLB-stores
     55.06            +1.1       56.14        perf-stat.i.iTLB-load-miss-rate%
 2.161e+08            -8.0%  1.989e+08 ±  3%  perf-stat.i.iTLB-load-misses
 1.927e+08           -14.7%  1.643e+08 ±  4%  perf-stat.i.iTLB-loads
 2.102e+11           -17.7%  1.731e+11 ±  3%  perf-stat.i.instructions
      1069 ±  2%      -8.4%     979.54 ±  3%  perf-stat.i.instructions-per-iTLB-miss
      0.56            +3.8%       0.58        perf-stat.i.ipc
    124005           -22.0%      96666 ±  3%  perf-stat.i.minor-faults
    277317           -17.9%     227661 ±  3%  perf-stat.i.msec
     63.61            -8.3       55.35 ±  2%  perf-stat.i.node-load-miss-rate%
  14378837 ±  2%      -5.7%   13566343 ±  3%  perf-stat.i.node-loads
     21.74 ±  2%      -4.3       17.40        perf-stat.i.node-store-miss-rate%
   5331671           -16.6%    4445492 ±  3%  perf-stat.i.node-store-misses
    124005           -22.0%      96666 ±  3%  perf-stat.i.page-faults
      3.76           +23.4%       4.64        perf-stat.overall.MPKI
      1.78            -0.0        1.76        perf-stat.overall.branch-miss-rate%
     12.84            -1.6       11.22        perf-stat.overall.cache-miss-rate%
      1.49            +1.5%       1.52        perf-stat.overall.cpi
      0.27            +0.1        0.38 ±  5%  perf-stat.overall.dTLB-load-miss-rate%
      0.02            +0.0        0.03        perf-stat.overall.dTLB-store-miss-rate%
     52.87            +1.9       54.77        perf-stat.overall.iTLB-load-miss-rate%
    972.65           -10.6%     869.88        perf-stat.overall.instructions-per-iTLB-miss
      0.67            -1.4%       0.66        perf-stat.overall.ipc
     55.35            +1.8       57.18        perf-stat.overall.node-load-miss-rate%
     23.25            -2.9       20.38        perf-stat.overall.node-store-miss-rate%
     32464            +6.1%      34453        perf-stat.overall.path-length
 1.579e+13            +2.0%   1.61e+13        perf-stat.total.branch-instructions
 3.969e+10            +9.6%  4.352e+10        perf-stat.total.cache-misses
  3.09e+11           +25.5%  3.879e+11        perf-stat.total.cache-references
 1.378e+09           +72.1%  2.371e+09        perf-stat.total.context-switches
 1.229e+14            +3.2%  1.268e+14        perf-stat.total.cpu-cycles
  44320452 ±  2%    +113.8%   94752867 ±  7%  perf-stat.total.cpu-migrations
 6.719e+10           +44.1%  9.685e+10 ±  5%  perf-stat.total.dTLB-load-misses
 2.502e+13            +1.1%  2.531e+13        perf-stat.total.dTLB-loads
 3.189e+09           +65.0%  5.262e+09        perf-stat.total.dTLB-store-misses
 8.459e+10           +13.7%  9.621e+10        perf-stat.total.iTLB-load-misses
 7.542e+10            +5.3%  7.945e+10        perf-stat.total.iTLB-loads
 8.228e+13            +1.7%  8.368e+13        perf-stat.total.instructions
  48538538            -3.7%   46749483        perf-stat.total.minor-faults
 1.086e+08            +1.4%  1.101e+08        perf-stat.total.msec
 6.976e+09           +25.7%  8.768e+09 ±  3%  perf-stat.total.node-load-misses
 5.627e+09           +16.6%  6.562e+09        perf-stat.total.node-loads
 2.087e+09            +3.0%   2.15e+09        perf-stat.total.node-store-misses
 6.891e+09           +21.9%    8.4e+09        perf-stat.total.node-stores
  48538482            -3.7%   46749452        perf-stat.total.page-faults
     44380 ±  9%     +36.4%      60536 ±  3%  softirqs.CPU0.RCU
     42700           +26.8%      54157        softirqs.CPU0.SCHED
     41219 ±  3%     +51.8%      62583 ±  3%  softirqs.CPU1.RCU
     40291 ±  2%     +53.7%      61915 ±  2%  softirqs.CPU10.RCU
     39712 ±  2%     +60.0%      63530 ±  2%  softirqs.CPU11.RCU
     39626           +58.3%      62721        softirqs.CPU12.RCU
     39746 ±  3%     +56.8%      62327        softirqs.CPU13.RCU
     40013 ±  2%     +62.5%      65013 ±  4%  softirqs.CPU14.RCU
     44176 ±  8%     +49.0%      65821 ±  3%  softirqs.CPU15.RCU
     43495 ±  8%     +50.4%      65412 ±  2%  softirqs.CPU16.RCU
     41320           +53.6%      63484        softirqs.CPU17.RCU
     41735           +60.7%      67060 ±  7%  softirqs.CPU18.RCU
     44197 ±  9%     +51.0%      66754 ±  2%  softirqs.CPU19.RCU
     40193 ±  3%     +52.0%      61104 ±  2%  softirqs.CPU2.RCU
     44564 ± 10%     +52.5%      67952 ±  6%  softirqs.CPU20.RCU
     42527           +55.1%      65966 ±  3%  softirqs.CPU21.RCU
     42153           +58.6%      66857 ±  5%  softirqs.CPU22.RCU
     42327 ±  3%     +68.9%      71495 ±  5%  softirqs.CPU23.RCU
     42306 ±  2%     +59.1%      67318 ±  9%  softirqs.CPU24.RCU
     41475 ±  2%     +66.9%      69214 ± 14%  softirqs.CPU25.RCU
     41655 ±  2%     +60.5%      66866 ±  3%  softirqs.CPU26.RCU
     41766 ±  2%     +51.7%      63373        softirqs.CPU27.RCU
     41548 ±  2%     +60.0%      66490 ±  5%  softirqs.CPU28.RCU
     44279 ±  6%     +50.4%      66601 ±  9%  softirqs.CPU29.RCU
     42001 ±  8%     +51.4%      63605 ±  5%  softirqs.CPU3.RCU
     40890 ±  6%     +50.8%      61645 ±  2%  softirqs.CPU30.RCU
     40491 ±  2%     +53.8%      62280 ±  3%  softirqs.CPU31.RCU
     41471 ±  2%     +56.5%      64894 ±  4%  softirqs.CPU32.RCU
     40349 ±  2%     +62.0%      65360 ±  5%  softirqs.CPU33.RCU
     40479 ±  2%     +62.5%      65781 ±  3%  softirqs.CPU34.RCU
     42319 ± 10%     +48.6%      62876 ±  3%  softirqs.CPU35.RCU
     41799           +55.1%      64841 ±  3%  softirqs.CPU36.RCU
     43731 ±  8%     +54.5%      67558 ±  3%  softirqs.CPU37.RCU
     41031 ±  3%     +73.3%      71116 ±  8%  softirqs.CPU38.RCU
     42978 ±  6%     +53.1%      65803 ±  7%  softirqs.CPU39.RCU
     39754 ±  2%     +55.0%      61636 ±  3%  softirqs.CPU4.RCU
     46367 ±  9%     +41.4%      65558 ±  9%  softirqs.CPU40.RCU
     40343 ±  2%     +59.9%      64513 ±  3%  softirqs.CPU41.RCU
     42227 ±  3%     +55.1%      65511 ±  4%  softirqs.CPU42.RCU
     40727 ±  3%     +65.6%      67457 ±  3%  softirqs.CPU43.RCU
     40253 ±  4%     +45.8%      58700 ±  2%  softirqs.CPU44.RCU
     40121 ±  2%     +52.7%      61278 ±  2%  softirqs.CPU45.RCU
     40516 ±  3%     +51.1%      61219 ±  2%  softirqs.CPU46.RCU
     41106 ± 10%     +48.5%      61035 ±  6%  softirqs.CPU47.RCU
     39554           +58.7%      62780 ±  6%  softirqs.CPU48.RCU
     38899           +53.9%      59879 ±  2%  softirqs.CPU49.RCU
     39677 ±  2%     +59.7%      63369 ±  6%  softirqs.CPU5.RCU
     42427 ±  9%     +47.9%      62754        softirqs.CPU50.RCU
     41751 ±  7%     +47.8%      61711 ±  3%  softirqs.CPU51.RCU
     39992           +65.4%      66128 ±  7%  softirqs.CPU52.RCU
     39026           +54.8%      60393 ±  3%  softirqs.CPU53.RCU
     42161 ± 10%     +47.6%      62218 ±  7%  softirqs.CPU54.RCU
     40080 ±  3%     +60.2%      64206 ±  6%  softirqs.CPU55.RCU
     39427 ±  3%     +57.5%      62094        softirqs.CPU56.RCU
     39673 ±  3%     +65.7%      65735 ±  7%  softirqs.CPU57.RCU
     40865           +53.4%      62669 ±  3%  softirqs.CPU58.RCU
     40346           +55.4%      62713 ±  3%  softirqs.CPU59.RCU
     39668 ±  2%     +58.5%      62879 ±  2%  softirqs.CPU6.RCU
     39930 ±  2%     +68.1%      67122 ±  5%  softirqs.CPU60.RCU
     40045 ±  2%     +55.1%      62093 ±  2%  softirqs.CPU61.RCU
     40057           +57.6%      63145 ±  5%  softirqs.CPU62.RCU
     40579           +57.4%      63889 ±  7%  softirqs.CPU63.RCU
     40947 ±  2%     +51.6%      62075 ±  2%  softirqs.CPU64.RCU
     42989 ±  8%     +45.6%      62592        softirqs.CPU65.RCU
     44393 ±  8%     +53.9%      68301 ±  6%  softirqs.CPU66.RCU
     41370 ±  3%     +57.9%      65307 ±  5%  softirqs.CPU67.RCU
     41532           +59.4%      66200 ±  2%  softirqs.CPU68.RCU
     40604 ±  3%     +58.8%      64470 ±  6%  softirqs.CPU69.RCU
     39822           +53.2%      61021        softirqs.CPU7.RCU
     40977 ±  2%     +58.1%      64805 ±  6%  softirqs.CPU70.RCU
     42698 ±  7%     +52.4%      65068 ±  6%  softirqs.CPU71.RCU
     41420           +70.9%      70802 ±  3%  softirqs.CPU72.RCU
     41458           +52.1%      63040 ±  2%  softirqs.CPU73.RCU
     41010 ±  2%     +64.8%      67576 ±  7%  softirqs.CPU74.RCU
     38997 ±  9%     +58.5%      61793 ±  5%  softirqs.CPU75.RCU
     37004 ±  3%     +55.5%      57533 ±  2%  softirqs.CPU76.RCU
     37324 ±  2%     +61.2%      60153 ±  6%  softirqs.CPU77.RCU
     40007 ±  8%     +45.9%      58373 ±  3%  softirqs.CPU78.RCU
     37264 ±  4%     +52.2%      56698 ±  3%  softirqs.CPU79.RCU
     39957 ±  2%     +54.1%      61588 ±  2%  softirqs.CPU8.RCU
     38811 ±  2%     +51.8%      58931 ±  2%  softirqs.CPU80.RCU
     40186 ±  6%     +53.6%      61730 ±  6%  softirqs.CPU81.RCU
     37439 ±  2%     +77.3%      66389 ±  6%  softirqs.CPU82.RCU
     37454 ±  3%     +62.9%      61016 ±  8%  softirqs.CPU83.RCU
     39574 ±  7%     +49.7%      59255 ±  3%  softirqs.CPU84.RCU
     36603 ±  2%     +77.5%      64978 ±  7%  softirqs.CPU85.RCU
     40354 ±  8%     +46.8%      59245 ±  5%  softirqs.CPU86.RCU
     37744 ±  3%     +65.6%      62495 ±  5%  softirqs.CPU87.RCU
     39090           +58.6%      62000        softirqs.CPU9.RCU
   3594895           +56.1%    5612323        softirqs.RCU
    500558           +17.5%     588306 ±  2%  softirqs.SCHED
    555.75 ± 24%     -31.4%     381.50 ± 16%  interrupts.34:PCI-MSI.3145731-edge.eth0-TxRx-2
    352056           +19.6%     421007 ±  2%  interrupts.CAL:Function_call_interrupts
      3999 ±  3%     +18.8%       4752 ±  6%  interrupts.CPU0.CAL:Function_call_interrupts
      4038           +19.2%       4812 ±  2%  interrupts.CPU1.CAL:Function_call_interrupts
      3898 ±  7%     +23.6%       4816 ±  3%  interrupts.CPU10.CAL:Function_call_interrupts
      2771 ± 32%    +160.5%       7220 ± 17%  interrupts.CPU10.NMI:Non-maskable_interrupts
      2771 ± 32%    +160.5%       7220 ± 17%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
      3957 ±  4%     +17.7%       4657 ±  7%  interrupts.CPU11.CAL:Function_call_interrupts
      2930 ± 28%    +145.8%       7204 ± 18%  interrupts.CPU11.NMI:Non-maskable_interrupts
      2930 ± 28%    +145.8%       7204 ± 18%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
      3996           +21.3%       4848 ±  2%  interrupts.CPU12.CAL:Function_call_interrupts
      2777 ± 32%    +159.5%       7207 ± 18%  interrupts.CPU12.NMI:Non-maskable_interrupts
      2777 ± 32%    +159.5%       7207 ± 18%  interrupts.CPU12.PMI:Performance_monitoring_interrupts
    555.75 ± 24%     -31.4%     381.50 ± 16%  interrupts.CPU13.34:PCI-MSI.3145731-edge.eth0-TxRx-2
      4059           +18.7%       4819 ±  3%  interrupts.CPU13.CAL:Function_call_interrupts
      2781 ± 32%    +159.2%       7207 ± 18%  interrupts.CPU13.NMI:Non-maskable_interrupts
      2781 ± 32%    +159.2%       7207 ± 18%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
      3994 ±  2%     +21.0%       4834 ±  3%  interrupts.CPU14.CAL:Function_call_interrupts
      2773 ± 32%    +168.6%       7448 ± 12%  interrupts.CPU14.NMI:Non-maskable_interrupts
      2773 ± 32%    +168.6%       7448 ± 12%  interrupts.CPU14.PMI:Performance_monitoring_interrupts
      4047           +19.2%       4823 ±  3%  interrupts.CPU15.CAL:Function_call_interrupts
      2774 ± 32%    +159.6%       7201 ± 18%  interrupts.CPU15.NMI:Non-maskable_interrupts
      2774 ± 32%    +159.6%       7201 ± 18%  interrupts.CPU15.PMI:Performance_monitoring_interrupts
      4056           +19.4%       4844 ±  3%  interrupts.CPU16.CAL:Function_call_interrupts
      2781 ± 32%    +158.8%       7198 ± 18%  interrupts.CPU16.NMI:Non-maskable_interrupts
      2781 ± 32%    +158.8%       7198 ± 18%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
      3921 ±  5%     +23.2%       4829 ±  2%  interrupts.CPU17.CAL:Function_call_interrupts
      3303 ± 31%    +118.2%       7209 ± 18%  interrupts.CPU17.NMI:Non-maskable_interrupts
      3303 ± 31%    +118.2%       7209 ± 18%  interrupts.CPU17.PMI:Performance_monitoring_interrupts
      4048           +18.8%       4808 ±  3%  interrupts.CPU18.CAL:Function_call_interrupts
      3315 ± 31%    +117.3%       7204 ± 18%  interrupts.CPU18.NMI:Non-maskable_interrupts
      3315 ± 31%    +117.3%       7204 ± 18%  interrupts.CPU18.PMI:Performance_monitoring_interrupts
      4056           +18.4%       4803 ±  3%  interrupts.CPU19.CAL:Function_call_interrupts
      3316 ± 31%    +117.2%       7204 ± 18%  interrupts.CPU19.NMI:Non-maskable_interrupts
      3316 ± 31%    +117.2%       7204 ± 18%  interrupts.CPU19.PMI:Performance_monitoring_interrupts
      4021 ±  2%     +20.5%       4844 ±  2%  interrupts.CPU2.CAL:Function_call_interrupts
      3307 ± 31%    +117.9%       7204 ± 18%  interrupts.CPU20.NMI:Non-maskable_interrupts
      3307 ± 31%    +117.9%       7204 ± 18%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
      4039           +19.5%       4824 ±  3%  interrupts.CPU21.CAL:Function_call_interrupts
      3307 ± 31%    +117.9%       7206 ± 18%  interrupts.CPU21.NMI:Non-maskable_interrupts
      3307 ± 31%    +117.9%       7206 ± 18%  interrupts.CPU21.PMI:Performance_monitoring_interrupts
      4018           +17.6%       4725 ±  4%  interrupts.CPU22.CAL:Function_call_interrupts
      3306 ± 31%    +117.1%       7179 ± 19%  interrupts.CPU22.NMI:Non-maskable_interrupts
      3306 ± 31%    +117.1%       7179 ± 19%  interrupts.CPU22.PMI:Performance_monitoring_interrupts
      4060           +18.1%       4794        interrupts.CPU23.CAL:Function_call_interrupts
      3853 ± 23%     +86.0%       7169 ± 19%  interrupts.CPU23.NMI:Non-maskable_interrupts
      3853 ± 23%     +86.0%       7169 ± 19%  interrupts.CPU23.PMI:Performance_monitoring_interrupts
      4009 ±  2%     +20.7%       4838 ±  3%  interrupts.CPU24.CAL:Function_call_interrupts
      3840 ± 23%     +86.7%       7172 ± 19%  interrupts.CPU24.NMI:Non-maskable_interrupts
      3840 ± 23%     +86.7%       7172 ± 19%  interrupts.CPU24.PMI:Performance_monitoring_interrupts
      3886 ±  5%     +22.9%       4775 ±  3%  interrupts.CPU25.CAL:Function_call_interrupts
      3874 ± 23%     +85.2%       7175 ± 19%  interrupts.CPU25.NMI:Non-maskable_interrupts
      3874 ± 23%     +85.2%       7175 ± 19%  interrupts.CPU25.PMI:Performance_monitoring_interrupts
      4008           +19.2%       4779 ±  4%  interrupts.CPU26.CAL:Function_call_interrupts
      3848 ± 23%     +86.4%       7173 ± 19%  interrupts.CPU26.NMI:Non-maskable_interrupts
      3848 ± 23%     +86.4%       7173 ± 19%  interrupts.CPU26.PMI:Performance_monitoring_interrupts
      4051           +19.6%       4846 ±  2%  interrupts.CPU27.CAL:Function_call_interrupts
      3307 ± 31%    +116.9%       7175 ± 19%  interrupts.CPU27.NMI:Non-maskable_interrupts
      3307 ± 31%    +116.9%       7175 ± 19%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
      3885 ±  6%     +23.4%       4793        interrupts.CPU28.CAL:Function_call_interrupts
      3305 ± 31%    +117.4%       7186 ± 18%  interrupts.CPU28.NMI:Non-maskable_interrupts
      3305 ± 31%    +117.4%       7186 ± 18%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
      3303 ± 31%    +117.4%       7182 ± 19%  interrupts.CPU29.NMI:Non-maskable_interrupts
      3303 ± 31%    +117.4%       7182 ± 19%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
      4041           +19.6%       4831 ±  2%  interrupts.CPU3.CAL:Function_call_interrupts
      3753 ± 13%     +30.1%       4882 ±  2%  interrupts.CPU30.CAL:Function_call_interrupts
      3848 ± 23%     +86.4%       7173 ± 19%  interrupts.CPU30.NMI:Non-maskable_interrupts
      3848 ± 23%     +86.4%       7173 ± 19%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
      3954           +22.5%       4845 ±  2%  interrupts.CPU31.CAL:Function_call_interrupts
      3851 ± 24%     +86.1%       7165 ± 19%  interrupts.CPU31.NMI:Non-maskable_interrupts
      3851 ± 24%     +86.1%       7165 ± 19%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
      4032           +19.4%       4814 ±  2%  interrupts.CPU32.CAL:Function_call_interrupts
      3898 ±  6%     +23.8%       4825 ±  3%  interrupts.CPU33.CAL:Function_call_interrupts
      3919 ±  2%     +20.4%       4718 ±  6%  interrupts.CPU34.CAL:Function_call_interrupts
      3863 ± 24%     +85.7%       7173 ± 19%  interrupts.CPU34.NMI:Non-maskable_interrupts
      3863 ± 24%     +85.7%       7173 ± 19%  interrupts.CPU34.PMI:Performance_monitoring_interrupts
      3993           +16.2%       4641 ±  5%  interrupts.CPU35.CAL:Function_call_interrupts
      3855 ± 24%     +85.9%       7168 ± 19%  interrupts.CPU35.NMI:Non-maskable_interrupts
      3855 ± 24%     +85.9%       7168 ± 19%  interrupts.CPU35.PMI:Performance_monitoring_interrupts
      3911 ±  2%     +23.2%       4817 ±  2%  interrupts.CPU36.CAL:Function_call_interrupts
      3856 ± 24%     +85.9%       7171 ± 19%  interrupts.CPU36.NMI:Non-maskable_interrupts
      3856 ± 24%     +85.9%       7171 ± 19%  interrupts.CPU36.PMI:Performance_monitoring_interrupts
      3928 ±  4%     +22.3%       4802 ±  2%  interrupts.CPU37.CAL:Function_call_interrupts
      4026           +15.0%       4629 ±  6%  interrupts.CPU38.CAL:Function_call_interrupts
      4034           +19.7%       4829        interrupts.CPU39.CAL:Function_call_interrupts
      4036           +19.1%       4805 ±  3%  interrupts.CPU4.CAL:Function_call_interrupts
      4031           +20.5%       4856 ±  2%  interrupts.CPU40.CAL:Function_call_interrupts
      4035           +17.9%       4758 ±  5%  interrupts.CPU41.CAL:Function_call_interrupts
      3919 ±  4%     +20.0%       4704 ±  6%  interrupts.CPU42.CAL:Function_call_interrupts
      3950           +19.2%       4710 ±  4%  interrupts.CPU43.CAL:Function_call_interrupts
      3931 ±  7%     +22.5%       4816 ±  3%  interrupts.CPU44.CAL:Function_call_interrupts
      2836 ± 32%    +119.7%       6232 ± 28%  interrupts.CPU44.NMI:Non-maskable_interrupts
      2836 ± 32%    +119.7%       6232 ± 28%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
      4018           +20.5%       4841 ±  2%  interrupts.CPU45.CAL:Function_call_interrupts
      2770 ± 32%    +124.5%       6218 ± 28%  interrupts.CPU45.NMI:Non-maskable_interrupts
      2770 ± 32%    +124.5%       6218 ± 28%  interrupts.CPU45.PMI:Performance_monitoring_interrupts
      4043           +19.1%       4817 ±  3%  interrupts.CPU46.CAL:Function_call_interrupts
      2778 ± 32%    +102.1%       5616 ± 42%  interrupts.CPU46.NMI:Non-maskable_interrupts
      2778 ± 32%    +102.1%       5616 ± 42%  interrupts.CPU46.PMI:Performance_monitoring_interrupts
      4064           +19.3%       4846 ±  3%  interrupts.CPU47.CAL:Function_call_interrupts
      2771 ± 32%    +138.2%       6600 ± 35%  interrupts.CPU47.NMI:Non-maskable_interrupts
      2771 ± 32%    +138.2%       6600 ± 35%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
      4037           +19.5%       4825 ±  3%  interrupts.CPU48.CAL:Function_call_interrupts
      2774 ± 32%    +102.6%       5622 ± 42%  interrupts.CPU48.NMI:Non-maskable_interrupts
      2774 ± 32%    +102.6%       5622 ± 42%  interrupts.CPU48.PMI:Performance_monitoring_interrupts
      4054           +18.6%       4809 ±  3%  interrupts.CPU49.CAL:Function_call_interrupts
      2769 ± 32%    +102.9%       5618 ± 42%  interrupts.CPU49.NMI:Non-maskable_interrupts
      2769 ± 32%    +102.9%       5618 ± 42%  interrupts.CPU49.PMI:Performance_monitoring_interrupts
      4036           +15.9%       4680 ±  6%  interrupts.CPU5.CAL:Function_call_interrupts
      4056           +19.0%       4826 ±  3%  interrupts.CPU50.CAL:Function_call_interrupts
      2770 ± 32%    +138.3%       6600 ± 35%  interrupts.CPU50.NMI:Non-maskable_interrupts
      2770 ± 32%    +138.3%       6600 ± 35%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
      4039           +19.6%       4832 ±  3%  interrupts.CPU51.CAL:Function_call_interrupts
      2774 ± 32%    +137.9%       6602 ± 35%  interrupts.CPU51.NMI:Non-maskable_interrupts
      2774 ± 32%    +137.9%       6602 ± 35%  interrupts.CPU51.PMI:Performance_monitoring_interrupts
      4059           +18.5%       4811 ±  3%  interrupts.CPU52.CAL:Function_call_interrupts
      2768 ± 32%    +138.5%       6602 ± 35%  interrupts.CPU52.NMI:Non-maskable_interrupts
      2768 ± 32%    +138.5%       6602 ± 35%  interrupts.CPU52.PMI:Performance_monitoring_interrupts
      4040           +18.2%       4775 ±  3%  interrupts.CPU53.CAL:Function_call_interrupts
      2768 ± 32%    +138.4%       6599 ± 35%  interrupts.CPU53.NMI:Non-maskable_interrupts
      2768 ± 32%    +138.4%       6599 ± 35%  interrupts.CPU53.PMI:Performance_monitoring_interrupts
      3982 ±  4%     +21.5%       4839 ±  2%  interrupts.CPU54.CAL:Function_call_interrupts
      4036 ±  2%     +16.0%       4681 ±  7%  interrupts.CPU55.CAL:Function_call_interrupts
      4002 ±  2%     +17.5%       4701 ±  7%  interrupts.CPU56.CAL:Function_call_interrupts
      3319 ± 31%    +117.0%       7201 ± 18%  interrupts.CPU56.NMI:Non-maskable_interrupts
      3319 ± 31%    +117.0%       7201 ± 18%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
      4049           +18.9%       4816 ±  4%  interrupts.CPU57.CAL:Function_call_interrupts
      3312 ± 31%    +117.3%       7197 ± 18%  interrupts.CPU57.NMI:Non-maskable_interrupts
      3312 ± 31%    +117.3%       7197 ± 18%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
      3973 ±  3%     +22.3%       4861 ±  2%  interrupts.CPU58.CAL:Function_call_interrupts
      3314 ± 31%    +139.8%       7948        interrupts.CPU58.NMI:Non-maskable_interrupts
      3314 ± 31%    +139.8%       7948        interrupts.CPU58.PMI:Performance_monitoring_interrupts
      4046           +20.2%       4861 ±  2%  interrupts.CPU59.CAL:Function_call_interrupts
      2779 ± 32%    +158.8%       7191 ± 18%  interrupts.CPU59.NMI:Non-maskable_interrupts
      2779 ± 32%    +158.8%       7191 ± 18%  interrupts.CPU59.PMI:Performance_monitoring_interrupts
      4030           +20.2%       4845 ±  2%  interrupts.CPU6.CAL:Function_call_interrupts
      3315 ± 32%    +117.2%       7199 ± 18%  interrupts.CPU6.NMI:Non-maskable_interrupts
      3315 ± 32%    +117.2%       7199 ± 18%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
      4047           +19.0%       4817 ±  3%  interrupts.CPU60.CAL:Function_call_interrupts
      2782 ± 32%    +158.6%       7194 ± 18%  interrupts.CPU60.NMI:Non-maskable_interrupts
      2782 ± 32%    +158.6%       7194 ± 18%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
      3935 ±  5%     +22.3%       4812 ±  3%  interrupts.CPU61.CAL:Function_call_interrupts
      2775 ± 32%    +159.4%       7200 ± 18%  interrupts.CPU61.NMI:Non-maskable_interrupts
      2775 ± 32%    +159.4%       7200 ± 18%  interrupts.CPU61.PMI:Performance_monitoring_interrupts
      4055           +19.9%       4861 ±  2%  interrupts.CPU62.CAL:Function_call_interrupts
      2776 ± 32%    +159.3%       7198 ± 18%  interrupts.CPU62.NMI:Non-maskable_interrupts
      2776 ± 32%    +159.3%       7198 ± 18%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
      4055           +19.0%       4827 ±  3%  interrupts.CPU63.CAL:Function_call_interrupts
      2781 ± 32%    +158.8%       7197 ± 18%  interrupts.CPU63.NMI:Non-maskable_interrupts
      2781 ± 32%    +158.8%       7197 ± 18%  interrupts.CPU63.PMI:Performance_monitoring_interrupts
      4044           +16.9%       4729 ±  5%  interrupts.CPU64.CAL:Function_call_interrupts
      2770 ± 32%    +159.7%       7195 ± 18%  interrupts.CPU64.NMI:Non-maskable_interrupts
      2770 ± 32%    +159.7%       7195 ± 18%  interrupts.CPU64.PMI:Performance_monitoring_interrupts
      4040           +19.6%       4833 ±  3%  interrupts.CPU65.CAL:Function_call_interrupts
      2782 ± 32%    +158.9%       7204 ± 18%  interrupts.CPU65.NMI:Non-maskable_interrupts
      2782 ± 32%    +158.9%       7204 ± 18%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
      4047           +15.9%       4691 ±  6%  interrupts.CPU66.CAL:Function_call_interrupts
      2778 ± 32%    +158.0%       7168 ± 19%  interrupts.CPU66.NMI:Non-maskable_interrupts
      2778 ± 32%    +158.0%       7168 ± 19%  interrupts.CPU66.PMI:Performance_monitoring_interrupts
      4069           +13.7%       4626 ±  4%  interrupts.CPU67.CAL:Function_call_interrupts
      2783 ± 31%    +157.4%       7165 ± 19%  interrupts.CPU67.NMI:Non-maskable_interrupts
      2783 ± 31%    +157.4%       7165 ± 19%  interrupts.CPU67.PMI:Performance_monitoring_interrupts
      4045           +19.9%       4851 ±  2%  interrupts.CPU68.CAL:Function_call_interrupts
      2779 ± 32%    +157.8%       7166 ± 19%  interrupts.CPU68.NMI:Non-maskable_interrupts
      2779 ± 32%    +157.8%       7166 ± 19%  interrupts.CPU68.PMI:Performance_monitoring_interrupts
      3897 ±  5%     +23.9%       4828 ±  3%  interrupts.CPU69.CAL:Function_call_interrupts
      2784 ± 31%    +157.6%       7171 ± 19%  interrupts.CPU69.NMI:Non-maskable_interrupts
      2784 ± 31%    +157.6%       7171 ± 19%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
      4045           +19.7%       4841 ±  2%  interrupts.CPU7.CAL:Function_call_interrupts
      2774 ± 32%    +159.5%       7200 ± 18%  interrupts.CPU7.NMI:Non-maskable_interrupts
      2774 ± 32%    +159.5%       7200 ± 18%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
      4040           +17.1%       4730 ±  5%  interrupts.CPU70.CAL:Function_call_interrupts
      2784 ± 32%    +157.7%       7174 ± 18%  interrupts.CPU70.NMI:Non-maskable_interrupts
      2784 ± 32%    +157.7%       7174 ± 18%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
      4051           +19.3%       4833 ±  2%  interrupts.CPU71.CAL:Function_call_interrupts
      2784 ± 32%    +157.5%       7169 ± 19%  interrupts.CPU71.NMI:Non-maskable_interrupts
      2784 ± 32%    +157.5%       7169 ± 19%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
      2785 ± 33%    +159.1%       7217 ± 17%  interrupts.CPU72.NMI:Non-maskable_interrupts
      2785 ± 33%    +159.1%       7217 ± 17%  interrupts.CPU72.PMI:Performance_monitoring_interrupts
      3999 ±  3%     +17.1%       4682 ±  5%  interrupts.CPU73.CAL:Function_call_interrupts
      2780 ± 32%    +157.8%       7169 ± 19%  interrupts.CPU73.NMI:Non-maskable_interrupts
      2780 ± 32%    +157.8%       7169 ± 19%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
      3834 ±  8%     +26.8%       4861 ±  2%  interrupts.CPU74.CAL:Function_call_interrupts
      2778 ± 32%    +158.4%       7177 ± 19%  interrupts.CPU74.NMI:Non-maskable_interrupts
      2778 ± 32%    +158.4%       7177 ± 19%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
      4063           +19.1%       4840 ±  3%  interrupts.CPU75.CAL:Function_call_interrupts
      3331 ± 32%    +115.3%       7171 ± 19%  interrupts.CPU75.NMI:Non-maskable_interrupts
      3331 ± 32%    +115.3%       7171 ± 19%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
      4021           +19.4%       4801 ±  3%  interrupts.CPU76.CAL:Function_call_interrupts
      3461 ± 26%    +108.5%       7214 ± 18%  interrupts.CPU76.NMI:Non-maskable_interrupts
      3461 ± 26%    +108.5%       7214 ± 18%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
      3984 ±  3%     +20.9%       4819 ±  3%  interrupts.CPU77.CAL:Function_call_interrupts
      3653 ± 39%     +96.4%       7176 ± 19%  interrupts.CPU77.NMI:Non-maskable_interrupts
      3653 ± 39%     +96.4%       7176 ± 19%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
      4024           +18.9%       4786 ±  4%  interrupts.CPU78.CAL:Function_call_interrupts
      2782 ± 32%    +158.1%       7181 ± 19%  interrupts.CPU78.NMI:Non-maskable_interrupts
      2782 ± 32%    +158.1%       7181 ± 19%  interrupts.CPU78.PMI:Performance_monitoring_interrupts
      3897 ±  3%     +22.8%       4787 ±  2%  interrupts.CPU79.CAL:Function_call_interrupts
      2784 ± 32%    +157.7%       7174 ± 19%  interrupts.CPU79.NMI:Non-maskable_interrupts
      2784 ± 32%    +157.7%       7174 ± 19%  interrupts.CPU79.PMI:Performance_monitoring_interrupts
      4024 ±  2%     +19.5%       4808 ±  3%  interrupts.CPU8.CAL:Function_call_interrupts
      2770 ± 32%    +159.9%       7201 ± 18%  interrupts.CPU8.NMI:Non-maskable_interrupts
      2770 ± 32%    +159.9%       7201 ± 18%  interrupts.CPU8.PMI:Performance_monitoring_interrupts
      3710 ± 10%     +30.5%       4840 ±  2%  interrupts.CPU80.CAL:Function_call_interrupts
      3323 ± 32%    +115.8%       7172 ± 19%  interrupts.CPU80.NMI:Non-maskable_interrupts
      3323 ± 32%    +115.8%       7172 ± 19%  interrupts.CPU80.PMI:Performance_monitoring_interrupts
      3997           +18.4%       4733 ±  2%  interrupts.CPU81.CAL:Function_call_interrupts
      3311 ± 32%    +117.4%       7198 ± 18%  interrupts.CPU81.NMI:Non-maskable_interrupts
      3311 ± 32%    +117.4%       7198 ± 18%  interrupts.CPU81.PMI:Performance_monitoring_interrupts
      4049           +17.6%       4763 ±  2%  interrupts.CPU82.CAL:Function_call_interrupts
      3319 ± 32%    +116.1%       7172 ± 19%  interrupts.CPU82.NMI:Non-maskable_interrupts
      3319 ± 32%    +116.1%       7172 ± 19%  interrupts.CPU82.PMI:Performance_monitoring_interrupts
      4013           +20.3%       4826 ±  2%  interrupts.CPU83.CAL:Function_call_interrupts
      3318 ± 32%    +116.9%       7199 ± 18%  interrupts.CPU83.NMI:Non-maskable_interrupts
      3318 ± 32%    +116.9%       7199 ± 18%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
      4040           +19.6%       4834 ±  2%  interrupts.CPU84.CAL:Function_call_interrupts
      3345 ± 32%    +114.5%       7175 ± 19%  interrupts.CPU84.NMI:Non-maskable_interrupts
      3345 ± 32%    +114.5%       7175 ± 19%  interrupts.CPU84.PMI:Performance_monitoring_interrupts
      3991 ±  2%     +19.8%       4781 ±  4%  interrupts.CPU86.CAL:Function_call_interrupts
      3350 ± 33%    +113.7%       7162 ± 19%  interrupts.CPU86.NMI:Non-maskable_interrupts
      3350 ± 33%    +113.7%       7162 ± 19%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
      3326 ± 32%    +117.0%       7216 ± 17%  interrupts.CPU87.NMI:Non-maskable_interrupts
      3326 ± 32%    +117.0%       7216 ± 17%  interrupts.CPU87.PMI:Performance_monitoring_interrupts
      4058           +19.1%       4835 ±  3%  interrupts.CPU9.CAL:Function_call_interrupts
      2782 ± 32%    +159.5%       7220 ± 18%  interrupts.CPU9.NMI:Non-maskable_interrupts
      2782 ± 32%    +159.5%       7220 ± 18%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
     51.00 ± 59%    +150.0%     127.50 ± 23%  interrupts.IWI:IRQ_work_interrupts
    283310 ± 21%    +114.7%     608405 ± 20%  interrupts.NMI:Non-maskable_interrupts
    283310 ± 21%    +114.7%     608405 ± 20%  interrupts.PMI:Performance_monitoring_interrupts
  38196883          +161.0%   99690427 ±  2%  interrupts.RES:Rescheduling_interrupts



***************************************************************************************************
lkp-skl-2sp4: 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.2/100%/debian-x86_64-2018-04-03.cgz/300s/lkp-skl-2sp4/custom/reaim/0x200004d

commit: 
  24d0c1d6e6 ("sched/fair: Do not migrate due to a sync wakeup on exit")
  2c83362734 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")

24d0c1d6e65f635b 2c83362734dad8e48ccc0710b5c 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          2:4          -50%            :4     kmsg.do_IRQ:#No_irq_handler_for_vector
           :4           25%           1:4     dmesg.WARNING:at#for_ip_error_entry/0x
           :4           25%           1:4     dmesg.WARNING:at#for_ip_ret_from_intr/0x
           :4           25%           1:4     dmesg.WARNING:stack_going_in_the_wrong_direction?ip=__slab_free/0x
          2:4           17%           2:4     perf-profile.calltrace.cycles-pp.error_entry
          2:4           24%           3:4     perf-profile.children.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
      4.77           +11.4%       5.32 ±  2%  reaim.std_dev_percent
      0.03            +8.2%       0.03        reaim.std_dev_time
   2025938           -10.6%    1811037        reaim.time.involuntary_context_switches
   8797312            +1.0%    8887670        reaim.time.voluntary_context_switches
      9078           +12.0%      10169        meminfo.PageTables
      2269           +11.6%       2531        proc-vmstat.nr_page_table_pages
     41893            +6.7%      44683 ±  4%  softirqs.CPU98.RCU
      4.65 ±  2%     -15.3%       3.94 ± 10%  turbostat.Pkg%pc6
    178.24            +2.0%     181.79        turbostat.PkgWatt
     12123 ± 18%     +33.1%      16134 ±  3%  numa-meminfo.node0.Mapped
     41009 ±  5%     +12.4%      46109 ±  4%  numa-meminfo.node0.SReclaimable
    131166 ±  5%      +4.8%     137483 ±  5%  numa-meminfo.node0.SUnreclaim
    172176 ±  2%      +6.6%     183593 ±  3%  numa-meminfo.node0.Slab
      3014 ± 18%     +34.4%       4052 ±  2%  numa-vmstat.node0.nr_mapped
     10257 ±  6%     +12.4%      11530 ±  4%  numa-vmstat.node0.nr_slab_reclaimable
     32780 ±  5%      +4.9%      34383 ±  5%  numa-vmstat.node0.nr_slab_unreclaimable
      3664 ± 16%     -25.6%       2725 ±  2%  numa-vmstat.node1.nr_mapped
      1969 ±  4%      +2.7%       2022 ±  5%  slabinfo.kmalloc-4096.active_objs
      5584            +9.8%       6130 ±  3%  slabinfo.mm_struct.active_objs
      5614            +9.7%       6158 ±  3%  slabinfo.mm_struct.num_objs
    877.00           +11.5%     978.25 ±  3%  slabinfo.names_cache.active_objs
    877.00           +11.5%     978.25 ±  3%  slabinfo.names_cache.num_objs
      1495            -9.8%       1348 ±  5%  slabinfo.nsproxy.active_objs
      1495            -9.8%       1348 ±  5%  slabinfo.nsproxy.num_objs
      1313 ± 15%     +27.1%       1668 ±  2%  sched_debug.cpu.nr_load_updates.stddev
      2664           +44.4%       3847 ± 14%  sched_debug.cpu.nr_switches.stddev
      0.02 ± 77%     -89.5%       0.00 ±223%  sched_debug.cpu.nr_uninterruptible.avg
     81.22 ± 12%     +30.5%     106.02 ±  8%  sched_debug.cpu.nr_uninterruptible.max
     30.10           +53.8%      46.29 ±  7%  sched_debug.cpu.nr_uninterruptible.stddev
      2408 ±  3%     +52.2%       3665 ± 18%  sched_debug.cpu.sched_count.stddev
      1066 ±  6%     +68.1%       1792 ± 17%  sched_debug.cpu.sched_goidle.stddev
      1413 ±  3%     +36.5%       1929 ± 19%  sched_debug.cpu.ttwu_count.stddev
    553.65 ±  2%      -8.6%     505.96 ±  2%  sched_debug.cpu.ttwu_local.stddev
      3.19            -0.2        2.99 ±  2%  perf-stat.i.branch-miss-rate%
      7.45 ±  2%      -0.7        6.79 ±  2%  perf-stat.i.cache-miss-rate%
  56644639           -11.5%   50150131        perf-stat.i.cache-misses
 5.795e+10            +1.5%  5.883e+10        perf-stat.i.cpu-cycles
      0.33            -0.0        0.29 ±  3%  perf-stat.i.dTLB-load-miss-rate%
      0.10            -0.0        0.09 ±  3%  perf-stat.i.dTLB-store-miss-rate%
     44.46            -1.5       42.99 ±  2%  perf-stat.i.iTLB-load-miss-rate%
   3761774            +5.5%    3969797        perf-stat.i.iTLB-load-misses
     79.67            -4.1       75.54        perf-stat.i.node-load-miss-rate%
  16697015           -17.2%   13819973        perf-stat.i.node-load-misses
   2830435            -2.3%    2764205        perf-stat.i.node-loads
     60.87 ±  2%      -6.0       54.85 ±  3%  perf-stat.i.node-store-miss-rate%
   3631713           -15.0%    3088593 ±  3%  perf-stat.i.node-store-misses
   2047418            +7.2%    2195246        perf-stat.i.node-stores
      6.61            -0.9        5.71        perf-stat.overall.cache-miss-rate%
     44.36            +0.7       45.06        perf-stat.overall.iTLB-load-miss-rate%
     11359            -3.1%      11011        perf-stat.overall.instructions-per-iTLB-miss
     85.51            -2.2       83.33        perf-stat.overall.node-load-miss-rate%
     63.95            -5.5       58.44        perf-stat.overall.node-store-miss-rate%
 1.782e+10           -13.1%  1.548e+10        perf-stat.total.cache-misses
   5160156            -1.5%    5081874        perf-stat.total.cpu-migrations
 1.184e+09            +3.5%  1.225e+09        perf-stat.total.iTLB-load-misses
 5.254e+09           -18.8%  4.266e+09        perf-stat.total.node-load-misses
 8.905e+08            -4.2%  8.532e+08        perf-stat.total.node-loads
 1.143e+09           -16.6%  9.537e+08 ±  3%  perf-stat.total.node-store-misses
 6.442e+08            +5.2%  6.776e+08        perf-stat.total.node-stores
      2.51            -0.3        2.23        perf-profile.calltrace.cycles-pp.copy_page_range.copy_process._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.07 ±  2%      -0.2        0.84 ±  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.release_pages.tlb_flush_mmu_free.arch_tlb_finish_mmu.tlb_finish_mmu
      1.04 ±  2%      -0.2        0.81 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.release_pages.tlb_flush_mmu_free.arch_tlb_finish_mmu
      1.37 ±  3%      -0.2        1.21 ±  9%  perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      0.88 ±  6%      -0.1        0.77 ±  6%  perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.flush_old_exec.load_elf_binary
      0.81 ±  6%      -0.1        0.72 ±  6%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.flush_old_exec
      0.60 ±  8%      +0.1        0.72 ± 16%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_read_lock_slowpath.do_wait.kernel_wait4.SYSC_wait4
      0.17 ±141%      +0.4        0.57 ±  3%  perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.do_exit.do_group_exit.__wake_up_parent.do_syscall_64
      1.29 ± 10%      +0.4        1.70 ± 12%  perf-profile.calltrace.cycles-pp.do_wait.kernel_wait4.SYSC_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
     25.70            +1.0       26.66        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      4.15 ± 23%      -0.9        3.20 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock
     14.46            -0.5       14.00 ±  2%  perf-profile.children.cycles-pp.exit_mmap
     10.89            -0.5       10.43        perf-profile.children.cycles-pp._do_fork
      2.46            -0.4        2.09 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      3.90            -0.3        3.56 ±  5%  perf-profile.children.cycles-pp.apic_timer_interrupt
      3.87            -0.3        3.53 ±  5%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
     10.20            -0.3        9.88        perf-profile.children.cycles-pp.copy_process
      2.53            -0.3        2.24 ±  2%  perf-profile.children.cycles-pp.copy_page_range
      1.64 ±  5%      -0.2        1.41 ±  2%  perf-profile.children.cycles-pp.get_page_from_freelist
      1.73 ±  4%      -0.2        1.51        perf-profile.children.cycles-pp.__alloc_pages_nodemask
      1.29 ±  3%      -0.2        1.10 ±  4%  perf-profile.children.cycles-pp.__schedule
      1.22 ±  8%      -0.2        1.04 ±  5%  perf-profile.children.cycles-pp.ret_from_fork
      0.59 ±  3%      -0.1        0.45 ±  5%  perf-profile.children.cycles-pp.pick_next_task_fair
      0.43 ±  2%      -0.1        0.30 ±  6%  perf-profile.children.cycles-pp.free_pgd_range
      0.64            -0.1        0.51 ±  4%  perf-profile.children.cycles-pp.wake_up_new_task
      0.30            -0.1        0.18 ±  6%  perf-profile.children.cycles-pp.free_unref_page
      0.98 ±  2%      -0.1        0.86 ±  2%  perf-profile.children.cycles-pp.rcu_process_callbacks
      0.50 ±  4%      -0.1        0.38 ±  4%  perf-profile.children.cycles-pp.load_balance
      0.51 ±  4%      -0.1        0.40 ±  5%  perf-profile.children.cycles-pp.__list_del_entry_valid
      1.22            -0.1        1.10 ±  4%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.39 ±  9%      -0.1        0.28 ±  7%  perf-profile.children.cycles-pp.schedule_tail
      0.29            -0.1        0.18 ±  4%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.35 ±  4%      -0.1        0.25 ±  3%  perf-profile.children.cycles-pp.do_task_dead
      0.88            -0.1        0.78 ±  2%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.35            -0.1        0.26 ±  4%  perf-profile.children.cycles-pp.free_unref_page_commit
      0.13 ±  7%      -0.1        0.04 ± 58%  perf-profile.children.cycles-pp.sched_ttwu_pending
      1.04            -0.1        0.96 ±  3%  perf-profile.children.cycles-pp.kmem_cache_free
      0.70 ±  2%      -0.1        0.62 ±  3%  perf-profile.children.cycles-pp.__pte_alloc
      0.21 ±  3%      -0.1        0.14 ±  7%  perf-profile.children.cycles-pp.idle_cpu
      0.32 ±  6%      -0.1        0.25 ±  6%  perf-profile.children.cycles-pp.find_busiest_group
      0.36 ±  4%      -0.1        0.29 ±  5%  perf-profile.children.cycles-pp.finish_task_switch
      0.97            -0.1        0.91        perf-profile.children.cycles-pp.anon_vma_interval_tree_insert
      0.32            -0.1        0.25 ±  6%  perf-profile.children.cycles-pp.mm_init
      0.28 ±  3%      -0.1        0.22 ±  4%  perf-profile.children.cycles-pp.pgd_alloc
      0.14 ± 10%      -0.1        0.08 ±  8%  perf-profile.children.cycles-pp.free_one_page
      0.56 ±  4%      -0.1        0.50 ±  4%  perf-profile.children.cycles-pp.schedule
      0.23 ±  2%      -0.1        0.18 ±  8%  perf-profile.children.cycles-pp.__get_free_pages
      0.81 ±  2%      -0.1        0.75 ±  3%  perf-profile.children.cycles-pp.__slab_free
      0.31 ±  9%      -0.1        0.25 ±  7%  perf-profile.children.cycles-pp.__put_user_4
      0.19 ±  2%      -0.1        0.14 ±  5%  perf-profile.children.cycles-pp.dup_userfaultfd
      0.15 ±  6%      -0.1        0.10 ±  9%  perf-profile.children.cycles-pp.__free_pages_ok
      2.25            -0.0        2.20        perf-profile.children.cycles-pp.anon_vma_clone
      0.08 ±  5%      -0.0        0.04 ± 60%  perf-profile.children.cycles-pp.unfreeze_partials
      1.00            -0.0        0.96        perf-profile.children.cycles-pp.sys_write
      0.20 ±  4%      -0.0        0.16 ± 13%  perf-profile.children.cycles-pp.devkmsg_write
      0.20 ±  4%      -0.0        0.16 ± 13%  perf-profile.children.cycles-pp.printk_emit
      0.21 ±  3%      -0.0        0.18 ±  6%  perf-profile.children.cycles-pp.update_curr
      0.89            -0.0        0.86 ±  2%  perf-profile.children.cycles-pp.__vfs_write
      0.09            -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.new_slab
      0.16 ±  7%      -0.0        0.13 ± 11%  perf-profile.children.cycles-pp.__mmdrop
      0.09 ±  9%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.put_cpu_partial
      0.44            -0.0        0.41        perf-profile.children.cycles-pp.remove_vma
      0.52            -0.0        0.49 ±  3%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.07            -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.lapic_next_deadline
      0.20 ±  4%      -0.0        0.18 ±  6%  perf-profile.children.cycles-pp.__put_task_struct
      0.17 ±  7%      -0.0        0.15 ±  5%  perf-profile.children.cycles-pp.__lock_text_start
      0.14 ±  5%      -0.0        0.12 ±  5%  perf-profile.children.cycles-pp.do_send_sig_info
      0.09            -0.0        0.07        perf-profile.children.cycles-pp.entry_SYSCALL_64_stage2
      0.13 ±  6%      -0.0        0.11 ±  3%  perf-profile.children.cycles-pp.put_task_stack
      0.09 ±  9%      -0.0        0.07 ±  5%  perf-profile.children.cycles-pp.__queue_work
      0.14 ±  3%      -0.0        0.12 ±  5%  perf-profile.children.cycles-pp.unmap_single_vma
      0.10            -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.tcp_sendmsg_locked
      0.28 ±  4%      +0.0        0.31 ±  2%  perf-profile.children.cycles-pp.vfs_statx
      0.26 ±  4%      +0.0        0.29 ±  3%  perf-profile.children.cycles-pp.SYSC_newstat
      0.56 ±  3%      +0.0        0.59 ±  3%  perf-profile.children.cycles-pp.elf_map
      0.45 ±  5%      +0.0        0.48 ±  3%  perf-profile.children.cycles-pp.__wake_up_common
      1.12            +0.2        1.32 ±  3%  perf-profile.children.cycles-pp.queued_read_lock_slowpath
      1.26 ±  2%      +0.2        1.49 ±  3%  perf-profile.children.cycles-pp.queued_write_lock_slowpath
      2.16 ±  2%      +0.2        2.39 ±  2%  perf-profile.children.cycles-pp.do_wait
      2.19 ±  2%      +0.2        2.43 ±  2%  perf-profile.children.cycles-pp.SYSC_wait4
      2.18 ±  2%      +0.2        2.42 ±  2%  perf-profile.children.cycles-pp.kernel_wait4
     25.95            +1.0       26.92        perf-profile.children.cycles-pp.intel_idle
      1.46            -0.1        1.31 ±  3%  perf-profile.self.cycles-pp.copy_page_range
      0.51 ±  4%      -0.1        0.40 ±  5%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.21 ±  3%      -0.1        0.14 ±  7%  perf-profile.self.cycles-pp.idle_cpu
      0.96            -0.1        0.89        perf-profile.self.cycles-pp.anon_vma_interval_tree_insert
      0.19            -0.1        0.14 ±  3%  perf-profile.self.cycles-pp.dup_userfaultfd
      0.35 ±  2%      -0.0        0.30 ±  5%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.25 ±  5%      -0.0        0.20 ±  6%  perf-profile.self.cycles-pp.find_busiest_group
      0.28 ±  2%      -0.0        0.24 ±  3%  perf-profile.self.cycles-pp.unlink_anon_vmas
      0.57            -0.0        0.53 ±  3%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.15 ±  3%      -0.0        0.11 ±  7%  perf-profile.self.cycles-pp.free_pcppages_bulk
      0.55            -0.0        0.52 ±  2%  perf-profile.self.cycles-pp.__slab_free
      0.09            -0.0        0.07        perf-profile.self.cycles-pp.entry_SYSCALL_64_stage2
      0.07 ± 11%      -0.0        0.05        perf-profile.self.cycles-pp.update_rq_clock
      0.14 ±  6%      +0.0        0.16        perf-profile.self.cycles-pp.handle_mm_fault
     25.94            +1.0       26.91        perf-profile.self.cycles-pp.intel_idle
      2592 ± 11%     -20.4%       2062 ± 11%  interrupts.CPU1.RES:Rescheduling_interrupts
      1584           -20.9%       1254 ±  8%  interrupts.CPU10.RES:Rescheduling_interrupts
      1405 ±  3%     -21.0%       1110 ±  2%  interrupts.CPU100.RES:Rescheduling_interrupts
      1420 ±  4%     -23.8%       1082 ± 10%  interrupts.CPU101.RES:Rescheduling_interrupts
      1421 ±  2%     -19.7%       1141 ± 11%  interrupts.CPU102.RES:Rescheduling_interrupts
      1501 ± 27%     +35.5%       2033 ±  8%  interrupts.CPU103.NMI:Non-maskable_interrupts
      1501 ± 27%     +35.5%       2033 ±  8%  interrupts.CPU103.PMI:Performance_monitoring_interrupts
      1394           -23.0%       1074 ±  5%  interrupts.CPU103.RES:Rescheduling_interrupts
      1566           -19.1%       1266 ± 11%  interrupts.CPU11.RES:Rescheduling_interrupts
      1531           -17.2%       1267 ±  7%  interrupts.CPU12.RES:Rescheduling_interrupts
      1559 ±  2%     -22.6%       1206 ±  8%  interrupts.CPU13.RES:Rescheduling_interrupts
      1503           -23.4%       1151 ±  6%  interrupts.CPU15.RES:Rescheduling_interrupts
      1584 ±  3%     -24.2%       1201 ±  8%  interrupts.CPU16.RES:Rescheduling_interrupts
      1528 ±  6%     -18.8%       1240 ± 13%  interrupts.CPU17.RES:Rescheduling_interrupts
      1518 ±  3%     -21.1%       1197 ±  6%  interrupts.CPU18.RES:Rescheduling_interrupts
      2303 ± 11%     -19.5%       1854        interrupts.CPU19.NMI:Non-maskable_interrupts
      2303 ± 11%     -19.5%       1854        interrupts.CPU19.PMI:Performance_monitoring_interrupts
      1457 ±  4%     -18.0%       1194 ±  4%  interrupts.CPU19.RES:Rescheduling_interrupts
      1884 ±  5%     -15.3%       1596 ±  4%  interrupts.CPU2.RES:Rescheduling_interrupts
      1543 ±  4%     -22.9%       1189 ±  7%  interrupts.CPU20.RES:Rescheduling_interrupts
      1480           -19.4%       1193 ±  5%  interrupts.CPU21.RES:Rescheduling_interrupts
      1492 ±  2%     -17.5%       1231 ±  6%  interrupts.CPU22.RES:Rescheduling_interrupts
      1482 ±  2%     -17.0%       1230 ±  7%  interrupts.CPU24.RES:Rescheduling_interrupts
      1434 ±  3%     -17.4%       1184 ±  6%  interrupts.CPU25.RES:Rescheduling_interrupts
      1568 ±  4%     -12.7%       1368 ±  4%  interrupts.CPU26.RES:Rescheduling_interrupts
      1544           -16.5%       1289 ±  3%  interrupts.CPU27.RES:Rescheduling_interrupts
      1486 ±  3%     -16.6%       1238 ±  5%  interrupts.CPU29.RES:Rescheduling_interrupts
      1856 ±  3%     -14.3%       1591 ±  8%  interrupts.CPU3.RES:Rescheduling_interrupts
      1507           -18.8%       1224 ±  9%  interrupts.CPU30.RES:Rescheduling_interrupts
      1561 ±  2%     -19.9%       1250 ±  3%  interrupts.CPU31.RES:Rescheduling_interrupts
      1551           -23.4%       1187 ±  3%  interrupts.CPU32.RES:Rescheduling_interrupts
      1449 ±  4%     -16.6%       1208 ±  9%  interrupts.CPU33.RES:Rescheduling_interrupts
      1521 ±  2%     -21.6%       1193 ±  6%  interrupts.CPU34.RES:Rescheduling_interrupts
      1644 ± 15%     -26.2%       1212 ±  7%  interrupts.CPU35.RES:Rescheduling_interrupts
      1498           -22.5%       1161 ±  2%  interrupts.CPU36.RES:Rescheduling_interrupts
      1487 ±  3%     -19.8%       1192 ±  4%  interrupts.CPU37.RES:Rescheduling_interrupts
      1538 ±  4%     -25.0%       1153 ±  5%  interrupts.CPU38.RES:Rescheduling_interrupts
      1486           -20.6%       1181 ±  7%  interrupts.CPU39.RES:Rescheduling_interrupts
      1488 ±  2%     -20.2%       1187 ±  3%  interrupts.CPU40.RES:Rescheduling_interrupts
      1503           -22.3%       1168 ± 10%  interrupts.CPU41.RES:Rescheduling_interrupts
      1560 ± 23%     +24.5%       1942 ±  4%  interrupts.CPU43.NMI:Non-maskable_interrupts
      1560 ± 23%     +24.5%       1942 ±  4%  interrupts.CPU43.PMI:Performance_monitoring_interrupts
      1654 ±  7%     -26.5%       1216 ±  4%  interrupts.CPU43.RES:Rescheduling_interrupts
      1501 ±  5%     -23.5%       1148 ±  4%  interrupts.CPU44.RES:Rescheduling_interrupts
      1473 ±  3%     -21.0%       1164 ±  7%  interrupts.CPU45.RES:Rescheduling_interrupts
      1424 ±  3%     -18.0%       1167 ±  6%  interrupts.CPU46.RES:Rescheduling_interrupts
      1481           -25.1%       1109 ±  4%  interrupts.CPU47.RES:Rescheduling_interrupts
      1436           -19.8%       1152 ±  4%  interrupts.CPU48.RES:Rescheduling_interrupts
      1688 ±  2%     -20.2%       1347 ±  8%  interrupts.CPU5.RES:Rescheduling_interrupts
      1440 ±  2%     -20.9%       1139 ±  7%  interrupts.CPU50.RES:Rescheduling_interrupts
      1462           -23.5%       1118 ±  7%  interrupts.CPU51.RES:Rescheduling_interrupts
      1410 ±  2%     -14.7%       1203 ±  5%  interrupts.CPU52.RES:Rescheduling_interrupts
      1524 ±  2%     -24.6%       1149 ±  5%  interrupts.CPU53.RES:Rescheduling_interrupts
      1438           -16.5%       1201 ±  9%  interrupts.CPU54.RES:Rescheduling_interrupts
      1454 ±  2%     -19.5%       1170 ±  6%  interrupts.CPU55.RES:Rescheduling_interrupts
      1468           -20.1%       1173 ±  4%  interrupts.CPU56.RES:Rescheduling_interrupts
      1461           -20.6%       1159 ±  4%  interrupts.CPU57.RES:Rescheduling_interrupts
      1410 ±  2%     -18.1%       1155 ±  4%  interrupts.CPU58.RES:Rescheduling_interrupts
      1452 ±  3%     -19.0%       1176 ±  6%  interrupts.CPU59.RES:Rescheduling_interrupts
      1621 ±  4%     -16.3%       1357 ±  5%  interrupts.CPU6.RES:Rescheduling_interrupts
      1455 ±  2%     -22.7%       1124 ±  8%  interrupts.CPU60.RES:Rescheduling_interrupts
      1491 ±  3%     -25.8%       1106 ± 11%  interrupts.CPU61.RES:Rescheduling_interrupts
      1401           -18.4%       1143 ±  3%  interrupts.CPU62.RES:Rescheduling_interrupts
      1429           -19.4%       1152 ±  9%  interrupts.CPU63.RES:Rescheduling_interrupts
      1437           -22.8%       1109 ±  8%  interrupts.CPU64.RES:Rescheduling_interrupts
      1499           -26.4%       1104 ±  7%  interrupts.CPU65.RES:Rescheduling_interrupts
      1485 ±  4%     -23.0%       1144 ±  7%  interrupts.CPU66.RES:Rescheduling_interrupts
      1405 ±  3%     -19.0%       1138 ±  8%  interrupts.CPU67.RES:Rescheduling_interrupts
      1492 ±  2%     -22.4%       1159 ± 12%  interrupts.CPU68.RES:Rescheduling_interrupts
      1435 ±  4%     -19.9%       1149 ± 14%  interrupts.CPU69.RES:Rescheduling_interrupts
      1625 ±  3%     -15.6%       1371 ±  6%  interrupts.CPU7.RES:Rescheduling_interrupts
      1480 ±  3%     -21.4%       1164 ± 12%  interrupts.CPU70.RES:Rescheduling_interrupts
      2355 ± 10%     -30.9%       1627 ± 25%  interrupts.CPU71.NMI:Non-maskable_interrupts
      2355 ± 10%     -30.9%       1627 ± 25%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
      1428 ±  3%     -19.4%       1151 ±  8%  interrupts.CPU71.RES:Rescheduling_interrupts
      1427 ±  2%     -19.7%       1145 ±  9%  interrupts.CPU72.RES:Rescheduling_interrupts
      1452 ±  4%     -17.5%       1198 ±  7%  interrupts.CPU73.RES:Rescheduling_interrupts
      1419 ±  2%     -19.0%       1149 ±  6%  interrupts.CPU74.RES:Rescheduling_interrupts
      1441 ±  2%     -18.4%       1176 ±  9%  interrupts.CPU75.RES:Rescheduling_interrupts
      1435 ±  3%     -16.1%       1204 ±  6%  interrupts.CPU76.RES:Rescheduling_interrupts
      1445           -22.2%       1124 ±  6%  interrupts.CPU77.RES:Rescheduling_interrupts
      1481 ±  4%     -23.8%       1128 ±  8%  interrupts.CPU78.RES:Rescheduling_interrupts
      1392           -20.7%       1104 ±  9%  interrupts.CPU79.RES:Rescheduling_interrupts
      1621 ±  4%     -22.7%       1252 ±  9%  interrupts.CPU8.RES:Rescheduling_interrupts
      1478 ±  2%     -24.3%       1118 ±  6%  interrupts.CPU80.RES:Rescheduling_interrupts
      1481 ±  4%     -23.2%       1137 ±  8%  interrupts.CPU81.RES:Rescheduling_interrupts
      1453 ±  2%     -20.8%       1151 ±  4%  interrupts.CPU82.RES:Rescheduling_interrupts
      1431           -22.5%       1110 ± 10%  interrupts.CPU83.RES:Rescheduling_interrupts
      1477 ±  4%     -25.9%       1094 ±  7%  interrupts.CPU84.RES:Rescheduling_interrupts
      1467 ±  2%     -21.4%       1153 ±  6%  interrupts.CPU85.RES:Rescheduling_interrupts
      1427 ±  3%     -20.1%       1140 ± 12%  interrupts.CPU86.RES:Rescheduling_interrupts
      1512 ±  5%     -25.5%       1126 ±  5%  interrupts.CPU87.RES:Rescheduling_interrupts
      1409           -20.8%       1115 ±  5%  interrupts.CPU88.RES:Rescheduling_interrupts
      1408 ±  2%     -18.8%       1143 ±  5%  interrupts.CPU89.RES:Rescheduling_interrupts
      1659 ±  2%     -22.0%       1294 ± 11%  interrupts.CPU9.RES:Rescheduling_interrupts
      1475 ±  3%     -23.7%       1126 ±  5%  interrupts.CPU90.RES:Rescheduling_interrupts
      1444           -22.8%       1114 ±  7%  interrupts.CPU91.RES:Rescheduling_interrupts
      1442 ±  6%     -21.3%       1135 ±  7%  interrupts.CPU92.RES:Rescheduling_interrupts
      1466 ±  2%     -21.7%       1148 ±  2%  interrupts.CPU93.RES:Rescheduling_interrupts
      1413 ±  2%     -17.7%       1163 ±  6%  interrupts.CPU94.RES:Rescheduling_interrupts
      1611 ±  3%     -29.8%       1131 ±  5%  interrupts.CPU95.RES:Rescheduling_interrupts
      1406           -20.1%       1123 ±  5%  interrupts.CPU96.RES:Rescheduling_interrupts
      1386 ±  3%     -20.0%       1109 ±  9%  interrupts.CPU97.RES:Rescheduling_interrupts
      1406 ±  4%     -21.6%       1102 ±  7%  interrupts.CPU98.RES:Rescheduling_interrupts
      1379 ±  4%     -18.9%       1118 ±  8%  interrupts.CPU99.RES:Rescheduling_interrupts
    163356           -19.1%     132229        interrupts.RES:Rescheduling_interrupts



***************************************************************************************************
lkp-skl-2sp4: 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.2/100%/debian-x86_64-2018-04-03.cgz/300s/lkp-skl-2sp4/alltests/reaim/0x200004d

commit: 
  24d0c1d6e6 ("sched/fair: Do not migrate due to a sync wakeup on exit")
  2c83362734 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")

24d0c1d6e65f635b 2c83362734dad8e48ccc0710b5c 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.WARNING:at#for_ip_error_entry/0x
           :4           25%           1:4     dmesg.WARNING:stack_going_in_the_wrong_direction?ip=__slab_free/0x
          1:4            9%           1:4     perf-profile.children.cycles-pp.error_entry
          0:4            4%           0:4     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
      8.37            -4.0%       8.03        reaim.child_systime
      0.42            +2.4%       0.43        reaim.std_dev_percent
    294600            -7.7%     271774        reaim.time.involuntary_context_switches
    329.71            -4.0%     316.42        reaim.time.system_time
   1675279            +1.3%    1696674        reaim.time.voluntary_context_switches
 4.516e+08 ±  6%     +17.8%  5.322e+08 ± 10%  cpuidle.POLL.time
    200189 ±  3%     +10.4%     220923        meminfo.AnonHugePages
    154.84            +0.8%     156.13        turbostat.PkgWatt
     64.74 ± 20%     +95.3%     126.45 ± 19%  boot-time.boot
      6459 ± 21%     +99.3%      12874 ± 20%  boot-time.idle
     40.50 ± 43%     +96.3%      79.50 ± 32%  numa-vmstat.node1.nr_anon_transparent_hugepages
     12122 ±  7%     -20.2%       9674 ± 14%  numa-vmstat.node1.nr_slab_reclaimable
    153053 ±  3%     +10.6%     169298 ±  7%  numa-meminfo.node0.Slab
     84083 ± 42%     +95.4%     164331 ± 32%  numa-meminfo.node1.AnonHugePages
     48467 ±  7%     -20.2%      38679 ± 14%  numa-meminfo.node1.SReclaimable
     97.25 ±  3%     +10.8%     107.75        proc-vmstat.nr_anon_transparent_hugepages
     26303            -1.8%      25827        proc-vmstat.nr_kernel_stack
      4731            -5.2%       4485 ±  3%  proc-vmstat.nr_page_table_pages
      2463 ±134%     -91.1%     220.50 ± 98%  proc-vmstat.numa_hint_faults
     16885 ±  6%      -9.6%      15258 ±  4%  softirqs.CPU32.RCU
     30855 ± 13%     -14.3%      26453        softirqs.CPU4.TIMER
     17409 ±  7%     -12.5%      15231        softirqs.CPU51.RCU
     18157 ±  9%     -12.9%      15809 ±  2%  softirqs.CPU55.RCU
     16588 ± 10%     -11.6%      14662 ±  2%  softirqs.CPU78.RCU
     17307 ±  7%     -12.6%      15130 ±  2%  softirqs.CPU87.RCU
     17146 ±  9%     -13.2%      14880 ±  2%  softirqs.CPU90.RCU
      2593 ±  6%     -17.3%       2145 ± 10%  slabinfo.Acpi-ParseExt.active_objs
      2593 ±  6%     -17.3%       2145 ± 10%  slabinfo.Acpi-ParseExt.num_objs
    682.50 ±  6%     -17.1%     565.50 ±  5%  slabinfo.bdev_cache.active_objs
    682.50 ±  6%     -17.1%     565.50 ±  5%  slabinfo.bdev_cache.num_objs
      6102 ±  5%     -30.6%       4234 ± 13%  slabinfo.eventpoll_epi.active_objs
      6102 ±  5%     -30.6%       4234 ± 13%  slabinfo.eventpoll_epi.num_objs
      5340 ±  5%     -30.6%       3704 ± 13%  slabinfo.eventpoll_pwq.active_objs
      5340 ±  5%     -30.6%       3704 ± 13%  slabinfo.eventpoll_pwq.num_objs
      1018 ±  9%     -17.6%     839.00 ±  2%  slabinfo.file_lock_cache.active_objs
      1018 ±  9%     -17.6%     839.00 ±  2%  slabinfo.file_lock_cache.num_objs
      3359 ±  7%     -12.7%       2933 ±  8%  slabinfo.fsnotify_mark_connector.active_objs
      3359 ±  7%     -12.7%       2933 ±  8%  slabinfo.fsnotify_mark_connector.num_objs
      1485 ±  3%     -10.9%       1323 ±  6%  slabinfo.nsproxy.active_objs
      1485 ±  3%     -10.9%       1323 ±  6%  slabinfo.nsproxy.num_objs
      1.67            -0.0        1.64        perf-stat.branch-miss-rate%
 1.383e+10            -1.6%  1.361e+10        perf-stat.branch-misses
      7.52            -1.0        6.54        perf-stat.cache-miss-rate%
 3.062e+09           -13.7%  2.643e+09        perf-stat.cache-misses
 1.041e+13            +2.4%  1.066e+13        perf-stat.cpu-cycles
    744682            -1.2%     735818        perf-stat.cpu-migrations
 4.046e+08            +3.2%  4.177e+08        perf-stat.iTLB-load-misses
 4.821e+08            +3.3%   4.98e+08        perf-stat.iTLB-loads
     16668            -2.8%      16200        perf-stat.instructions-per-iTLB-miss
     87.92            -2.1       85.80        perf-stat.node-load-miss-rate%
 8.016e+08           -20.8%  6.351e+08        perf-stat.node-load-misses
 1.102e+08            -4.6%  1.051e+08        perf-stat.node-loads
     59.53            -6.5       53.07        perf-stat.node-store-miss-rate%
 1.435e+08           -24.7%  1.081e+08 ±  2%  perf-stat.node-store-misses
  97539783            -2.0%   95550500        perf-stat.node-stores
    552.25 ± 27%     +67.3%     923.75 ± 24%  interrupts.CPU10.NMI:Non-maskable_interrupts
    552.25 ± 27%     +67.3%     923.75 ± 24%  interrupts.CPU10.PMI:Performance_monitoring_interrupts
    455.50           +32.9%     605.50 ± 19%  interrupts.CPU15.RES:Rescheduling_interrupts
    361.75 ±  6%     +58.9%     574.75 ± 23%  interrupts.CPU26.RES:Rescheduling_interrupts
    321.25 ±  7%     +22.0%     392.00 ±  6%  interrupts.CPU30.RES:Rescheduling_interrupts
    278.25 ±  9%     +18.1%     328.75 ± 13%  interrupts.CPU41.RES:Rescheduling_interrupts
    746.75 ± 11%     +60.5%       1198 ± 37%  interrupts.CPU44.NMI:Non-maskable_interrupts
    746.75 ± 11%     +60.5%       1198 ± 37%  interrupts.CPU44.PMI:Performance_monitoring_interrupts
    645.25 ± 32%     +43.0%     922.50 ± 13%  interrupts.CPU47.NMI:Non-maskable_interrupts
    645.25 ± 32%     +43.0%     922.50 ± 13%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
    631.25 ± 23%     +37.4%     867.25 ± 12%  interrupts.CPU58.NMI:Non-maskable_interrupts
    631.25 ± 23%     +37.4%     867.25 ± 12%  interrupts.CPU58.PMI:Performance_monitoring_interrupts
    713.50 ± 12%     +22.2%     871.75 ± 10%  interrupts.CPU65.NMI:Non-maskable_interrupts
    713.50 ± 12%     +22.2%     871.75 ± 10%  interrupts.CPU65.PMI:Performance_monitoring_interrupts
    620.00 ± 14%     +95.4%       1211 ± 56%  interrupts.CPU72.NMI:Non-maskable_interrupts
    620.00 ± 14%     +95.4%       1211 ± 56%  interrupts.CPU72.PMI:Performance_monitoring_interrupts
    620.75 ± 30%     +72.8%       1072 ± 33%  interrupts.CPU83.NMI:Non-maskable_interrupts
    620.75 ± 30%     +72.8%       1072 ± 33%  interrupts.CPU83.PMI:Performance_monitoring_interrupts
    779.83 ±  4%     +53.9%       1200 ± 16%  sched_debug.cfs_rq:/.exec_clock.stddev
     43531 ±  4%     +69.1%      73628 ±  7%  sched_debug.cfs_rq:/.min_vruntime.stddev
      1.00            +9.3%       1.09 ±  3%  sched_debug.cfs_rq:/.nr_spread_over.avg
      2.39 ± 75%    +256.7%       8.54 ± 53%  sched_debug.cfs_rq:/.removed.load_avg.avg
     16.68 ± 62%    +171.4%      45.26 ± 35%  sched_debug.cfs_rq:/.removed.load_avg.stddev
    110.67 ± 75%    +254.3%     392.14 ± 53%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
    771.19 ± 62%    +170.1%       2082 ± 35%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
     43530 ±  4%     +69.1%      73628 ±  7%  sched_debug.cfs_rq:/.spread0.stddev
    216468 ±  5%     +36.6%     295649 ±  6%  sched_debug.cpu.clock.avg
    216487 ±  5%     +36.6%     295669 ±  6%  sched_debug.cpu.clock.max
    216210 ±  5%     +36.6%     295436 ±  6%  sched_debug.cpu.clock.min
    216468 ±  5%     +36.6%     295649 ±  6%  sched_debug.cpu.clock_task.avg
    216487 ±  5%     +36.6%     295669 ±  6%  sched_debug.cpu.clock_task.max
    216210 ±  5%     +36.6%     295436 ±  6%  sched_debug.cpu.clock_task.min
      1593 ± 11%      -7.6%       1473 ±  5%  sched_debug.cpu.curr->pid.avg
     34247           +11.9%      38310 ±  6%  sched_debug.cpu.nr_switches.max
      2145 ±  2%     +19.3%       2559 ±  5%  sched_debug.cpu.nr_switches.stddev
     31588           +16.2%      36707 ±  6%  sched_debug.cpu.sched_count.max
      1791 ±  3%     +27.7%       2288 ±  7%  sched_debug.cpu.sched_count.stddev
     13661           +18.6%      16205 ±  8%  sched_debug.cpu.sched_goidle.max
    838.65 ±  4%     +25.7%       1054 ±  9%  sched_debug.cpu.sched_goidle.stddev
     12457 ±  3%     +33.1%      16586 ± 13%  sched_debug.cpu.ttwu_count.max
    887.04 ±  3%     +37.4%       1218 ± 13%  sched_debug.cpu.ttwu_count.stddev
    264.20 ±  4%     +25.9%     332.56 ± 11%  sched_debug.cpu.ttwu_local.stddev
    216473 ±  5%     +36.6%     295655 ±  6%  sched_debug.cpu_clk
    216473 ±  5%     +36.6%     295655 ±  6%  sched_debug.ktime
    217713 ±  5%     +36.3%     296841 ±  6%  sched_debug.sched_clk
      0.54 ±  4%      +0.1        0.67 ±  8%  perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      1.60 ±  5%      +0.3        1.89 ±  9%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      1.40 ± 12%      +0.9        2.29 ± 17%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.tick_do_update_jiffies64.tick_irq_enter.irq_enter
      1.60 ± 12%      +1.0        2.60 ± 17%  perf-profile.calltrace.cycles-pp._raw_spin_lock.tick_do_update_jiffies64.tick_irq_enter.irq_enter.smp_apic_timer_interrupt
      1.61 ± 12%      +1.0        2.62 ± 17%  perf-profile.calltrace.cycles-pp.tick_do_update_jiffies64.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt
      3.06 ±  6%      +1.3        4.36 ± 12%  perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state
      3.13 ±  5%      +1.3        4.45 ± 12%  perf-profile.calltrace.cycles-pp.irq_enter.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle
      5.58 ±  2%      +1.7        7.32 ± 12%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry
      5.60 ±  2%      +1.8        7.36 ± 12%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      0.07 ± 17%      +0.0        0.10 ±  8%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
      0.10 ±  4%      +0.0        0.13 ± 11%  perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
      0.10 ±  4%      +0.0        0.14 ± 12%  perf-profile.children.cycles-pp.vfs_statx
      0.09 ±  4%      +0.0        0.12 ± 13%  perf-profile.children.cycles-pp.SYSC_newstat
      0.06 ± 14%      +0.0        0.09 ± 24%  perf-profile.children.cycles-pp.__vmalloc_node_range
      0.11 ± 14%      +0.0        0.14 ± 25%  perf-profile.children.cycles-pp.tlb_flush_mmu_tlbonly
      0.06 ± 63%      +0.0        0.10 ± 18%  perf-profile.children.cycles-pp.terminate_walk
      0.00            +0.1        0.06 ± 14%  perf-profile.children.cycles-pp.find_vmap_area
      0.01 ±173%      +0.1        0.08 ± 11%  perf-profile.children.cycles-pp.remove_vm_area
      0.00            +0.1        0.07 ± 38%  perf-profile.children.cycles-pp.__get_vm_area_node
      0.18 ± 11%      +0.1        0.26 ± 15%  perf-profile.children.cycles-pp.creat
      0.58 ±  5%      +0.1        0.73 ±  8%  perf-profile.children.cycles-pp.ktime_get_update_offsets_now
      1.57 ± 11%      +1.0        2.52 ± 16%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      1.66 ± 14%      +1.0        2.67 ± 16%  perf-profile.children.cycles-pp.tick_do_update_jiffies64
      2.28 ±  7%      +1.1        3.35 ± 14%  perf-profile.children.cycles-pp._raw_spin_lock
      3.15 ±  7%      +1.3        4.44 ± 12%  perf-profile.children.cycles-pp.tick_irq_enter
      3.21 ±  6%      +1.3        4.53 ± 12%  perf-profile.children.cycles-pp.irq_enter
      6.39            +1.8        8.22 ± 11%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      6.42            +1.9        8.29 ± 11%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.07 ± 13%      +0.0        0.08 ± 13%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
      0.12 ± 22%      +0.0        0.16 ± 21%  perf-profile.self.cycles-pp.do_syscall_64
      0.27 ±  8%      +0.1        0.35 ± 12%  perf-profile.self.cycles-pp.tick_irq_enter
      0.55 ±  4%      +0.1        0.69 ±  9%  perf-profile.self.cycles-pp.ktime_get_update_offsets_now
      1.57 ± 11%      +1.0        2.52 ± 16%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath



***************************************************************************************************
lkp-skl-2sp4: 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/testcase/ucode:
  gcc-7/performance/x86_64-rhel-7.2/50%/debian-x86_64-2018-04-03.cgz/300s/lkp-skl-2sp4/pft/0x200004d

commit: 
  24d0c1d6e6 ("sched/fair: Do not migrate due to a sync wakeup on exit")
  2c83362734 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")

24d0c1d6e65f635b 2c83362734dad8e48ccc0710b5c 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_error_entry/0x
          1:4          -25%            :4     dmesg.WARNING:at#for_ip_ret_from_intr/0x
         %stddev     %change         %stddev
             \          |                \  
    230070           -42.7%     131828        pft.faults_per_sec_per_cpu
     24154            -4.1%      23160 ±  3%  pft.time.involuntary_context_switches
   7817260           -32.7%    5262150        pft.time.minor_page_faults
      4048           +18.0%       4778        pft.time.percent_of_cpu_this_job_got
     11917           +16.9%      13933        pft.time.system_time
    250.86           +73.7%     435.77 ± 10%  pft.time.user_time
    142515           +59.4%     227173 ±  5%  pft.time.voluntary_context_switches
   7334244 ±  7%     -64.3%    2618153 ± 65%  numa-numastat.node1.local_node
   7344576 ±  7%     -64.2%    2628307 ± 65%  numa-numastat.node1.numa_hit
     59.50           -11.3%      52.75        vmstat.cpu.id
     39.75           +20.1%      47.75        vmstat.procs.r
     59.92            -6.7       53.19        mpstat.cpu.idle%
      0.00 ± 26%      +0.0        0.01 ±  7%  mpstat.cpu.soft%
     39.08            +6.1       45.22        mpstat.cpu.sys%
      0.99            +0.6        1.58 ± 10%  mpstat.cpu.usr%
   4673420 ± 11%     +30.2%    6084101 ± 13%  cpuidle.C1.time
 2.712e+08 ±  5%     +30.1%  3.529e+08 ±  2%  cpuidle.C1E.time
    624111 ±  2%     +36.3%     850658 ±  4%  cpuidle.C1E.usage
 1.791e+10           -12.1%  1.574e+10        cpuidle.C6.time
  18607622           -13.1%   16169546        cpuidle.C6.usage
 1.361e+08 ± 19%    +118.3%  2.971e+08 ± 13%  cpuidle.POLL.time
   5103004 ±  2%     +26.6%    6459744        meminfo.Active
   5067740 ±  2%     +27.0%    6436925        meminfo.Active(anon)
    756326 ±  8%    +124.1%    1695158 ± 25%  meminfo.AnonHugePages
   4887369 ±  3%     +26.7%    6193791        meminfo.AnonPages
  5.41e+08           +19.0%  6.437e+08        meminfo.Committed_AS
      6892           +33.2%       9179 ± 10%  meminfo.PageTables
    574999 ±  9%    +115.7%    1240273 ± 28%  numa-vmstat.node0.nr_active_anon
    554302 ±  7%    +116.5%    1200168 ± 29%  numa-vmstat.node0.nr_anon_pages
    574609 ±  9%    +115.8%    1240281 ± 28%  numa-vmstat.node0.nr_zone_active_anon
     10839 ±  9%     -25.3%       8092 ±  8%  numa-vmstat.node1.nr_slab_reclaimable
   4140994 ±  7%     -49.5%    2091526 ± 62%  numa-vmstat.node1.numa_hit
   3996598 ±  7%     -51.3%    1947593 ± 68%  numa-vmstat.node1.numa_local
   2314942 ± 12%    +115.3%    4985067 ± 27%  numa-meminfo.node0.Active
   2297543 ± 13%    +116.5%    4973510 ± 27%  numa-meminfo.node0.Active(anon)
    359259 ± 13%    +249.8%    1256563 ± 42%  numa-meminfo.node0.AnonHugePages
   2191741 ± 12%    +118.8%    4794936 ± 28%  numa-meminfo.node0.AnonPages
   3351366 ±  9%     +80.5%    6050637 ± 27%  numa-meminfo.node0.MemUsed
      3133 ±  9%     +91.1%       5990 ± 32%  numa-meminfo.node0.PageTables
     43356 ±  9%     -25.3%      32370 ±  8%  numa-meminfo.node1.SReclaimable
      1177           +15.6%       1360        turbostat.Avg_MHz
     42.29            +6.6       48.84        turbostat.Busy%
    602767 ±  3%     +35.6%     817443 ±  2%  turbostat.C1E
      0.86 ±  6%      +0.3        1.11        turbostat.C1E%
  18604760           -13.1%   16166704        turbostat.C6
     57.00            -6.9       50.14        turbostat.C6%
     56.09           -12.8%      48.90 ±  4%  turbostat.CPU%c1
    255.40            -8.2%     234.42        turbostat.PkgWatt
     65.03           -17.0%      53.98        turbostat.RAMWatt
      6202 ±  2%     -28.7%       4424 ±  6%  slabinfo.eventpoll_epi.active_objs
      6202 ±  2%     -28.7%       4424 ±  6%  slabinfo.eventpoll_epi.num_objs
      5427 ±  2%     -28.7%       3871 ±  6%  slabinfo.eventpoll_pwq.active_objs
      5427 ±  2%     -28.7%       3871 ±  6%  slabinfo.eventpoll_pwq.num_objs
     14969 ±  4%     -21.6%      11739 ±  3%  slabinfo.files_cache.active_objs
     15026 ±  4%     -21.3%      11821 ±  3%  slabinfo.files_cache.num_objs
      5869           -25.8%       4356 ± 11%  slabinfo.mm_struct.active_objs
      5933           -25.8%       4402 ± 12%  slabinfo.mm_struct.num_objs
      6573 ±  2%     -19.3%       5303 ± 12%  slabinfo.sighand_cache.active_objs
      6602 ±  2%     -19.1%       5339 ± 12%  slabinfo.sighand_cache.num_objs
      1424 ±  6%     -17.5%       1175 ±  3%  slabinfo.skbuff_fclone_cache.active_objs
      1424 ±  6%     -17.5%       1175 ±  3%  slabinfo.skbuff_fclone_cache.num_objs
      1804 ±  4%     -12.8%       1573 ±  8%  slabinfo.task_group.active_objs
      1804 ±  4%     -12.8%       1573 ±  8%  slabinfo.task_group.num_objs
     74.90            -7.1       67.78        perf-stat.cache-miss-rate%
 1.917e+10           -30.7%  1.329e+10 ±  2%  perf-stat.cache-misses
 2.559e+10           -23.4%   1.96e+10 ±  2%  perf-stat.cache-references
 3.608e+13           +15.1%  4.153e+13        perf-stat.cpu-cycles
     10929           -21.3%       8606 ± 15%  perf-stat.cpu-migrations
      0.08            -0.0        0.05 ± 13%  perf-stat.dTLB-store-miss-rate%
 1.187e+08           -48.3%   61361329 ±  7%  perf-stat.dTLB-store-misses
  1.41e+11           -12.2%  1.238e+11 ±  6%  perf-stat.dTLB-stores
 2.149e+08           -30.2%    1.5e+08 ±  7%  perf-stat.iTLB-loads
   8598974           -29.6%    6054449        perf-stat.minor-faults
 1.421e+08 ±  3%     -21.4%  1.117e+08 ±  6%  perf-stat.node-load-misses
   1.5e+09           -27.5%  1.088e+09        perf-stat.node-loads
      0.85 ± 70%      -0.5        0.38 ± 13%  perf-stat.node-store-miss-rate%
  98747708 ± 70%     -67.5%   32046709 ± 13%  perf-stat.node-store-misses
 1.146e+10           -26.0%  8.481e+09        perf-stat.node-stores
   8598995           -29.6%    6054463        perf-stat.page-faults
   1300724 ±  2%     +22.6%    1594074        proc-vmstat.nr_active_anon
   1256932 ±  2%     +22.5%    1539865        proc-vmstat.nr_anon_pages
    406.25 ±  8%     +79.9%     731.00 ± 14%  proc-vmstat.nr_anon_transparent_hugepages
   1461780            -2.2%    1429319        proc-vmstat.nr_dirty_background_threshold
   2927135            -2.2%    2862134        proc-vmstat.nr_dirty_threshold
  14564117            -1.9%   14293825        proc-vmstat.nr_free_pages
      8545            +6.6%       9109 ±  2%  proc-vmstat.nr_mapped
      1771 ±  2%     +23.5%       2188 ±  5%  proc-vmstat.nr_page_table_pages
   1300723 ±  2%     +22.6%    1594070        proc-vmstat.nr_zone_active_anon
  13839779           -30.8%    9576432        proc-vmstat.numa_hit
  13819218           -30.9%    9555852        proc-vmstat.numa_local
 2.725e+09           -32.6%  1.837e+09        proc-vmstat.pgalloc_normal
   8634919           -29.6%    6080480        proc-vmstat.pgfault
 2.725e+09           -32.6%  1.836e+09        proc-vmstat.pgfree
   5304368           -32.6%    3574443        proc-vmstat.thp_deferred_split_page
   5305872           -32.6%    3575998        proc-vmstat.thp_fault_alloc
    402915            +1.3%     408011        interrupts.CAL:Function_call_interrupts
    186176 ±  6%     -21.8%     145644 ± 12%  interrupts.CPU0.RES:Rescheduling_interrupts
     12448 ± 16%     -71.5%       3551 ± 58%  interrupts.CPU2.RES:Rescheduling_interrupts
    334.25 ± 58%    +181.7%     941.50 ± 24%  interrupts.CPU21.RES:Rescheduling_interrupts
    202.75 ± 50%   +1023.3%       2277 ± 98%  interrupts.CPU22.RES:Rescheduling_interrupts
    138.50 ± 59%   +1086.3%       1643 ± 36%  interrupts.CPU23.RES:Rescheduling_interrupts
    179.00 ± 55%    +910.3%       1808 ±106%  interrupts.CPU25.RES:Rescheduling_interrupts
    485.50 ± 29%   +8854.8%      43475 ± 37%  interrupts.CPU26.RES:Rescheduling_interrupts
    248.75 ± 38%   +1876.2%       4915 ± 52%  interrupts.CPU27.RES:Rescheduling_interrupts
    116.75 ± 12%    +297.9%     464.50 ± 11%  interrupts.CPU29.RES:Rescheduling_interrupts
      8061 ± 28%     -54.5%       3669 ± 61%  interrupts.CPU3.RES:Rescheduling_interrupts
      3674 ±  6%     -57.9%       1546 ± 95%  interrupts.CPU31.NMI:Non-maskable_interrupts
      3674 ±  6%     -57.9%       1546 ± 95%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
     79.25 ± 40%     -56.8%      34.25 ± 97%  interrupts.CPU50.RES:Rescheduling_interrupts
     86.75 ± 56%     -77.8%      19.25 ± 71%  interrupts.CPU51.RES:Rescheduling_interrupts
    669.25 ± 78%     -73.4%     177.75 ±155%  interrupts.CPU53.RES:Rescheduling_interrupts
    498.25 ± 80%     -95.1%      24.50 ± 37%  interrupts.CPU55.RES:Rescheduling_interrupts
    238.00 ± 58%     -82.7%      41.25 ± 81%  interrupts.CPU58.RES:Rescheduling_interrupts
    278.50 ± 28%     -92.8%      20.00 ± 52%  interrupts.CPU59.RES:Rescheduling_interrupts
    256.75 ± 47%     -90.4%      24.75 ± 50%  interrupts.CPU60.RES:Rescheduling_interrupts
    225.25 ± 71%     -91.2%      19.75 ± 27%  interrupts.CPU61.RES:Rescheduling_interrupts
    236.00 ± 92%     -88.2%      27.75 ± 80%  interrupts.CPU63.RES:Rescheduling_interrupts
    171.25 ± 73%     -91.2%      15.00 ± 22%  interrupts.CPU64.RES:Rescheduling_interrupts
    239.00 ± 36%     -76.4%      56.50 ±130%  interrupts.CPU65.RES:Rescheduling_interrupts
    196.75 ± 51%     -89.8%      20.00 ± 15%  interrupts.CPU66.RES:Rescheduling_interrupts
    196.50 ± 53%     -78.1%      43.00 ±111%  interrupts.CPU70.RES:Rescheduling_interrupts
    191.00 ± 45%     -90.2%      18.75 ± 45%  interrupts.CPU71.RES:Rescheduling_interrupts
    203.25 ± 81%     -93.7%      12.75 ± 23%  interrupts.CPU72.RES:Rescheduling_interrupts
    103.25 ± 59%     -78.9%      21.75 ± 24%  interrupts.CPU73.RES:Rescheduling_interrupts
    111.25 ± 79%     -80.9%      21.25 ± 76%  interrupts.CPU74.RES:Rescheduling_interrupts
     93.50 ±106%     -78.1%      20.50 ± 67%  interrupts.CPU78.RES:Rescheduling_interrupts
    400.50 ±155%     -97.0%      12.00 ± 21%  interrupts.CPU81.RES:Rescheduling_interrupts
    347.50 ±156%     -96.3%      13.00 ± 75%  interrupts.CPU82.RES:Rescheduling_interrupts
    285.00 ±149%     -96.7%       9.50 ± 49%  interrupts.CPU83.RES:Rescheduling_interrupts
    265.00 ±136%     -94.2%      15.25 ± 48%  interrupts.CPU84.RES:Rescheduling_interrupts
    153.50 ±145%     -89.7%      15.75 ± 63%  interrupts.CPU85.RES:Rescheduling_interrupts
    167.00 ±101%     -91.0%      15.00 ± 54%  interrupts.CPU87.RES:Rescheduling_interrupts
     81.50 ± 79%     -87.4%      10.25 ± 78%  interrupts.CPU88.RES:Rescheduling_interrupts
    114.00 ± 92%     -79.2%      23.75 ±103%  interrupts.CPU98.RES:Rescheduling_interrupts
     54567 ± 11%     +54.4%      84277 ± 12%  sched_debug.cfs_rq:/.exec_clock.avg
    102845 ± 10%     +41.8%     145786 ±  2%  sched_debug.cfs_rq:/.exec_clock.max
     12134 ± 15%    +286.2%      46865 ± 36%  sched_debug.cfs_rq:/.exec_clock.stddev
    190887 ± 88%     -83.1%      32327 ± 43%  sched_debug.cfs_rq:/.load.max
     26347 ± 56%     -61.1%      10240 ± 19%  sched_debug.cfs_rq:/.load.stddev
   2704219 ± 11%     +61.5%    4367328 ± 12%  sched_debug.cfs_rq:/.min_vruntime.avg
   4650229 ±  8%     +60.8%    7478802 ±  3%  sched_debug.cfs_rq:/.min_vruntime.max
    546421 ± 15%    +343.4%    2422973 ± 37%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.41 ±  4%     +38.1%       0.56 ±  8%  sched_debug.cfs_rq:/.nr_running.avg
      1.65 ± 14%     +40.8%       2.32 ± 21%  sched_debug.cfs_rq:/.nr_spread_over.avg
      1.07 ± 23%     +49.1%       1.59 ± 20%  sched_debug.cfs_rq:/.nr_spread_over.stddev
      8.96 ±  5%     +21.6%      10.90 ±  6%  sched_debug.cfs_rq:/.runnable_load_avg.avg
     12.33 ± 22%     -28.3%       8.84 ±  6%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
    188871 ± 89%     -83.6%      30944 ± 47%  sched_debug.cfs_rq:/.runnable_weight.max
     26233 ± 57%     -61.0%      10219 ± 20%  sched_debug.cfs_rq:/.runnable_weight.stddev
   3417321 ± 11%     +37.6%    4701239 ± 20%  sched_debug.cfs_rq:/.spread0.max
   -122012         +1597.6%   -2071325        sched_debug.cfs_rq:/.spread0.min
    546405 ± 15%    +343.4%    2422990 ± 37%  sched_debug.cfs_rq:/.spread0.stddev
    425.53 ±  5%     +34.3%     571.59 ±  6%  sched_debug.cfs_rq:/.util_avg.avg
    172409 ± 17%     -51.9%      83000 ± 34%  sched_debug.cpu.avg_idle.min
     15.80 ± 35%     -55.0%       7.11 ± 31%  sched_debug.cpu.clock.stddev
     15.80 ± 35%     -55.0%       7.11 ± 31%  sched_debug.cpu.clock_task.stddev
     11.99 ± 20%     -22.2%       9.34 ±  4%  sched_debug.cpu.cpu_load[0].stddev
     25155 ±  9%     -18.1%      20603        sched_debug.cpu.curr->pid.max
     11971 ±  8%     -16.4%      10012 ±  2%  sched_debug.cpu.curr->pid.stddev
    139384 ± 77%     -46.1%      75152 ±117%  sched_debug.cpu.load.max
    110514 ± 11%     +33.6%     147689 ±  4%  sched_debug.cpu.nr_load_updates.max
      9429 ± 13%    +240.3%      32086 ± 33%  sched_debug.cpu.nr_load_updates.stddev
      6703 ± 11%     +31.8%       8832 ±  9%  sched_debug.cpu.nr_switches.avg
     42299 ±  4%    +297.2%     168004 ± 44%  sched_debug.cpu.nr_switches.max
      6781 ±  7%    +161.4%      17726 ± 37%  sched_debug.cpu.nr_switches.stddev
     16.50 ± 22%     +35.1%      22.29 ±  9%  sched_debug.cpu.nr_uninterruptible.max
      5906 ± 12%     +37.3%       8110 ±  8%  sched_debug.cpu.sched_count.avg
     41231          +301.8%     165649 ± 45%  sched_debug.cpu.sched_count.max
      6649 ±  4%    +162.1%      17430 ± 38%  sched_debug.cpu.sched_count.stddev
      2794 ± 14%     +37.8%       3849 ± 11%  sched_debug.cpu.sched_goidle.avg
     19770 ±  5%    +318.5%      82741 ± 45%  sched_debug.cpu.sched_goidle.max
      3237 ±  7%    +168.2%       8684 ± 39%  sched_debug.cpu.sched_goidle.stddev
      2707 ± 14%     +42.0%       3845 ± 10%  sched_debug.cpu.ttwu_count.avg
     24144 ±  4%    +222.3%      77807 ± 43%  sched_debug.cpu.ttwu_count.max
      3710 ±  6%    +149.1%       9243 ± 35%  sched_debug.cpu.ttwu_count.stddev
    890.11 ± 12%     +63.4%       1454 ±  3%  sched_debug.cpu.ttwu_local.avg
      3405 ± 20%     +65.9%       5649 ±  9%  sched_debug.cpu.ttwu_local.max
    599.71 ± 10%     +47.1%     881.99 ± 12%  sched_debug.cpu.ttwu_local.stddev
    101950 ±  5%     -22.2%      79276 ± 13%  softirqs.CPU0.SCHED
     76082 ±  8%     +65.3%     125732 ± 20%  softirqs.CPU10.TIMER
     87879 ± 13%     -33.2%      58674 ± 30%  softirqs.CPU102.TIMER
     84916 ± 12%     -28.8%      60445 ± 28%  softirqs.CPU103.TIMER
     79117 ±  7%     +60.4%     126925 ± 19%  softirqs.CPU11.TIMER
     80478 ±  7%     +57.9%     127096 ± 19%  softirqs.CPU12.TIMER
     79602 ±  8%     +58.9%     126500 ± 19%  softirqs.CPU13.TIMER
     76463 ±  9%     +63.9%     125308 ± 21%  softirqs.CPU14.TIMER
     71283 ± 15%     +76.9%     126120 ± 21%  softirqs.CPU15.TIMER
     75177 ± 10%     +71.9%     129197 ± 19%  softirqs.CPU16.TIMER
     75260 ± 13%     +69.9%     127848 ± 20%  softirqs.CPU17.TIMER
     78227 ± 14%     +49.7%     117133 ± 28%  softirqs.CPU18.TIMER
     76725 ± 15%     +60.8%     123342 ± 24%  softirqs.CPU19.TIMER
     14186 ± 12%     -60.0%       5675 ± 37%  softirqs.CPU2.SCHED
     78243 ± 12%     +57.5%     123220 ± 23%  softirqs.CPU20.TIMER
     74069 ± 13%     +66.7%     123496 ± 24%  softirqs.CPU21.TIMER
     72445 ±  8%     +71.5%     124276 ± 23%  softirqs.CPU24.TIMER
     71429 ±  7%     +69.8%     121306 ± 23%  softirqs.CPU25.TIMER
      6838 ± 11%    +291.4%      26765 ± 31%  softirqs.CPU26.SCHED
     84914 ± 11%     -44.7%      46972 ± 35%  softirqs.CPU26.TIMER
     85536 ± 11%     -37.9%      53112 ± 36%  softirqs.CPU27.TIMER
     92319 ± 11%     -39.7%      55695 ± 41%  softirqs.CPU29.TIMER
     11818 ± 11%     -50.4%       5865 ± 27%  softirqs.CPU3.SCHED
     91562 ±  6%     -37.7%      57040 ± 41%  softirqs.CPU30.TIMER
     94268 ±  6%     -43.9%      52887 ± 45%  softirqs.CPU31.TIMER
     93396 ±  4%     -40.8%      55292 ± 45%  softirqs.CPU32.TIMER
      6065 ± 24%     +65.3%      10023 ± 19%  softirqs.CPU34.SCHED
      5558 ± 18%     +49.6%       8313 ± 19%  softirqs.CPU35.SCHED
      5181 ± 19%     +91.7%       9930 ± 23%  softirqs.CPU36.SCHED
     10638 ±  5%     -45.1%       5843 ± 17%  softirqs.CPU4.SCHED
     82385 ±  7%     -30.8%      57037 ± 20%  softirqs.CPU42.TIMER
     85276 ± 10%     -42.1%      49344 ± 47%  softirqs.CPU45.TIMER
     90182 ± 11%     -38.8%      55156 ± 54%  softirqs.CPU48.TIMER
      9407 ± 11%     -33.4%       6268 ± 36%  softirqs.CPU5.SCHED
     86739 ±  5%     +50.2%     130246 ± 16%  softirqs.CPU5.TIMER
     90646 ± 10%     -40.1%      54319 ± 46%  softirqs.CPU51.TIMER
      8726 ± 21%     -54.2%       3998 ± 40%  softirqs.CPU55.SCHED
     77984 ±  6%     +59.3%     124232 ± 21%  softirqs.CPU55.TIMER
      8399 ± 18%     -53.5%       3905 ± 25%  softirqs.CPU57.SCHED
      8031 ± 17%     -39.5%       4859 ± 35%  softirqs.CPU58.SCHED
     77083 ± 15%     +56.7%     120805 ± 18%  softirqs.CPU58.TIMER
      8371 ± 17%     -50.9%       4107 ± 37%  softirqs.CPU59.SCHED
     80820 ± 10%     +54.3%     124710 ± 21%  softirqs.CPU59.TIMER
     84277 ± 10%     +51.3%     127533 ± 11%  softirqs.CPU6.TIMER
     77675 ±  7%     +57.8%     122547 ± 19%  softirqs.CPU60.TIMER
     79746 ±  7%     +56.4%     124719 ± 19%  softirqs.CPU61.TIMER
     75401 ± 16%     +64.4%     123975 ± 21%  softirqs.CPU62.TIMER
     80622 ±  9%     +54.4%     124509 ± 20%  softirqs.CPU63.TIMER
     78498 ±  7%     +57.4%     123570 ± 22%  softirqs.CPU64.TIMER
     78854 ± 11%     +58.7%     125157 ± 19%  softirqs.CPU65.TIMER
     78398 ± 10%     +56.5%     122679 ± 22%  softirqs.CPU66.TIMER
     70518 ± 14%     +77.2%     124944 ± 21%  softirqs.CPU67.TIMER
     76676 ± 18%     +65.8%     127094 ± 20%  softirqs.CPU68.TIMER
     79804 ± 13%     +59.8%     127489 ± 19%  softirqs.CPU69.TIMER
     87607 ±  7%     +47.0%     128769 ± 16%  softirqs.CPU7.TIMER
     72877 ± 11%     +64.6%     119922 ± 24%  softirqs.CPU74.TIMER
     72901 ±  8%     +69.5%     123554 ± 22%  softirqs.CPU77.TIMER
     89804 ±  9%     -49.5%      45318 ± 45%  softirqs.CPU78.TIMER
     83519 ±  4%     +44.1%     120383 ± 16%  softirqs.CPU8.TIMER
     88956 ± 14%     -37.8%      55317 ± 39%  softirqs.CPU81.TIMER
     85839 ± 10%     -40.4%      51145 ± 51%  softirqs.CPU83.TIMER
      5980 ± 25%     +65.2%       9881 ± 13%  softirqs.CPU86.SCHED
      5232 ± 15%     +68.0%       8790 ± 17%  softirqs.CPU87.SCHED
     79808 ±  6%     +57.2%     125474 ± 20%  softirqs.CPU9.TIMER
     19.41 ±  4%     -16.0        3.44 ±  9%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
     21.19 ±  2%     -15.3        5.84 ± 17%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     20.82 ±  2%     -15.2        5.66 ± 19%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     20.82 ±  2%     -15.2        5.66 ± 19%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     20.77 ±  2%     -15.2        5.61 ± 19%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     20.81 ±  2%     -15.2        5.66 ± 19%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.23 ±  2%      +0.2        1.46 ±  2%  perf-profile.calltrace.cycles-pp.tlb_flush_mmu_free.unmap_page_range.unmap_vmas.exit_mmap.mmput
      1.22 ±  2%      +0.2        1.45 ±  2%  perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu_free.unmap_page_range.unmap_vmas.exit_mmap
      0.59 ±  5%      +0.4        0.95 ±  2%  perf-profile.calltrace.cycles-pp.___might_sleep
      1.39            +0.5        1.89        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.39            +0.5        1.89        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.38            +0.5        1.88        perf-profile.calltrace.cycles-pp.__wake_up_parent.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.38            +0.5        1.88        perf-profile.calltrace.cycles-pp.do_group_exit.__wake_up_parent.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.38            +0.5        1.88        perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__wake_up_parent.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.37            +0.5        1.87        perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__wake_up_parent
      1.37            +0.5        1.88        perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__wake_up_parent.do_syscall_64
      1.33            +0.5        1.84        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit
      1.34            +0.5        1.85        perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit
      1.45 ±  2%      +0.5        2.00        perf-profile.calltrace.cycles-pp._cond_resched.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      0.93 ±  2%      +0.6        1.50 ±  3%  perf-profile.calltrace.cycles-pp.clear_huge_page
      0.28 ±173%      +1.0        1.25 ± 44%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      1.71 ±  2%      +1.0        2.73        perf-profile.calltrace.cycles-pp.___might_sleep.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      0.70 ±  9%      +1.1        1.81 ±  2%  perf-profile.calltrace.cycles-pp.clear_page_erms
      0.83 ±  7%      +1.8        2.62 ±  6%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.do_huge_pmd_anonymous_page
      0.83 ±  7%      +1.8        2.63 ±  6%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.do_huge_pmd_anonymous_page.__handle_mm_fault
      2.51 ±  2%      +2.2        4.69 ±  4%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      2.52 ±  2%      +2.2        4.71 ±  4%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
     61.29            +8.6       69.94        perf-profile.calltrace.cycles-pp.clear_page_erms.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
     69.75           +10.3       80.06        perf-profile.calltrace.cycles-pp.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
     72.75           +12.6       85.32        perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     72.78           +12.6       85.35        perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     72.64           +12.6       85.23        perf-profile.calltrace.cycles-pp.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
     72.84           +12.6       85.45        perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
     72.84           +12.6       85.45        perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
     72.84           +12.6       85.47        perf-profile.calltrace.cycles-pp.page_fault
     19.73 ±  3%     -16.2        3.58 ± 10%  perf-profile.children.cycles-pp.intel_idle
     21.19 ±  2%     -15.3        5.84 ± 17%  perf-profile.children.cycles-pp.secondary_startup_64
     21.19 ±  2%     -15.3        5.84 ± 17%  perf-profile.children.cycles-pp.cpu_startup_entry
     21.19 ±  2%     -15.3        5.84 ± 17%  perf-profile.children.cycles-pp.do_idle
     21.16 ±  2%     -15.3        5.82 ± 17%  perf-profile.children.cycles-pp.cpuidle_enter_state
     20.82 ±  2%     -15.2        5.66 ± 19%  perf-profile.children.cycles-pp.start_secondary
      0.37 ± 18%      -0.2        0.18 ± 74%  perf-profile.children.cycles-pp.start_kernel
      0.10 ± 15%      -0.1        0.04 ±107%  perf-profile.children.cycles-pp.read
      0.18 ±  6%      -0.0        0.14 ±  5%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.07 ±  7%      +0.0        0.08        perf-profile.children.cycles-pp.__list_del_entry_valid
      0.08 ±  5%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.__lru_cache_add
      0.08 ±  5%      +0.0        0.11 ±  4%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.07 ± 13%      +0.0        0.09 ± 11%  perf-profile.children.cycles-pp.cmd_stat
      0.07 ± 13%      +0.0        0.10 ±  9%  perf-profile.children.cycles-pp.__run_perf_stat
      0.07 ± 13%      +0.0        0.10 ±  9%  perf-profile.children.cycles-pp.process_interval
      0.07 ± 13%      +0.0        0.10 ±  9%  perf-profile.children.cycles-pp.read_counters
      0.07 ± 13%      +0.0        0.10 ±  9%  perf-profile.children.cycles-pp.perf_evsel__read_counter
      0.07 ± 13%      +0.0        0.10 ±  9%  perf-profile.children.cycles-pp.__read_nocancel
      0.06 ± 15%      +0.0        0.09 ±  7%  perf-profile.children.cycles-pp.perf_event_read
      0.06 ± 20%      +0.0        0.10 ±  9%  perf-profile.children.cycles-pp.perf_read
      0.06 ± 15%      +0.0        0.09 ±  4%  perf-profile.children.cycles-pp.smp_call_function_single
      0.03 ±100%      +0.0        0.07 ±  7%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.09 ±  9%      +0.0        0.14 ±  3%  perf-profile.children.cycles-pp.pte_alloc_one
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.___perf_sw_event
      0.06 ± 20%      +0.1        0.11 ± 25%  perf-profile.children.cycles-pp.ktime_get
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.__perf_sw_event
      0.01 ±173%      +0.1        0.08 ± 11%  perf-profile.children.cycles-pp.__perf_event_read_value
      0.13 ±  6%      +0.1        0.23 ±  5%  perf-profile.children.cycles-pp.__put_compound_page
      0.40            +0.1        0.50        perf-profile.children.cycles-pp.rcu_all_qs
      0.11 ±  7%      +0.1        0.22 ±  5%  perf-profile.children.cycles-pp.__page_cache_release
      0.03 ±100%      +0.1        0.17 ±  4%  perf-profile.children.cycles-pp.free_transhuge_page
      0.22 ±  6%      +0.2        0.42        perf-profile.children.cycles-pp.free_one_page
      1.23 ±  2%      +0.2        1.46 ±  2%  perf-profile.children.cycles-pp.tlb_flush_mmu_free
      1.23 ±  2%      +0.2        1.47 ±  2%  perf-profile.children.cycles-pp.release_pages
      0.06 ± 11%      +0.3        0.33 ± 11%  perf-profile.children.cycles-pp.deferred_split_huge_page
      0.09 ±  4%      +0.3        0.37 ±  9%  perf-profile.children.cycles-pp.zap_huge_pmd
      1.38            +0.5        1.88        perf-profile.children.cycles-pp.__wake_up_parent
      1.38            +0.5        1.88        perf-profile.children.cycles-pp.do_group_exit
      1.38            +0.5        1.88        perf-profile.children.cycles-pp.do_exit
      1.37            +0.5        1.88        perf-profile.children.cycles-pp.mmput
      1.37            +0.5        1.88        perf-profile.children.cycles-pp.exit_mmap
      1.33 ±  2%      +0.5        1.84        perf-profile.children.cycles-pp.unmap_page_range
      1.34            +0.5        1.85        perf-profile.children.cycles-pp.unmap_vmas
      1.75            +0.6        2.37        perf-profile.children.cycles-pp._cond_resched
      0.54 ± 63%      +0.7        1.25 ± 44%  perf-profile.children.cycles-pp.poll_idle
      2.31 ±  3%      +1.4        3.69        perf-profile.children.cycles-pp.___might_sleep
      2.61 ±  2%      +2.2        4.83 ±  4%  perf-profile.children.cycles-pp.get_page_from_freelist
      2.62 ±  2%      +2.2        4.85 ±  4%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      1.06 ±  6%      +2.3        3.36 ±  4%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.69 ± 11%      +2.5        4.17 ± 10%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     62.42            +9.8       72.18        perf-profile.children.cycles-pp.clear_page_erms
     70.69           +10.9       81.56        perf-profile.children.cycles-pp.clear_huge_page
     72.77           +12.6       85.33        perf-profile.children.cycles-pp.__handle_mm_fault
     72.79           +12.6       85.36        perf-profile.children.cycles-pp.handle_mm_fault
     72.64           +12.6       85.23        perf-profile.children.cycles-pp.do_huge_pmd_anonymous_page
     72.86           +12.6       85.47        perf-profile.children.cycles-pp.__do_page_fault
     72.86           +12.6       85.47        perf-profile.children.cycles-pp.do_page_fault
     72.86           +12.6       85.48        perf-profile.children.cycles-pp.page_fault
     19.73 ±  3%     -16.2        3.58 ± 10%  perf-profile.self.cycles-pp.intel_idle
      0.78 ±  3%      -0.2        0.60 ±  6%  perf-profile.self.cycles-pp.__free_pages_ok
      0.18 ±  6%      -0.0        0.14 ±  5%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.07 ±  7%      +0.0        0.08        perf-profile.self.cycles-pp.__list_del_entry_valid
      0.06 ± 15%      +0.0        0.09 ±  4%  perf-profile.self.cycles-pp.smp_call_function_single
      0.03 ±102%      +0.1        0.10 ± 29%  perf-profile.self.cycles-pp.ktime_get
      0.39            +0.1        0.49        perf-profile.self.cycles-pp.rcu_all_qs
      1.62            +0.3        1.97 ±  2%  perf-profile.self.cycles-pp.get_page_from_freelist
      1.50            +0.6        2.08        perf-profile.self.cycles-pp._cond_resched
      6.08            +0.7        6.78 ±  2%  perf-profile.self.cycles-pp.clear_huge_page
      0.54 ± 64%      +0.7        1.25 ± 44%  perf-profile.self.cycles-pp.poll_idle
      2.29 ±  3%      +1.4        3.66        perf-profile.self.cycles-pp.___might_sleep
      1.69 ± 11%      +2.5        4.17 ± 10%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
     61.81            +9.9       71.69        perf-profile.self.cycles-pp.clear_page_erms



***************************************************************************************************
lkp-bdw-ep4: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
=========================================================================================
array_size/compiler/cpufreq_governor/kconfig/nr_threads/omp/rootfs/tbox_group/testcase:
  50000000/gcc-7/performance/x86_64-rhel-7.2/50%/true/debian-x86_64-2018-04-03.cgz/lkp-bdw-ep4/stream

commit: 
  24d0c1d6e6 ("sched/fair: Do not migrate due to a sync wakeup on exit")
  2c83362734 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")

24d0c1d6e65f635b 2c83362734dad8e48ccc0710b5c 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     70867 ± 16%     -28.8%      50456 ±  6%  stream.add_bandwidth_MBps
     11358 ±  9%     +13.4%      12879        stream.time.minor_page_faults
    263.61 ± 16%     +36.1%     358.87        stream.time.user_time
      9477 ± 14%     -22.8%       7312 ±  6%  stream.time.voluntary_context_switches
     34.65            -2.3%      33.85 ±  2%  boot-time.boot
     14973 ±  5%      -7.9%      13783 ±  6%  softirqs.CPU6.TIMER
     67.50            -3.7%      65.00        vmstat.cpu.id
     31.50 ±  2%      +7.9%      34.00 ±  3%  vmstat.cpu.us
      3619 ± 15%     -29.4%       2555 ±  7%  vmstat.system.cs
   2863126 ±  6%     -15.5%    2419307 ±  2%  cpuidle.C3.time
      8360 ±  6%     -15.7%       7050 ±  2%  cpuidle.C3.usage
 4.188e+08 ± 14%     +32.5%  5.548e+08 ±  2%  cpuidle.C6.time
    428117 ± 14%     +31.9%     564534 ±  2%  cpuidle.C6.usage
      3694 ± 23%    +130.5%       8515 ± 73%  proc-vmstat.numa_hint_faults
     98740 ±  2%      +8.8%     107468 ±  3%  proc-vmstat.numa_hit
     81603 ±  2%     +10.7%      90367 ±  4%  proc-vmstat.numa_local
     48999 ±  5%     +25.0%      61261 ± 11%  proc-vmstat.pgfault
      7862 ±  6%     -14.0%       6758 ±  2%  turbostat.C3
      0.38 ± 17%      -0.1        0.24 ±  4%  turbostat.C3%
    428197 ± 14%     +32.2%     566126 ±  2%  turbostat.C6
     19.93 ± 28%     +53.7%      30.64 ±  5%  turbostat.CPU%c6
    960324 ± 12%     +23.7%    1187618        turbostat.IRQ
      2392 ±  3%     -26.4%       1760 ±  8%  slabinfo.eventpoll_epi.active_objs
      2392 ±  3%     -26.4%       1760 ±  8%  slabinfo.eventpoll_epi.num_objs
      4186 ±  3%     -26.4%       3080 ±  8%  slabinfo.eventpoll_pwq.active_objs
      4186 ±  3%     -26.4%       3080 ±  8%  slabinfo.eventpoll_pwq.num_objs
    650.00 ±  7%     -13.2%     564.50 ±  3%  slabinfo.file_lock_cache.active_objs
    650.00 ±  7%     -13.2%     564.50 ±  3%  slabinfo.file_lock_cache.num_objs
     28794 ± 14%     -24.0%      21889 ±  5%  sched_debug.cfs_rq:/.min_vruntime.max
      4667 ± 33%     -60.6%       1837 ± 65%  sched_debug.cfs_rq:/.min_vruntime.min
      3081 ± 24%     -73.8%     808.08 ±142%  sched_debug.cfs_rq:/.spread0.avg
     -2493           +70.7%      -4255        sched_debug.cfs_rq:/.spread0.min
    295.31 ±  3%     -16.9%     245.52 ± 11%  sched_debug.cfs_rq:/.util_avg.stddev
      0.00 ±  8%     +19.3%       0.00 ±  7%  sched_debug.cpu.next_balance.stddev
    240.50 ±  4%     -14.9%     204.75 ± 11%  sched_debug.cpu.nr_switches.min
      2.62 ±  3%     +32.0%       3.46 ±  7%  sched_debug.cpu.nr_uninterruptible.stddev
 3.769e+10 ±  5%      -7.8%  3.476e+10 ±  4%  perf-stat.branch-instructions
 9.742e+09            +6.0%  1.032e+10 ±  4%  perf-stat.cache-references
     28360 ±  6%     -14.5%      24239 ±  8%  perf-stat.context-switches
      4.31 ± 20%     +40.5%       6.06        perf-stat.cpi
 8.342e+11 ± 17%     +34.0%  1.118e+12 ±  4%  perf-stat.cpu-cycles
    451.50 ± 10%     -27.8%     326.00 ±  8%  perf-stat.cpu-migrations
      0.02 ± 19%      +0.0        0.03 ±  8%  perf-stat.dTLB-load-miss-rate%
   9450196 ± 21%     +41.5%   13368418 ±  4%  perf-stat.dTLB-load-misses
      0.24 ± 17%     -31.3%       0.17        perf-stat.ipc
     47611 ±  7%     +22.9%      58520 ± 10%  perf-stat.minor-faults
     47365 ±  6%     +23.6%      58532 ± 10%  perf-stat.page-faults
      1.19 ±173%      +5.6        6.76 ± 54%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
      1.19 ±173%      +5.6        6.76 ± 54%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      1.19 ±173%      +5.6        6.76 ± 54%  perf-profile.calltrace.cycles-pp.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      1.19 ±173%      +5.6        6.76 ± 54%  perf-profile.calltrace.cycles-pp.vfs_read.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.00            +6.6        6.55 ± 54%  perf-profile.calltrace.cycles-pp.kstat_irqs_cpu.show_interrupts.seq_read.proc_reg_read.__vfs_read
      1.19 ±173%      +6.7        7.84 ± 43%  perf-profile.calltrace.cycles-pp.read
      1.19 ±173%     +10.0       11.20 ± 56%  perf-profile.calltrace.cycles-pp.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.19 ±173%     +10.0       11.20 ± 56%  perf-profile.calltrace.cycles-pp.vfs_read.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.78 ±173%     +10.2       11.99 ± 52%  perf-profile.calltrace.cycles-pp.show_interrupts.seq_read.proc_reg_read.__vfs_read.vfs_read
      2.38 ±173%     +14.5       16.87 ± 38%  perf-profile.calltrace.cycles-pp.proc_reg_read.__vfs_read.vfs_read.sys_read.do_syscall_64
      2.38 ±173%     +14.5       16.87 ± 38%  perf-profile.calltrace.cycles-pp.seq_read.proc_reg_read.__vfs_read.vfs_read.sys_read
      2.38 ±173%     +15.6       17.96 ± 38%  perf-profile.calltrace.cycles-pp.__vfs_read.vfs_read.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +6.5        6.55 ± 54%  perf-profile.children.cycles-pp.kstat_irqs_cpu
      1.19 ±173%      +6.7        7.84 ± 43%  perf-profile.children.cycles-pp.read
      1.78 ±173%     +10.2       11.98 ± 52%  perf-profile.children.cycles-pp.show_interrupts
      2.38 ±173%     +14.5       16.87 ± 38%  perf-profile.children.cycles-pp.proc_reg_read
      2.38 ±173%     +15.6       17.95 ± 38%  perf-profile.children.cycles-pp.sys_read
      2.38 ±173%     +15.6       17.95 ± 38%  perf-profile.children.cycles-pp.vfs_read
      2.38 ±173%     +15.6       17.95 ± 38%  perf-profile.children.cycles-pp.__vfs_read
      2.38 ±173%     +15.6       17.95 ± 38%  perf-profile.children.cycles-pp.seq_read
     17.64 ± 77%     +28.9       46.59 ± 26%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     17.64 ± 77%     +28.9       46.59 ± 26%  perf-profile.children.cycles-pp.do_syscall_64
      8103 ± 15%     +30.9%      10608        interrupts.CPU0.LOC:Local_timer_interrupts
      8095 ± 15%     +30.5%      10561        interrupts.CPU1.LOC:Local_timer_interrupts
      8078 ± 15%     +31.3%      10610        interrupts.CPU10.LOC:Local_timer_interrupts
      8026 ± 14%     +32.0%      10598        interrupts.CPU11.LOC:Local_timer_interrupts
      8103 ± 15%     +30.9%      10603        interrupts.CPU12.LOC:Local_timer_interrupts
      8070 ± 15%     +31.3%      10593        interrupts.CPU13.LOC:Local_timer_interrupts
      8068 ± 15%     +32.4%      10678 ±  2%  interrupts.CPU14.LOC:Local_timer_interrupts
      8067 ± 15%     +31.5%      10609        interrupts.CPU15.LOC:Local_timer_interrupts
      8103 ± 15%     +30.7%      10588        interrupts.CPU16.LOC:Local_timer_interrupts
      8192 ± 16%     +29.3%      10596        interrupts.CPU17.LOC:Local_timer_interrupts
      8099 ± 15%     +31.2%      10628        interrupts.CPU18.LOC:Local_timer_interrupts
      8061 ± 15%     +31.6%      10604        interrupts.CPU19.LOC:Local_timer_interrupts
      8114 ± 15%     +30.8%      10610        interrupts.CPU2.LOC:Local_timer_interrupts
      8086 ± 15%     +31.2%      10610        interrupts.CPU20.LOC:Local_timer_interrupts
      8088 ± 15%     +31.4%      10629        interrupts.CPU21.LOC:Local_timer_interrupts
      8064 ± 16%     +31.3%      10591        interrupts.CPU22.LOC:Local_timer_interrupts
      8069 ± 15%     +30.5%      10530        interrupts.CPU23.LOC:Local_timer_interrupts
      8079 ± 15%     +31.3%      10611        interrupts.CPU24.LOC:Local_timer_interrupts
      8089 ± 15%     +31.0%      10598        interrupts.CPU25.LOC:Local_timer_interrupts
      8114 ± 15%     +30.6%      10597        interrupts.CPU26.LOC:Local_timer_interrupts
      8093 ± 15%     +31.3%      10630        interrupts.CPU27.LOC:Local_timer_interrupts
      8092 ± 15%     +30.9%      10589        interrupts.CPU28.LOC:Local_timer_interrupts
      8084 ± 15%     +31.1%      10599        interrupts.CPU29.LOC:Local_timer_interrupts
      8083 ± 15%     +30.5%      10547        interrupts.CPU3.LOC:Local_timer_interrupts
      8096 ± 15%     +31.2%      10624        interrupts.CPU30.LOC:Local_timer_interrupts
      8154 ± 15%     +31.0%      10680 ±  2%  interrupts.CPU31.LOC:Local_timer_interrupts
      8129 ± 15%     +30.7%      10628        interrupts.CPU32.LOC:Local_timer_interrupts
      8096 ± 15%     +31.9%      10676 ±  2%  interrupts.CPU33.LOC:Local_timer_interrupts
      8119 ± 15%     +30.8%      10620        interrupts.CPU34.LOC:Local_timer_interrupts
      8085 ± 15%     +31.2%      10612        interrupts.CPU35.LOC:Local_timer_interrupts
      8083 ± 15%     +31.2%      10602        interrupts.CPU36.LOC:Local_timer_interrupts
    819.25 ± 35%    +162.8%       2153 ± 47%  interrupts.CPU37.CAL:Function_call_interrupts
      8100 ± 15%     +31.5%      10655        interrupts.CPU37.LOC:Local_timer_interrupts
     32.75 ±173%   +4387.0%       1469 ± 71%  interrupts.CPU37.TLB:TLB_shootdowns
      8085 ± 15%     +31.2%      10612        interrupts.CPU38.LOC:Local_timer_interrupts
    985.25 ± 24%    +124.5%       2211 ± 52%  interrupts.CPU39.CAL:Function_call_interrupts
      8154 ± 15%     +30.3%      10625        interrupts.CPU39.LOC:Local_timer_interrupts
    162.75 ±110%    +852.4%       1550 ± 74%  interrupts.CPU39.TLB:TLB_shootdowns
      8171 ± 14%     +29.8%      10603        interrupts.CPU4.LOC:Local_timer_interrupts
      8088 ± 15%     +31.2%      10609        interrupts.CPU40.LOC:Local_timer_interrupts
      8091 ± 15%     +31.6%      10652 ±  2%  interrupts.CPU41.LOC:Local_timer_interrupts
      8097 ± 15%     +30.9%      10601        interrupts.CPU42.LOC:Local_timer_interrupts
      8105 ± 15%     +30.7%      10595 ±  2%  interrupts.CPU43.LOC:Local_timer_interrupts
      8069 ± 15%     +31.3%      10593        interrupts.CPU44.LOC:Local_timer_interrupts
      8075 ± 15%     +31.7%      10631        interrupts.CPU45.LOC:Local_timer_interrupts
      8070 ± 15%     +31.3%      10596        interrupts.CPU46.LOC:Local_timer_interrupts
      8093 ± 15%     +31.2%      10617        interrupts.CPU47.LOC:Local_timer_interrupts
      8085 ± 15%     +30.9%      10586        interrupts.CPU48.LOC:Local_timer_interrupts
      8093 ± 15%     +31.8%      10668        interrupts.CPU49.LOC:Local_timer_interrupts
      2611 ± 26%     -41.5%       1526 ± 15%  interrupts.CPU5.CAL:Function_call_interrupts
      8078 ± 15%     +31.1%      10592        interrupts.CPU5.LOC:Local_timer_interrupts
      1801 ± 23%     -51.2%     878.50 ± 27%  interrupts.CPU5.TLB:TLB_shootdowns
      8082 ± 15%     +31.1%      10596        interrupts.CPU50.LOC:Local_timer_interrupts
      8093 ± 15%     +31.3%      10628        interrupts.CPU51.LOC:Local_timer_interrupts
      8075 ± 15%     +31.3%      10600        interrupts.CPU52.LOC:Local_timer_interrupts
      8057 ± 15%     +31.6%      10602        interrupts.CPU53.LOC:Local_timer_interrupts
      8079 ± 15%     +30.8%      10566        interrupts.CPU54.LOC:Local_timer_interrupts
      8068 ± 15%     +31.3%      10596        interrupts.CPU55.LOC:Local_timer_interrupts
      8085 ± 15%     +31.0%      10593        interrupts.CPU56.LOC:Local_timer_interrupts
      8059 ± 15%     +31.6%      10605        interrupts.CPU57.LOC:Local_timer_interrupts
      8069 ± 15%     +31.3%      10599        interrupts.CPU58.LOC:Local_timer_interrupts
      8065 ± 15%     +31.5%      10608        interrupts.CPU59.LOC:Local_timer_interrupts
      8077 ± 15%     +31.2%      10600        interrupts.CPU6.LOC:Local_timer_interrupts
      8079 ± 15%     +31.3%      10607        interrupts.CPU60.LOC:Local_timer_interrupts
      8155 ± 15%     +30.3%      10628        interrupts.CPU61.LOC:Local_timer_interrupts
      8090 ± 15%     +31.4%      10633        interrupts.CPU62.LOC:Local_timer_interrupts
      8127 ± 16%     +30.5%      10603        interrupts.CPU63.LOC:Local_timer_interrupts
      8091 ± 15%     +31.8%      10664        interrupts.CPU64.LOC:Local_timer_interrupts
      8090 ± 16%     +31.1%      10604        interrupts.CPU65.LOC:Local_timer_interrupts
      8078 ± 15%     +32.6%      10714        interrupts.CPU66.LOC:Local_timer_interrupts
      8090 ± 15%     +31.2%      10611        interrupts.CPU67.LOC:Local_timer_interrupts
      8087 ± 15%     +31.1%      10606        interrupts.CPU68.LOC:Local_timer_interrupts
      8059 ± 15%     +31.4%      10588        interrupts.CPU69.LOC:Local_timer_interrupts
      7999 ± 16%     +32.6%      10610        interrupts.CPU7.LOC:Local_timer_interrupts
      8069 ± 15%     +31.7%      10625        interrupts.CPU70.LOC:Local_timer_interrupts
      8074 ± 15%     +31.1%      10586        interrupts.CPU71.LOC:Local_timer_interrupts
      8076 ± 15%     +31.3%      10602        interrupts.CPU72.LOC:Local_timer_interrupts
      8112 ± 16%     +30.9%      10618        interrupts.CPU73.LOC:Local_timer_interrupts
      8075 ± 15%     +31.7%      10635        interrupts.CPU74.LOC:Local_timer_interrupts
      8075 ± 15%     +31.2%      10595        interrupts.CPU75.LOC:Local_timer_interrupts
      8077 ± 15%     +31.2%      10598        interrupts.CPU76.LOC:Local_timer_interrupts
      8191 ± 14%     +30.4%      10682        interrupts.CPU77.LOC:Local_timer_interrupts
      8025 ± 16%     +32.2%      10609        interrupts.CPU78.LOC:Local_timer_interrupts
      8086 ± 15%     +31.3%      10615        interrupts.CPU79.LOC:Local_timer_interrupts
      8072 ± 15%     +31.3%      10600        interrupts.CPU8.LOC:Local_timer_interrupts
      3.00 ±137%   +1808.3%      57.25 ± 87%  interrupts.CPU8.RES:Rescheduling_interrupts
      8085 ± 15%     +32.1%      10681        interrupts.CPU80.LOC:Local_timer_interrupts
      8088 ± 15%     +31.2%      10608        interrupts.CPU81.LOC:Local_timer_interrupts
      8082 ± 15%     +31.5%      10625        interrupts.CPU82.LOC:Local_timer_interrupts
      8092 ± 15%     +31.2%      10620        interrupts.CPU83.LOC:Local_timer_interrupts
      8110 ± 15%     +30.7%      10600        interrupts.CPU84.LOC:Local_timer_interrupts
      8082 ± 15%     +31.4%      10623        interrupts.CPU85.LOC:Local_timer_interrupts
      8088 ± 15%     +31.0%      10597        interrupts.CPU86.LOC:Local_timer_interrupts
      8081 ± 15%     +31.2%      10600        interrupts.CPU87.LOC:Local_timer_interrupts
      8082 ± 15%     +31.1%      10598        interrupts.CPU9.LOC:Local_timer_interrupts
    207.25 ±170%    +301.3%     831.75 ± 52%  interrupts.CPU9.TLB:TLB_shootdowns
    711846 ± 15%     +31.2%     933907        interrupts.LOC:Local_timer_interrupts
      5731 ± 21%     +26.1%       7229 ± 16%  interrupts.RES:Rescheduling_interrupts



***************************************************************************************************
lkp-bdw-ep4: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
=========================================================================================
array_size/compiler/cpufreq_governor/kconfig/nr_threads/omp/rootfs/tbox_group/testcase:
  10000000/gcc-7/performance/x86_64-rhel-7.2/50%/true/debian-x86_64-2018-04-03.cgz/lkp-bdw-ep4/stream

commit: 
  24d0c1d6e6 ("sched/fair: Do not migrate due to a sync wakeup on exit")
  2c83362734 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")

24d0c1d6e65f635b 2c83362734dad8e48ccc0710b5c 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
           :4           25%           1:4     dmesg.WARNING:stack_recursion
         %stddev     %change         %stddev
             \          |                \  
     68287           -30.6%      47413 ±  3%  stream.add_bandwidth_MBps
     74377 ±  3%     -38.8%      45504 ±  3%  stream.copy_bandwidth_MBps
     71751           -39.8%      43172 ±  4%  stream.scale_bandwidth_MBps
     51.82           +51.5%      78.49 ±  3%  stream.time.user_time
    221.50 ± 45%    +431.6%       1177 ± 35%  stream.time.voluntary_context_switches
     74941           -34.4%      49156 ±  3%  stream.triad_bandwidth_MBps
      1792 ±  9%     +28.3%       2299 ± 14%  numa-meminfo.node0.PageTables
    447.25 ±  9%     +31.8%     589.50 ± 15%  numa-vmstat.node0.nr_page_table_pages
    983.50            +4.8%       1031 ±  2%  proc-vmstat.nr_page_table_pages
     18576 ± 24%     -29.5%      13093 ± 19%  softirqs.CPU44.TIMER
      6562 ± 14%     -31.8%       4477 ± 44%  turbostat.C1
     35.21 ± 11%     -31.6%      24.09 ±  9%  turbostat.CPU%c1
      2376 ± 12%     -27.3%       1728 ± 10%  slabinfo.eventpoll_epi.active_objs
      2376 ± 12%     -27.3%       1728 ± 10%  slabinfo.eventpoll_epi.num_objs
      4158 ± 12%     -27.3%       3024 ± 10%  slabinfo.eventpoll_pwq.active_objs
      4158 ± 12%     -27.3%       3024 ± 10%  slabinfo.eventpoll_pwq.num_objs
      3904 ±  5%      -9.2%       3544 ±  3%  slabinfo.skbuff_head_cache.active_objs
      1475 ±  4%     -13.1%       1282 ± 10%  slabinfo.task_group.active_objs
      1475 ±  4%     -13.1%       1282 ± 10%  slabinfo.task_group.num_objs
     40.50 ± 50%    +210.5%     125.75 ± 52%  interrupts.CPU1.RES:Rescheduling_interrupts
      1.00         +7075.0%      71.75 ±141%  interrupts.CPU22.RES:Rescheduling_interrupts
    150.25 ± 12%     -57.4%      64.00 ±100%  interrupts.CPU27.TLB:TLB_shootdowns
    166.50 ± 19%     -61.6%      64.00 ±100%  interrupts.CPU28.TLB:TLB_shootdowns
    160.50 ± 21%     -60.1%      64.00 ±100%  interrupts.CPU40.TLB:TLB_shootdowns
    174.25 ± 19%     -62.7%      65.00 ±100%  interrupts.CPU74.TLB:TLB_shootdowns
    149.25 ±  5%     -57.1%      64.00 ±100%  interrupts.CPU82.TLB:TLB_shootdowns
      7124 ± 14%     -29.4%       5030 ±  3%  interrupts.TLB:TLB_shootdowns
      4984 ± 45%     -78.7%       1060 ± 46%  sched_debug.cfs_rq:/.min_vruntime.min
     25.89 ± 57%     -66.2%       8.74 ±111%  sched_debug.cfs_rq:/.removed.load_avg.avg
    152.68 ± 29%     -57.6%      64.71 ±103%  sched_debug.cfs_rq:/.removed.load_avg.stddev
      1199 ± 56%     -66.4%     402.43 ±111%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
      7076 ± 29%     -57.9%       2981 ±103%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
      2404 ± 88%    -368.5%      -6456        sched_debug.cfs_rq:/.spread0.avg
     20903 ± 15%     -52.0%      10029 ± 29%  sched_debug.cfs_rq:/.spread0.max
     -2008          +507.1%     -12193        sched_debug.cfs_rq:/.spread0.min
    225.00 ±  8%     -16.7%     187.50 ± 15%  sched_debug.cpu.nr_switches.min
 1.356e+10 ±  9%     -28.4%  9.705e+09 ± 17%  perf-stat.branch-instructions
      3.35 ±  4%     +69.5%       5.67 ±  9%  perf-stat.cpi
   1694005 ± 15%     -32.0%    1152591 ± 21%  perf-stat.iTLB-loads
 6.601e+10 ±  9%     -24.9%  4.959e+10 ± 17%  perf-stat.instructions
      0.30 ±  4%     -40.5%       0.18 ± 10%  perf-stat.ipc
     39.73 ±  4%     -36.0        3.70 ± 35%  perf-stat.node-load-miss-rate%
  18391383 ±  6%     -90.9%    1670191 ± 34%  perf-stat.node-load-misses
  27864581 ±  2%     +56.6%   43648277 ±  3%  perf-stat.node-loads
     40.45 ±  4%     -38.9        1.51 ± 85%  perf-stat.node-store-miss-rate%
  44193260 ±  5%     -96.7%    1470898 ± 85%  perf-stat.node-store-misses
  65038165 ±  3%     +48.9%   96838822 ±  4%  perf-stat.node-stores
      7.68 ±133%      -4.9        2.78 ±173%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__ioctl.perf_evlist__disable.cmd_record.run_builtin
      7.68 ±133%      -4.9        2.78 ±173%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__ioctl.perf_evlist__disable.cmd_record
      7.68 ±133%      -4.9        2.78 ±173%  perf-profile.calltrace.cycles-pp.__ioctl.perf_evlist__disable.cmd_record.run_builtin.main
      7.68 ±133%      -4.9        2.78 ±173%  perf-profile.calltrace.cycles-pp.sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.__ioctl.perf_evlist__disable
      7.68 ±133%      -4.9        2.78 ±173%  perf-profile.calltrace.cycles-pp.perf_evlist__disable.cmd_record.run_builtin.main.generic_start_main
      4.37 ±111%      -4.4        0.00        perf-profile.calltrace.cycles-pp.__mutex_lock.show_interrupts.seq_read.proc_reg_read.__vfs_read
      4.37 ±111%      -4.4        0.00        perf-profile.calltrace.cycles-pp.mutex_spin_on_owner.__mutex_lock.show_interrupts.seq_read.proc_reg_read
      4.33 ±109%      -4.3        0.00        perf-profile.calltrace.cycles-pp.seq_printf.show_interrupts.seq_read.proc_reg_read.__vfs_read
      4.33 ±109%      -4.3        0.00        perf-profile.calltrace.cycles-pp.seq_vprintf.seq_printf.show_interrupts.seq_read.proc_reg_read
      4.33 ±109%      -4.3        0.00        perf-profile.calltrace.cycles-pp.vsnprintf.seq_vprintf.seq_printf.show_interrupts.seq_read
     13.76 ± 91%      -3.8       10.00 ±173%  perf-profile.calltrace.cycles-pp.proc_reg_read.__vfs_read.vfs_read.sys_read.do_syscall_64
     13.76 ± 91%      -3.8       10.00 ±173%  perf-profile.calltrace.cycles-pp.seq_read.proc_reg_read.__vfs_read.vfs_read.sys_read
     12.31 ± 90%      -2.3       10.00 ±173%  perf-profile.calltrace.cycles-pp.show_interrupts.seq_read.proc_reg_read.__vfs_read.vfs_read
      3.51 ±103%      -1.2        2.31 ±173%  perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64
      3.51 ±103%      -1.2        2.31 ±173%  perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.27 ±104%      +2.7       10.00 ±173%  perf-profile.calltrace.cycles-pp.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      7.27 ±104%      +2.7       10.00 ±173%  perf-profile.calltrace.cycles-pp.vfs_read.sys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.78 ±105%      -5.8        0.00        perf-profile.children.cycles-pp.seq_printf
      5.78 ±105%      -5.8        0.00        perf-profile.children.cycles-pp.seq_vprintf
      5.78 ±105%      -5.8        0.00        perf-profile.children.cycles-pp.vsnprintf
      7.68 ±133%      -4.9        2.78 ±173%  perf-profile.children.cycles-pp.perf_evlist__disable
     14.46 ± 83%      -4.5       10.00 ±173%  perf-profile.children.cycles-pp.seq_read
      4.37 ±111%      -4.4        0.00        perf-profile.children.cycles-pp.__mutex_lock
      4.37 ±111%      -4.4        0.00        perf-profile.children.cycles-pp.mutex_spin_on_owner
     13.76 ± 91%      -3.8       10.00 ±173%  perf-profile.children.cycles-pp.proc_reg_read
     12.31 ± 90%      -2.3       10.00 ±173%  perf-profile.children.cycles-pp.show_interrupts
      3.51 ±103%      -1.2        2.31 ±173%  perf-profile.children.cycles-pp.do_filp_open
      3.51 ±103%      -1.2        2.31 ±173%  perf-profile.children.cycles-pp.path_openat
      4.37 ±111%      -4.4        0.00        perf-profile.self.cycles-pp.mutex_spin_on_owner



***************************************************************************************************
lkp-skl-2sp4: 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/testcase:
  gcc-7/performance/x86_64-rhel-7.2/50%/debian-x86_64-2018-04-03.cgz/300s/lkp-skl-2sp4/pft

commit: 
  24d0c1d6e6 ("sched/fair: Do not migrate due to a sync wakeup on exit")
  2c83362734 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")

24d0c1d6e65f635b 2c83362734dad8e48ccc0710b5c 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.WARNING:at#for_ip_apic_timer_interrupt/0x
         %stddev     %change         %stddev
             \          |                \  
    230747           -42.5%     132607        pft.faults_per_sec_per_cpu
   7861494           -33.0%    5265420        pft.time.minor_page_faults
      4057           +17.5%       4767        pft.time.percent_of_cpu_this_job_got
     11947           +16.2%      13877        pft.time.system_time
    249.55           +86.6%     465.61 ±  7%  pft.time.user_time
    143006           +47.0%     210176 ±  4%  pft.time.voluntary_context_switches
   6501752 ±  9%     -40.5%    3870149 ± 34%  numa-numastat.node0.local_node
   6515543 ±  9%     -40.4%    3881040 ± 34%  numa-numastat.node0.numa_hit
     59.50           -10.1%      53.50        vmstat.cpu.id
     39.75 ±  2%     +20.1%      47.75        vmstat.procs.r
      5101 ±  2%     +31.8%       6723 ±  9%  vmstat.system.cs
     59.84            -6.1       53.74        mpstat.cpu.idle%
      0.00 ± 61%      +0.0        0.01 ± 33%  mpstat.cpu.soft%
     39.14            +5.4       44.54        mpstat.cpu.sys%
      0.99            +0.7        1.67 ±  7%  mpstat.cpu.usr%
     32208 ± 14%     +37.3%      44213 ±  4%  numa-meminfo.node0.SReclaimable
    411790 ± 16%    +161.6%    1077435 ± 17%  numa-meminfo.node1.AnonHugePages
      3769 ± 13%     +29.0%       4860 ±  9%  numa-meminfo.node1.PageTables
     41429 ± 11%     -31.3%      28458 ±  7%  numa-meminfo.node1.SReclaimable
     45429 ± 49%     -59.8%      18251 ±108%  numa-meminfo.node1.Shmem
     14247           -16.1%      11952 ±  4%  slabinfo.files_cache.active_objs
     14326           -15.5%      12108 ±  4%  slabinfo.files_cache.num_objs
      5797           -18.6%       4718 ±  8%  slabinfo.mm_struct.active_objs
      5863           -18.2%       4797 ±  8%  slabinfo.mm_struct.num_objs
      1193 ±  3%     -10.7%       1066 ±  2%  slabinfo.nsproxy.active_objs
      1193 ±  3%     -10.7%       1066 ±  2%  slabinfo.nsproxy.num_objs
   6180787 ±  3%     -15.2%    5242821 ± 14%  cpuidle.C1.time
    342482 ±  4%     +37.0%     469078 ±  7%  cpuidle.C1.usage
 2.939e+08 ±  3%     +31.0%   3.85e+08 ±  3%  cpuidle.C1E.time
    848558 ±  6%     +30.9%    1110683 ±  2%  cpuidle.C1E.usage
 1.796e+10           -10.1%  1.615e+10 ±  2%  cpuidle.C6.time
  18678065           -10.9%   16650357        cpuidle.C6.usage
 1.116e+08 ± 11%    +128.8%  2.554e+08 ± 10%  cpuidle.POLL.time
      6702 ±  3%     +38.5%       9283        cpuidle.POLL.usage
   5173027 ±  2%     +19.2%    6166000 ±  2%  meminfo.Active
   5128144 ±  2%     +19.4%    6121960 ±  2%  meminfo.Active(anon)
    733868          +130.3%    1690255 ± 33%  meminfo.AnonHugePages
   4988775 ±  2%     +19.2%    5948484 ±  3%  meminfo.AnonPages
 5.318e+08           +18.8%  6.319e+08        meminfo.Committed_AS
      1082 ± 84%     +87.0%       2024 ±  9%  meminfo.Mlocked
      7070           +26.5%       8941 ± 12%  meminfo.PageTables
      1083 ± 84%     +87.1%       2027 ±  9%  meminfo.Unevictable
      8051 ± 14%     +37.3%      11053 ±  4%  numa-vmstat.node0.nr_slab_reclaimable
   3703595 ±  7%     -43.4%    2095655 ± 15%  numa-vmstat.node0.numa_hit
   3689371 ±  7%     -43.5%    2084541 ± 15%  numa-vmstat.node0.numa_local
    191.75 ± 16%    +164.5%     507.25 ± 34%  numa-vmstat.node1.nr_anon_transparent_hugepages
     11353 ± 49%     -59.8%       4565 ±108%  numa-vmstat.node1.nr_shmem
     10358 ± 11%     -31.3%       7114 ±  7%  numa-vmstat.node1.nr_slab_reclaimable
   4161642 ±  6%     -12.2%    3653839 ±  9%  numa-vmstat.node1.numa_hit
   4020032 ±  6%     -12.7%    3508992 ±  9%  numa-vmstat.node1.numa_local
      1177           +14.0%       1341        turbostat.Avg_MHz
     42.29            +5.9       48.23        turbostat.Busy%
    340371 ±  4%     +37.3%     467253 ±  7%  turbostat.C1
    842290 ±  6%     +31.3%    1105626 ±  2%  turbostat.C1E
      0.93 ±  3%      +0.3        1.21 ±  4%  turbostat.C1E%
  18676469           -10.9%   16648172        turbostat.C6
     56.92            -6.2       50.68        turbostat.C6%
     56.18           -13.0%      48.89 ±  3%  turbostat.CPU%c1
    255.16            -8.8%     232.68        turbostat.PkgWatt
     65.13           -17.8%      53.51        turbostat.RAMWatt
     25420 ± 11%     -32.7%      17115 ± 18%  softirqs.CPU1.SCHED
     16168 ± 10%     -56.0%       7113 ± 22%  softirqs.CPU2.SCHED
     91499 ± 10%     +18.0%     107985 ± 15%  softirqs.CPU29.TIMER
     13561 ± 16%     -51.4%       6584 ± 16%  softirqs.CPU3.SCHED
     84774 ±  6%     +27.2%     107811 ± 18%  softirqs.CPU30.TIMER
     82435 ±  8%     +24.5%     102670 ± 21%  softirqs.CPU31.TIMER
     77710 ± 13%     +35.6%     105402 ± 20%  softirqs.CPU34.TIMER
     11043 ±  8%     -41.2%       6493 ± 19%  softirqs.CPU4.SCHED
     87767 ±  4%     +22.1%     107187 ± 15%  softirqs.CPU48.TIMER
     10229 ± 10%     -40.4%       6098 ± 21%  softirqs.CPU5.SCHED
     86042 ±  4%     +22.1%     105097 ± 17%  softirqs.CPU80.TIMER
     86912 ±  5%     +23.7%     107494 ± 16%  softirqs.CPU81.TIMER
     82985 ± 10%     +32.6%     110071 ± 13%  softirqs.CPU87.TIMER
     79387 ± 12%     +40.0%     111162 ± 13%  softirqs.CPU90.TIMER
     82466 ± 13%     +39.0%     114662 ± 11%  softirqs.CPU93.TIMER
      6797 ± 31%     -45.9%       3674 ± 34%  softirqs.CPU99.SCHED
    355628 ±  2%     +14.3%     406464 ±  6%  softirqs.RCU
   8446116            +9.7%    9267049        softirqs.TIMER
   1262046           +22.3%    1542932        proc-vmstat.nr_active_anon
   1228899           +22.1%    1500715        proc-vmstat.nr_anon_pages
    356.25 ±  9%    +100.1%     712.75 ±  3%  proc-vmstat.nr_anon_transparent_hugepages
   1473503            -1.8%    1446646        proc-vmstat.nr_dirty_background_threshold
   2950611            -1.8%    2896831        proc-vmstat.nr_dirty_threshold
  14627049            -1.8%   14358270        proc-vmstat.nr_free_pages
    269.75 ± 84%     +87.8%     506.50 ±  8%  proc-vmstat.nr_mlock
      1758           +21.4%       2136 ±  2%  proc-vmstat.nr_page_table_pages
     18410            -1.3%      18167        proc-vmstat.nr_slab_reclaimable
     52708            -2.6%      51354        proc-vmstat.nr_slab_unreclaimable
    270.00 ± 84%     +87.8%     507.00 ±  8%  proc-vmstat.nr_unevictable
   1262043           +22.3%    1542927        proc-vmstat.nr_zone_active_anon
    270.00 ± 84%     +87.8%     507.00 ±  8%  proc-vmstat.nr_zone_unevictable
  13918060           -31.1%    9583409        proc-vmstat.numa_hit
  13897019           -31.2%    9562209        proc-vmstat.numa_local
 2.743e+09           -32.9%  1.841e+09        proc-vmstat.pgalloc_normal
   8687624           -30.0%    6084756        proc-vmstat.pgfault
 2.742e+09           -32.9%  1.841e+09        proc-vmstat.pgfree
   5339231           -32.9%    3584118        proc-vmstat.thp_deferred_split_page
   5339481           -32.9%    3584076        proc-vmstat.thp_fault_alloc
 4.798e+11           +20.9%  5.802e+11 ±  7%  perf-stat.branch-instructions
      0.45 ±  4%      -0.1        0.37 ±  3%  perf-stat.branch-miss-rate%
     73.80            -8.9       64.94        perf-stat.cache-miss-rate%
 1.921e+10           -30.4%  1.337e+10 ±  2%  perf-stat.cache-misses
 2.603e+10           -20.9%  2.059e+10 ±  2%  perf-stat.cache-references
   1527973 ±  2%     +34.0%    2047421 ±  9%  perf-stat.context-switches
 3.625e+13           +15.2%  4.178e+13        perf-stat.cpu-cycles
     11029           -14.8%       9402 ±  5%  perf-stat.cpu-migrations
 5.477e+11           +18.3%  6.481e+11 ±  5%  perf-stat.dTLB-loads
      0.08            -0.0        0.05 ±  8%  perf-stat.dTLB-store-miss-rate%
 1.202e+08           -48.5%   61867822 ±  4%  perf-stat.dTLB-store-misses
 1.521e+11           -11.0%  1.354e+11 ±  4%  perf-stat.dTLB-stores
     22.62           +10.0       32.67 ± 10%  perf-stat.iTLB-load-miss-rate%
  63435492           +20.8%   76661143 ±  8%  perf-stat.iTLB-load-misses
  2.17e+08           -26.9%  1.586e+08 ±  7%  perf-stat.iTLB-loads
  1.91e+12           +16.2%  2.219e+12 ±  4%  perf-stat.instructions
   8642807           -30.0%    6049660        perf-stat.minor-faults
      8.59 ±  2%      -0.7        7.90 ±  6%  perf-stat.node-load-miss-rate%
 1.413e+08 ±  3%     -34.4%   92653965 ±  6%  perf-stat.node-load-misses
 1.504e+09           -28.1%  1.081e+09        perf-stat.node-loads
 1.152e+10           -27.4%   8.37e+09        perf-stat.node-stores
   8642829           -30.0%    6049673        perf-stat.page-faults
    611362 ± 11%     +11.6%     682342        interrupts.CAL:Function_call_interrupts
     34040 ± 11%     -33.7%      22553 ± 16%  interrupts.CPU1.RES:Rescheduling_interrupts
     94.00 ±115%     -82.7%      16.25 ± 84%  interrupts.CPU103.RES:Rescheduling_interrupts
      5539 ± 10%     +18.1%       6544        interrupts.CPU16.CAL:Function_call_interrupts
      5560 ±  9%     +18.5%       6589        interrupts.CPU17.CAL:Function_call_interrupts
     15417 ± 18%     -66.0%       5237 ± 47%  interrupts.CPU2.RES:Rescheduling_interrupts
    150.00 ± 74%    +346.3%     669.50 ± 32%  interrupts.CPU23.RES:Rescheduling_interrupts
    145.25 ± 65%    +430.1%     770.00 ± 61%  interrupts.CPU24.RES:Rescheduling_interrupts
    522.00 ± 75%   +3786.5%      20287 ± 52%  interrupts.CPU26.RES:Rescheduling_interrupts
    268.00 ± 57%   +1782.2%       5044 ± 80%  interrupts.CPU27.RES:Rescheduling_interrupts
    129.25 ± 48%    +713.9%       1052 ± 82%  interrupts.CPU28.RES:Rescheduling_interrupts
    101.75 ± 31%    +397.8%     506.50 ± 41%  interrupts.CPU29.RES:Rescheduling_interrupts
      9029 ± 10%     -72.3%       2498 ± 65%  interrupts.CPU3.RES:Rescheduling_interrupts
      5206 ± 16%     +28.1%       6672        interrupts.CPU36.CAL:Function_call_interrupts
      5821 ± 12%     -73.2%       1561 ± 51%  interrupts.CPU4.RES:Rescheduling_interrupts
     83.00 ± 17%     -32.5%      56.00 ± 29%  interrupts.CPU40.RES:Rescheduling_interrupts
      4349 ± 30%     -80.8%     836.00 ± 50%  interrupts.CPU5.RES:Rescheduling_interrupts
      2458 ± 44%     -68.3%     780.00 ±114%  interrupts.CPU55.NMI:Non-maskable_interrupts
      2458 ± 44%     -68.3%     780.00 ±114%  interrupts.CPU55.PMI:Performance_monitoring_interrupts
      3153 ± 48%     -78.7%     673.25 ± 74%  interrupts.CPU56.NMI:Non-maskable_interrupts
      3153 ± 48%     -78.7%     673.25 ± 74%  interrupts.CPU56.PMI:Performance_monitoring_interrupts
      3769 ± 13%     -80.7%     728.75 ± 48%  interrupts.CPU6.RES:Rescheduling_interrupts
    336.75 ±126%     -91.2%      29.50 ± 95%  interrupts.CPU74.RES:Rescheduling_interrupts
      1766 ± 54%     -65.7%     606.50 ± 54%  interrupts.CPU8.RES:Rescheduling_interrupts
     30.00 ± 28%    +432.5%     159.75 ± 92%  interrupts.CPU85.RES:Rescheduling_interrupts
    320.65 ±173%   +2126.1%       7138 ±100%  sched_debug.cfs_rq:/.MIN_vruntime.avg
     60894           +24.1%      75590 ± 14%  sched_debug.cfs_rq:/.exec_clock.avg
     13190 ±  9%    +144.9%      32297 ± 29%  sched_debug.cfs_rq:/.exec_clock.stddev
    320.65 ±173%   +2126.1%       7138 ±100%  sched_debug.cfs_rq:/.max_vruntime.avg
   3022932           +29.2%    3906103 ± 13%  sched_debug.cfs_rq:/.min_vruntime.avg
   5123973 ± 16%     +40.1%    7181169 ±  4%  sched_debug.cfs_rq:/.min_vruntime.max
    593877 ±  8%    +180.2%    1664130 ± 28%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.40 ±  6%     +45.8%       0.59 ±  3%  sched_debug.cfs_rq:/.nr_running.avg
      1.71 ±  2%     +67.3%       2.86 ± 19%  sched_debug.cfs_rq:/.nr_spread_over.avg
      1.29 ± 21%     +52.4%       1.97 ± 15%  sched_debug.cfs_rq:/.nr_spread_over.stddev
      9.02 ±  3%     +24.5%      11.23 ±  3%  sched_debug.cfs_rq:/.runnable_load_avg.avg
     28.38 ± 16%     -30.8%      19.62        sched_debug.cfs_rq:/.runnable_load_avg.max
     11.55 ± 11%     -26.2%       8.53 ±  3%  sched_debug.cfs_rq:/.runnable_load_avg.stddev
   3730816 ± 19%     +54.5%    5765499        sched_debug.cfs_rq:/.spread0.max
    593850 ±  8%    +180.2%    1664097 ± 28%  sched_debug.cfs_rq:/.spread0.stddev
    406.87 ±  7%     +48.3%     603.22 ±  3%  sched_debug.cfs_rq:/.util_avg.avg
     28.33 ± 16%     -30.6%      19.67        sched_debug.cpu.cpu_load[0].max
     11.23 ±  9%     -19.2%       9.07        sched_debug.cpu.cpu_load[0].stddev
     48.62 ± 23%     -59.1%      19.88        sched_debug.cpu.cpu_load[1].max
     12.53 ±  7%     -28.2%       9.00        sched_debug.cpu.cpu_load[1].stddev
     67.50 ± 44%     -68.6%      21.21 ± 11%  sched_debug.cpu.cpu_load[2].max
     14.14 ± 24%     -36.6%       8.97        sched_debug.cpu.cpu_load[2].stddev
    121.54 ±111%     -81.0%      23.08 ± 12%  sched_debug.cpu.cpu_load[3].max
     19.34 ± 71%     -53.4%       9.01        sched_debug.cpu.cpu_load[3].stddev
    139.17 ±118%     -80.4%      27.33 ± 14%  sched_debug.cpu.cpu_load[4].max
     20.88 ± 79%     -55.4%       9.30 ±  3%  sched_debug.cpu.cpu_load[4].stddev
     27826           -25.9%      20621        sched_debug.cpu.curr->pid.max
     12953 ±  3%     -21.4%      10185        sched_debug.cpu.curr->pid.stddev
      0.00 ± 17%     +24.4%       0.00 ± 12%  sched_debug.cpu.next_balance.stddev
    122120 ±  3%     +15.7%     141347 ±  2%  sched_debug.cpu.nr_load_updates.max
     10094 ±  7%    +127.8%      22993 ± 25%  sched_debug.cpu.nr_load_updates.stddev
      0.37 ±  6%     +15.8%       0.43        sched_debug.cpu.nr_running.avg
      7428 ±  2%     +23.6%       9182 ±  7%  sched_debug.cpu.nr_switches.avg
     42101 ±  7%    +105.4%      86497 ± 25%  sched_debug.cpu.nr_switches.max
      6842 ±  3%     +82.1%      12458 ± 10%  sched_debug.cpu.nr_switches.stddev
      6724           +24.8%       8389 ±  7%  sched_debug.cpu.sched_count.avg
     40147 ±  7%    +109.1%      83942 ± 26%  sched_debug.cpu.sched_count.max
      6608 ±  3%     +83.2%      12108 ± 11%  sched_debug.cpu.sched_count.stddev
      3230           +19.5%       3861 ±  6%  sched_debug.cpu.sched_goidle.avg
     19924 ±  7%    +110.0%      41838 ± 26%  sched_debug.cpu.sched_goidle.max
      3301 ±  3%     +78.9%       5906 ± 12%  sched_debug.cpu.sched_goidle.stddev
      3123 ±  2%     +29.1%       4032 ±  7%  sched_debug.cpu.ttwu_count.avg
     27111 ± 20%    +105.9%      55815 ± 34%  sched_debug.cpu.ttwu_count.max
      4227 ± 10%     +83.8%       7771 ± 11%  sched_debug.cpu.ttwu_count.stddev
    992.95           +56.0%       1548 ± 10%  sched_debug.cpu.ttwu_local.avg
      3551 ± 15%    +159.8%       9228 ± 48%  sched_debug.cpu.ttwu_local.max
    194.04 ± 71%     +90.8%     370.25 ±  9%  sched_debug.cpu.ttwu_local.min
    603.41 ±  7%    +114.2%       1292 ± 45%  sched_debug.cpu.ttwu_local.stddev
     19.92           -16.7        3.21 ± 15%  perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
     21.42           -16.0        5.42 ±  8%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     21.46           -16.0        5.49 ±  8%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     21.46           -16.0        5.49 ±  8%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     21.46           -16.0        5.49 ±  8%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     21.74 ±  2%     -15.9        5.86 ±  6%  perf-profile.calltrace.cycles-pp.secondary_startup_64
      1.17 ±  3%      +0.3        1.43 ±  2%  perf-profile.calltrace.cycles-pp.release_pages.tlb_flush_mmu_free.unmap_page_range.unmap_vmas.exit_mmap
      1.17 ±  3%      +0.3        1.44 ±  2%  perf-profile.calltrace.cycles-pp.tlb_flush_mmu_free.unmap_page_range.unmap_vmas.exit_mmap.mmput
      0.55            +0.4        0.95 ±  2%  perf-profile.calltrace.cycles-pp.___might_sleep
      1.43 ±  2%      +0.5        1.97        perf-profile.calltrace.cycles-pp._cond_resched.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      1.31 ±  3%      +0.6        1.88 ±  3%  perf-profile.calltrace.cycles-pp.__wake_up_parent.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.31 ±  3%      +0.6        1.88 ±  3%  perf-profile.calltrace.cycles-pp.do_group_exit.__wake_up_parent.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.31 ±  3%      +0.6        1.88 ±  3%  perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__wake_up_parent.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.30 ±  3%      +0.6        1.88 ±  3%  perf-profile.calltrace.cycles-pp.mmput.do_exit.do_group_exit.__wake_up_parent.do_syscall_64
      1.29 ±  3%      +0.6        1.88 ±  3%  perf-profile.calltrace.cycles-pp.exit_mmap.mmput.do_exit.do_group_exit.__wake_up_parent
      1.32 ±  3%      +0.6        1.90 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.32 ±  3%      +0.6        1.90 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.26 ±  3%      +0.6        1.85 ±  3%  perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.mmput.do_exit.do_group_exit
      1.26 ±  3%      +0.6        1.85 ±  3%  perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.mmput.do_exit
      0.92 ±  2%      +0.6        1.54        perf-profile.calltrace.cycles-pp.clear_huge_page
      0.55 ±102%      +0.9        1.42 ± 11%  perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.do_idle.cpu_startup_entry.start_secondary
      0.69 ±  4%      +1.0        1.69 ±  2%  perf-profile.calltrace.cycles-pp.clear_page_erms
      1.66 ±  2%      +1.1        2.72        perf-profile.calltrace.cycles-pp.___might_sleep.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      0.77 ±  4%      +1.7        2.44 ±  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.do_huge_pmd_anonymous_page.__handle_mm_fault
      0.76 ±  4%      +1.7        2.43 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_page_from_freelist.__alloc_pages_nodemask.do_huge_pmd_anonymous_page
      2.43 ±  2%      +2.1        4.53 ±  2%  perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
      2.44 ±  2%      +2.1        4.55 ±  2%  perf-profile.calltrace.cycles-pp.__alloc_pages_nodemask.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
     60.91            +9.2       70.07        perf-profile.calltrace.cycles-pp.clear_page_erms.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault
     69.30           +10.9       80.25        perf-profile.calltrace.cycles-pp.clear_huge_page.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault
     72.23           +13.1       85.35        perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     72.25           +13.1       85.38        perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     72.11           +13.1       85.25        perf-profile.calltrace.cycles-pp.do_huge_pmd_anonymous_page.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
     72.31           +13.2       85.48        perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
     72.31           +13.2       85.48        perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
     72.32           +13.2       85.50        perf-profile.calltrace.cycles-pp.page_fault
     20.15 ±  2%     -16.6        3.55 ±  4%  perf-profile.children.cycles-pp.intel_idle
     21.46           -16.0        5.49 ±  8%  perf-profile.children.cycles-pp.start_secondary
     21.71 ±  2%     -15.9        5.82 ±  6%  perf-profile.children.cycles-pp.cpuidle_enter_state
     21.74 ±  2%     -15.9        5.86 ±  6%  perf-profile.children.cycles-pp.do_idle
     21.74 ±  2%     -15.9        5.86 ±  6%  perf-profile.children.cycles-pp.secondary_startup_64
     21.74 ±  2%     -15.9        5.86 ±  6%  perf-profile.children.cycles-pp.cpu_startup_entry
      0.14 ± 16%      -0.1        0.08 ± 10%  perf-profile.children.cycles-pp.read
      0.17 ±  6%      -0.0        0.15 ±  4%  perf-profile.children.cycles-pp.native_irq_return_iret
      0.07 ±  6%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.08 ± 17%      +0.0        0.11 ±  9%  perf-profile.children.cycles-pp.cmd_stat
      0.08 ± 17%      +0.0        0.11 ±  9%  perf-profile.children.cycles-pp.__run_perf_stat
      0.08 ± 17%      +0.0        0.11 ±  9%  perf-profile.children.cycles-pp.process_interval
      0.08 ± 17%      +0.0        0.11 ±  9%  perf-profile.children.cycles-pp.read_counters
      0.07 ± 17%      +0.0        0.10 ± 10%  perf-profile.children.cycles-pp.perf_event_read
      0.08 ± 19%      +0.0        0.11 ±  9%  perf-profile.children.cycles-pp.perf_evsel__read_counter
      0.08 ± 19%      +0.0        0.11 ±  9%  perf-profile.children.cycles-pp.__read_nocancel
      0.08 ± 16%      +0.0        0.11 ± 11%  perf-profile.children.cycles-pp.perf_read
      0.07 ± 15%      +0.0        0.11 ±  7%  perf-profile.children.cycles-pp.smp_call_function_single
      0.03 ±100%      +0.0        0.06 ±  6%  perf-profile.children.cycles-pp.__pagevec_lru_add_fn
      0.57 ±  5%      +0.0        0.61 ±  4%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.00            +0.1        0.05        perf-profile.children.cycles-pp.___perf_sw_event
      0.08 ±  8%      +0.1        0.13 ±  3%  perf-profile.children.cycles-pp.pte_alloc_one
      0.00            +0.1        0.06 ±  7%  perf-profile.children.cycles-pp.__perf_sw_event
      0.05 ± 62%      +0.1        0.12 ± 19%  perf-profile.children.cycles-pp.ktime_get
      0.40 ±  2%      +0.1        0.49 ±  2%  perf-profile.children.cycles-pp.rcu_all_qs
      0.13 ±  3%      +0.1        0.25 ± 17%  perf-profile.children.cycles-pp.__put_compound_page
      0.10 ±  8%      +0.1        0.23 ± 18%  perf-profile.children.cycles-pp.__page_cache_release
      0.00            +0.2        0.16 ±  4%  perf-profile.children.cycles-pp.free_transhuge_page
      0.20 ±  4%      +0.2        0.41 ±  5%  perf-profile.children.cycles-pp.free_one_page
      1.17 ±  3%      +0.3        1.44 ±  2%  perf-profile.children.cycles-pp.tlb_flush_mmu_free
      1.18 ±  3%      +0.3        1.46 ±  2%  perf-profile.children.cycles-pp.release_pages
      0.07 ±  6%      +0.3        0.39 ± 11%  perf-profile.children.cycles-pp.zap_huge_pmd
      0.00            +0.3        0.34 ± 12%  perf-profile.children.cycles-pp.deferred_split_huge_page
      1.30 ±  3%      +0.6        1.88 ±  3%  perf-profile.children.cycles-pp.mmput
      1.31 ±  3%      +0.6        1.89 ±  3%  perf-profile.children.cycles-pp.__wake_up_parent
      1.31 ±  3%      +0.6        1.89 ±  3%  perf-profile.children.cycles-pp.do_group_exit
      1.31 ±  3%      +0.6        1.89 ±  3%  perf-profile.children.cycles-pp.do_exit
      1.30 ±  3%      +0.6        1.88 ±  3%  perf-profile.children.cycles-pp.exit_mmap
      1.27 ±  3%      +0.6        1.85 ±  3%  perf-profile.children.cycles-pp.unmap_vmas
      1.26 ±  3%      +0.6        1.85 ±  3%  perf-profile.children.cycles-pp.unmap_page_range
      1.71            +0.6        2.34        perf-profile.children.cycles-pp._cond_resched
      0.67 ± 73%      +0.8        1.42 ± 11%  perf-profile.children.cycles-pp.poll_idle
      2.22 ±  2%      +1.5        3.67        perf-profile.children.cycles-pp.___might_sleep
      2.54 ±  2%      +2.2        4.69 ±  2%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      2.52 ±  2%      +2.2        4.68 ±  2%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.97 ±  4%      +2.2        3.19 ±  5%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.57 ± 14%      +2.3        3.85 ± 10%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     62.03           +10.2       72.23        perf-profile.children.cycles-pp.clear_page_erms
     70.21           +11.6       81.78        perf-profile.children.cycles-pp.clear_huge_page
     72.25           +13.1       85.37        perf-profile.children.cycles-pp.__handle_mm_fault
     72.27           +13.1       85.40        perf-profile.children.cycles-pp.handle_mm_fault
     72.11           +13.1       85.26        perf-profile.children.cycles-pp.do_huge_pmd_anonymous_page
     72.33           +13.2       85.50        perf-profile.children.cycles-pp.do_page_fault
     72.33           +13.2       85.50        perf-profile.children.cycles-pp.__do_page_fault
     72.34           +13.2       85.51        perf-profile.children.cycles-pp.page_fault
     20.15 ±  2%     -16.6        3.55 ±  4%  perf-profile.self.cycles-pp.intel_idle
      0.78 ±  4%      -0.2        0.59 ±  4%  perf-profile.self.cycles-pp.__free_pages_ok
      0.17 ±  6%      -0.0        0.15 ±  4%  perf-profile.self.cycles-pp.native_irq_return_iret
      0.07 ±  6%      +0.0        0.08 ±  5%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.07 ± 15%      +0.0        0.11 ±  8%  perf-profile.self.cycles-pp.smp_call_function_single
      0.03 ±100%      +0.1        0.11 ± 23%  perf-profile.self.cycles-pp.ktime_get
      0.39 ±  2%      +0.1        0.48 ±  2%  perf-profile.self.cycles-pp.rcu_all_qs
      1.61            +0.4        2.00        perf-profile.self.cycles-pp.get_page_from_freelist
      1.46            +0.6        2.07        perf-profile.self.cycles-pp._cond_resched
      0.66 ± 73%      +0.8        1.42 ± 11%  perf-profile.self.cycles-pp.poll_idle
      6.08            +0.8        6.88        perf-profile.self.cycles-pp.clear_huge_page
      2.19            +1.5        3.65        perf-profile.self.cycles-pp.___might_sleep
      1.57 ± 14%      +2.3        3.85 ± 10%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
     61.40           +10.3       71.66        perf-profile.self.cycles-pp.clear_page_erms



***************************************************************************************************
lkp-hsx04: 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory
=========================================================================================
compiler/cpufreq_governor/iterations/kconfig/nr_task/rootfs/tbox_group/test/testcase:
  gcc-7/performance/30/x86_64-rhel-7.2/1600%/debian-x86_64-2016-08-31.cgz/lkp-hsx04/compute/reaim

commit: 
  24d0c1d6e6 ("sched/fair: Do not migrate due to a sync wakeup on exit")
  2c83362734 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")

24d0c1d6e65f635b 2c83362734dad8e48ccc0710b5c 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:4          -25%            :4     dmesg.WARNING:at#for_ip_error_entry/0x
          1:4          -25%            :4     dmesg.WARNING:at#for_ip_retint_user/0x
          1:4          -25%            :4     dmesg.WARNING:stack_going_in_the_wrong_direction?ip=__slab_free/0x
           :4           50%           2:4     dmesg.WARNING:stack_going_in_the_wrong_direction?ip=schedule_tail/0x
          6:4            0%           6:4     perf-profile.calltrace.cycles-pp.error_entry
          7:4           -1%           7:4     perf-profile.children.cycles-pp.error_entry
          5:4            1%           5:4     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
    124.00            -1.4%     122.25        reaim.child_systime
     87.40            -9.5%      79.08        reaim.jti
     12.09 ±  2%     +69.5%      20.49 ±  2%  reaim.std_dev_percent
      1.53           +50.7%       2.30 ±  2%  reaim.std_dev_time
  17534460            -5.9%   16498575        reaim.time.involuntary_context_switches
      3734            -1.4%       3683        reaim.time.system_time
  49689016 ± 17%     +26.1%   62671778 ±  6%  cpuidle.POLL.time
      0.39 ±  3%     -15.5%       0.33 ± 10%  turbostat.Pkg%pc6
     70.96            -2.2%      69.38        turbostat.RAMWatt
      1695           -10.0%       1525        vmstat.procs.r
     62117            -3.2%      60144        vmstat.system.cs
      7903 ±  4%     -56.9%       3405 ± 24%  slabinfo.eventpoll_epi.active_objs
      7903 ±  4%     -56.9%       3405 ± 24%  slabinfo.eventpoll_epi.num_objs
      6915 ±  4%     -56.9%       2979 ± 24%  slabinfo.eventpoll_pwq.active_objs
      6915 ±  4%     -56.9%       2979 ± 24%  slabinfo.eventpoll_pwq.num_objs
   1872863           -15.2%    1587464 ±  2%  meminfo.Active
   1789974           -15.8%    1506890        meminfo.Active(anon)
   1680128           -17.4%    1388186        meminfo.AnonPages
   3268127           -13.1%    2840098        meminfo.Committed_AS
      1855           -58.7%     766.75        meminfo.Mlocked
      1861           -58.5%     773.25        meminfo.Unevictable
      1.69            -0.2        1.52        perf-stat.cache-miss-rate%
 2.587e+10           -10.6%  2.313e+10        perf-stat.cache-misses
  30767306            -3.3%   29760895        perf-stat.context-switches
   5874734           -14.8%    5007295        perf-stat.cpu-migrations
     71.33            -0.8       70.58        perf-stat.node-load-miss-rate%
 8.155e+09           -10.6%  7.291e+09        perf-stat.node-load-misses
 3.277e+09            -7.3%  3.039e+09        perf-stat.node-loads
 2.765e+09           -11.5%  2.448e+09 ±  2%  perf-stat.node-store-misses
 1.519e+10           -13.1%   1.32e+10        perf-stat.node-stores
    396395 ±  4%     -18.4%     323621 ±  7%  numa-meminfo.node1.AnonPages
    281797 ±  8%      -9.8%     254296 ±  3%  numa-meminfo.node1.Inactive(file)
    521557 ±  6%     -17.7%     429170 ±  5%  numa-meminfo.node2.Active
    500694 ±  7%     -18.6%     407636 ±  5%  numa-meminfo.node2.Active(anon)
    478540 ±  8%     -17.2%     396013 ± 12%  numa-meminfo.node3.Active
    459279 ±  7%     -17.9%     377244 ± 13%  numa-meminfo.node3.Active(anon)
     32735 ± 41%     -97.8%     716.00 ±100%  numa-meminfo.node3.AnonHugePages
    434193 ±  4%     -23.8%     330780 ± 11%  numa-meminfo.node3.AnonPages
    285808 ±  5%     -10.7%     255254 ±  4%  numa-meminfo.node3.Inactive
    280289 ±  6%      -9.2%     254380 ±  3%  numa-meminfo.node3.Inactive(file)
    445067           -15.6%     375712        proc-vmstat.nr_active_anon
    417525           -17.1%     346011        proc-vmstat.nr_anon_pages
     56599            -5.0%      53752        proc-vmstat.nr_kernel_stack
    463.50           -58.9%     190.50        proc-vmstat.nr_mlock
    100103            -2.7%      97448        proc-vmstat.nr_slab_unreclaimable
    465.00           -58.7%     192.25        proc-vmstat.nr_unevictable
    445067           -15.6%     375712        proc-vmstat.nr_zone_active_anon
    465.00           -58.7%     192.25        proc-vmstat.nr_zone_unevictable
    195.25 ± 56%    +729.4%       1619 ± 87%  proc-vmstat.numa_hint_faults_local
      7687 ±  3%      +4.5%       8036 ±  2%  proc-vmstat.numa_pte_updates
    634.12 ±  6%     -26.8%     464.15 ± 12%  sched_debug.cfs_rq:/.exec_clock.stddev
  49884777 ±  5%     +21.8%   60745074        sched_debug.cfs_rq:/.min_vruntime.avg
  56196226 ±  6%     +25.2%   70352699        sched_debug.cfs_rq:/.min_vruntime.max
  43890810 ±  4%     +15.7%   50767701 ±  3%  sched_debug.cfs_rq:/.min_vruntime.min
   2523222 ±  6%     +98.7%    5013049 ± 19%  sched_debug.cfs_rq:/.min_vruntime.stddev
      1.56 ±  5%     +12.6%       1.76 ±  4%  sched_debug.cfs_rq:/.nr_spread_over.avg
  -5929434           +78.4%  -10580036        sched_debug.cfs_rq:/.spread0.min
   2487301 ±  6%    +101.3%    5007639 ± 19%  sched_debug.cfs_rq:/.spread0.stddev
    243.96 ±  2%     -10.0%     219.50        sched_debug.cfs_rq:/.util_avg.stddev
    670504 ±  5%     -15.4%     566996 ±  6%  sched_debug.cpu.avg_idle.min
     58664 ± 11%     +40.9%      82632 ± 12%  sched_debug.cpu.avg_idle.stddev
    470.51 ± 17%     -47.7%     246.06 ± 20%  sched_debug.cpu.clock.stddev
    470.51 ± 17%     -47.7%     246.06 ± 20%  sched_debug.cpu.clock_task.stddev
      7233 ±  9%     +18.2%       8551 ± 12%  sched_debug.cpu.load.avg
    137222 ± 62%    +119.4%     301049 ± 35%  sched_debug.cpu.load.max
     15787 ± 43%     +85.8%      29335 ± 30%  sched_debug.cpu.load.stddev
      0.00 ± 17%     -35.3%       0.00 ± 16%  sched_debug.cpu.next_balance.stddev
    169.75 ± 13%     -25.0%     127.31 ±  8%  sched_debug.cpu.sched_goidle.min
      3.38           +39.4%       4.71 ± 15%  sched_debug.rt_rq:/.rt_runtime.stddev
    131.75 ± 23%     -58.3%      55.00 ± 28%  numa-vmstat.node0.nr_mlock
    132.00 ± 23%     -58.1%      55.25 ± 27%  numa-vmstat.node0.nr_unevictable
    132.00 ± 23%     -58.1%      55.25 ± 27%  numa-vmstat.node0.nr_zone_unevictable
     98455 ±  4%     -18.2%      80516 ±  6%  numa-vmstat.node1.nr_anon_pages
    319.25 ±142%    -100.0%       0.00        numa-vmstat.node1.nr_dirtied
     70449 ±  8%      -9.8%      63574 ±  3%  numa-vmstat.node1.nr_inactive_file
     99.50 ±  3%     -60.1%      39.75        numa-vmstat.node1.nr_mlock
    100.00 ±  4%     -60.2%      39.75        numa-vmstat.node1.nr_unevictable
    319.25 ±142%    -100.0%       0.00        numa-vmstat.node1.nr_written
     70449 ±  8%      -9.8%      63574 ±  3%  numa-vmstat.node1.nr_zone_inactive_file
    100.00 ±  4%     -60.2%      39.75        numa-vmstat.node1.nr_zone_unevictable
    124407 ±  7%     -18.3%     101653 ±  5%  numa-vmstat.node2.nr_active_anon
    124405 ±  7%     -18.3%     101652 ±  5%  numa-vmstat.node2.nr_zone_active_anon
    114276 ±  7%     -17.8%      93899 ± 12%  numa-vmstat.node3.nr_active_anon
    108032 ±  4%     -23.8%      82312 ± 11%  numa-vmstat.node3.nr_anon_pages
     70071 ±  6%      -9.2%      63594 ±  3%  numa-vmstat.node3.nr_inactive_file
    134.00 ± 21%     -64.6%      47.50 ± 28%  numa-vmstat.node3.nr_mlock
    135.00 ± 21%     -64.6%      47.75 ± 28%  numa-vmstat.node3.nr_unevictable
    114275 ±  7%     -17.8%      93901 ± 12%  numa-vmstat.node3.nr_zone_active_anon
     70071 ±  6%      -9.2%      63594 ±  3%  numa-vmstat.node3.nr_zone_inactive_file
    135.00 ± 21%     -64.6%      47.75 ± 28%  numa-vmstat.node3.nr_zone_unevictable
 1.411e+09 ±  8%  -3.3e+08   1.077e+09 ± 14%  syscalls.sys_brk.noise.100%
  1.42e+09 ±  7%  -3.3e+08   1.086e+09 ± 14%  syscalls.sys_brk.noise.2%
 1.416e+09 ±  7%  -3.3e+08   1.082e+09 ± 14%  syscalls.sys_brk.noise.25%
  1.42e+09 ±  7%  -3.3e+08   1.085e+09 ± 14%  syscalls.sys_brk.noise.5%
 1.414e+09 ±  8%  -3.3e+08    1.08e+09 ± 14%  syscalls.sys_brk.noise.50%
 1.413e+09 ±  8%  -3.3e+08   1.079e+09 ± 14%  syscalls.sys_brk.noise.75%
 4.046e+09 ± 13%  -1.3e+09   2.793e+09 ±  6%  syscalls.sys_newstat.noise.100%
 4.119e+09 ± 12%  -1.3e+09   2.868e+09 ±  6%  syscalls.sys_newstat.noise.2%
 4.101e+09 ± 12%  -1.3e+09   2.849e+09 ±  6%  syscalls.sys_newstat.noise.25%
 4.117e+09 ± 12%  -1.3e+09   2.866e+09 ±  6%  syscalls.sys_newstat.noise.5%
  4.08e+09 ± 12%  -1.3e+09   2.828e+09 ±  6%  syscalls.sys_newstat.noise.50%
 4.062e+09 ± 12%  -1.3e+09   2.811e+09 ±  6%  syscalls.sys_newstat.noise.75%
 1.541e+11 ± 10%  -4.7e+10   1.072e+11 ± 15%  syscalls.sys_read.noise.100%
 1.541e+11 ± 10%  -4.7e+10   1.073e+11 ± 15%  syscalls.sys_read.noise.2%
 1.541e+11 ± 10%  -4.7e+10   1.073e+11 ± 15%  syscalls.sys_read.noise.25%
 1.541e+11 ± 10%  -4.7e+10   1.073e+11 ± 15%  syscalls.sys_read.noise.5%
 1.541e+11 ± 10%  -4.7e+10   1.073e+11 ± 15%  syscalls.sys_read.noise.50%
 1.541e+11 ± 10%  -4.7e+10   1.072e+11 ± 15%  syscalls.sys_read.noise.75%
    130453 ± 16%     -69.2%      40150 ±103%  syscalls.sys_rt_sigaction.max
  19777092 ±  4%  -1.3e+07     6543904 ±100%  syscalls.sys_rt_sigaction.noise.100%
  27560343 ±  2%  -1.7e+07    10538623 ±100%  syscalls.sys_rt_sigaction.noise.2%
  26718095 ±  3%  -1.7e+07     9971159 ±100%  syscalls.sys_rt_sigaction.noise.25%
  27550355 ±  2%  -1.7e+07    10510393 ±100%  syscalls.sys_rt_sigaction.noise.5%
  24718035 ±  3%  -1.6e+07     8356079 ±100%  syscalls.sys_rt_sigaction.noise.50%
  22249116 ±  4%  -1.5e+07     7149959 ±100%  syscalls.sys_rt_sigaction.noise.75%
  27266292 ± 11%  -1.6e+07    11532735 ±100%  syscalls.sys_times.noise.100%
  32337364 ±  9%  -1.8e+07    14209280 ±100%  syscalls.sys_times.noise.2%
  31159606 ±  9%  -1.8e+07    13621578 ±100%  syscalls.sys_times.noise.25%
  32279406 ±  9%  -1.8e+07    14182805 ±100%  syscalls.sys_times.noise.5%
  30086951 ±  9%  -1.7e+07    13027260 ±100%  syscalls.sys_times.noise.50%
  28978220 ± 10%  -1.7e+07    12426543 ±100%  syscalls.sys_times.noise.75%
      4922 ± 13%     -12.0%       4333        interrupts.CPU102.CAL:Function_call_interrupts
     15763 ± 12%     -17.4%      13021 ±  3%  interrupts.CPU104.RES:Rescheduling_interrupts
      4930 ± 14%     -12.3%       4325        interrupts.CPU11.CAL:Function_call_interrupts
      4910 ± 13%     -12.1%       4318        interrupts.CPU118.CAL:Function_call_interrupts
      4935 ± 14%     -17.6%       4064 ± 10%  interrupts.CPU119.CAL:Function_call_interrupts
      4926 ± 13%     -12.0%       4332        interrupts.CPU120.CAL:Function_call_interrupts
      4838 ± 32%     +34.0%       6483 ± 21%  interrupts.CPU122.NMI:Non-maskable_interrupts
      4838 ± 32%     +34.0%       6483 ± 21%  interrupts.CPU122.PMI:Performance_monitoring_interrupts
      4913 ± 14%     -17.4%       4058 ±  8%  interrupts.CPU123.CAL:Function_call_interrupts
      4907 ± 13%     -11.7%       4330        interrupts.CPU124.CAL:Function_call_interrupts
     14843 ±  3%     -10.8%      13246 ±  3%  interrupts.CPU126.RES:Rescheduling_interrupts
     15606           -17.6%      12854 ±  4%  interrupts.CPU130.RES:Rescheduling_interrupts
     15198 ±  8%     -14.3%      13028        interrupts.CPU131.RES:Rescheduling_interrupts
      4902 ± 14%     -12.6%       4286        interrupts.CPU134.CAL:Function_call_interrupts
      4878 ± 14%     -12.2%       4285        interrupts.CPU14.CAL:Function_call_interrupts
      4945 ± 14%     -13.1%       4297        interrupts.CPU140.CAL:Function_call_interrupts
     15827 ±  4%     -13.6%      13669 ±  5%  interrupts.CPU141.RES:Rescheduling_interrupts
      4786 ± 12%     -14.4%       4097 ±  7%  interrupts.CPU18.CAL:Function_call_interrupts
     15216 ± 12%     -15.3%      12883 ±  2%  interrupts.CPU18.RES:Rescheduling_interrupts
     15525 ±  5%     -15.0%      13200 ±  2%  interrupts.CPU19.RES:Rescheduling_interrupts
     14771 ±  3%      -7.8%      13620        interrupts.CPU2.RES:Rescheduling_interrupts
     15066 ±  7%     -10.6%      13468 ±  5%  interrupts.CPU20.RES:Rescheduling_interrupts
      4822 ± 10%      -9.7%       4352        interrupts.CPU21.CAL:Function_call_interrupts
      4908 ± 14%     -12.3%       4305        interrupts.CPU26.CAL:Function_call_interrupts
      4944 ± 31%     +54.2%       7623 ±  5%  interrupts.CPU27.NMI:Non-maskable_interrupts
      4944 ± 31%     +54.2%       7623 ±  5%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
      4947 ± 13%     -14.2%       4246 ±  4%  interrupts.CPU3.CAL:Function_call_interrupts
      4031 ±  5%     +73.1%       6977 ± 22%  interrupts.CPU3.NMI:Non-maskable_interrupts
      4031 ±  5%     +73.1%       6977 ± 22%  interrupts.CPU3.PMI:Performance_monitoring_interrupts
     15219 ±  4%     -10.9%      13568 ±  5%  interrupts.CPU3.RES:Rescheduling_interrupts
      4849 ± 32%     +54.9%       7510 ±  8%  interrupts.CPU30.NMI:Non-maskable_interrupts
      4849 ± 32%     +54.9%       7510 ±  8%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
     15365 ±  5%     -10.0%      13833 ±  6%  interrupts.CPU33.RES:Rescheduling_interrupts
      4758 ± 21%     +63.9%       7797        interrupts.CPU5.NMI:Non-maskable_interrupts
      4758 ± 21%     +63.9%       7797        interrupts.CPU5.PMI:Performance_monitoring_interrupts
      4937 ± 14%     -12.5%       4321        interrupts.CPU56.CAL:Function_call_interrupts
      4932 ± 14%     -12.0%       4340        interrupts.CPU58.CAL:Function_call_interrupts
      4935 ± 14%     -11.9%       4347        interrupts.CPU60.CAL:Function_call_interrupts
      4836 ± 32%     +35.3%       6542 ± 21%  interrupts.CPU60.NMI:Non-maskable_interrupts
      4836 ± 32%     +35.3%       6542 ± 21%  interrupts.CPU60.PMI:Performance_monitoring_interrupts
      4867 ± 14%     -12.5%       4260        interrupts.CPU62.CAL:Function_call_interrupts
      4922 ± 14%     -12.0%       4333        interrupts.CPU64.CAL:Function_call_interrupts
     15118 ±  7%     -14.0%      13008 ±  8%  interrupts.CPU64.RES:Rescheduling_interrupts
      4922 ± 13%     -12.1%       4329        interrupts.CPU65.CAL:Function_call_interrupts
     15324 ±  9%     -12.5%      13415 ±  3%  interrupts.CPU67.RES:Rescheduling_interrupts
      4884 ± 14%     -13.0%       4248        interrupts.CPU71.CAL:Function_call_interrupts
      4890 ± 14%     -11.8%       4311        interrupts.CPU77.CAL:Function_call_interrupts
      4889 ± 13%     -11.4%       4330        interrupts.CPU80.CAL:Function_call_interrupts
     14898 ±  3%      -9.4%      13504 ±  2%  interrupts.CPU80.RES:Rescheduling_interrupts
     15793 ± 12%     -13.6%      13651 ±  3%  interrupts.CPU83.RES:Rescheduling_interrupts
     14835 ±  3%      -9.2%      13466 ±  3%  interrupts.CPU85.RES:Rescheduling_interrupts
      4831 ± 32%     +41.0%       6809 ± 15%  interrupts.CPU86.NMI:Non-maskable_interrupts
      4831 ± 32%     +41.0%       6809 ± 15%  interrupts.CPU86.PMI:Performance_monitoring_interrupts
     15141 ± 11%     -14.7%      12921 ±  2%  interrupts.CPU91.RES:Rescheduling_interrupts
      4919 ± 13%     -12.0%       4328        interrupts.CPU94.CAL:Function_call_interrupts
     15256 ± 11%     -10.1%      13721 ±  6%  interrupts.CPU94.RES:Rescheduling_interrupts
     15869 ±  7%     -13.2%      13771 ±  4%  interrupts.CPU96.RES:Rescheduling_interrupts
      4919 ± 13%     -12.3%       4316        interrupts.CPU97.CAL:Function_call_interrupts
      4.50 ±  3%      -0.3        4.23        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve
      4.50 ±  3%      -0.3        4.23        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      4.48 ±  3%      -0.3        4.22        perf-profile.calltrace.cycles-pp.do_execveat_common.sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
      4.55 ±  3%      -0.2        4.35        perf-profile.calltrace.cycles-pp._do_fork.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.45 ±  2%      -0.1        2.31        perf-profile.calltrace.cycles-pp.__libc_fork
      1.67 ±  2%      -0.1        1.55 ±  2%  perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.mmput.flush_old_exec.load_elf_binary
      1.22 ±  6%      -0.1        1.13 ±  6%  perf-profile.calltrace.cycles-pp.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      2.17            -0.1        2.10        perf-profile.calltrace.cycles-pp.filemap_map_pages.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
      1.60            -0.1        1.53        perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.69 ±  4%      -0.1        0.62 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sync
      0.69 ±  4%      -0.1        0.63 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sync
      0.68 ±  4%      -0.1        0.62 ±  2%  perf-profile.calltrace.cycles-pp.sys_sync.do_syscall_64.entry_SYSCALL_64_after_hwframe.sync
      0.62 ±  4%      -0.1        0.56 ±  3%  perf-profile.calltrace.cycles-pp.iterate_supers.sys_sync.do_syscall_64.entry_SYSCALL_64_after_hwframe.sync
      0.92 ±  2%      -0.0        0.88        perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.71 ±  3%      -0.0        0.67        perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_open.do_syscall_64.entry_SYSCALL_64_after_hwframe.setlocale
      0.72            -0.0        0.68 ±  3%  perf-profile.calltrace.cycles-pp.rcu_process_callbacks.__softirqentry_text_start.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt
      0.66 ±  5%      +0.0        0.71 ±  2%  perf-profile.calltrace.cycles-pp.__pagevec_lru_add_fn.pagevec_lru_move_fn.__lru_cache_add.__handle_mm_fault.handle_mm_fault
      1.23 ±  3%      +0.1        1.30 ±  2%  perf-profile.calltrace.cycles-pp.__lru_cache_add.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault
      1.06 ±  4%      +0.1        1.16        perf-profile.calltrace.cycles-pp.__perf_sw_event.__do_page_fault.do_page_fault.page_fault
      1.50 ±  3%      +0.1        1.61 ±  4%  perf-profile.calltrace.cycles-pp.task_tick_fair.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer
      1.89            +0.1        2.02 ±  2%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
     12.85            +0.2       13.05        perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     13.47            +0.2       13.67        perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault
     16.66            +0.2       16.91        perf-profile.calltrace.cycles-pp.page_fault
     16.49            +0.3       16.75        perf-profile.calltrace.cycles-pp.do_page_fault.page_fault
     16.46            +0.3       16.72        perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault
      0.26 ±100%      +0.3        0.54 ±  3%  perf-profile.calltrace.cycles-pp.selinux_vm_enough_memory.security_vm_enough_memory_mm.do_brk_flags.sys_brk.do_syscall_64
      0.00            +0.5        0.51 ±  2%  perf-profile.calltrace.cycles-pp.__xstat64
      1.24 ± 16%      -0.3        0.90 ± 10%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      4.98 ±  2%      -0.2        4.80        perf-profile.children.cycles-pp.unmap_vmas
      1.90 ±  4%      -0.1        1.76 ±  5%  perf-profile.children.cycles-pp.__softirqentry_text_start
      2.45 ±  2%      -0.1        2.31        perf-profile.children.cycles-pp.__libc_fork
      2.38            -0.1        2.27        perf-profile.children.cycles-pp.do_mmap
      2.73            -0.1        2.63        perf-profile.children.cycles-pp.free_pgtables
      2.10            -0.1        2.00        perf-profile.children.cycles-pp.mmap_region
      2.54            -0.1        2.45        perf-profile.children.cycles-pp.vm_mmap_pgoff
      0.74 ±  3%      -0.1        0.66 ±  3%  perf-profile.children.cycles-pp.wake_up_new_task
      2.11            -0.1        2.03        perf-profile.children.cycles-pp.path_openat
      2.12            -0.1        2.04        perf-profile.children.cycles-pp.do_filp_open
      0.69 ±  3%      -0.1        0.62 ±  2%  perf-profile.children.cycles-pp.sys_sync
      0.36 ±  4%      -0.1        0.30 ±  5%  perf-profile.children.cycles-pp.update_rq_clock
      0.62 ±  4%      -0.1        0.57 ±  3%  perf-profile.children.cycles-pp.iterate_supers
      0.15 ±  6%      -0.0        0.11 ±  9%  perf-profile.children.cycles-pp.mark_page_accessed
      0.09 ± 11%      -0.0        0.06 ± 13%  perf-profile.children.cycles-pp.alloc_vmap_area
      0.78 ±  2%      -0.0        0.75        perf-profile.children.cycles-pp.lookup_fast
      0.11 ±  6%      -0.0        0.08 ± 17%  perf-profile.children.cycles-pp.__get_vm_area_node
      0.15 ±  7%      -0.0        0.12 ±  5%  perf-profile.children.cycles-pp.__vunmap
      0.15 ±  5%      -0.0        0.12 ±  8%  perf-profile.children.cycles-pp.__d_alloc
      0.15 ±  5%      -0.0        0.13 ±  5%  perf-profile.children.cycles-pp.free_work
      0.31 ±  3%      -0.0        0.29        perf-profile.children.cycles-pp.__update_load_avg_se
      0.10 ±  7%      +0.0        0.12 ±  7%  perf-profile.children.cycles-pp.generic_permission
      0.10 ±  8%      +0.0        0.12 ±  6%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.10 ± 14%      +0.0        0.14 ± 10%  perf-profile.children.cycles-pp.__do_fault
      0.47 ±  3%      +0.0        0.51 ±  2%  perf-profile.children.cycles-pp.__xstat64
      0.01 ±173%      +0.0        0.05 ±  9%  perf-profile.children.cycles-pp.__alloc_fd
      0.68            +0.0        0.73 ±  3%  perf-profile.children.cycles-pp.sys_read
      0.18 ±  4%      +0.1        0.23 ± 16%  perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
      0.41 ±  6%      +0.1        0.48 ±  8%  perf-profile.children.cycles-pp.mem_cgroup_try_charge
      2.12 ±  3%      +0.1        2.20        perf-profile.children.cycles-pp.vfs_statx
      1.60 ±  2%      +0.1        1.70 ±  3%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      1.64 ±  2%      +0.1        1.75 ±  3%  perf-profile.children.cycles-pp.task_tick_fair
      1.21 ±  4%      +0.1        1.31        perf-profile.children.cycles-pp.__perf_sw_event
      0.35 ±  9%      +0.1        0.46 ±  4%  perf-profile.children.cycles-pp.wait_consider_task
      2.10 ±  2%      +0.1        2.22        perf-profile.children.cycles-pp.syscall_return_via_sysret
      3.41 ±  2%      +0.1        3.54        perf-profile.children.cycles-pp.tlb_flush_mmu_free
      0.75 ±  4%      +0.1        0.89 ±  4%  perf-profile.children.cycles-pp.SYSC_wait4
      0.75 ±  4%      +0.1        0.89 ±  4%  perf-profile.children.cycles-pp.kernel_wait4
      0.70 ±  5%      +0.1        0.84 ±  5%  perf-profile.children.cycles-pp.do_wait
      5.25            +0.2        5.42        perf-profile.children.cycles-pp.alloc_pages_vma
     19.59            +0.2       19.82        perf-profile.children.cycles-pp.do_page_fault
      1.23 ± 16%      -0.3        0.90 ± 10%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      3.01            -0.1        2.90        perf-profile.self.cycles-pp.unmap_page_range
      0.32 ±  6%      -0.1        0.26 ±  6%  perf-profile.self.cycles-pp.update_rq_clock
      0.19 ± 13%      -0.0        0.15 ±  7%  perf-profile.self.cycles-pp.scheduler_tick
      1.20 ±  2%      -0.0        1.16        perf-profile.self.cycles-pp._dl_addr
      0.84            -0.0        0.81        perf-profile.self.cycles-pp.copy_page_range
      0.15 ±  7%      -0.0        0.11 ±  9%  perf-profile.self.cycles-pp.mark_page_accessed
      0.09            -0.0        0.07 ± 16%  perf-profile.self.cycles-pp.__d_alloc
      0.15 ±  3%      -0.0        0.14 ±  6%  perf-profile.self.cycles-pp.lru_cache_add_active_or_unevictable
      0.22 ±  8%      +0.0        0.26 ±  5%  perf-profile.self.cycles-pp.__perf_sw_event
      0.18 ±  4%      +0.0        0.23 ± 16%  perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
      0.07            +0.1        0.12 ±  4%  perf-profile.self.cycles-pp.queued_write_lock_slowpath
      0.69 ±  5%      +0.1        0.76 ±  2%  perf-profile.self.cycles-pp.__pagevec_lru_add_fn
      0.69 ±  7%      +0.1        0.78 ±  4%  perf-profile.self.cycles-pp.__vma_adjust
      0.05 ± 62%      +0.1        0.14 ± 20%  perf-profile.self.cycles-pp.wait_consider_task
      2.10 ±  2%      +0.1        2.22        perf-profile.self.cycles-pp.syscall_return_via_sysret



***************************************************************************************************
lkp-bdw-ep6: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
=========================================================================================
class/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/testcase/testtime:
  pipe/gcc-7/performance/x86_64-rhel-7.2/100%/debian-x86_64-2016-08-31.cgz/lkp-bdw-ep6/stress-ng/1s

commit: 
  24d0c1d6e6 ("sched/fair: Do not migrate due to a sync wakeup on exit")
  2c83362734 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")

24d0c1d6e65f635b 2c83362734dad8e48ccc0710b5c 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   3463075 ± 10%     +76.2%    6101037 ± 32%  stress-ng.fifo.ops
   3460753 ± 10%     +76.2%    6096158 ± 32%  stress-ng.fifo.ops_per_sec
  22252433 ±  4%     -41.9%   12934864 ± 38%  cpuidle.C1.time
   1240735 ±  5%     -41.6%     724182 ± 38%  cpuidle.C1.usage
      9537 ± 12%     +72.0%      16402 ± 19%  sched_debug.cpu.nr_switches.max
      1567 ±  5%     +28.0%       2006 ± 11%  sched_debug.cpu.nr_switches.stddev
   1239038 ±  5%     -41.7%     722719 ± 38%  turbostat.C1
      3.32 ±  5%      -1.4        1.89 ± 41%  turbostat.C1%
    696934 ±  3%      +7.9%     751814 ±  6%  turbostat.IRQ
      2473 ±  9%     -25.8%       1834 ± 10%  slabinfo.eventpoll_epi.active_objs
      2473 ±  9%     -25.8%       1834 ± 10%  slabinfo.eventpoll_epi.num_objs
      4267 ±  9%     -25.8%       3165 ± 10%  slabinfo.eventpoll_pwq.active_objs
      4267 ±  9%     -25.8%       3165 ± 10%  slabinfo.eventpoll_pwq.num_objs
    316.70 ± 42%    +286.3%       1223 ± 88%  interrupts.CPU1.RES:Rescheduling_interrupts
    214.30 ± 35%    +225.2%     697.00 ± 67%  interrupts.CPU14.RES:Rescheduling_interrupts
    249.60 ± 36%    +241.5%     852.30 ± 82%  interrupts.CPU15.RES:Rescheduling_interrupts
    280.70 ± 45%    +166.1%     746.90 ± 68%  interrupts.CPU17.RES:Rescheduling_interrupts
    290.10 ± 31%    +178.8%     808.70 ± 88%  interrupts.CPU2.RES:Rescheduling_interrupts
    221.70 ± 27%    +239.7%     753.10 ± 63%  interrupts.CPU20.RES:Rescheduling_interrupts
    256.40 ± 28%    +174.5%     703.90 ± 97%  interrupts.CPU49.RES:Rescheduling_interrupts
     22379 ± 11%    +107.5%      46443 ± 20%  interrupts.RES:Rescheduling_interrupts



***************************************************************************************************
lkp-bdw-ep6: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory
=========================================================================================
class/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/testcase/testtime:
  cpu/gcc-7/performance/x86_64-rhel-7.2/100%/debian-x86_64-2016-08-31.cgz/lkp-bdw-ep6/stress-ng/1s

commit: 
  24d0c1d6e6 ("sched/fair: Do not migrate due to a sync wakeup on exit")
  2c83362734 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")

24d0c1d6e65f635b 2c83362734dad8e48ccc0710b5c 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    189975           -25.8%     140873 ±  7%  stress-ng.context.ops
    190032           -25.9%     140895 ±  7%  stress-ng.context.ops_per_sec
    180580           -15.3%     152874 ±  9%  stress-ng.hsearch.ops
    180604           -15.4%     152852 ±  9%  stress-ng.hsearch.ops_per_sec
     47965            +6.3%      50971        stress-ng.time.involuntary_context_switches
   4076749            +6.0%    4319630        stress-ng.time.minor_page_faults
      6259 ±  3%      -8.8%       5706 ±  2%  stress-ng.time.percent_of_cpu_this_job_got
      1601            -8.3%       1468        stress-ng.time.user_time
    836.00           -17.3%     691.50 ±  3%  stress-ng.tsearch.ops
    806.28           -17.1%     668.40 ±  3%  stress-ng.tsearch.ops_per_sec
    103796 ± 48%     -49.0%      52979 ±  9%  meminfo.AnonHugePages
     54.87 ±  3%      -6.2       48.67 ±  6%  mpstat.cpu.usr%
     64134 ± 45%     -62.1%      24298 ± 56%  numa-meminfo.node0.AnonHugePages
      3.66 ±105%      -3.7        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      3.66 ±105%      -3.7        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
    338.42 ±  4%     -10.1%     304.30        sched_debug.cfs_rq:/.util_avg.avg
     10253 ± 21%     +66.7%      17088 ±  6%  sched_debug.cpu.nr_switches.max
      1602 ±  7%     +25.6%       2013 ±  5%  sched_debug.cpu.nr_switches.stddev
      2071 ± 15%     -18.7%       1683 ±  8%  slabinfo.eventpoll_epi.num_objs
      3605 ± 15%     -19.2%       2912 ±  8%  slabinfo.eventpoll_pwq.num_objs
    490.00 ± 10%     -28.0%     353.00 ±  3%  slabinfo.file_lock_cache.num_objs
     11628 ±  3%      -8.3%      10665 ±  2%  slabinfo.kmalloc-96.active_objs
  33103068 ±  5%     -14.1%   28438786 ±  7%  cpuidle.C3.time
    141695 ±  5%      -8.0%     130380 ±  6%  cpuidle.C3.usage
 6.907e+08 ± 10%     +29.0%  8.912e+08 ± 11%  cpuidle.C6.time
    642637 ± 11%     +20.4%     773937 ±  8%  cpuidle.C6.usage
   3162428 ± 53%     +81.0%    5724968 ± 26%  cpuidle.POLL.time
     22657 ±  2%      -9.2%      20570        softirqs.CPU46.TIMER
     22637 ±  3%      -7.4%      20972 ±  4%  softirqs.CPU5.TIMER
     21862 ±  3%      -7.9%      20139 ±  2%  softirqs.CPU52.TIMER
     22165 ±  4%      -9.8%      19993        softirqs.CPU54.TIMER
     21851 ±  4%      -8.6%      19977 ±  2%  softirqs.CPU59.TIMER
      9771            +2.0%       9966        proc-vmstat.nr_mapped
      2767 ±  2%      -5.0%       2629 ±  2%  proc-vmstat.nr_page_table_pages
      1602 ± 13%    +188.0%       4614 ± 27%  proc-vmstat.numa_hint_faults
    379391 ±  2%     -38.9%     231826 ± 16%  proc-vmstat.numa_pte_updates
   4840033            +4.8%    5074161        proc-vmstat.pgfault
      2502 ± 80%     -81.7%     457.50 ±  7%  proc-vmstat.thp_fault_alloc
      2011 ±  2%      -8.6%       1839 ±  3%  turbostat.Avg_MHz
      1.12 ±  7%      -0.2        0.95 ± 11%  turbostat.C3%
    644225 ± 10%     +20.2%     774089 ±  8%  turbostat.C6
     23.43 ±  9%      +6.3       29.72 ±  7%  turbostat.C6%
     12.89 ± 10%     +17.2%      15.10 ±  7%  turbostat.CPU%c1
     14.33 ±  6%     +28.9%      18.47 ± 11%  turbostat.CPU%c6
    220.42            -4.7%     210.07 ±  2%  turbostat.PkgWatt
 1.206e+11 ±  7%     -22.6%  9.329e+10 ±  9%  perf-stat.branch-instructions
      4.75 ±  5%      +0.5        5.25        perf-stat.branch-miss-rate%
 5.708e+09           -14.0%  4.908e+09 ± 10%  perf-stat.branch-misses
 1.676e+09 ± 19%     -24.5%  1.266e+09 ±  4%  perf-stat.cache-references
 1.017e+12 ±  3%     -16.3%  8.519e+11 ±  3%  perf-stat.cpu-cycles
      0.02 ±  3%      +0.0        0.03 ±  5%  perf-stat.dTLB-load-miss-rate%
 1.196e+11 ±  5%     -16.7%  9.964e+10 ±  5%  perf-stat.dTLB-loads
      0.01 ±  4%      +0.0        0.01 ±  4%  perf-stat.dTLB-store-miss-rate%
   5273411 ±  5%      +9.0%    5746770 ±  3%  perf-stat.dTLB-store-misses
 4.876e+10 ±  8%     -21.0%  3.853e+10 ±  4%  perf-stat.dTLB-stores
     42.05 ±  2%      -4.0       38.06        perf-stat.iTLB-load-miss-rate%
 2.264e+08 ±  3%     -31.9%  1.542e+08 ±  7%  perf-stat.iTLB-load-misses
 3.119e+08           -19.6%  2.508e+08 ±  6%  perf-stat.iTLB-loads
  6.53e+11 ±  6%     -19.6%  5.253e+11 ±  5%  perf-stat.instructions
  13642943 ±  9%     -14.0%   11738663 ±  4%  perf-stat.node-load-misses
   3805381 ± 34%     -32.1%    2584140 ±  8%  perf-stat.node-stores
     30946 ±  6%     -15.8%      26047 ±  7%  interrupts.CPU0.LOC:Local_timer_interrupts
      1326 ±  8%     +22.0%       1618 ± 12%  interrupts.CPU1.RES:Rescheduling_interrupts
    271.50 ±  4%      -9.8%     245.00 ±  5%  interrupts.CPU12.RES:Rescheduling_interrupts
     31204 ±  4%     -13.8%      26901 ±  3%  interrupts.CPU13.LOC:Local_timer_interrupts
    407.50 ± 25%     -41.7%     237.75 ± 17%  interrupts.CPU13.RES:Rescheduling_interrupts
     37.75 ± 61%    +341.7%     166.75 ± 68%  interrupts.CPU14.34:IR-PCI-MSI.1572865-edge.eth0-TxRx-0
    527.00 ± 24%     +45.3%     765.75 ± 13%  interrupts.CPU2.RES:Rescheduling_interrupts
    359.50 ± 14%     +67.1%     600.75 ± 18%  interrupts.CPU22.RES:Rescheduling_interrupts
    350.50 ± 25%     -42.4%     201.75 ±  3%  interrupts.CPU27.RES:Rescheduling_interrupts
    318.25 ± 23%     +90.7%     606.75 ± 25%  interrupts.CPU3.RES:Rescheduling_interrupts
      6321 ±  7%     +36.9%       8655 ±  4%  interrupts.CPU30.CAL:Function_call_interrupts
      3534 ± 13%     +74.1%       6154 ±  6%  interrupts.CPU30.TLB:TLB_shootdowns
    287.75 ± 11%     -28.7%     205.25 ±  6%  interrupts.CPU31.RES:Rescheduling_interrupts
    326.75 ± 13%     -26.9%     239.00 ± 16%  interrupts.CPU32.RES:Rescheduling_interrupts
    265.25 ±  7%     -22.9%     204.50 ± 14%  interrupts.CPU35.RES:Rescheduling_interrupts
      6704 ± 13%     +21.5%       8146 ±  7%  interrupts.CPU36.CAL:Function_call_interrupts
      3893 ± 20%     +46.3%       5696 ±  8%  interrupts.CPU36.TLB:TLB_shootdowns
      6798 ± 18%     +25.9%       8555 ±  5%  interrupts.CPU37.CAL:Function_call_interrupts
    273.00 ± 19%     -30.1%     190.75 ± 11%  interrupts.CPU37.RES:Rescheduling_interrupts
      4075 ± 29%     +48.7%       6058 ±  7%  interrupts.CPU37.TLB:TLB_shootdowns
    303.75 ± 12%     -38.5%     186.75 ± 22%  interrupts.CPU41.RES:Rescheduling_interrupts
      6346 ±  9%     +46.8%       9315 ± 12%  interrupts.CPU42.CAL:Function_call_interrupts
      3690 ± 15%     +82.6%       6736 ± 20%  interrupts.CPU42.TLB:TLB_shootdowns
    255.75 ± 10%     -37.8%     159.00 ± 12%  interrupts.CPU43.RES:Rescheduling_interrupts
      7579 ±  9%     -25.8%       5620 ± 33%  interrupts.CPU50.CAL:Function_call_interrupts
    398.75 ± 47%     -50.3%     198.00 ± 22%  interrupts.CPU51.RES:Rescheduling_interrupts
     31904 ±  5%     -13.3%      27660 ±  9%  interrupts.CPU52.LOC:Local_timer_interrupts
      7487 ± 11%     -19.2%       6048 ±  9%  interrupts.CPU54.CAL:Function_call_interrupts
      4747 ± 18%     -26.8%       3474 ± 16%  interrupts.CPU54.TLB:TLB_shootdowns
      7486 ± 17%     -30.7%       5187 ± 11%  interrupts.CPU57.CAL:Function_call_interrupts
    270.75 ± 11%     -16.9%     225.00 ± 11%  interrupts.CPU57.RES:Rescheduling_interrupts
      4782 ± 28%     -46.9%       2537 ± 31%  interrupts.CPU57.TLB:TLB_shootdowns
    238.25 ±  7%     +12.7%     268.50 ±  5%  interrupts.CPU63.RES:Rescheduling_interrupts
    286.75 ± 10%     -29.6%     202.00 ± 10%  interrupts.CPU64.RES:Rescheduling_interrupts
    290.25 ± 14%     -41.3%     170.25 ± 29%  interrupts.CPU65.RES:Rescheduling_interrupts
      6570 ± 31%     +30.0%       8538 ±  4%  interrupts.CPU66.CAL:Function_call_interrupts
      3829 ± 58%     +60.1%       6130 ±  7%  interrupts.CPU66.TLB:TLB_shootdowns
    267.75 ± 18%     -34.9%     174.25 ± 22%  interrupts.CPU67.RES:Rescheduling_interrupts
    257.00 ± 12%     -30.4%     178.75 ± 20%  interrupts.CPU68.RES:Rescheduling_interrupts
      4377 ± 28%     +47.4%       6450 ± 17%  interrupts.CPU68.TLB:TLB_shootdowns
    244.50 ± 17%     -22.4%     189.75        interrupts.CPU69.RES:Rescheduling_interrupts
      4263 ± 30%     +41.9%       6050 ±  7%  interrupts.CPU69.TLB:TLB_shootdowns
    262.50 ± 17%     -28.9%     186.75 ±  9%  interrupts.CPU71.RES:Rescheduling_interrupts
      6230 ± 10%     +43.2%       8922 ± 12%  interrupts.CPU72.CAL:Function_call_interrupts
     30763 ±  6%     -11.1%      27340 ±  7%  interrupts.CPU72.LOC:Local_timer_interrupts
      3562 ± 18%     +84.8%       6584 ± 19%  interrupts.CPU72.TLB:TLB_shootdowns
    259.25 ±  2%     -19.7%     208.25 ± 10%  interrupts.CPU74.RES:Rescheduling_interrupts
    237.75 ± 21%     -24.5%     179.50 ± 11%  interrupts.CPU76.RES:Rescheduling_interrupts
    297.25 ± 29%     -40.2%     177.75 ±  9%  interrupts.CPU79.RES:Rescheduling_interrupts
    271.75 ± 14%     -33.0%     182.00 ± 10%  interrupts.CPU80.RES:Rescheduling_interrupts
    221.25 ± 18%     -23.1%     170.25 ± 13%  interrupts.CPU81.RES:Rescheduling_interrupts
    323.75 ± 25%     -34.7%     211.50 ±  9%  interrupts.CPU86.RES:Rescheduling_interrupts
    308.00 ± 17%     -26.2%     227.25 ± 31%  interrupts.CPU87.RES:Rescheduling_interrupts





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


[-- Attachment #2: config-4.16.0-rc2-00007-g2c83362 --]
[-- Type: text/plain, Size: 165857 bytes --]

#
# Automatically generated file; DO NOT EDIT.
# Linux/x86_64 4.16.0-rc2 Kernel Configuration
#
CONFIG_64BIT=y
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_INSTRUCTION_DECODER=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_MMU=y
CONFIG_ARCH_MMAP_RND_BITS_MIN=28
CONFIG_ARCH_MMAP_RND_BITS_MAX=32
CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MIN=8
CONFIG_ARCH_MMAP_RND_COMPAT_BITS_MAX=16
CONFIG_NEED_DMA_MAP_STATE=y
CONFIG_NEED_SG_DMA_LENGTH=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y
CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y
CONFIG_ARCH_WANT_GENERAL_HUGETLB=y
CONFIG_ZONE_DMA32=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_HAVE_INTEL_TXT=y
CONFIG_X86_64_SMP=y
CONFIG_ARCH_SUPPORTS_UPROBES=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_PGTABLE_LEVELS=4
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_IRQ_WORK=y
CONFIG_BUILDTIME_EXTABLE_SORT=y
CONFIG_THREAD_INFO_IN_TASK=y

#
# General setup
#
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_CROSS_COMPILE=""
# CONFIG_COMPILE_TEST is not set
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_HAVE_KERNEL_XZ=y
CONFIG_HAVE_KERNEL_LZO=y
CONFIG_HAVE_KERNEL_LZ4=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
# CONFIG_KERNEL_XZ is not set
# CONFIG_KERNEL_LZO is not set
# CONFIG_KERNEL_LZ4 is not set
CONFIG_DEFAULT_HOSTNAME="(none)"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_CROSS_MEMORY_ATTACH=y
CONFIG_USELIB=y
CONFIG_AUDIT=y
CONFIG_HAVE_ARCH_AUDITSYSCALL=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_WATCH=y
CONFIG_AUDIT_TREE=y

#
# IRQ subsystem
#
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_IRQ_SHOW=y
CONFIG_GENERIC_IRQ_EFFECTIVE_AFF_MASK=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_GENERIC_IRQ_MIGRATION=y
CONFIG_IRQ_DOMAIN=y
CONFIG_IRQ_SIM=y
CONFIG_IRQ_DOMAIN_HIERARCHY=y
CONFIG_GENERIC_MSI_IRQ=y
CONFIG_GENERIC_MSI_IRQ_DOMAIN=y
CONFIG_GENERIC_IRQ_MATRIX_ALLOCATOR=y
CONFIG_GENERIC_IRQ_RESERVATION_MODE=y
CONFIG_IRQ_FORCED_THREADING=y
CONFIG_SPARSE_IRQ=y
# CONFIG_GENERIC_IRQ_DEBUGFS is not set
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_ARCH_CLOCKSOURCE_DATA=y
CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y
CONFIG_GENERIC_CMOS_UPDATE=y

#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
# CONFIG_NO_HZ_IDLE is not set
CONFIG_NO_HZ_FULL=y
# CONFIG_NO_HZ_FULL_ALL is not set
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y

#
# CPU/Task time and stats accounting
#
CONFIG_VIRT_CPU_ACCOUNTING=y
CONFIG_VIRT_CPU_ACCOUNTING_GEN=y
# CONFIG_IRQ_TIME_ACCOUNTING is not set
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_CPU_ISOLATION=y

#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_RCU_EXPERT is not set
CONFIG_SRCU=y
CONFIG_TREE_SRCU=y
CONFIG_TASKS_RCU=y
CONFIG_RCU_STALL_COMMON=y
CONFIG_RCU_NEED_SEGCBLIST=y
CONFIG_CONTEXT_TRACKING=y
# CONFIG_CONTEXT_TRACKING_FORCE is not set
CONFIG_RCU_NOCB_CPU=y
CONFIG_BUILD_BIN2C=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=20
CONFIG_LOG_CPU_MAX_BUF_SHIFT=12
CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT=13
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_ARCH_SUPPORTS_NUMA_BALANCING=y
CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH=y
CONFIG_ARCH_SUPPORTS_INT128=y
CONFIG_NUMA_BALANCING=y
CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
CONFIG_CGROUPS=y
CONFIG_PAGE_COUNTER=y
CONFIG_MEMCG=y
CONFIG_MEMCG_SWAP=y
CONFIG_MEMCG_SWAP_ENABLED=y
CONFIG_BLK_CGROUP=y
# CONFIG_DEBUG_BLK_CGROUP is not set
CONFIG_CGROUP_WRITEBACK=y
CONFIG_CGROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_CFS_BANDWIDTH=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_RDMA=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_DEVICE=y
# CONFIG_CGROUP_CPUACCT is not set
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_BPF=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_SOCK_CGROUP_DATA=y
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_SCHED_AUTOGROUP=y
# CONFIG_SYSFS_DEPRECATED is not set
CONFIG_RELAY=y
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_RD_GZIP=y
CONFIG_RD_BZIP2=y
CONFIG_RD_LZMA=y
CONFIG_RD_XZ=y
CONFIG_RD_LZO=y
CONFIG_RD_LZ4=y
CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_HAVE_UID16=y
CONFIG_SYSCTL_EXCEPTION_TRACE=y
CONFIG_HAVE_PCSPKR_PLATFORM=y
CONFIG_BPF=y
CONFIG_EXPERT=y
CONFIG_UID16=y
CONFIG_MULTIUSER=y
CONFIG_SGETMASK_SYSCALL=y
CONFIG_SYSFS_SYSCALL=y
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_FHANDLE=y
CONFIG_POSIX_TIMERS=y
CONFIG_PRINTK=y
CONFIG_PRINTK_NMI=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_FUTEX_PI=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_ADVISE_SYSCALLS=y
CONFIG_MEMBARRIER=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_ABSOLUTE_PERCPU=y
CONFIG_KALLSYMS_BASE_RELATIVE=y
CONFIG_BPF_SYSCALL=y
# CONFIG_BPF_JIT_ALWAYS_ON is not set
CONFIG_USERFAULTFD=y
CONFIG_ARCH_HAS_MEMBARRIER_SYNC_CORE=y
CONFIG_EMBEDDED=y
CONFIG_HAVE_PERF_EVENTS=y
# CONFIG_PC104 is not set

#
# Kernel Performance Events And Counters
#
CONFIG_PERF_EVENTS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_SLUB_MEMCG_SYSFS_ON is not set
# CONFIG_COMPAT_BRK is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
CONFIG_SLAB_MERGE_DEFAULT=y
# CONFIG_SLAB_FREELIST_RANDOM is not set
# CONFIG_SLAB_FREELIST_HARDENED is not set
CONFIG_SLUB_CPU_PARTIAL=y
CONFIG_SYSTEM_DATA_VERIFICATION=y
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
CONFIG_CRASH_CORE=y
CONFIG_KEXEC_CORE=y
CONFIG_OPROFILE=m
CONFIG_OPROFILE_EVENT_MULTIPLEX=y
CONFIG_HAVE_OPROFILE=y
CONFIG_OPROFILE_NMI_TIMER=y
CONFIG_KPROBES=y
CONFIG_JUMP_LABEL=y
# CONFIG_STATIC_KEYS_SELFTEST is not set
CONFIG_OPTPROBES=y
CONFIG_KPROBES_ON_FTRACE=y
CONFIG_UPROBES=y
# CONFIG_HAVE_64BIT_ALIGNED_ACCESS is not set
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_ARCH_USE_BUILTIN_BSWAP=y
CONFIG_KRETPROBES=y
CONFIG_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_OPTPROBES=y
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_HAVE_FUNCTION_ERROR_INJECTION=y
CONFIG_HAVE_NMI=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_CONTIGUOUS=y
CONFIG_GENERIC_SMP_IDLE_THREAD=y
CONFIG_ARCH_HAS_FORTIFY_SOURCE=y
CONFIG_ARCH_HAS_SET_MEMORY=y
CONFIG_HAVE_ARCH_THREAD_STRUCT_WHITELIST=y
CONFIG_ARCH_WANTS_DYNAMIC_TASK_STRUCT=y
CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
CONFIG_HAVE_CLK=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y
CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
CONFIG_HAVE_USER_RETURN_NOTIFIER=y
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_HARDLOCKUP_DETECTOR_PERF=y
CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_HAVE_ARCH_JUMP_LABEL=y
CONFIG_HAVE_RCU_TABLE_FREE=y
CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
CONFIG_HAVE_CMPXCHG_LOCAL=y
CONFIG_HAVE_CMPXCHG_DOUBLE=y
CONFIG_ARCH_WANT_COMPAT_IPC_PARSE_VERSION=y
CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP_FILTER=y
CONFIG_HAVE_GCC_PLUGINS=y
# CONFIG_GCC_PLUGINS is not set
CONFIG_HAVE_CC_STACKPROTECTOR=y
CONFIG_CC_STACKPROTECTOR_NONE=y
# CONFIG_CC_STACKPROTECTOR_REGULAR is not set
# CONFIG_CC_STACKPROTECTOR_STRONG is not set
# CONFIG_CC_STACKPROTECTOR_AUTO is not set
CONFIG_THIN_ARCHIVES=y
CONFIG_HAVE_ARCH_WITHIN_STACK_FRAMES=y
CONFIG_HAVE_CONTEXT_TRACKING=y
CONFIG_HAVE_VIRT_CPU_ACCOUNTING_GEN=y
CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD=y
CONFIG_HAVE_ARCH_HUGE_VMAP=y
CONFIG_HAVE_ARCH_SOFT_DIRTY=y
CONFIG_HAVE_MOD_ARCH_SPECIFIC=y
CONFIG_MODULES_USE_ELF_RELA=y
CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK=y
CONFIG_ARCH_HAS_ELF_RANDOMIZE=y
CONFIG_HAVE_ARCH_MMAP_RND_BITS=y
CONFIG_HAVE_EXIT_THREAD=y
CONFIG_ARCH_MMAP_RND_BITS=28
CONFIG_HAVE_ARCH_MMAP_RND_COMPAT_BITS=y
CONFIG_ARCH_MMAP_RND_COMPAT_BITS=8
CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES=y
CONFIG_HAVE_COPY_THREAD_TLS=y
CONFIG_HAVE_STACK_VALIDATION=y
# CONFIG_HAVE_ARCH_HASH is not set
# CONFIG_ISA_BUS_API is not set
CONFIG_OLD_SIGSUSPEND3=y
CONFIG_COMPAT_OLD_SIGACTION=y
# CONFIG_CPU_NO_EFFICIENT_FFS is not set
CONFIG_HAVE_ARCH_VMAP_STACK=y
CONFIG_VMAP_STACK=y
# CONFIG_ARCH_OPTIONAL_KERNEL_RWX is not set
# CONFIG_ARCH_OPTIONAL_KERNEL_RWX_DEFAULT is not set
CONFIG_ARCH_HAS_STRICT_KERNEL_RWX=y
CONFIG_STRICT_KERNEL_RWX=y
CONFIG_ARCH_HAS_STRICT_MODULE_RWX=y
CONFIG_STRICT_MODULE_RWX=y
CONFIG_ARCH_HAS_PHYS_TO_DMA=y
CONFIG_ARCH_HAS_REFCOUNT=y
# CONFIG_REFCOUNT_FULL is not set

#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
# CONFIG_MODULE_SIG is not set
# CONFIG_MODULE_COMPRESS is not set
# CONFIG_TRIM_UNUSED_KSYMS is not set
CONFIG_MODULES_TREE_LOOKUP=y
CONFIG_BLOCK=y
CONFIG_BLK_SCSI_REQUEST=y
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_BSGLIB=y
CONFIG_BLK_DEV_INTEGRITY=y
CONFIG_BLK_DEV_ZONED=y
CONFIG_BLK_DEV_THROTTLING=y
# CONFIG_BLK_DEV_THROTTLING_LOW is not set
# CONFIG_BLK_CMDLINE_PARSER is not set
# CONFIG_BLK_WBT is not set
CONFIG_BLK_DEBUG_FS=y
# CONFIG_BLK_SED_OPAL is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_AIX_PARTITION is not set
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
# CONFIG_ATARI_PARTITION is not set
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
# CONFIG_LDM_PARTITION is not set
CONFIG_SGI_PARTITION=y
# CONFIG_ULTRIX_PARTITION is not set
CONFIG_SUN_PARTITION=y
CONFIG_KARMA_PARTITION=y
CONFIG_EFI_PARTITION=y
# CONFIG_SYSV68_PARTITION is not set
# CONFIG_CMDLINE_PARTITION is not set
CONFIG_BLOCK_COMPAT=y
CONFIG_BLK_MQ_PCI=y
CONFIG_BLK_MQ_VIRTIO=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_CFQ_GROUP_IOSCHED=y
CONFIG_DEFAULT_DEADLINE=y
# CONFIG_DEFAULT_CFQ is not set
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="deadline"
CONFIG_MQ_IOSCHED_DEADLINE=y
CONFIG_MQ_IOSCHED_KYBER=y
# CONFIG_IOSCHED_BFQ is not set
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_PADATA=y
CONFIG_ASN1=y
CONFIG_INLINE_SPIN_UNLOCK_IRQ=y
CONFIG_INLINE_READ_UNLOCK=y
CONFIG_INLINE_READ_UNLOCK_IRQ=y
CONFIG_INLINE_WRITE_UNLOCK=y
CONFIG_INLINE_WRITE_UNLOCK_IRQ=y
CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y
CONFIG_MUTEX_SPIN_ON_OWNER=y
CONFIG_RWSEM_SPIN_ON_OWNER=y
CONFIG_LOCK_SPIN_ON_OWNER=y
CONFIG_ARCH_USE_QUEUED_SPINLOCKS=y
CONFIG_QUEUED_SPINLOCKS=y
CONFIG_ARCH_USE_QUEUED_RWLOCKS=y
CONFIG_QUEUED_RWLOCKS=y
CONFIG_ARCH_HAS_SYNC_CORE_BEFORE_USERMODE=y
CONFIG_FREEZER=y

#
# Processor type and features
#
CONFIG_ZONE_DMA=y
CONFIG_SMP=y
CONFIG_X86_FEATURE_NAMES=y
CONFIG_X86_FAST_FEATURE_TESTS=y
CONFIG_X86_X2APIC=y
CONFIG_X86_MPPARSE=y
# CONFIG_GOLDFISH is not set
CONFIG_RETPOLINE=y
CONFIG_INTEL_RDT=y
CONFIG_X86_EXTENDED_PLATFORM=y
# CONFIG_X86_NUMACHIP is not set
# CONFIG_X86_VSMP is not set
CONFIG_X86_UV=y
# CONFIG_X86_GOLDFISH is not set
# CONFIG_X86_INTEL_MID is not set
CONFIG_X86_INTEL_LPSS=y
# CONFIG_X86_AMD_PLATFORM_DEVICE is not set
CONFIG_IOSF_MBI=y
# CONFIG_IOSF_MBI_DEBUG is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
# CONFIG_SCHED_OMIT_FRAME_POINTER is not set
CONFIG_HYPERVISOR_GUEST=y
CONFIG_PARAVIRT=y
# CONFIG_PARAVIRT_DEBUG is not set
CONFIG_PARAVIRT_SPINLOCKS=y
# CONFIG_QUEUED_LOCK_STAT is not set
CONFIG_XEN=y
CONFIG_XEN_PV=y
CONFIG_XEN_PV_SMP=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_PVHVM_SMP=y
CONFIG_XEN_512GB=y
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
# CONFIG_XEN_PVH is not set
CONFIG_KVM_GUEST=y
# CONFIG_KVM_DEBUG_FS is not set
CONFIG_PARAVIRT_TIME_ACCOUNTING=y
CONFIG_PARAVIRT_CLOCK=y
# CONFIG_JAILHOUSE_GUEST is not set
CONFIG_NO_BOOTMEM=y
# CONFIG_MK8 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
CONFIG_GENERIC_CPU=y
CONFIG_X86_INTERNODE_CACHE_SHIFT=6
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
# CONFIG_PROCESSOR_SELECT is not set
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
# CONFIG_CALGARY_IOMMU is not set
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
CONFIG_MAXSMP=y
CONFIG_NR_CPUS_RANGE_BEGIN=8192
CONFIG_NR_CPUS_RANGE_END=8192
CONFIG_NR_CPUS_DEFAULT=8192
CONFIG_NR_CPUS=8192
CONFIG_SCHED_SMT=y
CONFIG_SCHED_MC=y
CONFIG_SCHED_MC_PRIO=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_PREEMPT_COUNT=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_MCELOG_LEGACY=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
CONFIG_X86_MCE_INJECT=m
CONFIG_X86_THERMAL_VECTOR=y

#
# Performance monitoring
#
CONFIG_PERF_EVENTS_INTEL_UNCORE=y
CONFIG_PERF_EVENTS_INTEL_RAPL=y
CONFIG_PERF_EVENTS_INTEL_CSTATE=y
# CONFIG_PERF_EVENTS_AMD_POWER is not set
# CONFIG_VM86 is not set
CONFIG_X86_16BIT=y
CONFIG_X86_ESPFIX64=y
CONFIG_X86_VSYSCALL_EMULATION=y
CONFIG_I8K=m
CONFIG_MICROCODE=y
CONFIG_MICROCODE_INTEL=y
CONFIG_MICROCODE_AMD=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
# CONFIG_X86_5LEVEL is not set
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_ARCH_DMA_ADDR_T_64BIT=y
CONFIG_X86_DIRECT_GBPAGES=y
CONFIG_ARCH_HAS_MEM_ENCRYPT=y
# CONFIG_AMD_MEM_ENCRYPT is not set
CONFIG_NUMA=y
CONFIG_AMD_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NODES_SPAN_OTHER_NODES=y
# CONFIG_NUMA_EMU is not set
CONFIG_NODES_SHIFT=10
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_MEMORY_PROBE=y
CONFIG_ARCH_PROC_KCORE_TEXT=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_NEED_MULTIPLE_NODES=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y
CONFIG_SPARSEMEM_VMEMMAP=y
CONFIG_HAVE_MEMBLOCK=y
CONFIG_HAVE_MEMBLOCK_NODE_MAP=y
CONFIG_HAVE_GENERIC_GUP=y
CONFIG_ARCH_DISCARD_MEMBLOCK=y
CONFIG_MEMORY_ISOLATION=y
CONFIG_HAVE_BOOTMEM_INFO_NODE=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTPLUG_SPARSE=y
# CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is not set
CONFIG_MEMORY_HOTREMOVE=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_ARCH_ENABLE_SPLIT_PMD_PTLOCK=y
CONFIG_MEMORY_BALLOON=y
CONFIG_BALLOON_COMPACTION=y
CONFIG_COMPACTION=y
CONFIG_MIGRATION=y
CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION=y
CONFIG_ARCH_ENABLE_THP_MIGRATION=y
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MMU_NOTIFIER=y
CONFIG_KSM=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y
CONFIG_MEMORY_FAILURE=y
CONFIG_HWPOISON_INJECT=m
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
# CONFIG_TRANSPARENT_HUGEPAGE_MADVISE is not set
CONFIG_ARCH_WANTS_THP_SWAP=y
CONFIG_THP_SWAP=y
CONFIG_TRANSPARENT_HUGE_PAGECACHE=y
CONFIG_CLEANCACHE=y
CONFIG_FRONTSWAP=y
CONFIG_CMA=y
# CONFIG_CMA_DEBUG is not set
# CONFIG_CMA_DEBUGFS is not set
CONFIG_CMA_AREAS=7
# CONFIG_MEM_SOFT_DIRTY is not set
CONFIG_ZSWAP=y
CONFIG_ZPOOL=y
CONFIG_ZBUD=y
# CONFIG_Z3FOLD is not set
CONFIG_ZSMALLOC=y
# CONFIG_PGTABLE_MAPPING is not set
# CONFIG_ZSMALLOC_STAT is not set
CONFIG_GENERIC_EARLY_IOREMAP=y
# CONFIG_DEFERRED_STRUCT_PAGE_INIT is not set
CONFIG_IDLE_PAGE_TRACKING=y
CONFIG_ARCH_HAS_ZONE_DEVICE=y
CONFIG_ZONE_DEVICE=y
CONFIG_ARCH_HAS_HMM=y
# CONFIG_HMM_MIRROR is not set
# CONFIG_DEVICE_PRIVATE is not set
# CONFIG_DEVICE_PUBLIC is not set
CONFIG_FRAME_VECTOR=y
CONFIG_ARCH_USES_HIGH_VMA_FLAGS=y
CONFIG_ARCH_HAS_PKEYS=y
# CONFIG_PERCPU_STATS is not set
# CONFIG_GUP_BENCHMARK is not set
CONFIG_X86_PMEM_LEGACY_DEVICE=y
CONFIG_X86_PMEM_LEGACY=m
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
# CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK is not set
CONFIG_X86_RESERVE_LOW=64
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
CONFIG_ARCH_RANDOM=y
CONFIG_X86_SMAP=y
CONFIG_X86_INTEL_UMIP=y
CONFIG_X86_INTEL_MPX=y
CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS=y
CONFIG_EFI=y
CONFIG_EFI_STUB=y
# CONFIG_EFI_MIXED is not set
CONFIG_SECCOMP=y
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
# CONFIG_KEXEC_FILE is not set
CONFIG_CRASH_DUMP=y
CONFIG_KEXEC_JUMP=y
CONFIG_PHYSICAL_START=0x1000000
CONFIG_RELOCATABLE=y
# CONFIG_RANDOMIZE_BASE is not set
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
CONFIG_BOOTPARAM_HOTPLUG_CPU0=y
# CONFIG_DEBUG_HOTPLUG_CPU0 is not set
# CONFIG_COMPAT_VDSO is not set
# CONFIG_LEGACY_VSYSCALL_NATIVE is not set
CONFIG_LEGACY_VSYSCALL_EMULATE=y
# CONFIG_LEGACY_VSYSCALL_NONE is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_MODIFY_LDT_SYSCALL=y
CONFIG_HAVE_LIVEPATCH=y
# CONFIG_LIVEPATCH is not set
CONFIG_ARCH_HAS_ADD_PAGES=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
CONFIG_USE_PERCPU_NUMA_NODE_ID=y

#
# Power management and ACPI options
#
CONFIG_ARCH_HIBERNATION_HEADER=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
# CONFIG_SUSPEND_SKIP_SYNC is not set
CONFIG_HIBERNATE_CALLBACKS=y
CONFIG_HIBERNATION=y
CONFIG_PM_STD_PARTITION=""
CONFIG_PM_SLEEP=y
CONFIG_PM_SLEEP_SMP=y
# CONFIG_PM_AUTOSLEEP is not set
# CONFIG_PM_WAKELOCKS is not set
CONFIG_PM=y
CONFIG_PM_DEBUG=y
CONFIG_PM_ADVANCED_DEBUG=y
CONFIG_PM_TEST_SUSPEND=y
CONFIG_PM_SLEEP_DEBUG=y
# CONFIG_DPM_WATCHDOG is not set
# CONFIG_PM_TRACE_RTC is not set
CONFIG_PM_CLK=y
# CONFIG_WQ_POWER_EFFICIENT_DEFAULT is not set
CONFIG_ACPI=y
CONFIG_ACPI_LEGACY_TABLES_LOOKUP=y
CONFIG_ARCH_MIGHT_HAVE_ACPI_PDC=y
CONFIG_ACPI_SYSTEM_POWER_STATES_SUPPORT=y
# CONFIG_ACPI_DEBUGGER is not set
CONFIG_ACPI_SPCR_TABLE=y
CONFIG_ACPI_LPIT=y
CONFIG_ACPI_SLEEP=y
# CONFIG_ACPI_PROCFS_POWER is not set
CONFIG_ACPI_REV_OVERRIDE_POSSIBLE=y
CONFIG_ACPI_EC_DEBUGFS=m
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_CPU_FREQ_PSS=y
CONFIG_ACPI_PROCESSOR_CSTATE=y
CONFIG_ACPI_PROCESSOR_IDLE=y
CONFIG_ACPI_CPPC_LIB=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_IPMI=m
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_PROCESSOR_AGGREGATOR=m
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_NUMA=y
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ARCH_HAS_ACPI_TABLE_UPGRADE=y
CONFIG_ACPI_TABLE_UPGRADE=y
CONFIG_ACPI_DEBUG=y
CONFIG_ACPI_PCI_SLOT=y
CONFIG_ACPI_CONTAINER=y
CONFIG_ACPI_HOTPLUG_MEMORY=y
CONFIG_ACPI_HOTPLUG_IOAPIC=y
CONFIG_ACPI_SBS=m
CONFIG_ACPI_HED=y
CONFIG_ACPI_CUSTOM_METHOD=m
CONFIG_ACPI_BGRT=y
# CONFIG_ACPI_REDUCED_HARDWARE_ONLY is not set
CONFIG_ACPI_NFIT=m
CONFIG_HAVE_ACPI_APEI=y
CONFIG_HAVE_ACPI_APEI_NMI=y
CONFIG_ACPI_APEI=y
CONFIG_ACPI_APEI_GHES=y
CONFIG_ACPI_APEI_PCIEAER=y
CONFIG_ACPI_APEI_MEMORY_FAILURE=y
CONFIG_ACPI_APEI_EINJ=m
CONFIG_ACPI_APEI_ERST_DEBUG=y
# CONFIG_DPTF_POWER is not set
CONFIG_ACPI_WATCHDOG=y
CONFIG_ACPI_EXTLOG=m
# CONFIG_PMIC_OPREGION is not set
# CONFIG_ACPI_CONFIGFS is not set
CONFIG_X86_PM_TIMER=y
CONFIG_SFI=y

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_GOV_ATTR_SET=y
CONFIG_CPU_FREQ_GOV_COMMON=y
# CONFIG_CPU_FREQ_STAT is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_SCHEDUTIL is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
# CONFIG_CPU_FREQ_GOV_SCHEDUTIL is not set

#
# CPU frequency scaling drivers
#
CONFIG_X86_INTEL_PSTATE=y
CONFIG_X86_PCC_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ_CPB=y
CONFIG_X86_POWERNOW_K8=m
CONFIG_X86_AMD_FREQ_SENSITIVITY=m
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
CONFIG_X86_P4_CLOCKMOD=m

#
# shared options
#
CONFIG_X86_SPEEDSTEP_LIB=m

#
# CPU Idle
#
CONFIG_CPU_IDLE=y
# CONFIG_CPU_IDLE_GOV_LADDER is not set
CONFIG_CPU_IDLE_GOV_MENU=y
# CONFIG_ARCH_NEEDS_CPU_IDLE_COUPLED is not set
CONFIG_INTEL_IDLE=y

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_XEN=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCI_CNB20LE_QUIRK is not set
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=y
CONFIG_PCIEAER=y
CONFIG_PCIE_ECRC=y
CONFIG_PCIEAER_INJECT=m
CONFIG_PCIEASPM=y
# CONFIG_PCIEASPM_DEBUG is not set
CONFIG_PCIEASPM_DEFAULT=y
# CONFIG_PCIEASPM_POWERSAVE is not set
# CONFIG_PCIEASPM_POWER_SUPERSAVE is not set
# CONFIG_PCIEASPM_PERFORMANCE is not set
CONFIG_PCIE_PME=y
# CONFIG_PCIE_DPC is not set
# CONFIG_PCIE_PTM is not set
CONFIG_PCI_BUS_ADDR_T_64BIT=y
CONFIG_PCI_MSI=y
CONFIG_PCI_MSI_IRQ_DOMAIN=y
CONFIG_PCI_QUIRKS=y
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_REALLOC_ENABLE_AUTO is not set
CONFIG_PCI_STUB=y
# CONFIG_XEN_PCIDEV_FRONTEND is not set
CONFIG_PCI_ATS=y
CONFIG_PCI_LOCKLESS_CONFIG=y
CONFIG_PCI_IOV=y
CONFIG_PCI_PRI=y
CONFIG_PCI_PASID=y
CONFIG_PCI_LABEL=y
# CONFIG_PCI_HYPERV is not set
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_ACPI=y
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
# CONFIG_HOTPLUG_PCI_CPCI is not set
CONFIG_HOTPLUG_PCI_SHPC=m

#
# Cadence PCIe controllers support
#

#
# DesignWare PCI Core Support
#
# CONFIG_PCIE_DW_PLAT is not set

#
# PCI host controller drivers
#
# CONFIG_VMD is not set

#
# PCI Endpoint
#
# CONFIG_PCI_ENDPOINT is not set

#
# PCI switch controller drivers
#
# CONFIG_PCI_SW_SWITCHTEC is not set
# CONFIG_ISA_BUS is not set
CONFIG_ISA_DMA_API=y
CONFIG_AMD_NB=y
CONFIG_PCCARD=y
# CONFIG_PCMCIA is not set
CONFIG_CARDBUS=y

#
# PC-card bridges
#
CONFIG_YENTA=m
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
# CONFIG_RAPIDIO is not set
# CONFIG_X86_SYSFB is not set

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_ELFCORE=y
CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y
CONFIG_BINFMT_SCRIPT=y
# CONFIG_HAVE_AOUT is not set
CONFIG_BINFMT_MISC=m
CONFIG_COREDUMP=y
CONFIG_IA32_EMULATION=y
# CONFIG_IA32_AOUT is not set
# CONFIG_X86_X32 is not set
CONFIG_COMPAT_32=y
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_X86_DEV_DMA_OPS=y
CONFIG_NET=y
CONFIG_COMPAT_NETLINK_MESSAGES=y
CONFIG_NET_INGRESS=y
CONFIG_NET_EGRESS=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_DIAG=m
CONFIG_UNIX=y
CONFIG_UNIX_DIAG=m
# CONFIG_TLS is not set
CONFIG_XFRM=y
CONFIG_XFRM_ALGO=y
CONFIG_XFRM_USER=y
CONFIG_XFRM_SUB_POLICY=y
CONFIG_XFRM_MIGRATE=y
CONFIG_XFRM_STATISTICS=y
CONFIG_XFRM_IPCOMP=m
CONFIG_NET_KEY=m
CONFIG_NET_KEY_MIGRATE=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_FIB_TRIE_STATS=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_ROUTE_CLASSID=y
CONFIG_IP_PNP=y
CONFIG_IP_PNP_DHCP=y
# CONFIG_IP_PNP_BOOTP is not set
# CONFIG_IP_PNP_RARP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE_DEMUX=m
CONFIG_NET_IP_TUNNEL=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_MROUTE_MULTIPLE_TABLES=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
CONFIG_SYN_COOKIES=y
CONFIG_NET_IPVTI=m
CONFIG_NET_UDP_TUNNEL=m
CONFIG_NET_FOU=m
CONFIG_NET_FOU_IP_TUNNELS=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
# CONFIG_INET_ESP_OFFLOAD is not set
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
CONFIG_INET_UDP_DIAG=m
# CONFIG_INET_RAW_DIAG is not set
# CONFIG_INET_DIAG_DESTROY is not set
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_TCP_CONG_HSTCP=m
CONFIG_TCP_CONG_HYBLA=m
CONFIG_TCP_CONG_VEGAS=m
# CONFIG_TCP_CONG_NV is not set
CONFIG_TCP_CONG_SCALABLE=m
CONFIG_TCP_CONG_LP=m
CONFIG_TCP_CONG_VENO=m
CONFIG_TCP_CONG_YEAH=m
CONFIG_TCP_CONG_ILLINOIS=m
# CONFIG_TCP_CONG_DCTCP is not set
# CONFIG_TCP_CONG_CDG is not set
# CONFIG_TCP_CONG_BBR is not set
CONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_TCP_MD5SIG=y
CONFIG_IPV6=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
CONFIG_IPV6_OPTIMISTIC_DAD=y
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
# CONFIG_INET6_ESP_OFFLOAD is not set
CONFIG_INET6_IPCOMP=m
CONFIG_IPV6_MIP6=m
# CONFIG_IPV6_ILA is not set
CONFIG_INET6_XFRM_TUNNEL=m
CONFIG_INET6_TUNNEL=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=m
CONFIG_INET6_XFRM_MODE_TUNNEL=m
CONFIG_INET6_XFRM_MODE_BEET=m
CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=m
CONFIG_IPV6_VTI=m
CONFIG_IPV6_SIT=m
CONFIG_IPV6_SIT_6RD=y
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
# CONFIG_IPV6_GRE is not set
CONFIG_IPV6_FOU=m
CONFIG_IPV6_FOU_TUNNEL=m
CONFIG_IPV6_MULTIPLE_TABLES=y
# CONFIG_IPV6_SUBTREES is not set
CONFIG_IPV6_MROUTE=y
CONFIG_IPV6_MROUTE_MULTIPLE_TABLES=y
CONFIG_IPV6_PIMSM_V2=y
CONFIG_IPV6_SEG6_LWTUNNEL=y
# CONFIG_IPV6_SEG6_HMAC is not set
CONFIG_NETLABEL=y
CONFIG_NETWORK_SECMARK=y
CONFIG_NET_PTP_CLASSIFY=y
CONFIG_NETWORK_PHY_TIMESTAMPING=y
CONFIG_NETFILTER=y
CONFIG_NETFILTER_ADVANCED=y
CONFIG_BRIDGE_NETFILTER=m

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_INGRESS=y
CONFIG_NETFILTER_NETLINK=m
CONFIG_NETFILTER_FAMILY_BRIDGE=y
CONFIG_NETFILTER_FAMILY_ARP=y
CONFIG_NETFILTER_NETLINK_ACCT=m
CONFIG_NETFILTER_NETLINK_QUEUE=m
CONFIG_NETFILTER_NETLINK_LOG=m
CONFIG_NF_CONNTRACK=m
CONFIG_NF_LOG_COMMON=m
# CONFIG_NF_LOG_NETDEV is not set
CONFIG_NETFILTER_CONNCOUNT=m
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_ZONES=y
CONFIG_NF_CONNTRACK_PROCFS=y
CONFIG_NF_CONNTRACK_EVENTS=y
# CONFIG_NF_CONNTRACK_TIMEOUT is not set
CONFIG_NF_CONNTRACK_TIMESTAMP=y
CONFIG_NF_CONNTRACK_LABELS=y
CONFIG_NF_CT_PROTO_DCCP=y
CONFIG_NF_CT_PROTO_GRE=m
CONFIG_NF_CT_PROTO_SCTP=y
CONFIG_NF_CT_PROTO_UDPLITE=y
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_BROADCAST=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_SNMP=m
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
# CONFIG_NF_CT_NETLINK_TIMEOUT is not set
# CONFIG_NETFILTER_NETLINK_GLUE_CT is not set
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_NF_NAT_PROTO_DCCP=y
CONFIG_NF_NAT_PROTO_UDPLITE=y
CONFIG_NF_NAT_PROTO_SCTP=y
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_SIP=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_REDIRECT=m
CONFIG_NETFILTER_SYNPROXY=m
CONFIG_NF_TABLES=m
# CONFIG_NF_TABLES_INET is not set
# CONFIG_NF_TABLES_NETDEV is not set
CONFIG_NFT_EXTHDR=m
CONFIG_NFT_META=m
# CONFIG_NFT_RT is not set
# CONFIG_NFT_NUMGEN is not set
CONFIG_NFT_CT=m
# CONFIG_NFT_SET_RBTREE is not set
# CONFIG_NFT_SET_HASH is not set
# CONFIG_NFT_SET_BITMAP is not set
CONFIG_NFT_COUNTER=m
CONFIG_NFT_LOG=m
CONFIG_NFT_LIMIT=m
# CONFIG_NFT_MASQ is not set
# CONFIG_NFT_REDIR is not set
CONFIG_NFT_NAT=m
# CONFIG_NFT_OBJREF is not set
# CONFIG_NFT_QUEUE is not set
# CONFIG_NFT_QUOTA is not set
# CONFIG_NFT_REJECT is not set
CONFIG_NFT_COMPAT=m
CONFIG_NFT_HASH=m
# CONFIG_NF_FLOW_TABLE is not set
CONFIG_NETFILTER_XTABLES=y

#
# Xtables combined modules
#
CONFIG_NETFILTER_XT_MARK=m
CONFIG_NETFILTER_XT_CONNMARK=m
CONFIG_NETFILTER_XT_SET=m

#
# Xtables targets
#
CONFIG_NETFILTER_XT_TARGET_AUDIT=m
CONFIG_NETFILTER_XT_TARGET_CHECKSUM=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m
CONFIG_NETFILTER_XT_TARGET_CT=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_HL=m
CONFIG_NETFILTER_XT_TARGET_HMARK=m
CONFIG_NETFILTER_XT_TARGET_IDLETIMER=m
CONFIG_NETFILTER_XT_TARGET_LED=m
CONFIG_NETFILTER_XT_TARGET_LOG=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_NAT=m
CONFIG_NETFILTER_XT_TARGET_NETMAP=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
CONFIG_NETFILTER_XT_TARGET_RATEEST=m
CONFIG_NETFILTER_XT_TARGET_REDIRECT=m
CONFIG_NETFILTER_XT_TARGET_TEE=m
CONFIG_NETFILTER_XT_TARGET_TPROXY=m
CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NETFILTER_XT_TARGET_SECMARK=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m

#
# Xtables matches
#
CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=m
CONFIG_NETFILTER_XT_MATCH_BPF=m
# CONFIG_NETFILTER_XT_MATCH_CGROUP is not set
CONFIG_NETFILTER_XT_MATCH_CLUSTER=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLABEL=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_CPU=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DEVGROUP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ECN=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_HL=m
# CONFIG_NETFILTER_XT_MATCH_IPCOMP is not set
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_IPVS=m
CONFIG_NETFILTER_XT_MATCH_L2TP=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_NFACCT=m
CONFIG_NETFILTER_XT_MATCH_OSF=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_RATEEST=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_RECENT=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NETFILTER_XT_MATCH_U32=m
CONFIG_IP_SET=m
CONFIG_IP_SET_MAX=256
CONFIG_IP_SET_BITMAP_IP=m
CONFIG_IP_SET_BITMAP_IPMAC=m
CONFIG_IP_SET_BITMAP_PORT=m
CONFIG_IP_SET_HASH_IP=m
# CONFIG_IP_SET_HASH_IPMARK is not set
CONFIG_IP_SET_HASH_IPPORT=m
CONFIG_IP_SET_HASH_IPPORTIP=m
CONFIG_IP_SET_HASH_IPPORTNET=m
# CONFIG_IP_SET_HASH_IPMAC is not set
# CONFIG_IP_SET_HASH_MAC is not set
# CONFIG_IP_SET_HASH_NETPORTNET is not set
CONFIG_IP_SET_HASH_NET=m
# CONFIG_IP_SET_HASH_NETNET is not set
CONFIG_IP_SET_HASH_NETPORT=m
CONFIG_IP_SET_HASH_NETIFACE=m
CONFIG_IP_SET_LIST_SET=m
CONFIG_IP_VS=m
CONFIG_IP_VS_IPV6=y
# CONFIG_IP_VS_DEBUG is not set
CONFIG_IP_VS_TAB_BITS=12

#
# IPVS transport protocol load balancing support
#
CONFIG_IP_VS_PROTO_TCP=y
CONFIG_IP_VS_PROTO_UDP=y
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y
CONFIG_IP_VS_PROTO_AH=y
CONFIG_IP_VS_PROTO_SCTP=y

#
# IPVS scheduler
#
CONFIG_IP_VS_RR=m
CONFIG_IP_VS_WRR=m
CONFIG_IP_VS_LC=m
CONFIG_IP_VS_WLC=m
# CONFIG_IP_VS_FO is not set
# CONFIG_IP_VS_OVF is not set
CONFIG_IP_VS_LBLC=m
CONFIG_IP_VS_LBLCR=m
CONFIG_IP_VS_DH=m
CONFIG_IP_VS_SH=m
CONFIG_IP_VS_SED=m
CONFIG_IP_VS_NQ=m

#
# IPVS SH scheduler
#
CONFIG_IP_VS_SH_TAB_BITS=8

#
# IPVS application helper
#
CONFIG_IP_VS_FTP=m
CONFIG_IP_VS_NFCT=y
CONFIG_IP_VS_PE_SIP=m

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=m
CONFIG_NF_CONNTRACK_IPV4=m
# CONFIG_NF_SOCKET_IPV4 is not set
# CONFIG_NF_TABLES_IPV4 is not set
# CONFIG_NF_TABLES_ARP is not set
CONFIG_NF_DUP_IPV4=m
# CONFIG_NF_LOG_ARP is not set
CONFIG_NF_LOG_IPV4=m
CONFIG_NF_REJECT_IPV4=m
CONFIG_NF_NAT_IPV4=m
CONFIG_NF_NAT_MASQUERADE_IPV4=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_RPFILTER=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_SYNPROXY=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_SECURITY=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m

#
# IPv6: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV6=m
CONFIG_NF_CONNTRACK_IPV6=m
# CONFIG_NF_SOCKET_IPV6 is not set
# CONFIG_NF_TABLES_IPV6 is not set
CONFIG_NF_DUP_IPV6=m
CONFIG_NF_REJECT_IPV6=m
CONFIG_NF_LOG_IPV6=m
CONFIG_NF_NAT_IPV6=m
# CONFIG_NF_NAT_MASQUERADE_IPV6 is not set
CONFIG_IP6_NF_IPTABLES=m
CONFIG_IP6_NF_MATCH_AH=m
CONFIG_IP6_NF_MATCH_EUI64=m
CONFIG_IP6_NF_MATCH_FRAG=m
CONFIG_IP6_NF_MATCH_OPTS=m
CONFIG_IP6_NF_MATCH_HL=m
CONFIG_IP6_NF_MATCH_IPV6HEADER=m
CONFIG_IP6_NF_MATCH_MH=m
CONFIG_IP6_NF_MATCH_RPFILTER=m
CONFIG_IP6_NF_MATCH_RT=m
# CONFIG_IP6_NF_MATCH_SRH is not set
CONFIG_IP6_NF_TARGET_HL=m
CONFIG_IP6_NF_FILTER=m
CONFIG_IP6_NF_TARGET_REJECT=m
CONFIG_IP6_NF_TARGET_SYNPROXY=m
CONFIG_IP6_NF_MANGLE=m
CONFIG_IP6_NF_RAW=m
CONFIG_IP6_NF_SECURITY=m
# CONFIG_IP6_NF_NAT is not set
# CONFIG_NF_TABLES_BRIDGE is not set
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_IP6=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_NFLOG=m
CONFIG_IP_DCCP=m
CONFIG_INET_DCCP_DIAG=m

#
# DCCP CCIDs Configuration
#
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
CONFIG_IP_DCCP_CCID3=y
# CONFIG_IP_DCCP_CCID3_DEBUG is not set
CONFIG_IP_DCCP_TFRC_LIB=y

#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
CONFIG_IP_SCTP=m
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_DEFAULT_COOKIE_HMAC_MD5 is not set
CONFIG_SCTP_DEFAULT_COOKIE_HMAC_SHA1=y
# CONFIG_SCTP_DEFAULT_COOKIE_HMAC_NONE is not set
CONFIG_SCTP_COOKIE_HMAC_MD5=y
CONFIG_SCTP_COOKIE_HMAC_SHA1=y
CONFIG_INET_SCTP_DIAG=m
# CONFIG_RDS is not set
CONFIG_TIPC=m
CONFIG_TIPC_MEDIA_UDP=y
CONFIG_ATM=m
CONFIG_ATM_CLIP=m
# CONFIG_ATM_CLIP_NO_ICMP is not set
CONFIG_ATM_LANE=m
# CONFIG_ATM_MPOA is not set
CONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set
CONFIG_L2TP=m
CONFIG_L2TP_DEBUGFS=m
CONFIG_L2TP_V3=y
CONFIG_L2TP_IP=m
CONFIG_L2TP_ETH=m
CONFIG_STP=m
CONFIG_GARP=m
CONFIG_MRP=m
CONFIG_BRIDGE=m
CONFIG_BRIDGE_IGMP_SNOOPING=y
CONFIG_BRIDGE_VLAN_FILTERING=y
CONFIG_HAVE_NET_DSA=y
# CONFIG_NET_DSA is not set
CONFIG_VLAN_8021Q=m
CONFIG_VLAN_8021Q_GVRP=y
CONFIG_VLAN_8021Q_MVRP=y
# CONFIG_DECNET is not set
CONFIG_LLC=m
# CONFIG_LLC2 is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_PHONET is not set
# CONFIG_6LOWPAN is not set
CONFIG_IEEE802154=m
# CONFIG_IEEE802154_NL802154_EXPERIMENTAL is not set
CONFIG_IEEE802154_SOCKET=m
CONFIG_MAC802154=m
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_ATM=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_MULTIQ=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFB=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
# CONFIG_NET_SCH_CBS is not set
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_DRR=m
CONFIG_NET_SCH_MQPRIO=m
CONFIG_NET_SCH_CHOKE=m
CONFIG_NET_SCH_QFQ=m
CONFIG_NET_SCH_CODEL=m
CONFIG_NET_SCH_FQ_CODEL=m
# CONFIG_NET_SCH_FQ is not set
# CONFIG_NET_SCH_HHF is not set
# CONFIG_NET_SCH_PIE is not set
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_SCH_PLUG=m
# CONFIG_NET_SCH_DEFAULT is not set

#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_CLS_FLOW=m
CONFIG_NET_CLS_CGROUP=y
CONFIG_NET_CLS_BPF=m
# CONFIG_NET_CLS_FLOWER is not set
# CONFIG_NET_CLS_MATCHALL is not set
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
# CONFIG_NET_EMATCH_CANID is not set
CONFIG_NET_EMATCH_IPSET=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
# CONFIG_NET_ACT_SAMPLE is not set
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_NAT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_ACT_SKBEDIT=m
CONFIG_NET_ACT_CSUM=m
# CONFIG_NET_ACT_VLAN is not set
# CONFIG_NET_ACT_BPF is not set
# CONFIG_NET_ACT_CONNMARK is not set
# CONFIG_NET_ACT_SKBMOD is not set
# CONFIG_NET_ACT_IFE is not set
# CONFIG_NET_ACT_TUNNEL_KEY is not set
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y
CONFIG_DCB=y
CONFIG_DNS_RESOLVER=m
# CONFIG_BATMAN_ADV is not set
CONFIG_OPENVSWITCH=m
CONFIG_OPENVSWITCH_GRE=m
CONFIG_OPENVSWITCH_VXLAN=m
CONFIG_VSOCKETS=m
CONFIG_VSOCKETS_DIAG=m
CONFIG_VMWARE_VMCI_VSOCKETS=m
# CONFIG_VIRTIO_VSOCKETS is not set
# CONFIG_HYPERV_VSOCKETS is not set
CONFIG_NETLINK_DIAG=m
CONFIG_MPLS=y
CONFIG_NET_MPLS_GSO=m
# CONFIG_MPLS_ROUTING is not set
CONFIG_NET_NSH=m
# CONFIG_HSR is not set
# CONFIG_NET_SWITCHDEV is not set
CONFIG_NET_L3_MASTER_DEV=y
# CONFIG_NET_NCSI is not set
CONFIG_RPS=y
CONFIG_RFS_ACCEL=y
CONFIG_XPS=y
# CONFIG_CGROUP_NET_PRIO is not set
CONFIG_CGROUP_NET_CLASSID=y
CONFIG_NET_RX_BUSY_POLL=y
CONFIG_BQL=y
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_NET_FLOW_LIMIT=y

#
# Network testing
#
CONFIG_NET_PKTGEN=m
CONFIG_NET_DROP_MONITOR=y
# CONFIG_HAMRADIO is not set
CONFIG_CAN=m
CONFIG_CAN_RAW=m
CONFIG_CAN_BCM=m
CONFIG_CAN_GW=m

#
# CAN Device Drivers
#
CONFIG_CAN_VCAN=m
# CONFIG_CAN_VXCAN is not set
# CONFIG_CAN_SLCAN is not set
CONFIG_CAN_DEV=m
CONFIG_CAN_CALC_BITTIMING=y
# CONFIG_CAN_LEDS is not set
# CONFIG_CAN_C_CAN is not set
# CONFIG_CAN_CC770 is not set
# CONFIG_CAN_IFI_CANFD is not set
# CONFIG_CAN_M_CAN is not set
# CONFIG_CAN_PEAK_PCIEFD is not set
# CONFIG_CAN_SJA1000 is not set
# CONFIG_CAN_SOFTING is not set

#
# CAN SPI interfaces
#
# CONFIG_CAN_HI311X is not set
# CONFIG_CAN_MCP251X is not set

#
# CAN USB interfaces
#
# CONFIG_CAN_EMS_USB is not set
# CONFIG_CAN_ESD_USB2 is not set
# CONFIG_CAN_GS_USB is not set
# CONFIG_CAN_KVASER_USB is not set
# CONFIG_CAN_PEAK_USB is not set
# CONFIG_CAN_8DEV_USB is not set
# CONFIG_CAN_MCBA_USB is not set
# CONFIG_CAN_DEBUG_DEVICES is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
# CONFIG_AF_KCM is not set
CONFIG_STREAM_PARSER=y
CONFIG_FIB_RULES=y
CONFIG_WIRELESS=y
CONFIG_WIRELESS_EXT=y
CONFIG_WEXT_CORE=y
CONFIG_WEXT_PROC=y
CONFIG_WEXT_PRIV=y
CONFIG_CFG80211=m
# CONFIG_NL80211_TESTMODE is not set
# CONFIG_CFG80211_DEVELOPER_WARNINGS is not set
# CONFIG_CFG80211_CERTIFICATION_ONUS is not set
CONFIG_CFG80211_REQUIRE_SIGNED_REGDB=y
CONFIG_CFG80211_USE_KERNEL_REGDB_KEYS=y
CONFIG_CFG80211_DEFAULT_PS=y
# CONFIG_CFG80211_DEBUGFS is not set
CONFIG_CFG80211_CRDA_SUPPORT=y
CONFIG_CFG80211_WEXT=y
CONFIG_LIB80211=m
# CONFIG_LIB80211_DEBUG is not set
CONFIG_MAC80211=m
CONFIG_MAC80211_HAS_RC=y
CONFIG_MAC80211_RC_MINSTREL=y
CONFIG_MAC80211_RC_MINSTREL_HT=y
# CONFIG_MAC80211_RC_MINSTREL_VHT is not set
CONFIG_MAC80211_RC_DEFAULT_MINSTREL=y
CONFIG_MAC80211_RC_DEFAULT="minstrel_ht"
CONFIG_MAC80211_MESH=y
CONFIG_MAC80211_LEDS=y
CONFIG_MAC80211_DEBUGFS=y
# CONFIG_MAC80211_MESSAGE_TRACING is not set
# CONFIG_MAC80211_DEBUG_MENU is not set
CONFIG_MAC80211_STA_HASH_MAX_SIZE=0
# CONFIG_WIMAX is not set
CONFIG_RFKILL=m
CONFIG_RFKILL_LEDS=y
CONFIG_RFKILL_INPUT=y
# CONFIG_RFKILL_GPIO is not set
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
# CONFIG_NET_9P_XEN is not set
# CONFIG_NET_9P_DEBUG is not set
# CONFIG_CAIF is not set
# CONFIG_CEPH_LIB is not set
# CONFIG_NFC is not set
# CONFIG_PSAMPLE is not set
# CONFIG_NET_IFE is not set
CONFIG_LWTUNNEL=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_DST_CACHE=y
CONFIG_GRO_CELLS=y
# CONFIG_NET_DEVLINK is not set
CONFIG_MAY_USE_DEVLINK=y
CONFIG_HAVE_EBPF_JIT=y

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER=y
CONFIG_UEVENT_HELPER_PATH=""
CONFIG_DEVTMPFS=y
CONFIG_DEVTMPFS_MOUNT=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_EXTRA_FIRMWARE=""
CONFIG_FW_LOADER_USER_HELPER=y
CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
CONFIG_ALLOW_DEV_COREDUMP=y
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_DEBUG_TEST_DRIVER_REMOVE is not set
# CONFIG_TEST_ASYNC_DRIVER_PROBE is not set
CONFIG_SYS_HYPERVISOR=y
# CONFIG_GENERIC_CPU_DEVICES is not set
CONFIG_GENERIC_CPU_AUTOPROBE=y
CONFIG_GENERIC_CPU_VULNERABILITIES=y
CONFIG_REGMAP=y
CONFIG_REGMAP_I2C=y
CONFIG_REGMAP_SPI=y
CONFIG_DMA_SHARED_BUFFER=y
# CONFIG_DMA_FENCE_TRACE is not set
CONFIG_DMA_CMA=y

#
# Default contiguous memory area size:
#
CONFIG_CMA_SIZE_MBYTES=200
CONFIG_CMA_SIZE_SEL_MBYTES=y
# CONFIG_CMA_SIZE_SEL_PERCENTAGE is not set
# CONFIG_CMA_SIZE_SEL_MIN is not set
# CONFIG_CMA_SIZE_SEL_MAX is not set
CONFIG_CMA_ALIGNMENT=8

#
# Bus devices
#
CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
CONFIG_MTD=m
# CONFIG_MTD_TESTS is not set
# CONFIG_MTD_REDBOOT_PARTS is not set
# CONFIG_MTD_CMDLINE_PARTS is not set
# CONFIG_MTD_AR7_PARTS is not set

#
# Partition parsers
#

#
# User Modules And Translation Layers
#
CONFIG_MTD_BLKDEVS=m
CONFIG_MTD_BLOCK=m
# CONFIG_MTD_BLOCK_RO is not set
# CONFIG_FTL is not set
# CONFIG_NFTL is not set
# CONFIG_INFTL is not set
# CONFIG_RFD_FTL is not set
# CONFIG_SSFDC is not set
# CONFIG_SM_FTL is not set
# CONFIG_MTD_OOPS is not set
# CONFIG_MTD_SWAP is not set
# CONFIG_MTD_PARTITIONED_MASTER is not set

#
# RAM/ROM/Flash chip drivers
#
# CONFIG_MTD_CFI is not set
# CONFIG_MTD_JEDECPROBE is not set
CONFIG_MTD_MAP_BANK_WIDTH_1=y
CONFIG_MTD_MAP_BANK_WIDTH_2=y
CONFIG_MTD_MAP_BANK_WIDTH_4=y
# CONFIG_MTD_MAP_BANK_WIDTH_8 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_16 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_32 is not set
CONFIG_MTD_CFI_I1=y
CONFIG_MTD_CFI_I2=y
# CONFIG_MTD_CFI_I4 is not set
# CONFIG_MTD_CFI_I8 is not set
# CONFIG_MTD_RAM is not set
# CONFIG_MTD_ROM is not set
# CONFIG_MTD_ABSENT is not set

#
# Mapping drivers for chip access
#
# CONFIG_MTD_COMPLEX_MAPPINGS is not set
# CONFIG_MTD_INTEL_VR_NOR is not set
# CONFIG_MTD_PLATRAM is not set

#
# Self-contained MTD device drivers
#
# CONFIG_MTD_PMC551 is not set
# CONFIG_MTD_DATAFLASH is not set
# CONFIG_MTD_MCHP23K256 is not set
# CONFIG_MTD_SST25L is not set
# CONFIG_MTD_SLRAM is not set
# CONFIG_MTD_PHRAM is not set
# CONFIG_MTD_MTDRAM is not set
# CONFIG_MTD_BLOCK2MTD is not set

#
# Disk-On-Chip Device Drivers
#
# CONFIG_MTD_DOCG3 is not set
# CONFIG_MTD_NAND is not set
# CONFIG_MTD_ONENAND is not set

#
# LPDDR & LPDDR2 PCM memory drivers
#
# CONFIG_MTD_LPDDR is not set
# CONFIG_MTD_SPI_NOR is not set
CONFIG_MTD_UBI=m
CONFIG_MTD_UBI_WL_THRESHOLD=4096
CONFIG_MTD_UBI_BEB_LIMIT=20
# CONFIG_MTD_UBI_FASTMAP is not set
# CONFIG_MTD_UBI_GLUEBI is not set
# CONFIG_MTD_UBI_BLOCK is not set
# CONFIG_OF is not set
CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_SERIAL=m
# CONFIG_PARPORT_PC_FIFO is not set
# CONFIG_PARPORT_PC_SUPERIO is not set
# CONFIG_PARPORT_GSC is not set
# CONFIG_PARPORT_AX88796 is not set
CONFIG_PARPORT_1284=y
CONFIG_PARPORT_NOT_PC=y
CONFIG_PNP=y
# CONFIG_PNP_DEBUG_MESSAGES is not set

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_NULL_BLK=m
CONFIG_BLK_DEV_NULL_BLK_FAULT_INJECTION=y
CONFIG_BLK_DEV_FD=m
CONFIG_CDROM=m
# CONFIG_PARIDE is not set
CONFIG_BLK_DEV_PCIESSD_MTIP32XX=m
# CONFIG_ZRAM is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_LOOP_MIN_COUNT=0
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
# CONFIG_BLK_DEV_DRBD is not set
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SKD is not set
CONFIG_BLK_DEV_SX8=m
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=16384
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
CONFIG_ATA_OVER_ETH=m
CONFIG_XEN_BLKDEV_FRONTEND=m
# CONFIG_XEN_BLKDEV_BACKEND is not set
CONFIG_VIRTIO_BLK=y
# CONFIG_VIRTIO_BLK_SCSI is not set
# CONFIG_BLK_DEV_RBD is not set
CONFIG_BLK_DEV_RSXX=m

#
# NVME Support
#
CONFIG_NVME_CORE=m
CONFIG_BLK_DEV_NVME=m
CONFIG_NVME_MULTIPATH=y
CONFIG_NVME_FABRICS=m
# CONFIG_NVME_FC is not set
CONFIG_NVME_TARGET=m
CONFIG_NVME_TARGET_LOOP=m
# CONFIG_NVME_TARGET_FC is not set

#
# Misc devices
#
CONFIG_SENSORS_LIS3LV02D=m
# CONFIG_AD525X_DPOT is not set
# CONFIG_DUMMY_IRQ is not set
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
CONFIG_SGI_IOC4=m
CONFIG_TIFM_CORE=m
CONFIG_TIFM_7XX1=m
# CONFIG_ICS932S401 is not set
CONFIG_ENCLOSURE_SERVICES=m
CONFIG_SGI_XP=m
CONFIG_HP_ILO=m
CONFIG_SGI_GRU=m
# CONFIG_SGI_GRU_DEBUG is not set
CONFIG_APDS9802ALS=m
CONFIG_ISL29003=m
CONFIG_ISL29020=m
CONFIG_SENSORS_TSL2550=m
CONFIG_SENSORS_BH1770=m
CONFIG_SENSORS_APDS990X=m
# CONFIG_HMC6352 is not set
# CONFIG_DS1682 is not set
CONFIG_VMWARE_BALLOON=m
# CONFIG_USB_SWITCH_FSA9480 is not set
# CONFIG_LATTICE_ECP3_CONFIG is not set
# CONFIG_SRAM is not set
# CONFIG_PCI_ENDPOINT_TEST is not set
# CONFIG_MISC_RTSX is not set
# CONFIG_C2PORT is not set

#
# EEPROM support
#
CONFIG_EEPROM_AT24=m
# CONFIG_EEPROM_AT25 is not set
CONFIG_EEPROM_LEGACY=m
CONFIG_EEPROM_MAX6875=m
CONFIG_EEPROM_93CX6=m
# CONFIG_EEPROM_93XX46 is not set
# CONFIG_EEPROM_IDT_89HPESX is not set
CONFIG_CB710_CORE=m
# CONFIG_CB710_DEBUG is not set
CONFIG_CB710_DEBUG_ASSUMPTIONS=y

#
# Texas Instruments shared transport line discipline
#
# CONFIG_TI_ST is not set
CONFIG_SENSORS_LIS3_I2C=m
CONFIG_ALTERA_STAPL=m
CONFIG_INTEL_MEI=y
CONFIG_INTEL_MEI_ME=y
# CONFIG_INTEL_MEI_TXE is not set
CONFIG_VMWARE_VMCI=m

#
# Intel MIC & related support
#

#
# Intel MIC Bus Driver
#
# CONFIG_INTEL_MIC_BUS is not set

#
# SCIF Bus Driver
#
# CONFIG_SCIF_BUS is not set

#
# VOP Bus Driver
#
# CONFIG_VOP_BUS is not set

#
# Intel MIC Host Driver
#

#
# Intel MIC Card Driver
#

#
# SCIF Driver
#

#
# Intel MIC Coprocessor State Management (COSM) Drivers
#

#
# VOP Driver
#
# CONFIG_GENWQE is not set
# CONFIG_ECHO is not set
# CONFIG_CXL_BASE is not set
# CONFIG_CXL_AFU_DRIVER_OPS is not set
# CONFIG_CXL_LIB is not set
# CONFIG_OCXL_BASE is not set
# CONFIG_MISC_RTSX_PCI is not set
# CONFIG_MISC_RTSX_USB is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set

#
# SCSI device support
#
CONFIG_SCSI_MOD=y
CONFIG_RAID_ATTRS=m
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_NETLINK=y
# CONFIG_SCSI_MQ_DEFAULT is not set
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
CONFIG_CHR_DEV_ST=m
CONFIG_CHR_DEV_OSST=m
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=m
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_ENCLOSURE=m
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SCAN_ASYNC=y

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
CONFIG_SCSI_SAS_LIBSAS=m
CONFIG_SCSI_SAS_ATA=y
CONFIG_SCSI_SAS_HOST_SMP=y
CONFIG_SCSI_SRP_ATTRS=m
CONFIG_SCSI_LOWLEVEL=y
CONFIG_ISCSI_TCP=m
CONFIG_ISCSI_BOOT_SYSFS=m
CONFIG_SCSI_CXGB3_ISCSI=m
CONFIG_SCSI_CXGB4_ISCSI=m
CONFIG_SCSI_BNX2_ISCSI=m
CONFIG_SCSI_BNX2X_FCOE=m
CONFIG_BE2ISCSI=m
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
CONFIG_SCSI_HPSA=m
CONFIG_SCSI_3W_9XXX=m
CONFIG_SCSI_3W_SAS=m
# CONFIG_SCSI_ACARD is not set
CONFIG_SCSI_AACRAID=m
# CONFIG_SCSI_AIC7XXX is not set
CONFIG_SCSI_AIC79XX=m
CONFIG_AIC79XX_CMDS_PER_DEVICE=4
CONFIG_AIC79XX_RESET_DELAY_MS=15000
# CONFIG_AIC79XX_DEBUG_ENABLE is not set
CONFIG_AIC79XX_DEBUG_MASK=0
# CONFIG_AIC79XX_REG_PRETTY_PRINT is not set
# CONFIG_SCSI_AIC94XX is not set
CONFIG_SCSI_MVSAS=m
# CONFIG_SCSI_MVSAS_DEBUG is not set
CONFIG_SCSI_MVSAS_TASKLET=y
CONFIG_SCSI_MVUMI=m
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
CONFIG_SCSI_ARCMSR=m
# CONFIG_SCSI_ESAS2R is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
CONFIG_MEGARAID_SAS=m
CONFIG_SCSI_MPT3SAS=m
CONFIG_SCSI_MPT2SAS_MAX_SGE=128
CONFIG_SCSI_MPT3SAS_MAX_SGE=128
CONFIG_SCSI_MPT2SAS=m
# CONFIG_SCSI_SMARTPQI is not set
CONFIG_SCSI_UFSHCD=m
CONFIG_SCSI_UFSHCD_PCI=m
# CONFIG_SCSI_UFS_DWC_TC_PCI is not set
# CONFIG_SCSI_UFSHCD_PLATFORM is not set
CONFIG_SCSI_HPTIOP=m
# CONFIG_SCSI_BUSLOGIC is not set
CONFIG_VMWARE_PVSCSI=m
# CONFIG_XEN_SCSI_FRONTEND is not set
CONFIG_HYPERV_STORAGE=m
CONFIG_LIBFC=m
CONFIG_LIBFCOE=m
CONFIG_FCOE=m
CONFIG_FCOE_FNIC=m
# CONFIG_SCSI_SNIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
CONFIG_SCSI_ISCI=m
# CONFIG_SCSI_IPS is not set
CONFIG_SCSI_INITIO=m
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
CONFIG_SCSI_STEX=m
# CONFIG_SCSI_SYM53C8XX_2 is not set
CONFIG_SCSI_IPR=m
CONFIG_SCSI_IPR_TRACE=y
CONFIG_SCSI_IPR_DUMP=y
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLA_FC=m
# CONFIG_TCM_QLA2XXX is not set
CONFIG_SCSI_QLA_ISCSI=m
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_AM53C974 is not set
# CONFIG_SCSI_WD719X is not set
CONFIG_SCSI_DEBUG=m
CONFIG_SCSI_PMCRAID=m
CONFIG_SCSI_PM8001=m
# CONFIG_SCSI_BFA_FC is not set
CONFIG_SCSI_VIRTIO=m
CONFIG_SCSI_CHELSIO_FCOE=m
CONFIG_SCSI_DH=y
CONFIG_SCSI_DH_RDAC=y
CONFIG_SCSI_DH_HP_SW=y
CONFIG_SCSI_DH_EMC=y
CONFIG_SCSI_DH_ALUA=y
CONFIG_SCSI_OSD_INITIATOR=m
CONFIG_SCSI_OSD_ULD=m
CONFIG_SCSI_OSD_DPRINT_SENSE=1
# CONFIG_SCSI_OSD_DEBUG is not set
CONFIG_ATA=m
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_VERBOSE_ERROR=y
CONFIG_ATA_ACPI=y
# CONFIG_SATA_ZPODD is not set
CONFIG_SATA_PMP=y

#
# Controllers with non-SFF native interface
#
CONFIG_SATA_AHCI=m
CONFIG_SATA_MOBILE_LPM_POLICY=0
CONFIG_SATA_AHCI_PLATFORM=m
# CONFIG_SATA_INIC162X is not set
CONFIG_SATA_ACARD_AHCI=m
CONFIG_SATA_SIL24=m
CONFIG_ATA_SFF=y

#
# SFF controllers with custom DMA interface
#
CONFIG_PDC_ADMA=m
CONFIG_SATA_QSTOR=m
CONFIG_SATA_SX4=m
CONFIG_ATA_BMDMA=y

#
# SATA SFF controllers with BMDMA
#
CONFIG_ATA_PIIX=m
# CONFIG_SATA_DWC is not set
CONFIG_SATA_MV=m
CONFIG_SATA_NV=m
CONFIG_SATA_PROMISE=m
CONFIG_SATA_SIL=m
CONFIG_SATA_SIS=m
CONFIG_SATA_SVW=m
CONFIG_SATA_ULI=m
CONFIG_SATA_VIA=m
CONFIG_SATA_VITESSE=m

#
# PATA SFF controllers with BMDMA
#
CONFIG_PATA_ALI=m
CONFIG_PATA_AMD=m
CONFIG_PATA_ARTOP=m
CONFIG_PATA_ATIIXP=m
CONFIG_PATA_ATP867X=m
CONFIG_PATA_CMD64X=m
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
CONFIG_PATA_HPT366=m
CONFIG_PATA_HPT37X=m
CONFIG_PATA_HPT3X2N=m
CONFIG_PATA_HPT3X3=m
# CONFIG_PATA_HPT3X3_DMA is not set
CONFIG_PATA_IT8213=m
CONFIG_PATA_IT821X=m
CONFIG_PATA_JMICRON=m
CONFIG_PATA_MARVELL=m
CONFIG_PATA_NETCELL=m
CONFIG_PATA_NINJA32=m
# CONFIG_PATA_NS87415 is not set
CONFIG_PATA_OLDPIIX=m
# CONFIG_PATA_OPTIDMA is not set
CONFIG_PATA_PDC2027X=m
CONFIG_PATA_PDC_OLD=m
# CONFIG_PATA_RADISYS is not set
CONFIG_PATA_RDC=m
CONFIG_PATA_SCH=m
CONFIG_PATA_SERVERWORKS=m
CONFIG_PATA_SIL680=m
CONFIG_PATA_SIS=m
CONFIG_PATA_TOSHIBA=m
# CONFIG_PATA_TRIFLEX is not set
CONFIG_PATA_VIA=m
# CONFIG_PATA_WINBOND is not set

#
# PIO-only SFF controllers
#
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_PLATFORM is not set
# CONFIG_PATA_RZ1000 is not set

#
# Generic fallback / legacy drivers
#
CONFIG_PATA_ACPI=m
CONFIG_ATA_GENERIC=m
# CONFIG_PATA_LEGACY is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
CONFIG_MD_LINEAR=m
CONFIG_MD_RAID0=m
CONFIG_MD_RAID1=m
CONFIG_MD_RAID10=m
CONFIG_MD_RAID456=m
CONFIG_MD_MULTIPATH=m
CONFIG_MD_FAULTY=m
# CONFIG_MD_CLUSTER is not set
# CONFIG_BCACHE is not set
CONFIG_BLK_DEV_DM_BUILTIN=y
CONFIG_BLK_DEV_DM=m
# CONFIG_DM_MQ_DEFAULT is not set
CONFIG_DM_DEBUG=y
CONFIG_DM_BUFIO=m
# CONFIG_DM_DEBUG_BLOCK_MANAGER_LOCKING is not set
CONFIG_DM_BIO_PRISON=m
CONFIG_DM_PERSISTENT_DATA=m
# CONFIG_DM_UNSTRIPED is not set
CONFIG_DM_CRYPT=m
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_THIN_PROVISIONING=m
CONFIG_DM_CACHE=m
CONFIG_DM_CACHE_SMQ=m
# CONFIG_DM_ERA is not set
CONFIG_DM_MIRROR=m
CONFIG_DM_LOG_USERSPACE=m
CONFIG_DM_RAID=m
CONFIG_DM_ZERO=m
CONFIG_DM_MULTIPATH=m
CONFIG_DM_MULTIPATH_QL=m
CONFIG_DM_MULTIPATH_ST=m
CONFIG_DM_DELAY=m
CONFIG_DM_UEVENT=y
CONFIG_DM_FLAKEY=m
CONFIG_DM_VERITY=m
# CONFIG_DM_VERITY_FEC is not set
CONFIG_DM_SWITCH=m
CONFIG_DM_LOG_WRITES=m
# CONFIG_DM_INTEGRITY is not set
# CONFIG_DM_ZONED is not set
CONFIG_TARGET_CORE=m
CONFIG_TCM_IBLOCK=m
CONFIG_TCM_FILEIO=m
CONFIG_TCM_PSCSI=m
# CONFIG_TCM_USER2 is not set
CONFIG_LOOPBACK_TARGET=m
CONFIG_TCM_FC=m
CONFIG_ISCSI_TARGET=m
# CONFIG_ISCSI_TARGET_CXGB4 is not set
# CONFIG_SBP_TARGET is not set
CONFIG_FUSION=y
CONFIG_FUSION_SPI=m
# CONFIG_FUSION_FC is not set
CONFIG_FUSION_SAS=m
CONFIG_FUSION_MAX_SGE=128
CONFIG_FUSION_CTL=m
CONFIG_FUSION_LOGGING=y

#
# IEEE 1394 (FireWire) support
#
CONFIG_FIREWIRE=m
CONFIG_FIREWIRE_OHCI=m
CONFIG_FIREWIRE_SBP2=m
CONFIG_FIREWIRE_NET=m
# CONFIG_FIREWIRE_NOSY is not set
CONFIG_MACINTOSH_DRIVERS=y
CONFIG_MAC_EMUMOUSEBTN=y
CONFIG_NETDEVICES=y
CONFIG_MII=y
CONFIG_NET_CORE=y
CONFIG_BONDING=m
CONFIG_DUMMY=m
# CONFIG_EQUALIZER is not set
CONFIG_NET_FC=y
CONFIG_IFB=m
CONFIG_NET_TEAM=m
CONFIG_NET_TEAM_MODE_BROADCAST=m
CONFIG_NET_TEAM_MODE_ROUNDROBIN=m
CONFIG_NET_TEAM_MODE_RANDOM=m
CONFIG_NET_TEAM_MODE_ACTIVEBACKUP=m
CONFIG_NET_TEAM_MODE_LOADBALANCE=m
CONFIG_MACVLAN=m
CONFIG_MACVTAP=m
# CONFIG_IPVLAN is not set
CONFIG_VXLAN=m
# CONFIG_GENEVE is not set
# CONFIG_GTP is not set
CONFIG_MACSEC=y
CONFIG_NETCONSOLE=m
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_TUN=m
CONFIG_TAP=m
# CONFIG_TUN_VNET_CROSS_LE is not set
CONFIG_VETH=m
CONFIG_VIRTIO_NET=y
CONFIG_NLMON=m
CONFIG_NET_VRF=y
# CONFIG_ARCNET is not set
# CONFIG_ATM_DRIVERS is not set

#
# CAIF transport drivers
#

#
# Distributed Switch Architecture drivers
#
CONFIG_ETHERNET=y
CONFIG_MDIO=y
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_NET_VENDOR_ADAPTEC is not set
CONFIG_NET_VENDOR_AGERE=y
# CONFIG_ET131X is not set
CONFIG_NET_VENDOR_ALACRITECH=y
# CONFIG_SLICOSS is not set
# CONFIG_NET_VENDOR_ALTEON is not set
# CONFIG_ALTERA_TSE is not set
CONFIG_NET_VENDOR_AMAZON=y
# CONFIG_ENA_ETHERNET is not set
# CONFIG_NET_VENDOR_AMD is not set
CONFIG_NET_VENDOR_AQUANTIA=y
# CONFIG_AQTION is not set
CONFIG_NET_VENDOR_ARC=y
CONFIG_NET_VENDOR_ATHEROS=y
CONFIG_ATL2=m
CONFIG_ATL1=m
CONFIG_ATL1E=m
CONFIG_ATL1C=m
CONFIG_ALX=m
# CONFIG_NET_VENDOR_AURORA is not set
CONFIG_NET_CADENCE=y
# CONFIG_MACB is not set
CONFIG_NET_VENDOR_BROADCOM=y
CONFIG_B44=m
CONFIG_B44_PCI_AUTOSELECT=y
CONFIG_B44_PCICORE_AUTOSELECT=y
CONFIG_B44_PCI=y
CONFIG_BNX2=m
CONFIG_CNIC=m
CONFIG_TIGON3=y
CONFIG_TIGON3_HWMON=y
# CONFIG_BNX2X is not set
# CONFIG_BNXT is not set
CONFIG_NET_VENDOR_BROCADE=y
CONFIG_BNA=m
CONFIG_NET_VENDOR_CAVIUM=y
# CONFIG_THUNDER_NIC_PF is not set
# CONFIG_THUNDER_NIC_VF is not set
# CONFIG_THUNDER_NIC_BGX is not set
# CONFIG_THUNDER_NIC_RGX is not set
CONFIG_CAVIUM_PTP=y
# CONFIG_LIQUIDIO is not set
# CONFIG_LIQUIDIO_VF is not set
CONFIG_NET_VENDOR_CHELSIO=y
# CONFIG_CHELSIO_T1 is not set
CONFIG_CHELSIO_T3=m
CONFIG_CHELSIO_T4=m
# CONFIG_CHELSIO_T4_DCB is not set
CONFIG_CHELSIO_T4VF=m
CONFIG_CHELSIO_LIB=m
CONFIG_NET_VENDOR_CISCO=y
CONFIG_ENIC=m
CONFIG_NET_VENDOR_CORTINA=y
# CONFIG_CX_ECAT is not set
CONFIG_DNET=m
CONFIG_NET_VENDOR_DEC=y
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_DE2104X_DSL=0
CONFIG_TULIP=y
# CONFIG_TULIP_MWI is not set
CONFIG_TULIP_MMIO=y
# CONFIG_TULIP_NAPI is not set
CONFIG_DE4X5=m
CONFIG_WINBOND_840=m
CONFIG_DM9102=m
CONFIG_ULI526X=m
CONFIG_PCMCIA_XIRCOM=m
# CONFIG_NET_VENDOR_DLINK is not set
CONFIG_NET_VENDOR_EMULEX=y
CONFIG_BE2NET=m
CONFIG_BE2NET_HWMON=y
CONFIG_NET_VENDOR_EZCHIP=y
# CONFIG_NET_VENDOR_EXAR is not set
# CONFIG_NET_VENDOR_HP is not set
CONFIG_NET_VENDOR_HUAWEI=y
# CONFIG_HINIC is not set
CONFIG_NET_VENDOR_INTEL=y
# CONFIG_E100 is not set
CONFIG_E1000=y
CONFIG_E1000E=y
CONFIG_E1000E_HWTS=y
CONFIG_IGB=y
CONFIG_IGB_HWMON=y
CONFIG_IGBVF=m
CONFIG_IXGB=m
CONFIG_IXGBE=y
CONFIG_IXGBE_HWMON=y
CONFIG_IXGBE_DCB=y
CONFIG_IXGBEVF=m
CONFIG_I40E=m
# CONFIG_I40E_DCB is not set
# CONFIG_I40EVF is not set
# CONFIG_FM10K is not set
# CONFIG_NET_VENDOR_I825XX is not set
CONFIG_JME=m
CONFIG_NET_VENDOR_MARVELL=y
CONFIG_MVMDIO=m
CONFIG_SKGE=m
CONFIG_SKGE_DEBUG=y
CONFIG_SKGE_GENESIS=y
CONFIG_SKY2=m
CONFIG_SKY2_DEBUG=y
CONFIG_NET_VENDOR_MELLANOX=y
CONFIG_MLX4_EN=m
CONFIG_MLX4_EN_DCB=y
CONFIG_MLX4_CORE=m
CONFIG_MLX4_DEBUG=y
CONFIG_MLX4_CORE_GEN2=y
# CONFIG_MLX5_CORE is not set
# CONFIG_MLXSW_CORE is not set
# CONFIG_MLXFW is not set
# CONFIG_NET_VENDOR_MICREL is not set
CONFIG_NET_VENDOR_MICROCHIP=y
# CONFIG_ENC28J60 is not set
# CONFIG_ENCX24J600 is not set
CONFIG_NET_VENDOR_MYRI=y
CONFIG_MYRI10GE=m
# CONFIG_FEALNX is not set
# CONFIG_NET_VENDOR_NATSEMI is not set
CONFIG_NET_VENDOR_NETRONOME=y
# CONFIG_NFP is not set
# CONFIG_NET_VENDOR_NVIDIA is not set
CONFIG_NET_VENDOR_OKI=y
CONFIG_ETHOC=m
CONFIG_NET_PACKET_ENGINE=y
# CONFIG_HAMACHI is not set
CONFIG_YELLOWFIN=m
CONFIG_NET_VENDOR_QLOGIC=y
CONFIG_QLA3XXX=m
CONFIG_QLCNIC=m
CONFIG_QLCNIC_SRIOV=y
CONFIG_QLCNIC_DCB=y
CONFIG_QLCNIC_HWMON=y
CONFIG_QLGE=m
CONFIG_NETXEN_NIC=m
# CONFIG_QED is not set
CONFIG_NET_VENDOR_QUALCOMM=y
# CONFIG_QCOM_EMAC is not set
# CONFIG_RMNET is not set
CONFIG_NET_VENDOR_REALTEK=y
# CONFIG_ATP is not set
CONFIG_8139CP=y
CONFIG_8139TOO=y
CONFIG_8139TOO_PIO=y
# CONFIG_8139TOO_TUNE_TWISTER is not set
CONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not set
CONFIG_R8169=y
CONFIG_NET_VENDOR_RENESAS=y
# CONFIG_NET_VENDOR_RDC is not set
CONFIG_NET_VENDOR_ROCKER=y
CONFIG_NET_VENDOR_SAMSUNG=y
# CONFIG_SXGBE_ETH is not set
# CONFIG_NET_VENDOR_SEEQ is not set
# CONFIG_NET_VENDOR_SILAN is not set
# CONFIG_NET_VENDOR_SIS is not set
CONFIG_NET_VENDOR_SOLARFLARE=y
CONFIG_SFC=m
CONFIG_SFC_MTD=y
CONFIG_SFC_MCDI_MON=y
CONFIG_SFC_SRIOV=y
CONFIG_SFC_MCDI_LOGGING=y
# CONFIG_SFC_FALCON is not set
CONFIG_NET_VENDOR_SMSC=y
CONFIG_EPIC100=m
# CONFIG_SMSC911X is not set
CONFIG_SMSC9420=m
CONFIG_NET_VENDOR_SOCIONEXT=y
# CONFIG_NET_VENDOR_STMICRO is not set
# CONFIG_NET_VENDOR_SUN is not set
# CONFIG_NET_VENDOR_TEHUTI is not set
# CONFIG_NET_VENDOR_TI is not set
# CONFIG_NET_VENDOR_VIA is not set
# CONFIG_NET_VENDOR_WIZNET is not set
CONFIG_NET_VENDOR_SYNOPSYS=y
# CONFIG_DWC_XLGMAC is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_NET_SB1000 is not set
CONFIG_MDIO_DEVICE=y
CONFIG_MDIO_BUS=y
CONFIG_MDIO_BITBANG=m
# CONFIG_MDIO_GPIO is not set
# CONFIG_MDIO_THUNDER is not set
CONFIG_PHYLIB=y
CONFIG_SWPHY=y
# CONFIG_LED_TRIGGER_PHY is not set

#
# MII PHY device drivers
#
CONFIG_AMD_PHY=m
# CONFIG_AQUANTIA_PHY is not set
CONFIG_AT803X_PHY=m
# CONFIG_BCM7XXX_PHY is not set
CONFIG_BCM87XX_PHY=m
CONFIG_BCM_NET_PHYLIB=m
CONFIG_BROADCOM_PHY=m
CONFIG_CICADA_PHY=m
# CONFIG_CORTINA_PHY is not set
CONFIG_DAVICOM_PHY=m
# CONFIG_DP83822_PHY is not set
# CONFIG_DP83848_PHY is not set
# CONFIG_DP83867_PHY is not set
CONFIG_FIXED_PHY=y
CONFIG_ICPLUS_PHY=m
# CONFIG_INTEL_XWAY_PHY is not set
CONFIG_LSI_ET1011C_PHY=m
CONFIG_LXT_PHY=m
CONFIG_MARVELL_PHY=m
# CONFIG_MARVELL_10G_PHY is not set
CONFIG_MICREL_PHY=m
# CONFIG_MICROCHIP_PHY is not set
# CONFIG_MICROSEMI_PHY is not set
CONFIG_NATIONAL_PHY=m
CONFIG_QSEMI_PHY=m
CONFIG_REALTEK_PHY=m
# CONFIG_RENESAS_PHY is not set
# CONFIG_ROCKCHIP_PHY is not set
CONFIG_SMSC_PHY=m
CONFIG_STE10XP=m
# CONFIG_TERANETICS_PHY is not set
CONFIG_VITESSE_PHY=m
# CONFIG_XILINX_GMII2RGMII is not set
# CONFIG_MICREL_KS8995MA is not set
# CONFIG_PLIP is not set
CONFIG_PPP=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_FILTER=y
CONFIG_PPP_MPPE=m
CONFIG_PPP_MULTILINK=y
CONFIG_PPPOATM=m
CONFIG_PPPOE=m
CONFIG_PPTP=m
CONFIG_PPPOL2TP=m
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_SLIP=m
CONFIG_SLHC=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLIP_SMART=y
# CONFIG_SLIP_MODE_SLIP6 is not set
CONFIG_USB_NET_DRIVERS=y
CONFIG_USB_CATC=y
CONFIG_USB_KAWETH=y
CONFIG_USB_PEGASUS=y
CONFIG_USB_RTL8150=y
CONFIG_USB_RTL8152=m
# CONFIG_USB_LAN78XX is not set
CONFIG_USB_USBNET=y
CONFIG_USB_NET_AX8817X=y
CONFIG_USB_NET_AX88179_178A=m
CONFIG_USB_NET_CDCETHER=y
CONFIG_USB_NET_CDC_EEM=y
CONFIG_USB_NET_CDC_NCM=m
# CONFIG_USB_NET_HUAWEI_CDC_NCM is not set
CONFIG_USB_NET_CDC_MBIM=m
CONFIG_USB_NET_DM9601=y
# CONFIG_USB_NET_SR9700 is not set
# CONFIG_USB_NET_SR9800 is not set
CONFIG_USB_NET_SMSC75XX=y
CONFIG_USB_NET_SMSC95XX=y
CONFIG_USB_NET_GL620A=y
CONFIG_USB_NET_NET1080=y
CONFIG_USB_NET_PLUSB=y
CONFIG_USB_NET_MCS7830=y
CONFIG_USB_NET_RNDIS_HOST=y
CONFIG_USB_NET_CDC_SUBSET_ENABLE=y
CONFIG_USB_NET_CDC_SUBSET=y
CONFIG_USB_ALI_M5632=y
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
CONFIG_USB_ARMLINUX=y
CONFIG_USB_EPSON2888=y
CONFIG_USB_KC2190=y
CONFIG_USB_NET_ZAURUS=y
CONFIG_USB_NET_CX82310_ETH=m
CONFIG_USB_NET_KALMIA=m
CONFIG_USB_NET_QMI_WWAN=m
CONFIG_USB_HSO=m
CONFIG_USB_NET_INT51X1=y
CONFIG_USB_IPHETH=y
CONFIG_USB_SIERRA_NET=y
CONFIG_USB_VL600=m
# CONFIG_USB_NET_CH9200 is not set
CONFIG_WLAN=y
# CONFIG_WIRELESS_WDS is not set
CONFIG_WLAN_VENDOR_ADMTEK=y
# CONFIG_ADM8211 is not set
CONFIG_WLAN_VENDOR_ATH=y
# CONFIG_ATH_DEBUG is not set
# CONFIG_ATH5K is not set
# CONFIG_ATH5K_PCI is not set
# CONFIG_ATH9K is not set
# CONFIG_ATH9K_HTC is not set
# CONFIG_CARL9170 is not set
# CONFIG_ATH6KL is not set
# CONFIG_AR5523 is not set
# CONFIG_WIL6210 is not set
# CONFIG_ATH10K is not set
# CONFIG_WCN36XX is not set
CONFIG_WLAN_VENDOR_ATMEL=y
# CONFIG_ATMEL is not set
# CONFIG_AT76C50X_USB is not set
CONFIG_WLAN_VENDOR_BROADCOM=y
# CONFIG_B43 is not set
# CONFIG_B43LEGACY is not set
# CONFIG_BRCMSMAC is not set
# CONFIG_BRCMFMAC is not set
CONFIG_WLAN_VENDOR_CISCO=y
# CONFIG_AIRO is not set
CONFIG_WLAN_VENDOR_INTEL=y
# CONFIG_IPW2100 is not set
# CONFIG_IPW2200 is not set
# CONFIG_IWL4965 is not set
# CONFIG_IWL3945 is not set
# CONFIG_IWLWIFI is not set
CONFIG_WLAN_VENDOR_INTERSIL=y
# CONFIG_HOSTAP is not set
# CONFIG_HERMES is not set
# CONFIG_P54_COMMON is not set
# CONFIG_PRISM54 is not set
CONFIG_WLAN_VENDOR_MARVELL=y
# CONFIG_LIBERTAS is not set
# CONFIG_LIBERTAS_THINFIRM is not set
# CONFIG_MWIFIEX is not set
# CONFIG_MWL8K is not set
CONFIG_WLAN_VENDOR_MEDIATEK=y
# CONFIG_MT7601U is not set
# CONFIG_MT76x2E is not set
CONFIG_WLAN_VENDOR_RALINK=y
# CONFIG_RT2X00 is not set
CONFIG_WLAN_VENDOR_REALTEK=y
# CONFIG_RTL8180 is not set
# CONFIG_RTL8187 is not set
CONFIG_RTL_CARDS=m
# CONFIG_RTL8192CE is not set
# CONFIG_RTL8192SE is not set
# CONFIG_RTL8192DE is not set
# CONFIG_RTL8723AE is not set
# CONFIG_RTL8723BE is not set
# CONFIG_RTL8188EE is not set
# CONFIG_RTL8192EE is not set
# CONFIG_RTL8821AE is not set
# CONFIG_RTL8192CU is not set
# CONFIG_RTL8XXXU is not set
CONFIG_WLAN_VENDOR_RSI=y
# CONFIG_RSI_91X is not set
CONFIG_WLAN_VENDOR_ST=y
# CONFIG_CW1200 is not set
CONFIG_WLAN_VENDOR_TI=y
# CONFIG_WL1251 is not set
# CONFIG_WL12XX is not set
# CONFIG_WL18XX is not set
# CONFIG_WLCORE is not set
CONFIG_WLAN_VENDOR_ZYDAS=y
# CONFIG_USB_ZD1201 is not set
# CONFIG_ZD1211RW is not set
CONFIG_WLAN_VENDOR_QUANTENNA=y
# CONFIG_QTNFMAC_PEARL_PCIE is not set
CONFIG_MAC80211_HWSIM=m
# CONFIG_USB_NET_RNDIS_WLAN is not set

#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#
CONFIG_WAN=y
# CONFIG_LANMEDIA is not set
CONFIG_HDLC=m
CONFIG_HDLC_RAW=m
# CONFIG_HDLC_RAW_ETH is not set
CONFIG_HDLC_CISCO=m
CONFIG_HDLC_FR=m
CONFIG_HDLC_PPP=m

#
# X.25/LAPB support is disabled
#
# CONFIG_PCI200SYN is not set
# CONFIG_WANXL is not set
# CONFIG_PC300TOO is not set
# CONFIG_FARSYNC is not set
# CONFIG_DSCC4 is not set
CONFIG_DLCI=m
CONFIG_DLCI_MAX=8
# CONFIG_SBNI is not set
CONFIG_IEEE802154_DRIVERS=m
CONFIG_IEEE802154_FAKELB=m
# CONFIG_IEEE802154_AT86RF230 is not set
# CONFIG_IEEE802154_MRF24J40 is not set
# CONFIG_IEEE802154_CC2520 is not set
# CONFIG_IEEE802154_ATUSB is not set
# CONFIG_IEEE802154_ADF7242 is not set
# CONFIG_IEEE802154_CA8210 is not set
CONFIG_XEN_NETDEV_FRONTEND=m
# CONFIG_XEN_NETDEV_BACKEND is not set
CONFIG_VMXNET3=m
# CONFIG_FUJITSU_ES is not set
CONFIG_HYPERV_NET=m
CONFIG_NETDEVSIM=m
CONFIG_ISDN=y
CONFIG_ISDN_I4L=m
CONFIG_ISDN_PPP=y
CONFIG_ISDN_PPP_VJ=y
CONFIG_ISDN_MPP=y
CONFIG_IPPP_FILTER=y
# CONFIG_ISDN_PPP_BSDCOMP is not set
CONFIG_ISDN_AUDIO=y
CONFIG_ISDN_TTY_FAX=y

#
# ISDN feature submodules
#
CONFIG_ISDN_DIVERSION=m

#
# ISDN4Linux hardware drivers
#

#
# Passive cards
#
# CONFIG_ISDN_DRV_HISAX is not set
CONFIG_ISDN_CAPI=m
# CONFIG_CAPI_TRACE is not set
CONFIG_ISDN_CAPI_CAPI20=m
CONFIG_ISDN_CAPI_MIDDLEWARE=y
CONFIG_ISDN_CAPI_CAPIDRV=m
# CONFIG_ISDN_CAPI_CAPIDRV_VERBOSE is not set

#
# CAPI hardware drivers
#
CONFIG_CAPI_AVM=y
CONFIG_ISDN_DRV_AVMB1_B1PCI=m
CONFIG_ISDN_DRV_AVMB1_B1PCIV4=y
CONFIG_ISDN_DRV_AVMB1_T1PCI=m
CONFIG_ISDN_DRV_AVMB1_C4=m
# CONFIG_CAPI_EICON is not set
CONFIG_ISDN_DRV_GIGASET=m
CONFIG_GIGASET_CAPI=y
# CONFIG_GIGASET_I4L is not set
# CONFIG_GIGASET_DUMMYLL is not set
CONFIG_GIGASET_BASE=m
CONFIG_GIGASET_M105=m
CONFIG_GIGASET_M101=m
# CONFIG_GIGASET_DEBUG is not set
CONFIG_HYSDN=m
CONFIG_HYSDN_CAPI=y
CONFIG_MISDN=m
CONFIG_MISDN_DSP=m
CONFIG_MISDN_L1OIP=m

#
# mISDN hardware drivers
#
CONFIG_MISDN_HFCPCI=m
CONFIG_MISDN_HFCMULTI=m
CONFIG_MISDN_HFCUSB=m
CONFIG_MISDN_AVMFRITZ=m
CONFIG_MISDN_SPEEDFAX=m
CONFIG_MISDN_INFINEON=m
CONFIG_MISDN_W6692=m
CONFIG_MISDN_NETJET=m
CONFIG_MISDN_IPAC=m
CONFIG_MISDN_ISAR=m
CONFIG_ISDN_HDLC=m
# CONFIG_NVM is not set

#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_LEDS=y
CONFIG_INPUT_FF_MEMLESS=m
CONFIG_INPUT_POLLDEV=m
CONFIG_INPUT_SPARSEKMAP=m
# CONFIG_INPUT_MATRIXKMAP is not set

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
# CONFIG_KEYBOARD_ADP5588 is not set
# CONFIG_KEYBOARD_ADP5589 is not set
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_QT1070 is not set
# CONFIG_KEYBOARD_QT2160 is not set
# CONFIG_KEYBOARD_DLINK_DIR685 is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_GPIO is not set
# CONFIG_KEYBOARD_GPIO_POLLED is not set
# CONFIG_KEYBOARD_TCA6416 is not set
# CONFIG_KEYBOARD_TCA8418 is not set
# CONFIG_KEYBOARD_MATRIX is not set
# CONFIG_KEYBOARD_LM8323 is not set
# CONFIG_KEYBOARD_LM8333 is not set
# CONFIG_KEYBOARD_MAX7359 is not set
# CONFIG_KEYBOARD_MCS is not set
# CONFIG_KEYBOARD_MPR121 is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_OPENCORES is not set
# CONFIG_KEYBOARD_SAMSUNG is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_TM2_TOUCHKEY is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_BYD=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_SYNAPTICS_SMBUS=y
CONFIG_MOUSE_PS2_CYPRESS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
CONFIG_MOUSE_PS2_ELANTECH=y
CONFIG_MOUSE_PS2_SENTELIC=y
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
CONFIG_MOUSE_PS2_FOCALTECH=y
# CONFIG_MOUSE_PS2_VMMOUSE is not set
CONFIG_MOUSE_PS2_SMBUS=y
CONFIG_MOUSE_SERIAL=m
CONFIG_MOUSE_APPLETOUCH=m
CONFIG_MOUSE_BCM5974=m
CONFIG_MOUSE_CYAPA=m
# CONFIG_MOUSE_ELAN_I2C is not set
CONFIG_MOUSE_VSXXXAA=m
# CONFIG_MOUSE_GPIO is not set
CONFIG_MOUSE_SYNAPTICS_I2C=m
CONFIG_MOUSE_SYNAPTICS_USB=m
# CONFIG_INPUT_JOYSTICK is not set
CONFIG_INPUT_TABLET=y
CONFIG_TABLET_USB_ACECAD=m
CONFIG_TABLET_USB_AIPTEK=m
CONFIG_TABLET_USB_GTCO=m
# CONFIG_TABLET_USB_HANWANG is not set
CONFIG_TABLET_USB_KBTAB=m
# CONFIG_TABLET_USB_PEGASUS is not set
# CONFIG_TABLET_SERIAL_WACOM4 is not set
CONFIG_INPUT_TOUCHSCREEN=y
CONFIG_TOUCHSCREEN_PROPERTIES=y
# CONFIG_TOUCHSCREEN_ADS7846 is not set
# CONFIG_TOUCHSCREEN_AD7877 is not set
# CONFIG_TOUCHSCREEN_AD7879 is not set
# CONFIG_TOUCHSCREEN_ATMEL_MXT is not set
# CONFIG_TOUCHSCREEN_AUO_PIXCIR is not set
# CONFIG_TOUCHSCREEN_BU21013 is not set
# CONFIG_TOUCHSCREEN_CY8CTMG110 is not set
# CONFIG_TOUCHSCREEN_CYTTSP_CORE is not set
# CONFIG_TOUCHSCREEN_CYTTSP4_CORE is not set
# CONFIG_TOUCHSCREEN_DYNAPRO is not set
# CONFIG_TOUCHSCREEN_HAMPSHIRE is not set
# CONFIG_TOUCHSCREEN_EETI is not set
# CONFIG_TOUCHSCREEN_EGALAX_SERIAL is not set
# CONFIG_TOUCHSCREEN_EXC3000 is not set
# CONFIG_TOUCHSCREEN_FUJITSU is not set
# CONFIG_TOUCHSCREEN_GOODIX is not set
# CONFIG_TOUCHSCREEN_HIDEEP is not set
# CONFIG_TOUCHSCREEN_ILI210X is not set
# CONFIG_TOUCHSCREEN_S6SY761 is not set
# CONFIG_TOUCHSCREEN_GUNZE is not set
# CONFIG_TOUCHSCREEN_EKTF2127 is not set
# CONFIG_TOUCHSCREEN_ELAN is not set
# CONFIG_TOUCHSCREEN_ELO is not set
CONFIG_TOUCHSCREEN_WACOM_W8001=m
CONFIG_TOUCHSCREEN_WACOM_I2C=m
# CONFIG_TOUCHSCREEN_MAX11801 is not set
# CONFIG_TOUCHSCREEN_MCS5000 is not set
# CONFIG_TOUCHSCREEN_MMS114 is not set
# CONFIG_TOUCHSCREEN_MELFAS_MIP4 is not set
# CONFIG_TOUCHSCREEN_MTOUCH is not set
# CONFIG_TOUCHSCREEN_INEXIO is not set
# CONFIG_TOUCHSCREEN_MK712 is not set
# CONFIG_TOUCHSCREEN_PENMOUNT is not set
# CONFIG_TOUCHSCREEN_EDT_FT5X06 is not set
# CONFIG_TOUCHSCREEN_TOUCHRIGHT is not set
# CONFIG_TOUCHSCREEN_TOUCHWIN is not set
# CONFIG_TOUCHSCREEN_PIXCIR is not set
# CONFIG_TOUCHSCREEN_WDT87XX_I2C is not set
# CONFIG_TOUCHSCREEN_WM97XX is not set
# CONFIG_TOUCHSCREEN_USB_COMPOSITE is not set
# CONFIG_TOUCHSCREEN_TOUCHIT213 is not set
# CONFIG_TOUCHSCREEN_TSC_SERIO is not set
# CONFIG_TOUCHSCREEN_TSC2004 is not set
# CONFIG_TOUCHSCREEN_TSC2005 is not set
# CONFIG_TOUCHSCREEN_TSC2007 is not set
# CONFIG_TOUCHSCREEN_RM_TS is not set
# CONFIG_TOUCHSCREEN_SILEAD is not set
# CONFIG_TOUCHSCREEN_SIS_I2C is not set
# CONFIG_TOUCHSCREEN_ST1232 is not set
# CONFIG_TOUCHSCREEN_STMFTS is not set
# CONFIG_TOUCHSCREEN_SUR40 is not set
# CONFIG_TOUCHSCREEN_SURFACE3_SPI is not set
# CONFIG_TOUCHSCREEN_SX8654 is not set
# CONFIG_TOUCHSCREEN_TPS6507X is not set
# CONFIG_TOUCHSCREEN_ZET6223 is not set
# CONFIG_TOUCHSCREEN_ZFORCE is not set
# CONFIG_TOUCHSCREEN_ROHM_BU21023 is not set
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_AD714X is not set
# CONFIG_INPUT_BMA150 is not set
# CONFIG_INPUT_E3X0_BUTTON is not set
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_MMA8450 is not set
CONFIG_INPUT_APANEL=m
# CONFIG_INPUT_GP2A is not set
# CONFIG_INPUT_GPIO_BEEPER is not set
# CONFIG_INPUT_GPIO_DECODER is not set
CONFIG_INPUT_ATLAS_BTNS=m
CONFIG_INPUT_ATI_REMOTE2=m
CONFIG_INPUT_KEYSPAN_REMOTE=m
# CONFIG_INPUT_KXTJ9 is not set
CONFIG_INPUT_POWERMATE=m
CONFIG_INPUT_YEALINK=m
CONFIG_INPUT_CM109=m
CONFIG_INPUT_UINPUT=m
# CONFIG_INPUT_PCF8574 is not set
# CONFIG_INPUT_PWM_BEEPER is not set
# CONFIG_INPUT_PWM_VIBRA is not set
# CONFIG_INPUT_GPIO_ROTARY_ENCODER is not set
# CONFIG_INPUT_ADXL34X is not set
# CONFIG_INPUT_IMS_PCU is not set
# CONFIG_INPUT_CMA3000 is not set
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=m
# CONFIG_INPUT_IDEAPAD_SLIDEBAR is not set
# CONFIG_INPUT_DRV260X_HAPTICS is not set
# CONFIG_INPUT_DRV2665_HAPTICS is not set
# CONFIG_INPUT_DRV2667_HAPTICS is not set
# CONFIG_RMI4_CORE is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_ARCH_MIGHT_HAVE_PC_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=m
CONFIG_SERIO_ALTERA_PS2=m
# CONFIG_SERIO_PS2MULT is not set
CONFIG_SERIO_ARC_PS2=m
CONFIG_HYPERV_KEYBOARD=m
# CONFIG_SERIO_GPIO_PS2 is not set
# CONFIG_USERIO is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_TTY=y
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_VT_CONSOLE_SLEEP=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_SERIAL_NONSTANDARD=y
# CONFIG_ROCKETPORT is not set
CONFIG_CYCLADES=m
# CONFIG_CYZ_INTR is not set
CONFIG_MOXA_INTELLIO=m
CONFIG_MOXA_SMARTIO=m
CONFIG_SYNCLINK=m
CONFIG_SYNCLINKMP=m
CONFIG_SYNCLINK_GT=m
CONFIG_NOZOMI=m
# CONFIG_ISI is not set
CONFIG_N_HDLC=m
CONFIG_N_GSM=m
# CONFIG_TRACE_SINK is not set
CONFIG_DEVMEM=y
# CONFIG_DEVKMEM is not set

#
# Serial drivers
#
CONFIG_SERIAL_EARLYCON=y
CONFIG_SERIAL_8250=y
# CONFIG_SERIAL_8250_DEPRECATED_OPTIONS is not set
CONFIG_SERIAL_8250_PNP=y
# CONFIG_SERIAL_8250_FINTEK is not set
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_DMA=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_EXAR=y
CONFIG_SERIAL_8250_NR_UARTS=32
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
# CONFIG_SERIAL_8250_DETECT_IRQ is not set
CONFIG_SERIAL_8250_RSA=y
# CONFIG_SERIAL_8250_FSL is not set
CONFIG_SERIAL_8250_DW=y
# CONFIG_SERIAL_8250_RT288X is not set
CONFIG_SERIAL_8250_LPSS=y
CONFIG_SERIAL_8250_MID=y
# CONFIG_SERIAL_8250_MOXA is not set

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_MAX3100 is not set
# CONFIG_SERIAL_MAX310X is not set
# CONFIG_SERIAL_UARTLITE is not set
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_SERIAL_JSM=m
# CONFIG_SERIAL_SCCNXP is not set
# CONFIG_SERIAL_SC16IS7XX is not set
# CONFIG_SERIAL_ALTERA_JTAGUART is not set
# CONFIG_SERIAL_ALTERA_UART is not set
# CONFIG_SERIAL_IFX6X60 is not set
CONFIG_SERIAL_ARC=m
CONFIG_SERIAL_ARC_NR_PORTS=1
# CONFIG_SERIAL_RP2 is not set
# CONFIG_SERIAL_FSL_LPUART is not set
# CONFIG_SERIAL_DEV_BUS is not set
# CONFIG_TTY_PRINTK is not set
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
CONFIG_PPDEV=m
CONFIG_HVC_DRIVER=y
CONFIG_HVC_IRQ=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
CONFIG_VIRTIO_CONSOLE=y
CONFIG_IPMI_HANDLER=m
CONFIG_IPMI_DMI_DECODE=y
CONFIG_IPMI_PROC_INTERFACE=y
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
# CONFIG_IPMI_SSIF is not set
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_TIMERIOMEM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_HW_RANDOM_VIA=m
CONFIG_HW_RANDOM_VIRTIO=y
CONFIG_NVRAM=y
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_MWAVE is not set
CONFIG_RAW_DRIVER=y
CONFIG_MAX_RAW_DEVS=8192
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
# CONFIG_HPET_MMAP_DEFAULT is not set
CONFIG_HANGCHECK_TIMER=m
CONFIG_UV_MMTIMER=m
CONFIG_TCG_TPM=y
CONFIG_HW_RANDOM_TPM=y
CONFIG_TCG_TIS_CORE=y
CONFIG_TCG_TIS=y
# CONFIG_TCG_TIS_SPI is not set
# CONFIG_TCG_TIS_I2C_ATMEL is not set
# CONFIG_TCG_TIS_I2C_INFINEON is not set
# CONFIG_TCG_TIS_I2C_NUVOTON is not set
CONFIG_TCG_NSC=m
CONFIG_TCG_ATMEL=m
CONFIG_TCG_INFINEON=m
# CONFIG_TCG_XEN is not set
CONFIG_TCG_CRB=y
# CONFIG_TCG_VTPM_PROXY is not set
# CONFIG_TCG_TIS_ST33ZP24_I2C is not set
# CONFIG_TCG_TIS_ST33ZP24_SPI is not set
CONFIG_TELCLOCK=m
CONFIG_DEVPORT=y
# CONFIG_XILLYBUS is not set

#
# I2C support
#
CONFIG_I2C=y
CONFIG_ACPI_I2C_OPREGION=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_COMPAT=y
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_MUX=m

#
# Multiplexer I2C Chip support
#
# CONFIG_I2C_MUX_GPIO is not set
# CONFIG_I2C_MUX_LTC4306 is not set
# CONFIG_I2C_MUX_PCA9541 is not set
# CONFIG_I2C_MUX_PCA954x is not set
# CONFIG_I2C_MUX_REG is not set
# CONFIG_I2C_MUX_MLXCPLD is not set
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_SMBUS=y
CONFIG_I2C_ALGOBIT=y
CONFIG_I2C_ALGOPCA=m

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD756_S4882=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=y
CONFIG_I2C_ISCH=m
CONFIG_I2C_ISMT=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_NFORCE2_S4985=m
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
CONFIG_I2C_SIS96X=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m

#
# ACPI drivers
#
CONFIG_I2C_SCMI=m

#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_CBUS_GPIO is not set
CONFIG_I2C_DESIGNWARE_CORE=m
CONFIG_I2C_DESIGNWARE_PLATFORM=m
# CONFIG_I2C_DESIGNWARE_SLAVE is not set
CONFIG_I2C_DESIGNWARE_PCI=m
# CONFIG_I2C_DESIGNWARE_BAYTRAIL is not set
# CONFIG_I2C_EMEV2 is not set
# CONFIG_I2C_GPIO is not set
# CONFIG_I2C_OCORES is not set
CONFIG_I2C_PCA_PLATFORM=m
# CONFIG_I2C_PXA_PCI is not set
CONFIG_I2C_SIMTEC=m
# CONFIG_I2C_XILINX is not set

#
# External I2C/SMBus adapter drivers
#
CONFIG_I2C_DIOLAN_U2C=m
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
# CONFIG_I2C_ROBOTFUZZ_OSIF is not set
# CONFIG_I2C_TAOS_EVM is not set
CONFIG_I2C_TINY_USB=m
CONFIG_I2C_VIPERBOARD=m

#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_MLXCPLD is not set
CONFIG_I2C_STUB=m
# CONFIG_I2C_SLAVE is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
CONFIG_SPI=y
# CONFIG_SPI_DEBUG is not set
CONFIG_SPI_MASTER=y

#
# SPI Master Controller Drivers
#
# CONFIG_SPI_ALTERA is not set
# CONFIG_SPI_AXI_SPI_ENGINE is not set
# CONFIG_SPI_BITBANG is not set
# CONFIG_SPI_BUTTERFLY is not set
# CONFIG_SPI_CADENCE is not set
CONFIG_SPI_DESIGNWARE=m
# CONFIG_SPI_DW_PCI is not set
# CONFIG_SPI_DW_MMIO is not set
# CONFIG_SPI_GPIO is not set
# CONFIG_SPI_LM70_LLP is not set
# CONFIG_SPI_OC_TINY is not set
CONFIG_SPI_PXA2XX=m
CONFIG_SPI_PXA2XX_PCI=m
# CONFIG_SPI_ROCKCHIP is not set
# CONFIG_SPI_SC18IS602 is not set
# CONFIG_SPI_XCOMM is not set
# CONFIG_SPI_XILINX is not set
# CONFIG_SPI_ZYNQMP_GQSPI is not set

#
# SPI Protocol Masters
#
# CONFIG_SPI_SPIDEV is not set
# CONFIG_SPI_LOOPBACK_TEST is not set
# CONFIG_SPI_TLE62X0 is not set
# CONFIG_SPI_SLAVE is not set
# CONFIG_SPMI is not set
# CONFIG_HSI is not set
CONFIG_PPS=y
# CONFIG_PPS_DEBUG is not set

#
# PPS clients support
#
# CONFIG_PPS_CLIENT_KTIMER is not set
CONFIG_PPS_CLIENT_LDISC=m
CONFIG_PPS_CLIENT_PARPORT=m
CONFIG_PPS_CLIENT_GPIO=m

#
# PPS generators support
#

#
# PTP clock support
#
CONFIG_PTP_1588_CLOCK=y
CONFIG_DP83640_PHY=m
CONFIG_PTP_1588_CLOCK_KVM=y
CONFIG_PINCTRL=y
CONFIG_PINMUX=y
CONFIG_PINCONF=y
CONFIG_GENERIC_PINCONF=y
# CONFIG_DEBUG_PINCTRL is not set
# CONFIG_PINCTRL_AMD is not set
# CONFIG_PINCTRL_MCP23S08 is not set
# CONFIG_PINCTRL_SX150X is not set
CONFIG_PINCTRL_BAYTRAIL=y
# CONFIG_PINCTRL_CHERRYVIEW is not set
# CONFIG_PINCTRL_BROXTON is not set
# CONFIG_PINCTRL_CANNONLAKE is not set
# CONFIG_PINCTRL_CEDARFORK is not set
# CONFIG_PINCTRL_DENVERTON is not set
# CONFIG_PINCTRL_GEMINILAKE is not set
# CONFIG_PINCTRL_LEWISBURG is not set
# CONFIG_PINCTRL_SUNRISEPOINT is not set
CONFIG_GPIOLIB=y
CONFIG_GPIO_ACPI=y
CONFIG_GPIOLIB_IRQCHIP=y
# CONFIG_DEBUG_GPIO is not set
CONFIG_GPIO_SYSFS=y

#
# Memory mapped GPIO drivers
#
# CONFIG_GPIO_AMDPT is not set
# CONFIG_GPIO_DWAPB is not set
# CONFIG_GPIO_EXAR is not set
# CONFIG_GPIO_GENERIC_PLATFORM is not set
# CONFIG_GPIO_ICH is not set
CONFIG_GPIO_LYNXPOINT=m
# CONFIG_GPIO_MB86S7X is not set
CONFIG_GPIO_MOCKUP=y
# CONFIG_GPIO_VX855 is not set

#
# Port-mapped I/O GPIO drivers
#
# CONFIG_GPIO_F7188X is not set
# CONFIG_GPIO_IT87 is not set
# CONFIG_GPIO_SCH is not set
# CONFIG_GPIO_SCH311X is not set

#
# I2C GPIO expanders
#
# CONFIG_GPIO_ADP5588 is not set
# CONFIG_GPIO_MAX7300 is not set
# CONFIG_GPIO_MAX732X is not set
# CONFIG_GPIO_PCA953X is not set
# CONFIG_GPIO_PCF857X is not set
# CONFIG_GPIO_TPIC2810 is not set

#
# MFD GPIO expanders
#

#
# PCI GPIO expanders
#
# CONFIG_GPIO_AMD8111 is not set
# CONFIG_GPIO_ML_IOH is not set
# CONFIG_GPIO_PCI_IDIO_16 is not set
# CONFIG_GPIO_PCIE_IDIO_24 is not set
# CONFIG_GPIO_RDC321X is not set

#
# SPI GPIO expanders
#
# CONFIG_GPIO_MAX3191X is not set
# CONFIG_GPIO_MAX7301 is not set
# CONFIG_GPIO_MC33880 is not set
# CONFIG_GPIO_PISOSR is not set
# CONFIG_GPIO_XRA1403 is not set

#
# USB GPIO expanders
#
# CONFIG_GPIO_VIPERBOARD is not set
# CONFIG_W1 is not set
# CONFIG_POWER_AVS is not set
CONFIG_POWER_RESET=y
# CONFIG_POWER_RESET_RESTART is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_TEST_POWER is not set
# CONFIG_BATTERY_DS2780 is not set
# CONFIG_BATTERY_DS2781 is not set
# CONFIG_BATTERY_DS2782 is not set
# CONFIG_BATTERY_SBS is not set
# CONFIG_CHARGER_SBS is not set
# CONFIG_MANAGER_SBS is not set
# CONFIG_BATTERY_BQ27XXX is not set
# CONFIG_BATTERY_MAX17040 is not set
# CONFIG_BATTERY_MAX17042 is not set
# CONFIG_CHARGER_ISP1704 is not set
# CONFIG_CHARGER_MAX8903 is not set
# CONFIG_CHARGER_LP8727 is not set
# CONFIG_CHARGER_GPIO is not set
# CONFIG_CHARGER_LTC3651 is not set
# CONFIG_CHARGER_BQ2415X is not set
# CONFIG_CHARGER_BQ24190 is not set
# CONFIG_CHARGER_BQ24257 is not set
# CONFIG_CHARGER_BQ24735 is not set
# CONFIG_CHARGER_BQ25890 is not set
CONFIG_CHARGER_SMB347=m
# CONFIG_BATTERY_GAUGE_LTC2941 is not set
# CONFIG_CHARGER_RT9455 is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=m
# CONFIG_HWMON_DEBUG_CHIP is not set

#
# Native drivers
#
CONFIG_SENSORS_ABITUGURU=m
CONFIG_SENSORS_ABITUGURU3=m
# CONFIG_SENSORS_AD7314 is not set
CONFIG_SENSORS_AD7414=m
CONFIG_SENSORS_AD7418=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
CONFIG_SENSORS_ADM1029=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ADM9240=m
CONFIG_SENSORS_ADT7X10=m
# CONFIG_SENSORS_ADT7310 is not set
CONFIG_SENSORS_ADT7410=m
CONFIG_SENSORS_ADT7411=m
CONFIG_SENSORS_ADT7462=m
CONFIG_SENSORS_ADT7470=m
CONFIG_SENSORS_ADT7475=m
CONFIG_SENSORS_ASC7621=m
CONFIG_SENSORS_K8TEMP=m
CONFIG_SENSORS_K10TEMP=m
CONFIG_SENSORS_FAM15H_POWER=m
CONFIG_SENSORS_APPLESMC=m
CONFIG_SENSORS_ASB100=m
# CONFIG_SENSORS_ASPEED is not set
CONFIG_SENSORS_ATXP1=m
CONFIG_SENSORS_DS620=m
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_DELL_SMM=m
CONFIG_SENSORS_I5K_AMB=m
CONFIG_SENSORS_F71805F=m
CONFIG_SENSORS_F71882FG=m
CONFIG_SENSORS_F75375S=m
CONFIG_SENSORS_FSCHMD=m
# CONFIG_SENSORS_FTSTEUTATES is not set
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
CONFIG_SENSORS_G760A=m
# CONFIG_SENSORS_G762 is not set
# CONFIG_SENSORS_HIH6130 is not set
CONFIG_SENSORS_IBMAEM=m
CONFIG_SENSORS_IBMPEX=m
# CONFIG_SENSORS_I5500 is not set
CONFIG_SENSORS_CORETEMP=m
CONFIG_SENSORS_IT87=m
# CONFIG_SENSORS_JC42 is not set
# CONFIG_SENSORS_POWR1220 is not set
CONFIG_SENSORS_LINEAGE=m
# CONFIG_SENSORS_LTC2945 is not set
# CONFIG_SENSORS_LTC2990 is not set
CONFIG_SENSORS_LTC4151=m
CONFIG_SENSORS_LTC4215=m
# CONFIG_SENSORS_LTC4222 is not set
CONFIG_SENSORS_LTC4245=m
# CONFIG_SENSORS_LTC4260 is not set
CONFIG_SENSORS_LTC4261=m
# CONFIG_SENSORS_MAX1111 is not set
CONFIG_SENSORS_MAX16065=m
CONFIG_SENSORS_MAX1619=m
CONFIG_SENSORS_MAX1668=m
CONFIG_SENSORS_MAX197=m
# CONFIG_SENSORS_MAX31722 is not set
# CONFIG_SENSORS_MAX6621 is not set
CONFIG_SENSORS_MAX6639=m
CONFIG_SENSORS_MAX6642=m
CONFIG_SENSORS_MAX6650=m
CONFIG_SENSORS_MAX6697=m
# CONFIG_SENSORS_MAX31790 is not set
CONFIG_SENSORS_MCP3021=m
# CONFIG_SENSORS_TC654 is not set
# CONFIG_SENSORS_ADCXX is not set
CONFIG_SENSORS_LM63=m
# CONFIG_SENSORS_LM70 is not set
CONFIG_SENSORS_LM73=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
CONFIG_SENSORS_LM93=m
CONFIG_SENSORS_LM95234=m
CONFIG_SENSORS_LM95241=m
CONFIG_SENSORS_LM95245=m
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_PC87427=m
CONFIG_SENSORS_NTC_THERMISTOR=m
# CONFIG_SENSORS_NCT6683 is not set
CONFIG_SENSORS_NCT6775=m
# CONFIG_SENSORS_NCT7802 is not set
# CONFIG_SENSORS_NCT7904 is not set
CONFIG_SENSORS_PCF8591=m
CONFIG_PMBUS=m
CONFIG_SENSORS_PMBUS=m
CONFIG_SENSORS_ADM1275=m
# CONFIG_SENSORS_IBM_CFFPS is not set
# CONFIG_SENSORS_IR35221 is not set
CONFIG_SENSORS_LM25066=m
CONFIG_SENSORS_LTC2978=m
# CONFIG_SENSORS_LTC3815 is not set
CONFIG_SENSORS_MAX16064=m
# CONFIG_SENSORS_MAX20751 is not set
# CONFIG_SENSORS_MAX31785 is not set
CONFIG_SENSORS_MAX34440=m
CONFIG_SENSORS_MAX8688=m
# CONFIG_SENSORS_TPS40422 is not set
# CONFIG_SENSORS_TPS53679 is not set
CONFIG_SENSORS_UCD9000=m
CONFIG_SENSORS_UCD9200=m
CONFIG_SENSORS_ZL6100=m
# CONFIG_SENSORS_SHT15 is not set
CONFIG_SENSORS_SHT21=m
# CONFIG_SENSORS_SHT3x is not set
# CONFIG_SENSORS_SHTC1 is not set
CONFIG_SENSORS_SIS5595=m
CONFIG_SENSORS_DME1737=m
CONFIG_SENSORS_EMC1403=m
# CONFIG_SENSORS_EMC2103 is not set
CONFIG_SENSORS_EMC6W201=m
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47M192=m
CONFIG_SENSORS_SMSC47B397=m
CONFIG_SENSORS_SCH56XX_COMMON=m
CONFIG_SENSORS_SCH5627=m
CONFIG_SENSORS_SCH5636=m
# CONFIG_SENSORS_STTS751 is not set
# CONFIG_SENSORS_SMM665 is not set
# CONFIG_SENSORS_ADC128D818 is not set
CONFIG_SENSORS_ADS1015=m
CONFIG_SENSORS_ADS7828=m
# CONFIG_SENSORS_ADS7871 is not set
CONFIG_SENSORS_AMC6821=m
CONFIG_SENSORS_INA209=m
CONFIG_SENSORS_INA2XX=m
# CONFIG_SENSORS_INA3221 is not set
# CONFIG_SENSORS_TC74 is not set
CONFIG_SENSORS_THMC50=m
CONFIG_SENSORS_TMP102=m
# CONFIG_SENSORS_TMP103 is not set
# CONFIG_SENSORS_TMP108 is not set
CONFIG_SENSORS_TMP401=m
CONFIG_SENSORS_TMP421=m
CONFIG_SENSORS_VIA_CPUTEMP=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_VT1211=m
CONFIG_SENSORS_VT8231=m
# CONFIG_SENSORS_W83773G is not set
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83793=m
CONFIG_SENSORS_W83795=m
# CONFIG_SENSORS_W83795_FANCTRL is not set
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83L786NG=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
# CONFIG_SENSORS_XGENE is not set

#
# ACPI drivers
#
CONFIG_SENSORS_ACPI_POWER=m
CONFIG_SENSORS_ATK0110=m
CONFIG_THERMAL=y
CONFIG_THERMAL_EMERGENCY_POWEROFF_DELAY_MS=0
CONFIG_THERMAL_HWMON=y
CONFIG_THERMAL_WRITABLE_TRIPS=y
CONFIG_THERMAL_DEFAULT_GOV_STEP_WISE=y
# CONFIG_THERMAL_DEFAULT_GOV_FAIR_SHARE is not set
# CONFIG_THERMAL_DEFAULT_GOV_USER_SPACE is not set
# CONFIG_THERMAL_DEFAULT_GOV_POWER_ALLOCATOR is not set
CONFIG_THERMAL_GOV_FAIR_SHARE=y
CONFIG_THERMAL_GOV_STEP_WISE=y
CONFIG_THERMAL_GOV_BANG_BANG=y
CONFIG_THERMAL_GOV_USER_SPACE=y
# CONFIG_THERMAL_GOV_POWER_ALLOCATOR is not set
# CONFIG_CLOCK_THERMAL is not set
# CONFIG_DEVFREQ_THERMAL is not set
# CONFIG_THERMAL_EMULATION is not set
CONFIG_INTEL_POWERCLAMP=m
CONFIG_X86_PKG_TEMP_THERMAL=m
# CONFIG_INTEL_SOC_DTS_THERMAL is not set

#
# ACPI INT340X thermal drivers
#
# CONFIG_INT340X_THERMAL is not set
CONFIG_INTEL_PCH_THERMAL=m
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_CORE=y
# CONFIG_WATCHDOG_NOWAYOUT is not set
CONFIG_WATCHDOG_HANDLE_BOOT_ENABLED=y
# CONFIG_WATCHDOG_SYSFS is not set

#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
CONFIG_WDAT_WDT=m
# CONFIG_XILINX_WATCHDOG is not set
# CONFIG_ZIIRAVE_WATCHDOG is not set
# CONFIG_CADENCE_WATCHDOG is not set
# CONFIG_DW_WATCHDOG is not set
# CONFIG_MAX63XX_WATCHDOG is not set
# CONFIG_ACQUIRE_WDT is not set
# CONFIG_ADVANTECH_WDT is not set
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=m
CONFIG_F71808E_WDT=m
CONFIG_SP5100_TCO=m
CONFIG_SBC_FITPC2_WATCHDOG=m
# CONFIG_EUROTECH_WDT is not set
CONFIG_IB700_WDT=m
CONFIG_IBMASR=m
# CONFIG_WAFER_WDT is not set
CONFIG_I6300ESB_WDT=y
CONFIG_IE6XX_WDT=m
CONFIG_ITCO_WDT=y
CONFIG_ITCO_VENDOR_SUPPORT=y
CONFIG_IT8712F_WDT=m
CONFIG_IT87_WDT=m
CONFIG_HP_WATCHDOG=m
CONFIG_HPWDT_NMI_DECODING=y
# CONFIG_SC1200_WDT is not set
# CONFIG_PC87413_WDT is not set
CONFIG_NV_TCO=m
# CONFIG_60XX_WDT is not set
# CONFIG_CPU5_WDT is not set
CONFIG_SMSC_SCH311X_WDT=m
# CONFIG_SMSC37B787_WDT is not set
CONFIG_VIA_WDT=m
CONFIG_W83627HF_WDT=m
CONFIG_W83877F_WDT=m
CONFIG_W83977F_WDT=m
CONFIG_MACHZ_WDT=m
# CONFIG_SBC_EPX_C3_WATCHDOG is not set
# CONFIG_INTEL_MEI_WDT is not set
# CONFIG_NI903X_WDT is not set
# CONFIG_NIC7018_WDT is not set
# CONFIG_MEN_A21_WDT is not set
CONFIG_XEN_WDT=m

#
# PCI-based Watchdog Cards
#
CONFIG_PCIPCWATCHDOG=m
CONFIG_WDTPCI=m

#
# USB-based Watchdog Cards
#
CONFIG_USBPCWATCHDOG=m

#
# Watchdog Pretimeout Governors
#
# CONFIG_WATCHDOG_PRETIMEOUT_GOV is not set
CONFIG_SSB_POSSIBLE=y
CONFIG_SSB=m
CONFIG_SSB_SPROM=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
# CONFIG_SSB_B43_PCI_BRIDGE is not set
CONFIG_SSB_SDIOHOST_POSSIBLE=y
CONFIG_SSB_SDIOHOST=y
# CONFIG_SSB_SILENT is not set
# CONFIG_SSB_DEBUG is not set
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y
# CONFIG_SSB_DRIVER_GPIO is not set
CONFIG_BCMA_POSSIBLE=y
CONFIG_BCMA=m
CONFIG_BCMA_HOST_PCI_POSSIBLE=y
CONFIG_BCMA_HOST_PCI=y
# CONFIG_BCMA_HOST_SOC is not set
CONFIG_BCMA_DRIVER_PCI=y
CONFIG_BCMA_DRIVER_GMAC_CMN=y
# CONFIG_BCMA_DRIVER_GPIO is not set
# CONFIG_BCMA_DEBUG is not set

#
# Multifunction device drivers
#
CONFIG_MFD_CORE=y
# CONFIG_MFD_AS3711 is not set
# CONFIG_PMIC_ADP5520 is not set
# CONFIG_MFD_AAT2870_CORE is not set
# CONFIG_MFD_BCM590XX is not set
# CONFIG_MFD_BD9571MWV is not set
# CONFIG_MFD_AXP20X_I2C is not set
# CONFIG_MFD_CROS_EC is not set
# CONFIG_PMIC_DA903X is not set
# CONFIG_MFD_DA9052_SPI is not set
# CONFIG_MFD_DA9052_I2C is not set
# CONFIG_MFD_DA9055 is not set
# CONFIG_MFD_DA9062 is not set
# CONFIG_MFD_DA9063 is not set
# CONFIG_MFD_DA9150 is not set
# CONFIG_MFD_DLN2 is not set
# CONFIG_MFD_MC13XXX_SPI is not set
# CONFIG_MFD_MC13XXX_I2C is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_HTC_I2CPLD is not set
# CONFIG_MFD_INTEL_QUARK_I2C_GPIO is not set
CONFIG_LPC_ICH=y
CONFIG_LPC_SCH=m
# CONFIG_INTEL_SOC_PMIC is not set
# CONFIG_INTEL_SOC_PMIC_CHTWC is not set
# CONFIG_INTEL_SOC_PMIC_CHTDC_TI is not set
# CONFIG_MFD_INTEL_LPSS_ACPI is not set
# CONFIG_MFD_INTEL_LPSS_PCI is not set
# CONFIG_MFD_JANZ_CMODIO is not set
# CONFIG_MFD_KEMPLD is not set
# CONFIG_MFD_88PM800 is not set
# CONFIG_MFD_88PM805 is not set
# CONFIG_MFD_88PM860X is not set
# CONFIG_MFD_MAX14577 is not set
# CONFIG_MFD_MAX77693 is not set
# CONFIG_MFD_MAX77843 is not set
# CONFIG_MFD_MAX8907 is not set
# CONFIG_MFD_MAX8925 is not set
# CONFIG_MFD_MAX8997 is not set
# CONFIG_MFD_MAX8998 is not set
# CONFIG_MFD_MT6397 is not set
# CONFIG_MFD_MENF21BMC is not set
# CONFIG_EZX_PCAP is not set
CONFIG_MFD_VIPERBOARD=m
# CONFIG_MFD_RETU is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_UCB1400_CORE is not set
# CONFIG_MFD_RDC321X is not set
# CONFIG_MFD_RT5033 is not set
# CONFIG_MFD_RC5T583 is not set
# CONFIG_MFD_SEC_CORE is not set
# CONFIG_MFD_SI476X_CORE is not set
CONFIG_MFD_SM501=m
# CONFIG_MFD_SM501_GPIO is not set
# CONFIG_MFD_SKY81452 is not set
# CONFIG_MFD_SMSC is not set
# CONFIG_ABX500_CORE is not set
# CONFIG_MFD_SYSCON is not set
# CONFIG_MFD_TI_AM335X_TSCADC is not set
# CONFIG_MFD_LP3943 is not set
# CONFIG_MFD_LP8788 is not set
# CONFIG_MFD_TI_LMU is not set
# CONFIG_MFD_PALMAS is not set
# CONFIG_TPS6105X is not set
# CONFIG_TPS65010 is not set
# CONFIG_TPS6507X is not set
# CONFIG_MFD_TPS65086 is not set
# CONFIG_MFD_TPS65090 is not set
# CONFIG_MFD_TPS68470 is not set
# CONFIG_MFD_TI_LP873X is not set
# CONFIG_MFD_TPS6586X is not set
# CONFIG_MFD_TPS65910 is not set
# CONFIG_MFD_TPS65912_I2C is not set
# CONFIG_MFD_TPS65912_SPI is not set
# CONFIG_MFD_TPS80031 is not set
# CONFIG_TWL4030_CORE is not set
# CONFIG_TWL6040_CORE is not set
# CONFIG_MFD_WL1273_CORE is not set
# CONFIG_MFD_LM3533 is not set
# CONFIG_MFD_TMIO is not set
CONFIG_MFD_VX855=m
# CONFIG_MFD_ARIZONA_I2C is not set
# CONFIG_MFD_ARIZONA_SPI is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_WM831X_I2C is not set
# CONFIG_MFD_WM831X_SPI is not set
# CONFIG_MFD_WM8350_I2C is not set
# CONFIG_MFD_WM8994 is not set
# CONFIG_REGULATOR is not set
CONFIG_RC_CORE=m
CONFIG_RC_MAP=m
# CONFIG_LIRC is not set
CONFIG_RC_DECODERS=y
CONFIG_IR_NEC_DECODER=m
CONFIG_IR_RC5_DECODER=m
CONFIG_IR_RC6_DECODER=m
CONFIG_IR_JVC_DECODER=m
CONFIG_IR_SONY_DECODER=m
CONFIG_IR_SANYO_DECODER=m
CONFIG_IR_SHARP_DECODER=m
CONFIG_IR_MCE_KBD_DECODER=m
CONFIG_IR_XMP_DECODER=m
CONFIG_RC_DEVICES=y
CONFIG_RC_ATI_REMOTE=m
CONFIG_IR_ENE=m
CONFIG_IR_IMON=m
CONFIG_IR_MCEUSB=m
CONFIG_IR_ITE_CIR=m
CONFIG_IR_FINTEK=m
CONFIG_IR_NUVOTON=m
CONFIG_IR_REDRAT3=m
CONFIG_IR_STREAMZAP=m
CONFIG_IR_WINBOND_CIR=m
# CONFIG_IR_IGORPLUGUSB is not set
CONFIG_IR_IGUANA=m
CONFIG_IR_TTUSBIR=m
CONFIG_RC_LOOPBACK=m
# CONFIG_IR_SERIAL is not set
# CONFIG_IR_SIR is not set
CONFIG_MEDIA_SUPPORT=m

#
# Multimedia core support
#
CONFIG_MEDIA_CAMERA_SUPPORT=y
CONFIG_MEDIA_ANALOG_TV_SUPPORT=y
CONFIG_MEDIA_DIGITAL_TV_SUPPORT=y
CONFIG_MEDIA_RADIO_SUPPORT=y
# CONFIG_MEDIA_SDR_SUPPORT is not set
# CONFIG_MEDIA_CEC_SUPPORT is not set
# CONFIG_MEDIA_CONTROLLER is not set
CONFIG_VIDEO_DEV=m
CONFIG_VIDEO_V4L2=m
# CONFIG_VIDEO_ADV_DEBUG is not set
# CONFIG_VIDEO_FIXED_MINOR_RANGES is not set
CONFIG_VIDEO_TUNER=m
CONFIG_VIDEOBUF_GEN=m
CONFIG_VIDEOBUF_DMA_SG=m
CONFIG_VIDEOBUF_VMALLOC=m
CONFIG_VIDEOBUF_DVB=m
CONFIG_DVB_CORE=m
# CONFIG_DVB_MMAP is not set
CONFIG_DVB_NET=y
CONFIG_TTPCI_EEPROM=m
CONFIG_DVB_MAX_ADAPTERS=8
CONFIG_DVB_DYNAMIC_MINORS=y
# CONFIG_DVB_DEMUX_SECTION_LOSS_LOG is not set
# CONFIG_DVB_ULE_DEBUG is not set

#
# Media drivers
#
CONFIG_MEDIA_USB_SUPPORT=y

#
# Webcam devices
#
CONFIG_USB_VIDEO_CLASS=m
CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y
CONFIG_USB_GSPCA=m
CONFIG_USB_M5602=m
CONFIG_USB_STV06XX=m
CONFIG_USB_GL860=m
CONFIG_USB_GSPCA_BENQ=m
CONFIG_USB_GSPCA_CONEX=m
CONFIG_USB_GSPCA_CPIA1=m
# CONFIG_USB_GSPCA_DTCS033 is not set
CONFIG_USB_GSPCA_ETOMS=m
CONFIG_USB_GSPCA_FINEPIX=m
CONFIG_USB_GSPCA_JEILINJ=m
CONFIG_USB_GSPCA_JL2005BCD=m
# CONFIG_USB_GSPCA_KINECT is not set
CONFIG_USB_GSPCA_KONICA=m
CONFIG_USB_GSPCA_MARS=m
CONFIG_USB_GSPCA_MR97310A=m
CONFIG_USB_GSPCA_NW80X=m
CONFIG_USB_GSPCA_OV519=m
CONFIG_USB_GSPCA_OV534=m
CONFIG_USB_GSPCA_OV534_9=m
CONFIG_USB_GSPCA_PAC207=m
CONFIG_USB_GSPCA_PAC7302=m
CONFIG_USB_GSPCA_PAC7311=m
CONFIG_USB_GSPCA_SE401=m
CONFIG_USB_GSPCA_SN9C2028=m
CONFIG_USB_GSPCA_SN9C20X=m
CONFIG_USB_GSPCA_SONIXB=m
CONFIG_USB_GSPCA_SONIXJ=m
CONFIG_USB_GSPCA_SPCA500=m
CONFIG_USB_GSPCA_SPCA501=m
CONFIG_USB_GSPCA_SPCA505=m
CONFIG_USB_GSPCA_SPCA506=m
CONFIG_USB_GSPCA_SPCA508=m
CONFIG_USB_GSPCA_SPCA561=m
CONFIG_USB_GSPCA_SPCA1528=m
CONFIG_USB_GSPCA_SQ905=m
CONFIG_USB_GSPCA_SQ905C=m
CONFIG_USB_GSPCA_SQ930X=m
CONFIG_USB_GSPCA_STK014=m
# CONFIG_USB_GSPCA_STK1135 is not set
CONFIG_USB_GSPCA_STV0680=m
CONFIG_USB_GSPCA_SUNPLUS=m
CONFIG_USB_GSPCA_T613=m
CONFIG_USB_GSPCA_TOPRO=m
# CONFIG_USB_GSPCA_TOUPTEK is not set
CONFIG_USB_GSPCA_TV8532=m
CONFIG_USB_GSPCA_VC032X=m
CONFIG_USB_GSPCA_VICAM=m
CONFIG_USB_GSPCA_XIRLINK_CIT=m
CONFIG_USB_GSPCA_ZC3XX=m
CONFIG_USB_PWC=m
# CONFIG_USB_PWC_DEBUG is not set
CONFIG_USB_PWC_INPUT_EVDEV=y
# CONFIG_VIDEO_CPIA2 is not set
CONFIG_USB_ZR364XX=m
CONFIG_USB_STKWEBCAM=m
CONFIG_USB_S2255=m
# CONFIG_VIDEO_USBTV is not set

#
# Analog TV USB devices
#
CONFIG_VIDEO_PVRUSB2=m
CONFIG_VIDEO_PVRUSB2_SYSFS=y
CONFIG_VIDEO_PVRUSB2_DVB=y
# CONFIG_VIDEO_PVRUSB2_DEBUGIFC is not set
CONFIG_VIDEO_HDPVR=m
CONFIG_VIDEO_USBVISION=m
# CONFIG_VIDEO_STK1160_COMMON is not set
# CONFIG_VIDEO_GO7007 is not set

#
# Analog/digital TV USB devices
#
CONFIG_VIDEO_AU0828=m
CONFIG_VIDEO_AU0828_V4L2=y
# CONFIG_VIDEO_AU0828_RC is not set
CONFIG_VIDEO_CX231XX=m
CONFIG_VIDEO_CX231XX_RC=y
CONFIG_VIDEO_CX231XX_ALSA=m
CONFIG_VIDEO_CX231XX_DVB=m
CONFIG_VIDEO_TM6000=m
CONFIG_VIDEO_TM6000_ALSA=m
CONFIG_VIDEO_TM6000_DVB=m

#
# Digital TV USB devices
#
CONFIG_DVB_USB=m
# CONFIG_DVB_USB_DEBUG is not set
CONFIG_DVB_USB_DIB3000MC=m
CONFIG_DVB_USB_A800=m
CONFIG_DVB_USB_DIBUSB_MB=m
# CONFIG_DVB_USB_DIBUSB_MB_FAULTY is not set
CONFIG_DVB_USB_DIBUSB_MC=m
CONFIG_DVB_USB_DIB0700=m
CONFIG_DVB_USB_UMT_010=m
CONFIG_DVB_USB_CXUSB=m
CONFIG_DVB_USB_M920X=m
CONFIG_DVB_USB_DIGITV=m
CONFIG_DVB_USB_VP7045=m
CONFIG_DVB_USB_VP702X=m
CONFIG_DVB_USB_GP8PSK=m
CONFIG_DVB_USB_NOVA_T_USB2=m
CONFIG_DVB_USB_TTUSB2=m
CONFIG_DVB_USB_DTT200U=m
CONFIG_DVB_USB_OPERA1=m
CONFIG_DVB_USB_AF9005=m
CONFIG_DVB_USB_AF9005_REMOTE=m
CONFIG_DVB_USB_PCTV452E=m
CONFIG_DVB_USB_DW2102=m
CONFIG_DVB_USB_CINERGY_T2=m
CONFIG_DVB_USB_DTV5100=m
CONFIG_DVB_USB_FRIIO=m
CONFIG_DVB_USB_AZ6027=m
CONFIG_DVB_USB_TECHNISAT_USB2=m
CONFIG_DVB_USB_V2=m
CONFIG_DVB_USB_AF9015=m
CONFIG_DVB_USB_AF9035=m
CONFIG_DVB_USB_ANYSEE=m
CONFIG_DVB_USB_AU6610=m
CONFIG_DVB_USB_AZ6007=m
CONFIG_DVB_USB_CE6230=m
CONFIG_DVB_USB_EC168=m
CONFIG_DVB_USB_GL861=m
CONFIG_DVB_USB_LME2510=m
CONFIG_DVB_USB_MXL111SF=m
CONFIG_DVB_USB_RTL28XXU=m
# CONFIG_DVB_USB_DVBSKY is not set
# CONFIG_DVB_USB_ZD1301 is not set
CONFIG_DVB_TTUSB_BUDGET=m
CONFIG_DVB_TTUSB_DEC=m
CONFIG_SMS_USB_DRV=m
CONFIG_DVB_B2C2_FLEXCOP_USB=m
# CONFIG_DVB_B2C2_FLEXCOP_USB_DEBUG is not set
# CONFIG_DVB_AS102 is not set

#
# Webcam, TV (analog/digital) USB devices
#
CONFIG_VIDEO_EM28XX=m
# CONFIG_VIDEO_EM28XX_V4L2 is not set
CONFIG_VIDEO_EM28XX_ALSA=m
CONFIG_VIDEO_EM28XX_DVB=m
CONFIG_VIDEO_EM28XX_RC=m
CONFIG_MEDIA_PCI_SUPPORT=y

#
# Media capture support
#
# CONFIG_VIDEO_MEYE is not set
# CONFIG_VIDEO_SOLO6X10 is not set
# CONFIG_VIDEO_TW5864 is not set
# CONFIG_VIDEO_TW68 is not set
# CONFIG_VIDEO_TW686X is not set
# CONFIG_VIDEO_ZORAN is not set

#
# Media capture/analog TV support
#
CONFIG_VIDEO_IVTV=m
# CONFIG_VIDEO_IVTV_DEPRECATED_IOCTLS is not set
# CONFIG_VIDEO_IVTV_ALSA is not set
CONFIG_VIDEO_FB_IVTV=m
# CONFIG_VIDEO_HEXIUM_GEMINI is not set
# CONFIG_VIDEO_HEXIUM_ORION is not set
# CONFIG_VIDEO_MXB is not set
# CONFIG_VIDEO_DT3155 is not set

#
# Media capture/analog/hybrid TV support
#
CONFIG_VIDEO_CX18=m
CONFIG_VIDEO_CX18_ALSA=m
CONFIG_VIDEO_CX23885=m
CONFIG_MEDIA_ALTERA_CI=m
# CONFIG_VIDEO_CX25821 is not set
CONFIG_VIDEO_CX88=m
CONFIG_VIDEO_CX88_ALSA=m
CONFIG_VIDEO_CX88_BLACKBIRD=m
CONFIG_VIDEO_CX88_DVB=m
CONFIG_VIDEO_CX88_ENABLE_VP3054=y
CONFIG_VIDEO_CX88_VP3054=m
CONFIG_VIDEO_CX88_MPEG=m
CONFIG_VIDEO_BT848=m
CONFIG_DVB_BT8XX=m
CONFIG_VIDEO_SAA7134=m
CONFIG_VIDEO_SAA7134_ALSA=m
CONFIG_VIDEO_SAA7134_RC=y
CONFIG_VIDEO_SAA7134_DVB=m
CONFIG_VIDEO_SAA7164=m

#
# Media digital TV PCI Adapters
#
CONFIG_DVB_AV7110_IR=y
CONFIG_DVB_AV7110=m
CONFIG_DVB_AV7110_OSD=y
CONFIG_DVB_BUDGET_CORE=m
CONFIG_DVB_BUDGET=m
CONFIG_DVB_BUDGET_CI=m
CONFIG_DVB_BUDGET_AV=m
CONFIG_DVB_BUDGET_PATCH=m
CONFIG_DVB_B2C2_FLEXCOP_PCI=m
# CONFIG_DVB_B2C2_FLEXCOP_PCI_DEBUG is not set
CONFIG_DVB_PLUTO2=m
CONFIG_DVB_DM1105=m
CONFIG_DVB_PT1=m
# CONFIG_DVB_PT3 is not set
CONFIG_MANTIS_CORE=m
CONFIG_DVB_MANTIS=m
CONFIG_DVB_HOPPER=m
CONFIG_DVB_NGENE=m
CONFIG_DVB_DDBRIDGE=m
# CONFIG_DVB_DDBRIDGE_MSIENABLE is not set
# CONFIG_DVB_SMIPCIE is not set
# CONFIG_DVB_NETUP_UNIDVB is not set
# CONFIG_V4L_PLATFORM_DRIVERS is not set
# CONFIG_V4L_MEM2MEM_DRIVERS is not set
# CONFIG_V4L_TEST_DRIVERS is not set
# CONFIG_DVB_PLATFORM_DRIVERS is not set

#
# Supported MMC/SDIO adapters
#
CONFIG_SMS_SDIO_DRV=m
CONFIG_RADIO_ADAPTERS=y
CONFIG_RADIO_TEA575X=m
# CONFIG_RADIO_SI470X is not set
# CONFIG_RADIO_SI4713 is not set
# CONFIG_USB_MR800 is not set
# CONFIG_USB_DSBR is not set
# CONFIG_RADIO_MAXIRADIO is not set
# CONFIG_RADIO_SHARK is not set
# CONFIG_RADIO_SHARK2 is not set
# CONFIG_USB_KEENE is not set
# CONFIG_USB_RAREMONO is not set
# CONFIG_USB_MA901 is not set
# CONFIG_RADIO_TEA5764 is not set
# CONFIG_RADIO_SAA7706H is not set
# CONFIG_RADIO_TEF6862 is not set
# CONFIG_RADIO_WL1273 is not set

#
# Texas Instruments WL128x FM driver (ST based)
#

#
# Supported FireWire (IEEE 1394) Adapters
#
CONFIG_DVB_FIREDTV=m
CONFIG_DVB_FIREDTV_INPUT=y
CONFIG_MEDIA_COMMON_OPTIONS=y

#
# common driver options
#
CONFIG_VIDEO_CX2341X=m
CONFIG_VIDEO_TVEEPROM=m
CONFIG_CYPRESS_FIRMWARE=m
CONFIG_VIDEOBUF2_CORE=m
CONFIG_VIDEOBUF2_MEMOPS=m
CONFIG_VIDEOBUF2_VMALLOC=m
CONFIG_VIDEOBUF2_DMA_SG=m
CONFIG_VIDEOBUF2_DVB=m
CONFIG_DVB_B2C2_FLEXCOP=m
CONFIG_VIDEO_SAA7146=m
CONFIG_VIDEO_SAA7146_VV=m
CONFIG_SMS_SIANO_MDTV=m
CONFIG_SMS_SIANO_RC=y
# CONFIG_SMS_SIANO_DEBUGFS is not set

#
# Media ancillary drivers (tuners, sensors, i2c, spi, frontends)
#
CONFIG_MEDIA_SUBDRV_AUTOSELECT=y
CONFIG_MEDIA_ATTACH=y
CONFIG_VIDEO_IR_I2C=m

#
# Audio decoders, processors and mixers
#
CONFIG_VIDEO_TVAUDIO=m
CONFIG_VIDEO_TDA7432=m
CONFIG_VIDEO_MSP3400=m
CONFIG_VIDEO_CS3308=m
CONFIG_VIDEO_CS5345=m
CONFIG_VIDEO_CS53L32A=m
CONFIG_VIDEO_WM8775=m
CONFIG_VIDEO_WM8739=m
CONFIG_VIDEO_VP27SMPX=m

#
# RDS decoders
#
CONFIG_VIDEO_SAA6588=m

#
# Video decoders
#
CONFIG_VIDEO_SAA711X=m

#
# Video and audio decoders
#
CONFIG_VIDEO_SAA717X=m
CONFIG_VIDEO_CX25840=m

#
# Video encoders
#
CONFIG_VIDEO_SAA7127=m

#
# Camera sensor devices
#

#
# Flash devices
#

#
# Video improvement chips
#
CONFIG_VIDEO_UPD64031A=m
CONFIG_VIDEO_UPD64083=m

#
# Audio/Video compression chips
#
CONFIG_VIDEO_SAA6752HS=m

#
# SDR tuner chips
#

#
# Miscellaneous helper chips
#
CONFIG_VIDEO_M52790=m

#
# Sensors used on soc_camera driver
#
CONFIG_MEDIA_TUNER=m
CONFIG_MEDIA_TUNER_SIMPLE=m
CONFIG_MEDIA_TUNER_TDA18250=m
CONFIG_MEDIA_TUNER_TDA8290=m
CONFIG_MEDIA_TUNER_TDA827X=m
CONFIG_MEDIA_TUNER_TDA18271=m
CONFIG_MEDIA_TUNER_TDA9887=m
CONFIG_MEDIA_TUNER_TEA5761=m
CONFIG_MEDIA_TUNER_TEA5767=m
CONFIG_MEDIA_TUNER_MT20XX=m
CONFIG_MEDIA_TUNER_MT2060=m
CONFIG_MEDIA_TUNER_MT2063=m
CONFIG_MEDIA_TUNER_MT2266=m
CONFIG_MEDIA_TUNER_MT2131=m
CONFIG_MEDIA_TUNER_QT1010=m
CONFIG_MEDIA_TUNER_XC2028=m
CONFIG_MEDIA_TUNER_XC5000=m
CONFIG_MEDIA_TUNER_XC4000=m
CONFIG_MEDIA_TUNER_MXL5005S=m
CONFIG_MEDIA_TUNER_MXL5007T=m
CONFIG_MEDIA_TUNER_MC44S803=m
CONFIG_MEDIA_TUNER_MAX2165=m
CONFIG_MEDIA_TUNER_TDA18218=m
CONFIG_MEDIA_TUNER_FC0011=m
CONFIG_MEDIA_TUNER_FC0012=m
CONFIG_MEDIA_TUNER_FC0013=m
CONFIG_MEDIA_TUNER_TDA18212=m
CONFIG_MEDIA_TUNER_E4000=m
CONFIG_MEDIA_TUNER_FC2580=m
CONFIG_MEDIA_TUNER_M88RS6000T=m
CONFIG_MEDIA_TUNER_TUA9001=m
CONFIG_MEDIA_TUNER_SI2157=m
CONFIG_MEDIA_TUNER_IT913X=m
CONFIG_MEDIA_TUNER_R820T=m
CONFIG_MEDIA_TUNER_QM1D1C0042=m

#
# Multistandard (satellite) frontends
#
CONFIG_DVB_STB0899=m
CONFIG_DVB_STB6100=m
CONFIG_DVB_STV090x=m
CONFIG_DVB_STV0910=m
CONFIG_DVB_STV6110x=m
CONFIG_DVB_STV6111=m
CONFIG_DVB_MXL5XX=m
CONFIG_DVB_M88DS3103=m

#
# Multistandard (cable + terrestrial) frontends
#
CONFIG_DVB_DRXK=m
CONFIG_DVB_TDA18271C2DD=m
CONFIG_DVB_SI2165=m
CONFIG_DVB_MN88472=m
CONFIG_DVB_MN88473=m

#
# DVB-S (satellite) frontends
#
CONFIG_DVB_CX24110=m
CONFIG_DVB_CX24123=m
CONFIG_DVB_MT312=m
CONFIG_DVB_ZL10036=m
CONFIG_DVB_ZL10039=m
CONFIG_DVB_S5H1420=m
CONFIG_DVB_STV0288=m
CONFIG_DVB_STB6000=m
CONFIG_DVB_STV0299=m
CONFIG_DVB_STV6110=m
CONFIG_DVB_STV0900=m
CONFIG_DVB_TDA8083=m
CONFIG_DVB_TDA10086=m
CONFIG_DVB_TDA8261=m
CONFIG_DVB_VES1X93=m
CONFIG_DVB_TUNER_ITD1000=m
CONFIG_DVB_TUNER_CX24113=m
CONFIG_DVB_TDA826X=m
CONFIG_DVB_TUA6100=m
CONFIG_DVB_CX24116=m
CONFIG_DVB_CX24117=m
CONFIG_DVB_CX24120=m
CONFIG_DVB_SI21XX=m
CONFIG_DVB_TS2020=m
CONFIG_DVB_DS3000=m
CONFIG_DVB_MB86A16=m
CONFIG_DVB_TDA10071=m

#
# DVB-T (terrestrial) frontends
#
CONFIG_DVB_SP8870=m
CONFIG_DVB_SP887X=m
CONFIG_DVB_CX22700=m
CONFIG_DVB_CX22702=m
CONFIG_DVB_DRXD=m
CONFIG_DVB_L64781=m
CONFIG_DVB_TDA1004X=m
CONFIG_DVB_NXT6000=m
CONFIG_DVB_MT352=m
CONFIG_DVB_ZL10353=m
CONFIG_DVB_DIB3000MB=m
CONFIG_DVB_DIB3000MC=m
CONFIG_DVB_DIB7000M=m
CONFIG_DVB_DIB7000P=m
CONFIG_DVB_TDA10048=m
CONFIG_DVB_AF9013=m
CONFIG_DVB_EC100=m
CONFIG_DVB_STV0367=m
CONFIG_DVB_CXD2820R=m
CONFIG_DVB_CXD2841ER=m
CONFIG_DVB_RTL2830=m
CONFIG_DVB_RTL2832=m
CONFIG_DVB_SI2168=m
# CONFIG_DVB_AS102_FE is not set
CONFIG_DVB_GP8PSK_FE=m

#
# DVB-C (cable) frontends
#
CONFIG_DVB_VES1820=m
CONFIG_DVB_TDA10021=m
CONFIG_DVB_TDA10023=m
CONFIG_DVB_STV0297=m

#
# ATSC (North American/Korean Terrestrial/Cable DTV) frontends
#
CONFIG_DVB_NXT200X=m
CONFIG_DVB_OR51211=m
CONFIG_DVB_OR51132=m
CONFIG_DVB_BCM3510=m
CONFIG_DVB_LGDT330X=m
CONFIG_DVB_LGDT3305=m
CONFIG_DVB_LGDT3306A=m
CONFIG_DVB_LG2160=m
CONFIG_DVB_S5H1409=m
CONFIG_DVB_AU8522=m
CONFIG_DVB_AU8522_DTV=m
CONFIG_DVB_AU8522_V4L=m
CONFIG_DVB_S5H1411=m

#
# ISDB-T (terrestrial) frontends
#
CONFIG_DVB_S921=m
CONFIG_DVB_DIB8000=m
CONFIG_DVB_MB86A20S=m

#
# ISDB-S (satellite) & ISDB-T (terrestrial) frontends
#
CONFIG_DVB_TC90522=m

#
# Digital terrestrial only tuners/PLL
#
CONFIG_DVB_PLL=m
CONFIG_DVB_TUNER_DIB0070=m
CONFIG_DVB_TUNER_DIB0090=m

#
# SEC control devices for DVB-S
#
CONFIG_DVB_DRX39XYJ=m
CONFIG_DVB_LNBH25=m
CONFIG_DVB_LNBP21=m
CONFIG_DVB_LNBP22=m
CONFIG_DVB_ISL6405=m
CONFIG_DVB_ISL6421=m
CONFIG_DVB_ISL6423=m
CONFIG_DVB_A8293=m
CONFIG_DVB_LGS8GXX=m
CONFIG_DVB_ATBM8830=m
CONFIG_DVB_TDA665x=m
CONFIG_DVB_IX2505V=m
CONFIG_DVB_M88RS2000=m
CONFIG_DVB_AF9033=m

#
# Tools to develop new frontends
#
# CONFIG_DVB_DUMMY_FE is not set

#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_AGP_SIS=y
CONFIG_AGP_VIA=y
CONFIG_INTEL_GTT=y
CONFIG_VGA_ARB=y
CONFIG_VGA_ARB_MAX_GPUS=64
CONFIG_VGA_SWITCHEROO=y
CONFIG_DRM=m
CONFIG_DRM_MIPI_DSI=y
# CONFIG_DRM_DP_AUX_CHARDEV is not set
# CONFIG_DRM_DEBUG_MM_SELFTEST is not set
CONFIG_DRM_KMS_HELPER=m
CONFIG_DRM_KMS_FB_HELPER=y
CONFIG_DRM_FBDEV_EMULATION=y
CONFIG_DRM_FBDEV_OVERALLOC=100
CONFIG_DRM_LOAD_EDID_FIRMWARE=y
CONFIG_DRM_TTM=m

#
# I2C encoder or helper chips
#
CONFIG_DRM_I2C_CH7006=m
CONFIG_DRM_I2C_SIL164=m
CONFIG_DRM_I2C_NXP_TDA998X=m
# CONFIG_DRM_RADEON is not set
# CONFIG_DRM_AMDGPU is not set

#
# ACP (Audio CoProcessor) Configuration
#

#
# AMD Library routines
#
# CONFIG_CHASH is not set
# CONFIG_DRM_NOUVEAU is not set
CONFIG_DRM_I915=m
# CONFIG_DRM_I915_ALPHA_SUPPORT is not set
CONFIG_DRM_I915_CAPTURE_ERROR=y
CONFIG_DRM_I915_COMPRESS_ERROR=y
CONFIG_DRM_I915_USERPTR=y
# CONFIG_DRM_I915_GVT is not set

#
# drm/i915 Debugging
#
# CONFIG_DRM_I915_WERROR is not set
# CONFIG_DRM_I915_DEBUG is not set
# CONFIG_DRM_I915_SW_FENCE_DEBUG_OBJECTS is not set
# CONFIG_DRM_I915_SW_FENCE_CHECK_DAG is not set
# CONFIG_DRM_I915_SELFTEST is not set
# CONFIG_DRM_I915_LOW_LEVEL_TRACEPOINTS is not set
# CONFIG_DRM_I915_DEBUG_VBLANK_EVADE is not set
CONFIG_DRM_VGEM=m
CONFIG_DRM_VMWGFX=m
CONFIG_DRM_VMWGFX_FBCON=y
CONFIG_DRM_GMA500=m
CONFIG_DRM_GMA600=y
CONFIG_DRM_GMA3600=y
CONFIG_DRM_UDL=m
CONFIG_DRM_AST=m
CONFIG_DRM_MGAG200=m
CONFIG_DRM_CIRRUS_QEMU=m
CONFIG_DRM_QXL=m
# CONFIG_DRM_BOCHS is not set
# CONFIG_DRM_VIRTIO_GPU is not set
CONFIG_DRM_PANEL=y

#
# Display Panels
#
# CONFIG_DRM_PANEL_RASPBERRYPI_TOUCHSCREEN is not set
CONFIG_DRM_BRIDGE=y
CONFIG_DRM_PANEL_BRIDGE=y

#
# Display Interface Bridges
#
# CONFIG_DRM_ANALOGIX_ANX78XX is not set
# CONFIG_DRM_HISI_HIBMC is not set
# CONFIG_DRM_TINYDRM is not set
# CONFIG_DRM_LEGACY is not set
CONFIG_DRM_PANEL_ORIENTATION_QUIRKS=y
# CONFIG_DRM_LIB_RANDOM is not set

#
# Frame buffer Devices
#
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
CONFIG_FB_CMDLINE=y
CONFIG_FB_NOTIFY=y
# CONFIG_FB_DDC is not set
CONFIG_FB_BOOT_VESA_SUPPORT=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
CONFIG_FB_SYS_FILLRECT=m
CONFIG_FB_SYS_COPYAREA=m
CONFIG_FB_SYS_IMAGEBLIT=m
# CONFIG_FB_PROVIDE_GET_FB_UNMAPPED_AREA is not set
# CONFIG_FB_FOREIGN_ENDIAN is not set
CONFIG_FB_SYS_FOPS=m
CONFIG_FB_DEFERRED_IO=y
# CONFIG_FB_SVGALIB is not set
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
# CONFIG_FB_MODE_HELPERS is not set
CONFIG_FB_TILEBLITTING=y

#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_UVESA is not set
CONFIG_FB_VESA=y
CONFIG_FB_EFI=y
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_OPENCORES is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_I740 is not set
# CONFIG_FB_LE80578 is not set
# CONFIG_FB_INTEL is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_VIA is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_VT8623 is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_SM501 is not set
# CONFIG_FB_SMSCUFX is not set
# CONFIG_FB_UDL is not set
# CONFIG_FB_IBM_GXT4500 is not set
# CONFIG_FB_VIRTUAL is not set
# CONFIG_XEN_FBDEV_FRONTEND is not set
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
# CONFIG_FB_BROADSHEET is not set
# CONFIG_FB_AUO_K190X is not set
CONFIG_FB_HYPERV=m
# CONFIG_FB_SIMPLE is not set
# CONFIG_FB_SM712 is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=m
# CONFIG_LCD_L4F00242T03 is not set
# CONFIG_LCD_LMS283GF05 is not set
# CONFIG_LCD_LTV350QV is not set
# CONFIG_LCD_ILI922X is not set
# CONFIG_LCD_ILI9320 is not set
# CONFIG_LCD_TDO24M is not set
# CONFIG_LCD_VGG2432A4 is not set
CONFIG_LCD_PLATFORM=m
# CONFIG_LCD_S6E63M0 is not set
# CONFIG_LCD_LD9040 is not set
# CONFIG_LCD_AMS369FG06 is not set
# CONFIG_LCD_LMS501KF03 is not set
# CONFIG_LCD_HX8357 is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
# CONFIG_BACKLIGHT_GENERIC is not set
# CONFIG_BACKLIGHT_PWM is not set
CONFIG_BACKLIGHT_APPLE=m
# CONFIG_BACKLIGHT_PM8941_WLED is not set
# CONFIG_BACKLIGHT_SAHARA is not set
# CONFIG_BACKLIGHT_ADP8860 is not set
# CONFIG_BACKLIGHT_ADP8870 is not set
# CONFIG_BACKLIGHT_LM3630A is not set
# CONFIG_BACKLIGHT_LM3639 is not set
# CONFIG_BACKLIGHT_LP855X is not set
# CONFIG_BACKLIGHT_GPIO is not set
# CONFIG_BACKLIGHT_LV5207LP is not set
# CONFIG_BACKLIGHT_BD6107 is not set
# CONFIG_BACKLIGHT_ARCXCNN is not set
# CONFIG_VGASTATE is not set
CONFIG_HDMI=y

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
# CONFIG_VGACON_SOFT_SCROLLBACK_PERSISTENT_ENABLE_BY_DEFAULT is not set
CONFIG_DUMMY_CONSOLE=y
CONFIG_DUMMY_CONSOLE_COLUMNS=80
CONFIG_DUMMY_CONSOLE_ROWS=25
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y
CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=m
CONFIG_SOUND_OSS_CORE=y
CONFIG_SOUND_OSS_CORE_PRECLAIM=y
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_SEQ_DEVICE=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_JACK=y
CONFIG_SND_JACK_INPUT_DEV=y
CONFIG_SND_OSSEMUL=y
# CONFIG_SND_MIXER_OSS is not set
# CONFIG_SND_PCM_OSS is not set
CONFIG_SND_PCM_TIMER=y
CONFIG_SND_HRTIMER=m
CONFIG_SND_DYNAMIC_MINORS=y
CONFIG_SND_MAX_CARDS=32
# CONFIG_SND_SUPPORT_OLD_API is not set
CONFIG_SND_PROC_FS=y
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
CONFIG_SND_DMA_SGBUF=y
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_SEQUENCER_OSS=m
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_SEQ_MIDI_EVENT=m
CONFIG_SND_SEQ_MIDI=m
CONFIG_SND_SEQ_MIDI_EMUL=m
CONFIG_SND_SEQ_VIRMIDI=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_OPL3_LIB=m
CONFIG_SND_OPL3_LIB_SEQ=m
# CONFIG_SND_OPL4_LIB_SEQ is not set
CONFIG_SND_VX_LIB=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DRIVERS=y
CONFIG_SND_PCSP=m
CONFIG_SND_DUMMY=m
CONFIG_SND_ALOOP=m
CONFIG_SND_VIRMIDI=m
CONFIG_SND_MTPAV=m
# CONFIG_SND_MTS64 is not set
# CONFIG_SND_SERIAL_U16550 is not set
CONFIG_SND_MPU401=m
# CONFIG_SND_PORTMAN2X4 is not set
CONFIG_SND_AC97_POWER_SAVE=y
CONFIG_SND_AC97_POWER_SAVE_DEFAULT=5
CONFIG_SND_PCI=y
CONFIG_SND_AD1889=m
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
CONFIG_SND_ALI5451=m
CONFIG_SND_ASIHPI=m
CONFIG_SND_ATIIXP=m
CONFIG_SND_ATIIXP_MODEM=m
CONFIG_SND_AU8810=m
CONFIG_SND_AU8820=m
CONFIG_SND_AU8830=m
# CONFIG_SND_AW2 is not set
# CONFIG_SND_AZT3328 is not set
CONFIG_SND_BT87X=m
# CONFIG_SND_BT87X_OVERCLOCK is not set
CONFIG_SND_CA0106=m
CONFIG_SND_CMIPCI=m
CONFIG_SND_OXYGEN_LIB=m
CONFIG_SND_OXYGEN=m
# CONFIG_SND_CS4281 is not set
CONFIG_SND_CS46XX=m
CONFIG_SND_CS46XX_NEW_DSP=y
CONFIG_SND_CTXFI=m
CONFIG_SND_DARLA20=m
CONFIG_SND_GINA20=m
CONFIG_SND_LAYLA20=m
CONFIG_SND_DARLA24=m
CONFIG_SND_GINA24=m
CONFIG_SND_LAYLA24=m
CONFIG_SND_MONA=m
CONFIG_SND_MIA=m
CONFIG_SND_ECHO3G=m
CONFIG_SND_INDIGO=m
CONFIG_SND_INDIGOIO=m
CONFIG_SND_INDIGODJ=m
CONFIG_SND_INDIGOIOX=m
CONFIG_SND_INDIGODJX=m
CONFIG_SND_EMU10K1=m
CONFIG_SND_EMU10K1_SEQ=m
CONFIG_SND_EMU10K1X=m
CONFIG_SND_ENS1370=m
CONFIG_SND_ENS1371=m
# CONFIG_SND_ES1938 is not set
CONFIG_SND_ES1968=m
CONFIG_SND_ES1968_INPUT=y
CONFIG_SND_ES1968_RADIO=y
# CONFIG_SND_FM801 is not set
CONFIG_SND_HDSP=m
CONFIG_SND_HDSPM=m
CONFIG_SND_ICE1712=m
CONFIG_SND_ICE1724=m
CONFIG_SND_INTEL8X0=m
CONFIG_SND_INTEL8X0M=m
CONFIG_SND_KORG1212=m
CONFIG_SND_LOLA=m
CONFIG_SND_LX6464ES=m
CONFIG_SND_MAESTRO3=m
CONFIG_SND_MAESTRO3_INPUT=y
CONFIG_SND_MIXART=m
# CONFIG_SND_NM256 is not set
CONFIG_SND_PCXHR=m
# CONFIG_SND_RIPTIDE is not set
CONFIG_SND_RME32=m
CONFIG_SND_RME96=m
CONFIG_SND_RME9652=m
# CONFIG_SND_SONICVIBES is not set
CONFIG_SND_TRIDENT=m
CONFIG_SND_VIA82XX=m
CONFIG_SND_VIA82XX_MODEM=m
CONFIG_SND_VIRTUOSO=m
CONFIG_SND_VX222=m
# CONFIG_SND_YMFPCI is not set

#
# HD-Audio
#
CONFIG_SND_HDA=m
CONFIG_SND_HDA_INTEL=m
CONFIG_SND_HDA_HWDEP=y
# CONFIG_SND_HDA_RECONFIG is not set
CONFIG_SND_HDA_INPUT_BEEP=y
CONFIG_SND_HDA_INPUT_BEEP_MODE=0
# CONFIG_SND_HDA_PATCH_LOADER is not set
CONFIG_SND_HDA_CODEC_REALTEK=m
CONFIG_SND_HDA_CODEC_ANALOG=m
CONFIG_SND_HDA_CODEC_SIGMATEL=m
CONFIG_SND_HDA_CODEC_VIA=m
CONFIG_SND_HDA_CODEC_HDMI=m
CONFIG_SND_HDA_CODEC_CIRRUS=m
CONFIG_SND_HDA_CODEC_CONEXANT=m
CONFIG_SND_HDA_CODEC_CA0110=m
CONFIG_SND_HDA_CODEC_CA0132=m
CONFIG_SND_HDA_CODEC_CA0132_DSP=y
CONFIG_SND_HDA_CODEC_CMEDIA=m
CONFIG_SND_HDA_CODEC_SI3054=m
CONFIG_SND_HDA_GENERIC=m
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=0
CONFIG_SND_HDA_CORE=m
CONFIG_SND_HDA_DSP_LOADER=y
CONFIG_SND_HDA_I915=y
CONFIG_SND_HDA_PREALLOC_SIZE=512
CONFIG_SND_SPI=y
CONFIG_SND_USB=y
CONFIG_SND_USB_AUDIO=m
CONFIG_SND_USB_UA101=m
CONFIG_SND_USB_USX2Y=m
CONFIG_SND_USB_CAIAQ=m
CONFIG_SND_USB_CAIAQ_INPUT=y
CONFIG_SND_USB_US122L=m
CONFIG_SND_USB_6FIRE=m
# CONFIG_SND_USB_HIFACE is not set
# CONFIG_SND_BCD2000 is not set
# CONFIG_SND_USB_POD is not set
# CONFIG_SND_USB_PODHD is not set
# CONFIG_SND_USB_TONEPORT is not set
# CONFIG_SND_USB_VARIAX is not set
CONFIG_SND_FIREWIRE=y
CONFIG_SND_FIREWIRE_LIB=m
# CONFIG_SND_DICE is not set
# CONFIG_SND_OXFW is not set
CONFIG_SND_ISIGHT=m
# CONFIG_SND_FIREWORKS is not set
# CONFIG_SND_BEBOB is not set
# CONFIG_SND_FIREWIRE_DIGI00X is not set
# CONFIG_SND_FIREWIRE_TASCAM is not set
# CONFIG_SND_FIREWIRE_MOTU is not set
# CONFIG_SND_FIREFACE is not set
# CONFIG_SND_SOC is not set
CONFIG_SND_X86=y
# CONFIG_HDMI_LPE_AUDIO is not set
CONFIG_SND_SYNTH_EMUX=m
CONFIG_AC97_BUS=m

#
# HID support
#
CONFIG_HID=y
CONFIG_HID_BATTERY_STRENGTH=y
CONFIG_HIDRAW=y
CONFIG_UHID=m
CONFIG_HID_GENERIC=y

#
# Special HID drivers
#
CONFIG_HID_A4TECH=y
# CONFIG_HID_ACCUTOUCH is not set
CONFIG_HID_ACRUX=m
# CONFIG_HID_ACRUX_FF is not set
CONFIG_HID_APPLE=y
CONFIG_HID_APPLEIR=m
# CONFIG_HID_ASUS is not set
CONFIG_HID_AUREAL=m
CONFIG_HID_BELKIN=y
# CONFIG_HID_BETOP_FF is not set
CONFIG_HID_CHERRY=y
CONFIG_HID_CHICONY=y
# CONFIG_HID_CORSAIR is not set
CONFIG_HID_PRODIKEYS=m
# CONFIG_HID_CMEDIA is not set
# CONFIG_HID_CP2112 is not set
CONFIG_HID_CYPRESS=y
CONFIG_HID_DRAGONRISE=m
# CONFIG_DRAGONRISE_FF is not set
# CONFIG_HID_EMS_FF is not set
CONFIG_HID_ELECOM=m
# CONFIG_HID_ELO is not set
CONFIG_HID_EZKEY=y
# CONFIG_HID_GEMBIRD is not set
# CONFIG_HID_GFRM is not set
CONFIG_HID_HOLTEK=m
# CONFIG_HOLTEK_FF is not set
# CONFIG_HID_GT683R is not set
CONFIG_HID_KEYTOUCH=m
CONFIG_HID_KYE=m
CONFIG_HID_UCLOGIC=m
CONFIG_HID_WALTOP=m
CONFIG_HID_GYRATION=m
CONFIG_HID_ICADE=m
# CONFIG_HID_ITE is not set
# CONFIG_HID_JABRA is not set
CONFIG_HID_TWINHAN=m
CONFIG_HID_KENSINGTON=y
CONFIG_HID_LCPOWER=m
CONFIG_HID_LED=m
# CONFIG_HID_LENOVO is not set
CONFIG_HID_LOGITECH=y
CONFIG_HID_LOGITECH_DJ=m
CONFIG_HID_LOGITECH_HIDPP=m
# CONFIG_LOGITECH_FF is not set
# CONFIG_LOGIRUMBLEPAD2_FF is not set
# CONFIG_LOGIG940_FF is not set
# CONFIG_LOGIWHEELS_FF is not set
CONFIG_HID_MAGICMOUSE=y
# CONFIG_HID_MAYFLASH is not set
CONFIG_HID_MICROSOFT=y
CONFIG_HID_MONTEREY=y
CONFIG_HID_MULTITOUCH=m
# CONFIG_HID_NTI is not set
CONFIG_HID_NTRIG=y
CONFIG_HID_ORTEK=m
CONFIG_HID_PANTHERLORD=m
# CONFIG_PANTHERLORD_FF is not set
# CONFIG_HID_PENMOUNT is not set
CONFIG_HID_PETALYNX=m
CONFIG_HID_PICOLCD=m
CONFIG_HID_PICOLCD_FB=y
CONFIG_HID_PICOLCD_BACKLIGHT=y
CONFIG_HID_PICOLCD_LCD=y
CONFIG_HID_PICOLCD_LEDS=y
CONFIG_HID_PICOLCD_CIR=y
CONFIG_HID_PLANTRONICS=y
CONFIG_HID_PRIMAX=m
# CONFIG_HID_RETRODE is not set
CONFIG_HID_ROCCAT=m
CONFIG_HID_SAITEK=m
CONFIG_HID_SAMSUNG=m
CONFIG_HID_SONY=m
# CONFIG_SONY_FF is not set
CONFIG_HID_SPEEDLINK=m
CONFIG_HID_STEELSERIES=m
CONFIG_HID_SUNPLUS=m
# CONFIG_HID_RMI is not set
CONFIG_HID_GREENASIA=m
# CONFIG_GREENASIA_FF is not set
CONFIG_HID_HYPERV_MOUSE=m
CONFIG_HID_SMARTJOYPLUS=m
# CONFIG_SMARTJOYPLUS_FF is not set
CONFIG_HID_TIVO=m
CONFIG_HID_TOPSEED=m
CONFIG_HID_THINGM=m
CONFIG_HID_THRUSTMASTER=m
# CONFIG_THRUSTMASTER_FF is not set
# CONFIG_HID_UDRAW_PS3 is not set
CONFIG_HID_WACOM=m
CONFIG_HID_WIIMOTE=m
# CONFIG_HID_XINMO is not set
CONFIG_HID_ZEROPLUS=m
# CONFIG_ZEROPLUS_FF is not set
CONFIG_HID_ZYDACRON=m
# CONFIG_HID_SENSOR_HUB is not set
# CONFIG_HID_ALPS is not set

#
# USB HID support
#
CONFIG_USB_HID=y
CONFIG_HID_PID=y
CONFIG_USB_HIDDEV=y

#
# I2C HID support
#
CONFIG_I2C_HID=m

#
# Intel ISH HID support
#
# CONFIG_INTEL_ISH_HID is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_COMMON=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB=y
CONFIG_USB_PCI=y
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y

#
# Miscellaneous USB options
#
CONFIG_USB_DEFAULT_PERSIST=y
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_OTG is not set
# CONFIG_USB_OTG_WHITELIST is not set
# CONFIG_USB_OTG_BLACKLIST_HUB is not set
# CONFIG_USB_LEDS_TRIGGER_USBPORT is not set
CONFIG_USB_MON=y
CONFIG_USB_WUSB=m
CONFIG_USB_WUSB_CBAF=m
# CONFIG_USB_WUSB_CBAF_DEBUG is not set

#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_XHCI_HCD=y
# CONFIG_USB_XHCI_DBGCAP is not set
CONFIG_USB_XHCI_PCI=y
CONFIG_USB_XHCI_PLATFORM=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_USB_EHCI_PCI=y
# CONFIG_USB_EHCI_HCD_PLATFORM is not set
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1362_HCD is not set
# CONFIG_USB_FOTG210_HCD is not set
# CONFIG_USB_MAX3421_HCD is not set
CONFIG_USB_OHCI_HCD=y
CONFIG_USB_OHCI_HCD_PCI=y
# CONFIG_USB_OHCI_HCD_PLATFORM is not set
CONFIG_USB_UHCI_HCD=y
# CONFIG_USB_U132_HCD is not set
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_WHCI_HCD is not set
CONFIG_USB_HWA_HCD=m
# CONFIG_USB_HCD_BCMA is not set
# CONFIG_USB_HCD_SSB is not set
# CONFIG_USB_HCD_TEST_MODE is not set

#
# USB Device Class drivers
#
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
CONFIG_USB_WDM=m
CONFIG_USB_TMC=m

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#

#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
CONFIG_USB_STORAGE_REALTEK=m
CONFIG_REALTEK_AUTOPM=y
CONFIG_USB_STORAGE_DATAFAB=m
CONFIG_USB_STORAGE_FREECOM=m
CONFIG_USB_STORAGE_ISD200=m
CONFIG_USB_STORAGE_USBAT=m
CONFIG_USB_STORAGE_SDDR09=m
CONFIG_USB_STORAGE_SDDR55=m
CONFIG_USB_STORAGE_JUMPSHOT=m
CONFIG_USB_STORAGE_ALAUDA=m
CONFIG_USB_STORAGE_ONETOUCH=m
CONFIG_USB_STORAGE_KARMA=m
CONFIG_USB_STORAGE_CYPRESS_ATACB=m
CONFIG_USB_STORAGE_ENE_UB6250=m
CONFIG_USB_UAS=m

#
# USB Imaging devices
#
CONFIG_USB_MDC800=m
CONFIG_USB_MICROTEK=m
# CONFIG_USBIP_CORE is not set
# CONFIG_USB_MUSB_HDRC is not set
CONFIG_USB_DWC3=y
# CONFIG_USB_DWC3_HOST is not set
CONFIG_USB_DWC3_GADGET=y
# CONFIG_USB_DWC3_DUAL_ROLE is not set

#
# Platform Glue Driver Support
#
CONFIG_USB_DWC3_PCI=y
# CONFIG_USB_DWC2 is not set
# CONFIG_USB_CHIPIDEA is not set
# CONFIG_USB_ISP1760 is not set

#
# USB port drivers
#
CONFIG_USB_USS720=m
CONFIG_USB_SERIAL=y
CONFIG_USB_SERIAL_CONSOLE=y
CONFIG_USB_SERIAL_GENERIC=y
# CONFIG_USB_SERIAL_SIMPLE is not set
CONFIG_USB_SERIAL_AIRCABLE=m
CONFIG_USB_SERIAL_ARK3116=m
CONFIG_USB_SERIAL_BELKIN=m
CONFIG_USB_SERIAL_CH341=m
CONFIG_USB_SERIAL_WHITEHEAT=m
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
CONFIG_USB_SERIAL_CP210X=m
CONFIG_USB_SERIAL_CYPRESS_M8=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
CONFIG_USB_SERIAL_IR=m
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
# CONFIG_USB_SERIAL_F81232 is not set
# CONFIG_USB_SERIAL_F8153X is not set
CONFIG_USB_SERIAL_GARMIN=m
CONFIG_USB_SERIAL_IPW=m
CONFIG_USB_SERIAL_IUU=m
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
# CONFIG_USB_SERIAL_METRO is not set
CONFIG_USB_SERIAL_MOS7720=m
CONFIG_USB_SERIAL_MOS7715_PARPORT=y
CONFIG_USB_SERIAL_MOS7840=m
# CONFIG_USB_SERIAL_MXUPORT is not set
CONFIG_USB_SERIAL_NAVMAN=m
CONFIG_USB_SERIAL_PL2303=m
CONFIG_USB_SERIAL_OTI6858=m
CONFIG_USB_SERIAL_QCAUX=m
CONFIG_USB_SERIAL_QUALCOMM=m
CONFIG_USB_SERIAL_SPCP8X5=m
CONFIG_USB_SERIAL_SAFE=m
CONFIG_USB_SERIAL_SAFE_PADDED=y
CONFIG_USB_SERIAL_SIERRAWIRELESS=m
CONFIG_USB_SERIAL_SYMBOL=m
# CONFIG_USB_SERIAL_TI is not set
CONFIG_USB_SERIAL_CYBERJACK=m
CONFIG_USB_SERIAL_XIRCOM=m
CONFIG_USB_SERIAL_WWAN=m
CONFIG_USB_SERIAL_OPTION=m
CONFIG_USB_SERIAL_OMNINET=m
CONFIG_USB_SERIAL_OPTICON=m
CONFIG_USB_SERIAL_XSENS_MT=m
# CONFIG_USB_SERIAL_WISHBONE is not set
CONFIG_USB_SERIAL_SSU100=m
CONFIG_USB_SERIAL_QT2=m
# CONFIG_USB_SERIAL_UPD78F0730 is not set
CONFIG_USB_SERIAL_DEBUG=m

#
# USB Miscellaneous drivers
#
CONFIG_USB_EMI62=m
CONFIG_USB_EMI26=m
CONFIG_USB_ADUTUX=m
CONFIG_USB_SEVSEG=m
# CONFIG_USB_RIO500 is not set
CONFIG_USB_LEGOTOWER=m
CONFIG_USB_LCD=m
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
CONFIG_USB_IDMOUSE=m
CONFIG_USB_FTDI_ELAN=m
CONFIG_USB_APPLEDISPLAY=m
CONFIG_USB_SISUSBVGA=m
CONFIG_USB_SISUSBVGA_CON=y
CONFIG_USB_LD=m
# CONFIG_USB_TRANCEVIBRATOR is not set
CONFIG_USB_IOWARRIOR=m
# CONFIG_USB_TEST is not set
# CONFIG_USB_EHSET_TEST_FIXTURE is not set
CONFIG_USB_ISIGHTFW=m
# CONFIG_USB_YUREX is not set
CONFIG_USB_EZUSB_FX2=m
# CONFIG_USB_HUB_USB251XB is not set
CONFIG_USB_HSIC_USB3503=m
# CONFIG_USB_HSIC_USB4604 is not set
# CONFIG_USB_LINK_LAYER_TEST is not set
# CONFIG_USB_CHAOSKEY is not set
CONFIG_USB_ATM=m
CONFIG_USB_SPEEDTOUCH=m
CONFIG_USB_CXACRU=m
CONFIG_USB_UEAGLEATM=m
CONFIG_USB_XUSBATM=m

#
# USB Physical Layer drivers
#
CONFIG_USB_PHY=y
CONFIG_NOP_USB_XCEIV=y
# CONFIG_USB_GPIO_VBUS is not set
# CONFIG_USB_ISP1301 is not set
CONFIG_USB_GADGET=y
# CONFIG_USB_GADGET_DEBUG is not set
# CONFIG_USB_GADGET_DEBUG_FILES is not set
# CONFIG_USB_GADGET_DEBUG_FS is not set
CONFIG_USB_GADGET_VBUS_DRAW=2
CONFIG_USB_GADGET_STORAGE_NUM_BUFFERS=2

#
# USB Peripheral Controller
#
# CONFIG_USB_FOTG210_UDC is not set
# CONFIG_USB_GR_UDC is not set
# CONFIG_USB_R8A66597 is not set
# CONFIG_USB_PXA27X is not set
# CONFIG_USB_MV_UDC is not set
# CONFIG_USB_MV_U3D is not set
# CONFIG_USB_M66592 is not set
# CONFIG_USB_BDC_UDC is not set
# CONFIG_USB_AMD5536UDC is not set
# CONFIG_USB_NET2272 is not set
# CONFIG_USB_NET2280 is not set
# CONFIG_USB_GOKU is not set
# CONFIG_USB_EG20T is not set
# CONFIG_USB_DUMMY_HCD is not set
CONFIG_USB_LIBCOMPOSITE=m
CONFIG_USB_F_MASS_STORAGE=m
# CONFIG_USB_CONFIGFS is not set
# CONFIG_USB_ZERO is not set
# CONFIG_USB_AUDIO is not set
# CONFIG_USB_ETH is not set
# CONFIG_USB_G_NCM is not set
# CONFIG_USB_GADGETFS is not set
# CONFIG_USB_FUNCTIONFS is not set
CONFIG_USB_MASS_STORAGE=m
# CONFIG_USB_GADGET_TARGET is not set
# CONFIG_USB_G_SERIAL is not set
# CONFIG_USB_MIDI_GADGET is not set
# CONFIG_USB_G_PRINTER is not set
# CONFIG_USB_CDC_COMPOSITE is not set
# CONFIG_USB_G_ACM_MS is not set
# CONFIG_USB_G_MULTI is not set
# CONFIG_USB_G_HID is not set
# CONFIG_USB_G_DBGP is not set
# CONFIG_USB_G_WEBCAM is not set
# CONFIG_TYPEC is not set
# CONFIG_USB_LED_TRIG is not set
# CONFIG_USB_ULPI_BUS is not set
CONFIG_UWB=m
CONFIG_UWB_HWA=m
CONFIG_UWB_WHCI=m
CONFIG_UWB_I1480U=m
CONFIG_MMC=m
CONFIG_MMC_BLOCK=m
CONFIG_MMC_BLOCK_MINORS=8
CONFIG_SDIO_UART=m
# CONFIG_MMC_TEST is not set

#
# MMC/SD/SDIO Host Controller Drivers
#
# CONFIG_MMC_DEBUG is not set
CONFIG_MMC_SDHCI=m
CONFIG_MMC_SDHCI_PCI=m
CONFIG_MMC_RICOH_MMC=y
CONFIG_MMC_SDHCI_ACPI=m
CONFIG_MMC_SDHCI_PLTFM=m
# CONFIG_MMC_SDHCI_F_SDH30 is not set
# CONFIG_MMC_WBSD is not set
CONFIG_MMC_TIFM_SD=m
# CONFIG_MMC_SPI is not set
CONFIG_MMC_CB710=m
CONFIG_MMC_VIA_SDMMC=m
CONFIG_MMC_VUB300=m
CONFIG_MMC_USHC=m
# CONFIG_MMC_USDHI6ROL0 is not set
CONFIG_MMC_CQHCI=m
# CONFIG_MMC_TOSHIBA_PCI is not set
# CONFIG_MMC_MTK is not set
# CONFIG_MMC_SDHCI_XENON is not set
CONFIG_MEMSTICK=m
# CONFIG_MEMSTICK_DEBUG is not set

#
# MemoryStick drivers
#
# CONFIG_MEMSTICK_UNSAFE_RESUME is not set
CONFIG_MSPRO_BLOCK=m
# CONFIG_MS_BLOCK is not set

#
# MemoryStick Host Controller Drivers
#
CONFIG_MEMSTICK_TIFM_MS=m
CONFIG_MEMSTICK_JMICRON_38X=m
CONFIG_MEMSTICK_R592=m
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y
# CONFIG_LEDS_CLASS_FLASH is not set
# CONFIG_LEDS_BRIGHTNESS_HW_CHANGED is not set

#
# LED drivers
#
# CONFIG_LEDS_APU is not set
CONFIG_LEDS_LM3530=m
# CONFIG_LEDS_LM3642 is not set
# CONFIG_LEDS_PCA9532 is not set
# CONFIG_LEDS_GPIO is not set
CONFIG_LEDS_LP3944=m
# CONFIG_LEDS_LP3952 is not set
CONFIG_LEDS_LP55XX_COMMON=m
CONFIG_LEDS_LP5521=m
CONFIG_LEDS_LP5523=m
CONFIG_LEDS_LP5562=m
# CONFIG_LEDS_LP8501 is not set
CONFIG_LEDS_CLEVO_MAIL=m
# CONFIG_LEDS_PCA955X is not set
# CONFIG_LEDS_PCA963X is not set
# CONFIG_LEDS_DAC124S085 is not set
# CONFIG_LEDS_PWM is not set
# CONFIG_LEDS_BD2802 is not set
CONFIG_LEDS_INTEL_SS4200=m
# CONFIG_LEDS_LT3593 is not set
# CONFIG_LEDS_TCA6507 is not set
# CONFIG_LEDS_TLC591XX is not set
# CONFIG_LEDS_LM355x is not set

#
# LED driver for blink(1) USB RGB LED is under Special HID drivers (HID_THINGM)
#
CONFIG_LEDS_BLINKM=m
# CONFIG_LEDS_MLXCPLD is not set
# CONFIG_LEDS_USER is not set
# CONFIG_LEDS_NIC78BX is not set

#
# LED Triggers
#
CONFIG_LEDS_TRIGGERS=y
CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_ONESHOT=m
# CONFIG_LEDS_TRIGGER_DISK is not set
# CONFIG_LEDS_TRIGGER_MTD is not set
CONFIG_LEDS_TRIGGER_HEARTBEAT=m
CONFIG_LEDS_TRIGGER_BACKLIGHT=m
# CONFIG_LEDS_TRIGGER_CPU is not set
# CONFIG_LEDS_TRIGGER_ACTIVITY is not set
# CONFIG_LEDS_TRIGGER_GPIO is not set
CONFIG_LEDS_TRIGGER_DEFAULT_ON=m

#
# iptables trigger is under Netfilter config (LED target)
#
CONFIG_LEDS_TRIGGER_TRANSIENT=m
CONFIG_LEDS_TRIGGER_CAMERA=m
# CONFIG_LEDS_TRIGGER_PANIC is not set
# CONFIG_LEDS_TRIGGER_NETDEV is not set
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
CONFIG_EDAC_ATOMIC_SCRUB=y
CONFIG_EDAC_SUPPORT=y
CONFIG_EDAC=y
CONFIG_EDAC_LEGACY_SYSFS=y
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_DECODE_MCE=m
# CONFIG_EDAC_GHES is not set
CONFIG_EDAC_AMD64=m
# CONFIG_EDAC_AMD64_ERROR_INJECTION is not set
CONFIG_EDAC_E752X=m
CONFIG_EDAC_I82975X=m
CONFIG_EDAC_I3000=m
CONFIG_EDAC_I3200=m
# CONFIG_EDAC_IE31200 is not set
CONFIG_EDAC_X38=m
CONFIG_EDAC_I5400=m
CONFIG_EDAC_I7CORE=m
CONFIG_EDAC_I5000=m
CONFIG_EDAC_I5100=m
CONFIG_EDAC_I7300=m
CONFIG_EDAC_SBRIDGE=m
CONFIG_EDAC_SKX=m
# CONFIG_EDAC_PND2 is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_MC146818_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
# CONFIG_RTC_SYSTOHC is not set
# CONFIG_RTC_DEBUG is not set
CONFIG_RTC_NVMEM=y

#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set

#
# I2C RTC drivers
#
# CONFIG_RTC_DRV_ABB5ZES3 is not set
# CONFIG_RTC_DRV_ABX80X is not set
CONFIG_RTC_DRV_DS1307=m
CONFIG_RTC_DRV_DS1307_HWMON=y
# CONFIG_RTC_DRV_DS1307_CENTURY is not set
CONFIG_RTC_DRV_DS1374=m
# CONFIG_RTC_DRV_DS1374_WDT is not set
CONFIG_RTC_DRV_DS1672=m
CONFIG_RTC_DRV_MAX6900=m
CONFIG_RTC_DRV_RS5C372=m
CONFIG_RTC_DRV_ISL1208=m
CONFIG_RTC_DRV_ISL12022=m
CONFIG_RTC_DRV_X1205=m
CONFIG_RTC_DRV_PCF8523=m
# CONFIG_RTC_DRV_PCF85063 is not set
# CONFIG_RTC_DRV_PCF85363 is not set
CONFIG_RTC_DRV_PCF8563=m
CONFIG_RTC_DRV_PCF8583=m
CONFIG_RTC_DRV_M41T80=m
CONFIG_RTC_DRV_M41T80_WDT=y
CONFIG_RTC_DRV_BQ32K=m
# CONFIG_RTC_DRV_S35390A is not set
CONFIG_RTC_DRV_FM3130=m
# CONFIG_RTC_DRV_RX8010 is not set
CONFIG_RTC_DRV_RX8581=m
CONFIG_RTC_DRV_RX8025=m
CONFIG_RTC_DRV_EM3027=m
# CONFIG_RTC_DRV_RV8803 is not set

#
# SPI RTC drivers
#
# CONFIG_RTC_DRV_M41T93 is not set
# CONFIG_RTC_DRV_M41T94 is not set
# CONFIG_RTC_DRV_DS1302 is not set
# CONFIG_RTC_DRV_DS1305 is not set
# CONFIG_RTC_DRV_DS1343 is not set
# CONFIG_RTC_DRV_DS1347 is not set
# CONFIG_RTC_DRV_DS1390 is not set
# CONFIG_RTC_DRV_MAX6916 is not set
# CONFIG_RTC_DRV_R9701 is not set
# CONFIG_RTC_DRV_RX4581 is not set
# CONFIG_RTC_DRV_RX6110 is not set
# CONFIG_RTC_DRV_RS5C348 is not set
# CONFIG_RTC_DRV_MAX6902 is not set
# CONFIG_RTC_DRV_PCF2123 is not set
# CONFIG_RTC_DRV_MCP795 is not set
CONFIG_RTC_I2C_AND_SPI=y

#
# SPI and I2C RTC drivers
#
CONFIG_RTC_DRV_DS3232=m
CONFIG_RTC_DRV_DS3232_HWMON=y
# CONFIG_RTC_DRV_PCF2127 is not set
CONFIG_RTC_DRV_RV3029C2=m
CONFIG_RTC_DRV_RV3029_HWMON=y

#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=y
CONFIG_RTC_DRV_DS1286=m
CONFIG_RTC_DRV_DS1511=m
CONFIG_RTC_DRV_DS1553=m
# CONFIG_RTC_DRV_DS1685_FAMILY is not set
CONFIG_RTC_DRV_DS1742=m
CONFIG_RTC_DRV_DS2404=m
CONFIG_RTC_DRV_STK17TA8=m
# CONFIG_RTC_DRV_M48T86 is not set
CONFIG_RTC_DRV_M48T35=m
CONFIG_RTC_DRV_M48T59=m
CONFIG_RTC_DRV_MSM6242=m
CONFIG_RTC_DRV_BQ4802=m
CONFIG_RTC_DRV_RP5C01=m
CONFIG_RTC_DRV_V3020=m

#
# on-CPU RTC drivers
#
# CONFIG_RTC_DRV_FTRTC010 is not set

#
# HID Sensor RTC drivers
#
# CONFIG_RTC_DRV_HID_SENSOR_TIME is not set
CONFIG_DMADEVICES=y
# CONFIG_DMADEVICES_DEBUG is not set

#
# DMA Devices
#
CONFIG_DMA_ENGINE=y
CONFIG_DMA_VIRTUAL_CHANNELS=y
CONFIG_DMA_ACPI=y
# CONFIG_ALTERA_MSGDMA is not set
# CONFIG_INTEL_IDMA64 is not set
# CONFIG_INTEL_IOATDMA is not set
# CONFIG_QCOM_HIDMA_MGMT is not set
# CONFIG_QCOM_HIDMA is not set
CONFIG_DW_DMAC_CORE=y
CONFIG_DW_DMAC=m
CONFIG_DW_DMAC_PCI=y
CONFIG_HSU_DMA=y

#
# DMA Clients
#
CONFIG_ASYNC_TX_DMA=y
CONFIG_DMATEST=m
CONFIG_DMA_ENGINE_RAID=y

#
# DMABUF options
#
CONFIG_SYNC_FILE=y
CONFIG_SW_SYNC=y
CONFIG_AUXDISPLAY=y
# CONFIG_HD44780 is not set
CONFIG_KS0108=m
CONFIG_KS0108_PORT=0x378
CONFIG_KS0108_DELAY=2
CONFIG_CFAG12864B=m
CONFIG_CFAG12864B_RATE=20
# CONFIG_IMG_ASCII_LCD is not set
# CONFIG_PANEL is not set
CONFIG_UIO=m
CONFIG_UIO_CIF=m
CONFIG_UIO_PDRV_GENIRQ=m
# CONFIG_UIO_DMEM_GENIRQ is not set
CONFIG_UIO_AEC=m
CONFIG_UIO_SERCOS3=m
CONFIG_UIO_PCI_GENERIC=m
# CONFIG_UIO_NETX is not set
# CONFIG_UIO_PRUSS is not set
# CONFIG_UIO_MF624 is not set
# CONFIG_UIO_HV_GENERIC is not set
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO_VIRQFD=m
CONFIG_VFIO=m
# CONFIG_VFIO_NOIOMMU is not set
CONFIG_VFIO_PCI=m
# CONFIG_VFIO_PCI_VGA is not set
CONFIG_VFIO_PCI_MMAP=y
CONFIG_VFIO_PCI_INTX=y
CONFIG_VFIO_PCI_IGD=y
# CONFIG_VFIO_MDEV is not set
CONFIG_IRQ_BYPASS_MANAGER=m
# CONFIG_VIRT_DRIVERS is not set
CONFIG_VIRTIO=y
CONFIG_VIRTIO_MENU=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_PCI_LEGACY=y
CONFIG_VIRTIO_BALLOON=y
# CONFIG_VIRTIO_INPUT is not set
# CONFIG_VIRTIO_MMIO is not set

#
# Microsoft Hyper-V guest support
#
CONFIG_HYPERV=m
CONFIG_HYPERV_TSCPAGE=y
CONFIG_HYPERV_UTILS=m
CONFIG_HYPERV_BALLOON=m

#
# Xen driver support
#
CONFIG_XEN_BALLOON=y
# CONFIG_XEN_SELFBALLOONING is not set
# CONFIG_XEN_BALLOON_MEMORY_HOTPLUG is not set
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=m
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=m
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
# CONFIG_XEN_GNTDEV is not set
# CONFIG_XEN_GRANT_DEV_ALLOC is not set
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_TMEM=m
CONFIG_XEN_PCIDEV_BACKEND=m
# CONFIG_XEN_PVCALLS_FRONTEND is not set
# CONFIG_XEN_PVCALLS_BACKEND is not set
# CONFIG_XEN_SCSI_BACKEND is not set
CONFIG_XEN_PRIVCMD=m
CONFIG_XEN_ACPI_PROCESSOR=m
# CONFIG_XEN_MCE_LOG is not set
CONFIG_XEN_HAVE_PVMMU=y
CONFIG_XEN_EFI=y
CONFIG_XEN_AUTO_XLATE=y
CONFIG_XEN_ACPI=y
CONFIG_XEN_SYMS=y
CONFIG_XEN_HAVE_VPMU=y
CONFIG_STAGING=y
# CONFIG_IRDA is not set
# CONFIG_IPX is not set
# CONFIG_NCP_FS is not set
# CONFIG_PRISM2_USB is not set
# CONFIG_COMEDI is not set
# CONFIG_RTL8192U is not set
CONFIG_RTLLIB=m
CONFIG_RTLLIB_CRYPTO_CCMP=m
CONFIG_RTLLIB_CRYPTO_TKIP=m
CONFIG_RTLLIB_CRYPTO_WEP=m
CONFIG_RTL8192E=m
# CONFIG_RTL8723BS is not set
CONFIG_R8712U=m
# CONFIG_R8188EU is not set
# CONFIG_R8822BE is not set
# CONFIG_RTS5208 is not set
# CONFIG_VT6655 is not set
# CONFIG_VT6656 is not set
# CONFIG_FB_SM750 is not set
# CONFIG_FB_XGI is not set

#
# Speakup console speech
#
# CONFIG_SPEAKUP is not set
# CONFIG_STAGING_MEDIA is not set

#
# Android
#
# CONFIG_LTE_GDM724X is not set
CONFIG_FIREWIRE_SERIAL=m
CONFIG_FWTTY_MAX_TOTAL_PORTS=64
CONFIG_FWTTY_MAX_CARD_PORTS=32
# CONFIG_LNET is not set
# CONFIG_DGNC is not set
# CONFIG_GS_FPGABOOT is not set
# CONFIG_CRYPTO_SKEIN is not set
# CONFIG_UNISYSSPAR is not set
# CONFIG_FB_TFT is not set
# CONFIG_WILC1000_SDIO is not set
# CONFIG_WILC1000_SPI is not set
# CONFIG_MOST is not set
# CONFIG_KS7010 is not set
# CONFIG_GREYBUS is not set

#
# USB Power Delivery and Type-C drivers
#
# CONFIG_DRM_VBOXVIDEO is not set
# CONFIG_PI433 is not set
CONFIG_X86_PLATFORM_DEVICES=y
CONFIG_ACER_WMI=m
# CONFIG_ACER_WIRELESS is not set
CONFIG_ACERHDF=m
# CONFIG_ALIENWARE_WMI is not set
CONFIG_ASUS_LAPTOP=m
# CONFIG_DELL_SMBIOS_WMI is not set
# CONFIG_DELL_SMBIOS_SMM is not set
# CONFIG_DELL_LAPTOP is not set
# CONFIG_DELL_WMI is not set
CONFIG_DELL_WMI_AIO=m
# CONFIG_DELL_WMI_LED is not set
# CONFIG_DELL_SMO8800 is not set
# CONFIG_DELL_RBTN is not set
CONFIG_FUJITSU_LAPTOP=m
CONFIG_FUJITSU_TABLET=m
CONFIG_AMILO_RFKILL=m
# CONFIG_GPD_POCKET_FAN is not set
CONFIG_HP_ACCEL=m
# CONFIG_HP_WIRELESS is not set
CONFIG_HP_WMI=m
CONFIG_MSI_LAPTOP=m
CONFIG_PANASONIC_LAPTOP=m
CONFIG_COMPAL_LAPTOP=m
CONFIG_SONY_LAPTOP=m
CONFIG_SONYPI_COMPAT=y
CONFIG_IDEAPAD_LAPTOP=m
# CONFIG_SURFACE3_WMI is not set
CONFIG_THINKPAD_ACPI=m
CONFIG_THINKPAD_ACPI_ALSA_SUPPORT=y
# CONFIG_THINKPAD_ACPI_DEBUGFACILITIES is not set
# CONFIG_THINKPAD_ACPI_DEBUG is not set
# CONFIG_THINKPAD_ACPI_UNSAFE_LEDS is not set
CONFIG_THINKPAD_ACPI_VIDEO=y
CONFIG_THINKPAD_ACPI_HOTKEY_POLL=y
CONFIG_SENSORS_HDAPS=m
# CONFIG_INTEL_MENLOW is not set
CONFIG_EEEPC_LAPTOP=m
CONFIG_ASUS_WMI=m
CONFIG_ASUS_NB_WMI=m
CONFIG_EEEPC_WMI=m
# CONFIG_ASUS_WIRELESS is not set
CONFIG_ACPI_WMI=m
CONFIG_WMI_BMOF=m
# CONFIG_INTEL_WMI_THUNDERBOLT is not set
CONFIG_MSI_WMI=m
# CONFIG_PEAQ_WMI is not set
CONFIG_TOPSTAR_LAPTOP=m
CONFIG_TOSHIBA_BT_RFKILL=m
# CONFIG_TOSHIBA_HAPS is not set
# CONFIG_TOSHIBA_WMI is not set
CONFIG_ACPI_CMPC=m
# CONFIG_INTEL_INT0002_VGPIO is not set
# CONFIG_INTEL_HID_EVENT is not set
# CONFIG_INTEL_VBTN is not set
CONFIG_INTEL_IPS=m
# CONFIG_INTEL_PMC_CORE is not set
# CONFIG_IBM_RTL is not set
CONFIG_SAMSUNG_LAPTOP=m
CONFIG_MXM_WMI=m
CONFIG_INTEL_OAKTRAIL=m
CONFIG_SAMSUNG_Q10=m
CONFIG_APPLE_GMUX=m
# CONFIG_INTEL_RST is not set
# CONFIG_INTEL_SMARTCONNECT is not set
CONFIG_PVPANIC=y
# CONFIG_INTEL_PMC_IPC is not set
# CONFIG_SURFACE_PRO3_BUTTON is not set
# CONFIG_INTEL_PUNIT_IPC is not set
# CONFIG_MLX_PLATFORM is not set
# CONFIG_INTEL_TURBO_MAX_3 is not set
CONFIG_PMC_ATOM=y
# CONFIG_CHROME_PLATFORMS is not set
# CONFIG_MELLANOX_PLATFORM is not set
CONFIG_CLKDEV_LOOKUP=y
CONFIG_HAVE_CLK_PREPARE=y
CONFIG_COMMON_CLK=y

#
# Common Clock Framework
#
# CONFIG_COMMON_CLK_SI5351 is not set
# CONFIG_COMMON_CLK_CDCE706 is not set
# CONFIG_COMMON_CLK_CS2000_CP is not set
# CONFIG_COMMON_CLK_NXP is not set
# CONFIG_COMMON_CLK_PWM is not set
# CONFIG_COMMON_CLK_PXA is not set
# CONFIG_COMMON_CLK_PIC32 is not set
# CONFIG_HWSPINLOCK is not set

#
# Clock Source drivers
#
CONFIG_CLKEVT_I8253=y
CONFIG_I8253_LOCK=y
CONFIG_CLKBLD_I8253=y
# CONFIG_ATMEL_PIT is not set
# CONFIG_SH_TIMER_CMT is not set
# CONFIG_SH_TIMER_MTU2 is not set
# CONFIG_SH_TIMER_TMU is not set
# CONFIG_EM_TIMER_STI is not set
CONFIG_MAILBOX=y
CONFIG_PCC=y
# CONFIG_ALTERA_MBOX is not set
CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y

#
# Generic IOMMU Pagetable Support
#
CONFIG_IOMMU_IOVA=y
CONFIG_AMD_IOMMU=y
CONFIG_AMD_IOMMU_V2=m
CONFIG_DMAR_TABLE=y
CONFIG_INTEL_IOMMU=y
# CONFIG_INTEL_IOMMU_SVM is not set
# CONFIG_INTEL_IOMMU_DEFAULT_ON is not set
CONFIG_INTEL_IOMMU_FLOPPY_WA=y
CONFIG_IRQ_REMAP=y

#
# Remoteproc drivers
#
# CONFIG_REMOTEPROC is not set

#
# Rpmsg drivers
#
# CONFIG_RPMSG_QCOM_GLINK_RPM is not set
# CONFIG_RPMSG_VIRTIO is not set
# CONFIG_SOUNDWIRE is not set

#
# SOC (System On Chip) specific Drivers
#

#
# Amlogic SoC drivers
#

#
# Broadcom SoC drivers
#

#
# i.MX SoC drivers
#

#
# Qualcomm SoC drivers
#
# CONFIG_SUNXI_SRAM is not set
# CONFIG_SOC_TI is not set

#
# Xilinx SoC drivers
#
# CONFIG_XILINX_VCU is not set
CONFIG_PM_DEVFREQ=y

#
# DEVFREQ Governors
#
CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=m
# CONFIG_DEVFREQ_GOV_PERFORMANCE is not set
# CONFIG_DEVFREQ_GOV_POWERSAVE is not set
# CONFIG_DEVFREQ_GOV_USERSPACE is not set
# CONFIG_DEVFREQ_GOV_PASSIVE is not set

#
# DEVFREQ Drivers
#
# CONFIG_PM_DEVFREQ_EVENT is not set
CONFIG_EXTCON=y

#
# Extcon Device Drivers
#
# CONFIG_EXTCON_GPIO is not set
# CONFIG_EXTCON_INTEL_INT3496 is not set
# CONFIG_EXTCON_MAX3355 is not set
# CONFIG_EXTCON_RT8973A is not set
# CONFIG_EXTCON_SM5502 is not set
# CONFIG_EXTCON_USB_GPIO is not set
# CONFIG_MEMORY is not set
# CONFIG_IIO is not set
CONFIG_NTB=m
# CONFIG_NTB_AMD is not set
# CONFIG_NTB_IDT is not set
# CONFIG_NTB_INTEL is not set
# CONFIG_NTB_SWITCHTEC is not set
# CONFIG_NTB_PINGPONG is not set
# CONFIG_NTB_TOOL is not set
# CONFIG_NTB_PERF is not set
# CONFIG_NTB_TRANSPORT is not set
# CONFIG_VME_BUS is not set
CONFIG_PWM=y
CONFIG_PWM_SYSFS=y
CONFIG_PWM_LPSS=m
CONFIG_PWM_LPSS_PCI=m
CONFIG_PWM_LPSS_PLATFORM=m
# CONFIG_PWM_PCA9685 is not set

#
# IRQ chip support
#
CONFIG_ARM_GIC_MAX_NR=1
# CONFIG_ARM_GIC_V3_ITS is not set
# CONFIG_IPACK_BUS is not set
# CONFIG_RESET_CONTROLLER is not set
# CONFIG_FMC is not set

#
# PHY Subsystem
#
CONFIG_GENERIC_PHY=y
# CONFIG_BCM_KONA_USB2_PHY is not set
# CONFIG_PHY_PXA_28NM_HSIC is not set
# CONFIG_PHY_PXA_28NM_USB2 is not set
CONFIG_POWERCAP=y
CONFIG_INTEL_RAPL=m
# CONFIG_MCB is not set

#
# Performance monitor support
#
CONFIG_RAS=y
# CONFIG_RAS_CEC is not set
# CONFIG_THUNDERBOLT is not set

#
# Android
#
# CONFIG_ANDROID is not set
CONFIG_LIBNVDIMM=m
CONFIG_BLK_DEV_PMEM=m
CONFIG_ND_BLK=m
CONFIG_ND_CLAIM=y
CONFIG_ND_BTT=m
CONFIG_BTT=y
CONFIG_ND_PFN=m
CONFIG_NVDIMM_PFN=y
CONFIG_NVDIMM_DAX=y
CONFIG_DAX=y
CONFIG_DEV_DAX=m
CONFIG_DEV_DAX_PMEM=m
CONFIG_NVMEM=y
CONFIG_STM=m
CONFIG_STM_DUMMY=m
CONFIG_STM_SOURCE_CONSOLE=m
CONFIG_STM_SOURCE_HEARTBEAT=m
CONFIG_STM_SOURCE_FTRACE=m
CONFIG_INTEL_TH=m
CONFIG_INTEL_TH_PCI=m
CONFIG_INTEL_TH_GTH=m
CONFIG_INTEL_TH_STH=m
CONFIG_INTEL_TH_MSU=m
CONFIG_INTEL_TH_PTI=m
# CONFIG_INTEL_TH_DEBUG is not set
# CONFIG_FPGA is not set
# CONFIG_FSI is not set
CONFIG_PM_OPP=y
# CONFIG_UNISYS_VISORBUS is not set
# CONFIG_SIOX is not set
# CONFIG_SLIMBUS is not set

#
# Firmware Drivers
#
CONFIG_EDD=m
# CONFIG_EDD_OFF is not set
CONFIG_FIRMWARE_MEMMAP=y
CONFIG_DELL_RBU=m
CONFIG_DCDBAS=m
CONFIG_DMIID=y
CONFIG_DMI_SYSFS=y
CONFIG_DMI_SCAN_MACHINE_NON_EFI_FALLBACK=y
CONFIG_ISCSI_IBFT_FIND=y
CONFIG_ISCSI_IBFT=m
# CONFIG_FW_CFG_SYSFS is not set
# CONFIG_GOOGLE_FIRMWARE is not set

#
# EFI (Extensible Firmware Interface) Support
#
CONFIG_EFI_VARS=y
CONFIG_EFI_ESRT=y
CONFIG_EFI_VARS_PSTORE=y
CONFIG_EFI_VARS_PSTORE_DEFAULT_DISABLE=y
CONFIG_EFI_RUNTIME_MAP=y
# CONFIG_EFI_FAKE_MEMMAP is not set
CONFIG_EFI_RUNTIME_WRAPPERS=y
# CONFIG_EFI_BOOTLOADER_CONTROL is not set
# CONFIG_EFI_CAPSULE_LOADER is not set
# CONFIG_EFI_TEST is not set
# CONFIG_APPLE_PROPERTIES is not set
# CONFIG_RESET_ATTACK_MITIGATION is not set
CONFIG_UEFI_CPER=y
# CONFIG_EFI_DEV_PATH_PARSER is not set

#
# Tegra firmware driver
#

#
# File systems
#
CONFIG_DCACHE_WORD_ACCESS=y
CONFIG_FS_IOMAP=y
# CONFIG_EXT2_FS is not set
# CONFIG_EXT3_FS is not set
CONFIG_EXT4_FS=y
CONFIG_EXT4_USE_FOR_EXT2=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_EXT4_ENCRYPTION=y
CONFIG_EXT4_FS_ENCRYPTION=y
# CONFIG_EXT4_DEBUG is not set
CONFIG_JBD2=y
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
CONFIG_XFS_FS=m
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
CONFIG_XFS_RT=y
CONFIG_XFS_ONLINE_SCRUB=y
CONFIG_XFS_DEBUG=y
CONFIG_XFS_ASSERT_FATAL=y
CONFIG_GFS2_FS=m
CONFIG_GFS2_FS_LOCKING_DLM=y
CONFIG_OCFS2_FS=m
CONFIG_OCFS2_FS_O2CB=m
CONFIG_OCFS2_FS_USERSPACE_CLUSTER=m
CONFIG_OCFS2_FS_STATS=y
CONFIG_OCFS2_DEBUG_MASKLOG=y
# CONFIG_OCFS2_DEBUG_FS is not set
CONFIG_BTRFS_FS=m
CONFIG_BTRFS_FS_POSIX_ACL=y
# CONFIG_BTRFS_FS_CHECK_INTEGRITY is not set
# CONFIG_BTRFS_FS_RUN_SANITY_TESTS is not set
# CONFIG_BTRFS_DEBUG is not set
# CONFIG_BTRFS_ASSERT is not set
# CONFIG_BTRFS_FS_REF_VERIFY is not set
# CONFIG_NILFS2_FS is not set
CONFIG_F2FS_FS=m
CONFIG_F2FS_STAT_FS=y
CONFIG_F2FS_FS_XATTR=y
CONFIG_F2FS_FS_POSIX_ACL=y
# CONFIG_F2FS_FS_SECURITY is not set
# CONFIG_F2FS_CHECK_FS is not set
CONFIG_F2FS_FS_ENCRYPTION=y
# CONFIG_F2FS_IO_TRACE is not set
# CONFIG_F2FS_FAULT_INJECTION is not set
CONFIG_FS_DAX=y
CONFIG_FS_DAX_PMD=y
CONFIG_FS_POSIX_ACL=y
CONFIG_EXPORTFS=y
# CONFIG_EXPORTFS_BLOCK_OPS is not set
CONFIG_FILE_LOCKING=y
CONFIG_MANDATORY_FILE_LOCKING=y
CONFIG_FS_ENCRYPTION=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_FANOTIFY=y
CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
CONFIG_PRINT_QUOTA_WARNING=y
# CONFIG_QUOTA_DEBUG is not set
CONFIG_QUOTA_TREE=y
# CONFIG_QFMT_V1 is not set
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
CONFIG_QUOTACTL_COMPAT=y
CONFIG_AUTOFS4_FS=y
CONFIG_FUSE_FS=m
CONFIG_CUSE=m
CONFIG_OVERLAY_FS=m
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set

#
# Caches
#
CONFIG_FSCACHE=m
CONFIG_FSCACHE_STATS=y
# CONFIG_FSCACHE_HISTOGRAM is not set
# CONFIG_FSCACHE_DEBUG is not set
# CONFIG_FSCACHE_OBJECT_LIST is not set
CONFIG_CACHEFILES=m
# CONFIG_CACHEFILES_DEBUG is not set
# CONFIG_CACHEFILES_HISTOGRAM is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="ascii"
# CONFIG_FAT_DEFAULT_UTF8 is not set
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_VMCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_PROC_CHILDREN=y
CONFIG_KERNFS=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS_XATTR=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_ARCH_HAS_GIGANTIC_PAGE=y
CONFIG_CONFIGFS_FS=y
CONFIG_EFIVAR_FS=y
CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ORANGEFS_FS is not set
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_ECRYPT_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_JFFS2_FS is not set
# CONFIG_UBIFS_FS is not set
CONFIG_CRAMFS=m
CONFIG_CRAMFS_BLOCKDEV=y
# CONFIG_CRAMFS_MTD is not set
CONFIG_SQUASHFS=m
CONFIG_SQUASHFS_FILE_CACHE=y
# CONFIG_SQUASHFS_FILE_DIRECT is not set
CONFIG_SQUASHFS_DECOMP_SINGLE=y
# CONFIG_SQUASHFS_DECOMP_MULTI is not set
# CONFIG_SQUASHFS_DECOMP_MULTI_PERCPU is not set
CONFIG_SQUASHFS_XATTR=y
CONFIG_SQUASHFS_ZLIB=y
# CONFIG_SQUASHFS_LZ4 is not set
CONFIG_SQUASHFS_LZO=y
CONFIG_SQUASHFS_XZ=y
# CONFIG_SQUASHFS_ZSTD is not set
# CONFIG_SQUASHFS_4K_DEVBLK_SIZE is not set
# CONFIG_SQUASHFS_EMBEDDED is not set
CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3
# CONFIG_VXFS_FS is not set
CONFIG_MINIX_FS=m
# CONFIG_OMFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_QNX6FS_FS is not set
# CONFIG_ROMFS_FS is not set
CONFIG_PSTORE=y
CONFIG_PSTORE_ZLIB_COMPRESS=y
# CONFIG_PSTORE_LZO_COMPRESS is not set
# CONFIG_PSTORE_LZ4_COMPRESS is not set
CONFIG_PSTORE_CONSOLE=y
CONFIG_PSTORE_PMSG=y
# CONFIG_PSTORE_FTRACE is not set
CONFIG_PSTORE_RAM=m
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
# CONFIG_EXOFS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
# CONFIG_NFS_V2 is not set
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=m
# CONFIG_NFS_SWAP is not set
CONFIG_NFS_V4_1=y
CONFIG_NFS_V4_2=y
CONFIG_PNFS_FILE_LAYOUT=m
CONFIG_PNFS_BLOCK=m
CONFIG_PNFS_FLEXFILE_LAYOUT=m
CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org"
# CONFIG_NFS_V4_1_MIGRATION is not set
CONFIG_NFS_V4_SECURITY_LABEL=y
CONFIG_ROOT_NFS=y
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFS_DEBUG=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
# CONFIG_NFSD_BLOCKLAYOUT is not set
# CONFIG_NFSD_SCSILAYOUT is not set
# CONFIG_NFSD_FLEXFILELAYOUT is not set
CONFIG_NFSD_V4_SECURITY_LABEL=y
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_GRACE_PERIOD=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_NFS_ACL_SUPPORT=y
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=m
CONFIG_SUNRPC_BACKCHANNEL=y
CONFIG_RPCSEC_GSS_KRB5=m
CONFIG_SUNRPC_DEBUG=y
# CONFIG_CEPH_FS is not set
CONFIG_CIFS=m
CONFIG_CIFS_STATS=y
# CONFIG_CIFS_STATS2 is not set
CONFIG_CIFS_WEAK_PW_HASH=y
CONFIG_CIFS_UPCALL=y
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
CONFIG_CIFS_ACL=y
CONFIG_CIFS_DEBUG=y
# CONFIG_CIFS_DEBUG2 is not set
# CONFIG_CIFS_DEBUG_DUMP_KEYS is not set
CONFIG_CIFS_DFS_UPCALL=y
# CONFIG_CIFS_SMB311 is not set
# CONFIG_CIFS_FSCACHE is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
CONFIG_9P_FS=y
CONFIG_9P_FS_POSIX_ACL=y
# CONFIG_9P_FS_SECURITY is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_737=m
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_855=m
CONFIG_NLS_CODEPAGE_857=m
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=m
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=m
CONFIG_NLS_CODEPAGE_865=m
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
CONFIG_NLS_CODEPAGE_932=m
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=m
CONFIG_NLS_ISO8859_3=m
CONFIG_NLS_ISO8859_4=m
CONFIG_NLS_ISO8859_5=m
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=m
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=m
CONFIG_NLS_KOI8_R=m
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_MAC_ROMAN=m
CONFIG_NLS_MAC_CELTIC=m
CONFIG_NLS_MAC_CENTEURO=m
CONFIG_NLS_MAC_CROATIAN=m
CONFIG_NLS_MAC_CYRILLIC=m
CONFIG_NLS_MAC_GAELIC=m
CONFIG_NLS_MAC_GREEK=m
CONFIG_NLS_MAC_ICELAND=m
CONFIG_NLS_MAC_INUIT=m
CONFIG_NLS_MAC_ROMANIAN=m
CONFIG_NLS_MAC_TURKISH=m
CONFIG_NLS_UTF8=m
CONFIG_DLM=m
CONFIG_DLM_DEBUG=y

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y

#
# printk and dmesg options
#
CONFIG_PRINTK_TIME=y
CONFIG_CONSOLE_LOGLEVEL_DEFAULT=7
CONFIG_MESSAGE_LOGLEVEL_DEFAULT=4
CONFIG_BOOT_PRINTK_DELAY=y
CONFIG_DYNAMIC_DEBUG=y

#
# Compile-time checks and compiler options
#
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_INFO_REDUCED=y
# CONFIG_DEBUG_INFO_SPLIT is not set
# CONFIG_DEBUG_INFO_DWARF4 is not set
# CONFIG_GDB_SCRIPTS is not set
# CONFIG_ENABLE_WARN_DEPRECATED is not set
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_STRIP_ASM_SYMS=y
# CONFIG_READABLE_ASM is not set
# CONFIG_UNUSED_SYMBOLS is not set
# CONFIG_PAGE_OWNER is not set
CONFIG_DEBUG_FS=y
CONFIG_HEADERS_CHECK=y
CONFIG_DEBUG_SECTION_MISMATCH=y
CONFIG_SECTION_MISMATCH_WARN_ONLY=y
CONFIG_STACK_VALIDATION=y
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
CONFIG_MAGIC_SYSRQ=y
CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
CONFIG_MAGIC_SYSRQ_SERIAL=y
CONFIG_DEBUG_KERNEL=y

#
# Memory Debugging
#
# CONFIG_PAGE_EXTENSION is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_PAGE_POISONING is not set
# CONFIG_DEBUG_PAGE_REF is not set
CONFIG_DEBUG_RODATA_TEST=y
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
CONFIG_HAVE_DEBUG_KMEMLEAK=y
# CONFIG_DEBUG_KMEMLEAK is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_VM is not set
CONFIG_ARCH_HAS_DEBUG_VIRTUAL=y
# CONFIG_DEBUG_VIRTUAL is not set
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_MEMORY_NOTIFIER_ERROR_INJECT=m
# CONFIG_DEBUG_PER_CPU_MAPS is not set
CONFIG_HAVE_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_HAVE_ARCH_KASAN=y
# CONFIG_KASAN is not set
CONFIG_ARCH_HAS_KCOV=y
# CONFIG_KCOV is not set
CONFIG_DEBUG_SHIRQ=y

#
# Debug Lockups and Hangs
#
CONFIG_LOCKUP_DETECTOR=y
CONFIG_SOFTLOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR_PERF=y
CONFIG_HARDLOCKUP_CHECK_TIMESTAMP=y
CONFIG_HARDLOCKUP_DETECTOR=y
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC=y
CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=1
# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
# CONFIG_DETECT_HUNG_TASK is not set
# CONFIG_WQ_WATCHDOG is not set
CONFIG_PANIC_ON_OOPS=y
CONFIG_PANIC_ON_OOPS_VALUE=1
CONFIG_PANIC_TIMEOUT=0
CONFIG_SCHED_DEBUG=y
CONFIG_SCHED_INFO=y
CONFIG_SCHEDSTATS=y
# CONFIG_SCHED_STACK_END_CHECK is not set
# CONFIG_DEBUG_TIMEKEEPING is not set

#
# Lock Debugging (spinlocks, mutexes, etc...)
#
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_WW_MUTEX_SLOWPATH is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
CONFIG_DEBUG_ATOMIC_SLEEP=y
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_LOCK_TORTURE_TEST=m
CONFIG_WW_MUTEX_SELFTEST=m
CONFIG_STACKTRACE=y
# CONFIG_WARN_ALL_UNSEEDED_RANDOM is not set
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_LIST=y
# CONFIG_DEBUG_PI_LIST is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set

#
# RCU Debugging
#
# CONFIG_PROVE_RCU is not set
CONFIG_TORTURE_TEST=m
CONFIG_RCU_PERF_TEST=m
CONFIG_RCU_TORTURE_TEST=m
CONFIG_RCU_CPU_STALL_TIMEOUT=60
# CONFIG_RCU_TRACE is not set
# CONFIG_RCU_EQS_DEBUG is not set
# CONFIG_DEBUG_WQ_FORCE_RR_CPU is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_CPU_HOTPLUG_STATE_CONTROL is not set
CONFIG_NOTIFIER_ERROR_INJECTION=m
CONFIG_PM_NOTIFIER_ERROR_INJECT=m
# CONFIG_NETDEV_NOTIFIER_ERROR_INJECT is not set
CONFIG_FAULT_INJECTION=y
CONFIG_FUNCTION_ERROR_INJECTION=y
# CONFIG_FAILSLAB is not set
# CONFIG_FAIL_PAGE_ALLOC is not set
CONFIG_FAIL_MAKE_REQUEST=y
# CONFIG_FAIL_IO_TIMEOUT is not set
# CONFIG_FAIL_MMC_REQUEST is not set
# CONFIG_FAIL_FUTEX is not set
# CONFIG_FAIL_FUNCTION is not set
CONFIG_FAULT_INJECTION_DEBUG_FS=y
CONFIG_LATENCYTOP=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_HAVE_FENTRY=y
CONFIG_HAVE_C_RECORDMCOUNT=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACE_CLOCK=y
CONFIG_RING_BUFFER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
CONFIG_TRACING=y
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
# CONFIG_PREEMPTIRQ_EVENTS is not set
# CONFIG_IRQSOFF_TRACER is not set
CONFIG_SCHED_TRACER=y
# CONFIG_HWLAT_TRACER is not set
CONFIG_FTRACE_SYSCALLS=y
CONFIG_TRACER_SNAPSHOT=y
# CONFIG_TRACER_SNAPSHOT_PER_CPU_SWAP is not set
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
CONFIG_STACK_TRACER=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_KPROBE_EVENTS=y
CONFIG_UPROBE_EVENTS=y
CONFIG_BPF_EVENTS=y
CONFIG_PROBE_EVENTS=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_FUNCTION_PROFILER=y
# CONFIG_BPF_KPROBE_OVERRIDE is not set
CONFIG_FTRACE_MCOUNT_RECORD=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_MMIOTRACE is not set
CONFIG_TRACING_MAP=y
CONFIG_HIST_TRIGGERS=y
# CONFIG_TRACEPOINT_BENCHMARK is not set
CONFIG_RING_BUFFER_BENCHMARK=m
# CONFIG_RING_BUFFER_STARTUP_TEST is not set
# CONFIG_TRACE_EVAL_MAP_FILE is not set
CONFIG_TRACING_EVENTS_GPIO=y
CONFIG_PROVIDE_OHCI1394_DMA_INIT=y
# CONFIG_DMA_API_DEBUG is not set
CONFIG_RUNTIME_TESTING_MENU=y
CONFIG_LKDTM=m
# CONFIG_TEST_LIST_SORT is not set
# CONFIG_TEST_SORT is not set
# CONFIG_KPROBES_SANITY_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
CONFIG_RBTREE_TEST=m
CONFIG_INTERVAL_TREE_TEST=m
CONFIG_PERCPU_TEST=m
CONFIG_ATOMIC64_SELFTEST=y
CONFIG_ASYNC_RAID6_TEST=m
# CONFIG_TEST_HEXDUMP is not set
# CONFIG_TEST_STRING_HELPERS is not set
CONFIG_TEST_KSTRTOX=m
CONFIG_TEST_PRINTF=m
CONFIG_TEST_BITMAP=m
# CONFIG_TEST_UUID is not set
# CONFIG_TEST_RHASHTABLE is not set
# CONFIG_TEST_HASH is not set
CONFIG_TEST_LKM=m
CONFIG_TEST_USER_COPY=m
CONFIG_TEST_BPF=m
# CONFIG_FIND_BIT_BENCHMARK is not set
CONFIG_TEST_FIRMWARE=m
CONFIG_TEST_SYSCTL=m
CONFIG_TEST_UDELAY=m
CONFIG_TEST_STATIC_KEYS=m
CONFIG_TEST_KMOD=m
# CONFIG_MEMTEST is not set
# CONFIG_BUG_ON_DATA_CORRUPTION is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_ARCH_HAS_UBSAN_SANITIZE_ALL=y
# CONFIG_ARCH_WANTS_UBSAN_NO_NULL is not set
# CONFIG_UBSAN is not set
CONFIG_ARCH_HAS_DEVMEM_IS_ALLOWED=y
CONFIG_STRICT_DEVMEM=y
# CONFIG_IO_STRICT_DEVMEM is not set
CONFIG_EARLY_PRINTK_USB=y
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
CONFIG_EARLY_PRINTK_DBGP=y
# CONFIG_EARLY_PRINTK_EFI is not set
# CONFIG_EARLY_PRINTK_USB_XDBC is not set
# CONFIG_X86_PTDUMP_CORE is not set
# CONFIG_X86_PTDUMP is not set
# CONFIG_EFI_PGT_DUMP is not set
# CONFIG_DEBUG_WX is not set
CONFIG_DOUBLEFAULT=y
# CONFIG_DEBUG_TLBFLUSH is not set
# CONFIG_IOMMU_DEBUG is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_X86_DECODER_SELFTEST=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_DEBUG_BOOT_PARAMS=y
# CONFIG_CPA_DEBUG is not set
CONFIG_OPTIMIZE_INLINING=y
# CONFIG_DEBUG_ENTRY is not set
# CONFIG_DEBUG_NMI_SELFTEST is not set
CONFIG_X86_DEBUG_FPU=y
# CONFIG_PUNIT_ATOM_DEBUG is not set
CONFIG_UNWINDER_ORC=y
# CONFIG_UNWINDER_FRAME_POINTER is not set
# CONFIG_UNWINDER_GUESS is not set

#
# Security options
#
CONFIG_KEYS=y
CONFIG_KEYS_COMPAT=y
CONFIG_PERSISTENT_KEYRINGS=y
CONFIG_BIG_KEYS=y
CONFIG_TRUSTED_KEYS=y
CONFIG_ENCRYPTED_KEYS=y
# CONFIG_KEY_DH_OPERATIONS is not set
# CONFIG_SECURITY_DMESG_RESTRICT is not set
CONFIG_SECURITY=y
CONFIG_SECURITY_WRITABLE_HOOKS=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_PAGE_TABLE_ISOLATION=y
CONFIG_SECURITY_NETWORK_XFRM=y
CONFIG_SECURITY_PATH=y
CONFIG_INTEL_TXT=y
CONFIG_LSM_MMAP_MIN_ADDR=65535
CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR=y
# CONFIG_HARDENED_USERCOPY is not set
# CONFIG_FORTIFY_SOURCE is not set
# CONFIG_STATIC_USERMODEHELPER is not set
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
CONFIG_SECURITY_SELINUX_DISABLE=y
CONFIG_SECURITY_SELINUX_DEVELOP=y
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SMACK is not set
# CONFIG_SECURITY_TOMOYO is not set
CONFIG_SECURITY_APPARMOR=y
CONFIG_SECURITY_APPARMOR_BOOTPARAM_VALUE=1
CONFIG_SECURITY_APPARMOR_HASH=y
CONFIG_SECURITY_APPARMOR_HASH_DEFAULT=y
# CONFIG_SECURITY_APPARMOR_DEBUG is not set
# CONFIG_SECURITY_LOADPIN is not set
# CONFIG_SECURITY_YAMA is not set
CONFIG_INTEGRITY=y
CONFIG_INTEGRITY_SIGNATURE=y
CONFIG_INTEGRITY_ASYMMETRIC_KEYS=y
CONFIG_INTEGRITY_TRUSTED_KEYRING=y
CONFIG_INTEGRITY_AUDIT=y
CONFIG_IMA=y
CONFIG_IMA_MEASURE_PCR_IDX=10
CONFIG_IMA_LSM_RULES=y
# CONFIG_IMA_TEMPLATE is not set
CONFIG_IMA_NG_TEMPLATE=y
# CONFIG_IMA_SIG_TEMPLATE is not set
CONFIG_IMA_DEFAULT_TEMPLATE="ima-ng"
CONFIG_IMA_DEFAULT_HASH_SHA1=y
# CONFIG_IMA_DEFAULT_HASH_SHA256 is not set
# CONFIG_IMA_DEFAULT_HASH_SHA512 is not set
CONFIG_IMA_DEFAULT_HASH="sha1"
# CONFIG_IMA_WRITE_POLICY is not set
# CONFIG_IMA_READ_POLICY is not set
CONFIG_IMA_APPRAISE=y
CONFIG_IMA_APPRAISE_BOOTPARAM=y
CONFIG_IMA_TRUSTED_KEYRING=y
# CONFIG_IMA_BLACKLIST_KEYRING is not set
# CONFIG_IMA_LOAD_X509 is not set
CONFIG_EVM=y
CONFIG_EVM_ATTR_FSUUID=y
# CONFIG_EVM_LOAD_X509 is not set
CONFIG_DEFAULT_SECURITY_SELINUX=y
# CONFIG_DEFAULT_SECURITY_APPARMOR is not set
# CONFIG_DEFAULT_SECURITY_DAC is not set
CONFIG_DEFAULT_SECURITY="selinux"
CONFIG_XOR_BLOCKS=m
CONFIG_ASYNC_CORE=m
CONFIG_ASYNC_MEMCPY=m
CONFIG_ASYNC_XOR=m
CONFIG_ASYNC_PQ=m
CONFIG_ASYNC_RAID6_RECOV=m
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_RNG_DEFAULT=y
CONFIG_CRYPTO_AKCIPHER2=y
CONFIG_CRYPTO_AKCIPHER=y
CONFIG_CRYPTO_KPP2=y
CONFIG_CRYPTO_ACOMP2=y
CONFIG_CRYPTO_RSA=y
# CONFIG_CRYPTO_DH is not set
# CONFIG_CRYPTO_ECDH is not set
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_USER=m
CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y
CONFIG_CRYPTO_GF128MUL=y
CONFIG_CRYPTO_NULL=y
CONFIG_CRYPTO_NULL2=y
CONFIG_CRYPTO_PCRYPT=m
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_CRYPTO_CRYPTD=m
CONFIG_CRYPTO_MCRYPTD=m
CONFIG_CRYPTO_AUTHENC=m
CONFIG_CRYPTO_TEST=m
CONFIG_CRYPTO_ABLK_HELPER=m
CONFIG_CRYPTO_SIMD=m
CONFIG_CRYPTO_GLUE_HELPER_X86=m
CONFIG_CRYPTO_ENGINE=m

#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=y
# CONFIG_CRYPTO_CHACHA20POLY1305 is not set
CONFIG_CRYPTO_SEQIV=y
CONFIG_CRYPTO_ECHAINIV=m

#
# Block modes
#
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_CTR=y
CONFIG_CRYPTO_CTS=y
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=y
# CONFIG_CRYPTO_KEYWRAP is not set

#
# Hash modes
#
CONFIG_CRYPTO_CMAC=m
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_VMAC=m

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_CRC32C_INTEL=m
CONFIG_CRYPTO_CRC32=m
CONFIG_CRYPTO_CRC32_PCLMUL=m
CONFIG_CRYPTO_CRCT10DIF=y
CONFIG_CRYPTO_CRCT10DIF_PCLMUL=m
CONFIG_CRYPTO_GHASH=y
# CONFIG_CRYPTO_POLY1305 is not set
# CONFIG_CRYPTO_POLY1305_X86_64 is not set
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_RMD128=m
CONFIG_CRYPTO_RMD160=m
CONFIG_CRYPTO_RMD256=m
CONFIG_CRYPTO_RMD320=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA1_SSSE3=m
CONFIG_CRYPTO_SHA256_SSSE3=m
CONFIG_CRYPTO_SHA512_SSSE3=m
CONFIG_CRYPTO_SHA1_MB=m
CONFIG_CRYPTO_SHA256_MB=m
CONFIG_CRYPTO_SHA512_MB=m
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_SHA512=y
CONFIG_CRYPTO_SHA3=m
# CONFIG_CRYPTO_SM3 is not set
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL=m

#
# Ciphers
#
CONFIG_CRYPTO_AES=y
# CONFIG_CRYPTO_AES_TI is not set
CONFIG_CRYPTO_AES_X86_64=y
CONFIG_CRYPTO_AES_NI_INTEL=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_BLOWFISH_COMMON=m
CONFIG_CRYPTO_BLOWFISH_X86_64=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAMELLIA_X86_64=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX_X86_64=m
CONFIG_CRYPTO_CAMELLIA_AESNI_AVX2_X86_64=m
CONFIG_CRYPTO_CAST_COMMON=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST5_AVX_X86_64=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_CAST6_AVX_X86_64=m
CONFIG_CRYPTO_DES=m
# CONFIG_CRYPTO_DES3_EDE_X86_64 is not set
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
CONFIG_CRYPTO_SALSA20_X86_64=m
# CONFIG_CRYPTO_CHACHA20 is not set
# CONFIG_CRYPTO_CHACHA20_X86_64 is not set
CONFIG_CRYPTO_SEED=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_SERPENT_SSE2_X86_64=m
CONFIG_CRYPTO_SERPENT_AVX_X86_64=m
CONFIG_CRYPTO_SERPENT_AVX2_X86_64=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m
CONFIG_CRYPTO_TWOFISH_X86_64_3WAY=m
CONFIG_CRYPTO_TWOFISH_AVX_X86_64=m

#
# Compression
#
CONFIG_CRYPTO_DEFLATE=y
CONFIG_CRYPTO_LZO=y
# CONFIG_CRYPTO_842 is not set
# CONFIG_CRYPTO_LZ4 is not set
# CONFIG_CRYPTO_LZ4HC is not set

#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=m
CONFIG_CRYPTO_DRBG_MENU=y
CONFIG_CRYPTO_DRBG_HMAC=y
# CONFIG_CRYPTO_DRBG_HASH is not set
# CONFIG_CRYPTO_DRBG_CTR is not set
CONFIG_CRYPTO_DRBG=y
CONFIG_CRYPTO_JITTERENTROPY=y
CONFIG_CRYPTO_USER_API=y
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CRYPTO_USER_API_SKCIPHER=y
# CONFIG_CRYPTO_USER_API_RNG is not set
# CONFIG_CRYPTO_USER_API_AEAD is not set
CONFIG_CRYPTO_HASH_INFO=y
CONFIG_CRYPTO_HW=y
CONFIG_CRYPTO_DEV_PADLOCK=m
CONFIG_CRYPTO_DEV_PADLOCK_AES=m
CONFIG_CRYPTO_DEV_PADLOCK_SHA=m
# CONFIG_CRYPTO_DEV_FSL_CAAM_CRYPTO_API_DESC is not set
# CONFIG_CRYPTO_DEV_CCP is not set
# CONFIG_CRYPTO_DEV_QAT_DH895xCC is not set
# CONFIG_CRYPTO_DEV_QAT_C3XXX is not set
# CONFIG_CRYPTO_DEV_QAT_C62X is not set
# CONFIG_CRYPTO_DEV_QAT_DH895xCCVF is not set
# CONFIG_CRYPTO_DEV_QAT_C3XXXVF is not set
# CONFIG_CRYPTO_DEV_QAT_C62XVF is not set
# CONFIG_CRYPTO_DEV_NITROX_CNN55XX is not set
# CONFIG_CRYPTO_DEV_CHELSIO is not set
CONFIG_CRYPTO_DEV_VIRTIO=m
CONFIG_ASYMMETRIC_KEY_TYPE=y
CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE=y
CONFIG_X509_CERTIFICATE_PARSER=y
CONFIG_PKCS7_MESSAGE_PARSER=y
# CONFIG_PKCS7_TEST_KEY is not set
# CONFIG_SIGNED_PE_FILE_VERIFICATION is not set

#
# Certificates for signature checking
#
CONFIG_SYSTEM_TRUSTED_KEYRING=y
CONFIG_SYSTEM_TRUSTED_KEYS=""
# CONFIG_SYSTEM_EXTRA_CERTIFICATE is not set
# CONFIG_SECONDARY_TRUSTED_KEYRING is not set
# CONFIG_SYSTEM_BLACKLIST_KEYRING is not set
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_IRQFD=y
CONFIG_HAVE_KVM_IRQ_ROUTING=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
CONFIG_KVM_VFIO=y
CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=y
CONFIG_KVM_COMPAT=y
CONFIG_HAVE_KVM_IRQ_BYPASS=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
CONFIG_KVM_AMD=m
CONFIG_KVM_MMU_AUDIT=y
CONFIG_VHOST_NET=m
# CONFIG_VHOST_SCSI is not set
# CONFIG_VHOST_VSOCK is not set
CONFIG_VHOST=m
# CONFIG_VHOST_CROSS_ENDIAN_LEGACY is not set
CONFIG_BINARY_PRINTF=y

#
# Library routines
#
CONFIG_RAID6_PQ=m
CONFIG_BITREVERSE=y
# CONFIG_HAVE_ARCH_BITREVERSE is not set
CONFIG_RATIONAL=y
CONFIG_GENERIC_STRNCPY_FROM_USER=y
CONFIG_GENERIC_STRNLEN_USER=y
CONFIG_GENERIC_NET_UTILS=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_PCI_IOMAP=y
CONFIG_GENERIC_IOMAP=y
CONFIG_ARCH_USE_CMPXCHG_LOCKREF=y
CONFIG_ARCH_HAS_FAST_MULTIPLIER=y
CONFIG_CRC_CCITT=y
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
# CONFIG_CRC32_SELFTEST is not set
CONFIG_CRC32_SLICEBY8=y
# CONFIG_CRC32_SLICEBY4 is not set
# CONFIG_CRC32_SARWATE is not set
# CONFIG_CRC32_BIT is not set
# CONFIG_CRC4 is not set
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=y
CONFIG_CRC8=m
CONFIG_XXHASH=m
# CONFIG_AUDIT_ARCH_COMPAT_GENERIC is not set
# CONFIG_RANDOM32_SELFTEST is not set
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_LZ4_DECOMPRESS=y
CONFIG_ZSTD_COMPRESS=m
CONFIG_ZSTD_DECOMPRESS=m
CONFIG_XZ_DEC=y
CONFIG_XZ_DEC_X86=y
CONFIG_XZ_DEC_POWERPC=y
CONFIG_XZ_DEC_IA64=y
CONFIG_XZ_DEC_ARM=y
CONFIG_XZ_DEC_ARMTHUMB=y
CONFIG_XZ_DEC_SPARC=y
CONFIG_XZ_DEC_BCJ=y
# CONFIG_XZ_DEC_TEST is not set
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
CONFIG_DECOMPRESS_XZ=y
CONFIG_DECOMPRESS_LZO=y
CONFIG_DECOMPRESS_LZ4=y
CONFIG_GENERIC_ALLOCATOR=y
CONFIG_REED_SOLOMON=m
CONFIG_REED_SOLOMON_ENC8=y
CONFIG_REED_SOLOMON_DEC8=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_BTREE=y
CONFIG_INTERVAL_TREE=y
CONFIG_RADIX_TREE_MULTIORDER=y
CONFIG_ASSOCIATIVE_ARRAY=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT_MAP=y
CONFIG_HAS_DMA=y
CONFIG_SGL_ALLOC=y
# CONFIG_DMA_DIRECT_OPS is not set
# CONFIG_DMA_VIRT_OPS is not set
CONFIG_CHECK_SIGNATURE=y
CONFIG_CPUMASK_OFFSTACK=y
CONFIG_CPU_RMAP=y
CONFIG_DQL=y
CONFIG_GLOB=y
# CONFIG_GLOB_SELFTEST is not set
CONFIG_NLATTR=y
CONFIG_CLZ_TAB=y
CONFIG_CORDIC=m
# CONFIG_DDR is not set
CONFIG_IRQ_POLL=y
CONFIG_MPILIB=y
CONFIG_SIGNATURE=y
CONFIG_OID_REGISTRY=y
CONFIG_UCS2_STRING=y
CONFIG_FONT_SUPPORT=y
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
# CONFIG_SG_SPLIT is not set
CONFIG_SG_POOL=y
CONFIG_ARCH_HAS_SG_CHAIN=y
CONFIG_ARCH_HAS_PMEM_API=y
CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE=y
CONFIG_SBITMAP=y
# CONFIG_STRING_SELFTEST is not set

[-- Attachment #3: job-script.ksh --]
[-- Type: text/plain, Size: 7081 bytes --]

#!/bin/sh

export_top_env()
{
	export suite='pft'
	export testcase='pft'
	export category='benchmark'
	export runtime=300
	export nr_task=44
	export job_origin='/lkp/lkp/.src-20190204-151053/allot/cyclic:default:stable:linux-4.19.y/lkp-bdw-ep3/pft.yaml'
	export queue_cmdline_keys='branch
commit'
	export queue='validate'
	export testbox='lkp-bdw-ep3'
	export tbox_group='lkp-bdw-ep3'
	export submit_id='5c69b07b0b9a932e5da44a99'
	export job_file='/lkp/jobs/scheduled/lkp-bdw-ep3/pft-performance-50%-300s-ucode=0xb00002e-debian-x86_64-2018-04-03-20190218-77405-q5wdb0-3.yaml'
	export id='a73f2f27d22a2f3393f42b75592f0d4864371c5c'
	export queuer_version='/lkp/lkp/.src-20190216-121343'
	export commit='2c83362734dad8e48ccc0710b5cd2436a0323893'
	export kconfig='x86_64-rhel-7.2'
	export compiler='gcc-7'
	export rootfs='debian-x86_64-2018-04-03.cgz'
	export enqueue_time='2019-02-18 03:05:31 +0800'
	export _id='5c69b07c0b9a932e5da44a9a'
	export _rt='/result/pft/performance-50%-300s-ucode=0xb00002e/lkp-bdw-ep3/debian-x86_64-2018-04-03.cgz/x86_64-rhel-7.2/gcc-7/2c83362734dad8e48ccc0710b5cd2436a0323893'
	export user='lkp'
	export head_commit='e3185123541204ca4f715eeaaa1f9929c09ff3b4'
	export base_commit='7876320f88802b22d4e2daf7eb027dd14175a0f8'
	export branch='stable/linux-4.19.y'
	export result_root='/result/pft/performance-50%-300s-ucode=0xb00002e/lkp-bdw-ep3/debian-x86_64-2018-04-03.cgz/x86_64-rhel-7.2/gcc-7/2c83362734dad8e48ccc0710b5cd2436a0323893/3'
	export scheduler_version='/lkp/lkp/.src-20190216-121343'
	export LKP_SERVER='inn'
	export max_uptime=1500
	export initrd='/osimage/debian/debian-x86_64-2018-04-03.cgz'
	export bootloader_append='root=/dev/ram0
user=lkp
job=/lkp/jobs/scheduled/lkp-bdw-ep3/pft-performance-50%-300s-ucode=0xb00002e-debian-x86_64-2018-04-03-20190218-77405-q5wdb0-3.yaml
ARCH=x86_64
kconfig=x86_64-rhel-7.2
branch=stable/linux-4.19.y
commit=2c83362734dad8e48ccc0710b5cd2436a0323893
BOOT_IMAGE=/pkg/linux/x86_64-rhel-7.2/gcc-7/2c83362734dad8e48ccc0710b5cd2436a0323893/vmlinuz-4.16.0-rc2-00007-g2c83362
max_uptime=1500
RESULT_ROOT=/result/pft/performance-50%-300s-ucode=0xb00002e/lkp-bdw-ep3/debian-x86_64-2018-04-03.cgz/x86_64-rhel-7.2/gcc-7/2c83362734dad8e48ccc0710b5cd2436a0323893/3
LKP_SERVER=inn
debug
apic=debug
sysrq_always_enabled
rcupdate.rcu_cpu_stall_timeout=100
net.ifnames=0
printk.devkmsg=on
panic=-1
softlockup_panic=1
nmi_watchdog=panic
oops=panic
load_ramdisk=2
prompt_ramdisk=0
drbd.minor_count=8
systemd.log_level=err
ignore_loglevel
console=tty0
earlyprintk=ttyS0,115200
console=ttyS0,115200
vga=normal
rw'
	export modules_initrd='/pkg/linux/x86_64-rhel-7.2/gcc-7/2c83362734dad8e48ccc0710b5cd2436a0323893/modules.cgz'
	export bm_initrd='/osimage/deps/debian-x86_64-2018-04-03.cgz/run-ipconfig_2018-04-03.cgz,/osimage/deps/debian-x86_64-2018-04-03.cgz/lkp_2018-04-03.cgz,/osimage/deps/debian-x86_64-2018-04-03.cgz/rsync-rootfs_2018-04-03.cgz,/osimage/deps/debian-x86_64-2018-04-03.cgz/perf_2019-01-01.cgz,/osimage/pkg/debian-x86_64-2018-04-03.cgz/perf-x86_64-e1ef035d272e_2019-01-01.cgz,/osimage/deps/debian-x86_64-2018-04-03.cgz/pft_2018-05-17.cgz,/osimage/pkg/debian-x86_64-2018-04-03.cgz/pft-x86_64-b5cfe1b_2018-05-18.cgz,/osimage/deps/debian-x86_64-2018-04-03.cgz/mpstat_2018-06-19.cgz,/osimage/deps/debian-x86_64-2018-04-03.cgz/turbostat_2018-05-17.cgz,/osimage/pkg/debian-x86_64-2018-04-03.cgz/turbostat-x86_64-d5256b2_2018-05-18.cgz,/osimage/deps/debian-x86_64-2018-04-03.cgz/hw_2016-11-15.cgz'
	export lkp_initrd='/lkp/lkp/lkp-x86_64.cgz'
	export site='inn'
	export LKP_CGI_PORT=80
	export LKP_CIFS_PORT=139
	export repeat_to=4
	export good_samples='200614.53096422766
253185.63281241903
199033.9087635469'
	export schedule_notify_address=
	export model='Broadwell-EP'
	export nr_cpu=88
	export memory='64G'
	export swap_partitions=
	export rootfs_partition='LABEL=LKP-ROOTFS'
	export hdd_partitions='/dev/disk/by-id/ata-ST250DM000-1BD141_W2ADD5CZ-part1'
	export brand='Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz'
	export ucode='0xb00002e'
	export kernel='/pkg/linux/x86_64-rhel-7.2/gcc-7/2c83362734dad8e48ccc0710b5cd2436a0323893/vmlinuz-4.16.0-rc2-00007-g2c83362'
	export dequeue_time='2019-02-18 03:41:24 +0800'
	export job_initrd='/lkp/jobs/scheduled/lkp-bdw-ep3/pft-performance-50%-300s-ucode=0xb00002e-debian-x86_64-2018-04-03-20190218-77405-q5wdb0-3.cgz'

	[ -n "$LKP_SRC" ] ||
	export LKP_SRC=/lkp/${user:-lkp}/src
}

run_job()
{
	echo $$ > $TMP/run-job.pid

	. $LKP_SRC/lib/http.sh
	. $LKP_SRC/lib/job.sh
	. $LKP_SRC/lib/env.sh

	export_top_env

	run_setup $LKP_SRC/setup/cpufreq_governor 'performance'

	run_monitor delay=30 $LKP_SRC/monitors/no-stdout/wrapper perf-profile
	run_monitor $LKP_SRC/monitors/wrapper kmsg
	run_monitor $LKP_SRC/monitors/no-stdout/wrapper boot-time
	run_monitor $LKP_SRC/monitors/wrapper iostat
	run_monitor $LKP_SRC/monitors/wrapper heartbeat
	run_monitor $LKP_SRC/monitors/wrapper vmstat
	run_monitor $LKP_SRC/monitors/wrapper numa-numastat
	run_monitor $LKP_SRC/monitors/wrapper numa-vmstat
	run_monitor $LKP_SRC/monitors/wrapper numa-meminfo
	run_monitor $LKP_SRC/monitors/wrapper proc-vmstat
	run_monitor $LKP_SRC/monitors/wrapper proc-stat
	run_monitor $LKP_SRC/monitors/wrapper meminfo
	run_monitor $LKP_SRC/monitors/wrapper slabinfo
	run_monitor $LKP_SRC/monitors/wrapper interrupts
	run_monitor $LKP_SRC/monitors/wrapper lock_stat
	run_monitor $LKP_SRC/monitors/wrapper latency_stats
	run_monitor $LKP_SRC/monitors/wrapper softirqs
	run_monitor $LKP_SRC/monitors/one-shot/wrapper bdi_dev_mapping
	run_monitor $LKP_SRC/monitors/wrapper diskstats
	run_monitor $LKP_SRC/monitors/wrapper nfsstat
	run_monitor $LKP_SRC/monitors/wrapper cpuidle
	run_monitor $LKP_SRC/monitors/wrapper cpufreq-stats
	run_monitor $LKP_SRC/monitors/wrapper turbostat
	run_monitor $LKP_SRC/monitors/wrapper sched_debug
	run_monitor $LKP_SRC/monitors/wrapper perf-stat
	run_monitor $LKP_SRC/monitors/wrapper mpstat
	run_monitor $LKP_SRC/monitors/wrapper oom-killer
	run_monitor $LKP_SRC/monitors/plain/watchdog

	run_test $LKP_SRC/tests/wrapper pft
}

extract_stats()
{
	export stats_part_begin=
	export stats_part_end=

	$LKP_SRC/stats/wrapper perf-profile
	$LKP_SRC/stats/wrapper pft
	$LKP_SRC/stats/wrapper kmsg
	$LKP_SRC/stats/wrapper boot-time
	$LKP_SRC/stats/wrapper iostat
	$LKP_SRC/stats/wrapper vmstat
	$LKP_SRC/stats/wrapper numa-numastat
	$LKP_SRC/stats/wrapper numa-vmstat
	$LKP_SRC/stats/wrapper numa-meminfo
	$LKP_SRC/stats/wrapper proc-vmstat
	$LKP_SRC/stats/wrapper meminfo
	$LKP_SRC/stats/wrapper slabinfo
	$LKP_SRC/stats/wrapper interrupts
	$LKP_SRC/stats/wrapper lock_stat
	$LKP_SRC/stats/wrapper latency_stats
	$LKP_SRC/stats/wrapper softirqs
	$LKP_SRC/stats/wrapper diskstats
	$LKP_SRC/stats/wrapper nfsstat
	$LKP_SRC/stats/wrapper cpuidle
	$LKP_SRC/stats/wrapper turbostat
	$LKP_SRC/stats/wrapper sched_debug
	$LKP_SRC/stats/wrapper perf-stat
	$LKP_SRC/stats/wrapper mpstat

	$LKP_SRC/stats/wrapper time pft.time
	$LKP_SRC/stats/wrapper time
	$LKP_SRC/stats/wrapper dmesg
	$LKP_SRC/stats/wrapper kmsg
	$LKP_SRC/stats/wrapper stderr
	$LKP_SRC/stats/wrapper last_state
}

"$@"

[-- Attachment #4: job.yaml --]
[-- Type: text/plain, Size: 4538 bytes --]

---
suite: pft
testcase: pft
category: benchmark
perf-profile:
  delay: 30
runtime: 300s
nr_task: 50%
pft: 
job_origin: "/lkp/lkp/.src-20190204-151053/allot/cyclic:default:stable:linux-4.19.y/lkp-bdw-ep3/pft.yaml"
queue_cmdline_keys:
- branch
- commit
queue: bisect
testbox: lkp-bdw-ep3
tbox_group: lkp-bdw-ep3
submit_id: 5c69a69f0b9a93846d9ce7b1
job_file: "/lkp/jobs/scheduled/lkp-bdw-ep3/pft-performance-50%-300s-ucode=0xb00002e-debian-x86_64-2018-04-03.c-20190218-33901-1ollmjl-0.yaml"
id: 203107233327347f56615902c6a0812da89be5e1
queuer_version: "/lkp/lkp/.src-20190216-121343"
kmsg: 
boot-time: 
iostat: 
heartbeat: 
vmstat: 
numa-numastat: 
numa-vmstat: 
numa-meminfo: 
proc-vmstat: 
proc-stat: 
meminfo: 
slabinfo: 
interrupts: 
lock_stat: 
latency_stats: 
softirqs: 
bdi_dev_mapping: 
diskstats: 
nfsstat: 
cpuidle: 
cpufreq-stats: 
turbostat: 
sched_debug: 
perf-stat: 
mpstat: 
cpufreq_governor: performance
commit: 2c83362734dad8e48ccc0710b5cd2436a0323893
kconfig: x86_64-rhel-7.2
compiler: gcc-7
rootfs: debian-x86_64-2018-04-03.cgz
enqueue_time: 2019-02-18 02:23:27.858707861 +08:00
_id: 5c69a69f0b9a93846d9ce7b1
_rt: "/result/pft/performance-50%-300s-ucode=0xb00002e/lkp-bdw-ep3/debian-x86_64-2018-04-03.cgz/x86_64-rhel-7.2/gcc-7/2c83362734dad8e48ccc0710b5cd2436a0323893"
user: lkp
head_commit: e3185123541204ca4f715eeaaa1f9929c09ff3b4
base_commit: 7876320f88802b22d4e2daf7eb027dd14175a0f8
branch: stable/linux-4.19.y
result_root: "/result/pft/performance-50%-300s-ucode=0xb00002e/lkp-bdw-ep3/debian-x86_64-2018-04-03.cgz/x86_64-rhel-7.2/gcc-7/2c83362734dad8e48ccc0710b5cd2436a0323893/0"
scheduler_version: "/lkp/lkp/.src-20190216-121343"
LKP_SERVER: inn
max_uptime: 1500
initrd: "/osimage/debian/debian-x86_64-2018-04-03.cgz"
bootloader_append:
- root=/dev/ram0
- user=lkp
- job=/lkp/jobs/scheduled/lkp-bdw-ep3/pft-performance-50%-300s-ucode=0xb00002e-debian-x86_64-2018-04-03.c-20190218-33901-1ollmjl-0.yaml
- ARCH=x86_64
- kconfig=x86_64-rhel-7.2
- branch=stable/linux-4.19.y
- commit=2c83362734dad8e48ccc0710b5cd2436a0323893
- BOOT_IMAGE=/pkg/linux/x86_64-rhel-7.2/gcc-7/2c83362734dad8e48ccc0710b5cd2436a0323893/vmlinuz-4.16.0-rc2-00007-g2c83362
- max_uptime=1500
- RESULT_ROOT=/result/pft/performance-50%-300s-ucode=0xb00002e/lkp-bdw-ep3/debian-x86_64-2018-04-03.cgz/x86_64-rhel-7.2/gcc-7/2c83362734dad8e48ccc0710b5cd2436a0323893/0
- LKP_SERVER=inn
- debug
- apic=debug
- sysrq_always_enabled
- rcupdate.rcu_cpu_stall_timeout=100
- net.ifnames=0
- printk.devkmsg=on
- panic=-1
- softlockup_panic=1
- nmi_watchdog=panic
- oops=panic
- load_ramdisk=2
- prompt_ramdisk=0
- drbd.minor_count=8
- systemd.log_level=err
- ignore_loglevel
- console=tty0
- earlyprintk=ttyS0,115200
- console=ttyS0,115200
- vga=normal
- rw
modules_initrd: "/pkg/linux/x86_64-rhel-7.2/gcc-7/2c83362734dad8e48ccc0710b5cd2436a0323893/modules.cgz"
bm_initrd: "/osimage/deps/debian-x86_64-2018-04-03.cgz/run-ipconfig_2018-04-03.cgz,/osimage/deps/debian-x86_64-2018-04-03.cgz/lkp_2018-04-03.cgz,/osimage/deps/debian-x86_64-2018-04-03.cgz/rsync-rootfs_2018-04-03.cgz,/osimage/deps/debian-x86_64-2018-04-03.cgz/perf_2019-01-01.cgz,/osimage/pkg/debian-x86_64-2018-04-03.cgz/perf-x86_64-e1ef035d272e_2019-01-01.cgz,/osimage/deps/debian-x86_64-2018-04-03.cgz/pft_2018-05-17.cgz,/osimage/pkg/debian-x86_64-2018-04-03.cgz/pft-x86_64-b5cfe1b_2018-05-18.cgz,/osimage/deps/debian-x86_64-2018-04-03.cgz/mpstat_2018-06-19.cgz,/osimage/deps/debian-x86_64-2018-04-03.cgz/turbostat_2018-05-17.cgz,/osimage/pkg/debian-x86_64-2018-04-03.cgz/turbostat-x86_64-d5256b2_2018-05-18.cgz,/osimage/deps/debian-x86_64-2018-04-03.cgz/hw_2016-11-15.cgz"
lkp_initrd: "/lkp/lkp/lkp-x86_64.cgz"
site: inn
LKP_CGI_PORT: 80
LKP_CIFS_PORT: 139
oom-killer: 
watchdog: 
repeat_to: 2
good_samples:
- 200614.53096422766
- 253185.63281241903
- 199033.9087635469

#! queue options
schedule_notify_address: 

#! hosts/lkp-bdw-ep3
model: Broadwell-EP
nr_cpu: 88
memory: 64G
swap_partitions: 
rootfs_partition: LABEL=LKP-ROOTFS
hdd_partitions: "/dev/disk/by-id/ata-ST250DM000-1BD141_W2ADD5CZ-part1"
brand: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
ucode: '0xb00002e'

#! user overrides

#! schedule options
kernel: "/pkg/linux/x86_64-rhel-7.2/gcc-7/2c83362734dad8e48ccc0710b5cd2436a0323893/vmlinuz-4.16.0-rc2-00007-g2c83362"
dequeue_time: 2019-02-18 02:37:47.959031905 +08:00

#! /lkp/lkp/.src-20190216-121343/include/site/inn

#! runtime status
job_state: finished
loadavg: 34.83 25.14 11.21 2/741 33530
start_time: '1550428734'
end_time: '1550429035'
version: "/lkp/lkp/.src-20190216-121343"

[-- Attachment #5: reproduce.ksh --]
[-- Type: text/plain, Size: 16156 bytes --]


for cpu_dir in /sys/devices/system/cpu/cpu[0-9]*
do
	online_file="$cpu_dir"/online
	[ -f "$online_file" ] && [ "$(cat "$online_file")" -eq 0 ] && continue

	file="$cpu_dir"/cpufreq/scaling_governor
	[ -f "$file" ] && echo "performance" > "$file"
done

 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F
 ./pft -m 13743895347 -n 44 -F

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [LKP] [sched/fair]  2c83362734:  pft.faults_per_sec_per_cpu -41.4% regression
  2019-02-28  7:17 [sched/fair] 2c83362734: pft.faults_per_sec_per_cpu -41.4% regression kernel test robot
@ 2019-02-28 11:10   ` Mel Gorman
  0 siblings, 0 replies; 4+ messages in thread
From: Mel Gorman @ 2019-02-28 11:10 UTC (permalink / raw)
  To: kernel test robot
  Cc: Ingo Molnar, Peter Zijlstra, Giovanni Gherdovich, Linus Torvalds,
	Matt Fleming, Mike Galbraith, Thomas Gleixner, LKML, lkp

On Thu, Feb 28, 2019 at 03:17:51PM +0800, kernel test robot wrote:
> Greeting,
> 
> FYI, we noticed a -41.4% regression of pft.faults_per_sec_per_cpu due to commit:
> 
> 
> commit: 2c83362734dad8e48ccc0710b5cd2436a0323893 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> 
> in testcase: pft
> on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory
> with following parameters:
> 
> 	runtime: 300s
> 	nr_task: 50%
> 	cpufreq_governor: performance
> 	ucode: 0xb00002e
> 

The headline regression looks high but it's also a known consequence for
some microbenchmarks, particularly those that are short-lived and consist
of non-communicating tasks.

The impact of the patch is to favour starting a new task on the local
node unless the socket is saturated. This is to avoid a pattern where a
task that clones a helper that it communicates with starts on a remote
node. Starting remote negatively impacts basis workloads like
shellscripts, client/server workloads or pipelined tasks. The workloads
that benefit from spreading early are parallelised tasks that do not
communicate until end of the task.

PFT is an example of the latter. If spread early, it maximises the total
memory bandwidth of the machine early in the lifetime of the machine. It
would quickly recover if it run long enough, the early measurements are
low as it saturates the bandwidth of the local node. This configuration
is at 50% and the machine is likely to be 2-socket so it has half the
bandwidth in all likelihood and hence the 41.4% regression (very close
to half so some tasks probably got load-balanced).

On to the other examples;

> test-description: Pft is the page fault test micro benchmark.
> test-url: https://github.com/gormanm/pft
> 
> In addition to that, the commit also has significant impact on the following tests:
> 
> +------------------+--------------------------------------------------------------------------+
> | testcase: change | stream:                                                                  |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
> | test parameters  | array_size=10000000                                                      |
> |                  | cpufreq_governor=performance                                             |
> |                  | nr_threads=25%                                                           |
> |                  | omp=true                                                                 |
> |                  | ucode=0xb00002e                                                          |

STREAM is typically short-lived. Again, it benefits from spreading early
to maximise memory bandwidth. 25% of threads would fit in one node. For
parallelised stream tests it's usually the case that OMP is used to bind 1
thread per memory channel using the openmp directives to measure the total
machine memory bandwidth rather than using it as a scaling tests. I'm
guessing this machine didn't have 22 memory channels that would make
nr_thread=25% a sensible configuration.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | reaim: reaim.jobs_per_min 1.3% improvement                               |
> | test machine     | 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory    |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | nr_job=3000                                                              |
> |                  | nr_task=100%                                                             |
> |                  | runtime=300s                                                             |
> |                  | test=custom                                                              |
> |                  | ucode=0x3d                                                               |

reaim is generally a mess so in this case it's unclear. The load is a
mix of task creation, IO operations, signal and others. It might have
benefitted slightly from running local. One reason I don't particularly
like reaim is that historically it was dominated by sending/receiving
signals. In my own tests, signal is typically removed as well as it's
tendency to sync the entire filesystem at high frequency.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | stream: stream.add_bandwidth_MBps -32.0% regression                      |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
> | test parameters  | array_size=10000000                                                      |
> |                  | cpufreq_governor=performance                                             |
> |                  | nr_threads=50%                                                           |
> |                  | omp=true                                                                 |
> |                  | ucode=0xb00002e                                                          |

STREAM covered already other than noting that it's unlikely it has 44
memory channels to work with so any imbalance in the task distribution
should show up as a regression. Again, the patch favours using local node
first which would saturate the local memory channel earlier.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | plzip:                                                                   |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | nr_threads=100%                                                          |
> |                  | ucode=0xb00002e                                                          |

Doesn't state what change happened be it positive or negative.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | reaim: reaim.jobs_per_min -11.9% regression                              |
> | test machine     | 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 512G memory   |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | nr_task=100%                                                             |
> |                  | runtime=300s                                                             |
> |                  | test=all_utime                                                           |
> |                  | ucode=0xb00002e                                                          |

This is completely user-space bound running basic math operations. Not
clear why it would suffer *but* if hyperthreading is enabled, the patch
might mean that hyperthread siblings were used early due to favouring
the local node.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | hackbench: hackbench.throughput -7.3% regression                         |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory     |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | ipc=pipe                                                                 |
> |                  | mode=process                                                             |
> |                  | nr_threads=1600%                                                         |
> |                  | ucode=0xb00002e                                                          |

Hackbench very short-lived but the workload is also heavily saturating the
machine to an extent where it would be hard to tell from this report if
the 7.3% is statically significant or not. The patch might mean a socket
is severely over-saturated in the very early phases of the workload.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | reaim: reaim.std_dev_percent 11.4% undefined                             |
> | test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | nr_task=100%                                                             |
> |                  | runtime=300s                                                             |
> |                  | test=custom                                                              |
> |                  | ucode=0x200004d                                                          |

Not sure what the change is saying. Possibly that it's less variable.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | reaim: boot-time.boot 95.3% regression                                   |
> | test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | nr_task=100%                                                             |
> |                  | runtime=300s                                                             |
> |                  | test=alltests                                                            |
> |                  | ucode=0x200004d                                                          |

boot-time.boot?

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | pft: pft.faults_per_sec_per_cpu -42.7% regression                        |
> | test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | nr_task=50%                                                              |
> |                  | runtime=300s                                                             |
> |                  | ucode=0x200004d                                                          |

PFT already discussed.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | stream: stream.add_bandwidth_MBps -28.8% regression                      |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
> | test parameters  | array_size=50000000                                                      |
> |                  | cpufreq_governor=performance                                             |
> |                  | nr_threads=50%                                                           |
> |                  | omp=true                                                                 |
> +------------------+--------------------------------------------------------------------------+
> | testcase: change | stream: stream.add_bandwidth_MBps -30.6% regression                      |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
> | test parameters  | array_size=10000000                                                      |
> |                  | cpufreq_governor=performance                                             |
> |                  | nr_threads=50%                                                           |
> |                  | omp=true                                                                 |
> +------------------+--------------------------------------------------------------------------+
> | testcase: change | pft: pft.faults_per_sec_per_cpu -42.5% regression                        |
> | test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | nr_task=50%                                                              |
> |                  | runtime=300s                                                             |

Already discussed.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | reaim: reaim.child_systime -1.4% undefined                               |
> | test machine     | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory   |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | iterations=30                                                            |
> |                  | nr_task=1600%                                                            |
> |                  | test=compute                                                             |

1.4% change in system time could be overhead in the fork phase as it
looks for local idle cores then remote idle cores early but the
difference is tiny.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | stress-ng: stress-ng.fifo.ops_per_sec 76.2% improvement                  |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
> | test parameters  | class=pipe                                                               |
> |                  | cpufreq_governor=performance                                             |
> |                  | nr_threads=100%                                                          |
> |                  | testtime=1s                                                              |

A case where short-lived communicating tasks benefit by starting local.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | stress-ng: stress-ng.tsearch.ops_per_sec -17.1% regression               |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
> | test parameters  | class=cpu                                                                |
> |                  | cpufreq_governor=performance                                             |
> |                  | nr_threads=100%                                                          |
> |                  | testtime=1s                                                              |
> +------------------+--------------------------------------------------------------------------+
> 

Given full machine utilisation and a 1 second duration, it's a case
where saturating the local node early was sub-optimal and 1 second is
too long for load balancing or other factors to correct it.

Bottom line, the patch is a trade off but from a range of tests, I found
that on balance we benefit more from having tasks start local until
there is evidence that the kernel is justified to spread the load to
remote nodes.

-- 
Mel Gorman
SUSE Labs

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [sched/fair] 2c83362734: pft.faults_per_sec_per_cpu -41.4% regression
@ 2019-02-28 11:10   ` Mel Gorman
  0 siblings, 0 replies; 4+ messages in thread
From: Mel Gorman @ 2019-02-28 11:10 UTC (permalink / raw)
  To: lkp

[-- Attachment #1: Type: text/plain, Size: 15724 bytes --]

On Thu, Feb 28, 2019 at 03:17:51PM +0800, kernel test robot wrote:
> Greeting,
> 
> FYI, we noticed a -41.4% regression of pft.faults_per_sec_per_cpu due to commit:
> 
> 
> commit: 2c83362734dad8e48ccc0710b5cd2436a0323893 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> 
> in testcase: pft
> on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory
> with following parameters:
> 
> 	runtime: 300s
> 	nr_task: 50%
> 	cpufreq_governor: performance
> 	ucode: 0xb00002e
> 

The headline regression looks high but it's also a known consequence for
some microbenchmarks, particularly those that are short-lived and consist
of non-communicating tasks.

The impact of the patch is to favour starting a new task on the local
node unless the socket is saturated. This is to avoid a pattern where a
task that clones a helper that it communicates with starts on a remote
node. Starting remote negatively impacts basis workloads like
shellscripts, client/server workloads or pipelined tasks. The workloads
that benefit from spreading early are parallelised tasks that do not
communicate until end of the task.

PFT is an example of the latter. If spread early, it maximises the total
memory bandwidth of the machine early in the lifetime of the machine. It
would quickly recover if it run long enough, the early measurements are
low as it saturates the bandwidth of the local node. This configuration
is at 50% and the machine is likely to be 2-socket so it has half the
bandwidth in all likelihood and hence the 41.4% regression (very close
to half so some tasks probably got load-balanced).

On to the other examples;

> test-description: Pft is the page fault test micro benchmark.
> test-url: https://github.com/gormanm/pft
> 
> In addition to that, the commit also has significant impact on the following tests:
> 
> +------------------+--------------------------------------------------------------------------+
> | testcase: change | stream:                                                                  |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
> | test parameters  | array_size=10000000                                                      |
> |                  | cpufreq_governor=performance                                             |
> |                  | nr_threads=25%                                                           |
> |                  | omp=true                                                                 |
> |                  | ucode=0xb00002e                                                          |

STREAM is typically short-lived. Again, it benefits from spreading early
to maximise memory bandwidth. 25% of threads would fit in one node. For
parallelised stream tests it's usually the case that OMP is used to bind 1
thread per memory channel using the openmp directives to measure the total
machine memory bandwidth rather than using it as a scaling tests. I'm
guessing this machine didn't have 22 memory channels that would make
nr_thread=25% a sensible configuration.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | reaim: reaim.jobs_per_min 1.3% improvement                               |
> | test machine     | 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory    |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | nr_job=3000                                                              |
> |                  | nr_task=100%                                                             |
> |                  | runtime=300s                                                             |
> |                  | test=custom                                                              |
> |                  | ucode=0x3d                                                               |

reaim is generally a mess so in this case it's unclear. The load is a
mix of task creation, IO operations, signal and others. It might have
benefitted slightly from running local. One reason I don't particularly
like reaim is that historically it was dominated by sending/receiving
signals. In my own tests, signal is typically removed as well as it's
tendency to sync the entire filesystem at high frequency.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | stream: stream.add_bandwidth_MBps -32.0% regression                      |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
> | test parameters  | array_size=10000000                                                      |
> |                  | cpufreq_governor=performance                                             |
> |                  | nr_threads=50%                                                           |
> |                  | omp=true                                                                 |
> |                  | ucode=0xb00002e                                                          |

STREAM covered already other than noting that it's unlikely it has 44
memory channels to work with so any imbalance in the task distribution
should show up as a regression. Again, the patch favours using local node
first which would saturate the local memory channel earlier.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | plzip:                                                                   |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | nr_threads=100%                                                          |
> |                  | ucode=0xb00002e                                                          |

Doesn't state what change happened be it positive or negative.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | reaim: reaim.jobs_per_min -11.9% regression                              |
> | test machine     | 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 512G memory   |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | nr_task=100%                                                             |
> |                  | runtime=300s                                                             |
> |                  | test=all_utime                                                           |
> |                  | ucode=0xb00002e                                                          |

This is completely user-space bound running basic math operations. Not
clear why it would suffer *but* if hyperthreading is enabled, the patch
might mean that hyperthread siblings were used early due to favouring
the local node.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | hackbench: hackbench.throughput -7.3% regression                         |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory     |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | ipc=pipe                                                                 |
> |                  | mode=process                                                             |
> |                  | nr_threads=1600%                                                         |
> |                  | ucode=0xb00002e                                                          |

Hackbench very short-lived but the workload is also heavily saturating the
machine to an extent where it would be hard to tell from this report if
the 7.3% is statically significant or not. The patch might mean a socket
is severely over-saturated in the very early phases of the workload.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | reaim: reaim.std_dev_percent 11.4% undefined                             |
> | test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | nr_task=100%                                                             |
> |                  | runtime=300s                                                             |
> |                  | test=custom                                                              |
> |                  | ucode=0x200004d                                                          |

Not sure what the change is saying. Possibly that it's less variable.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | reaim: boot-time.boot 95.3% regression                                   |
> | test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | nr_task=100%                                                             |
> |                  | runtime=300s                                                             |
> |                  | test=alltests                                                            |
> |                  | ucode=0x200004d                                                          |

boot-time.boot?

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | pft: pft.faults_per_sec_per_cpu -42.7% regression                        |
> | test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | nr_task=50%                                                              |
> |                  | runtime=300s                                                             |
> |                  | ucode=0x200004d                                                          |

PFT already discussed.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | stream: stream.add_bandwidth_MBps -28.8% regression                      |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
> | test parameters  | array_size=50000000                                                      |
> |                  | cpufreq_governor=performance                                             |
> |                  | nr_threads=50%                                                           |
> |                  | omp=true                                                                 |
> +------------------+--------------------------------------------------------------------------+
> | testcase: change | stream: stream.add_bandwidth_MBps -30.6% regression                      |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
> | test parameters  | array_size=10000000                                                      |
> |                  | cpufreq_governor=performance                                             |
> |                  | nr_threads=50%                                                           |
> |                  | omp=true                                                                 |
> +------------------+--------------------------------------------------------------------------+
> | testcase: change | pft: pft.faults_per_sec_per_cpu -42.5% regression                        |
> | test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | nr_task=50%                                                              |
> |                  | runtime=300s                                                             |

Already discussed.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | reaim: reaim.child_systime -1.4% undefined                               |
> | test machine     | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory   |
> | test parameters  | cpufreq_governor=performance                                             |
> |                  | iterations=30                                                            |
> |                  | nr_task=1600%                                                            |
> |                  | test=compute                                                             |

1.4% change in system time could be overhead in the fork phase as it
looks for local idle cores then remote idle cores early but the
difference is tiny.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | stress-ng: stress-ng.fifo.ops_per_sec 76.2% improvement                  |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
> | test parameters  | class=pipe                                                               |
> |                  | cpufreq_governor=performance                                             |
> |                  | nr_threads=100%                                                          |
> |                  | testtime=1s                                                              |

A case where short-lived communicating tasks benefit by starting local.

> +------------------+--------------------------------------------------------------------------+
> | testcase: change | stress-ng: stress-ng.tsearch.ops_per_sec -17.1% regression               |
> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
> | test parameters  | class=cpu                                                                |
> |                  | cpufreq_governor=performance                                             |
> |                  | nr_threads=100%                                                          |
> |                  | testtime=1s                                                              |
> +------------------+--------------------------------------------------------------------------+
> 

Given full machine utilisation and a 1 second duration, it's a case
where saturating the local node early was sub-optimal and 1 second is
too long for load balancing or other factors to correct it.

Bottom line, the patch is a trade off but from a range of tests, I found
that on balance we benefit more from having tasks start local until
there is evidence that the kernel is justified to spread the load to
remote nodes.

-- 
Mel Gorman
SUSE Labs

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [sched/fair] 2c83362734: pft.faults_per_sec_per_cpu -41.4% regression
  2019-02-28 11:10   ` Mel Gorman
  (?)
@ 2019-03-02  8:15   ` Huang, Ying
  -1 siblings, 0 replies; 4+ messages in thread
From: Huang, Ying @ 2019-03-02  8:15 UTC (permalink / raw)
  To: lkp

[-- Attachment #1: Type: text/plain, Size: 16104 bytes --]

Mel Gorman <mgorman@techsingularity.net> writes:

> On Thu, Feb 28, 2019 at 03:17:51PM +0800, kernel test robot wrote:
>> Greeting,
>> 
>> FYI, we noticed a -41.4% regression of pft.faults_per_sec_per_cpu due to commit:
>> 
>> 
>> commit: 2c83362734dad8e48ccc0710b5cd2436a0323893 ("sched/fair: Consider SD_NUMA when selecting the most idle group to schedule on")
>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>> 
>> in testcase: pft
>> on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory
>> with following parameters:
>> 
>> 	runtime: 300s
>> 	nr_task: 50%
>> 	cpufreq_governor: performance
>> 	ucode: 0xb00002e
>> 
>
> The headline regression looks high but it's also a known consequence for
> some microbenchmarks, particularly those that are short-lived and consist
> of non-communicating tasks.
>
> The impact of the patch is to favour starting a new task on the local
> node unless the socket is saturated. This is to avoid a pattern where a
> task that clones a helper that it communicates with starts on a remote
> node. Starting remote negatively impacts basis workloads like
> shellscripts, client/server workloads or pipelined tasks. The workloads
> that benefit from spreading early are parallelised tasks that do not
> communicate until end of the task.
>
> PFT is an example of the latter. If spread early, it maximises the total
> memory bandwidth of the machine early in the lifetime of the machine. It
> would quickly recover if it run long enough, the early measurements are
> low as it saturates the bandwidth of the local node. This configuration
> is at 50% and the machine is likely to be 2-socket so it has half the
> bandwidth in all likelihood and hence the 41.4% regression (very close
> to half so some tasks probably got load-balanced).
>
> On to the other examples;
>
>> test-description: Pft is the page fault test micro benchmark.
>> test-url: https://github.com/gormanm/pft
>> 
>> In addition to that, the commit also has significant impact on the following tests:
>> 
>> +------------------+--------------------------------------------------------------------------+
>> | testcase: change | stream:                                                                  |
>> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
>> | test parameters  | array_size=10000000                                                      |
>> |                  | cpufreq_governor=performance                                             |
>> |                  | nr_threads=25%                                                           |
>> |                  | omp=true                                                                 |
>> |                  | ucode=0xb00002e                                                          |
>
> STREAM is typically short-lived. Again, it benefits from spreading early
> to maximise memory bandwidth. 25% of threads would fit in one node. For
> parallelised stream tests it's usually the case that OMP is used to bind 1
> thread per memory channel using the openmp directives to measure the total
> machine memory bandwidth rather than using it as a scaling tests. I'm
> guessing this machine didn't have 22 memory channels that would make
> nr_thread=25% a sensible configuration.
>
>> +------------------+--------------------------------------------------------------------------+
>> | testcase: change | reaim: reaim.jobs_per_min 1.3% improvement                               |
>> | test machine     | 72 threads Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz with 256G memory    |
>> | test parameters  | cpufreq_governor=performance                                             |
>> |                  | nr_job=3000                                                              |
>> |                  | nr_task=100%                                                             |
>> |                  | runtime=300s                                                             |
>> |                  | test=custom                                                              |
>> |                  | ucode=0x3d                                                               |
>
> reaim is generally a mess so in this case it's unclear. The load is a
> mix of task creation, IO operations, signal and others. It might have
> benefitted slightly from running local. One reason I don't particularly
> like reaim is that historically it was dominated by sending/receiving
> signals. In my own tests, signal is typically removed as well as it's
> tendency to sync the entire filesystem at high frequency.
>
>> +------------------+--------------------------------------------------------------------------+
>> | testcase: change | stream: stream.add_bandwidth_MBps -32.0% regression                      |
>> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
>> | test parameters  | array_size=10000000                                                      |
>> |                  | cpufreq_governor=performance                                             |
>> |                  | nr_threads=50%                                                           |
>> |                  | omp=true                                                                 |
>> |                  | ucode=0xb00002e                                                          |
>
> STREAM covered already other than noting that it's unlikely it has 44
> memory channels to work with so any imbalance in the task distribution
> should show up as a regression. Again, the patch favours using local node
> first which would saturate the local memory channel earlier.
>
>> +------------------+--------------------------------------------------------------------------+
>> | testcase: change | plzip:                                                                   |
>> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
>> | test parameters  | cpufreq_governor=performance                                             |
>> |                  | nr_threads=100%                                                          |
>> |                  | ucode=0xb00002e                                                          |
>
> Doesn't state what change happened be it positive or negative.
>
>> +------------------+--------------------------------------------------------------------------+
>> | testcase: change | reaim: reaim.jobs_per_min -11.9% regression                              |
>> | test machine     | 192 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 512G memory   |
>> | test parameters  | cpufreq_governor=performance                                             |
>> |                  | nr_task=100%                                                             |
>> |                  | runtime=300s                                                             |
>> |                  | test=all_utime                                                           |
>> |                  | ucode=0xb00002e                                                          |
>
> This is completely user-space bound running basic math operations. Not
> clear why it would suffer *but* if hyperthreading is enabled, the patch
> might mean that hyperthread siblings were used early due to favouring
> the local node.
>
>> +------------------+--------------------------------------------------------------------------+
>> | testcase: change | hackbench: hackbench.throughput -7.3% regression                         |
>> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory     |
>> | test parameters  | cpufreq_governor=performance                                             |
>> |                  | ipc=pipe                                                                 |
>> |                  | mode=process                                                             |
>> |                  | nr_threads=1600%                                                         |
>> |                  | ucode=0xb00002e                                                          |
>
> Hackbench very short-lived but the workload is also heavily saturating the
> machine to an extent where it would be hard to tell from this report if
> the 7.3% is statically significant or not. The patch might mean a socket
> is severely over-saturated in the very early phases of the workload.
>
>> +------------------+--------------------------------------------------------------------------+
>> | testcase: change | reaim: reaim.std_dev_percent 11.4% undefined                             |
>> | test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
>> | test parameters  | cpufreq_governor=performance                                             |
>> |                  | nr_task=100%                                                             |
>> |                  | runtime=300s                                                             |
>> |                  | test=custom                                                              |
>> |                  | ucode=0x200004d                                                          |
>
> Not sure what the change is saying. Possibly that it's less variable.
>
>> +------------------+--------------------------------------------------------------------------+
>> | testcase: change | reaim: boot-time.boot 95.3% regression                                   |
>> | test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
>> | test parameters  | cpufreq_governor=performance                                             |
>> |                  | nr_task=100%                                                             |
>> |                  | runtime=300s                                                             |
>> |                  | test=alltests                                                            |
>> |                  | ucode=0x200004d                                                          |
>
> boot-time.boot?
>
>> +------------------+--------------------------------------------------------------------------+
>> | testcase: change | pft: pft.faults_per_sec_per_cpu -42.7% regression                        |
>> | test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
>> | test parameters  | cpufreq_governor=performance                                             |
>> |                  | nr_task=50%                                                              |
>> |                  | runtime=300s                                                             |
>> |                  | ucode=0x200004d                                                          |
>
> PFT already discussed.
>
>> +------------------+--------------------------------------------------------------------------+
>> | testcase: change | stream: stream.add_bandwidth_MBps -28.8% regression                      |
>> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
>> | test parameters  | array_size=50000000                                                      |
>> |                  | cpufreq_governor=performance                                             |
>> |                  | nr_threads=50%                                                           |
>> |                  | omp=true                                                                 |
>> +------------------+--------------------------------------------------------------------------+
>> | testcase: change | stream: stream.add_bandwidth_MBps -30.6% regression                      |
>> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
>> | test parameters  | array_size=10000000                                                      |
>> |                  | cpufreq_governor=performance                                             |
>> |                  | nr_threads=50%                                                           |
>> |                  | omp=true                                                                 |
>> +------------------+--------------------------------------------------------------------------+
>> | testcase: change | pft: pft.faults_per_sec_per_cpu -42.5% regression                        |
>> | test machine     | 104 threads Intel(R) Xeon(R) Platinum 8170 CPU @ 2.10GHz with 64G memory |
>> | test parameters  | cpufreq_governor=performance                                             |
>> |                  | nr_task=50%                                                              |
>> |                  | runtime=300s                                                             |
>
> Already discussed.
>
>> +------------------+--------------------------------------------------------------------------+
>> | testcase: change | reaim: reaim.child_systime -1.4% undefined                               |
>> | test machine     | 144 threads Intel(R) Xeon(R) CPU E7-8890 v3 @ 2.50GHz with 512G memory   |
>> | test parameters  | cpufreq_governor=performance                                             |
>> |                  | iterations=30                                                            |
>> |                  | nr_task=1600%                                                            |
>> |                  | test=compute                                                             |
>
> 1.4% change in system time could be overhead in the fork phase as it
> looks for local idle cores then remote idle cores early but the
> difference is tiny.
>
>> +------------------+--------------------------------------------------------------------------+
>> | testcase: change | stress-ng: stress-ng.fifo.ops_per_sec 76.2% improvement                  |
>> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
>> | test parameters  | class=pipe                                                               |
>> |                  | cpufreq_governor=performance                                             |
>> |                  | nr_threads=100%                                                          |
>> |                  | testtime=1s                                                              |
>
> A case where short-lived communicating tasks benefit by starting local.
>
>> +------------------+--------------------------------------------------------------------------+
>> | testcase: change | stress-ng: stress-ng.tsearch.ops_per_sec -17.1% regression               |
>> | test machine     | 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 128G memory    |
>> | test parameters  | class=cpu                                                                |
>> |                  | cpufreq_governor=performance                                             |
>> |                  | nr_threads=100%                                                          |
>> |                  | testtime=1s                                                              |
>> +------------------+--------------------------------------------------------------------------+
>> 
>
> Given full machine utilisation and a 1 second duration, it's a case
> where saturating the local node early was sub-optimal and 1 second is
> too long for load balancing or other factors to correct it.
>
> Bottom line, the patch is a trade off but from a range of tests, I found
> that on balance we benefit more from having tasks start local until
> there is evidence that the kernel is justified to spread the load to
> remote nodes.

Thanks a lot for detailed explanation!

Best Regards,
Huang, Ying

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2019-03-02  8:15 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-02-28  7:17 [sched/fair] 2c83362734: pft.faults_per_sec_per_cpu -41.4% regression kernel test robot
2019-02-28 11:10 ` [LKP] " Mel Gorman
2019-02-28 11:10   ` Mel Gorman
2019-03-02  8:15   ` Huang, Ying

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.