[PATCH v4 0/4] make vm_committed_as_batch aware of vm overcommit policy

[PATCH v4 0/4] make vm_committed_as_batch aware of vm overcommit policy

[Feng Tang]

When checking a performance change for will-it-scale scalability
mmap test [1], we found very high lock contention for spinlock of
percpu counter 'vm_committed_as':

    94.14%     0.35%  [kernel.kallsyms]         [k] _raw_spin_lock_irqsave
    48.21% _raw_spin_lock_irqsave;percpu_counter_add_batch;__vm_enough_memory;mmap_region;do_mmap;
    45.91% _raw_spin_lock_irqsave;percpu_counter_add_batch;__do_munmap;

Actually this heavy lock contention is not always necessary. The
'vm_committed_as' needs to be very precise when the strict
OVERCOMMIT_NEVER policy is set, which requires a rather small batch
number for the percpu counter.

So keep 'batch' number unchanged for strict OVERCOMMIT_NEVER policy,
and enlarge it for not-so-strict  OVERCOMMIT_ALWAYS and OVERCOMMIT_GUESS
policies.

Benchmark with the same testcase in [1] shows 53% improvement on a
8C/16T desktop, and 2097%(20X) on a 4S/72C/144T server. And for that
case, whether it shows improvements depends on if the test mmap size
is bigger than the batch number computed.

We tested 10+ platforms in 0day (server, desktop and laptop). If we
lift it to 64X, 80%+ platforms show improvements, and for 16X lift,
1/3 of the platforms will show improvements.

And generally it should help the mmap/unmap usage,as Michal Hocko
mentioned:
"
I believe that there are non-synthetic worklaods which would benefit
from a larger batch. E.g. large in memory databases which do large
mmaps during startups from multiple threads.
"

Note: There are some style complain from checkpatch for patch 3,
as sysctl handler declaration follows the similar format of sibling
functions

patch1: a cleanup for /proc/meminfo
patch2: a preparation patch which also improve the accuracy of
        vm_memory_committed
patch3: remove the VM_WARN_ONCE for vm_committed_as underflow check
patch4: main change

This is against today's linux-mm git tree on github.

Please help to review, thanks!

- Feng

----------------------------------------------------------------
Changelog:

  v4:
    * Remove the VM_WARN_ONCE check for vm_committed_as underflow,
      thanks to Qian Cai for finding and testing the warning

  v3:
    * refine commit log and cleanup code, according to comments
      from Michal Hocko and Matthew Wilcox
    * change the lift from 16X and 64X after test 
  
  v2:
     * add the sysctl handler to cover runtime overcommit policy
       change, as suggested by Andres Morton 
     * address the accuracy concern of vm_memory_committed()
       from Andi Kleen 


Feng Tang (4):
  proc/meminfo: avoid open coded reading of vm_committed_as
  mm/util.c: make vm_memory_committed() more accurate
  mm/util.c: remove the VM_WARN_ONCE for vm_committed_as underflow check
  mm: adjust vm_committed_as_batch according to vm overcommit policy

 fs/proc/meminfo.c    |  2 +-
 include/linux/mm.h   |  2 ++
 include/linux/mman.h |  4 ++++
 kernel/sysctl.c      |  2 +-
 mm/mm_init.c         | 18 ++++++++++++++----
 mm/util.c            | 22 +++++++++++++---------
 6 files changed, 35 insertions(+), 15 deletions(-)

-- 
2.7.4

[PATCH v4 1/4] proc/meminfo: avoid open coded reading of vm_committed_as

[Feng Tang]

Use the existing vm_memory_committed() instead, which is also
convenient for future change.

Signed-off-by: Feng Tang <feng.tang@intel.com>
Acked-by: Michal Hocko <mhocko@suse.com>
---
 fs/proc/meminfo.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c
index b030d8b..e3d14ee 100644
--- a/fs/proc/meminfo.c
+++ b/fs/proc/meminfo.c
@@ -41,7 +41,7 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
 
 	si_meminfo(&i);
 	si_swapinfo(&i);
-	committed = percpu_counter_read_positive(&vm_committed_as);
+	committed = vm_memory_committed();
 
 	cached = global_node_page_state(NR_FILE_PAGES) -
 			total_swapcache_pages() - i.bufferram;
-- 
2.7.4

[PATCH v4 2/4] mm/util.c: make vm_memory_committed() more accurate

[Feng Tang]

percpu_counter_sum_positive() will provide more accurate info.

As with percpu_counter_read_positive(), in worst case the deviation
could be 'batch * nr_cpus', which is totalram_pages/256 for now,
and will be more when the batch gets enlarged.

Its time cost is about 800 nanoseconds on a 2C/4T platform and
2~3 microseconds on a 2S/36C/72T server in normal case, and in
worst case where vm_committed_as's spinlock is under severe
contention, it costs 30~40 microseconds for the 2S/36C/72T sever,
which should be fine for its only two users: /proc/meminfo and
HyperV balloon driver's status trace per second.

Signed-off-by: Feng Tang <feng.tang@intel.com>
---
 mm/util.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/util.c b/mm/util.c
index 9b3be03..3c7a08c 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -790,7 +790,7 @@ struct percpu_counter vm_committed_as ____cacheline_aligned_in_smp;
  */
 unsigned long vm_memory_committed(void)
 {
-	return percpu_counter_read_positive(&vm_committed_as);
+	return percpu_counter_sum_positive(&vm_committed_as);
 }
 EXPORT_SYMBOL_GPL(vm_memory_committed);
 
-- 
2.7.4

Re: [PATCH v4 2/4] mm/util.c: make vm_memory_committed() more accurate

[Michal Hocko]

On Fri 29-05-20 09:06:08, Feng Tang wrote:
> percpu_counter_sum_positive() will provide more accurate info.
> 
> As with percpu_counter_read_positive(), in worst case the deviation
> could be 'batch * nr_cpus', which is totalram_pages/256 for now,
> and will be more when the batch gets enlarged.
> 
> Its time cost is about 800 nanoseconds on a 2C/4T platform and
> 2~3 microseconds on a 2S/36C/72T server in normal case, and in
> worst case where vm_committed_as's spinlock is under severe
> contention, it costs 30~40 microseconds for the 2S/36C/72T sever,
> which should be fine for its only two users: /proc/meminfo and
> HyperV balloon driver's status trace per second.
> 
> Signed-off-by: Feng Tang <feng.tang@intel.com>

I cannot speak for HyperV part. Cc maintainers but this shouldn't be a
problem for meminfo.

Acked-by: Michal Hocko <mhocko@suse.com>

> ---
>  mm/util.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/mm/util.c b/mm/util.c
> index 9b3be03..3c7a08c 100644
> --- a/mm/util.c
> +++ b/mm/util.c
> @@ -790,7 +790,7 @@ struct percpu_counter vm_committed_as ____cacheline_aligned_in_smp;
>   */
>  unsigned long vm_memory_committed(void)
>  {
> -	return percpu_counter_read_positive(&vm_committed_as);
> +	return percpu_counter_sum_positive(&vm_committed_as);
>  }
>  EXPORT_SYMBOL_GPL(vm_memory_committed);
>  
> -- 
> 2.7.4
> 

-- 
Michal Hocko
SUSE Labs

Re: [PATCH v4 2/4] mm/util.c: make vm_memory_committed() more accurate

[Andi Kleen]

> Its time cost is about 800 nanoseconds on a 2C/4T platform and
> 2~3 microseconds on a 2S/36C/72T server in normal case, and in
> worst case where vm_committed_as's spinlock is under severe
> contention, it costs 30~40 microseconds for the 2S/36C/72T sever,

This will be likely 40-80us on larger systems, although the overhead
is often non linear so it might get worse.

> which should be fine for its only two users: /proc/meminfo and
> HyperV balloon driver's status trace per second.

There are some setups who do frequent sampling of /proc/meminfo
in the background.  Increased overhead could be a problem for them.
But not proposing a change now. If someone complains have to 
revisit I guess, perhaps adding a rate limit of some sort.

-Andi

Re: [PATCH v4 2/4] mm/util.c: make vm_memory_committed() more accurate

[Feng Tang]

On Wed, Jun 03, 2020 at 07:28:53AM -0700, Andi Kleen wrote:
> > Its time cost is about 800 nanoseconds on a 2C/4T platform and
> > 2~3 microseconds on a 2S/36C/72T server in normal case, and in
> > worst case where vm_committed_as's spinlock is under severe
> > contention, it costs 30~40 microseconds for the 2S/36C/72T sever,
> 
> This will be likely 40-80us on larger systems, although the overhead
> is often non linear so it might get worse.
> 
> > which should be fine for its only two users: /proc/meminfo and
> > HyperV balloon driver's status trace per second.
> 
> There are some setups who do frequent sampling of /proc/meminfo
> in the background.  Increased overhead could be a problem for them.
> But not proposing a change now. If someone complains have to 
> revisit I guess, perhaps adding a rate limit of some sort.

Agree. Maybe I should also put the time cost info into the code
comments in case someone noticed the slowdown.

Thanks,
Feng

> 
> -Andi

[PATCH v4 3/4] mm/util.c: remove the VM_WARN_ONCE for vm_committed_as underflow check

[Feng Tang]

As is explained by Michal Hocko:

: Looking at the history, this has been added by 82f71ae4a2b8
: ("mm: catch memory commitment underflow") to have a safety check
: for issues which have been fixed. There doesn't seem to be any bug
: reports mentioning this splat since then so it is likely just
: spending cycles for a hot path (yes many people run with DEBUG_VM)
: without a strong reason.

Signed-off-by: Feng Tang <feng.tang@intel.com>
Cc: Konstantin Khlebnikov <koct9i@gmail.com>
Cc: Qian Cai <cai@lca.pw>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Andi Kleen <andi.kleen@intel.com>
---
 mm/util.c | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/mm/util.c b/mm/util.c
index 3c7a08c..fe63271 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -814,14 +814,6 @@ int __vm_enough_memory(struct mm_struct *mm, long pages, int cap_sys_admin)
 {
 	long allowed;
 
-	/*
-	 * A transient decrease in the value is unlikely, so no need
-	 * READ_ONCE() for vm_committed_as.count.
-	 */
-	VM_WARN_ONCE(data_race(percpu_counter_read(&vm_committed_as) <
-			-(s64)vm_committed_as_batch * num_online_cpus()),
-			"memory commitment underflow");
-
 	vm_acct_memory(pages);
 
 	/*
-- 
2.7.4

Re: [PATCH v4 3/4] mm/util.c: remove the VM_WARN_ONCE for vm_committed_as underflow check

[Qian Cai]

On Fri, May 29, 2020 at 09:06:09AM +0800, Feng Tang wrote:
> As is explained by Michal Hocko:
> 
> : Looking at the history, this has been added by 82f71ae4a2b8
> : ("mm: catch memory commitment underflow") to have a safety check
> : for issues which have been fixed. There doesn't seem to be any bug
> : reports mentioning this splat since then so it is likely just
> : spending cycles for a hot path (yes many people run with DEBUG_VM)
> : without a strong reason.

Hmm, it looks like the warning is still useful to catch issues in,

https://lore.kernel.org/linux-mm/20140624201606.18273.44270.stgit@zurg
https://lore.kernel.org/linux-mm/54BB9A32.7080703@oracle.com/

After read the whole discussion in that thread, I actually disagree with
Michal. In order to get ride of this existing warning, it is rather
someone needs a strong reason that could prove the performance hit is
noticeable with some data.

> 
> Signed-off-by: Feng Tang <feng.tang@intel.com>
> Cc: Konstantin Khlebnikov <koct9i@gmail.com>
> Cc: Qian Cai <cai@lca.pw>
> Cc: Michal Hocko <mhocko@suse.com>
> Cc: Andi Kleen <andi.kleen@intel.com>
> ---
>  mm/util.c | 8 --------
>  1 file changed, 8 deletions(-)
> 
> diff --git a/mm/util.c b/mm/util.c
> index 3c7a08c..fe63271 100644
> --- a/mm/util.c
> +++ b/mm/util.c
> @@ -814,14 +814,6 @@ int __vm_enough_memory(struct mm_struct *mm, long pages, int cap_sys_admin)
>  {
>  	long allowed;
>  
> -	/*
> -	 * A transient decrease in the value is unlikely, so no need
> -	 * READ_ONCE() for vm_committed_as.count.
> -	 */
> -	VM_WARN_ONCE(data_race(percpu_counter_read(&vm_committed_as) <
> -			-(s64)vm_committed_as_batch * num_online_cpus()),
> -			"memory commitment underflow");
> -
>  	vm_acct_memory(pages);
>  
>  	/*
> -- 
> 2.7.4
> 

Re: [PATCH v4 3/4] mm/util.c: remove the VM_WARN_ONCE for vm_committed_as underflow check

[Feng Tang]

On Thu, May 28, 2020 at 10:49:28PM -0400, Qian Cai wrote:
> On Fri, May 29, 2020 at 09:06:09AM +0800, Feng Tang wrote:
> > As is explained by Michal Hocko:
> > 
> > : Looking at the history, this has been added by 82f71ae4a2b8
> > : ("mm: catch memory commitment underflow") to have a safety check
> > : for issues which have been fixed. There doesn't seem to be any bug
> > : reports mentioning this splat since then so it is likely just
> > : spending cycles for a hot path (yes many people run with DEBUG_VM)
> > : without a strong reason.
> 
> Hmm, it looks like the warning is still useful to catch issues in,
> 
> https://lore.kernel.org/linux-mm/20140624201606.18273.44270.stgit@zurg
> https://lore.kernel.org/linux-mm/54BB9A32.7080703@oracle.com/
> 
> After read the whole discussion in that thread, I actually disagree with
> Michal. In order to get ride of this existing warning, it is rather
> someone needs a strong reason that could prove the performance hit is
> noticeable with some data.

One problem with current check is percpu_counter_read(&vm_committed_as)
is not accurate, and percpu_counter_sum() is way too heavy.

Thanks,
Feng

Re: [PATCH v4 3/4] mm/util.c: remove the VM_WARN_ONCE for vm_committed_as underflow check

[Feng Tang]

Hi Qian,

On Thu, May 28, 2020 at 10:49:28PM -0400, Qian Cai wrote:
> On Fri, May 29, 2020 at 09:06:09AM +0800, Feng Tang wrote:
> > As is explained by Michal Hocko:
> > 
> > : Looking at the history, this has been added by 82f71ae4a2b8
> > : ("mm: catch memory commitment underflow") to have a safety check
> > : for issues which have been fixed. There doesn't seem to be any bug
> > : reports mentioning this splat since then so it is likely just
> > : spending cycles for a hot path (yes many people run with DEBUG_VM)
> > : without a strong reason.
> 
> Hmm, it looks like the warning is still useful to catch issues in,
> 
> https://lore.kernel.org/linux-mm/20140624201606.18273.44270.stgit@zurg
> https://lore.kernel.org/linux-mm/54BB9A32.7080703@oracle.com/
> 
> After read the whole discussion in that thread, I actually disagree with
> Michal. In order to get ride of this existing warning, it is rather
> someone needs a strong reason that could prove the performance hit is
> noticeable with some data.

I re-run the same benchmark with v5.7 and 5.7+remove_warning kernels,
the overall performance change is trivial (which is expected)

   1330147            +0.1%    1331032        will-it-scale.72.processes

But the perf stats of "self" shows big change for __vm_enough_memory() 

      0.27            -0.3        0.00        pp.self.__vm_enough_memory

I post the full compare result in the end.

Thanks,
Feng


=========================================================================================
tbox_group/testcase/rootfs/kconfig/compiler/nr_task/mode/test/cpufreq_governor/ucode:
  lkp-skl-2sp7/will-it-scale/debian-x86_64-20191114.cgz/x86_64-rhel-7.6-vm-debug/gcc-7/100%/process/mmap2/performance/0x2000065

commit: 
  v5.7
  af3eca72dc43078e1ee4a38b0ecc0225b659f345

            v5.7 af3eca72dc43078e1ee4a38b0ec 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
        850:3       -12130%         486:2     dmesg.timestamp:last
          2:3          -67%            :2     kmsg.Firmware_Bug]:the_BIOS_has_corrupted_hw-PMU_resources(MSR#is#bb)
           :3           33%           1:2     kmsg.Firmware_Bug]:the_BIOS_has_corrupted_hw-PMU_resources(MSR#is#e08)
          5:3         -177%            :2     kmsg.timestamp:Firmware_Bug]:the_BIOS_has_corrupted_hw-PMU_resources(MSR#is#bb)
           :3           88%           2:2     kmsg.timestamp:Firmware_Bug]:the_BIOS_has_corrupted_hw-PMU_resources(MSR#is#e08)
        398:3        -4444%         265:2     kmsg.timestamp:last
         %stddev     %change         %stddev
             \          |                \  
   1330147            +0.1%    1331032        will-it-scale.72.processes
      0.02            +0.0%       0.02        will-it-scale.72.processes_idle
     18474            +0.1%      18486        will-it-scale.per_process_ops
    301.18            -0.0%     301.16        will-it-scale.time.elapsed_time
    301.18            -0.0%     301.16        will-it-scale.time.elapsed_time.max
      1.00 ± 81%    +100.0%       2.00        will-it-scale.time.involuntary_context_switches
      9452            +0.0%       9452        will-it-scale.time.maximum_resident_set_size
      5925            +0.1%       5932        will-it-scale.time.minor_page_faults
      4096            +0.0%       4096        will-it-scale.time.page_size
      0.01 ± 35%     +12.5%       0.01 ± 33%  will-it-scale.time.system_time
      0.03 ± 14%      +5.0%       0.04 ± 14%  will-it-scale.time.user_time
     83.33            +0.2%      83.50        will-it-scale.time.voluntary_context_switches
   1330147            +0.1%    1331032        will-it-scale.workload
      0.45 ± 29%      +0.0        0.50 ± 28%  mpstat.cpu.all.idle%
     98.41            -0.1       98.34        mpstat.cpu.all.sys%
      1.14            +0.0        1.16        mpstat.cpu.all.usr%
    200395 ± 18%     +11.9%     224282 ± 14%  cpuidle.C1.time
      4008 ± 38%      -2.1%       3924 ± 15%  cpuidle.C1.usage
 1.222e+08 ± 19%     -29.2%   86444161        cpuidle.C1E.time
    254203 ± 19%     -23.2%     195198 ±  4%  cpuidle.C1E.usage
   8145747 ± 31%    +339.9%   35830338 ± 72%  cpuidle.C6.time
     22878 ±  9%    +288.2%      88823 ± 70%  cpuidle.C6.usage
      8891 ±  7%      -7.4%       8229        cpuidle.POLL.time
      3111 ± 18%     -11.1%       2766        cpuidle.POLL.usage
      0.00          -100.0%       0.00        numa-numastat.node0.interleave_hit
    314399 ±  2%      -1.0%     311244 ±  3%  numa-numastat.node0.local_node
    322209            +2.4%     329909        numa-numastat.node0.numa_hit
      7814 ± 73%    +138.9%      18670 ± 24%  numa-numastat.node0.other_node
      0.00          -100.0%       0.00        numa-numastat.node1.interleave_hit
    343026 ±  2%      -0.3%     341980        numa-numastat.node1.local_node
    358632            -3.3%     346708        numa-numastat.node1.numa_hit
     15613 ± 36%     -69.7%       4728 ± 98%  numa-numastat.node1.other_node
    301.18            -0.0%     301.16        time.elapsed_time
    301.18            -0.0%     301.16        time.elapsed_time.max
      1.00 ± 81%    +100.0%       2.00        time.involuntary_context_switches
      9452            +0.0%       9452        time.maximum_resident_set_size
      5925            +0.1%       5932        time.minor_page_faults
      4096            +0.0%       4096        time.page_size
      0.01 ± 35%     +12.5%       0.01 ± 33%  time.system_time
      0.03 ± 14%      +5.0%       0.04 ± 14%  time.user_time
     83.33            +0.2%      83.50        time.voluntary_context_switches
      0.33 ±141%     +50.0%       0.50 ±100%  vmstat.cpu.id
     97.00            +0.0%      97.00        vmstat.cpu.sy
      1.00            +0.0%       1.00        vmstat.cpu.us
      0.00          -100.0%       0.00        vmstat.io.bi
      4.00            +0.0%       4.00        vmstat.memory.buff
   1391751            +0.1%    1392746        vmstat.memory.cache
 1.294e+08            -0.0%  1.294e+08        vmstat.memory.free
     71.00            +0.0%      71.00        vmstat.procs.r
      1315            -0.7%       1305        vmstat.system.cs
    147433            -0.0%     147369        vmstat.system.in
      0.00          -100.0%       0.00        proc-vmstat.compact_isolated
     85060            +0.4%      85431        proc-vmstat.nr_active_anon
     37.00            -1.4%      36.50        proc-vmstat.nr_active_file
     71111            +0.1%      71200        proc-vmstat.nr_anon_pages
     77.33 ± 17%     +12.5%      87.00        proc-vmstat.nr_anon_transparent_hugepages
     54.00            +1.9%      55.00        proc-vmstat.nr_dirtied
      5.00            +0.0%       5.00        proc-vmstat.nr_dirty
   3215506            -0.0%    3215471        proc-vmstat.nr_dirty_background_threshold
   6438875            -0.0%    6438805        proc-vmstat.nr_dirty_threshold
    327936            +0.1%     328237        proc-vmstat.nr_file_pages
     50398            +0.0%      50398        proc-vmstat.nr_free_cma
  32356721            -0.0%   32356374        proc-vmstat.nr_free_pages
      4640            -0.1%       4636        proc-vmstat.nr_inactive_anon
     82.67 ±  2%      -0.8%      82.00 ±  2%  proc-vmstat.nr_inactive_file
     13256            -0.3%      13211        proc-vmstat.nr_kernel_stack
      8057            -0.5%       8017        proc-vmstat.nr_mapped
    134.00 ±141%     +49.3%     200.00 ±100%  proc-vmstat.nr_mlock
      2229            +0.2%       2234        proc-vmstat.nr_page_table_pages
     18609 ±  3%      +1.6%      18898 ±  3%  proc-vmstat.nr_shmem
     19964            -0.3%      19901        proc-vmstat.nr_slab_reclaimable
     34003            +0.1%      34025        proc-vmstat.nr_slab_unreclaimable
    309227            +0.0%     309249        proc-vmstat.nr_unevictable
      0.00          -100.0%       0.00        proc-vmstat.nr_writeback
     53.00            +0.0%      53.00        proc-vmstat.nr_written
     85060            +0.4%      85431        proc-vmstat.nr_zone_active_anon
     37.00            -1.4%      36.50        proc-vmstat.nr_zone_active_file
      4640            -0.1%       4636        proc-vmstat.nr_zone_inactive_anon
     82.67 ±  2%      -0.8%      82.00 ±  2%  proc-vmstat.nr_zone_inactive_file
    309227            +0.0%     309249        proc-vmstat.nr_zone_unevictable
      5.00            +0.0%       5.00        proc-vmstat.nr_zone_write_pending
      2181 ±124%     -68.6%     685.50 ± 80%  proc-vmstat.numa_hint_faults
     37.67 ±109%     +77.9%      67.00 ± 91%  proc-vmstat.numa_hint_faults_local
    702373            +0.7%     707116        proc-vmstat.numa_hit
     35.33 ± 85%     -70.3%      10.50 ±  4%  proc-vmstat.numa_huge_pte_updates
      0.00          -100.0%       0.00        proc-vmstat.numa_interleave
    678938            +0.7%     683714        proc-vmstat.numa_local
     23435            -0.1%      23401        proc-vmstat.numa_other
      4697 ± 68%     -86.8%     618.50 ± 98%  proc-vmstat.numa_pages_migrated
     25844 ± 52%     -79.1%       5406 ±  4%  proc-vmstat.numa_pte_updates
     20929 ±  4%      +1.9%      21332 ±  5%  proc-vmstat.pgactivate
      0.00          -100.0%       0.00        proc-vmstat.pgalloc_dma32
    760325            -0.4%     756908        proc-vmstat.pgalloc_normal
    801566            -0.5%     797832        proc-vmstat.pgfault
    714690            -0.2%     713286        proc-vmstat.pgfree
      4697 ± 68%     -86.8%     618.50 ± 98%  proc-vmstat.pgmigrate_success
      0.00          -100.0%       0.00        proc-vmstat.pgpgin
    103.00            +0.5%     103.50        proc-vmstat.thp_collapse_alloc
      5.00            +0.0%       5.00        proc-vmstat.thp_fault_alloc
      0.00          -100.0%       0.00        proc-vmstat.thp_zero_page_alloc
     41.00 ± 98%     +35.4%      55.50 ± 80%  proc-vmstat.unevictable_pgs_culled
    183.00 ±141%     +50.0%     274.50 ±100%  proc-vmstat.unevictable_pgs_mlocked
      2.59            +0.5%       2.60        perf-stat.i.MPKI
 4.854e+09            +0.0%  4.856e+09        perf-stat.i.branch-instructions
      0.45            -0.0        0.43        perf-stat.i.branch-miss-rate%
  21296577            -2.3%   20817170        perf-stat.i.branch-misses
     39.98            -0.2       39.81        perf-stat.i.cache-miss-rate%
  21372778            +0.0%   21380457        perf-stat.i.cache-misses
  53441942            +0.5%   53705724        perf-stat.i.cache-references
      1285            -0.7%       1277        perf-stat.i.context-switches
     10.67            -0.0%      10.67        perf-stat.i.cpi
     71998            -0.0%      71998        perf-stat.i.cpu-clock
  2.21e+11            +0.0%   2.21e+11        perf-stat.i.cpu-cycles
    117.36            +0.3%     117.71        perf-stat.i.cpu-migrations
     10322            -0.0%      10321        perf-stat.i.cycles-between-cache-misses
      0.05            +0.0        0.05        perf-stat.i.dTLB-load-miss-rate%
   2709233            +0.1%    2712427        perf-stat.i.dTLB-load-misses
 5.785e+09            +0.0%  5.787e+09        perf-stat.i.dTLB-loads
      0.00            +0.0        0.00 ±  2%  perf-stat.i.dTLB-store-miss-rate%
      8967            -3.0%       8701        perf-stat.i.dTLB-store-misses
  1.97e+09            +0.1%  1.971e+09        perf-stat.i.dTLB-stores
     94.02            +0.2       94.24        perf-stat.i.iTLB-load-miss-rate%
   2732366            -1.4%    2694372        perf-stat.i.iTLB-load-misses
    173049            -5.7%     163172 ±  2%  perf-stat.i.iTLB-loads
  2.07e+10            +0.0%  2.071e+10        perf-stat.i.instructions
      7671            +1.0%       7747        perf-stat.i.instructions-per-iTLB-miss
      0.10            +0.1%       0.10        perf-stat.i.ipc
      3.07            +0.0%       3.07        perf-stat.i.metric.GHz
      0.42            +0.5%       0.43        perf-stat.i.metric.K/sec
    175.98            +0.1%     176.08        perf-stat.i.metric.M/sec
      2565            -0.7%       2547        perf-stat.i.minor-faults
     99.55            -0.0       99.53        perf-stat.i.node-load-miss-rate%
   5949351            +0.3%    5969805        perf-stat.i.node-load-misses
     22301 ±  6%      +5.6%      23543 ±  8%  perf-stat.i.node-loads
     99.73            -0.0       99.72        perf-stat.i.node-store-miss-rate%
   5314673            -0.1%    5310449        perf-stat.i.node-store-misses
      4704 ±  4%      -1.8%       4619        perf-stat.i.node-stores
      2565            -0.7%       2547        perf-stat.i.page-faults
     71998            -0.0%      71998        perf-stat.i.task-clock
      2.58            +0.5%       2.59        perf-stat.overall.MPKI
      0.44            -0.0        0.43        perf-stat.overall.branch-miss-rate%
     39.99            -0.2       39.81        perf-stat.overall.cache-miss-rate%
     10.67            -0.0%      10.67        perf-stat.overall.cpi
     10340            -0.0%      10337        perf-stat.overall.cycles-between-cache-misses
      0.05            +0.0        0.05        perf-stat.overall.dTLB-load-miss-rate%
      0.00            -0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
     94.04            +0.2       94.29        perf-stat.overall.iTLB-load-miss-rate%
      7577            +1.4%       7686        perf-stat.overall.instructions-per-iTLB-miss
      0.09            +0.0%       0.09        perf-stat.overall.ipc
     99.62            -0.0       99.60        perf-stat.overall.node-load-miss-rate%
     99.91            +0.0       99.91        perf-stat.overall.node-store-miss-rate%
   4691551            +0.0%    4693151        perf-stat.overall.path-length
 4.838e+09            +0.0%   4.84e+09        perf-stat.ps.branch-instructions
  21230930            -2.3%   20750859        perf-stat.ps.branch-misses
  21302195            +0.0%   21309444        perf-stat.ps.cache-misses
  53273375            +0.5%   53531696        perf-stat.ps.cache-references
      1281            -0.7%       1273        perf-stat.ps.context-switches
     71760            -0.0%      71759        perf-stat.ps.cpu-clock
 2.203e+11            +0.0%  2.203e+11        perf-stat.ps.cpu-cycles
    117.02            +0.3%     117.33        perf-stat.ps.cpu-migrations
   2702184            +0.1%    2704689        perf-stat.ps.dTLB-load-misses
 5.766e+09            +0.0%  5.767e+09        perf-stat.ps.dTLB-loads
      9028            -3.2%       8736        perf-stat.ps.dTLB-store-misses
 1.963e+09            +0.1%  1.965e+09        perf-stat.ps.dTLB-stores
   2723237            -1.4%    2685365        perf-stat.ps.iTLB-load-misses
    172573            -5.7%     162735 ±  2%  perf-stat.ps.iTLB-loads
 2.063e+10            +0.0%  2.064e+10        perf-stat.ps.instructions
      2559            -0.7%       2540        perf-stat.ps.minor-faults
   5929506            +0.3%    5949863        perf-stat.ps.node-load-misses
     22689 ±  5%      +4.7%      23766 ±  8%  perf-stat.ps.node-loads
   5296902            -0.1%    5292690        perf-stat.ps.node-store-misses
      4724 ±  4%      -2.2%       4622        perf-stat.ps.node-stores
      2559            -0.7%       2540        perf-stat.ps.page-faults
     71760            -0.0%      71759        perf-stat.ps.task-clock
  6.24e+12            +0.1%  6.247e+12        perf-stat.total.instructions
     47.20            -0.2       47.05        pp.bt.percpu_counter_add_batch.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
     50.12            -0.2       49.97        pp.bt.entry_SYSCALL_64_after_hwframe.munmap
     50.10            -0.1       49.95        pp.bt.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
     46.75            -0.1       46.60        pp.bt._raw_spin_lock_irqsave.percpu_counter_add_batch.__do_munmap.__vm_munmap.__x64_sys_munmap
     49.36            -0.1       49.22        pp.bt.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
     49.78            -0.1       49.64        pp.bt.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
     49.75            -0.1       49.61        pp.bt.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.munmap
     50.48            -0.1       50.34        pp.bt.munmap
     46.56            -0.1       46.41        pp.bt.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.percpu_counter_add_batch.__do_munmap.__vm_munmap
      1.88            -0.0        1.88        pp.bt.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
      1.32            +0.0        1.33        pp.bt.unmap_page_range.unmap_vmas.unmap_region.__do_munmap.__vm_munmap
      1.42            +0.0        1.44        pp.bt.unmap_vmas.unmap_region.__do_munmap.__vm_munmap.__x64_sys_munmap
      0.51            +0.0        0.53        pp.bt.___might_sleep.unmap_page_range.unmap_vmas.unmap_region.__do_munmap
     48.27            +0.1       48.39        pp.bt.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
     48.67            +0.1       48.80        pp.bt.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64
     48.51            +0.1       48.65        pp.bt.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
     47.10            +0.1       47.24        pp.bt.__vm_enough_memory.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
     48.74            +0.1       48.88        pp.bt.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64
     49.08            +0.1       49.23        pp.bt.entry_SYSCALL_64_after_hwframe.mmap64
     46.48            +0.1       46.62        pp.bt.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.percpu_counter_add_batch.__vm_enough_memory.mmap_region
     49.06            +0.1       49.21        pp.bt.do_syscall_64.entry_SYSCALL_64_after_hwframe.mmap64
     49.46            +0.1       49.61        pp.bt.mmap64
     46.66            +0.1       46.80        pp.bt._raw_spin_lock_irqsave.percpu_counter_add_batch.__vm_enough_memory.mmap_region.do_mmap
     46.84            +0.4       47.23        pp.bt.percpu_counter_add_batch.__vm_enough_memory.mmap_region.do_mmap.vm_mmap_pgoff
     49.78            -0.1       49.64        pp.child.__x64_sys_munmap
     50.51            -0.1       50.36        pp.child.munmap
     49.76            -0.1       49.61        pp.child.__vm_munmap
     49.36            -0.1       49.22        pp.child.__do_munmap
      0.45            -0.0        0.41        pp.child.perf_event_mmap
      0.03 ± 70%      -0.0        0.00        pp.child.strlen
      0.02 ±141%      -0.0        0.00        pp.child.common_file_perm
      0.30            -0.0        0.29        pp.child.free_pgd_range
      1.88            -0.0        1.88        pp.child.unmap_region
      0.28            -0.0        0.27        pp.child.free_p4d_range
      0.28 ±  4%      -0.0        0.27        pp.child.apic_timer_interrupt
      0.19 ±  4%      -0.0        0.18 ±  2%  pp.child.hrtimer_interrupt
      0.09            -0.0        0.08 ±  5%  pp.child.find_vma
      0.40 ±  2%      -0.0        0.40 ±  3%  pp.child.vm_area_alloc
      0.35            -0.0        0.34        pp.child.syscall_return_via_sysret
      0.35 ±  2%      -0.0        0.34        pp.child.up_read
     93.41            -0.0       93.41        pp.child._raw_spin_lock_irqsave
      0.05 ±  8%      -0.0        0.05        pp.child.kmem_cache_alloc_trace
      0.11 ±  4%      -0.0        0.11        pp.child.d_path
      0.11 ±  4%      -0.0        0.11        pp.child.perf_iterate_sb
      0.08 ±  5%      -0.0        0.08        pp.child.prepend_path
     99.23            -0.0       99.23        pp.child.entry_SYSCALL_64_after_hwframe
     93.04            -0.0       93.04        pp.child.native_queued_spin_lock_slowpath
      0.09            -0.0        0.09        pp.child.tick_sched_timer
      0.05            -0.0        0.05        pp.child.unlink_file_vma
      0.05            -0.0        0.05        pp.child.task_tick_fair
      0.05            -0.0        0.05        pp.child.perf_event_mmap_output
     99.20            +0.0       99.20        pp.child.do_syscall_64
      0.21 ±  3%      +0.0        0.21        pp.child.smp_apic_timer_interrupt
      0.08            +0.0        0.08        pp.child.update_process_times
      0.06            +0.0        0.06        pp.child.down_write_killable
      0.15            +0.0        0.15 ±  6%  pp.child.rcu_all_qs
      0.37 ±  2%      +0.0        0.37 ±  5%  pp.child.kmem_cache_alloc
      0.28            +0.0        0.29 ±  5%  pp.child._cond_resched
      0.12 ±  3%      +0.0        0.12 ±  4%  pp.child.vma_link
      0.09 ±  5%      +0.0        0.10 ±  5%  pp.child.security_mmap_file
      0.06 ±  7%      +0.0        0.07 ±  7%  pp.child.scheduler_tick
      0.13 ±  3%      +0.0        0.14 ±  3%  pp.child.__hrtimer_run_queues
      0.32 ±  2%      +0.0        0.32        pp.child._raw_spin_unlock_irqrestore
      0.08            +0.0        0.08 ±  5%  pp.child.tick_sched_handle
      0.06            +0.0        0.07 ±  7%  pp.child.down_write
      0.06            +0.0        0.07 ±  7%  pp.child.remove_vma
      1.43            +0.0        1.44        pp.child.unmap_vmas
      0.05            +0.0        0.06        pp.child.__vma_rb_erase
      0.08            +0.0        0.09        pp.child.free_pgtables
      1.39            +0.0        1.40        pp.child.unmap_page_range
      0.35            +0.0        0.36 ±  4%  pp.child.entry_SYSCALL_64
      0.10 ±  4%      +0.0        0.11        pp.child.arch_get_unmapped_area_topdown
      0.58            +0.0        0.59        pp.child.___might_sleep
      0.06            +0.0        0.08 ±  6%  pp.child.shmem_mmap
      0.03 ± 70%      +0.0        0.05        pp.child.up_write
      0.03 ± 70%      +0.0        0.05        pp.child.vm_unmapped_area
      0.03 ± 70%      +0.0        0.05        pp.child.__vma_link_rb
      0.15 ±  3%      +0.0        0.17 ±  3%  pp.child.shmem_get_unmapped_area
      0.18 ±  2%      +0.0        0.20 ±  2%  pp.child.get_unmapped_area
      0.00            +0.0        0.03 ±100%  pp.child.prepend_name
      0.02 ±141%      +0.0        0.05        pp.child.touch_atime
     48.28            +0.1       48.39        pp.child.mmap_region
     47.10            +0.1       47.24        pp.child.__vm_enough_memory
     48.51            +0.1       48.65        pp.child.do_mmap
     48.74            +0.1       48.88        pp.child.ksys_mmap_pgoff
     48.67            +0.1       48.81        pp.child.vm_mmap_pgoff
     49.49            +0.1       49.64        pp.child.mmap64
     94.04            +0.2       94.28        pp.child.percpu_counter_add_batch
      0.27            -0.3        0.00        pp.self.__vm_enough_memory
      0.03 ± 70%      -0.0        0.00        pp.self.strlen
      0.02 ±141%      -0.0        0.00        pp.self.prepend_path
      0.28            -0.0        0.27        pp.self.free_p4d_range
      0.07            -0.0        0.06        pp.self.perf_iterate_sb
      0.08            -0.0        0.07        pp.self.perf_event_mmap
      0.35 ±  2%      -0.0        0.34 ±  5%  pp.self.kmem_cache_alloc
      0.11            -0.0        0.11 ±  4%  pp.self.rcu_all_qs
      0.37            -0.0        0.36        pp.self._raw_spin_lock_irqsave
      0.35            -0.0        0.34        pp.self.syscall_return_via_sysret
      0.35 ±  2%      -0.0        0.34        pp.self.up_read
      0.14 ±  3%      -0.0        0.14 ±  3%  pp.self._cond_resched
     93.04            -0.0       93.04        pp.self.native_queued_spin_lock_slowpath
      0.31            +0.0        0.32 ±  4%  pp.self.entry_SYSCALL_64
      0.69            +0.0        0.69        pp.self.unmap_page_range
      0.10 ±  4%      +0.0        0.10        pp.self._raw_spin_unlock_irqrestore
      0.06 ±  8%      +0.0        0.06        pp.self.find_vma
      0.06 ±  8%      +0.0        0.06        pp.self.__do_munmap
      0.06 ±  7%      +0.0        0.07        pp.self.mmap_region
      0.61 ±  3%      +0.0        0.62        pp.self.do_syscall_64
      0.02 ±141%      +0.0        0.03 ±100%  pp.self.up_write
      0.05            +0.0        0.06        pp.self.__vma_rb_erase
      0.54            +0.0        0.56        pp.self.___might_sleep
      0.03 ± 70%      +0.0        0.05        pp.self.vm_unmapped_area
      0.03 ± 70%      +0.0        0.05        pp.self.shmem_get_unmapped_area
      0.03 ± 70%      +0.0        0.05        pp.self.__vma_link_rb
      0.00            +0.0        0.03 ±100%  pp.self.prepend_name
      0.00            +0.0        0.03 ±100%  pp.self.do_mmap
      0.00            +0.0        0.03 ±100%  pp.self.arch_get_unmapped_area_topdown
      0.02 ±141%      +0.0        0.05        pp.self.perf_event_mmap_output
      0.32 ±  2%      +0.2        0.56        pp.self.percpu_counter_add_batch
    552.67 ±  2%      -5.2%     524.00 ±  6%  softirqs.BLOCK
      2.00            +0.0%       2.00        softirqs.HI
    911.00 ± 47%     -31.4%     625.00 ±  2%  softirqs.NET_RX
     63.67 ±  3%      -5.0%      60.50 ±  4%  softirqs.NET_TX
    312414            -1.1%     309101        softirqs.RCU
    228903            -1.4%     225602 ±  3%  softirqs.SCHED
    265.67            -1.0%     263.00        softirqs.TASKLET
   8777267            +0.1%    8789634        softirqs.TIMER
     23504            -0.1%      23472        interrupts.CAL:Function_call_interrupts
    144.00            -0.3%     143.50        interrupts.IWI:IRQ_work_interrupts
  43641147            -0.0%   43621882        interrupts.LOC:Local_timer_interrupts
      0.00          -100.0%       0.00        interrupts.MCP:Machine_check_polls
    570736 ±  5%      +2.1%     582977 ±  5%  interrupts.NMI:Non-maskable_interrupts
    570736 ±  5%      +2.1%     582977 ±  5%  interrupts.PMI:Performance_monitoring_interrupts
     45097            +0.7%      45407        interrupts.RES:Rescheduling_interrupts
      0.00          -100.0%       0.00        interrupts.RTR:APIC_ICR_read_retries
    193.67 ± 26%     -21.3%     152.50 ± 29%  interrupts.TLB:TLB_shootdowns

[PATCH v4 4/4] mm: adjust vm_committed_as_batch according to vm overcommit policy

[Feng Tang]

When checking a performance change for will-it-scale scalability mmap test
[1], we found very high lock contention for spinlock of percpu counter
'vm_committed_as':

    94.14%     0.35%  [kernel.kallsyms]         [k] _raw_spin_lock_irqsave
    48.21% _raw_spin_lock_irqsave;percpu_counter_add_batch;__vm_enough_memory;mmap_region;do_mmap;
    45.91% _raw_spin_lock_irqsave;percpu_counter_add_batch;__do_munmap;

Actually this heavy lock contention is not always necessary.  The
'vm_committed_as' needs to be very precise when the strict
OVERCOMMIT_NEVER policy is set, which requires a rather small batch number
for the percpu counter.

So keep 'batch' number unchanged for strict OVERCOMMIT_NEVER policy, and
lift it to 64X for OVERCOMMIT_ALWAYS and OVERCOMMIT_GUESS policies.  Also
add a sysctl handler to adjust it when the policy is reconfigured.

Benchmark with the same testcase in [1] shows 53% improvement on a 8C/16T
desktop, and 2097%(20X) on a 4S/72C/144T server.  We tested with test
platforms in 0day (server, desktop and laptop), and 80%+ platforms shows
improvements with that test.  And whether it shows improvements depends on
if the test mmap size is bigger than the batch number computed.

And if the lift is 16X, 1/3 of the platforms will show improvements,
though it should help the mmap/unmap usage generally, as Michal Hocko
mentioned:

: I believe that there are non-synthetic worklaods which would benefit from
: a larger batch.  E.g.  large in memory databases which do large mmaps
: during startups from multiple threads.

[1] https://lore.kernel.org/lkml/20200305062138.GI5972@shao2-debian/

Link: http://lkml.kernel.org/r/1589611660-89854-4-git-send-email-feng.tang@intel.com
Signed-off-by: Feng Tang <feng.tang@intel.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Mel Gorman <mgorman@suse.de>
Cc: Kees Cook <keescook@chromium.org>
Cc: Andi Kleen <andi.kleen@intel.com>
Cc: Tim Chen <tim.c.chen@intel.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Huang Ying <ying.huang@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
 include/linux/mm.h   |  2 ++
 include/linux/mman.h |  4 ++++
 kernel/sysctl.c      |  2 +-
 mm/mm_init.c         | 18 ++++++++++++++----
 mm/util.c            | 12 ++++++++++++
 5 files changed, 33 insertions(+), 5 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 573947c..c2efea6 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -206,6 +206,8 @@ int overcommit_ratio_handler(struct ctl_table *, int, void *, size_t *,
 		loff_t *);
 int overcommit_kbytes_handler(struct ctl_table *, int, void *, size_t *,
 		loff_t *);
+int overcommit_policy_handler(struct ctl_table *, int, void *, size_t *,
+		loff_t *);
 
 #define nth_page(page,n) pfn_to_page(page_to_pfn((page)) + (n))
 
diff --git a/include/linux/mman.h b/include/linux/mman.h
index 4b08e9c..91c93c1 100644
--- a/include/linux/mman.h
+++ b/include/linux/mman.h
@@ -57,8 +57,12 @@ extern struct percpu_counter vm_committed_as;
 
 #ifdef CONFIG_SMP
 extern s32 vm_committed_as_batch;
+extern void mm_compute_batch(void);
 #else
 #define vm_committed_as_batch 0
+static inline void mm_compute_batch(void)
+{
+}
 #endif
 
 unsigned long vm_memory_committed(void);
diff --git a/kernel/sysctl.c b/kernel/sysctl.c
index db1ce7a..9456c86 100644
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -2650,7 +2650,7 @@ static struct ctl_table vm_table[] = {
 		.data		= &sysctl_overcommit_memory,
 		.maxlen		= sizeof(sysctl_overcommit_memory),
 		.mode		= 0644,
-		.proc_handler	= proc_dointvec_minmax,
+		.proc_handler	= overcommit_policy_handler,
 		.extra1		= SYSCTL_ZERO,
 		.extra2		= &two,
 	},
diff --git a/mm/mm_init.c b/mm/mm_init.c
index 435e5f7..c5a6fb1 100644
--- a/mm/mm_init.c
+++ b/mm/mm_init.c
@@ -13,6 +13,7 @@
 #include <linux/memory.h>
 #include <linux/notifier.h>
 #include <linux/sched.h>
+#include <linux/mman.h>
 #include "internal.h"
 
 #ifdef CONFIG_DEBUG_MEMORY_INIT
@@ -144,14 +145,23 @@ EXPORT_SYMBOL_GPL(mm_kobj);
 #ifdef CONFIG_SMP
 s32 vm_committed_as_batch = 32;
 
-static void __meminit mm_compute_batch(void)
+void mm_compute_batch(void)
 {
 	u64 memsized_batch;
 	s32 nr = num_present_cpus();
 	s32 batch = max_t(s32, nr*2, 32);
-
-	/* batch size set to 0.4% of (total memory/#cpus), or max int32 */
-	memsized_batch = min_t(u64, (totalram_pages()/nr)/256, 0x7fffffff);
+	unsigned long ram_pages = totalram_pages();
+
+	/*
+	 * For policy of OVERCOMMIT_NEVER, set batch size to 0.4%
+	 * of (total memory/#cpus), and lift it to 25% for other
+	 * policies to easy the possible lock contention for percpu_counter
+	 * vm_committed_as, while the max limit is INT_MAX
+	 */
+	if (sysctl_overcommit_memory == OVERCOMMIT_NEVER)
+		memsized_batch = min_t(u64, ram_pages/nr/256, INT_MAX);
+	else
+		memsized_batch = min_t(u64, ram_pages/nr/4, INT_MAX);
 
 	vm_committed_as_batch = max_t(s32, memsized_batch, batch);
 }
diff --git a/mm/util.c b/mm/util.c
index fe63271..580d268 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -746,6 +746,18 @@ int overcommit_ratio_handler(struct ctl_table *table, int write, void *buffer,
 	return ret;
 }
 
+int overcommit_policy_handler(struct ctl_table *table, int write, void *buffer,
+		size_t *lenp, loff_t *ppos)
+{
+	int ret;
+
+	ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
+	if (ret == 0 && write)
+		mm_compute_batch();
+
+	return ret;
+}
+
 int overcommit_kbytes_handler(struct ctl_table *table, int write, void *buffer,
 		size_t *lenp, loff_t *ppos)
 {
-- 
2.7.4

Re: [PATCH v4 4/4] mm: adjust vm_committed_as_batch according to vm overcommit policy

[Michal Hocko]

On Fri 29-05-20 09:06:10, Feng Tang wrote:
> When checking a performance change for will-it-scale scalability mmap test
> [1], we found very high lock contention for spinlock of percpu counter
> 'vm_committed_as':
> 
>     94.14%     0.35%  [kernel.kallsyms]         [k] _raw_spin_lock_irqsave
>     48.21% _raw_spin_lock_irqsave;percpu_counter_add_batch;__vm_enough_memory;mmap_region;do_mmap;
>     45.91% _raw_spin_lock_irqsave;percpu_counter_add_batch;__do_munmap;
> 
> Actually this heavy lock contention is not always necessary.  The
> 'vm_committed_as' needs to be very precise when the strict
> OVERCOMMIT_NEVER policy is set, which requires a rather small batch number
> for the percpu counter.
> 
> So keep 'batch' number unchanged for strict OVERCOMMIT_NEVER policy, and
> lift it to 64X for OVERCOMMIT_ALWAYS and OVERCOMMIT_GUESS policies.  Also
> add a sysctl handler to adjust it when the policy is reconfigured.
> 
> Benchmark with the same testcase in [1] shows 53% improvement on a 8C/16T
> desktop, and 2097%(20X) on a 4S/72C/144T server.  We tested with test
> platforms in 0day (server, desktop and laptop), and 80%+ platforms shows
> improvements with that test.  And whether it shows improvements depends on
> if the test mmap size is bigger than the batch number computed.
> 
> And if the lift is 16X, 1/3 of the platforms will show improvements,
> though it should help the mmap/unmap usage generally, as Michal Hocko
> mentioned:
> 
> : I believe that there are non-synthetic worklaods which would benefit from
> : a larger batch.  E.g.  large in memory databases which do large mmaps
> : during startups from multiple threads.
> 
> [1] https://lore.kernel.org/lkml/20200305062138.GI5972@shao2-debian/
> 
> Link: http://lkml.kernel.org/r/1589611660-89854-4-git-send-email-feng.tang@intel.com
> Signed-off-by: Feng Tang <feng.tang@intel.com>
> Cc: Michal Hocko <mhocko@suse.com>
> Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Mel Gorman <mgorman@suse.de>
> Cc: Kees Cook <keescook@chromium.org>
> Cc: Andi Kleen <andi.kleen@intel.com>
> Cc: Tim Chen <tim.c.chen@intel.com>
> Cc: Dave Hansen <dave.hansen@intel.com>
> Cc: Huang Ying <ying.huang@intel.com>
> Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

Acked-by: Michal Hocko <mhocko@suse.com>

> ---
>  include/linux/mm.h   |  2 ++
>  include/linux/mman.h |  4 ++++
>  kernel/sysctl.c      |  2 +-
>  mm/mm_init.c         | 18 ++++++++++++++----
>  mm/util.c            | 12 ++++++++++++
>  5 files changed, 33 insertions(+), 5 deletions(-)
> 
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 573947c..c2efea6 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -206,6 +206,8 @@ int overcommit_ratio_handler(struct ctl_table *, int, void *, size_t *,
>  		loff_t *);
>  int overcommit_kbytes_handler(struct ctl_table *, int, void *, size_t *,
>  		loff_t *);
> +int overcommit_policy_handler(struct ctl_table *, int, void *, size_t *,
> +		loff_t *);
>  
>  #define nth_page(page,n) pfn_to_page(page_to_pfn((page)) + (n))
>  
> diff --git a/include/linux/mman.h b/include/linux/mman.h
> index 4b08e9c..91c93c1 100644
> --- a/include/linux/mman.h
> +++ b/include/linux/mman.h
> @@ -57,8 +57,12 @@ extern struct percpu_counter vm_committed_as;
>  
>  #ifdef CONFIG_SMP
>  extern s32 vm_committed_as_batch;
> +extern void mm_compute_batch(void);
>  #else
>  #define vm_committed_as_batch 0
> +static inline void mm_compute_batch(void)
> +{
> +}
>  #endif
>  
>  unsigned long vm_memory_committed(void);
> diff --git a/kernel/sysctl.c b/kernel/sysctl.c
> index db1ce7a..9456c86 100644
> --- a/kernel/sysctl.c
> +++ b/kernel/sysctl.c
> @@ -2650,7 +2650,7 @@ static struct ctl_table vm_table[] = {
>  		.data		= &sysctl_overcommit_memory,
>  		.maxlen		= sizeof(sysctl_overcommit_memory),
>  		.mode		= 0644,
> -		.proc_handler	= proc_dointvec_minmax,
> +		.proc_handler	= overcommit_policy_handler,
>  		.extra1		= SYSCTL_ZERO,
>  		.extra2		= &two,
>  	},
> diff --git a/mm/mm_init.c b/mm/mm_init.c
> index 435e5f7..c5a6fb1 100644
> --- a/mm/mm_init.c
> +++ b/mm/mm_init.c
> @@ -13,6 +13,7 @@
>  #include <linux/memory.h>
>  #include <linux/notifier.h>
>  #include <linux/sched.h>
> +#include <linux/mman.h>
>  #include "internal.h"
>  
>  #ifdef CONFIG_DEBUG_MEMORY_INIT
> @@ -144,14 +145,23 @@ EXPORT_SYMBOL_GPL(mm_kobj);
>  #ifdef CONFIG_SMP
>  s32 vm_committed_as_batch = 32;
>  
> -static void __meminit mm_compute_batch(void)
> +void mm_compute_batch(void)
>  {
>  	u64 memsized_batch;
>  	s32 nr = num_present_cpus();
>  	s32 batch = max_t(s32, nr*2, 32);
> -
> -	/* batch size set to 0.4% of (total memory/#cpus), or max int32 */
> -	memsized_batch = min_t(u64, (totalram_pages()/nr)/256, 0x7fffffff);
> +	unsigned long ram_pages = totalram_pages();
> +
> +	/*
> +	 * For policy of OVERCOMMIT_NEVER, set batch size to 0.4%
> +	 * of (total memory/#cpus), and lift it to 25% for other
> +	 * policies to easy the possible lock contention for percpu_counter
> +	 * vm_committed_as, while the max limit is INT_MAX
> +	 */
> +	if (sysctl_overcommit_memory == OVERCOMMIT_NEVER)
> +		memsized_batch = min_t(u64, ram_pages/nr/256, INT_MAX);
> +	else
> +		memsized_batch = min_t(u64, ram_pages/nr/4, INT_MAX);
>  
>  	vm_committed_as_batch = max_t(s32, memsized_batch, batch);
>  }
> diff --git a/mm/util.c b/mm/util.c
> index fe63271..580d268 100644
> --- a/mm/util.c
> +++ b/mm/util.c
> @@ -746,6 +746,18 @@ int overcommit_ratio_handler(struct ctl_table *table, int write, void *buffer,
>  	return ret;
>  }
>  
> +int overcommit_policy_handler(struct ctl_table *table, int write, void *buffer,
> +		size_t *lenp, loff_t *ppos)
> +{
> +	int ret;
> +
> +	ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
> +	if (ret == 0 && write)
> +		mm_compute_batch();
> +
> +	return ret;
> +}
> +
>  int overcommit_kbytes_handler(struct ctl_table *table, int write, void *buffer,
>  		size_t *lenp, loff_t *ppos)
>  {
> -- 
> 2.7.4
> 

-- 
Michal Hocko
SUSE Labs