From: Rik van Riel <riel@surriel.com>
To: linux-kernel@vger.kernel.org
Cc: x86@kernel.org, luto@kernel.org, dave.hansen@linux.intel.com,
mingo@kernel.org, kernel-team@fb.com, tglx@linutronix.de,
efault@gmx.de, songliubraving@fb.com, hpa@zytor.com,
Rik van Riel <riel@surriel.com>
Subject: [PATCH 7/7] x86,switch_mm: skip atomic operations for init_mm
Date: Fri, 29 Jun 2018 10:29:18 -0400 [thread overview]
Message-ID: <20180629142918.26601-8-riel@surriel.com> (raw)
In-Reply-To: <20180629142918.26601-1-riel@surriel.com>
Song noticed switch_mm_irqs_off taking a lot of CPU time in recent
kernels,using 1.8% of a 48 CPU system during a netperf to localhost run.
Digging into the profile, we noticed that cpumask_clear_cpu and
cpumask_set_cpu together take about half of the CPU time taken by
switch_mm_irqs_off.
However, the CPUs running netperf end up switching back and forth
between netperf and the idle task, which does not require changes
to the mm_cpumask. Furthermore, the init_mm cpumask ends up being
the most heavily contended one in the system.
Simply skipping changes to mm_cpumask(&init_mm) reduces overhead.
Signed-off-by: Rik van Riel <riel@surriel.com>
Reported-and-tested-by: Song Liu <songliubraving@fb.com>
---
arch/x86/mm/tlb.c | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index 5a01fcb22a7e..b55e6b7df7c9 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -311,14 +311,17 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,
}
/* Stop remote flushes for the previous mm */
- VM_WARN_ON_ONCE(!cpumask_test_cpu(cpu, mm_cpumask(real_prev)) &&
- real_prev != &init_mm);
- cpumask_clear_cpu(cpu, mm_cpumask(real_prev));
+ if (real_prev != &init_mm) {
+ VM_WARN_ON_ONCE(!cpumask_test_cpu(cpu,
+ mm_cpumask(real_prev)));
+ cpumask_clear_cpu(cpu, mm_cpumask(real_prev));
+ }
/*
* Start remote flushes and then read tlb_gen.
*/
- cpumask_set_cpu(cpu, mm_cpumask(next));
+ if (next != &init_mm)
+ cpumask_set_cpu(cpu, mm_cpumask(next));
next_tlb_gen = atomic64_read(&next->context.tlb_gen);
choose_new_asid(next, next_tlb_gen, &new_asid, &need_flush);
--
2.14.4
next prev parent reply other threads:[~2018-06-29 14:29 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-06-29 14:29 [PATCH v3 0/7] x86,tlb,mm: make lazy TLB mode even lazier Rik van Riel
2018-06-29 14:29 ` [PATCH 1/7] mm: allocate mm_cpumask dynamically based on nr_cpu_ids Rik van Riel
2018-06-30 4:30 ` kbuild test robot
2018-06-29 14:29 ` [PATCH 2/7] x86,tlb: leave lazy TLB mode at page table free time Rik van Riel
2018-06-29 16:39 ` Dave Hansen
2018-06-29 16:56 ` Rik van Riel
2018-06-29 14:29 ` [PATCH 3/7] x86,mm: restructure switch_mm_irqs_off Rik van Riel
2018-06-29 16:49 ` Dave Hansen
2018-06-29 14:29 ` [PATCH 4/7] x86,tlb: make lazy TLB mode lazier Rik van Riel
2018-06-29 17:05 ` Dave Hansen
2018-06-29 17:29 ` Rik van Riel
2018-06-29 14:29 ` [PATCH 5/7] x86,tlb: only send page table free TLB flush to lazy TLB CPUs Rik van Riel
2018-07-07 12:26 ` Mike Galbraith
2018-07-07 13:39 ` Mike Galbraith
2018-07-07 21:21 ` Rik van Riel
2018-06-29 14:29 ` [PATCH 6/7] x86,mm: always use lazy TLB mode Rik van Riel
2018-06-29 14:29 ` Rik van Riel [this message]
2018-06-29 17:06 ` [PATCH v3 0/7] x86,tlb,mm: make lazy TLB mode even lazier Dave Hansen
2018-07-06 21:56 [PATCH v4 " Rik van Riel
2018-07-06 21:56 ` [PATCH 7/7] x86,switch_mm: skip atomic operations for init_mm Rik van Riel
2018-07-10 14:28 [PATCH v5 0/7] x86,tlb,mm: make lazy TLB mode even lazier Rik van Riel
2018-07-10 14:28 ` [PATCH 7/7] x86,switch_mm: skip atomic operations for init_mm Rik van Riel
2018-07-15 23:04 ` Ingo Molnar
2018-07-15 23:49 ` Rik van Riel
2018-07-16 1:04 ` Ingo Molnar
2018-07-16 18:37 ` Rik van Riel
2018-07-16 19:03 [PATCH v6 0/7] x86,tlb,mm: make lazy TLB mode even lazier Rik van Riel
2018-07-16 19:03 ` [PATCH 7/7] x86,switch_mm: skip atomic operations for init_mm Rik van Riel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180629142918.26601-8-riel@surriel.com \
--to=riel@surriel.com \
--cc=dave.hansen@linux.intel.com \
--cc=efault@gmx.de \
--cc=hpa@zytor.com \
--cc=kernel-team@fb.com \
--cc=linux-kernel@vger.kernel.org \
--cc=luto@kernel.org \
--cc=mingo@kernel.org \
--cc=songliubraving@fb.com \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).