From: Nicholas Piggin <npiggin@gmail.com> To: linux-kernel@vger.kernel.org Cc: Nicholas Piggin <npiggin@gmail.com>, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, Anton Blanchard <anton@ozlabs.org>, Andy Lutomirski <luto@kernel.org> Subject: [PATCH v2 3/5] lazy tlb: shoot lazies, a non-refcounting lazy tlb option Date: Mon, 14 Dec 2020 16:53:10 +1000 [thread overview] Message-ID: <20201214065312.270062-4-npiggin@gmail.com> (raw) In-Reply-To: <20201214065312.270062-1-npiggin@gmail.com> On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread). Abandoning lazy tlb slows switching down quite a bit in the important user->idle->user cases, so instead implement a non-refcounted scheme that causes __mmdrop() to IPI all CPUs in the mm_cpumask and shoot down any remaining lazy ones. Shootdown IPIs are some concern, but they have not been observed to be a big problem with this scheme (the powerpc implementation generated 314 additional interrupts on a 144 CPU system during a kernel compile). There are a number of strategies that could be employed to reduce IPIs if they turn out to be a problem for some workload. Signed-off-by: Nicholas Piggin <npiggin@gmail.com> --- arch/Kconfig | 17 +++++++++++++++-- kernel/fork.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+), 2 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 84faaba66364..e69c974369cc 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -443,9 +443,22 @@ config MMU_LAZY_TLB config MMU_LAZY_TLB_REFCOUNT def_bool y depends on MMU_LAZY_TLB + depends on !MMU_LAZY_TLB_SHOOTDOWN help - This must be enabled if MMU_LAZY_TLB is enabled until the next - patch. + This refcounts the mm that is used as the lazy TLB mm when switching + switching to a kernel thread. + +config MMU_LAZY_TLB_SHOOTDOWN + bool + depends on MMU_LAZY_TLB + help + Instead of refcounting the "lazy tlb" mm struct, which can cause + contention with multi-threaded apps on large multiprocessor systems, + this option causes __mmdrop to IPI all CPUs in the mm_cpumask and + switch to init_mm if they were using the to-be-freed mm as the lazy + tlb. To implement this, architectures must use _lazy_tlb variants of + mm refcounting, and mm_cpumask must include at least all possible + CPUs in which mm might be lazy. config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/kernel/fork.c b/kernel/fork.c index 6d266388d380..74b972d2d8a9 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -669,6 +669,53 @@ static void check_mm(struct mm_struct *mm) #define allocate_mm() (kmem_cache_alloc(mm_cachep, GFP_KERNEL)) #define free_mm(mm) (kmem_cache_free(mm_cachep, (mm))) +static void do_shoot_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + + if (current->active_mm == mm) { + WARN_ON_ONCE(current->mm); + current->active_mm = &init_mm; + switch_mm(mm, &init_mm, current); + } +} + +static void do_check_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + + WARN_ON_ONCE(current->active_mm == mm); +} + +static void shoot_lazy_tlbs(struct mm_struct *mm) +{ + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_SHOOTDOWN)) { + /* + * IPI overheads have not found to be expensive, but they could + * be reduced in a number of possible ways, for example (in + * roughly increasing order of complexity): + * - A batch of mms requiring IPIs could be gathered and freed + * at once. + * - CPUs could store their active mm somewhere that can be + * remotely checked without a lock, to filter out + * false-positives in the cpumask. + * - After mm_users or mm_count reaches zero, switching away + * from the mm could clear mm_cpumask to reduce some IPIs + * (some batching or delaying would help). + * - A delayed freeing and RCU-like quiescing sequence based on + * mm switching to avoid IPIs completely. + */ + on_each_cpu_mask(mm_cpumask(mm), do_shoot_lazy_tlb, (void *)mm, 1); + if (IS_ENABLED(CONFIG_DEBUG_VM)) + on_each_cpu(do_check_lazy_tlb, (void *)mm, 1); + } else { + /* + * In this case, lazy tlb mms are refounted and would not reach + * __mmdrop until all CPUs have switched away and mmdrop()ed. + */ + } +} + /* * Called when the last reference to the mm * is dropped: either by a lazy thread or by @@ -678,7 +725,12 @@ void __mmdrop(struct mm_struct *mm) { BUG_ON(mm == &init_mm); WARN_ON_ONCE(mm == current->mm); + + /* Ensure no CPUs are using this as their lazy tlb mm */ + shoot_lazy_tlbs(mm); + WARN_ON_ONCE(mm == current->active_mm); + mm_free_pgd(mm); destroy_context(mm); mmu_notifier_subscriptions_destroy(mm); -- 2.23.0
WARNING: multiple messages have this Message-ID (diff)
From: Nicholas Piggin <npiggin@gmail.com> To: linux-kernel@vger.kernel.org Cc: linux-arch@vger.kernel.org, Nicholas Piggin <npiggin@gmail.com>, linux-mm@kvack.org, Andy Lutomirski <luto@kernel.org>, linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 3/5] lazy tlb: shoot lazies, a non-refcounting lazy tlb option Date: Mon, 14 Dec 2020 16:53:10 +1000 [thread overview] Message-ID: <20201214065312.270062-4-npiggin@gmail.com> (raw) In-Reply-To: <20201214065312.270062-1-npiggin@gmail.com> On big systems, the mm refcount can become highly contented when doing a lot of context switching with threaded applications (particularly switching between the idle thread and an application thread). Abandoning lazy tlb slows switching down quite a bit in the important user->idle->user cases, so instead implement a non-refcounted scheme that causes __mmdrop() to IPI all CPUs in the mm_cpumask and shoot down any remaining lazy ones. Shootdown IPIs are some concern, but they have not been observed to be a big problem with this scheme (the powerpc implementation generated 314 additional interrupts on a 144 CPU system during a kernel compile). There are a number of strategies that could be employed to reduce IPIs if they turn out to be a problem for some workload. Signed-off-by: Nicholas Piggin <npiggin@gmail.com> --- arch/Kconfig | 17 +++++++++++++++-- kernel/fork.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+), 2 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 84faaba66364..e69c974369cc 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -443,9 +443,22 @@ config MMU_LAZY_TLB config MMU_LAZY_TLB_REFCOUNT def_bool y depends on MMU_LAZY_TLB + depends on !MMU_LAZY_TLB_SHOOTDOWN help - This must be enabled if MMU_LAZY_TLB is enabled until the next - patch. + This refcounts the mm that is used as the lazy TLB mm when switching + switching to a kernel thread. + +config MMU_LAZY_TLB_SHOOTDOWN + bool + depends on MMU_LAZY_TLB + help + Instead of refcounting the "lazy tlb" mm struct, which can cause + contention with multi-threaded apps on large multiprocessor systems, + this option causes __mmdrop to IPI all CPUs in the mm_cpumask and + switch to init_mm if they were using the to-be-freed mm as the lazy + tlb. To implement this, architectures must use _lazy_tlb variants of + mm refcounting, and mm_cpumask must include at least all possible + CPUs in which mm might be lazy. config ARCH_HAVE_NMI_SAFE_CMPXCHG bool diff --git a/kernel/fork.c b/kernel/fork.c index 6d266388d380..74b972d2d8a9 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -669,6 +669,53 @@ static void check_mm(struct mm_struct *mm) #define allocate_mm() (kmem_cache_alloc(mm_cachep, GFP_KERNEL)) #define free_mm(mm) (kmem_cache_free(mm_cachep, (mm))) +static void do_shoot_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + + if (current->active_mm == mm) { + WARN_ON_ONCE(current->mm); + current->active_mm = &init_mm; + switch_mm(mm, &init_mm, current); + } +} + +static void do_check_lazy_tlb(void *arg) +{ + struct mm_struct *mm = arg; + + WARN_ON_ONCE(current->active_mm == mm); +} + +static void shoot_lazy_tlbs(struct mm_struct *mm) +{ + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_SHOOTDOWN)) { + /* + * IPI overheads have not found to be expensive, but they could + * be reduced in a number of possible ways, for example (in + * roughly increasing order of complexity): + * - A batch of mms requiring IPIs could be gathered and freed + * at once. + * - CPUs could store their active mm somewhere that can be + * remotely checked without a lock, to filter out + * false-positives in the cpumask. + * - After mm_users or mm_count reaches zero, switching away + * from the mm could clear mm_cpumask to reduce some IPIs + * (some batching or delaying would help). + * - A delayed freeing and RCU-like quiescing sequence based on + * mm switching to avoid IPIs completely. + */ + on_each_cpu_mask(mm_cpumask(mm), do_shoot_lazy_tlb, (void *)mm, 1); + if (IS_ENABLED(CONFIG_DEBUG_VM)) + on_each_cpu(do_check_lazy_tlb, (void *)mm, 1); + } else { + /* + * In this case, lazy tlb mms are refounted and would not reach + * __mmdrop until all CPUs have switched away and mmdrop()ed. + */ + } +} + /* * Called when the last reference to the mm * is dropped: either by a lazy thread or by @@ -678,7 +725,12 @@ void __mmdrop(struct mm_struct *mm) { BUG_ON(mm == &init_mm); WARN_ON_ONCE(mm == current->mm); + + /* Ensure no CPUs are using this as their lazy tlb mm */ + shoot_lazy_tlbs(mm); + WARN_ON_ONCE(mm == current->active_mm); + mm_free_pgd(mm); destroy_context(mm); mmu_notifier_subscriptions_destroy(mm); -- 2.23.0
next prev parent reply other threads:[~2020-12-14 7:02 UTC|newest] Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-12-14 6:53 [PATCH v2 0/5] shoot lazy tlbs Nicholas Piggin 2020-12-14 6:53 ` Nicholas Piggin 2020-12-14 6:53 ` [PATCH v2 1/5] lazy tlb: introduce lazy mm refcount helper functions Nicholas Piggin 2020-12-14 6:53 ` Nicholas Piggin 2020-12-14 6:53 ` [PATCH v2 2/5] lazy tlb: allow lazy tlb mm switching to be configurable Nicholas Piggin 2020-12-14 6:53 ` Nicholas Piggin 2020-12-14 6:53 ` Nicholas Piggin [this message] 2020-12-14 6:53 ` [PATCH v2 3/5] lazy tlb: shoot lazies, a non-refcounting lazy tlb option Nicholas Piggin 2020-12-14 7:04 ` Randy Dunlap 2020-12-14 7:04 ` Randy Dunlap 2020-12-14 6:53 ` [PATCH v2 4/5] powerpc: use lazy mm refcount helper functions Nicholas Piggin 2020-12-14 6:53 ` Nicholas Piggin 2020-12-14 6:53 ` [PATCH v2 5/5] powerpc/64s: enable MMU_LAZY_TLB_SHOOTDOWN Nicholas Piggin 2020-12-14 6:53 ` Nicholas Piggin 2021-02-04 8:21 ` [PATCH v2 0/5] shoot lazy tlbs Nicholas Piggin 2021-02-04 8:21 ` Nicholas Piggin
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20201214065312.270062-4-npiggin@gmail.com \ --to=npiggin@gmail.com \ --cc=anton@ozlabs.org \ --cc=linux-arch@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linuxppc-dev@lists.ozlabs.org \ --cc=luto@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.