From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5E76C28CF6 for ; Wed, 1 Aug 2018 10:03:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 71483208A2 for ; Wed, 1 Aug 2018 10:03:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 71483208A2 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=surriel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389314AbeHALsM (ORCPT ); Wed, 1 Aug 2018 07:48:12 -0400 Received: from shelob.surriel.com ([96.67.55.147]:56014 "EHLO shelob.surriel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389216AbeHALsL (ORCPT ); Wed, 1 Aug 2018 07:48:11 -0400 Received: from imladris.surriel.com ([96.67.55.152]) by shelob.surriel.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.90_1) (envelope-from ) id 1fkny6-00010M-CY; Wed, 01 Aug 2018 06:02:58 -0400 From: Rik van Riel To: linux-kernel@vger.kernel.org Cc: kernel-team@fb.com, mingo@kernel.org, peterz@infradead.org, luto@kernel.org, x86@kernel.org, efault@gmx.de, dave.hansen@intel.com, Rik van Riel Subject: [PATCH 04/11] x86,mm: use on_each_cpu_cond for TLB flushes Date: Wed, 1 Aug 2018 06:02:48 -0400 Message-Id: <20180801100255.4278-5-riel@surriel.com> X-Mailer: git-send-email 2.14.4 In-Reply-To: <20180801100255.4278-1-riel@surriel.com> References: <20180801100255.4278-1-riel@surriel.com> Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Instead of open coding bitmap magic, use on_each_cpu_cond to determine which CPUs to send TLB flush IPIs to. This might be a little bit slower than examining the bitmaps, but it should be a lot easier to maintain in the long run. Suggested-by: Peter Zijlstra Signed-off-by: Rik van Riel --- arch/x86/mm/tlb.c | 75 +++++++++++-------------------------------------------- 1 file changed, 15 insertions(+), 60 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 5321e02c4e09..671cc66df801 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -582,12 +582,19 @@ static void flush_tlb_func_remote(void *info) flush_tlb_func_common(f, false, TLB_REMOTE_SHOOTDOWN); } +static bool tlb_is_lazy(int cpu, void *data) +{ + return per_cpu(cpu_tlbstate.is_lazy, cpu); +} + +static bool tlb_is_not_lazy(int cpu, void *data) +{ + return !per_cpu(cpu_tlbstate.is_lazy, cpu); +} + void native_flush_tlb_others(const struct cpumask *cpumask, const struct flush_tlb_info *info) { - cpumask_var_t lazymask; - unsigned int cpu; - count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); if (info->end == TLB_FLUSH_ALL) trace_tlb_flush(TLB_REMOTE_SEND_IPI, TLB_FLUSH_ALL); @@ -596,6 +603,7 @@ void native_flush_tlb_others(const struct cpumask *cpumask, (info->end - info->start) >> PAGE_SHIFT); if (is_uv_system()) { + unsigned int cpu; /* * This whole special case is confused. UV has a "Broadcast * Assist Unit", which seems to be a fancy way to send IPIs. @@ -619,28 +627,8 @@ void native_flush_tlb_others(const struct cpumask *cpumask, return; } - /* - * A temporary cpumask is used in order to skip sending IPIs - * to CPUs in lazy TLB state, while keeping them in mm_cpumask(mm). - * If the allocation fails, simply IPI every CPU in mm_cpumask. - */ - if (!alloc_cpumask_var(&lazymask, GFP_ATOMIC)) { - smp_call_function_many(cpumask, flush_tlb_func_remote, - (void *)info, 1); - return; - } - - cpumask_copy(lazymask, cpumask); - - for_each_cpu(cpu, lazymask) { - if (per_cpu(cpu_tlbstate.is_lazy, cpu)) - cpumask_clear_cpu(cpu, lazymask); - } - - smp_call_function_many(lazymask, flush_tlb_func_remote, - (void *)info, 1); - - free_cpumask_var(lazymask); + on_each_cpu_cond_mask(tlb_is_not_lazy, flush_tlb_func_remote, + (void *)info, 1, GFP_ATOMIC, cpumask); } /* @@ -709,50 +697,17 @@ void tlb_flush_remove_tables_local(void *arg) } } -static void mm_fill_lazy_tlb_cpu_mask(struct mm_struct *mm, - struct cpumask *lazy_cpus) -{ - int cpu; - - for_each_cpu(cpu, mm_cpumask(mm)) { - if (!per_cpu(cpu_tlbstate.is_lazy, cpu)) - cpumask_set_cpu(cpu, lazy_cpus); - } -} - void tlb_flush_remove_tables(struct mm_struct *mm) { int cpu = get_cpu(); - cpumask_var_t lazy_cpus; if (cpumask_any_but(mm_cpumask(mm), cpu) >= nr_cpu_ids) { put_cpu(); return; } - if (!zalloc_cpumask_var(&lazy_cpus, GFP_ATOMIC)) { - /* - * If the cpumask allocation fails, do a brute force flush - * on all the CPUs that have this mm loaded. - */ - smp_call_function_many(mm_cpumask(mm), - tlb_flush_remove_tables_local, (void *)mm, 1); - put_cpu(); - return; - } - - /* - * CPUs with !is_lazy either received a TLB flush IPI while the user - * pages in this address range were unmapped, or have context switched - * and reloaded %CR3 since then. - * - * Shootdown IPIs at page table freeing time only need to be sent to - * CPUs that may have out of date TLB contents. - */ - mm_fill_lazy_tlb_cpu_mask(mm, lazy_cpus); - smp_call_function_many(lazy_cpus, - tlb_flush_remove_tables_local, (void *)mm, 1); - free_cpumask_var(lazy_cpus); + on_each_cpu_cond_mask(tlb_is_lazy, tlb_flush_remove_tables_local, + (void *)mm, 1, GFP_ATOMIC, mm_cpumask(mm)); put_cpu(); } -- 2.14.4