From: Peter Zijlstra <peterz@infradead.org>
To: dave.hansen@intel.com, luto@kernel.org, peterz@infradead.org
Cc: x86@kernel.org, Tom.StDenis@amd.com, linux-kernel@vger.kernel.org
Subject: [RFC][PATCH 06/10] x86/mm/cpa: Optimize cpa_flush_array() TLB invalidation
Date: Mon, 03 Dec 2018 18:03:49 +0100 [thread overview]
Message-ID: <20181203171043.430001980@infradead.org> (raw)
In-Reply-To: 20181203170343.054618965@infradead.org
Instead of punting and doing tlb_flush_all(), do the same as
flush_tlb_kernel_range() does and use single page invalidations.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
arch/x86/mm/mm_internal.h | 2 ++
arch/x86/mm/pageattr.c | 42 ++++++++++++++++++++++++------------------
arch/x86/mm/tlb.c | 4 +++-
3 files changed, 29 insertions(+), 19 deletions(-)
--- a/arch/x86/mm/mm_internal.h
+++ b/arch/x86/mm/mm_internal.h
@@ -19,4 +19,6 @@ extern int after_bootmem;
void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache);
+extern unsigned long tlb_single_page_flush_ceiling;
+
#endif /* __X86_MM_INTERNAL_H */
--- a/arch/x86/mm/pageattr.c
+++ b/arch/x86/mm/pageattr.c
@@ -26,6 +26,8 @@
#include <asm/pat.h>
#include <asm/set_memory.h>
+#include "mm_internal.h"
+
/*
* The current flushing context - we pass it instead of 5 arguments:
*/
@@ -346,16 +348,26 @@ static void cpa_flush_range(unsigned lon
}
}
-static void cpa_flush_array(unsigned long baddr, unsigned long *start,
- int numpages, int cache,
- int in_flags, struct page **pages)
+void __cpa_flush_array(void *data)
{
- unsigned int i, level;
+ struct cpa_data *cpa = data;
+ unsigned int i;
- if (__inv_flush_all(cache))
+ for (i = 0; i < cpa->numpages; i++)
+ __flush_tlb_one_kernel(__cpa_addr(cpa, i));
+}
+
+static void cpa_flush_array(struct cpa_data *cpa, int cache)
+{
+ unsigned int i;
+
+ if (cpa_check_flush_all(cache))
return;
- flush_tlb_all();
+ if (cpa->numpages <= tlb_single_page_flush_ceiling)
+ on_each_cpu(__cpa_flush_array, cpa, 1);
+ else
+ flush_tlb_all();
if (!cache)
return;
@@ -366,15 +378,11 @@ static void cpa_flush_array(unsigned lon
* will cause all other CPUs to flush the same
* cachelines:
*/
- for (i = 0; i < numpages; i++) {
- unsigned long addr;
+ for (i = 0; i < cpa->numpages; i++) {
+ unsigned long addr = __cpa_addr(cpa, i);
+ unsigned int level;
pte_t *pte;
- if (in_flags & CPA_PAGES_ARRAY)
- addr = (unsigned long)page_address(pages[i]);
- else
- addr = start[i];
-
pte = lookup_address(addr, &level);
/*
@@ -1771,12 +1779,10 @@ static int change_page_attr_set_clr(unsi
goto out;
}
- if (cpa.flags & (CPA_PAGES_ARRAY | CPA_ARRAY)) {
- cpa_flush_array(baddr, addr, numpages, cache,
- cpa.flags, pages);
- } else {
+ if (cpa.flags & (CPA_PAGES_ARRAY | CPA_ARRAY))
+ cpa_flush_array(&cpa, cache);
+ else
cpa_flush_range(baddr, numpages, cache);
- }
out:
return ret;
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -15,6 +15,8 @@
#include <asm/apic.h>
#include <asm/uv/uv.h>
+#include "mm_internal.h"
+
/*
* TLB flushing, formerly SMP-only
* c/o Linus Torvalds.
@@ -721,7 +723,7 @@ void native_flush_tlb_others(const struc
*
* This is in units of pages.
*/
-static unsigned long tlb_single_page_flush_ceiling __read_mostly = 33;
+unsigned long tlb_single_page_flush_ceiling __read_mostly = 33;
void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
unsigned long end, unsigned int stride_shift,
next prev parent reply other threads:[~2018-12-03 17:14 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-12-03 17:03 [RFC][PATCH 00/10] x86/mm/cpa: Various fixes and improvements Peter Zijlstra
2018-12-03 17:03 ` [PATCH 01/10] x86/mm/cpa: Fix cpa_flush_array() TLB invalidation Peter Zijlstra
2018-12-17 18:20 ` [tip:x86/mm] " tip-bot for Peter Zijlstra
2018-12-03 17:03 ` [RFC][PATCH 02/10] x86/mm/cpa: Add ARRAY and PAGES_ARRAY selftests Peter Zijlstra
2018-12-17 18:21 ` [tip:x86/mm] " tip-bot for Peter Zijlstra
2018-12-03 17:03 ` [RFC][PATCH 03/10] x86/mm/cpa: Add __cpa_addr() helper Peter Zijlstra
2018-12-17 18:21 ` [tip:x86/mm] " tip-bot for Peter Zijlstra
2018-12-03 17:03 ` [RFC][PATCH 04/10] x86/mm/cpa: Make cpa_data::vaddr invariant Peter Zijlstra
2018-12-17 18:22 ` [tip:x86/mm] " tip-bot for Peter Zijlstra
2018-12-03 17:03 ` [RFC][PATCH 05/10] x86/mm/cpa: Cleanup Peter Zijlstra
2018-12-17 18:23 ` [tip:x86/mm] x86/mm/cpa: Simplify the code after making cpa->vaddr invariant tip-bot for Peter Zijlstra
2018-12-03 17:03 ` Peter Zijlstra [this message]
2018-12-17 18:23 ` [tip:x86/mm] x86/mm/cpa: Optimize cpa_flush_array() TLB invalidation tip-bot for Peter Zijlstra
2018-12-03 17:03 ` [RFC][PATCH 07/10] x86/mm/cpa: Make cpa_data::numpages invariant Peter Zijlstra
2018-12-17 18:24 ` [tip:x86/mm] " tip-bot for Peter Zijlstra
2018-12-03 17:03 ` [RFC][PATCH 08/10] x86/mm/cpa: Fold cpa_flush_range() and cpa_flush_array() Peter Zijlstra
2018-12-17 18:24 ` [tip:x86/mm] x86/mm/cpa: Fold cpa_flush_range() and cpa_flush_array() into a single cpa_flush() function tip-bot for Peter Zijlstra
2018-12-03 17:03 ` [RFC][PATCH 09/10] x86/mm/cpa: Better use clflushopt Peter Zijlstra
2018-12-17 18:25 ` [tip:x86/mm] x86/mm/cpa: Better use CLFLUSHOPT tip-bot for Peter Zijlstra
2018-12-03 17:03 ` [RFC][PATCH 10/10] x86/mm/cpa: Rename @addrinarray Peter Zijlstra
2018-12-17 18:26 ` [tip:x86/mm] x86/mm/cpa: Rename @addrinarray to @numpages tip-bot for Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20181203171043.430001980@infradead.org \
--to=peterz@infradead.org \
--cc=Tom.StDenis@amd.com \
--cc=dave.hansen@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=luto@kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).