* [PATCH v3 1/3] x86/mm: Add INVPCID helpers
2016-01-29 19:42 [PATCH v3 0/3] x86/mm: INVPCID support Andy Lutomirski
@ 2016-01-29 19:42 ` Andy Lutomirski
2016-01-29 19:42 ` [PATCH v3 2/3] x86/mm: Add a noinvpcid option to turn off INVPCID Andy Lutomirski
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Andy Lutomirski @ 2016-01-29 19:42 UTC (permalink / raw)
To: x86, linux-kernel
Cc: Borislav Petkov, Brian Gerst, Dave Hansen, Linus Torvalds,
Oleg Nesterov, linux-mm, Andrey Ryabinin, Andy Lutomirski
This adds helpers for each of the four currently-specified INVPCID
modes.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
---
arch/x86/include/asm/tlbflush.h | 48 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 48 insertions(+)
diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h
index 6df2029405a3..8b576832777e 100644
--- a/arch/x86/include/asm/tlbflush.h
+++ b/arch/x86/include/asm/tlbflush.h
@@ -7,6 +7,54 @@
#include <asm/processor.h>
#include <asm/special_insns.h>
+static inline void __invpcid(unsigned long pcid, unsigned long addr,
+ unsigned long type)
+{
+ u64 desc[2] = { pcid, addr };
+
+ /*
+ * The memory clobber is because the whole point is to invalidate
+ * stale TLB entries and, especially if we're flushing global
+ * mappings, we don't want the compiler to reorder any subsequent
+ * memory accesses before the TLB flush.
+ *
+ * The hex opcode is invpcid (%ecx), %eax in 32-bit mode and
+ * invpcid (%rcx), %rax in long mode.
+ */
+ asm volatile (".byte 0x66, 0x0f, 0x38, 0x82, 0x01"
+ : : "m" (desc), "a" (type), "c" (desc) : "memory");
+}
+
+#define INVPCID_TYPE_INDIV_ADDR 0
+#define INVPCID_TYPE_SINGLE_CTXT 1
+#define INVPCID_TYPE_ALL_INCL_GLOBAL 2
+#define INVPCID_TYPE_ALL_NON_GLOBAL 3
+
+/* Flush all mappings for a given pcid and addr, not including globals. */
+static inline void invpcid_flush_one(unsigned long pcid,
+ unsigned long addr)
+{
+ __invpcid(pcid, addr, INVPCID_TYPE_INDIV_ADDR);
+}
+
+/* Flush all mappings for a given PCID, not including globals. */
+static inline void invpcid_flush_single_context(unsigned long pcid)
+{
+ __invpcid(pcid, 0, INVPCID_TYPE_SINGLE_CTXT);
+}
+
+/* Flush all mappings, including globals, for all PCIDs. */
+static inline void invpcid_flush_all(void)
+{
+ __invpcid(0, 0, INVPCID_TYPE_ALL_INCL_GLOBAL);
+}
+
+/* Flush all mappings for all PCIDs except globals. */
+static inline void invpcid_flush_all_nonglobals(void)
+{
+ __invpcid(0, 0, INVPCID_TYPE_ALL_NON_GLOBAL);
+}
+
#ifdef CONFIG_PARAVIRT
#include <asm/paravirt.h>
#else
--
2.5.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH v3 2/3] x86/mm: Add a noinvpcid option to turn off INVPCID
2016-01-29 19:42 [PATCH v3 0/3] x86/mm: INVPCID support Andy Lutomirski
2016-01-29 19:42 ` [PATCH v3 1/3] x86/mm: Add INVPCID helpers Andy Lutomirski
@ 2016-01-29 19:42 ` Andy Lutomirski
2016-01-29 19:42 ` [PATCH v3 3/3] x86/mm: If INVPCID is available, use it to flush global mappings Andy Lutomirski
2016-02-01 10:51 ` [PATCH v3 0/3] x86/mm: INVPCID support Borislav Petkov
3 siblings, 0 replies; 5+ messages in thread
From: Andy Lutomirski @ 2016-01-29 19:42 UTC (permalink / raw)
To: x86, linux-kernel
Cc: Borislav Petkov, Brian Gerst, Dave Hansen, Linus Torvalds,
Oleg Nesterov, linux-mm, Andrey Ryabinin, Andy Lutomirski
This adds a chicken bit to turn off INVPCID in case something goes
wrong. It's an early_param because we do TLB flushes before we
parse __setup parameters.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
---
Documentation/kernel-parameters.txt | 2 ++
arch/x86/kernel/cpu/common.c | 16 ++++++++++++++++
2 files changed, 18 insertions(+)
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 742f69d18fc8..b34e55e00bae 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -2508,6 +2508,8 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
nointroute [IA-64]
+ noinvpcid [X86] Disable the INVPCID cpu feature.
+
nojitter [IA-64] Disables jitter checking for ITC timers.
no-kvmclock [X86,KVM] Disable paravirtualized KVM clock driver
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index c2b7522cbf35..48196980c1c7 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -162,6 +162,22 @@ static int __init x86_mpx_setup(char *s)
}
__setup("nompx", x86_mpx_setup);
+static int __init x86_noinvpcid_setup(char *s)
+{
+ /* noinvpcid doesn't accept parameters */
+ if (s)
+ return -EINVAL;
+
+ /* do not emit a message if the feature is not present */
+ if (!boot_cpu_has(X86_FEATURE_INVPCID))
+ return 0;
+
+ setup_clear_cpu_cap(X86_FEATURE_INVPCID);
+ pr_info("noinvpcid: INVPCID feature disabled\n");
+ return 0;
+}
+early_param("noinvpcid", x86_noinvpcid_setup);
+
#ifdef CONFIG_X86_32
static int cachesize_override = -1;
static int disable_x86_serial_nr = 1;
--
2.5.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH v3 3/3] x86/mm: If INVPCID is available, use it to flush global mappings
2016-01-29 19:42 [PATCH v3 0/3] x86/mm: INVPCID support Andy Lutomirski
2016-01-29 19:42 ` [PATCH v3 1/3] x86/mm: Add INVPCID helpers Andy Lutomirski
2016-01-29 19:42 ` [PATCH v3 2/3] x86/mm: Add a noinvpcid option to turn off INVPCID Andy Lutomirski
@ 2016-01-29 19:42 ` Andy Lutomirski
2016-02-01 10:51 ` [PATCH v3 0/3] x86/mm: INVPCID support Borislav Petkov
3 siblings, 0 replies; 5+ messages in thread
From: Andy Lutomirski @ 2016-01-29 19:42 UTC (permalink / raw)
To: x86, linux-kernel
Cc: Borislav Petkov, Brian Gerst, Dave Hansen, Linus Torvalds,
Oleg Nesterov, linux-mm, Andrey Ryabinin, Andy Lutomirski
On my Skylake laptop, INVPCID function 2 (flush absolutely
everything) takes about 376ns, whereas saving flags, twiddling
CR4.PGE to flush global mappings, and restoring flags takes about
539ns.
Signed-off-by: Andy Lutomirski <luto@kernel.org>
---
arch/x86/include/asm/tlbflush.h | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h
index 8b576832777e..985f1162c505 100644
--- a/arch/x86/include/asm/tlbflush.h
+++ b/arch/x86/include/asm/tlbflush.h
@@ -152,6 +152,15 @@ static inline void __native_flush_tlb_global(void)
{
unsigned long flags;
+ if (static_cpu_has_safe(X86_FEATURE_INVPCID)) {
+ /*
+ * Using INVPCID is considerably faster than a pair of writes
+ * to CR4 sandwiched inside an IRQ flag save/restore.
+ */
+ invpcid_flush_all();
+ return;
+ }
+
/*
* Read-modify-write to CR4 - protect it from preemption and
* from interrupts. (Use the raw variant because this code can
--
2.5.0
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH v3 0/3] x86/mm: INVPCID support
2016-01-29 19:42 [PATCH v3 0/3] x86/mm: INVPCID support Andy Lutomirski
` (2 preceding siblings ...)
2016-01-29 19:42 ` [PATCH v3 3/3] x86/mm: If INVPCID is available, use it to flush global mappings Andy Lutomirski
@ 2016-02-01 10:51 ` Borislav Petkov
3 siblings, 0 replies; 5+ messages in thread
From: Borislav Petkov @ 2016-02-01 10:51 UTC (permalink / raw)
To: Andy Lutomirski, Ingo Molnar
Cc: x86, linux-kernel, Brian Gerst, Dave Hansen, Linus Torvalds,
Oleg Nesterov, linux-mm, Andrey Ryabinin
On Fri, Jan 29, 2016 at 11:42:56AM -0800, Andy Lutomirski wrote:
> Boris, I think you already have these prerequisites queued up:
>
> http://lkml.kernel.org/g/1452516679-32040-2-git-send-email-aryabinin@virtuozzo.com
> http://lkml.kernel.org/g/1452516679-32040-3-git-send-email-aryabinin@virtuozzo.com
>
> This is a straightforward speedup on Ivy Bridge and newer, IIRC.
> (I tested on Skylake. INVPCID is not available on Sandy Bridge.
> I don't have Ivy Bridge, Haswell or Broadwell to test on, so I
> could be wrong as to when the feature was introduced.)
>
> I think we should consider these patches separately from the rest
> of the PCID stuff -- they barely interact, and this part is much
> simpler and is useful on its own.
>
> Changes from v2:
> - Add macros for the INVPCID mode numbers.
> - Add a changelog message for the chicken bit.
>
> v1 was exactly identical to patches 2-4 of the PCID RFC series.
> Andy Lutomirski (3):
> x86/mm: Add INVPCID helpers
> x86/mm: Add a noinvpcid option to turn off INVPCID
> x86/mm: If INVPCID is available, use it to flush global mappings
>
> Documentation/kernel-parameters.txt | 2 ++
> arch/x86/include/asm/tlbflush.h | 57 +++++++++++++++++++++++++++++++++++++
> arch/x86/kernel/cpu/common.c | 16 +++++++++++
> 3 files changed, 75 insertions(+)
All 5 (3 INVPCID + 2 KASAN ones at the URLs above):
Reviewed-by: Borislav Petkov <bp@suse.de>
--
Regards/Gruss,
Boris.
ECO tip #101: Trim your mails when you reply.
^ permalink raw reply [flat|nested] 5+ messages in thread