* [RFC PATCH v3 0/7] Kernel Address Space Isolation
@ 2020-02-26 16:21 Alexandre Chartre
2020-02-26 16:21 ` [RFC PATCH v3 1/7] mm/x86: Introduce kernel Address Space Isolation (ASI) Alexandre Chartre
` (6 more replies)
0 siblings, 7 replies; 16+ messages in thread
From: Alexandre Chartre @ 2020-02-26 16:21 UTC (permalink / raw)
To: rkrcmar, tglx, mingo, bp, hpa, dave.hansen, luto, peterz, x86,
linux-mm, linux-kernel
Cc: pbonzini, konrad.wilk, jan.setjeeilers, liran.alon, junaids,
graf, rppt, kuzuno, mgross, alexandre.chartre
Hi,
This is version 3 of the "Kernel Address Space Isolation" (ASI) RFC
[1][2]. This version focus on providing the ASI infrastructure and
implementing PTI with ASI.
Compared to version 2, this RFC is much smaller with only 7 patches
(RFCv2 has 26 patches). This is because this RFC contains patches only
for the ASI infrastructure and for implementing PTI with ASI, but it
doesn't contain patches for using ASI with KVM and all related changes
regarding page-table creation and filling.
This patchset is based on the Linux 5.5 source code.
Background
==========
Kernel Address Space Isolation aims to use address spaces to isolate some
parts of the kernel (for example KVM) to prevent leaking sensitive data
between CPU hyper-threads under speculative execution attacks.
Over the past years, various speculative attacks (like L1TF or MDS) have
highlighted that data can leak between CPU threads through the CPU (micro)
architecture. In particular, a malicious virtual machine running on a CPU
thread can target data used by a sibling CPU thread from the same CPU core.
Thus, a malicious VM can potentially access data from another VM or from
the host system if they are running on sibling CPU threads.
Core Scheduling [3] can prevent a malicious VM from attacking another VM
by running the same VM on all CPU threads of a CPU core. However a
malicious VM can still target the host system when the sibling CPU thread
exits the VM and returns to the host.
Address Space Isolation can be applied to KVM to mitigate this VM-to-host
attack by removing secrets from the kernel address space used when running
KVM, thus preventing a malicious VM from collecting any sensitive data
from host.
Address Space Isolation can also be used to implement Page Table Isolation
(PTI [4]) which reduces kernel mappings present in user address spaces to
prevent the Meltdown attack.
Details
=======
ASI
---
An ASI is created by calling asi_create() with a specified ASI type. The
ASI type manages data common to all ASI of the same type. It is used, in
particular, to manage per-ASI type TLB/PCID information.
Then the ASI can be entered with asi_enter() and exited with asi_exit().
When an ASI is in used, any interrupt/exception/NMI will cause the ASI to
be interrupted (ASI_INTERRUPT) and the ASI will be resumed (ASI_RESUME)
when the interrupt/exception/NMI returns.
asi_enter()/asi_exit() and ASI_INTERRUPT/ASI_RESUME switch between the
ASI and the full kernel page-table by updating the CR3 register.
If a task using ASI is scheduled out then its ASI state is saved and it
will be restored when the task is scheduled back.
Page fault occurring while ASI is used will either cause the ASI to be
aborted (switch back to the full kernel pagetable) or to be preserved.
The behavior depends on the ASI type. For example, for PTI the ASI is
preserved and the kernel page fault handler handles the fault on behalf
of the ASI. But for KVM ASI, the ASI will be aborted and the fault will
be retried with the full kernel page-table.
PTI
---
PTI is now implemented with ASI (user ASI) if both CONFIG_ADDRESS_SPACE_ISOLATION
and CONFIG_PAGE_TABLE_ISOLATION are set. The behavior of PTI is unchanged
but it is now using the ASI infrastructure.
For each user process, a user ASI is defined with the PTI pagetable. The
user ASI is used when running userland code, and it is exited when entering
a syscall. The user ASI is re-entered when the syscall returns to userland.
KVM
---
As already mentioned, KVM ASI is not present in this patchset. KVM ASI
will be implemented ontop of this infrastructure. Basically, the KVM ASI
patchset will:
- define a KVM ASI type (DEFINE_ASI_TYPE)
- create and fill a page-table to be used by the KVM ASI
- create a KVM ASI (asi_create_kvm())
- enter the KVM ASI (asi_enter()) on KVM_RUN ioctl
- exit the KVM ASI (asi_exit())
Fault occuring when KVM ASI is in used will cause the ASI to be aborted,
and the code will continue running with the full kernel page-table,
until KVM ASI is explicitly reentered.
Status
======
The code looks stable and it supports running a full kernel build and
also ltp tests. Performance impact is expected to be limited as the new
code only adds a small number of assembly instructions on syscall and
interrupts. There's probably also room for reducing this number of
instructions.
Changes
=======
Summary of changes compared to RFCv2:
- Add ASI Type
- Add generic TLB flushing mechanism for ASI. This mechanism is similar
to the context tracking done when switching mm.
- When ASI is in used, it is interrupted on interrupt/exception/NMI and
resumed when the interrupt/exception/NMI returns.
- If a task using ASI is scheduled in/out then save/restore the corresponding
ASI and update the cpu ASI session.
- Implement PTI with ASI.
- Remove KVM ASI from the patchset. KVM ASI will be provided in a separated
patchset ontop of the ASI infrastructure.
- Remove functions to manage, populate and clear page-tables. These functions
were only used to build to the KVM ASI page-table. Also such functions should
be generic page-table functions and not specific to ASI. Mike Rapoport is also
looking at making these functions generic.
References
==========
[1] ASI RFCv1 - https://lkml.org/lkml/2019/5/13/515
[2] ASI RFCv2 - https://lore.kernel.org/lkml/1562855138-19507-1-git-send-email-alexandre.chartre@oracle.com
[3] Core Scheduling - https://lwn.net/Articles/803652
[4] Page Table Isolation (PTI) - https://www.kernel.org/doc/html/latest/x86/pti.html
---
Alexandre Chartre (7):
mm/x86: Introduce kernel Address Space Isolation (ASI)
mm/asi: ASI entry/exit interface
mm/asi: Improve TLB flushing when switching to an ASI pagetable
mm/asi: Interrupt ASI on interrupt/exception/NMI
mm/asi: Switch ASI on task switch
mm/asi: ASI fault handler
mm/asi: Implement PTI with ASI
arch/x86/entry/calling.h | 31 +++-
arch/x86/entry/common.c | 29 +++-
arch/x86/entry/entry_64.S | 28 +++
arch/x86/include/asm/asi.h | 289 ++++++++++++++++++++++++++
arch/x86/include/asm/asi_session.h | 24 +++
arch/x86/include/asm/mmu_context.h | 20 ++-
arch/x86/include/asm/tlbflush.h | 23 ++-
arch/x86/kernel/asm-offsets.c | 5 +
arch/x86/mm/Makefile | 1 +
arch/x86/mm/asi.c | 402 ++++++++++++++++++++++++++++++++++++
arch/x86/mm/fault.c | 20 ++
arch/x86/mm/pti.c | 28 ++-
include/linux/mm_types.h | 5 +
include/linux/sched.h | 9 +
kernel/fork.c | 17 ++
kernel/sched/core.c | 8 +
security/Kconfig | 10 +
17 files changed, 931 insertions(+), 18 deletions(-)
create mode 100644 arch/x86/include/asm/asi.h
create mode 100644 arch/x86/include/asm/asi_session.h
create mode 100644 arch/x86/mm/asi.c
^ permalink raw reply [flat|nested] 16+ messages in thread
* [RFC PATCH v3 1/7] mm/x86: Introduce kernel Address Space Isolation (ASI)
2020-02-26 16:21 [RFC PATCH v3 0/7] Kernel Address Space Isolation Alexandre Chartre
@ 2020-02-26 16:21 ` Alexandre Chartre
2020-02-26 16:21 ` [RFC PATCH v3 2/7] mm/asi: ASI entry/exit interface Alexandre Chartre
` (5 subsequent siblings)
6 siblings, 0 replies; 16+ messages in thread
From: Alexandre Chartre @ 2020-02-26 16:21 UTC (permalink / raw)
To: rkrcmar, tglx, mingo, bp, hpa, dave.hansen, luto, peterz, x86,
linux-mm, linux-kernel
Cc: pbonzini, konrad.wilk, jan.setjeeilers, liran.alon, junaids,
graf, rppt, kuzuno, mgross, alexandre.chartre
Introduce core functions and structures for implementing Address Space
Isolation (ASI). Kernel address space isolation provides the ability to
run some kernel code with a reduced kernel address space.
An address space isolation is defined with a struct asi structure and
associated with an ASI type and a pagetable.
Signed-off-by: Alexandre Chartre <alexandre.chartre@oracle.com>
---
arch/x86/include/asm/asi.h | 88 ++++++++++++++++++++++++++++++++++++++++++++
arch/x86/mm/Makefile | 1 +
arch/x86/mm/asi.c | 60 ++++++++++++++++++++++++++++++
security/Kconfig | 10 +++++
4 files changed, 159 insertions(+), 0 deletions(-)
create mode 100644 arch/x86/include/asm/asi.h
create mode 100644 arch/x86/mm/asi.c
diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h
new file mode 100644
index 0000000..844a81f
--- /dev/null
+++ b/arch/x86/include/asm/asi.h
@@ -0,0 +1,88 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef ARCH_X86_MM_ASI_H
+#define ARCH_X86_MM_ASI_H
+
+#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
+
+/*
+ * An Address Space Isolation (ASI) is defined with a struct asi and
+ * associated with an ASI type (struct asi_type). All ASIs of the same
+ * type reference the same ASI type.
+ *
+ * An ASI type has a unique PCID prefix (a value in the range [1, 255])
+ * which is used to define the PCID used for the ASI CR3 value. The
+ * first four bits of the ASI PCID come from the kernel PCID (a value
+ * between 1 and 6, see TLB_NR_DYN_ASIDS). The remaining 8 bits are
+ * filled with the ASI PCID prefix.
+ *
+ * ASI PCID = (ASI Type PCID Prefix << 4) | Kernel PCID
+ *
+ * The ASI PCID is used to optimize TLB flushing when switching between
+ * the kernel and ASI pagetables. The optimization is valid only when
+ * a task switches between ASI of different types. If a task switches
+ * between different ASIs with the same type then the ASI TLB the task
+ * is switching to will always be flushed.
+ */
+
+#define ASI_PCID_PREFIX_SHIFT 4
+#define ASI_PCID_PREFIX_MASK 0xff0
+#define ASI_KERNEL_PCID_MASK 0x00f
+
+/*
+ * We use bit 12 of a pagetable pointer (and so of the CR3 value) as
+ * a way to know if a pointer/CR3 is referencing a full kernel page
+ * table or an ASI page table.
+ *
+ * A full kernel pagetable is always located on the first half of an
+ * 8K buffer, while an ASI pagetable is always located on the second
+ * half of an 8K buffer.
+ */
+#define ASI_PGTABLE_BIT PAGE_SHIFT
+#define ASI_PGTABLE_MASK (1 << ASI_PGTABLE_BIT)
+
+#ifndef __ASSEMBLY__
+
+#include <linux/export.h>
+
+struct asi_type {
+ int pcid_prefix; /* PCID prefix */
+};
+
+/*
+ * Macro to define and declare an ASI type.
+ *
+ * Declaring an ASI type will also define an inline function
+ * (asi_create_<typename>()) to easily create an ASI of the
+ * specified type.
+ */
+#define DEFINE_ASI_TYPE(name, pcid_prefix) \
+ struct asi_type asi_type_ ## name = { \
+ pcid_prefix, \
+ }; \
+ EXPORT_SYMBOL(asi_type_ ## name)
+
+#define DECLARE_ASI_TYPE(name) \
+ extern struct asi_type asi_type_ ## name; \
+ DECLARE_ASI_CREATE(name)
+
+#define DECLARE_ASI_CREATE(name) \
+static inline struct asi *asi_create_ ## name(void) \
+{ \
+ return asi_create(&asi_type_ ## name); \
+}
+
+struct asi {
+ struct asi_type *type; /* ASI type */
+ pgd_t *pagetable; /* ASI pagetable */
+ unsigned long base_cr3; /* base ASI CR3 */
+};
+
+extern struct asi *asi_create(struct asi_type *type);
+extern void asi_destroy(struct asi *asi);
+extern void asi_set_pagetable(struct asi *asi, pgd_t *pagetable);
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* CONFIG_ADDRESS_SPACE_ISOLATION */
+
+#endif
diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile
index 3b89c20..99c56ed 100644
--- a/arch/x86/mm/Makefile
+++ b/arch/x86/mm/Makefile
@@ -49,6 +49,7 @@ obj-$(CONFIG_X86_INTEL_MPX) += mpx.o
obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o
obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o
obj-$(CONFIG_PAGE_TABLE_ISOLATION) += pti.o
+obj-$(CONFIG_ADDRESS_SPACE_ISOLATION) += asi.o
obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt.o
obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o
diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c
new file mode 100644
index 0000000..0a0ac9d
--- /dev/null
+++ b/arch/x86/mm/asi.c
@@ -0,0 +1,60 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2019, 2020, Oracle and/or its affiliates.
+ *
+ * Kernel Address Space Isolation (ASI)
+ */
+
+#include <linux/mm.h>
+#include <linux/slab.h>
+
+#include <asm/asi.h>
+#include <asm/bug.h>
+
+struct asi *asi_create(struct asi_type *type)
+{
+ struct asi *asi;
+
+ if (!type)
+ return NULL;
+
+ asi = kzalloc(sizeof(*asi), GFP_KERNEL);
+ if (!asi)
+ return NULL;
+
+ asi->type = type;
+
+ return asi;
+}
+EXPORT_SYMBOL(asi_create);
+
+void asi_destroy(struct asi *asi)
+{
+ kfree(asi);
+}
+EXPORT_SYMBOL(asi_destroy);
+
+void asi_set_pagetable(struct asi *asi, pgd_t *pagetable)
+{
+ /*
+ * Check that the specified pagetable is properly aligned to be
+ * used as an ASI pagetable. If not, the pagetable is ignored
+ * and entering/exiting ASI will do nothing.
+ */
+ if (!(((unsigned long)pagetable) & ASI_PGTABLE_MASK)) {
+ WARN(1, "ASI %p: invalid ASI pagetable", asi);
+ asi->pagetable = NULL;
+ return;
+ }
+ asi->pagetable = pagetable;
+
+ /*
+ * Initialize the invariant part of the ASI CR3 value. We will
+ * just have to complete the PCID with the kernel PCID before
+ * using it.
+ */
+ asi->base_cr3 = __sme_pa(asi->pagetable) |
+ (asi->type->pcid_prefix << ASI_PCID_PREFIX_SHIFT);
+
+}
+EXPORT_SYMBOL(asi_set_pagetable);
diff --git a/security/Kconfig b/security/Kconfig
index 2a1a2d3..fe0515a 100644
--- a/security/Kconfig
+++ b/security/Kconfig
@@ -65,6 +65,16 @@ config PAGE_TABLE_ISOLATION
See Documentation/x86/pti.rst for more details.
+config ADDRESS_SPACE_ISOLATION
+ bool "Allow code to run with a reduced kernel address space"
+ default y
+ depends on (X86_64 || X86_PAE) && !UML
+ help
+ This feature provides the ability to run some kernel code
+ with a reduced kernel address space. This can be used to
+ mitigate speculative execution attacks which are able to
+ leak data between sibling CPU hyper-threads.
+
config SECURITY_INFINIBAND
bool "Infiniband Security Hooks"
depends on SECURITY && INFINIBAND
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [RFC PATCH v3 2/7] mm/asi: ASI entry/exit interface
2020-02-26 16:21 [RFC PATCH v3 0/7] Kernel Address Space Isolation Alexandre Chartre
2020-02-26 16:21 ` [RFC PATCH v3 1/7] mm/x86: Introduce kernel Address Space Isolation (ASI) Alexandre Chartre
@ 2020-02-26 16:21 ` Alexandre Chartre
2020-02-26 20:38 ` kbuild test robot
2020-02-26 22:02 ` Ira Weiny
2020-02-26 16:21 ` [RFC PATCH v3 3/7] mm/asi: Improve TLB flushing when switching to an ASI pagetable Alexandre Chartre
` (4 subsequent siblings)
6 siblings, 2 replies; 16+ messages in thread
From: Alexandre Chartre @ 2020-02-26 16:21 UTC (permalink / raw)
To: rkrcmar, tglx, mingo, bp, hpa, dave.hansen, luto, peterz, x86,
linux-mm, linux-kernel
Cc: pbonzini, konrad.wilk, jan.setjeeilers, liran.alon, junaids,
graf, rppt, kuzuno, mgross, alexandre.chartre
Address Space Isolation (ASI) is entered by calling asi_enter() which
switches the kernel page-table to the ASI page-table. Isolation is then
exited by calling asi_exit() which switches the page-table back to the
original kernel page-table.
The ASI being used and its state is tracked in a per-cpu ASI session
structure (struct asi_session).
Signed-off-by: Alexandre Chartre <alexandre.chartre@oracle.com>
---
arch/x86/include/asm/asi.h | 4 ++
arch/x86/include/asm/asi_session.h | 17 +++++++
arch/x86/include/asm/mmu_context.h | 19 +++++++-
arch/x86/include/asm/tlbflush.h | 12 +++++
arch/x86/mm/asi.c | 90 ++++++++++++++++++++++++++++++++++++
5 files changed, 140 insertions(+), 2 deletions(-)
create mode 100644 arch/x86/include/asm/asi_session.h
diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h
index 844a81f..29b745a 100644
--- a/arch/x86/include/asm/asi.h
+++ b/arch/x86/include/asm/asi.h
@@ -44,6 +44,8 @@
#include <linux/export.h>
+#include <asm/asi_session.h>
+
struct asi_type {
int pcid_prefix; /* PCID prefix */
};
@@ -80,6 +82,8 @@ struct asi {
extern struct asi *asi_create(struct asi_type *type);
extern void asi_destroy(struct asi *asi);
extern void asi_set_pagetable(struct asi *asi, pgd_t *pagetable);
+extern int asi_enter(struct asi *asi);
+extern void asi_exit(struct asi *asi);
#endif /* __ASSEMBLY__ */
diff --git a/arch/x86/include/asm/asi_session.h b/arch/x86/include/asm/asi_session.h
new file mode 100644
index 0000000..9d39c93
--- /dev/null
+++ b/arch/x86/include/asm/asi_session.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef ARCH_X86_MM_ASI_SESSION_H
+#define ARCH_X86_MM_ASI_SESSION_H
+
+#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
+
+struct asi;
+
+struct asi_session {
+ struct asi *asi; /* ASI for this session */
+ unsigned long isolation_cr3; /* cr3 when ASI is active */
+ unsigned long original_cr3; /* cr3 before entering ASI */
+};
+
+#endif /* CONFIG_ADDRESS_SPACE_ISOLATION */
+
+#endif
diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h
index 5f33924..2d65443 100644
--- a/arch/x86/include/asm/mmu_context.h
+++ b/arch/x86/include/asm/mmu_context.h
@@ -14,6 +14,7 @@
#include <asm/paravirt.h>
#include <asm/mpx.h>
#include <asm/debugreg.h>
+#include <asm/asi.h>
extern atomic64_t last_mm_ctx_id;
@@ -349,8 +350,22 @@ static inline bool arch_vma_access_permitted(struct vm_area_struct *vma,
*/
static inline unsigned long __get_current_cr3_fast(void)
{
- unsigned long cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd,
- this_cpu_read(cpu_tlbstate.loaded_mm_asid));
+ unsigned long cr3;
+
+ /*
+ * If isolation is active then we need to return the CR3 for the
+ * currently active ASI. This value is stored in the isolation_cr3
+ * field of the ASI session.
+ */
+ if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION) &&
+ this_cpu_read(cpu_asi_session.asi)) {
+ cr3 = this_cpu_read(cpu_asi_session.isolation_cr3);
+ /* CR3 read never returns with the NOFLUSH bit */
+ cr3 &= ~X86_CR3_PCID_NOFLUSH;
+ } else {
+ cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd,
+ this_cpu_read(cpu_tlbstate.loaded_mm_asid));
+ }
/* For now, be very restrictive about when this can be called. */
VM_WARN_ON(in_nmi() || preemptible());
diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h
index 6f66d84..241058f 100644
--- a/arch/x86/include/asm/tlbflush.h
+++ b/arch/x86/include/asm/tlbflush.h
@@ -12,6 +12,7 @@
#include <asm/invpcid.h>
#include <asm/pti.h>
#include <asm/processor-flags.h>
+#include <asm/asi.h>
/*
* The x86 feature is called PCID (Process Context IDentifier). It is similar
@@ -239,9 +240,20 @@ struct tlb_state {
* context 0.
*/
struct tlb_context ctxs[TLB_NR_DYN_ASIDS];
+
+#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
+ /*
+ * The ASI session tracks the ASI being used and its state.
+ */
+ struct asi_session asi_session;
+#endif
};
DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_state, cpu_tlbstate);
+#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
+#define cpu_asi_session (cpu_tlbstate.asi_session)
+#endif
+
/*
* Blindly accessing user memory from NMI context can be dangerous
* if we're in the middle of switching the current user task or
diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c
index 0a0ac9d..9fbc921 100644
--- a/arch/x86/mm/asi.c
+++ b/arch/x86/mm/asi.c
@@ -10,6 +10,8 @@
#include <asm/asi.h>
#include <asm/bug.h>
+#include <asm/mmu_context.h>
+#include <asm/tlbflush.h>
struct asi *asi_create(struct asi_type *type)
{
@@ -58,3 +60,91 @@ void asi_set_pagetable(struct asi *asi, pgd_t *pagetable)
}
EXPORT_SYMBOL(asi_set_pagetable);
+
+static void asi_switch_to_asi_cr3(struct asi *asi)
+{
+ unsigned long original_cr3, asi_cr3;
+ struct asi_session *asi_session;
+ u16 pcid;
+
+ WARN_ON(!irqs_disabled());
+
+ original_cr3 = __get_current_cr3_fast();
+
+ /* build the ASI cr3 value */
+ asi_cr3 = asi->base_cr3;
+ if (boot_cpu_has(X86_FEATURE_PCID)) {
+ pcid = original_cr3 & ASI_KERNEL_PCID_MASK;
+ asi_cr3 |= pcid;
+ }
+
+ /* get the ASI session ready for entering ASI */
+ asi_session = &get_cpu_var(cpu_asi_session);
+ asi_session->asi = asi;
+ asi_session->original_cr3 = original_cr3;
+ asi_session->isolation_cr3 = asi_cr3;
+
+ /* Update CR3 to immediately enter ASI */
+ native_write_cr3(asi_cr3);
+}
+
+static void asi_switch_to_kernel_cr3(struct asi *asi)
+{
+ struct asi_session *asi_session;
+ unsigned long original_cr3;
+
+ WARN_ON(!irqs_disabled());
+
+ original_cr3 = this_cpu_read(cpu_asi_session.original_cr3);
+ if (boot_cpu_has(X86_FEATURE_PCID))
+ original_cr3 |= X86_CR3_PCID_NOFLUSH;
+ native_write_cr3(original_cr3);
+
+ asi_session = &get_cpu_var(cpu_asi_session);
+ asi_session->asi = NULL;
+}
+
+int asi_enter(struct asi *asi)
+{
+ struct asi *current_asi;
+ unsigned long flags;
+
+ /*
+ * We can re-enter isolation, but only with the same ASI (we don't
+ * support nesting isolation).
+ */
+ current_asi = this_cpu_read(cpu_asi_session.asi);
+ if (current_asi) {
+ if (current_asi != asi) {
+ WARN_ON(1);
+ return -EBUSY;
+ }
+ return 0;
+ }
+
+ local_irq_save(flags);
+ asi_switch_to_asi_cr3(asi);
+ local_irq_restore(flags);
+
+ return 0;
+}
+EXPORT_SYMBOL(asi_enter);
+
+void asi_exit(struct asi *asi)
+{
+ struct asi *current_asi;
+ unsigned long flags;
+
+ current_asi = this_cpu_read(cpu_asi_session.asi);
+ if (!current_asi) {
+ /* already exited */
+ return;
+ }
+
+ WARN_ON(current_asi != asi);
+
+ local_irq_save(flags);
+ asi_switch_to_kernel_cr3(asi);
+ local_irq_restore(flags);
+}
+EXPORT_SYMBOL(asi_exit);
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [RFC PATCH v3 3/7] mm/asi: Improve TLB flushing when switching to an ASI pagetable
2020-02-26 16:21 [RFC PATCH v3 0/7] Kernel Address Space Isolation Alexandre Chartre
2020-02-26 16:21 ` [RFC PATCH v3 1/7] mm/x86: Introduce kernel Address Space Isolation (ASI) Alexandre Chartre
2020-02-26 16:21 ` [RFC PATCH v3 2/7] mm/asi: ASI entry/exit interface Alexandre Chartre
@ 2020-02-26 16:21 ` Alexandre Chartre
2020-02-26 16:21 ` [RFC PATCH v3 4/7] mm/asi: Interrupt ASI on interrupt/exception/NMI Alexandre Chartre
` (3 subsequent siblings)
6 siblings, 0 replies; 16+ messages in thread
From: Alexandre Chartre @ 2020-02-26 16:21 UTC (permalink / raw)
To: rkrcmar, tglx, mingo, bp, hpa, dave.hansen, luto, peterz, x86,
linux-mm, linux-kernel
Cc: pbonzini, konrad.wilk, jan.setjeeilers, liran.alon, junaids,
graf, rppt, kuzuno, mgross, alexandre.chartre
When switching to an ASI pagetable, the TLB doesn't need to be flushed
if it was previously used with the same PCID. So, to avoid unnecessary
TLB flushing, we track which pagetables are used with the different
ASI PCIDs. If an ASI PCID is being used with a different ASI pagetable,
or if we have a new generation of the same ASI pagetable, then the TLB
needs to be flushed. This behavior is similar to the context tracking
done when switching mm.
Signed-off-by: Alexandre Chartre <alexandre.chartre@oracle.com>
---
arch/x86/include/asm/asi.h | 23 +++++++++++++++++++++++
arch/x86/mm/asi.c | 34 ++++++++++++++++++++++++++++++++--
2 files changed, 55 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h
index 29b745a..bcfb68e 100644
--- a/arch/x86/include/asm/asi.h
+++ b/arch/x86/include/asm/asi.h
@@ -46,8 +46,26 @@
#include <asm/asi_session.h>
+/*
+ * ASI_NR_DYN_ASIDS is the same as TLB_NR_DYN_ASIDS. We can't directly
+ * use TLB_NR_DYN_ASIDS because asi.h and tlbflush.h can't both include
+ * each other.
+ */
+#define ASI_TLB_NR_DYN_ASIDS 6
+
+struct asi_tlb_pgtable {
+ u64 id;
+ u64 gen;
+};
+
+struct asi_tlb_state {
+ struct asi_tlb_pgtable tlb_pgtables[ASI_TLB_NR_DYN_ASIDS];
+};
+
struct asi_type {
int pcid_prefix; /* PCID prefix */
+ struct asi_tlb_state *tlb_state; /* percpu ASI TLB state */
+ atomic64_t last_pgtable_id; /* last id for this type */
};
/*
@@ -58,8 +76,11 @@ struct asi_type {
* specified type.
*/
#define DEFINE_ASI_TYPE(name, pcid_prefix) \
+ DEFINE_PER_CPU(struct asi_tlb_state, asi_tlb_ ## name); \
struct asi_type asi_type_ ## name = { \
pcid_prefix, \
+ &asi_tlb_ ## name, \
+ ATOMIC64_INIT(1), \
}; \
EXPORT_SYMBOL(asi_type_ ## name)
@@ -76,6 +97,8 @@ struct asi_type {
struct asi {
struct asi_type *type; /* ASI type */
pgd_t *pagetable; /* ASI pagetable */
+ u64 pgtable_id; /* ASI pagetable ID */
+ atomic64_t pgtable_gen; /* ASI pagetable generation */
unsigned long base_cr3; /* base ASI CR3 */
};
diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c
index 9fbc921..cf0d122 100644
--- a/arch/x86/mm/asi.c
+++ b/arch/x86/mm/asi.c
@@ -25,6 +25,8 @@ struct asi *asi_create(struct asi_type *type)
return NULL;
asi->type = type;
+ asi->pgtable_id = atomic64_inc_return(&type->last_pgtable_id);
+ atomic64_set(&asi->pgtable_gen, 0);
return asi;
}
@@ -61,6 +63,33 @@ void asi_set_pagetable(struct asi *asi, pgd_t *pagetable)
}
EXPORT_SYMBOL(asi_set_pagetable);
+/*
+ * Update ASI TLB flush information for the specified ASI CR3 value.
+ * Return an updated ASI CR3 value which specified if TLB needs to
+ * be flushed or not.
+ */
+static unsigned long asi_update_flush(struct asi *asi, unsigned long asi_cr3)
+{
+ struct asi_tlb_pgtable *tlb_pgtable;
+ struct asi_tlb_state *tlb_state;
+ s64 pgtable_gen;
+ u16 pcid;
+
+ pcid = asi_cr3 & ASI_KERNEL_PCID_MASK;
+ tlb_state = get_cpu_ptr(asi->type->tlb_state);
+ tlb_pgtable = &tlb_state->tlb_pgtables[pcid - 1];
+ pgtable_gen = atomic64_read(&asi->pgtable_gen);
+ if (tlb_pgtable->id == asi->pgtable_id &&
+ tlb_pgtable->gen == pgtable_gen) {
+ asi_cr3 |= X86_CR3_PCID_NOFLUSH;
+ } else {
+ tlb_pgtable->id = asi->pgtable_id;
+ tlb_pgtable->gen = pgtable_gen;
+ }
+
+ return asi_cr3;
+}
+
static void asi_switch_to_asi_cr3(struct asi *asi)
{
unsigned long original_cr3, asi_cr3;
@@ -72,10 +101,11 @@ static void asi_switch_to_asi_cr3(struct asi *asi)
original_cr3 = __get_current_cr3_fast();
/* build the ASI cr3 value */
- asi_cr3 = asi->base_cr3;
if (boot_cpu_has(X86_FEATURE_PCID)) {
pcid = original_cr3 & ASI_KERNEL_PCID_MASK;
- asi_cr3 |= pcid;
+ asi_cr3 = asi_update_flush(asi, asi->base_cr3 | pcid);
+ } else {
+ asi_cr3 = asi->base_cr3;
}
/* get the ASI session ready for entering ASI */
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [RFC PATCH v3 4/7] mm/asi: Interrupt ASI on interrupt/exception/NMI
2020-02-26 16:21 [RFC PATCH v3 0/7] Kernel Address Space Isolation Alexandre Chartre
` (2 preceding siblings ...)
2020-02-26 16:21 ` [RFC PATCH v3 3/7] mm/asi: Improve TLB flushing when switching to an ASI pagetable Alexandre Chartre
@ 2020-02-26 16:21 ` Alexandre Chartre
2020-02-26 20:38 ` kbuild test robot
2020-02-26 16:21 ` [RFC PATCH v3 5/7] mm/asi: Switch ASI on task switch Alexandre Chartre
` (2 subsequent siblings)
6 siblings, 1 reply; 16+ messages in thread
From: Alexandre Chartre @ 2020-02-26 16:21 UTC (permalink / raw)
To: rkrcmar, tglx, mingo, bp, hpa, dave.hansen, luto, peterz, x86,
linux-mm, linux-kernel
Cc: pbonzini, konrad.wilk, jan.setjeeilers, liran.alon, junaids,
graf, rppt, kuzuno, mgross, alexandre.chartre
If an interrupt/exception/NMI is triggered while using ASI then
ASI is interrupted and the system switches back to the (kernel)
page-table used before entering ASI.
When the interrupt/exception/NMI handler returns then ASI is
resumed by switching back to the ASI page-table.
Signed-off-by: Alexandre Chartre <alexandre.chartre@oracle.com>
---
arch/x86/entry/calling.h | 26 +++++++-
arch/x86/entry/entry_64.S | 22 +++++++
arch/x86/include/asm/asi.h | 122 ++++++++++++++++++++++++++++++++++++
arch/x86/include/asm/asi_session.h | 7 ++
arch/x86/include/asm/mmu_context.h | 3 +-
arch/x86/kernel/asm-offsets.c | 5 ++
arch/x86/mm/asi.c | 67 +++++++++++++++++---
7 files changed, 242 insertions(+), 10 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index 0789e13..ca23b79 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -6,6 +6,7 @@
#include <asm/percpu.h>
#include <asm/asm-offsets.h>
#include <asm/processor-flags.h>
+#include <asm/asi.h>
/*
@@ -172,7 +173,30 @@
.endif
.endm
-#ifdef CONFIG_PAGE_TABLE_ISOLATION
+#if defined(CONFIG_ADDRESS_SPACE_ISOLATION)
+
+/*
+ * For now, ASI is not compatible with PTI.
+ */
+
+.macro SWITCH_TO_KERNEL_CR3 scratch_reg:req
+.endm
+
+.macro SWITCH_TO_USER_CR3_NOSTACK scratch_reg:req scratch_reg2:req
+.endm
+
+.macro SWITCH_TO_USER_CR3_STACK scratch_reg:req
+.endm
+
+.macro SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg:req save_reg:req
+ ASI_INTERRUPT_AND_SAVE_CR3 \scratch_reg \save_reg
+.endm
+
+.macro RESTORE_CR3 scratch_reg:req save_reg:req
+ ASI_RESUME_AND_RESTORE_CR3 \save_reg
+.endm
+
+#elif defined(CONFIG_PAGE_TABLE_ISOLATION)
/*
* PAGE_TABLE_ISOLATION PGDs are 8k. Flip bit 12 to switch between the two
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 76942cb..fddb820 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -573,7 +573,15 @@ SYM_CODE_START(interrupt_entry)
CALL_enter_from_user_mode
+#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
+ jmp 2f
+#endif
1:
+#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
+ /* Interrupt address space isolation if it is active */
+ ASI_INTERRUPT scratch_reg=%rdi
+2:
+#endif
ENTER_IRQ_STACK old_rsp=%rdi save_ret=1
/* We entered an interrupt context - irqs are off: */
TRACE_IRQS_OFF
@@ -674,6 +682,10 @@ retint_kernel:
call preempt_schedule_irq
1:
#endif
+#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
+ ASI_PREPARE_RESUME
+ ASI_RESUME scratch_reg=%rdi
+#endif
/*
* The iretq could re-enable interrupts:
*/
@@ -1238,6 +1250,9 @@ SYM_CODE_START_LOCAL(paranoid_entry)
* This is also why CS (stashed in the "iret frame" by the
* hardware at entry) can not be used: this may be a return
* to kernel code, but with a user CR3 value.
+ *
+ * If ASI is enabled, this also handles the case where we are
+ * using an ASI CR3 value.
*/
SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg=%rax save_reg=%r14
@@ -1313,6 +1328,13 @@ SYM_CODE_START_LOCAL(error_entry)
.Lerror_entry_done_lfence:
FENCE_SWAPGS_KERNEL_ENTRY
+#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
+ /*
+ * Interrupt address space isolation if it is active. This will restore
+ * the original kernel CR3.
+ */
+ ASI_INTERRUPT scratch_reg=%rdi
+#endif
.Lerror_entry_done:
ret
diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h
index bcfb68e..d240954 100644
--- a/arch/x86/include/asm/asi.h
+++ b/arch/x86/include/asm/asi.h
@@ -108,6 +108,128 @@ struct asi {
extern int asi_enter(struct asi *asi);
extern void asi_exit(struct asi *asi);
+#else /* __ASSEMBLY__ */
+
+#include <asm/alternative-asm.h>
+#include <asm/asm-offsets.h>
+#include <asm/cpufeatures.h>
+#include <asm/percpu.h>
+#include <asm/processor-flags.h>
+
+#define THIS_ASI_SESSION_asi \
+ PER_CPU_VAR(cpu_tlbstate + TLB_STATE_asi)
+#define THIS_ASI_SESSION_isolation_cr3 \
+ PER_CPU_VAR(cpu_tlbstate + TLB_STATE_asi_isolation_cr3)
+#define THIS_ASI_SESSION_original_cr3 \
+ PER_CPU_VAR(cpu_tlbstate + TLB_STATE_asi_original_cr3)
+#define THIS_ASI_SESSION_idepth \
+ PER_CPU_VAR(cpu_tlbstate + TLB_STATE_asi_idepth)
+
+.macro SET_NOFLUSH_BIT reg:req
+ bts $X86_CR3_PCID_NOFLUSH_BIT, \reg
+.endm
+
+/*
+ * Switch CR3 to the original kernel CR3 value. This is used when exiting
+ * interrupting ASI.
+ */
+.macro ASI_SWITCH_TO_KERNEL_CR3 scratch_reg:req
+ /*
+ * KERNEL pages can always resume with NOFLUSH as we do
+ * explicit flushes.
+ */
+ movq THIS_ASI_SESSION_original_cr3, \scratch_reg
+ ALTERNATIVE "", "SET_NOFLUSH_BIT \scratch_reg", X86_FEATURE_PCID
+ movq \scratch_reg, %cr3
+.endm
+
+/*
+ * Interrupt ASI, when there's an interrupt or exception while we
+ * were running with ASI.
+ */
+.macro ASI_INTERRUPT scratch_reg:req
+ movq THIS_ASI_SESSION_asi, \scratch_reg
+ testq \scratch_reg, \scratch_reg
+ jz .Lasi_interrupt_done_\@
+ incl THIS_ASI_SESSION_idepth
+ cmp $1, THIS_ASI_SESSION_idepth
+ jne .Lasi_interrupt_done_\@
+ ASI_SWITCH_TO_KERNEL_CR3 \scratch_reg
+.Lasi_interrupt_done_\@:
+.endm
+
+.macro ASI_PREPARE_RESUME
+ call asi_prepare_resume
+.endm
+
+/*
+ * Resume ASI, after it was interrupted by an interrupt or an exception.
+ */
+.macro ASI_RESUME scratch_reg:req
+ movq THIS_ASI_SESSION_asi, \scratch_reg
+ testq \scratch_reg, \scratch_reg
+ jz .Lasi_resume_done_\@
+ decl THIS_ASI_SESSION_idepth
+ jnz .Lasi_resume_done_\@
+ movq THIS_ASI_SESSION_isolation_cr3, \scratch_reg
+ mov \scratch_reg, %cr3
+.Lasi_resume_done_\@:
+.endm
+
+/*
+ * Interrupt ASI, special processing when ASI is interrupted by a NMI
+ * or a paranoid interrupt/exception.
+ */
+.macro ASI_INTERRUPT_AND_SAVE_CR3 scratch_reg:req save_reg:req
+ movq %cr3, \save_reg
+ /*
+ * Test the ASI PCID bits. If set, then an ASI page table
+ * is active. If clear, CR3 already has the kernel page table
+ * active.
+ */
+ bt $ASI_PGTABLE_BIT, \save_reg
+ jnc .Ldone_\@
+ incl THIS_ASI_SESSION_idepth
+ ASI_SWITCH_TO_KERNEL_CR3 \scratch_reg
+.Ldone_\@:
+.endm
+
+/*
+ * Resume ASI, special processing when ASI is resumed from a NMI
+ * or a paranoid interrupt/exception.
+ */
+.macro ASI_RESUME_AND_RESTORE_CR3 save_reg:req
+
+ ALTERNATIVE "jmp .Lwrite_cr3_\@", "", X86_FEATURE_PCID
+
+ bt $ASI_PGTABLE_BIT, \save_reg
+ jnc .Lrestore_kernel_cr3_\@
+
+ /*
+ * Restore ASI CR3. We need to update TLB flushing
+ * information.
+ */
+ movq THIS_ASI_SESSION_asi, %rdi
+ movq \save_reg, %rsi
+ call asi_update_flush
+ movq %rax, THIS_ASI_SESSION_isolation_cr3
+ decl THIS_ASI_SESSION_idepth
+ movq %rax, %cr3
+ jmp .Ldone_\@
+
+.Lrestore_kernel_cr3_\@:
+ /*
+ * Restore kernel CR3. KERNEL pages can always resume
+ * with NOFLUSH as we do explicit flushes.
+ */
+ SET_NOFLUSH_BIT \save_reg
+
+.Lwrite_cr3_\@:
+ movq \save_reg, %cr3
+
+.Ldone_\@:
+.endm
+
#endif /* __ASSEMBLY__ */
#endif /* CONFIG_ADDRESS_SPACE_ISOLATION */
diff --git a/arch/x86/include/asm/asi_session.h b/arch/x86/include/asm/asi_session.h
index 9d39c93..85968f7 100644
--- a/arch/x86/include/asm/asi_session.h
+++ b/arch/x86/include/asm/asi_session.h
@@ -10,6 +10,13 @@ struct asi_session {
struct asi *asi; /* ASI for this session */
unsigned long isolation_cr3; /* cr3 when ASI is active */
unsigned long original_cr3; /* cr3 before entering ASI */
+ /*
+ * The interrupt depth (idepth) tracks interrupt (actually
+ * interrupt/exception/NMI) nesting. ASI is interrupted on
+ * the first interrupt, and it is resumed when that interrupt
+ * handler returns.
+ */
+ unsigned int idepth; /* interrupt depth */
};
#endif /* CONFIG_ADDRESS_SPACE_ISOLATION */
diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h
index 2d65443..b29e866 100644
--- a/arch/x86/include/asm/mmu_context.h
+++ b/arch/x86/include/asm/mmu_context.h
@@ -358,7 +358,8 @@ static inline unsigned long __get_current_cr3_fast(void)
* field of the ASI session.
*/
if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION) &&
- this_cpu_read(cpu_asi_session.asi)) {
+ this_cpu_read(cpu_asi_session.asi) &&
+ !this_cpu_read(cpu_asi_session.idepth)) {
cr3 = this_cpu_read(cpu_asi_session.isolation_cr3);
/* CR3 read never returns with the NOFLUSH bit */
cr3 &= ~X86_CR3_PCID_NOFLUSH;
diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c
index 5c7ee3d..a4316aa 100644
--- a/arch/x86/kernel/asm-offsets.c
+++ b/arch/x86/kernel/asm-offsets.c
@@ -95,6 +95,11 @@ static void __used common(void)
/* TLB state for the entry code */
OFFSET(TLB_STATE_user_pcid_flush_mask, tlb_state, user_pcid_flush_mask);
+ OFFSET(TLB_STATE_asi, tlb_state, asi_session.asi);
+ OFFSET(TLB_STATE_asi_isolation_cr3, tlb_state,
+ asi_session.isolation_cr3);
+ OFFSET(TLB_STATE_asi_original_cr3, tlb_state, asi_session.original_cr3);
+ OFFSET(TLB_STATE_asi_idepth, tlb_state, asi_session.idepth);
/* Layout info for cpu_entry_area */
OFFSET(CPU_ENTRY_AREA_entry_stack, cpu_entry_area, entry_stack_page);
diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c
index cf0d122..c91ba82 100644
--- a/arch/x86/mm/asi.c
+++ b/arch/x86/mm/asi.c
@@ -68,7 +68,7 @@ void asi_set_pagetable(struct asi *asi, pgd_t *pagetable)
* Return an updated ASI CR3 value which specified if TLB needs to
* be flushed or not.
*/
-static unsigned long asi_update_flush(struct asi *asi, unsigned long asi_cr3)
+unsigned long asi_update_flush(struct asi *asi, unsigned long asi_cr3)
{
struct asi_tlb_pgtable *tlb_pgtable;
struct asi_tlb_state *tlb_state;
@@ -90,7 +90,24 @@ static unsigned long asi_update_flush(struct asi *asi, unsigned long asi_cr3)
return asi_cr3;
}
-static void asi_switch_to_asi_cr3(struct asi *asi)
+
+/*
+ * Switch to the ASI pagetable.
+ *
+ * If schedule is ASI_SWITCH_NOW, then immediately switch to the ASI
+ * pagetable by updating the CR3 register with the ASI CR3 value.
+ * Otherwise, if schedule is ASI_SWITCH_ON_RESUME, prepare everything
+ * for switching to ASI pagetable but do not update the CR3 register
+ * yet. This will be done by the next ASI_RESUME call.
+ */
+
+enum asi_switch_schedule {
+ ASI_SWITCH_NOW,
+ ASI_SWITCH_ON_RESUME,
+};
+
+static void asi_switch_to_asi_cr3(struct asi *asi,
+ enum asi_switch_schedule schedule)
{
unsigned long original_cr3, asi_cr3;
struct asi_session *asi_session;
@@ -114,8 +131,16 @@ static void asi_switch_to_asi_cr3(struct asi *asi)
asi_session->original_cr3 = original_cr3;
asi_session->isolation_cr3 = asi_cr3;
- /* Update CR3 to immediately enter ASI */
- native_write_cr3(asi_cr3);
+ if (schedule == ASI_SWITCH_ON_RESUME) {
+ /*
+ * Defer the CR3 update the next ASI resume by setting
+ * the interrupt depth to 1.
+ */
+ asi_session->idepth = 1;
+ } else {
+ /* Update CR3 to immediately enter ASI */
+ native_write_cr3(asi_cr3);
+ }
}
static void asi_switch_to_kernel_cr3(struct asi *asi)
@@ -132,6 +157,7 @@ static void asi_switch_to_kernel_cr3(struct asi *asi)
asi_session = &get_cpu_var(cpu_asi_session);
asi_session->asi = NULL;
+ asi_session->idepth = 0;
}
int asi_enter(struct asi *asi)
@@ -153,7 +179,7 @@ int asi_enter(struct asi *asi)
}
local_irq_save(flags);
- asi_switch_to_asi_cr3(asi);
+ asi_switch_to_asi_cr3(asi, ASI_SWITCH_NOW);
local_irq_restore(flags);
return 0;
@@ -162,8 +188,10 @@ int asi_enter(struct asi *asi)
void asi_exit(struct asi *asi)
{
+ struct asi_session *asi_session;
struct asi *current_asi;
unsigned long flags;
+ int idepth;
current_asi = this_cpu_read(cpu_asi_session.asi);
if (!current_asi) {
@@ -173,8 +201,31 @@ void asi_exit(struct asi *asi)
WARN_ON(current_asi != asi);
- local_irq_save(flags);
- asi_switch_to_kernel_cr3(asi);
- local_irq_restore(flags);
+ idepth = this_cpu_read(cpu_asi_session.idepth);
+ if (!idepth) {
+ local_irq_save(flags);
+ asi_switch_to_kernel_cr3(asi);
+ local_irq_restore(flags);
+ } else {
+ /*
+ * ASI was interrupted so we already switched back
+ * to the back to the kernel page table and we just
+ * need to clear the ASI session.
+ */
+ asi_session = &get_cpu_var(cpu_asi_session);
+ asi_session->asi = NULL;
+ asi_session->idepth = 0;
+ }
}
EXPORT_SYMBOL(asi_exit);
+
+void asi_prepare_resume(void)
+{
+ struct asi_session *asi_session;
+
+ asi_session = &get_cpu_var(cpu_asi_session);
+ if (!asi_session->asi || asi_session->idepth > 1)
+ return;
+
+ asi_switch_to_asi_cr3(asi_session->asi, ASI_SWITCH_ON_RESUME);
+}
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [RFC PATCH v3 5/7] mm/asi: Switch ASI on task switch
2020-02-26 16:21 [RFC PATCH v3 0/7] Kernel Address Space Isolation Alexandre Chartre
` (3 preceding siblings ...)
2020-02-26 16:21 ` [RFC PATCH v3 4/7] mm/asi: Interrupt ASI on interrupt/exception/NMI Alexandre Chartre
@ 2020-02-26 16:21 ` Alexandre Chartre
2020-02-26 20:50 ` kbuild test robot
` (2 more replies)
2020-02-26 16:21 ` [RFC PATCH v3 6/7] mm/asi: ASI fault handler Alexandre Chartre
2020-02-26 16:22 ` [RFC PATCH v3 7/7] mm/asi: Implement PTI with ASI Alexandre Chartre
6 siblings, 3 replies; 16+ messages in thread
From: Alexandre Chartre @ 2020-02-26 16:21 UTC (permalink / raw)
To: rkrcmar, tglx, mingo, bp, hpa, dave.hansen, luto, peterz, x86,
linux-mm, linux-kernel
Cc: pbonzini, konrad.wilk, jan.setjeeilers, liran.alon, junaids,
graf, rppt, kuzuno, mgross, alexandre.chartre
If a task using ASI is scheduled in/out then save/restore the
corresponding ASI and update the cpu ASI session.
Signed-off-by: Alexandre Chartre <alexandre.chartre@oracle.com>
---
arch/x86/include/asm/asi.h | 3 ++
arch/x86/mm/asi.c | 67 ++++++++++++++++++++++++++++++++++++++++++++
include/linux/sched.h | 9 ++++++
kernel/sched/core.c | 8 +++++
4 files changed, 87 insertions(+), 0 deletions(-)
diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h
index d240954..a0733f1 100644
--- a/arch/x86/include/asm/asi.h
+++ b/arch/x86/include/asm/asi.h
@@ -102,6 +102,9 @@ struct asi {
unsigned long base_cr3; /* base ASI CR3 */
};
+void asi_schedule_out(struct task_struct *task);
+void asi_schedule_in(struct task_struct *task);
+
extern struct asi *asi_create(struct asi_type *type);
extern void asi_destroy(struct asi *asi);
extern void asi_set_pagetable(struct asi *asi, pgd_t *pagetable);
diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c
index c91ba82..9955eb2 100644
--- a/arch/x86/mm/asi.c
+++ b/arch/x86/mm/asi.c
@@ -229,3 +229,70 @@ void asi_prepare_resume(void)
asi_switch_to_asi_cr3(asi_session->asi, ASI_SWITCH_ON_RESUME);
}
+
+void asi_schedule_out(struct task_struct *task)
+{
+ struct asi_session *asi_session;
+ unsigned long flags;
+ struct asi *asi;
+
+ asi = this_cpu_read(cpu_asi_session.asi);
+ if (!asi)
+ return;
+
+ /*
+ * Save the ASI session.
+ *
+ * Exit the session if it hasn't been interrupted, otherwise
+ * just save the session state.
+ */
+ local_irq_save(flags);
+ if (!this_cpu_read(cpu_asi_session.idepth)) {
+ asi_switch_to_kernel_cr3(asi);
+ task->asi_session.asi = asi;
+ task->asi_session.idepth = 0;
+ } else {
+ asi_session = &get_cpu_var(cpu_asi_session);
+ task->asi_session = *asi_session;
+ asi_session->asi = NULL;
+ asi_session->idepth = 0;
+ }
+ local_irq_restore(flags);
+}
+
+void asi_schedule_in(struct task_struct *task)
+{
+ struct asi_session *asi_session;
+ unsigned long flags;
+ struct asi *asi;
+
+ asi = task->asi_session.asi;
+ if (!asi)
+ return;
+
+ /*
+ * At this point, the CPU shouldn't be using ASI because the
+ * ASI session is expected to be cleared in asi_schedule_out().
+ */
+ WARN_ON(this_cpu_read(cpu_asi_session.asi));
+
+ /*
+ * Restore ASI.
+ *
+ * If the task was scheduled out while using ASI, then the ASI
+ * is already setup and we can immediately switch to ASI page
+ * table.
+ *
+ * Otherwise, if the task was scheduled out while ASI was
+ * interrupted, just restore the ASI session.
+ */
+ local_irq_save(flags);
+ if (!task->asi_session.idepth) {
+ asi_switch_to_asi_cr3(asi, ASI_SWITCH_NOW);
+ } else {
+ asi_session = &get_cpu_var(cpu_asi_session);
+ *asi_session = task->asi_session;
+ task->asi_session.asi = NULL;
+ }
+ local_irq_restore(flags);
+}
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 716ad1d..66cc583 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -10,6 +10,7 @@
#include <uapi/linux/sched.h>
#include <asm/current.h>
+#include <asm/asi_session.h>
#include <linux/pid.h>
#include <linux/sem.h>
@@ -1281,6 +1282,14 @@ struct task_struct {
unsigned long prev_lowest_stack;
#endif
+#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
+ /*
+ * ASI session is saved here when the task is scheduled out
+ * while an ASI session was active or interrupted.
+ */
+ struct asi_session asi_session;
+#endif
+
/*
* New fields for task_struct should be added above here, so that
* they are included in the randomized portion of task_struct.
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 90e4b00..a2c8604 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -14,6 +14,7 @@
#include <asm/switch_to.h>
#include <asm/tlb.h>
+#include <asm/asi.h>
#include "../workqueue_internal.h"
#include "../../fs/io-wq.h"
@@ -3153,6 +3154,9 @@ static inline void finish_lock_switch(struct rq *rq)
prepare_task_switch(struct rq *rq, struct task_struct *prev,
struct task_struct *next)
{
+ if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION))
+ asi_schedule_out(prev);
+
kcov_prepare_switch(prev);
sched_info_switch(rq, prev, next);
perf_event_task_sched_out(prev, next);
@@ -3259,6 +3263,10 @@ static inline void finish_lock_switch(struct rq *rq)
}
tick_nohz_task_switch();
+
+ if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION))
+ asi_schedule_in(current);
+
return rq;
}
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [RFC PATCH v3 6/7] mm/asi: ASI fault handler
2020-02-26 16:21 [RFC PATCH v3 0/7] Kernel Address Space Isolation Alexandre Chartre
` (4 preceding siblings ...)
2020-02-26 16:21 ` [RFC PATCH v3 5/7] mm/asi: Switch ASI on task switch Alexandre Chartre
@ 2020-02-26 16:21 ` Alexandre Chartre
2020-02-26 16:22 ` [RFC PATCH v3 7/7] mm/asi: Implement PTI with ASI Alexandre Chartre
6 siblings, 0 replies; 16+ messages in thread
From: Alexandre Chartre @ 2020-02-26 16:21 UTC (permalink / raw)
To: rkrcmar, tglx, mingo, bp, hpa, dave.hansen, luto, peterz, x86,
linux-mm, linux-kernel
Cc: pbonzini, konrad.wilk, jan.setjeeilers, liran.alon, junaids,
graf, rppt, kuzuno, mgross, alexandre.chartre
Add an ASI fault handler and options to define the handler behavior.
Depending on the ASI, the ASI fault handler can either abort the
isolation and retry the faulty instruction with the full kernel
page-table, or preserve the isolation and process the fault like
any regular fault. If isolation is aborted then the location and
address of the fault can be logged and optionally include a stack
trace.
Signed-off-by: Alexandre Chartre <alexandre.chartre@oracle.com>
---
arch/x86/include/asm/asi.h | 42 +++++++++++++++++++-
arch/x86/mm/asi.c | 95 ++++++++++++++++++++++++++++++++++++++++++++
arch/x86/mm/fault.c | 20 +++++++++
3 files changed, 156 insertions(+), 1 deletions(-)
diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h
index a0733f1..b8d7b93 100644
--- a/arch/x86/include/asm/asi.h
+++ b/arch/x86/include/asm/asi.h
@@ -66,6 +66,7 @@ struct asi_type {
int pcid_prefix; /* PCID prefix */
struct asi_tlb_state *tlb_state; /* percpu ASI TLB state */
atomic64_t last_pgtable_id; /* last id for this type */
+ bool fault_abort; /* abort ASI on fault? */
};
/*
@@ -75,12 +76,13 @@ struct asi_type {
* (asi_create_<typename>()) to easily create an ASI of the
* specified type.
*/
-#define DEFINE_ASI_TYPE(name, pcid_prefix) \
+#define DEFINE_ASI_TYPE(name, pcid_prefix, fault_abort) \
DEFINE_PER_CPU(struct asi_tlb_state, asi_tlb_ ## name); \
struct asi_type asi_type_ ## name = { \
pcid_prefix, \
&asi_tlb_ ## name, \
ATOMIC64_INIT(1), \
+ fault_abort \
}; \
EXPORT_SYMBOL(asi_type_ ## name)
@@ -94,16 +96,49 @@ struct asi_type {
return asi_create(&asi_type_ ## name); \
}
+/* ASI fault log size */
+#define ASI_FAULT_LOG_SIZE 128
+
+/*
+ * Options to specify the fault log policy when a fault occurs
+ * while using ASI.
+ *
+ * When set, ASI_FAULT_LOG_KERNEL|USER log the address and location
+ * of the fault. In addition, if ASI_FAULT_LOG_STACK is set, the stack
+ * trace where the fault occurred is also logged.
+ *
+ * Faults are logged only for ASIs with a type which aborts ASI on an
+ * ASI fault (see fault_abort in struct asi_type).
+ */
+#define ASI_FAULT_LOG_KERNEL 0x01 /* log kernel faults */
+#define ASI_FAULT_LOG_USER 0x02 /* log user faults */
+#define ASI_FAULT_LOG_STACK 0x04 /* log stack trace */
+
+enum asi_fault_origin {
+ ASI_FAULT_KERNEL = ASI_FAULT_LOG_KERNEL,
+ ASI_FAULT_USER = ASI_FAULT_LOG_USER,
+};
+
+struct asi_fault_log {
+ unsigned long address; /* fault address */
+ unsigned long count; /* fault count */
+};
+
struct asi {
struct asi_type *type; /* ASI type */
pgd_t *pagetable; /* ASI pagetable */
u64 pgtable_id; /* ASI pagetable ID */
atomic64_t pgtable_gen; /* ASI pagetable generation */
unsigned long base_cr3; /* base ASI CR3 */
+ spinlock_t fault_lock; /* protect fault_log_* */
+ struct asi_fault_log fault_log[ASI_FAULT_LOG_SIZE];
+ int fault_log_policy; /* fault log policy */
};
void asi_schedule_out(struct task_struct *task);
void asi_schedule_in(struct task_struct *task);
+bool asi_fault(struct pt_regs *regs, unsigned long error_code,
+ unsigned long address, enum asi_fault_origin fault_origin);
extern struct asi *asi_create(struct asi_type *type);
extern void asi_destroy(struct asi *asi);
@@ -111,6 +146,11 @@ struct asi {
extern int asi_enter(struct asi *asi);
extern void asi_exit(struct asi *asi);
+static inline void asi_set_log_policy(struct asi *asi, int policy)
+{
+ asi->fault_log_policy = policy;
+}
+
#else /* __ASSEMBLY__ */
#include <asm/alternative-asm.h>
diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c
index 9955eb2..6c94d29 100644
--- a/arch/x86/mm/asi.c
+++ b/arch/x86/mm/asi.c
@@ -6,6 +6,7 @@
*/
#include <linux/mm.h>
+#include <linux/sched/debug.h>
#include <linux/slab.h>
#include <asm/asi.h>
@@ -13,6 +14,97 @@
#include <asm/mmu_context.h>
#include <asm/tlbflush.h>
+static void asi_log_fault(struct asi *asi, struct pt_regs *regs,
+ unsigned long error_code, unsigned long address,
+ enum asi_fault_origin fault_origin)
+{
+ int i;
+
+ /*
+ * Log information about the fault only if this is a fault
+ * we don't know about yet (and the fault log is not full).
+ */
+ spin_lock(&asi->fault_lock);
+ if (!(asi->fault_log_policy & fault_origin)) {
+ spin_unlock(&asi->fault_lock);
+ return;
+ }
+ for (i = 0; i < ASI_FAULT_LOG_SIZE; i++) {
+ if (asi->fault_log[i].address == regs->ip) {
+ asi->fault_log[i].count++;
+ spin_unlock(&asi->fault_lock);
+ return;
+ }
+ if (!asi->fault_log[i].address) {
+ asi->fault_log[i].address = regs->ip;
+ asi->fault_log[i].count = 1;
+ break;
+ }
+ }
+
+ if (i >= ASI_FAULT_LOG_SIZE) {
+ pr_warn("ASI %p: fault log buffer is full [%d]\n",
+ asi, i);
+ }
+
+ pr_info("ASI %p: PF#%d (%ld) at %pS on %px\n", asi, i,
+ error_code, (void *)regs->ip, (void *)address);
+
+ if (asi->fault_log_policy & ASI_FAULT_LOG_STACK)
+ show_stack(NULL, (unsigned long *)regs->sp);
+
+ spin_unlock(&asi->fault_lock);
+}
+
+bool asi_fault(struct pt_regs *regs, unsigned long error_code,
+ unsigned long address, enum asi_fault_origin fault_origin)
+{
+ struct asi_session *asi_session;
+
+ /*
+ * If address space isolation was active when the fault occurred
+ * then the page fault handler has interrupted the isolation
+ * (exception handlers interrupt isolation very early) and switched
+ * CR3 back to its original kernel value. So we can safely retrieved
+ * the CPU ASI session.
+ */
+ asi_session = &get_cpu_var(cpu_asi_session);
+
+ /*
+ * If address space isolation is not active, or we have a fault
+ * after isolation was aborted then this was not a fault while
+ * using ASI and we don't handle it.
+ */
+ if (!asi_session->asi || asi_session->idepth > 1)
+ return false;
+
+ /*
+ * We have a fault while the CPU is using address space isolation.
+ * Depending on the ASI fault policy, either:
+ *
+ * - Abort the isolation. The ASI used when the fault occurred is
+ * aborted, and the faulty instruction is immediately retried.
+ * The fault is not processed by the system fault handler. The
+ * fault handler will return immediately, the system will not
+ * restore the ASI pagetable and will continue to run with the
+ * full kernel pagetable.
+ *
+ * - Or preserve the isolation. The system fault handler will
+ * process the fault like any regular fault. The ASI pagetable
+ * be restored after the fault has been handled and the system
+ * fault handler returns.
+ */
+ if (asi_session->asi->type->fault_abort) {
+ asi_log_fault(asi_session->asi, regs, error_code,
+ address, fault_origin);
+ asi_session->asi = NULL;
+ asi_session->idepth = 0;
+ return true;
+ }
+
+ return false;
+}
+
struct asi *asi_create(struct asi_type *type)
{
struct asi *asi;
@@ -27,6 +119,9 @@ struct asi *asi_create(struct asi_type *type)
asi->type = type;
asi->pgtable_id = atomic64_inc_return(&type->last_pgtable_id);
atomic64_set(&asi->pgtable_gen, 0);
+ spin_lock_init(&asi->fault_lock);
+ /* by default, log ASI kernel faults */
+ asi->fault_log_policy = ASI_FAULT_LOG_KERNEL;
return asi;
}
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index 304d31d..d50676f 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -29,6 +29,7 @@
#include <asm/efi.h> /* efi_recover_from_page_fault()*/
#include <asm/desc.h> /* store_idt(), ... */
#include <asm/cpu_entry_area.h> /* exception stack */
+#include <asm/asi.h> /* asi_fault() */
#define CREATE_TRACE_POINTS
#include <asm/trace/exceptions.h>
@@ -1235,6 +1236,15 @@ static int fault_in_kernel_space(unsigned long address)
WARN_ON_ONCE(hw_error_code & X86_PF_PK);
/*
+ * Check if the fault occurs with ASI and if the ASI handler
+ * handles it.
+ */
+ if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION) &&
+ asi_fault(regs, hw_error_code, address, ASI_FAULT_KERNEL)) {
+ return;
+ }
+
+ /*
* We can fault-in kernel-space virtual memory on-demand. The
* 'reference' page table is init_mm.pgd.
*
@@ -1289,6 +1299,16 @@ void do_user_addr_fault(struct pt_regs *regs,
vm_fault_t fault, major = 0;
unsigned int flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+
+ /*
+ * Check if the fault occurs with ASI and if the ASI handler
+ * handles it.
+ */
+ if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION) &&
+ asi_fault(regs, hw_error_code, address, ASI_FAULT_USER)) {
+ return;
+ }
+
tsk = current;
mm = tsk->mm;
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [RFC PATCH v3 7/7] mm/asi: Implement PTI with ASI
2020-02-26 16:21 [RFC PATCH v3 0/7] Kernel Address Space Isolation Alexandre Chartre
` (5 preceding siblings ...)
2020-02-26 16:21 ` [RFC PATCH v3 6/7] mm/asi: ASI fault handler Alexandre Chartre
@ 2020-02-26 16:22 ` Alexandre Chartre
2020-02-26 20:32 ` kbuild test robot
6 siblings, 1 reply; 16+ messages in thread
From: Alexandre Chartre @ 2020-02-26 16:22 UTC (permalink / raw)
To: rkrcmar, tglx, mingo, bp, hpa, dave.hansen, luto, peterz, x86,
linux-mm, linux-kernel
Cc: pbonzini, konrad.wilk, jan.setjeeilers, liran.alon, junaids,
graf, rppt, kuzuno, mgross, alexandre.chartre
ASI supersedes PTI. If both CONFIG_ADDRESS_SPACE_ISOLATION and
CONFIG_PAGE_TABLE_ISOLATION are set then PTI is implemented using
ASI. For each user process, a "user" ASI is then defined with the
PTI pagetable. The user ASI is used when running userland code, and
it is exited when entering a syscall. The user ASI is re-entered
when the syscall returns to userland.
As with any ASI, interrupts/exceptions/NMIs will interrupt the
ASI, the ASI will resume when the interrupt/exception/NMI has
completed. Faults won't abort the user ASI as user faults are
handled by the kernel before returning to userland.
Signed-off-by: Alexandre Chartre <alexandre.chartre@oracle.com>
---
arch/x86/entry/calling.h | 7 ++++++-
arch/x86/entry/common.c | 29 ++++++++++++++++++++++++-----
arch/x86/entry/entry_64.S | 6 ++++++
arch/x86/include/asm/asi.h | 9 +++++++++
arch/x86/include/asm/tlbflush.h | 11 +++++++++--
arch/x86/mm/asi.c | 9 +++++++++
arch/x86/mm/pti.c | 28 ++++++++++++++++++++--------
include/linux/mm_types.h | 5 +++++
kernel/fork.c | 17 +++++++++++++++++
9 files changed, 105 insertions(+), 16 deletions(-)
diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h
index ca23b79..ce0fccd 100644
--- a/arch/x86/entry/calling.h
+++ b/arch/x86/entry/calling.h
@@ -176,16 +176,21 @@
#if defined(CONFIG_ADDRESS_SPACE_ISOLATION)
/*
- * For now, ASI is not compatible with PTI.
+ * ASI supersedes the entry points used by PTI. If both
+ * CONFIG_ADDRESS_SPACE_ISOLATION and CONFIG_PAGE_TABLE_ISOLATION are
+ * set then PTI is implemented using ASI.
*/
.macro SWITCH_TO_KERNEL_CR3 scratch_reg:req
+ ASI_INTERRUPT \scratch_reg
.endm
.macro SWITCH_TO_USER_CR3_NOSTACK scratch_reg:req scratch_reg2:req
+ ASI_RESUME \scratch_reg
.endm
.macro SWITCH_TO_USER_CR3_STACK scratch_reg:req
+ ASI_RESUME \scratch_reg
.endm
.macro SAVE_AND_SWITCH_TO_KERNEL_CR3 scratch_reg:req save_reg:req
diff --git a/arch/x86/entry/common.c b/arch/x86/entry/common.c
index 9747876..a437de3 100644
--- a/arch/x86/entry/common.c
+++ b/arch/x86/entry/common.c
@@ -34,6 +34,7 @@
#include <asm/fpu/api.h>
#include <asm/nospec-branch.h>
#include <asm/io_bitmap.h>
+#include <asm/asi.h>
#define CREATE_TRACE_POINTS
#include <trace/events/syscalls.h>
@@ -49,6 +50,13 @@ __visible inline void enter_from_user_mode(void)
static inline void enter_from_user_mode(void) {}
#endif
+static inline void syscall_enter(void)
+{
+ /* syscall enter has interrupted ASI, now exit ASI */
+ asi_exit(current->mm->user_asi);
+ enter_from_user_mode();
+}
+
static void do_audit_syscall_entry(struct pt_regs *regs, u32 arch)
{
#ifdef CONFIG_X86_64
@@ -224,6 +232,17 @@ __visible inline void prepare_exit_to_usermode(struct pt_regs *regs)
mds_user_clear_cpu_buffers();
}
+static inline void prepare_syscall_return(struct pt_regs *regs)
+{
+ prepare_exit_to_usermode(regs);
+
+ /*
+ * Syscall return will resume ASI, prepare resume to enter
+ * user ASI.
+ */
+ asi_deferred_enter(current->mm->user_asi);
+}
+
#define SYSCALL_EXIT_WORK_FLAGS \
(_TIF_SYSCALL_TRACE | _TIF_SYSCALL_AUDIT | \
_TIF_SINGLESTEP | _TIF_SYSCALL_TRACEPOINT)
@@ -275,7 +294,7 @@ __visible inline void syscall_return_slowpath(struct pt_regs *regs)
syscall_slow_exit_work(regs, cached_flags);
local_irq_disable();
- prepare_exit_to_usermode(regs);
+ prepare_syscall_return(regs);
}
#ifdef CONFIG_X86_64
@@ -283,7 +302,7 @@ __visible void do_syscall_64(unsigned long nr, struct pt_regs *regs)
{
struct thread_info *ti;
- enter_from_user_mode();
+ syscall_enter();
local_irq_enable();
ti = current_thread_info();
if (READ_ONCE(ti->flags) & _TIF_WORK_SYSCALL_ENTRY)
@@ -355,7 +374,7 @@ static __always_inline void do_syscall_32_irqs_on(struct pt_regs *regs)
/* Handles int $0x80 */
__visible void do_int80_syscall_32(struct pt_regs *regs)
{
- enter_from_user_mode();
+ syscall_enter();
local_irq_enable();
do_syscall_32_irqs_on(regs);
}
@@ -378,7 +397,7 @@ __visible long do_fast_syscall_32(struct pt_regs *regs)
*/
regs->ip = landing_pad;
- enter_from_user_mode();
+ syscall_enter();
local_irq_enable();
@@ -400,7 +419,7 @@ __visible long do_fast_syscall_32(struct pt_regs *regs)
/* User code screwed up. */
local_irq_disable();
regs->ax = -EFAULT;
- prepare_exit_to_usermode(regs);
+ prepare_syscall_return(regs);
return 0; /* Keep it simple: use IRET. */
}
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index fddb820..9042ba1 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -627,6 +627,9 @@ ret_from_intr:
.Lretint_user:
mov %rsp,%rdi
call prepare_exit_to_usermode
+#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
+ ASI_PREPARE_RESUME
+#endif
TRACE_IRQS_IRETQ
SYM_INNER_LABEL(swapgs_restore_regs_and_return_to_usermode, SYM_L_GLOBAL)
@@ -1491,6 +1494,9 @@ SYM_CODE_START(nmi)
movq %rsp, %rdi
movq $-1, %rsi
call do_nmi
+#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
+ ASI_PREPARE_RESUME
+#endif
/*
* Return back to user mode. We must *not* do the normal exit
diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h
index b8d7b93..ac0594d 100644
--- a/arch/x86/include/asm/asi.h
+++ b/arch/x86/include/asm/asi.h
@@ -62,6 +62,10 @@ struct asi_tlb_state {
struct asi_tlb_pgtable tlb_pgtables[ASI_TLB_NR_DYN_ASIDS];
};
+#ifdef CONFIG_PAGE_TABLE_ISOLATION
+#define ASI_PCID_PREFIX_USER 0x80 /* user ASI */
+#endif
+
struct asi_type {
int pcid_prefix; /* PCID prefix */
struct asi_tlb_state *tlb_state; /* percpu ASI TLB state */
@@ -139,6 +143,7 @@ struct asi {
void asi_schedule_in(struct task_struct *task);
bool asi_fault(struct pt_regs *regs, unsigned long error_code,
unsigned long address, enum asi_fault_origin fault_origin);
+void asi_deferred_enter(struct asi *asi);
extern struct asi *asi_create(struct asi_type *type);
extern void asi_destroy(struct asi *asi);
@@ -146,6 +151,10 @@ bool asi_fault(struct pt_regs *regs, unsigned long error_code,
extern int asi_enter(struct asi *asi);
extern void asi_exit(struct asi *asi);
+#ifdef CONFIG_PAGE_TABLE_ISOLATION
+DECLARE_ASI_TYPE(user);
+#endif
+
static inline void asi_set_log_policy(struct asi *asi, int policy)
{
asi->fault_log_policy = policy;
diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h
index 241058f..db114de 100644
--- a/arch/x86/include/asm/tlbflush.h
+++ b/arch/x86/include/asm/tlbflush.h
@@ -390,6 +390,8 @@ static inline void cr4_set_bits_and_update_boot(unsigned long mask)
*/
static inline void invalidate_user_asid(u16 asid)
{
+ struct asi_tlb_state *tlb_state;
+
/* There is no user ASID if address space separation is off */
if (!IS_ENABLED(CONFIG_PAGE_TABLE_ISOLATION))
return;
@@ -404,8 +406,13 @@ static inline void invalidate_user_asid(u16 asid)
if (!static_cpu_has(X86_FEATURE_PTI))
return;
- __set_bit(kern_pcid(asid),
- (unsigned long *)this_cpu_ptr(&cpu_tlbstate.user_pcid_flush_mask));
+ if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION)) {
+ tlb_state = get_cpu_ptr(asi_type_user.tlb_state);
+ tlb_state->tlb_pgtables[asid].id = 0;
+ } else {
+ __set_bit(kern_pcid(asid),
+ (unsigned long *)this_cpu_ptr(&cpu_tlbstate.user_pcid_flush_mask));
+ }
}
/*
diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c
index 6c94d29..3448413 100644
--- a/arch/x86/mm/asi.c
+++ b/arch/x86/mm/asi.c
@@ -14,6 +14,10 @@
#include <asm/mmu_context.h>
#include <asm/tlbflush.h>
+#ifdef CONFIG_PAGE_TABLE_ISOLATION
+DEFINE_ASI_TYPE(user, ASI_PCID_PREFIX_USER, false);
+#endif
+
static void asi_log_fault(struct asi *asi, struct pt_regs *regs,
unsigned long error_code, unsigned long address,
enum asi_fault_origin fault_origin)
@@ -314,6 +318,11 @@ void asi_exit(struct asi *asi)
}
EXPORT_SYMBOL(asi_exit);
+void asi_deferred_enter(struct asi *asi)
+{
+ asi_switch_to_asi_cr3(asi, ASI_SWITCH_ON_RESUME);
+}
+
void asi_prepare_resume(void)
{
struct asi_session *asi_session;
diff --git a/arch/x86/mm/pti.c b/arch/x86/mm/pti.c
index 44a9f06..9f91a93 100644
--- a/arch/x86/mm/pti.c
+++ b/arch/x86/mm/pti.c
@@ -429,6 +429,18 @@ static void __init pti_clone_p4d(unsigned long addr)
*user_p4d = *kernel_p4d;
}
+static void __init pti_map_va(unsigned long va)
+{
+ phys_addr_t pa = per_cpu_ptr_to_phys((void *)va);
+ pte_t *target_pte;
+
+ target_pte = pti_user_pagetable_walk_pte(va);
+ if (WARN_ON(!target_pte))
+ return;
+
+ *target_pte = pfn_pte(pa >> PAGE_SHIFT, PAGE_KERNEL);
+}
+
/*
* Clone the CPU_ENTRY_AREA and associated data into the user space visible
* page table.
@@ -456,15 +468,15 @@ static void __init pti_clone_user_shared(void)
* is set up.
*/
- unsigned long va = (unsigned long)&per_cpu(cpu_tss_rw, cpu);
- phys_addr_t pa = per_cpu_ptr_to_phys((void *)va);
- pte_t *target_pte;
-
- target_pte = pti_user_pagetable_walk_pte(va);
- if (WARN_ON(!target_pte))
- return;
+ pti_map_va((unsigned long)&per_cpu(cpu_tss_rw, cpu));
- *target_pte = pfn_pte(pa >> PAGE_SHIFT, PAGE_KERNEL);
+ if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION)) {
+ /*
+ * Map the ASI session. We need to always be able
+ * to access the ASI session.
+ */
+ pti_map_va((unsigned long)&per_cpu(cpu_tlbstate, cpu));
+ }
}
}
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 270aa8f..0152f73 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -25,6 +25,7 @@
struct address_space;
struct mem_cgroup;
+struct asi;
/*
* Each physical page in the system has a struct page associated with
@@ -524,6 +525,10 @@ struct mm_struct {
atomic_long_t hugetlb_usage;
#endif
struct work_struct async_put_work;
+#if defined(CONFIG_ADDRESS_SPACE_ISOLATION) && defined(CONFIG_PAGE_TABLE_ISOLATION)
+ /* ASI used for user address space */
+ struct asi *user_asi;
+#endif
} __randomize_layout;
/*
diff --git a/kernel/fork.c b/kernel/fork.c
index 0808095..d245cc0 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -101,6 +101,7 @@
#include <asm/mmu_context.h>
#include <asm/cacheflush.h>
#include <asm/tlbflush.h>
+#include <asm/asi.h>
#include <trace/events/sched.h>
@@ -695,6 +696,10 @@ void __mmdrop(struct mm_struct *mm)
mmu_notifier_mm_destroy(mm);
check_mm(mm);
put_user_ns(mm->user_ns);
+ if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION) &&
+ IS_ENABLED(CONFIG_PAGE_TABLE_ISOLATION)) {
+ asi_destroy(mm->user_asi);
+ }
free_mm(mm);
}
EXPORT_SYMBOL_GPL(__mmdrop);
@@ -1046,6 +1051,18 @@ static void mm_init_uprobes_state(struct mm_struct *mm)
if (init_new_context(p, mm))
goto fail_nocontext;
+ if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION) &&
+ IS_ENABLED(CONFIG_PAGE_TABLE_ISOLATION)) {
+ /*
+ * If we have PTI and ASI then use ASI to switch between
+ * user and kernel spaces, so create an ASI for this mm.
+ */
+ mm->user_asi = asi_create_user();
+ if (!mm->user_asi)
+ goto fail_nocontext;
+ asi_set_pagetable(mm->user_asi, kernel_to_user_pgdp(mm->pgd));
+ }
+
mm->user_ns = get_user_ns(user_ns);
return mm;
--
1.7.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [RFC PATCH v3 7/7] mm/asi: Implement PTI with ASI
2020-02-26 16:22 ` [RFC PATCH v3 7/7] mm/asi: Implement PTI with ASI Alexandre Chartre
@ 2020-02-26 20:32 ` kbuild test robot
0 siblings, 0 replies; 16+ messages in thread
From: kbuild test robot @ 2020-02-26 20:32 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 9970 bytes --]
Hi Alexandre,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/x86/core]
[also build test ERROR on tip/x86/asm]
[cannot apply to tip/x86/mm v5.6-rc3 next-20200226]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Alexandre-Chartre/Kernel-Address-Space-Isolation/20200227-034033
base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 248ed51048c40d36728e70914e38bffd7821da57
config: i386-tinyconfig (attached as .config)
compiler: gcc-7 (Debian 7.5.0-5) 7.5.0
reproduce:
# save the attached .config to linux build tree
make ARCH=i386
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>
All error/warnings (new ones prefixed by >>):
In file included from include/asm-generic/percpu.h:7:0,
from arch/x86/include/asm/percpu.h:556,
from arch/x86/include/asm/preempt.h:6,
from include/linux/preempt.h:78,
from include/linux/spinlock.h:51,
from include/linux/mmzone.h:8,
from include/linux/gfp.h:6,
from include/linux/slab.h:15,
from include/linux/crypto.h:19,
from arch/x86/kernel/asm-offsets.c:9:
arch/x86/include/asm/tlbflush.h: In function 'invalidate_user_asid':
>> arch/x86/include/asm/tlbflush.h:410:27: error: 'asi_type_user' undeclared (first use in this function)
tlb_state = get_cpu_ptr(asi_type_user.tlb_state);
^
include/linux/percpu-defs.h:220:47: note: in definition of macro '__verify_pcpu_ptr'
const void __percpu *__vpp_verify = (typeof((ptr) + 0))NULL; \
^~~
>> include/linux/percpu-defs.h:264:47: note: in expansion of macro 'VERIFY_PERCPU_PTR'
#define per_cpu_ptr(ptr, cpu) ({ (void)(cpu); VERIFY_PERCPU_PTR(ptr); })
^~~~~~~~~~~~~~~~~
>> include/linux/percpu-defs.h:265:26: note: in expansion of macro 'per_cpu_ptr'
#define raw_cpu_ptr(ptr) per_cpu_ptr(ptr, 0)
^~~~~~~~~~~
>> include/linux/percpu-defs.h:266:27: note: in expansion of macro 'raw_cpu_ptr'
#define this_cpu_ptr(ptr) raw_cpu_ptr(ptr)
^~~~~~~~~~~
>> include/linux/percpu-defs.h:295:2: note: in expansion of macro 'this_cpu_ptr'
this_cpu_ptr(var); \
^~~~~~~~~~~~
>> arch/x86/include/asm/tlbflush.h:410:15: note: in expansion of macro 'get_cpu_ptr'
tlb_state = get_cpu_ptr(asi_type_user.tlb_state);
^~~~~~~~~~~
arch/x86/include/asm/tlbflush.h:410:27: note: each undeclared identifier is reported only once for each function it appears in
tlb_state = get_cpu_ptr(asi_type_user.tlb_state);
^
include/linux/percpu-defs.h:220:47: note: in definition of macro '__verify_pcpu_ptr'
const void __percpu *__vpp_verify = (typeof((ptr) + 0))NULL; \
^~~
>> include/linux/percpu-defs.h:264:47: note: in expansion of macro 'VERIFY_PERCPU_PTR'
#define per_cpu_ptr(ptr, cpu) ({ (void)(cpu); VERIFY_PERCPU_PTR(ptr); })
^~~~~~~~~~~~~~~~~
>> include/linux/percpu-defs.h:265:26: note: in expansion of macro 'per_cpu_ptr'
#define raw_cpu_ptr(ptr) per_cpu_ptr(ptr, 0)
^~~~~~~~~~~
>> include/linux/percpu-defs.h:266:27: note: in expansion of macro 'raw_cpu_ptr'
#define this_cpu_ptr(ptr) raw_cpu_ptr(ptr)
^~~~~~~~~~~
>> include/linux/percpu-defs.h:295:2: note: in expansion of macro 'this_cpu_ptr'
this_cpu_ptr(var); \
^~~~~~~~~~~~
>> arch/x86/include/asm/tlbflush.h:410:15: note: in expansion of macro 'get_cpu_ptr'
tlb_state = get_cpu_ptr(asi_type_user.tlb_state);
^~~~~~~~~~~
In file included from arch/x86/kernel/asm-offsets.c:20:0:
>> arch/x86/include/asm/tlbflush.h:411:12: error: dereferencing pointer to incomplete type 'struct asi_tlb_state'
tlb_state->tlb_pgtables[asid].id = 0;
^~
In file included from arch/x86/kernel/asm-offsets.c:14:0:
arch/x86/kernel/asm-offsets.c: In function 'common':
include/linux/compiler_types.h:129:35: error: 'struct tlb_state' has no member named 'asi_session'
#define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
^
include/linux/kbuild.h:6:62: note: in definition of macro 'DEFINE'
asm volatile("\n.ascii \"->" #sym " %0 " #val "\"" : : "i" (val))
^~~
include/linux/stddef.h:17:32: note: in expansion of macro '__compiler_offsetof'
#define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
^~~~~~~~~~~~~~~~~~~
include/linux/kbuild.h:11:14: note: in expansion of macro 'offsetof'
DEFINE(sym, offsetof(struct str, mem))
^~~~~~~~
arch/x86/kernel/asm-offsets.c:98:2: note: in expansion of macro 'OFFSET'
OFFSET(TLB_STATE_asi, tlb_state, asi_session.asi);
^~~~~~
include/linux/compiler_types.h:129:35: error: 'struct tlb_state' has no member named 'asi_session'
#define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
^
include/linux/kbuild.h:6:62: note: in definition of macro 'DEFINE'
asm volatile("\n.ascii \"->" #sym " %0 " #val "\"" : : "i" (val))
^~~
include/linux/stddef.h:17:32: note: in expansion of macro '__compiler_offsetof'
#define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
^~~~~~~~~~~~~~~~~~~
include/linux/kbuild.h:11:14: note: in expansion of macro 'offsetof'
DEFINE(sym, offsetof(struct str, mem))
^~~~~~~~
arch/x86/kernel/asm-offsets.c:99:2: note: in expansion of macro 'OFFSET'
OFFSET(TLB_STATE_asi_isolation_cr3, tlb_state,
^~~~~~
include/linux/compiler_types.h:129:35: error: 'struct tlb_state' has no member named 'asi_session'
#define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
^
include/linux/kbuild.h:6:62: note: in definition of macro 'DEFINE'
asm volatile("\n.ascii \"->" #sym " %0 " #val "\"" : : "i" (val))
^~~
include/linux/stddef.h:17:32: note: in expansion of macro '__compiler_offsetof'
#define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
^~~~~~~~~~~~~~~~~~~
include/linux/kbuild.h:11:14: note: in expansion of macro 'offsetof'
DEFINE(sym, offsetof(struct str, mem))
^~~~~~~~
arch/x86/kernel/asm-offsets.c:101:2: note: in expansion of macro 'OFFSET'
OFFSET(TLB_STATE_asi_original_cr3, tlb_state, asi_session.original_cr3);
^~~~~~
include/linux/compiler_types.h:129:35: error: 'struct tlb_state' has no member named 'asi_session'
#define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
^
include/linux/kbuild.h:6:62: note: in definition of macro 'DEFINE'
asm volatile("\n.ascii \"->" #sym " %0 " #val "\"" : : "i" (val))
^~~
include/linux/stddef.h:17:32: note: in expansion of macro '__compiler_offsetof'
#define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
^~~~~~~~~~~~~~~~~~~
include/linux/kbuild.h:11:14: note: in expansion of macro 'offsetof'
DEFINE(sym, offsetof(struct str, mem))
^~~~~~~~
arch/x86/kernel/asm-offsets.c:102:2: note: in expansion of macro 'OFFSET'
OFFSET(TLB_STATE_asi_idepth, tlb_state, asi_session.idepth);
^~~~~~
make[2]: *** [scripts/Makefile.build:99: arch/x86/kernel/asm-offsets.s] Error 1
make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1113: prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:179: sub-make] Error 2
55 real 6 user 6 sys 24.45% cpu make prepare
vim +/asi_type_user +410 arch/x86/include/asm/tlbflush.h
384
385 /*
386 * Given an ASID, flush the corresponding user ASID. We can delay this
387 * until the next time we switch to it.
388 *
389 * See SWITCH_TO_USER_CR3.
390 */
391 static inline void invalidate_user_asid(u16 asid)
392 {
393 struct asi_tlb_state *tlb_state;
394
395 /* There is no user ASID if address space separation is off */
396 if (!IS_ENABLED(CONFIG_PAGE_TABLE_ISOLATION))
397 return;
398
399 /*
400 * We only have a single ASID if PCID is off and the CR3
401 * write will have flushed it.
402 */
403 if (!cpu_feature_enabled(X86_FEATURE_PCID))
404 return;
405
406 if (!static_cpu_has(X86_FEATURE_PTI))
407 return;
408
409 if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION)) {
> 410 tlb_state = get_cpu_ptr(asi_type_user.tlb_state);
> 411 tlb_state->tlb_pgtables[asid].id = 0;
412 } else {
413 __set_bit(kern_pcid(asid),
414 (unsigned long *)this_cpu_ptr(&cpu_tlbstate.user_pcid_flush_mask));
415 }
416 }
417
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 7284 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH v3 2/7] mm/asi: ASI entry/exit interface
2020-02-26 16:21 ` [RFC PATCH v3 2/7] mm/asi: ASI entry/exit interface Alexandre Chartre
@ 2020-02-26 20:38 ` kbuild test robot
2020-02-26 22:02 ` Ira Weiny
1 sibling, 0 replies; 16+ messages in thread
From: kbuild test robot @ 2020-02-26 20:38 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 5763 bytes --]
Hi Alexandre,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/x86/core]
[also build test ERROR on tip/x86/asm tip/x86/mm v5.6-rc3 next-20200226]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Alexandre-Chartre/Kernel-Address-Space-Isolation/20200227-034033
base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 248ed51048c40d36728e70914e38bffd7821da57
config: i386-tinyconfig (attached as .config)
compiler: gcc-7 (Debian 7.5.0-5) 7.5.0
reproduce:
# save the attached .config to linux build tree
make ARCH=i386
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>
All error/warnings (new ones prefixed by >>):
In file included from include/asm-generic/percpu.h:7:0,
from arch/x86/include/asm/percpu.h:556,
from arch/x86/include/asm/current.h:6,
from include/linux/sched.h:12,
from include/linux/ptrace.h:6,
from include/uapi/asm-generic/bpf_perf_event.h:4,
from ./arch/x86/include/generated/uapi/asm/bpf_perf_event.h:1,
from include/uapi/linux/bpf_perf_event.h:11,
from include/linux/perf_event.h:18,
from arch/x86/events/core.c:15:
arch/x86/include/asm/mmu_context.h: In function '__get_current_cr3_fast':
>> arch/x86/include/asm/mmu_context.h:361:20: error: 'cpu_asi_session' undeclared (first use in this function); did you mean 'task_session'?
this_cpu_read(cpu_asi_session.asi)) {
^
include/linux/percpu-defs.h:319:9: note: in definition of macro '__pcpu_size_call_return'
typeof(variable) pscr_ret__; \
^~~~~~~~
>> arch/x86/include/asm/mmu_context.h:361:6: note: in expansion of macro 'this_cpu_read'
this_cpu_read(cpu_asi_session.asi)) {
^~~~~~~~~~~~~
arch/x86/include/asm/mmu_context.h:361:20: note: each undeclared identifier is reported only once for each function it appears in
this_cpu_read(cpu_asi_session.asi)) {
^
include/linux/percpu-defs.h:319:9: note: in definition of macro '__pcpu_size_call_return'
typeof(variable) pscr_ret__; \
^~~~~~~~
>> arch/x86/include/asm/mmu_context.h:361:6: note: in expansion of macro 'this_cpu_read'
this_cpu_read(cpu_asi_session.asi)) {
^~~~~~~~~~~~~
--
In file included from include/asm-generic/percpu.h:7:0,
from arch/x86/include/asm/percpu.h:556,
from arch/x86/include/asm/current.h:6,
from include/linux/mutex.h:14,
from include/linux/notifier.h:14,
from include/linux/reboot.h:6,
from arch/x86/kernel/reboot.c:5:
arch/x86/include/asm/mmu_context.h: In function '__get_current_cr3_fast':
>> arch/x86/include/asm/mmu_context.h:361:20: error: 'cpu_asi_session' undeclared (first use in this function); did you mean 'cpumask_size'?
this_cpu_read(cpu_asi_session.asi)) {
^
include/linux/percpu-defs.h:319:9: note: in definition of macro '__pcpu_size_call_return'
typeof(variable) pscr_ret__; \
^~~~~~~~
>> arch/x86/include/asm/mmu_context.h:361:6: note: in expansion of macro 'this_cpu_read'
this_cpu_read(cpu_asi_session.asi)) {
^~~~~~~~~~~~~
arch/x86/include/asm/mmu_context.h:361:20: note: each undeclared identifier is reported only once for each function it appears in
this_cpu_read(cpu_asi_session.asi)) {
^
include/linux/percpu-defs.h:319:9: note: in definition of macro '__pcpu_size_call_return'
typeof(variable) pscr_ret__; \
^~~~~~~~
>> arch/x86/include/asm/mmu_context.h:361:6: note: in expansion of macro 'this_cpu_read'
this_cpu_read(cpu_asi_session.asi)) {
^~~~~~~~~~~~~
vim +361 arch/x86/include/asm/mmu_context.h
343
344 /*
345 * This can be used from process context to figure out what the value of
346 * CR3 is without needing to do a (slow) __read_cr3().
347 *
348 * It's intended to be used for code like KVM that sneakily changes CR3
349 * and needs to restore it. It needs to be used very carefully.
350 */
351 static inline unsigned long __get_current_cr3_fast(void)
352 {
353 unsigned long cr3;
354
355 /*
356 * If isolation is active then we need to return the CR3 for the
357 * currently active ASI. This value is stored in the isolation_cr3
358 * field of the ASI session.
359 */
360 if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION) &&
> 361 this_cpu_read(cpu_asi_session.asi)) {
362 cr3 = this_cpu_read(cpu_asi_session.isolation_cr3);
363 /* CR3 read never returns with the NOFLUSH bit */
364 cr3 &= ~X86_CR3_PCID_NOFLUSH;
365 } else {
366 cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd,
367 this_cpu_read(cpu_tlbstate.loaded_mm_asid));
368 }
369
370 /* For now, be very restrictive about when this can be called. */
371 VM_WARN_ON(in_nmi() || preemptible());
372
373 VM_BUG_ON(cr3 != __read_cr3());
374 return cr3;
375 }
376
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 7284 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH v3 4/7] mm/asi: Interrupt ASI on interrupt/exception/NMI
2020-02-26 16:21 ` [RFC PATCH v3 4/7] mm/asi: Interrupt ASI on interrupt/exception/NMI Alexandre Chartre
@ 2020-02-26 20:38 ` kbuild test robot
0 siblings, 0 replies; 16+ messages in thread
From: kbuild test robot @ 2020-02-26 20:38 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 5849 bytes --]
Hi Alexandre,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/x86/core]
[also build test ERROR on tip/x86/asm tip/x86/mm v5.6-rc3 next-20200226]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Alexandre-Chartre/Kernel-Address-Space-Isolation/20200227-034033
base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 248ed51048c40d36728e70914e38bffd7821da57
config: i386-tinyconfig (attached as .config)
compiler: gcc-7 (Debian 7.5.0-5) 7.5.0
reproduce:
# save the attached .config to linux build tree
make ARCH=i386
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>
All error/warnings (new ones prefixed by >>):
In file included from arch/x86/kernel/asm-offsets.c:14:0:
arch/x86/kernel/asm-offsets.c: In function 'common':
>> include/linux/compiler_types.h:129:35: error: 'struct tlb_state' has no member named 'asi_session'
#define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
^
include/linux/kbuild.h:6:62: note: in definition of macro 'DEFINE'
asm volatile("\n.ascii \"->" #sym " %0 " #val "\"" : : "i" (val))
^~~
>> include/linux/stddef.h:17:32: note: in expansion of macro '__compiler_offsetof'
#define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
^~~~~~~~~~~~~~~~~~~
>> include/linux/kbuild.h:11:14: note: in expansion of macro 'offsetof'
DEFINE(sym, offsetof(struct str, mem))
^~~~~~~~
>> arch/x86/kernel/asm-offsets.c:98:2: note: in expansion of macro 'OFFSET'
OFFSET(TLB_STATE_asi, tlb_state, asi_session.asi);
^~~~~~
>> include/linux/compiler_types.h:129:35: error: 'struct tlb_state' has no member named 'asi_session'
#define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
^
include/linux/kbuild.h:6:62: note: in definition of macro 'DEFINE'
asm volatile("\n.ascii \"->" #sym " %0 " #val "\"" : : "i" (val))
^~~
>> include/linux/stddef.h:17:32: note: in expansion of macro '__compiler_offsetof'
#define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
^~~~~~~~~~~~~~~~~~~
>> include/linux/kbuild.h:11:14: note: in expansion of macro 'offsetof'
DEFINE(sym, offsetof(struct str, mem))
^~~~~~~~
arch/x86/kernel/asm-offsets.c:99:2: note: in expansion of macro 'OFFSET'
OFFSET(TLB_STATE_asi_isolation_cr3, tlb_state,
^~~~~~
>> include/linux/compiler_types.h:129:35: error: 'struct tlb_state' has no member named 'asi_session'
#define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
^
include/linux/kbuild.h:6:62: note: in definition of macro 'DEFINE'
asm volatile("\n.ascii \"->" #sym " %0 " #val "\"" : : "i" (val))
^~~
>> include/linux/stddef.h:17:32: note: in expansion of macro '__compiler_offsetof'
#define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
^~~~~~~~~~~~~~~~~~~
>> include/linux/kbuild.h:11:14: note: in expansion of macro 'offsetof'
DEFINE(sym, offsetof(struct str, mem))
^~~~~~~~
arch/x86/kernel/asm-offsets.c:101:2: note: in expansion of macro 'OFFSET'
OFFSET(TLB_STATE_asi_original_cr3, tlb_state, asi_session.original_cr3);
^~~~~~
>> include/linux/compiler_types.h:129:35: error: 'struct tlb_state' has no member named 'asi_session'
#define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
^
include/linux/kbuild.h:6:62: note: in definition of macro 'DEFINE'
asm volatile("\n.ascii \"->" #sym " %0 " #val "\"" : : "i" (val))
^~~
>> include/linux/stddef.h:17:32: note: in expansion of macro '__compiler_offsetof'
#define offsetof(TYPE, MEMBER) __compiler_offsetof(TYPE, MEMBER)
^~~~~~~~~~~~~~~~~~~
>> include/linux/kbuild.h:11:14: note: in expansion of macro 'offsetof'
DEFINE(sym, offsetof(struct str, mem))
^~~~~~~~
arch/x86/kernel/asm-offsets.c:102:2: note: in expansion of macro 'OFFSET'
OFFSET(TLB_STATE_asi_idepth, tlb_state, asi_session.idepth);
^~~~~~
make[2]: *** [scripts/Makefile.build:99: arch/x86/kernel/asm-offsets.s] Error 1
make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1113: prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:179: sub-make] Error 2
30 real 5 user 5 sys 34.18% cpu make prepare
vim +129 include/linux/compiler_types.h
71391bdd2e9aab Xiaozhou Liu 2018-12-14 128
71391bdd2e9aab Xiaozhou Liu 2018-12-14 @129 #define __compiler_offsetof(a, b) __builtin_offsetof(a, b)
71391bdd2e9aab Xiaozhou Liu 2018-12-14 130
:::::: The code at line 129 was first introduced by commit
:::::: 71391bdd2e9aab188f86bf1ecd9b232531ec7eea include/linux/compiler_types.h: don't pollute userspace with macro definitions
:::::: TO: Xiaozhou Liu <liuxiaozhou@bytedance.com>
:::::: CC: Miguel Ojeda <miguel.ojeda.sandonis@gmail.com>
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 7284 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH v3 5/7] mm/asi: Switch ASI on task switch
2020-02-26 16:21 ` [RFC PATCH v3 5/7] mm/asi: Switch ASI on task switch Alexandre Chartre
@ 2020-02-26 20:50 ` kbuild test robot
2020-02-26 20:59 ` kbuild test robot
2020-02-28 12:44 ` kbuild test robot
2 siblings, 0 replies; 16+ messages in thread
From: kbuild test robot @ 2020-02-26 20:50 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 7099 bytes --]
Hi Alexandre,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/x86/core]
[also build test ERROR on tip/x86/asm tip/x86/mm v5.6-rc3]
[cannot apply to next-20200226]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Alexandre-Chartre/Kernel-Address-Space-Isolation/20200227-034033
base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 248ed51048c40d36728e70914e38bffd7821da57
config: um-x86_64_defconfig (attached as .config)
compiler: gcc-7 (Debian 7.5.0-5) 7.5.0
reproduce:
# save the attached .config to linux build tree
make ARCH=um SUBARCH=x86_64
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
kernel/sched/core.c: In function 'prepare_task_switch':
>> kernel/sched/core.c:3158:3: error: implicit declaration of function 'asi_schedule_out'; did you mean 'io_schedule'? [-Werror=implicit-function-declaration]
asi_schedule_out(prev);
^~~~~~~~~~~~~~~~
io_schedule
kernel/sched/core.c: In function 'finish_task_switch':
>> kernel/sched/core.c:3268:3: error: implicit declaration of function 'asi_schedule_in'; did you mean 'io_schedule'? [-Werror=implicit-function-declaration]
asi_schedule_in(current);
^~~~~~~~~~~~~~~
io_schedule
cc1: some warnings being treated as errors
vim +3158 kernel/sched/core.c
3139
3140 /**
3141 * prepare_task_switch - prepare to switch tasks
3142 * @rq: the runqueue preparing to switch
3143 * @prev: the current task that is being switched out
3144 * @next: the task we are going to switch to.
3145 *
3146 * This is called with the rq lock held and interrupts off. It must
3147 * be paired with a subsequent finish_task_switch after the context
3148 * switch.
3149 *
3150 * prepare_task_switch sets up locking and calls architecture specific
3151 * hooks.
3152 */
3153 static inline void
3154 prepare_task_switch(struct rq *rq, struct task_struct *prev,
3155 struct task_struct *next)
3156 {
3157 if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION))
> 3158 asi_schedule_out(prev);
3159
3160 kcov_prepare_switch(prev);
3161 sched_info_switch(rq, prev, next);
3162 perf_event_task_sched_out(prev, next);
3163 rseq_preempt(prev);
3164 fire_sched_out_preempt_notifiers(prev, next);
3165 prepare_task(next);
3166 prepare_arch_switch(next);
3167 }
3168
3169 /**
3170 * finish_task_switch - clean up after a task-switch
3171 * @prev: the thread we just switched away from.
3172 *
3173 * finish_task_switch must be called after the context switch, paired
3174 * with a prepare_task_switch call before the context switch.
3175 * finish_task_switch will reconcile locking set up by prepare_task_switch,
3176 * and do any other architecture-specific cleanup actions.
3177 *
3178 * Note that we may have delayed dropping an mm in context_switch(). If
3179 * so, we finish that here outside of the runqueue lock. (Doing it
3180 * with the lock held can cause deadlocks; see schedule() for
3181 * details.)
3182 *
3183 * The context switch have flipped the stack from under us and restored the
3184 * local variables which were saved when this task called schedule() in the
3185 * past. prev == current is still correct but we need to recalculate this_rq
3186 * because prev may have moved to another CPU.
3187 */
3188 static struct rq *finish_task_switch(struct task_struct *prev)
3189 __releases(rq->lock)
3190 {
3191 struct rq *rq = this_rq();
3192 struct mm_struct *mm = rq->prev_mm;
3193 long prev_state;
3194
3195 /*
3196 * The previous task will have left us with a preempt_count of 2
3197 * because it left us after:
3198 *
3199 * schedule()
3200 * preempt_disable(); // 1
3201 * __schedule()
3202 * raw_spin_lock_irq(&rq->lock) // 2
3203 *
3204 * Also, see FORK_PREEMPT_COUNT.
3205 */
3206 if (WARN_ONCE(preempt_count() != 2*PREEMPT_DISABLE_OFFSET,
3207 "corrupted preempt_count: %s/%d/0x%x\n",
3208 current->comm, current->pid, preempt_count()))
3209 preempt_count_set(FORK_PREEMPT_COUNT);
3210
3211 rq->prev_mm = NULL;
3212
3213 /*
3214 * A task struct has one reference for the use as "current".
3215 * If a task dies, then it sets TASK_DEAD in tsk->state and calls
3216 * schedule one last time. The schedule call will never return, and
3217 * the scheduled task must drop that reference.
3218 *
3219 * We must observe prev->state before clearing prev->on_cpu (in
3220 * finish_task), otherwise a concurrent wakeup can get prev
3221 * running on another CPU and we could rave with its RUNNING -> DEAD
3222 * transition, resulting in a double drop.
3223 */
3224 prev_state = prev->state;
3225 vtime_task_switch(prev);
3226 perf_event_task_sched_in(prev, current);
3227 finish_task(prev);
3228 finish_lock_switch(rq);
3229 finish_arch_post_lock_switch();
3230 kcov_finish_switch(current);
3231
3232 fire_sched_in_preempt_notifiers(current);
3233 /*
3234 * When switching through a kernel thread, the loop in
3235 * membarrier_{private,global}_expedited() may have observed that
3236 * kernel thread and not issued an IPI. It is therefore possible to
3237 * schedule between user->kernel->user threads without passing though
3238 * switch_mm(). Membarrier requires a barrier after storing to
3239 * rq->curr, before returning to userspace, so provide them here:
3240 *
3241 * - a full memory barrier for {PRIVATE,GLOBAL}_EXPEDITED, implicitly
3242 * provided by mmdrop(),
3243 * - a sync_core for SYNC_CORE.
3244 */
3245 if (mm) {
3246 membarrier_mm_sync_core_before_usermode(mm);
3247 mmdrop(mm);
3248 }
3249 if (unlikely(prev_state == TASK_DEAD)) {
3250 if (prev->sched_class->task_dead)
3251 prev->sched_class->task_dead(prev);
3252
3253 /*
3254 * Remove function-return probe instances associated with this
3255 * task and put them back on the free list.
3256 */
3257 kprobe_flush_task(prev);
3258
3259 /* Task is done with its stack. */
3260 put_task_stack(prev);
3261
3262 put_task_struct_rcu_user(prev);
3263 }
3264
3265 tick_nohz_task_switch();
3266
3267 if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION))
> 3268 asi_schedule_in(current);
3269
3270 return rq;
3271 }
3272
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 8440 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH v3 5/7] mm/asi: Switch ASI on task switch
2020-02-26 16:21 ` [RFC PATCH v3 5/7] mm/asi: Switch ASI on task switch Alexandre Chartre
2020-02-26 20:50 ` kbuild test robot
@ 2020-02-26 20:59 ` kbuild test robot
2020-02-28 12:44 ` kbuild test robot
2 siblings, 0 replies; 16+ messages in thread
From: kbuild test robot @ 2020-02-26 20:59 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 2026 bytes --]
Hi Alexandre,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/x86/core]
[also build test ERROR on tip/x86/asm tip/x86/mm v5.6-rc3]
[cannot apply to next-20200226]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Alexandre-Chartre/Kernel-Address-Space-Isolation/20200227-034033
base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 248ed51048c40d36728e70914e38bffd7821da57
config: nds32-defconfig (attached as .config)
compiler: nds32le-linux-gcc (GCC) 9.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross
chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
GCC_VERSION=9.2.0 make.cross ARCH=nds32
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
In file included from arch/nds32/kernel/asm-offsets.c:4:
>> include/linux/sched.h:13:10: fatal error: asm/asi_session.h: No such file or directory
13 | #include <asm/asi_session.h>
| ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [scripts/Makefile.build:99: arch/nds32/kernel/asm-offsets.s] Error 1
make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1113: prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:179: sub-make] Error 2
28 real 5 user 8 sys 50.72% cpu make prepare
vim +13 include/linux/sched.h
11
12 #include <asm/current.h>
> 13 #include <asm/asi_session.h>
14
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 10731 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH v3 2/7] mm/asi: ASI entry/exit interface
2020-02-26 16:21 ` [RFC PATCH v3 2/7] mm/asi: ASI entry/exit interface Alexandre Chartre
2020-02-26 20:38 ` kbuild test robot
@ 2020-02-26 22:02 ` Ira Weiny
2020-02-27 8:51 ` Alexandre Chartre
1 sibling, 1 reply; 16+ messages in thread
From: Ira Weiny @ 2020-02-26 22:02 UTC (permalink / raw)
To: Alexandre Chartre
Cc: rkrcmar, tglx, mingo, bp, hpa, dave.hansen, luto, peterz, x86,
linux-mm, linux-kernel, pbonzini, konrad.wilk, jan.setjeeilers,
liran.alon, junaids, graf, rppt, kuzuno, mgross
On Wed, Feb 26, 2020 at 05:21:55PM +0100, Alexandre Chartre wrote:
> Address Space Isolation (ASI) is entered by calling asi_enter() which
> switches the kernel page-table to the ASI page-table. Isolation is then
> exited by calling asi_exit() which switches the page-table back to the
> original kernel page-table.
>
> The ASI being used and its state is tracked in a per-cpu ASI session
> structure (struct asi_session).
>
> Signed-off-by: Alexandre Chartre <alexandre.chartre@oracle.com>
> ---
> arch/x86/include/asm/asi.h | 4 ++
> arch/x86/include/asm/asi_session.h | 17 +++++++
> arch/x86/include/asm/mmu_context.h | 19 +++++++-
> arch/x86/include/asm/tlbflush.h | 12 +++++
> arch/x86/mm/asi.c | 90 ++++++++++++++++++++++++++++++++++++
> 5 files changed, 140 insertions(+), 2 deletions(-)
> create mode 100644 arch/x86/include/asm/asi_session.h
>
> diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h
> index 844a81f..29b745a 100644
> --- a/arch/x86/include/asm/asi.h
> +++ b/arch/x86/include/asm/asi.h
> @@ -44,6 +44,8 @@
>
> #include <linux/export.h>
>
> +#include <asm/asi_session.h>
> +
> struct asi_type {
> int pcid_prefix; /* PCID prefix */
> };
> @@ -80,6 +82,8 @@ struct asi {
> extern struct asi *asi_create(struct asi_type *type);
> extern void asi_destroy(struct asi *asi);
> extern void asi_set_pagetable(struct asi *asi, pgd_t *pagetable);
> +extern int asi_enter(struct asi *asi);
> +extern void asi_exit(struct asi *asi);
>
> #endif /* __ASSEMBLY__ */
>
> diff --git a/arch/x86/include/asm/asi_session.h b/arch/x86/include/asm/asi_session.h
> new file mode 100644
> index 0000000..9d39c93
> --- /dev/null
> +++ b/arch/x86/include/asm/asi_session.h
> @@ -0,0 +1,17 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef ARCH_X86_MM_ASI_SESSION_H
> +#define ARCH_X86_MM_ASI_SESSION_H
> +
> +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
> +
> +struct asi;
> +
> +struct asi_session {
> + struct asi *asi; /* ASI for this session */
> + unsigned long isolation_cr3; /* cr3 when ASI is active */
> + unsigned long original_cr3; /* cr3 before entering ASI */
> +};
> +
> +#endif /* CONFIG_ADDRESS_SPACE_ISOLATION */
> +
> +#endif
> diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h
> index 5f33924..2d65443 100644
> --- a/arch/x86/include/asm/mmu_context.h
> +++ b/arch/x86/include/asm/mmu_context.h
> @@ -14,6 +14,7 @@
> #include <asm/paravirt.h>
> #include <asm/mpx.h>
> #include <asm/debugreg.h>
> +#include <asm/asi.h>
>
> extern atomic64_t last_mm_ctx_id;
>
> @@ -349,8 +350,22 @@ static inline bool arch_vma_access_permitted(struct vm_area_struct *vma,
> */
> static inline unsigned long __get_current_cr3_fast(void)
> {
> - unsigned long cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd,
> - this_cpu_read(cpu_tlbstate.loaded_mm_asid));
> + unsigned long cr3;
> +
> + /*
> + * If isolation is active then we need to return the CR3 for the
> + * currently active ASI. This value is stored in the isolation_cr3
> + * field of the ASI session.
> + */
> + if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION) &&
> + this_cpu_read(cpu_asi_session.asi)) {
> + cr3 = this_cpu_read(cpu_asi_session.isolation_cr3);
> + /* CR3 read never returns with the NOFLUSH bit */
> + cr3 &= ~X86_CR3_PCID_NOFLUSH;
> + } else {
> + cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd,
> + this_cpu_read(cpu_tlbstate.loaded_mm_asid));
> + }
>
> /* For now, be very restrictive about when this can be called. */
> VM_WARN_ON(in_nmi() || preemptible());
> diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h
> index 6f66d84..241058f 100644
> --- a/arch/x86/include/asm/tlbflush.h
> +++ b/arch/x86/include/asm/tlbflush.h
> @@ -12,6 +12,7 @@
> #include <asm/invpcid.h>
> #include <asm/pti.h>
> #include <asm/processor-flags.h>
> +#include <asm/asi.h>
>
> /*
> * The x86 feature is called PCID (Process Context IDentifier). It is similar
> @@ -239,9 +240,20 @@ struct tlb_state {
> * context 0.
> */
> struct tlb_context ctxs[TLB_NR_DYN_ASIDS];
> +
> +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
> + /*
> + * The ASI session tracks the ASI being used and its state.
> + */
> + struct asi_session asi_session;
> +#endif
> };
> DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_state, cpu_tlbstate);
>
> +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
> +#define cpu_asi_session (cpu_tlbstate.asi_session)
> +#endif
> +
> /*
> * Blindly accessing user memory from NMI context can be dangerous
> * if we're in the middle of switching the current user task or
> diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c
> index 0a0ac9d..9fbc921 100644
> --- a/arch/x86/mm/asi.c
> +++ b/arch/x86/mm/asi.c
> @@ -10,6 +10,8 @@
>
> #include <asm/asi.h>
> #include <asm/bug.h>
> +#include <asm/mmu_context.h>
> +#include <asm/tlbflush.h>
>
> struct asi *asi_create(struct asi_type *type)
> {
> @@ -58,3 +60,91 @@ void asi_set_pagetable(struct asi *asi, pgd_t *pagetable)
>
> }
> EXPORT_SYMBOL(asi_set_pagetable);
> +
> +static void asi_switch_to_asi_cr3(struct asi *asi)
> +{
> + unsigned long original_cr3, asi_cr3;
> + struct asi_session *asi_session;
> + u16 pcid;
> +
> + WARN_ON(!irqs_disabled());
> +
> + original_cr3 = __get_current_cr3_fast();
> +
> + /* build the ASI cr3 value */
> + asi_cr3 = asi->base_cr3;
> + if (boot_cpu_has(X86_FEATURE_PCID)) {
> + pcid = original_cr3 & ASI_KERNEL_PCID_MASK;
> + asi_cr3 |= pcid;
> + }
> +
> + /* get the ASI session ready for entering ASI */
> + asi_session = &get_cpu_var(cpu_asi_session);
> + asi_session->asi = asi;
> + asi_session->original_cr3 = original_cr3;
> + asi_session->isolation_cr3 = asi_cr3;
> +
> + /* Update CR3 to immediately enter ASI */
> + native_write_cr3(asi_cr3);
> +}
> +
> +static void asi_switch_to_kernel_cr3(struct asi *asi)
asi is not used in this function?
> +{
> + struct asi_session *asi_session;
> + unsigned long original_cr3;
> +
> + WARN_ON(!irqs_disabled());
> +
> + original_cr3 = this_cpu_read(cpu_asi_session.original_cr3);
> + if (boot_cpu_has(X86_FEATURE_PCID))
> + original_cr3 |= X86_CR3_PCID_NOFLUSH;
> + native_write_cr3(original_cr3);
> +
> + asi_session = &get_cpu_var(cpu_asi_session);
> + asi_session->asi = NULL;
> +}
> +
> +int asi_enter(struct asi *asi)
> +{
> + struct asi *current_asi;
> + unsigned long flags;
> +
> + /*
> + * We can re-enter isolation, but only with the same ASI (we don't
> + * support nesting isolation).
> + */
> + current_asi = this_cpu_read(cpu_asi_session.asi);
> + if (current_asi) {
> + if (current_asi != asi) {
> + WARN_ON(1);
> + return -EBUSY;
> + }
if (WARN_ON(current_asi != asi)) {
return -EBUSY;
???
Ira
> + return 0;
> + }
> +
> + local_irq_save(flags);
> + asi_switch_to_asi_cr3(asi);
> + local_irq_restore(flags);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL(asi_enter);
> +
> +void asi_exit(struct asi *asi)
> +{
> + struct asi *current_asi;
> + unsigned long flags;
> +
> + current_asi = this_cpu_read(cpu_asi_session.asi);
> + if (!current_asi) {
> + /* already exited */
> + return;
> + }
> +
> + WARN_ON(current_asi != asi);
> +
> + local_irq_save(flags);
> + asi_switch_to_kernel_cr3(asi);
> + local_irq_restore(flags);
> +}
> +EXPORT_SYMBOL(asi_exit);
> --
> 1.7.1
>
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH v3 2/7] mm/asi: ASI entry/exit interface
2020-02-26 22:02 ` Ira Weiny
@ 2020-02-27 8:51 ` Alexandre Chartre
0 siblings, 0 replies; 16+ messages in thread
From: Alexandre Chartre @ 2020-02-27 8:51 UTC (permalink / raw)
To: Ira Weiny
Cc: rkrcmar, tglx, mingo, bp, hpa, dave.hansen, luto, peterz, x86,
linux-mm, linux-kernel, pbonzini, konrad.wilk, jan.setjeeilers,
liran.alon, junaids, graf, rppt, kuzuno, mgross
On 2/26/20 11:02 PM, Ira Weiny wrote:
> On Wed, Feb 26, 2020 at 05:21:55PM +0100, Alexandre Chartre wrote:
>> Address Space Isolation (ASI) is entered by calling asi_enter() which
>> switches the kernel page-table to the ASI page-table. Isolation is then
>> exited by calling asi_exit() which switches the page-table back to the
>> original kernel page-table.
>>
>> The ASI being used and its state is tracked in a per-cpu ASI session
>> structure (struct asi_session).
>>
>> Signed-off-by: Alexandre Chartre <alexandre.chartre@oracle.com>
>> ---
>> arch/x86/include/asm/asi.h | 4 ++
>> arch/x86/include/asm/asi_session.h | 17 +++++++
>> arch/x86/include/asm/mmu_context.h | 19 +++++++-
>> arch/x86/include/asm/tlbflush.h | 12 +++++
>> arch/x86/mm/asi.c | 90 ++++++++++++++++++++++++++++++++++++
>> 5 files changed, 140 insertions(+), 2 deletions(-)
>> create mode 100644 arch/x86/include/asm/asi_session.h
>>
>> diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h
>> index 844a81f..29b745a 100644
>> --- a/arch/x86/include/asm/asi.h
>> +++ b/arch/x86/include/asm/asi.h
>> @@ -44,6 +44,8 @@
>>
>> #include <linux/export.h>
>>
>> +#include <asm/asi_session.h>
>> +
>> struct asi_type {
>> int pcid_prefix; /* PCID prefix */
>> };
>> @@ -80,6 +82,8 @@ struct asi {
>> extern struct asi *asi_create(struct asi_type *type);
>> extern void asi_destroy(struct asi *asi);
>> extern void asi_set_pagetable(struct asi *asi, pgd_t *pagetable);
>> +extern int asi_enter(struct asi *asi);
>> +extern void asi_exit(struct asi *asi);
>>
>> #endif /* __ASSEMBLY__ */
>>
>> diff --git a/arch/x86/include/asm/asi_session.h b/arch/x86/include/asm/asi_session.h
>> new file mode 100644
>> index 0000000..9d39c93
>> --- /dev/null
>> +++ b/arch/x86/include/asm/asi_session.h
>> @@ -0,0 +1,17 @@
>> +/* SPDX-License-Identifier: GPL-2.0 */
>> +#ifndef ARCH_X86_MM_ASI_SESSION_H
>> +#define ARCH_X86_MM_ASI_SESSION_H
>> +
>> +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
>> +
>> +struct asi;
>> +
>> +struct asi_session {
>> + struct asi *asi; /* ASI for this session */
>> + unsigned long isolation_cr3; /* cr3 when ASI is active */
>> + unsigned long original_cr3; /* cr3 before entering ASI */
>> +};
>> +
>> +#endif /* CONFIG_ADDRESS_SPACE_ISOLATION */
>> +
>> +#endif
>> diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h
>> index 5f33924..2d65443 100644
>> --- a/arch/x86/include/asm/mmu_context.h
>> +++ b/arch/x86/include/asm/mmu_context.h
>> @@ -14,6 +14,7 @@
>> #include <asm/paravirt.h>
>> #include <asm/mpx.h>
>> #include <asm/debugreg.h>
>> +#include <asm/asi.h>
>>
>> extern atomic64_t last_mm_ctx_id;
>>
>> @@ -349,8 +350,22 @@ static inline bool arch_vma_access_permitted(struct vm_area_struct *vma,
>> */
>> static inline unsigned long __get_current_cr3_fast(void)
>> {
>> - unsigned long cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd,
>> - this_cpu_read(cpu_tlbstate.loaded_mm_asid));
>> + unsigned long cr3;
>> +
>> + /*
>> + * If isolation is active then we need to return the CR3 for the
>> + * currently active ASI. This value is stored in the isolation_cr3
>> + * field of the ASI session.
>> + */
>> + if (IS_ENABLED(CONFIG_ADDRESS_SPACE_ISOLATION) &&
>> + this_cpu_read(cpu_asi_session.asi)) {
>> + cr3 = this_cpu_read(cpu_asi_session.isolation_cr3);
>> + /* CR3 read never returns with the NOFLUSH bit */
>> + cr3 &= ~X86_CR3_PCID_NOFLUSH;
>> + } else {
>> + cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd,
>> + this_cpu_read(cpu_tlbstate.loaded_mm_asid));
>> + }
>>
>> /* For now, be very restrictive about when this can be called. */
>> VM_WARN_ON(in_nmi() || preemptible());
>> diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h
>> index 6f66d84..241058f 100644
>> --- a/arch/x86/include/asm/tlbflush.h
>> +++ b/arch/x86/include/asm/tlbflush.h
>> @@ -12,6 +12,7 @@
>> #include <asm/invpcid.h>
>> #include <asm/pti.h>
>> #include <asm/processor-flags.h>
>> +#include <asm/asi.h>
>>
>> /*
>> * The x86 feature is called PCID (Process Context IDentifier). It is similar
>> @@ -239,9 +240,20 @@ struct tlb_state {
>> * context 0.
>> */
>> struct tlb_context ctxs[TLB_NR_DYN_ASIDS];
>> +
>> +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
>> + /*
>> + * The ASI session tracks the ASI being used and its state.
>> + */
>> + struct asi_session asi_session;
>> +#endif
>> };
>> DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_state, cpu_tlbstate);
>>
>> +#ifdef CONFIG_ADDRESS_SPACE_ISOLATION
>> +#define cpu_asi_session (cpu_tlbstate.asi_session)
>> +#endif
>> +
>> /*
>> * Blindly accessing user memory from NMI context can be dangerous
>> * if we're in the middle of switching the current user task or
>> diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c
>> index 0a0ac9d..9fbc921 100644
>> --- a/arch/x86/mm/asi.c
>> +++ b/arch/x86/mm/asi.c
>> @@ -10,6 +10,8 @@
>>
>> #include <asm/asi.h>
>> #include <asm/bug.h>
>> +#include <asm/mmu_context.h>
>> +#include <asm/tlbflush.h>
>>
>> struct asi *asi_create(struct asi_type *type)
>> {
>> @@ -58,3 +60,91 @@ void asi_set_pagetable(struct asi *asi, pgd_t *pagetable)
>>
>> }
>> EXPORT_SYMBOL(asi_set_pagetable);
>> +
>> +static void asi_switch_to_asi_cr3(struct asi *asi)
>> +{
>> + unsigned long original_cr3, asi_cr3;
>> + struct asi_session *asi_session;
>> + u16 pcid;
>> +
>> + WARN_ON(!irqs_disabled());
>> +
>> + original_cr3 = __get_current_cr3_fast();
>> +
>> + /* build the ASI cr3 value */
>> + asi_cr3 = asi->base_cr3;
>> + if (boot_cpu_has(X86_FEATURE_PCID)) {
>> + pcid = original_cr3 & ASI_KERNEL_PCID_MASK;
>> + asi_cr3 |= pcid;
>> + }
>> +
>> + /* get the ASI session ready for entering ASI */
>> + asi_session = &get_cpu_var(cpu_asi_session);
>> + asi_session->asi = asi;
>> + asi_session->original_cr3 = original_cr3;
>> + asi_session->isolation_cr3 = asi_cr3;
>> +
>> + /* Update CR3 to immediately enter ASI */
>> + native_write_cr3(asi_cr3);
>> +}
>> +
>> +static void asi_switch_to_kernel_cr3(struct asi *asi)
>
> asi is not used in this function?
>
You are right, I was probably using it in a previous version of the code
and forgot to remote it. I will remove it.
>> +{
>> + struct asi_session *asi_session;
>> + unsigned long original_cr3;
>> +
>> + WARN_ON(!irqs_disabled());
>> +
>> + original_cr3 = this_cpu_read(cpu_asi_session.original_cr3);
>> + if (boot_cpu_has(X86_FEATURE_PCID))
>> + original_cr3 |= X86_CR3_PCID_NOFLUSH;
>> + native_write_cr3(original_cr3);
>> +
>> + asi_session = &get_cpu_var(cpu_asi_session);
>> + asi_session->asi = NULL;
>> +}
>> +
>> +int asi_enter(struct asi *asi)
>> +{
>> + struct asi *current_asi;
>> + unsigned long flags;
>> +
>> + /*
>> + * We can re-enter isolation, but only with the same ASI (we don't
>> + * support nesting isolation).
>> + */
>> + current_asi = this_cpu_read(cpu_asi_session.asi);
>> + if (current_asi) {
>> + if (current_asi != asi) {
>> + WARN_ON(1);
>> + return -EBUSY;
>> + }
>
> if (WARN_ON(current_asi != asi)) {
> return -EBUSY;
>
Yes, much better, will change.
Thanks,
alex.
>
>> + return 0;
>> + }
>> +
>> + local_irq_save(flags);
>> + asi_switch_to_asi_cr3(asi);
>> + local_irq_restore(flags);
>> +
>> + return 0;
>> +}
>> +EXPORT_SYMBOL(asi_enter);
>> +
>> +void asi_exit(struct asi *asi)
>> +{
>> + struct asi *current_asi;
>> + unsigned long flags;
>> +
>> + current_asi = this_cpu_read(cpu_asi_session.asi);
>> + if (!current_asi) {
>> + /* already exited */
>> + return;
>> + }
>> +
>> + WARN_ON(current_asi != asi);
>> +
>> + local_irq_save(flags);
>> + asi_switch_to_kernel_cr3(asi);
>> + local_irq_restore(flags);
>> +}
>> +EXPORT_SYMBOL(asi_exit);
>> --
>> 1.7.1
>>
>>
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [RFC PATCH v3 5/7] mm/asi: Switch ASI on task switch
2020-02-26 16:21 ` [RFC PATCH v3 5/7] mm/asi: Switch ASI on task switch Alexandre Chartre
2020-02-26 20:50 ` kbuild test robot
2020-02-26 20:59 ` kbuild test robot
@ 2020-02-28 12:44 ` kbuild test robot
2 siblings, 0 replies; 16+ messages in thread
From: kbuild test robot @ 2020-02-28 12:44 UTC (permalink / raw)
To: kbuild-all
[-- Attachment #1: Type: text/plain, Size: 2464 bytes --]
Hi Alexandre,
[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on tip/x86/core]
[also build test ERROR on tip/x86/asm tip/x86/mm v5.6-rc3]
[cannot apply to next-20200227]
[if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Alexandre-Chartre/Kernel-Address-Space-Isolation/20200227-034033
base: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git 248ed51048c40d36728e70914e38bffd7821da57
config: arm64-defconfig (attached as .config)
compiler: clang version 11.0.0 (git://gitmirror/llvm_project 949134e2fefd34a38ed71de90dffe2300e2e1139)
reproduce:
# FIXME the reproduce steps for clang is not ready yet
If you fix the issue, kindly add following tag
Reported-by: kbuild test robot <lkp@intel.com>
All errors (new ones prefixed by >>):
In file included from arch/arm64/kernel/asm-offsets.c:10:
In file included from include/linux/arm_sdei.h:8:
In file included from include/acpi/ghes.h:5:
In file included from include/acpi/apei.h:9:
In file included from include/linux/acpi.h:13:
In file included from include/linux/irqdomain.h:35:
In file included from include/linux/of.h:17:
In file included from include/linux/kobject.h:20:
In file included from include/linux/sysfs.h:22:
In file included from include/linux/stat.h:6:
In file included from arch/arm64/include/asm/stat.h:13:
In file included from arch/arm64/include/asm/compat.h:13:
>> include/linux/sched.h:13:10: fatal error: 'asm/asi_session.h' file not found
#include <asm/asi_session.h>
^~~~~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [scripts/Makefile.build:99: arch/arm64/kernel/asm-offsets.s] Error 1
make[2]: Target '__build' not remade because of errors.
make[1]: *** [Makefile:1113: prepare0] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:179: sub-make] Error 2
348 real 47 user 100 sys 42.47% cpu make prepare
vim +13 include/linux/sched.h
11
12 #include <asm/current.h>
> 13 #include <asm/asi_session.h>
14
---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 46438 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2020-02-28 12:44 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-02-26 16:21 [RFC PATCH v3 0/7] Kernel Address Space Isolation Alexandre Chartre
2020-02-26 16:21 ` [RFC PATCH v3 1/7] mm/x86: Introduce kernel Address Space Isolation (ASI) Alexandre Chartre
2020-02-26 16:21 ` [RFC PATCH v3 2/7] mm/asi: ASI entry/exit interface Alexandre Chartre
2020-02-26 20:38 ` kbuild test robot
2020-02-26 22:02 ` Ira Weiny
2020-02-27 8:51 ` Alexandre Chartre
2020-02-26 16:21 ` [RFC PATCH v3 3/7] mm/asi: Improve TLB flushing when switching to an ASI pagetable Alexandre Chartre
2020-02-26 16:21 ` [RFC PATCH v3 4/7] mm/asi: Interrupt ASI on interrupt/exception/NMI Alexandre Chartre
2020-02-26 20:38 ` kbuild test robot
2020-02-26 16:21 ` [RFC PATCH v3 5/7] mm/asi: Switch ASI on task switch Alexandre Chartre
2020-02-26 20:50 ` kbuild test robot
2020-02-26 20:59 ` kbuild test robot
2020-02-28 12:44 ` kbuild test robot
2020-02-26 16:21 ` [RFC PATCH v3 6/7] mm/asi: ASI fault handler Alexandre Chartre
2020-02-26 16:22 ` [RFC PATCH v3 7/7] mm/asi: Implement PTI with ASI Alexandre Chartre
2020-02-26 20:32 ` kbuild test robot
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.