[PATCH 0/3] series to optionally disable tlbie for 64s/radix

[PATCH 0/3] series to optionally disable tlbie for 64s/radix

[Nicholas Piggin]

Since the RFC I accounted for feedback, and also made the patch more
complete so we can actually boot and run a 64s/radix kernel without
using tlbie at all.

KVM and bare metal hash are harder to support. Bare metal hash because
it does some TLB invation with interrupts disabled (can't use IPIs).
It might be possible those invalidates could be avoided or the paths
changed to enable interrupts, but it would be a much bigger change.

KVM with radix guests might be acutally quite simple to support, so I
can look at that next if we get this merged.

KVM with hash guests might not be feasible to do in a performant way,
because of the design of virtualised hash architecture (host flushes
guest effective addresses), at least it would need special real mode
IPI handlers for TLB flushes.

Thanks,
Nick

Nicholas Piggin (3):
  powerpc/64s: Remove mmu_partition_table_set
  powerpc/64s/radix: all CPUs should flush local translation structure
    before turning MMU on
  powerpc/64s: introduce options to disable use of the tlbie instruction

 .../admin-guide/kernel-parameters.txt         |   4 +
 arch/powerpc/include/asm/book3s/64/tlbflush.h |   9 +
 arch/powerpc/include/asm/mmu.h                |   2 -
 arch/powerpc/kvm/book3s_hv.c                  |  10 +-
 arch/powerpc/kvm/book3s_hv_nested.c           |  35 +++-
 arch/powerpc/mm/book3s64/hash_utils.c         |   4 +-
 arch/powerpc/mm/book3s64/mmu_context.c        |   4 +-
 arch/powerpc/mm/book3s64/pgtable.c            |  77 +++++---
 arch/powerpc/mm/book3s64/radix_pgtable.c      |  22 +--
 arch/powerpc/mm/book3s64/radix_tlb.c          | 182 ++++++++++++++++--
 arch/powerpc/mm/pgtable_64.c                  |   5 +-
 drivers/misc/cxl/main.c                       |   4 +
 drivers/misc/ocxl/main.c                      |   4 +
 13 files changed, 282 insertions(+), 80 deletions(-)

-- 
2.22.0

[PATCH 1/3] powerpc/64s: Remove mmu_partition_table_set

[Nicholas Piggin]

This was an awkward interface that didn't quite suit bare metal boot,
because the host page table base is stored into the partition table at
a different place than the process table base, and the TLB flushing is
unnecessary at boot with the MMU off, provided all CPUs invalidate
all their translation structures before turning their MMU on.

So inline this into kvmhv_set_ptbl_entry, where it pairs up nicely
with the nested case.

Bare metal modifies the partition table directly, and the TLBIEs are
dropped because in HVMODE, all CPUs invalidate their translation
structures before enabling their MMU.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 arch/powerpc/include/asm/mmu.h           |  2 --
 arch/powerpc/kvm/book3s_hv.c             |  4 +--
 arch/powerpc/kvm/book3s_hv_nested.c      | 35 +++++++++++++++++++-----
 arch/powerpc/mm/book3s64/hash_utils.c    |  4 ++-
 arch/powerpc/mm/book3s64/mmu_context.c   |  4 +--
 arch/powerpc/mm/book3s64/pgtable.c       | 30 --------------------
 arch/powerpc/mm/book3s64/radix_pgtable.c | 11 +++-----
 arch/powerpc/mm/pgtable_64.c             |  5 ++--
 8 files changed, 42 insertions(+), 53 deletions(-)

diff --git a/arch/powerpc/include/asm/mmu.h b/arch/powerpc/include/asm/mmu.h
index ba94ce8c22d7..b973a708e8b4 100644
--- a/arch/powerpc/include/asm/mmu.h
+++ b/arch/powerpc/include/asm/mmu.h
@@ -256,8 +256,6 @@ extern void radix__mmu_cleanup_all(void);
 
 /* Functions for creating and updating partition table on POWER9 */
 extern void mmu_partition_table_init(void);
-extern void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,
-					  unsigned long dw1);
 #endif /* CONFIG_PPC64 */
 
 struct mm_struct;
diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index cde3f5a4b3e4..6ec2e4fe86d3 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c
@@ -4786,8 +4786,8 @@ static int kvmppc_core_init_vm_hv(struct kvm *kvm)
 	 * Since we don't flush the TLB when tearing down a VM,
 	 * and this lpid might have previously been used,
 	 * make sure we flush on each core before running the new VM.
-	 * On POWER9, the tlbie in mmu_partition_table_set_entry()
-	 * does this flush for us.
+	 * On POWER9, the flush setting the partition table entry
+	 * in kvmhv_set_ptbl_entry takes care of this.
 	 */
 	if (!cpu_has_feature(CPU_FTR_ARCH_300))
 		cpumask_setall(&kvm->arch.need_tlb_flush);
diff --git a/arch/powerpc/kvm/book3s_hv_nested.c b/arch/powerpc/kvm/book3s_hv_nested.c
index 735e0ac6f5b2..76f37dd8a327 100644
--- a/arch/powerpc/kvm/book3s_hv_nested.c
+++ b/arch/powerpc/kvm/book3s_hv_nested.c
@@ -411,14 +411,35 @@ static void kvmhv_flush_lpid(unsigned int lpid)
 void kvmhv_set_ptbl_entry(unsigned int lpid, u64 dw0, u64 dw1)
 {
 	if (!kvmhv_on_pseries()) {
-		mmu_partition_table_set_entry(lpid, dw0, dw1);
-		return;
-	}
+		unsigned long old = be64_to_cpu(partition_tb[lpid].patb0);
+
+		partition_tb[lpid].patb0 = cpu_to_be64(dw0);
+		partition_tb[lpid].patb1 = cpu_to_be64(dw1);
+
+		/*
+		 * Global flush of TLBs and partition table caches for this
+		 * lpid. The type of flush (hash or radix) depends on what the
+		 * previous use of this partition ID was, not the new use.
+		 */
+		asm volatile("ptesync" : : : "memory");
+		if (old & PATB_HR) {
+			asm volatile(PPC_TLBIE_5(%0,%1,2,0,1) : :
+				     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+			asm volatile(PPC_TLBIE_5(%0,%1,2,1,1) : :
+				     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		} else {
+			asm volatile(PPC_TLBIE_5(%0,%1,2,0,0) : :
+				     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
+		}
+		/* do we need fixup here ?*/
+		asm volatile("eieio; tlbsync; ptesync" : : : "memory");
 
-	pseries_partition_tb[lpid].patb0 = cpu_to_be64(dw0);
-	pseries_partition_tb[lpid].patb1 = cpu_to_be64(dw1);
-	/* L0 will do the necessary barriers */
-	kvmhv_flush_lpid(lpid);
+	} else {
+		pseries_partition_tb[lpid].patb0 = cpu_to_be64(dw0);
+		pseries_partition_tb[lpid].patb1 = cpu_to_be64(dw1);
+		/* L0 will do the necessary barriers */
+		kvmhv_flush_lpid(lpid);
+	}
 }
 
 static void kvmhv_set_nested_ptbl(struct kvm_nested_guest *gp)
diff --git a/arch/powerpc/mm/book3s64/hash_utils.c b/arch/powerpc/mm/book3s64/hash_utils.c
index b8ad14bb1170..e19bc3a5d9a5 100644
--- a/arch/powerpc/mm/book3s64/hash_utils.c
+++ b/arch/powerpc/mm/book3s64/hash_utils.c
@@ -823,7 +823,9 @@ static void __init hash_init_partition_table(phys_addr_t hash_table,
 	 * For now, UPRT is 0 and we have no segment table.
 	 */
 	htab_size =  __ilog2(htab_size) - 18;
-	mmu_partition_table_set_entry(0, hash_table | htab_size, 0);
+	partition_tb[0].patb0 = cpu_to_be64(hash_table | htab_size);
+	partition_tb[0].patb1 = 0;
+
 	pr_info("Partition table %p\n", partition_tb);
 }
 
diff --git a/arch/powerpc/mm/book3s64/mmu_context.c b/arch/powerpc/mm/book3s64/mmu_context.c
index 2d0cb5ba9a47..7344bea5d243 100644
--- a/arch/powerpc/mm/book3s64/mmu_context.c
+++ b/arch/powerpc/mm/book3s64/mmu_context.c
@@ -276,8 +276,8 @@ void arch_exit_mmap(struct mm_struct *mm)
 		 * and 0 is invalid. So this will do.
 		 *
 		 * This runs before the "fullmm" tlb flush in exit_mmap,
-		 * which does a RIC=2 tlbie to clear the process table
-		 * entry. See the "fullmm" comments in tlb-radix.c.
+		 * which does a RIC=2 tlb invalidate to clear the process
+		 * table entry. See the "fullmm" comments in tlb-radix.c.
 		 *
 		 * No barrier required here after the store because
 		 * this process will do the invalidate, which starts with
diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
index 7d0e0d0d22c4..bbedd7f7ab3f 100644
--- a/arch/powerpc/mm/book3s64/pgtable.c
+++ b/arch/powerpc/mm/book3s64/pgtable.c
@@ -209,36 +209,6 @@ void __init mmu_partition_table_init(void)
 	powernv_set_nmmu_ptcr(ptcr);
 }
 
-void mmu_partition_table_set_entry(unsigned int lpid, unsigned long dw0,
-				   unsigned long dw1)
-{
-	unsigned long old = be64_to_cpu(partition_tb[lpid].patb0);
-
-	partition_tb[lpid].patb0 = cpu_to_be64(dw0);
-	partition_tb[lpid].patb1 = cpu_to_be64(dw1);
-
-	/*
-	 * Global flush of TLBs and partition table caches for this lpid.
-	 * The type of flush (hash or radix) depends on what the previous
-	 * use of this partition ID was, not the new use.
-	 */
-	asm volatile("ptesync" : : : "memory");
-	if (old & PATB_HR) {
-		asm volatile(PPC_TLBIE_5(%0,%1,2,0,1) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		asm volatile(PPC_TLBIE_5(%0,%1,2,1,1) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 1);
-	} else {
-		asm volatile(PPC_TLBIE_5(%0,%1,2,0,0) : :
-			     "r" (TLBIEL_INVAL_SET_LPID), "r" (lpid));
-		trace_tlbie(lpid, 0, TLBIEL_INVAL_SET_LPID, lpid, 2, 0, 0);
-	}
-	/* do we need fixup here ?*/
-	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
-}
-EXPORT_SYMBOL_GPL(mmu_partition_table_set_entry);
-
 static pmd_t *get_pmd_from_cache(struct mm_struct *mm)
 {
 	void *pmd_frag, *ret;
diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c
index b4ca9e95e678..d60cfa05447a 100644
--- a/arch/powerpc/mm/book3s64/radix_pgtable.c
+++ b/arch/powerpc/mm/book3s64/radix_pgtable.c
@@ -36,12 +36,7 @@ unsigned int mmu_base_pid;
 static int native_register_process_table(unsigned long base, unsigned long pg_sz,
 					 unsigned long table_size)
 {
-	unsigned long patb0, patb1;
-
-	patb0 = be64_to_cpu(partition_tb[0].patb0);
-	patb1 = base | table_size | PATB_GR;
-
-	mmu_partition_table_set_entry(0, patb0, patb1);
+	partition_tb[0].patb1 = cpu_to_be64(base | table_size | PATB_GR);
 
 	return 0;
 }
@@ -417,7 +412,9 @@ static void __init radix_init_partition_table(void)
 	mmu_partition_table_init();
 	rts_field = radix__get_tree_size();
 	dw0 = rts_field | __pa(init_mm.pgd) | RADIX_PGD_INDEX_SIZE | PATB_HR;
-	mmu_partition_table_set_entry(0, dw0, 0);
+
+	partition_tb[0].patb0 = cpu_to_be64(dw0);
+	partition_tb[0].patb1 = 0;
 
 	pr_info("Initializing Radix MMU\n");
 	pr_info("Partition table %p\n", partition_tb);
diff --git a/arch/powerpc/mm/pgtable_64.c b/arch/powerpc/mm/pgtable_64.c
index 9ad59b733984..5bd008e6eb41 100644
--- a/arch/powerpc/mm/pgtable_64.c
+++ b/arch/powerpc/mm/pgtable_64.c
@@ -54,8 +54,9 @@
 /*
  * partition table and process table for ISA 3.0
  */
-struct prtb_entry *process_tb;
-struct patb_entry *partition_tb;
+struct prtb_entry *process_tb __read_mostly;
+struct patb_entry *partition_tb __read_mostly;
+EXPORT_SYMBOL_GPL(partition_tb); /* KVM only */
 /*
  * page table size
  */
-- 
2.22.0

[PATCH 2/3] powerpc/64s/radix: all CPUs should flush local translation structure before turning MMU on

[Nicholas Piggin]

Rather than sprinkle various translation structure invalidations
around different places in early boot, have each CPU flush everything
from its local translation structures before enabling its MMU.

Radix guests can execute tlbie(l), so have them tlbiel_all in the same
place as radix host does.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 arch/powerpc/mm/book3s64/radix_pgtable.c | 11 ++---------
 1 file changed, 2 insertions(+), 9 deletions(-)

diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c
index d60cfa05447a..839e01795211 100644
--- a/arch/powerpc/mm/book3s64/radix_pgtable.c
+++ b/arch/powerpc/mm/book3s64/radix_pgtable.c
@@ -382,11 +382,6 @@ static void __init radix_init_pgtable(void)
 	 */
 	register_process_table(__pa(process_tb), 0, PRTB_SIZE_SHIFT - 12);
 	pr_info("Process table %p and radix root for kernel: %p\n", process_tb, init_mm.pgd);
-	asm volatile("ptesync" : : : "memory");
-	asm volatile(PPC_TLBIE_5(%0,%1,2,1,1) : :
-		     "r" (TLBIEL_INVAL_SET_LPID), "r" (0));
-	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
-	trace_tlbie(0, 0, TLBIEL_INVAL_SET_LPID, 0, 2, 1, 1);
 
 	/*
 	 * The init_mm context is given the first available (non-zero) PID,
@@ -633,8 +628,7 @@ void __init radix__early_init_mmu(void)
 	radix_init_pgtable();
 	/* Switch to the guard PID before turning on MMU */
 	radix__switch_mmu_context(NULL, &init_mm);
-	if (cpu_has_feature(CPU_FTR_HVMODE))
-		tlbiel_all();
+	tlbiel_all();
 }
 
 void radix__early_init_mmu_secondary(void)
@@ -653,8 +647,7 @@ void radix__early_init_mmu_secondary(void)
 	}
 
 	radix__switch_mmu_context(NULL, &init_mm);
-	if (cpu_has_feature(CPU_FTR_HVMODE))
-		tlbiel_all();
+	tlbiel_all();
 }
 
 void radix__mmu_cleanup_all(void)
-- 
2.22.0

[PATCH 3/3] powerpc/64s: introduce options to disable use of the tlbie instruction

[Nicholas Piggin]

Introduce two options to control the use of the tlbie instruction. A
boot time option which completely disables the kernel using the
instruction, this is currently incompatible with HASH MMU, KVM, and
coherent accelerators.

And a debugfs option can be switched at runtime and avoids using tlbie
for invalidating CPU TLBs for normal process and kernel address
mappings. Coherent accelerators are still managed with tlbie, as will
KVM partition scope translations.

Cross-CPU TLB flushing is implemented with IPIs and tlbiel. This is a
basic implementation which does not attempt to make any optimisation
beyond the tlbie implementation.

This is useful for performance testing among other things. For example
in certain situations on large systems, using IPIs may be faster than
tlbie as they can be directed rather than broadcast. Later we may also
take advantage of the IPIs to do more interesting things such as trim
the mm cpumask more aggressively.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 .../admin-guide/kernel-parameters.txt         |   4 +
 arch/powerpc/include/asm/book3s/64/tlbflush.h |   9 +
 arch/powerpc/kvm/book3s_hv.c                  |   6 +
 arch/powerpc/mm/book3s64/pgtable.c            |  47 +++++
 arch/powerpc/mm/book3s64/radix_tlb.c          | 182 ++++++++++++++++--
 drivers/misc/cxl/main.c                       |   4 +
 drivers/misc/ocxl/main.c                      |   4 +
 7 files changed, 238 insertions(+), 18 deletions(-)

diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
index 7ccd158b3894..5e5206acc64e 100644
--- a/Documentation/admin-guide/kernel-parameters.txt
+++ b/Documentation/admin-guide/kernel-parameters.txt
@@ -860,6 +860,10 @@
 	disable_radix	[PPC]
 			Disable RADIX MMU mode on POWER9
 
+	disable_tlbie	[PPC]
+			Disable TLBIE instruction. Currently does not work
+			with KVM, with HASH MMU, or with coherent accelerators.
+
 	disable_cpu_apicid= [X86,APIC,SMP]
 			Format: <int>
 			The number of initial APIC ID for the
diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h
index ebf572ea621e..4b4568b7d4f2 100644
--- a/arch/powerpc/include/asm/book3s/64/tlbflush.h
+++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h
@@ -162,4 +162,13 @@ static inline void flush_tlb_pgtable(struct mmu_gather *tlb, unsigned long addre
 
 	radix__flush_tlb_pwc(tlb, address);
 }
+
+extern bool tlbie_capable;
+extern bool tlbie_enabled;
+
+static inline bool use_tlbie(void)
+{
+	return tlbie_enabled;
+}
+
 #endif /*  _ASM_POWERPC_BOOK3S_64_TLBFLUSH_H */
diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index 6ec2e4fe86d3..886d92639f1e 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c
@@ -5462,6 +5462,12 @@ static int kvmppc_radix_possible(void)
 static int kvmppc_book3s_init_hv(void)
 {
 	int r;
+
+	if (!tlbie_capable) {
+		pr_err("KVM-HV: Host does not support TLBIE\n");
+		return -ENODEV;
+	}
+
 	/*
 	 * FIXME!! Do we need to check on all cpus ?
 	 */
diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
index bbedd7f7ab3f..73ba67d6ca37 100644
--- a/arch/powerpc/mm/book3s64/pgtable.c
+++ b/arch/powerpc/mm/book3s64/pgtable.c
@@ -8,6 +8,7 @@
 #include <linux/memblock.h>
 #include <misc/cxl-base.h>
 
+#include <asm/debugfs.h>
 #include <asm/pgalloc.h>
 #include <asm/tlb.h>
 #include <asm/trace.h>
@@ -437,3 +438,49 @@ int ioremap_range(unsigned long ea, phys_addr_t pa, unsigned long size, pgprot_t
 
 	return 0;
 }
+
+/*
+ * Can tlbie be used.
+ */
+bool tlbie_capable __read_mostly = true;
+EXPORT_SYMBOL(tlbie_capable);
+
+/*
+ * Avoid using tlbie for management of CPU TLBs for kernel and process address
+ * spaces. tlbie may still be used for accelerators, and by KVM.
+ */
+bool tlbie_enabled __read_mostly = true;
+
+
+static int __init setup_disable_tlbie(char *str)
+{
+	if (!radix_enabled()) {
+		pr_err("disable_tlbie: Unable to disable TLBIE with Hash MMU.\n");
+		return 1;
+	}
+
+	tlbie_capable = false;
+	tlbie_enabled = false;
+
+        return 1;
+}
+__setup("disable_tlbie", setup_disable_tlbie);
+
+static int __init pgtable_debugfs_setup(void)
+{
+	if (!tlbie_capable)
+		return 0;
+
+	/*
+	 * There is no locking vs tlb flushing when changing this value.
+	 * The tlb flushers will see one value or another, and use either
+	 * tlbie or tlbiel with IPIs. In both cases the TLBs will be
+	 * invalidated as expected.
+	 */
+	debugfs_create_bool("tlbie_enabled", 0600,
+			powerpc_debugfs_root,
+			&tlbie_enabled);
+
+	return 0;
+}
+arch_initcall(pgtable_debugfs_setup);
diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c
index 71f7fede2fa4..7d17564f12c1 100644
--- a/arch/powerpc/mm/book3s64/radix_tlb.c
+++ b/arch/powerpc/mm/book3s64/radix_tlb.c
@@ -285,6 +285,39 @@ static inline void _tlbie_pid(unsigned long pid, unsigned long ric)
 	asm volatile("eieio; tlbsync; ptesync": : :"memory");
 }
 
+struct tlbiel_pid {
+	unsigned long pid;
+	unsigned long ric;
+};
+
+static void do_tlbiel_pid(void *info)
+{
+	struct tlbiel_pid *t = info;
+
+	if (t->ric == RIC_FLUSH_TLB)
+		_tlbiel_pid(t->pid, RIC_FLUSH_TLB);
+	else if (t->ric == RIC_FLUSH_PWC)
+		_tlbiel_pid(t->pid, RIC_FLUSH_PWC);
+	else
+		_tlbiel_pid(t->pid, RIC_FLUSH_ALL);
+}
+
+static inline void _tlbiel_pid_multicast(struct mm_struct *mm,
+				unsigned long pid, unsigned long ric)
+{
+	struct cpumask *cpus = mm_cpumask(mm);
+	struct tlbiel_pid t = { .pid = pid, .ric = ric };
+
+	on_each_cpu_mask(cpus, do_tlbiel_pid, &t, 1);
+	/*
+	 * Always want the CPU translations to be invalidated with tlbiel in
+	 * these paths, so while coprocessors must use tlbie, we can not
+	 * optimise away the tlbiel component.
+	 */
+	if (atomic_read(&mm->context.copros) > 0)
+		_tlbie_pid(pid, RIC_FLUSH_ALL);
+}
+
 static inline void _tlbiel_lpid(unsigned long lpid, unsigned long ric)
 {
 	int set;
@@ -420,6 +453,53 @@ static __always_inline void _tlbie_va(unsigned long va, unsigned long pid,
 	asm volatile("eieio; tlbsync; ptesync": : :"memory");
 }
 
+struct tlbiel_va {
+	unsigned long pid;
+	unsigned long va;
+	unsigned long psize;
+	unsigned long ric;
+};
+
+static void do_tlbiel_va(void *info)
+{
+	struct tlbiel_va *t = info;
+
+	if (t->ric == RIC_FLUSH_TLB)
+		_tlbiel_va(t->va, t->pid, t->psize, RIC_FLUSH_TLB);
+	else if (t->ric == RIC_FLUSH_PWC)
+		_tlbiel_va(t->va, t->pid, t->psize, RIC_FLUSH_PWC);
+	else
+		_tlbiel_va(t->va, t->pid, t->psize, RIC_FLUSH_ALL);
+}
+
+static inline void _tlbiel_va_multicast(struct mm_struct *mm,
+				unsigned long va, unsigned long pid,
+				unsigned long psize, unsigned long ric)
+{
+	struct cpumask *cpus = mm_cpumask(mm);
+	struct tlbiel_va t = { .va = va, .pid = pid, .psize = psize, .ric = ric };
+	on_each_cpu_mask(cpus, do_tlbiel_va, &t, 1);
+	if (atomic_read(&mm->context.copros) > 0)
+		_tlbie_va(va, pid, psize, RIC_FLUSH_TLB);
+}
+
+struct tlbiel_va_range {
+	unsigned long pid;
+	unsigned long start;
+	unsigned long end;
+	unsigned long page_size;
+	unsigned long psize;
+	bool also_pwc;
+};
+
+static void do_tlbiel_va_range(void *info)
+{
+	struct tlbiel_va_range *t = info;
+
+	_tlbiel_va_range(t->start, t->end, t->pid, t->page_size,
+				    t->psize, t->also_pwc);
+}
+
 static __always_inline void _tlbie_lpid_va(unsigned long va, unsigned long lpid,
 			      unsigned long psize, unsigned long ric)
 {
@@ -443,6 +523,21 @@ static inline void _tlbie_va_range(unsigned long start, unsigned long end,
 	asm volatile("eieio; tlbsync; ptesync": : :"memory");
 }
 
+static inline void _tlbiel_va_range_multicast(struct mm_struct *mm,
+				unsigned long start, unsigned long end,
+				unsigned long pid, unsigned long page_size,
+				unsigned long psize, bool also_pwc)
+{
+	struct cpumask *cpus = mm_cpumask(mm);
+	struct tlbiel_va_range t = { .start = start, .end = end,
+				.pid = pid, .page_size = page_size,
+				.psize = psize, .also_pwc = also_pwc };
+
+	on_each_cpu_mask(cpus, do_tlbiel_va_range, &t, 1);
+	if (atomic_read(&mm->context.copros) > 0)
+		_tlbie_va_range(start, end, pid, page_size, psize, also_pwc);
+}
+
 /*
  * Base TLB flushing operations:
  *
@@ -580,10 +675,14 @@ void radix__flush_tlb_mm(struct mm_struct *mm)
 			goto local;
 		}
 
-		if (mm_needs_flush_escalation(mm))
-			_tlbie_pid(pid, RIC_FLUSH_ALL);
-		else
-			_tlbie_pid(pid, RIC_FLUSH_TLB);
+		if (use_tlbie()) {
+			if (mm_needs_flush_escalation(mm))
+				_tlbie_pid(pid, RIC_FLUSH_ALL);
+			else if (use_tlbie())
+				_tlbie_pid(pid, RIC_FLUSH_TLB);
+		} else {
+			_tlbiel_pid_multicast(mm, pid, RIC_FLUSH_TLB);
+		}
 	} else {
 local:
 		_tlbiel_pid(pid, RIC_FLUSH_TLB);
@@ -609,7 +708,10 @@ static void __flush_all_mm(struct mm_struct *mm, bool fullmm)
 				goto local;
 			}
 		}
-		_tlbie_pid(pid, RIC_FLUSH_ALL);
+		if (use_tlbie())
+			_tlbie_pid(pid, RIC_FLUSH_ALL);
+		else
+			_tlbiel_pid_multicast(mm, pid, RIC_FLUSH_ALL);
 	} else {
 local:
 		_tlbiel_pid(pid, RIC_FLUSH_ALL);
@@ -644,7 +746,10 @@ void radix__flush_tlb_page_psize(struct mm_struct *mm, unsigned long vmaddr,
 			exit_flush_lazy_tlbs(mm);
 			goto local;
 		}
-		_tlbie_va(vmaddr, pid, psize, RIC_FLUSH_TLB);
+		if (use_tlbie())
+			_tlbie_va(vmaddr, pid, psize, RIC_FLUSH_TLB);
+		else
+			_tlbiel_va_multicast(mm, vmaddr, pid, psize, RIC_FLUSH_TLB);
 	} else {
 local:
 		_tlbiel_va(vmaddr, pid, psize, RIC_FLUSH_TLB);
@@ -666,6 +771,16 @@ EXPORT_SYMBOL(radix__flush_tlb_page);
 #define radix__flush_all_mm radix__local_flush_all_mm
 #endif /* CONFIG_SMP */
 
+static void do_tlbiel_kernel(void *info)
+{
+	_tlbiel_pid(0, RIC_FLUSH_ALL);
+}
+
+static inline void _tlbiel_kernel_broadcast(void)
+{
+	on_each_cpu(do_tlbiel_kernel, NULL, 1);
+}
+
 /*
  * If kernel TLBIs ever become local rather than global, then
  * drivers/misc/ocxl/link.c:ocxl_link_add_pe will need some work, as it
@@ -673,7 +788,10 @@ EXPORT_SYMBOL(radix__flush_tlb_page);
  */
 void radix__flush_tlb_kernel_range(unsigned long start, unsigned long end)
 {
-	_tlbie_pid(0, RIC_FLUSH_ALL);
+	if (tlbie_capable)
+		_tlbie_pid(0, RIC_FLUSH_ALL);
+	else
+		_tlbiel_kernel_broadcast();
 }
 EXPORT_SYMBOL(radix__flush_tlb_kernel_range);
 
@@ -729,10 +847,14 @@ static inline void __radix__flush_tlb_range(struct mm_struct *mm,
 		if (local) {
 			_tlbiel_pid(pid, RIC_FLUSH_TLB);
 		} else {
-			if (mm_needs_flush_escalation(mm))
-				_tlbie_pid(pid, RIC_FLUSH_ALL);
-			else
-				_tlbie_pid(pid, RIC_FLUSH_TLB);
+			if (use_tlbie()) {
+				if (mm_needs_flush_escalation(mm))
+					_tlbie_pid(pid, RIC_FLUSH_ALL);
+				else
+					_tlbie_pid(pid, RIC_FLUSH_TLB);
+			} else {
+				_tlbiel_pid_multicast(mm, pid, RIC_FLUSH_TLB);
+			}
 		}
 	} else {
 		bool hflush = flush_all_sizes;
@@ -757,8 +879,8 @@ static inline void __radix__flush_tlb_range(struct mm_struct *mm,
 				gflush = false;
 		}
 
-		asm volatile("ptesync": : :"memory");
 		if (local) {
+			asm volatile("ptesync": : :"memory");
 			__tlbiel_va_range(start, end, pid, page_size, mmu_virtual_psize);
 			if (hflush)
 				__tlbiel_va_range(hstart, hend, pid,
@@ -767,7 +889,8 @@ static inline void __radix__flush_tlb_range(struct mm_struct *mm,
 				__tlbiel_va_range(gstart, gend, pid,
 						PUD_SIZE, MMU_PAGE_1G);
 			asm volatile("ptesync": : :"memory");
-		} else {
+		} else if (use_tlbie()) {
+			asm volatile("ptesync": : :"memory");
 			__tlbie_va_range(start, end, pid, page_size, mmu_virtual_psize);
 			if (hflush)
 				__tlbie_va_range(hstart, hend, pid,
@@ -777,6 +900,15 @@ static inline void __radix__flush_tlb_range(struct mm_struct *mm,
 						PUD_SIZE, MMU_PAGE_1G);
 			fixup_tlbie();
 			asm volatile("eieio; tlbsync; ptesync": : :"memory");
+		} else {
+			_tlbiel_va_range_multicast(mm,
+					start, end, pid, page_size, mmu_virtual_psize, false);
+			if (hflush)
+				_tlbiel_va_range_multicast(mm,
+					hstart, hend, pid, PMD_SIZE, MMU_PAGE_2M, false);
+			if (gflush)
+				_tlbiel_va_range_multicast(mm,
+					gstart, gend, pid, PUD_SIZE, MMU_PAGE_1G, false);
 		}
 	}
 	preempt_enable();
@@ -966,16 +1098,26 @@ static __always_inline void __radix__flush_tlb_range_psize(struct mm_struct *mm,
 		if (local) {
 			_tlbiel_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB);
 		} else {
-			if (mm_needs_flush_escalation(mm))
-				also_pwc = true;
+			if (use_tlbie()) {
+				if (mm_needs_flush_escalation(mm))
+					also_pwc = true;
+
+				_tlbie_pid(pid,
+					also_pwc ?  RIC_FLUSH_ALL : RIC_FLUSH_TLB);
+			} else {
+				_tlbiel_pid_multicast(mm, pid,
+					also_pwc ?  RIC_FLUSH_ALL : RIC_FLUSH_TLB);
+			}
 
-			_tlbie_pid(pid, also_pwc ? RIC_FLUSH_ALL : RIC_FLUSH_TLB);
 		}
 	} else {
 		if (local)
 			_tlbiel_va_range(start, end, pid, page_size, psize, also_pwc);
-		else
+		else if (use_tlbie())
 			_tlbie_va_range(start, end, pid, page_size, psize, also_pwc);
+		else
+			_tlbiel_va_range_multicast(mm,
+					start, end, pid, page_size, psize, also_pwc);
 	}
 	preempt_enable();
 }
@@ -1017,7 +1159,11 @@ void radix__flush_tlb_collapsed_pmd(struct mm_struct *mm, unsigned long addr)
 			exit_flush_lazy_tlbs(mm);
 			goto local;
 		}
-		_tlbie_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true);
+		if (use_tlbie())
+			_tlbie_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true);
+		else
+			_tlbiel_va_range_multicast(mm,
+					addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true);
 	} else {
 local:
 		_tlbiel_va_range(addr, end, pid, PAGE_SIZE, mmu_virtual_psize, true);
diff --git a/drivers/misc/cxl/main.c b/drivers/misc/cxl/main.c
index 482a2c1b340a..43b312d06e3e 100644
--- a/drivers/misc/cxl/main.c
+++ b/drivers/misc/cxl/main.c
@@ -18,6 +18,7 @@
 #include <linux/sched/task.h>
 
 #include <asm/cputable.h>
+#include <asm/mmu.h>
 #include <misc/cxl-base.h>
 
 #include "cxl.h"
@@ -315,6 +316,9 @@ static int __init init_cxl(void)
 {
 	int rc = 0;
 
+	if (!tlbie_capable)
+		return -EINVAL;
+
 	if ((rc = cxl_file_init()))
 		return rc;
 
diff --git a/drivers/misc/ocxl/main.c b/drivers/misc/ocxl/main.c
index 7210d9e059be..ef73cf35dda2 100644
--- a/drivers/misc/ocxl/main.c
+++ b/drivers/misc/ocxl/main.c
@@ -2,12 +2,16 @@
 // Copyright 2017 IBM Corp.
 #include <linux/module.h>
 #include <linux/pci.h>
+#include <asm/mmu.h>
 #include "ocxl_internal.h"
 
 static int __init init_ocxl(void)
 {
 	int rc = 0;
 
+	if (!tlbie_capable)
+		return -EINVAL;
+
 	rc = ocxl_file_init();
 	if (rc)
 		return rc;
-- 
2.22.0

Re: [PATCH 2/3] powerpc/64s/radix: all CPUs should flush local translation structure before turning MMU on

[Michael Ellerman]

Nicholas Piggin <npiggin@gmail.com> writes:
> Rather than sprinkle various translation structure invalidations
> around different places in early boot, have each CPU flush everything
> from its local translation structures before enabling its MMU.
>
> Radix guests can execute tlbie(l), so have them tlbiel_all in the same
> place as radix host does.
>
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> ---
>  arch/powerpc/mm/book3s64/radix_pgtable.c | 11 ++---------
>  1 file changed, 2 insertions(+), 9 deletions(-)
>
> diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c
> index d60cfa05447a..839e01795211 100644
> --- a/arch/powerpc/mm/book3s64/radix_pgtable.c
> +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c
> @@ -382,11 +382,6 @@ static void __init radix_init_pgtable(void)
>  	 */
>  	register_process_table(__pa(process_tb), 0, PRTB_SIZE_SHIFT - 12);
>  	pr_info("Process table %p and radix root for kernel: %p\n", process_tb, init_mm.pgd);
> -	asm volatile("ptesync" : : : "memory");
> -	asm volatile(PPC_TLBIE_5(%0,%1,2,1,1) : :
> -		     "r" (TLBIEL_INVAL_SET_LPID), "r" (0));
> -	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
> -	trace_tlbie(0, 0, TLBIEL_INVAL_SET_LPID, 0, 2, 1, 1);
>  
>  	/*
>  	 * The init_mm context is given the first available (non-zero) PID,
> @@ -633,8 +628,7 @@ void __init radix__early_init_mmu(void)
>  	radix_init_pgtable();
>  	/* Switch to the guard PID before turning on MMU */
>  	radix__switch_mmu_context(NULL, &init_mm);
> -	if (cpu_has_feature(CPU_FTR_HVMODE))
> -		tlbiel_all();
> +	tlbiel_all();
>  }

This is oopsing for me in a guest on Power9:

  [    0.000000] radix-mmu: Page sizes from device-tree:
  [    0.000000] radix-mmu: Page size shift = 12 AP=0x0
  [    0.000000] radix-mmu: Page size shift = 16 AP=0x5
  [    0.000000] radix-mmu: Page size shift = 21 AP=0x1
  [    0.000000] radix-mmu: Page size shift = 30 AP=0x2
  [    0.000000]  -> fw_vec5_feature_init()
  [    0.000000]  <- fw_vec5_feature_init()
  [    0.000000]  -> fw_hypertas_feature_init()
  [    0.000000]  <- fw_hypertas_feature_init()
  [    0.000000] radix-mmu: Activating Kernel Userspace Execution Prevention
  [    0.000000] radix-mmu: Activating Kernel Userspace Access Prevention
  [    0.000000] lpar: Using radix MMU under hypervisor
  [    0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000040000000 with 1.00 GiB pages (exec)
  [    0.000000] radix-mmu: Mapped 0x0000000040000000-0x0000000100000000 with 1.00 GiB pages
  [    0.000000] radix-mmu: Process table (____ptrval____) and radix root for kernel: (____ptrval____)
  [    0.000000] Oops: Exception in kernel mode, sig: 4 [#1]
  [    0.000000] LE PAGE_SIZE=64K MMU=Radix MMU=Hash SMP NR_CPUS=2048 NUMA 
  [    0.000000] Modules linked in:
  [    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.3.0-rc2-gcc-8.2.0-00063-gef906dcf7b75 #633
  [    0.000000] NIP:  c0000000000838f8 LR: c000000001066864 CTR: c0000000000838c0
  [    0.000000] REGS: c000000001647c40 TRAP: 0700   Not tainted  (5.3.0-rc2-gcc-8.2.0-00063-gef906dcf7b75)
  [    0.000000] MSR:  8000000000043003 <SF,FP,ME,RI,LE>  CR: 48000222  XER: 20040000
  [    0.000000] CFAR: c0000000000839b4 IRQMASK: 1 
  [    0.000000] GPR00: c000000001066864 c000000001647ed0 c000000001649700 0000000000000000 
  [    0.000000] GPR04: c000000001608830 0000000000000000 0000000000000010 2000000000000000 
  [    0.000000] GPR08: 0000000000000c00 0000000000000000 0000000000000002 726f6620746f6f72 
  [    0.000000] GPR12: c0000000000838c0 c000000001930000 000000000dc5bef0 0000000001309e10 
  [    0.000000] GPR16: 0000000001309c90 fffffffffffffffd 000000000dc5bef0 0000000001339800 
  [    0.000000] GPR20: 0000000000000014 0000000001ac0000 000000000dc5bf38 000000000daf0000 
  [    0.000000] GPR24: 0000000001f4000c c000000000000000 0000000000400000 c000000001802858 
  [    0.000000] GPR28: c007ffffffffffff c000000001803954 c000000001681cb0 c000000001608830 
  [    0.000000] NIP [c0000000000838f8] radix__tlbiel_all+0x48/0x110
  [    0.000000] LR [c000000001066864] radix__early_init_mmu+0x494/0x4c8
  [    0.000000] Call Trace:
  [    0.000000] [c000000001647ed0] [c000000001066820] radix__early_init_mmu+0x450/0x4c8 (unreliable)
  [    0.000000] [c000000001647f60] [c00000000105c628] early_setup+0x160/0x198
  [    0.000000] [c000000001647f90] [000000000000b460] 0xb460
  [    0.000000] Instruction dump:
  [    0.000000] 2b830001 39000002 409e00e8 3d220003 3929c318 e9290000 e9290010 75290002 
  [    0.000000] 41820088 7c4004ac 39200000 79085564 <7d294224> 3940007f 39201000 38e00000 
  [    0.000000] random: get_random_bytes called from print_oops_end_marker+0x40/0x80 with crng_init=0
  [    0.000000] ---[ end trace 0000000000000000 ]---


So I think we still need a HV check in there somewhere.

cheers

Re: [PATCH 2/3] powerpc/64s/radix: all CPUs should flush local translation structure before turning MMU on

[Nicholas Piggin]

Michael Ellerman's on August 19, 2019 12:00 pm:
> Nicholas Piggin <npiggin@gmail.com> writes:
>> Rather than sprinkle various translation structure invalidations
>> around different places in early boot, have each CPU flush everything
>> from its local translation structures before enabling its MMU.
>>
>> Radix guests can execute tlbie(l), so have them tlbiel_all in the same
>> place as radix host does.
>>
>> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
>> ---
>>  arch/powerpc/mm/book3s64/radix_pgtable.c | 11 ++---------
>>  1 file changed, 2 insertions(+), 9 deletions(-)
>>
>> diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c
>> index d60cfa05447a..839e01795211 100644
>> --- a/arch/powerpc/mm/book3s64/radix_pgtable.c
>> +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c
>> @@ -382,11 +382,6 @@ static void __init radix_init_pgtable(void)
>>  	 */
>>  	register_process_table(__pa(process_tb), 0, PRTB_SIZE_SHIFT - 12);
>>  	pr_info("Process table %p and radix root for kernel: %p\n", process_tb, init_mm.pgd);
>> -	asm volatile("ptesync" : : : "memory");
>> -	asm volatile(PPC_TLBIE_5(%0,%1,2,1,1) : :
>> -		     "r" (TLBIEL_INVAL_SET_LPID), "r" (0));
>> -	asm volatile("eieio; tlbsync; ptesync" : : : "memory");
>> -	trace_tlbie(0, 0, TLBIEL_INVAL_SET_LPID, 0, 2, 1, 1);
>>  
>>  	/*
>>  	 * The init_mm context is given the first available (non-zero) PID,
>> @@ -633,8 +628,7 @@ void __init radix__early_init_mmu(void)
>>  	radix_init_pgtable();
>>  	/* Switch to the guard PID before turning on MMU */
>>  	radix__switch_mmu_context(NULL, &init_mm);
>> -	if (cpu_has_feature(CPU_FTR_HVMODE))
>> -		tlbiel_all();
>> +	tlbiel_all();
>>  }
> 
> This is oopsing for me in a guest on Power9:
> 
>   [    0.000000] radix-mmu: Page sizes from device-tree:
>   [    0.000000] radix-mmu: Page size shift = 12 AP=0x0
>   [    0.000000] radix-mmu: Page size shift = 16 AP=0x5
>   [    0.000000] radix-mmu: Page size shift = 21 AP=0x1
>   [    0.000000] radix-mmu: Page size shift = 30 AP=0x2
>   [    0.000000]  -> fw_vec5_feature_init()
>   [    0.000000]  <- fw_vec5_feature_init()
>   [    0.000000]  -> fw_hypertas_feature_init()
>   [    0.000000]  <- fw_hypertas_feature_init()
>   [    0.000000] radix-mmu: Activating Kernel Userspace Execution Prevention
>   [    0.000000] radix-mmu: Activating Kernel Userspace Access Prevention
>   [    0.000000] lpar: Using radix MMU under hypervisor
>   [    0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000040000000 with 1.00 GiB pages (exec)
>   [    0.000000] radix-mmu: Mapped 0x0000000040000000-0x0000000100000000 with 1.00 GiB pages
>   [    0.000000] radix-mmu: Process table (____ptrval____) and radix root for kernel: (____ptrval____)
>   [    0.000000] Oops: Exception in kernel mode, sig: 4 [#1]
>   [    0.000000] LE PAGE_SIZE=64K MMU=Radix MMU=Hash SMP NR_CPUS=2048 NUMA 
>   [    0.000000] Modules linked in:
>   [    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.3.0-rc2-gcc-8.2.0-00063-gef906dcf7b75 #633
>   [    0.000000] NIP:  c0000000000838f8 LR: c000000001066864 CTR: c0000000000838c0
>   [    0.000000] REGS: c000000001647c40 TRAP: 0700   Not tainted  (5.3.0-rc2-gcc-8.2.0-00063-gef906dcf7b75)
>   [    0.000000] MSR:  8000000000043003 <SF,FP,ME,RI,LE>  CR: 48000222  XER: 20040000
>   [    0.000000] CFAR: c0000000000839b4 IRQMASK: 1 
>   [    0.000000] GPR00: c000000001066864 c000000001647ed0 c000000001649700 0000000000000000 
>   [    0.000000] GPR04: c000000001608830 0000000000000000 0000000000000010 2000000000000000 
>   [    0.000000] GPR08: 0000000000000c00 0000000000000000 0000000000000002 726f6620746f6f72 
>   [    0.000000] GPR12: c0000000000838c0 c000000001930000 000000000dc5bef0 0000000001309e10 
>   [    0.000000] GPR16: 0000000001309c90 fffffffffffffffd 000000000dc5bef0 0000000001339800 
>   [    0.000000] GPR20: 0000000000000014 0000000001ac0000 000000000dc5bf38 000000000daf0000 
>   [    0.000000] GPR24: 0000000001f4000c c000000000000000 0000000000400000 c000000001802858 
>   [    0.000000] GPR28: c007ffffffffffff c000000001803954 c000000001681cb0 c000000001608830 
>   [    0.000000] NIP [c0000000000838f8] radix__tlbiel_all+0x48/0x110
>   [    0.000000] LR [c000000001066864] radix__early_init_mmu+0x494/0x4c8
>   [    0.000000] Call Trace:
>   [    0.000000] [c000000001647ed0] [c000000001066820] radix__early_init_mmu+0x450/0x4c8 (unreliable)
>   [    0.000000] [c000000001647f60] [c00000000105c628] early_setup+0x160/0x198
>   [    0.000000] [c000000001647f90] [000000000000b460] 0xb460
>   [    0.000000] Instruction dump:
>   [    0.000000] 2b830001 39000002 409e00e8 3d220003 3929c318 e9290000 e9290010 75290002 
>   [    0.000000] 41820088 7c4004ac 39200000 79085564 <7d294224> 3940007f 39201000 38e00000 
>   [    0.000000] random: get_random_bytes called from print_oops_end_marker+0x40/0x80 with crng_init=0
>   [    0.000000] ---[ end trace 0000000000000000 ]---
> 
> 
> So I think we still need a HV check in there somewhere.

Oh it's the partition scope flush by the looks. Simulator doesn't seem 
to be trapping that one.

I'll resend, have to fix the kernel invalidation to also do tlbie for
nest.

Thanks,
Nick