[PATCH 0/2] ERAT invalidation fixes

[PATCH 0/2] ERAT invalidation fixes

[Nicholas Piggin]

First one is a real bug-fix for POWER8 and earlier CPUs, second I'm
not exactly sure how POWER9's ERAT behaves in this situation but it
seems to match what we do in other places.

Thanks,
Nick

Nicholas Piggin (2):
  powerpc/64s/hash: do not use PPC_INVALIDATE_ERAT on CPUs before POWER9
  powerpc/64s/radix: Explicitly flush ERAT with local LPID invalidation

 arch/powerpc/kernel/mce_power.c  | 7 +++++++
 arch/powerpc/mm/hash_native_64.c | 4 ++--
 arch/powerpc/mm/tlb-radix.c      | 1 +
 3 files changed, 10 insertions(+), 2 deletions(-)

-- 
2.18.0

[PATCH 1/2] powerpc/64s/hash: do not use PPC_INVALIDATE_ERAT on CPUs before POWER9

[Nicholas Piggin]

PPC_INVALIDATE_ERAT is slbia IH=7 which is a new variant introduced
with POWER9, and the result is undefined on earlier CPUs.

Commits 7b9f71f974 ("powerpc/64s: POWER9 machine check handler") and
d4748276ae ("powerpc/64s: Improve local TLB flush for boot and MCE on
POWER9") caused POWER7/8 code to use this instruction. Remove it. An
ERAT flush can be made by invalidatig the SLB, but before POWER9 that
requires a flush and rebolt.

Fixes: 7b9f71f974 ("powerpc/64s: POWER9 machine check handler")
Fixes: d4748276ae ("powerpc/64s: Improve local TLB flush for boot and
MCE on POWER9")
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 arch/powerpc/kernel/mce_power.c  | 7 +++++++
 arch/powerpc/mm/hash_native_64.c | 4 ++--
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/kernel/mce_power.c b/arch/powerpc/kernel/mce_power.c
index 3497c8329c1d..3022d67f0c48 100644
--- a/arch/powerpc/kernel/mce_power.c
+++ b/arch/powerpc/kernel/mce_power.c
@@ -89,6 +89,13 @@ static void flush_and_reload_slb(void)
 
 static void flush_erat(void)
 {
+#ifdef CONFIG_PPC_BOOK3S_64
+	if (!early_cpu_has_feature(CPU_FTR_ARCH_300)) {
+		flush_and_reload_slb();
+		return;
+	}
+#endif
+	/* PPC_INVALIDATE_ERAT can only be used on ISA v3 and newer */
 	asm volatile(PPC_INVALIDATE_ERAT : : :"memory");
 }
 
diff --git a/arch/powerpc/mm/hash_native_64.c b/arch/powerpc/mm/hash_native_64.c
index 729f02df8290..aaa28fd918fe 100644
--- a/arch/powerpc/mm/hash_native_64.c
+++ b/arch/powerpc/mm/hash_native_64.c
@@ -115,6 +115,8 @@ static void tlbiel_all_isa300(unsigned int num_sets, unsigned int is)
 	tlbiel_hash_set_isa300(0, is, 0, 2, 1);
 
 	asm volatile("ptesync": : :"memory");
+
+	asm volatile(PPC_INVALIDATE_ERAT "; isync" : : :"memory");
 }
 
 void hash__tlbiel_all(unsigned int action)
@@ -140,8 +142,6 @@ void hash__tlbiel_all(unsigned int action)
 		tlbiel_all_isa206(POWER7_TLB_SETS, is);
 	else
 		WARN(1, "%s called on pre-POWER7 CPU\n", __func__);
-
-	asm volatile(PPC_INVALIDATE_ERAT "; isync" : : :"memory");
 }
 
 static inline unsigned long  ___tlbie(unsigned long vpn, int psize,
-- 
2.18.0

[PATCH 2/2] powerpc/64s/radix: Explicitly flush ERAT with local LPID invalidation

[Nicholas Piggin]

Local radix TLB flush operations that operate on congruence classes
have explicit ERAT flushes for POWER9. The process scoped LPID flush
did not have a flush, so add it.

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 arch/powerpc/mm/tlb-radix.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c
index fef3e1eb3a19..4e798f33c530 100644
--- a/arch/powerpc/mm/tlb-radix.c
+++ b/arch/powerpc/mm/tlb-radix.c
@@ -366,6 +366,7 @@ static inline void _tlbiel_lpid_guest(unsigned long lpid, unsigned long ric)
 		__tlbiel_lpid_guest(lpid, set, RIC_FLUSH_TLB);
 
 	asm volatile("ptesync": : :"memory");
+	asm volatile(PPC_INVALIDATE_ERAT : : :"memory");
 }
 
 
-- 
2.18.0

Re: [PATCH 2/2] powerpc/64s/radix: Explicitly flush ERAT with local LPID invalidation

[Benjamin Herrenschmidt]

On Mon, 2018-08-27 at 13:03 +1000, Nicholas Piggin wrote:
> Local radix TLB flush operations that operate on congruence classes
> have explicit ERAT flushes for POWER9. The process scoped LPID flush
> did not have a flush, so add it.

Paul, is that an actual bug ? I think the ERAT is flushed on LPID
changes...

> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> ---
>  arch/powerpc/mm/tlb-radix.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c
> index fef3e1eb3a19..4e798f33c530 100644
> --- a/arch/powerpc/mm/tlb-radix.c
> +++ b/arch/powerpc/mm/tlb-radix.c
> @@ -366,6 +366,7 @@ static inline void _tlbiel_lpid_guest(unsigned long lpid, unsigned long ric)
>  		__tlbiel_lpid_guest(lpid, set, RIC_FLUSH_TLB);
>  
>  	asm volatile("ptesync": : :"memory");
> +	asm volatile(PPC_INVALIDATE_ERAT : : :"memory");
>  }
>  
>  

Re: [PATCH 2/2] powerpc/64s/radix: Explicitly flush ERAT with local LPID invalidation

[Nicholas Piggin]

On Mon, 27 Aug 2018 18:16:05 +1000
Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

> On Mon, 2018-08-27 at 13:03 +1000, Nicholas Piggin wrote:
> > Local radix TLB flush operations that operate on congruence classes
> > have explicit ERAT flushes for POWER9. The process scoped LPID flush
> > did not have a flush, so add it. =20
>=20
> Paul, is that an actual bug ? I think the ERAT is flushed on LPID
> changes...

We also have a PPC_INVALIDATE_ERAT in tlbiel_lpid. I'd like to add some
comments for these things because I'm not entirely clear on them myself.

The P9 UM says, "Additionally, mtpidr and mtlpidr instructions perform
an implicit slbia with IH =3D x=E2=80=983=E2=80=99."

Although slbia IH=3D3 does not appear to invalidate LPID!=3D0 && PID=3D=3D0
entries (guest OS mappigs).

The UM also says that tlbiels will clear the involved ERATs, but this
might not match hardware?

Thanks,
Nick

>=20
> > Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> > ---
> >  arch/powerpc/mm/tlb-radix.c | 1 +
> >  1 file changed, 1 insertion(+)
> >=20
> > diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c
> > index fef3e1eb3a19..4e798f33c530 100644
> > --- a/arch/powerpc/mm/tlb-radix.c
> > +++ b/arch/powerpc/mm/tlb-radix.c
> > @@ -366,6 +366,7 @@ static inline void _tlbiel_lpid_guest(unsigned long=
 lpid, unsigned long ric)
> >  		__tlbiel_lpid_guest(lpid, set, RIC_FLUSH_TLB);
> > =20
> >  	asm volatile("ptesync": : :"memory");
> > +	asm volatile(PPC_INVALIDATE_ERAT : : :"memory");
> >  }
> > =20
> >   =20
>=20

Re: [PATCH 2/2] powerpc/64s/radix: Explicitly flush ERAT with local LPID invalidation

[Nicholas Piggin]

On Mon, 27 Aug 2018 18:16:05 +1000
Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

> On Mon, 2018-08-27 at 13:03 +1000, Nicholas Piggin wrote:
> > Local radix TLB flush operations that operate on congruence classes
> > have explicit ERAT flushes for POWER9. The process scoped LPID flush
> > did not have a flush, so add it.  
> 
> Paul, is that an actual bug ? I think the ERAT is flushed on LPID
> changes...

FWIW I'd like to add the following patch after these fixes. While I
have your attention...

[PATCH] powerpc/64s/radix: keep kernel ERAT over local process/guest
 invalidates

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
 arch/powerpc/include/asm/ppc-opcode.h | 10 +++++++++-
 arch/powerpc/mm/tlb-radix.c           |  6 +++---
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/arch/powerpc/include/asm/ppc-opcode.h b/arch/powerpc/include/asm/ppc-opcode.h
index 665af14850e4..5fe617ab680a 100644
--- a/arch/powerpc/include/asm/ppc-opcode.h
+++ b/arch/powerpc/include/asm/ppc-opcode.h
@@ -568,7 +568,15 @@
 
 #define PPC_SLBIA(IH)	stringify_in_c(.long PPC_INST_SLBIA | \
 				       ((IH & 0x7) << 21))
-#define PPC_INVALIDATE_ERAT	PPC_SLBIA(7)
+
+/*
+ * These may only be used by ARCH_300.
+ * GUEST/USER invalidates should only be used by radix mode, on HPT they also
+ * invalidate SLBs so the SLBIA instruction should be used directly.
+ */
+#define PPC_INVALIDATE_ERAT		PPC_SLBIA(7)
+#define PPC_INVALIDATE_GUEST_ERAT	PPC_SLBIA(6)
+#define PPC_INVALIDATE_USER_ERAT	PPC_SLBIA(3)
 
 #define VCMPEQUD_RC(vrt, vra, vrb)	stringify_in_c(.long PPC_INST_VCMPEQUD | \
 			      ___PPC_RT(vrt) | ___PPC_RA(vra) | \
diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c
index 4e798f33c530..6887e4b2568b 100644
--- a/arch/powerpc/mm/tlb-radix.c
+++ b/arch/powerpc/mm/tlb-radix.c
@@ -262,7 +262,7 @@ static inline void _tlbiel_pid(unsigned long pid, unsigned long ric)
 		__tlbiel_pid(pid, set, RIC_FLUSH_TLB);
 
 	asm volatile("ptesync": : :"memory");
-	asm volatile(PPC_INVALIDATE_ERAT "; isync" : : :"memory");
+	asm volatile(PPC_INVALIDATE_USER_ERAT "; isync" : : :"memory");
 }
 
 static inline void _tlbie_pid(unsigned long pid, unsigned long ric)
@@ -314,7 +314,7 @@ static inline void _tlbiel_lpid(unsigned long lpid, unsigned long ric)
 		__tlbiel_lpid(lpid, set, RIC_FLUSH_TLB);
 
 	asm volatile("ptesync": : :"memory");
-	asm volatile(PPC_INVALIDATE_ERAT "; isync" : : :"memory");
+	asm volatile(PPC_INVALIDATE_GUEST_ERAT "; isync" : : :"memory");
 }
 
 static inline void _tlbie_lpid(unsigned long lpid, unsigned long ric)
@@ -366,7 +366,7 @@ static inline void _tlbiel_lpid_guest(unsigned long lpid, unsigned long ric)
 		__tlbiel_lpid_guest(lpid, set, RIC_FLUSH_TLB);
 
 	asm volatile("ptesync": : :"memory");
-	asm volatile(PPC_INVALIDATE_ERAT : : :"memory");
+	asm volatile(PPC_INVALIDATE_GUEST_ERAT : : :"memory");
 }
 
 
-- 

Re: [1/2] powerpc/64s/hash: do not use PPC_INVALIDATE_ERAT on CPUs before POWER9

[Michael Ellerman]

On Mon, 2018-08-27 at 03:03:01 UTC, Nicholas Piggin wrote:
> PPC_INVALIDATE_ERAT is slbia IH=7 which is a new variant introduced
> with POWER9, and the result is undefined on earlier CPUs.
> 
> Commits 7b9f71f974 ("powerpc/64s: POWER9 machine check handler") and
> d4748276ae ("powerpc/64s: Improve local TLB flush for boot and MCE on
> POWER9") caused POWER7/8 code to use this instruction. Remove it. An
> ERAT flush can be made by invalidatig the SLB, but before POWER9 that
> requires a flush and rebolt.
> 
> Fixes: 7b9f71f974 ("powerpc/64s: POWER9 machine check handler")
> Fixes: d4748276ae ("powerpc/64s: Improve local TLB flush for boot and
> MCE on POWER9")
> Signed-off-by: Nicholas Piggin <npiggin@gmail.com>

Series applied to powerpc next, thanks.

https://git.kernel.org/powerpc/c/bc276ecba132caccb1fda5863a652c

cheers