[PATCH 0/5] arm64: Add workaround for Cortex-A77 erratum 1542418

[PATCH 0/5] arm64: Add workaround for Cortex-A77 erratum 1542418

[Suzuki K Poulose]

This series adds workaround for Arm erratum 1542418 which affects
Cortex-A77 cores (r0p0 - r1p0). Affected cores may execute stale
instructions from the L0 macro-op cache violating the
prefetch-speculation-protection guaranteed by the architecture.
This happens when the when the branch predictor bases its predictions
on a branch at this address on the stale history due to ASID or VMID
reuse.

The workaround is to invalidate the branch history before reusing
any ASID for a new address space. This is done by ensuring 60 ASIDs
are selected before any ASID is reused.


James Morse (5):
  arm64: Add MIDR encoding for Arm Cortex-A77
  arm64: mm: Workaround Cortex-A77 erratum 1542418 on ASID rollover
  arm64: Workaround Cortex-A77 erratum 1542418 on boot due to kexec
  KVM: arm64: Workaround Cortex-A77 erratum 1542418 on VMID rollover
  KVM: arm/arm64: Don't invoke defacto-CnP on first run

 Documentation/arm64/silicon-errata.rst |  2 +
 arch/arm/include/asm/kvm_mmu.h         |  5 ++
 arch/arm64/Kconfig                     | 16 ++++++
 arch/arm64/include/asm/cpucaps.h       |  3 +-
 arch/arm64/include/asm/cputype.h       |  2 +
 arch/arm64/include/asm/kvm_mmu.h       | 15 ++++++
 arch/arm64/include/asm/mmu_context.h   |  1 +
 arch/arm64/kernel/cpu_errata.c         | 21 ++++++++
 arch/arm64/mm/context.c                | 73 +++++++++++++++++++++++++-
 virt/kvm/arm/arm.c                     | 23 +++++---
 10 files changed, 151 insertions(+), 10 deletions(-)

-- 
2.23.0

[PATCH 1/5] arm64: Add MIDR encoding for Arm Cortex-A77

[Suzuki K Poulose]

From: James Morse <james.morse@arm.com>

Add Arm Cortex-A77's part-number so we can match against its MIDR_EL1.

Signed-off-by: James Morse <james.morse@arm.com>
Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
---
 arch/arm64/include/asm/cputype.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/arm64/include/asm/cputype.h b/arch/arm64/include/asm/cputype.h
index aca07c2f6e6e..3c8c1580527d 100644
--- a/arch/arm64/include/asm/cputype.h
+++ b/arch/arm64/include/asm/cputype.h
@@ -71,6 +71,7 @@
 #define ARM_CPU_PART_CORTEX_A55		0xD05
 #define ARM_CPU_PART_CORTEX_A76		0xD0B
 #define ARM_CPU_PART_NEOVERSE_N1	0xD0C
+#define ARM_CPU_PART_CORTEX_A77		0xD0D
 
 #define APM_CPU_PART_POTENZA		0x000
 
@@ -101,6 +102,7 @@
 #define MIDR_CORTEX_A35 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A35)
 #define MIDR_CORTEX_A55 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A55)
 #define MIDR_CORTEX_A76	MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A76)
+#define MIDR_CORTEX_A77	MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A77)
 #define MIDR_NEOVERSE_N1 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_NEOVERSE_N1)
 #define MIDR_THUNDERX	MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX)
 #define MIDR_THUNDERX_81XX MIDR_CPU_MODEL(ARM_CPU_IMP_CAVIUM, CAVIUM_CPU_PART_THUNDERX_81XX)
-- 
2.23.0

[PATCH 2/5] arm64: mm: Workaround Cortex-A77 erratum 1542418 on ASID rollover

[Suzuki K Poulose]

From: James Morse <james.morse@arm.com>

On affected Cortex-A77 cores, software relying on the
prefetch-speculation-protection instead of explicit synchronisation may
fetch a stale instruction from a CPU-specific cache. This violates the
ordering rules for instruction fetches.

This can only happen when the CPU correctly predicts the modified branch
based on a previous ASID/VMID. The workaround is to prevent these
predictions by selecting 60 ASIDs before an ASID is reused.

Add this logic as a workaround in the asid-alloctor's per-cpu rollover
path. When the first asid of the new generation is about to be used,
select 60 different ASIDs before we do the TLB maintenance.

Signed-off-by: James Morse <james.morse@arm.com>
[ Added/modified commentary ]
Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
---
 Documentation/arm64/silicon-errata.rst |  2 +
 arch/arm64/Kconfig                     | 16 ++++++++
 arch/arm64/include/asm/cpucaps.h       |  3 +-
 arch/arm64/kernel/cpu_errata.c         |  7 ++++
 arch/arm64/mm/context.c                | 56 +++++++++++++++++++++++++-
 5 files changed, 82 insertions(+), 2 deletions(-)

diff --git a/Documentation/arm64/silicon-errata.rst b/Documentation/arm64/silicon-errata.rst
index 5a09661330fc..a6a5ece00392 100644
--- a/Documentation/arm64/silicon-errata.rst
+++ b/Documentation/arm64/silicon-errata.rst
@@ -84,6 +84,8 @@ stable kernels.
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Cortex-A76      | #1463225        | ARM64_ERRATUM_1463225       |
 +----------------+-----------------+-----------------+-----------------------------+
+| ARM            | Cortex-A77      | #1542418        | ARM64_ERRATUM_1542418       |
++----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-N1     | #1188873,1418040| ARM64_ERRATUM_1418040       |
 +----------------+-----------------+-----------------+-----------------------------+
 | ARM            | Neoverse-N1     | #1349291        | N/A                         |
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 3f047afb982c..f0fc570ce05f 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -558,6 +558,22 @@ config ARM64_ERRATUM_1463225
 
 	  If unsure, say Y.
 
+config ARM64_ERRATUM_1542418
+	bool "Cortex-A77: The core might fetch a stale instuction, violating the ordering of instruction fetches"
+	default y
+	help
+	  This option adds a workaround for Arm Cortex-A77 erratum 1542418.
+
+	  On the affected Cortex-A77 cores (r0p0 and r1p0), software relying
+	  on the prefetch-speculation-protection instead of explicit
+	  synchronisation may fetch a stale instruction from a CPU-specific
+	  cache. This violates the ordering rules for instruction fetches.
+
+	  Work around the erratum by ensuring that 60 ASIDs are selected
+	  before any ASID is reused.
+
+	  If unsure, say Y.
+
 config CAVIUM_ERRATUM_22375
 	bool "Cavium erratum 22375, 24313"
 	default y
diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index ac1dbca3d0cd..1f90084e8a59 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -54,7 +54,8 @@
 #define ARM64_WORKAROUND_1463225		44
 #define ARM64_WORKAROUND_CAVIUM_TX2_219_TVM	45
 #define ARM64_WORKAROUND_CAVIUM_TX2_219_PRFM	46
+#define ARM64_WORKAROUND_1542418		47
 
-#define ARM64_NCAPS				47
+#define ARM64_NCAPS				48
 
 #endif /* __ASM_CPUCAPS_H */
diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index 93f34b4eca25..a66d433d0113 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -926,6 +926,13 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
 		.capability = ARM64_WORKAROUND_CAVIUM_TX2_219_PRFM,
 		ERRATA_MIDR_RANGE_LIST(tx2_family_cpus),
 	},
+#endif
+#ifdef CONFIG_ARM64_ERRATUM_1542418
+	{
+		.desc = "ARM erratum 1542418",
+		.capability = ARM64_WORKAROUND_1542418,
+		ERRATA_MIDR_RANGE(MIDR_CORTEX_A77, 0, 0, 1, 0),
+	},
 #endif
 	{
 	}
diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c
index b5e329fde2dd..ae3ee8e101d6 100644
--- a/arch/arm64/mm/context.c
+++ b/arch/arm64/mm/context.c
@@ -77,6 +77,58 @@ void verify_cpu_asid_bits(void)
 	}
 }
 
+
+/*
+ * When the CnP is active, the caller must have set the ttbr0 to reserved
+ * before calling this function.
+ * Upon completion, the caller must ensure to:
+ *   - restore the ttbr0
+ *   - execute isb() to synchronize the change.
+ */
+static void __arm64_workaround_1542418_asid_rollover(void)
+{
+	phys_addr_t ttbr1_baddr;
+	u64 idx, ttbr1;	/* ASID is in ttbr1 due to TCR_EL1.A1 */
+
+	if (!IS_ENABLED(CONFIG_ARM64_ERRATUM_1542418) ||
+	    !cpus_have_const_cap(ARM64_WORKAROUND_1542418) ||
+	    !this_cpu_has_cap(ARM64_WORKAROUND_1542418))
+		return;
+
+	/*
+	 * We're about to use an arbitrary set of ASIDs, which may have
+	 * live entries in the TLB (and on other CPUs with CnP). Ensure
+	 * that we can't allocate conflicting entries using this task's
+	 * TTBR0.
+	 */
+	if (!system_supports_cnp())
+		cpu_set_reserved_ttbr0();
+	/* else: the caller must have already set this */
+
+	ttbr1 = read_sysreg(ttbr1_el1);
+	ttbr1_baddr = ttbr1 & ~TTBR_ASID_MASK;
+
+	/*
+	 * Select 60 asids to invalidate the branch history for this generation.
+	 * If kpti is in use we avoid selecting a user asid as
+	 * __sdei_asm_entry_trampoline() uses USER_ASID_FLAG to determine if
+	 * the NMI interrupted the kpti trampoline. Avoid using the reserved
+	 * asid 0.
+	 */
+	for (idx = 1; idx <= 61; idx++) {
+		write_sysreg((idx2asid(idx) << 48) | ttbr1_baddr, ttbr1_el1);
+		isb();
+	}
+
+	/* restore the current ASID */
+	write_sysreg(ttbr1, ttbr1_el1);
+
+	/*
+	 * Rely on local_flush_tlb_all()'s isb to complete the ASID restore.
+	 * check_and_switch_context() will call cpu_switch_mm() to (re)set ttbr0_el1.
+	 */
+}
+
 static void flush_context(void)
 {
 	int i;
@@ -219,8 +271,10 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu)
 		atomic64_set(&mm->context.id, asid);
 	}
 
-	if (cpumask_test_and_clear_cpu(cpu, &tlb_flush_pending))
+	if (cpumask_test_and_clear_cpu(cpu, &tlb_flush_pending)) {
+		__arm64_workaround_1542418_asid_rollover();
 		local_flush_tlb_all();
+	}
 
 	atomic64_set(&per_cpu(active_asids, cpu), asid);
 	raw_spin_unlock_irqrestore(&cpu_asid_lock, flags);
-- 
2.23.0

[PATCH 3/5] arm64: Workaround Cortex-A77 erratum 1542418 on boot due to kexec

[Suzuki K Poulose]

From: James Morse <james.morse@arm.com>

Kexec allows us to inherit dirty ASIDs from a previous kernel. We can't
wait until the next ASID rollover to cleanup, do it early as part of
the cpu-errata's enable callback.

This extends __arm64_workaround_1542418_asid_rollover() to put everything
back as it was.

Signed-off-by: James Morse <james.morse@arm.com>
[ skip CPUs not affected, refactor cpu_enable callback ]
Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
---
 arch/arm64/include/asm/mmu_context.h |  1 +
 arch/arm64/kernel/cpu_errata.c       | 14 ++++++++++++++
 arch/arm64/mm/context.c              | 17 +++++++++++++++++
 3 files changed, 32 insertions(+)

diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h
index 3827ff4040a3..434a5c661d78 100644
--- a/arch/arm64/include/asm/mmu_context.h
+++ b/arch/arm64/include/asm/mmu_context.h
@@ -247,6 +247,7 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next,
 
 void verify_cpu_asid_bits(void);
 void post_ttbr_update_workaround(void);
+void arm64_workaround_1542418_asid_rollover(void);
 
 #endif /* !__ASSEMBLY__ */
 
diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index a66d433d0113..4656157ffa36 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -12,6 +12,7 @@
 #include <asm/cpu.h>
 #include <asm/cputype.h>
 #include <asm/cpufeature.h>
+#include <asm/mmu_context.h>
 #include <asm/smp_plat.h>
 
 static bool __maybe_unused
@@ -650,6 +651,18 @@ needs_tx2_tvm_workaround(const struct arm64_cpu_capabilities *entry,
 	return false;
 }
 
+#ifdef CONFIG_ARM64_ERRATUM_1542418
+static void run_workaround_1542418_asid_rollover(const struct arm64_cpu_capabilities *c)
+{
+	/*
+	 * If this CPU is affected by the erratum, run the workaround
+	 * to protect us in case we are running on a kexec'ed kernel.
+	 */
+	if (c->matches(c, SCOPE_LOCAL_CPU))
+		arm64_workaround_1542418_asid_rollover();
+}
+#endif
+
 #ifdef CONFIG_HARDEN_EL2_VECTORS
 
 static const struct midr_range arm64_harden_el2_vectors[] = {
@@ -932,6 +945,7 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
 		.desc = "ARM erratum 1542418",
 		.capability = ARM64_WORKAROUND_1542418,
 		ERRATA_MIDR_RANGE(MIDR_CORTEX_A77, 0, 0, 1, 0),
+		.cpu_enable = run_workaround_1542418_asid_rollover,
 	},
 #endif
 	{
diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c
index ae3ee8e101d6..ad4e78bb68ed 100644
--- a/arch/arm64/mm/context.c
+++ b/arch/arm64/mm/context.c
@@ -129,6 +129,23 @@ static void __arm64_workaround_1542418_asid_rollover(void)
 	 */
 }
 
+void arm64_workaround_1542418_asid_rollover(void)
+{
+	u64 ttbr0 = read_sysreg(ttbr0_el1);
+
+	lockdep_assert_irqs_disabled();
+
+	/* Mirror check_and_switch_context() */
+	if (system_supports_cnp())
+		cpu_set_reserved_ttbr0();
+
+	__arm64_workaround_1542418_asid_rollover();
+	isb();
+
+	write_sysreg(ttbr0, ttbr0_el1);
+	isb();
+}
+
 static void flush_context(void)
 {
 	int i;
-- 
2.23.0

[PATCH 4/5] KVM: arm64: Workaround Cortex-A77 erratum 1542418 on VMID rollover

[Suzuki K Poulose]

From: James Morse <james.morse@arm.com>

Cortex-A77's erratum 1542418 workaround needs to be applied for VMID
re-use too. This prevents the CPU correctly predicting a modified branch
based on a previous user of the VMID and ASID.

KVM doesn't use force_vm_exit or exit_vm_noop for anything other than
vmid rollover. Rename them, and use this to invoke the VMID workaround
on each CPU.

Another case where VMID and ASID may get reused is if the system is
over-provisioned and two vCPUs of the same VMID are scheduled on
one physical CPU. KVM invalidates the TLB to prevent ASID sharing
in this case, invoke the asid-rollover workaround too so we avoid
the ASID sharing tripping the erratum.

Signed-off-by: James Morse <james.morse@arm.com>
Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
---
 arch/arm/include/asm/kvm_mmu.h   |  5 +++++
 arch/arm64/include/asm/kvm_mmu.h | 15 +++++++++++++++
 virt/kvm/arm/arm.c               | 20 ++++++++++++++------
 3 files changed, 34 insertions(+), 6 deletions(-)

diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index 0d84d50bf9ba..8a5702e0c3f8 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -430,6 +430,11 @@ static __always_inline u64 kvm_get_vttbr(struct kvm *kvm)
 	return kvm_phys_to_vttbr(baddr) | vmid_field;
 }
 
+static inline void kvm_workaround_1542418_vmid_rollover(void)
+{
+	/* not affected */
+}
+
 #endif	/* !__ASSEMBLY__ */
 
 #endif /* __ARM_KVM_MMU_H__ */
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index befe37d4bc0e..5776e53c296d 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -9,6 +9,7 @@
 
 #include <asm/page.h>
 #include <asm/memory.h>
+#include <asm/mmu_context.h>
 #include <asm/cpufeature.h>
 
 /*
@@ -603,5 +604,19 @@ static __always_inline u64 kvm_get_vttbr(struct kvm *kvm)
 	return kvm_phys_to_vttbr(baddr) | vmid_field | cnp;
 }
 
+static inline void kvm_workaround_1542418_vmid_rollover(void)
+{
+	unsigned long flags;
+
+	if (!IS_ENABLED(CONFIG_ARM64_ERRATUM_1542418) ||
+	    !cpus_have_const_cap(ARM64_WORKAROUND_1542418))
+		return;
+
+	local_irq_save(flags);
+	arm64_workaround_1542418_asid_rollover();
+	local_irq_restore(flags);
+
+}
+
 #endif /* __ASSEMBLY__ */
 #endif /* __ARM64_KVM_MMU_H__ */
diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
index 86c6aa1cb58e..ac9e017df7c9 100644
--- a/virt/kvm/arm/arm.c
+++ b/virt/kvm/arm/arm.c
@@ -368,6 +368,13 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 	 */
 	if (*last_ran != vcpu->vcpu_id) {
 		kvm_call_hyp(__kvm_tlb_flush_local_vmid, vcpu);
+
+		/*
+		 * 'last_ran' and this vcpu may share an ASID and hit the
+		 *  conditions for Cortex-A77 erratum 1542418.
+		 */
+		kvm_workaround_1542418_vmid_rollover();
+
 		*last_ran = vcpu->vcpu_id;
 	}
 
@@ -458,15 +465,16 @@ bool kvm_arch_vcpu_in_kernel(struct kvm_vcpu *vcpu)
 	return vcpu_mode_priv(vcpu);
 }
 
-/* Just ensure a guest exit from a particular CPU */
-static void exit_vm_noop(void *info)
+static void exit_vmid_rollover(void *info)
 {
+	kvm_workaround_1542418_vmid_rollover();
 }
 
-void force_vm_exit(const cpumask_t *mask)
+static void force_vmid_rollover_exit(const cpumask_t *mask)
 {
 	preempt_disable();
-	smp_call_function_many(mask, exit_vm_noop, NULL, true);
+	smp_call_function_many(mask, exit_vmid_rollover, NULL, true);
+	kvm_workaround_1542418_vmid_rollover();
 	preempt_enable();
 }
 
@@ -518,10 +526,10 @@ static void update_vmid(struct kvm_vmid *vmid)
 
 		/*
 		 * On SMP we know no other CPUs can use this CPU's or each
-		 * other's VMID after force_vm_exit returns since the
+		 * other's VMID after force_vmid_rollover_exit returns since the
 		 * kvm_vmid_lock blocks them from reentry to the guest.
 		 */
-		force_vm_exit(cpu_all_mask);
+		force_vmid_rollover_exit(cpu_all_mask);
 		/*
 		 * Now broadcast TLB + ICACHE invalidation over the inner
 		 * shareable domain to make sure all data structures are
-- 
2.23.0

[PATCH 5/5] KVM: arm/arm64: Don't invoke defacto-CnP on first run

[Suzuki K Poulose]

From: James Morse <james.morse@arm.com>

When KVM finds itself switching between two vCPUs of the same VM
on one physical CPU it has to invalidate the TLB for this VMID
to avoid unintended sharing of TLB entries between vCPU.

This is done by tracking the 'last_vcpu_ran' as a percpu variable
for each vm.

kvm_arch_init_vm() is careful to initialise these to an impossible
vcpu id, but we never check for this. The first time
vm_arch_vcpu_load() is called on a new physical CPU, we will fail
the last_ran check and invalidate the TLB.

Now that we have an errata workaround in this path, it means we
trigger the workaround whenever a guest is migrated to a new CPU.

Check for the impossible vcpu id, and skip defacto-CnP.

Signed-off-by: James Morse <james.morse@arm.com>
Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
---
 virt/kvm/arm/arm.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
index ac9e017df7c9..6f729739cf6f 100644
--- a/virt/kvm/arm/arm.c
+++ b/virt/kvm/arm/arm.c
@@ -366,7 +366,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 	 * We might get preempted before the vCPU actually runs, but
 	 * over-invalidation doesn't affect correctness.
 	 */
-	if (*last_ran != vcpu->vcpu_id) {
+	if (*last_ran != -1 && *last_ran != vcpu->vcpu_id) {
 		kvm_call_hyp(__kvm_tlb_flush_local_vmid, vcpu);
 
 		/*
@@ -374,9 +374,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 		 *  conditions for Cortex-A77 erratum 1542418.
 		 */
 		kvm_workaround_1542418_vmid_rollover();
-
-		*last_ran = vcpu->vcpu_id;
 	}
+	*last_ran = vcpu->vcpu_id;
 
 	vcpu->cpu = cpu;
 	vcpu->arch.host_cpu_context = &cpu_data->host_ctxt;
-- 
2.23.0

Re: [PATCH 0/5] arm64: Add workaround for Cortex-A77 erratum 1542418

[Will Deacon]

Hi Suzuki,

On Thu, Nov 14, 2019 at 02:59:13PM +0000, Suzuki K Poulose wrote:
> This series adds workaround for Arm erratum 1542418 which affects

Searching for that erratum number doesn't find me a description :(

> Cortex-A77 cores (r0p0 - r1p0). Affected cores may execute stale
> instructions from the L0 macro-op cache violating the
> prefetch-speculation-protection guaranteed by the architecture.
> This happens when the when the branch predictor bases its predictions
> on a branch at this address on the stale history due to ASID or VMID
> reuse.

Two immediate questions:

 1. Can we disable the L0 MOP cache?
 2. Can we invalidate the branch predictor? If Spectre-v2 taught us
    anything it's that removing those instructions was a mistake!

Moving on...

Have you reproduced this at top-level? If I recall the
prefetch-speculation-protection, it's designed to protect against the
case where you have a direct branch:

addr:	B	foo

and another CPU writes out a new function:

bar:
	insn0
	...
	insnN

before doing any necessary maintenance and then patches the original
branch to:

addr:	B	bar

The idea is that a concurrently executing CPU could mispredict the original
branch to point at 'bar', fetch the instructions before they've been written
out and then confirm the prediction by looking at the newly written branch
instruction. Even without the prefetch-speculation-protection, that's
fairly difficult to achieve in practice: you'd need to be doing something
like reusing memory to hold the instructions so that the initial
misprediction occurs.

How does A77 stop this from occurring when the ASID is not reallocated (e.g.
the example above)? Is the MOP cache flushed somehow?

With this erratum, it sounds like you have to end up reusing an ASID from
a task that had a branch at 'addr' in its address space that branched to
the address of 'bar' (again. in its address space). Is that right? That
sounds super rare to me, particularly with ASLR: not only does the aliasing
branch need to exist, but it needs to be held in the branch predictor while
we cycle through 64k ASIDs *and* the race with the writer needs to happen
so that we get stale instructions from the MOP cache.

Is there something I'm missing that makes this remotely plausible?

Will

Re: [PATCH 0/5] arm64: Add workaround for Cortex-A77 erratum 1542418

[Suzuki K Poulose]

Hi Will

On 11/14/2019 04:39 PM, Will Deacon wrote:
> Hi Suzuki,
> 
> On Thu, Nov 14, 2019 at 02:59:13PM +0000, Suzuki K Poulose wrote:
>> This series adds workaround for Arm erratum 1542418 which affects
> 
> Searching for that erratum number doesn't find me a description :(

I believe this was published in the Cortex-A77 SDEN v9.0. I will chase
it internally.

> 
>> Cortex-A77 cores (r0p0 - r1p0). Affected cores may execute stale
>> instructions from the L0 macro-op cache violating the
>> prefetch-speculation-protection guaranteed by the architecture.
>> This happens when the when the branch predictor bases its predictions
>> on a branch at this address on the stale history due to ASID or VMID
>> reuse.
> 
> Two immediate questions:
> 
>   1. Can we disable the L0 MOP cache?
Yes, but it hurts performance.

>   2. Can we invalidate the branch predictor? If Spectre-v2 taught us
>      anything it's that removing those instructions was a mistake!

The workaround suggested is actually invalidating the branch history
but in a costly way. I am unaware of any.
> Moving on...
> 
> Have you reproduced this at top-level? If I recall the
> prefetch-speculation-protection, it's designed to protect against the
> case where you have a direct branch:

No, see below.

> 
> addr:	B	foo
> 
> and another CPU writes out a new function:
> 
> bar:
> 	insn0
> 	...
> 	insnN
> 
> before doing any necessary maintenance and then patches the original
> branch to:
> 
> addr:	B	bar
> 
> The idea is that a concurrently executing CPU could mispredict the original
> branch to point at 'bar', fetch the instructions before they've been written
> out and then confirm the prediction by looking at the newly written branch
> instruction. Even without the prefetch-speculation-protection, that's
> fairly difficult to achieve in practice: you'd need to be doing something
> like reusing memory to hold the instructions so that the initial
> misprediction occurs.
> 
> How does A77 stop this from occurring when the ASID is not reallocated (e.g.
> the example above)? Is the MOP cache flushed somehow?

IIUC, The MOP cache is flushed on I-cache invalidate, thus it is fine.	

> 
> With this erratum, it sounds like you have to end up reusing an ASID from
> a task that had a branch at 'addr' in its address space that branched to
> the address of 'bar' (again. in its address space). Is that right? That
> sounds super rare to me, particularly with ASLR: not only does the aliasing

AFAICS, yes and on top of that, it should also miss "addr" in MOP-cache
and hit "bar" before the I-cache invalidate is received. This may cause
the "bar" to be fetched from mop (and is not canceled even though there
was a mop-flush triggered by the i-cache invalidate after the hit) and
"addr" should miss in I-cache, causing it to fetch the updated instruction.

Also this means that the new context must not have executed "addr"
(which would give a hit in MOP-cache) while "bar" was fetched. So,
this adds on more constraints to actually hit it.

> branch need to exist, but it needs to be held in the branch predictor while
> we cycle through 64k ASIDs *and* the race with the writer needs to happen
> so that we get stale instructions from the MOP cache.
> 
> Is there something I'm missing that makes this remotely plausible?

No :-)

Cheers
Suzuki

Re: [PATCH 0/5] arm64: Add workaround for Cortex-A77 erratum 1542418

[Will Deacon]

On Fri, Nov 15, 2019 at 01:14:07AM +0000, Suzuki K Poulose wrote:
> On 11/14/2019 04:39 PM, Will Deacon wrote:
> > On Thu, Nov 14, 2019 at 02:59:13PM +0000, Suzuki K Poulose wrote:
> > > This series adds workaround for Arm erratum 1542418 which affects
> > > Cortex-A77 cores (r0p0 - r1p0). Affected cores may execute stale
> > > instructions from the L0 macro-op cache violating the
> > > prefetch-speculation-protection guaranteed by the architecture.
> > > This happens when the when the branch predictor bases its predictions
> > > on a branch at this address on the stale history due to ASID or VMID
> > > reuse.
> > 
> > Two immediate questions:
> > 
> >   1. Can we disable the L0 MOP cache?
> Yes, but it hurts performance.
> 
> >   2. Can we invalidate the branch predictor? If Spectre-v2 taught us
> >      anything it's that removing those instructions was a mistake!
> 
> The workaround suggested is actually invalidating the branch history
> but in a costly way. I am unaware of any.
> > Moving on...
> > 
> > Have you reproduced this at top-level? If I recall the
> > prefetch-speculation-protection, it's designed to protect against the
> > case where you have a direct branch:
> 
> No, see below.
> 
> > 
> > addr:	B	foo
> > 
> > and another CPU writes out a new function:
> > 
> > bar:
> > 	insn0
> > 	...
> > 	insnN
> > 
> > before doing any necessary maintenance and then patches the original
> > branch to:
> > 
> > addr:	B	bar
> > 
> > The idea is that a concurrently executing CPU could mispredict the original
> > branch to point at 'bar', fetch the instructions before they've been written
> > out and then confirm the prediction by looking at the newly written branch
> > instruction. Even without the prefetch-speculation-protection, that's
> > fairly difficult to achieve in practice: you'd need to be doing something
> > like reusing memory to hold the instructions so that the initial
> > misprediction occurs.
> > 
> > How does A77 stop this from occurring when the ASID is not reallocated (e.g.
> > the example above)? Is the MOP cache flushed somehow?
> 
> IIUC, The MOP cache is flushed on I-cache invalidate, thus it is fine.	

Hmm, so this is interesting. Does that mean we could do a local I-cache
invalidation in check_and_switch_context() at the same as doing the local
TLBI after a rollover?

I still don't grok the failure case, though, because assuming A77 has IDC=0,
then won't you see the I-cache maintenance from userspace anyway?

Will

Re: [PATCH 0/5] arm64: Add workaround for Cortex-A77 erratum 1542418

[Will Deacon]

Hi Suzuki,

On Wed, Nov 20, 2019 at 07:18:55PM +0000, Will Deacon wrote:
> On Fri, Nov 15, 2019 at 01:14:07AM +0000, Suzuki K Poulose wrote:
> > On 11/14/2019 04:39 PM, Will Deacon wrote:
> > > addr:	B	foo
> > > 
> > > and another CPU writes out a new function:
> > > 
> > > bar:
> > > 	insn0
> > > 	...
> > > 	insnN
> > > 
> > > before doing any necessary maintenance and then patches the original
> > > branch to:
> > > 
> > > addr:	B	bar
> > > 
> > > The idea is that a concurrently executing CPU could mispredict the original
> > > branch to point at 'bar', fetch the instructions before they've been written
> > > out and then confirm the prediction by looking at the newly written branch
> > > instruction. Even without the prefetch-speculation-protection, that's
> > > fairly difficult to achieve in practice: you'd need to be doing something
> > > like reusing memory to hold the instructions so that the initial
> > > misprediction occurs.
> > > 
> > > How does A77 stop this from occurring when the ASID is not reallocated (e.g.
> > > the example above)? Is the MOP cache flushed somehow?
> > 
> > IIUC, The MOP cache is flushed on I-cache invalidate, thus it is fine.	
> 
> Hmm, so this is interesting. Does that mean we could do a local I-cache
> invalidation in check_and_switch_context() at the same as doing the local
> TLBI after a rollover?
> 
> I still don't grok the failure case, though, because assuming A77 has IDC=0,
> then won't you see the I-cache maintenance from userspace anyway?

Please could you explain why the userspace I-cache maintenance doesn't
resolve the issue here?

Thanks,

Will