[PATCH 00/27] KVM: arm64: NV trap forwarding infrastructure

[PATCH 00/27] KVM: arm64: NV trap forwarding infrastructure

[Marc Zyngier]

As people are getting tired of seeing the full NV series, I've
extracted some of the easy stuff which I'm targeting for 6.6.

This implements the so called "trap forwarding" infrastructure, which
gets used when we take a trap from an L2 guest and that the L1 guest
wants to see the trap for itself.

Most of the series is pretty boring stuff, mostly a long list of
encodings which are mapped to a set of trap bits. I swear they are
correct. Sort of.

The interesting bit is around how we compute the trap result, which is
pretty complex due to the layers of crap the architecture has piled
over the years (a single op can be trapped by multiple coarse grained
trap bits, or a fine grained trap bit, which may itself be conditioned
by another control bit -- madness).

This also results in some rework of both the FGT stuff (for which I
carry a patch from Mark) and newly introduced the HCRX support.

With that (and the rest of the NV series[1]), FGT gets exposed to guests
and the trapping seems to work as expected.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/nv-6.6-WIP

Marc Zyngier (26):
  arm64: Add missing VA CMO encodings
  arm64: Add missing ERX*_EL1 encodings
  arm64: Add missing DC ZVA/GVA/GZVA encodings
  arm64: Add TLBI operation encodings
  arm64: Add AT operation encodings
  arm64: Add debug registers affected by HDFGxTR_EL2
  arm64: Add missing BRB/CFP/DVP/CPP instructions
  arm64: Fix HFGxTR_EL2 field naming
  arm64: Add HDFGRTR_EL2 and HDFGWTR_EL2 layouts
  KVM: arm64: Correctly handle ACCDATA_EL1 traps
  KVM: arm64: Add missing HCR_EL2 trap bits
  KVM: arm64: nv: Add FGT registers
  KVM: arm64: Restructure FGT register switching
  KVM: arm64: nv: Add trap forwarding infrastructure
  KVM: arm64: nv: Add trap forwarding for HCR_EL2
  KVM: arm64: nv: Expose FEAT_EVT to nested guests
  KVM: arm64: nv: Add trap forwarding for MDCR_EL2
  KVM: arm64: nv: Add trap forwarding for CNTHCTL_EL2
  KVM: arm64: nv: Add trap forwarding for HFGxTR_EL2
  KVM: arm64: nv: Add trap forwarding for HFGITR_EL2
  KVM: arm64: nv: Add trap forwarding for HDFGxTR_EL2
  KVM: arm64: nv: Add SVC trap forwarding
  KVM: arm64: nv: Add switching support for HFGxTR/HDFGxTR
  KVM: arm64: nv: Expose FGT to nested guests
  KVM: arm64: Move HCRX_EL2 switch to load/put on VHE systems
  KVM: arm64: nv: Add support for HCRX_EL2

Mark Brown (1):
  arm64: Add feature detection for fine grained traps

 arch/arm64/include/asm/kvm_arm.h        |   50 +
 arch/arm64/include/asm/kvm_host.h       |    7 +
 arch/arm64/include/asm/kvm_nested.h     |    2 +
 arch/arm64/include/asm/sysreg.h         |  270 +++-
 arch/arm64/kernel/cpufeature.c          |   11 +
 arch/arm64/kvm/arm.c                    |    4 +
 arch/arm64/kvm/emulate-nested.c         | 1703 +++++++++++++++++++++++
 arch/arm64/kvm/handle_exit.c            |   12 +
 arch/arm64/kvm/hyp/include/hyp/switch.h |  126 +-
 arch/arm64/kvm/nested.c                 |   11 +-
 arch/arm64/kvm/sys_regs.c               |   15 +
 arch/arm64/kvm/trace_arm.h              |   19 +
 arch/arm64/tools/cpucaps                |    1 +
 arch/arm64/tools/sysreg                 |  141 +-
 14 files changed, 2326 insertions(+), 46 deletions(-)

-- 
2.34.1

[PATCH 01/27] arm64: Add missing VA CMO encodings

[Marc Zyngier]

Add the missing VA-based CMOs encodings.

Reviewed-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/sysreg.h | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index b481935e9314..85447e68951a 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -124,6 +124,32 @@
 #define SYS_DC_CIGSW			sys_insn(1, 0, 7, 14, 4)
 #define SYS_DC_CIGDSW			sys_insn(1, 0, 7, 14, 6)
 
+#define SYS_IC_IALLUIS			sys_insn(1, 0, 7, 1, 0)
+#define SYS_IC_IALLU			sys_insn(1, 0, 7, 5, 0)
+#define SYS_IC_IVAU			sys_insn(1, 3, 7, 5, 1)
+
+#define SYS_DC_IVAC			sys_insn(1, 0, 7, 6, 1)
+#define SYS_DC_IGVAC			sys_insn(1, 0, 7, 6, 3)
+#define SYS_DC_IGDVAC			sys_insn(1, 0, 7, 6, 5)
+
+#define SYS_DC_CVAC			sys_insn(1, 3, 7, 10, 1)
+#define SYS_DC_CGVAC			sys_insn(1, 3, 7, 10, 3)
+#define SYS_DC_CGDVAC			sys_insn(1, 3, 7, 10, 5)
+
+#define SYS_DC_CVAU			sys_insn(1, 3, 7, 11, 1)
+
+#define SYS_DC_CVAP			sys_insn(1, 3, 7, 12, 1)
+#define SYS_DC_CGVAP			sys_insn(1, 3, 7, 12, 3)
+#define SYS_DC_CGDVAP			sys_insn(1, 3, 7, 12, 5)
+
+#define SYS_DC_CVADP			sys_insn(1, 3, 7, 13, 1)
+#define SYS_DC_CGVADP			sys_insn(1, 3, 7, 13, 3)
+#define SYS_DC_CGDVADP			sys_insn(1, 3, 7, 13, 5)
+
+#define SYS_DC_CIVAC			sys_insn(1, 3, 7, 14, 1)
+#define SYS_DC_CIGVAC			sys_insn(1, 3, 7, 14, 3)
+#define SYS_DC_CIGDVAC			sys_insn(1, 3, 7, 14, 5)
+
 /*
  * Automatically generated definitions for system registers, the
  * manual encodings below are in the process of being converted to
-- 
2.34.1

[PATCH 02/27] arm64: Add missing ERX*_EL1 encodings

[Marc Zyngier]

We only describe a few of the ERX*_EL1 registers. Add the missing
ones (ERXPFGF_EL1, ERXPFGCTL_EL1, ERXPFGCDN_EL1, ERXMISC2_EL1 and
ERXMISC3_EL1).

Reviewed-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/sysreg.h | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index 85447e68951a..ed2739897859 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -229,8 +229,13 @@
 #define SYS_ERXCTLR_EL1			sys_reg(3, 0, 5, 4, 1)
 #define SYS_ERXSTATUS_EL1		sys_reg(3, 0, 5, 4, 2)
 #define SYS_ERXADDR_EL1			sys_reg(3, 0, 5, 4, 3)
+#define SYS_ERXPFGF_EL1			sys_reg(3, 0, 5, 4, 4)
+#define SYS_ERXPFGCTL_EL1		sys_reg(3, 0, 5, 4, 5)
+#define SYS_ERXPFGCDN_EL1		sys_reg(3, 0, 5, 4, 6)
 #define SYS_ERXMISC0_EL1		sys_reg(3, 0, 5, 5, 0)
 #define SYS_ERXMISC1_EL1		sys_reg(3, 0, 5, 5, 1)
+#define SYS_ERXMISC2_EL1		sys_reg(3, 0, 5, 5, 2)
+#define SYS_ERXMISC3_EL1		sys_reg(3, 0, 5, 5, 3)
 #define SYS_TFSR_EL1			sys_reg(3, 0, 5, 6, 0)
 #define SYS_TFSRE0_EL1			sys_reg(3, 0, 5, 6, 1)
 
-- 
2.34.1

[PATCH 03/27] arm64: Add missing DC ZVA/GVA/GZVA encodings

[Marc Zyngier]

Add the missing DC *VA encodings.

Reviewed-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/sysreg.h | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index ed2739897859..5084add86897 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -150,6 +150,11 @@
 #define SYS_DC_CIGVAC			sys_insn(1, 3, 7, 14, 3)
 #define SYS_DC_CIGDVAC			sys_insn(1, 3, 7, 14, 5)
 
+/* Data cache zero operations */
+#define SYS_DC_ZVA			sys_insn(1, 3, 7, 4, 1)
+#define SYS_DC_GVA			sys_insn(1, 3, 7, 4, 3)
+#define SYS_DC_GZVA			sys_insn(1, 3, 7, 4, 4)
+
 /*
  * Automatically generated definitions for system registers, the
  * manual encodings below are in the process of being converted to
-- 
2.34.1

[PATCH 04/27] arm64: Add TLBI operation encodings

[Marc Zyngier]

Add all the TLBI encodings that are usable from Non-Secure.

Reviewed-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/sysreg.h | 128 ++++++++++++++++++++++++++++++++
 1 file changed, 128 insertions(+)

diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index 5084add86897..72e18480ce62 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -514,6 +514,134 @@
 
 #define SYS_SP_EL2			sys_reg(3, 6,  4, 1, 0)
 
+/* TLBI instructions */
+#define OP_TLBI_VMALLE1OS		sys_insn(1, 0, 8, 1, 0)
+#define OP_TLBI_VAE1OS			sys_insn(1, 0, 8, 1, 1)
+#define OP_TLBI_ASIDE1OS		sys_insn(1, 0, 8, 1, 2)
+#define OP_TLBI_VAAE1OS			sys_insn(1, 0, 8, 1, 3)
+#define OP_TLBI_VALE1OS			sys_insn(1, 0, 8, 1, 5)
+#define OP_TLBI_VAALE1OS		sys_insn(1, 0, 8, 1, 7)
+#define OP_TLBI_RVAE1IS			sys_insn(1, 0, 8, 2, 1)
+#define OP_TLBI_RVAAE1IS		sys_insn(1, 0, 8, 2, 3)
+#define OP_TLBI_RVALE1IS		sys_insn(1, 0, 8, 2, 5)
+#define OP_TLBI_RVAALE1IS		sys_insn(1, 0, 8, 2, 7)
+#define OP_TLBI_VMALLE1IS		sys_insn(1, 0, 8, 3, 0)
+#define OP_TLBI_VAE1IS			sys_insn(1, 0, 8, 3, 1)
+#define OP_TLBI_ASIDE1IS		sys_insn(1, 0, 8, 3, 2)
+#define OP_TLBI_VAAE1IS			sys_insn(1, 0, 8, 3, 3)
+#define OP_TLBI_VALE1IS			sys_insn(1, 0, 8, 3, 5)
+#define OP_TLBI_VAALE1IS		sys_insn(1, 0, 8, 3, 7)
+#define OP_TLBI_RVAE1OS			sys_insn(1, 0, 8, 5, 1)
+#define OP_TLBI_RVAAE1OS		sys_insn(1, 0, 8, 5, 3)
+#define OP_TLBI_RVALE1OS		sys_insn(1, 0, 8, 5, 5)
+#define OP_TLBI_RVAALE1OS		sys_insn(1, 0, 8, 5, 7)
+#define OP_TLBI_RVAE1			sys_insn(1, 0, 8, 6, 1)
+#define OP_TLBI_RVAAE1			sys_insn(1, 0, 8, 6, 3)
+#define OP_TLBI_RVALE1			sys_insn(1, 0, 8, 6, 5)
+#define OP_TLBI_RVAALE1			sys_insn(1, 0, 8, 6, 7)
+#define OP_TLBI_VMALLE1			sys_insn(1, 0, 8, 7, 0)
+#define OP_TLBI_VAE1			sys_insn(1, 0, 8, 7, 1)
+#define OP_TLBI_ASIDE1			sys_insn(1, 0, 8, 7, 2)
+#define OP_TLBI_VAAE1			sys_insn(1, 0, 8, 7, 3)
+#define OP_TLBI_VALE1			sys_insn(1, 0, 8, 7, 5)
+#define OP_TLBI_VAALE1			sys_insn(1, 0, 8, 7, 7)
+#define OP_TLBI_VMALLE1OSNXS		sys_insn(1, 0, 9, 1, 0)
+#define OP_TLBI_VAE1OSNXS		sys_insn(1, 0, 9, 1, 1)
+#define OP_TLBI_ASIDE1OSNXS		sys_insn(1, 0, 9, 1, 2)
+#define OP_TLBI_VAAE1OSNXS		sys_insn(1, 0, 9, 1, 3)
+#define OP_TLBI_VALE1OSNXS		sys_insn(1, 0, 9, 1, 5)
+#define OP_TLBI_VAALE1OSNXS		sys_insn(1, 0, 9, 1, 7)
+#define OP_TLBI_RVAE1ISNXS		sys_insn(1, 0, 9, 2, 1)
+#define OP_TLBI_RVAAE1ISNXS		sys_insn(1, 0, 9, 2, 3)
+#define OP_TLBI_RVALE1ISNXS		sys_insn(1, 0, 9, 2, 5)
+#define OP_TLBI_RVAALE1ISNXS		sys_insn(1, 0, 9, 2, 7)
+#define OP_TLBI_VMALLE1ISNXS		sys_insn(1, 0, 9, 3, 0)
+#define OP_TLBI_VAE1ISNXS		sys_insn(1, 0, 9, 3, 1)
+#define OP_TLBI_ASIDE1ISNXS		sys_insn(1, 0, 9, 3, 2)
+#define OP_TLBI_VAAE1ISNXS		sys_insn(1, 0, 9, 3, 3)
+#define OP_TLBI_VALE1ISNXS		sys_insn(1, 0, 9, 3, 5)
+#define OP_TLBI_VAALE1ISNXS		sys_insn(1, 0, 9, 3, 7)
+#define OP_TLBI_RVAE1OSNXS		sys_insn(1, 0, 9, 5, 1)
+#define OP_TLBI_RVAAE1OSNXS		sys_insn(1, 0, 9, 5, 3)
+#define OP_TLBI_RVALE1OSNXS		sys_insn(1, 0, 9, 5, 5)
+#define OP_TLBI_RVAALE1OSNXS		sys_insn(1, 0, 9, 5, 7)
+#define OP_TLBI_RVAE1NXS		sys_insn(1, 0, 9, 6, 1)
+#define OP_TLBI_RVAAE1NXS		sys_insn(1, 0, 9, 6, 3)
+#define OP_TLBI_RVALE1NXS		sys_insn(1, 0, 9, 6, 5)
+#define OP_TLBI_RVAALE1NXS		sys_insn(1, 0, 9, 6, 7)
+#define OP_TLBI_VMALLE1NXS		sys_insn(1, 0, 9, 7, 0)
+#define OP_TLBI_VAE1NXS			sys_insn(1, 0, 9, 7, 1)
+#define OP_TLBI_ASIDE1NXS		sys_insn(1, 0, 9, 7, 2)
+#define OP_TLBI_VAAE1NXS		sys_insn(1, 0, 9, 7, 3)
+#define OP_TLBI_VALE1NXS		sys_insn(1, 0, 9, 7, 5)
+#define OP_TLBI_VAALE1NXS		sys_insn(1, 0, 9, 7, 7)
+#define OP_TLBI_IPAS2E1IS		sys_insn(1, 4, 8, 0, 1)
+#define OP_TLBI_RIPAS2E1IS		sys_insn(1, 4, 8, 0, 2)
+#define OP_TLBI_IPAS2LE1IS		sys_insn(1, 4, 8, 0, 5)
+#define OP_TLBI_RIPAS2LE1IS		sys_insn(1, 4, 8, 0, 6)
+#define OP_TLBI_ALLE2OS			sys_insn(1, 4, 8, 1, 0)
+#define OP_TLBI_VAE2OS			sys_insn(1, 4, 8, 1, 1)
+#define OP_TLBI_ALLE1OS			sys_insn(1, 4, 8, 1, 4)
+#define OP_TLBI_VALE2OS			sys_insn(1, 4, 8, 1, 5)
+#define OP_TLBI_VMALLS12E1OS		sys_insn(1, 4, 8, 1, 6)
+#define OP_TLBI_RVAE2IS			sys_insn(1, 4, 8, 2, 1)
+#define OP_TLBI_RVALE2IS		sys_insn(1, 4, 8, 2, 5)
+#define OP_TLBI_ALLE2IS			sys_insn(1, 4, 8, 3, 0)
+#define OP_TLBI_VAE2IS			sys_insn(1, 4, 8, 3, 1)
+#define OP_TLBI_ALLE1IS			sys_insn(1, 4, 8, 3, 4)
+#define OP_TLBI_VALE2IS			sys_insn(1, 4, 8, 3, 5)
+#define OP_TLBI_VMALLS12E1IS		sys_insn(1, 4, 8, 3, 6)
+#define OP_TLBI_IPAS2E1OS		sys_insn(1, 4, 8, 4, 0)
+#define OP_TLBI_IPAS2E1			sys_insn(1, 4, 8, 4, 1)
+#define OP_TLBI_RIPAS2E1		sys_insn(1, 4, 8, 4, 2)
+#define OP_TLBI_RIPAS2E1OS		sys_insn(1, 4, 8, 4, 3)
+#define OP_TLBI_IPAS2LE1OS		sys_insn(1, 4, 8, 4, 4)
+#define OP_TLBI_IPAS2LE1		sys_insn(1, 4, 8, 4, 5)
+#define OP_TLBI_RIPAS2LE1		sys_insn(1, 4, 8, 4, 6)
+#define OP_TLBI_RIPAS2LE1OS		sys_insn(1, 4, 8, 4, 7)
+#define OP_TLBI_RVAE2OS			sys_insn(1, 4, 8, 5, 1)
+#define OP_TLBI_RVALE2OS		sys_insn(1, 4, 8, 5, 5)
+#define OP_TLBI_RVAE2			sys_insn(1, 4, 8, 6, 1)
+#define OP_TLBI_RVALE2			sys_insn(1, 4, 8, 6, 5)
+#define OP_TLBI_ALLE2			sys_insn(1, 4, 8, 7, 0)
+#define OP_TLBI_VAE2			sys_insn(1, 4, 8, 7, 1)
+#define OP_TLBI_ALLE1			sys_insn(1, 4, 8, 7, 4)
+#define OP_TLBI_VALE2			sys_insn(1, 4, 8, 7, 5)
+#define OP_TLBI_VMALLS12E1		sys_insn(1, 4, 8, 7, 6)
+#define OP_TLBI_IPAS2E1ISNXS		sys_insn(1, 4, 9, 0, 1)
+#define OP_TLBI_RIPAS2E1ISNXS		sys_insn(1, 4, 9, 0, 2)
+#define OP_TLBI_IPAS2LE1ISNXS		sys_insn(1, 4, 9, 0, 5)
+#define OP_TLBI_RIPAS2LE1ISNXS		sys_insn(1, 4, 9, 0, 6)
+#define OP_TLBI_ALLE2OSNXS		sys_insn(1, 4, 9, 1, 0)
+#define OP_TLBI_VAE2OSNXS		sys_insn(1, 4, 9, 1, 1)
+#define OP_TLBI_ALLE1OSNXS		sys_insn(1, 4, 9, 1, 4)
+#define OP_TLBI_VALE2OSNXS		sys_insn(1, 4, 9, 1, 5)
+#define OP_TLBI_VMALLS12E1OSNXS		sys_insn(1, 4, 9, 1, 6)
+#define OP_TLBI_RVAE2ISNXS		sys_insn(1, 4, 9, 2, 1)
+#define OP_TLBI_RVALE2ISNXS		sys_insn(1, 4, 9, 2, 5)
+#define OP_TLBI_ALLE2ISNXS		sys_insn(1, 4, 9, 3, 0)
+#define OP_TLBI_VAE2ISNXS		sys_insn(1, 4, 9, 3, 1)
+#define OP_TLBI_ALLE1ISNXS		sys_insn(1, 4, 9, 3, 4)
+#define OP_TLBI_VALE2ISNXS		sys_insn(1, 4, 9, 3, 5)
+#define OP_TLBI_VMALLS12E1ISNXS		sys_insn(1, 4, 9, 3, 6)
+#define OP_TLBI_IPAS2E1OSNXS		sys_insn(1, 4, 9, 4, 0)
+#define OP_TLBI_IPAS2E1NXS		sys_insn(1, 4, 9, 4, 1)
+#define OP_TLBI_RIPAS2E1NXS		sys_insn(1, 4, 9, 4, 2)
+#define OP_TLBI_RIPAS2E1OSNXS		sys_insn(1, 4, 9, 4, 3)
+#define OP_TLBI_IPAS2LE1OSNXS		sys_insn(1, 4, 9, 4, 4)
+#define OP_TLBI_IPAS2LE1NXS		sys_insn(1, 4, 9, 4, 5)
+#define OP_TLBI_RIPAS2LE1NXS		sys_insn(1, 4, 9, 4, 6)
+#define OP_TLBI_RIPAS2LE1OSNXS		sys_insn(1, 4, 9, 4, 7)
+#define OP_TLBI_RVAE2OSNXS		sys_insn(1, 4, 9, 5, 1)
+#define OP_TLBI_RVALE2OSNXS		sys_insn(1, 4, 9, 5, 5)
+#define OP_TLBI_RVAE2NXS		sys_insn(1, 4, 9, 6, 1)
+#define OP_TLBI_RVALE2NXS		sys_insn(1, 4, 9, 6, 5)
+#define OP_TLBI_ALLE2NXS		sys_insn(1, 4, 9, 7, 0)
+#define OP_TLBI_VAE2NXS			sys_insn(1, 4, 9, 7, 1)
+#define OP_TLBI_ALLE1NXS		sys_insn(1, 4, 9, 7, 4)
+#define OP_TLBI_VALE2NXS		sys_insn(1, 4, 9, 7, 5)
+#define OP_TLBI_VMALLS12E1NXS		sys_insn(1, 4, 9, 7, 6)
+
 /* Common SCTLR_ELx flags. */
 #define SCTLR_ELx_ENTP2	(BIT(60))
 #define SCTLR_ELx_DSSBS	(BIT(44))
-- 
2.34.1

[PATCH 05/27] arm64: Add AT operation encodings

[Marc Zyngier]

Add the encodings for the AT operation that are usable from NS.

Reviewed-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/sysreg.h | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index 72e18480ce62..76289339b43b 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -514,6 +514,23 @@
 
 #define SYS_SP_EL2			sys_reg(3, 6,  4, 1, 0)
 
+/* AT instructions */
+#define AT_Op0 1
+#define AT_CRn 7
+
+#define OP_AT_S1E1R	sys_insn(AT_Op0, 0, AT_CRn, 8, 0)
+#define OP_AT_S1E1W	sys_insn(AT_Op0, 0, AT_CRn, 8, 1)
+#define OP_AT_S1E0R	sys_insn(AT_Op0, 0, AT_CRn, 8, 2)
+#define OP_AT_S1E0W	sys_insn(AT_Op0, 0, AT_CRn, 8, 3)
+#define OP_AT_S1E1RP	sys_insn(AT_Op0, 0, AT_CRn, 9, 0)
+#define OP_AT_S1E1WP	sys_insn(AT_Op0, 0, AT_CRn, 9, 1)
+#define OP_AT_S1E2R	sys_insn(AT_Op0, 4, AT_CRn, 8, 0)
+#define OP_AT_S1E2W	sys_insn(AT_Op0, 4, AT_CRn, 8, 1)
+#define OP_AT_S12E1R	sys_insn(AT_Op0, 4, AT_CRn, 8, 4)
+#define OP_AT_S12E1W	sys_insn(AT_Op0, 4, AT_CRn, 8, 5)
+#define OP_AT_S12E0R	sys_insn(AT_Op0, 4, AT_CRn, 8, 6)
+#define OP_AT_S12E0W	sys_insn(AT_Op0, 4, AT_CRn, 8, 7)
+
 /* TLBI instructions */
 #define OP_TLBI_VMALLE1OS		sys_insn(1, 0, 8, 1, 0)
 #define OP_TLBI_VAE1OS			sys_insn(1, 0, 8, 1, 1)
-- 
2.34.1

[PATCH 06/27] arm64: Add debug registers affected by HDFGxTR_EL2

[Marc Zyngier]

The HDFGxTR_EL2 registers trap a (huge) set of debug and trace
related registers. Add their encodings (and only that, because
we really don't care about what these registers actually do at
this stage).

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/sysreg.h | 78 +++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)

diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index 76289339b43b..9dfd127be55a 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -194,6 +194,84 @@
 #define SYS_DBGDTRTX_EL0		sys_reg(2, 3, 0, 5, 0)
 #define SYS_DBGVCR32_EL2		sys_reg(2, 4, 0, 7, 0)
 
+#define SYS_BRBINF_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 0))
+#define SYS_BRBINFINJ_EL1		sys_reg(2, 1, 9, 1, 0)
+#define SYS_BRBSRC_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 1))
+#define SYS_BRBSRCINJ_EL1		sys_reg(2, 1, 9, 1, 1)
+#define SYS_BRBTGT_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 2))
+#define SYS_BRBTGTINJ_EL1		sys_reg(2, 1, 9, 1, 2)
+#define SYS_BRBTS_EL1			sys_reg(2, 1, 9, 0, 2)
+
+#define SYS_BRBCR_EL1			sys_reg(2, 1, 9, 0, 0)
+#define SYS_BRBFCR_EL1			sys_reg(2, 1, 9, 0, 1)
+#define SYS_BRBIDR0_EL1			sys_reg(2, 1, 9, 2, 0)
+
+#define SYS_TRCITECR_EL1		sys_reg(3, 0, 1, 2, 3)
+#define SYS_TRCITECR_EL1		sys_reg(3, 0, 1, 2, 3)
+#define SYS_TRCACATR(m)			sys_reg(2, 1, 2, ((m & 7) << 1), (2 | (m >> 3)))
+#define SYS_TRCACVR(m)			sys_reg(2, 1, 2, ((m & 7) << 1), (0 | (m >> 3)))
+#define SYS_TRCAUTHSTATUS		sys_reg(2, 1, 7, 14, 6)
+#define SYS_TRCAUXCTLR			sys_reg(2, 1, 0, 6, 0)
+#define SYS_TRCBBCTLR			sys_reg(2, 1, 0, 15, 0)
+#define SYS_TRCCCCTLR			sys_reg(2, 1, 0, 14, 0)
+#define SYS_TRCCIDCCTLR0		sys_reg(2, 1, 3, 0, 2)
+#define SYS_TRCCIDCCTLR1		sys_reg(2, 1, 3, 1, 2)
+#define SYS_TRCCIDCVR(m)		sys_reg(2, 1, 3, ((m & 7) << 1), 0)
+#define SYS_TRCCLAIMCLR			sys_reg(2, 1, 7, 9, 6)
+#define SYS_TRCCLAIMSET			sys_reg(2, 1, 7, 8, 6)
+#define SYS_TRCCNTCTLR(m)		sys_reg(2, 1, 0, (4 | (m & 3)), 5)
+#define SYS_TRCCNTRLDVR(m)		sys_reg(2, 1, 0, (0 | (m & 3)), 5)
+#define SYS_TRCCNTVR(m)			sys_reg(2, 1, 0, (8 | (m & 3)), 5)
+#define SYS_TRCCONFIGR			sys_reg(2, 1, 0, 4, 0)
+#define SYS_TRCDEVARCH			sys_reg(2, 1, 7, 15, 6)
+#define SYS_TRCDEVID			sys_reg(2, 1, 7, 2, 7)
+#define SYS_TRCEVENTCTL0R		sys_reg(2, 1, 0, 8, 0)
+#define SYS_TRCEVENTCTL1R		sys_reg(2, 1, 0, 9, 0)
+#define SYS_TRCEXTINSELR(m)		sys_reg(2, 1, 0, (8 | (m & 3)), 4)
+#define SYS_TRCIDR0			sys_reg(2, 1, 0, 8, 7)
+#define SYS_TRCIDR10			sys_reg(2, 1, 0, 2, 6)
+#define SYS_TRCIDR11			sys_reg(2, 1, 0, 3, 6)
+#define SYS_TRCIDR12			sys_reg(2, 1, 0, 4, 6)
+#define SYS_TRCIDR13			sys_reg(2, 1, 0, 5, 6)
+#define SYS_TRCIDR1			sys_reg(2, 1, 0, 9, 7)
+#define SYS_TRCIDR2			sys_reg(2, 1, 0, 10, 7)
+#define SYS_TRCIDR3			sys_reg(2, 1, 0, 11, 7)
+#define SYS_TRCIDR4			sys_reg(2, 1, 0, 12, 7)
+#define SYS_TRCIDR5			sys_reg(2, 1, 0, 13, 7)
+#define SYS_TRCIDR6			sys_reg(2, 1, 0, 14, 7)
+#define SYS_TRCIDR7			sys_reg(2, 1, 0, 15, 7)
+#define SYS_TRCIDR8			sys_reg(2, 1, 0, 0, 6)
+#define SYS_TRCIDR9			sys_reg(2, 1, 0, 1, 6)
+#define SYS_TRCIMSPEC0			sys_reg(2, 1, 0, 0, 7)
+#define SYS_TRCIMSPEC(m)		sys_reg(2, 1, 0, (m & 7), 7)
+#define SYS_TRCITEEDCR			sys_reg(2, 1, 0, 2, 1)
+#define SYS_TRCOSLSR			sys_reg(2, 1, 1, 1, 4)
+#define SYS_TRCPRGCTLR			sys_reg(2, 1, 0, 1, 0)
+#define SYS_TRCQCTLR			sys_reg(2, 1, 0, 1, 1)
+#define SYS_TRCRSCTLR(m)		sys_reg(2, 1, 1, (m & 15), (0 | (m >> 4)))
+#define SYS_TRCRSR			sys_reg(2, 1, 0, 10, 0)
+#define SYS_TRCSEQEVR(m)		sys_reg(2, 1, 0, (m & 3), 4)
+#define SYS_TRCSEQRSTEVR		sys_reg(2, 1, 0, 6, 4)
+#define SYS_TRCSEQSTR			sys_reg(2, 1, 0, 7, 4)
+#define SYS_TRCSSCCR(m)			sys_reg(2, 1, 1, (m & 7), 2)
+#define SYS_TRCSSCSR(m)			sys_reg(2, 1, 1, (8 | (m & 7)), 2)
+#define SYS_TRCSSPCICR(m)		sys_reg(2, 1, 1, (m & 7), 3)
+#define SYS_TRCSTALLCTLR		sys_reg(2, 1, 0, 11, 0)
+#define SYS_TRCSTATR			sys_reg(2, 1, 0, 3, 0)
+#define SYS_TRCSYNCPR			sys_reg(2, 1, 0, 13, 0)
+#define SYS_TRCTRACEIDR			sys_reg(2, 1, 0, 0, 1)
+#define SYS_TRCTSCTLR			sys_reg(2, 1, 0, 12, 0)
+#define SYS_TRCVICTLR			sys_reg(2, 1, 0, 0, 2)
+#define SYS_TRCVIIECTLR			sys_reg(2, 1, 0, 1, 2)
+#define SYS_TRCVIPCSSCTLR		sys_reg(2, 1, 0, 3, 2)
+#define SYS_TRCVISSCTLR			sys_reg(2, 1, 0, 2, 2)
+#define SYS_TRCVMIDCCTLR0		sys_reg(2, 1, 3, 2, 2)
+#define SYS_TRCVMIDCCTLR1		sys_reg(2, 1, 3, 3, 2)
+#define SYS_TRCVMIDCVR(m)		sys_reg(2, 1, 3, ((m & 7) << 1), 1)
+
+/* ETM */
+#define SYS_TRCOSLAR			sys_reg(2, 1, 1, 0, 4)
+
 #define SYS_MIDR_EL1			sys_reg(3, 0, 0, 0, 0)
 #define SYS_MPIDR_EL1			sys_reg(3, 0, 0, 0, 5)
 #define SYS_REVIDR_EL1			sys_reg(3, 0, 0, 0, 6)
-- 
2.34.1

[PATCH 07/27] arm64: Add missing BRB/CFP/DVP/CPP instructions

[Marc Zyngier]

HFGITR_EL2 traps a bunch of instructions for which we don't have
encodings yet. Add them.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/sysreg.h | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index 9dfd127be55a..e2357529c633 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -737,6 +737,13 @@
 #define OP_TLBI_VALE2NXS		sys_insn(1, 4, 9, 7, 5)
 #define OP_TLBI_VMALLS12E1NXS		sys_insn(1, 4, 9, 7, 6)
 
+/* Misc instructions */
+#define OP_BRB_IALL			sys_insn(1, 1, 7, 2, 4)
+#define OP_BRB_INJ			sys_insn(1, 1, 7, 2, 5)
+#define OP_CFP_RCTX			sys_insn(1, 3, 7, 3, 4)
+#define OP_DVP_RCTX			sys_insn(1, 3, 7, 3, 5)
+#define OP_CPP_RCTX			sys_insn(1, 3, 7, 3, 7)
+
 /* Common SCTLR_ELx flags. */
 #define SCTLR_ELx_ENTP2	(BIT(60))
 #define SCTLR_ELx_DSSBS	(BIT(44))
-- 
2.34.1

[PATCH 08/27] arm64: Fix HFGxTR_EL2 field naming

[Marc Zyngier]

The HFGxTR_EL2 fields do not always follow the naming described
in the spec, nor do they match the name of the register they trap
in the rest of the kernel.

It is a bit sad that they were written by hand despite the availability
of a machine readable version...

Fixes: cc077e7facbe ("arm64/sysreg: Convert HFG[RW]TR_EL2 to automatic generation")
Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Mark Brown <broonie@kernel.org>
Cc: Will Deacon <will@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
---
 arch/arm64/tools/sysreg | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg
index 1ea4a3dc68f8..65866bf819c3 100644
--- a/arch/arm64/tools/sysreg
+++ b/arch/arm64/tools/sysreg
@@ -2017,7 +2017,7 @@ Field	0	SM
 EndSysreg
 
 SysregFields	HFGxTR_EL2
-Field	63	nAMIAIR2_EL1
+Field	63	nAMAIR2_EL1
 Field	62	nMAIR2_EL1
 Field	61	nS2POR_EL1
 Field	60	nPOR_EL1
@@ -2032,9 +2032,9 @@ Field	52	nGCS_EL0
 Res0	51
 Field	50	nACCDATA_EL1
 Field	49	ERXADDR_EL1
-Field	48	EXRPFGCDN_EL1
-Field	47	EXPFGCTL_EL1
-Field	46	EXPFGF_EL1
+Field	48	ERXPFGCDN_EL1
+Field	47	ERXPFGCTL_EL1
+Field	46	ERXPFGF_EL1
 Field	45	ERXMISCn_EL1
 Field	44	ERXSTATUS_EL1
 Field	43	ERXCTLR_EL1
@@ -2049,8 +2049,8 @@ Field	35	TPIDR_EL0
 Field	34	TPIDRRO_EL0
 Field	33	TPIDR_EL1
 Field	32	TCR_EL1
-Field	31	SCTXNUM_EL0
-Field	30	SCTXNUM_EL1
+Field	31	SCXTNUM_EL0
+Field	30	SCXTNUM_EL1
 Field	29	SCTLR_EL1
 Field	28	REVIDR_EL1
 Field	27	PAR_EL1
-- 
2.34.1

[PATCH 09/27] arm64: Add HDFGRTR_EL2 and HDFGWTR_EL2 layouts

[Marc Zyngier]

As we're about to implement full support for FEAT_FGT, add the
full HDFGRTR_EL2 and HDFGWTR_EL2 layouts.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/sysreg.h |   2 -
 arch/arm64/tools/sysreg         | 129 ++++++++++++++++++++++++++++++++
 2 files changed, 129 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index e2357529c633..6d3d16fac227 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -497,8 +497,6 @@
 #define SYS_VTCR_EL2			sys_reg(3, 4, 2, 1, 2)
 
 #define SYS_TRFCR_EL2			sys_reg(3, 4, 1, 2, 1)
-#define SYS_HDFGRTR_EL2			sys_reg(3, 4, 3, 1, 4)
-#define SYS_HDFGWTR_EL2			sys_reg(3, 4, 3, 1, 5)
 #define SYS_HAFGRTR_EL2			sys_reg(3, 4, 3, 1, 6)
 #define SYS_SPSR_EL2			sys_reg(3, 4, 4, 0, 0)
 #define SYS_ELR_EL2			sys_reg(3, 4, 4, 0, 1)
diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg
index 65866bf819c3..2517ef7c21cf 100644
--- a/arch/arm64/tools/sysreg
+++ b/arch/arm64/tools/sysreg
@@ -2156,6 +2156,135 @@ Field	1	ICIALLU
 Field	0	ICIALLUIS
 EndSysreg
 
+Sysreg HDFGRTR_EL2	3	4	3	1	4
+Field	63	PMBIDR_EL1
+Field	62	nPMSNEVFR_EL1
+Field	61	nBRBDATA
+Field	60	nBRBCTL
+Field	59	nBRBIDR
+Field	58	PMCEIDn_EL0
+Field	57	PMUSERENR_EL0
+Field	56	TRBTRG_EL1
+Field	55	TRBSR_EL1
+Field	54	TRBPTR_EL1
+Field	53	TRBMAR_EL1
+Field	52	TRBLIMITR_EL1
+Field	51	TRBIDR_EL1
+Field	50	TRBBASER_EL1
+Res0	49
+Field	48	TRCVICTLR
+Field	47	TRCSTATR
+Field	46	TRCSSCSRn
+Field	45	TRCSEQSTR
+Field	44	TRCPRGCTLR
+Field	43	TRCOSLSR
+Res0	42
+Field	41	TRCIMSPECn
+Field	40	TRCID
+Res0	39:38
+Field	37	TRCCNTVRn
+Field	36	TRCCLAIM
+Field	35	TRCAUXCTLR
+Field	34	TRCAUTHSTATUS
+Field	33	TRC
+Field	32	PMSLATFR_EL1
+Field	31	PMSIRR_EL1
+Field	30	PMSIDR_EL1
+Field	29	PMSICR_EL1
+Field	28	PMSFCR_EL1
+Field	27	PMSEVFR_EL1
+Field	26	PMSCR_EL1
+Field	25	PMBSR_EL1
+Field	24	PMBPTR_EL1
+Field	23	PMBLIMITR_EL1
+Field	22	PMMIR_EL1
+Res0	21:20
+Field	19	PMSELR_EL0
+Field	18	PMOVS
+Field	17	PMINTEN
+Field	16	PMCNTEN
+Field	15	PMCCNTR_EL0
+Field	14	PMCCFILTR_EL0
+Field	13	PMEVTYPERn_EL0
+Field	12	PMEVCNTRn_EL0
+Field	11	OSDLR_EL1
+Field	10	OSECCR_EL1
+Field	9	OSLSR_EL1
+Res0	8
+Field	7	DBGPRCR_EL1
+Field	6	DBGAUTHSTATUS_EL1
+Field	5	DBGCLAIM
+Field	4	MDSCR_EL1
+Field	3	DBGWVRn_EL1
+Field	2	DBGWCRn_EL1
+Field	1	DBGBVRn_EL1
+Field	0	DBGBCRn_EL1
+EndSysreg
+
+Sysreg HDFGWTR_EL2	3	4	3	1	5
+Res0	63
+Field	62	nPMSNEVFR_EL1
+Field	61	nBRBDATA
+Field	60	nBRBCTL
+Res0	59:58
+Field	57	PMUSERENR_EL0
+Field	56	TRBTRG_EL1
+Field	55	TRBSR_EL1
+Field	54	TRBPTR_EL1
+Field	53	TRBMAR_EL1
+Field	52	TRBLIMITR_EL1
+Res0	51
+Field	50	TRBBASER_EL1
+Field	49	TRFCR_EL1
+Field	48	TRCVICTLR
+Res0	47
+Field	46	TRCSSCSRn
+Field	45	TRCSEQSTR
+Field	44	TRCPRGCTLR
+Res0	43
+Field	42	TRCOSLAR
+Field	41	TRCIMSPECn
+Res0	40:38
+Field	37	TRCCNTVRn
+Field	36	TRCCLAIM
+Field	35	TRCAUXCTLR
+Res0	34
+Field	33	TRC
+Field	32	PMSLATFR_EL1
+Field	31	PMSIRR_EL1
+Res0	30
+Field	29	PMSICR_EL1
+Field	28	PMSFCR_EL1
+Field	27	PMSEVFR_EL1
+Field	26	PMSCR_EL1
+Field	25	PMBSR_EL1
+Field	24	PMBPTR_EL1
+Field	23	PMBLIMITR_EL1
+Res0	22
+Field	21	PMCR_EL0
+Field	20	PMSWINC_EL0
+Field	19	PMSELR_EL0
+Field	18	PMOVS
+Field	17	PMINTEN
+Field	16	PMCNTEN
+Field	15	PMCCNTR_EL0
+Field	14	PMCCFILTR_EL0
+Field	13	PMEVTYPERn_EL0
+Field	12	PMEVCNTRn_EL0
+Field	11	OSDLR_EL1
+Field	10	OSECCR_EL1
+Res0	9
+Field	8	OSLAR_EL1
+Field	7	DBGPRCR_EL1
+Res0	6
+Field	5	DBGCLAIM
+Field	4	MDSCR_EL1
+Field	3	DBGWVRn_EL1
+Field	2	DBGWCRn_EL1
+Field	1	DBGBVRn_EL1
+Field	0	DBGBCRn_EL1
+EndSysreg
+
 Sysreg	ZCR_EL2	3	4	1	2	0
 Fields	ZCR_ELx
 EndSysreg
-- 
2.34.1

[PATCH 10/27] arm64: Add feature detection for fine grained traps

[Marc Zyngier]

From: Mark Brown <broonie@kernel.org>

In order to allow us to have shared code for managing fine grained traps
for KVM guests add it as a detected feature rather than relying on it
being a dependency of other features.

Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Mark Brown <broonie@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20230301-kvm-arm64-fgt-v4-1-1bf8d235ac1f@kernel.org
---
 arch/arm64/kernel/cpufeature.c | 11 +++++++++++
 arch/arm64/tools/cpucaps       |  1 +
 2 files changed, 12 insertions(+)

diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index f9d456fe132d..0768f98c49cc 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -2627,6 +2627,17 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.matches = has_cpuid_feature,
 		ARM64_CPUID_FIELDS(ID_AA64ISAR1_EL1, LRCPC, IMP)
 	},
+	{
+		.desc = "Fine Grained Traps",
+		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+		.capability = ARM64_HAS_FGT,
+		.sys_reg = SYS_ID_AA64MMFR0_EL1,
+		.sign = FTR_UNSIGNED,
+		.field_pos = ID_AA64MMFR0_EL1_FGT_SHIFT,
+		.field_width = 4,
+		.min_field_value = 1,
+		.matches = has_cpuid_feature,
+	},
 #ifdef CONFIG_ARM64_SME
 	{
 		.desc = "Scalable Matrix Extension",
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index c80ed4f3cbce..c3f06fdef609 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -26,6 +26,7 @@ HAS_ECV
 HAS_ECV_CNTPOFF
 HAS_EPAN
 HAS_EVT
+HAS_FGT
 HAS_GENERIC_AUTH
 HAS_GENERIC_AUTH_ARCH_QARMA3
 HAS_GENERIC_AUTH_ARCH_QARMA5
-- 
2.34.1

[PATCH 11/27] KVM: arm64: Correctly handle ACCDATA_EL1 traps

[Marc Zyngier]

As we blindly reset some HFGxTR_EL2 bits to 0, we also randomly trap
unsuspecting sysregs that have their trap bits with a negative
polarity.

ACCDATA_EL1 is one such register that can be accessed by the guest,
causing a splat on the host as we don't have a proper handler for
it.

Adding such handler addresses the issue, though there are a number
of other registers missing as the current architecture documentation
doesn't describe them yet.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/sysreg.h | 2 ++
 arch/arm64/kvm/sys_regs.c       | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
index 6d3d16fac227..d528a79417a0 100644
--- a/arch/arm64/include/asm/sysreg.h
+++ b/arch/arm64/include/asm/sysreg.h
@@ -389,6 +389,8 @@
 #define SYS_ICC_IGRPEN0_EL1		sys_reg(3, 0, 12, 12, 6)
 #define SYS_ICC_IGRPEN1_EL1		sys_reg(3, 0, 12, 12, 7)
 
+#define SYS_ACCDATA_EL1			sys_reg(3, 0, 13, 0, 5)
+
 #define SYS_CNTKCTL_EL1			sys_reg(3, 0, 14, 1, 0)
 
 #define SYS_AIDR_EL1			sys_reg(3, 1, 0, 0, 7)
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index bd3431823ec5..3a6f678ca67d 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -2151,6 +2151,8 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ SYS_DESC(SYS_CONTEXTIDR_EL1), access_vm_reg, reset_val, CONTEXTIDR_EL1, 0 },
 	{ SYS_DESC(SYS_TPIDR_EL1), NULL, reset_unknown, TPIDR_EL1 },
 
+	{ SYS_DESC(SYS_ACCDATA_EL1), undef_access },
+
 	{ SYS_DESC(SYS_SCXTNUM_EL1), undef_access },
 
 	{ SYS_DESC(SYS_CNTKCTL_EL1), NULL, reset_val, CNTKCTL_EL1, 0},
-- 
2.34.1

[PATCH 12/27] KVM: arm64: Add missing HCR_EL2 trap bits

[Marc Zyngier]

We're still missing a handfull of HCR_EL2 trap bits. Add them.

Reviewed-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/kvm_arm.h | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index 58e5eb27da68..028049b147df 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -18,10 +18,19 @@
 #define HCR_DCT		(UL(1) << 57)
 #define HCR_ATA_SHIFT	56
 #define HCR_ATA		(UL(1) << HCR_ATA_SHIFT)
+#define HCR_TTLBOS	(UL(1) << 55)
+#define HCR_TTLBIS	(UL(1) << 54)
+#define HCR_ENSCXT	(UL(1) << 53)
+#define HCR_TOCU	(UL(1) << 52)
 #define HCR_AMVOFFEN	(UL(1) << 51)
+#define HCR_TICAB	(UL(1) << 50)
 #define HCR_TID4	(UL(1) << 49)
 #define HCR_FIEN	(UL(1) << 47)
 #define HCR_FWB		(UL(1) << 46)
+#define HCR_NV2		(UL(1) << 45)
+#define HCR_AT		(UL(1) << 44)
+#define HCR_NV1		(UL(1) << 43)
+#define HCR_NV		(UL(1) << 42)
 #define HCR_API		(UL(1) << 41)
 #define HCR_APK		(UL(1) << 40)
 #define HCR_TEA		(UL(1) << 37)
-- 
2.34.1

[PATCH 13/27] KVM: arm64: nv: Add FGT registers

[Marc Zyngier]

Add the 5 registers covering FEAT_FGT. The AMU-related registers
are currently left out as we don't have a plan for them. Yet.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/kvm_host.h | 5 +++++
 arch/arm64/kvm/sys_regs.c         | 5 +++++
 2 files changed, 10 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 8b6096753740..1200f29282ba 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -400,6 +400,11 @@ enum vcpu_sysreg {
 	TPIDR_EL2,	/* EL2 Software Thread ID Register */
 	CNTHCTL_EL2,	/* Counter-timer Hypervisor Control register */
 	SP_EL2,		/* EL2 Stack Pointer */
+	HFGRTR_EL2,
+	HFGWTR_EL2,
+	HFGITR_EL2,
+	HDFGRTR_EL2,
+	HDFGWTR_EL2,
 	CNTHP_CTL_EL2,
 	CNTHP_CVAL_EL2,
 	CNTHV_CTL_EL2,
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index 3a6f678ca67d..f88cd1390998 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -2367,6 +2367,9 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	EL2_REG(MDCR_EL2, access_rw, reset_val, 0),
 	EL2_REG(CPTR_EL2, access_rw, reset_val, CPTR_NVHE_EL2_RES1),
 	EL2_REG(HSTR_EL2, access_rw, reset_val, 0),
+	EL2_REG(HFGRTR_EL2, access_rw, reset_val, 0),
+	EL2_REG(HFGWTR_EL2, access_rw, reset_val, 0),
+	EL2_REG(HFGITR_EL2, access_rw, reset_val, 0),
 	EL2_REG(HACR_EL2, access_rw, reset_val, 0),
 
 	EL2_REG(TTBR0_EL2, access_rw, reset_val, 0),
@@ -2376,6 +2379,8 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	EL2_REG(VTCR_EL2, access_rw, reset_val, 0),
 
 	{ SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 },
+	EL2_REG(HDFGRTR_EL2, access_rw, reset_val, 0),
+	EL2_REG(HDFGWTR_EL2, access_rw, reset_val, 0),
 	EL2_REG(SPSR_EL2, access_rw, reset_val, 0),
 	EL2_REG(ELR_EL2, access_rw, reset_val, 0),
 	{ SYS_DESC(SYS_SP_EL1), access_sp_el1},
-- 
2.34.1

[PATCH 14/27] KVM: arm64: Restructure FGT register switching

[Marc Zyngier]

As we're about to majorly extend the handling of FGT registers,
restructure the code to actually save/restore the registers
as required. This is made easy thanks to the previous addition
of the EL2 registers, allowing us to use the host context for
this purpose.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/kvm_arm.h        | 21 ++++++++++
 arch/arm64/kvm/hyp/include/hyp/switch.h | 55 +++++++++++++------------
 2 files changed, 49 insertions(+), 27 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index 028049b147df..85908aa18908 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -333,6 +333,27 @@
 				 BIT(18) |		\
 				 GENMASK(16, 15))
 
+/*
+ * FGT register definitions
+ *
+ * RES0 and polarity masks as of DDI0487J.a, to be updated as needed.
+ * We're not using the generated masks as they are usually ahead of
+ * the published ARM ARM, which we use as a reference.
+ *
+ * Once we get to a point where the two describe the same thing, we'll
+ * merge the definitions. One day.
+ */
+#define __HFGRTR_EL2_RES0	(GENMASK(63, 56) | GENMASK(53, 51))
+#define __HFGRTR_EL2_MASK	GENMASK(49, 0)
+#define __HFGRTR_EL2_nMASK	(GENMASK(55, 54) | BIT(50))
+
+#define __HFGWTR_EL2_RES0	(GENMASK(63, 56) | GENMASK(53, 51) |	\
+				 BIT(46) | BIT(42) | BIT(40) | BIT(28) | \
+				 GENMASK(26, 25) | BIT(21) | BIT(18) |	\
+				 GENMASK(15, 14) | GENMASK(10, 9) | BIT(2))
+#define __HFGWTR_EL2_MASK	GENMASK(49, 0)
+#define __HFGWTR_EL2_nMASK	(GENMASK(55, 54) | BIT(50))
+
 /* Hyp Prefetch Fault Address Register (HPFAR/HDFAR) */
 #define HPFAR_MASK	(~UL(0xf))
 /*
diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
index 4bddb8541bec..9781e79a5127 100644
--- a/arch/arm64/kvm/hyp/include/hyp/switch.h
+++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
@@ -70,20 +70,19 @@ static inline void __activate_traps_fpsimd32(struct kvm_vcpu *vcpu)
 	}
 }
 
-static inline bool __hfgxtr_traps_required(void)
-{
-	if (cpus_have_final_cap(ARM64_SME))
-		return true;
-
-	if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
-		return true;
 
-	return false;
-}
 
-static inline void __activate_traps_hfgxtr(void)
+static inline void __activate_traps_hfgxtr(struct kvm_vcpu *vcpu)
 {
+	struct kvm_cpu_context *hctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
 	u64 r_clr = 0, w_clr = 0, r_set = 0, w_set = 0, tmp;
+	u64 r_val, w_val;
+
+	if (!cpus_have_final_cap(ARM64_HAS_FGT))
+		return;
+
+	ctxt_sys_reg(hctxt, HFGRTR_EL2) = read_sysreg_s(SYS_HFGRTR_EL2);
+	ctxt_sys_reg(hctxt, HFGWTR_EL2) = read_sysreg_s(SYS_HFGWTR_EL2);
 
 	if (cpus_have_final_cap(ARM64_SME)) {
 		tmp = HFGxTR_EL2_nSMPRI_EL1_MASK | HFGxTR_EL2_nTPIDR2_EL0_MASK;
@@ -98,26 +97,30 @@ static inline void __activate_traps_hfgxtr(void)
 	if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
 		w_set |= HFGxTR_EL2_TCR_EL1_MASK;
 
-	sysreg_clear_set_s(SYS_HFGRTR_EL2, r_clr, r_set);
-	sysreg_clear_set_s(SYS_HFGWTR_EL2, w_clr, w_set);
+
+	r_val = __HFGRTR_EL2_nMASK & ~HFGxTR_EL2_nACCDATA_EL1;
+	r_val |= r_set;
+	r_val &= ~r_clr;
+
+	w_val = __HFGWTR_EL2_nMASK & ~HFGxTR_EL2_nACCDATA_EL1;
+	w_val |= w_set;
+	w_val &= ~w_clr;
+
+	write_sysreg_s(r_val, SYS_HFGRTR_EL2);
+	write_sysreg_s(w_val, SYS_HFGWTR_EL2);
 }
 
-static inline void __deactivate_traps_hfgxtr(void)
+static inline void __deactivate_traps_hfgxtr(struct kvm_vcpu *vcpu)
 {
-	u64 r_clr = 0, w_clr = 0, r_set = 0, w_set = 0, tmp;
+	struct kvm_cpu_context *hctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
 
-	if (cpus_have_final_cap(ARM64_SME)) {
-		tmp = HFGxTR_EL2_nSMPRI_EL1_MASK | HFGxTR_EL2_nTPIDR2_EL0_MASK;
+	if (!cpus_have_final_cap(ARM64_HAS_FGT))
+		return;
 
-		r_set |= tmp;
-		w_set |= tmp;
-	}
+	write_sysreg_s(ctxt_sys_reg(hctxt, HFGRTR_EL2), SYS_HFGRTR_EL2);
+	write_sysreg_s(ctxt_sys_reg(hctxt, HFGWTR_EL2), SYS_HFGWTR_EL2);
 
-	if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
-		w_clr |= HFGxTR_EL2_TCR_EL1_MASK;
 
-	sysreg_clear_set_s(SYS_HFGRTR_EL2, r_clr, r_set);
-	sysreg_clear_set_s(SYS_HFGWTR_EL2, w_clr, w_set);
 }
 
 static inline void __activate_traps_common(struct kvm_vcpu *vcpu)
@@ -145,8 +148,7 @@ static inline void __activate_traps_common(struct kvm_vcpu *vcpu)
 	vcpu->arch.mdcr_el2_host = read_sysreg(mdcr_el2);
 	write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
 
-	if (__hfgxtr_traps_required())
-		__activate_traps_hfgxtr();
+	__activate_traps_hfgxtr(vcpu);
 }
 
 static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu)
@@ -162,8 +164,7 @@ static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu)
 		vcpu_clear_flag(vcpu, PMUSERENR_ON_CPU);
 	}
 
-	if (__hfgxtr_traps_required())
-		__deactivate_traps_hfgxtr();
+	__deactivate_traps_hfgxtr(vcpu);
 }
 
 static inline void ___activate_traps(struct kvm_vcpu *vcpu)
-- 
2.34.1

[PATCH 15/27] KVM: arm64: nv: Add trap forwarding infrastructure

[Marc Zyngier]

A significant part of what a NV hypervisor needs to do is to decide
whether a trap from a L2+ guest has to be forwarded to a L1 guest
or handled locally. This is done by checking for the trap bits that
the guest hypervisor has set and acting accordingly, as described by
the architecture.

A previous approach was to sprinkle a bunch of checks in all the
system register accessors, but this is pretty error prone and doesn't
help getting an overview of what is happening.

Instead, implement a set of global tables that describe a trap bit,
combinations of trap bits, behaviours on trap, and what bits must
be evaluated on a system register trap.

Although this is painful to describe, this allows to specify each
and every control bit in a static manner. To make it efficient,
the table is inserted in an xarray that is global to the system,
and checked each time we trap a system register while running
a L2 guest.

Add the basic infrastructure for now, while additional patches will
implement configuration registers.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/kvm_host.h   |   1 +
 arch/arm64/include/asm/kvm_nested.h |   2 +
 arch/arm64/kvm/emulate-nested.c     | 210 ++++++++++++++++++++++++++++
 arch/arm64/kvm/sys_regs.c           |   6 +
 arch/arm64/kvm/trace_arm.h          |  19 +++
 5 files changed, 238 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 1200f29282ba..e498cca78a8d 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -986,6 +986,7 @@ int kvm_handle_cp10_id(struct kvm_vcpu *vcpu);
 void kvm_reset_sys_regs(struct kvm_vcpu *vcpu);
 
 int __init kvm_sys_reg_table_init(void);
+void __init populate_nv_trap_config(void);
 
 bool lock_all_vcpus(struct kvm *kvm);
 void unlock_all_vcpus(struct kvm *kvm);
diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h
index 8fb67f032fd1..fa23cc9c2adc 100644
--- a/arch/arm64/include/asm/kvm_nested.h
+++ b/arch/arm64/include/asm/kvm_nested.h
@@ -11,6 +11,8 @@ static inline bool vcpu_has_nv(const struct kvm_vcpu *vcpu)
 		test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features));
 }
 
+extern bool __check_nv_sr_forward(struct kvm_vcpu *vcpu);
+
 struct sys_reg_params;
 struct sys_reg_desc;
 
diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
index b96662029fb1..5bab2e85d70c 100644
--- a/arch/arm64/kvm/emulate-nested.c
+++ b/arch/arm64/kvm/emulate-nested.c
@@ -14,6 +14,216 @@
 
 #include "trace.h"
 
+enum trap_behaviour {
+	BEHAVE_HANDLE_LOCALLY	= 0,
+	BEHAVE_FORWARD_READ	= BIT(0),
+	BEHAVE_FORWARD_WRITE	= BIT(1),
+	BEHAVE_FORWARD_ANY	= BEHAVE_FORWARD_READ | BEHAVE_FORWARD_WRITE,
+};
+
+struct trap_bits {
+	const enum vcpu_sysreg		index;
+	const enum trap_behaviour	behaviour;
+	const u64			value;
+	const u64			mask;
+};
+
+enum coarse_grain_trap_id {
+	/* Indicates no coarse trap control */
+	__RESERVED__,
+
+	/*
+	 * The first batch of IDs denote coarse trapping that are used
+	 * on their own instead of being part of a combination of
+	 * trap controls.
+	 */
+
+	/*
+	 * Anything after this point is a combination of trap controls,
+	 * which all must be evaluated to decide what to do.
+	 */
+	__MULTIPLE_CONTROL_BITS__,
+
+	/*
+	 * Anything after this point requires a callback evaluating a
+	 * complex trap condition. Hopefully we'll never need this...
+	 */
+	__COMPLEX_CONDITIONS__,
+};
+
+static const struct trap_bits coarse_trap_bits[] = {
+};
+
+#define MCB(id, ...)					\
+	[id - __MULTIPLE_CONTROL_BITS__]	=	\
+		(const enum coarse_grain_trap_id []){	\
+			__VA_ARGS__ , __RESERVED__	\
+		}
+
+static const enum coarse_grain_trap_id *coarse_control_combo[] = {
+};
+
+typedef enum trap_behaviour (*complex_condition_check)(struct kvm_vcpu *);
+
+#define CCC(id, fn)	[id - __COMPLEX_CONDITIONS__] = fn
+
+static const complex_condition_check ccc[] = {
+};
+
+/*
+ * Bit assignment for the trap controls. We use a 64bit word with the
+ * following layout for each trapped sysreg:
+ *
+ * [9:0]	enum coarse_grain_trap_id (10 bits)
+ * [13:10]	enum fgt_group_id (4 bits)
+ * [19:14]	bit number in the FGT register (6 bits)
+ * [20]		trap polarity (1 bit)
+ * [62:21]	Unused (42 bits)
+ * [63]		RES0 - Must be zero, as lost on insertion in the xarray
+ */
+union trap_config {
+	u64	val;
+	struct {
+		unsigned long	cgt:10;	/* Coarse trap id */
+		unsigned long 	fgt:4;	/* Fing Grained Trap id */
+		unsigned long	bit:6;	/* Bit number */
+		unsigned long	pol:1;	/* Polarity */
+		unsigned long	unk:42;	/* Unknown */
+		unsigned long	mbz:1;	/* Must Be Zero */
+	};
+};
+
+struct encoding_to_trap_config {
+	const u32			encoding;
+	const u32			end;
+	const union trap_config		tc;
+};
+
+#define SR_RANGE_TRAP(sr_start, sr_end, trap_id)			\
+	{								\
+		.encoding	= sr_start,				\
+		.end		= sr_end,				\
+		.tc		= {					\
+			.cgt		= trap_id,			\
+		},							\
+	}
+
+#define SR_TRAP(sr, trap_id)		SR_RANGE_TRAP(sr, sr, trap_id)
+
+/*
+ * Map encoding to trap bits for exception reported with EC=0x18.
+ * These must only be evaluated when running a nested hypervisor, but
+ * that the current context is not a hypervisor context. When the
+ * trapped access matches one of the trap controls, the exception is
+ * re-injected in the nested hypervisor.
+ */
+static const struct encoding_to_trap_config encoding_to_cgt[] __initdata = {
+};
+
+static DEFINE_XARRAY(sr_forward_xa);
+
+static union trap_config get_trap_config(u32 sysreg)
+{
+	return (union trap_config) {
+		.val = xa_to_value(xa_load(&sr_forward_xa, sysreg)),
+	};
+}
+
+void __init populate_nv_trap_config(void)
+{
+	for (int i = 0; i < ARRAY_SIZE(encoding_to_cgt); i++) {
+		const struct encoding_to_trap_config *cgt = &encoding_to_cgt[i];
+		void *prev;
+
+		prev = xa_store_range(&sr_forward_xa, cgt->encoding, cgt->end,
+				      xa_mk_value(cgt->tc.val), GFP_KERNEL);
+		WARN_ON(prev);
+	}
+
+	kvm_info("nv: %ld coarse grained trap handlers\n",
+		 ARRAY_SIZE(encoding_to_cgt));
+
+}
+
+static enum trap_behaviour get_behaviour(struct kvm_vcpu *vcpu,
+					 const struct trap_bits *tb)
+{
+	enum trap_behaviour b = BEHAVE_HANDLE_LOCALLY;
+	u64 val;
+
+	val = __vcpu_sys_reg(vcpu, tb->index);
+	if ((val & tb->mask) == tb->value)
+		b |= tb->behaviour;
+
+	return b;
+}
+
+static enum trap_behaviour __do_compute_behaviour(struct kvm_vcpu *vcpu,
+						  const enum coarse_grain_trap_id id,
+						  enum trap_behaviour b)
+{
+	switch (id) {
+		const enum coarse_grain_trap_id *cgids;
+
+	case __RESERVED__ ... __MULTIPLE_CONTROL_BITS__ - 1:
+		if (likely(id != __RESERVED__))
+			b |= get_behaviour(vcpu, &coarse_trap_bits[id]);
+		break;
+	case __MULTIPLE_CONTROL_BITS__ ... __COMPLEX_CONDITIONS__ - 1:
+		/* Yes, this is recursive. Don't do anything stupid. */
+		cgids = coarse_control_combo[id - __MULTIPLE_CONTROL_BITS__];
+		for (int i = 0; cgids[i] != __RESERVED__; i++)
+			b |= __do_compute_behaviour(vcpu, cgids[i], b);
+		break;
+	default:
+		if (ARRAY_SIZE(ccc))
+			b |= ccc[id -  __COMPLEX_CONDITIONS__](vcpu);
+		break;
+	}
+
+	return b;
+}
+
+static enum trap_behaviour compute_behaviour(struct kvm_vcpu *vcpu,
+					     const union trap_config tc)
+{
+	enum trap_behaviour b = BEHAVE_HANDLE_LOCALLY;
+
+	return __do_compute_behaviour(vcpu, tc.cgt, b);
+}
+
+bool __check_nv_sr_forward(struct kvm_vcpu *vcpu)
+{
+	union trap_config tc;
+	enum trap_behaviour b;
+	bool is_read;
+	u32 sysreg;
+	u64 esr;
+
+	if (!vcpu_has_nv(vcpu) || is_hyp_ctxt(vcpu))
+		return false;
+
+	esr = kvm_vcpu_get_esr(vcpu);
+	sysreg = esr_sys64_to_sysreg(esr);
+	is_read = (esr & ESR_ELx_SYS64_ISS_DIR_MASK) == ESR_ELx_SYS64_ISS_DIR_READ;
+
+	tc = get_trap_config(sysreg);
+
+	b = compute_behaviour(vcpu, tc);
+
+	if (((b & BEHAVE_FORWARD_READ) && is_read) ||
+	    ((b & BEHAVE_FORWARD_WRITE) && !is_read))
+		goto inject;
+
+	return false;
+
+inject:
+	trace_kvm_forward_sysreg_trap(vcpu, sysreg, is_read);
+
+	kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu));
+	return true;
+}
+
 static u64 kvm_check_illegal_exception_return(struct kvm_vcpu *vcpu, u64 spsr)
 {
 	u64 mode = spsr & PSR_MODE_MASK;
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index f88cd1390998..cc10ecaff98a 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -3177,6 +3177,9 @@ int kvm_handle_sys_reg(struct kvm_vcpu *vcpu)
 
 	trace_kvm_handle_sys_reg(esr);
 
+	if (__check_nv_sr_forward(vcpu))
+		return 1;
+
 	params = esr_sys64_to_params(esr);
 	params.regval = vcpu_get_reg(vcpu, Rt);
 
@@ -3594,5 +3597,8 @@ int __init kvm_sys_reg_table_init(void)
 	if (!first_idreg)
 		return -EINVAL;
 
+	if (kvm_get_mode() == KVM_MODE_NV)
+		populate_nv_trap_config();
+		
 	return 0;
 }
diff --git a/arch/arm64/kvm/trace_arm.h b/arch/arm64/kvm/trace_arm.h
index 6ce5c025218d..1f0f3f653606 100644
--- a/arch/arm64/kvm/trace_arm.h
+++ b/arch/arm64/kvm/trace_arm.h
@@ -364,6 +364,25 @@ TRACE_EVENT(kvm_inject_nested_exception,
 		  __entry->hcr_el2)
 );
 
+TRACE_EVENT(kvm_forward_sysreg_trap,
+	    TP_PROTO(struct kvm_vcpu *vcpu, u32 sysreg, bool is_read),
+	    TP_ARGS(vcpu, sysreg, is_read),
+
+	    TP_STRUCT__entry(
+		__field(struct kvm_vcpu *, vcpu)
+		__field(u32,		   sysreg)
+		__field(bool,		   is_read)
+	    ),
+
+	    TP_fast_assign(
+		__entry->vcpu = vcpu;
+		__entry->sysreg = sysreg;
+		__entry->is_read = is_read;
+	    ),
+
+	    TP_printk("%c %x", __entry->is_read ? 'R' : 'W', __entry->sysreg)
+);
+
 #endif /* _TRACE_ARM_ARM64_KVM_H */
 
 #undef TRACE_INCLUDE_PATH
-- 
2.34.1

[PATCH 16/27] KVM: arm64: nv: Add trap forwarding for HCR_EL2

[Marc Zyngier]

Describe the HCR_EL2 register, and associate it with all the sysregs
it allows to trap.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/kvm/emulate-nested.c | 475 ++++++++++++++++++++++++++++++++
 1 file changed, 475 insertions(+)

diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
index 5bab2e85d70c..51901e85e43d 100644
--- a/arch/arm64/kvm/emulate-nested.c
+++ b/arch/arm64/kvm/emulate-nested.c
@@ -37,12 +37,47 @@ enum coarse_grain_trap_id {
 	 * on their own instead of being part of a combination of
 	 * trap controls.
 	 */
+	CGT_HCR_TID1,
+	CGT_HCR_TID2,
+	CGT_HCR_TID3,
+	CGT_HCR_IMO,
+	CGT_HCR_FMO,
+	CGT_HCR_TIDCP,
+	CGT_HCR_TACR,
+	CGT_HCR_TSW,
+	CGT_HCR_TPC,
+	CGT_HCR_TPU,
+	CGT_HCR_TTLB,
+	CGT_HCR_TVM,
+	CGT_HCR_TDZ,
+	CGT_HCR_TRVM,
+	CGT_HCR_TLOR,
+	CGT_HCR_TERR,
+	CGT_HCR_APK,
+	CGT_HCR_NV,
+	CGT_HCR_NV1,
+	CGT_HCR_AT,
+	CGT_HCR_FIEN,
+	CGT_HCR_TID4,
+	CGT_HCR_TICAB,
+	CGT_HCR_TOCU,
+	CGT_HCR_ENSCXT,
+	CGT_HCR_TTLBIS,
+	CGT_HCR_TTLBOS,
 
 	/*
 	 * Anything after this point is a combination of trap controls,
 	 * which all must be evaluated to decide what to do.
 	 */
 	__MULTIPLE_CONTROL_BITS__,
+	CGT_HCR_IMO_FMO = __MULTIPLE_CONTROL_BITS__,
+	CGT_HCR_TID2_TID4,
+	CGT_HCR_TTLB_TTLBIS,
+	CGT_HCR_TTLB_TTLBOS,
+	CGT_HCR_TVM_TRVM,
+	CGT_HCR_TPU_TICAB,
+	CGT_HCR_TPU_TOCU,
+	CGT_HCR_NV1_ENSCXT,
 
 	/*
 	 * Anything after this point requires a callback evaluating a
@@ -52,6 +87,168 @@ enum coarse_grain_trap_id {
 };
 
 static const struct trap_bits coarse_trap_bits[] = {
+	[CGT_HCR_TID1] = {
+		.index		= HCR_EL2,
+		.value 		= HCR_TID1,
+		.mask		= HCR_TID1,
+		.behaviour	= BEHAVE_FORWARD_READ,
+	},
+	[CGT_HCR_TID2] = {
+		.index		= HCR_EL2,
+		.value 		= HCR_TID2,
+		.mask		= HCR_TID2,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_TID3] = {
+		.index		= HCR_EL2,
+		.value 		= HCR_TID3,
+		.mask		= HCR_TID3,
+		.behaviour	= BEHAVE_FORWARD_READ,
+	},
+	[CGT_HCR_IMO] = {
+		.index		= HCR_EL2,
+		.value 		= HCR_IMO,
+		.mask		= HCR_IMO,
+		.behaviour	= BEHAVE_FORWARD_WRITE,
+	},
+	[CGT_HCR_FMO] = {
+		.index		= HCR_EL2,
+		.value 		= HCR_FMO,
+		.mask		= HCR_FMO,
+		.behaviour	= BEHAVE_FORWARD_WRITE,
+	},
+	[CGT_HCR_TIDCP] = {
+		.index		= HCR_EL2,
+		.value		= HCR_TIDCP,
+		.mask		= HCR_TIDCP,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_TACR] = {
+		.index		= HCR_EL2,
+		.value		= HCR_TACR,
+		.mask		= HCR_TACR,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_TSW] = {
+		.index		= HCR_EL2,
+		.value		= HCR_TSW,
+		.mask		= HCR_TSW,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_TPC] = {
+		.index		= HCR_EL2,
+		.value		= HCR_TPC,
+		.mask		= HCR_TPC,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_TPU] = {
+		.index		= HCR_EL2,
+		.value		= HCR_TPU,
+		.mask		= HCR_TPU,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_TTLB] = {
+		.index		= HCR_EL2,
+		.value		= HCR_TTLB,
+		.mask		= HCR_TTLB,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_TVM] = {
+		.index		= HCR_EL2,
+		.value		= HCR_TVM,
+		.mask		= HCR_TVM,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_TDZ] = {
+		.index		= HCR_EL2,
+		.value		= HCR_TDZ,
+		.mask		= HCR_TDZ,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_TRVM] = {
+		.index		= HCR_EL2,
+		.value		= HCR_TRVM,
+		.mask		= HCR_TRVM,
+		.behaviour	= BEHAVE_FORWARD_READ,
+	},
+	[CGT_HCR_TLOR] = {
+		.index		= HCR_EL2,
+		.value		= HCR_TLOR,
+		.mask		= HCR_TLOR,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_TERR] = {
+		.index		= HCR_EL2,
+		.value		= HCR_TERR,
+		.mask		= HCR_TERR,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_APK] = {
+		.index		= HCR_EL2,
+		.value		= 0,
+		.mask		= HCR_APK,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_NV] = {
+		.index		= HCR_EL2,
+		.value		= HCR_NV,
+		.mask		= HCR_NV | HCR_NV2,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_NV1] = {
+		.index		= HCR_EL2,
+		.value		= HCR_NV | HCR_NV1,
+		.mask		= HCR_NV | HCR_NV1 | HCR_NV2,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_AT] = {
+		.index		= HCR_EL2,
+		.value		= HCR_AT,
+		.mask		= HCR_AT,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_FIEN] = {
+		.index		= HCR_EL2,
+		.value		= HCR_FIEN,
+		.mask		= HCR_FIEN,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_TID4] = {
+		.index		= HCR_EL2,
+		.value 		= HCR_TID4,
+		.mask		= HCR_TID4,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_TICAB] = {
+		.index		= HCR_EL2,
+		.value 		= HCR_TICAB,
+		.mask		= HCR_TICAB,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_TOCU] = {
+		.index		= HCR_EL2,
+		.value 		= HCR_TOCU,
+		.mask		= HCR_TOCU,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_ENSCXT] = {
+		.index		= HCR_EL2,
+		.value 		= 0,
+		.mask		= HCR_ENSCXT,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_TTLBIS] = {
+		.index		= HCR_EL2,
+		.value		= HCR_TTLBIS,
+		.mask		= HCR_TTLBIS,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_TTLBOS] = {
+		.index		= HCR_EL2,
+		.value		= HCR_TTLBOS,
+		.mask		= HCR_TTLBOS,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
 };
 
 #define MCB(id, ...)					\
@@ -61,6 +258,14 @@ static const struct trap_bits coarse_trap_bits[] = {
 		}
 
 static const enum coarse_grain_trap_id *coarse_control_combo[] = {
+	MCB(CGT_HCR_IMO_FMO,		CGT_HCR_IMO, CGT_HCR_FMO),
+	MCB(CGT_HCR_TID2_TID4,		CGT_HCR_TID2, CGT_HCR_TID4),
+	MCB(CGT_HCR_TTLB_TTLBIS,	CGT_HCR_TTLB, CGT_HCR_TTLBIS),
+	MCB(CGT_HCR_TTLB_TTLBOS,	CGT_HCR_TTLB, CGT_HCR_TTLBOS),
+	MCB(CGT_HCR_TVM_TRVM,		CGT_HCR_TVM, CGT_HCR_TRVM),
+	MCB(CGT_HCR_TPU_TICAB,		CGT_HCR_TPU, CGT_HCR_TICAB),
+	MCB(CGT_HCR_TPU_TOCU,		CGT_HCR_TPU, CGT_HCR_TOCU),
+	MCB(CGT_HCR_NV1_ENSCXT,		CGT_HCR_NV1, CGT_HCR_ENSCXT),
 };
 
 typedef enum trap_behaviour (*complex_condition_check)(struct kvm_vcpu *);
@@ -118,6 +323,276 @@ struct encoding_to_trap_config {
  * re-injected in the nested hypervisor.
  */
 static const struct encoding_to_trap_config encoding_to_cgt[] __initdata = {
+	SR_TRAP(SYS_REVIDR_EL1,		CGT_HCR_TID1),
+	SR_TRAP(SYS_AIDR_EL1,		CGT_HCR_TID1),
+	SR_TRAP(SYS_SMIDR_EL1,		CGT_HCR_TID1),
+	SR_TRAP(SYS_CTR_EL0,		CGT_HCR_TID2),
+	SR_TRAP(SYS_CCSIDR_EL1,		CGT_HCR_TID2_TID4),
+	SR_TRAP(SYS_CCSIDR2_EL1,	CGT_HCR_TID2_TID4),
+	SR_TRAP(SYS_CLIDR_EL1,		CGT_HCR_TID2_TID4),
+	SR_TRAP(SYS_CSSELR_EL1,		CGT_HCR_TID2_TID4),
+	SR_RANGE_TRAP(SYS_ID_PFR0_EL1,
+		      sys_reg(3, 0, 0, 7, 7), CGT_HCR_TID3),
+	SR_TRAP(SYS_ICC_SGI0R_EL1,	CGT_HCR_IMO_FMO),
+	SR_TRAP(SYS_ICC_ASGI1R_EL1,	CGT_HCR_IMO_FMO),
+	SR_TRAP(SYS_ICC_SGI1R_EL1,	CGT_HCR_IMO_FMO),
+	SR_RANGE_TRAP(sys_reg(3, 0, 11, 0, 0),
+		      sys_reg(3, 0, 11, 15, 7), CGT_HCR_TIDCP),
+	SR_RANGE_TRAP(sys_reg(3, 1, 11, 0, 0),
+		      sys_reg(3, 1, 11, 15, 7), CGT_HCR_TIDCP),
+	SR_RANGE_TRAP(sys_reg(3, 2, 11, 0, 0),
+		      sys_reg(3, 2, 11, 15, 7), CGT_HCR_TIDCP),
+	SR_RANGE_TRAP(sys_reg(3, 3, 11, 0, 0),
+		      sys_reg(3, 3, 11, 15, 7), CGT_HCR_TIDCP),
+	SR_RANGE_TRAP(sys_reg(3, 4, 11, 0, 0),
+		      sys_reg(3, 4, 11, 15, 7), CGT_HCR_TIDCP),
+	SR_RANGE_TRAP(sys_reg(3, 5, 11, 0, 0),
+		      sys_reg(3, 5, 11, 15, 7), CGT_HCR_TIDCP),
+	SR_RANGE_TRAP(sys_reg(3, 6, 11, 0, 0),
+		      sys_reg(3, 6, 11, 15, 7), CGT_HCR_TIDCP),
+	SR_RANGE_TRAP(sys_reg(3, 7, 11, 0, 0),
+		      sys_reg(3, 7, 11, 15, 7), CGT_HCR_TIDCP),
+	SR_RANGE_TRAP(sys_reg(3, 0, 15, 0, 0),
+		      sys_reg(3, 0, 15, 15, 7), CGT_HCR_TIDCP),
+	SR_RANGE_TRAP(sys_reg(3, 1, 15, 0, 0),
+		      sys_reg(3, 1, 15, 15, 7), CGT_HCR_TIDCP),
+	SR_RANGE_TRAP(sys_reg(3, 2, 15, 0, 0),
+		      sys_reg(3, 2, 15, 15, 7), CGT_HCR_TIDCP),
+	SR_RANGE_TRAP(sys_reg(3, 3, 15, 0, 0),
+		      sys_reg(3, 3, 15, 15, 7), CGT_HCR_TIDCP),
+	SR_RANGE_TRAP(sys_reg(3, 4, 15, 0, 0),
+		      sys_reg(3, 4, 15, 15, 7), CGT_HCR_TIDCP),
+	SR_RANGE_TRAP(sys_reg(3, 5, 15, 0, 0),
+		      sys_reg(3, 5, 15, 15, 7), CGT_HCR_TIDCP),
+	SR_RANGE_TRAP(sys_reg(3, 6, 15, 0, 0),
+		      sys_reg(3, 6, 15, 15, 7), CGT_HCR_TIDCP),
+	SR_RANGE_TRAP(sys_reg(3, 7, 15, 0, 0),
+		      sys_reg(3, 7, 15, 15, 7), CGT_HCR_TIDCP),
+	SR_TRAP(SYS_ACTLR_EL1,		CGT_HCR_TACR),
+	SR_TRAP(SYS_DC_ISW,		CGT_HCR_TSW),
+	SR_TRAP(SYS_DC_CSW,		CGT_HCR_TSW),
+	SR_TRAP(SYS_DC_CISW,		CGT_HCR_TSW),
+	SR_TRAP(SYS_DC_IGSW,		CGT_HCR_TSW),
+	SR_TRAP(SYS_DC_IGDSW,		CGT_HCR_TSW),
+	SR_TRAP(SYS_DC_CGSW,		CGT_HCR_TSW),
+	SR_TRAP(SYS_DC_CGDSW,		CGT_HCR_TSW),
+	SR_TRAP(SYS_DC_CIGSW,		CGT_HCR_TSW),
+	SR_TRAP(SYS_DC_CIGDSW,		CGT_HCR_TSW),
+	SR_TRAP(SYS_DC_CIVAC,		CGT_HCR_TPC),
+	SR_TRAP(SYS_DC_CVAC,		CGT_HCR_TPC),
+	SR_TRAP(SYS_DC_CVAP,		CGT_HCR_TPC),
+	SR_TRAP(SYS_DC_IVAC,		CGT_HCR_TPC),
+	SR_TRAP(SYS_DC_CIGVAC,		CGT_HCR_TPC),
+	SR_TRAP(SYS_DC_CIGDVAC,		CGT_HCR_TPC),
+	SR_TRAP(SYS_DC_IGVAC,		CGT_HCR_TPC),
+	SR_TRAP(SYS_DC_IGDVAC,		CGT_HCR_TPC),
+	SR_TRAP(SYS_DC_CGVAC,		CGT_HCR_TPC),
+	SR_TRAP(SYS_DC_CGDVAC,		CGT_HCR_TPC),
+	SR_TRAP(SYS_DC_CGVAP,		CGT_HCR_TPC),
+	SR_TRAP(SYS_DC_CGDVAP,		CGT_HCR_TPC),
+	SR_TRAP(SYS_DC_CGVADP,		CGT_HCR_TPC),
+	SR_TRAP(SYS_DC_CGDVADP,		CGT_HCR_TPC),
+	SR_TRAP(SYS_IC_IVAU,		CGT_HCR_TPU_TOCU),
+	SR_TRAP(SYS_IC_IALLU,		CGT_HCR_TPU_TOCU),
+	SR_TRAP(SYS_IC_IALLUIS,		CGT_HCR_TPU_TICAB),
+	SR_TRAP(SYS_DC_CVAU,		CGT_HCR_TPU_TOCU),
+	SR_TRAP(OP_TLBI_RVAE1,		CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_RVAAE1,		CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_RVALE1,		CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_RVAALE1,	CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_VMALLE1,	CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_VAE1,		CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_ASIDE1,		CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_VAAE1,		CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_VALE1,		CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_VAALE1,		CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_RVAE1NXS,	CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_RVAAE1NXS,	CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_RVALE1NXS,	CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_RVAALE1NXS,	CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_VMALLE1NXS,	CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_VAE1NXS,	CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_ASIDE1NXS,	CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_VAAE1NXS,	CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_VALE1NXS,	CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_VAALE1NXS,	CGT_HCR_TTLB),
+	SR_TRAP(OP_TLBI_RVAE1IS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_RVAAE1IS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_RVALE1IS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_RVAALE1IS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_VMALLE1IS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_VAE1IS,		CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_ASIDE1IS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_VAAE1IS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_VALE1IS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_VAALE1IS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_RVAE1ISNXS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_RVAAE1ISNXS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_RVALE1ISNXS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_RVAALE1ISNXS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_VMALLE1ISNXS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_VAE1ISNXS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_ASIDE1ISNXS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_VAAE1ISNXS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_VALE1ISNXS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_VAALE1ISNXS,	CGT_HCR_TTLB_TTLBIS),
+	SR_TRAP(OP_TLBI_VMALLE1OS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_VAE1OS,		CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_ASIDE1OS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_VAAE1OS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_VALE1OS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_VAALE1OS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_RVAE1OS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_RVAAE1OS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_RVALE1OS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_RVAALE1OS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_VMALLE1OSNXS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_VAE1OSNXS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_ASIDE1OSNXS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_VAAE1OSNXS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_VALE1OSNXS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_VAALE1OSNXS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_RVAE1OSNXS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_RVAAE1OSNXS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_RVALE1OSNXS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(OP_TLBI_RVAALE1OSNXS,	CGT_HCR_TTLB_TTLBOS),
+	SR_TRAP(SYS_SCTLR_EL1,		CGT_HCR_TVM_TRVM),
+	SR_TRAP(SYS_TTBR0_EL1,		CGT_HCR_TVM_TRVM),
+	SR_TRAP(SYS_TTBR1_EL1,		CGT_HCR_TVM_TRVM),
+	SR_TRAP(SYS_TCR_EL1,		CGT_HCR_TVM_TRVM),
+	SR_TRAP(SYS_ESR_EL1,		CGT_HCR_TVM_TRVM),
+	SR_TRAP(SYS_FAR_EL1,		CGT_HCR_TVM_TRVM),
+	SR_TRAP(SYS_AFSR0_EL1,		CGT_HCR_TVM_TRVM),
+	SR_TRAP(SYS_AFSR1_EL1,		CGT_HCR_TVM_TRVM),
+	SR_TRAP(SYS_MAIR_EL1,		CGT_HCR_TVM_TRVM),
+	SR_TRAP(SYS_AMAIR_EL1,		CGT_HCR_TVM_TRVM),
+	SR_TRAP(SYS_CONTEXTIDR_EL1,	CGT_HCR_TVM_TRVM),
+	SR_TRAP(SYS_DC_ZVA,		CGT_HCR_TDZ),
+	SR_TRAP(SYS_DC_GVA,		CGT_HCR_TDZ),
+	SR_TRAP(SYS_DC_GZVA,		CGT_HCR_TDZ),
+	SR_TRAP(SYS_LORSA_EL1,		CGT_HCR_TLOR),
+	SR_TRAP(SYS_LOREA_EL1, 		CGT_HCR_TLOR),
+	SR_TRAP(SYS_LORN_EL1, 		CGT_HCR_TLOR),
+	SR_TRAP(SYS_LORC_EL1, 		CGT_HCR_TLOR),
+	SR_TRAP(SYS_LORID_EL1,		CGT_HCR_TLOR),
+	SR_TRAP(SYS_ERRIDR_EL1,		CGT_HCR_TERR),
+	SR_TRAP(SYS_ERRSELR_EL1,	CGT_HCR_TERR),
+	SR_TRAP(SYS_ERXADDR_EL1,	CGT_HCR_TERR),
+	SR_TRAP(SYS_ERXCTLR_EL1,	CGT_HCR_TERR),
+	SR_TRAP(SYS_ERXFR_EL1,		CGT_HCR_TERR),
+	SR_TRAP(SYS_ERXMISC0_EL1,	CGT_HCR_TERR),
+	SR_TRAP(SYS_ERXMISC1_EL1,	CGT_HCR_TERR),
+	SR_TRAP(SYS_ERXMISC2_EL1,	CGT_HCR_TERR),
+	SR_TRAP(SYS_ERXMISC3_EL1,	CGT_HCR_TERR),
+	SR_TRAP(SYS_ERXSTATUS_EL1,	CGT_HCR_TERR),
+	SR_TRAP(SYS_APIAKEYLO_EL1,	CGT_HCR_APK),
+	SR_TRAP(SYS_APIAKEYHI_EL1,	CGT_HCR_APK),
+	SR_TRAP(SYS_APIBKEYLO_EL1,	CGT_HCR_APK),
+	SR_TRAP(SYS_APIBKEYHI_EL1,	CGT_HCR_APK),
+	SR_TRAP(SYS_APDAKEYLO_EL1,	CGT_HCR_APK),
+	SR_TRAP(SYS_APDAKEYHI_EL1,	CGT_HCR_APK),
+	SR_TRAP(SYS_APDBKEYLO_EL1,	CGT_HCR_APK),
+	SR_TRAP(SYS_APDBKEYHI_EL1,	CGT_HCR_APK),
+	SR_TRAP(SYS_APGAKEYLO_EL1,	CGT_HCR_APK),
+	SR_TRAP(SYS_APGAKEYHI_EL1,	CGT_HCR_APK),
+	/* All _EL2 registers */
+	SR_RANGE_TRAP(sys_reg(3, 4, 0, 0, 0),
+		      sys_reg(3, 4, 10, 15, 7), CGT_HCR_NV),
+	SR_RANGE_TRAP(sys_reg(3, 4, 12, 0, 0),
+		      sys_reg(3, 4, 14, 15, 7), CGT_HCR_NV),
+	/* All _EL02, _EL12 registers */
+	SR_RANGE_TRAP(sys_reg(3, 5, 0, 0, 0),
+		      sys_reg(3, 5, 10, 15, 7), CGT_HCR_NV),
+	SR_RANGE_TRAP(sys_reg(3, 5, 12, 0, 0),
+		      sys_reg(3, 5, 14, 15, 7), CGT_HCR_NV),
+	SR_TRAP(SYS_SP_EL1,		CGT_HCR_NV),
+	SR_TRAP(OP_AT_S1E2R,		CGT_HCR_NV),
+	SR_TRAP(OP_AT_S1E2W,		CGT_HCR_NV),
+	SR_TRAP(OP_AT_S12E1R,		CGT_HCR_NV),
+	SR_TRAP(OP_AT_S12E1W,		CGT_HCR_NV),
+	SR_TRAP(OP_AT_S12E0R,		CGT_HCR_NV),
+	SR_TRAP(OP_AT_S12E0W,		CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_IPAS2E1,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RIPAS2E1,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_IPAS2LE1,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RIPAS2LE1,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RVAE2,		CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RVALE2,		CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_ALLE2,		CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VAE2,		CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_ALLE1,		CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VALE2,		CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VMALLS12E1,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_IPAS2E1NXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RIPAS2E1NXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_IPAS2LE1NXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RIPAS2LE1NXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RVAE2NXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RVALE2NXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_ALLE2NXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VAE2NXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_ALLE1NXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VALE2NXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VMALLS12E1NXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_IPAS2E1IS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RIPAS2E1IS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_IPAS2LE1IS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RIPAS2LE1IS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RVAE2IS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RVALE2IS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_ALLE2IS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VAE2IS,		CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_ALLE1IS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VALE2IS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VMALLS12E1IS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_IPAS2E1ISNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RIPAS2E1ISNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_IPAS2LE1ISNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RIPAS2LE1ISNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RVAE2ISNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RVALE2ISNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_ALLE2ISNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VAE2ISNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_ALLE1ISNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VALE2ISNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VMALLS12E1ISNXS,CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_ALLE2OS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VAE2OS,		CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_ALLE1OS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VALE2OS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VMALLS12E1OS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_IPAS2E1OS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RIPAS2E1OS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_IPAS2LE1OS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RIPAS2LE1OS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RVAE2OS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RVALE2OS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_ALLE2OSNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VAE2OSNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_ALLE1OSNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VALE2OSNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_VMALLS12E1OSNXS,CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_IPAS2E1OSNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RIPAS2E1OSNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_IPAS2LE1OSNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RIPAS2LE1OSNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RVAE2OSNXS,	CGT_HCR_NV),
+	SR_TRAP(OP_TLBI_RVALE2OSNXS,	CGT_HCR_NV),
+	SR_TRAP(SYS_VBAR_EL1,		CGT_HCR_NV1),
+	SR_TRAP(SYS_ELR_EL1,		CGT_HCR_NV1),
+	SR_TRAP(SYS_SPSR_EL1,		CGT_HCR_NV1),
+	SR_TRAP(SYS_SCXTNUM_EL1,	CGT_HCR_NV1_ENSCXT),
+	SR_TRAP(SYS_SCXTNUM_EL0,	CGT_HCR_ENSCXT),
+	SR_TRAP(OP_AT_S1E1R, 		CGT_HCR_AT),
+	SR_TRAP(OP_AT_S1E1W, 		CGT_HCR_AT),
+	SR_TRAP(OP_AT_S1E0R, 		CGT_HCR_AT),
+	SR_TRAP(OP_AT_S1E0W, 		CGT_HCR_AT),
+	SR_TRAP(OP_AT_S1E1RP, 		CGT_HCR_AT),
+	SR_TRAP(OP_AT_S1E1WP, 		CGT_HCR_AT),
+	SR_TRAP(SYS_ERXPFGF_EL1,	CGT_HCR_FIEN),
+	SR_TRAP(SYS_ERXPFGCTL_EL1,	CGT_HCR_FIEN),
+	SR_TRAP(SYS_ERXPFGCDN_EL1,	CGT_HCR_FIEN),
+	SR_TRAP(SYS_SCXTNUM_EL0,	CGT_HCR_ENSCXT),
 };
 
 static DEFINE_XARRAY(sr_forward_xa);
-- 
2.34.1

[PATCH 17/27] KVM: arm64: nv: Expose FEAT_EVT to nested guests

[Marc Zyngier]

Now that we properly implement FEAT_EVT (as we correctly forward
traps), expose it to guests.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/kvm/nested.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index 315354d27978..7f80f385d9e8 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -124,8 +124,7 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
 		break;
 
 	case SYS_ID_AA64MMFR2_EL1:
-		val &= ~(NV_FTR(MMFR2, EVT)	|
-			 NV_FTR(MMFR2, BBM)	|
+		val &= ~(NV_FTR(MMFR2, BBM)	|
 			 NV_FTR(MMFR2, TTL)	|
 			 GENMASK_ULL(47, 44)	|
 			 NV_FTR(MMFR2, ST)	|
-- 
2.34.1

[PATCH 18/27] KVM: arm64: nv: Add trap forwarding for MDCR_EL2

[Marc Zyngier]

Describe the MDCR_EL2 register, and associate it with all the sysregs
it allows to trap.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/kvm/emulate-nested.c | 262 ++++++++++++++++++++++++++++++++
 1 file changed, 262 insertions(+)

diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
index 51901e85e43d..25e4842ac334 100644
--- a/arch/arm64/kvm/emulate-nested.c
+++ b/arch/arm64/kvm/emulate-nested.c
@@ -65,6 +65,18 @@ enum coarse_grain_trap_id {
 	CGT_HCR_TTLBIS,
 	CGT_HCR_TTLBOS,
 
+	CGT_MDCR_TPMCR,
+	CGT_MDCR_TPM,
+	CGT_MDCR_TDE,
+	CGT_MDCR_TDA,
+	CGT_MDCR_TDOSA,
+	CGT_MDCR_TDRA,
+	CGT_MDCR_E2PB,
+	CGT_MDCR_TPMS,
+	CGT_MDCR_TTRF,
+	CGT_MDCR_E2TB,
+	CGT_MDCR_TDCC,
+
 	/*
 	 * Anything after this point is a combination of trap controls,
 	 * which all must be evaluated to decide what to do.
@@ -78,6 +90,11 @@ enum coarse_grain_trap_id {
 	CGT_HCR_TPU_TICAB,
 	CGT_HCR_TPU_TOCU,
 	CGT_HCR_NV1_ENSCXT,
+	CGT_MDCR_TPM_TPMCR,
+	CGT_MDCR_TDE_TDA,
+	CGT_MDCR_TDE_TDOSA,
+	CGT_MDCR_TDE_TDRA,
+	CGT_MDCR_TDCC_TDE_TDA,
 
 	/*
 	 * Anything after this point requires a callback evaluating a
@@ -249,6 +266,72 @@ static const struct trap_bits coarse_trap_bits[] = {
 		.mask		= HCR_TTLBOS,
 		.behaviour	= BEHAVE_FORWARD_ANY,
 	},
+	[CGT_MDCR_TPMCR] = {
+		.index		= MDCR_EL2,
+		.value		= MDCR_EL2_TPMCR,
+		.mask		= MDCR_EL2_TPMCR,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_MDCR_TPM] = {
+		.index		= MDCR_EL2,
+		.value		= MDCR_EL2_TPM,
+		.mask		= MDCR_EL2_TPM,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_MDCR_TDE] = {
+		.index		= MDCR_EL2,
+		.value		= MDCR_EL2_TDE,
+		.mask		= MDCR_EL2_TDE,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_MDCR_TDA] = {
+		.index		= MDCR_EL2,
+		.value		= MDCR_EL2_TDA,
+		.mask		= MDCR_EL2_TDA,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_MDCR_TDOSA] = {
+		.index		= MDCR_EL2,
+		.value		= MDCR_EL2_TDOSA,
+		.mask		= MDCR_EL2_TDOSA,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_MDCR_TDRA] = {
+		.index		= MDCR_EL2,
+		.value		= MDCR_EL2_TDRA,
+		.mask		= MDCR_EL2_TDRA,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_MDCR_E2PB] = {
+		.index		= MDCR_EL2,
+		.value		= 0,
+		.mask		= BIT(MDCR_EL2_E2PB_SHIFT),
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_MDCR_TPMS] = {
+		.index		= MDCR_EL2,
+		.value		= MDCR_EL2_TPMS,
+		.mask		= MDCR_EL2_TPMS,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_MDCR_TTRF] = {
+		.index		= MDCR_EL2,
+		.value		= MDCR_EL2_TTRF,
+		.mask		= MDCR_EL2_TTRF,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_MDCR_E2TB] = {
+		.index		= MDCR_EL2,
+		.value		= 0,
+		.mask		= BIT(MDCR_EL2_E2TB_SHIFT),
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_MDCR_TDCC] = {
+		.index		= MDCR_EL2,
+		.value		= MDCR_EL2_TDCC,
+		.mask		= MDCR_EL2_TDCC,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
 };
 
 #define MCB(id, ...)					\
@@ -266,6 +349,11 @@ static const enum coarse_grain_trap_id *coarse_control_combo[] = {
 	MCB(CGT_HCR_TPU_TICAB,		CGT_HCR_TPU, CGT_HCR_TICAB),
 	MCB(CGT_HCR_TPU_TOCU,		CGT_HCR_TPU, CGT_HCR_TOCU),
 	MCB(CGT_HCR_NV1_ENSCXT,		CGT_HCR_NV1, CGT_HCR_ENSCXT),
+	MCB(CGT_MDCR_TPM_TPMCR,		CGT_MDCR_TPM, CGT_MDCR_TPMCR),
+	MCB(CGT_MDCR_TDE_TDA,		CGT_MDCR_TDE, CGT_MDCR_TDA),
+	MCB(CGT_MDCR_TDE_TDOSA,		CGT_MDCR_TDE, CGT_MDCR_TDOSA),
+	MCB(CGT_MDCR_TDE_TDRA,		CGT_MDCR_TDE, CGT_MDCR_TDRA),
+	MCB(CGT_MDCR_TDCC_TDE_TDA,	CGT_MDCR_TDCC, CGT_MDCR_TDE_TDA),
 };
 
 typedef enum trap_behaviour (*complex_condition_check)(struct kvm_vcpu *);
@@ -593,6 +681,180 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initdata = {
 	SR_TRAP(SYS_ERXPFGCTL_EL1,	CGT_HCR_FIEN),
 	SR_TRAP(SYS_ERXPFGCDN_EL1,	CGT_HCR_FIEN),
 	SR_TRAP(SYS_SCXTNUM_EL0,	CGT_HCR_ENSCXT),
+	SR_TRAP(SYS_PMCR_EL0,		CGT_MDCR_TPM_TPMCR),
+	SR_TRAP(SYS_PMCNTENSET_EL0,	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMCNTENCLR_EL0,	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMOVSSET_EL0,	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMOVSCLR_EL0,	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMCEID0_EL0,	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMCEID1_EL0,	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMXEVTYPER_EL0,	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMSWINC_EL0,	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMSELR_EL0,		CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMXEVCNTR_EL0,	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMCCNTR_EL0,	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMUSERENR_EL0,	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMINTENSET_EL1,	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMINTENCLR_EL1,	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMMIR_EL1,		CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(0),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(1),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(2),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(3),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(4),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(5),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(6),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(7),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(8),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(9),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(10),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(11),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(12),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(13),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(14),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(15),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(16),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(17),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(18),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(19),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(20),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(21),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(22),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(23),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(24),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(25),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(26),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(27),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(28),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(29),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVCNTRn_EL0(30),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(0),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(1),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(2),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(3),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(4),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(5),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(6),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(7),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(8),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(9),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(10),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(11),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(12),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(13),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(14),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(15),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(16),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(17),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(18),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(19),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(20),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(21),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(22),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(23),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(24),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(25),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(26),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(27),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(28),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(29),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMEVTYPERn_EL0(30),	CGT_MDCR_TPM),
+	SR_TRAP(SYS_PMCCFILTR_EL0,	CGT_MDCR_TPM),
+	SR_TRAP(SYS_MDCCSR_EL0,		CGT_MDCR_TDCC_TDE_TDA),
+	SR_TRAP(SYS_MDCCINT_EL1,	CGT_MDCR_TDCC_TDE_TDA),
+	SR_TRAP(SYS_OSDTRRX_EL1,	CGT_MDCR_TDCC_TDE_TDA),
+	SR_TRAP(SYS_OSDTRTX_EL1,	CGT_MDCR_TDCC_TDE_TDA),
+	SR_TRAP(SYS_MDSCR_EL1,		CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_OSECCR_EL1,		CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(0),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(1),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(2),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(3),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(4),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(5),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(6),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(7),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(8),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(9),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(10),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(11),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(12),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(13),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(14),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBVRn_EL1(15),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(0),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(1),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(2),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(3),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(4),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(5),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(6),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(7),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(8),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(9),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(10),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(11),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(12),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(13),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(14),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGBCRn_EL1(15),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(0),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(1),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(2),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(3),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(4),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(5),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(6),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(7),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(8),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(9),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(10),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(11),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(12),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(13),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(14),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWVRn_EL1(15),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWCRn_EL1(0),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWCRn_EL1(1),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWCRn_EL1(2),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWCRn_EL1(3),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWCRn_EL1(4),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWCRn_EL1(5),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWCRn_EL1(6),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWCRn_EL1(7),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWCRn_EL1(8),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWCRn_EL1(9),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWCRn_EL1(10),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWCRn_EL1(11),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWCRn_EL1(12),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWCRn_EL1(13),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGWCRn_EL1(14),	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGCLAIMSET_EL1,	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGCLAIMCLR_EL1,	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_DBGAUTHSTATUS_EL1,	CGT_MDCR_TDE_TDA),
+	SR_TRAP(SYS_OSLAR_EL1,		CGT_MDCR_TDE_TDOSA),
+	SR_TRAP(SYS_OSLSR_EL1,		CGT_MDCR_TDE_TDOSA),
+	SR_TRAP(SYS_OSDLR_EL1,		CGT_MDCR_TDE_TDOSA),
+	SR_TRAP(SYS_DBGPRCR_EL1,	CGT_MDCR_TDE_TDOSA),
+	SR_TRAP(SYS_MDRAR_EL1,		CGT_MDCR_TDE_TDRA),
+	SR_TRAP(SYS_PMBLIMITR_EL1,	CGT_MDCR_E2PB),
+	SR_TRAP(SYS_PMBPTR_EL1,		CGT_MDCR_E2PB),
+	SR_TRAP(SYS_PMBSR_EL1,		CGT_MDCR_E2PB),
+	SR_TRAP(SYS_PMSCR_EL1,		CGT_MDCR_TPMS),
+	SR_TRAP(SYS_PMSEVFR_EL1,	CGT_MDCR_TPMS),
+	SR_TRAP(SYS_PMSFCR_EL1,		CGT_MDCR_TPMS),
+	SR_TRAP(SYS_PMSICR_EL1,		CGT_MDCR_TPMS),
+	SR_TRAP(SYS_PMSIDR_EL1,		CGT_MDCR_TPMS),
+	SR_TRAP(SYS_PMSIRR_EL1,		CGT_MDCR_TPMS),
+	SR_TRAP(SYS_PMSLATFR_EL1,	CGT_MDCR_TPMS),
+	SR_TRAP(SYS_PMSNEVFR_EL1,	CGT_MDCR_TPMS),
+	SR_TRAP(SYS_TRFCR_EL1,		CGT_MDCR_TTRF),
+	SR_TRAP(SYS_TRBBASER_EL1,	CGT_MDCR_E2TB),
+	SR_TRAP(SYS_TRBLIMITR_EL1,	CGT_MDCR_E2TB),
+	SR_TRAP(SYS_TRBMAR_EL1, 	CGT_MDCR_E2TB),
+	SR_TRAP(SYS_TRBPTR_EL1, 	CGT_MDCR_E2TB),
+	SR_TRAP(SYS_TRBSR_EL1, 		CGT_MDCR_E2TB),
+	SR_TRAP(SYS_TRBTRG_EL1,		CGT_MDCR_E2TB),
 };
 
 static DEFINE_XARRAY(sr_forward_xa);
-- 
2.34.1

[PATCH 19/27] KVM: arm64: nv: Add trap forwarding for CNTHCTL_EL2

[Marc Zyngier]

Describe the CNTHCTL_EL2 register, and associate it with all the sysregs
it allows to trap.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/kvm/emulate-nested.c | 37 ++++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
index 25e4842ac334..c07c0f3361d7 100644
--- a/arch/arm64/kvm/emulate-nested.c
+++ b/arch/arm64/kvm/emulate-nested.c
@@ -98,9 +98,11 @@ enum coarse_grain_trap_id {
 
 	/*
 	 * Anything after this point requires a callback evaluating a
-	 * complex trap condition. Hopefully we'll never need this...
+	 * complex trap condition. Ugly stuff.
 	 */
 	__COMPLEX_CONDITIONS__,
+	CGT_CNTHCTL_EL1PCTEN = __COMPLEX_CONDITIONS__,
+	CGT_CNTHCTL_EL1PTEN,
 };
 
 static const struct trap_bits coarse_trap_bits[] = {
@@ -358,9 +360,37 @@ static const enum coarse_grain_trap_id *coarse_control_combo[] = {
 
 typedef enum trap_behaviour (*complex_condition_check)(struct kvm_vcpu *);
 
+static u64 get_sanitized_cnthctl(struct kvm_vcpu *vcpu)
+{
+	u64 val = __vcpu_sys_reg(vcpu, CNTHCTL_EL2);
+
+	if (!vcpu_el2_e2h_is_set(vcpu))
+		val = (val & (CNTHCTL_EL1PCEN | CNTHCTL_EL1PCTEN)) << 10;
+
+	return val;
+}
+
+static enum trap_behaviour check_cnthctl_el1pcten(struct kvm_vcpu *vcpu)
+{
+	if (get_sanitized_cnthctl(vcpu) & (CNTHCTL_EL1PCTEN << 10))
+		return BEHAVE_HANDLE_LOCALLY;
+
+	return BEHAVE_FORWARD_ANY;
+}
+
+static enum trap_behaviour check_cnthctl_el1pten(struct kvm_vcpu *vcpu)
+{
+	if (get_sanitized_cnthctl(vcpu) & (CNTHCTL_EL1PCEN << 10))
+		return BEHAVE_HANDLE_LOCALLY;
+
+	return BEHAVE_FORWARD_ANY;
+}
+
 #define CCC(id, fn)	[id - __COMPLEX_CONDITIONS__] = fn
 
 static const complex_condition_check ccc[] = {
+	CCC(CGT_CNTHCTL_EL1PCTEN, check_cnthctl_el1pcten),
+	CCC(CGT_CNTHCTL_EL1PTEN, check_cnthctl_el1pten),
 };
 
 /*
@@ -855,6 +885,11 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initdata = {
 	SR_TRAP(SYS_TRBPTR_EL1, 	CGT_MDCR_E2TB),
 	SR_TRAP(SYS_TRBSR_EL1, 		CGT_MDCR_E2TB),
 	SR_TRAP(SYS_TRBTRG_EL1,		CGT_MDCR_E2TB),
+	SR_TRAP(SYS_CNTP_TVAL_EL0,	CGT_CNTHCTL_EL1PTEN),
+	SR_TRAP(SYS_CNTP_CVAL_EL0,	CGT_CNTHCTL_EL1PTEN),
+	SR_TRAP(SYS_CNTP_CTL_EL0,	CGT_CNTHCTL_EL1PTEN),
+	SR_TRAP(SYS_CNTPCT_EL0,		CGT_CNTHCTL_EL1PCTEN),
+	SR_TRAP(SYS_CNTPCTSS_EL0,	CGT_CNTHCTL_EL1PCTEN),
 };
 
 static DEFINE_XARRAY(sr_forward_xa);
-- 
2.34.1

[PATCH 20/27] KVM: arm64: nv: Add trap forwarding for HFGxTR_EL2

[Marc Zyngier]

Fine Grained Traps are fun. Not.

Implement the trap forwarding for traps describer by HFGxTR_EL2,
reusing the Coarse Grained Traps infrastructure previously implemented.

Each sysreg/instruction inserted in the xarray gets a FGT group
(vaguely equivalent to a register number), a bit number in that register,
and a polarity.

It is then pretty easy to check the FGT state at handling time, just
like we do for the coarse version (it is just faster).

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/kvm/emulate-nested.c | 133 +++++++++++++++++++++++++++++++-
 1 file changed, 132 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
index c07c0f3361d7..51b9df19bdfa 100644
--- a/arch/arm64/kvm/emulate-nested.c
+++ b/arch/arm64/kvm/emulate-nested.c
@@ -894,6 +894,88 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initdata = {
 
 static DEFINE_XARRAY(sr_forward_xa);
 
+enum fgt_group_id {
+	__NO_FGT_GROUP__,
+	HFGxTR_GROUP,
+};
+
+#define SR_FGT(sr, g, b, p)					\
+	{							\
+		.encoding	= sr,				\
+		.end		= sr,				\
+		.tc		= {				\
+			.fgt = g ## _GROUP,			\
+			.bit = g ## _EL2_ ## b ## _SHIFT,	\
+			.pol = p,				\
+		},						\
+	}
+
+static const struct encoding_to_trap_config encoding_to_fgt[] __initdata = {
+	/* HFGTR_EL2, HFGWTR_EL2 */
+	SR_FGT(SYS_TPIDR2_EL0,		HFGxTR, nTPIDR2_EL0, 0),
+	SR_FGT(SYS_SMPRI_EL1,		HFGxTR, nSMPRI_EL1, 0),
+	SR_FGT(SYS_ACCDATA_EL1,		HFGxTR, nACCDATA_EL1, 0),
+	SR_FGT(SYS_ERXADDR_EL1,		HFGxTR, ERXADDR_EL1, 1),
+	SR_FGT(SYS_ERXPFGCDN_EL1,	HFGxTR, ERXPFGCDN_EL1, 1),
+	SR_FGT(SYS_ERXPFGCTL_EL1,	HFGxTR, ERXPFGCTL_EL1, 1),
+	SR_FGT(SYS_ERXPFGF_EL1,		HFGxTR, ERXPFGF_EL1, 1),
+	SR_FGT(SYS_ERXMISC0_EL1,	HFGxTR, ERXMISCn_EL1, 1),
+	SR_FGT(SYS_ERXMISC1_EL1,	HFGxTR, ERXMISCn_EL1, 1),
+	SR_FGT(SYS_ERXMISC2_EL1,	HFGxTR, ERXMISCn_EL1, 1),
+	SR_FGT(SYS_ERXMISC3_EL1,	HFGxTR, ERXMISCn_EL1, 1),
+	SR_FGT(SYS_ERXSTATUS_EL1,	HFGxTR, ERXSTATUS_EL1, 1),
+	SR_FGT(SYS_ERXCTLR_EL1,		HFGxTR, ERXCTLR_EL1, 1),
+	SR_FGT(SYS_ERXFR_EL1,		HFGxTR, ERXFR_EL1, 1),
+	SR_FGT(SYS_ERRSELR_EL1,		HFGxTR, ERRSELR_EL1, 1),
+	SR_FGT(SYS_ERRIDR_EL1,		HFGxTR, ERRIDR_EL1, 1),
+	SR_FGT(SYS_ICC_IGRPEN0_EL1,	HFGxTR, ICC_IGRPENn_EL1, 1),
+	SR_FGT(SYS_ICC_IGRPEN1_EL1,	HFGxTR, ICC_IGRPENn_EL1, 1),
+	SR_FGT(SYS_VBAR_EL1,		HFGxTR, VBAR_EL1, 1),
+	SR_FGT(SYS_TTBR1_EL1,		HFGxTR, TTBR1_EL1, 1),
+	SR_FGT(SYS_TTBR0_EL1,		HFGxTR, TTBR0_EL1, 1),
+	SR_FGT(SYS_TPIDR_EL0,		HFGxTR, TPIDR_EL0, 1),
+	SR_FGT(SYS_TPIDRRO_EL0,		HFGxTR, TPIDRRO_EL0, 1),
+	SR_FGT(SYS_TPIDR_EL1,		HFGxTR, TPIDR_EL1, 1),
+	SR_FGT(SYS_TCR_EL1,		HFGxTR, TCR_EL1, 1),
+	SR_FGT(SYS_SCXTNUM_EL0,		HFGxTR, SCXTNUM_EL0, 1),
+	SR_FGT(SYS_SCXTNUM_EL1, 	HFGxTR, SCXTNUM_EL1, 1),
+	SR_FGT(SYS_SCTLR_EL1, 		HFGxTR, SCTLR_EL1, 1),
+	SR_FGT(SYS_REVIDR_EL1, 		HFGxTR, REVIDR_EL1, 1),
+	SR_FGT(SYS_PAR_EL1, 		HFGxTR, PAR_EL1, 1),
+	SR_FGT(SYS_MPIDR_EL1, 		HFGxTR, MPIDR_EL1, 1),
+	SR_FGT(SYS_MIDR_EL1, 		HFGxTR, MIDR_EL1, 1),
+	SR_FGT(SYS_MAIR_EL1, 		HFGxTR, MAIR_EL1, 1),
+	SR_FGT(SYS_LORSA_EL1, 		HFGxTR, LORSA_EL1, 1),
+	SR_FGT(SYS_LORN_EL1, 		HFGxTR, LORN_EL1, 1),
+	SR_FGT(SYS_LORID_EL1, 		HFGxTR, LORID_EL1, 1),
+	SR_FGT(SYS_LOREA_EL1, 		HFGxTR, LOREA_EL1, 1),
+	SR_FGT(SYS_LORC_EL1, 		HFGxTR, LORC_EL1, 1),
+	SR_FGT(SYS_ISR_EL1, 		HFGxTR, ISR_EL1, 1),
+	SR_FGT(SYS_FAR_EL1, 		HFGxTR, FAR_EL1, 1),
+	SR_FGT(SYS_ESR_EL1, 		HFGxTR, ESR_EL1, 1),
+	SR_FGT(SYS_DCZID_EL0, 		HFGxTR, DCZID_EL0, 1),
+	SR_FGT(SYS_CTR_EL0, 		HFGxTR, CTR_EL0, 1),
+	SR_FGT(SYS_CSSELR_EL1, 		HFGxTR, CSSELR_EL1, 1),
+	SR_FGT(SYS_CPACR_EL1, 		HFGxTR, CPACR_EL1, 1),
+	SR_FGT(SYS_CONTEXTIDR_EL1, 	HFGxTR, CONTEXTIDR_EL1, 1),
+	SR_FGT(SYS_CLIDR_EL1, 		HFGxTR, CLIDR_EL1, 1),
+	SR_FGT(SYS_CCSIDR_EL1, 		HFGxTR, CCSIDR_EL1, 1),
+	SR_FGT(SYS_APIBKEYLO_EL1, 	HFGxTR, APIBKey, 1),
+	SR_FGT(SYS_APIBKEYHI_EL1, 	HFGxTR, APIBKey, 1),
+	SR_FGT(SYS_APIAKEYLO_EL1, 	HFGxTR, APIAKey, 1),
+	SR_FGT(SYS_APIAKEYHI_EL1, 	HFGxTR, APIAKey, 1),
+	SR_FGT(SYS_APGAKEYLO_EL1, 	HFGxTR, APGAKey, 1),
+	SR_FGT(SYS_APGAKEYHI_EL1, 	HFGxTR, APGAKey, 1),
+	SR_FGT(SYS_APDBKEYLO_EL1, 	HFGxTR, APDBKey, 1),
+	SR_FGT(SYS_APDBKEYHI_EL1, 	HFGxTR, APDBKey, 1),
+	SR_FGT(SYS_APDAKEYLO_EL1, 	HFGxTR, APDAKey, 1),
+	SR_FGT(SYS_APDAKEYHI_EL1, 	HFGxTR, APDAKey, 1),
+	SR_FGT(SYS_AMAIR_EL1, 		HFGxTR, AMAIR_EL1, 1),
+	SR_FGT(SYS_AIDR_EL1, 		HFGxTR, AIDR_EL1, 1),
+	SR_FGT(SYS_AFSR1_EL1, 		HFGxTR, AFSR1_EL1, 1),
+	SR_FGT(SYS_AFSR0_EL1, 		HFGxTR, AFSR0_EL1, 1),
+};
+
 static union trap_config get_trap_config(u32 sysreg)
 {
 	return (union trap_config) {
@@ -915,6 +997,27 @@ void __init populate_nv_trap_config(void)
 	kvm_info("nv: %ld coarse grained trap handlers\n",
 		 ARRAY_SIZE(encoding_to_cgt));
 
+	for (int i = 0; i < ARRAY_SIZE(encoding_to_fgt); i++) {
+		const struct encoding_to_trap_config *fgt = &encoding_to_fgt[i];
+		union trap_config tc;
+
+		tc = get_trap_config(fgt->encoding);
+
+		WARN(tc.fgt,
+		     "Duplicate FGT for sys_reg(%d, %d, %d, %d, %d)\n",
+		     sys_reg_Op0(fgt->encoding),
+		     sys_reg_Op1(fgt->encoding),
+		     sys_reg_CRn(fgt->encoding),
+		     sys_reg_CRm(fgt->encoding),
+		     sys_reg_Op2(fgt->encoding));
+
+		tc.val |= fgt->tc.val;
+		xa_store(&sr_forward_xa, fgt->encoding,
+			 xa_mk_value(tc.val), GFP_KERNEL);
+	}
+
+	kvm_info("nv: %ld fine grained trap handlers\n",
+		 ARRAY_SIZE(encoding_to_fgt));
 }
 
 static enum trap_behaviour get_behaviour(struct kvm_vcpu *vcpu,
@@ -964,13 +1067,26 @@ static enum trap_behaviour compute_behaviour(struct kvm_vcpu *vcpu,
 	return __do_compute_behaviour(vcpu, tc.cgt, b);
 }
 
+static bool check_fgt_bit(u64 val, const union trap_config tc)
+{
+	return ((val >> tc.bit) & 1) == tc.pol;
+}
+
+#define sanitised_sys_reg(vcpu, reg)			\
+	({						\
+		u64 __val;				\
+		__val = __vcpu_sys_reg(vcpu, reg);	\
+		__val &= ~__ ## reg ## _RES0;		\
+		(__val);				\
+	})
+
 bool __check_nv_sr_forward(struct kvm_vcpu *vcpu)
 {
 	union trap_config tc;
 	enum trap_behaviour b;
 	bool is_read;
 	u32 sysreg;
-	u64 esr;
+	u64 esr, val;
 
 	if (!vcpu_has_nv(vcpu) || is_hyp_ctxt(vcpu))
 		return false;
@@ -981,6 +1097,21 @@ bool __check_nv_sr_forward(struct kvm_vcpu *vcpu)
 
 	tc = get_trap_config(sysreg);
 
+	switch ((enum fgt_group_id)tc.fgt) {
+	case __NO_FGT_GROUP__:
+		break;
+
+	case HFGxTR_GROUP:
+		if (is_read)
+			val = sanitised_sys_reg(vcpu, HFGRTR_EL2);
+		else
+			val = sanitised_sys_reg(vcpu, HFGWTR_EL2);
+		break;
+	}
+
+	if (tc.fgt != __NO_FGT_GROUP__ && check_fgt_bit(val, tc))
+		goto inject;
+
 	b = compute_behaviour(vcpu, tc);
 
 	if (((b & BEHAVE_FORWARD_READ) && is_read) ||
-- 
2.34.1

[PATCH 21/27] KVM: arm64: nv: Add trap forwarding for HFGITR_EL2

[Marc Zyngier]

Similarly, implement the trap forwarding for instructions affected
by HFGITR_EL2.

Note that the TLBI*nXS instructions should be affected by HCRX_EL2,
which will be dealt with down the line.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/kvm_arm.h |   4 ++
 arch/arm64/kvm/emulate-nested.c  | 109 +++++++++++++++++++++++++++++++
 2 files changed, 113 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index 85908aa18908..809bc86acefd 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -354,6 +354,10 @@
 #define __HFGWTR_EL2_MASK	GENMASK(49, 0)
 #define __HFGWTR_EL2_nMASK	(GENMASK(55, 54) | BIT(50))
 
+#define __HFGITR_EL2_RES0	GENMASK(63, 57)
+#define __HFGITR_EL2_MASK	GENMASK(54, 0)
+#define __HFGITR_EL2_nMASK	GENMASK(56, 55)
+
 /* Hyp Prefetch Fault Address Register (HPFAR/HDFAR) */
 #define HPFAR_MASK	(~UL(0xf))
 /*
diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
index 51b9df19bdfa..d4838b395566 100644
--- a/arch/arm64/kvm/emulate-nested.c
+++ b/arch/arm64/kvm/emulate-nested.c
@@ -897,6 +897,7 @@ static DEFINE_XARRAY(sr_forward_xa);
 enum fgt_group_id {
 	__NO_FGT_GROUP__,
 	HFGxTR_GROUP,
+	HFGITR_GROUP,
 };
 
 #define SR_FGT(sr, g, b, p)					\
@@ -974,6 +975,110 @@ static const struct encoding_to_trap_config encoding_to_fgt[] __initdata = {
 	SR_FGT(SYS_AIDR_EL1, 		HFGxTR, AIDR_EL1, 1),
 	SR_FGT(SYS_AFSR1_EL1, 		HFGxTR, AFSR1_EL1, 1),
 	SR_FGT(SYS_AFSR0_EL1, 		HFGxTR, AFSR0_EL1, 1),
+	/* HFGITR_EL2 */
+	SR_FGT(OP_BRB_IALL, 		HFGITR, nBRBIALL, 0),
+	SR_FGT(OP_BRB_INJ, 		HFGITR, nBRBINJ, 0),
+	SR_FGT(SYS_DC_CVAC, 		HFGITR, DCCVAC, 1),
+	SR_FGT(SYS_DC_CGVAC, 		HFGITR, DCCVAC, 1),
+	SR_FGT(SYS_DC_CGDVAC, 		HFGITR, DCCVAC, 1),
+	SR_FGT(OP_CPP_RCTX, 		HFGITR, CPPRCTX, 1),
+	SR_FGT(OP_DVP_RCTX, 		HFGITR, DVPRCTX, 1),
+	SR_FGT(OP_CFP_RCTX, 		HFGITR, CFPRCTX, 1),
+	SR_FGT(OP_TLBI_VAALE1, 		HFGITR, TLBIVAALE1, 1),
+	SR_FGT(OP_TLBI_VALE1, 		HFGITR, TLBIVALE1, 1),
+	SR_FGT(OP_TLBI_VAAE1, 		HFGITR, TLBIVAAE1, 1),
+	SR_FGT(OP_TLBI_ASIDE1, 		HFGITR, TLBIASIDE1, 1),
+	SR_FGT(OP_TLBI_VAE1, 		HFGITR, TLBIVAE1, 1),
+	SR_FGT(OP_TLBI_VMALLE1, 	HFGITR, TLBIVMALLE1, 1),
+	SR_FGT(OP_TLBI_RVAALE1, 	HFGITR, TLBIRVAALE1, 1),
+	SR_FGT(OP_TLBI_RVALE1, 		HFGITR, TLBIRVALE1, 1),
+	SR_FGT(OP_TLBI_RVAAE1, 		HFGITR, TLBIRVAAE1, 1),
+	SR_FGT(OP_TLBI_RVAE1, 		HFGITR, TLBIRVAE1, 1),
+	SR_FGT(OP_TLBI_RVAALE1IS, 	HFGITR, TLBIRVAALE1IS, 1),
+	SR_FGT(OP_TLBI_RVALE1IS, 	HFGITR, TLBIRVALE1IS, 1),
+	SR_FGT(OP_TLBI_RVAAE1IS, 	HFGITR, TLBIRVAAE1IS, 1),
+	SR_FGT(OP_TLBI_RVAE1IS, 	HFGITR, TLBIRVAE1IS, 1),
+	SR_FGT(OP_TLBI_VAALE1IS, 	HFGITR, TLBIVAALE1IS, 1),
+	SR_FGT(OP_TLBI_VALE1IS, 	HFGITR, TLBIVALE1IS, 1),
+	SR_FGT(OP_TLBI_VAAE1IS, 	HFGITR, TLBIVAAE1IS, 1),
+	SR_FGT(OP_TLBI_ASIDE1IS, 	HFGITR, TLBIASIDE1IS, 1),
+	SR_FGT(OP_TLBI_VAE1IS, 		HFGITR, TLBIVAE1IS, 1),
+	SR_FGT(OP_TLBI_VMALLE1IS, 	HFGITR, TLBIVMALLE1IS, 1),
+	SR_FGT(OP_TLBI_RVAALE1OS, 	HFGITR, TLBIRVAALE1OS, 1),
+	SR_FGT(OP_TLBI_RVALE1OS, 	HFGITR, TLBIRVALE1OS, 1),
+	SR_FGT(OP_TLBI_RVAAE1OS, 	HFGITR, TLBIRVAAE1OS, 1),
+	SR_FGT(OP_TLBI_RVAE1OS, 	HFGITR, TLBIRVAE1OS, 1),
+	SR_FGT(OP_TLBI_VAALE1OS, 	HFGITR, TLBIVAALE1OS, 1),
+	SR_FGT(OP_TLBI_VALE1OS, 	HFGITR, TLBIVALE1OS, 1),
+	SR_FGT(OP_TLBI_VAAE1OS, 	HFGITR, TLBIVAAE1OS, 1),
+	SR_FGT(OP_TLBI_ASIDE1OS, 	HFGITR, TLBIASIDE1OS, 1),
+	SR_FGT(OP_TLBI_VAE1OS, 		HFGITR, TLBIVAE1OS, 1),
+	SR_FGT(OP_TLBI_VMALLE1OS, 	HFGITR, TLBIVMALLE1OS, 1),
+	/* FIXME: nXS variants must be checked against HCRX_EL2.FGTnXS */
+	SR_FGT(OP_TLBI_VAALE1NXS, 	HFGITR, TLBIVAALE1, 1),
+	SR_FGT(OP_TLBI_VALE1NXS, 	HFGITR, TLBIVALE1, 1),
+	SR_FGT(OP_TLBI_VAAE1NXS, 	HFGITR, TLBIVAAE1, 1),
+	SR_FGT(OP_TLBI_ASIDE1NXS, 	HFGITR, TLBIASIDE1, 1),
+	SR_FGT(OP_TLBI_VAE1NXS, 	HFGITR, TLBIVAE1, 1),
+	SR_FGT(OP_TLBI_VMALLE1NXS, 	HFGITR, TLBIVMALLE1, 1),
+	SR_FGT(OP_TLBI_RVAALE1NXS, 	HFGITR, TLBIRVAALE1, 1),
+	SR_FGT(OP_TLBI_RVALE1NXS, 	HFGITR, TLBIRVALE1, 1),
+	SR_FGT(OP_TLBI_RVAAE1NXS, 	HFGITR, TLBIRVAAE1, 1),
+	SR_FGT(OP_TLBI_RVAE1NXS, 	HFGITR, TLBIRVAE1, 1),
+	SR_FGT(OP_TLBI_RVAALE1ISNXS, 	HFGITR, TLBIRVAALE1IS, 1),
+	SR_FGT(OP_TLBI_RVALE1ISNXS, 	HFGITR, TLBIRVALE1IS, 1),
+	SR_FGT(OP_TLBI_RVAAE1ISNXS, 	HFGITR, TLBIRVAAE1IS, 1),
+	SR_FGT(OP_TLBI_RVAE1ISNXS, 	HFGITR, TLBIRVAE1IS, 1),
+	SR_FGT(OP_TLBI_VAALE1ISNXS, 	HFGITR, TLBIVAALE1IS, 1),
+	SR_FGT(OP_TLBI_VALE1ISNXS, 	HFGITR, TLBIVALE1IS, 1),
+	SR_FGT(OP_TLBI_VAAE1ISNXS, 	HFGITR, TLBIVAAE1IS, 1),
+	SR_FGT(OP_TLBI_ASIDE1ISNXS, 	HFGITR, TLBIASIDE1IS, 1),
+	SR_FGT(OP_TLBI_VAE1ISNXS, 	HFGITR, TLBIVAE1IS, 1),
+	SR_FGT(OP_TLBI_VMALLE1ISNXS, 	HFGITR, TLBIVMALLE1IS, 1),
+	SR_FGT(OP_TLBI_RVAALE1OSNXS, 	HFGITR, TLBIRVAALE1OS, 1),
+	SR_FGT(OP_TLBI_RVALE1OSNXS, 	HFGITR, TLBIRVALE1OS, 1),
+	SR_FGT(OP_TLBI_RVAAE1OSNXS, 	HFGITR, TLBIRVAAE1OS, 1),
+	SR_FGT(OP_TLBI_RVAE1OSNXS, 	HFGITR, TLBIRVAE1OS, 1),
+	SR_FGT(OP_TLBI_VAALE1OSNXS, 	HFGITR, TLBIVAALE1OS, 1),
+	SR_FGT(OP_TLBI_VALE1OSNXS, 	HFGITR, TLBIVALE1OS, 1),
+	SR_FGT(OP_TLBI_VAAE1OSNXS, 	HFGITR, TLBIVAAE1OS, 1),
+	SR_FGT(OP_TLBI_ASIDE1OSNXS, 	HFGITR, TLBIASIDE1OS, 1),
+	SR_FGT(OP_TLBI_VAE1OSNXS, 	HFGITR, TLBIVAE1OS, 1),
+	SR_FGT(OP_TLBI_VMALLE1OSNXS, 	HFGITR, TLBIVMALLE1OS, 1),
+	SR_FGT(OP_AT_S1E1WP, 		HFGITR, ATS1E1WP, 1),
+	SR_FGT(OP_AT_S1E1RP, 		HFGITR, ATS1E1RP, 1),
+	SR_FGT(OP_AT_S1E0W, 		HFGITR, ATS1E0W, 1),
+	SR_FGT(OP_AT_S1E0R, 		HFGITR, ATS1E0R, 1),
+	SR_FGT(OP_AT_S1E1W, 		HFGITR, ATS1E1W, 1),
+	SR_FGT(OP_AT_S1E1R, 		HFGITR, ATS1E1R, 1),
+	SR_FGT(SYS_DC_ZVA, 		HFGITR, DCZVA, 1),
+	SR_FGT(SYS_DC_GVA, 		HFGITR, DCZVA, 1),
+	SR_FGT(SYS_DC_GZVA, 		HFGITR, DCZVA, 1),
+	SR_FGT(SYS_DC_CIVAC, 		HFGITR, DCCIVAC, 1),
+	SR_FGT(SYS_DC_CIGVAC, 		HFGITR, DCCIVAC, 1),
+	SR_FGT(SYS_DC_CIGDVAC, 		HFGITR, DCCIVAC, 1),
+	SR_FGT(SYS_DC_CVADP, 		HFGITR, DCCVADP, 1),
+	SR_FGT(SYS_DC_CGVADP, 		HFGITR, DCCVADP, 1),
+	SR_FGT(SYS_DC_CGDVADP, 		HFGITR, DCCVADP, 1),
+	SR_FGT(SYS_DC_CVAP, 		HFGITR, DCCVAP, 1),
+	SR_FGT(SYS_DC_CGVAP, 		HFGITR, DCCVAP, 1),
+	SR_FGT(SYS_DC_CGDVAP, 		HFGITR, DCCVAP, 1),
+	SR_FGT(SYS_DC_CVAU, 		HFGITR, DCCVAU, 1),
+	SR_FGT(SYS_DC_CISW, 		HFGITR, DCCISW, 1),
+	SR_FGT(SYS_DC_CIGSW, 		HFGITR, DCCISW, 1),
+	SR_FGT(SYS_DC_CIGDSW, 		HFGITR, DCCISW, 1),
+	SR_FGT(SYS_DC_CSW, 		HFGITR, DCCSW, 1),
+	SR_FGT(SYS_DC_CGSW, 		HFGITR, DCCSW, 1),
+	SR_FGT(SYS_DC_CGDSW, 		HFGITR, DCCSW, 1),
+	SR_FGT(SYS_DC_ISW, 		HFGITR, DCISW, 1),
+	SR_FGT(SYS_DC_IGSW, 		HFGITR, DCISW, 1),
+	SR_FGT(SYS_DC_IGDSW, 		HFGITR, DCISW, 1),
+	SR_FGT(SYS_DC_IVAC, 		HFGITR, DCIVAC, 1),
+	SR_FGT(SYS_DC_IGVAC, 		HFGITR, DCIVAC, 1),
+	SR_FGT(SYS_DC_IGDVAC, 		HFGITR, DCIVAC, 1),
+	SR_FGT(SYS_IC_IVAU, 		HFGITR, ICIVAU, 1),
+	SR_FGT(SYS_IC_IALLU, 		HFGITR, ICIALLU, 1),
+	SR_FGT(SYS_IC_IALLUIS, 		HFGITR, ICIALLUIS, 1),
 };
 
 static union trap_config get_trap_config(u32 sysreg)
@@ -1107,6 +1212,10 @@ bool __check_nv_sr_forward(struct kvm_vcpu *vcpu)
 		else
 			val = sanitised_sys_reg(vcpu, HFGWTR_EL2);
 		break;
+
+	case HFGITR_GROUP:
+		val = sanitised_sys_reg(vcpu, HFGITR_EL2);
+		break;
 	}
 
 	if (tc.fgt != __NO_FGT_GROUP__ && check_fgt_bit(val, tc))
-- 
2.34.1

[PATCH 22/27] KVM: arm64: nv: Add trap forwarding for HDFGxTR_EL2

[Marc Zyngier]

... and finally, the Debug version of FGT, with its *enormous*
list of trapped registers.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/kvm_arm.h |  11 +
 arch/arm64/kvm/emulate-nested.c  | 460 +++++++++++++++++++++++++++++++
 2 files changed, 471 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index 809bc86acefd..d229f238c3b6 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -358,6 +358,17 @@
 #define __HFGITR_EL2_MASK	GENMASK(54, 0)
 #define __HFGITR_EL2_nMASK	GENMASK(56, 55)
 
+#define __HDFGRTR_EL2_RES0	(BIT(49) | BIT(42) | GENMASK(39, 38) |	\
+				 GENMASK(21, 20) | BIT(8))
+#define __HDFGRTR_EL2_MASK	~__HDFGRTR_EL2_nMASK
+#define __HDFGRTR_EL2_nMASK	GENMASK(62, 59)
+
+#define __HDFGWTR_EL2_RES0	(BIT(63) | GENMASK(59, 58) | BIT(51) | BIT(47) | \
+				 BIT(43) | GENMASK(40, 38) | BIT(34) | BIT(30) | \
+				 BIT(22) | BIT(9) | BIT(6))
+#define __HDFGWTR_EL2_MASK	~__HDFGWTR_EL2_nMASK
+#define __HDFGWTR_EL2_nMASK	GENMASK(62, 60)
+
 /* Hyp Prefetch Fault Address Register (HPFAR/HDFAR) */
 #define HPFAR_MASK	(~UL(0xf))
 /*
diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
index d4838b395566..739b35e6ef18 100644
--- a/arch/arm64/kvm/emulate-nested.c
+++ b/arch/arm64/kvm/emulate-nested.c
@@ -897,6 +897,8 @@ static DEFINE_XARRAY(sr_forward_xa);
 enum fgt_group_id {
 	__NO_FGT_GROUP__,
 	HFGxTR_GROUP,
+	HDFGRTR_GROUP,
+	HDFGWTR_GROUP,
 	HFGITR_GROUP,
 };
 
@@ -1079,6 +1081,456 @@ static const struct encoding_to_trap_config encoding_to_fgt[] __initdata = {
 	SR_FGT(SYS_IC_IVAU, 		HFGITR, ICIVAU, 1),
 	SR_FGT(SYS_IC_IALLU, 		HFGITR, ICIALLU, 1),
 	SR_FGT(SYS_IC_IALLUIS, 		HFGITR, ICIALLUIS, 1),
+	/* HDFGRTR_EL2 */
+	SR_FGT(SYS_PMBIDR_EL1, 		HDFGRTR, PMBIDR_EL1, 1),
+	SR_FGT(SYS_PMSNEVFR_EL1, 	HDFGRTR, nPMSNEVFR_EL1, 0),
+	SR_FGT(SYS_BRBINF_EL1(0), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(1), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(2), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(3), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(4), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(5), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(6), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(7), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(8), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(9), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(10), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(11), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(12), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(13), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(14), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(15), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(16), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(17), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(18), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(19), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(20), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(21), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(22), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(23), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(24), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(25), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(26), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(27), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(28), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(29), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(30), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINF_EL1(31), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBINFINJ_EL1, 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(0), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(1), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(2), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(3), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(4), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(5), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(6), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(7), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(8), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(9), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(10), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(11), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(12), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(13), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(14), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(15), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(16), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(17), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(18), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(19), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(20), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(21), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(22), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(23), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(24), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(25), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(26), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(27), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(28), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(29), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(30), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRC_EL1(31), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBSRCINJ_EL1, 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(0), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(1), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(2), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(3), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(4), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(5), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(6), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(7), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(8), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(9), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(10), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(11), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(12), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(13), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(14), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(15), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(16), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(17), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(18), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(19), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(20), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(21), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(22), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(23), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(24), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(25), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(26), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(27), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(28), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(29), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(30), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGT_EL1(31), 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTGTINJ_EL1, 	HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBTS_EL1, 		HDFGRTR, nBRBDATA, 0),
+	SR_FGT(SYS_BRBCR_EL1, 		HDFGRTR, nBRBCTL, 0),
+	SR_FGT(SYS_BRBFCR_EL1, 		HDFGRTR, nBRBCTL, 0),
+	SR_FGT(SYS_BRBIDR0_EL1, 	HDFGRTR, nBRBIDR, 0),
+	SR_FGT(SYS_PMCEID0_EL0, 	HDFGRTR, PMCEIDn_EL0, 1),
+	SR_FGT(SYS_PMCEID1_EL0, 	HDFGRTR, PMCEIDn_EL0, 1),
+	SR_FGT(SYS_PMUSERENR_EL0, 	HDFGRTR, PMUSERENR_EL0, 1),
+	SR_FGT(SYS_TRBTRG_EL1, 		HDFGRTR, TRBTRG_EL1, 1),
+	SR_FGT(SYS_TRBSR_EL1, 		HDFGRTR, TRBSR_EL1, 1),
+	SR_FGT(SYS_TRBPTR_EL1, 		HDFGRTR, TRBPTR_EL1, 1),
+	SR_FGT(SYS_TRBMAR_EL1, 		HDFGRTR, TRBMAR_EL1, 1),
+	SR_FGT(SYS_TRBLIMITR_EL1, 	HDFGRTR, TRBLIMITR_EL1, 1),
+	SR_FGT(SYS_TRBIDR_EL1, 		HDFGRTR, TRBIDR_EL1, 1),
+	SR_FGT(SYS_TRBBASER_EL1, 	HDFGRTR, TRBBASER_EL1, 1),
+	SR_FGT(SYS_TRCVICTLR, 		HDFGRTR, TRCVICTLR, 1),
+	SR_FGT(SYS_TRCSTATR, 		HDFGRTR, TRCSTATR, 1),
+	SR_FGT(SYS_TRCSSCSR(0), 	HDFGRTR, TRCSSCSRn, 1),
+	SR_FGT(SYS_TRCSSCSR(1), 	HDFGRTR, TRCSSCSRn, 1),
+	SR_FGT(SYS_TRCSSCSR(2), 	HDFGRTR, TRCSSCSRn, 1),
+	SR_FGT(SYS_TRCSSCSR(3), 	HDFGRTR, TRCSSCSRn, 1),
+	SR_FGT(SYS_TRCSSCSR(4), 	HDFGRTR, TRCSSCSRn, 1),
+	SR_FGT(SYS_TRCSSCSR(5), 	HDFGRTR, TRCSSCSRn, 1),
+	SR_FGT(SYS_TRCSSCSR(6), 	HDFGRTR, TRCSSCSRn, 1),
+	SR_FGT(SYS_TRCSSCSR(7), 	HDFGRTR, TRCSSCSRn, 1),
+	SR_FGT(SYS_TRCSEQSTR, 		HDFGRTR, TRCSEQSTR, 1),
+	SR_FGT(SYS_TRCPRGCTLR, 		HDFGRTR, TRCPRGCTLR, 1),
+	SR_FGT(SYS_TRCOSLSR, 		HDFGRTR, TRCOSLSR, 1),
+	SR_FGT(SYS_TRCIMSPEC0, 		HDFGRTR, TRCIMSPECn, 1),
+	SR_FGT(SYS_TRCIMSPEC(1), 	HDFGRTR, TRCIMSPECn, 1),
+	SR_FGT(SYS_TRCIMSPEC(2), 	HDFGRTR, TRCIMSPECn, 1),
+	SR_FGT(SYS_TRCIMSPEC(3), 	HDFGRTR, TRCIMSPECn, 1),
+	SR_FGT(SYS_TRCIMSPEC(4), 	HDFGRTR, TRCIMSPECn, 1),
+	SR_FGT(SYS_TRCIMSPEC(5), 	HDFGRTR, TRCIMSPECn, 1),
+	SR_FGT(SYS_TRCIMSPEC(6), 	HDFGRTR, TRCIMSPECn, 1),
+	SR_FGT(SYS_TRCIMSPEC(7), 	HDFGRTR, TRCIMSPECn, 1),
+	SR_FGT(SYS_TRCDEVARCH, 		HDFGRTR, TRCID, 1),
+	SR_FGT(SYS_TRCDEVID, 		HDFGRTR, TRCID, 1),
+	SR_FGT(SYS_TRCCNTVR(0), 	HDFGRTR, TRCCNTVRn, 1),
+	SR_FGT(SYS_TRCCNTVR(1), 	HDFGRTR, TRCCNTVRn, 1),
+	SR_FGT(SYS_TRCCNTVR(2), 	HDFGRTR, TRCCNTVRn, 1),
+	SR_FGT(SYS_TRCCNTVR(3), 	HDFGRTR, TRCCNTVRn, 1),
+	SR_FGT(SYS_TRCCLAIMCLR, 	HDFGRTR, TRCCLAIM, 1),
+	SR_FGT(SYS_TRCCLAIMSET, 	HDFGRTR, TRCCLAIM, 1),
+	SR_FGT(SYS_TRCAUXCTLR, 		HDFGRTR, TRCAUXCTLR, 1),
+	SR_FGT(SYS_TRCAUTHSTATUS, 	HDFGRTR, TRCAUTHSTATUS, 1),
+	SR_FGT(SYS_TRCACATR(0), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACATR(1), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACATR(2), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACATR(3), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACATR(4), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACATR(5), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACATR(6), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACATR(7), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACATR(8), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACATR(9), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACATR(10), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACATR(11), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACATR(12), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACATR(13), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACATR(14), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACATR(15), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(0), 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(1), 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(2), 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(3), 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(4), 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(5), 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(6), 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(7), 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(8), 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(9), 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(10), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(11), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(12), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(13), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(14), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCACVR(15), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCBBCTLR, 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCCCTLR, 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCIDCCTLR0, 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCIDCCTLR1, 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCIDCVR(0), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCIDCVR(1), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCIDCVR(2), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCIDCVR(3), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCIDCVR(4), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCIDCVR(5), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCIDCVR(6), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCIDCVR(7), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCNTCTLR(0), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCNTCTLR(1), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCNTCTLR(2), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCNTCTLR(3), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCNTRLDVR(0), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCNTRLDVR(1), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCNTRLDVR(2), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCNTRLDVR(3), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCCONFIGR, 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCEVENTCTL0R, 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCEVENTCTL1R, 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCEXTINSELR(0), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCEXTINSELR(1), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCEXTINSELR(2), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCEXTINSELR(3), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(2), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(3), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(4), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(5), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(6), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(7), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(8), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(9), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(10), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(11), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(12), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(13), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(14), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(15), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(16), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(17), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(18), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(19), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(20), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(21), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(22), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(23), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(24), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(25), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(26), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(27), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(28), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(29), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(30), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSCTLR(31), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCQCTLR, 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCRSR, 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSEQEVR(0), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSEQEVR(1), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSEQEVR(2), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSEQRSTEVR, 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSCCR(0), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSCCR(1), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSCCR(2), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSCCR(3), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSCCR(4), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSCCR(5), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSCCR(6), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSCCR(7), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSPCICR(0), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSPCICR(1), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSPCICR(2), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSPCICR(3), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSPCICR(4), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSPCICR(5), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSPCICR(6), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSSPCICR(7), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSTALLCTLR, 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCSYNCPR, 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCTRACEIDR, 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCTSCTLR, 		HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCVIIECTLR, 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCVIPCSSCTLR, 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCVISSCTLR, 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCVMIDCCTLR0, 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCVMIDCCTLR1, 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCVMIDCVR(0), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCVMIDCVR(1), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCVMIDCVR(2), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCVMIDCVR(3), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCVMIDCVR(4), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCVMIDCVR(5), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCVMIDCVR(6), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_TRCVMIDCVR(7), 	HDFGRTR, TRC, 1),
+	SR_FGT(SYS_PMSLATFR_EL1, 	HDFGRTR, PMSLATFR_EL1, 1),
+	SR_FGT(SYS_PMSIRR_EL1, 		HDFGRTR, PMSIRR_EL1, 1),
+	SR_FGT(SYS_PMSIDR_EL1, 		HDFGRTR, PMSIDR_EL1, 1),
+	SR_FGT(SYS_PMSICR_EL1, 		HDFGRTR, PMSICR_EL1, 1),
+	SR_FGT(SYS_PMSFCR_EL1, 		HDFGRTR, PMSFCR_EL1, 1),
+	SR_FGT(SYS_PMSEVFR_EL1, 	HDFGRTR, PMSEVFR_EL1, 1),
+	SR_FGT(SYS_PMSCR_EL1, 		HDFGRTR, PMSCR_EL1, 1),
+	SR_FGT(SYS_PMBSR_EL1, 		HDFGRTR, PMBSR_EL1, 1),
+	SR_FGT(SYS_PMBPTR_EL1, 		HDFGRTR, PMBPTR_EL1, 1),
+	SR_FGT(SYS_PMBLIMITR_EL1, 	HDFGRTR, PMBLIMITR_EL1, 1),
+	SR_FGT(SYS_PMMIR_EL1, 		HDFGRTR, PMMIR_EL1, 1),
+	SR_FGT(SYS_PMSELR_EL0, 		HDFGRTR, PMSELR_EL0, 1),
+	SR_FGT(SYS_PMOVSCLR_EL0, 	HDFGRTR, PMOVS, 1),
+	SR_FGT(SYS_PMOVSSET_EL0, 	HDFGRTR, PMOVS, 1),
+	SR_FGT(SYS_PMINTENCLR_EL1, 	HDFGRTR, PMINTEN, 1),
+	SR_FGT(SYS_PMINTENSET_EL1, 	HDFGRTR, PMINTEN, 1),
+	SR_FGT(SYS_PMCNTENCLR_EL0, 	HDFGRTR, PMCNTEN, 1),
+	SR_FGT(SYS_PMCNTENSET_EL0, 	HDFGRTR, PMCNTEN, 1),
+	SR_FGT(SYS_PMCCNTR_EL0, 	HDFGRTR, PMCCNTR_EL0, 1),
+	SR_FGT(SYS_PMCCFILTR_EL0, 	HDFGRTR, PMCCFILTR_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(0), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(1), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(2), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(3), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(4), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(5), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(6), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(7), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(8), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(9), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(10), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(11), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(12), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(13), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(14), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(15), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(16), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(17), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(18), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(19), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(20), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(21), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(22), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(23), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(24), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(25), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(26), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(27), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(28), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(29), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVTYPERn_EL0(30), 	HDFGRTR, PMEVTYPERn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(0), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(1), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(2), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(3), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(4), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(5), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(6), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(7), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(8), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(9), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(10), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(11), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(12), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(13), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(14), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(15), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(16), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(17), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(18), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(19), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(20), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(21), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(22), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(23), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(24), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(25), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(26), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(27), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(28), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(29), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_PMEVCNTRn_EL0(30), 	HDFGRTR, PMEVCNTRn_EL0, 1),
+	SR_FGT(SYS_OSDLR_EL1, 		HDFGRTR, OSDLR_EL1, 1),
+	SR_FGT(SYS_OSECCR_EL1, 		HDFGRTR, OSECCR_EL1, 1),
+	SR_FGT(SYS_OSLSR_EL1, 		HDFGRTR, OSLSR_EL1, 1),
+	SR_FGT(SYS_DBGPRCR_EL1, 	HDFGRTR, DBGPRCR_EL1, 1),
+	SR_FGT(SYS_DBGAUTHSTATUS_EL1, 	HDFGRTR, DBGAUTHSTATUS_EL1, 1),
+	SR_FGT(SYS_DBGCLAIMSET_EL1, 	HDFGRTR, DBGCLAIM, 1),
+	SR_FGT(SYS_DBGCLAIMCLR_EL1, 	HDFGRTR, DBGCLAIM, 1),
+	SR_FGT(SYS_MDSCR_EL1, 		HDFGRTR, MDSCR_EL1, 1),
+	/*
+	 * The trap bits capture *64* debug registers per bit, but the
+	 * ARM ARM only describes the encoding for the first 16, and
+	 * we don't really support more than that anyway.
+	 */
+	SR_FGT(SYS_DBGWVRn_EL1(0), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWVRn_EL1(1), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWVRn_EL1(2), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWVRn_EL1(3), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWVRn_EL1(4), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWVRn_EL1(5), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWVRn_EL1(6), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWVRn_EL1(7), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWVRn_EL1(8), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWVRn_EL1(9), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWVRn_EL1(10), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWVRn_EL1(11), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWVRn_EL1(12), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWVRn_EL1(13), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWVRn_EL1(14), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWVRn_EL1(15), 	HDFGRTR, DBGWVRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(0), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(1), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(2), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(3), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(4), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(5), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(6), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(7), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(8), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(9), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(10), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(11), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(12), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(13), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(14), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGWCRn_EL1(15), 	HDFGRTR, DBGWCRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(0), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(1), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(2), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(3), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(4), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(5), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(6), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(7), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(8), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(9), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(10), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(11), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(12), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(13), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(14), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBVRn_EL1(15), 	HDFGRTR, DBGBVRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(0), 	HDFGRTR, DBGBCRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(1), 	HDFGRTR, DBGBCRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(2), 	HDFGRTR, DBGBCRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(3), 	HDFGRTR, DBGBCRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(4), 	HDFGRTR, DBGBCRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(5), 	HDFGRTR, DBGBCRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(6), 	HDFGRTR, DBGBCRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(7), 	HDFGRTR, DBGBCRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(8), 	HDFGRTR, DBGBCRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(9), 	HDFGRTR, DBGBCRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(10), 	HDFGRTR, DBGBCRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(11), 	HDFGRTR, DBGBCRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(12), 	HDFGRTR, DBGBCRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(13), 	HDFGRTR, DBGBCRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(14), 	HDFGRTR, DBGBCRn_EL1, 1),
+	SR_FGT(SYS_DBGBCRn_EL1(15), 	HDFGRTR, DBGBCRn_EL1, 1),
+	/*
+	 * HDFGWTR_EL2
+	 *
+	 * Although HDFGRTR_EL2 and HDFGWTR_EL2 registers largely
+	 * overlap in their bit assignment, there are a number of bits
+	 * that are RES0 on one side, and an actual trap bit on the
+	 * other.  The policy chosen here is to describe all the
+	 * read-side mappings, and only the write-side mappings that
+	 * differ from the write side, and the trap handler will pick
+	 * the correct shadow register based on the access type.
+	 */
+	SR_FGT(SYS_TRFCR_EL1,		HDFGWTR, TRFCR_EL1, 1),
+	SR_FGT(SYS_TRCOSLAR,		HDFGWTR, TRCOSLAR, 1),
+	SR_FGT(SYS_PMCR_EL0,		HDFGWTR, PMCR_EL0, 1),
+	SR_FGT(SYS_PMSWINC_EL0,		HDFGWTR, PMSWINC_EL0, 1),
+	SR_FGT(SYS_OSLAR_EL1,		HDFGWTR, OSLAR_EL1, 1),
 };
 
 static union trap_config get_trap_config(u32 sysreg)
@@ -1213,6 +1665,14 @@ bool __check_nv_sr_forward(struct kvm_vcpu *vcpu)
 			val = sanitised_sys_reg(vcpu, HFGWTR_EL2);
 		break;
 
+	case HDFGRTR_GROUP:
+	case HDFGWTR_GROUP:
+		if (is_read)
+			val = sanitised_sys_reg(vcpu, HDFGRTR_EL2);
+		else
+			val = sanitised_sys_reg(vcpu, HDFGWTR_EL2);
+		break;
+
 	case HFGITR_GROUP:
 		val = sanitised_sys_reg(vcpu, HFGITR_EL2);
 		break;
-- 
2.34.1

[PATCH 23/27] KVM: arm64: nv: Add SVC trap forwarding

[Marc Zyngier]

HFGITR_EL2 allows the trap of SVC instructions to EL2. Allow these
traps to be forwarded. Take this opportunity to deny any 32bit activity
when NV is enabled.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/kvm/arm.c         |  4 ++++
 arch/arm64/kvm/handle_exit.c | 12 ++++++++++++
 2 files changed, 16 insertions(+)

diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index c2c14059f6a8..f876e1bbaad2 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -36,6 +36,7 @@
 #include <asm/kvm_arm.h>
 #include <asm/kvm_asm.h>
 #include <asm/kvm_mmu.h>
+#include <asm/kvm_nested.h>
 #include <asm/kvm_pkvm.h>
 #include <asm/kvm_emulate.h>
 #include <asm/sections.h>
@@ -811,6 +812,9 @@ static bool vcpu_mode_is_bad_32bit(struct kvm_vcpu *vcpu)
 	if (likely(!vcpu_mode_is_32bit(vcpu)))
 		return false;
 
+	if (vcpu_has_nv(vcpu))
+		return true;
+
 	return !kvm_supports_32bit_el0();
 }
 
diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c
index 6dcd6604b6bc..3b86d534b995 100644
--- a/arch/arm64/kvm/handle_exit.c
+++ b/arch/arm64/kvm/handle_exit.c
@@ -226,6 +226,17 @@ static int kvm_handle_eret(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+static int handle_svc(struct kvm_vcpu *vcpu)
+{
+	/*
+	 * So far, SVC traps only for NV via HFGITR_EL2. A SVC from a
+	 * 32bit guest would be caught by vpcu_mode_is_bad_32bit(), so
+	 * we should only have to deal with a 64 bit exception.
+	 */
+	kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu));
+	return 1;
+}
+
 static exit_handle_fn arm_exit_handlers[] = {
 	[0 ... ESR_ELx_EC_MAX]	= kvm_handle_unknown_ec,
 	[ESR_ELx_EC_WFx]	= kvm_handle_wfx,
@@ -239,6 +250,7 @@ static exit_handle_fn arm_exit_handlers[] = {
 	[ESR_ELx_EC_SMC32]	= handle_smc,
 	[ESR_ELx_EC_HVC64]	= handle_hvc,
 	[ESR_ELx_EC_SMC64]	= handle_smc,
+	[ESR_ELx_EC_SVC64]	= handle_svc,
 	[ESR_ELx_EC_SYS64]	= kvm_handle_sys_reg,
 	[ESR_ELx_EC_SVE]	= handle_sve,
 	[ESR_ELx_EC_ERET]	= kvm_handle_eret,
-- 
2.34.1

[PATCH 24/27] KVM: arm64: nv: Add switching support for HFGxTR/HDFGxTR

[Marc Zyngier]

Now that we can evaluate the FGT registers, allow them to be merged
with the hypervisor's own configuration (in the case of HFG{RW}TR_EL2)
or simply set for HFGITR_EL2, HDGFRTR_EL2 and HDFGWTR_EL2.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/kvm/hyp/include/hyp/switch.h | 48 +++++++++++++++++++++++++
 1 file changed, 48 insertions(+)

diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
index 9781e79a5127..5fa1ae26f075 100644
--- a/arch/arm64/kvm/hyp/include/hyp/switch.h
+++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
@@ -70,6 +70,13 @@ static inline void __activate_traps_fpsimd32(struct kvm_vcpu *vcpu)
 	}
 }
 
+#define compute_clr_set(vcpu, reg, clr, set)				\
+	do {								\
+		u64 hfg;						\
+		hfg = __vcpu_sys_reg(vcpu, reg) & ~__ ## reg ## _RES0;	\
+		set |= hfg & __ ## reg ## _MASK; 			\
+		clr |= ~hfg & __ ## reg ## _nMASK; 			\
+	} while(0)
 
 
 static inline void __activate_traps_hfgxtr(struct kvm_vcpu *vcpu)
@@ -97,6 +104,10 @@ static inline void __activate_traps_hfgxtr(struct kvm_vcpu *vcpu)
 	if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
 		w_set |= HFGxTR_EL2_TCR_EL1_MASK;
 
+	if (vcpu_has_nv(vcpu) && !is_hyp_ctxt(vcpu)) {
+		compute_clr_set(vcpu, HFGRTR_EL2, r_clr, r_set);
+		compute_clr_set(vcpu, HFGWTR_EL2, w_clr, w_set);
+	}
 
 	r_val = __HFGRTR_EL2_nMASK & ~HFGxTR_EL2_nACCDATA_EL1;
 	r_val |= r_set;
@@ -108,6 +119,38 @@ static inline void __activate_traps_hfgxtr(struct kvm_vcpu *vcpu)
 
 	write_sysreg_s(r_val, SYS_HFGRTR_EL2);
 	write_sysreg_s(w_val, SYS_HFGWTR_EL2);
+
+	if (!vcpu_has_nv(vcpu) || is_hyp_ctxt(vcpu))
+		return;
+
+	ctxt_sys_reg(hctxt, HFGITR_EL2) = read_sysreg_s(SYS_HFGITR_EL2);
+
+	r_set = r_clr = 0;
+	compute_clr_set(vcpu, HFGITR_EL2, r_clr, r_set);
+	r_val = __HFGITR_EL2_nMASK;
+	r_val |= r_set;
+	r_val &= ~r_clr;
+
+	write_sysreg_s(r_val, SYS_HFGITR_EL2);
+
+	ctxt_sys_reg(hctxt, HDFGRTR_EL2) = read_sysreg_s(SYS_HDFGRTR_EL2);
+	ctxt_sys_reg(hctxt, HDFGWTR_EL2) = read_sysreg_s(SYS_HDFGWTR_EL2);
+
+	r_clr = r_set = w_clr = w_set = 0;
+
+	compute_clr_set(vcpu, HDFGRTR_EL2, r_clr, r_set);
+	compute_clr_set(vcpu, HDFGWTR_EL2, w_clr, w_set);
+
+	r_val = __HDFGRTR_EL2_nMASK;
+	r_val |= r_set;
+	r_val &= ~r_clr;
+
+	w_val = __HDFGWTR_EL2_nMASK;
+	w_val |= w_set;
+	w_val &= ~w_clr;
+
+	write_sysreg_s(r_val, SYS_HDFGRTR_EL2);
+	write_sysreg_s(w_val, SYS_HDFGWTR_EL2);
 }
 
 static inline void __deactivate_traps_hfgxtr(struct kvm_vcpu *vcpu)
@@ -120,7 +163,12 @@ static inline void __deactivate_traps_hfgxtr(struct kvm_vcpu *vcpu)
 	write_sysreg_s(ctxt_sys_reg(hctxt, HFGRTR_EL2), SYS_HFGRTR_EL2);
 	write_sysreg_s(ctxt_sys_reg(hctxt, HFGWTR_EL2), SYS_HFGWTR_EL2);
 
+	if (!vcpu_has_nv(vcpu) || is_hyp_ctxt(vcpu))
+		return;
 
+	write_sysreg_s(ctxt_sys_reg(hctxt, HFGITR_EL2), SYS_HFGITR_EL2);
+	write_sysreg_s(ctxt_sys_reg(hctxt, HDFGRTR_EL2), SYS_HDFGRTR_EL2);
+	write_sysreg_s(ctxt_sys_reg(hctxt, HDFGWTR_EL2), SYS_HDFGWTR_EL2);
 }
 
 static inline void __activate_traps_common(struct kvm_vcpu *vcpu)
-- 
2.34.1

[PATCH 25/27] KVM: arm64: nv: Expose FGT to nested guests

[Marc Zyngier]

Now that we have FGT support, expose the feature to NV guests.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/kvm/nested.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index 7f80f385d9e8..3facd8918ae3 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -71,8 +71,9 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
 		break;
 
 	case SYS_ID_AA64MMFR0_EL1:
-		/* Hide ECV, FGT, ExS, Secure Memory */
-		val &= ~(GENMASK_ULL(63, 43)		|
+		/* Hide ECV, ExS, Secure Memory */
+		val &= ~(NV_FTR(MMFR0, ECV)		|
+			 NV_FTR(MMFR0, EXS)		|
 			 NV_FTR(MMFR0, TGRAN4_2)	|
 			 NV_FTR(MMFR0, TGRAN16_2)	|
 			 NV_FTR(MMFR0, TGRAN64_2)	|
-- 
2.34.1

[PATCH 26/27] KVM: arm64: Move HCRX_EL2 switch to load/put on VHE systems

[Marc Zyngier]

Although the nVHE behaviour requires HCRX_EL2 to be switched
on each switch between host and guest, there is nothing in
this register that would affect a VHE host.

It is thus possible to save/restore this register on load/put
on VHE systems, avoiding unnecessary sysreg access on the hot
path. Additionally, it avoids unnecessary traps when running
with NV.

To achieve this, simply move the read/writes to the *_common()
helpers, which are called on load/put on VHE, and more eagerly
on nVHE.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/kvm/hyp/include/hyp/switch.h | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
index 5fa1ae26f075..f07523d7d4ec 100644
--- a/arch/arm64/kvm/hyp/include/hyp/switch.h
+++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
@@ -196,6 +196,9 @@ static inline void __activate_traps_common(struct kvm_vcpu *vcpu)
 	vcpu->arch.mdcr_el2_host = read_sysreg(mdcr_el2);
 	write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
 
+	if (cpus_have_final_cap(ARM64_HAS_HCX))
+		write_sysreg_s(HCRX_GUEST_FLAGS, SYS_HCRX_EL2);
+
 	__activate_traps_hfgxtr(vcpu);
 }
 
@@ -212,6 +215,9 @@ static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu)
 		vcpu_clear_flag(vcpu, PMUSERENR_ON_CPU);
 	}
 
+	if (cpus_have_final_cap(ARM64_HAS_HCX))
+		write_sysreg_s(HCRX_HOST_FLAGS, SYS_HCRX_EL2);
+
 	__deactivate_traps_hfgxtr(vcpu);
 }
 
@@ -226,9 +232,6 @@ static inline void ___activate_traps(struct kvm_vcpu *vcpu)
 
 	if (cpus_have_final_cap(ARM64_HAS_RAS_EXTN) && (hcr & HCR_VSE))
 		write_sysreg_s(vcpu->arch.vsesr_el2, SYS_VSESR_EL2);
-
-	if (cpus_have_final_cap(ARM64_HAS_HCX))
-		write_sysreg_s(HCRX_GUEST_FLAGS, SYS_HCRX_EL2);
 }
 
 static inline void ___deactivate_traps(struct kvm_vcpu *vcpu)
@@ -243,9 +246,6 @@ static inline void ___deactivate_traps(struct kvm_vcpu *vcpu)
 		vcpu->arch.hcr_el2 &= ~HCR_VSE;
 		vcpu->arch.hcr_el2 |= read_sysreg(hcr_el2) & HCR_VSE;
 	}
-
-	if (cpus_have_final_cap(ARM64_HAS_HCX))
-		write_sysreg_s(HCRX_HOST_FLAGS, SYS_HCRX_EL2);
 }
 
 static inline bool __populate_fault_info(struct kvm_vcpu *vcpu)
-- 
2.34.1

[PATCH 27/27] KVM: arm64: nv: Add support for HCRX_EL2

[Marc Zyngier]

HCRX_EL2 has an interesting effect on HFGITR_EL2, as it conditions
the traps of TLBI*nXS.

Expand the FGT support to add a new Fine Grained Filter that will
get checked when the instruction gets trapped, allowing the shadow
register to override the trap as needed.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/kvm_arm.h        |  5 ++
 arch/arm64/include/asm/kvm_host.h       |  1 +
 arch/arm64/kvm/emulate-nested.c         | 89 +++++++++++++++----------
 arch/arm64/kvm/hyp/include/hyp/switch.h | 15 ++++-
 arch/arm64/kvm/nested.c                 |  3 +-
 arch/arm64/kvm/sys_regs.c               |  2 +
 6 files changed, 78 insertions(+), 37 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index d229f238c3b6..137f732789c9 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -369,6 +369,11 @@
 #define __HDFGWTR_EL2_MASK	~__HDFGWTR_EL2_nMASK
 #define __HDFGWTR_EL2_nMASK	GENMASK(62, 60)
 
+/* Similar definitions for HCRX_EL2 */
+#define __HCRX_EL2_RES0		(GENMASK(63, 16) | GENMASK(13, 12))
+#define __HCRX_EL2_MASK		(0)
+#define __HCRX_EL2_nMASK	(GENMASK(15, 14) | GENMASK(4, 0))
+
 /* Hyp Prefetch Fault Address Register (HPFAR/HDFAR) */
 #define HPFAR_MASK	(~UL(0xf))
 /*
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index e498cca78a8d..c446f7527f5e 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -380,6 +380,7 @@ enum vcpu_sysreg {
 	CPTR_EL2,	/* Architectural Feature Trap Register (EL2) */
 	HSTR_EL2,	/* Hypervisor System Trap Register */
 	HACR_EL2,	/* Hypervisor Auxiliary Control Register */
+	HCRX_EL2,	/* Extended Hypervisor Configuration Register */
 	TTBR0_EL2,	/* Translation Table Base Register 0 (EL2) */
 	TTBR1_EL2,	/* Translation Table Base Register 1 (EL2) */
 	TCR_EL2,	/* Translation Control Register (EL2) */
diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
index 739b35e6ef18..ee7f3de6e7c2 100644
--- a/arch/arm64/kvm/emulate-nested.c
+++ b/arch/arm64/kvm/emulate-nested.c
@@ -401,7 +401,8 @@ static const complex_condition_check ccc[] = {
  * [13:10]	enum fgt_group_id (4 bits)
  * [19:14]	bit number in the FGT register (6 bits)
  * [20]		trap polarity (1 bit)
- * [62:21]	Unused (42 bits)
+ * [25:21]	FG filter (5 bits)
+ * [62:26]	Unused (37 bits)
  * [63]		RES0 - Must be zero, as lost on insertion in the xarray
  */
 union trap_config {
@@ -411,7 +412,8 @@ union trap_config {
 		unsigned long 	fgt:4;	/* Fing Grained Trap id */
 		unsigned long	bit:6;	/* Bit number */
 		unsigned long	pol:1;	/* Polarity */
-		unsigned long	unk:42;	/* Unknown */
+		unsigned long	fgf:5;	/* Fine Grained Filter */
+		unsigned long	unk:37;	/* Unknown */
 		unsigned long	mbz:1;	/* Must Be Zero */
 	};
 };
@@ -902,7 +904,12 @@ enum fgt_group_id {
 	HFGITR_GROUP,
 };
 
-#define SR_FGT(sr, g, b, p)					\
+enum fg_filter_id {
+	__NO_FGF__,
+	HCRX_FGTnXS,
+};
+
+#define SR_FGF(sr, g, b, p, f)					\
 	{							\
 		.encoding	= sr,				\
 		.end		= sr,				\
@@ -910,9 +917,12 @@ enum fgt_group_id {
 			.fgt = g ## _GROUP,			\
 			.bit = g ## _EL2_ ## b ## _SHIFT,	\
 			.pol = p,				\
+			.fgf = f,				\
 		},						\
 	}
 
+#define SR_FGT(sr, g, b, p)	SR_FGF(sr, g, b, p, __NO_FGF__)
+
 static const struct encoding_to_trap_config encoding_to_fgt[] __initdata = {
 	/* HFGTR_EL2, HFGWTR_EL2 */
 	SR_FGT(SYS_TPIDR2_EL0,		HFGxTR, nTPIDR2_EL0, 0),
@@ -1016,37 +1026,37 @@ static const struct encoding_to_trap_config encoding_to_fgt[] __initdata = {
 	SR_FGT(OP_TLBI_ASIDE1OS, 	HFGITR, TLBIASIDE1OS, 1),
 	SR_FGT(OP_TLBI_VAE1OS, 		HFGITR, TLBIVAE1OS, 1),
 	SR_FGT(OP_TLBI_VMALLE1OS, 	HFGITR, TLBIVMALLE1OS, 1),
-	/* FIXME: nXS variants must be checked against HCRX_EL2.FGTnXS */
-	SR_FGT(OP_TLBI_VAALE1NXS, 	HFGITR, TLBIVAALE1, 1),
-	SR_FGT(OP_TLBI_VALE1NXS, 	HFGITR, TLBIVALE1, 1),
-	SR_FGT(OP_TLBI_VAAE1NXS, 	HFGITR, TLBIVAAE1, 1),
-	SR_FGT(OP_TLBI_ASIDE1NXS, 	HFGITR, TLBIASIDE1, 1),
-	SR_FGT(OP_TLBI_VAE1NXS, 	HFGITR, TLBIVAE1, 1),
-	SR_FGT(OP_TLBI_VMALLE1NXS, 	HFGITR, TLBIVMALLE1, 1),
-	SR_FGT(OP_TLBI_RVAALE1NXS, 	HFGITR, TLBIRVAALE1, 1),
-	SR_FGT(OP_TLBI_RVALE1NXS, 	HFGITR, TLBIRVALE1, 1),
-	SR_FGT(OP_TLBI_RVAAE1NXS, 	HFGITR, TLBIRVAAE1, 1),
-	SR_FGT(OP_TLBI_RVAE1NXS, 	HFGITR, TLBIRVAE1, 1),
-	SR_FGT(OP_TLBI_RVAALE1ISNXS, 	HFGITR, TLBIRVAALE1IS, 1),
-	SR_FGT(OP_TLBI_RVALE1ISNXS, 	HFGITR, TLBIRVALE1IS, 1),
-	SR_FGT(OP_TLBI_RVAAE1ISNXS, 	HFGITR, TLBIRVAAE1IS, 1),
-	SR_FGT(OP_TLBI_RVAE1ISNXS, 	HFGITR, TLBIRVAE1IS, 1),
-	SR_FGT(OP_TLBI_VAALE1ISNXS, 	HFGITR, TLBIVAALE1IS, 1),
-	SR_FGT(OP_TLBI_VALE1ISNXS, 	HFGITR, TLBIVALE1IS, 1),
-	SR_FGT(OP_TLBI_VAAE1ISNXS, 	HFGITR, TLBIVAAE1IS, 1),
-	SR_FGT(OP_TLBI_ASIDE1ISNXS, 	HFGITR, TLBIASIDE1IS, 1),
-	SR_FGT(OP_TLBI_VAE1ISNXS, 	HFGITR, TLBIVAE1IS, 1),
-	SR_FGT(OP_TLBI_VMALLE1ISNXS, 	HFGITR, TLBIVMALLE1IS, 1),
-	SR_FGT(OP_TLBI_RVAALE1OSNXS, 	HFGITR, TLBIRVAALE1OS, 1),
-	SR_FGT(OP_TLBI_RVALE1OSNXS, 	HFGITR, TLBIRVALE1OS, 1),
-	SR_FGT(OP_TLBI_RVAAE1OSNXS, 	HFGITR, TLBIRVAAE1OS, 1),
-	SR_FGT(OP_TLBI_RVAE1OSNXS, 	HFGITR, TLBIRVAE1OS, 1),
-	SR_FGT(OP_TLBI_VAALE1OSNXS, 	HFGITR, TLBIVAALE1OS, 1),
-	SR_FGT(OP_TLBI_VALE1OSNXS, 	HFGITR, TLBIVALE1OS, 1),
-	SR_FGT(OP_TLBI_VAAE1OSNXS, 	HFGITR, TLBIVAAE1OS, 1),
-	SR_FGT(OP_TLBI_ASIDE1OSNXS, 	HFGITR, TLBIASIDE1OS, 1),
-	SR_FGT(OP_TLBI_VAE1OSNXS, 	HFGITR, TLBIVAE1OS, 1),
-	SR_FGT(OP_TLBI_VMALLE1OSNXS, 	HFGITR, TLBIVMALLE1OS, 1),
+	/* nXS variants must be checked against HCRX_EL2.FGTnXS */
+	SR_FGF(OP_TLBI_VAALE1NXS, 	HFGITR, TLBIVAALE1, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_VALE1NXS, 	HFGITR, TLBIVALE1, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_VAAE1NXS, 	HFGITR, TLBIVAAE1, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_ASIDE1NXS, 	HFGITR, TLBIASIDE1, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_VAE1NXS, 	HFGITR, TLBIVAE1, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_VMALLE1NXS, 	HFGITR, TLBIVMALLE1, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_RVAALE1NXS, 	HFGITR, TLBIRVAALE1, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_RVALE1NXS, 	HFGITR, TLBIRVALE1, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_RVAAE1NXS, 	HFGITR, TLBIRVAAE1, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_RVAE1NXS, 	HFGITR, TLBIRVAE1, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_RVAALE1ISNXS, 	HFGITR, TLBIRVAALE1IS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_RVALE1ISNXS, 	HFGITR, TLBIRVALE1IS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_RVAAE1ISNXS, 	HFGITR, TLBIRVAAE1IS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_RVAE1ISNXS, 	HFGITR, TLBIRVAE1IS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_VAALE1ISNXS, 	HFGITR, TLBIVAALE1IS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_VALE1ISNXS, 	HFGITR, TLBIVALE1IS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_VAAE1ISNXS, 	HFGITR, TLBIVAAE1IS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_ASIDE1ISNXS, 	HFGITR, TLBIASIDE1IS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_VAE1ISNXS, 	HFGITR, TLBIVAE1IS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_VMALLE1ISNXS, 	HFGITR, TLBIVMALLE1IS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_RVAALE1OSNXS, 	HFGITR, TLBIRVAALE1OS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_RVALE1OSNXS, 	HFGITR, TLBIRVALE1OS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_RVAAE1OSNXS, 	HFGITR, TLBIRVAAE1OS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_RVAE1OSNXS, 	HFGITR, TLBIRVAE1OS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_VAALE1OSNXS, 	HFGITR, TLBIVAALE1OS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_VALE1OSNXS, 	HFGITR, TLBIVALE1OS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_VAAE1OSNXS, 	HFGITR, TLBIVAAE1OS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_ASIDE1OSNXS, 	HFGITR, TLBIASIDE1OS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_VAE1OSNXS, 	HFGITR, TLBIVAE1OS, 1, HCRX_FGTnXS),
+	SR_FGF(OP_TLBI_VMALLE1OSNXS, 	HFGITR, TLBIVMALLE1OS, 1, HCRX_FGTnXS),
 	SR_FGT(OP_AT_S1E1WP, 		HFGITR, ATS1E1WP, 1),
 	SR_FGT(OP_AT_S1E1RP, 		HFGITR, ATS1E1RP, 1),
 	SR_FGT(OP_AT_S1E0W, 		HFGITR, ATS1E0W, 1),
@@ -1675,6 +1685,17 @@ bool __check_nv_sr_forward(struct kvm_vcpu *vcpu)
 
 	case HFGITR_GROUP:
 		val = sanitised_sys_reg(vcpu, HFGITR_EL2);
+		switch (tc.fgf) {
+			u64 tmp;
+
+		case __NO_FGF__:
+			break;
+
+		case HCRX_FGTnXS:
+			tmp = sanitised_sys_reg(vcpu, HCRX_EL2);
+			if (tmp & HCRX_EL2_FGTnXS)
+				tc.fgt = __NO_FGT_GROUP__;
+		}
 		break;
 	}
 
diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
index f07523d7d4ec..39d79bbbfe73 100644
--- a/arch/arm64/kvm/hyp/include/hyp/switch.h
+++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
@@ -196,8 +196,19 @@ static inline void __activate_traps_common(struct kvm_vcpu *vcpu)
 	vcpu->arch.mdcr_el2_host = read_sysreg(mdcr_el2);
 	write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
 
-	if (cpus_have_final_cap(ARM64_HAS_HCX))
-		write_sysreg_s(HCRX_GUEST_FLAGS, SYS_HCRX_EL2);
+	if (cpus_have_final_cap(ARM64_HAS_HCX)) {
+		u64 hcrx = HCRX_GUEST_FLAGS;
+		if (vcpu_has_nv(vcpu) && !is_hyp_ctxt(vcpu)) {
+			u64 clr = 0, set = 0;
+
+			compute_clr_set(vcpu, HCRX_EL2, clr, set);
+
+			hcrx |= set;
+			hcrx &= ~clr;
+		}
+
+		write_sysreg_s(hcrx, SYS_HCRX_EL2);
+	}
 
 	__activate_traps_hfgxtr(vcpu);
 }
diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
index 3facd8918ae3..042695a210ce 100644
--- a/arch/arm64/kvm/nested.c
+++ b/arch/arm64/kvm/nested.c
@@ -117,7 +117,8 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
 		break;
 
 	case SYS_ID_AA64MMFR1_EL1:
-		val &= (NV_FTR(MMFR1, PAN)	|
+		val &= (NV_FTR(MMFR1, HCX)	|
+			NV_FTR(MMFR1, PAN)	|
 			NV_FTR(MMFR1, LO)	|
 			NV_FTR(MMFR1, HPDS)	|
 			NV_FTR(MMFR1, VH)	|
diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
index cc10ecaff98a..e79a9b821e20 100644
--- a/arch/arm64/kvm/sys_regs.c
+++ b/arch/arm64/kvm/sys_regs.c
@@ -2372,6 +2372,8 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	EL2_REG(HFGITR_EL2, access_rw, reset_val, 0),
 	EL2_REG(HACR_EL2, access_rw, reset_val, 0),
 
+	EL2_REG(HCRX_EL2, access_rw, reset_val, 0),
+
 	EL2_REG(TTBR0_EL2, access_rw, reset_val, 0),
 	EL2_REG(TTBR1_EL2, access_rw, reset_val, 0),
 	EL2_REG(TCR_EL2, access_rw, reset_val, TCR_EL2_RES1),
-- 
2.34.1

Re: [PATCH 00/27] KVM: arm64: NV trap forwarding infrastructure

[Eric Auger]

Hi Marc,

On 7/12/23 16:57, Marc Zyngier wrote:
> As people are getting tired of seeing the full NV series, I've
> extracted some of the easy stuff which I'm targeting for 6.6.
>
> This implements the so called "trap forwarding" infrastructure, which
> gets used when we take a trap from an L2 guest and that the L1 guest
> wants to see the trap for itself.
>
> Most of the series is pretty boring stuff, mostly a long list of
> encodings which are mapped to a set of trap bits. I swear they are
> correct. Sort of.
>
> The interesting bit is around how we compute the trap result, which is
> pretty complex due to the layers of crap the architecture has piled
> over the years (a single op can be trapped by multiple coarse grained
> trap bits, or a fine grained trap bit, which may itself be conditioned
> by another control bit -- madness).
>
> This also results in some rework of both the FGT stuff (for which I
> carry a patch from Mark) and newly introduced the HCRX support.
>
> With that (and the rest of the NV series[1]), FGT gets exposed to guests
> and the trapping seems to work as expected.
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/nv-6.6-WIP

I have not received patches 10-27 and I do not see them on lore archive
https://lore.kernel.org/all/20230712145810.3864793-1-maz@kernel.org/#r

Thanks

Eric
>
> Marc Zyngier (26):
>   arm64: Add missing VA CMO encodings
>   arm64: Add missing ERX*_EL1 encodings
>   arm64: Add missing DC ZVA/GVA/GZVA encodings
>   arm64: Add TLBI operation encodings
>   arm64: Add AT operation encodings
>   arm64: Add debug registers affected by HDFGxTR_EL2
>   arm64: Add missing BRB/CFP/DVP/CPP instructions
>   arm64: Fix HFGxTR_EL2 field naming
>   arm64: Add HDFGRTR_EL2 and HDFGWTR_EL2 layouts
>   KVM: arm64: Correctly handle ACCDATA_EL1 traps
>   KVM: arm64: Add missing HCR_EL2 trap bits
>   KVM: arm64: nv: Add FGT registers
>   KVM: arm64: Restructure FGT register switching
>   KVM: arm64: nv: Add trap forwarding infrastructure
>   KVM: arm64: nv: Add trap forwarding for HCR_EL2
>   KVM: arm64: nv: Expose FEAT_EVT to nested guests
>   KVM: arm64: nv: Add trap forwarding for MDCR_EL2
>   KVM: arm64: nv: Add trap forwarding for CNTHCTL_EL2
>   KVM: arm64: nv: Add trap forwarding for HFGxTR_EL2
>   KVM: arm64: nv: Add trap forwarding for HFGITR_EL2
>   KVM: arm64: nv: Add trap forwarding for HDFGxTR_EL2
>   KVM: arm64: nv: Add SVC trap forwarding
>   KVM: arm64: nv: Add switching support for HFGxTR/HDFGxTR
>   KVM: arm64: nv: Expose FGT to nested guests
>   KVM: arm64: Move HCRX_EL2 switch to load/put on VHE systems
>   KVM: arm64: nv: Add support for HCRX_EL2
>
> Mark Brown (1):
>   arm64: Add feature detection for fine grained traps
>
>  arch/arm64/include/asm/kvm_arm.h        |   50 +
>  arch/arm64/include/asm/kvm_host.h       |    7 +
>  arch/arm64/include/asm/kvm_nested.h     |    2 +
>  arch/arm64/include/asm/sysreg.h         |  270 +++-
>  arch/arm64/kernel/cpufeature.c          |   11 +
>  arch/arm64/kvm/arm.c                    |    4 +
>  arch/arm64/kvm/emulate-nested.c         | 1703 +++++++++++++++++++++++
>  arch/arm64/kvm/handle_exit.c            |   12 +
>  arch/arm64/kvm/hyp/include/hyp/switch.h |  126 +-
>  arch/arm64/kvm/nested.c                 |   11 +-
>  arch/arm64/kvm/sys_regs.c               |   15 +
>  arch/arm64/kvm/trace_arm.h              |   19 +
>  arch/arm64/tools/cpucaps                |    1 +
>  arch/arm64/tools/sysreg                 |  141 +-
>  14 files changed, 2326 insertions(+), 46 deletions(-)
>

Re: [PATCH 00/27] KVM: arm64: NV trap forwarding infrastructure

[Eric Auger]

Hi,

On 7/12/23 17:16, Eric Auger wrote:
> Hi Marc,
> 
> On 7/12/23 16:57, Marc Zyngier wrote:
>> As people are getting tired of seeing the full NV series, I've
>> extracted some of the easy stuff which I'm targeting for 6.6.
>>
>> This implements the so called "trap forwarding" infrastructure, which
>> gets used when we take a trap from an L2 guest and that the L1 guest
>> wants to see the trap for itself.
>>
>> Most of the series is pretty boring stuff, mostly a long list of
>> encodings which are mapped to a set of trap bits. I swear they are
>> correct. Sort of.
>>
>> The interesting bit is around how we compute the trap result, which is
>> pretty complex due to the layers of crap the architecture has piled
>> over the years (a single op can be trapped by multiple coarse grained
>> trap bits, or a fine grained trap bit, which may itself be conditioned
>> by another control bit -- madness).
>>
>> This also results in some rework of both the FGT stuff (for which I
>> carry a patch from Mark) and newly introduced the HCRX support.
>>
>> With that (and the rest of the NV series[1]), FGT gets exposed to guests
>> and the trapping seems to work as expected.
>>
>> [1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/nv-6.6-WIP
> 
> I have not received patches 10-27 and I do not see them on lore archive
> https://lore.kernel.org/all/20230712145810.3864793-1-maz@kernel.org/#r

Hum I was too eager to see them. Now I have received everything and they
are all visible on lore.

Sorry for the noise

Eric
> 
> Thanks
> 
> Eric
>>
>> Marc Zyngier (26):
>>   arm64: Add missing VA CMO encodings
>>   arm64: Add missing ERX*_EL1 encodings
>>   arm64: Add missing DC ZVA/GVA/GZVA encodings
>>   arm64: Add TLBI operation encodings
>>   arm64: Add AT operation encodings
>>   arm64: Add debug registers affected by HDFGxTR_EL2
>>   arm64: Add missing BRB/CFP/DVP/CPP instructions
>>   arm64: Fix HFGxTR_EL2 field naming
>>   arm64: Add HDFGRTR_EL2 and HDFGWTR_EL2 layouts
>>   KVM: arm64: Correctly handle ACCDATA_EL1 traps
>>   KVM: arm64: Add missing HCR_EL2 trap bits
>>   KVM: arm64: nv: Add FGT registers
>>   KVM: arm64: Restructure FGT register switching
>>   KVM: arm64: nv: Add trap forwarding infrastructure
>>   KVM: arm64: nv: Add trap forwarding for HCR_EL2
>>   KVM: arm64: nv: Expose FEAT_EVT to nested guests
>>   KVM: arm64: nv: Add trap forwarding for MDCR_EL2
>>   KVM: arm64: nv: Add trap forwarding for CNTHCTL_EL2
>>   KVM: arm64: nv: Add trap forwarding for HFGxTR_EL2
>>   KVM: arm64: nv: Add trap forwarding for HFGITR_EL2
>>   KVM: arm64: nv: Add trap forwarding for HDFGxTR_EL2
>>   KVM: arm64: nv: Add SVC trap forwarding
>>   KVM: arm64: nv: Add switching support for HFGxTR/HDFGxTR
>>   KVM: arm64: nv: Expose FGT to nested guests
>>   KVM: arm64: Move HCRX_EL2 switch to load/put on VHE systems
>>   KVM: arm64: nv: Add support for HCRX_EL2
>>
>> Mark Brown (1):
>>   arm64: Add feature detection for fine grained traps
>>
>>  arch/arm64/include/asm/kvm_arm.h        |   50 +
>>  arch/arm64/include/asm/kvm_host.h       |    7 +
>>  arch/arm64/include/asm/kvm_nested.h     |    2 +
>>  arch/arm64/include/asm/sysreg.h         |  270 +++-
>>  arch/arm64/kernel/cpufeature.c          |   11 +
>>  arch/arm64/kvm/arm.c                    |    4 +
>>  arch/arm64/kvm/emulate-nested.c         | 1703 +++++++++++++++++++++++
>>  arch/arm64/kvm/handle_exit.c            |   12 +
>>  arch/arm64/kvm/hyp/include/hyp/switch.h |  126 +-
>>  arch/arm64/kvm/nested.c                 |   11 +-
>>  arch/arm64/kvm/sys_regs.c               |   15 +
>>  arch/arm64/kvm/trace_arm.h              |   19 +
>>  arch/arm64/tools/cpucaps                |    1 +
>>  arch/arm64/tools/sysreg                 |  141 +-
>>  14 files changed, 2326 insertions(+), 46 deletions(-)
>>
> 

Re: [PATCH 00/27] KVM: arm64: NV trap forwarding infrastructure

[Marc Zyngier]

On Wed, 12 Jul 2023 16:16:59 +0100,
Eric Auger <eric.auger@redhat.com> wrote:
> 
> Hi Marc,
> 
> On 7/12/23 16:57, Marc Zyngier wrote:
> > As people are getting tired of seeing the full NV series, I've
> > extracted some of the easy stuff which I'm targeting for 6.6.
> >
> > This implements the so called "trap forwarding" infrastructure, which
> > gets used when we take a trap from an L2 guest and that the L1 guest
> > wants to see the trap for itself.
> >
> > Most of the series is pretty boring stuff, mostly a long list of
> > encodings which are mapped to a set of trap bits. I swear they are
> > correct. Sort of.
> >
> > The interesting bit is around how we compute the trap result, which is
> > pretty complex due to the layers of crap the architecture has piled
> > over the years (a single op can be trapped by multiple coarse grained
> > trap bits, or a fine grained trap bit, which may itself be conditioned
> > by another control bit -- madness).
> >
> > This also results in some rework of both the FGT stuff (for which I
> > carry a patch from Mark) and newly introduced the HCRX support.
> >
> > With that (and the rest of the NV series[1]), FGT gets exposed to guests
> > and the trapping seems to work as expected.
> >
> > [1] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=kvm-arm64/nv-6.6-WIP
> 
> I have not received patches 10-27 and I do not see them on lore archive
> https://lore.kernel.org/all/20230712145810.3864793-1-maz@kernel.org/#r

You should see them shortly. My server throttles outgoing bursts of
more than 10 emails, which is why the rest still in limbo. It will
eventually time out and spam you for good! ;p

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 09/27] arm64: Add HDFGRTR_EL2 and HDFGWTR_EL2 layouts

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 404 bytes --]

On Wed, Jul 12, 2023 at 03:57:52PM +0100, Marc Zyngier wrote:

> As we're about to implement full support for FEAT_FGT, add the
> full HDFGRTR_EL2 and HDFGWTR_EL2 layouts.

Checked against DDI0601 2023-06:

Reviewed-by: Mark Brown <broonie@kernel.org>

FWIW we now also have HDFGTxTR2_EL2, those can be added incrementally
of course (and generally the fewer registers in an individual patch the
better).

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 14/27] KVM: arm64: Restructure FGT register switching

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 963 bytes --]

On Wed, Jul 12, 2023 at 03:57:57PM +0100, Marc Zyngier wrote:

> As we're about to majorly extend the handling of FGT registers,
> restructure the code to actually save/restore the registers
> as required. This is made easy thanks to the previous addition
> of the EL2 registers, allowing us to use the host context for
> this purpose.

> +/*
> + * FGT register definitions
> + *
> + * RES0 and polarity masks as of DDI0487J.a, to be updated as needed.
> + * We're not using the generated masks as they are usually ahead of
> + * the published ARM ARM, which we use as a reference.
> + *
> + * Once we get to a point where the two describe the same thing, we'll
> + * merge the definitions. One day.
> + */

What's the issue here?  The generated definitions should be aligned with
what's published in DDI0601.  That AIUI exists in large part due to
concerns people were having with the amount of time it can take to fold
new features into the ARM, it's official.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 14/27] KVM: arm64: Restructure FGT register switching

[Marc Zyngier]

On Wed, 12 Jul 2023 18:15:41 +0100,
Mark Brown <broonie@kernel.org> wrote:
> 
> On Wed, Jul 12, 2023 at 03:57:57PM +0100, Marc Zyngier wrote:
>
> > As we're about to majorly extend the handling of FGT registers,
> > restructure the code to actually save/restore the registers
> > as required. This is made easy thanks to the previous addition
> > of the EL2 registers, allowing us to use the host context for
> > this purpose.
> 
> > +/*
> > + * FGT register definitions
> > + *
> > + * RES0 and polarity masks as of DDI0487J.a, to be updated as needed.
> > + * We're not using the generated masks as they are usually ahead of
> > + * the published ARM ARM, which we use as a reference.
> > + *
> > + * Once we get to a point where the two describe the same thing, we'll
> > + * merge the definitions. One day.
> > + */
> 
> What's the issue here?  The generated definitions should be aligned with
> what's published in DDI0601.  That AIUI exists in large part due to
> concerns people were having with the amount of time it can take to fold
> new features into the ARM, it's official.

For multiple reasons:

- What's published as DDI0601 is a list of registers, without any
  context and no relation to the wider architecture (it is basically
  the XML dumped as a PDF). That's not enough to implement the
  architecture as it is missing all the content of the engineering
  specs, which are not public documents.

- I have no motivation in supporting the latest and greatest. NV is
  hard enough without all the (still evolving) crop of 8.9/9.4
  extensions.  As long as what I have is a legal implementation and
  runs on the HW I have access to, that's good enough for me.

- I want to look at a single document and support what's in there. Not
  two. Because it is hard enough to follow when you're implementing
  this crap, and even harder for someone trying to review it.

So I firmly intend to totally ignore most of what's outside of the
published ARM ARM unless it makes my life so much easier that I can't
afford not to implement it.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 14/27] KVM: arm64: Restructure FGT register switching

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 2429 bytes --]

On Wed, Jul 12, 2023 at 09:06:08PM +0100, Marc Zyngier wrote:
> Mark Brown <broonie@kernel.org> wrote:

> > > + * RES0 and polarity masks as of DDI0487J.a, to be updated as needed.
> > > + * We're not using the generated masks as they are usually ahead of
> > > + * the published ARM ARM, which we use as a reference.

> > What's the issue here?  The generated definitions should be aligned with
> > what's published in DDI0601.  That AIUI exists in large part due to
> > concerns people were having with the amount of time it can take to fold
> > new features into the ARM, it's official.

> For multiple reasons:

> - What's published as DDI0601 is a list of registers, without any
>   context and no relation to the wider architecture (it is basically
>   the XML dumped as a PDF). That's not enough to implement the
>   architecture as it is missing all the content of the engineering
>   specs, which are not public documents.

Right, it's not the full spec - I was just thinking it was enough to
cover the use here with finding RES0 bits.  The actual XML is
downloadable as well, via 

    https://developer.arm.com/downloads/-/exploration-tools

if that's more convenient (I am not sure why that's not available if you
go looking for DDI0601), not that that addresses the issue with not
having the non-XML part of things.  I know the people responsible for
producing the ARM are actively working on improving the production
process to address the lag so the ARM is available much more promptly.

> - I have no motivation in supporting the latest and greatest. NV is
>   hard enough without all the (still evolving) crop of 8.9/9.4
>   extensions.  As long as what I have is a legal implementation and
>   runs on the HW I have access to, that's good enough for me.

> - I want to look at a single document and support what's in there. Not
>   two. Because it is hard enough to follow when you're implementing
>   this crap, and even harder for someone trying to review it.

> So I firmly intend to totally ignore most of what's outside of the
> published ARM ARM unless it makes my life so much easier that I can't
> afford not to implement it.

That's definitely fair, my concern here is the risk that we might end up
with issues due to the manual definitions drifting from the generated
ones without people noticing as things go forwards.  Hopefully that's a
minor risk.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 16/27] KVM: arm64: nv: Add trap forwarding for HCR_EL2

[Eric Auger]

Hi Marc,

On 7/12/23 16:57, Marc Zyngier wrote:
> Describe the HCR_EL2 register, and associate it with all the sysregs
> it allows to trap.
>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  arch/arm64/kvm/emulate-nested.c | 475 ++++++++++++++++++++++++++++++++
>  1 file changed, 475 insertions(+)
>
> diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
> index 5bab2e85d70c..51901e85e43d 100644
> --- a/arch/arm64/kvm/emulate-nested.c
> +++ b/arch/arm64/kvm/emulate-nested.c
> @@ -37,12 +37,47 @@ enum coarse_grain_trap_id {
>  	 * on their own instead of being part of a combination of
>  	 * trap controls.
>  	 */
> +	CGT_HCR_TID1,
> +	CGT_HCR_TID2,
> +	CGT_HCR_TID3,
> +	CGT_HCR_IMO,
> +	CGT_HCR_FMO,
> +	CGT_HCR_TIDCP,
> +	CGT_HCR_TACR,
> +	CGT_HCR_TSW,
> +	CGT_HCR_TPC,
> +	CGT_HCR_TPU,
> +	CGT_HCR_TTLB,
> +	CGT_HCR_TVM,
> +	CGT_HCR_TDZ,
> +	CGT_HCR_TRVM,
> +	CGT_HCR_TLOR,
> +	CGT_HCR_TERR,
> +	CGT_HCR_APK,
> +	CGT_HCR_NV,
> +	CGT_HCR_NV1,
> +	CGT_HCR_AT,
> +	CGT_HCR_FIEN,
> +	CGT_HCR_TID4,
> +	CGT_HCR_TICAB,
> +	CGT_HCR_TOCU,
> +	CGT_HCR_ENSCXT,
> +	CGT_HCR_TTLBIS,
> +	CGT_HCR_TTLBOS,
>  
>  	/*
>  	 * Anything after this point is a combination of trap controls,
>  	 * which all must be evaluated to decide what to do.
>  	 */
>  	__MULTIPLE_CONTROL_BITS__,
> +	CGT_HCR_IMO_FMO = __MULTIPLE_CONTROL_BITS__,
> +	CGT_HCR_TID2_TID4,
> +	CGT_HCR_TTLB_TTLBIS,
> +	CGT_HCR_TTLB_TTLBOS,
> +	CGT_HCR_TVM_TRVM,
> +	CGT_HCR_TPU_TICAB,
> +	CGT_HCR_TPU_TOCU,
> +	CGT_HCR_NV1_ENSCXT,
>  
>  	/*
>  	 * Anything after this point requires a callback evaluating a
> @@ -52,6 +87,168 @@ enum coarse_grain_trap_id {
>  };
>  
>  static const struct trap_bits coarse_trap_bits[] = {
> +	[CGT_HCR_TID1] = {
> +		.index		= HCR_EL2,
> +		.value 		= HCR_TID1,
> +		.mask		= HCR_TID1,
> +		.behaviour	= BEHAVE_FORWARD_READ,
> +	},
> +	[CGT_HCR_TID2] = {
> +		.index		= HCR_EL2,
> +		.value 		= HCR_TID2,
> +		.mask		= HCR_TID2,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_TID3] = {
> +		.index		= HCR_EL2,
> +		.value 		= HCR_TID3,
> +		.mask		= HCR_TID3,
> +		.behaviour	= BEHAVE_FORWARD_READ,
> +	},
> +	[CGT_HCR_IMO] = {
> +		.index		= HCR_EL2,
> +		.value 		= HCR_IMO,
> +		.mask		= HCR_IMO,
> +		.behaviour	= BEHAVE_FORWARD_WRITE,
> +	},
> +	[CGT_HCR_FMO] = {
> +		.index		= HCR_EL2,
> +		.value 		= HCR_FMO,
> +		.mask		= HCR_FMO,
> +		.behaviour	= BEHAVE_FORWARD_WRITE,
> +	},
> +	[CGT_HCR_TIDCP] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_TIDCP,
> +		.mask		= HCR_TIDCP,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_TACR] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_TACR,
> +		.mask		= HCR_TACR,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_TSW] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_TSW,
> +		.mask		= HCR_TSW,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_TPC] = {
modern revisions now refer to TPCP, maybe worth a comment?
> +		.index		= HCR_EL2,
> +		.value		= HCR_TPC,
> +		.mask		= HCR_TPC,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_TPU] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_TPU,
> +		.mask		= HCR_TPU,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_TTLB] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_TTLB,
> +		.mask		= HCR_TTLB,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_TVM] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_TVM,
> +		.mask		= HCR_TVM,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_TDZ] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_TDZ,
> +		.mask		= HCR_TDZ,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_TRVM] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_TRVM,
> +		.mask		= HCR_TRVM,
> +		.behaviour	= BEHAVE_FORWARD_READ,
> +	},
> +	[CGT_HCR_TLOR] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_TLOR,
> +		.mask		= HCR_TLOR,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_TERR] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_TERR,
> +		.mask		= HCR_TERR,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_APK] = {
> +		.index		= HCR_EL2,
> +		.value		= 0,
> +		.mask		= HCR_APK,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_NV] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_NV,
> +		.mask		= HCR_NV | HCR_NV2,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_NV1] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_NV | HCR_NV1,
> +		.mask		= HCR_NV | HCR_NV1 | HCR_NV2,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_AT] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_AT,
> +		.mask		= HCR_AT,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_FIEN] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_FIEN,
> +		.mask		= HCR_FIEN,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_TID4] = {
> +		.index		= HCR_EL2,
> +		.value 		= HCR_TID4,
> +		.mask		= HCR_TID4,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_TICAB] = {
> +		.index		= HCR_EL2,
> +		.value 		= HCR_TICAB,
> +		.mask		= HCR_TICAB,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_TOCU] = {
> +		.index		= HCR_EL2,
> +		.value 		= HCR_TOCU,
> +		.mask		= HCR_TOCU,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_ENSCXT] = {
> +		.index		= HCR_EL2,
> +		.value 		= 0,
> +		.mask		= HCR_ENSCXT,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_TTLBIS] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_TTLBIS,
> +		.mask		= HCR_TTLBIS,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_TTLBOS] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_TTLBOS,
> +		.mask		= HCR_TTLBOS,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
>  };
>  
>  #define MCB(id, ...)					\
> @@ -61,6 +258,14 @@ static const struct trap_bits coarse_trap_bits[] = {
>  		}
>  
>  static const enum coarse_grain_trap_id *coarse_control_combo[] = {
> +	MCB(CGT_HCR_IMO_FMO,		CGT_HCR_IMO, CGT_HCR_FMO),
> +	MCB(CGT_HCR_TID2_TID4,		CGT_HCR_TID2, CGT_HCR_TID4),
> +	MCB(CGT_HCR_TTLB_TTLBIS,	CGT_HCR_TTLB, CGT_HCR_TTLBIS),
> +	MCB(CGT_HCR_TTLB_TTLBOS,	CGT_HCR_TTLB, CGT_HCR_TTLBOS),
> +	MCB(CGT_HCR_TVM_TRVM,		CGT_HCR_TVM, CGT_HCR_TRVM),
> +	MCB(CGT_HCR_TPU_TICAB,		CGT_HCR_TPU, CGT_HCR_TICAB),
> +	MCB(CGT_HCR_TPU_TOCU,		CGT_HCR_TPU, CGT_HCR_TOCU),
> +	MCB(CGT_HCR_NV1_ENSCXT,		CGT_HCR_NV1, CGT_HCR_ENSCXT),
>  };
>  
>  typedef enum trap_behaviour (*complex_condition_check)(struct kvm_vcpu *);
> @@ -118,6 +323,276 @@ struct encoding_to_trap_config {
>   * re-injected in the nested hypervisor.
>   */
>  static const struct encoding_to_trap_config encoding_to_cgt[] __initdata = {
> +	SR_TRAP(SYS_REVIDR_EL1,		CGT_HCR_TID1),
> +	SR_TRAP(SYS_AIDR_EL1,		CGT_HCR_TID1),
> +	SR_TRAP(SYS_SMIDR_EL1,		CGT_HCR_TID1),
> +	SR_TRAP(SYS_CTR_EL0,		CGT_HCR_TID2),
> +	SR_TRAP(SYS_CCSIDR_EL1,		CGT_HCR_TID2_TID4),
> +	SR_TRAP(SYS_CCSIDR2_EL1,	CGT_HCR_TID2_TID4),
> +	SR_TRAP(SYS_CLIDR_EL1,		CGT_HCR_TID2_TID4),
> +	SR_TRAP(SYS_CSSELR_EL1,		CGT_HCR_TID2_TID4),
> +	SR_RANGE_TRAP(SYS_ID_PFR0_EL1,
> +		      sys_reg(3, 0, 0, 7, 7), CGT_HCR_TID3),
in the spec I see this upper limit in the FEAT_FGT section. Out of
curiosity how were you able to convert the sys reg names into this Op0,
Op1, CRn, CRm, Op2. Is there any ordering logic documented somewhere for
those group3 regs?

I checked Table D18-2 and this looks good but I wonder if there isn't
any more efficient way to review this.
> +	SR_TRAP(SYS_ICC_SGI0R_EL1,	CGT_HCR_IMO_FMO),
> +	SR_TRAP(SYS_ICC_ASGI1R_EL1,	CGT_HCR_IMO_FMO),
> +	SR_TRAP(SYS_ICC_SGI1R_EL1,	CGT_HCR_IMO_FMO),
> +	SR_RANGE_TRAP(sys_reg(3, 0, 11, 0, 0),
> +		      sys_reg(3, 0, 11, 15, 7), CGT_HCR_TIDCP),
> +	SR_RANGE_TRAP(sys_reg(3, 1, 11, 0, 0),
> +		      sys_reg(3, 1, 11, 15, 7), CGT_HCR_TIDCP),
> +	SR_RANGE_TRAP(sys_reg(3, 2, 11, 0, 0),
> +		      sys_reg(3, 2, 11, 15, 7), CGT_HCR_TIDCP),
> +	SR_RANGE_TRAP(sys_reg(3, 3, 11, 0, 0),
> +		      sys_reg(3, 3, 11, 15, 7), CGT_HCR_TIDCP),
> +	SR_RANGE_TRAP(sys_reg(3, 4, 11, 0, 0),
> +		      sys_reg(3, 4, 11, 15, 7), CGT_HCR_TIDCP),
> +	SR_RANGE_TRAP(sys_reg(3, 5, 11, 0, 0),
> +		      sys_reg(3, 5, 11, 15, 7), CGT_HCR_TIDCP),
> +	SR_RANGE_TRAP(sys_reg(3, 6, 11, 0, 0),
> +		      sys_reg(3, 6, 11, 15, 7), CGT_HCR_TIDCP),
> +	SR_RANGE_TRAP(sys_reg(3, 7, 11, 0, 0),
> +		      sys_reg(3, 7, 11, 15, 7), CGT_HCR_TIDCP),
> +	SR_RANGE_TRAP(sys_reg(3, 0, 15, 0, 0),
> +		      sys_reg(3, 0, 15, 15, 7), CGT_HCR_TIDCP),
> +	SR_RANGE_TRAP(sys_reg(3, 1, 15, 0, 0),
> +		      sys_reg(3, 1, 15, 15, 7), CGT_HCR_TIDCP),
> +	SR_RANGE_TRAP(sys_reg(3, 2, 15, 0, 0),
> +		      sys_reg(3, 2, 15, 15, 7), CGT_HCR_TIDCP),
> +	SR_RANGE_TRAP(sys_reg(3, 3, 15, 0, 0),
> +		      sys_reg(3, 3, 15, 15, 7), CGT_HCR_TIDCP),
> +	SR_RANGE_TRAP(sys_reg(3, 4, 15, 0, 0),
> +		      sys_reg(3, 4, 15, 15, 7), CGT_HCR_TIDCP),
> +	SR_RANGE_TRAP(sys_reg(3, 5, 15, 0, 0),
> +		      sys_reg(3, 5, 15, 15, 7), CGT_HCR_TIDCP),
> +	SR_RANGE_TRAP(sys_reg(3, 6, 15, 0, 0),
> +		      sys_reg(3, 6, 15, 15, 7), CGT_HCR_TIDCP),
> +	SR_RANGE_TRAP(sys_reg(3, 7, 15, 0, 0),
> +		      sys_reg(3, 7, 15, 15, 7), CGT_HCR_TIDCP),
> +	SR_TRAP(SYS_ACTLR_EL1,		CGT_HCR_TACR),
> +	SR_TRAP(SYS_DC_ISW,		CGT_HCR_TSW),
> +	SR_TRAP(SYS_DC_CSW,		CGT_HCR_TSW),
> +	SR_TRAP(SYS_DC_CISW,		CGT_HCR_TSW),
> +	SR_TRAP(SYS_DC_IGSW,		CGT_HCR_TSW),
> +	SR_TRAP(SYS_DC_IGDSW,		CGT_HCR_TSW),
> +	SR_TRAP(SYS_DC_CGSW,		CGT_HCR_TSW),
> +	SR_TRAP(SYS_DC_CGDSW,		CGT_HCR_TSW),
> +	SR_TRAP(SYS_DC_CIGSW,		CGT_HCR_TSW),
> +	SR_TRAP(SYS_DC_CIGDSW,		CGT_HCR_TSW),
> +	SR_TRAP(SYS_DC_CIVAC,		CGT_HCR_TPC),
I don't see CVADP?
> +	SR_TRAP(SYS_DC_CVAC,		CGT_HCR_TPC),
> +	SR_TRAP(SYS_DC_CVAP,		CGT_HCR_TPC),
> +	SR_TRAP(SYS_DC_IVAC,		CGT_HCR_TPC),
> +	SR_TRAP(SYS_DC_CIGVAC,		CGT_HCR_TPC),
> +	SR_TRAP(SYS_DC_CIGDVAC,		CGT_HCR_TPC),
> +	SR_TRAP(SYS_DC_IGVAC,		CGT_HCR_TPC),
> +	SR_TRAP(SYS_DC_IGDVAC,		CGT_HCR_TPC),
> +	SR_TRAP(SYS_DC_CGVAC,		CGT_HCR_TPC),
> +	SR_TRAP(SYS_DC_CGDVAC,		CGT_HCR_TPC),
> +	SR_TRAP(SYS_DC_CGVAP,		CGT_HCR_TPC),
> +	SR_TRAP(SYS_DC_CGDVAP,		CGT_HCR_TPC),
> +	SR_TRAP(SYS_DC_CGVADP,		CGT_HCR_TPC),
> +	SR_TRAP(SYS_DC_CGDVADP,		CGT_HCR_TPC),
> +	SR_TRAP(SYS_IC_IVAU,		CGT_HCR_TPU_TOCU),
> +	SR_TRAP(SYS_IC_IALLU,		CGT_HCR_TPU_TOCU),
> +	SR_TRAP(SYS_IC_IALLUIS,		CGT_HCR_TPU_TICAB),
> +	SR_TRAP(SYS_DC_CVAU,		CGT_HCR_TPU_TOCU),
> +	SR_TRAP(OP_TLBI_RVAE1,		CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_RVAAE1,		CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_RVALE1,		CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_RVAALE1,	CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_VMALLE1,	CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_VAE1,		CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_ASIDE1,		CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_VAAE1,		CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_VALE1,		CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_VAALE1,		CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_RVAE1NXS,	CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_RVAAE1NXS,	CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_RVALE1NXS,	CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_RVAALE1NXS,	CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_VMALLE1NXS,	CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_VAE1NXS,	CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_ASIDE1NXS,	CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_VAAE1NXS,	CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_VALE1NXS,	CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_VAALE1NXS,	CGT_HCR_TTLB),*
> +	SR_TRAP(OP_TLBI_RVAE1IS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_RVAAE1IS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_RVALE1IS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_RVAALE1IS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_VMALLE1IS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_VAE1IS,		CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_ASIDE1IS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_VAAE1IS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_VALE1IS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_VAALE1IS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_RVAE1ISNXS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_RVAAE1ISNXS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_RVALE1ISNXS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_RVAALE1ISNXS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_VMALLE1ISNXS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_VAE1ISNXS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_ASIDE1ISNXS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_VAAE1ISNXS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_VALE1ISNXS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_VAALE1ISNXS,	CGT_HCR_TTLB_TTLBIS),*
> +	SR_TRAP(OP_TLBI_VMALLE1OS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_VAE1OS,		CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_ASIDE1OS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_VAAE1OS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_VALE1OS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_VAALE1OS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_RVAE1OS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_RVAAE1OS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_RVALE1OS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_RVAALE1OS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_VMALLE1OSNXS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_VAE1OSNXS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_ASIDE1OSNXS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_VAAE1OSNXS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_VALE1OSNXS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_VAALE1OSNXS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_RVAE1OSNXS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_RVAAE1OSNXS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_RVALE1OSNXS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(OP_TLBI_RVAALE1OSNXS,	CGT_HCR_TTLB_TTLBOS),*
> +	SR_TRAP(SYS_SCTLR_EL1,		CGT_HCR_TVM_TRVM),
> +	SR_TRAP(SYS_TTBR0_EL1,		CGT_HCR_TVM_TRVM),
> +	SR_TRAP(SYS_TTBR1_EL1,		CGT_HCR_TVM_TRVM),
> +	SR_TRAP(SYS_TCR_EL1,		CGT_HCR_TVM_TRVM),
> +	SR_TRAP(SYS_ESR_EL1,		CGT_HCR_TVM_TRVM),
> +	SR_TRAP(SYS_FAR_EL1,		CGT_HCR_TVM_TRVM),
> +	SR_TRAP(SYS_AFSR0_EL1,		CGT_HCR_TVM_TRVM),*
Looking at the SFSR0_EL1 MRS/MSR pseudo code I understand TRVM is tested
on read and
TVM is tested on write. However CGT_HCR_TVM has FORWARD_ANY behaviour
while TRVM looks good as FORWARD_READ? Do I miss something.

> +	SR_TRAP(SYS_AFSR1_EL1,		CGT_HCR_TVM_TRVM),*
same here and below
> +	SR_TRAP(SYS_MAIR_EL1,		CGT_HCR_TVM_TRVM),
> +	SR_TRAP(SYS_AMAIR_EL1,		CGT_HCR_TVM_TRVM),
> +	SR_TRAP(SYS_CONTEXTIDR_EL1,	CGT_HCR_TVM_TRVM),
> +	SR_TRAP(SYS_DC_ZVA,		CGT_HCR_TDZ),*
> +	SR_TRAP(SYS_DC_GVA,		CGT_HCR_TDZ),*
> +	SR_TRAP(SYS_DC_GZVA,		CGT_HCR_TDZ),*
> +	SR_TRAP(SYS_LORSA_EL1,		CGT_HCR_TLOR),*
> +	SR_TRAP(SYS_LOREA_EL1, 		CGT_HCR_TLOR),*
> +	SR_TRAP(SYS_LORN_EL1, 		CGT_HCR_TLOR),*
> +	SR_TRAP(SYS_LORC_EL1, 		CGT_HCR_TLOR),*
> +	SR_TRAP(SYS_LORID_EL1,		CGT_HCR_TLOR),*
> +	SR_TRAP(SYS_ERRIDR_EL1,		CGT_HCR_TERR),*
> +	SR_TRAP(SYS_ERRSELR_EL1,	CGT_HCR_TERR),*
> +	SR_TRAP(SYS_ERXADDR_EL1,	CGT_HCR_TERR),*
> +	SR_TRAP(SYS_ERXCTLR_EL1,	CGT_HCR_TERR),*
> +	SR_TRAP(SYS_ERXFR_EL1,		CGT_HCR_TERR),*
> +	SR_TRAP(SYS_ERXMISC0_EL1,	CGT_HCR_TERR),*
> +	SR_TRAP(SYS_ERXMISC1_EL1,	CGT_HCR_TERR),*
> +	SR_TRAP(SYS_ERXMISC2_EL1,	CGT_HCR_TERR),*
> +	SR_TRAP(SYS_ERXMISC3_EL1,	CGT_HCR_TERR),*
> +	SR_TRAP(SYS_ERXSTATUS_EL1,	CGT_HCR_TERR),*
> +	SR_TRAP(SYS_APIAKEYLO_EL1,	CGT_HCR_APK),*
> +	SR_TRAP(SYS_APIAKEYHI_EL1,	CGT_HCR_APK),*
> +	SR_TRAP(SYS_APIBKEYLO_EL1,	CGT_HCR_APK),*
> +	SR_TRAP(SYS_APIBKEYHI_EL1,	CGT_HCR_APK),*
> +	SR_TRAP(SYS_APDAKEYLO_EL1,	CGT_HCR_APK),*
> +	SR_TRAP(SYS_APDAKEYHI_EL1,	CGT_HCR_APK),*
> +	SR_TRAP(SYS_APDBKEYLO_EL1,	CGT_HCR_APK),*
> +	SR_TRAP(SYS_APDBKEYHI_EL1,	CGT_HCR_APK),*
> +	SR_TRAP(SYS_APGAKEYLO_EL1,	CGT_HCR_APK),*
> +	SR_TRAP(SYS_APGAKEYHI_EL1,	CGT_HCR_APK),*
> +	/* All _EL2 registers */
> +	SR_RANGE_TRAP(sys_reg(3, 4, 0, 0, 0),
> +		      sys_reg(3, 4, 10, 15, 7), CGT_HCR_NV),
> +	SR_RANGE_TRAP(sys_reg(3, 4, 12, 0, 0),
> +		      sys_reg(3, 4, 14, 15, 7), CGT_HCR_NV),
> +	/* All _EL02, _EL12 registers */
> +	SR_RANGE_TRAP(sys_reg(3, 5, 0, 0, 0),
> +		      sys_reg(3, 5, 10, 15, 7), CGT_HCR_NV),
> +	SR_RANGE_TRAP(sys_reg(3, 5, 12, 0, 0),
> +		      sys_reg(3, 5, 14, 15, 7), CGT_HCR_NV),
same question as bove, where in the ARM ARM do you find those ranges?
> +	SR_TRAP(SYS_SP_EL1,		CGT_HCR_NV),*
> +	SR_TRAP(OP_AT_S1E2R,		CGT_HCR_NV),*
> +	SR_TRAP(OP_AT_S1E2W,		CGT_HCR_NV),*
> +	SR_TRAP(OP_AT_S12E1R,		CGT_HCR_NV),*
> +	SR_TRAP(OP_AT_S12E1W,		CGT_HCR_NV),*
> +	SR_TRAP(OP_AT_S12E0R,		CGT_HCR_NV),*
according to the pseudo code NV2 is not checked
shouldn't we have a separate CGT? Question also valid for a bunch of ops
below
> +	SR_TRAP(OP_AT_S12E0W,		CGT_HCR_NV),*
> +	SR_TRAP(OP_TLBI_IPAS2E1,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RIPAS2E1,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_IPAS2LE1,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RIPAS2LE1,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RVAE2,		CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RVALE2,		CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_ALLE2,		CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VAE2,		CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_ALLE1,		CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VALE2,		CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VMALLS12E1,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_IPAS2E1NXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RIPAS2E1NXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_IPAS2LE1NXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RIPAS2LE1NXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RVAE2NXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RVALE2NXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_ALLE2NXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VAE2NXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_ALLE1NXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VALE2NXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VMALLS12E1NXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_IPAS2E1IS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RIPAS2E1IS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_IPAS2LE1IS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RIPAS2LE1IS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RVAE2IS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RVALE2IS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_ALLE2IS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VAE2IS,		CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_ALLE1IS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VALE2IS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VMALLS12E1IS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_IPAS2E1ISNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RIPAS2E1ISNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_IPAS2LE1ISNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RIPAS2LE1ISNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RVAE2ISNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RVALE2ISNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_ALLE2ISNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VAE2ISNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_ALLE1ISNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VALE2ISNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VMALLS12E1ISNXS,CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_ALLE2OS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VAE2OS,		CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_ALLE1OS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VALE2OS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VMALLS12E1OS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_IPAS2E1OS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RIPAS2E1OS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_IPAS2LE1OS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RIPAS2LE1OS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RVAE2OS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RVALE2OS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_ALLE2OSNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VAE2OSNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_ALLE1OSNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VALE2OSNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_VMALLS12E1OSNXS,CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_IPAS2E1OSNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RIPAS2E1OSNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_IPAS2LE1OSNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RIPAS2LE1OSNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RVAE2OSNXS,	CGT_HCR_NV),
> +	SR_TRAP(OP_TLBI_RVALE2OSNXS,	CGT_HCR_NV),
> +	SR_TRAP(SYS_VBAR_EL1,		CGT_HCR_NV1),
> +	SR_TRAP(SYS_ELR_EL1,		CGT_HCR_NV1),

CIGDPAE?
CIPAE?
CFP/CPP/DVP RCTX?
> +	SR_TRAP(SYS_SPSR_EL1,		CGT_HCR_NV1),*
> +	SR_TRAP(SYS_SCXTNUM_EL1,	CGT_HCR_NV1_ENSCXT),
> +	SR_TRAP(SYS_SCXTNUM_EL0,	CGT_HCR_ENSCXT),
> +	SR_TRAP(OP_AT_S1E1R, 		CGT_HCR_AT),
> +	SR_TRAP(OP_AT_S1E1W, 		CGT_HCR_AT),
> +	SR_TRAP(OP_AT_S1E0R, 		CGT_HCR_AT),
> +	SR_TRAP(OP_AT_S1E0W, 		CGT_HCR_AT),
> +	SR_TRAP(OP_AT_S1E1RP, 		CGT_HCR_AT),
> +	SR_TRAP(OP_AT_S1E1WP, 		CGT_HCR_AT),
> +	SR_TRAP(SYS_ERXPFGF_EL1,	CGT_HCR_FIEN),
> +	SR_TRAP(SYS_ERXPFGCTL_EL1,	CGT_HCR_FIEN),
> +	SR_TRAP(SYS_ERXPFGCDN_EL1,	CGT_HCR_FIEN),
> +	SR_TRAP(SYS_SCXTNUM_EL0,	CGT_HCR_ENSCXT),
>  };
>  
>  static DEFINE_XARRAY(sr_forward_xa);
Thanks

Eric

Re: [PATCH 16/27] KVM: arm64: nv: Add trap forwarding for HCR_EL2

[Marc Zyngier]

Hey Eric,

Thanks for looking into this, much appreciated given how tedious it
is.

On Thu, 13 Jul 2023 15:05:33 +0100,
Eric Auger <eric.auger@redhat.com> wrote:
> 
> Hi Marc,
> 
> On 7/12/23 16:57, Marc Zyngier wrote:
> > Describe the HCR_EL2 register, and associate it with all the sysregs
> > it allows to trap.
> >
> > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > ---
> >  arch/arm64/kvm/emulate-nested.c | 475 ++++++++++++++++++++++++++++++++
> >  1 file changed, 475 insertions(+)
> >
> > diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
> > index 5bab2e85d70c..51901e85e43d 100644
> > --- a/arch/arm64/kvm/emulate-nested.c
> > +++ b/arch/arm64/kvm/emulate-nested.c

[...]

> > +	[CGT_HCR_TPC] = {
> modern revisions now refer to TPCP, maybe worth a comment?

Absolutely.

[...]

> > +	SR_RANGE_TRAP(SYS_ID_PFR0_EL1,
> > +		      sys_reg(3, 0, 0, 7, 7), CGT_HCR_TID3),
> in the spec I see this upper limit in the FEAT_FGT section. Out of
> curiosity how were you able to convert the sys reg names into this Op0,
> Op1, CRn, CRm, Op2. Is there any ordering logic documented somewhere for
> those group3 regs?

If you look at the sysreg encoding described on page D18-6308 if
version J.a of the ARM ARM, you will find a block of 56 contiguous
encodings ranging from (3, 0, 0, 1, 0), which happens to be
ID_PFR0_EL1, all the way to a reserved range ending in (3, 0, 0, 7,
7).

This is the block of register that is controlled by TID3.

> I checked Table D18-2 and this looks good but I wonder if there isn't
> any more efficient way to review this.

Not that I know of, unfortunately. Even the pseudocode isn't enough
for this as it doesn't described the trapping of unallocated regions.

> > +	SR_TRAP(SYS_ICC_SGI0R_EL1,	CGT_HCR_IMO_FMO),
> > +	SR_TRAP(SYS_ICC_ASGI1R_EL1,	CGT_HCR_IMO_FMO),
> > +	SR_TRAP(SYS_ICC_SGI1R_EL1,	CGT_HCR_IMO_FMO),
> > +	SR_RANGE_TRAP(sys_reg(3, 0, 11, 0, 0),
> > +		      sys_reg(3, 0, 11, 15, 7), CGT_HCR_TIDCP),
> > +	SR_RANGE_TRAP(sys_reg(3, 1, 11, 0, 0),
> > +		      sys_reg(3, 1, 11, 15, 7), CGT_HCR_TIDCP),
> > +	SR_RANGE_TRAP(sys_reg(3, 2, 11, 0, 0),
> > +		      sys_reg(3, 2, 11, 15, 7), CGT_HCR_TIDCP),
> > +	SR_RANGE_TRAP(sys_reg(3, 3, 11, 0, 0),
> > +		      sys_reg(3, 3, 11, 15, 7), CGT_HCR_TIDCP),
> > +	SR_RANGE_TRAP(sys_reg(3, 4, 11, 0, 0),
> > +		      sys_reg(3, 4, 11, 15, 7), CGT_HCR_TIDCP),
> > +	SR_RANGE_TRAP(sys_reg(3, 5, 11, 0, 0),
> > +		      sys_reg(3, 5, 11, 15, 7), CGT_HCR_TIDCP),
> > +	SR_RANGE_TRAP(sys_reg(3, 6, 11, 0, 0),
> > +		      sys_reg(3, 6, 11, 15, 7), CGT_HCR_TIDCP),
> > +	SR_RANGE_TRAP(sys_reg(3, 7, 11, 0, 0),
> > +		      sys_reg(3, 7, 11, 15, 7), CGT_HCR_TIDCP),
> > +	SR_RANGE_TRAP(sys_reg(3, 0, 15, 0, 0),
> > +		      sys_reg(3, 0, 15, 15, 7), CGT_HCR_TIDCP),
> > +	SR_RANGE_TRAP(sys_reg(3, 1, 15, 0, 0),
> > +		      sys_reg(3, 1, 15, 15, 7), CGT_HCR_TIDCP),
> > +	SR_RANGE_TRAP(sys_reg(3, 2, 15, 0, 0),
> > +		      sys_reg(3, 2, 15, 15, 7), CGT_HCR_TIDCP),
> > +	SR_RANGE_TRAP(sys_reg(3, 3, 15, 0, 0),
> > +		      sys_reg(3, 3, 15, 15, 7), CGT_HCR_TIDCP),
> > +	SR_RANGE_TRAP(sys_reg(3, 4, 15, 0, 0),
> > +		      sys_reg(3, 4, 15, 15, 7), CGT_HCR_TIDCP),
> > +	SR_RANGE_TRAP(sys_reg(3, 5, 15, 0, 0),
> > +		      sys_reg(3, 5, 15, 15, 7), CGT_HCR_TIDCP),
> > +	SR_RANGE_TRAP(sys_reg(3, 6, 15, 0, 0),
> > +		      sys_reg(3, 6, 15, 15, 7), CGT_HCR_TIDCP),
> > +	SR_RANGE_TRAP(sys_reg(3, 7, 15, 0, 0),
> > +		      sys_reg(3, 7, 15, 15, 7), CGT_HCR_TIDCP),
> > +	SR_TRAP(SYS_ACTLR_EL1,		CGT_HCR_TACR),
> > +	SR_TRAP(SYS_DC_ISW,		CGT_HCR_TSW),
> > +	SR_TRAP(SYS_DC_CSW,		CGT_HCR_TSW),
> > +	SR_TRAP(SYS_DC_CISW,		CGT_HCR_TSW),
> > +	SR_TRAP(SYS_DC_IGSW,		CGT_HCR_TSW),
> > +	SR_TRAP(SYS_DC_IGDSW,		CGT_HCR_TSW),
> > +	SR_TRAP(SYS_DC_CGSW,		CGT_HCR_TSW),
> > +	SR_TRAP(SYS_DC_CGDSW,		CGT_HCR_TSW),
> > +	SR_TRAP(SYS_DC_CIGSW,		CGT_HCR_TSW),
> > +	SR_TRAP(SYS_DC_CIGDSW,		CGT_HCR_TSW),
> > +	SR_TRAP(SYS_DC_CIVAC,		CGT_HCR_TPC),
> I don't see CVADP?

Me neither! Good catch!

[...]

> > +	SR_TRAP(SYS_SCTLR_EL1,		CGT_HCR_TVM_TRVM),
> > +	SR_TRAP(SYS_TTBR0_EL1,		CGT_HCR_TVM_TRVM),
> > +	SR_TRAP(SYS_TTBR1_EL1,		CGT_HCR_TVM_TRVM),
> > +	SR_TRAP(SYS_TCR_EL1,		CGT_HCR_TVM_TRVM),
> > +	SR_TRAP(SYS_ESR_EL1,		CGT_HCR_TVM_TRVM),
> > +	SR_TRAP(SYS_FAR_EL1,		CGT_HCR_TVM_TRVM),
> > +	SR_TRAP(SYS_AFSR0_EL1,		CGT_HCR_TVM_TRVM),*
> Looking at the SFSR0_EL1 MRS/MSR pseudo code I understand TRVM is tested
> on read and
> TVM is tested on write. However CGT_HCR_TVM has FORWARD_ANY behaviour
> while TRVM looks good as FORWARD_READ? Do I miss something.

You're not missing anything. For some reason, I had in my head that
TVM was trapping both reads and writes, while the spec is clear that
it only traps writes.

> 
> > +	SR_TRAP(SYS_AFSR1_EL1,		CGT_HCR_TVM_TRVM),*
> same here and below

Yup, I need to fix the TVM encoding like this:

@@ -176,7 +176,7 @@ static const struct trap_bits coarse_trap_bits[] = {
 		.index		= HCR_EL2,
 		.value		= HCR_TVM,
 		.mask		= HCR_TVM,
-		.behaviour	= BEHAVE_FORWARD_ANY,
+		.behaviour	= BEHAVE_FORWARD_WRITE,
 	},
 	[CGT_HCR_TDZ] = {
 		.index		= HCR_EL2,

[...]

> > +	/* All _EL2 registers */
> > +	SR_RANGE_TRAP(sys_reg(3, 4, 0, 0, 0),
> > +		      sys_reg(3, 4, 10, 15, 7), CGT_HCR_NV),
> > +	SR_RANGE_TRAP(sys_reg(3, 4, 12, 0, 0),
> > +		      sys_reg(3, 4, 14, 15, 7), CGT_HCR_NV),
> > +	/* All _EL02, _EL12 registers */
> > +	SR_RANGE_TRAP(sys_reg(3, 5, 0, 0, 0),
> > +		      sys_reg(3, 5, 10, 15, 7), CGT_HCR_NV),
> > +	SR_RANGE_TRAP(sys_reg(3, 5, 12, 0, 0),
> > +		      sys_reg(3, 5, 14, 15, 7), CGT_HCR_NV),
> same question as bove, where in the ARM ARM do you find those
> ranges?

I went over the encoding with a fine comb, and realised that all the
(3, 4, ...) encodings are EL2, and all the (3, 5, ...) ones are EL02
and EL12.

I appreciate that this is taking a massive bet on the future, but
there is no such rule in the ARM ARM as such...

> > +	SR_TRAP(SYS_SP_EL1,		CGT_HCR_NV),*
> > +	SR_TRAP(OP_AT_S1E2R,		CGT_HCR_NV),*
> > +	SR_TRAP(OP_AT_S1E2W,		CGT_HCR_NV),*
> > +	SR_TRAP(OP_AT_S12E1R,		CGT_HCR_NV),*
> > +	SR_TRAP(OP_AT_S12E1W,		CGT_HCR_NV),*
> > +	SR_TRAP(OP_AT_S12E0R,		CGT_HCR_NV),*
> according to the pseudo code NV2 is not checked
> shouldn't we have a separate CGT? Question also valid for a bunch of ops
> below

Hmmm. Yes, this is wrong. Well spotted. I guess I need a
CGT_HCR_NV_nNV2 for the cases that want that particular condition (NV1
probably needs a similar fix).

[...]

> CIGDPAE?
> CIPAE?

These two are part of RME (well, technically MEC, but that an RME
thing), and I have no plan to support this with NV -- yet.

> CFP/CPP/DVP RCTX?

These are definitely missing. I'll add them.

Thanks again for going through this list, this is awesome work!

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 18/27] KVM: arm64: nv: Add trap forwarding for MDCR_EL2

[Eric Auger]

Hi Marc,

On 7/12/23 16:58, Marc Zyngier wrote:
> Describe the MDCR_EL2 register, and associate it with all the sysregs
> it allows to trap.
>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  arch/arm64/kvm/emulate-nested.c | 262 ++++++++++++++++++++++++++++++++
>  1 file changed, 262 insertions(+)
>
> diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
> index 51901e85e43d..25e4842ac334 100644
> --- a/arch/arm64/kvm/emulate-nested.c
> +++ b/arch/arm64/kvm/emulate-nested.c
> @@ -65,6 +65,18 @@ enum coarse_grain_trap_id {
>  	CGT_HCR_TTLBIS,
>  	CGT_HCR_TTLBOS,
>  
> +	CGT_MDCR_TPMCR,
> +	CGT_MDCR_TPM,
> +	CGT_MDCR_TDE,
> +	CGT_MDCR_TDA,
> +	CGT_MDCR_TDOSA,
> +	CGT_MDCR_TDRA,
> +	CGT_MDCR_E2PB,
> +	CGT_MDCR_TPMS,
> +	CGT_MDCR_TTRF,
> +	CGT_MDCR_E2TB,
> +	CGT_MDCR_TDCC,
> +
>  	/*
>  	 * Anything after this point is a combination of trap controls,
>  	 * which all must be evaluated to decide what to do.
> @@ -78,6 +90,11 @@ enum coarse_grain_trap_id {
>  	CGT_HCR_TPU_TICAB,
>  	CGT_HCR_TPU_TOCU,
>  	CGT_HCR_NV1_ENSCXT,
> +	CGT_MDCR_TPM_TPMCR,
> +	CGT_MDCR_TDE_TDA,
> +	CGT_MDCR_TDE_TDOSA,
> +	CGT_MDCR_TDE_TDRA,
> +	CGT_MDCR_TDCC_TDE_TDA,
>  
>  	/*
>  	 * Anything after this point requires a callback evaluating a
> @@ -249,6 +266,72 @@ static const struct trap_bits coarse_trap_bits[] = {
>  		.mask		= HCR_TTLBOS,
>  		.behaviour	= BEHAVE_FORWARD_ANY,
>  	},
> +	[CGT_MDCR_TPMCR] = {
> +		.index		= MDCR_EL2,
> +		.value		= MDCR_EL2_TPMCR,
> +		.mask		= MDCR_EL2_TPMCR,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_MDCR_TPM] = {
> +		.index		= MDCR_EL2,
> +		.value		= MDCR_EL2_TPM,
> +		.mask		= MDCR_EL2_TPM,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_MDCR_TDE] = {
> +		.index		= MDCR_EL2,
> +		.value		= MDCR_EL2_TDE,
> +		.mask		= MDCR_EL2_TDE,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_MDCR_TDA] = {
> +		.index		= MDCR_EL2,
> +		.value		= MDCR_EL2_TDA,
> +		.mask		= MDCR_EL2_TDA,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_MDCR_TDOSA] = {
> +		.index		= MDCR_EL2,
> +		.value		= MDCR_EL2_TDOSA,
> +		.mask		= MDCR_EL2_TDOSA,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_MDCR_TDRA] = {
> +		.index		= MDCR_EL2,
> +		.value		= MDCR_EL2_TDRA,
> +		.mask		= MDCR_EL2_TDRA,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_MDCR_E2PB] = {
> +		.index		= MDCR_EL2,
> +		.value		= 0,
> +		.mask		= BIT(MDCR_EL2_E2PB_SHIFT),
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_MDCR_TPMS] = {
> +		.index		= MDCR_EL2,
> +		.value		= MDCR_EL2_TPMS,
> +		.mask		= MDCR_EL2_TPMS,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_MDCR_TTRF] = {
> +		.index		= MDCR_EL2,
> +		.value		= MDCR_EL2_TTRF,
> +		.mask		= MDCR_EL2_TTRF,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_MDCR_E2TB] = {
> +		.index		= MDCR_EL2,
> +		.value		= 0,
> +		.mask		= BIT(MDCR_EL2_E2TB_SHIFT),
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_MDCR_TDCC] = {
> +		.index		= MDCR_EL2,
> +		.value		= MDCR_EL2_TDCC,
> +		.mask		= MDCR_EL2_TDCC,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
>  };
>  
>  #define MCB(id, ...)					\
> @@ -266,6 +349,11 @@ static const enum coarse_grain_trap_id *coarse_control_combo[] = {
>  	MCB(CGT_HCR_TPU_TICAB,		CGT_HCR_TPU, CGT_HCR_TICAB),
>  	MCB(CGT_HCR_TPU_TOCU,		CGT_HCR_TPU, CGT_HCR_TOCU),
>  	MCB(CGT_HCR_NV1_ENSCXT,		CGT_HCR_NV1, CGT_HCR_ENSCXT),
> +	MCB(CGT_MDCR_TPM_TPMCR,		CGT_MDCR_TPM, CGT_MDCR_TPMCR),
> +	MCB(CGT_MDCR_TDE_TDA,		CGT_MDCR_TDE, CGT_MDCR_TDA),
> +	MCB(CGT_MDCR_TDE_TDOSA,		CGT_MDCR_TDE, CGT_MDCR_TDOSA),
> +	MCB(CGT_MDCR_TDE_TDRA,		CGT_MDCR_TDE, CGT_MDCR_TDRA),
> +	MCB(CGT_MDCR_TDCC_TDE_TDA,	CGT_MDCR_TDCC, CGT_MDCR_TDE_TDA),
>  };
>  
>  typedef enum trap_behaviour (*complex_condition_check)(struct kvm_vcpu *);
> @@ -593,6 +681,180 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initdata = {
>  	SR_TRAP(SYS_ERXPFGCTL_EL1,	CGT_HCR_FIEN),
>  	SR_TRAP(SYS_ERXPFGCDN_EL1,	CGT_HCR_FIEN),
>  	SR_TRAP(SYS_SCXTNUM_EL0,	CGT_HCR_ENSCXT),
> +	SR_TRAP(SYS_PMCR_EL0,		CGT_MDCR_TPM_TPMCR), *
> +	SR_TRAP(SYS_PMCNTENSET_EL0,	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMCNTENCLR_EL0,	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMOVSSET_EL0,	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMOVSCLR_EL0,	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMCEID0_EL0,	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMCEID1_EL0,	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMXEVTYPER_EL0,	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMSWINC_EL0,	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMSELR_EL0,		CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMXEVCNTR_EL0,	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMCCNTR_EL0,	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMUSERENR_EL0,	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMINTENSET_EL1,	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMINTENCLR_EL1,	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMMIR_EL1,		CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(0),	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(1),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(2),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(3),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(4),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(5),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(6),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(7),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(8),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(9),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(10),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(11),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(12),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(13),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(14),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(15),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(16),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(17),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(18),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(19),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(20),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(21),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(22),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(23),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(24),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(25),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(26),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(27),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(28),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(29),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVCNTRn_EL0(30),	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(0),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(1),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(2),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(3),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(4),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(5),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(6),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(7),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(8),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(9),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(10),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(11),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(12),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(13),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(14),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(15),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(16),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(17),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(18),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(19),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(20),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(21),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(22),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(23),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(24),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(25),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(26),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(27),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(28),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(29),	CGT_MDCR_TPM),
> +	SR_TRAP(SYS_PMEVTYPERn_EL0(30),	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_PMCCFILTR_EL0,	CGT_MDCR_TPM),*
> +	SR_TRAP(SYS_MDCCSR_EL0,		CGT_MDCR_TDCC_TDE_TDA),*
> +	SR_TRAP(SYS_MDCCINT_EL1,	CGT_MDCR_TDCC_TDE_TDA),*
> +	SR_TRAP(SYS_OSDTRRX_EL1,	CGT_MDCR_TDCC_TDE_TDA),*
> +	SR_TRAP(SYS_OSDTRTX_EL1,	CGT_MDCR_TDCC_TDE_TDA),*
Please also double check DBGDTR_EL0, DBGDTRRX/TX_EL0. I understand from
the spec they may end up in CGT_MDCR_TDCC_TDE_TDA too
> +	SR_TRAP(SYS_MDSCR_EL1,		CGT_MDCR_TDE_TDA),*
> +	SR_TRAP(SYS_OSECCR_EL1,		CGT_MDCR_TDE_TDA),*
> +	SR_TRAP(SYS_DBGBVRn_EL1(0),	CGT_MDCR_TDE_TDA),*
> +	SR_TRAP(SYS_DBGBVRn_EL1(1),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBVRn_EL1(2),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBVRn_EL1(3),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBVRn_EL1(4),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBVRn_EL1(5),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBVRn_EL1(6),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBVRn_EL1(7),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBVRn_EL1(8),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBVRn_EL1(9),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBVRn_EL1(10),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBVRn_EL1(11),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBVRn_EL1(12),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBVRn_EL1(13),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBVRn_EL1(14),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBVRn_EL1(15),	CGT_MDCR_TDE_TDA),*
> +	SR_TRAP(SYS_DBGBCRn_EL1(0),	CGT_MDCR_TDE_TDA),*
> +	SR_TRAP(SYS_DBGBCRn_EL1(1),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBCRn_EL1(2),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBCRn_EL1(3),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBCRn_EL1(4),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBCRn_EL1(5),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBCRn_EL1(6),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBCRn_EL1(7),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBCRn_EL1(8),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBCRn_EL1(9),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBCRn_EL1(10),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBCRn_EL1(11),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBCRn_EL1(12),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBCRn_EL1(13),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBCRn_EL1(14),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGBCRn_EL1(15),	CGT_MDCR_TDE_TDA),*
> +	SR_TRAP(SYS_DBGWVRn_EL1(0),	CGT_MDCR_TDE_TDA),*
> +	SR_TRAP(SYS_DBGWVRn_EL1(1),	CGT_MDCR_TDE_TDA),*
> +	SR_TRAP(SYS_DBGWVRn_EL1(2),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWVRn_EL1(3),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWVRn_EL1(4),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWVRn_EL1(5),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWVRn_EL1(6),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWVRn_EL1(7),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWVRn_EL1(8),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWVRn_EL1(9),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWVRn_EL1(10),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWVRn_EL1(11),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWVRn_EL1(12),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWVRn_EL1(13),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWVRn_EL1(14),	CGT_MDCR_TDE_TDA),*
> +	SR_TRAP(SYS_DBGWVRn_EL1(15),	CGT_MDCR_TDE_TDA),*
> +	SR_TRAP(SYS_DBGWCRn_EL1(0),	CGT_MDCR_TDE_TDA),*
> +	SR_TRAP(SYS_DBGWCRn_EL1(1),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWCRn_EL1(2),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWCRn_EL1(3),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWCRn_EL1(4),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWCRn_EL1(5),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWCRn_EL1(6),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWCRn_EL1(7),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWCRn_EL1(8),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWCRn_EL1(9),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWCRn_EL1(10),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWCRn_EL1(11),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWCRn_EL1(12),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWCRn_EL1(13),	CGT_MDCR_TDE_TDA),
> +	SR_TRAP(SYS_DBGWCRn_EL1(14),	CGT_MDCR_TDE_TDA),*
> +	SR_TRAP(SYS_DBGCLAIMSET_EL1,	CGT_MDCR_TDE_TDA),*
> +	SR_TRAP(SYS_DBGCLAIMCLR_EL1,	CGT_MDCR_TDE_TDA),*
> +	SR_TRAP(SYS_DBGAUTHSTATUS_EL1,	CGT_MDCR_TDE_TDA),*
> +	SR_TRAP(SYS_OSLAR_EL1,		CGT_MDCR_TDE_TDOSA),*
> +	SR_TRAP(SYS_OSLSR_EL1,		CGT_MDCR_TDE_TDOSA),*
> +	SR_TRAP(SYS_OSDLR_EL1,		CGT_MDCR_TDE_TDOSA),*
> +	SR_TRAP(SYS_DBGPRCR_EL1,	CGT_MDCR_TDE_TDOSA),*
> +	SR_TRAP(SYS_MDRAR_EL1,		CGT_MDCR_TDE_TDRA),*
> +	SR_TRAP(SYS_PMBLIMITR_EL1,	CGT_MDCR_E2PB),*
> +	SR_TRAP(SYS_PMBPTR_EL1,		CGT_MDCR_E2PB),*
> +	SR_TRAP(SYS_PMBSR_EL1,		CGT_MDCR_E2PB),*
> +	SR_TRAP(SYS_PMSCR_EL1,		CGT_MDCR_TPMS),*
> +	SR_TRAP(SYS_PMSEVFR_EL1,	CGT_MDCR_TPMS),*
> +	SR_TRAP(SYS_PMSFCR_EL1,		CGT_MDCR_TPMS),*
> +	SR_TRAP(SYS_PMSICR_EL1,		CGT_MDCR_TPMS),*
> +	SR_TRAP(SYS_PMSIDR_EL1,		CGT_MDCR_TPMS),*
> +	SR_TRAP(SYS_PMSIRR_EL1,		CGT_MDCR_TPMS),*
> +	SR_TRAP(SYS_PMSLATFR_EL1,	CGT_MDCR_TPMS),*
> +	SR_TRAP(SYS_PMSNEVFR_EL1,	CGT_MDCR_TPMS),*
> +	SR_TRAP(SYS_TRFCR_EL1,		CGT_MDCR_TTRF),*
> +	SR_TRAP(SYS_TRBBASER_EL1,	CGT_MDCR_E2TB),*
> +	SR_TRAP(SYS_TRBLIMITR_EL1,	CGT_MDCR_E2TB),*
> +	SR_TRAP(SYS_TRBMAR_EL1, 	CGT_MDCR_E2TB),*
> +	SR_TRAP(SYS_TRBPTR_EL1, 	CGT_MDCR_E2TB),*
> +	SR_TRAP(SYS_TRBSR_EL1, 		CGT_MDCR_E2TB),*
> +	SR_TRAP(SYS_TRBTRG_EL1,		CGT_MDCR_E2TB),*
>  };
>  
>  static DEFINE_XARRAY(sr_forward_xa);
Besides

Reviewed-by: Eric Auger <eric.auger@redhat.com>

Eric

Re: [PATCH 10/27] arm64: Add feature detection for fine grained traps

[Eric Auger]

Hi,

On 7/12/23 16:57, Marc Zyngier wrote:
> From: Mark Brown <broonie@kernel.org>
>
> In order to allow us to have shared code for managing fine grained traps
> for KVM guests add it as a detected feature rather than relying on it
> being a dependency of other features.
>
> Acked-by: Catalin Marinas <catalin.marinas@arm.com>
> Signed-off-by: Mark Brown <broonie@kernel.org>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> Link: https://lore.kernel.org/r/20230301-kvm-arm64-fgt-v4-1-1bf8d235ac1f@kernel.org
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Eric
> ---
>  arch/arm64/kernel/cpufeature.c | 11 +++++++++++
>  arch/arm64/tools/cpucaps       |  1 +
>  2 files changed, 12 insertions(+)
>
> diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
> index f9d456fe132d..0768f98c49cc 100644
> --- a/arch/arm64/kernel/cpufeature.c
> +++ b/arch/arm64/kernel/cpufeature.c
> @@ -2627,6 +2627,17 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
>  		.matches = has_cpuid_feature,
>  		ARM64_CPUID_FIELDS(ID_AA64ISAR1_EL1, LRCPC, IMP)
>  	},
> +	{
> +		.desc = "Fine Grained Traps",
> +		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
> +		.capability = ARM64_HAS_FGT,
> +		.sys_reg = SYS_ID_AA64MMFR0_EL1,
> +		.sign = FTR_UNSIGNED,
> +		.field_pos = ID_AA64MMFR0_EL1_FGT_SHIFT,
> +		.field_width = 4,
> +		.min_field_value = 1,
> +		.matches = has_cpuid_feature,
> +	},
>  #ifdef CONFIG_ARM64_SME
>  	{
>  		.desc = "Scalable Matrix Extension",
> diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
> index c80ed4f3cbce..c3f06fdef609 100644
> --- a/arch/arm64/tools/cpucaps
> +++ b/arch/arm64/tools/cpucaps
> @@ -26,6 +26,7 @@ HAS_ECV
>  HAS_ECV_CNTPOFF
>  HAS_EPAN
>  HAS_EVT
> +HAS_FGT
>  HAS_GENERIC_AUTH
>  HAS_GENERIC_AUTH_ARCH_QARMA3
>  HAS_GENERIC_AUTH_ARCH_QARMA5

Re: [PATCH 16/27] KVM: arm64: nv: Add trap forwarding for HCR_EL2

[Marc Zyngier]

On Thu, 13 Jul 2023 16:53:40 +0100,
Marc Zyngier <maz@kernel.org> wrote:
> 
> Hey Eric,
> 
> Thanks for looking into this, much appreciated given how tedious it
> is.

FWIW, here are the changes I'm going to squash in that patch. Shout if
you spot something that looks odd...

Thanks,

	M.

diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
index c4057f4ff72d..f5978b463aca 100644
--- a/arch/arm64/kvm/emulate-nested.c
+++ b/arch/arm64/kvm/emulate-nested.c
@@ -55,7 +55,8 @@ enum coarse_grain_trap_id {
 	CGT_HCR_TERR,
 	CGT_HCR_APK,
 	CGT_HCR_NV,
-	CGT_HCR_NV1,
+	CGT_HCR_NV_nNV2,
+	CGT_HCR_NV1_nNV2,
 	CGT_HCR_AT,
 	CGT_HCR_FIEN,
 	CGT_HCR_TID4,
@@ -89,7 +90,7 @@ enum coarse_grain_trap_id {
 	CGT_HCR_TVM_TRVM,
 	CGT_HCR_TPU_TICAB,
 	CGT_HCR_TPU_TOCU,
-	CGT_HCR_NV1_ENSCXT,
+	CGT_HCR_NV1_nNV2_ENSCXT,
 	CGT_MDCR_TPM_TPMCR,
 	CGT_MDCR_TDE_TDA,
 	CGT_MDCR_TDE_TDOSA,
@@ -154,7 +155,7 @@ static const struct trap_bits coarse_trap_bits[] = {
 		.mask		= HCR_TSW,
 		.behaviour	= BEHAVE_FORWARD_ANY,
 	},
-	[CGT_HCR_TPC] = {
+	[CGT_HCR_TPC] = { /* Also called TCPC when FEAT_DPB is implemented */
 		.index		= HCR_EL2,
 		.value		= HCR_TPC,
 		.mask		= HCR_TPC,
@@ -176,7 +177,7 @@ static const struct trap_bits coarse_trap_bits[] = {
 		.index		= HCR_EL2,
 		.value		= HCR_TVM,
 		.mask		= HCR_TVM,
-		.behaviour	= BEHAVE_FORWARD_ANY,
+		.behaviour	= BEHAVE_FORWARD_WRITE,
 	},
 	[CGT_HCR_TDZ] = {
 		.index		= HCR_EL2,
@@ -209,12 +210,18 @@ static const struct trap_bits coarse_trap_bits[] = {
 		.behaviour	= BEHAVE_FORWARD_ANY,
 	},
 	[CGT_HCR_NV] = {
+		.index		= HCR_EL2,
+		.value		= HCR_NV,
+		.mask		= HCR_NV,
+		.behaviour	= BEHAVE_FORWARD_ANY,
+	},
+	[CGT_HCR_NV_nNV2] = {
 		.index		= HCR_EL2,
 		.value		= HCR_NV,
 		.mask		= HCR_NV | HCR_NV2,
 		.behaviour	= BEHAVE_FORWARD_ANY,
 	},
-	[CGT_HCR_NV1] = {
+	[CGT_HCR_NV1_nNV2] = {
 		.index		= HCR_EL2,
 		.value		= HCR_NV | HCR_NV1,
 		.mask		= HCR_NV | HCR_NV1 | HCR_NV2,
@@ -350,7 +357,7 @@ static const enum coarse_grain_trap_id *coarse_control_combo[] = {
 	MCB(CGT_HCR_TVM_TRVM,		CGT_HCR_TVM, CGT_HCR_TRVM),
 	MCB(CGT_HCR_TPU_TICAB,		CGT_HCR_TPU, CGT_HCR_TICAB),
 	MCB(CGT_HCR_TPU_TOCU,		CGT_HCR_TPU, CGT_HCR_TOCU),
-	MCB(CGT_HCR_NV1_ENSCXT,		CGT_HCR_NV1, CGT_HCR_ENSCXT),
+	MCB(CGT_HCR_NV1_nNV2_ENSCXT,	CGT_HCR_NV1_nNV2, CGT_HCR_ENSCXT),
 	MCB(CGT_MDCR_TPM_TPMCR,		CGT_MDCR_TPM, CGT_MDCR_TPMCR),
 	MCB(CGT_MDCR_TDE_TDA,		CGT_MDCR_TDE, CGT_MDCR_TDA),
 	MCB(CGT_MDCR_TDE_TDOSA,		CGT_MDCR_TDE, CGT_MDCR_TDOSA),
@@ -501,6 +508,7 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initdata = {
 	SR_TRAP(SYS_DC_CIVAC,		CGT_HCR_TPC),
 	SR_TRAP(SYS_DC_CVAC,		CGT_HCR_TPC),
 	SR_TRAP(SYS_DC_CVAP,		CGT_HCR_TPC),
+	SR_TRAP(SYS_DC_CVADP,		CGT_HCR_TPC),
 	SR_TRAP(SYS_DC_IVAC,		CGT_HCR_TPC),
 	SR_TRAP(SYS_DC_CIGVAC,		CGT_HCR_TPC),
 	SR_TRAP(SYS_DC_CIGDVAC,		CGT_HCR_TPC),
@@ -625,7 +633,6 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initdata = {
 		      sys_reg(3, 5, 10, 15, 7), CGT_HCR_NV),
 	SR_RANGE_TRAP(sys_reg(3, 5, 12, 0, 0),
 		      sys_reg(3, 5, 14, 15, 7), CGT_HCR_NV),
-	SR_TRAP(SYS_SP_EL1,		CGT_HCR_NV),
 	SR_TRAP(OP_AT_S1E2R,		CGT_HCR_NV),
 	SR_TRAP(OP_AT_S1E2W,		CGT_HCR_NV),
 	SR_TRAP(OP_AT_S12E1R,		CGT_HCR_NV),
@@ -698,10 +705,14 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initdata = {
 	SR_TRAP(OP_TLBI_RIPAS2LE1OSNXS,	CGT_HCR_NV),
 	SR_TRAP(OP_TLBI_RVAE2OSNXS,	CGT_HCR_NV),
 	SR_TRAP(OP_TLBI_RVALE2OSNXS,	CGT_HCR_NV),
-	SR_TRAP(SYS_VBAR_EL1,		CGT_HCR_NV1),
-	SR_TRAP(SYS_ELR_EL1,		CGT_HCR_NV1),
-	SR_TRAP(SYS_SPSR_EL1,		CGT_HCR_NV1),
-	SR_TRAP(SYS_SCXTNUM_EL1,	CGT_HCR_NV1_ENSCXT),
+	SR_TRAP(OP_CPP_RCTX, 		CGT_HCR_NV),
+	SR_TRAP(OP_DVP_RCTX, 		CGT_HCR_NV),
+	SR_TRAP(OP_CFP_RCTX, 		CGT_HCR_NV),
+	SR_TRAP(SYS_SP_EL1,		CGT_HCR_NV_nNV2),
+	SR_TRAP(SYS_VBAR_EL1,		CGT_HCR_NV1_nNV2),
+	SR_TRAP(SYS_ELR_EL1,		CGT_HCR_NV1_nNV2),
+	SR_TRAP(SYS_SPSR_EL1,		CGT_HCR_NV1_nNV2),
+	SR_TRAP(SYS_SCXTNUM_EL1,	CGT_HCR_NV1_nNV2_ENSCXT),
 	SR_TRAP(SYS_SCXTNUM_EL0,	CGT_HCR_ENSCXT),
 	SR_TRAP(OP_AT_S1E1R, 		CGT_HCR_AT),
 	SR_TRAP(OP_AT_S1E1W, 		CGT_HCR_AT),

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 18/27] KVM: arm64: nv: Add trap forwarding for MDCR_EL2

[Marc Zyngier]

Hi Eric,

On Thu, 13 Jul 2023 18:34:39 +0100,
Eric Auger <eric.auger@redhat.com> wrote:
> 
> Hi Marc,
> 
> Please also double check DBGDTR_EL0, DBGDTRRX/TX_EL0. I understand from
> the spec they may end up in CGT_MDCR_TDCC_TDE_TDA too

Indeed they do. More traps! ;-)

[...]

> >  static DEFINE_XARRAY(sr_forward_xa);
> Besides
> 
> Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks!

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 06/27] arm64: Add debug registers affected by HDFGxTR_EL2

[Eric Auger]

Hi Marc,

On 7/12/23 16:57, Marc Zyngier wrote:
> The HDFGxTR_EL2 registers trap a (huge) set of debug and trace
> related registers. Add their encodings (and only that, because
> we really don't care about what these registers actually do at
> this stage).
>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  arch/arm64/include/asm/sysreg.h | 78 +++++++++++++++++++++++++++++++++
>  1 file changed, 78 insertions(+)
>
> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
> index 76289339b43b..9dfd127be55a 100644
> --- a/arch/arm64/include/asm/sysreg.h
> +++ b/arch/arm64/include/asm/sysreg.h
> @@ -194,6 +194,84 @@
>  #define SYS_DBGDTRTX_EL0		sys_reg(2, 3, 0, 5, 0)*
>  #define SYS_DBGVCR32_EL2		sys_reg(2, 4, 0, 7, 0)*
>  
> +#define SYS_BRBINF_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 0))*
> +#define SYS_BRBINFINJ_EL1		sys_reg(2, 1, 9, 1, 0)*
> +#define SYS_BRBSRC_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 1))*
> +#define SYS_BRBSRCINJ_EL1		sys_reg(2, 1, 9, 1, 1)*
> +#define SYS_BRBTGT_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 2))*
> +#define SYS_BRBTGTINJ_EL1		sys_reg(2, 1, 9, 1, 2)*
> +#define SYS_BRBTS_EL1			sys_reg(2, 1, 9, 0, 2)*
> +
> +#define SYS_BRBCR_EL1			sys_reg(2, 1, 9, 0, 0)*
> +#define SYS_BRBFCR_EL1			sys_reg(2, 1, 9, 0, 1)*
> +#define SYS_BRBIDR0_EL1			sys_reg(2, 1, 9, 2, 0)*
> +
> +#define SYS_TRCITECR_EL1		sys_reg(3, 0, 1, 2, 3)
> +#define SYS_TRCITECR_EL1		sys_reg(3, 0, 1, 2, 3)
I cannot find this one - which is duplicated by the way - in DDI0487Jaa
> +#define SYS_TRCACATR(m)			sys_reg(2, 1, 2, ((m & 7) << 1), (2 | (m >> 3)))*
> +#define SYS_TRCACVR(m)			sys_reg(2, 1, 2, ((m & 7) << 1), (0 | (m >> 3)))*
> +#define SYS_TRCAUTHSTATUS		sys_reg(2, 1, 7, 14, 6)*
> +#define SYS_TRCAUXCTLR			sys_reg(2, 1, 0, 6, 0)*
> +#define SYS_TRCBBCTLR			sys_reg(2, 1, 0, 15, 0)*
> +#define SYS_TRCCCCTLR			sys_reg(2, 1, 0, 14, 0)*
> +#define SYS_TRCCIDCCTLR0		sys_reg(2, 1, 3, 0, 2)*
> +#define SYS_TRCCIDCCTLR1		sys_reg(2, 1, 3, 1, 2)*
> +#define SYS_TRCCIDCVR(m)		sys_reg(2, 1, 3, ((m & 7) << 1), 0)*
> +#define SYS_TRCCLAIMCLR			sys_reg(2, 1, 7, 9, 6)*
> +#define SYS_TRCCLAIMSET			sys_reg(2, 1, 7, 8, 6)*
> +#define SYS_TRCCNTCTLR(m)		sys_reg(2, 1, 0, (4 | (m & 3)), 5)*
> +#define SYS_TRCCNTRLDVR(m)		sys_reg(2, 1, 0, (0 | (m & 3)), 5)*
> +#define SYS_TRCCNTVR(m)			sys_reg(2, 1, 0, (8 | (m & 3)), 5)*
> +#define SYS_TRCCONFIGR			sys_reg(2, 1, 0, 4, 0)*
> +#define SYS_TRCDEVARCH			sys_reg(2, 1, 7, 15, 6)*
> +#define SYS_TRCDEVID			sys_reg(2, 1, 7, 2, 7)*
> +#define SYS_TRCEVENTCTL0R		sys_reg(2, 1, 0, 8, 0)*
> +#define SYS_TRCEVENTCTL1R		sys_reg(2, 1, 0, 9, 0)*
> +#define SYS_TRCEXTINSELR(m)		sys_reg(2, 1, 0, (8 | (m & 3)), 4)*
> +#define SYS_TRCIDR0			sys_reg(2, 1, 0, 8, 7)*
> +#define SYS_TRCIDR10			sys_reg(2, 1, 0, 2, 6)*
> +#define SYS_TRCIDR11			sys_reg(2, 1, 0, 3, 6)*
> +#define SYS_TRCIDR12			sys_reg(2, 1, 0, 4, 6)*
> +#define SYS_TRCIDR13			sys_reg(2, 1, 0, 5, 6)*
> +#define SYS_TRCIDR1			sys_reg(2, 1, 0, 9, 7)*
> +#define SYS_TRCIDR2			sys_reg(2, 1, 0, 10, 7)*
> +#define SYS_TRCIDR3			sys_reg(2, 1, 0, 11, 7)*
> +#define SYS_TRCIDR4			sys_reg(2, 1, 0, 12, 7)*
> +#define SYS_TRCIDR5			sys_reg(2, 1, 0, 13, 7)*
> +#define SYS_TRCIDR6			sys_reg(2, 1, 0, 14, 7)*
> +#define SYS_TRCIDR7			sys_reg(2, 1, 0, 15, 7)*
> +#define SYS_TRCIDR8			sys_reg(2, 1, 0, 0, 6)*
> +#define SYS_TRCIDR9			sys_reg(2, 1, 0, 1, 6)*
> +#define SYS_TRCIMSPEC0			sys_reg(2, 1, 0, 0, 7)*
> +#define SYS_TRCIMSPEC(m)		sys_reg(2, 1, 0, (m & 7), 7)*
> +#define SYS_TRCITEEDCR			sys_reg(2, 1, 0, 2, 1)
I cannot find this one in D18-1 or elsewhere in DDI0487Jaa
> +#define SYS_TRCOSLSR			sys_reg(2, 1, 1, 1, 4)*
> +#define SYS_TRCPRGCTLR			sys_reg(2, 1, 0, 1, 0)*
> +#define SYS_TRCQCTLR			sys_reg(2, 1, 0, 1, 1)*
> +#define SYS_TRCRSCTLR(m)		sys_reg(2, 1, 1, (m & 15), (0 | (m >> 4)))*
> +#define SYS_TRCRSR			sys_reg(2, 1, 0, 10, 0)*
> +#define SYS_TRCSEQEVR(m)		sys_reg(2, 1, 0, (m & 3), 4)*
> +#define SYS_TRCSEQRSTEVR		sys_reg(2, 1, 0, 6, 4)*
> +#define SYS_TRCSEQSTR			sys_reg(2, 1, 0, 7, 4)*
> +#define SYS_TRCSSCCR(m)			sys_reg(2, 1, 1, (m & 7), 2)*
> +#define SYS_TRCSSCSR(m)			sys_reg(2, 1, 1, (8 | (m & 7)), 2)*
> +#define SYS_TRCSSPCICR(m)		sys_reg(2, 1, 1, (m & 7), 3)*
> +#define SYS_TRCSTALLCTLR		sys_reg(2, 1, 0, 11, 0)*
> +#define SYS_TRCSTATR			sys_reg(2, 1, 0, 3, 0)*
> +#define SYS_TRCSYNCPR			sys_reg(2, 1, 0, 13, 0)*
> +#define SYS_TRCTRACEIDR			sys_reg(2, 1, 0, 0, 1)*
> +#define SYS_TRCTSCTLR			sys_reg(2, 1, 0, 12, 0)*
> +#define SYS_TRCVICTLR			sys_reg(2, 1, 0, 0, 2)*
> +#define SYS_TRCVIIECTLR			sys_reg(2, 1, 0, 1, 2)*
> +#define SYS_TRCVIPCSSCTLR		sys_reg(2, 1, 0, 3, 2)*
> +#define SYS_TRCVISSCTLR			sys_reg(2, 1, 0, 2, 2)*
> +#define SYS_TRCVMIDCCTLR0		sys_reg(2, 1, 3, 2, 2)*
> +#define SYS_TRCVMIDCCTLR1		sys_reg(2, 1, 3, 3, 2)*
> +#define SYS_TRCVMIDCVR(m)		sys_reg(2, 1, 3, ((m & 7) << 1), 1)*
> +
> +/* ETM */
> +#define SYS_TRCOSLAR			sys_reg(2, 1, 1, 0, 4)
not able to locate this one either. I see the bit of HDFGWTR_EL2 though

Eric
> +
>  #define SYS_MIDR_EL1			sys_reg(3, 0, 0, 0, 0)
>  #define SYS_MPIDR_EL1			sys_reg(3, 0, 0, 0, 5)
>  #define SYS_REVIDR_EL1			sys_reg(3, 0, 0, 0, 6)

Re: [PATCH 16/27] KVM: arm64: nv: Add trap forwarding for HCR_EL2

[Eric Auger]

Hi Marc,

On 7/13/23 17:53, Marc Zyngier wrote:
> Hey Eric,
>
> Thanks for looking into this, much appreciated given how tedious it
> is.
>
> On Thu, 13 Jul 2023 15:05:33 +0100,
> Eric Auger <eric.auger@redhat.com> wrote:
>> Hi Marc,
>>
>> On 7/12/23 16:57, Marc Zyngier wrote:
>>> Describe the HCR_EL2 register, and associate it with all the sysregs
>>> it allows to trap.
>>>
>>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>>> ---
>>>  arch/arm64/kvm/emulate-nested.c | 475 ++++++++++++++++++++++++++++++++
>>>  1 file changed, 475 insertions(+)
>>>
>>> diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
>>> index 5bab2e85d70c..51901e85e43d 100644
>>> --- a/arch/arm64/kvm/emulate-nested.c
>>> +++ b/arch/arm64/kvm/emulate-nested.c
> [...]
>
>>> +	[CGT_HCR_TPC] = {
>> modern revisions now refer to TPCP, maybe worth a comment?
> Absolutely.
>
> [...]
>
>>> +	SR_RANGE_TRAP(SYS_ID_PFR0_EL1,
>>> +		      sys_reg(3, 0, 0, 7, 7), CGT_HCR_TID3),
>> in the spec I see this upper limit in the FEAT_FGT section. Out of
>> curiosity how were you able to convert the sys reg names into this Op0,
>> Op1, CRn, CRm, Op2. Is there any ordering logic documented somewhere for
>> those group3 regs?
> If you look at the sysreg encoding described on page D18-6308 if
> version J.a of the ARM ARM, you will find a block of 56 contiguous
> encodings ranging from (3, 0, 0, 1, 0), which happens to be
> ID_PFR0_EL1, all the way to a reserved range ending in (3, 0, 0, 7,
> 7).
>
> This is the block of register that is controlled by TID3.

OK thanks
>
>> I checked Table D18-2 and this looks good but I wonder if there isn't
>> any more efficient way to review this.
> Not that I know of, unfortunately. Even the pseudocode isn't enough
> for this as it doesn't described the trapping of unallocated regions.

OK
>
>>> +	SR_TRAP(SYS_ICC_SGI0R_EL1,	CGT_HCR_IMO_FMO),
>>> +	SR_TRAP(SYS_ICC_ASGI1R_EL1,	CGT_HCR_IMO_FMO),
>>> +	SR_TRAP(SYS_ICC_SGI1R_EL1,	CGT_HCR_IMO_FMO),
>>> +	SR_RANGE_TRAP(sys_reg(3, 0, 11, 0, 0),
>>> +		      sys_reg(3, 0, 11, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_RANGE_TRAP(sys_reg(3, 1, 11, 0, 0),
>>> +		      sys_reg(3, 1, 11, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_RANGE_TRAP(sys_reg(3, 2, 11, 0, 0),
>>> +		      sys_reg(3, 2, 11, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_RANGE_TRAP(sys_reg(3, 3, 11, 0, 0),
>>> +		      sys_reg(3, 3, 11, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_RANGE_TRAP(sys_reg(3, 4, 11, 0, 0),
>>> +		      sys_reg(3, 4, 11, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_RANGE_TRAP(sys_reg(3, 5, 11, 0, 0),
>>> +		      sys_reg(3, 5, 11, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_RANGE_TRAP(sys_reg(3, 6, 11, 0, 0),
>>> +		      sys_reg(3, 6, 11, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_RANGE_TRAP(sys_reg(3, 7, 11, 0, 0),
>>> +		      sys_reg(3, 7, 11, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_RANGE_TRAP(sys_reg(3, 0, 15, 0, 0),
>>> +		      sys_reg(3, 0, 15, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_RANGE_TRAP(sys_reg(3, 1, 15, 0, 0),
>>> +		      sys_reg(3, 1, 15, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_RANGE_TRAP(sys_reg(3, 2, 15, 0, 0),
>>> +		      sys_reg(3, 2, 15, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_RANGE_TRAP(sys_reg(3, 3, 15, 0, 0),
>>> +		      sys_reg(3, 3, 15, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_RANGE_TRAP(sys_reg(3, 4, 15, 0, 0),
>>> +		      sys_reg(3, 4, 15, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_RANGE_TRAP(sys_reg(3, 5, 15, 0, 0),
>>> +		      sys_reg(3, 5, 15, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_RANGE_TRAP(sys_reg(3, 6, 15, 0, 0),
>>> +		      sys_reg(3, 6, 15, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_RANGE_TRAP(sys_reg(3, 7, 15, 0, 0),
>>> +		      sys_reg(3, 7, 15, 15, 7), CGT_HCR_TIDCP),
>>> +	SR_TRAP(SYS_ACTLR_EL1,		CGT_HCR_TACR),
>>> +	SR_TRAP(SYS_DC_ISW,		CGT_HCR_TSW),
>>> +	SR_TRAP(SYS_DC_CSW,		CGT_HCR_TSW),
>>> +	SR_TRAP(SYS_DC_CISW,		CGT_HCR_TSW),
>>> +	SR_TRAP(SYS_DC_IGSW,		CGT_HCR_TSW),
>>> +	SR_TRAP(SYS_DC_IGDSW,		CGT_HCR_TSW),
>>> +	SR_TRAP(SYS_DC_CGSW,		CGT_HCR_TSW),
>>> +	SR_TRAP(SYS_DC_CGDSW,		CGT_HCR_TSW),
>>> +	SR_TRAP(SYS_DC_CIGSW,		CGT_HCR_TSW),
>>> +	SR_TRAP(SYS_DC_CIGDSW,		CGT_HCR_TSW),
>>> +	SR_TRAP(SYS_DC_CIVAC,		CGT_HCR_TPC),
>> I don't see CVADP?
> Me neither! Good catch!
>
> [...]
>
>>> +	SR_TRAP(SYS_SCTLR_EL1,		CGT_HCR_TVM_TRVM),
>>> +	SR_TRAP(SYS_TTBR0_EL1,		CGT_HCR_TVM_TRVM),
>>> +	SR_TRAP(SYS_TTBR1_EL1,		CGT_HCR_TVM_TRVM),
>>> +	SR_TRAP(SYS_TCR_EL1,		CGT_HCR_TVM_TRVM),
>>> +	SR_TRAP(SYS_ESR_EL1,		CGT_HCR_TVM_TRVM),
>>> +	SR_TRAP(SYS_FAR_EL1,		CGT_HCR_TVM_TRVM),
>>> +	SR_TRAP(SYS_AFSR0_EL1,		CGT_HCR_TVM_TRVM),*
>> Looking at the SFSR0_EL1 MRS/MSR pseudo code I understand TRVM is tested
>> on read and
>> TVM is tested on write. However CGT_HCR_TVM has FORWARD_ANY behaviour
>> while TRVM looks good as FORWARD_READ? Do I miss something.
> You're not missing anything. For some reason, I had in my head that
> TVM was trapping both reads and writes, while the spec is clear that
> it only traps writes.
>
>>> +	SR_TRAP(SYS_AFSR1_EL1,		CGT_HCR_TVM_TRVM),*
>> same here and below
> Yup, I need to fix the TVM encoding like this:
>
> @@ -176,7 +176,7 @@ static const struct trap_bits coarse_trap_bits[] = {
>  		.index		= HCR_EL2,
>  		.value		= HCR_TVM,
>  		.mask		= HCR_TVM,
> -		.behaviour	= BEHAVE_FORWARD_ANY,
> +		.behaviour	= BEHAVE_FORWARD_WRITE,
yes matches my understanding
>  	},
>  	[CGT_HCR_TDZ] = {
>  		.index		= HCR_EL2,
>
> [...]
>
>>> +	/* All _EL2 registers */
>>> +	SR_RANGE_TRAP(sys_reg(3, 4, 0, 0, 0),
>>> +		      sys_reg(3, 4, 10, 15, 7), CGT_HCR_NV),
>>> +	SR_RANGE_TRAP(sys_reg(3, 4, 12, 0, 0),
>>> +		      sys_reg(3, 4, 14, 15, 7), CGT_HCR_NV),
>>> +	/* All _EL02, _EL12 registers */
>>> +	SR_RANGE_TRAP(sys_reg(3, 5, 0, 0, 0),
>>> +		      sys_reg(3, 5, 10, 15, 7), CGT_HCR_NV),
>>> +	SR_RANGE_TRAP(sys_reg(3, 5, 12, 0, 0),
>>> +		      sys_reg(3, 5, 14, 15, 7), CGT_HCR_NV),
>> same question as bove, where in the ARM ARM do you find those
>> ranges?
> I went over the encoding with a fine comb, and realised that all the
> (3, 4, ...) encodings are EL2, and all the (3, 5, ...) ones are EL02
> and EL12.
Oh good catch
>
> I appreciate that this is taking a massive bet on the future, but
> there is no such rule in the ARM ARM as such...

yeah that's unfortunate that rule is not stated anywhere
>
>>> +	SR_TRAP(SYS_SP_EL1,		CGT_HCR_NV),*
>>> +	SR_TRAP(OP_AT_S1E2R,		CGT_HCR_NV),*
>>> +	SR_TRAP(OP_AT_S1E2W,		CGT_HCR_NV),*
>>> +	SR_TRAP(OP_AT_S12E1R,		CGT_HCR_NV),*
>>> +	SR_TRAP(OP_AT_S12E1W,		CGT_HCR_NV),*
>>> +	SR_TRAP(OP_AT_S12E0R,		CGT_HCR_NV),*
>> according to the pseudo code NV2 is not checked
>> shouldn't we have a separate CGT? Question also valid for a bunch of ops
>> below
> Hmmm. Yes, this is wrong. Well spotted. I guess I need a
> CGT_HCR_NV_nNV2 for the cases that want that particular condition (NV1
> probably needs a similar fix).
NV1 looks good. There NV2 is checked
>
> [...]
>
>> CIGDPAE?
>> CIPAE?
> These two are part of RME (well, technically MEC, but that an RME
> thing), and I have no plan to support this with NV -- yet.
OK
>
>> CFP/CPP/DVP RCTX?
> These are definitely missing. I'll add them.
>
> Thanks again for going through this list, this is awesome work!
you are welcome. This is cumbersome to review but less than writing it I
suspect ;-)

Eric
>
> 	M.
>

Re: [PATCH 16/27] KVM: arm64: nv: Add trap forwarding for HCR_EL2

[Eric Auger]

Hi Marc,

On 7/14/23 12:10, Marc Zyngier wrote:
> On Thu, 13 Jul 2023 16:53:40 +0100,
> Marc Zyngier <maz@kernel.org> wrote:
>> Hey Eric,
>>
>> Thanks for looking into this, much appreciated given how tedious it
>> is.
> FWIW, here are the changes I'm going to squash in that patch. Shout if
> you spot something that looks odd...
>
> Thanks,
>
> 	M.
>
> diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
> index c4057f4ff72d..f5978b463aca 100644
> --- a/arch/arm64/kvm/emulate-nested.c
> +++ b/arch/arm64/kvm/emulate-nested.c
> @@ -55,7 +55,8 @@ enum coarse_grain_trap_id {
>  	CGT_HCR_TERR,
>  	CGT_HCR_APK,
>  	CGT_HCR_NV,
> -	CGT_HCR_NV1,
> +	CGT_HCR_NV_nNV2,
> +	CGT_HCR_NV1_nNV2,
>  	CGT_HCR_AT,
>  	CGT_HCR_FIEN,
>  	CGT_HCR_TID4,
> @@ -89,7 +90,7 @@ enum coarse_grain_trap_id {
>  	CGT_HCR_TVM_TRVM,
>  	CGT_HCR_TPU_TICAB,
>  	CGT_HCR_TPU_TOCU,
> -	CGT_HCR_NV1_ENSCXT,
> +	CGT_HCR_NV1_nNV2_ENSCXT,
>  	CGT_MDCR_TPM_TPMCR,
>  	CGT_MDCR_TDE_TDA,
>  	CGT_MDCR_TDE_TDOSA,
> @@ -154,7 +155,7 @@ static const struct trap_bits coarse_trap_bits[] = {
>  		.mask		= HCR_TSW,
>  		.behaviour	= BEHAVE_FORWARD_ANY,
>  	},
> -	[CGT_HCR_TPC] = {
> +	[CGT_HCR_TPC] = { /* Also called TCPC when FEAT_DPB is implemented */
>  		.index		= HCR_EL2,
>  		.value		= HCR_TPC,
>  		.mask		= HCR_TPC,
> @@ -176,7 +177,7 @@ static const struct trap_bits coarse_trap_bits[] = {
>  		.index		= HCR_EL2,
>  		.value		= HCR_TVM,
>  		.mask		= HCR_TVM,
> -		.behaviour	= BEHAVE_FORWARD_ANY,
> +		.behaviour	= BEHAVE_FORWARD_WRITE,
>  	},
>  	[CGT_HCR_TDZ] = {
>  		.index		= HCR_EL2,
> @@ -209,12 +210,18 @@ static const struct trap_bits coarse_trap_bits[] = {
>  		.behaviour	= BEHAVE_FORWARD_ANY,
>  	},
>  	[CGT_HCR_NV] = {
> +		.index		= HCR_EL2,
> +		.value		= HCR_NV,
> +		.mask		= HCR_NV,
> +		.behaviour	= BEHAVE_FORWARD_ANY,
> +	},
> +	[CGT_HCR_NV_nNV2] = {
>  		.index		= HCR_EL2,
>  		.value		= HCR_NV,
>  		.mask		= HCR_NV | HCR_NV2,
>  		.behaviour	= BEHAVE_FORWARD_ANY,
>  	},
> -	[CGT_HCR_NV1] = {
> +	[CGT_HCR_NV1_nNV2] = {
>  		.index		= HCR_EL2,
>  		.value		= HCR_NV | HCR_NV1,
>  		.mask		= HCR_NV | HCR_NV1 | HCR_NV2,
> @@ -350,7 +357,7 @@ static const enum coarse_grain_trap_id *coarse_control_combo[] = {
>  	MCB(CGT_HCR_TVM_TRVM,		CGT_HCR_TVM, CGT_HCR_TRVM),
>  	MCB(CGT_HCR_TPU_TICAB,		CGT_HCR_TPU, CGT_HCR_TICAB),
>  	MCB(CGT_HCR_TPU_TOCU,		CGT_HCR_TPU, CGT_HCR_TOCU),
> -	MCB(CGT_HCR_NV1_ENSCXT,		CGT_HCR_NV1, CGT_HCR_ENSCXT),
> +	MCB(CGT_HCR_NV1_nNV2_ENSCXT,	CGT_HCR_NV1_nNV2, CGT_HCR_ENSCXT),
>  	MCB(CGT_MDCR_TPM_TPMCR,		CGT_MDCR_TPM, CGT_MDCR_TPMCR),
>  	MCB(CGT_MDCR_TDE_TDA,		CGT_MDCR_TDE, CGT_MDCR_TDA),
>  	MCB(CGT_MDCR_TDE_TDOSA,		CGT_MDCR_TDE, CGT_MDCR_TDOSA),
> @@ -501,6 +508,7 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initdata = {
>  	SR_TRAP(SYS_DC_CIVAC,		CGT_HCR_TPC),
>  	SR_TRAP(SYS_DC_CVAC,		CGT_HCR_TPC),
>  	SR_TRAP(SYS_DC_CVAP,		CGT_HCR_TPC),
> +	SR_TRAP(SYS_DC_CVADP,		CGT_HCR_TPC),
>  	SR_TRAP(SYS_DC_IVAC,		CGT_HCR_TPC),
>  	SR_TRAP(SYS_DC_CIGVAC,		CGT_HCR_TPC),
>  	SR_TRAP(SYS_DC_CIGDVAC,		CGT_HCR_TPC),
> @@ -625,7 +633,6 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initdata = {
>  		      sys_reg(3, 5, 10, 15, 7), CGT_HCR_NV),
>  	SR_RANGE_TRAP(sys_reg(3, 5, 12, 0, 0),
>  		      sys_reg(3, 5, 14, 15, 7), CGT_HCR_NV),
> -	SR_TRAP(SYS_SP_EL1,		CGT_HCR_NV),
>  	SR_TRAP(OP_AT_S1E2R,		CGT_HCR_NV),
>  	SR_TRAP(OP_AT_S1E2W,		CGT_HCR_NV),
>  	SR_TRAP(OP_AT_S12E1R,		CGT_HCR_NV),
> @@ -698,10 +705,14 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initdata = {
>  	SR_TRAP(OP_TLBI_RIPAS2LE1OSNXS,	CGT_HCR_NV),
>  	SR_TRAP(OP_TLBI_RVAE2OSNXS,	CGT_HCR_NV),
>  	SR_TRAP(OP_TLBI_RVALE2OSNXS,	CGT_HCR_NV),
> -	SR_TRAP(SYS_VBAR_EL1,		CGT_HCR_NV1),
> -	SR_TRAP(SYS_ELR_EL1,		CGT_HCR_NV1),
> -	SR_TRAP(SYS_SPSR_EL1,		CGT_HCR_NV1),
> -	SR_TRAP(SYS_SCXTNUM_EL1,	CGT_HCR_NV1_ENSCXT),
> +	SR_TRAP(OP_CPP_RCTX, 		CGT_HCR_NV),
> +	SR_TRAP(OP_DVP_RCTX, 		CGT_HCR_NV),
> +	SR_TRAP(OP_CFP_RCTX, 		CGT_HCR_NV),
> +	SR_TRAP(SYS_SP_EL1,		CGT_HCR_NV_nNV2),
> +	SR_TRAP(SYS_VBAR_EL1,		CGT_HCR_NV1_nNV2),
> +	SR_TRAP(SYS_ELR_EL1,		CGT_HCR_NV1_nNV2),
> +	SR_TRAP(SYS_SPSR_EL1,		CGT_HCR_NV1_nNV2),
> +	SR_TRAP(SYS_SCXTNUM_EL1,	CGT_HCR_NV1_nNV2_ENSCXT),
>  	SR_TRAP(SYS_SCXTNUM_EL0,	CGT_HCR_ENSCXT),
>  	SR_TRAP(OP_AT_S1E1R, 		CGT_HCR_AT),
>  	SR_TRAP(OP_AT_S1E1W, 		CGT_HCR_AT),
>
Looks good to me. Feel free to add my
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric

Re: [PATCH 06/27] arm64: Add debug registers affected by HDFGxTR_EL2

[Marc Zyngier]

Hey Eric,

On Fri, 14 Jul 2023 15:47:20 +0100,
Eric Auger <eric.auger@redhat.com> wrote:
> 
> Hi Marc,
> 
> On 7/12/23 16:57, Marc Zyngier wrote:
> > The HDFGxTR_EL2 registers trap a (huge) set of debug and trace
> > related registers. Add their encodings (and only that, because
> > we really don't care about what these registers actually do at
> > this stage).
> >
> > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > ---
> >  arch/arm64/include/asm/sysreg.h | 78 +++++++++++++++++++++++++++++++++
> >  1 file changed, 78 insertions(+)
> >
> > diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
> > index 76289339b43b..9dfd127be55a 100644
> > --- a/arch/arm64/include/asm/sysreg.h
> > +++ b/arch/arm64/include/asm/sysreg.h
> > @@ -194,6 +194,84 @@
> >  #define SYS_DBGDTRTX_EL0		sys_reg(2, 3, 0, 5, 0)*
> >  #define SYS_DBGVCR32_EL2		sys_reg(2, 4, 0, 7, 0)*
> >  
> > +#define SYS_BRBINF_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 0))*
> > +#define SYS_BRBINFINJ_EL1		sys_reg(2, 1, 9, 1, 0)*
> > +#define SYS_BRBSRC_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 1))*
> > +#define SYS_BRBSRCINJ_EL1		sys_reg(2, 1, 9, 1, 1)*
> > +#define SYS_BRBTGT_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 2))*
> > +#define SYS_BRBTGTINJ_EL1		sys_reg(2, 1, 9, 1, 2)*
> > +#define SYS_BRBTS_EL1			sys_reg(2, 1, 9, 0, 2)*
> > +
> > +#define SYS_BRBCR_EL1			sys_reg(2, 1, 9, 0, 0)*
> > +#define SYS_BRBFCR_EL1			sys_reg(2, 1, 9, 0, 1)*
> > +#define SYS_BRBIDR0_EL1			sys_reg(2, 1, 9, 2, 0)*
> > +
> > +#define SYS_TRCITECR_EL1		sys_reg(3, 0, 1, 2, 3)
> > +#define SYS_TRCITECR_EL1		sys_reg(3, 0, 1, 2, 3)
> I cannot find this one - which is duplicated by the way - in DDI0487Jaa

Ah, that's one of the sucker I got from peeking at the 2023-03 XML and
wrote it twice for a good measure. You can see it there:

https://developer.arm.com/documentation/ddi0601/2023-03/AArch64-Registers/TRCITECR-EL1--Instrumentation-Trace-Control-Register--EL1-

> > +#define SYS_TRCACATR(m)			sys_reg(2, 1, 2, ((m & 7) << 1), (2 | (m >> 3)))*
> > +#define SYS_TRCACVR(m)			sys_reg(2, 1, 2, ((m & 7) << 1), (0 | (m >> 3)))*
> > +#define SYS_TRCAUTHSTATUS		sys_reg(2, 1, 7, 14, 6)*
> > +#define SYS_TRCAUXCTLR			sys_reg(2, 1, 0, 6, 0)*
> > +#define SYS_TRCBBCTLR			sys_reg(2, 1, 0, 15, 0)*
> > +#define SYS_TRCCCCTLR			sys_reg(2, 1, 0, 14, 0)*
> > +#define SYS_TRCCIDCCTLR0		sys_reg(2, 1, 3, 0, 2)*
> > +#define SYS_TRCCIDCCTLR1		sys_reg(2, 1, 3, 1, 2)*
> > +#define SYS_TRCCIDCVR(m)		sys_reg(2, 1, 3, ((m & 7) << 1), 0)*
> > +#define SYS_TRCCLAIMCLR			sys_reg(2, 1, 7, 9, 6)*
> > +#define SYS_TRCCLAIMSET			sys_reg(2, 1, 7, 8, 6)*
> > +#define SYS_TRCCNTCTLR(m)		sys_reg(2, 1, 0, (4 | (m & 3)), 5)*
> > +#define SYS_TRCCNTRLDVR(m)		sys_reg(2, 1, 0, (0 | (m & 3)), 5)*
> > +#define SYS_TRCCNTVR(m)			sys_reg(2, 1, 0, (8 | (m & 3)), 5)*
> > +#define SYS_TRCCONFIGR			sys_reg(2, 1, 0, 4, 0)*
> > +#define SYS_TRCDEVARCH			sys_reg(2, 1, 7, 15, 6)*
> > +#define SYS_TRCDEVID			sys_reg(2, 1, 7, 2, 7)*
> > +#define SYS_TRCEVENTCTL0R		sys_reg(2, 1, 0, 8, 0)*
> > +#define SYS_TRCEVENTCTL1R		sys_reg(2, 1, 0, 9, 0)*
> > +#define SYS_TRCEXTINSELR(m)		sys_reg(2, 1, 0, (8 | (m & 3)), 4)*
> > +#define SYS_TRCIDR0			sys_reg(2, 1, 0, 8, 7)*
> > +#define SYS_TRCIDR10			sys_reg(2, 1, 0, 2, 6)*
> > +#define SYS_TRCIDR11			sys_reg(2, 1, 0, 3, 6)*
> > +#define SYS_TRCIDR12			sys_reg(2, 1, 0, 4, 6)*
> > +#define SYS_TRCIDR13			sys_reg(2, 1, 0, 5, 6)*
> > +#define SYS_TRCIDR1			sys_reg(2, 1, 0, 9, 7)*
> > +#define SYS_TRCIDR2			sys_reg(2, 1, 0, 10, 7)*
> > +#define SYS_TRCIDR3			sys_reg(2, 1, 0, 11, 7)*
> > +#define SYS_TRCIDR4			sys_reg(2, 1, 0, 12, 7)*
> > +#define SYS_TRCIDR5			sys_reg(2, 1, 0, 13, 7)*
> > +#define SYS_TRCIDR6			sys_reg(2, 1, 0, 14, 7)*
> > +#define SYS_TRCIDR7			sys_reg(2, 1, 0, 15, 7)*
> > +#define SYS_TRCIDR8			sys_reg(2, 1, 0, 0, 6)*
> > +#define SYS_TRCIDR9			sys_reg(2, 1, 0, 1, 6)*
> > +#define SYS_TRCIMSPEC0			sys_reg(2, 1, 0, 0, 7)*
> > +#define SYS_TRCIMSPEC(m)		sys_reg(2, 1, 0, (m & 7), 7)*
> > +#define SYS_TRCITEEDCR			sys_reg(2, 1, 0, 2, 1)
> I cannot find this one in D18-1 or elsewhere in DDI0487Jaa

Same thing. You can find it here:

https://developer.arm.com/documentation/ddi0601/2023-03/AArch64-Registers/TRCITEEDCR--Instrumentation-Trace-Extension-External-Debug-Control-Register

> > +#define SYS_TRCOSLSR			sys_reg(2, 1, 1, 1, 4)*
> > +#define SYS_TRCPRGCTLR			sys_reg(2, 1, 0, 1, 0)*
> > +#define SYS_TRCQCTLR			sys_reg(2, 1, 0, 1, 1)*
> > +#define SYS_TRCRSCTLR(m)		sys_reg(2, 1, 1, (m & 15), (0 | (m >> 4)))*
> > +#define SYS_TRCRSR			sys_reg(2, 1, 0, 10, 0)*
> > +#define SYS_TRCSEQEVR(m)		sys_reg(2, 1, 0, (m & 3), 4)*
> > +#define SYS_TRCSEQRSTEVR		sys_reg(2, 1, 0, 6, 4)*
> > +#define SYS_TRCSEQSTR			sys_reg(2, 1, 0, 7, 4)*
> > +#define SYS_TRCSSCCR(m)			sys_reg(2, 1, 1, (m & 7), 2)*
> > +#define SYS_TRCSSCSR(m)			sys_reg(2, 1, 1, (8 | (m & 7)), 2)*
> > +#define SYS_TRCSSPCICR(m)		sys_reg(2, 1, 1, (m & 7), 3)*
> > +#define SYS_TRCSTALLCTLR		sys_reg(2, 1, 0, 11, 0)*
> > +#define SYS_TRCSTATR			sys_reg(2, 1, 0, 3, 0)*
> > +#define SYS_TRCSYNCPR			sys_reg(2, 1, 0, 13, 0)*
> > +#define SYS_TRCTRACEIDR			sys_reg(2, 1, 0, 0, 1)*
> > +#define SYS_TRCTSCTLR			sys_reg(2, 1, 0, 12, 0)*
> > +#define SYS_TRCVICTLR			sys_reg(2, 1, 0, 0, 2)*
> > +#define SYS_TRCVIIECTLR			sys_reg(2, 1, 0, 1, 2)*
> > +#define SYS_TRCVIPCSSCTLR		sys_reg(2, 1, 0, 3, 2)*
> > +#define SYS_TRCVISSCTLR			sys_reg(2, 1, 0, 2, 2)*
> > +#define SYS_TRCVMIDCCTLR0		sys_reg(2, 1, 3, 2, 2)*
> > +#define SYS_TRCVMIDCCTLR1		sys_reg(2, 1, 3, 3, 2)*
> > +#define SYS_TRCVMIDCVR(m)		sys_reg(2, 1, 3, ((m & 7) << 1), 1)*
> > +
> > +/* ETM */
> > +#define SYS_TRCOSLAR			sys_reg(2, 1, 1, 0, 4)
> not able to locate this one either. I see the bit of HDFGWTR_EL2 though

This one lives in the ETM spec:

https://documentation-service.arm.com/static/60017fbb3f22832ff1d6872b

Page 7-342 has the register number, and the encoding is computed as
per the formula in 4.3.6 "System instructions", page 4-169.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 16/27] KVM: arm64: nv: Add trap forwarding for HCR_EL2

[Marc Zyngier]

On Fri, 14 Jul 2023 16:06:18 +0100,
Eric Auger <eric.auger@redhat.com> wrote:
> 
> Hi Marc,
> 
> On 7/14/23 12:10, Marc Zyngier wrote:
> > On Thu, 13 Jul 2023 16:53:40 +0100,
> > Marc Zyngier <maz@kernel.org> wrote:
> >> Hey Eric,
> >>
> >> Thanks for looking into this, much appreciated given how tedious it
> >> is.
> > FWIW, here are the changes I'm going to squash in that patch. Shout if
> > you spot something that looks odd...

[...]

> Looks good to me. Feel free to add my
> Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thank you!

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 07/27] arm64: Add missing BRB/CFP/DVP/CPP instructions

[Miguel Luis]

Hi Marc,

> On 12 Jul 2023, at 14:57, Marc Zyngier <maz@kernel.org> wrote:
> 
> HFGITR_EL2 traps a bunch of instructions for which we don't have
> encodings yet. Add them.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
> arch/arm64/include/asm/sysreg.h | 7 +++++++
> 1 file changed, 7 insertions(+)
> 
> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
> index 9dfd127be55a..e2357529c633 100644
> --- a/arch/arm64/include/asm/sysreg.h
> +++ b/arch/arm64/include/asm/sysreg.h
> @@ -737,6 +737,13 @@
> #define OP_TLBI_VALE2NXS sys_insn(1, 4, 9, 7, 5)
> #define OP_TLBI_VMALLS12E1NXS sys_insn(1, 4, 9, 7, 6)
> 
> +/* Misc instructions */
> +#define OP_BRB_IALL sys_insn(1, 1, 7, 2, 4)
> +#define OP_BRB_INJ sys_insn(1, 1, 7, 2, 5)
> +#define OP_CFP_RCTX sys_insn(1, 3, 7, 3, 4)
> +#define OP_DVP_RCTX sys_insn(1, 3, 7, 3, 5)
> +#define OP_CPP_RCTX sys_insn(1, 3, 7, 3, 7)
> +

As documented in DDI0487J.a

Reviewed-by: Miguel Luis <miguel.luis@oracle.com>

Miguel

> /* Common SCTLR_ELx flags. */
> #define SCTLR_ELx_ENTP2 (BIT(60))
> #define SCTLR_ELx_DSSBS (BIT(44))
> -- 
> 2.34.1
> 

Re: [PATCH 06/27] arm64: Add debug registers affected by HDFGxTR_EL2

[Suzuki K Poulose]

On 12/07/2023 15:57, Marc Zyngier wrote:
> The HDFGxTR_EL2 registers trap a (huge) set of debug and trace
> related registers. Add their encodings (and only that, because
> we really don't care about what these registers actually do at
> this stage).
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>   arch/arm64/include/asm/sysreg.h | 78 +++++++++++++++++++++++++++++++++
>   1 file changed, 78 insertions(+)
> 
> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
> index 76289339b43b..9dfd127be55a 100644
> --- a/arch/arm64/include/asm/sysreg.h
> +++ b/arch/arm64/include/asm/sysreg.h
> @@ -194,6 +194,84 @@
>   #define SYS_DBGDTRTX_EL0		sys_reg(2, 3, 0, 5, 0)
>   #define SYS_DBGVCR32_EL2		sys_reg(2, 4, 0, 7, 0)
>   
> +#define SYS_BRBINF_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 0))
> +#define SYS_BRBINFINJ_EL1		sys_reg(2, 1, 9, 1, 0)
> +#define SYS_BRBSRC_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 1))
> +#define SYS_BRBSRCINJ_EL1		sys_reg(2, 1, 9, 1, 1)
> +#define SYS_BRBTGT_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 2))
> +#define SYS_BRBTGTINJ_EL1		sys_reg(2, 1, 9, 1, 2)
> +#define SYS_BRBTS_EL1			sys_reg(2, 1, 9, 0, 2)
> +
> +#define SYS_BRBCR_EL1			sys_reg(2, 1, 9, 0, 0)
> +#define SYS_BRBFCR_EL1			sys_reg(2, 1, 9, 0, 1)
> +#define SYS_BRBIDR0_EL1			sys_reg(2, 1, 9, 2, 0)
> +
> +#define SYS_TRCITECR_EL1		sys_reg(3, 0, 1, 2, 3)
> +#define SYS_TRCITECR_EL1		sys_reg(3, 0, 1, 2, 3)
> +#define SYS_TRCACATR(m)			sys_reg(2, 1, 2, ((m & 7) << 1), (2 | (m >> 3)))
> +#define SYS_TRCACVR(m)			sys_reg(2, 1, 2, ((m & 7) << 1), (0 | (m >> 3)))
> +#define SYS_TRCAUTHSTATUS		sys_reg(2, 1, 7, 14, 6)
> +#define SYS_TRCAUXCTLR			sys_reg(2, 1, 0, 6, 0)
> +#define SYS_TRCBBCTLR			sys_reg(2, 1, 0, 15, 0)
> +#define SYS_TRCCCCTLR			sys_reg(2, 1, 0, 14, 0)
> +#define SYS_TRCCIDCCTLR0		sys_reg(2, 1, 3, 0, 2)
> +#define SYS_TRCCIDCCTLR1		sys_reg(2, 1, 3, 1, 2)
> +#define SYS_TRCCIDCVR(m)		sys_reg(2, 1, 3, ((m & 7) << 1), 0)
> +#define SYS_TRCCLAIMCLR			sys_reg(2, 1, 7, 9, 6)
> +#define SYS_TRCCLAIMSET			sys_reg(2, 1, 7, 8, 6)
> +#define SYS_TRCCNTCTLR(m)		sys_reg(2, 1, 0, (4 | (m & 3)), 5)
> +#define SYS_TRCCNTRLDVR(m)		sys_reg(2, 1, 0, (0 | (m & 3)), 5)
> +#define SYS_TRCCNTVR(m)			sys_reg(2, 1, 0, (8 | (m & 3)), 5)
> +#define SYS_TRCCONFIGR			sys_reg(2, 1, 0, 4, 0)
> +#define SYS_TRCDEVARCH			sys_reg(2, 1, 7, 15, 6)
> +#define SYS_TRCDEVID			sys_reg(2, 1, 7, 2, 7)
> +#define SYS_TRCEVENTCTL0R		sys_reg(2, 1, 0, 8, 0)
> +#define SYS_TRCEVENTCTL1R		sys_reg(2, 1, 0, 9, 0)
> +#define SYS_TRCEXTINSELR(m)		sys_reg(2, 1, 0, (8 | (m & 3)), 4)
> +#define SYS_TRCIDR0			sys_reg(2, 1, 0, 8, 7)
> +#define SYS_TRCIDR10			sys_reg(2, 1, 0, 2, 6)
> +#define SYS_TRCIDR11			sys_reg(2, 1, 0, 3, 6)
> +#define SYS_TRCIDR12			sys_reg(2, 1, 0, 4, 6)
> +#define SYS_TRCIDR13			sys_reg(2, 1, 0, 5, 6)
> +#define SYS_TRCIDR1			sys_reg(2, 1, 0, 9, 7)
> +#define SYS_TRCIDR2			sys_reg(2, 1, 0, 10, 7)
> +#define SYS_TRCIDR3			sys_reg(2, 1, 0, 11, 7)
> +#define SYS_TRCIDR4			sys_reg(2, 1, 0, 12, 7)
> +#define SYS_TRCIDR5			sys_reg(2, 1, 0, 13, 7)
> +#define SYS_TRCIDR6			sys_reg(2, 1, 0, 14, 7)
> +#define SYS_TRCIDR7			sys_reg(2, 1, 0, 15, 7)
> +#define SYS_TRCIDR8			sys_reg(2, 1, 0, 0, 6)
> +#define SYS_TRCIDR9			sys_reg(2, 1, 0, 1, 6)
> +#define SYS_TRCIMSPEC0			sys_reg(2, 1, 0, 0, 7)
> +#define SYS_TRCIMSPEC(m)		sys_reg(2, 1, 0, (m & 7), 7)
> +#define SYS_TRCITEEDCR			sys_reg(2, 1, 0, 2, 1)
> +#define SYS_TRCOSLSR			sys_reg(2, 1, 1, 1, 4)
> +#define SYS_TRCPRGCTLR			sys_reg(2, 1, 0, 1, 0)
> +#define SYS_TRCQCTLR			sys_reg(2, 1, 0, 1, 1)
> +#define SYS_TRCRSCTLR(m)		sys_reg(2, 1, 1, (m & 15), (0 | (m >> 4)))
> +#define SYS_TRCRSR			sys_reg(2, 1, 0, 10, 0)
> +#define SYS_TRCSEQEVR(m)		sys_reg(2, 1, 0, (m & 3), 4)
> +#define SYS_TRCSEQRSTEVR		sys_reg(2, 1, 0, 6, 4)
> +#define SYS_TRCSEQSTR			sys_reg(2, 1, 0, 7, 4)
> +#define SYS_TRCSSCCR(m)			sys_reg(2, 1, 1, (m & 7), 2)
> +#define SYS_TRCSSCSR(m)			sys_reg(2, 1, 1, (8 | (m & 7)), 2)
> +#define SYS_TRCSSPCICR(m)		sys_reg(2, 1, 1, (m & 7), 3)
> +#define SYS_TRCSTALLCTLR		sys_reg(2, 1, 0, 11, 0)
> +#define SYS_TRCSTATR			sys_reg(2, 1, 0, 3, 0)
> +#define SYS_TRCSYNCPR			sys_reg(2, 1, 0, 13, 0)
> +#define SYS_TRCTRACEIDR			sys_reg(2, 1, 0, 0, 1)
> +#define SYS_TRCTSCTLR			sys_reg(2, 1, 0, 12, 0)
> +#define SYS_TRCVICTLR			sys_reg(2, 1, 0, 0, 2)
> +#define SYS_TRCVIIECTLR			sys_reg(2, 1, 0, 1, 2)
> +#define SYS_TRCVIPCSSCTLR		sys_reg(2, 1, 0, 3, 2)
> +#define SYS_TRCVISSCTLR			sys_reg(2, 1, 0, 2, 2)
> +#define SYS_TRCVMIDCCTLR0		sys_reg(2, 1, 3, 2, 2)
> +#define SYS_TRCVMIDCCTLR1		sys_reg(2, 1, 3, 3, 2)
> +#define SYS_TRCVMIDCVR(m)		sys_reg(2, 1, 3, ((m & 7) << 1), 1)
> +
> +/* ETM */
> +#define SYS_TRCOSLAR			sys_reg(2, 1, 1, 0, 4)

For all ETM Trace related registers. I compared them with the list
we have in ETM/ETE driver and used a script to generate sys_reg encoding
from the macros we have (which consumes an MMIO offset). The list
covers the registers accessible only via system instructions
and the encodings match.

Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>

Re: [PATCH 06/27] arm64: Add debug registers affected by HDFGxTR_EL2

[Suzuki K Poulose]

Hi Marc

On 12/07/2023 15:57, Marc Zyngier wrote:
> The HDFGxTR_EL2 registers trap a (huge) set of debug and trace
> related registers. Add their encodings (and only that, because
> we really don't care about what these registers actually do at
> this stage).
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>   arch/arm64/include/asm/sysreg.h | 78 +++++++++++++++++++++++++++++++++
>   1 file changed, 78 insertions(+)
> 
> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
> index 76289339b43b..9dfd127be55a 100644
> --- a/arch/arm64/include/asm/sysreg.h
> +++ b/arch/arm64/include/asm/sysreg.h
> @@ -194,6 +194,84 @@
>   #define SYS_DBGDTRTX_EL0		sys_reg(2, 3, 0, 5, 0)
>   #define SYS_DBGVCR32_EL2		sys_reg(2, 4, 0, 7, 0)
>   
> +#define SYS_BRBINF_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 0))
> +#define SYS_BRBINFINJ_EL1		sys_reg(2, 1, 9, 1, 0)
> +#define SYS_BRBSRC_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 1))
> +#define SYS_BRBSRCINJ_EL1		sys_reg(2, 1, 9, 1, 1)
> +#define SYS_BRBTGT_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 2))
> +#define SYS_BRBTGTINJ_EL1		sys_reg(2, 1, 9, 1, 2)
> +#define SYS_BRBTS_EL1			sys_reg(2, 1, 9, 0, 2)
> +
> +#define SYS_BRBCR_EL1			sys_reg(2, 1, 9, 0, 0)
> +#define SYS_BRBFCR_EL1			sys_reg(2, 1, 9, 0, 1)
> +#define SYS_BRBIDR0_EL1			sys_reg(2, 1, 9, 2, 0)
> +

Merge conflict alert. The above are being added as part of the BRBE 
support series from Anshuman [0], though in a slightly different scheme
for registers with numbers. e.g, SYS_BRBINF_EL1(0) vs SYS_BRBINF0_EL1.
That series is not merged yet, but might go in this cycle.

[0] 
https://lkml.kernel.org/r/20230711082455.215983-1-anshuman.khandual@arm.com

Suzuki



> +#define SYS_TRCITECR_EL1		sys_reg(3, 0, 1, 2, 3)
> +#define SYS_TRCITECR_EL1		sys_reg(3, 0, 1, 2, 3)
> +#define SYS_TRCACATR(m)			sys_reg(2, 1, 2, ((m & 7) << 1), (2 | (m >> 3)))
> +#define SYS_TRCACVR(m)			sys_reg(2, 1, 2, ((m & 7) << 1), (0 | (m >> 3)))
> +#define SYS_TRCAUTHSTATUS		sys_reg(2, 1, 7, 14, 6)
> +#define SYS_TRCAUXCTLR			sys_reg(2, 1, 0, 6, 0)
> +#define SYS_TRCBBCTLR			sys_reg(2, 1, 0, 15, 0)
> +#define SYS_TRCCCCTLR			sys_reg(2, 1, 0, 14, 0)
> +#define SYS_TRCCIDCCTLR0		sys_reg(2, 1, 3, 0, 2)
> +#define SYS_TRCCIDCCTLR1		sys_reg(2, 1, 3, 1, 2)
> +#define SYS_TRCCIDCVR(m)		sys_reg(2, 1, 3, ((m & 7) << 1), 0)
> +#define SYS_TRCCLAIMCLR			sys_reg(2, 1, 7, 9, 6)
> +#define SYS_TRCCLAIMSET			sys_reg(2, 1, 7, 8, 6)
> +#define SYS_TRCCNTCTLR(m)		sys_reg(2, 1, 0, (4 | (m & 3)), 5)
> +#define SYS_TRCCNTRLDVR(m)		sys_reg(2, 1, 0, (0 | (m & 3)), 5)
> +#define SYS_TRCCNTVR(m)			sys_reg(2, 1, 0, (8 | (m & 3)), 5)
> +#define SYS_TRCCONFIGR			sys_reg(2, 1, 0, 4, 0)
> +#define SYS_TRCDEVARCH			sys_reg(2, 1, 7, 15, 6)
> +#define SYS_TRCDEVID			sys_reg(2, 1, 7, 2, 7)
> +#define SYS_TRCEVENTCTL0R		sys_reg(2, 1, 0, 8, 0)
> +#define SYS_TRCEVENTCTL1R		sys_reg(2, 1, 0, 9, 0)
> +#define SYS_TRCEXTINSELR(m)		sys_reg(2, 1, 0, (8 | (m & 3)), 4)
> +#define SYS_TRCIDR0			sys_reg(2, 1, 0, 8, 7)
> +#define SYS_TRCIDR10			sys_reg(2, 1, 0, 2, 6)
> +#define SYS_TRCIDR11			sys_reg(2, 1, 0, 3, 6)
> +#define SYS_TRCIDR12			sys_reg(2, 1, 0, 4, 6)
> +#define SYS_TRCIDR13			sys_reg(2, 1, 0, 5, 6)
> +#define SYS_TRCIDR1			sys_reg(2, 1, 0, 9, 7)
> +#define SYS_TRCIDR2			sys_reg(2, 1, 0, 10, 7)
> +#define SYS_TRCIDR3			sys_reg(2, 1, 0, 11, 7)
> +#define SYS_TRCIDR4			sys_reg(2, 1, 0, 12, 7)
> +#define SYS_TRCIDR5			sys_reg(2, 1, 0, 13, 7)
> +#define SYS_TRCIDR6			sys_reg(2, 1, 0, 14, 7)
> +#define SYS_TRCIDR7			sys_reg(2, 1, 0, 15, 7)
> +#define SYS_TRCIDR8			sys_reg(2, 1, 0, 0, 6)
> +#define SYS_TRCIDR9			sys_reg(2, 1, 0, 1, 6)
> +#define SYS_TRCIMSPEC0			sys_reg(2, 1, 0, 0, 7)
> +#define SYS_TRCIMSPEC(m)		sys_reg(2, 1, 0, (m & 7), 7)
> +#define SYS_TRCITEEDCR			sys_reg(2, 1, 0, 2, 1)
> +#define SYS_TRCOSLSR			sys_reg(2, 1, 1, 1, 4)
> +#define SYS_TRCPRGCTLR			sys_reg(2, 1, 0, 1, 0)
> +#define SYS_TRCQCTLR			sys_reg(2, 1, 0, 1, 1)
> +#define SYS_TRCRSCTLR(m)		sys_reg(2, 1, 1, (m & 15), (0 | (m >> 4)))
> +#define SYS_TRCRSR			sys_reg(2, 1, 0, 10, 0)
> +#define SYS_TRCSEQEVR(m)		sys_reg(2, 1, 0, (m & 3), 4)
> +#define SYS_TRCSEQRSTEVR		sys_reg(2, 1, 0, 6, 4)
> +#define SYS_TRCSEQSTR			sys_reg(2, 1, 0, 7, 4)
> +#define SYS_TRCSSCCR(m)			sys_reg(2, 1, 1, (m & 7), 2)
> +#define SYS_TRCSSCSR(m)			sys_reg(2, 1, 1, (8 | (m & 7)), 2)
> +#define SYS_TRCSSPCICR(m)		sys_reg(2, 1, 1, (m & 7), 3)
> +#define SYS_TRCSTALLCTLR		sys_reg(2, 1, 0, 11, 0)
> +#define SYS_TRCSTATR			sys_reg(2, 1, 0, 3, 0)
> +#define SYS_TRCSYNCPR			sys_reg(2, 1, 0, 13, 0)
> +#define SYS_TRCTRACEIDR			sys_reg(2, 1, 0, 0, 1)
> +#define SYS_TRCTSCTLR			sys_reg(2, 1, 0, 12, 0)
> +#define SYS_TRCVICTLR			sys_reg(2, 1, 0, 0, 2)
> +#define SYS_TRCVIIECTLR			sys_reg(2, 1, 0, 1, 2)
> +#define SYS_TRCVIPCSSCTLR		sys_reg(2, 1, 0, 3, 2)
> +#define SYS_TRCVISSCTLR			sys_reg(2, 1, 0, 2, 2)
> +#define SYS_TRCVMIDCCTLR0		sys_reg(2, 1, 3, 2, 2)
> +#define SYS_TRCVMIDCCTLR1		sys_reg(2, 1, 3, 3, 2)
> +#define SYS_TRCVMIDCVR(m)		sys_reg(2, 1, 3, ((m & 7) << 1), 1)
> +
> +/* ETM */
> +#define SYS_TRCOSLAR			sys_reg(2, 1, 1, 0, 4)
> +
>   #define SYS_MIDR_EL1			sys_reg(3, 0, 0, 0, 0)
>   #define SYS_MPIDR_EL1			sys_reg(3, 0, 0, 0, 5)
>   #define SYS_REVIDR_EL1			sys_reg(3, 0, 0, 0, 6)

Re: [PATCH 07/27] arm64: Add missing BRB/CFP/DVP/CPP instructions

[Eric Auger]

Hi Marc,

On 7/12/23 16:57, Marc Zyngier wrote:
> HFGITR_EL2 traps a bunch of instructions for which we don't have
> encodings yet. Add them.
>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  arch/arm64/include/asm/sysreg.h | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
> index 9dfd127be55a..e2357529c633 100644
> --- a/arch/arm64/include/asm/sysreg.h
> +++ b/arch/arm64/include/asm/sysreg.h
> @@ -737,6 +737,13 @@
>  #define OP_TLBI_VALE2NXS		sys_insn(1, 4, 9, 7, 5)
>  #define OP_TLBI_VMALLS12E1NXS		sys_insn(1, 4, 9, 7, 6)
>  
> +/* Misc instructions */
> +#define OP_BRB_IALL			sys_insn(1, 1, 7, 2, 4)
> +#define OP_BRB_INJ			sys_insn(1, 1, 7, 2, 5)
> +#define OP_CFP_RCTX			sys_insn(1, 3, 7, 3, 4)
> +#define OP_DVP_RCTX			sys_insn(1, 3, 7, 3, 5)
> +#define OP_CPP_RCTX			sys_insn(1, 3, 7, 3, 7)
> +
>  /* Common SCTLR_ELx flags. */
>  #define SCTLR_ELx_ENTP2	(BIT(60))
>  #define SCTLR_ELx_DSSBS	(BIT(44))
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric

Re: [PATCH 06/27] arm64: Add debug registers affected by HDFGxTR_EL2

[Eric Auger]

Hi Marc,

On 7/14/23 18:09, Marc Zyngier wrote:
> Hey Eric,
>
> On Fri, 14 Jul 2023 15:47:20 +0100,
> Eric Auger <eric.auger@redhat.com> wrote:
>> Hi Marc,
>>
>> On 7/12/23 16:57, Marc Zyngier wrote:
>>> The HDFGxTR_EL2 registers trap a (huge) set of debug and trace
>>> related registers. Add their encodings (and only that, because
>>> we really don't care about what these registers actually do at
>>> this stage).
>>>
>>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>>> ---
>>>  arch/arm64/include/asm/sysreg.h | 78 +++++++++++++++++++++++++++++++++
>>>  1 file changed, 78 insertions(+)
>>>
>>> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
>>> index 76289339b43b..9dfd127be55a 100644
>>> --- a/arch/arm64/include/asm/sysreg.h
>>> +++ b/arch/arm64/include/asm/sysreg.h
>>> @@ -194,6 +194,84 @@
>>>  #define SYS_DBGDTRTX_EL0		sys_reg(2, 3, 0, 5, 0)*
>>>  #define SYS_DBGVCR32_EL2		sys_reg(2, 4, 0, 7, 0)*
>>>  
>>> +#define SYS_BRBINF_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 0))*
>>> +#define SYS_BRBINFINJ_EL1		sys_reg(2, 1, 9, 1, 0)*
>>> +#define SYS_BRBSRC_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 1))*
>>> +#define SYS_BRBSRCINJ_EL1		sys_reg(2, 1, 9, 1, 1)*
>>> +#define SYS_BRBTGT_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 2) | 2))*
>>> +#define SYS_BRBTGTINJ_EL1		sys_reg(2, 1, 9, 1, 2)*
>>> +#define SYS_BRBTS_EL1			sys_reg(2, 1, 9, 0, 2)*
>>> +
>>> +#define SYS_BRBCR_EL1			sys_reg(2, 1, 9, 0, 0)*
>>> +#define SYS_BRBFCR_EL1			sys_reg(2, 1, 9, 0, 1)*
>>> +#define SYS_BRBIDR0_EL1			sys_reg(2, 1, 9, 2, 0)*
>>> +
>>> +#define SYS_TRCITECR_EL1		sys_reg(3, 0, 1, 2, 3)
>>> +#define SYS_TRCITECR_EL1		sys_reg(3, 0, 1, 2, 3)
>> I cannot find this one - which is duplicated by the way - in DDI0487Jaa
> Ah, that's one of the sucker I got from peeking at the 2023-03 XML and
> wrote it twice for a good measure. You can see it there:
>
> https://developer.arm.com/documentation/ddi0601/2023-03/AArch64-Registers/TRCITECR-EL1--Instrumentation-Trace-Control-Register--EL1-
>
>>> +#define SYS_TRCACATR(m)			sys_reg(2, 1, 2, ((m & 7) << 1), (2 | (m >> 3)))*
>>> +#define SYS_TRCACVR(m)			sys_reg(2, 1, 2, ((m & 7) << 1), (0 | (m >> 3)))*
>>> +#define SYS_TRCAUTHSTATUS		sys_reg(2, 1, 7, 14, 6)*
>>> +#define SYS_TRCAUXCTLR			sys_reg(2, 1, 0, 6, 0)*
>>> +#define SYS_TRCBBCTLR			sys_reg(2, 1, 0, 15, 0)*
>>> +#define SYS_TRCCCCTLR			sys_reg(2, 1, 0, 14, 0)*
>>> +#define SYS_TRCCIDCCTLR0		sys_reg(2, 1, 3, 0, 2)*
>>> +#define SYS_TRCCIDCCTLR1		sys_reg(2, 1, 3, 1, 2)*
>>> +#define SYS_TRCCIDCVR(m)		sys_reg(2, 1, 3, ((m & 7) << 1), 0)*
>>> +#define SYS_TRCCLAIMCLR			sys_reg(2, 1, 7, 9, 6)*
>>> +#define SYS_TRCCLAIMSET			sys_reg(2, 1, 7, 8, 6)*
>>> +#define SYS_TRCCNTCTLR(m)		sys_reg(2, 1, 0, (4 | (m & 3)), 5)*
>>> +#define SYS_TRCCNTRLDVR(m)		sys_reg(2, 1, 0, (0 | (m & 3)), 5)*
>>> +#define SYS_TRCCNTVR(m)			sys_reg(2, 1, 0, (8 | (m & 3)), 5)*
>>> +#define SYS_TRCCONFIGR			sys_reg(2, 1, 0, 4, 0)*
>>> +#define SYS_TRCDEVARCH			sys_reg(2, 1, 7, 15, 6)*
>>> +#define SYS_TRCDEVID			sys_reg(2, 1, 7, 2, 7)*
>>> +#define SYS_TRCEVENTCTL0R		sys_reg(2, 1, 0, 8, 0)*
>>> +#define SYS_TRCEVENTCTL1R		sys_reg(2, 1, 0, 9, 0)*
>>> +#define SYS_TRCEXTINSELR(m)		sys_reg(2, 1, 0, (8 | (m & 3)), 4)*
>>> +#define SYS_TRCIDR0			sys_reg(2, 1, 0, 8, 7)*
>>> +#define SYS_TRCIDR10			sys_reg(2, 1, 0, 2, 6)*
>>> +#define SYS_TRCIDR11			sys_reg(2, 1, 0, 3, 6)*
>>> +#define SYS_TRCIDR12			sys_reg(2, 1, 0, 4, 6)*
>>> +#define SYS_TRCIDR13			sys_reg(2, 1, 0, 5, 6)*
>>> +#define SYS_TRCIDR1			sys_reg(2, 1, 0, 9, 7)*
>>> +#define SYS_TRCIDR2			sys_reg(2, 1, 0, 10, 7)*
>>> +#define SYS_TRCIDR3			sys_reg(2, 1, 0, 11, 7)*
>>> +#define SYS_TRCIDR4			sys_reg(2, 1, 0, 12, 7)*
>>> +#define SYS_TRCIDR5			sys_reg(2, 1, 0, 13, 7)*
>>> +#define SYS_TRCIDR6			sys_reg(2, 1, 0, 14, 7)*
>>> +#define SYS_TRCIDR7			sys_reg(2, 1, 0, 15, 7)*
>>> +#define SYS_TRCIDR8			sys_reg(2, 1, 0, 0, 6)*
>>> +#define SYS_TRCIDR9			sys_reg(2, 1, 0, 1, 6)*
>>> +#define SYS_TRCIMSPEC0			sys_reg(2, 1, 0, 0, 7)*
>>> +#define SYS_TRCIMSPEC(m)		sys_reg(2, 1, 0, (m & 7), 7)*
>>> +#define SYS_TRCITEEDCR			sys_reg(2, 1, 0, 2, 1)
>> I cannot find this one in D18-1 or elsewhere in DDI0487Jaa
> Same thing. You can find it here:
>
> https://developer.arm.com/documentation/ddi0601/2023-03/AArch64-Registers/TRCITEEDCR--Instrumentation-Trace-Extension-External-Debug-Control-Register
>
>>> +#define SYS_TRCOSLSR			sys_reg(2, 1, 1, 1, 4)*
>>> +#define SYS_TRCPRGCTLR			sys_reg(2, 1, 0, 1, 0)*
>>> +#define SYS_TRCQCTLR			sys_reg(2, 1, 0, 1, 1)*
>>> +#define SYS_TRCRSCTLR(m)		sys_reg(2, 1, 1, (m & 15), (0 | (m >> 4)))*
>>> +#define SYS_TRCRSR			sys_reg(2, 1, 0, 10, 0)*
>>> +#define SYS_TRCSEQEVR(m)		sys_reg(2, 1, 0, (m & 3), 4)*
>>> +#define SYS_TRCSEQRSTEVR		sys_reg(2, 1, 0, 6, 4)*
>>> +#define SYS_TRCSEQSTR			sys_reg(2, 1, 0, 7, 4)*
>>> +#define SYS_TRCSSCCR(m)			sys_reg(2, 1, 1, (m & 7), 2)*
>>> +#define SYS_TRCSSCSR(m)			sys_reg(2, 1, 1, (8 | (m & 7)), 2)*
>>> +#define SYS_TRCSSPCICR(m)		sys_reg(2, 1, 1, (m & 7), 3)*
>>> +#define SYS_TRCSTALLCTLR		sys_reg(2, 1, 0, 11, 0)*
>>> +#define SYS_TRCSTATR			sys_reg(2, 1, 0, 3, 0)*
>>> +#define SYS_TRCSYNCPR			sys_reg(2, 1, 0, 13, 0)*
>>> +#define SYS_TRCTRACEIDR			sys_reg(2, 1, 0, 0, 1)*
>>> +#define SYS_TRCTSCTLR			sys_reg(2, 1, 0, 12, 0)*
>>> +#define SYS_TRCVICTLR			sys_reg(2, 1, 0, 0, 2)*
>>> +#define SYS_TRCVIIECTLR			sys_reg(2, 1, 0, 1, 2)*
>>> +#define SYS_TRCVIPCSSCTLR		sys_reg(2, 1, 0, 3, 2)*
>>> +#define SYS_TRCVISSCTLR			sys_reg(2, 1, 0, 2, 2)*
>>> +#define SYS_TRCVMIDCCTLR0		sys_reg(2, 1, 3, 2, 2)*
>>> +#define SYS_TRCVMIDCCTLR1		sys_reg(2, 1, 3, 3, 2)*
>>> +#define SYS_TRCVMIDCVR(m)		sys_reg(2, 1, 3, ((m & 7) << 1), 1)*
>>> +
>>> +/* ETM */
>>> +#define SYS_TRCOSLAR			sys_reg(2, 1, 1, 0, 4)
>> not able to locate this one either. I see the bit of HDFGWTR_EL2 though
> This one lives in the ETM spec:
>
> https://documentation-service.arm.com/static/60017fbb3f22832ff1d6872b
>
> Page 7-342 has the register number, and the encoding is computed as
> per the formula in 4.3.6 "System instructions", page 4-169.

OK then feel free to add my
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric
>
> Thanks,
>
> 	M.
>

Re: [PATCH 08/27] arm64: Fix HFGxTR_EL2 field naming

[Eric Auger]

On 7/12/23 16:57, Marc Zyngier wrote:
> The HFGxTR_EL2 fields do not always follow the naming described
> in the spec, nor do they match the name of the register they trap
> in the rest of the kernel.
>
> It is a bit sad that they were written by hand despite the availability
> of a machine readable version...
>
> Fixes: cc077e7facbe ("arm64/sysreg: Convert HFG[RW]TR_EL2 to automatic generation")
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> Reviewed-by: Mark Brown <broonie@kernel.org>
> Cc: Will Deacon <will@kernel.org>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> ---
>  arch/arm64/tools/sysreg | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
>
> diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg
> index 1ea4a3dc68f8..65866bf819c3 100644
> --- a/arch/arm64/tools/sysreg
> +++ b/arch/arm64/tools/sysreg
> @@ -2017,7 +2017,7 @@ Field	0	SM
>  EndSysreg
>  
>  SysregFields	HFGxTR_EL2
> -Field	63	nAMIAIR2_EL1
> +Field	63	nAMAIR2_EL1
>  Field	62	nMAIR2_EL1
>  Field	61	nS2POR_EL1
>  Field	60	nPOR_EL1
> @@ -2032,9 +2032,9 @@ Field	52	nGCS_EL0
>  Res0	51
>  Field	50	nACCDATA_EL1
>  Field	49	ERXADDR_EL1
> -Field	48	EXRPFGCDN_EL1
> -Field	47	EXPFGCTL_EL1
> -Field	46	EXPFGF_EL1
> +Field	48	ERXPFGCDN_EL1
> +Field	47	ERXPFGCTL_EL1
> +Field	46	ERXPFGF_EL1
>  Field	45	ERXMISCn_EL1
>  Field	44	ERXSTATUS_EL1
>  Field	43	ERXCTLR_EL1
> @@ -2049,8 +2049,8 @@ Field	35	TPIDR_EL0
>  Field	34	TPIDRRO_EL0
>  Field	33	TPIDR_EL1
>  Field	32	TCR_EL1
> -Field	31	SCTXNUM_EL0
> -Field	30	SCTXNUM_EL1
> +Field	31	SCXTNUM_EL0
> +Field	30	SCXTNUM_EL1
>  Field	29	SCTLR_EL1
>  Field	28	REVIDR_EL1
>  Field	27	PAR_EL1
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Eric

Re: [PATCH 11/27] KVM: arm64: Correctly handle ACCDATA_EL1 traps

[Eric Auger]

Hi Marc,
On 7/12/23 16:57, Marc Zyngier wrote:
> As we blindly reset some HFGxTR_EL2 bits to 0, we also randomly trap
> unsuspecting sysregs that have their trap bits with a negative
> polarity.
>
> ACCDATA_EL1 is one such register that can be accessed by the guest,
> causing a splat on the host as we don't have a proper handler for
> it.
>
> Adding such handler addresses the issue, though there are a number
> of other registers missing as the current architecture documentation
> doesn't describe them yet.
>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  arch/arm64/include/asm/sysreg.h | 2 ++
>  arch/arm64/kvm/sys_regs.c       | 2 ++
>  2 files changed, 4 insertions(+)
>
> diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h
> index 6d3d16fac227..d528a79417a0 100644
> --- a/arch/arm64/include/asm/sysreg.h
> +++ b/arch/arm64/include/asm/sysreg.h
> @@ -389,6 +389,8 @@
>  #define SYS_ICC_IGRPEN0_EL1		sys_reg(3, 0, 12, 12, 6)
>  #define SYS_ICC_IGRPEN1_EL1		sys_reg(3, 0, 12, 12, 7)
>  
> +#define SYS_ACCDATA_EL1			sys_reg(3, 0, 13, 0, 5)
> +
>  #define SYS_CNTKCTL_EL1			sys_reg(3, 0, 14, 1, 0)
>  
>  #define SYS_AIDR_EL1			sys_reg(3, 1, 0, 0, 7)
> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> index bd3431823ec5..3a6f678ca67d 100644
> --- a/arch/arm64/kvm/sys_regs.c
> +++ b/arch/arm64/kvm/sys_regs.c
> @@ -2151,6 +2151,8 @@ static const struct sys_reg_desc sys_reg_descs[] = {
>  	{ SYS_DESC(SYS_CONTEXTIDR_EL1), access_vm_reg, reset_val, CONTEXTIDR_EL1, 0 },
>  	{ SYS_DESC(SYS_TPIDR_EL1), NULL, reset_unknown, TPIDR_EL1 },
>  
> +	{ SYS_DESC(SYS_ACCDATA_EL1), undef_access },
> +
>  	{ SYS_DESC(SYS_SCXTNUM_EL1), undef_access },
>  
>  	{ SYS_DESC(SYS_CNTKCTL_EL1), NULL, reset_val, CNTKCTL_EL1, 0},
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Eric

Re: [PATCH 13/27] KVM: arm64: nv: Add FGT registers

[Eric Auger]

Hi Marc,

On 7/12/23 16:57, Marc Zyngier wrote:
> Add the 5 registers covering FEAT_FGT. The AMU-related registers
> are currently left out as we don't have a plan for them. Yet.
>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  arch/arm64/include/asm/kvm_host.h | 5 +++++
>  arch/arm64/kvm/sys_regs.c         | 5 +++++
>  2 files changed, 10 insertions(+)
>
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index 8b6096753740..1200f29282ba 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -400,6 +400,11 @@ enum vcpu_sysreg {
>  	TPIDR_EL2,	/* EL2 Software Thread ID Register */
>  	CNTHCTL_EL2,	/* Counter-timer Hypervisor Control register */
>  	SP_EL2,		/* EL2 Stack Pointer */
> +	HFGRTR_EL2,
> +	HFGWTR_EL2,
> +	HFGITR_EL2,
> +	HDFGRTR_EL2,
> +	HDFGWTR_EL2,
>  	CNTHP_CTL_EL2,
>  	CNTHP_CVAL_EL2,
>  	CNTHV_CTL_EL2,
> diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c
> index 3a6f678ca67d..f88cd1390998 100644
> --- a/arch/arm64/kvm/sys_regs.c
> +++ b/arch/arm64/kvm/sys_regs.c
> @@ -2367,6 +2367,9 @@ static const struct sys_reg_desc sys_reg_descs[] = {
>  	EL2_REG(MDCR_EL2, access_rw, reset_val, 0),
>  	EL2_REG(CPTR_EL2, access_rw, reset_val, CPTR_NVHE_EL2_RES1),
>  	EL2_REG(HSTR_EL2, access_rw, reset_val, 0),
> +	EL2_REG(HFGRTR_EL2, access_rw, reset_val, 0),
> +	EL2_REG(HFGWTR_EL2, access_rw, reset_val, 0),
> +	EL2_REG(HFGITR_EL2, access_rw, reset_val, 0),
>  	EL2_REG(HACR_EL2, access_rw, reset_val, 0),
>  
>  	EL2_REG(TTBR0_EL2, access_rw, reset_val, 0),
> @@ -2376,6 +2379,8 @@ static const struct sys_reg_desc sys_reg_descs[] = {
>  	EL2_REG(VTCR_EL2, access_rw, reset_val, 0),
>  
>  	{ SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 },
> +	EL2_REG(HDFGRTR_EL2, access_rw, reset_val, 0),
> +	EL2_REG(HDFGWTR_EL2, access_rw, reset_val, 0),
>  	EL2_REG(SPSR_EL2, access_rw, reset_val, 0),
>  	EL2_REG(ELR_EL2, access_rw, reset_val, 0),
>  	{ SYS_DESC(SYS_SP_EL1), access_sp_el1},
Reviewed-by: Eric Auger <eric.auger@redhat.com>


Eric

Re: [PATCH 14/27] KVM: arm64: Restructure FGT register switching

[Eric Auger]

Hi Marc,

On 7/12/23 16:57, Marc Zyngier wrote:
> As we're about to majorly extend the handling of FGT registers,
> restructure the code to actually save/restore the registers
> as required. This is made easy thanks to the previous addition
> of the EL2 registers, allowing us to use the host context for
> this purpose.
>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  arch/arm64/include/asm/kvm_arm.h        | 21 ++++++++++
>  arch/arm64/kvm/hyp/include/hyp/switch.h | 55 +++++++++++++------------
>  2 files changed, 49 insertions(+), 27 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
> index 028049b147df..85908aa18908 100644
> --- a/arch/arm64/include/asm/kvm_arm.h
> +++ b/arch/arm64/include/asm/kvm_arm.h
> @@ -333,6 +333,27 @@
>  				 BIT(18) |		\
>  				 GENMASK(16, 15))
>  
> +/*
> + * FGT register definitions
> + *
> + * RES0 and polarity masks as of DDI0487J.a, to be updated as needed.
> + * We're not using the generated masks as they are usually ahead of
> + * the published ARM ARM, which we use as a reference.
> + *
> + * Once we get to a point where the two describe the same thing, we'll
> + * merge the definitions. One day.
> + */
> +#define __HFGRTR_EL2_RES0	(GENMASK(63, 56) | GENMASK(53, 51))
> +#define __HFGRTR_EL2_MASK	GENMASK(49, 0)
> +#define __HFGRTR_EL2_nMASK	(GENMASK(55, 54) | BIT(50))
> +
> +#define __HFGWTR_EL2_RES0	(GENMASK(63, 56) | GENMASK(53, 51) |	\
> +				 BIT(46) | BIT(42) | BIT(40) | BIT(28) | \
> +				 GENMASK(26, 25) | BIT(21) | BIT(18) |	\
> +				 GENMASK(15, 14) | GENMASK(10, 9) | BIT(2))
> +#define __HFGWTR_EL2_MASK	GENMASK(49, 0)
> +#define __HFGWTR_EL2_nMASK	(GENMASK(55, 54) | BIT(50))
> +
>  /* Hyp Prefetch Fault Address Register (HPFAR/HDFAR) */
>  #define HPFAR_MASK	(~UL(0xf))
>  /*
> diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
> index 4bddb8541bec..9781e79a5127 100644
> --- a/arch/arm64/kvm/hyp/include/hyp/switch.h
> +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
> @@ -70,20 +70,19 @@ static inline void __activate_traps_fpsimd32(struct kvm_vcpu *vcpu)
>  	}
>  }
>  
> -static inline bool __hfgxtr_traps_required(void)
> -{
> -	if (cpus_have_final_cap(ARM64_SME))
> -		return true;
> -
> -	if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
> -		return true;
>  
> -	return false;
> -}
>  
> -static inline void __activate_traps_hfgxtr(void)
> +static inline void __activate_traps_hfgxtr(struct kvm_vcpu *vcpu)
>  {
> +	struct kvm_cpu_context *hctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
>  	u64 r_clr = 0, w_clr = 0, r_set = 0, w_set = 0, tmp;
> +	u64 r_val, w_val;
> +
> +	if (!cpus_have_final_cap(ARM64_HAS_FGT))
> +		return;
> +
> +	ctxt_sys_reg(hctxt, HFGRTR_EL2) = read_sysreg_s(SYS_HFGRTR_EL2);
> +	ctxt_sys_reg(hctxt, HFGWTR_EL2) = read_sysreg_s(SYS_HFGWTR_EL2);
>  
>  	if (cpus_have_final_cap(ARM64_SME)) {
>  		tmp = HFGxTR_EL2_nSMPRI_EL1_MASK | HFGxTR_EL2_nTPIDR2_EL0_MASK;
> @@ -98,26 +97,30 @@ static inline void __activate_traps_hfgxtr(void)
>  	if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
>  		w_set |= HFGxTR_EL2_TCR_EL1_MASK;
>  
> -	sysreg_clear_set_s(SYS_HFGRTR_EL2, r_clr, r_set);
> -	sysreg_clear_set_s(SYS_HFGWTR_EL2, w_clr, w_set);
> +
> +	r_val = __HFGRTR_EL2_nMASK & ~HFGxTR_EL2_nACCDATA_EL1;
I don't get why you do

& ~HFGxTR_EL2_nACCDATA_EL1 as this latter also has a negative polarity. 

Please could you explain what is special with this bit/add a comment?

Thanks

Eric

> +	r_val |= r_set;
> +	r_val &= ~r_clr;
> +
> +	w_val = __HFGWTR_EL2_nMASK & ~HFGxTR_EL2_nACCDATA_EL1;
> +	w_val |= w_set;
> +	w_val &= ~w_clr;
> +
> +	write_sysreg_s(r_val, SYS_HFGRTR_EL2);
> +	write_sysreg_s(w_val, SYS_HFGWTR_EL2);
>  }
>  
> -static inline void __deactivate_traps_hfgxtr(void)
> +static inline void __deactivate_traps_hfgxtr(struct kvm_vcpu *vcpu)
>  {
> -	u64 r_clr = 0, w_clr = 0, r_set = 0, w_set = 0, tmp;
> +	struct kvm_cpu_context *hctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
>  
> -	if (cpus_have_final_cap(ARM64_SME)) {
> -		tmp = HFGxTR_EL2_nSMPRI_EL1_MASK | HFGxTR_EL2_nTPIDR2_EL0_MASK;
> +	if (!cpus_have_final_cap(ARM64_HAS_FGT))
> +		return;
>  
> -		r_set |= tmp;
> -		w_set |= tmp;
> -	}
> +	write_sysreg_s(ctxt_sys_reg(hctxt, HFGRTR_EL2), SYS_HFGRTR_EL2);
> +	write_sysreg_s(ctxt_sys_reg(hctxt, HFGWTR_EL2), SYS_HFGWTR_EL2);
>  
> -	if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
> -		w_clr |= HFGxTR_EL2_TCR_EL1_MASK;
>  
> -	sysreg_clear_set_s(SYS_HFGRTR_EL2, r_clr, r_set);
> -	sysreg_clear_set_s(SYS_HFGWTR_EL2, w_clr, w_set);
>  }
>  
>  static inline void __activate_traps_common(struct kvm_vcpu *vcpu)
> @@ -145,8 +148,7 @@ static inline void __activate_traps_common(struct kvm_vcpu *vcpu)
>  	vcpu->arch.mdcr_el2_host = read_sysreg(mdcr_el2);
>  	write_sysreg(vcpu->arch.mdcr_el2, mdcr_el2);
>  
> -	if (__hfgxtr_traps_required())
> -		__activate_traps_hfgxtr();
> +	__activate_traps_hfgxtr(vcpu);
>  }
>  
>  static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu)
> @@ -162,8 +164,7 @@ static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu)
>  		vcpu_clear_flag(vcpu, PMUSERENR_ON_CPU);
>  	}
>  
> -	if (__hfgxtr_traps_required())
> -		__deactivate_traps_hfgxtr();
> +	__deactivate_traps_hfgxtr(vcpu);
>  }
>  
>  static inline void ___activate_traps(struct kvm_vcpu *vcpu)

Re: [PATCH 17/27] KVM: arm64: nv: Expose FEAT_EVT to nested guests

[Eric Auger]

On 7/12/23 16:58, Marc Zyngier wrote:
> Now that we properly implement FEAT_EVT (as we correctly forward
> traps), expose it to guests.
>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Eric
> ---
>  arch/arm64/kvm/nested.c | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
>
> diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c
> index 315354d27978..7f80f385d9e8 100644
> --- a/arch/arm64/kvm/nested.c
> +++ b/arch/arm64/kvm/nested.c
> @@ -124,8 +124,7 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
>  		break;
>  
>  	case SYS_ID_AA64MMFR2_EL1:
> -		val &= ~(NV_FTR(MMFR2, EVT)	|
> -			 NV_FTR(MMFR2, BBM)	|
> +		val &= ~(NV_FTR(MMFR2, BBM)	|
>  			 NV_FTR(MMFR2, TTL)	|
>  			 GENMASK_ULL(47, 44)	|
>  			 NV_FTR(MMFR2, ST)	|

Re: [PATCH 19/27] KVM: arm64: nv: Add trap forwarding for CNTHCTL_EL2

[Eric Auger]

Hi Marc,

On 7/12/23 16:58, Marc Zyngier wrote:
> Describe the CNTHCTL_EL2 register, and associate it with all the sysregs
> it allows to trap.
>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  arch/arm64/kvm/emulate-nested.c | 37 ++++++++++++++++++++++++++++++++-
>  1 file changed, 36 insertions(+), 1 deletion(-)
>
> diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
> index 25e4842ac334..c07c0f3361d7 100644
> --- a/arch/arm64/kvm/emulate-nested.c
> +++ b/arch/arm64/kvm/emulate-nested.c
> @@ -98,9 +98,11 @@ enum coarse_grain_trap_id {
>  
>  	/*
>  	 * Anything after this point requires a callback evaluating a
> -	 * complex trap condition. Hopefully we'll never need this...
> +	 * complex trap condition. Ugly stuff.
>  	 */
>  	__COMPLEX_CONDITIONS__,
> +	CGT_CNTHCTL_EL1PCTEN = __COMPLEX_CONDITIONS__,
> +	CGT_CNTHCTL_EL1PTEN,
>  };
>  
>  static const struct trap_bits coarse_trap_bits[] = {
> @@ -358,9 +360,37 @@ static const enum coarse_grain_trap_id *coarse_control_combo[] = {
>  
>  typedef enum trap_behaviour (*complex_condition_check)(struct kvm_vcpu *);
>  
> +static u64 get_sanitized_cnthctl(struct kvm_vcpu *vcpu)
> +{
> +	u64 val = __vcpu_sys_reg(vcpu, CNTHCTL_EL2);
> +
> +	if (!vcpu_el2_e2h_is_set(vcpu))
> +		val = (val & (CNTHCTL_EL1PCEN | CNTHCTL_EL1PCTEN)) << 10;
> +
> +	return val;
don't you want to return only bits 10 & 11 to match the other condition?

I would add a comment saying that When FEAT_VHE is implemented and
HCR_EL2.E2H == 1:

sanitized_cnthctl[11:10] = [EL1PTEN, EL1PCTEN]
otherwise
sanitized_cnthctl[11:10] = [EL1PCEN, EL1PCTEN]

> +}
> +
> +static enum trap_behaviour check_cnthctl_el1pcten(struct kvm_vcpu *vcpu)
> +{
> +	if (get_sanitized_cnthctl(vcpu) & (CNTHCTL_EL1PCTEN << 10))
> +		return BEHAVE_HANDLE_LOCALLY;
> +
> +	return BEHAVE_FORWARD_ANY;
> +}
> +
> +static enum trap_behaviour check_cnthctl_el1pten(struct kvm_vcpu *vcpu)
or pcen. This is a bit confusing to see EL1PCEN below. But this is due
to above sanitized CNTHCTL. Worth a comment to me.
> +{
> +	if (get_sanitized_cnthctl(vcpu) & (CNTHCTL_EL1PCEN << 10))
> +		return BEHAVE_HANDLE_LOCALLY;
> +
> +	return BEHAVE_FORWARD_ANY;
> +}
> +
>  #define CCC(id, fn)	[id - __COMPLEX_CONDITIONS__] = fn
>  
>  static const complex_condition_check ccc[] = {
> +	CCC(CGT_CNTHCTL_EL1PCTEN, check_cnthctl_el1pcten),
> +	CCC(CGT_CNTHCTL_EL1PTEN, check_cnthctl_el1pten),
>  };
>  
>  /*
> @@ -855,6 +885,11 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initdata = {
>  	SR_TRAP(SYS_TRBPTR_EL1, 	CGT_MDCR_E2TB),
>  	SR_TRAP(SYS_TRBSR_EL1, 		CGT_MDCR_E2TB),
>  	SR_TRAP(SYS_TRBTRG_EL1,		CGT_MDCR_E2TB),
> +	SR_TRAP(SYS_CNTP_TVAL_EL0,	CGT_CNTHCTL_EL1PTEN),
> +	SR_TRAP(SYS_CNTP_CVAL_EL0,	CGT_CNTHCTL_EL1PTEN),
> +	SR_TRAP(SYS_CNTP_CTL_EL0,	CGT_CNTHCTL_EL1PTEN),
> +	SR_TRAP(SYS_CNTPCT_EL0,		CGT_CNTHCTL_EL1PCTEN),
> +	SR_TRAP(SYS_CNTPCTSS_EL0,	CGT_CNTHCTL_EL1PCTEN),
>  };
>  
>  static DEFINE_XARRAY(sr_forward_xa);
Otherwise looks good to me
Reviewed-by: Eric Auger <eric.auger@redhat.com>


Eric

Re: [PATCH 14/27] KVM: arm64: Restructure FGT register switching

[Marc Zyngier]

On Tue, 25 Jul 2023 17:39:52 +0100,
Eric Auger <eric.auger@redhat.com> wrote:
> 
> Hi Marc,
> 
> On 7/12/23 16:57, Marc Zyngier wrote:
> > As we're about to majorly extend the handling of FGT registers,
> > restructure the code to actually save/restore the registers
> > as required. This is made easy thanks to the previous addition
> > of the EL2 registers, allowing us to use the host context for
> > this purpose.
> >
> > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > ---
> >  arch/arm64/include/asm/kvm_arm.h        | 21 ++++++++++
> >  arch/arm64/kvm/hyp/include/hyp/switch.h | 55 +++++++++++++------------
> >  2 files changed, 49 insertions(+), 27 deletions(-)
> >
> > diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
> > index 028049b147df..85908aa18908 100644
> > --- a/arch/arm64/include/asm/kvm_arm.h
> > +++ b/arch/arm64/include/asm/kvm_arm.h
> > @@ -333,6 +333,27 @@
> >  				 BIT(18) |		\
> >  				 GENMASK(16, 15))
> >  
> > +/*
> > + * FGT register definitions
> > + *
> > + * RES0 and polarity masks as of DDI0487J.a, to be updated as needed.
> > + * We're not using the generated masks as they are usually ahead of
> > + * the published ARM ARM, which we use as a reference.
> > + *
> > + * Once we get to a point where the two describe the same thing, we'll
> > + * merge the definitions. One day.
> > + */
> > +#define __HFGRTR_EL2_RES0	(GENMASK(63, 56) | GENMASK(53, 51))
> > +#define __HFGRTR_EL2_MASK	GENMASK(49, 0)
> > +#define __HFGRTR_EL2_nMASK	(GENMASK(55, 54) | BIT(50))
> > +
> > +#define __HFGWTR_EL2_RES0	(GENMASK(63, 56) | GENMASK(53, 51) |	\
> > +				 BIT(46) | BIT(42) | BIT(40) | BIT(28) | \
> > +				 GENMASK(26, 25) | BIT(21) | BIT(18) |	\
> > +				 GENMASK(15, 14) | GENMASK(10, 9) | BIT(2))
> > +#define __HFGWTR_EL2_MASK	GENMASK(49, 0)
> > +#define __HFGWTR_EL2_nMASK	(GENMASK(55, 54) | BIT(50))
> > +
> >  /* Hyp Prefetch Fault Address Register (HPFAR/HDFAR) */
> >  #define HPFAR_MASK	(~UL(0xf))
> >  /*
> > diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
> > index 4bddb8541bec..9781e79a5127 100644
> > --- a/arch/arm64/kvm/hyp/include/hyp/switch.h
> > +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
> > @@ -70,20 +70,19 @@ static inline void __activate_traps_fpsimd32(struct kvm_vcpu *vcpu)
> >  	}
> >  }
> >  
> > -static inline bool __hfgxtr_traps_required(void)
> > -{
> > -	if (cpus_have_final_cap(ARM64_SME))
> > -		return true;
> > -
> > -	if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
> > -		return true;
> >  
> > -	return false;
> > -}
> >  
> > -static inline void __activate_traps_hfgxtr(void)
> > +static inline void __activate_traps_hfgxtr(struct kvm_vcpu *vcpu)
> >  {
> > +	struct kvm_cpu_context *hctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
> >  	u64 r_clr = 0, w_clr = 0, r_set = 0, w_set = 0, tmp;
> > +	u64 r_val, w_val;
> > +
> > +	if (!cpus_have_final_cap(ARM64_HAS_FGT))
> > +		return;
> > +
> > +	ctxt_sys_reg(hctxt, HFGRTR_EL2) = read_sysreg_s(SYS_HFGRTR_EL2);
> > +	ctxt_sys_reg(hctxt, HFGWTR_EL2) = read_sysreg_s(SYS_HFGWTR_EL2);
> >  
> >  	if (cpus_have_final_cap(ARM64_SME)) {
> >  		tmp = HFGxTR_EL2_nSMPRI_EL1_MASK | HFGxTR_EL2_nTPIDR2_EL0_MASK;
> > @@ -98,26 +97,30 @@ static inline void __activate_traps_hfgxtr(void)
> >  	if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
> >  		w_set |= HFGxTR_EL2_TCR_EL1_MASK;
> >  
> > -	sysreg_clear_set_s(SYS_HFGRTR_EL2, r_clr, r_set);
> > -	sysreg_clear_set_s(SYS_HFGWTR_EL2, w_clr, w_set);
> > +
> > +	r_val = __HFGRTR_EL2_nMASK & ~HFGxTR_EL2_nACCDATA_EL1;
> I don't get why you do
> 
> & ~HFGxTR_EL2_nACCDATA_EL1 as this latter also has a negative polarity. 
> 
> Please could you explain what is special with this bit/add a comment?

Nothing is really special with this bit.

But it is currently always cleared (we blindly write a big fat zero),
and I wanted to explicitly show all the instructions for which we
enable trapping for (ACCDATA_EL1 being the only one that is currently
documented in the ARM ARM, although there are more already).

So the construct I came up with is the above, initialising the
register value with the nMASK bits (i.e. not trapping the
corresponding instructions), and then clearing the bit for the stuff
we want to trap. Maybe adding something like:

/* Default to no trapping anything but ACCDATA_EL1 */

would help?

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 19/27] KVM: arm64: nv: Add trap forwarding for CNTHCTL_EL2

[Marc Zyngier]

On 2023-07-25 18:37, Eric Auger wrote:
> Hi Marc,
> 
> On 7/12/23 16:58, Marc Zyngier wrote:
>> Describe the CNTHCTL_EL2 register, and associate it with all the 
>> sysregs
>> it allows to trap.
>> 
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> ---
>>  arch/arm64/kvm/emulate-nested.c | 37 
>> ++++++++++++++++++++++++++++++++-
>>  1 file changed, 36 insertions(+), 1 deletion(-)
>> 
>> diff --git a/arch/arm64/kvm/emulate-nested.c 
>> b/arch/arm64/kvm/emulate-nested.c
>> index 25e4842ac334..c07c0f3361d7 100644
>> --- a/arch/arm64/kvm/emulate-nested.c
>> +++ b/arch/arm64/kvm/emulate-nested.c
>> @@ -98,9 +98,11 @@ enum coarse_grain_trap_id {
>> 
>>  	/*
>>  	 * Anything after this point requires a callback evaluating a
>> -	 * complex trap condition. Hopefully we'll never need this...
>> +	 * complex trap condition. Ugly stuff.
>>  	 */
>>  	__COMPLEX_CONDITIONS__,
>> +	CGT_CNTHCTL_EL1PCTEN = __COMPLEX_CONDITIONS__,
>> +	CGT_CNTHCTL_EL1PTEN,
>>  };
>> 
>>  static const struct trap_bits coarse_trap_bits[] = {
>> @@ -358,9 +360,37 @@ static const enum coarse_grain_trap_id 
>> *coarse_control_combo[] = {
>> 
>>  typedef enum trap_behaviour (*complex_condition_check)(struct 
>> kvm_vcpu *);
>> 
>> +static u64 get_sanitized_cnthctl(struct kvm_vcpu *vcpu)
>> +{
>> +	u64 val = __vcpu_sys_reg(vcpu, CNTHCTL_EL2);
>> +
>> +	if (!vcpu_el2_e2h_is_set(vcpu))
>> +		val = (val & (CNTHCTL_EL1PCEN | CNTHCTL_EL1PCTEN)) << 10;
>> +
>> +	return val;
> don't you want to return only bits 10 & 11 to match the other 
> condition?
> 
> I would add a comment saying that When FEAT_VHE is implemented and
> HCR_EL2.E2H == 1:
> 
> sanitized_cnthctl[11:10] = [EL1PTEN, EL1PCTEN]
> otherwise
> sanitized_cnthctl[11:10] = [EL1PCEN, EL1PCTEN]
> 
>> +}
>> +
>> +static enum trap_behaviour check_cnthctl_el1pcten(struct kvm_vcpu 
>> *vcpu)
>> +{
>> +	if (get_sanitized_cnthctl(vcpu) & (CNTHCTL_EL1PCTEN << 10))
>> +		return BEHAVE_HANDLE_LOCALLY;
>> +
>> +	return BEHAVE_FORWARD_ANY;
>> +}
>> +
>> +static enum trap_behaviour check_cnthctl_el1pten(struct kvm_vcpu 
>> *vcpu)
> or pcen. This is a bit confusing to see EL1PCEN below. But this is due
> to above sanitized CNTHCTL. Worth a comment to me.

I'm adding the following:

/*
  * Warning, maximum confusion ahead.
  *
  * When E2H=0, CNTHCTL_EL2[1:0] are defined as EL1PCEN:EL1PCTEN
  * When E2H=1, CNTHCTL_EL2[11:10] are defined as EL1PTEN:EL1PCTEN
  *
  * Note the single letter difference? Yet, the bits have the same
  * function despite a different layout and a different name.
  *
  * We don't try to reconcile this mess. We just use the E2H=0 bits
  * to generate something that is in the E2H=1 format, and live with
  * it. You're welcome.
  */

Hopefully this will make things clearer. Not completely sure though.

>> +{
>> +	if (get_sanitized_cnthctl(vcpu) & (CNTHCTL_EL1PCEN << 10))
>> +		return BEHAVE_HANDLE_LOCALLY;
>> +
>> +	return BEHAVE_FORWARD_ANY;
>> +}
>> +
>>  #define CCC(id, fn)	[id - __COMPLEX_CONDITIONS__] = fn
>> 
>>  static const complex_condition_check ccc[] = {
>> +	CCC(CGT_CNTHCTL_EL1PCTEN, check_cnthctl_el1pcten),
>> +	CCC(CGT_CNTHCTL_EL1PTEN, check_cnthctl_el1pten),
>>  };
>> 
>>  /*
>> @@ -855,6 +885,11 @@ static const struct encoding_to_trap_config 
>> encoding_to_cgt[] __initdata = {
>>  	SR_TRAP(SYS_TRBPTR_EL1, 	CGT_MDCR_E2TB),
>>  	SR_TRAP(SYS_TRBSR_EL1, 		CGT_MDCR_E2TB),
>>  	SR_TRAP(SYS_TRBTRG_EL1,		CGT_MDCR_E2TB),
>> +	SR_TRAP(SYS_CNTP_TVAL_EL0,	CGT_CNTHCTL_EL1PTEN),
>> +	SR_TRAP(SYS_CNTP_CVAL_EL0,	CGT_CNTHCTL_EL1PTEN),
>> +	SR_TRAP(SYS_CNTP_CTL_EL0,	CGT_CNTHCTL_EL1PTEN),
>> +	SR_TRAP(SYS_CNTPCT_EL0,		CGT_CNTHCTL_EL1PCTEN),
>> +	SR_TRAP(SYS_CNTPCTSS_EL0,	CGT_CNTHCTL_EL1PCTEN),
>>  };
>> 
>>  static DEFINE_XARRAY(sr_forward_xa);
> Otherwise looks good to me
> Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks!

         M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 06/27] arm64: Add debug registers affected by HDFGxTR_EL2

[Marc Zyngier]

On 2023-07-19 12:00, Suzuki K Poulose wrote:
> Hi Marc
> 
> On 12/07/2023 15:57, Marc Zyngier wrote:
>> The HDFGxTR_EL2 registers trap a (huge) set of debug and trace
>> related registers. Add their encodings (and only that, because
>> we really don't care about what these registers actually do at
>> this stage).
>> 
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> ---
>>   arch/arm64/include/asm/sysreg.h | 78 
>> +++++++++++++++++++++++++++++++++
>>   1 file changed, 78 insertions(+)
>> 
>> diff --git a/arch/arm64/include/asm/sysreg.h 
>> b/arch/arm64/include/asm/sysreg.h
>> index 76289339b43b..9dfd127be55a 100644
>> --- a/arch/arm64/include/asm/sysreg.h
>> +++ b/arch/arm64/include/asm/sysreg.h
>> @@ -194,6 +194,84 @@
>>   #define SYS_DBGDTRTX_EL0		sys_reg(2, 3, 0, 5, 0)
>>   #define SYS_DBGVCR32_EL2		sys_reg(2, 4, 0, 7, 0)
>>   +#define SYS_BRBINF_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 
>> 2) | 0))
>> +#define SYS_BRBINFINJ_EL1		sys_reg(2, 1, 9, 1, 0)
>> +#define SYS_BRBSRC_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 
>> 2) | 1))
>> +#define SYS_BRBSRCINJ_EL1		sys_reg(2, 1, 9, 1, 1)
>> +#define SYS_BRBTGT_EL1(n)		sys_reg(2, 1, 8, (n & 15), (((n & 16) >> 
>> 2) | 2))
>> +#define SYS_BRBTGTINJ_EL1		sys_reg(2, 1, 9, 1, 2)
>> +#define SYS_BRBTS_EL1			sys_reg(2, 1, 9, 0, 2)
>> +
>> +#define SYS_BRBCR_EL1			sys_reg(2, 1, 9, 0, 0)
>> +#define SYS_BRBFCR_EL1			sys_reg(2, 1, 9, 0, 1)
>> +#define SYS_BRBIDR0_EL1			sys_reg(2, 1, 9, 2, 0)
>> +
> 
> Merge conflict alert. The above are being added as part of the BRBE
> support series from Anshuman [0], though in a slightly different
> scheme
> for registers with numbers. e.g, SYS_BRBINF_EL1(0) vs SYS_BRBINF0_EL1.
> That series is not merged yet, but might go in this cycle.

Thanks for the heads up!

I'm pretty confident that this will not conflict too badly, as the
#defines are different. But I can always drag a prefix of Anshuman's
series into the KVM tree and resolve it there.

Cheers,

         M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 14/27] KVM: arm64: Restructure FGT register switching

[Eric Auger]

Hi Marc,

On 7/26/23 09:23, Marc Zyngier wrote:
> On Tue, 25 Jul 2023 17:39:52 +0100,
> Eric Auger <eric.auger@redhat.com> wrote:
>> Hi Marc,
>>
>> On 7/12/23 16:57, Marc Zyngier wrote:
>>> As we're about to majorly extend the handling of FGT registers,
>>> restructure the code to actually save/restore the registers
>>> as required. This is made easy thanks to the previous addition
>>> of the EL2 registers, allowing us to use the host context for
>>> this purpose.
>>>
>>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>>> ---
>>>  arch/arm64/include/asm/kvm_arm.h        | 21 ++++++++++
>>>  arch/arm64/kvm/hyp/include/hyp/switch.h | 55 +++++++++++++------------
>>>  2 files changed, 49 insertions(+), 27 deletions(-)
>>>
>>> diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
>>> index 028049b147df..85908aa18908 100644
>>> --- a/arch/arm64/include/asm/kvm_arm.h
>>> +++ b/arch/arm64/include/asm/kvm_arm.h
>>> @@ -333,6 +333,27 @@
>>>  				 BIT(18) |		\
>>>  				 GENMASK(16, 15))
>>>  
>>> +/*
>>> + * FGT register definitions
>>> + *
>>> + * RES0 and polarity masks as of DDI0487J.a, to be updated as needed.
>>> + * We're not using the generated masks as they are usually ahead of
>>> + * the published ARM ARM, which we use as a reference.
>>> + *
>>> + * Once we get to a point where the two describe the same thing, we'll
>>> + * merge the definitions. One day.
>>> + */
>>> +#define __HFGRTR_EL2_RES0	(GENMASK(63, 56) | GENMASK(53, 51))
>>> +#define __HFGRTR_EL2_MASK	GENMASK(49, 0)
>>> +#define __HFGRTR_EL2_nMASK	(GENMASK(55, 54) | BIT(50))
>>> +
>>> +#define __HFGWTR_EL2_RES0	(GENMASK(63, 56) | GENMASK(53, 51) |	\
>>> +				 BIT(46) | BIT(42) | BIT(40) | BIT(28) | \
>>> +				 GENMASK(26, 25) | BIT(21) | BIT(18) |	\
>>> +				 GENMASK(15, 14) | GENMASK(10, 9) | BIT(2))
>>> +#define __HFGWTR_EL2_MASK	GENMASK(49, 0)
>>> +#define __HFGWTR_EL2_nMASK	(GENMASK(55, 54) | BIT(50))
>>> +
>>>  /* Hyp Prefetch Fault Address Register (HPFAR/HDFAR) */
>>>  #define HPFAR_MASK	(~UL(0xf))
>>>  /*
>>> diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
>>> index 4bddb8541bec..9781e79a5127 100644
>>> --- a/arch/arm64/kvm/hyp/include/hyp/switch.h
>>> +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
>>> @@ -70,20 +70,19 @@ static inline void __activate_traps_fpsimd32(struct kvm_vcpu *vcpu)
>>>  	}
>>>  }
>>>  
>>> -static inline bool __hfgxtr_traps_required(void)
>>> -{
>>> -	if (cpus_have_final_cap(ARM64_SME))
>>> -		return true;
>>> -
>>> -	if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
>>> -		return true;
>>>  
>>> -	return false;
>>> -}
>>>  
>>> -static inline void __activate_traps_hfgxtr(void)
>>> +static inline void __activate_traps_hfgxtr(struct kvm_vcpu *vcpu)
>>>  {
>>> +	struct kvm_cpu_context *hctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
>>>  	u64 r_clr = 0, w_clr = 0, r_set = 0, w_set = 0, tmp;
>>> +	u64 r_val, w_val;
>>> +
>>> +	if (!cpus_have_final_cap(ARM64_HAS_FGT))
>>> +		return;
>>> +
>>> +	ctxt_sys_reg(hctxt, HFGRTR_EL2) = read_sysreg_s(SYS_HFGRTR_EL2);
>>> +	ctxt_sys_reg(hctxt, HFGWTR_EL2) = read_sysreg_s(SYS_HFGWTR_EL2);
>>>  
>>>  	if (cpus_have_final_cap(ARM64_SME)) {
>>>  		tmp = HFGxTR_EL2_nSMPRI_EL1_MASK | HFGxTR_EL2_nTPIDR2_EL0_MASK;
>>> @@ -98,26 +97,30 @@ static inline void __activate_traps_hfgxtr(void)
>>>  	if (cpus_have_final_cap(ARM64_WORKAROUND_AMPERE_AC03_CPU_38))
>>>  		w_set |= HFGxTR_EL2_TCR_EL1_MASK;
>>>  
>>> -	sysreg_clear_set_s(SYS_HFGRTR_EL2, r_clr, r_set);
>>> -	sysreg_clear_set_s(SYS_HFGWTR_EL2, w_clr, w_set);
>>> +
>>> +	r_val = __HFGRTR_EL2_nMASK & ~HFGxTR_EL2_nACCDATA_EL1;
>> I don't get why you do
>>
>> & ~HFGxTR_EL2_nACCDATA_EL1 as this latter also has a negative polarity. 
>>
>> Please could you explain what is special with this bit/add a comment?
> Nothing is really special with this bit.
>
> But it is currently always cleared (we blindly write a big fat zero),
> and I wanted to explicitly show all the instructions for which we
> enable trapping for (ACCDATA_EL1 being the only one that is currently
> documented in the ARM ARM, although there are more already).
>
> So the construct I came up with is the above, initialising the
> register value with the nMASK bits (i.e. not trapping the
> corresponding instructions), and then clearing the bit for the stuff
> we want to trap. Maybe adding something like:
>
> /* Default to no trapping anything but ACCDATA_EL1 */
>
> would help?
thank you for the explanation. Yes it does.

Eric
>
> Thanks,
>
> 	M.
>