Re: [PATCH v4 3/3] arm64/kvm: Fine grain _EL2 system registers list that affect nested virtualization

From: Miguel Luis <miguel.luis@oracle.com>
To: Marc Zyngier <maz@kernel.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>,
	Will Deacon <will@kernel.org>,
	Oliver Upton <oliver.upton@linux.dev>,
	James Morse <james.morse@arm.com>,
	Suzuki K Poulose <suzuki.poulose@arm.com>,
	Zenghui Yu <yuzenghui@huawei.com>,
	Eric Auger <eric.auger@redhat.com>,
	Jing Zhang <jingzhangos@google.com>,
	"linux-arm-kernel@lists.infradead.org" 
	<linux-arm-kernel@lists.infradead.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"kvmarm@lists.linux.dev" <kvmarm@lists.linux.dev>
Subject: Re: [PATCH v4 3/3] arm64/kvm: Fine grain _EL2 system registers list that affect nested virtualization
Date: Thu, 19 Oct 2023 14:46:41 +0000	[thread overview]
Message-ID: <92F481A7-7A36-470C-9275-046ACDC958ED@oracle.com> (raw)
In-Reply-To: <86il72n5si.wl-maz@kernel.org>

Hi Marc,

> On 19 Oct 2023, at 12:41, Marc Zyngier <maz@kernel.org> wrote:
> 
> On Mon, 16 Oct 2023 12:17:42 +0100,
> Miguel Luis <miguel.luis@oracle.com> wrote:
>> 
>> Implement a fine grained approach in the _EL2 sysreg ranges.
>> 
>> Fixes: d0fc0a2519a6 ("KVM: arm64: nv: Add trap forwarding for HCR_EL2")
>> Reviewed-by: Eric Auger <eric.auger@redhat.com>
>> Signed-off-by: Miguel Luis <miguel.luis@oracle.com>
>> ---
>> arch/arm64/kvm/emulate-nested.c | 89 ++++++++++++++++++++++++++++++---
>> 1 file changed, 83 insertions(+), 6 deletions(-)
>> 
>> diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c
>> index 9ced1bf0c2b7..3a7d4003fc2b 100644
>> --- a/arch/arm64/kvm/emulate-nested.c
>> +++ b/arch/arm64/kvm/emulate-nested.c
>> @@ -648,15 +648,92 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initconst = {
>> SR_TRAP(SYS_APGAKEYLO_EL1, CGT_HCR_APK),
>> SR_TRAP(SYS_APGAKEYHI_EL1, CGT_HCR_APK),
>> /* All _EL2 registers */
>> - SR_RANGE_TRAP(sys_reg(3, 4, 0, 0, 0),
>> -       sys_reg(3, 4, 3, 15, 7), CGT_HCR_NV),
>> + SR_TRAP(SYS_BRBCR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_VPIDR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_VMPIDR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_SCTLR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_ACTLR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_SCTLR2_EL2, CGT_HCR_NV),
>> + SR_RANGE_TRAP(SYS_HCR_EL2,
>> +       SYS_HCRX_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_SMPRIMAP_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_SMCR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_SDER32_EL2, CGT_HCR_NV),
> 
> No. This is a *secure* register. How could it be trapped?

Ack. Please see below.

> 
>> + SR_RANGE_TRAP(SYS_TTBR0_EL2,
>> +       SYS_TCR2_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_VTTBR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_VTCR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_VNCR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_VSTTBR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_VSTCR_EL2, CGT_HCR_NV),
> 
> Secure registers.

Ack.

> 
>> + SR_TRAP(SYS_DACR32_EL2, CGT_HCR_NV),
> 
> This only exists if EL1 is AArch32 capable. Which contradicts the
> basic principle that we don't support AArch32 with NV. Why would you
> want to forward such a trap?

Ack. Please see below.

> 
>> + SR_RANGE_TRAP(SYS_HDFGRTR_EL2,
>> +       SYS_HAFGRTR_EL2, CGT_HCR_NV),
>> /* Skip the SP_EL1 encoding... */
>> SR_TRAP(SYS_SPSR_EL2, CGT_HCR_NV),
>> SR_TRAP(SYS_ELR_EL2, CGT_HCR_NV),
>> - SR_RANGE_TRAP(sys_reg(3, 4, 4, 1, 1),
>> -       sys_reg(3, 4, 10, 15, 7), CGT_HCR_NV),
>> - SR_RANGE_TRAP(sys_reg(3, 4, 12, 0, 0),
>> -       sys_reg(3, 4, 14, 15, 7), CGT_HCR_NV),
>> + /* SPSR_irq, SPSR_abt, SPSR_und, SPSR_fiq */
>> + SR_RANGE_TRAP(sys_reg(3, 4, 4, 3, 0),
>> +       sys_reg(3, 4, 4, 3, 3), CGT_HCR_NV),
>> + SR_TRAP(SYS_IFSR32_EL2, CGT_HCR_NV),
> 
> Again: AArch32 related register. The spec is very clear that it UNDEFs
> when AArch32 doesn't exist. Even the SPSR_* registers should be
> removed and handled as RES0 without reinjection of the trap.
> 

OK.

>> + SR_TRAP(SYS_AFSR0_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_AFSR1_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_ESR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_VSESR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_FPEXC32_EL2, CGT_HCR_NV),
> 
> AArch32.

Got it.

> 
>> + SR_TRAP(SYS_TFSR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_FAR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_HPFAR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_PMSCR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_MAIR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_AMAIR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_MPAMHCR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_MPAMVPMV_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_MPAM2_EL2, CGT_HCR_NV),
>> + SR_RANGE_TRAP(SYS_MPAMVPM0_EL2,
>> +       SYS_MPAMVPM7_EL2, CGT_HCR_NV),
>> + /*
>> +  * Note that the spec. describes a group of MEC registers
>> +  * whose access should not trap, therefore skip the following:
>> +  * MECID_A0_EL2, MECID_A1_EL2, MECID_P0_EL2,
>> +  * MECID_P1_EL2, MECIDR_EL2, VMECID_A_EL2,
>> +  * VMECID_P_EL2.
>> +  */
>> + SR_RANGE_TRAP(SYS_VBAR_EL2,
>> +       SYS_RMR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_VDISR_EL2, CGT_HCR_NV),
>> + /* ICH_AP0R<m>_EL2 */
>> + SR_RANGE_TRAP(SYS_ICH_AP0R0_EL2,
>> +       SYS_ICH_AP0R3_EL2, CGT_HCR_NV),
>> + /* ICH_AP1R<m>_EL2 */
>> + SR_RANGE_TRAP(SYS_ICH_AP1R0_EL2,
>> +       SYS_ICH_AP1R3_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_ICC_SRE_EL2, CGT_HCR_NV),
>> + SR_RANGE_TRAP(SYS_ICH_HCR_EL2,
>> +       SYS_ICH_EISR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_ICH_ELRSR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_ICH_VMCR_EL2, CGT_HCR_NV),
>> + /* ICH_LR<m>_EL2 */
>> + SR_RANGE_TRAP(SYS_ICH_LR0_EL2,
>> +       SYS_ICH_LR15_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_CONTEXTIDR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_TPIDR_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_SCXTNUM_EL2, CGT_HCR_NV),
>> + /* AMEVCNTVOFF0<n>_EL2, AMEVCNTVOFF1<n>_EL2  */
>> + SR_RANGE_TRAP(SYS_AMEVCNTVOFF0n_EL2(0),
>> +       SYS_AMEVCNTVOFF1n_EL2(15), CGT_HCR_NV),
>> + /* CNT*_EL2 */
>> + SR_TRAP(SYS_CNTVOFF_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_CNTPOFF_EL2, CGT_HCR_NV),
>> + SR_TRAP(SYS_CNTHCTL_EL2, CGT_HCR_NV),
>> + SR_RANGE_TRAP(SYS_CNTHP_TVAL_EL2,
>> +       SYS_CNTHP_CVAL_EL2, CGT_HCR_NV),
>> + SR_RANGE_TRAP(SYS_CNTHV_TVAL_EL2,
>> +       SYS_CNTHV_CVAL_EL2, CGT_HCR_NV),
>> + SR_RANGE_TRAP(SYS_CNTHVS_TVAL_EL2,
>> +       SYS_CNTHVS_CVAL_EL2, CGT_HCR_NV),
>> + SR_RANGE_TRAP(SYS_CNTHPS_TVAL_EL2,
>> +       SYS_CNTHPS_CVAL_EL2, CGT_HCR_NV),
> 
> None of these secure registers can be accessed, and they will UNDEF at
> EL1.
> 

In summary, the refinement on this patch started by considering the
spec statement that register accesses using MRS or MSR with a name
ending in _EL2 but the exceptions stated, should trap. Solely by that
statement this patch would, indeed, include registers ending in _EL2 which
should not be contemplated.

NV won’t support Aarch32, so those Aarch32 registers must be removed and I wasn’t
aware that KVM runs always in Non-secure state so secure registers must also
be removed.

Should I spin v5 ?

Thank you
Miguel

> M.
> 
> -- 
> Without deviation from the norm, progress is not possible.