From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF8FDC6379D for ; Mon, 23 Nov 2020 14:20:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 76BBA2075A for ; Mon, 23 Nov 2020 14:20:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="1rORItQv" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389149AbgKWOUO (ORCPT ); Mon, 23 Nov 2020 09:20:14 -0500 Received: from mail.kernel.org ([198.145.29.99]:44590 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389030AbgKWOUL (ORCPT ); Mon, 23 Nov 2020 09:20:11 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 62C2D20781; Mon, 23 Nov 2020 14:20:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1606141210; bh=xAdNybpsYh2E0+792QH2eMxvVzwUFSgdE96e/bSFE6c=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=1rORItQvoz/zaPOIkqicusx02B492b3DEaRv+XMYId8M2qBmmnkyk7/9BOnkKF9vZ ZtvPJuhbsaeN6DJbOuDeZnzwo+y9/+24Mt8DWBFcXxLwVxIO8SPdKC28awNVXH9eAh AswYJ7dWlspM5DPdwNUg9yPnDn64PkopIsvi9CKw= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=wait-a-minute.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1khChM-00CyLj-AO; Mon, 23 Nov 2020 14:20:08 +0000 Date: Mon, 23 Nov 2020 14:20:07 +0000 Message-ID: <87lfes5f54.wl-maz@kernel.org> From: Marc Zyngier To: David Brazdil Cc: kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, James Morse , Julien Thierry , Suzuki K Poulose , Catalin Marinas , Will Deacon , Dennis Zhou , Tejun Heo , Christoph Lameter , Mark Rutland , Lorenzo Pieralisi , Quentin Perret , Andrew Scull , Andrew Walbran , kernel-team@android.com Subject: Re: [PATCH v2 06/24] kvm: arm64: Move hyp-init params to a per-CPU struct In-Reply-To: <20201116204318.63987-7-dbrazdil@google.com> References: <20201116204318.63987-1-dbrazdil@google.com> <20201116204318.63987-7-dbrazdil@google.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: dbrazdil@google.com, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org, dennis@kernel.org, tj@kernel.org, cl@linux.com, mark.rutland@arm.com, lorenzo.pieralisi@arm.com, qperret@google.com, ascull@google.com, qwandor@google.com, kernel-team@android.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 16 Nov 2020 20:43:00 +0000, David Brazdil wrote: > > Once we start initializing KVM on newly booted cores before the rest of > the kernel, parameters to __do_hyp_init will need to be provided by EL2 > rather than EL1. At that point it will not be possible to pass its four > arguments directly because PSCI_CPU_ON only supports one context > argument. > > Refactor __do_hyp_init to accept its parameters in a struct. This > prepares the code for KVM booting cores as well as removes any limits on > the number of __do_hyp_init arguments. > > Signed-off-by: David Brazdil > --- > arch/arm64/include/asm/kvm_asm.h | 7 +++++++ > arch/arm64/include/asm/kvm_hyp.h | 4 ++++ > arch/arm64/kernel/asm-offsets.c | 4 ++++ > arch/arm64/kvm/arm.c | 26 ++++++++++++++------------ > arch/arm64/kvm/hyp/nvhe/hyp-init.S | 21 ++++++++++----------- > arch/arm64/kvm/hyp/nvhe/hyp-main.c | 2 ++ > 6 files changed, 41 insertions(+), 23 deletions(-) > > diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h > index 54387ccd1ab2..01904e88cead 100644 > --- a/arch/arm64/include/asm/kvm_asm.h > +++ b/arch/arm64/include/asm/kvm_asm.h > @@ -150,6 +150,13 @@ extern void *__vhe_undefined_symbol; > > #endif > > +struct kvm_nvhe_init_params { > + unsigned long tpidr_el2; > + unsigned long vector_hyp_va; > + unsigned long stack_hyp_va; > + phys_addr_t pgd_pa; > +}; > + > /* Translate a kernel address @ptr into its equivalent linear mapping */ > #define kvm_ksym_ref(ptr) \ > ({ \ > diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h > index 6b664de5ec1f..a3289071f3d8 100644 > --- a/arch/arm64/include/asm/kvm_hyp.h > +++ b/arch/arm64/include/asm/kvm_hyp.h > @@ -15,6 +15,10 @@ > DECLARE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt); > DECLARE_PER_CPU(unsigned long, kvm_hyp_vector); > > +#ifdef __KVM_NVHE_HYPERVISOR__ > +DECLARE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); > +#endif I'm not sure we should bother with this #ifdefery. Having the declaration present at all times doesn't really hurt, since it is only defined in the HYP code. Cutting down on the conditionals would certainly help readability. > + > #define read_sysreg_elx(r,nvh,vh) \ > ({ \ > u64 reg; \ > diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c > index 7d32fc959b1a..4435ad8be938 100644 > --- a/arch/arm64/kernel/asm-offsets.c > +++ b/arch/arm64/kernel/asm-offsets.c > @@ -110,6 +110,10 @@ int main(void) > DEFINE(CPU_APGAKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APGAKEYLO_EL1])); > DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu)); > DEFINE(HOST_DATA_CONTEXT, offsetof(struct kvm_host_data, host_ctxt)); > + DEFINE(NVHE_INIT_TPIDR_EL2, offsetof(struct kvm_nvhe_init_params, tpidr_el2)); > + DEFINE(NVHE_INIT_VECTOR_HYP_VA, offsetof(struct kvm_nvhe_init_params, vector_hyp_va)); > + DEFINE(NVHE_INIT_STACK_HYP_VA, offsetof(struct kvm_nvhe_init_params, stack_hyp_va)); > + DEFINE(NVHE_INIT_PGD_PA, offsetof(struct kvm_nvhe_init_params, pgd_pa)); > #endif > #ifdef CONFIG_CPU_PM > DEFINE(CPU_CTX_SP, offsetof(struct cpu_suspend_ctx, sp)); > diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c > index c0ffb019ca8b..4838556920fb 100644 > --- a/arch/arm64/kvm/arm.c > +++ b/arch/arm64/kvm/arm.c > @@ -50,6 +50,7 @@ DECLARE_KVM_HYP_PER_CPU(unsigned long, kvm_hyp_vector); > > static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page); > unsigned long kvm_arm_hyp_percpu_base[NR_CPUS]; > +DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); > > /* The VMID used in the VTTBR */ > static atomic64_t kvm_vmid_gen = ATOMIC64_INIT(1); > @@ -1347,10 +1348,7 @@ static int kvm_map_vectors(void) > > static void cpu_init_hyp_mode(void) > { > - phys_addr_t pgd_ptr; > - unsigned long hyp_stack_ptr; > - unsigned long vector_ptr; > - unsigned long tpidr_el2; > + struct kvm_nvhe_init_params *params = this_cpu_ptr_nvhe_sym(kvm_init_params); > struct arm_smccc_res res; > > /* Switch from the HYP stub to our own HYP init vector */ > @@ -1361,13 +1359,18 @@ static void cpu_init_hyp_mode(void) > * kernel's mapping to the linear mapping, and store it in tpidr_el2 > * so that we can use adr_l to access per-cpu variables in EL2. > */ > - tpidr_el2 = (unsigned long)this_cpu_ptr_nvhe_sym(__per_cpu_start) - > - (unsigned long)kvm_ksym_ref(CHOOSE_NVHE_SYM(__per_cpu_start)); > + params->tpidr_el2 = (unsigned long)this_cpu_ptr_nvhe_sym(__per_cpu_start) - > + (unsigned long)kvm_ksym_ref(CHOOSE_NVHE_SYM(__per_cpu_start)); > > - pgd_ptr = kvm_mmu_get_httbr(); > - hyp_stack_ptr = __this_cpu_read(kvm_arm_hyp_stack_page) + PAGE_SIZE; > - hyp_stack_ptr = kern_hyp_va(hyp_stack_ptr); > - vector_ptr = (unsigned long)kern_hyp_va(kvm_ksym_ref(__kvm_hyp_host_vector)); > + params->vector_hyp_va = (unsigned long)kern_hyp_va(kvm_ksym_ref(__kvm_hyp_host_vector)); > + params->stack_hyp_va = kern_hyp_va(__this_cpu_read(kvm_arm_hyp_stack_page) + PAGE_SIZE); > + params->pgd_pa = kvm_mmu_get_httbr(); Note to self: rename this to kvm_mmu_get_hyp_pgd() (another AArch32-ism). > + > + /* > + * Flush the init params from the data cache because the struct will > + * be read while the MMU is off. > + */ > + __flush_dcache_area(params, sizeof(*params)); nit: please use kvm_flush_dcache_to_poc(), as it clearly indicates to which point we are flushing. Thanks, M. -- Without deviation from the norm, progress is not possible.